Experimentation in Design: How To Make Data-Driven Decisions

Experimentation in design helps provide information for more assertive decision-making, so designers can continuously upgrade products and the overall user experience.

But instead of relying on their intuition to solve a problem or improve a specific feature, product teams can test different options through continuous experimentation. This way, designers can find out what is working and what needs to change to iterate in the next round of experimentation.

This article will explain what experimentation in design is and provide you with a simple framework so you can start running your own investigations.

What is experimentation?

Experimentation is about testing and experimenting to validate a hypothesis.

This is why experimentation is strongly linked to science and scientific methodology. Because of experimentation, hypotheses can be validated or not, discoveries are made, and advances in science are possible.

The scientific method of iterative experiments ensures that product teams make data-driven decisions about how the product and the user experience should evolve.

The scientific method

UX relies on the scientific method to investigate and test hypotheses to answer questions that help designers solve UX problems.

If you still have questions about the scientific basis of UX, check the article: UX Design: Art or Science?

Experimentation in Product Design

Experimentation in Product Design brings the scientific method into the development processes of digital products.

The idea is to test (experiment) different hypotheses that can improve the product in question.

You can contemplate minor design tweaks, such as buttons, colors, and typography, or modify entire journeys, new features, and functions.

The test results will provide information, validating or not the hypotheses and, thus, supporting decision-making.

Experimentation in this process is essential because it avoids spending a large part of your budget on a feature that might prove to be a dead end. And it prevents your team from a lot of rework and frustration.

The 3 principles of experimentation in Design

There are 3 principles that guide experimentation properly:

1. Scientific method: without the scientific method, it is impossible to run UX Design experiments. The scientific method is the basis for transparent, efficient, result-oriented, and bias-free experimentation;
2. Prioritization: during the experimentation process, it is normal to come up with several questions and hypotheses to be answered and validated. Therefore, it is important to understand what the project's priorities are, defining which are the most relevant hypotheses to be tested;
3. Iteration: iteration is what will transform the experimentation process into a system. Iterate means doing something again and again. In the case of experimentation, it means taking the results, learning from them, and defining new questions and hypotheses to be tested in future experiments.

Why should you run an experiment?

Experimentation is important to validate hypotheses about the product and how people use it.

But going further, working with experimentation allows significant advantages and benefits for the project and the business. Some of them are:

Identify and measure impacts of changes: with several factors influencing KPIs and business results, it can be complex to understand which actions had the greatest impact.

So, experimentation helps track and measure which changes had the most significant impact and which indicators. This is because the experimentation process is controlled, so you know precisely what is being tested and the effects of that specific experiment.

Constant and relevant product updates: implementing a culture of experimentation allows for continuous and relevant updates to the product, keeping it competitive and providing good experiences.

Mitigate risks: testing and experimentation mitigate risks because it is done in a controlled environment and on an evaluation basis. This way, there is room for mistakes and lessons.

If you launch a product – or an update – without prior testing, you're not sure of what is being launched and the impact on the business and the people using it. This lack of predictability can be very harmful to the company.

The elements behind proper experimentation

In addition to the principles of the experimentation process, some fundamental elements make up this process:

• Problem: without a problem to investigate or solve, there is no experimentation. For this reason, conduct proper research before defining the issue you'll be addressing in the experimentation process;
• Users: it is essential to know who the users are so that experimentation is directed to the target audience. Without your users, there is no way to conduct the necessary tests;
• Features and benefits: to experiment, it is crucial to have something to test besides a problem to solve. In this sense, know which features will be tested and what benefit they propose to users;
• Metrics for analysis: experimentation has a goal, and to measure the success of this goal, you need to know the metrics to analyze the tests and their results. Testing without knowing how to measure the results is like being in a dark room.

Besides these specific elements, three other factors influence a successful experimentation process. They are:

• Realistic objectives: working with well-defined and realistic goals is essential to measure the experimentation and validate the hypotheses. In this sense, the more specific and detailed the objective, the better.
• Spread the word: team support is fundamental for successful experimentation. Therefore, talk to your team, peers, stakeholders, and directors to align the importance of experimentation and thus start building a culture around this process.
• Build processes: part of the job is to make it official and build strategies, so everyone learns and knows how to run experiments. Document the steps, create a glossary, research and define tools, and train people.

How to conduct experimentation in Design?

We can summarize an experimentation framework in 4 major steps:

1. Ask a question (define the problem);
2. Create a hypothesis (possible solutions);
3. Run the experiment and analyze results;
4. Communicate the results.

1) Ask a question

One of the foundations of experimentation is to have a question to be answered. However, defining a query may be more complex.

It is not enough to ask a question. We have to ask the right ones. It is not about how many questions either, but the quality of them.

Eventually, within an iterative cycle of experimentation, you will be able to test several hypotheses for different questions. Therefore, it is worth thinking about the right ones to ask.

The first step is to understand what area you want to investigate about the product, the context, and the reasons for you to run an experiment.

Then you can refine your question with the 5 W's technique:

• What is the problem?
• Who is impacted by the problem?
• Where does the problem occur?
• When does the problem occur?
• Why does the problem happen?

From this, you can build the right questions to guide the experimentation and bring important information to the table for decision-making.

This first step of the experimentation process is really not simple. To help you better understand how to identify a problem and form a question, we suggest reading this article:

• Problem Investigation In UX Design

2) Create a hypothesis

Before understanding how to create a hypothesis, it is worth first explaining what a hypothesis is.

A hypothesis is a statement that can be tested based on the relationship between two variables.

For example: If we increase the size of the sign-up button, then the number of people signing up will double.

Notice that the hypothesis above states a causal relationship between the variables. If we change the button, we will get more people signed up. More importantly, you can test this hypothesis.

Going further, we can add a reason for the hypothesis as justification.

For example: If we increase the subscribe button size, the number of people will double because it will attract more attention.

This is just an example of a hypothesis created from a question: "How do we improve our number of signups?"

But from one question, we can create several hypotheses to test. So, in addition to knowing how to develop hypotheses, it is also essential to know how to prioritize them.

While there is no rule about how to prioritize hypotheses, you can think about the following factors:

• Impact: Which hypotheses will impact the business most if validated or invalidated?
• Evidence: what other evidence do you have that supports your hypothesis?

This way, you can make a more thorough test based on solid results from the research stage.

What to consider when creating hypotheses?

It is important to remember that hypotheses will lead to experimentation, which in turn is tied to a central goal – and usually involves business metrics.

In this sense, when creating hypotheses, be sure to consider some important factors:

Which business metric/outcome indicator should serve as a guide? Which KPI should be improved?

What benefits should the people who use the software be taken into consideration?

What habits and behaviors of the people using the product need to be changed when interacting with it?

How do the interface design elements influence the users and their behavior when interacting with the product?

Reading tip: KPIs in UX Design: Why Do We Need Them?

3) Experiment and analyze

Once the hypotheses are created, the next step is to do the actual experimentation.

Once you know what you are going to test, who are the users for the test, and what metrics are used to measure the result, choose the best method to run the experiment.

For this, there are some standard procedures:

• A/B testing: two versions are made, the existing page/screen/feature is exposed to the control group, and a new experimental version is exposed to the experiment group. The idea is to understand if the experimentation version will perform better than the control version;
• Multi-variable testing: is similar to A/B testing but includes more variables, that is, more versions for experimentation;
• Funnel experimentation: a more complex A/B test where you test different journeys through multiple web pages/screens.

No matter which method you use, some additional factors are important for any experimentation:

• Ensure that the entire group (control and experiment) takes part in the experimentation;
• Set parameters to tracking results before the test, i.e., plan how you will collect the information from the experimentation;
• Trust the results: we often have expectations about the results, but regardless if they turn out to be different from what you first thought, trust the results and make your decisions based on them.

Also, since experimentation is a test, consider not investing too much money or effort in your chosen methods. In that sense, keep in mind some models, such as:

• MVP: is the minimum viable product, built with minimal resources to work and be tested;
• Low/medium fidelity prototypes consume less budget, effort, and time. For experimentation, they are great for getting hypotheses validated or invalidated.

How to evaluate and measure the experimentation results?

Overall, the experiment results are quantitative and should not be difficult to analyze or interpret.

If a larger subscribe button (experiment group) captured twice as many subscriptions as the control group (smaller button/current version), we have a validated hypothesis.

Nevertheless, don't forget to consider the statistical significance, which is the probability that the results did not happen by chance or coincidence.

Best practices indicate that we should consider a level of statistical significance equal to or greater than 95%. This means we should assume that 5% of the results could have been merely the result of randomness.

We also talk about statistical significance in this article:

• UX Survey: How To Collect Trustful Data

What can affect the experimentation results?

It is also essential to consider the elements that can affect the results of your experimentation. Therefore, beware of the following:

• Results "contaminated" by other experimentations running in parallel: ideally, the best way is to run one investigation at a time, to avoid this situation;
• Not enough participants: a small audience or sample can provide biased results and make them inconclusive;
• Concluding the experiment too soon: stopping the investigation earlier than ideal can harm the results, compromising the data;
• External factors: a pandemic, for example, can affect the results of your experiments.

4) Communicate the results

After running and analyzing the results, the final step of the process is to consolidate the information, communicate to stakeholders, and outline action plans for the next steps.

At this point, it is essential to be transparent while communicating the results. Sometimes, the results are not exactly what you were expecting. But this is no reason for frustration because that's what experimentation is all about. Testing, validating, invalidating, and making the best decisions.

So prepare a presentation with the information, the methods, and the results that have been collected. Schedule a meeting and present the findings.

Don't forget to create an action plan from the results obtained. Either a timeline to implement the changes for real, or even redefine the objectives and test other hypotheses.

After that, it is important to celebrate! The experimentation process is not easy, so it is important to celebrate these victories and achievements with the team.

Don't forget that the experimentation process is part of a mindset of constant change and improvement. So when you finish one experiment, you can prepare for the next one!