A/B testing Q&A

With giosg A/B tests, you can compare different actions and interactions to each other, and see which one performs best.

What is giosg A/B testing?

A/B testing enables you to test what works and what doesn’t work for your visitors. With giosg A/B tests, you can compare different actions and interactions to each other, and see which one performs best.

What can I test with giosg A/B testing?

You can test any actions. If you are familiar with giosg Rules, you have already used actions. For example, you might want to find the best autosuggest message by comparing different wordings or you might want to compare different versions of a banner. You can also compare totally different interactions, like whether you should show a chatbot or rather a lead form to get most leads.

How do I start with giosg A/B testing?

First, you need a goal. What is it you want to improve? In other words, what should be the goal for your A/B test? Common examples would be getting more sales, more leads, more bookings, or more visitors to a certain page. When you have figured out what you are after, configure goal tracking in giosg. Read more about creating goals in our guide to goals.

Second, you need an idea how you plan to reach that goal. What could you do marketing-wise to get more of what you want? This is your A/B testing hypothesis. Often, these actions are small and incremental - trying a new chat button text, a different visualisation for a banner, or a new flow for a chatbot. Sometimes, you might want to try something completely new. These will be the actions you are testing in your test.

Third, you need enough traffic on your site to split the traffic between the different alternatives. If you have very little traffic, or you have chosen an action which triggers very rarely, the results are often ambiguous.

What is an alternative?

Alternative is what we call each version being tested. Alternatives are visualised as branches in your test setup.

What is the Control alternative?

The alternatives you have set up in your A/B test are all compared to the so-called Control alternative. In other words, all the alternatives are competing against the Control alternative.

Often, the Control alternative is the “status quo” of your website. In that case, you should not set any action to the alternative.

Sometimes, you already know you want to do something, but want data on what the best approach could be. In that case, you should set an action also to the Control alternative. For example, you already know you want to show a banner, but want to test a blue banner, red banner, and green banner. Define your favourite banner to be the control alternative, and get the results of the others relative to that.

How many different alternatives can I test simultaneously?

You can test an unlimited number of alternatives. But bear in mind that the more alternatives you have, the smaller your traffic is split and the longer it takes to get statistically significant results. If you split the traffic too small, the test will never reach reliable results.

What is traffic distribution?

Traffic distribution is the amount of visitors an alternative gets. By default, we split the traffic equally across your alternatives. Depending on your testing hypothesis, you might want to adjust them. If you are doing something risky, just put a small share of the traffic to that alternative. If you already are convinced that your alternatives will work well, you can drive more traffic to them.

Why isn’t the traffic split exactly like I configured in the traffic distribution?

To avoid any bias, we assign visitors to testing groups as soon as they arrive on your site. Especially if you are using conditions that filter your traffic a lot, the real split of the visitors you see on the A/B test report might not exactly match your configurations.

Does a visitor always see the same alternative during an A/B test?

If we can recognise them as the same visitor, we keep giving them the same experience every time they visit the site. This depends on the cookies: if a visitor is using the same device and has not cleared cookies, we can keep treating them the same way. But if they arrive on site on a different device, or if they have cleared cookies, they will be randomly assigned to an alternative.

What are rooms and conditions?

Rooms and conditions tell the A/B test which of your visitors the test should be targeted to. For example, you can target the test only to visitors from a certain country, or only to visitors who have put something in their shopping basket. Or, you might already have filtered your traffic into custom rooms, and want to run your test only in certain room(s). Read more about custom rooms here.

By using rooms and conditions, you can make sure your test only runs to visitor segments for whom it is relevant.

Should I always use rooms and conditions in my A/B testing?

Not necessarily. Conditions and rooms filter the traffic. If you want to test with all your visitors, you should not set conditions, and you should run the test in all rooms.

Why can’t I create an A/B test for different conditions?

Conditions are filters which affect your traffic. If you would use different conditions for different alternatives, the resulting visitor groups would not be random samples from the same base population, and hence the result you get would no longer be reliable.

How do I know if my findings are valid?

We use a Bayesian method to estimate the probability that the conversion rate of an alternative is better than the conversion rate of the control alternative. This probability is reported as confidence in the A/B test report. Anything over 95 % confidence is usually considered standard for statistical significance. This means that the conversion rate of the group under investigation is higher than the conversion rate with 95 % probability.

How long should my A/B test run?

Depends a lot on the amount of traffic in the test. On a big site, running an A/B test only for a day might be enough to get reliable results. More often, a few weeks should yield enough data to yield good results.

Can I stop my A/B test as soon as I see good results?

We would advise against stopping the A/B test before you have planned to, to avoid stepping into the pit of false discovery. If you stop the test as soon as you see results you like, it might lead to wrong conclusions - maybe the effect was only temporary but vanishes over a longer time.

I don’t want to run my A/B test any more. Where can I stop it?

You can stop your A/B test by setting the end date of the test period to the current day. 

Can I extend the test period of my test if the results are not significant?

Yes, it is safe to do so, and in most cases extending the test period (in other words, giving it more visitors) helps make the results more reliable. While this strictly speaking violates some assumptions of statistical tests, the violation is small. To extend your test, simply give your test a new end date. 

Why can’t I edit the settings of my A/B test?

Changing things in your A/B test changes it in ways which would render the results unreliable. Therefore, you can edit all settings of your test when it has not yet been running. If your test is already running, you can only change the traffic distribution, and the name of the test and the names of the alternatives. If you want to test something different, just create a new test.

How is A/B testing related to Rules?

A/B tests operate using giosg Rules, and they have the same actions and conditions. Creating an A/B test will create the necessary rules automatically. You will see the A/B test rules also in giosg Rules, with label "Experiment", to keep track of everything that’s happening on your site. 

Why can’t I edit the rules related to an A/B test in the Rule settings?

We thought it would be best to have an overview of your test and not edit the rules individually. There are also things you should not touch while the test is running. Therefore, editing the test settings and editing the related rules can only be done in giosg A/B tests.