AB testing is important to gauge the effectiveness of changes you make to your site. Depending on the type of test you are running, or the different phase of testing you are in, we recommend targeting your tests to different proportions of users. Here are the four different testing “modes” that we recommend for a successful testing strategy, which are now available with Deliver. These modes are Pilot, Split, Supervised and All Traffic: four different ways of allocating the visitor traffic that test your hypotheses. By testing through each of these modes, you will be able to have a more confident testing program which tests your hypotheses at the beginning of the test, proves its effectiveness, and then tracks its progress over time.
Pilot: prototyping your ideas (20%)
Are you considering making a change to your website which is potentially high risk? Perhaps you would like to smooth out the process at the checkout? Or re-design the entire look and feel of the homepage? Or, even, changing the call-to-action buttons on the product pages? The downside of a test like this is potentially significant, and if pushed live could have a negative impact on your site’s performance.
In this situation, you would want to reduce the risk by only showing it to a smaller proportion of users. But you would also want to find out if it is having a statistically catastrophic impact as soon as you possibly could.
This is why we suggest that you use Pilot mode in your traffic allocation where only 1 in 5 users will be exposed to the new variation that you are running. This is what we have determined as the most productive ratio to test whether the hypothesis having a big impact, in the shortest amount of time. Once you have established that your test has no major negative impact, you can then move to Split allocation (where 50% of the traffic see the test).
Split: testing campaigns (50%)
This is the standard A/B test mode. The one you all know and probably use for most of your tests. This test mode should be used whenever you are making changes to your site. A 50/50 split means that you can evaluate the effectiveness of the change quickly and can reach the results of the test as quickly as possible. Here you would be able to clearly determine the true impact of the test - positive, negative and by what level of magnitude it affects your visitors. If you had a test running in pilot mode that did not lead to a large negative effect, this is where you would determine its actual impact on your visitors.
Supervised: validate your tests (95%)
Once the split test is seen as successful, we would recommend that you use supervised mode. This is where you show the test to 95% of the users, but hide it from 5% (who act as the control). This is to ensure that if the test becomes unsuccessful in the long term, you can spot the change as soon as possible and reduce possible damage. It also acts as a useful point of reference, to prove its value and measure the uplift in conversions.
All traffic: seasonal flash campaign (100%)
Sometimes the changes you want to make to your site are relevant to every visitor. For example, a flash notification, or a service update message. This mode should be used with caution and we recommend only using these if the changes are absolutely necessary. As there is no control in the test, you cannot determine the effectiveness of the test. If a test that won in Split mode was to be pushed to 100% traffic, there would be no way of knowing if it was continuing to drive an uplift. We have seen in the past that winning tests only have an impact during certain seasons. Tests that were successful for a time, can become non effective, or at worst damaging to conversions, which these mode would be blind to.
By devising a testing strategy that develops over time, using our recommended workflow, not only will you be able to gauge a more accurate sense of what your visitors actually want on your site, you will also be able to avoid the risks attached to testing on sites. Use this workflow to make your testing more robust!
If you want to learn more check out our indepth FAQ.