What is A/B Testing?

A/B Testing (also known as Split Testing or A/B/n Testing) defines a way to compare two versions of an application or a web page that enables you to determine which one performs better. It is one of the easiest ways to analyze an application or a web page to create a new version. Thereafter, both these versions can be compared to find the conversion rate, which further helps in finding the better performer of these two.

Simply put, it’s a way to figure out how we can get our website users to click that shiny red button instead of yesterday’s green one. Or maybe discover if the green one was the better choice to begin and thus forcing us back to the previous version.

Put in another way, it’s a method for measuring two versions of one element so we can determine which one is more successful. Is the winner version A or version B?

If we have a website or an app, we have activities that we want our users to complete (e.g., make a purchase, sign up for a newsletter) and/or metrics that we want to improve (e.g., revenue, session duration, bounce rate). With A/B Testing, we can test which version of a page results in the greatest improvement in conversions (i.e. completed activities that you measure as goals) or metric value.

A/B testing is a fantastic method for figuring out the best online promotional and marketing strategies for a business. It can be used to test everything from website copy to sales emails to search ads. Well-planned A/B testing can make a huge difference in the effectiveness of the marketing efforts.

The first thing to do when planning an A/B test is to figure out what we want to test. It might be  an on-site test, or an off-site test. Now, what is an on-site test or an off-site test?

Let’s take an example of a site that sells subscription boxes and also the individual products. If we’re running an on-site test, we’ll want to think of all the sales-related pieces of our website, and then figure out which elements we want to test. In this case, it might be the number of subscription boxes sold from the cart and the number of customers ordering individual products.

With off-site tests, we’re probably testing either an ad, or a sales email. In this case, we might also want to test whether users are more comfortable with tapping banner ads or interstitial ads. Testing ad copy to see which ad brings in more converting visitors can help us focus our advertising efforts. Once we know the particular ad is converting as well as possible, it’s easier to justify spending more money and time on it. The same goes for emails. If we send out two versions to our list (randomly selecting which half gets which email), and then track which one converts better, we can send only that version to all the contacts next time.

But before we start testing, we need to have a very clear idea about the results we are looking for. We should already be aware of our baseline results. We want to test option A and B against each other, but we also want to know that whichever one does better in the test is also doing better than your current results. Alternatively, you can use A as your control (whatever you’re currently using) and then use something new for B.

Tests need to be run simultaneously to account for any variations in timing. We cannot test one variation today and the other one tomorrow, because various factors might change between today and tomorrow on which we might not have control. Instead, we need to split the traffic seeing your variations at the same time.

The option A (current live version) is known as the control and the option B (modified version) is known as the variation.

Before we think about optimization and testing, however, solidifying our high-level strategy is required. So the order should be:

1. Defining business objectives
2. Defining website goals
3. Defining Key Performance Indicators
4. Defining target metrics

Case studies of A/B Testing :

Two Magical Words Increased Conversion Rate by 28% –

In this experiment, two simple words increased the sign-ups on this button by 28%, demonstrating the significance of measuring call-to-action buttons. Apparently, even the smallest alterations can have surprisingly effective results. The two magic words in question? “It’s Free!”

Changing a Sign-Up From Green to Red Led to 34% Growth –

Sometimes even the most trivial change to a website can fuel substantial results. A website increased their conversion rate by 34% by changing the color range of their sign-up button from green to red.

Changing the view of the banner image resulted in more conversion rate –

The original page and variations are:

Original

Variation #1: Banner Image

Variation #2: Sub-heading

Variation #3: Button Text

Variation #4: Button Color

This is the landing page of a travel and tourism website. Variation #1 was found to have much more positive response from the audience (even more than the original one) than the rest of the variations. Variation #1 just had a different view of the same banner image.

Image courtesy : Google Analytics

Add-to-Cart Experiment –

A website, decided to test their purchase process by making three variations of an “Add to Cart” button to increase sales. One being icon focused, one with larger text, and one with a simple “+Cart”. The test had a major impact on “Add to Cart” clicks. Variation 2 increased conversion by 15% compared to the original and Variation 1, the larger button with the clear call-to-action text, increased “Add to Cart” clicks by a staggering 49%.

Few tools that can be used for A/B testing :

• Optimizely

Optimizely is one of the used A/B testing platforms around. It is a cost-efficient premium option for entry-level developers. It has an intuitive interface that is easy to implement. It also offers the best-in-class integrations with a code-free visual editor.

• Five Second Test

The Five Second Test tool employs a simple, yet amazing mechanism of showing your design to people for five seconds. After five seconds, those who saw your design will be asked questions about they remembered.

The Five Second Test gives you an insight into a customer’s first impressions of your mobile application.

• Apptimize

Apptimize is a mobile application optimization company that offers A/B testing. It claims to be the most powerful tool around, with the most reliable SDK and robust analytics.

It comes with no coding and no tagging, and it deploys changes immediately. Apptimize lets you change the color, copy, position, images, and navigation items within your app.

• Google Analytics Experiments

Google Analytics is a complete A/B testing platform that utilizes Google’s manifold approach. It allows you to split-test up to 10 versions of each page and delivers a page at random to customers via a separate URL. You can also choose the objective you would like to test and get updates via email on how the experiment is performing.

• Kissmetrics

Kissmetrics is an outstanding A/B testing tool that lets you analyze what is working and what is not working. It prompts customers with the right interaction at the right moment, and it advertises its ability to increase conversion and decrease churn.

There are other tools for AB testing as well like, Adobe Target, AB Tasty, VWO, etc. Which tool to use will depend on the time, requirements and feasibility.