What Is A/B Testing and How Is It Done?

An A/B test is, at its core, an experiment. Visitors are shown a control group alongside two or more variations, and the data collected from their behavior determines which variation performs better.

You can run A/B tests on virtually anything you can think of: websites, mobile apps, ads, and more.

Looking at your digital assets, A/B tests let you experiment with elements such as:

• Copy itself, along with its font and size,
• The color, size, and placement of buttons,
• The position and size of images,
• The dimensions and placement of banners,
• The number of products shown and their color options.

With A/B testing you can test the design or copy of your website or mobile app across endless variations, then serve the version that converts best to all of your visitors.

So how does it work in practice?

Let us start with a few tools that can help you run A/B tests.

What are the main A/B testing tools?

Once you decide to run an A/B test, there are plenty of tools available, both free and paid.

• Google Website Optimizer: A free A/B testing tool offered by Google.
• VWO / Visual Website Optimizer: A paid, easy-to-use tool that, as the name suggests, focuses primarily on visual changes.
• Optimizely: Like VWO, Optimizely is a paid optimization tool.
• Max A/B: A plugin that lets you run tests on WordPress sites.
• Unbounce: A paid tool that lets you build landing pages and run tests on the pages you create.
• Webtrends: Another paid tool you can use for A/B testing.

Beyond these, there are many other testing tools on the market. You can try the demo versions of the paid tools and, once you find one that fits your needs, purchase it to start your conversion optimization work.

Before answering how to run an A/B test, it helps to restate what an A/B test is. It is an observation-based experiment designed to measure which of two possibilities delivers better results. Since the test can be applied to almost anything, there is no real limitation on its scope. Applicable to web design, social media, advertising, marketing, and many other areas, A/B testing has become a highly practical option for measurement and evaluation in recent years. When used to measure website performance, the elements it most frequently tests are:

• Font type and size,
• Design colors, dimensions, and placements,
• Images,
• Banners,
• Products

Which tools are used when running an A/B test?

When you want to run an A/B test, you will find plenty of applications to choose from, both free and paid. Finding a tool is rarely the hard part. So which programs are used most often for this kind of testing? Here are the answers.

Google Website Optimizer: Offered free of charge by Google, this program is used, as the name suggests, to optimize websites, and it delivers solid results.

Visual Website Optimizer: Commonly known as VWO, this is a paid program. Used for the visual optimization of websites, VWO helps polish the visual elements on your pages.

Max A/B: This application, which enables optimization for sites built on WordPress, is also quite capable.

What to watch out for when running an A/B test

If you are going to run an A/B test, there are several points to consider before, during, and after the test. Keep the following in mind:

• Determine the visitor volume of the page being tested and the improvement rate you expect from it accurately.
• Keep the variations live for at least two weeks and make sure visitors are actually seeing them.
• Analyze the test results correctly and compare them against the results reported by your testing tool to evaluate whether the test truly succeeded.

A/B testing checklist

Before the A/B test

• The daily visitor count of the page you will test
• The current conversion rate of the metric you aim to improve
• The estimated improvement percentage you expect from the test
• The number of variations you will use in the test

Make a note of the metrics above before you start any A/B test.

During the A/B test

• Keep your test live for at least two weeks.
• Make sure the tested variations receive enough traffic and conversions.
• Make sure the same visitor is not shown different variations of the test.

After the A/B test

• Analyze the test results.
• Analyze the results from your own analytics tool.
• Compare the results from your A/B testing tool with your analytics tool to confirm the win is real.
• Roll out the winning version on your site to the relevant segment.