5 Most Common A/B Testing Mistakes and Tips on How to Avoid Them

5 Most Common A/B Testing Mistakes and Tips on How to Avoid Them

A/B Testing

Interested in improving your conversion rate? Of course you are. Everyone is. The trouble is, not everyone has a clue how to go about it. The solution is really obvious: A/B testing. It’s efficient, cost-effective, and in the end, you will wind up with an answer to what works best for your website. However, the reason why A/B testing is not more popular, despite all the benefits it has to offer, is because useful tests are hard to put together, and it’s perhaps even harder to figure out what to make of the results, due to so many factors you need to take into account.

Then there is the way you interpret the results, which is where most people go wrong by oversimplifying things. Fortunately, we have teamed up with experts from BestEssays  in order to help you become a master of A/B testing. According to their chief marketing officer, John Hammond, A/B testing makes up for much of their success:

“A/B testing is a powerful tool, yet it doesn’t help as many people as it should, due to their lack of knowledge on how it works. And rest assured, it does work.”

This is why we have put together a list of 5 most common A/B testing mistakes, and advice on how to avoid them.

1. A/B Tests Are Ended Too Soon

Probably the most common mistake related to A/B testing. Whether you are short on time, or in favor of a particular option, you will be tempted to end the test early. According to research conducted by SumoMe, the higher the percentage of statistical significance, the lesser probability that the test result is coincidence. In other words, the longer you test, the lower the change the result is a fluke. This means that cutting your test short at 50% is pretty much pointless, because it doesn’t really tell you anything.

And while 75 or 80 percent is way better, even that is too early to call it quits, since the sample is still not large enough, as can be seen in this study conducted by CXL. The experts in the field maintain that you should end the test at 95%! This means there is only a 5% chance that the result of the test is random.

2. Tests Are Conducted without a Hypothesis

The whole point of testing is to gather data and numbers, which will help you learn if the solution you are proposing actually works. The best way to test would be to treat testing as a scientific experiment, which it certainly is, and base it around a hypothesis. Hypothesis is a proposed statement made on the basis of limited evidence that can be proved or disproved and is used as a starting point for further investigation. Performing a completely random test will certainly produce results, but your goal is not just that. You also need to learn how and why something has worked, so that you can fully understand it and apply it further down the road.

Of course, you won’t have all the data you want when you are putting together your hypothesis right away, but that’s what you are testing for: to gather more data and evidence. That way, you can figure out what, why, and how something works, determine its relevance, and ultimately, come up with better tests and hypotheses.

3. Ignoring Failed Tests

Your test hasn’t produced a lift? Good, that means you should keep doing further tests. According CXL, they often perform as much as 6 different tests on the same page before they receive the kind of lift they were expecting. Instead of going and performing a new test on a different page, you should focus on analyzing its data, and improving your theory, because there is a high chance that your theory is flawed, and not the test. Tweak it, and then run the test again. Lather, rinse, repeat.

Another reason why your test has failed is because of the human nature of your audience. Sometimes, it’s hard to predict how your readers will behave. Plus, there are simply too many variables, which is why it’s almost impossible to know something in advance with 100% certainty. Each new test, even a failed one, serves as a piece of the puzzle, as long as you interpret the data constructively.  

4. Test Involving Overlapping Traffic

Running multiple test on several different pages at once may seem like a clever way to cut down on the costs and time, but there are pitfalls involved. For instance, if the pages you are trying to test simultaneously have overlapping traffic, the tests will yield results which can be misleading or inconclusive. In other words, you will operate under a false assumption that the data gathered through this type of testing is accurate.

So what can you do? Well, you need to make sure that you have gotten the traffic distribution right. This means ensuring a 50/50 traffic split between pages A and B, as well as C and D. The same goes for any other page involved in the test.

5. Not Testing Regularly

If your test has been successful and you have managed to improve your conversion rate, the last thing you should do is stop testing. It’s all the more reason to keep on doing tests. Just because a particular solution has worked now, doesn’t mean it will keep on working forever, and tests can help you keep an eye on the changes. Also, by doing more tests, you will be able hone your skills, come up with more sound hypotheses, more effective tests, and improve your conversion rates even more. Finally, you will gain a better insight into how your audience thinks and acts, which is essential for every successful marketing campaign.


While testing is not without its challenges, it should be the cornerstone of your marketing efforts, because it allows you to assume more control over running your own business, instead of merely reacting to trends and your audience’s responses. Sure, it’s hard work, but it will definitely pay off.

Post to Twitter Post to Facebook

Leave a Reply

Your email address will not be published. Required fields are marked *


About WossJr

Sasha Woss is a content marketer at BestEssays and passionate guest blogger. When she's not researching ways to make you better at digital marketing, she finds herself at foreign countries or cooking classes. Follow Sasha on Facebook and Twitter.