Blog

How do a SaaS Startup use A/B testing to increase conversions?

8 Jun, 2020
Pattern
How do a Subscription Management Startup use A/B testing to increase conversions?

Since you are here you have probably heard about A/B testing before, and perhaps maybe you already understand the context about how it works. Nevertheless, this blog post is for you, to help you understand how your company can benefit from it. Below is your total guide to understanding what A/B testing is, what to expect, and how to start working on it.

What is A/B testing and how can it help your business?

Easily explained A/B testing is a way to further develop and improve your website. The only thing you need to get started is an idea, one idea about design or user experience change that could improve the results of the website. It might be a larger CTA button (Call To Action) or maybe the checkout cart should be green instead of red. The A/B test is the perfect way to base your decisions on facts and not assumptions. If you already feel that your website is optimized, why not simply run two different campaigns to see which one performs best.

Let Substly keep track of your subscriptions

Substly is your go-to subscription management tool. Stop spending money on subscriptions not being used.

Try for free

So instead of making a decision, publishing the change, and hoping for the best, you do so-called A/B test (also known as a split test). When you got your idea and are ready to go you launch your changes on the website and then send 50% of the traffic to the new version and 50% to the original. By doing so, you can analyze the end results and see if your changes had a positive or negative effect. Either way, you’ll be able to learn from the A/B test even if you get a false-positive (in this case a false-positive is when you have performed a test that doesn’t show improved goal conversion, but then you’ve learned that this idea can be excluded). If you need to, you can try out more than one change at a time, however, we recommend not to try more then three things simultaneously.

Why is continuous A/B testing so important?

When you work continuously with A/B testing you say that you get a compound interest also know as the interest-on-interest effect that increases with every conversion. What this means is that when you increase 2% on a test, this is added to the already existing conversion rate. The graph below shows how a conversion increase of 3% per month can double the amount of conversion during a two year period. Companies like Amazon, Airbnb & Facebook have announced that they’ve had a lot of success in the number of A/B tests that’s been performed over the years.

Start working with A/B testing

Now that you know what an A/B test is and why you should include it in your work, we will move on with how to start working on it with only 3 simple steps.

Step 1: The hypothesis

As I’ve mentioned before, all A/B tests start with having an idea of ??an improvement that you want to test to see if it is correct or not. It can be something as simple as making the USPs clearer under the purchase button. So in this case, the hypothesis might be the following, “By making USPs (ex. Free Shipping, 1-2 days delivery, Safe E-commerce) clearer, the result will be that more customers go through with their purchases.”

Step 2: Create different versions

Now it’s time to create one or two different versions that you want to try out. Here you need a person with some UX & programming knowledge.

Step 3: A/B testing

Finally, the test is ready to run, and depending on the amount of traffic at your website, you should discuss and pin down the number of weeks required to get a 100% fair result. If you find it hard to figure out the right amount of time to run an A/B test, a ground-rule to try it out for a minimum of 14 days but it’s even better if you got the opportunity to try it during an entire month. When the test is complete, you hopefully get a conversion increase on the new version which then tells you that your hypothesis was correct.

Most common mistakes in A/B testing of a website

When considering improvements and hypotheses for an A/B test, there are a few things to avoid. To fully succeed and get the best results out of your A/B test I really recommend you follow these simple but important steps below.

Test several things at the same time – in the same test

The most common mistake is making several changes to their A/B tests. The problem then becomes to find out which of these changes actually led to the number of increased conversions or that might have had a negative input. An example is that you decide to change USPs in both the menu and under the payment button on the purchase page. When you, later on, get the results you might not be able to tell which of them that worked well or perhaps did not work at all. Instead, divide your hypothesis in two so that it becomes two different tests. That way, by running them after each other, you know exactly how the different versions have performed.

Test several things at the same time in different tests

For the same reasons as I listed above, it’s best practice to only run one A/B test at a time even if you followed the step above and divided your hypothesis. This way you minimize the chances of getting an incorrect or not reliable result.

Run too many variations

The more versions you try the longer the test needs to be run in order for the results to be accurate. Therefore, you should consider how many different versions you really want to test to make the result as safe as possible.

Useful tools for A/B testing

Last, there’s a couple of different tools you could use when working with A/B testing on your website. But most of the time, Google’s tool: Google Optimize is more than enough. It’s 100% free and easy to understand. If you still want to try something different I would really recommend Optimizely and Sitegainer. They both charge a small fee but is definitely a very good tool that’s worth trying out.


As the subscription economy is booming and spending on subscription services increases each year the only question is how businesses will take advantage of this – or fall victim to it. When you need to get the most out of your company subscriptions, Substly is here to support you. Substly is perfect for small and medium-sized companies that want to improve their subscription and user management.

Ready to spend less on subscriptions?

Try for free Contact sales