Two identical doors in a concrete wall, the right door painted blue

Website A/B Testing: How to Validate Hypotheses After a UX Audit

A usability audit ends with a list: people abandon the form here, they miss the price there, on mobile they can't find the button. Every item on that list is a hypothesis, not a proven fact. Some fixes will genuinely bring in more leads, others will change nothing, and a few may even hurt.

This article is the second half of the audit conversation. The audit produces hypotheses; website A/B testing checks which of them really increase conversions. We'll cover when a split test is worth running and when traffic is too low and you should simply ship, how to frame a hypothesis, how long to run a test, and the mistakes that most often ruin the result.

What an A/B test is and why it matters after an audit

An A/B test is an experiment in which visitors are randomly split into two groups. One sees the current version of the page (A), the other sees the modified version (B). Both groups exist in the same period, with the same ads and the same season, so any difference in results can be attributed to the change itself.

Without a test, judging a change is harder. If you redesign a form and see more leads a month later, you can't tell what worked: the new form, a strong ad campaign, the start of peak season, or a competitor leaving the market. A before-and-after comparison mixes all of these factors together. Testing hypotheses with an A/B test separates them.

That's especially useful after an audit. Audit findings rely on session recordings, click maps and tests with five to seven users. They explain well why people leave, but they don't tell you how much conversions will grow once the issue is fixed. An A/B test answers that question with numbers.

When you need an A/B test and when you should just ship

The main constraint on A/B testing is traffic. To tell a real effect from noise, you need a large enough sample: plenty of visitors and, above all, plenty of conversions in each group. If your site gets a handful of leads a week, a difference of one or two leads between variants means nothing. Chance could produce it just as easily.

Statistical significance in plain terms

Significance answers one question: how likely are you to see a difference this large if the variants are actually identical? A result is usually considered reliable when that probability is low, typically with the threshold set at 5%. The smaller the effect you want to detect and the lower your baseline conversion rate, the more visitors you need to spot it.

The required sample size is calculated before launch. You enter your current conversion rate, the minimum effect that matters to the business and the significance level into a calculator. It returns the number of visitors per variant. Divide that by the page's daily traffic, and you'll know whether the test takes two weeks or six months.

What to do with low traffic

For small sites and B2B companies with infrequent leads, a different approach works. Obvious issues from the audit, like a broken phone input mask, unnecessary fields or a hidden price, get fixed right away without a test because the damage is clear. Debatable changes are tested with users on a prototype before development. After release, you compare the same reports and funnels in Google Analytics across comparable periods, keeping in mind that this comparison is less precise.

Another technique is to test not the final lead but a more frequent intermediate action: adding to cart, starting to fill out a form, tapping the phone number. These events happen more often, so the sample builds faster. But the conclusion then applies to that step only, not to final sales.

How to frame a hypothesis

A weak hypothesis sounds like "let's make the button green and see what happens." A strong one links the change, the expected result and the reason. A handy template: "If we change X, metric Y will increase because Z," where Z comes from audit data.

Example: "If we remove the Company field from the lead form, the share of submitted forms will increase, because form analytics show a noticeable portion of visitors abandon at that field, and session recordings show mobile users skipping it and leaving." A hypothesis like this is easy to test and easy to explain to the team, even if it doesn't hold up.

One metric and test duration

Before launch, lock in one primary metric: the one the change is meant to move. Usually it's a conversion event in your analytics: a form submission, a completed order, a click to a messaging app. You can look at secondary metrics, but the decision is made on the primary one. Otherwise, among a dozen numbers there will always be one that "went up," and any variant can be declared the winner.

It also helps to add a guardrail metric, something that must not get worse. A new form might produce more leads, for instance, but with a higher share of junk inquiries. In that case, check lead quality in the CRM, not just the number of submitted forms.

Duration comes from the sample size, but there's also a floor: at least one full week, ideally two. Behavior on weekdays and weekends differs noticeably, especially in B2B, and a test that covers only part of a week will paint a distorted picture.

Decide on a test using one metric chosen before launch. Otherwise, any variant can be declared the winner.

How to run an A/B test, step by step

  1. Check your analytics. Conversion events in Google Analytics are set up and fire correctly on every device. Without this, the test will be comparing noise.
  2. Pick a hypothesis from the audit list. Start with one where a strong impact on leads meets enough traffic on the page.
  3. Calculate sample size and duration. Work out the number of visitors per variant and convert it into weeks at current traffic.
  4. Build variant B. Make the change in design and code, check it on mobile and in major browsers, and make sure the page hasn't gotten slower.
  5. Set up the experiment. Dedicated platforms such as VWO, Optimizely or the open-source GrowthBook handle this and connect to your analytics. Google Optimize, once the go-to option, was shut down by Google in 2023. You can also run the test in-house: the site splits visitors itself and passes the variant ID to analytics.
  6. Launch and leave it alone. Don't change ads, prices or the page itself until the planned duration is over.
  7. Wrap up. Compare the variants on the primary metric, check significance and the guardrail metric, and write down the conclusion.

Common A/B testing mistakes

Other mistakes are just as common:

What to test first

The priorities are the same as after an audit: first, the steps every customer goes through and the points in the funnel with the steepest drop-off. A change on the checkout page affects every buyer, while a tweak on the About page reaches a small share of visitors.

Details like button color or corner radius rarely produce a noticeable effect, and on modest traffic the difference simply won't show up. Starting with them means spending weeks on a test that will almost certainly end in a draw.

What to do after the test

A test has three possible outcomes, and each one is useful. If variant B wins by a significant margin, roll it out to everyone and remove the test logic from the code. If it loses, revert to the original and note why the hypothesis didn't hold. If there's no difference, you can still ship the change for other reasons, say because it simplifies maintenance, or drop it.

Keep results in a single log: hypothesis, duration, metric, outcome, takeaway. After a few months, that log becomes your most valuable knowledge about your customers: what works on them and what doesn't. New hypotheses for the next round come from the same audit, a fresh look at funnels, and feedback from the sales team.

A/B testing is an ongoing part of working on a website after launch, not a separate project. Sphere is a digital agency that handles custom website development from research to launch and beyond: we analyze your analytics data, design the changes, build the variant in design and code, and set up tracking so results show up in the numbers. See examples in our web development projects.

FAQ

How much traffic do you need for an A/B test?

There's no fixed threshold: the required volume depends on your current conversion rate and the size of the difference you want to detect. The rarer the target action and the smaller the expected effect, the more visitors you need. Before launching, calculate the sample size with a calculator and estimate how many weeks the test will take at your traffic. If it comes out to months, ship the change and compare periods instead.

What's the difference between an A/B test and a split test?

Often they're synonyms. Sometimes "split test" refers to comparing two separate versions of a page at different URLs, while "A/B test" means changing one element on the same page. The principle is the same: traffic is split randomly, and variants are compared on one metric over the same period.

Can you run several A/B tests at the same time?

Yes, as long as they run on different pages or in different flows and don't affect the same metric. Two simultaneous tests on the same lead form will blend their effects, and you won't be able to honestly attribute the result to either change.

What if the test shows no difference?

That's a result too: the change doesn't noticeably affect leads, so you can ship it for other reasons or drop it. Log the takeaway and move on to the next hypothesis. Sometimes it's worth testing a bolder version of the same change, since a subtle tweak may simply not produce an effect visible at your traffic level.

Let’s discuss your project
Related reading