How to Run an A/B Test: An Eight-Step Guide With Sample Size Formula and Common Mistakes

An A/B test compares two versions of a page, email or feature by showing each to a random share of visitors and measuring a single outcome, such as sign-ups or purchases. Done well, it replaces opinion with evidence. Done badly, it produces false winners that quietly lower conversion.

The statistics behind it are standard. The NIST engineering statistics handbook covers comparing two proportions and the logic of hypothesis tests, which are the foundation of every testing tool. This guide turns them into an eight-step process and shows where tools from our A/B testing software category fit.

Step 1: Write a testable hypothesis

A good hypothesis names the change, the audience, the expected effect and the reason. For example: shortening the signup form from seven fields to four will raise completed sign-ups for new visitors because it lowers effort. Vague goals like improving the page cannot be tested.

• Base ideas on data: analytics funnels, session recordings, support tickets and user interviews.
• Change one meaningful thing per test so the result is interpretable.
• Rank ideas by expected impact, confidence and effort.

Step 2: Choose one primary metric and guardrails

Choose a single primary metric before launching, such as trial starts per visitor. Add guardrail metrics that must not get worse, such as revenue per visitor, page speed or refund rate. This prevents winning on clicks while losing on revenue.

Pick a metric close to the change. A button color test should not be judged on annual revenue, which is too distant and noisy.

Step 3: Calculate the sample size

Sample size tells you how many visitors each version needs before the result is trustworthy. It depends on your baseline conversion rate, the smallest lift you care about, the significance level and statistical power. Smaller lifts need far more traffic.

A widely used rule of thumb for 80% power and a 5% significance level is:

Here n is the visitors needed in each variant, p is the baseline conversion rate and δ is the minimum absolute lift you want to detect. With a 5% baseline and a one percentage point lift (δ = 0.01), n is about 16 × 0.05 × 0.95 ÷ 0.0001, or roughly 7,600 visitors per variant. If your page gets 1,000 visitors a week, that is more than 15 weeks, so test a bigger change or a higher-traffic page.

Step 4: Set up the test and check quality

Quality assurance prevents invalid data. Verify that visitors are randomly assigned and stay in the same variant, that conversion events fire in both versions, and that the variant renders without flicker on common browsers and devices.

Setup checklist

• Random assignment at the visitor level, persisted across visits.
• Sample ratio check: traffic split should match the plan, for example 50/50.
• Exclude internal traffic and bots.
• Confirm consent handling and privacy settings for your regions.
• Test the variant on mobile, desktop and slow connections.

Step 5: Run for full cycles and do not peek

Let the test run until the planned sample is reached and for at least one or two full weeks, so weekday and weekend behavior are both included. Stopping the moment the result looks significant inflates false positives.

• Do not change traffic allocation or the variant mid-test.
• Avoid launching during unusual periods such as major promotions unless that is the target.
• If you must monitor early, use a tool or method designed for sequential testing.

Step 6: Analyze the result properly

Look at the effect size and its confidence interval, not only whether a p-value crossed a threshold. A result can be statistically significant yet too small to matter, or inconclusive because traffic was low.

If control converts at 5.0% and the variant at 5.6%, the relative lift is 12%. Check guardrails and segment results, but treat segment findings as new hypotheses unless each segment was sized for the test.

Step 7: Decide, ship and document

Decide using the plan you wrote before the test. Ship a clear winner, iterate on a flat result with a new hypothesis, and drop a loser. Record the hypothesis, screenshots, dates, sample sizes and outcome in a shared log so future teams avoid repeats.

Choosing A/B testing software

The right tool depends on your monthly traffic, who builds experiments, and how well it fits your analytics and consent stack. Compare statistics engines and pricing limits before committing. The SaaSPodium directory lists Optimizely Web Experimentation, VWO, AB Tasty, Kameleoon and Convert Experiences, among others, and our guides compare them in best A/B testing software and Optimizely vs VWO vs AB Tasty.

• Editor type: visual editor for marketers, code-based experiments for developers.
• Statistics engine: fixed-horizon or sequential, and how it handles multiple metrics.
• Integrations: analytics, CDP, data warehouse and consent platforms.
• Flicker control and performance impact on page speed.
• Pricing: the directory lists starting prices of $99 for Convert Experiences and $199 for VWO as of October 2026; confirm traffic limits.

Teams comparing a visual-editor platform can start with VWO and benchmark its statistics approach against the other options.

Frequently Asked Questions

How long should an A/B test run?
Run it until you reach the sample size calculated in advance and for at least one or two full weeks to cover weekday and weekend patterns. Low-traffic sites may need many weeks, in which case test bigger changes or higher-traffic pages.

How much does A/B testing software cost?
Prices vary by traffic and features. The directory lists starting prices of $99 for Convert Experiences and $199 for VWO as of October 2026, while enterprise tools such as Optimizely and Adobe Target list higher starting points. Verify traffic caps and add-ons before comparing.

Can small websites run A/B tests?
Yes, but they need larger changes. With limited traffic, test high-impact elements such as offers, headlines or checkout steps, and consider qualitative research instead when the required sample size would take months to collect.