A/B Testing Statistical Significance: Sample Size, P-Values and Mistakes That Create False Winners

Teams misuse significance more often than they misread it. They stop tests the moment a dashboard turns green, run many variants without correction, or declare a winner from too few conversions. Each habit raises the chance of shipping a change that does nothing, or hurts.

The underlying math is two-proportion testing, which the NIST e-Handbook section on comparing two proportions covers in detail, and sample-size planning is explained in the NIST guidance on required sample sizes. If you want the execution side, read our guide to running an A/B test. Tools are listed in the A/B testing software category.

What statistical significance actually means

Statistical significance means that, if there were truly no difference between variants, a result at least this extreme would occur less often than your chosen alpha level. It does not tell you the size or business value of the lift.

Think of alpha as your tolerance for false positives. At 0.05, about one in twenty tests of a truly useless change will still look like a winner. A p-value is not the probability that your variant is better, which is the most common misreading, and it says nothing about practical importance.

• Null hypothesis: the variants perform the same.
• Alpha: the false positive rate you accept before the test starts.
• Power: the chance of detecting a real effect, often set at 80%.
• Minimum detectable effect: the smallest lift worth finding.
• Confidence interval: the range of lifts consistent with your data.

How to plan sample size before launch

Plan sample size from your baseline conversion rate, minimum detectable effect, alpha and power. Smaller expected lifts and lower baselines need far more traffic, which is why many tests on low-traffic pages never reach a conclusion.

Here n is the sample size per variant, p is the baseline conversion rate, δ is the absolute lift you want to detect, Z_α/2 is the critical value for your significance level, and Z_β reflects statistical power. At 95% confidence and 80% power, the two Z values sum to about 2.80. Most testing platforms include a calculator, so use the formula to understand the trade-offs, not to replace the tool.

Mistakes that create false winners

The most damaging mistakes are peeking, stopping early, testing too many variants and ignoring sample ratio mismatches. All of them inflate false positives while the dashboard still looks authoritative.

• Peeking: checking results repeatedly and stopping at the first significant reading. Use fixed horizons or sequential methods built into your platform.
• Multiple comparisons: more variants or metrics mean more chances for a fluke. Correct for it or limit the number of primary metrics.
• Short duration: run at least one or two full business cycles to capture weekday effects.
• Sample ratio mismatch: if traffic split deviates from the plan, assignment or tracking is broken, so do not trust the result.
• Novelty effects: early lifts from new designs can fade.

Frequentist versus Bayesian reporting

Frequentist tools report p-values and confidence intervals against a fixed sample plan, while Bayesian tools report the probability that a variant is better given the data. Both are valid when used with discipline, and they differ mostly in how they communicate uncertainty.

Many commercial platforms offer one or both approaches, and some add sequential testing to allow safe early stopping. Ask each vendor what stats engine it uses, how it handles peeking and whether you can export raw data to verify results.

Choosing software with a sound statistics engine

Look for transparent methodology, support for your traffic level, and integrations with your analytics stack. A strong statistics engine will document its assumptions and guard against common errors.

• Optimizely Web Experimentation: positioned for medium and enterprise experimentation programs.
• VWO: experience optimization platform for small, medium and enterprise teams.
• Convert Experiences: A/B testing aimed at data-driven agencies and experts.
• Kameleoon: advanced testing and personalization for medium and enterprise teams.
• AB Tasty: experience optimization with feature management.

Compare these in more depth in our best A/B testing software roundup or the Optimizely vs VWO vs AB Tasty comparison.

A practical decision checklist

Before declaring a winner, confirm the planned sample is reached, the split is healthy, the lift is large enough to matter and no guardrail metric moved the wrong way. This takes five minutes and prevents costly rollbacks.

• Was the sample size and duration set before launch?
• Does the traffic split match the design?
• Is the confidence interval above your minimum useful lift?
• Did revenue, latency or error-rate guardrails hold?
• Will you validate with a holdout or follow-up test for high-stakes changes?

For a closer look at one of the established platforms, see Optimizely.

Frequently Asked Questions

What sample size do I need for an A/B test?
It depends on your baseline conversion rate, the smallest lift you care about, and your chosen confidence and power. Low baselines and small lifts need many thousands of visitors per variant, so use a calculator before you launch.

Which A/B testing tool has the best statistics?
No tool is best for every case. Compare how Optimizely, VWO, Convert Experiences, Kameleoon and AB Tasty handle peeking, multiple comparisons and data export, and validate with a trial test on your own traffic.

Can I trust a result at 95% significance?
Only if the test was planned and run correctly. A 95% threshold still allows false positives, and peeking, many variants or broken tracking make errors far more likely than five percent.