How to use this calculator
Run an A/B test by sending roughly equal traffic to two versions of a page, ad, or email. Once your test has been running long enough to collect data, enter the number of visitors and conversions for each variant and click Calculate.
This tool runs a two-proportion z-test (two-tailed) to determine whether the difference in conversion rates is statistically significant or likely due to chance. A 95% confidence threshold means there's only a 5% probability that the observed difference occurred randomly.
Before you launch a test, use the Sample Size tab to figure out how many visitors you need per variant - stopping early is one of the most common A/B testing mistakes.
For context on what conversion rates to benchmark against, see our guide: What is a good conversion rate?
Frequently asked questions
What does statistical significance mean in A/B testing?
Statistical significance tells you how likely it is that the difference between your two variants is real and not due to random chance. A result is considered statistically significant at 95% confidence when there is only a 5% probability that the observed difference occurred by chance alone. In practice, most marketers use 95% confidence as the minimum threshold before acting on A/B test results.
How many visitors do I need for a valid A/B test?
The required sample size depends on your baseline conversion rate and the minimum lift you want to detect. To detect a 20% improvement on a 3% baseline conversion rate at 95% confidence and 80% statistical power, you typically need around 10,000–15,000 visitors per variant. Smaller improvements require larger samples. Use the Sample Size tab in this tool to calculate your specific number before launching a test.
What is a p-value in an A/B test?
The p-value represents the probability of seeing a difference this large (or larger) between your variants purely by chance, assuming there is no real difference. A p-value below 0.05 means the result is statistically significant at the 95% confidence level. A lower p-value means stronger evidence that your variant's performance is genuinely different from the control - not just random noise.
Should I use one-tailed or two-tailed A/B tests?
Use a two-tailed test in almost all marketing A/B tests. A two-tailed test asks: "Is the variant different from control in either direction?" This is more conservative and appropriate because your variant could perform better or worse than control. One-tailed tests are only valid when you have a strong prior reason to believe the variant can only outperform - which is rarely true in practice. This calculator uses a two-tailed test.
