Determine if your A/B test results are statistically significant. Enter visitors and conversions for control and treatment to get p-value, confidence level, relative lift, and a clear pass/fail verdict.
Free, no sign-up — one of the SaaSQuickStart Data & Analytics tools.
This calculator uses a two-proportion z-test to compare conversion rates between your control and variant. It computes a pooled standard error from the combined conversion rate, then derives a z-score that measures how many standard deviations apart the two rates are. The p-value is the probability of seeing a difference this large (or larger) purely by chance if there is actually no effect. A p-value below 0.05 means you can be 95% confident the difference is real; below 0.01 means 99% confidence. The normal CDF is approximated using the Horner's method error function — accurate to within ±1.5×10⁻⁷.
It means the observed difference between variants is unlikely to have occurred by random chance alone. At 95% confidence, there is less than a 5% probability the result is a fluke. It does not tell you the difference is large or practically important — that is measured by relative improvement.
Running a test until you see p < 0.05 leads to inflated false-positive rates. Determine your required sample size before starting the test (use the recommendation above), commit to running until you reach it, and only then read the result. Stopping early because results look good ("peeking") is the most common mistake in A/B testing.
The MDE is the smallest improvement worth detecting. Smaller MDE requires more traffic. For most SaaS products, a 5–10% relative improvement is a meaningful threshold — detecting smaller effects requires tens of thousands of visitors per variant and weeks of runtime.
This calculator uses a two-tailed test, which tests whether Variant B differs from Control in either direction. Use two-tailed unless you are certain ahead of time that the variant can only improve the metric — which is rare in practice. Two-tailed tests are more conservative and reduce false positives.