A/B Test Significance Calculator

Check whether the gap between two versions is bigger than chance would explain, and how big the difference might really be.

How to use

What it checks and how it works

The tool runs a two-proportion z-test with a pooled standard error, the test the NIST/SEMATECH e-Handbook sets out for comparing the proportion of defectives in two processes: z = (p1 - p2) / sqrt(p(1 - p)(1/n1 + 1/n2)), where p is the pooled proportion. The p-value comes from the normal distribution, and the interval for the difference uses the unpooled standard error. The test assumes the visitors were assigned at random, each visit is counted once, and the sample size was fixed in advance. It does not correct for stopping early, for several variants or for several metrics, and it is a poor fit when any cell holds fewer than 5 conversions or non-conversions, which the tool warns about. A significant result says the gap is unlikely to be chance. It does not say the effect is large or worth shipping.

What does the p-value mean?

It is the probability of seeing a gap at least this large if the two versions truly convert at the same rate. A small p-value is evidence against that, but it is not the probability that the variant is better.

One-sided or two-sided?

Choose two-sided unless you decided before the test that you would act only on an increase. Two-sided is the cautious default and is what most teams report.

Can I stop the test when it turns significant?

Stopping the moment p dips below the line inflates false wins. Fix the sample size in advance with the sample size tool, run to that number, then read the result once.

Sources

Related tools

Privacy

Runs in your browser; nothing is uploaded. The interactive tool needs JavaScript.