A/B Test Significance Calculator

1. 01Enter variant AThe control: how many visitors saw it, and how many of them converted. A conversion is whatever you decided it was before the test started.
2. 02Enter variant BThe challenger, counted the same way over the same period. Comparing different windows is the quickest way to get a meaningless answer.
3. 03Pick a confidence level and directionNinety-five per cent two-sided is the usual choice. A one-sided test is available but should be chosen before you look at the data, never after.
4. 04Read the result and the caveatsYou get the verdict, the p-value, the interval, and any cautions about sample size. Open "how this was calculated" to see the pooled rate and standard errors behind it.

• Deciding whether to ship a landing pageTwo page versions, one conversion goal. Find out whether the version that looks better is actually distinguishable from the old one.
• Checking an email subject lineCompare opens or clicks across two sends of similar size and see whether the winner is a real winner.
• Sanity-checking a reported liftA dashboard says a variant is up 18%. Put the raw counts in here and see whether that lift survives a significance test.
• Knowing when to keep the test runningA result that is not significant yet, with an interval that still straddles zero, usually means more data rather than a decision.

• 10% versus 12% on 1,000 visitors eachz = 1.43, p = 0.1529. Not significant at 95%, and the interval runs from −0.74 to +4.74 percentage points, so it still includes no difference at all.
• The same rates on 5,000 visitors eachz = 3.20, p = 0.0014. Identical rates, five times the traffic, and now significant — which is exactly how sample size works.
• A result sitting on the line200/2,000 against 240/2,000 gives p = 0.0432: significant at 95%, not significant at 99%. Same data, different threshold.

• Decide the confidence level before you lookChoosing 90% after 95% failed is not analysis, it is moving the goalposts. Pick the threshold when you design the test.
• Significant is not the same as importantWith enough traffic a 0.1 point difference becomes significant. Decide in advance what size of change would actually be worth acting on.
• Do not stop the moment it turns greenChecking repeatedly and stopping at the first significant reading inflates your false-positive rate well beyond the level you picked. Set the sample size in advance and read it once.
• Watch for the small-sample warningWith fewer than about five expected conversions in an arm, the normal approximation this test rests on stops being trustworthy. The p-value is still shown, flagged as indicative only.

With visitors n₁, n₂ and conversions x₁, x₂, the observed rates are p̂₁ = x₁/n₁ and p̂₂ = x₂/n₂. Under the null hypothesis that both come from one underlying rate, that shared rate is estimated by the pooled p̄ = (x₁+x₂)/(n₁+n₂), giving a standard error of √(p̄(1−p̄)(1/n₁+1/n₂)) and a test statistic z = (p̂₂−p̂₁) divided by it. The two-sided p-value is 2(1−Φ(|z|)), where Φ is the standard normal cumulative distribution. A one-sided test uses 1−Φ(z) and answers only the narrower question of whether B beats A.

The test and the confidence interval are answering different questions, so they use different standard errors. The test asks whether one shared rate could have produced both samples, so it pools. The interval estimates the size of a real difference between two distinct rates, so it does not: it uses √(p̂₁(1−p̂₁)/n₁ + p̂₂(1−p̂₂)/n₂). Using the pooled error for both is a common shortcut and it is wrong either way round — it mis-sizes the interval or mis-states the test.

There is no Bayesian posterior, no sequential-testing correction, no adjustment for testing several variants at once, and no minimum-detectable-effect planner. It answers one question — could a difference this size plausibly be noise — and reports the assumptions it made to get there. If you are testing five variants against one control, a plain two-proportion test on each pair will overstate your confidence, and you need a multiple-comparison correction this tool does not apply.

What test does this use?

A two-proportion z-test, two-sided by default. It compares two independent samples of a binary outcome — each visitor either converted or did not. The pooled rate, both standard errors and the z-score are all shown on the page so the result can be checked by hand.

What does the p-value actually mean?

It is the probability of seeing a difference at least this large if the two variants genuinely had the same conversion rate. A small p-value means chance alone rarely produces a gap this big at this sample size. It is not the probability that B is better.

What does "not statistically significant" tell me?

That this test cannot separate the two variants with the data you have. It is not evidence that they perform the same. Often it means the sample is still too small — the confidence interval will show how much room is left on either side of zero.

Should I use a one-sided test?

Usually not. A one-sided test only asks whether B beats A and ignores the possibility that B is worse, which makes it easier to reach significance. It is legitimate only when you genuinely decided in advance that a drop would lead to the same action as no change. Switching to it after seeing the numbers is not.

Why does it warn me about small samples?

The z-test relies on a normal approximation that needs enough expected conversions and non-conversions in each arm — conventionally at least five of each. Below that the p-value can be well off. The tool computes it anyway but flags it clearly rather than presenting a shaky number as solid.

What if one variant had no conversions at all?

The test still runs and is flagged, because the interval around a zero rate is very wide. If neither variant converted anyone, there is genuinely nothing to compare and the calculator says so instead of returning a number.

Does a significant result mean I should ship variant B?

Not on its own. Significance says the difference is unlikely to be noise; it says nothing about whether the difference is large enough to matter, whether it will hold over time, or what it costs to implement. Those are judgements the test cannot make for you.