Tool

A/B test significance calculator

Significance and confidence interval, plus whether the test was ever capable of answering the question.

Every significance calculator tells you whether a result cleared a threshold. Almost none tell you whether the test could have cleared it, which for most accounts is the more useful answer. This one reports the significance, the confidence interval that significance is usually mistaken for, and the sample size and runtime your chosen sensitivity actually requires, including when that runtime makes the test not worth running.

What your data says

Verdict: Significant at 95% confidence
p-value, two-tailed: 0.0283
Control conversion rate: 5.00%
Variant conversion rate: 6.00%
Relative lift: 20.00%
Confidence interval on the difference, at 95%: 0.11% to 1.89%

The interval is the number to argue from, not the p-value. If it spans zero, the variant may well be worse, and reporting the midpoint as the expected gain is how a flat test becomes a case study.

Whether the test could work at all

Visitors needed per variant to detect a 10% lift: 31,234
Total visitors needed: 62,468
Weeks at your weekly traffic: 31.2 weeks

Your account is too small to A/B test this

At this traffic and this sensitivity the test outlasts the seasonality, the ad accounts and probably the landing page. Stop testing increments and change something big enough that it does not need a statistician: the offer, the audience, the page, or what the form asks for. A change large enough to see without a calculator is the one your traffic can actually evaluate.

Not enough traffic to test?

That is the common case, and it changes what conversion work should look like rather than ruling it out. The alternatives are on the conversion rate optimization page.

Talk it through with us

Method

How is significance calculated here?

A two-tailed two-proportion z-test, with the interval built on the unpooled standard error rather than the pooled one.

The test statistic compares the two conversion rates against the standard error you would expect if they came from the same underlying rate, which is the pooled proportion. The p-value is the two-tailed probability of seeing a gap at least that large by chance alone.

The confidence interval is built differently, on the unpooled standard error, because it is estimating the size of a difference rather than testing whether one exists. Using the pooled figure for both is a common shortcut and it makes the interval and the p-value disagree at the margins. The interval is the number worth arguing from: it says how large the effect might plausibly be, and whether zero is still on the table.

The sample size uses your control rate as the baseline, your chosen confidence and power, and the smallest relative lift you said was worth detecting. It is the standard two-proportion calculation, and it produces uncomfortable numbers because the underlying reality is uncomfortable, not because the model is pessimistic.

One assumption worth naming: this is a fixed-sample test, so it assumes you decide the sample size first and read the result once. Watching a running test and stopping when it turns significant inflates the false positive rate well beyond the level you chose, and no calculator can correct for it after the fact. If low traffic is the constraint, the useful response is on the conversion rate optimization page.

Questions

What do people ask about A/B significance?

What significance means, why two-tailed, why stopping early breaks it, and what to do when traffic is too low.

What does statistical significance actually tell me?

Only how surprising your result would be if the two variants were identical. It is not the probability that the variant is better, it says nothing about how much better, and it can be reached by a difference too small to be worth shipping. The confidence interval answers the question people think significance answers.

Why is a two-tailed test used?

Because a one-tailed test halves the p-value and is only legitimate when the direction was committed to before any data was seen. In practice nobody does that, so a one-tailed test in a conversion tool is a way of reaching significance sooner rather than a methodological choice.

Can I stop the test as soon as it goes significant?

No, and this is the most common way tests produce false winners. Checking repeatedly and stopping at the first significant reading inflates the false positive rate far above the level you chose. Fix the sample size before starting and read the result when you reach it.

What if my account is too small to run a valid test?

That is the usual case, and the calculator says so rather than hiding it. The response is not to test smaller things but larger ones: the offer, the audience, the page, or what the form asks for. Changes big enough to see without a calculator are the only ones low-traffic accounts can evaluate.

Why does the required sample size look so large?

Because detecting a small relative lift on a low base rate takes a great deal of data, and the arithmetic does not care how much anyone wants it to be otherwise. Detecting a 10% relative lift on a 5% conversion rate at conventional confidence and power takes roughly thirty thousand visitors per variant.

Does anything I type get sent to you?

No. The calculator runs entirely in your browser, there is no backend, and nothing is transmitted. The only way your numbers leave your machine is if you copy the shareable link and send it yourself.

Told you cannot test?

Most accounts cannot, and it is not the end of conversion work. Tell us what your traffic looks like and we will say which changes are worth making without proof.

Talk it through Conversion rate optimization