Test for the Difference of Two Proportions

Tests whether two groups differ in proportion. The two samples are pooled to form the standard error, and the difference between the proportions is divided by it to give z. This is the test behind A/B testing.

Page A converted 60 of 500 visitors; page B converted 85 of 500. Is B genuinely better? Testing whether two groups differ in proportion is what this does.

z = \dfrac{\hat{p}_A - \hat{p}_B}{\sqrt{\bar{p}(1 - \bar{p})\left(\dfrac{1}{n_A} + \dfrac{1}{n_B}\right)}}

$\hat{p}_A$ and $\hat{p}_B$ are the sample proportions of each group
$\bar{p}$ is the pooled proportion, total successes divided by total size
$n_A$ and $n_B$ are the group sizes

The pooled $\bar{p}$ in the denominator is the essential part. The null hypothesis states that both groups share the same population proportion, so that common value has to be estimated, and the best estimate available is the two samples counted together. The standard error is built from the pooled figure rather than from either group alone.

Example

Take the defaults: 60 of 500 in group A, 85 of 500 in group B, at a 5% significance level.

The proportions are 12% and 17%, a gap of five percentage points. Pooled, they give $(60 + 85) \div (500 + 500)$ , or 14.5%.

The standard error is $\sqrt{0.145 \times 0.855 \times (1 \div 500 + 1 \div 500)}$ , about 0.022269, so z is $-0.05 \div 0.022269$ , about −2.2453.

The two-sided p-value is about 0.024749. Since the magnitude of z exceeds the critical value of 1.9600, at the 5% level the two proportions are judged to differ.

Points to watch

The most common mistake in A/B testing is checking the result repeatedly and stopping as soon as it turns significant. Peeking while data still accumulates gives chance many opportunities to produce a significant-looking moment even when nothing is happening. Repeated peeking at the 5% level pushes the real false-positive rate far above 5%.

The remedy is to fix the sample size in advance, collect all of it, and test exactly once. This calculator's sample size tool for proportions will give you that number.

Significance and importance are different things. With a large enough sample, a trivial difference becomes significant. Whether a gap between 12% and 17% justifies the change is a business question, not a statistical one.

The normal approximation assumes reasonable counts: at least about 5 successes and 5 failures in each group.