How to Calculate Statistical Power

Finds the chance of detecting a difference that is really there. It follows from the effect size, the sample size and the significance level. A study with low power misses real differences, so a result that is not significant proves nothing.

Power is the chance that a test finds a difference that genuinely exists. It follows from the effect size, the sample size and the significance level.

\text{power} = \Phi(d\sqrt{n} - z_{\alpha/2})

$d$ is the effect size, $n$ the sample size and $z_{\alpha/2}$ the critical value for the significance level. The chance of missing the difference, $\beta$ , is one minus the power, and is known as a type II error.

Example

At an effect size of 0.5 with 30 observations at the 5% level, the power is 78.2%. Even with a real difference present, one study in five will miss it. Reaching 80% power takes a sample of 32.

Not significant does not mean no difference

When an underpowered study fails to reach significance, that is not evidence of no difference. It may simply have lacked the observations to find one. Any non-significant result should be read alongside the power of the study that produced it.

Sample size goes with the square of the effect

Halving the effect size quadruples the sample needed for the same power. The smaller the difference being chased, the more sharply the requirement climbs. The proper order is to fix the sample size in advance from the size of difference worth finding.

Notes

This uses a normal approximation. In small samples it differs slightly from the more exact calculation based on the t distribution.