Effect Size (Cohen's d)

Expresses how large the gap between two group means is relative to their spread. A p-value can be driven down simply by collecting more data, but the effect size does not depend on sample size, so it measures the size of the difference itself.

A p-value says whether a difference can be claimed; it says nothing about how big that difference is. Given enough data, even a negligible gap eventually turns significant. Effect size measures the size of the gap itself.

The most widely used measure is Cohen's d, which expresses the difference in means using the spread as its unit.

d = \dfrac{\bar{x}_1 - \bar{x}_2}{s_p}

s_p = \sqrt{\dfrac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}

$d$ is the effect size
$\bar{x}_1$ and $\bar{x}_2$ are the group means
$s_p$ is the pooled standard deviation, combining both groups' spreads weighted by their degrees of freedom
$n_1$ and $n_2$ are the group sizes

Since the denominator is a spread, d reads as a number of standard deviations. It carries no units, so an experiment measured in centimetres can be set beside one measured in kilograms.

Example

Take the defaults, A as 12, 15, 18, 20, 25 and B as 10, 14, 15, 17, 19.

The means are 18 and 15, a difference of 3. The unbiased variances are 24.5 and 11.5, so the pooled standard deviation is $\sqrt{(4 \times 24.5 + 4 \times 11.5) \div 8}$ , or $\sqrt{18}$ , about 4.2426.

The effect size is $3 \div 4.2426$ , about 0.7071. The two group means sit roughly seven tenths of a standard deviation apart.

Rules of thumb

Cohen suggested 0.2 as a small effect, 0.5 as medium and 0.8 as large. The 0.7071 here falls between medium and large.

Those thresholds are a fallback for when nothing better is available, and their meaning shifts by field. In education and psychology 0.5 is often considered substantial, while fields that can control conditions tightly routinely see much larger values. Comparing against effect sizes previously reported in the same field is far more informative.

Points to watch

P-values and effect sizes answer different questions. A t-test on this data would not reach significance, yet the effect size of 0.7071 is far from trivial — five values per group simply lack the power to detect it. Conversely, with thousands of observations a d of 0.05 becomes significant. This is why papers are expected to report both.

Meta-analysis converts many studies, measured in different units by different methods, into effect sizes before pooling them. The absence of units in d is exactly what makes that possible.