How to Run a Two-sample t-test (Welch and Student)

Tests whether the means of two independently collected samples differ. Choose welch, which does not assume equal variances and is the safer default, or student, which pools the variances and assumes they are equal.

Explanation

A two-sample t-test asks whether two independently collected samples have different means: two classes of students, two production lines, an old design against a new one. The samples are not paired, and they need not be the same size.

t = \dfrac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}

$\bar{x}_1, \bar{x}_2$ — the two sample means
$s_1^2, s_2^2$ — the two sample variances
$n_1, n_2$ — the number of values in each

That is welch's method, which does not assume that the two populations share a variance. Its degrees of freedom come from the Welch-Satterthwaite formula and are usually not a whole number.

\nu = \dfrac{\left(\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}\right)^2}{\dfrac{(s_1^2/n_1)^2}{n_1 - 1} + \dfrac{(s_2^2/n_2)^2}{n_2 - 1}}

Choosing student instead assumes the variances are equal and pools them into one, on $n_1 + n_2 - 2$ degrees of freedom.

Example

With the defaults, Data A is 12, 15, 18, 20, 25 and Data B is 10, 14, 15, 17, 19. The means are 18 and 15, a difference of 3, and the sample variances are 24.5 and 11.5. Welch gives $t \approx 1.1180$ on 7.0771 degrees of freedom, with a two-sided p-value of 0.3001. The critical value at 5% is 2.3594, so the means cannot be said to differ.

Which method

When in doubt, use welch. You rarely know that two populations share a variance, and welch costs almost nothing when they happen to. Student's method is there for textbook problems that ask for the pooled formula.

When the two samples are the same size, both methods give the same $t$ and differ only in the degrees of freedom. Switching the example above to student gives 8 degrees of freedom and a p-value of 0.2960.

Watch out

Do not use this test on before-and-after measurements of the same subjects. Those are paired, and ignoring the pairing lets the differences between individuals leak in as noise, hiding the change.