Probability and Statistics 4: The Chi-Square, t, and F Distributions

Probability and Statistics 3 introduced the sample mean $\bar X_n$ and sample variance $S^2$, but only covered their expected values, not their own complete distributions as random variables. Building a confidence interval for a mean, or comparing whether two datasets’ variances differ, both require knowing the true sampling distribution of these statistics. The chi-square, t, and F distributions are three distribution families derived from the standard normal distribution, specifically for describing sample variance and for estimating a mean when the variance is unknown.

1. From the Standard Normal to the Chi-Square Distribution

1.1 Definition of the chi-square distribution

If $Z_1,\dots,Z_k$ are $k$ independent standard normal random variables, then the sum of squares $Q=Z_1^2+\cdots+Z_k^2$ follows the chi-square distribution, written $Q\sim\chi^2_k$, where $k$ is called the degrees of freedom. The chi-square distribution only takes nonnegative values, and its shape changes with the degrees of freedom: it’s noticeably right-skewed for small degrees of freedom, and approaches a normal distribution as the degrees of freedom grow. The chi-square distribution’s expected value is $E[Q]=k$ and its variance is $\text{Var}(Q)=2k$; for example, $\chi^2_9$ has an expected value of 9 and a variance of 18.

1.2 The sampling distribution of sample variance

For $X_1,\dots,X_n\sim N(\mu,\sigma^2)$ drawn independently, it can be shown that $\dfrac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}$, where $S^2$ is the unbiased sample variance defined in Part 3 of this series. The degrees of freedom is $n-1$ rather than $n$, for a reason consistent with $S^2$‘s own definition: the sample mean $\bar X_n$ has already used up one degree of freedom from the data, leaving $n-1$ independent pieces of deviation-from-mean information. This relationship lets $S^2$‘s variability be described precisely by the chi-square distribution, and is the foundation for constructing a confidence interval for variance.

2. The t Distribution: Estimating a Mean When Variance Is Unknown

2.1 Definition and motivation

If the population variance $\sigma^2$ is known, $\dfrac{\bar X_n-\mu}{\sigma/\sqrt n}$ follows a standard normal distribution. But in practice $\sigma^2$ is usually unknown, and can only be replaced with the sample variance $S^2$ — in that case, the statistic $\dfrac{\bar X_n-\mu}{S/\sqrt n}$ is no longer standard normal, but instead follows the t distribution (Student’s t-distribution) with $n-1$ degrees of freedom, written $t_{n-1}$. The t distribution can be defined as a standard normal divided by the root-mean-square of an independent chi-square variable: $T=Z/\sqrt{V/k}$, where $Z\sim N(0,1)$ and $V\sim\chi^2_k$.

The t distribution’s shape resembles the normal distribution — symmetric around 0 — but with heavier tails than the normal; the smaller the degrees of freedom, the heavier the tails, because replacing $\sigma$ with $S$ introduces extra uncertainty from estimating the variance. As the degrees of freedom go to infinity, the t distribution approaches the standard normal.

2.2 Building a confidence interval for a mean with the t distribution

When the sample size is small and $\sigma^2$ is unknown, the 95% confidence interval for the mean is $\bar x\pm t_{0.025,\,n-1}\times s/\sqrt n$, where $t_{0.025,\,n-1}$ is the two-tailed critical value of the t distribution.

Suppose a measurement has $n=10$ samples, sample mean $\bar x=52.3$, sample standard deviation $s=4.8$, and looking up the table gives $t_{0.025,9}\approx2.262$. The margin of error for the confidence interval is $2.262\times4.8/\sqrt{10}\approx3.43$, so the 95% confidence interval is approximately $(48.87,55.73)$. With small degrees of freedom, $t_{0.025,9}$ is noticeably larger than the corresponding normal-distribution critical value of 1.96 — this is exactly the t distribution’s heavy-tail property showing up in the confidence interval: the smaller the sample size, the less certain the estimate of variance, so the confidence interval should widen accordingly.

3. The F Distribution: Comparing Two Variances

3.1 Definition of the F distribution

If $U\sim\chi^2_{d_1}$ and $V\sim\chi^2_{d_2}$ are independent, then the ratio $F=\dfrac{U/d_1}{V/d_2}$ follows the F distribution with degrees of freedom $(d_1,d_2)$, written $F\sim F_{d_1,d_2}$. The F distribution only takes nonnegative values, and its shape is determined jointly by the two degrees of freedom; like the chi-square and t distributions, it can also be derived from the standard normal — all three are natural results of the same “normal sampling theory.”

3.2 Comparing the variances of two samples

For two independent normal samples, their respective unbiased variances $S_1^2,S_2^2$ satisfy $\dfrac{S_1^2/\sigma_1^2}{S_2^2/\sigma_2^2}\sim F_{n_1-1,n_2-1}$. If the two population variances are equal ($\sigma_1^2=\sigma_2^2$), this statistic simplifies to $S_1^2/S_2^2\sim F_{n_1-1,n_2-1}$, letting the observed variance ratio be used directly to judge whether the hypothesis “the two variances are equal” is reasonable.

Suppose two samples have sizes $n_1=13$ and $n_2=16$, with sample variances $S_1^2=15.2$ and $S_2^2=8.7$, giving a ratio $F=15.2/8.7\approx1.747$. The critical value for degrees of freedom $(12,15)$ is $F_{0.05,12,15}\approx2.475$. Since the observed ratio of 1.747 is less than the critical value of 2.475, this data isn’t enough to conclude, at the 5% significance level, that the two population variances differ. The F distribution is also the core distribution used by analysis of variance (ANOVA) when comparing means across multiple groups.

4. Summary: Sampling Distributions Connect Estimation and Inference

The $\chi^2$, t, and F distributions are all built on “combinations of independent standard normal variables”: the chi-square distribution describes a sum of squares, the t distribution describes the ratio of a normal to a chi-square, and the F distribution describes the ratio of two chi-squares. The three correspond to the three most common problems in statistical inference: the chi-square distribution describes the sampling distribution of a single variance, the t distribution lets a mean still be estimated when variance is unknown, and the F distribution lets two variances be compared directly. These sampling distributions are the shared computational foundation for both confidence intervals and hypothesis testing.