Stochastic Processes 4: The Law of Large Numbers and the Central Limit Theorem

A stochastic process is often made up of a large number of random variables: partial sums, cumulative rewards, total request counts, and random walks all accumulate new variables as the index increases. Individual results remain unpredictable, yet the long-run average and an appropriately scaled sum can still exhibit stable patterns.

The Law of Large Numbers (LLN) answers whether the sample mean gets close to the expected value; the Central Limit Theorem (CLT) answers how large a scale the sample mean fluctuates at around the expected value. Understanding the difference between the two theorems is necessary preparation for entering random walks, Brownian motion, Monte Carlo methods, and statistical inference. For the relationship between random variables and sample paths, see Stochastic Processes 1: What Is a Stochastic Process? first.

1. Why the Sample Mean Can Be Stable

1.1 Setting up the problem with a fair die

Let $X_1,X_2,\dots$ be i.i.d. fair-die rolls. Each $X_i$ can take a value from 1 to 6, with $E[X_i]=3.5$ and $\operatorname{Var}(X_i)=35/12$. The sample mean of the first $n$ results is $\bar X_n=n^{-1}\sum_{i=1}^nX_i$.

When $n$ is small, $\bar X_n$ can deviate noticeably from 3.5. As the sample size grows, results that are too high and too low tend to cancel out in the average. This cancellation doesn’t guarantee that every small stretch of data is close to 3.5 — rather, it means the probability of exceeding a fixed error threshold gradually decreases.

Independence keeps different observations from all drifting in the same direction together; identical distribution means every observation shares the same mean and fluctuation scale; finite expectation or finite variance limits the influence of extreme values on the average. Different versions of limit theorems can relax some of these conditions, but you can’t apply the i.i.d. conclusion directly without checking the assumptions first.

1.2 The Markov and Chebyshev inequalities

Markov’s inequality states that a nonnegative random variable $Y$ satisfies $P(Y\ge a)\le E[Y]/a$ for any $a>0$. Replacing $Y$ with the squared deviation $(X-\mu)^2$ gives Chebyshev’s inequality: if $E[X]=\mu$ and $\operatorname{Var}(X)=\sigma^2$, then $P(|X-\mu|\ge\varepsilon)\le\sigma^2/\varepsilon^2$.

Chebyshev’s inequality only uses the mean and variance — it doesn’t require a normal distribution, or even a symmetric one. The cost is that the bound is usually fairly loose. Writing the threshold as $k$ standard deviations gives $P(|X-\mu|\ge k\sigma)\le1/k^2$: regardless of the distribution’s shape, the probability of landing more than two standard deviations from the mean is at most 1/4, and more than three standard deviations is at most 1/9.

For i.i.d. samples, $E[\bar X_n]=\mu$ and $\operatorname{Var}(\bar X_n)=\sigma^2/n$. Applying Chebyshev’s inequality to the sample mean gives $P(|\bar X_n-\mu|\ge\varepsilon)\le\sigma^2/(n\varepsilon^2)$. The right side approaches 0 as $n$ grows, directly giving the weak law of large numbers under a finite-variance condition.

2. The Law of Large Numbers Describes the Long-Run Average

2.1 The weak and strong laws of large numbers

The weak law of large numbers describes convergence in probability: for any $\varepsilon>0$, $P(|\bar X_n-\mu|>\varepsilon)\to0$. Once an acceptable error is fixed, the probability of the sample mean exceeding it vanishes as the sample size grows.

The strong law of large numbers describes almost sure convergence: $P(\lim_{n\to\infty}\bar X_n=\mu)=1$. The strong law states its result from the perspective of an entire, infinite sample path — except for a set of paths with total probability 0, the sample mean converges to $\mu$.

Almost sure convergence is stronger than convergence in probability. The two versions can’t simply be swapped by name, since different theorems can rely on different moment conditions, independence conditions, and proof techniques. For i.i.d. variables, a finite first absolute moment $E[|X_1|]<\infty$ is already enough to support the classical strong law of large numbers; the simple weak-law proof using Chebyshev’s inequality instead relies on a finite variance.

2.2 A worked example with actual numbers

A fair die has variance $35/12\approx2.917$. If it’s rolled 1,000 times and we want the sample mean to differ from 3.5 by less than 0.2, Chebyshev’s inequality gives $P(|\bar X_{1000}-3.5|\ge0.2)\le(35/12)/(1000\times0.2^2)\approx0.0729$. This bound says the probability of a deviation of at least 0.2 is no more than about 7.29%.

The true probability would be much smaller than the Chebyshev bound, because Chebyshev’s inequality doesn’t use the boundedness or shape of the die’s distribution. Even though the bound is conservative, it demonstrates a conclusion that doesn’t rely on a normal approximation: the probability of failing a fixed error threshold decreases at least as fast as a bound of $1/n$.

The law of large numbers doesn’t mean any single $X_n$ gets close to $\mu$. The millionth die roll can still only produce 1 through 6; what converges is the average of the first $n$ results. The law of large numbers also doesn’t guarantee that a short-run average approaches the expected value monotonically — the sample mean can cross back and forth over $\mu$ repeatedly during convergence.

3. The Central Limit Theorem Quantifies the Remaining Fluctuation

3.1 Standardization and the $1/\sqrt n$ scale

Suppose $X_1,X_2,\dots$ are i.i.d., with a common mean $\mu$ and a common finite, positive variance $\sigma^2$. The classical central limit theorem states that the standardized sample mean $\sqrt n(\bar X_n-\mu)/\sigma$ converges in distribution to $N(0,1)$. Equivalently, for a large sample, $\bar X_n\approx N(\mu,\sigma^2/n)$ can be used.

The sample mean’s standard deviation, $\sigma/\sqrt n$, is called the mean’s standard error. To halve the standard error, the sample size needs to grow fourfold; to shrink it to a tenth, the sample size needs to grow a hundredfold. The same square-root cost shows up repeatedly in statistical experiments, A/B tests, and Monte Carlo estimation.

The central limit theorem doesn’t require each individual $X_i$ to follow a normal distribution. The average of Bernoulli variables, die rolls, and many skewed distributions all gradually approach a normal distribution under suitable conditions. The normal approximation describes the sampling distribution of a standardized sum or average — it doesn’t turn the raw data itself into a normal distribution.

3.2 An approximate calculation for a Bernoulli success rate

Let $X_i\sim\operatorname{Bernoulli}(p)$ represent whether the $i$th request succeeds, recorded as 1 for success and 0 for failure. The sample mean $\hat p=\bar X_n$ is the success proportion, satisfying $E[\hat p]=p$ and $\operatorname{Var}(\hat p)=p(1-p)/n$.

Suppose the true success rate is $p=0.8$, observed over $n=400$ independent requests. The standard error is $\sqrt{0.8\times0.2/400}=0.02$. The CLT approximation says the success proportion is roughly $N(0.8,0.02^2)$, so about 95% of sample proportions would fall within $0.8\pm1.96\times0.02$, that is, roughly $[0.761,0.839]$.

The 95% range in this example describes how the sample proportion varies across repeated draws of 400 requests. If $p$ is unknown, in practice an estimated value has to be substituted into the standard error, along with consideration of finite-sample and interval methods.

4. From Independent Samples to Stochastic Processes

4.1 Partial sums and random walks

Once the partial sum $S_n=\sum_{i=1}^nX_i$ is defined, $\{S_n\}_{n\ge1}$ is itself a stochastic process. The law of large numbers studies $S_n/n\to\mu$, and the central limit theorem studies $(S_n-n\mu)/(\sigma\sqrt n)\Rightarrow N(0,1)$. The LLN determines the first-order macroscopic speed $n\mu$, while the CLT describes the $\sqrt n$ random fluctuation around that speed.

A symmetric random walk’s increments $\epsilon_i$ take $+1$ or $-1$ with equal probability, so $E[\epsilon_i]=0$ and $\operatorname{Var}(\epsilon_i)=1$. The position $S_n=\sum_i\epsilon_i$ satisfies $S_n/n\to0$, but the conclusion isn’t that the position returns to the origin — it’s that the average displacement per step approaches 0. The CLT further gives $S_n/\sqrt n\Rightarrow N(0,1)$, showing that the typical distance after $n$ steps is on the order of $\sqrt n$. For the complete path behavior, continue with Stochastic Processes 7: Random Walks.

The ordinary CLT only looks at the endpoint $S_n$. The functional central limit theorem scales both time and space at once, studying the entire partial-sum path. Donsker’s theorem states that a random walk, suitably interpolated and rescaled, converges to Brownian motion. The normal limit at the endpoint can therefore be extended into a Brownian-motion limit in path space.

4.2 Dependent data needs extra conditions

Observations in a time series or stochastic process are usually dependent. If adjacent $X_t$‘s are positively correlated, collecting 1,000 consecutive observations may provide far less information than 1,000 independent samples. A stationary sequence’s sample-mean variance includes the autocovariance at every lag; positive autocorrelation usually inflates the standard error, while negative autocorrelation can reduce it.

Dependent versions of the LLN and CLT need ergodicity, mixing, martingale-difference conditions, or other constraints controlling long-range dependence. Simply substituting the time-series length for $n$ in the i.i.d. formulas can seriously underestimate uncertainty. For dependence structure, see Stochastic Processes 2: Joint Distributions, Correlation, and Dependence first; the relationship between a long-run average and an ensemble average is discussed in Stochastic Processes 14: Ergodicity.

5. What to Check When Using a Limit Theorem

5.1 Separate the object of convergence from the conclusion

The LLN describes the unscaled sample mean getting close to $\mu$; the CLT describes the limiting distribution after multiplying the mean’s error by $\sqrt n$. The LLN doesn’t provide the exact shape of finite-sample error, and the CLT doesn’t guarantee that an arbitrarily small sample is already close to normal.

Extremely heavy-tailed distributions may not have a finite variance, or even a finite mean. The average of Cauchy samples doesn’t stabilize toward a finite expectation as the sample size grows; the standardized sum of an infinite-variance distribution can also converge to a non-Gaussian stable law instead. Strong dependence, nonstationarity, and sampling bias can likewise break the simple versions of these theorems.

5.2 From macroscopic stability to the continuous limit

The law of large numbers connects the theoretical expectation to the long-run empirical average; the central limit theorem connects the remaining error to the $1/\sqrt n$ scale and the Gaussian limit. For a partial-sum process, the two theorems respectively characterize the deterministic macroscopic trend and the random, second-order fluctuation.

A random walk accumulates discrete increments into a path, and Brownian motion emerges as the continuous limit after rescaling. With a grasp of the LLN, the CLT, and the limitations of dependent data, the Markov processes, random walks, Brownian motion, and stochastic differential equations that follow will share a clear common foundation.