Probability & Statistics 2: Common Distributions, Random Vectors, and Correlation

A random variable turns the outcome of a trial into a number, and a distribution describes the probability of each of those numbers occurring. Memorizing distribution names and formulas alone isn’t enough: when modeling, you also need to know what the parameters actually represent, whether the data fits the model’s assumptions, and what information the mean and variance leave out.

This article first builds up the numerical tools used to describe a single distribution, then compares common discrete and continuous distributions, and finally generalizes from a single random variable to a random vector. Whenever a PMF, PDF, or CDF is used, the necessary definitions are given inline, so this article doesn’t require having read the other articles in the series first.

1. Describing a Random Variable with a Distribution and Numerical Summaries

1.1 The PMF, PDF, and CDF Answer Different Questions

A discrete random variable $X$ uses a probability mass function (PMF) $p_X(x)=P(X=x)$, and the probabilities of all possible values must satisfy $p_X(x)\ge 0$ and $\sum_x p_X(x)=1$. A continuous random variable uses a probability density function (PDF) $f_X(x)$, with interval probability given by $P(a\le X\le b)=\int_a^b f_X(x)\,dx$. A density value itself is not a probability; for a continuous variable, $P(X=x)=0$ typically holds, and what’s actually meaningful is the area under a stretch of the curve.

A cumulative distribution function (CDF) $F_X(x)=P(X\le x)$ applies equally to discrete and continuous variables. The CDF can compute interval and tail probabilities directly, for example $P(X>x)=1-F_X(x)$; for a continuous $X$, $P(a<X\le b)=F_X(b)-F_X(a)$. When comparing two models, first confirm whether you’re working with a point probability, a density, or a cumulative probability — doing so avoids mistaking a PDF’s height for an event’s probability.

1.2 Expectation, Variance, and Moments

Expectation describes the center of gravity of a distribution. In the discrete case, $E[X]=\sum_x xp_X(x)$; in the continuous case, $E[X]=\int_{-\infty}^{\infty}xf_X(x)\,dx$. Expectation isn’t necessarily a value you could actually observe: a fair six-sided die has an expected value of $3.5$, but no single roll will ever come up $3.5$. Expectation also isn’t guaranteed to exist; a distribution whose tail decays too slowly can make the integral diverge.

Variance $\operatorname{Var}(X)=E[(X-\mu)^2]=E[X^2]-\mu^2$ measures the squared deviation of observations relative to the mean $\mu=E[X]$. The standard deviation $\sigma=\sqrt{\operatorname{Var}(X)}$ shares the same units as $X$, making it easier to interpret than the variance. If a server’s latency has a mean of 100 ms and a standard deviation of 10 ms, the standard deviation directly describes the fluctuation in milliseconds; the variance’s units are squared milliseconds instead.

The $k$-th order raw moment is $E[X^k]$, and the $k$-th order central moment is $E[(X-\mu)^k]$. The first-order raw moment is just the mean, and the second-order central moment is just the variance; the standardized third-order central moment forms skewness, used to describe left-right asymmetry; the standardized fourth-order central moment relates to kurtosis, used to describe tail weight and concentration. A finite number of moments generally can’t fully determine a distribution, and some distributions don’t have higher-order moments at all, so moments are a summary, not the distribution itself.

1.3 Median, Quantiles, and Tail Risk

The median is the 0.5 quantile. More generally, the $p$ quantile can be defined as $q_p=\inf\{x:F_X(x)\ge p\}$; the 25th, 50th, and 75th percentiles respectively describe a distribution’s quartile positions. The interquartile range $IQR=q_{0.75}-q_{0.25}$ is less sensitive to extreme values, making it suitable for describing skewed data or data containing outliers.

The difference between the mean and the median can reveal skew. When nine latency measurements are 20 ms and one is 1020 ms, the mean is 120 ms while the median is still 20 ms. The mean reflects the total cost caused by extreme latency, while the median reflects the typical request; the two numbers answer different questions. Service reliability reporting usually also needs to report $p_{95}$, $p_{99}$, or the tail probability of exceeding a threshold, $P(X>c)$, since distributions sharing the same mean and standard deviation can still have drastically different tails.

2. Common Discrete Distributions

2.1 Bernoulli, Binomial, and Geometric

The Bernoulli distribution describes a single trial with only success or failure. If $X\in\{0,1\}$ with success probability $p$, this is written $X\sim\operatorname{Bernoulli}(p)$; its expectation is $p$, and its variance is $p(1-p)$. The parameter $p$ must have a clear meaning within each specific trial it’s applied to — for example, the probability that “a request completes within 200 ms” — rather than an abstract success rate with no time frame or scope attached.

The Binomial distribution describes the total number of successes across $n$ independent Bernoulli trials sharing the same success rate. If $X\sim\operatorname{Binomial}(n,p)$, then $P(X=k)={n\choose k}p^k(1-p)^{n-k}$, $E[X]=np$, $\operatorname{Var}(X)=np(1-p)$. Suppose 20 independent requests each have a 5% chance of failing; the number of failures $X$ can be modeled as $\operatorname{Binomial}(20,0.05)$, and the probability of exactly two failures is ${20\choose2}(0.05)^2(0.95)^{18}\approx0.189$. If the requests share the same failing node, failure events could be highly correlated, and the Binomial’s independence assumption would no longer hold.

The Geometric distribution describes how many trials are needed before the first success. If the trial number on which success occurs is denoted $T\in\{1,2,\ldots\}$, then $P(T=k)=(1-p)^{k-1}p$ and $E[T]=1/p$. Some textbooks instead denote the number of failures as a variable starting at 0; you must confirm where the counting starts before applying a formula. The Geometric distribution has the discrete memoryless property, making it suitable for waiting-time problems with a fixed success rate and independent trials.

2.2 The Poisson Distribution

The Poisson distribution describes the number of events within a fixed interval. If $X\sim\operatorname{Poisson}(\lambda)$, then $P(X=k)=e^{-\lambda}\lambda^k/k!$, and $E[X]=\operatorname{Var}(X)=\lambda$. The parameter $\lambda$ is the average number of events within the specified interval; if the average is 3 requests per minute, the mean for a ten-minute interval, under a homogeneity assumption, is 30 — you can’t still plug in a parameter of 3.

The Poisson model generally assumes events occur individually and independently of one another within very short intervals, and that the average rate is fixed across the interval being studied. A large gap between the mean and the sample variance may indicate overdispersion in the data, a rate that changes over time, or clustering of events. The Poisson distribution for a fixed interval only describes a single count variable; to describe how a count accumulates over time, and the relationship between a count and the waiting time, see Stochastic Processes 9: The Poisson Process.

2.3 Categorical and Multinomial

The Categorical distribution extends Bernoulli’s two outcomes to $K$ mutually exclusive categories, with parameters given by a probability vector $(p_1,\dots,p_K)$, where every $p_i\ge0$ and they sum to 1. A single request’s status might be split into success, client error, and server error, with the three probabilities together defining one Categorical trial.

The Multinomial distribution further extends a single Categorical trial to $n$ independent, identically distributed trials, giving the count of each category $(X_1,\dots,X_K)$, satisfying $\sum_iX_i=n$. The counts of the different categories can’t be independent of one another, since one more count in one category necessarily reduces the total available for the others. This constraint foreshadows an important idea: every component of multi-dimensional data may each have a simple marginal distribution, yet the components remain constrained by their joint structure.

3. Common Continuous Distributions

3.1 Uniform, Exponential, and Gamma

The Uniform distribution $X\sim\operatorname{Uniform}(a,b)$ has a fixed density of $1/(b-a)$ on the interval $[a,b]$, with expectation $(a+b)/2$ and variance $(b-a)^2/12$. The model assumes every equal-length subinterval has the same probability, making it suitable for describing a location with no preference within a bounded range; Uniform shouldn’t be adopted just because the data happens to fall within some range — the observation mechanism still needs to support the uniformity assumption.

The Exponential distribution $T\sim\operatorname{Exponential}(\lambda)$ is commonly used to describe the gaps between events in a Poisson process, with density $f_T(t)=\lambda e^{-\lambda t}$ ($t\ge0$), expectation $1/\lambda$, and variance $1/\lambda^2$. The larger the rate $\lambda$, the shorter the average waiting time. The Exponential’s memoryless property, $P(T>s+t\mid T>s)=P(T>t)$, means that time already waited doesn’t change the distribution of the remaining waiting time; lifetime data exhibiting aging, wear, or queuing clustering usually doesn’t satisfy this assumption.

The Gamma distribution can describe the accumulation of multiple independent Exponential waiting times. Using the shape-rate parameterization $T\sim\operatorname{Gamma}(\alpha,\lambda)$, the expectation is $\alpha/\lambda$ and the variance is $\alpha/\lambda^2$. Different software may use the scale $\theta=1/\lambda$ instead of the rate; when you encounter Gamma parameters, you must first confirm whether the second parameter is a rate or a scale.

3.2 The Normal Distribution and Standardization

The Normal distribution $X\sim N(\mu,\sigma^2)$ is determined by the location parameter $\mu$ and the scale parameter $\sigma>0$, with expectation $\mu$ and variance $\sigma^2$. Standardizing via $Z=(X-\mu)/\sigma$ gives $Z\sim N(0,1)$, so observations in different units can be converted into a number of standard deviations from the mean.

If a part’s length can be approximated as $N(100,2^2)$ mm, and the spec requires 96 to 104 mm, the standardized bounds are -2 and 2, so the pass rate is approximately $P(-2\le Z\le2)\approx0.9545$. This computed result depends on the Normal assumption; if the distribution is skewed, has heavy tails, or mixes several production batches together, applying the 68-95-99.7 rule based on the mean and standard deviation alone will underestimate tail probability.

The Normal distribution commonly shows up in measurement errors and sample means arising from many small effects added together, but “having a lot of data” doesn’t automatically make the raw data itself follow a Normal distribution. The central limit theorem typically describes a centered and rescaled sum or mean, not every individual observation from an arbitrary population.

3.3 Choosing a Candidate Model Based on the Data’s Type

A distribution’s name should be determined jointly by the random mechanism and the support set. For a binary outcome, consider Bernoulli first; for the number of successes within a fixed number of trials, consider Binomial; for a sparse event count within a fixed time period, consider Poisson; for a positive waiting time, compare Exponential, Gamma, or other lifetime distributions; for a location with roughly no preference within upper and lower bounds, consider Uniform; for a real-valued quantity formed by summing many small errors, consider Normal.

A candidate model still needs to be checked against the data. A researcher can compare histograms, the empirical CDF, sample quantiles against model quantiles, and check whether the mean and variance are consistent with the parameter constraints. A model is an approximation carrying assumptions; when choosing a distribution, the generating mechanism, the support set, and tail behavior matter more than the shape of the curve.

4. Joint, Marginal, and Conditional Distributions

4.1 A Joint Distribution Preserves How Variables Occur Together

The joint PMF of two discrete random variables is $p_{X,Y}(x,y)=P(X=x,Y=y)$; two continuous random variables can instead use the joint density $f_{X,Y}(x,y)$. A joint distribution needs to be normalized over every possible combination, and it contains the complete information about both the individual variables’ distributions and the dependence between them.

Consider a data table sampled with equal probability: $(X,Y)$ each take the values $(0,0),(0,1),(1,1),(1,2)$ with probability $1/4$. $X$’s distribution alone only tells the reader $P(X=0)=P(X=1)=1/2$, and $Y$‘s distribution alone only tells the reader $P(Y=0)=1/4$, $P(Y=1)=1/2$, $P(Y=2)=1/4$. The joint PMF additionally preserves the dependence structure that $Y$ never equals 2 when $X=0$, and $Y$ never equals 0 when $X=1$.

A joint distribution can be generalized to a random vector $\mathbf X=(X_1,\dots,X_d)^\mathsf T$. Each observation is no longer a single number, but a point in $d$-dimensional space. For example, a request might be represented by a three-dimensional random vector made up of latency, response size, and CPU time; a multivariate distribution describes how each of the three components varies on its own, as well as how they vary together.

4.2 Marginal Distributions and Conditional Distributions

Summing a joint PMF over a variable you don’t care about gives the marginal PMF: $p_X(x)=\sum_y p_{X,Y}(x,y)$. In the continuous case, this becomes an integral instead: $f_X(x)=\int_{-\infty}^{\infty}f_{X,Y}(x,y)\,dy$. This operation is called marginalization, meaning that the specific value of $Y$ is ignored, keeping only $X$‘s distribution.

After observing $Y=y$, the discrete conditional distribution is $p_{X\mid Y}(x\mid y)=p_{X,Y}(x,y)/p_Y(y)$, provided $p_Y(y)>0$. In the four-point example above, $P(Y=2)=1/4$ and only $(1,2)$ satisfies that condition, so $P(X=1\mid Y=2)=1$; without that conditioning information, $P(X=1)=1/2$. A conditional distribution quantifies how newly added information changes uncertainty, and it’s also the shared foundation of regression, classification, Bayesian inference, and sequence models.

Knowing all of the marginal distributions still isn’t enough to reconstruct the joint distribution. Two Bernoulli$(1/2)$ variables could be mutually independent, could satisfy $Y=X$, or could satisfy $Y=1-X$; all three cases have the same marginal distributions, but completely different joint behavior. Stochastic processes therefore must study the joint distribution across different points in time, rather than just listing distributions point by point; for a more complete extension, see Stochastic Processes 2: Joint Distributions, Correlation, and Dependence Structure.

5. Independence, Covariance, and Correlation

5.1 Independence Is a Requirement on the Entire Joint Distribution

Discrete variables $X$ and $Y$ are independent when every possible combination satisfies $p_{X,Y}(x,y)=p_X(x)p_Y(y)$; continuous variables are independent when $f_{X,Y}(x,y)=f_X(x)f_Y(y)$. Equivalently, knowing $Y$ doesn’t change $X$‘s conditional distribution. Independence is a stronger claim than “the two variables don’t appear to show a trend” — it must be supported by the joint distribution itself, or by a reasonable generating mechanism.

If $X$ and $Y$ are independent and the relevant expectations exist, then $E[g(X)h(Y)]=E[g(X)]E[h(Y)]$ holds for any suitable functions $g,h$. The reverse generally doesn’t hold: one particular expectation factoring doesn’t prove that the entire joint distribution factors.

5.2 Covariance and the Correlation Coefficient

Covariance $\operatorname{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])]=E[XY]-E[X]E[Y]$ describes linear co-movement. A positive value indicates the two variables tend to deviate from their respective means in the same direction, a negative value indicates opposite directions, and zero indicates no linear co-movement. Covariance is affected by units; converting meters to centimeters scales the covariance up by a factor of 100.

The Pearson correlation coefficient $\rho_{X,Y}=\operatorname{Cov}(X,Y)/(\sigma_X\sigma_Y)$ removes the scale, landing within $[-1,1]$, provided both standard deviations are greater than 0. $\rho$ close to 1 or -1 indicates a strong linear relationship, while $\rho$ close to 0 only indicates a weak linear relationship. A correlation coefficient provides no direction of causation, and it can also be misled by extreme values, mixed subgroups, or nonlinear relationships.

Zero correlation doesn’t imply independence. Let $X$ be uniformly distributed on $[-1,1]$, and let $Y=X^2$. Symmetry gives $E[X]=0$ and $E[X^3]=0$, so $\operatorname{Cov}(X,Y)=E[X^3]-E[X]E[X^2]=0$; yet $Y$ is entirely determined by $X$, and the two are clearly not independent. Independence together with the existence of second-order moments implies zero covariance; zero covariance only implies independence when additional conditions hold, such as jointly Normal.

5.3 The Covariance Matrix and Linear Combinations

For a $d$-dimensional random vector $\mathbf X$, the mean vector is written $\boldsymbol\mu=E[\mathbf X]$, and the covariance matrix is written $\boldsymbol\Sigma=E[(\mathbf X-\boldsymbol\mu)(\mathbf X-\boldsymbol\mu)^\mathsf T]$. The matrix entries are $\Sigma_{ij}=\operatorname{Cov}(X_i,X_j)$; the diagonal holds each component’s variance, and the off-diagonal holds the pairwise covariances. A covariance matrix is always symmetric and positive semi-definite, so any vector $\mathbf a$ satisfies $\operatorname{Var}(\mathbf a^\mathsf T\mathbf X)=\mathbf a^\mathsf T\boldsymbol\Sigma\mathbf a\ge0$. The multivariate Normal distribution $N_d(\boldsymbol\mu,\boldsymbol\Sigma)$ is determined by the mean vector and covariance matrix; within this distribution family, zero covariance also implies independence between the corresponding components.

Suppose two services’ per-minute load are $X_1,X_2$, each with a standard deviation of 10. If the correlation coefficient is 0.8, the variance of their combined load is $10^2+10^2+2(0.8)(10)(10)=360$, giving a standard deviation of about 18.97; if the two are independent instead, the combined load’s standard deviation is $\sqrt{200}\approx14.14$. With the same per-machine fluctuation, positive correlation still increases the uncertainty of the total capacity. Resource allocation, portfolio construction, and multi-sensor fusion all need to preserve the covariance matrix — you can’t just keep each component’s standard deviation separately.

A distribution describes the possible values and their probabilities, a numerical summary compresses the features of a distribution, and a joint distribution fills in the patterns of how multiple variables occur together. With these three levels of distinction in place, the parameter estimation, confidence intervals, the law of large numbers, and the central limit theorem that follow all have a clear subject: which distribution the data comes from, what the unknown parameters are, and how a statistic varies under repeated sampling.