Chapter 5 — Front Matter
Purpose and Overview
The first four chapters of this book studied one random variable at a time. A single die, a single waiting time, a single measurement drawn from a normal distribution — these are important and instructive models. But real uncertainty rarely comes alone.
A patient's blood pressure and cholesterol level vary together. The temperature and humidity at a weather station are related. The height and weight of an individual move in concert. The demand for two products in a market may be positively associated, or they may compete for the same customers and move in opposite directions. The error in a sensor at one time is related to the error at the next. In each of these cases, understanding one quantity requires paying attention to the other.
This chapter develops the mathematics for studying two or more random variables simultaneously. The central object is the joint distribution, which describes how probability is spread across all combinations of values for a collection of random variables. From the joint distribution, we can recover information about each variable separately, describe the conditional behavior of one variable given what we know about another, measure the degree to which variables move together, and decide whether variables are statistically independent.
Opening Question
A system contains two components with lifetimes X and Y, measured in months. The system fails when the first component fails. To compute the probability that the system survives for at least t months, we need the probability that both X > t and Y > t simultaneously. No single random variable is sufficient — we need a model that assigns probability to pairs (X, Y) at once. By the end of this chapter, you will have the tools to build and analyze exactly such models.
14Sections
12Objectives
18Key Terms
20+Worked Examples
Learning Objectives
- 1Explain why multiple random variables are needed to model real uncertainty and identify the four core questions a joint distribution answers.
- 2Define the joint CDF, joint PMF, and joint PDF and verify that a given function is a valid joint distribution.
- 3Compute probabilities from a joint PMF table and from a joint PDF by integration over regions.
- 4Derive marginal distributions from joint distributions by summing or integrating out one variable, including over non-rectangular supports.
- 5Compute and interpret conditional PMFs, conditional PDFs, and conditional expectations.
- 6Test whether two random variables are independent using the factorization condition and explain why a non-rectangular support rules out independence.
- 7Compute the expected value of a function of two random variables using the bivariate law of the unconscious statistician.
- 8Define covariance, compute it from a joint distribution, and apply the variance-of-a-sum formula.
- 9Define and compute the correlation coefficient and interpret its value, including distinguishing correlation from causation.
- 10Find the mean and variance of any linear combination of random variables using the appropriate formulas.
- 11Apply the convolution formula to compute the distribution of a sum of independent random variables and identify closure results for common families.
- 12Identify and correct common errors in joint distribution work, including misapplied limits, incorrect variance formulas, and the independence-versus-zero-covariance confusion.
Sections 5.1 – 5.4
The Joint Distribution: Structure and Specification
Why Multiple Variables Matter · Joint CDF · Joint PMF · Joint PDF
Section 5.1
Why Multiple Random Variables Matter
When a single quantity is uncertain, a random variable assigns probability to its possible values. When two or more quantities are simultaneously uncertain — and when those quantities may be related — a single random variable is no longer sufficient.
Example 5.1.1
A medical study — sleep and cognition
- SetupA study records, for each participant, the number of hours of sleep per night X and the score on a cognitive function test Y. Both X and Y are random: different participants have different sleep habits and different test scores.
- ProblemPeople who sleep fewer hours tend to score lower. Any useful model must describe both variables together, not in isolation.
- ConclusionA single random variable is not enough. We need a joint model that assigns probabilities to pairs (X, Y) simultaneously.
Example 5.1.2
Component lifetimes and system reliability
- SetupA system contains two components with lifetimes X and Y (months). The system fails when the first component fails.
- QuestionWhat is P(system survives at least t months) = P(X > t and Y > t)?
- ConclusionThis requires a model that assigns probability to pairs (X, Y) simultaneously — a joint distribution.
Definition 5.1.1 — Joint Distribution
Let X and Y be random variables defined on the same probability space. The joint distribution of X and Y is a complete specification of the probabilities assigned to all pairs (X, Y). It describes how probability is distributed across the two-dimensional space of possible outcomes.
The Four Core Questions
| Question | Answer Comes From |
|---|
| P(X and Y fall in some region)? | Direct calculation from the joint distribution |
| What is the behavior of X alone, ignoring Y? | The marginal distribution of X |
| How does X behave given we know Y? | The conditional distribution of X given Y |
| How do X and Y move together? | Covariance and correlation |
Why Calculus Is Involved
For discrete random variables, joint distributions involve sums over pairs of values. For continuous random variables, they involve double integrals over regions of the plane. Probability is still area — now area in two dimensions, computed by integrating a surface rather than a curve.
Section 5.2
Joint Distributions
When X and Y are random variables, an outcome of the underlying experiment produces a pair (X(ω), Y(ω)) for each sample point ω. The joint distribution of X and Y is determined by asking: for any set A in two-dimensional space, what is the probability that (X, Y) falls in A?
Definition 5.2.2 — Joint CDF
The joint cumulative distribution function of X and Y is
F_(X,Y)(x, y) = P(X ≤ x, Y ≤ y)
defined for all real numbers x and y. The notation P(X ≤ x, Y ≤ y) means P(X ≤ x and Y ≤ y). Its value ranges from 0 to 1 and is non-decreasing in each argument separately.
Properties of the Joint CDF
- (i)F(x,y) is non-decreasing in x for fixed y, and non-decreasing in y for fixed x.
- (ii)lim F(x,y) → 0 as x → −∞ or y → −∞.
- (iii)lim F(x,y) → 1 as x → ∞ and y → ∞.
- (iv)F is right-continuous in each argument.
- (v)Rectangle formula: P(a < X ≤ b, c < Y ≤ d) = F(b,d) − F(a,d) − F(b,c) + F(a,c).
Example 5.2.1
Using the rectangle formula
- ProblemCompute P(1 < X ≤ 3, 2 < Y ≤ 5) from a known joint CDF.
- ApplyBy property (v), evaluate the joint CDF at all four corners of the rectangle and combine with alternating signs:
P(1 < X ≤ 3, 2 < Y ≤ 5)
= F(3,5) − F(1,5) − F(3,2) + F(1,2)
- NoteThis is the two-dimensional analogue of P(a < X ≤ b) = F(b) − F(a).
Section 5.3
Joint Probability Mass Functions
Definition 5.3.1 — Joint PMF
Let X and Y be discrete random variables. The joint probability mass function is
p(x, y) = P(X = x, Y = y)
A function p(x,y) is a valid joint PMF if and only if: (i) p(x,y) ≥ 0 for all (x,y), and (ii) the double sum over all (x,y) in the support equals 1.
Example 5.3.2
A joint PMF table
- Table
| Y = 1 | Y = 2 | Y = 3 |
|---|
| X = 1 | 0.10 | 0.20 | 0.15 |
| X = 2 | 0.15 | 0.25 | 0.15 |
Verification: 0.10 + 0.20 + 0.15 + 0.15 + 0.25 + 0.15 = 1.00 ✓
Example 5.3.3
Computing a probability from a joint PMF
- ProblemUsing the table above, compute P(X + Y ≤ 3).
- ThinkIdentify all pairs (x, y) with x + y ≤ 3: these are (1,1), (1,2), and (2,1).
- Solve
P(X + Y ≤ 3) = p(1,1) + p(1,2) + p(2,1)
= 0.10 + 0.20 + 0.15 = 0.45
Answer: P(X + Y ≤ 3) = 0.45
Section 5.4
Joint Probability Density Functions
Definition 5.4.1 — Joint PDF
Let X and Y be continuous. A function f(x,y) is the joint probability density function if
P((X, Y) ∈ A) = ∬_A f(x, y) dx dy
for all appropriate regions A. A valid joint PDF satisfies: (i) f(x,y) ≥ 0 everywhere, and (ii) the double integral over all of ℝ² equals 1. Geometrically, probability is the volume under the density surface above A.
Connection to the Joint CDF
When f(x,y) is continuous: F(x,y) = ∫∫ f(s,t) ds dt (iterated from −∞), and conversely f(x,y) = ∂²F(x,y) / (∂x ∂y). This mirrors the one-dimensional CDF–PDF relationship.
Example 5.4.1
Finding a normalization constant
- AttemptTest whether f(x,y) = 6xy on [0,1]² is a valid joint PDF.
- Compute
∫₀¹ ∫₀¹ 6xy dx dy = ∫₀¹ 6y · [x²/2]₀¹ dy = ∫₀¹ 3y dy = 3/2
- Conclusion3/2 ≠ 1, so f(x,y) = 6xy is not valid. The correct normalization gives f(x,y) = 4xy, since ∫₀¹ ∫₀¹ 4xy dx dy = 1. ✓
Valid joint PDF: f(x,y) = 4xy on [0,1]²
Example 5.4.2
Computing a probability from a joint PDF
- ProblemUsing f(x,y) = 4xy on [0,1]², compute P(X > 1/2, Y > 1/2).
- Solve
P(X > 1/2, Y > 1/2) = ∫_{1/2}^{1} ∫_{1/2}^{1} 4xy dx dy
= ∫_{1/2}^{1} 4y · [x²/2]_{1/2}^{1} dy
= ∫_{1/2}^{1} 4y · 3/8 dy = (3/2) · [y²/2]_{1/2}^{1}
= (3/2) · (1/2 − 1/8) = (3/2)(3/8) = 9/16
Answer: P(X > 1/2, Y > 1/2) = 9/16
Example 5.4.3
Non-rectangular support — finding c
- ProblemLet f(x,y) = c for 0 < x < y < 1 and 0 otherwise. Find c.
- ThinkThe support is a triangle. The outer integral runs over y from 0 to 1; for fixed y, x runs from 0 to y.
- Solve
∫₀¹ ∫₀^y c dx dy = ∫₀¹ cy dy = c/2 = 1 → c = 2
Answer: c = 2. So f(x,y) = 2 on the triangle {0 < x < y < 1}.
Student Warning — Limits on Non-Rectangular Supports
When the support is triangular or otherwise non-rectangular, the limits of integration depend on which variable you integrate first. Always draw a sketch of the support region and set up limits from the geometry, not by rote.
Sections 5.5 – 5.7
Marginal Distributions, Conditional Distributions, and Independence
Marginalizing · Conditioning · Independence and Factorization
Section 5.5
Marginal Distributions
The joint distribution contains complete information about both variables. When we want to focus on just one variable and ignore the other, we recover the marginal distribution by summing or integrating out the unwanted variable. The word "marginal" comes from the practice of writing row and column totals in the margins of a joint probability table.
Marginal Distributions — Formulas
Discrete: p_X(x) = Σ_y p(x,y) p_Y(y) = Σ_x p(x,y)
Continuous: f_X(x) = ∫ f(x,y) dy f_Y(y) = ∫ f(x,y) dx
Each marginal is itself a valid distribution and must integrate (or sum) to 1.
Example 5.5.1
Marginals from a joint PMF table
- Table with Marginals
| Y = 1 | Y = 2 | Y = 3 | p_X(x) |
|---|
| X = 1 | 0.10 | 0.20 | 0.15 | 0.45 |
| X = 2 | 0.15 | 0.25 | 0.15 | 0.55 |
| p_Y(y) | 0.25 | 0.45 | 0.30 | 1.00 |
Each marginal PMF sums to 1 independently. ✓
Example 5.5.2
Marginals from f(x,y) = 4xy on [0,1]²
- Solve for f_X
f_X(x) = ∫₀¹ 4xy dy = 4x · [y²/2]₀¹ = 2x, 0 ≤ x ≤ 1
- Solve for f_Y
f_Y(y) = ∫₀¹ 4xy dx = 4y · [x²/2]₀¹ = 2y, 0 ≤ y ≤ 1
- Check∫₀¹ 2x dx = 1 ✓ ∫₀¹ 2y dy = 1 ✓
f_X(x) = 2x and f_Y(y) = 2y on [0,1].
Example 5.5.3
Marginals on a triangular support: f(x,y) = 2 on 0 < x < y < 1
- f_X(x)For fixed x, y ranges from x to 1:
f_X(x) = ∫_x^1 2 dy = 2(1 − x), 0 < x < 1
- f_Y(y)For fixed y, x ranges from 0 to y:
f_Y(y) = ∫₀^y 2 dx = 2y, 0 < y < 1
- NoteThe limits depend on the variable not being integrated — this is the hallmark of a non-rectangular support.
Section 5.6
Conditional Distributions
Conditional distributions extend the idea of conditional probability P(A|B) = P(A ∩ B)/P(B) from events to entire distributions. They answer: given that Y = y, how does X behave? A conditional distribution is a complete probability distribution in its own right.
Conditional Distributions — Formulas
Discrete: p_{X|Y}(x|y) = p(x,y) / p_Y(y)
Continuous: f_{X|Y}(x|y) = f(x,y) / f_Y(y)
Conditional Expectation:
Discrete: E[X | Y = y] = Σ_x x · p_{X|Y}(x|y)
Continuous: E[X | Y = y] = ∫ x · f_{X|Y}(x|y) dx
The conditional expectation is a function of y — it tells how the average of X changes with what we know about Y.
Example 5.6.1
Conditional PMF given Y = 2
- FromJoint PMF table (Example 5.3.2); marginal p_Y(2) = 0.45.
- Solve
p_{X|Y}(1|2) = 0.20 / 0.45 = 4/9
p_{X|Y}(2|2) = 0.25 / 0.45 = 5/9
- Check4/9 + 5/9 = 1 ✓. Compare with marginal: p_X(1) = 0.45. Knowing Y = 2 shifts probability slightly toward X = 2.
Conditional PMF of X given Y = 2: p = 4/9 at X = 1, p = 5/9 at X = 2.
Example 5.6.2
Conditional PDF: f(x,y) = 4xy on [0,1]²
- Compute
f_{X|Y}(x|y) = 4xy / f_Y(y) = 4xy / 2y = 2x, 0 ≤ x ≤ 1
- ObserveThe conditional distribution of X given Y = y does not depend on y. This is the signature of independence — explored next section.
f_{X|Y}(x|y) = 2x — independent of y.
Example 5.6.3
Conditional PDF on triangular support
- Setupf(x,y) = 2 on 0 < x < y < 1; marginal f_Y(y) = 2y.
- Solve
f_{X|Y}(x|y) = 2 / 2y = 1/y, 0 < x < y
- InterpretGiven Y = y, X is Uniform(0, y). The conditional distribution depends on y: larger y allows X to range more widely.
X | Y = y ~ Uniform(0, y). So E[X | Y = y] = y/2.
Section 5.7
Independence of Random Variables
Definition 5.7.1 — Independence
Random variables X and Y are independent if and only if for all x and y:
F(x, y) = F_X(x) · F_Y(y)
Equivalently, for discrete: p(x,y) = p_X(x) · p_Y(y)
For continuous: f(x,y) = f_X(x) · f_Y(y)
Independence means the joint distribution is entirely determined by the two marginals — knowing Y provides no information about X.
Example 5.7.1
Verifying independence: f(x,y) = 4xy on [0,1]²
- Checkf_X(x) = 2x, f_Y(y) = 2y.
f_X(x) · f_Y(y) = 2x · 2y = 4xy = f(x,y) ✓
X and Y are independent.
Example 5.7.2
Verifying dependence: f(x,y) = 2 on triangle 0 < x < y < 1
- Checkf_X(x) = 2(1−x), f_Y(y) = 2y.
f_X(x) · f_Y(y) = 2(1−x) · 2y = 4y(1−x) ≠ 2 = f(x,y)
X and Y are not independent. Knowing Y constrains where X can be.
Independence Requires a Rectangular (Product) Support
A joint distribution whose support is not a rectangle cannot have independent components. If the range of X depends on the value of Y — as in any triangular support — then X and Y are dependent by definition. Always check support geometry first.
Sections 5.8 – 5.10
Expectation, Covariance, and Correlation
Bivariate LOTUS · Covariance · Variance of Sums · Correlation
Section 5.8
Expected Value of Functions of Random Variables
Theorem 5.8.1 — Bivariate Law of the Unconscious Statistician
Let g(X, Y) be any function of two random variables. Then:
Discrete: E[g(X,Y)] = Σ_x Σ_y g(x,y) · p(x,y)
Continuous: E[g(X,Y)] = ∫∫ g(x,y) · f(x,y) dx dy
We do not need the distribution of g(X,Y) first. We weight values of g by the joint probabilities directly. Linearity of expectation — E[aX + bY + c] = aE[X] + bE[Y] + c — holds regardless of any dependence structure.
Example 5.8.1
E[X], E[Y], and E[X + Y] on the triangle f(x,y) = 2
- E[X]
E[X] = ∫₀¹ ∫_x^1 x · 2 dy dx = ∫₀¹ 2x(1−x) dx = 1 − 2/3 = 1/3
- E[Y]
E[Y] = ∫₀¹ ∫₀^y y · 2 dx dy = ∫₀¹ 2y² dy = 2/3
- E[X+Y]By linearity: E[X + Y] = 1/3 + 2/3 = 1. No need to find the distribution of X + Y first.
E[X] = 1/3, E[Y] = 2/3, E[X + Y] = 1.
Independence and Expected Products
If X and Y are independent, then E[XY] = E[X] · E[Y]. This simplification does not hold in general when X and Y are dependent.
Section 5.9
Covariance
Definition 5.9.1 — Covariance
The covariance of X and Y is
Cov(X,Y) = E[(X − μ_X)(Y − μ_Y)] = E[XY] − E[X]·E[Y]
Positive covariance: X and Y tend to move in the same direction.
Negative covariance: they tend to move in opposite directions.
Zero covariance: no linear relationship — but this does not imply independence.
Properties of Covariance
- (i)Cov(X, X) = Var(X)
- (ii)Cov(X, Y) = Cov(Y, X) — symmetry
- (iii)Cov(aX + b, cY + d) = ac · Cov(X, Y)
- (iv)Cov(X + Z, Y) = Cov(X, Y) + Cov(Z, Y) — bilinearity
- (v)If X and Y are independent, then Cov(X, Y) = 0. (The converse does not hold.)
Variance of a Sum
Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y)
When X and Y are independent, Cov(X,Y) = 0 and the cross term vanishes: Var(X + Y) = Var(X) + Var(Y).
Example 5.9.1
Computing Cov(X,Y) on the triangle
- KnownE[X] = 1/3, E[Y] = 2/3 from Example 5.8.1.
- E[XY]
E[XY] = ∫₀¹ ∫₀^y xy · 2 dx dy = ∫₀¹ 2y · y²/2 dy = ∫₀¹ y³ dy = 1/4
- Cov
Cov(X,Y) = 1/4 − (1/3)(2/3) = 1/4 − 2/9 = 9/36 − 8/36 = 1/36
- InterpretPositive covariance: large Y allows larger X, so they tend to move upward together.
Cov(X, Y) = 1/36 > 0.
Example 5.9.2
Portfolio variance and diversification
- SetupTwo assets with Var(X) = 4, Var(Y) = 9, and Cov(X,Y) = −2.
- Solve
Var(X + Y) = 4 + 9 + 2(−2) = 9
- InterpretNegative covariance reduces portfolio variance below 4 + 9 = 13. This is the mathematical foundation of diversification.
Var(X + Y) = 9, substantially below 13.
Section 5.10
Correlation
Covariance measures direction and presence of a linear relationship, but its magnitude depends on the units of X and Y. The correlation coefficient standardizes covariance to produce a dimensionless measure that always lies in [−1, 1].
Definition 5.10.1 — Correlation Coefficient
ρ_{X,Y} = Cov(X,Y) / (σ_X · σ_Y)
Properties: (i) −1 ≤ ρ ≤ 1 always. (ii) ρ = 1 iff Y = aX + b with a > 0 (perfect positive linear). (iii) ρ = −1 iff Y = aX + b with a < 0. (iv) ρ = 0 means no linear relationship — but not necessarily independence. (v) ρ is unitless and scale-invariant.
Example 5.10.1
Computing ρ on the triangular distribution
- Var(X)
E[X²] = ∫₀¹∫_x^1 x² · 2 dy dx = ∫₀¹ 2x²(1−x) dx = 2/3 − 1/2 = 1/6
Var(X) = 1/6 − (1/3)² = 1/6 − 1/9 = 1/18
- Var(Y)
E[Y²] = ∫₀¹ y² · 2y dy = ∫₀¹ 2y³ dy = 1/2
Var(Y) = 1/2 − (2/3)² = 1/2 − 4/9 = 1/18
- ρ
ρ = (1/36) / √((1/18)(1/18)) = (1/36) / (1/18) = 1/2
ρ_{X,Y} = 0.5 — a moderate positive linear relationship.
Correlation Is Not Causation
A nonzero ρ describes a statistical association in a probability model. It carries no information about which variable drives which. Causal conclusions require study design, mechanism, or intervention — not just a correlation coefficient. This theme reappears in Chapter 11 on regression.
Sections 5.11 – 5.14
Linear Combinations, Sums, Common Mistakes, and Looking Ahead
Portfolio Math · Convolution · Closure Results · Error Analysis · Bridge to Ch. 6 & 7
Section 5.11
Linear Combinations of Random Variables
Many practically important quantities are linear combinations of random variables: portfolio returns, sample means, test composite scores. Their means and variances follow directly from the properties of expectation and covariance.
Mean and Variance of a Linear Combination
Let W = a₁X₁ + a₂X₂ + ⋯ + aₙXₙ + c.
Mean (always): E[W] = a₁E[X₁] + ⋯ + aₙE[Xₙ] + c
Variance (general):
Var(W) = Σᵢ aᵢ² Var(Xᵢ) + 2 Σ_{i<j} aᵢaⱼ Cov(Xᵢ, Xⱼ)
Variance (when all mutually independent):
Var(W) = Σᵢ aᵢ² Var(Xᵢ)
Example 5.11.1
An equal-weighted three-asset portfolio
- SetupThree independent investments: μ₁ = 5%, μ₂ = 8%, μ₃ = 3%; variances: σ₁² = 4, σ₂² = 16, σ₃² = 1. Weights: a₁ = a₂ = a₃ = 1/3.
- E[W]
E[W] = (5 + 8 + 3)/3 = 16/3 ≈ 5.33%
- Var(W)
Var(W) = (1/9)(4 + 16 + 1) = 21/9 = 7/3 ≈ 2.33
E[W] ≈ 5.33%, Var(W) ≈ 2.33.
The Sample Mean
If X₁, X₂, …, Xₙ are independent and identically distributed with mean μ and variance σ², then the sample mean X̄ = (X₁ + ⋯ + Xₙ)/n satisfies:
E[X̄] = μ Var(X̄) = σ²/n
The sample mean is an unbiased estimator of μ, and its variance shrinks as n grows — a foundational result for statistical inference in Chapter 7.
Section 5.12
Sums of Independent Random Variables
For independent random variables, we can often identify the complete distribution of X + Y — not just its mean and variance. The mechanism is convolution.
Convolution Formula
For independent X and Y, the distribution of S = X + Y is:
Discrete: p_S(s) = Σ_x p_X(x) · p_Y(s − x)
Continuous: f_S(s) = ∫ f_X(x) · f_Y(s − x) dx
Also: if X and Y are independent, the MGF of X + Y is M_{X+Y}(t) = M_X(t) · M_Y(t).
Example 5.12.1
Sum of two independent Bernoulli(p) variables
- Solve
p_S(0) = (1−p)²
p_S(1) = p(1−p) + (1−p)p = 2p(1−p)
p_S(2) = p²
- RecognizeThese are exactly the Binomial(2, p) probabilities. Generalizing: the sum of n independent Bernoulli(p) variables is Binomial(n, p).
X + Y ~ Binomial(2, p).
Closure Under Convolution
| Family | Condition | Distribution of Sum |
|---|
| Binomial(nᵢ, p) | Same p; independent | Binomial(Σnᵢ, p) |
| Poisson(λᵢ) | Independent | Poisson(Σλᵢ) |
| Normal(μᵢ, σᵢ²) | Independent | Normal(Σμᵢ, Σσᵢ²) |
| Exponential(λ) | Same λ; independent | Gamma(n, λ) |
Example 5.12.2
Total request rate at a data center
- SetupThree independent Poisson sources with rates λ₁ = 10, λ₂ = 15, λ₃ = 5 requests/minute.
- ResultBy closure: total requests ~ Poisson(30).
- UseP(total ≤ 25) = Σ_{k=0}^{25} e^{−30} · 30^k / k! — computed directly from Poisson(30) without combining three separate distributions.
Total requests per minute ~ Poisson(30).
Section 5.13
Common Joint Distribution Mistakes
Mistake 1 — Confusing the Joint PDF with a Marginal
E[X] = ∫₀¹ x · f(x,y) dx (missing integration over y)
Correct: E[X] = ∫∫ x · f(x,y) dx dy, or first find f_X(x) = ∫ f(x,y) dy and then compute ∫ x · f_X(x) dx.
Mistake 2 — Zero Covariance Does Not Imply Independence
"Cov(X,Y) = 0, so X and Y are independent, so we can factor P(X > 1, Y > 1)."
Counterexample: Let X ~ Uniform(−1,1) and Y = X². Then Cov(X,Y) = E[X³] − E[X]E[X²] = 0 by symmetry, yet Y is completely determined by X. Zero covariance means no linear relationship — not independence.
Mistake 3 — Wrong Limits on Non-Rectangular Supports
f_X(x) = ∫₀¹ 2 dy = 2 for f(x,y) = 2 on 0 < x < y < 1
Correct: For fixed x, y ranges from x to 1 (not 0 to 1). f_X(x) = ∫_x^1 2 dy = 2(1 − x). Always sketch the support and set limits from the geometry.
Mistake 4 — Variance of a Sum Without the Covariance Term
Var(X + Y) = Var(X) + Var(Y) = 3 + 7 = 10 (always)
Correct: Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X,Y). The simplified formula Var(X) + Var(Y) holds only when Cov(X,Y) = 0, which requires independence or another special structure.
Mistake 5 — Applying Convolution to Dependent Variables
p_S(s) = Σ_x p_X(x) · p_Y(s − x) (for any X, Y)
Correct: The convolution formula assumes independence. For dependent X and Y, the distribution of X + Y must be computed directly from the joint distribution.
Mistake 6 — Treating Correlation as Causation
"The correlation between study hours and scores is ρ = 0.85, so more study causes higher scores."
Correct: ρ measures the strength of a linear association, not its direction of causation. Establishing causation requires controlled study design, randomization, or mechanism — not a correlation coefficient.
Section 5.14
Preparing for Transformations and Limit Theorems
This chapter developed the structure of joint distributions. The next two chapters extend this foundation in two distinct directions.
Chapter 6 will study what happens when we apply functions to random variables — individually or jointly. How is the distribution of g(X, Y) related to the joint distribution of (X, Y)? What is the change-of-variables formula in two dimensions? What do moment generating functions tell us about distributions of functions and sums? The joint PDFs, marginal distributions, and convolution ideas from this chapter are direct prerequisites.
Chapter 7 will use the structure of sums and linear combinations to prove two of the most important results in all of probability: the Law of Large Numbers and the Central Limit Theorem. The result Var(X̄) = σ²/n from Section 5.11 is already a step toward understanding why sample means concentrate around the true mean as n grows.
Students who have studied Linear Algebra will recognize that the covariance structure of a collection X₁, …, Xₙ can be assembled into a covariance matrix Σ with entries Σᵢⱼ = Cov(Xᵢ, Xⱼ). This matrix plays a central role in multivariate statistics, principal component analysis, and the multivariate normal distribution.
Chapter 5 — Back Matter
Chapter Summary
- Sec 5.1Real uncertainty often involves several quantities at once. The joint distribution describes probability over all combinations of values. Four core questions — joint probability, marginal behavior, conditional behavior, and co-movement — organize the chapter.
- Sec 5.2The joint CDF F(x,y) = P(X ≤ x, Y ≤ y) is defined for all (x,y). It satisfies monotonicity, boundary conditions, and a rectangle-probability formula with alternating-sign corner evaluations.
- Sec 5.3The joint PMF assigns probability to each pair (x,y) for discrete variables. Probabilities for any event are computed by summing p(x,y) over all pairs in that event.
- Sec 5.4The joint PDF f(x,y) is a non-negative surface over the plane with total volume 1. Probability is the volume under f above a region A, computed by double integration. Supports need not be rectangular.
- Sec 5.5Marginal distributions are obtained by summing or integrating out the other variable. For non-rectangular supports, limits depend on the variable not being integrated — always follow the support geometry.
- Sec 5.6Conditional distributions describe how one variable behaves given the value of another. The conditional PDF (or PMF) is the joint divided by the marginal of the conditioning variable. Conditional expectation is a function of the conditioning value.
- Sec 5.7X and Y are independent if and only if the joint distribution factors into the product of the marginals. A non-rectangular support rules out independence. Zero covariance does not imply independence.
- Sec 5.8E[g(X,Y)] is computed by weighting g(x,y) by the joint distribution. Linearity of expectation holds always. If X and Y are independent, E[XY] = E[X]·E[Y].
- Sec 5.9Cov(X,Y) = E[XY] − E[X]E[Y] measures linear co-movement. Var(X+Y) = Var(X) + Var(Y) + 2 Cov(X,Y). Negative covariance enables portfolio diversification.
- Sec 5.10The correlation coefficient ρ = Cov(X,Y)/(σ_X σ_Y) lies in [−1,1], is dimensionless, and measures linear association. ρ = 0 does not imply independence. Correlation is not causation.
- Sec 5.11For W = Σ aᵢXᵢ + c: E[W] = Σ aᵢE[Xᵢ] + c always; variance includes covariance cross-terms unless variables are independent. The sample mean satisfies E[X̄] = μ and Var(X̄) = σ²/n.
- Sec 5.12For independent X, Y the distribution of X + Y is computed by convolution. Binomial (same p), Poisson, Normal, and Exponential (same λ to Gamma) families are closed under convolution.
- Sec 5.13Six common errors: using a marginal instead of the full joint for E[X]; concluding independence from zero covariance; wrong integration limits on non-rectangular supports; dropping covariance from variance-of-sum; applying convolution to dependent variables; treating correlation as causation.
- Sec 5.14Chapter 6 extends joint distributions through transformation methods and moment generating functions. Chapter 7 applies sums and linear combinations to prove the Law of Large Numbers and Central Limit Theorem. The covariance matrix extends joint structure to multivariate settings.
Key Terms Review
joint distribution
A specification of probabilities for all pairs (or tuples) of outcomes of two or more random variables.
joint CDF
F(x,y) = P(X ≤ x, Y ≤ y); the two-dimensional cumulative distribution function.
joint PMF
p(x,y) = P(X = x, Y = y); for discrete random variables. Must be non-negative with double sum equal to 1.
joint PDF
f(x,y) such that P((X,Y) ∈ A) = ∬_A f(x,y) dx dy; for continuous random variables.
marginal distribution
The distribution of a single variable obtained by summing or integrating out the others from a joint distribution.
marginal PMF
p_X(x) = Σ_y p(x,y); the row (or column) totals in a joint PMF table.
marginal PDF
f_X(x) = ∫ f(x,y) dy; integrates out the other variable, with limits following the support geometry.
conditional distribution
The distribution of one variable given a fixed value of another; computed by dividing the joint by the marginal of the conditioning variable.
conditional PMF
p_{X|Y}(x|y) = p(x,y)/p_Y(y); a valid PMF in x for each fixed y with p_Y(y) > 0.
conditional PDF
f_{X|Y}(x|y) = f(x,y)/f_Y(y); a valid PDF in x for each fixed y with f_Y(y) > 0.
conditional expectation
E[X | Y = y]; the expected value of X under the conditional distribution given Y = y. A function of y.
independence
X and Y are independent if f(x,y) = f_X(x)f_Y(y) (or PMF equivalent). Requires a product (rectangular) support.
covariance
Cov(X,Y) = E[(X−μ_X)(Y−μ_Y)] = E[XY] − E[X]E[Y]; measures linear co-movement direction and magnitude.
correlation coefficient
ρ_{X,Y} = Cov(X,Y)/(σ_X σ_Y); dimensionless, always in [−1,1], measures linear association strength.
linear combination
W = a₁X₁ + ⋯ + aₙXₙ + c; a weighted sum. Its mean adds linearly; its variance uses the full covariance structure.
convolution
The formula computing the distribution of X + Y for independent X and Y; sums or integrates all ways the total can equal a fixed value.
closure under convolution
A family is closed when the sum of independent members stays in the family (e.g., Poisson, Normal, Binomial with same p).
covariance matrix
The matrix Σ with (i,j) entry Cov(Xᵢ, Xⱼ); summarizes the full joint variability structure of a collection of random variables.
Joint Distribution Practice
- Let X and Y have the joint PMF table below. (a) Verify this is a valid joint PMF. (b) Find P(X = Y). (c) Find P(X + Y > 2). (d) Find P(X ≤ 1, Y ≤ 1).
| Y = 0 | Y = 1 | Y = 2 |
|---|
| X = 0 | 0.05 | 0.10 | 0.15 |
| X = 1 | 0.20 | 0.25 | 0.10 |
| X = 2 | 0.05 | 0.05 | 0.05 |
- A fair coin is flipped three times. Let X be the number of heads on the first two flips and Y the number of heads on all three. (a) Write the complete joint PMF as a table. (b) Find the marginal PMFs of X and Y. (c) Are X and Y independent? Justify.
- Let f(x,y) = c(x + y) for 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1. (a) Find c. (b) Compute P(X < 1/2, Y < 1/2). (c) Compute P(X + Y < 1).
- Let f(x,y) = e^{−x−y} for x > 0, y > 0. (a) Verify this is a valid joint PDF. (b) Find P(X > 1, Y > 1). (c) Find P(X < Y).
- Let f(x,y) = 6x for 0 < x < y < 1. (a) Verify this is valid. (b) Find the joint CDF on the triangular support. (c) Compute P(X > 1/3, Y < 2/3).
Marginal and Conditional Practice
- Using f(x,y) = c(x + y) from Part B Problem 3 (with your c): (a) Find f_X(x). (b) Find f_Y(y). (c) Find E[X] and E[Y].
- Using f(x,y) = 6x for 0 < x < y < 1: (a) Find f_X(x). (b) Find f_Y(y). (c) Verify each integrates to 1.
- Using the joint PMF from Part A Problem 1: (a) Find the conditional PMF of Y given X = 1. (b) Find E[Y | X = 1]. (c) Compare E[Y | X = 1] with the marginal E[Y].
- Using f(x,y) = e^{−x−y} for x, y > 0: (a) Find f_{Y|X}(y|x). (b) Find E[Y | X = x]. (c) Does the conditional expectation depend on x? What does this imply?
Independence Practice
- For each joint PDF, determine whether X and Y are independent and justify algebraically. (a) f(x,y) = e^{−x−y} for x > 0, y > 0. (b) f(x,y) = c(x+y) for 0 ≤ x,y ≤ 1. (c) f(x,y) = 4xy for 0 ≤ x,y ≤ 1. (d) f(x,y) = 2 for 0 < x < y < 1.
- Suppose X and Y are independent with X ~ Exponential(2) and Y ~ Exponential(3). (a) Write the joint PDF. (b) Find P(X < Y). (c) Find P(X + Y > 1).
- Let X and Y be independent with E[X] = 3, E[Y] = 4, Var(X) = 2, Var(Y) = 5. (a) Find E[XY]. (b) Find Var(X + Y). (c) Find Var(X − 2Y).
- Give an explicit example of two random variables with Cov(X,Y) = 0 that are not independent. Verify both properties from the joint distribution.
Covariance and Correlation Practice
- Using the joint PMF from Part A Problem 1: (a) Find E[XY]. (b) Compute Cov(X,Y). (c) Find Var(X) and Var(Y). (d) Compute ρ_{X,Y}.
- Investment A: mean 6%, SD 4%. Investment B: mean 9%, SD 8%. Correlation ρ = 0.3. An investor holds weight w in A and (1−w) in B. (a) Find E[portfolio] as a function of w. (b) Find Var(portfolio) as a function of w. (c) Find the w that minimizes portfolio variance.
- Let W = 2X − 3Y + 1, where Cov(X,Y) = −1, Var(X) = 4, Var(Y) = 9. (a) Find Var(W). (b) If E[X] = 5 and E[Y] = 2, find E[W].
- Let X ~ Uniform(0,1) and Y = 1 − X. (a) Find Cov(X,Y) directly. (b) Find ρ_{X,Y}. (c) Does the result make geometric sense?
Word Problems and Applications
For each problem, set up the relevant joint distribution, identify the method, and interpret your answer in context.
- Engineering. Two system components have independent lifetimes X ~ Exponential(1) and Y ~ Exponential(2) (years). (a) Find the probability both survive at least 2 years. (b) Let T = min(X,Y) be the system lifetime. Find the CDF of T. (c) Show T ~ Exponential(3) and explain why this makes sense intuitively.
- Finance. Daily returns X and Y have E[X] = 0.1%, E[Y] = 0.15%, Var(X) = 1, Var(Y) = 4, and Cov(X,Y) = 0.5. (a) For an equal-weight portfolio W = (X + Y)/2, find E[W] and Var(W). (b) What correlation ρ would make Var(W) = Var(X)? (c) If ρ = −1, what weight gives zero variance?
- Computing. Two independent Poisson request streams arrive at rates λ₁ = 20 and λ₂ = 30 per minute. (a) What is the distribution of total requests per minute? (b) Find P(total > 60 in one minute). (c) Find the covariance between the number of stream-1 requests and the total.
- Medicine. In a clinical trial, let X = diastolic blood pressure reduction (mmHg) and Y = systolic reduction (mmHg), with marginals X ~ N(8,4) and Y ~ N(12,9) and ρ = 0.6. (a) Find Cov(X,Y). (b) Let S = X + Y be the composite score. Find E[S] and Var(S). (c) Explain in words why ρ > 0 is medically plausible.
Error Analysis
Each problem contains a conceptual or computational error. Identify the error and write the correct approach.
Problem 1
"Cov(X,Y) = 0, so X and Y are independent, so P(X > 1, Y > 1) = P(X > 1)·P(Y > 1)."
What is wrong? When does zero covariance actually allow this factorization?
Problem 2
For f(x,y) = 2 on 0 < x < y < 1: f_X(x) = ∫₀¹ 2 dy = 2.
What error was made? State the correct computation and the correct marginal.
Problem 3
"Var(X + Y) = Var(X) + Var(Y) = 3 + 7 = 10" (no additional information given).
Under what condition is this correct? What is the general formula when Cov(X,Y) = 2?
Problem 4
E[X] = ∫₀¹ x · f(x, 0.5) dx (computing expected value at a single slice y = 0.5).
What error was made? What is the correct formula using the joint PDF?
Problem 5
"The correlation between study hours and exam scores is ρ = 0.85, so more study causes higher scores."
What logical error is made? What additional evidence would support a causal conclusion?
Chapter 5 Checkpoint
Chapter 5 Checkpoint
Complete without assistance — assesses all key skills and concepts from Chapter 5
Section 1 — Joint PMF
- Let X and Y have the joint PMF:
| Y = 0 | Y = 1 |
|---|
| X = 0 | 0.30 | 0.20 |
| X = 1 | 0.10 | 0.40 |
(a) Verify this is a valid PMF. (b) Find P(X = Y). (c) Find the marginal PMF of X. (d) Find the conditional PMF of Y given X = 1. (e) Find Cov(X,Y). (f) Are X and Y independent? Justify algebraically.
Section 2 — Joint PDF
- Let f(x,y) = λ² e^{−λ(x+y)} for x > 0, y > 0, where λ > 0. (a) Verify this is a valid joint PDF. (b) Find the marginal PDFs of X and Y. (c) Are X and Y independent? (d) Find Cov(X,Y). (e) Identify the distribution of X + Y.
- Let f(x,y) = 8xy for 0 < x < y < 1. (a) Verify this is a valid joint PDF. (b) Find the marginal PDF of Y. (c) Find the conditional PDF of X given Y = y. (d) Find E[X | Y = 1/2]. (e) Find Cov(X,Y).
Section 3 — Linear Combinations
- Random variables X, Y, Z are mutually independent with E[X] = 1, E[Y] = 2, E[Z] = 3 and Var(X) = 1, Var(Y) = 4, Var(Z) = 9. (a) Find E[2X − Y + Z]. (b) Find Var(2X − Y + Z). (c) Let W̄ = (X + Y + Z)/3. Find E[W̄] and Var(W̄).
Section 4 — Conceptual Questions
- Explain in two to four sentences why a joint distribution on a triangular support cannot correspond to independent random variables.
- Explain the difference between Cov(X,Y) = 0 and X and Y being independent. Give a specific counterexample.
- Explain why Var(X̄) = σ²/n decreases as n increases, and state what this implies for statistical estimation.
Looking Ahead
Chapter 6: Transformations, Expectation, and Moment Methods
Chapter 5 gave you the vocabulary and mechanics of joint distributions. Chapter 6 will deepen distribution theory through transformations and moments. You will study how to find the distribution of g(X, Y), including the change-of-variables technique in two dimensions. You will also develop moment generating functions — powerful tools that characterize distributions through their moments and make sums of independent variables tractable. The convolution results and linear combination formulas from this chapter are direct prerequisites for everything in Chapter 6, which in turn prepares you for the Law of Large Numbers and Central Limit Theorem in Chapter 7.
MGU Library Connection
- Distribution Reference — Appendix F (Discrete) and Appendix G (Continuous) of this textbook
- Calculus Readiness Review — Appendix A: double integrals, iterated integrals, integration over non-rectangular regions
- Linear Algebra, Chapter 5: vectors and matrices — for the covariance matrix extension in multivariate statistics
- Chapters 3 and 4 of this textbook: single-variable PMFs, PDFs, expected value, variance, and common distributions
- MGU Dictionary entries: joint distribution, marginal distribution, conditional distribution, covariance, correlation, convolution, independence