The Central Idea
A joint distribution describes the probability behavior of multiple random variables simultaneously.
For two random variables \(X\) and \(Y\), the joint distribution tells us how probability is assigned to pairs
\[ (X,Y). \]Instead of asking only
\[ P(X=x), \]we may ask
\[ \boxed{P(X=x,\;Y=y)}. \]Why Joint Distributions Matter
Joint distributions are needed whenever two quantities may be related.
Examples include:
- height and weight;
- price and demand;
- temperature and energy use;
- two asset returns;
- arrival time and service duration;
- the outcomes of two related measurements.
Discrete Joint Distributions
If \(X\) and \(Y\) are discrete, their joint probability mass function is
\[ \boxed{ p_{X,Y}(x,y)=P(X=x,\;Y=y) }. \]The function assigns probability directly to ordered pairs of possible values.
Requirements for a Joint PMF
A valid joint PMF must satisfy:
\[ \boxed{p_{X,Y}(x,y)\ge0} \]for every pair \((x,y)\), and
\[ \boxed{ \sum_x\sum_y p_{X,Y}(x,y)=1 }. \]The total probability across all possible pairs must equal \(1\).
Example: A Joint Probability Table
Suppose \(X\in\{0,1\}\) and \(Y\in\{0,1,2\}\), with joint PMF:
| \(X\backslash Y\) | \(0\) | \(1\) | \(2\) |
|---|---|---|---|
| \(0\) | \(0.10\) | \(0.20\) | \(0.15\) |
| \(1\) | \(0.15\) | \(0.25\) | \(0.15\) |
The entries sum to
\[ 0.10+0.20+0.15+0.15+0.25+0.15=1. \]Therefore the table defines a valid joint PMF.
Reading a Joint Probability
From the table,
\[ P(X=1,Y=1)=0.25. \]Therefore,
\[ \boxed{p_{X,Y}(1,1)=0.25}. \]A single table cell corresponds to a probability involving both random variables simultaneously.
Probabilities Over Several Pairs
If an event contains several possible pairs, add their joint probabilities.
For example,
\[ P(X=1,\;Y\ge1) = P(X=1,Y=1)+P(X=1,Y=2). \]Using the table,
\[ P(X=1,\;Y\ge1) = 0.25+0.15 = \boxed{0.40}. \]Marginal Distributions
A marginal distribution describes one variable by itself, ignoring the value of the other variable.
For discrete variables,
\[ \boxed{ p_X(x)=\sum_y p_{X,Y}(x,y) } \]and
\[ \boxed{ p_Y(y)=\sum_x p_{X,Y}(x,y) }. \]Finding the Marginal Distribution of \(X\)
Using the example table,
\[ P(X=0) = 0.10+0.20+0.15 = 0.45. \]Also,
\[ P(X=1) = 0.15+0.25+0.15 = 0.55. \]Therefore,
\[ \boxed{ p_X(0)=0.45,\qquad p_X(1)=0.55 }. \]Finding the Marginal Distribution of \(Y\)
\[ P(Y=0) = 0.10+0.15 = 0.25, \] \[ P(Y=1) = 0.20+0.25 = 0.45, \]and
\[ P(Y=2) = 0.15+0.15 = 0.30. \]Thus,
\[ \boxed{ p_Y(0)=0.25,\quad p_Y(1)=0.45,\quad p_Y(2)=0.30 }. \]Why They Are Called Marginals
In a joint probability table, marginal probabilities are often written in the row and column margins.
Row sums give the distribution of one variable, and column sums give the distribution of the other.
Conditional Distributions
A conditional distribution describes one random variable after information about another variable is known.
For discrete variables,
\[ \boxed{ P(X=x\mid Y=y) = \frac{P(X=x,Y=y)}{P(Y=y)} } \]whenever
\[ P(Y=y)>0. \]Example: Conditional Probability from a Joint Table
Using the joint table, find
\[ P(X=1\mid Y=1). \]We have
\[ P(X=1,Y=1)=0.25 \]and
\[ P(Y=1)=0.45. \]Therefore,
\[ \boxed{ P(X=1\mid Y=1) = \frac{0.25}{0.45} = \frac59 \approx0.5556 }. \]Conditional PMFs Must Sum to 1
Once \(Y=y\) is fixed, the conditional probabilities over all possible values of \(X\) must sum to \(1\):
\[ \boxed{ \sum_x P(X=x\mid Y=y)=1 }. \]Conditioning creates a new probability distribution.
Independence of Random Variables
Random variables \(X\) and \(Y\) are independent when knowledge of one does not change the probability distribution of the other.
For discrete variables, independence means
\[ \boxed{ p_{X,Y}(x,y)=p_X(x)p_Y(y) } \]for every possible pair \((x,y)\).
Checking Independence
Using the example table,
\[ p_{X,Y}(0,0)=0.10. \]But
\[ p_X(0)p_Y(0) = 0.45(0.25) = 0.1125. \]Since
\[ 0.10\ne0.1125, \]the variables are not independent.
Independence and Conditional Distributions
If \(X\) and \(Y\) are independent, then
\[ P(X=x\mid Y=y)=P(X=x) \]whenever the conditional probability is defined.
Learning \(Y=y\) provides no new information about \(X\).
Continuous Joint Distributions
If \(X\) and \(Y\) are continuous, their joint probability density function is denoted
\[ f_{X,Y}(x,y). \]Probabilities are obtained by integrating over regions in the \(xy\)-plane.
Requirements for a Joint PDF
A valid joint density satisfies
\[ \boxed{ f_{X,Y}(x,y)\ge0 } \]and
\[ \boxed{ \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dy\,dx = 1 }. \]Probability Over a Region
If \(R\) is a region in the plane, then
\[ \boxed{ P((X,Y)\in R) = \iint_R f_{X,Y}(x,y)\,dA }. \]Joint continuous probability is represented by volume under the density surface over the region \(R\).
Example: Constant Joint Density on a Rectangle
Suppose
\[ f_{X,Y}(x,y)=c \]on
\[ 0\le x\le2,\qquad 0\le y\le3, \]and \(0\) elsewhere.
The support rectangle has area
\[ 2\cdot3=6. \]Since total probability must equal \(1\),
\[ 6c=1. \]Therefore,
\[ \boxed{c=\frac16}. \]Probability in a Subrectangle
Using the preceding density, find
\[ P(0\le X\le1,\;0\le Y\le2). \]The subrectangle has area
\[ 1\cdot2=2. \]Therefore,
\[ P = 2\left(\frac16\right) = \boxed{\frac13}. \]Marginal Densities
For continuous random variables,
\[ \boxed{ f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dy } \]and
\[ \boxed{ f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dx }. \]Integration removes the variable that is not being retained.
Marginals from the Rectangle Example
Since
\[ f_{X,Y}(x,y)=\frac16 \]on \(0\le x\le2\), \(0\le y\le3\),
\[ f_X(x) = \int_0^3\frac16\,dy = \frac12 \]for \(0\le x\le2\).
Similarly,
\[ f_Y(y) = \int_0^2\frac16\,dx = \frac13 \]for \(0\le y\le3\).
Conditional Densities
For continuous random variables, the conditional density of \(X\) given \(Y=y\) is
\[ \boxed{ f_{X\mid Y}(x\mid y) = \frac{f_{X,Y}(x,y)}{f_Y(y)} } \]whenever
\[ f_Y(y)>0. \]Continuous Independence
Continuous random variables \(X\) and \(Y\) are independent when
\[ \boxed{ f_{X,Y}(x,y)=f_X(x)f_Y(y) } \]for all relevant values.
In the rectangle example,
\[ f_X(x)f_Y(y) = \frac12\cdot\frac13 = \frac16, \]which equals the joint density on the rectangular support. Therefore those variables are independent.
Support Shape Can Reveal Dependence
If the joint support is not a product of separate \(x\)- and \(y\)-ranges, independence may be impossible.
For example, suppose probability is restricted to
\[ 0\le y\le x\le1. \]Knowing \(X=x\) changes the allowable range of \(Y\), which signals dependence.
Joint Cumulative Distribution Function
The joint CDF of \(X\) and \(Y\) is
\[ \boxed{ F_{X,Y}(x,y) = P(X\le x,\;Y\le y) }. \]It accumulates probability over the lower-left region determined by \((x,y)\).
Recovering a Continuous Joint Density
If the joint CDF is sufficiently differentiable, then
\[ \boxed{ f_{X,Y}(x,y) = \frac{\partial^2}{\partial x\,\partial y} F_{X,Y}(x,y) }. \]This extends the one-variable relationship between a CDF and a PDF.
Expected Value of a Function of Two Variables
For discrete \(X\) and \(Y\),
\[ \boxed{ \mathbb{E}[g(X,Y)] = \sum_x\sum_y g(x,y)p_{X,Y}(x,y) }. \]For continuous variables,
\[ \boxed{ \mathbb{E}[g(X,Y)] = \iint g(x,y)f_{X,Y}(x,y)\,dA }. \]Expected Value of a Sum
Regardless of dependence,
\[ \boxed{ \mathbb{E}[X+Y] = \mathbb{E}[X]+\mathbb{E}[Y] }. \]Joint distributions are not needed to establish linearity of expectation, although they are needed for many other combined quantities.
Expected Product
The expected product is
\[ \boxed{ \mathbb{E}[XY] }. \]For discrete variables,
\[ \mathbb{E}[XY] = \sum_x\sum_y xy\,p_{X,Y}(x,y). \]This quantity plays a central role in covariance.
Covariance
The covariance of \(X\) and \(Y\) measures whether they tend to vary together:
\[ \boxed{ \operatorname{Cov}(X,Y) = \mathbb{E} \left[ (X-\mu_X)(Y-\mu_Y) \right] }. \]Computational Covariance Formula
Expanding the product gives
\[ \boxed{ \operatorname{Cov}(X,Y) = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y] }. \]This form is often easier to calculate.
Interpreting Covariance
- Positive covariance: larger values of one variable tend to occur with larger values of the other.
- Negative covariance: larger values of one tend to occur with smaller values of the other.
- Covariance near zero: there may be little linear co-movement, although other forms of dependence may still exist.
Independence Implies Zero Covariance
If \(X\) and \(Y\) are independent and the relevant expectations exist, then
\[ \mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]. \]Therefore,
\[ \boxed{ \operatorname{Cov}(X,Y)=0 }. \]Zero Covariance Does Not Usually Imply Independence
The converse is generally false.
Two random variables can have
\[ \operatorname{Cov}(X,Y)=0 \]and still be dependent through a nonlinear relationship.
Independence is a stronger condition than zero covariance.
Correlation
Covariance depends on the units of \(X\) and \(Y\). Correlation removes those units by standardizing:
\[ \boxed{ \rho_{X,Y} = \frac{ \operatorname{Cov}(X,Y) }{ \sigma_X\sigma_Y } }. \]This requires both standard deviations to be positive.
Range of Correlation
Correlation always satisfies
\[ \boxed{ -1\le\rho_{X,Y}\le1 }. \]- \(\rho=1\): perfect positive linear relationship;
- \(\rho=-1\): perfect negative linear relationship;
- \(\rho=0\): no linear correlation.
Correlation Does Not Establish Causation
A strong correlation does not by itself show that changes in one variable cause changes in the other.
Correlation describes association within a joint distribution. Causal interpretation requires additional assumptions, design, or evidence.
Variance of a Sum
Joint behavior affects the variance of sums:
\[ \boxed{ \operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y) }. \]Dependence enters through the covariance term.
Variance of a Difference
Similarly,
\[ \boxed{ \operatorname{Var}(X-Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) - 2\operatorname{Cov}(X,Y) }. \]Independent Sums
If \(X\) and \(Y\) are independent, then covariance is zero, so
\[ \boxed{ \operatorname{Var}(X+Y) = \operatorname{Var}(X)+\operatorname{Var}(Y) }. \]This is the special case used in many earlier probability models.
Common Error: Confusing Joint and Marginal Probability
The joint probability
\[ P(X=x,Y=y) \]describes two conditions simultaneously.
The marginal probability
\[ P(X=x) \]sums over all possible values of \(Y\).
Common Error: Reading a Row or Column as a Joint Probability
A single cell in a joint PMF table is a joint probability.
A row sum or column sum is a marginal probability.
Common Error: Forgetting to Normalize a Conditional Distribution
Conditional probability requires division by the probability of the conditioning event:
\[ P(X=x\mid Y=y) = \frac{P(X=x,Y=y)}{P(Y=y)}. \]The numerator alone is only a joint probability.
Common Error: Checking Independence at Only One Pair
Independence requires
\[ p_{X,Y}(x,y)=p_X(x)p_Y(y) \]for every possible pair.
One matching pair is not sufficient to establish independence.
Common Error: Assuming Zero Correlation Means Independence
Zero correlation only rules out linear association in the covariance sense.
Nonlinear dependence may still remain.
Common Error: Adding Variances Under Dependence
In general,
\[ \operatorname{Var}(X+Y) \ne \operatorname{Var}(X)+\operatorname{Var}(Y). \]The covariance term must be included unless independence or zero covariance is justified.
Common Error: Treating a Joint Density as Point Probability
For continuous random variables,
\[ P(X=x,Y=y)=0. \]The value \(f_{X,Y}(x,y)\) is a density, not a point probability.
A Reliable Procedure for Discrete Joint Distributions
- Identify the possible values of \(X\) and \(Y\).
- Verify that all joint PMF entries are nonnegative.
- Verify that all joint probabilities sum to \(1\).
- Use row or column sums to find marginals.
- Use joint divided by marginal probability for conditionals.
- Check factorization into marginals to test independence.
- Use the joint PMF for expectations involving both variables.
A Reliable Procedure for Continuous Joint Distributions
- Identify the support region.
- Check that the joint density is nonnegative.
- Verify that the total double integral equals \(1\).
- Integrate over regions to find event probabilities.
- Integrate out one variable to find marginals.
- Divide joint density by the appropriate marginal for conditionals.
- Check factorization into marginal densities for independence.
Practice 1: Read a Joint Probability
Suppose
\[ P(X=1,Y=2)=0.18. \]Interpret this statement.
Practice 2: Find a Marginal Probability
Suppose
\[ P(X=0,Y=0)=0.20, \] \[ P(X=0,Y=1)=0.30, \] \[ P(X=1,Y=0)=0.10, \]and
\[ P(X=1,Y=1)=0.40. \]Find \(P(X=1)\).
Practice 3: Conditional Probability
Using Practice 2, find
\[ P(X=1\mid Y=1). \]Practice 4: Check Independence
Using Practice 2, determine whether \(X\) and \(Y\) are independent.
Practice 5: Constant Joint Density
Suppose
\[ f_{X,Y}(x,y)=c \]on
\[ 0\le x\le4,\qquad 0\le y\le2, \]and \(0\) elsewhere.
Find \(c\).
Practice 6: Probability Over a Subregion
Using Practice 5, find
\[ P(0\le X\le2,\;0\le Y\le1). \]Practice 7: Covariance
Suppose
\[ \mathbb{E}[X]=2, \qquad \mathbb{E}[Y]=5, \qquad \mathbb{E}[XY]=13. \]Find \(\operatorname{Cov}(X,Y)\).
Practice 8: Correlation
Suppose
\[ \operatorname{Cov}(X,Y)=6, \qquad \sigma_X=2, \qquad \sigma_Y=4. \]Find the correlation.
Practice 9: Variance of a Sum
Suppose
\[ \operatorname{Var}(X)=3, \qquad \operatorname{Var}(Y)=5, \qquad \operatorname{Cov}(X,Y)=2. \]Find
\[ \operatorname{Var}(X+Y). \]Practice 10: Identify the Error
A student says:
“Because \(\operatorname{Cov}(X,Y)=0\), the variables must be independent.”
Explain the error.
Practice Answers
Practice 1
The probability that \(X\) equals \(1\) and \(Y\) equals \(2\) in the same outcome is
\[ \boxed{0.18}. \]Practice 2
\[ P(X=1) = P(X=1,Y=0)+P(X=1,Y=1). \] \[ = 0.10+0.40 = \boxed{0.50}. \]Practice 3
First,
\[ P(Y=1)=0.30+0.40=0.70. \]Therefore,
\[ P(X=1\mid Y=1) = \frac{0.40}{0.70} = \boxed{\frac47\approx0.5714}. \]Practice 4
We have
\[ P(X=1)=0.50 \]and
\[ P(Y=1)=0.70. \]If independent, we would need
\[ P(X=1,Y=1)=0.50(0.70)=0.35. \]But the joint probability is \(0.40\). Therefore,
\[ \boxed{X\text{ and }Y\text{ are not independent}}. \]Practice 5
The support rectangle has area
\[ 4\cdot2=8. \]Therefore,
\[ 8c=1 \]and
\[ \boxed{c=\frac18}. \]Practice 6
The subrectangle has area
\[ 2\cdot1=2. \]Therefore,
\[ P = 2\left(\frac18\right) = \boxed{\frac14}. \]Practice 7
\[ \operatorname{Cov}(X,Y) = 13-(2)(5) = \boxed{3}. \]Practice 8
\[ \rho = \frac{6}{(2)(4)} = \boxed{0.75}. \]Practice 9
\[ \operatorname{Var}(X+Y) = 3+5+2(2). \]Therefore,
\[ \boxed{\operatorname{Var}(X+Y)=12}. \]Practice 10
Zero covariance means there is no linear co-movement in the covariance sense, but nonlinear dependence may still exist. Independence implies zero covariance when expectations exist, but the converse is not true in general.
What to Remember
For discrete variables,
\[ \boxed{ p_{X,Y}(x,y)=P(X=x,Y=y) }. \]Marginals are obtained by summing:
\[ \boxed{ p_X(x)=\sum_y p_{X,Y}(x,y) }. \]For continuous variables,
\[ \boxed{ f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dy }. \]Independence requires factorization:
\[ \boxed{ p_{X,Y}(x,y)=p_X(x)p_Y(y) } \]or, in the continuous case,
\[ \boxed{ f_{X,Y}(x,y)=f_X(x)f_Y(y) }. \]Covariance is
\[ \boxed{ \operatorname{Cov}(X,Y) = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y] }. \]Correlation is
\[ \boxed{ \rho_{X,Y} = \frac{\operatorname{Cov}(X,Y)} {\sigma_X\sigma_Y} }. \]The central interpretation is: a joint distribution describes how random variables behave together, while marginals, conditionals, covariance, and correlation summarize different aspects of that joint behavior.
Where We Go Next
Joint distributions describe several random variables simultaneously. Another important question is how the distribution changes when a random variable is algebraically transformed.
The next lesson, Transformations of Random Variables, develops methods for finding distributions, means, and variances after applying functions to random variables.