The Central Idea
Suppose
\[ X_1,X_2,\ldots,X_n \]are repeated observations from the same probability model.
The Law of Large Numbers says that the sample average tends to stabilize near the population mean:
\[ \boxed{ \overline{X}_n = \frac{X_1+\cdots+X_n}{n} \longrightarrow \mu }. \]The Central Limit Theorem says that, under standard conditions, the properly standardized sum or sample mean becomes approximately normal as \(n\) grows.
The Sample Mean
Given observations
\[ X_1,X_2,\ldots,X_n, \]the sample mean is
\[ \boxed{ \overline{X} = \frac{1}{n} \sum_{i=1}^{n}X_i }. \]It is itself a random variable because it depends on random observations.
Expected Value of the Sample Mean
If each \(X_i\) has mean
\[ \mu, \]then by linearity of expectation,
\[ \mathbb{E}[\overline{X}] = \frac1n \sum_{i=1}^{n} \mathbb{E}[X_i]. \]Therefore,
\[ \boxed{ \mathbb{E}[\overline{X}]=\mu }. \]The sample mean is centered at the population mean.
Variance of the Sample Mean
If the \(X_i\) are independent and each has variance
\[ \sigma^2, \]then
\[ \operatorname{Var}(\overline{X}) = \operatorname{Var} \left( \frac1n \sum_{i=1}^{n}X_i \right). \]Independence allows the variances to add:
\[ \boxed{ \operatorname{Var}(\overline{X}) = \frac{\sigma^2}{n} }. \]Standard Error of the Sample Mean
Taking the positive square root gives
\[ \boxed{ \sigma_{\overline{X}} = \frac{\sigma}{\sqrt{n}} }. \]This quantity is called the standard error of the sample mean when the population standard deviation \(\sigma\) is known.
Why Larger Samples Produce More Stable Means
As \(n\) increases,
\[ \frac{\sigma}{\sqrt{n}} \]decreases.
Therefore the distribution of the sample mean becomes more concentrated around
\[ \mu. \]This shrinking variability is one mathematical reason larger samples tend to produce more stable averages.
The Law of Large Numbers
The Law of Large Numbers states, in broad form, that repeated averages tend toward the expected value when observations satisfy appropriate conditions.
Symbolically,
\[ \boxed{ \overline{X}_n\to\mu } \]as
\[ n\to\infty. \]What the Law of Large Numbers Means
Suppose a random experiment has expected value
\[ \mu. \]If the experiment is repeated many times and the outcomes are averaged, the average tends to get closer to
\[ \mu. \]The theorem describes stabilization of averages, not stabilization of individual outcomes.
Example: Fair Die Averages
A fair six-sided die has expected value
\[ \mu=3.5. \]A single roll must be one of
\[ 1,2,3,4,5,6. \]But the average of many independent rolls tends to move toward
\[ \boxed{3.5}. \]Long-Run Frequency as a Special Case
Let
\[ I_i= \begin{cases} 1, & \text{success on trial }i,\\ 0, & \text{failure on trial }i. \end{cases} \]If the success probability is \(p\), then
\[ \mathbb{E}[I_i]=p. \]The sample average
\[ \frac{I_1+\cdots+I_n}{n} \]is the observed proportion of successes.
By the Law of Large Numbers, that proportion tends toward
\[ \boxed{p}. \]Weak Law of Large Numbers
One standard form is the Weak Law of Large Numbers.
Under suitable assumptions, for every
\[ \varepsilon>0, \]we have
\[ \boxed{ P\left( |\overline{X}_n-\mu|>\varepsilon \right) \to0 }. \]The probability that the sample mean differs from \(\mu\) by more than a fixed amount tends to zero.
Strong Law of Large Numbers
A stronger form states that, under appropriate assumptions,
\[ \boxed{ \overline{X}_n\to\mu \quad\text{with probability }1 }. \]The strong and weak laws formalize related notions of long-run convergence.
At an introductory level, the essential message is that repeated averages stabilize near the expected value.
What the Law of Large Numbers Does Not Say
The theorem does not say that:
- individual observations become less random;
- short runs must resemble long-run probabilities;
- deviations must alternate in order to “balance out”;
- the average must move monotonically toward the mean;
- past outcomes force future outcomes to compensate.
The Gambler's Fallacy
Suppose a fair coin produces several heads in a row.
The Law of Large Numbers does not imply that tails has become more likely on the next toss.
If tosses are independent,
\[ P(\text{tails on next toss})=\frac12 \]regardless of the earlier sequence.
The Central Limit Theorem
The Central Limit Theorem, or CLT, describes the distributional shape of sums and averages.
Suppose \(X_1,\ldots,X_n\) are independent and identically distributed with finite mean
\[ \mu \]and finite positive variance
\[ \sigma^2. \]Then the standardized sample mean
\[ \boxed{ Z_n = \frac{ \overline{X}-\mu }{ \sigma/\sqrt{n} } } \]approaches the standard normal distribution as \(n\) becomes large.
Equivalent CLT Form for Sums
Let
\[ S_n=X_1+\cdots+X_n. \]Then
\[ \mathbb{E}[S_n]=n\mu \]and
\[ \operatorname{Var}(S_n)=n\sigma^2. \]The standardized sum is
\[ \boxed{ \frac{S_n-n\mu}{\sigma\sqrt{n}} }. \]This also approaches
\[ N(0,1). \]Approximate Distribution of the Sample Mean
For sufficiently large \(n\),
\[ \boxed{ \overline{X} \approx N\left( \mu, \frac{\sigma^2}{n} \right) }. \]Equivalently, the sample mean has approximate standard deviation
\[ \frac{\sigma}{\sqrt{n}}. \]Approximate Distribution of the Sum
For sufficiently large \(n\),
\[ \boxed{ S_n \approx N(n\mu,n\sigma^2) }. \]The corresponding standard deviation is
\[ \boxed{ \sigma_{S_n}=\sigma\sqrt{n} }. \]Why the Central Limit Theorem Is Remarkable
The original distribution of each \(X_i\) does not need to be normal.
Under standard CLT conditions, the distribution of the standardized sum or sample mean becomes approximately normal as the sample size grows.
This explains why normal distributions appear throughout statistical practice.
LLN Versus CLT
| Law of Large Numbers | Central Limit Theorem |
|---|---|
| Explains where the sample mean goes. | Explains the shape of its distribution after standardization. |
| \(\overline{X}\to\mu\) | \((\overline{X}-\mu)/(\sigma/\sqrt n)\to N(0,1)\) |
| Focuses on convergence to the expected value. | Focuses on approximate normality. |
| Explains stabilization. | Explains sampling-distribution shape. |
The Two Results Work Together
The Law of Large Numbers tells us that large-sample averages concentrate near
\[ \mu. \]The Central Limit Theorem tells us approximately how those averages are distributed around
\[ \mu. \]Together they provide both a center and a probability model for random averages.
Example: Sample Mean from a Population
Suppose a population has
\[ \mu=50 \]and
\[ \sigma=12. \]Independent samples of size
\[ n=36 \]are taken.
The sample mean has mean
\[ \boxed{50} \]and standard error
\[ \frac{12}{\sqrt{36}} = \boxed{2}. \]Approximate Probability for a Sample Mean
Using the preceding example, find approximately
\[ P(\overline{X}>54). \]Standardize:
\[ z = \frac{54-50}{2} = 2. \]Therefore,
\[ P(\overline{X}>54) \approx P(Z>2). \]From the standard normal distribution,
\[ \boxed{ P(\overline{X}>54)\approx0.0228 }. \]When the Original Population Is Normal
If the original observations are normally distributed,
\[ X_i\sim N(\mu,\sigma^2), \]then the sample mean is exactly normal for every positive integer \(n\):
\[ \boxed{ \overline{X} \sim N\left( \mu,\frac{\sigma^2}{n} \right) }. \]In this special case, the CLT approximation is not needed.
How Large Must \(n\) Be?
There is no universal sample size that guarantees a good CLT approximation for every population.
The required \(n\) depends on the shape of the original distribution.
- Nearly symmetric distributions may require only modest \(n\).
- Strongly skewed distributions may require larger \(n\).
- Heavy-tailed distributions may converge more slowly.
The Common \(n\ge30\) Rule
The guideline
\[ n\ge30 \]is often taught as a rough introductory rule.
It is not a theorem and should not be treated as universally sufficient. Distribution shape and tail behavior matter.
Central Limit Theorem for Bernoulli Variables
Let
\[ X_i= \begin{cases} 1, & \text{success},\\ 0, & \text{failure}. \end{cases} \]with success probability \(p\).
Then
\[ \mu=p \]and
\[ \sigma^2=p(1-p). \]Sample Proportion
If
\[ X=X_1+\cdots+X_n \]is the number of successes, then the sample proportion is
\[ \boxed{ \widehat{p}=\frac{X}{n} }. \]Since \(\widehat p\) is the average of Bernoulli indicators,
\[ \mathbb{E}[\widehat p]=p \]and
\[ \operatorname{Var}(\widehat p) = \frac{p(1-p)}{n}. \]Approximate Normality of a Sample Proportion
Under suitable conditions,
\[ \boxed{ \widehat p \approx N\left( p, \frac{p(1-p)}{n} \right) }. \]Its standard error is
\[ \boxed{ \sqrt{\frac{p(1-p)}{n}} }. \]Normal Approximation to the Binomial
If
\[ X\sim\operatorname{Binomial}(n,p), \]then for suitable \(n\) and \(p\),
\[ \boxed{ X \approx N\left( np, np(1-p) \right) }. \]This follows from the CLT because a binomial random variable is a sum of Bernoulli variables.
Conditions for a Useful Binomial Normal Approximation
A common practical guideline is to require both
\[ np \]and
\[ n(1-p) \]to be reasonably large.
This helps avoid extremely skewed binomial distributions.
Continuity Correction
The binomial distribution is discrete, while the normal distribution is continuous.
A continuity correction adjusts integer boundaries by \(0.5\) when using a normal approximation.
For example,
\[ P(X\le k) \]is approximated by
\[ \boxed{ P(Y\le k+0.5) } \]for the corresponding normal variable \(Y\).
Example: Continuity Correction
Suppose a binomial random variable is approximated by a normal random variable \(Y\).
Then
\[ P(X=10) \]is approximated by the continuous interval
\[ \boxed{ P(9.5CLT for Sums of Non-Normal Variables
Suppose each \(X_i\) is highly non-normal but has finite mean and variance.
The sum
\[ S_n=X_1+\cdots+X_n \]can still become approximately normal after standardization for large enough \(n\).
The CLT does not require each individual observation to look normal.
The Distribution of Individual Observations Does Not Change
The CLT concerns the distribution of
\[ \overline X \]or
\[ S_n. \]It does not say that the original random variable \(X\) itself becomes normally distributed as more observations are collected.
Standard Error Versus Standard Deviation
The population standard deviation
\[ \sigma \]describes the spread of individual observations.
The standard error
\[ \boxed{ \frac{\sigma}{\sqrt n} } \]describes the spread of the sample mean across repeated samples.
Example: Individual Variation Versus Mean Variation
Suppose individual measurements have
\[ \sigma=20. \]For samples of size
\[ n=100, \]the standard error is
\[ \frac{20}{\sqrt{100}} = \boxed{2}. \]Individual measurements remain highly variable, while sample means vary much less.
Increasing Sample Size by a Factor of Four
Since standard error is
\[ \frac{\sigma}{\sqrt n}, \]multiplying the sample size by \(4\) divides the standard error by
\[ \sqrt4=2. \]Thus four times as many observations halves the standard error.
Increasing Sample Size by a Factor of Nine
Multiplying \(n\) by \(9\) divides standard error by
\[ \sqrt9=3. \]This square-root relationship is important when evaluating the benefit of larger samples.
Sampling Distributions
A sampling distribution is the probability distribution of a statistic across repeated samples.
The CLT primarily concerns the sampling distribution of sums and sample means.
The random variable
\[ \overline X \]has its own mean, variance, and approximate shape.
Why Sampling Distributions Matter
Statistical inference depends on understanding how a statistic would vary across repeated random samples.
The CLT makes this variation mathematically manageable by providing an approximate normal model in many settings.
Common Error: Confusing the LLN with the CLT
The Law of Large Numbers says that averages approach the expected value.
The Central Limit Theorem says that properly standardized averages or sums approach a normal distribution.
They answer different questions.
Common Error: Saying the Data Become Normal
The CLT does not say the original data become normally distributed.
It describes the distribution of sums or sample means across repeated samples.
Common Error: Forgetting the Square Root of \(n\)
The standard error of the sample mean is
\[ \boxed{ \frac{\sigma}{\sqrt n} }, \]not
\[ \frac{\sigma}{n}. \]Common Error: Using \(\sigma/\sqrt n\) for the Sum
For the sum
\[ S_n=X_1+\cdots+X_n, \]the standard deviation is
\[ \boxed{ \sigma\sqrt n }. \]The quantity \(\sigma/\sqrt n\) belongs to the sample mean.
Common Error: Assuming Independence Without Checking
Standard textbook versions of the CLT often assume independent observations.
Strong dependence can change the behavior of sums and averages and may invalidate the simple formulas used here.
Common Error: Treating \(n\ge30\) as a Universal Theorem
The number \(30\) is only a rough introductory guideline.
Highly skewed or heavy-tailed populations may require much larger samples for a good normal approximation.
Common Error: Applying the CLT When Variance Is Not Finite
The classical finite-variance CLT requires
\[ \sigma^2<\infty. \]Extremely heavy-tailed distributions can violate this condition and may exhibit different limiting behavior.
Common Error: Believing Large Samples Remove Bias
A large sample can reduce random sampling variability, but it does not automatically correct a systematically biased data-collection process.
Probability theory cannot rescue a sample that does not represent the target population appropriately.
A Reliable Procedure for Sample-Mean Problems
- Identify the population mean \(\mu\).
- Identify the population standard deviation \(\sigma\).
- Identify the sample size \(n\).
- Compute \(\mathbb{E}[\overline X]=\mu\).
- Compute the standard error \(\sigma/\sqrt n\).
- Determine whether exact normality or a CLT approximation is justified.
- Standardize the desired sample-mean boundary.
- Use the standard normal distribution to find the probability.
A Reliable Procedure for Sum Problems
- Identify \(\mu\), \(\sigma\), and \(n\).
- Compute the sum mean \(n\mu\).
- Compute the sum standard deviation \(\sigma\sqrt n\).
- Check whether a normal model or CLT approximation is appropriate.
- Standardize using \((S_n-n\mu)/(\sigma\sqrt n)\).
- Use the standard normal distribution for the desired probability.
Practice 1: Mean of the Sample Mean
Suppose a population has
\[ \mu=80 \]and
\[ \sigma=15. \]Samples of size \(25\) are taken. Find
\[ \mathbb{E}[\overline X]. \]Practice 2: Standard Error
Using Practice 1, find the standard error of the sample mean.
Practice 3: Sample-Mean z-Score
Using Practice 1, find the z-score corresponding to
\[ \overline X=86. \]Practice 4: Law of Large Numbers
A fair coin is tossed independently many times.
What value should the observed proportion of heads tend toward according to the Law of Large Numbers?
Practice 5: Sum Mean and Variance
Independent random variables \(X_1,\ldots,X_{16}\) each have
\[ \mathbb{E}[X_i]=5 \]and
\[ \operatorname{Var}(X_i)=9. \]Let
\[ S=X_1+\cdots+X_{16}. \]Find \(\mathbb{E}[S]\) and \(\operatorname{Var}(S)\).
Practice 6: Sum Standard Deviation
Using Practice 5, find the standard deviation of \(S\).
Practice 7: Sample Proportion Mean
Independent Bernoulli trials have success probability
\[ p=0.4. \]Let \(\widehat p\) be the sample proportion of successes. Find
\[ \mathbb{E}[\widehat p]. \]Practice 8: Sample Proportion Standard Error
Using
\[ p=0.4 \]and
\[ n=100, \]find the standard error of \(\widehat p\).
Practice 9: Distinguish LLN and CLT
Identify which result best matches each statement:
- The sample mean becomes increasingly concentrated near \(\mu\).
- The standardized sample mean becomes approximately normal.
Practice 10: Identify the Error
A student says:
“If I collect enough observations from a skewed population, the individual observations themselves will become normally distributed.”
Explain the error.
Practice Answers
Practice 1
\[ \boxed{ \mathbb{E}[\overline X]=80 }. \]Practice 2
\[ \sigma_{\overline X} = \frac{15}{\sqrt{25}} = \frac{15}{5} = \boxed{3}. \]Practice 3
\[ z = \frac{86-80}{3} = \boxed{2}. \]Practice 4
The success probability for heads is
\[ p=\frac12. \]Therefore the observed proportion of heads tends toward
\[ \boxed{0.5}. \]Practice 5
\[ \mathbb{E}[S] = 16(5) = \boxed{80}. \]Independence allows the variances to add:
\[ \operatorname{Var}(S) = 16(9) = \boxed{144}. \]Practice 6
\[ \sigma_S = \sqrt{144} = \boxed{12}. \]Practice 7
\[ \boxed{ \mathbb{E}[\widehat p]=0.4 }. \]Practice 8
\[ \sigma_{\widehat p} = \sqrt{ \frac{0.4(0.6)}{100} }. \] \[ = \sqrt{0.0024} \approx \boxed{0.0490}. \]Practice 9
- Law of Large Numbers.
- Central Limit Theorem.
Practice 10
The CLT concerns the distribution of sums or sample means across repeated samples. It does not change the distribution of the individual observations. A skewed population remains skewed even when many observations are collected.
What to Remember
The sample mean is
\[ \boxed{ \overline X = \frac1n\sum_{i=1}^{n}X_i }. \]For independent identically distributed observations with mean \(\mu\) and variance \(\sigma^2\),
\[ \boxed{ \mathbb{E}[\overline X]=\mu } \]and
\[ \boxed{ \operatorname{Var}(\overline X)=\frac{\sigma^2}{n} }. \]The Law of Large Numbers explains
\[ \boxed{ \overline X_n\to\mu }. \]The Central Limit Theorem explains the approximate normality of
\[ \boxed{ \frac{\overline X-\mu}{\sigma/\sqrt n} }. \]For sufficiently large \(n\),
\[ \boxed{ \overline X \approx N\left( \mu,\frac{\sigma^2}{n} \right) }. \]The central distinction is: the Law of Large Numbers explains why averages stabilize near the expected value, while the Central Limit Theorem explains why the distribution of those averages often becomes approximately normal.
Where We Go Next
The Law of Large Numbers and Central Limit Theorem complete this sequence on random variables and probability distributions. They connect individual probability models to the behavior of large collections of observations and provide the foundation for later work in estimation, confidence intervals, hypothesis testing, and statistical inference.
With these ideas in place, we can move from describing probability distributions to using sample data to reason about larger populations and uncertain systems.