Point estimates, bias, variance, mean squared error, and the construction and interpretation of confidence intervals for means, proportions, differences, and variances
Every statistical investigation begins with a question about the world that cannot be answered by examining every individual in a population. A physician studying blood pressure in adults cannot measure every adult. An engineer testing the lifetime of a manufactured component cannot test every component to failure. A public health researcher estimating vaccine effectiveness cannot enroll an entire country. In each case, the investigator collects a sample, analyzes it carefully, and uses the results to draw conclusions about the larger population. This process is estimation.
Estimation is not guessing. It is disciplined uncertainty: the principled use of probability theory, sampling distributions, and careful assumptions to say not only what we think a parameter is, but how confident we are in that judgment and under what conditions that confidence is warranted. Every concept from Chapters 1 through 7 flows into estimation — probability, random variables, distributions, joint distributions, transformations, limit theorems, and sampling distributions now converge into their main applied purpose.
This chapter introduces two forms of estimation. A point estimate is a single number representing our best guess at a population parameter. A confidence interval surrounds that guess with a carefully calculated range reflecting how much uncertainty the sampling process introduces. Both forms require understanding what estimators are, how to judge their quality, and how to interpret results correctly — including the most commonly misunderstood idea in statistics: what a confidence interval actually means.
A polling organization surveys 1,000 registered voters and finds that 480 plan to vote for a particular candidate. They report "48% support — margin of error ±3%." Where does that margin come from? Why 3% and not 1% or 10%? What exactly does "margin of error" mean? What would it mean to say we are "95% confident"? By the end of this chapter, you will be able to derive every number in that report from first principles — and explain, precisely, why the phrase "there is a 95% probability the true support is between 45% and 51%" is incorrect.
Science, engineering, medicine, economics, and public policy all depend on knowledge of populations: the full set of individuals, items, or outcomes under study. But populations are almost never fully observable. The gap between what we can observe and what we need to know is where statistics lives. Estimation bridges that gap.
Estimation is the process of using information from a sample to make inferences about numerical characteristics of a population. Estimation is not about removing uncertainty — uncertainty cannot be removed. It is about quantifying uncertainty honestly, working within it rigorously, and communicating conclusions that are both useful and defensible.
| Mode | What It Produces | Example |
|---|---|---|
| Point estimation | A single number — the best guess at the parameter | "48% of voters support Candidate A" |
| Interval estimation | A range of plausible values with a stated confidence level | "48% ± 3%, with 95% confidence" |
A point estimate without an interval gives the impression of false precision. An interval without a point estimate gives less information than is available. Good statistical practice reports both. Estimation rests on probability theory: because sample statistics are random variables that vary from sample to sample according to known distributions, we can use probability to describe how far a statistic is likely to be from the parameter it estimates.
Why is a population parameter considered a fixed quantity while a sample statistic is considered a random variable?
A nutritionist reports that the average caloric intake in a sample of 200 adults was 2,150 calories per day. Identify the population, the parameter, the sample, and the statistic in this scenario.
A parameter is a numerical characteristic of a population — fixed but unknown. A statistic is any function of observed sample data. When a statistic is used to estimate a parameter, it is called an estimator. When specific data are collected and the rule is applied, the resulting number is called an estimate.
This distinction matters deeply. An estimator is a rule — a function, a random variable with its own distribution. An estimate is the specific number produced by applying that rule to our data. The quality of an estimator is judged not by any single estimate but by the behavior of the estimator over many hypothetical repetitions of the sampling process.
| Parameter | Symbol | Estimator | Formula |
|---|---|---|---|
| Population mean | μ | Sample mean | X̄ = (1/n) Σ Xᵢ |
| Population proportion | p | Sample proportion | p̂ = X/n |
| Population variance | σ² | Sample variance | S² = (1/(n−1)) Σ(Xᵢ − X̄)² |
| Difference of means | μ₁ − μ₂ | Difference of sample means | X̄₁ − X̄₂ |
The denominator n − 1 rather than n is not arbitrary. It corrects for a systematic underestimation that arises when the sample mean x̄ is used in place of the unknown μ. This correction — Bessel's correction — is developed fully in Section 8.4.
For each parameter, state an appropriate estimator: (a) the population mean μ of diastolic blood pressure, (b) the population proportion p of defective items in a shipment, (c) the population variance σ² of exam scores.
A point estimate is the numerical value an estimator produces when applied to a specific sample. It is our single best guess at the population parameter. Point estimates are easy to communicate and easy to compute — their limitation is that they convey no information about uncertainty. Every confidence interval is built around a point estimate; we cannot quantify uncertainty about a quantity without first having an estimate of it.
Point estimation often follows the plug-in principle: replace the population quantity with its sample analog. The population mean μ is estimated by x̄. The population variance σ² is estimated by s². The population proportion p is estimated by p̂. The population median is estimated by the sample median. This principle is intuitive and often produces good estimators.
x̄ = 2,130 / 30 = 71.0 beats per minutes² = (Σxᵢ² − n·x̄²) / (n−1)
= (152,910 − 30·(71.0)²) / 29
= (152,910 − 151,230) / 29
= 1,680 / 29 ≈ 57.93s = √57.93 ≈ 7.61 beats per minutep̂ = 184 / 400 = 0.46An estimator T for parameter θ is unbiased if E[T] = θ. If E[T] ≠ θ, the estimator is biased. The bias of T is:
Bias(T) = E[T] − θ
Positive bias: T tends to overestimate θ on average. Negative bias: T tends to underestimate. Bias is a property of the estimator — the rule — not of any single estimate.
Let X₁, X₂, …, Xₙ be a random sample from a population with mean μ. Then:
E[X̄] = E[(1/n)ΣXᵢ] = (1/n)ΣE[Xᵢ] = (1/n)·nμ = μ
The second equality uses linearity of expectation. The third uses the fact that each Xᵢ has expected value μ, since all observations are drawn from the same population. This result holds for any population distribution — the sample mean is always unbiased for the population mean, as long as the sample is random.
The naive estimator S̃² = (1/n)Σ(Xᵢ − X̄)² satisfies E[S̃²] = ((n−1)/n)σ², which underestimates σ² systematically. The bias is −σ²/n.
Defining S² = (1/(n−1))Σ(Xᵢ − X̄)² instead gives E[S²] = (n/(n−1))·((n−1)/n)σ² = σ². This is Bessel's correction.
Intuition. When we compute S², we use x̄ instead of the unknown μ. Since x̄ is computed from the same data, it is automatically closer to each xᵢ than μ typically would be — the deviations xᵢ − x̄ are systematically smaller than xᵢ − μ. Dividing by n − 1 compensates for this shrinkage.
Even though S² is unbiased for σ², the sample standard deviation S = √S² is not an unbiased estimator for σ. By Jensen's inequality, E[√S²] ≤ √E[S²] = σ for a concave function. In practice, the bias in S is small for large n, and S is still used as the standard point estimator for σ.
Show that the estimator T = X₁ (just the first observation) is an unbiased estimator of μ. Despite being unbiased, why is T a poor estimator compared to X̄?
Two unbiased estimators can both center on the true parameter θ and yet behave very differently. One might fluctuate wildly from sample to sample; another might stay relatively close to θ in almost every sample. The variance of an estimator measures this fluctuation. Lower variance means the estimator is more reliable: in most samples, it will be close to the truth.
For a random sample of size n from a population with mean μ and variance σ²:
Var(X̄) = σ²/n
SE(X̄) = σ/√n
This formula encodes two fundamental truths: (1) Larger samples give more reliable estimates — as n grows, SE shrinks. (2) More variable populations require larger samples to achieve the same precision.
Var(X̄) = Var((1/n)ΣXᵢ) = (1/n²)ΣVar(Xᵢ) [independence: covariance terms vanish] = (1/n²)·nσ² = σ²/n
For a random sample of n Bernoulli observations with success probability p:
Var(p̂) = p(1−p)/n
SE(p̂) = √(p(1−p)/n)
Since p is unknown, in practice substitute p̂: SE_hat(p̂) = √(p̂(1−p̂)/n)
SE(X̄) = σ/√n = 16/√64 = 16/8 = 2When comparing two unbiased estimators T₁ and T₂ for the same parameter, the one with smaller variance is called more efficient. The ratio Var(T₁)/Var(T₂) is the relative efficiency. For example, the sample mean is more efficient than the sample median when sampling from a normal population — the median needs roughly 57% more observations to achieve the same precision.
A population has σ = 24. (a) What is SE(X̄) for n = 36? (b) What sample size is needed to achieve SE = 2? (c) By what factor does the sample size need to increase relative to n = 36?
The mean squared error of an estimator T for parameter θ combines bias and variance:
MSE(T) = E[(T − θ)²]
A fundamental identity decomposes MSE into two components:
MSE(T) = Var(T) + [Bias(T)]²
If T is unbiased, MSE(T) = Var(T). If T is biased, a small amount of bias can sometimes dramatically reduce variance, giving a lower MSE overall — the bias-variance tradeoff.
Let b = E[T] − θ (the bias). Write T − θ = (T − E[T]) + (E[T] − θ) = (T − E[T]) + b. Then:
E[(T−θ)²] = E[(T−E[T])²] + 2b·E[T−E[T]] + b² = Var(T) + 0 + b²
since E[T − E[T]] = 0 by definition.
MSE(A) = 12 + 0² = 12MSE(B) = 3 + 2² = 3 + 4 = 7Regularization in machine learning and ridge regression in statistics intentionally introduce a small amount of bias to substantially reduce variance, achieving a lower MSE overall. This is not hypothetical — it is the basis of some of the most widely used modeling techniques. Chapter 12 will revisit this idea in the context of regression.
An estimator has Var(T) = 5 and Bias(T) = −1. A competing estimator has Var(T′) = 9 and Bias(T′) = 0. Compare their MSEs. Which would you choose in a context where bias is penalized heavily?
A point estimate communicates our best guess but nothing about uncertainty. A confidence interval answers the uncertainty question by constructing a range of values that is designed to capture the true parameter with a specified probability under repeated sampling. Before stating the definition, a critical conceptual issue must be addressed — one that trips up students and professionals alike.
The true parameter θ is fixed. It does not vary. What is random is the interval itself — each time we draw a new sample, we compute a new confidence interval. A 95% confidence interval means: the procedure that produces this interval, when applied repeatedly to random samples from the same population, will yield intervals that capture the true parameter 95% of the time in the long run. This is a statement about the procedure, not about any specific computed interval.
Let X₁, …, Xₙ be a random sample and θ an unknown parameter. Two statistics L = L(X₁, …, Xₙ) and U = U(X₁, …, Xₙ) form a 100(1−α)% confidence interval for θ if:
P(L ≤ θ ≤ U) = 1 − α
The value 1 − α is the confidence level. Common choices: 90% (α = 0.10), 95% (α = 0.05), 99% (α = 0.01). Intervals are constructed from pivotal quantities whose distributions do not depend on unknown parameters.
| Pivotal Quantity | Distribution | Used When |
|---|---|---|
| Z = (X̄ − μ)/(σ/√n) | N(0,1) | σ known, or large n |
| T = (X̄ − μ)/(S/√n) | t with n−1 df | σ unknown, population approximately normal |
| (p̂ − p)/√(p(1−p)/n) | N(0,1) approximately | Large samples, proportions |
| (n−1)S²/σ² | χ²(n−1) | Variance, normal population |
Higher confidence levels produce wider intervals. To be 99% confident, we must accept a wider range of plausible values. There is a fundamental tradeoff: width measures precision (narrower is more informative), while confidence measures reliability (higher means the procedure rarely fails). We cannot simultaneously maximize both. This tradeoff is an expression of unavoidable uncertainty in statistical inference.
The margin of error (ME) of a confidence interval is half the width of the interval:
ME = (U − L) / 2
Equivalently, the confidence interval is written as: Estimate ± ME
For a Z-interval for a mean: ME = z_{α/2} · σ/√n
Three factors control ME: (1) the critical value z_{α/2} (confidence level); (2) the population standard deviation σ; (3) the sample size n.
| Confidence Level | α | α/2 | zα/2 |
|---|---|---|---|
| 90% | 0.10 | 0.05 | 1.645 |
| 95% | 0.05 | 0.025 | 1.960 |
| 99% | 0.01 | 0.005 | 2.576 |
To achieve a desired margin of error ME₀ at confidence level 1 − α, solve ME₀ = z_{α/2} · σ/√n for n:
n = (z_{α/2} · σ / ME₀)²
= (1.960 × 15 / 3)²
= (29.4 / 3)²
= (9.8)²
= 96.04n = (2.576 × 15 / 3)²
= (12.88)²
= 165.9 → n = 166A polling firm wants to estimate a population proportion with ME ≤ 0.02 at 95% confidence. Using the conservative value p̂ = 0.5, what sample size is required? (Hint: for a proportion, ME = z_{α/2}·√(p(1−p)/n). Use p(1−p) ≤ 1/4.)
When σ is known, the pivotal quantity Z = (X̄ − μ)/(σ/√n) ~ N(0,1). We want P(L ≤ μ ≤ U) = 1 − α. Starting from:
P(−z_{α/2} ≤ (X̄ − μ)/(σ/√n) ≤ z_{α/2}) = 1 − α
Rearranging to isolate μ yields the confidence interval.
z_{0.025} = 1.960
SE = σ/√n = 0.12/√36 = 0.12/6 = 0.020
ME = 1.960 × 0.020 = 0.039295% CI: 10.04 ± 0.039
= (10.001, 10.079)In practice, σ is almost never known. When σ must be estimated by s, the pivot changes to T = (X̄ − μ)/(S/√n) ~ t_{n−1}. The t-distribution has heavier tails than the normal, reflecting the additional uncertainty from estimating σ. As n → ∞, t_{n−1} → N(0,1).
The sample is a simple random sample. Observations are independent. Either the population is approximately normal, or n ≥ 30 (common guideline for CLT approximation). The t-interval is reasonably robust to mild non-normality for moderate and large samples.
df = n − 1 = 15
t_{15, 0.025} = 2.131
SE = s/√n = 3.1/√16 = 3.1/4 = 0.775
ME = 2.131 × 0.775 = 1.65295% CI: 22.4 ± 1.652
= (20.748, 22.052)
≈ (20.7, 22.1) gt_{15, 0.005} = 2.947
ME = 2.947 × 0.775 = 2.284
99% CI: 22.4 ± 2.284
= (20.116, 24.684) g| Situation | Use |
|---|---|
| σ known, normal population | Z-interval |
| σ unknown, normal population, any n | t-interval |
| σ unknown, non-normal population, large n (≥ 30) | t-interval (approximately valid by CLT) |
| σ unknown, non-normal population, small n | Special methods (beyond this chapter) |
A sample of n = 25 measurements has x̄ = 83.7 and s = 10.4. Assuming a normal population, construct a 90% CI for μ. Identify the point estimate, the critical value, the standard error, and the margin of error. Then interpret the interval in a complete sentence.
Many real-world quantities are proportions: the fraction of voters supporting a candidate, the proportion of patients responding to a treatment, the defect rate in manufacturing. We estimate the population proportion p using the sample proportion p̂ = X/n. By the Central Limit Theorem, for large samples the standardized p̂ is approximately standard normal.
The normal approximation is reasonable when: n·p̂ ≥ 10 and n·(1−p̂) ≥ 10. If these conditions fail, alternative methods — the Wilson interval or exact binomial intervals — are preferred. Statistical software computes these automatically.
p̂ = 320/500 = 0.640
n·p̂ = 320 ≥ 10 ✓
n·(1−p̂) = 180 ≥ 10 ✓SE = √(0.640 × 0.360 / 500)
= √(0.0004608)
≈ 0.0214795% CI: 0.640 ± 1.960 × 0.02147
= 0.640 ± 0.042
= (0.598, 0.682)p̂ = 4/80 = 0.05
n·p̂ = 4 ✗ (less than 10)When planning a study and p is unknown, use the conservative bound p(1 − p) ≤ 1/4 (achieved at p = 0.5):
n = (1.960)² / (4 × (0.03)²)
= 3.8416 / 0.0036
≈ 1,067.1 → n = 1,068A survey of 750 adults finds that 420 believe climate change is primarily caused by human activity. Check the large-sample conditions, then construct a 90% CI for p. Interpret the interval.
Many research questions compare two groups: does a new drug reduce blood pressure more than a placebo? Do two teaching methods produce different mean scores? These questions are answered by estimating the difference between two population parameters: μ₁ − μ₂ or p₁ − p₂.
| Case | Interval Formula | Degrees of Freedom |
|---|---|---|
| A: σ₁, σ₂ known | (x̄₁ − x̄₂) ± z_{α/2}√(σ₁²/n₁ + σ₂²/n₂) | — |
| B: Unequal variances (Welch) | (x̄₁ − x̄₂) ± t*√(s₁²/n₁ + s₂²/n₂) | Welch–Satterthwaite ν |
| C: Equal variances (pooled) | (x̄₁ − x̄₂) ± t_{n₁+n₂−2}·Sₚ√(1/n₁+1/n₂) | n₁ + n₂ − 2 |
When σ₁ = σ₂ = σ is assumed (equal but unknown), estimate σ² by the pooled variance:
Sₚ² = [(n₁−1)S₁² + (n₂−1)S₂²] / (n₁ + n₂ − 2)
x̄₁ − x̄₂ = 78.3 − 82.7 = −4.4SE = √(9.1²/40 + 10.6²/45)
= √(2.070 + 2.497)
= √4.567 ≈ 2.137ν ≈ (2.070+2.497)² / [(2.070)²/39 + (2.497)²/45]
≈ 20.86 / 0.2485 ≈ 83.9 → use ν = 83
t_{83, 0.025} ≈ 1.99095% CI: −4.4 ± 1.990 × 2.137
= −4.4 ± 4.25
= (−8.65, −0.15)SE = √(0.60×0.40/200 + 0.45×0.55/200)
= √(0.0012 + 0.0012375)
= √0.0024375 ≈ 0.0493795% CI: 0.15 ± 1.960 × 0.04937
= 0.15 ± 0.097
= (0.053, 0.247)When the two groups are not independent — for example, before-and-after measurements on the same individuals — define Dᵢ = X₁ᵢ − X₂ᵢ as the difference for subject i. Then construct a one-sample t-interval on the differences:
t_{11, 0.025} = 2.201
SE = 12.3/√12 = 12.3/3.464 ≈ 3.551
ME = 2.201 × 3.551 ≈ 7.815
95% CI: 18.4 ± 7.815
= (10.58, 26.22) mg/dLTwo independent groups are compared: n₁ = 30, x̄₁ = 47.2, s₁ = 6.8 and n₂ = 35, x̄₂ = 43.9, s₂ = 7.4. Construct a 90% CI for μ₁ − μ₂ using Welch's method. What does the interval tell you about the difference?
Estimating variability is as important as estimating center. A manufacturer needs to certify that a production process is consistent; a financial analyst needs to quantify volatility. Confidence intervals for the population variance σ² use the chi-square distribution.
When sampling from a normal population:
(n−1)S²/σ² ~ χ²(n−1)
The chi-square distribution is not symmetric — it is right-skewed and takes only nonnegative values. This means the confidence interval for σ² is not symmetric around S².
df = n − 1 = 19
χ²_{19, 0.025} = 32.852 (upper 2.5% critical value)
χ²_{19, 0.975} = 8.907 (lower 2.5% critical value)
(n−1)s² = 19 × 36 = 68495% CI for σ²:
(684/32.852 , 684/8.907)
= (20.82, 76.79)95% CI for σ:
(√20.82, √76.79)
= (4.56, 8.76)The interval (20.82, 76.79) is not symmetric around s² = 36. The center of the interval is not 36. This is a consequence of the right-skewed chi-square distribution and is expected. Never be tempted to center the interval at s² or treat it as a symmetric ± interval. The chi-square interval is also sensitive to the normality assumption; use it only when normality is reasonable.
A sample of n = 10 from a normal distribution has s = 5.2. Construct a 90% CI for σ². Use χ²_{9, 0.05} = 16.919 and χ²_{9, 0.95} = 3.325. Then interpret the result and comment on the asymmetry.
After computing a 95% confidence interval, students and professionals alike frequently say: "There is a 95% probability that the true parameter lies in this interval." This statement is wrong in the classical framework — and understanding why is one of the most important lessons in this entire course.
Once the data are collected and the interval is computed, the interval is fixed. The true parameter is fixed. Either it is in the interval or it is not — the probability is either 1 or 0; we just do not know which. The 95% refers to the long-run success rate of the procedure, not the probability for any specific interval.
| Correct Interpretation | Incorrect Interpretation |
|---|---|
| "We are 95% confident that μ lies in [L, U]." (Acceptable shorthand for the long-run procedure statement.) | "There is a 95% probability that μ is in [L, U]." (Treats a fixed parameter as if it were random.) |
| "The method that produced this interval captures the true parameter in 95% of applications." | "95% of all sample means lie in [L, U]." (Confuses a parameter interval with a data interval.) |
| "If we repeated this study many times, about 95% of the intervals would contain the true parameter." | "The true mean is most likely near the center of the interval." (No frequentist basis for this claim.) |
A confidence interval that excludes zero is "statistically significant" — zero is not a plausible value for the parameter. But statistical detectability is not the same as practical relevance. A 95% CI for a drug's effect on blood pressure of (0.1 mmHg, 0.8 mmHg) excludes zero but is clinically meaningless. Always interpret intervals in the context of what differences actually matter.
Once the data are collected, the interval is fixed. The parameter is fixed. The 95% refers to the long-run procedure, not the specific computed interval. Correct: "We are 95% confident in this interval" — understood as a statement about the procedure.
A confidence interval for a mean is about the parameter, not about individual observations. An interval for individual measurements is a prediction interval or tolerance interval — a different construction entirely.
A wider interval is more likely to contain the parameter, but also less informative. The interval (−∞, ∞) has 100% confidence but says nothing. Width and confidence must be balanced given the goals of the analysis.
Always identify whether σ is known or unknown. Use n − 1 degrees of freedom for one-sample t-intervals. Use z_{α/2} (two-tailed), not z_α (one-tailed), for a two-sided interval.
Before computing any interval, verify: (1) random sample, (2) independence, (3) distributional conditions. The stated confidence level is not guaranteed when conditions are violated.
Statistical significance (the interval excludes zero) is not the same as practical relevance. A detectable effect can be trivially small. Always interpret intervals in the context of what differences actually matter.
Always accompany point estimates with a confidence interval or at minimum a standard error. A point estimate alone implies false precision.
Higher confidence with the same sample size produces a wider interval. To maintain precision while increasing confidence, the sample size must increase. These are not independent choices.
Always round UP when computing required sample size. Rounding down gives a margin of error slightly larger than the desired ME₀ — the opposite of the intended guarantee.
Identify the error. What critical value should be used? Compute the correct interval.
Identify two distinct problems with this approach: one involving the conditions for the Wald interval and one involving the truncation.
Identify the error. What should n be?
Identify all errors in this calculation. Describe the correct approach and what distribution should be used.
Explain precisely why this statement is incorrect in the frequentist framework. Write a correct interpretation.
Chapter 8 established the machinery of estimation: how to use samples to produce informed, probabilistic claims about population parameters. Chapter 9 carries this work forward into hypothesis testing — a complementary form of inference that asks not "what is the parameter?" but "is the parameter consistent with a specific claim?" The connection between the two chapters is deep and precise. If a 95% CI for μ excludes the value μ₀, then a two-sided test of H₀: μ = μ₀ at significance level α = 0.05 will reject H₀ — and vice versa. This duality between confidence intervals and hypothesis tests is one of the most important structural properties of classical statistical inference. Students who understand the estimation framework — sampling distributions, standard errors, pivotal quantities, and the meaning of confidence — will find hypothesis testing to be a natural extension of the same tools, applied to a different question.