Likelihood functions, maximum likelihood estimation, Bayesian updating, conjugate priors, and model-based reasoning — the frameworks connecting data to parameters throughout modern statistics and data science
Every inference method rests on a question: given data, what can we learn about the unknown process that produced it? Chapters 8 and 9 developed two classical answers. Confidence intervals quantify uncertainty in estimation. Hypothesis tests evaluate evidence against a null claim. Both approaches use sample statistics whose sampling distributions are known — from the central limit theorem, from earlier distribution theory, from the careful analysis of Part II.
This chapter introduces a different level of the question. Rather than asking how a fixed procedure behaves across many samples, we ask: for this specific dataset, how does the evidence bear on different possible parameter values? That question leads to two powerful frameworks: maximum likelihood estimation and Bayesian inference. Both use the same mathematical object — the likelihood function — as their foundation, though they ask different questions of it.
By the end of this chapter, you should be able to construct likelihood functions for discrete and continuous models, find maximum likelihood estimates using calculus, understand priors and posteriors in the Bayesian framework, and connect these ideas to practice in statistics, machine learning, and data science.
A clinical trial tests a new drug. Of 20 patients enrolled, 14 respond positively. What is the most plausible value of the true response rate p? And if prior evidence from similar drugs suggests response rates cluster around 50%, how should that prior knowledge change your estimate? By the end of this chapter, you will have two rigorous answers — one from maximum likelihood, one from Bayesian inference — and the tools to choose between them in context.
When you work with a probability distribution, two different things are in play: the parameter that defines the distribution and the data that the distribution generates. In most of Chapters 3 through 9, parameters were fixed or assumed unknown, while data was the random quantity. Probability answers: if the parameter is θ, what is the probability of observing data x?
Likelihood reverses the question. Data has already been observed — it is fixed. The parameter is unknown. Likelihood asks: given that we observed x, how does the probability of x change as we vary θ?
Probability (before observing data): For fixed θ, the probability of observing x is P(X = x ; θ) in the discrete case, or the density f(x ; θ) in the continuous case.
Likelihood (after observing data x): The same mathematical expression — now evaluated as a function of θ with x held fixed:
L(θ) = P(X = x ; θ) [discrete] L(θ) = f(x ; θ) [continuous]
Likelihood is not a probability distribution over θ. The values L(θ) do not need to integrate or sum to 1 over the parameter space. Saying "the likelihood that θ = 0.5 is 0.3" is not a probability statement about θ — it is a comparative measure of how consistent θ = 0.5 is with the data.
Suppose X ~ Binomial(n = 10, p) for unknown p. We observe x = 7 successes.
| Question | Formula | What Varies | What Is Fixed |
|---|---|---|---|
| Probability — if p = 0.5, how likely is x = 7? | P(X = 7 ; p = 0.5) = C(10,7)(0.5)⁷(0.5)³ ≈ 0.117 | Data x | Parameter p |
| Likelihood — given x = 7, how does p affect the probability? | L(p) = C(10,7) p⁷(1−p)³ for p ∈ [0,1] | Parameter p | Data x = 7 |
L(0.3) = 120 · (0.3)⁷ · (0.7)³ ≈ 120 · 0.000219 · 0.343 ≈ 0.009
L(0.5) = 120 · (0.5)⁷ · (0.5)³ = 120 · (0.5)¹⁰ ≈ 0.117
L(0.7) = 120 · (0.7)⁷ · (0.3)³ ≈ 120 · 0.0824 · 0.027 ≈ 0.267
L(0.9) = 120 · (0.9)⁷ · (0.1)³ ≈ 120 · 0.478 · 0.001 ≈ 0.057
The distinction between probability and likelihood is one of the most important conceptual shifts in statistics. Probability flows from parameter to data — it predicts what we might see. Likelihood flows from data back to parameters — it evaluates what is consistent with what we saw. Both use the same formula but ask fundamentally different questions.
When a model has a single parameter θ, the likelihood L(θ) is a curve — a function from the parameter space into the non-negative real numbers. High values indicate parameter values that make the observed data relatively probable. Low values indicate parameter values under which the data would be rare.
One of the most useful properties of likelihood is that it supports direct comparison through the likelihood ratio:
L(θ₁) / L(θ₂)
If this ratio equals 8, then θ₁ makes the observed data 8 times more probable than θ₂ does. Likelihood ratios do not depend on the overall scale of the likelihood — only on relative values. This is a natural tool for comparing competing parameter values or hypotheses.
L(0.5) = 5(0.5)⁴(0.5) = 5/32 ≈ 0.156
L(0.8) = 5(0.8)⁴(0.2) = 5 · 0.4096 · 0.2 ≈ 0.410
L(0.8) / L(0.5) = 0.410 / 0.156 ≈ 2.63The likelihood function is the data's voice about the parameter. It does not tell you the parameter with certainty — it tells you, relatively speaking, which parameter values the data finds more or less plausible. The shape of the likelihood — sharp peak versus broad plateau — also reflects how much information the data contains about θ.
In most problems, we observe not one data point but a sample of n independent observations. Because the observations are independent, the probability of the entire dataset is the product of individual probabilities. Viewed as a function of θ, this product is the likelihood.
Let x₁, x₂, …, xₙ be observed values from independent random variables each with PMF P(Xᵢ = xᵢ ; θ). The likelihood is:
L(θ) = P(X₁ = x₁ ; θ) · P(X₂ = x₂ ; θ) · ⋯ · P(Xₙ = xₙ ; θ)
L(p) = p⁵(1−p)³L(0.5) = (0.5)⁵(0.5)³ = (0.5)⁸ ≈ 0.00391
L(0.625) ≈ (0.625)⁵(0.375)³ ≈ 0.00502
L(0.8) = (0.8)⁵(0.2)³ ≈ 0.00262
L(λ) = [e⁻λλ²/2!] · [e⁻λλ⁰/0!] · [e⁻λλ⁴/4!]
= e⁻³λ · λ⁶ / (2 · 1 · 24)
= e⁻³λ · λ⁶ / 48
For discrete exponential family distributions (Bernoulli, Binomial, Poisson, Geometric), the likelihood always depends on the data through a sufficient statistic — a summary that captures all the likelihood information. For Bernoulli models, it is the count of successes and failures. For Poisson, it is the total count. Identifying the sufficient statistic is the first step toward finding the MLE efficiently.
For continuous random variables, individual probabilities are zero — P(X = x) = 0 for any specific x. The likelihood uses the probability density function evaluated at the observed values. For independent observations, the likelihood is the product of densities:
L(λ) = ∏ᵢ λe^(−λxᵢ) = λⁿ · exp(−λ∑xᵢ) = λⁿe^(−λS)
where S = x₁ + x₂ + ⋯ + xₙ.
L(0.87) ≈ (0.87)⁴ e^(−0.87·4.6) ≈ 0.01048
L(0.5) = (0.5)⁴ e^(−0.5·4.6) ≈ 0.00627
L(1.5) = (1.5)⁴ e^(−1.5·4.6) ≈ 0.00510
L(μ) = (1/√(2πσ²))ⁿ · exp(−(1/2σ²)∑(xᵢ − μ)²)
When both μ and σ² are unknown in the normal model, the MLEs are μ̂ = x̄ and σ̂² = (1/n)∑(xᵢ − x̄)². Note the MLE divides by n, not n − 1. The MLE for σ² is biased downward — a point examined in Section 10.8. The standard unbiased estimator S² = (1/(n−1))∑(xᵢ − x̄)² corrects this.
The likelihood function is a product of probabilities or densities. For large samples this product becomes extremely small, causing numerical precision problems. More importantly, products are harder to differentiate than sums. The natural logarithm transforms products into sums and does not change where the maximum occurs — because the logarithm is strictly increasing. If L(θ) is maximized at θ*, then so is ln L(θ).
The log-likelihood function is:
ℓ(θ) = ln L(θ) = ∑ᵢ ln f(xᵢ ; θ)
We denote the log-likelihood by ℓ (lowercase ell). The maximum of L(θ) and ℓ(θ) occur at the same θ. In virtually all applied statistics, machine learning, and software, the log-likelihood is the quantity being maximized.
ℓ(p) = 5 ln p + 3 ln(1−p)dℓ/dp = 5/p − 3/(1−p)5(1−p) = 3p → 5 − 5p = 3p → p̂ = 5/8 = 0.625ℓ(λ) = S ln λ − nλ (plus constants)dℓ/dλ = S/λ − n = 0 → λ̂ = S/n = x̄ℓ(μ, σ²) = −(n/2)ln(2π) − (n/2)ln(σ²) − (1/2σ²)∑(xᵢ − μ)²
∂ℓ/∂μ = (1/σ²)∑(xᵢ − μ) = 0 → μ̂ = x̄∂ℓ/∂v = −n/(2v) + ∑(xᵢ−μ)²/(2v²) = 0 → σ̂² = (1/n)∑(xᵢ − x̄)²The maximum likelihood estimate (MLE) of parameter θ, given observed data, is the value θ̂ that maximizes the likelihood — equivalently, the log-likelihood:
For differentiable log-likelihoods, the MLE satisfies the score equation:
dℓ/dθ = 0
For multiple parameters, solve the system ∂ℓ/∂θⱼ = 0 simultaneously. Always verify the second derivative is negative (confirming a maximum, not a minimum or saddle point).
ℓ(p) = x ln p + (n−x) ln(1−p) + constantdℓ/dp = x/p − (n−x)/(1−p) = 0
→ x(1−p) = (n−x)p
→ x − xp = np − xp
→ x = np
→ p̂ = x/nℓ(λ) = n ln λ − λ∑xᵢdℓ/dλ = n/λ − ∑xᵢ = 0 → λ̂ = n/∑xᵢ = 1/x̄ℓ(λ) = −nλ + (∑xᵢ) ln λ − ∑ ln(xᵢ!)dℓ/dλ = −n + ∑xᵢ/λ = 0 → λ̂ = ∑xᵢ/n = x̄| Model | Unknown Parameter(s) | MLE | Sufficient Statistic |
|---|---|---|---|
| Bernoulli(p) | p | x̄ = S/n | S = ∑xᵢ |
| Binomial(n, p), n known | p | x/n | x (observed successes) |
| Poisson(λ) | λ | x̄ | ∑xᵢ |
| Geometric(p) | p | 1/x̄ | ∑xᵢ |
| Exponential(λ) | λ | 1/x̄ | ∑xᵢ |
| Normal(μ, σ²), both unknown | μ, σ² | x̄ and (1/n)∑(xᵢ−x̄)² | ∑xᵢ and ∑xᵢ² |
| Normal(μ, σ²), σ² known | μ | x̄ | ∑xᵢ |
| Uniform(0, θ) | θ | max(x₁,…,xₙ) | x₍ₙ₎ |
The Uniform(0, θ) case shows that the score equation approach can fail. The likelihood L(θ) = (1/θⁿ) · 1[θ ≥ max(xᵢ)] is decreasing in θ (wherever it is positive), so it is maximized by making θ as small as possible — subject to θ ≥ max(x₁, …, xₙ). Therefore θ̂ = max(x₁, …, xₙ). This estimate is biased downward: E[x₍ₙ₎] = n/(n+1) · θ < θ. An unbiased correction is (n+1)/n · x₍ₙ₎.
μ̂ = x̄ = (312 + 298 + 325 + 307 + 319)/5 = 1561/5 = 312.2 MPaσ̂² = 438.8/5 = 87.76 σ̂ ≈ 9.37 MPa
λ̂ = x̄ = (3+1+4+2+3+0+2+5+1+3)/10 = 24/10 = 2.4 birds/minuteMLE is not just a computational recipe — it produces estimators with strong theoretical properties. This section introduces the most important properties; full proofs are left to mathematical statistics courses.
| Property | Statement | Significance |
|---|---|---|
| Consistency | θ̂ converges in probability to the true θ* as n → ∞ | Larger samples produce more accurate estimates |
| Asymptotic normality | Standardized MLE is approximately N(0,1) for large n | Justifies large-sample confidence intervals and tests |
| Asymptotic efficiency | MLE achieves the Cramér–Rao lower bound asymptotically | Most precise consistent estimator in large samples |
| Invariance | If θ̂ is MLE of θ, then g(θ̂) is MLE of g(θ) | No need to re-derive likelihood for transformed parameters |
| Possible bias | MLE may be biased in finite samples | Unbiased correction may be needed (e.g., divide by n−1 for variance) |
For any unbiased estimator T of θ:
Var(T) ≥ 1/I(θ)
where I(θ) is the Fisher information:
I(θ) = E[(d/dθ ln f(X;θ))²] = −E[d²/dθ² ln f(X;θ)]
Fisher information measures how sensitive the log-likelihood is to the parameter. The MLE achieves this bound asymptotically — in large samples, it is the most precise consistent estimator.
If λ̂ = 1/x̄ is the MLE for the Exponential rate λ, then the MLE for the mean μ = 1/λ is μ̂ = x̄. The MLE for P(X > t) = e^(−λt) is e^(−λ̂t) = e^(−t/x̄). No re-derivation is needed — plug in the MLE wherever the parameter appears.
Bayesian inference takes a different view of uncertainty from frequentist inference. Rather than treating parameters as fixed unknown constants, the Bayesian framework expresses uncertainty about parameters directly as a probability distribution. Before seeing the data, the analyst specifies a prior distribution P(θ) encoding beliefs or prior knowledge. After observing data x, the prior is updated through Bayes' theorem to produce a posterior distribution P(θ | x).
In the inferential context, Bayes' theorem becomes:
P(θ | x) = [P(x | θ) · P(θ)] / P(x)
Or compactly: Posterior ∝ Likelihood × Prior
The symbol ∝ means "proportional to." The marginal likelihood P(x) is a normalizing constant that ensures the posterior integrates to 1. Since it does not depend on θ, the essential computation is the product of likelihood and prior.
| Term | Symbol | Meaning |
|---|---|---|
| Prior | P(θ) or f(θ) | Probability distribution over the parameter before seeing data |
| Likelihood | P(x | θ) = L(θ) | Probability of observed data given parameter θ |
| Marginal likelihood | P(x) | Normalizing constant — probability of data averaged over all θ |
| Posterior | P(θ | x) | Updated distribution over θ after observing data |
P(HH | Type A) = (0.5)² = 0.25
P(HH | Type B) = (0.8)² = 0.64
P(HH and Type A) = 0.60 × 0.25 = 0.150
P(HH and Type B) = 0.40 × 0.64 = 0.256
P(HH) = 0.150 + 0.256 = 0.406
P(Type A | HH) = 0.150/0.406 ≈ 0.370
P(Type B | HH) = 0.256/0.406 ≈ 0.630
A prior distribution f(θ) encodes uncertainty about the parameter θ before the data is observed. Priors must be valid probability distributions — they integrate (or sum) to 1 over the parameter space. The choice of prior is one of the most discussed topics in Bayesian statistics.
| Prior Type | Description | When to Use |
|---|---|---|
| Informative | Reflects genuine prior knowledge; concentrated near plausible values | When previous studies, expert knowledge, or historical data are available |
| Weakly informative | Encodes mild constraints (e.g., non-negativity, plausible range) without being restrictive | General default; guards against implausible extremes while letting data dominate |
| Non-informative (flat) | Uniform over parameter space; all values equal probability | When little prior knowledge exists; requires care — flatness depends on parameterization |
| Improper | Integrates to infinity; not a valid distribution but may yield valid posteriors | Advanced settings; used with caution |
A uniform prior on p ∈ [0, 1] looks assumption-free, but flatness depends on parameterization. A flat prior on p does not correspond to a flat prior on the log-odds logit(p) = log(p/(1−p)). The notion of "knowing nothing" is more subtle than it appears. Weakly informative priors that encode basic physical constraints are generally preferred to flat priors.
Prior: p ~ Beta(12, 12)
Given a prior f(θ) and likelihood L(θ) = f(x | θ), the posterior is proportional to their product. For conjugate priors (Section 10.12), the posterior falls in a recognizable family and the normalizing constant is automatic. From the posterior distribution, Bayesian inference extracts point estimates and intervals.
| Posterior Summary | Definition | Interpretation |
|---|---|---|
| Posterior mean | Expected value E[θ | x] under the posterior | Point estimate that minimizes expected squared error |
| MAP estimate | Mode of the posterior density | Most probable parameter value; equals MLE when prior is flat |
| Credible interval | [a, b] such that P(a ≤ θ ≤ b | x) ≥ 0.95 | Direct probability statement: 95% posterior probability that θ ∈ [a, b] |
f(p | x) ∝ p^(x+α−1)(1−p)^(n−x+β−1)
This is the kernel of Beta(α + x, β + n − x).
p | x ~ Beta(α + x, β + n − x)Posterior: Beta(2+14, 2+6) = Beta(16, 8)
Posterior mean = 16/(16+8) = 16/24 ≈ 0.667
Prior mean = 0.5 MLE = 14/20 = 0.70
A prior is conjugate to a likelihood if the posterior falls in the same distributional family as the prior. Conjugacy produces closed-form posteriors and simple update rules. It is not required for Bayesian inference — modern computational methods handle non-conjugate settings — but it provides clean intuition and efficient computation for many standard models.
Posterior: λ | x ~ Gamma(2 + 8, 1 + 5) = Gamma(10, 6)
Posterior mean = 10/6 ≈ 1.67 crashes/hour
Prior mean = 2.0
MLE = 8/5 = 1.6
1/τ₁² = 1/τ² + n/σ² = 1/25 + 9/64 = 0.04 + 0.1406 = 0.1806
→ τ₁² ≈ 5.54, τ₁ ≈ 2.35 bpm
μ₁ = τ₁² · (μ₀/τ² + nx̄/σ²)
= 5.54 · (72/25 + 9·68/64)
= 5.54 · (2.88 + 9.5625)
≈ 68.93 bpm
| Likelihood | Prior Family | Posterior Family | Update Rule |
|---|---|---|---|
| Binomial(n, p) | Beta(α, β) | Beta(α + x, β + n − x) | Add successes to α; failures to β |
| Poisson(λ) | Gamma(α, β) | Gamma(α + S, β + n) | Add total count to α; n to β |
| Normal(μ, σ²), σ² known | Normal(μ₀, τ²) | Normal(μ₁, τ₁²) | Precisions add; means precision-weighted |
| Exponential(λ) | Gamma(α, β) | Gamma(α + n, β + S) | Add n to α; sample sum to β |
| Geometric(p) | Beta(α, β) | Beta(α + n, β + S − n) | Add successes and failures |
The frameworks developed in this chapter are not confined to traditional statistics. They underlie the training of modern machine learning models, the comparison of competing models, and probabilistic forecasting across science, engineering, and industry.
| Application | Connection to Likelihood/Bayesian Methods |
|---|---|
| Linear regression | Least squares = MLE under Normal error assumption. Minimizing residual sum of squares is maximizing the Normal log-likelihood. |
| Logistic regression | Parameter estimation = maximizing Bernoulli log-likelihood. Cross-entropy loss = negative log-likelihood. |
| Neural network training | Training with cross-entropy loss = maximizing log-likelihood of a categorical distribution. Gradient descent on the log-likelihood is MLE in high dimensions. |
| Bayesian optimization | Uses Gaussian process prior over objective function; updates posterior as new evaluations arrive. Widely used for hyperparameter tuning. |
| Naïve Bayes classifiers | Direct application of Bayesian inference: prior probabilities over categories updated by word likelihoods to classify spam/not spam, sentiment, etc. |
| Model comparison | AIC = 2k − 2ℓ(θ̂); BIC = k ln n − 2ℓ(θ̂); Likelihood ratio test: Λ = −2 ln[L(θ̂₀)/L(θ̂₁)], which follows a chi-squared distribution under the null. |
Maximum likelihood estimation is the conceptual foundation of most machine learning training procedures. When a neural network minimizes cross-entropy loss, it is performing MLE on a categorical model. When a regression minimizes squared error, it is performing MLE on a Gaussian model. The mathematical framework of this chapter — likelihood as a function of parameters, differentiation, maximization — is the underlying principle behind a vast range of modern methods.
Correction: Likelihood is not a probability distribution over θ. L(θ) values do not sum or integrate to 1. Likelihood is used comparatively — L(θ₁)/L(θ₂) > 1 means θ₁ is more consistent with the data than θ₂. It is not a probability statement about θ.
Correction: The MLE is an estimate — the most probable parameter given observed data. For a different sample, the MLE would differ. The MLE is a random variable with its own sampling distribution. Always accompany it with a standard error or confidence interval.
Correction: MLE is consistent and asymptotically efficient, but not always unbiased in finite samples. The normal variance MLE σ̂² = (1/n)∑(xᵢ−x̄)² is biased downward; the unbiased estimator S² divides by n−1. The uniform MLE x₍ₙ₎ underestimates θ systematically.
Correction: A 95% confidence interval is a random procedure that captures the true θ in 95% of repeated samples. It says nothing about the probability that θ lies in the specific interval already computed. A 95% credible interval does carry that direct interpretation: P(θ ∈ [a, b] | x) = 0.95.
Correction: All statistical inference involves choices — model family, estimator, significance level. Bayesian inference makes the prior explicit and open to scrutiny. When prior information is genuine, incorporating it improves inference. Sensitivity analysis (varying the prior) checks how robust conclusions are.
Correction: Flatness depends on parameterization. A uniform prior on p ∈ [0,1] is not flat when reparameterized as logit(p). No prior is assumption-free. Weakly informative priors that encode basic physical constraints are generally preferred.
Correction: In multi-parameter problems, both partial derivatives must be set to zero and solved simultaneously. Optimizing over only one parameter while holding others fixed does not give the joint MLE — it gives a conditional optimum, which may be different.
Correction: The MAP estimate (posterior mode) and the posterior mean coincide only when the posterior is symmetric. For skewed posteriors (Beta, Gamma), they differ. The posterior mean minimizes expected squared error loss; the posterior mode is the most probable single value. Choose the appropriate summary for context.
Correction: Log-likelihoods are always ≤ 0 (since probabilities ≤ 1 and log(1) = 0). They cannot be interpreted as probabilities. Log-likelihoods are used for comparison and optimization — a less negative log-likelihood means a better fit. Use likelihood ratios or AIC/BIC for formal comparison.
Each statement below contains an error. Identify it and write a correct version.
Chapter 10 developed two foundational frameworks: maximum likelihood estimation and Bayesian reasoning. Both use the likelihood function to connect data to parameters. Both produce principled estimates with interpretable uncertainty.
Chapter 11 applies these frameworks — and the estimation theory built throughout Part III — to regression analysis. Regression models the relationship between two or more variables as a function with unknown parameters. The most important case is linear regression, where the relationship is modeled as a line and the parameters (intercept and slope) must be estimated from data.
The connection runs deep. As Section 10.4 showed, maximizing the Normal log-likelihood with unknown mean is equivalent to minimizing the sum of squared deviations from that mean. In linear regression, the unknown "mean" depends on a predictor variable x, and the parameters govern how the mean changes with x. Minimizing the sum of squared residuals — the least squares criterion — is therefore maximum likelihood estimation under the assumption that errors are normally distributed. Every idea from this chapter carries forward.