Maximum-Likelihood Methods

The Central Idea

Suppose observed data are represented by

and the statistical model depends on unknown parameters

The model assigns probabilities or probability densities to possible data:

Maximum likelihood chooses the parameter value that makes the observed data most probable or most plausible under the model:

Probability Model Versus Likelihood Function

The same expression

can be viewed in two different ways.

As a probability model, \(\boldsymbol{\theta}\) is fixed and the possible data \(\mathbf{y}\) vary.

As a likelihood function, the observed data are fixed and the parameter \(\boldsymbol{\theta}\) varies.

We therefore write

Likelihood Is Not a Probability Distribution Over Parameters

In maximum-likelihood estimation, the parameter is treated as an unknown fixed quantity.

Therefore,

measures relative support for different parameter values, but it is not itself required to integrate to \(1\) over parameter space.

This distinction separates likelihood-based inference from Bayesian posterior probability.

A Simple Discrete Example

Suppose a coin has unknown probability \(p\) of landing heads. We observe \(h\) heads and \(t\) tails.

Assuming independent tosses, the likelihood is proportional to

Maximum likelihood chooses the value of \(p\) that maximizes this expression.

Maximum-Likelihood Estimate for a Bernoulli Probability

Let

The log-likelihood is

Differentiating gives

Setting this equal to zero gives

Therefore,

Why We Use the Log-Likelihood

Likelihoods for independent observations are often products:

Products of many probabilities can become numerically very small.

Taking logarithms converts products into sums:

Because the logarithm is strictly increasing, maximizing \(L\) is equivalent to maximizing \(\ell\).

Negative Log-Likelihood

Numerical optimization software is often written to minimize rather than maximize.

Therefore, we commonly define

Maximum likelihood can then be written as

Independent Observations

If observations are conditionally independent given the parameters, then

The log-likelihood becomes

This additive structure is one reason likelihood methods scale naturally to large data sets.

Gaussian Measurement Error

Suppose a forward model predicts

and measurement errors are independent Gaussian variables:

The density of one observation is

Gaussian Log-Likelihood

For independent observations,

If \(\sigma^2\) is fixed, the first term is constant with respect to \(\boldsymbol{\theta}\).

Therefore, maximizing the Gaussian likelihood is equivalent to minimizing

Least Squares as Maximum Likelihood

This gives the important relationship:

Least squares is therefore not an isolated fitting rule. It is one member of a broader family of likelihood-based estimators.

Unequal Gaussian Variances

Suppose

Then the negative log-likelihood contains

Therefore, inverse-variance weighted least squares is the maximum-likelihood estimator under independent Gaussian errors with known unequal variances.

Correlated Gaussian Errors

Let the observation vector satisfy

The negative log-likelihood includes

This recovers generalized least squares.

Estimating the Noise Variance

The noise variance \(\sigma^2\) may itself be unknown.

In a Gaussian model with residuals

the likelihood depends on both

Maximum likelihood can estimate them jointly.

Gaussian MLE of Variance

If the mean structure is fitted by maximum likelihood, the Gaussian MLE of the variance has the form

This differs from the common unbiased residual-variance estimator that divides by \(n-p\).

Maximum-likelihood estimation and unbiased estimation are different goals.

Poisson Likelihood

Count data are often modeled using a Poisson distribution.

Suppose

where

Then

Poisson Log-Likelihood

Taking logarithms gives

The final term does not depend on the parameters and can be omitted when performing optimization.

Why Squared Error Can Be Poor for Counts

A Poisson variable has variance tied to its mean:

Therefore, the noise scale changes naturally with the expected count.

A Poisson likelihood can represent this structure more directly than an unweighted squared-error objective.

Binomial Likelihood

Suppose observation \(y_i\) counts successes among \(m_i\) trials:

The probability mass function is

If \(p_i\) depends on model parameters, maximum likelihood estimates those parameters from the observed success counts.

Binary Outcomes

For a single binary outcome

with success probability \(p_i\),

This is the Bernoulli likelihood.

Logistic regression is a major example of maximum-likelihood estimation for binary data.

Exponential Waiting-Time Likelihood

If event waiting times are modeled as exponential with rate \(\lambda\),

For independent observations \(t_1,\ldots,t_n\),

The maximum-likelihood estimate is

Choosing the Error Distribution

Maximum likelihood forces us to state an explicit model for observational variability.

Possible choices include:

• Gaussian errors for approximately symmetric additive noise;
• Poisson distributions for event counts;
• Binomial or Bernoulli distributions for success-failure outcomes;
• lognormal errors for positive multiplicative variability;
• exponential or related distributions for waiting times;
• heavy-tailed distributions when extreme deviations are more common.

Additive Versus Multiplicative Noise

Additive noise can be written

Multiplicative noise can be written

These two assumptions lead to different likelihoods and potentially different fitted parameters.

Lognormal Measurement Models

If positive observations have multiplicative variation, a lognormal model may be appropriate:

The likelihood then corresponds naturally to squared differences on the logarithmic scale:

Likelihood and the Forward Model

In inverse problems, a useful conceptual decomposition is

The forward model determines the expected or latent physical response.

The statistical observation model determines how measured data vary around that response.

Model Parameters and Noise Parameters

Parameters may enter different parts of the model.

For example,

The physical parameters control the forward model, while noise parameters control quantities such as variance, dispersion, or correlation.

Maximum likelihood can estimate both classes jointly.

Score Function

The gradient of the log-likelihood is called the score:

At an interior maximum under regular conditions,

This is the likelihood analogue of setting the gradient of a least-squares objective equal to zero.

Observed Curvature

The second derivative matrix

describes local curvature of the log-likelihood.

Near a well-defined maximum, the log-likelihood typically curves downward in identifiable parameter directions.

Observed Information

The negative Hessian of the log-likelihood is often called the observed information:

Large curvature means the likelihood changes rapidly as the parameter moves away from the optimum.

Small curvature indicates weak local information.

Fisher Information

The Fisher information matrix is an expected measure of likelihood curvature:

Under regular conditions, it can also be written as

Information and Parameter Precision

Greater Fisher information generally corresponds to greater potential precision in parameter estimation.

In regular large-sample settings,

Weak-information directions therefore produce large uncertainty.

Fisher Information and Identifiability

If the Fisher information matrix is singular, at least one parameter direction is not locally distinguishable under the model.

If it is nearly singular, some parameter combinations are only weakly distinguishable.

Thus likelihood methods connect naturally to the identifiability analysis developed earlier.

Asymptotic Normality

Under suitable regularity conditions and with enough data, maximum-likelihood estimators are often approximately normally distributed:

This approximation supports standard-error and confidence-interval calculations.

Approximate Standard Errors

After fitting, an approximate covariance matrix can be obtained from the inverse observed information:

The square roots of the diagonal entries provide approximate standard errors.

Likelihood-Based Confidence Intervals

Confidence intervals do not have to rely only on quadratic curvature.

One can examine parameter values for which the log-likelihood remains close to its maximum.

This often provides a better picture when the likelihood is asymmetric or strongly nonlinear.

Profile Likelihood

Suppose \(\theta_1\) is the parameter of interest and the remaining parameters are nuisance parameters.

Fix \(\theta_1\) at a chosen value and maximize the likelihood over all remaining parameters:

The resulting function is the profile likelihood.

What a Profile Likelihood Reveals

A sharply peaked profile indicates that the data strongly constrain the parameter.

A broad profile indicates weak practical identifiability.

A profile with several peaks suggests multiple competing parameter regions.

A profile that does not close on one side may indicate an effectively unbounded confidence region.

Likelihood-Ratio Comparisons

Two parameter values can be compared through their log-likelihoods.

A common quantity is

Under regular conditions, likelihood-ratio statistics have useful large-sample reference distributions.

Constrained Maximum Likelihood

Parameters may be subject to physical restrictions:

or

Maximum likelihood can be performed over the admissible parameter region rather than over all real values.

Boundary Estimates

Sometimes the maximum occurs on a parameter boundary.

For example,

may be the best estimate when the parameter is constrained to be nonnegative.

Standard asymptotic formulas may behave differently at boundaries, so such solutions deserve special attention.

Reparameterization

Constraints can often be handled through transformations.

To guarantee

write

To guarantee a probability in \((0,1)\), use the logistic transformation

Such transformations can simplify optimization.

Optimization of the Likelihood

Simple likelihoods can sometimes be maximized analytically.

More often, especially in nonlinear inverse problems, numerical optimization is required.

Methods may use:

• gradients;
• Hessians;
• quasi-Newton approximations;
• trust regions;
• derivative-free searches;
• multi-start strategies.

Local Maxima

A nonlinear likelihood can contain several local maxima.

Therefore, a successful numerical optimization does not necessarily establish that the global maximum has been found.

Multiple initial guesses and likelihood-surface exploration can reveal competing modes.

Flat Likelihood Regions

If the likelihood changes very little over a wide parameter region, then the data weakly constrain the parameter.

Numerically, this may appear as:

• slow optimization;
• large parameter changes with little likelihood change;
• large standard errors;
• unstable fitted values;
• high parameter correlation.

Likelihood Ridges

Two or more parameters may trade off against each other so that many combinations have nearly equal likelihood.

Geometrically, the likelihood then contains a ridge.

Such ridges are closely related to confounding and practical nonidentifiability.

Parameter Scaling

Likelihood optimization can be difficult when parameters have very different numerical scales.

Scaled variables such as

can improve optimizer behavior.

Scaling does not alter the underlying statistical model when implemented correctly.

Likelihood Contributions

For independent observations,

Inspecting individual contributions

can reveal observations that are especially surprising under the fitted model.

Deviance

In many likelihood-based models, fit quality is summarized by comparing the fitted likelihood with a more flexible reference model.

A common deviance form is

Smaller deviance generally corresponds to closer fit relative to the reference model.

Likelihood and Model Comparison

More flexible models often achieve larger maximum likelihood simply because they contain more parameters.

Therefore, raw likelihood alone is not sufficient for comparing models of different complexity.

Later model-selection methods introduce penalties or predictive criteria to balance fit against complexity.

Maximum Likelihood Does Not Guarantee a Good Model

Maximum likelihood finds the best parameter value within the assumed model family.

If the entire model family is wrong, the MLE can still be the best available member of a poor family.

Therefore, residual checks, predictive validation, and model criticism remain necessary.

Model Misspecification

Misspecification can arise from:

• an incorrect forward model;
• an incorrect noise distribution;
• unmodeled correlation;
• incorrect variance structure;
• missing covariates or states;
• incorrect censoring or truncation assumptions.

A carefully optimized incorrect likelihood remains an incorrect model.

Residuals in Likelihood Models

Residuals remain useful even when ordinary differences are not the most natural measure.

Depending on the model, analysts may examine:

• raw residuals;
• standardized residuals;
• Pearson residuals;
• deviance residuals;
• probability-scale diagnostics.

Standardized Residuals

If the model predicts mean \(\mu_i\) and standard deviation \(\sigma_i\), a standardized residual is

This measures discrepancy relative to the model's expected noise scale.

Checking Distributional Assumptions

If Gaussian errors are assumed, residual diagnostics should examine whether the errors are approximately:

• centered around zero;
• roughly symmetric;
• consistent in variance when homoscedasticity is assumed;
• free from strong serial correlation when independence is assumed.

The same principle applies to other assumed distributions.

Censored Data

Sometimes an observation is known only to lie above or below a detection threshold.

Maximum likelihood can incorporate this information directly.

For example, if only

Missing Data

Missing observations complicate the likelihood because some components of the complete data are unobserved.

Depending on the model and missingness mechanism, one may integrate or sum over unobserved quantities rather than simply discard incomplete records.

Latent Variables

Some models include unobserved internal variables

The observed-data likelihood may require

Such integrals can make likelihood evaluation significantly more difficult.

The EM Idea

For some latent-variable problems, the expectation-maximization method alternates between:

1. estimating expected latent information under current parameters;
2. maximizing an expected complete-data log-likelihood.

This transforms a difficult direct optimization into repeated simpler steps.

Likelihood in Dynamic Models

In differential-equation or state-space models, a parameter vector may first determine an evolving state:

An observation model then connects the hidden state to measured data:

Likelihood evaluation may therefore require solving the forward model many times.

Computational Cost

If one forward simulation is expensive, likelihood optimization can become computationally demanding because each candidate parameter vector may require a new simulation.

Efficient methods may use:

• analytic gradients;
• automatic differentiation;
• adjoint methods;
• surrogate models;
• parallel evaluation;
• careful parameter initialization.

Synthetic Recovery Tests

A likelihood-based implementation should be tested using synthetic data generated from known parameters:

Repeated recovery experiments can test whether the estimation procedure is approximately unbiased, stable, and consistent with its claimed uncertainty.

Likelihood Surface Exploration

For one or two parameters, direct likelihood plots are especially useful.

They can reveal:

• multiple maxima;
• flat regions;
• ridges;
• boundary optima;
• asymmetric uncertainty;
• strong parameter interactions.

Maximum Likelihood and Data Volume

As the amount of informative data increases, regular maximum-likelihood estimators often become more concentrated around the true parameter value.

However, additional data do not resolve structural nonidentifiability if all observations carry the same inseparable parameter dependence.

Likelihood and Experimental Design

Experimental design can be chosen to increase expected information about parameters.

One strategy is to select experimental conditions that increase the Fisher information matrix or improve a scalar summary such as:

or reduce predicted parameter uncertainty.

Maximum Likelihood Versus Least Squares

Least squares starts with a numerical discrepancy.

Maximum likelihood starts with a probability model.

When the probability model is independent Gaussian noise with fixed variance, the two methods coincide.

Under other error models, likelihood produces different objective functions better matched to the data type.

Maximum Likelihood Versus Bayesian Estimation

Maximum likelihood uses

and selects the parameter value that maximizes it.

Bayesian estimation combines the likelihood with a prior:

The next lesson develops that broader framework.

Common Error: Maximizing Probability Over the Data

Once the data have been observed, maximum likelihood does not vary the data.

It varies the parameter:

Confusing these roles leads to incorrect reasoning about likelihood.

Common Error: Treating Likelihood as a Parameter Probability

A likelihood value does not directly mean

Such a probability requires a prior distribution and Bayesian updating.

Common Error: Ignoring Constants Too Early

Terms constant with respect to the parameter can be omitted when optimizing the likelihood.

But a term that appears constant may depend on another parameter being estimated, such as \(\sigma^2\).

Always determine which quantities are actually variable before simplifying.

Common Error: Using the Wrong Noise Model

A Gaussian likelihood for strongly skewed counts or a constant-variance model for rapidly changing measurement variance can distort parameter estimates and uncertainty.

The observation model is part of the scientific model and should be justified.

Common Error: Trusting Only the MLE

A single maximum-likelihood point estimate does not reveal:

• parameter uncertainty;
• multiple modes;
• flat likelihood directions;
• boundary sensitivity;
• model misspecification.

Likelihood shape and diagnostics are essential companions to the optimum.

Common Error: Ignoring Identifiability

A numerical optimizer can return one maximum even if an entire ridge of parameter combinations has nearly identical likelihood.

Numerical uniqueness does not prove scientific identifiability.

A Reliable Maximum-Likelihood Workflow

1. Define the forward model and predicted observables.
2. Specify the probability model for measurement variability.
3. Write the likelihood \(L(\boldsymbol{\theta};\mathbf{y})\).
4. Convert to a log-likelihood for numerical stability.
5. Identify parameter constraints and useful transformations.
6. Check structural and practical identifiability.
7. Choose an appropriate optimizer and starting values.
8. Use multiple starts or likelihood-surface exploration when needed.
9. Inspect curvature, profiles, and parameter correlations.
10. Compute uncertainty estimates.
11. Inspect residual and distributional diagnostics.
12. Validate the model on independent or synthetic data.
13. Document the likelihood assumptions explicitly.

Practice 1: Likelihood Versus Probability

In

which quantity is treated as fixed when viewing the expression as a likelihood?

Practice 2: Bernoulli MLE

A coin is tossed \(20\) times and produces \(14\) heads.

Compute the maximum-likelihood estimate of the probability of heads.

Practice 3: Log-Likelihood

Suppose

Write the corresponding log-likelihood.

Practice 4: Gaussian Connection

Under independent Gaussian errors with common known variance, what familiar objective is equivalent to maximizing the likelihood?

Practice 5: Poisson Data

A model predicts event counts with mean \(\lambda(\theta)\).

Which distribution is a natural first choice when counts arise as independent events in a fixed interval?

Practice 6: Score

What is the score function in maximum-likelihood estimation?

Practice 7: Fisher Information

What does a nearly singular Fisher information matrix suggest about parameter estimation?

Practice 8: Profile Likelihood

A profile likelihood for a parameter is very broad and almost flat near its maximum.

What does this indicate?

Practice 9: Boundary Constraint

A rate parameter must satisfy

Give one reparameterization that enforces this constraint automatically.

Practice 10: Model Misspecification

An optimizer finds the global maximum of a likelihood, but residual diagnostics show strong systematic patterns.

What should be concluded?

Practice Answers

Practice 1

The observed data \(y\) are fixed, while the parameter \(\theta\) is varied.

Practice 2

Practice 3

Practice 4

Maximizing the Gaussian likelihood is equivalent to minimizing the sum of squared residuals:

Practice 5

A natural first model is the Poisson distribution.

Practice 6

The score is the gradient of the log-likelihood:

Practice 7

A nearly singular Fisher information matrix indicates weakly informed or highly correlated parameter directions. Parameter uncertainty may be large, and practical identifiability may be poor.

Practice 8

The data weakly constrain the parameter. A broad flat profile is a signature of large uncertainty or practical nonidentifiability.

Practice 9

Set

where \(\eta\in\mathbb{R}\). Then \(k>0\) automatically.

Practice 10

Finding the global optimum does not establish that the model is adequate. Systematic residual structure indicates that the forward model, observation model, or error assumptions may be misspecified and should be investigated.

What to Remember

Maximum likelihood estimates parameters by asking which values make the observed data most plausible under an explicit probability model.

• The likelihood is \(L(\boldsymbol{\theta};\mathbf{y})=p(\mathbf{y}\mid\boldsymbol{\theta})\).
• The observed data are fixed while the parameters vary.
• The log-likelihood converts products into sums and improves numerical stability.
• Maximum likelihood often becomes minimization of a negative log-likelihood.
• Least squares is the Gaussian-error special case.
• Poisson, binomial, lognormal, and other likelihoods handle different data types.
• The score and likelihood curvature describe local parameter information.
• Fisher information connects likelihood shape to parameter precision.
• Profile likelihoods expose uncertainty, asymmetry, and weak identifiability.
• Likelihood optimization can contain local maxima, ridges, and boundary solutions.
• A maximum-likelihood estimate is only as credible as the forward model and error model.
• Residual diagnostics and validation remain essential.

The central principle is:

Where We Go Next

Maximum likelihood uses the observed data to measure how strongly different parameter values are supported by an assumed probability model.

The next lesson, Bayesian Estimation, adds another source of information: prior knowledge about the parameters. Bayes' rule combines the likelihood with a prior distribution to produce a posterior distribution that represents updated uncertainty after observing the data.