Least-Squares Estimation

The Central Idea

Suppose a model predicts observations from a parameter vector \(\boldsymbol{\theta}\). The observed data are

If the model prediction is

then the residual vector is

Least-squares estimation chooses the parameter vector that minimizes

Why Least Squares Is Needed

In many inverse problems, there are more observations than unknown parameters.

If

where \(n\) is the number of observations and \(p\) is the number of parameters, the system is often overdetermined.

Because noisy measurements generally do not lie exactly on the model, there may be no parameter vector that satisfies every observation simultaneously.

Least squares provides a principled best compromise.

Linear Least Squares

The basic linear least-squares problem has the form

where:

• \(\mathbf{y}\in\mathbb{R}^n\) is the observation vector;
• \(X\in\mathbb{R}^{n\times p}\) is the design matrix;
• \(\boldsymbol{\theta}\in\mathbb{R}^p\) is the unknown parameter vector;
• \(\boldsymbol{\varepsilon}\) represents measurement or model error.

The Linear Residual

For a trial parameter vector \(\boldsymbol{\theta}\), the predicted data are

Therefore,

The least-squares objective is

Expanding the Objective

The objective can be written

Expanding gives

This is a quadratic function of the unknown parameter vector.

Deriving the Normal Equations

Differentiate the objective with respect to \(\boldsymbol{\theta}\):

At a least-squares minimizer,

Therefore,

These are the normal equations.

The Closed-Form Solution

If \(X\) has full column rank, then

is invertible.

The least-squares estimate is then

This formula is important conceptually, although numerical software often avoids explicitly forming the inverse.

A One-Parameter Example

Suppose

The least-squares objective is

Differentiate:

Set this equal to zero:

Therefore,

Fitting a Line with an Intercept

For the model

the design matrix is

and

Least squares selects the intercept and slope that minimize the total squared vertical residuals.

The Geometric Interpretation

The vector

lies in the column space of \(X\).

If the observed vector \(\mathbf{y}\) does not lie in that column space, an exact solution is impossible.

Least squares chooses the point in the column space closest to \(\mathbf{y}\).

Thus,

The Residual Is Orthogonal to the Model Space

At the least-squares solution,

Since

we obtain

Therefore, the least-squares residual is orthogonal to every column of the design matrix.

Why the Normal Equations Are Called Normal

In geometry, a vector is normal to a subspace when it is perpendicular to that subspace.

The least-squares residual satisfies

The normal equations express exactly this orthogonality condition.

The Projection Matrix

When \(X\) has full column rank, the fitted vector is

Define

Then

The matrix \(P\) projects observation vectors onto the column space of \(X\).

The Residual Projection

Since

we may write

Thus \(I-P\) projects onto the orthogonal complement of the model space.

Pythagorean Decomposition

Because the fitted vector and residual are orthogonal,

gives

Least-squares geometry is therefore a direct application of orthogonal decomposition.

Uniqueness of the Least-Squares Estimate

The fitted prediction

is unique as the orthogonal projection of \(\mathbf{y}\) onto a subspace.

The parameter vector itself is unique only if \(X\) has full column rank.

If the columns of \(X\) are linearly dependent, multiple parameter vectors may produce exactly the same fitted prediction.

Rank Deficiency

If

Near Rank Deficiency

A design matrix can have full rank and still be nearly rank deficient.

This occurs when one column is almost a linear combination of others.

Then parameter estimates may become extremely sensitive to small changes in the data.

The problem is mathematically identifiable but practically ill conditioned.

Conditioning of the Normal Equations

Forming

can worsen numerical conditioning.

Roughly,

where \(\kappa\) is a condition number.

Therefore, direct solution of the normal equations can lose accuracy when the design matrix is poorly conditioned.

Why We Usually Do Not Form the Inverse

The expression

is useful theoretically.

Numerically, explicitly computing a matrix inverse is often less stable and less efficient than solving a linear system or using matrix factorizations.

Practical least-squares software usually uses QR decomposition, singular value decomposition, or specialized sparse solvers.

QR Factorization

Suppose

where \(Q\) has orthonormal columns and \(R\) is upper triangular.

Then

The least-squares problem can be reduced to solving a triangular system

Why QR Is Useful

QR factorization avoids explicitly forming \(X^TX\), which generally improves numerical stability.

It is a standard method when the design matrix has full or nearly full column rank.

Householder reflections and Givens rotations are common computational tools for constructing QR factorizations.

Singular Value Decomposition

The singular value decomposition writes

The diagonal entries of \(\Sigma\),

are the singular values.

They reveal how strongly different parameter directions affect the observations.

SVD and Identifiability

If a singular value is exactly zero, there is a parameter direction that does not affect the predicted observations.

If a singular value is very small, the corresponding direction affects the observations only weakly.

Therefore:

The Pseudoinverse

The Moore-Penrose pseudoinverse provides a generalized solution when \(X\) is rectangular or rank deficient.

If

then

where nonzero singular values are replaced by their reciprocals.

A least-squares solution can then be written

Minimum-Norm Solutions

When an underdetermined system has infinitely many exact solutions, the pseudoinverse selects the solution with the smallest Euclidean norm:

subject to satisfying the observational constraints.

This is one natural way to choose among many equivalent parameter vectors.

Weighted Least Squares

If observations have unequal uncertainty, we may minimize

where \(W\) is a positive-definite weighting matrix.

The weighted normal equations are

Weighted Closed-Form Estimate

If \(X^TWX\) is invertible, then

Observations receiving larger weights have greater influence on the fit.

Inverse-Variance Weighting

If observation \(i\) has variance

and measurement errors are independent, a natural choice is

More precise observations receive more weight.

Correlated Errors

If measurement errors have covariance matrix

then generalized least squares uses

The objective becomes

This accounts for both unequal variances and correlation among errors.

Whitening the Residuals

If

where \(L\) is a matrix factor, define transformed quantities

Then generalized least squares becomes ordinary least squares:

This transformation is often called whitening.

Least Squares and Gaussian Error

Suppose measurement errors are independent and Gaussian:

Then the probability density of observing the data contains the factor

Maximizing this likelihood is equivalent to minimizing

This is the key connection between least squares and maximum likelihood.

Least Squares Does Not Require Exact Gaussianity

Least squares can still be useful even when the errors are not perfectly Gaussian.

However, the interpretation of standard errors, confidence intervals, and optimality may change when the error assumptions are violated.

The loss function should be chosen with the data-generating process in mind.

Residual Sum of Squares

The minimized least-squares objective is often called the residual sum of squares:

Smaller RSS indicates closer agreement between model and observations, holding the data and model scaling fixed.

Mean Squared Error

A normalized residual measure is

In statistical estimation, a degrees-of-freedom adjustment is often used:

Under suitable assumptions, \(s^2\) estimates the error variance \(\sigma^2\).

Root Mean Squared Error

The root mean squared error is

Unlike squared error, RMSE has the same physical units as the observed quantity.

Parameter Covariance in Linear Least Squares

Under the model

the least-squares estimator has covariance

What the Covariance Matrix Tells Us

Diagonal entries describe parameter variances:

Off-diagonal entries describe how parameter estimates vary together.

Large variances or strong correlations often indicate weak information or nearly dependent model effects.

Standard Errors

The standard error of parameter \(j\) is

A large standard error relative to the parameter magnitude signals substantial uncertainty.

Confidence Intervals

Under suitable statistical assumptions, a parameter confidence interval can be constructed from the estimate and its standard error.

A generic form is

where \(c\) depends on the desired confidence level and reference distribution.

Confidence intervals describe uncertainty in estimation, not guaranteed physical bounds.

Leverage

In linear least squares, the projection matrix

has diagonal entries

These are called leverages.

Observations with high leverage occupy unusual locations in predictor space and can strongly influence fitted parameters.

Influence

An observation can have a small residual yet still strongly affect the fit if it has high leverage.

Influence diagnostics examine how parameter estimates or fitted values change when individual observations are removed or perturbed.

Residual size alone therefore does not fully describe the importance of an observation.

Least Squares and Scaling

Poorly scaled columns of \(X\) can make the numerical problem difficult.

Suppose one column is of order \(10^{-8}\) and another is of order \(10^6\). Rescaling parameters or columns can improve numerical behavior without changing the underlying model.

Scaling is especially important before interpreting condition numbers or singular values.

Centering Predictors

In some regression problems, replacing \(x_i\) by

can reduce correlation between the intercept and slope terms.

Centering often improves interpretation and sometimes improves numerical conditioning.

Polynomial Least Squares

A quadratic model can be written

The design matrix becomes

Although the model is nonlinear in \(x\), it is linear in the unknown coefficients and is therefore a linear least-squares problem.

Linear in Parameters Versus Linear in Variables

The phrase linear least squares refers to linearity in the unknown parameters.

For example,

is linear in \(a\) and \(b\), even though the predictors involve nonlinear functions of \(x\).

Therefore, the problem can still be written

Nonlinear Least Squares

If the model is nonlinear in the unknown parameters,

then the objective is

There is usually no direct closed-form solution.

Local Linearization

Near a current estimate \(\boldsymbol{\theta}_k\), approximate the model by

where

This converts the nonlinear problem locally into a linear least-squares update.

Gauss-Newton Method

Let

The Gauss-Newton step approximately solves

The corresponding normal equations are

Then

When Gauss-Newton Works Well

Gauss-Newton is often effective when:

• the residuals are reasonably small near the solution;
• the model is smooth;
• the Jacobian has adequate rank;
• the initial guess is reasonably close to the solution.

Performance can deteriorate with poor initialization or severe nonlinearity.

Levenberg-Marquardt Idea

A common modification adds a stabilizing term:

When \(\lambda\) is large, the step is more conservative.

When \(\lambda\) is small, the method behaves more like Gauss-Newton.

This provides a bridge between gradient descent and Gauss-Newton behavior.

Residual Diagnostics Still Matter

Least squares produces an optimum relative to the selected objective, but the residual pattern must still be inspected.

A good residual pattern should be consistent with the assumed error process.

Systematic trends can indicate that the model family itself is inadequate.

Heteroscedasticity

Heteroscedasticity means that the measurement variance changes across observations.

For example,

may increase with the magnitude of the response.

In such cases, ordinary least squares may give inefficient or misleading uncertainty estimates, while weighted least squares can better reflect the error structure.

Outliers and Squared Loss

Squared loss grows rapidly:

Therefore, one very large residual can have substantial influence on the fitted parameters.

Least squares is highly effective when the assumed error model is appropriate, but it is not automatically robust to outliers.

Regularized Least Squares

When the inverse problem is unstable or underdetermined, a penalty can be added:

The first term measures data mismatch.

The second term penalizes undesirable parameter behavior.

This leads toward the later lesson on regularization.

Ridge-Type Least Squares

If

then

The corresponding normal equations become

The added diagonal term can improve stability in weak parameter directions.

Bias-Variance Tradeoff

Regularization generally introduces bias because the solution is pulled toward values favored by the penalty.

In return, it can substantially reduce variance and sensitivity to noise.

This creates a fundamental tradeoff:

Least Squares with Constraints

Parameters may be subject to constraints such as

Then the problem becomes

subject to the stated restrictions.

Nonnegative least squares is a common example.

Why Constraints Matter

In physical estimation problems, negative values may be impossible for quantities such as:

• concentrations;
• masses;
• rate constants;
• intensities;
• component fractions.

Constrained least squares incorporates such knowledge directly into the estimation problem.

Recursive Least Squares

Sometimes observations arrive sequentially rather than all at once.

Recursive least-squares methods update the parameter estimate as new data become available without solving the full problem from scratch each time.

This is useful in:

• online calibration;
• adaptive control;
• tracking systems;
• streaming estimation;
• real-time monitoring.

Batch Versus Sequential Estimation

Batch least squares uses all available observations at once.

Sequential least squares updates the estimate as observations arrive.

Both are built on the same principle of minimizing accumulated squared prediction errors.

Least Squares and Experimental Design

The precision of parameter estimates depends on the structure of \(X\).

Experimental design can therefore improve estimation by choosing conditions that make the columns of \(X\) more independent and informative.

In nonlinear problems, the same idea applies to the sensitivity matrix \(J\).

More Observations Are Not Always Better

Additional observations improve estimation only when they contribute useful information.

Repeating nearly identical measurements can reduce random uncertainty but may do little to resolve parameter correlations or rank deficiency.

Informative diversity in the design can matter more than raw data volume.

Checking a Least-Squares Implementation

A robust implementation should be tested with synthetic problems for which the answer is known.

Useful checks include:

• exactly determined systems;
• noise-free overdetermined systems;
• known noisy systems;
• rank-deficient cases;
• scaled and badly scaled cases;
• weighted problems with known covariance.

Common Error: Solving \(X\boldsymbol{\theta}=\mathbf{y}\) Exactly

In an overdetermined noisy problem, there may be no exact solution.

Least squares does not require

Instead, it minimizes the residual norm:

Common Error: Assuming the Residual Must Be Zero

A nonzero residual is normal when observations contain noise or the model is approximate.

The relevant question is whether the residual is as small as the model allows and whether its remaining structure is consistent with the assumed errors.

Common Error: Using the Normal Equations Blindly

The normal equations are mathematically correct under appropriate rank conditions, but forming \(X^TX\) can magnify conditioning problems.

QR or SVD methods are often preferable for numerical work.

Common Error: Ignoring Rank

If \(X\) is rank deficient, the parameter vector is not uniquely determined.

Numerical software may still return one answer, but that answer should not be interpreted as uniquely identified by the data.

Common Error: Ignoring Weights

If measurement uncertainties differ strongly, unweighted least squares can give disproportionately noisy observations too much influence.

Weighting should reflect the measurement model whenever justified.

Common Error: Reporting Only the Best-Fit Parameters

A complete least-squares analysis should include more than \(\hat{\boldsymbol{\theta}}\).

Also inspect:

• residual patterns;
• uncertainty estimates;
• conditioning;
• parameter correlations;
• validation performance;
• sensitivity to model assumptions.

A Reliable Linear Least-Squares Workflow

1. Write the model in the form \(\mathbf{y}\approx X\boldsymbol{\theta}\).
2. Check units and parameter scaling.
3. Inspect the rank and conditioning of \(X\).
4. Choose weights if observation uncertainties differ.
5. Solve using a numerically stable method such as QR or SVD.
6. Compute fitted values and residuals.
7. Verify the orthogonality condition \(X^T\hat{\mathbf{r}}\approx0\).
8. Inspect residual structure.
9. Estimate parameter uncertainty.
10. Validate the fitted model where possible.

A Reliable Nonlinear Least-Squares Workflow

1. Define the nonlinear forward model \(F(\boldsymbol{\theta})\).
2. Define the residual vector.
3. Check identifiability and parameter scaling.
4. Choose initial parameter values.
5. Compute or approximate the Jacobian.
6. Use Gauss-Newton, Levenberg-Marquardt, or another suitable solver.
7. Repeat from multiple starts when necessary.
8. Inspect convergence and residual diagnostics.
9. Assess local curvature and uncertainty.
10. Test the fit on independent or synthetic data.

Practice 1: Residual Vector

Let

Compute the residual vector.

Practice 2: Residual Sum of Squares

For

compute the residual sum of squares.

Practice 3: One-Parameter Least Squares

For the model

suppose

Compute

Practice 4: Normal Equations

Starting from

state the normal equations satisfied by the least-squares estimate.

Practice 5: Orthogonality

At a linear least-squares solution, what relationship must hold between the residual vector and the columns of \(X\)?

Practice 6: Rank Deficiency

Suppose

Explain why the two parameters are not uniquely identifiable.

Practice 7: Weighted Least Squares

One observation has standard deviation \(1\), while another has standard deviation \(4\).

Under inverse-variance weighting, what are their relative diagonal weights?

Practice 8: QR Versus Normal Equations

Give one numerical reason QR factorization is often preferred to explicitly solving the normal equations.

Practice 9: Nonlinear Least Squares

In the model

explain why least-squares estimation of \(A\) and \(k\) is nonlinear.

Practice 10: Regularized Least Squares

What effect does the term

have when added to an ordinary least-squares objective?

Practice Answers

Practice 1

Practice 2

Practice 3

The numerator is

The denominator is

Therefore,

Practice 4

Practice 5

The residual must be orthogonal to every column of \(X\):

Practice 6

The second column is twice the first:

Therefore,

The two parameter effects are not independent, so multiple parameter pairs can produce the same fitted values.

Practice 7

Inverse-variance weights are

and

Therefore, the first observation receives

times the weight of the second.

Practice 8

Forming \(X^TX\) can square the condition number approximately, magnifying numerical sensitivity. QR avoids explicitly forming \(X^TX\) and is usually more stable.

Practice 9

The parameter \(k\) appears inside the exponential:

Therefore, the model is not linear in the unknown parameter vector \((A,k)\), so iterative nonlinear least-squares methods are generally needed.

Practice 10

The penalty discourages large parameter norms and can stabilize weak or poorly conditioned directions. It trades some bias for reduced sensitivity to noise and instability.

What to Remember

Least squares chooses parameters by minimizing squared model-data mismatch.

• The linear objective is \(\|\mathbf{y}-X\boldsymbol{\theta}\|_2^2\).
• The normal equations are \(X^TX\hat{\boldsymbol{\theta}}=X^T\mathbf{y}\).
• The fitted vector is the projection of the observations onto the column space of \(X\).
• The residual is orthogonal to the model space.
• Full column rank gives a unique linear parameter estimate.
• Rank deficiency indicates nonidentifiable parameter directions.
• QR and SVD are generally more stable than explicitly forming a matrix inverse.
• Weighted least squares accounts for unequal or correlated measurement uncertainty.
• Parameter covariance is tied to the geometry and conditioning of the design matrix.
• Nonlinear least squares uses local linearization and iterative updates.
• Regularization can stabilize weak parameter directions.
• Residual diagnostics remain essential after minimization.

The central relationship is:

Where We Go Next

Least-squares estimation converts model fitting into a precise optimization problem with a rich geometric and linear-algebraic structure.

The next lesson, Maximum-Likelihood Methods, broadens the estimation framework by starting from a probability model for the observed data. Least squares will appear as an important special case when measurement errors are Gaussian.