Parameter Fitting

The Central Idea

Suppose a model depends on parameters

and predicts observations through

We also have measured data

Parameter fitting seeks a parameter vector for which

agrees well with the observed data.

In compact form,

where \(\Phi\) measures model-data disagreement.

From Forward Model to Fitting Problem

The forward problem starts with parameters and predicts data:

The fitting problem begins with observations and searches for parameters:

Because the inverse direction is usually not available as a simple formula, parameter fitting is commonly performed by repeated forward evaluations.

The Residual

The difference between an observed value and a predicted value is called a residual.

For one observation,

Since

we may write

The Residual Vector

With \(n\) observations, collect the residuals into a vector:

Equivalently,

The residual vector is the fundamental bridge between prediction and estimation.

Why Residuals Matter

Residuals tell us both the magnitude and direction of mismatch.

If

then the model prediction is smaller than the observation.

If

then the model prediction is larger than the observation.

If

then the model matches that observation exactly.

From Residuals to an Objective Function

An optimizer usually requires a single scalar quantity to minimize.

The residual vector must therefore be summarized by an objective function or loss function.

A common choice is the sum of squared residuals:

In vector notation,

Why Square the Residuals?

Squaring residuals provides several useful properties:

• positive and negative residuals do not cancel;
• larger errors receive greater penalty;
• the objective is differentiable in many common models;
• the method connects naturally to Gaussian measurement error;
• efficient numerical algorithms are available.

The resulting method is called least squares, which will be developed in greater detail in the next lesson.

A Simple One-Parameter Fit

Suppose the model is

and the observed data are

The predicted values are

The residuals are

A least-squares fit chooses \(\theta\) to minimize

Exact Fit Versus Approximate Fit

If there exists a parameter vector such that

then the residual vector is zero:

Real data usually contain noise, however, so an exact match is often neither possible nor desirable.

Parameter fitting usually seeks the best compromise among many imperfect observations.

Overdetermined Fitting Problems

A fitting problem is often overdetermined, meaning that the number of observations exceeds the number of parameters:

In this situation, the model generally cannot satisfy every observation exactly.

The objective function provides a systematic way to choose the parameter vector that best balances the discrepancies.

Underdetermined Fitting Problems

If

Parameter Fitting Depends on Identifiability

A fitting algorithm can produce a numerical answer even when the model is nonidentifiable.

If many parameter vectors produce nearly the same predictions, the optimizer may simply select one member of a broad family of solutions.

Therefore,

Weighted Residuals

Not all observations should necessarily contribute equally to the objective.

Suppose observation \(i\) has scale or uncertainty \(\sigma_i\). A scaled residual may be defined as

The corresponding weighted least-squares objective is

Why Weighting Matters

Suppose one measurement is expressed in thousands while another is expressed in tenths.

If raw residuals are squared without scaling, the numerically larger variable may dominate the fit even when it is not more scientifically important.

Weighting can account for:

• measurement uncertainty;
• different physical units;
• different response magnitudes;
• different confidence levels;
• different sampling frequencies.

Matrix Form of a Weighted Objective

Let \(W\) be a weighting matrix. Then a general quadratic objective can be written

If the measurement errors are independent with variances \(\sigma_i^2\), a common choice is

Correlated Measurement Errors

Measurement errors are not always independent.

If the error covariance matrix is

then a natural quadratic objective is

Off-diagonal covariance terms represent statistical relationships among measurement errors.

Absolute-Error Objectives

Squared residuals are not the only possible choice.

An absolute-error objective is

Compared with squared error, absolute error reduces the influence of very large residuals.

The choice of objective should reflect the assumed error structure and the goals of the analysis.

Relative-Error Objectives

When observations span several orders of magnitude, relative errors may be more informative than absolute errors.

One form is

A corresponding objective is

Care is required when observed values are zero or very close to zero.

Logarithmic Residuals

For positive quantities spanning wide ranges, one may compare logarithms:

This treats multiplicative differences more naturally than raw additive differences.

For example, predicting \(20\) instead of \(10\) produces the same logarithmic ratio error as predicting \(200\) instead of \(100\).

Choosing the Parameterization

A parameter can often be represented in several equivalent ways.

If a parameter must remain positive, instead of fitting

directly, one may introduce

Then

is automatically positive for every real \(\eta\).

Why Reparameterization Can Help

Reparameterization can improve:

• constraint handling;
• parameter scaling;
• numerical conditioning;
• optimization geometry;
• interpretation of identifiable combinations.

A good parameterization makes the structure of the fitting problem easier for both the optimizer and the analyst to interpret.

Parameter Bounds

Physical knowledge may restrict parameters to an interval:

Examples include:

• positive rate constants;
• probabilities between \(0\) and \(1\);
• material properties within feasible ranges;
• geometric dimensions constrained by design.

Bounds can prevent meaningless trial values and improve computational efficiency.

Bounds Do Not Replace Identifiability

If an optimizer returns a parameter at a boundary, this can indicate that:

• the true optimum lies outside the allowed region;
• the data are weakly informative;
• the model is misspecified;
• the parameter is poorly identifiable;
• the bound itself is influencing the result.

A fitted boundary value should therefore be interpreted carefully.

Initialization

Many nonlinear fitting algorithms require an initial guess

The initial guess can influence:

• which local minimum is reached;
• how quickly the algorithm converges;
• whether the solver enters an invalid region;
• whether the fit succeeds at all.

Initialization is therefore part of the modeling workflow, not merely a software detail.

Local and Global Minima

A nonlinear objective function can possess several minima.

A local minimum is better than nearby parameter values.

A global minimum is at least as good as every admissible parameter value.

A local optimization algorithm can converge successfully while still missing the global best fit.

Multi-Start Fitting

One practical strategy is to solve the fitting problem repeatedly from several initial guesses.

If many starts converge to the same parameter vector and objective value, confidence in the optimization result increases.

If different starts converge to substantially different solutions, the objective may contain multiple basins or flat directions that deserve further investigation.

Grid Searches

For one or two parameters, a direct grid evaluation can be highly informative.

Evaluate

over a range of parameter values.

The resulting surface can reveal:

• multiple minima;
• flat valleys;
• parameter correlations;
• boundary effects;
• poor scaling.

Gradient-Based Fitting

If the objective is differentiable, gradient-based methods use

to determine directions of decrease.

These methods can converge efficiently when the objective is smooth and the derivatives are accurate.

They can be less reliable when the model is discontinuous, noisy, poorly scaled, or contains many local minima.

Derivative-Free Fitting

Derivative-free methods estimate progress using function evaluations rather than explicit gradients.

They may be useful when:

• the forward model is a black box;
• derivatives are unavailable;
• the model includes nondifferentiable components;
• numerical noise corrupts finite differences.

Their efficiency can deteriorate as the number of parameters grows.

The Jacobian of the Residuals

For nonlinear least squares, an important matrix is the residual Jacobian

Since

we have

Thus the fitting Jacobian is directly connected to the model sensitivity matrix.

Gradient of a Least-Squares Objective

For

the gradient is

This equation connects residual size and parameter sensitivity.

A direction is important to the optimizer when changing parameters both affects the predictions and reduces the observed residuals.

Linearization Near a Current Estimate

Around a current parameter estimate

the residual vector can be approximated by

Many nonlinear least-squares algorithms use this local linear approximation to choose the next parameter update.

Gauss-Newton Structure

In a least-squares problem, the Gauss-Newton step approximately solves

The parameter estimate is then updated:

This connects nonlinear parameter fitting to repeated linear least-squares problems.

Ill-Conditioned Parameter Updates

If the columns of \(J_r\) are nearly dependent, then

can be poorly conditioned.

Small changes in the data or residuals may then cause large changes in the computed parameter update.

This is one way practical identifiability problems appear inside fitting algorithms.

Parameter Scaling

Suppose one parameter is of order

while another is of order

Treating both parameters on the same raw numerical scale can make optimization inefficient.

Scaled parameters may be introduced as

where \(s_j\) is a characteristic parameter scale.

Residual Scaling

Residual components may also require scaling.

If one observable naturally has values near \(10^6\) and another near \(10^{-2}\), raw squared residuals may overwhelmingly favor the first.

Good residual scaling makes the objective reflect scientific relevance, measurement uncertainty, or meaningful relative error rather than arbitrary numerical units.

Fitting Multiple Data Sets

A single parameter vector may need to explain several experiments.

Suppose experiments \(1,\ldots,m\) produce residual vectors

A combined objective may be

Joint fitting can reveal whether one parameter set consistently explains multiple operating conditions.

Shared and Experiment-Specific Parameters

In multi-experiment fitting, some parameters may be shared while others vary by experiment.

For example,

This structure can separate universal model properties from experiment-specific initial conditions, offsets, or nuisance parameters.

Nuisance Parameters

A nuisance parameter affects the observations but is not the main scientific quantity of interest.

Examples include:

• instrument offsets;
• background levels;
• calibration factors;
• unknown initial states;
• timing shifts.

Nuisance parameters may still need to be fitted so that the parameters of interest are not biased.

Data Preprocessing and Fitting

Fitting results can depend strongly on preprocessing choices.

Examples include:

• baseline subtraction;
• smoothing;
• normalization;
• outlier removal;
• time alignment;
• unit conversion.

Preprocessing should be documented because it changes the data presented to the inverse problem.

Outliers

An outlier is an observation that differs substantially from the pattern of the remaining data.

Under squared-error fitting, large residuals receive disproportionately large weight because

grows rapidly with \(|r|\).

Outliers can therefore pull fitted parameters away from values supported by most of the data.

Robust Fitting

Robust loss functions reduce the influence of unusually large residuals.

A robust objective may behave approximately like

for small residuals but grow more slowly for large residuals.

Robust fitting can be appropriate when occasional contamination or outliers are expected, but the choice should reflect the data-generation process.

Fitting Is Not Interpolation

Interpolation constructs a function that passes exactly through specified data points.

Parameter fitting instead chooses parameters within a prescribed model family.

The fitted model generally does not pass through every observation.

Its purpose is to infer a parameterized relationship, not merely to reproduce each data point exactly.

Fitting Is Not Model Validation

A low fitting error does not prove that a model is correct.

A flexible model may fit one data set closely while failing under new conditions.

Model validation asks whether the fitted model makes adequate predictions outside the information directly used to determine its parameters.

Training and Validation Data

One strategy is to divide the available data into:

• a fitting or training set;
• a validation or testing set.

Parameters are estimated using the fitting data.

The resulting parameter values are then used to predict observations that were not used during fitting.

Good out-of-sample performance provides stronger evidence than fit quality alone.

Residual Plots

After fitting, residuals should be inspected rather than summarized only by a single objective value.

Useful plots include residuals versus:

• time;
• predicted value;
• input value;
• experimental condition;
• measurement index.

Structured residual patterns can reveal model deficiencies that a single goodness-of-fit number hides.

Residuals Should Not Contain Obvious Structure

Ideally, residuals should look consistent with the assumed measurement noise.

Persistent trends, oscillations, clusters, or systematic sign changes can indicate:

• missing dynamics;
• incorrect functional form;
• unmodeled dependence;
• measurement bias;
• incorrect weighting.

Underfitting

Underfitting occurs when the model is too simple to capture important structure in the data.

Symptoms can include:

• large residuals;
• systematic residual patterns;
• poor prediction across conditions;
• consistent bias.

Underfitting cannot be repaired simply by optimizing the same inadequate model more aggressively.

Overfitting

Overfitting occurs when a model adapts too closely to the particular noise or accidental features of the fitting data.

An overfit model may have:

• very small training residuals;
• unstable parameter estimates;
• poor performance on new data;
• unnecessarily large parameter count.

Model selection and regularization help address this tradeoff.

Parameter Uncertainty After Fitting

A best-fit value alone does not describe how strongly the parameter is determined.

A complete analysis should ask how much the parameter could vary while remaining consistent with the data.

Uncertainty can be investigated through:

• covariance approximations;
• confidence intervals;
• profile likelihoods;
• bootstrap methods;
• Bayesian posterior distributions.

Approximate Covariance Near a Least-Squares Solution

Under suitable assumptions, local parameter uncertainty is connected to

Roughly speaking, if this matrix is well conditioned and has large curvature in all parameter directions, the data strongly constrain the parameters.

Small eigenvalues indicate directions in which the objective changes only weakly.

Confidence Regions and Objective Geometry

In two parameters, a well-constrained fit may produce a compact region around the optimum.

A long narrow region indicates strong parameter correlation.

A nearly open valley indicates weak identifiability.

The shape of the objective surface is therefore as informative as the location of its minimum.

Parameter Fitting with Constraints

Some models require equality or inequality constraints in addition to simple bounds.

For example,

or

Constrained optimization methods can enforce these relationships during the fitting process.

Transforming a Sum-to-One Constraint

Suppose

Instead of fitting both independently, write

The fitting problem then contains only one independent parameter.

Reparameterizing exact constraints can reduce dimensionality and eliminate unnecessary degrees of freedom.

Stopping Criteria

Numerical fitting algorithms need rules for deciding when to stop.

Common stopping criteria include:

• small change in objective value;
• small change in parameter values;
• small gradient norm;
• small step size;
• maximum iteration count.

Stopping is a numerical decision and does not by itself establish model adequacy.

Convergence Diagnostics

After optimization, check:

• whether the solver reported convergence;
• whether parameter bounds were active;
• whether different starting values give similar results;
• whether gradients are small;
• whether residuals are sensible;
• whether the final objective is substantially better than the initial one.

Verification with Synthetic Data

Before fitting experimental data, a useful test is to generate synthetic data from a known parameter vector

Then fit the model back to those generated observations.

If the fitting pipeline cannot recover known parameters under controlled conditions, the implementation should be investigated before real-data conclusions are trusted.

Noise-Free Recovery Test

The strongest initial test uses noise-free synthetic data:

Fit

using the same forward model.

For an identifiable and correctly implemented problem, the recovered parameter vector should closely match \(\boldsymbol{\theta}_{\text{true}}\).

Noisy Recovery Test

Next add simulated measurement noise:

Repeated fits show how parameter estimates vary under realistic noise.

This helps assess:

• bias;
• variance;
• practical identifiability;
• robustness of initialization;
• coverage of uncertainty intervals.

Fit Quality and Scientific Meaning

A parameter estimate should satisfy more than a numerical criterion.

Ask whether the fitted value:

• lies in a physically plausible range;
• is consistent with independent measurements;
• remains stable across related data sets;
• supports accurate predictions;
• has uncertainty appropriate to the available information.

Common Error: Minimizing the Wrong Quantity

A fitting algorithm faithfully minimizes the objective it is given.

If the objective poorly represents the scientific problem, the resulting parameters can be mathematically optimal but scientifically inappropriate.

Objective design is therefore part of modeling.

Common Error: Ignoring Measurement Uncertainty

Suppose one observation has uncertainty \(0.01\) while another has uncertainty \(10\).

Treating the two raw residuals equally may give the noisy measurement far too much influence.

Weighting by uncertainty can better reflect how much information each observation contributes.

Common Error: Overinterpreting the Best-Fit Value

A reported estimate such as

can create an illusion of precision.

If the data support values from \(1.8\) to \(3.1\) almost equally well, four decimal places are not scientifically meaningful.

Reported precision should reflect parameter uncertainty.

Common Error: Trusting a Single Starting Point

A nonlinear optimizer may converge to a local minimum determined partly by its initial guess.

Repeating the fit from multiple plausible starting points is a simple and valuable diagnostic.

Common Error: Ignoring Residual Structure

A small total objective can hide systematic model failure.

Residual plots may show persistent oscillations, curvature, drift, or condition-specific bias even when the overall error is numerically modest.

Always inspect the residuals themselves.

Common Error: Fitting Parameters That the Data Cannot Distinguish

If two parameters affect the model in nearly identical ways, the optimizer may trade one against the other.

The final predictions can appear excellent while the individual parameter values remain unstable.

Identifiability analysis should therefore precede or accompany fitting.

A Reliable Parameter-Fitting Workflow

1. Define the forward model and observation operator.
2. Identify the unknown parameters.
3. Check structural and practical identifiability.
4. Define residuals in meaningful units.
5. Choose an objective function consistent with the error model.
6. Apply appropriate weighting and scaling.
7. Set physically justified parameter bounds or constraints.
8. Choose plausible initial guesses.
9. Run the optimization.
10. Repeat from multiple starts when appropriate.
11. Inspect residuals and convergence diagnostics.
12. Estimate parameter uncertainty.
13. Validate predictions on independent or withheld data.
14. Document preprocessing, assumptions, and solver settings.

Practice 1: Residual Calculation

An observed value is

and the model predicts

Compute the residual using

Practice 2: Sum of Squared Residuals

Suppose the residual vector is

Compute

Practice 3: Weighted Residual

Suppose

Compute the scaled residual

Practice 4: One-Parameter Model

For the model

with data point

write the residual as a function of \(\theta\).

Practice 5: Objective at a Trial Parameter

Using

compute the squared-error objective at

Practice 6: Parameter Bounds

A rate parameter must satisfy

Give one reason why enforcing a positive bound during fitting can be useful.

Practice 7: Multiple Starts

Three optimization runs from different starting points produce:

What does this suggest about the fitting problem?

Practice 8: Residual Pattern

After fitting a time-dependent model, residuals are negative at early times and positive at late times.

Why should this pattern be investigated?

Practice 9: Synthetic Recovery

Synthetic data are generated using

In a noise-free test, the fitting procedure consistently returns

What should be concluded?

Practice 10: Fit Versus Validation

A model fits its training data almost perfectly but performs poorly on a withheld validation data set.

What modeling concern does this raise?

Practice Answers

Practice 1

Practice 2

Practice 3

Practice 4

The prediction is

Therefore,

Practice 5

At \(\theta=2\),

Therefore,

Practice 6

Enforcing positivity prevents the optimizer from testing physically meaningless negative rate constants and can reduce the search region.

Practice 7

The different outcomes suggest multiple local minima, dependence on initialization, weak identifiability, or another complicated feature of the objective surface. The result should be investigated rather than accepting a single optimization run.

Practice 8

The residuals contain systematic structure rather than random-looking variation. This may indicate a missing trend, incorrect dynamics, biased measurement model, or another form of model misspecification.

Practice 9

A correctly implemented identifiable model should recover the known parameter in a noise-free self-consistency test. The discrepancy indicates an implementation, optimization, observation-model, or identifiability problem that should be resolved before fitting real data.

Practice 10

This raises concern about overfitting. The model may have adapted too closely to the particular fitting data without capturing a relationship that generalizes.

What to Remember

Parameter fitting converts model-data mismatch into a numerical estimation problem.

• Residuals measure the difference between observations and predictions.
• An objective function summarizes residual disagreement.
• Least squares minimizes the sum of squared residuals.
• Weights can represent uncertainty, scaling, or relative importance.
• Bounds and reparameterizations can enforce physically meaningful values.
• Nonlinear fitting can contain local minima and depend on initialization.
• Parameter and residual scaling affect numerical performance.
• Residual inspection is essential after optimization.
• A good fit does not prove identifiability or model validity.
• Synthetic recovery tests are valuable implementation checks.
• Uncertainty and validation should accompany the best-fit parameter values.

The key principle is:

Where We Go Next

Parameter fitting provides the general framework for comparing model predictions with observations and selecting parameter values that minimize disagreement.

The next lesson, Least-Squares Estimation, develops the most widely used fitting method in greater detail. We will examine linear and nonlinear least squares, normal equations, projection, weighting, residual geometry, and the numerical structure of least-squares solutions.