The Central Idea
Most models contain parameters whose values must be supplied.
Consider
\[ y=mx+b. \]The equation gives a model family, but it does not determine a particular model until values of \(m\) and \(b\) are chosen.
Calibration chooses or estimates those values using observations, measurements, known conditions, or other evidence.
Validation asks whether the resulting model behaves adequately when tested against relevant evidence.
A useful summary is:
\[ \boxed{\text{calibration fits the model; validation tests the model}.} \]Calibration
Calibration is the process of determining parameter values so that a model agrees with selected observations or known conditions.
Calibration may use:
- direct measurement;
- algebraic conditions;
- averages or summary statistics;
- curve fitting;
- optimization;
- maximum likelihood;
- simulation-based search;
- expert or engineering estimates.
The exact method depends on the model and the available evidence.
Validation
Validation examines whether the calibrated model is sufficiently accurate, reliable, or useful for its intended purpose.
Validation may compare model outputs with:
- new observations;
- held-out data;
- known physical behavior;
- historical periods not used for calibration;
- benchmark models;
- experimental results;
- expert expectations;
- operational requirements.
Calibration and Validation Are Different Jobs
Suppose a line is chosen to fit observed data.
\[ \hat y=2.1x+4.7. \]Determining the values \(2.1\) and \(4.7\) is a calibration task.
Checking whether this line predicts additional observations accurately is a validation task.
A model should not be considered validated merely because it reproduces the data used to construct it.
Why the Distinction Matters
A sufficiently flexible model can often match past data very closely.
But modeling usually aims at something beyond memorizing past observations. We may want to:
- predict future outcomes;
- estimate unobserved quantities;
- compare scenarios;
- support decisions;
- explain mechanisms;
- control a system.
Validation asks whether the model can support those intended uses.
A Simple Two-Point Calibration
Suppose a linear model has the form
\[ y=mx+b \]and two observed points are
\[ (2,7) \]and
\[ (6,19). \]The slope is
\[ m=\frac{19-7}{6-2}=3. \]Substituting \((2,7)\),
\[ 7=3(2)+b, \]so
\[ b=1. \]The calibrated model is
\[ \boxed{y=3x+1}. \]Calibration from Known Conditions
Calibration does not always require a large data set.
Suppose
\[ P(t)=P_0e^{rt} \]and we know
\[ P(0)=500 \]and
\[ P(4)=800. \]Then \(P_0=500\), and
\[ 800=500e^{4r}. \]Therefore,
\[ e^{4r}=1.6 \]and
\[ r=\frac{\ln(1.6)}{4}. \]The known conditions calibrate the model parameters.
Parameter Estimation
In many models, parameters cannot be measured directly.
Instead, they are estimated from data.
Suppose
\[ y_i\approx mx_i+b. \]With many observations, no single line may pass through every point.
We then choose \(m\) and \(b\) according to a fitting criterion.
Observed and Predicted Values
For observation \(i\), let
\[ y_i \]be the observed value and
\[ \hat y_i \]be the value predicted by the model.
The difference between them is the residual:
\[ \boxed{e_i=y_i-\hat y_i}. \]Residuals
Residuals measure how far model predictions are from observed values.
If
\[ y_i=12 \]and
\[ \hat y_i=10.5, \]then
\[ e_i=12-10.5=1.5. \]A positive residual means the model predicted too low. A negative residual means the model predicted too high.
Least Squares
One common calibration method chooses parameters that minimize the sum of squared residuals:
\[ \boxed{ \sum_{i=1}^{n}(y_i-\hat y_i)^2 }. \]Squaring prevents positive and negative residuals from canceling and gives larger errors greater influence.
Least squares is widely used, but it is not the only possible calibration criterion.
Absolute Error
Another error measure is absolute error:
\[ |y_i-\hat y_i|. \]The mean absolute error is
\[ \boxed{ \operatorname{MAE} = \frac{1}{n} \sum_{i=1}^{n}|y_i-\hat y_i| }. \]MAE is expressed in the same units as the modeled quantity.
Mean Squared Error
The mean squared error is
\[ \boxed{ \operatorname{MSE} = \frac{1}{n} \sum_{i=1}^{n}(y_i-\hat y_i)^2 }. \]Because errors are squared, large misses receive extra weight.
Root Mean Squared Error
Taking the square root of MSE gives
\[ \boxed{ \operatorname{RMSE} = \sqrt{ \frac{1}{n} \sum_{i=1}^{n}(y_i-\hat y_i)^2 } }. \]RMSE returns the error measure to the original units of \(y\).
Relative and Percentage Error
In some applications, scale matters.
Relative error may be written
\[ \frac{|y-\hat y|}{|y|}, \]when \(y\neq 0\).
Percentage error is then
\[ 100\% \cdot \frac{|y-\hat y|}{|y|}. \]A one-unit error may be minor when the true value is \(10{,}000\) and major when the true value is \(2\).
Calibration Data
The observations used to estimate model parameters are the calibration data.
In statistical and machine-learning contexts, this set is often called the training data.
The model is allowed to use these observations while its parameters are being selected.
Validation Data
Validation data are observations not used in the same way to fit the model.
The purpose is to obtain a more independent test of model performance.
A basic workflow is:
- use one set of observations for calibration;
- freeze the model structure and estimated parameters;
- evaluate performance on separate observations.
Holdout Validation
One simple strategy divides available data into separate portions.
For example:
- calibration set;
- validation set.
Parameters are fitted using the calibration set.
Performance is then measured on the validation set.
This reduces the risk of judging the model only by data it has already seen.
Training, Validation, and Test Sets
In larger modeling workflows, data may be divided into three roles:
- training: estimate parameters;
- validation: compare model choices or tune settings;
- test: provide a final independent performance check.
The exact terminology varies by field, but the underlying principle is the same: avoid evaluating a model only on information used to construct it.
Overfitting
Overfitting occurs when a model follows the calibration data too closely, including noise or accidental features that do not generalize.
An overfit model may have very small calibration error but much larger validation error.
High flexibility can improve fit while reducing reliability outside the calibration sample.
A Simple Overfitting Example
Suppose five observed points are available.
A fourth-degree polynomial can often be constructed to pass exactly through all five points.
That perfect fit does not imply the polynomial represents the underlying mechanism.
A simpler line or quadratic may predict new observations better.
Underfitting
Underfitting occurs when a model is too simple to capture important structure.
For example, fitting a constant value to a strongly increasing trend may produce large errors on both calibration and validation data.
Good modeling balances simplicity with adequate structure.
Fit Is Not the Same as Validity
A model can fit data well and still be unsuitable.
Reasons include:
- wrong mechanism;
- wrong domain;
- unstable extrapolation;
- biased data;
- omitted variables;
- incorrect units;
- violated assumptions;
- poor performance on new observations.
Residual Patterns Matter
Residual size is important, but residual structure also matters.
If residuals are systematically positive for small \(x\) and negative for large \(x\), the model may be missing curvature.
If residual size grows with \(x\), the variability may not be constant.
Patterns in residuals can reveal model inadequacy.
Bias
A model is biased in a practical sense when it systematically predicts too high or too low under relevant conditions.
If residuals
\[ e_i=y_i-\hat y_i \]are mostly positive, the model tends to underpredict.
If they are mostly negative, the model tends to overpredict.
Validation Against Physical Principles
Validation is not always purely statistical.
A model of probability should satisfy
\[ 0\leq p\leq 1. \]A population model should not produce negative population.
A mass-balance model should respect conservation when conservation is part of the mechanism.
Violating known structure is evidence against the model.
Validation Against Limiting Behavior
A model can also be checked at extreme or limiting cases.
If
\[ C(q)=F+cq, \]then at zero production,
\[ C(0)=F. \]If the interpretation says fixed cost should remain when production is zero, this is sensible.
Such checks test whether the model behaves correctly at important boundaries.
Validation Against Known Cases
A useful model should often reproduce known special cases.
Suppose a more complicated formula is intended to generalize a simpler model.
When the added mechanism is turned off, the new model should reduce to the known simpler case if the theory says it should.
This kind of structural check can reveal algebraic and conceptual errors.
Validation for Prediction
If prediction is the purpose, validation should measure predictive performance.
Questions include:
- How large are prediction errors?
- Are errors acceptable for the decision being made?
- Does performance deteriorate in certain regions?
- Does the model outperform a simpler benchmark?
- Does performance remain stable over time?
Validation for Explanation
A model intended to explain a mechanism faces additional requirements.
Accurate prediction alone may not establish that the internal mechanism is correct.
An explanatory model should also be evaluated for:
- plausible causal structure;
- interpretable parameters;
- consistency with established knowledge;
- behavior under interventions or changed conditions.
Validation for Decision-Making
A model used for decisions should be evaluated in terms of the consequences of error.
A forecast error of \(5\%\) may be acceptable in one context and unacceptable in another.
The validation standard should therefore depend on the intended use of the model.
Benchmarks
A model should often be compared with a simpler alternative.
A forecasting model might be compared with:
- the previous observed value;
- a historical average;
- a simple linear trend;
- a standard industry rule.
A complicated model that does not outperform a simple benchmark may not justify its additional complexity.
Cross-Validation
When data are limited, repeated data splitting can provide a more stable assessment than one holdout split.
In \(k\)-fold cross-validation, the data are divided into \(k\) groups.
The model is repeatedly calibrated on \(k-1\) groups and evaluated on the remaining group.
The validation results are then combined.
Time-Series Validation
Time-ordered data require special care.
A forecasting model should generally be calibrated using earlier observations and evaluated on later observations.
Using future data to calibrate a model that is supposed to predict the past creates information leakage.
Validation should preserve the information structure of the real task.
Calibration Windows
Dynamic systems may change over time.
A model calibrated on very old data may no longer represent current behavior.
Modelers therefore sometimes use a moving calibration window, such as the most recent \(n\) periods.
This introduces a tradeoff between using more data and using more relevant data.
Parameter Identifiability
A parameter is identifiable when the available evidence can distinguish its value sufficiently from alternatives.
Suppose a model contains parameters \(a\) and \(b\), but observations depend only on their product:
\[ y=(ab)x. \]Data may identify the product \(ab\) without determining \(a\) and \(b\) separately.
Calibration cannot recover information the data do not contain.
Multiple Parameter Sets May Fit
In complex models, different parameter combinations can sometimes produce similar outputs.
This creates calibration ambiguity.
A close fit does not always mean the estimated parameters are uniquely or precisely known.
Sensitivity and uncertainty analysis help investigate this issue.
Calibration by Optimization
Many calibration problems can be expressed as optimization.
Let \(\theta\) denote the vector of model parameters.
Choose
\[ \boxed{ \theta^* = \arg\min_{\theta} L(\theta) } \]where \(L(\theta)\) is a loss or error function.
The loss function defines what “best fit” means.
The Choice of Loss Function Matters
Minimizing squared error,
\[ \sum e_i^2, \]is not the same as minimizing absolute error,
\[ \sum |e_i|. \]Different loss functions can produce different calibrated parameters.
The loss should reflect the purpose of the model and the consequences of different kinds of error.
Weighted Calibration
Some observations may deserve more influence than others.
A weighted squared-error objective is
\[ \sum_{i=1}^{n}w_i(y_i-\hat y_i)^2. \]Larger \(w_i\) gives observation \(i\) greater importance.
Weights may reflect measurement precision, business importance, sample size, or other justified considerations.
Measurement Error
Calibration data are not always exact.
Instruments have limited precision, surveys contain reporting error, and operational data can contain missing or miscoded values.
A model calibrated to noisy observations should not be expected to reproduce every measured value exactly.
Data Quality Matters
Calibration cannot rescue fundamentally poor evidence.
Important questions include:
- Were the quantities measured correctly?
- Are units consistent?
- Are important values missing?
- Are there duplicated observations?
- Does the sample represent the intended domain?
- Did measurement procedures change?
Validation Across the Intended Domain
A model should be tested where it is intended to be used.
Suppose a model is calibrated using temperatures from \(10^\circ\)C to \(25^\circ\)C.
Validation only within that same narrow range does not establish reliability at \(60^\circ\)C.
Domain matters during validation.
Extrapolation Requires Extra Caution
A model may interpolate well within observed conditions and extrapolate poorly beyond them.
For example, a polynomial fitted on
\[ 0\leq x\leq 10 \]may behave wildly for
\[ x=100. \]Validation evidence should not be silently extended far beyond its support.
Model Validation Is Purpose-Dependent
There is no universal threshold that makes every model “valid.”
A weather model, a classroom demonstration, a bridge design model, and a retail sales forecast have different acceptable error levels and different consequences of failure.
Validation should be judged against the model's stated objective.
Example: Calibrating a Linear Cost Model
Suppose total cost is modeled as
\[ C(q)=F+cq. \]Measurements show
\[ C(100)=1400 \]and
\[ C(300)=2600. \]Subtracting,
\[ 1200=200c, \]so
\[ c=6. \]Then
\[ 1400=F+6(100), \]giving
\[ F=800. \]Thus,
\[ \boxed{C(q)=800+6q}. \]Validating the Cost Model
Suppose a later observation at \(q=200\) gives actual cost
\[ C_{\text{obs}}=2050. \]The model predicts
\[ C(200)=800+6(200)=2000. \]The residual is
\[ e=2050-2000=50. \]Whether a \(\$50\) error is acceptable depends on the intended use of the model.
Example: Growth Model
Suppose
\[ P(t)=P_0e^{rt} \]is calibrated using early population data.
If later observations consistently fall below the predicted trajectory, this may indicate:
- the growth rate changed;
- resource limitations became important;
- the exponential mechanism is inadequate;
- the calibration interval was not representative.
Model Revision
Failed validation is not the end of modeling.
It provides information.
A model may be revised by:
- changing parameter values;
- changing assumptions;
- adding or removing variables;
- changing functional form;
- changing the time structure;
- restricting the intended domain;
- adding stochastic variation;
- collecting better data.
Recalibration versus Structural Revision
If a model fails validation, there are at least two broad possibilities.
The model structure may be reasonable but its parameters may need updating. This suggests recalibration.
Alternatively, the equations or mechanisms may be inadequate. This requires structural revision.
Changing parameters cannot repair every structural error.
Common Error: Validating on Calibration Data
A model that is evaluated only on the observations used to fit it has not received an independent validation test.
Calibration error measures fit.
Validation error measures generalization to evidence not used in the same fitting process.
Common Error: Equating Low Error with Truth
A low numerical error does not prove that a model is the true description of reality.
Several different models can fit the same observations.
Validation provides evidence of adequacy for a purpose, not mathematical proof that the model is universally correct.
Common Error: Ignoring the Baseline
A model may appear accurate until it is compared with a simple alternative.
If a sophisticated forecasting model has RMSE \(=10\) but predicting the previous value has RMSE \(=8\), the sophisticated model has not improved predictive performance.
Validation should include meaningful benchmarks.
Common Error: Tuning Repeatedly to the Validation Set
If a model is repeatedly changed until it performs well on a validation set, that set gradually becomes part of the calibration process.
A separate final test set or new evidence may then be needed for a more independent evaluation.
Common Error: Ignoring Scale
An error of \(10\) units can mean very different things depending on the modeled quantity.
Always interpret error relative to:
- the scale of the quantity;
- measurement precision;
- decision thresholds;
- operational consequences.
Common Error: Using Data Outside the Model's Scope
A model calibrated for one population, region, time period, or operating regime may not automatically transfer to another.
Validation should reflect the domain in which the model will actually be used.
A Reliable Calibration Procedure
- State the model structure.
- Identify the parameters that must be determined.
- Specify the calibration data or known conditions.
- Check units and data quality.
- Choose a fitting criterion or estimation method.
- Estimate the parameters.
- Compute fitted values and residuals.
- Inspect residual size and patterns.
- Check whether the parameter values are plausible.
- Record the calibration domain and assumptions.
A Reliable Validation Procedure
- State the intended use of the model.
- Select evidence not used in the same calibration step when possible.
- Generate predictions without retuning the model.
- Compute appropriate error measures.
- Inspect residual patterns and systematic bias.
- Compare against meaningful benchmarks.
- Check physical, logical, and dimensional behavior.
- Evaluate performance across the intended domain.
- Identify conditions under which the model fails.
- Decide whether to accept, restrict, recalibrate, or revise the model.
Practice 1: Calibration or Validation?
A modeler uses ten observations to estimate the slope and intercept of a linear model.
Is this calibration or validation?
Practice 2: Residual
An observed value is
\[ y=18 \]and the model predicts
\[ \hat y=16.5. \]Find the residual.
Practice 3: Linear Calibration
A linear model passes through
\[ (1,5) \]and
\[ (4,14). \]Find the calibrated model.
Practice 4: MAE
Three prediction errors are
\[ -2,\qquad 3,\qquad -1. \]Find the mean absolute error.
Practice 5: MSE
Using the same errors
\[ -2,\qquad 3,\qquad -1, \]find the mean squared error.
Practice 6: Overfitting
Model A has calibration error \(1.2\) and validation error \(12.0\). Model B has calibration error \(3.5\) and validation error \(4.1\).
Which model shows stronger evidence of overfitting?
Practice 7: Validation Domain
A model was calibrated and validated only for
\[ 0\leq x\leq 20. \]What should be said about a prediction at \(x=200\)?
Practice 8: Benchmark
A new forecasting model has RMSE \(=9\). A simple historical-average model has RMSE \(=7\) on the same validation data.
Which performs better under this metric?
Practice 9: Recalibrate or Revise?
A model's residuals show a strong curved pattern across the entire validation domain.
Does this suggest only a parameter update, or might the model structure need revision?
Practice 10: Validation Purpose
Two models have similar average prediction error, but one occasionally produces extremely large errors that would create serious operational losses.
Why might average error alone be an inadequate validation criterion?
Practice Answers
Practice 1
This is calibration because the observations are being used to estimate model parameters.
Practice 2
\[ e=y-\hat y. \] \[ e=18-16.5. \] \[ \boxed{e=1.5}. \]Practice 3
The slope is
\[ m=\frac{14-5}{4-1}=3. \]Using \((1,5)\),
\[ 5=3(1)+b, \]so
\[ b=2. \]Therefore,
\[ \boxed{y=3x+2}. \]Practice 4
\[ \operatorname{MAE} = \frac{|-2|+|3|+|-1|}{3} = \frac{6}{3}. \] \[ \boxed{\operatorname{MAE}=2}. \]Practice 5
\[ \operatorname{MSE} = \frac{(-2)^2+3^2+(-1)^2}{3} = \frac{4+9+1}{3}. \] \[ \boxed{\operatorname{MSE}=\frac{14}{3}\approx 4.67}. \]Practice 6
Model A shows stronger evidence of overfitting because its calibration error is very low while its validation error is much larger.
Practice 7
The prediction is a major extrapolation beyond the validated domain. Reliability at \(x=200\) has not been established by the available validation evidence.
Practice 8
The historical-average model performs better under RMSE because
\[ 7<9. \]Practice 9
A systematic curved residual pattern suggests that the functional form may be missing important structure. Structural revision should be considered, not merely recalibration.
Practice 10
Average error can hide rare but severe failures. If extreme errors carry major consequences, validation should also examine tail behavior, maximum error, threshold violations, or other risk-sensitive measures.
What to Remember
- Calibration determines parameter values from evidence or known conditions.
- Validation tests whether the calibrated model is adequate for its intended use.
- Residuals are observed values minus predicted values.
- MAE, MSE, and RMSE summarize prediction error in different ways.
- A close calibration fit does not guarantee good validation performance.
- Overfitting occurs when a model captures calibration-specific noise or accidental structure.
- Validation should use evidence not employed in the same fitting process when possible.
- Benchmarks help determine whether added complexity produces real improvement.
- Validation includes physical, logical, dimensional, and domain checks as well as numerical error measures.
- Failed validation can lead to recalibration, structural revision, domain restriction, or better data collection.
- A model is validated relative to a purpose and domain, not declared universally true.
The central habit is: never confuse a model's ability to fit the past with evidence that it will work where you intend to use it.
Where We Go Next
Calibration gives parameter estimates, and validation measures how well the resulting model performs. But those parameter values and predictions are rarely known with perfect certainty.
The next lesson, Sensitivity and Uncertainty, examines how changes in assumptions, parameters, inputs, and random variation affect model outputs and conclusions.