Noise and Measurement Error

The Central Idea

Suppose a forward model predicts

The actual observation is often better represented as

where

represents measurement error or observational noise.

Measurement Error Is Part of the Model

A complete inverse model has at least two components:

1. a forward model describing the underlying system;
2. an observation model describing how measurements differ from the underlying system.

Conceptually,

Ignoring the second stage can produce misleading parameter estimates and uncertainty statements.

Random Error

Random error varies unpredictably from one measurement to another.

Repeated measurements of the same quantity may produce values scattered around a central value.

A common model is

Random error contributes uncertainty but does not necessarily introduce systematic bias.

Systematic Error

Systematic error shifts measurements consistently in one direction or according to a repeatable pattern.

Examples include:

• incorrect calibration;
• sensor offset;
• scale-factor error;
• persistent environmental interference;
• misaligned timing;
• modeling the wrong physical quantity.

Bias

The bias of a measurement error is

If

the measurement error is unbiased in expectation.

If

the observation process contains systematic bias.

Variance

The variance measures the spread of random measurement error:

Large variance means repeated measurements are widely scattered.

Small variance means the measurements are relatively precise.

Standard Deviation

The standard deviation is

Because \(\sigma\) has the same units as the measured quantity, it is often easier to interpret than variance.

Accuracy Versus Precision

Accuracy concerns closeness to the true value.

Precision concerns repeatability or spread.

Measurements can be:

• accurate and precise;
• accurate on average but imprecise;
• precise but biased;
• both inaccurate and imprecise.

A Precise but Biased Sensor

Suppose a true value is

A sensor repeatedly reports values near

The readings may have very small variance while still being systematically wrong.

Precision alone does not guarantee accuracy.

Gaussian Noise

A common model for additive measurement error is

Then

This model is appropriate when many small independent effects combine to produce approximately symmetric noise.

Gaussian Likelihood

If the forward prediction is

and

then the negative log-likelihood contains

Measurement variance therefore determines the natural weighting of residuals.

Homoscedastic Noise

Noise is homoscedastic when all observations share the same variance:

Under independent Gaussian errors, ordinary least squares is then a natural estimator.

Heteroscedastic Noise

Noise is heteroscedastic when the measurement variance differs across observations:

Variance may depend on:

• signal magnitude;
• time;
• sensor type;
• experimental condition;
• distance or spatial location;
• count intensity.

Inverse-Variance Weighting

If observation \(i\) has known variance

a common residual weight is

The weighted least-squares objective becomes

Why More Precise Measurements Receive More Weight

If one measurement has small uncertainty, a given residual is more surprising than the same residual from a noisy measurement.

Inverse-variance weighting formalizes this idea.

A residual of \(1\) unit may be severe if

but modest if

Covariance

Errors from different observations may vary together.

The covariance between errors \(i\) and \(j\) is

Nonzero covariance means the errors are statistically related.

The Covariance Matrix

For a noise vector

the covariance matrix is

Diagonal and Off-Diagonal Terms

The diagonal entries

are error variances.

The off-diagonal entries

are error covariances.

A diagonal covariance matrix represents uncorrelated errors.

Correlated Measurement Error

Correlation can arise when several measurements share:

• the same instrument;
• the same calibration constant;
• a common environmental disturbance;
• the same preprocessing pipeline;
• overlapping spatial or temporal windows.

Treating correlated measurements as independent can overstate the amount of information in the data.

Generalized Least Squares

If

then a natural quadratic residual measure is

This accounts for both unequal variances and correlations.

Whitening

If

define transformed residuals

Then

The transformation converts correlated, unequal-variance residuals into standardized residual coordinates.

Correlation

Covariance depends on measurement scale.

The correlation coefficient standardizes covariance:

It lies between

and

Temporal Correlation

Time-series measurements often have residuals that are correlated across nearby times.

For example,

may decrease gradually as the lag \(\Delta\) increases.

Ignoring this structure can lead to overly optimistic uncertainty estimates.

Spatial Correlation

Measurements taken at nearby locations may share environmental or physical disturbances.

A covariance model may therefore depend on distance:

Spatially correlated error is common in imaging, geophysics, climate data, and remote sensing.

Calibration Error

A sensor may follow

Here:

• \(a\) is an offset error;
• \(b\) is a scale-factor error;
• \(\varepsilon\) is random noise.

Calibration parameters may need to be estimated alongside the physical parameters.

Additive Bias

An additive sensor bias can be modeled as

If \(b\) is ignored, the inverse method may incorrectly force physical model parameters to compensate for the sensor offset.

Multiplicative Error

Some instruments exhibit proportional error:

The absolute error then grows naturally with signal magnitude.

Relative-error or logarithmic observation models may be more appropriate than constant-variance additive noise.

Lognormal Error

For positive quantities with multiplicative variability, one may model

where

Then the original measurement follows a lognormal-type distribution.

Count Noise

Count measurements may naturally follow a Poisson model:

Then

and

The variance is therefore tied to the expected signal.

Quantization Error

Digital instruments record values on a finite grid.

If the measurement resolution is

then continuous values are rounded or truncated to nearby representable values.

This introduces quantization error even when the underlying sensor is otherwise noise-free.

Finite Resolution

Some instruments do not measure a point value but average over a spatial, temporal, or spectral window.

The observation may therefore be better modeled as

Instrument resolution is then part of the observation operator, not merely an additive noise term.

Timing Error

Suppose a measurement is intended at time

but actually occurs at

Then the observation may be

In rapidly changing systems, timing uncertainty can produce substantial apparent measurement error.

Error in the Independent Variable

Standard least squares often assumes predictors are known exactly.

If both \(x\) and \(y\) contain measurement error, the problem changes.

For example,

Ignoring predictor error can bias fitted parameters.

Errors-in-Variables Models

When explanatory variables are noisy, one may model their latent true values explicitly.

This leads to errors-in-variables formulations or total least-squares methods in suitable linear settings.

Outliers

An outlier is an observation unusually far from the pattern predicted by the model.

Possible causes include:

• instrument failure;
• data-entry error;
• unmodeled physical events;
• rare but genuine stochastic variation;
• incorrect synchronization.

Outliers should be investigated rather than automatically deleted.

Heavy-Tailed Noise

Gaussian models assign rapidly decreasing probability to extreme residuals.

If large deviations occur more often than Gaussian theory predicts, a heavy-tailed distribution may be more appropriate.

Heavy-tailed likelihoods reduce the influence of extreme observations.

Robust Estimation

Robust methods reduce sensitivity to a small number of extreme residuals.

Instead of penalizing every residual quadratically, a robust loss may grow more slowly when

becomes large.

This can improve stability when contamination or outliers are plausible.

Detection Limits

Instruments may not resolve values below a threshold.

A reported measurement may therefore mean only

Censoring

If the instrument reports only that

Missing Measurements

Missing data should not automatically be treated as zero or replaced by an arbitrary value.

The correct treatment depends on why the data are missing and whether the missingness mechanism is related to the underlying system.

Preprocessing Error

Data often undergo processing before estimation:

• filtering;
• smoothing;
• baseline subtraction;
• normalization;
• interpolation;
• registration;
• unit conversion.

These operations can alter both the mean structure and covariance of the measurement errors.

Smoothing Creates Correlation

Suppose raw measurement errors are independent.

If neighboring measurements are averaged, the processed values share some of the same raw observations.

The resulting errors become correlated even if the original noise was not.

Replication

Repeating a measurement under nominally identical conditions helps estimate random variability.

If measurements are

the sample mean is

Replicates provide direct information about repeatability.

Variance of an Average

If independent replicated measurements have common variance

then

Therefore, averaging independent replicates reduces random uncertainty.

Replication Does Not Remove Bias

If every measurement contains the same additive bias

then averaging many measurements does not eliminate it.

Random noise decreases through averaging, but systematic error remains.

Error Propagation Through a Function

Suppose

and \(x\) has small uncertainty.

Linearization gives

Therefore,

Multivariable Error Propagation

Let

If \(\mathbf{x}\) has covariance matrix

then local linearization gives

where \(J_g\) is the Jacobian of \(g\).

Inverse Error Propagation

In inverse problems, the mapping from data to parameters may be highly sensitive.

If

then approximately

Large entries in \(J_G\) indicate strong amplification of measurement uncertainty.

Linear Inverse Error Propagation

If

then

This formula makes the effect of measurement covariance explicit.

Noise Amplification and Ill-Posedness

If the inverse operator \(A\) has very large gain in some directions, then modest measurement covariance can produce enormous parameter covariance.

This is the uncertainty form of ill-posedness.

Standardized Residuals

If a measurement has standard deviation

a standardized residual is

This measures model-data mismatch in units of expected measurement uncertainty.

Interpreting Standardized Residuals

A raw residual of \(2\) units cannot be judged without knowing the noise scale.

If

then \(2\) units is extremely large relative to the expected noise.

If

the same raw residual is modest.

Residual Diagnostics

After fitting, residuals should be inspected for:

• nonzero mean;
• changing variance;
• time correlation;
• spatial correlation;
• outliers;
• systematic trends;
• distributional mismatch.

Residual structure often reveals an inadequate observation model.

Residual Mean

If residuals have a persistent nonzero average,

this may indicate:

• sensor bias;
• incorrect model offset;
• systematic model misspecification.

Residual Variance Patterns

If residual spread increases with predicted magnitude, constant-variance Gaussian noise may be inappropriate.

A heteroscedastic or multiplicative error model may better represent the data-generating process.

Autocorrelation Diagnostics

For time-indexed residuals, examine whether

is correlated with

Persistent autocorrelation can indicate unmodeled dynamics or correlated measurement noise.

Noise Model Misspecification

If the assumed noise model is wrong, several consequences can follow:

• inefficient parameter estimates;
• biased estimates in some settings;
• incorrect uncertainty intervals;
• misleading likelihood values;
• poor experimental design decisions.

Estimating Noise Parameters

Noise parameters such as

correlation length, or dispersion may be unknown.

They can sometimes be estimated jointly with the physical parameters.

This creates a larger inverse problem in which both signal and uncertainty structure are inferred.

Separating Process Noise and Measurement Noise

In dynamic systems, uncertainty may enter both the system evolution and the measurement process.

A state model may include

while observations satisfy

Here \(\mathbf{w}_k\) is process noise and \(\mathbf{v}_k\) is measurement noise.

Why the Distinction Matters

Measurement noise describes imperfect observation of a system state.

Process noise describes genuine uncertainty in how the system evolves.

Treating one as the other can distort state and parameter estimates.

Instrument Specifications

Real instruments may report uncertainty through:

• resolution;
• repeatability;
• accuracy;
• linearity;
• drift;
• temperature dependence;
• calibration tolerance.

These specifications should be translated carefully into the mathematical observation model.

Error Budgets

An error budget separates different uncertainty sources.

For example,

when independent components can reasonably be added in variance.

Error budgets help identify which source dominates total uncertainty.

Repeated Calibration

Calibration uncertainty may change over time because instruments drift.

Repeated calibration allows analysts to distinguish:

• short-term random noise;
• long-term drift;
• persistent bias;
• changes in sensitivity.

Data Quality Versus Data Quantity

More measurements are not always equivalent to more information.

A large number of highly correlated, biased, or low-resolution measurements may be less useful than a smaller number of precise independent measurements.

Experimental Design for Noise Reduction

Measurement uncertainty can often be reduced through:

• replication;
• longer integration time;
• better calibration;
• improved sensor placement;
• more informative sampling times;
• independent measurement modalities.

Noise and Regularization

Regularization strength should depend partly on measurement uncertainty.

If noise is large, fitting every detail of the observations is undesirable.

If noise is very small, stronger data fidelity may be justified.

Discrepancy Principle

If the expected noise magnitude is known, one may choose regularization so that

is comparable to the expected observational error.

Residuals far below the noise level may indicate overfitting.

Noise and Bayesian Estimation

In Bayesian inference, the noise model enters the likelihood:

Larger assumed noise generally produces a broader likelihood and weaker data-driven posterior concentration.

Smaller assumed noise gives the data greater influence.

Underestimating Noise

If the analysis assumes

then residuals appear more informative than they really are.

Parameter uncertainty may be reported too narrowly.

Overestimating Noise

If

the analysis may underuse the information in the observations.

Parameter uncertainty can become unnecessarily broad.

Common Error: Assuming All Error Is Random

Repeated averaging can reduce random noise but cannot remove a persistent calibration bias.

Systematic and random errors should be modeled separately when possible.

Common Error: Assuming Independence

If observations share a sensor, preprocessing step, or environmental disturbance, their errors may be correlated.

Treating them as independent can overstate effective sample size.

Common Error: Using Equal Weights Automatically

Equal weighting is appropriate only when observations have comparable error scales and the objective is defined accordingly.

Large differences in precision should generally influence residual weighting.

Common Error: Deleting Outliers Without Investigation

An extreme observation may be erroneous, but it may also reflect genuine system behavior that the model fails to explain.

Outliers should be investigated before removal.

Common Error: Ignoring Error in Predictors

Standard regression assumptions may fail when explanatory variables are also measured with uncertainty.

Predictor error can bias slope and parameter estimates.

Common Error: Treating Instrument Resolution as Pure Noise

Finite resolution may represent averaging or blurring by the instrument.

That effect belongs in the forward observation operator, not only in an additive error term.

Common Error: Ignoring Preprocessing Effects

Smoothing, interpolation, and normalization can change error variance and induce correlation.

The statistical model should reflect the data actually used in estimation, not merely the raw instrument output.

A Reliable Measurement-Error Workflow

1. Identify the physical quantity actually observed.
2. Separate forward-model output from observation-model behavior.
3. List random and systematic error sources.
4. Estimate or justify measurement variance.
5. Check whether variance changes with condition or signal level.
6. Assess temporal, spatial, or instrument-induced correlation.
7. Model calibration and predictor uncertainty when relevant.
8. Translate uncertainty into residual weights or a likelihood.
9. Inspect standardized residuals after fitting.
10. Test for bias, heteroscedasticity, and correlation.
11. Propagate measurement uncertainty into parameter uncertainty.
12. Use replication or calibration experiments where possible.
13. Document preprocessing and uncertainty assumptions.

Practice 1: Bias

If a measurement error satisfies

is the measurement process unbiased?

Practice 2: Variance and Standard Deviation

If

what is the standard deviation?

Practice 3: Inverse-Variance Weighting

Two observations have standard deviations

and

What are their inverse-variance weights?

Practice 4: Replication

Independent repeated measurements each have variance

If \(4\) such measurements are averaged, what is the variance of the average?

Practice 5: Systematic Error

A sensor always reads \(0.7\) units too high.

What type of measurement error is this?

Practice 6: Correlation

Why can smoothing independent raw measurements create correlated processed errors?

Practice 7: Standardized Residual

Suppose

Compute the standardized residual.

Practice 8: Error Propagation

Suppose

and

Approximate \(\operatorname{Var}(z)\).

Practice 9: Multiplicative Noise

If the absolute measurement spread grows in proportion to the signal level, which may be more appropriate: constant-variance additive noise or a multiplicative error model?

Practice 10: Bias and Averaging

Will averaging many repeated measurements eliminate a fixed calibration offset?

Practice Answers

Practice 1

No. The measurement process is biased because

Practice 2

Practice 3

The inverse-variance weights are

and

Practice 4

For \(m=4\) independent measurements,

Practice 5

This is a systematic additive bias.

Practice 6

Neighboring processed values can share some of the same raw observations. Because the same random errors contribute to several processed points, their resulting errors become statistically correlated.

Practice 7

Practice 8

Since

local variance propagation gives

Practice 9

A multiplicative error model is generally more appropriate because the noise scale grows with the signal.

Practice 10

No. Averaging reduces random variance but does not remove a fixed systematic calibration offset.

What to Remember

Measurement error determines how strongly observations should influence an inverse estimate and how much uncertainty remains afterward.

• Random error contributes spread; systematic error contributes bias.
• Variance and standard deviation quantify random measurement uncertainty.
• Heteroscedasticity means uncertainty changes across observations.
• Inverse-variance weighting gives more precise measurements greater influence.
• Covariance describes dependence among measurement errors.
• Correlated observations contain less independent information than their count alone suggests.
• Calibration, timing, resolution, predictor error, and preprocessing can all alter the observation model.
• Replication reduces random error but not systematic bias.
• Error propagation connects measurement uncertainty to parameter uncertainty.
• Standardized residuals measure mismatch relative to expected noise.
• Noise assumptions directly affect likelihoods, confidence intervals, and Bayesian posteriors.
• Measurement uncertainty should be modeled, not treated as an afterthought.

The central principle is:

Where We Go Next

Once measurement uncertainty is represented explicitly, the next question is how to decide among competing model structures that may explain the same observations.

The next lesson, Model Selection, develops methods for comparing candidate models while balancing fit, complexity, predictive performance, and uncertainty.