Sources and Propagation of Error

The Central Idea

Suppose the exact quantity is \(x\), but we use an approximation \(\tilde{x}\).

The error is

Numerical analysis asks several questions:

• How large is \(e\)?
• What caused it?
• How does it affect later calculations?
• Can the final result still be trusted?
• Can the error be bounded or reduced?

Error Does Not Mean Mistake

In numerical analysis, error usually means the difference between an exact mathematical quantity and an approximation.

It does not necessarily mean that someone made a mistake.

For example,

contains approximation error even though the approximation may be intentional and useful.

Exact Value, Approximate Value, and Error

Let

and suppose we use

Then the signed error is

Because \(\tilde{x}

Absolute Error

The absolute error ignores the sign and measures the size of the discrepancy:

Absolute error uses the same units as the quantity itself.

If a measured length is off by \(0.02\) meter, then the absolute error is

Relative Error

Absolute error does not tell us whether the error is large compared with the quantity being measured.

The relative error is

provided \(x\neq0\).

Relative error is dimensionless and gives the error in proportion to the size of the exact quantity.

Percentage Error

Relative error is often expressed as a percentage:

For example, a relative error of

corresponds to

Example: Absolute and Relative Error

Suppose the exact value is

and the approximation is

The absolute error is

The relative error is

Therefore, the percentage error is

Why Relative Error Often Matters More

An absolute error of \(1\) can have very different meanings.

If the exact quantity is

then an error of \(1\) is relatively tiny.

If the exact quantity is

then an error of \(1\) is enormous.

Relative error places the discrepancy in scale.

The Major Sources of Numerical Error

Several recurring sources of error appear throughout numerical work.

• measurement and data error;
• modeling error;
• representation and rounding error;
• truncation error;
• iteration error;
• algorithmic amplification of existing error.

A reliable analysis separates these sources whenever possible.

Measurement Error

Physical data are often measured rather than known exactly.

A measured length reported as

may actually represent an interval of plausible values rather than one exact length.

Numerical methods cannot recover information that was never present in the original data.

Modeling Error

A mathematical model deliberately simplifies reality.

For example, a projectile model may neglect:

• air resistance;
• wind;
• rotation;
• variation in gravitational acceleration.

Even if the equations are solved exactly, the model can still differ from the physical system.

Representation Error

A finite-precision computer cannot represent every real number exactly.

If \(x\) is replaced by its floating-point representation \(\operatorname{fl}(x)\), then

may introduce representation error before any arithmetic begins.

The previous lesson developed this source in detail.

Rounding Error

Rounding error occurs when a value is replaced by a nearby representable value.

If a system keeps only four significant decimal digits, then

may be stored as

Every later calculation begins with that rounded approximation.

Truncation Error

Truncation error appears when an infinite mathematical process is replaced by a finite one.

For example,

is an infinite series.

If we stop after the quadratic term,

the omitted remainder creates truncation error.

Iteration Error

Iterative methods approach a solution through a sequence

rather than reaching the answer in one exact step.

If the process is stopped at \(x_n\), the remaining difference from the desired limiting solution is an iteration error.

Error Can Be Present Before the Algorithm Starts

A common mistake is to blame every discrepancy on the numerical algorithm.

But the input may already contain:

• measurement uncertainty;
• rounded constants;
• estimated parameters;
• simplified model assumptions.

The algorithm inherits these uncertainties.

Propagation of Error

Error propagation describes how uncertainty or approximation in inputs affects the result of later calculations.

Suppose

and \(x\) is replaced by \(x+\Delta x\).

Then the output changes by

Numerical analysis studies the relationship between \(\Delta x\) and \(\Delta y\).

Linear Approximation of Propagated Error

For sufficiently small \(\Delta x\), differentiability gives

Therefore,

The derivative measures how strongly a small input error is transmitted into the output.

Absolute Error Propagation Through a Function

If the input uncertainty satisfies

then, for small perturbations,

Thus a useful local estimate is

Example: Squaring an Approximate Number

Let

Then

A small input error \(\Delta x\) therefore produces approximately

At \(x=100\), even a small absolute error in \(x\) is magnified by a factor of about \(200\) in the absolute output error.

Relative Error Propagation Through Powers

Suppose

Differentiating gives

Divide by \(y=x^n\):

Thus, for small errors,

The relative error is approximately multiplied by \(|n|\).

Error in Sums and Differences

Suppose

If the approximations carry errors \(e_x\) and \(e_y\), then

Therefore,

In absolute value,

Error Bounds for Addition

If

and

then

This is a worst-case bound. The actual errors may partially cancel.

Error in Subtraction

For

the propagated signed error is

Therefore,

Absolute error is still controlled by the sum of the input error magnitudes.

Why Subtraction Can Be More Dangerous Than Addition

Subtraction becomes numerically delicate when two nearly equal numbers are subtracted.

If

then

may be very small.

Even modest absolute input errors can then become very large relative to the small result.

Catastrophic Cancellation

Catastrophic cancellation occurs when subtraction of nearly equal quantities destroys many significant digits.

Suppose the exact values are

and

Their exact difference is

If both values are rounded too coarsely before subtraction, that tiny difference can be severely distorted or lost entirely.

Error in Products

Suppose

and the inputs become \(x+\Delta x\) and \(y+\Delta y\).

Then

Thus,

First-Order Product Error

When \(\Delta x\) and \(\Delta y\) are small, the product

is second-order and often much smaller than the first-order terms.

Therefore,

Dividing by \(z=xy\) gives

Relative Errors in Products Approximately Add

The first-order product formula explains a widely useful rule:

This approximation is especially useful when the input errors are small.

Error in Quotients

Suppose

A first-order differential calculation gives

For error magnitudes, a useful bound is

Division by a Small Quantity

Division becomes sensitive when the denominator is close to zero.

For

small perturbations in \(y\) can have a large effect when \(|y|\) is small.

This is one reason near-zero denominators require special attention in numerical algorithms.

Functions of Several Variables

Suppose

For small perturbations,

Each partial derivative measures sensitivity to one input while the others are held fixed.

A Multivariable Error Bound

Taking absolute values gives the first-order estimate

This gives a practical way to combine bounded input uncertainties.

Example: Area from Measured Dimensions

Let

Then

Dividing by \(A=LW\),

Thus the relative error in area is approximately the sum of the relative errors in length and width.

Example: Volume of a Sphere

The volume is

Differentiating,

Divide by \(V\):

Therefore,

Forward Error

Suppose the exact problem has exact solution \(y\), while an algorithm returns \(\tilde{y}\).

The forward error measures the difference between the computed and exact outputs:

Forward error asks:

How close is the computed answer to the true answer?

Backward Error

Backward error asks a different question.

Instead of comparing outputs directly, we ask:

For what nearby input would the computed result be exact?

If \(\tilde{y}\) is exactly the solution to a slightly perturbed problem, the algorithm may be considered backward accurate even when the exact output is difficult to obtain directly.

Why Backward Error Is Useful

Suppose we want to solve

An algorithm returns \(\tilde{x}\).

Instead of asking only how far \(\tilde{x}\) is from the exact solution, we may ask how much \(b\) would need to change so that

holds exactly.

A small \(\Delta b\) indicates a small backward error.

Residual

For an approximate solution \(\tilde{x}\) to

the residual is

A small residual means the approximate solution nearly satisfies the equation.

However, a small residual does not automatically guarantee a small forward error.

Small Residual Does Not Always Mean Small Solution Error

Suppose \(f\) is nearly flat near the solution.

Then a large change in \(x\) may produce only a small change in \(f(x)\).

In such a case,

can be small even though \(\tilde{x}\) is not especially close to the exact root.

This leads directly toward the study of conditioning.

Conditioning and Error Propagation

Some problems naturally amplify input error more than others.

This sensitivity belongs to the problem, not necessarily to the algorithm.

A well-conditioned problem changes only modestly when the input changes slightly.

An ill-conditioned problem can change dramatically under tiny perturbations.

A Local Sensitivity Measure

For

the derivative

measures local absolute sensitivity.

A large value of

means small absolute input errors can create larger absolute output changes.

Relative Condition Measure

For a scalar differentiable function with \(x\neq0\) and \(f(x)\neq0\), a local relative sensitivity measure is

It connects relative input error to relative output error approximately:

Example: Conditioning of \(f(x)=x^2\)

Let

Then

Therefore,

Small relative input errors are approximately doubled in the output.

Algorithmic Stability

Even when a problem is well-conditioned, a poor algorithm may magnify rounding or intermediate errors.

Stability describes how an algorithm manages unavoidable perturbations during computation.

A stable algorithm generally does not introduce disproportionate additional error beyond the sensitivity already present in the problem.

Problem Sensitivity and Algorithm Stability Are Different

Two distinct questions must be kept separate.

1. Conditioning: Is the mathematical problem sensitive to small input changes?
2. Stability: Does the numerical method control the errors created during computation?

The next lesson will develop this distinction systematically.

Error Accumulation Across Many Steps

A long computation may perform thousands or millions of operations.

Each operation may introduce a small rounding error.

The final error depends on how these local errors interact:

• some may cancel;
• some may reinforce one another;
• some may be damped;
• some may be amplified by the algorithm.

Worst-Case Error Bounds

If several independent contributions satisfy

then the triangle inequality gives

This produces a safe worst-case bound, although the actual error may be much smaller.

Why Worst-Case Bounds Can Be Conservative

Worst-case analysis assumes the errors align in the most damaging possible direction.

In real computations, signs may vary and partial cancellation may occur.

Therefore,

is often larger than the observed total error.

A conservative bound is still valuable when reliability must be guaranteed.

Significant Digits and Error

If an approximation is accurate to several significant digits, its relative error must be correspondingly small.

Numerical analysis often uses relative error to assess how many leading digits can be trusted.

The exact relationship depends on the rounding convention and the position of the leading digits, but the principle is simple:

smaller relative error means more reliable significant digits.

Errors in Constants

Constants may also be rounded.

For example, using

rather than a higher-precision approximation introduces a known input error.

In repeated or sensitive calculations, even a small constant error can propagate substantially.

Errors in Coefficients

Consider a polynomial

If the coefficients \(a_i\) are measured, estimated, or rounded, the polynomial itself is perturbed.

Numerical analysis must sometimes distinguish between:

• error in the input variable \(x\);
• error in the coefficients \(a_i\);
• error introduced while evaluating the polynomial.

Errors in Repeated Addition

Repeatedly summing many floating-point values can accumulate rounding error.

The order of summation may matter because floating-point addition is not exactly associative.

Thus,

and

may produce slightly different stored results.

Summing Numbers of Very Different Magnitudes

If a very small number is added to a very large number, the smaller contribution may be below the available floating-point spacing.

The small addend can then disappear after rounding.

This is why numerical summation strategies often pay attention to magnitude and ordering.

Error Growth in Recurrence Relations

A recurrence computes each new value from earlier approximations:

If \(x_n\) contains error, then \(x_{n+1}\) begins with that inherited error before new rounding occurs.

Depending on \(F\), errors may shrink, remain comparable, or grow rapidly.

A Simple Recurrence Error Model

Suppose

If the current error is \(e_n\), then the inherited part of the next error is approximately

Therefore:

• if \(|a|<1\), inherited error tends to shrink;
• if \(|a|=1\), it tends to persist;
• if \(|a|>1\), it tends to grow.

Local Error and Global Error

In step-by-step numerical methods, two ideas often appear.

Local error measures the error introduced in one step, often assuming the step begins from exact data.

Global error measures the accumulated discrepancy after many steps.

A method can have tiny local errors that still accumulate into a meaningful global error.

Truncation Error in Taylor Approximation

Suppose

If we retain only the first \(n\) terms, the omitted remainder is the truncation error.

Taylor's theorem provides a structured way to bound that remainder under suitable smoothness assumptions.

Example: Approximating \(e^{0.1}\)

Using

a second-order approximation at \(x=0.1\) is

Therefore,

The difference between this and the exact exponential value includes truncation error.

Rounding Error and Truncation Error Can Compete

Numerical methods often involve a tradeoff.

Taking more steps or retaining more terms can reduce truncation error.

But additional arithmetic may introduce more rounding error.

A method must therefore balance approximation quality against finite-precision effects.

Stopping Too Early

If an iterative method is terminated prematurely, the iteration error may still be large.

For example, stopping after only a few root-finding iterations may produce a residual that is not yet within the required tolerance.

A good stopping rule links computational effort to a measurable error target.

Stopping Too Late

More iterations are not always automatically better.

Once the method reaches the practical precision limit of the computation, additional steps may:

• fail to improve the result;
• waste computation;
• amplify accumulated rounding;
• trigger numerical instability.

Error Bounds

An error bound does not necessarily give the exact error.

Instead, it guarantees that the error is no larger than some quantity:

Error bounds are central to trustworthy numerical computation because the exact answer is often unknown.

A Priori Error Estimates

An a priori estimate is obtained before the full numerical computation is completed.

It may depend on:

• the method;
• step size;
• derivative bounds;
• problem parameters;
• known convergence rates.

A priori analysis helps choose a method and computational resolution.

A Posteriori Error Estimates

An a posteriori estimate uses information obtained during or after the computation.

Examples include:

• residuals;
• differences between successive iterates;
• comparison of coarse and refined approximations;
• embedded error estimates.

These estimates support adaptive numerical methods.

Absolute Tolerance

An absolute stopping condition may require

This is appropriate when a fixed error scale has a direct practical meaning.

For example, a geometric computation might require accuracy within

Relative Tolerance

A relative criterion may require

This is useful when the acceptable error should scale with the magnitude of the quantity.

Combined Tolerances

Near zero, relative error can become awkward because division by a very small reference value magnifies the ratio.

Practical numerical tests often combine absolute and relative ideas, for example

This gives meaningful control over both small and large magnitudes.

A Reliable Error-Analysis Workflow

1. Identify the exact mathematical quantity of interest.
2. Identify all approximated or uncertain inputs.
3. Classify the likely error sources.
4. Measure absolute and relative input error when possible.
5. Determine how the computation transforms those errors.
6. Check for subtraction, small denominators, or sensitive functions.
7. Estimate or bound the resulting output error.
8. Compare the result with the required tolerance.
9. Refine the data, precision, or algorithm if necessary.

A Reliable Procedure for First-Order Propagation

1. Write the output as a function of the uncertain input or inputs.
2. Differentiate with respect to each uncertain variable.
3. Multiply each derivative by the corresponding small input change.
4. Add the first-order contributions.
5. Take absolute values for a conservative bound.
6. Divide by the output when a relative error estimate is desired.

Common Error: Treating Approximate Input as Exact

If a measured quantity is uncertain, every output depending on it inherits that uncertainty.

Carrying many decimal digits through the calculation does not magically create additional trustworthy information.

Common Error: Reporting Too Many Digits

A calculator may display many digits even when the input data justify only a few.

Numerical reporting should reflect the actual accuracy of the data and method.

Extra displayed digits can create a false impression of certainty.

Common Error: Ignoring Relative Scale

An absolute error may look small without being small relative to the quantity.

Always ask whether the scale of the error is appropriate for the scale of the result.

Common Error: Assuming Small Residual Means Small Error

A residual measures how well the approximate solution satisfies the equation.

It does not by itself measure how close the approximate solution is to the exact solution.

Conditioning determines how reliably one can convert residual information into solution-error information.

Common Error: Ignoring Cancellation

Subtracting nearly equal quantities can destroy significant digits.

If a formula contains such subtraction, consider whether an algebraically equivalent form could reduce cancellation.

Equivalent formulas can behave very differently numerically.

Common Error: Mixing Error Sources

Measurement error, truncation error, rounding error, and algorithmic instability are not the same phenomenon.

Separating them helps identify the correct remedy.

• better measurement reduces data error;
• more terms can reduce truncation error;
• higher precision can reduce rounding error;
• a better algorithm can reduce numerical amplification.

Example: Error in a Product

Suppose

and

The nominal product is

The approximate first-order absolute error bound is

Therefore,

Example: Error in a Quotient

Suppose the relative errors in \(x\) and \(y\) are approximately

and

For

a conservative first-order relative error estimate is approximately

Example: Error Through a Nonlinear Function

Let

Then

A small input error therefore propagates approximately as

The sensitivity changes with \(x\).

Example: Relative Error in a Square Root

Since

the power rule for relative errors gives

Therefore,

Practice 1: Absolute Error

The exact value is

and the approximation is

Find the absolute error.

Practice 2: Relative and Percentage Error

Using

find the relative error and percentage error.

Practice 3: Identify the Error Source

A computer stores \(0.1\) as a nearby binary floating-point number.

What type of error is involved?

Practice 4: Truncation Error

The exponential function is approximated by

What source of error is created by omitting the remaining series terms?

Practice 5: Propagating Error Through a Square

Let

At \(x=10\), estimate the output error when

Practice 6: Product Relative Error

Two quantities are multiplied. Their relative errors are approximately

and

Estimate a conservative first-order relative error in the product.

Practice 7: Power Error

If

estimate the relative error in \(y\) when the relative error in \(x\) is

Practice 8: Residual

For

calculate the residual at

Practice 9: Cancellation

Explain why computing

can be numerically sensitive if the two numbers have already been rounded.

Practice 10: Forward or Backward Error?

Identify the concept described by each question.

1. How close is the computed solution to the exact solution?
2. For what nearby input would the computed solution be exact?

Practice Answers

Practice 1

Practice 2

Absolute error:

Relative error:

Percentage error:

Practice 3

This is representation error, and the selection of the nearby representable value is associated with rounding.

Practice 4

Omitting the remaining terms creates truncation error.

Practice 5

Since

at \(x=10\),

Practice 6

Relative errors approximately add:

Practice 7

For \(y=x^4\),

Therefore,

Practice 8

The residual is

Practice 9

The exact difference is very small compared with the two inputs. If the inputs have already lost lower-order digits through rounding, subtraction can expose those errors and destroy most of the reliable significant digits. This is cancellation.

Practice 10

1. Forward error.
2. Backward error.

What to Remember

Numerical error begins with the distinction between an exact quantity \(x\) and an approximation \(\tilde{x}\).

and, when \(x\neq0\),

• measurement error comes from uncertain data;
• modeling error comes from simplifying reality;
• representation and rounding error come from finite precision;
• truncation error comes from replacing infinite processes with finite ones;
• iteration error remains when an iterative process is stopped;
• errors propagate through functions according to their sensitivity;
• relative errors in products and quotients approximately add in magnitude;
• subtraction of nearly equal quantities can cause catastrophic cancellation;
• forward error measures output discrepancy;
• backward error asks whether the computed result exactly solves a nearby problem;
• residuals, tolerances, and error bounds help assess numerical reliability.

Where We Go Next

Once error has been identified and tracked, the next question is whether the mathematical problem itself is sensitive and whether the chosen algorithm controls unavoidable perturbations.

The next lesson, Conditioning and Stability, separates these two ideas and develops the framework used to judge whether a numerical result is reliable because the problem is well-conditioned, the algorithm is stable, or both.