MGU Mathematics Series — Numerical Methods
Part I: Foundations of Numerical Thinking and Error
Error · Floating-Point · Loss of Significance · Conditioning · Stability
Chapter 2 of 14
2
MGU Mathematics Series  ·  Numerical Methods  ·  Library Textbook Edition

Chapter 2:
Error, Floating-Point
Arithmetic, and Stability

How numerical answers are measured, how computers represent real numbers, and why approximation without error awareness is dangerous

■ Part I of V  ·  Foundations of Numerical Thinking and Error ■

■ 14 Sections  •  Chapter 2 of 14 ■

Prerequisite: Chapter 1 — What Numerical Methods Study

Chapter 2 — Front Matter

Chapter Introduction and Learning Objectives

Every numerical method produces an approximation. That approximation differs from the exact answer by some amount — and that amount matters. Before students can responsibly use any numerical technique, they must understand how errors arise, how they are measured, how they propagate through computation, and how the structure of a computation can either control errors or allow them to grow out of hand.

This chapter introduces the language and tools of numerical error analysis. Students will learn to distinguish absolute error from relative error, rounding error from truncation error, and a well-conditioned problem from an ill-conditioned one. They will encounter floating-point arithmetic — the system by which computers represent real numbers — and will see that this system introduces subtle but consequential limitations. They will study how errors propagate through arithmetic operations, what it means for an algorithm to be stable or unstable, and how to judge whether the answer a method produces can be trusted.

This chapter is not a detour. It is the foundation on which every later chapter rests. Bisection, Newton's method, numerical integration, linear system solvers, ODE methods — every technique in this book must be understood through the lens of error, stability, and reliability. A numerical answer without error awareness is not mathematics. It is a number without meaning.

Opening Question

Suppose a student uses a calculator to evaluate the expression 1 / (1 − cos x) for a small value of x, say x = 0.001 radians. The calculator returns a large number. The student refines the computation and gets a slightly different large number. A third computation, using a slightly different algebraic form of the same expression, gives a third answer. All three look reasonable, but they cannot all be correct. What is happening? Why does a straightforward mathematical expression produce inconsistent results? And how can a student know which answer — if any — to trust? By the end of this chapter, the answer will be clear.

14Sections
10Objectives
20Key Terms
14Total Chapters

Learning Objectives

By the end of this chapter, you should be able to:

  1. 1Define and compute absolute error, relative error, and percent error for a given approximation.
  2. 2Identify the number of significant digits in a computed or measured result.
  3. 3Distinguish between rounding error and truncation error, and explain when each arises.
  4. 4Explain how real numbers are stored in floating-point format and why this representation is inexact.
  5. 5Describe machine precision (machine epsilon) and explain what it means for numerical computation.
  6. 6Identify loss of significance and explain why catastrophic cancellation is dangerous.
  7. 7Compute and interpret how errors propagate through arithmetic operations and function evaluations.
  8. 8Define the condition number of a problem and classify it as well-conditioned or ill-conditioned.
  9. 9Distinguish between a stable algorithm and an unstable one, and explain forward and backward error.
  10. 10Apply error concepts to judge the reliability of numerical results from later chapters.
Chapter 2 — Contents

Sections at a Glance

Sections 2.1 – 2.5
Measuring Error: Absolute, Relative, and Significant Digits
Absolute Error · Relative Error · Percent Error · Significant Figures
Section 2.1

Why Error Analysis Matters

In symbolic mathematics, when we write √2, we mean the unique positive real number whose square is exactly 2. This number is irrational: its decimal expansion never terminates and never repeats. A computer cannot store this number exactly. It stores an approximation — perhaps 1.41421356237310 — and uses that approximation in every subsequent calculation involving it.

This gap between the exact mathematical value and the stored numerical value is a representation error, and it exists for nearly every real number. By itself, this gap is small enough to be harmless for most purposes. The danger arises when tiny representation errors combine, amplify, and accumulate through many steps of computation.

Consider the following situations, each of which can cause an initially tiny error to become significant:

Subtraction
Near-Equal Values
Subtracting two nearly equal numbers amplifies relative error dramatically. The small difference may have far fewer reliable significant digits than either operand.
Division
Very Small Denominators
Dividing by a very small number magnifies any error in the numerator. A small absolute error in the numerator becomes a large absolute error in the quotient.
Sensitive Functions
Large Derivative
Applying a function with a large derivative near the evaluation point amplifies input errors. If |f′(x)| ≫ 1, even a small error in x produces a large error in f(x).
Accumulation
Many Small Errors
Repeating a computation thousands of times, where each step introduces a tiny error, can cause those errors to accumulate into a result that is significantly wrong.

Understanding error is not pessimism about computation. It is professionalism. Engineers designing aircraft, scientists modeling climate systems, and data scientists training algorithms must all understand what their computed answers mean — and what they do not mean. A number printed to twelve decimal places is not necessarily accurate to twelve decimal places. Significant digits, floating-point representation, conditioning, and stability together determine whether a computed answer is reliable.

Student Note

Error analysis is not a sign that numerical methods are weak. It is evidence that numerical methods are honest. Symbolic algebra can produce exact answers only because it avoids representing those answers as finite decimal numbers. The moment any computation happens in a computer, approximation enters. Error analysis is the discipline of understanding that approximation rigorously.

Section 2.2

Absolute Error

Let p denote the exact value of a quantity and let p* denote an approximation to p. The absolute error of the approximation is the magnitude of their difference.

Definition — Absolute Error
Eabs = |p − p*|
Absolute error measures the size of the discrepancy between the exact value and the approximate value, using the same units as the quantity being measured.
Example 2.2.1
Computing absolute error for an approximation to π
  • Problem The exact value of π is 3.14159265…. A student uses the approximation p* = 3.14. What is the absolute error?
  • Think The approximation 3.14 truncates π after two decimal places. The error is the magnitude of the difference between the true value and the approximation.
  • Compute Eabs = |3.14159265… − 3.14| = |0.00159265…| ≈ 0.00159
  • Interpret The approximation p* = 3.14 differs from π by about 0.00159 in absolute terms. Whether this error is acceptable depends entirely on the application. For a rough area estimate it may be fine; for precision engineering it may not be.

Absolute error is intuitive and easy to compute, but it has a significant limitation: it does not account for the scale of the quantity being approximated. An absolute error of 0.001 is negligible if we are measuring distances in kilometers, but it is enormous if we are measuring atomic radii in nanometers. This limitation motivates the concept of relative error.

Section 2.3

Relative Error

Relative error corrects the scale problem by expressing the absolute error as a fraction of the exact value, making the measure independent of the units involved.

Definition — Relative Error
Erel = |p − p*| / |p|    (p ≠ 0)
Relative error expresses the error as a fraction of the exact value. A relative error of 0.02 means the approximation differs from the exact value by about 2 parts per hundred.
Example 2.3.1
Relative error reveals what absolute error conceals
  • Problem Consider two measurements. In the first, an exact distance is p₁ = 1000 meters and the approximation is p₁* = 999 meters. In the second, an exact length is p₂ = 0.005 mm and the approximation is p₂* = 0.004 mm. Compare the absolute and relative errors.
  • Compute — First Eabs,1 = |1000 − 999| = 1 meter
    Erel,1 = 1 / 1000 = 0.001 = 0.1%
  • Compute — Second Eabs,2 = |0.005 − 0.004| = 0.001 mm
    Erel,2 = 0.001 / 0.005 = 0.2 = 20%
  • Interpret The second measurement has a far smaller absolute error (0.001 mm vs. 1 m), yet its relative error is vastly larger (20% vs. 0.1%). For the application, the second approximation is far worse. Relative error captures this distinction correctly; absolute error alone does not.
Reliability Note

When comparing the accuracy of two approximations, always consider relative error unless there is a specific reason that absolute error is the appropriate measure. For most numerical applications, relative error is the more informative and practical standard.

Section 2.4

Percent Error

Percent error is simply the relative error expressed as a percentage. It is widely used in science and engineering because it is immediately interpretable without further conversion.

Definition — Percent Error
E% = (|p − p*| / |p|) × 100%
A percent error of 2% means the approximation differs from the exact value by about 2 parts per hundred.
Example 2.4.1
Percent error in a physical measurement
  • Problem A student estimates the acceleration due to gravity as 9.75 m/s². The accepted value is 9.81 m/s². What is the percent error?
  • Compute E% = (|9.81 − 9.75| / 9.81) × 100% = (0.06 / 9.81) × 100% ≈ 0.61%
  • Interpret The estimate differs from the accepted value by less than 1%. In most classroom physics applications, this is acceptable. For precision engineering — say, designing a guidance system — it might not be.
Section 2.5

Significant Digits

The concept of significant digits (also called significant figures) describes how many digits of a number are reliably meaningful. When a number is the result of a measurement or computation, not all digits in its decimal representation are trustworthy. The rules for counting significant digits are as follows:

  • All nonzero digits are significant.
  • Zeros between nonzero digits are significant.
  • Leading zeros (zeros before the first nonzero digit) are not significant.
  • Trailing zeros in a decimal number are significant.
  • Trailing zeros in a whole number are ambiguous unless a decimal point is written explicitly.
NumberSignificant DigitsCount
3.14159all digits6
0.004074, 0, 73
1200ambiguous without context2, 3, or 4
1200.1, 2, 0, 04
0.050305, 0, 3, 04

There is a formal connection between significant digits and relative error. We say that an approximation p* approximates p to t significant digits if

|p − p*| / |p| < 5 × 10−t

This definition captures the idea that having t correct significant digits means the relative error is smaller than half a unit in the tth decimal place. In numerical computation, the number of significant digits in a result reflects the precision of that computation. When a result has fewer significant digits than expected, it often signals that digits have been lost — usually through subtraction of nearly equal numbers, which Section 2.10 addresses in detail.

Sections 2.6 – 2.9
How Computers Represent Numbers: Rounding, Truncation, and Floating-Point
Rounding Error · Truncation Error · IEEE 754 · Machine Epsilon
Section 2.6

Rounding Error

Rounding error arises when a number with more digits than a system can store is rounded to fit within the available precision. Every time a real number is stored in a computer's floating-point system, a rounding error may occur. This error is small but unavoidable. Rounding occurs in two standard ways:

  • Round-to-nearest: The stored value is the floating-point number closest to the exact value.
  • Chopping (truncation toward zero): The stored value is obtained by simply dropping all digits beyond the available precision.
Example 2.6.1
Rounding versus chopping π to five significant digits
  • Problem Store the number π = 3.14159265… with five significant decimal digits using (a) rounding and (b) chopping.
  • Rounding The fifth significant digit is 5. The next digit is 9 ≥ 5, so we round up: p*round = 3.1416
  • Chopping Chopping simply drops all digits after the fifth: p*chop = 3.1415
  • Errors Eround = |3.14159265… − 3.1416| ≈ 0.0000074
    Echop = |3.14159265… − 3.1415| ≈ 0.0000926
  • Interpret Rounding introduces roughly one-tenth the error of chopping here. Round-to-nearest is standard in modern computing systems for exactly this reason: it produces the smallest possible average rounding error.
Student Warning

Rounding errors in individual computations are typically tiny. Their danger comes from accumulation. In a computation involving thousands of operations, each contributing a tiny rounding error, the total accumulated error can become significant. This is one reason why the stability of an algorithm matters greatly.

Section 2.7

Truncation Error

Truncation error is fundamentally different from rounding error. It arises not from the limitations of number storage, but from replacing an infinite or exact mathematical process with a finite approximation.

The clearest example comes from Taylor series. The exponential function has the infinite expansion:

ex = 1 + x + x²/2! + x³/3! + x⁴/4! + ···

If we compute ex using only the first four terms, we introduce a truncation error — the error caused by cutting off an infinite process at a finite number of terms. A second example comes from the derivative definition. The derivative at a point x is the limit

f′(x) = limh→0 [f(x+h) − f(x)] / h

If a computer approximates this using a small but nonzero h, say h = 0.001, the formula

f′(x) ≈ [f(x + 0.001) − f(x)] / 0.001

introduces a truncation error: we have stopped the limit at h = 0.001 rather than actually taking h → 0. Both types of error coexist in nearly every numerical computation and must be managed separately.

Error TypeSourceCan Be Reduced By
Rounding error Finite digit storage in floating-point systems Using higher-precision arithmetic
Truncation error Finite approximation of an infinite process Using more terms, smaller steps, or higher-order methods
Section 2.8

Floating-Point Arithmetic

Real computers do not work with real numbers in the mathematical sense. They work with floating-point numbers: a finite set of rational numbers that approximates the real number line. A floating-point number in base β with t digits of precision has the form

± d1.d2d3···dt × βe

where β is the base (almost always 2 in modern computers), d1.d2···dt is the mantissa (also called the significand, with d1 ≠ 0 in a normalized representation), and e is the exponent, an integer in some range [emin, emax].

This representation can store a wide range of magnitudes — controlled by the exponent — with a fixed number of significant digits, controlled by the mantissa length t.

The IEEE 754 Standard

Modern computers overwhelmingly use the IEEE 754 standard for floating-point arithmetic. The two most common formats are single precision and double precision.

FormatBitsSignificant Decimal DigitsApproximate Range
Single precision 32 ~7 10−38 to 1038
Double precision 64 ~15–16 10−308 to 10308

Most numerical software uses double precision by default. Because floating-point systems have finitely many members, most real numbers are not representable exactly. The number ⅓ = 0.333… has no terminating binary or decimal representation and is always approximated. Even the simple decimal 0.1 does not have an exact binary floating-point representation — this surprises many students, but it follows directly from the fact that 1/10 in binary is an infinitely repeating fraction.

Example 2.8.1
Floating-point representation in base 10
  • Problem Express the number −0.00432 in normalized floating-point form in base 10 with 4 significant digits.
  • Compute −0.00432 = −4.320 × 10−3
  • Interpret The mantissa is 4.320, the exponent is −3, and the sign is negative. The stored number has exactly four significant digits. Moving the decimal point to achieve d₁ ≠ 0 is what normalization means.

Overflow and Underflow

When a computation produces a number larger than the maximum representable floating-point number, overflow occurs. The result is typically represented as ∞ in IEEE 754 systems, poisoning subsequent calculations. When a computation produces a nonzero number smaller in magnitude than the smallest representable positive floating-point number, underflow occurs. The result is typically rounded to zero, which can cause division-by-near-zero problems downstream.

Student Warning

Never assume that a computer will automatically warn you when overflow or underflow occurs. Some programming environments do; many do not. Part of responsible numerical work is monitoring the magnitude of intermediate results and checking that they remain in a sensible range.

Section 2.9

Machine Precision

Among the most important concepts in floating-point arithmetic is machine precision (also called machine epsilon), denoted εmach. It is defined as the smallest positive floating-point number ε such that

1 + ε ≠ 1    in floating-point arithmetic

Equivalently, machine epsilon is the gap between 1 and the next larger floating-point number. For IEEE 754 double precision (64-bit), machine epsilon is approximately

IEEE 754 Double Precision Machine Epsilon
εmach ≈ 2.22 × 10−16
This is the fundamental precision limit of double-precision computation. Any relative error smaller than εmach is unresolvable at this precision level — indistinguishable from zero.

Machine epsilon quantifies the fundamental precision limitation of double-precision computation. If a computed result has a relative error smaller than εmach, the computation is as accurate as the floating-point system allows — the best achievable without switching to higher-precision arithmetic. On the other hand, if a computed result has a relative error close to 1 — meaning essentially no correct significant digits remain — something has gone badly wrong, and the culprit is usually loss of significance or a poorly conditioned problem.

Interpretation

Double precision carries roughly 15–16 significant decimal digits. When you see an answer like 2.718281828459045, that last digit is at the edge of what double precision can reliably represent. Treating all 16 digits as accurate without verifying the computation is unwarranted confidence.

Sections 2.10 – 2.13
Catastrophic Cancellation, Error Propagation, Conditioning, and Stability
Loss of Significance · Propagation · Condition Number · Stability · Backward Error
Section 2.10

Loss of Significance

One of the most dangerous phenomena in floating-point arithmetic is loss of significance, also called catastrophic cancellation. It occurs when two nearly equal floating-point numbers are subtracted, causing the significant digits to cancel and leaving a result dominated by rounding error.

Example 2.10.1
How subtraction destroys significant digits
  • Setup Suppose we compute a − b where a = 1.000000001 and b = 1.000000000 in a floating-point system with 8 significant decimal digits.
  • Problem Both a and b are stored with only 8 significant digits. The ninth digit — the one that distinguishes them — is beyond the system's precision and may be lost to rounding.
  • Result If both values round to 1.0000000 in the 8-digit system, the computed difference is 0. But the true difference is 10−9. The relative error is 100%.
  • Interpret All significant digits have been destroyed by cancellation. The computed answer is zero; the exact answer is 10−9. This is catastrophic cancellation.

This is precisely what happens in the expression 1/(1 − cos x) for small x. When x is small, cos x is very close to 1, so 1 − cos x is the difference of two nearly equal numbers. That subtraction loses significance, producing a result with few reliable significant digits. Dividing by that small inaccurate number amplifies the error dramatically.

The remedy is to rewrite the expression algebraically before computing. Using the trigonometric identity 1 − cos x = 2 sin²(x/2):

1 / (1 − cos x) = 1 / (2 sin²(x/2))

This form avoids the catastrophic subtraction entirely and produces an accurate result for small x. The opening question is now resolved.

Example 2.10.2
The quadratic formula and loss of significance
  • Problem Use the quadratic formula to find the roots of x² − 1000x + 1 = 0.
  • Compute x = (1000 ± √(1000² − 4)) / 2 = (1000 ± √999996) / 2
    √999996 ≈ 999.998000
    x₁ = (1000 + 999.998000) / 2 ≈ 999.999000
    x₂ = (1000 − 999.998000) / 2 = 0.002000 / 2 = 0.001000
  • Problem The computation of x₁ is fine. But x₂ requires subtracting 1000 and 999.998000 — two nearly equal numbers. Small rounding errors in √999996 will produce a result for x₂ with far fewer reliable significant digits than the inputs.
  • Remedy For the smaller root, use Vieta's formula: x₁ · x₂ = c/a = 1, so x₂ = 1/x₁ ≈ 1/999.999000 ≈ 0.0010000010. This avoids catastrophic cancellation entirely.
Key Principle

When a computation requires subtracting two quantities that may be nearly equal, look for an algebraically equivalent form that avoids the subtraction. This is one of the most practical and important skills in numerical computing. Mathematical equivalence does not guarantee numerical equivalence in floating-point arithmetic.

Section 2.11

Error Propagation

Errors do not stay fixed when we perform arithmetic with approximate numbers. They propagate: when we add, subtract, multiply, or divide approximate quantities, the errors in those quantities combine to produce an error in the result. Understanding how this combination works is essential for judging whether a multi-step computation can be trusted.

Addition and Subtraction

Let x* approximate x with error δx = x − x*, and let y* approximate y with error δy = y − y*. The absolute error in a sum satisfies

|δ(x ± y)| ≤ |δx| + |δy|

Absolute errors add when summing approximate numbers. This is why adding many approximate numbers — say, summing a thousand measured values — can accumulate significant total error even if each individual error is tiny.

Multiplication and Division

For products and quotients, relative errors approximately add (for small relative errors):

|δ(xy)| / |xy| ≈ |δx| / |x| + |δy| / |y|

The same formula applies for division. Relative errors combine additively through multiplication and division, meaning that these operations do not amplify errors beyond the sum of the input relative errors.

Functions of One Variable

For a smooth function f, the error in f(x*) as an approximation to f(x) is approximately

|f(x) − f(x*)| ≈ |f′(x)| · |x − x*|

The derivative |f′(x)| is an amplification factor: if |f′(x)| is large near the point of evaluation, even a small input error can produce a large output error.

Example 2.11.1
Error amplification in a function evaluation
  • Problem Suppose x = 2.00 approximates an exact value in [1.99, 2.01], so |δx| ≤ 0.01. Estimate the resulting error in f(x) = x³.
  • Compute f′(x) = 3x². At x = 2: f′(2) = 12.
    |δf| ≈ |f′(2)| · |δx| = 12 · 0.01 = 0.12
  • Interpret An input error of at most 0.01 produces an output error of at most approximately 0.12 in x³. The function amplifies the input error by a factor of about 12 at this point. The amplification factor grows as x increases, since f′(x) = 3x² grows.
Section 2.12

Conditioning

A fundamental distinction in numerical analysis is between the difficulty of a problem and the quality of the algorithm used to solve it. Some problems are inherently sensitive: small changes in input produce large changes in output, regardless of what algorithm is used. These problems are called ill-conditioned.

The condition number of a problem measures this inherent sensitivity. For a function f evaluated at a point x ≠ 0, the condition number is defined as

Condition Number — Function of One Variable
κ = |x · f′(x)| / |f(x)|
This ratio measures the relative change in output per unit relative change in input. A large condition number means small relative errors in the input produce large relative errors in the output.

If κ ≈ 1, the problem is well-conditioned: errors in input are not significantly amplified. If κ ≫ 1, the problem is ill-conditioned: small input errors produce large output errors, and the computed result may be unreliable regardless of the algorithm used.

Critical Distinction

Conditioning is a property of the problem, not the algorithm. Even a perfect algorithm, with no rounding error whatsoever, will produce an inaccurate result when applied to an ill-conditioned problem if the input data are not exact. No algorithm can compensate for a fundamentally sensitive problem.

Example 2.12.1
Condition number of √x near x = 4
  • Problem Estimate the condition number of f(x) = √x near x = 4.
  • Compute f′(x) = 1/(2√x). At x = 4: f(4) = 2, f′(4) = 1/4.
    κ = |4 · (1/4)| / |2| = 1/2
  • Interpret The condition number is 0.5, less than 1. The function √x near x = 4 is well-conditioned: a 1% relative error in x produces only about a 0.5% relative error in √x. Square root is a smoothing function.
Application Note — Linear Systems

For a linear system Ax = b, the relevant condition number is the matrix condition number κ(A), studied in detail in Chapter 9. When κ(A) is very large — say, 10¹⁰ — a relative error of 10⁻¹⁶ in the input data can produce a relative error approaching 10⁻⁶ in the solution, a loss of ten significant digits. Matrix conditioning is one of the central concerns of numerical linear algebra.

Section 2.13

Stability of Algorithms

Where conditioning describes sensitivity of the problem, stability describes sensitivity of the algorithm. Two different algorithms may solve the same problem with the same conditioning, but one may amplify rounding errors at each step while the other keeps them under control.

An algorithm is stable if small perturbations in the input (or small rounding errors introduced during computation) produce only small perturbations in the output. An algorithm is unstable if rounding errors introduced during the computation grow and eventually dominate the result.

Forward and Backward Error

Two complementary ways of measuring algorithm error give different insights:

Forward Error
Distance from the True Answer
The difference between the computed answer and the exact answer: |p* − p|. This is the most direct measure, but can be hard to compute since we rarely know p exactly.
Backward Error
How Much the Problem Changed
How much would we need to change the input so that the computed answer becomes the exact answer to that perturbed problem? A small backward error means "I solved a problem very close to the intended one."

A backward stable algorithm is one whose backward error is small — roughly the size of machine epsilon. This is often the most practical goal: an algorithm that produces a result which is the exact answer to a slightly perturbed version of the input. Forward error then depends on both backward error and condition number: forward error ≈ condition number × backward error.

Example 2.13.1
Instability in a recurrence relation — direction matters
  • Setup The integrals Eₙ = ∫₀¹ xⁿeˣ dx satisfy the recurrence Eₙ = e − n·Eₙ₋₁, derived by integration by parts. We compute E₀ = e − 1 ≈ 1.71828 and use the forward recurrence to generate E₁, E₂, …
  • Analysis Each step multiplies the previous result by −n, where n grows. An initial error ε in E₀ becomes an error of magnitude n! · ε after n steps. For n = 30 with ε = 10⁻¹⁶, the accumulated error is 30! × 10⁻¹⁶ — an astronomically large number. The forward recurrence is unstable for large n.
  • Remedy The backward recurrence — starting from a known approximation for large n and working downward — is stable, because the factor (−n)⁻¹ contracts errors at each step rather than amplifying them.
  • Key Insight Stability is not a property of a formula alone. It is a property of how and in what direction the formula is evaluated. The same mathematical recurrence, run forward or backward, can be stable in one direction and unstable in the other.
Section 2.14

Common Error Analysis Mistakes

Students encountering numerical error analysis for the first time make certain characteristic mistakes. Recognizing these now will prevent much confusion in later chapters.

Mistake 1
"Rounding error and truncation error are the same thing."
Rounding error comes from the finite precision of the floating-point system: a number that has more digits than can be stored gets rounded. Truncation error comes from using a finite approximation for an infinite mathematical process — stopping a Taylor series early, or using a finite step size instead of the true limit. They coexist in nearly every numerical computation and must be managed separately.
Mistake 2
"More decimal digits in the answer means greater accuracy."
A computed answer printed to 15 decimal places is not accurate to 15 decimal places unless the entire computation was carefully controlled. If loss of significance occurred early in the computation, only the first few digits may be reliable. The number of significant digits in the output depends on the whole computation, not just the final display precision.
Mistake 3
"Making the step size smaller always improves accuracy in numerical differentiation."
Reducing the step size h reduces truncation error but increases rounding error, because we must subtract nearly equal function values. There is an optimal step size, and making h too small actually worsens the result. Chapter 6 analyzes this tradeoff in detail. Smaller is not always better.
Mistake 4
"Conditioning and stability are different words for the same thing."
Conditioning describes the problem; stability describes the algorithm. A well-conditioned problem solved by an unstable algorithm will give a bad answer. An ill-conditioned problem solved by a stable algorithm will give the best answer the problem allows — which may still not be good, because the problem itself is inherently sensitive. They must be assessed separately.
Mistake 5
"Individual rounding errors are tiny, so I don't need to worry about them."
Tiny errors accumulate. A computation with 10⁶ steps, each introducing a rounding error of 10⁻¹⁶, could produce an accumulated error of order 10⁻¹⁰. For high-precision applications — GPS positioning, financial derivatives pricing, scientific simulation — this matters enormously.
Mistake 6
"The condition number is a property of my algorithm, so I can improve it by choosing a better method."
The condition number is a property of the mathematical problem, not of any particular algorithm for solving it. An algorithm cannot improve a problem's conditioning. It can only solve the problem well or poorly given that conditioning. When a problem is ill-conditioned, the remedy is to reformulate the problem — perhaps by choosing better coordinates or scaling — not to choose a different algorithm for the same formulation.
Chapter 2 — Back Matter

Chapter Summary

Key Terms

absolute error
|p − p*|: the magnitude of the difference between the exact value p and the approximation p*.
relative error
|p − p*| / |p|: the absolute error as a fraction of the exact value; scale-independent.
percent error
Relative error expressed as a percentage: (|p − p*| / |p|) × 100%.
significant digits
The number of reliably meaningful digits in an approximation; related to relative error by the formal definition |p − p*| / |p| < 5 × 10⁻ᵗ.
rounding error
Error from storing a number with fewer digits than it has; arises in every floating-point representation of a non-representable real number.
truncation error
Error from replacing an infinite mathematical process (limit, series) with a finite approximation.
floating-point number
A finite representation of a real number using a fixed-precision mantissa and integer exponent in a given base.
mantissa
The significand of a floating-point number: the sequence of significant digits, normalized so the leading digit is nonzero.
machine epsilon
The smallest positive ε such that 1 + ε ≠ 1 in floating-point arithmetic; ≈ 2.22 × 10⁻¹⁶ for IEEE 754 double precision.
overflow
Occurs when a computation produces a number larger than the maximum representable floating-point value; result is typically ∞.
underflow
Occurs when a computation produces a nonzero number smaller than the smallest representable positive value; result is typically rounded to zero.
loss of significance
The destruction of significant digits when two nearly equal floating-point numbers are subtracted; also called catastrophic cancellation.
catastrophic cancellation
See loss of significance; emphasizes the severity of the digit loss in subtraction of nearly equal values.
error propagation
How input errors combine and transfer to output errors through arithmetic and function evaluation.
condition number
κ = |x · f′(x)| / |f(x)|: measures how sensitively a problem's output responds to perturbations in its input.
well-conditioned
A problem with small condition number; small relative errors in input produce small relative errors in output.
ill-conditioned
A problem with large condition number; small relative errors in input may produce large relative errors in output.
stable algorithm
An algorithm in which rounding errors introduced during computation remain controlled and do not grow to dominate the result.
forward error
The difference between the computed answer and the exact answer: |p* − p|.
backward error
The size of the perturbation to the input that would make the computed answer exact for the perturbed problem.
ReviewConcept Review Questions
  1. Define absolute error and relative error. Which is more informative when comparing two approximations of quantities at very different magnitudes? Give a one-sentence justification.
  2. What is the difference between rounding error and truncation error? Give an original example of each that does not appear in this chapter.
  3. Explain in your own words what machine epsilon represents. If machine epsilon for a system is 10⁻⁷, approximately how many significant decimal digits does the system carry?
  4. Describe catastrophic cancellation. Under what arithmetic circumstance does it arise? Give a mathematical scenario where it could occur.
  5. A student says: "My result agrees with the exact answer to 10 decimal places, so my algorithm must be stable." Why might this reasoning be incomplete?
  6. Explain the difference between a well-conditioned problem and a stable algorithm. Is it possible to have a well-conditioned problem solved by an unstable algorithm? An ill-conditioned problem solved by a stable algorithm? What are the consequences in each case?
  7. What does it mean for errors to propagate through a computation? In what arithmetic context can errors be most dramatically amplified?
  8. Define backward error. Why is backward error often a more practical measure of algorithmic quality than forward error?
PracticeSkill Practice
  1. For each of the following, compute the absolute error, relative error, and percent error.

    (a) Exact value p = 2.71828… (e), approximation p* = 2.718.
    (b) Exact value p = 100, approximation p* = 97.3.
    (c) Exact value p = 0.000125, approximation p* = 0.000130.
  2. How many significant digits are in each of the following numbers?

    (a) 0.004050    (b) 30000 (no decimal point)    (c) 30000.    (d) 1.00320
  3. Round each of the following to four significant digits.

    (a) 3.14159265    (b) 0.00271828    (c) 149600000
  4. A floating-point system in base 10 stores four significant decimal digits with exponents from −5 to 5.

    (a) What is the largest positive number this system can represent?
    (b) What is the smallest positive normalized number?
    (c) Represent 0.003476 in normalized form.
    (d) What happens if you try to store 12345 in this system?
  5. In double-precision floating-point arithmetic, machine epsilon is approximately 2.22 × 10⁻¹⁶. Approximately how many significant decimal digits does a double-precision computation carry?
PracticeError Propagation and Stability
  1. Let f(x) = x² − x. Suppose x* = 2.01 approximates x = 2.

    (a) Compute the absolute error in x.
    (b) Use the linear approximation |δf| ≈ |f′(x)| · |δx| to estimate the error in f.
    (c) Compute f(2) and f(2.01) and find the actual absolute error. Compare with your estimate.
  2. Let a* = 1.23 ± 0.01 and b* = 4.56 ± 0.02 be measured quantities with the given absolute errors.

    (a) Estimate the absolute error in a* + b*.
    (b) Estimate the relative error in a* · b*.
    (c) Estimate the relative error in a* / b*.
  3. The expression √(x+1) − √x is prone to loss of significance for large x.

    (a) Explain why subtraction becomes dangerous for large x.
    (b) Show algebraically that √(x+1) − √x = 1 / (√(x+1) + √x).
    (c) For x = 1000, compute both forms numerically and compare. Which is more reliable?
  4. Compute the condition number of f(x) = eˣ at x = 1 and at x = 10. Interpret what these numbers mean for the relative error in evaluating f.
  5. Consider the recurrence aₙ₊₁ = 2aₙ − 1 starting from a₀ = 0.5.

    (a) What is the exact value of aₙ for all n?
    (b) Suppose a₀ = 0.5 + ε for a small error ε. What is aₙ as a function of n and ε?
    (c) Is this recurrence stable? Explain.
ApplicationsReal-World Applications
  1. Surveying. A surveyor measures a distance as d* = 152.3 meters with an absolute error of at most 0.1 meters. The surveyor squares this distance to compute an area.

    (a) Estimate the absolute error in the computed area.
    (b) Estimate the relative error in the computed area and compare it to the relative error in the measured distance.
  2. Medication Dosing. A dosing formula requires computing e−0.693t for values of t from 0 to 10. The coefficient −0.693 approximates −ln 2 rounded to three decimal places.

    (a) Estimate the relative error in the exponent at t = 10 due to this rounding.
    (b) Estimate the resulting relative error in e−0.693·10.
    (c) Is this level of error likely to be acceptable for dosing calculations? Discuss briefly.
  3. Polynomial Evaluation. Consider two algorithms for evaluating p(x) = aₙxⁿ + aₙ₋₁xⁿ⁻¹ + ··· + a₁x + a₀:

    Algorithm A: Evaluate each term aₖxᵏ separately, computing a separate power for each, and sum all terms.

    Algorithm B (Horner's method): Rewrite as p(x) = (···((aₙx + aₙ₋₁)x + aₙ₋₂)x + ··· + a₁)x + a₀ and evaluate from inside out.

    (a) Count the multiplications each algorithm requires for a degree-n polynomial.
    (b) Why does Horner's method tend to introduce less rounding error?
  4. Climate Modeling. Global climate simulations involve billions of floating-point operations per timestep. In your own words, explain why error analysis matters for long-range simulations, and identify one type of error (rounding or truncation) that climate scientists would need to control carefully.
Error AnalysisReasoning About Reliability
  1. Explain in two to three sentences why reducing the number of floating-point operations in a computation tends to reduce total rounding error.
  2. Suppose a numerical algorithm applied to a well-conditioned problem (condition number κ ≈ 1) produces a result with a relative error of 10⁻³, even though double-precision arithmetic with εmach ≈ 10⁻¹⁶ is used. What might explain this discrepancy?
  3. Suppose a numerical algorithm applied to an ill-conditioned problem (condition number κ = 10⁸) produces a result with a relative error of 10⁻⁸. Is this surprising? Is it acceptable? What is the best achievable relative error given double-precision arithmetic?
  4. A student evaluates cos(0.001) − 1 as part of a larger computation. They note the result is very small.

    (a) Identify the source of potential error in this computation.
    (b) Using the Maclaurin series cos x ≈ 1 − x²/2 + x⁴/24 − ···, propose a reformulation that avoids catastrophic cancellation.
Methods Reference — Chapter 2 Formulas
Absolute Error
Eabs = |p − p*|
Relative Error
Erel = |p − p*| / |p|
Percent Error
E% = (|p − p*| / |p|) × 100%
t Significant Digits
|p − p*| / |p| < 5 × 10−t
Error in Function Evaluation
|f(x) − f(x*)| ≈ |f′(x)| · |x − x*|
Condition Number (function)
κ = |x · f′(x)| / |f(x)|
Absolute Error Propagation (sums)
|δ(x ± y)| ≤ |δx| + |δy|
Relative Error Propagation (products)
|δ(xy)| / |xy| ≈ |δx| / |x| + |δy| / |y|
IEEE 754 Double Precision εmach
εmach ≈ 2.22 × 10−16
Chapter 2 Checkpoint
Confirm your understanding before moving to Chapter 3
Error Measurement
  • I can compute absolute error, relative error, and percent error given an exact value and an approximation.
  • I can count the significant digits in a given numerical result.
  • I understand why relative error is usually more informative than absolute error when comparing approximations at different scales.
  • I can explain the formal relationship between significant digits and relative error.
Sources of Error
  • I can distinguish rounding error from truncation error and explain when each arises.
  • I understand how floating-point numbers are represented using a mantissa and exponent.
  • I know that most real numbers — including 0.1 — are not exactly representable in binary floating-point.
  • I can explain what machine epsilon is and what it means for the resolution of computation.
  • I understand what overflow and underflow are and why they must be monitored.
Loss of Significance and Propagation
  • I can identify a computation that is prone to catastrophic cancellation.
  • I can propose an algebraically equivalent reformulation that avoids catastrophic cancellation.
  • I can estimate how errors propagate through arithmetic operations using the rules for sums, products, and function evaluations.
  • I understand the role of the derivative as an amplification factor in error propagation.
Conditioning and Stability
  • I can compute the condition number of a function at a given point.
  • I can classify a problem as well-conditioned or ill-conditioned based on its condition number.
  • I understand that conditioning is a property of the problem, not the algorithm.
  • I can distinguish between a stable algorithm and an unstable one.
  • I can explain the difference between forward error and backward error.

Checkpoint Problems

  1. Checkpoint 2.1. The exact value of ln 3 is approximately 1.09861228867…. A student uses the approximation 1.099. Compute the absolute error, relative error, and percent error. How many significant digits does the approximation have?
  2. Checkpoint 2.2. Explain in your own words why the expression cos x − 1 for small x is prone to loss of significance, and describe a strategy using a trigonometric identity to evaluate it accurately.
  3. Checkpoint 2.3. The formula p(x) = (x−1)⁶ expands to x⁶ − 6x⁵ + 15x⁴ − 20x³ + 15x² − 6x + 1. Evaluate p(1.0001) using the expanded form with a 6-digit decimal calculator (round each intermediate result to 6 significant digits). Then evaluate using the factored form (x−1)⁶. Compare the two results and explain any discrepancy.
  4. Checkpoint 2.4. Compute the condition number of f(x) = tan x at x = π/4. Then compute it near x = π/2 − 0.01, close to where tan x diverges. What do these numbers suggest about the reliability of computing tan x near π/2?
  5. Checkpoint 2.5. Consider the recurrence yₙ = 10yₙ₋₁ − 9 with y₀ = 1.
    (a) Show that the exact solution is yₙ = 1 for all n.
    (b) Start with y₀ = 1 + ε for small ε and compute how the error grows with n.
    (c) Is this recurrence stable? What does this imply for using it in floating-point arithmetic starting from y₀ = 1.000000000000001?
Bridge Note — Chapters 3 through 14
Error Analysis Is the Thread of the Entire Book

The error concepts introduced in this chapter will reappear in every chapter that follows. In Chapter 3, stopping criteria for root-finding methods depend directly on absolute and relative error tolerances. In Chapters 4 and 5, interpolation error bounds describe how polynomial approximations deviate from the true function. In Chapters 6 and 7, the tradeoff between truncation error and rounding error determines how to choose step sizes. In Chapter 8, Taylor remainder terms provide truncation error bounds for polynomial approximations. In Chapter 9, the matrix condition number determines how accurately a linear system can be solved. In Chapters 12 and 13, local and global errors describe the accuracy of ODE and PDE solvers. In Chapter 14, responsible scientific computing requires communicating error honestly. Error analysis is not a topic studied once and set aside. It is the thread connecting the entire discipline of numerical methods.

MGU Library Connection
  • Numerical Methods — Appendix E: Error Analysis Reference
  • Numerical Methods — Appendix F: Floating-Point Arithmetic Reference
  • MGU Dictionary entries: floating-point arithmetic, significant figures, condition number, algorithm stability, numerical precision
  • MGU Calculus — Chapter on Taylor Series (connects to truncation error in Section 2.7)
  • MGU Computer Science Foundations — Binary number representation and IEEE 754 arithmetic
  • Related chapter: Chapter 9 (Numerical Linear Algebra) — matrix condition numbers
  • Related chapter: Chapter 6 (Numerical Differentiation) — step size and the rounding/truncation tradeoff