Chapter 6 — Front Matter

Chapter Introduction and Learning Objectives

A weather monitoring station records temperature every ten minutes throughout the day. A meteorologist wants to know how quickly the temperature is rising at noon. There is no algebraic formula for the temperature — only a table of recorded values. How do you estimate a rate of change when you have data but no formula? How close is your estimate to the true rate of change? And if the measurements contain small errors — as real sensor data always does — how do those errors affect your answer?

These are the central questions of numerical differentiation. This chapter introduces the art and science of estimating derivatives from function values or data when exact symbolic differentiation is unavailable, impractical, or simply not the right tool.

Students who have studied calculus know that the derivative of a function is defined as the limit of a difference quotient. Numerical differentiation replaces that limit with a carefully chosen finite approximation — one that is close enough to be useful but subject to errors that must be understood and managed. This chapter develops the three fundamental finite difference formulas, analyzes their accuracy using Taylor series, and explores what happens when the step size is too large, too small, or poorly chosen.

Opening Question

A GPS device records the position of a vehicle at one-second intervals. How do you estimate velocity from position data? How large can the estimation error be? If the sensor has measurement noise of ±1 meter, does a finer time resolution help or hurt? By the end of this chapter, you will be able to answer all three questions precisely — and apply the same reasoning to data from any source.

14Sections

9Objectives

3Core Formulas

14Total Chapters

Learning Objectives

By the end of Chapter 6, students should be able to:

1Explain what numerical differentiation is and why it is needed when symbolic differentiation is unavailable.
2Derive the forward difference, backward difference, and central difference formulas from the derivative definition and from Taylor series.
3Apply these formulas to estimate first derivatives from function values or tabular data.
4Identify the order of accuracy of each formula using Taylor series error analysis.
5Explain the tradeoff between truncation error and roundoff error in choosing a step size.
6Apply central difference formulas for second derivatives and higher-order approximations.
7Discuss how noisy data affects numerical differentiation and describe strategies for managing this challenge.
8Recognize common mistakes in numerical differentiation and know how to avoid them.
9Connect numerical differentiation to interpolation, Taylor series, ODE solvers, and PDE methods.

Chapter 6 — Key Terms

Vocabulary for This Chapter

numerical differentiation

Estimation of a derivative from function values rather than by symbolic computation.

finite difference

A discrete approximation to a derivative, formed from differences of function values at nearby points.

forward difference formula

An approximation to \( f'(x) \) using \( f(x) \) and \( f(x+h) \); first-order accurate.

backward difference formula

An approximation to \( f'(x) \) using \( f(x) \) and \( f(x-h) \); first-order accurate.

central difference formula

An approximation to \( f'(x) \) using \( f(x+h) \) and \( f(x-h) \); second-order accurate.

step size

The spacing \( h \) between evaluation points in a finite difference formula; critically affects accuracy.

truncation error

Error caused by replacing a limiting operation with a finite difference; decreases as \( h \to 0 \).

roundoff error

Error caused by limited floating-point precision; increases as \( h \to 0 \) in differentiation.

order of accuracy

Rate at which truncation error decreases as \( h \) shrinks; a method of order \( p \) has error proportional to \( h^p \).

second-order method

A method with error \( O(h^2) \); error shrinks by factor of 4 when \( h \) is halved.

higher-order difference formula

A finite difference approximation using more evaluation points to achieve better-than-second-order accuracy.

noisy data

Data containing measurement errors that distort numerical differentiation, especially with small step sizes.

Chapter 6 — Contents

Sections at a Glance

6.1 Why Numerical Differentiation Is Needed When symbolic differentiation fails; data without formulas; computation pipelines; applications in science, engineering, and finance
6.2 Derivatives from Data Estimating velocity from position tables; the intuition behind finite differences; average rates of change as approximations
6.3 Difference Quotients The forward, backward, and central difference quotients; their relationship to the derivative definition
6.4 Forward Difference Formula Derivation from Taylor series; first-order accuracy; error term; worked example with \( e^x \); step size table
6.5 Backward Difference Formula Derivation; first-order accuracy; when to use it; worked example at a data table endpoint
6.6 Central Difference Formula Derivation; second-order accuracy; why symmetry cancels the \( h \) error term; comparison of all three formulas
6.7 Higher-Order Difference Approximations Second derivative by central differences; four-point first-derivative formula; Richardson extrapolation preview
6.8 Taylor Series and Error Terms Systematic error analysis; formal error statements; bounding truncation error with known derivatives
6.9 Step Size and Roundoff Error Two sources of error; total error model; optimal step size; double-precision arithmetic; observing the crossover
6.10 Differentiating Interpolating Polynomials Polynomial interpolation as a route to derivatives; three-point endpoint formulas; second-order endpoint accuracy
6.11 Numerical Differentiation with Noisy Data Why noise is amplified by differentiation; noise contribution formula; smoothing strategies; reliability warnings
6.12 Numerical Derivatives in Science and Engineering GPS velocity; image edge detection; finite element methods; option sensitivities in finance
6.13 Common Numerical Differentiation Mistakes Step size too small; using first-order when second-order is available; noise amplification; unequal spacing errors
6.14 Preparing for Numerical Integration Differentiation versus integration; noise-amplifying vs. noise-averaging; preview of Chapter 7 methods

Section 6.1

Why Numerical Differentiation Is Needed

Calculus defines the derivative of a function \( f \) at a point \( x \) as a limit:

f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}

This definition is elegant, precise, and exact. Given a formula for \( f \), we can differentiate symbolically and compute \( f'(x) \) exactly. But this approach requires several things that are not always available.

A symbolic formula for \( f \). If \( f \) is defined by experimental measurements, simulation output, or a black-box computer program, there is no formula to differentiate. We have only a table of numbers.

Differentiability in a form we can exploit. Even when a symbolic formula exists, it may be so complicated — products of exponentials, trigonometric functions, and implicit constraints — that symbolic differentiation is correct but cumbersome to compute and verify in practice.

Access to the derivative of each component. In some computation pipelines, a subroutine computes \( f(x) \) for any input \( x \), but does not return derivative information. The only way to estimate \( f'(x) \) is to call the subroutine at two or more nearby points and compare the outputs.

Numerical differentiation solves all of these problems in the same way: it evaluates \( f \) at one or more nearby points and forms a finite difference — a ratio of changes — to approximate the derivative. The challenge, as we will see throughout this chapter, is choosing the spacing wisely. Too large, and the approximation is inaccurate. Too small, and roundoff error corrupts the result.

Scientific Computing

Rates from Simulation Output

When a simulation produces a time series of values but no symbolic expression, finite differences estimate rates of change at each time step.

ODE Solvers

Stepping Forward in Time

Numerical ODE solvers approximate derivatives to predict the next state of a differential equation. Every step is a finite difference computation.

Optimization

Gradients from Black-Box Functions

When an objective function has no closed-form gradient, finite differences estimate the gradient numerically for use in gradient descent.

Signal Processing

Detecting Changes in Digital Signals

Edges, peaks, and transitions in digital images and time-series signals correspond to large numerical derivatives computed from discrete samples.

Finance

Option Sensitivities (The Greeks)

The sensitivity of an option price to its underlying parameters — the Greeks — is estimated by finite differences of the pricing function.

Engineering

Field Data Analysis

Stresses, strains, flow rates, and reaction rates are computed from field measurements using finite difference formulas applied to sensor data.

Section 6.2

Derivatives from Data

Before introducing formal formulas, it helps to think about what a derivative means when all we have is data. Suppose we know the position \( s(t) \) of a moving object at several times:

\( t \) (seconds)	\( s(t) \) (meters)
0.0	0.00
0.5	1.23
1.0	2.48
1.5	3.72
2.0	4.95

We want to estimate the velocity \( s'(t) \) at \( t = 1.0 \).

Between \( t = 1.0 \) and \( t = 1.5 \), the position increases by \( 3.72 - 2.48 = 1.24 \) meters over \( 0.5 \) seconds. The average rate of change is \( 1.24 / 0.5 = 2.48 \) m/s. Between \( t = 0.5 \) and \( t = 1.0 \), the change is \( 2.48 - 1.23 = 1.25 \) meters over \( 0.5 \) seconds, giving \( 2.50 \) m/s. Using the symmetric interval from \( t = 0.5 \) to \( t = 1.5 \), the change is \( 3.72 - 1.23 = 2.49 \) meters over \( 1.0 \) second, giving \( 2.49 \) m/s.

Each of these is a finite difference approximation to \( s'(1.0) \). The formulas we develop in this chapter give these estimates a precise mathematical form and allow us to say exactly how large the approximation error can be.

Section 6.3

Difference Quotients

The derivative definition begins with the difference quotient. For any fixed nonzero \( h \), a difference quotient is an approximation to \( f'(x) \). There are three natural ways to form one, depending on which nearby points we choose:

The Three Fundamental Difference Quotients

Forward difference — step forward from \( x \):

\( \displaystyle\frac{f(x+h) - f(x)}{h} \)

Backward difference — step backward to \( x \):

\( \displaystyle\frac{f(x) - f(x-h)}{h} \)

Central difference — straddle \( x \) symmetrically:

\( \displaystyle\frac{f(x+h) - f(x-h)}{2h} \)

All three approach \( f'(x) \) as \( h \to 0 \). They differ in accuracy, required points, and error behavior.

Section 6.4

Forward Difference Formula

The forward difference formula for the first derivative is:

f'(x) \approx \frac{f(x+h) - f(x)}{h}

To understand its accuracy, we use the Taylor series expansion of \( f(x+h) \) around \( x \):

f(x+h) = f(x) + h f'(x) + \frac{h^2}{2} f''(x) + \frac{h^3}{6} f'''(x) + \cdots

Subtracting \( f(x) \) and dividing by \( h \):

\frac{f(x+h) - f(x)}{h} = f'(x) + \frac{h}{2} f''(x) + O(h^2)

The dominant error term is \( \frac{h}{2} f''(x) \), which is proportional to \( h \). The forward difference formula is a first-order method: when \( h \) is halved, the truncation error is approximately halved. We write this as \( O(h) \).

Worked Example 6.4.1

Forward difference estimate for \( f'(1) \) with \( f(x) = e^x \), \( h = 0.1 \)

Problem Estimate \( f'(1) \) for \( f(x) = e^x \) using the forward difference formula with \( h = 0.1 \). Compare with the exact value.
Think The exact derivative is \( f'(x) = e^x \), so \( f'(1) = e \approx 2.71828 \). We need \( f(1) \) and \( f(1.1) \).
Method Apply the forward difference: \( f'(1) \approx \dfrac{f(1.1) - f(1)}{0.1} \)
Compute \( f(1) = e^1 \approx 2.71828 \), \( f(1.1) = e^{1.1} \approx 3.00417 \)
\( \dfrac{3.00417 - 2.71828}{0.1} = \dfrac{0.28589}{0.1} = 2.8589 \)
Check Absolute error: \( |2.8589 - 2.71828| = 0.1406 \), about \( 5.2\% \).
Interpret With \( h = 0.1 \), the forward difference gives a reasonable but rough estimate. Reducing \( h \) to \( 0.01 \) reduces the error to roughly \( 0.5\% \), consistent with first-order accuracy.

The table below shows how the error decreases as \( h \) decreases — approximately by a factor of 10 each time \( h \) is divided by 10, confirming first-order accuracy.

\( h \)	Forward Difference Estimate	Absolute Error	Error Ratio
0.1	2.85890	0.14062	—
0.05	2.78712	0.06884	≈ 2.0
0.01	2.73191	0.01363	≈ 10.2
0.001	2.71964	0.00136	≈ 10.0

Section 6.5

Backward Difference Formula

f'(x) \approx \frac{f(x) - f(x-h)}{h}

Using the Taylor expansion of \( f(x-h) \) around \( x \):

f(x-h) = f(x) - h f'(x) + \frac{h^2}{2} f''(x) - \frac{h^3}{6} f'''(x) + \cdots

Rearranging:

\frac{f(x) - f(x-h)}{h} = f'(x) - \frac{h}{2} f''(x) + O(h^2) = f'(x) + O(h)

The backward difference formula is also first-order accurate. The error term has opposite sign to the forward difference: where the forward formula overshoots, the backward formula undershoots (when \( f'' > 0 \)). The backward formula is most useful when \( x \) is at the right end of a data table and \( f(x+h) \) is not available.

Worked Example 6.5.1

Backward difference at a data table endpoint

Problem A data table records \( f(x) \) at \( x = 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 \). Estimate \( f'(1.0) \) using the backward difference with \( h = 0.2 \), given \( f(0.8) = 3.41 \) and \( f(1.0) = 3.95 \).
Think Since \( x = 1.0 \) is the last point, we cannot use a forward difference. The backward formula is the natural choice.
Compute \( f'(1.0) \approx \dfrac{f(1.0) - f(0.8)}{0.2} = \dfrac{3.95 - 3.41}{0.2} = \dfrac{0.54}{0.2} = 2.70 \)
Interpret We accept \( f'(1.0) \approx 2.70 \) with error \( O(0.2) \). A smaller spacing \( h \) would give a more accurate estimate.

Section 6.6

Central Difference Formula

f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}

The central difference formula uses function values symmetrically placed around \( x \). Expanding both:

f(x+h) = f(x) + h f'(x) + \frac{h^2}{2} f''(x) + \frac{h^3}{6} f'''(x) + \cdots \] \[ f(x-h) = f(x) - h f'(x) + \frac{h^2}{2} f''(x) - \frac{h^3}{6} f'''(x) + \cdots

Subtracting the second from the first, all even-order terms cancel:

f(x+h) - f(x-h) = 2h f'(x) + \frac{h^3}{3} f'''(x) + \cdots

Dividing by \( 2h \):

\frac{f(x+h) - f(x-h)}{2h} = f'(x) + \frac{h^2}{6} f'''(x) + O(h^4) = f'(x) + O(h^2)

The central difference formula is second-order accurate. The leading error term is proportional to \( h^2 \), not \( h \). When \( h \) is halved, the truncation error decreases by a factor of 4. The key reason is symmetry: the \( h^2 \) terms in the two expansions are identical and cancel in the subtraction, leaving the first surviving error term at \( h^3 \) — giving \( O(h^2) \) after dividing by \( 2h \).

Worked Example 6.6.1

Comparing all three formulas for \( f'(1) \) with \( f(x) = e^x \), \( h = 0.1 \)

Values needed \( f(0.9) = e^{0.9} \approx 2.45960 \), \( f(1.0) = e^{1.0} \approx 2.71828 \), \( f(1.1) = e^{1.1} \approx 3.00417 \)
Central difference \( \dfrac{f(1.1) - f(0.9)}{2(0.1)} = \dfrac{3.00417 - 2.45960}{0.2} = \dfrac{0.54457}{0.2} = 2.72285 \)
Check Exact value: \( e \approx 2.71828 \). Absolute error of central difference: \( |2.72285 - 2.71828| = 0.00457 \).
Interpret The central difference error is about 30 times smaller than the forward difference error at the same \( h \). This is the power of second-order accuracy.

Comparison of All Three Formulas at \( h = 0.1 \)

Formula	Estimate	Absolute Error	Order
Forward difference	2.85890	0.14062	\( O(h) \)
Backward difference	2.62712	0.09116	\( O(h) \)
Central difference	2.72285	0.00457	\( O(h^2) \)

Best Practice

Whenever both \( f(x+h) \) and \( f(x-h) \) are available, use the central difference formula. It provides dramatically better accuracy at no additional computational cost — both formulas require exactly two function evaluations.

Section 6.7

Higher-Order Difference Approximations

Second Derivative by Central Differences

The second derivative \( f''(x) \) measures the rate of change of the rate of change. Adding the two Taylor expansions rather than subtracting them:

f(x+h) + f(x-h) = 2f(x) + h^2 f''(x) + \frac{h^4}{12} f^{(4)}(x) + \cdots

Solving for \( f''(x) \):

f''(x) \approx \frac{f(x+h) - 2f(x) + f(x-h)}{h^2}

This is the central difference formula for the second derivative. Its truncation error is \( O(h^2) \): it uses three points and is second-order accurate. This formula appears constantly in numerical ODE and PDE solvers.

Worked Example 6.7.1

Second derivative estimate for \( f(x) = \sin(x) \) at \( x = 1 \), \( h = 0.1 \)

Values \( f(0.9) = \sin(0.9) \approx 0.78333 \), \( f(1.0) \approx 0.84147 \), \( f(1.1) \approx 0.89121 \)
Compute \( \dfrac{0.89121 - 2(0.84147) + 0.78333}{(0.1)^2} = \dfrac{-0.00840}{0.01} = -0.840 \)
Check Exact: \( f''(x) = -\sin(x) \), so \( f''(1) \approx -0.84147 \). Absolute error ≈ \( 0.0015 \), consistent with \( O(h^2) \) accuracy.

Higher-Order Formula for the First Derivative

Using four evaluation points, we can construct a fourth-order approximation to \( f'(x) \) by combining two central difference formulas of different step sizes and canceling the \( h^2 \) error term:

f'(x) \approx \frac{-f(x+2h) + 8f(x+h) - 8f(x-h) + f(x-2h)}{12h} \quad \text{error } O(h^4)

Halving \( h \) reduces the truncation error by a factor of 16. This formula is standard in scientific computing libraries and is closely related to the general technique of Richardson extrapolation.

Formula Reference

Core Finite Difference Formulas

Forward diff. \( \displaystyle f'(x) \approx \frac{f(x+h) - f(x)}{h} \) \( O(h) \)

Backward diff. \( \displaystyle f'(x) \approx \frac{f(x) - f(x-h)}{h} \) \( O(h) \)

Central diff. \( \displaystyle f'(x) \approx \frac{f(x+h) - f(x-h)}{2h} \) \( O(h^2) \)

2nd deriv. \( \displaystyle f''(x) \approx \frac{f(x+h) - 2f(x) + f(x-h)}{h^2} \) \( O(h^2) \)

4-point diff. \( \displaystyle f'(x) \approx \frac{-f(x+2h) + 8f(x+h) - 8f(x-h) + f(x-2h)}{12h} \) \( O(h^4) \)

MGU Library Connection

Richardson extrapolation is a general technique for improving finite difference accuracy by combining results at different step sizes. It appears in the MGU Numerical Analysis reference and the Scientific Computing guide.

Section 6.8

Taylor Series and Error Terms

The Taylor series is the foundation of all finite difference error analysis. The process is systematic: expand each function value in a Taylor series around the differentiation point, form the finite difference combination, and identify which terms cancel and which survive as the leading error.

For the central difference formula, the formal error statement is:

\frac{f(x+h) - f(x-h)}{2h} = f'(x) + \frac{h^2}{6} f'''(x) + O(h^4)

If we know that \( |f'''(x)| \leq M \) on an interval, then the absolute truncation error satisfies:

\left| \frac{f(x+h) - f(x-h)}{2h} - f'(x) \right| \leq \frac{h^2}{6} M + O(h^4)

Worked Example 6.8.1

Bounding the truncation error for \( f(x) = e^x \) at \( x = 1 \), \( h = 0.1 \)

Think For \( f(x) = e^x \), all derivatives equal \( e^x \). Since \( e^{1.1} \approx 3.004 \), we have \( |f'''(\xi)| \leq 3.004 \) for \( \xi \in (0.9, 1.1) \).
Bound \( \text{error} \leq \dfrac{(0.1)^2}{6} \cdot 3.004 = \dfrac{0.01}{6} \cdot 3.004 \approx 0.00501 \)
Check The actual error was \( 0.00457 \), which is less than \( 0.00501 \). The bound is tight and reliable.

Section 6.9

Step Size and Roundoff Error

The analysis so far assumed that function values can be computed exactly. In practice, every floating-point computation introduces a small roundoff error. This changes the picture when \( h \) is very small.

Two Sources of Error in Numerical Differentiation

Truncation error arises from replacing the limit with a finite difference. It is approximately \( \frac{h^2}{6}|f'''(x)| \) for the central difference formula. It decreases as \( h \) decreases.

Roundoff error arises because each function value carries floating-point error. After dividing by \( 2h \), the roundoff contribution is approximately \( \frac{\epsilon_{\text{mach}} |f(x)|}{h} \). It increases as \( h \) decreases.

The total error in the central difference formula is approximately:

\text{total error} \approx \frac{h^2}{6} |f'''(x)| + \frac{\epsilon_{\text{mach}} |f(x)|}{h}

This sum has a minimum at an optimal step size. Setting the derivative with respect to \( h \) equal to zero gives:

h_{\text{opt}} \approx \left( \frac{3 \,\epsilon_{\text{mach}} |f(x)|}{|f'''(x)|} \right)^{1/3}

For double-precision arithmetic, \( \epsilon_{\text{mach}} \approx 10^{-16} \). With \( |f(x)| \approx |f'''(x)| \approx 1 \):

h_{\text{opt}} \approx (3 \times 10^{-16})^{1/3} \approx 10^{-5}

For typical smooth functions in double precision, the optimal step size for the central difference formula is around \( h = 10^{-5} \).

Worked Example 6.9.1

Observing the step size effect: central difference for \( f'(1) \), \( f(x) = e^x \)

Table

\( h \)	Estimate	Absolute Error
\( 0.1 \)	2.72285	\( 4.57 \times 10^{-3} \)
\( 0.01 \)	2.71832	\( 4.0 \times 10^{-5} \)
\( 0.001 \)	2.71828	\( \approx 4 \times 10^{-7} \)
\( 10^{-5} \)	2.71828	\( \approx 10^{-10} \) (near optimal)
\( 10^{-7} \)	2.71835	\( \approx 7 \times 10^{-5} \) (roundoff dominates)
\( 10^{-9} \)	2.71892	\( \approx 6 \times 10^{-4} \) (worse still)

Interpret The error decreases as \( h \) shrinks from 0.1 toward \( 10^{-5} \), then increases as roundoff takes over. The minimum occurs near the predicted optimal step size.

Critical Warning

Never assume that a smaller step size always gives a more accurate numerical derivative. Below the optimal step size, roundoff error dominates and the estimate deteriorates. In practice, step sizes of \( h \approx 10^{-5} \) to \( 10^{-6} \) are typical for central differences in double precision. Do not use \( h = 10^{-12} \) or smaller without expecting significant degradation.

Section 6.10

Differentiating Interpolating Polynomials

Another approach to numerical differentiation is through interpolation. If we have data at \( n+1 \) equally spaced points, we can construct an interpolating polynomial through those points (as in Chapter 4) and differentiate the polynomial exactly.

For three equally spaced points \( x_0 \), \( x_1 = x_0 + h \), \( x_2 = x_0 + 2h \), differentiating the interpolating quadratic polynomial gives:

f'(x_0) \approx \frac{-3f(x_0) + 4f(x_1) - f(x_2)}{2h} \quad \text{(left endpoint, } O(h^2)\text{)} \] \[ f'(x_1) \approx \frac{f(x_2) - f(x_0)}{2h} \quad \text{(midpoint, central difference)} \] \[ f'(x_2) \approx \frac{f(x_0) - 4f(x_1) + 3f(x_2)}{2h} \quad \text{(right endpoint, } O(h^2)\text{)}

The left and right endpoint formulas are called three-point endpoint formulas. They provide second-order accuracy at a boundary where the central difference is unavailable — they use only data already in the table.

Worked Example 6.10.1

Three-point endpoint formula with \( h = 0.5 \)

Data \( f(0) = 1.000 \), \( f(0.5) = 1.649 \), \( f(1.0) = 2.718 \) (values of \( e^x \)). Estimate \( f'(0) \).
Compute \( f'(0) \approx \dfrac{-3(1.000) + 4(1.649) - (2.718)}{2(0.5)} = \dfrac{-3.000 + 6.596 - 2.718}{1.0} = \dfrac{0.878}{1.0} = 0.878 \)
Check Exact value: \( f'(0) = e^0 = 1.000 \). Absolute error: \( |0.878 - 1.000| = 0.122 \).
Interpret The error is \( O(h^2) = O(0.25) \), which is large because \( h = 0.5 \) is large. A smaller spacing would reduce the error substantially.

Section 6.11

Numerical Differentiation with Noisy Data

When data comes from physical measurements, it always contains noise. Numerical differentiation of noisy data is one of the most challenging problems in applied mathematics, because differentiation amplifies high-frequency variation.

If a signal \( f(x) \) contains a noise component \( \eta(x) \), the central difference formula gives:

\frac{f(x+h) - f(x-h)}{2h} = f'_{\text{true}}(x) + O(h^2) + \frac{\eta(x+h) - \eta(x-h)}{2h}

If the noise amplitude is \( \delta \), the noise contribution to the derivative estimate can be as large as \( \delta / h \). As \( h \) decreases, the noise contribution grows. For clean data, smaller \( h \) is better (up to the roundoff limit). For noisy data, smaller \( h \) is worse.

Worked Example 6.11.1

Noise contribution to a central difference estimate

Setup True function: \( f(x) = \sin(x) \), so \( f'(1) = \cos(1) \approx 0.5403 \). Measurement noise: amplitude \( \pm 0.01 \). Step size: \( h = 0.01 \).
Noise bound Noise in \( f(x+h) - f(x-h) \) can be at most \( 2 \times 0.01 = 0.02 \). After dividing by \( 2h = 0.02 \), the noise contribution is \( 0.02 / 0.02 = 1.00 \).
Interpret The true derivative is about \( 0.54 \), but the noise alone could contribute an error as large as \( 1.00 \) — nearly twice the true value. The estimate is completely unreliable at this step size.
Remedy Using \( h = 0.5 \) instead reduces the noise contribution to \( 0.02 / 1.0 = 0.02 \), about \( 4\% \) of the true derivative — much more manageable.

Strategies for Noisy Data

Larger Step Size

Accept Some Truncation Error

Use a larger \( h \) to reduce noise amplification, accepting more truncation error in exchange. Practical when the noise level is known.

Smooth First

Filter Before Differentiating

Apply a moving average, Gaussian filter, or other smoother to reduce noise in the data before applying finite differences.

Fit and Differentiate

Differentiate the Fitted Model

Fit a polynomial or spline to the noisy data (Chapter 5), then differentiate the fitted model analytically. The fit absorbs noise; the model derivative is smooth.

Reliability Note

Before differentiating measured data, always ask: how large is the noise amplitude \( \delta \) compared to the step size \( h \)? If \( \delta / h \) is comparable to the expected derivative, numerical differentiation is unreliable and smoothing is required first.

Section 6.12

Numerical Derivatives in Science and Engineering

Numerical differentiation appears throughout applied mathematics and the sciences. The four examples below illustrate how the same finite difference ideas appear in very different contexts.

Velocity from GPS Position Data

A GPS device records position at one-second intervals. Applying central differences at interior time points and three-point endpoint formulas at the first and last points gives a complete velocity profile. With position accuracy \( \pm 1 \) meter and step size \( h = 1 \) second, the velocity noise is approximately \( 1 \) m/s — about \( 3\% \) of a typical vehicle speed of 30 m/s.

Edge Detection in Digital Images

Edges in a digital image correspond to rapid changes in pixel intensity. The gradient magnitude at each pixel is estimated by finite differences of the intensity values. The Sobel operator — a standard edge detection algorithm — is essentially a pair of finite difference formulas applied to a digital image grid.

Finite Difference PDE Methods

Engineering simulations of heat transfer, fluid flow, and structural stress are governed by partial differential equations involving spatial derivatives. Finite difference methods replace these derivatives with the central difference approximations developed in this chapter, turning differential equations into algebraic systems that can be solved numerically. We will develop this fully in Chapter 13.

Option Sensitivity in Finance

The delta of a financial option — how much its value changes per unit change in the underlying asset price — can be estimated by central difference:

\Delta \approx \frac{V(S+h) - V(S-h)}{2h}

where \( V(S) \) is the option value and \( S \) is the current stock price. The same formula applies to estimate other Greeks (Gamma, Vega, Rho) by varying the appropriate parameter.

Section 6.13

Common Numerical Differentiation Mistakes

Mistake 1

"A smaller step size always gives a more accurate derivative."

Below the optimal step size (roughly \( 10^{-5} \) for central differences in double precision), roundoff error dominates and the estimate deteriorates. The error is U-shaped in \( h \) — it decreases, reaches a minimum, then increases. Always verify the estimate is stable before using a very small \( h \).

Mistake 2

"When both \( f(x+h) \) and \( f(x-h) \) are available, I can use the forward formula to save work."

The forward and central difference formulas both require exactly two function evaluations. The central difference provides second-order accuracy versus first-order for the forward formula. There is no tradeoff — always use the central difference when symmetric data is available.

Mistake 3

"I can differentiate noisy sensor data with a small step size for better accuracy."

With noisy data, the noise contribution to the derivative estimate is \( \delta / h \), which grows as \( h \) shrinks. A smaller step size makes things worse. Increase \( h \) or smooth the data before differentiating.

Mistake 4

"A second-order formula is always more accurate than a first-order formula."

Order describes how accuracy scales with \( h \), not the absolute error at a specific \( h \). A first-order formula with very small \( h \) may outperform a second-order formula with large \( h \) in particular cases. Compare actual error estimates, not just order labels.

Mistake 5

"Finite difference formulas work everywhere."

Standard finite difference formulas require \( f \) to be sufficiently smooth — high-order derivatives must be bounded. Near discontinuities, corners, or rapid oscillations, the formulas may give misleading or meaningless results. Always verify function smoothness at the point of differentiation.

Mistake 6

"The standard formulas work for irregularly spaced data."

The formulas in this chapter assume equal spacing \( h \) between evaluation points. Irregularly spaced data requires formulas derived specifically for the given spacing, typically through interpolation. Using the equal-spacing formulas on unequal data introduces additional systematic error.

Section 6.14

Preparing for Numerical Integration

Numerical differentiation estimates rates of change from function values. Numerical integration, the subject of Chapter 7, estimates accumulated totals from function values. The two are mathematically inverse operations, but their numerical properties differ in one important way.

Numerical differentiation is noise-amplifying: small errors in function values produce proportionally larger errors in derivative estimates, especially with small \( h \). This makes numerical differentiation a delicate operation requiring careful step size selection.

Numerical integration is noise-averaging: the errors in function values tend to cancel when summed, making integration far more stable and forgiving than differentiation. This is why numerical integration is generally easier to perform reliably.

The Taylor series framework used throughout this chapter — expanding function values around a central point and analyzing error terms — will reappear in identical form in Chapter 7. The methods of this chapter and the next are mirror images of each other: the same tools, the same error analysis, opposite mathematical operations.

Bridge to Chapter 7: Numerical Integration

Chapter 7 develops the rectangle rules, midpoint rule, trapezoidal rule, and Simpson's rule. The Taylor series error analysis is structurally identical to Chapter 6 — the same expansions, the same order analysis, the same step size tradeoffs. The key difference is that integration sums function values rather than subtracting them, so errors tend to cancel rather than amplify.

Bridge to Chapters 12–13: ODE and PDE Solvers

Every formula in this chapter reappears in the numerical solution of differential equations. Euler's method for ODEs is the forward difference formula used as a predictor. The second-derivative central difference formula discretizes the Laplacian in PDE methods. Students who understand Chapter 6 will recognize the structure of every time-stepping and spatial discretization formula in the later chapters.

Chapter 6 — Back Matter

Chapter Summary

Section 6.1 Numerical differentiation is needed whenever symbolic derivatives are unavailable — in experimental data, simulation output, or black-box computation pipelines. It replaces the limiting operation in the derivative definition with a computable finite approximation.
Section 6.2 When only tabular data is available, average rates of change over small intervals approximate instantaneous rates of change. This intuition underlies every finite difference formula.
Section 6.3 Difference quotients — forward, backward, and central — are the three natural ways to approximate \( f'(x) \) from nearby function values. They differ in accuracy, required points, and error behavior.
Sections 6.4–6.5 The forward and backward difference formulas are first-order accurate, with error proportional to \( h \). Both require only two function evaluations. The backward formula is used at right-end data boundaries.
Section 6.6 The central difference formula is second-order accurate, with error proportional to \( h^2 \). The even-order Taylor terms cancel by symmetry, leaving an error 10–30 times smaller than the first-order formulas at the same step size. Always prefer central differences when both neighboring points are available.
Section 6.7 The second derivative can be approximated by \( [f(x+h) - 2f(x) + f(x-h)] / h^2 \) with error \( O(h^2) \). A four-point formula achieves \( O(h^4) \) for the first derivative by canceling the \( h^2 \) error term.
Section 6.8 Taylor series analysis gives explicit bounds on truncation error. For the central difference formula, the bound is \( h^2 M / 6 \) where \( M = \max |f'''| \) on the relevant interval.
Section 6.9 Two error sources compete: truncation error (decreases with \( h \)) and roundoff error (increases with \( h \)). The total error is minimized at the optimal step size \( h_{\text{opt}} \approx 10^{-5} \) for central differences in double precision. Smaller \( h \) below this optimum worsens results.
Section 6.10 Differentiating the polynomial interpolant through three equally spaced points yields second-order endpoint formulas that estimate derivatives at data table boundaries with the same accuracy as the central difference at interior points.
Section 6.11 Noise is amplified by differentiation. The noise contribution to a central difference estimate is \( \delta / h \), growing as \( h \) decreases. For noisy data, use a larger step size, smooth the data first, or fit a smooth model and differentiate the fit.
Sections 6.12–6.14 Numerical differentiation appears in GPS velocity estimation, image processing, finite difference PDE solvers, and financial risk analysis. The six common mistakes center on step size, formula choice, noise management, and the assumptions of equal spacing and function smoothness.

Key Terms Review

numerical differentiation

Estimation of a derivative from discrete function values rather than by symbolic computation.

finite difference

A discrete ratio of function value differences that approximates a derivative.

forward difference

\( [f(x+h)-f(x)]/h \); first-order accurate; uses one step ahead of \( x \).

backward difference

\( [f(x)-f(x-h)]/h \); first-order accurate; useful at right-end data boundaries.

central difference

\( [f(x+h)-f(x-h)]/(2h) \); second-order accurate; preferred when symmetric data is available.

step size \( h \)

The spacing between evaluation points in a finite difference formula; must be chosen carefully.

truncation error

Error from replacing a limit with a finite approximation; decreases as \( h \to 0 \).

roundoff error

Error from limited floating-point precision; increases as \( h \to 0 \) in differentiation.

order of accuracy

The exponent \( p \) such that truncation error is \( O(h^p) \); describes rate of error decrease.

optimal step size

The step size that minimizes total error (truncation + roundoff); approximately \( 10^{-5} \) for central differences in double precision.

noise amplification

The tendency of differentiation to magnify measurement errors, especially with small \( h \).

Richardson extrapolation

A technique for improving accuracy by combining estimates at different step sizes to cancel leading error terms.

ReviewConcept Review Questions

What is numerical differentiation, and when is it needed instead of symbolic differentiation? Give three distinct situations.
State the forward, backward, and central difference formulas for the first derivative. What is the order of accuracy of each?
How does Taylor series analysis reveal the truncation error of a finite difference formula? Illustrate the argument for the central difference formula, identifying which terms cancel and why.
Why is the central difference formula more accurate than the forward or backward difference formula for the same step size? What property of the Taylor expansion is responsible?
Why does decreasing \( h \) toward zero eventually make a numerical derivative estimate worse rather than better? What two effects are in competition?
What is the optimal step size for the central difference formula in double-precision arithmetic, and what principle determines it?
Why is numerical differentiation especially difficult when the data is noisy? State the formula for the noise contribution to the error and explain how it depends on \( h \).
State the central difference formula for the second derivative. What is its order of accuracy, and how is it derived?
When would you use a three-point endpoint formula rather than the central difference formula? What order of accuracy does the endpoint formula achieve?
How does numerical differentiation connect to numerical integration (Chapter 7), ODE solvers (Chapter 12), and PDE methods (Chapter 13)?

Skill PracticeComputation and Analysis

Concept Check

Without computing, explain why the central difference formula is more accurate than the forward difference formula for the same step size.
A student computes a central difference estimate with \( h = 10^{-10} \) and obtains a less accurate result than with \( h = 10^{-5} \). Explain why this happens.
At which points in a uniformly spaced data table can you apply the central difference formula? What formula would you use at the endpoints?
State two situations where numerical differentiation of data is unreliable. For each, state what you would do instead.

Skill Practice

Compute the forward difference approximation to \( f'(0) \) for \( f(x) = \cos(x) \) with \( h = 0.2 \). State the absolute and relative error.
Compute the backward difference approximation to \( f'(2) \) for \( f(x) = \ln(x) \) with \( h = 0.1 \). State the absolute error.
Compute the central difference approximation to \( f'(1) \) for \( f(x) = x^3 - 2x \) with \( h = 0.1 \). Compare with the exact value \( f'(1) = 1 \).
Estimate \( f''(0) \) for \( f(x) = e^x \) using the second-derivative central difference formula with \( h = 0.1 \). State the exact value and the error.
Given \( f(0) = 0 \), \( f(0.5) = 0.479 \), \( f(1.0) = 0.841 \) (approximate values of \( \sin(x) \)), estimate \( f'(0.5) \) using all three first-derivative formulas. Compare the estimates.
Using the same data as Problem 9, estimate \( f'(0) \) using the three-point left-endpoint formula. What is the additional data requirement compared to the central difference at \( x = 0.5 \)?
For \( f(x) = e^{2x} \), compute the central difference approximation to \( f'(0) \) with \( h = 0.05, 0.01, 0.005 \). Verify that the error decreases by a factor of approximately 4 each time \( h \) is halved.
Compute the four-point approximation to \( f'(0) \) for \( f(x) = e^x \) using the formula \( \frac{-f(2h) + 8f(h) - 8f(-h) + f(-2h)}{12h} \) with \( h = 0.1 \). Compare the accuracy with the central difference at the same \( h \).

Algorithm PracticePseudocode and Procedure

Algorithm Reference

Numerical Derivative from Uniformly Spaced Data

PurposeEstimate \( f'(x_i) \) at each point in a uniformly spaced data set using second-order accurate formulas

InputsArray \( y[0..n] \) of function values; step size \( h > 0 \)

Steps d[0] ← (−3·y[0] + 4·y[1] − y[2]) / (2h) {left endpoint formula}
for i from 1 to n−1:
d[i] ← (y[i+1] − y[i−1]) / (2h) {central difference}
d[n] ← (y[n−2] − 4·y[n−1] + 3·y[n]) / (2h) {right endpoint formula}

OutputArray \( d[0..n] \) of derivative estimates

ReliabilityAll estimates are second-order accurate. For noisy data, check whether \( \delta/h \) is comparable to the expected derivative; if so, increase \( h \) or smooth the data first.

Using the algorithm above, trace through the computation for the data set \( y = [1.00, 1.22, 1.49, 1.82, 2.23] \) with \( h = 0.5 \). Write out all five derivative estimates.
Describe how you would modify the algorithm to flag points where the noise contribution \( \delta/h \) exceeds a user-specified tolerance. What additional input would you require?
Write pseudocode for an algorithm that computes the second derivative at each interior point of a uniformly spaced data set using the three-point second-derivative formula.

InterpretationComputational Interpretation Problems

A researcher uses the forward difference formula to estimate \( f'(x) \) with \( h = 0.01 \) and obtains 3.456. She then uses \( h = 0.001 \) and obtains 3.312. What does the change suggest about which estimate is more accurate?
The central difference estimates of \( f'(1) \) with \( h = 0.1, 0.05, 0.025 \) are 2.3421, 2.3367, 2.3354. Is this behavior consistent with second-order accuracy? Explain how you check this.
A student computes central difference estimates with \( h = 10^{-5}, 10^{-7}, 10^{-9}, 10^{-11} \) and finds the estimates first improve and then deteriorate. At approximately which step size does the improvement stop, and what is responsible for the deterioration?
Two students estimate \( f'(2) \) for the same function. Student A uses the forward difference with \( h = 0.001 \) and gets 4.128. Student B uses the central difference with \( h = 0.1 \) and gets 4.103. Whose estimate is likely more accurate, and how would you verify this without knowing the true value?

ApplicationsReal-World Numerical Differentiation

GPS Velocity. A GPS receiver records positions (in meters) at one-second intervals: \( s(0) = 0 \), \( s(1) = 12.3 \), \( s(2) = 24.7 \), \( s(3) = 37.1 \), \( s(4) = 49.5 \). Estimate the velocity at each time using central differences at interior points and endpoint formulas at the boundaries.
Temperature Rate of Change. A sensor records temperatures at 30-minute intervals: \( T(0) = 15.2 \)°C, \( T(0.5) = 16.8 \)°C, \( T(1.0) = 18.9 \)°C, \( T(1.5) = 21.4 \)°C, \( T(2.0) = 23.0 \)°C (time in hours). Estimate the rate of temperature change at each time point.
Population Growth Rate. A study records population (in thousands) at five-year intervals: \( P(0) = 100 \), \( P(5) = 118 \), \( P(10) = 139 \), \( P(15) = 164 \), \( P(20) = 193 \). Estimate the growth rate (thousands per year) at each time.
Option Delta. An option pricing model gives values \( V(48) = 3.82 \), \( V(50) = 4.56 \), \( V(52) = 5.34 \). Estimate the option delta \( \Delta = dV/dS \) at \( S = 50 \) using the central difference formula.
Signal Derivative. A digital signal has values at sample points \( f(0) = 1.00 \), \( f(1) = 2.14 \), \( f(2) = 3.57 \), \( f(3) = 4.98 \), \( f(4) = 6.22 \) (uniform spacing). Estimate the derivative at each sample point and identify the point where the rate of change appears to be decreasing.
Noise Assessment. For the GPS problem in Problem 16, suppose the GPS device has position accuracy \( \pm 1 \) meter. Estimate the noise contribution to the velocity error at each interior time point. Is the error small enough to be practically useful?

Error AnalysisQuantitative Error Problems

For the forward difference formula applied to \( f(x) = e^x \) at \( x = 0 \): (a) Compute the absolute error for \( h = 0.5, 0.1, 0.05, 0.01 \). (b) Verify that the error is approximately proportional to \( h \). (c) Estimate the step size \( h \) that would give absolute error less than \( 0.001 \).
For the central difference formula applied to \( f(x) = \sin(x) \) at \( x = \pi/4 \): (a) Compute the absolute error for \( h = 0.5, 0.1, 0.05, 0.01 \). (b) Verify that the error is approximately proportional to \( h^2 \). (c) Bound the truncation error using \( h^2 |f'''(x)| / 6 \) and compare with the actual error.
A function satisfies \( |f'''(x)| \leq 5 \) on the interval \( [0, 2] \). What step size \( h \) is needed so that the central difference formula has truncation error less than \( 0.001 \)?
Suppose function values are computed with relative error \( 10^{-8} \) and \( |f(x)| \approx 1 \). Estimate the optimal step size for the central difference formula and the corresponding minimum total error.
A student uses the forward difference formula with \( h = 0.001 \) to estimate \( f'(x) \) from data with noise amplitude \( \pm 0.005 \). Estimate the noise contribution to the derivative error and determine whether the estimate is likely to be useful.
For \( f(x) = e^{2x} \), bound the truncation error of the central difference approximation to \( f'(0) \) with \( h = 0.1 \). Compare with the actual error.

Chapter 6 Checkpoint

Complete without notes. Assess readiness to proceed to Chapter 7.

Part A — Formulas and Definitions

I can write the forward, backward, and central difference formulas and state the order of accuracy of each.
I can write the central difference formula for the second derivative and state its order of accuracy.
I can state the two types of error in numerical differentiation and describe how each depends on step size.
I can state the approximate optimal step size for the central difference formula in double precision.

Part B — Computation

Given \( f(x) = x^2 + 3x \), I can estimate \( f'(2) \) using the central difference formula with \( h = 0.1 \) and compare with the exact value \( f'(2) = 7 \).
Given \( f(x) = \ln(x) \), I can estimate \( f''(1) \) using the second-derivative formula with \( h = 0.1 \) and compare with the exact value \( f''(1) = -1 \).
Given three data points, I can apply the three-point endpoint formula to estimate the derivative at a boundary point.

Part C — Error and Interpretation

I can bound the truncation error of a central difference estimate given a bound on the third derivative.
I can explain why numerical differentiation of noisy data with a small step size is unreliable and state what to do instead.
Given two central difference estimates at different step sizes, I can assess the apparent order of accuracy.

Part D — Checkpoint Problems

C6.1 Estimate \( f'(1) \) for \( f(x) = e^x \) using the central difference formula with \( h = 0.05 \). State the absolute error.
C6.2 Estimate \( f''(\pi/2) \) for \( f(x) = \sin(x) \) using the second-derivative formula with \( h = 0.1 \). State the exact value and the error.
C6.3 For \( f(x) = \cos(x) \), bound the truncation error of the central difference formula at \( x = 0 \) with \( h = 0.2 \). (Hint: \( |f'''| = |\sin(x)| \leq 1 \).)
C6.4 A student estimates \( f'(1) \) as 4.327 using \( h = 0.1 \) and as 4.321 using \( h = 0.05 \). What does this tell us about the order of accuracy of the formula being used?
C6.5 Why is applying the central difference formula with \( h = 10^{-12} \) to a smooth function in double precision a poor choice? What step size would you recommend?

Preview — Chapter 7

Numerical Integration

Chapter 6 developed the tools for estimating rates of change from function values. Chapter 7 applies the same building blocks — function values at discrete points, Taylor series error analysis, and step size considerations — to the opposite problem: estimating accumulated totals. The rectangle rules, midpoint rule, trapezoidal rule, and Simpson's rule each replace a definite integral by a weighted sum of function values. The Taylor series framework of this chapter reappears in identical form to analyze their accuracy. The key difference from differentiation: integration is noise-averaging rather than noise-amplifying, making it a more forgiving and robust numerical operation.