The Central Idea
The derivative is defined by the limit
\[ f'(x) = \lim_{h\to0} \frac{f(x+h)-f(x)}{h}. \]Numerical differentiation replaces the limiting process with a small finite \(h\):
\[ \boxed{ f'(x) \approx \frac{f(x+h)-f(x)}{h} }. \]This is the forward-difference approximation.
Why Numerical Differentiation Is Needed
An analytic derivative may be unavailable or inconvenient when:
- the function is known only through measured data;
- the function is produced by a simulation;
- the symbolic derivative is complicated;
- only tabulated values are available;
- the derivative must be estimated inside a larger numerical algorithm.
Finite Differences
A finite difference compares function values at separated points.
Common first-difference forms are:
\[ f(x+h)-f(x), \] \[ f(x)-f(x-h), \]and
\[ f(x+h)-f(x-h). \]Dividing these differences by appropriate powers of \(h\) produces derivative approximations.
The Forward-Difference Formula
The first derivative can be approximated using
\[ \boxed{ f'(x) \approx \frac{f(x+h)-f(x)}{h} }. \]This formula uses the current point and a point one step forward.
Deriving Forward Difference from Taylor's Formula
Expand \(f(x+h)\):
\[ f(x+h) = f(x) + hf'(x) + \frac{h^2}{2}f''(x) + \frac{h^3}{6}f'''(x) +\cdots. \]Subtract \(f(x)\) and divide by \(h\):
\[ \frac{f(x+h)-f(x)}{h} = f'(x) + \frac{h}{2}f''(x) + O(h^2). \]Forward-Difference Truncation Error
Therefore,
\[ \boxed{ f'(x) = \frac{f(x+h)-f(x)}{h} +O(h) }. \]The leading truncation error is proportional to \(h\).
Forward difference is therefore a first-order method.
The Backward-Difference Formula
A backward estimate uses
\[ \boxed{ f'(x) \approx \frac{f(x)-f(x-h)}{h} }. \]It is useful when data to the right of \(x\) are unavailable.
Backward-Difference Error
Taylor expansion gives
\[ f(x-h) = f(x) - hf'(x) + \frac{h^2}{2}f''(x) - \frac{h^3}{6}f'''(x) +\cdots. \]Rearranging,
\[ \boxed{ f'(x) = \frac{f(x)-f(x-h)}{h} +O(h) }. \]Backward difference is also first-order accurate.
The Central-Difference Formula
Use values on both sides of \(x\):
\[ \boxed{ f'(x) \approx \frac{f(x+h)-f(x-h)}{2h} }. \]Why Central Difference Is More Accurate
Subtract the Taylor expansions of \(f(x+h)\) and \(f(x-h)\):
\[ f(x+h)-f(x-h) = 2hf'(x) + \frac{h^3}{3}f'''(x) +\cdots. \]Divide by \(2h\):
\[ \frac{f(x+h)-f(x-h)}{2h} = f'(x) + \frac{h^2}{6}f'''(x) +O(h^4). \]Central-Difference Truncation Error
Thus,
\[ \boxed{ f'(x) = \frac{f(x+h)-f(x-h)}{2h} +O(h^2) }. \]Central difference is second-order accurate. Halving \(h\) reduces the leading truncation error by approximately a factor of \(4\).
Forward, Backward, and Central Comparison
\[ \text{Forward:}\qquad \frac{f(x+h)-f(x)}{h}, \] \[ \text{Backward:}\qquad \frac{f(x)-f(x-h)}{h}, \] \[ \text{Central:}\qquad \frac{f(x+h)-f(x-h)}{2h}. \]Their leading truncation orders are
\[ O(h),\qquad O(h),\qquad O(h^2). \]Example: Forward Difference
Let \(f(x)=x^2\). At \(x=2\) with \(h=0.1\),
\[ f'(2) \approx \frac{f(2.1)-f(2)}{0.1} = \frac{4.41-4}{0.1} = \boxed{4.1}. \]The exact derivative is \(4\).
Example: Central Difference
Again let \(f(x)=x^2\), \(x=2\), and \(h=0.1\).
\[ f'(2) \approx \frac{f(2.1)-f(1.9)}{0.2} = \frac{4.41-3.61}{0.2} = \boxed{4}. \]For a quadratic, the symmetric central formula recovers the derivative exactly.
One-Sided Differences at Boundaries
Central difference requires data on both sides of the evaluation point.
At a boundary, one side may not be available. Forward or backward formulas then become necessary.
A Higher-Order Forward Formula
\[ \boxed{ f'(x) \approx \frac{ -3f(x)+4f(x+h)-f(x+2h) }{2h} }. \]Its truncation error is
\[ O(h^2). \]A Higher-Order Backward Formula
\[ \boxed{ f'(x) \approx \frac{ 3f(x)-4f(x-h)+f(x-2h) }{2h} }. \]This is also second-order accurate.
A Five-Point Central Formula
\[ \boxed{ f'(x) \approx \frac{ f(x-2h)-8f(x-h)+8f(x+h)-f(x+2h) }{12h} }. \]Its truncation error is
\[ O(h^4). \]Second Derivatives
Adding the Taylor expansions of \(f(x+h)\) and \(f(x-h)\) gives
\[ f(x+h)+f(x-h) = 2f(x) + h^2f''(x) + O(h^4). \]Therefore,
\[ \boxed{ f''(x) \approx \frac{ f(x+h)-2f(x)+f(x-h) }{h^2} }. \]Second-Derivative Error
\[ \boxed{ f''(x) = \frac{ f(x+h)-2f(x)+f(x-h) }{h^2} +O(h^2) }. \]This formula becomes central to finite-difference methods for differential equations.
Finite-Difference Stencils
A stencil describes the pattern of sample points used by a finite-difference formula.
The central first derivative uses
\[ x-h,\qquad x+h, \]while the central second derivative uses
\[ x-h,\qquad x,\qquad x+h. \]Stencil Coefficients
The central first derivative has coefficient pattern
\[ \frac{1}{2h} [-1,0,1], \]while the central second derivative has
\[ \frac{1}{h^2} [1,-2,1]. \]Wider stencils produce higher-order formulas.
Deriving Finite Differences Systematically
- Expand nearby values with Taylor series.
- Assign coefficients to those expansions.
- Cancel unwanted derivative terms.
- Scale the remaining desired derivative term.
- Identify the first uncanceled term as the leading truncation error.
Order of Accuracy
A formula has order \(p\) when its leading truncation error behaves like
\[ \boxed{Ch^p}. \]Halving \(h\) then reduces that error by approximately
\[ 2^p. \]Richardson Extrapolation
Suppose
\[ D(h) = f'(x)+Ch^p+\text{higher-order terms}. \]Then
\[ D(h/2) = f'(x) + C\left(\frac{h}{2}\right)^p +\cdots. \]Combining the two can eliminate the leading error term.
Richardson Formula
\[ \boxed{ D_{\text{rich}} = \frac{ 2^pD(h/2)-D(h) }{ 2^p-1 } }. \]This raises the effective order when the assumed error expansion applies.
Example: Richardson with Central Differences
Central difference has \(p=2\), so
\[ \boxed{ D_{\text{rich}} = \frac{ 4D(h/2)-D(h) }{3} }. \]Why Smaller \(h\) Is Not Always Better
In exact mathematics, the derivative uses \(h\to0\).
In floating-point arithmetic, extremely small \(h\) makes
\[ f(x+h) \]and
\[ f(x) \]nearly equal, making subtraction vulnerable to cancellation.
Cancellation in Numerical Differentiation
The forward difference contains
\[ f(x+h)-f(x). \]For very small \(h\), these values may share most leading digits. Their subtraction can destroy significant information before division by \(h\).
Truncation Error Versus Rounding Error
A simple model is
\[ \boxed{ E(h) \approx C_1h^p + \frac{C_2u}{h} }, \]where the first term models truncation and the second models floating-point effects.
The Optimal Step-Size Idea
- large \(h\): truncation error dominates;
- moderate \(h\): total error decreases;
- very small \(h\): rounding error dominates.
The best numerical step is therefore usually neither very large nor extremely small.
Forward-Difference Step-Size Scale
For the simplified model
\[ E(h) \approx C_1h+\frac{C_2u}{h}, \]balancing the terms suggests a step scale proportional to
\[ \boxed{\sqrt{u}}. \]Central-Difference Step-Size Scale
For
\[ E(h) \approx C_1h^2+\frac{C_2u}{h}, \]balancing the terms suggests a scale proportional to
\[ \boxed{u^{1/3}}. \]Noise Amplification
If data errors have magnitude about \(\varepsilon\), a difference quotient can amplify them on the scale
\[ \boxed{ \frac{\varepsilon}{h} }. \]Numerical differentiation is therefore highly sensitive to noisy data.
Differentiation and Conditioning
Small high-frequency perturbations can have tiny amplitude while possessing large derivatives.
Two functions may therefore be close in function value but far apart in derivative value.
Differentiation is an inherently sensitive operation on noisy data.
Smoothing Before Differentiation
For noisy samples, first fit a smooth approximation
\[ p(x)\approx f(x) \]and then differentiate:
\[ \boxed{ f'(x)\approx p'(x) }. \]This connects numerical differentiation to the previous lessons on interpolation and approximation.
Differentiating an Interpolating Polynomial
If \(p_n\) interpolates nearby data, then
\[ \boxed{ f'(x)\approx p_n'(x) }. \]Finite-difference formulas can be interpreted as derivatives of local interpolating polynomials.
Unequally Spaced Data
When nodes are not equally spaced, standard uniform-grid formulas no longer apply directly.
Construct a local Lagrange or Newton interpolant using the actual node locations and differentiate that polynomial.
Higher Derivatives
Finite differences can approximate
\[ f''(x),\qquad f'''(x),\qquad f^{(4)}(x), \]and higher derivatives.
Higher derivative estimation becomes progressively more sensitive to noise and rounding.
Numerical Differentiation on a Grid
Let
\[ x_i=x_0+ih. \]Then
\[ \boxed{ f'(x_i) \approx \frac{f_{i+1}-f_{i-1}}{2h} } \]and
\[ \boxed{ f''(x_i) \approx \frac{f_{i+1}-2f_i+f_{i-1}}{h^2} }. \]Finite-Difference Matrices
Applying derivative formulas at many grid points produces a matrix operator:
\[ \boxed{ \mathbf{f}' \approx D\mathbf{f} }. \]This connects numerical differentiation to numerical linear algebra and later differential-equation discretizations.
Boundary Stencils
Interior points can use symmetric central formulas. Boundary points may require:
- forward differences;
- backward differences;
- higher-order one-sided formulas;
- ghost points in differential-equation schemes.
Complex-Step Differentiation
For analytic functions that support complex arithmetic,
\[ \boxed{ f'(x) \approx \frac{\operatorname{Im}(f(x+ih))}{h} }. \]This avoids subtracting two nearly equal real numbers.
Why Complex Step Avoids Cancellation
Expanding gives
\[ f(x+ih) = f(x) + ihf'(x) - \frac{h^2}{2}f''(x) - i\frac{h^3}{6}f'''(x) +\cdots. \]Therefore,
\[ \operatorname{Im}(f(x+ih)) = hf'(x)+O(h^3). \]Automatic Differentiation Is Different
Automatic differentiation does not estimate a derivative from a finite difference quotient.
It propagates derivative rules through the operations used to compute a function.
Symbolic, Automatic, and Numerical Differentiation
- symbolic differentiation: manipulates formulas;
- automatic differentiation: propagates derivative rules through a computation;
- numerical differentiation: estimates derivatives from sampled values.
Derivative Checks in Optimization
Finite differences are frequently used to validate analytic or automatically computed gradients.
A claimed derivative \(g(x)\) should agree approximately with
\[ \frac{f(x+h)-f(x-h)}{2h} \]over an appropriate range of \(h\).
Directional Derivative Checks
For \(f(\mathbf{x})\) and direction \(\mathbf{v}\),
\[ D_{\mathbf{v}}f(\mathbf{x}) = \nabla f(\mathbf{x})^T\mathbf{v}. \]Numerically,
\[ \boxed{ D_{\mathbf{v}}f(\mathbf{x}) \approx \frac{ f(\mathbf{x}+h\mathbf{v}) - f(\mathbf{x}-h\mathbf{v}) }{2h} }. \]Approximating a Gradient
\[ \boxed{ \frac{\partial f}{\partial x_i} \approx \frac{ f(\mathbf{x}+h\mathbf{e}_i) - f(\mathbf{x}-h\mathbf{e}_i) }{2h} }. \]Repeating this for each coordinate approximates the full gradient.
Cost of a Finite-Difference Gradient
For \(n\) variables, forward differences require roughly \(n+1\) function evaluations.
Central differences require roughly
\[ 2n \]evaluations.
When each function evaluation is expensive, derivative estimation can become a major computational cost.
Approximating the Hessian
\[ \boxed{ \frac{\partial^2f}{\partial x_i^2} \approx \frac{ f(\mathbf{x}+h\mathbf{e}_i) - 2f(\mathbf{x}) + f(\mathbf{x}-h\mathbf{e}_i) }{h^2} }. \]Mixed derivatives require multi-directional stencils.
Choosing \(h\) Empirically
Compute estimates with
\[ h,\qquad \frac{h}{2},\qquad \frac{h}{4},\ldots \]and watch how they change.
Improvement followed by stagnation or deterioration often signals the transition from truncation-dominated error to rounding- or noise-dominated error.
Convergence Tables
Tabulate
\[ D(h),\qquad D(h/2),\qquad D(h/4),\ldots \]and compare successive differences.
In the asymptotic regime, an order-\(p\) method should exhibit the expected reduction factor \(2^p\).
Observed Order of Accuracy
If the exact derivative is known for validation,
\[ \boxed{ p_{\text{obs}} = \log_2 \left( \frac{E(h)}{E(h/2)} \right) }. \]Values near the theoretical \(p\) support the expected convergence rate.
Derivative Estimation from Tables
Uniform tables support standard difference formulas directly.
For noisy data, wider local fits or smoothing methods may be more reliable than raw two-point differences.
Local Polynomial Differentiation
Fit a polynomial \(p\) to nearby samples and estimate
\[ \boxed{ f^{(k)}(x)\approx p^{(k)}(x) }. \]This unifies finite differences, interpolation, and local smoothing.
A Reliable First-Derivative Workflow
- Determine whether analytic or automatic differentiation is available.
- Inspect smoothness and data quality.
- Prefer central differences at interior points when appropriate.
- Use one-sided formulas at boundaries.
- Choose \(h\) to balance truncation and rounding.
- Repeat with several step sizes.
- Use Richardson extrapolation when the error model is reliable.
- Verify against independent derivative information when available.
A Reliable Data-Differentiation Workflow
- Inspect sampling intervals and measurement noise.
- Avoid excessively small differences when noise dominates.
- Use symmetric local data where available.
- Fit a local polynomial or smoothing model if raw differences are unstable.
- Differentiate the fitted model.
- Report uncertainty consistent with the data quality.
A Reliable Gradient-Checking Workflow
- Choose a representative test point.
- Evaluate the claimed gradient.
- Select one or more directions.
- Compare directional derivatives with central finite differences.
- Vary \(h\) across several scales.
- Look for the expected convergence region before rounding dominates.
Common Error: Choosing \(h\) Extremely Small
Smaller \(h\) does not guarantee a better result.
Once cancellation and rounding dominate, reducing \(h\) can make the derivative estimate worse.
Common Error: Ignoring the Order of the Formula
Forward and central formulas have different truncation orders.
Accuracy comparisons should account for both stencil cost and convergence order.
Common Error: Using Central Difference at a Boundary
A central stencil requires data on both sides.
Use a suitable one-sided formula when the domain does not extend beyond the boundary.
Common Error: Differentiating Noisy Data Directly
Differentiation amplifies high-frequency noise.
A smooth local approximation may provide a more meaningful derivative than a raw difference quotient.
Common Error: Using Only One Step Size
One derivative estimate gives little evidence about convergence.
A sequence of step sizes helps reveal whether truncation, rounding, or noise is controlling the result.
Common Error: Ignoring Scale
A step appropriate near \(x=1\) may be inappropriate near \(x=10^{10}\).
Step selection should reflect the scale of both the independent variable and the function.
Common Error: Confusing Numerical and Automatic Differentiation
Finite differences estimate derivatives from nearby values.
Automatic differentiation applies derivative rules through the computational graph.
Common Error: Ignoring Function Evaluation Error
If \(f(x)\) is itself computed approximately, the derivative estimate inherits and can amplify that error.
Example: Backward Difference
Let
\[ f(x)=x^3, \qquad x=2, \qquad h=0.1. \]Then
\[ f'(2) \approx \frac{8-6.859}{0.1} = \boxed{11.41}. \]The exact derivative is \(12\).
Example: Central Difference for \(x^3\)
\[ f'(2) \approx \frac{9.261-6.859}{0.2} = \boxed{12.01}. \]This is much closer to the exact derivative \(12\).
Example: Second Derivative
Let \(f(x)=x^2\), \(x=3\), and \(h=0.1\).
\[ f''(3) \approx \frac{9.61-18+8.41}{0.01} = \frac{0.02}{0.01} = \boxed{2}. \]Practice 1: Forward Difference
Let
\[ f(x)=x^2, \qquad x=1, \qquad h=0.01. \]Estimate \(f'(1)\) with the forward-difference formula.
Practice 2: Backward Difference
Using the same function and step, estimate \(f'(1)\) with the backward-difference formula.
Practice 3: Central Difference
Using the same function and step, estimate \(f'(1)\) with the central-difference formula.
Practice 4: Orders of Accuracy
State the leading truncation order of:
- forward difference;
- backward difference;
- central difference.
Practice 5: Second Derivative
Write the standard three-point central approximation for \(f''(x)\).
Practice 6: Richardson Extrapolation
A derivative formula has leading error \(O(h^2)\). Write the Richardson combination using \(D(h)\) and \(D(h/2)\).
Practice 7: Step-Size Tradeoff
Explain why taking \(h\) extremely small can make a finite-difference derivative less accurate.
Practice 8: Noise
Why does numerical differentiation amplify measurement noise?
Practice 9: Boundary Method
Which basic finite-difference family is most natural at the left endpoint when no data are available to the left?
Practice 10: Gradient Approximation
Write a central finite-difference approximation for
\[ \frac{\partial f}{\partial x_i} \]at \(\mathbf{x}\).
Practice Answers
Practice 1
\[ f'(1) \approx \frac{1.0201-1}{0.01} = \boxed{2.01}. \]Practice 2
\[ f'(1) \approx \frac{1-0.9801}{0.01} = \boxed{1.99}. \]Practice 3
\[ f'(1) \approx \frac{1.0201-0.9801}{0.02} = \boxed{2}. \]Practice 4
- forward difference: \(\boxed{O(h)}\);
- backward difference: \(\boxed{O(h)}\);
- central difference: \(\boxed{O(h^2)}\).
Practice 5
\[ \boxed{ f''(x) \approx \frac{ f(x+h)-2f(x)+f(x-h) }{h^2} }. \]Practice 6
\[ \boxed{ D_{\text{rich}} = \frac{ 4D(h/2)-D(h) }{3} }. \]Practice 7
Very small \(h\) makes nearby function values nearly equal. Their subtraction loses significant digits, and division by the small step amplifies the resulting floating-point error.
Practice 8
Difference quotients divide the difference between noisy measurements by \(h\). Noise is therefore amplified roughly on the scale \(\varepsilon/h\).
Practice 9
A forward-difference formula.
Practice 10
\[ \boxed{ \frac{\partial f}{\partial x_i} \approx \frac{ f(\mathbf{x}+h\mathbf{e}_i) - f(\mathbf{x}-h\mathbf{e}_i) }{2h} }. \]What to Remember
Numerical differentiation replaces derivative limits with finite differences.
\[ \boxed{ f'(x) \approx \frac{f(x+h)-f(x-h)}{2h} } \]- forward and backward differences are first-order accurate;
- central differences are typically second-order accurate;
- wider stencils can produce higher-order formulas;
- second derivatives use second differences divided by \(h^2\);
- Taylor series reveal truncation error and order of accuracy;
- Richardson extrapolation cancels leading error terms;
- smaller \(h\) reduces truncation error only until rounding begins to dominate;
- differentiation amplifies floating-point error and measurement noise;
- local polynomial fitting can stabilize derivative estimation from noisy data;
- boundary points require one-sided or specialized stencils;
- finite differences extend naturally to gradients, Hessians, and differentiation matrices;
- convergence studies across several step sizes are essential for validation.
Where We Go Next
Numerical differentiation estimates local rates of change from discrete information. The complementary problem is numerical integration: estimating accumulated quantities from sampled function values.
The next lesson, Numerical Integration, develops rectangle, trapezoidal, Simpson, Gaussian, and adaptive quadrature methods together with error formulas, convergence rates, and practical strategies for reliable accumulation.