The Central Idea
Suppose a mathematical problem maps input \(x\) to output
\[ y=f(x). \]Numerical computation must distinguish two effects:
- small changes in \(x\) may naturally cause changes in \(y\);
- the algorithm may introduce additional error while computing an approximation to \(y\).
These correspond to
\[ \boxed{\text{conditioning}} \]and
\[ \boxed{\text{stability}}. \]Conditioning Belongs to the Problem
A problem is well-conditioned when small changes in the input produce only modest changes in the output.
A problem is ill-conditioned when tiny input perturbations can produce large changes in the output.
Conditioning is therefore a property of the mathematical relationship being solved or evaluated.
Stability Belongs to the Algorithm
A numerical algorithm is stable when it controls the effects of rounding, intermediate approximation, and perturbation during computation.
An unstable algorithm can magnify small numerical errors even when the original problem is well-conditioned.
Stability is therefore a property of the computational procedure.
Why the Distinction Matters
A poor result can arise for two fundamentally different reasons.
- The problem may be inherently sensitive.
- The chosen algorithm may be numerically unstable.
The remedies are different.
If the problem is ill-conditioned, higher-quality data or reformulation may be required. If the algorithm is unstable, a better computational method may solve the issue.
A Simple Sensitivity Example
Consider
\[ f(x)=10x. \]If \(x\) changes by
\[ \Delta x, \]then
\[ \Delta f=10\Delta x. \]The output perturbation is ten times the absolute input perturbation.
Absolute Sensitivity
For a differentiable scalar function
\[ y=f(x), \]a small perturbation satisfies approximately
\[ \Delta y\approx f'(x)\Delta x. \]The quantity
\[ \boxed{|f'(x)|} \]measures local absolute sensitivity.
Absolute Condition Number
A natural local absolute condition number is
\[ \boxed{\kappa_{\text{abs}}(x)=|f'(x)|}. \]Then small input perturbations approximately satisfy
\[ |\Delta y| \approx \kappa_{\text{abs}}(x)|\Delta x|. \]A larger value indicates greater absolute sensitivity.
Relative Sensitivity
Absolute sensitivity depends on the scale of the quantities.
Relative conditioning compares proportional changes:
\[ \frac{|\Delta y|}{|y|} \]versus
\[ \frac{|\Delta x|}{|x|}. \]This often gives a more meaningful measure across different magnitudes.
Relative Condition Number
For \(x\neq0\) and \(f(x)\neq0\), the scalar relative condition number is
\[ \boxed{ \kappa(x) = \left| \frac{x f'(x)}{f(x)} \right| }. \]For sufficiently small perturbations,
\[ \boxed{ \frac{|\Delta y|}{|y|} \approx \kappa(x) \frac{|\Delta x|}{|x|} }. \]Interpreting a Condition Number
If
\[ \kappa(x)\approx1, \]then relative input error is transmitted at roughly the same scale.
If
\[ \kappa(x)\gg1, \]then small relative input errors can be strongly amplified.
If
\[ \kappa(x)\ll1, \]the function locally damps relative perturbations.
Example: Conditioning of \(f(x)=x^n\)
Let
\[ f(x)=x^n. \]Then
\[ f'(x)=nx^{n-1}. \]Therefore,
\[ \kappa(x) = \left| \frac{x(nx^{n-1})}{x^n} \right| = \boxed{|n|}. \]Relative input error is locally multiplied by approximately \(|n|\).
Example: Conditioning of the Square Root
Let
\[ f(x)=\sqrt{x}=x^{1/2}. \]Then
\[ \kappa(x)=\frac12. \]Thus a small relative error in \(x\) produces approximately half as much relative error in \(\sqrt{x}\).
The square-root operation is relatively well-conditioned away from problematic domain boundaries.
Example: Conditioning Near a Zero of the Output
Consider
\[ f(x)=x-1. \]Then
\[ f'(x)=1 \]and
\[ \kappa(x) = \left| \frac{x}{x-1} \right|. \]As \(x\) approaches \(1\),
\[ |x-1|\to0, \]so the relative condition number becomes very large.
Why Subtraction Can Be Ill-Conditioned
Suppose
\[ z=x-y \]and \(x\) and \(y\) are nearly equal.
The output \(z\) is small compared with the inputs.
Small relative perturbations in \(x\) or \(y\) can therefore create a large relative perturbation in \(z\).
This sensitivity belongs to the subtraction problem itself.
Conditioning of Addition
For positive quantities of similar scale, addition is usually well-conditioned.
If
\[ z=x+y \]with \(x>0\) and \(y>0\), small relative input errors generally produce a comparably small relative error in the sum.
The situation changes when signs differ and cancellation occurs.
Cancellation and Conditioning
Cancellation is not always an algorithmic defect.
If the mathematical problem asks for the difference of two nearly equal uncertain quantities, the problem itself may be ill-conditioned.
But an algebraically equivalent reformulation can sometimes avoid unnecessary cancellation and improve algorithmic stability.
A Classic Algebraic Reformulation
Consider
\[ \sqrt{x+1}-\sqrt{x} \]for large \(x\).
Direct subtraction involves two nearly equal quantities.
Rationalize:
\[ \sqrt{x+1}-\sqrt{x} = \frac{1}{\sqrt{x+1}+\sqrt{x}}. \]The second form avoids subtracting nearly equal numbers and is often more numerically stable.
Same Mathematics, Different Numerical Behavior
Algebraically equivalent expressions can behave differently in floating-point arithmetic.
Exact algebra guarantees
\[ A=B. \]But finite precision can produce
\[ \operatorname{fl}(A)\neq\operatorname{fl}(B). \]Numerical formulation therefore matters.
Forward Stability
An algorithm is informally forward stable when its computed output is close to the exact output.
If the exact result is \(y\) and the algorithm returns \(\tilde{y}\), forward error measures
\[ \boxed{|\tilde{y}-y|}. \]A small forward error is the most direct statement about the final answer.
Backward Stability
An algorithm is backward stable when its computed result is the exact solution to a nearby problem.
Symbolically, if the original input is \(x\), a backward-stable algorithm produces
\[ \tilde{y}=f(x+\Delta x) \]for a small perturbation \(\Delta x\).
Why Backward Stability Is Powerful
Backward stability says that the algorithm behaves as though the input had been perturbed slightly and then solved exactly.
This is often the strongest practical guarantee available in finite-precision computation.
If the problem is also well-conditioned, a small backward error usually implies a small forward error.
The Conditioning-Stability Principle
A useful conceptual relationship is
\[ \boxed{ \text{forward error} \approx \text{condition number} \times \text{backward error} }. \]This is not a universal exact equality, but it captures the central logic of numerical reliability.
A stable algorithm cannot overcome severe ill-conditioning.
Well-Conditioned Problem, Stable Algorithm
This is the ideal case.
- small input perturbations produce small output changes;
- the algorithm introduces only controlled additional error;
- the computed result is likely to be reliable.
Well-Conditioned Problem, Unstable Algorithm
The mathematical problem is not especially sensitive, but the method magnifies numerical errors.
In this situation, replacing the algorithm may dramatically improve the result.
Ill-Conditioned Problem, Stable Algorithm
The algorithm behaves well, but the problem itself amplifies small input perturbations.
Even an excellent algorithm may then produce an output with limited accuracy.
Better data, additional precision, reformulation, or regularization may be required.
Ill-Conditioned Problem, Unstable Algorithm
This is the most difficult case.
The problem amplifies input perturbations and the algorithm introduces additional numerical amplification.
Reliable computation may require changes to both the formulation and the algorithm.
Residuals and Stability
Suppose an approximate solution \(\tilde{x}\) is intended to satisfy
\[ f(x)=0. \]The residual is
\[ r=f(\tilde{x}). \]A small residual indicates that the computed solution nearly satisfies the equation.
But the condition of the problem determines whether that small residual implies a small solution error.
Residual Versus Forward Error
Consider
\[ f(x)=\varepsilon x \]with very small \(\varepsilon\).
A large error in \(x\) may produce only a small residual because the function is nearly flat.
Thus,
\[ |r|\text{ small} \]does not automatically imply
\[ |\tilde{x}-x^*|\text{ small}. \]Perturbation Analysis
Perturbation analysis studies how a solution changes when the input data are changed slightly.
The typical pattern is
\[ x\longrightarrow x+\Delta x \]and then
\[ f(x)\longrightarrow f(x+\Delta x). \]Condition numbers summarize this local sensitivity.
Multivariable Conditioning
For
\[ y=f(x_1,\ldots,x_n), \]first-order perturbation gives
\[ \Delta y \approx \sum_{i=1}^{n} \frac{\partial f}{\partial x_i}\Delta x_i. \]The gradient therefore controls local absolute sensitivity:
\[ \nabla f = \left( \frac{\partial f}{\partial x_1}, \ldots, \frac{\partial f}{\partial x_n} \right). \]Norms and Conditioning
In vector and matrix problems, error is measured with a norm.
For example,
\[ \|\Delta x\| \]measures input perturbation magnitude and
\[ \|\Delta y\| \]measures output perturbation magnitude.
Condition numbers then compare these quantities in an appropriate absolute or relative form.
Matrix Conditioning
For a nonsingular linear system
\[ A\mathbf{x}=\mathbf{b}, \]the matrix \(A\) controls how perturbations in the data can affect the solution.
A standard matrix condition number is
\[ \boxed{ \kappa(A)=\|A\|\,\|A^{-1}\| }. \]The exact value depends on the chosen matrix norm.
Interpreting Matrix Condition Numbers
A condition number near \(1\) indicates relatively favorable sensitivity.
A very large condition number indicates that small perturbations in the matrix or right-hand side may cause large changes in the solution.
Matrix conditioning will become especially important in Numerical Linear Algebra.
Near Singularity
A matrix that is close to singular is often ill-conditioned.
If \(A^{-1}\) has very large norm, then
\[ \kappa(A)=\|A\|\|A^{-1}\| \]becomes large.
This means small data perturbations may be strongly amplified in the solution.
Condition Numbers and Lost Digits
A large condition number can indicate loss of reliable digits.
Roughly, if
\[ \kappa\approx10^m, \]then up to about \(m\) decimal digits of relative accuracy may be lost compared with the input accuracy.
This is a rule of thumb rather than a universal exact statement.
Stability of Repeated Iteration
Consider an iteration
\[ x_{n+1}=F(x_n). \]If a small perturbation \(e_n\) is present in \(x_n\), then locally
\[ e_{n+1}\approx F'(x_n)e_n. \]Thus the magnitude of \(F'\) helps indicate whether perturbations shrink or grow.
Error Damping
If
\[ |F'(x)|<1, \]then nearby perturbations may contract under iteration.
This can contribute to numerical stability and convergence.
Error Amplification
If
\[ |F'(x)|>1, \]then perturbations may grow under repeated iteration.
Even small rounding errors can then become significant after many steps.
Stable and Unstable Recurrence Directions
A recurrence relation may behave well in one direction and poorly in another.
Reversing a numerically stable recurrence can turn contraction into amplification.
The mathematically equivalent direction is therefore not always the numerically reliable one.
Scaling
Numerical problems often behave better when variables are placed on comparable scales.
If one variable is of order
\[ 10^{-9} \]while another is of order
\[ 10^{12}, \]poor scaling can complicate numerical computation.
Rescaling or nondimensionalization can improve numerical behavior without changing the underlying problem.
Why Scaling Can Help
Good scaling can:
- reduce extreme coefficient disparities;
- improve the effectiveness of stopping tolerances;
- reduce loss of significance;
- improve iterative convergence;
- make error estimates easier to interpret.
Stability and Intermediate Quantities
An algorithm may produce a modest final result while passing through enormous intermediate values.
Large intermediate magnitudes can increase the risk of:
- overflow;
- loss of significance;
- rounding amplification;
- subtraction of nearly equal large values.
Stable formulations often avoid unnecessary extremes.
Example: Evaluating a Polynomial
Consider
\[ p(x)=a_0+a_1x+a_2x^2+\cdots+a_nx^n. \]One approach computes every power \(x^k\) separately.
Another rewrites the polynomial using Horner's form:
\[ p(x) = a_0+x(a_1+x(a_2+\cdots+x(a_{n-1}+xa_n)\cdots)). \]The second form usually uses fewer arithmetic operations and is often numerically preferable.
Fewer Operations Can Mean Less Error
Every floating-point arithmetic operation can introduce rounding.
Reducing unnecessary operations can therefore reduce opportunities for error accumulation.
Efficiency and stability are not identical, but they often reinforce each other.
Stable Does Not Mean Exact
A stable algorithm may still produce a nonzero error.
Stability means the error is controlled relative to the unavoidable perturbations of finite-precision computation.
Exact arithmetic is rarely the realistic standard for large numerical problems.
High Precision Does Not Guarantee Stability
Using more digits can reduce rounding error, but it does not automatically fix an unstable algorithm.
If an error is multiplied repeatedly by a large amplification factor, additional precision may only delay the breakdown.
Algorithmic design still matters.
Stability Does Not Fix Ill-Conditioning
A backward-stable algorithm can solve a nearby problem extremely accurately.
But if the original problem is ill-conditioned, even a tiny backward perturbation can cause a large forward change.
This is why condition number and stability must be evaluated together.
Diagnostic Question 1: Is the Input Reliable?
Before analyzing the algorithm, inspect the data.
- Are measurements noisy?
- Were constants rounded?
- Are parameters estimated?
- Are units and scales consistent?
Poor input quality can dominate every later numerical effect.
Diagnostic Question 2: Is the Problem Sensitive?
Perturb the input slightly and observe whether the output changes dramatically.
Analytically, use derivatives, Jacobians, matrix norms, or condition numbers.
Computationally, controlled perturbation experiments can provide practical evidence of sensitivity.
Diagnostic Question 3: Is the Formula Numerically Safe?
Look for:
- subtraction of nearly equal numbers;
- division by small numbers;
- very large intermediate quantities;
- very small intermediate quantities;
- repeated multiplication of error-amplifying factors.
Diagnostic Question 4: Does More Precision Help?
Repeating a calculation with higher precision can reveal useful information.
If the result stabilizes, rounding may have been significant.
If the result remains highly sensitive, the problem itself may be ill-conditioned.
Diagnostic Question 5: Do Independent Methods Agree?
When practical, compare:
- different algorithms;
- different formulations;
- different step sizes;
- different precision levels;
- analytic checks or known special cases.
Agreement across independent approaches increases confidence.
A Reliable Conditioning Workflow
- Write the mathematical mapping from input to output.
- Identify the input quantities that may be perturbed.
- Estimate absolute or relative input perturbations.
- Compute derivatives, Jacobians, or an appropriate condition number.
- Estimate how input errors can be amplified.
- Identify points or parameter regimes where the condition number becomes large.
- Decide whether reformulation, rescaling, or better data are needed.
A Reliable Stability Workflow
- Identify where rounding or truncation enters the algorithm.
- Track how local errors move through later steps.
- Inspect subtraction, division, and intermediate scaling.
- Estimate forward error, backward error, or both.
- Compare alternative algebraic formulations.
- Test sensitivity to precision and evaluation order.
- Prefer methods that control error amplification.
Example: Stable Reformulation
Evaluate
\[ \sqrt{1000001}-1000. \]Direct subtraction involves two nearly equal numbers.
Rationalizing gives
\[ \frac{1}{\sqrt{1000001}+1000}. \]The second form avoids subtracting large nearly equal quantities and is numerically safer.
Example: Condition Number of \(1/x\)
Let
\[ f(x)=\frac1x. \]Then
\[ f'(x)=-\frac1{x^2}. \]The relative condition number is
\[ \kappa(x) = \left| \frac{x(-1/x^2)}{1/x} \right| = \boxed{1}. \]Away from \(x=0\), reciprocal evaluation is relatively well-conditioned.
Example: Absolute Sensitivity of \(1/x\)
Although the relative condition number is \(1\), the absolute condition number is
\[ \kappa_{\text{abs}}(x) = \frac1{x^2}. \]As \(x\to0\), this becomes large.
This illustrates why absolute and relative conditioning answer different questions.
Example: Sensitivity of \(e^x\)
Let
\[ f(x)=e^x. \]Then
\[ f'(x)=e^x. \]Therefore,
\[ \kappa(x) = \left| \frac{x e^x}{e^x} \right| = \boxed{|x|}. \]Relative sensitivity increases with \(|x|\).
Example: A Sensitive Root
Suppose \(x^*\) satisfies
\[ f(x^*)=0. \]If
\[ |f'(x^*)| \]is very small, then the graph is nearly flat at the root.
A small change in function value can correspond to a large change in the root location.
Root-finding near such points can be ill-conditioned.
Example: Stable Versus Unstable Summation
Suppose many positive floating-point numbers vary greatly in magnitude.
Adding the largest values first can cause tiny terms to be lost when added to a much larger partial sum.
Summing smaller magnitudes first can sometimes preserve more low-order information.
The arithmetic is mathematically equivalent, but the numerical behavior can differ.
Common Error: Blaming the Algorithm for Ill-Conditioning
If tiny input changes produce large output changes even under exact arithmetic, the problem is ill-conditioned.
Changing algorithms alone cannot remove that inherent sensitivity.
Common Error: Calling Every Accurate Method Stable
A method may happen to produce an accurate answer for one input without being stable in general.
Stability concerns the method's behavior across perturbations and finite precision, not one fortunate result.
Common Error: Using Only the Residual
A small residual is useful evidence, but it is not a complete error analysis.
Always ask how residual size relates to solution error under the conditioning of the problem.
Common Error: Ignoring Units and Scale
Absolute conditioning can change when units change.
Relative conditioning is often more invariant to scale, but numerical implementations can still benefit from careful rescaling.
Conditioning should always be interpreted in the context of the actual problem.
Common Error: Assuming More Iterations Always Improve Accuracy
If an iteration is unstable, additional steps can amplify errors.
Once rounding and model limitations dominate, more iterations may no longer improve the result.
Common Error: Ignoring Algebraic Reformulation
A formula may be mathematically correct yet numerically poor.
Before increasing precision or computational cost, check whether an equivalent formulation avoids cancellation, overflow, underflow, or unnecessary operations.
Practice 1: Condition Number of a Power
Find the relative condition number of
\[ f(x)=x^5. \]Practice 2: Condition Number of a Square Root
Find the relative condition number of
\[ f(x)=\sqrt{x}. \]Practice 3: Sensitive Subtraction
Explain why
\[ 1000.001-1000 \]can be more sensitive to input perturbations than
\[ 1000.001+1000. \]Practice 4: Conditioning or Stability?
Identify whether each statement describes conditioning or stability.
- A tiny perturbation in the input causes a huge change in the exact solution.
- An algorithm magnifies rounding errors through repeated subtraction.
Practice 5: Backward Stability
In your own words, explain what it means for a computed solution to be the exact solution of a nearby problem.
Practice 6: Relative Sensitivity
Let
\[ f(x)=e^x. \]Find the relative condition number at
\[ x=3. \]Practice 7: Stable Reformulation
Rewrite
\[ \sqrt{x+1}-\sqrt{x} \]in a form that avoids direct subtraction of nearly equal square roots.
Practice 8: Matrix Conditioning
A matrix has
\[ \|A\|=12 \]and
\[ \|A^{-1}\|=50. \]Compute the corresponding condition number.
Practice 9: Interpreting a Large Condition Number
What does a very large condition number indicate about the sensitivity of a problem?
Practice 10: Residual and Error
Why does a small residual not automatically prove that an approximate solution is close to the exact solution?
Practice Answers
Practice 1
For
\[ f(x)=x^5, \]the relative condition number is
\[ \boxed{5}. \]Practice 2
Since
\[ f(x)=x^{1/2}, \]the relative condition number is
\[ \boxed{\frac12}. \]Practice 3
The subtraction produces a very small result from two large nearby inputs. Small perturbations in either input can therefore become large relative to the difference. The sum remains large and is usually much less sensitive in relative terms.
Practice 4
- Conditioning.
- Stability.
Practice 5
Backward stability means the computed result can be interpreted as an exact solution to a slightly perturbed version of the original input problem.
Practice 6
For
\[ f(x)=e^x, \]the relative condition number is
\[ \kappa(x)=|x|. \]At \(x=3\),
\[ \boxed{\kappa(3)=3}. \]Practice 7
Rationalize:
\[ \boxed{ \sqrt{x+1}-\sqrt{x} = \frac{1}{\sqrt{x+1}+\sqrt{x}} }. \]Practice 8
\[ \kappa(A) = \|A\|\,\|A^{-1}\| = 12(50) = \boxed{600}. \]Practice 9
A very large condition number indicates that small relative changes in the input can produce much larger relative changes in the output. The problem is highly sensitive or ill-conditioned.
Practice 10
A small residual only shows that the approximate solution nearly satisfies the equation. If the problem is ill-conditioned, a small residual can correspond to a much larger error in the solution itself.
What to Remember
Conditioning and stability answer different questions:
\[ \boxed{ \text{conditioning} = \text{sensitivity of the mathematical problem} } \]and
\[ \boxed{ \text{stability} = \text{error behavior of the numerical algorithm} }. \]- well-conditioned problems respond modestly to small input perturbations;
- ill-conditioned problems amplify input perturbations;
- stable algorithms control rounding and intermediate errors;
- backward stability means the computed answer exactly solves a nearby problem;
- a small backward error gives a small forward error only when the problem is sufficiently well-conditioned;
- large condition numbers signal sensitivity and potential loss of reliable digits;
- small residuals do not automatically imply small solution errors;
- algebraically equivalent formulas can have very different numerical stability;
- scaling, reformulation, higher precision, and better algorithms address different sources of numerical difficulty.
Where We Go Next
Conditioning and stability provide the framework for judging whether a numerical method can be trusted. The next step is to apply that framework to a fundamental computational task: solving nonlinear equations.
The next lesson, Root-Finding, develops bisection, fixed-point iteration, Newton's method, secant methods, stopping criteria, convergence, and the numerical tradeoffs involved in locating zeros of functions.