The Central Idea
A numerical solution replaces the question
\[ \text{“Can we isolate the exact answer symbolically?”} \]with
\[ \text{“Can we compute an approximation accurate enough for the problem?”} \]For a single equation,
\[ f(x)=0, \]we seek
\[ x\approx r \]such that
\[ \boxed{|f(r)|\text{ is sufficiently small}}. \]From an Equation to a Root-Finding Problem
Many equations can be rewritten in the standard form
\[ \boxed{f(x)=0}. \]For example,
\[ e^x=3x \]becomes
\[ \boxed{f(x)=e^x-3x}. \]Numerical root-finding methods can then be applied to \(f\).
Residuals
If \(x=a\) is a proposed approximation to a root of
\[ f(x)=0, \]then
\[ \boxed{f(a)} \]is the residual.
An exact root gives residual \(0\).
An approximate root should normally give a residual whose magnitude is sufficiently small for the application.
Example: Checking a Numerical Root
Suppose
\[ f(x)=x^3-x-2 \]and a numerical method gives
\[ x\approx1.52138. \]Evaluate
\[ f(1.52138). \]The result is close to zero, confirming that the approximation nearly satisfies the equation.
Several Numerical Methods Can Solve the Same Equation
The equation
\[ f(x)=0 \]may be approached using:
- numerical tables;
- successive approximation;
- bracketing and bisection;
- fixed-point iteration;
- Newton's method;
- secant methods;
- computer algebra or numerical solver routines.
The best method depends on the available information and the required reliability.
Method Selection
A numerical method may be chosen according to whether:
- a root bracket is available;
- a derivative is available;
- function evaluations are expensive;
- multiple roots may be present;
- a guaranteed interval is required;
- rapid convergence is important;
- the problem involves one variable or several.
Example: One Equation, Several Methods
Consider
\[ x^2-5=0. \]Bisection can begin with
\[ [2,3]. \]Newton's method can begin with
\[ x_0=2. \]The secant method can begin with
\[ x_0=2,\qquad x_1=3. \]All three methods can approximate
\[ \boxed{\sqrt5\approx2.236068}. \]Numerical Solutions Need a Stopping Rule
Iterative methods do not simply stop because a decimal “looks stable.”
Common stopping conditions include:
\[ |x_{n+1}-x_n|<\varepsilon, \] \[ |f(x_n)|<\varepsilon, \]or, for a bracket,
\[ b_n-a_n<\varepsilon. \]Relative versus Absolute Tolerance
An absolute tolerance uses a fixed threshold such as
\[ |x_{n+1}-x_n|<10^{-6}. \]A relative tolerance scales the change by the size of the current value:
\[ \boxed{ \frac{|x_{n+1}-x_n|}{\max(1,|x_{n+1}|)} <\varepsilon }. \]Relative criteria are useful when solutions can vary greatly in magnitude.
More Than One Root
A nonlinear equation may have several real roots.
For example,
\[ f(x)=x^3-x \]factors as
\[ x(x-1)(x+1), \]so the exact roots are
\[ -1,\quad0,\quad1. \]A numerical method may converge to different roots depending on its starting information.
Root Isolation
Before solving numerically, it is often useful to isolate individual roots in separate regions.
For example,
\[ [-1.5,-0.5], \qquad [-0.5,0.5], \qquad [0.5,1.5] \]isolate the three roots of \(x^3-x=0\).
Root isolation helps prevent a solver from returning the wrong solution.
What Is a System of Equations?
A system of equations contains two or more equations that must be satisfied simultaneously.
For example,
\[ \begin{cases} x+y=5,\\ x-y=1. \end{cases} \]A solution is an ordered pair
\[ (x,y) \]that satisfies both equations.
Exact Systems versus Numerical Systems
Linear systems often have efficient exact methods such as elimination, matrix reduction, or inverse methods.
Nonlinear systems may be much harder to solve symbolically.
Numerical methods become especially important when the system contains nonlinear terms such as
\[ x^2,\qquad xy,\qquad e^x,\qquad\sin y. \]A Nonlinear System
Consider
\[ \begin{cases} x^2+y^2=5,\\ x-y=1. \end{cases} \]The first equation represents a circle.
The second represents a line.
Their intersections are solutions of the system.
Systems as Vector Equations
A system can be written compactly as
\[ \boxed{ \mathbf{F}(\mathbf{x})=\mathbf{0} }. \]For two variables,
\[ \mathbf{x} = \begin{bmatrix} x\\ y \end{bmatrix}, \]and
\[ \mathbf{F}(\mathbf{x}) = \begin{bmatrix} f_1(x,y)\\ f_2(x,y) \end{bmatrix}. \]Residual Vector
For a proposed numerical solution
\[ \mathbf{x}= \begin{bmatrix} x\\ y \end{bmatrix}, \]the residual vector is
\[ \boxed{ \mathbf{F}(\mathbf{x}) }. \]An exact solution gives
\[ \mathbf{F}(\mathbf{x})=\mathbf{0}. \]A numerical solution should make all residual components small.
Residual Size for a System
One way to summarize the residual vector is with a norm.
For example, the Euclidean norm is
\[ \boxed{ \|\mathbf{F}\| = \sqrt{ f_1^2+f_2^2+\cdots+f_m^2 } }. \]A small norm indicates that all equations are being satisfied closely in aggregate.
Graphical Interpretation of a Two-Equation System
For
\[ \begin{cases} f(x,y)=0,\\ g(x,y)=0, \end{cases} \]each equation defines a curve in the coordinate plane.
A solution occurs where the curves intersect.
A graph can supply an initial estimate for a numerical solver.
Substitution Can Reduce a System to One Equation
Sometimes a system can be simplified before numerical work begins.
Consider
\[ \begin{cases} x^2+y^2=5,\\ x-y=1. \end{cases} \]From
\[ x-y=1, \]we get
\[ y=x-1. \]Substitute into the first equation.
Reducing the Example
Substitute
\[ y=x-1 \]into
\[ x^2+y^2=5: \] \[ x^2+(x-1)^2=5. \]Expand:
\[ 2x^2-2x-4=0. \]Divide by \(2\):
\[ x^2-x-2=0. \]This system happens to reduce to an exactly solvable equation.
Numerical Reduction Can Still Be Useful
If substitution produces an equation such as
\[ e^x+x^2-4=0, \]the system has been reduced to a one-variable numerical root-finding problem.
Bisection, Newton's method, or a secant method can then be applied.
Fixed-Point Iteration for Systems
A nonlinear system may sometimes be rearranged into
\[ \boxed{ \begin{aligned} x&=g_1(x,y),\\ y&=g_2(x,y). \end{aligned} } \]This suggests the iteration
\[ \boxed{ \begin{aligned} x_{n+1}&=g_1(x_n,y_n),\\ y_{n+1}&=g_2(x_n,y_n). \end{aligned} } \]Example: A Simple Coupled Iteration
Consider
\[ \begin{aligned} x_{n+1}&=\frac{y_n+3}{2},\\ y_{n+1}&=\frac{x_n+1}{2}. \end{aligned} \]Starting from
\[ (x_0,y_0)=(0,0), \]gives
\[ (x_1,y_1) = \left(\frac32,\frac12\right). \]Continuing the Coupled Iteration
Next,
\[ x_2 = \frac{0.5+3}{2} = 1.75, \] \[ y_2 = \frac{1.5+1}{2} = 1.25. \]Then
\[ (x_3,y_3)=(2.125,1.375). \]The sequence of ordered pairs approaches a fixed point of the coupled update.
Simultaneous versus Sequential Updates
In a simultaneous update, all new values are computed from the old iterate:
\[ x_{n+1}=g_1(x_n,y_n), \] \[ y_{n+1}=g_2(x_n,y_n). \]In a sequential update, a newly computed value may be used immediately:
\[ x_{n+1}=g_1(x_n,y_n), \] \[ y_{n+1}=g_2(x_{n+1},y_n). \]These are different algorithms and can have different convergence behavior.
Jacobi-Type Iteration
A simultaneous update pattern is often called Jacobi-type iteration.
Every component of the new vector
\[ \mathbf{x}_{n+1} \]is computed using only components of
\[ \mathbf{x}_n. \]Gauss-Seidel-Type Iteration
A sequential update pattern is often called Gauss-Seidel-type iteration.
Newly computed components are used as soon as they become available.
This can improve convergence for some systems, though convergence is not automatic.
Newton's Method for Systems
Newton's method extends naturally from one equation to several.
For
\[ \mathbf{F}(\mathbf{x})=\mathbf{0}, \]the update is
\[ \boxed{ \mathbf{x}_{n+1} = \mathbf{x}_n - J(\mathbf{x}_n)^{-1} \mathbf{F}(\mathbf{x}_n) }, \]when the Jacobian matrix \(J\) is invertible.
The Jacobian Matrix
For a two-variable system
\[ \begin{cases} f_1(x,y)=0,\\ f_2(x,y)=0, \end{cases} \]the Jacobian is
\[ \boxed{ J(x,y) = \begin{bmatrix} \frac{\partial f_1}{\partial x} & \frac{\partial f_1}{\partial y} \\ \frac{\partial f_2}{\partial x} & \frac{\partial f_2}{\partial y} \end{bmatrix} }. \]It plays the role that the derivative plays in one-variable Newton iteration.
A More Practical Newton-System Form
Numerical software usually avoids explicitly computing
\[ J^{-1}. \]Instead, solve the linear system
\[ \boxed{ J(\mathbf{x}_n)\Delta\mathbf{x} = -\mathbf{F}(\mathbf{x}_n) } \]and then update
\[ \boxed{ \mathbf{x}_{n+1} = \mathbf{x}_n+\Delta\mathbf{x} }. \]Example Nonlinear System for Newton's Method
Consider
\[ \begin{cases} x^2+y^2-4=0,\\ x-y=0. \end{cases} \]Define
\[ \mathbf{F}(x,y) = \begin{bmatrix} x^2+y^2-4\\ x-y \end{bmatrix}. \]Then
\[ J(x,y) = \begin{bmatrix} 2x & 2y\\ 1 & -1 \end{bmatrix}. \]Interpreting the Newton Correction
The correction vector
\[ \Delta\mathbf{x} = \begin{bmatrix} \Delta x\\ \Delta y \end{bmatrix} \]estimates how the current point should move to reduce the residual vector.
Each Newton step therefore solves a local linear approximation of the nonlinear system.
Why Systems Are Harder
Several complications appear in systems:
- multiple variables interact;
- several residuals must be reduced at once;
- the Jacobian may become singular or ill-conditioned;
- different starting guesses may converge to different solutions;
- the system may have no solution or many solutions.
Initial Guesses for Systems
A nonlinear system requires an initial vector such as
\[ \boxed{ \mathbf{x}_0 = \begin{bmatrix} x_0\\ y_0 \end{bmatrix} }. \]Useful guesses can come from:
- graphs or contour plots;
- physical constraints;
- nearby parameter cases;
- simplified versions of the system;
- coarse grid searches.
Grid Search for a Two-Variable System
A simple exploratory method is to evaluate residuals over a grid of points.
For example:
\[ x=-2,-1.5,-1,\ldots,2 \]and
\[ y=-2,-1.5,-1,\ldots,2. \]Points producing small residuals can supply starting guesses for more efficient iterative methods.
Convergence for Systems
A sequence of vectors
\[ \mathbf{x}_0,\quad \mathbf{x}_1,\quad \mathbf{x}_2,\quad\ldots \]converges when all components approach stable limiting values.
We write
\[ \boxed{ \mathbf{x}_n\to\mathbf{r} }. \]Stopping Conditions for Systems
Common criteria include
\[ \boxed{ \|\mathbf{x}_{n+1}-\mathbf{x}_n\|<\varepsilon_x } \]and
\[ \boxed{ \|\mathbf{F}(\mathbf{x}_{n+1})\|<\varepsilon_f }. \]These generalize iterate-change and residual tests from one variable.
Maximum Iteration Limits
Numerical solvers should include a maximum iteration count.
A system solver may fail because:
- the starting guess is poor;
- the Jacobian becomes singular;
- the iteration diverges;
- the tolerance is unrealistically strict;
- the system itself is inconsistent.
Linear Systems and Numerical Solution
Even linear systems may be solved numerically when they are very large.
A system
\[ A\mathbf{x}=\mathbf{b} \]may be solved using direct matrix methods or iterative methods.
The choice depends on matrix size, sparsity, conditioning, and computational cost.
Direct versus Iterative Linear Solvers
Direct methods attempt to produce a solution after a finite sequence of elimination steps.
Iterative methods produce approximations
\[ \mathbf{x}_0, \mathbf{x}_1, \mathbf{x}_2,\ldots \]that ideally converge to the solution.
Large sparse systems often favor iterative methods.
Conditioning
A problem is well-conditioned when small changes in the input produce relatively small changes in the solution.
A problem is ill-conditioned when small input changes can cause large solution changes.
Numerical difficulty may come from the problem itself, not merely from the algorithm.
Algorithmic Error versus Problem Sensitivity
Two different issues must be separated:
- algorithmic error from approximation, stopping, or roundoff;
- problem sensitivity from the mathematical structure of the equations.
A highly accurate algorithm cannot remove sensitivity that is inherent in the original problem.
Scaling
Numerical systems can become difficult when different variables or equations operate on very different scales.
For example, one residual may naturally be near
\[ 10^6 \]while another is near
\[ 10^{-3}. \]Rescaling equations or variables can improve numerical behavior.
Verification Matters
After a solver returns a candidate solution, verify it independently.
For one equation, evaluate
\[ f(x). \]For a system, evaluate every equation or the residual vector
\[ \mathbf{F}(\mathbf{x}). \]A solver's status message should not replace mathematical checking.
Check the Original Equations
If the equations were rearranged, scaled, squared, or otherwise transformed, verify the final approximation in the original system.
Transformations can introduce:
- extraneous solutions;
- lost domain restrictions;
- numerical instability;
- unintended branches.
Common Error: Solving Only One Equation in a System
A pair \((x,y)\) is a solution only if it satisfies every equation in the system.
Making one residual small while another remains large is not enough.
Common Error: Ignoring Multiple Solutions
A numerical solver may return one valid solution without revealing others.
Use graphs, root isolation, multiple starting guesses, or structural analysis when the problem may have several solutions.
Common Error: Assuming Convergence Means Correctness
An iterative method may converge to:
- the desired solution;
- a different solution;
- a spurious fixed point of a rearranged system;
- a numerically stable but inaccurate value.
Always verify the result in the original equations.
Common Error: Using an Unjustified Starting Guess
Nonlinear solvers can be sensitive to initialization.
A good starting guess should come from mathematical information whenever possible rather than arbitrary choice.
Common Error: Using Too Strict a Tolerance
Demanding accuracy far beyond the precision of the data or machine arithmetic can waste computation or prevent convergence criteria from ever being satisfied.
Tolerance should match the purpose and scale of the problem.
Common Error: Reporting Solver Digits as Exact
A numerical solver may display many digits.
Those digits are not automatically exact or even all meaningful.
Report only the precision supported by the residuals, tolerances, input data, and conditioning of the problem.
A Reliable Workflow for One Equation
- Rewrite the problem as \(f(x)=0\).
- Identify the domain and possible number of roots.
- Use algebra, graphs, or tables to estimate root locations.
- Choose an appropriate numerical method.
- Select starting values or a bracket.
- Apply the iteration with stated tolerances.
- Monitor failure conditions and iteration count.
- Check the residual.
- Confirm that the intended root was found.
- Report the answer with justified precision.
A Reliable Workflow for a System
- Write all equations clearly.
- Identify variables, domains, and constraints.
- Simplify or reduce the system when practical.
- Estimate solution regions using graphs, tables, or physical reasoning.
- Choose a numerical method appropriate to the system.
- Select an initial vector.
- Iterate while monitoring step size and residual norm.
- Stop at the stated tolerance or failure condition.
- Substitute the candidate solution into every original equation.
- Search for additional solutions when the structure suggests they may exist.
Example: Numerical Solution of One Equation
Solve
\[ e^{-x}=x. \]Rewrite:
\[ f(x)=e^{-x}-x. \]Since
\[ f(0)=1 \]and
\[ f(1)=e^{-1}-1<0, \]the root lies in
\[ \boxed{(0,1)}. \]Approximating the Root
A numerical method gives
\[ \boxed{x\approx0.567143}. \]Check:
\[ e^{-0.567143}\approx0.567143. \]Thus the approximation satisfies the original equation closely.
Example: Numerical System
Consider
\[ \begin{cases} x^2+y^2=4,\\ e^x+y=2. \end{cases} \]This system combines a circle with an exponential relationship.
Closed-form elimination is not especially convenient.
A graph or numerical search can provide an initial estimate, followed by a system solver.
Numerical-System Verification
Suppose a solver returns
\[ (x,y)\approx(a,b). \]Verify both residuals:
\[ r_1=a^2+b^2-4, \] \[ r_2=e^a+b-2. \]A reliable approximation should make both
\[ |r_1| \]and
\[ |r_2| \]acceptably small.
Practice 1: Standard Form
Rewrite
\[ 2^x=4x \]as a root-finding problem \(f(x)=0\).
Practice 2: Residual
For
\[ f(x)=x^2-7 \]and
\[ x=2.646, \]compute the residual.
Practice 3: Method Choice
Which of the following methods requires an explicit derivative?
- bisection;
- Newton's method;
- secant method.
Practice 4: System Residuals
For
\[ \begin{cases} x+y=3,\\ x-y=1, \end{cases} \]evaluate both residuals at
\[ (x,y)=(2,1). \]Practice 5: Vector Form
Write
\[ \begin{cases} x^2+y-3=0,\\ x+y^2-3=0 \end{cases} \]as
\[ \mathbf{F}(x,y)=\mathbf{0}. \]Practice 6: Jacobian
Find the Jacobian of
\[ \mathbf{F}(x,y) = \begin{bmatrix} x^2+y-3\\ x+y^2-3 \end{bmatrix}. \]Practice 7: Stopping Condition
Suppose
\[ \|\mathbf{x}_{n+1}-\mathbf{x}_n\| = 4\times10^{-7}. \]Does the condition
\[ \|\mathbf{x}_{n+1}-\mathbf{x}_n\|<10^{-6} \]hold?
Practice 8: Multiple Solutions
Why might one numerical run be insufficient for a nonlinear system with several possible intersections?
Practice 9: Simultaneous Update
In the iteration
\[ x_{n+1}=g_1(x_n,y_n), \qquad y_{n+1}=g_2(x_n,y_n), \]are the two new values computed from old values or from newly updated values?
Practice 10: Verification
State the most important final check after a numerical solver returns a solution to a system.
Practice Answers
Practice 1
Move all terms to one side:
\[ \boxed{ f(x)=2^x-4x }. \]Then solve
\[ f(x)=0. \]Practice 2
\[ f(2.646) = 2.646^2-7. \] \[ = 7.001316-7. \] \[ \boxed{f(2.646)=0.001316}. \]Practice 3
Newton's method requires
\[ \boxed{f'(x)}. \]Bisection and the standard secant method do not require an explicit derivative.
Practice 4
First residual:
\[ 2+1-3=0. \]Second residual:
\[ 2-1-1=0. \]Therefore,
\[ \boxed{(2,1)} \]satisfies both equations exactly.
Practice 5
\[ \boxed{ \mathbf{F}(x,y) = \begin{bmatrix} x^2+y-3\\ x+y^2-3 \end{bmatrix} = \begin{bmatrix} 0\\ 0 \end{bmatrix} }. \]Practice 6
\[ \boxed{ J(x,y) = \begin{bmatrix} 2x & 1\\ 1 & 2y \end{bmatrix} }. \]Practice 7
Since
\[ 4\times10^{-7}<10^{-6}, \]the stopping condition is satisfied.
Practice 8
Different starting guesses may converge to different solutions.
One run may therefore find only one intersection while missing others.
Practice 9
Both new values are computed from the old pair
\[ \boxed{(x_n,y_n)}. \]This is a simultaneous update.
Practice 10
Substitute the reported values into every original equation and verify that all residuals are sufficiently small.
What to Remember
- Numerical equation solving seeks approximations that make residuals small.
- Single equations are commonly rewritten as \(f(x)=0\).
- Different numerical methods trade speed, information requirements, and reliability.
- Systems require several equations to be satisfied simultaneously.
- A system can be written as \(\mathbf{F}(\mathbf{x})=\mathbf{0}\).
- Residual vectors and norms help measure how well a numerical solution satisfies a system.
- Fixed-point and Newton-style methods extend naturally to multiple variables.
- Newton's method for systems uses the Jacobian matrix.
- Initial guesses, scaling, conditioning, and multiple solutions all affect numerical behavior.
- Every numerical result should be verified in the original equations.
Where We Go Next
Numerical equation solving depends on repeated evaluation of expressions and structured computational procedures. Many of those procedures can be made faster and more reliable by using specialized algebraic algorithms.
The next lesson, Polynomial Evaluation and Synthetic Division Algorithms, develops efficient methods for evaluating polynomials, dividing by linear factors, and organizing repeated polynomial computations.