Searching for the best value when exact calculus solutions are unavailable — one-dimensional search, Newton's method, gradient descent, and the geometry of minima
Every field of applied mathematics eventually encounters the same fundamental question: among all possible choices, which one is best? That question is the domain of optimization. In calculus, students learn to find maxima and minima by setting derivatives equal to zero and solving. That approach is powerful when functions are simple enough to differentiate symbolically and when the resulting equations can be solved exactly. But in real applications, objective functions are often complicated, multivariable, noisy, or entirely lacking a symbolic formula. Exact differentiation may be unavailable. Exact equation-solving may be computationally impossible. In those situations, numerical optimization takes over.
This chapter introduces the core ideas of numerical optimization: how to search for best values when exact formulas are unavailable or impractical. Students will review optimization from calculus, then study one-dimensional search methods, Newton's method applied to optimization problems, gradient descent, multivariable optimization, and a preview of constrained optimization. Applications range from engineering design to financial portfolio modeling to machine learning.
Numerical optimization should feel like intelligent, structured mathematical searching. The goal is always to find a value — an input, a parameter, a decision — that makes some measure of performance as good as possible, while managing error, convergence, and the ever-present risk of finding only a local optimum rather than a global one.
An engineer is designing a cylindrical storage tank with a fixed volume of 1000 cubic meters. The cost of material depends on the surface area: the curved side costs twice as much per square meter as the flat top and bottom. What radius and height minimize the total material cost?
If you set up the cost function, substitute the volume constraint, and differentiate, you can find an exact answer in this case. But suppose the cost relationship between the curved surface and the flat ends is not a simple ratio — suppose it depends on manufacturing tolerances, which themselves depend on the radius in a nonlinear way given by an empirical table. There is no longer a clean symbolic formula. You cannot differentiate and set equal to zero. You must search numerically. That is the situation numerical optimization was built for.
Optimization is embedded in nearly every quantitative field. Engineers minimize weight, maximize strength, or balance cost against performance. Economists maximize utility subject to budget constraints. Statisticians minimize the sum of squared residuals to fit models to data. Machine learning algorithms minimize loss functions over millions of parameters. Financial analysts maximize expected return for a given level of risk. Climate modelers optimize parameters to match observational data.
In all these fields, the mathematical problems are too complex, too large, or too data-dependent for symbolic calculus alone. Numerical optimization provides the computational tools to find answers. Beyond applications, optimization deepens understanding of calculus itself: every root-finding method from Chapter 3 can be interpreted as an optimization step, Taylor approximations from Chapter 8 underlie Newton's optimization method, and the linear algebra of Chapter 9 appears directly in multivariable gradient computations. Numerical optimization is where much of the mathematics from earlier chapters comes together into a unified computational practice.
By the end of this chapter, you should be able to:
Optimization is the process of finding the input value, or collection of input values, that makes a function as large or as small as possible. The function being optimized is called the objective function. When seeking the smallest possible value, the problem is a minimization problem. When seeking the largest possible value, it is a maximization problem.
In calculus, these problems are solved by finding critical points: values where the derivative equals zero or fails to exist, then testing to determine whether each critical point is a maximum, minimum, or neither. This strategy works beautifully for well-behaved functions of one variable, and extends to several variables using partial derivatives and the second derivative test.
Numerical optimization addresses the situation when this symbolic strategy is unavailable, impractical, or too slow. There are several reasons this happens.
In all these cases, numerical optimization searches for a good value by generating a sequence of trial inputs, evaluating the objective function at each, and using that information to guide the next trial. Different methods generate the sequence differently, using various amounts of function value information, derivative information, and geometric reasoning.
Finding the global minimum requires examining, at least indirectly, the entire feasible region. For difficult problems, this is computationally expensive or even impossible. Practical numerical optimization frequently settles for finding a good local minimum quickly, with some awareness of the risk that the global minimum has not been found.
Before turning to numerical methods, it is worth reviewing the exact framework from calculus, because numerical methods are designed to approximate exactly what calculus would do if it could.
For a differentiable function \( f(x) \) of a single variable, a critical point is a value \( x^* \) where
or where \( f'(x^*) \) does not exist. At a critical point, the function is neither increasing nor decreasing; it is momentarily flat. The second derivative test then classifies the critical point:
For a function of several variables, \( f(x_1, x_2, \ldots, x_n) \), a critical point requires that all partial derivatives vanish simultaneously:
The gradient of \( f \) is the vector of all partial derivatives:
A critical point satisfies \( \nabla f = \mathbf{0} \), the zero vector. Whether that critical point is a minimum, maximum, or saddle point is determined by the Hessian matrix, which contains all second-order partial derivatives.
Numerical optimization methods are, in essence, structured ways to find a point where the gradient is zero — or close enough to zero to be acceptable — without needing to solve the gradient equation symbolically.
Every numerical optimization method implicitly relies on the same calculus principle: at a minimum, the function stops decreasing. Methods differ in how they detect that moment, how efficiently they approach it, and how reliably they avoid being fooled by a local minimum.
An objective function is the mathematical expression of what is being optimized. Defining the objective function carefully is often the most important and most underappreciated step in any optimization problem. A poorly defined objective function will produce an answer that is mathematically optimal but practically wrong.
Structure of an objective function. An objective function takes some input — a number, a vector, a set of parameters — and returns a scalar value representing performance, cost, error, likelihood, or whatever quantity the problem seeks to optimize. The choice of what to measure and how to measure it encodes the problem's values.
| Problem Domain | Typical Objective Function |
|---|---|
| Curve fitting | Sum of squared residuals between a model and data |
| Engineering design | Total weight of a structure, subject to strength requirements |
| Finance | Expected portfolio return minus a penalty for variance |
| Machine learning | Cross-entropy loss between predicted probabilities and true labels |
Properties that matter for optimization. Whether a numerical method will work well depends heavily on the properties of the objective function.
In applied problems, objective functions are frequently nonconvex and multimodal. Numerical optimization can find a local minimum efficiently, but finding the global minimum of a multimodal function generally requires specialized global search strategies — random restarts, population-based methods, or simulated annealing — that are beyond the scope of this chapter. "The optimization converged" does not mean "the global minimum was found."
The simplest numerical optimization problem is minimizing a function of a single variable over a bounded interval. Even this simple case requires care. If the function is not differentiable, or if its derivative is unavailable, methods that rely only on function values must be used.
Bracket reduction. The key idea in one-dimensional search is bracket reduction: maintaining an interval \( [a, b] \) that is known to contain a minimum, then shrinking it step by step until the minimum is located to within a desired tolerance.
For this strategy to work, the function must be unimodal on the initial interval. A function \( f \) is unimodal on \( [a, b] \) if there exists a unique minimum \( x^* \in [a, b] \) such that \( f \) is decreasing on \( [a, x^*] \) and increasing on \( [x^*, b] \). If the function is unimodal, then comparing function values at two interior points provides useful information about which side of the interval contains the minimum.
Suppose \( c \) and \( d \) are two points with \( a < c < d < b \). There are two cases.
If \( f(c) < f(d) \): the minimum must lie in \( [a, d] \), because if the function is unimodal and lower on the left interior point than the right, the minimum cannot be to the right of \( d \).
If \( f(c) > f(d) \): the minimum must lie in \( [c, b] \) by symmetric reasoning.
If \( f(c) = f(d) \): the minimum lies in \( [c, d] \).
After each comparison, the interval shrinks. Repeating this process many times locates the minimum to arbitrary precision. The question is how to choose \( c \) and \( d \) optimally to shrink the interval as fast as possible with as few function evaluations as possible. Golden section search answers this question.
Golden section search is a bracket-reduction method for one-dimensional unimodal minimization that uses the golden ratio to place interior evaluation points optimally. The key feature of golden section search is that it reduces the interval by a constant factor at each step while reusing one of the two function evaluations from the previous step.
The golden ratio. The golden ratio is
Its reciprocal is
Golden section search places the two interior points at distances \( \rho(b - a) \) from each end of the interval:
Why the golden ratio? After one comparison and interval reduction, one of the old interior points becomes an endpoint of the new interval, and the other old interior point lies exactly in the golden ratio position within the new interval. This means only one new function evaluation is needed per step, making the method efficient.
Reduction rate. Each step of golden section search reduces the interval to a fraction \( \rho \approx 0.618 \) of its previous length. After \( n \) steps, the interval has width approximately \( \rho^n (b - a) \).
Algorithm: Golden Section Search
Purpose: Find the minimum of a unimodal function f on [a, b]
Inputs: f (function), a (left endpoint), b (right endpoint), tol (tolerance)
Steps:
Set rho = (sqrt(5) - 1) / 2
Set c = a + (1 - rho) * (b - a)
Set d = a + rho * (b - a)
Evaluate fc = f(c), fd = f(d)
While (b - a) > tol:
If fc < fd:
Set b = d
Set d = c, fd = fc
Set c = a + (1 - rho) * (b - a)
Evaluate fc = f(c)
Else:
Set a = c
Set c = d, fc = fd
Set d = a + rho * (b - a)
Evaluate fd = f(d)
Return (a + b) / 2 as approximate minimizer
Stopping criterion: interval width (b - a) < tol
Output: approximate location of minimum
Reliability notes:
- Requires unimodality on [a, b]
- Does not require derivatives
- Converges linearly; each step reduces interval by factor 0.618
- Cannot distinguish local from global minimum if function is multimodalWhen the objective function is smooth and derivatives are available, Newton's method for optimization is far more efficient than golden section search. The idea is to use both the first and second derivative to model the function locally as a parabola, then jump directly to the minimum of that parabola.
Derivation. Let \( x_n \) be the current estimate of the minimizer. Near \( x_n \), the Taylor expansion of \( f \) is approximately
This quadratic approximation has its minimum where its derivative with respect to \( x \) equals zero:
Solving for \( x \):
This update is equivalent to applying Newton's root-finding method (from Chapter 3) to the equation \( f'(x) = 0 \), because finding a minimum means finding a root of the derivative.
Algorithm: Newton's Method for Optimization
Purpose: Find a local minimum of a smooth function f
Inputs: f', f'' (first and second derivatives), x0 (initial guess), tol
Steps:
Set n = 0
Repeat:
Compute f'(x_n) and f''(x_n)
If f''(x_n) <= 0: warn (not near a minimum; second derivative non-positive)
Set x_{n+1} = x_n - f'(x_n) / f''(x_n)
Set n = n + 1
Until |x_n - x_{n-1}| < tol or |f'(x_n)| < tol
Output: x_n as approximate minimizer
Reliability notes:
- Requires f'' > 0 near the minimum (positive curvature)
- Converges quadratically near a minimum when f'' is nonzero
- Can diverge or find a maximum if started poorly
- Not reliable for multimodal functions without a good starting pointConvergence. When started near a minimum where \( f''(x^*) \neq 0 \), Newton's method for optimization converges quadratically: the number of correct decimal places roughly doubles at each step. This is much faster than the linear convergence of golden section search.
Newton's method requires that \( f''(x_n) > 0 \) at each iterate. If \( f''(x_n) \leq 0 \), the method is not near a minimum; it may be near a maximum or a saddle point. The step would move toward a maximum rather than a minimum.
For functions of several variables, the gradient generalizes the derivative. The negative gradient points in the direction of steepest decrease. Gradient descent follows this direction iteratively, moving toward a local minimum step by step.
The update rule. Given a current point \( \mathbf{x}_n \), the gradient descent update is
where \( \alpha > 0 \) is the step size or learning rate, and \( \nabla f(\mathbf{x}_n) \) is the gradient vector evaluated at the current point.
Imagine standing on a hilly landscape and wanting to reach the lowest point. At each step, you look at the slope of the ground beneath you and take a step in the direction the hill descends most steeply. That is gradient descent. The step size determines how far you step each time.
For one variable. In one dimension, \( \nabla f(x) = f'(x) \), and the update becomes
Algorithm: Gradient Descent
Purpose: Find a local minimum of f by following the negative gradient
Inputs: f (objective function), grad_f (gradient function), x0 (initial point),
alpha (step size), tol (tolerance), max_iter (maximum iterations)
Steps:
Set n = 0, x = x0
Repeat:
Compute g = grad_f(x)
Set x_new = x - alpha * g
If ||x_new - x|| < tol or ||g|| < tol: stop
Set x = x_new
Set n = n + 1
If n >= max_iter: stop with warning (convergence not confirmed)
Output: x as approximate minimizer
Reliability notes:
- Convergence rate depends heavily on alpha and the problem's curvature
- Too large alpha causes oscillation or divergence; too small causes slow convergence
- Finds a local minimum; global minimum not guaranteed
- Works in any number of dimensionsConvergence. Gradient descent converges linearly when the step size is chosen well. Near the minimum, each step reduces the error by a fixed factor. Convergence is slower when the function has elongated level curves — meaning the curvature is very different in different directions. In that case, gradient descent zigzags inefficiently toward the minimum.
In training neural networks, the objective function is a loss function over millions of parameters. Gradient descent — and variants like stochastic gradient descent — repeatedly updates the parameters to reduce the loss. Understanding gradient descent is therefore essential for understanding modern machine learning.
| Step | \( x_n \) | \( f'(x_n) \) | \( x_{n+1} \) |
|---|---|---|---|
| 0 | 0.000 | -4.000 | 1.200 |
| 1 | 1.200 | -1.600 | 1.680 |
| 2 | 1.680 | -0.640 | 1.872 |
| 3 | 1.872 | -0.256 | 1.949 |
| 4 | 1.949 | -0.102 | 1.980 |
| 5 | 1.980 | -0.040 | 1.992 |
The step size \( \alpha \) in gradient descent is the single most consequential parameter in the algorithm. Choosing it well is an art as much as a science, and a great deal of practical optimization expertise concerns getting it right.
Too large a step size. If \( \alpha \) is too large, the update \( \mathbf{x}_{n+1} = \mathbf{x}_n - \alpha \nabla f(\mathbf{x}_n) \) overshoots the minimum. The iterates may oscillate across the minimum, alternating between two sides without converging. If the overshoot is severe enough, the iterates may actually diverge, moving farther from the minimum with each step.
Too small a step size. If \( \alpha \) is too small, the iterates move correctly toward the minimum but very slowly. Each step makes only tiny progress. A method that would converge in 100 steps with a good \( \alpha \) may require 100,000 steps with an \( \alpha \) that is 1000 times too small. This is wasteful computationally.
Effect of curvature. The optimal step size depends on the curvature of the objective function. If \( f''(x) \) is large (sharp curvature), a smaller \( \alpha \) is needed. If \( f''(x) \) is small (flat curvature), a larger \( \alpha \) can be used safely. For one-dimensional problems, the optimal fixed step size is \( \alpha = 1/f''(x^*) \), matching the Newton step.
Picture three plots side by side showing gradient descent trajectories on a bowl-shaped (convex) function. In the left plot, a small \( \alpha \) produces a slow but steady descent. In the middle plot, a well-chosen \( \alpha \) produces rapid convergence in a few steps. In the right plot, a large \( \alpha \) produces oscillation, jumping back and forth across the minimum.
In machine learning, the word "learning rate" is preferred over "step size," but the concept is identical. When reading about machine learning, students can translate "learning rate" directly to "step size in gradient descent."
| \( \alpha \) | Factor \( (1-2\alpha) \) | Iterates | Behavior |
|---|---|---|---|
| 0.1 | 0.8 | 4, 3.2, 2.56, 2.048, ... | Converges steadily |
| 0.9 | -0.8 | 4, -3.2, 2.56, -2.048, ... | Oscillates, but converges |
| 1.1 | -1.2 | 4, -4.8, 5.76, -6.912, ... | Diverges |
The distinction between local and global optima is among the most important ideas in optimization, and it is where many practical applications face their hardest challenges.
Local minimum. A point \( x^* \) is a local minimum of \( f \) if \( f(x^*) \leq f(x) \) for all \( x \) in some neighborhood of \( x^* \). The function is smaller at \( x^* \) than at all nearby points, but there may be distant points where the function is even smaller.
Global minimum. A point \( x^* \) is the global minimum of \( f \) if \( f(x^*) \leq f(x) \) for all \( x \) in the entire feasible region. The function is smallest everywhere, not just locally.
Convex functions. For a convex function, every local minimum is the global minimum. This is the great advantage of convexity: any optimization method that finds a local minimum has found the best possible answer. Least squares problems, linear regression, and many machine learning objectives are convex.
Nonconvex functions. For nonconvex functions, local minima may be many, scattered across the domain, and far from the global minimum. A gradient-based method started at one initial point will find one local minimum. Started at another initial point, it may find a different local minimum. Without additional information, there is no guarantee which is best.
These advanced strategies are mentioned here for awareness. This chapter focuses on methods that find local minima reliably.
Picture the graph of a function with two local minima and one global minimum on an interval. Each local minimum is marked with a dot, with the global minimum labeled clearly. The basin of attraction around each local minimum is shaded to show where gradient descent from different starting points would converge.
"The algorithm converged" is not the same as "the global minimum was found." Students should always verify whether the problem is convex before trusting a numerical minimum as globally optimal.
When the objective function depends on two or more variables, the geometry becomes richer and the algorithms more involved.
The gradient in multiple dimensions. For a function \( f(x_1, x_2, \ldots, x_n) \), the gradient
is a vector that points in the direction of steepest increase of \( f \). Moving in the direction of \( -\nabla f \) decreases \( f \) most rapidly. At a local minimum, \( \nabla f = \mathbf{0} \): all partial derivatives vanish.
Gradient descent in multiple dimensions. The gradient descent update in multiple dimensions is
where \( \mathbf{x}_n \) is now a vector and \( \nabla f(\mathbf{x}_n) \) is the gradient vector at \( \mathbf{x}_n \). This is the same formula as before, but now applied to vectors.
The Hessian matrix. The second-order information in a multivariable function is captured by the Hessian matrix, whose entry in row \( i \) and column \( j \) is the second mixed partial derivative:
The Hessian plays the role of the second derivative in the multivariable second derivative test. If the Hessian at a critical point is positive definite (all eigenvalues positive), then the critical point is a local minimum. If any eigenvalue is negative, the critical point is a saddle point.
Newton's method in multiple dimensions. The multivariable version of Newton's optimization method uses the Hessian:
This requires computing and inverting the Hessian at each step, which is computationally expensive for large \( n \) but converges very quickly near the minimum.
| Step | \( (x_n, y_n) \) | \( \nabla f(x_n, y_n) \) | \( (x_{n+1}, y_{n+1}) \) |
|---|---|---|---|
| 0 | (0, 0) | (-2, -16) | (0.20, 1.60) |
| 1 | (0.20, 1.60) | (-1.6, -3.2) | (0.36, 1.92) |
| 2 | (0.36, 1.92) | (-1.28, -0.64) | (0.488, 1.984) |
When the curvature of \( f \) differs greatly in different directions, gradient descent zigzags inefficiently toward the minimum. This is related to the conditioning concepts from Chapter 9. Methods that account for the Hessian, such as Newton's method, handle ill-conditioning much better by rescaling the gradient step according to the local curvature.
So far, all optimization problems have been unconstrained: the minimizer can be any point in the domain. Many real problems impose constraints: restrictions on which inputs are allowed.
Types of constraints. An equality constraint requires that a function of the variables equals a specified value:
An inequality constraint requires that a function of the variables satisfies an inequality:
The set of all points satisfying all constraints is the feasible region. The constrained minimum is the smallest value of \( f \) over the feasible region, which may be different from the unconstrained minimum.
Why constraints matter. In engineering, design parameters are constrained by physical limits: a beam cannot be thinner than a manufacturing minimum, a budget cannot be exceeded. In finance, a portfolio must be fully invested and individual positions cannot be negative (if short-selling is prohibited). In machine learning, regularization constraints prevent parameters from growing too large.
Lagrange multipliers (preview). For equality-constrained optimization, Lagrange multipliers provide an elegant method from calculus. At a constrained minimum of \( f \) subject to \( g = 0 \), the gradients of \( f \) and \( g \) must be parallel:
for some scalar \( \lambda \) called the Lagrange multiplier. This condition, combined with the constraint equation, gives a system of equations to solve for the minimizer and the multiplier. Numerical methods for constrained optimization extend this idea to handle inequality constraints, nonlinear constraints, and large-scale problems.
As the penalty parameter \( \mu \) increases, minimizing \( \tilde{f} \) forces the minimizer toward the feasible region. Constrained optimization is a rich subject. This preview is sufficient to recognize when a problem requires constraints and to understand that constrained problems require different algorithms than unconstrained ones.
Numerical optimization is not an abstract exercise. It is one of the most widely applied areas of mathematics. The following examples illustrate the breadth and importance of the field.
An aerospace engineer designs a wing shape to minimize drag while maintaining sufficient lift. The design variables are parameters describing the wing cross-section geometry: curvature, thickness, and angle. The objective function is drag, computed by a computational fluid dynamics simulation at each candidate design. Constraints require lift to exceed a minimum, stress to remain below material limits, and manufacturing tolerances to be satisfied. The optimization problem has dozens to hundreds of variables and nonlinear constraints. Gradient-based methods with adjoint sensitivity analysis are used to compute gradients efficiently despite the complexity of the simulation.
A fund manager selects portfolio weights \( w_1, w_2, \ldots, w_n \) for \( n \) assets to maximize expected return minus a penalty for risk:
where \( \mathbf{r} \) is the vector of expected returns, \( \Sigma \) is the covariance matrix of asset returns, and \( \lambda \) controls the trade-off between return and risk. The constraint \( \sum_i w_i = 1 \) requires full investment. This is a quadratic programming problem: the objective is quadratic and the constraints are linear. Efficient algorithms solve it in milliseconds for typical portfolio sizes.
Training a neural network means finding model parameters \( \mathbf{w} \) (weights and biases) that minimize a loss function:
where \( \ell \) measures the discrepancy between the model's prediction \( f(\mathbf{x}_i; \mathbf{w}) \) and the true label \( y_i \) for training example \( i \). The parameter vector \( \mathbf{w} \) may have millions or billions of components. Stochastic gradient descent (SGD) and its variants minimize this loss by computing gradients on random subsets (mini-batches) of the training data at each step, rather than the full dataset. This makes each gradient computation affordable even when \( N \) is enormous.
A civil engineer designs a truss structure with minimum total weight subject to stress constraints at every joint. This is a topology optimization problem: the design variables determine not just the sizes of members but their very presence in the structure. Modern topology optimization uses iterative algorithms that alternately update the design and solve the structural equations.
Scientists frequently observe data and want to find the parameters of a physical model that best explain the observations. This is a parameter estimation or inverse problem, solved by minimizing the discrepancy between model predictions and observations. Numerical optimization is the tool that makes this minimization computationally feasible.
Students learning numerical optimization encounter a characteristic set of mistakes. Awareness of these pitfalls is as important as mastery of the algorithms.
| Method | Update Rule | Requires Derivatives? | Convergence Rate |
|---|---|---|---|
| Golden Section Search | Bracket reduction using \( \rho \approx 0.618 \) | No | Linear, factor 0.618 per step |
| Newton's Method (1D) | \( x_{n+1} = x_n - f'(x_n)/f''(x_n) \) | First and second | Quadratic near minimum |
| Gradient Descent | \( \mathbf{x}_{n+1} = \mathbf{x}_n - \alpha \nabla f(\mathbf{x}_n) \) | First (gradient) | Linear, depends on \( \alpha \) and curvature |
| Newton's Method (multivariable) | \( \mathbf{x}_{n+1} = \mathbf{x}_n - H(\mathbf{x}_n)^{-1} \nabla f(\mathbf{x}_n) \) | Gradient and Hessian | Quadratic near minimum |
The checkpoint assesses command of the key ideas and skills of Chapter 11.
Chapter 11 introduced numerical optimization as a structured search for best values when symbolic calculus cannot provide an answer directly. Chapter 12 turns to a related but distinct computational challenge: simulating how a system changes over time. Students will study initial value problems, Euler's method, improved Euler method, Runge-Kutta methods, systems of differential equations, step size, local and global error, and numerical stability. Just as optimization algorithms iterate toward a best value, ODE solvers iterate forward through time, and many of the same ideas — step size selection, error analysis, and stability — return in a new computational setting.
Numerical optimization in advanced mathematics, science, and engineering. The ideas in this chapter are entry points into a vast and active field. In advanced mathematics, convex optimization studies the rich theory of optimization over convex sets and functions, with guaranteed global convergence and deep duality theory. Applications include compressed sensing, semidefinite programming, and optimal transport.
In scientific computing, large-scale optimization addresses problems with millions or billions of variables. Stochastic gradient descent, quasi-Newton methods (which approximate the Hessian without computing it explicitly), and conjugate gradient methods are standard tools for problems at this scale.
In machine learning, virtually every training procedure is an optimization algorithm. Understanding gradient descent, adaptive learning rates, and the geometry of loss landscapes is fundamental to understanding why neural networks work and how to train them effectively.
In operations research, linear programming and integer programming optimize linear and integer-valued objectives over polyhedral feasible regions. The simplex method and interior point methods solve these problems at industrial scale in logistics, scheduling, and resource allocation.
In control theory, optimal control minimizes a cost functional over the trajectory of a dynamical system, extending optimization ideas to differential equations and time evolution. This connects directly to Chapter 12's numerical ODE methods.
Students continuing in data science, engineering, computational science, or applied mathematics will encounter numerical optimization at every level. The methods introduced here — gradient descent, Newton's method, golden section search, constrained optimization — are the conceptual foundation for all that follows.