The Central Idea
An unconstrained optimization problem asks us to maximize or minimize an objective function without imposing separate external constraints.
In one variable, the problem may be written
\[ \boxed{ \text{minimize or maximize } f(x) }. \]In several variables,
\[ \boxed{ \text{minimize or maximize } f(\mathbf{x}) }. \]The main task is to locate candidate points and determine whether they are local or global extrema.
What Does Unconstrained Mean?
Unconstrained means there are no additional conditions such as
\[ g(x)\le0 \]or
\[ h(x)=0 \]restricting the decision.
The variable is free to vary over the natural domain of the objective function.
The Natural Domain Still Matters
Unconstrained does not mean that every real number is automatically allowed.
Consider
\[ f(x)=\ln x. \]The natural domain is
\[ x>0. \]This domain restriction comes from the function itself rather than from an external constraint.
One-Variable Unconstrained Optimization
For a differentiable one-variable function, interior local extrema often occur at critical points satisfying
\[ \boxed{f'(x)=0}. \]These points are candidates that must then be classified.
Critical Points as Candidates
A critical number \(c\) occurs when
\[ f'(c)=0 \]or when \(f'(c)\) does not exist while \(f(c)\) is defined.
Optimization methods begin by locating such points because local extrema often occur there.
Example: Quadratic Minimization
Minimize
\[ f(x)=x^2-8x+20. \]Differentiate:
\[ f'(x)=2x-8. \]Set the derivative equal to zero:
\[ 2x-8=0. \]Therefore,
\[ x=4. \]Evaluate:
\[ f(4)=16-32+20=4. \]Thus the minimum candidate is
\[ \boxed{x^*=4} \]with objective value
\[ \boxed{f(x^*)=4}. \]Why the Candidate Is Global
Complete the square:
\[ f(x)=x^2-8x+20 \] \[ =(x-4)^2+4. \]Since
\[ (x-4)^2\ge0, \]we have
\[ f(x)\ge4. \]Therefore \(x=4\) is not merely a local minimum; it is the global minimum.
First-Derivative Test
The sign of the first derivative describes whether the objective is increasing or decreasing.
- If \(f'\) changes from positive to negative, there is a local maximum.
- If \(f'\) changes from negative to positive, there is a local minimum.
- If the sign does not change, the critical point is not a local extremum.
Example: First-Derivative Test
Consider
\[ f(x)=x^3-6x^2+9x. \]Differentiate:
\[ f'(x)=3x^2-12x+9. \]Factor:
\[ f'(x)=3(x-1)(x-3). \]Therefore, the critical points are
\[ x=1 \]and
\[ x=3. \]The derivative changes from positive to negative at \(x=1\), so \(x=1\) is a local maximum.
It changes from negative to positive at \(x=3\), so \(x=3\) is a local minimum.
Second-Derivative Test
If
\[ f'(c)=0, \]then curvature can help classify the point.
If
\[ f''(c)>0, \]then \(c\) is a local minimum.
If
\[ f''(c)<0, \]then \(c\) is a local maximum.
When the Second-Derivative Test Fails
If
\[ f''(c)=0, \]the test is inconclusive.
The point may be a minimum, maximum, or neither.
Another method, such as the first-derivative test or higher-order analysis, must be used.
Example: Inconclusive Second Derivative
Consider
\[ f(x)=x^4. \]Then
\[ f'(x)=4x^3 \]and
\[ f''(x)=12x^2. \]At \(x=0\),
\[ f'(0)=0 \]and
\[ f''(0)=0. \]Yet
\[ x^4\ge0, \]so \(x=0\) is the global minimum.
Nondifferentiable Optima
Unconstrained extrema can occur where the derivative does not exist.
Consider
\[ f(x)=|x|. \]At \(x=0\), the derivative does not exist, but
\[ f(x)\ge0 \]for all \(x\).
Therefore,
\[ \boxed{x=0} \]is the global minimizer.
Unbounded Objectives
An unconstrained problem may have no finite optimum.
Consider
\[ f(x)=x. \]As \(x\to\infty\),
\[ f(x)\to\infty. \]Therefore, \(f\) has no global maximum over \(\mathbb{R}\).
Asymptotic Behavior Matters
When optimizing over an unbounded domain, it is important to examine behavior as
\[ x\to\infty \]and
\[ x\to-\infty. \]Critical points alone do not reveal whether the objective ultimately escapes toward larger or smaller values.
Example: No Global Maximum
Consider
\[ f(x)=x^3-3x. \]The function has local extrema, but
\[ f(x)\to\infty \]as \(x\to\infty\), and
\[ f(x)\to-\infty \]as \(x\to-\infty\).
Therefore it has neither a global maximum nor a global minimum over all real numbers.
Completing the Square as an Optimization Method
Some quadratic objectives can be optimized algebraically.
Consider
\[ f(x)=x^2+6x+11. \]Complete the square:
\[ f(x) = (x+3)^2+2. \]Therefore,
\[ \boxed{x=-3} \]minimizes the function and
\[ \boxed{f(-3)=2}. \]Optimization by Known Function Shape
Sometimes the structure of a familiar function identifies the optimum immediately.
For example,
\[ f(x)=-(x-a)^2+b \]has a global maximum of
\[ b \]at
\[ x=a. \]Multivariable Unconstrained Optimization
For a differentiable objective
\[ f(x_1,\ldots,x_n), \]interior stationary points satisfy
\[ \boxed{ \nabla f(\mathbf{x})=\mathbf{0} }. \]This means every first partial derivative is zero.
The Gradient
The gradient of
\[ f(x_1,\ldots,x_n) \]is
\[ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ \vdots\\ \frac{\partial f}{\partial x_n} \end{bmatrix}. \]At an interior differentiable optimum, the gradient is typically zero.
Example: Two-Variable Quadratic
Minimize
\[ f(x,y)=x^2+y^2-4x+6y+20. \]Compute the partial derivatives:
\[ \frac{\partial f}{\partial x}=2x-4, \] \[ \frac{\partial f}{\partial y}=2y+6. \]Set both equal to zero:
\[ 2x-4=0, \] \[ 2y+6=0. \]Therefore,
\[ x=2, \qquad y=-3. \]The stationary point is
\[ \boxed{(2,-3)}. \]Verifying the Multivariable Minimum
Rewrite the objective:
\[ f(x,y) = (x-2)^2+(y+3)^2+7. \]Since both squared terms are nonnegative,
\[ f(x,y)\ge7. \]Equality occurs at
\[ (2,-3). \]Therefore,
\[ \boxed{(2,-3)} \]is the global minimizer and
\[ \boxed{f(2,-3)=7}. \]The Hessian Matrix
The multivariable analogue of the second derivative is the Hessian matrix.
For a function of two variables,
\[ H_f(x,y) = \begin{bmatrix} f_{xx} & f_{xy}\\ f_{yx} & f_{yy} \end{bmatrix}. \]The Hessian summarizes local curvature.
Second-Derivative Test in Two Variables
At a stationary point, define
\[ D = f_{xx}f_{yy} - (f_{xy})^2. \]Then:
- if \(D>0\) and \(f_{xx}>0\), there is a local minimum;
- if \(D>0\) and \(f_{xx}<0\), there is a local maximum;
- if \(D<0\), there is a saddle point;
- if \(D=0\), the test is inconclusive.
Example: Hessian Classification
Consider
\[ f(x,y)=x^2+2y^2. \]The gradient is
\[ \nabla f= \begin{bmatrix} 2x\\ 4y \end{bmatrix}. \]The stationary point is
\[ (0,0). \]The Hessian is
\[ H_f = \begin{bmatrix} 2 & 0\\ 0 & 4 \end{bmatrix}. \]Its curvature is positive in every direction, so
\[ \boxed{(0,0)} \]is a local minimum.
Since \(f(x,y)\ge0\), it is also global.
Positive Definite Hessians
If the Hessian is positive definite at a stationary point, the objective curves upward in every local direction.
This indicates a strict local minimum.
Negative Definite Hessians
If the Hessian is negative definite, the objective curves downward in every local direction.
This indicates a strict local maximum.
Indefinite Hessians
If the Hessian is positive in some directions and negative in others, it is indefinite.
A stationary point with an indefinite Hessian is a saddle point.
Example: Saddle Point
Consider
\[ f(x,y)=x^2-y^2. \]The stationary point is
\[ (0,0). \]The Hessian is
\[ H_f = \begin{bmatrix} 2 & 0\\ 0 & -2 \end{bmatrix}. \]Because the curvature has opposite signs in different directions,
\[ \boxed{(0,0)\text{ is a saddle point}.} \]Local Versus Global Unconstrained Solutions
Solving
\[ \nabla f=\mathbf{0} \]produces stationary candidates.
Classification determines local behavior.
Additional structure is often needed to prove that a local optimum is global.
Convexity and Global Minima
If \(f\) is convex over its domain, then every local minimum is also a global minimum.
If \(f\) is strictly convex, the global minimizer is unique when it exists.
This is one of the most important global guarantees in optimization.
Concavity and Global Maxima
If \(f\) is concave, then every local maximum is global.
Strict concavity provides uniqueness when an optimum exists.
Quadratic Objectives
A general quadratic objective can be written
\[ f(\mathbf{x}) = \frac12\mathbf{x}^TQ\mathbf{x} + \mathbf{c}^T\mathbf{x} + d. \]The matrix \(Q\) controls curvature.
When \(Q\) is positive definite, the objective is strictly convex and has at most one unconstrained global minimizer.
Solving a Quadratic Objective with Linear Algebra
For
\[ f(\mathbf{x}) = \frac12\mathbf{x}^TQ\mathbf{x} + \mathbf{c}^T\mathbf{x}, \]the gradient is
\[ \nabla f = Q\mathbf{x} + \mathbf{c} \]when \(Q\) is symmetric.
Setting the gradient equal to zero gives
\[ Q\mathbf{x}+\mathbf{c}=0. \]If \(Q\) is invertible,
\[ \boxed{ \mathbf{x}^* = -Q^{-1}\mathbf{c} }. \]Optimization and Root-Finding
Many optimization problems reduce locally to solving
\[ f'(x)=0 \]or
\[ \nabla f(\mathbf{x})=\mathbf{0}. \]Therefore, numerical root-finding methods can also play an important role in optimization.
Newton's Method for Optimization
In one variable, Newton's method for locating a stationary point uses
\[ \boxed{ x_{k+1} = x_k - \frac{f'(x_k)}{f''(x_k)} }. \]This is Newton's root-finding method applied to
\[ f'(x)=0. \]Gradient Descent Preview
For multivariable minimization, a basic iterative method is gradient descent.
Starting from \(\mathbf{x}_k\), update according to
\[ \boxed{ \mathbf{x}_{k+1} = \mathbf{x}_k - \alpha_k\nabla f(\mathbf{x}_k) }. \]The parameter \(\alpha_k>0\) is called a step size or learning rate.
Gradient-based methods are developed later in this optimization sequence.
Why Move Against the Gradient?
The gradient points in the direction of greatest local increase.
Therefore,
\[ -\nabla f \]points in the direction of greatest local decrease.
This makes it a natural search direction for minimization.
Step Size Matters
An optimization algorithm may move in a good direction but still perform poorly if the step size is badly chosen.
A step that is too small can produce slow progress.
A step that is too large can overshoot a minimum or cause instability.
Initial Conditions Matter in Nonconvex Problems
When an objective has several local minima, an iterative optimization method may converge to different points depending on the starting value.
This is one reason global optimization can be substantially harder than local optimization.
Example: Multiple Local Minima
A polynomial such as
\[ f(x)=x^4-5x^2+4 \]can contain several stationary points.
Different candidates must be classified and compared before a global minimum can be identified.
Objective Scaling
Multiplying an objective by a positive constant does not change its minimizing or maximizing points.
Thus
\[ f(x) \]and
\[ 100f(x) \]have the same unconstrained optimizers.
Equivalent Objectives
Sometimes an objective can be transformed into a simpler equivalent form.
For example, minimizing
\[ \sqrt{g(x)} \]over a domain where \(g(x)\ge0\) is equivalent to minimizing
\[ g(x) \]because the square-root function is strictly increasing.
Least-Squares Optimization
A central unconstrained optimization problem is minimizing squared error.
For residuals
\[ r_i(\boldsymbol{\theta}), \]define
\[ L(\boldsymbol{\theta}) = \sum_{i=1}^{n} r_i(\boldsymbol{\theta})^2. \]The problem is
\[ \boxed{ \min_{\boldsymbol{\theta}} L(\boldsymbol{\theta}) }. \]This structure appears throughout statistics, regression, estimation, and machine learning.
Physical Energy Minimization
Some physical systems seek configurations that minimize an energy function.
\[ \boxed{ \min_{\mathbf{x}} E(\mathbf{x}) }. \]Stationary conditions then describe possible equilibrium configurations.
Economic Profit Maximization
If profit is
\[ P(q)=R(q)-C(q), \]then an unconstrained interior optimum satisfies
\[ P'(q)=0. \]Since
\[ P'(q)=R'(q)-C'(q), \]the condition becomes
\[ \boxed{ R'(q)=C'(q) }. \]Marginal revenue equals marginal cost at an interior stationary profit point.
Common Error: Assuming Unconstrained Means No Domain
A function may still have a restricted natural domain.
Logarithms, radicals, rational expressions, and other functions can impose intrinsic restrictions.
Common Error: Stopping at \(f'(x)=0\)
Solving
\[ f'(x)=0 \]identifies candidates only.
Each candidate must still be classified and, if global optimality is claimed, compared with the rest of the domain.
Common Error: Ignoring Nondifferentiable Points
A corner or cusp can contain an optimum even though
\[ f'(x) \]does not exist there.
Common Error: Ignoring Behavior at Infinity
On an unbounded domain, asymptotic behavior may determine whether a global optimum exists.
Common Error: Treating a Saddle Point as an Optimum
In several variables,
\[ \nabla f=\mathbf{0} \]may describe a saddle point rather than a maximum or minimum.
Curvature must be examined.
Common Error: Assuming a Local Minimum Is Global
Without additional structure such as convexity, a local minimum may be worse than another minimum elsewhere.
A Reliable One-Variable Workflow
- Identify the natural domain of the objective.
- Compute \(f'(x)\).
- Find points where \(f'(x)=0\).
- Include points where \(f'\) does not exist but \(f\) does.
- Classify candidates using sign changes, curvature, or direct comparison.
- Inspect behavior near domain boundaries.
- Inspect behavior as \(x\to\pm\infty\) when relevant.
- Determine whether global extrema exist.
- State both the optimizer and the optimal value.
A Reliable Multivariable Workflow
- Identify the natural domain.
- Compute the gradient \(\nabla f\).
- Solve \(\nabla f=\mathbf{0}\).
- Compute the Hessian when appropriate.
- Classify stationary points.
- Check for nondifferentiable candidates.
- Study global structure such as convexity or concavity.
- Inspect behavior at large distances if the domain is unbounded.
- Compare candidate objective values when necessary.
- State the optimizing point and objective value.
Practice 1: Quadratic Minimum
Minimize
\[ f(x)=x^2-10x+29. \]Practice 2: Quadratic Maximum
Maximize
\[ f(x)=-2x^2+8x+3. \]Practice 3: Critical Points
Find the critical points of
\[ f(x)=x^3-3x^2-9x. \]Practice 4: Nondifferentiable Minimum
Find the global minimum of
\[ f(x)=|x+5|. \]Practice 5: Unbounded Objective
Determine whether
\[ f(x)=2x+1 \]has a global maximum or minimum over \(\mathbb{R}\).
Practice 6: Two-Variable Stationary Point
Find the stationary point of
\[ f(x,y)=x^2+y^2-6x+8y+30. \]Practice 7: Saddle Point
Classify the stationary point of
\[ f(x,y)=x^2-4y^2. \]Practice 8: Profit Optimization
Suppose
\[ R(q)=80q-q^2 \]and
\[ C(q)=20q+100. \]Find the stationary profit level.
Practice 9: Gradient
Compute the gradient of
\[ f(x,y)=3x^2+2xy+y^2. \]Practice 10: Identify the Error
A student solves
\[ \nabla f=\mathbf{0} \]and declares every resulting point to be a global minimum.
Explain the error.
Practice Answers
Practice 1
Differentiate:
\[ f'(x)=2x-10. \]Set equal to zero:
\[ x=5. \]Then
\[ f(5)=25-50+29=4. \]Therefore,
\[ \boxed{x^*=5,\qquad f(x^*)=4}. \]Practice 2
Differentiate:
\[ f'(x)=-4x+8. \]Thus
\[ x=2. \]Evaluate:
\[ f(2)=-8+16+3=11. \]Therefore,
\[ \boxed{x^*=2,\qquad f(x^*)=11}. \]Practice 3
Differentiate:
\[ f'(x)=3x^2-6x-9. \]Factor:
\[ f'(x)=3(x-3)(x+1). \]Therefore, the critical points are
\[ \boxed{x=-1,\qquad x=3}. \]Practice 4
Since
\[ |x+5|\ge0, \]the global minimum occurs at
\[ \boxed{x=-5} \]with value
\[ \boxed{0}. \]Practice 5
Because the linear function increases without bound as \(x\to\infty\) and decreases without bound as \(x\to-\infty\),
\[ \boxed{\text{no global maximum or minimum exists}.} \]Practice 6
Compute
\[ \frac{\partial f}{\partial x}=2x-6, \] \[ \frac{\partial f}{\partial y}=2y+8. \]Therefore,
\[ x=3, \qquad y=-4. \]The stationary point is
\[ \boxed{(3,-4)}. \]Practice 7
The stationary point is
\[ (0,0). \]The function curves upward in the \(x\)-direction and downward in the \(y\)-direction.
Therefore,
\[ \boxed{(0,0)\text{ is a saddle point}.} \]Practice 8
Profit is
\[ P(q)=R(q)-C(q). \]Thus
\[ P(q) = 80q-q^2-(20q+100) \] \[ = 60q-q^2-100. \]Differentiate:
\[ P'(q)=60-2q. \]Set equal to zero:
\[ q=30. \]Therefore, the stationary profit level occurs at
\[ \boxed{q=30}. \]Practice 9
The partial derivatives are
\[ f_x=6x+2y, \] \[ f_y=2x+2y. \]Therefore,
\[ \boxed{ \nabla f = \begin{bmatrix} 6x+2y\\ 2x+2y \end{bmatrix} }. \]Practice 10
Solving
\[ \nabla f=\mathbf{0} \]identifies stationary candidates only.
A stationary point may be a local minimum, local maximum, saddle point, or other degenerate point.
Global minimality requires additional analysis such as convexity, comparison, bounds, or other global structure.
What to Remember
Unconstrained optimization studies extrema of an objective without separate equality or inequality constraints.
- the natural domain still matters;
- one-variable stationary points often satisfy \(f'(x)=0\);
- multivariable stationary points often satisfy \(\nabla f=\mathbf{0}\);
- critical points are candidates, not automatic optima;
- the first and second derivatives classify local behavior;
- the Hessian describes multivariable curvature;
- saddle points are stationary but not optimal;
- asymptotic behavior matters on unbounded domains;
- convexity can convert a local minimum into a global guarantee;
- numerical methods often search iteratively for stationary or improving points.
The central pattern is
\[ \boxed{ \text{find candidates} \;\rightarrow\; \text{classify them} \;\rightarrow\; \text{determine globality} }. \]Where We Go Next
Unconstrained optimization allows the objective to determine the search without separate external restrictions.
Many practical problems, however, impose budgets, capacities, equality requirements, physical limits, and other conditions that restrict which stationary points are admissible.
The next lesson, Constrained Optimization, develops the general strategy for optimizing an objective over a feasible region and prepares the way for Lagrange multipliers and other constrained methods.