Constrained Optimization

The Central Idea

In constrained optimization, we optimize an objective only over the set of decisions that satisfy all restrictions.

A general problem can be written

The objective tells us what should become best. The constraints tell us what choices are allowed.

How Constrained Optimization Differs from Unconstrained Optimization

In an unconstrained problem, an interior differentiable optimum often satisfies

In a constrained problem, the optimal point may lie on a boundary where this condition does not hold.

The constraints can therefore change both the location of the solution and the mathematical method needed to find it.

The Feasible Set

The collection of all points satisfying every constraint is the feasible set.

If

then the feasible set is the triangular region in the first quadrant on or below

Feasibility Comes First

A candidate cannot be optimal unless it is feasible.

This gives the fundamental sequence

A point with an excellent objective value but a violated constraint is not an admissible solution.

Equality-Constrained Optimization

An equality constraint requires

For example,

restricts feasible points to a line.

In higher dimensions, equality constraints may restrict decisions to a curve, surface, or other lower-dimensional set.

Inequality-Constrained Optimization

An inequality constraint may take the form

For example,

restricts points to the disk of radius \(5\).

The interior and boundary of the region may both contain candidate optima.

Active and Inactive Constraints

An inequality constraint is active at a point when it holds with equality.

If

and

then the constraint is active.

If instead

the constraint is inactive and has slack.

Why Boundaries Matter

In constrained problems, the objective may improve as we move toward the edge of the feasible region.

The best feasible point can therefore occur where one or more constraints become active.

Boundary analysis is one of the defining features of constrained optimization.

Example: A Constraint Changes the Solution

Consider

Without constraints, the global minimum is

Now impose

The point \((0,0)\) is no longer feasible.

The constrained optimizer must lie somewhere on the line

Substitution as a Constrained Optimization Method

When an equality constraint can be solved for one variable, substitution can reduce the constrained problem to an unconstrained problem in fewer variables.

Suppose

Then

Any objective \(f(x,y)\) can now be rewritten as a one-variable function

Example: Minimizing Distance with Substitution

Minimize

subject to

Substitute

Then

Expand:

Differentiate:

Set equal to zero:

Therefore,

The constrained minimizer is

Why the Minimum Occurs at \((5,5)\)

Along the line

the point \((5,5)\) is closest to the origin.

Since

is squared distance from the origin, minimizing the objective finds the nearest feasible point.

Geometric Interpretation

Objective level sets can be compared with the feasible region.

For

the contours are circles

Minimization asks for the smallest circle that touches the feasible set.

Tangency at a Smooth Equality-Constrained Optimum

At many smooth constrained optima, an objective contour becomes tangent to the constraint curve.

The two curves then share a normal direction.

This observation leads directly to Lagrange multipliers.

Interior Candidates

If a constrained optimum lies strictly inside the feasible region, then no inequality constraint is active there.

In that case, the point behaves locally like an unconstrained optimum and often satisfies

Boundary Candidates

Boundary points require separate analysis because the feasible directions are restricted.

An objective may still be improving in the ambient space while no feasible direction allows further improvement.

Example: Boundary Maximum

Maximize

subject to

The unconstrained objective has no maximum because \(x+y\) can grow indefinitely.

The disk constraint makes the problem bounded.

The maximum occurs on the boundary

Constraints Can Create Extrema

A constrained optimum can exist even when the unconstrained problem has no optimum.

This happens because the feasible set may prevent the decision variables from moving indefinitely in an improving direction.

Constraints Can Remove Extrema

A point that is optimal in the unconstrained problem may be excluded by the constraints.

The constrained problem must then select the best remaining feasible alternative.

Candidate Sets in Constrained Optimization

Depending on the problem, relevant candidates may include:

• interior stationary points;
• points where the objective is nondifferentiable;
• boundary stationary points;
• vertices or corners;
• intersection points of active constraints;
• endpoints of feasible intervals;
• other special feasible boundary points.

Closed and Bounded Feasible Sets

If the objective is continuous and the feasible set is nonempty, closed, and bounded, then a global maximum and minimum are attained.

This is a direct consequence of the Extreme Value Theorem in higher dimensions.

Unbounded Feasible Sets

An unbounded feasible set does not automatically prevent an optimum.

For example,

has the global minimum

over the unbounded feasible set

But unbounded feasible sets require careful analysis of the objective.

Empty Feasible Sets

If the constraints contradict one another, there may be no feasible solution.

In that case,

and optimization cannot proceed until the model is revised.

Linear Constraints

Constraints such as

define half-spaces.

Intersections of linear inequalities create polygonal or polyhedral feasible regions.

These structures are central to linear programming.

Nonlinear Constraints

Constraints may also be nonlinear.

Examples include

and

Nonlinear constraints can produce curved and more complicated feasible regions.

Discrete Constraints

Some constrained problems require

or

These restrictions create discrete feasible sets and often require different optimization methods from those used in continuous calculus.

Multiple Constraints

A realistic model may contain many constraints simultaneously:

The feasible set is their intersection.

Example: Production Planning

Suppose profit is

Labor imposes

Machine capacity imposes

Production must satisfy

The complete problem is

Corner Points in Linear Problems

For a linear objective over a polygonal feasible region, an optimum—when it is attained—can be found at a vertex of the feasible region.

This is why corner-point analysis is fundamental in linear programming.

Example: Candidate Vertices

Suppose the feasible region has vertices

For the objective

evaluate:

Therefore, among these vertices,

gives the largest value.

Binding Constraints at an Optimum

At a constrained optimum, one or more resource limits are often fully used.

Such constraints satisfy equality and are called binding or active.

Their presence frequently determines the geometry of the optimal point.

Slack at an Optimum

Not every constraint must be active at an optimum.

If

then the corresponding constraint has slack.

This means some permitted capacity remains unused.

Why Some Constraints Become Active

If increasing a resource allows the objective to improve, the optimizer may push against the corresponding resource limit.

This is one reason binding constraints often have important economic or engineering interpretations.

Equality Constraints and Degrees of Freedom

Each independent equality constraint typically removes one degree of freedom.

For example, two variables subject to

effectively leave one independent variable.

This explains why substitution can reduce the dimension of a constrained problem.

Constraint Qualification Preview

More advanced constrained optimization methods require regularity conditions on the constraints.

These conditions ensure that the geometry of the feasible set is sufficiently well behaved for multiplier and derivative methods to apply.

Lagrange Multipliers Preview

For an equality-constrained problem

subject to

a smooth constrained optimum often satisfies

The scalar \(\lambda\) is called a Lagrange multiplier.

Why the Gradients Become Parallel

At a smooth constrained optimum, feasible movement occurs tangent to the constraint surface.

The objective cannot improve to first order in a feasible tangent direction.

Therefore, the gradient of the objective must align with the normal to the constraint.

Since \(\nabla g\) is normal to the constraint surface,

Inequality Constraints and Optimality Conditions

Inequality-constrained problems require distinguishing active from inactive constraints.

This leads to more general optimality conditions such as the Karush–Kuhn–Tucker conditions, often abbreviated KKT.

These extend the Lagrange multiplier idea to inequality constraints.

KKT Conditions Preview

For a minimization problem with constraints

the KKT framework introduces multipliers

Active constraints may contribute to the stationarity condition, while inactive constraints have zero multiplier under complementary slackness.

Complementary Slackness Preview

A central KKT relation is

This means either the constraint is active or its multiplier is zero.

The relation connects feasibility with marginal value.

Constrained Global Optimality

A feasible point \(\mathbf{x}^*\) is a global minimizer if

for every feasible \(\mathbf{x}\).

The comparison is made only within the feasible set.

Convexity and Constrained Optimization

When the objective is convex and the feasible set is convex, local minima become global minima.

This dramatically simplifies the interpretation of first-order conditions.

Convexity will be developed in a later lesson.

Sensitivity and Constraint Values

Constraints often contain parameters such as

Changing \(b\) changes the feasible set and can change the optimal objective value.

Studying this dependence is part of sensitivity analysis.

Economic Interpretation of Constraints

A binding resource constraint may have economic value because relaxing it slightly could improve the optimal objective.

In many optimization models, multiplier values quantify this marginal value.

Engineering Interpretation of Constraints

Constraints can represent:

• maximum stress;
• minimum strength;
• temperature limits;
• mass limits;
• power limits;
• geometric requirements;
• safety factors.

The optimizer seeks the best design that respects all such requirements.

Common Error: Optimizing Before Checking Feasibility

A stationary point of the objective is irrelevant if it violates a constraint.

Always test feasibility before accepting a candidate.

Common Error: Checking Only the Interior

Constrained extrema often occur on boundaries.

Searching only for

can therefore miss the actual solution.

Common Error: Ignoring Constraint Intersections

Corners where several constraints are active can be important candidates, especially in linear and piecewise-smooth problems.

Common Error: Treating Equality and Inequality Constraints the Same

An equality constraint restricts the solution to a boundary-like set.

An inequality constraint permits a region and may or may not be active at the optimum.

Common Error: Forgetting Domain Restrictions

The natural domain of the objective and the explicit constraints must both be satisfied.

A point can satisfy every written constraint and still be invalid if the objective is undefined there.

Common Error: Assuming Every Active Constraint Determines the Optimum

A constraint can be active without being the sole reason the point is optimal.

Classification requires the full objective-and-feasible-set structure.

A Reliable Constrained Optimization Workflow

1. Define the decision variables.
2. Write the objective function.
3. Write every equality and inequality constraint.
4. Identify the natural domain.
5. Determine the feasible set.
6. Check whether the feasible set is empty, bounded, or unbounded.
7. Find interior stationary candidates.
8. Analyze equality-constrained boundaries.
9. Analyze active inequality boundaries.
10. Check vertices, intersections, and nondifferentiable points.
11. Evaluate the objective at all relevant feasible candidates.
12. Compare values to identify local or global extrema.
13. Interpret active constraints and slack.
14. State the optimizer and optimal objective value clearly.

Practice 1: Feasibility

Determine whether

is feasible for

Practice 2: Equality Constraint

Rewrite

so that \(y\) is expressed in terms of \(x\).

Practice 3: Substitution

Minimize

subject to

Practice 4: Active Constraint

At

determine whether

is active or inactive.

Practice 5: Slack

Find the slack in

at

Practice 6: Boundary Candidate

Explain why maximizing

over

should lead us to inspect the boundary.

Practice 7: Corner Comparison

Evaluate

at

Which point gives the largest value?

Practice 8: Empty Feasible Set

Determine whether the system

has a feasible solution.

Practice 9: Build a Constrained Model

Profit is

Labor requires

Total production satisfies

Write the complete constrained maximization model including nonnegativity.

Practice 10: Identify the Error

A student solves

and stops, even though the problem contains inequality constraints.

Explain what has been omitted.

Practice Answers

Practice 1

Since

and both coordinates are nonnegative,

Practice 2

Practice 3

Substitute

Then

Differentiate:

Therefore,

The constrained minimizer is

Practice 4

At \((4,6)\),

Therefore,

Practice 5

Resource use is

Slack is

Practice 6

The objective increases in the direction

Inside the disk, movement in that direction remains possible until the boundary is reached.

Therefore the maximum must be sought on

Practice 7

The largest value is

attained at both

Practice 8

No real number satisfies both constraints.

Therefore,

Practice 9

Practice 10

The student has checked only unconstrained interior stationary points.

Boundary points, active constraints, vertices, intersections, and other feasible candidates must also be analyzed.

What to Remember

Constrained optimization seeks the best objective value among feasible decisions.

• constraints define the feasible set;
• feasibility must be checked before optimality;
• interior optima may satisfy unconstrained stationary conditions;
• boundary points can contain the true optimum;
• active constraints hold with equality;
• substitution can reduce equality-constrained problems;
• vertices and constraint intersections are important candidates;
• closed and bounded feasible sets support global existence results;
• convexity can provide global guarantees;
• Lagrange multipliers formalize smooth equality-constrained optimality.

The governing principle is

Where We Go Next

Substitution works well when an equality constraint can be solved easily, but many constrained problems involve several variables and constraints for which direct elimination becomes cumbersome.

The next lesson, Lagrange Multipliers, develops a systematic derivative- based method for locating smooth extrema subject to equality constraints.