MGU Mathematics Series — Numerical Methods
Part II: Roots, Interpolation, and Approximation of Functions
Splines · Least Squares · Curve Fitting · Residuals · Normal Equations
Chapter 5 of 14
5
MGU Mathematics Series  ·  Numerical Methods  ·  Library Textbook Edition

Chapter 5:
Splines, Curve Fitting,
and Least Squares

Piecewise polynomial smoothness and the mathematics of fitting curves to noisy data — when a perfect match is not the right answer

■ Part II of V  ·  Roots, Interpolation, and Approximation ■

■ 14 Sections  •  Chapter 5 of 14 ■

Prerequisite: Chapter 4 (Interpolation and Polynomial Approximation)

Chapter 5 — Front Matter

Chapter Introduction and Learning Objectives

Chapter 4 introduced interpolation as the construction of a polynomial that passes through every given data point exactly. That approach works well in many situations, but it carries hidden risks: a high-degree polynomial may oscillate wildly between data points, the model may be sensitive to small changes in the data, and when data contain measurement noise, forcing a curve to pass through every point exactly is not modeling at all — it is memorizing noise.

This chapter extends the idea of approximation beyond exact interpolation. Students will study two major families of methods: splines, which join low-degree polynomial pieces smoothly across intervals, and least squares fitting, which finds a curve that passes close to data without passing through every point exactly. Both approaches reflect a mature understanding of what approximation means: not perfect agreement with every datum, but a smooth, reliable, interpretable model that captures the structure of the data.

Opening Question

A structural engineer measures the deflection of a bridge beam at five equally spaced points along its length and obtains the values 0, 2.31, 3.84, 3.15, and 0 mm. She needs a smooth curve describing deflection at every point between the measurements. Should she use the degree-4 interpolating polynomial that passes through all five points exactly? Or would a different model serve her better? This chapter answers that question — and along the way develops the mathematical ideas that make modern curve fitting and data modeling possible.

14Sections
10Objectives
15Key Terms
14Total Chapters

Learning Objectives

By the end of this chapter, you should be able to:

  1. 1Explain why exact polynomial interpolation is not always the best approximation method.
  2. 2Construct and evaluate linear splines from given data.
  3. 3Describe the properties that quadratic and cubic splines are designed to achieve.
  4. 4Define residuals and explain what they measure.
  5. 5State the least squares criterion and explain its mathematical meaning.
  6. 6Set up and solve a linear least squares problem for a line or low-degree polynomial.
  7. 7Interpret the normal equations as the system that minimizes the sum of squared residuals.
  8. 8Distinguish between overfitting and underfitting and explain the tradeoffs involved.
  9. 9Identify common mistakes in curve fitting and explain how to avoid them.
  10. 10Apply the problem-solving method — Understand, Model, Choose Method, Compute, Estimate Error, Check, Explain — to spline and least squares problems.
Chapter 5 — Contents

Sections at a Glance

Part II — Chapter 5  ·  Sections 5.1 – 5.5
Splines: Piecewise Polynomial Smoothness
Knots · Linear Splines · Quadratic Splines · Cubic Splines · Natural Boundary Conditions
Section 5.1

Why Exact Interpolation Is Not Always Best

Interpolation constructs a function that passes through every given data point exactly. This is appropriate when the data points themselves are exact — when they represent precise values of a known function, or when the measurement process is highly controlled and essentially error-free.

But in many practical situations, data are not exact. Physical measurements carry experimental error. Sensor readings fluctuate. Survey data are rounded. Economic observations are imprecise estimates. When data carry error, insisting that a curve pass through every data point exactly means forcing the model to reproduce the error faithfully — which is precisely the wrong thing to do.

Even when data are exact, polynomial interpolation faces a structural problem: the degree of the interpolating polynomial grows with the number of data points. Ten data points require a degree-9 polynomial. Fifty data points require a degree-49 polynomial. High-degree polynomials are mathematically unstable and tend to oscillate between data points, sometimes producing approximations that are wildly inaccurate at intermediate values. Chapter 4 introduced this phenomenon as Runge's phenomenon.

These problems suggest two important alternative strategies, both of which are developed in this chapter.

Strategy One
Splines
Keep the degree of each polynomial piece low — typically degree 1, 2, or 3 — and join many pieces together smoothly. The spline passes through the data points exactly but avoids the oscillation problems of high-degree global polynomials.
Strategy Two
Least Squares Fitting
Allow the curve to miss data points by small residuals, seeking instead a smooth model that captures the trend in the data rather than every individual measurement. This is the right approach when data are noisy.

Understanding when each strategy is appropriate requires thinking carefully about the source, quality, and purpose of the data — a habit of mind that separates mature numerical reasoning from algorithmic button-pushing.

Fundamental Question

Before choosing a numerical approximation method, ask: Are my data exact or noisy? If the data are exact and a smooth curve through all points is needed, use a spline. If the data carry measurement error and a trend model is needed, use least squares fitting. The answer shapes every subsequent decision.

Section 5.2

Piecewise Approximation

The fundamental idea behind splines is simple: rather than using a single polynomial of high degree over the entire interval, divide the interval into smaller subintervals and use a separate low-degree polynomial on each subinterval.

Suppose data points are given at \( x_0 < x_1 < x_2 < \cdots < x_n \). The interval \([x_0, x_n]\) is divided into \(n\) subintervals:

\[ [x_0, x_1], \quad [x_1, x_2], \quad \ldots, \quad [x_{n-1}, x_n] \]

On each subinterval \([x_{i-1}, x_i]\), a separate polynomial \(S_i(x)\) is used. Collectively, these polynomials form the piecewise polynomial approximation. The points \(x_0, x_1, \ldots, x_n\) where the pieces join are called knots.

At each knot, adjacent polynomial pieces must agree on the function value — they must both pass through the data point at that knot. Depending on the type of spline, they may also be required to agree on the derivative and the second derivative. Enforcing these smoothness conditions at the knots is what makes a spline feel smooth rather than jagged.

Piecewise approximation is extremely flexible. By using many short subintervals with low-degree pieces and strong smoothness conditions, any smooth function can be well approximated without the oscillation risks of high-degree global polynomials.

Section 5.3

Linear Splines

The simplest piecewise polynomial approximation is the linear spline, also called the piecewise linear interpolant. On each subinterval \([x_{i-1}, x_i]\), the linear spline is simply the straight line connecting the two data points \((x_{i-1}, y_{i-1})\) and \((x_i, y_i)\). The formula on subinterval \([x_{i-1}, x_i]\) is:

\[ S_i(x) = y_{i-1} + \frac{y_i - y_{i-1}}{x_i - x_{i-1}} (x - x_{i-1}) \]

This is the point-slope form of the line through the two endpoints of the subinterval.

Property
Exact at Data Points
Each piece passes through both of its endpoint data values: \(S_i(x_{i-1}) = y_{i-1}\) and \(S_i(x_i) = y_i\).
Property
Continuous
The pieces join without gaps, since each piece is defined to match the data value at both of its endpoints.
Limitation
Not Differentiable at Knots
The slope changes abruptly at each knot. The linear spline has corners at the data points. If the underlying function is smooth, the corners indicate modeling error.

Linear splines are easy to construct and useful for many purposes — approximating integrals (as in the trapezoidal rule), displaying data, and basic interpolation. When a smoother approximation is needed, quadratic or cubic splines are used instead.

Example 5.3.1
Constructing a linear spline and evaluating it at interior points
  • Problem Data are given at three points: \((1, 2)\), \((3, 6)\), \((5, 3)\). Construct the linear spline and evaluate it at \(x = 2\) and \(x = 4\).
  • Think The linear spline connects consecutive data points with straight lines. There are two subintervals: \([1,3]\) and \([3,5]\). A different formula applies on each subinterval.
  • Method Compute the slope between the two endpoints of each subinterval and write the point-slope line.
  • Compute On \([1,3]\): slope \(= (6-2)/(3-1) = 2\), so \(S_1(x) = 2 + 2(x-1)\).
    On \([3,5]\): slope \(= (3-6)/(5-3) = -3/2\), so \(S_2(x) = 6 - \tfrac{3}{2}(x-3)\).

    At \(x = 2\): \(S_1(2) = 2 + 2(1) = 4\).
    At \(x = 4\): \(S_2(4) = 6 - \tfrac{3}{2}(1) = 4.5\).
  • Check At the knot \(x = 3\): \(S_1(3) = 2 + 2(2) = 6\) ✓ and \(S_2(3) = 6 - 0 = 6\) ✓. The spline is continuous at the knot.
  • Interpret Between \(x=1\) and \(x=3\) the function increases at a constant rate; at \(x=2\) the estimated value is 4. Between \(x=3\) and \(x=5\) the function decreases; at \(x=4\) the estimate is 4.5. The linear spline captures the overall shape but has a corner at \(x=3\), which a smooth underlying function would not have.
Section 5.4

Quadratic Splines as an Introduction

Linear splines are continuous but not smooth — they have corners at each knot. When a smooth curve is needed, the pieces must agree not only on function values at the knots but also on slopes.

A quadratic spline uses a quadratic polynomial on each subinterval. Given \(n\) subintervals, there are \(n\) quadratic pieces, each determined by 3 coefficients, for a total of \(3n\) unknowns. The conditions imposed are:

  • Data fit: each piece passes through both endpoint data values — \(2n\) equations.
  • Continuity of first derivative: at each interior knot \(x_1, \ldots, x_{n-1}\), adjacent pieces have equal slopes — \(n-1\) equations.

This gives \(2n + (n-1) = 3n - 1\) equations for \(3n\) unknowns. One degree of freedom remains, and is typically resolved by specifying the slope of the first piece at the leftmost data point.

Quadratic splines are smoother than linear splines but still have a limitation: the second derivative (curvature) may change abruptly at knots. For applications where smooth curvature is important — mechanical design, animation, geometric modeling — cubic splines are preferred.

Section 5.5

Cubic Splines as an Introduction

The cubic spline is the most widely used spline in scientific and engineering computing. It uses a cubic polynomial on each subinterval and enforces continuity of the function value, the first derivative, and the second derivative at each interior knot.

Given \(n+1\) data points defining \(n\) subintervals, the cubic spline consists of \(n\) cubic pieces. Each piece on \([x_{i-1}, x_i]\) has the form:

\[ S_i(x) = a_i + b_i(x - x_{i-1}) + c_i(x - x_{i-1})^2 + d_i(x - x_{i-1})^3 \]

with 4 unknowns per piece, giving \(4n\) unknowns total. The conditions are:

ConditionDescriptionCount
Data fit at left endpoints\(S_i(x_{i-1}) = y_{i-1}\) for each piece\(n\)
Data fit at right endpoints\(S_i(x_i) = y_i\) for each piece\(n\)
Continuity of first derivative\(S_i'(x_i) = S_{i+1}'(x_i)\) at interior knots\(n-1\)
Continuity of second derivative\(S_i''(x_i) = S_{i+1}''(x_i)\) at interior knots\(n-1\)
Total so far\(4n - 2\)

Two additional conditions are needed. The most common choice produces the natural cubic spline, which requires zero curvature at the endpoints:

\[ S_1''(x_0) = 0 \qquad \text{and} \qquad S_n''(x_n) = 0 \]

With these conditions, the cubic spline is fully determined. Computing it requires solving a tridiagonal linear system — a system that is both well-conditioned and efficiently solvable (Chapter 9).

Why Cubic Splines Are Preferred

They are smooth through the second derivative, so they look smooth to the eye and to measuring instruments. Each piece is only degree 3, so they avoid the oscillation of high-degree polynomials. The resulting linear system is well-conditioned and tridiagonal, solvable in O(n) operations. Cubic splines are optimal: among all twice-differentiable functions that pass through the data, the natural cubic spline minimizes the integral of the squared second derivative — a measure of total curvature.

Physical Interpretation

The word "spline" comes from a tool used by draftsmen and shipbuilders before the age of computers: a thin, flexible strip of wood or metal pinned to pass through a set of points. The strip naturally took on the shape that minimized its internal bending energy — which is mathematically equivalent to the natural cubic spline. When you compute a cubic spline, you are reproducing exactly this physical behavior in a mathematical model.

Sections 5.6 – 5.11
Least Squares: Fitting Curves to Noisy Data
Residuals · Least Squares Criterion · Normal Equations · Linear and Polynomial Regression
Section 5.6

Curve Fitting

Splines are useful when the data are precise and a smooth interpolating curve is needed. But many real datasets do not meet this requirement. Physical measurements contain experimental uncertainty. Economic data are estimates. Survey measurements are rounded or biased. Sensor readings fluctuate around the true value.

When data are noisy, the goal changes. Rather than finding a curve that passes through every data point exactly, the goal is to find a curve that fits the trend in the data — passing close to the points without reproducing every fluctuation. This is the problem of curve fitting.

Curve fitting asks: given a collection of data points \((x_1, y_1), (x_2, y_2), \ldots, (x_m, y_m)\), find a function \(f(x)\) from some family (lines, polynomials, exponentials, and so on) that best represents the overall relationship.

The key difference from interpolation is this: the number of data points \(m\) may be much larger than the number of parameters in the model, so it will generally be impossible to find a model that passes through every data point exactly. Instead, the model passes close to all data points, with the definition of "close" made precise by a criterion.

Section 5.7

Residuals

When a model \(f(x)\) is fitted to data, the residual for each data point measures how far the model is from the observed value. For data point \((x_i, y_i)\) and model prediction \(\hat{y}_i = f(x_i)\), the residual is:

\[ r_i = y_i - \hat{y}_i = y_i - f(x_i) \]

A positive residual means the model underpredicts at that point; a negative residual means it overpredicts; a zero residual means the model passes exactly through the data point.

The full collection of residuals \(r_1, r_2, \ldots, r_m\) describes the error of the model across all data. A good fit produces residuals that are small and scattered without obvious patterns. A poor fit produces residuals that are large, or that show a systematic pattern — such as all positive in one region and all negative in another — indicating that the model is missing structure in the data.

Residuals play the same role in curve fitting that absolute error plays in root-finding and interpolation: they are the direct measure of how well the model agrees with the data.

Section 5.8

Least Squares Criterion

Given residuals \(r_1, r_2, \ldots, r_m\), how should they be combined into a single measure of goodness of fit? Consider three options:

CriterionFormulaProblem
Sum of residuals \(\sum r_i\) Positive and negative residuals cancel; a model can fit poorly yet have zero sum.
Sum of absolute values \(\sum |r_i|\) Avoids cancellation, but the absolute value is not differentiable everywhere, making optimization harder.
Sum of squares \(\sum r_i^2\) No cancellation, weights large residuals more heavily, produces a smooth differentiable objective. This is the least squares criterion.

The least squares criterion chooses the model parameters that minimize the sum of squared residuals:

\[ \text{Minimize} \quad S = \sum_{i=1}^m r_i^2 = \sum_{i=1}^m \left(y_i - f(x_i)\right)^2 \]

This principle — minimize the sum of squared residuals — is one of the most important ideas in all of applied mathematics. It is the foundation of linear regression, of least squares approximation in numerical analysis, and of many optimization methods in data science and statistics.

The least squares criterion has a natural geometric interpretation: among all curves in the chosen family, the least squares fit is the one that is closest to the data in the sense of minimizing the total squared vertical distance from the data points to the curve.

Section 5.9

Linear Least Squares

The simplest and most important case of least squares fitting is fitting a straight line to data. Given \(m\) data points, the goal is to find constants \(a\) and \(b\) such that the line \(f(x) = a + bx\) minimizes:

\[ S(a, b) = \sum_{i=1}^m \left(y_i - a - bx_i\right)^2 \]

To minimize \(S\), take partial derivatives with respect to \(a\) and \(b\) and set them equal to zero. Rearranging the resulting two equations gives the normal equations for linear least squares:

\[ ma + b\sum_{i=1}^m x_i = \sum_{i=1}^m y_i \] \[ a\sum_{i=1}^m x_i + b\sum_{i=1}^m x_i^2 = \sum_{i=1}^m x_i y_i \]

This is a \(2 \times 2\) linear system in the unknowns \(a\) and \(b\). Solving it gives the least squares line. The solutions can be written in closed form. Let \(\bar{x}\) and \(\bar{y}\) denote the sample means. Then:

\[ b = \frac{\displaystyle\sum_{i=1}^m (x_i - \bar{x})(y_i - \bar{y})}{\displaystyle\sum_{i=1}^m (x_i - \bar{x})^2}, \qquad a = \bar{y} - b\bar{x} \]
Example 5.9.1
Fitting a least squares line to velocity measurements
  • Problem A physics experiment records velocity at five times: \(t = 0, 1, 2, 3, 4\) s with velocities \(v = 1.2, 3.8, 6.1, 8.3, 10.9\) m/s. Fit a least squares line \(v = a + bt\) and estimate velocity at \(t = 5\) s.
  • Think The data appear to increase approximately linearly. Since there are 5 data points and only 2 parameters, the line will generally not pass through all points exactly. Least squares gives the best-fitting line in the sum-of-squares sense.
  • Compute \(\bar{t} = 2\), \(\bar{v} = 6.06\).

    \(\displaystyle\sum (t_i - \bar{t})^2 = 4 + 1 + 0 + 1 + 4 = 10\)

    \(\displaystyle\sum (t_i - \bar{t})(v_i - \bar{v}) = 9.72 + 2.26 + 0 + 2.24 + 9.68 = 23.90\)

    \(b = 23.90 / 10 = 2.39\), \(\quad a = 6.06 - (2.39)(2) = 1.28\)

    Least squares line: \(v = 1.28 + 2.39t\)
  • Check
    \(t\)Observed \(v\)Predicted \(\hat{v}\)Residual
    01.21.28−0.08
    13.83.670.13
    26.16.060.04
    38.38.45−0.15
    410.910.840.06
    Residuals are small and show no systematic pattern — the fit is good.
  • Interpret The slope \(b = 2.39\) m/s² estimates the constant acceleration. The intercept \(a = 1.28\) m/s estimates the initial velocity. At \(t = 5\): \(v = 1.28 + 2.39(5) = 13.23\) m/s. This is an extrapolation — it assumes the linear trend continues, which may not hold if the object decelerates or reaches a terminal velocity.
Reliability Note: Extrapolation

Predictions beyond the range of the data (extrapolation) are less reliable than predictions within the range (interpolation). Always state clearly when a prediction involves extrapolation, and consider whether the model relationship is likely to continue outside the observed range.

Section 5.10

Polynomial Least Squares

The least squares principle extends naturally to polynomials of any degree \(d\). The model is:

\[ f(x) = c_0 + c_1 x + c_2 x^2 + \cdots + c_d x^d \]

The goal is to choose the coefficients \(c_0, c_1, \ldots, c_d\) to minimize:

\[ S = \sum_{i=1}^m \left(y_i - c_0 - c_1 x_i - c_2 x_i^2 - \cdots - c_d x_i^d\right)^2 \]

Taking partial derivatives and setting them to zero produces a system of \(d+1\) linear equations. In matrix form, the polynomial least squares problem seeks the coefficient vector \(\mathbf{c}\) satisfying:

\[ \mathbf{A}^T \mathbf{A} \, \mathbf{c} = \mathbf{A}^T \mathbf{y} \]

where \(\mathbf{A}\) is the design matrix with entries \(A_{ij} = x_i^{j-1}\) and \(\mathbf{y} = (y_1, \ldots, y_m)^T\). This system — the normal equations in matrix form — is always solvable when the \(x\)-values are distinct and \(d < m\).

The choice of degree \(d\) is a modeling decision. If \(d\) is too small, the model may not capture the structure of the data. If \(d\) is too large, the model may fit noise rather than the underlying relationship. These failure modes are examined in Section 5.12.

Section 5.11

Normal Equations as an Introduction

The phrase "normal equations" appears in many contexts across mathematics, statistics, and numerical analysis. Understanding their geometric meaning connects these contexts.

In general, the least squares problem asks for the vector \(\mathbf{c}\) that minimizes \(\| \mathbf{A}\mathbf{c} - \mathbf{y} \|^2\) — the squared length of the residual vector. When the system \(\mathbf{A}\mathbf{c} = \mathbf{y}\) is overdetermined (more equations than unknowns, as is typical in curve fitting), there is no exact solution. The least squares solution minimizes the residual norm.

The key result is that the least squares solution satisfies the normal equations \(\mathbf{A}^T \mathbf{A} \, \mathbf{c} = \mathbf{A}^T \mathbf{y}\). The name comes from the geometric fact that the residual vector \(\mathbf{y} - \mathbf{A}\mathbf{c}\) is normal (perpendicular) to the column space of \(\mathbf{A}\) at the least squares solution. The least squares fit is the point in the model space closest to the data.

Computational Note on Conditioning

Although the normal equations can be solved directly using Gaussian elimination (Chapter 9), the matrix \(\mathbf{A}^T \mathbf{A}\) can be ill-conditioned when the polynomial degree is high or when the \(x\)-values are clustered. For high-quality computation, numerically stable approaches such as QR decomposition are preferred. For this chapter, direct solution of the normal equations is appropriate for low-degree models with well-distributed data.

Example 5.11.1
Setting up the normal equations for a quadratic least squares fit
  • Problem Data: \((1, 2.1),\; (2, 3.9),\; (3, 8.2),\; (4, 14.8)\). Set up the normal equations for \(f(x) = c_0 + c_1 x + c_2 x^2\).
  • Think This is a quadratic polynomial least squares problem with \(m = 4\) data points and \(d = 2\), giving 3 unknown coefficients. The normal equations will be a \(3 \times 3\) symmetric linear system.
  • Compute Design matrix: \(\mathbf{A} = \begin{pmatrix} 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \\ 1 & 4 & 16 \end{pmatrix}\)

    \(\mathbf{A}^T \mathbf{A} = \begin{pmatrix} 4 & 10 & 30 \\ 10 & 30 & 100 \\ 30 & 100 & 354 \end{pmatrix}, \quad \mathbf{A}^T \mathbf{y} = \begin{pmatrix} 29.0 \\ 88.5 \\ 296.5 \end{pmatrix}\)
  • Check The matrix \(\mathbf{A}^T \mathbf{A}\) is symmetric (as it always must be) and its entries are sums of powers of the \(x\)-values, which is consistent with the expected structure.
  • Interpret Solving this \(3\times 3\) system by Gaussian elimination (Chapter 9) gives the coefficients \(c_0, c_1, c_2\) of the best-fitting parabola. The quadratic model is appropriate here because the data values increase in a clearly curved, accelerating pattern.
Sections 5.12 – 5.14
Reliability, Mistakes, and Transition
Overfitting · Underfitting · Common Errors · Bridge to Numerical Calculus
Section 5.12

Overfitting and Underfitting

One of the most important judgments in curve fitting is choosing the degree of the polynomial model. This choice involves a fundamental tradeoff.

Underfitting occurs when the model is too simple to capture the structure of the data. A linear model fitted to data that follow a curve will produce large, systematic residuals — positive in the middle of the range and negative at the ends, or vice versa. The pattern in the residuals reveals that the model is missing structure.

Overfitting occurs when the model is too complex relative to the data. A degree-9 polynomial fitted to 10 noisy data points will pass through every point exactly (or nearly so), but it will oscillate wildly between data points. The model has absorbed the noise in the data and will not generalize reliably to new observations.

Failure Mode
Underfitting
Model is too simple. Residuals are large and show a systematic pattern — all positive in one region, all negative in another. Increasing the model degree will substantially reduce the sum of squared residuals.
Failure Mode
Overfitting
Model is too complex. Residuals are small for the training data but the fitted curve oscillates wildly. The model fits noise, not structure. Adding a new data point would dramatically change the fit.
Goal
Good Fit
Residuals are small and show no systematic pattern. The model is the simplest one that captures the genuine structure of the data. Further complexity does not substantially reduce the sum of squared residuals.

The key principle is: choose the simplest model that captures the genuine structure of the data. In practice, this means plotting the data and the fitted curve together, plotting the residuals against \(x\), and increasing the polynomial degree only as long as it significantly reduces the sum of squared residuals. When further increases in degree produce diminishing improvements, stop.

Common Misconception

In curve fitting, a perfect fit is not necessarily a good fit. If a polynomial of degree \(m-1\) is fitted to \(m\) data points, it passes through every point exactly — but this is just polynomial interpolation. The model has no ability to distinguish the true relationship from the noise in the measurements. When fitting noisy data, always prefer a lower-degree model that still captures the trend.

Section 5.13

Common Curve Fitting Mistakes

Mistake 1
Interpolating noisy data.
When data contain measurement error, using an interpolating polynomial or spline that passes through every point exactly will reproduce the noise. Use least squares fitting instead, choosing a degree appropriate to the underlying relationship.
Mistake 2
Choosing a model degree based on the number of data points rather than the structure of the data.
The degree of the fitting polynomial should reflect the physical or mathematical relationship being modeled (linear motion gives a linear model; constant acceleration gives a quadratic model), not merely the count of available data points.
Mistake 3
Extrapolating far beyond the data without qualification.
Least squares models are valid within or near the range of the data. Predicting values far outside this range assumes the model relationship continues to hold, which is often unjustified. Always note when a prediction involves extrapolation.
Mistake 4
Ignoring the residual pattern.
Always plot the residuals against \(x\). A systematic pattern — all positive in one region, alternating signs, or a visible curve — means the model is inadequate. Random-looking scatter around zero indicates a good fit.
Mistake 5
Treating a low sum of squared residuals as proof of a good model.
A low sum of squared residuals means the model fits the current data well, but does not guarantee the model is physically meaningful, correctly specified, or predictively reliable on new data.
Mistake 6
Using high-degree polynomial least squares without checking conditioning.
The matrix \(\mathbf{A}^T \mathbf{A}\) can be very ill-conditioned when the polynomial degree is high. Direct solution of the normal equations may give inaccurate coefficients. Prefer low-degree models, or use numerically stable alternatives such as QR decomposition.
Mistake 7
Confusing the least squares line with the true physical relationship.
A least squares line is a mathematical model that minimizes squared vertical deviations. It is not a physical law. Its interpretation depends entirely on the context, and it carries no causal force by itself.
Section 5.14

Preparing for Numerical Differentiation and Integration

Splines and least squares fitting are not only approximation tools in their own right — they are also building blocks for the numerical calculus methods developed in Part III.

The trapezoidal rule for numerical integration (Chapter 7) is exactly the integral of a linear spline fitted to function values. Simpson's rule is the integral of a quadratic spline. Understanding linear and quadratic splines, therefore, is not a detour from numerical integration — it is the conceptual foundation for it. When Part III asks how much error the trapezoidal rule makes, the answer comes directly from the analysis of how well a linear spline approximates a smooth curve over each subinterval.

Similarly, the finite difference formulas for numerical differentiation (Chapter 6) can be derived by differentiating an interpolating polynomial fitted to nearby function values. The accuracy and error analysis of these formulas follows directly from the interpolation error theory of Chapter 4 and the smoothness ideas of this chapter.

Least squares fitting connects forward to linear algebra (Chapter 9) through the normal equations, and to optimization (Chapter 11) through the general problem of minimizing an objective function. Students who understand how the normal equations arise from minimizing the sum of squared residuals will find that the gradient descent and Newton methods of Chapter 11 are natural extensions of exactly the same idea.

Chapter 5 — Back Matter

Chapter Summary

Key Terms

spline
A piecewise polynomial function designed to be smooth at the points where the pieces join.
knot
A point where two adjacent polynomial pieces of a spline meet.
linear spline
A piecewise linear function that interpolates data by connecting data points with straight line segments; continuous but not differentiable at knots.
quadratic spline
A piecewise quadratic polynomial function with continuous slope at knots.
cubic spline
A piecewise cubic polynomial function with continuous slope and curvature at knots; the most widely used spline type.
natural cubic spline
A cubic spline with the additional condition that the second derivative is zero at the two endpoints.
curve fitting
The process of finding a mathematical function that approximates the relationship described by data, allowing nonzero residuals.
residual
\(r_i = y_i - f(x_i)\); the difference between an observed data value and the value predicted by the model.
least squares criterion
The principle that the best-fitting curve minimizes \(\sum r_i^2\), the sum of the squares of the residuals.
normal equations
The linear system \(\mathbf{A}^T \mathbf{A}\,\mathbf{c} = \mathbf{A}^T \mathbf{y}\) whose solution gives the least squares coefficients.
design matrix
The matrix \(\mathbf{A}\) with entries \(A_{ij} = x_i^{j-1}\) used to set up polynomial least squares problems.
overfitting
Using a model that is too complex, so that it fits the noise in the data rather than the underlying structure; does not generalize to new data.
underfitting
Using a model that is too simple, so that it misses real structure in the data; residuals show a systematic pattern.
extrapolation
Predicting values of the model outside the range of the data; less reliable than interpolation within the data range.
condition number
A measure of how sensitive a linear system is to small changes in its inputs; relevant to the reliability of normal equation solutions for high-degree polynomial fits.
PracticeConcept Check
  1. Explain in your own words the difference between interpolation and least squares fitting. When would you choose each approach?
  2. What is a knot in a spline? What conditions must be satisfied at a knot for a linear spline? For a cubic spline?
  3. Why are cubic splines preferred over high-degree global polynomials for smooth curve fitting through precise data?
  4. What is a residual? If a residual is positive, what does that tell you about the model's prediction at that data point?
  5. State the least squares criterion. Why is the sum of squares used rather than the sum of absolute values or the sum of residuals?
  6. What are the normal equations, and what mathematical problem do they solve?
  7. What is overfitting? What is underfitting? How would you recognize each in a residual plot?
PracticeSkill Practice
  1. Data are given at the points \((0, 1), (2, 4), (4, 2), (6, 5)\). Construct the linear spline and evaluate it at \(x = 1\), \(x = 3\), and \(x = 5\).
  2. Verify that the linear spline from Problem 8 is continuous at \(x = 2\) and \(x = 4\).
  3. For the data in Problem 8, would a linear spline or a cubic spline be more appropriate if the underlying function is physically smooth? Explain.
  4. Data: \((1, 3.2), (2, 5.1), (3, 7.3), (4, 9.8), (5, 11.5)\). (a) Compute the least squares line. (b) Find the residuals at each data point. (c) Estimate the value at \(x = 6\). Note any reservations about this estimate.
  5. Set up (but do not solve) the normal equations for a least squares quadratic fit \(f(x) = c_0 + c_1 x + c_2 x^2\) to the data \((0,1), (1,3), (2,8), (3,20)\).
PracticeAlgorithm Practice
Algorithm Linear Spline Evaluation
PurposeEvaluate the linear spline at a query point \(x^*\), given sorted knots \(x_0 < x_1 < \cdots < x_n\) and corresponding values \(y_0, y_1, \ldots, y_n\).
InputsArrays \(x[0..n]\) and \(y[0..n]\); query point \(x^*\) with \(x_0 \le x^* \le x_n\).
Steps
  1. If \(x^* = x_n\), return \(y_n\). (Boundary case.)
  2. Find the index \(i\) such that \(x_{i-1} \le x^* < x_i\). (Scan or binary search.)
  3. Compute slope: \(m = (y_i - y_{i-1}) / (x_i - x_{i-1})\).
  4. Return \(y_{i-1} + m \cdot (x^* - x_{i-1})\).
OutputThe linear spline value at \(x^*\).
ReliabilityThe result is exact at knots. Error at interior points depends on the curvature of the underlying function and the width of the subinterval.
  1. Trace through the algorithm above for the data \((0,0), (2,4), (4,3)\) and the query point \(x^* = 3\). Show each step.
  2. Write pseudocode for computing the least squares line coefficients \(a\) and \(b\) from a dataset of \(m\) points using the closed-form formulas derived in Section 5.9. Include computation of the sample means as explicit steps.
PracticeComputational Interpretation
  1. A student fits a least squares line to 20 data points. She notices the first 10 residuals are all negative and the last 10 are all positive (sorted by \(x\)). What does this pattern tell her about the model?
  2. A cubic spline is constructed through 6 data points on \([0, 5]\). How many subintervals are used? How many cubic pieces? How many total coefficients must be determined? List all the conditions that uniquely determine the natural cubic spline.
  3. Explain why a polynomial of degree \(m-1\) fitted to \(m\) data points by least squares passes through every data point exactly. Why is this a problem if the data contain measurement noise?
  4. Two students fit polynomials to the same dataset: Student A uses degree 2, Student B uses degree 6. Student B's sum of squared residuals is much smaller. Is Student B's model necessarily better? Explain.
PracticeApplications
  1. Structural Engineering. A bridge beam's deflection is measured at 6 equally spaced points along its 10-meter length, giving deflections (in millimeters) of \(0, 3.2, 5.8, 5.9, 3.4, 0\). The deflection must be modeled by a smooth curve that is zero at both endpoints and has its maximum near the center. Which type of spline would you recommend? Justify your answer with reference to the conditions at endpoints and knots.
  2. Economics. A researcher fits a least squares line to data on household income \(x\) (thousands of dollars) and annual spending on education \(y\) (thousands of dollars) for 15 families. The slope is 0.12 and the intercept is 0.5. Interpret these values in economic terms. What does the slope represent? What prediction does the model make for a family earning $80,000 per year? State any reservations about this prediction.
  3. Physics. A ball is dropped from rest. Heights (in meters) are recorded at \(t = 0, 0.2, 0.4, 0.6, 0.8, 1.0\) s: \(10.0, 9.80, 9.22, 8.24, 6.86, 5.10\) m. Fit a least squares quadratic to this data. Compare the coefficient of \(t^2\) to the value expected from free-fall physics (\(-\frac{1}{2}g = -4.9\)). Does the data support the physical model?
  4. Data Science. A data scientist fits models of degree 1 through 5 to a dataset of 50 points. The sums of squared residuals are 452, 118, 103, 99, and 97. At what degree would you stop increasing complexity? Justify your choice in terms of diminishing returns and the risk of overfitting.
PracticeError Analysis
  1. Explain how the conditioning of the normal equations worsens as the polynomial degree increases for a fixed dataset. What practical problem can this cause when computing the least squares coefficients?
  2. A researcher uses a degree-8 polynomial to fit 9 noisy data points. Explain why the result is equivalent to polynomial interpolation and why this fails to model the underlying relationship.
  3. For the linear spline in Example 5.3.1, the underlying function is actually \(f(x) = x^2 - x + 2\). Compute the true function values at \(x = 2\) and \(x = 4\) and compare them to the spline approximations. What does the error tell you about the adequacy of a linear spline for this function?
  4. A least squares line is computed for data with \(m = 100\) points. One data point is far from the fitted line. Explain qualitatively how this outlier affects the slope and intercept, and why outlier detection before fitting is important.

Concept Review Questions

  1. What are the two major limitations of exact polynomial interpolation that motivate splines and least squares fitting?
  2. How does a cubic spline differ from the high-degree interpolating polynomial? What specific smoothness conditions does the cubic spline enforce that the interpolating polynomial does not?
  3. What additional two conditions make a cubic spline uniquely determined? What is the natural spline choice?
  4. Why is the sum of squared residuals minimized rather than the sum of residuals?
  5. Describe in words what the normal equations represent geometrically. What does it mean for the residual vector to be normal to the column space?
  6. Give two practical examples where least squares fitting is more appropriate than polynomial interpolation.
  7. Define overfitting and underfitting. What kind of residual patterns indicate each problem?
  8. Why does conditioning of the normal equations worsen as polynomial degree increases? What does poor conditioning imply about the computed coefficients?
Methods Reference — Chapter 5
Linear Spline on Subinterval \([x_{i-1}, x_i]\)
\[ S_i(x) = y_{i-1} + \frac{y_i - y_{i-1}}{x_i - x_{i-1}}(x - x_{i-1}) \]
Residual
\[ r_i = y_i - f(x_i) \]
Least Squares Criterion
\[ \text{Minimize} \quad S = \sum_{i=1}^m r_i^2 \]
Least Squares Line Coefficients
\[ b = \frac{\displaystyle\sum_{i=1}^m (x_i - \bar{x})(y_i - \bar{y})}{\displaystyle\sum_{i=1}^m (x_i - \bar{x})^2}, \qquad a = \bar{y} - b\bar{x} \]
Normal Equations (Matrix Form)
\[ \mathbf{A}^T \mathbf{A} \, \mathbf{c} = \mathbf{A}^T \mathbf{y}, \qquad A_{ij} = x_i^{j-1} \]
Chapter 5 Checkpoint
Confirm your understanding before moving to Chapter 6
Part A — Concepts
  • I can explain why exact polynomial interpolation may be inappropriate for noisy data or many data points.
  • I can define a spline and explain what a knot is and what conditions are satisfied there.
  • I can construct a linear spline from given data and evaluate it at interior points.
  • I can state the smoothness conditions for cubic splines and explain what the natural boundary conditions require.
  • I can define a residual and explain how residual patterns indicate underfitting or overfitting.
  • I can state the least squares criterion and explain why the sum of squares is preferred.
Part B — Calculations
  • I can construct the linear spline through the data \((1,4), (3,8), (5,3)\) and evaluate it at \(x = 2\) and \(x = 4\).
  • I can fit a least squares line to five or more data points, compute residuals, and comment on fit quality.
  • I can set up (but need not solve) the normal equations for a quadratic least squares fit to four data points.
Part C — Analysis
  • I can look at a table of sums of squared residuals for increasing polynomial degree and recommend an appropriate stopping degree.
  • I can explain why solving the normal equations for a high-degree polynomial fit may be poorly conditioned and what this implies.
  • I can identify which curve fitting mistakes are present in a described scenario and explain how to correct them.
Preview — Chapter 6
Numerical Differentiation

Chapter 5 completed Part II by developing splines and least squares as approximation tools. Part III now turns to numerical calculus — computing derivatives and integrals from discrete data or function values, when symbolic formulas are unavailable. Chapter 6 introduces finite differences: formulas that estimate the derivative of a function from nearby function values. These formulas can be derived by differentiating interpolating polynomials, connecting directly to the approximation ideas of Chapters 4 and 5. Students will see that numerical differentiation is sensitive to step size and to noise in the data — and that making the step size smaller does not always improve the approximation, because floating-point errors eventually dominate. Understanding this tradeoff is one of the central insights of numerical calculus.

Bridge Note — Chapter 5 to Advanced Topics

Numerical Integration (Chapter 7) uses linear and quadratic splines as the basis for the trapezoidal and Simpson's rules. The error analysis of these methods follows directly from the interpolation error theory in Chapter 4 and the smoothness ideas here.

Numerical Linear Algebra (Chapter 9) provides the tools for solving the normal equations stably and efficiently. The tridiagonal system for cubic spline coefficients is also solved there.

Numerical Optimization (Chapter 11) generalizes the least squares principle. Minimizing the sum of squared residuals is a special case of optimization; the normal equations are the first-order optimality conditions for that special case.

Data Science and Machine Learning build on least squares fitting as a foundational primitive. Linear regression, polynomial regression, ridge regression, and regularized models all minimize variants of the sum of squared residuals.

MGU Library Connection
  • Numerical Methods — Chapter 4: Interpolation and Polynomial Approximation (prerequisite)
  • Numerical Methods — Chapter 9: Numerical Linear Algebra (solving the normal equations and the cubic spline system)
  • Numerical Methods — Chapter 11: Numerical Optimization (generalization of the least squares criterion)
  • Appendix I: Least Squares Reference
  • Appendix J: Numerical Differentiation Formula Sheet (builds on spline differentiation)
  • Companion: MGU Probability and Statistics — Linear Regression and Statistical Modeling
  • Companion: MGU Scientific Computing — Splines and Curve Fitting in Practice