Chapter 11 — Front Matter
Purpose and Overview
This chapter introduces one of the most geometrically beautiful ideas in linear algebra: the structure of perpendicularity in vector spaces. You already understand perpendicular lines from coordinate geometry. In this chapter, you will extend that idea into higher dimensions, where two vectors can be perpendicular regardless of how many components they have. From perpendicularity, you will develop the idea of projection — the mathematical process of finding the closest vector in a given direction or subspace. Projection is the bridge between exact answers and approximate ones, and it appears throughout data science, physics, statistics, computer graphics, and machine learning.
By the end of this chapter, you should understand why orthogonality is not just a geometric nicety but a powerful structural tool, and why projections allow linear algebra to solve problems that have no exact solution.
Opening Question
Suppose you are a scientist collecting measurements of a physical process. You record many data points, and you want to fit a straight line through them. But the data does not fall perfectly on any line — no line passes through all the points at once. How do you find the best line?
This problem cannot be solved by substitution or elimination. The system of equations you would write to force every point onto a line is inconsistent. But linear algebra gives you a way to find the closest possible solution. The answer uses projection: you project the data onto the space of all possible linear relationships and find the vector in that space nearest to your data. This is least squares, and it rests entirely on orthogonality and projection.
14Sections
11Objectives
20+Key Terms
12+Worked Examples
Why This Chapter Matters
| Field | Role of Orthogonality and Projection |
|---|
| Geometry | Perpendicular vectors, distances, and angles in any dimension |
| Data fitting | Least squares regression; fitting lines and planes to data without calculus |
| Computer graphics | Decomposing motion, lighting, and coordinates into independent directions |
| Physics readiness | Resolving forces and velocities into perpendicular components |
| Machine learning | Principal components, feature decorrelation, signal decomposition |
| Statistics | Residuals, variance decomposition, linear regression |
| Signal processing | Decomposing signals into orthogonal frequency components |
Learning Objectives
- 1Compute dot products of vectors in \(\mathbb{R}^n\) and interpret the result geometrically.
- 2Find the length of a vector and the distance between two vectors.
- 3Use the dot product to compute the angle between two vectors.
- 4Identify orthogonal vectors and verify orthogonality.
- 5Define and identify orthogonal sets and orthogonal bases.
- 6Find the orthogonal complement of a subspace.
- 7Compute the projection of a vector onto a line (a one-dimensional subspace).
- 8Compute the projection of a vector onto a subspace using the projection formula.
- 9Describe what least squares means and find the least squares solution to an inconsistent system.
- 10Apply the Gram-Schmidt process to convert a linearly independent set into an orthogonal basis.
- 11Identify and avoid common errors in orthogonality and projection problems.
Sections 11.1 – 11.3
The Dot Product: Length, Distance, and Angle
Dot Product · Norm · Unit Vectors · Distance · Angle Formula
Section 11.1
Dot Product Review
You first encountered the dot product in Chapter 2 when studying vectors in two and three dimensions. Before extending these ideas, it is worth reviewing the dot product carefully, because every major idea in this chapter depends on it.
Definition — The Dot Product
Let \(\mathbf{u}\) and \(\mathbf{v}\) be vectors in \(\mathbb{R}^n\). The dot product of \(\mathbf{u}\) and \(\mathbf{v}\) is the scalar:
$$\mathbf{u} \cdot \mathbf{v} = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n$$
Notice that the dot product produces a number, not a vector. In matrix notation: \(\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^T \mathbf{v}\).
The dot product satisfies the following algebraic properties for all vectors \(\mathbf{u}\), \(\mathbf{v}\), \(\mathbf{w}\) in \(\mathbb{R}^n\) and all scalars \(c\):
| Property | Statement |
|---|
| Commutativity | \(\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}\) |
| Distributivity | \(\mathbf{u} \cdot (\mathbf{v} + \mathbf{w}) = \mathbf{u} \cdot \mathbf{v} + \mathbf{u} \cdot \mathbf{w}\) |
| Scalar factoring | \((c\mathbf{u}) \cdot \mathbf{v} = c(\mathbf{u} \cdot \mathbf{v})\) |
| Dot with itself | \(\mathbf{v} \cdot \mathbf{v} \geq 0\), and \(\mathbf{v} \cdot \mathbf{v} = 0\) only when \(\mathbf{v} = \mathbf{0}\) |
Inner Products
The dot product is the most familiar example of an inner product. In later mathematics, you will meet inner products on function spaces and polynomial spaces — objects where "vectors" are functions, and the inner product is an integral. The dot product on \(\mathbb{R}^n\) is the finite-dimensional model for all of them.
Example 11.1.1
Computing a dot product in \(\mathbb{R}^3\)
- ProblemFind \(\mathbf{u} \cdot \mathbf{v}\) for \(\mathbf{u} = \begin{pmatrix}3\\-1\\2\end{pmatrix}\) and \(\mathbf{v} = \begin{pmatrix}4\\5\\-2\end{pmatrix}\).
- ThinkMultiply corresponding components and sum. Three terms, one for each component.
- Calculate$$\mathbf{u} \cdot \mathbf{v} = (3)(4) + (-1)(5) + (2)(-2) = 12 - 5 - 4 = 3$$
- InterpretThe dot product is positive, which means the vectors form an acute angle. We will make this precise in Section 11.3.
Answer: \(\mathbf{u} \cdot \mathbf{v} = 3\)
Example 11.1.2
Dot product in \(\mathbb{R}^4\)
- ProblemFind \(\mathbf{a} \cdot \mathbf{b}\) for \(\mathbf{a} = \begin{pmatrix}1\\0\\-3\\2\end{pmatrix}\) and \(\mathbf{b} = \begin{pmatrix}2\\7\\1\\-1\end{pmatrix}\).
- Calculate$$\mathbf{a} \cdot \mathbf{b} = (1)(2) + (0)(7) + (-3)(1) + (2)(-1) = 2 + 0 - 3 - 2 = -3$$
- InterpretThe dot product is negative. The two vectors in \(\mathbb{R}^4\) form an obtuse angle. Even though we cannot draw \(\mathbb{R}^4\), the dot product still gives meaningful geometric information.
Answer: \(\mathbf{a} \cdot \mathbf{b} = -3\)
Section 11.2
Length and Distance
Definition — Length (Norm) of a Vector
The length or norm of a vector \(\mathbf{v}\) in \(\mathbb{R}^n\) is:
$$\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}} = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2}$$
This extends the Pythagorean distance formula into any number of dimensions. A unit vector has length exactly 1. To normalize: \(\hat{\mathbf{v}} = \dfrac{\mathbf{v}}{\|\mathbf{v}\|}\). The distance between \(\mathbf{u}\) and \(\mathbf{v}\) is \(d(\mathbf{u},\mathbf{v}) = \|\mathbf{u} - \mathbf{v}\|\).
Example 11.2.1
Finding length and a unit vector
- ProblemFind the length of \(\mathbf{v} = \begin{pmatrix}2\\-3\\6\end{pmatrix}\) and find the unit vector in the direction of \(\mathbf{v}\).
- Calculate$$\|\mathbf{v}\| = \sqrt{(2)^2 + (-3)^2 + (6)^2} = \sqrt{4 + 9 + 36} = \sqrt{49} = 7$$
The unit vector: \(\hat{\mathbf{v}} = \dfrac{1}{7}\begin{pmatrix}2\\-3\\6\end{pmatrix} = \begin{pmatrix}2/7\\-3/7\\6/7\end{pmatrix}\)
- CheckVerify the unit vector has length 1: \(\|\hat{\mathbf{v}}\| = \sqrt{\frac{4+9+36}{49}} = \sqrt{\frac{49}{49}} = 1\) ✓
Answer: \(\|\mathbf{v}\| = 7\); unit vector: \(\bigl(2/7,\,-3/7,\,6/7\bigr)\)
Example 11.2.2
Distance between two vectors
- ProblemFind the distance between \(\mathbf{u} = \begin{pmatrix}1\\4\end{pmatrix}\) and \(\mathbf{v} = \begin{pmatrix}5\\1\end{pmatrix}\).
- Calculate$$\mathbf{u} - \mathbf{v} = \begin{pmatrix}-4\\3\end{pmatrix}, \quad d(\mathbf{u},\mathbf{v}) = \left\|\begin{pmatrix}-4\\3\end{pmatrix}\right\| = \sqrt{16+9} = \sqrt{25} = 5$$
- InterpretThe two points \((1,4)\) and \((5,1)\) are 5 units apart. This matches the standard coordinate-geometry distance formula.
Answer: \(d(\mathbf{u},\mathbf{v}) = 5\)
Section 11.3
Angles Between Vectors
The Angle Formula
For any nonzero vectors \(\mathbf{u}\) and \(\mathbf{v}\) in \(\mathbb{R}^n\), the angle \(\theta\) between them satisfies:
$$\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}, \quad 0 \leq \theta \leq \pi$$
The sign of the dot product tells you the type of angle: positive → acute, negative → obtuse, zero → exactly \(90°\) (orthogonal).
Example 11.3.1
Finding the angle between two vectors
- ProblemFind the angle between \(\mathbf{u} = \begin{pmatrix}1\\2\end{pmatrix}\) and \(\mathbf{v} = \begin{pmatrix}3\\1\end{pmatrix}\).
- Calculate$$\mathbf{u}\cdot\mathbf{v} = 5, \quad \|\mathbf{u}\| = \sqrt{5}, \quad \|\mathbf{v}\| = \sqrt{10}$$
$$\cos\theta = \frac{5}{\sqrt{5}\cdot\sqrt{10}} = \frac{5}{\sqrt{50}} = \frac{5}{5\sqrt{2}} = \frac{1}{\sqrt{2}}$$
$$\theta = \arccos\!\left(\tfrac{1}{\sqrt{2}}\right) = 45°$$
- InterpretThe two vectors make a \(45°\) angle. Draw them from the origin on a coordinate grid and confirm they look roughly like this.
Answer: \(\theta = 45°\)
Diagram
Draw \(\mathbf{u} = (1,2)\) as an arrow from the origin pointing up and to the right. Draw \(\mathbf{v} = (3,1)\) as an arrow from the origin pointing to the right and slightly up. The angle between them, measured at the origin where both arrows begin, is 45°.
- Compute \(\mathbf{u}\cdot\mathbf{v}\) for \(\mathbf{u}=\begin{pmatrix}2\\-3\end{pmatrix}\), \(\mathbf{v}=\begin{pmatrix}4\\5\end{pmatrix}\).
- Compute \(\mathbf{u}\cdot\mathbf{v}\) for \(\mathbf{u}=\begin{pmatrix}1\\0\\-2\end{pmatrix}\), \(\mathbf{v}=\begin{pmatrix}3\\7\\-1\end{pmatrix}\).
- Find \(\|\begin{pmatrix}5\\12\end{pmatrix}\|\) and the unit vector in its direction.
- Find \(\|\begin{pmatrix}3\\0\\4\end{pmatrix}\|\) and the unit vector in its direction.
- Find the distance between \(\begin{pmatrix}1\\2\\3\end{pmatrix}\) and \(\begin{pmatrix}4\\6\\3\end{pmatrix}\).
- Find the angle between \(\begin{pmatrix}1\\1\end{pmatrix}\) and \(\begin{pmatrix}1\\-1\end{pmatrix}\). Is it acute, right, or obtuse?
Sections 11.4 – 11.7
Orthogonality: Perpendicular Structure in Vector Spaces
Orthogonal Vectors · Orthogonal Sets · Orthogonal Bases · Orthogonal Complements
Section 11.4
Orthogonal Vectors
Definition — Orthogonality
Two vectors \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal if and only if:
$$\mathbf{u} \cdot \mathbf{v} = 0$$
Orthogonal means perpendicular in the sense of the dot product. In \(\mathbb{R}^2\) and \(\mathbb{R}^3\), orthogonal vectors are geometrically perpendicular. The zero vector is orthogonal to every vector, but this is a trivial case. In applications, "orthogonal" typically refers to nonzero vectors.
If \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal, the Pythagorean Theorem generalizes:
Pythagorean Theorem for Vectors
If \(\mathbf{u} \cdot \mathbf{v} = 0\), then:
$$\|\mathbf{u} + \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2$$
Proof: \(\|\mathbf{u}+\mathbf{v}\|^2 = (\mathbf{u}+\mathbf{v})\cdot(\mathbf{u}+\mathbf{v}) = \|\mathbf{u}\|^2 + 2(\mathbf{u}\cdot\mathbf{v}) + \|\mathbf{v}\|^2\). When \(\mathbf{u}\cdot\mathbf{v}=0\), the middle term vanishes.
Example 11.4.1
Verifying orthogonality
- ProblemAre \(\mathbf{u} = \begin{pmatrix}3\\4\\0\end{pmatrix}\) and \(\mathbf{v} = \begin{pmatrix}-4\\3\\0\end{pmatrix}\) orthogonal?
- Calculate$$\mathbf{u} \cdot \mathbf{v} = (3)(-4) + (4)(3) + (0)(0) = -12 + 12 + 0 = 0$$
- InterpretYes, they are orthogonal. In the \(xy\)-plane they are perpendicular. Notice that \(\mathbf{u}\) has slope \(4/3\) and \(\mathbf{v}\) has slope \(-3/4\) — these are negative reciprocals, confirming perpendicularity.
Answer: Yes — \(\mathbf{u}\cdot\mathbf{v} = 0\) ✓
Example 11.4.2
Finding a vector orthogonal to a given vector
- ProblemFind a nonzero vector in \(\mathbb{R}^2\) orthogonal to \(\mathbf{v} = \begin{pmatrix}5\\-2\end{pmatrix}\).
- ThinkWe need \(\mathbf{u} = \begin{pmatrix}a\\b\end{pmatrix}\) such that \(5a - 2b = 0\).
- SolveChoose \(a = 2\), then \(b = 5\). So \(\mathbf{u} = \begin{pmatrix}2\\5\end{pmatrix}\).
- Check\((5)(2) + (-2)(5) = 10 - 10 = 0\) ✓
- InterpretThe vectors \((5,-2)\) and \((2,5)\) are perpendicular. One has slope \(-2/5\) and the other has slope \(5/2\) — negative reciprocals.
Answer: \(\mathbf{u} = \begin{pmatrix}2\\5\end{pmatrix}\) (or any nonzero scalar multiple)
Section 11.5
Orthogonal Sets
Definition — Orthogonal and Orthonormal Sets
A set \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}\) is an orthogonal set if \(\mathbf{v}_i \cdot \mathbf{v}_j = 0\) whenever \(i \neq j\).
It is an orthonormal set if, additionally, every vector has length 1.
Key fact: Every orthogonal set of nonzero vectors is automatically linearly independent.
The independence proof uses orthogonality cleanly. Suppose \(c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0}\). Take the dot product of both sides with \(\mathbf{v}_1\). All cross terms vanish because the set is orthogonal, leaving \(c_1\|\mathbf{v}_1\|^2 = 0\). Since \(\mathbf{v}_1\) is nonzero, \(c_1 = 0\). The same argument forces every coefficient to zero. This result means that building an orthogonal set automatically builds an independent set — orthogonality is a stronger condition than independence alone.
Example 11.5.1
Verifying an orthogonal set
- ProblemShow that \(\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}\) is an orthogonal set, where \(\mathbf{v}_1 = \begin{pmatrix}1\\1\\0\end{pmatrix}\), \(\mathbf{v}_2 = \begin{pmatrix}1\\-1\\0\end{pmatrix}\), \(\mathbf{v}_3 = \begin{pmatrix}0\\0\\1\end{pmatrix}\).
- Check all pairs$$\mathbf{v}_1\cdot\mathbf{v}_2 = (1)(1)+(1)(-1)+(0)(0) = 0 \checkmark$$
$$\mathbf{v}_1\cdot\mathbf{v}_3 = 0 \checkmark \quad \mathbf{v}_2\cdot\mathbf{v}_3 = 0 \checkmark$$
- InterpretAll three pairs are orthogonal, so the set is orthogonal. It is not orthonormal because \(\|\mathbf{v}_1\| = \|\mathbf{v}_2\| = \sqrt{2} \neq 1\). To normalize, divide \(\mathbf{v}_1\) and \(\mathbf{v}_2\) each by \(\sqrt{2}\).
Answer: Orthogonal ✓ but not orthonormal (\(\mathbf{v}_1\) and \(\mathbf{v}_2\) have length \(\sqrt{2}\))
Section 11.6
Orthogonal Bases
An orthogonal basis for a subspace \(W\) is a basis for \(W\) that is also an orthogonal set. An orthonormal basis is an orthogonal basis where every vector has length 1. The standard basis vectors \(\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_n\) for \(\mathbb{R}^n\) form the most familiar orthonormal basis.
When you have an orthogonal basis \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}\) for a subspace \(W\), you can find the coordinates of any vector \(\mathbf{w}\) in \(W\) using simple dot product formulas — no row reduction required:
Coordinates via an Orthogonal Basis
If \(\mathbf{w}\) is in \(W\) and \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}\) is an orthogonal basis for \(W\):
$$\mathbf{w} = \frac{\mathbf{w}\cdot\mathbf{v}_1}{\|\mathbf{v}_1\|^2}\mathbf{v}_1 + \frac{\mathbf{w}\cdot\mathbf{v}_2}{\|\mathbf{v}_2\|^2}\mathbf{v}_2 + \cdots + \frac{\mathbf{w}\cdot\mathbf{v}_k}{\|\mathbf{v}_k\|^2}\mathbf{v}_k$$
For an orthonormal basis \(\{\mathbf{q}_1,\ldots,\mathbf{q}_k\}\), the denominators are all 1: each coordinate is simply \(\mathbf{w}\cdot\mathbf{q}_j\).
Section 11.7
Orthogonal Complements
Definition — Orthogonal Complement
Let \(W\) be a subspace of \(\mathbb{R}^n\). The orthogonal complement of \(W\), written \(W^\perp\) ("W perp"), is:
$$W^\perp = \{\mathbf{v} \in \mathbb{R}^n : \mathbf{v} \cdot \mathbf{w} = 0 \text{ for all } \mathbf{w} \in W\}$$
Key properties: \(\dim(W) + \dim(W^\perp) = n\); \(\;W \cap W^\perp = \{\mathbf{0}\}\); \(\;(W^\perp)^\perp = W\).
Every vector in \(\mathbb{R}^n\) can be written uniquely as \(\mathbf{v} = \mathbf{w} + \mathbf{w}^\perp\) where \(\mathbf{w}\in W\) and \(\mathbf{w}^\perp \in W^\perp\). This is the orthogonal decomposition and is the foundation of projection.
Connection to Fundamental Subspaces
The null space of \(A\) is the orthogonal complement of the row space of \(A\). The left null space of \(A\) is the orthogonal complement of the column space of \(A\). These relationships reveal deep structural symmetry in linear systems — they appeared in Chapter 10 and reappear in least squares.
Example 11.7.1
Finding the orthogonal complement of a subspace
- ProblemLet \(W = \operatorname{span}\left\{\begin{pmatrix}1\\2\\3\end{pmatrix}\right\}\) in \(\mathbb{R}^3\). Describe \(W^\perp\).
- ThinkWe need all \(\mathbf{v} = \begin{pmatrix}x\\y\\z\end{pmatrix}\) such that \(\mathbf{v} \cdot \begin{pmatrix}1\\2\\3\end{pmatrix} = 0\).
- Equation$$x + 2y + 3z = 0$$
- Interpret\(W^\perp\) is the plane through the origin with normal vector \((1,2,3)\). Since \(\dim(W)=1\) and \(n=3\), we have \(\dim(W^\perp) = 3-1 = 2\). ✓
Answer: the plane \(x + 2y + 3z = 0\) — a two-dimensional subspace of \(\mathbb{R}^3\)
Diagram
Draw a single line through the origin in \(\mathbb{R}^3\) in the direction of \((1,2,3)\) — this is \(W\). The orthogonal complement \(W^\perp\) is the plane perpendicular to this line, passing through the origin. Every vector in \(\mathbb{R}^3\) can be uniquely decomposed into one component along the line and one component in the plane.
- Determine whether each pair is orthogonal: \(\begin{pmatrix}1\\-3\\2\end{pmatrix}\) and \(\begin{pmatrix}4\\2\\1\end{pmatrix}\).
- Determine whether each pair is orthogonal: \(\begin{pmatrix}2\\0\\-1\end{pmatrix}\) and \(\begin{pmatrix}1\\5\\2\end{pmatrix}\).
- Find a nonzero vector in \(\mathbb{R}^3\) orthogonal to both \(\begin{pmatrix}1\\0\\1\end{pmatrix}\) and \(\begin{pmatrix}0\\1\\1\end{pmatrix}\).
- Is \(\left\{\begin{pmatrix}1\\0\\1\end{pmatrix},\begin{pmatrix}1\\0\\-1\end{pmatrix},\begin{pmatrix}0\\1\\0\end{pmatrix}\right\}\) an orthogonal set? Is it orthonormal?
- If \(W\) is a 2-dimensional subspace of \(\mathbb{R}^5\), what is \(\dim(W^\perp)\)?
- Describe \(W^\perp\) geometrically when \(W\) is a two-dimensional subspace of \(\mathbb{R}^3\).
Sections 11.8 – 11.9
Projections: The Closest Vector in a Given Direction
Projection onto Lines · Projection Matrix · Projection onto Subspaces
Section 11.8
Projections onto Lines
Projection is the process of finding the "shadow" of one vector onto another. Imagine shining a light perpendicular to the direction of a vector \(\mathbf{u}\). The shadow that \(\mathbf{v}\) casts along \(\mathbf{u}\) is the projection of \(\mathbf{v}\) onto \(\mathbf{u}\). More precisely: among all scalar multiples of \(\mathbf{u}\), which one is closest to \(\mathbf{v}\)? The answer is the projection.
The Projection Formula
The projection of \(\mathbf{v}\) onto the line spanned by \(\mathbf{u}\):
$$\operatorname{proj}_{\mathbf{u}} \mathbf{v} = \frac{\mathbf{v} \cdot \mathbf{u}}{\|\mathbf{u}\|^2}\,\mathbf{u}$$
The error vector (residual) \(\mathbf{z} = \mathbf{v} - \operatorname{proj}_{\mathbf{u}}\mathbf{v}\) is perpendicular to \(\mathbf{u}\): \(\mathbf{z}\cdot\mathbf{u} = 0\).
The decomposition \(\mathbf{v} = \operatorname{proj}_{\mathbf{u}}\mathbf{v} + \mathbf{z}\) splits \(\mathbf{v}\) into a component along \(\mathbf{u}\) and a component perpendicular to \(\mathbf{u}\).
Example 11.8.1
Projecting a vector onto a line in \(\mathbb{R}^2\)
- ProblemFind the projection of \(\mathbf{v} = \begin{pmatrix}3\\4\end{pmatrix}\) onto \(\mathbf{u} = \begin{pmatrix}1\\2\end{pmatrix}\).
- Calculate$$\mathbf{v}\cdot\mathbf{u} = 3+8=11, \quad \|\mathbf{u}\|^2 = 1+4=5$$
$$\operatorname{proj}_{\mathbf{u}}\mathbf{v} = \tfrac{11}{5}\begin{pmatrix}1\\2\end{pmatrix} = \begin{pmatrix}11/5\\22/5\end{pmatrix}$$
- Error vector$$\mathbf{z} = \begin{pmatrix}3\\4\end{pmatrix} - \begin{pmatrix}11/5\\22/5\end{pmatrix} = \begin{pmatrix}4/5\\-2/5\end{pmatrix}$$
- Check\(\mathbf{z}\cdot\mathbf{u} = \tfrac{4}{5}(1)+(-\tfrac{2}{5})(2) = \tfrac{4}{5}-\tfrac{4}{5} = 0\) ✓
- InterpretThe projection \(\bigl(11/5,\,22/5\bigr)\) is the closest point on the line through \(\mathbf{u}\) to \(\mathbf{v}\). The error vector \(\bigl(4/5,\,-2/5\bigr)\) is perpendicular to \(\mathbf{u}\) and measures how far \(\mathbf{v}\) is from that line.
Answer: \(\operatorname{proj}_{\mathbf{u}}\mathbf{v} = \begin{pmatrix}11/5\\22/5\end{pmatrix}\); error: \(\begin{pmatrix}4/5\\-2/5\end{pmatrix}\)
Diagram
Draw the line through the origin in the direction of \((1,2)\). Draw \(\mathbf{v}=(3,4)\) as an arrow from the origin. Drop a perpendicular from the tip of \(\mathbf{v}\) to the line. The foot of the perpendicular is the tip of the projection vector \((11/5,\,22/5)\). The perpendicular segment from that foot back to the tip of \(\mathbf{v}\) is the error vector \(\mathbf{z}\). The three vectors form a right triangle.
Example 11.8.2
Projection in \(\mathbb{R}^3\)
- ProblemFind the projection of \(\mathbf{v} = \begin{pmatrix}1\\1\\1\end{pmatrix}\) onto \(\mathbf{u} = \begin{pmatrix}2\\0\\-1\end{pmatrix}\).
- Calculate$$\mathbf{v}\cdot\mathbf{u} = 2+0-1=1, \quad \|\mathbf{u}\|^2 = 4+0+1=5$$
$$\operatorname{proj}_{\mathbf{u}}\mathbf{v} = \tfrac{1}{5}\begin{pmatrix}2\\0\\-1\end{pmatrix} = \begin{pmatrix}2/5\\0\\-1/5\end{pmatrix}$$
- Check\(\mathbf{z} = \begin{pmatrix}3/5\\1\\6/5\end{pmatrix}\); then \(\mathbf{z}\cdot\mathbf{u} = \tfrac{6}{5}+0-\tfrac{6}{5}=0\) ✓
Answer: \(\operatorname{proj}_{\mathbf{u}}\mathbf{v} = \begin{pmatrix}2/5\\0\\-1/5\end{pmatrix}\)
The projection onto the line spanned by \(\mathbf{u}\) can also be written as a matrix transformation. The projection matrix onto the line is:
Projection Matrix onto a Line
$$P = \frac{\mathbf{u}\mathbf{u}^T}{\mathbf{u}^T\mathbf{u}}$$
where \(\mathbf{u}\mathbf{u}^T\) is the outer product (an \(n\times n\) matrix) and \(\mathbf{u}^T\mathbf{u}\) is a scalar. Then \(\operatorname{proj}_{\mathbf{u}}\mathbf{v} = P\mathbf{v}\).
Key properties: \(P^2 = P\) (idempotent — projecting twice gives the same result as projecting once); \(P^T = P\) (symmetric); eigenvalues of \(P\) are 0s and 1s.
Section 11.9
Projections onto Subspaces
The projection formula from Section 11.8 handles lines. Now we extend to projecting onto a subspace of dimension greater than one. Let \(W\) be a subspace with basis \(\{\mathbf{a}_1, \ldots, \mathbf{a}_k\}\). Form the matrix \(A\) whose columns are these basis vectors. The projection of any vector \(\mathbf{b}\) onto \(W\) is:
Projection onto a General Subspace
$$\hat{\mathbf{b}} = A(A^T A)^{-1} A^T \mathbf{b}$$
The matrix \(P = A(A^TA)^{-1}A^T\) is the projection matrix onto \(W\). The error vector \(\mathbf{b}-\hat{\mathbf{b}}\) is always orthogonal to every vector in \(W\).
When the columns of \(A\) are already orthogonal, \(A^TA\) becomes diagonal and the formula simplifies to a sum of scalar projections — one per basis vector.
Example 11.9.1
Projecting onto a plane in \(\mathbb{R}^3\)
- ProblemFind the projection of \(\mathbf{b} = \begin{pmatrix}1\\2\\3\end{pmatrix}\) onto the plane \(W\) spanned by \(\mathbf{v}_1 = \begin{pmatrix}1\\1\\0\end{pmatrix}\) and \(\mathbf{v}_2 = \begin{pmatrix}0\\0\\1\end{pmatrix}\).
- Check orthogonality\(\mathbf{v}_1 \cdot \mathbf{v}_2 = 0\) ✓ — so we can use the simpler formula (sum of two scalar projections).
- Calculate$$\frac{\mathbf{b}\cdot\mathbf{v}_1}{\|\mathbf{v}_1\|^2} = \frac{1+2+0}{1+1+0} = \frac{3}{2}, \quad \frac{\mathbf{b}\cdot\mathbf{v}_2}{\|\mathbf{v}_2\|^2} = \frac{0+0+3}{1} = 3$$
$$\hat{\mathbf{b}} = \tfrac{3}{2}\begin{pmatrix}1\\1\\0\end{pmatrix} + 3\begin{pmatrix}0\\0\\1\end{pmatrix} = \begin{pmatrix}3/2\\3/2\\3\end{pmatrix}$$
- CheckError: \(\mathbf{e} = \begin{pmatrix}-1/2\\1/2\\0\end{pmatrix}\). Then \(\mathbf{e}\cdot\mathbf{v}_1 = -\tfrac{1}{2}+\tfrac{1}{2}+0 = 0\) ✓ and \(\mathbf{e}\cdot\mathbf{v}_2 = 0\) ✓
- InterpretThe projection \((3/2,\,3/2,\,3)\) is the closest point to \((1,2,3)\) inside the plane \(W\). The error vector \((-1/2,\,1/2,\,0)\) lies outside the plane, perpendicular to it.
Answer: \(\hat{\mathbf{b}} = \begin{pmatrix}3/2\\3/2\\3\end{pmatrix}\); error: \(\begin{pmatrix}-1/2\\1/2\\0\end{pmatrix}\)
- Compute \(\operatorname{proj}_{\mathbf{u}}\mathbf{v}\) and the error vector for \(\mathbf{v} = \begin{pmatrix}5\\2\end{pmatrix}\), \(\mathbf{u} = \begin{pmatrix}4\\-1\end{pmatrix}\). Verify the error is orthogonal to \(\mathbf{u}\).
- Compute \(\operatorname{proj}_{\mathbf{u}}\mathbf{v}\) and the error vector for \(\mathbf{v} = \begin{pmatrix}2\\3\\1\end{pmatrix}\), \(\mathbf{u} = \begin{pmatrix}1\\1\\1\end{pmatrix}\). Verify orthogonality.
- Let \(\mathbf{v}_1 = \begin{pmatrix}2\\1\\0\end{pmatrix}\) and \(\mathbf{v}_2 = \begin{pmatrix}0\\0\\1\end{pmatrix}\) be an orthogonal basis for a subspace \(W\). Find the projection of \(\mathbf{b} = \begin{pmatrix}3\\-1\\4\end{pmatrix}\) onto \(W\).
- If \(\mathbf{v} \cdot \mathbf{u} = 0\), the projection of \(\mathbf{v}\) onto \(\mathbf{u}\) is the zero vector. Explain why this makes geometric sense.
- Show that if \(P = \dfrac{\mathbf{u}\mathbf{u}^T}{\mathbf{u}^T\mathbf{u}}\), then \(P^2 = P\). (Compute \(PP\) step by step.)
Sections 11.10 – 11.12
Least Squares, Gram-Schmidt, and Applications
Normal Equations · Line Fitting · Gram-Schmidt Process · Real-World Orthogonality
Section 11.10
Least Squares as an Algebraic Introduction
Consider trying to solve \(A\mathbf{x} = \mathbf{b}\) where \(\mathbf{b}\) does not lie in the column space of \(A\) — the system is inconsistent. No exact solution exists. This is the common situation in data science and statistics: you have more equations (data points) than unknowns (model parameters).
The least squares approach asks: find \(\hat{\mathbf{x}}\) that minimizes the size of the error \(\|A\hat{\mathbf{x}} - \mathbf{b}\|\). This is asking for the vector \(A\hat{\mathbf{x}}\) in the column space of \(A\) that is closest to \(\mathbf{b}\) — which is a projection.
The Normal Equations
The least squares solution \(\hat{\mathbf{x}}\) satisfies:
$$A^T A\hat{\mathbf{x}} = A^T \mathbf{b}$$
When the columns of \(A\) are linearly independent, \(A^TA\) is invertible and the unique solution is \(\hat{\mathbf{x}} = (A^TA)^{-1}A^T\mathbf{b}\).
The word "normal" means perpendicular: the residual \(\mathbf{b}-A\hat{\mathbf{x}}\) is perpendicular (normal) to every column of \(A\). This geometric condition is what the normal equations encode.
Why No Calculus?
Even though least squares is often derived by minimizing a sum of squares in a calculus course, we can understand and apply it entirely through projection. The residual being orthogonal to the column space is the geometric condition, and geometry is enough to derive and apply the normal equations.
Least Squares for Line Fitting
One common application is fitting a line \(y = mx + b\) to data points \((x_1,y_1), \ldots, (x_n,y_n)\). Write this as \(A\mathbf{c} = \mathbf{y}\) where:
Line Fitting Setup
$$A = \begin{pmatrix}x_1 & 1\\x_2 & 1\\\vdots & \vdots\\x_n & 1\end{pmatrix}, \quad \mathbf{c} = \begin{pmatrix}m\\b\end{pmatrix}, \quad \mathbf{y} = \begin{pmatrix}y_1\\y_2\\\vdots\\y_n\end{pmatrix}$$
Solve the normal equations \(A^TA\hat{\mathbf{c}} = A^T\mathbf{y}\) to find the slope \(m\) and intercept \(b\) of the best-fit line.
Example 11.10.1
Least squares line through three points
- ProblemFind the best-fit line \(y = mx + b\) through \((0,1)\), \((1,3)\), \((2,4)\).
- RepresentThree points overdetermine a line. Write \(A = \begin{pmatrix}0&1\\1&1\\2&1\end{pmatrix}\), \(\mathbf{y} = \begin{pmatrix}1\\3\\4\end{pmatrix}\).
- Compute \(A^TA\)$$A^TA = \begin{pmatrix}0&1&2\\1&1&1\end{pmatrix}\begin{pmatrix}0&1\\1&1\\2&1\end{pmatrix} = \begin{pmatrix}5&3\\3&3\end{pmatrix}$$
- Compute \(A^T\mathbf{y}\)$$A^T\mathbf{y} = \begin{pmatrix}0&1&2\\1&1&1\end{pmatrix}\begin{pmatrix}1\\3\\4\end{pmatrix} = \begin{pmatrix}11\\8\end{pmatrix}$$
- SolveFrom the system \(\begin{pmatrix}5&3\\3&3\end{pmatrix}\begin{pmatrix}m\\b\end{pmatrix} = \begin{pmatrix}11\\8\end{pmatrix}\): subtract equations to get \(2m = 3\), so \(m = 3/2\). Then \(b = 8/3 - 3/2 = 7/6\).
- InterpretNo line passes through all three points exactly. The least squares line minimizes the total squared vertical distance from the line to each data point — this is linear regression.
Answer: best-fit line \(y = \dfrac{3}{2}x + \dfrac{7}{6}\)
Diagram
Plot the three points \((0,1)\), \((1,3)\), \((2,4)\) on a coordinate grid. Draw the line \(y = \tfrac{3}{2}x + \tfrac{7}{6}\). Draw short vertical segments from each point to the line — these are the residuals. The least squares line minimizes the sum of the squares of these residual lengths. No line could achieve a smaller total.
MGU Library Connection
- The least squares method connects to statistics (linear regression), data science (model fitting), and machine learning (loss minimization).
- See the MGU Data Science Foundations and Machine Learning Mathematics chapters for extensions of this topic.
- The residual vector and its geometric interpretation are developed further in the MGU Probability and Statistics chapter.
Section 11.11
Gram-Schmidt Process as an Introduction
Given an arbitrary linearly independent set of vectors, how do you convert it into an orthogonal basis for the same subspace? The Gram-Schmidt process answers this question.
The Gram-Schmidt Algorithm
Given a linearly independent set \(\{\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_k\}\), produce an orthogonal set \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}\) spanning the same subspace:
Step 1: Set \(\mathbf{v}_1 = \mathbf{x}_1\).
Step 2: \(\mathbf{v}_2 = \mathbf{x}_2 - \dfrac{\mathbf{x}_2\cdot\mathbf{v}_1}{\|\mathbf{v}_1\|^2}\mathbf{v}_1\)
Step 3: \(\mathbf{v}_3 = \mathbf{x}_3 - \dfrac{\mathbf{x}_3\cdot\mathbf{v}_1}{\|\mathbf{v}_1\|^2}\mathbf{v}_1 - \dfrac{\mathbf{x}_3\cdot\mathbf{v}_2}{\|\mathbf{v}_2\|^2}\mathbf{v}_2\)
General step: \(\mathbf{v}_j = \mathbf{x}_j - \displaystyle\sum_{i=1}^{j-1} \dfrac{\mathbf{x}_j\cdot\mathbf{v}_i}{\|\mathbf{v}_i\|^2}\mathbf{v}_i\)
To obtain an orthonormal basis, normalize: \(\mathbf{q}_j = \dfrac{\mathbf{v}_j}{\|\mathbf{v}_j\|}\).
Each step removes the components of the new vector that lie along previously processed directions. What remains is exactly the part of \(\mathbf{x}_j\) perpendicular to all earlier directions. The result is orthogonal by construction.
Example 11.11.1
Gram-Schmidt in \(\mathbb{R}^3\)
- ProblemApply Gram-Schmidt to \(\mathbf{x}_1 = \begin{pmatrix}1\\1\\1\end{pmatrix}\), \(\mathbf{x}_2 = \begin{pmatrix}1\\1\\0\end{pmatrix}\), \(\mathbf{x}_3 = \begin{pmatrix}1\\0\\0\end{pmatrix}\).
- Step 1$$\mathbf{v}_1 = \begin{pmatrix}1\\1\\1\end{pmatrix}$$
- Step 2$$\mathbf{x}_2\cdot\mathbf{v}_1 = 2, \quad \|\mathbf{v}_1\|^2 = 3$$
$$\mathbf{v}_2 = \begin{pmatrix}1\\1\\0\end{pmatrix} - \tfrac{2}{3}\begin{pmatrix}1\\1\\1\end{pmatrix} = \begin{pmatrix}1/3\\1/3\\-2/3\end{pmatrix}$$
- Step 3$$\mathbf{x}_3\cdot\mathbf{v}_1 = 1, \quad \mathbf{x}_3\cdot\mathbf{v}_2 = \tfrac{1}{3}, \quad \|\mathbf{v}_2\|^2 = \tfrac{2}{3}$$
$$\mathbf{v}_3 = \begin{pmatrix}1\\0\\0\end{pmatrix} - \tfrac{1}{3}\begin{pmatrix}1\\1\\1\end{pmatrix} - \tfrac{1}{2}\begin{pmatrix}1/3\\1/3\\-2/3\end{pmatrix} = \begin{pmatrix}1/2\\-1/2\\0\end{pmatrix}$$
- Check\(\mathbf{v}_1\cdot\mathbf{v}_2 = \tfrac{1}{3}+\tfrac{1}{3}-\tfrac{2}{3} = 0\) ✓; \(\mathbf{v}_1\cdot\mathbf{v}_3 = \tfrac{1}{2}-\tfrac{1}{2}+0=0\) ✓; \(\mathbf{v}_2\cdot\mathbf{v}_3 = \tfrac{1}{6}-\tfrac{1}{6}+0=0\) ✓
- InterpretWe have converted the original basis into an orthogonal basis for \(\mathbb{R}^3\). To get an orthonormal basis, normalize each \(\mathbf{v}_j\). The relationship between the original and orthogonal vectors is captured in the QR factorization \(A = QR\), where \(Q\) has orthonormal columns and \(R\) is upper triangular.
Orthogonal basis: \(\mathbf{v}_1 = (1,1,1)\), \(\mathbf{v}_2 = (1/3,1/3,-2/3)\), \(\mathbf{v}_3 = (1/2,-1/2,0)\)
MGU Library Connection
- The QR decomposition, obtained from Gram-Schmidt, is fundamental in numerical linear algebra and stable least squares computation.
- See the MGU Scientific Computing and Numerical Methods chapters for QR applications and stability analysis.
Section 11.12
Orthogonality in Data, Geometry, and Physics Readiness
| Context | How Orthogonality and Projection Appear |
|---|
| Force decomposition | Decompose a force \(\mathbf{F}\) into components along and perpendicular to a surface. The effective force is \(\operatorname{proj}_{\mathbf{d}}\mathbf{F}\); the wasted force is the error vector. Every inclined-plane problem in physics is a projection. |
| Signal processing | A signal is represented as a vector. Orthogonal basis functions (sines and cosines) decompose it into frequency components. Each component's strength is a dot product — the foundation of the Fourier transform. |
| Computer graphics | Camera frames and object orientations use orthonormal bases. Projecting vectors onto local coordinate axes uses the dot product formulas from this chapter. |
| Data science / ML | Orthogonal features carry independent information. Principal Component Analysis (PCA) finds the most informative orthogonal directions in data using projection. |
| Statistics | The residual in linear regression is the error vector. The residual sum of squares is \(\|\mathbf{b} - A\hat{\mathbf{x}}\|^2\). Understanding regression geometrically means understanding projection. |
- Set up and solve the normal equations for \(A = \begin{pmatrix}1&1\\1&-1\\2&0\end{pmatrix}\), \(\mathbf{b} = \begin{pmatrix}2\\0\\3\end{pmatrix}\). Find \(\hat{\mathbf{x}}\).
- Data points \((1,3)\), \((2,4)\), \((4,5)\), \((5,4)\): set up the least squares system and solve for the best-fit line \(y = mx+b\).
- Apply Gram-Schmidt to \(\mathbf{x}_1 = \begin{pmatrix}1\\0\end{pmatrix}\), \(\mathbf{x}_2 = \begin{pmatrix}1\\2\end{pmatrix}\). Confirm your result is orthogonal and spans \(\mathbb{R}^2\).
- Apply Gram-Schmidt to \(\mathbf{x}_1 = \begin{pmatrix}2\\-1\\1\end{pmatrix}\), \(\mathbf{x}_2 = \begin{pmatrix}1\\0\\3\end{pmatrix}\). Verify orthogonality.
- A force \(\mathbf{F} = \begin{pmatrix}10\\6\end{pmatrix}\) acts on an object moving in the direction \(\mathbf{d} = \begin{pmatrix}3\\1\end{pmatrix}\). Find the component of force along the direction of motion and the component perpendicular to it.
Sections 11.13 – 11.14
Common Mistakes and Preparing for Eigenvalues
Eight Recurring Errors · Bridge to Chapter 12
Section 11.13
Common Orthogonality Mistakes
Mistake 1 — Confusing Orthogonality and Linear Independence
Orthogonal nonzero vectors are always linearly independent, but independent vectors are not necessarily orthogonal. The vectors \(\begin{pmatrix}1\\0\end{pmatrix}\) and \(\begin{pmatrix}1\\1\end{pmatrix}\) are independent but not orthogonal (their dot product is 1, not 0). Do not assume that any independent set is orthogonal.
Mistake 2 — Projecting onto a Vector Instead of a Subspace (Wrong Formula)
When projecting \(\mathbf{b}\) onto a subspace \(W\) spanned by two or more vectors, you cannot use the single-vector formula unless the basis vectors are already orthogonal. If the basis is not orthogonal, use the matrix formula \(P = A(A^TA)^{-1}A^T\) or first apply Gram-Schmidt to get an orthogonal basis.
Mistake 3 — Forgetting to Subtract the Projection in Gram-Schmidt
The orthogonal vector is \(\mathbf{v}_j = \mathbf{x}_j - \text{(projections)}\), not just the projections. A common error is computing the projection terms but forgetting to subtract them from \(\mathbf{x}_j\). Always check by computing dot products to confirm the result is orthogonal.
Mistake 4 — Confusing the Scalar Component with the Vector Projection
scalar component = vector projection
The scalar component of \(\mathbf{v}\) along \(\mathbf{u}\) is the number \(\dfrac{\mathbf{v}\cdot\mathbf{u}}{\|\mathbf{u}\|}\). The vector projection is \(\dfrac{\mathbf{v}\cdot\mathbf{u}}{\|\mathbf{u}\|^2}\mathbf{u}\) — note the different denominator and the factor of \(\mathbf{u}\). These are related but distinct results.
Mistake 5 — Expecting the Projection to Equal the Original Vector
The projection of \(\mathbf{v}\) onto a subspace equals \(\mathbf{v}\) itself only when \(\mathbf{v}\) already lies in that subspace. If \(\mathbf{v}\) is not in the subspace, the projection is strictly "shorter" in a sense and the error vector is nonzero.
Mistake 6 — Wrong Setup of Normal Equations
\(AA^T\hat{\mathbf{x}} = A^T\mathbf{b}\)
The normal equations are \(A^TA\hat{\mathbf{x}} = A^T\mathbf{b}\). Students sometimes write \(AA^T\) instead of \(A^TA\), which is a completely different (and wrong) matrix. Carefully track the order and placement of transposes.
Mistake 7 — Assuming Normal Equations Always Have a Unique Solution
If the columns of \(A\) are linearly dependent, then \(A^TA\) is not invertible and the normal equations have infinitely many solutions. Every solution minimizes the same residual, but there is no unique least squares solution. This happens when model variables are redundant.
Mistake 8 — Confusing Orthogonal Complement with the Null Space
The null space of \(A\) equals the orthogonal complement of the row space of \(A\) — not the orthogonal complement of the column space. These are different subspaces of different ambient spaces. Confusion between them leads to incorrect dimension counts and structural conclusions.
Section 11.14
Preparing for Eigenvalues and Eigenvectors
This chapter has developed the geometric language of perpendicularity, projection, and orthogonal structure. Chapter 12 will use some of this language as you study eigenvalues and eigenvectors.
Eigenvalues and eigenvectors describe directions that remain on their own line under a matrix transformation — directions of pure stretching or compression. Orthogonality will reappear when you study symmetric matrices, which always have orthogonal eigenvectors. This result, called the Spectral Theorem, means that symmetric matrices can always be diagonalized using an orthonormal basis of eigenvectors.
Least squares is also connected to eigenvalues: the principal components in data analysis are eigenvectors of a symmetric matrix related to the covariance of the data. As you move into Chapter 12, bring these tools: the dot product and orthogonality condition; the projection formula; the understanding that a subspace can be described by an orthogonal basis; and the connection between symmetric matrices and clean geometric properties.
- A student computes the projection of \(\mathbf{v}=\begin{pmatrix}6\\4\end{pmatrix}\) onto \(\mathbf{u}=\begin{pmatrix}3\\-1\end{pmatrix}\) using the formula \(\dfrac{\mathbf{v}\cdot\mathbf{u}}{\|\mathbf{v}\|^2}\mathbf{u}\). What is the error? Write the correct formula and compute the correct answer.
- A student checks whether \(\mathbf{v}_1=\begin{pmatrix}2\\1\end{pmatrix}\) and \(\mathbf{v}_2=\begin{pmatrix}-1\\2\end{pmatrix}\) form an orthogonal set and concludes they do not because "they clearly point in different directions." What should the student check instead? Are they orthogonal?
- A student applies Gram-Schmidt to \(\mathbf{x}_1=\begin{pmatrix}2\\1\end{pmatrix}\), \(\mathbf{x}_2=\begin{pmatrix}3\\1\end{pmatrix}\) and sets \(\mathbf{v}_2 = \operatorname{proj}_{\mathbf{v}_1}\mathbf{x}_2\). What is the error?
- A student writes the normal equations as \(AA^T\hat{\mathbf{x}} = A\mathbf{b}\). Identify all errors and write the correct form.
- True or false: every set of linearly independent vectors is also an orthogonal set. Justify with an example or a counterexample.
Chapter 11 — Back Matter
Chapter Summary
- Sec 11.1The dot product \(\mathbf{u}\cdot\mathbf{v} = u_1v_1 + \cdots + u_nv_n\) produces a scalar. It satisfies commutativity, distributivity, and scalar factoring. Also written \(\mathbf{u}^T\mathbf{v}\).
- Sec 11.2The length of \(\mathbf{v}\) is \(\|\mathbf{v}\| = \sqrt{\mathbf{v}\cdot\mathbf{v}}\). A unit vector has length 1; normalize by dividing by the length. Distance between \(\mathbf{u}\) and \(\mathbf{v}\) is \(\|\mathbf{u}-\mathbf{v}\|\).
- Sec 11.3The angle between vectors satisfies \(\cos\theta = \dfrac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}\). Positive dot product → acute; negative → obtuse; zero → right angle (orthogonal).
- Sec 11.4Two vectors are orthogonal iff \(\mathbf{u}\cdot\mathbf{v}=0\). Orthogonal vectors satisfy the Pythagorean theorem: \(\|\mathbf{u}+\mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2\).
- Sec 11.5An orthogonal set has all pairwise dot products equal to zero. An orthonormal set also has all lengths equal to 1. Every orthogonal set of nonzero vectors is automatically linearly independent.
- Sec 11.6An orthogonal basis for a subspace allows coordinates to be computed by dot products instead of row reduction. The formula is simpler still when the basis is orthonormal.
- Sec 11.7The orthogonal complement \(W^\perp\) contains all vectors perpendicular to every vector in \(W\). Dimensions satisfy \(\dim(W)+\dim(W^\perp)=n\). Null space of \(A\) equals the orthogonal complement of the row space of \(A\).
- Sec 11.8Projection of \(\mathbf{v}\) onto \(\mathbf{u}\): \(\operatorname{proj}_{\mathbf{u}}\mathbf{v} = \frac{\mathbf{v}\cdot\mathbf{u}}{\|\mathbf{u}\|^2}\mathbf{u}\). The error vector \(\mathbf{v}-\operatorname{proj}_{\mathbf{u}}\mathbf{v}\) is perpendicular to \(\mathbf{u}\). Projection matrix \(P = \frac{\mathbf{u}\mathbf{u}^T}{\mathbf{u}^T\mathbf{u}}\) satisfies \(P^2=P\) and \(P^T=P\).
- Sec 11.9Projection onto a subspace uses the matrix formula \(P = A(A^TA)^{-1}A^T\). If the basis for the subspace is already orthogonal, the projection is a sum of individual scalar projections — no matrix inverse needed.
- Sec 11.10When \(A\mathbf{x}=\mathbf{b}\) is inconsistent, the least squares solution minimizes \(\|A\hat{\mathbf{x}}-\mathbf{b}\|\) and satisfies the normal equations \(A^TA\hat{\mathbf{x}}=A^T\mathbf{b}\). The residual is orthogonal to the column space of \(A\).
- Sec 11.11Gram-Schmidt converts any linearly independent set into an orthogonal basis for the same subspace, by iteratively subtracting projections onto previously built basis vectors. Normalizing gives an orthonormal basis.
- Sec 11.12Orthogonality and projection appear in force decomposition, signal processing, computer graphics, data science (PCA), machine learning, and statistics (residuals, regression geometry).
- Sec 11.13Eight common mistakes: confusing orthogonality and independence; wrong projection formula for subspaces; forgetting to subtract in Gram-Schmidt; scalar vs. vector projection confusion; wrong normal equations; non-uniqueness of least squares when columns are dependent; confusing null space and orthogonal complement.
- Sec 11.14Orthogonality prepares for eigenvalues: symmetric matrices have orthogonal eigenvectors (Spectral Theorem). Least squares connects to eigenvalues through PCA and covariance matrices.
Key Terms Review
dot product
The scalar \(\mathbf{u}\cdot\mathbf{v} = u_1v_1 + \cdots + u_nv_n\). Also written \(\mathbf{u}^T\mathbf{v}\). Measures angle and perpendicularity.
inner product
A generalization of the dot product to abstract vector spaces. The dot product on \(\mathbb{R}^n\) is the standard example.
norm (length)
\(\|\mathbf{v}\| = \sqrt{\mathbf{v}\cdot\mathbf{v}}\). Extends the Pythagorean distance formula into any dimension.
unit vector
A vector of length 1. Obtained by normalizing: \(\hat{\mathbf{v}} = \mathbf{v}/\|\mathbf{v}\|\).
normalizing
Dividing a nonzero vector by its length to obtain a unit vector in the same direction.
distance
\(d(\mathbf{u},\mathbf{v}) = \|\mathbf{u}-\mathbf{v}\|\). The distance between the tips of two vectors.
orthogonal
Two vectors are orthogonal when their dot product is zero — perpendicular in the sense of the inner product.
orthogonal set
A set where every pair of distinct vectors has zero dot product. Automatically linearly independent.
orthonormal set
An orthogonal set where every vector also has length 1.
orthogonal basis
A basis that is also an orthogonal set. Coordinates computed by dot products.
orthogonal complement
\(W^\perp\): all vectors orthogonal to every vector in \(W\). Satisfies \(\dim(W)+\dim(W^\perp)=n\).
projection
The closest vector in a given direction or subspace to a given vector; found via the projection formula.
scalar component
The number \(\mathbf{v}\cdot\mathbf{u}/\|\mathbf{u}\|\). The signed length of the projection of \(\mathbf{v}\) along \(\mathbf{u}\).
projection matrix
The matrix \(P\) such that \(P\mathbf{b} = \hat{\mathbf{b}}\). Satisfies \(P^2=P\) (idempotent) and \(P^T=P\) (symmetric).
residual (error vector)
\(\mathbf{b} - A\hat{\mathbf{x}}\). Always orthogonal to the column space of \(A\) in a least squares problem.
least squares solution
Minimizes \(\|A\mathbf{x}-\mathbf{b}\|\). Satisfies the normal equations when the original system is inconsistent.
normal equations
\(A^TA\hat{\mathbf{x}} = A^T\mathbf{b}\). The system whose solution gives the least squares approximation.
Gram-Schmidt process
Algorithm to convert a linearly independent set into an orthogonal (or orthonormal) basis for the same subspace.
QR decomposition
Factorization \(A = QR\) where \(Q\) has orthonormal columns and \(R\) is upper triangular; produced by Gram-Schmidt.
idempotent matrix
A matrix \(P\) satisfying \(P^2=P\). Projection matrices are idempotent because projecting twice gives the same result as projecting once.
Concept Review Questions
- What does the dot product measure geometrically when both vectors are unit vectors?
- How does the sign of a dot product tell you whether an angle is acute or obtuse?
- Why is the zero vector considered orthogonal to every vector? Is this a useful or trivial case?
- Explain in your own words why every orthogonal set of nonzero vectors is linearly independent.
- If \(W\) is a two-dimensional subspace of \(\mathbb{R}^5\), what is \(\dim(W^\perp)\)? Explain using the dimension formula.
- What property does the error vector always satisfy in a projection? Why is this the geometric condition for "closeness"?
- What are the normal equations, and why are they called "normal"?
- Why does least squares give a useful approximate answer even when the original system \(A\mathbf{x}=\mathbf{b}\) is inconsistent?
- What is the purpose of Gram-Schmidt? What does it produce, and why is the result useful?
- Is it possible for two vectors to be linearly independent but not orthogonal? Give an example.
Skill Practice
- \(\mathbf{u}=\begin{pmatrix}4\\-2\\1\end{pmatrix}\), \(\mathbf{v}=\begin{pmatrix}3\\5\\-2\end{pmatrix}\): compute \(\mathbf{u}\cdot\mathbf{v}\)
- \(\mathbf{a}=\begin{pmatrix}1\\2\\3\\4\end{pmatrix}\), \(\mathbf{b}=\begin{pmatrix}-4\\3\\-2\\1\end{pmatrix}\): compute \(\mathbf{a}\cdot\mathbf{b}\)
- Find \(\|\begin{pmatrix}1\\-2\\2\end{pmatrix}\|\) and normalize it.
- Find the distance between \(\begin{pmatrix}0\\0\end{pmatrix}\) and \(\begin{pmatrix}-3\\4\end{pmatrix}\).
- Find the angle between \(\begin{pmatrix}2\\1\end{pmatrix}\) and \(\begin{pmatrix}1\\3\end{pmatrix}\).
- Verify: \(\begin{pmatrix}1\\2\\2\end{pmatrix}\) and \(\begin{pmatrix}4\\0\\-3\end{pmatrix}\) — are they orthogonal? Find \(\|\mathbf{u}\|\), \(\|\mathbf{v}\|\), and the angle.
- Verify that \(\left\{\begin{pmatrix}3\\0\\4\end{pmatrix},\begin{pmatrix}0\\1\\0\end{pmatrix},\begin{pmatrix}4\\0\\-3\end{pmatrix}\right\}\) is orthogonal. Normalize to form an orthonormal set.
- Find a nonzero vector in \(\mathbb{R}^2\) orthogonal to \(\begin{pmatrix}7\\-3\end{pmatrix}\).
- Show that \(\left\{\begin{pmatrix}1\\1\\0\end{pmatrix},\begin{pmatrix}0\\0\\1\end{pmatrix}\right\}\) is an orthogonal basis for the plane \(x_1 = x_2,\,x_3\) arbitrary. Find the coordinates of \(\begin{pmatrix}3\\3\\5\end{pmatrix}\) in this basis.
- Let \(W = \operatorname{span}\!\left\{\begin{pmatrix}2\\1\\0\end{pmatrix}\right\}\). Describe \(W^\perp\) as an equation. What is its dimension?
- Project \(\begin{pmatrix}7\\2\end{pmatrix}\) onto \(\begin{pmatrix}1\\3\end{pmatrix}\). Find the error vector and verify perpendicularity.
- Project \(\begin{pmatrix}4\\1\\-1\end{pmatrix}\) onto \(\begin{pmatrix}2\\0\\1\end{pmatrix}\). Find the error vector and verify.
- The orthogonal set \(\mathbf{v}_1 = \begin{pmatrix}1\\0\\1\end{pmatrix}\), \(\mathbf{v}_2 = \begin{pmatrix}0\\1\\0\end{pmatrix}\) spans a subspace \(W\). Project \(\mathbf{b} = \begin{pmatrix}2\\3\\4\end{pmatrix}\) onto \(W\).
- Write the projection matrix \(P\) for projection onto \(\mathbf{u} = \begin{pmatrix}1\\0\end{pmatrix}\). Verify \(P^2 = P\).
- Normal equations for \(A = \begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}\), \(\mathbf{b}=\begin{pmatrix}6\\5\\7\end{pmatrix}\). Find \(\hat{\mathbf{x}}\) and interpret as a line.
- Apply Gram-Schmidt to \(\mathbf{x}_1=\begin{pmatrix}1\\1\\0\end{pmatrix}\), \(\mathbf{x}_2=\begin{pmatrix}2\\1\\1\end{pmatrix}\). Verify orthogonality.
- Apply Gram-Schmidt to produce an orthonormal basis from: \(\mathbf{x}_1=\begin{pmatrix}0\\0\\1\end{pmatrix}\), \(\mathbf{x}_2=\begin{pmatrix}1\\0\\1\end{pmatrix}\).
- Data: \((0,1),(1,2),(2,2),(3,4)\). Set up and solve the normal equations for the best-fit line.
Word Problems and Applications
- Navigation. A hiker's displacement is \(\mathbf{d} = \begin{pmatrix}4\\3\end{pmatrix}\) km. Find the East component and North component of the displacement.
- Force decomposition. A force \(\mathbf{F} = \begin{pmatrix}5\\8\\2\end{pmatrix}\) is applied to an object. Find the component along \(\mathbf{d}=\begin{pmatrix}1\\1\\0\end{pmatrix}\) and the component perpendicular to \(\mathbf{d}\). Verify the two components are orthogonal.
- Graphics. A surface has outward normal \(\mathbf{n}=\begin{pmatrix}0\\1\\0\end{pmatrix}\). A light direction vector is \(\mathbf{L}=\begin{pmatrix}1\\1\\0\end{pmatrix}\). Find the component of the light direction along the normal — the diffuse lighting intensity factor in a basic shading model.
- Data fitting. A researcher records \((1,2),(2,3),(4,5),(5,4)\). Set up and solve the least squares system for the best-fit line \(y=mx+b\).
- Structural check. You project vector \(\mathbf{b}\) onto a subspace \(W\) and get projection \(\hat{\mathbf{b}} = \mathbf{b}\). What does this tell you about \(\mathbf{b}\) in relation to \(W\)?
Error Analysis
Each problem shows a student's work. Identify the error and write the correct solution.
Problem 1
Projection of \(\mathbf{v}=\begin{pmatrix}6\\4\end{pmatrix}\) onto \(\mathbf{u}=\begin{pmatrix}3\\-1\end{pmatrix}\): student writes \(\dfrac{\mathbf{v}\cdot\mathbf{u}}{\|\mathbf{v}\|^2}\mathbf{u}\)
What is wrong with the denominator? What is the correct formula and the correct numerical answer?
Problem 2
A student claims \(\mathbf{v}_1=\begin{pmatrix}1\\2\end{pmatrix}\) and \(\mathbf{v}_2=\begin{pmatrix}3\\4\end{pmatrix}\) are orthogonal because "they are linearly independent."
What is the error in this reasoning? Are they actually orthogonal?
Problem 3
Gram-Schmidt: starting with \(\mathbf{v}_1 = \mathbf{x}_1\), student sets \(\mathbf{v}_2 = \dfrac{\mathbf{x}_2\cdot\mathbf{v}_1}{\|\mathbf{v}_1\|^2}\mathbf{v}_1\) (the projection term).
What was forgotten? Write the correct formula for \(\mathbf{v}_2\).
Problem 4
Normal equations: \(A^T A \hat{\mathbf{x}} = A\mathbf{b}\)
Identify the error on the right-hand side and write the correct normal equations.
Problem 5
A student projects \(\mathbf{b}\) onto a subspace spanned by two non-orthogonal vectors using the formula \(\hat{\mathbf{b}} = \dfrac{\mathbf{b}\cdot\mathbf{v}_1}{\|\mathbf{v}_1\|^2}\mathbf{v}_1 + \dfrac{\mathbf{b}\cdot\mathbf{v}_2}{\|\mathbf{v}_2\|^2}\mathbf{v}_2\).
What condition must the basis satisfy for this formula to be valid? What should the student do when that condition is not met?
Challenge and Extension
- Prove that if \(\mathbf{u}\) and \(\mathbf{v}\) are orthogonal, then \(\|\mathbf{u}+\mathbf{v}\|^2 = \|\mathbf{u}\|^2+\|\mathbf{v}\|^2\). Use the dot product definition of length, and show every step.
- Show that the projection matrix \(P = \dfrac{\mathbf{u}\mathbf{u}^T}{\mathbf{u}^T\mathbf{u}}\) satisfies \(P^2=P\). Compute \(PP\) carefully, using the associativity of matrix multiplication.
- Suppose \(\{\mathbf{v}_1, \mathbf{v}_2\}\) is an orthogonal basis for a subspace \(W\) and \(\mathbf{b}\in W\). Show that the projection formula gives exactly \(\mathbf{b}\) — that is, the projection of a vector onto the subspace it already lies in equals itself.
- (Extension) The Cauchy-Schwarz inequality states \(|\mathbf{u}\cdot\mathbf{v}| \leq \|\mathbf{u}\|\|\mathbf{v}\|\). Explain why this guarantees that the angle formula \(\cos\theta = \dfrac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}\) always gives a value between \(-1\) and \(1\), making \(\theta\) a valid angle.
- (Extension) Show that a projection matrix \(P\) satisfying \(P^2=P\) and \(P^T=P\) has only eigenvalues 0 and 1. (Hint: if \(P\mathbf{v}=\lambda\mathbf{v}\), apply \(P\) again.)
Chapter 11 Checkpoint
Chapter 11 Checkpoint
Complete without assistance — assesses all key skills and concepts from Chapter 11
Part 1 — Dot Products, Length, and Angles
- Compute \(\mathbf{u}\cdot\mathbf{v}\), \(\|\mathbf{u}\|\), \(\|\mathbf{v}\|\), and the angle between \(\mathbf{u}=\begin{pmatrix}1\\2\\2\end{pmatrix}\) and \(\mathbf{v}=\begin{pmatrix}4\\0\\-3\end{pmatrix}\).
- Find the distance between \(\mathbf{p}=\begin{pmatrix}1\\3\\-1\end{pmatrix}\) and \(\mathbf{q}=\begin{pmatrix}4\\-1\\2\end{pmatrix}\).
Part 2 — Orthogonality
- Determine whether the set \(\left\{\begin{pmatrix}3\\0\\4\end{pmatrix},\begin{pmatrix}0\\1\\0\end{pmatrix},\begin{pmatrix}4\\0\\-3\end{pmatrix}\right\}\) is orthogonal. Normalize each vector to form an orthonormal set.
- Find a nonzero vector in \(\mathbb{R}^3\) orthogonal to both \(\begin{pmatrix}1\\2\\0\end{pmatrix}\) and \(\begin{pmatrix}0\\1\\1\end{pmatrix}\).
Part 3 — Projections
- Find the projection of \(\mathbf{b}=\begin{pmatrix}5\\3\\-2\end{pmatrix}\) onto the orthogonal set from CP3 above. Find the error vector and verify it is orthogonal to both basis vectors.
- Compute \(\operatorname{proj}_{\mathbf{u}}\mathbf{v}\) for \(\mathbf{v}=\begin{pmatrix}4\\6\end{pmatrix}\), \(\mathbf{u}=\begin{pmatrix}3\\1\end{pmatrix}\). Find the error vector and verify orthogonality.
Part 4 — Least Squares
- Data points: \((1,3),(2,4),(3,5),(4,7)\). Set up the matrix \(A\) and vector \(\mathbf{y}\). Compute \(A^TA\) and \(A^T\mathbf{y}\). Solve the normal equations for the best-fit line \(y=mx+b\).
Part 5 — Gram-Schmidt
- Apply Gram-Schmidt to \(\mathbf{x}_1=\begin{pmatrix}2\\-1\\1\end{pmatrix}\), \(\mathbf{x}_2=\begin{pmatrix}1\\0\\3\end{pmatrix}\). Show all steps and verify the result is orthogonal.
Part 6 — Error Analysis and Concepts
- A force \(\mathbf{F}=\begin{pmatrix}5\\8\\2\end{pmatrix}\) is applied to an object. Find the component of \(\mathbf{F}\) along \(\mathbf{d}=\begin{pmatrix}1\\1\\0\end{pmatrix}\) and the component perpendicular to \(\mathbf{d}\). Verify the two components are orthogonal.
- True or false, with full justification: "If two vectors are orthogonal, they must be linearly independent."
- A student sets up the normal equations as \(A^TA\hat{\mathbf{x}} = A\mathbf{b}\). Identify the error and write the correct form.
Answers may be placed in the answer key.
Looking Ahead
Chapter 12: Eigenvalues and Eigenvectors
Chapter 11 gave you the geometric language of perpendicularity and projection. Chapter 12 introduces eigenvalues and eigenvectors — the directions in space that a linear transformation leaves unchanged in direction, only scaling them. The dot product and orthogonality will return when you study symmetric matrices, which have a beautiful property: their eigenvectors are always orthogonal. This result, the Spectral Theorem, connects everything you learned in Chapter 11 to the deepest structure of square matrices. Bring the projection formula and orthogonality condition forward — you will use them.
Bridge to Calculus, Multivariable Mathematics, and Advanced Fields
Multivariable Calculus. Dot products and projections appear throughout. The gradient vector is orthogonal to level curves and level surfaces. Finding the component of a gradient in a particular direction — the directional derivative — is a projection. The tangent plane to a surface uses normal vectors, which are defined by orthogonality.
Differential Equations. Many solution methods rely on orthogonal function spaces. Fourier series decompose functions using orthogonal sine and cosine functions — the analogue of projecting a vector onto an orthogonal basis.
Physics. Work is the dot product of force and displacement. The normal force on an inclined plane is the projection of gravity perpendicular to the surface. Stress tensors and angular momentum use orthogonal decompositions.
Data Science and Machine Learning. Least squares is the foundation of linear regression. Principal Component Analysis uses orthogonal projections to find the most informative directions in high-dimensional data. Orthogonality is central to virtually every major machine learning algorithm.
Statistics. The geometry of linear regression is exactly the projection of a response vector onto the column space of the predictor matrix. Variance decomposition is an orthogonal decomposition of the response vector.
Computer Graphics and Robotics. Orthonormal frames, rotation matrices, and projection operators are used constantly. Camera matrices, lighting calculations, and robot joint coordinates all rely on the ideas of this chapter.
MGU Library Connection
- Appendix M: Orthogonality and Projection Reference
- Appendix P: Least Squares Reference
- Appendix R: Calculus Readiness Bridge Guide — orthogonal gradients and directional derivatives
- MGU Calculus and Multivariable Calculus chapters: gradient, directional derivative, tangent planes
- MGU Data Science Foundations and Machine Learning Mathematics chapters: PCA, linear regression, feature decorrelation
- MGU Scientific Computing and Numerical Methods chapters: QR decomposition, stable least squares