Inner Products and Orthogonality | Malone Math

The Central Idea

In ordinary Euclidean space, the dot product of two vectors

\[ u= \begin{bmatrix} u_1\\ \vdots\\ u_n \end{bmatrix}, \qquad v= \begin{bmatrix} v_1\\ \vdots\\ v_n \end{bmatrix} \]

\[ \boxed{ u\cdot v = u_1v_1+\cdots+u_nv_n}. \]

This single operation lets us define length, angle, orthogonality, projection, and distance.

What Is an Inner Product?

An inner product is a rule

\[ \langle u,v\rangle \]

that assigns a scalar to a pair of vectors while satisfying certain structural properties.

In \(\mathbb{R}^n\), the standard inner product is the dot product:

\[ \boxed{ \langle u,v\rangle = u^Tv = u_1v_1+\cdots+u_nv_n}. \]

Properties of the Standard Inner Product

For vectors \(u,v,w\) and scalar \(c\),

\[ \langle u,v\rangle=\langle v,u\rangle, \] \[ \langle cu,v\rangle=c\langle u,v\rangle, \] \[ \langle u+v,w\rangle = \langle u,w\rangle+\langle v,w\rangle, \]

and

\[ \boxed{ \langle v,v\rangle\ge0} \]

with equality only when \(v=0\).

Example: Compute a Dot Product

Let

\[ u= \begin{bmatrix} 2\\ -1\\ 3 \end{bmatrix}, \qquad v= \begin{bmatrix} 4\\ 5\\ -2 \end{bmatrix}. \]

Then

\[ u\cdot v = 2(4)+(-1)(5)+3(-2). \]

Therefore,

\[ \boxed{u\cdot v=-3}. \]

Norm and Length

The norm induced by the standard inner product is

\[ \boxed{ \|v\| = \sqrt{\langle v,v\rangle}}. \]

In \(\mathbb{R}^n\),

\[ \|v\| = \sqrt{v_1^2+\cdots+v_n^2}. \]

This generalizes the familiar distance formula.

Example: Compute a Norm

Let

\[ v= \begin{bmatrix} 3\\ 4 \end{bmatrix}. \]

Then

\[ \|v\| = \sqrt{3^2+4^2} = \boxed{5}. \]

Unit Vectors

A vector \(u\) is a unit vector when

\[ \boxed{\|u\|=1}. \]

If \(v\neq0\), then

\[ \boxed{ \frac{v}{\|v\|}} \]

is a unit vector pointing in the same direction as \(v\).

This process is called normalization.

Example: Normalize a Vector

For

\[ v= \begin{bmatrix} 3\\ 4 \end{bmatrix}, \]

we found

\[ \|v\|=5. \]

Therefore the corresponding unit vector is

\[ \boxed{ \frac15 \begin{bmatrix} 3\\ 4 \end{bmatrix} = \begin{bmatrix} 3/5\\ 4/5 \end{bmatrix}}. \]

Distance Between Vectors

The distance between vectors \(u\) and \(v\) is

\[ \boxed{d(u,v)=\|u-v\|}. \]

This is the length of the displacement from one vector endpoint to the other.

Angles Between Vectors

For nonzero vectors \(u\) and \(v\), the angle \(\theta\) between them satisfies

\[ \boxed{ \cos\theta = \frac{u\cdot v}{\|u\|\|v\|}}. \]

Therefore,

\[ \theta = \arccos \left( \frac{u\cdot v}{\|u\|\|v\|} \right). \]

Sign of the Dot Product

The sign of

\[ u\cdot v \]

gives a quick angular interpretation:

positive: the angle is acute;
zero: the vectors are perpendicular;
negative: the angle is obtuse.

Orthogonal Vectors

Two vectors are orthogonal when their inner product is zero:

\[ \boxed{\langle u,v\rangle=0}. \]

In Euclidean space, this means the vectors meet at a right angle.

Example: Test Orthogonality

Let

\[ u= \begin{bmatrix} 1\\ 2 \end{bmatrix}, \qquad v= \begin{bmatrix} 2\\ -1 \end{bmatrix}. \]

Then

\[ u\cdot v = 1(2)+2(-1) = 0. \]

Therefore,

\[ \boxed{u\perp v}. \]

Orthogonal Sets

A set of nonzero vectors

\[ \{v_1,\ldots,v_k\} \]

is orthogonal if

\[ \boxed{ \langle v_i,v_j\rangle=0 \quad\text{whenever }i\neq j}. \]

Every distinct pair must be orthogonal.

Orthogonal Nonzero Vectors Are Independent

Any orthogonal set of nonzero vectors is linearly independent.

Suppose

\[ c_1v_1+\cdots+c_kv_k=0. \]

Take the inner product with \(v_j\):

\[ c_j\langle v_j,v_j\rangle=0. \]

Since \(v_j\neq0\),

\[ \langle v_j,v_j\rangle>0, \]

so \(c_j=0\).

This holds for every \(j\).

Orthonormal Sets

A set is orthonormal if its vectors are mutually orthogonal and each has norm \(1\).

Thus

\[ \boxed{ \langle q_i,q_j\rangle = \begin{cases} 1,&i=j,\\ 0,&i\neq j. \end{cases}} \]

Orthonormal Bases

An orthonormal basis is a basis whose vectors form an orthonormal set.

The standard basis

\[ e_1,\ldots,e_n \]

of \(\mathbb{R}^n\) is orthonormal.

Orthonormal bases make coordinate calculations especially simple.

Coordinates in an Orthonormal Basis

Suppose

\[ B=(q_1,\ldots,q_n) \]

is an orthonormal basis.

Every vector \(v\) can be written

\[ v=c_1q_1+\cdots+c_nq_n. \]

The coefficients are simply

\[ \boxed{c_i=\langle v,q_i\rangle}. \]

Orthonormal Expansion Formula

Therefore,

\[ \boxed{ v = \langle v,q_1\rangle q_1 +\cdots+ \langle v,q_n\rangle q_n}. \]

No system of equations is required to recover the coordinates.

Projection onto a Vector

The projection of \(v\) onto a nonzero vector \(u\) is

\[ \boxed{ \operatorname{proj}_u(v) = \frac{\langle v,u\rangle}{\langle u,u\rangle}u}. \]

This gives the component of \(v\) pointing along \(u\).

Projection onto a Unit Vector

If \(\|u\|=1\), then

\[ \langle u,u\rangle=1. \]

Therefore,

\[ \boxed{ \operatorname{proj}_u(v) = \langle v,u\rangle u}. \]

Example: Projection onto a Vector

Let

\[ v= \begin{bmatrix} 3\\ 4 \end{bmatrix}, \qquad u= \begin{bmatrix} 1\\ 0 \end{bmatrix}. \]

Then

\[ v\cdot u=3, \qquad u\cdot u=1. \]

Hence

\[ \boxed{ \operatorname{proj}_u(v) = \begin{bmatrix} 3\\ 0 \end{bmatrix}}. \]

Orthogonal Decomposition

A vector \(v\) can be decomposed into a part parallel to \(u\) and a part orthogonal to \(u\):

\[ \boxed{ v = \operatorname{proj}_u(v) + \left( v-\operatorname{proj}_u(v) \right)}. \]

The second term is orthogonal to \(u\).

Projection onto a Subspace

Suppose

\[ W=\operatorname{span}\{q_1,\ldots,q_k\} \]

where the \(q_i\) form an orthonormal basis for \(W\).

Then

\[ \boxed{ \operatorname{proj}_W(v) = \sum_{i=1}^k \langle v,q_i\rangle q_i}. \]

The Orthogonal Error

\[ p=\operatorname{proj}_W(v), \]

then the error vector

\[ e=v-p \]

is orthogonal to every vector in \(W\):

\[ \boxed{e\in W^\perp}. \]

Orthogonal Complement

The orthogonal complement of a subspace \(W\subseteq \mathbb{R}^n\) is

\[ \boxed{ W^\perp = \{v\in\mathbb{R}^n: \langle v,w\rangle=0 \text{ for every }w\in W\}}. \]

It contains all vectors perpendicular to the entire subspace.

Orthogonal Complements Are Subspaces

If \(u,v\in W^\perp\), then for every \(w\in W\),

\[ \langle au+bv,w\rangle = a\langle u,w\rangle+b\langle v,w\rangle = 0. \]

Therefore,

\[ \boxed{W^\perp\text{ is a subspace}}. \]

Dimension of an Orthogonal Complement

If \(W\subseteq\mathbb{R}^n\), then

\[ \boxed{ \dim(W)+\dim(W^\perp)=n}. \]

The subspace and its orthogonal complement together account for all independent directions in the ambient space.

Orthogonal Direct-Sum Decomposition

Every vector \(v\in\mathbb{R}^n\) can be written uniquely as

\[ \boxed{v=w+z} \]

where

\[ w\in W, \qquad z\in W^\perp. \]

This is written

\[ \boxed{\mathbb{R}^n=W\oplus W^\perp}. \]

Row Space and Null Space

For a matrix \(A\), the null space is orthogonal to the row space:

\[ \boxed{ \operatorname{Null}(A) = \operatorname{Row}(A)^\perp}. \]

This follows because

\[ Ax=0 \]

means that \(x\) has zero dot product with every row of \(A\).

Column Space and Left Null Space

The orthogonal complement of the column space is the null space of \(A^T\):

\[ \boxed{ \operatorname{Col}(A)^\perp = \operatorname{Null}(A^T)}. \]

This relationship becomes central in least-squares theory.

The Four Fundamental Subspaces

A matrix naturally produces four subspaces:

column space;
null space;
row space;
left null space.

Orthogonality pairs them:

\[ \operatorname{Row}(A)^\perp = \operatorname{Null}(A), \] \[ \operatorname{Col}(A)^\perp = \operatorname{Null}(A^T). \]

The Pythagorean Theorem for Vectors

\[ u\perp v, \]

then

\[ \boxed{ \|u+v\|^2 = \|u\|^2+\|v\|^2}. \]

This is the vector-space form of the Pythagorean theorem.

Why the Pythagorean Identity Works

Expand:

\[ \|u+v\|^2 = \langle u+v,u+v\rangle. \]

This becomes

\[ \langle u,u\rangle + 2\langle u,v\rangle + \langle v,v\rangle. \]

Since \(u\perp v\),

\[ \langle u,v\rangle=0. \]

Therefore,

\[ \|u+v\|^2 = \|u\|^2+\|v\|^2. \]

Cauchy-Schwarz Inequality

For any vectors \(u,v\),

\[ \boxed{ |\langle u,v\rangle| \le \|u\|\|v\|}. \]

Equality occurs exactly when the vectors are linearly dependent.

This inequality guarantees that the cosine formula for angles produces a value between \(-1\) and \(1\).

Triangle Inequality

Norms satisfy

\[ \boxed{ \|u+v\| \le \|u\|+\|v\|}. \]

Geometrically, traveling directly from one point to another is no longer than traveling through an intermediate point.

Orthogonal Matrices

A square matrix \(Q\) is orthogonal if its columns form an orthonormal basis.

Equivalently,

\[ \boxed{Q^TQ=I}. \]

Therefore,

\[ \boxed{Q^{-1}=Q^T}. \]

Orthogonal Matrices Preserve Length

If \(Q\) is orthogonal, then

\[ \|Qx\|=\|x\|. \]

Indeed,

\[ \|Qx\|^2 = (Qx)^T(Qx) = x^TQ^TQx = x^Tx. \]

Orthogonal Matrices Preserve Inner Products

For orthogonal \(Q\),

\[ \boxed{ \langle Qu,Qv\rangle = \langle u,v\rangle}. \]

Therefore orthogonal transformations preserve lengths and angles.

Rotations and reflections are standard examples.

Determinant of an Orthogonal Matrix

Since

\[ Q^TQ=I, \]

taking determinants gives

\[ \det(Q)^2=1. \]

Therefore,

\[ \boxed{\det(Q)=\pm1}. \]

Gram-Schmidt Orthogonalization

The Gram-Schmidt process converts a linearly independent set into an orthogonal or orthonormal set spanning the same subspace.

Starting from

\[ v_1,\ldots,v_k, \]

we remove from each new vector the components lying in directions already constructed.

First Gram-Schmidt Step

Set

\[ \boxed{u_1=v_1}. \]

If an orthonormal basis is desired, normalize:

\[ q_1=\frac{u_1}{\|u_1\|}. \]

Second Gram-Schmidt Step

Remove from \(v_2\) its projection onto \(u_1\):

\[ \boxed{ u_2 = v_2 - \operatorname{proj}_{u_1}(v_2)}. \]

Then \(u_2\perp u_1\).

Normalize if needed:

\[ q_2=\frac{u_2}{\|u_2\|}. \]

General Gram-Schmidt Step

For \(j\ge2\),

\[ \boxed{ u_j = v_j - \sum_{i=1}^{j-1} \operatorname{proj}_{u_i}(v_j)}. \]

After normalization,

\[ q_j=\frac{u_j}{\|u_j\|}. \]

The vectors \(q_1,\ldots,q_k\) form an orthonormal basis for the same span.

Example: Gram-Schmidt in \(\mathbb{R}^2\)

Let

\[ v_1= \begin{bmatrix} 1\\ 1 \end{bmatrix}, \qquad v_2= \begin{bmatrix} 1\\ 0 \end{bmatrix}. \]

Set

\[ u_1=v_1. \]

Then

\[ \operatorname{proj}_{u_1}(v_2) = \frac{v_2\cdot u_1}{u_1\cdot u_1}u_1. \]

Since

\[ v_2\cdot u_1=1 \]

and

\[ u_1\cdot u_1=2, \]

we get

\[ \operatorname{proj}_{u_1}(v_2) = \frac12 \begin{bmatrix} 1\\ 1 \end{bmatrix}. \]

Finish the Gram-Schmidt Example

Therefore,

\[ u_2 = \begin{bmatrix} 1\\ 0 \end{bmatrix} - \frac12 \begin{bmatrix} 1\\ 1 \end{bmatrix} = \begin{bmatrix} 1/2\\ -1/2 \end{bmatrix}. \]

One normalized pair is

\[ \boxed{ q_1= \frac1{\sqrt2} \begin{bmatrix} 1\\ 1 \end{bmatrix}, \qquad q_2= \frac1{\sqrt2} \begin{bmatrix} 1\\ -1 \end{bmatrix}}. \]

Gram-Schmidt Preserves Span

Gram-Schmidt changes the generating vectors but not the subspace they generate:

\[ \boxed{ \operatorname{span}\{v_1,\ldots,v_k\} = \operatorname{span}\{u_1,\ldots,u_k\}}. \]

The result is a geometrically cleaner basis for the same space.

QR Factorization

Applying Gram-Schmidt to the independent columns of a matrix \(A\) produces an orthonormal matrix \(Q\) and an upper triangular matrix \(R\) such that

\[ \boxed{A=QR}. \]

QR factorization is a major computational tool in least squares and eigenvalue algorithms.

Inner Products Beyond \(\mathbb{R}^n\)

Inner products can be defined on many vector spaces.

For continuous functions on an interval, one common example is

\[ \boxed{ \langle f,g\rangle = \int_a^b f(x)g(x)\,dx}. \]

This lets us discuss orthogonal functions just as we discuss orthogonal vectors.

Orthogonal Functions

Functions \(f\) and \(g\) are orthogonal under the integral inner product when

\[ \boxed{ \int_a^b f(x)g(x)\,dx=0}. \]

Orthogonal function systems appear in Fourier analysis, differential equations, approximation theory, and signal processing.

Inner Products on Matrix Spaces

For real matrices \(A\) and \(B\) of the same size, the Frobenius inner product is

\[ \boxed{ \langle A,B\rangle = \operatorname{tr}(A^TB)}. \]

This equals the sum of the products of corresponding entries.

It turns matrix spaces into geometric inner-product spaces.

Weighted Inner Products

Geometry can be modified by assigning different importance to different directions.

If \(M\) is symmetric positive definite, then

\[ \boxed{ \langle u,v\rangle_M = u^TMv} \]

defines an inner product.

Such weighted inner products appear in optimization, statistics, mechanics, and numerical analysis.

Orthogonality Depends on the Inner Product

Two vectors may be orthogonal under one inner product and not under another.

Therefore orthogonality is not only a property of the vectors; it also depends on the geometric structure chosen for the space.

Projection as Best Approximation

Let \(W\) be a subspace and let

\[ p=\operatorname{proj}_W(v). \]

Then \(p\) is the vector in \(W\) closest to \(v\):

\[ \boxed{ \|v-p\| \le \|v-w\| \quad \text{for every }w\in W}. \]

This is the central geometric idea behind least squares.

Why Orthogonality Produces the Closest Point

For any \(w\in W\),

\[ v-w = (v-p)+(p-w). \]

Here

\[ v-p\in W^\perp \]

while

\[ p-w\in W. \]

Therefore the two pieces are orthogonal.

By the Pythagorean theorem,

\[ \|v-w\|^2 = \|v-p\|^2+\|p-w\|^2 \ge \|v-p\|^2. \]

Projection Matrices

If the columns of \(Q\) form an orthonormal basis for a subspace \(W\), then the projection onto \(W\) is

\[ \boxed{P=QQ^T}. \]

For any vector \(v\),

\[ \operatorname{proj}_W(v)=Pv. \]

Properties of Projection Matrices

An orthogonal projection matrix satisfies

\[ \boxed{P^2=P} \]

and

\[ \boxed{P^T=P}. \]

Applying the projection twice has the same effect as applying it once.

Symmetric Matrices and Orthogonality

Real symmetric matrices have orthogonal eigenspaces corresponding to distinct eigenvalues.

This fact allows a real symmetric matrix to be diagonalized by an orthogonal matrix:

\[ \boxed{A=QDQ^T}. \]

This is one form of the spectral theorem.

Why Orthonormal Coordinates Are Numerically Useful

Orthonormal bases avoid unnecessary stretching in coordinate conversion.

Because

\[ Q^{-1}=Q^T, \]

inversion becomes inexpensive and stable.

Lengths and angles are also preserved exactly in ideal arithmetic.

Common Error: Confusing Orthogonal with Independent

Orthogonal nonzero vectors are always independent.

But independent vectors do not need to be orthogonal.

Orthogonality is a stronger geometric relationship.

Common Error: Forgetting to Normalize

An orthogonal set is not automatically orthonormal.

Each vector must also have norm \(1\).

Normalize using

\[ q=\frac{v}{\|v\|}. \]

Common Error: Dividing a Projection by \(\|u\|\)

The general projection formula is

\[ \boxed{ \operatorname{proj}_u(v) = \frac{v\cdot u}{u\cdot u}u}. \]

The denominator is

\[ u\cdot u=\|u\|^2, \]

not merely \(\|u\|\).

Common Error: Using the Unit-Vector Formula on a Nonunit Vector

The simplified formula

\[ \operatorname{proj}_u(v) = (v\cdot u)u \]

is valid only when

\[ \|u\|=1. \]

Common Error: Assuming Orthogonal Complement Means Set Difference

The notation

\[ W^\perp \]

does not mean the vectors outside \(W\).

It means the vectors orthogonal to every vector in \(W\).

Common Error: Including the Zero Vector in an Orthogonal Basis

The zero vector is orthogonal to every vector, but it cannot belong to a basis because it makes the set linearly dependent.

Orthogonal bases consist of nonzero vectors.

Common Error: Skipping Earlier Directions in Gram-Schmidt

At step \(j\), the vector \(v_j\) must have all components along

\[ u_1,\ldots,u_{j-1} \]

removed.

Subtracting only the most recent projection generally fails to produce a fully orthogonal set.

A Reliable Procedure for Dot Products and Angles

Multiply corresponding vector components.
Add the products to obtain \(u\cdot v\).
Compute \(\|u\|\) and \(\|v\|\) if an angle is needed.
Use \(\cos\theta=(u\cdot v)/(\|u\|\|v\|)\).
Interpret the sign of the dot product when a qualitative angle test is enough.

A Reliable Procedure for Projection onto a Vector

Compute \(v\cdot u\).
Compute \(u\cdot u\).
Form the scalar \((v\cdot u)/(u\cdot u)\).
Multiply that scalar by \(u\).
Subtract from \(v\) if the orthogonal residual is needed.

A Reliable Procedure for Projection onto a Subspace

Find an orthonormal basis \(q_1,\ldots,q_k\) for the subspace.
Compute each coefficient \(\langle v,q_i\rangle\).
Multiply each coefficient by its basis vector.
Add the components.
Check that the residual \(v-\operatorname{proj}_W(v)\) is orthogonal to the subspace.

A Reliable Procedure for Gram-Schmidt

Start with a linearly independent list \(v_1,\ldots,v_k\).
Set \(u_1=v_1\).
For each new \(v_j\), subtract its projections onto all earlier \(u_i\).
Verify that the resulting \(u_j\) is nonzero.
Normalize each \(u_j\) if an orthonormal basis is desired.
Check pairwise inner products when needed.

Example: Full Projection Workflow

Let

\[ v= \begin{bmatrix} 4\\ 2\\ 3 \end{bmatrix} \]

and let

\[ W= \operatorname{span} \left\{ e_1,e_2 \right\}. \]

The vectors \(e_1,e_2\) form an orthonormal basis for the \(xy\)-plane.

Therefore,

\[ \operatorname{proj}_W(v) = (v\cdot e_1)e_1 + (v\cdot e_2)e_2. \]

Since

\[ v\cdot e_1=4, \qquad v\cdot e_2=2, \]

we obtain

\[ \boxed{ \operatorname{proj}_W(v) = \begin{bmatrix} 4\\ 2\\ 0 \end{bmatrix}}. \]

The residual is

\[ \boxed{ v-\operatorname{proj}_W(v) = \begin{bmatrix} 0\\ 0\\ 3 \end{bmatrix}}, \]

which is orthogonal to the \(xy\)-plane.

Example: Full Orthogonal-Basis Workflow

Let

\[ v_1= \begin{bmatrix} 1\\ 1\\ 0 \end{bmatrix}, \qquad v_2= \begin{bmatrix} 1\\ 0\\ 1 \end{bmatrix}. \]

Start with

\[ u_1=v_1. \]

Compute

\[ \operatorname{proj}_{u_1}(v_2) = \frac{v_2\cdot u_1}{u_1\cdot u_1}u_1. \]

Here

\[ v_2\cdot u_1=1, \qquad u_1\cdot u_1=2. \]

Thus

\[ u_2 = v_2-\frac12u_1 = \begin{bmatrix} 1/2\\ -1/2\\ 1 \end{bmatrix}. \]

The vectors \(u_1,u_2\) are orthogonal and span the same subspace as \(v_1,v_2\).

Why Inner Products Matter in Multidimensional Systems

Inner products let us compare directions, measure error, define distance, and identify independent geometric components in high-dimensional spaces.

Orthogonality makes those components easier to separate and interpret. Projection then turns geometric approximation into an algebraic process.

These ideas support least squares, QR factorization, Fourier methods, principal directions, spectral analysis, numerical optimization, and dimensional reduction.

Practice 1: Dot Product

Compute

\[ \begin{bmatrix} 2\\ 3 \end{bmatrix} \cdot \begin{bmatrix} 4\\ -1 \end{bmatrix}. \]

Practice 2: Norm

Find the norm of

\[ \begin{bmatrix} 5\\ 12 \end{bmatrix}. \]

Practice 3: Orthogonality

Are

\[ \begin{bmatrix} 1\\ 2 \end{bmatrix} \quad\text{and}\quad \begin{bmatrix} 4\\ -2 \end{bmatrix} \]

orthogonal?

Practice 4: Normalize a Vector

Normalize

\[ \begin{bmatrix} 6\\ 8 \end{bmatrix}. \]

Practice 5: Projection

Project

\[ v= \begin{bmatrix} 3\\ 4 \end{bmatrix} \]

onto

\[ u= \begin{bmatrix} 1\\ 0 \end{bmatrix}. \]

Practice 6: Orthogonal Complement Dimension

If \(W\subseteq\mathbb{R}^7\) has dimension \(3\), what is \(\dim(W^\perp)\)?

Practice 7: Orthonormal Coordinates

If \(q_1,q_2\) form an orthonormal basis and

\[ v=5q_1-2q_2, \]

what are

\[ \langle v,q_1\rangle \quad\text{and}\quad \langle v,q_2\rangle? \]

Practice 8: Pythagorean Identity

If \(u\perp v\), \(\|u\|=3\), and \(\|v\|=4\), find \(\|u+v\|\).

Practice 9: Orthogonal Matrix

If \(Q\) is orthogonal, what is \(Q^{-1}\)?

Practice 10: Projection Matrix

If the columns of \(Q\) form an orthonormal basis for \(W\), what matrix projects vectors onto \(W\)?

Practice 11: Gram-Schmidt

Let

\[ v_1= \begin{bmatrix} 1\\ 0 \end{bmatrix}, \qquad v_2= \begin{bmatrix} 1\\ 1 \end{bmatrix}. \]

Use one Gram-Schmidt step to find a vector \(u_2\) orthogonal to \(v_1\).

Practice 12: Best Approximation

If \(p=\operatorname{proj}_W(v)\), what geometric property does the error

\[ e=v-p \]

satisfy?

Practice Answers

Practice 1

\[ 2(4)+3(-1)=8-3=\boxed{5}. \]

Practice 2

\[ \sqrt{5^2+12^2} = \sqrt{25+144} = \sqrt{169} = \boxed{13}. \]

Practice 3

\[ 1(4)+2(-2)=4-4=0. \]

Therefore,

\[ \boxed{\text{the vectors are orthogonal}}. \]

Practice 4

The norm is

\[ \sqrt{6^2+8^2}=10. \]

Therefore,

\[ \boxed{ \frac1{10} \begin{bmatrix} 6\\ 8 \end{bmatrix} = \begin{bmatrix} 3/5\\ 4/5 \end{bmatrix}}. \]

Practice 5

\[ \operatorname{proj}_u(v) = \frac{v\cdot u}{u\cdot u}u = \frac{3}{1} \begin{bmatrix} 1\\ 0 \end{bmatrix}. \]

Thus

\[ \boxed{ \begin{bmatrix} 3\\ 0 \end{bmatrix}}. \]

Practice 6

\[ \dim(W^\perp) = 7-3 = \boxed{4}. \]

Practice 7

In an orthonormal basis, coefficients are inner products:

\[ \boxed{\langle v,q_1\rangle=5}, \qquad \boxed{\langle v,q_2\rangle=-2}. \]

Practice 8

\[ \|u+v\|^2 = 3^2+4^2 = 25. \]

Therefore,

\[ \boxed{\|u+v\|=5}. \]

Practice 9

\[ \boxed{Q^{-1}=Q^T}. \]

Practice 10

\[ \boxed{P=QQ^T}. \]

Practice 11

Since

\[ \operatorname{proj}_{v_1}(v_2) = \begin{bmatrix} 1\\ 0 \end{bmatrix}, \]

we obtain

\[ u_2 = v_2-\operatorname{proj}_{v_1}(v_2) = \boxed{ \begin{bmatrix} 0\\ 1 \end{bmatrix}}. \]

Practice 12

The error is orthogonal to the entire subspace:

\[ \boxed{e\in W^\perp}. \]

This is why \(p\) is the closest vector in \(W\) to \(v\).

What to Remember

The standard inner product in \(\mathbb{R}^n\) is

\[ \boxed{\langle u,v\rangle=u^Tv}. \]

Norm is defined by \(\|v\|=\sqrt{\langle v,v\rangle}\).
Orthogonal vectors have zero inner product.
Orthogonal nonzero vectors are automatically independent.
Orthonormal vectors are orthogonal unit vectors.
Projection extracts the component of one vector along another vector or subspace.
The projection error lies in the orthogonal complement.
\(\dim(W)+\dim(W^\perp)=n\) in \(\mathbb{R}^n\).
Gram-Schmidt converts an independent set into an orthogonal or orthonormal basis.
Orthogonal matrices satisfy \(Q^{-1}=Q^T\).
Projection onto an orthonormal-column space has matrix \(QQ^T\).

The central geometric principle is:

\[ \boxed{ \text{orthogonality separates a vector into independent geometric components}}. \]

Where We Go Next

Projection onto a subspace gives the closest possible vector in that subspace to a target. This becomes especially useful when a linear system has no exact solution.

The next lesson, Least Squares, uses orthogonal projection to find best-fit solutions to inconsistent systems and develops the normal equations, residual vectors, regression geometry, and QR-based solution methods.