The Central Idea
In ordinary Euclidean geometry, a vector can be decomposed into components along perpendicular directions. If \(\mathbf e_1,\mathbf e_2,\ldots,\mathbf e_n\) are mutually orthogonal basis vectors, then a vector \(\mathbf v\) can be written as
\[ \mathbf v=c_1\mathbf e_1+c_2\mathbf e_2+\cdots+c_n\mathbf e_n. \]An orthogonal expansion applies the same idea to functions. Instead of decomposing a vector into perpendicular vectors, we decompose a function into mutually orthogonal functions:
\[ f(x)\approx c_0\phi_0(x)+c_1\phi_1(x)+c_2\phi_2(x)+\cdots. \]The functions \(\phi_0,\phi_1,\phi_2,\ldots\) form the directions of a function space. The coefficients \(c_k\) tell us how much of each direction is present in \(f\).
From Vectors to Functions
The familiar dot product of two vectors,
\[ \mathbf u\cdot\mathbf v, \]measures how strongly the vectors point in the same direction. Perpendicular vectors satisfy
\[ \mathbf u\cdot\mathbf v=0. \]For functions, integration can play a similar role. On an interval \([a,b]\), a common inner product is
\[ \boxed{\langle f,g\rangle=\int_a^b f(x)g(x)\,dx}. \]If this integral is zero, the functions are orthogonal relative to that inner product.
What Orthogonality Means for Functions
Two functions \(f\) and \(g\) are orthogonal on \([a,b]\) when
\[ \boxed{\langle f,g\rangle=0}. \]Under the standard integral inner product, this means
\[ \int_a^b f(x)g(x)\,dx=0. \]Orthogonality does not mean that one function is visually perpendicular to another on a graph. It is an algebraic relationship defined by the inner product chosen for the problem.
Example: Two Orthogonal Functions
Consider the functions
\[ f(x)=1, \qquad g(x)=x \]on the symmetric interval \([-1,1]\). Their inner product is
\[ \langle 1,x\rangle =\int_{-1}^{1}x\,dx. \]Because \(x\) is odd and the interval is symmetric,
\[ \int_{-1}^{1}x\,dx=0. \]Therefore,
\[ \boxed{1\perp x\text{ on }[-1,1]}. \]Orthogonal Sets
A collection of functions
\[ \phi_0,\phi_1,\phi_2,\ldots \]is an orthogonal set when every distinct pair is orthogonal:
\[ \boxed{\langle \phi_j,\phi_k\rangle=0\quad\text{whenever }j\ne k}. \]The crucial benefit is separation. When one basis function is projected against another, the cross-term disappears. This lets us solve for expansion coefficients one at a time rather than solving a large coupled system.
The Norm of a Function
An inner product also gives a notion of length. The norm of a function is
\[ \boxed{\|f\|=\sqrt{\langle f,f\rangle}}. \]With the standard integral inner product,
\[ \|f\| =\sqrt{\int_a^b |f(x)|^2\,dx}. \]The squared norm,
\[ \|f\|^2=\langle f,f\rangle, \]behaves much like the squared length of a vector.
Normalization and Orthonormal Functions
An orthogonal function does not necessarily have norm \(1\). We can normalize a nonzero function \(\phi_k\) by dividing by its norm:
\[ e_k(x)=\frac{\phi_k(x)}{\|\phi_k\|}. \]A set is orthonormal when
\[ \boxed{\langle e_j,e_k\rangle= \begin{cases} 1,&j=k,\\ 0,&j\ne k. \end{cases}} \]Orthonormal systems are especially convenient because coefficient formulas become simpler.
Projection onto One Function
Suppose we want to approximate a function \(f\) by a multiple of a nonzero function \(\phi\):
\[ f(x)\approx c\phi(x). \]The best coefficient in the least-squares sense is obtained by projection:
\[ \boxed{c=\frac{\langle f,\phi\rangle}{\langle \phi,\phi\rangle}}. \]This is exactly analogous to projecting a vector onto another vector.
Why the Projection Formula Works
Let
\[ r=f-c\phi \]be the residual, or approximation error. For the projection to be optimal, the residual should be orthogonal to the direction \(\phi\):
\[ \langle r,\phi\rangle=0. \]Substitute \(r=f-c\phi\):
\[ \langle f-c\phi,\phi\rangle=0. \]By linearity,
\[ \langle f,\phi\rangle-c\langle \phi,\phi\rangle=0. \]Therefore,
\[ \boxed{c=\frac{\langle f,\phi\rangle}{\langle \phi,\phi\rangle}}. \]Projection onto an Orthogonal Set
Suppose \(\phi_0,\phi_1,\ldots,\phi_n\) are mutually orthogonal and we seek an approximation
\[ f_n(x)=\sum_{k=0}^{n}c_k\phi_k(x). \]Each coefficient is found independently:
\[ \boxed{c_k=\frac{\langle f,\phi_k\rangle} {\langle \phi_k,\phi_k\rangle}}. \]This coefficient formula is one of the main engines behind orthogonal expansions.
The Orthonormal Coefficient Formula
If the basis functions are orthonormal, then
\[ \langle e_k,e_k\rangle=1. \]The denominator disappears, leaving
\[ \boxed{c_k=\langle f,e_k\rangle}. \]The expansion becomes
\[ f(x)\approx\sum_{k=0}^{n}\langle f,e_k\rangle e_k(x). \]This compact form is a major reason orthonormal bases are preferred in both theory and computation.
Exact Expansion versus Approximation
In a finite-dimensional vector space, a complete basis represents every vector exactly using finitely many terms. Function spaces can be infinite-dimensional, so an exact representation may require infinitely many basis functions:
\[ f(x)=\sum_{k=0}^{\infty}c_k\phi_k(x). \]In computation, we usually truncate the expansion after \(n\) terms:
\[ \boxed{f_n(x)=\sum_{k=0}^{n}c_k\phi_k(x)}. \]The practical question then becomes how rapidly \(f_n\) approaches \(f\) as more terms are included.
A Finite-Dimensional Analogy
Suppose
\[ \mathbf v=3\mathbf e_1-2\mathbf e_2+5\mathbf e_3 \]in an orthonormal basis. The coefficients are recovered by dot products:
\[ 3=\mathbf v\cdot\mathbf e_1, \qquad -2=\mathbf v\cdot\mathbf e_2, \qquad 5=\mathbf v\cdot\mathbf e_3. \]Orthogonal function expansions use the same architecture. Replace vector dot products with function inner products, and replace basis vectors with basis functions.
Example: Projection onto the Constant Function
Approximate \(f(x)=x^2\) on \([-1,1]\) using only the basis function \(\phi_0(x)=1\).
The coefficient is
\[ c_0= \frac{\langle x^2,1\rangle}{\langle 1,1\rangle}. \]Compute the numerator:
\[ \langle x^2,1\rangle =\int_{-1}^{1}x^2\,dx =\frac{2}{3}. \]Compute the denominator:
\[ \langle 1,1\rangle =\int_{-1}^{1}1\,dx =2. \]Therefore,
\[ \boxed{c_0=\frac13}. \]The best constant approximation in this inner-product sense is
\[ \boxed{x^2\approx\frac13}. \]The Constant Projection Is an Average
When the basis function is \(1\) on \([a,b]\), the projection coefficient is
\[ c= \frac{\int_a^b f(x)\,dx}{\int_a^b 1\,dx} =\frac{1}{b-a}\int_a^b f(x)\,dx. \]Therefore,
\[ \boxed{c=f_{\text{avg}}}. \]The ordinary average value of a function is itself a simple orthogonal projection.
Example: Two-Term Orthogonal Approximation
Approximate \(f(x)=1+x+x^2\) on \([-1,1]\) using
\[ \phi_0(x)=1, \qquad \phi_1(x)=x. \]These functions are orthogonal because
\[ \int_{-1}^{1}x\,dx=0. \]The constant coefficient is
\[ c_0= \frac{\int_{-1}^{1}(1+x+x^2)\,dx}{\int_{-1}^{1}1\,dx} =\frac{2+0+\frac23}{2} =\frac43. \]The linear coefficient is
\[ c_1= \frac{\int_{-1}^{1}(1+x+x^2)x\,dx}{\int_{-1}^{1}x^2\,dx}. \]The numerator becomes
\[ \int_{-1}^{1}(x+x^2+x^3)\,dx =\frac23, \]while the denominator is also \(\frac23\). Thus
\[ c_1=1. \]The two-term projection is
\[ \boxed{f_1(x)=\frac43+x}. \]Why Symmetry Simplifies Coefficients
On symmetric intervals such as \([-L,L]\), parity often eliminates integrals. Recall:
- an odd integrand integrates to zero over \([-L,L]\);
- an even integrand can be integrated over \([0,L]\) and doubled.
These symmetry rules become extremely useful in trigonometric expansions because sine functions are odd and cosine functions are even.
Weighted Inner Products
Some problems use a weight function \(w(x)\) instead of the standard inner product. A weighted inner product has the form
\[ \boxed{\langle f,g\rangle_w =\int_a^b f(x)g(x)w(x)\,dx}. \]The weight changes the geometry of the function space. Regions where \(w(x)\) is larger contribute more strongly to the inner product and therefore to the notion of approximation error.
Orthogonality Depends on the Inner Product
A pair of functions can be orthogonal under one inner product and not under another. For example, changing the interval or introducing a weight can change whether
\[ \langle f,g\rangle=0. \]For that reason, statements about orthogonality should always be interpreted relative to a specified domain and inner product.
Orthogonal Polynomials
Trigonometric functions are not the only useful orthogonal systems. Polynomial families can also be constructed so that distinct members are orthogonal. Examples encountered in advanced analysis include Legendre, Chebyshev, Hermite, and Laguerre polynomials.
Different families are adapted to different intervals, weights, equations, and computational goals. This flexibility is one reason orthogonal expansions appear throughout approximation theory and spectral methods.
Building Orthogonal Functions by Gram-Schmidt
The Gram-Schmidt process used for vectors can be extended to functions. Starting from linearly independent functions \(v_0,v_1,v_2,\ldots\), we can remove from each new function the components lying in directions already constructed.
For example,
\[ u_0=v_0, \] \[ u_1=v_1- \frac{\langle v_1,u_0\rangle}{\langle u_0,u_0\rangle}u_0, \]and the process continues similarly. The resulting functions are mutually orthogonal when the required inner products exist and the denominators are nonzero.
Example: Orthogonalizing \(1\) and \(x\)
On \([-1,1]\), begin with
\[ v_0(x)=1, \qquad v_1(x)=x. \]Set
\[ u_0(x)=1. \]Then
\[ u_1(x)=x- \frac{\langle x,1\rangle}{\langle 1,1\rangle}. \]Since \(\langle x,1\rangle=0\), we obtain
\[ \boxed{u_1(x)=x}. \]The pair was already orthogonal, so Gram-Schmidt leaves it unchanged.
Example: Orthogonalizing \(x^2\) against \(1\)
On \([-1,1]\), let
\[ v_0(x)=1, \qquad v_1(x)=x^2. \]The projection of \(x^2\) onto \(1\) is \(\frac13\), so
\[ u_1(x)=x^2-\frac13. \]Check orthogonality:
\[ \int_{-1}^{1}\left(x^2-\frac13\right)\,dx =\frac23-\frac23 =0. \]Therefore,
\[ \boxed{1\perp\left(x^2-\frac13\right)}. \]Least-Squares Approximation
Suppose
\[ V_n=\operatorname{span}\{\phi_0,\phi_1,\ldots,\phi_n\}. \]The orthogonal projection of \(f\) onto \(V_n\) is the member of that subspace that minimizes the squared error
\[ \|f-f_n\|^2. \]Thus orthogonal expansion coefficients are not merely convenient algebraic quantities. They produce a best approximation within the chosen finite-dimensional subspace under the selected norm.
The Residual Is Orthogonal to the Approximation Space
If
\[ r_n=f-f_n, \]then the projection condition is
\[ \boxed{\langle r_n,\phi_k\rangle=0 \qquad k=0,1,\ldots,n}. \]In words, once the best approximation has been chosen, none of the remaining error points in any direction available inside the approximation space.
Energy and Coefficient Size
In an orthogonal expansion, the squared norm of a finite sum separates cleanly. If
\[ f_n=\sum_{k=0}^{n}c_k\phi_k, \]then orthogonality gives
\[ \boxed{\|f_n\|^2 =\sum_{k=0}^{n}|c_k|^2\|\phi_k\|^2}. \]For an orthonormal system this simplifies to
\[ \boxed{\|f_n\|^2=\sum_{k=0}^{n}|c_k|^2}. \]This is the function-space version of the Pythagorean theorem.
Bessel's Inequality
For an orthonormal set \(e_k\), the projection coefficients cannot collectively contain more squared magnitude than the original function. In a suitable inner- product space,
\[ \boxed{\sum_k |\langle f,e_k\rangle|^2\le \|f\|^2}. \]This result is known as Bessel's inequality. It formalizes the idea that projections capture portions of the total size or energy of a function.
Parseval-Type Identities
When an orthonormal system is complete for the function under consideration, the inequality can become an equality:
\[ \boxed{\|f\|^2=\sum_k |c_k|^2}. \]Such identities are often called Parseval-type relations. In Fourier analysis, they connect the energy measured in the original function to the energy stored in its expansion coefficients.
Completeness
Orthogonality alone does not guarantee that every function can be reconstructed from a given collection. The collection must also be sufficiently rich.
Informally, an orthogonal system is complete when no nonzero direction remains outside the span of its basis functions. In a complete system, increasingly long expansions can represent a broad class of functions with arbitrarily small error in the relevant norm.
Truncation Error
If the full expansion is
\[ f=\sum_{k=0}^{\infty}c_k\phi_k, \]and we retain only the first \(n+1\) terms, then
\[ f_n=\sum_{k=0}^{n}c_k\phi_k. \]The truncation error is
\[ \boxed{r_n=f-f_n}. \]A useful expansion is one in which the important information is concentrated in relatively few coefficients, so that the residual decreases rapidly as \(n\) increases.
Coefficient Decay
In many orthogonal expansions, smooth functions tend to produce coefficients that decrease in magnitude as the index increases. Faster coefficient decay usually means that fewer terms are needed for a good approximation.
Discontinuities, sharp corners, singular behavior, or poor basis choices can slow coefficient decay. This connection between regularity and spectral content becomes important in Fourier analysis and numerical spectral methods.
Choosing a Useful Basis
A mathematically valid basis is not automatically a computationally efficient one. A useful orthogonal system should reflect the structure of the problem. Important considerations include:
- the interval or geometric domain;
- boundary conditions;
- symmetry;
- periodicity;
- smoothness of the target function;
- the differential operator involved;
- the weight used in the inner product;
- computational cost and numerical stability.
Eigenfunctions as Natural Expansion Directions
Differential and integral operators often possess special functions called eigenfunctions. If an operator \(L\) satisfies
\[ L\phi_k=\lambda_k\phi_k, \]then \(\phi_k\) is an eigenfunction with eigenvalue \(\lambda_k\). Under suitable conditions, eigenfunctions belonging to distinct eigenvalues may form orthogonal families.
Expanding a solution in such functions can transform an operator equation into simpler algebraic equations for the coefficients. This is one of the principal ideas behind spectral decomposition.
Orthogonal Expansions and Boundary-Value Problems
Boundary-value problems frequently lead to orthogonal eigenfunction systems. Instead of solving directly for an unknown function \(u(x)\), we may seek
\[ u(x)=\sum_{k=0}^{\infty}c_k\phi_k(x), \]where the \(\phi_k\) satisfy the relevant boundary conditions. Substitution into the differential equation can reduce the original problem to equations for the coefficients \(c_k\).
Orthogonal Expansions and Data Compression
Orthogonal coordinates are useful beyond symbolic analysis. If most of a signal, image, or dataset is captured by a small number of large coefficients, the smaller coefficients may contribute comparatively little to reconstruction.
This principle supports dimensional reduction, compression, denoising, and feature extraction. Different transform families provide different coordinate systems for exposing structure in the data.
Orthogonal Expansions and Filtering
Once a function or signal has been decomposed into orthogonal components, we can modify the coefficients selectively. For example, we may suppress components associated with unwanted oscillation or retain components associated with a desired scale.
The reconstructed function then reflects those coefficient-level changes. This idea will reappear in frequency-domain filtering and signal analysis.
Discrete Orthogonality
Orthogonality does not require an integral. For sampled data, an inner product can be defined by a finite sum:
\[ \boxed{\langle f,g\rangle =\sum_{j=0}^{N-1} f_j\overline{g_j}}. \]This discrete viewpoint connects orthogonal expansions to matrices, numerical linear algebra, and the discrete Fourier transform.
Matrix View of an Orthogonal Expansion
After sampling or discretization, basis functions can be arranged as columns of a matrix \(Q\). If those columns are orthonormal, then
\[ Q^{*}Q=I. \]The coefficient vector is obtained by
\[ \boxed{\mathbf c=Q^{*}\mathbf f}, \]while reconstruction is
\[ \boxed{\mathbf f\approx Q\mathbf c}. \]This gives a direct bridge between continuous orthogonal expansions and computational linear algebra.
Complex-Valued Inner Products
Fourier analysis often uses complex exponentials. For complex-valued functions, the standard inner product includes complex conjugation:
\[ \boxed{\langle f,g\rangle =\int_a^b f(x)\overline{g(x)}\,dx}. \]This convention ensures that
\[ \langle f,f\rangle =\int_a^b |f(x)|^2\,dx\ge 0. \]The same projection ideas then extend naturally to complex bases.
Common Error: Assuming Visual Perpendicularity
Orthogonal functions do not have to cross at right angles on a graph. The definition is determined by an inner product, such as
\[ \int_a^b f(x)g(x)\,dx=0. \]Always test the inner product rather than relying on the appearance of the graphs.
Common Error: Forgetting the Denominator
For an orthogonal but not normalized function \(\phi_k\), the coefficient is
\[ c_k= \frac{\langle f,\phi_k\rangle} {\langle \phi_k,\phi_k\rangle}, \]not merely \(\langle f,\phi_k\rangle\). The simpler formula applies only when the basis function has unit norm.
Common Error: Changing the Interval Unnoticed
Orthogonality can depend on the integration interval. A pair that is orthogonal on \([-1,1]\) may fail to be orthogonal on \([0,1]\).
Always keep the interval, weighting, and boundary conditions attached to the expansion problem.
Common Error: Treating Truncation as Exact Equality
A finite expansion
\[ f_n=\sum_{k=0}^{n}c_k\phi_k \]is usually an approximation unless \(f\) already lies in the span of the chosen basis functions. Use an approximation sign when exact equality has not been established.
A Reliable Procedure for Orthogonal Expansion
- Identify the interval or domain.
- Specify the inner product and any weight function.
- Confirm that the proposed basis functions are orthogonal.
- Normalize the basis if an orthonormal form is desirable.
- Compute each coefficient using projection.
- Assemble the finite or infinite expansion.
- Check symmetry before evaluating unnecessary integrals.
- Measure or estimate the residual when using a truncated expansion.
- Interpret the coefficients in the context of the problem.
Example: Full Projection Workflow
Approximate
\[ f(x)=x^2+x \]on \([-1,1]\) using \(\phi_0(x)=1\) and \(\phi_1(x)=x\).
First confirm orthogonality:
\[ \langle 1,x\rangle =\int_{-1}^{1}x\,dx =0. \]Constant coefficient:
\[ c_0= \frac{\int_{-1}^{1}(x^2+x)\,dx}{2} =\frac{\frac23}{2} =\frac13. \]Linear coefficient:
\[ c_1= \frac{\int_{-1}^{1}(x^2+x)x\,dx} {\int_{-1}^{1}x^2\,dx}. \]The numerator is
\[ \int_{-1}^{1}(x^3+x^2)\,dx=\frac23, \]and the denominator is \(\frac23\). Therefore,
\[ c_1=1. \]The projection is
\[ \boxed{f_1(x)=\frac13+x}. \]The remaining term \(x^2-\frac13\) is orthogonal to both \(1\) and \(x\) on this interval.
Practice 1: Test Orthogonality
Determine whether \(f(x)=1\) and \(g(x)=x^3\) are orthogonal on \([-2,2]\) under the standard integral inner product.
Practice 2: Compute a Norm
Find \(\|x\|\) on \([-1,1]\) using
\[ \langle f,g\rangle=\int_{-1}^{1}f(x)g(x)\,dx. \]Practice 3: Normalize a Function
Normalize the constant function \(\phi(x)=1\) on \([-1,1]\).
Practice 4: Projection onto a Constant
Find the best constant approximation to \(f(x)=x\) on \([0,2]\) under the standard integral inner product.
Practice 5: Orthogonal Coefficients
On \([-1,1]\), use \(\phi_0(x)=1\) and \(\phi_1(x)=x\) to approximate
\[ f(x)=2+3x+x^2. \]Practice 6: Weighted Inner Product
Write the weighted inner product of \(f\) and \(g\) on \([0,1]\) when \(w(x)=x\).
Practice 7: Identify the Error
A student uses an orthogonal function \(\phi\) with \(\|\phi\|=2\) and writes
\[ c=\langle f,\phi\rangle. \]Explain what is missing.
Practice 8: Residual Orthogonality
If \(f_n\) is the orthogonal projection of \(f\) onto \(\operatorname{span}\{\phi_0,\phi_1\}\), what inner-product conditions must the residual \(r=f-f_n\) satisfy?
Practice Answers
Practice 1
Because \(x^3\) is odd and \([-2,2]\) is symmetric,
\[ \langle 1,x^3\rangle =\int_{-2}^{2}x^3\,dx =0. \]Therefore the functions are orthogonal.
Practice 2
\[ \|x\| =\sqrt{\int_{-1}^{1}x^2\,dx} =\sqrt{\frac23}. \]Thus,
\[ \boxed{\|x\|=\sqrt{\frac23}}. \]Practice 3
Since
\[ \|1\|=\sqrt{\int_{-1}^{1}1\,dx}=\sqrt2, \]the normalized function is
\[ \boxed{e(x)=\frac{1}{\sqrt2}}. \]Practice 4
The best constant is the average value:
\[ c=\frac{1}{2-0}\int_0^2x\,dx =\frac12\left[\frac{x^2}{2}\right]_0^2 =1. \]Therefore,
\[ \boxed{f(x)\approx1}. \]Practice 5
The constant coefficient is
\[ c_0= \frac{\int_{-1}^{1}(2+3x+x^2)\,dx}{2} =\frac{4+\frac23}{2} =\frac73. \]The linear coefficient is
\[ c_1= \frac{\int_{-1}^{1}(2+3x+x^2)x\,dx}{\int_{-1}^{1}x^2\,dx} =3. \]Therefore,
\[ \boxed{f_1(x)=\frac73+3x}. \]Practice 6
\[ \boxed{\langle f,g\rangle_w =\int_0^1 f(x)g(x)x\,dx}. \]Practice 7
Since \(\|\phi\|=2\), we have
\[ \langle \phi,\phi\rangle=\|\phi\|^2=4. \]Therefore the correct coefficient is
\[ \boxed{c=\frac{\langle f,\phi\rangle}{4}}. \]Practice 8
The residual must be orthogonal to every basis direction used in the projection:
\[ \boxed{\langle r,\phi_0\rangle=0, \qquad \langle r,\phi_1\rangle=0}. \]What to Remember
Orthogonal expansions transfer the geometry of perpendicular vectors into function spaces.
- an inner product defines orthogonality;
- orthogonal functions satisfy \(\langle \phi_j,\phi_k\rangle=0\) for \(j\ne k\);
- orthonormal functions also have unit norm;
- projection determines expansion coefficients;
- orthogonality separates coefficient calculations;
- truncated expansions provide least-squares approximations;
- the residual is orthogonal to the approximation space;
- complete orthogonal systems can reconstruct broad classes of functions;
- coefficient size and decay reveal how information is distributed among basis directions.
The central coefficient formula is
\[ \boxed{c_k= \frac{\langle f,\phi_k\rangle} {\langle \phi_k,\phi_k\rangle}}. \]For an orthonormal basis, this becomes
\[ \boxed{c_k=\langle f,e_k\rangle}. \]Where We Go Next
Orthogonal expansions provide the general framework. The next lesson, Fourier Series, specializes that framework to sines, cosines, and complex exponentials. Those trigonometric basis functions are naturally adapted to periodic behavior and create one of the most important bridges between time-domain functions and frequency-domain representations.