Orthogonality & Orthonormal Bases

🎯 Learning Objectives

• Determine whether vectors are orthogonal
• Define orthogonal complements and find them
• Compute the orthogonal projection of a vector onto a subspace
• Explain why orthonormal bases are convenient
• Express a vector in an orthonormal basis easily via inner products

📋 Prerequisites

• Inner Products & Norms (Week 7) — inner products, angles
• Subspaces (Week 3) — linear subspaces
• Orthogonality is the geometric heart of linear algebra

1. Intuition: Right Angles Everywhere

• Orthogonal vectors are automatically linearly independent
• Coordinates in an orthonormal basis are computed with simple inner products (no solving equations!)
• Projections onto orthogonal subspaces are easy (Diagram)

2. Orthogonal Vectors and Sets

2.1 Orthogonal Set

Definition (Orthogonal Set). A set of vectors $\{\mathbf{v}_1, \dots, \mathbf{v}_k\}$ is orthogonal if $\langle \mathbf{v}_i, \mathbf{v}_j \rangle = 0$ for all $i \neq j$. Theorem. An orthogonal set of non-zero vectors is linearly independent. Proof: Suppose $c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0}$. Take inner product with $\mathbf{v}_i$:

2.2 Orthonormal Set

Definition (Orthonormal Set). An orthonormal set is an orthogonal set where each vector has norm 1:

Example 1: Orthogonal and orthonormal sets

$\{(1,2,-3), (3,0,1), (1,-5,-3)\}$ is orthogonal (check dot products = 0). Normalising gives an orthonormal set.

2.3 Orthonormal Basis

Definition (Orthonormal Basis). An orthonormal basis of $V$ is a basis that is also an orthonormal set. The beauty of orthonormal bases: If $\{\mathbf{e}_1, \dots, \mathbf{e}_n\}$ is orthonormal, then for any $\mathbf{v} \in V$:

Example 2: Coordinates in orthonormal basis

Standard basis of $\mathbb{R}^2$ is orthonormal. For $\mathbf{v} = (3,4)$: $\langle \mathbf{v}, \mathbf{e}_1 \rangle = 3$, $\langle \mathbf{v}, \mathbf{e}_2 \rangle = 4$. So $\mathbf{v} = 3\mathbf{e}_1 + 4\mathbf{e}_2$ — we read coordinates directly. Example 3: A non-standard orthonormal basis

$B = \left\{\left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right), \left(-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right)\right\}$ is orthonormal.

For $\mathbf{v} = (3,1)$: $\langle \mathbf{v}, \mathbf{b}_1 \rangle = \frac{3+1}{\sqrt{2}} = \frac{4}{\sqrt{2}} = 2\sqrt{2}$ $\langle \mathbf{v}, \mathbf{b}_2 \rangle = \frac{-3+1}{\sqrt{2}} = \frac{-2}{\sqrt{2}} = -\sqrt{2}$

Check: $2\sqrt{2}\left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right) + (-\sqrt{2})\left(-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right) = (2,2) + (1,-1) = (3,1)$ ✓

3. Orthogonal Complements

Definition (Orthogonal Complement). For a subspace $W \subseteq V$, the orthogonal complement is:

• $W^\perp$ is a subspace of $V$
• $W \cap W^\perp = \{\mathbf{0}\}$
• $\dim(W) + \dim(W^\perp) = \dim(V)$
• $(W^\perp)^\perp = W$

Example 4: Orthogonal complement in ℝ²

$W = \text{span}\{(1,2)\}$ (a line). $W^\perp = \{(x,y) \mid x + 2y = 0\}$ (the perpendicular line). Basis for $W^\perp$: $\{(-2,1)\}$. Example 5: Orthogonal complement in ℝ³

$W = xy$-plane $\{(x,y,0)\}$. $W^\perp = \{(0,0,z)\}$ (the $z$-axis).

4. Orthogonal Projections

Definition (Orthogonal Projection). For a subspace $W \subseteq V$ with orthonormal basis $\{\mathbf{e}_1, \dots, \mathbf{e}_k\}$, the orthogonal projection of $\mathbf{v}$ onto $W$ is:

Example 6: Projection onto a line

Project $\mathbf{v} = (3,4)$ onto the line $W = \text{span}\{(1,2)\}$.

Normalise: $\mathbf{e} = \frac{(1,2)}{\sqrt{5}}$. $\langle \mathbf{v}, \mathbf{e} \rangle = \frac{3+8}{\sqrt{5}} = \frac{11}{\sqrt{5}}$. $\text{proj}_W(\mathbf{v}) = \frac{11}{\sqrt{5}} \cdot \frac{(1,2)}{\sqrt{5}} = \frac{11}{5}(1,2) = (2.2, 4.4)$.

Check: $\mathbf{v} - \text{proj} = (0.8, -0.4)$, which is orthogonal to $(1,2)$: $0.8(1) + (-0.4)(2) = 0$ ✓

5. Edge Cases & Gotchas

6. Common Pitfalls

❌ Pitfall 1: Thinking orthogonal = independent but not vice versa

❌ Pitfall 2: Normalising before checking orthogonality

❌ Pitfall 3: Forgetting to normalise basis vectors for projection formula

7. Formula Summary Table

8. 📝 Practice Questions

Q1: Orthogonal set?

Is $\{(1,1,1), (1,-1,0), (1,1,-2)\}$ orthogonal?

Solution: $(1,1,1) \cdot (1,-1,0) = 1-1+0 = 0$ ✓ $(1,1,1) \cdot (1,1,-2) = 1+1-2 = 0$ ✓ $(1,-1,0) \cdot (1,1,-2) = 1-1+0 = 0$ ✓

Yes, orthogonal. Q2: Orthonormal from orthogonal

Normalise the orthogonal set from Q1.

Solution: $\|(1,1,1)\| = \sqrt{3}$, so $\mathbf{e}_1 = \left(\frac{1}{\sqrt{3}}, \frac{1}{\sqrt{3}}, \frac{1}{\sqrt{3}}\right)$. $\|(1,-1,0)\| = \sqrt{2}$, so $\mathbf{e}_2 = \left(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}, 0\right)$. $\|(1,1,-2)\| = \sqrt{6}$, so $\mathbf{e}_3 = \left(\frac{1}{\sqrt{6}}, \frac{1}{\sqrt{6}}, -\frac{2}{\sqrt{6}}\right)$. Q3: Coordinates in orthonormal basis

Express $(2,5,3)$ in the orthonormal basis from Q2.

Solution: $c_1 = \langle (2,5,3), \mathbf{e}_1 \rangle = \frac{2+5+3}{\sqrt{3}} = \frac{10}{\sqrt{3}}$ $c_2 = \langle (2,5,3), \mathbf{e}_2 \rangle = \frac{2-5+0}{\sqrt{2}} = -\frac{3}{\sqrt{2}}$ $c_3 = \langle (2,5,3), \mathbf{e}_3 \rangle = \frac{2+5-6}{\sqrt{6}} = \frac{1}{\sqrt{6}}$

So $(2,5,3) = \frac{10}{\sqrt{3}}\mathbf{e}_1 - \frac{3}{\sqrt{2}}\mathbf{e}_2 + \frac{1}{\sqrt{6}}\mathbf{e}_3$. Q4: Orthogonal complement

Find $W^\perp$ where $W = \text{span}\{(1,1,1), (1,0,-1)\}$.

Solution: Solve $\mathbf{w} \perp \text{both}$: $(x,y,z) \cdot (1,1,1) = x+y+z = 0$ $(x,y,z) \cdot (1,0,-1) = x - z = 0 \Rightarrow x = z$. Then $x+y+x = 2x + y = 0 \Rightarrow y = -2x$.

$W^\perp = \text{span}\{(1,-2,1)\}$. Q5: Projection onto a line

Project $(2,1)$ onto the line through $(1,1)$.

Solution: $\mathbf{e} = \frac{(1,1)}{\sqrt{2}}$. $\langle (2,1), \mathbf{e} \rangle = \frac{2+1}{\sqrt{2}} = \frac{3}{\sqrt{2}}$. $\text{proj} = \frac{3}{\sqrt{2}} \cdot \frac{(1,1)}{\sqrt{2}} = \frac{3}{2}(1,1) = (1.5, 1.5)$. Q6: Projection onto a plane

Project $(1,2,3)$ onto the plane $x + y + z = 0$ (with orthonormal basis from Q2's $\mathbf{e}_2, \mathbf{e}_3$).

Solution: $c_2 = \langle (1,2,3), \mathbf{e}_2 \rangle = \frac{1-2+0}{\sqrt{2}} = -\frac{1}{\sqrt{2}}$. $c_3 = \langle (1,2,3), \mathbf{e}_3 \rangle = \frac{1+2-6}{\sqrt{6}} = -\frac{3}{\sqrt{6}} = -\sqrt{\frac{3}{2}}$.

$\text{proj} = -\frac{1}{\sqrt{2}}\mathbf{e}_2 - \sqrt{\frac{3}{2}}\mathbf{e}_3$.

Simplify: $\mathbf{e}_2 + \sqrt{3}\mathbf{e}_3 = \left(\frac{1}{\sqrt{2}}+\sqrt{3}\frac{1}{\sqrt{6}}, -\frac{1}{\sqrt{2}}+\sqrt{3}\frac{1}{\sqrt{6}}, 0+\sqrt{3}\left(-\frac{2}{\sqrt{6}}\right)\right)$ $= \left(\frac{1}{\sqrt{2}}+\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}+\frac{1}{\sqrt{2}}, -\frac{2\sqrt{3}}{\sqrt{6}}\right) = (\sqrt{2}, 0, -\sqrt{2})$.

So $\text{proj} = \frac{-1}{\sqrt{2}}(\sqrt{2}, 0, -\sqrt{2}) = (-1, 0, 1)$.

Check: $(-1,0,1)$ is in the plane $x+y+z=0$ ✓. And $(1,2,3) - (-1,0,1) = (2,2,2)$ is along $(1,1,1)$, perpendicular to the plane ✓. Q7: Orthogonal decomposition

Decompose $(4,7)$ into components parallel and perpendicular to $(2,3)$.

Solution: Project onto $(2,3)$: $\text{proj}_{\mathbf{u}} \mathbf{v} = \frac{\mathbf{v}\cdot\mathbf{u}}{\|\mathbf{u}\|^2}\mathbf{u} = \frac{8+21}{4+9}(2,3) = \frac{29}{13}(2,3) = \left(\frac{58}{13}, \frac{87}{13}\right)$. $\mathbf{v}_\perp = (4,7) - \left(\frac{58}{13}, \frac{87}{13}\right) = \left(\frac{52-58}{13}, \frac{91-87}{13}\right) = \left(-\frac{6}{13}, \frac{4}{13}\right)$. Q8: Orthogonal projection formula check

If $\{\mathbf{e}_1, \dots, \mathbf{e}_k\}$ is orthonormal, show $\|\text{proj}_W(\mathbf{v})\|^2 = \sum_{i=1}^k \langle \mathbf{v}, \mathbf{e}_i \rangle^2$.

Proof: $\|\text{proj}\|^2 = \langle \sum \langle \mathbf{v}, \mathbf{e}_i \rangle \mathbf{e}_i, \sum \langle \mathbf{v}, \mathbf{e}_j \rangle \mathbf{e}_j \rangle = \sum_i \sum_j \langle \mathbf{v}, \mathbf{e}_i \rangle \langle \mathbf{v}, \mathbf{e}_j \rangle \langle \mathbf{e}_i, \mathbf{e}_j \rangle$. Since $\langle \mathbf{e}_i, \mathbf{e}_j \rangle = 0$ for $i \neq j$ and $=1$ for $i=j$: $\|\text{proj}\|^2 = \sum_{i=1}^k \langle \mathbf{v}, \mathbf{e}_i \rangle^2$. ✓ Q9: Orthogonal complement of a sum

Show $(W_1 + W_2)^\perp = W_1^\perp \cap W_2^\perp$.

Proof: $\mathbf{v} \in (W_1 + W_2)^\perp$ iff $\langle \mathbf{v}, \mathbf{w}_1 + \mathbf{w}_2 \rangle = 0$ for all $\mathbf{w}_1 \in W_1, \mathbf{w}_2 \in W_2$. This holds iff $\langle \mathbf{v}, \mathbf{w}_1 \rangle = 0$ for all $\mathbf{w}_1$ and $\langle \mathbf{v}, \mathbf{w}_2 \rangle = 0$ for all $\mathbf{w}_2$, i.e., $\mathbf{v} \in W_1^\perp \cap W_2^\perp$. ✓ Q10: Best approximation

Show that $\text{proj}_W(\mathbf{v})$ is the closest vector in $W$ to $\mathbf{v}$ — i.e., for any $\mathbf{w} \in W$: $\|\mathbf{v} - \text{proj}_W(\mathbf{v})\| \leq \|\mathbf{v} - \mathbf{w}\|$.

Proof: Write $\mathbf{v} - \mathbf{w} = (\mathbf{v} - \text{proj}) + (\text{proj} - \mathbf{w})$. Since $\text{proj} - \mathbf{w} \in W$ and $\mathbf{v} - \text{proj} \in W^\perp$, they're orthogonal. By Pythagoras: $\|\mathbf{v} - \mathbf{w}\|^2 = \|\mathbf{v} - \text{proj}\|^2 + \|\text{proj} - \mathbf{w}\|^2 \geq \|\mathbf{v} - \text{proj}\|^2$. ✓

🔗 Cross-References

• Next topic: Gram-Schmidt & QR Decomposition
• Week 8 (Gram-Schmidt): Builds orthonormal bases from any basis
• Week 12 (Eigenvalues): Spectral theorem — symmetric matrices have orthogonal eigenvectors
• BSCS2004 (ML Foundations): Orthogonal features, PCA Join Discord Previous7.3 Inner Products & NormsNext8.2 Gram-Schmidt & QR