Gram-Schmidt Process & QR Decomposition

🎯 Learning Objectives

• Apply the Gram-Schmidt process to convert any basis into an orthonormal basis
• Compute the $QR$ decomposition of a matrix
• Use Gram-Schmidt to find orthogonal bases for subspaces
• Understand the geometric intuition of subtracting projections

📋 Prerequisites

• Orthogonality (this week) — orthonormal bases, projections
• Inner Products (Week 7) — needed for projections
• Gram-Schmidt is the algorithm that makes orthonormal bases computable

1. Intuition: Building Orthogonality Step by Step

2. The Gram-Schmidt Algorithm

Gram-Schmidt Process. Given a basis $\{\mathbf{v}_1, \dots, \mathbf{v}_n\}$, define:

Continue: $\mathbf{w}_k = \mathbf{v}_k - \sum_{i=1}^{k-1} \langle \mathbf{v}_k, \mathbf{u}_i \rangle \mathbf{u}_i$, then $\mathbf{u}_k = \frac{\mathbf{w}_k}{\|\mathbf{w}_k\|}$. Example 1: Gram-Schmidt in ℝ²

Basis: $\mathbf{v}_1 = (3, 1)$, $\mathbf{v}_2 = (2, 2)$.

Step 1: $\|\mathbf{v}_1\| = \sqrt{9+1} = \sqrt{10}$, $\mathbf{u}_1 = \left(\frac{3}{\sqrt{10}}, \frac{1}{\sqrt{10}}\right)$.

Step 2: $\langle \mathbf{v}_2, \mathbf{u}_1 \rangle = \frac{6+2}{\sqrt{10}} = \frac{8}{\sqrt{10}}$.

$\mathbf{w}_2 = (2,2) - \frac{8}{\sqrt{10}}\left(\frac{3}{\sqrt{10}}, \frac{1}{\sqrt{10}}\right) = (2,2) - \frac{8}{10}(3,1) = (2,2) - (2.4, 0.8) = (-0.4, 1.2)$.

$\|\mathbf{w}_2\| = \sqrt{0.16 + 1.44} = \sqrt{1.6} = \frac{4}{\sqrt{10}}$.

$\mathbf{u}_2 = \frac{(-0.4, 1.2)}{4/\sqrt{10}} = \left(-\frac{1}{\sqrt{10}}, \frac{3}{\sqrt{10}}\right)$.

Check: $\langle \mathbf{u}_1, \mathbf{u}_2 \rangle = \frac{3}{\sqrt{10}}\left(-\frac{1}{\sqrt{10}}\right) + \frac{1}{\sqrt{10}}\left(\frac{3}{\sqrt{10}}\right) = -\frac{3}{10} + \frac{3}{10} = 0$ ✓ Example 2: Gram-Schmidt in ℝ³

Basis: $\mathbf{v}_1 = (1, 2, 2)$, $\mathbf{v}_2 = (-1, 0, 2)$, $\mathbf{v}_3 = (0, 0, 1)$.

Step 1: $\|\mathbf{v}_1\| = \sqrt{1+4+4} = 3$, $\mathbf{u}_1 = \left(\frac{1}{3}, \frac{2}{3}, \frac{2}{3}\right)$.

Step 2: $\langle \mathbf{v}_2, \mathbf{u}_1 \rangle = \frac{-1+0+4}{3} = \frac{3}{3} = 1$.

$\mathbf{w}_2 = (-1,0,2) - 1\left(\frac{1}{3}, \frac{2}{3}, \frac{2}{3}\right) = \left(-\frac{4}{3}, -\frac{2}{3}, \frac{4}{3}\right)$.

$\|\mathbf{w}_2\| = \sqrt{\frac{16+4+16}{9}} = \sqrt{\frac{36}{9}} = 2$.

$\mathbf{u}_2 = \left(-\frac{2}{3}, -\frac{1}{3}, \frac{2}{3}\right)$.

Step 3: $\langle \mathbf{v}_3, \mathbf{u}_1 \rangle = \frac{0+0+2}{3} = \frac{2}{3}$. $\langle \mathbf{v}_3, \mathbf{u}_2 \rangle = 0 + 0 + \frac{2}{3} = \frac{2}{3}$.

$\mathbf{w}_3 = (0,0,1) - \frac{2}{3}\left(\frac{1}{3}, \frac{2}{3}, \frac{2}{3}\right) - \frac{2}{3}\left(-\frac{2}{3}, -\frac{1}{3}, \frac{2}{3}\right)$

$= (0,0,1) - \left(\frac{2}{9}, \frac{4}{9}, \frac{4}{9}\right) - \left(-\frac{4}{9}, -\frac{2}{9}, \frac{4}{9}\right)$

$= \left(0-\frac{2}{9}+\frac{4}{9}, 0-\frac{4}{9}+\frac{2}{9}, 1-\frac{4}{9}-\frac{4}{9}\right) = \left(\frac{2}{9}, -\frac{2}{9}, \frac{1}{9}\right)$.

$\|\mathbf{w}_3\| = \sqrt{\frac{4+4+1}{81}} = \sqrt{\frac{9}{81}} = \frac{1}{3}$.

$\mathbf{u}_3 = \left(\frac{2}{3}, -\frac{2}{3}, \frac{1}{3}\right)$.

Orthonormal basis: $\left\{\left(\frac{1}{3}, \frac{2}{3}, \frac{2}{3}\right), \left(-\frac{2}{3}, -\frac{1}{3}, \frac{2}{3}\right), \left(\frac{2}{3}, -\frac{2}{3}, \frac{1}{3}\right)\right\}$.

3. QR Decomposition

Definition (QR Decomposition). For an $m \times n$ matrix $A$ with linearly independent columns, the $QR$ decomposition is:

where $Q$ is $m \times n$ with orthonormal columns and $R$ is $n \times n$ upper triangular. How to get $R$: When Gram-Schmidt produces $\mathbf{q}_j$ from $\mathbf{a}_j$, the entries of $R$ are:

• $r_{ii} = \|\mathbf{w}_i\|$ (norm before normalisation)
• $r_{ji} = \langle \mathbf{a}_i, \mathbf{q}_j \rangle$ for $j < i$
• $r_{ji} = 0$ for $j > i$

Example 3: QR of a 2×2 matrix

.

From Example 1: $\mathbf{q}_1 = \left(\frac{3}{\sqrt{10}}, \frac{1}{\sqrt{10}}\right)$, $\mathbf{q}_2 = \left(-\frac{1}{\sqrt{10}}, \frac{3}{\sqrt{10}}\right)$.

So

.

$R$: $r_{11} = \|\mathbf{v}_1\| = \sqrt{10}$. $r_{12} = \langle \mathbf{v}_2, \mathbf{q}_1 \rangle = \frac{8}{\sqrt{10}}$. $r_{22} = \|\mathbf{w}_2\| = \frac{4}{\sqrt{10}}$.

.

Check:

✓

4. Modified Gram-Schmidt (Numerical Stability)

• For each $k$, subtract projections one at a time
• This reduces rounding errors But the classical Gram-Schmidt (described above) is what you'll use on exams.

5. Edge Cases & Gotchas

6. Common Pitfalls

❌ Pitfall 1: Arithmetic errors in subtraction

❌ Pitfall 2: Forgetting to subtract ALL previous projections

❌ Pitfall 3: Normalising too early

7. Formula Summary Table

8. 📝 Practice Questions

Q1: Gram-Schmidt in ℝ²

Orthonormalise $\{(1,2), (3,4)\}$.

Solution: $\mathbf{u}_1 = \frac{(1,2)}{\sqrt{5}} = \left(\frac{1}{\sqrt{5}}, \frac{2}{\sqrt{5}}\right)$.

$\langle (3,4), \mathbf{u}_1 \rangle = \frac{3+8}{\sqrt{5}} = \frac{11}{\sqrt{5}}$. $\mathbf{w}_2 = (3,4) - \frac{11}{\sqrt{5}}\left(\frac{1}{\sqrt{5}}, \frac{2}{\sqrt{5}}\right) = (3,4) - \frac{11}{5}(1,2) = \left(\frac{15-11}{5}, \frac{20-22}{5}\right) = \left(\frac{4}{5}, -\frac{2}{5}\right)$.

$\|\mathbf{w}_2\| = \sqrt{\frac{16+4}{25}} = \sqrt{\frac{20}{25}} = \frac{2}{\sqrt{5}}$. $\mathbf{u}_2 = \frac{(4/5, -2/5)}{2/\sqrt{5}} = \left(\frac{2}{\sqrt{5}}, -\frac{1}{\sqrt{5}}\right)$. Q2: Gram-Schmidt in ℝ³

Orthonormalise $\{(1,0,1), (1,1,0), (0,1,1)\}$.

Solution: $\mathbf{u}_1 = \frac{(1,0,1)}{\sqrt{2}} = \left(\frac{1}{\sqrt{2}}, 0, \frac{1}{\sqrt{2}}\right)$.

$\langle (1,1,0), \mathbf{u}_1 \rangle = \frac{1+0+0}{\sqrt{2}} = \frac{1}{\sqrt{2}}$. $\mathbf{w}_2 = (1,1,0) - \frac{1}{2}(1,0,1) = \left(\frac{1}{2}, 1, -\frac{1}{2}\right)$. $\|\mathbf{w}_2\| = \sqrt{\frac{1}{4}+1+\frac{1}{4}} = \sqrt{\frac{3}{2}} = \frac{\sqrt{6}}{2}$. $\mathbf{u}_2 = \frac{(1/2, 1, -1/2)}{\sqrt{6}/2} = \left(\frac{1}{\sqrt{6}}, \frac{2}{\sqrt{6}}, -\frac{1}{\sqrt{6}}\right)$.

$\langle (0,1,1), \mathbf{u}_1 \rangle = \frac{0+0+1}{\sqrt{2}} = \frac{1}{\sqrt{2}}$. $\langle (0,1,1), \mathbf{u}_2 \rangle = \frac{0+2-1}{\sqrt{6}} = \frac{1}{\sqrt{6}}$.

$\mathbf{w}_3 = (0,1,1) - \frac{1}{2}(1,0,1) - \frac{1}{6}(1,2,-1)$. $= (0,1,1) - \left(\frac{1}{2}, 0, \frac{1}{2}\right) - \left(\frac{1}{6}, \frac{1}{3}, -\frac{1}{6}\right)$. $= \left(0-\frac{1}{2}-\frac{1}{6}, 1-0-\frac{1}{3}, 1-\frac{1}{2}+\frac{1}{6}\right) = \left(-\frac{2}{3}, \frac{2}{3}, \frac{2}{3}\right)$.

$\|\mathbf{w}_3\| = \sqrt{\frac{4+4+4}{9}} = \sqrt{\frac{12}{9}} = \frac{2}{\sqrt{3}}$. $\mathbf{u}_3 = \left(-\frac{1}{\sqrt{3}}, \frac{1}{\sqrt{3}}, \frac{1}{\sqrt{3}}\right)$. Q3: QR decomposition

Find the QR decomposition of

.

Solution: $\mathbf{v}_1 = (1,0,1)$, $\mathbf{u}_1 = \frac{(1,0,1)}{\sqrt{2}}$. $\mathbf{v}_2 = (0,1,1)$, $\langle \mathbf{v}_2, \mathbf{u}_1 \rangle = \frac{0+0+1}{\sqrt{2}} = \frac{1}{\sqrt{2}}$. $\mathbf{w}_2 = (0,1,1) - \frac{1}{2}(1,0,1) = \left(-\frac{1}{2}, 1, \frac{1}{2}\right)$. $\|\mathbf{w}_2\| = \sqrt{\frac{1}{4}+1+\frac{1}{4}} = \sqrt{\frac{3}{2}}$. $\mathbf{u}_2 = \left(-\frac{1}{\sqrt{6}}, \frac{2}{\sqrt{6}}, \frac{1}{\sqrt{6}}\right)$.

,

. Q4: Gram-Schmidt on dependent set

What happens if you apply Gram-Schmidt to $\{(1,2), (2,4)\}$?

Solution: $\mathbf{u}_1 = \frac{(1,2)}{\sqrt{5}}$. $\langle (2,4), \mathbf{u}_1 \rangle = \frac{2+8}{\sqrt{5}} = \frac{10}{\sqrt{5}}$. $\mathbf{w}_2 = (2,4) - \frac{10}{5}(1,2) = (2,4) - (2,4) = (0,0)$. $\|\mathbf{w}_2\| = 0$ — the algorithm fails because the set is dependent. Q5: Gram-Schmidt with different inner product

Orthonormalise $\{1, x\}$ in $P_1$ with $\langle p, q \rangle = \int_0^1 p(x)q(x) dx$.

Solution: $\|1\|^2 = \int_0^1 1 dx = 1$, so $\mathbf{u}_1 = 1$.

$\langle x, 1 \rangle = \int_0^1 x dx = \frac{1}{2}$. $\mathbf{w}_2 = x - \frac{1}{2} \cdot 1 = x - \frac{1}{2}$. $\|x-\frac{1}{2}\|^2 = \int_0^1 (x-\frac{1}{2})^2 dx = \int_0^1 (x^2 - x + \frac{1}{4}) dx = \frac{1}{3} - \frac{1}{2} + \frac{1}{4} = \frac{4-6+3}{12} = \frac{1}{12}$. $\|x-\frac{1}{2}\| = \frac{1}{2\sqrt{3}}$.

$\mathbf{u}_2 = \frac{x - 1/2}{1/(2\sqrt{3})} = 2\sqrt{3}(x - 1/2) = 2\sqrt{3}x - \sqrt{3}$.

Orthonormal basis: $\{1, 2\sqrt{3}x - \sqrt{3}\}$. Q6: QR for solving Ax=b

Using QR, solve $A\mathbf{x} = \mathbf{b}$ where

,

.

Solution: From Example 3, $A = QR$ with

,

.

.

Solve $R\mathbf{x} = Q^T\mathbf{b}$: $\sqrt{10}x_1 + \frac{8}{\sqrt{10}}x_2 = \frac{18}{\sqrt{10}}$ $\frac{4}{\sqrt{10}}x_2 = \frac{4}{\sqrt{10}} \Rightarrow x_2 = 1$ $\sqrt{10}x_1 + \frac{8}{\sqrt{10}} = \frac{18}{\sqrt{10}} \Rightarrow \sqrt{10}x_1 = \frac{10}{\sqrt{10}} \Rightarrow x_1 = 1$. Solution: $\mathbf{x} = (1,1)$. Q7: Orthogonal basis for a plane

Find an orthonormal basis for $W = \{(x,y,z) \mid x + y + z = 0\}$.

Solution: Find any basis. Let $x = 1, y = 0 \Rightarrow z = -1$: $\mathbf{v}_1 = (1,0,-1)$. Let $x = 0, y = 1 \Rightarrow z = -1$: $\mathbf{v}_2 = (0,1,-1)$.

Apply Gram-Schmidt: $\mathbf{u}_1 = \frac{(1,0,-1)}{\sqrt{2}}$.

$\langle (0,1,-1), \mathbf{u}_1 \rangle = \frac{0+0+1}{\sqrt{2}} = \frac{1}{\sqrt{2}}$. $\mathbf{w}_2 = (0,1,-1) - \frac{1}{2}(1,0,-1) = \left(-\frac{1}{2}, 1, -\frac{1}{2}\right)$. $\|\mathbf{w}_2\| = \sqrt{\frac{1}{4}+1+\frac{1}{4}} = \sqrt{\frac{3}{2}}$. $\mathbf{u}_2 = \left(-\frac{1}{\sqrt{6}}, \frac{2}{\sqrt{6}}, -\frac{1}{\sqrt{6}}\right)$. Q8: Projection using QR

Project $\mathbf{b} = (1,2,3)$ onto the column space of $A$ from Q3 using QR.

Solution: From Q3, $\text{proj} = QQ^T\mathbf{b}$.

.

.

Interesting — $\mathbf{b}$ is already in $C(A)$! Check: $A(1,1) = (1,0,1)+(0,1,1) = (1,1,2)$. Wait, that doesn't give (1,2,3). Let me recheck...

Actually $(1,2,3)$ is in $C(A)$:

. Solve $A\mathbf{x} = (1,2,3)$: $x=1$, $y=2$, $1+2=3$ ✓. So $\mathbf{b}$ is indeed in $C(A)$ and projection = itself. Q9: Orthogonal polynomials

Apply Gram-Schmidt to $\{1, x, x^2\}$ on $[-1,1]$ with $\langle f,g \rangle = \int_{-1}^1 f(x)g(x) dx$.

Solution: $\|1\|^2 = \int_{-1}^1 1 dx = 2$, $\mathbf{u}_1 = \frac{1}{\sqrt{2}}$.

$\langle x, 1 \rangle = \int_{-1}^1 x dx = 0$ (odd). $\mathbf{w}_2 = x - 0 = x$. $\|x\|^2 = \int_{-1}^1 x^2 dx = \frac{2}{3}$, $\mathbf{u}_2 = \frac{x}{\sqrt{2/3}} = \sqrt{\frac{3}{2}}x$.

$\langle x^2, 1 \rangle = \int_{-1}^1 x^2 dx = \frac{2}{3}$. $\langle x^2, x \rangle = \int_{-1}^1 x^3 dx = 0$. $\mathbf{w}_3 = x^2 - \frac{2/3}{2} \cdot 1 - 0 = x^2 - \frac{1}{3}$. $\|x^2 - \frac{1}{3}\|^2 = \int_{-1}^1 (x^4 - \frac{2}{3}x^2 + \frac{1}{9}) dx = \frac{2}{5} - \frac{4}{9} + \frac{2}{9} = \frac{2}{5} - \frac{2}{9} = \frac{18-10}{45} = \frac{8}{45}$. $\mathbf{u}_3 = \frac{x^2 - 1/3}{\sqrt{8/45}} = \sqrt{\frac{45}{8}}(x^2 - \frac{1}{3})$.

These are (up to scaling) the Legendre polynomials: $P_0=1$, $P_1=x$, $P_2=\frac{1}{2}(3x^2-1)$. Q10: QR and least squares

Why is QR useful for least squares?

Solution: The least squares solution to $A\mathbf{x} \approx \mathbf{b}$ (when $A$ is full column rank) solves $A^T A \mathbf{x} = A^T \mathbf{b}$. With $A = QR$ where $Q$ has orthonormal columns: $A^T A = R^T Q^T Q R = R^T R$ (since $Q^T Q = I$). $A^T \mathbf{b} = R^T Q^T \mathbf{b}$. So $R^T R \mathbf{x} = R^T Q^T \mathbf{b} \Rightarrow R\mathbf{x} = Q^T\mathbf{b}$. This triangular system is easy to solve by back substitution — no need to form $A^T A$!

🔗 Cross-References

• Next topic: Orthogonal Transformations
• Week 12 (Eigenvalues): QR algorithm for computing eigenvalues
• BSCS2004 (ML Foundations): QR for linear regression, numerical stability Join Discord Previous8.1 OrthogonalityNext8.3 Orthogonal Transformations