Inner Products & Norms

🎯 Learning Objectives

• Define an inner product and a norm axiomatically
• Compute the inner product for vectors in $\mathbb{R}^n$, polynomials, and matrices
• Use the Cauchy-Schwarz inequality to bound inner products
• Prove the triangle inequality from Cauchy-Schwarz
• Compute distances and angles in arbitrary inner product spaces

📋 Prerequisites

• Vector Spaces (Week 3) — the abstract setting
• Dot Product (Week 1) — the motivating example
• Inner products add geometry (lengths, angles) to vector spaces

1. Intuition: Adding Geometry to Vector Spaces

• Define the length (norm) of a vector
• Define the distance between two vectors
• Define the angle between two vectors
• Talk about orthogonality (Diagram)

2. Definition of an Inner Product

Definition (Inner Product). An inner product on a real vector space $V$ is a function $\langle \cdot, \cdot \rangle: V \times V \to \mathbb{R}$ satisfying:

1. Linearity in first argument: $\langle a\mathbf{u} + b\mathbf{v}, \mathbf{w} \rangle = a\langle \mathbf{u}, \mathbf{w} \rangle + b\langle \mathbf{v}, \mathbf{w} \rangle$
2. Symmetry: $\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle$
3. Positive definiteness: $\langle \mathbf{v}, \mathbf{v} \rangle \geq 0$, and $\langle \mathbf{v}, \mathbf{v} \rangle = 0$ iff $\mathbf{v} = \mathbf{0}$ A vector space equipped with an inner product is called an inner product space.

2.1 Examples

Example 1: Standard dot product on ℝⁿ

Example 2: Weighted inner product on ℝ²

Check: Positive definite? $\langle (x_1, x_2), (x_1, x_2) \rangle = 2x_1^2 + 3x_2^2 \geq 0$ and $=0$ only when $x_1=x_2=0$. ✓ Example 3: Inner product on polynomials

For $p, q \in P_n$: $\langle p, q \rangle = \int_{-1}^1 p(x) q(x) \, dx$

Check: $\langle p, p \rangle = \int_{-1}^1 p(x)^2 dx \geq 0$, and $=0$ only if $p=0$. ✓ Example 4: Frobenius inner product on matrices

For $A, B \in \mathbb{R}^{m \times n}$: $\langle A, B \rangle = \text{tr}(A^T B) = \sum_{i,j} a_{ij} b_{ij}$

3. Norm

3.1 Definition

Definition (Norm). A norm on a vector space $V$ is a function $\|\cdot\|: V \to \mathbb{R}$ satisfying:

1. Positive definiteness: $\|\mathbf{v}\| \geq 0$, and $\|\mathbf{v}\| = 0$ iff $\mathbf{v} = \mathbf{0}$
2. Absolute homogeneity: $\|c\mathbf{v}\| = |c| \|\mathbf{v}\|$ for all $c \in \mathbb{R}$
3. Triangle inequality: $\|\mathbf{u} + \mathbf{v}\| \leq \|\mathbf{u}\| + \|\mathbf{v}\|$

3.2 The Norm Induced by an Inner Product

Example 5: Norms from inner products

• ℝⁿ (standard): $\|\mathbf{x}\| = \sqrt{x_1^2 + \cdots + x_n^2}$ (Euclidean norm)
• Weighted ℝ²: $\|(x_1, x_2)\| = \sqrt{2x_1^2 + 3x_2^2}$
• Polynomials: $\|p\| = \sqrt{\int_{-1}^1 p(x)^2 dx}$
• Frobenius norm: $\|A\|_F = \sqrt{\text{tr}(A^T A)} = \sqrt{\sum a_{ij}^2}$

4. Cauchy-Schwarz Inequality

Theorem (Cauchy-Schwarz). For any $\mathbf{u}, \mathbf{v}$ in an inner product space:

Equality holds iff $\mathbf{u}$ and $\mathbf{v}$ are linearly dependent (one is a scalar multiple of the other). Proof

Consider $\|\mathbf{u} - t\mathbf{v}\|^2 \geq 0$ for any $t \in \mathbb{R}$:

This is a quadratic in $t$ that is always $\geq 0$, so its discriminant must be $\leq 0$:

Dividing by 4: $\langle \mathbf{u}, \mathbf{v} \rangle^2 \leq \|\mathbf{u}\|^2 \|\mathbf{v}\|^2$. Taking square roots gives the result. ∎ Example 6: Using Cauchy-Schwarz

For $\mathbf{u} = (1, 2, 3)$, $\mathbf{v} = (4, 5, 6)$: $\langle \mathbf{u}, \mathbf{v} \rangle = 4 + 10 + 18 = 32$ $\|\mathbf{u}\| = \sqrt{14} \approx 3.74$, $\|\mathbf{v}\| = \sqrt{77} \approx 8.77$ $\|\mathbf{u}\|\|\mathbf{v}\| \approx 32.83$ $|\langle \mathbf{u}, \mathbf{v} \rangle| = 32 \leq 32.83$ ✓

5. Triangle Inequality

Theorem (Triangle Inequality). For any $\mathbf{u}, \mathbf{v}$ in an inner product space:

6. Angle and Distance

6.1 Angle

6.2 Distance

Example 7: Angle between polynomials

$p(x) = 1$, $q(x) = x$ with inner product $\langle p, q \rangle = \int_{-1}^1 p(x) q(x) dx$.

$\langle p, q \rangle = \int_{-1}^1 1 \cdot x dx = 0$ (odd function over symmetric interval). So $\cos\theta = 0$, $\theta = 90^\circ$ — the polynomials 1 and $x$ are orthogonal!

7. Edge Cases & Gotchas

8. Common Pitfalls

❌ Pitfall 1: Assuming all norms come from inner products

❌ Pitfall 2: Forgetting absolute value in Cauchy-Schwarz

❌ Pitfall 3: Confusing inner product spaces with normed spaces

9. Formula Summary Table

10. 📝 Practice Questions

Q1: Weighted inner product

For $\langle (x_1,x_2), (y_1,y_2) \rangle = 2x_1y_1 - x_1y_2 - x_2y_1 + 3x_2y_2$, compute $\langle (1,2), (3,4) \rangle$.

Solution: $2(1)(3) - 1(4) - 2(3) + 3(2)(4) = 6 - 4 - 6 + 24 = 20$. Q2: Checking inner product axioms

Is $\langle (x_1,x_2), (y_1,y_2) \rangle = x_1y_1 + x_2y_2 + 1$ an inner product?

Solution: Fails linearity (the constant 1 breaks it). Also, $\langle (0,0), (0,0) \rangle = 1 \neq 0$, so positive definiteness fails. Q3: Norm from inner product

Compute $\|(3,4)\|$ using the standard inner product and the weighted inner product from Example 2.

Solution: Standard: $\sqrt{9+16} = 5$. Weighted ($2x_1^2 + 3x_2^2$): $\sqrt{2(9) + 3(16)} = \sqrt{18+48} = \sqrt{66} \approx 8.12$. Q4: Cauchy-Schwarz verification

Verify Cauchy-Schwarz for $p(x) = x$, $q(x) = x^2$ with $\langle p,q \rangle = \int_0^1 p(x)q(x) dx$.

Solution: $\langle p,q \rangle = \int_0^1 x^3 dx = \frac{1}{4}$. $\|p\|^2 = \int_0^1 x^2 dx = \frac{1}{3}$, so $\|p\| = \frac{1}{\sqrt{3}}$. $\|q\|^2 = \int_0^1 x^4 dx = \frac{1}{5}$, so $\|q\| = \frac{1}{\sqrt{5}}$. $\|p\|\|q\| = \frac{1}{\sqrt{15}} \approx 0.258$. $|\langle p,q \rangle| = 0.25 \leq 0.258$ ✓ Q5: Triangle inequality

Verify the triangle inequality for $\mathbf{u} = (1,2)$, $\mathbf{v} = (3,4)$.

Solution: $\mathbf{u}+\mathbf{v} = (4,6)$, $\|(4,6)\| = \sqrt{16+36} = \sqrt{52} \approx 7.21$. $\|\mathbf{u}\| = \sqrt{5} \approx 2.24$, $\|\mathbf{v}\| = 5$. $2.24 + 5 = 7.24 > 7.21$ ✓ Q6: Angle in weighted inner product

Find the angle between $(1,0)$ and $(0,1)$ using the inner product $\langle (x_1,x_2), (y_1,y_2) \rangle = x_1y_1 + 2x_2y_2$.

Solution: $\langle (1,0), (0,1) \rangle = 0$. So $\cos\theta = 0$, $\theta = 90^\circ$. They are orthogonal. Q7: Distance between functions

Find the distance between $f(x)=1$ and $g(x)=x$ on $[0,1]$ using $\langle f,g \rangle = \int_0^1 fg$.

Solution: $f-g = 1-x$. $\|f-g\|^2 = \int_0^1 (1-x)^2 dx = \int_0^1 (1 - 2x + x^2) dx = [x - x^2 + x^3/3]_0^1 = 1 - 1 + 1/3 = 1/3$. Distance = $\frac{1}{\sqrt{3}}$. Q8: Parallelogram law

Verify the parallelogram law for $\mathbf{u} = (1,2)$, $\mathbf{v} = (3,4)$.

Solution: $\mathbf{u}+\mathbf{v} = (4,6)$, $\|\mathbf{u}+\mathbf{v}\|^2 = 52$. $\mathbf{u}-\mathbf{v} = (-2,-2)$, $\|\mathbf{u}-\mathbf{v}\|^2 = 8$. LHS: $52 + 8 = 60$. RHS: $2\|(1,2)\|^2 + 2\|(3,4)\|^2 = 2(5) + 2(25) = 10 + 50 = 60$ ✓ Q9: Orthogonal vectors in weighted inner product

Find $k$ such that $(1,2)$ and $(3,k)$ are orthogonal in the inner product $\langle (x_1,x_2), (y_1,y_2) \rangle = x_1y_1 + 2x_2y_2$.

Solution: $\langle (1,2), (3,k) \rangle = 1(3) + 2(2)(k) = 3 + 4k = 0 \Rightarrow k = -\frac{3}{4}$. Q10: Frobenius inner product

Compute $\langle A, B \rangle$ where

,

using $\langle A, B \rangle = \text{tr}(A^T B)$.

Solution:

,

. $\text{tr}(A^T B) = -3 + 2 = -1$. Check entry-wise: $\sum a_{ij}b_{ij} = 1(0) + 2(1) + 3(-1) + 4(0) = 0 + 2 - 3 + 0 = -1$ ✓

🔗 Cross-References

• Next topic: Orthogonality
• Week 8 (Gram-Schmidt): Uses inner products to build orthonormal bases
• Week 12 (Eigenvalues): Symmetric matrices have orthogonal eigenvectors
• BSCS2004 (ML Foundations): Kernel methods use inner products in feature spaces
• BSMA1001 (Maths 1): Dot product review Join Discord Previous7.2 Affine SubspacesNext8.1 Orthogonality