Linear Independence & Span

🎯 Learning Objectives

• Define the span of a set of vectors
• Determine whether a set of vectors is linearly independent or dependent
• Find whether a given vector lies in the span of a set of vectors
• Use Gaussian elimination to test linear independence
• Describe the geometric meaning of dependence/independence in $\mathbb{R}^2$ and $\mathbb{R}^3$
• Prove basic facts about independence

📋 Prerequisites

• Vector Spaces and Subspaces (this week) — context for where vectors live
• Gaussian Elimination (Week 2) — the computational tool for testing independence
• Solving homogeneous systems (Week 2) — $A\mathbf{x} = \mathbf{0}$ is the key equation

1. Intuition: Redundant vs. Essential Vectors

1.1 What Does "Independent" Mean?

• A shirt
• Pants
• Shoes
• A jacket These are all independent — none can be made from the others. Now imagine you also pack:
• A shirt that's exactly like the first one (redundant)
• An outfit that's just shirt + pants (composite) These are dependent — they add nothing new to your wardrobe. Linear independence captures this same idea for vectors: a set of vectors is independent if no vector can be expressed as a combination of the others. Every vector contributes a new "direction." (Diagram)

1.2 Span: The Set of All Combinations

• If you have one vector in $\mathbb{R}^2$, its span is a line through the origin
• If you have two linearly independent vectors in $\mathbb{R}^2$, their span is the entire plane
• If you have three or more vectors in $\mathbb{R}^2$, their span is still just the plane (the extra vectors are redundant)

2. Span

Definition (Span). The span of a set of vectors $\{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_k\}$ in a vector space $V$ is:

It is the set of all linear combinations of the vectors. Key fact: The span of any set of vectors is always a subspace of $V$. Example 1: Span of two vectors in ℝ³

$\mathbf{v}_1 = (1, 0, 1)$, $\mathbf{v}_2 = (0, 1, 1)$

This is the plane $x - y + z = 0$ (check: $x = c_1$, $y = c_2$, $z = c_1 + c_2$, so $x - y + z = c_1 - c_2 + c_1 + c_2 = 2c_1$, wait... let me recheck). Actually: $\text{span}\{\mathbf{v}_1, \mathbf{v}_2\} = \{(c_1, c_2, c_1 + c_2) \mid c_1, c_2 \in \mathbb{R}\}$. This is the plane through origin with equation $x + y - z = 0$ (since $x + y - z = c_1 + c_2 - (c_1 + c_2) = 0$).

3. Linear Independence

3.1 Definition

Definition (Linear Independence). A set of vectors $\{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_k\}$ is linearly independent if the only solution to:

is the trivial solution $c_1 = c_2 = \cdots = c_k = 0$.

If there exists a non-trivial solution (at least one $c_i \neq 0$), the set is linearly dependent. Intuition: A set is dependent if some vector can be written as a combination of the others. In that case, we have:

3.2 Geometric Interpretation

3.3 Testing Independence with Gaussian Elimination

1. Form matrix $A$ with $\mathbf{v}_i$ as columns
2. Solve $A\mathbf{c} = \mathbf{0}$ (row reduce to RREF)
3. If the only solution is $\mathbf{c} = \mathbf{0}$, the vectors are independent
4. If there are free variables, they are dependent Alternatively: form a matrix with vectors as rows and compute the rank. The vectors are independent iff the number of vectors equals the rank.

Example 2: Independent vectors in ℝ³

Test $\mathbf{v}_1 = (1, 2, 1)$, $\mathbf{v}_2 = (0, 1, 2)$, $\mathbf{v}_3 = (1, 0, 1)$.

Row reduce $A\mathbf{c} = \mathbf{0}$:

All columns are pivot columns, so only trivial solution. Independent. Example 3: Dependent vectors in ℝ³

Test $\mathbf{v}_1 = (1, 2, 3)$, $\mathbf{v}_2 = (4, 5, 6)$, $\mathbf{v}_3 = (7, 8, 9)$.

Row reduce:

Column 3 has no pivot (free variable). Dependent. Indeed, $\mathbf{v}_3 = 2\mathbf{v}_2 - \mathbf{v}_1$ (check: $7 = 2(4)-1$, $8 = 2(5)-2$, $9 = 2(6)-3$).

3.4 Important Facts

4. Wronskian Test for Functions

Example 4: Wronskian of $\{1, x, x^2\}$

So $\{1, x, x^2\}$ is independent. (This makes sense — no polynomial of degree 2 is a combination of lower-degree polynomials.)

5. Edge Cases & Gotchas

6. Common Pitfalls

❌ Pitfall 1: Thinking "independent" means "orthogonal"

❌ Pitfall 2: Testing independence by looking at rows vs. columns

❌ Pitfall 3: Assuming more vectors than the dimension are independent

❌ Pitfall 4: Forgetting the zero vector case

7. Formula Summary Table

8. 📝 Practice Questions

Q1: Span description

Describe $\text{span}\{(1,2), (2,4)\}$.

Solution: The second vector is $2$ times the first, so they are collinear. $\text{span}\{(1,2), (2,4)\} = \text{span}\{(1,2)\} = \{(c, 2c) \mid c \in \mathbb{R}\}$, a line through origin. Q2: Independence test in ℝ²

Are $\mathbf{v}_1 = (1,3)$ and $\mathbf{v}_2 = (2,-1)$ linearly independent?

Solution: Check if one is a scalar multiple of the other. If $\mathbf{v}_2 = c\mathbf{v}_1$, then $2 = c(1)$ and $-1 = c(3)$ ⇒ $c=2$ and $c=-1/3$, impossible. They are independent. Q3: Independence test in ℝ³

Are $\mathbf{v}_1 = (1,0,1)$, $\mathbf{v}_2 = (0,1,2)$, $\mathbf{v}_3 = (1,1,1)$ independent?

Solution: Solve $c_1(1,0,1) + c_2(0,1,2) + c_3(1,1,1) = (0,0,0)$:

From first two: $c_1 = -c_3$, $c_2 = -c_3$. Substitute into third: $(-c_3) + 2(-c_3) + c_3 = -2c_3 = 0 \Rightarrow c_3 = 0$, then $c_1 = 0$, $c_2 = 0$.

Only trivial solution. Independent. Q4: Finding a dependency relation

Find a dependency relation among $\mathbf{v}_1 = (1,2,1)$, $\mathbf{v}_2 = (2,3,4)$, $\mathbf{v}_3 = (3,5,5)$.

Solution: Solve $A\mathbf{c} = \mathbf{0}$:

$c_2 = -c_3$, $c_1 = -2c_2 - 3c_3 = 2c_3 - 3c_3 = -c_3$. Choose $c_3 = 1$: $c_1 = -1$, $c_2 = -1$, $c_3 = 1$.

Dependency: $-\mathbf{v}_1 - \mathbf{v}_2 + \mathbf{v}_3 = \mathbf{0}$, i.e., $\mathbf{v}_3 = \mathbf{v}_1 + \mathbf{v}_2$.

Check: $(3,5,5) = (1,2,1) + (2,3,4) = (3,5,5)$ ✓ Q5: Zero vector set

Is $\{(1,2), (0,0), (3,4)\}$ linearly independent?

Solution: No, because it contains the zero vector. A non-trivial combination is $0(1,2) + 1(0,0) + 0(3,4) = (0,0)$. Q6: Independence of polynomials

Are $p_1(x) = 1 + x$, $p_2(x) = x + x^2$, $p_3(x) = 1 - x^2$ linearly independent in $P_2$?

Solution: Set $c_1(1+x) + c_2(x+x^2) + c_3(1 - x^2) = 0$ (the zero polynomial).

Group like powers: $(c_1 + c_3) + (c_1 + c_2)x + (c_2 - c_3)x^2 = 0$.

For this to be the zero polynomial, all coefficients must be zero:

From first: $c_3 = -c_1$. From third: $c_2 = c_3 = -c_1$. From second: $c_1 + (-c_1) = 0$ (satisfied automatically).

So $c_1$ is free; pick $c_1 = 1$, then $c_2 = -1$, $c_3 = -1$.

$p_1 - p_2 - p_3 = (1+x) - (x+x^2) - (1-x^2) = 1+x-x-x^2-1+x^2 = 0$.

Dependent. Q7: Maximum independent set

What is the maximum number of linearly independent vectors in $\mathbb{R}^4$?

Solution: At most 4 (the dimension of $\mathbb{R}^4$). Any 5 vectors in $\mathbb{R}^4$ are linearly dependent. Q8: Wronskian test

Are $f(x) = \sin x$ and $g(x) = \cos x$ linearly independent?

Solution: Compute the Wronskian:

Independent. Q9: Span and independence

If $\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$ is linearly independent in $\mathbb{R}^3$, what is $\text{span}\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$?

Solution: Since $\mathbb{R}^3$ has dimension 3 and we have 3 independent vectors, they span the entire space: $\text{span}\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\} = \mathbb{R}^3$. Q10: Proving a property

Prove: If $\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$ is independent, then $\{\mathbf{v}_1, \mathbf{v}_1+\mathbf{v}_2, \mathbf{v}_1+\mathbf{v}_2+\mathbf{v}_3\}$ is also independent.

Proof: Suppose $c_1\mathbf{v}_1 + c_2(\mathbf{v}_1+\mathbf{v}_2) + c_3(\mathbf{v}_1+\mathbf{v}_2+\mathbf{v}_3) = \mathbf{0}$. Group terms: $(c_1 + c_2 + c_3)\mathbf{v}_1 + (c_2 + c_3)\mathbf{v}_2 + c_3\mathbf{v}_3 = \mathbf{0}$.

Since $\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$ is independent:

Back-substitute: $c_3 = 0 \Rightarrow c_2 = 0 \Rightarrow c_1 = 0$. Only trivial solution, so the new set is independent. ∎

🔗 Cross-References

• Next topic: Basis & Dimension
• Week 4: A basis is a spanning set that is linearly independent
• Week 1 (Vectors): Dot products provide another way to test independence (orthogonal ⇒ independent)
• Week 8 (Gram-Schmidt): Builds orthogonal/orthonormal independent sets
• BSCS2004 (ML Foundations): Independent features are non-redundant predictors Join Discord Previous3.2 SubspacesNext4.1 Basis & Dimension