Gaussian Elimination: Solving Linear Systems

🎯 Learning Objectives

• Identify row echelon form (REF) and reduced row echelon form (RREF)
• Perform the three elementary row operations
• Carry out Gaussian elimination to transform any matrix to REF
• Carry out Gauss-Jordan elimination to transform any matrix to RREF
• Determine whether a linear system has zero, one, or infinitely many solutions
• Express solution sets in parametric form using free variables
• Compute the rank of a matrix from its RREF

📋 Prerequisites

• Matrices Introduction (Week 1) — matrix notation, matrix-vector multiplication
• Determinants (Week 1) — testing invertibility
• Vectors Introduction (Week 1) — column vectors and linear combinations
• Gaussian elimination is the master algorithm of linear algebra — it underlies finding bases, ranks, null spaces, and inverses

1. The Problem: Solving $A\mathbf{x} = \mathbf{b}$

1.1 Intuition: From Messy to Clean

2. Row Echelon Form (REF)

2.1 Definition

A matrix is in row echelon form if:

1. All non-zero rows are above any zero rows
2. The leading (first non-zero) entry of each row is to the right of the leading entry in the row above
3. All entries below a leading entry are zero The leading entry (or pivot) is the first non-zero element in a row. In REF, pivots are typically normalised to 1 for convenience (though the definition doesn't require it).

2.2 Examples

2.3 Reduced Row Echelon Form (RREF)

A matrix is in RREF if it is in REF and additionally:

1. The leading entry in each non-zero row is $1$
2. Each leading $1$ is the only non-zero entry in its column ✓ RREF (correct):

3. Elementary Row Operations

Key insight: Adding a multiple of one row to another preserves the determinant. Swapping flips its sign. Scaling scales the determinant by $c$.

4. Gaussian Elimination Algorithm

4.1 Forward Elimination (to REF)

1. Start with the leftmost non-zero column (pivot column)
2. Find a pivot: select a non-zero entry in this column. If none exists, move to the next column
3. Swap (if needed): bring the pivot row to the top
4. Scale (optional): make the pivot $1$
5. Eliminate below: use row operations to create zeros below the pivot
6. Repeat: ignore the pivot row and repeat steps 1-5 on the remaining submatrix

Example 1: Gaussian elimination (3 equations, 3 unknowns)

Solve:

Augmented matrix:

Step 1: Already have pivot $(1,1) = 1$.

Step 2: Eliminate column 1 below pivot:

Step 3: Next pivot is in column 2. Row 2 has a 0 in column 2, so swap $R_2 \leftrightarrow R_3$:

Step 4: Scale pivots (optional but helpful):

Step 5: Back substitution. From row 3: $z = \frac{17}{3}$.

From row 2: $y + \frac{5}{4}z = \frac{25}{4} \Rightarrow y = \frac{25}{4} - \frac{5}{4}\cdot\frac{17}{3} = \frac{75 - 85}{12} = -\frac{10}{12} = -\frac{5}{6}$

From row 1: $x + 2y + z = 8 \Rightarrow x = 8 - 2(-\frac{5}{6}) - \frac{17}{3} = 8 + \frac{5}{3} - \frac{17}{3} = \frac{24}{3} + \frac{5}{3} - \frac{17}{3} = \frac{12}{3} = 4$

Solution: $x = 4$, $y = -\frac{5}{6}$, $z = \frac{17}{3}$.

4.2 Gauss-Jordan Elimination (to RREF)

1. Perform forward elimination (to REF)
2. Scale each row so pivots are $1$
3. Eliminate upward: for each pivot, use row operations to create zeros above it

Example 2: Gauss-Jordan (same system)

Start from REF:

Step 1: Eliminate above pivot in column 3:

Step 2: Eliminate above pivot in column 2:

Solution is now read directly from the RREF: $x=4, y=-\frac{5}{6}, z=\frac{17}{3}$.

5. Three Cases for Solution Sets

5.1 Case 1: Unique Solution (already seen in Example 1)

5.2 Case 2: No Solution (Inconsistent)

Example 3: Inconsistent system

Augmented matrix and row reduction:

Row 2 says $0x + 0y = 1$, which is impossible. No solution.

5.3 Case 3: Infinitely Many Solutions

Example 4: Infinitely many solutions

Augmented matrix:

RREF:

Columns 1 and 3 have pivots? Actually column 2 has no pivot (free variable). Column 1 has pivot. Column 3 has no pivot.

Wait — let me re-check. We have pivots in column 1 only. So columns 2 and 3 are free.

Solution: $x = 1 - 2y + z$, where $y, z$ are free variables (can be any real number).

In parametric vector form:

6. Rank of a Matrix

Definition (Rank). The rank of a matrix $A$ is the number of non-zero rows in its RREF (equivalently, the number of pivots).

• $\text{rank}(A) \leq \min(m, n)$ for $A_{m \times n}$
• $\text{rank}(A) = \text{rank}(A^T)$
• $\text{rank}(A) = \dim(\text{column space of } A)$
• $\text{rank}(AB) \leq \min(\text{rank}(A), \text{rank}(B))$

Example 5: Finding rank

Rank = 1 (only one pivot).

7. Homogeneous Systems ($A\mathbf{x} = \mathbf{0}$)

Definition (Homogeneous). A system $A\mathbf{x} = \mathbf{0}$ (all right-hand sides are zero) is homogeneous. Key facts:

• Always has at least the trivial solution $\mathbf{x} = \mathbf{0}$
• Has non-trivial solutions iff there is at least one free variable (i.e., rank < number of columns)
• The solution set is a vector space (the null space of $A$)

Example 6: Homogeneous system

Pivots in columns 1 and 3. Column 2 is free.

$x = -2y$, $z = 0$, $y$ free.

Solution:

.

8. Algorithm Summary

9. Edge Cases & Gotchas

10. Common Pitfalls

❌ Pitfall 1: Arithmetic errors in elimination

❌ Pitfall 2: Forgetting to apply operations to the RHS

❌ Pitfall 3: Misidentifying free variables after Gauss-Jordan

❌ Pitfall 4: Thinking REF requires pivots to be 1

11. Formula Summary Table

12. 📝 Practice Questions

Q1: Simple elimination (unique solution)

Solve using Gaussian elimination:

Strategy: Augmented matrix → REF → back substitute.

Solution:

From $R_2$: $-3y = -6 \Rightarrow y = 2$. From $R_1$: $x + 2 = 3 \Rightarrow x = 1$.

Solution: $x = 1$, $y = 2$. Q2: Inconsistent system

Solve:

Strategy: Check for inconsistency.

Solution:

Row 2: $0x + 0y = -1$ — impossible. No solution. Q3: Infinitely many solutions

Solve:

Solution:

Pivot in column 1 (basic variable $x$). $y, z$ are free.

$x = 1 - y + 2z$, $y, z \in \mathbb{R}$.

Parametric form:

. Q4: Gauss-Jordan elimination

Solve using Gauss-Jordan:

Solution:

$R_2 \to R_2 - 2R_1$, $R_3 \to R_3 - R_1$:

$R_2 \to -\frac{1}{5}R_2$:

$R_3 \to R_3 + 3R_2$:

$R_3 \to \frac{5}{3}R_3$:

Eliminate above: $R_1 \to R_1 - R_3$, $R_2 \to R_2 - \frac{1}{5}R_3$:

$R_1 \to R_1 - 2R_2$:

Solution: $x = 2$, $y = -\frac{1}{3}$, $z = -\frac{4}{3}$. Q5: Homogeneous system with free variables

Find all solutions to:

Solution:

$R_3 \to R_3 - R_2$:

$R_2 \to -R_2$, $R_1 \to R_1 - 2R_2$:

Pivots in columns 1, 2. $z$ is free. $x = -5z$, $y = 3z$.

Solution:

, $z \in \mathbb{R}$. Q6: Rank from RREF

Find the rank of

.

Solution:

Swap $R_2 \leftrightarrow R_3$:

Rank = 2 (two pivots). Q7: Parameterising solution set

Solve and parameterise:

Solution:

Pivot in column 1. $x_1$ basic; $x_2, x_3, x_4$ free.

$x_1 = 1 - 2x_2 - x_3 + x_4$, $x_2, x_3, x_4 \in \mathbb{R}$.

Parametric:

. Q8: System with parameters

For what value(s) of $k$ does the system have (a) unique solution, (b) no solution, (c) infinitely many solutions?

Solution:

• If $k \neq 2$, unique solution ($y = \frac{1}{k-2}$, $x = 1 - \frac{1}{k-2}$).
• If $k = 2$, row 2 says $0x + 0y = 1$ — no solution.
• Infinitely many solutions are impossible here (2 equations, 2 unknowns, RHS not all zero). Q9: Rank of transpose

If $A$ is $3 \times 4$ with rank 2, what is the rank of $A^T$?

Solution: $\text{rank}(A^T) = \text{rank}(A) = 2$. Q10: Application — network flow

A network has nodes A, B, C with flow equations:

Does a unique flow exist?

Solution:

Row 3 says $0 = 10$ — impossible. No solution exists (the equations are inconsistent).

Wait — let me recheck. The sum of equations 1 and 2 minus equation 3: $(x_1+x_2)+(x_2-x_3)-(x_1+x_3) = 10+5-15 \Rightarrow 2x_2-2x_3 = 0 \Rightarrow x_2 = x_3$. Then eq 2: $x_2-x_3 = 0 \neq 5$. Indeed inconsistent.

🔗 Cross-References

• Next topic: Cramer's Rule — solving systems using determinants
• Week 3 (Vector Spaces): The solution set of $A\mathbf{x} = \mathbf{0}$ is a vector space
• Week 4 (Basis & Dimension): Gaussian elimination finds bases for vector spaces
• Week 5 (Rank-Nullity): Rank from elimination feeds into the rank-nullity theorem
• BSCS2004 (ML Foundations): Solving normal equations in linear regression uses Gaussian elimination Join Discord Previous1.3 DeterminantsNext2.2 Cramer's Rule & Inverses