03. Complexity Analysis — Recurrence Relations & Master Theorem

What problem does this solve? While big-O gives us a way to express complexity, we need systematic methods to derive the complexity of recursive algorithms. The Master Theorem is a recipe that instantly gives us the complexity of any divide-and-conquer recurrence.

1. Recurrence Relations — The Language of Recursive Algorithms

Mental Model

• (a) = number of subproblems
• (n/b) = size of each subproblem
• (f(n)) = cost of dividing and combining

Step-by-Step: Writing Recurrences

python
Run Inline
# runnable
def binary_search(arr, target, left, right):
    if left > right:           # Base case: T(0) = 1
        return -1
    mid = (left + right) // 2  # O(1) work
    if arr[mid] == target:     # Check found
        return mid
    elif arr[mid] > target:
        return binary_search(arr, target, left, mid - 1)  # 1 subproblem, size n/2
    else:
        return binary_search(arr, target, mid + 1, right)
# Recurrence: T(n) = T(n/2) + 1

python
Run Inline
# runnable
def merge_sort(arr):
    if len(arr) <= 1:         # Base case: T(1) = 1
        return arr
    mid = len(arr) // 2
    left = merge_sort(arr[:mid])   # T(n/2)
    right = merge_sort(arr[mid:])  # T(n/2)
    return merge(left, right)      # O(n) — merging
# Recurrence: T(n) = 2T(n/2) + n

python
Run Inline
# runnable
def hanoi(n, source, target, aux):
    if n == 1:                     # T(1) = 1
        print(f"Move disk 1: {source} -> {target}")
        return
    hanoi(n - 1, source, aux, target)  # T(n-1)
    print(f"Move disk {n}: {source} -> {target}")  # 1 move
    hanoi(n - 1, aux, target, source)  # T(n-1)
# Recurrence: T(n) = 2T(n-1) + 1

python
Run Inline
# runnable
def quicksort(arr, low, high):
    if low >= high:              # Base case
        return
    pivot = partition(arr, low, high)  # O(n) partition
    if pivot == low:                    # Worst case: pivot at one extreme
        quicksort(arr, low + 1, high)   # T(n-1)
    else:
        quicksort(arr, low, pivot - 1)  # T(n/2)
        quicksort(arr, pivot + 1, high) # T(n/2)
# Best case: T(n) = 2T(n/2) + n → O(n log n)
# Worst case: T(n) = T(n-1) + n → O(n²)

2. Solving Recurrences by Unwinding (Substitution)

Method

1. Substitute the recurrence into itself repeatedly
2. Look for the pattern
3. Stop when you reach the base case
4. Simplify

Example: Binary Search (T(n) = T(n/2) + 1)

pseudo
Step 1:  T(n) = T(n/2) + 1
Step 2:  T(n) = [T(n/4) + 1] + 1 = T(n/4) + 2
Step 3:  T(n) = [T(n/8) + 1] + 2 = T(n/8) + 3
Step 4:  T(n) = T(n/2^k) + k

Example: Merge Sort (T(n) = 2T(n/2) + n)

pseudo
Step 1:  T(n) = 2T(n/2) + n
Step 2:  T(n) = 2[2T(n/4) + n/2] + n = 4T(n/4) + 2n
Step 3:  T(n) = 4[2T(n/8) + n/4] + 2n = 8T(n/8) + 3n
Step 4:  T(n) = 2^k T(n/2^k) + kn

Example: Towers of Hanoi (T(n) = 2T(n-1) + 1)

pseudo
Step 1:  T(n) = 2T(n-1) + 1
Step 2:  T(n) = 2[2T(n-2) + 1] + 1 = 2^2 T(n-2) + 2 + 1
Step 3:  T(n) = 2^2[2T(n-3) + 1] + 3 = 2^3 T(n-3) + 4 + 2 + 1
Step 4:  T(n) = 2^k T(n-k) + (2^k - 1)

3. The Master Theorem

What It Is

The Three Cases

Step-by-Step Examples

4. Recursion Tree Method

Mental Model

Three Patterns of Recursion Trees

5. Quick Reference: Common Recurrences and Their Solutions

Practice Questions

Answers

A1. a=9, b=3, (\log_b a = \log_3 9 = 2), (n^{\log_b a} = n^2). f(n) = n = (O(n^{2-\varepsilon})) for (\varepsilon = 1). Case 1: (T(n) = \Theta(n^2)).

A2. a=1, b=3/2, (\log_b a = \log_{1.5} 1 = 0), (n^0 = 1). f(n) = 1 = (\Theta(1) = \Theta(n^0)). Case 2 (k=0): (T(n) = \Theta(\log n)).

A3. Root: (n^2). Level 1: (2 \cdot (n/2)^2 = n^2/2). Level 2: (4 \cdot (n/4)^2 = n^2/4). Total = (n^2(1 + 1/2 + 1/4 + ...) = 2n^2 = \Theta(n^2)).

A4. a=8, b=2, (\log_b a = \log_2 8 = 3), (n^3). f(n) = n^3 = (\Theta(n^3)). Case 2 (k=0): (T(n) = \Theta(n^3 \log n)).

A5. Unwinding: (T(n) = 3^k T(n-k) + (3^k - 1)/2). When n-k = 1, k = n-1. (T(n) = 3^{n-1} \cdot 1 + (3^{n-1} - 1)/2 = O(3^n)).

A6. a=2, b=4, (\log_b a = \log_4 2 = 0.5), (n^{0.5}). f(n) = n^{0.5} = (\Theta(n^{0.5})). Case 2 (k=0): (T(n) = \Theta(n^{0.5} \log n)).

A7. Best case: (T(n) = 2T(n/2) + n = O(n \log n)). Worst case: (T(n) = T(n-1) + n = O(n^2)).

A8. (T(n) = T(n-1) + 2 = T(n-2) + 4 = ... = T(1) + 2(n-1) = O(n)).

A9. a=7, b=3, (\log_b a = \log_3 7 \approx 1.77). (n^{\log_b a} \approx n^{1.77}). f(n) = n^2. Compare: n^2 vs n^{1.77}. Since n^2 grows faster, (f(n) = \Omega(n^{\log_b a + \varepsilon})), Case 3: (T(n) = \Theta(n^2)).

A10. For n > threshold, (n/2 + 17 \le 0.6n) (say). So (T(n) \le 2T(0.6n) + n). Master Theorem: a=2, b=1/0.6 ≈ 1.67, (\log_b a = \log_{1.67} 2 \approx 1.36). f(n) = n is dominated by (n^{1.36}), so Case 1: (T(n) = O(n^{1.36})). Actually, the "+17" just shifts the base case — the divide step still splits roughly in half for large n, so the recurrence is essentially (2T(n/2) + n = O(n \log n)). Join Discord Previous02. Algorithm Analysis & Big-O NotationNext04. Searching Algorithms — Linear Search & Binary Search