02. Algorithm Analysis & Big-O Notation

What problem does this solve? When you write a program, how do you know if it will finish in 1 second or 1 year? Algorithm analysis gives us a mathematical language to talk about how runtime grows with input size — without running the code on any specific machine.

1. Why Analyze Algorithms?

Mental Model

• Naïve method (selection sort): Find the max, put it aside, repeat. For 1000 papers, you scan 1000 times = ~1 million operations.
• Smart method (merge sort): Split into piles, sort each pile, merge. For 1000 papers, ~10,000 operations. The difference isn't a constant factor — it's the rate of growth. For 1 billion papers, naïve takes 10^18 operations (centuries), smart takes ~30 billion operations (minutes on a supercomputer). We measure time complexity (how runtime grows) and space complexity (how memory grows).

Key Parameters

• Input size (n) — length of list, number of vertices, number of digits
• Basic operation — comparison, assignment, arithmetic operation
• Worst case — the input that makes the algorithm slowest

2. Asymptotic Notation — Big O, Ω, Θ

Big O — Upper Bound (O(g(n)))

Intuition: (f(n) = O(g(n))) means (f(n)) grows no faster than (g(n)). Formal Definition: (f(n) = O(g(n))) if there exist constants (c > 0) and (n_0 \ge 0) such that:

python
Run Inline
# runnable
# Example: f(n) = 100n + 5 is O(n^2)
# Because: 100n + 5 ≤ 100n + 5n = 105n ≤ 105n² for n ≥ 1
# So: c = 105, n₀ = 1
# But it's also O(n)!
# 100n + 5 ≤ 101n for n ≥ 5
# So: c = 101, n₀ = 5
# Tightest bound: O(n) — we always want the tightest!

Omega — Lower Bound (\Omega(g(n)))

Intuition: (f(n) = \Omega(g(n))) means (f(n)) grows at least as fast as (g(n)). Formal Definition: (f(n) = \Omega(g(n))) if there exist constants (c > 0) and (n_0 \ge 0) such that:

Theta — Tight Bound (\Theta(g(n)))

Intuition: (f(n) = \Theta(g(n))) means (f(n)) grows at the same rate as (g(n)). Formal Definition: (f(n) = \Theta(g(n))) if (f(n) = O(g(n))) AND (f(n) = \Omega(g(n))).

python
Run Inline
# runnable
# Example: f(n) = n(n-1)/2 is Θ(n²)
# Upper bound: n(n-1)/2 ≤ n²/2 ≤ n² for n ≥ 1
# Lower bound: n(n-1)/2 ≥ n²/4 for n ≥ 2
# So c₁ = 1/4, c₂ = 1, n₀ = 2

Growth Hierarchy (from slowest to fastest)

3. Analyzing Iterative Programs

Rule 1: Single loop → O(n)

python
Run Inline
# runnable
def find_max(L):
    """O(n) — one pass through the list."""
    max_val = L[0]          # 1 operation
    for x in L:             # n iterations
        if x > max_val:     # 1 comparison per iteration
            max_val = x     # at most 1 assignment
    return max_val
# Total: 1 + n * (1 + up to 1) + 1 = O(n)

Rule 2: Nested loops → O(n²)

python
Run Inline
# runnable
def has_duplicates(L):
    """O(n²) — compare every pair."""
    n = len(L)
    for i in range(n):           # n iterations
        for j in range(i + 1, n):  # n-i-1 iterations
            if L[i] == L[j]:      # 1 comparison
                return True
    return False
# Total comparisons: n(n-1)/2 = O(n²)

Rule 3: Halving loop → O(log n)

python
Run Inline
# runnable
def count_bits(n):
    """O(log n) — n halves each iteration."""
    count = 0
    while n > 0:
        n = n // 2
        count += 1
    return count
print(count_bits(32))   # 6 (32→16→8→4→2→1→0)
print(count_bits(1000)) # 10
# Why? n halves each time → number of iterations = log₂(n) + 1

Rule 4: Loop with constant work → Multiply

python
Run Inline
# runnable
def matrix_multiply(A, B):
    """O(mnp) — three nested loops for matrix multiplication."""
    m, n = len(A), len(A[0])
    p = len(B[0])
    C = [[0 for _ in range(p)] for _ in range(m)]
    for i in range(m):       # m iterations
        for j in range(p):   # p iterations
            for k in range(n):  # n iterations
                C[i][j] += A[i][k] * B[k][j]
    return C
# If all are n: O(n³)

Rule 5: Sum of loop iterations

python
Run Inline
# runnable
def print_triangle(n):
    """O(n²) — rows decrease linearly."""
    for i in range(n, 0, -1):  # n, n-1, n-2, ..., 1
        print('*' * i)
# Total operations: n + (n-1) + ... + 1 = n(n+1)/2 = O(n²)

4. Analyzing Recursive Programs

Step 1: Write the Recurrence Relation

• (a) = number of recursive calls
• (b) = fraction of input each call processes
• (f(n)) = work done outside recursion (combining/partitioning)

Common Recurrences

Step-by-Step: Solving by Unwinding

pseudo
T(n) = T(n/2) + 1
     = [T(n/4) + 1] + 1 = T(n/4) + 2
     = [T(n/8) + 1] + 2 = T(n/8) + 3
     = ...
     = T(n/2^k) + k

pseudo
T(n) = 2T(n/2) + n
     = 2[2T(n/4) + n/2] + n = 4T(n/4) + 2n
     = 4[2T(n/8) + n/4] + 2n = 8T(n/8) + 3n
     = ...
     = 2^k T(n/2^k) + kn

5. Sum and Max Rules

python
Run Inline
# runnable
# Phase 1: Sort (O(n log n))
# Phase 2: Search (O(log n))
# Total: O(n log n) — the larger term dominates
def process(L):
    L.sort()                          # O(n log n)
    target = 42
    # Binary search
    left, right = 0, len(L) - 1
    while left <= right:              # O(log n)
        mid = (left + right) // 2
        if L[mid] == target:
            return True
        elif L[mid] < target:
            left = mid + 1
        else:
            right = mid - 1
    return False
# Total: O(n log n) + O(log n) = O(n log n)

6. Common Pitfalls

python
Run Inline
# PITFALL 1: O(n) + O(n) ≠ O(2n) ... it's O(n)
def two_loops(L):
    for x in L:    # O(n)
        print(x)
    for y in L:    # O(n)
        print(y)
# Total: O(n) + O(n) = O(n)
# PITFALL 2: Hidden O(n) operations
def bad_duplicate_check(L):
    result = []
    for x in L:              # n iterations
        if x not in result:  # 'not in' on list is O(n)!
            result.append(x)
    return result
# Total: O(n²) — not O(n)!
# PITFALL 3: Python slicing creates copies
def bad_average(L):
    if len(L) <= 1:
        return L
    # L[:len(L)//2] creates a copy — O(n)!
    left_sum = sum(L[:len(L)//2])  # O(n)
    right_sum = sum(L[len(L)//2:]) # O(n)
    return (left_sum + right_sum) / len(L)
# Total: O(n) but creates O(n) memory
# PITFALL 4: Integer input size
def is_prime(n):
    """n is the number itself, not number of digits."""
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True
# If n is the magnitude, input size = log₂(n) bits
# Complexity: O(√n) = O(2^(bits/2)) — EXPONENTIAL in input size!

7. Complexity Comparison Table

Practice Questions

python
Run Inline
def f(n):
    s = 0
    for i in range(n):
        for j in range(i, n):
            s += 1
    return s

python
Run Inline
def g(n):
    if n <= 1:
        return 1
    return g(n-1) + g(n-1) + 1

python
Run Inline
def find(L, target):
    for x in L:
        if x == target:
            return True
    return False
# Claimed: O(log n) because "we stop early if found"

Answers

A1. (O(n^2)). The inner loop runs n, n-1, n-2, ..., 1 times. Sum = n(n+1)/2 = O(n²).

A2. (3n^2 + 2n + 1 \le 3n^2 + 2n^2 + n^2 = 6n^2) for (n \ge 1). So (c = 6), (n_0 = 1).

A3. (T(n) = 2T(n-1) + 1). Unwinding: (T(n) = 2^n - 1 = O(2^n)).

A4. Unwinding:

pseudo
T(n) = 4T(n/2) + n
     = 4[4T(n/4) + n/2] + n = 16T(n/4) + 2n + n
     = 16[4T(n/8) + n/4] + 3n = 64T(n/8) + 4n + 2n + n
     = ...
     = 4^k T(n/2^k) + n(2^k - 1)

When (n/2^k = 1), (k = \log_2 n), (4^k = n^2). (T(n) = n² \cdot T(1) + n(2^{\log n} - 1) = n² + n² - n = O(n²))

A5. Slowest to fastest: (n \log n < n^{1.5} < n^3 < 2^n)

A6. (O(n)) — linked lists have no random access; you must traverse from the head.

A7. Yes. Upper bound: (n^2 + 100n + 1000 \le 3n^2) for n ≥ 101. Lower bound: (n^2 + 100n + 1000 \ge n^2) for all n ≥ 0. So (c_1 = 1, c_2 = 3, n_0 = 101).

A8. Worst-case analysis must consider the worst-case input. In worst case, target isn't in the list, so we scan all n elements — O(n). Best-case isn't used as the standard measure.

A9. (T(n) = T(n/2) + n). Unwinding:

pseudo
T(n) = T(n/2) + n
     = T(n/4) + n/2 + n
     = T(n/8) + n/4 + n/2 + n
     = ...
     = T(1) + n(1 + 1/2 + 1/4 + ...) = 1 + 2n = O(n)

This is Quick Select / Fast Select (finding kth smallest element).

A10.

python
Run Inline
def all_unique(s):
    # Without extra space: O(n²) compare all pairs
    for i in range(len(s)):
        for j in range(i + 1, len(s)):
            if s[i] == s[j]:
                return False
    return True

(With a set: O(n), but that's "extra data structure".) Join Discord Previous01. Python Refresher — Classes, Exceptions, Timing, RecursionNext03. Complexity Analysis — Recurrence Relations & Master Theorem