08. Sorting Algorithms Comparison — Heap Sort, Counting Sort, Radix Sort

What problem does this solve? You now know three sorting algorithms (selection, insertion, merge, quick). But there's a whole zoo of sorting algorithms, each with unique strengths. Heap sort guarantees O(n log n) without extra memory. Counting and radix sort break the O(n log n) barrier for integer data. This module helps you choose the right tool.

1. Heap Sort — Guaranteed O(n log n), In-Place

Mental Model

Full heap details are in [18-heaps-priority-queues.md] and [19-heap-sort.md].

Implementation

python
Run Inline
# runnable
def heap_sort(arr):
    """Sort arr in-place using heap sort.
    Time: O(n log n) — always (best, average, worst)
    Space: O(1) — in-place
    Not stable
    """
    n = len(arr)
    # Step 1: Build max-heap (heapify entire array)
    for i in range(n // 2 - 1, -1, -1):
        _sift_down(arr, n, i)
    # Step 2: Extract elements one by one
    for i in range(n - 1, 0, -1):
        arr[0], arr[i] = arr[i], arr[0]  # Swap max to end
        _sift_down(arr, i, 0)            # Restore heap on reduced array
    return arr
def _sift_down(arr, n, i):
    """Sift down element at index i in heap of size n."""
    largest = i
    left = 2 * i + 1
    right = 2 * i + 2
    if left < n and arr[left] > arr[largest]:
        largest = left
    if right < n and arr[right] > arr[largest]:
        largest = right
    if largest != i:
        arr[i], arr[largest] = arr[largest], arr[i]
        _sift_down(arr, n, largest)
# Test
arr = [12, 11, 13, 5, 6, 7]
print(f"Heap sort: {heap_sort(arr)}")

2. Counting Sort — O(n + k) for Integers

Mental Model

Implementation

python
Run Inline
# runnable
def counting_sort(arr):
    """Sort arr (non-negative integers) in O(n + k) time.
    Time: O(n + k) where k = max(arr) - min(arr)
    Space: O(n + k)
    Stable: Yes (when using cumulative counts)
    """
    if not arr:
        return arr
    k = max(arr)  # Range of values
    # Step 1: Count occurrences
    count = [0] * (k + 1)
    for num in arr:
        count[num] += 1
    # Step 2: Compute cumulative counts (prefix sums)
    for i in range(1, k + 1):
        count[i] += count[i - 1]
    # Step 3: Build output (stable — traverse input in reverse)
    output = [0] * len(arr)
    for num in reversed(arr):
        output[count[num] - 1] = num
        count[num] -= 1
    return output
# Test
arr = [4, 2, 2, 8, 3, 3, 1]
print(f"Counting sort: {counting_sort(arr)}")
# Output: [1, 2, 2, 3, 3, 4, 8]
# Step-by-step trace
arr = [4, 2, 2, 8, 3, 3, 1]
k = 8
# After counting: count = [0, 1, 2, 2, 1, 0, 0, 0, 1]
# After cumulative: count = [0, 1, 3, 5, 6, 6, 6, 6, 7]
# Count now means: "position of last occurrence of value i in sorted output"
# For value 4: count[4] = 6, so 4 goes at position 5 (0-indexed)

Complexity

3. Radix Sort — Sorting Digit by Digit

Mental Model

Implementation

python
Run Inline
# runnable
def counting_sort_by_digit(arr, exp):
    """Sort arr by digit at position 10^exp using counting sort."""
    n = len(arr)
    output = [0] * n
    count = [0] * 10  # 10 possible digits (0-9)
    # Count occurrences of each digit
    for num in arr:
        digit = (num // exp) % 10
        count[digit] += 1
    # Cumulative counts
    for i in range(1, 10):
        count[i] += count[i - 1]
    # Build output (reverse for stability)
    for num in reversed(arr):
        digit = (num // exp) % 10
        output[count[digit] - 1] = num
        count[digit] -= 1
    # Copy back
    for i in range(n):
        arr[i] = output[i]
def radix_sort(arr):
    """Sort arr using radix sort (LSD first).
    Time: O(d * (n + k)) where d = number of digits, k = 10
    Space: O(n + k)
    Stable: Yes (uses stable counting sort internally)
    """
    if not arr:
        return arr
    # Find maximum to know number of digits
    max_val = max(arr)
    # Do counting sort for every digit
    exp = 1
    while max_val // exp > 0:
        counting_sort_by_digit(arr, exp)
        exp *= 10
    return arr
# Test
arr = [170, 45, 75, 90, 802, 24, 2, 66]
print(f"Radix sort: {radix_sort(arr)}")
# Output: [2, 24, 45, 66, 75, 90, 170, 802]

Complexity

4. Comprehensive Sorting Comparison

Decision Tree

5. The Sorting Stability Concept

What is Stability?

python
Run Inline
# runnable
# Sort by name, then by grade (stable second sort keeps name order)
students = [
    ("Alice", 85),
    ("Bob", 92),
    ("Charlie", 85),
    ("Diana", 92),
]
# Stable sort by grade
# If stable: (Alice, 85) stays before (Charlie, 85)
#            (Bob, 92) stays before (Diana, 92)

Which Sorts Are Stable?

6. When Each Sort Works Best — Decision Problems

Practice Questions

Answers

A1. (a) When swaps are expensive. (b) Nearly sorted data or online sorting. (c) Guaranteed O(n log n) needed, or linked list. (d) General purpose with random access. (e) Guaranteed O(n log n) with limited memory. (f) Integer data with small range. (g) Fixed-width integer keys.

A2. Heap sort swaps the root with the last element (long distance). Example: [5, 3, 3'] with max-heap. 5 swaps with 3', then sift-down swaps 3 and 3' past each other.

A3. 32-bit = 32 bits. In base-2 radix sort: d = 32, n = 1000, k = 2. O(32 × 1000) = O(32,000) operations. Very fast!

A4. Counting sort uses array indices as keys. Non-integer values can't be used as indices.

A5. Insertion sort is excellent for small subproblems (its O(n²) cost is negligible for small n). TimSort uses insertion sort for small chunks (typically ≤ 64 elements).

A6. No. k = 10⁶ is much larger than n log n (100 × ~7 = 700). The O(n + k) = O(10⁶) counting sort is slower than O(n log n) comparison sort.

A7. Build-heap calls sift-down on the bottom n/2 elements. Each sift-down takes O(h) where h varies. The sum of heights is O(n). Formal proof: The total cost is (\sum_{h=0}^{\log n} \frac{n}{2^{h+1}} \cdot O(h) = O(n)).

A8. Insertion Sort — O(n) on nearly sorted data.

A9. Comparison-based: Selection, Insertion, Merge, Quick, Heap. Non-comparison: Counting, Radix, Bucket sort.

A10. Counting Sort. Range k = 101 (marks 0-100), n = 10,000. O(n + k) = O(10,101) — very fast. Join Discord Previous07. Quick Sort — The Fastest Practical SortNext09. Linked Lists — Singly, Doubly, Circular