Clustering: k-Means, Hierarchical, DBSCAN

🎯 Learning Objectives

• Explain unsupervised learning and when clustering is appropriate
• Implement and interpret k-means clustering with proper k selection
• Understand hierarchical clustering and dendrogram interpretation
• Use DBSCAN for density-based clustering with arbitrary shapes
• Evaluate cluster quality using internal and external metrics

📋 Prerequisites

• Unsupervised learning concepts — learning without labels
• Distance metrics — Euclidean, Manhattan (from k-NN)
• Basic statistics — mean, variance, standard deviation

📖 Core Content

9.1 Intuition: Finding Natural Groups Without a Teacher

• Customer segmentation (marketing)
• Image compression (color quantization)
• Anomaly detection (outliers are tiny clusters)
• Recommendation systems (cluster users, recommend within cluster)

9.2 k-Means Clustering

1. Choose k (number of clusters)
2. Initialize k centroids (randomly or with k-means++)
3. Repeat until convergence:
   • Assignment step: Assign each point to nearest centroid
   • Update step: Recompute centroid as mean of assigned points Objective (Inertia):

python
Run Inline
# runnable
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
import numpy as np
# Generate synthetic data
X, _ = make_blobs(n_samples=300, centers=4, cluster_std=1.0, random_state=42)
# Fit k-means
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
kmeans.fit(X)
print(f"Inertia: {kmeans.inertia_:.2f}")
print(f"Centroids:\n{kmeans.cluster_centers_}")
print(f"Labels: {kmeans.labels_[:10]}")
# Visualize
plt.scatter(X[:, 0], X[:, 1], c=kmeans.labels_, cmap='viridis', alpha=0.7)
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1],
            marker='X', s=200, c='red', label='Centroids')
plt.title('k-Means Clustering')
plt.legend()
plt.grid(True)
plt.show()

9.2.1 Worked Example 1: k-Means by Hand

• μ₁ = mean(A, B, C, D) = mean([1,2,4,5], [2,1,4,5]) = (3, 3)
• μ₂ = mean(E, F) = mean([8,9], [1,2]) = (8.5, 1.5) Step 4: Reassign. Now check distances.

9.2.2 Choosing k: The Elbow Method

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
# Calculate inertia for k=1 to k=10
inertias = []
K = range(1, 11)
for k in K:
    kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
    kmeans.fit(X)
    inertias.append(kmeans.inertia_)
plt.plot(K, inertias, 'bo-')
plt.xlabel('k')
plt.ylabel('Inertia')
plt.title('Elbow Method for Optimal k')
plt.grid(True)
plt.show()

9.3 Hierarchical Clustering

• Agglomerative (bottom-up): Start with each point as its own cluster, merge closest pairs
• Divisive (top-down): Start with all points in one cluster, split recursively

python
Run Inline
# runnable
from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage
import matplotlib.pyplot as plt
# Agglomerative clustering
hc = AgglomerativeClustering(n_clusters=4, linkage='ward')
hc.fit(X)
print(f"Hierarchical labels: {hc.labels_}")
# Dendrogram
Z = linkage(X, method='ward')
plt.figure(figsize=(10, 5))
dendrogram(Z)
plt.title('Dendrogram (Ward Linkage)')
plt.ylabel('Distance')
plt.show()

9.4 DBSCAN: Density-Based Spatial Clustering

• eps: Maximum distance between two points to be considered neighbors
• min_samples: Minimum points to form a dense region Point Types:
• Core point: ≥ min_samples neighbors within eps
• Border point: < min_samples neighbors, but reachable from a core point
• Noise point: Not reachable from any core point

python
Run Inline
# runnable
from sklearn.cluster import DBSCAN
from sklearn.datasets import make_moons
# Non-spherical data
X, _ = make_moons(n_samples=200, noise=0.05, random_state=42)
# k-means fails on this
kmeans = KMeans(n_clusters=2, random_state=42)
print(f"k-means on moons: {kmeans.fit_predict(X)}")
# DBSCAN succeeds
db = DBSCAN(eps=0.3, min_samples=5)
db_labels = db.fit_predict(X)
print(f"DBSCAN labels: {db_labels}")
print(f"Number of clusters found: {len(set(db_labels)) - (1 if -1 in db_labels else 0)}")
print(f"Noise points: {list(db_labels).count(-1)}")

9.5 Cluster Evaluation

9.6 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Wrong k in k-Means

Pitfall 2: Inconsistent Scales

Pitfall 3: k-Means on Non-Spherical Data

📝 Practice Questions

Q1: Compute the 2nd step of k-means for points (0,0), (2,0), (5,5), (7,5) with k=2, centroids at (0,0) and (7,5).

Assignment:

• (0,0) → 0 (dist=0 vs √74=8.6)
• (2,0) → 0 (dist=2 vs √[25+25]=√50=7.07)
• (5,5) → 1 (dist=√50=7.07 vs √[4+0]=2)
• (7,5) → 1 (dist=√74=8.6 vs 0)

Update:

• μ₀ = mean([0,0], [2,0]) = (1, 0)
• μ₁ = mean([5,5], [7,5]) = (6, 5) Q2: What does a silhouette score close to 1 mean?

Each point is much closer to its own cluster than to the nearest other cluster. The clusters are dense and well-separated. By contrast, a score near 0 means clusters overlap, and negative means many points are assigned to the wrong cluster. Q3: Why does DBSCAN label some points as -1 (noise)?

Points labeled -1 cannot be reached from any core point within ε distance. They're in sparse regions that don't meet the density threshold. These are outliers that don't belong to any cluster. Q4: When would you use hierarchical over k-means?

1. You want a dendrogram — visualize the hierarchy at all levels
2. You don't know k — the dendrogram helps choose k post-hoc
3. Small dataset (< 2000 points) — hierarchical is O(m²)
4. Non-spherical clusters — with single/complete linkage Q5: What is k-means++ initialization?

Standard random initialization can lead to poor convergence. k-means++ spreads out initial centroids:

1. Pick first centroid randomly
2. For each point, compute distance to nearest centroid
3. Pick next centroid with probability proportional to distance²
4. Repeat until k centroids chosen

This leads to better, faster convergence. It's the default in sklearn (init='k-means++'). Q6: Implement k-means with silhouette score to find optimal k.

python
Run Inline
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import numpy as np

best_k, best_score = 2, -1
for k in range(2, 11):
    kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels = kmeans.fit_predict(X)
    score = silhouette_score(X, labels)
    print(f"k={k}: silhouette = {score:.3f}")
    if score > best_score:
        best_k, best_score = k, score

print(f"Best k = {best_k} with silhouette = {best_score:.3f}")

Q7: What happens in DBSCAN if eps is too small or too large?

• eps too small: Most points become noise (-1), clusters fragment into tiny groups
• eps too large: Clusters merge together, most points become core points of one big cluster

The optimal eps creates enough density within clusters while keeping them separated. Use the k-distance plot to find the optimal eps value. Q8: Compare centroid-based vs density-based clustering.

Q9: A customer segmentation has silhouette score 0.15. What does this indicate?

0.15 is low — clusters are not well-separated. The customer segments likely overlap significantly. Possible issues:

1. Wrong k (try different values)
2. Features don't separate customers well (try different features or PCA first)
3. Customers genuinely form a continuum, not discrete segments Q10: How would you cluster a dataset of 1 million customer records?

Use k-means (scalable) or Mini-Batch k-Means (even faster). Avoid hierarchical (O(m²)) and DBSCAN (O(m²) naive). Mini-Batch k-Means processes data in small batches, making it suitable for million-scale data with minimal quality loss.

python
Run Inline
from sklearn.cluster import MiniBatchKMeans
mbk = MiniBatchKMeans(n_clusters=10, batch_size=1000, random_state=42)

Q11: Explain single vs complete linkage in hierarchical clustering with an example.

• Single linkage: Distance between clusters = minimum distance between any point in A and any point in B. Creates long "chained" clusters (good for elongated shapes).
• Complete linkage: Distance = maximum distance between any point in A and any point in B. Creates compact clusters (good for spherical shapes).

Example: Two circular clusters connected by a thin line of points. Single linkage would merge them into one chain. Complete linkage would keep them separate (the line's nearest-endpoint distance is small, but its farthest is large). Q12: Your k-means gives different results each run. Why?

k-means is sensitive to initialization. Different random starts produce different final clusters. To make it reproducible:

1. Use random_state in sklearn
2. Use k-means++ initialization (default)
3. Run multiple restarts (n_init=10+) and keep the one with lowest inertia

sklearn's defaults handle this, but the variability is inherent to the algorithm.

🔗 Cross-References

• Next Topic: Dimensionality Reduction
• Related: k-Nearest Neighbors — distance metrics
• Related: PCA — pre-processing for clustering
• External: IITM BSCS2004 Week 9, Hands-On ML Ch. 9, ISLR Ch. 10 Join Discord PreviousSupport Vector MachinesNextDimensionality Reduction