k-Nearest Neighbors (k-NN)

🎯 Learning Objectives

• Explain the k-NN algorithm in plain English and why it's called "lazy learning"
• Compute distances using Euclidean, Manhattan, and Minkowski metrics
• Determine the optimal k using cross-validation
• Implement k-NN for classification and regression
• Understand the curse of dimensionality and how to mitigate it

📋 Prerequisites

• Basic geometry — distance between points in n-dimensional space
• Probability — voting/majority rule concepts
• Classification basics — what it means to predict a class

📖 Core Content

5.1 Intuition: "Birds of a Feather Flock Together"

5.2 Formal Definition

1. Compute distance between $x$ and every training example
2. Select the $k$ training examples with smallest distance
3. Classification (majority vote):

5.3 Distance Metrics

• $p = 1$: Manhattan distance
• $p = 2$: Euclidean distance
• $p \to \infty$: Chebyshev distance

5.4 Worked Example 1: Hand Calculation

• d(A) = √[(3-1)² + (4-2)²] = √(4 + 4) = √8 = 2.83
• d(B) = √[(3-2)² + (4-3)²] = √(1 + 1) = √2 = 1.41
• d(C) = √[(3-3)² + (4-5)²] = √(0 + 1) = 1.00
• d(D) = √[(3-5)² + (4-1)²] = √(4 + 9) = √13 = 3.61
• d(E) = √[(3-6)² + (4-4)²] = √(9 + 0) = 3.00 Step 2: Sort by distance:

1. C (1.00) — Blue
2. B (1.41) — Red
3. A (2.83) — Red Step 3: Majority vote: Red (2) > Blue (1). Predict: Red. Step 4: What if k=5? Votes: Red (A,B,D) = 3, Blue (C,E) = 2. Still predicts Red.

5.5 Choosing k

python
Run Inline
# runnable
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
import numpy as np
import matplotlib.pyplot as plt
# Generate sample data
np.random.seed(42)
X = np.random.randn(100, 2)
y = (X[:, 0]**2 + X[:, 1]**2 > 1).astype(int)
# Find optimal k using cross-validation
k_range = range(1, 30)
cv_scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(knn, X, y, cv=5, scoring='accuracy')
    cv_scores.append(scores.mean())
optimal_k = k_range[np.argmax(cv_scores)]
print(f"Optimal k: {optimal_k}")
plt.plot(k_range, cv_scores, marker='o')
plt.xlabel('k')
plt.ylabel('Cross-validation accuracy')
plt.title('Optimal k Selection')
plt.grid(True)
plt.show()

5.6 The Curse of Dimensionality

python
Run Inline
# runnable
import numpy as np
for dim in [1, 2, 5, 10, 50, 100]:
    X = np.random.randn(1000, dim)
    # Distance from first point to all others
    dists = np.linalg.norm(X - X[0], axis=1)
    nearest = dists[dists > 0].min()
    farthest = dists.max()
    ratio = nearest / farthest
    print(f"Dim {dim:3d}: nearest={nearest:.3f}, farthest={farthest:.3f}, ratio={ratio:.3f}")

1. Feature selection (remove irrelevant features)
2. Dimensionality reduction (PCA before k-NN)
3. Use Manhattan distance (more robust in high dimensions)
4. Increase k (smooths over the distance problem)

5.7 Weighted k-NN

5.8 Python Implementation

python
Run Inline
# runnable
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# Load data
iris = load_iris()
X, y = iris.data, iris.target
# IMPORTANT: Scale features for k-NN!
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Split
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.3, random_state=42)
# Train k-NN
knn = KNeighborsClassifier(n_neighbors=5, weights='distance', metric='euclidean')
knn.fit(X_train, y_train)
# Evaluate
accuracy = knn.score(X_test, y_test)
print(f"Accuracy: {accuracy:.3f}")
# Predict a new point
new_point = scaler.transform(5.0, 3.5, 1.5, 0.3)
pred = knn.predict(new_point)
probs = knn.predict_proba(new_point)
print(f"Predicted class: {pred[0]} ({iris.target_names[pred[0]]})")
print(f"Probabilities: {probs[0]}")

5.9 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Not Scaling Features

Pitfall 2: Choosing k = 1 or k = N

Pitfall 3: Ignoring the Curse of Dimensionality

📝 Practice Questions

Q1: Compute Manhattan distance between (3,4) and (1,2).

$d_{manhattan} = |3-1| + |4-2| = 2 + 2 = 4$

Euclidean would be √8 ≈ 2.83. Manhattan is always ≥ Euclidean. Q2: For points A(0,0,Red), B(1,1,Red), C(2,2,Blue), what does k=3 predict for (1.5,1.5)?

Distances from (1.5,1.5):

• d(A) = √[(1.5-0)² + (1.5-0)²] = √(2.25+2.25) = √4.5 = 2.12
• d(B) = √[(1.5-1)² + (1.5-1)²] = √(0.25+0.25) = √0.5 = 0.71
• d(C) = √[(1.5-2)² + (1.5-2)²] = √(0.25+0.25) = √0.5 = 0.71

k=3: Red (A=2.12, B=0.71) = 2 votes, Blue (C=0.71) = 1 vote. Predict: Red.

With weighted voting, B and C have equal weight (both 0.71), so B (Red) and C (Blue) tie, and A (Red) breaks the tie. Still Red. Q3: What happens to k-NN if all features are on different scales?

Features with larger magnitudes dominate the distance calculation. If one feature ranges 0-1 and another ranges 0-1000, the second feature effectively determines the neighbors. This makes the first feature irrelevant. Always scale features to equalize their influence. Q4: Why is k-NN called "lazy learning"?

Because it defers all computation to prediction time. During training, it simply memorizes the dataset (no model building). During prediction, it computes distances to ALL training points and finds the k nearest. This is the opposite of "eager" learning (like neural networks) where training is expensive but prediction is cheap. Q5: With 1000 training examples and 20 features, estimate prediction time complexity.

For each prediction: O(m × n) = O(1000 × 20) = O(20,000) distance computations. With efficient data structures (KD-tree, Ball tree), this can reduce to O(log m × n) in low dimensions. But in 20 dimensions, tree-based structures degrade. Q6: What's the best k if the data is very noisy?

Larger k — averaging over more neighbors smooths out noise. The tradeoff is that the decision boundary becomes less flexible. Use cross-validation to find the optimal balance between noise reduction and underfitting. Q7: How does weighted k-NN differ from standard k-NN?

Standard k-NN gives each neighbor one vote regardless of distance. Weighted k-NN gives closer neighbors more votes (weight = 1/distance). This makes the algorithm more robust — distant neighbors have minimal influence. It also eliminates ties more effectively. Q8: Implement k-NN regression for house price prediction.

python
Run Inline
from sklearn.neighbors import KNeighborsRegressor
from sklearn.preprocessing import StandardScaler
import numpy as np

# Sample: sq_ft, bedrooms, location_score → price
X = np.array([1000, 2, 5], [1500, 3, 7], [2000, 3, 8], [2500, 4, 9](/courses/bscs2004/notes/1000%2C%202%2C%205%5D%2C%20%5B1500%2C%203%2C%207%5D%2C%20%5B2000%2C%203%2C%208%5D%2C%20%5B2500%2C%204%2C%209))
y = np.array([200000, 300000, 350000, 450000])

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

knn = KNeighborsRegressor(n_neighbors=2, weights='distance')
knn.fit(X_scaled, y)

# Predict for a new house
new = scaler.transform(1800, 3, 7.5)
pred = knn.predict(new)
print(f"Predicted price: ${pred[0]:.0f}")

Q9: In 2D, draw the decision boundary for k=1 vs k=5 on the same data. What's different?

k=1: Decision boundary is very jagged — every training point has its own "territory" (Voronoi diagram). The boundary passes exactly between every pair of points with different labels.

k=5: Decision boundary is smooth and wavy — it ignores local noise and follows the broader pattern. Small islands of one class inside another disappear.

More neighbors = smoother boundary = higher bias = lower variance. Q10: Your k-NN model has high training accuracy but low test accuracy. What do you do?

This is overfitting — the model has memorized noise. Solutions:

1. Increase k — smoother decision boundary
2. Use weighted voting — reduce influence of far neighbors
3. Add more training data — helps all models
4. Remove irrelevant features — reduces noise dimensions

Start by increasing k and checking cross-validation score. Q11: Why is cosine distance preferred for text data in k-NN?

Cosine distance measures the angle between vectors, ignoring magnitude. In text data, document length varies hugely (a 100-word article vs. a 1000-word article). Cosine similarity captures the content/theme without being dominated by length. Example: two articles about "machine learning" will have high cosine similarity regardless of their lengths. Q12: Compare k-NN and Logistic Regression.

Choose k-NN for non-linear, low-dimensional data. Choose Logistic Regression for interpretability or high-dimensional data.

🔗 Cross-References

• Next Topic: Decision Trees — interpretable tree-based models
• Related: Feature Scaling — why scaling matters
• Related: Dimensionality Reduction — mitigating curse of dimensionality
• External: IITM BSCS2004 Week 5, Hands-On ML Ch. 5 (distance-based models) Join Discord PreviousLogistic RegressionNextDecision Trees