Support Vector Machines

🎯 Learning Objectives

• Explain how SVM finds the maximum margin hyperplane
• Understand the role of support vectors
• Use the kernel trick to handle non-linear data
• Tune SVM hyperparameters (C, gamma, kernel)
• Implement SVM for classification and regression

📋 Prerequisites

• Linear Algebra — hyperplanes, dot products, vector projections
• Convex Optimization — understanding Lagrange multipliers (conceptual)
• Logistic Regression — linear decision boundaries

📖 Core Content

8.1 Intuition: The Widest Possible Street

8.2 Formal Definition

• All positive examples satisfy $w^T x + b \geq 1$
• All negative examples satisfy $w^T x + b \leq -1$
• No points fall inside the margin Soft Margin SVM (allows misclassifications):

8.3 Worked Example 1: Finding the Support Vectors

• A(1,1): $-2(1) -2(1) + 5 = 1$ → on the margin boundary ✓
• B(2,3): $-2(2) -2(3) + 5 = -5$ → far from boundary ✓
• C(3,2): $-2(3) -2(2) + 5 = -5$ → far from boundary ✓ Support vectors: Only A is a support vector! B and C don't influence the boundary.

8.4 The Kernel Trick

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
from sklearn.svm import SVC
# Create non-linear data (concentric circles)
np.random.seed(42)
X = np.random.randn(200, 2)
y = (X[:, 0]**2 + X[:, 1]**2 > 1.5).astype(int)
# SVM with RBF kernel
svm = SVC(kernel='rbf', C=10, gamma=1.0)
svm.fit(X, y)
print(f"Accuracy: {svm.score(X, y):.3f}")
print(f"Number of support vectors: {len(svm.support_)}")
# Plot decision boundary
xx, yy = np.meshgrid(np.linspace(-3, 3, 50), np.linspace(-3, 3, 50))
Z = svm.decision_function(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, levels=20, cmap='RdBu', alpha=0.7)
plt.scatter(X[y==0,0], X[y==0,1], color='blue', label='Class 0')
plt.scatter(X[y==1,0], X[y==1,1], color='red', label='Class 1')
plt.scatter(svm.support_vectors_[:,0], svm.support_vectors_[:,1],
            s=100, facecolors='none', edgecolors='k', label='Support Vectors')
plt.legend()
plt.grid(True)
plt.show()

8.5 Hyperparameter Tuning

• Small C: Large margin allowed, more training errors tolerated (high bias)
• Large C: Small margin, few training errors tolerated (high variance) γ (Gamma - for RBF kernel):
• Small γ: Large influence radius → smooth decision boundary (high bias)
• Large γ: Small influence radius → wiggly boundary (high variance)

python
Run Inline
# runnable
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_iris
iris = load_iris()
X, y = iris.data, iris.target
param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': [0.01, 0.1, 1, 'auto', 'scale'],
    'kernel': ['rbf']
}
grid = GridSearchCV(SVC(), param_grid, cv=5, scoring='accuracy', n_jobs=-1)
grid.fit(X, y)
print(f"Best params: {grid.best_params_}")
print(f"Best CV accuracy: {grid.best_score_:.3f}")

8.6 SVM for Regression (SVR)

8.7 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Not Scaling Features

Pitfall 2: Choosing Wrong C Value

Pitfall 3: Using Linear Kernel on Non-Linear Data

📝 Practice Questions

Q1: What makes a point a "support vector"?

A point $x^{(i)}$ is a support vector if it lies on the margin boundary or inside the margin. Mathematically: $y^{(i)}(w^T x^{(i)} + b) \leq 1$. Support vectors are the only points that determine the SVM's decision boundary — removing any non-support vector doesn't change the model. Q2: How does the C parameter affect SVM?

C controls penalty for margin violations:

• C large (e.g., 100): Heavy penalty for misclassifications → narrower margin, fewer support vectors, may overfit
• C small (e.g., 0.01): Light penalty → wider margin, more support vectors, may underfit

C is inversely related to regularization strength (like 1/α in ridge regression). Q3: What is the kernel trick and why is it useful?

The kernel trick computes dot products in a transformed feature space without explicitly computing the transformation $φ(x)$. This:

1. Saves computation (mapping to infinite dimensions would be impossible)
2. Allows SVM to find non-linear decision boundaries efficiently
3. Works because SVM only needs dot products, not the transformed coordinates themselves Q4: For an RBF kernel, what does gamma control?

Gamma ($γ$) controls the influence radius of each training example:

• Small γ: Each example influences a large area → smooth boundary (high bias)
• Large γ: Each example has tiny influence → wiggly boundary (high variance)

Default: γ = 1/(n_features × variance(X)) or just 1/n_features. Q5: Why is SVM sensitive to feature scaling?

The margin is measured in feature space — it's the perpendicular distance from the hyperplane to the nearest points. If one feature has range [0, 1000] and another [0, 1], the first feature dominates distance calculations. The margin will be defined almost entirely by that feature.

Solution: Standardize all features to have zero mean and unit variance. Q6: Compute $\|w\|$ if the margin width is 2.

Margin $= 2/\|w\| = 2 \implies \|w\| = 1$

The norm of w is inversely proportional to the margin. To maximize the margin, we minimize $\|w\|$. Q7: Your SVM takes too long to train on 100k examples. What do you do?

SVM training is O(m²) to O(m³) for RBF kernel. Options:

1. Use LinearSVC (linear kernel, O(m) training) — much faster
2. Reduce dataset size (random subset)
3. Use SGDClassifier with hinge loss (online SVM approximation)
4. Use a different model (Random Forest scales better)

SVM is best for small-to-medium datasets (< 10k is ideal). Q8: Implement SVM with polynomial kernel.

python
Run Inline
from sklearn.svm import SVC
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=200, noise=0.15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

svm = SVC(kernel='poly', degree=3, C=10, gamma='auto')
svm.fit(X_train, y_train)
print(f"Accuracy: {svm.score(X_test, y_test):.3f}")

Q9: What is the dual formulation of SVM?

The dual formulation expresses SVM optimization in terms of Lagrange multipliers $α_i$: $\max_α \sum α_i - \frac{1}{2}\sum\sum α_i α_j y_i y_j K(x_i, x_j)$

Subject to $0 \leq α_i \leq C$ and $\sum α_i y_i = 0$.

The dual is important because:

1. It introduces the kernel trick naturally
2. Most $α_i = 0$ — only support vectors have $α_i > 0$
3. The decision function becomes: $f(x) = \sum α_i y_i K(x_i, x) + b$ Q10: When would you choose SVM over Random Forest?

Choose SVM over RF when:

1. Data is well-structured with clear margin (text, images with good features)
2. n_features > m_samples (high-dim, low-sample — SVM excels)
3. You need a compact model (only support vectors stored)
4. Binary classification with interpretable margin

Choose RF when:

1. Data is large (> 10k examples)
2. Feature types are mixed
3. You want feature importance rankings
4. Training speed matters Q11: How does the ε parameter work in SVR?

ε-insensitive loss: points with error $|y - ŷ| < ε$ cost nothing (they're "inside the tube"). Points outside the tube are penalized linearly: $L(y, ŷ) = \max(0, |y - ŷ| - ε)$

Small ε: tight tube, more support vectors, may overfit. Large ε: loose tube, simpler model. Q12: Your SVM gives poor probability estimates. Why?

SVM is not designed for probability estimation — it outputs distance to the hyperplane, not a probability. sklearn's probability=True uses Platt scaling (fitting a logistic regression on SVM outputs), but:

1. It requires extra cross-validation (slower)
2. Calibration may be poor, especially with small data
3. Probabilities don't represent true class probabilities

Use logistic regression or calibrated Random Forest if probabilities are critical.

🔗 Cross-References

• Next Topic: Clustering — unsupervised learning
• Related: Logistic Regression — alternative linear classifier
• Related: Kernel Methods — deeper on kernels
• External: IITM BSCS2004 Week 8, Hands-On ML Ch. 5, ISLR Ch. 9 Join Discord PreviousEnsemble BoostingNextClustering