Kernel Methods

🎯 Learning Objectives

• Explain the kernel trick and why it's computationally efficient
• Implement kernelized SVM with RBF kernel
• Understand Mercer's theorem and valid kernels
• Use kernel PCA for non-linear dimensionality reduction

📖 Core Content

4.1 Intuition: The Kernel Trick

4.2 Mathematical Formulation

4.3 Common Kernels

4.4 The Representer Theorem

4.5 Kernelized SVM

python
Run Inline
# runnable
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
# Non-linear data
X, y = make_circles(n_samples=300, noise=0.1, factor=0.5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# RBF kernel
svm_rbf = SVC(kernel='rbf', C=10, gamma=1.0)
svm_rbf.fit(X_train, y_train)
print(f"RBF SVM accuracy: {svm_rbf.score(X_test, y_test):.3f}")
# Compare with linear kernel
svm_linear = SVC(kernel='linear', C=10)
svm_linear.fit(X_train, y_train)
print(f"Linear SVM accuracy: {svm_linear.score(X_test, y_test):.3f}")

4.6 Kernel PCA

python
Run Inline
# runnable
from sklearn.decomposition import KernelPCA
from sklearn.datasets import make_circles
X, y = make_circles(n_samples=200, noise=0.05, random_state=42)
kpca = KernelPCA(n_components=2, kernel='rbf', gamma=10)
X_kpca = kpca.fit_transform(X)
print(f"Original shape: {X.shape}, KPCA shape: {X_kpca.shape}")
print(f"Explained variance ratio: {kpca.eigenvalues_}")

📝 Practice Questions

Q1: What makes a valid kernel function?

Mercer's theorem: $K(x, x')$ is a valid kernel if for any finite set of points, the kernel matrix $K_{ij} = K(x_i, x_j)$ is positive semi-definite (all eigenvalues ≥ 0). This ensures there exists some feature map $\phi$ such that $K(x, x') = \phi(x)^T \phi(x')$. Q2: Why is RBF kernel infinite-dimensional?

The RBF kernel $\exp(-\gamma\|x-x'\|^2)$ can be expanded as a Taylor series: $\exp(-\gamma\|x-x'\|^2) = \sum_{n=0}^{\infty} \frac{(\gamma)^n}{n!} (x^T x')^n$

Each term $(x^T x')^n$ corresponds to a feature space of polynomial degree n. The sum over all n means the implicit feature space is a sum of all polynomial feature spaces — infinite-dimensional. Q3: When does the kernel trick save computation?

If $\phi(x)$ maps to a very high (or infinite) dimensional space, computing $\phi(x)$ explicitly is impossible. The kernel trick computes $K(x_i, x_j) = \phi(x_i)^T \phi(x_j)$ directly without computing $\phi$. For RBF, this is just $\exp(-\gamma\|x_i-x_j\|^2)$ — O(n) time per pair, regardless of the infinite-dimensional feature space. Join Discord PreviousNeural NetworksNextTime Series