Kernel Methods

549 words

3 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# Kernel Methods ## 🎯 Learning Objectives - Explain the kernel trick and why it's computationally efficient - Implement kernelized SVM with RBF kernel - Understand Mercer's theorem and valid kernels - Use kernel PCA for non-linear dimensionality reduction ## 📖 Core Content ### 4.1 Intuition: The Kernel Trick Some...

Kernel Methods

🎯 Learning Objectives

Explain the kernel trick and why it's computationally efficient
Implement kernelized SVM with RBF kernel
Understand Mercer's theorem and valid kernels
Use kernel PCA for non-linear dimensionality reduction

📖 Core Content

4.1 Intuition: The Kernel Trick

Some problems that aren't linearly separable become separable in higher dimensions. But explicitly mapping data to high dimensions is expensive. The kernel trick computes dot products in the transformed space without ever computing the transformation.

4.2 Mathematical Formulation

Feature map:

\phi: \mathbb{R}^n \to \mathbb{R}^d

, where

d \gg n

(often

d = \infty

). Kernel function:

K(x, x') = \phi(x)^T \phi(x')

Mercer's theorem: A symmetric function

K(x, x')

is a valid kernel iff the kernel matrix

K_{ij} = K(x_i, x_j)

is positive semi-definite for all finite sets

\{x_i\}

4.3 Common Kernels

Kernel	Formula	Feature Space	Use
Linear	$x^T x'$	Same	Baseline
Polynomial	$(\gamma x^T x' + r)^d$	All monomials up to degree d	Images
RBF (Gaussian)	$\exp(-\gamma\	x - x'\	^2)$
Sigmoid	$\tanh(\gamma x^T x' + r)$	Neural network-like	Certain text problems

4.4 The Representer Theorem

For kernel methods, the optimal weight vector can be expressed as:

w = \sum_{i=1}^{m} \alpha_i \phi(x^{(i)})

So the decision function becomes:

f(x) = w^T \phi(x) + b = \sum_{i=1}^{m} \alpha_i K(x^{(i)}, x) + b

This is the "kernel trick": we only need

K(x_i, x)

, never

\phi(x_i)

4.5 Kernelized SVM

The dual formulation only uses kernel evaluations:

\max_\alpha \sum \alpha_i - \frac{1}{2} \sum\sum \alpha_i \alpha_j y_i y_j K(x_i, x_j)

Subject to

0 \leq \alpha_i \leq C

\sum \alpha_i y_i = 0


python
# runnable
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
# Non-linear data
X, y = make_circles(n_samples=300, noise=0.1, factor=0.5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# RBF kernel
svm_rbf = SVC(kernel='rbf', C=10, gamma=1.0)
svm_rbf.fit(X_train, y_train)
print(f"RBF SVM accuracy: {svm_rbf.score(X_test, y_test):.3f}")
# Compare with linear kernel
svm_linear = SVC(kernel='linear', C=10)
svm_linear.fit(X_train, y_train)
print(f"Linear SVM accuracy: {svm_linear.score(X_test, y_test):.3f}")

4.6 Kernel PCA


python
# runnable
from sklearn.decomposition import KernelPCA
from sklearn.datasets import make_circles
X, y = make_circles(n_samples=200, noise=0.05, random_state=42)
kpca = KernelPCA(n_components=2, kernel='rbf', gamma=10)
X_kpca = kpca.fit_transform(X)
print(f"Original shape: {X.shape}, KPCA shape: {X_kpca.shape}")
print(f"Explained variance ratio: {kpca.eigenvalues_}")

📝 Practice Questions

Q1: What makes a valid kernel function?

Mercer's theorem: $K(x, x')$ is a valid kernel if for any finite set of points, the kernel matrix $K_{ij} = K(x_i, x_j)$ is positive semi-definite (all eigenvalues ≥ 0). This ensures there exists some feature map $\phi$ such that $K(x, x') = \phi(x)^T \phi(x')$ . Q2: Why is RBF kernel infinite-dimensional?

The RBF kernel $\exp(-\gamma\|x-x'\|^2)$ can be expanded as a Taylor series: $\exp(-\gamma\|x-x'\|^2) = \sum_{n=0}^{\infty} \frac{(\gamma)^n}{n!} (x^T x')^n$

Each term $(x^T x')^n$ corresponds to a feature space of polynomial degree n. The sum over all n means the implicit feature space is a sum of all polynomial feature spaces — infinite-dimensional. Q3: When does the kernel trick save computation?

If $\phi(x)$ maps to a very high (or infinite) dimensional space, computing $\phi(x)$ explicitly is impossible. The kernel trick computes $K(x_i, x_j) = \phi(x_i)^T \phi(x_j)$ directly without computing $\phi$ . For RBF, this is just $\exp(-\gamma\|x_i-x_j\|^2)$ — O(n) time per pair, regardless of the infinite-dimensional feature space. Join Discord PreviousNeural NetworksNextTime Series