Dimensionality Reduction: PCA & t-SNE

🎯 Learning Objectives

• Explain why dimensionality reduction is necessary and when to use it
• Understand PCA: eigendecomposition, explained variance, and projection
• Implement PCA for visualization and noise reduction
• Use t-SNE for high-dimensional data visualization
• Compare linear (PCA) vs non-linear (t-SNE) dimensionality reduction

📋 Prerequisites

• Linear algebra — eigenvalues, eigenvectors, covariance matrix
• Statistics — variance, covariance, standardization
• Visualization — scatter plots

📖 Core Content

10.1 Intuition: Seeing in 4D (and Beyond)

10.2 PCA: Principal Component Analysis

1. Standardize the data (mean=0, std=1 for each feature)
2. Compute covariance matrix
3. Compute eigenvectors and eigenvalues of covariance matrix
4. Sort eigenvectors by decreasing eigenvalue
5. Select top k eigenvectors as principal components
6. Project data: $X_{reduced} = X \cdot W_k$

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
iris = load_iris()
X, y = iris.data, iris.target
# Standardize
X_scaled = StandardScaler().fit_transform(X)
# PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Cumulative: {pca.explained_variance_ratio_.cumsum()}")
print(f"Components:\n{pca.components_}")
# Plot
for target, color, label in zip([0, 1, 2], ['r', 'g', 'b'], iris.target_names):
    plt.scatter(X_pca[y==target, 0], X_pca[y==target, 1],
                c=color, label=label, alpha=0.7)
plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.2%})')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.2%})')
plt.title('PCA of Iris Dataset')
plt.legend()
plt.grid(True)
plt.show()

10.2.1 Worked Example 1: 2D → 1D PCA by Hand

• A: (-1.5, -0.75)
• B: (-0.5, -0.75)
• C: (0.5, 1.25)
• D: (1.5, 0.25) Step 2: Covariance matrix.

• A: (-1.5, -0.75) · (0.827, 0.562) = -1.24 - 0.42 = -1.66
• B: (-0.5, -0.75) · (0.827, 0.562) = -0.41 - 0.42 = -0.83
• C: (0.5, 1.25) · (0.827, 0.562) = 0.41 + 0.70 = 1.11
• D: (1.5, 0.25) · (0.827, 0.562) = 1.24 + 0.14 = 1.38 The 4 points are now 1D values: [-1.66, -0.83, 1.11, 1.38].

10.3 Choosing the Number of Components

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
digits = load_digits()
X = digits.data  # 64 features (8×8 pixel images)
pca = PCA()
pca.fit(X)
plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
plt.plot(range(1, len(pca.explained_variance_ratio_)+1),
         pca.explained_variance_ratio_.cumsum(), 'bo-')
plt.xlabel('Number of components')
plt.ylabel('Cumulative explained variance')
plt.axhline(y=0.95, color='r', linestyle='--', label='95% variance')
plt.title('Scree Plot')
plt.grid(True)
plt.legend()
plt.subplot(1, 2, 2)
plt.bar(range(1, len(pca.explained_variance_ratio_)+1), pca.explained_variance_ratio_)
plt.xlabel('Component')
plt.ylabel('Variance explained')
plt.title('Individual Component Variance')
plt.tight_layout()
plt.show()
# How many components for 95% variance?
cumsum = np.cumsum(pca.explained_variance_ratio_)
k = np.argmax(cumsum >= 0.95) + 1
print(f"Components needed for 95% variance: {k}")

10.4 PCA for Noise Reduction

python
Run Inline
# runnable
from sklearn.datasets import load_digits
import numpy as np
digits = load_digits()
X = digits.data
np.random.seed(42)
X_noisy = X + np.random.randn(*X.shape) * 15  # Add noise
# PCA denoise
pca = PCA(n_components=32)
X_pca = pca.fit_transform(X_noisy)
X_denoised = pca.inverse_transform(X_pca)
print(f"Original noise variance: {np.var(X_noisy - X):.2f}")
print(f"Denoised residual variance: {np.var(X_denoised - X):.2f}")

10.5 t-SNE: Non-Linear Visualization

1. Compute pairwise similarities in high-D space (Gaussian distribution)
2. Compute pairwise similarities in low-D space (Student-t distribution)
3. Minimize KL divergence between the two similarity distributions

python
Run Inline
# runnable
from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
digits = load_digits()
X, y = digits.data, digits.target
tsne = TSNE(n_components=2, random_state=42, perplexity=30)
X_tsne = tsne.fit_transform(X)
plt.figure(figsize=(10, 8))
scatter = plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap='tab10', alpha=0.7)
plt.colorbar(scatter)
plt.title('t-SNE Visualization of Digits Dataset')
plt.grid(True)
plt.show()

10.6 PCA vs t-SNE

10.7 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: PCA Without Standardization

Pitfall 2: Misinterpreting t-SNE Distances

Pitfall 3: Using PCA on Non-Linear Data

📝 Practice Questions

Q1: What does PCA maximize?

PCA maximizes the variance of projected data. The first principal component is the line that maximizes the variance of the projected points. Equivalently, it minimizes the reconstruction error (sum of squared distances from points to their projections). Q2: If 3 components explain 90% of variance, how many components should you keep?

Either keep 3 (enough for 90%) or keep however many reach your threshold (95% is common). For visualization, keep 2. For downstream ML, retain enough to preserve signal — often 90-99%. Q3: Why are PCA components orthogonal?

Because the covariance matrix is symmetric. Eigenvectors of a symmetric matrix are orthogonal. This ensures each component captures unique variance not explained by previous components. Q4: What is the perplexity parameter in t-SNE?

Perplexity controls the balance between local and global aspects. It's roughly the number of nearest neighbors considered. Common range: 5-50. Lower perplexity focuses on very local structure; higher perplexity considers more global structure. The method is robust to perplexity choice between 5-50. Q5: You apply PCA and get explained variance [0.4, 0.3, 0.2, 0.1]. How many components for 95%?

Cumulative: 0.4 + 0.3 + 0.2 = 0.9 (90%). Adding the 4th: 0.9 + 0.1 = 1.0 (100%). So 4 components reach 95%. Wait — 0.9 < 0.95, so we need 4 components to exceed 95%. Q6: Can PCA be used for anomaly detection?

Yes! Project data onto top k components, then reconstruct. If reconstruction error is high for a point, it's likely an anomaly — the normal structure (captured by PCA) doesn't explain it well. This is useful for detecting unusual patterns in manufacturing, network traffic, etc. Q7: Compare PCA and LDA (Linear Discriminant Analysis).

LDA finds projections that best separate known classes. Q8: Why can't t-SNE be used for out-of-sample extension?

t-SNE is a non-parametric method — it creates a new representation for the given data points but doesn't learn a function to map new points. To add new points, you'd need to re-run t-SNE on the full dataset including new points. Some variants (parametric t-SNE) address this. Q9: What happens if you use PCA with n_components = n_features?

You'd get all components that explain 100% of the variance. The transformed data would have the same dimensionality as the input but rotated to align with the principal components. No compression happens. This is useful for decorrelating features. Q10: Your t-SNE plot shows well-separated clusters but they don't correspond to any known labels. What might be happening?

t-SNE may be finding structure you haven't thought of — perhaps there's a natural (but unknown) grouping in the data. It could also be:

1. Over-interpreting noise: t-SNE can create false clusters from random noise (especially with small perplexity)
2. Batch effects: Data collected from different batches/sensors
3. A continuous gradient: t-SNE can break a continuum into apparent clusters

Check by running t-SNE with different perplexities and random seeds. Q11: How does the reconstruction error relate to explained variance?

Reconstruction error = total variance - explained variance. If you keep k components, the reconstruction error equals the sum of the variances of the discarded (n-k) components. Lower reconstruction error = better data preservation. Q12: Is it valid to fit PCA on training data and transform test data?

Yes — and this is the correct approach:

1. Fit PCA on training data only
2. Use the same PCA transformation to transform test data
3. Never fit PCA on the full dataset before splitting — this leaks information from test to training

python
Run Inline
pca = PCA(n_components=10)
X_train_pca = pca.fit_transform(X_train)
X_test_pca = pca.transform(X_test)  # Use same transform

🔗 Cross-References

• Next Topic: Model Evaluation & Cross-Validation
• Related: Feature Selection — alternative to dimensionality reduction
• Related: Clustering — often used with PCA for visualization
• External: IITM BSCS2004 Week 10, Hands-On ML Ch. 8 Join Discord PreviousClusteringNextModel Evaluation & CV