Semi-supervised & Self-supervised Learning

🎯 Learning Objectives

• Explain why unlabeled data helps learning
• Implement self-training and pseudo-labeling
• Understand label propagation algorithms
• Apply consistency regularization

📖 Core Content

7.1 Intuition: Using Unlabeled Data as a Free Lunch

7.2 Self-Training (Pseudo-Labeling)

1. Train a model on labeled data
2. Predict on unlabeled data (pseudo-labels)
3. Add high-confidence predictions to training set
4. Retrain the model
5. Repeat until convergence

7.3 Label Propagation

7.4 Consistency Regularization

python
Run Inline
# runnable
import numpy as np
from sklearn.semi_supervised import SelfTrainingClassifier, LabelPropagation
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
iris = load_iris()
X, y = iris.data, iris.target
# Simulate: only 10% labeled
np.random.seed(42)
n_labels = 15
labeled_idx = np.random.choice(len(X), n_labels, replace=False)
y_mixed = np.full_like(y, -1)  # -1 = unlabeled
y_mixed[labeled_idx] = y[labeled_idx]
# Self-training with Random Forest
base_clf = RandomForestClassifier(random_state=42)
self_training = SelfTrainingClassifier(base_clf, threshold=0.8, verbose=False)
self_training.fit(X, y_mixed)
print(f"Self-training accuracy: {self_training.score(X, y):.3f}")
# Compare with supervised only (trained on labeled subset only)
supervised = RandomForestClassifier(random_state=42)
supervised.fit(X[labeled_idx], y[labeled_idx])
print(f"Supervised only accuracy: {supervised.score(X, y):.3f}")

📝 Practice Questions

Q1: When does semi-supervised learning help most?

When: (1) labeled data is scarce, (2) unlabeled data is abundant, (3) the data has inherent cluster structure (classes are separated). It's less helpful when classes overlap heavily or when the labeled set is already representative. Q2: What's the risk of self-training?

Confirmation bias: Early mistakes get reinforced. If the model confidently mislabels an unlabeled point, that error propagates in subsequent iterations. Mitigations: use high-confidence thresholds, ensemble methods, or mix labels. Q3: How does label propagation work algorithmically?

1. Build a k-NN graph connecting all points (edge weights = similarity)
2. Initialize: labeled points have one-hot label vectors; unlabeled = zeros
3. Iteratively update: $F_{t+1} = \alpha P F_t + (1-\alpha) Y$, where P is the transition matrix
4. Fixed labeled points at each iteration
5. Converges to a harmonic solution Join Discord PreviousAnomaly DetectionNextML System Design