EM Algorithm

5.1 Intuition: When Data Is Incomplete

• E-step: Compute expected log-likelihood given current parameters
• M-step: Maximize expected log-likelihood to update parameters

5.2 Gaussian Mixture Model

python
Run Inline
import numpy as np
from scipy.stats import norm
# Simple EM for 2-component Gaussian mixture
np.random.seed(42)
# Generate data
true_mu = [-2, 3]
data = np.concatenate([np.random.randn(300) + true_mu[0],
                       np.random.randn(200) + true_mu[1]])
# Initialize
mu = [-1, 1]
sigma = [1, 1]
pi = [0.5, 0.5]
for iteration in range(20):
    # E-step: responsibilities
    resp = np.zeros((len(data), 2))
    for k in range(2):
        resp[:, k] = pi[k] * norm.pdf(data, mu[k], sigma[k])
    resp /= resp.sum(axis=1, keepdims=True)
    # M-step
    Nk = resp.sum(axis=0)
    mu = [np.sum(resp[:, k] * data) / Nk[k] for k in range(2)]
    pi = Nk / len(data)
print(f"Estimated means: {mu}")