Quiz 2

Week 7.2: Maximum Likelihood Estimation (MLE)

622 words
3 min read
Python Week 1: the first filter for runtime behavior
Visual companion
Python
Type and operator map

Python Week 1: the first filter for runtime behavior

Open galleryRelated surface
View
Revision summary

What this note is really saying

Short form

# Week 7.2: Maximum Likelihood Estimation (MLE) > **Prerequisites:** Independence ([Week 2.3: Independence of Random Variables](/notes/01-foundation-bsma1004-stats-2-week02-07-independence)), MME ([Week 7.1: Point Estimation — Method of Moments (MME)](/notes/01-foundation-bsma1004-stats-2-week07-20-point-estimation-...

Week 7.2: Maximum Likelihood Estimation (MLE)

Prerequisites: Independence (Week 2.3: Independence of Random Variables), MME (Week 7.1: Point Estimation — Method of Moments (MME)) Cross-links: BSMA3012 (Linear Stat Models) — MLE for regression Core question: Which parameter value makes the observed data most probable?

1. Intuition: "What value of θ\theta best explains the data?"

The likelihood function L(θ)=∏i=1nfX(xi;θ)L(\theta) = \prod_{i=1}^n f_X(x_i; \theta) measures how "likely" the observed data is for a given θ\theta. The maximum likelihood estimator (MLE) θ^MLE\hat{\theta}_{\text{MLE}} is the value of θ\theta that maximises L(θ)L(\theta) — the value that makes the data most probable.
Key insight: We maximise the log-likelihood ℓ(θ)=ln⁡L(θ)\ell(\theta) = \ln L(\theta) for computational convenience — the logarithm is monotonic, so the maximiser is the same.

2. The Recipe

Algorithm:
  1. Write the likelihood: L(θ)=∏i=1nf(xi;θ)L(\theta) = \prod_{i=1}^n f(x_i; \theta).
  2. Take log: ℓ(θ)=ln⁡L(θ)\ell(\theta) = \ln L(\theta).
  3. Differentiate: dℓdθ\frac{d\ell}{d\theta} (or partial derivatives for vector θ\theta).
  4. Set to zero and solve for θ\theta.
  5. Verify second derivative is negative (maximum).

3. Worked Examples

Example 1: Bernoulli(pp)

Xi∼Bernoulli(p)X_i \sim \text{Bernoulli}(p) i.i.d. L(p)=∏i=1npxi(1−p)1−xi=p∑xi(1−p)n−∑xiL(p) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^{\sum x_i}(1-p)^{n-\sum x_i}. ℓ(p)=(∑xi)ln⁡p+(n−∑xi)ln⁡(1−p)\ell(p) = (\sum x_i)\ln p + (n-\sum x_i)\ln(1-p). dℓdp=∑xip−n−∑xi1−p=0  ⟹  p^MLE=∑xin=Xˉ\frac{d\ell}{dp} = \frac{\sum x_i}{p} - \frac{n-\sum x_i}{1-p} = 0 \implies \hat{p}_{\text{MLE}} = \frac{\sum x_i}{n} = \bar{X}.

Example 2: Normal(μ,σ2)(\mu, \sigma^2)

Xi∼N(μ,σ2)X_i \sim \mathcal{N}(\mu, \sigma^2) i.i.d. L(μ,σ2)=∏i=1n12πσ2exp⁡(−(xi−μ)22σ2)=(2πσ2)−n/2exp⁡(−∑(xi−μ)22σ2)L(\mu, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i-\mu)^2}{2\sigma^2}\right) = (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{\sum (x_i-\mu)^2}{2\sigma^2}\right). ℓ(μ,σ2)=−n2ln⁡(2π)−n2ln⁡σ2−12σ2∑(xi−μ)2\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln\sigma^2 - \frac{1}{2\sigma^2}\sum (x_i-\mu)^2. For μ\mu: ∂ℓ∂μ=1σ2∑(xi−μ)=0  ⟹  μ^MLE=Xˉ\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum (x_i-\mu) = 0 \implies \hat{\mu}_{\text{MLE}} = \bar{X}. For σ2\sigma^2: ∂ℓ∂σ2=−n2σ2+12σ4∑(xi−μ)2=0  ⟹  σ^MLE2=1n∑(xi−Xˉ)2\frac{\partial \ell}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum (x_i-\mu)^2 = 0 \implies \hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum (x_i-\bar{X})^2. Note: MLE divides by nn (biased), MME gave the same.

Example 3: Poisson(λ\lambda)

Xi∼Poisson(λ)X_i \sim \text{Poisson}(\lambda) i.i.d. L(λ)=∏i=1ne−λλxixi!=e−nλλ∑xi∏xi!L(\lambda) = \prod_{i=1}^n \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{-n\lambda}\lambda^{\sum x_i}}{\prod x_i!}. ℓ(λ)=−nλ+(∑xi)ln⁡λ−ln⁡(∏xi!)\ell(\lambda) = -n\lambda + (\sum x_i)\ln\lambda - \ln(\prod x_i!). dℓdλ=−n+∑xiλ=0  ⟹  λ^MLE=∑xin=Xˉ\frac{d\ell}{d\lambda} = -n + \frac{\sum x_i}{\lambda} = 0 \implies \hat{\lambda}_{\text{MLE}} = \frac{\sum x_i}{n} = \bar{X}.

Example 4: Exponential(λ\lambda)

Xi∼Exp(λ)X_i \sim \text{Exp}(\lambda) i.i.d. L(λ)=∏i=1nλe−λxi=λne−λ∑xiL(\lambda) = \prod_{i=1}^n \lambda e^{-\lambda x_i} = \lambda^n e^{-\lambda \sum x_i}. ℓ(λ)=nln⁡λ−λ∑xi\ell(\lambda) = n\ln\lambda - \lambda\sum x_i. dℓdλ=nλ−∑xi=0  ⟹  λ^MLE=n∑xi=1Xˉ\frac{d\ell}{d\lambda} = \frac{n}{\lambda} - \sum x_i = 0 \implies \hat{\lambda}_{\text{MLE}} = \frac{n}{\sum x_i} = \frac{1}{\bar{X}}.

4. MLE for Uniform(0,θ)(0,\theta): A Special Case

fX(x;θ)=1θf_X(x; \theta) = \frac{1}{\theta}, 0≤x≤θ0 \leq x \leq \theta. L(θ)=1θnL(\theta) = \frac{1}{\theta^n} for θ≥max⁡(x1,…,xn)\theta \geq \max(x_1,\dots,x_n), 00 otherwise. L(θ)L(\theta) is maximised by the smallest possible θ\theta that makes the data possible:
θ^MLE=max⁡(X1,…,Xn).\hat{\theta}_{\text{MLE}} = \max(X_1,\dots,X_n).
Note: No differentiation needed — this is a boundary problem.

5. Mermaid: MLE Flowchart

(Diagram)

6. Practice Questions

Q1 (Easy)

Find the MLE of pp for Geometric(pp) given X1,…,XnX_1,\dots,X_n i.i.d.
Full Solution
L(p)=∏p(1−p)xi−1=pn(1−p)∑xi−nL(p) = \prod p(1-p)^{x_i-1} = p^n (1-p)^{\sum x_i - n}.
ℓ(p)=nln⁡p+(∑xi−n)ln⁡(1−p)\ell(p) = n\ln p + (\sum x_i - n)\ln(1-p).
dℓdp=np−∑xi−n1−p=0  ⟹  p^=n∑xi=1Xˉ\frac{d\ell}{dp} = \frac{n}{p} - \frac{\sum x_i - n}{1-p} = 0 \implies \hat{p} = \frac{n}{\sum x_i} = \frac{1}{\bar{X}}.

Q2 (Medium)

Find MLE for μ\mu of N(μ,1)\mathcal{N}(\mu, 1) (known variance = 1).
Full Solution
ℓ(μ)=−n2ln⁡(2π)−12∑(xi−μ)2\ell(\mu) = -\frac{n}{2}\ln(2\pi) - \frac{1}{2}\sum (x_i-\mu)^2.
dℓdμ=∑(xi−μ)=0  ⟹  μ^=Xˉ\frac{d\ell}{d\mu} = \sum (x_i-\mu) = 0 \implies \hat{\mu} = \bar{X}.

Q3 (Hard)

For Xi∼Uniform(−θ,θ)X_i \sim \text{Uniform}(-\theta, \theta), find MLE of θ\theta.
Full Solution
fX(x;θ)=12θf_X(x; \theta) = \frac{1}{2\theta} for −θ≤x≤θ-\theta \leq x \leq \theta.
L(θ)=1(2θ)nL(\theta) = \frac{1}{(2\theta)^n} for θ≥max⁡(∣x1∣,…,∣xn∣)\theta \geq \max(|x_1|,\dots,|x_n|).
Maximising LL means minimising θ\theta, so θ^MLE=max⁡(∣X1∣,…,∣Xn∣)\hat{\theta}_{\text{MLE}} = \max(|X_1|,\dots,|X_n|).
Next topic: Week 8.1: Properties of Estimators — Properties of estimators (bias, variance, consistency). Join Discord PreviousWeek 7.1: Point Estimation — Method of Moments (MME)NextWeek 8.1: Properties of Estimators
Document outline

Keep your place and jump directly to a heading.

Table of Contents
System Normal // Awaiting Context

Intelligence Hub

Navigate the knowledge graph to generate context. The Hub adapts dynamically to surface backlinks, related notes, and metadata insights.

Open library →Launch Assessment Engine