Week 7.2: Maximum Likelihood Estimation (MLE)

622 words

3 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# Week 7.2: Maximum Likelihood Estimation (MLE) > **Prerequisites:** Independence ([Week 2.3: Independence of Random Variables](/notes/01-foundation-bsma1004-stats-2-week02-07-independence)), MME ([Week 7.1: Point Estimation — Method of Moments (MME)](/notes/01-foundation-bsma1004-stats-2-week07-20-point-estimation-...

Week 7.2: Maximum Likelihood Estimation (MLE)

Prerequisites: Independence (Week 2.3: Independence of Random Variables), MME (Week 7.1: Point Estimation — Method of Moments (MME)) Cross-links: BSMA3012 (Linear Stat Models) — MLE for regression Core question: Which parameter value makes the observed data most probable?

1. Intuition: "What value of $\theta$ best explains the data?"

The likelihood function

L(\theta) = \prod_{i=1}^n f_X(x_i; \theta)

measures how "likely" the observed data is for a given

\theta

. The maximum likelihood estimator (MLE)

\hat{\theta}_{\text{MLE}}

is the value of

\theta

that maximises

L(\theta)

— the value that makes the data most probable.

Key insight: We maximise the log-likelihood $\ell(\theta) = \ln L(\theta)$ for computational convenience — the logarithm is monotonic, so the maximiser is the same.

2. The Recipe

Algorithm:

Write the likelihood: $L(\theta) = \prod_{i=1}^n f(x_i; \theta)$ .
Take log: $\ell(\theta) = \ln L(\theta)$ .
Differentiate: $\frac{d\ell}{d\theta}$ (or partial derivatives for vector $\theta$ ).
Set to zero and solve for $\theta$ .
Verify second derivative is negative (maximum).

3. Worked Examples

Example 1: Bernoulli( $p$ )

X_i \sim \text{Bernoulli}(p)

i.i.d.

L(p) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^{\sum x_i}(1-p)^{n-\sum x_i}

\ell(p) = (\sum x_i)\ln p + (n-\sum x_i)\ln(1-p)

\frac{d\ell}{dp} = \frac{\sum x_i}{p} - \frac{n-\sum x_i}{1-p} = 0 \implies \hat{p}_{\text{MLE}} = \frac{\sum x_i}{n} = \bar{X}

Example 2: Normal $(\mu, \sigma^2)$

X_i \sim \mathcal{N}(\mu, \sigma^2)

i.i.d.

L(\mu, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i-\mu)^2}{2\sigma^2}\right) = (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{\sum (x_i-\mu)^2}{2\sigma^2}\right)

\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln\sigma^2 - \frac{1}{2\sigma^2}\sum (x_i-\mu)^2

. For $\mu$ :

\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum (x_i-\mu) = 0 \implies \hat{\mu}_{\text{MLE}} = \bar{X}

. For $\sigma^2$ :

\frac{\partial \ell}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum (x_i-\mu)^2 = 0 \implies \hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum (x_i-\bar{X})^2

. Note: MLE divides by

n

(biased), MME gave the same.

Example 3: Poisson( $\lambda$ )

X_i \sim \text{Poisson}(\lambda)

i.i.d.

L(\lambda) = \prod_{i=1}^n \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{-n\lambda}\lambda^{\sum x_i}}{\prod x_i!}

\ell(\lambda) = -n\lambda + (\sum x_i)\ln\lambda - \ln(\prod x_i!)

\frac{d\ell}{d\lambda} = -n + \frac{\sum x_i}{\lambda} = 0 \implies \hat{\lambda}_{\text{MLE}} = \frac{\sum x_i}{n} = \bar{X}

Example 4: Exponential( $\lambda$ )

X_i \sim \text{Exp}(\lambda)

i.i.d.

L(\lambda) = \prod_{i=1}^n \lambda e^{-\lambda x_i} = \lambda^n e^{-\lambda \sum x_i}

\ell(\lambda) = n\ln\lambda - \lambda\sum x_i

\frac{d\ell}{d\lambda} = \frac{n}{\lambda} - \sum x_i = 0 \implies \hat{\lambda}_{\text{MLE}} = \frac{n}{\sum x_i} = \frac{1}{\bar{X}}

4. MLE for Uniform $(0,\theta)$ : A Special Case

f_X(x; \theta) = \frac{1}{\theta}

0 \leq x \leq \theta

L(\theta) = \frac{1}{\theta^n}

for

\theta \geq \max(x_1,\dots,x_n)

0

otherwise.

L(\theta)

is maximised by the smallest possible

\theta

that makes the data possible:

\hat{\theta}_{\text{MLE}} = \max(X_1,\dots,X_n).

Note: No differentiation needed — this is a boundary problem.

5. Mermaid: MLE Flowchart

(Diagram)

6. Practice Questions

Q1 (Easy)

Find the MLE of

p

for Geometric(

p

) given

X_1,\dots,X_n

i.i.d.

Full Solution

$L(p) = \prod p(1-p)^{x_i-1} = p^n (1-p)^{\sum x_i - n}$ .

$\ell(p) = n\ln p + (\sum x_i - n)\ln(1-p)$ .

$\frac{d\ell}{dp} = \frac{n}{p} - \frac{\sum x_i - n}{1-p} = 0 \implies \hat{p} = \frac{n}{\sum x_i} = \frac{1}{\bar{X}}$ .

Q2 (Medium)

Find MLE for

\mu

\mathcal{N}(\mu, 1)

(known variance = 1).

Full Solution

$\ell(\mu) = -\frac{n}{2}\ln(2\pi) - \frac{1}{2}\sum (x_i-\mu)^2$ .

$\frac{d\ell}{d\mu} = \sum (x_i-\mu) = 0 \implies \hat{\mu} = \bar{X}$ .

Q3 (Hard)

For

X_i \sim \text{Uniform}(-\theta, \theta)

, find MLE of

\theta

Full Solution

$f_X(x; \theta) = \frac{1}{2\theta}$ for $-\theta \leq x \leq \theta$ .

$L(\theta) = \frac{1}{(2\theta)^n}$ for $\theta \geq \max(|x_1|,\dots,|x_n|)$ .

Maximising $L$ means minimising $\theta$ , so $\hat{\theta}_{\text{MLE}} = \max(|X_1|,\dots,|X_n|)$ .

Next topic: Week 8.1: Properties of Estimators — Properties of estimators (bias, variance, consistency). Join Discord PreviousWeek 7.1: Point Estimation — Method of Moments (MME)NextWeek 8.1: Properties of Estimators

What this note is really saying

Week 7.2: Maximum Likelihood Estimation (MLE)

1. Intuition: "What value of θ\thetaθ best explains the data?"

2. The Recipe

3. Worked Examples

Example 1: Bernoulli(ppp)

Example 2: Normal(μ,σ2)(\mu, \sigma^2)(μ,σ2)

Example 3: Poisson(λ\lambdaλ)

Example 4: Exponential(λ\lambdaλ)

4. MLE for Uniform(0,θ)(0,\theta)(0,θ): A Special Case

5. Mermaid: MLE Flowchart

6. Practice Questions

Q1 (Easy)

Q2 (Medium)

Q3 (Hard)

1. Intuition: "What value of $\theta$ best explains the data?"

Example 1: Bernoulli( $p$ )

Example 2: Normal $(\mu, \sigma^2)$

Example 3: Poisson( $\lambda$ )

Example 4: Exponential( $\lambda$ )

4. MLE for Uniform $(0,\theta)$ : A Special Case