Quiz 2

Week 7.1: Point Estimation — Method of Moments (MME)

543 words
3 min read
Python Week 1: the first filter for runtime behavior
Visual companion
Python
Type and operator map

Python Week 1: the first filter for runtime behavior

Open galleryRelated surface
View
Revision summary

What this note is really saying

Short form

# Week 7.1: Point Estimation — Method of Moments (MME) > **Prerequisites:** Expectation ([Week 3.1: Expected Value](/notes/01-foundation-bsma1004-stats-2-week03-08-expectation)), Sampling Distributions ([Week 6.2: Sampling Distributions](/notes/01-foundation-bsma1004-stats-2-week06-19-sampling-distributions)) > **Cr...

Week 7.1: Point Estimation — Method of Moments (MME)

Prerequisites: Expectation (Week 3.1: Expected Value), Sampling Distributions (Week 6.2: Sampling Distributions) Cross-links: BSMA3012 (Linear Stat Models) — GMM Core question: How do we estimate a parameter by matching sample moments to population moments?

1. Intuition: Matching What We Observe

The method of moments (MME) is the oldest and simplest parameter estimation technique. Idea: If a distribution has kk unknown parameters, compute the first kk sample moments and set them equal to the population moments (which are functions of the parameters). Solve for the parameters.
Sample moment≈Population moment\text{Sample moment} \approx \text{Population moment}

2. The Recipe

Algorithm:
  1. Identify the unknown parameters θ1,…,θk\theta_1, \dots, \theta_k.
  2. Compute population moments μj′=E[Xj]\mu_j' = \mathbb{E}[X^j] as functions of θ\theta.
  3. Compute sample moments μ^j′=1n∑i=1nXij\hat{\mu}_j' = \frac{1}{n}\sum_{i=1}^n X_i^j.
  4. Set μj′=μ^j′\mu_j' = \hat{\mu}_j' for j=1,…,kj = 1,\dots,k and solve.

3. Worked Examples

Example 1: MME for Normal(μ,σ2)(\mu, \sigma^2)

Parameters: μ\mu, σ2\sigma^2 (2 parameters → need 2 moments). Population moments: μ1′=μ\mu_1' = \mu, μ2′=μ2+σ2\mu_2' = \mu^2 + \sigma^2. Sample moments: μ^1′=Xˉ\hat{\mu}_1' = \bar{X}, μ^2′=1n∑Xi2\hat{\mu}_2' = \frac{1}{n}\sum X_i^2. Solve:
Xˉ=μ  ⟹  μ^MME=Xˉ1n∑Xi2=μ2+σ2  ⟹  σ^MME2=1n∑Xi2−Xˉ2=1n∑(Xi−Xˉ)2.\begin{aligned} \bar{X} &= \mu \implies \hat{\mu}_{\text{MME}} = \bar{X} \\ \frac{1}{n}\sum X_i^2 &= \mu^2 + \sigma^2 \implies \hat{\sigma}^2_{\text{MME}} = \frac{1}{n}\sum X_i^2 - \bar{X}^2 = \frac{1}{n}\sum (X_i - \bar{X})^2. \end{aligned}
Note: This is the biased sample variance (divides by nn, not n−1n-1).

Example 2: MME for Poisson(λ)(\lambda)

Parameter: λ\lambda (1 parameter → need 1 moment). Population moment: E[X]=λ\mathbb{E}[X] = \lambda. Sample moment: Xˉ=1n∑Xi\bar{X} = \frac{1}{n}\sum X_i. MME: λ^MME=Xˉ\hat{\lambda}_{\text{MME}} = \bar{X}.

Example 3: MME for Uniform(0,θ)(0, \theta)

Parameter: θ\theta (1 parameter). Population moment: E[X]=θ/2\mathbb{E}[X] = \theta/2. Sample moment: Xˉ\bar{X}. Solve: Xˉ=θ/2  ⟹  θ^MME=2Xˉ\bar{X} = \theta/2 \implies \hat{\theta}_{\text{MME}} = 2\bar{X}.

Example 4: MME for Gamma(α,β)(\alpha, \beta)

Parameters: α\alpha, β\beta (2 parameters). Population moments: E[X]=α/β\mathbb{E}[X] = \alpha/\beta, E[X2]=α(α+1)/β2=(α/β)2+α/β2\mathbb{E}[X^2] = \alpha(\alpha+1)/\beta^2 = (\alpha/\beta)^2 + \alpha/\beta^2. Sample moments: Xˉ\bar{X}, 1n∑Xi2\frac{1}{n}\sum X_i^2. Solve:
Xˉ=αβ1n∑Xi2=α(α+1)β2\begin{aligned} \bar{X} &= \frac{\alpha}{\beta} \\ \frac{1}{n}\sum X_i^2 &= \frac{\alpha(\alpha+1)}{\beta^2} \end{aligned}
From first: α=βXˉ\alpha = \beta\bar{X}. Substitute into second:
1n∑Xi2=βXˉ(βXˉ+1)β2=Xˉ2+Xˉβ.\frac{1}{n}\sum X_i^2 = \frac{\beta\bar{X}(\beta\bar{X}+1)}{\beta^2} = \bar{X}^2 + \frac{\bar{X}}{\beta}.
Thus 1n∑Xi2−Xˉ2=Xˉβ  ⟹  β^MME=Xˉ1n∑Xi2−Xˉ2=XˉSn2\frac{1}{n}\sum X_i^2 - \bar{X}^2 = \frac{\bar{X}}{\beta} \implies \hat{\beta}_{\text{MME}} = \frac{\bar{X}}{\frac{1}{n}\sum X_i^2 - \bar{X}^2} = \frac{\bar{X}}{S_n^2}. And α^MME=βXˉ=Xˉ2Sn2\hat{\alpha}_{\text{MME}} = \beta\bar{X} = \frac{\bar{X}^2}{S_n^2}.

4. Practice Questions

Q1 (Easy)

Find the MME of pp for Bernoulli(pp) distribution.
Full Solution
E[X]=p\mathbb{E}[X] = p. Sample mean Xˉ=p^MME\bar{X} = \hat{p}_{\text{MME}}. So p^=Xˉ\hat{p} = \bar{X} (the sample proportion).

Q2 (Medium)

Find the MME of λ\lambda for Exponential(λ\lambda).
Full Solution
E[X]=1/λ\mathbb{E}[X] = 1/\lambda. So Xˉ=1/λ  ⟹  λ^MME=1/Xˉ\bar{X} = 1/\lambda \implies \hat{\lambda}_{\text{MME}} = 1/\bar{X}.

Q3 (Hard)

Find MME for Uniform(a,b)(a,b) (two parameters).
Full Solution
Population: E[X]=(a+b)/2\mathbb{E}[X] = (a+b)/2, Var⁡(X)=(b−a)2/12\operatorname{Var}(X) = (b-a)^2/12.
Sample: Xˉ\bar{X}, Sn2=1n∑(Xi−Xˉ)2S_n^2 = \frac{1}{n}\sum (X_i-\bar{X})^2.
Solve:
>a+b2=Xˉ  ⟹  a+b=2Xˉ(b−a)212=Sn2  ⟹  b−a=12Sn2=23Sn.>> \begin{aligned} \frac{a+b}{2} &= \bar{X} \implies a+b = 2\bar{X} \\ \frac{(b-a)^2}{12} &= S_n^2 \implies b-a = \sqrt{12 S_n^2} = 2\sqrt{3} S_n. \end{aligned} >
Thus a^=Xˉ−3Sn\hat{a} = \bar{X} - \sqrt{3} S_n, b^=Xˉ+3Sn\hat{b} = \bar{X} + \sqrt{3} S_n.
Next topic: Week 7.2: Maximum Likelihood Estimation (MLE) — Maximum Likelihood Estimation. Join Discord PreviousWeek 6.2: Sampling DistributionsNextWeek 7.2: Maximum Likelihood Estimation (MLE)
Document outline

Keep your place and jump directly to a heading.

Table of Contents
System Normal // Awaiting Context

Intelligence Hub

Navigate the knowledge graph to generate context. The Hub adapts dynamically to surface backlinks, related notes, and metadata insights.

Open library →Launch Assessment Engine