Week 8.1: Properties of Estimators

Prerequisites: MME (Week 7.1: Point Estimation — Method of Moments (MME)), MLE (Week 7.2: Maximum Likelihood Estimation (MLE)) Cross-links: BSMA3012 (Linear Stat Models) — estimator evaluation Core question: What makes a "good" estimator?

1. Intuition: Evaluating Our Guesses

1. Bias: Is it correct on average? $\text{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta$.
2. Variance: How much does it fluctuate? $\operatorname{Var}(\hat{\theta})$.
3. Consistency: Does it converge to the truth as $n \to \infty$?

2. Bias and Unbiasedness

Definition (Bias) $\text{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta$. $\hat{\theta}$ is unbiased if $\text{Bias}(\hat{\theta}) = 0$, i.e., $\mathbb{E}[\hat{\theta}] = \theta$. Examples:

• $\bar{X}$ is unbiased for $\mu$: $\mathbb{E}[\bar{X}] = \mu$.
• $S^2 = \frac{1}{n-1}\sum (X_i-\bar{X})^2$ is unbiased for $\sigma^2$.
• $\hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum (X_i-\bar{X})^2$ is biased: $\mathbb{E}[\hat{\sigma}^2_{\text{MLE}}] = \frac{n-1}{n}\sigma^2$. Bias correction: Multiply MLE variance by $\frac{n}{n-1}$ to get unbiased estimate.

3. Variance and MSE

Definition (Mean Squared Error)

4. Consistency

Definition (Consistency) $\hat{\theta}_n$ is consistent if $\hat{\theta}_n \xrightarrow{p} \theta$ as $n \to \infty$ (converges in probability). Sufficient condition: If $\text{Bias}(\hat{\theta}_n) \to 0$ and $\operatorname{Var}(\hat{\theta}_n) \to 0$, then $\hat{\theta}_n$ is consistent (by Chebyshev). Examples:

• $\bar{X}$ is consistent for $\mu$ (by LLN).
• MLEs are generally consistent under regularity conditions.

5. Efficiency

Definition (Efficiency) $\hat{\theta}_1$ is more efficient than $\hat{\theta}_2$ if $\operatorname{Var}(\hat{\theta}_1) < \operatorname{Var}(\hat{\theta}_2)$ (both unbiased). Cramér-Rao Lower Bound (CRLB): For unbiased estimators:

6. Summary Table

7. Practice Questions

Q1 (Easy)

Full Solution

Yes. $\mathbb{E}[S^2] = \sigma^2$. This is why we divide by $n-1$ — to correct the bias of the MLE.

Q2 (Medium)

Full Solution

$\mathbb{E}[\bar{X}] = \mu$ (unbiased), $\operatorname{Var}(\bar{X}) = \sigma^2/n \to 0$ as $n \to \infty$.

By Chebyshev: $\mathbb{P}(|\bar{X} - \mu| \geq \epsilon) \leq \frac{\sigma^2}{n\epsilon^2} \to 0$. So $\bar{X} \xrightarrow{p} \mu$.

Q3 (Hard)

Full Solution

CDF of max: $F_{X_{(n)}}(x) = (x/\theta)^n$, $0 \leq x \leq \theta$.

PDF: $f_{X_{(n)}}(x) = n x^{n-1}/\theta^n$.

$\mathbb{E}[X_{(n)}] = \int_0^\theta x \cdot n x^{n-1}/\theta^n \, dx = \frac{n}{\theta^n} \cdot \frac{\theta^{n+1}}{n+1} = \frac{n}{n+1}\theta$.

So $\text{Bias} = \frac{n}{n+1}\theta - \theta = -\frac{\theta}{n+1}$ (negatively biased).

Unbiased estimator: $\hat{\theta}_{\text{unbiased}} = \frac{n+1}{n} X_{(n)}$.

Next topic: Week 8.2: Confidence Intervals — Confidence Intervals. Join Discord PreviousWeek 7.2: Maximum Likelihood Estimation (MLE)NextWeek 8.2: Confidence Intervals