Quiz 2

Week 9: Bayesian Estimation

624 words
3 min read
Python Week 1: the first filter for runtime behavior
Visual companion
Python
Type and operator map

Python Week 1: the first filter for runtime behavior

Open galleryRelated surface
View
Revision summary

What this note is really saying

Short form

# Week 9: Bayesian Estimation > **Prerequisites:** MLE ([Week 7.2: Maximum Likelihood Estimation (MLE)](/notes/01-foundation-bsma1004-stats-2-week07-21-point-estimation-mle)), Conditional Distributions ([Week 1.3: Conditional Distributions](/notes/01-foundation-bsma1004-stats-2-week01-03-conditional-distributions))...

Week 9: Bayesian Estimation

Prerequisites: MLE (Week 7.2: Maximum Likelihood Estimation (MLE)), Conditional Distributions (Week 1.3: Conditional Distributions) Cross-links: BSMA3012 (Linear Stat Models) — Bayesian regression Core question: How do we incorporate prior knowledge into parameter estimation?

1. Intuition: The Bayesian Philosophy

The frequentist approach treats θ\theta as an unknown but fixed constant. The Bayesian approach treats θ\theta as a random variable with its own distribution. Bayes' theorem for parameters:
π(θ∣x)⏟Posterior=f(x∣θ)⏞Likelihood⋅π(θ)⏞Priorm(x)⏟Marginal likelihood∝Likelihood×Prior.\underbrace{\pi(\theta \mid \mathbf{x})}_{\text{Posterior}} = \frac{\overbrace{f(\mathbf{x} \mid \theta)}^{\text{Likelihood}} \cdot \overbrace{\pi(\theta)}^{\text{Prior}}}{\underbrace{m(\mathbf{x})}_{\text{Marginal likelihood}}} \propto \text{Likelihood} \times \text{Prior}.
  • Prior π(θ)\pi(\theta): Your belief about θ\theta before seeing data.
  • Posterior π(θ∣x)\pi(\theta \mid \mathbf{x}): Your updated belief after seeing data.
  • Likelihood f(x∣θ)f(\mathbf{x} \mid \theta): Same as in MLE.

2. The Bayesian Workflow

(Diagram)

3. Conjugate Priors

A prior is conjugate if the posterior belongs to the same family as the prior.
LikelihoodConjugate PriorPosterior
Bernoulli( pp )Beta( α,β\alpha,\beta )Beta( α+∑xi,β+n−∑xi\alpha+\sum x_i, \beta+n-\sum x_i )
Poisson( λ\lambda )Gamma( α,β\alpha,\beta )Gamma( α+∑xi,β+n\alpha+\sum x_i, \beta+n )
Normal( μ\mu , known σ2\sigma^2 )Normal( μ0,τ2\mu_0, \tau^2 )Normal( μ0/τ2+nxˉ/σ21/τ2+n/σ2,11/τ2+n/σ2\frac{\mu_0/\tau^2 + n\bar{x}/\sigma^2}{1/\tau^2 + n/\sigma^2}, \frac{1}{1/\tau^2 + n/\sigma^2} )
Normal(known μ\mu , σ2\sigma^2 )Inverse-Gamma( α,β\alpha,\beta )Inverse-Gamma( α+n/2,β+12∑(xi−μ)2\alpha+n/2, \beta+\frac12\sum(x_i-\mu)^2 )
Exponential( λ\lambda )Gamma( α,β\alpha,\beta )Gamma( α+n,β+∑xi\alpha+n, \beta+\sum x_i )

4. Worked Example: Bernoulli-Beta Conjugate

Model: Xi∼Bernoulli(p)X_i \sim \text{Bernoulli}(p) i.i.d. Prior: p∼Beta(α,β)p \sim \text{Beta}(\alpha,\beta) with PDF π(p)∝pα−1(1−p)β−1\pi(p) \propto p^{\alpha-1}(1-p)^{\beta-1}. Data: ∑xi\sum x_i successes in nn trials. Posterior:
π(p∣x)∝p∑xi(1−p)n−∑xi⋅pα−1(1−p)β−1=pα+∑xi−1(1−p)β+n−∑xi−1.\pi(p \mid \mathbf{x}) \propto p^{\sum x_i}(1-p)^{n-\sum x_i} \cdot p^{\alpha-1}(1-p)^{\beta-1} = p^{\alpha+\sum x_i - 1}(1-p)^{\beta+n-\sum x_i - 1}.
So p∣x∼Beta(α+∑xi,β+n−∑xi)p \mid \mathbf{x} \sim \text{Beta}(\alpha + \sum x_i, \beta + n - \sum x_i). Posterior mean (under squared error loss):
p^Bayes=α+∑xiα+β+n.\hat{p}_{\text{Bayes}} = \frac{\alpha + \sum x_i}{\alpha + \beta + n}.
This is a weighted average of the prior mean α/(α+β)\alpha/(\alpha+\beta) and the sample proportion xˉ\bar{x}.

Numerical Example

Suppose prior Beta(2,2)\text{Beta}(2,2) (prior mean 0.50.5). Observe n=10n=10, ∑xi=7\sum x_i = 7. Posterior: Beta(9,5)\text{Beta}(9, 5). Posterior mean: 914≈0.643\frac{9}{14} \approx 0.643. Sample proportion: 0.70.7. Prior mean: 0.50.5. The posterior mean (0.643)(0.643) is pulled towards the prior relative to 0.70.7.

5. Non-Informative Priors

When we have little prior information, we use non-informative (flat) priors:
  • Uniform prior: π(θ)∝1\pi(\theta) \propto 1 (improper if range unbounded)
  • Jeffreys prior: π(θ)∝I(θ)\pi(\theta) \propto \sqrt{I(\theta)} (invariant to reparameterisation) With a flat prior, the posterior mode equals the MLE.

6. Bayesian vs Frequentist

AspectFrequentistBayesian
θ\thetaFixed constantRandom variable
InferenceBased on sampling distributionBased on posterior
PriorNot usedRequired
IntervalConfidence interval (random)Credible interval (fixed given data)
Interpretation95% of intervals contain θ\theta95% probability θ\theta lies in interval

7. Practice Questions

Q1 (Easy)

Prior: p∼Beta(1,1)p \sim \text{Beta}(1,1) (uniform). Data: 3 successes in 10 trials. Find posterior.
Full Solution
Posterior: Beta(1+3,1+7)=Beta(4,8)\text{Beta}(1+3, 1+7) = \text{Beta}(4, 8). Posterior mean: 4/12=1/3≈0.3334/12 = 1/3 \approx 0.333.

Q2 (Medium)

Prior: λ∼Gamma(2,1)\lambda \sim \text{Gamma}(2, 1). Data: n=5n=5, ∑xi=15\sum x_i = 15. Find posterior for Poisson(λ\lambda).
Full Solution
Posterior: Gamma(2+15,1+5)=Gamma(17,6)\text{Gamma}(2+15, 1+5) = \text{Gamma}(17, 6). Posterior mean: 17/6≈2.8317/6 \approx 2.83.

Q3 (Hard — Normal-Normal)

Prior: μ∼N(0,4)\mu \sim \mathcal{N}(0, 4). Data: n=10n=10, xˉ=2\bar{x}=2, known σ2=1\sigma^2=1. Find posterior.
Full Solution
Posterior precision: 1/τ2+n/σ2=1/4+10/1=10.251/\tau^2 + n/\sigma^2 = 1/4 + 10/1 = 10.25. Posterior mean: 0/4+10⋅2/110.25=2010.25≈1.951\frac{0/4 + 10\cdot2/1}{10.25} = \frac{20}{10.25} \approx 1.951. Posterior variance: 1/10.25≈0.09761/10.25 \approx 0.0976.
So μ∣x∼N(1.951,0.0976)\mu \mid \mathbf{x} \sim \mathcal{N}(1.951, 0.0976).
Next topic: Week 10.1: Introduction to Hypothesis Testing — Introduction to Hypothesis Testing. Join Discord PreviousWeek 8.2: Confidence IntervalsNextWeek 10.1: Introduction to Hypothesis Testing
Document outline

Keep your place and jump directly to a heading.

Table of Contents
System Normal // Awaiting Context

Intelligence Hub

Navigate the knowledge graph to generate context. The Hub adapts dynamically to surface backlinks, related notes, and metadata insights.

Open library →Launch Assessment Engine