Neural Sync Active

Week 9: Bayesian Estimation

Registry Synced

Week 9: Bayesian Estimation

624 words

3 min read

Reading compass

Now · 1. Intuition: The Bayesian Philosophy

Week 9: Bayesian Estimation

Prerequisites: MLE (Week 7.2: Maximum Likelihood Estimation (MLE)), Conditional Distributions (Week 1.3: Conditional Distributions) Cross-links: BSMA3012 (Linear Stat Models) — Bayesian regression Core question: How do we incorporate prior knowledge into parameter estimation?

1. Intuition: The Bayesian Philosophy

The frequentist approach treats

\theta

as an unknown but fixed constant. The Bayesian approach treats

\theta

as a random variable with its own distribution. Bayes' theorem for parameters:

\underbrace{\pi(\theta \mid \mathbf{x})}_{\text{Posterior}} = \frac{\overbrace{f(\mathbf{x} \mid \theta)}^{\text{Likelihood}} \cdot \overbrace{\pi(\theta)}^{\text{Prior}}}{\underbrace{m(\mathbf{x})}_{\text{Marginal likelihood}}} \propto \text{Likelihood} \times \text{Prior}.

Prior $\pi(\theta)$ : Your belief about $\theta$ before seeing data.
Posterior $\pi(\theta \mid \mathbf{x})$ : Your updated belief after seeing data.
Likelihood $f(\mathbf{x} \mid \theta)$ : Same as in MLE.

2. The Bayesian Workflow

(Diagram)

3. Conjugate Priors

A prior is conjugate if the posterior belongs to the same family as the prior.

Likelihood	Conjugate Prior	Posterior
Bernoulli( $p$ )	Beta( $\alpha,\beta$ )	Beta( $\alpha+\sum x_i, \beta+n-\sum x_i$ )
Poisson( $\lambda$ )	Gamma( $\alpha,\beta$ )	Gamma( $\alpha+\sum x_i, \beta+n$ )
Normal( $\mu$ , known $\sigma^2$ )	Normal( $\mu_0, \tau^2$ )	Normal( $\frac{\mu_0/\tau^2 + n\bar{x}/\sigma^2}{1/\tau^2 + n/\sigma^2}, \frac{1}{1/\tau^2 + n/\sigma^2}$ )
Normal(known $\mu$ , $\sigma^2$ )	Inverse-Gamma( $\alpha,\beta$ )	Inverse-Gamma( $\alpha+n/2, \beta+\frac12\sum(x_i-\mu)^2$ )
Exponential( $\lambda$ )	Gamma( $\alpha,\beta$ )	Gamma( $\alpha+n, \beta+\sum x_i$ )

4. Worked Example: Bernoulli-Beta Conjugate

Model:

X_i \sim \text{Bernoulli}(p)

i.i.d. Prior:

p \sim \text{Beta}(\alpha,\beta)

with PDF

\pi(p) \propto p^{\alpha-1}(1-p)^{\beta-1}

. Data:

\sum x_i

successes in

n

trials. Posterior:

\pi(p \mid \mathbf{x}) \propto p^{\sum x_i}(1-p)^{n-\sum x_i} \cdot p^{\alpha-1}(1-p)^{\beta-1} = p^{\alpha+\sum x_i - 1}(1-p)^{\beta+n-\sum x_i - 1}.

p \mid \mathbf{x} \sim \text{Beta}(\alpha + \sum x_i, \beta + n - \sum x_i)

. Posterior mean (under squared error loss):

\hat{p}_{\text{Bayes}} = \frac{\alpha + \sum x_i}{\alpha + \beta + n}.

This is a weighted average of the prior mean

\alpha/(\alpha+\beta)

and the sample proportion

\bar{x}

Numerical Example

Suppose prior

\text{Beta}(2,2)

(prior mean

0.5

). Observe

n=10

\sum x_i = 7

. Posterior:

\text{Beta}(9, 5)

. Posterior mean:

\frac{9}{14} \approx 0.643

. Sample proportion:

0.7

. Prior mean:

0.5

. The posterior mean

(0.643)

is pulled towards the prior relative to

0.7

5. Non-Informative Priors

When we have little prior information, we use non-informative (flat) priors:

Uniform prior: $\pi(\theta) \propto 1$ (improper if range unbounded)
Jeffreys prior: $\pi(\theta) \propto \sqrt{I(\theta)}$ (invariant to reparameterisation) With a flat prior, the posterior mode equals the MLE.

6. Bayesian vs Frequentist

Aspect	Frequentist	Bayesian
$\theta$	Fixed constant	Random variable
Inference	Based on sampling distribution	Based on posterior
Prior	Not used	Required
Interval	Confidence interval (random)	Credible interval (fixed given data)
Interpretation	95% of intervals contain $\theta$	95% probability $\theta$ lies in interval

7. Practice Questions

Q1 (Easy)

Prior:

p \sim \text{Beta}(1,1)

(uniform). Data: 3 successes in 10 trials. Find posterior.

Full Solution

Posterior: $\text{Beta}(1+3, 1+7) = \text{Beta}(4, 8)$ . Posterior mean: $4/12 = 1/3 \approx 0.333$ .

Q2 (Medium)

Prior:

\lambda \sim \text{Gamma}(2, 1)

. Data:

n=5

\sum x_i = 15

. Find posterior for Poisson(

\lambda

Full Solution

Posterior: $\text{Gamma}(2+15, 1+5) = \text{Gamma}(17, 6)$ . Posterior mean: $17/6 \approx 2.83$ .

Q3 (Hard — Normal-Normal)

Prior:

\mu \sim \mathcal{N}(0, 4)

. Data:

n=10

\bar{x}=2

, known

\sigma^2=1

. Find posterior.

Full Solution

Posterior precision: $1/\tau^2 + n/\sigma^2 = 1/4 + 10/1 = 10.25$ . Posterior mean: $\frac{0/4 + 10\cdot2/1}{10.25} = \frac{20}{10.25} \approx 1.951$ . Posterior variance: $1/10.25 \approx 0.0976$ .

So $\mu \mid \mathbf{x} \sim \mathcal{N}(1.951, 0.0976)$ .

Next topic: Week 10.1: Introduction to Hypothesis Testing — Introduction to Hypothesis Testing. Join Discord PreviousWeek 8.2: Confidence IntervalsNextWeek 10.1: Introduction to Hypothesis Testing