Bayes' Theorem — Reversing Conditional Probability

🎯 Learning Objectives

• State and apply Bayes' theorem to compute $P(A|B)$ from $P(B|A)$
• Understand the relationship between prior, likelihood, and posterior probabilities
• Solve diagnostic testing problems (sensitivity, specificity, predictive values)
• Recognize when Bayes' theorem is needed and when it's not
• Build and interpret Bayesian updating tables

📋 Prerequisites

• Conditional Probability (13-conditional-probability) — essential foundation
• Probability Rules (12-probability-rules) — multiplication rule, law of total probability
• Basic algebra

---

📖 Core Content

15.1 Intuition: The "Reverse" Probability

Everyday analogy: You hear a car alarm at night. The alarm is very reliable (95% of car thefts trigger it). But the alarm also goes off due to other causes (wind, animals, etc.). What's the probability a car is actually being stolen given the alarm? Bayes' theorem tells you it depends on how rare car thefts are in your neighborhood. 🔑 Key Insight: Bayes' theorem lets us update our beliefs when we see new evidence. It's the mathematical formulation of learning from experience.

15.2 The Formula

Two-Event Form

$$P(A|B) = \frac{P(B|A) \times P(A)}{P(B)}$$

• $P(A)$ = prior probability (before seeing evidence B)
• $P(B|A)$ = likelihood (probability of evidence if A is true)
• $P(B)$ = marginal probability of evidence
• $P(A|B)$ = posterior probability (after updating with evidence)

Extended Form (using Law of Total Probability)

$$P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_{j=1}^{k} P(B|A_j)P(A_j)}$$

15.3 The Intuitive Approach: Table Method

• Has disease: 10,000 × 0.01 = 100
• No disease: 10,000 - 100 = 9,900 Step 3: Fill in test results using sensitivity and specificity.
• True positives: 100 × 0.95 = 95
• False negatives: 100 - 95 = 5
• True negatives: 9,900 × 0.90 = 8,910
• False positives: 9,900 - 8,910 = 990 Complete table:

$$P(\text{disease} | \text{positive}) = \frac{95}{1,085} \approx 0.0876 = 8.76\%$$

15.4 The Three Components

15.5 Bayesian Updating

15.6 Worked Examples

Example 1: Simple Bayes (Easy)

• $A_1$ = Right-handed, $A_2$ = Left-handed, $A_3$ = Ambidextrous
• $B$ = Curly hair Step 2: Priors: $P(A_1)=0.6$, $P(A_2)=0.3$, $P(A_3)=0.1$ Likelihoods: $P(B|A_1)=0.4$, $P(B|A_2)=0.8$, $P(B|A_3)=0.5$ Step 3: $P(B) = 0.4(0.6) + 0.8(0.3) + 0.5(0.1) = 0.24 + 0.24 + 0.05 = 0.53$ Step 4: $P(A_2|B) = \frac{0.8 \times 0.3}{0.53} = \frac{0.24}{0.53} \approx 0.453$ So about 45.3% of curly-haired people are left-handed.

Example 2: Medical Testing (Medium)

$$P(\text{disease} | \text{positive}) = \frac{49.5}{248.5} \approx 0.199 = 19.9\%$$ $$P(D|+) = \frac{0.99 \times 0.005}{0.99 \times 0.005 + 0.02 \times 0.995} = \frac{0.00495}{0.00495 + 0.0199} = \frac{0.00495}{0.02485} \approx 0.199$$

Example 3: Prospecting for Oil (Harder)

• $P(\text{oil}) = 0.2$ (20% chance oil is present)
• A seismic test can detect oil with 90% sensitivity
• The test has 85% specificity The test comes back positive. Should the company drill? Solution: Table method (1,000 sites):

$$P(\text{oil} | \text{positive}) = \frac{180}{300} = 0.60$$

15.7 The Relationship Between Sensitivity, Specificity, and Predictive Values

15.8 Edge Cases & Gotchas

When Prevalence Changes

• General population (prevalence = 1%): PPV ≈ 9%
• High-risk population (prevalence = 20%): PPV ≈ 92% Lesson: Bayes' theorem shows that even "accurate" tests can be poor predictors for rare conditions.

The Base Rate Fallacy

15.9 Why This Matters

• Machine learning: Naive Bayes classifiers are fast and effective
• Spam filtering: $P(\text{spam} | \text{words in email})$
• Medical diagnosis: Interpreting test results
• A/B testing: Bayesian A/B testing frameworks
• Scientific inference: Updating beliefs based on experimental data

---

📐 Key Formulas / Concepts

---

⚠️ Common Pitfalls

Pitfall 1: The Base Rate Fallacy

Pitfall 2: Forgetting the Denominator

Pitfall 3: Not Updating the Prior for Repeat Testing

---

📝 Practice Questions

Q1: Basic Bayes

</strong>

$P(A) = 0.3$, $P(B|A) = 0.7$, $P(B|\bar{A}) = 0.2$. Find $P(A|B)$.

<details> <strong>Solution</strong>

Step 1: $P(B) = P(B|A)P(A) + P(B|\bar{A})P(\bar{A}) = 0.7(0.3) + 0.2(0.7) = 0.21 + 0.14 = 0.35$

Step 2: $P(A|B) = \frac{0.7 \times 0.3}{0.35} = \frac{0.21}{0.35} = 0.6$

$\boxed{0.6}$

</details> > **Q2: Drug Testing** > > </strong> > > A drug test has 99% sensitivity and 98% specificity. 3% of employees use drugs. If an employee tests positive, what's the probability they actually use drugs? > > <details> <strong>Solution</strong> > > **Table (10,000 employees):** > >

$P(\text{drugs} | \text{positive}) = 297/491 \approx 0.605 = 60.5\%$

$\boxed{60.5\%}$

</details> > **Q3: Spam Detection** > > </strong> > > 50% of emails are spam. The word "free" appears in 80% of spam and 10% of non-spam. If an email contains "free," what's the probability it's spam? > > <details> <strong>Solution</strong> > > $P(\text{spam} | \text{"free"}) = \frac{0.80 \times 0.50}{0.80 \times 0.50 + 0.10 \times 0.50} = \frac{0.40}{0.40 + 0.05} = \frac{0.40}{0.45} = \frac{8}{9} \approx 0.889$ > > $\boxed{8/9 \approx 88.9\%}$ </details> > **Q4: Multiple Tests** > > </strong> > > Using the test from Q2 (99% sens, 98% spec, 3% prevalence), suppose an employee tests positive twice (independent tests). What's the updated probability? > > <details> <strong>Solution</strong> > > After the first test, posterior = 60.5%. This becomes the new prior. > > **Table (10,000 people, prior = 60.5%):** > >

$P(\text{drugs} | \text{positive again}) = 5,989.5/6,068.5 \approx 0.987 = 98.7\%$

After two positive tests, the probability jumps from 3% → 60.5% → 98.7%.

$\boxed{98.7\%}$

</details> > **Q5: Diagnostics Comparison** > > </strong> > > Disease A affects 10% of the population; Disease B affects 0.1%. Both have tests with 95% sensitivity and 90% specificity. Compare PPV. > > <details> <strong>Solution</strong> > > **Disease A (prevalence = 10%):** $PPV_A = \frac{0.95 \times 0.10}{0.95 \times 0.10 + 0.10 \times 0.90} = \frac{0.095}{0.095 + 0.09} = \frac{0.095}{0.185} \approx 0.514 = 51.4\%$ > > **Disease B (prevalence = 0.1%):** $PPV_B = \frac{0.95 \times 0.001}{0.95 \times 0.001 + 0.10 \times 0.999} = \frac{0.00095}{0.00095 + 0.0999} = \frac{0.00095}{0.10085} \approx 0.0094 = 0.94\%$ > > **Conclusion:** The same test has PPV of 51.4% for the common disease and 0.94% for the rare one. This shows why mass screening for rare diseases is problematic — most positive results are false positives. </details> > **Q6: Jury Problem** > > </strong> > > A crime occurs in a city of 1 million people. A DNA match is found. The DNA test falsely matches 1 in 10,000 people (0.01% false positive rate). The suspect is one person. What's the probability they're guilty given the DNA match? (Assume no other evidence.) > > <details> <strong>Solution</strong> > > Prior: $P(\text{suspect is guilty}) = 1/1,000,000$ (no other evidence, equally likely among all people) > > Test characteristics: > > - Sensitivity ≈ 100% (the true perpetrator always matches) > - False positive rate = 0.01% = 0.0001 > > $P(\text{guilty} | \text{match}) = \frac{1 \times 1/1,000,000}{1 \times 1/1,000,000 + 0.0001 \times 999,999/1,000,000}$ > > $= \frac{0.000001}{0.000001 + 0.0000999999} = \frac{0.000001}{0.0001009999} \approx 0.0099 = 0.99\%$ > > Even with a DNA match, the probability the suspect is guilty is only about 1%! This is the DNA base rate fallacy. > > $\boxed{\approx 0.99\%}$ </details> > **Q7: Manufacturing** > > </strong> > > Factory has 3 machines. Machine A makes 50% of items (2% defective), Machine B makes 30% (4% defective), Machine C makes 20% (1% defective). A random item is found defective. Probability it came from Machine B? > > <details> <strong>Solution</strong> > > $P(B|D) = \frac{P(D|B)P(B)}{P(D|A)P(A) + P(D|B)P(B) + P(D|C)P(C)}$ > > $= \frac{0.04 \times 0.30}{0.02 \times 0.50 + 0.04 \times 0.30 + 0.01 \times 0.20}$ > > $= \frac{0.012}{0.01 + 0.012 + 0.002} = \frac{0.012}{0.024} = 0.5$ > > $\boxed{0.5 \text{ or } 50\%}$ </details> > **Q8: Weather Prediction** > > </strong> > > A weather model predicts rain. Historically: > > - It actually rains on 15% of days > - When it will rain, the model predicts rain 90% of the time > - When it won't rain, the model predicts rain 10% of the time > > If the model predicts rain today, what's the probability it actually rains? > > <details> <strong>Solution</strong> > > $P(\text{rain} | \text{predicted}) = \frac{0.90 \times 0.15}{0.90 \times 0.15 + 0.10 \times 0.85} = \frac{0.135}{0.135 + 0.085} = \frac{0.135}{0.22} \approx 0.614 = 61.4\%$ > > So a rain prediction means only 61.4% chance of actual rain. > > $\boxed{61.4\%}$ </details> > **Q9: Monty Hall Problem (Bayesian Version)** > > </strong> > > In the Monty Hall problem: 3 doors, 1 car (prize), 2 goats. You pick Door 1. Monty (who knows where the car is) opens Door 3, showing a goat. Should you switch to Door 2? > > Compute $P(\text{car behind Door 2} | \text{Monty opened Door 3})$. > > <details> <strong>Solution</strong> > > **Step 1 — Priors:** $P(\text{car at 1}) = P(\text{car at 2}) = P(\text{car at 3}) = 1/3$ > > **Step 2 — Likelihood:** What's the probability Monty opens Door 3 given where the car is? > > - If car at 1: Monty opens either Door 2 or 3. $P(\text{opens 3} | \text{car at 1}) = 1/2$ > - If car at 2: Monty MUST open Door 3 (he can't open the door with the car or your pick). $P(\text{opens 3} | \text{car at 2}) = 1$ > - If car at 3: Monty can't open Door 3 (it has the car). $P(\text{opens 3} | \text{car at 3}) = 0$ > > **Step 3 — Total probability Monty opens Door 3:** $P(\text{opens 3}) = (1/2)(1/3) + (1)(1/3) + (0)(1/3) = 1/6 + 1/3 = 1/2$ > > **Step 4 — Posterior:** $P(\text{car at 2} | \text{opens 3}) = \frac{1 \times 1/3}{1/2} = \frac{1/3}{1/2} = 2/3$ > > Switching gives you 2/3 probability of winning. Stick with Door 1 gives 1/3. </details> > **Q10: Real-World Application — Rare Disease Screening** > > </strong> > > A government is considering mandatory screening for a rare disease (prevalence 0.01%). The test has 99.9% sensitivity and 99.8% specificity. The population is 100 million. > > a) How many true positives, false positives, true negatives, false negatives? b) What's the PPV? c) Should the government implement mandatory screening? Discuss. > > <details> <strong>Solution</strong> > > **a) Table (100 million people):** > > - Disease: 100M × 0.0001 = 10,000 > - No disease: 100M - 10,000 = 99,990,000 > - True positives: 10,000 × 0.999 = 9,990 > - False negatives: 10,000 - 9,990 = 10 > - True negatives: 99,990,000 × 0.998 = 99,790,020 > - False positives: 99,990,000 - 99,790,020 = 199,980 > > **b) PPV = 9,990 / (9,990 + 199,980) = 9,990 / 209,970 ≈ 0.0476 = 4.76%** > > Only 4.76% of positive results are true positives! > > **c) Discussion:** > > - **Against screening:** 199,980 people would be told they "might" have the disease, causing enormous anxiety and triggering expensive follow-up tests. Cost of follow-up for 200K people is huge. > - **For screening:** 9,990 cases detected early. The 10 false negatives are a tiny number. > - **Better approach:** Screen only high-risk populations (higher prevalence → higher PPV). Or use a two-stage approach (initial test → confirmatory test). > > $\boxed{PPV \approx 4.76\%}$ </details> * * * ## 🔗 Cross-References - **Previous:** [Conditional Probability](/notes/01-foundation-bsma1002-stats-1-week08-13-conditional-probability) — foundation for Bayes - **Week 9 (Random Variables):** Bayesian inference with continuous distributions - **Week 11 (Bernoulli & Binomial):** Bayesian inference for proportions - **BSMA1004 (Stats 2):** Bayesian statistics, conjugate priors - **BSCS2004 (ML Foundations):** Naive Bayes classifier - **BSMA1001-maths-1:** Partitions of sets (law of total probability) [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Conditional Probability**](/notes/01-foundation-bsma1002-stats-1-week08-13-conditional-probability)[Next**Discrete Random Variables**](/notes/01-foundation-bsma1002-stats-1-week09-15-discrete-random-variables)