Neural Sync Active

Bayes' Theorem — Reversing Conditional Probability

Registry Synced

Bayes' Theorem — Reversing Conditional Probability

2723 words

14 min read

Reading compass

Now · 🎯 Learning Objectives

Bayes' Theorem — Reversing Conditional Probability

🎯 Learning Objectives

After completing this topic, you will be able to:

State and apply Bayes' theorem to compute $P(A|B)$ from $P(B|A)$
Understand the relationship between prior, likelihood, and posterior probabilities
Solve diagnostic testing problems (sensitivity, specificity, predictive values)
Recognize when Bayes' theorem is needed and when it's not
Build and interpret Bayesian updating tables

📋 Prerequisites

Conditional Probability (13-conditional-probability) — essential foundation
Probability Rules (12-probability-rules) — multiplication rule, law of total probability
Basic algebra

📖 Core Content

15.1 Intuition: The "Reverse" Probability

Bayes' theorem answers the question: "Given that I observed an effect, what's the probability of a particular cause?" The classic example: A medical test is 95% accurate (sensitivity). You test positive. What's the probability you actually have the disease? Most people say "95%!" — and they're wrong. The actual answer depends on how rare the disease is. If the disease affects 1% of the population, a positive test means you have about a 16% chance of having the disease — not 95%. Bayes' theorem is how we correct for base rates.

Everyday analogy: You hear a car alarm at night. The alarm is very reliable (95% of car thefts trigger it). But the alarm also goes off due to other causes (wind, animals, etc.). What's the probability a car is actually being stolen given the alarm? Bayes' theorem tells you it depends on how rare car thefts are in your neighborhood. 🔑 Key Insight: Bayes' theorem lets us update our beliefs when we see new evidence. It's the mathematical formulation of learning from experience.

15.2 The Formula

Two-Event Form

P(A|B) = \frac{P(B|A) \times P(A)}{P(B)}

Where:

$P(A)$ = prior probability (before seeing evidence B)
$P(B|A)$ = likelihood (probability of evidence if A is true)
$P(B)$ = marginal probability of evidence
$P(A|B)$ = posterior probability (after updating with evidence)

Extended Form (using Law of Total Probability)

A_1, A_2, ..., A_k

form a partition of the sample space:

P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_{j=1}^{k} P(B|A_j)P(A_j)}

15.3 The Intuitive Approach: Table Method

Many find Bayes' theorem easier to understand using a contingency table approach. Example: Disease prevalence = 1%, test sensitivity = 95%, test specificity = 90%. Step 1: Assume a population of 10,000 people.

	Has Disease	No Disease	Total
Test Positive
Test Negative
Total			10,000

Step 2: Fill in the disease totals using prevalence.

Has disease: 10,000 × 0.01 = 100
No disease: 10,000 - 100 = 9,900 Step 3: Fill in test results using sensitivity and specificity.
True positives: 100 × 0.95 = 95
False negatives: 100 - 95 = 5
True negatives: 9,900 × 0.90 = 8,910
False positives: 9,900 - 8,910 = 990 Complete table:

	Has Disease	No Disease	Total
Test Positive	95	990	1,085
Test Negative	5	8,910	8,915
Total	100	9,900	10,000

Step 4: Compute $P(\text{disease} | \text{positive})$ :

P(\text{disease} | \text{positive}) = \frac{95}{1,085} \approx 0.0876 = 8.76\%

Only 8.76%! Far from the 95% sensitivity.

15.4 The Three Components

(Diagram)

Term	Name	Example
$P(A)$	Prior probability	Disease prevalence: 1%
$P(B	A)$	Likelihood
$P(B	\bar{A})$	False positive rate
$P(A	B)$	Posterior probability

15.5 Bayesian Updating

Bayes' theorem can be applied repeatedly. As new evidence comes in, today's posterior becomes tomorrow's prior. Example: After a positive test (posterior ≈ 8.76%), suppose you take a second, independent test (same sensitivity/specificity). Now the prior is 8.76%.

	Has Disease	No Disease	Total
Test Positive	8.76% × 95% = 8.32%	91.24% × 10% = 9.12%	17.44%
Test Negative	8.76% × 5% = 0.44%	91.24% × 90% = 82.12%	82.56%
Total	8.76%	91.24%	100%

New posterior:

8.32 / 17.44 \approx 47.7\%

After two positive tests, the probability jumps from 1% → 8.76% → 47.7%. This is why confirmatory testing is so important.

15.6 Worked Examples

Example 1: Simple Bayes (Easy)

Scenario: In a city, 60% are right-handed, 30% are left-handed, 10% are ambidextrous. Among left-handed people, 80% have curly hair. Among right-handed, 40% have curly hair. Among ambidextrous, 50% have curly hair. If you see a person with curly hair, what's the probability they're left-handed? Solution: Step 1: Identify the events:

$A_1$ = Right-handed, $A_2$ = Left-handed, $A_3$ = Ambidextrous
$B$ = Curly hair Step 2: Priors: $P(A_1)=0.6$ , $P(A_2)=0.3$ , $P(A_3)=0.1$ Likelihoods: $P(B|A_1)=0.4$ , $P(B|A_2)=0.8$ , $P(B|A_3)=0.5$ Step 3: $P(B) = 0.4(0.6) + 0.8(0.3) + 0.5(0.1) = 0.24 + 0.24 + 0.05 = 0.53$ Step 4: $P(A_2|B) = \frac{0.8 \times 0.3}{0.53} = \frac{0.24}{0.53} \approx 0.453$ So about 45.3% of curly-haired people are left-handed.

Example 2: Medical Testing (Medium)

Scenario: A disease affects 0.5% of the population. A test has 99% sensitivity and 98% specificity. Find the probability that someone who tests positive actually has the disease. Solution: Table method (10,000 people):

	Disease	No Disease	Total
Positive	50 × 0.99 = 49.5	9,950 × 0.02 = 199	248.5
Negative	50 × 0.01 = 0.5	9,950 × 0.98 = 9,751	9,751.5
Total	50	9,950	10,000

P(\text{disease} | \text{positive}) = \frac{49.5}{248.5} \approx 0.199 = 19.9\%

Formula:

P(D|+) = \frac{0.99 \times 0.005}{0.99 \times 0.005 + 0.02 \times 0.995} = \frac{0.00495}{0.00495 + 0.0199} = \frac{0.00495}{0.02485} \approx 0.199

Example 3: Prospecting for Oil (Harder)

Scenario: A company is considering drilling for oil at a site. Based on geological data:

$P(\text{oil}) = 0.2$ (20% chance oil is present)
A seismic test can detect oil with 90% sensitivity
The test has 85% specificity The test comes back positive. Should the company drill? Solution: Table method (1,000 sites):

	Oil	No Oil	Total
Test Positive	200 × 0.9 = 180	800 × 0.15 = 120	300
Test Negative	200 × 0.1 = 20	800 × 0.85 = 680	700
Total	200	800	1,000

P(\text{oil} | \text{positive}) = \frac{180}{300} = 0.60

Decision: Before the test, the probability was 20%. After a positive test, it's 60%. Whether to drill depends on costs (drilling cost, potential revenue). If revenue is high enough, 60% might justify drilling.

15.7 The Relationship Between Sensitivity, Specificity, and Predictive Values

Term	Definition	Formula
Sensitivity	$P(\text{positive} \	\text{disease})$
Specificity	$P(\text{negative} \	\text{no disease})$
PPV (Positive Predictive Value)	$P(\text{disease} \	\text{positive})$
NPV (Negative Predictive Value)	$P(\text{no disease} \	\text{negative})$

15.8 Edge Cases & Gotchas

When Prevalence Changes

The same test has different PPV depending on the population:

General population (prevalence = 1%): PPV ≈ 9%
High-risk population (prevalence = 20%): PPV ≈ 92% Lesson: Bayes' theorem shows that even "accurate" tests can be poor predictors for rare conditions.

The Base Rate Fallacy

Ignoring the base rate (prevalence) is one of the most common errors in statistical reasoning. Even trained professionals make this mistake. Example: A prosecutor presents evidence that a DNA match occurs with probability 1 in 10 million. The jury thinks: "That's almost certainly him!" But they forget the base rate — there are 300 million people, so about 30 people would match. The probability it's the defendant is much lower than 99.99999%.

15.9 Why This Matters

Bayes' theorem is one of the most important ideas in data science:

Machine learning: Naive Bayes classifiers are fast and effective
Spam filtering: $P(\text{spam} | \text{words in email})$
Medical diagnosis: Interpreting test results
A/B testing: Bayesian A/B testing frameworks
Scientific inference: Updating beliefs based on experimental data

📐 Key Formulas / Concepts

Concept	Formula	Use
Bayes' Theorem (simple)	$P(A	B) = \frac{P(B
Bayes (extended)	$P(A_i	B) = \frac{P(B
Positive Predictive Value	$P(D\	+) = \frac{\text{sens} \times \text{prev}}{\text{sens} \times \text{prev} + (1-\text{spec})(1-\text{prev})}$
Prior	$P(A)$	Initial belief
Posterior	$P(A	B)$
Likelihood	$P(B	A)$

⚠️ Common Pitfalls

Pitfall 1: The Base Rate Fallacy

The mistake: Thinking

P(\text{disease} | \text{positive}) = P(\text{positive} | \text{disease})

. Why it happens: The test "95% accurate" sounds like a 95% probability of being right. Correction: Always factor in the base rate. A 95% accurate test for a 1% disease gives PPV of about 16%, not 95%.

Pitfall 2: Forgetting the Denominator

The mistake: Computing

P(A|B) = P(B|A)P(A)

without dividing by

P(B)

. Why it happens: Students memorize the numerator but forget the denominator. Correction: The posterior must be a probability (≤ 1). If you computed

0.95 \times 0.01 = 0.0095

as your answer, you forgot to divide by

P(B)

Pitfall 3: Not Updating the Prior for Repeat Testing

The mistake: Using the original prior when computing the probability after a second positive test. Why it happens: It's easier to reuse the same numbers. Correction: After the first test, the posterior becomes the new prior. Bayes' theorem is iterative.

📝 Practice Questions

Q1: Basic Bayes

$P(A) = 0.3$ , $P(B|A) = 0.7$ , $P(B|\bar{A}) = 0.2$ . Find $P(A|B)$ .
<details> Solution
Step 1: $P(B) = P(B|A)P(A) + P(B|\bar{A})P(\bar{A}) = 0.7(0.3) + 0.2(0.7) = 0.21 + 0.14 = 0.35$

Step 2: $P(A|B) = \frac{0.7 \times 0.3}{0.35} = \frac{0.21}{0.35} = 0.6$

$\boxed{0.6}$

</details> > **Q2: Drug Testing** > > > > A drug test has 99% sensitivity and 98% specificity. 3% of employees use drugs. If an employee tests positive, what's the probability they actually use drugs? > > <details> Solution > > **Table (10,000 employees):** > >

	Drug User	Not User	Total
Positive	300 × 0.99 = 297	9,700 × 0.02 = 194	491
Negative	300 × 0.01 = 3	9,700 × 0.98 = 9,506	9,509
Total	300	9,700	10,000

$P(\text{drugs} | \text{positive}) = 297/491 \approx 0.605 = 60.5\%$

$\boxed{60.5\%}$

</details> > **Q3: Spam Detection** > > > > 50% of emails are spam. The word "free" appears in 80% of spam and 10% of non-spam. If an email contains "free," what's the probability it's spam? > > <details> Solution > > $P(\text{spam} | \text{"free"}) = \frac{0.80 \times 0.50}{0.80 \times 0.50 + 0.10 \times 0.50} = \frac{0.40}{0.40 + 0.05} = \frac{0.40}{0.45} = \frac{8}{9} \approx 0.889$ > > $\boxed{8/9 \approx 88.9\%}$ </details> > **Q4: Multiple Tests** > > > > Using the test from Q2 (99% sens, 98% spec, 3% prevalence), suppose an employee tests positive twice (independent tests). What's the updated probability? > > <details> Solution > > After the first test, posterior = 60.5%. This becomes the new prior. > > **Table (10,000 people, prior = 60.5%):** > >

	Drug User	Not User	Total
Positive	6,050 × 0.99 = 5,989.5	3,950 × 0.02 = 79	6,068.5
Negative	6,050 × 0.01 = 60.5	3,950 × 0.98 = 3,871	3,931.5
Total	6,050	3,950	10,000

$P(\text{drugs} | \text{positive again}) = 5,989.5/6,068.5 \approx 0.987 = 98.7\%$

After two positive tests, the probability jumps from 3% → 60.5% → 98.7%.

$\boxed{98.7\%}$

</details> > **Q5: Diagnostics Comparison** > > > > Disease A affects 10% of the population; Disease B affects 0.1%. Both have tests with 95% sensitivity and 90% specificity. Compare PPV. > > <details> Solution > > **Disease A (prevalence = 10%):** $PPV_A = \frac{0.95 \times 0.10}{0.95 \times 0.10 + 0.10 \times 0.90} = \frac{0.095}{0.095 + 0.09} = \frac{0.095}{0.185} \approx 0.514 = 51.4\%$ > > **Disease B (prevalence = 0.1%):** $PPV_B = \frac{0.95 \times 0.001}{0.95 \times 0.001 + 0.10 \times 0.999} = \frac{0.00095}{0.00095 + 0.0999} = \frac{0.00095}{0.10085} \approx 0.0094 = 0.94\%$ > > **Conclusion:** The same test has PPV of 51.4% for the common disease and 0.94% for the rare one. This shows why mass screening for rare diseases is problematic — most positive results are false positives. </details> > **Q6: Jury Problem** > > > > A crime occurs in a city of 1 million people. A DNA match is found. The DNA test falsely matches 1 in 10,000 people (0.01% false positive rate). The suspect is one person. What's the probability they're guilty given the DNA match? (Assume no other evidence.) > > <details> Solution > > Prior: $P(\text{suspect is guilty}) = 1/1,000,000$ (no other evidence, equally likely among all people) > > Test characteristics: > > - Sensitivity ≈ 100% (the true perpetrator always matches) > - False positive rate = 0.01% = 0.0001 > > $P(\text{guilty} | \text{match}) = \frac{1 \times 1/1,000,000}{1 \times 1/1,000,000 + 0.0001 \times 999,999/1,000,000}$ > > $= \frac{0.000001}{0.000001 + 0.0000999999} = \frac{0.000001}{0.0001009999} \approx 0.0099 = 0.99\%$ > > Even with a DNA match, the probability the suspect is guilty is only about 1%! This is the DNA base rate fallacy. > > $\boxed{\approx 0.99\%}$ </details> > **Q7: Manufacturing** > > > > Factory has 3 machines. Machine A makes 50% of items (2% defective), Machine B makes 30% (4% defective), Machine C makes 20% (1% defective). A random item is found defective. Probability it came from Machine B? > > <details> Solution > > $P(B|D) = \frac{P(D|B)P(B)}{P(D|A)P(A) + P(D|B)P(B) + P(D|C)P(C)}$ > > $= \frac{0.04 \times 0.30}{0.02 \times 0.50 + 0.04 \times 0.30 + 0.01 \times 0.20}$ > > $= \frac{0.012}{0.01 + 0.012 + 0.002} = \frac{0.012}{0.024} = 0.5$ > > $\boxed{0.5 \text{ or } 50\%}$ </details> > **Q8: Weather Prediction** > > > > A weather model predicts rain. Historically: > > - It actually rains on 15% of days > - When it will rain, the model predicts rain 90% of the time > - When it won't rain, the model predicts rain 10% of the time > > If the model predicts rain today, what's the probability it actually rains? > > <details> Solution > > $P(\text{rain} | \text{predicted}) = \frac{0.90 \times 0.15}{0.90 \times 0.15 + 0.10 \times 0.85} = \frac{0.135}{0.135 + 0.085} = \frac{0.135}{0.22} \approx 0.614 = 61.4\%$ > > So a rain prediction means only 61.4% chance of actual rain. > > $\boxed{61.4\%}$ </details> > **Q9: Monty Hall Problem (Bayesian Version)** > > > > In the Monty Hall problem: 3 doors, 1 car (prize), 2 goats. You pick Door 1. Monty (who knows where the car is) opens Door 3, showing a goat. Should you switch to Door 2? > > Compute $P(\text{car behind Door 2} | \text{Monty opened Door 3})$. > > <details> Solution > > **Step 1 — Priors:** $P(\text{car at 1}) = P(\text{car at 2}) = P(\text{car at 3}) = 1/3$ > > **Step 2 — Likelihood:** What's the probability Monty opens Door 3 given where the car is? > > - If car at 1: Monty opens either Door 2 or 3. $P(\text{opens 3} | \text{car at 1}) = 1/2$ > - If car at 2: Monty MUST open Door 3 (he can't open the door with the car or your pick). $P(\text{opens 3} | \text{car at 2}) = 1$ > - If car at 3: Monty can't open Door 3 (it has the car). $P(\text{opens 3} | \text{car at 3}) = 0$ > > **Step 3 — Total probability Monty opens Door 3:** $P(\text{opens 3}) = (1/2)(1/3) + (1)(1/3) + (0)(1/3) = 1/6 + 1/3 = 1/2$ > > **Step 4 — Posterior:** $P(\text{car at 2} | \text{opens 3}) = \frac{1 \times 1/3}{1/2} = \frac{1/3}{1/2} = 2/3$ > > Switching gives you 2/3 probability of winning. Stick with Door 1 gives 1/3. </details> > **Q10: Real-World Application — Rare Disease Screening** > > > > A government is considering mandatory screening for a rare disease (prevalence 0.01%). The test has 99.9% sensitivity and 99.8% specificity. The population is 100 million. > > a) How many true positives, false positives, true negatives, false negatives? b) What's the PPV? c) Should the government implement mandatory screening? Discuss. > > <details> Solution > > **a) Table (100 million people):** > > - Disease: 100M × 0.0001 = 10,000 > - No disease: 100M - 10,000 = 99,990,000 > - True positives: 10,000 × 0.999 = 9,990 > - False negatives: 10,000 - 9,990 = 10 > - True negatives: 99,990,000 × 0.998 = 99,790,020 > - False positives: 99,990,000 - 99,790,020 = 199,980 > > **b) PPV = 9,990 / (9,990 + 199,980) = 9,990 / 209,970 ≈ 0.0476 = 4.76%** > > Only 4.76% of positive results are true positives! > > **c) Discussion:** > > - **Against screening:** 199,980 people would be told they "might" have the disease, causing enormous anxiety and triggering expensive follow-up tests. Cost of follow-up for 200K people is huge. > - **For screening:** 9,990 cases detected early. The 10 false negatives are a tiny number. > - **Better approach:** Screen only high-risk populations (higher prevalence → higher PPV). Or use a two-stage approach (initial test → confirmatory test). > > $\boxed{PPV \approx 4.76\%}$ </details> * * * ## 🔗 Cross-References - **Previous:** [Conditional Probability](/notes/01-foundation-bsma1002-stats-1-week08-13-conditional-probability) — foundation for Bayes - **Week 9 (Random Variables):** Bayesian inference with continuous distributions - **Week 11 (Bernoulli & Binomial):** Bayesian inference for proportions - **BSMA1004 (Stats 2):** Bayesian statistics, conjugate priors - **BSCS2004 (ML Foundations):** Naive Bayes classifier - **BSMA1001-maths-1:** Partitions of sets (law of total probability) [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Conditional Probability**](/notes/01-foundation-bsma1002-stats-1-week08-13-conditional-probability)[Next**Discrete Random Variables**](/notes/01-foundation-bsma1002-stats-1-week09-15-discrete-random-variables)