Bayes' Theorem — Reversing Conditional Probability, Base Rates & Updating (Layered)

2618 words

13 min read

2026-09-12

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

Layered chapter on Bayes' theorem: reversing conditional probabilities, partition theorem, false-positive paradox, frequency grids, and question-pattern anatomy. # Bayes' Theorem — Reversing Conditional Probability, Base Rates & Updating > **Acceptance Standard:** This chapter serves as a stand-alone textbook for an...

Bayes' Theorem — Reversing Conditional Probability, Base Rates & Updating

Acceptance Standard: This chapter serves as a stand-alone textbook for any beginner studying IIT Madras BSMA1002 (Statistics for Data Science I). It demystifies belief updating, the base-rate fallacy, medical diagnostic testing, and the Law of Total Probability with complete mathematical rigor and intuitive frequency grids.

1. Orientation & Mental Model: Reversing Cause and Effect

The Core Question: Diagnosing Causes from Observed Effects

In forward probability, we ask:

\text{"Given that someone is sick, what is the chance they test positive?"} \quad \longrightarrow \quad P(\text{Positive} \mid \text{Sick})

This is easy to measure in a medical laboratory: take 1,000 infected patients, run the test, and see how many turn positive. If 990 test positive, the test is

99\%

accurate.

However, in the real world, patients never walk into a doctor's clinic knowing they are sick. They walk in with only the test result in hand, and ask the opposite question:

\text{"Given that I tested positive, what is the actual chance that I am sick?"} \quad \longrightarrow \quad P(\text{Sick} \mid \text{Positive})

Most humans instinctively assume

P(\text{Sick} \mid \text{Positive}) = P(\text{Positive} \mid \text{Sick}) = 99\%

. This assumption is catastrophic, and mathematically false.

The Base Rate Intuition: The Rare Diamond Analogy

Suppose an AI airport scanner detects diamonds inside luggage with

99\%

accuracy (

1\%

false positive rate). The alarm sounds on a passenger's bag. Does that passenger have a

99\%

chance of carrying a diamond?

No! Because out of $1,000,000$ airline passengers, perhaps only $1$ is smuggling diamonds.
The $1\%$ false alarm rate on the $999,999$ innocent passengers will generate roughly $10,000$ false alarms.
Among the $10,001$ total alarms sounded, only $1$ person actually carries a diamond!
The true probability is $\frac{1}{10,001} \approx 0.01\%$ —not $99\%$ !

Bayes' Theorem is the exact mathematical engine that corrects for the rarity of events (the base rate) when new evidence arrives.

2. Terminology, Notation & Etymology

Term	Mathematical Symbol	Etymology / Origin	Exact Technical Meaning
Prior Probability	$P(A)$ or $P(A_i)$	Latin prior ("earlier, former, preceding").	The unconditional probability of an event or hypothesis before any new empirical evidence is observed (e.g. population disease prevalence).
Likelihood	$P(B \mid A)$	Old English geliclic ("likely, probable").	The conditional probability of observing evidence $B$ given that hypothesis $A$ is true.
Marginal Evidence	$P(B)$	Latin marginalis ("written in the margin").	The total, unconditional probability of observing evidence $B$ across all possible states of reality.
Posterior Probability	$P(A \mid B)$	Latin posterior ("coming after, latter").	The updated probability of hypothesis $A$ after observing evidence $B$ .
Partition	$\{A_1, A_2, \dots, A_k\}$	Latin partitionem ("division, portioning").	A collection of mutually exclusive ( $A_i \cap A_j = \emptyset$ ) and collectively exhaustive ( $\bigcup A_i = S$ ) events.
Sensitivity	$P(T^+ \mid D^+)$	Latin sentire ("to perceive, feel").	True Positive Rate (TPR): The probability that the test correctly detects the disease in a sick individual.
Specificity	$P(T^- \mid D^-)$	Latin specificus ("forming a particular kind").	True Negative Rate (TNR): The probability that the test correctly identifies a healthy person as negative.
False Positive Rate	$P(T^+ \mid D^-) = 1 - \text{Spec}$	—	Probability that a healthy individual incorrectly tests positive.
False Negative Rate	$P(T^- \mid D^+) = 1 - \text{Sens}$	—	Probability that an infected individual incorrectly tests negative.

3. Mathematical Foundations: The Exact Derivation

Bayes' Theorem is not an arbitrary formula; it is a direct consequence of the definition of conditional probability.

Step 1: Symmetry of the Joint Intersection

By the multiplication rule of conditional probability:

P(A \cap B) = P(A \mid B) \cdot P(B)

P(B \cap A) = P(B \mid A) \cdot P(A)

Because set intersection is commutative (

A \cap B = B \cap A

P(A \mid B) \cdot P(B) = P(B \mid A) \cdot P(A)

Step 2: The Two-Event Form

Dividing both sides by

P(B)

(assuming

P(B) > 0

\bbox[10px,border:2px solid var(--teal)]{P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)} = \frac{\text{Likelihood} \times \text{Prior}}{\text{Marginal Evidence}}}

Step 3: Expansion via the Law of Total Probability

In practice, the denominator

P(B)

is rarely given directly. We compute it by partitioning the sample space into the event occurring (

A

) and not occurring (

A^c

P(B) = P(B \cap A) + P(B \cap A^c) = P(B \mid A)P(A) + P(B \mid A^c)P(A^c)

Substituting this gives the standard Binary Diagnostic Form:

P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B \mid A)P(A) + P(B \mid A^c)P(A^c)}

Step 4: The General Partition Form

\{A_1, A_2, \dots, A_k\}

partitions the universe

S

into

k

distinct causes (e.g. 3 manufacturing machines producing bolts):

\bbox[10px,border:2px solid var(--amber)]{P(A_i \mid B) = \frac{P(B \mid A_i) \cdot P(A_i)}{\displaystyle \sum_{j=1}^k P(B \mid A_j) \cdot P(A_j)}}}

4. Worked Anatomy: Two Parallel Problem-Solving Frameworks

Let us analyze the standard IITM benchmark problem:

Prevalence: Disease $D$ affects $1\%$ of the population $\implies P(D) = 0.01$ , $P(D^c) = 0.99$ .
Sensitivity: The test is $95\%$ accurate on sick patients $\implies P(T^+ \mid D) = 0.95$ .
Specificity: The test is $90\%$ $90%$ accurate on healthy patients $\implies P(T^- \mid D^c) = 0.90$ $⟹ P (T^{-} ∣ D^{c}) = 0.90$ .
- Therefore, the False Positive Rate is $P(T^+ \mid D^c) = 1 - 0.90 = 0.10$ .

A randomly selected patient tests positive (

T^+

). What is the probability they actually have the disease?

Method 1: The Algebraic Bayes Formulation

Prior: $P(D) = 0.01$
Likelihood: $P(T^+ \mid D) = 0.95$
Denominator (Marginal Probability of Testing Positive): $P(T^+) = P(T^+ \mid D)P(D) + P(T^+ \mid D^c)P(D^c)$ $P(T^+) = (0.95 \times 0.01) + (0.10 \times 0.99) = 0.0095 + 0.0990 = 0.1085$
Posterior Probability: $P(D \mid T^+) = \frac{P(T^+ \mid D)P(D)}{P(T^+)} = \frac{0.0095}{0.1085} = \frac{95}{1085} \approx \mathbf{0.0876} \quad (8.76\%)$

Despite a

95\%

sensitive test, a person who tests positive has less than an $9\%$ chance of actually being ill!

Method 2: The 10,000-Person Frequency Grid (The Intuitive Approach)

For students who find formulas prone to arithmetic slips under exam pressure, the Natural Frequency Grid eliminates decimal confusion completely:

Assume a representative cohort of $10,000$ individuals:


Step 1: Divide by disease prevalence (1% sick, 99% healthy)
Total: 10,000 people
├── Sick (1%): 100 people
└── Healthy (99%): 9,900 people

Step 2: Apply test sensitivity (95% of sick test positive)
From 100 Sick people:
├── True Positives (95%): 95 test positive
└── False Negatives (5%): 5 test negative

Step 3: Apply test specificity (90% of healthy test negative)
From 9,900 Healthy people:
├── True Negatives (90%): 8,910 test negative
└── False Positives (10%): 990 test positive

Construct the complete

2 \times 2

Contingency Table:

Test Result	Has Disease ( $D$ )	No Disease ( $D^c$ )	Total Tested
Test Positive ( $T^+$ )	95 (True Positives)	990 (False Positives)	1,085 (All Positives)
Test Negative ( $T^-$ )	5 (False Negatives)	8,910 (True Negatives)	8,915 (All Negatives)
Total Population	100	9,900	10,000

Now answer the question directly from the table:

\text{"Of all people who tested positive (1,085), how many are actually sick (95)?"}

P(D \mid T^+) = \frac{95}{1,085} \approx \mathbf{8.76\%}

5. Failure Modes & Common Exam Pitfalls


        The Confusion Axis:
      P(T+ | D)  ≠  P(D | T+)
          │             │
          │             └─► Positive Predictive Value (Posterior)
          └───────────────► Sensitivity (Likelihood)

Pitfall 1: Base Rate Neglect (The Prosecutor's Fallacy)

Ignoring

P(D)

and treating

P(D \mid T^+) = P(T^+ \mid D)

. Whenever an exam problem specifies a rare event (e.g. disease prevalence

< 5\%

, fraud

< 1\%

), the posterior is dominated by false positives from the healthy majority.

Pitfall 2: Confusing Specificity with False Positive Rate

Specificity: Probability healthy people test negative: $P(T^- \mid D^c) = 90\%$ .
False Positive Rate: Probability healthy people test positive: $P(T^+ \mid D^c) = 1 - 0.90 = 10\%$ .
Exam blunder: Plugging $0.90$ directly into the false positive term in the denominator.

Pitfall 3: Partition Non-Exhaustiveness

When solving 3-machine factory problems (e.g. Machine A, B, C): Ensure the priors sum to

1

P(A) + P(B) + P(C) = 1

If a student misreads machine production percentages (e.g.

50\%, 30\%, 20\%

) and writes

0.5, 0.3, 0.3

, the normalization fails completely.

6. Question-Pattern Anatomy (8 Finite Exam Shapes)

Shape 1: Two-State Medical Diagnostic Testing

Cue: "Given sensitivity 98%, specificity 95%, and prevalence 0.5%, find probability of disease given positive test."
Attack: Set up the 2-event formula or 10,000-person grid. Compute $P(T^+) = (0.98)(0.005) + (0.05)(0.995)$ . Divide $(0.98)(0.005)$ by $P(T^+)$ .

Shape 2: Multi-Factory Defective Item Attribution

Cue: "Factory has 3 machines A, B, C producing 40%, 35%, 25% of bolts. Defect rates are 2%, 3%, 5%. A bolt is found defective. Find probability it came from machine C."
Attack:
1. $P(\text{Defect}) = (0.40)(0.02) + (0.35)(0.03) + (0.25)(0.05) = 0.008 + 0.0105 + 0.0125 = 0.0310$ .
2. $P(C \mid \text{Defect}) = \frac{(0.25)(0.05)}{0.0310} = \frac{0.0125}{0.0310} = \frac{25}{62} \approx 0.4032$ .

Shape 3: Sequential Updating (Two Consecutive Tests)

Cue: "A patient tests positive. A second independent test is administered and also tests positive. What is the updated probability?"
Attack: The posterior of the first test becomes the prior for the second test! $\text{Prior}_2 = P(D \mid T_1^+) = 0.0876$ Apply Bayes' Theorem again using $\text{Prior}_2$ .

Shape 4: False Negative Discovery

Cue: "What is the probability that a person who tested negative actually has the disease?"
Attack: $P(D \mid T^-) = \frac{P(T^- \mid D)P(D)}{P(T^- \mid D)P(D) + P(T^- \mid D^c)P(D^c)}$ Where $P(T^- \mid D) = 1 - \text{Sensitivity}$ , and $P(T^- \mid D^c) = \text{Specificity}$ .

Shape 5: Multiple Choice Question Answering (Guessing vs Knowing)

Cue: "A student knows the answer with probability $p$ . If they do not know, they guess from $m$ options. Given that they answered correctly, what is the probability they knew the answer?"
Attack:
- $P(\text{Know}) = p$ , $P(\text{Guess}) = 1 - p$ .
- $P(\text{Correct} \mid \text{Know}) = 1$ .
- $P(\text{Correct} \mid \text{Guess}) = \frac{1}{m}$ .
- $P(\text{Know} \mid \text{Correct}) = \frac{1 \cdot p}{1 \cdot p + \frac{1}{m}(1 - p)} = \frac{m \cdot p}{m \cdot p + (1 - p)}$ .

Shape 6: Lie Detector / Polygraph Reliability

Cue: "Suspect tells the truth with probability 90%. Polygraph correctly detects lies 80% of the time, and truth 85% of the time..."
Attack: Identify states: Truth ( $T$ ) vs Lie ( $L$ ). Identify signal: Flagged as Liar ( $F$ ). Solve $P(L \mid F)$ .

Shape 7: Finding the Threshold Prevalence for Credibility

Cue: "At what minimum disease prevalence $P(D)$ does a positive test result imply $P(D \mid T^+) > 0.5$ ?"
Attack: Set $\frac{P(T^+ \mid D)p}{P(T^+ \mid D)p + P(T^+ \mid D^c)(1-p)} > 0.5 \implies P(T^+ \mid D)p > P(T^+ \mid D^c)(1-p)$ and solve for $p$ .

Shape 8: Urn and Coin Selection Traps

Cue: "A box contains 2 fair coins and 1 two-headed coin. A coin is picked at random and flipped 3 times, showing 3 heads. What is the probability it was the two-headed coin?"
Attack:
- $P(\text{Fair}) = \frac{2}{3}, \quad P(3H \mid \text{Fair}) = (1/2)^3 = \frac{1}{8}$ .
- $P(\text{Biased}) = \frac{1}{3}, \quad P(3H \mid \text{Biased}) = 1^3 = 1$ .
- $P(\text{Biased} \mid 3H) = \frac{1 \cdot (1/3)}{1 \cdot (1/3) + (1/8)(2/3)} = \frac{1/3}{1/3 + 1/12} = \frac{4/12}{5/12} = \frac{4}{5} = 0.80$ .

7. Interactive Verification & Attempt-First Self-Checks

Question 1

Type: MCQ | Marks: 1 | Topic: 3-way partition laboratory error rate

A laboratory has three technicians: Alice, Bob, and Charlie.

Alice processes $50\%$ of all blood samples, with an error rate of $1\%$ .
Bob processes $30\%$ of all blood samples, with an error rate of $3\%$ .
Charlie processes $20\%$ of all blood samples, with an error rate of $5\%$ .

A sample is selected at random and found to have an error. What is the probability that Charlie processed this sample?

MCQ

3-way partition laboratory error rate

3 Unit Assessment

0.2000

0.4167

0.5000

0.2500

Question 2

Type: MCQ | Marks: 1 | Topic: Insurance risk classification updating

An insurance company divides drivers into two risk classes:

Safe drivers ( $70\%$ of the population), who have a $1\%$ chance of an accident in any given year.
High-risk drivers ( $30\%$ of the population), who have a $10\%$ chance of an accident in any given year.

A newly insured customer gets into an accident in their first year. What is the probability that this customer belongs to the high-risk class?

MCQ

Insurance risk classification updating

3 Unit Assessment

0.3000

0.8108

0.1892

0.9000

Question 3

Type: MCQ | Marks: 1 | Topic: False negative posterior given rare disease

A test has sensitivity

90\%

and specificity

90\%

. The base rate of the disease is

10\%

. If a patient tests negative, what is the probability that the patient actually has the disease (the False Negative Posterior)?

MCQ

False negative posterior given rare disease

3 Unit Assessment

0.1000

0.0122

0.0500

0.0889

8. Next Action & Spacing Rail

Next Invariant Chapter: Proceed to Discrete Random Variables — PMF, CDF & Properties to see how conditional probabilities map into distributions.
Machine Learning Foundations Connection: In BSCS2007 (MLF), Bayes' theorem forms the bedrock of the Naive Bayes Classifier, where conditional independence simplifies the likelihood calculation across text tokens.
Self-Check Diagnostic: Review the 10,000-person contingency table method; ensure you can reconstruct the 4-cell table from memory in under 60 seconds.

Chapter provenance: authored 2026-09-12 to the layered-chapter standard (docs/roadmaps/2026-09-12-claude-content-ecosystem.md, syntax per docs/authoring/AUTHORING_COOKBOOK.md), scoped against the BSMA1002 syllabus vault and the imported 721 - Bayes' Theorem — Reversing Conditional Probability.md (untouched; this file is its new sibling).