Week 8: Conditional Probability & Bayes’ Theorem

3594 words

18 min read

2026-09-12T00:00:00.000Z

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

Conditional probability, multiplication rule, independent events, law of total probability, and Bayes’ Theorem. # Statistics for Data Science I — Week 8: Conditional Probability & Bayes’ Theorem IITM’s Week 8 covers **conditional probability, the multiplication rule, independence, the law of total probability, and B...

Statistics for Data Science I — Week 8: Conditional Probability & Bayes’ Theorem

IITM’s Week 8 covers conditional probability, the multiplication rule, independence, the law of total probability, and Bayes’ theorem.

TL;DR

Week 7 asked:

P(A)

Week 8 asks:

\boxed{ P(A\mid B) }

meaning:

What is the probability of $A$ , given that $B$ is already known to have happened?

The most important formulas are:

\boxed{ P(A\mid B) = \frac{P(A\cap B)}{P(B)} }

\boxed{ P(A\cap B) = P(A\mid B)P(B) }

For independent events:

\boxed{ P(A\cap B)=P(A)P(B) }

Law of total probability:

\boxed{ P(A) = \sum_i P(A\mid B_i)P(B_i) }

Bayes:

\boxed{ P(B_i\mid A) = \frac{P(A\mid B_i)P(B_i)} {P(A)} }

1. Conditional probability

Conditional probability asks:

What is the probability of event $A$ once we know that event $B$ occurred?

Notation:

\boxed{ P(A\mid B) }

Read:

probability of A given B.

2. Conditional probability formula

If:

P(B)>0,

then:

\boxed{ P(A\mid B) = \frac{P(A\cap B)}{P(B)} }

The denominator changes from the whole sample space to event

B

3. Core intuition

Originally:

S

is the universe.

Once we know

B

happened, we restrict attention to:

B.

Then we ask:

Inside $B$ , what fraction also belongs to $A$ ?

So:

\boxed{ P(A\mid B) = \frac{\text{A and B}} {\text{B}} }

4. Example from a table

Suppose:

	Pass	Fail	Total
Group A	40	10	50
Group B	30	20	50
Total	70	30	100

Find:

P(\text{Pass}\mid\text{Group A}).

Among Group A students:

50

total.

Of those:

40

passed.

Therefore:

P(\text{Pass}\mid\text{Group A}) = \frac{40}{50} = \boxed{0.8}.

5. Reverse conditional

Now find:

P(\text{Group A}\mid\text{Pass}).

Among students who passed:

70

total.

Of those:

40

are in Group A.

Therefore:

P(\text{Group A}\mid\text{Pass}) = \frac{40}{70} = \boxed{\frac47}.

These are not equal.

6. Critical distinction

In general:

\boxed{ P(A\mid B)\neq P(B\mid A) }

This is one of the most important probability distinctions.

The conditioning event determines the denominator.

7. Denominator rule

For:

P(A\mid B),

ask:

Given what?

The answer is:

B.

So the denominator is:

P(B).

For:

P(B\mid A),

the denominator becomes:

P(A).

8. Deriving the multiplication rule

From:

P(A\mid B) = \frac{P(A\cap B)}{P(B)},

multiply by:

P(B).

Then:

\boxed{ P(A\cap B) = P(A\mid B)P(B) }

9. Equivalent version

Similarly:

P(B\mid A) = \frac{P(A\cap B)}{P(A)}.

Therefore:

\boxed{ P(A\cap B) = P(B\mid A)P(A) }

Hence:

\boxed{ P(A\mid B)P(B) = P(B\mid A)P(A) }

This identity leads directly to Bayes’ theorem.

10. Multiplication rule intuition

To obtain:

A\cap B,

think sequentially:

event $B$ happens
given $B$ , event $A$ happens

So:

P(B)\times P(A\mid B).

This is the probability analogue of the multiplication principle in counting.

11. Example

Suppose:

P(B)=0.4

and:

P(A\mid B)=0.7.

Then:

P(A\cap B) = 0.4\times0.7 = \boxed{0.28}.

12. Three-event multiplication rule

For three events:

\boxed{ P(A\cap B\cap C) = P(A) P(B\mid A) P(C\mid A\cap B) }

This extends naturally to longer sequences.

13. Tree-diagram intuition

Suppose:

first event has branches $A,A^c$
after each branch, another event $B,B^c$

Then a complete path probability is obtained by multiplication.

Conceptually:

For example:

P(A\cap B) = P(A)P(B\mid A).

14. Independence

Two events are independent if knowing one occurred gives no information about whether the other occurs.

Formally:

\boxed{ P(A\mid B)=P(A) }

provided:

P(B)>0.

Likewise:

P(B\mid A)=P(B).

15. Independence multiplication criterion

For independent events:

\boxed{ P(A\cap B) = P(A)P(B) }

This is often the easiest condition to test.

16. Example: independent events

Toss a fair coin and roll a fair die.

Let:

A=\{\text{Head}\}

and:

B=\{\text{die shows 6}\}.

Then:

P(A)=\frac12

P(B)=\frac16.

The coin result does not affect the die result.

Therefore:

P(A\cap B) = \frac12\cdot\frac16 = \boxed{\frac1{12}}.

17. Testing independence numerically

Suppose:

P(A)=0.5,

P(B)=0.4,

and:

P(A\cap B)=0.2.

Check:

P(A)P(B) = 0.5\cdot0.4 = 0.2.

Therefore:

\boxed{A\text{ and }B\text{ are independent}}

18. Example of dependence

Suppose:

P(A)=0.5

P(B)=0.4

but:

P(A\cap B)=0.3.

If independent, intersection would have been:

0.5\cdot0.4=0.2.

Since:

0.3\neq0.2,

the events are dependent.

19. Independent vs mutually exclusive

These are very different concepts.

Mutually exclusive

A\cap B=\varnothing

They cannot occur together.

Independent

Occurrence of one does not affect probability of the other.

If both events have positive probability, mutually exclusive events are not independent.

20. Why mutually exclusive events are dependent

Suppose:

P(A)>0,\qquad P(B)>0

and:

A\cap B=\varnothing.

Then:

P(A\mid B)=0.

But:

P(A)>0.

Therefore:

P(A\mid B)\neq P(A).

So they are dependent.

21. Example

One die roll.

Let:

A=\{1\}

and:

B=\{2\}.

They are mutually exclusive.

If you know

B

happened, probability of

A

is:

0.

So occurrence of B absolutely affects A.

22. Independence of complements

A

and

B

are independent, then so are:

A^c\text{ and }B

A\text{ and }B^c

A^c\text{ and }B^c.

For example:

P(A^c\cap B) = P(A^c)P(B).

23. Repeated independent trials

Suppose each trial succeeds with probability:

p.

If trials are independent, probability all

n

succeed:

\boxed{ p^n }

Probability all fail:

\boxed{ (1-p)^n }

24. At least one success

Using complement:

P(\text{at least one success}) = 1-P(\text{no successes}).

For independent trials:

\boxed{ P(\text{at least one}) = 1-(1-p)^n }

25. Example

A machine succeeds with probability:

0.9

on each independent attempt.

For 3 attempts:

Probability at least one succeeds:

1-(0.1)^3

=1-0.001

=\boxed{0.999}.

26. Conditional probability with cards

Suppose one card is drawn from a standard deck.

Let:

A=\{\text{King}\}

B=\{\text{Face card}\}.

There are:

12

face cards.

There are:

4

kings.

All kings are face cards.

Therefore:

P(A\mid B) = \frac4{12} = \boxed{\frac13}.

27. Conditioning can increase probability

Unconditional:

P(\text{King})=\frac4{52}=\frac1{13}.

Given face card:

P(\text{King}\mid\text{Face}) = \frac13.

Knowing extra information changed the probability dramatically.

That's the entire point of conditional probability.

28. Conditioning can decrease probability

Suppose:

A=\{\text{King}\}

and:

B=\{\text{Heart}\}.

Then:

P(A)=\frac1{13}

and:

P(A\mid B)=\frac1{13}.

Here it stays unchanged because suit and rank are independent in a standard deck.

But other conditions can reduce the probability.

29. Partitions

A collection:

B_1,B_2,\dots,B_n

forms a partition of the sample space if:

they are mutually exclusive
their union is the entire sample space

Formally:

B_i\cap B_j=\varnothing

for:

i\neq j

and:

\bigcup_i B_i=S.

30. Partition intuition

Imagine dividing the sample space into non-overlapping exhaustive cases:


text
S
+-------------------------+
| B₁ | B₂ | B₃ | B₄      |
+-------------------------+

Every outcome belongs to exactly one partition component.

Partitions are the foundation of total probability.

31. Law of total probability

Suppose:

B_1,\dots,B_n

form a partition.

Then for any event

A

\boxed{ P(A) = \sum_{i=1}^{n} P(A\mid B_i)P(B_i) }

32. Why total probability works

Because:

A = (A\cap B_1) \cup (A\cap B_2) \cup \cdots \cup (A\cap B_n).

These pieces are mutually exclusive.

Therefore:

P(A) = \sum_iP(A\cap B_i).

Using the multiplication rule:

P(A\cap B_i) = P(A\mid B_i)P(B_i).

Hence the formula.

33. Two-case total probability

A particularly common partition is:

B,\quad B^c.

Then:

\boxed{ P(A) = P(A\mid B)P(B) + P(A\mid B^c)P(B^c) }

This form appears constantly.

34. Example: defective products

Suppose a factory uses two machines.

Machine 1 makes:

60\%

of products.

Machine 2 makes:

40\%.

Defect rates:

P(D\mid M_1)=0.02

P(D\mid M_2)=0.05.

Find overall defect probability.

35. Apply total probability

P(D) = P(D\mid M_1)P(M_1) + P(D\mid M_2)P(M_2).

So:

P(D) = 0.02(0.60)+0.05(0.40).

=0.012+0.020.

Thus:

\boxed{P(D)=0.032}.

Overall defect rate:

3.2\%.

36. Weighted-average interpretation

Notice:

0.032

is a weighted average of the machine-specific defect rates:

2\%

and:

5\%.

Weights are machine production shares:

60\%,\quad40\%.

This is a useful intuition for total probability.

37. Bayes’ theorem

Bayes' theorem reverses conditioning.

Suppose we know:

P(A\mid B).

Bayes helps obtain:

P(B\mid A).

Start from:

P(A\cap B) = P(A\mid B)P(B)

and also:

P(A\cap B) = P(B\mid A)P(A).

Equate them.

38. Basic Bayes formula

P(A\mid B)P(B) = P(B\mid A)P(A).

Therefore:

\boxed{ P(B\mid A) = \frac{P(A\mid B)P(B)} {P(A)} }

provided:

P(A)>0.

39. General Bayes formula

If:

B_1,\dots,B_n

form a partition, then:

\boxed{ P(B_j\mid A) = \frac{ P(A\mid B_j)P(B_j) }{ \sum_iP(A\mid B_i)P(B_i) } }

The denominator is simply the law of total probability.

40. Bayes interpretation

Bayes updates:

\boxed{ \text{Prior} \rightarrow \text{Evidence} \rightarrow \text{Posterior} }

Where:

prior = $P(B)$
likelihood = $P(A\mid B)$
posterior = $P(B\mid A)$

41. Medical-test example

Suppose:

P(D)=0.01

for having a disease.

Test sensitivity:

P(+\mid D)=0.99.

False-positive rate:

P(+\mid D^c)=0.05.

Question:

If the test is positive, what is the probability the person actually has the disease?

We want:

P(D\mid +).

42. First compute total positive probability

P(+) = P(+\mid D)P(D) + P(+\mid D^c)P(D^c).

Since:

P(D^c)=0.99,

P(+) = 0.99(0.01)+0.05(0.99).

=0.0099+0.0495

=0.0594.

43. Apply Bayes

P(D\mid +) = \frac{ P(+\mid D)P(D) }{ P(+) }.

Thus:

= \frac{0.99(0.01)}{0.0594}.

= \frac{0.0099}{0.0594}

\approx \boxed{0.1667}.

So only about:

16.7\%.

44. Why this feels surprising

The test has:

99\%

sensitivity.

But the disease is rare:

1\%.

There are many more healthy people than diseased people.

Even a modest false-positive rate creates many false positives.

This is the base-rate effect.

45. Natural-frequency version

Imagine:

10,000

people.

Diseased:

100.

Positive among diseased:

99.

Healthy:

9900.

False positives:

0.05(9900)=495.

Total positives:

99+495=594.

Actually diseased among positives:

\frac{99}{594} \approx16.7\%.

Same Bayes result, often easier to understand.

46. Base-rate neglect

A common mistake is to focus on:

P(+\mid D)

while ignoring:

P(D).

That confuses:

probability of positive given disease

with:

probability of disease given positive.

They are fundamentally different.

47. Bayes with the machine example

Recall:

P(M_1)=0.60

P(M_2)=0.40

P(D\mid M_1)=0.02

P(D\mid M_2)=0.05.

We found:

P(D)=0.032.

Suppose a defective product is selected.

What is the probability it came from Machine 2?

48. Bayes calculation

P(M_2\mid D) = \frac{ P(D\mid M_2)P(M_2) }{ P(D) }.

So:

= \frac{0.05(0.40)}{0.032}

= \frac{0.020}{0.032}

= \boxed{0.625}.

So:

62.5\%

of defective products come from Machine 2 even though it makes only

40\%

of all products.

49. Prior vs posterior in that example

Before observing defect:

P(M_2)=0.40.

After learning product is defective:

P(M_2\mid D)=0.625.

The evidence "defective" raises belief that Machine 2 produced it because Machine 2 has the higher defect rate.

50. Bayes as reweighting

Bayes effectively reweights prior possibilities according to how compatible each is with the observed evidence.

Conceptually:

\boxed{ \text{Posterior weight} \propto \text{Likelihood}\times\text{Prior weight} }

Then normalize so probabilities sum to:

1.

51. Bayes workflow

For a Bayes question:

Identify hypotheses: $B_1,B_2,\dots$
Identify priors: $P(B_i)$
Identify likelihoods: $P(A\mid B_i)$
Compute: $P(A)$ using total probability.
Apply: $P(B_j\mid A).$

52. Tree diagrams for Bayes

Using the machine example:


text
                Defect
             / 
        M1
       /     \ No defect
Start
       \
        M2
             \ Defect
              \ No defect

Branch probabilities:

P(M_1)

then:

P(D\mid M_1).

Multiply along paths:

P(M_1\cap D) = P(M_1)P(D\mid M_1).

Add relevant terminal paths for total probability.

53. Table method for Bayes

Sometimes a contingency-table interpretation is easier.

Suppose 1000 products:

Machine 1 makes:

600.

At 2% defective:

12

defective.

Machine 2 makes:

400.

At 5% defective:

20

defective.

Total defective:

32.

Among defective products, Machine 2 share:

\frac{20}{32} = 0.625.

Same answer.

54. Conditional probability from intersection

Suppose:

P(A\cap B)=0.12

and:

P(B)=0.30.

Then:

P(A\mid B) = \frac{0.12}{0.30} = \boxed{0.4}.

55. Recover intersection from conditional probability

Suppose:

P(A\mid B)=0.4

and:

P(B)=0.3.

Then:

P(A\cap B) = 0.4\cdot0.3 = \boxed{0.12}.

These two operations should become automatic.

56. Recover conditional from independence

If:

A,B

are independent:

P(A\mid B)=P(A).

So if:

P(A)=0.7,

then:

\boxed{ P(A\mid B)=0.7 }

regardless of

B

, assuming

P(B)>0

57. Pairwise vs mutual independence

For several events, pairwise independence is weaker than full mutual independence.

For three events, mutual independence requires appropriate factorization not just pairwise checks.

At introductory level, the key warning is:

Don't assume that pairwise independence automatically means all events behave jointly independently.

58. Independence of repeated draws: with replacement

Suppose a card is drawn, returned to the deck, shuffled, then another card is drawn.

Because the first card is replaced, the distribution for the second draw remains unchanged.

This often gives independence between draws.

59. Without replacement

If the card is not replaced, the composition changes.

Suppose drawing an ace first.

Then the probability of ace on second draw becomes:

\frac3{51}

instead of:

\frac4{52}.

So draws are dependent.

60. Example: two aces without replacement

Probability first card is ace:

\frac4{52}.

Given first was ace:

P(\text{second ace}\mid\text{first ace}) = \frac3{51}.

Thus:

P(\text{two aces}) = \frac4{52}\cdot\frac3{51}.

61. With replacement version

If the first card is replaced:

P(\text{ace on first}) = \frac4{52}

and:

P(\text{ace on second}) = \frac4{52}.

Thus:

P(\text{two aces}) = \left(\frac4{52}\right)^2.

62. Conditional probability is not necessarily smaller

A common intuition error:

conditioning always reduces probability.

No.

Conditioning can:

increase probability
decrease probability
leave it unchanged

depending on the relationship.

63. Example of increase

P(\text{King})=\frac1{13}.

Given face card:

P(\text{King}\mid\text{Face}) = \frac13.

Increase.

64. Example of no change

For a standard card:

P(\text{King})=\frac1{13}.

Given heart:

P(\text{King}\mid\text{Heart}) = \frac1{13}.

No change.

This indicates independence between rank and suit.

Common mistake: reversing Bayes

Wrong:

P(A\mid B)=P(B\mid A).

Correct relationship:

\boxed{ P(A\mid B) = \frac{P(B\mid A)P(A)} {P(B)} }

Common mistake: multiplying without conditioning

For dependent events:

Wrong:

P(A\cap B)=P(A)P(B).

Correct:

\boxed{ P(A\cap B) = P(A)P(B\mid A) }

Only for independence may you replace:

P(B\mid A)

with:

P(B).

Common mistake: confusing mutually exclusive and independent

Again:

\boxed{ \text{mutually exclusive}\neq\text{independent} }

A useful memory device:

mutually exclusive → can't happen together
independent → can happen together, but don't influence each other's probability

Common mistake: wrong conditional denominator

For:

P(A\mid B),

denominator is:

\boxed{P(B)}

not:

P(A).

The event after the vertical bar defines the restricted universe.

Common mistake: forgetting priors in Bayes

If one cause is extremely rare, even a strong likelihood may not make it highly probable after evidence.

Always include:

P(B_i).

Bayes is not likelihood alone.

Common mistake: total probability without exhaustive cases

The

B_i

events must cover the full relevant sample space:

\bigcup_iB_i=S.

Otherwise the sum misses possible routes to

A

Common mistake: overlapping partition cases

For total probability, the partition pieces must also be mutually exclusive.

If:

B_1\cap B_2\neq\varnothing,

simply adding path probabilities can double-count.

72. Probability language map

Wording	Expression
A given B	$P(A\mid B)$
A and B	$P(A\cap B)$
A then B	$P(A)P(B\mid A)$
A independent of B	$P(A\mid B)=P(A)$
either source produced evidence	total probability
source given evidence	Bayes

73. Full worked example

Suppose:

P(A)=0.3

P(B\mid A)=0.8

P(B\mid A^c)=0.2.

Find:

$P(B)$
$P(A\cap B)$
$P(A\mid B)$

74. Step 1: complement

P(A^c)=1-0.3=0.7.

75. Step 2: total probability

P(B) = P(B\mid A)P(A) + P(B\mid A^c)P(A^c).

Thus:

P(B) = 0.8(0.3)+0.2(0.7).

=0.24+0.14

=\boxed{0.38}.

76. Step 3: intersection

P(A\cap B) = P(B\mid A)P(A)

= 0.8(0.3)

= \boxed{0.24}.

77. Step 4: Bayes

P(A\mid B) = \frac{P(B\mid A)P(A)} {P(B)}.

Therefore:

= \frac{0.24}{0.38}

\approx \boxed{0.632}.

78. Interpretation

Before seeing

B

P(A)=0.30.

After seeing

B

P(A\mid B)\approx0.632.

B

is evidence strongly favoring

A

That is Bayesian updating in its simplest form.

79. Core formula sheet

Conditional probability

\boxed{ P(A\mid B) = \frac{P(A\cap B)} {P(B)} }

Multiplication rule

\boxed{ P(A\cap B) = P(A\mid B)P(B) }

or:

\boxed{ P(A\cap B) = P(B\mid A)P(A) }

Independence

\boxed{ P(A\cap B)=P(A)P(B) }

equivalently:

\boxed{ P(A\mid B)=P(A) }

Total probability

\boxed{ P(A) = \sum_iP(A\mid B_i)P(B_i) }

Bayes

\boxed{ P(B_j\mid A) = \frac{ P(A\mid B_j)P(B_j) }{ \sum_iP(A\mid B_i)P(B_i) } }

80. Week 8 mental map


text
Conditional probability
        |
        v
P(A|B) = P(A∩B)/P(B)
        |
        v
Multiplication rule
        |
        +----> Independence
        |
        v
Partition sample space
        |
        v
Law of total probability
        |
        v
Bayes' theorem

Everything in Week 8 is tightly connected rather than five separate topics.

81. Weeks 7 → 8 connection

Week 7:

P(A)

asks for an unconditional probability.

Week 8:

P(A\mid B)

asks what happens after gaining information.

Then Bayes goes one step further:

\boxed{ \text{new evidence} \rightarrow \text{updated probabilities} }

82. What you should actually master

By the end of Stats 1 Week 8, you should be able to:

Interpret: $P(A\mid B).$
Calculate conditional probability.
Choose the correct conditioning denominator.
Apply: $P(A\cap B)=P(A)P(B\mid A).$
Define independence.
Test independence using: $P(A\cap B)=P(A)P(B).$
Distinguish independence from mutual exclusivity.
Handle repeated independent trials.
Handle dependent sampling without replacement.
Use complements for "at least one."
Recognize a partition of the sample space.
Apply the law of total probability.
Apply Bayes’ theorem.
Distinguish:

P(A\mid B)

from:

P(B\mid A).

Understand prior, likelihood, and posterior intuitively.
Avoid base-rate neglect.

If those are comfortable, Stats 1 Week 8 is under control.

Quick Recall & Flashcards

Test your active recall before proceeding to practice problems. Format:

Question :: Answer.

Week 8 Diagnostic: If P(A) = 0.4, P(B) = 0.5, and P(A and B) = 0.2, are A and B independent? :: Answer: A — P(A) * P(B) = 0.4 * 0.5 = 0.2. Since P(A and B) = P(A)*P(B), events A and B are independent.
Week 8 Diagnostic: What is P(A|B) if P(A and B) = 0.15 and P(B) = 0.30? :: Answer: 0.5 — P(A|B) = P(A and B) / P(B) = 0.15 / 0.30 = 0.5.

Practice & Self-Check

Attempt these diagnostic questions to verify your operational understanding.

Question 1

If P(A) = 0.4, P(B) = 0.5, and P(A and B) = 0.2, are A and B independent?

MCQ

3 Unit Assessment

A. Yes, because P(A and B) = P(A) * P(B)

B. No, because P(A and B) != P(A) + P(B)

C. Only if A and B are mutually exclusive

D. Insufficient information

Question 2

What is P(A|B) if P(A and B) = 0.15 and P(B) = 0.30?

NAT

3 Unit Assessment