Week 2.3: Independence of Random Variables

Prerequisites: Joint PMF (Week 1.1: Joint Probability Mass Function (Joint PMF)), Marginal Distributions (Week 1.2: Marginal Distributions) Cross-links: BSMA1002 (Stats 1) — independence of events Core question: When does knowing one random variable tell us nothing about another?

1. Intuition: The Product Condition

2. Formal Definition

Definition (Independence of two RVs) $X$ and $Y$ are independent if for all $x \in T_X$, $y \in T_Y$:

Equivalently: the conditional PMF equals the marginal PMF:

Definition (Mutual independence of $n$ RVs) $X_1,\dots,X_n$ are mutually independent if for all $t_i \in T_{X_i}$:

3. How to Check Independence

1. Compute marginal PMFs $f_X$ and $f_Y$ by summing over the other variable.
2. For every $(x,y)$ pair, check if $f_{XY}(x,y) = f_X(x) \cdot f_Y(y)$.
3. If any pair violates this, $X$ and $Y$ are dependent.

Shortcut: If there exists a cell $(x,y)$ where $f_{XY}(x,y) = 0$ but $f_X(x) > 0$ and $f_Y(y) > 0$, then $X$ and $Y$ are dependent.

Examples

4. Important Properties

1. $g(X)$ and $h(Y)$ are independent for any functions $g, h$.
2. $E[XY] = E[X]E[Y]$
3. $\operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$
4. $\operatorname{Cov}(X,Y) = 0$ (but the converse is false!)
5. $M_{X+Y}(t) = M_X(t) \cdot M_Y(t)$ (moment generating functions multiply) For mutually independent $X_1,\dots,X_n$:

• Functions of non-overlapping subsets are independent
• Example: $g(X_1,X_2)$ is independent of $h(X_3,X_4)$

5. i.i.d. Random Variables

Definition (i.i.d.) $X_1,\dots,X_n$ are independent and identically distributed (i.i.d.) if:

1. They are mutually independent.
2. All have the same marginal distribution: $f_{X_i} = f_X$. The i.i.d. assumption underlies most of statistical inference — it models repeated independent draws from the same population. Example: Repeated tosses of the same coin produce i.i.d. Bernoulli($p$) random variables.

6. Pairwise vs Mutual Independence

Warning: Pairwise independence does NOT imply mutual independence. Example (Even parity): $X_1, X_2, X_3$ each Uniform$\{0,1\}$, but the triple has even parity (sum is even).

• Any pair is independent (check: $f_{X_1X_2}(0,0)=1/4 = (1/2)(1/2)$).
• But the triple is dependent: $f_{X_1X_2X_3}(0,0,1) = 0 \neq (1/2)(1/2)(1/2) = 1/8$.

7. Mermaid: Independence Flowchart

8. Practice Questions

Q1 (Easy)

Full Solution

Marginals: $f_X(x) = 1/3$ for $x=0,1,2$, $f_Y(y) = 1/3$ for $y=0,1,2$.

$f_{XY}(x,y) = 1/9 = (1/3)(1/3) = f_X(x)f_Y(y)$ for all 9 cells → Independent.

Q2 (Medium)

Full Solution

From the joint table: $f_{XY}(0,0) = 1/20$, $f_X(0) = 1/10$, $f_Y(0) = 1/4$.

$f_X(0)f_Y(0) = (1/10)(1/4) = 1/40 \neq 1/20$ → Dependent.

Indeed, $f_{XY}(1,0) = 0$ while $f_X(1)f_Y(0) = (1/10)(1/4) = 1/40$ confirms dependence.

Q3 (Hard — Even Parity)

Full Solution

From the table: $f_{X_1X_2}(0,0) = 1/4$, $f_{X_1}(0) = 1/2$, $f_{X_2}(0) = 1/2$ → $1/4 = (1/2)(1/2)$ ✓ $f_{X_1X_2}(0,1) = 1/4 = (1/2)(1/2)$ ✓ $f_{X_1X_2}(1,0) = 1/4 = (1/2)(1/2)$ ✓ $f_{X_1X_2}(1,1) = 1/4 = (1/2)(1/2)$ ✓

So pairwise independent.

But $f_{X_1X_2X_3}(0,0,1) = 0 \neq (1/2)^3 = 1/8$. Not mutually independent.

Next topic: Week 3.1: Expected Value — Expected value of a random variable. Join Discord PreviousWeek 2.2: Sum (Convolution), Min & Max of Random VariablesNextWeek 3.1: Expected Value