Neural Sync Active

Week 1.2: Marginal Distributions

Registry Synced

Week 1.2: Marginal Distributions

1674 words

8 min read

Reading compass

Now · 1. Intuition: "Summing Away" the Other Variable

Week 1.2: Marginal Distributions

Prerequisites: Joint PMF (Week 1.1: Joint Probability Mass Function (Joint PMF)) Cross-links: BSMA3012 (Linear Stat Models) — marginalisation in multivariate regression Core question: Given the full joint distribution, how do we recover the distribution of just one variable?

1. Intuition: "Summing Away" the Other Variable

The joint PMF

f_{XY}(x, y)

tells us everything about the pair

(X, Y)

. But sometimes we only care about

X

alone. How do we get

f_X(x)

from

f_{XY}

? Intuition: The event

(X = x)

can be partitioned into the disjoint union of events

(X = x, Y = y)

for all possible

y

. Since these events are mutually exclusive:

\mathbb{P}(X = x) = \sum_{y \in T_Y} \mathbb{P}(X = x, Y = y).

We are summing over (or marginalising out)

Y

. The resulting PMF is called the marginal PMF because it appears in the margins of a contingency table.

2. Formal Definition

Definition (Marginal PMF) Let $X$ and $Y$ be jointly distributed discrete random variables with joint PMF $f_{XY}$ . The marginal PMF of $X$ is

> f_X(t) = \mathbb{P}(X = t) = \sum_{t' \in T_Y} f_{XY}(t, t'), \quad t \in T_X. >

Similarly, the marginal PMF of $Y$ is

> f_Y(t) = \mathbb{P}(Y = t) = \sum_{t' \in T_X} f_{XY}(t', t), \quad t \in T_Y. >

Proof sketch: Let

T_Y = \{y_1, y_2, \dots, y_K\}

. Then:

\begin{aligned} (X = t) &= (X = t, Y = y_1) \cup (X = t, Y = y_2) \cup \dots \cup (X = t, Y = y_K) \\ \mathbb{P}(X = t) &= \sum_{j=1}^K \mathbb{P}(X = t, Y = y_j) = \sum_{j=1}^K f_{XY}(t, y_j). \end{aligned}

The union is disjoint because

Y

can only take one value at a time.

3. Visualised: The Margins of a Table

The name "marginal" comes from where these values appear in a contingency table — they are written in the margins (the extra row and column).

Example: Toss a Fair Coin Twice

$X_2 \backslash X_1$	0	1	$f_{X_2}$
0	1/4	1/4	1/2
1	1/4	1/4	1/2
$f_{X_1}$	1/2	1/2

$f_{X_1}(0) = 1/4 + 1/4 = 1/2$ (column sum)
$f_{X_1}(1) = 1/4 + 1/4 = 1/2$ (column sum)
$f_{X_2}(0) = 1/4 + 1/4 = 1/2$ (row sum)
$f_{X_2}(1) = 1/4 + 1/4 = 1/2$ (row sum)

4. Key Insight: Same Marginals, Different Joints

It is entirely possible for different joint PMFs to produce the same marginal PMFs. This is a critical observation — the marginals alone do not capture the dependence structure.

Example

Consider: Case 1 (Independent):

$Y \backslash X$	0	1	$f_Y$
0	1/4	1/4	1/2
1	1/4	1/4	1/2
$f_X$	1/2	1/2

Case 2 (Dependent, parameter $x$ ):

$Y \backslash X$	0	1	$f_Y$
0	$x$	$1/2 - x$	1/2
1	$1/2 - x$	$x$	1/2
$f_X$	1/2	1/2

For any

x \in [0, 1/2]

, both marginals are

f_X(0)=f_X(1)=1/2

f_Y(0)=f_Y(1)=1/2

. But the joint distributions differ!

$x = 1/4$ → independent case
$x = 0$ → perfect negative dependence ( $X$ and $Y$ always differ)
$x = 1/2$ → perfect positive dependence ( $X$ and $Y$ always equal)

Takeaway: The joint PMF contains strictly more information than the two marginals. Marginalisation loses information about dependence.

5. Marginalisation for Multiple RVs

For

n

random variables

X_1, \dots, X_n

with joint PMF

f_{X_1\dots X_n}

Individual marginal: Sum over all other variables.

f_{X_1}(t) = \sum_{t_2 \in T_{X_2}} \cdots \sum_{t_n \in T_{X_n}} f_{X_1\dots X_n}(t, t_2, \dots, t_n).

Joint marginal of a subset: Sum over the variables not in the subset.

f_{X_1X_2}(t_1, t_2) = \sum_{t_3 \in T_{X_3}} \cdots \sum_{t_n \in T_{X_n}} f_{X_1\dots X_n}(t_1, t_2, t_3, \dots, t_n).

Mnemonic: Sum over everything you do NOT want.

6. Worked Examples

Example 1: Random 2-Digit Number

From the joint PMF table in Week 1.1: Joint Probability Mass Function (Joint PMF):

$Y \backslash X$	0	1	2	3	4	5	6	7	8	9	$f_Y$
0	1/20	0	1/20	0	1/20	0	1/20	0	1/20	0	1/4
1	0	1/20	0	1/20	0	1/20	0	1/20	0	1/20	1/4
2	1/20	0	1/20	0	1/20	0	1/20	0	1/20	0	1/4
3	0	1/20	0	1/20	0	1/20	0	1/20	0	1/20	1/4
$f_X$	1/10	1/10	1/10	1/10	1/10	1/10	1/10	1/10	1/10	1/10

$f_X(x) = 1/10$ for $x = 0,1,\dots,9$ → $X \sim \text{Uniform}\{0,\dots,9\}$
$f_Y(y) = 1/4$ for $y = 0,1,2,3$ → $Y \sim \text{Uniform}\{0,1,2,3\}$ Despite both being uniform, $X$ and $Y$ are not independent (the joint is not the product of marginals).

Example 2: Triple Coin Toss

$X_1$	$X_2$	$X_3$	$f_{X_1X_2X_3}$
0	0	0	1/8
0	0	1	1/8
0	1	0	1/8
0	1	1	1/8
1	0	0	1/8
1	0	1	1/8
1	1	0	1/8
1	1	1	1/8

Marginal of $X_1$ :

\begin{aligned} f_{X_1}(0) &= f(0,0,0) + f(0,0,1) + f(0,1,0) + f(0,1,1) = 1/8+1/8+1/8+1/8 = 1/2 \\ f_{X_1}(1) &= f(1,0,0) + f(1,0,1) + f(1,1,0) + f(1,1,1) = 1/2 \end{aligned}

X_1 \sim \text{Bernoulli}(1/2)

. Joint marginal of $(X_1, X_2)$ :

\begin{aligned} f_{X_1X_2}(0,0) &= f(0,0,0) + f(0,0,1) = 1/8+1/8 = 1/4 \\ f_{X_1X_2}(0,1) &= f(0,1,0) + f(0,1,1) = 1/4 \\ f_{X_1X_2}(1,0) &= f(1,0,0) + f(1,0,1) = 1/4 \\ f_{X_1X_2}(1,1) &= f(1,1,0) + f(1,1,1) = 1/4 \end{aligned}

Note

(X_1, X_2)

are independent and uniform — same as the 2-coin toss.

Example 3: IPL Over

From the IPL example, marginal of

X

(runs) can be obtained by summing over

Y

f_X(x) = \sum_{y=0}^2 f_{XY}(x,y) = \sum_{y=0}^2 f_Y(y) \cdot f_{X \mid Y=y}(x).

For

x = 6

f_X(6) = \frac{13}{16}\cdot\frac{1}{7} + \frac{1}{8}\cdot\frac{1}{7} + \frac{1}{16}\cdot\frac{1}{7} = \frac{1}{7}

7. Formula Summary

Operation	Formula	Mnemonic
Marginal of $X$ from $(X,Y)$	$f_X(x) = \sum_y f_{XY}(x,y)$	Sum over $y$
Marginal of $Y$ from $(X,Y)$	$f_Y(y) = \sum_x f_{XY}(x,y)$	Sum over $x$
Joint marginal $(X_i, X_j)$ from $n$ RVs	$f_{X_iX_j}(u,v) = \sum_{\text{others}} f_{X_1\dots X_n}$	Sum out the rest
Individual marginal from $n$ RVs	$f_{X_i}(t) = \sum_{\text{all others}} f_{X_1\dots X_n}$	Sum out everything else

8. Practice Questions

Q1 (Easy)

For the joint PMF below, find the marginal PMFs of

X

and

Y

$Y \backslash X$	0	1	2
0	0.1	0.2	0.1
1	0.15	0.05	0.1
2	0.05	0.15	0.1

Full Solution

Marginal of $X$ (column sums):

$f_X(0) = 0.1 + 0.15 + 0.05 = 0.30$

$f_X(1) = 0.2 + 0.05 + 0.15 = 0.40$

$f_X(2) = 0.1 + 0.1 + 0.1 = 0.30$

Marginal of $Y$ (row sums):

$f_Y(0) = 0.1 + 0.2 + 0.1 = 0.40$

$f_Y(1) = 0.15 + 0.05 + 0.1 = 0.30$

$f_Y(2) = 0.05 + 0.15 + 0.1 = 0.30$

Check: $\sum f_X = 0.3+0.4+0.3 = 1$ , $\sum f_Y = 0.4+0.3+0.3 = 1$ . ✓

Q2 (Medium)

For the same joint PMF, compute

\mathbb{P}(X + Y \leq 1)

using the marginal distributions. Can you compute it using only marginals?

Strategy Hint

The event $X+Y \leq 1$ involves both variables together; you need the joint PMF, not just the marginals. Full Solution

We need the joint probabilities for pairs where $x+y \leq 1$ :

$(0,0)$ : 0.1

$(0,1)$ : 0.15

$(1,0)$ : 0.2

$(0,0)$ already counted

$\mathbb{P}(X+Y \leq 1) = 0.1 + 0.15 + 0.2 = 0.45$ .

Cannot be computed from marginals alone — we need the joint.

Q3 (Hard)

Three fair dice are rolled. Let

X_1, X_2, X_3

be the outcomes. Find

f_{X_1X_2}(i,j)

and

f_{X_1}(i)

Full Solution

Since rolls are independent and uniform $\{1,\dots,6\}$ :

$f_{X_1X_2}(i,j) = f_{X_1}(i) \cdot f_{X_2}(j) = \frac{1}{6} \cdot \frac{1}{6} = \frac{1}{36}$ for all $i,j \in \{1,\dots,6\}$ .

$f_{X_1}(i) = \frac{1}{6}$ for $i = 1,\dots,6$ .

Alternatively, marginalise from the triple:

> f_{X_1X_2}(i,j) = \sum_{k=1}^6 f_{X_1X_2X_3}(i,j,k) = \sum_{k=1}^6 \frac{1}{216} = \frac{6}{216} = \frac{1}{36}. >

Q4 (Exam-style)

The joint PMF of

(X,Y)

is given by

f_{XY}(x,y) = c \cdot (x+y)

for

x = 1,2

y = 1,2,3

. Find

c

and the marginal PMFs.

Strategy Hint

First find $c$ by requiring total probability = 1, then compute marginals. Full Solution

Find $c$ :

> \begin{aligned} 1 &= \sum_{x=1}^2 \sum_{y=1}^3 c(x+y) = c \sum_{x=1}^2 \sum_{y=1}^3 (x+y) \\ &= c \left[ \sum_{x=1}^2 \sum_{y=1}^3 x + \sum_{x=1}^2 \sum_{y=1}^3 y \right] \\ &= c \left[ (1+2) \cdot 3 + 2 \cdot (1+2+3) \right] \\ &= c \left[ 3 \cdot 3 + 2 \cdot 6 \right] = c (9 + 12) = 21c \\ \Rightarrow c &= \frac{1}{21}. \end{aligned} >

Marginal of $X$ :

> \begin{aligned} f_X(1) &= \sum_{y=1}^3 \frac{1}{21}(1+y) = \frac{1}{21}[(2)+(3)+(4)] = \frac{9}{21} = \frac{3}{7} \\ f_X(2) &= \sum_{y=1}^3 \frac{1}{21}(2+y) = \frac{1}{21}[(3)+(4)+(5)] = \frac{12}{21} = \frac{4}{7} \end{aligned} >

Marginal of $Y$ :

> \begin{aligned} f_Y(1) &= \frac{1}{21}[(1+1)+(2+1)] = \frac{1}{21}(2+3) = \frac{5}{21} \\ f_Y(2) &= \frac{1}{21}[(1+2)+(2+2)] = \frac{1}{21}(3+4) = \frac{7}{21} = \frac{1}{3} \\ f_Y(3) &= \frac{1}{21}[(1+3)+(2+3)] = \frac{1}{21}(4+5) = \frac{9}{21} = \frac{3}{7} \end{aligned} >

Check: $\sum f_X = 3/7+4/7=1$ , $\sum f_Y = 5/21+7/21+9/21=1$ . ✓

Next topic: Week 1.3: Conditional Distributions — The distribution of one variable given the value of another. Join Discord PreviousWeek 1.1: Joint Probability Mass Function (Joint PMF)NextWeek 1.3: Conditional Distributions