Week 3.3: Covariance & Correlation

601 words

3 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# Week 3.3: Covariance & Correlation > **Prerequisites:** Expectation ([Week 3.1: Expected Value](/notes/01-foundation-bsma1004-stats-2-week03-08-expectation)), Variance ([Week 3.2: Variance & Standard Deviation](/notes/01-foundation-bsma1004-stats-2-week03-09-variance)), Independence ([Week 2.3: Independence of Ran...

Week 3.3: Covariance & Correlation

Prerequisites: Expectation (Week 3.1: Expected Value), Variance (Week 3.2: Variance & Standard Deviation), Independence (Week 2.3: Independence of Random Variables) Cross-links: BSMA3012 (Linear Stat Models) — covariance matrix Core question: How do two random variables vary together?

1. Intuition: The Co-variation

Two RVs can have the same individual means and variances yet completely different relationships:

$Y \backslash X$	0	1
0	1/4	1/4
1	1/4	1/4

Independent:

\operatorname{Cov}(X,Y) = 0

$Y \backslash X$	0	1
0	0	1/2
1	1/2	0

Perfect negative dependence:

X

determines

Y

completely.

\operatorname{Cov}(X,Y) = -1/4

. Covariance quantifies this: positive when they move together, negative when they move opposite, zero when uncorrelated.

2. Formal Definitions

Definition (Covariance)

> \operatorname{Cov}(X,Y) = \mathbb{E}\left[(X - \mu_X)(Y - \mu_Y)\right]. >

Alternative formula:

\operatorname{Cov}(X,Y) = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y].

Definition (Correlation Coefficient)

> \rho_{XY} = \frac{\operatorname{Cov}(X,Y)}{\sigma_X \sigma_Y}, \quad -1 \leq \rho_{XY} \leq 1. >

The correlation coefficient is a dimensionless measure of linear dependence.

3. Properties of Covariance

$\operatorname{Cov}(X,X) = \operatorname{Var}(X)$
$\operatorname{Cov}(X,Y) = \operatorname{Cov}(Y,X)$ (symmetric)
$\operatorname{Cov}(aX, bY) = ab \cdot \operatorname{Cov}(X,Y)$
$\operatorname{Cov}(X+Y, Z) = \operatorname{Cov}(X,Z) + \operatorname{Cov}(Y,Z)$ (bilinear)
If $X,Y$ independent → $\operatorname{Cov}(X,Y) = 0$ (converse false)

Warning: Zero covariance does not imply independence. Example: $X \in \{-1,0,1\}$ , $Y = X^2$ .

4. Correlation: Interpretation

$\rho$	Interpretation
$+1$	Perfect positive linear relationship ( $Y = aX + b$ , $a>0$ )
$>0$	Positive association (tall→heavy)
$0$	No linear association
$<0$	Negative association (rain→debt)
$-1$	Perfect negative linear relationship ( $Y = aX + b$ , $a<0$ )

5. Computing Covariance: Example

$Y \backslash X$	-1	0	1
-1	1/15	2/15	2/15
0	2/15	1/15	2/15
1	2/15	2/15	1/15

Step 1 — Marginals:

f_X(-1) = 5/15 = 1/3

f_X(0) = 5/15 = 1/3

f_X(1) = 5/15 = 1/3

. Same for

Y

. Step 2 — Means:

\mu_X = 0

\mu_Y = 0

. Step 3 — $\mathbb{E}[XY]$ :

\mathbb{E}[XY] = \sum_{x,y} xy \cdot f_{XY}(x,y).

Non-zero terms:

(-1)(-1)(1/15) + (-1)(1)(2/15) + (1)(-1)(2/15) + (1)(1)(1/15)

= (1/15) - (2/15) - (2/15) + (1/15) = -2/15

. Step 4 — Covariance:

\operatorname{Cov}(X,Y) = \mathbb{E}[XY] - \mu_X\mu_Y = -2/15 - 0 = -2/15

6. Practice Questions

Q1 (Easy)

X \in \{0,1\}

Y \in \{0,1\}

with

\mathbb{P}(X=1)=0.6

\mathbb{P}(Y=1)=0.7

\mathbb{P}(X=1,Y=1)=0.5

. Find

\operatorname{Cov}(X,Y)

Full Solution

$\mathbb{E}[X] = 0.6$ , $\mathbb{E}[Y] = 0.7$ , $\mathbb{E}[XY] = \mathbb{P}(X=1,Y=1) = 0.5$ .

$\operatorname{Cov}(X,Y) = 0.5 - (0.6)(0.7) = 0.5 - 0.42 = 0.08$ (positive).

Q2 (Medium)

Given

f_{XY}

below, find

\rho_{XY}

$Y \backslash X$	0	1
0	$1/4-x$	$1/4+x$
1	$1/4+x$	$1/4-x$

Full Solution

$\mu_X = \mu_Y = 0.5$ . $\operatorname{Var}(X) = \operatorname{Var}(Y) = 0.25$ .

$\mathbb{E}[XY] = 0\cdot0(1/4-x) + 0\cdot1(1/4+x) + 1\cdot0(1/4+x) + 1\cdot1(1/4-x) = 1/4 - x$ .

$\operatorname{Cov}(X,Y) = (1/4 - x) - (0.5)(0.5) = -x$ .

$\rho_{XY} = \frac{-x}{\sqrt{0.25 \cdot 0.25}} = \frac{-x}{0.25} = -4x$ .

For $x > 0$ → negative correlation; $x = 0$ → uncorrelated; $x = 1/4$ → $\rho = -1$ (perfect negative).

Next topic: Week 3.4: Markov & Chebyshev Inequalities — Markov and Chebyshev inequalities. Join Discord PreviousWeek 3.2: Variance & Standard DeviationNextWeek 3.4: Markov & Chebyshev Inequalities

What this note is really saying

Week 3.3: Covariance & Correlation

1. Intuition: The Co-variation

2. Formal Definitions

3. Properties of Covariance

4. Correlation: Interpretation

5. Computing Covariance: Example

6. Practice Questions

Q1 (Easy)

Q2 (Medium)

Intelligence Hub