Correlation — Measuring Relationships Between Numerical Variables

🎯 Learning Objectives

• Create and interpret scatter plots for bivariate numerical data
• Compute and interpret covariance
• Calculate and interpret Pearson's correlation coefficient (r)
• Distinguish correlation from causation
• Recognize the effect of outliers and non-linear relationships on correlation
• Know when correlation is and isn't appropriate

📋 Prerequisites

• Central Tendency (03-central-tendency) — mean is used in covariance
• Dispersion (04-dispersion-percentiles) — standard deviation is in the formula
• Contingency Tables (06-contingency-tables) — understanding association for categorical data
• Basic algebra: summation notation, fractions

---

📖 Core Content

8.1 Intuition: How Two Variables Move Together

• As temperature rises, do ice cream sales increase? → Positive relationship
• As a car's age increases, does its value decrease? → Negative relationship
• Is there no connection between shoe size and IQ? → No relationship Correlation quantifies the strength and direction of a linear relationship between two numerical variables.

Everyday analogy: Imagine you and a friend are walking. If you both tend to walk forward together (both increase at the same time), you're positively correlated. If one of you walks forward while the other walks backward (one increases as the other decreases), you're negatively correlated. If your movements are unrelated (one walks while the other hops on one foot), there's no correlation. 🔑 Key Insight: Correlation measures linear relationships only. Two variables could be perfectly related (e.g., $y = x^2$) but have zero correlation because the relationship is curved, not straight.

8.2 Scatter Plots

Anatomy of a Scatter Plot

• X-axis: Independent variable (predictor)
• Y-axis: Dependent variable (response)
• Each point: One observation (x, y) (Diagram)

Reading Patterns from a Scatter Plot

8.3 Covariance

Definition

$$\text{Cov}(x, y) = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{n - 1}$$

Intuition

• If both $x$ and $y$ are above their means → $(x_i - \bar{x})(y_i - \bar{y}) > 0$ → positive contribution
• If both are below their means → both differences are negative, product positive → positive contribution
• If one is above and other below → product negative → negative contribution Sum all contributions → positive covariance means X and Y tend to move in the same direction.

Limitation

8.4 Pearson's Correlation Coefficient (r)

8.4.1 Intuition

$$r = \frac{\text{Cov}(x, y)}{s_x s_y} = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2} \sqrt{\sum (y_i - \bar{y})^2}}$$

8.4.2 Interpreting r

Note: These are rough guidelines. Context matters — in physics, r = 0.99 is expected; in social sciences, r = 0.3 might be considered strong.

8.4.3 The Coefficient of Determination (r²)

$$r^2 = \text{(correlation coefficient)}^2$$

8.5 Step-by-Step Calculation

$$\bar{x} = \frac{2+4+6+8+10}{5} = \frac{30}{5} = 6$$ $$\bar{y} = \frac{65+70+75+85+95}{5} = \frac{390}{5} = 78$$

• $\sum (x-\bar{x})(y-\bar{y}) = 52 + 16 + 0 + 14 + 68 = 150$
• $\sum (x-\bar{x})^2 = 16 + 4 + 0 + 4 + 16 = 40$
• $\sum (y-\bar{y})^2 = 169 + 64 + 9 + 49 + 289 = 580$ Step 4: Compute r

$$r = \frac{150}{\sqrt{40} \times \sqrt{580}} = \frac{150}{\sqrt{23200}} = \frac{150}{152.315} \approx 0.985$$ $$r^2 \approx 0.970 = 97\%$$

8.6 Properties of r

8.7 Correlation vs. Causation

1. Causation: X → Y (more studying causes higher scores)
2. Reverse causation: Y → X (higher scores cause more studying — maybe?)
3. Confounding variable: Z → X and Z → Y (a third variable causes both)
4. Coincidence: Random chance (especially with small samples)
5. Selection bias: The data was collected in a biased way

Classic Spurious Correlations

• Ice cream sales and drowning deaths are positively correlated. Does ice cream cause drowning? No! Both are caused by a confounder: hot weather (more people swim, more people eat ice cream).
• Number of storks and number of babies in a region are correlated. Do storks deliver babies? No — both are related to the region being rural (more space for storks, higher birth rates in rural areas).
• Chocolate consumption and Nobel laureates per capita are correlated. Does chocolate make you smarter? Probably not — both are correlated with the country's wealth.

Memory trick: "Correlation does not imply causation" is the first principle of statistical thinking.

8.8 Outliers and Correlation

8.9 Non-Linear Relationships

• The relationship is perfect (y is completely determined by x)
• But r ≈ 0 because the relationship is U-shaped (not linear) How to detect: Look at the scatter plot! If you see a curve, correlation is the wrong tool.

8.10 Worked Examples

Example 1: Computing r (Easy)

$$r = \frac{20}{\sqrt{10} \times \sqrt{40}} = \frac{20}{\sqrt{400}} = \frac{20}{20} = 1.0$$

Example 2: Interpreting r (Medium)

• TV watching might cause lower scores (by replacing study time)
• Lower-scoring students might watch more TV (reverse causation)
• A confounder like socioeconomic status might affect both TV access and educational outcomes
• Generally unmotivated students might watch more TV AND study less

Example 3: Comparing Relationships (Harder)

8.11 Edge Cases & Gotchas

All Points on a Vertical Line

Correlation of a Variable with Itself

The "Restriction of Range" Problem

8.12 Why This Matters

• Regression (linear models): Regression extends correlation to prediction
• Factor analysis: Correlations among variables reveal underlying patterns
• Machine learning: Feature selection uses correlations
• Portfolio theory: Asset correlations determine diversification benefits Later topics that build on correlation:
• BSMA1003 (Maths 2): Linear algebra view of correlation matrices
• BSMA1004 (Stats 2): Inference for correlation, multiple regression
• BSCS2004 (ML Foundations): Feature selection, multicollinearity

---

📐 Key Formulas / Concepts

---

⚠️ Common Pitfalls

Pitfall 1: Assuming Correlation Implies Causation

Pitfall 2: Only Reporting r Without a Scatter Plot

Pitfall 3: Treating r = 0 as "No Relationship"

Pitfall 4: Comparing r Across Different Contexts

---

📝 Practice Questions

Q1: Compute r

Find r for: (1,1), (2,3), (3,2), (4,4), (5,5)

<details> <strong>Solution</strong>

$\bar{x} = 3$, $\bar{y} = 3$

$r = \frac{9}{\sqrt{10}\sqrt{10}} = \frac{9}{10} = 0.9$

$\boxed{r = 0.9}$

</details> > **Q2: Interpret r²** > > A study finds r = 0.6 between exercise hours and sleep quality. Find and interpret r². > > <details> <strong>Solution</strong> > > r² = 0.6² = 0.36 > > **Interpretation:** 36% of the variation in sleep quality can be explained by exercise hours (in the linear model). The remaining 64% is due to other factors (diet, stress, medical conditions, etc.). </details> > **Q3: Identify the Pattern** > > For each scatter plot description, predict r: > > a) Points tightly clustered around a downward-sloping line b) Points randomly scattered with no pattern c) Points following a clear upward curve (U-shape opening upward) > > <details> <strong>Solution</strong> > > a) **r near -1** (strong negative linear relationship) > > b) **r near 0** (no linear relationship) > > c) **r near 0** but for the wrong reason! The relationship is clear (curved), but r only measures linearity. This is a classic example of why you need a scatter plot — r misses the non-linear relationship. </details> > **Q4: Outlier Effect** > > Main data: (1,2), (2,4), (3,6), (4,8) > > Add outlier: (10,1) > > Find r without and with the outlier. > > <details> <strong>Solution</strong> > > **Without outlier:** x̄ = 2.5, ȳ = 5 > >

r_without = 10 / √(5×20) = 10/10 = 1.0 ✅

With outlier (10,1): x̄ = (1+2+3+4+10)/5 = 20/5 = 4 ȳ = (2+4+6+8+1)/5 = 21/5 = 4.2

r_with = -14 / √(50×32.8) = -14 / √1640 = -14 / 40.497 ≈ -0.346

Dramatic effect: r went from +1.0 to -0.346! One outlier completely reversed the relationship. This shows why outliers must be investigated.

</details> > **Q5: Multiple Choice** > > </strong> > > Which of the following does NOT affect Pearson's r? > > a) An outlier b) Changing the units of measurement (cm → inches) c) A non-linear relationship d) Adding an extreme value > > <details> <strong>Solution</strong> > > **Answer: b)** Changing units does NOT affect r. > > - **a) Outlier:** YES — affects r significantly > - **b) Units:** **NO** — r is unitless, standardized by standard deviations > - **c) Non-linear:** YES — r depends on the relationship being linear > - **d) Extreme value:** YES — this is an outlier, which affects r </details> > **Q6: Spurious Correlation** > > </strong> > > Correlation between the number of fire stations in a city and the number of fires is r = 0.85. Does having more fire stations cause more fires? Explain. > > <details> <strong>Solution</strong> > > **No! This is a spurious correlation.** > > The fire department's interpretation: More fire stations don't cause more fires. Instead, cities with more fires build more fire stations to respond to them. This is **reverse causation**. > > Or, more likely: **confounding variable** — larger cities have more buildings (population), which means both more fires AND more fire stations. The size of the city is the confounder. > > **Alternative explanations:** > > 1. Larger population → more fires + more fire stations (confounding) > 2. Cities with more fires respond by building more stations (reverse causation) </details> > **Q7: Which Has Stronger Correlation?** > > </strong> > > Study A: r = 0.45, n = 500 Study B: r = 0.40, n = 50 > > Which correlation is stronger? Which is more reliable? > > <details> <strong>Solution</strong> > > **Strength:** Study A (r = 0.45) has a slightly stronger correlation than Study B (r = 0.40). > > **Reliability:** Study A (n = 500) is much more reliable. With n = 50, the margin of error around r is much larger — the true population correlation could be anywhere from 0.15 to 0.65. With n = 500, the estimate is more precise. > > **Lesson:** Strength (r value) and reliability (sample size) are different things. A strong correlation with a small sample might not be statistically significant. </details> > **Q8: Application** > > </strong> > > A real estate researcher collects data on 200 houses: > >

a) Which variable has the strongest linear relationship with price? b) Interpret r for Size vs Price. c) Why might Age be negatively correlated with Price? d) If you could only know ONE variable to predict price, which would you choose?

<details> <strong>Solution</strong>

a) Size (sq ft) has the strongest relationship with price (|r| = 0.72).

b) Size vs Price (r = 0.72): Strong positive linear relationship. Larger houses tend to have higher prices. r² = 0.52, so 52% of the variation in price can be explained by house size alone.

c) Age vs Price (r = -0.55): Older houses tend to be cheaper (negative correlation). Why?

• Wear and tear, need for renovations
• Older designs may be less desirable
• However, some old houses in good locations (heritage areas) may be expensive — this is why the correlation is only moderate, not perfect

d) Best predictor: Size (highest |r| = 0.72). Size alone explains 52% of price variation. But using ALL variables together (multiple regression) would give even better predictions.

</details> > **Q9: Simpson's Paradox with Correlation** > > </strong> > > A school has two programs. Within each program, there's a positive correlation between hours studied and grades (r = 0.5 in each). But when combining all students, the overall correlation is r = -0.2. How is this possible? > > <details> <strong>Solution</strong> > > **This is Simpson's Paradox for correlation.** It happens when: > > - **Program A** is harder (lower grades for the same study hours) AND has students who study more > - **Program B** is easier (higher grades) AND has students who study less > > Within each program: more study → higher grades ✅ Between programs: Program A students study more but get lower grades (because the program is harder) > > When you combine them, the between-program difference dominates, creating the illusion of a negative correlation. > > **Visualized:** > > ```pseudo > Grades ↑ > | ·· (Program B, easy, less study, higher grades) > |·· > | > | ·· (Program A, hard, more study, lower grades) > | ·· > |________________→ Hours studied > ``` > > **Lesson:** Check for subgroups before interpreting overall correlation! </details> > **Q10: What r Doesn't Tell You** > > Match each dataset to the correct r (choose from: r = 0, r = 0.5, r = 0.9, r = -0.5): > > a) Points forming a tight upward-sloping line b) Points forming a loose cloud, slightly upward c) Points forming a perfectly symmetric U-shape d) Points forming a slight downward slope > > <details> <strong>Solution</strong> > > a) **r = 0.9** — tight upward trend → strong positive correlation > > b) **r = 0.5** — loose upward trend → moderate positive correlation > > c) **r ≈ 0** — perfectly symmetric U-shape → no linear relationship (but clear non-linear relationship!) > > d) **r = -0.5** — slight downward slope → moderate negative correlation </details> * * * ## 🔗 Cross-References - **Next topic:** [Counting — Fundamental Principle of Counting](/notes/01-foundation-bsma1002-stats-1-week05-08-counting-fundamentals) — shifting gears to combinatorics, the foundation of probability - **Previous:** [Contingency Tables](/notes/01-foundation-bsma1002-stats-1-week04-06-contingency-tables) — association for categorical data - **Week 10 (Expectation):** Covariance and correlation for random variables - **BSMA1004 (Stats 2):** Inference for correlation, multiple regression - **BSCS2004 (ML Foundations):** Feature correlation, multicollinearity - **BSMA1001-maths-1:** Functions and linear relationships [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Contingency Tables**](/notes/01-foundation-bsma1002-stats-1-week04-06-contingency-tables)[Next**Counting Fundamentals**](/notes/01-foundation-bsma1002-stats-1-week05-08-counting-fundamentals)