Dispersion and Percentiles — Measuring Spread

🎯 Learning Objectives

• Explain why measuring spread is as important as measuring central tendency
• Compute range, variance, standard deviation, and interquartile range (IQR) for raw data
• Find and interpret percentiles and the five-number summary
• Create and interpret box plots
• Choose the appropriate measure of dispersion based on distribution shape
• Detect outliers using the IQR method

📋 Prerequisites

• Central Tendency (03-central-tendency) — understanding the mean and median is essential
• Data Types & Scales (01-data-types-scales) — variance requires interval/ratio data
• Basic algebra: squaring, square roots, summation notation

---

📖 Core Content

5.1 Intuition: Why "Average" Isn't Enough

• Are most values close to the mean, or far away?
• How much variability is there in the data?
• What's the range of values?

Everyday analogy: Two friends each make ₹50,000/month on average. One is a salaried employee (always ₹50K). The other is a freelancer (₹0 one month, ₹100K the next). The average is the same, but the spread tells a completely different story about financial stability. 🔑 Key Insight: Central tendency tells you where the data is centered. Dispersion tells you how much the data varies. You need both to understand a dataset.

5.2 Range

5.2.1 Intuition

5.2.2 Definition

$$\text{Range} = \text{Maximum Value} - \text{Minimum Value}$$

5.2.3 Example

• Max = 100, Min = 10
• Range = 100 - 10 = 90 Limitation: The range only uses the two most extreme values. It ignores all the data in between. A single outlier can dramatically inflate the range.

5.3 Variance and Standard Deviation

5.3.1 Intuition

5.3.2 Formal Definitions

$$\sigma^2 = \frac{\sum_{i=1}^{N} (x_i - \mu)^2}{N}$$ $$s^2 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n - 1}$$ $$\sigma = \sqrt{\sigma^2} = \sqrt{\frac{\sum_{i=1}^{N} (x_i - \mu)^2}{N}}$$ $$s = \sqrt{s^2} = \sqrt{\frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n - 1}}$$

Why divide by n-1 for the sample? This is called Bessel's correction. Dividing by n would give a biased estimate (consistently too small) of the population variance. Using n-1 corrects this bias. For this course: use n-1 for sample, N for population.

5.3.3 Step-by-Step Calculation

5.3.4 Computational Formula (for efficiency)

$$s^2 = \frac{\sum x_i^2 - \frac{(\sum x_i)^2}{n}}{n - 1}$$

• $\sum x_i = 26$
• $\sum x_i^2 = 4^2 + 8^2 + 6^2 + 5^2 + 3^2 = 16 + 64 + 36 + 25 + 9 = 150$
• $s^2 = \frac{150 - \frac{(26)^2}{5}}{4} = \frac{150 - \frac{676}{5}}{4} = \frac{150 - 135.2}{4} = \frac{14.8}{4} = 3.7$ ✅

5.3.5 Properties of Variance and Standard Deviation

5.4 Percentiles

5.4.1 Intuition

5.4.2 Definition

$$\text{Position} = \frac{k}{100} \times (n + 1)$$

5.4.3 Special Percentiles

5.4.4 Worked Example

• Position = $(10+1)/2 = 5.5$
• Q2 = average of 5th and 6th values
• Q2 = $(22 + 25)/2 = 23.5$ Q1:
• Position = $(10+1)/4 = 2.75$
• 2nd value = 15, 3rd value = 18
• Q1 = $15 + 0.75(18 - 15) = 15 + 0.75(3) = 15 + 2.25 = 17.25$ Q3:
• Position = $3(10+1)/4 = 8.25$
• 8th value = 30, 9th value = 35
• Q3 = $30 + 0.25(35 - 30) = 30 + 0.25(5) = 30 + 1.25 = 31.25$

5.5 Interquartile Range (IQR)

5.5.1 Intuition

5.5.2 Definition

$$\text{IQR} = Q_3 - Q_1$$

5.5.3 Example

5.6 Outlier Detection with IQR

• Below: $Q_1 - 1.5 \times \text{IQR}$
• Above: $Q_3 + 1.5 \times \text{IQR}$ Example: Using our data:
• Lower fence = $17.25 - 1.5(14) = 17.25 - 21 = -3.75$
• Upper fence = $31.25 + 1.5(14) = 31.25 + 21 = 52.25$ Any value below -3.75 or above 52.25 is an outlier. In this dataset, all values are within fences — no outliers.

5.7 Five-Number Summary and Box Plots

5.7.1 Five-Number Summary

1. Minimum
2. Q1 (25th percentile)
3. Q2 (Median)
4. Q3 (75th percentile)
5. Maximum Example (our data): Min = 12, Q1 = 17.25, Median = 23.5, Q3 = 31.25, Max = 40

5.7.2 Box Plot (Box-and-Whisker Plot)

• The box spans from Q1 to Q3 (contains the middle 50%)
• The line inside the box is the median
• The whiskers extend to min and max (or to the fences, with outliers plotted as individual points)

5.8 Choosing the Right Measure of Dispersion

5.9 Coefficient of Variation (CV)

5.9.1 Intuition

5.9.2 Definition

$$\text{CV} = \frac{s}{\bar{x}} \times 100\%$$

5.9.3 Example

5.10 Worked Examples

Example 1: Complete Analysis (Easy)

$$\bar{x} = \frac{65+70+72+75+78+80+82+85+90+95}{10} = \frac{792}{10} = 79.2$$

• $\sum x_i = 792$
• $\sum x_i^2 = 65^2 + 70^2 + ... + 95^2 = 4225+4900+5184+5625+6084+6400+6724+7225+8100+9025 = 63492$
• $s^2 = \frac{63492 - \frac{(792)^2}{10}}{9} = \frac{63492 - \frac{627264}{10}}{9} = \frac{63492 - 62726.4}{9} = \frac{765.6}{9} = 85.07$ Standard deviation: $s = \sqrt{85.07} \approx 9.22$ Five-number summary:
• Min = 65
• Q1 position = (10+1)/4 = 2.75 → Q1 = 70 + 0.75(72-70) = 70 + 1.5 = 71.5
• Q2 position = 5.5 → Q2 = (78+80)/2 = 79
• Q3 position = 3(10+1)/4 = 8.25 → Q3 = 85 + 0.25(90-85) = 85 + 1.25 = 86.25
• Max = 95 IQR: 86.25 - 71.5 = 14.75

Example 2: Outlier Detection (Medium)

• n = 15
• Min = 22
• Q1 position = (15+1)/4 = 4 → Q1 = 4th value = 30
• Q2 = median, position = (15+1)/2 = 8 → Q2 = 8th value = 38
• Q3 position = 3(15+1)/4 = 12 → Q3 = 12th value = 48
• Max = 120 Step 2: IQR = 48 - 30 = 18 Step 3: Fences
• Lower: Q1 - 1.5(IQR) = 30 - 1.5(18) = 30 - 27 = 3
• Upper: Q3 + 1.5(IQR) = 48 + 1.5(18) = 48 + 27 = 75 Step 4: Check each value
• 120 > 75 → Outlier!
• All other values are between 22 and 55 (within fences) Conclusion: ₹120K is an outlier. It's likely the CEO's salary.

Example 3: Comparing Two Distributions (Harder)

• Class A (s=5): Scores are tightly clustered around 75. Most students scored between 70-80 (±1 s.d.). Very consistent performance.
• Class B (s=15): Scores are widely spread. Most students scored between 60-90 (±1 s.d.). More variability — some students did much better, some much worse. b) Student preference:
• If you're an above-average student, you might prefer Class B — you have a chance to shine (score 90+).
• If you're a below-average student, you might prefer Class A — you're likely close to the mean and won't stand out as much.
• If you're an average student, you'd probably prefer Class A — the experience is more predictable.

5.11 Edge Cases & Gotchas

All Identical Values

Single Value

When Variance Is Large

Empirical Rule (68-95-99.7)

• 68% of values fall between 70 and 80
• 95% fall between 65 and 85
• 99.7% fall between 60 and 90 This is a powerful rule of thumb, but it only applies to roughly normal distributions!

5.12 Why This Matters

• Finance: Standard deviation measures risk (volatility of returns)
• Quality control: Variance tells you if manufacturing is consistent
• Education: Variance in test scores shows teaching effectiveness
• Sports: A consistent player (low variance) vs. an inconsistent one (high variance) Dispersion also connects directly to upcoming topics:
• Week 10 (Expectation & Variance): Formalizing variance for random variables
• Week 12 (Normal Distribution): The empirical rule and Z-scores
• BSMA1004 (Stats 2): Hypothesis tests compare means relative to variability

---

📐 Key Formulas / Concepts

---

⚠️ Common Pitfalls

Pitfall 1: Confusing Variance and Standard Deviation

Pitfall 2: Using n Instead of n-1 for Sample Variance

Pitfall 3: Forgetting That Range Ignores Most Data

Pitfall 4: Applying the Empirical Rule to Non-Normal Data

---

📝 Practice Questions

Q1: Range

Find the range of: 3, 15, 8, 22, 7, 31, 12

<details> <strong>Solution</strong>

Max = 31, Min = 3 Range = 31 - 3 = 28

$\boxed{\text{Range} = 28}$

</details> > **Q2: Variance and Standard Deviation** > > Find the sample variance and standard deviation of: 5, 7, 9, 11, 13 > > <details> <strong>Solution</strong> > > **Step 1:** n = 5, $\bar{x} = (5+7+9+11+13)/5 = 45/5 = 9$ > > **Step 2:** $\sum x_i^2 = 25+49+81+121+169 = 445$ > > **Step 3:** $(\sum x_i)^2 / n = 45^2 / 5 = 2025/5 = 405$ > > **Step 4:** $s^2 = \frac{445 - 405}{4} = \frac{40}{4} = 10$ > > **Step 5:** $s = \sqrt{10} \approx 3.162$ > > $\boxed{s^2 = 10,\quad s \approx 3.162}$ </details> > **Q3: Five-Number Summary** > > Find the five-number summary for: 2, 5, 7, 8, 10, 12, 15, 18, 20, 25, 30 > > <details> <strong>Solution</strong> > > n = 11 > > - **Min** = 2 > - **Q1:** Position = (11+1)/4 = 3 → 3rd value = 7 > - **Q2 (Median):** Position = (11+1)/2 = 6 → 6th value = 12 > - **Q3:** Position = 3(11+1)/4 = 9 → 9th value = 20 > - **Max** = 30 > > $\boxed{2,\ 7,\ 12,\ 20,\ 30}$ </details> > **Q4: IQR and Outliers** > > Dataset: 10, 12, 14, 15, 18, 20, 22, 25, 30, 35, 60 > > Find Q1, Q3, IQR, and determine if 60 is an outlier. > > <details> <strong>Solution</strong> > > n = 11 > > **Q1:** Position = 3 → Q1 = 14 **Q3:** Position = 9 → Q3 = 30 **IQR:** 30 - 14 = 16 > > **Upper fence:** Q3 + 1.5(IQR) = 30 + 1.5(16) = 30 + 24 = 54 > > **Check:** 60 > 54 → **60 is an outlier** ✅ > > **Lower fence:** Q1 - 1.5(IQR) = 14 - 24 = -10 > > No values below -10, so no lower outliers. > > $\boxed{Q_1=14,\ Q_3=30,\ \text{IQR}=16,\ \text{60 is an outlier}}$ </details> > **Q5: Coefficient of Variation** > > Which dataset has more relative variability? > > Dataset A: $\bar{x} = 50$, $s = 10$ Dataset B: $\bar{x} = 200$, $s = 30$ > > <details> <strong>Solution</strong> > > **Dataset A:** CV = 10/50 × 100% = 20% **Dataset B:** CV = 30/200 × 100% = 15% > > **Dataset A has more relative variability** (20% > 15%), even though its absolute standard deviation is smaller. > > Interpretation: Dataset A's spread is larger relative to its mean. </details> > **Q6: Effect of Adding Constant on Variance** > > If you add 10 to every value in a dataset, what happens to the variance and standard deviation? > > <details> <strong>Solution</strong> > > **They don't change!** > > Adding a constant shifts all values, so the mean shifts by the same amount. But the **deviations** $(x_i - \bar{x})$ stay the same because both $x_i$ and $\bar{x}$ increase by 10. Since variance depends on these deviations, it's unchanged. > > **Example:** Original {1, 2, 3}: $\bar{x}=2$, $s^2=1$. Add 10: {11, 12, 13}: $\bar{x}=12$, $s^2=1$. > > $\boxed{\text{Variance and standard deviation are unchanged}}$ </details> > **Q7: Effect of Multiplying on Variance** > > If you multiply every value by 3, what happens to the variance and standard deviation? > > <details> <strong>Solution</strong> > > **Variance is multiplied by $3^2 = 9$. Standard deviation is multiplied by 3.** > > Proof: If $y_i = c \cdot x_i$, then $\bar{y} = c \cdot \bar{x}$. $s_y^2 = \frac{\sum(cx_i - c\bar{x})^2}{n-1} = \frac{c^2\sum(x_i - \bar{x})^2}{n-1} = c^2 \cdot s_x^2$ > > So $s_y = c \cdot s_x$. > > **Example:** Original {1, 2, 3}: $s^2=1$. Multiply by 3: {3, 6, 9}: $s^2=9$. > > $\boxed{s_y^2 = c^2 s_x^2,\quad s_y = c s_x}$ </details> > **Q8: Box Plot Interpretation** > > A box plot shows: Min=5, Q1=15, Median=25, Q3=40, Max=80. a) What is the IQR? b) Are there likely outliers on the upper end? c) Describe the shape (symmetric or skewed?). > > <details> <strong>Solution</strong> > > **a) IQR:** 40 - 15 = 25 > > **b) Upper fence:** 40 + 1.5(25) = 40 + 37.5 = 77.5 Max = 80 > 77.5 → There is at least one outlier above 77.5. > > (Or the maximum value 80 is itself an outlier, and there may be other values near it.) > > **c) Shape:** The right whisker (40 to 80, length = 40) is much longer than the left whisker (5 to 15, length = 10). The box (15-40) has Q3- median = 15 and median - Q1 = 10, so the right half is slightly longer. This indicates **right-skewness** (positive skew). > > $\boxed{\text{IQR}=25,\ \text{Upper outliers likely},\ \text{Right-skewed}}$ </details> > **Q9: Complete Analysis** > > Dataset: 8, 12, 15, 18, 20, 22, 24, 26, 28, 35 > > Compute: mean, median, range, variance, standard deviation, Q1, Q3, IQR. Check for outliers. > > <details> <strong>Solution</strong> > > **Mean:** $\bar{x} = (8+12+15+18+20+22+24+26+28+35)/10 = 208/10 = 20.8$ > > **Median:** n=10, positions 5 and 6: (20+22)/2 = 21 > > **Range:** 35 - 8 = 27 > > **Variance (computational):** > > - $\sum x_i = 208$ > - $\sum x_i^2 = 64+144+225+324+400+484+576+676+784+1225 = 4902$ > - $s^2 = \frac{4902 - 208^2/10}{9} = \frac{4902 - 4326.4}{9} = \frac{575.6}{9} \approx 63.96$ > > **Standard deviation:** $s = \sqrt{63.96} \approx 8.00$ > > **Five-number summary:** > > - Min = 8 > - Q1: position = (11)/4 = 2.75 → Q1 = 12 + 0.75(15-12) = 12 + 2.25 = 14.25 > - Q2 = 21 > - Q3: position = 33/4 = 8.25 → Q3 = 26 + 0.25(28-26) = 26 + 0.5 = 26.5 > - Max = 35 > > **IQR:** 26.5 - 14.25 = 12.25 > > **Outlier check:** > > - Lower fence: 14.25 - 1.5(12.25) = 14.25 - 18.375 = -4.125 > - Upper fence: 26.5 + 1.5(12.25) = 26.5 + 18.375 = 44.875 > - All values within fences → **No outliers** > > $\boxed{\bar{x}=20.8,\ \text{Median}=21,\ s\approx8.00,\ \text{IQR}=12.25,\ \text{No outliers}}$ </details> > **Q10: Application** > > Two investment portfolios have the same average annual return of 8%. Portfolio A has a standard deviation of 3%. Portfolio B has a standard deviation of 12%. > > a) Which is riskier? b) In a given year, what range would contain about 95% of returns for Portfolio A (assuming normal distribution)? c) What does this mean for an investor? > > <details> <strong>Solution</strong> > > **a) Portfolio B is riskier** — its returns are more volatile (higher standard deviation). > > **b) Portfolio A (95% range):** Mean ± 2 s.d. = 8% ± 2(3%) = 8% ± 6% Range: 2% to 14% > > So in any given year, there's about a 95% chance the return is between 2% and 14%. > > **c) Investor implications:** > > - Portfolio A: Safer, more predictable returns (range of 2-14% with 95% confidence) > - Portfolio B: Riskier, could have much higher returns (8% + 24% = 32%!) or much lower (8% - 24% = -16% loss!) > - An investor near retirement (risk-averse) would prefer Portfolio A > - A young investor with a long time horizon might choose Portfolio B for higher potential returns </details> * * * ## 🔗 Cross-References - **Next topic:** [Data Visualization](/notes/01-foundation-bsma1002-stats-1-week03-05-data-visualization) — visualizing spread with histograms and box plots - **Previous:** [Central Tendency](/notes/01-foundation-bsma1002-stats-1-week03-03-central-tendency) — measures of center, the foundation for dispersion - **Week 10 (Expectation & Variance):** Formalizing variance for probability distributions - **Week 12 (Normal Distribution):** Z-scores use standard deviation - **BSMA1004 (Stats 2):** Standard deviation in hypothesis testing (t-tests, ANOVA) [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Central Tendency**](/notes/01-foundation-bsma1002-stats-1-week03-03-central-tendency)[Next**Data Visualization**](/notes/01-foundation-bsma1002-stats-1-week03-05-data-visualization)