Concentration Inequalities

348 words

2 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# Concentration Inequalities ## 🎯 Learning Objectives - State Markov's, Chebyshev's, and Chernoff's inequalities - Apply concentration bounds to analyze randomized algorithms - Derive tail bounds for sums of independent random variables - Understand the trade-off between tail decay and moment assumptions * * * ## 1...

Concentration Inequalities

🎯 Learning Objectives

State Markov's, Chebyshev's, and Chernoff's inequalities
Apply concentration bounds to analyze randomized algorithms
Derive tail bounds for sums of independent random variables
Understand the trade-off between tail decay and moment assumptions

1.1 Intuition: How Fast Does a Random Variable Concentrate?

Concentration inequalities quantify how quickly a random variable approaches its mean. They're the foundation of randomized algorithm analysis and statistical learning theory.

🔑 Key Insight: Under weak assumptions (finite variance → Chebyshev; boundedness → Hoeffding; subgaussian → Chernoff), sums of independent random variables concentrate exponentially fast around their mean.

1.2 Key Inequalities

Markov's Inequality

\Pr(X \geq t) \leq \frac{E[X]}{t} \quad \text{for } X \geq 0

Example: If average income is

50k, at most 1/4 of people earn

\geq 200k$.

Chebyshev's Inequality

\Pr(|X - E[X]| \geq t) \leq \frac{\text{Var}(X)}{t^2}

Example: For

\bar{X}

from

n

i.i.d. samples:

\Pr(|\bar{X} - \mu| \geq \epsilon) \leq \frac{\sigma^2}{n\epsilon^2}

Chernoff Bound (Multiplicative)

For

X = \sum X_i

where

X_i \in [0,1]

independent:

\Pr(X \geq (1+\delta)\mu) \leq \left(\frac{e^\delta}{(1+\delta)^{1+\delta}}\right)^\mu

Hoeffding's Inequality

For

X_i \in [a_i, b_i]

independent:

\Pr(|\bar{X} - E[\bar{X}]| \geq t) \leq 2\exp\left(-\frac{2n^2 t^2}{\sum(b_i - a_i)^2}\right)

1.3 Comparison

Bound	Assumptions	Decay	Use Case
Markov	$X \geq 0$	$1/t$	Generic
Chebyshev	Finite variance	$1/t^2$	Generic with variance
Chernoff	Independent, bounded	$\exp(-c\mu\delta^2)$	Sums of bounded variables
Hoeffding	Independent, bounded	$\exp(-2nt^2/(b-a)^2)$	Sample means

✅ Practice Questions

Q1: Use Chebyshev to bound

\Pr(|\bar{X} - \mu| \geq \epsilon)

for

n

i.i.d. samples with variance

\sigma^2

Solution

$\text{Var}(\bar{X}) = \sigma^2/n$ . By Chebyshev: $\Pr(|\bar{X} - \mu| \geq \epsilon) \leq \frac{\sigma^2}{n\epsilon^2}$ . Q2: Compare Hoeffding and Chebyshev bounds. Which is tighter for bounded variables? Solution

Hoeffding decays as $\exp(-cn\epsilon^2)$ , exponentially faster than Chebyshev's $1/(n\epsilon^2)$ . For bounded variables, Hoeffding is exponentially tighter. Q3: How many samples $n$ needed to estimate a mean within $\pm 0.01$ with 95% confidence for bounded $[0,1]$ variables? Solution

Hoeffding: $2\exp(-2n(0.01)^2) \leq 0.05$ . So $\exp(-0.0002n) \leq 0.025$ , $-0.0002n \leq \ln(0.025)$ , $n \geq \ln(40)/0.0002 \approx 18449$ . Join Discord NextRandomized SVD