A/B Testing & Experimentation

🎯 Learning Objectives

• Design rigorous A/B tests with proper sample size
• Understand p-values, confidence intervals, and statistical power
• Avoid common A/B testing pitfalls
• Analyze A/B test results correctly

📖 Core Content

2.1 The A/B Testing Framework

2.2 Hypothesis Testing

• $H_0$ (Null): No difference between A and B ($\mu_A = \mu_B$)
• $H_1$ (Alternative): There is a difference ($\mu_A \neq \mu_B$) Error types:
• Type I (α): False positive — concluding difference when none exists (typically α = 0.05)
• Type II (β): False negative — failing to detect a real difference
• Power (1-β): Probability of detecting a real effect (typically 0.80)

2.3 Sample Size Calculation

python
Run Inline
# runnable
import numpy as np
from scipy import stats
def sample_size(effect_size, std, alpha=0.05, power=0.80):
    z_alpha = stats.norm.ppf(1 - alpha/2)
    z_beta = stats.norm.ppf(power)
    n = (2 * std**2 * (z_alpha + z_beta)**2) / effect_size**2
    return int(np.ceil(n))
# Example: detect 5% conversion increase from 10% baseline
baseline = 0.10
effect = 0.05  # absolute increase to 0.15
std = np.sqrt(baseline * (1 - baseline))  # approx
n = sample_size(effect, std)
print(f"Need {n} users per variant to detect {effect:.0%} lift")

2.4 Common Pitfalls

📝 Practice Questions

Q1: A/B test runs for 2 days, p=0.04 on day 1, p=0.30 on day 2. Which is correct?

Day 2 result (p=0.30) — you must wait for the planned sample size. The day 1 result was likely noise (Type I error). Peeking at results and stopping early inflates false positive rates dramatically. Pre-register: "test will run for 2 weeks or until n=10,000 per variant." Q2: What does p=0.03 mean?

If the null hypothesis is true (no real difference), there's a 3% probability of observing a difference this large or larger purely by chance. It does NOT mean "97% chance the treatment is better." Common misinterpretation — p-value is about data given null, not about null given data. Q3: Your A/B test shows statistical significance (p=0.01) but the effect size is 0.1% improvement. What do you do?

Consider practical significance. A 0.1% improvement may not be worth the implementation cost, risk, or complexity. Even though the result is statistically significant, the business impact may be negligible. Always ask: "Is this effect big enough to matter?" Join Discord PreviousCourse OverviewNextCustomer Analytics