Data Cleaning & Preprocessing

🎯 Learning Objectives

• Identify and handle different types of missing data
• Detect and treat outliers using statistical methods
• Resolve data inconsistencies and duplicates
• Build robust data cleaning pipelines

📖 Core Content

2.1 Missing Data Mechanisms

2.2 Imputation Strategies

python
Run Inline
# runnable
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer, KNNImputer
# Create data with missing values
data = pd.DataFrame({
    'A': [1, 2, np.nan, 4, 5],
    'B': [np.nan, 3, 4, 5, 6],
    'C': [100, 200, 300, np.nan, 500]
})
# Mean imputation
mean_imputer = SimpleImputer(strategy='mean')
data_mean = pd.DataFrame(mean_imputer.fit_transform(data), columns=data.columns)
print("Mean imputation:\n", data_mean)
# Median imputation
median_imputer = SimpleImputer(strategy='median')
data_median = pd.DataFrame(median_imputer.fit_transform(data), columns=data.columns)
print("\nMedian imputation:\n", data_median)
# KNN imputation
knn_imputer = KNNImputer(n_neighbors=2)
data_knn = pd.DataFrame(knn_imputer.fit_transform(data), columns=data.columns)
print("\nKNN imputation:\n", data_knn)

2.3 Outlier Detection

python
Run Inline
# runnable
import numpy as np
# IQR method
data = np.random.randn(100)
data = np.append(data, [15, -12])  # Add outliers
Q1, Q3 = np.percentile(data, [25, 75])
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = data[(data < lower) | (data > upper)]
print(f"Found {len(outliers)} outliers using IQR method")
print(f"Outliers: {outliers}")

2.4 Key Decisions

📝 Practice Questions

Q1: Why not always drop rows with missing values?

Dropping rows reduces sample size (power), and if data is not MCAR, it introduces bias. For example, if wealthy people skip income questions, dropping them removes the wealthy from analysis. Better to impute thoughtfully. Q2: When would you use median vs mean imputation?

Median is robust to outliers; mean is not. For skewed distributions (income, house prices), use median. For symmetric distributions (height, temperature), mean is fine. Always check distribution before choosing. Q3: What's the danger of imputing a large proportion of missing values?

If > 30-40% of a feature is missing, imputation introduces significant uncertainty. The imputed values are "fake" — relationships inferred from imputed data may be artifacts. Consider whether the feature is worth keeping. Join Discord PreviousCourse OverviewNextEDA & Visualization