Learning Objectives

• Perform comprehensive EDA
• Visualize data distributions
• Identify patterns and anomalies

python
Run Inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load data
df = pd.read_csv('data/raw/dataset.csv')
# Basic info
print(df.info())
print(df.describe())
print(df.isnull().sum())
# Univariate analysis
fig, axes = plt.subplots(2, 2)
df['feature'].hist(ax=axes[0,0])
df['feature'].plot(kind='box', ax=axes[0,1])
sns.countplot(data=df, x='categorical', ax=axes[1,0])
axes[1,1].axis('off')
plt.tight_layout()
# Bivariate analysis
sns.scatterplot(data=df, x='feature1', y='feature2', hue='target')
sns.heatmap(df.corr(), annot=True, cmap='coolwarm')

1. Data types and missing values
2. Distribution of each feature
3. Target variable distribution (class balance)
4. Correlations between features
5. Outliers detection
6. Patterns across time (if temporal)

Q1: Why is EDA important?

Reveals data quality issues, outliers, missing patterns, feature distributions, relationships. Guides preprocessing and feature engineering. Q2: How to detect outliers?

Z-score (>3 std from mean), IQR method (below Q1-1.5_IQR or above Q3+1.5_IQR), visualization (box plots, scatter plots). Q3: What to do with missing values?

Drop (if few), impute (mean/median/mode), predict from other features, create "missing" indicator. Choice depends on missing mechanism (MCAR, MAR, MNAR). Q4: How to handle imbalanced classes?

Resampling (SMOTE, RandomUnderSampler), class weights, anomaly detection algorithms, collect more data, use proper metrics (F1, ROC-AUC, precision-recall). Join Discord PreviousMilestone 1: Problem Definition & Data CollectionNextMilestone 3: Data Preprocessing & Feature Engineering