Pandas for Deep Learning: DataFrames, Series, and GPU-Accelerated Data Processing

🎯 Learning Objectives

• Use Pandas DataFrames and Series for efficient data manipulation
• Perform groupby, apply, and aggregation operations for feature engineering
• Integrate Pandas with PyTorch/TensorFlow data pipelines
• Use GPU-accelerated data processing (cuDF) for large datasets
• Work with sklearn's dataset API for quick prototyping

📋 Prerequisites

• Python basics: Lists, dictionaries, functions
• PyTorch basics (Week 1): Tensor operations
• Data Pipelines (Week 2): Dataset and DataLoader concepts

1. 📖 Core Content

1.1 Intuition: Why Pandas for DL?

1. Load CSV/JSON/Parquet files
2. Clean missing values
3. Encode categorical features
4. Normalize numerical features
5. Create train/val/test splits
6. Convert to PyTorch tensors Pandas is the swiss army knife for all of this. It handles 90% of data preprocessing tasks with concise, readable code.

1.2 Pandas Fundamentals for DL

1.2.1 Loading Data

python
Run Inline
# runnable
import pandas as pd
import numpy as np
# From CSV
df = pd.read_csv('data.csv')
# From Parquet (faster for large datasets)
df = pd.read_parquet('data.parquet')
# From SQL
import sqlite3
conn = sqlite3.connect('database.db')
df = pd.read_sql('SELECT * FROM images', conn)
# Quick check
print(df.shape)         # (rows, columns)
print(df.head())        # First 5 rows
print(df.info())        # Types, non-null counts
print(df.describe())    # Summary statistics

1.2.2 Handling Missing Data

python
Run Inline
# runnable
import pandas as pd
import numpy as np
df = pd.DataFrame({
    'age': [25, 30, np.nan, 35, 28],
    'income': [50000, 60000, 75000, np.nan, 55000],
    'label': [0, 1, 0, 1, np.nan]
})
# Check missing values
print(df.isnull().sum())
# Fill strategies for ML
# For numerical features: median (robust to outliers)
df['age'] = df['age'].fillna(df['age'].median())
df['income'] = df['income'].fillna(df['income'].median())
# For categorical features: mode
# For label: drop rows with missing labels (can't train on them)
df = df.dropna(subset=['label'])
print(df)

1.2.3 Categorical Encoding

python
Run Inline
# runnable
import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
df = pd.DataFrame({
    'city': ['NYC', 'LA', 'SF', 'NYC', 'LA'],
    'price': [100, 200, 150, 120, 180]
})
# Label encoding (for ordinal categories)
le = LabelEncoder()
df['city_label'] = le.fit_transform(df['city'])
print("Label encoded:", df'city', 'city_label'.drop_duplicates())
# One-hot encoding (for nominal categories)
df_onehot = pd.get_dummies(df, columns=['city'], prefix='city')
print("\nOne-hot encoded:", df_onehot)

1.3 Groupby and Aggregation for Feature Engineering

python
Run Inline
# runnable
import pandas as pd
import numpy as np
# Transaction data: compute user-level features
transactions = pd.DataFrame({
    'user_id': [1, 1, 1, 2, 2, 3],
    'amount': [100, 200, 50, 300, 150, 500],
    'category': ['food', 'travel', 'food', 'travel', 'food', 'travel'],
    'timestamp': pd.date_range('2024-01-01', periods=6, freq='D')
})
# User-level aggregate features
user_features = transactions.groupby('user_id').agg({
    'amount': ['sum', 'mean', 'std', 'count'],
    'category': lambda x: x.nunique()
}).reset_index()
# Flatten column names
user_features.columns = ['user_id', 'total_spend', 'avg_spend',
                         'std_spend', 'txn_count', 'unique_categories']
print(user_features)

1.4 Pandas + PyTorch Integration

python
Run Inline
# runnable
import pandas as pd
import torch
from torch.utils.data import Dataset
class PandasDataset(Dataset):
    def __init__(self, df: pd.DataFrame, feature_cols, label_col):
        self.features = torch.tensor(df[feature_cols].values, dtype=torch.float32)
        self.labels = torch.tensor(df[label_col].values, dtype=torch.float32)
    def __len__(self):
        return len(self.features)
    def __getitem__(self, idx):
        return self.features[idx], self.labels[idx]
# Example usage
df = pd.DataFrame({
    'feature_1': np.random.randn(1000),
    'feature_2': np.random.randn(1000),
    'label': np.random.randint(0, 2, 1000)
})
dataset = PandasDataset(df, ['feature_1', 'feature_2'], 'label')
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)
for batch_x, batch_y in dataloader:
    print(f"Batch: X shape {batch_x.shape}, y shape {batch_y.shape}")
    break

1.5 GPU-Accelerated Pandas with cuDF

python
Run Inline
# runnable
# Note: cuDF requires NVIDIA GPU with RAPIDS installed
# import cudf
#
# gpu_df = cudf.read_csv('large_dataset.csv')
# gpu_df['features'] = gpu_df.groupby('user_id')['amount'].transform('mean')
# gpu_tensor = torch.as_tensor(gpu_df.to_cupy())  # Zero-copy to PyTorch
# CPU fallback for this example:
print("For >10M rows, use cuDF for 10-50x speedup over Pandas")

1.6 Worked Example: End-to-End Data Pipeline

python
Run Inline
# runnable
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import torch
# Step 1: Create synthetic data
np.random.seed(42)
n_samples = 10000
df = pd.DataFrame({
    'age': np.random.randint(18, 80, n_samples),
    'income': np.random.lognormal(mean=10, sigma=1, size=n_samples),
    'education_level': np.random.choice(['HS', 'BS', 'MS', 'PhD'], n_samples),
    'city': np.random.choice(['NYC', 'LA', 'SF', 'CHI'], n_samples),
    'target': np.random.randint(0, 2, n_samples)
})
print(f"Original shape: {df.shape}")
print(df.head())
# Step 2: Handle missing values (none in this synthetic data)
# Step 3: Encode categorical features
df = pd.get_dummies(df, columns=['education_level', 'city'], drop_first=True)
# Step 4: Split features and target
X = df.drop('target', axis=1)
y = df['target']
# Step 5: Train/val/test split
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)
print(f"Train: {X_train.shape}, Val: {X_val.shape}, Test: {X_test.shape}")
# Step 6: Normalize numerical features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)
X_test_scaled = scaler.transform(X_test)
# Step 7: Convert to PyTorch tensors
train_dataset = torch.utils.data.TensorDataset(
    torch.tensor(X_train_scaled, dtype=torch.float32),
    torch.tensor(y_train.values, dtype=torch.float32)
)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True)
print(f"Ready for training: {len(train_loader)} batches per epoch")

1.7 Why This Matters

• Faster prototyping (no waiting for ETL pipelines)
• Reproducible preprocessing (code, not manual steps)
• Integration with any deep learning framework

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Data Leakage Through Improper Split

Pitfall 2: Not Handling Categorical Variables

Pitfall 3: Using Pandas for GPU Training Data Loading

4. 📝 Practice Questions

Q1: Load a CSV with columns [age, income, city, purchased]. Handle missing ages by filling with median. One-hot encode city. Split 70/15/15. Write the full pipeline.

python
Run Inline
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load
df = pd.read_csv('data.csv')

# Handle missing
df['age'] = df['age'].fillna(df['age'].median())

# One-hot encode
df = pd.get_dummies(df, columns=['city'], drop_first=True)

# Split
X = df.drop('purchased', axis=1)
y = df['purchased']

X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

# Scale
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_val = scaler.transform(X_val)
X_test = scaler.transform(X_test)

Q2: You have transaction data (user_id, amount, timestamp). Create features: total amount in last 7 days, transaction count in last 7 days, days since last transaction.

python
Run Inline
import pandas as pd

df = pd.read_csv('transactions.csv')
df['timestamp'] = pd.to_datetime(df['timestamp'])
df = df.sort_values(['user_id', 'timestamp'])

# Window features (last 7 days per row)
def compute_window_features(group):
    group = group.sort_values('timestamp')
    group['total_7d'] = group['amount'].rolling('7D', on='timestamp').sum()
    group['count_7d'] = group['amount'].rolling('7D', on='timestamp').count()
    group['days_since_last'] = group['timestamp'].diff().dt.days.fillna(999)
    return group

df = df.groupby('user_id').apply(compute_window_features)

5. 🔗 Cross-References

• Previous: Data Loading & Pipelines (Week 2)
• Next: Training & Debugging (Week 4)
• Related: Distributed Training (Week 5) — Scaling data loading
• External: Pandas documentation (pandas.pydata.org) Join Discord PreviousData PipelinesNextTraining & Debugging