Feature Engineering: Creating, Transforming, and Selecting Features
419 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Feature Engineering: Creating, Transforming, and Selecting Features ## 🎯 Learning Objectives - Create informative features from raw data - Apply transformations (log, Box-Cox, polynomial) for better model performance - Select the most important features using multiple methods - Automate feature engineering with F...
Feature Engineering: Creating, Transforming, and Selecting Features
🎯 Learning Objectives
- Create informative features from raw data
- Apply transformations (log, Box-Cox, polynomial) for better model performance
- Select the most important features using multiple methods
- Automate feature engineering with Featuretools
📖 Core Content
1.1 Intuition: Why Feature Engineering?
Raw data is rarely ready for ML. Feature engineering transforms raw data into features that ML algorithms can learn from effectively. It's often the difference between a good model and a great one — many Kaggle competitions are won on feature engineering, not model architecture.
1.2 Feature Creation Techniques
| Technique | Description | Example |
|---|---|---|
| Domain features | Features from domain knowledge | Credit score from income/debt ratio |
| Aggregation | Groupby statistics | Avg purchase amount per user |
| Time-based | Day of week, hour, season | "Is weekend?" flag |
| Text features | Word count, TF-IDF | Review length sentiment |
| Interaction | Feature products | Age × Income interaction |
| Binning | Continuous → categorical | Age group: 18-25, 26-35, ... |
1.3 Feature Scaling
python# runnable from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler import numpy as np data = np.array([100, 0.001], [200, 0.01], [300, 0.1], [500, 1.0](/courses/bscs2008/notes/100%2C%200.001%5D%2C%20%5B200%2C%200.01%5D%2C%20%5B300%2C%200.1%5D%2C%20%5B500%2C%201.0)) # StandardScaler (z-score): best for linear models, neural networks scaler = StandardScaler() standardized = scaler.fit_transform(data) print(f"Standardized mean: {standardized.mean(axis=0)}") print(f"Standardized std: {standardized.std(axis=0)}") # MinMaxScaler: best for bounded data, neural nets minmax = MinMaxScaler() normalized = minmax.fit_transform(data) print(f"MinMax range: [{normalized.min():.2f}, {normalized.max():.2f}]") # RobustScaler: robust to outliers robust = RobustScaler() robust_scaled = robust.fit_transform(data)
1.4 Feature Selection Methods
| Method | Type | How It Works | Best For |
|---|---|---|---|
| Variance threshold | Filter | Remove low-variance features | Quick pruning |
| Correlation analysis | Filter | Remove highly correlated features | Redundancy reduction |
| Chi-squared test | Filter | Test independence with target | Classification |
| Mutual information | Filter | Non-linear dependency with target | General purpose |
| RFE | Wrapper | Recursively remove least important | Small feature sets |
| Lasso (L1) | Embedded | Shrinks coefficients to zero | Linear models |
| Tree importance | Embedded | Feature importance from trees | Non-linear models |
1.5 Why This Matters
Feature engineering is where domain expertise meets ML. It's the most impactful step in the ML pipeline — better features can improve model performance more than any algorithm choice.
2. 📝 Practice Questions
Q1: You have a dataset with 500 features but only 10K rows. Feature selection reduces it to 50. Explain which selection methods you'd use and why.Recommended approach:
- Filter first: Remove low-variance features (variance threshold) and highly correlated features (correlation > 0.95). This is cheap and removes obvious redundancy.
- Embedded: Train a Random Forest (fast to train) and keep only features with non-zero importance. This captures non-linear relationships.
- Optional wrapper: Apply RFE on the remaining ~100 features to get to 50. This is computationally expensive but thorough.
Why this order: Filter is cheapest, embedded adds non-linear signal, wrapper is most expensive. Always start cheap. Join Discord PreviousModel InterpretationNextEvaluation Metrics