Multiple & Polynomial Regression

🎯 Learning Objectives

• Formulate multiple linear regression with vectorized notation
• Understand feature scaling and why it's critical for gradient descent
• Implement polynomial regression by adding polynomial features
• Detect when a polynomial model is overfitting
• Compare linear, quadratic, and cubic models using validation curves

📋 Prerequisites

• Linear Regression — the foundation we extend here
• Matrix multiplication — for the vectorized hypothesis
• Gradient descent — we'll use the same optimization

📖 Core Content

3.1 Intuition: Why Multiple Features?

3.2 Multiple Linear Regression — Vectorized Form

3.3 Worked Example 1: Two-Feature Multiple Regression

• $\theta_0 = 45$: baseline score with 0 hours and 0 tests
• $\theta_1 = 5$: each additional hour adds 5 points (holding tests constant)
• $\theta_2 = 10$: each practice test adds 10 points (holding hours constant) Predictions:
• Student A: $45 + 5(2) + 10(1) = 65$ ✓
• Student C: $45 + 5(6) + 10(3) = 105$ — wait, that's > 100! Edge case break: Our model predicts Student C would score 105, but scores are capped at 100. This reveals that the linear relationship breaks down at the boundaries — a limitation of linear models.

3.4 Feature Scaling

3.5 Polynomial Regression

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
# Non-linear data: y = 2 + 3x - 0.5x² + noise
np.random.seed(42)
X = np.linspace(0, 5, 20).reshape(-1, 1)
y = 2 + 3*X.ravel() - 0.5*X.ravel()**2 + np.random.randn(20) * 0.5
# Linear model
linear = LinearRegression()
linear.fit(X, y)
# Polynomial model (degree 2)
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
poly_reg = LinearRegression()
poly_reg.fit(X_poly, y)
print(f"Linear:   y = {linear.intercept_:.2f} + {linear.coef_[0]:.2f}x")
print(f"Polynomial: y = {poly_reg.intercept_:.2f} + {poly_reg.coef_[0]:.2f}x + {poly_reg.coef_[1]:.2f}x²")
# Plot
X_plot = np.linspace(0, 5, 100).reshape(-1, 1)
plt.scatter(X, y, label='Data')
plt.plot(X_plot, linear.predict(X_plot), 'r--', label='Linear')
plt.plot(X_plot, poly_reg.predict(PolynomialFeatures(2).fit_transform(X_plot)), 'g-', label='Quadratic')
plt.legend()
plt.grid(True)
plt.show()

3.6 Worked Example 2: Polynomial Degree Selection

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
# Generate data
np.random.seed(0)
X = np.linspace(0, 3, 15).reshape(-1, 1)
y = np.sin(X).ravel() + np.random.randn(15) * 0.15
degrees = [1, 3, 9, 15]
plt.figure(figsize=(12, 8))
for i, deg in enumerate(degrees):
    poly = PolynomialFeatures(degree=deg, include_bias=False)
    X_poly = poly.fit_transform(X)
    model = LinearRegression()
    model.fit(X_poly, y)
    X_plot = np.linspace(0, 3, 200).reshape(-1, 1)
    y_plot = model.predict(PolynomialFeatures(deg).fit_transform(X_plot))
    plt.subplot(2, 2, i+1)
    plt.scatter(X, y, color='blue', alpha=0.7)
    plt.plot(X_plot, y_plot, 'r-')
    plt.title(f'Degree {deg}')
    plt.grid(True)
plt.tight_layout()
plt.show()

• Degree 1: Underfitting — too simple, misses the curve
• Degree 3: Good fit — captures the sine shape well
• Degree 9: Overfitting — wiggly, memorizes noise
• Degree 15: Severe overfitting — extreme oscillations between points

3.7 The Bias-Variance Tradeoff

3.8 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Not Scaling Features for Gradient Descent

Pitfall 2: Using Too High a Polynomial Degree

Pitfall 3: Extrapolating Polynomials

📝 Practice Questions

Q1: A model has 3 features. How many parameters does it have?

Answer: 4 parameters — $\theta_0$ (intercept) + $\theta_1, \theta_2, \theta_3$ (one per feature). The hypothesis is: $h_\theta(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_3$ Q2: Feature A has range [0.001, 0.01] and Feature B has range [100, 1000]. Why must we scale?

Answer: Without scaling, gradient descent updates for Feature A will be tiny (gradient ~ average error × 0.001) and for Feature B will be huge (gradient ~ average error × 100). This causes uneven convergence — B converges quickly while A barely changes. Standardization or min-max scaling fixes this imbalance. Q3: Compute (X^T X)^{-1} X^T y for points (1,1), (2,3), (3,6) with a quadratic model y = θ₀ + θ₁x + θ₂x²

Design Matrix:

Compute:

Model: $\hat{y} = 0 + 0.5x + 0.5x^2$

Check: x=1: 1.0 (actual 1), x=2: 3.0 (actual 3), x=3: 6.0 (actual 6) — perfect fit! Q4: What is the main advantage of vectorized implementation?

Answer: Vectorized code (using matrix operations) is much faster than explicit loops because:

1. It uses highly optimized BLAS linear algebra libraries
2. It leverages CPU cache and SIMD instructions
3. It avoids Python loop overhead

With NumPy, theta = np.linalg.inv(X.T @ X) @ X.T @ y is both cleaner and faster than any loop-based implementation. Q5: With 10 features and 1000 datapoints, which is better: Normal Equation or gradient descent?

Answer: Both work fine for 10 features (Normal Equation is O(n³) = O(1000), cheap for n=10). But gradient descent scales better to larger datasets and integrates with regularization. For n=10, m=1000, either choice is acceptable — use Normal Equation for simplicity. Q6: A degree-20 polynomial perfectly fits training data (R²=1) but fails on test data. What's happening and what do you do?

Answer: This is overfitting — the model has memorized the training data including noise. Solutions:

1. Reduce polynomial degree (try 2-5)
2. Add regularization (Ridge/Lasso)
3. Get more training data
4. Use cross-validation to select optimal degree

The training R² of 1.0 is misleading — it doesn't indicate generalization. Q7: Transform y = 3 + 2x + 0.5x² into polynomial features for sklearn

python
Run Inline
from sklearn.preprocessing import PolynomialFeatures
import numpy as np

X = np.array([1], [2], [3](/courses/bscs2004/notes/1%5D%2C%20%5B2%5D%2C%20%5B3))
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
# X_poly columns: [x, x²]
# For x=2: [2, 4]

PolynomialFeatures with include_bias=False creates columns [x, x², x³, ...]. The bias term (column of 1's) is added by the linear regression automatically. Q8: What's the difference between standardization and min-max scaling?

Answer:

• Standardization subtracts mean and divides by standard deviation. Result: mean=0, std=1. Not bounded — can be any value. Good with outliers.
• Min-Max scaling subtracts min and divides by range. Result: range [0, 1]. Bounded. Sensitive to outliers (a single outlier compresses most values into a small range).

Use standardization for algorithms that assume normally distributed data (SVM, PCA, linear regression). Use min-max for neural networks. Q9: Using our model ŷ = 45 + 5x₁ + 10x₂, interpret θ₂ = 10

Answer: Holding hours studied ($x_1$) constant, each additional practice test ($x_2$) is associated with a 10-point increase in exam score ($\hat{y}$). This is the ceteris paribus (all else equal) interpretation — it controls for the effect of the other feature. Q10: Your linear model performs poorly but adding x² helps. What does this suggest?

Answer: The relationship between X and Y is non-linear. Adding x² (a quadratic term) captures curvature. The sign of the x² coefficient tells you the direction:

• Positive coefficient: concave up (U-shaped)
• Negative coefficient: concave down (inverted-U) Q11: Why does the rank of X^T X matter for the Normal Equation?

Answer: If $X^T X$ is not full rank (determinant = 0), it's non-invertible. This happens when:

1. Features are linearly dependent (e.g., one column = 2× another)
2. More features than training examples (underdetermined system)

Solution: Remove collinear features, use pseudo-inverse np.linalg.pinv, or add L2 regularization (Ridge regression). Q12: Compare linear (deg=1), quadratic (deg=2), and cubic (deg=3) polynomial models.

Answer:

Rule of thumb: start with degree 1-3 and use validation to select.

🔗 Cross-References

• Next Topic: Logistic Regression — classification with linear decision boundaries
• Related: Regularization — Ridge/Lasso for polynomial models
• Related: Gradient Descent — optimization algorithms
• External: IITM BSCS2004 Week 3, Pattern Recognition and Machine Learning (Bishop) Ch. 3.1 Join Discord PreviousGradient Descent VariantsNextLogistic Regression