Linear Regression with One Variable

🎯 Learning Objectives

• Explain linear regression in plain English and when to use it
• Derive the cost function (MSE) for linear regression
• Implement gradient descent for a univariate linear regression model
• Solve linear regression analytically using the Normal Equation
• Evaluate model performance using R² and MSE

📋 Prerequisites

• Introduction to ML — basic ML concepts (features, labels, training)
• Differentiation basics — we'll need derivatives for gradient descent
• Matrix multiplication — for the Normal Equation

📖 Core Content

2.1 Intuition: What Problem Does Linear Regression Solve?

2.2 Formal Definition

• $h_\theta(x)$ or $\hat{y}$ is the predicted output
• $\theta_0$ (intercept/bias): the predicted value when $x = 0$
• $\theta_1$ (slope/weight): the change in $\hat{y}$ for a one-unit change in $x$
• $x$ is the input feature The Cost Function (Mean Squared Error):

2.3 Worked Example 1: Hand Calculation with Tiny Dataset

• $h(1) = 52$, error = 2
• $h(2) = 57$, error = 2
• $h(3) = 62$, error = -3
• $h(4) = 67$, error = -3
• $h(5) = 72$, error = -3

2.4 Gradient Descent: The Optimization Algorithm

1. Start with some $\theta_0, \theta_1$ (often 0 or random)
2. Simultaneously update:

1. Repeat until convergence Where $\alpha$ is the learning rate — how big a step we take each iteration. The derivatives (for MSE):

2.5 Worked Example 2: Gradient Descent Step-by-Step

• $(h - y) = [-50, -55, -65, -70, -75]$ Step 2: Compute gradient:

• $h(1) = 0.63 + 2.02(1) = 2.65$, error = 50 - 2.65 = 47.35
• $h(2) = 0.63 + 2.02(2) = 4.67$, error = 55 - 4.67 = 50.33
• $h(3) = 0.63 + 2.02(3) = 6.69$, error = 65 - 6.69 = 58.31
• $h(4) = 0.63 + 2.02(4) = 8.71$, error = 70 - 8.71 = 61.29
• $h(5) = 0.63 + 2.02(5) = 10.73$, error = 75 - 10.73 = 64.27 Step 5: New cost:

2.6 The Normal Equation (Closed-Form Solution)

• $x = 1$: $43.5 + 6.5 = 50.0$ (perfect match!)
• $x = 3$: $43.5 + 19.5 = 63.0$ (true = 65, off by 2)
• $x = 5$: $43.5 + 32.5 = 76.0$ (true = 75, off by 1) Pros of Normal Equation: No learning rate, no iterations, guaranteed global minimum. Cons: $O(n^3)$ matrix inversion — doesn't scale beyond ~10,000 features.

2.7 Model Evaluation Metrics

2.8 Python Implementation

python
Run Inline
# runnable
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
# Our study hours data
X = np.array([1, 2, 3, 4, 5]).reshape(-1, 1)  # Feature matrix (m x 1)
y = np.array([50, 55, 65, 70, 75])             # Target vector
# Create and train the model
model = LinearRegression()
model.fit(X, y)
# Get parameters
print(f"Intercept (θ₀): {model.intercept_:.2f}")
print(f"Slope (θ₁):     {model.coef_[0]:.2f}")
print(f"Model: ŷ = {model.intercept_:.2f} + {model.coef_[0]:.2f}x")
# Predict
y_pred = model.predict(X)
# Evaluate
mse = mean_squared_error(y, y_pred)
r2 = r2_score(y, y_pred)
print(f"MSE: {mse:.2f}")
print(f"RMSE: {np.sqrt(mse):.2f}")
print(f"R²:  {r2:.4f}")
# Predict a new value
x_new = np.array(3.5)
y_new = model.predict(x_new)
print(f"\nPrediction for x=3.5 hours: {y_new[0]:.1f}")
# Plot
plt.scatter(X, y, color='blue', label='Data points')
plt.plot(X, y_pred, color='red', label='Regression line')
plt.xlabel('Hours Studied')
plt.ylabel('Exam Score')
plt.title('Linear Regression: Study Hours vs Exam Score')
plt.legend()
plt.grid(True)
plt.show()

2.9 When to Use / Not Use Linear Regression

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Not Adding the Intercept Column

Pitfall 2: Using the Wrong Learning Rate

Pitfall 3: Not Normalizing Features with Gradient Descent

📝 Practice Questions

Q1: For the points (1,2), (2,3), (3,5), find the linear regression line by hand.

Step 1: Calculate means: $\bar{x} = 2$, $\bar{y} = (2+3+5)/3 = 3.33$

Step 2: Calculate $\theta_1 = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2}$ $= \frac{(1-2)(2-3.33) + (2-2)(3-3.33) + (3-2)(5-3.33)}{(1-2)^2 + (2-2)^2 + (3-2)^2}$ $= \frac{(-1)(-1.33) + 0 + (1)(1.67)}{1 + 0 + 1} = \frac{1.33 + 1.67}{2} = \frac{3}{2} = 1.5$

Step 3: Calculate $\theta_0 = \bar{y} - \theta_1\bar{x} = 3.33 - 1.5(2) = 0.33$

Answer: $\hat{y} = 0.33 + 1.5x$ Q2: If R² = 0.75, what does this mean?

Answer: 75% of the variance in the target variable is explained by the model. The remaining 25% is unexplained (due to noise or missing features). The model is decent but has room for improvement. Q3: With α = 0.1 and gradient = [-50, -200], what are the new θ values?

Answer: $\theta_0 = \theta_0 - 0.1(-50) = \theta_0 + 5$ $\theta_1 = \theta_1 - 0.1(-200) = \theta_1 + 20$

We move in the opposite direction of the gradient (since gradient points uphill, we subtract to go downhill). Since both gradients are negative, we increase θ. Q4: What happens if α is too large in gradient descent?

Answer: The algorithm diverges — the cost J increases instead of decreasing. Each step overshoots the minimum, potentially moving to a worse position. You'll see the cost oscillate or grow unbounded. Solution: reduce α. Q5: Why does the Normal Equation use $(X^T X)^{-1}$ and what if it's non-invertible?

Answer: The inversion finds the θ that minimizes MSE by setting derivatives to zero. If $X^T X$ is non-invertible (singular), it means:

1. Some features are linearly dependent (remove them)
2. More features than examples (use regularization or reduce features)

Solution: Use np.linalg.pinv (pseudo-inverse) or apply regularization. Q6: For the dataset (2,4), (4,6), (6,10), predict y when x=5 using the Normal Equation.

Step 1:

,

Step 2:

,

Step 3:

Step 4:

Step 5: $\hat{y} = 2.67 + 1(5) = 7.67$ Q7: What's the difference between MSE and RMSE?

Answer: RMSE is the square root of MSE. Both measure average prediction error, but RMSE is in the same units as the target variable. MSE is in squared units. If predicting house prices in dollars, MSE is in $² (hard to interpret), while RMSE is in$ (intuitive). Q8: Your model has R² = -0.5. What went wrong?

Answer: A negative R² means the model is worse than predicting the mean. This can happen if:

1. No regularization was used with highly overfitted model
2. The relationship is non-linear but you used linear regression
3. Wrong model entirely — the fit is terrible

Check your model assumptions and consider transformations or a different algorithm. Q9: Which is faster for 100 features — gradient descent or Normal Equation?

Answer: Normal Equation — it's O(n³) = O(100³) = 1,000,000 operations, which is fine for 100 features. Gradient descent would need many iterations over the full dataset. The Normal Equation's weakness is at n > 10,000 where O(n³) becomes prohibitive. Q10: Why is the 1/2 factor in the MSE cost function?

Answer: The 1/2 cancels the 2 that appears when we differentiate the squared error. Without it: $\frac{\partial}{\partial \theta} \frac{1}{m} \sum (h_\theta - y)^2 = \frac{2}{m} \sum (h_\theta - y) \frac{\partial h_\theta}{\partial \theta}$

With the 1/2, the 2 cancels out, giving cleaner gradient expressions. It doesn't change the optimal θ — scaling the cost by any positive constant doesn't shift the minimum. Q11: Implement linear regression using gradient descent for 50 iterations on the dataset (x = [1,2,3,4,5], y = [2,4,5,4,6]).

python
Run Inline
import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 6])
m = len(x)

theta0, theta1 = 0, 0
alpha = 0.01

for i in range(50):
    h = theta0 + theta1 * x
    error = h - y
    grad0 = (1/m) * np.sum(error)
    grad1 = (1/m) * np.sum(error * x)
    theta0 -= alpha * grad0
    theta1 -= alpha * grad1

print(f"After 50 iterations: θ₀ = {theta0:.4f}, θ₁ = {theta1:.4f}")

Q12: When would you choose gradient descent over the Normal Equation?

Answer: Choose gradient descent when:

1. Number of features is large (>10,000) — Normal Equation is O(n³)
2. Data is very large — use stochastic/mini-batch gradient descent
3. You want to add regularization easily
4. The model is not linear (neural networks, etc. — Normal Equation only works for linear regression)

Choose Normal Equation when n < 10,000 and you want the exact solution in one step.

🔗 Cross-References

• Next Topic: Multiple & Polynomial Regression — extending to multiple features
• Related: Gradient Descent Deep Dive — optimization variants
• Related: Regularization — preventing overfitting in linear models
• External: IITM BSCS2004 Week 2, Pattern Recognition and Machine Learning (Bishop) Ch. 3 Join Discord PreviousIntro to MLNextGradient Descent Variants