Introduction to Linear Regression

355 words

2 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# Introduction to Linear Regression ## 🎯 Learning Objectives - Formulate the simple linear regression model - Estimate parameters using **ordinary least squares (OLS)** - Interpret regression coefficients - Understand assumptions underpinning the model * * * ## 1.1 Intuition: Finding the Line of Best Fit Given a sc...

Introduction to Linear Regression

🎯 Learning Objectives

Formulate the simple linear regression model
Estimate parameters using ordinary least squares (OLS)
Interpret regression coefficients
Understand assumptions underpinning the model

1.1 Intuition: Finding the Line of Best Fit

Given a scatter plot of points

(x_i, y_i)

, we want to find the line that best predicts

y

from

x

. "Best" means minimising the sum of squared vertical distances.

🔑 Key Insight: Regression is about explaining variation in an outcome variable $y$ using one or more predictor variables $x$ .

1.2 The Simple Linear Regression Model

y_i = \beta_0 + \beta_1 x_i + \varepsilon_i

Where:

$y_i$ = response (dependent) variable
$x_i$ = predictor (independent) variable
$\beta_0$ = intercept (expected $y$ when $x = 0$ )
$\beta_1$ = slope (expected change in $y$ per unit change in $x$ )
$\varepsilon_i$ = random error term, $\varepsilon_i \sim N(0, \sigma^2)$

1.3 Least Squares Estimation

Minimise

S(\beta_0, \beta_1) = \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i)^2

Normal equations:

\frac{\partial S}{\partial \beta_0} = -2\sum(y_i - \beta_0 - \beta_1 x_i) = 0

\frac{\partial S}{\partial \beta_1} = -2\sum x_i(y_i - \beta_0 - \beta_1 x_i) = 0

Solution:

\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}

\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}

1.4 Interpretation

$\hat{\beta}_1$ = "a one-unit increase in $x$ is associated with a $\hat{\beta}_1$ change in $y$ "
$\hat{\beta}_0$ = "when $x = 0$ , the predicted $y$ is $\hat{\beta}_0$ " (may not be meaningful if $x=0$ is outside data range)

✅ Practice Questions

Q1: Given data: (1,2), (2,4), (3,5), find

\hat{\beta}_0

and

\hat{\beta}_1

Solution

$\bar{x} = 2$ , $\bar{y} = 11/3 \approx 3.67$ $S_{xx} = (1-2)^2 + (2-2)^2 + (3-2)^2 = 1 + 0 + 1 = 2$ $S_{xy} = (1-2)(2-3.67) + (2-2)(4-3.67) + (3-2)(5-3.67) = (-1)(-1.67) + 0 + (1)(1.33) = 1.67 + 1.33 = 3$ $\hat{\beta}_1 = 3/2 = 1.5$ $\hat{\beta}_0 = 3.67 - 1.5(2) = 0.67$ $\hat{y} = 0.67 + 1.5x$ Join Discord NextRegression Applications