Linear regression

Fitting a straight line to data, and what that line can and cannot tell you.

Linear regression models a response y as a linear function of one or more predictors x, plus noise. It is the oldest and most widely used tool in statistics: Legendre and Gauss published the method of least squares around 1805–1809 to fit the orbits of comets and asteroids.

The model

With a single predictor, simple linear regression assumes

yi = β0 + β1xi + εi,  i = 1, …, n

where β0 is the intercept, β1 is the slope (the expected change in y per unit increase of x), and εi is an error term with mean zero that captures everything the line does not.

Least squares

We choose the line that minimises the sum of squared vertical distances (residuals) between the observations and the line:

minimise  S(β0, β1) = Σi (yi − β0 − β1xi)2

Setting both partial derivatives to zero gives a closed-form solution:

β̂1 = Σ(xi − x̄)(yi − ȳ) ⁄ Σ(xi − x̄)2β̂0 = ȳ − β̂1x̄

The slope is the covariance of x and y divided by the variance of x, and the fitted line always passes through the point of means (x̄, ȳ).

A worked example

An ice-cream stand records the daily maximum temperature and the number of cones sold over seven days:

Max temp (°C)Cones soldFittedResidual
14212211.6+0.4
16241241.5−0.5
19287286.3+0.7
22334331.2+2.8
25371376.0−5.0
28420420.8−0.8
31468465.6+2.4

Here x̄ ≈ 22.14 °C and ȳ ≈ 333.3 cones. The cross-product sum is Σ(x − x̄)(y − ȳ) ≈ 3508.7 and Σ(x − x̄)2 ≈ 234.9, so

β̂1 ≈ 3508.7 ⁄ 234.9 ≈ 14.94β̂0 ≈ 333.3 − 14.94 × 22.14 ≈ 2.48

Each extra degree Celsius is associated with about 15 more cones. On a 20 °C day the model predicts 2.48 + 14.94 × 20 ≈ 301 cones.

101520 253035 °C 200300 400500
Cones sold against daily maximum temperature, with the least-squares line ŷ = 2.48 + 14.94x.

Goodness of fit

The coefficient of determination compares the residual sum of squares to the total variation in y:

R2 = 1 − Σ(yi − ŷi)2 ⁄ Σ(yi − ȳ)2

For the ice-cream data R2 ≈ 0.999: temperature explains almost all of the day-to-day variation in sales. Real data is rarely this tidy. A high R2 also does not show the model is correct, and a low one does not make it useless.

Multiple regression

With p predictors, stack the observations into a matrix X (one row per observation, with a leading column of ones for the intercept) and write

y = Xβ + ε⇒β̂ = (X⊤X)−1X⊤y

These are the normal equations. In practice software solves them with a QR or singular value decomposition instead of inverting X⊤X, which is numerically fragile when predictors are nearly collinear. Each coefficient is then the expected change in y per unit change in that predictor, holding the others fixed.

“Linear” refers to the parameters, not the predictors: y = β0 + β1x + β2x2 is still a linear regression.

Assumptions

Least squares always produces a line. Whether its standard errors, confidence intervals and p-values mean anything depends on these assumptions:

Under the first four, the Gauss–Markov theorem says ordinary least squares is the best linear unbiased estimator: no other linear unbiased estimator has lower variance.

Common pitfalls