Linear regression
Fitting a straight line to data, and what that line can and cannot tell you.
Linear regression models a response y as a linear function of one or more predictors x, plus noise. It is the oldest and most widely used tool in statistics: Legendre and Gauss published the method of least squares around 1805–1809 to fit the orbits of comets and asteroids.
The model
With a single predictor, simple linear regression assumes
where β0 is the intercept, β1 is the slope (the expected change in y per unit increase of x), and εi is an error term with mean zero that captures everything the line does not.
Least squares
We choose the line that minimises the sum of squared vertical distances (residuals) between the observations and the line:
Setting both partial derivatives to zero gives a closed-form solution:
The slope is the covariance of x and y divided by the variance of x, and the fitted line always passes through the point of means (x̄, ȳ).
A worked example
An ice-cream stand records the daily maximum temperature and the number of cones sold over seven days:
| Max temp (°C) | Cones sold | Fitted | Residual |
|---|---|---|---|
| 14 | 212 | 211.6 | +0.4 |
| 16 | 241 | 241.5 | −0.5 |
| 19 | 287 | 286.3 | +0.7 |
| 22 | 334 | 331.2 | +2.8 |
| 25 | 371 | 376.0 | −5.0 |
| 28 | 420 | 420.8 | −0.8 |
| 31 | 468 | 465.6 | +2.4 |
Here x̄ ≈ 22.14 °C and ȳ ≈ 333.3 cones. The cross-product sum is Σ(x − x̄)(y − ȳ) ≈ 3508.7 and Σ(x − x̄)2 ≈ 234.9, so
Each extra degree Celsius is associated with about 15 more cones. On a 20 °C day the model predicts 2.48 + 14.94 × 20 ≈ 301 cones.
Goodness of fit
The coefficient of determination compares the residual sum of squares to the total variation in y:
For the ice-cream data R2 ≈ 0.999: temperature explains almost all of the day-to-day variation in sales. Real data is rarely this tidy. A high R2 also does not show the model is correct, and a low one does not make it useless.
Multiple regression
With p predictors, stack the observations into a matrix X (one row per observation, with a leading column of ones for the intercept) and write
These are the normal equations. In practice software solves them with a QR or singular value decomposition instead of inverting X⊤X, which is numerically fragile when predictors are nearly collinear. Each coefficient is then the expected change in y per unit change in that predictor, holding the others fixed.
“Linear” refers to the parameters, not the predictors: y = β0 + β1x + β2x2 is still a linear regression.
Assumptions
Least squares always produces a line. Whether its standard errors, confidence intervals and p-values mean anything depends on these assumptions:
- Linearity: the expected value of y really is linear in the parameters.
- Independence: errors are uncorrelated with each other (often violated by time series).
- Homoscedasticity: errors have constant variance across all values of x.
- Exogeneity: errors are uncorrelated with the predictors, so no important omitted variable drives both.
- Normality of errors, needed for exact small-sample inference but not for the estimates themselves.
Under the first four, the Gauss–Markov theorem says ordinary least squares is the best linear unbiased estimator: no other linear unbiased estimator has lower variance.
Common pitfalls
- Extrapolation. The ice-cream line predicts about −147 cones at −10 °C. The model is only trustworthy near the data it was fitted on.
- Correlation is not causation. A slope measures association. Without a design that rules out confounders, it does not say what would happen if you changed x.
- Outliers and leverage. Squared errors let a single far-away point drag the line. Always plot the data and the residuals.
- Overfitting. Adding predictors never lowers R2. Use adjusted R2, cross-validation, or regularisation (ridge, lasso) instead.