Linear Regression

Linear regression fits the line ŷ = m·x + c to a cloud of points by picking the m and c that make the total squared miss as small as possible. It is the first machine-learning algorithm most people meet, and one of the models with a closed-form least-squares solution.

Two sums over the data give you the slope, one subtraction gives you the intercept, and that pair is provably the best of the infinitely many lines you could have drawn.

What makes it worth an animation rather than a formula sheet is the word "squared". Every point's error is charged as a literal square built on its miss — a point 1 away pays 1, a point 3 away pays 9, a point 9 away pays 81. Fitting a line means shrinking the total area of those squares, and once you can see the areas, two things become obvious that the formula hides: why the best line is unique when x varies and computable with a few aggregate sums, and why a single point flung far off the trend can drag that line away from every other point in the dataset.

What "Best Fit" Actually Means

"Best" is not a property of a line. It is a property of a line and a scoring rule, and the scoring rule here is a specific choice with specific consequences.

Start with one point (x, y) and one candidate line. The line predicts ŷ = m·x + c. The residual is r = y − ŷ, the vertical gap between where the point is and where the line says it should be. Vertical, not perpendicular — the model is only ever asked to predict y from x, so the only error that counts is the one in y. This trips people up constantly, and it is why swapping the roles of x and y generally gives a different line rather than the same one seen sideways (the two coincide only when the points lie exactly on one line).

Now square it. The cost of that point is r², which on a 1:1 plot is the area of a square with the residual as its side. Add up all n of them and you have the sum of squared errors:

SSE(m, c) = Σ (yᵢ − (m·xᵢ + c))²

Divide by n and you have the mean squared error, MSE = SSE / n, which is the same quantity rescaled so that datasets of different sizes are comparable. Least squares is the rule "pick the (m, c) that minimizes this". Nothing about that is inevitable. You could minimize Σ|r| instead, or the largest single |r|, or the perpendicular distances; each defines its own "best line", and on most data they give different answers. Squaring is a choice, and the last third of this lesson is about what that choice costs.

Why the Answer Needs No Search

SSE(m, c) is a quadratic in two variables — a bowl. Provided the x values are not all identical, that bowl has exactly one bottom, and you can find it by setting both partial derivatives to zero. (If every x is the same the bowl degenerates into a trough with no unique lowest point, which is the first entry under Edge Cases.) Doing that and solving gives two facts worth remembering separately.

The best line passes through the mean point. Setting ∂SSE/∂c = 0 says the residuals sum to zero, which rearranges to ȳ = m·x̄ + c, i.e. c = ȳ − m·x̄. So whatever slope you end up with, the line is pinned to (x̄, ȳ). This holds for every dataset ever collected, outliers included, and it means fitting a line is really only a question about the tilt around a point you already know.

The slope is a ratio of two sums.

m = Sxy / Sxx     where  Sxy = Σ (xᵢ − x̄)(yᵢ − ȳ)
                         Sxx = Σ (xᵢ − x̄)²

Sxy measures how x and y move together: positive when above-average x tends to come with above-average y, zero when the sample's centered cross-products cancel. Sxx measures how spread out x is on its own. The slope is the first divided by the second — covariance over variance, co-movement per unit of spread. Predicting x from y instead uses Sxy / Syy, not the reciprocal of this ratio, and generally gives a different line, which is the same point about verticality made arithmetically.

That is the whole algorithm: one pass to get the two means, one pass to get the two sums, one division, one subtraction. No iteration, no learning rate, no convergence to check. Gradient descent applied to this same SSE can converge toward this (m, c) with suitable step sizes and numerical stopping criteria — it is solving a problem that has a closed-form answer, which is why it makes such a good teaching example there and such a bad choice here.

What the Squares Cost You

Here is the part the area picture makes unmissable. Take a point sitting 0.2 away from the fitted line — its square is 0.04, essentially invisible. Move it, at the same x, until it is 9.2 away. Its miss grew by a factor of 46. Its square grew by a factor of 2116, from 0.04 to 84.64.

Now re-run the fit. The optimizer is not trying to pass through the crowd; it is trying to shrink a total, and 84.64 of that total belongs to one point. Tilting the line toward that point buys a huge reduction there in exchange for a modest increase spread over everyone else, and the arithmetic says take the deal. On the default dataset the crowd's collective bill rises from 1.56 to 21.96 so that the single flung point can fall from 84.64 to 33.64. Total: 86.2 down to 55.6. The line that results is the genuine least-squares optimum, and the eight points that never moved now pay fourteen times what they paid before.

This is not a bug in the fit. It is the loss function doing exactly what it was defined to do. Squared error hands disproportionate influence to whatever is furthest away, which makes it exquisitely sensitive to outliers — a data-entry typo, a broken sensor, one unusual customer. The standard responses all amount to changing the rule rather than the fitting: absolute error (Σ|r|, which weights every miss the same at the margin and has to be solved numerically), Huber loss (squared near zero, absolute far away), or removing the point on evidence that it does not belong to the population you are modeling.

The check that tells you which of those you need costs one plot. Fit the line, then plot the residuals against x. Structure in that plot — a bend, a widening fan, one residual ten times the size of the rest — is the model class or the loss function telling you it was the wrong choice for this data, and none of it is visible in the MSE, which is a single number that has already averaged the evidence away.

Linear Regression Step by Step

The default run uses nine points,

(−4,−3) (−3,−3) (−2,−1) (−1,−1) (0,0) (1,1) (2,1) (3,3)   and   P = (4, 3)

chosen so that Σx = Σy = 0, which puts the mean point exactly on the origin. The candidate line starts at m = 0, c = 0 — the "ignore x, always answer the mean" baseline, the honest zero-information prediction. P is the one point you can move.

The question (beat 1). Nine points on a 12 × 12 plane, and the model that has to explain them: ŷ = m·x + c. Two numbers, one straight line, every x gets a prediction.

The candidate (beat 2). The flat line ŷ = 0·x + 0 appears in amber. A slope of zero means x is ignored entirely — every point gets the same prediction — and since that prediction sits at ȳ, this is the zero-information baseline any real model has to beat. It is a guess, not an answer, and before improving it we have to be able to say how wrong it is.

One miss (beat 3). A dashed vertical segment drops from P at (4, 3) to the line at ŷ = 0. That gap is P's residual: r = 3 − 0 = 3. Note the direction — straight down the y-axis, not the shortest path to the line.

One square (beat 4). The square gets built on that residual: 3 units on each side, 9 unit² of area, sitting on the plane in red. This is what "squared error" looks like when you stop treating it as notation. A point twice as far off does not cost twice as much; it costs four times as much.

The whole bill (beat 5). The other eight squares appear together. Each is sized by its own miss (equal misses give equal squares: here 9, 9, 1, 1, 0, 1, 1, 9, 9), and the total area they cover is the entire content of the phrase "this line is wrong".

One number (beat 6). Sum the areas: SSE = 40. Divide by 9: MSE = 4.44. That is the number to beat, and the right-hand header carries it from here on.

The anchor (beat 7). A cross appears at (x̄, ȳ) = (0, 0). Whatever the best line turns out to be, it goes through this cross — that is c = ȳ − m·x̄, rearranged. The flat guess already passes through it, so the anchor was never the problem. The tilt is.

The tilt (beat 8). The left-hand ledger fills in: Sxy = 48, Sxx = 60, so m = 48 / 60 = 0.8. The plane holds still while the derivation happens; there is nothing to look at out there yet.

The swing (beat 9). c = ȳ − m·x̄ = 0, and the line rotates from flat to 0.8 — a pure pivot about the mean cross, because both poses pass through it. Watch the squares as it turns: they are recomputed every frame from the moving line, and the field visibly deflates.

The verdict (beat 10). The amber guess returns as a dashed ghost, and the meter fills a second bar. Your line: SSE 40, MSE 4.44. The fit: SSE 1.6, MSE 0.18 — twenty-five times less area. And not merely better: no (m, c) anywhere scores lower on this data. That optimality is the entire content of the word "least".

The fling (beat 11). P falls from (4, 3) to (4, −6) at fixed x. The line is deliberately held still, so you can watch one thing at a time. P's residual goes −0.2 → −9.2; its square goes 0.04 → 84.64 and swells across a whole quadrant of the plane. Total area is now 86.2, and 98% of it belongs to one point.

The re-solve (beat 12). The same two formulas run again on the moved data. x̄ is unchanged at 0, ȳ drops to −1, Sxy falls 48 → 12, and the line swings to ŷ = 0.2x − 1. Nothing about the method changed; only the data did. Watch where the line ends up: not through the crowd it was threading a moment ago.

The trade (beat 13). The meter seats the second bar and prices what just happened. The eight pinned points now pay 21.96 between them instead of 1.56, and P pays 33.64 instead of 84.64 — a total of 55.6, which really is the minimum for this data. Per point the deal is lopsided rather than universal: six of the nine fit worse, one is unchanged, and two (including P itself) fit better. The optimum and the sensible line have parted company.

Close (beat 14). Two centered sums and a subtraction found that line after computing the means, with no iteration anywhere — and squaring is what let a single point steer it. When that behavior is wrong for your data, the thing to change is the loss, not the solver. Gradient descent, in the next lesson, reaches a minimum by stepping downhill instead of solving for it — shown there on a one-parameter bowl, not on this line.

Three recipes vary one thing each. Eyeball it sets your line to ŷ = x, which threads the cloud so convincingly that most people would stop there — and still costs 2.5× the optimal area. Wrong anchor sets ŷ = 3, a line that misses the mean point entirely, so beat 7 reports the miss and the swing has to slide as well as turn. Data that bends swaps in seven points on y = x² − 3, where the symmetry makes Sxy exactly 0: the best line is flat at ȳ = 1, MSE 12, and a final beat says plainly that no (m, c) does better. That floor belongs to the model class, not to the fit.

Time and Space Complexity

Two passes over n points (one for the means, one for the sums) is O(n) time, and nothing is stored beyond a handful of running totals, so O(1) extra space. A single-pass version accumulating Σx, Σy, Σxy, Σx² is standard and gets the same answer with worse numerical conditioning. With d features instead of one, the closed form becomes (XᵀX)⁻¹Xᵀy at O(n·d² + d³), which is why iterative methods take over once d is large — not because the formula stops working, but because inverting a d × d matrix stops being cheap. Nor should you write that inverse literally even when d is small: numerically robust code can pass X and y directly to a QR- or SVD-based least-squares solver, avoiding an explicit inverse and the condition-number squaring caused by forming XᵀX.

Edge Cases

• Every x identical. Then Sxx = 0 and the slope is 0/0. Geometrically the data is a vertical stripe and no non-vertical line is preferred; algebraically the model is unidentifiable. This lesson's eight pinned points already span x, so no setting of P can reach it, but any real implementation needs the guard.
• Zero covariance. Try the Data that bends preset. Sxy = 0 makes m = 0 and the fit collapses to the horizontal line ŷ = ȳ, which is the "predict the mean, ignore x" baseline. That is the correct answer, and the residuals stay enormous. A slope of zero says "x tells me nothing linearly", not "x tells me nothing" — here x predicts y perfectly, just not with a line.
• A steep guess line. Set m = 3, c = 4 and the candidate crosses the visible square in one short stretch, leaving through the top and bottom edges, so most of its predictions land off the plot. The status panel and the meter still price every one of them exactly.
• Duplicate x values. Move P onto an x another point already occupies. Perfectly legal — the fit does not require distinct x values, only that they are not all equal.
• A point exactly on the line. Its residual is 0, its square has zero area, and it contributes nothing to the total. It still contributes to the means and to Sxy, so it is not the same as deleting it.

Common Mistakes

• Measuring perpendicular distance. The residual is vertical. Minimizing perpendicular distance is a different technique (total least squares, closely related to PCA) that generally gives a different line and answers a different question — one where x and y are symmetric rather than input and output.
• Believing the line has to be searched for. Simple linear regression has a closed form. If you are running gradient descent on one feature, you are iterating toward an answer you could have computed exactly in two passes. The iterative machinery earns its keep on models where no such formula exists.
• Reading SSE and MSE as the same number. SSE is the total area, MSE is the area per point. On this dataset they differ by a factor of nine, which is enough to make a "the error is 40" claim meaningless without saying which one you meant.
• Treating a low MSE as evidence the model is right. MSE is measured against the model you chose. The bending dataset bottoms out at MSE 12 and no straight line does better — the number is telling you about the ceiling of straight lines, not about how well straight lines describe this world.
• Blaming the fit for outlier sensitivity. The line that chases one flung point is the least-squares optimum; it is minimizing exactly what it was told to minimize. If that behavior is wrong for your problem, the thing to change is the loss function, not the solver.
• Assuming shifting the data changes the slope. Add a constant to every y and m does not move at all — Sxy and Sxx are both built from deviations around the means, and the means shift with the data. Only c absorbs the shift. Slope reads shape; intercept reads level.
• Confusing "passes through the mean point" with "passes through the origin". The default dataset is centered, so here they coincide. Move P and (x̄, ȳ) moves with it; the line keeps following the mean and passes through the origin only when the fitted c happens to be 0.