Machine Learning Evaluation Visualizations
Every other collection on this site animates how a model computes something. This one animates how you find out whether it is any good — and the recurring answer is that the obvious number lies to you in a specific, predictable direction.
Three numbers get most people into trouble. Training error falls forever as you add capacity, which makes "my loss went down" evidence of nothing. Test accuracy on a single held-out split is mostly a property of which rows you drew, which makes one run's score unrepeatable. Accuracy rises as your positive class gets rarer, which makes "99.7% accurate" a sentence you can produce by returning False. All three lessons are built to make the failure visible rather than asserted: you watch the good-looking number improve while the model gets worse, or stay put while the data does not.
What These Lessons Cover
| Lesson | What you watch |
|---|---|
| Overfitting & Validation | nine polynomials fit to the same ten noisy points — training error falling monotonically to an exact 0.0000 while held-out error turns around at degree 3 and ends 899× worse |
| Cross-Validation | twenty photos rotating through five folds — the first fold scoring a flawless 1.00 and the second 0.25 on the same data, same model, same code, while the averaged score reads 0.800 on both draws |
| Confusion Matrix | one threshold sliding across forty scored cases, sorting them into four counts — and three degenerate rules, two of which never read a score at all, each posting a flattering number on a different metric |
A Recommended Learning Order
Start with Overfitting & Validation, because it establishes the split that everything else depends on: data the model studied versus data it has never seen. The argument is algebraic rather than empirical, which is what makes it inescapable. Every degree-d polynomial is also a degree-(d+1) polynomial with a zero leading coefficient, so the best degree-(d+1) fit can never be worse on the training set. A number that can only move one direction carries no information about which direction is right. The lesson then runs the consequence to its end: ten coefficients, ten points, a curve threaded exactly through every one of them at a training error of 0.0000, swinging to ±353 in the gaps where the held-out points happen to sit. Its three recipes each kill a different misreading — shrink the sample and the wall moves to degree 7, double the data and the same degree 9 becomes nearly free, remove the noise entirely and the U-curve vanishes, because overfitting is the memorization of noise and there is none left to memorize.
Then Cross-Validation, which attacks the split itself. The first lesson treats "held-out error" as a number you can read; this one shows that on a small dataset it is mostly a draw. Twenty photos, a 1-NN model with nothing to hide, and an ordinary 80/20 holdout: over 400 random splits the reported accuracy has a standard deviation of 0.2005 and returns a perfect 1.00 on 35% of them, for a model that is right 80% of the time. k-fold does not average that noise away by luck — it removes the draw, because every row is held out exactly once, so the score is just "how many of the twenty did a model that had not seen them get right". Turn the split dial at k = 20 and it stops responding entirely. The failure case is the one nobody's evaluation catches: twenty photos of only ten animals, where random folds report a flawless 1.000 for a model that is honestly a coin flip, and raising k makes it worse rather than better.
Finish with Confusion Matrix, which asks the question the first two lessons left open: you have held-out data and a score for each case, so what number do you actually report? The lesson starts one step earlier than most explanations, at the fact that a classifier hands you a ranking rather than a decision, and that a threshold is what turns one into the other. Cases sit on a score axis with the truth as height and the threshold as a vertical line, so the four cells of the matrix arrive as the four quadrants you already had rather than a convention to memorize. Then it walks the threshold from "alarm on nobody" to "alarm on everybody" across one fixed set of forty cases and lets the three metrics disagree in public: accuracy peaks at 0.900 and ends below the do-nothing model's 0.850, precision sags but rises whenever the next case over the line is real, and recall reaches a perfect 1.000 on a model with no skill at all.
Read together, the three lessons say the same thing three times about different numbers. Training error cannot choose a model because it only moves one way. A single split's test accuracy cannot price a model because it mostly measures the draw. Accuracy cannot judge a classifier because it mostly measures your class balance. In all three cases the fix is the same shape: stop reading one summary, and look at the thing the summary threw away.
Where These Lessons Sit
Evaluation is the step after a model exists, so all three lessons take the model as a given. Where the scores come from is the classical machine learning collection's subject — logistic regression is the most direct feed into the confusion matrix, since its sigmoid output is exactly the per-case score the threshold cuts. Linear regression is the fit that the overfitting lesson escalates to nine degrees, and k-nearest neighbours is the model cross-validation puts on the stand — deliberately, because a model that fits instantly leaves nothing on screen except the protocol. On the neural-network side, the training loop is the same train/validate split running live, epoch by epoch, and it is where you watch the two curves separate in real time instead of degree by degree.
All ML Evaluation Visualizations
Pick a lesson below to open its interactive player — change the input, step through it, and watch the code highlight move in lockstep with the animation.