Confusion Matrix

A confusion matrix is a 2×2 table that splits a binary classifier's predictions into four counts (true positives, false positives, false negatives, true negatives) by crossing what the model said against what was actually true. Metrics such as accuracy, precision, and recall are ratios computed from the table at a selected threshold; ranking and calibration metrics require additional information.

Here is the part that should bother you. On the forty cases below, a "model" that raises no alarm on anybody scores 85% accuracy. One that alarms on everybody scores perfect recall. One that alarms on the single highest-scoring case and stops there scores perfect precision. The first two never look at a score; the third looks at exactly one and answers a question nobody asked. Each is useless, and each posts a number that looks like success — two of them perfect. What catches all three is the table: a single ratio can hide which of the four things happened, and the four counts cannot.

How a Confusion Matrix Works

Start one step earlier than most explanations do, because the missing step is where the confusion lives. A classifier does not hand you a decision. It hands you a score — one number per case, higher meaning "more likely positive". Logistic regression gives you a probability, a tree gives you a leaf frequency, a neural net gives you a softmax output. Nothing has been decided yet.

You decide, by picking a threshold τ and declaring: alarm when the score is at least τ. That single choice is what turns a ranking into predictions, and it is the reason the same trained model can produce wildly different confusion matrices.

Now every case has two independent facts attached: what it actually is, and what the model just said about it. Two binary facts make four combinations, and those four combinations are the confusion matrix.

• TP — actually positive, and the model alarmed. Caught.
• FP — actually negative, but the model alarmed anyway. A false alarm.
• FN — actually positive, and the model stayed quiet. A miss.
• TN — actually negative, and the model stayed quiet. Correctly left alone.

The animation makes the derivation literal. Cases sit on a horizontal score axis: above the line means actually positive, below means actually negative, and left-to-right is the score. Drop the threshold in as a vertical line and the strip is cut into four quadrants — and those quadrants slide, without crossing, straight into the four cells of the table. The matrix is not a convention you memorize. It is the two channels you already had, counted.

Three ratios come out of it, and each one reads a different piece:

• Accuracy = (TP + TN) / n — the correct cells (on a diagonal, depending on axis order).
• Precision = TP / (TP + FP) — the alarms column. Of everything I flagged, how much was real?
• Recall = TP / (TP + FN) — the actually-positive row. Of everything real, how much did I catch?

Precision never looks at TN. Recall never looks at FP or TN. Accuracy looks at all four but cannot tell a miss from a false alarm. Every one of them throws away part of the table, and which part it throws away is exactly where it can be fooled.

Confusion Matrix Step by Step

The default run holds 40 hold-out cases: 34 actually negative, 6 actually positive, so the positive rate, the prevalence, is 15%. The six positives scored 0.93, 0.81, 0.64, 0.55, 0.34, and 0.19. Nine of the negatives scored 0.20 or higher, topping out at 0.88; the other twenty-five are all below 0.18. The threshold starts at τ = 0.50.

Sort the four cells. Four positives clear 0.50 (0.93, 0.81, 0.64, 0.55), so TP = 4. Two do not (0.34, 0.19), so FN = 2. Three negatives clear it (0.88, 0.72, 0.52), so FP = 3, and the remaining thirty-one sit quietly on the left: TN = 31. Notice that a single threshold produced both kinds of error at once. Moving the threshold can change either count; some threshold intervals add only positives or only negatives.

Read the three ratios. Accuracy is (4 + 31) / 40 = 0.875. Precision is 4 / 7 = 0.571: of the seven alarms raised, four were real. Recall is 4 / 6 = 0.667 — of the six real cases, four were caught. One model, three numbers, three different stories, and a reader who hears only one of them has no idea what the other two are.

Now raise τ past every score. Nobody is alarmed: TP = FP = 0, FN = 6, TN = 34. Recall is 0.000, precision is undefined (see Edge Cases), and accuracy is 34 / 40 = 0.850. That is 1 − prevalence, exactly, and it will always be exactly that: get every negative right and you have already banked the negative share of the data. This model has no model in it — no threshold logic, no scores, nothing — and it lands 2.5 points behind the real classifier.

Then walk τ down, one positive at a time. This is the whole trade-off in one continuous experiment on one fixed set of cases.

Read the FP column, not the metrics. The first catch is free: no negative scores above 0.93, so precision is a perfect 1.000 on a model that has found one case in six. The fourth catch is also free, which is why precision goes up from 0.600 to 0.667 at τ = 0.55. The fifth costs three false alarms. The sixth costs four more. The price of a catch escalates because the positives you have not found yet are the ones the model scored low, and low is where the negatives live.

Drop τ to 0 and alarm on everybody. Recall is a perfect 1.000. Precision is 6/40 = 0.150, which is the prevalence exactly, and always will be, because when you flag everything, the share of your flags that are real is just the share of the data that is real. Accuracy is 0.150.

The verdict. Accuracy peaks at 0.900 around τ = 0.55, is back to the do-nothing model's 0.850 by τ = 0.44, and first falls below it at τ = 0.38. From there it touches 0.850 once more, at τ = 0.34, and never beats it again. The threshold that misses nobody reports an accuracy of 0.775, worse than a model that does nothing at all, while representing the highest threshold that catches every real case; lower thresholds also catch them. If accuracy were your objective, you would ship the model that misses two of six.

Why Every Single Metric Can Be Gamed

Three degenerate rules, three metrics, three trophies:

None of these is a trick or an adversarial edge case; each one is a real strategy people deploy without noticing. "Our fraud model is 99.7% accurate" is a sentence you can produce by returning False, if fraud is 0.3% of transactions. Push the prevalence knob down to 2 positives in 36 cases and the do-nothing accuracy rises to 0.944. The model got better without changing a line of code, because the data changed.

That is the point worth carrying out of here. Accuracy's failure is not a property of your classifier; it is a property of your class balance. The rarer the thing you are looking for, the more accuracy measures your dataset instead of your model.

Two structural facts explain why no single ratio escapes this:

Recall is monotone in τ; precision is not. Lower the threshold and cases can only move from "predicted negative" to "predicted positive", so TP can only rise and FN can only fall — recall never goes down. Precision has a numerator and a denominator both growing, and which grows faster depends on whether the next case over the line is real. That is why the table above shows precision sagging overall but rising at τ = 0.55, and why a precision-versus-recall curve is a sawtooth rather than a clean slope.

Accuracy weights the two errors by class size, which is almost never what you want. In this data one FN and one FP each cost 1/40 of the accuracy score. In cancer screening, a miss is a person going home untreated and a false alarm is a follow-up test. Those are not the same price, and no ratio in the table knows the difference — which is why the four counts, not any summary of them, are what you report.

How to Actually Pick a Threshold

"Which error you can afford is your call" is true and, on its own, useless. The move that turns it into a procedure is to stop optimizing and start constraining: name the error you cannot tolerate, turn it into a floor, and then maximize whatever you like among the thresholds that clear it.

Screening is the standard case. A miss sends someone home sick, so the floor goes on recall: recall ≥ 0.80, and among the thresholds that clear it, take the highest precision. Run that on the sweep table above and it is a two-line calculation. Two stops clear the floor — τ = 0.34 at recall 0.833 and τ = 0.19 at recall 1.000 — and of those two, τ = 0.34 has the better precision, 0.500 against 0.400. So you ship τ = 0.34: five of six caught, five false alarms, one miss. Flip the constraint for a system that spams a human reviewer (precision ≥ 0.65, then maximize recall) and the same table hands you τ = 0.55 instead.

Notice what the rule bought. It never asked which metric is "the good one", it never compared a precision to a recall, and it is auditable — someone can disagree with your floor, which is a conversation about the problem rather than about statistics. What it cannot do is invent the floor. That number comes from what a miss and a false alarm actually cost you, and no amount of staring at the matrix will produce it.

Edge Cases

• No alarms at all (TP + FP = 0). Precision is 0/0, which is undefined, not zero. The lesson's precision readout prints an em dash instead of a number. Reporting 0.00 here is a lie in a specific direction: it says the model raised alarms and they were all wrong, when it actually raised none. sklearn returns 0.0 with a warning; treat that warning as load-bearing.
• No actual positives (TP + FN = 0). Recall is undefined for the same reason. This lesson's config keeps at least two positives so it cannot happen, but a small held-out fold in cross-validation absolutely can.
• A score exactly equal to τ. The convention here is p >= tau, so a tie is alarmed on. p > tau moves that case from TP to FN, or FP to TN. All 58 available scores are distinct, so cases never tie with one another; a threshold set to a score still produces equality — but real scores clump on round values, and the two conventions can disagree on a meaningful number of cases. Say which one you use.
• F1, the usual escape hatch. F1 is the harmonic mean of precision and recall, 2PR/(P+R). It is genuinely harder to game than either alone: alarm on everybody here and F1 is 0.261, alarm on the top case only and it is 0.286, against 0.615 for the real classifier at τ = 0.50. But it still never looks at TN, and it still weights a miss and a false alarm equally by construction. It is a better default, not an answer to "which error can I afford".
• Specificity, the missing fourth ratio. TN / (TN + FP) is recall's mirror image on the negative row, and it is the one people forget. Its denominator is the whole negative class, which on imbalanced data makes it flattering: at τ = 0.19 above, specificity is 25/34 = 0.735 while precision on the same nine false alarms is 0.400. Precision never sees TN, so the two ratios can tell you opposite things about one model.
• sklearn prints it upside down from this lesson. confusion_matrix(y_true, y_pred) orders both axes by label, so with 0/1 labels you get [[TN, FP], [FN, TP]] — negatives first, and the correct cells land on the main diagonal. This lesson puts the positive row on top and the positive column on the right so each cell sits under its own quadrant on the score axis, which puts TP and TN on the anti-diagonal. Same four numbers, different corners. Check the axis order before you read anyone's matrix, including your own from six months ago.

Common Mistakes

• Reporting accuracy without prevalence. An accuracy number with no class balance beside it is uninterpretable. At 30% prevalence, a never-alarm baseline scores 70%; at 95% prevalence it scores 5%. An 85% score exceeds those two baselines, but equals the never-alarm baseline at this lesson’s 15% prevalence. Always publish n and the positive rate next to the score.
• Treating recall as the safe metric. "We optimized for recall because misses are expensive" sounds responsible and can end at τ = 0, where recall is perfect but 34 of 40 alarms are false. Recall alone cannot detect that it has been maxed out by a model that flags everything; it needs precision or specificity standing next to it.
• Believing the threshold is part of the model. It is not. It is a business decision layered on top of a ranking, and it is usually the cheapest thing in the whole pipeline to change. If precision is too low, you may not need a better model — you may need a higher τ and the willingness to accept the misses that come with it.
• Comparing two models at whatever threshold each happened to ship with. Model A at τ = 0.5 versus model B at τ = 0.3 compares two complete operating points and mixes ranking quality with threshold choice. Compare them at matched recall, at matched precision, or across the whole threshold range.
• Reading a matrix without checking n. TP = 4 means something different out of 6 positives than out of 6,000. The four counts are absolute; every ratio hides the denominator that produced it.

A Note on Simplification

The scores and labels here are designed, not exported from a trained classifier, so the threshold story lands on whole numbers a reader can verify by hand. Where the scores come from is a different lesson's subject — this one takes (y_true, y_score) as its input, exactly as sklearn.metrics does, and the model is a given throughout.

Real evaluation adds machinery this lesson deliberately leaves out. There is one fixed hold-out set here, and no cross-validation, so nothing accounts for the fact that a 40-case estimate of precision has a confidence interval you could drive a truck through. Everything is binary; multi-class problems get a k×k matrix where "precision" splits into per-class and averaged variants that disagree with each other. Nothing is calibrated, so a score of 0.9 is not a claim that the case is 90% likely to be positive. And no cost matrix appears anywhere — in practice, choosing τ means pricing a miss against a false alarm in whatever units your problem actually cares about, which is the one input no metric can supply for you.