Linear Classifier

A linear classifier gives every class its own row of weights and its own bias, scores an input once per class with s_k = w_k·x + b_k, and returns whichever score came out highest. The rows are stacked into a matrix W and the biases into a vector b, so the whole model is s = Wx + b followed by argmax.

That is three dot products for three classes — the same loop you already know from the dot product, run once per class.

What makes that worth an entire lesson is the shape it forces on the answer. Because every score is linear in x, the place where two classes change hands is exactly where their scores tie — a straight line in the nondegenerate case. Stack those lines up and the plane comes out carved into straight-edged, convex wedges: one per class, no curves, no holes, no islands. That constraint is invisible in the code and unmissable on the plane, and it is the reason a linear model can be hopeless on data that a human separates at a glance.

How the Scores Become a Decision

Read the score function one row at a time. Row w_A is a direction: points lying along it score high for class A, points lying opposite score low, and the bias b_A shifts every one of those scores by a constant — a head start that has nothing to do with direction. Do that for all K rows and you have K numbers describing the same input from K points of view. argmax then does the only remaining work, which is to pick the largest.

Notice what argmax never asks. It never measures distance to a cluster, never checks whether x looks like anything it has seen, never reports how sure it is. Feed it a point in the far corner of the plane, nowhere near any training data, and it will still return a class — usually with a huge margin, because a score grows without bound as you travel along its row's direction. A linear classifier has no "none of the above". A rejection rule would have to be added beyond this argmax.

Why Every Border Is Straight

Class i beats class j exactly when s_i > s_j. Write both out and subtract:

w_i·x + b_i  >  w_j·x + b_j
(w_i − w_j)·x + (b_i − b_j)  >  0

The equality case, (w_i − w_j)·x + (b_i − b_j) = 0, is a straight line with normal vector w_i − w_j when that vector is nonzero. So a decision boundary is not a special object the model stores. It is a tie, and the tie between two linear functions is always linear.

With two classes that line is the whole story. With three it is only a candidate. Where i and j tie is a real decision boundary just along the stretch where neither of them is losing to the third class; run it far enough and it slides into territory k owns outright, and there it separates nothing — both i and j are beaten on both sides of it. That is why the map in the animation draws each tie line only where its pair really is the top two, and why K classes produce up to K(K−1)/2 border segments, meeting at points where three scores come out equal at once.

Two consequences fall out of the same equation, and the lesson's two experiments are built to make each one visible on its own.

Turning a row swings its region. The normal of every border touching class i is w_i − w_j, so changing the direction of w_i turns all of class i's borders at once. Its wedge pivots to face wherever the row now points, and whatever used to be inside it — or was just outside — changes hands.

Changing a bias slides the borders without turning them. Add a constant to b_i and the differences w_i − w_j do not move at all; only the constant term does. Every border of class i therefore slides parallel to itself, and class i's region inflates or shrinks with every edge still facing exactly the way it did. With three classes that means the wedge keeps its angles precisely; with more classes an edge can appear or disappear as a constraint starts or stops binding, but no edge ever turns. Bias controls how much a class claims, never which way it faces. It also follows that only the gaps between biases matter: adding the same number to all K biases moves every score up equally and changes nothing.

Why a Linear Classifier Cannot Make Islands

Class k's region is the set {x : s_k ≥ s_j for every j}. Each of those conditions is a half-plane, so the region is an intersection of half-planes — and an intersection of half-planes is convex. That one sentence is the entire ceiling of the model, and it is worth slowing down for, because it is the part a picture makes obvious and an equation hides.

Convex means: own two points, own everything between them. Pick any two points class C wins, draw the segment, and every point on it is C's as well. There is no way to carve a dent, a hole, or a second lobe out of an intersection of half-planes.

Now give class C two clumps of data in opposite corners of the plane, with class A's points strung along the corridor between them. To get both clumps right, C's region has to contain both corners. Being convex, it then has to contain the corridor too — including every A point standing in it. No choice of W and b escapes that. It is not a training problem, and no learning rate fixes it: the model is being asked for a shape it cannot express.

This is the multi-class form of the wall the perceptron hits on XOR, and it is the argument for stacking layers. A hidden layer's job is to bend the space until the classes are convex in the new coordinates, so that a linear layer on top — exactly the one on this page — can finish the job.

Where W Comes From

In this lesson W and b are inputs. They were designed by hand so the geometry reads cleanly, and you can edit class C's row directly to see what each ingredient controls. Real models are not hand-designed: they learn W by measuring how wrong the current scores are and nudging every weight in the direction that reduces the error, over and over. That is what gradient descent, backpropagation and the training loop are for. A common approach turns scores into probabilities with softmax and minimizes cross-entropy; other losses, such as hinge loss, do not require probabilities — the direct sequel to this lesson.

Understanding what a fixed (W, b) does is the prerequisite for understanding what training is trying to do. Everything training changes, it changes through the geometry on this page.

Linear Classifier Step by Step

The default run uses nine labeled points in three clusters, the query x = (−1, −4), and the hand-designed rows

w_A = (1, −1), b_A = 0     w_B = (0, 2), b_B = 0     w_C = (−2, 0), b_C = 0

with all three biases at zero, which puts the whole border structure through the origin.

Setup (beats 1–2). Nine points appear in their class colors — A low and to the right, B up top, C out to the left — plus the query x, drawn hollow because it has no class yet. Then the three prototype arrows leave the origin, one per row of W. Each visibly points into its own cluster; that is what "a row is a direction the class scores highly along" looks like when the model is any good.

Scoring (beats 3–5). The same dot product, three times. For class A, s_A = 1×(−1) + (−1)×(−4) + 0 = 3; a dashed perpendicular drops from x onto w_A's line and the shadow it casts is that score in geometric form. For class B, s_B = 0×(−1) + 2×(−4) + 0 = −8: x sits almost exactly opposite the direction B likes, so its shadow falls behind the origin. For class C, s_C = (−2)×(−1) + 0×(−4) + 0 = 2. Each score commits to a row of the left-hand scoreboard as a signed bar, so all three are on screen at once by the end.

The decision (beat 6). Scores (3, −8, 2). The highest is s_A = 3, so argmax returns A and x takes class A's color. The margin over class C is exactly 1 — a close call, which is what makes the next four beats worth watching. Class B is out of the running by eleven.

The border (beat 7). Where would that verdict flip? Where A and C tie, which by the subtraction above is (w_A − w_C)·x + (b_A − b_C) = 0, i.e. (3, −1)·x = 0, the line x₂ = 3x₁. It appears on the plane, and x sits just barely on A's side of it — the same "barely" as the margin of 1.

Building one region (beat 8). Where does class C win outright, not just against A? Two conditions, and the beat tints them one at a time. s_C ≥ s_A is the half-plane on one side of the line you just met — the same x₂ = 3x₁, now drawn edge to edge, because as a constraint it runs the full width of the plane. s_C ≥ s_B is a second half-plane, bounded by x₁ + x₂ = 0. Where the two tints overlap is class C's region, and nothing else went into it: region(C) = {s_C ≥ s_A} ∩ {s_C ≥ s_B}. Two half-planes, intersected. Every region on this page is built that way, which is what makes "convex" a fact you can derive rather than a word the caption hands you.

The map (beat 9). All three regions at once, and the third tie line with them: A against B along x₁ − 3x₂ = 0. All three lines pass through the origin, because all three biases are zero, and they meet there in a triple point where the three scores are equal. Notice what got trimmed on the way in — each line is now drawn only along the stretch where its two classes really are the top two. Past the triple point, a tie between A and B is two losers agreeing. Every one of the nine labeled points sits in the right wedge: errors 0 of 9.

What convex costs (beat 10). A segment stretches from one end of class C's wedge to the other, a dot at each end. Both dots sit inside class C's region, and so does every point on the line between them — not by luck of this particular W, but forced by the construction two beats ago. Read the consequence straight off the picture: for C to hold two separated blobs, its region would have to reach both of them and skip the corridor in between, and it cannot skip anything. Whatever lives in that corridor, C takes.

Experiment one — direction alone (beats 11–12). Class C's row turns a quarter circle, (−2, 0) → (0, −2). Its length is unchanged, its bias is unchanged, so anything that moves is the direction's doing. C's arrow swings down, and its two borders swing with it. Re-scoring the nine points shows the damage: 3 of 9 now sit in the wrong wedge. C's region has rotated off one of C's own points, which now scores highest for B, and has swung across two of A's points, which now score highest for C. The query flips too, A → C.

Experiment two — bias alone (beats 13–14). W snaps back to the designed rows and only b_C moves, 0 → 3. This time the borders do not turn at all; they slide parallel to themselves, and C's wedge inflates in place. Re-scoring: all nine labeled points keep their class, errors 0 of 9 — the slide was not far enough to reach any of them. But it was far enough to cross x, which flips A → C again. Two experiments, two flips of the same query, by two completely different mechanisms.

Close (beat 15). The designed pose returns, clean map, zero errors. One row per class, one dot product per score, argmax on the result — and straight, convex regions as the standing price.

Switch the preset to Two islands and the same beats run over data no linear model can handle: class C occupies two opposite corners with class A's points strung along the segment between them, which is exactly the arrangement beat 10 rules out. The map arrives already carrying three errors, rotating C makes it four, biasing C makes it five, and an extra beat 15 draws the chord between the two islands before the close moves to beat 16. Any convex region holding both ends of that chord holds the whole segment, including the A points strung along it.

Time and Space Complexity

Prediction costs one dot product per class: O(K·d) time for K classes and d features, plus O(K) for the argmax, with O(1) extra space beyond the K scores. Storage is the matrix itself, O(K·d). Both scale linearly in every direction, which is the practical reason linear classifiers survive as the last layer of nearly every deep network — a 1000-class ImageNet head is exactly this operation, run on features a network has already computed.

Edge Cases

• A tie. Try the On the border preset, which puts x at (−1, −3) so s_A = s_C = 2. The point sits exactly on a boundary and there is genuinely no winner; scores.index(max(scores)) returns A only because A comes first in the list. Every argmax implementation breaks ties by some arbitrary rule, and the honest reading is "on the border", not "class A".
• A point far from everything. The Far corner preset puts x at (5, 5): s = (0, 10, −10), class B by a margin of ten, from a location nowhere near class B's cluster. Confidence in a linear score means "far along this row's direction", not "similar to the training data".
• Data no line can separate. The Two islands preset is the failure case, and it fails structurally rather than by bad luck. Sweeping every integer row for class C leaves at least one point wrong; the region has to be convex, and the data is not.
• A zero row. w_k = (0, 0) scores every point the same b_k, so class k has a constant score; it may still win only the region where every competing score is lower. The input panel rejects it.
• Two identical rows. If w_i = w_j and b_i = b_j, the two classes tie at every point in the plane and the border between them stops being a line. The panel rejects an exact duplicate of class A's or B's row; giving class C the same direction with a different bias is legal and shows the other degenerate case, where one class simply never wins and its region is empty.

Common Mistakes

• Thinking the model stores a boundary. It stores rows and biases. Borders are what you get when you ask where two scores tie; they are computed, not saved. This is why a K-class model has K rows and up to K(K−1)/2 borders rather than one boundary per class.
• Using a single class's zero line as the border. w_A·x + b_A = 0 is where class A's score turns negative, not where class A stops winning. With one class against a fixed reference the two coincide, which is why the binary case gets away with it; with three classes they are different lines, and only the difference of two rows gives the real one.
• Reading a score as a probability. Scores are unbounded and unnormalized: a 3 and a −8 tell you the order, not the odds. Softmax is the function that turns them into probabilities, and it is a separate step — a monotone one, so it changes no verdict and moves no border. Adding it does not buy the model a curve.
• Expecting bias to rotate the borders. It cannot rotate a nondegenerate pairwise border. b_k moves class k's borders parallel to themselves, so the wedge grows or shrinks while keeping its angles. Rows control border normals; in larger class systems, bias can also change which constraints are active and thus the polygon's shape.
• Blaming the optimizer for a convexity failure. When a linear model cannot fit non-convex classes, more training will not help, and neither will a better learning rate. The fix is a different function class — extra features, a kernel, or hidden layers.
• Assuming argmax means the classes are separable. Every point on the plane gets a class, including regions containing no training data at all. A confident answer is not evidence that the question was answerable.