Loss Function

A loss function is the rule a model uses to score how wrong a prediction is, and gradient-based optimization uses its derivatives to determine updates. Two losses can agree completely about which answers are wrong and still disagree wildly about how hard to push.

Mean squared error paired with a sigmoid output produces a gradient 25 times smaller than cross-entropy at exactly the moment the model is most confidently wrong.

That last sentence sounds like a footnote about numerical taste. It is not. Picture two identical students taking the same test, getting the same question wrong, and being graded by two different teachers. One teacher writes "wrong, and here is how far off you were." The other shrugs: "wrong, I guess." Same answer, same mistake — but one of them tells you how much to change. That gap is what this lesson puts on screen: one sigmoid unit, one training example, one learning rate, and two losses racing each other down their own curves.

How a Loss Function Works

A single sigmoid unit computes a number called the logit, written z — the raw score before squashing — and turns it into a probability with p = σ(z) = 1 / (1 + e⁻ᶻ). Training nudges z in whatever direction reduces the loss: z ← z − η·∂L/∂z, where η is the learning rate.

The whole lesson lives in that derivative. Squared error scores the prediction as L = (p − y)², and the chain rule gives

∂L/∂z = 2(p − y) · p(1 − p)

because the sigmoid contributes its own slope dp/dz = p(1 − p). Cross-entropy scores it as L = −[y log p + (1 − y) log(1 − p)], and the same chain rule gives

∂L/∂z = p − y

The p(1 − p) factor cancels exactly. That cancellation is not a lucky accident — cross-entropy is the matched loss for a sigmoid output, designed so the squashing washes out of the gradient.

Now divide one by the other. The ratio between the two learning signals is

1 / (2p(1 − p))

which depends on nothing but confidence. At p = 0.5 it bottoms out at exactly 2. At p = 0.02 it is 25.5. At p = 0.001 it is 500. Squared error's signal collapses precisely where the sigmoid is flattest — which is precisely where the model is most sure of the wrong answer, and precisely where you most want it to move.

One caveat about that number. The 25.5 assumes squared error is written L = (p − y)². Write it the other common way, L = ½(p − y)², and the ratio doubles to 51; multiply the loss by any constant c and it becomes 1 / (2c·p(1 − p)). The headline figure is a scaling convention. What no rescaling touches is the p(1 − p) inside it — a factor that changes with the prediction, and a constant cannot cancel something that varies.

On the canvas both curves share one pair of axes with the logit z running left to right. The x axis has to be z, not p: the saturation lives in dp/dz, so plotting loss against p would hide the entire phenomenon. Drawn against z, MSE is a plateau hugging the axis on the left, and a plateau has no slope to descend. Cross-entropy is a ramp falling from the top-left corner. Two markers — a hollow ring for MSE, a filled disc for cross-entropy — start stacked at the same prediction and then live very different lives.

One warning about the picture: the two curves share a vertical axis carrying two different units. MSE is in probability-squared, cross-entropy in nats. Their heights are not comparable, and the lesson never asks you to compare them. What is comparable is the step each run takes, because both multiply their gradient by the same η and add it to the same z.

Loss Functions Step by Step

The default run starts at p₀ = 0.02 with target y = 1 — the unit is 98% sure the answer is "no" and the answer is "yes" — and both runs share η = 1.00 and a budget of 8 steps. At η = 1 the update is z ← z − ∂L/∂z, so the number on each gradient bar is exactly how far the marker's logit moves (the bar and the logit axis are drawn at different scales).

Start. p = 0.02 corresponds to z = −3.892. Both markers sit at that same x, one on each curve.

Score. The same mistake, priced twice: MSE says (0.02 − 1)² = 0.9604, cross-entropy says −log 0.02 = 3.9120. Different units, so the two numbers say nothing about each other. What matters is what happens next.

Step 1. MSE's slope is 2(0.02 − 1)(0.02)(0.98) = −0.0384. Cross-entropy's is 0.02 − 1 = −0.9800. The ratio is 25.5, and you can see it in the tangents: over the same horizontal span, MSE's falls about six pixels and cross-entropy's falls about 145. The steps that follow are those slopes: MSE moves to p = 0.0208, cross-entropy to p = 0.0516.

Step 2. MSE's gradient is now −0.0398 — it grew, but by less than 4%. Cross-entropy's is −0.9484, still nearly maximal, because (p − y) only shrinks as the prediction actually improves. MSE: p = 0.0216. Cross-entropy: p = 0.1231.

Steps 3 and 4. Cross-entropy climbs 0.2523, then 0.4161 — closing on the decision boundary. MSE crawls to 0.0225, then 0.0234. Its trail is turning into a smudge of dots inside 55 pixels.

Step 5. Cross-entropy's logit crosses zero (z = 0.245), so p = 0.5610 — the prediction flips to the correct side. MSE is at p = 0.0245.

Steps 6 through 8. Cross-entropy settles up to 0.6647, 0.7349, 0.7832, its steps shrinking as the gradient (p − y) finally decays — which is what a well-behaved loss does near the answer. MSE ends at p = 0.0282.

Tally it. Cross-entropy moved the prediction by 0.7632; MSE moved it by 0.0082. Ninety-three times less progress from the same code, the same example, and the same learning rate. Cross-entropy's loss fell 94%; MSE's fell 1.7%. And MSE is not "nearly there" — at this η it needs 33 steps to reach p = 0.5 at all.

Why the Gradient Vanishes Where It Matters Most

The stall has a shape, and the shape is the point. p(1 − p) is a hump: 0.25 at p = 0.5, 0.0196 at p = 0.02, essentially zero at the extremes. Multiply MSE's honest error term 2(p − y) by that hump and you get a gradient that is largest somewhere in the middle and vanishing at both ends — including the end where the model is wrong.

That is the failure mode with a name: saturation. The unit is deep in the flat tail of the sigmoid, and squared error hands it a learning signal proportional to that flatness. The confidently-wrong model, the one most in need of correction, is the one squared error corrects most slowly. That is the shrugging teacher from the top of the page, written out: squared error still marks the answer wrong, but the force of its correction tracks how flat the sigmoid is, not how wrong the prediction is.

Cross-entropy is built to cancel exactly that factor. Its −log p term blows up as p → 0, and the blow-up is calibrated so that after the chain rule the surviving gradient is plain (p − y): proportional to how wrong you are, and to nothing else.

Two scoping notes, because this is where the lesson is easiest to over-learn.

First, this is a property of MSE composed with a saturating output, not of squared error itself. Swap the sigmoid for a linear output and ∂L/∂z becomes 2(p − y), with no vanishing factor anywhere. That is why MSE remains a standard loss for regression — the pathology needs the squashing function to exist.

Second, you can rescue the stall with a bigger learning rate, and the lesson lets you try: at η = 5.00 MSE does cross the boundary, on the very last step of the budget. But look at what that costs. η is one global number shared by every parameter in the network, while p(1 − p) is different for every unit and every example. Tuning η to compensate for saturation means tuning it to a factor that changes from example to example. Choosing the matched loss fixes it once.

Three decisions fall out of this, and they are the practical payload. For a sigmoid or softmax output, use the matched cross-entropy — not as a style preference, but because it is the loss whose gradient survives the squashing. Regression through a linear output keeps squared error, for the reason above. And in a real network the gradient does not stop at z: the chain rule carries it to the weights as ∂L/∂w = (∂L/∂z)·x, so anything that shrinks ∂L/∂z shrinks every weight update fed by it, and every layer behind it.

Edge Cases

• Starting unsure (p₀ = 0.50). The ratio bottoms out at exactly 2.0 and both losses learn — final predictions 0.775 and 0.886. The pathology is saturation-specific, not "MSE is bad."
• Starting already right for y = 1 (p₀ = 0.98). The ratio is 25.5 again, but now both gradients are tiny (0.0008 and 0.0200) and both runs correctly barely move. A small gradient is only a bug when the answer is wrong.
• The target y = 0. The lesson animates y = 1, but nothing in the argument is left-handed. Both gradients keep their form, so a unit sitting at p = 0.98 when the answer is "no" stalls in exactly the same way, in the mirrored tail. Confidently wrong is confidently wrong at either end.
• A learning rate too small (η = 0.25). Neither run reaches p = 0.5 inside 8 steps. The run reports exactly that, rather than claiming a convergence that did not happen.
• p never reaches 0 or 1. log(0) is undefined, and target y = 0 would need log(1 − p) at p = 1. For finite logits the mathematical sigmoid never attains its limits, but floating-point values can round to an endpoint; the generator clamps the probability before any logarithm — so cross-entropy stays finite for every configuration you can dial in.
• The markers can coincide. When both losses sit near zero, the ring and the disc land on the same point. That is not a rendering glitch: it is a ring drawn around a disc, both at their true positions.

Common Mistakes

• Comparing the two curve heights. MSE lives in probability-squared, cross-entropy in nats. "Cross-entropy's loss is bigger, so it must be a harsher grader" is a unit error. Compare the steps, which share η and the same z.
• Believing the loss value drives training. The loop never calls mse(p, y) or cross_entropy(p, y). Delete both functions and training is unchanged, because the update consumes the gradient, not the number. The loss value is a progress report for you; the slope is what the model gets.
• Reading "small gradient" as "broken". Near the correct answer a small gradient is exactly right — it means only small corrections remain. The bug is a small gradient far from the answer.
• Blaming squared error. Squared error is fine. Squared error on a saturating output is the problem, and the fix is either a matched loss (cross-entropy for sigmoid, softmax cross-entropy for multi-class) or an output that does not saturate.
• Computing cross-entropy from the probability in production. This lesson does it in the open — sigmoid first, then log(p) behind a clamp — because you need to see p on screen. Real code does not: BCEWithLogitsLoss in PyTorch and from_logits=True in Keras take the logit z directly and fuse the two steps into one numerically stable expression. Same math, no clamp, far fewer ways to produce a NaN.
• Reaching for a bigger learning rate first. It can paper over the stall, as the η = 5.00 recipe shows. It also amplifies every other gradient in the network by the same factor, and it does not adapt per example the way the missing p(1 − p) factor would need it to.

A Note on Simplification

Everything animated here is real arithmetic — a real sigmoid, real analytic gradients, a fixed 8-step budget — but the setup is deliberately small. A production loss averages over a mini-batch of many examples and many output units. A real unit's logit is z = w·x + b, so a step changes weights rather than z directly; stepping z itself keeps the loss geometry visible without a second variable in the way. There is no regularization, no momentum or Adam. And the two runs are two separate models drawn together for comparison — this comparison does not represent one model optimizing both losses; real training can combine multiple objectives.