Training Loop

The training loop is the procedure that turns a neural network from random weights into a working model: run the data forward, measure the loss, backpropagate the gradients, update every weight, repeat. Gradient-based neural-network training uses this cycle at many scales. The default two-input, four-hidden-unit model has 17 trainable parameters, including biases.

Here is the analogy to keep: training is studying for an exam with a practice set, while a sealed final exam sits in a drawer. The 40 filled points on the canvas are the practice problems — the network grades itself on them every epoch and adjusts. The 20 hollow points are the final: the network never sees their answers, never trains on them, and their results help assess generalization. Because you repeatedly inspect them while choosing settings, they function as a validation set in this playground, despite the UI label “test”; a final unbiased evaluation needs separate untouched data. This playground trains a real network in your browser, live, and the whole lesson is in watching those two scores diverge or agree.

How the Training Loop Works

The network is deliberately small: two inputs, one hidden layer of tanh neurons (you choose 0, 2, 4, or 8), one sigmoid output that reads as a probability. One epoch is one full trip around the loop:

1. Forward — every training point flows through the net: h = tanh(W1·x + b1), then p = sigmoid(W2·h + b2), the network's probability that the point belongs to the positive class. With no hidden layer it is just p = sigmoid(w·x + b).
2. Loss — binary cross-entropy scores every p against the true label. Being confidently right is cheap; being confidently wrong is brutally expensive.
3. Backward — the chain rule (the exact bookkeeping from the Backpropagation lesson) assigns each weight its share of the blame.
4. Update — every weight steps against its gradient, scaled by the learning rate η (the step from the Gradient Descent lesson).
5. Score the final — the loop also evaluates loss and accuracy on the 20 held-out test points. They contribute nothing to the gradients. Pure observation.

The playground runs this full-batch — all 40 practice problems per update — targeting about 160 epochs per wall-clock second at 1× playback (scaled by the speed setting; a busy device may manage fewer). The colored field behind the points is the network's current opinion of every location in the plane: indigo where it would answer "positive", magenta where "negative", fading to nothing along the p = 0.5 frontier where it genuinely can't decide. Every cell of that field is the same forward pass the loss uses, so the picture cannot drift from the math. And everything is seeded — data, initial weights, no shuffling — so Reset replays the identical experiment, every time.

The Training Loop Step by Step

The default run (XOR · η = 0.3 · hidden = 4 · clean) opens paused at epoch 0; press Play and it resolves the perceptron lesson's cliffhanger: a single line provably cycles forever on XOR. Four tanh neurons do not.

1. Epoch 0. Random weights. The field is a faint meaningless wash, and both losses start near ln 2 ≈ 0.69 — the cross-entropy of a coin flip. The network has studied nothing.
2. Early epochs. Both curves drop fast. For a stretch the field looks almost linear — splitting the plane crudely in half is the easiest progress, so that is what gradient descent buys first.
3. The carve. Then the field bends. The four hidden neurons combine into a curved frontier that hands the ++ and −− quadrants to one class and the mixed quadrants to the other — precisely the shape no single line can draw.
4. Auto-pause around epoch 878. Both curves flatten near zero; train and test accuracy read 100%. The run stops once train loss has stayed below 0.02 for 200 consecutive epochs — a window this visualization chose so runs end, not a principled stopping rule. The check runs once per rendered batch of epochs, so the pause can land a few epochs past 878. Real training stops on iteration budgets or validation loss.

Practice score and exam score agree the whole way down. That is what genuine learning looks like — and the three recipe buttons each break it — Too hot and Too simple by turning one knob, Overfit by switching to noisy circle data with 8 neurons:

• Too hot (η = 30): the loss spikes and crashes chaotically; the field flickers between near-solutions and garbage. Note what it does not do: explode to infinity. The derivative with respect to the output logit is bounded (p − y), but gradients with respect to all network parameters need not be. This seeded run shows thrashing rather than unbounded growth — unlike the quadratic bowl in Gradient Descent, where too-large η provably blows up. Instability wears a different costume on every loss.
• Too simple (no hidden layer): logistic regression on XOR. Both curves plateau near 0.69 and accuracy stalls far short of solving it, forever. The optimization is fine; capacity is the binding constraint. This is the 1969 wall, now with instrumentation.
• Overfit (circle · noisy · 8 neurons): the fork, and the reason the held-out exam exists. Test loss bottoms out within the first few hundred epochs, then climbs to more than double its minimum while train loss keeps falling toward zero. Final score: train accuracy 100%, test accuracy stuck below 80%. The network stopped learning the circle and started memorizing the answer key — noise and all.

Overfitting, Underfitting, and Reading the Loss Curves

The solid line is the practice score; the dashed line is the exam. The loop tries to lower the solid line, but a gradient update need not reduce it when the step is too large. The dashed line is the one that tells the truth: a falling dashed line is evidence the network is absorbing real structure, and it can rise as the model fits training-specific patterns or noise; a finite held-out curve is evidence, not an exact detector of that instant.

That gives you a two-curve diagnosis chart. Both falling: learning. Both flat and high can indicate underfitting, but also optimization or data problems; in the no-hidden XOR preset the capacity limit is known. Solid falling while dashed climbs: overfitting — the best network was the one a few hundred epochs ago, which is why real practice watches validation loss and stops early. Both oscillating wildly: the learning rate, not the model, is your problem.

One honest caveat. This is a 2-D toy trained with plain full-batch gradient descent, so every phenomenon plays out in seconds. Real training swaps in minibatches, adaptive optimizers like Adam, and regularization — more machinery around the loop, but the loop itself, forward–loss–backward–update–evaluate-honestly, survives the scale-up unchanged.

Edge Cases

• Linear mode draws a literal line. With no hidden layer the p = 0.5 frontier is exactly w·x + b = 0, so the playground draws it as a crisp line over the field — the same object the perceptron lesson rotates.
• The spiral is a capacity ladder. At the default η = 0.3, a linear model scores barely above chance on it; 2 neurons strain; 8 largely crack it. Capacity buys curvature, one neuron at a time — but the optimizer still has to find it: at η = 30, 8 neurons never do.
• A slow run is not a stuck run. On clean XOR at η = 0.03 both curves are still falling when the 4000-epoch cap lands; on noisy data the dashed one can already be turning up. Tell it apart from "too simple" by slope: slow-but-falling versus flat-forever.
• Resuming means different things. Pause a run mid-flight and Play picks up where it stopped; but once a run ends — converged, capped, or thrashed — Play replays it from epoch 0. Any knob change also restarts from scratch — same seeds, identical replay.

Common Mistakes

• Reading train loss as learning. Train loss falling is the loop doing its job; test loss falling is the evidence of learning. Grading your own practice problems is not an exam result.
• Treating more neurons as monotonically better. On the noisy circle at the default η = 0.3, 8 neurons does worse on held-out data than a smaller net would — surplus capacity plus noise is precisely the overfitting recipe.
• Expecting "too hot" to mean explosion. In this seeded run it means thrashing. A bounded derivative with respect to the output logit does not guarantee bounded gradients or parameters throughout a network. For a genuine blow-up, that's the divergent regime in Gradient Descent — different loss surface, different failure.
• Reading the auto-pause as convergence theory. "Loss below 0.02 for 200 epochs" is a window chosen so runs end on screen, not a stopping criterion anyone should ship.