Forward Propagation

Forward propagation is the computation a neural network runs to turn an input into a prediction: each layer multiplies the incoming values by its weights, adds a bias, and applies a nonlinearity, passing the result left to right until the output layer reports a score for every possible answer. It is inference — the network's only job once training ends.

Think of each layer as a panel of judges. Every judge reads all the evidence coming in, but each has a fixed scoring sheet — the weights — that says how much each piece of evidence counts, and in which direction. The judges never confer; they just total their sheets and pass their verdicts to the next panel. The network here recognizes handwritten digits: 784 pixel brightnesses go in, ten digit scores come out, and the largest score is the guess.

How Forward Propagation Works

The architecture is 784 → 25 → 25 → 10: the flattened 28×28 image, two hidden panels of 25 judges each, and a final panel of 10 — one per digit.

Every judge performs the same two-part computation, and the animation insists on showing the parts separately because conflating them is where intuition goes wrong. First the weighted sum, the pre-activation z = w·a + b: multiply each input by its weight, add everything up, add a bias. Nothing stops z from coming out negative — a judge can be actively repelled by what it sees. Then the ReLU nonlinearity, max(0, z): negative verdicts are clamped to zero and the judge goes silent; positive ones pass through unchanged. A weight is the strength of one connection — positive weights (drawn green) excite the next judge, negative weights (drawn magenta) inhibit it.

The output panel skips ReLU. Its ten raw scores — the logits — go through softmax, pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ), which exponentiates every score and normalizes so the results sum to 1. The exponential is aggressive: a logit gap becomes a probability ratio of e^gap, so a large lead concentrates nearly all the probability. The prediction is the argmax — whichever digit holds the most probability.

One number worth holding onto: the whole pass costs about 784·25 + 25·25 + 25·10 = 20,475 multiply-adds, one per connection, in a single layer-by-layer sweep. This counts connections, excluding bias additions and activation/softmax work; the implementation still uses loops or matrix operations. And it is the weights' values, not their count, that make this work — random values in the same 20,475 connections would not provide the learned digit recognition; chance-level accuracy is a baseline, not a guarantee for every draw. The count is capacity; training fills it. (The 19,600 connections into the first hidden layer are too dense to draw, so the canvas shows a representative subset between the later layers — a drawing constraint, not a property of the network.)

Forward Propagation Step by Step

The default input is a real MNIST test image of a handwritten 7, run through a real trained network — every number on screen is a genuine forward pass.

1. Input. The 28×28 image flattens into 784 pixel values. Ink is high, background near zero.
2. Layer 1 — weighted sum. Each of the 25 judges computes z₁ = x·W₁ + b₁ over all 784 pixels. Ten of the 25 come out negative (drawn blue); the rest positive (dark).
3. Layer 1 — ReLU. max(0, z₁) silences the ten negative judges — they fade to empty — while the positive ones keep their values as the activations a₁.
4. Layer 2 — weighted sum. The drawn connections light up and each layer-2 judge shows z₂ = a₁·W₂ + b₂. Nine of 25 are negative this time.
5. Layer 2 — ReLU. Negatives zeroed again, giving a₂.
6. Output — logits. Ten raw scores z₃ = a₂·W₃ + b₃ appear. The judge for 7 posts about 19.0; the runner-up, digit 2, manages roughly 1.2; most of the rest are negative.
7. Predict. Softmax exponentiates that 17.9-point gap until almost all of the probability mass lands on one digit. The network predicts 7 at ~100%.

The visual grammar to read as it plays: blue means a negative pre-activation, and the ReLU beats are where blue cells fade to empty — the two-step rhythm of "raw sum, then nonlinearity" repeats twice in the hidden layers and once more as "logits, then softmax" at the output. Same shape, three times.

Why a Confident Network Can Still Be Wrong

Softmax confidence measures decisiveness, not truth. The exponential rewards whichever logit leads, however the lead was earned — and the judges' scoring sheets are frozen, so an unusual image that happens to excite the wrong judges produces a confidently wrong verdict by the same honest arithmetic that nailed the 7.

The lesson ships this failure as a first-class input. The Misclassified variant loads, for each digit, the test image this exact network gets wrong with the highest confidence — real images, predictions above 90% on the wrong answer. The seven beats are identical; only the input changed. The Ambiguous variant sits in between: correctly classified, but at ~75%, so softmax leaves a visible runner-up. Run all three for one digit and you get the lesson's real takeaway — the network never changes, so where the probability lands is a property of the input-network pair. This network gets about 3.7% of the MNIST test set wrong, and some of those wrong answers are highly confident.

Where the scoring sheets came from is the next lesson: Gradient Descent shows the update rule and Backpropagation shows how every weight learns its share of the blame.

Edge Cases

• Dead-quiet judges. ReLU outputs exactly zero whenever a weighted sum is negative, so many hidden neurons stay dark for any given image. Normal, not a bug — different images wake different subsets.
• Saturated vs split softmax. A clean digit collapses the distribution to ~100% on one class; an ambiguous one leaves real mass on a runner-up. Both are correct forward passes — only the logit gap differs.
• The all-zero input. With no ink, the first layer’s weighted sums reduce to its biases; later layers also receive the activations those biases produce. the network still produces a full probability distribution. It always answers, even when there is no question.

Common Mistakes

• Confusing logits with probabilities. Logits are unbounded raw scores — 19.0 is not "1900% sure". Only after softmax do the numbers form a distribution you may read as confidence.
• Reading meaning into hidden neurons. Individual hidden judges rarely correspond to anything a human would name — "loop detector" stories are mostly retrofitted. Only the output layer has assigned meanings.
• Dropping the nonlinearity. Without ReLU between them, two linear layers multiply out to one linear layer — the hidden panels would buy you nothing.
• Treating confidence as accuracy. The Misclassified variant exists precisely because >90% softmax on a wrong answer is not an anomaly; it is what the exponential does to any decisive logit gap, deserved or not.