RNN / LSTM

A recurrent neural network (RNN) processes a sequence one element at a time, folding each new input into a single hidden state vector with the same weight matrix at every step. An LSTM is a recurrent cell with a second state — a memory line that is scaled by a forget gate and added to, rather than recomputed.

The distinction matters because a plain RNN can carry a value forward across a long span and still be unable to send a learning signal back along it.

Here is the thing nobody tells you first: a plain RNN can remember something perfectly well and still be unable to learn it. Those are two different paths through the same network. The forward path carries a value; the backward path carries the gradient that would teach the network which value to carry. In this lesson you watch a trained RNN answer a long-range question correctly, and then you watch the learning signal for that same question shrink to about one part in a hundred thousand on the way back. Both facts are true at once, and the tension between them is the reason the LSTM exists.

How an RNN Works

The whole machine is one line. At each timestep t the cell takes the current token x_t and the previous state h_{t−1}, and produces the next state:

h_t = tanh(W_x x_t + W_h h_{t−1} + b)

Nothing about W_h changes between steps. That is the point — a recurrent net handles a sentence of any length with a fixed number of parameters because it reuses the same matrix at every position. When the sequence ends, a readout layer reads the final state and produces an answer. The state is the network's entire recollection of everything it has seen.

The task on the canvas is subject–verb number agreement, the standard probe for long-range dependency in recurrent nets: the keys to the desk near the door ___ takes are; the key to the desk near the door ___ takes is. Only the second token decides it. Every word after the subject is filler drawn from the same pool for both answers, so nothing later can carry the information. If the model gets it right, the subject's number really did survive all the way down the chain.

Now run the sequence backwards. To learn from a mistake, the network needs to know how the final state would change if the state seven steps earlier had been slightly different. That derivative is ∂h_T/∂h_t, and the chain rule makes it a product:

∂h_t/∂h_{t−1} = diag(1 − h_t²) · W_h

one such matrix per step, multiplied together. Long matrix products can shrink, grow, or preserve different directions, depending on their factors. Which one depends on the factors — and here the factors are stacked against you, because 1 − h² is exactly the term that goes to zero when a unit saturates. And saturation is precisely how a tanh cell latches a value. In this trained latch, saturation preserves the forward value while weakening the backward gradient. That is not an implementation bug you could patch; it is the same equation read in two directions.

The LSTM's answer is to build a second lane that no matrix ever touches:

C_t = f_t ⊙ C_{t−1} + i_t ⊙ g_t, and h_t = o_t ⊙ tanh(C_t)

The cell state C is only ever multiplied elementwise by a forget gate and added to. So the direct path from C_T back to C_t is a product of gate values, not of matrices — and if the gates sit near 1, the product barely moves. (Direct, not total: the gates are computed from h_{t−1}, so there are indirect routes back through the recurrent matrix as well. Those can also shrink or grow through recurrent Jacobians. The straight lane is the one that survives, and it is the one the canvas plots.) That is the constant error carousel, and it is the entire trick. The gates i, f, g, o all come out of one affine map of (x_t, h_{t−1}); the network learns when to keep, when to write, and how much of the memory to expose.

RNN and LSTM Step by Step

The default run is the vanilla RNN on the keys to the desk near the door ___, six filler words between the subject and the blank. Nine timesteps, twelve hidden units, real trained weights.

Chapter one — read the sentence.

1. h₀ = 0. The network starts with no memory at all.
2. t = 1, "the". With h₀ = 0 the state comes entirely from W_x and the bias.
3. t = 2, "keys". The one token that decides the answer. Watch the outlined unit — that is the hidden unit whose final value moves most when you flip the subject, and it is about to latch.
4. t = 3…8: "to", "the", "desk", "near", "the", "door". Six filler words, six applications of the same W_h. The latched unit holds; the rest of the state churns.
5. t = 9, the blank. The readout fires on h₉: are at 100%, is at 0%. Correct, six words after the evidence.

Flip the subject to key in the panel and the answer flips to is. The model is reading the cue, not guessing a majority class.

The hinge. One beat holds that correct answer on screen, re-lights keys seven columns to its left, and draws an arrow from one to the other. The answer was right because of that word. So if it had been wrong, could the correction have reached that word? Everything below is the answer to that one question.

Chapter two — walk the same path backwards.

The gradient lane appears under the unroll on a log axis, and what it plots is the path gain: how much the backward pass scales a correction on its way from the readout to step t. Formally it is the Frobenius norm of ∂h_T/∂h_t divided by √H, which is exactly the root-mean-square gain over directions — point a randomly oriented correction back down the path and this is the factor it shrinks by, on average. At the answer itself the path is the identity and the gain is 1.000. Each hop applies one more Jacobian:

Read the right-hand column, not the left. Seven steps back, a correction leaving the readout arrives at the token that determines the answer scaled, on average, by 8.9 × 10⁻⁶ — one part in a hundred thousand. The forward answer was still perfect.

Two things the number is not. It is not the magnitude of any particular gradient vector — a real training step also depends on where the error points and on the other paths through the network. And it is not a ceiling: it is an average over directions, so some directions do better than 8.9 × 10⁻⁶. The best one is the product's largest singular value, and on this run that comes to 3.1 × 10⁻⁵ — about three and a half times the average, and still one part in thirty thousand. The gap between those two numbers is the difference between "typical" and "best possible"; the thing they agree on, is that the measured hidden-state path attenuates all directions strongly; whether the resulting update is useful also depends on the loss, input, precision, optimizer, and other paths.

Two honest details the table makes visible. The factor is not a constant ρ: it ranges from 0.066 to 1.312, because each hop applies a different Jacobian whose gain depends on how saturated that particular state is. And the last hop is above 1 — the state right after the subject is one of the unsaturated ones, so that step actually amplifies. None of that rescues the product. Averaged over the seven hops it works out to ×0.19 per step, and geometric decay at 0.19 does not care about one hop of 1.3.

The other cell, on the same sentence. You do not have to switch anything to see the difference. One beat near the end lays the LSTM's per-hop factors across the same columns the RNN's are on, so the comparison is a row of numbers rather than an argument: near 0.9 at every hop, against a matrix gain that spends most of the sentence below 0.2. Its gain is measured along the direct cell-state path, the lane built for long range:

The factors barely move. These displayed effective per-hop factors summarize the direct path formed by elementwise forget gates; their product is about 0.43 instead of 0.0000089. That is a factor of about 48,000 between the two cells on the same sentence — and at the widest gap this lesson offers, twelve filler words, it becomes a factor of forty billion.

Switch the cell to LSTM in the panel and you watch that lane get built forward too — the conveyor running under the states, scaled by f and added to at every step, never rewritten. Then drag the gap slider and watch which number moves. The forward answer stays correct at every setting. The reach at the subject falls off a cliff for the RNN and drifts gently for the LSTM. That difference is the entire lesson.

Why the LSTM Is Not Magic

It is tempting to file this under "LSTMs are better". The mechanism is narrower than that, and the animation shows exactly how narrow: the LSTM's memory path is Π f_t, so it is only as flat as the forget gates are close to 1. The verdict beat draws the counterfactual — if every forget gate sat at f = 0.5 instead of near 0.9, the same seven hops would leave 0.5⁷ ≈ 7.8e-3, and thirteen hops would leave 1.2e-4. A forget gate that half-closes turns the constant error carousel straight back into a vanishing product.

So the architecture does not guarantee anything. What it does is make "keep this" a learnable, per-unit decision instead of a fixed property of one shared matrix's spectrum. Forget-gate biases are usually initialised to 1 for exactly this reason — start the gates open, and let training close them where forgetting is actually useful. The models here follow that convention.

What this changes in practice. When a model handles long-range cases badly, two different failures look identical from the outside: it never carried the information, or it carried it and could never be taught to use it. They need different fixes — more capacity or a better input representation for the first, a shorter gradient path for the second — and dependency-path measurements help investigate the distinction. The full Jacobian or its norm may require more than one vector backward pass; a directional probe measures only one direction. Recurrence has not disappeared either: when input arrives one element at a time and the memory per step has to stay constant, as in streaming audio or a control loop, a gated cell can offer fixed-size streaming state; its cost relative to alternatives depends on the implementation and task. What this lesson measures is how far its training signal reaches, not whether it can run.

The same reframing explains why attention eventually replaced recurrence for long sequences. An LSTM shortens the gradient path from a product of T matrices to a product of T scalars. An allowed attention connection can link positions in one layer rather than traversing every intervening recurrent step. Masks, attention weights, depth, and other Jacobians still affect gradient strength; long distance is not cost-free in every sense. Recurrence is where you learn why that mattered.

Edge Cases

• Shortest gap (2 filler words). Three hops back, the RNN still has 7.8% of the signal and the LSTM 61%. This is the control: the collapse is a function of distance, not something wrong with the cell that shows up everywhere.

• Singular subject. Switching to key flips the answer to is at every gap, for both cells. If it did not, the model would be exploiting a class imbalance rather than reading the cue, and the whole lesson would be measuring nothing.

• A hop that amplifies. As the trace above shows, individual factors can exceed 1. A per-step ratio above 1 is normal; the product is what vanishes.

• The h-path of the LSTM decays too. The flat curve on the canvas is the direct cell-state path, Π f. The output path through h runs through o ⊙ tanh(C) and the recurrent matrix, and it decays much like the RNN's. The cell state is the long-range lane, not the whole network.

• Both cells learn this particular task. This is worth stating plainly, because a 48,000× gap in the memory path invites the conclusion that the RNN cannot be trained here, and our own measurement says otherwise. Training each cell from five random seeds under the identical recipe, on sentences with a fixed gap:

  fixed gap	RNN converged	LSTM converged	median iterations to solve
  20	5 / 5	5 / 5	RNN 331 · LSTM 48
  40	2 / 5	2 / 5	too few runs to report

  At gap 20 both cells solve it every time, the LSTM roughly seven times faster. At gap 40 neither is reliable at this size. A task that only has to latch one bit is a weak test of credit assignment — a single saturated tanh unit is a perfectly good latch, and gradient clipping plus Adam paper over a lot. What a healthy memory path buys is credit assignment over long ranges on tasks that need more than one preserved bit; this toy is not that task, and the honest reading of the measurement is "the paths differ by five orders of magnitude, and on this problem both cells still learn."

Common Mistakes

• Reading a correct answer as evidence of a healthy gradient. The most common misreading of this lesson, and the reason the default run shows a correct prediction. Forward retention and backward reach are separate quantities computed from the same weights.
• Reading the path gain as a ceiling. It is a root-mean-square over directions, not a bound. The best-preserved direction beats it — here by a factor of about three and a half — which changes none of the conclusions and all of the phrasing. If you want the honest upper bound, take the largest singular value of the product.
• Believing the decay is a fixed rate ρ per step. The literature's ρ^k is a bound, not a trace. The real product mixes factors from 0.07 to 1.3 depending on which states saturated.
• Thinking the LSTM removed the multiplication. It replaced a matrix product with a scalar product. Gates below 1 still compound; they just compound at a rate the network can control per unit.
• Assuming tanh is the culprit and ReLU would fix it. Swapping the nonlinearity changes which failure you get: non-saturating activations can still shrink or amplify the recurrent product depending on the weights, which is why gradient clipping is standard equipment in recurrent training. Additive state, not a different squashing function, is what fixes the path.
• Generalising from twelve hidden units. This model is a toy: 12 units, a 10-word vocabulary, synthetic sentences. The weights, the forward pass and the Jacobians are real and computed live in your browser, but a production language model is many orders of magnitude larger and stacks several recurrent or attention layers. The mechanism scales; the numbers do not.