Next-Token Prediction

Next-token prediction is the core generation operation of an autoregressive language model: given a sequence of tokens, it produces a probability distribution over its entire vocabulary — 50,257 numbers for GPT-2 — describing what might come next. Text appears because something outside the model picks one of those tokens, appends it to the prefix, and runs the whole thing again.

This lesson uses the simplest possible picker, greedy decoding: take the highest-probability token, every time.

That loop is the entire trick, and it is stranger than it sounds. Ask GPT-2 what follows The capital of France is and it does not answer "Paris". It ranks every token it knows, and the one it puts first is the — with the leading space, because in GPT-2 a space belongs to the word after it, which is why the animation prints those tokens as ·the. It gets p = 0.0846. Paris comes fifth, at p = 0.0322, behind now, a, and even France again. Two thirds of its probability mass sits outside the eight tokens you can fit on a chart. The model is far less certain than its fluent output later suggests; fluency is a property of the loop, not of confidence.

How Next-Token Prediction Works

One pass of the model is five lines of arithmetic with a very large matrix in the middle.

The transformer stack reads the whole prefix and produces position representations; next-token generation selects the vector for the last position: h, 768 numbers in GPT-2 small. Everything the model has concluded about what should come next is compressed into that vector. Multiply it by the unembedding matrix W_u — 50,257 × 768 — and you get one raw score, a logit, for every token in the vocabulary. Softmax exponentiates and normalizes those scores into probabilities that sum to 1.

The model uses embedding lookups, but this example has no external retrieval step or explicit geography-question branch. There is a matrix multiply that scores 50,257 candidates, and the candidates it scores highest after The capital of France is are mostly grammatical continuations — the, a, now — in this recorded prediction; the ranking alone does not establish why those scores were learned.

Two things you can do with that distribution, and the difference between them is the whole lesson.

Score real text. Take a sentence you already have. At each position, look up the probability the model assigned to the token that actually came next. Convert it to bits of surprise with −log₂(p), average over the sentence, and you have measured how well the model predicts real language. This is the negative-log-likelihood objective used in training. A causal forward pass can score multiple positions together. Here the browser displays fixed scores and does not update weights.

Generate. Run the same forward pass, take the highest-probability token, append it to the prefix, and repeat. Now the model is reading its own output.

Those two procedures differ by one operation. Teacher forcing reads the real next token from ids; greedy generation appends argmax(p) instead. Same model, same matrix, same softmax. One substitution turns a measuring instrument into a writer.

Next-Token Prediction Step by Step

The default run scores The capital of France is Paris. and hands the pen to the model after five tokens.

GPT-2 sees seven tokens, not seven words: The, capital, of, France, is, Paris, .

Act 1 — one prediction, taken apart. Feed the model the first five tokens. It returns h, the 768-d vector; h · W_uᵀ gives 50,257 logits; softmax makes them probabilities. The chart shows eight candidates; the first five are listed below, with logits shifted so the leader reads +0.00:

Those eight bars hold 32.02% between them. The remaining 50,249 tokens hold 67.98% — the flat, invisible bulk of the distribution, which is why the animation gives the tail its own meter instead of pretending it can be a ninth bar. Entropy is 8.65 bits: about as uncertain as picking uniformly among 400 options.

Act 2 — the training objective. Now score the real sentence, position by position. The prefix grows one token at a time and each position is priced in bits:

Position 0 is the honest one. After a single token The, the real continuation capital is rank 1,412 — not on the chart, not close to it, and it costs 13.08 bits. Training does not care that the model finds it absurd; it prices the token that actually came next and, during optimization, supplies a gradient for increasing its probability. Averaged over the six positions: 5.4797 bits per token, a perplexity of 44.62. Loosely: at each step the model is about as uncertain as if it were guessing among 45 equally likely tokens.

Act 3 — the loop. Throw away everything after position 5 and let the model continue on its own. Twelve steps of argmax, each pick appended to the prefix that produces the next distribution:

the capital of the French Republic, and the capital of the

Fluent, grammatical, and going nowhere — twelve tokens in, it has not finished a sentence, it has restarted a noun phrase. And it cost the model 1.9445 bits per token — steps 2 and 10 ( of, after the capital) cost 0.11 and 0.09 bits, because the model assigns that continuation high probability, though alternatives remain possible.

Act 4 — the verdict. Two averages land on one axis: 5.48 bits per token for the sentence a human wrote, 1.94 for the sentence the model wrote. Before reading anything into the gap between them, notice what those two numbers do not have in common. They start at different positions, run to different lengths, and describe different continuations. They are a summary of the two bit strips on screen, not a controlled experiment.

The controlled experiment is one step, and the animation has been sitting on it since Act 1. At the handoff, the real text and the model are reading the same distribution — one prefix, one forward pass, one set of 50,257 numbers. The text's next token is Paris, and it costs 4.95 bits. The model's argmax is the, and it costs 3.56. The only variable is who chose. And nothing in that vocabulary could have cost the model less than the did, because "the argmax" and "the cheapest token available" are two names for the same thing.

That much is guaranteed, at every step, by arithmetic. The 2.82× is not. One source of the gap is unrelated to the local greedy choice: the real text's average includes position 0, where the prefix was the single token The and the true continuation cost 13.08 bits. Score only the positions after the handoff and the same run reads 3.45 against 1.94. Report the per-step comparison, which holds every time; do not report the multiplier as if it were the size of the greedy effect.

Then watch what the greedy choice actually costs. Steps 0–3 are the capital of the. Steps 8–11 are the capital of the again, the identical four tokens. This is repeated text within the twelve-token budget, not proof of a state cycle: the full prefix has grown.

Why Every Greedy Pick Is Locally Cheapest

Real language is full of choices a predictor cannot see coming: which fact the writer will bring up, which name, which of six synonyms. Those choices are exactly the expensive positions — France at rank 14, capital at rank 1,412. Human text is expensive to the model because it carries information the model does not have.

Greedy generation removes every one of those choices. At each step it takes the argmax, which means it takes the token with the lowest surprise available. Do that twelve times and you have twelve low-surprise tokens by definition. That much is arithmetic: at every step, nothing in the vocabulary could have cost less than the token greedy took.

The gap between the two averages is a weaker thing. It compares different spans over different continuations, and neither its size nor its direction follows from the per-step rule — a greedy run that wanders into a genuinely uncertain region can be expensive too. What no version of the comparison measures is quality: perplexity computed on a model's own output is a fact about the decoder.

The counting preset makes that concrete. Switch the text to 1, 2, 3, 4, 5, 6 and hand over the pen after three tokens: there the argmax and the real next token are the same token, ,, at the identical 1.45 bits. Zero greedy advantage at the handoff — and the run still reports 2.77× cheaper, almost entirely because the counting sentence's opening positions are expensive and its later ones are not. A ratio between two averages over different spans is a description of two spans. The per-step inequality is the claim that survives.

Greedy decoding can favor recurring phrases because it always selects the local leader. Here a four-token phrase repeats within twelve tokens, but the full prefixes differ, so this does not demonstrate an infinite deterministic cycle. Sampling methods such as temperature, top-k, and top-p can change these choices; they do not guarantee that repetition disappears.

Edge Cases

• The real token can be off the chart. In this sentence, 2 of 6 positions have their true token outside the top 8, one of them at rank 1,412. The animation gives those a card of their own rather than silently dropping them; a chart of the top 8 is a viewport, not the distribution.
• The model can be very sure. Switch the text to 1, 2, 3, 4, 5, 6 and the picture inverts: at Position 9, where the real token is 6, it gets p = 0.9096 and entropy falls to 0.98 bits. Uncertainty is a property of the prefix, not a permanent state of the model. This recorded continuation keeps counting; that observation does not guarantee that every longer run will do so.
• Handing over the pen at the very end. If the prefix is the whole sentence, there is no "real next token" to look for, and the model simply continues past the period — often by reproducing what it just read, at p ≈ 1.00. A model conditioned on its own recent text is extremely good at predicting its own recent text.
• Fluent and false. Try Water boils at 100 degrees Celsius at sea level. with the pen handed over after three tokens. GPT-2 writes "about 1,000 degrees Fahrenheit" — grammatical but false; at sea level water boils at about 212 °F, or 100 °C. The animation prints the correction next to it, because a canvas that shows a falsehood without marking it teaches the falsehood. Low surprise is not truth. The model is optimizing the probability of the next token, and nothing in that objective mentions the world.

Common Mistakes

• Thinking the model outputs a word. It outputs 50,257 numbers. Which token you get depends on the decoding rule you wrap around it, and that rule is not part of the model.
• Reading a top-k bar chart as the distribution. Eight bars are 32% of the story here. The 67.98% you cannot draw is where most of the model's uncertainty lives, and it is why entropy — a number computed over the whole vocabulary — is worth more than the chart.
• Confusing rank with probability. Paris is rank 5 and the is rank 1, but the gap between them is 0.05 in probability. Ranks alone conceal the size of probability gaps. A different prefix may change both ranks and probabilities.
• Believing low perplexity means good writing. Repetitive text can have low perplexity under the model that produces it; this is not guaranteed for every repetition. Perplexity measures how well a model predicts a given text; turned on text the model generated itself, it describes the model–decoder–prefix combination rather than writing quality.
• Assuming the model has memory between calls. Each forward pass sees a prefix and nothing else. Earlier tokens remain available through the supplied context (and consistent cached representations). Removing one token changes that context, but later tokens may still carry information about it.

A Note on Simplification

Everything on screen is a real GPT-2 small (124M) forward pass, exported offline — but GPT-2 small is an untuned base model from 2019. Model training and decoding both affect repetition; greater size alone does not guarantee its absence. Decoding here is pure greedy argmax — no temperature, no top-k, no top-p, no repetition penalty — which is the strictest case and belongs to a separate lesson. The animation also compresses twelve transformer blocks into a single slab (attention and the transformer block have lessons of their own), shows 8 of 50,257 logits, and skips the KV cache, which reuses earlier key/value vectors instead of recomputing the prefix. Generation still makes a new-token pass per step and attention still reads the available context.

Surprise is reported in bits (log₂) here so it pairs directly with perplexity = 2^(bits/token). The softmax and cross-entropy lesson measures the same quantity in nats (natural log); 1 nat = 1.4427 bits.