NLP Visualizations
NLP visualizations show the part of language models that words alone make sound like magic: how text becomes tokens, how tokens acquire geometric representations, how context changes those representations, and how a model turns its final vector back into a probability distribution. Each lesson isolates one mechanism and makes its evidence inspectable, from real token IDs and GloVe vectors to a real GPT-2 next-token trace.
A language model never sees your sentence as prose. A classical N-gram model reads token IDs and predicts by counting exact recent histories. A modern neural model turns those IDs into vectors, repeatedly mixes information among the vectors, and finally scores every possible next token. The N-gram lesson is the inspectable baseline; embeddings, attention, transformer blocks, and next-token prediction show how current models replace exact string matches with learned context.
How a Language Model Reads Text
The path from raw text to a working model runs through a few stages, and each lesson here is one of them:
- Split it into tokens. Models read tokens, and a token can be a whole word, a piece of one, or a single byte. The token vocabulary is learned, not handed down, which is why "tokenization" and "training the tokenizer" are two different lessons.
- Count a transparent baseline. An N-gram model remembers the last
n - 1tokens, counts what followed the same history, and exposes why more exact context also creates more missing contexts. - Put tokens into a geometry. Embeddings turn discrete symbols into vectors. Similar contexts create nearby directions, which makes operations such as cosine search and analogy offsets possible — and also carries corpus bias into the representation.
- Turn tokens into vectors that can look at each other. Attention is the engine: every token forms a query and every other token offers a key, and the match between them decides how much one token reads from another.
- Supply position information. Sinusoidal position vectors add a slot-dependent signal to token embeddings, so repeated tokens can enter the model with different inputs.
- Stack it into an architecture. A transformer is attention and feed-forward layers repeated in a pile, which is what lets the same simple operation, run enough times, model language.
- Score what comes next. The final hidden vector becomes one probability per vocabulary token. A decoder chooses one, appends it, and runs the loop again.
The Lessons, From Text to Transformer
| Lesson | What you watch |
|---|---|
| Tokenization | a sentence splitting into the tokens a model actually consumes |
| BPE training | a vocabulary building itself by merging the most frequent adjacent pair |
| N-gram Language Models | exact histories becoming continuation counts, smoothed probabilities, and a visible sparsity failure |
| Word Embeddings | a real 50D relationship offset moving from woman → girl to queen, then exposing a bias failure |
| Attention | each token deciding how much to read from every other token |
| Positional Encoding | sine/cosine pairs turning slot indices into vectors, followed by a controlled encoding-on/off comparison |
| Transformer | attention and feed-forward layers stacked into the architecture behind GPT |
| Next-Token Prediction | GPT-2 turning one hidden vector into 50,257 probabilities, then feeding its own choice back in |
| Decoding Strategies | Greedy, Top-k, and Top-p choosing from the same real GPT-2 logits, with temperature and cutoff controls |
A Recommended Learning Order
Start with tokenization — the gate every input passes through. Watching a sentence break into tokens (and seeing where it breaks in surprising places) makes concrete the fact that a model can only receive symbols from a fixed vocabulary — 50,257 of them for GPT-2. Then BPE training, which answers where that vocabulary comes from: byte-pair encoding starts from single characters and merges the most frequent neighboring pair over and over, growing common words into single tokens while rare ones stay in pieces. Tokenization applies the vocabulary; BPE training is the lesson where you watch it get built.
Before introducing vectors, open N-gram Language Models. Its five-sentence corpus makes every prediction auditable: select a history length, count the exact matches, normalize, and then watch one changed word erase the evidence. Shortening the history recovers matches while losing specificity. That failure is the reason learned representations matter rather than a historical footnote about models that happened to be smaller.
With text turned into tokens, study word embeddings before attention. The GloVe lesson makes cosine similarity and vector offsets concrete with real 50D data, while its bias recipe shows why geometric regularity is evidence, not truth. Static embeddings give each word one vector; attention is where that vector can start depending on context. Each token looks at all the others and weights them by relevance, so bank can read from river or savings.
Before opening the transformer block, study positional encoding. Its repeated-token comparison isolates the position contribution: the same embedding cancels, leaving only the difference between the added position vectors. Then open the transformer, which stacks attention and feed-forward layers with residual connections. Finish with next-token prediction: the last vector is projected into a score for every token, softmax turns those scores into probabilities, and a decoder feeds one choice back into the model. At that point the whole trip from text to text is visible.
After next-token prediction, open decoding strategies to isolate the choice itself. Hold the prefix and random draw fixed, then compare Greedy, Top-k, and Top-p. Temperature changes how the probability is distributed; the retained set determines which tokens can be selected.
All NLP Visualizations
Pick any lesson below to open its interactive player, change its inputs, and step through the mechanism.