One character.
Many possibilities.
A language model makes one decision at a time. Give it a beginning, look at the probabilities, and see what comes next.
01 Choose a way to learn
Same task. Different machinery.02 Give it a beginning
BIGRAMReshapes the sampling probabilities. The learned weights stay the same.
Repeat the experiment
The same model, prefix, temperature and seed reproduce the same sequence.
03 What might come next?
LIVEThe model scores every possible next character.
A probability, not a promise. Each step samples from this distribution. END means the model chooses to finish the name.
WHAT THIS MODEL CAN SEE
INSIDE THE TRANSFORMER
Where the last position attends
Each row is one head. Darker cells carry more attention weight. Attention is one part of the computation, not a full explanation of a prediction.
Follow the decisions 0 steps
| Step | Context before sampling | Selected | Probability | Temperature |
|---|---|---|---|---|
| Take a step to see the decision recorded here. | ||||
One context, three predictions
Same temperature. Same starting information.These models have different parameter counts and were trained on the ONS baby names corpus (1996–2025). This is an exploration of mechanics, not a model ranking.
THE IDEA IN A MINUTE
Predict. Sample.
Repeat.
These models learned patterns from a list of names. Every generated character becomes part of the context for the next decision.
- 01
Look at the distribution.
Try "ma". Which letters are likely? Switch models and see how their predictions differ.
- 02
Make one decision.
Click Next character. The model samples a character, then calculates a fresh distribution for the new context.
- 03
Change one variable.
Lower the temperature. The distribution concentrates around its favourites. Raise it to give alternatives more of a chance.
Three models, three approaches
Same task. Different machinery.01 / COUNTS
Bigram
The simplest possible model: a lookup table of what tends to follow what.
- Context
- 1 character — only the immediately preceding character is considered.
- How it learns
- It counts how often each character follows every other character across all training names. No gradient descent — just tallying occurrences and dividing by totals.
- How it predicts
- Given the last character, it looks up that row in its table and returns the observed frequencies as probabilities.
- Limitation
- It has no memory beyond a single character, so it cannot learn that "qu" is common while "qz" almost never appears.
02 / CONNECTIONS
Neural network
A small learned function that maps a window of characters to a next-character prediction.
- Context
- 6 characters — a fixed-length window of recent characters.
- How it learns
- Each character is converted to a short vector of numbers called an embedding. The six embeddings are concatenated and passed through a hidden layer (with a non-linear activation), then to an output layer. Weights are adjusted by gradient descent to minimise prediction error.
- How it predicts
- The same forward pass runs at inference: embed → concatenate → hidden layer → output probabilities.
- What it gains
- It can learn that combinations like "ann" carry different meaning than "zzz" — something a bigram cannot capture — because it sees a wider context and learns richer representations.
03 / ATTENTION
Transformer
A model that lets every position look at every earlier position and decide what is relevant.
- Context
- 6 characters — but each position can attend to all of them simultaneously.
- How it learns
- Characters are embedded and positional information is added. Self-attention heads learn which earlier characters matter most to each position. Two heads run in parallel, each free to capture a different pattern. Weights are learned by gradient descent.
- How it predicts
- The final-position representation — enriched by what every attention head found relevant — is passed to an output layer. The attention grid in the playground shows this live.
- What it gains
- Heads can specialise. One might track vowel patterns; another might track name endings. The model can weigh distant characters directly, not just through a fixed window.
Why not always pick the most likely character?
The model outputs a probability distribution — a score for every possible next character that sums to 1. Rather than always taking the top-scoring character (called greedy decoding), this demo samples from that distribution.
Think of it like a weighted lottery. If "a" has a 40 % probability and "n" has 20 %, "a" is twice as likely to be drawn — but "n" still wins sometimes. This means the same prefix can produce many different names, all plausible given what the model learned.
Greedy decoding would always yield the same name from the same prefix. Sampling preserves the genuine variety that exists in the data.
- Greedy (always top)
- Deterministic. The same prefix always produces the same name. Tends towards the most "average" output and quickly becomes repetitive across runs.
- Sampling (weighted random)
- Stochastic. The distribution guides the choice but does not fix it. High-probability characters win most often; low-probability ones still get a turn. Produces variety that still reflects learned patterns.
- Temperature
- A dial that reshapes the distribution before sampling. Below 1, it concentrates probability mass on the favourites — closer to greedy. Above 1, it spreads mass more evenly, giving rare characters a better chance and producing more surprising outputs.
THE MODELS
TRAINING & INFERENCE
All weights were trained with the repository's NumPy engine. The MLP uses explicitly derived gradients. The Transformer uses the repository's own automatic differentiation engine. Browser inference is an independently checked JavaScript forward pass. There is no API call for prediction.
LIMITATIONS
The corpus is drawn from ONS baby name records (England and Wales, 1996–2025) and reflects names registered in that context. Generated strings may be unusual, repeat training names, or reflect biases present in the underlying records. Training and validation records are separated. Checkpoints are selected by validation loss, and test loss is evaluated after selection. These are character-level models, not chatbots or large language models.
CREDITS
Inspired by Andrej Karpathy's makemore. Source, training scripts and tests are included in this repository under the MIT license.