Part one of a five-part walkthrough of what happens between your prompt and the next token. Every part uses one reference model, Qwen2.5-0.5B, so the numbers line up from start to finish. See the whole series →
Type “The apple is sweet” and the model generates one token. To continue, it runs the same pipeline again with that token added to the context. This page takes one pass apart: CPU work in blue, GPU work in green, a diagram for each stage, and two 3-D benches that show the tensor moving through the model.
Blue is the host - i.e. the CPU. Green is the device - i.e. the GPU. The top green row is the forward pass through depth; the bottom green row is the head that turns 896 numbers into vocabulary scores. Click any block to jump to its explanation.
Each stop names the tensor it produces and its shape. Following those shapes keeps the operations tied to concrete data.
The tokenizer receives a byte sequence. After tokenization, the model answers one question: given the vocabulary entries so far, which entry comes next?
At this point nothing explicitly marks sweet as an adjective or says that the sentence may end. Those relationships have to emerge from the computations that follow.
Byte-pair encoding merges frequent adjacent pairs until nothing in its learned merge list applies. The result is a list of vocabulary entries, and each entry has a fixed integer id — position in the dictionary, nothing more.
The leading space is part of the token: ␣apple and apple are different ids. A rare word can split into several entries; Numidroid might use five tokens where apple uses one.
This is branchy string manipulation over a hash table. There is nothing here a GPU is good at, so it stays on the host. What crosses to the device is a few hundred bytes of integers.
The embedding table has one row per vocabulary entry, each row 896 numbers long. Turning an id into a vector is not a multiplication — it is a row copy. Id 23268 means “fetch row 23268”.
d_model is the width of the pipe that runs the length of the model: 896 for Qwen2.5-0.5B, 4096 for Llama 3 8B. Every hidden state, every residual, every attention output is this wide.
Each row is one token. The 896 numbers along a row are that token's features — 896 is the width of a row, the size of the vector, not a count of columns of anything. This convention lets one set of kernels serve a four-token prompt and a four-thousand-token one: the shape changes, the code does not.
Adding a batch dimension in front gives [batch, tokens, d_model], and that is the tensor shape you will see printed in every debugger from here to the end of the model.
The tensor now passes through 24 identical-in-shape, different-in-weights blocks. Once it is inside, stop calling it an embedding: it is a hidden state, and specifically a residual stream that each sublayer reads and writes back into.
One block contains four main operations: normalize, attend, add the result to the residual stream; normalize again, run the MLP, and add that result back. Each sublayer modifies the residual stream instead of replacing it.
Attention is the only place rows talk. With a causal mask, row 3 may look at rows 0–3 and no further, which is exactly what makes the model a next-token predictor rather than a fill-in-the-blank one.
After layer 24, a final RMSNorm sets the scale of the hidden states before the LM head. Omitting it changes the logit scale and therefore changes the effect of temperature.
Four rows went in, but only row 3 — the final prompt token — is read by the LM head for this generation step. Rows 0–2 were computed through all 24 layers; their final hidden states are not read out.
That waste is only waste at generation time. During training every row predicts its own successor, so one forward pass yields n training signals instead of one.
The LM head multiplies the final 896-number hidden state by a [896 × 151,936] matrix. The result is one logit for each vocabulary entry.
Logits are unbounded scores and are meaningful only relative to one another. A logit of 57.29 says little alone; its position among the other 151,935 scores determines its probability after softmax.
In Qwen2.5-0.5B this LM head matrix is the embedding table transposed — the same 136M parameters used at both ends. Llama 3 keeps them separate. Either way it is one of the largest single tensors in a small model.
Every logit is divided by T. Dividing by a small number less than 1 stretches the gaps between scores, so after the softmax the leader takes almost everything. Dividing by a large number squashes the gaps and the distribution flattens.
T does not add noise — it reshapes the distribution the sampler will draw from. Ordering never changes. T → 0 is argmax — no randomness at all; T = 1 leaves the model's own distribution alone.
Top-k and top-p are separate scissors, usually applied around this step: keep the best k entries, or the smallest set whose probability sums past p, and zero the rest before renormalizing. The bench below lets you move all three.
Softmax exponentiates each logit and divides by the sum of all exponentiated logits. The result is a probability distribution over 151,936 vocabulary entries that sums to 1. Because softmax is monotonic, the ranking stays the same.
Argmax chooses the highest-probability entry every time. Sampling assigns each entry a segment of the interval from 0 to 1, sized by its probability, then draws a random point on that interval. The leading token wins most often, but lower-probability tokens can still be selected.
The id goes back to the CPU and through the dictionary in reverse: 323 becomes " and". Streaming decoders buffer here, because byte-level Byte Pair Encoding (BPE) means one id can be an incomplete UTF-8 sequence — the reason an emoji sometimes lands a beat late.
The same id is embedded on the device as row 4, and the pipeline runs again with one more token of context. Autoregressive generation repeats this loop until a stopping condition is reached.
Twelve plausible continuations of “The apple is sweet”, with hand-picked logits. Everything downstream of the logits is the real arithmetic — divide, filter, exponentiate, normalize, draw.
One token's journey, on repeat. Prompt text enters on the blue host slab, becomes token rows on the green device, travels through 24 layers, collapses to one readout row, fans out into logits, passes through temperature and sampling, and returns as the selected token. Drag to orbit; scroll to zoom.
The same scene under manual control. Play advances stage by stage and stops; the stepper moves one stage; the dial scrubs anywhere in between so you can hold the tensor mid-flight.
One turn, end to end. Blue runs on the host, green on the device, and red marks a value rather than a step.