Research · vLLM bug investigation
vLLM is a popular open-source engine for serving large language models. From at least version 0.28 through 0.31, its MTP speculative decoding, a feature that speeds up generation by drafting tokens ahead, can change a model's output without raising any error: on hybrid models such as Qwen3.5, it repeats or skips tokens. MTP is not at fault; the bug is in vLLM's older V1 model runner. We caught it with our evaluation tools, used them to trace the cause, fixed it, and validated the fix with the same tools.
POSTED UPSTREAM ON GITHUB vllm-project/vllm Our reproduction and patch are posted in the GitHub repository where vLLM is developed. See our reproduction and patch on GitHub →Follow-up to our Qwen3.8-27B benchmark, where MTP cost up to 11.9 accuracy points.
The bug, who it affects, and what to do. A two-minute read.
Speculative decoding guesses a few tokens ahead and keeps only the guesses the model agrees with, so the output should be the same as without it. MTP uses the model's own built-in guesser. In our benchmark it cost up to 11.9 accuracy points instead. When we compared the replies, some had a token written twice: 214134 came out as 2144134.
A hybrid model, such as Qwen3.5 or Qwen3.6, mixes attention layers with recurrent layers that keep a running memory of the text. After each guess, vLLM must resume that memory at the right point. With several requests running, vLLM's older model runner sometimes gave a request another request's position, so it resumed from the wrong point and repeated or skipped a token.
MTP on a hybrid model, served by vLLM's older (V1) model runner with async scheduling and prefix caching on and several requests at once. We measured it on vLLM 0.28, 0.29 and 0.30, and 0.31 has the same code. vLLM's newer V2 model runner, the default for MTP on 0.30, did not show the bug, and vLLM fixed the related n-gram bug on its own.
On vLLM 0.28, add --no-async-scheduling or --no-enable-prefix-caching. On 0.30, keep the default model runner. Or apply our patch, which changes one file.
The full investigation below has every measurement, the root cause in vLLM's code, and the fix.
Every measurement before and after the fix, the root cause in vLLM's code, and the fix itself.
How we worked
Every step below used the same prompts, the same comparison and the same server settings, so the result before the fix and the result after it can be read side by side.
Two runs without speculation, compared token for token, set how much output changes from batching alone: 9 of 385 replies, none with a repeated token.
Every reply with MTP is compared token for token with the reply without it, and each difference is classified: a repeated token, a skipped token, or another change.
Async scheduling off, prefix caching off, one request at a time, a dense model, a newer vLLM runner. Each control says which conditions the defect needs.
Two partial patches, each changing one operation, showed which of the two operations is wrong.
The fix is a diff to vLLM's source, run as the server's own code rather than as a flag or a configuration change.
The same comparison and the same equivalence check under load, on two vLLM versions and two MTP depths, plus a throughput check.
01 · What we saw
With greedy decoding, speculative decoding is meant to return the tokens the model would produce without it, apart from small numerical differences. Our benchmark measured something far larger.
In our Qwen3.8-27B configuration study on the RTX 4090 GPU family with vLLM 0.28, enabling the model's built-in MTP head cost 6.5 accuracy points on short answers at depth 1 and 11.9 points at depth 3, against a repeat-run variation of three to four answers. n-gram speculative decoding cost 18.2 points. Two runs of the same MTP configuration also disagreed with each other more than two runs without speculation. A loss that size, from a feature designed to be lossless, pointed to a defect rather than to MTP.
02 · Testing before the fix
We sent 400 short-answer prompts to the same server with and without speculation (8 at a time, temperature 0, up to 512 tokens) and compared every reply token for token. The 15 longest prompts exceed the 8,192-token context in every run, which leaves 385 comparisons.
| Configuration | Identical | Different | Repeated a token |
|---|---|---|---|
| No speculation, second run (baseline) | 376 | 9 | 0 |
| MTP-1, defaults | 325 | 60 | 29 |
| MTP-3, defaults | 308 | 77 | 26 |
MTP-1, --no-async-scheduling | 377 | 8 | 0 |
MTP-1, --no-enable-prefix-caching | 377 | 8 | 0 |
| MTP-1, one request at a time | 375 | 10 | 0 |
| MTP-1, vLLM 0.29.0, V1 runner | 348 | 37 | 10 |
| MTP-1, vLLM 0.30.0, V1 runner | 346 | 39 | 13 |
| MTP-1, vLLM 0.30.0, default (V2) runner | 372 | 13 | 0 |
| MTP-3, vLLM 0.30.0, default (V2) runner | 379 | 6 | 0 |
Identical: the reply matches the no-speculation server's token for token. Even two runs without speculation differ on 9 replies, because eight requests at once reorder floating-point work and flip near-ties. In our runs, no reply without the defect repeated a token.
Our equivalence checker, rv-lossless, compares a feature with the same server without it and, wherever two replies diverge, measures how far apart the model scored the two competing tokens. Reordering arithmetic only flips near-ties of a fraction of a nat; a wrong computation moves the scores by many nats.
Measured with no other traffic, MTP on vLLM 0.28 passed: batch rows never move, so the defect never fires. With 16 other requests running it failed. Since then rv-lossless tests every feature under load as well as alone.
| Treatment | Verdict | Splits above 1 nat | Largest swing |
|---|---|---|---|
| Alone | equivalent | 0 | 0.25 nat |
| 16 other requests running | not equivalent | 19 of 48 | 22.2 nat |
16 other requests running, --no-async-scheduling | equivalent | 0 | 0.25 nat |
03 · Root cause
Qwen3.8-27B is a hybrid model: 48 GatedDeltaNet layers that keep a running recurrent state, 16 full-attention layers and one MTP layer. Speculative decoding has to restore that state correctly after every step.
In a speculative step the model checks the token it emitted last and the drafted tokens after it, all at once. For the recurrent layers, vLLM saves the state after each position in a separate slot. The next step must continue from the slot after the last accepted token, so the kernel loads slot num_accepted_tokens − 1. That per-request count is what goes wrong.
d1 accepted, d2 rejected: the accepted count is 2, so the next step must continue from slot 1.
Continues after d1. The reply matches the model without speculation.
The state lacks d1, so the model writes d1 again: a repeated token.
The state includes the rejected d2: a skipped token.
Under async scheduling with Mamba cache mode align, vLLM's V1 model runner (vllm/v1/worker/gpu_model_runner.py) handles the counts in three steps:
_update_states() adds, removes and moves batch rows as requests finish and arrive. swap_states() and condense() move each count with its row._prepare_inputs() then remaps the counts through prev_positions, the old-row-to-new-row map. In our runs the copy has already landed when the rows move, so this applies the row moves a second time, and a request whose row moved reads another request's count.Both conditions are needed. With prefix caching off, vLLM does not read these counts under async scheduling; without async scheduling, they are read in the right order. That is why each flag alone removes the symptom.
Two partial patches separated the two operations. Waiting for the copy before the rows move, while keeping the remap, changed nothing. Removing the remap removed the repeated tokens.
| Partial patch | Different | Repeated a token |
|---|---|---|
| Wait for the copy before rows move; keep the remap | 57 | 24 |
| Remove the remap; no extra wait | 12 | 0 |
04 · The fix
Removing the remap is right only if the copy lands before the rows move. The fix makes that order certain instead of leaving it to timing.
_update_states(), under async scheduling in align mode, wait for the copy of the accepted counts, so every count is in its request's row before any row moves and then moves with it._prepare_inputs(), read the counts directly instead of remapping them again. New rows already start at 1 in add_request().+ if (self.use_async_scheduling + and self.num_accepted_tokens_event is not None + and self.cache_config.mamba_cache_mode == "align"): + self.num_accepted_tokens_event.synchronize() # start of _update_states() ... - if self.use_async_scheduling and prev_req_id_to_index: - prev_idx = self.prev_positions.np[:num_reqs] - ... # gather the counts again through prev_positions + self.num_accepted_tokens.np[:num_reqs] = ( + self.input_batch.num_accepted_tokens_cpu[:num_reqs] + )
An excerpt. The full diff is in our reproduction and patch on vLLM's GitHub. It applies cleanly to vLLM 0.28.0, 0.30.0, 0.31.0 and main.
05 · Testing after the fix
We applied the fix as a source diff, loaded by the server in place of vLLM's own file, and repeated the comparison and the equivalence check.
| Configuration | Identical | Different | Repeated a token |
|---|---|---|---|
| vLLM 0.28.0, MTP-1 (before: 60 different, 29 repeated) | 374 | 11 | 0 |
| vLLM 0.28.0, MTP-3 (before: 77 different, 26 repeated) | 379 | 6 | 0 |
vLLM 0.28.0, MTP-1, --no-async-scheduling | 378 | 7 | 0 |
| vLLM 0.30.0, V1 runner, MTP-1 (before: 39 different, 13 repeated) | 377 | 8 | 0 |
| Treatment | Verdict | Splits above 1 nat | Largest swing |
|---|---|---|---|
| MTP-1 with the fix (before: not equivalent, 22.2 nat) | equivalent | 0 | 0.25 nat |
| MTP-3 with the fix | equivalent | 0 | 0.25 nat |
Every configuration is back in the no-speculation baseline range with no repeated tokens, and under load every divergence is a near-tie of at most 0.25 nat, the same as batching alone produces. The setting with async scheduling off, which the fix does not touch, stays clean.
Throughput stayed within run-to-run noise (MTP-1, one run each): 71.1 against 71.9 tokens per second for one request, 415 against 403 with eight at a time. The added wait gives up overlap that vLLM already gave up a little later in the same step; on faster hardware or smaller models its cost may be larger.
06 · n-gram
n-gram speculative decoding lost 18.2 points in the study. The same checks found a separate defect.
n-gram proposes drafts by matching recent text, so many steps have no draft for a request. vLLM treated such a request as a plain decode, which reads state slot 0, even when its previous step had accepted drafts and its state was in a later slot. The dense Qwen3-8B, which has no recurrent state, showed no repeated tokens with the same n-gram settings, so the draft verifier itself is fine.
vLLM has since fixed this on its main branch (2026-10-05, after the 0.31.0 release). We ran the same checks on that fix.
| Configuration | Identical | Different | Repeated a token |
|---|---|---|---|
| vLLM 0.28.0, before | 306 | 79 | 12 |
| vLLM 0.28.0, after (fix backported) | 379 | 6 | 0 |
| vLLM 0.30.0, before | 305 | 80 | 14 |
| vLLM 0.30.0, after | 374 | 11 | 0 |
rv-lossless judged unpatched n-gram not equivalent even with no other traffic (36 of 68 divergences above 1 nat, up to 16 nats), and equivalent after the fix, alone and with 16 other requests running.
07 · Questions
Short answers for anyone running MTP or n-gram speculative decoding on a hybrid model with vLLM.
A hybrid model mixes ordinary attention layers with recurrent layers, such as GatedDeltaNet or Mamba, that keep a running memory of the text instead of attending over every past token. Qwen3.5 and Qwen3.6 are hybrid; the Qwen3.8-27B we tested has 48 GatedDeltaNet layers and 16 attention layers. Not every Qwen model is hybrid: Qwen3-8B is a plain transformer, and with n-gram speculative decoding it showed no repeated tokens in our tests.
It is meant to be, up to the small numerical differences that batching causes. The large accuracy loss we measured with MTP on vLLM 0.28 came from a bug in vLLM's V1 model runner. With the bug worked around or fixed, MTP-1 and MTP-3 matched the no-speculation baseline.
The bug is in the V1 model runner. We measured it on vLLM 0.28.0, where the V1 runner is the default path for this model, and on 0.29.0 and 0.30.0 with the V1 runner. vLLM 0.31.0 and the main branch on 2026-10-06 contain the same code; we checked them by reading the code, not by running them. On those versions MTP reaches this code when it runs on the V1 runner: with VLLM_USE_V2_MODEL_RUNNER=0, when MTP is combined with a feature the V2 runner does not support yet, or on platforms without Triton. We did not test versions before 0.28. The V2 model runner, used by default for MTP on 0.30.0, did not show the bug.
A hybrid model with recurrent layers (GatedDeltaNet or Mamba, for example the Qwen3.5 family), MTP speculative decoding, async scheduling and prefix caching (Mamba cache mode align), with several requests running so that batch rows move as requests finish and arrive. With one request at a time it does not show.
On vLLM 0.28, add --no-async-scheduling or --no-enable-prefix-caching. Both removed the repeated tokens in our tests. Prefix caching off kept more of MTP's speed in our measurement but gives up prefix reuse. On 0.30.0, the version we measured, keep the default V2 model runner.
Yes. Our tested patch is in our comment on vLLM's GitHub, with the reproduction. It changes one file, vllm/v1/worker/gpu_model_runner.py, and applies cleanly to vLLM 0.28.0, 0.30.0, 0.31.0 and main.
n-gram has a separate bug with the same symptom on hybrid models. vLLM fixed it on its main branch on 2026-10-05, after 0.31.0 was released, and the fix passed our checks. Until a release includes it, leave n-gram off on these models.
What it took
Aggregate accuracy and single-request tests did not catch this. Comparing outputs token by token, under load, did.