Research · vLLM speculative decoding
On vLLM 0.32.0, MTP speculative decoding made Qwen3.8-27B generate 36 to 40% more tokens a second on a plain throughput benchmark. On agent tasks, the same server completed fewer than half as many tasks an hour. We ran controls that separate the two causes: the memory MTP takes from the KV cache, and less reuse of cached prompts.
Number 3 in our list of things to look out for when serving open models. The MTP and n-gram bugs that changed answers on earlier releases are not present in vLLM 0.32.0's default setup, which these runs use.
MTP at depth 1 raised tokens per second by 40% for one request and 36% for eight. On 192 agent tasks with four agents working at once, it cut successful tasks per hour from 752 to 330. Both servers solved 117 of the 192 tasks, so the difference is time, not accuracy.
MTP's draft layer needs GPU memory, and it comes out of the KV cache, which held 19,727 tokens instead of 36,253. A server without speculation, cut to the same cache size, completed 467 tasks an hour, so the smaller cache explains about two thirds of the loss. The rest is that vLLM reused less of each agent prompt from the cache with MTP on.
Workloads with long prompts that grow at each step and several requests at once, such as agents, on a card where memory is tight. We measured one hybrid model on one RTX 4090 GPU family card. A throughput benchmark with short, unrelated prompts does not show the effect.
Measure speculative decoding on the workload it will serve, by completed tasks per hour. For agents on this model and card, the server without speculation was the fastest: n-gram completed 14% fewer tasks an hour, and MTP more than half fewer.
01 · Two measurements
Each server ran Qwen3.8-27B at 4 bits on vLLM 0.32.0 with prefix caching on. Only the speculative decoding setting, or in the controls the KV cache size, changed between rows.
| Server | One request | Eight at once |
|---|---|---|
| No speculation | 54.1 | 392 |
| MTP, depth 1 | 75.5 (+40%) | 532 (+36%) |
| MTP, depth 3 | 87.8 (+62%) | 372 (−5%) |
| n-gram, depth 3 | 48.8 (−10%) | 323 (−18%) |
On this measure MTP at depth 1 is a clear gain. The agent workload is 192 multi-step tasks from our own task suite. An agent works through each task by calling tools over several steps, and each step sends the whole conversation so far. Four agents ran at once with a 16,384-token context and thinking off, and we evaluated whether each task was completed.
| Server | KV cache, tokens | Solved | Successful tasks per hour | Prompt tokens from the cache | GPU energy per successful task |
|---|---|---|---|---|---|
| No speculation | 36,253 | 117 | 752 | 78% | 1,813 J |
| No speculation, second run | 36,253 | 116 | 747 | 78% | 1,830 J |
| MTP, depth 1 | 19,727 | 117 | 330 | 16% | 4,777 J |
| MTP, depth 3 (context 8,192) | 11,936 | 118 | 274 | 2% | 5,775 J |
| n-gram, depth 3 | 27,405 | 114 | 648 | 61% | 2,306 J |
MTP at depth 3 does not start with a 16,384-token context on this card, so it ran at 8,192. Every server solved between 114 and 118 tasks; two runs of the same server without speculation solved 117 and 116.
02 · Why
To separate the causes, we ran the server without speculation twice more, with its KV cache cut to the size MTP and n-gram had left, and ran MTP once more with vLLM's experimental option to keep its last cache block.
| Server | KV cache, tokens | Successful tasks per hour | Reused all of it | One block short | Reused nothing |
|---|---|---|---|---|---|
| No speculation | 36,253 | 752 | 100% | 0% | 0% |
| MTP, depth 1 | 19,727 | 330 | 14% | 51% | 33% |
| No speculation, cache cut to MTP's size | 19,521 | 467 | 82% | 1% | 16% |
| MTP, depth 1, keeping its last cache block | 19,727 | 397 | 64% | 1% | 32% |
| n-gram, depth 3 | 27,405 | 648 | 89% | 0% | 10% |
| No speculation, cache cut to n-gram's size | 27,190 | 760 | 100% | 0% | 0% |
With its cache cut to MTP's size, the server without speculation completed 467 tasks an hour instead of 752. MTP completed 330, so the smaller cache accounts for 285 of the 422 tasks an hour that MTP lost, about two thirds. With four agents whose prompts grow at every step, a smaller cache holds fewer of them, and more steps have to recompute their prompt. Cut to n-gram's size, the server completed 760 tasks an hour, the same as with the full cache, so n-gram's smaller loss does not come from memory.
Qwen3.8-27B is a hybrid model. Most of its layers keep a running state instead of attending over every past token, so vLLM can resume a cached prompt only from a point where it saved that state, at the end of a cache block. Each agent step sends the previous prompt plus a little more, so at best a step reuses everything up to the previous prompt's last full block. Without speculation every step got that far.
With MTP, 14% of steps got that far and 51% stopped one block short, because vLLM drops the last cache block for drafters of MTP's kind. Speculation also holds extra state for each drafted token, and more steps found nothing to reuse: 33% for MTP against 16% for the server without speculation at the same cache size, and 10% for n-gram against none. With the experimental option to keep the last block, 64% of MTP's steps reused the whole previous prompt and MTP completed 397 tasks an hour, still below the 467 of the server without speculation at the same cache size.
03 · Upstream
Both causes are open on vLLM's issue tracker, so we have not filed a new report.
Reports that MTP lowers the prefix-cache hit rate on hybrid models are open as issue #38182 and issue #54360, and a report that a hybrid 27B-class model with MTP runs only about three sequences at a time is open as issue #55533. Two pull requests that change how speculative decoding uses the prefix cache are open, #61589 and #61911. The second keeps the last cache block for hybrid models, the behaviour behind the “one block short” column. We have not tested either.
Our runs add the effect on a whole agent workload, measured in completed tasks, and split it into its two causes with controls.
04 · Questions
Short answers for anyone deciding whether to turn speculative decoding on for agents.
No. On the plain benchmark, MTP at depth 1 was 40% faster for one request and 36% faster for eight. The loss came from agent-style traffic, long prompts that grow at each step with several agents at once, on a card where memory is tight. A card with more memory to spare for the KV cache would lose less to the first cause. We have not measured one.
Not in our runs. Every server solved between 114 and 118 of the 192 tasks, and on our 400-question check MTP and n-gram on vLLM 0.32.0's default setup matched the server without speculation. Earlier releases had bugs that did change answers.
Completed tasks per hour on traffic shaped like yours, with the KV cache size and the number of preemptions recorded for every server start. Tokens per second on short, unrelated prompts does not show this effect.
One model, Qwen3.8-27B at 4 bits, on one RTX 4090 GPU family card under WSL2, and one agent workload with four agents at once and thinking off. Each setting ran once, except the server without speculation, which ran twice and completed 752 and 747 tasks an hour.
What it took
A throughput benchmark called MTP 36% faster. Counting completed agent tasks showed it finishing less than half as much work.