Things to look out for when serving open models.

ANSWERS THAT CHANGED

1. vLLMDefault runner clean since 0.30

   MTP speculative decoding repeated or skipped tokens

   On hybrid models, vLLM's older V1 model runner sometimes resumed a request from another request's position when several requests ran at once, so a reply repeated or skipped a token. On vLLM 0.28, 29 of 385 replies with MTP at depth 1 repeated a token, and MTP cost up to 11.9 accuracy points in our benchmark. We traced the cause in vLLM's code, posted a tested one-file patch on vLLM's GitHub, and checked it on vLLM 0.28, 0.30 and 0.32.

   vLLM's newer V2 model runner, the default for MTP since 0.30, does not have the bug. On 0.32.0 the V1 runner still has it, with 11 replies repeating a token, and the patch still removes it.

   What to do: on vLLM 0.30 and later, keep the default model runner. On 0.28, add --no-async-scheduling.

   The investigation →Our patch on GitHub →Found by us · vLLM 0.28, 0.29, 0.30, 0.32 · Qwen3.8-27B
2. vLLMFixed in vLLM 0.32.0

   n-gram speculative decoding changed answers too

   This is a separate bug with the same symptom. n-gram drafts tokens by matching recent text, so many steps have no draft for a request, and after such a step vLLM resumed the request's recurrent state from the wrong slot. On vLLM 0.28 and 0.30, n-gram changed 79 and 80 of 385 replies, of which 12 and 14 repeated a token, and no flag avoided it. In our benchmark it cost 18.2 accuracy points.

   vLLM fixed it on 5 October, and the fix is in 0.32.0. In our runs on 0.32.0, n-gram answered 314 of 400 questions correctly, the same as without speculation, and no reply repeated a token.

   What to do: use vLLM 0.32.0 or later for n-gram on hybrid models. On earlier releases, leave it off.

   The n-gram measurements →vLLM's fix, PR #56531 →vLLM 0.28, 0.30, 0.32 · Qwen3.8-27B

SPEED THAT A BENCHMARK DOES NOT SHOW

3. vLLMReported upstream · fixes open

   Speculative decoding made agent work slower

   With both bugs out of the way, MTP at depth 1 on vLLM 0.32.0 generated 36% more tokens a second than no speculation on a plain benchmark with eight requests at once. On 192 agent tasks with four agents at once, it completed 330 successful tasks an hour, against 752 without speculation, while solving the same number of tasks.

   About two thirds of the loss comes from the memory MTP takes from the KV cache, and the rest from agents' long prompts being reused from the cache less often. Both effects are already reported on vLLM's issue tracker, and two fixes are open there.

   What to do: judge a speculative setting by the tasks it completes on your own workload, not by tokens a second. For agents on this model and card, no speculation was the fastest setting in our runs.

   The measurements →vLLM 0.32 · Qwen3.8-27B
4. SGLangUse one client for every engine

   The same SGLang server measured 1.7 times faster under its own benchmark client

   We sent the same load to the same SGLang 0.5.20 server running Qwen3-8B, 32 requests at once with 88 prompt tokens and 1,000 output tokens each, from two benchmark clients. Under vLLM's client, vllm bench serve, SGLang generated 799 and 800 tokens a second in two runs and never ran more than 29 requests at once. Under SGLang's own client it generated 1,337 to 1,342 tokens a second in three runs and ran all 32.

   The client decided whether SGLang trailed vLLM or matched it, since vLLM generated 1,274 and 1,378 tokens a second on two starts under vLLM's client. SGLang's client sends requests to SGLang's native /generate endpoint instead of the OpenAI-compatible /v1/completions, and we have not tested whether the endpoint or the client makes the difference. TensorRT-LLM gave the same result under both clients, 1,389 and 1,405 tokens a second.

   What to do: send the same load from the same client to every engine you compare, and name the client and the endpoint with the result.

   The runs, in inferlint's docs →SGLang 0.5.20 · Qwen3-8B
5. SGLangCheck what actually runs

   SGLang ran a hybrid model one request at a time

   Qwen3.8-27B is a hybrid model, which keeps a recurrent state for each running request. With its prefix cache on, which is the default, SGLang 0.5.20 needs more memory for that state per request, and on this card it lowered its running limit to one request. It said so once in its start-up log and raised no error. In a start with the limit set to 32, its server-info endpoint still reported 32 while it ran one request at a time.

   With the limit set to 8, it generated 52 tokens a second whether 1, 2, 4, 8 or 16 requests were waiting. With the prefix cache off and the same share of memory, it ran up to 6 requests at once and reached 232 tokens a second.

   What to do: read how many requests the server actually runs under load, from its running-requests gauge or its log, not from the limit you set.

   inferlint's check for it →SGLang 0.5.20 · Qwen3.8-27B

STARTING AND STOPPING THE SERVER

6. vLLMPin the cache size

   The KV cache size depended on whether vLLM compiled the model at start

   vLLM compiles the model on its first start and keeps the result for later starts. A start that compiled used about 2.2 GiB more memory and left that much less for the KV cache. With the same flags, one start printed a cache of 51,079 tokens and the next, which loaded the compiled model, printed 94,448. Under the same load the first peaked at 266 tokens a second and the second at 407, so two runs with identical settings were measuring two different servers.

   What to do: set --kv-cache-memory-bytes, which gave a start that compiled 95,412 tokens of cache and the same 403 to 407 tokens a second, and record the cache size of every start.

   inferlint's check for it →vLLM 0.30 · Qwen3.8-27B
7. vLLMStop the engine process too

   Stopping the server left the engine holding the GPU

   vLLM renames its engine process to VLLM::EngineCore, so a cleanup that looks for “vllm serve” stops the API server and misses the process holding the weights and the KV cache. When we killed the API server, the engine stayed alive and kept between 21,522 and 21,872 MiB of GPU memory on vLLM 0.28 to 0.30, and 22,153 MiB on 0.32.0. The next start then sizes its KV cache from whatever memory is left, and reports no error. When we killed SGLang's and TensorRT-LLM's servers the same way, their workers ended with them and the GPU was free.

   What to do: stop VLLM::EngineCore as well, and check that the card is free before the next start.

   inferlint's check for it →vLLM 0.28, 0.29, 0.30, 0.32

WHAT THE SERVER REPORTS

8. vLLMRead the counter

   Preempted requests left no line in the log

   When the KV cache has no room for a running request, vLLM preempts it: it frees the request's memory and later recomputes it from the prompt and the output so far. At the default log level it writes nothing about this. In one run on vLLM 0.28 it preempted 22 requests, and a search of the log for “preempt” found nothing. Only the counter vllm:num_preemptions_total showed it. A preempted request's latency includes waiting again and recomputing, so a latency result that assumed uninterrupted generation is wrong. SGLang logs each of these and counts them, and TensorRT-LLM logs them without counting them (number 10).

   What to do: read vllm:num_preemptions_total before and after every run.

   inferlint's check for it →vLLM 0.28, 0.29, 0.30
9. TensorRT-LLMCount the streams

   TensorRT-LLM counted requests waiting for memory as running

   TensorRT-LLM's default scheduling policy starts a request only when its whole output is sure to fit in the KV cache. We sent 32 requests at once, each about 1,550 tokens long, to a server whose 37,760-token cache held about 24 of them. Its running gauge reported 32, but at most 24 requests were streaming output at the same moment. The rest waited for room, and the slowest 5% of requests waited 34 seconds for their first token.

   This run used Qwen3-8B's shapes with placeholder weights, which makes the generated text meaningless but leaves the KV cache and the scheduler as they are.

   What to do: count the requests that are streaming output, on the client side, before you report how many ran at once.

   TensorRT-LLM 1.3.0rc29 · Qwen3-8B shapes
10. TensorRT-LLMCount the log lines

    TensorRT-LLM logged paused requests but kept no count of them

    When the KV cache runs out, TensorRT-LLM pauses a request and recomputes it later, which vLLM calls preemption. It logs each pause at INFO level (“request ID 72 -> pause”), and its metrics show how many requests are paused at that moment but keep no running total. Its default policy pauses none, because it admits a request only when its whole output fits (number 9). With the MAX_UTILIZATION policy and a 20,960-token cache, it paused 13 requests in one run, and the total was only in the log.

    What to do: run at INFO log level and count the pause lines, or keep the default policy and watch the time to first token instead.

    How each engine reports it →TensorRT-LLM 1.3.0rc29
11. TensorRT-LLMTime from two readings

    TensorRT-LLM's metrics moved only when a request finished

    TensorRT-LLM updates its gauges for running requests, KV cache use and paused requests, and adds a request's generated tokens to its counter, only when a request completes. In a run of 32 requests its token counter moved 3 times in 126 readings, against 89 times in 137 on vLLM. A tokens-a-second chart drawn from these metrics shows when requests finished, not how fast tokens came out, and when requests finish together a recording can miss the peak load. SGLang's token counter also moves only when a request finishes.

    What to do: compute tokens a second from one reading before the load and one after it, and take the peak number of requests at once from the client.

    How each engine reports it →TensorRT-LLM 1.3.0rc29 · Qwen3-8B