SGLang vs vLLM throughput: a deep dive.

Under vLLM's client, SGLang looked about 40% slower than vLLM. Under its own client, it was as fast as vLLM. TensorRT-LLM gave about the same speed under both clients, so the question was why SGLang's number moved.

Under vLLM's client, SGLang never ran more than 29 requests at once. Under its own client, it ran all 32. The two clients also use different endpoints. SGLang's client sends requests to SGLang's native /generate endpoint, and vLLM's client uses the OpenAI-compatible /v1/completions. So the difference could have come from the client or from the endpoint. We published what we had seen as number 4 in our findings, and planned runs to tell the two apart.

The check at each step

At each step, SGLang goes through the waiting requests and starts each one that fits in its KV cache, which held 32,096 tokens here. It counts a new request at its prompt plus its whole reply: 88 + 1,000 + 1 = 1,089 tokens. It counts a running request at the tokens it already holds plus about 70% of what it still has to write, which is about 788 tokens at the start. Counting running requests lower lets more of them run together. If the cache does fill up, SGLang pauses a request and resumes it later.

The cache had room for 29 of these requests at full length, so the last 3 of the 32 could start only on the lower count.

vLLM 0.32.0 works differently. It starts each request as it arrives, gives it more cache as its reply grows, and pauses the newest request when the cache runs out. So the way the requests arrive does not change what vLLM does.

Two runs, step by step

The slow runs: the rest arrived while only a few were running

In every slow run, one request had started on its own (five in two of the runs) when the other requests reached SGLang's queue together. With one request running, 28 of the 31 fit, and the next one did not. When a request does not fit while others are running, SGLang marks its running batch as full. It then skips the check at every following step until a running request finishes.

One step later, with the 28 counted as running, SGLang's own count left about 9,200 tokens free, and the 3 waiting requests needed about 3,300. But the check did not run again. Every reply in this benchmark is exactly 1,000 tokens long, so all 29 running requests finished at the same step, and the 3 waited for the whole first group.

The fast runs: the rest arrived while many were running

In the fast runs, 12 or more requests were already running when the rest arrived, for example 25 and then 6. Counted at about 788 tokens each, the running requests left room for every waiting one, so the batch was never marked full. With all 32 running, the cache filled up shortly before the end. SGLang paused 1 to 3 requests for about 2 seconds and then finished them, which cost far less than the 21-second wait in the slow runs.

What decides which path a run takes

Both clients send all 32 requests at once, but the requests reach SGLang's scheduler over a few milliseconds. How they fell across its steps changed from run to run, with nothing changed on the client or the server, and we saw both paths on the same server a few minutes apart. The number of requests already running when the rest arrive decides the path, and we tested that directly in the next section.

The client did not send fewer requests in the slow runs. The requests that waited were in SGLang's own queue, and every client reported all 32 completed. All 30 runs with a saved server log followed one of the two paths, and in each one the path matched the speed.

With the queue checked at every step, all 10 runs were fast, at the speed of the fast runs before. In 6 of them, the requests reached SGLang's queue exactly as in the slow runs and 28 started, but the last 3 started at the next step instead of about 21 seconds later.

With room for all 32 at full size, all 10 runs were fast, and they were the fastest we measured, at 1,428 to 1,473 output tokens a second. No request waited, and SGLang never needed to pause one near the end. Each run took about 22 seconds, against about 24 for a fast run on the smaller cache.

Cache size also explains part of the gap when each engine sizes its own cache. At 0.88 of GPU memory, vLLM 0.32.0 set aside 39,152 tokens on the same card, enough for all 32 requests at full length, and generated 1,457 to 1,461 output tokens a second with nothing paused. SGLang set aside 32,096 tokens at the same setting, and needs 0.90 to have room for all 32.

With the arrivals from section 04 set by our client, the larger cache was fast for every arrival on both SGLang 0.5.20 and 0.5.22, at 1,428 to 1,454 output tokens a second. What matters is the size, not the setting. With 1,120-token replies, the same cache had room for only 29 requests at full length, and on 0.5.20 one request first was slow again, at 796 to 798.

As released, SGLang took the slow path in 7 of these 10 runs. In one of them, 31 requests started and a single one waited, and that run took as long as the others, about 39 seconds.

With the queue checked at every step, the arrivals from section 04 that had been slow became fast. One request first, then the other 31, went from 802 to 1,337 output tokens a second with 1,000-token replies, and from 794 to 1,224 with 1,120-token replies. Every first-group size we tried was fast.

A setting with the same effect

SGLang reads an environment variable, SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION, that caps how many reply tokens its check counts for one request. The default is 4,096. Set to 512, a new request here counts as 601 tokens, all 32 start however they arrive, and SGLang pauses the 3 that the cache cannot finish near the end. That gave 1,330 to 1,348 output tokens a second on SGLang 0.5.20 and 0.5.22, for all three arrivals.

Lowering --schedule-conservativeness to 0.3 did not help, because it lowers only the count for running requests. With one request first, 30 started instead of 29, and the run stayed slow.

On SGLang 0.5.22, 29 of the 32 requests started in every run, however the requests arrived, and checking the queue at every step did not change that. The reason is a deliberate change in 0.5.22, pull request #43564, "Reserve remaining tokens for ignore_eos requests during prefill admission". For a request that ignores the end-of-sequence token, as fixed-length benchmarks do, SGLang now counts a running request at its whole remaining reply instead of about 70% of it. With that count, the requests the cache cannot hold at full length wait for the first group in every run, and SGLang never needs to pause one. The pull request was merged after 0.5.21 was released, so 0.5.22 is the first release with it. The change is still in SGLang's main branch as of Oct 2026, so later releases will likely behave the same way.

We ran 0.5.22 with only that change undone, and it matched 0.5.20 on all 12 cases, within 1%. Under the two benchmark clients we started with, it gave both speeds again. SGLang's client was fast in 4 of 8 runs and vLLM's in 2 of 8, and every run followed the rule from section 04.

The change applies only to requests that ignore the end-of-sequence token. We also fixed the reply length the other common way, with min_new_tokens set to the full length, which keeps the end-of-sequence token from being chosen before then. Every reply still came to exactly 1,000 tokens. Released 0.5.22 then counted running requests at about 70% of their reply again, and gave both speeds, case for case as 0.5.20: 1,323 to 1,335 output tokens a second with all 32 at the same step, and 795 to 798 with one request first. With the queue checked at every step, one request first was fast on 0.5.22 as well, at 1,318 to 1,320.

With SGLang's prefix cache turned off (--disable-radix-cache), both releases count fixed-length requests at their whole reply before starting them. Both held the same 3 requests back, with all 32 at the same step and with one request first, at 798 to 806 output tokens a second.

The line is sharp. With 914-token replies, all 32 requests fit in the cache at full length, and every run gave about 1,459 output tokens a second. With 916-token replies, they need more than the cache's 32,096 tokens, one request waited for the whole first group, and the runs gave 824.

A cache with room for all 32 at full size made every run fast on 0.5.22 too, at 1,453 to 1,474 output tokens a second. So for a fixed-length benchmark on SGLang 0.5.22, size the cache for every request at full length, or check how many requests ran, before comparing it with another engine.

What the change trades

The change was written for longer prompts, where fewer requests fit at full length, and where SGLang 0.5.20 starts more than fit and pauses some near the end. We measured that trade with vLLM's client keeping 32 requests in flight, replies of exactly 1,000 tokens, and prompts of 88, 1,000 and 3,000 tokens.

With 1,000-token prompts, 0.5.22 was 3% faster than 0.5.20, and with 3,000-token prompts 1% faster, without pausing any request. With 88-token prompts, it held 3 requests back in every run, at about 1,203 output tokens a second, against 1,351 and 1,352 for 0.5.20's fast runs. The first 0.5.20 run there was slow for the reason in section 04: one request started on its own, and the other 31 arrived at the next step.

vLLM matched 0.5.20's fast runs with 88-token prompts. With 1,000-token prompts, it was 8% faster than 0.5.22. It ran up to 31 requests at once against 16, and its median first token came after 2.8 seconds against about 24. With 3,000-token prompts, vLLM was 3% slower than 0.5.22.

With 64 in flight, the cost on short prompts grew. With 256 requests of 88-token prompts and 1,000-token replies, SGLang 0.5.22 never ran more than 29 at a time and gave 1,308 to 1,311 output tokens a second, with a median first token after 22 seconds. In the same comparison, SGLang 0.5.20 ran 64, paused 82 to 89 requests a run, and gave 1,516 to 1,525, which is 16% more, with a median first token after 11 seconds. vLLM gave 1,730 to 1,732.

The same holds for real conversations when the reply lengths are fixed, which is the usual way ShareGPT throughput is benchmarked (--ignore-eos). With 800 ShareGPT conversations at 128 in flight, SGLang 0.5.22 ran at most 64 to 81 requests at once and gave 1,475 to 1,568 output tokens a second. SGLang 0.5.20 ran up to 93 to 110 and gave 1,929 to 1,942, and 0.5.22 with only the change undone gave 1,734 to 1,934. Each run drew its own sample of conversations, and the gap is larger than the spread within each release. With replies free to stop, 0.5.22 matched 0.5.20, as the next section shows.

Several reports describe the full mark staying set in other situations, and some of those have been fixed. In issue #35537, for example, a request waited for a running one to finish on an almost empty server. Pull request #43901, which is an open draft, removes the full mark so that SGLang checks the waiting requests at every step. That is the change that removed the slow runs on 0.5.20 above. The full mark works the same way in SGLang's main branch as we read it in Oct 2026.

Pull request #43901 is built on a branch that already counts fixed-length requests at their whole reply, as 0.5.22 does. On this load, we would expect it to behave like 0.5.22 with the queue checked at every step, with 29 requests started. We have not run it. For fixed-length requests that use min_new_tokens instead, 0.5.22 with the queue checked at every step started all 32 in every run, so the change still matters there.