Measure the serving system before enabling CUDA graphs.
An Inference Optimization Benchmark that is Behavior-Aware
Beyond
tokens per second.
Most GPU reports stop at tokens per second and memory use. We wanted a more comprehensive evaluation of inference optimization and the trade-offs that matter in real deployments.
THE QUESTION
How do choices—from quantization and CUDA graphs to speculative decoding, concurrency and prompt design—affect throughput, latency, correctness and answer stability?
WHY RV-380
Instead of using a standard model benchmark, which we might do later, we generated 380 mixed questions - a new RadianVector benchmark. This was done to reflect example client workloads, to approximate the varied traffic of a newly onboarded customer. This set was then used for comparing inference configurations — not ranking general intelligence.
WHAT WE CHECKED
We compared four weight-quantized checkpoint builds—AWQ, GPTQ, AutoRound and Red Hat AI’s llm-compressor build—plus FP16 and FP8 KV caches, MTP and n-gram speculative decoding, concurrency, prompt design and reasoning prompts. We checked aggregate scores and individual answers.
Qwen3.8-27B was tested through 54 controlled inference configurations on the Nvidia RTX 4090 GPU. The model and RV-380 requests stayed fixed while the inference stack changed.
EXPERIMENT MAPMEASURED
FIXEDQwen3.8-27BNvidia RTX 4090 · 24 GiB
54 configurations
CHANGEDInference stack
WEIGHT-QUANTIZED BUILDS
AWQGPTQAutoRoundRed Hat AI
EXECUTION
EagerCUDA graphs
KV CACHE
FP16FP8
SPECULATIVE DECODING
None (control)MTPn-gram
REQUEST + LOAD
ConcurrencyPrompt designReasoning prompts
PERFORMANCESpeed + waittok/s · TTFT · memory
BEHAVIORQuality + drift380 scored questions
same modelsame questionspaired comparison
54configurations examined
380fixed, scored tasks
3.08×CUDA-graph speedup
−18.9largest quality shift, points
3–4natural correctness flips / 380
The serving baseline
First, establish
the vLLM baseline.
The 3.08× speed-up was measured after establishing a working vLLM baseline. It is an improvement within vLLM—not a comparison with a basic Hugging Face Transformers loop. Compared to the basic Hugging Face Transformers loop, the improvement achieved would have been 3.08 times the 14-24x speedup vLLM provides ~ 60x over the simple implementation.
Change the execution mode while keeping the model, GPU and single-user workload fixed.
Why this experiment
Optimization is a
multi-axis decision.
Faster generation is useful only when the model still behaves the way the product requires. We treated every supposedly “free” optimization as a hypothesis to test—not a property to assume.
THE FULL 4K VIEWfour key lenses · one deployment decision
01 · SPEEDHow fast?tok/s · throughput
02 · WAITHow responsive?TTFT · queue time
03 · CAPACITYHow much fits?memory · concurrent users
04 · BEHAVIORWhat changed?correctness · answer drift
A TYPICAL GPU REPORT
- 01How fast?
Peak tokens per second.
- 02Does it fit?
Model and cache memory.
Necessary measurements. Not a deployment decision.
THE RADIANVECTOR STUDY
- 01How fast—and for whom?
Generation rate, aggregate throughput, time to first token and queueing under load.
- 02Does behavior survive?
Correctness, exact output changes and question-level flips under each configuration.
- 03Is the conclusion stable?
Repeated boots, paired tests, multiple answer regimes and a measured noise floor.
MODELQwen3.8-27B W4A16
RUNTIMEvLLM 0.28 · WSL2
HARDWARERTX 4090 · 24 GiB
QUALITY INSTRUMENTRV-380 · 17 scored task types
DECODINGGreedy* · native thinking off
* Greedy means temperature 0: at each step the model takes the single highest-probability next token instead of sampling among the plausible ones. No randomness, so the same question gives the same answer every time — which is what makes a difference between two runs attributable to the configuration rather than to chance. How we read these numbers →
WHAT RV-380 COVERS
Five capability bands, checked by code.
The suite is a regression instrument for inference changes—not a claim to measure all of intelligence.
The behavior check
RV-380,
question by question.
Every configuration is evaluated on the same deterministic set of questions, with each response checked programmatically for correctness. The questions and subject areas are selected to reflect representative client workloads. RadianVector can also generate and simulate evaluation sets tailored to a client’s specific domains, use cases, and areas of interest, then run inference experiments against those workloads. This enables evaluations that more closely reflect real-world production requirements, rather than relying solely on generic benchmarks that may miss important workload-specific nuances.
380questions scored
17scored task types
5capability bands
fixedsame suite for every run
ALL 17 RV-380 TASK TYPES
See exactly what was tested.
WHAT FUTURE VERSIONS SHOULD ADD
A focused regression suite—not a complete measure of model capability.
RV-380 includes 25 simple general-knowledge questions about capitals, chemical symbols and basic facts. That is useful for checking one-word instruction following, but it is not broad knowledge coverage.
Future versions should expand general knowledge, safety, multilingual and multimodal evaluation. Code execution was deliberately left for a separate RadianVector benchmark and report focused on coding models, executable tests and development workflows.
01Broader knowledgeMore subjects, difficulty levels and current factual domains
02Code executionA dedicated coding benchmark and report is planned
03SafetyRefusal quality, boundary handling and robustness
04MultilingualInstruction following and reasoning across languages
05MultimodalImage and mixed-input understanding
Experimental design
How to read
these numbers.
01 · GREEDY DECODING
The model never rolls a die.
Greedy means temperature 0. At each step the model takes the single highest-probability next token, instead of sampling among the several plausible ones. There is no randomness, so the same question produces the same answer every time.
That is not how most products run — a little sampling usually makes output feel less mechanical. It is how a measurement has to run. With sampling on, two runs of one configuration already differ, and there is no way to tell a real effect from the dice. Greedy makes the configuration the only moving part, so a difference between two runs is attributable to the change we made.
It also turns one result on this page into a genuine finding rather than a curiosity. Speculative decoding is supposed to be exactly output-preserving under greedy — the verifier accepts a drafted token only when it matches what the model would have picked anyway. Under greedy that is a testable promise. It failed.
02 · PAIRED COMPARISON
Question by question, not score against score.
The obvious way to compare two configurations is to subtract their scores. We do not do that anywhere on this page. Instead every comparison walks all 380 questions and sorts each one into four buckets: both right, both wrong, only A right, only B right.
The two "both" buckets are then discarded. A question every configuration answers correctly tells you nothing about which is better, and neither does one they all fail. Only the disagreements carry information, and the test is simply whether one side won them far more often than a coin flip would explain.
That is why a line on this page can read paired 3 vs 4: the two arms disagreed on seven questions in total, one winning three and the other four. A coin flip. No difference is claimed, however far apart the raw percentages look.
The reason to bother is that RV-380 contains both easy questions and brutal ones. Comparing overall scores mostly measures which questions you happened to ask. Pairing cancels that out completely, because both arms face the identical question — the same logic as measuring each patient before and after a drug rather than comparing two different groups of people. It makes small real effects visible, and it refuses to dress up large-looking differences that rest on a handful of items.
Where a request’s time actually goes
Both settings above are about correctness. This one is about what the server feels like when more than one person is using it — and it is the most counter-intuitive thing we measured. Each bar is one request’s whole life, split into the four phases the server actually reports: time queued before anything starts, time spent reading your prompt, time spent writing the answer, and the scheduling remainder. The only thing that changes down the rows is how many people are asking at once.
ONE REQUEST’S LIFE UNDER LOAD · FOUR MEASURED PHASES · THINKING OFF
queued — not being served at all
prefill — reading the prompt
decode — writing the answer
other — scheduling remainder
hover any segment for its seconds and share
Load does not make the model slower. It makes you wait longer to start. Decode — the only phase that produces text — takes 4.78 s at one user and 7.72 s at thirty-two, a factor of 1.6. Queue goes from nothing at all to 22.40 s. At eight users the request is still only 3% queued; by thirty-two it is 72%, and the machine is working the whole time on somebody else’s request.
Prefill behaves differently again, and it is the one phase that gets better under load: 0.35 s alone, rising to 1.24 s at six users, then falling back to 0.77 s at thirty-two as the scheduler batches prompts together. It never dominates. Reading the question was never the expensive part.
This is why a throughput number quoted without its latency is close to meaningless, and why every configuration on this page is compared at a matched concurrency rather than at whatever load flattered it.
Config M1_mamba_bf16, CUDA graphs on, 64 scheduler slots, 27,306-token KV pool. Synthetic 1,024-token prompts with a 256-token cap, three rounds per load level, so the phase split is not confounded by questions of different lengths. The phases are the server’s own queue_s, prefill_s and decode_s; “other” is the residual against end-to-end.
One consequence worth stating plainly: pairing only works if both arms saw the same questions in the same rounds. That is why results from different suite hashes are never pooled, and why a comparison drawn across two campaigns is reported as such rather than merged.
Executive results
Eight experiments.
Four decisions.
Filter by the kind of decision each experiment informs. Every card distinguishes the observed result from its operational interpretation.
01USE
CUDA graphs
3.08×single-user generation
16.1 tok/s 49.6 tok/s
Accuracy was unchanged on the stated 380-item basis when re-tested under the harder reasoning regime: 95.18% versus 95.26%, paired 3 vs 4, p=1.0.
02PREFER AWQ
Four 4-bit checkpoint builds
2.4points between builds*
AWQ is the only build never significantly beaten across every tested scoring basis and both answer regimes. The order beneath it is not robust.
*These are four downloadable builds, not four independent algorithms. The Red Hat AI recipe combines AWQModifier and GPTQModifier. On the primary 380-item long-answer basis, the range is 2.4 points.03WORKLOAD-SPECIFIC
MTP-head speculative decoding
−11.9 → −1.2accuracy points
The model’s built-in MTP head proposed future tokens; no separate draft model was used. Depth 3 hurt answer-only output far more than full reasoning and never became output-preserving.
04AVOID
n-gram speculative decoding
−18.2 ptsand 2.6× slower
The n-gram prompt-lookup proposer used no draft model. It was already unsafe at depth 1 and degraded further with depth under long-form answering.
05NO QUALITY EFFECT FOUND
Cache preemption
3–5×more evictions forced
At matched concurrency, the starved cache produced correctness changes indistinguishable from repeat-run noise: p=1.0 at c=8 and p=0.45 at c=16.
06CAP AT 6
Production MTP configuration
+7.4%throughput at 6 users
MK17 preserved full-reasoning quality (90.09% vs 90.44%, p=0.60) but still lost 6.5 points on answer-only tasks.
07WATCH THE WAIT
Concurrency
0.32s → 28.37smean time to first token
Total throughput rises under load, but responsiveness collapses. A peak-throughput number without TTFT describes the wrong product.
08MEASURE THE FLOOR
Repeatability and audit
3–4 / 380natural correctness flips
Identical runs do not produce identical text. We measured that floor, paired each question, generated figures from raw results and attached every conclusion to its source campaign.
Behind the conclusions
What it took to reach
one defensible answer.
The polished result is one page. Behind it is a reproducible experiment system: hypotheses, controlled runs, raw telemetry, stored responses, paired quality checks, failed configurations, repeated measurements and a corrections ledger.
WHAT WE VARIED
More than a speed ladder.
Each technique was tested as a change to a serving system, with performance and behavioral consequences measured together.
THE EVIDENCE ARCHIVE
Results kept at run-level resolution.
The archive retains machine-readable metrics, individual model answers, logs and analysis—not just the winning number.
HOW IT BECAME TRUSTWORTHY
The corrections are part of the work.
- 01
Freeze the model, hardware and 380-item scored instrument.
- 02
Repeat runs and compare every question, not only the mean.
- 03
Separate measured relationships from mechanism hypotheses.
- 04
Generate figures from raw results and audit source citations.
- 05
Record corrections when a stronger test changes the conclusion.
WHAT COUNTS AS EVIDENCE
The number of files is not the evidence. The evidence is the traceable chain from a hypothesis, through a controlled run and stored outputs, to a conclusion another engineer can inspect.
Detailed evidence
The numbers behind
the decisions.
01 · QUANTIZATION
A ranking that moves when the scoring basis moves.
The headline table shows four RV-380 views: two answer regimes, each on the full suite and a stricter 360-item subset. “Leader separable” uses a paired McNemar test between first and second by mean. “Never significantly beaten” asks of every build, not just the leader, whether any other build significantly beats it on that basis — so the reader can see whether surviving is a distinction or merely common. AWQ is the only build that survives all four; AutoRound survives three.
What survives all four views
AWQ is the only build never significantly beaten on any of the four RV-380 views; AutoRound survives three and Red Hat AI one. GPTQ is significantly beaten on every view. Nothing below AWQ has a stable ordering. On the strict long-answer subset AWQ ranks third by mean and is still not beaten — “never beaten” means no other build separates from it, not that it scores highest.
| Asked | Counted | Ranking by mean | Leader separable from 2nd? | Never significantly beaten |
|---|
INTERACTIVE CHECKPOINT EXPLORER
Change what counts.
02 · SPECULATIVE DECODING
Answer length changes the MTP trade-off.
Same model, questions and matched control. Only the requested answer style changes. Run time is a proxy for how much text the regime elicited.
Measured vs inferred
The relationship is measured. Longer answers “absorbing” an early divergence is a working hypothesis; this experiment did not directly isolate that mechanism.
| Answer regime | Seconds / pass | Control | MTP depth 3 | Cost | Answers changed | Paired p |
|---|
03 · PROPOSER COMPARISON
MTP-head and prompt-lookup speculation do not fail together.
Both are forms of speculative decoding, and neither uses a separate draft model. At full reasoning, MTP depth 1 ties control while n-gram depth 1 immediately loses 13.1 points. Since both use the same verification path but diverge sharply, verification alone cannot explain the difference.
Mechanism status
The data localize the difference to the proposer: the model’s MTP head versus n-gram lookup in the prompt. The more specific story—that prompt lookup decays as generation moves away from prompt text—is a well-motivated inference, not a directly instrumented causal result.
04 · CACHE PRESSURE
We made preemption happen—and still found no quality effect.
The positive control matters. A starved cache demonstrably caused far more preemptions, while concurrency was held constant. That separates cache eviction from the batch-shape changes caused by comparing different concurrency levels.
05 · CONCURRENCY
Throughput and responsiveness pull in opposite directions.
Select or focus a point to inspect the measured mean TTFT and the range across four configurations.
The companion study
Thinking on,
thinking off.
Everything above was measured with the model’s own reasoning mode switched off. The results with the reasoning mode switched on are presented in the companion study, to make the results easier to understand and interpret.
WHY WE PINNED IT OFF FIRST
- 01One variable at a time
This report changes the serving stack — quantization, cache format, speculation, concurrency. Letting the model also decide how much to think would have put an uncontrolled variable inside every comparison.
- 02A stable floor to measure from
With thinking off the model answers directly and output length is governed by the question, not by an internal budget. That makes the paired comparisons above clean.
- 03It matches how the study began
Every request sent enable_thinking: false. Holding that fixed keeps the whole first body of work internally comparable.
The cost of that choice: none of the numbers above tell you what the model can do when it is allowed to reason.
WHAT THE SECOND STUDY FOUND
- 01Turning it on is worth ~4.4 points
RV-380 accuracy rises from 94.4% to 98.8% averaged across four checkpoint builds — roughly seventeen questions, four to five times the repeat-run noise floor.
- 02The gain is bought at the lowest setting
Almost all of it arrives at low. Every step above that buys 0.4 points or less while generating substantially more text, and the template’s own default is the most expensive rung of all.
- 03It does not rescue speculative decoding
MTP still costs accuracy with thinking on. The damage tracks how much the model writes, not why it writes it.
Same RV-380 questions, same GPU, same greedy decoding. The only thing that changes is whether the chat template may reason before answering.
STUDY WITH REASONING TURNED ON
The companion study - with Qwen3.8-27B with reasoning enabled - carries the full grid: four checkpoint builds by four reasoning-effort settings ie off, low, medium, and x-high and the paired significance tests behind every claim, and what the setting costs in generated tokens
Thinking on, thinking offDeployment decisions
What we would
actually run.
These recommendations are deliberately narrow: the exact model, runtime, hardware and workload are part of every conclusion.
TURN ON
CUDA graphs
The largest clean speed win: 3.08× single-user generation with no measurable aggregate accuracy cost.
DEFAULT BUILD
AWQ
The only checkpoint never significantly beaten across all tested regimes and scoring bases.
CONDITIONAL
MTP depth 1, capped at six sequences
Consider only for long-form workloads after an application-specific quality gate. Leave it off for short factual answers.
LEAVE OFF
n-gram speculative decoding
Worse quality and slower execution in the tested long-answer regime, including at the shallowest depth.
Experimental method
How we tried not
to fool ourselves.
The project’s corrections became part of the method. A trustworthy null needs a positive control; a small difference needs pairing; every statement needs a route back to the run that produced it.
Freeze the instrument
Fixed seed, deterministic task generation and a suite SHA. Runs with different hashes are not pooled.
Compare item by item
Every arm answers identical questions. McNemar tests the discordant questions rather than comparing rounded means.
Measure natural variation
Repeated identical runs establish the floor: 3–4 correctness flips and roughly 5% text variation per 380 items.
Prove the treatment happened
The preemption null is interpretable because counters showed the starved cache really did preempt 3–5× more.
Separate score from output
“Same accuracy” and “same answer” are different claims. CUDA graphs and MTP can alter text without changing the aggregate score.
Generate and audit claims
Figures are derived from raw results; conclusion blocks name their campaign; independent checks recompute published values.
WHAT CHANGED OUR MINDS
A correction is a result.
01“MTP costs about 13 points.”
Replaced by: MTP’s measured cost depends on answer regime: −11.9, −8.9 and −1.2 points across the length ladder.
02“The shared verifier is at fault.”
Replaced by: n-gram and MTP diverge at the same depth under the same verification path. The difference lies upstream in how proposals are made.
03“AutoRound is the default download.”
Replaced by: AutoRound falls to last under two long-answer bases. AWQ is the only build never significantly beaten across every view.
04“The preemption question is unresolved.”
Replaced by: a crossed cache-size × concurrency test with verified preemption counts found no answer-quality effect above the repeat-run floor.
Boundaries
THE SEARCH SPACE
IS THE PROBLEM.
Good inference research is as specific about its limits as it is about its results.
The configuration space for LLM inference is vast. Model format, quantization, KV-cache precision, batching, concurrency, speculative decoding, kernels, memory policies, runtime settings, prompt characteristics and hardware can interact in ways that make exhaustive testing impractical.
For most teams, exploring this space is not the product they are trying to build. Every experiment consumes engineering time, GPU time, and attention—and the search space continues to evolve as models, runtimes, kernels, and hardware change.
A client should not have to explore this space alone. RadianVector's role is to partner and systematically search, measure, and narrow it—so the client's engineering team can focus on the application while still benefiting from inference configurations tuned to its actual workloads.
This study therefore does not attempt to identify a universally optimal configuration. Its conclusions apply to the hardware, software versions, workloads and configurations tested here. Other combinations may produce different trade-offs.
Measure these configurations carefully, isolate the effects of specific changes where possible, and provide reproducible evidence that can guide the next experiment.
Prompted reasoning, not native thinking
enable_thinking remained false in this main study. Long-form results came from a prompt asking for working and an ANSWER: sentinel. See the thinking-enabled companion study →
The tool-call category is excluded
Its prompt forbids prose while the reasoning regime demands working, and it swings sharply between rounds. It cannot fairly rank these arms.
Some 8k arms saw 12k questions
Fifteen items were outside those configurations’ context window. Shared zeros do not bias paired flips, but they depress headline percentages.
Mechanisms remain hypotheses
The experiment identifies relationships and localizes likely components. It did not directly trace token acceptance or prove why longer outputs reduce MTP’s score cost.
RV-380 is a regression suite
It has limited general-knowledge coverage, not broad coverage. Future versions should expand knowledge, safety, multilingual and multimodal evaluation; code execution is reserved for a separate coding benchmark and report.
The deployment envelope is contained here
One model family, a 24 GiB GPU, one vLLM version and a bounded traffic shape. Different model variants, GPU generations, runtimes such as vLLM, SGLang, and TensorRT-LLM, and distributed multi-GPU deployments each add new dimensions to the configuration space — and multiply the interactions between them
The takeaway
Inference optimization is not “find the highest tokens per second.” It is finding the fastest configuration that preserves the behavior your application depends on—under the load your users will actually create.