Native reasoning, measured
Every quality number in the main study was measured with the model's own reasoning mode switched off. Here we turn it on across four checkpoint builds and up to four effort settings, then re-run the full RV-380 instrument.
The same RV-380 questions, the same GPU, the same greedy decoding. The only thing that changes is whether the chat template is allowed to reason before it answers. It is worth +4.4 points of accuracy for about 1.5× the generated tokens — and almost all of that is bought by the lowest setting.
The headline
Averaged over four checkpoints, turning native thinking on moves RV-380 from the low-nineties to the high-nineties. Against a measured noise floor of 3–4 question flips per run, this is roughly seventeen questions — four to five times the noise.
RV-380 ACCURACY BY REASONING EFFORT · 380-QUESTION BASIS
The full grid
Accuracy on the 380-question basis, with the mean number of generated tokens each setting costs. AutoRound comes from the 2026-09-08 run; the other three were measured on 2026-09-09.
FIRST — WHAT "OFF", "LOW", "MEDIUM" AND "XHIGH" MEAN
These are not four different models, and nothing about the model's weights changes between them. They are four positions on a single dial in the chat template called reasoning_effort, which controls how much the model works through a problem privately before it writes its answer.
The private reasoning is generated text, so it costs tokens and time even though the user never sees it. That is the entire cost of turning this dial up.
AND WHAT THE TOKEN COLUMNS ARE COUNTING
Every "tokens/answer" figure on this page is an average across individual answers, not a total. The scale behind it:
| Setting | answers graded | total tokens | average per answer | median per answer | 90th percentile |
|---|---|---|---|---|---|
| off | 3,040 | 392,510 | 129 | 43 | 283 |
| low | 3,040 | 575,673 | 189 | 134 | 321 |
| medium | 3,040 | 738,652 | 243 | 175 | 481 |
| xhigh | 1,520 | 599,791 | 395 | 306 | 761 |
The average overstates the typical answer, and badly at "off". The median answer with thinking off is 43 tokens — the mean is 129 because a small number of answers run away to thousands. One question type is responsible for most of it: date_reason is 20 of the 380 questions, 5.3% of the set, and it produces 39% of every token generated at "off". It is also the one question that hits the 4,096-token output cap. So "129 tokens" is a real average over real answers, but most answers are far shorter than it, and the tail is concentrated in one place.
xhigh has half as many answers because only two of the four checkpoints were measured at that setting.
| Checkpoint | off | tok/ans | low | tok/ans | medium | tok/ans | xhigh | tok/ans | off → best |
|---|---|---|---|---|---|---|---|---|---|
| AutoRound | 94.47% | 166 | 98.60% | 192 | 99.21% | 256 | 99.21% | 429 | +4.74 |
| AWQ | 95.13% | 91 | 98.95% | 188 | 98.95% | 234 | 99.21% | 362 | +4.08 |
| GPTQ | 94.21% | 138 | 98.68% | 207 | 98.95% | 248 | — | — | +4.74 |
| RedHat | 93.95% | 121 | 99.04% | 171 | 99.12% | 234 | — | — | +5.17 |
| Mean | 94.44% | 129 | 98.82% | 190 | 99.06% | 243 | 99.21% | 396 | +4.62 |
These four are downloadable builds, not four independent algorithms. Three labels name a quantization method; “Red Hat AI” names a publisher, and that build’s own recipe runs AWQModifier followed by GPTQModifier — a composition of two of the other arms, from the same toolchain and version as the AWQ build. Read the rows as four checkpoints you can download, not as a comparison of four separate techniques.
Source: results/thinkgrid_20260908-2016/ (AutoRound, 3 rounds at every setting) + results/thinkgrid2_20260909-0024/ (AWQ: 2 rounds at off/xhigh and 3 at low/medium; GPTQ and RedHat: 2 at off and 3 at low/medium, with xhigh not run). 380-question basis. Scores use the corrected numbers scorer throughout.
THE COST SIDE
+4.38 points for 47% more generated tokens. The only trade on this page that is obviously worth taking.
+0.24 points for a further 28% more tokens. Below or near the noise floor.
+0.26 points on AWQ for 55% more tokens; nothing at all on AutoRound.
All token figures are averages per answer. Tokens are the cost: at a fixed generation rate they are the user's wait.
NOT JUST BETTER THAN "OFF"
94.47% · 166 tokens. The regime every earlier RadianVector quality number was measured in.
92.89% · 56 tokens. Asking the model in the prompt to show its working.
98.60% · 192 tokens. The template's own reasoning channel.
All three on AutoRound, three rounds each, same 380 questions. Native reasoning and prompted reasoning are two different mechanisms, and they do not perform alike.
Experimental design
The tables that follow are full of paired tests and p-values. Two design decisions are doing the work behind them, and neither is visible in the numbers themselves.
01 · GREEDY DECODING
Greedy means temperature 0. At each step the model takes the single highest-probability next token, instead of sampling among the several plausible ones. There is no randomness, so the same question produces the same answer every time.
That matters more here than on the main study, because on this page the thing being changed is how much the model reasons before answering — a change that acts on the model's own output, not on the server. If sampling were on, every effort setting would produce different text on every run, and a 4.38-point gap would be impossible to separate from luck. Greedy makes reasoning effort the only moving part.
It is worth being clear about what greedy does not remove. Even at temperature 0 this stack is not perfectly deterministic: two identical runs of one configuration still differ on 3–4 answers in 380. That residual is the noise floor every claim on this page is measured against, and it is why no result rests on a single round.
02 · PAIRED COMPARISON
The obvious way to compare off against low is to subtract 94.44% from 98.82%. We do not do that anywhere on this page. Instead every comparison walks all 380 questions and sorts each into four buckets: both right, both wrong, only the first setting right, only the second right.
The two "both" buckets are then discarded. A question both settings answer correctly tells you nothing about which is better, and neither does one they both fail. Only the disagreements carry information, and the test asks whether one side won them far more often than a coin flip would explain.
The reason to bother is that RV-380 mixes easy questions with brutal ones, so comparing overall scores mostly measures which questions you happened to ask. Pairing cancels that completely — both settings face the identical question, the same logic as measuring each patient before and after a drug rather than comparing two different groups of people.
It cuts both ways, and on this page you can watch it do so. off versus low separates at p=2.7e−25 because the disagreements are lopsided nearly every time. But with thinking on, the four checkpoints disagree on only 0–8 questions out of 380 — so the same test finds nothing, not because they are proven equal but because it has almost no evidence left to weigh. That distinction is the subject of the closing section →
One consequence worth stating plainly: pairing only works if both arms answered the same questions in the same rounds. That is why every effort setting on this page was run against one server boot per checkpoint, why suite hashes are never pooled, and why the speculative-decoding section reports its own basis separately rather than merging it with the rest.
Is it real?
Every comparison below is paired: the same question, the same round, one setting against another. Nothing here is an average of two separate scores.
FIRST — WHERE THE NUMBER 760 COMES FROM
A model does not always give the same answer to the same question. Ask it twice and it can be right the first time and wrong the second. So measuring a setting once cannot tell a real improvement from a lucky run — which is why every setting here was measured more than once, on a separately restarted server each time. We call each full pass through the question set a round.
Some settings were actually run three times rather than two. The two tables below use two rounds for every row, so that every row is out of the same 760 and no row silently gets more evidence than its neighbours. Nothing is lost by doing so: every verdict in those tables is identical whether the third round is included or not. One later section — the speculative-decoding comparison — runs on a different basis for a stated reason, and shows its own arithmetic where it does.
THEN — HOW TO READ THE TABLES
Take one question in one round and ask it under two different settings. There are four possible outcomes: both right, both wrong, only the first right, only the second right. The first two tell you nothing about which setting is better — so they are set aside, and the whole test rests on the last two.
The p column asks a single question: if the two settings were genuinely equally good, how likely is a split this lopsided by luck alone? Small means the result is hard to explain away as chance. Anything above 0.05 we report as no difference.
WORKED EXAMPLE — THE REDHAT ROW, IN WORDS
380 questions, asked twice at each setting, is 760 comparisons. Thinking-off and thinking-low gave the same verdict on 721 of them — both right, or both wrong. They differed on 39. On all 39, the thinking-on answer was the correct one and the thinking-off answer was wrong. Not one went the other way.
A 39–0 split, if the two settings were really equally good, is exactly as likely as calling 39 coin flips in a row. That is the 3.6e−12 in the table — about one chance in 275 billion.
This is also why a small "differed on" number is a warning rather than a reassurance. Two settings that differ on 3 questions cannot be separated by any test — not because they are proven equal, but because there is almost nothing to weigh.
| Checkpoint | Compared | out of | only the harder setting right | only the easier setting right | they differed this many times | p by chance? | Verdict |
|---|---|---|---|---|---|---|---|
| AWQ | low vs off | 760 | 31 | 2 | 33 | 1.3e−07 | LOW BETTER |
| GPTQ | low vs off | 760 | 38 | 4 | 42 | 5.7e−08 | LOW BETTER |
| RedHat | low vs off | 760 | 39 | 0 | 39 | 3.6e−12 | LOW BETTER |
| AWQ | medium vs low | 760 | 0 | 0 | 0 | 1.00 | NO DIFFERENCE |
| GPTQ | medium vs low | 760 | 2 | 0 | 2 | 0.50 | NO DIFFERENCE |
| RedHat | medium vs low | 760 | 1 | 0 | 1 | 1.00 | NO DIFFERENCE |
| AWQ | xhigh vs medium | 760 | 2 | 0 | 2 | 0.50 | NO DIFFERENCE |
| Setting | Compared (A vs B) | out of | only A right | only B right | they differed this many times | p by chance? | Verdict |
|---|---|---|---|---|---|---|---|
| off | AWQ vs GPTQ | 760 | 23 | 16 | 39 | 0.34 | NO DIFFERENCE |
| off | AWQ vs RedHat | 760 | 20 | 11 | 31 | 0.15 | NO DIFFERENCE |
| off | GPTQ vs RedHat | 760 | 18 | 16 | 34 | 0.86 | NO DIFFERENCE |
| low | AWQ vs GPTQ | 760 | 2 | 0 | 2 | 0.50 | NO DIFFERENCE |
| low | AWQ vs RedHat | 760 | 1 | 2 | 3 | 1.00 | NO DIFFERENCE |
| low | GPTQ vs RedHat | 760 | 1 | 4 | 5 | 0.38 | NO DIFFERENCE |
| medium | AWQ vs GPTQ | 760 | 0 | 0 | 0 | 1.00 | IDENTICAL |
| medium | AWQ vs RedHat | 760 | 0 | 2 | 2 | 0.50 | NO DIFFERENCE |
| medium | GPTQ vs RedHat | 760 | 0 | 2 | 2 | 0.50 | NO DIFFERENCE |
READ THIS BEFORE CONCLUDING "ALL CHECKPOINTS ARE EQUAL"
No pair separates at any setting — but the reason differs by row, and that matters. With thinking off there are 31–39 disagreements per pair and the wins split roughly evenly: a genuine dead heat with plenty of evidence. With thinking on there are 0–8 disagreements: the checkpoints converge to near-identical behavior, so the test has almost nothing left to work with. "No difference" at medium is a statement about a ceiling, not a strong finding.
This also does not overturn the earlier conviction of GPTQ at p=0.004 and p=1.8e−08. Those were measured on harder instruments where the model scored in the seventies, leaving room to discriminate. Here every arm sits between 94% and 99%. A negative result is only as strong as the instrument's remaining headroom — and on this page that headroom is small by construction.
One naming caveat: these are four downloadable checkpoint builds, not four independent quantization algorithms. “RedHat” names Red Hat AI’s llm-compressor build; its recipe combines AWQModifier and GPTQModifier.
The positive control is on the same page: the identical method, the identical questions and the identical rounds separate off from low at p=2.7e−25. The instrument is not blind — it simply finds no separable difference among the three jointly run checkpoint builds in this paired table.
The other half of the trade
Everything above this point measures whether thinking makes the model right. It does. But nobody deploys an accuracy number — they deploy a server that has to answer people. So we ran the whole effort dial again on a second instrument, measuring wall-clock wait and how many users still fit.
Six checkpoints, four effort settings each, nine concurrency levels from one user to thirty-two — 216 measured rungs in one overnight campaign. Unlike the throughput numbers in the main study, this one does not pin the output length. That matters more than it sounds, and it is the whole reason a separate instrument was needed.
WHY THE MAIN STUDY'S BENCHMARK COULD NOT ANSWER THIS
Our standard throughput harness sends a fixed max_tokens and sets ignore_eos, so the model emits exactly as many tokens as it is told to and never stops early. That is deliberate, and it is the right way to measure raw decode speed: it removes the model's choices from the measurement.
It also makes that harness structurally blind to reasoning effort — because the only thing effort changes is how many tokens the model chooses to emit. Pin that, and thinking becomes invisible. Point the old instrument at this question and it reports "no difference," confidently and wrongly.
This is the finding everything else follows from, and it is unusually clean. Across all sixteen checkpoint-and-effort combinations at one user, the decode rate sits between 53.2 and 56.1 tokens per second — a spread of 5%, with no trend across the effort dial. The model does not think "harder" per token. It emits a longer answer.
So every cost on this page is a token-count cost, and it is predictable: mean tokens per answer go from 117 at off to 172 at low, 235 at medium and 417 at xhigh. A 3.6× longer answer at the same tokens per second is a 3.6× longer wait.
Those four numbers are close to, but not the same as, the 129 / 190 / 243 / 396 quoted in the accuracy sections above, and the difference is not an error. The accuracy campaign asks all 380 RV-380 questions; this one cycles a 48-question sample drawn evenly across all 17 kinds, because a latency ladder has to send the same prompts at every concurrency level to stay comparable. Different question mix, slightly different mean answer length. The shape is what carries over: answers grow roughly threefold from off to xhigh on either basis — 3.1× on the accuracy suite, 3.6× here.
MEDIAN WAIT FOR A COMPLETE ANSWER · ONE USER · BY REASONING EFFORT
The four checkpoints start apart and end together. AWQ is the fastest with thinking off (1.10 s) and GPTQ the slowest (2.55 s), a 2.3× spread; by xhigh they sit within 1.9 s of each other and the ordering has partly reversed. Once the answer is long enough, how the weights were quantised stops mattering to the clock — the token count dominates.
Putting accuracy and cost in one table is the only honest way to read the effort dial. Throughput here is quoted at twelve concurrent users, which is where all four effort settings reach their peak — comparing each setting at its own best rung would flatter whichever one peaks earliest.
| Effort | RV-380 accuracy | Tokens per answer | Wait, 1 user | Answers/min at 12 users | What the step bought |
|---|---|---|---|---|---|
| off | 94.44% | 117 | 1.86 s | 75.0 | baseline |
| low | 98.82% | 172 | 3.47 s | 61.1 | +4.38 pts for −18.6% throughput |
| medium | 99.06% | 235 | 4.50 s | 53.0 | +0.24 pts for −13.2% |
| xhigh | 99.21% | 417 | 6.93 s | 35.4 | +0.15 pts for −33.1% |
Turning thinking on costs about a fifth of your throughput and buys 4.38 accuracy points. Going the rest of the way to the template's default costs a further 42% of what is left, and buys 0.39 points. The cost side of the dial is roughly linear; the benefit side flattened two settings earlier.
COMPLETED ANSWERS PER MINUTE · BY OFFERED CONCURRENCY · MEAN OF FOUR CHECKPOINTS
Two things to read here. The curves flatten after twelve users on every setting, because the server stops accepting more work — see the ceiling below. And the dip every curve takes at sixteen users is real and reproduces on all four checkpoints; we do not yet have an explanation for it, so it is drawn rather than smoothed.
The tables above give the totals. This is the same data opened up: each bar is one request’s whole life, split into the part spent waiting for a turn and the part spent generating text. Every load level shows both settings, so the pairs can be read straight down.
ONE REQUEST’S LIFE UNDER LOAD · THINKING OFF vs XHIGH · MEAN OF FOUR QUANTISED CHECKPOINTS
Two readings, and they point opposite ways. At one user, effort is the whole cost: xhigh takes 8.12 s against off’s 2.49 s, and 96% of that is genuinely generating — the user is waiting for reasoning, not for a queue. Under load, the queue swallows the difference: by thirty-two users both settings spend about three-quarters of the request waiting, off at 74% and xhigh at 75%. What thinking changes is the scale — 20.6 s becomes 47.8 s — because every request holds its slot for longer and everyone else’s wait grows with it. Generating time itself grows only 1.5× across the whole ladder at xhigh, while waiting grows 117×.
We expected the capacity ceiling to fall as answers got longer, because a longer answer holds its share of cache for longer. It does not. The number of requests the server will actually run at once is a property of the checkpoint and its memory layout, and it is identical at every effort setting — we offered thirty-two concurrent users and watched what the scheduler actually admitted.
| Checkpoint | Ceiling at off | Ceiling at xhigh | Preemptions across the ladder | Reading |
|---|---|---|---|---|
| AWQ | 10 | 10 | 591 | unchanged by effort |
| AutoRound | 10 | 10 | 674 | unchanged by effort |
| GPTQ | 10 | 10 | 613 | unchanged by effort |
| RedHat | 10 | 10 | 616 | unchanged by effort |
| 8k context + fp16 SSM cache | 16 | 16 | 7 | reached every level we offered |
| MTP speculative depth 3 | 4 | 4 | 699 | capped at 4 regardless |
A preemption is the server evicting a half-finished request to make room, then recomputing it later. It is invisible in vLLM's logs and only appears in its metrics endpoint, which is why we assert the counter rather than reasoning about headroom.
THE CONFIGURATION THAT CHANGES THE ANSWER
The fifth row is not a quantisation choice — it is the same model served with a 16k context trimmed to 8k and the Mamba state cache in fp16. That one change lifts the ceiling from 10 concurrent to 16, and takes preemptions from roughly six hundred across the ladder to seven. At sixteen users with thinking off it answers in 0.42 s where AWQ takes 5.03 s.
It is on this page because it reframes the cost table above: the throughput that reasoning effort spends is throughput a serving-configuration change can partly give back. The two dials are independent, and the memory-layout dial is the cheaper of the two.
The main study records that multi-token prediction costs accuracy in proportion to how long the answer is. Thinking makes answers three to four times longer, so the two interact — and the interaction is worse than either alone. At sixteen users, MTP at depth 3 goes from 96.8 answers per minute with thinking off to 20.2 at xhigh, a 79% collapse, with 195 preemptions in that single rung and a 95th-percentile wait of 57 seconds. Every other checkpoint degrades by about half. If you are running speculative decoding, treat high reasoning effort as incompatible with it until this is understood.
ONE MEASUREMENT THAT IS NOT FINISHED — AND A CORRECTION
There is a wait this campaign did not originally capture. Because no reasoning parser is configured on this server, the model's thinking arrives inline in the response, wrapped in a <think> block. A first token therefore arrives almost immediately at every effort setting — but if your interface hides reasoning, the user sees nothing until that block closes.
Our first pass reported those two moments as identical, which would have meant thinking costs nothing before the answer appears. That was a measurement bug, not a finding: the chat template opens the <think> tag in the prompt itself, so the model only ever emits the closing tag, and a detector waiting for the opening one never fired. Corrected and re-measured on AWQ: the first token still arrives in 0.30 s at every setting, but the answer starts at 3.03 s at low and 3.69 s at medium, and under load at sixteen users the gap widens to 5.3 s and 6.4 s.
Three of the four effort settings have been re-measured on one checkpoint so far. We are publishing the figure with that limitation stated rather than holding the page, and the remaining ladders will be added when they finish. Nothing else on this page depends on it.
results/thinkperf_20260910-0141/ · 216 rungs, six checkpoints × four efforts × nine concurrency levels, one server boot per checkpoint so all four efforts share a KV pool. Corrected time-to-answer from results/thinkperf_20260910-1112/.
The practical answer
The template's default effort is xhigh. On this evidence that default is the wasteful end of a curve that has already flattened two settings earlier. On this suite — and that qualification matters →
Going from off to low buys 4.38 points for 61 extra tokens. Going from low all the way to xhigh buys a further 0.39 points for 206 extra tokens — five times the token cost for a ninth of the gain, at or below the run-to-run noise floor.
THE SAME CONCLUSION, REACHED A SECOND WAY
That paragraph is an argument about tokens. The cost measurements reach the same place from an independent direction: going from off to low costs 18.6% of your throughput and buys 4.38 points, while going from low to the template's default costs a further 42% and buys 0.39. Two instruments, two different failure modes, one answer — stop at low.
ONE HONEST WRINKLE
The earlier AutoRound run found medium and xhigh to be exactly identical — zero disagreements across 1,140 question-rounds. That does not quite reproduce. On AWQ there are 2 disagreements across 760 question-rounds, and both are the same single question, which xhigh gets right in both rounds. So "the curve saturates" holds; "medium and xhigh are identical" does not. The gap is one question in 380.
The limit of the result
Speculative decoding (MTP) costs this model accuracy, and the damage scales with how much text it generates. Native thinking generates more text — so the open question was whether it would land at the safe end of that curve or make things worse.
THE PAIRED TEST · 330 QUESTIONS × 3 ROUNDS = 990 COMPARISONS
Control wins 14, MTP wins 2, of 16 disagreements across three matched rounds. p = 0.0042. On two rounds it is 12 against 2, p = 0.013 — same verdict either way.
Run the same config twice: the control changes its mind on 0 and 1 questions. MTP changes its mind on 10 and 9.
Speculative decoding is supposed to be output-preserving under greedy decoding. It is not.
−1.21 points is almost exactly where the earlier length ladder put speculation at this output length (−1.2), against −11.9 for short answers and −8.9 when capped. So the damage tracks how much the model writes, not why it writes it. Native thinking does not fix MTP; it just puts you at the forgiving end of the same curve.
What this does not say
Four things a careful reader should know before reusing these numbers.
01 · THE SPECULATION ARMS USE A SMALLER BASIS
The two MTP-comparison configurations run with an 8,192-token context. With 4,096 tokens reserved for output, 50 long-context questions do not fit and the server refuses them outright with an HTTP error. A refused request is not a wrong answer, so those questions are scored on neither arm and that comparison runs on 330 questions, not 380. The refusals are identical on both arms, so the paired test and the −1.21 points are unaffected — but the absolute percentages in that section are not comparable to the rest of this page.
The main study page counts these differently. There, a refused request is counted as a wrong answer and the basis stays at 380, which drags those arms' headline percentages down by about 13 points without changing any paired comparison. Both conventions are defensible and neither moves a verdict; they are simply not interchangeable, so do not compare an absolute percentage across the two pages. The number of refusals also differs by suite — 15 there, 50 here — because this page reserves a larger output budget, which pushes more long-context questions past the limit.
02 · THE 20 EXCLUDED QUESTIONS ARE NOT HIDING ANYTHING
RV-380 excludes one 20-question task type whose prompt is ill-posed. On this instrument every checkpoint answers all 20 correctly at every effort setting — 760 of 760 question-rounds. They are perfectly non-discriminating: including them would raise every arm by 0.04–0.30 points and change no comparison on this page. They are excluded because they carry no information, not to flatter a result.
03 · ONE QUESTION TRUNCATES, AND ONLY WITH THINKING OFF
A single date-reasoning question hits the output cap on the thinking-off arm and on no other. The effective basis for those runs is therefore 379 rather than 380 — worth 0.26 points, below the noise floor, but it means the denominator is not perfectly constant across the page.
04 · THE ACCURACY AND COST FIGURES COME FROM DIFFERENT RUNS
Every accuracy number on this page comes from a quality run, where the token column is generated tokens rather than measured time. The wait, throughput and capacity figures in what it costs come from a separate campaign on a separate instrument, with a different question mix and a different output budget. They are two measurements of the same effort dial, not two columns of one measurement — which is why the token counts differ slightly between them, and why an accuracy percentage and a latency figure on this page should never be combined into a single derived number without checking that both rest on the same basis.
Boundaries
Good inference research is as specific about its limits as it is about its results.
The knee is a property of the workload, not of the model. Stopping at low is the right answer for RV-380. It is not a general law about reasoning effort, and that distinction matters more than any other sentence on this page.
By low, every checkpoint is already answering 98.82% of RV-380 correctly. That leaves about four and a half questions in 380 still to be won. No setting can buy more than the headroom that remains — so the curve had to flatten at low, whether or not medium and xhigh are genuinely reasoning harder.
What the saturation chart measures is the ceiling of this instrument, not the limit of the model's reasoning. The significance section says the same thing from the other side: with thinking on, the four checkpoints disagree on 0–8 questions out of 380. The test does not run out of significance because the settings are equivalent. It runs out of questions able to tell them apart.
On a harder or more varied workload — multi-step reasoning, long chains of dependent inference, genuinely ambiguous questions, anything where this model sits at sixty or seventy percent rather than ninety-nine — there is real headroom left, and medium or xhigh may well repay their throughput cost.
This project has already seen the mechanism from the other direction. Suite v2 could barely convict the GPTQ checkpoint (p=0.004). Suite v3, deliberately concentrated in discriminating questions, convicted the same checkpoint at p=1.8e−08. Same model, same method, same weights — a harder instrument, and more than five orders of magnitude more evidence. Difficulty is what creates headroom, and headroom is what lets a setting prove itself.
Two different things on this page generalise to different degrees.
The cost curve travels well. What each step of the dial costs — roughly 47% more tokens from off to low, 78% more from medium to xhigh, and the throughput that follows from them — is a property of the model and the server, not of the questions. Expect similar costs on your workload.
The benefit curve does not travel. Where it flattens depends entirely on how hard your questions are. So the transferable finding is not "low is where the universe is." It is: measure your own knee, because the knee moves with difficulty — and now you know what each step costs while you look for it.
It is built to catch a configuration change that breaks something, and it is good at that. It is not a general intelligence benchmark, and its ceiling under thinking is the direct consequence. Treat 98.82% as "this suite is close to exhausted", not as "the model is nearly perfect".
The dial exposes four named values. We cannot say where between off and low the gain actually arrives, only that almost all of it has arrived by low.
Short questions, a bounded output budget, a queue drained by a fixed number of workers. A workload of long documents or streaming chat would hold cache differently and move the ceilings.
One model family, one 24 GiB GPU, one vLLM version. Quantization, KV precision, batching, speculative decoding, kernels and memory policy all interact, and they change with every release. A result is a measurement at a date, not a constant.
The configuration space for LLM inference is vast, and for most teams exploring it is not the product they are trying to build — every experiment consumes engineering time, GPU time and attention, while the space itself keeps evolving as models, runtimes, kernels and hardware change. RadianVector's role is to search, measure and narrow it systematically, so a client's engineering team can stay on the application and still run on configurations tuned to its actual workload. This study does not claim a universally optimal setting; its conclusions apply to the hardware, software versions, workloads and configurations tested here.
In one line
The four words doing the work there are on a suite like this one. Low is where RV-380 stops paying, because RV-380 has almost nothing left to win by low. On harder questions the knee sits further right. Why that is, and what still transfers →
44 v7 quality runs archived · 4 checkpoint builds · up to 4 effort settings · RV-380, greedy decoding · Qwen3.8-27B W4A16 on one RTX 4090, vLLM 0.28 under WSL2.
Every setting was run at least twice on separately restarted servers; some were run a third time. Checkpoint and effort comparisons use two rounds — 760 question-round comparisons per row. The MTP section additionally reports its matched third round (990 comparisons) and gives the two-round result; both produce the same verdict.
Sources: results/thinkgrid_20260908-2016/ and results/thinkgrid2_20260909-0024/. Suite: scripts/tasks_v7.py, sha256 7c48ce4f.