Intelligence, Measured.

Build a better AI system for your workload. Better intelligence. Faster responses. Lower costs. Higher energy efficiency.

RadianVector measures and optimizes entire AI systems - not just models. We find the smartest AI setup for each task.

See the number of knobs we have identified.

Powered by proprietary optimization technology and original research, RadianVector systematically explores various possibilities to identify and validate the right setup for your workloads - with research-grade rigor.

We test and optimize offline, without requiring sensitive production data or live traces, and without slowing down your production pipeline.

Share only what you choose - not your entire production traffic or live traces.

Frontier Models Open Models
your workload your workload · model APIs your model · hosted GPUs behavior latency cost quality throughput GPU memory Intelligence, Measured AI MODELS AND AGENTS Frontier Models Open Models FRONTIER MODELS Claude · GPT · Gemini · Grok support → use model A claims → use model B regulatory → AI drafts, human checks OPEN MODELS vLLM · SGLang · TensorRT-LLM vllm · fp8 weights fp8 KV cache max_num_seqs 32 illustrative · hover either side

Keep your production traffic private

Teams changing their workflows

Deploying AI is not
deploying software.

Box CEO Aaron Levie put it plainly: changing workflows for AI takes far more work than anyone expects. Workflows need re-engineering for agents, evaluations need to be built and maintained, and the whole system needs updating every time a new model arrives.

If your organization is in the middle of that change, talk to us. Measuring what works, and checking it still works after every model update, is what RadianVector tools are built for.

LinkedIn post by Aaron Levie, CEO of Box: There's a huge opportunity right now in being the deployment layer for AI into the economy. The amount of work it takes to change out workflows in enterprises tends to be far greater than anyone realizes or would prefer.
Aaron Levie, CEO of Box, on LinkedIn ↗
Post on X by Brian Armstrong, CEO of Coinbase: We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at Coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud prevention in this case). I think we'll see more of this pattern inside companies. The frontier models are amazing generalists. But increasingly you'll see specialist models out perform on more narrow tasks, either working on teams, or by themselves. The attached chart shows the fine-tuned model improving fraud detection over each comparison model, including an Opus 4.5 decision agent.
Brian Armstrong, CEO of Coinbase, on X ↗
Teams running their own models

Open models can win.
When they are tuned well.

Coinbase fine-tuned Qwen3.5-9B, an open-weight model, on its own data. For fraud prevention it outperformed frontier models at much lower cost.

Running your own model is where RadianVector tools go furthest. A hosted API gives you the settings its provider offers; your own model exposes many model-level settings as well. We measure which combination delivers the answers, speed and cost you need, on your hardware and your workload.

  • vLLM · SGLang · TensorRT-LLM
  • Quantization
  • KV-cache precision
  • Speculative decoding
  • Batching & concurrency
  • KV-cache offload & preemption
  • Temperature · top-k · top-p
  • Reasoning budget

See every setting we tune →   Our Qwen benchmark →

Original RadianVector results

Frontier Model analysis.

We ran our own questions on models from Anthropic, OpenAI and xAI, through each vendor’s own command-line tool, and evaluated the answers. We have ten findings from October 2026, and here are a few of them.

Findings 02 and 07

Max reasoning is not always the right answer.

Opus 5.5 Medium1/17,584 tokens
VS
Opus 5.5 High1/18,932 tokens
VS
Opus 5.5 Max1/124,875 tokens

Right answers on a 50-step calculation

Opus 5.5 High2/233,411 and 36,387 tokens
VS
Opus 5.5 Max0/2139,699 and 514,985 tokens

Right answers on two 300-step calculations

GPT‑5.5 Low7/14
VS
GPT‑5.5 High12/14

Right answers on 14 level-10 questions

More effort helps on some tasks and wastes tokens on others.

On a 50-step calculation, max, high and medium effort all got it right, but max used 24,875 tokens, against 8,932 for high and 7,584 for medium. On two 300-step calculations, high effort solved both, while max kept reasoning past the 128,000 tokens one reply can hold and gave no answer.

More effort can also pay off. Raising GPT-5.5 from low to high effort took it from 7 to 12 right out of 14 level-10 questions.

What it means for you. Use higher effort where it gets more answers right, not by default. On everyday work it can cost several times more for the same answer, so test effort settings on your own questions. Read finding 2 → Read finding 7 →

Tested: Claude Opus 5.5 · max effort: October 8–15 · high and medium: October 15 · GPT-5.5: Codex (0.154.0), October 4, 2026

Finding 01

Do not retry without looking: a question that overflows the output limit will likely overflow again

514,985 output tokens on one question, and no answer ($10.33 at API rates)

On one question at max effort, Opus reached its 128,000-token output limit before it answered. Our test runner retried the question three times. Each try started from the beginning and ran out of room again, and after four replies there was still no answer.

What it means for you. Do not retry a failed request without looking at why it failed. Retrying after the model ran out of output tokens is especially costly, because the retry will likely fail for the same reason and use as many tokens again. Split the work into smaller steps, or try a lower effort setting first (finding 2). Read finding 1 →

Tested: Claude Opus 5.5, max effort · October 15, 2026

Finding 04

Tool use did better than GPT‑6 Astra’s top reasoning setting alone

Astra, tool useRight494 tokens
VS
Astra, no toolWrong47,931 tokens

One 700-step calculation, at xhigh, its highest setting

On 28 hard questions, GPT-6 Astra got all 28 right when it could use its built-in tool, and 26 right without it. With the tool, it also used less than half the tokens.

What it means for you. For exact work, tool use can do more than extra reasoning. On these 28 questions, it gave more right answers while using fewer tokens. Read finding 4 →

Tested: GPT-6 Astra, xhigh effort · Codex (0.154.0) · October 5, 2026

Read all ten findings → See the results tables →

Original RadianVector results

Open Model analysis.

Measured answer quality, request timings and serving behavior, including a tested vLLM fix.

Answer quality

Open models, with thinking on.

Results at levels 5 and 6 of our difficulty ladder, on the RTX 4090 GPU family.

Swipe to see all columns.

Thinking on · levels 5 and 6 · 28 questions per level
ModelLevel 5Level 6
gpt-oss-20b, high effort25/2822/28
Qwen3.8-27B, 4-bit21/2821/28
Qwen3-8B, 4-bit18/2810/28
GLM-4.7-Flash, 4-bit14/286/28

How the difficulty ladder works →

Response time

What happens as requests increase.

The tool report separates time spent waiting from time spent generating.

Swipe to see all columns.

27B model · vLLM 0.28 · no speculative decoding · all times in seconds
Requests
open		RunningWait for first token
median		Time to completion
mean		Time generating
median
110.054.84.8
220.135.35.2
440.115.55.4
880.115.65.5
16103.18.85.8
321011.816.65.8

At 8 open requests, mean completion time was 5.6 s. At 32, it was 16.6 s, while median generation time changed only from 5.5 to 5.8 s. Only 10 of the 32 requests were running; the others were waiting in line.

See all timing measurements →
Benchmark controls

Same model, three inference engines.

RadianVector’s inferlint measured what each server actually ran under the same benchmark load.

Swipe to see all columns.

Qwen3-8B BF16 · RTX 4090 GPU family · 16,384-token context · at most 32 running · same load from vllm bench serve
MeasurementvLLM 0.30SGLang 0.5.20TensorRT-LLM 1.3.0rc29
KV pool32,336 tokens
47,888 on a second start		32,096 tokens37,760 tokens
Most running at once322932
Output tokens per second1,2747991,389

The benchmark path matters. With vllm bench serve sending the same load to all three engines, SGLang ran up to 29 requests at once and produced 799 output tokens/s. With SGLang’s own client using its native /generate endpoint, the same server ran all 32 requests and produced 1,337–1,342 output tokens/s.

See the full engine comparison and run details →
Answer correctness

A vLLM bug, before and after the fix.

MTP speculative decoding repeated tokens under load. We traced the cause to the model runner, patched it and reran the tests.

Swipe to see all columns.

Qwen3.8-27B W4A16 · RTX 4090 GPU family · WSL2 · V1 model runner · 385 replies per run · 8 concurrent requests · async scheduling and prefix caching on
ConfigurationDifferent replies
before → after		Repeated-token replies
before → after
vLLM 0.28 · MTP-160 → 1129 → 0
vLLM 0.28 · MTP-377 → 626 → 0
vLLM 0.30 · MTP-139 → 813 → 0

Repeated-token replies fell to zero in these patched runs. “Different replies” counts token-for-token differences from the no-speculation run; it is not a correctness score.

Read the fix and full results →
Two paths

Offline AI model evaluation. Hosted and open models.

Offline evaluation compares AI configurations outside your live production pipeline. RadianVector tools support hosted model APIs and self-hosted open models, so you can measure answer quality, response time and cost before deploying a change.

Frontier Models

Compare hosted models and API settings.

Which model and prompt combination meets your requirements? Where does reasoning effort help? What changes when the output format or supplied context changes?

For engineering teams improving hosted-model applications, retrieval workflows and agents.

  • questions
  • reasoning effort
  • prompt sensitivity
  • tool use
  • severe errors
  • latency
  • cost
  • human review
Explore hosted-model evaluation
Open Models

Measure answers and the serving stack.

Where do requests queue? Does quantization change the answers? How do cache settings and concurrency affect latency, throughput and GPU use?

For teams serving open-weight models with vLLM, SGLang, TensorRT-LLM or llama.cpp on their own infrastructure.

  • model
  • runtime
  • quantization
  • KV cache
  • concurrency
  • decoding
  • throughput
  • latency
  • GPU utilization
  • cost
Explore open-model evaluation
What RadianVector measures

See beneath the headline score.

RadianVector compares performance and behavior on the same workload. See which tasks each configuration completes, where it fails and how much time and compute it uses.

Explore the engineering behind the measurements →
Workload-specific evaluation

Which tasks does the system get right?

Compare task results against your requirements. See successful completions, important errors and the tasks each configuration struggles with.

Controlled runs & effort analysis

Where does extra computation help?

Compare models, prompts and supported reasoning settings on the same questions. Measure where a change improves answers and what it costs in time and money.

inferlint · Open source

What happened inside the serving stack?

Use inferlint to inspect request queues, memory use and concurrency, and understand what your inference server actually did.

Inspect inferlint and an example report →
Regression & change analysis

Did the change improve the system?

Test before and after a change. See what improved, what regressed and how response time and cost shifted.

Illustrative study design

One document workflow. 1,000 representative questions.

For a workflow that extracts facts from research documents, a study could screen 10 model and reasoning-level settings with 5 prompt variations, then test output formats and sampling settings on promising candidates.

Baseline

Measure the current setup on representative questions.

Screen

Narrow the search using the quality and cost requirements.

Validate

Test the finalists on held-out cases and repeat runs to check stability.

Deliver

Provide configurations, measured trade-offs and failure examples.

The search is staged; the final validation cases stay separate from the cases used to select candidates. This is an illustrative design; published results are linked above. When a model, runtime or GPU changes, rerun the same tests to check the setup still holds.

Explore offline evaluation for hosted models →
Start a conversation

Want to understand your AI better?
Let’s measure it.

You don’t need to know what to ask. Tell us how your team works and how you use AI today, and we will work out with you what is worth measuring.

[email protected]