Pick a size. It sets the number of questions and how hard they are. Every size has at least 200 questions, so that the tests can separate the settings.
These counts are representative. We set the exact number for your tasks and sub-tasks. One price, with no other fees.
We change nothing on your server. We send requests and read the replies. If you can change the server, see server controls.
There are four ways to create the question set. Choosing your areas is all you have to do. The other three take optional input from you, for when you want a say. You do not have to provide the questions: if the price works for you, we write them all.
Pick as many as you like. Each tag is an area or a sub-area, and two related ones work well together.
One row per question, for anything you send us. Send as many as you have, and we write the rest. Only question is required. If you leave the answer and how_to_check fields empty, we will fill them in on our end. We send the question set back in the same format.
questions.csv question,answer,how_to_check,area,sub_area What is the total federal tax?,122175.83,"A number, within $2",Taxes,Investor return Which clause caps liability here?,,Must name the clause,Law,Contracts What dose for a 72 kg patient at 5 mg/kg?,360 mg,,Healthcare,Dosing
Questions say what to ask. Traffic says how the requests arrive. We build it from your areas, or copy the shape of your traces.
Independent questions that share little text.
Long prompts that fill the context window.
Each step sends the whole conversation so far.
Many requests sent at once.
You can read every question with its answer, change a group, and sign off. Or let our systems decide: the run starts once the set has at least 200 questions. If you sign, we record exactly which version you approved.
We run every question on your endpoint, on every viable combination of thinking, temperature, system prompt and output format. If we know a combination will do badly, we leave it out. What the research shows. Then we load it.
We raise the number of requests at once, and measure where each request's time goes.
Data from our research: Qwen3.8-27B on vLLM, RTX 4090 GPU family, thinking off. The benchmark.
The same requests can run at different speeds, depending on how they reach the server. So we send them in bursts, and spread out.
Data from our research: SGLang 0.5.20, Qwen3-8B, RTX 4090 GPU family. vLLM 0.32.0 ran both at about 1,341 tokens a second. The deep dive.
An agent sends its whole conversation at every step. If the start of the prompt is still in the KV cache, only the new part is computed. So we run several agents at once, and measure how much comes from the cache and how many tasks finish.
Data from our research: 192 agent tasks, four agents at once, Qwen3.8-27B on vLLM 0.32.0, RTX 4090 GPU family. The bars show the idea, and the totals are measured. The second server had MTP speculative decoding on. The measurements.
We ask the same questions under load, and compare the answers with those at one request at a time.
We grade every answer, then run the answers and timings through our repos, shown here as rv-1 to rv-n, and our statistical tests, including McNemar, Holm and repetition tests. You get a report of which settings to use for which task and sub-task, how many requests at once your endpoint can take, and how to send them.
| Requests at once | Waiting in the queue | Reading the prompt | Writing the reply | Whole request |
|---|---|---|---|---|
| 1 | 0.00 s | 0.35 s | 4.78 s | 5.1 s |
| 2 | 0.00 s | 0.60 s | 5.23 s | 5.9 s |
| 4 | 0.03 s | 1.08 s | 5.55 s | 6.8 s |
| 6 | 0.10 s | 1.24 s | 5.90 s | 7.5 s |
| 8 | 0.23 s | 1.06 s | 6.79 s | 8.3 s |
| 12 | 3.61 s | 0.95 s | 7.34 s | 12.2 s |
| 16 | 7.46 s | 0.89 s | 7.48 s | 16.1 s |
| 24 | 14.93 s | 0.81 s | 7.64 s | 23.6 s |
| 32 | 22.40 s | 0.77 s | 7.72 s | 31.1 s |
Data from our research: Qwen3.8-27B on vLLM, RTX 4090 GPU family, thinking off. On this setup, the queue starts between 8 and 12 requests at once. The benchmark.
| Sub-task | Thinking | Temperature | System prompt | Output |
|---|---|---|---|---|
| Loan terms | Off | 0 | Plain | JSON |
| Investor return | Long | 0 | Tuned with DSPy | Plain text |
| Retirement | Short | 0.3 | Expert | JSON |
| Tool calls | Off | 0 | Tuned with DSPy | JSON |
An illustration of the layout, with no results.
For teams that want more out of AI, on work where a right answer matters far more. We test what more can do: your model asked several times, a second model checking it, and several models voting. We find what gives the most for each task, work out what it costs, and tell you if it is worth it. How RV_Enhanced works.
Tokens a second is not the whole story. Count the work that finishes.
The same two servers, on two kinds of traffic. The second server is faster on short requests, and finishes fewer than half as many agent tasks. Both solved 117 of 192 tasks, so the difference is time, not accuracy.
Data from our research: Qwen3.8-27B on vLLM 0.32.0, RTX 4090 GPU family. The second server had MTP speculative decoding on at depth 1. The measurements.
With no other change, a smaller KV cache cut completed tasks from 752 to 467 an hour. Four agents with prompts that grow at every step do not all fit, so more steps compute their prompt again.
Data from our research: 192 agent tasks, Qwen3.8-27B on vLLM 0.32.0, RTX 4090 GPU family, no speculative decoding. The measurements.
Reach out and send us a note. We’ll take it forward from there.