Workflow · Frontier models

You choose a size and an area. We do the rest.

STEP 1

Place an order.

Pick a size. It sets the number of questions, how hard they are, and the time.

  • S100 questions, standard complexity, 1 week.
  • M300 questions, harder, 2 weeks.
  • L1,000 questions, hardest, 4 weeks.
STEP 2

Build the questions.

There are four ways to create the question set. Choosing your areas is all you have to do. The other three are optional, for when you want a say.

Start here

Choose your areas

Pick as many as you like. Each tag is an area or a sub-area, and two related ones work well together.

Questions
We write them.
Answers
We decide what a good answer is, and how to check it.
Optional

Let us look at your product

Questions
We write them from your site and docs.
Answers
We decide. You can check them at sign-off.
Optional

Describe your tasks

Questions
You flag the tasks. We write the questions.
Answers
We decide, or you tell us how to check.
Optional

Send your questions or traces

Questions
You flag the questions.
Answers
You flag the answers, or tell us how to check.

The format: a CSV file

One row per question, for anything you send us. Only question is required. Leave answer and how_to_check empty, and we decide. We send the question set back in the same format.

Download the template
questions.csv
question,answer,how_to_check,area,sub_area
What is the total federal tax?,122175.83,"A number, within $1",Taxes,Investor return
Which clause caps liability here?,,Must name the clause,Law,Contracts
What dose for a 72 kg patient at 5 mg/kg?,360 mg,,Healthcare,Dosing
STEP 3 · OPTIONAL

Sign off on the questions.

You can read every question with its answer, change a group, or skip this entirely. If you sign, we record exactly which version you approved.

STEP 4

Run them.

We run every question on every combination of model, effort, system prompt and output format, and repeat the runs. That includes system prompts tuned with DSPy, and JSON against plain text.

STEP 5

Get the report.

We grade every answer, then run the answers through our repos, shown here as rv-1 to rv-n, and our statistical tests, including McNemar, Holm and repetition tests. You get correctness, speed and token spend, and a report of which model to use for which task and sub-task. You keep the report and the question set.

Which model for which sub-task

Sub-taskClaude SonnetGPT-6 AstraGrok 4.7Use
ESPP, qualifying sale2 of 21,556 tokens2 of 2220 tokens2 of 28,568 tokensGPT-6 Astra
Qualified dividends2 of 22,366 tokens2 of 2209 tokens2 of 211,898 tokensGPT-6 Astra
Section 1256 contracts4 of 42,465 tokens4 of 4315 tokens4 of 47,401 tokensGPT-6 Astra
Wash sales4 of 41,681 tokens4 of 4259 tokens4 of 45,087 tokensGPT-6 Astra
ESPP, disposition4 of 42,888 tokens4 of 4712 tokens4 of 417,002 tokensGPT-6 Astra
ISO and AMT4 of 46,220 tokens4 of 41,645 tokens4 of 484,300 tokensGPT-6 Astra
Loss carryovers4 of 41,606 tokens4 of 4283 tokens4 of 436,000 tokensGPT-6 Astra
Whole investor return11 of 1231,781 tokens9 of 126,048 tokens12 of 12195,062 tokensGrok 4.7
ISO crossover2 of 25,607 tokens1 of 21,445 tokens2 of 251,451 tokensClaude Sonnet
Lot selection2 of 26,858 tokens2 of 2735 tokens2 of 228,138 tokensGPT-6 Astra

Real results from our run on 31 Oct 2026: 40 stock-market tax answers, each model at high effort, 2 to 12 per row. Ranked by right answers, then by fewest tokens. Rows this small show the form of the report. A project has more questions per row, and the tests say which gaps hold.

Two real examples

What the answer looks like.

Example 1

Which frontier model for hard finance questions?

20 stock-market tax questions at four levels of difficulty, each asked with the cited law in the prompt and without it, so 40 in all. Three frontier models, each at high effort.

Right answers, of 40

Grok 4.7
40
Claude Sonnet
39
GPT-6 Astra
36

Reply tokens, all 40 questions

Grok 4.7
444,907
Claude Sonnet
63,028
GPT-6 Astra
11,871

What we would recommend. Start with the model that uses the fewest tokens, after checking each vendor's price per token, and run a larger question set before settling. The widest gap in right answers is Grok over GPT-6 Astra, where 4 questions went one way and none the other, with an exact McNemar p of 0.125. So 40 questions do not separate these models. The tokens differ by about 37 times, and the median question took 8.6 seconds on GPT-6 Astra against 124.7 on Grok.

Run on 31 Oct 2026 through each vendor's own command-line tool, with X search on for Grok. Tokens are as each tool reported them.

Example 2

Which effort level?

The same model, at different effort levels, on the same long calculations.

Claude Opus 5.5, two 300-step calculations: right answers

High effort
2 of 2
Max effort
0 of 2

Output tokens, both calculations

High effort
69,798
Max effort
654,684

What we would recommend. Use high effort for long calculations like these. At max, Claude Opus 5.5 reasoned past the 128,000 tokens one reply can hold and gave no answer. Do not assume the same on every task: on one 50-step calculation, medium, high and max were all right, with 7,584, 8,932 and 24,875 tokens, and GPT-5.5 went from 7 to 12 of 14 level-10 questions right from low to high effort.

Observed in Oct 2026, and published on our frontier models page.

Watch them answer

Real questions, real answers.

We picked these to show each case. They are saved answers, played back evenly across the time each model took.

Recorded 31 Oct 2026