Which Claude model should run the seat? Four models on one bench

Fresh chats began with the same 20,000 to 25,000 tokens of start pages and could consult most of the same note store. Answer formats and lengths were fixed. Web access, messaging and consultation with other agents were out of scope. The team sealed the answer key or the grader’s proposed answers by hash before the candidates ran. Reviewers read answers shuffled under neutral labels.

Six chats ran 20 recall questions and two three-question judgment rounds. Eleven new chats ran the three solvable problems. Each setting ran once. This is an internal benchmark on one Windows machine.

Twenty questions each had a knowledge and a judgment part. The sealed key awarded 1, 0.5 or 0 per part. All six runs searched nearly the whole memory store, in 5 to 20 bulk reads. The one half point lost concerned judgment depth. Opus xhigh scored 40/40 at an estimated USD $1.35, while Fable’s three perfect runs cost $3.46 to $5.01. Opus max scored the same as xhigh at $2.99. Higher effort often added reversal conditions, inferences and weak points without changing scores.

One Fable run assumed the wrong time of day. Another corrected a weekday after checking a clock. Two saved answers into the restricted test folder. The question paper itself lived there, so the instruction and setup conflicted. See the field note on unobserved facts.

The same six chats received two sets of three choices about the team’s own affairs. The notes gave reasons on both sides. On the first set, all six agreed on every choice. Five of six suggested Opus high for one conversational role. The second set explicitly invited disagreement. A safety classifier stopped Opus max after it read the questions. It reported the stop. The other five agreed on two choices and split on the third.

The owner read the responses blind and preferred one Fable round for its account of causes. That preference comes from one reader and one round. The private questions and answers remain withheld.

The tasks asked for the cost of waking an idle teammate with a large context, the cause of a broken startup routine, and a plan for a data delivery. All 11 identified that one needed price was absent. All 11 traced the startup issue to multiple commands failing on the same overlong line, independent of the page or byte budget.

Sonnet max tested its fix, caught a second limit on escaped input, and recalled a dependency on an awake machine. Haiku max identified a selector gap and preserved a snapshot. Haiku high spotted a second injector with no line check. Both low-effort runs skipped executing a check on the bug. One medium run read outside the allowed area, and one xhigh run used the wrong weekday. The decoy did weakest on the delivery plan, counting data the system did not store. Sonnet high finished roughly 28 minutes after its start. It earned 9.2 and supplied the fullest cost table.

Grades average one reviewer’s five scores for recall, reasoning, depth, rigour and discipline. In this test, Sonnet max gained 0.4 grade points over high for about five times the estimated cost. Haiku max gained 0.2 points over high for 20 times the estimated cost.

These are the list rates used to reconstruct the October 2026 runs. The tier uses each call’s prompt length, including cache reads and writes. All four rates rise fivefold above the threshold. Haiku at high and below stayed within the lower tier on every call. At xhigh, 5 of 11 calls crossed the threshold. At max, 10 of 14 crossed it. The max run would have priced at $0.15 if every call had used the lower tier. Applying the tier call by call gives $0.60.

One grader grouped 28 of 30 responses by session using layout and citation habits. Model identification was harder. The owner named 4 of 6 models in the first judgment round and scored 16.5 of 30 in the next. Both readers scored below half on model identification for the fresh writers. A distinctive punctuation pattern helped the grader pick out the decoy. Record model and effort directly when they matter.

1. Separate tasks that have an answer key from tasks that need judgment. Choose the trial and scoring rule before running either.
2. Seal the key and blind the reviewer. Repeat settings before routing real work on small grade differences.
3. For a bounded task, test a smaller model at high effort and compare its quality and total cost. Check the longest calls against any price tier.
4. Supply a clock when a result needs a date. Record model, effort and per-call usage in the evidence.

Related notes cover effort and lookup costs, rule retrieval and guard behaviour and what a memory summary keeps.

The four tests ran on October 19, 2026 in Claude Code on one Windows machine, using 17 chats in total. The first three tests reused six chats. The two cohorts had different questions. Within-cohort results come from one run per setting and an Opus 5.5 grader scoring its own model family. Two grades were sensitive to individual slips. The owner’s preference on the open-ended work was subjective.

Calls and token counts come from transcript usage deduplicated by message. Estimated USD costs apply Anthropic’s October 23 list rates, assume one-hour cache writes and use the Haiku tier per call. Subscription quota consumption was outside the measurement. Open-book recall shows how readily these notes could be found when almost all were read. Routing under a budget, longer sessions, acting with tools and completed builds remain untested. The reported runs were not reproduced for publication.