Subagent Answer Benchmarks
2 min read
Measured results for the Subagent Answer agent. These pages are benchmarks: what was tested, on which model, and what to expect. The agent composes an answer from eligible library sources, with supporting arguments and as-of dates, and flags the gap when no eligible source determines the answer.
What users get. Claude Opus 5.5 at low reasoning matches the approved answers best on both content and tone: it keeps the content on 99% of the 133 questions the library answers, and its tone scores 0.95 on a 0-to-1 scale, level with Claude Opus 5. The other models keep the content almost as well (93–98%; Mistral Medium 3 62%) but drift further in tone, down to 0.87 for GPT-6.1 Sol. On questions the library does not answer, GPT-6.1 Sol at low reasoning flags the gap on 72%; the other models mostly compose an answer. An answer costs USD 0.01 to USD 1.18.
Answering models compared on the 133 questions with an approved library answer, out of 325 benchmark questions (October 2026). Cost and time are per answer over all 325 questions. Reasoning was set to low for every model that supports it. Mistral Medium 3 has no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model. GPT-4o, the production default, is not included: in this setup it did not return an answer within 15 minutes, while the models below take 13 to 68 s per answer on average. Claude Sonnet 5 is not in the charts: its run stopped at 282 of 325 questions when the Anthropic workspace reached its usage limit, and only complete runs were judged.
Content matches the approved answer on the 133 answerable questions (GPT-5.4 judge), against mean list-price cost per answer over all 325 questions.
Tone match with the approved answer on the 133 answerable questions (style-embedding similarity, mean with a 95% bootstrap interval), against mean list-price cost per answer. A verbatim copy scores 1; two different approved answers score 0.75 on average. A measurement, not a judge; see How it is scored in the report.
Content matches the approved answer (GPT-5.4 judge) against tone match with the approved answer, both on the 133 answerable questions. Top right is closest to the approved answers on both.
Content accuracy against mean answer time over all 325 questions (log scale). GPT-5.1 times are inflated: another benchmark shared its token limit in the QA environment.
Content accuracy on the 133 answerable questions, against the share of the 192 questions without a library answer that got an answer without flagging the gap (phrase rule, not a judge).