Agentic Table-Subagent Answer Benchmark

This page lists the answering models we evaluated for the Subagent Answer agent. Where the Library Answers agent returns the approved library text verbatim or refuses, the Subagent Answer agent composes an answer from eligible library sources, with supporting arguments and as-of dates, and flags the gap when no eligible source determines the answer. Its configuration is the Library Answers configuration with two changes: the Subagent Answer prompt and a search result limit of 200 instead of 50. Search scope, temperature and every other setting are the same.

Every model was given the 325 scored cases of the Library Answers Benchmark (133 with an approved answer in the Answer Library, 192 without), question only, with the same providers and routes as on the Library Answers Benchmark. Reasoning was set to low for every model that supports it. Mistral Medium 3 has no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model.

GPT-4o, the production default, is not included: in this setup it did not return an answer within 15 minutes, while the models below take 13 to 68 s per answer on average. Claude Sonnet 5's run stopped at 282 of 325 questions when the Anthropic workspace reached its usage limit, and only complete runs were judged, so it appears in the results table without content or tone figures. There is no production reference on this page, so every model is compared with the one with the highest content accuracy.

Hosting. GPT-5.1 runs on Azure in Switzerland and Mistral Medium 3 on Vertex AI in the EU. Anthropic, OpenAI through LiteLLM (GPT-6.1 Sol) and Together AI (Kimi K3 and DeepSeek V4.1 Flash) process globally. Which providers a client can use depends on its contract.

How it is scored

The verbatim pass rules of the Library Answers Benchmark do not fit this agent, because it rewrites library text by design. Instead, two LLM judges compare each answer with the approved library answer.

• Answerable question (133): content matches the approved answer. GPT-5.4 compares the answer with the approved library answer and scores two criteria from 0 to 10, using our G-Eval-style scoring rubric. Coverage: every material point, question part, condition and exception of the approved answer is kept. Contradiction: no claim conflicts with the approved answer, so no reversed meaning and no changed value, date or condition. Full marks mean the criterion holds completely. An answer counts as keeping the content when the judge rates both criteria as good or excellent. The judge sees only the direct answer; the supporting arguments and citations are removed first.

• Second judge. Claude Sonnet 5.5 reads the same pairs and labels specific differences: a missing required claim, a contradiction, a changed value, a reversed meaning (such as Yes for No), or a changed condition or exception. The two judges agree on 96% of the 1,029 answers both scored (Cohen's kappa 0.69), and both put Claude Opus 5.5 first and Mistral Medium 3 last. Claude Sonnet 5.5 is also one of the answering models; GPT-5.4 is not. Automatic measures that use no LLM agree with the judges; see How we checked the content number.

• Additions are not errors. The agent composes from every eligible source by design, so content beyond the approved answer does not fail an answer. The judges check that added content does not contradict the approved answer, not that it is true. How often each model adds content is reported for reference.

• Tone match (133). How closely the answer's writing style matches its approved answer: the similarity of two style embeddings (LUAR, with content words masked so that the facts do not count), from 0 to 1, averaged over the answerable questions. A verbatim copy scores 1; two different approved answers score 0.75 on average. This is a measurement, not a judge. It rises when an answer reuses the approved wording and falls as the answer's length moves away from the approved answer's (rank correlation −0.62), so a lower score often means a longer, rewritten answer rather than a wrong tone.

• Question without a library answer (192): gap flagged. There is no approved answer to compare with, so the judges do not score these. The answer passes when it declines or flags the gap in its first sentence ("cannot confirm", "requires confirmation", "no verified figure", or the Library Answers refusal). This is a phrase rule, not a judge; it was checked by reading answers from GPT-6.1 Sol and GPT-5.1.

• Firm Yes or No. Answering a question without a library answer is not automatically wrong, because the prompt asks the agent to compose an answer from eligible sources when no template exists. An answer that opens with a firm Yes or No is the sharper risk, and is reported separately.

Citations are not used to score accuracy. The benchmark records one expected library file per question, and composed answers often cite another entry with the same content. On the answerable questions, the citation check and the GPT-5.4 judge disagree on 130 of 1,050 answers: 99 keep the approved content without citing the expected file, and 31 cite the expected file with content that does not match.

Summary

• Claude Opus 5.5 at low reasoning keeps the content of the approved answers best, and its tone is the closest, level with Claude Opus 5. It keeps the content on 99% of the answerable questions. The other models except Mistral Medium 3 keep the content on 93–98%; between them the differences are a few answers each. Tone separates the models more clearly: from 0.95 for the Claude Opus models to 0.87 for GPT-6.1 Sol, whose answers are a median 1.2 times as long as the approved answer, with sentences 3 words longer on average. Tone is separate from content: Mistral Medium 3 has the lowest content accuracy but a closer tone (0.88) than GPT-6.1 Sol. On questions without a library answer, GPT-6.1 Sol at low reasoning flags the gap on 72%; the other judged models flag it on 0–11% and compose an answer on the rest, and up to 24% open with a firm Yes or No.

• Claude Opus 5.5, low: content matches the approved answer on 99% (132/133) [96–100%]. The second judge finds a material difference on 1% (1/132). On questions without a library answer it flags the gap on 6% and opens with a firm Yes or No on 6%. Mean answer time 29 s, USD 0.74 per answer, USD 241 per benchmark run. The highest content accuracy under both judges and the closest tone, level with Claude Opus 5, but it rarely flags the gap. About 1.5 times the cost of GPT-6.1 Sol per answer.

• DeepSeek V4.1 Flash, default: content matches the approved answer on 98% (128/130) [95–100%], no significant difference from Claude Opus 5.5 (p = 1). The second judge finds a material difference on 3% (4/131). On questions without a library answer it flags the gap on 5% and opens with a firm Yes or No on 24%. Mean answer time 38 s, USD 0.18 per answer, USD 60 per benchmark run. Close to the top on content accuracy, but rarely flags the gap and gives the most firm Yes or No answers where the library has none. Runs at its default reasoning, not at low. Not recommended.

• Claude Sonnet 5.5, low: content matches the approved answer on 95% (126/132) [90–98%], no significant difference from Claude Opus 5.5 (p = 0.062). The second judge finds a material difference on 2% (2/128). On questions without a library answer it flags the gap on 11% and opens with a firm Yes or No on 5%. Mean answer time 19 s, USD 0.32 per answer, USD 104 per benchmark run. Flags the gap on 11% of those questions, more than the other judged models except GPT-6.1 Sol.

• Claude Opus 5, low: content matches the approved answer on 95% (126/133) [90–97%], significantly lower than Claude Opus 5.5 (p = 0.031; not significant once corrected for the seven comparisons). The second judge finds a material difference on 4% (5/130). On questions without a library answer it flags the gap on 5% and opens with a firm Yes or No on 17%. Mean answer time 25 s, USD 1.18 per answer, USD 384 per benchmark run. The most expensive model, with no accuracy advantage over Claude Opus 5.5. Not recommended.

• Kimi K3, low: content matches the approved answer on 95% (125/132) [89–97%], significantly lower than Claude Opus 5.5 (p = 0.031; not significant once corrected for the seven comparisons). The second judge finds a material difference on 3% (4/131). On questions without a library answer it flags the gap on 1% and opens with a firm Yes or No on 23%. Mean answer time 30 s, USD 0.39 per answer, USD 127 per benchmark run. Almost never flags the gap. Not recommended.

• GPT-6.1 Sol, low: content matches the approved answer on 95% (125/132) [89–97%], significantly lower than Claude Opus 5.5 (p = 0.031; not significant once corrected for the seven comparisons). The second judge finds a material difference on 5% (6/131). On questions without a library answer it flags the gap on 72% and opens with a firm Yes or No on 1%. Mean answer time 19 s, USD 0.49 per answer, USD 159 per benchmark run. The only model that flags the gap on most questions without a library answer, with content accuracy in the same range as the others. Its tone is the furthest from the approved answers (0.87): its answers are a median 1.2 times as long as the approved answer, and it adds content beyond the approved answer on 89% of answerable questions, which the judges do not verify. The candidate for Subagent Answer; less expensive per answer than either Claude Opus model.

• GPT-5.1, low: content matches the approved answer on 93% (121/130) [87–96%], significantly lower than Claude Opus 5.5 (p = 0.008). The second judge finds a material difference on 5% (6/128). On questions without a library answer it flags the gap on 1% and opens with a firm Yes or No on 16%. Mean answer time 68 s, USD 0.17 per answer, USD 56 per benchmark run. The option when inference must run in Switzerland, but it almost never flags the gap. The slowest model; its answer times are inflated by a shared token limit.

• Mistral Medium 3, no reasoning: content matches the approved answer on 62% (80/128) [54–70%], significantly lower than Claude Opus 5.5 (p < 0.001). The second judge finds a material difference on 39% (51/132). On questions without a library answer it flags the gap on 0% and opens with a firm Yes or No on 21%. Mean answer time 13 s, USD 0.01 per answer, USD 3 per benchmark run. Often leaves out material content of the approved answer. Not suitable.

• Claude Sonnet 5, low: not judged for content or tone; its run stopped at 282 of 325 questions (see the notes). On the 149 questions without a library answer it reached, it flags the gap on 40% and opens with a firm Yes or No on 3%. Mean answer time 19 s, USD 0.34 per answer, USD 110 per benchmark run. Flags the gap more often than any model except GPT-6.1 Sol, so it is worth judging once the remaining questions are run.

How we checked the content number

The content figures come from an LLM judge, so we checked them three ways: against the second LLM judge, against 13 automatic measures that use no LLM (entailment and consistency models, embedding similarity, word-level similarity and word overlap), and by human review of the responses.

• Second judge. Claude Sonnet 5.5, from another provider, also puts Claude Opus 5.5 first: 99% with no material difference, against 99% under GPT-5.4. The two judges disagree on 2 to 10 answers per model, so the few-answer differences between the models other than Mistral Medium 3 are no larger than the judges' own disagreement.

• Same order at the top and the bottom. 10 of the 13 automatic measures put Claude Opus 5.5 first, and 9 put Mistral Medium 3 last.

• Same answers flagged. The automatic measures score the answers the judge says lose content well below the ones it says keep it: a randomly chosen kept answer scores higher than a randomly chosen flagged one 80–90% of the time on the measures that put Claude Opus 5.5 first, where 50% would be chance. The entailment model, for example, gives a median of 0.99 to kept answers and below 0.01 to flagged ones.

• Where they do not agree. The three measures that do not put Claude Opus 5.5 first, the contradiction scores of the NLI and SummaC models and AlignScore's alignment score, are also the ones that separate the judge's flagged answers least well: a kept answer scores higher 56–69% of the time.

• Human review. People verified the responses, and the automatic measures were used to pick out outlier answers for that review.

• What the automatic measures do not show. They reward staying close to the approved wording, so they favour answers that reuse it. On their own they confirm the ranking and which answers lose content, not the exact rate; that is what the human review is for.

Results

95% Wilson intervals in brackets. Content rows count only answers the judge returned a score for; the missing ones are listed in their own row. Answer time: mean per question with a 95% bootstrap interval. Cost per benchmark run: mean list price per answer times 325. Claude Sonnet 5: 133 answerable and 149 of 192 questions without a library answer.

OpenAI models

Other providers

Per 1,000 questions

In the benchmark mix, 41% of questions have an approved library answer (133 of 325) and 59% do not (192 of 325). Scaled to 1,000 questions in this mix, rounded so that the first four rows of each column add up to 1,000. The answerable rows use the GPT-5.4 judge; the other rows use the phrase rule.

Statistical tests

Paired exact McNemar tests on content accuracy against Claude Opus 5.5, low, the highest, on the answerable questions both models have a judge score for. After a Holm correction for the seven comparisons, only GPT-5.1 and Mistral Medium 3 remain significantly lower than Claude Opus 5.5.

Configurations

Times are measured per answer. Costs are list prices for the agent's calls, from the usage the platform records or reads back afterwards, not invoices. The longer prompt and the search result limit of 200 roughly double the input per answer compared with Library Answers, and the composed answers are longer, so costs per answer are not comparable between the two pages. For some answers the run did not capture usage data, so it was read back from the chat records in the QA environment after the run and priced with the same price sheet: Claude Opus 5.5 24, DeepSeek V4.1 Flash 11, Kimi K3 19, GPT-5.1 2, Mistral Medium 3 25. Answers with no usage data at all are scored and left out of the cost mean: GPT-5.1 2. Answers that failed for platform reasons (service restarts, provider rate limits) were retried and are not counted as wrong answers.

Content accuracy comes from LLM judges. People reviewed the responses, with the automatic measures used to pick out outliers for that review. Each answer was judged once, with no repeat and no planted test answers. GPT-5.4 returned no score on 14 of the 1,064 answers (it marked the criterion as not applicable or its reply could not be read) and Claude Sonnet 5.5 on 21; these are left out of the rates. The comparison is with the approved answer only, so content the agent adds from other eligible sources is not checked.

Mistral Medium 3's low content accuracy is not caused by how the direct answer is separated from the supporting arguments: its direct answers are about as long as the others' (median 33 words; Claude Opus 5.5: 35), and the second judge most often labels a missing required claim (44 of its 132 judged answers).

Claude Sonnet 5 answered all 133 answerable questions and 149 of the 192 without a library answer before the Anthropic workspace reached its usage limit. Only complete runs were judged, so it is left out of the content and tone rows, the per-1,000 table and the tests; its other rows cover the questions it answered. Its answers can be judged once the remaining questions are run.