Agentic Table-Composed Answers from Library Sources Benchmark
22 min read
This page lists the answering models we evaluated for the Composed answers from library sources agent. Where the Verbatim library answers agent returns the approved library text verbatim or refuses, this agent composes an answer from eligible library sources, with supporting arguments and as-of dates, and is designed to flag the gap when no eligible source determines the answer. The cases and scoring are described under How the benchmark works below.
The models were tested in two runs with the same questions, library, prompt, judges and scoring. Run 2 (Gemini 4 Argon, GPT-5.6 Terra, GPT-5.6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Luna, GPT-4o, GPT-6 Sol, Claude Haiku 5.5 and GPT-6.1 Terra) retrieved fewer library sources per search than run 1 (Claude Opus 5.5, DeepSeek V4.1 Flash, Claude Sonnet 5.5, Kimi K3, Claude Opus 5, GPT-5.1, GPT-6.1 Sol, Claude Sonnet 5 and Mistral Medium 3). That lowers the input per answer, so cost and answer time are compared only within a run.
Key findings
Claude Opus 5.5 at low reasoning and Gemini 4 Argon at low reasoning keep the approved content best (content score 9.9 of 10 each; content kept on 99% and 98%), and their tone is the closest, level with Claude Opus 5 at low reasoning. Gemini 4 Argon is much slower: a mean 178 s per answer, against 12 to 50 s for the other models in its run. On 3 of the 192 questions the library cannot answer, Gemini 4 Argon kept searching and returned no answer within 30 minutes. DeepSeek V4.1 Flash at its default reasoning, Claude Sonnet 5.5 at low reasoning, Kimi K3 at low reasoning and Claude Opus 5 score 9.6–9.8, no significant difference from Claude Opus 5.5 after correction. The other 12 models score significantly lower (8.4–9.5): 37–76% of their answers get full marks on both criteria, against 95% for both Claude Opus 5.5 and Gemini 4 Argon, and the second judge finds a material difference on 5–25% of their answers. Mistral Medium 3 (no reasoning) scores 7.2 and often leaves out material content. Every model except Claude Opus 5 and Gemini 4 Argon is significantly further from the approved answers in tone, down to 0.85 for GPT-6.1 Terra, whose answers are a median 1.2 times as long as the approved answer.
Results
Content score and tone match: mean with a 95% bootstrap interval. Content kept: 95% Wilson interval. Content rows count only answers the judge returned a score for; the missing ones are listed in their own row. Answer time: mean per question with a 95% bootstrap interval. Cost per benchmark run: mean list price per answer times 325. Models are ranked by content score, highest first.
Model | Reasoning effort | Provider | Content score, 0 to 10 (GPT-5.4 judge, 133 answerable), mean [95% interval] | Content kept: at least 7 of 10 on both criteria (GPT-5.4 judge) [95% interval] | Full marks on both criteria (GPT-5.4 judge) | Material difference found by the second judge (Claude Sonnet 5.5) | Adds content not in the approved answer (second judge) | Answers with no judge score (GPT-5.4 / Claude Sonnet 5.5) | Tone match with the approved answer, 0 to 1, mean [95% interval] | Answer time, mean [95% interval] | Answer time, median / 90th percentile | Cost per answer (mean) | Cost per benchmark run | Run (cost and time compare only within a run) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Claude Opus 5.5 | low | Anthropic | 9.9 [9.8–10.0] | 99% (132/133) [96–100%] | 95% (127/133) | 1% (1/132) | 10% (13/132) | 0 / 1 | 0.95 [0.93–0.96] | 29 s [28–30 s] | 28 s / 38 s | USD 0.74 | USD 241 | Run 1 |
Gemini 4 Argon | low | 9.9 [9.8–10.0] | 98% (130/133) [94–99%] | 95% (126/133) | 0% (0/131) | 13% (17/131) | 0 / 2 | 0.95 [0.94–0.97] | 178 s [158–199 s] | 107 s / 408 s | USD 0.51 | USD 166 | Run 2 | |
DeepSeek V4.1 Flash | default | DeepSeek | 9.8 [9.7–9.9] | 98% (128/130) [95–100%] | 86% (112/130) | 3% (4/131) | 29% (38/131) | 3 / 2 | 0.90 [0.88–0.93] | 38 s [35–41 s] | 32 s / 74 s | USD 0.18 | USD 60 | Run 1 |
Claude Sonnet 5.5 | low | Anthropic | 9.7 [9.5–9.9] | 95% (126/132) [90–98%] | 88% (116/132) | 2% (2/128) | 31% (40/128) | 1 / 5 | 0.90 [0.87–0.92] | 19 s [19–20 s] | 19 s / 25 s | USD 0.32 | USD 104 | Run 1 |
Kimi K3 | low | Moonshot AI | 9.7 [9.5–9.9] | 95% (125/132) [89–97%] | 92% (121/132) | 3% (4/131) | 23% (30/131) | 1 / 2 | 0.91 [0.89–0.93] | 30 s [29–31 s] | 28 s / 42 s | USD 0.39 | USD 127 | Run 1 |
Claude Opus 5 | low | Anthropic | 9.6 [9.3–9.8] | 95% (126/133) [90–97%] | 89% (118/133) | 4% (5/130) | 15% (19/130) | 0 / 3 | 0.95 [0.93–0.96] | 25 s [25–26 s] | 24 s / 34 s | USD 1.18 | USD 384 | Run 1 |
GPT-5.1 | low | OpenAI | 9.5 [9.2–9.7] | 93% (121/130) [87–96%] | 76% (99/130) | 5% (6/128) | 35% (45/128) | 3 / 5 | 0.90 [0.88–0.92] | 68 s [64–72 s] | 60 s / 115 s | USD 0.17 | USD 56 | Run 1 |
GPT-5.6 Terra | low | OpenAI | 9.3 [8.9–9.6] | 92% (119/130) [85–95%] | 75% (98/130) | 12% (16/129) | 32% (41/129) | 3 / 4 | 0.91 [0.89–0.93] | 16 s [16–17 s] | 16 s / 23 s | USD 0.11 | USD 36 | Run 2 |
GPT-6.1 Sol | low | OpenAI | 9.2 [9.0–9.5] | 95% (125/132) [89–97%] | 46% (61/132) | 5% (6/131) | 89% (116/131) | 1 / 2 | 0.87 [0.84–0.90] | 19 s [19–20 s] | 19 s / 27 s | USD 0.49 | USD 159 | Run 1 |
GPT-5.6 Sol | low | OpenAI | 9.2 [8.9–9.5] | 91% (120/132) [85–95%] | 67% (88/132) | 9% (11/129) | 62% (80/129) | 1 / 4 | 0.89 [0.87–0.92] | 19 s [19–20 s] | 19 s / 26 s | USD 0.31 | USD 100 | Run 2 |
Claude Sonnet 5 | low | Anthropic | 9.2 [8.9–9.5] | 89% (116/130) [83–93%] | 71% (92/130) | 7% (9/126) | 44% (56/126) | 3 / 7 | 0.88 [0.86–0.90] | 21 s [20–22 s] | 20 s / 30 s | USD 0.34 | USD 112 | Run 1 |
GPT-6 Astra | low | OpenAI | 9.1 [8.8–9.4] | 92% (123/133) [87–96%] | 45% (60/133) | 7% (9/131) | 88% (115/131) | 0 / 2 | 0.88 [0.85–0.90] | 20 s [19–21 s] | 20 s / 29 s | USD 0.44 | USD 143 | Run 2 |
GPT-6 Luna | low | OpenAI | 8.9 [8.4–9.3] | 87% (111/128) [80–92%] | 64% (82/128) | 20% (26/130) | 31% (40/130) | 5 / 3 | 0.89 [0.87–0.91] | 17 s [13–22 s] | 11 s / 15 s | USD 0.01 | USD 2 | Run 2 |
GPT-5.6 Luna | low | OpenAI | 8.7 [8.2–9.1] | 84% (107/128) [76–89%] | 55% (71/128) | 13% (17/130) | 48% (62/130) | 5 / 3 | 0.88 [0.86–0.90] | 12 s [12–13 s] | 12 s / 16 s | USD 0.01 | USD 4 | Run 2 |
GPT-4o (production default) | not applicable | OpenAI | 8.5 [8.0–9.0] | 84% (107/128) [76–89%] | 58% (74/128) | 22% (28/128) | 41% (53/128) | 5 / 5 | 0.86 [0.83–0.89] | 50 s [45–55 s] | 27 s / 121 s | USD 0.07 | USD 24 | Run 2 |
GPT-6 Sol | low | OpenAI | 8.5 [8.0–8.9] | 86% (113/131) [79–91%] | 37% (49/131) | 25% (33/133) | 84% (112/133) | 2 / 0 | 0.86 [0.83–0.88] | 22 s [21–23 s] | 22 s / 32 s | USD 0.16 | USD 51 | Run 2 |
Claude Haiku 5.5 | low | Anthropic | 8.4 [8.0–8.9] | 79% (102/129) [71–85%] | 51% (66/129) | 22% (28/126) | 60% (75/126) | 4 / 7 | 0.87 [0.84–0.89] | 27 s [26–27 s] | 25 s / 35 s | USD 0.02 | USD 8 | Run 2 |
GPT-6.1 Terra | low | OpenAI | 8.4 [7.8–8.8] | 81% (108/133) [74–87%] | 47% (62/133) | 18% (23/129) | 85% (110/129) | 0 / 4 | 0.85 [0.82–0.88] | 12 s [12–12 s] | 12 s / 15 s | USD 0.05 | USD 15 | Run 2 |
Mistral Medium 3 | not applicable | Mistral | 7.2 [6.5–7.8] | 63% (80/128) [54–70%] | 55% (70/128) | 39% (51/132) | 24% (32/132) | 5 / 1 | 0.88 [0.86–0.90] | 13 s [13–14 s] | 11 s / 24 s | USD 0.01 | USD 3 | Run 1 |
Model notes
Claude Opus 5.5, low: content score 9.9 of 10 [9.8–10.0]. The second judge finds a material difference on 1% (1/132). Mean answer time 29 s, USD 0.74 per answer, USD 241 per benchmark run. The highest content score, level with Gemini 4 Argon, and, with Gemini 4 Argon and Claude Opus 5, the closest tone. About 2.3 times the cost of Claude Sonnet 5.5 per answer.
Gemini 4 Argon, low: content score 9.9 of 10 [9.8–10.0], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 1). The second judge finds a material difference on 0% (0/131). Mean answer time 178 s, USD 0.51 per answer, USD 166 per benchmark run. Level with Claude Opus 5.5 on content and tone, and the second judge finds no material difference in any of its 131 judged answers. The slowest and the most expensive model in run 2. On 3 of the 192 questions the library cannot answer, Gemini 4 Argon kept searching and returned no answer within 30 minutes.
DeepSeek V4.1 Flash, default: content score 9.8 of 10 [9.7–9.9], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 0.522). The second judge finds a material difference on 3% (4/131). Mean answer time 38 s, USD 0.18 per answer, USD 60 per benchmark run. Close to the top on content. Runs at its default reasoning, not at low.
Claude Sonnet 5.5, low: content score 9.7 of 10 [9.5–9.9], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 0.687). The second judge finds a material difference on 2% (2/128). Mean answer time 19 s, USD 0.32 per answer, USD 104 per benchmark run. No significant difference from Claude Opus 5.5 on the content score, at less than half its cost per answer. Further from the approved answers in tone (0.90).
Kimi K3, low: content score 9.7 of 10 [9.5–9.9], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 1). The second judge finds a material difference on 3% (4/131). Mean answer time 30 s, USD 0.39 per answer, USD 127 per benchmark run. Further from the approved answers in tone (0.91).
Claude Opus 5, low: content score 9.6 of 10 [9.3–9.8], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 1). The second judge finds a material difference on 4% (5/130). Mean answer time 25 s, USD 1.18 per answer, USD 384 per benchmark run. The most expensive model, with no content or tone advantage over Claude Opus 5.5. Not recommended.
GPT-5.1, low: content score 9.5 of 10 [9.2–9.7], significantly lower than Claude Opus 5.5 (Holm-adjusted p = 0.002). The second judge finds a material difference on 5% (6/128). Mean answer time 68 s, USD 0.17 per answer, USD 56 per benchmark run. The option when inference must run in Switzerland. The slowest model in run 1; its answer times are inflated by a shared token limit.
GPT-5.6 Terra, low: content score 9.3 of 10 [8.9–9.6], significantly lower than Claude Opus 5.5 (Holm-adjusted p = 0.001). The second judge finds a material difference on 12% (16/129). Mean answer time 16 s, USD 0.11 per answer, USD 36 per benchmark run. The highest content score in run 2 after Gemini 4 Argon, significantly below Claude Opus 5.5, at about 36% of GPT-5.6 Sol's cost per answer.
GPT-6.1 Sol, low: content score 9.2 of 10 [9.0–9.5], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 5% (6/131). Mean answer time 19 s, USD 0.49 per answer, USD 159 per benchmark run. Significantly below Claude Opus 5.5 on content and far from the approved answers in tone (0.87): its answers are a median 1.2 times as long as the approved answer, and the second judge finds content beyond the approved answer in 89% of its judged answers, which the judges do not verify.
GPT-5.6 Sol, low: content score 9.2 of 10 [8.9–9.5], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 9% (11/129). Mean answer time 19 s, USD 0.31 per answer, USD 100 per benchmark run. Significantly below Claude Opus 5.5 on content. The second judge finds content beyond the approved answer in 62% of its judged answers.
Claude Sonnet 5, low: content score 9.2 of 10 [8.9–9.5], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 7% (9/126). Mean answer time 21 s, USD 0.34 per answer, USD 112 per benchmark run. Claude Sonnet 5.5 costs about the same per answer with no significant content difference from Claude Opus 5.5. Not recommended.
GPT-6 Astra, low: content score 9.1 of 10 [8.8–9.4], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 7% (9/131). Mean answer time 20 s, USD 0.44 per answer, USD 143 per benchmark run. The most expensive OpenAI model in run 2 (USD 0.44 per answer), significantly below Claude Opus 5.5 on content.
GPT-6 Luna, low: content score 8.9 of 10 [8.4–9.3], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 20% (26/130). Mean answer time 17 s, USD 0.01 per answer, USD 2 per benchmark run. The least expensive model in run 2, but the second judge finds a material difference on 20% of its answers.
GPT-5.6 Luna, low: content score 8.7 of 10 [8.2–9.1], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 13% (17/130). Mean answer time 12 s, USD 0.01 per answer, USD 4 per benchmark run. About USD 0.01 per answer; the second judge finds a material difference on 13% of its answers.
GPT-4o, no reasoning (production default): content score 8.5 of 10 [8.0–9.0], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 22% (28/128). Mean answer time 50 s, USD 0.07 per answer, USD 24 per benchmark run. The production default. The second judge finds a material difference on 22% of its answers. Not recommended.
GPT-6 Sol, low: content score 8.5 of 10 [8.0–8.9], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 25% (33/133). Mean answer time 22 s, USD 0.16 per answer, USD 51 per benchmark run. After Mistral Medium 3, the second judge finds a material difference most often for this model (25% of its answers). Not suitable.
Claude Haiku 5.5, low: content score 8.4 of 10 [8.0–8.9], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 22% (28/126). Mean answer time 27 s, USD 0.02 per answer, USD 8 per benchmark run. Significantly below Claude Opus 5.5 on content; its answers are a median 1.5 times as long as the approved answer. Not suitable.
GPT-6.1 Terra, low: content score 8.4 of 10 [7.8–8.8], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 18% (23/129). Mean answer time 12 s, USD 0.05 per answer, USD 15 per benchmark run. The lowest content score after Mistral Medium 3, with the furthest tone from the approved answers (0.85): the second judge finds content beyond the approved answer in 85% of its judged answers.
Mistral Medium 3, no reasoning: content score 7.2 of 10 [6.5–7.8], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 39% (51/132). Mean answer time 13 s, USD 0.01 per answer, USD 3 per benchmark run. Often leaves out material content of the approved answer. Not suitable.
Statistical tests
Paired Wilcoxon signed-rank tests against Claude Opus 5.5, low, the highest content score, on the answerable questions both models have a score for, with a Holm correction for the eighteen comparisons in each column. The difference is the other model's mean minus Claude Opus 5.5's on those questions. On content, GPT-5.1, GPT-5.6 Terra, GPT-6.1 Sol, GPT-5.6 Sol, Claude Sonnet 5, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Luna, GPT-4o, GPT-6 Sol, Claude Haiku 5.5, GPT-6.1 Terra and Mistral Medium 3 are significantly lower; on tone, every model except Claude Opus 5 and Gemini 4 Argon is.
Model | Reasoning effort | Content score (GPT-5.4 judge) | Tone match |
|---|---|---|---|
Gemini 4 Argon | low | no significant difference (difference below 0.01; p = 0.879; Holm-adjusted p = 1; 133 shared questions) | no significant difference (difference +0.01; p = 0.875; Holm-adjusted p = 1; 133 shared questions) |
DeepSeek V4.1 Flash | default | no significant difference (difference −0.08; p = 0.104; Holm-adjusted p = 0.522; 130 shared questions) | significantly lower (difference −0.04; p < 0.001; Holm-adjusted p = 0.001; 133 shared questions) |
Claude Sonnet 5.5 | low | no significant difference (difference −0.17; p = 0.172; Holm-adjusted p = 0.687; 132 shared questions) | significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
Kimi K3 | low | no significant difference (difference −0.20; p = 0.34; Holm-adjusted p = 1; 132 shared questions) | significantly lower (difference −0.03; p < 0.001; Holm-adjusted p = 0.002; 133 shared questions) |
Claude Opus 5 | low | no significant difference (difference −0.29; p = 0.341; Holm-adjusted p = 1; 133 shared questions) | no significant difference (difference below 0.01; p = 0.64; Holm-adjusted p = 1; 133 shared questions) |
GPT-5.1 | low | significantly lower (difference −0.42; p < 0.001; Holm-adjusted p = 0.002; 130 shared questions) | significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-5.6 Terra | low | significantly lower (difference −0.62; p < 0.001; Holm-adjusted p = 0.001; 130 shared questions) | significantly lower (difference −0.03; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-6.1 Sol | low | significantly lower (difference −0.64; p < 0.001; Holm-adjusted p < 0.001; 132 shared questions) | significantly lower (difference −0.08; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-5.6 Sol | low | significantly lower (difference −0.68; p < 0.001; Holm-adjusted p < 0.001; 132 shared questions) | significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
Claude Sonnet 5 | low | significantly lower (difference −0.68; p < 0.001; Holm-adjusted p < 0.001; 130 shared questions) | significantly lower (difference −0.07; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-6 Astra | low | significantly lower (difference −0.77; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) | significantly lower (difference −0.07; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-6 Luna | low | significantly lower (difference −0.98; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions) | significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-5.6 Luna | low | significantly lower (difference −1.22; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions) | significantly lower (difference −0.06; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-4o (production default) | not applicable | significantly lower (difference −1.36; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions) | significantly lower (difference −0.09; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-6 Sol | low | significantly lower (difference −1.41; p < 0.001; Holm-adjusted p < 0.001; 131 shared questions) | significantly lower (difference −0.09; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
Claude Haiku 5.5 | low | significantly lower (difference −1.47; p < 0.001; Holm-adjusted p < 0.001; 129 shared questions) | significantly lower (difference −0.08; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
GPT-6.1 Terra | low | significantly lower (difference −1.54; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) | significantly lower (difference −0.10; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
Mistral Medium 3 | not applicable | significantly lower (difference −2.75; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions) | significantly lower (difference −0.07; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions) |
How the benchmark works
The verbatim pass rules of the Verbatim library answers benchmark do not fit this agent, because it rewrites library text by design. Instead, two LLM judges compare each answer with the approved library answer.
Answerable question (133): content compared with the approved answer. GPT-5.4 compares the answer with the approved library answer and scores two criteria from 0 to 10, using our G-Eval-style scoring rubric. Coverage: every material point, question part, condition and exception of the approved answer is kept. Contradiction: no claim conflicts with the approved answer, so no reversed meaning and no changed value, date or condition. 10 means the criterion holds completely. The scores compare models; they are not a pass mark. A model's content score is the mean of the two criteria over the answerable questions, and models are compared with a paired test on the same questions. The judge sees only the direct answer; the supporting arguments and citations are removed first.
Content kept. The share of answers that score at least 7 of 10 on both criteria, shown next to the content score for an at-a-glance figure. It ignores differences above the cut-off, so models that are significantly apart on the content score can show the same share; the tests use the content score.
Second judge. Claude Sonnet 5.5 reads the same pairs and labels specific differences: a missing required claim, a contradiction, a changed value, a reversed meaning (such as Yes for No), or a changed condition or exception. The two judges agree: on the 2,423 answers both scored, the ones Claude Sonnet 5.5 labels with a material difference get a median GPT-5.4 score of 5.8, against 10.0 for the others. GPT-5.4 puts Claude Opus 5.5 first, level with Gemini 4 Argon; Claude Sonnet 5.5 puts Gemini 4 Argon first and Claude Opus 5.5 second; both put Mistral Medium 3 last. Claude Sonnet 5.5 is from Anthropic, a different provider from GPT-5.4 but the same as the Claude answering models, including Claude Opus 5.5, and it is one of the answering models itself; GPT-5.4 is not an answering model. GPT-5.4 is from OpenAI, the same provider as the GPT answering models, and all of them score significantly below Claude Opus 5.5 on content, so there is no sign that the judge favours them. Automatic measures that use no LLM agree with the judges; see How we checked the content score.
Additions are not errors. The agent composes from every eligible source by design, so the rubric does not count content beyond the approved answer against an answer. The judges check that added content does not contradict the approved answer, not that it is true. How often each model adds content is reported for reference.
Tone match (133). How closely the answer's writing style matches its approved answer: the similarity of two style embeddings (LUAR, with content words masked so that the facts do not count), from 0 to 1, averaged over the answerable questions. A verbatim copy scores 1; two different approved answers score 0.75 on average. This is a measurement, not a judge. It rises when an answer reuses the approved wording and falls as the answer's length moves away from the approved answer's (rank correlation −0.64), so a lower score often means a longer, rewritten answer rather than a wrong tone. The order of the models holds when the length difference is taken out: Gemini 4 Argon and the Claude Opus models stay closest and GPT-6.1 Terra furthest. Answers with a lower content score also score lower on tone (rank correlation 0.30), so the masking does not separate style from content completely. Tone match has not been checked against people's ratings of tone.
Question without a library answer (192). These questions count only in cost and answer time.
Citations are not used to score accuracy. The benchmark records one expected library file per question, and composed answers often cite another entry with the same content. On the answerable questions, the citation check and the second judge disagree on 414 of 2,465 answers: 233 have no material difference from the approved answer without citing the expected file, and 181 cite the expected file with a material difference.
How we checked the content score
The content score comes from an LLM judge, so we checked it two ways: against the second LLM judge and against 13 automatic measures that use no LLM (entailment and consistency models, embedding similarity, word-level similarity and word overlap).
Second judge. Claude Sonnet 5.5 puts the same two models at the top: it finds no material difference in 100% of Gemini 4 Argon's judged answers and 99% of Claude Opus 5.5's. Comparing every answer without a material difference with every answer with one, the first has the higher GPT-5.4 score in 87% of pairs and the same score in 8%. Claude Sonnet 5.5 is from the same provider as Claude Opus 5.5, and it ranks Gemini 4 Argon, a Google model, above it.
Same order at the top and the bottom. 12 of the 13 automatic measures put Claude Opus 5.5 or Gemini 4 Argon first (9 of them Gemini 4 Argon), and 4 put Mistral Medium 3 last.
Same answers scored low. The automatic measures rise with the judge's content score: rank correlation 0.19–0.52 on 12 of the 13 measures, where 0 would be no relation. The contradiction score of the NLI model barely follows it (rank correlation 0.09).
Where they do not agree. AlignScore's alignment score puts GPT-5.6 Terra first and ranks Gemini 4 Argon 15th of 19 (rank correlation with the judge's score 0.19).
What the automatic measures do not show. They reward staying close to the approved wording, so they favour answers that reuse it. On their own they confirm the models at the top and which answers lose content, not the order of the models in between or the exact score.
Model | Reasoning effort | Content score, 0 to 10 (GPT-5.4 judge) | No material difference (Claude Sonnet 5.5) | Entailment, 0 to 1 (DeBERTa NLI) | Embedding similarity, 0 to 1 (BGE-M3) | Word-level similarity, 0 to 1 (BERTScore) |
|---|---|---|---|---|---|---|
Claude Opus 5.5 | low | 9.9 | 99% | 0.91 | 0.95 | 0.98 |
Gemini 4 Argon | low | 9.9 | 100% | 0.93 | 0.96 | 0.99 |
DeepSeek V4.1 Flash | default | 9.8 | 97% | 0.84 | 0.87 | 0.97 |
Claude Sonnet 5.5 | low | 9.7 | 98% | 0.76 | 0.86 | 0.96 |
Kimi K3 | low | 9.7 | 97% | 0.81 | 0.89 | 0.97 |
Claude Opus 5 | low | 9.6 | 96% | 0.88 | 0.94 | 0.98 |
GPT-5.1 | low | 9.5 | 95% | 0.78 | 0.87 | 0.95 |
GPT-5.6 Terra | low | 9.3 | 88% | 0.73 | 0.88 | 0.95 |
GPT-6.1 Sol | low | 9.2 | 95% | 0.84 | 0.85 | 0.96 |
GPT-5.6 Sol | low | 9.2 | 91% | 0.77 | 0.85 | 0.95 |
Claude Sonnet 5 | low | 9.2 | 93% | 0.75 | 0.82 | 0.95 |
GPT-6 Astra | low | 9.1 | 93% | 0.84 | 0.84 | 0.96 |
GPT-6 Luna | low | 8.9 | 80% | 0.69 | 0.86 | 0.94 |
GPT-5.6 Luna | low | 8.7 | 87% | 0.63 | 0.83 | 0.93 |
GPT-4o (production default) | not applicable | 8.5 | 78% | 0.67 | 0.77 | 0.92 |
GPT-6 Sol | low | 8.5 | 75% | 0.66 | 0.77 | 0.93 |
Claude Haiku 5.5 | low | 8.4 | 78% | 0.58 | 0.79 | 0.93 |
GPT-6.1 Terra | low | 8.4 | 82% | 0.71 | 0.80 | 0.94 |
Mistral Medium 3 | not applicable | 7.2 | 61% | 0.53 | 0.78 | 0.92 |
Setup
The agent's configuration is the Verbatim library answers configuration with two changes: its own prompt and a larger search result limit, so each search returns more library sources. Search scope, temperature and every other setting are the same.
Every model was given the 325 scored cases of the Verbatim library answers benchmark (133 with an approved answer in the Answer Library, 192 without), question only, with the same providers and routes as on that benchmark. Reasoning was set to low for every model that supports it. GPT-4o and Mistral Medium 3 have no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model.
Every model is compared with the one with the highest content score.
Hosting. GPT-5.1 runs on Azure in Switzerland, GPT-4o on Azure OpenAI and Mistral Medium 3 and Gemini 4 Argon on Vertex AI in the EU. Anthropic, Azure OpenAI global deployments (GPT-6.1 Terra, GPT-6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna), OpenAI through LiteLLM (GPT-6.1 Sol) and Together AI (Kimi K3 and DeepSeek V4.1 Flash) process globally. Which providers a client can use depends on its contract.
Configurations
Model | Model deployment | Reasoning effort |
|---|---|---|
Claude Opus 5.5 | Claude Opus 5.5, Anthropic through LiteLLM | low |
Gemini 4 Argon | Gemini 4 Argon, Google Vertex AI EU through LiteLLM | low |
DeepSeek V4.1 Flash | DeepSeek V4.1 Flash, Together AI through LiteLLM | default (Together AI rejects the reasoning setting the platform sends, so it is removed and the model runs at its default) |
Claude Sonnet 5.5 | Claude Sonnet 5.5, Anthropic through LiteLLM | low |
Kimi K3 | Kimi K3, Together AI through LiteLLM | low |
Claude Opus 5 | Claude Opus 5, Anthropic through LiteLLM | low |
GPT-5.1 | GPT-5.1 (2025-11-13), Azure, Swiss inference | low |
GPT-5.6 Terra | GPT-5.6 Terra (2026-07-09), Azure OpenAI | low |
GPT-6.1 Sol | GPT-6.1 Sol, OpenAI through LiteLLM | low |
GPT-5.6 Sol | GPT-5.6 Sol (2026-07-09), Azure OpenAI | low |
Claude Sonnet 5 | Claude Sonnet 5, Anthropic through LiteLLM | low |
GPT-6 Astra | GPT-6 Astra (2026-09-03), Azure OpenAI | low |
GPT-6 Luna | GPT-6 Luna (2026-09-22), Azure OpenAI | low |
GPT-5.6 Luna | GPT-5.6 Luna (2026-07-09), Azure OpenAI | low |
GPT-4o (production default) | GPT-4o (2024-11-20), Azure OpenAI | not applicable (not a reasoning model) |
GPT-6 Sol | GPT-6 Sol (2026-09-22), Azure OpenAI | low |
Claude Haiku 5.5 | Claude Haiku 5.5, Anthropic through LiteLLM | low |
GPT-6.1 Terra | GPT-6.1 Terra (2026-10-26), Azure OpenAI | low |
Mistral Medium 3 | Mistral Medium 3, Google Vertex AI EU through LiteLLM | not applicable (no reasoning setting) |
Notes
Times are measured per answer. Costs are list prices for the agent's calls, from the usage the platform records or reads back afterwards, not invoices. The longer prompt and the larger search result limit add input per answer compared with Verbatim library answers, and the composed answers are longer, so costs per answer are not comparable between the two pages. For some answers the run did not capture usage data, so it was read back from the chat records after the run and priced with the same price sheet: Claude Opus 5.5 24, DeepSeek V4.1 Flash 11, Kimi K3 19, GPT-5.1 2, Mistral Medium 3 25. Answers with no usage data at all are scored and left out of the cost mean: GPT-5.1 2, GPT-4o 1. Answers that failed for platform reasons (service restarts, provider rate limits) were retried and are not counted as wrong answers. On 3 of the 192 questions the library cannot answer, Gemini 4 Argon kept searching and returned no answer within 30 minutes. Each of the 3 was tried 3 to 8 times and every one ended in a timeout, the last at the 30-minute limit, so they count as a result for the model, not as platform failures. Gemini 4 Argon's cost and time are over the 322 questions it answered and leave out these 3.
Content scores come from LLM judges, not from people. Each answer was judged once, with no repeat and no planted test answers. GPT-5.4 returned no score on 42 of the 2,527 answers (it marked a criterion as not applicable or its reply could not be read) and Claude Sonnet 5.5 on 62; these are left out of the scores and rates. The comparison is with the approved answer only, so content the agent adds from other eligible sources is not checked.
Mistral Medium 3's low content score is not caused by how the direct answer is separated from the supporting arguments: its direct answers are about as long as the others' (median 33 words; Claude Opus 5.5: 35), and the second judge most often labels a missing required claim (44 of its 132 judged answers).