Agentic Table-Composed Answers from Library Sources Benchmark

22 min read

This page lists the answering models we evaluated for the Composed answers from library sources agent. Where the Verbatim library answers agent returns the approved library text verbatim or refuses, this agent composes an answer from eligible library sources, with supporting arguments and as-of dates, and is designed to flag the gap when no eligible source determines the answer. The cases and scoring are described under How the benchmark works below.

The models were tested in two runs with the same questions, library, prompt, judges and scoring. Run 2 (Gemini 4 Argon, GPT-5.6 Terra, GPT-5.6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Luna, GPT-4o, GPT-6 Sol, Claude Haiku 5.5 and GPT-6.1 Terra) retrieved fewer library sources per search than run 1 (Claude Opus 5.5, DeepSeek V4.1 Flash, Claude Sonnet 5.5, Kimi K3, Claude Opus 5, GPT-5.1, GPT-6.1 Sol, Claude Sonnet 5 and Mistral Medium 3). That lowers the input per answer, so cost and answer time are compared only within a run.

Key findings

  • Claude Opus 5.5 at low reasoning and Gemini 4 Argon at low reasoning keep the approved content best (content score 9.9 of 10 each; content kept on 99% and 98%), and their tone is the closest, level with Claude Opus 5 at low reasoning. Gemini 4 Argon is much slower: a mean 178 s per answer, against 12 to 50 s for the other models in its run. On 3 of the 192 questions the library cannot answer, Gemini 4 Argon kept searching and returned no answer within 30 minutes. DeepSeek V4.1 Flash at its default reasoning, Claude Sonnet 5.5 at low reasoning, Kimi K3 at low reasoning and Claude Opus 5 score 9.6–9.8, no significant difference from Claude Opus 5.5 after correction. The other 12 models score significantly lower (8.4–9.5): 37–76% of their answers get full marks on both criteria, against 95% for both Claude Opus 5.5 and Gemini 4 Argon, and the second judge finds a material difference on 5–25% of their answers. Mistral Medium 3 (no reasoning) scores 7.2 and often leaves out material content. Every model except Claude Opus 5 and Gemini 4 Argon is significantly further from the approved answers in tone, down to 0.85 for GPT-6.1 Terra, whose answers are a median 1.2 times as long as the approved answer.

Results

Content score and tone match: mean with a 95% bootstrap interval. Content kept: 95% Wilson interval. Content rows count only answers the judge returned a score for; the missing ones are listed in their own row. Answer time: mean per question with a 95% bootstrap interval. Cost per benchmark run: mean list price per answer times 325. Models are ranked by content score, highest first.

Model

Reasoning effort

Provider

Content score, 0 to 10 (GPT-5.4 judge, 133 answerable), mean [95% interval]

Content kept: at least 7 of 10 on both criteria (GPT-5.4 judge) [95% interval]

Full marks on both criteria (GPT-5.4 judge)

Material difference found by the second judge (Claude Sonnet 5.5)

Adds content not in the approved answer (second judge)

Answers with no judge score (GPT-5.4 / Claude Sonnet 5.5)

Tone match with the approved answer, 0 to 1, mean [95% interval]

Answer time, mean [95% interval]

Answer time, median / 90th percentile

Cost per answer (mean)

Cost per benchmark run

Run (cost and time compare only within a run)

Claude Opus 5.5

low

Anthropic

9.9 [9.8–10.0]

99% (132/133) [96–100%]

95% (127/133)

1% (1/132)

10% (13/132)

0 / 1

0.95 [0.93–0.96]

29 s [28–30 s]

28 s / 38 s

USD 0.74

USD 241

Run 1

Gemini 4 Argon

low

Google

9.9 [9.8–10.0]

98% (130/133) [94–99%]

95% (126/133)

0% (0/131)

13% (17/131)

0 / 2

0.95 [0.94–0.97]

178 s [158–199 s]

107 s / 408 s

USD 0.51

USD 166

Run 2

DeepSeek V4.1 Flash

default

DeepSeek

9.8 [9.7–9.9]

98% (128/130) [95–100%]

86% (112/130)

3% (4/131)

29% (38/131)

3 / 2

0.90 [0.88–0.93]

38 s [35–41 s]

32 s / 74 s

USD 0.18

USD 60

Run 1

Claude Sonnet 5.5

low

Anthropic

9.7 [9.5–9.9]

95% (126/132) [90–98%]

88% (116/132)

2% (2/128)

31% (40/128)

1 / 5

0.90 [0.87–0.92]

19 s [19–20 s]

19 s / 25 s

USD 0.32

USD 104

Run 1

Kimi K3

low

Moonshot AI

9.7 [9.5–9.9]

95% (125/132) [89–97%]

92% (121/132)

3% (4/131)

23% (30/131)

1 / 2

0.91 [0.89–0.93]

30 s [29–31 s]

28 s / 42 s

USD 0.39

USD 127

Run 1

Claude Opus 5

low

Anthropic

9.6 [9.3–9.8]

95% (126/133) [90–97%]

89% (118/133)

4% (5/130)

15% (19/130)

0 / 3

0.95 [0.93–0.96]

25 s [25–26 s]

24 s / 34 s

USD 1.18

USD 384

Run 1

GPT-5.1

low

OpenAI

9.5 [9.2–9.7]

93% (121/130) [87–96%]

76% (99/130)

5% (6/128)

35% (45/128)

3 / 5

0.90 [0.88–0.92]

68 s [64–72 s]

60 s / 115 s

USD 0.17

USD 56

Run 1

GPT-5.6 Terra

low

OpenAI

9.3 [8.9–9.6]

92% (119/130) [85–95%]

75% (98/130)

12% (16/129)

32% (41/129)

3 / 4

0.91 [0.89–0.93]

16 s [16–17 s]

16 s / 23 s

USD 0.11

USD 36

Run 2

GPT-6.1 Sol

low

OpenAI

9.2 [9.0–9.5]

95% (125/132) [89–97%]

46% (61/132)

5% (6/131)

89% (116/131)

1 / 2

0.87 [0.84–0.90]

19 s [19–20 s]

19 s / 27 s

USD 0.49

USD 159

Run 1

GPT-5.6 Sol

low

OpenAI

9.2 [8.9–9.5]

91% (120/132) [85–95%]

67% (88/132)

9% (11/129)

62% (80/129)

1 / 4

0.89 [0.87–0.92]

19 s [19–20 s]

19 s / 26 s

USD 0.31

USD 100

Run 2

Claude Sonnet 5

low

Anthropic

9.2 [8.9–9.5]

89% (116/130) [83–93%]

71% (92/130)

7% (9/126)

44% (56/126)

3 / 7

0.88 [0.86–0.90]

21 s [20–22 s]

20 s / 30 s

USD 0.34

USD 112

Run 1

GPT-6 Astra

low

OpenAI

9.1 [8.8–9.4]

92% (123/133) [87–96%]

45% (60/133)

7% (9/131)

88% (115/131)

0 / 2

0.88 [0.85–0.90]

20 s [19–21 s]

20 s / 29 s

USD 0.44

USD 143

Run 2

GPT-6 Luna

low

OpenAI

8.9 [8.4–9.3]

87% (111/128) [80–92%]

64% (82/128)

20% (26/130)

31% (40/130)

5 / 3

0.89 [0.87–0.91]

17 s [13–22 s]

11 s / 15 s

USD 0.01

USD 2

Run 2

GPT-5.6 Luna

low

OpenAI

8.7 [8.2–9.1]

84% (107/128) [76–89%]

55% (71/128)

13% (17/130)

48% (62/130)

5 / 3

0.88 [0.86–0.90]

12 s [12–13 s]

12 s / 16 s

USD 0.01

USD 4

Run 2

GPT-4o (production default)

not applicable

OpenAI

8.5 [8.0–9.0]

84% (107/128) [76–89%]

58% (74/128)

22% (28/128)

41% (53/128)

5 / 5

0.86 [0.83–0.89]

50 s [45–55 s]

27 s / 121 s

USD 0.07

USD 24

Run 2

GPT-6 Sol

low

OpenAI

8.5 [8.0–8.9]

86% (113/131) [79–91%]

37% (49/131)

25% (33/133)

84% (112/133)

2 / 0

0.86 [0.83–0.88]

22 s [21–23 s]

22 s / 32 s

USD 0.16

USD 51

Run 2

Claude Haiku 5.5

low

Anthropic

8.4 [8.0–8.9]

79% (102/129) [71–85%]

51% (66/129)

22% (28/126)

60% (75/126)

4 / 7

0.87 [0.84–0.89]

27 s [26–27 s]

25 s / 35 s

USD 0.02

USD 8

Run 2

GPT-6.1 Terra

low

OpenAI

8.4 [7.8–8.8]

81% (108/133) [74–87%]

47% (62/133)

18% (23/129)

85% (110/129)

0 / 4

0.85 [0.82–0.88]

12 s [12–12 s]

12 s / 15 s

USD 0.05

USD 15

Run 2

Mistral Medium 3

not applicable

Mistral

7.2 [6.5–7.8]

63% (80/128) [54–70%]

55% (70/128)

39% (51/132)

24% (32/132)

5 / 1

0.88 [0.86–0.90]

13 s [13–14 s]

11 s / 24 s

USD 0.01

USD 3

Run 1

Model notes

  • Claude Opus 5.5, low: content score 9.9 of 10 [9.8–10.0]. The second judge finds a material difference on 1% (1/132). Mean answer time 29 s, USD 0.74 per answer, USD 241 per benchmark run. The highest content score, level with Gemini 4 Argon, and, with Gemini 4 Argon and Claude Opus 5, the closest tone. About 2.3 times the cost of Claude Sonnet 5.5 per answer.

  • Gemini 4 Argon, low: content score 9.9 of 10 [9.8–10.0], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 1). The second judge finds a material difference on 0% (0/131). Mean answer time 178 s, USD 0.51 per answer, USD 166 per benchmark run. Level with Claude Opus 5.5 on content and tone, and the second judge finds no material difference in any of its 131 judged answers. The slowest and the most expensive model in run 2. On 3 of the 192 questions the library cannot answer, Gemini 4 Argon kept searching and returned no answer within 30 minutes.

  • DeepSeek V4.1 Flash, default: content score 9.8 of 10 [9.7–9.9], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 0.522). The second judge finds a material difference on 3% (4/131). Mean answer time 38 s, USD 0.18 per answer, USD 60 per benchmark run. Close to the top on content. Runs at its default reasoning, not at low.

  • Claude Sonnet 5.5, low: content score 9.7 of 10 [9.5–9.9], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 0.687). The second judge finds a material difference on 2% (2/128). Mean answer time 19 s, USD 0.32 per answer, USD 104 per benchmark run. No significant difference from Claude Opus 5.5 on the content score, at less than half its cost per answer. Further from the approved answers in tone (0.90).

  • Kimi K3, low: content score 9.7 of 10 [9.5–9.9], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 1). The second judge finds a material difference on 3% (4/131). Mean answer time 30 s, USD 0.39 per answer, USD 127 per benchmark run. Further from the approved answers in tone (0.91).

  • Claude Opus 5, low: content score 9.6 of 10 [9.3–9.8], no significant difference from Claude Opus 5.5 (Holm-adjusted p = 1). The second judge finds a material difference on 4% (5/130). Mean answer time 25 s, USD 1.18 per answer, USD 384 per benchmark run. The most expensive model, with no content or tone advantage over Claude Opus 5.5. Not recommended.

  • GPT-5.1, low: content score 9.5 of 10 [9.2–9.7], significantly lower than Claude Opus 5.5 (Holm-adjusted p = 0.002). The second judge finds a material difference on 5% (6/128). Mean answer time 68 s, USD 0.17 per answer, USD 56 per benchmark run. The option when inference must run in Switzerland. The slowest model in run 1; its answer times are inflated by a shared token limit.

  • GPT-5.6 Terra, low: content score 9.3 of 10 [8.9–9.6], significantly lower than Claude Opus 5.5 (Holm-adjusted p = 0.001). The second judge finds a material difference on 12% (16/129). Mean answer time 16 s, USD 0.11 per answer, USD 36 per benchmark run. The highest content score in run 2 after Gemini 4 Argon, significantly below Claude Opus 5.5, at about 36% of GPT-5.6 Sol's cost per answer.

  • GPT-6.1 Sol, low: content score 9.2 of 10 [9.0–9.5], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 5% (6/131). Mean answer time 19 s, USD 0.49 per answer, USD 159 per benchmark run. Significantly below Claude Opus 5.5 on content and far from the approved answers in tone (0.87): its answers are a median 1.2 times as long as the approved answer, and the second judge finds content beyond the approved answer in 89% of its judged answers, which the judges do not verify.

  • GPT-5.6 Sol, low: content score 9.2 of 10 [8.9–9.5], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 9% (11/129). Mean answer time 19 s, USD 0.31 per answer, USD 100 per benchmark run. Significantly below Claude Opus 5.5 on content. The second judge finds content beyond the approved answer in 62% of its judged answers.

  • Claude Sonnet 5, low: content score 9.2 of 10 [8.9–9.5], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 7% (9/126). Mean answer time 21 s, USD 0.34 per answer, USD 112 per benchmark run. Claude Sonnet 5.5 costs about the same per answer with no significant content difference from Claude Opus 5.5. Not recommended.

  • GPT-6 Astra, low: content score 9.1 of 10 [8.8–9.4], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 7% (9/131). Mean answer time 20 s, USD 0.44 per answer, USD 143 per benchmark run. The most expensive OpenAI model in run 2 (USD 0.44 per answer), significantly below Claude Opus 5.5 on content.

  • GPT-6 Luna, low: content score 8.9 of 10 [8.4–9.3], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 20% (26/130). Mean answer time 17 s, USD 0.01 per answer, USD 2 per benchmark run. The least expensive model in run 2, but the second judge finds a material difference on 20% of its answers.

  • GPT-5.6 Luna, low: content score 8.7 of 10 [8.2–9.1], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 13% (17/130). Mean answer time 12 s, USD 0.01 per answer, USD 4 per benchmark run. About USD 0.01 per answer; the second judge finds a material difference on 13% of its answers.

  • GPT-4o, no reasoning (production default): content score 8.5 of 10 [8.0–9.0], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 22% (28/128). Mean answer time 50 s, USD 0.07 per answer, USD 24 per benchmark run. The production default. The second judge finds a material difference on 22% of its answers. Not recommended.

  • GPT-6 Sol, low: content score 8.5 of 10 [8.0–8.9], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 25% (33/133). Mean answer time 22 s, USD 0.16 per answer, USD 51 per benchmark run. After Mistral Medium 3, the second judge finds a material difference most often for this model (25% of its answers). Not suitable.

  • Claude Haiku 5.5, low: content score 8.4 of 10 [8.0–8.9], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 22% (28/126). Mean answer time 27 s, USD 0.02 per answer, USD 8 per benchmark run. Significantly below Claude Opus 5.5 on content; its answers are a median 1.5 times as long as the approved answer. Not suitable.

  • GPT-6.1 Terra, low: content score 8.4 of 10 [7.8–8.8], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 18% (23/129). Mean answer time 12 s, USD 0.05 per answer, USD 15 per benchmark run. The lowest content score after Mistral Medium 3, with the furthest tone from the approved answers (0.85): the second judge finds content beyond the approved answer in 85% of its judged answers.

  • Mistral Medium 3, no reasoning: content score 7.2 of 10 [6.5–7.8], significantly lower than Claude Opus 5.5 (Holm-adjusted p < 0.001). The second judge finds a material difference on 39% (51/132). Mean answer time 13 s, USD 0.01 per answer, USD 3 per benchmark run. Often leaves out material content of the approved answer. Not suitable.

Statistical tests

Paired Wilcoxon signed-rank tests against Claude Opus 5.5, low, the highest content score, on the answerable questions both models have a score for, with a Holm correction for the eighteen comparisons in each column. The difference is the other model's mean minus Claude Opus 5.5's on those questions. On content, GPT-5.1, GPT-5.6 Terra, GPT-6.1 Sol, GPT-5.6 Sol, Claude Sonnet 5, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Luna, GPT-4o, GPT-6 Sol, Claude Haiku 5.5, GPT-6.1 Terra and Mistral Medium 3 are significantly lower; on tone, every model except Claude Opus 5 and Gemini 4 Argon is.

Model

Reasoning effort

Content score (GPT-5.4 judge)

Tone match

Gemini 4 Argon

low

no significant difference (difference below 0.01; p = 0.879; Holm-adjusted p = 1; 133 shared questions)

no significant difference (difference +0.01; p = 0.875; Holm-adjusted p = 1; 133 shared questions)

DeepSeek V4.1 Flash

default

no significant difference (difference −0.08; p = 0.104; Holm-adjusted p = 0.522; 130 shared questions)

significantly lower (difference −0.04; p < 0.001; Holm-adjusted p = 0.001; 133 shared questions)

Claude Sonnet 5.5

low

no significant difference (difference −0.17; p = 0.172; Holm-adjusted p = 0.687; 132 shared questions)

significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

Kimi K3

low

no significant difference (difference −0.20; p = 0.34; Holm-adjusted p = 1; 132 shared questions)

significantly lower (difference −0.03; p < 0.001; Holm-adjusted p = 0.002; 133 shared questions)

Claude Opus 5

low

no significant difference (difference −0.29; p = 0.341; Holm-adjusted p = 1; 133 shared questions)

no significant difference (difference below 0.01; p = 0.64; Holm-adjusted p = 1; 133 shared questions)

GPT-5.1

low

significantly lower (difference −0.42; p < 0.001; Holm-adjusted p = 0.002; 130 shared questions)

significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-5.6 Terra

low

significantly lower (difference −0.62; p < 0.001; Holm-adjusted p = 0.001; 130 shared questions)

significantly lower (difference −0.03; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-6.1 Sol

low

significantly lower (difference −0.64; p < 0.001; Holm-adjusted p < 0.001; 132 shared questions)

significantly lower (difference −0.08; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-5.6 Sol

low

significantly lower (difference −0.68; p < 0.001; Holm-adjusted p < 0.001; 132 shared questions)

significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

Claude Sonnet 5

low

significantly lower (difference −0.68; p < 0.001; Holm-adjusted p < 0.001; 130 shared questions)

significantly lower (difference −0.07; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-6 Astra

low

significantly lower (difference −0.77; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

significantly lower (difference −0.07; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-6 Luna

low

significantly lower (difference −0.98; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions)

significantly lower (difference −0.05; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-5.6 Luna

low

significantly lower (difference −1.22; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions)

significantly lower (difference −0.06; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-4o (production default)

not applicable

significantly lower (difference −1.36; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions)

significantly lower (difference −0.09; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-6 Sol

low

significantly lower (difference −1.41; p < 0.001; Holm-adjusted p < 0.001; 131 shared questions)

significantly lower (difference −0.09; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

Claude Haiku 5.5

low

significantly lower (difference −1.47; p < 0.001; Holm-adjusted p < 0.001; 129 shared questions)

significantly lower (difference −0.08; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

GPT-6.1 Terra

low

significantly lower (difference −1.54; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

significantly lower (difference −0.10; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

Mistral Medium 3

not applicable

significantly lower (difference −2.75; p < 0.001; Holm-adjusted p < 0.001; 128 shared questions)

significantly lower (difference −0.07; p < 0.001; Holm-adjusted p < 0.001; 133 shared questions)

How the benchmark works

The verbatim pass rules of the Verbatim library answers benchmark do not fit this agent, because it rewrites library text by design. Instead, two LLM judges compare each answer with the approved library answer.

  • Answerable question (133): content compared with the approved answer. GPT-5.4 compares the answer with the approved library answer and scores two criteria from 0 to 10, using our G-Eval-style scoring rubric. Coverage: every material point, question part, condition and exception of the approved answer is kept. Contradiction: no claim conflicts with the approved answer, so no reversed meaning and no changed value, date or condition. 10 means the criterion holds completely. The scores compare models; they are not a pass mark. A model's content score is the mean of the two criteria over the answerable questions, and models are compared with a paired test on the same questions. The judge sees only the direct answer; the supporting arguments and citations are removed first.

  • Content kept. The share of answers that score at least 7 of 10 on both criteria, shown next to the content score for an at-a-glance figure. It ignores differences above the cut-off, so models that are significantly apart on the content score can show the same share; the tests use the content score.

  • Second judge. Claude Sonnet 5.5 reads the same pairs and labels specific differences: a missing required claim, a contradiction, a changed value, a reversed meaning (such as Yes for No), or a changed condition or exception. The two judges agree: on the 2,423 answers both scored, the ones Claude Sonnet 5.5 labels with a material difference get a median GPT-5.4 score of 5.8, against 10.0 for the others. GPT-5.4 puts Claude Opus 5.5 first, level with Gemini 4 Argon; Claude Sonnet 5.5 puts Gemini 4 Argon first and Claude Opus 5.5 second; both put Mistral Medium 3 last. Claude Sonnet 5.5 is from Anthropic, a different provider from GPT-5.4 but the same as the Claude answering models, including Claude Opus 5.5, and it is one of the answering models itself; GPT-5.4 is not an answering model. GPT-5.4 is from OpenAI, the same provider as the GPT answering models, and all of them score significantly below Claude Opus 5.5 on content, so there is no sign that the judge favours them. Automatic measures that use no LLM agree with the judges; see How we checked the content score.

  • Additions are not errors. The agent composes from every eligible source by design, so the rubric does not count content beyond the approved answer against an answer. The judges check that added content does not contradict the approved answer, not that it is true. How often each model adds content is reported for reference.

  • Tone match (133). How closely the answer's writing style matches its approved answer: the similarity of two style embeddings (LUAR, with content words masked so that the facts do not count), from 0 to 1, averaged over the answerable questions. A verbatim copy scores 1; two different approved answers score 0.75 on average. This is a measurement, not a judge. It rises when an answer reuses the approved wording and falls as the answer's length moves away from the approved answer's (rank correlation −0.64), so a lower score often means a longer, rewritten answer rather than a wrong tone. The order of the models holds when the length difference is taken out: Gemini 4 Argon and the Claude Opus models stay closest and GPT-6.1 Terra furthest. Answers with a lower content score also score lower on tone (rank correlation 0.30), so the masking does not separate style from content completely. Tone match has not been checked against people's ratings of tone.

  • Question without a library answer (192). These questions count only in cost and answer time.

Citations are not used to score accuracy. The benchmark records one expected library file per question, and composed answers often cite another entry with the same content. On the answerable questions, the citation check and the second judge disagree on 414 of 2,465 answers: 233 have no material difference from the approved answer without citing the expected file, and 181 cite the expected file with a material difference.

How we checked the content score

The content score comes from an LLM judge, so we checked it two ways: against the second LLM judge and against 13 automatic measures that use no LLM (entailment and consistency models, embedding similarity, word-level similarity and word overlap).

  • Second judge. Claude Sonnet 5.5 puts the same two models at the top: it finds no material difference in 100% of Gemini 4 Argon's judged answers and 99% of Claude Opus 5.5's. Comparing every answer without a material difference with every answer with one, the first has the higher GPT-5.4 score in 87% of pairs and the same score in 8%. Claude Sonnet 5.5 is from the same provider as Claude Opus 5.5, and it ranks Gemini 4 Argon, a Google model, above it.

  • Same order at the top and the bottom. 12 of the 13 automatic measures put Claude Opus 5.5 or Gemini 4 Argon first (9 of them Gemini 4 Argon), and 4 put Mistral Medium 3 last.

  • Same answers scored low. The automatic measures rise with the judge's content score: rank correlation 0.19–0.52 on 12 of the 13 measures, where 0 would be no relation. The contradiction score of the NLI model barely follows it (rank correlation 0.09).

  • Where they do not agree. AlignScore's alignment score puts GPT-5.6 Terra first and ranks Gemini 4 Argon 15th of 19 (rank correlation with the judge's score 0.19).

  • What the automatic measures do not show. They reward staying close to the approved wording, so they favour answers that reuse it. On their own they confirm the models at the top and which answers lose content, not the order of the models in between or the exact score.

Model

Reasoning effort

Content score, 0 to 10 (GPT-5.4 judge)

No material difference (Claude Sonnet 5.5)

Entailment, 0 to 1 (DeBERTa NLI)

Embedding similarity, 0 to 1 (BGE-M3)

Word-level similarity, 0 to 1 (BERTScore)

Claude Opus 5.5

low

9.9

99%

0.91

0.95

0.98

Gemini 4 Argon

low

9.9

100%

0.93

0.96

0.99

DeepSeek V4.1 Flash

default

9.8

97%

0.84

0.87

0.97

Claude Sonnet 5.5

low

9.7

98%

0.76

0.86

0.96

Kimi K3

low

9.7

97%

0.81

0.89

0.97

Claude Opus 5

low

9.6

96%

0.88

0.94

0.98

GPT-5.1

low

9.5

95%

0.78

0.87

0.95

GPT-5.6 Terra

low

9.3

88%

0.73

0.88

0.95

GPT-6.1 Sol

low

9.2

95%

0.84

0.85

0.96

GPT-5.6 Sol

low

9.2

91%

0.77

0.85

0.95

Claude Sonnet 5

low

9.2

93%

0.75

0.82

0.95

GPT-6 Astra

low

9.1

93%

0.84

0.84

0.96

GPT-6 Luna

low

8.9

80%

0.69

0.86

0.94

GPT-5.6 Luna

low

8.7

87%

0.63

0.83

0.93

GPT-4o (production default)

not applicable

8.5

78%

0.67

0.77

0.92

GPT-6 Sol

low

8.5

75%

0.66

0.77

0.93

Claude Haiku 5.5

low

8.4

78%

0.58

0.79

0.93

GPT-6.1 Terra

low

8.4

82%

0.71

0.80

0.94

Mistral Medium 3

not applicable

7.2

61%

0.53

0.78

0.92

Setup

The agent's configuration is the Verbatim library answers configuration with two changes: its own prompt and a larger search result limit, so each search returns more library sources. Search scope, temperature and every other setting are the same.

Every model was given the 325 scored cases of the Verbatim library answers benchmark (133 with an approved answer in the Answer Library, 192 without), question only, with the same providers and routes as on that benchmark. Reasoning was set to low for every model that supports it. GPT-4o and Mistral Medium 3 have no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model.

Every model is compared with the one with the highest content score.

Hosting. GPT-5.1 runs on Azure in Switzerland, GPT-4o on Azure OpenAI and Mistral Medium 3 and Gemini 4 Argon on Vertex AI in the EU. Anthropic, Azure OpenAI global deployments (GPT-6.1 Terra, GPT-6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna), OpenAI through LiteLLM (GPT-6.1 Sol) and Together AI (Kimi K3 and DeepSeek V4.1 Flash) process globally. Which providers a client can use depends on its contract.

Configurations

Model

Model deployment

Reasoning effort

Claude Opus 5.5

Claude Opus 5.5, Anthropic through LiteLLM

low

Gemini 4 Argon

Gemini 4 Argon, Google Vertex AI EU through LiteLLM

low

DeepSeek V4.1 Flash

DeepSeek V4.1 Flash, Together AI through LiteLLM

default (Together AI rejects the reasoning setting the platform sends, so it is removed and the model runs at its default)

Claude Sonnet 5.5

Claude Sonnet 5.5, Anthropic through LiteLLM

low

Kimi K3

Kimi K3, Together AI through LiteLLM

low

Claude Opus 5

Claude Opus 5, Anthropic through LiteLLM

low

GPT-5.1

GPT-5.1 (2025-11-13), Azure, Swiss inference

low

GPT-5.6 Terra

GPT-5.6 Terra (2026-07-09), Azure OpenAI

low

GPT-6.1 Sol

GPT-6.1 Sol, OpenAI through LiteLLM

low

GPT-5.6 Sol

GPT-5.6 Sol (2026-07-09), Azure OpenAI

low

Claude Sonnet 5

Claude Sonnet 5, Anthropic through LiteLLM

low

GPT-6 Astra

GPT-6 Astra (2026-09-03), Azure OpenAI

low

GPT-6 Luna

GPT-6 Luna (2026-09-22), Azure OpenAI

low

GPT-5.6 Luna

GPT-5.6 Luna (2026-07-09), Azure OpenAI

low

GPT-4o (production default)

GPT-4o (2024-11-20), Azure OpenAI

not applicable (not a reasoning model)

GPT-6 Sol

GPT-6 Sol (2026-09-22), Azure OpenAI

low

Claude Haiku 5.5

Claude Haiku 5.5, Anthropic through LiteLLM

low

GPT-6.1 Terra

GPT-6.1 Terra (2026-10-26), Azure OpenAI

low

Mistral Medium 3

Mistral Medium 3, Google Vertex AI EU through LiteLLM

not applicable (no reasoning setting)

Notes

Times are measured per answer. Costs are list prices for the agent's calls, from the usage the platform records or reads back afterwards, not invoices. The longer prompt and the larger search result limit add input per answer compared with Verbatim library answers, and the composed answers are longer, so costs per answer are not comparable between the two pages. For some answers the run did not capture usage data, so it was read back from the chat records after the run and priced with the same price sheet: Claude Opus 5.5 24, DeepSeek V4.1 Flash 11, Kimi K3 19, GPT-5.1 2, Mistral Medium 3 25. Answers with no usage data at all are scored and left out of the cost mean: GPT-5.1 2, GPT-4o 1. Answers that failed for platform reasons (service restarts, provider rate limits) were retried and are not counted as wrong answers. On 3 of the 192 questions the library cannot answer, Gemini 4 Argon kept searching and returned no answer within 30 minutes. Each of the 3 was tried 3 to 8 times and every one ended in a timeout, the last at the 30-minute limit, so they count as a result for the model, not as platform failures. Gemini 4 Argon's cost and time are over the 322 questions it answered and leave out these 3.

Content scores come from LLM judges, not from people. Each answer was judged once, with no repeat and no planted test answers. GPT-5.4 returned no score on 42 of the 2,527 answers (it marked a criterion as not applicable or its reply could not be read) and Claude Sonnet 5.5 on 62; these are left out of the scores and rates. The comparison is with the approved answer only, so content the agent adds from other eligible sources is not checked.

Mistral Medium 3's low content score is not caused by how the direct answer is separated from the supporting arguments: its direct answers are about as long as the others' (median 33 words; Claude Opus 5.5: 35), and the second judge most often labels a missing required claim (44 of its 132 judged answers).

Last updated