Agentic Table-Verbatim Library Answers Benchmark

18 min read

The Verbatim library answers agent answers a questionnaire question with the approved answer from the Answer Library, cited to its library entry, and replies "No matching documents found" when the library has no answer. This page reports how well it does that with each answering model we evaluated. The cases and pass rules are described under How the benchmark works below.

Key findings

  • The models differ on the approved text and on refusing, more than on finding the file. Every model except Mistral Medium 3 and GPT-4o cites the right library file on 89–100% of answerable questions (GPT-4o 82%, Mistral Medium 3 67%). What separates them is whether they return the approved text unchanged (25–93%) and whether they decline the questions the library does not answer (45–100%).

Results

95% Wilson intervals in brackets. Answer coverage is the share of answerable questions that got an answer, answer accuracy the share of those answers with the approved text from the right file, safe refusal the share of questions without a library answer that got the refusal. Answer time: mean per question with a 95% bootstrap interval. Cost per benchmark run: mean list price per answer times 325. Models are ranked by overall pass rate, highest first.

Model

Reasoning effort

Provider

Overall pass rate (325)

Answer coverage: answerable questions answered (133)

Answer accuracy: approved text from the right file, of answered

Approved text from the right file, of all answerable (133)

Right file cited, in an answer or a refusal (133)

Safe refusal (192 questions without a library answer)

Answer time, mean [95% interval]

Answer time, median / 90th percentile

Cost per answer (mean)

Cost per benchmark run

Claude Opus 5.5

low

Anthropic

97% (316/325) [95–99%]

97% (129/133) [93–99%]

96% (124/129) [91–98%]

93% (124/133) [88–96%]

100% (133/133)

100% (192/192) [98–100%]

15 s [15–16 s]

14 s / 21 s

USD 0.22

USD 71

GPT-6 Sol

low

OpenAI

96% (311/325) [93–97%]

96% (128/133) [92–98%]

94% (120/128) [88–97%]

90% (120/133) [84–94%]

94% (125/133)

99% (191/192) [97–100%]

20 s [19–21 s]

19 s / 29 s

USD 0.13

USD 42

GPT-6 Astra

low

OpenAI

95% (310/325) [93–97%]

93% (124/133) [88–96%]

95% (118/124) [90–98%]

89% (118/133) [82–93%]

94% (125/133)

100% (192/192) [98–100%]

17 s [16–17 s]

16 s / 22 s

USD 0.37

USD 121

GPT-6.1 Sol

low

OpenAI

94% (306/325) [91–96%]

89% (118/133) [82–93%]

97% (114/118) [92–99%]

86% (114/133) [79–91%]

95% (127/133)

100% (192/192) [98–100%]

19 s [18–19 s]

17 s / 27 s

USD 0.14

USD 45

GPT-6.1 Terra

low

OpenAI

94% (306/325) [91–96%]

92% (123/133) [87–96%]

93% (114/123) [87–96%]

86% (114/133) [79–91%]

92% (122/133)

100% (192/192) [98–100%]

11 s [11–12 s]

12 s / 14 s

USD 0.04

USD 13

GPT-5.6 Sol

low

OpenAI

91% (296/325) [87–94%]

97% (129/133) [93–99%]

91% (117/129) [84–95%]

88% (117/133) [81–92%]

90% (120/133)

93% (179/192) [89–96%]

23 s [19–27 s]

18 s / 25 s

USD 0.23

USD 75

GPT-5.6 Terra

low

OpenAI

88% (287/325) [84–91%]

95% (127/133) [91–98%]

89% (113/127) [82–93%]

85% (113/133) [78–90%]

89% (118/133)

91% (174/192) [86–94%]

15 s [15–16 s]

14 s / 20 s

USD 0.10

USD 32

GPT-6 Luna

low

OpenAI

87% (283/325) [83–90%]

92% (122/133) [86–95%]

89% (108/122) [82–93%]

81% (108/133) [74–87%]

91% (121/133)

91% (175/192) [86–94%]

13 s [11–15 s]

11 s / 15 s

USD 0.01

USD 2

GPT-5.1

low

OpenAI

85% (275/325) [80–88%]

95% (127/133) [91–98%]

89% (113/127) [82–93%]

85% (113/133) [78–90%]

94% (125/133)

84% (162/192) [79–89%]

47 s [42–52 s]

30 s / 107 s

USD 0.09

USD 30

GPT-5.6 Luna

low

OpenAI

84% (272/325) [79–87%]

95% (126/133) [90–97%]

93% (117/126) [87–96%]

88% (117/133) [81–92%]

90% (120/133)

81% (155/192) [75–86%]

12 s [12–13 s]

12 s / 17 s

USD 0.01

USD 4

DeepSeek V4.1 Flash

default

DeepSeek

83% (271/325) [79–87%]

100% (133/133) [97–100%]

71% (95/133) [63–78%]

71% (95/133) [63–78%]

90% (120/133)

92% (176/192) [87–95%]

30 s [27–33 s]

20 s / 66 s

USD 0.13

USD 44

Claude Sonnet 5.5

low

Anthropic

82% (267/325) [78–86%]

96% (128/133) [92–98%]

60% (77/128) [51–68%]

58% (77/133) [49–66%]

94% (125/133)

99% (190/192) [96–100%]

12 s [12–12 s]

11 s / 16 s

USD 0.11

USD 35

Gemini 4 Argon

low

Google

79% (258/325) [75–83%]

100% (133/133) [97–100%]

50% (66/133) [41–58%]

50% (66/133) [41–58%]

100% (133/133)

100% (192/192) [98–100%]

82 s [73–92 s]

51 s / 177 s

USD 0.22

USD 73

Claude Opus 5

low

Anthropic

75% (245/325) [70–80%]

100% (133/133) [97–100%]

62% (83/133) [54–70%]

62% (83/133) [54–70%]

100% (133/133)

84% (162/192) [79–89%]

15 s [14–15 s]

14 s / 19 s

USD 0.33

USD 107

Kimi K3

low

Moonshot AI

75% (244/325) [70–79%]

100% (133/133) [97–100%]

82% (109/133) [75–88%]

82% (109/133) [75–88%]

96% (128/133)

70% (135/192) [64–76%]

23 s [22–24 s]

21 s / 33 s

USD 0.19

USD 62

Claude Haiku 5.5

low

Anthropic

75% (243/325) [70–79%]

100% (133/133) [97–100%]

47% (62/133) [38–55%]

47% (62/133) [38–55%]

90% (120/133)

94% (181/192) [90–97%]

19 s [18–20 s]

18 s / 26 s

USD 0.02

USD 6

GPT-4o (production default)

not applicable

OpenAI

65% (210/325) [59–70%]

99% (132/133) [96–100%]

35% (46/132) [27–43%]

35% (46/133) [27–43%]

82% (109/133)

85% (164/192) [80–90%]

28 s [25–30 s]

18 s / 53 s

USD 0.08

USD 25

Claude Sonnet 5

low

Anthropic

40% (130/325) [35–45%]

100% (133/133) [97–100%]

25% (33/133) [18–33%]

25% (33/133) [18–33%]

91% (121/133)

51% (97/192) [44–58%]

13 s [12–13 s]

12 s / 16 s

USD 0.12

USD 39

Mistral Medium 3

not applicable

Mistral

37% (119/325) [32–42%]

98% (130/133) [94–99%]

25% (33/130) [19–33%]

25% (33/133) [18–33%]

67% (89/133)

45% (86/192) [38–52%]

12 s [11–12 s]

10 s / 21 s

USD 0.01

USD 3

Per 1,000 questions

In the benchmark mix, 41% of questions have an approved library answer (133 of 325) and 59% do not (192 of 325). Scaled to 1,000 questions in this mix, rounded so that each row adds up to 1,000:

Model

Reasoning effort

Provider

Right outcome

Answerable, other text or wrong file

Answerable, declined

No library answer, answered anyway

Claude Opus 5.5

low

Anthropic

about 972

about 16

about 12

about 0

GPT-6 Sol

low

OpenAI

about 957

about 25

about 15

about 3

GPT-6 Astra

low

OpenAI

about 954

about 18

about 28

about 0

GPT-6.1 Sol

low

OpenAI

about 942

about 12

about 46

about 0

GPT-6.1 Terra

low

OpenAI

about 941

about 28

about 31

about 0

GPT-5.6 Sol

low

OpenAI

about 911

about 37

about 12

about 40

GPT-5.6 Terra

low

OpenAI

about 883

about 43

about 19

about 55

GPT-6 Luna

low

OpenAI

about 871

about 43

about 34

about 52

GPT-5.1

low

OpenAI

about 846

about 43

about 19

about 92

GPT-5.6 Luna

low

OpenAI

about 837

about 28

about 21

about 114

DeepSeek V4.1 Flash

default

DeepSeek

about 834

about 117

about 0

about 49

Claude Sonnet 5.5

low

Anthropic

about 822

about 157

about 15

about 6

Gemini 4 Argon

low

Google

about 794

about 206

about 0

about 0

Claude Opus 5

low

Anthropic

about 754

about 154

about 0

about 92

Kimi K3

low

Moonshot AI

about 751

about 74

about 0

about 175

Claude Haiku 5.5

low

Anthropic

about 748

about 218

about 0

about 34

GPT-4o (production default)

not applicable

OpenAI

about 646

about 265

about 3

about 86

Claude Sonnet 5

low

Anthropic

about 400

about 308

about 0

about 292

Mistral Medium 3

not applicable

Mistral

about 366

about 299

about 9

about 326

The mix is set by the benchmark, not by production. A questionnaire where most questions have a library answer weights the first columns more; one with many new questions weights the last column more.

Model notes

  • Claude Opus 5.5, low: the right outcome on 97% of questions (316/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 93% of answerable questions (answer accuracy 96%), refusal on 100% of questions without a library answer. Mean answer time 15 s, USD 71 per benchmark run. The most accurate model we measured. Against GPT-6 Sol: no significant difference (p = 0.302), at about 1.7 times its cost per answer.

  • GPT-6 Sol, low: the right outcome on 96% of questions (311/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 90% of answerable questions (answer accuracy 94%), refusal on 99% of questions without a library answer. Mean answer time 20 s, USD 42 per benchmark run. Level with GPT-6 Astra, behind only Claude Opus 5.5. Against Claude Opus 5.5: no significant difference (p = 0.302), at about 58% of its cost per answer.

  • GPT-6 Astra, low: the right outcome on 95% of questions (310/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 89% of answerable questions (answer accuracy 95%), refusal on 100% of questions without a library answer. Mean answer time 17 s, USD 121 per benchmark run. Close to GPT-6 Sol on accuracy, at about 2.9 times its cost per answer: the most expensive model we measured.

  • GPT-6.1 Sol, low: the right outcome on 94% of questions (306/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 86% of answerable questions (answer accuracy 97%), refusal on 100% of questions without a library answer. Mean answer time 19 s, USD 45 per benchmark run. Refuses every question without a library answer, but also declines 15 answerable questions that the library does answer.

  • GPT-6.1 Terra, low: the right outcome on 94% of questions (306/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 86% of answerable questions (answer accuracy 93%), refusal on 100% of questions without a library answer. Mean answer time 11 s, USD 13 per benchmark run. Against Claude Opus 5.5: significantly lower (p = 0.021), at about 19% of its cost per answer.

  • GPT-5.6 Sol, low: the right outcome on 91% of questions (296/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 88% of answerable questions (answer accuracy 91%), refusal on 93% of questions without a library answer. Mean answer time 23 s, USD 75 per benchmark run. Answers 13 of the 192 questions without a library answer instead of refusing. Against GPT-6.1 Sol: no significant difference (p = 0.064), at a higher cost per answer.

  • GPT-5.6 Terra, low: the right outcome on 88% of questions (287/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 85% of answerable questions (answer accuracy 89%), refusal on 91% of questions without a library answer. Mean answer time 15 s, USD 32 per benchmark run. Against GPT-5.6 Sol: no significant difference (p = 0.136), at about 43% of its cost per answer.

  • GPT-6 Luna, low: the right outcome on 87% of questions (283/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 81% of answerable questions (answer accuracy 89%), refusal on 91% of questions without a library answer. Mean answer time 13 s, USD 2 per benchmark run. The least expensive model we measured (USD 0.006 per answer), but it answers 17 of the 192 questions without a library answer instead of refusing.

  • GPT-5.1, low: the right outcome on 85% of questions (275/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 85% of answerable questions (answer accuracy 89%), refusal on 84% of questions without a library answer. Mean answer time 47 s, USD 30 per benchmark run. The option when inference must run in Switzerland. It answers 30 of the 192 questions without a library answer instead of refusing. Its answer times are inflated by a shared token limit.

  • GPT-5.6 Luna, low: the right outcome on 84% of questions (272/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 88% of answerable questions (answer accuracy 93%), refusal on 81% of questions without a library answer. Mean answer time 12 s, USD 4 per benchmark run. About USD 0.01 per answer, but it answers 37 of the 192 questions without a library answer instead of refusing.

  • DeepSeek V4.1 Flash, default: the right outcome on 83% of questions (271/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 71% of answerable questions (answer accuracy 71%), refusal on 92% of questions without a library answer. Mean answer time 30 s, USD 44 per benchmark run. Close to GPT-5.1 overall: fewer approved texts, more correct refusals. Runs at its default reasoning, not at low.

  • Claude Sonnet 5.5, low: the right outcome on 82% of questions (267/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 58% of answerable questions (answer accuracy 60%), refusal on 99% of questions without a library answer. Mean answer time 12 s, USD 35 per benchmark run. Refuses correctly and cites the right file on 94% of answerable questions, but often returns text that differs from the approved answer.

  • Gemini 4 Argon, low: the right outcome on 79% of questions (258/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 50% of answerable questions (answer accuracy 50%), refusal on 100% of questions without a library answer. Mean answer time 82 s, USD 73 per benchmark run. Cites the right file on every answerable question, but returns the approved text unchanged on only 50% of them. The slowest model we measured.

  • Claude Opus 5, low: the right outcome on 75% of questions (245/325), significantly higher than GPT-4o (65%, p = 0.001). Approved text from the right file on 62% of answerable questions (answer accuracy 62%), refusal on 84% of questions without a library answer. Mean answer time 15 s, USD 107 per benchmark run. Answers 30 questions without a library answer, at USD 0.33 per answer. Not recommended.

  • Kimi K3, low: the right outcome on 75% of questions (244/325), significantly higher than GPT-4o (65%, p = 0.002). Approved text from the right file on 82% of answerable questions (answer accuracy 82%), refusal on 70% of questions without a library answer. Mean answer time 23 s, USD 62 per benchmark run. Answers 30% of the questions without a library answer. Not recommended.

  • Claude Haiku 5.5, low: the right outcome on 75% of questions (243/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 47% of answerable questions (answer accuracy 47%), refusal on 94% of questions without a library answer. Mean answer time 19 s, USD 6 per benchmark run. Inexpensive and refuses reliably, but changes the approved text on about half of the answerable questions.

  • GPT-4o, no reasoning (production default): the right outcome on 65% of questions (210/325). Approved text from the right file on 35% of answerable questions (answer accuracy 35%), refusal on 85% of questions without a library answer. Mean answer time 28 s, USD 25 per benchmark run. The production default. It cites the right file on 82% of answerable questions but changes the approved text on most of them.

  • Claude Sonnet 5, low: the right outcome on 40% of questions (130/325), significantly lower than GPT-4o (65%, p < 0.001). Approved text from the right file on 25% of answerable questions (answer accuracy 25%), refusal on 51% of questions without a library answer. Mean answer time 13 s, USD 39 per benchmark run. Answers 49% of the questions without a library answer and returns the approved text on 25% of the answerable ones. Not suitable.

  • Mistral Medium 3, no reasoning: the right outcome on 37% of questions (119/325), significantly lower than GPT-4o (65%, p < 0.001). Approved text from the right file on 25% of answerable questions (answer accuracy 25%), refusal on 45% of questions without a library answer. Mean answer time 12 s, USD 3 per benchmark run. Among the least expensive models (about USD 0.01 per answer), but cites the right file on only 67% of answerable questions. Not suitable.

Statistical tests

Paired exact McNemar tests against GPT-4o, the production default, on the same questions:

Model

Reasoning effort

Questions with the right outcome

Claude Opus 5.5

low

significantly more (passes 110 that GPT-4o fails, fails 4; p < 0.001)

GPT-6 Sol

low

significantly more (passes 103 that GPT-4o fails, fails 2; p < 0.001)

GPT-6 Astra

low

significantly more (passes 103 that GPT-4o fails, fails 3; p < 0.001)

GPT-6.1 Sol

low

significantly more (passes 99 that GPT-4o fails, fails 3; p < 0.001)

GPT-6.1 Terra

low

significantly more (passes 101 that GPT-4o fails, fails 5; p < 0.001)

GPT-5.6 Sol

low

significantly more (passes 96 that GPT-4o fails, fails 10; p < 0.001)

GPT-5.6 Terra

low

significantly more (passes 94 that GPT-4o fails, fails 17; p < 0.001)

GPT-6 Luna

low

significantly more (passes 84 that GPT-4o fails, fails 11; p < 0.001)

GPT-5.1

low

significantly more (passes 90 that GPT-4o fails, fails 25; p < 0.001)

GPT-5.6 Luna

low

significantly more (passes 85 that GPT-4o fails, fails 23; p < 0.001)

DeepSeek V4.1 Flash

default

significantly more (passes 81 that GPT-4o fails, fails 20; p < 0.001)

Claude Sonnet 5.5

low

significantly more (passes 77 that GPT-4o fails, fails 20; p < 0.001)

Gemini 4 Argon

low

significantly more (passes 72 that GPT-4o fails, fails 24; p < 0.001)

Claude Opus 5

low

significantly more (passes 73 that GPT-4o fails, fails 38; p = 0.001)

Kimi K3

low

significantly more (passes 76 that GPT-4o fails, fails 42; p = 0.002)

Claude Haiku 5.5

low

significantly more (passes 63 that GPT-4o fails, fails 30; p < 0.001)

Claude Sonnet 5

low

significantly fewer (passes 27 that GPT-4o fails, fails 107; p < 0.001)

Mistral Medium 3

not applicable

significantly fewer (passes 24 that GPT-4o fails, fails 115; p < 0.001)

Paired exact McNemar tests against Claude Opus 5.5, low, the most accurate model:

Model

Reasoning effort

Questions with the right outcome

GPT-6 Sol

low

no significant difference (5 vs 10; p = 0.302)

GPT-6 Astra

low

no significant difference (6 vs 12; p = 0.238)

GPT-6.1 Sol

low

significantly lower (4 vs 14; p = 0.031)

GPT-6.1 Terra

low

significantly lower (3 vs 13; p = 0.021)

GPT-5.6 Sol

low

significantly lower (4 vs 24; p < 0.001)

GPT-5.6 Terra

low

significantly lower (4 vs 33; p < 0.001)

GPT-6 Luna

low

significantly lower (4 vs 37; p < 0.001)

GPT-5.1

low

significantly lower (4 vs 45; p < 0.001)

GPT-5.6 Luna

low

significantly lower (4 vs 48; p < 0.001)

DeepSeek V4.1 Flash

default

significantly lower (4 vs 49; p < 0.001)

Claude Sonnet 5.5

low

significantly lower (2 vs 51; p < 0.001)

Gemini 4 Argon

low

significantly lower (6 vs 64; p < 0.001)

Claude Opus 5

low

significantly lower (4 vs 75; p < 0.001)

Kimi K3

low

significantly lower (6 vs 78; p < 0.001)

Claude Haiku 5.5

low

significantly lower (2 vs 75; p < 0.001)

GPT-4o (production default)

not applicable

significantly lower (4 vs 110; p < 0.001)

Claude Sonnet 5

low

significantly lower (0 vs 186; p < 0.001)

Mistral Medium 3

not applicable

significantly lower (2 vs 199; p < 0.001)

How the benchmark works

The benchmark is a frozen set of questionnaire questions run against a frozen Answer Library scope. Each question is sent on its own, and the agent's answer and every cited source are recorded.

Part

Cases

What it tests

Answerable: the library holds an approved answer and its source

133

Whether the agent finds the entry and returns the approved answer unchanged

Without a library answer: realistic, in-scope questions the library does not answer

192

Whether the agent refuses instead of improvising

Total

325

Pass rules. Scoring is deterministic; no model judges the answers.

  • Answerable case: passes when the answer cites the expected library entry and its text equals the approved answer, ignoring letter case. Before comparing, the scorer removes what the agent adds around the answer (citation markers, collapsible detail blocks, the outdated-entry flag) and normalizes whitespace and surrounding quotes. Partial, similar or reworded answers fail.

  • Duplicate entries: the library holds some answers more than once. Citing a different entry passes only when that entry's stored question and answer are the same as the expected ones.

  • Case without a library answer: passes only when the answer begins with "No matching documents". Any other answer fails.

The overall pass rate depends on the mix of 133 answerable to 192 unanswerable cases; a different mix changes it even when the agent behaves the same. Answer coverage, answer accuracy and safe refusal do not depend on the mix. This is a controlled benchmark: it does not measure performance under other topics, languages, phrasings or library content.

Setup

Every model answered the same 325 scored cases (133 with an approved answer in the library, 192 without) through the same agent configuration, the same prompt and the same frozen library scope; only the model and its reasoning level change between rows.

GPT-4o, the production default, ran on 29 October with the same library, configuration, questions and release as the other models. It has no reasoning setting. Gemini 4 Argon ran on 4 November, GPT-6.1 Terra on 12 November, and GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, GPT-6 Sol, GPT-6 Astra, GPT-6 Luna and Claude Haiku 5.5 on 13 November, with the same library, configuration and questions. Reasoning was set to low for every model that supports it. Mistral Medium 3 has no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model. Every model is compared with GPT-4o. A repeat run of 6 of the models (GPT-5.1, GPT-6.1 Sol, Claude Opus 5.5, Claude Opus 5, Kimi K3 and Mistral Medium 3) on the same configuration moved the overall pass rate by at most 1.5 points, which is the run-to-run noise to expect.

Hosting. GPT-5.1 runs on Azure in Switzerland, GPT-4o on Azure OpenAI and Mistral Medium 3 and Gemini 4 Argon on Vertex AI in the EU. Anthropic, Azure OpenAI global deployments (GPT-6.1 Terra, GPT-6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna), OpenAI through LiteLLM (GPT-6.1 Sol) and Together AI (Kimi K3 and DeepSeek V4.1 Flash) process globally. Which providers a client can use depends on its contract.

Configurations

Model

Model deployment

Reasoning effort

Claude Opus 5.5

Claude Opus 5.5, Anthropic through LiteLLM

low

GPT-6 Sol

GPT-6 Sol (2026-09-22), Azure OpenAI

low

GPT-6 Astra

GPT-6 Astra (2026-09-03), Azure OpenAI

low

GPT-6.1 Sol

GPT-6.1 Sol, OpenAI through LiteLLM

low

GPT-6.1 Terra

GPT-6.1 Terra (2026-10-26), Azure OpenAI

low

GPT-5.6 Sol

GPT-5.6 Sol (2026-07-09), Azure OpenAI

low

GPT-5.6 Terra

GPT-5.6 Terra (2026-07-09), Azure OpenAI

low

GPT-6 Luna

GPT-6 Luna (2026-09-22), Azure OpenAI

low

GPT-5.1

GPT-5.1 (2025-11-13), Azure, Swiss inference

low

GPT-5.6 Luna

GPT-5.6 Luna (2026-07-09), Azure OpenAI

low

DeepSeek V4.1 Flash

DeepSeek V4.1 Flash, Together AI through LiteLLM

default (Together AI rejects the reasoning setting the platform sends, so it is removed and the model runs at its default)

Claude Sonnet 5.5

Claude Sonnet 5.5, Anthropic through LiteLLM

low

Gemini 4 Argon

Gemini 4 Argon, Google Vertex AI EU through LiteLLM

low

Claude Opus 5

Claude Opus 5, Anthropic through LiteLLM

low

Kimi K3

Kimi K3, Together AI through LiteLLM

low

Claude Haiku 5.5

Claude Haiku 5.5, Anthropic through LiteLLM

low

GPT-4o (production default)

GPT-4o (2024-11-20), Azure OpenAI

not applicable (not a reasoning model)

Claude Sonnet 5

Claude Sonnet 5, Anthropic through LiteLLM

low

Mistral Medium 3

Mistral Medium 3, Google Vertex AI EU through LiteLLM

not applicable (no reasoning setting)

Notes

Times are measured per answer across all 325 questions. Costs are list prices for the agent's calls, from the usage the platform records or reads back afterwards, not invoices. For some answers the run did not capture usage data, so it was read back from the chat records after the run and priced with the same price sheet: GPT-4o 253, GPT-5.1 2, Claude Opus 5.5 1, Claude Opus 5 7, Claude Sonnet 5.5 1, Kimi K3 2, DeepSeek V4.1 Flash 1. Answers with no usage data at all are scored and left out of the cost mean: GPT-4o 4. Answers that failed for platform reasons (service restarts, provider rate limits) were retried and are not counted as wrong answers. Every model returned a response to all 325 questions.

Last updated