Agentic Table-Verbatim Library Answers Benchmark
18 min read
The Verbatim library answers agent answers a questionnaire question with the approved answer from the Answer Library, cited to its library entry, and replies "No matching documents found" when the library has no answer. This page reports how well it does that with each answering model we evaluated. The cases and pass rules are described under How the benchmark works below.
Key findings
The models differ on the approved text and on refusing, more than on finding the file. Every model except Mistral Medium 3 and GPT-4o cites the right library file on 89–100% of answerable questions (GPT-4o 82%, Mistral Medium 3 67%). What separates them is whether they return the approved text unchanged (25–93%) and whether they decline the questions the library does not answer (45–100%).
Results
95% Wilson intervals in brackets. Answer coverage is the share of answerable questions that got an answer, answer accuracy the share of those answers with the approved text from the right file, safe refusal the share of questions without a library answer that got the refusal. Answer time: mean per question with a 95% bootstrap interval. Cost per benchmark run: mean list price per answer times 325. Models are ranked by overall pass rate, highest first.
Model | Reasoning effort | Provider | Overall pass rate (325) | Answer coverage: answerable questions answered (133) | Answer accuracy: approved text from the right file, of answered | Approved text from the right file, of all answerable (133) | Right file cited, in an answer or a refusal (133) | Safe refusal (192 questions without a library answer) | Answer time, mean [95% interval] | Answer time, median / 90th percentile | Cost per answer (mean) | Cost per benchmark run |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Claude Opus 5.5 | low | Anthropic | 97% (316/325) [95–99%] | 97% (129/133) [93–99%] | 96% (124/129) [91–98%] | 93% (124/133) [88–96%] | 100% (133/133) | 100% (192/192) [98–100%] | 15 s [15–16 s] | 14 s / 21 s | USD 0.22 | USD 71 |
GPT-6 Sol | low | OpenAI | 96% (311/325) [93–97%] | 96% (128/133) [92–98%] | 94% (120/128) [88–97%] | 90% (120/133) [84–94%] | 94% (125/133) | 99% (191/192) [97–100%] | 20 s [19–21 s] | 19 s / 29 s | USD 0.13 | USD 42 |
GPT-6 Astra | low | OpenAI | 95% (310/325) [93–97%] | 93% (124/133) [88–96%] | 95% (118/124) [90–98%] | 89% (118/133) [82–93%] | 94% (125/133) | 100% (192/192) [98–100%] | 17 s [16–17 s] | 16 s / 22 s | USD 0.37 | USD 121 |
GPT-6.1 Sol | low | OpenAI | 94% (306/325) [91–96%] | 89% (118/133) [82–93%] | 97% (114/118) [92–99%] | 86% (114/133) [79–91%] | 95% (127/133) | 100% (192/192) [98–100%] | 19 s [18–19 s] | 17 s / 27 s | USD 0.14 | USD 45 |
GPT-6.1 Terra | low | OpenAI | 94% (306/325) [91–96%] | 92% (123/133) [87–96%] | 93% (114/123) [87–96%] | 86% (114/133) [79–91%] | 92% (122/133) | 100% (192/192) [98–100%] | 11 s [11–12 s] | 12 s / 14 s | USD 0.04 | USD 13 |
GPT-5.6 Sol | low | OpenAI | 91% (296/325) [87–94%] | 97% (129/133) [93–99%] | 91% (117/129) [84–95%] | 88% (117/133) [81–92%] | 90% (120/133) | 93% (179/192) [89–96%] | 23 s [19–27 s] | 18 s / 25 s | USD 0.23 | USD 75 |
GPT-5.6 Terra | low | OpenAI | 88% (287/325) [84–91%] | 95% (127/133) [91–98%] | 89% (113/127) [82–93%] | 85% (113/133) [78–90%] | 89% (118/133) | 91% (174/192) [86–94%] | 15 s [15–16 s] | 14 s / 20 s | USD 0.10 | USD 32 |
GPT-6 Luna | low | OpenAI | 87% (283/325) [83–90%] | 92% (122/133) [86–95%] | 89% (108/122) [82–93%] | 81% (108/133) [74–87%] | 91% (121/133) | 91% (175/192) [86–94%] | 13 s [11–15 s] | 11 s / 15 s | USD 0.01 | USD 2 |
GPT-5.1 | low | OpenAI | 85% (275/325) [80–88%] | 95% (127/133) [91–98%] | 89% (113/127) [82–93%] | 85% (113/133) [78–90%] | 94% (125/133) | 84% (162/192) [79–89%] | 47 s [42–52 s] | 30 s / 107 s | USD 0.09 | USD 30 |
GPT-5.6 Luna | low | OpenAI | 84% (272/325) [79–87%] | 95% (126/133) [90–97%] | 93% (117/126) [87–96%] | 88% (117/133) [81–92%] | 90% (120/133) | 81% (155/192) [75–86%] | 12 s [12–13 s] | 12 s / 17 s | USD 0.01 | USD 4 |
DeepSeek V4.1 Flash | default | DeepSeek | 83% (271/325) [79–87%] | 100% (133/133) [97–100%] | 71% (95/133) [63–78%] | 71% (95/133) [63–78%] | 90% (120/133) | 92% (176/192) [87–95%] | 30 s [27–33 s] | 20 s / 66 s | USD 0.13 | USD 44 |
Claude Sonnet 5.5 | low | Anthropic | 82% (267/325) [78–86%] | 96% (128/133) [92–98%] | 60% (77/128) [51–68%] | 58% (77/133) [49–66%] | 94% (125/133) | 99% (190/192) [96–100%] | 12 s [12–12 s] | 11 s / 16 s | USD 0.11 | USD 35 |
Gemini 4 Argon | low | 79% (258/325) [75–83%] | 100% (133/133) [97–100%] | 50% (66/133) [41–58%] | 50% (66/133) [41–58%] | 100% (133/133) | 100% (192/192) [98–100%] | 82 s [73–92 s] | 51 s / 177 s | USD 0.22 | USD 73 | |
Claude Opus 5 | low | Anthropic | 75% (245/325) [70–80%] | 100% (133/133) [97–100%] | 62% (83/133) [54–70%] | 62% (83/133) [54–70%] | 100% (133/133) | 84% (162/192) [79–89%] | 15 s [14–15 s] | 14 s / 19 s | USD 0.33 | USD 107 |
Kimi K3 | low | Moonshot AI | 75% (244/325) [70–79%] | 100% (133/133) [97–100%] | 82% (109/133) [75–88%] | 82% (109/133) [75–88%] | 96% (128/133) | 70% (135/192) [64–76%] | 23 s [22–24 s] | 21 s / 33 s | USD 0.19 | USD 62 |
Claude Haiku 5.5 | low | Anthropic | 75% (243/325) [70–79%] | 100% (133/133) [97–100%] | 47% (62/133) [38–55%] | 47% (62/133) [38–55%] | 90% (120/133) | 94% (181/192) [90–97%] | 19 s [18–20 s] | 18 s / 26 s | USD 0.02 | USD 6 |
GPT-4o (production default) | not applicable | OpenAI | 65% (210/325) [59–70%] | 99% (132/133) [96–100%] | 35% (46/132) [27–43%] | 35% (46/133) [27–43%] | 82% (109/133) | 85% (164/192) [80–90%] | 28 s [25–30 s] | 18 s / 53 s | USD 0.08 | USD 25 |
Claude Sonnet 5 | low | Anthropic | 40% (130/325) [35–45%] | 100% (133/133) [97–100%] | 25% (33/133) [18–33%] | 25% (33/133) [18–33%] | 91% (121/133) | 51% (97/192) [44–58%] | 13 s [12–13 s] | 12 s / 16 s | USD 0.12 | USD 39 |
Mistral Medium 3 | not applicable | Mistral | 37% (119/325) [32–42%] | 98% (130/133) [94–99%] | 25% (33/130) [19–33%] | 25% (33/133) [18–33%] | 67% (89/133) | 45% (86/192) [38–52%] | 12 s [11–12 s] | 10 s / 21 s | USD 0.01 | USD 3 |
Per 1,000 questions
In the benchmark mix, 41% of questions have an approved library answer (133 of 325) and 59% do not (192 of 325). Scaled to 1,000 questions in this mix, rounded so that each row adds up to 1,000:
Model | Reasoning effort | Provider | Right outcome | Answerable, other text or wrong file | Answerable, declined | No library answer, answered anyway |
|---|---|---|---|---|---|---|
Claude Opus 5.5 | low | Anthropic | about 972 | about 16 | about 12 | about 0 |
GPT-6 Sol | low | OpenAI | about 957 | about 25 | about 15 | about 3 |
GPT-6 Astra | low | OpenAI | about 954 | about 18 | about 28 | about 0 |
GPT-6.1 Sol | low | OpenAI | about 942 | about 12 | about 46 | about 0 |
GPT-6.1 Terra | low | OpenAI | about 941 | about 28 | about 31 | about 0 |
GPT-5.6 Sol | low | OpenAI | about 911 | about 37 | about 12 | about 40 |
GPT-5.6 Terra | low | OpenAI | about 883 | about 43 | about 19 | about 55 |
GPT-6 Luna | low | OpenAI | about 871 | about 43 | about 34 | about 52 |
GPT-5.1 | low | OpenAI | about 846 | about 43 | about 19 | about 92 |
GPT-5.6 Luna | low | OpenAI | about 837 | about 28 | about 21 | about 114 |
DeepSeek V4.1 Flash | default | DeepSeek | about 834 | about 117 | about 0 | about 49 |
Claude Sonnet 5.5 | low | Anthropic | about 822 | about 157 | about 15 | about 6 |
Gemini 4 Argon | low | about 794 | about 206 | about 0 | about 0 | |
Claude Opus 5 | low | Anthropic | about 754 | about 154 | about 0 | about 92 |
Kimi K3 | low | Moonshot AI | about 751 | about 74 | about 0 | about 175 |
Claude Haiku 5.5 | low | Anthropic | about 748 | about 218 | about 0 | about 34 |
GPT-4o (production default) | not applicable | OpenAI | about 646 | about 265 | about 3 | about 86 |
Claude Sonnet 5 | low | Anthropic | about 400 | about 308 | about 0 | about 292 |
Mistral Medium 3 | not applicable | Mistral | about 366 | about 299 | about 9 | about 326 |
The mix is set by the benchmark, not by production. A questionnaire where most questions have a library answer weights the first columns more; one with many new questions weights the last column more.
Model notes
Claude Opus 5.5, low: the right outcome on 97% of questions (316/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 93% of answerable questions (answer accuracy 96%), refusal on 100% of questions without a library answer. Mean answer time 15 s, USD 71 per benchmark run. The most accurate model we measured. Against GPT-6 Sol: no significant difference (p = 0.302), at about 1.7 times its cost per answer.
GPT-6 Sol, low: the right outcome on 96% of questions (311/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 90% of answerable questions (answer accuracy 94%), refusal on 99% of questions without a library answer. Mean answer time 20 s, USD 42 per benchmark run. Level with GPT-6 Astra, behind only Claude Opus 5.5. Against Claude Opus 5.5: no significant difference (p = 0.302), at about 58% of its cost per answer.
GPT-6 Astra, low: the right outcome on 95% of questions (310/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 89% of answerable questions (answer accuracy 95%), refusal on 100% of questions without a library answer. Mean answer time 17 s, USD 121 per benchmark run. Close to GPT-6 Sol on accuracy, at about 2.9 times its cost per answer: the most expensive model we measured.
GPT-6.1 Sol, low: the right outcome on 94% of questions (306/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 86% of answerable questions (answer accuracy 97%), refusal on 100% of questions without a library answer. Mean answer time 19 s, USD 45 per benchmark run. Refuses every question without a library answer, but also declines 15 answerable questions that the library does answer.
GPT-6.1 Terra, low: the right outcome on 94% of questions (306/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 86% of answerable questions (answer accuracy 93%), refusal on 100% of questions without a library answer. Mean answer time 11 s, USD 13 per benchmark run. Against Claude Opus 5.5: significantly lower (p = 0.021), at about 19% of its cost per answer.
GPT-5.6 Sol, low: the right outcome on 91% of questions (296/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 88% of answerable questions (answer accuracy 91%), refusal on 93% of questions without a library answer. Mean answer time 23 s, USD 75 per benchmark run. Answers 13 of the 192 questions without a library answer instead of refusing. Against GPT-6.1 Sol: no significant difference (p = 0.064), at a higher cost per answer.
GPT-5.6 Terra, low: the right outcome on 88% of questions (287/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 85% of answerable questions (answer accuracy 89%), refusal on 91% of questions without a library answer. Mean answer time 15 s, USD 32 per benchmark run. Against GPT-5.6 Sol: no significant difference (p = 0.136), at about 43% of its cost per answer.
GPT-6 Luna, low: the right outcome on 87% of questions (283/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 81% of answerable questions (answer accuracy 89%), refusal on 91% of questions without a library answer. Mean answer time 13 s, USD 2 per benchmark run. The least expensive model we measured (USD 0.006 per answer), but it answers 17 of the 192 questions without a library answer instead of refusing.
GPT-5.1, low: the right outcome on 85% of questions (275/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 85% of answerable questions (answer accuracy 89%), refusal on 84% of questions without a library answer. Mean answer time 47 s, USD 30 per benchmark run. The option when inference must run in Switzerland. It answers 30 of the 192 questions without a library answer instead of refusing. Its answer times are inflated by a shared token limit.
GPT-5.6 Luna, low: the right outcome on 84% of questions (272/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 88% of answerable questions (answer accuracy 93%), refusal on 81% of questions without a library answer. Mean answer time 12 s, USD 4 per benchmark run. About USD 0.01 per answer, but it answers 37 of the 192 questions without a library answer instead of refusing.
DeepSeek V4.1 Flash, default: the right outcome on 83% of questions (271/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 71% of answerable questions (answer accuracy 71%), refusal on 92% of questions without a library answer. Mean answer time 30 s, USD 44 per benchmark run. Close to GPT-5.1 overall: fewer approved texts, more correct refusals. Runs at its default reasoning, not at low.
Claude Sonnet 5.5, low: the right outcome on 82% of questions (267/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 58% of answerable questions (answer accuracy 60%), refusal on 99% of questions without a library answer. Mean answer time 12 s, USD 35 per benchmark run. Refuses correctly and cites the right file on 94% of answerable questions, but often returns text that differs from the approved answer.
Gemini 4 Argon, low: the right outcome on 79% of questions (258/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 50% of answerable questions (answer accuracy 50%), refusal on 100% of questions without a library answer. Mean answer time 82 s, USD 73 per benchmark run. Cites the right file on every answerable question, but returns the approved text unchanged on only 50% of them. The slowest model we measured.
Claude Opus 5, low: the right outcome on 75% of questions (245/325), significantly higher than GPT-4o (65%, p = 0.001). Approved text from the right file on 62% of answerable questions (answer accuracy 62%), refusal on 84% of questions without a library answer. Mean answer time 15 s, USD 107 per benchmark run. Answers 30 questions without a library answer, at USD 0.33 per answer. Not recommended.
Kimi K3, low: the right outcome on 75% of questions (244/325), significantly higher than GPT-4o (65%, p = 0.002). Approved text from the right file on 82% of answerable questions (answer accuracy 82%), refusal on 70% of questions without a library answer. Mean answer time 23 s, USD 62 per benchmark run. Answers 30% of the questions without a library answer. Not recommended.
Claude Haiku 5.5, low: the right outcome on 75% of questions (243/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 47% of answerable questions (answer accuracy 47%), refusal on 94% of questions without a library answer. Mean answer time 19 s, USD 6 per benchmark run. Inexpensive and refuses reliably, but changes the approved text on about half of the answerable questions.
GPT-4o, no reasoning (production default): the right outcome on 65% of questions (210/325). Approved text from the right file on 35% of answerable questions (answer accuracy 35%), refusal on 85% of questions without a library answer. Mean answer time 28 s, USD 25 per benchmark run. The production default. It cites the right file on 82% of answerable questions but changes the approved text on most of them.
Claude Sonnet 5, low: the right outcome on 40% of questions (130/325), significantly lower than GPT-4o (65%, p < 0.001). Approved text from the right file on 25% of answerable questions (answer accuracy 25%), refusal on 51% of questions without a library answer. Mean answer time 13 s, USD 39 per benchmark run. Answers 49% of the questions without a library answer and returns the approved text on 25% of the answerable ones. Not suitable.
Mistral Medium 3, no reasoning: the right outcome on 37% of questions (119/325), significantly lower than GPT-4o (65%, p < 0.001). Approved text from the right file on 25% of answerable questions (answer accuracy 25%), refusal on 45% of questions without a library answer. Mean answer time 12 s, USD 3 per benchmark run. Among the least expensive models (about USD 0.01 per answer), but cites the right file on only 67% of answerable questions. Not suitable.
Statistical tests
Paired exact McNemar tests against GPT-4o, the production default, on the same questions:
Model | Reasoning effort | Questions with the right outcome |
|---|---|---|
Claude Opus 5.5 | low | significantly more (passes 110 that GPT-4o fails, fails 4; p < 0.001) |
GPT-6 Sol | low | significantly more (passes 103 that GPT-4o fails, fails 2; p < 0.001) |
GPT-6 Astra | low | significantly more (passes 103 that GPT-4o fails, fails 3; p < 0.001) |
GPT-6.1 Sol | low | significantly more (passes 99 that GPT-4o fails, fails 3; p < 0.001) |
GPT-6.1 Terra | low | significantly more (passes 101 that GPT-4o fails, fails 5; p < 0.001) |
GPT-5.6 Sol | low | significantly more (passes 96 that GPT-4o fails, fails 10; p < 0.001) |
GPT-5.6 Terra | low | significantly more (passes 94 that GPT-4o fails, fails 17; p < 0.001) |
GPT-6 Luna | low | significantly more (passes 84 that GPT-4o fails, fails 11; p < 0.001) |
GPT-5.1 | low | significantly more (passes 90 that GPT-4o fails, fails 25; p < 0.001) |
GPT-5.6 Luna | low | significantly more (passes 85 that GPT-4o fails, fails 23; p < 0.001) |
DeepSeek V4.1 Flash | default | significantly more (passes 81 that GPT-4o fails, fails 20; p < 0.001) |
Claude Sonnet 5.5 | low | significantly more (passes 77 that GPT-4o fails, fails 20; p < 0.001) |
Gemini 4 Argon | low | significantly more (passes 72 that GPT-4o fails, fails 24; p < 0.001) |
Claude Opus 5 | low | significantly more (passes 73 that GPT-4o fails, fails 38; p = 0.001) |
Kimi K3 | low | significantly more (passes 76 that GPT-4o fails, fails 42; p = 0.002) |
Claude Haiku 5.5 | low | significantly more (passes 63 that GPT-4o fails, fails 30; p < 0.001) |
Claude Sonnet 5 | low | significantly fewer (passes 27 that GPT-4o fails, fails 107; p < 0.001) |
Mistral Medium 3 | not applicable | significantly fewer (passes 24 that GPT-4o fails, fails 115; p < 0.001) |
Paired exact McNemar tests against Claude Opus 5.5, low, the most accurate model:
Model | Reasoning effort | Questions with the right outcome |
|---|---|---|
GPT-6 Sol | low | no significant difference (5 vs 10; p = 0.302) |
GPT-6 Astra | low | no significant difference (6 vs 12; p = 0.238) |
GPT-6.1 Sol | low | significantly lower (4 vs 14; p = 0.031) |
GPT-6.1 Terra | low | significantly lower (3 vs 13; p = 0.021) |
GPT-5.6 Sol | low | significantly lower (4 vs 24; p < 0.001) |
GPT-5.6 Terra | low | significantly lower (4 vs 33; p < 0.001) |
GPT-6 Luna | low | significantly lower (4 vs 37; p < 0.001) |
GPT-5.1 | low | significantly lower (4 vs 45; p < 0.001) |
GPT-5.6 Luna | low | significantly lower (4 vs 48; p < 0.001) |
DeepSeek V4.1 Flash | default | significantly lower (4 vs 49; p < 0.001) |
Claude Sonnet 5.5 | low | significantly lower (2 vs 51; p < 0.001) |
Gemini 4 Argon | low | significantly lower (6 vs 64; p < 0.001) |
Claude Opus 5 | low | significantly lower (4 vs 75; p < 0.001) |
Kimi K3 | low | significantly lower (6 vs 78; p < 0.001) |
Claude Haiku 5.5 | low | significantly lower (2 vs 75; p < 0.001) |
GPT-4o (production default) | not applicable | significantly lower (4 vs 110; p < 0.001) |
Claude Sonnet 5 | low | significantly lower (0 vs 186; p < 0.001) |
Mistral Medium 3 | not applicable | significantly lower (2 vs 199; p < 0.001) |
How the benchmark works
The benchmark is a frozen set of questionnaire questions run against a frozen Answer Library scope. Each question is sent on its own, and the agent's answer and every cited source are recorded.
Part | Cases | What it tests |
|---|---|---|
Answerable: the library holds an approved answer and its source | 133 | Whether the agent finds the entry and returns the approved answer unchanged |
Without a library answer: realistic, in-scope questions the library does not answer | 192 | Whether the agent refuses instead of improvising |
Total | 325 |
Pass rules. Scoring is deterministic; no model judges the answers.
Answerable case: passes when the answer cites the expected library entry and its text equals the approved answer, ignoring letter case. Before comparing, the scorer removes what the agent adds around the answer (citation markers, collapsible detail blocks, the outdated-entry flag) and normalizes whitespace and surrounding quotes. Partial, similar or reworded answers fail.
Duplicate entries: the library holds some answers more than once. Citing a different entry passes only when that entry's stored question and answer are the same as the expected ones.
Case without a library answer: passes only when the answer begins with "No matching documents". Any other answer fails.
The overall pass rate depends on the mix of 133 answerable to 192 unanswerable cases; a different mix changes it even when the agent behaves the same. Answer coverage, answer accuracy and safe refusal do not depend on the mix. This is a controlled benchmark: it does not measure performance under other topics, languages, phrasings or library content.
Setup
Every model answered the same 325 scored cases (133 with an approved answer in the library, 192 without) through the same agent configuration, the same prompt and the same frozen library scope; only the model and its reasoning level change between rows.
GPT-4o, the production default, ran on 29 October with the same library, configuration, questions and release as the other models. It has no reasoning setting. Gemini 4 Argon ran on 4 November, GPT-6.1 Terra on 12 November, and GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, GPT-6 Sol, GPT-6 Astra, GPT-6 Luna and Claude Haiku 5.5 on 13 November, with the same library, configuration and questions. Reasoning was set to low for every model that supports it. Mistral Medium 3 has no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model. Every model is compared with GPT-4o. A repeat run of 6 of the models (GPT-5.1, GPT-6.1 Sol, Claude Opus 5.5, Claude Opus 5, Kimi K3 and Mistral Medium 3) on the same configuration moved the overall pass rate by at most 1.5 points, which is the run-to-run noise to expect.
Hosting. GPT-5.1 runs on Azure in Switzerland, GPT-4o on Azure OpenAI and Mistral Medium 3 and Gemini 4 Argon on Vertex AI in the EU. Anthropic, Azure OpenAI global deployments (GPT-6.1 Terra, GPT-6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna), OpenAI through LiteLLM (GPT-6.1 Sol) and Together AI (Kimi K3 and DeepSeek V4.1 Flash) process globally. Which providers a client can use depends on its contract.
Configurations
Model | Model deployment | Reasoning effort |
|---|---|---|
Claude Opus 5.5 | Claude Opus 5.5, Anthropic through LiteLLM | low |
GPT-6 Sol | GPT-6 Sol (2026-09-22), Azure OpenAI | low |
GPT-6 Astra | GPT-6 Astra (2026-09-03), Azure OpenAI | low |
GPT-6.1 Sol | GPT-6.1 Sol, OpenAI through LiteLLM | low |
GPT-6.1 Terra | GPT-6.1 Terra (2026-10-26), Azure OpenAI | low |
GPT-5.6 Sol | GPT-5.6 Sol (2026-07-09), Azure OpenAI | low |
GPT-5.6 Terra | GPT-5.6 Terra (2026-07-09), Azure OpenAI | low |
GPT-6 Luna | GPT-6 Luna (2026-09-22), Azure OpenAI | low |
GPT-5.1 | GPT-5.1 (2025-11-13), Azure, Swiss inference | low |
GPT-5.6 Luna | GPT-5.6 Luna (2026-07-09), Azure OpenAI | low |
DeepSeek V4.1 Flash | DeepSeek V4.1 Flash, Together AI through LiteLLM | default (Together AI rejects the reasoning setting the platform sends, so it is removed and the model runs at its default) |
Claude Sonnet 5.5 | Claude Sonnet 5.5, Anthropic through LiteLLM | low |
Gemini 4 Argon | Gemini 4 Argon, Google Vertex AI EU through LiteLLM | low |
Claude Opus 5 | Claude Opus 5, Anthropic through LiteLLM | low |
Kimi K3 | Kimi K3, Together AI through LiteLLM | low |
Claude Haiku 5.5 | Claude Haiku 5.5, Anthropic through LiteLLM | low |
GPT-4o (production default) | GPT-4o (2024-11-20), Azure OpenAI | not applicable (not a reasoning model) |
Claude Sonnet 5 | Claude Sonnet 5, Anthropic through LiteLLM | low |
Mistral Medium 3 | Mistral Medium 3, Google Vertex AI EU through LiteLLM | not applicable (no reasoning setting) |
Notes
Times are measured per answer across all 325 questions. Costs are list prices for the agent's calls, from the usage the platform records or reads back afterwards, not invoices. For some answers the run did not capture usage data, so it was read back from the chat records after the run and priced with the same price sheet: GPT-4o 253, GPT-5.1 2, Claude Opus 5.5 1, Claude Opus 5 7, Claude Sonnet 5.5 1, Kimi K3 2, DeepSeek V4.1 Flash 1. Answers with no usage data at all are scored and left out of the cost mean: GPT-4o 4. Answers that failed for platform reasons (service restarts, provider rate limits) were retried and are not counted as wrong answers. Every model returned a response to all 325 questions.