Agentic Table-Library Answers Benchmark
11 min read
The Library Answers agent answers a questionnaire question with the approved answer from the Answer Library, cited to its library entry, and replies "No matching documents found" when the library has no answer. This page reports how well it does that with each answering model we evaluated. Every model answered the same 325 scored cases (133 with an approved answer in the library, 192 without) through the same agent configuration, the same prompt and the same frozen library scope; only the model and its reasoning level change between columns. The cases and pass rules are described under How the benchmark works below.
GPT-4o, the production default, ran on 29 October with the same library, configuration, questions and release as the other models. It has no reasoning setting. Reasoning was set to low for every model that supports it. Mistral Medium 3 has no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model. Every model is compared with GPT-4o. A repeat run of 6 of the models on the same configuration moved the overall pass rate by at most 1.5 points, which is the run-to-run noise to expect.
Hosting. GPT-5.1 runs on Azure in Switzerland, GPT-4o on Azure OpenAI and Mistral Medium 3 on Vertex AI in the EU. Anthropic, OpenAI through LiteLLM (GPT-6.1 Sol) and Together AI (Kimi K3 and DeepSeek V4.1 Flash) process globally. Which providers a client can use depends on its contract.
Summary
The models differ on the approved text and on refusing, more than on finding the file. Every model except Mistral Medium 3 and GPT-4o cites the right library file on 90–100% of answerable questions (GPT-4o 82%, Mistral Medium 3 67%). What separates them is whether they return the approved text unchanged (25–93%) and whether they decline the questions the library does not answer (45–100%).
Claude Opus 5.5, low: the right outcome on 97% of questions (316/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 93% of answerable questions (answer accuracy 96%), refusal on 100% of questions without a library answer. Mean answer time 15 s, USD 71 per benchmark run. Against GPT-6.1 Sol: significantly higher (p = 0.031). The most accurate model we measured, at about 1.6 times the cost of GPT-6.1 Sol.
GPT-6.1 Sol, low: the right outcome on 94% of questions (306/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 86% of answerable questions (answer accuracy 97%), refusal on 100% of questions without a library answer. Mean answer time 19 s, USD 45 per benchmark run. Refuses every question without a library answer, but also declines 15 answerable questions that the library does answer.
GPT-5.1, low: the right outcome on 85% of questions (275/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 85% of answerable questions (answer accuracy 89%), refusal on 84% of questions without a library answer. Mean answer time 47 s, USD 30 per benchmark run. The option when inference must run in Switzerland. It answers 30 of the 192 questions without a library answer instead of refusing. Its answer times are inflated by a shared token limit.
DeepSeek V4.1 Flash, default: the right outcome on 83% of questions (271/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 71% of answerable questions (answer accuracy 71%), refusal on 92% of questions without a library answer. Mean answer time 30 s, USD 44 per benchmark run. Close to GPT-5.1 overall: fewer approved texts, more correct refusals. Runs at its default reasoning, not at low.
Claude Sonnet 5.5, low: the right outcome on 82% of questions (267/325), significantly higher than GPT-4o (65%, p < 0.001). Approved text from the right file on 58% of answerable questions (answer accuracy 60%), refusal on 99% of questions without a library answer. Mean answer time 12 s, USD 35 per benchmark run. Refuses correctly and cites the right file on 94% of answerable questions, but often returns text that differs from the approved answer.
Claude Opus 5, low: the right outcome on 75% of questions (245/325), significantly higher than GPT-4o (65%, p = 0.001). Approved text from the right file on 62% of answerable questions (answer accuracy 62%), refusal on 84% of questions without a library answer. Mean answer time 15 s, USD 107 per benchmark run. Answers 30 questions without a library answer and is the most expensive model. Not recommended.
Kimi K3, low: the right outcome on 75% of questions (244/325), significantly higher than GPT-4o (65%, p = 0.002). Approved text from the right file on 82% of answerable questions (answer accuracy 82%), refusal on 70% of questions without a library answer. Mean answer time 23 s, USD 62 per benchmark run. Answers 30% of the questions without a library answer. Not recommended.
GPT-4o, no reasoning (production default): the right outcome on 65% of questions (210/325). Approved text from the right file on 35% of answerable questions (answer accuracy 35%), refusal on 85% of questions without a library answer. Mean answer time 28 s, USD 25 per benchmark run. The production default. It cites the right file on 82% of answerable questions but changes the approved text on most of them.
Claude Sonnet 5, low: the right outcome on 40% of questions (130/325), significantly lower than GPT-4o (65%, p < 0.001). Approved text from the right file on 25% of answerable questions (answer accuracy 25%), refusal on 51% of questions without a library answer. Mean answer time 13 s, USD 39 per benchmark run. Answers 49% of the questions without a library answer and returns the approved text on 25% of the answerable ones. Not suitable.
Mistral Medium 3, no reasoning: the right outcome on 37% of questions (119/325), significantly lower than GPT-4o (65%, p < 0.001). Approved text from the right file on 25% of answerable questions (answer accuracy 25%), refusal on 45% of questions without a library answer. Mean answer time 12 s, USD 3 per benchmark run. The cheapest model, but cites the right file on only 67% of answerable questions. Not suitable.
Results
95% Wilson intervals in brackets. Answer coverage is the share of answerable questions that got an answer, answer accuracy the share of those answers with the approved text from the right file, safe refusal the share of questions without a library answer that got the refusal. Answer time: mean per question with a 95% bootstrap interval. Cost per benchmark run: mean list price per answer times 325.
OpenAI models
GPT-4o, no reasoning (production default) | GPT-5.1, low | GPT-6.1 Sol, low | |
|---|---|---|---|
Overall pass rate (325) | 65% (210/325) [59–70%] | 85% (275/325) [80–88%] | 94% (306/325) [91–96%] |
Answer coverage: answerable questions answered (133) | 99% (132/133) [96–100%] | 95% (127/133) [91–98%] | 89% (118/133) [82–93%] |
Answer accuracy: approved text from the right file, of answered | 35% (46/132) [27–43%] | 89% (113/127) [82–93%] | 97% (114/118) [92–99%] |
Approved text from the right file, of all answerable (133) | 35% (46/133) [27–43%] | 85% (113/133) [78–90%] | 86% (114/133) [79–91%] |
Right file cited, in an answer or a refusal (133) | 82% (109/133) | 94% (125/133) | 95% (127/133) |
Safe refusal (192 questions without a library answer) | 85% (164/192) [80–90%] | 84% (162/192) [79–89%] | 100% (192/192) [98–100%] |
Answer time, mean [95% interval] | 28 s [25–30 s] | 47 s [42–52 s] | 19 s [18–19 s] |
Answer time, median / 90th percentile | 18 s / 53 s | 30 s / 107 s | 17 s / 27 s |
Cost per answer (mean) | USD 0.08 | USD 0.09 | USD 0.14 |
Cost per benchmark run | USD 25 | USD 30 | USD 45 |
Other providers
GPT-4o, no reasoning (production default); for comparison | Claude Opus 5.5, low | Claude Opus 5, low | Claude Sonnet 5.5, low | Claude Sonnet 5, low | Kimi K3, low | Mistral Medium 3, no reasoning | DeepSeek V4.1 Flash, default | |
|---|---|---|---|---|---|---|---|---|
Overall pass rate (325) | 65% (210/325) [59–70%] | 97% (316/325) [95–99%] | 75% (245/325) [70–80%] | 82% (267/325) [78–86%] | 40% (130/325) [35–45%] | 75% (244/325) [70–79%] | 37% (119/325) [32–42%] | 83% (271/325) [79–87%] |
Answer coverage: answerable questions answered (133) | 99% (132/133) [96–100%] | 97% (129/133) [93–99%] | 100% (133/133) [97–100%] | 96% (128/133) [92–98%] | 100% (133/133) [97–100%] | 100% (133/133) [97–100%] | 98% (130/133) [94–99%] | 100% (133/133) [97–100%] |
Answer accuracy: approved text from the right file, of answered | 35% (46/132) [27–43%] | 96% (124/129) [91–98%] | 62% (83/133) [54–70%] | 60% (77/128) [51–68%] | 25% (33/133) [18–33%] | 82% (109/133) [75–88%] | 25% (33/130) [19–33%] | 71% (95/133) [63–78%] |
Approved text from the right file, of all answerable (133) | 35% (46/133) [27–43%] | 93% (124/133) [88–96%] | 62% (83/133) [54–70%] | 58% (77/133) [49–66%] | 25% (33/133) [18–33%] | 82% (109/133) [75–88%] | 25% (33/133) [18–33%] | 71% (95/133) [63–78%] |
Right file cited, in an answer or a refusal (133) | 82% (109/133) | 100% (133/133) | 100% (133/133) | 94% (125/133) | 91% (121/133) | 96% (128/133) | 67% (89/133) | 90% (120/133) |
Safe refusal (192 questions without a library answer) | 85% (164/192) [80–90%] | 100% (192/192) [98–100%] | 84% (162/192) [79–89%] | 99% (190/192) [96–100%] | 51% (97/192) [44–58%] | 70% (135/192) [64–76%] | 45% (86/192) [38–52%] | 92% (176/192) [87–95%] |
Answer time, mean [95% interval] | 28 s [25–30 s] | 15 s [15–16 s] | 15 s [14–15 s] | 12 s [12–12 s] | 13 s [12–13 s] | 23 s [22–24 s] | 12 s [11–12 s] | 30 s [27–33 s] |
Answer time, median / 90th percentile | 18 s / 53 s | 14 s / 21 s | 14 s / 19 s | 11 s / 16 s | 12 s / 16 s | 21 s / 33 s | 10 s / 21 s | 20 s / 66 s |
Cost per answer (mean) | USD 0.08 | USD 0.22 | USD 0.33 | USD 0.11 | USD 0.12 | USD 0.19 | USD 0.01 | USD 0.13 |
Cost per benchmark run | USD 25 | USD 71 | USD 107 | USD 35 | USD 39 | USD 62 | USD 3 | USD 44 |
Per 1,000 questions
In the benchmark mix, 41% of questions have an approved library answer (133 of 325) and 59% do not (192 of 325). Scaled to 1,000 questions in this mix, rounded so that each column adds up to 1,000:
Per 1,000 questions | GPT-4o, no reasoning (production default) | GPT-5.1, low | GPT-6.1 Sol, low | Claude Opus 5.5, low | Claude Opus 5, low | Claude Sonnet 5.5, low | Claude Sonnet 5, low | Kimi K3, low | Mistral Medium 3, no reasoning | DeepSeek V4.1 Flash, default |
|---|---|---|---|---|---|---|---|---|---|---|
Right outcome | about 646 | about 846 | about 942 | about 972 | about 754 | about 822 | about 400 | about 751 | about 366 | about 834 |
Answerable, other text or wrong file | about 265 | about 43 | about 12 | about 16 | about 154 | about 157 | about 308 | about 74 | about 299 | about 117 |
Answerable, declined | about 3 | about 19 | about 46 | about 12 | about 0 | about 15 | about 0 | about 0 | about 9 | about 0 |
No library answer, answered anyway | about 86 | about 92 | about 0 | about 0 | about 92 | about 6 | about 292 | about 175 | about 326 | about 49 |
The mix is set by the benchmark, not by production. A questionnaire where most questions have a library answer weights the first rows more; one with many new questions weights the last row more.
Statistical tests
Paired exact McNemar tests against GPT-4o, the production default, on the same questions:
Compared with GPT-4o, no reasoning (production default) | Questions with the right outcome |
|---|---|
GPT-5.1, low | significantly more (passes 90 that GPT-4o fails, fails 25; p < 0.001) |
GPT-6.1 Sol, low | significantly more (passes 99 that GPT-4o fails, fails 3; p < 0.001) |
Claude Opus 5.5, low | significantly more (passes 110 that GPT-4o fails, fails 4; p < 0.001) |
Claude Opus 5, low | significantly more (passes 73 that GPT-4o fails, fails 38; p = 0.001) |
Claude Sonnet 5.5, low | significantly more (passes 77 that GPT-4o fails, fails 20; p < 0.001) |
Claude Sonnet 5, low | significantly fewer (passes 27 that GPT-4o fails, fails 107; p < 0.001) |
Kimi K3, low | significantly more (passes 76 that GPT-4o fails, fails 42; p = 0.002) |
Mistral Medium 3, no reasoning | significantly fewer (passes 24 that GPT-4o fails, fails 115; p < 0.001) |
DeepSeek V4.1 Flash, default | significantly more (passes 81 that GPT-4o fails, fails 20; p < 0.001) |
Against Claude Opus 5.5, low, every other model scores significantly lower (p < 0.001 for the others; GPT-6.1 Sol, low p = 0.031).
How the benchmark works
The benchmark is a frozen set of questionnaire questions run against a frozen Answer Library scope. Each question is sent on its own, and the agent's answer and every cited source are recorded.
Part | Cases | What it tests |
|---|---|---|
Answerable: the library holds an approved answer and its source | 133 | Whether the agent finds the entry and returns the approved answer unchanged |
Without a library answer: realistic, in-scope questions the library does not answer | 192 | Whether the agent refuses instead of improvising |
Total | 325 |
Pass rules. Scoring is deterministic; no model judges the answers.
Answerable case: passes when the answer cites the expected library entry and its text equals the approved answer, ignoring letter case. Before comparing, the scorer removes what the agent adds around the answer (citation markers, collapsible detail blocks, the outdated-entry flag) and normalizes whitespace and surrounding quotes. Partial, similar or reworded answers fail.
Duplicate entries: the library holds some answers more than once. Citing a different entry passes only when that entry's stored question and answer are the same as the expected ones.
Case without a library answer: passes only when the answer begins with "No matching documents". Any other answer fails.
The overall pass rate depends on the mix of 133 answerable to 192 unanswerable cases; a different mix changes it even when the agent behaves the same. Answer coverage, answer accuracy and safe refusal do not depend on the mix. This is a controlled benchmark: it does not measure performance under other topics, languages, phrasings or library content.
Configurations
Configuration | Model deployment | Reasoning |
|---|---|---|
GPT-4o, no reasoning (production default) | GPT-4o (2024-11-20), Azure OpenAI | not applicable (not a reasoning model) |
GPT-5.1, low | GPT-5.1 (2025-11-13), Azure, Swiss inference | low |
GPT-6.1 Sol, low | GPT-6.1 Sol, OpenAI through LiteLLM | low |
Claude Opus 5.5, low | Claude Opus 5.5, Anthropic through LiteLLM | low |
Claude Opus 5, low | Claude Opus 5, Anthropic through LiteLLM | low |
Claude Sonnet 5.5, low | Claude Sonnet 5.5, Anthropic through LiteLLM | low |
Claude Sonnet 5, low | Claude Sonnet 5, Anthropic through LiteLLM | low |
Kimi K3, low | Kimi K3, Together AI through LiteLLM | low |
Mistral Medium 3, no reasoning | Mistral Medium 3, Google Vertex AI europe-west4 through LiteLLM | not applicable (no reasoning setting) |
DeepSeek V4.1 Flash, default | DeepSeek V4.1 Flash, Together AI through LiteLLM | default (Together AI rejects the reasoning setting the platform sends, so it is removed and the model runs at its default) |
Times are measured per answer across all 325 questions. Costs are list prices for the agent's calls, from the usage the platform records or reads back afterwards, not invoices. For some answers the run did not capture usage data, so it was read back from the chat records in the QA environment after the run and priced with the same price sheet: GPT-4o 253, GPT-5.1 2, Claude Opus 5.5 1, Claude Opus 5 7, Claude Sonnet 5.5 1, Kimi K3 2, DeepSeek V4.1 Flash 1. Answers with no usage data at all are scored and left out of the cost mean: GPT-4o 4. Answers that failed for platform reasons (service restarts, provider rate limits) were retried and are not counted as wrong answers. Every model answered all 325 questions.