Library Answers Benchmarks
1 min read
Measured results for the Library Answers agent. These pages are benchmarks: what was tested, on which model, and what to expect. The agent answers a questionnaire question with the approved answer from the Answer Library, cited to its library entry, and replies "No matching documents found" when the library has no answer.
What users get. With Claude Opus 5.5 at low reasoning, 97% of questions get the right outcome: the approved answer from the right library entry, or the refusal when the library has no answer. GPT-6.1 Sol at low reasoning reaches 94%. With Swiss inference (GPT-5.1 at low reasoning) it is 85%: per 1,000 questions, about 62 answerable ones get something other than the approved answer, and about 92 get an answer where the library has none, instead of a refusal. GPT-4o, the production default, gets the right outcome on 65% (210 of 325).
Answering models compared on 325 benchmark questions (October 2026). Reasoning was set to low for every model that supports it. GPT-4o and Mistral Medium 3 have no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model.
Right outcome on 325 questions against mean list-price cost per answer.
Right outcome on 325 questions against mean answer time (log scale). GPT-5.1 times are inflated: another benchmark shared its token limit in the QA environment.
Approved text from the right library file on the 133 answerable questions, against the share of the 192 questions without a library answer that got an answer instead of the refusal.