Composed Answers from Library Sources Benchmarks

3 min read

Measured results for the Composed answers from library sources agent. These pages are benchmarks: what was tested, on which model, and what to expect. The agent composes an answer from eligible library sources, with supporting arguments and as-of dates, and is designed to flag the gap when no eligible source determines the answer.

What users get. On the 133 questions with an approved library answer, Claude Opus 5.5 at low reasoning and Gemini 4 Argon at low reasoning keep the approved content best (content score 9.9 of 10 each from an LLM judge, no significant difference; 99% and 98% of their answers score at least 7 on both criteria) and, with Claude Opus 5 at low reasoning, stay closest to the approved answers in tone (0.95 on a 0-to-1 scale). Gemini 4 Argon is much slower: a mean 178 s per answer, against 12 to 50 s for the other models in its run. On 3 of the 192 questions the library cannot answer, Gemini 4 Argon kept searching and returned no answer within 30 minutes. DeepSeek V4.1 Flash at its default reasoning, Claude Sonnet 5.5 at low reasoning, Kimi K3 at low reasoning and Claude Opus 5 at low reasoning are close behind (9.6–9.8). The other models score significantly lower (8.4–9.5), among them GPT-4o, the production default (8.5; the second judge finds a material difference on 22% of its answers). Mistral Medium 3 (no reasoning, 7.2) often leaves out material content. Every model except Claude Opus 5 and Gemini 4 Argon is significantly further from the approved answers in tone, down to 0.85 for GPT-6.1 Terra. An answer costs USD 0.01 to USD 1.18 in run 1 and USD 0.01 to USD 0.51 in run 2.

Answering models compared on 325 benchmark questions (October–November 2026): content and tone on the 133 with an approved library answer. Cost and time are per answer over all 325 questions (322 for Gemini 4 Argon). Reasoning was set to low for every model that supports it. GPT-4o and Mistral Medium 3 have no reasoning setting, and DeepSeek V4.1 Flash runs at its default because Together AI does not accept the setting for this model. The models were tested in two runs with the same questions, library, prompt, judges and scoring. Run 2 (Gemini 4 Argon, GPT-5.6 Terra, GPT-5.6 Sol, GPT-6 Astra, GPT-6 Luna, GPT-5.6 Luna, GPT-4o, GPT-6 Sol, Claude Haiku 5.5 and GPT-6.1 Terra) retrieved fewer library sources per search than run 1 (Claude Opus 5.5, DeepSeek V4.1 Flash, Claude Sonnet 5.5, Kimi K3, Claude Opus 5, GPT-5.1, GPT-6.1 Sol, Claude Sonnet 5 and Mistral Medium 3). That lowers the input per answer, so cost and answer time are compared only within a run.

subagent_accuracy_vs_tone.png

Content score (GPT-5.4 judge) against tone match with the approved answer, both on the 133 answerable questions, for every model. Top right is closest to the approved answers on both.

subagent_accuracy_vs_cost.png

Content score on the 133 answerable questions (GPT-5.4 judge, 0 to 10, mean with a 95% bootstrap interval), against mean list-price cost per answer over all 325 questions. Run 1. Cost and time are shown per run because run 2 retrieved fewer library sources per search.

subagent_accuracy_vs_cost_run2.png

Content score on the 133 answerable questions (GPT-5.4 judge, 0 to 10, mean with a 95% bootstrap interval), against mean list-price cost per answer over all 325 questions. Run 2. Cost and time are shown per run because run 2 retrieved fewer library sources per search. Gemini 4 Argon's cost and time are over the 322 questions it answered; on 3 questions without a library answer it returned no answer within 30 minutes.

subagent_accuracy_vs_time.png

Content score against mean answer time over all 325 questions (log scale). Run 1. GPT-5.1 times are inflated: another benchmark shared its token limit during the run.

subagent_accuracy_vs_time_run2.png

Content score against mean answer time over all 325 questions (log scale). Run 2. Cost and time are shown per run because run 2 retrieved fewer library sources per search. Gemini 4 Argon's cost and time are over the 322 questions it answered; on 3 questions without a library answer it returned no answer within 30 minutes.

Last updated