Evaluating Models and Harnesses: Overview
6 min read
Evaluations of our agents and workflows with the language models that run in them. Each table lists only the model and reasoning-effort combinations tested in that benchmark. Every score is a percentage where higher is better.
Agents
Unique AI chat
Provider | Model | Reasoning effort | Web search | Internal search |
|---|---|---|---|---|
OpenAI | GPT-6.1 Sol | medium | 94% | 94% |
OpenAI | GPT-6.1 Terra | medium | 96% | 92% |
OpenAI | GPT-5.1 (Swiss inference) | none | 87% | 90% |
OpenAI | GPT-5.1 (Swiss inference) | low | 87% | 90% |
OpenAI | GPT-5.1 (Swiss inference) | medium | 88% | 91% |
OpenAI | GPT-5.1 (Swiss inference) | high | 89% | 91% |
Anthropic | Claude Opus 5.5 | medium | 82% | 88% |
Anthropic | Claude Opus 5 | medium | 79% | 87% |
Anthropic | Claude Sonnet 5.5 | medium | 84% | 89% |
Anthropic | Claude Sonnet 5 | medium | 83% | 87% |
DeepSeek | DeepSeek V4.1 Flash | default | 83% | 87% |
Gemini 4 Argon | medium | 85% | 87% |
Web search and internal search. Overall score in %: the average of five LLM judges (factual accuracy, sources or next steps, intent, conciseness, tone) on 50 web-search and 150 internal-search questions. Details: Unique AI Chat Benchmarks.
The two columns use different question sets, so they are not combined into one overall value. GPT-6.1 Terra was added in November 2026 and scored with the November judges; against GPT-6.1 Sol scored the same way it is 1 point ahead on web search and 1 point behind on internal search.
Agentic Table subagents
Provider | Model | Reasoning effort | Verbatim library answers | Composed answers from library sources: content kept |
|---|---|---|---|---|
OpenAI | GPT-6.1 Sol | low | 94% | 95% |
OpenAI | GPT-6.1 Terra | low | 94% | 81% |
OpenAI | GPT-6 Sol | low | 96% | 86% |
OpenAI | GPT-6 Astra | low | 95% | 92% |
OpenAI | GPT-6 Luna | low | 87% | 87% |
OpenAI | GPT-5.1 (Swiss inference) | low | 85% | 93% |
OpenAI | GPT-5.6 Sol | low | 91% | 91% |
OpenAI | GPT-5.6 Terra | low | 88% | 92% |
OpenAI | GPT-5.6 Luna | low | 84% | 84% |
OpenAI | GPT-4o (production default) | not applicable | 65% | 84% |
Anthropic | Claude Opus 5.5 | low | 97% | 99% |
Anthropic | Claude Opus 5 | low | 75% | 95% |
Anthropic | Claude Sonnet 5.5 | low | 82% | 95% |
Anthropic | Claude Sonnet 5 | low | 40% | 89% |
Anthropic | Claude Haiku 5.5 | low | 75% | 79% |
DeepSeek | DeepSeek V4.1 Flash | default | 83% | 98% |
Moonshot AI | Kimi K3 | low | 75% | 95% |
Gemini 4 Argon | low | 79% | 98% | |
Mistral AI | Mistral Medium 3 | not applicable | 37% | 63% |
Verbatim library answers (Agentic Table; returns the approved answer unchanged, or refuses). Overall pass rate: the share of 325 benchmark questions with the right outcome, meaning the approved answer from the right library entry, or the refusal when the library has no answer. Details: Verbatim Library Answers Benchmarks.
Composed answers from library sources (Agentic Table; writes an answer from eligible library sources and is designed to flag gaps). Content kept: the share of judged answers, among the 133 questions with an approved library answer, where an LLM judge (GPT-5.4) gives the composed answer at least 7 of 10 both for covering the approved content and for not contradicting it. Details: Composed Answers from Library Sources Benchmarks.
The two columns use different scores and question sets, so they are not combined into one overall value. Default: DeepSeek V4.1 Flash runs at its own default reasoning, because the provider does not accept the setting.
Questionnaire question extraction
Provider | Model | Reasoning effort | Questions captured |
|---|---|---|---|
OpenAI | GPT-6.1 Sol | low | 86% |
OpenAI | GPT-6.1 Terra | low | 91% |
OpenAI | GPT-6 Sol | low | 91% |
OpenAI | GPT-6 Luna | low | 89% |
OpenAI | GPT-6 Astra | low | 86% |
OpenAI | GPT-5.6 Sol | low | 88% |
OpenAI | GPT-5.6 Terra | low | 91% |
OpenAI | GPT-5.6 Luna | low | 80% |
OpenAI | GPT-5.4 | low | 84% |
OpenAI | GPT-5.2 | low | 84% |
OpenAI | GPT-5.1 | low | 81% |
OpenAI | GPT-5 Mini | low | 84% |
OpenAI | GPT-4o (production default) | not applicable | 31% |
Anthropic | Claude Opus 5.5 | low | 86% |
Anthropic | Claude Opus 5 | low | 85% |
Anthropic | Claude Sonnet 5.5 | low | 58% |
Anthropic | Claude Sonnet 5 | low | 45% |
Anthropic | Claude Haiku 5.5 | low | 39% |
Moonshot AI | Kimi K3 | low | 68% |
Gemini 4 Argon | low | 25% |
Questionnaire question extraction (Agentic Table; turns an uploaded questionnaire into the list of questions to answer). Questions captured: the share of the 1,495 questions on 18 standard documents that the extraction returns, graded by an aligner and a two-vote semantic judge; a document a model does not complete counts as zero. Details: Questionnaire Question Extraction Benchmarks.
GPT-6.1 Terra, GPT-5.6 Terra and GPT-6 Sol are within 0.1 points of each other. With 18 documents only large differences are significant: every model down to Claude Sonnet 5.5 is not significantly different from GPT-6.1 Terra. Claude models were scored with the extraction schema in the prompt, because Anthropic rejects the production schema; the product cannot use them for extraction as it is. Gemini 4 Argon returns an empty question list on 9 of the 18 documents.
Conduct
Provider | Model | Reasoning effort | Harness | PowerPoint generation |
|---|---|---|---|---|
OpenAI | GPT-6.1 Sol | medium | Codex | 98% |
OpenAI | GPT-6.1 Terra | high | Codex | 100% |
OpenAI | GPT-6.1 Terra | medium | Codex | 96% |
OpenAI | GPT-6.1 Terra | low | Codex | 96% |
OpenAI | GPT-5.1 (Swiss inference) | high | Codex | 72% |
OpenAI | GPT-5.1 (Swiss inference) | high | pi | 28% |
OpenAI | GPT-5.1 (Swiss inference) | medium | Codex | 63% |
Anthropic | Claude Opus 5.5 | medium | Claude Code | 99% |
PowerPoint generation (builds a deck in the bank’s own template). Client-ready slides: the share of the requested slides with no defect the user must fix before sending, on 8 requests on 4 company templates nobody tuned the setup on, graded blind slide by slide. Each harness runs with the setup currently in the product: GPT-5.1 on Codex with the optimised setup; GPT-5.1 on pi with the standard skill, because the optimised setup is not on pi yet; GPT-6.1 Sol, GPT-6.1 Terra and Claude Opus 5.5 with the standard skill. GPT-6.1 Terra ran on 12 November 2026 with the same requests and grading. On templates already reviewed with us, GPT-5.1 on Codex with the optimised setup reaches 97% at high and at medium.
Provider | Model | Reasoning effort | Harness | Web search | Internal search |
|---|---|---|---|---|---|
OpenAI | GPT-6.1 Sol | medium | none (standard chat) | 95% | 95% |
OpenAI | GPT-6.1 Sol | medium | Codex | 93% | 94% |
OpenAI | GPT-6.1 Sol | medium | pi | 94% | 94% |
OpenAI | GPT-5.1 (Swiss inference) | medium | none (standard chat) | 91% | 91% |
OpenAI | GPT-5.1 (Swiss inference) | medium | Codex | 88% | 92% |
OpenAI | GPT-5.1 (Swiss inference) | medium | pi | 92% | 91% |
Anthropic | Claude Opus 5.5 | medium | none (standard chat) | 84% | 89% |
Anthropic | Claude Opus 5.5 | auto | Claude Code | 89% | 92% |
Anthropic | Claude Opus 5.5 | medium | pi | 88% | 92% |
Web search and internal search with Conduct. Core score in %: the mean of the LLM judges that score every answer (internal search: factual accuracy against the reference answer, intent, conciseness, tone; web search: factual accuracy against the retrieved pages, source quality, intent, conciseness, tone), on 50 web-search and 150 internal-search questions, November 2026. The rows without a harness are standard Unique AI chat in the same run. These runs use revised judges, so they differ from the Unique AI chat table above, which will be updated.
AI Workflows
Hallucination check
Provider | Model | Reasoning effort | Accuracy |
|---|---|---|---|
OpenAI | GPT-6.1 Sol | low | 91% |
OpenAI | GPT-6.1 Sol | medium | 91% |
OpenAI | GPT-6.1 Sol | high | 91% |
OpenAI | GPT-6.1 Sol | xhigh | 91% |
OpenAI | GPT-6.1 Terra | low | 89% |
OpenAI | GPT-6 Sol | none | 90% |
OpenAI | GPT-6 Sol | medium | 90% |
OpenAI | GPT-5.1 (Swiss inference) | minimal | 78% |
OpenAI | GPT-5.1 (Swiss inference) | low | 87% |
OpenAI | GPT-5.1 (Swiss inference) | medium | 87% |
OpenAI | GPT-5.4-mini | none | 72% |
OpenAI | GPT-5.6 Luna | none | 80% |
OpenAI | GPT-5.6 Luna | low | 85% |
OpenAI | GPT-6 Luna | none | 84% |
OpenAI | GPT-5-mini | minimal | 68% |
Anthropic | Claude Opus 5.5 | low | 92% |
Anthropic | Claude Sonnet 5.5 | low | 88% |
Anthropic | Claude Sonnet 5 | low | 83% |
Gemini 4 Argon | low | 91% | |
DeepSeek | DeepSeek V4.1 Flash | default | 89% |
Moonshot AI | Kimi K3 | low | 88% |
Mistral AI | Mistral Medium 3 | not applicable | 67% |
Accuracy is the share of 834 benchmark answers the judge model gets right: it warns when an answer contradicts a source the agent cited and stays quiet when the answer is correct. Details: Hallucination Check. Not scorable: Claude Opus 5 returned no verdict on 102 of the 834 answers.