Models and Harness Eval Overview
3 min read
Evaluations of our agents and workflows with the language models that run in them. Each table lists only the model and reasoning-effort combinations tested in that benchmark. Every score is a percentage where higher is better.
Agents
Unique AI chat
Provider | Model | Reasoning effort | Web search | Internal search |
|---|---|---|---|---|
OpenAI | GPT-6.1 Sol | medium | 94% | 94% |
OpenAI | GPT-5.1 (Swiss inference) | none | 87% | 90% |
OpenAI | GPT-5.1 (Swiss inference) | low | 87% | 90% |
OpenAI | GPT-5.1 (Swiss inference) | medium | 88% | 91% |
OpenAI | GPT-5.1 (Swiss inference) | high | 89% | 91% |
Anthropic | Claude Opus 5.5 | medium | 82% | 88% |
Anthropic | Claude Opus 5 | medium | 79% | 87% |
Anthropic | Claude Sonnet 5.5 | medium | 84% | 89% |
Anthropic | Claude Sonnet 5 | medium | 83% | 87% |
Web search and internal search. Overall score in %: the average of five LLM judges (factual accuracy, sources or next steps, intent, conciseness, tone) on 50 web-search and 150 internal-search questions. Details: Unique AI Chat Benchmarks.
The two columns use different question sets, so they are not combined into one overall value.
Agentic Table subagents
Provider | Model | Reasoning effort | Verbatim library answers | Composed answers from library sources: content kept |
|---|---|---|---|---|
OpenAI | GPT-6.1 Sol | low | 94% | 95% |
OpenAI | GPT-5.1 (Swiss inference) | low | 85% | 93% |
OpenAI | GPT-4o (production default) | not applicable | 65% | no answer |
Anthropic | Claude Opus 5.5 | low | 97% | 99% |
Anthropic | Claude Opus 5 | low | 75% | 95% |
Anthropic | Claude Sonnet 5.5 | low | 82% | 95% |
Anthropic | Claude Sonnet 5 | low | 40% | run incomplete |
DeepSeek | DeepSeek V4.1 Flash | default | 83% | 98% |
Moonshot AI | Kimi K3 | low | 75% | 95% |
Mistral AI | Mistral Medium 3 | not applicable | 37% | 62% |
Verbatim library answers (Agentic Table; returns the approved answer unchanged, or refuses). Overall pass rate: the share of 325 benchmark questions with the right outcome, meaning the approved answer from the right library entry, or the refusal when the library has no answer. Details: Verbatim Library Answers Benchmarks.
Composed answers from library sources (Agentic Table; writes an answer from eligible library sources and is designed to flag gaps). Content kept: the share of the 133 questions with an approved library answer where an LLM judge (GPT-5.4) gives the composed answer at least 7 of 10 both for covering the approved content and for not contradicting it. Details: Composed Answers from Library Sources Benchmarks.
The two columns use different scores and question sets, so they are not combined into one overall value. Default: DeepSeek V4.1 Flash runs at its own default reasoning, because the provider does not accept the setting. No answer: GPT-4o did not return a composed answer within 15 minutes. Run incomplete: the Claude Sonnet 5 composed-answers run stopped at 282 of 325 questions, and only complete runs were scored.
Conduct
Provider | Model | Reasoning effort | Harness | PowerPoint generation |
|---|---|---|---|---|
OpenAI | GPT-6.1 Sol | medium | Codex | 98% |
OpenAI | GPT-5.1 (Swiss inference) | high | Codex | 72% |
OpenAI | GPT-5.1 (Swiss inference) | high | pi | 28% |
OpenAI | GPT-5.1 (Swiss inference) | medium | Codex | 63% |
Anthropic | Claude Opus 5.5 | medium | Claude Code | 99% |
PowerPoint generation (builds a deck in the bank’s own template). Client-ready slides: the share of the requested slides with no defect the user must fix before sending, on 8 requests on 4 company templates nobody tuned the setup on, graded blind slide by slide. Each harness runs as on master: GPT-5.1 on Codex with the optimised setup; GPT-5.1 on pi with the standard skill, because the optimised setup is not on pi yet; GPT-6.1 Sol and Claude Opus 5.5 with the standard skill. On templates already reviewed with us, GPT-5.1 on Codex with the optimised setup reaches 97% at high and at medium. Details: Client-Ready Decks with GPT-5.1: Evaluation.
Web search and internal search per harness will follow.
AI Workflows
Hallucination check
Provider | Model | Reasoning effort | Accuracy |
|---|---|---|---|
OpenAI | GPT-6.1 Sol | low | 91% |
OpenAI | GPT-6.1 Sol | medium | 91% |
OpenAI | GPT-6.1 Sol | high | 91% |
OpenAI | GPT-6.1 Sol | xhigh | 91% |
OpenAI | GPT-6.1 Terra | low | 89% |
OpenAI | GPT-6 Sol | none | 90% |
OpenAI | GPT-6 Sol | medium | 90% |
OpenAI | GPT-5.1 (Swiss inference) | minimal | 78% |
OpenAI | GPT-5.1 (Swiss inference) | low | 87% |
OpenAI | GPT-5.1 (Swiss inference) | medium | 87% |
OpenAI | GPT-5.4-mini | none | 72% |
OpenAI | GPT-5.6 luna | none | 80% |
OpenAI | GPT-5.6 luna | low | 85% |
Anthropic | Claude Opus 5.5 | low | 92% |
Anthropic | Claude Opus 5 | low | not scorable |
Anthropic | Claude Sonnet 5.5 | low | 88% |
Anthropic | Claude Sonnet 5 | low | 83% |
Gemini 4 Argon | low | 91% | |
DeepSeek | DeepSeek V4.1 Flash | not applicable | 89% |
Moonshot AI | Kimi K3 | low | 88% |
Mistral AI | Mistral Medium 3 | not applicable | 67% |
Accuracy is the share of 834 benchmark answers the judge model gets right: it warns when an answer contradicts a source the agent cited and stays quiet when the answer is correct. Details: Hallucination Check. Not scorable: Claude Opus 5 returned no verdict on 102 of the 834 answers.