Evaluating Models and Harnesses: Overview

6 min read

Evaluations of our agents and workflows with the language models that run in them. Each table lists only the model and reasoning-effort combinations tested in that benchmark. Every score is a percentage where higher is better.

Agents

Unique AI chat

Provider

Model

Reasoning effort

Web search

Internal search

OpenAI

GPT-6.1 Sol

medium

94%

94%

OpenAI

GPT-6.1 Terra

medium

96%

92%

OpenAI

GPT-5.1 (Swiss inference)

none

87%

90%

OpenAI

GPT-5.1 (Swiss inference)

low

87%

90%

OpenAI

GPT-5.1 (Swiss inference)

medium

88%

91%

OpenAI

GPT-5.1 (Swiss inference)

high

89%

91%

Anthropic

Claude Opus 5.5

medium

82%

88%

Anthropic

Claude Opus 5

medium

79%

87%

Anthropic

Claude Sonnet 5.5

medium

84%

89%

Anthropic

Claude Sonnet 5

medium

83%

87%

DeepSeek

DeepSeek V4.1 Flash

default

83%

87%

Google

Gemini 4 Argon

medium

85%

87%

Web search and internal search. Overall score in %: the average of five LLM judges (factual accuracy, sources or next steps, intent, conciseness, tone) on 50 web-search and 150 internal-search questions. Details: Unique AI Chat Benchmarks.

The two columns use different question sets, so they are not combined into one overall value. GPT-6.1 Terra was added in November 2026 and scored with the November judges; against GPT-6.1 Sol scored the same way it is 1 point ahead on web search and 1 point behind on internal search.

Agentic Table subagents

Provider

Model

Reasoning effort

Verbatim library answers

Composed answers from library sources: content kept

OpenAI

GPT-6.1 Sol

low

94%

95%

OpenAI

GPT-6.1 Terra

low

94%

81%

OpenAI

GPT-6 Sol

low

96%

86%

OpenAI

GPT-6 Astra

low

95%

92%

OpenAI

GPT-6 Luna

low

87%

87%

OpenAI

GPT-5.1 (Swiss inference)

low

85%

93%

OpenAI

GPT-5.6 Sol

low

91%

91%

OpenAI

GPT-5.6 Terra

low

88%

92%

OpenAI

GPT-5.6 Luna

low

84%

84%

OpenAI

GPT-4o (production default)

not applicable

65%

84%

Anthropic

Claude Opus 5.5

low

97%

99%

Anthropic

Claude Opus 5

low

75%

95%

Anthropic

Claude Sonnet 5.5

low

82%

95%

Anthropic

Claude Sonnet 5

low

40%

89%

Anthropic

Claude Haiku 5.5

low

75%

79%

DeepSeek

DeepSeek V4.1 Flash

default

83%

98%

Moonshot AI

Kimi K3

low

75%

95%

Google

Gemini 4 Argon

low

79%

98%

Mistral AI

Mistral Medium 3

not applicable

37%

63%

Verbatim library answers (Agentic Table; returns the approved answer unchanged, or refuses). Overall pass rate: the share of 325 benchmark questions with the right outcome, meaning the approved answer from the right library entry, or the refusal when the library has no answer. Details: Verbatim Library Answers Benchmarks.

Composed answers from library sources (Agentic Table; writes an answer from eligible library sources and is designed to flag gaps). Content kept: the share of judged answers, among the 133 questions with an approved library answer, where an LLM judge (GPT-5.4) gives the composed answer at least 7 of 10 both for covering the approved content and for not contradicting it. Details: Composed Answers from Library Sources Benchmarks.

The two columns use different scores and question sets, so they are not combined into one overall value. Default: DeepSeek V4.1 Flash runs at its own default reasoning, because the provider does not accept the setting.

Questionnaire question extraction

Provider

Model

Reasoning effort

Questions captured

OpenAI

GPT-6.1 Sol

low

86%

OpenAI

GPT-6.1 Terra

low

91%

OpenAI

GPT-6 Sol

low

91%

OpenAI

GPT-6 Luna

low

89%

OpenAI

GPT-6 Astra

low

86%

OpenAI

GPT-5.6 Sol

low

88%

OpenAI

GPT-5.6 Terra

low

91%

OpenAI

GPT-5.6 Luna

low

80%

OpenAI

GPT-5.4

low

84%

OpenAI

GPT-5.2

low

84%

OpenAI

GPT-5.1

low

81%

OpenAI

GPT-5 Mini

low

84%

OpenAI

GPT-4o (production default)

not applicable

31%

Anthropic

Claude Opus 5.5

low

86%

Anthropic

Claude Opus 5

low

85%

Anthropic

Claude Sonnet 5.5

low

58%

Anthropic

Claude Sonnet 5

low

45%

Anthropic

Claude Haiku 5.5

low

39%

Moonshot AI

Kimi K3

low

68%

Google

Gemini 4 Argon

low

25%

Questionnaire question extraction (Agentic Table; turns an uploaded questionnaire into the list of questions to answer). Questions captured: the share of the 1,495 questions on 18 standard documents that the extraction returns, graded by an aligner and a two-vote semantic judge; a document a model does not complete counts as zero. Details: Questionnaire Question Extraction Benchmarks.

GPT-6.1 Terra, GPT-5.6 Terra and GPT-6 Sol are within 0.1 points of each other. With 18 documents only large differences are significant: every model down to Claude Sonnet 5.5 is not significantly different from GPT-6.1 Terra. Claude models were scored with the extraction schema in the prompt, because Anthropic rejects the production schema; the product cannot use them for extraction as it is. Gemini 4 Argon returns an empty question list on 9 of the 18 documents.

Conduct

Provider

Model

Reasoning effort

Harness

PowerPoint generation

OpenAI

GPT-6.1 Sol

medium

Codex

98%

OpenAI

GPT-6.1 Terra

high

Codex

100%

OpenAI

GPT-6.1 Terra

medium

Codex

96%

OpenAI

GPT-6.1 Terra

low

Codex

96%

OpenAI

GPT-5.1 (Swiss inference)

high

Codex

72%

OpenAI

GPT-5.1 (Swiss inference)

high

pi

28%

OpenAI

GPT-5.1 (Swiss inference)

medium

Codex

63%

Anthropic

Claude Opus 5.5

medium

Claude Code

99%

PowerPoint generation (builds a deck in the bank’s own template). Client-ready slides: the share of the requested slides with no defect the user must fix before sending, on 8 requests on 4 company templates nobody tuned the setup on, graded blind slide by slide. Each harness runs with the setup currently in the product: GPT-5.1 on Codex with the optimised setup; GPT-5.1 on pi with the standard skill, because the optimised setup is not on pi yet; GPT-6.1 Sol, GPT-6.1 Terra and Claude Opus 5.5 with the standard skill. GPT-6.1 Terra ran on 12 November 2026 with the same requests and grading. On templates already reviewed with us, GPT-5.1 on Codex with the optimised setup reaches 97% at high and at medium.

Provider

Model

Reasoning effort

Harness

Web search

Internal search

OpenAI

GPT-6.1 Sol

medium

none (standard chat)

95%

95%

OpenAI

GPT-6.1 Sol

medium

Codex

93%

94%

OpenAI

GPT-6.1 Sol

medium

pi

94%

94%

OpenAI

GPT-5.1 (Swiss inference)

medium

none (standard chat)

91%

91%

OpenAI

GPT-5.1 (Swiss inference)

medium

Codex

88%

92%

OpenAI

GPT-5.1 (Swiss inference)

medium

pi

92%

91%

Anthropic

Claude Opus 5.5

medium

none (standard chat)

84%

89%

Anthropic

Claude Opus 5.5

auto

Claude Code

89%

92%

Anthropic

Claude Opus 5.5

medium

pi

88%

92%

Web search and internal search with Conduct. Core score in %: the mean of the LLM judges that score every answer (internal search: factual accuracy against the reference answer, intent, conciseness, tone; web search: factual accuracy against the retrieved pages, source quality, intent, conciseness, tone), on 50 web-search and 150 internal-search questions, November 2026. The rows without a harness are standard Unique AI chat in the same run. These runs use revised judges, so they differ from the Unique AI chat table above, which will be updated.

AI Workflows

Hallucination check

Provider

Model

Reasoning effort

Accuracy

OpenAI

GPT-6.1 Sol

low

91%

OpenAI

GPT-6.1 Sol

medium

91%

OpenAI

GPT-6.1 Sol

high

91%

OpenAI

GPT-6.1 Sol

xhigh

91%

OpenAI

GPT-6.1 Terra

low

89%

OpenAI

GPT-6 Sol

none

90%

OpenAI

GPT-6 Sol

medium

90%

OpenAI

GPT-5.1 (Swiss inference)

minimal

78%

OpenAI

GPT-5.1 (Swiss inference)

low

87%

OpenAI

GPT-5.1 (Swiss inference)

medium

87%

OpenAI

GPT-5.4-mini

none

72%

OpenAI

GPT-5.6 Luna

none

80%

OpenAI

GPT-5.6 Luna

low

85%

OpenAI

GPT-6 Luna

none

84%

OpenAI

GPT-5-mini

minimal

68%

Anthropic

Claude Opus 5.5

low

92%

Anthropic

Claude Sonnet 5.5

low

88%

Anthropic

Claude Sonnet 5

low

83%

Google

Gemini 4 Argon

low

91%

DeepSeek

DeepSeek V4.1 Flash

default

89%

Moonshot AI

Kimi K3

low

88%

Mistral AI

Mistral Medium 3

not applicable

67%

Accuracy is the share of 834 benchmark answers the judge model gets right: it warns when an answer contradicts a source the agent cited and stays quiet when the answer is correct. Details: Hallucination Check. Not scorable: Claude Opus 5 returned no verdict on 102 of the 834 answers.

Last updated