Unique AI Chat Benchmarks

3 min read

Measured answer quality of Unique AI chat in three benchmarks: internal search (answers from the bank's own documents), web search and library answers (approved answers from the answer library). For internal and web search, five LLM judges built for financial institutions grade every answer, and twelve chat configurations (GPT-6.1 Sol, GPT-6.1 Terra, GPT-5.1 at four thinking levels, four Claude models, DeepSeek V4.1 Flash and Gemini 4 Argon) answered the same questions in October and November 2026.

Questions answered from the bank's own documents: fund due diligence (DDQs, fund terms, org charts), EU and Swiss financial regulation, internal HR and expense policies. Each answer is compared with a golden answer.

What relationship managers get. With GPT-6.1 Sol (thinking medium), 91 in 100 answers match the golden answer and about 3 in 100 are materially wrong. With GPT-5.1 (Swiss inference) about 8 in 100 are materially wrong, for example a net return target given as gross, or a "Yes" in a DDQ that the documents do not support. Three of the four Claude models (Claude Sonnet 5.5, Claude Opus 5.5 and Claude Opus 5) are as accurate as GPT-6.1 Sol, but all four answer two to three times as long. DeepSeek V4.1 Flash is the cheapest (USD 0.03 per answer) and about as accurate as GPT-5.1, but its long answers put it last overall. Gemini 4 Argon is about as accurate as GPT-6.1 Sol, but gives the longest answers and is the most expensive (USD 0.40 per answer) and slowest configuration. GPT-6.1 Terra is as accurate as GPT-6.1 Sol (5 of 150 answers materially wrong, as with Sol), gives the shortest answers and costs less than half as much (USD 0.05 per answer).

Chat configurations compared on 150 internal-search questions (October and November 2026):

chat_internal_score_vs_cost.pngchat_internal_score_vs_time.pngchat_internal_accuracy_vs_length.png

Questions that need a live web search: central-bank rates, FX, prices, company figures, financial regulation and current events. Each answer is checked against the web sources it retrieved.

What relationship managers get. GPT-6.1 Terra (thinking medium) scores highest (96.2%): sources as strong as GPT-6.1 Sol's, the shortest answers and about half Sol's cost (USD 0.06 per answer). GPT-6.1 Sol (thinking medium) follows with 94.0% against 88.0% for GPT-5.1 (Swiss inference). When the search returns nothing, GPT-6.1 Sol says so; GPT-5.1 usually answers from its 2024 training data, for example a Fed Funds range of 5.25–5.50% instead of the current 3.75–4.00%. Claude answers are four to six times as long. DeepSeek V4.1 Flash is the cheapest (USD 0.04 per answer) and fully grounded, but its answers are as long as Claude's. Gemini 4 Argon (85.1%) lands between GPT-5.1 and Claude, with the longest answers and the slowest start (median 20 s).

Chat configurations compared on 50 web-search questions (October and November 2026):

chat_web_score_vs_cost.pngchat_web_score_vs_time.pngchat_web_sources_vs_length.png

Library answers

Questions answered from the bank's approved answer library in Agentic Table, for example RFP and DDQ answers. Each answer must be the approved answer from the right library entry, or a refusal when the library has no answer.

Chat configuration

The first ten configurations ran with the standard Unique AI chat space settings. Only the language model and the thinking level differ between them. Gemini 4 Argon ran in its own benchmark space, whose settings were not checked against these. GPT-6.1 Terra also ran in its own space; apart from the model, its settings match the GPT-6.1 Sol space of the November runs. The key settings:

Area

Setting

Agent loop

At most 8 loop iterations per answer and 10 tool calls per iteration. 20% of the model's input window is reserved for the conversation history.

Tools

Internal Search, Web Search, Deep Research and Code Interpreter enabled. Hallucination check, to-do list, open-file tool and skills off.

Internal Search

Combined vector and full-text search, up to 10 search strings per call and 40 chunks per search. Sources are capped at 30,000 tokens. No reranker, no relevance threshold.

Web Search

Mode v1: one search round per tool call, without planning steps (a step limit applies only in mode v2). Google, up to 5 queries per call and 10 results per query. HTML pages only, no PDFs. Each page is cut at 10,000 tokens and split into chunks of 1,000 tokens. Sources are capped at 12,000 tokens per call.

Last updated