Internal Search Answer Quality
13 min read
Key facts
What we test: how well Unique AI chat answers questions from the bank's own documents (internal search). 150 questions, each with a reference (golden) answer: 69 questions on a fund due-diligence case (DDQ, fund terms, org charts) and 81 questions on EU and Swiss financial regulation, FINMA circulars, Swiss employment law, internal HR and expense policies and OECD / World Bank investment-policy reports.
How we test: every answer is graded by five LLM judges built for financial institutions: factual accuracy against the golden answer, interpretation of intent, conciseness, professional tone and recommended next steps. Twelve chat configurations answered the same 150 questions; DeepSeek V4.1 Flash, Gemini 4 Argon and GPT-6.1 Terra were added in November 2026.
Best configuration: GPT-6.1 Sol (thinking medium), overall score 94.4% [93.2–95.4%], significantly ahead of every other configuration (p < 0.001). It gives short answers (median 848 characters; only GPT-6.1 Terra's are shorter) and few materially wrong ones.
Materially wrong answers: about 3 in 100 answers with GPT-6.1 Sol (5/150) contradict the golden answer on a key fact, against about 8 in 100 with GPT-5.1 (Swiss inference, thinking medium: 12/149).
Claude is as accurate, but longer. Claude Sonnet 5.5, Claude Opus 5.5 and Claude Opus 5 reach the same factual accuracy as GPT-6.1 Sol (no significant difference, p ≥ 0.13). Claude Sonnet 5.5 and Claude Opus 5 are the most accurate on fund due diligence. Their answers are two to three times as long, which costs them on conciseness and puts them behind overall.
Swiss inference: GPT-5.1. When client-identifying data (CID) must stay in Switzerland, GPT-5.1 is the option. Its thinking level makes no significant difference to the answers (none, low, medium, high: p ≥ 0.18), but thinking none starts answering in a median of 5 s instead of 9 s.
DeepSeek V4.1 Flash: cheapest, but long. At USD 0.03 per answer it costs about half as much as GPT-5.1 and starts answering in a median of 4 s. Its factual accuracy (89.2%) is level with GPT-5.1 (thinking medium: +1.7 pp, p = 0.37) but significantly below GPT-6.1 Sol (−4.2 pp, p = 0.007); 7 of 149 answers are materially wrong. Its answers are the second longest (median 2,378 characters), which puts it last overall (86.5%).
Gemini 4 Argon: accurate, but long, slow and expensive. Its factual accuracy (91.7%) is level with GPT-6.1 Sol (−1.7 pp, p = 0.17) and above GPT-5.1 (thinking medium: +4.1 pp, p = 0.005). Seven of its eight materially wrong answers are on fund due diligence (89.4% there, 93.7% on the other questions). Its answers are the longest (median 3,639 characters), so it scores 86.8% overall, level with Claude Sonnet 5 and Claude Opus 5. It is the most expensive configuration (USD 0.40 per answer) and the slowest to start (median 16 s; 1 in 10 answers waits more than 2 minutes).
GPT-6.1 Terra: as accurate as GPT-6.1 Sol, at less than half the cost. Its factual accuracy (92.6%) is level with GPT-6.1 Sol (−0.9 pp, p = 0.24) and above GPT-5.1 (thinking medium: +5.0 pp, p < 0.001); 5 of 150 answers are materially wrong, as with GPT-6.1 Sol, all of them on fund due diligence (90.6% there, 94.3% on the other questions). It gives the shortest answers (median 638 characters), starts answering in a median of 6 s and costs USD 0.05 per answer, against USD 0.12 for GPT-6.1 Sol. Overall it is second at 92.5%, behind GPT-6.1 Sol (−1.9 pp, p < 0.001), mainly on intent (−3.1 pp). Against GPT-6.1 Sol scored by the same judges in November the gap is 0.8 pp (p = 0.012), with factual accuracy level (+0.2 pp, p = 0.84).
Caveat: the judge model for this run is GPT-6.1 Sol (thinking low), the same model family as the best chat configuration. Self-preference cannot be ruled out, especially on style (conciseness, tone). The factual-accuracy judge compares with a fixed golden answer and is less exposed.
Why this matters for relationship managers
Relationship managers (RMs) and their support teams use internal search to answer questions they would otherwise look up by hand: due-diligence questionnaires on a fund, fee terms, return targets, who sits on which committee, what the internal travel or expense policy allows, what a directive or FINMA circular requires. The answer often goes straight into a client e-mail, a DDQ response or a meeting note.
A wrong answer here is not a typo. It is a statement the bank makes to a client or a regulator:
A figure on the wrong basis. Asked for the gross target TVPI of a fund, GPT-5.1 answered 1.8x. The documents give 1.8x as a net figure and do not disclose a gross target.
A commitment the documents do not make. Asked whether the fund manager in the due-diligence case provides online reporting (Yes or No), GPT-5.1 answered Yes. The documents do not say so. Written into a DDQ, that is a promise to the client.
The wrong date. Asked for the fundraising start date of a fund, GPT-5.1 gave the first-closing date. The documents do not disclose the start date, and the first closing is a different event.
In all three cases GPT-6.1 Sol gave the correct answer (scores 90–100%). Every answer carries references, but an RM preparing a DDQ does not have the time to open every reference. This benchmark measures how often the answer itself is right.
What we test: five judges
Each answer is graded by five judges. Each judge sees the full conversation and the golden answer and returns a score from 0 to 100% with a justification. Pass is 75% or more, borderline 50–74%, fail below 50%. The overall score is the plain average of the five judges.
| Category | Judge | What it checks |
|---|---|---|
| Factual accuracy | Accuracy of Information | Are the key facts, figures, conclusions and relationships the same as in the golden answer? No contradictions, no fabricated claims. Wording does not matter. |
| Interpretation of intent | Intent Comprehension | Did the assistant understand what the user was trying to achieve, including implied goals, constraints and sub-requests? |
| Conciseness and relevance | Response Conciseness | Is the answer focused and actionable, without filler, repetition or tangents, but long enough to be useful? |
| Professional tone | Tone Appropriateness | Is the tone clear, respectful and approachable for a financial-services environment, neither too casual nor robotic? |
| Resource recommendation | Recommendation Appropriateness | When the assistant cannot fully resolve the question, does it suggest specific, practical next steps? Not applicable when the question is fully answered. |
The judges are written for financial institutions (banks, asset managers, insurers). The factual-accuracy judge is the one that matters most for compliance: it measures whether the RM can rely on the answer.
What to expect
Per 100 internal-search questions (95% Wilson intervals in brackets):
| Per 100 questions | GPT-6.1 Sol (thinking medium) | GPT-6.1 Terra (thinking medium) | Claude Sonnet 5.5 (thinking medium) | GPT-5.1, Swiss inference (thinking medium) |
|---|---|---|---|---|
| Correct (factual accuracy 75% or more) | 91 | 91 | 88 | 84 |
| Partly correct (50–74%) | 6 | 5 | 9 | 8 |
| Materially wrong (below 50%) | about 3 [1–8] | about 3 [1–8] | about 3 [1–8] | about 8 [5–14] |
| Time until the answer starts, median | 8 s | 6 s | 4 s | 9 s |
| Cost per answer | USD 0.12 | USD 0.05 | USD 0.11 | USD 0.07 |
Results
Overall score: average of the five judges, 95% bootstrap interval over questions. Judge columns: mean score, with the share of answers that pass in brackets. Same 150 questions for every configuration, one answer per question.
| Configuration | Overall | Factual accuracy | Intent | Conciseness | Tone | Next steps* |
|---|---|---|---|---|---|---|
| GPT-6.1 Sol, thinking medium | 94.4% [93.2–95.4%] | 93.4% (91%) | 97.6% (99%) | 93.7% (100%) | 95.8% (100%) | 65.5% (44%, n = 25) |
| GPT-6.1 Terra, thinking medium | 92.5% [91.4–93.5%] | 92.6% (91%) | 94.5% (97%) | 94.7% (99%) | 95.4% (100%) | 60.2% (39%, n = 44) |
| GPT-5.1, thinking high | 91.1% [89.8–92.3%] | 89.3% (86%) | 95.8% (98%) | 86.7% (83%) | 93.6% (100%) | 69.5% (50%, n = 16) |
| GPT-5.1, thinking medium | 90.6% [89.0–92.0%] | 87.5% (84%) | 95.2% (97%) | 86.8% (89%) | 93.8% (100%) | 63.9% (50%, n = 14) |
| GPT-5.1, thinking low | 89.7% [88.2–91.1%] | 86.9% (83%) | 94.6% (96%) | 84.6% (83%) | 93.4% (100%) | 68.4% (50%, n = 16) |
| GPT-5.1, thinking none | 89.6% [88.1–91.0%] | 87.4% (85%) | 94.9% (97%) | 83.5% (79%) | 93.3% (100%) | 69.3% (56%, n = 16) |
| Claude Sonnet 5.5, thinking medium | 89.4% [88.5–90.2%] | 91.7% (88%) | 96.3% (100%) | 77.1% (57%) | 92.1% (100%) | 86.8% (93%, n = 57) |
| Claude Opus 5.5, thinking medium | 88.1% [87.3–88.9%] | 91.6% (89%) | 96.4% (99%) | 73.3% (34%) | 90.8% (100%) | 86.7% (91%, n = 47) |
| Claude Sonnet 5, thinking medium | 86.8% [85.4–87.9%] | 88.9% (85%) | 94.4% (95%) | 75.2% (51%) | 88.7% (100%) | 73.2% (71%, n = 17) |
| Claude Opus 5, thinking medium | 86.8% [85.8–87.6%] | 91.6% (89%) | 95.8% (98%) | 70.4% (25%) | 88.3% (100%) | 86.7% (94%, n = 35) |
| Gemini 4 Argon, thinking medium | 86.8% [85.8–87.7%] | 91.7% (91%) | 94.1% (99%) | 71.5% (31%) | 91.6% (100%) | 58.6% (15%, n = 13) |
| DeepSeek V4.1 Flash, thinking default | 86.5% [85.2–87.7%] | 89.2% (85%) | 95.0% (96%) | 71.0% (32%) | 90.7% (100%) | 80.1% (81%, n = 36) |
* Next steps is scored only on answers the judge considered not fully resolved, so each configuration is scored on a different, small set (13 to 57 answers). It is not comparable across configurations and pulls the overall score down most for the configurations where it was scored often.
Factual accuracy in detail
| Configuration | Materially wrong (below 50%) | Partly correct (50–74%) | Fund due diligence (69 questions), mean | Other questions (81), mean | Answer length, median |
|---|---|---|---|---|---|
| GPT-6.1 Sol, thinking medium | 3.3% (5/150) [1.4–7.6%] | 9 | 91.4% (5 wrong) | 95.1% | 848 characters |
| GPT-6.1 Terra, thinking medium | 3.3% (5/150) [1.4–7.6%] | 8 | 90.6% (5 wrong) | 94.3% | 638 |
| GPT-5.1, thinking high | 5.3% (8/150) [2.7–10.2%] | 13 | 89.3% (6 wrong) | 89.3% | 992 |
| GPT-5.1, thinking medium | 8.1% (12/149) [4.7–13.5%] | 12 | 86.6% (9 wrong) | 88.2% | 950 |
| GPT-5.1, thinking low | 8.1% (12/149) [4.7–13.5%] | 13 | 86.4% (8 wrong) | 87.3% | 1,119 |
| GPT-5.1, thinking none | 7.3% (11/150) [4.1–12.7%] | 12 | 86.2% (7 wrong) | 88.3% | 1,252 |
| Claude Sonnet 5.5, thinking medium | 3.3% (5/150) [1.4–7.6%] | 13 | 93.0% (2 wrong) | 90.5% | 1,927 |
| Claude Opus 5.5, thinking medium | 4.0% (6/149) [1.9–8.5%] | 11 | 90.3% (4 wrong) | 92.7% | 2,156 |
| Claude Sonnet 5, thinking medium | 6.0% (9/149) [3.2–11.1%] | 13 | 88.9% (6 wrong) | 88.9% | 1,932 |
| Claude Opus 5, thinking medium | 2.7% (4/150) [1.0–6.7%] | 12 | 92.8% (1 wrong) | 90.6% | 2,475 |
| Gemini 4 Argon, thinking medium | 5.3% (8/150) [2.7–10.2%] | 6 | 89.4% (7 wrong) | 93.7% | 3,639 |
| DeepSeek V4.1 Flash, thinking default | 4.7% (7/149) [2.3–9.4%] | 15 | 90.0% (3 wrong) | 88.4% | 2,378 |
On fund due diligence, the question type closest to an RM's daily work, Claude Sonnet 5.5 and Claude Opus 5 are the most accurate (93.0% and 92.8%, with two and one materially wrong answers out of 69). GPT-6.1 Sol leads on the other questions (95.1%).
Time and cost
| Configuration | Time until the answer starts, median / 90th percentile | Cost per answer (estimated) |
|---|---|---|
| GPT-6.1 Sol, thinking medium | 8 s / 18 s | USD 0.12 |
| GPT-6.1 Terra, thinking medium | 6 s / 12 s | USD 0.05 |
| GPT-5.1, thinking high | 9 s / 20 s | USD 0.08 |
| GPT-5.1, thinking medium | 9 s / 22 s | USD 0.07 |
| GPT-5.1, thinking low | 8 s / 18 s | USD 0.06 |
| GPT-5.1, thinking none | 5 s / 8 s | USD 0.06 |
| Claude Sonnet 5.5, thinking medium | 4 s / 8 s | USD 0.11 |
| Claude Opus 5.5, thinking medium | 6 s / 13 s | USD 0.25 |
| Claude Sonnet 5, thinking medium | 4 s / 6 s | USD 0.12 |
| Claude Opus 5, thinking medium | 6 s / 11 s | USD 0.37 |
| Gemini 4 Argon, thinking medium | 16 s / 128 s | USD 0.40 |
| DeepSeek V4.1 Flash, thinking default | 4 s / 6 s | USD 0.03 |
The time is measured until the answer starts streaming (search included), not until it is complete. Longer answers take longer to finish, so the Claude models lose part of their head start.
Statistical tests
Paired sign-flip permutation tests on the same questions (20,000 permutations, two-sided). Differences in percentage points (pp):
| Compared with GPT-6.1 Sol, thinking medium | Overall score | Factual accuracy |
|---|---|---|
| GPT-6.1 Terra, thinking medium | significantly lower (−1.9 pp; p < 0.001) | no significant difference (−0.9 pp; p = 0.24) |
| GPT-5.1, thinking high | significantly lower (−3.3 pp; p < 0.001) | significantly lower (−4.1 pp; p < 0.001) |
| GPT-5.1, thinking medium | significantly lower (−3.8 pp; p < 0.001) | significantly lower (−5.9 pp; p < 0.001) |
| GPT-5.1, thinking low | significantly lower (−4.7 pp; p < 0.001) | significantly lower (−6.5 pp; p < 0.001) |
| GPT-5.1, thinking none | significantly lower (−4.8 pp; p < 0.001) | significantly lower (−6.0 pp; p < 0.001) |
| Claude Sonnet 5.5, thinking medium | significantly lower (−5.0 pp; p < 0.001) | no significant difference (−1.8 pp; p = 0.18) |
| Claude Opus 5.5, thinking medium | significantly lower (−6.3 pp; p < 0.001) | no significant difference (−1.8 pp; p = 0.13) |
| Claude Sonnet 5, thinking medium | significantly lower (−7.6 pp; p < 0.001) | significantly lower (−4.5 pp; p = 0.003) |
| Claude Opus 5, thinking medium | significantly lower (−7.6 pp; p < 0.001) | no significant difference (−1.8 pp; p = 0.14) |
| Gemini 4 Argon, thinking medium | significantly lower (−7.6 pp; p < 0.001) | no significant difference (−1.7 pp; p = 0.17) |
| DeepSeek V4.1 Flash, thinking default | significantly lower (−7.9 pp; p < 0.001) | significantly lower (−4.2 pp; p = 0.007) |
GPT-5.1 thinking levels against thinking medium, overall score: high +0.5 pp (p = 0.47), low −0.9 pp (p = 0.24), none −1.0 pp (p = 0.20); factual accuracy: high +1.7 pp (p = 0.18), low −0.6 pp (p = 0.72), none −0.2 pp (p = 0.91). None of the differences is significant.
Without the conciseness judge GPT-6.1 Sol still leads (94.7%), but the gap to Claude Sonnet 5.5 (93.1%, p = 0.01) and Claude Opus 5.5 (92.7%, p = 0.002) shrinks to about 2 percentage points. Most of the Claude deficit is answer length. DeepSeek V4.1 Flash reaches 91.4% without it, level with GPT-5.1, Gemini 4 Argon 91.8% and GPT-6.1 Terra 91.9%.
Examples
Gross vs net (fund due diligence). Question: what is the target TVPI (gross) of the fund in the due-diligence case?
GPT-5.1 (thinking medium): "1.8x". Score 20%.
Golden answer: the gross TVPI is not disclosed; the target is ">1.8x after seven years", and the separate 1.8x figure is net to investors.
GPT-6.1 Sol: score 90%.
An unsupported Yes (DDQ). Question: does the fund manager in the due-diligence case provide online reporting? Choose from: Yes or No.
GPT-5.1 (thinking medium): "Yes". Score 0%.
Golden answer: not stated in the knowledge base.
GPT-6.1 Sol: score 100%.
Wrong document (regulation). Question: summarise the changes in the scope of the authorisation.
GPT-5.1 (thinking medium) summarised FINMA licensing and stock-exchange approvals. Score 10%.
Golden answer: Article 10 of the AIFMD (notification of material changes to the home Member State, one-month assessment period).
GPT-6.1 Sol: score 100%.
Missing the document. Question: what was CPT sold for?
GPT-5.1 (thinking medium): the internal documents contain no record of a CPT sale. Score 5%.
Golden answer: Compañía Peruana de Teléfonos, sold for USD 2 billion (OECD Policy Framework for Investment review, 2006, page 18).
GPT-6.1 Sol: score 95%.
Next steps
Rule out self-preference: rerun the judges with a model from another family (for example Claude Opus 5.5) and compare the ranking.
Shorter Claude answers: test a system prompt that asks for concise answers. Three of the four Claude models are already as accurate as GPT-6.1 Sol, and Claude Sonnet 5.5 and Claude Opus 5 are more accurate on fund due diligence.
Repeat runs: each question was answered once. Repeat runs measure the run-to-run noise.
GPT-6.1 Sol thinking levels: only medium was tested.
Method notes
Benchmark: the Unique benchmarking platform, specification "Internal Search Judges" (exported 27 October 2026; DeepSeek V4.1 Flash exported 3 November 2026, Gemini 4 Argon 7 November 2026, GPT-6.1 Terra 12 November 2026). 150 conversations, 149 distinct questions (one asked twice), one answer per question and configuration, October and November 2026.
Chat configurations: GPT-6.1 Sol (deployment of 29 September 2026), thinking medium; GPT-5.1 (deployment of 13 November 2025, Swiss inference), thinking none, low, medium and high; Claude Opus 5.5, Claude Opus 5, Claude Sonnet 5.5 and Claude Sonnet 5 through LiteLLM (Anthropic), thinking medium; DeepSeek V4.1 Flash through LiteLLM, default thinking level (the platform is set to medium, but whether the provider applies it is not verified; it returned reasoning tokens on 142 of 150 answers); Gemini 4 Argon through LiteLLM (Google Vertex AI), thinking medium (reasoning tokens on 113 of 150 answers); GPT-6.1 Terra (deployment of 26 October 2026), thinking medium (reasoning tokens on 82 of 150 answers). Gemini 4 Argon and GPT-6.1 Terra ran in their own benchmark spaces. Gemini's settings were not checked against the standard ones; GPT-6.1 Terra's match the GPT-6.1 Sol space of the November runs apart from the model. All other configurations use the standard space settings; see <![CDATA[Chat configuration]]> for loop limits, chunk limits and token limits.
Judge model: GPT-6.1 Sol (deployment of 29 September 2026), thinking low, for all five judges. Each judge receives the full conversation and the golden answer. The Gemini 4 Argon and GPT-6.1 Terra exports have a sixth judge (grounding against the retrieved sources) that the earlier exports do not have; it is left out so that every configuration is scored on the same five judges.
October and November judges: GPT-6.1 Terra was scored with the judges of the November runs. In the run of 5 November, scored the same way, GPT-6.1 Sol reaches 93.3% overall (94.4% here); against that run GPT-6.1 Terra is 0.8 pp behind (p = 0.012) rather than 1.9 pp, with factual accuracy level (+0.2 pp, p = 0.84).
Scores: 0 to 100% per judge; pass 75% or more, borderline 50–74%, fail below 50%. The overall score is the unweighted average of the judges that scored the answer. Not-applicable verdicts (next steps on fully resolved questions) are left out.
Missing answers: four configurations returned no answer for one question each (GPT-5.1 low and medium, Claude Opus 5.5, Claude Sonnet 5). These are left out of the scores. DeepSeek V4.1 Flash answered one question (the entity to be invoiced in the due-diligence case) with a search call only and no text; the accuracy and tone judges marked it not applicable.
Intervals and tests: 95% bootstrap intervals over questions (5,000 resamples) for mean scores, Wilson intervals for proportions, paired sign-flip permutation tests for differences.
Cost: estimated per answer from token counts, chat model only (judging not included). Answers without token counts in the export (up to 8 per configuration) are left out.
Fund due diligence: the questions of the due-diligence case (fund manager, fund and DDQ).
Limitation: all scores come from LLM judges. The factual-accuracy judge is only as good as the golden answers.