Web Search Answer Quality
14 min read
Key facts
What we test: how well Unique AI chat answers questions that need a live web search. 50 questions: 30 on markets and finance (central-bank rates, FX, index levels, commodity and share prices, company figures, financial regulation) and 20 on current events (sport, culture, festivals) that change over time.
How we test: every answer is graded by five LLM judges built for financial institutions: factual accuracy (is every claim backed by the retrieved web sources?), source quality, interpretation of intent, conciseness and professional tone. Twelve chat configurations answered the same 50 questions; DeepSeek V4.1 Flash, Gemini 4 Argon and GPT-6.1 Terra were added in November 2026.
Best configuration: GPT-6.1 Terra (thinking medium), overall score 96.2% [95.6–96.7%], significantly ahead of every other configuration, GPT-6.1 Sol included (+2.2 pp, p < 0.001). Its sources are as good as GPT-6.1 Sol's (91.7% against 92.5%, p = 0.41), its answers are the shortest (median 165 characters; the conciseness judge passes all 50) and it costs about half as much (USD 0.06 per answer against USD 0.11). Part of the lead comes from the November judges: against GPT-6.1 Sol scored the same way it is 1.1 pp (p = 0.002), see Method notes.
GPT-6.1 Sol (thinking medium) follows at 94.0% [92.4–95.0%], significantly ahead of all other configurations (p < 0.001), thanks to better sources (92.5% against 84% or less) and short, focused answers (median 275 characters).
Grounding is high for all configurations. Almost every answer is backed by its sources (94–100%), with no significant difference between configurations. The differences are in the edge cases: GPT-6.1 Terra and GPT-6.1 Sol had no answer whose main claim was unsupported or contradicted; GPT-5.1 had up to two per 50, for example a Fed Funds range from 2024 presented as current.
When the search returns nothing, GPT-6.1 Sol and Claude Sonnet 5.5 say so and point to the official source. GPT-5.1 usually falls back on its training knowledge from 2024 (5 of 6 cases), for example the Fed Funds target range of 5.25–5.50% instead of the current 3.75–4.00%.
Claude: correct, but too long. Claude answers are four to six times as long as GPT-6.1 Sol's (median 1,230–1,570 characters). The conciseness judge passes only 0–6% of them. Without the conciseness judge, Claude Sonnet 5.5 is level with GPT-5.1.
DeepSeek V4.1 Flash: cheapest and grounded, but long. At USD 0.04 per answer, every answer is backed by its sources (100%) and no search came back empty. Its answers are as long as Claude's (median 1,535 characters; the conciseness judge passes 1 of 50) and its sources are weaker than GPT-6.1 Sol's (80.8% against 92.5%, p < 0.001), so it scores 82.8% overall, between Claude Sonnet 5.5 and Claude Sonnet 5.
Gemini 4 Argon: understands the question, but long and slow. It scores 85.1% overall, between GPT-5.1 (thinking none, 86.7%) and Claude Sonnet 5.5, and significantly below GPT-5.1 thinking medium (−2.9 pp, p = 0.021). Its sources are as good as GPT-5.1's (83.1%, p = 0.75) and it reads intent better (96.8%, +3.7 pp, p = 0.016), but its answers are the longest (median 1,967 characters; the conciseness judge passes 3 of 50). It is the slowest to start (median 20 s, 90th percentile 75 s) and costs USD 0.15 per answer.
Swiss inference: GPT-5.1. The thinking level makes no significant difference (none, low, medium, high: p ≥ 0.29).
Caveat: 50 questions, one answer each. The judge model is recorded only in the DeepSeek V4.1 Flash, Gemini 4 Argon and GPT-6.1 Terra exports (GPT-6.1 Sol, thinking low).
Why this matters for relationship managers
Relationship managers (RMs) use web search to prepare client conversations: where the SNB policy rate stands, the EUR/USD rate this morning, the latest Apple close, what MiCA or SFDR require. The answer has to be current, it has to come from a trustworthy source, and it has to be short enough to use in the five minutes before a call.
Three risks are specific to web search:
Stale data presented as current. Search results contain old articles. An answer that quotes the 2024 Fed Funds range as "current" is wrong, even though a source supports the number.
Weak sources. A figure from an aggregator, a forum or an SEO page is not a figure an RM can quote to a client. Central banks, regulators, exchanges and company releases are.
Answering without searching. When the search fails, a model can answer from memory. Its training data is months or years old, and the RM cannot tell the difference.
The Hallucination Check catches contradictions between an answer and its sources after the turn. This benchmark measures the quality of the answer before any check: how current, how well sourced, how focused.
What we test: five judges
Each answer is graded by five judges. Factual accuracy is scored from 0 to 1 and shown here as 0 to 100% (well grounded from 80%, poorly grounded below 40%). The other judges score from 0 to 100% (pass from 75%, fail below 50%). The overall score is the plain average of the five judges.
| Category | Judge | What it checks |
|---|---|---|
| Factual accuracy | Accuracy of Information | Is every factual claim in the latest answer supported by the retrieved web sources, and does each citation point to a source that actually supports the claim? Contradictions, overstatements and wrong citations reduce the score. No outside knowledge. |
| Source quality | Quality of References | Are the cited sources authoritative, direct, current and appropriate for the claim? Primary sources (central banks, regulators, company sites, official documentation) score higher; aggregators, user-generated content and SEO pages lower. |
| Interpretation of intent | Intent Comprehension | Did the assistant understand what the user wanted, including implied needs, knowing that it has live web search? |
| Conciseness and relevance | Response Conciseness | Does every sentence contribute to the user's goal? Repetition, preambles, unnecessary caveats and secondary detail reduce the score. |
| Professional tone | Empathetic and Professional Tone | Is the tone professional, natural and approachable for a financial-services environment, without generic or overly formal phrasing? |
The judges apply a demanding standard: scores above 90% are reserved for answers with virtually no weaknesses. Source tags such as [source3] are ignored when judging tone and conciseness.
What to expect
Per 50 web-search questions:
| Per 50 questions | GPT-6.1 Terra (thinking medium) | GPT-6.1 Sol (thinking medium) | Claude Sonnet 5.5 (thinking medium) | GPT-5.1, Swiss inference (thinking medium) |
|---|---|---|---|---|
| Main claim unsupported or contradicted by the sources | 0 | 0 | 0 | 1 |
| Search returned nothing: the answer says so | no empty search | 1 of 1 | 1 of 1 | 1 of 2 |
| Search returned nothing: answered from training knowledge | 0 | 0 | 0 | 1 |
| Answers the conciseness judge passes | 50 | 49 | 3 | 41 |
| Answers with high-quality sources (75% or more) | 48 of 50 | 48 of 49 | 42 of 49 | 38 of 46 |
| Time until the answer starts, median | 12 s | 13 s | 11 s | 13 s |
| Cost per answer | USD 0.06 | USD 0.11 | USD 0.15 | USD 0.07 |
Results
Overall score: average of the five judges, 95% bootstrap interval over questions. Judge columns: mean score, with the share of answers that pass in brackets. Same 50 questions for every configuration, one answer per question.
| Configuration | Overall | Factual accuracy (grounding) | Source quality | Intent | Conciseness | Tone | Finance questions only (30) |
|---|---|---|---|---|---|---|---|
| GPT-6.1 Terra, thinking medium | 96.2% [95.6–96.7%] | 99.5% (100%) | 91.7% (96%) | 97.6% (100%) | 97.3% (100%) | 94.7% (100%) | 96.4% |
| GPT-6.1 Sol, thinking medium | 94.0% [92.4–95.0%] | 100% (100%) | 92.5% (98%) | 97.0% (98%) | 91.3% (98%) | 90.4% (100%) | 94.8% |
| GPT-5.1, thinking high | 88.9% [87.1–90.4%] | 100% (100%) | 84.0% (83%) | 95.0% (98%) | 81.3% (86%) | 85.6% (100%) | 89.3% |
| GPT-5.1, thinking medium | 88.0% [85.6–90.1%] | 97.9% (98%) | 84.0% (83%) | 93.2% (92%) | 81.0% (82%) | 86.4% (100%) | 88.6% |
| GPT-5.1, thinking low | 86.8% [84.1–89.0%] | 95.9% (96%) | 83.2% (83%) | 92.8% (90%) | 78.2% (74%) | 85.2% (100%) | 88.2% |
| GPT-5.1, thinking none | 86.7% [84.1–89.0%] | 95.8% (96%) | 82.7% (85%) | 93.0% (94%) | 78.8% (84%) | 85.5% (100%) | 87.8% |
| Gemini 4 Argon, thinking medium | 85.1% [84.5–85.7%] | 98.5% (100%) | 83.1% (84%) | 96.8% (100%) | 66.9% (6%) | 80.1% (100%) | 85.9% |
| Claude Sonnet 5.5, thinking medium | 83.9% [82.9–84.8%] | 100% (100%) | 83.9% (86%) | 95.2% (100%) | 59.8% (6%) | 81.6% (100%) | 85.0% |
| DeepSeek V4.1 Flash, thinking default | 82.8% [81.9–83.7%] | 100% (100%) | 80.8% (86%) | 95.5% (100%) | 56.7% (2%) | 81.0% (100%) | 83.0% |
| Claude Sonnet 5, thinking medium | 82.7% [81.9–83.5%] | 100% (100%) | 80.3% (78%) | 94.9% (98%) | 59.9% (2%) | 78.5% (90%) | 83.3% |
| Claude Opus 5.5, thinking medium | 82.5% [81.7–83.4%] | 100% (100%) | 81.0% (82%) | 95.4% (100%) | 55.1% (2%) | 80.9% (98%) | 83.5% |
| Claude Opus 5, thinking medium | 79.3% [77.5–80.8%] | 93.6% (94%) | 81.3% (91%) | 94.2% (100%) | 51.4% (0%) | 78.3% (96%) | 80.6% |
On the 30 finance questions the ranking is the same as on all 50, except in the middle: Claude Opus 5.5, Claude Sonnet 5 and DeepSeek V4.1 Flash are within 0.5 percentage points of each other (83.5%, 83.3% and 83.0%).
Gemini 4 Argon's factual-accuracy scores are graded (21 of 50 answers between 91% and 99%, none lower), while all earlier exports contain only 0% or 100%. The judge's scale changed between exports, so its 98.5% is not directly comparable with the others; it lowers Gemini's overall score by at most 0.3 percentage points. GPT-6.1 Terra's scores are graded as well (7 of 50 answers between 94% and 98%), which lowers its overall score by at most 0.1 percentage points.
Search failures, ungrounded answers and length
| Configuration | Search returned nothing | Of which answered from training knowledge | Main claim unsupported or contradicted | Answer length, median |
|---|---|---|---|---|
| GPT-6.1 Terra, thinking medium | 0 | 0 | 0 | 165 characters |
| GPT-6.1 Sol, thinking medium | 1 | 0 | 0 | 275 |
| GPT-5.1, thinking high | 1 | 1 | 0 | 572 |
| GPT-5.1, thinking medium | 2 | 1 | 1 | 503 |
| GPT-5.1, thinking low | 1 | 1 | 2 | 631 |
| GPT-5.1, thinking none | 2 | 2 | 2 | 624 |
| Gemini 4 Argon, thinking medium | 0 | 0 | 0 | 1,967 |
| Claude Sonnet 5.5, thinking medium | 1 | 0 | 0 | 1,230 |
| DeepSeek V4.1 Flash, thinking default | 0 | 0 | 0 | 1,535 |
| Claude Sonnet 5, thinking medium | 0 | 0 | 0 | 1,232 |
| Claude Opus 5.5, thinking medium | 0 | 0 | 0 | 1,450 |
| Claude Opus 5, thinking medium | 3 | 1 (flagged as unverified) | 3 (wrong citation numbers) | 1,566 |
"Search returned nothing" means no web sources were retrieved, so grounding and source quality do not apply; these answers are left out of those two judges. "Main claim unsupported or contradicted" means a grounding score below 40%.
Time and cost
| Configuration | Time until the answer starts, median / 90th percentile | Cost per answer (estimated) |
|---|---|---|
| GPT-6.1 Terra, thinking medium | 12 s / 16 s | USD 0.06 |
| GPT-6.1 Sol, thinking medium | 13 s / 29 s | USD 0.11 |
| GPT-5.1, thinking high | 15 s / 31 s | USD 0.08 |
| GPT-5.1, thinking medium | 13 s / 24 s | USD 0.07 |
| GPT-5.1, thinking low | 13 s / 26 s | USD 0.08 |
| GPT-5.1, thinking none | 13 s / 22 s | USD 0.08 |
| Gemini 4 Argon, thinking medium | 20 s / 75 s | USD 0.15 |
| Claude Sonnet 5.5, thinking medium | 11 s / 22 s | USD 0.15 |
| DeepSeek V4.1 Flash, thinking default | 12 s / 24 s | USD 0.04 |
| Claude Sonnet 5, thinking medium | 12 s / 25 s | USD 0.16 |
| Claude Opus 5.5, thinking medium | 12 s / 33 s | USD 0.31 |
| Claude Opus 5, thinking medium | 14 s / 31 s | USD 0.55 |
The time is measured until the answer starts streaming, web search included. The search itself takes most of it, so the configurations are close.
Statistical tests
Paired sign-flip permutation tests on the same questions (20,000 permutations, two-sided). Differences in percentage points (pp):
| Compared with GPT-6.1 Sol, thinking medium | Overall score | Source quality | Factual accuracy (grounding) |
|---|---|---|---|
| GPT-6.1 Terra, thinking medium | significantly higher (+2.2 pp; p < 0.001) | no significant difference (−0.9 pp; p = 0.41) | not comparable (−0.5 pp; graded scores, see Results) |
| GPT-5.1, thinking high | significantly lower (−5.1 pp; p < 0.001) | significantly lower (−8.6 pp; p < 0.001) | no significant difference (0.0 pp; p = 1.0) |
| GPT-5.1, thinking medium | significantly lower (−6.0 pp; p < 0.001) | significantly lower (−8.6 pp; p < 0.001) | no significant difference (−2.1 pp; p = 1.0) |
| GPT-5.1, thinking low | significantly lower (−7.2 pp; p < 0.001) | significantly lower (−9.4 pp; p < 0.001) | no significant difference (−4.1 pp; p = 0.50) |
| GPT-5.1, thinking none | significantly lower (−7.3 pp; p < 0.001) | significantly lower (−9.8 pp; p < 0.001) | no significant difference (−4.2 pp; p = 0.51) |
| Gemini 4 Argon, thinking medium | significantly lower (−8.9 pp; p < 0.001) | significantly lower (−9.1 pp; p < 0.001) | not comparable (−1.5 pp; graded scores, see Results) |
| Claude Sonnet 5.5, thinking medium | significantly lower (−10.1 pp; p < 0.001) | significantly lower (−8.6 pp; p < 0.001) | no significant difference (0.0 pp; p = 1.0) |
| DeepSeek V4.1 Flash, thinking default | significantly lower (−11.2 pp; p < 0.001) | significantly lower (−11.6 pp; p < 0.001) | no significant difference (0.0 pp; p = 1.0) |
| Claude Sonnet 5, thinking medium | significantly lower (−11.3 pp; p < 0.001) | significantly lower (−12.1 pp; p < 0.001) | no significant difference (0.0 pp; p = 1.0) |
| Claude Opus 5.5, thinking medium | significantly lower (−11.5 pp; p < 0.001) | significantly lower (−11.2 pp; p < 0.001) | no significant difference (0.0 pp; p = 1.0) |
| Claude Opus 5, thinking medium | significantly lower (−14.7 pp; p < 0.001) | significantly lower (−11.2 pp; p < 0.001) | no significant difference (−6.4 pp; p = 0.25) |
GPT-5.1 thinking levels against thinking medium, overall score: high +0.9 pp (p = 0.41), low −1.2 pp (p = 0.30), none −1.3 pp (p = 0.29). None of the differences is significant.
Without the conciseness judge GPT-6.1 Terra leads (95.9%; +1.1 pp over GPT-6.1 Sol, p = 0.018), and GPT-6.1 Sol (94.7%) stays ahead of GPT-5.1 high (90.9%), Claude Sonnet 5.5 (90.1%) and GPT-5.1 medium (90.0%); all differences to GPT-6.1 Sol remain significant (p < 0.001). Source quality and intent keep GPT-6.1 Sol ahead. DeepSeek V4.1 Flash reaches 89.3% without it, level with Claude Opus 5.5, and Gemini 4 Argon 89.6%.
Examples
Stale rate presented as current. Question: the current target range for the US Federal Funds Rate.
GPT-5.1 (thinking low): "the current target range for the U.S. federal funds rate is 4.75% to 5.00% following a 50-basis-point cut announced by the Federal Reserve on September 18, 2024".
Retrieved sources: the move of 16 September 2026 brought the range to 3.75–4.00%. The cited 2024 article is real, but no longer current. Grounding score 0%.
GPT-6.1 Sol, GPT-5.1 with thinking medium and high, and all four Claude models gave the current range of 3.75–4.00%.
Answering from memory when the search fails. Same question, GPT-5.1 (thinking none), search returned no sources:
"As of my last built-in knowledge update (late 2024), the Federal Reserve had the target range at 5.25% to 5.50%. However, this may have changed since then."
The range was already out of date in late 2024. An RM skimming the answer takes away a figure that is 1.5 percentage points too high.
On the one question where GPT-6.1 Sol's search also returned nothing (the most recent UEFA Champions League final), it answered: "the search tool returned no relevant sources, so I couldn't verify", with a link to the official site.
Right citation, wrong conclusion. Question: which film was the highest-grossing worldwide last year?
GPT-5.1 (thinking none, low and medium): "Zootopia 2", citing Disney's release that it is the "#1 global MPA release of 2025".
Other retrieved sources show Ne Zha 2 grossing more in 2025. "Biggest MPA release" is not "biggest film worldwide". Grounding score 0%.
Unsupported figure. Question: Tesla's latest quarterly deliveries.
GPT-5.1 (thinking none): "Delivered 341,000 vehicles in the third quarter of 2026", attributed to Tesla's official release.
The cited Reuters snippet mentions third-quarter deliveries but gives no total. Grounding score 0%.
Wrong citation numbers (Claude Opus 5). In three answers (next FIFA World Cup host, Fed Funds range, Coachella headliners) the content was largely correct, but most [sourceN] tags pointed to the wrong source. For an RM the reference then does not show where the figure comes from.
Next steps
More finance questions with golden answers. The grounding judge checks the answer against the retrieved sources, not against the truth: a stale source can pass. A golden answer per question, as in internal search, measures correctness directly.
Shorter Claude answers: test a system prompt that asks for concise answers.
No answers from memory: instruct GPT-5.1 not to answer time-sensitive questions from training knowledge when the search fails.
Repeat runs and a judge from another model family, to measure run-to-run noise and rule out self-preference.
Method notes
Benchmark: the Unique benchmarking platform (exported 26 October 2026; DeepSeek V4.1 Flash exported 3 November 2026, Gemini 4 Argon 7 November 2026, GPT-6.1 Terra 12 November 2026). 50 questions, one answer per question and configuration, October and November 2026.
Chat configurations: GPT-6.1 Sol (deployment of 29 September 2026), thinking medium; GPT-5.1 (deployment of 13 November 2025, Swiss inference), thinking none, low, medium and high; Claude Opus 5.5, Claude Opus 5, Claude Sonnet 5.5 and Claude Sonnet 5 through LiteLLM (Anthropic), thinking medium; DeepSeek V4.1 Flash through LiteLLM, default thinking level (the platform is set to medium, but whether the provider applies it is not verified; it returned reasoning tokens on 48 of 50 answers); Gemini 4 Argon through LiteLLM (Google Vertex AI), thinking medium (reasoning tokens on 34 of 50 answers); GPT-6.1 Terra (deployment of 26 October 2026), thinking medium (reasoning tokens on 36 of 50 answers). Gemini 4 Argon and GPT-6.1 Terra ran in their own benchmark spaces. Gemini's settings were not checked against the standard ones; GPT-6.1 Terra's match the GPT-6.1 Sol space of the November runs apart from the model. All other configurations use the standard space settings; see <![CDATA[Chat configuration]]> for loop limits, chunk limits and token limits.
Judge model: not recorded in the exports of the first nine configurations. The DeepSeek V4.1 Flash, Gemini 4 Argon and GPT-6.1 Terra exports record GPT-6.1 Sol (deployment of 29 September 2026), thinking low.
October and November judges: the November runs grade conciseness higher than the October exports for answers of similar length (GPT-5.1 thinking medium: 91.2% against 81.0%). In the run of 5 November, scored like GPT-6.1 Terra, GPT-6.1 Sol reaches 95.0% overall (94.0% here) and GPT-5.1 thinking medium 91.0% (88.0% here). Against that GPT-6.1 Sol run, GPT-6.1 Terra leads by 1.1 pp (p = 0.002) rather than 2.2 pp; source quality +1.1 pp (p = 0.37), conciseness level.
Judges: factual accuracy is a G-Eval judge that sees only the latest answer and all retrieved chunks, and maps the agent's zero-based [sourceN] tags to chunk N+1. Source quality sees the conversation and the retrieved chunks. Intent, conciseness and tone see the full conversation.
Scores: factual accuracy 0 to 1 (shown as 0 to 100%), the others 0 to 100%. The overall score is the unweighted average of the judges that scored the answer; not-applicable verdicts (no sources retrieved) are left out.
Finance questions: 30 of the 50 (rates, FX, prices, company figures, financial regulators and regulation).
Intervals and tests: 95% bootstrap intervals over questions (5,000 resamples), paired sign-flip permutation tests.
Cost: estimated per answer from token counts, chat model only (judging not included). Answers without token counts in the export (up to 4 per configuration) are left out.
Limitation: 50 questions is a small set; differences of one or two answers are within noise. All scores come from LLM judges.