Remy Hallucination Benchmarks
3 min read
Measured results for two capabilities relationship managers use with Remy. These pages are benchmarks: what was tested, on which model, and what to expect. The tracks above describe how Remy works; this section holds the evidence behind two of its controls.
Hallucination check
A second compliance layer after every answer. It compares the answer with the sources the agent used and warns the relationship manager when a statement contradicts those sources.
What relationship managers get. About 2% of answers contain a hallucination. The hallucination check warns on 96% of them (GPT-6.1 Sol, the main model), so only 2% × (1 − 0.96) ≈ 0.1% of answers reach the relationship manager with a hallucination and no warning: about 1 in 1,000 answers instead of 20.
Judge models compared on 834 benchmark answers (October and November 2026):
The number in front of each model on the charts is its rank in this table, by overall accuracy. Click a column header to sort, or use Ctrl+F to find a model. Overall accuracy counts all 834 answers; hallucinations caught pools the 297 planted and real hallucinations; wrongly warned is the share of the 537 correct answers that got a warning. Answers a judge could not finish count as no warning, as in production.
Rank | Model | Reasoning effort | Provider | Overall accuracy [95% interval] | Hallucinations caught (297) | Correct answers wrongly warned (537) | Mean check time | Cost per checked answer | Inference |
|---|---|---|---|---|---|---|---|---|---|
1 | Claude Opus 5.5 | low | Anthropic | 92.3% [90.3–93.9%] | 80% (239/297) | 1.1% (6/537) | 7.6 s | USD 0.238 | Anthropic (US) |
2 | GPT-6.1 Sol (production, main) | low | OpenAI | 91.5% [89.4–93.2%] | 89% (264/297) | 7.1% (38/537) | 7.4 s | USD 0.174 | OpenAI (US) |
3 | GPT-6.1 Sol | high | OpenAI | 91.4% [89.3–93.1%] | 88% (262/297) | 6.9% (37/537) | 29.5 s | USD 0.160 | OpenAI (US) |
4 | Gemini 4 Argon | low | 91.0% [88.9–92.8%] | 77% (229/297) | 1.3% (7/537) | 12.9 s | USD 0.092 | Google Vertex AI (EU) | |
5 | GPT-6.1 Sol | medium | OpenAI | 90.8% [88.6–92.5%] | 88% (261/297) | 7.6% (41/537) | 10.0 s | USD 0.168 | OpenAI (US) |
6 | GPT-6.1 Sol | xhigh | OpenAI | 90.6% [88.5–92.4%] | 85% (252/297) | 6.1% (33/537) | 53.7 s | USD 0.185 | OpenAI (US) |
7 | GPT-6 Sol (previous main) | none | OpenAI | 89.8% [87.6–91.7%] | 77% (230/297) | 3.4% (18/537) | 5.4 s | USD 0.161 | Azure |
8 | GPT-6 Sol | medium | OpenAI | 89.7% [87.4–91.6%] | 83% (247/297) | 6.7% (36/537) | 8.4 s | USD 0.164 | Azure |
9 | GPT-6.1 Terra | low | OpenAI | 89.2% [86.9–91.1%] | 82% (244/297) | 6.9% (37/537) | 4.9 s | USD 0.087 | Azure |
10 | DeepSeek V4.1 Flash | not applicable | DeepSeek | 88.8% [86.5–90.8%] | 73% (216/297) | 2.2% (12/537) | 19.2 s | USD 0.017 | Together AI (US) |
11 | Claude Sonnet 5.5 | low | Anthropic | 88.4% [86.0–90.4%] | 69% (205/297) | 0.9% (5/537) | 4.3 s | USD 0.114 | Anthropic (US) |
12 | Kimi K3 | low | Moonshot AI | 87.5% [85.1–89.6%] | 68% (201/297) | 1.5% (8/537) | 12.9 s | USD 0.150 | Together AI (US) |
13 | GPT-5.1 (production, Swiss) | medium | OpenAI | 87.1% [84.6–89.2%] | 71% (211/297) | 4.1% (22/537) | 43.6 s | USD 0.055 | Swiss inference |
14 | GPT-5.1 | low | OpenAI | 86.6% [84.1–88.7%] | 67% (200/297) | 2.8% (15/537) | 21.4 s | USD 0.035 | Swiss inference |
15 | GPT-5.6 luna | low | OpenAI | 85.1% [82.6–87.4%] | 74% (219/297) | 8.6% (46/537) | 6.2 s | USD 0.018 | Azure |
16 | GPT-6 Luna | none | OpenAI | 83.6% [80.9–85.9%] | 61% (181/297) | 3.9% (21/537) | 4.5 s | USD 0.009 | Azure |
17 | Claude Sonnet 5 | low | Anthropic | 82.6% [79.9–85.0%] | 63% (188/297) | 6.7% (36/537) | 6.8 s | USD 0.117 | Anthropic (US) |
18 | GPT-5.6 luna | none | OpenAI | 80.0% [77.1–82.6%] | 65% (193/297) | 11.7% (63/537) | 4.5 s | USD 0.020 | Azure |
19 | GPT-5.1 | minimal | OpenAI | 78.3% [75.4–81.0%] | 49% (146/297) | 5.6% (30/537) | 15.7 s | USD 0.041 | Swiss inference |
20 | GPT-5.4-mini | none | OpenAI | 72.3% [69.2–75.2%] | 26% (77/297) | 2.0% (11/537) | 3.6 s | USD 0.027 | Azure |
21 | GPT-5-mini | minimal | OpenAI | 67.7% [64.5–70.8%] | 10% (31/297) | 0.6% (3/537) | 23.0 s | USD 0.005 | Azure |
22 | Mistral Medium 3 | not applicable | Mistral | 67.3% [64.0–70.4%] | 13% (39/297) | 2.8% (15/537) | 18.0 s | USD 0.012 | Google Vertex AI (EU) |