Remy Hallucination Benchmarks

3 min read

Measured results for two capabilities relationship managers use with Remy. These pages are benchmarks: what was tested, on which model, and what to expect. The tracks above describe how Remy works; this section holds the evidence behind two of its controls.

Hallucination check

A second compliance layer after every answer. It compares the answer with the sources the agent used and warns the relationship manager when a statement contradicts those sources.

What relationship managers get. About 2% of answers contain a hallucination. The hallucination check warns on 96% of them (GPT-6.1 Sol, the main model), so only 2% × (1 − 0.96) ≈ 0.1% of answers reach the relationship manager with a hallucination and no warning: about 1 in 1,000 answers instead of 20.

Judge models compared on 834 benchmark answers (October and November 2026):

hallucination_check_accuracy_vs_cost.pnghallucination_check_accuracy_vs_latency.pnghallucination_check_caught_vs_false_alarms.png

The number in front of each model on the charts is its rank in this table, by overall accuracy. Click a column header to sort, or use Ctrl+F to find a model. Overall accuracy counts all 834 answers; hallucinations caught pools the 297 planted and real hallucinations; wrongly warned is the share of the 537 correct answers that got a warning. Answers a judge could not finish count as no warning, as in production.

Rank

Model

Reasoning effort

Provider

Overall accuracy [95% interval]

Hallucinations caught (297)

Correct answers wrongly warned (537)

Mean check time

Cost per checked answer

Inference

1

Claude Opus 5.5

low

Anthropic

92.3% [90.3–93.9%]

80% (239/297)

1.1% (6/537)

7.6 s

USD 0.238

Anthropic (US)

2

GPT-6.1 Sol (production, main)

low

OpenAI

91.5% [89.4–93.2%]

89% (264/297)

7.1% (38/537)

7.4 s

USD 0.174

OpenAI (US)

3

GPT-6.1 Sol

high

OpenAI

91.4% [89.3–93.1%]

88% (262/297)

6.9% (37/537)

29.5 s

USD 0.160

OpenAI (US)

4

Gemini 4 Argon

low

Google

91.0% [88.9–92.8%]

77% (229/297)

1.3% (7/537)

12.9 s

USD 0.092

Google Vertex AI (EU)

5

GPT-6.1 Sol

medium

OpenAI

90.8% [88.6–92.5%]

88% (261/297)

7.6% (41/537)

10.0 s

USD 0.168

OpenAI (US)

6

GPT-6.1 Sol

xhigh

OpenAI

90.6% [88.5–92.4%]

85% (252/297)

6.1% (33/537)

53.7 s

USD 0.185

OpenAI (US)

7

GPT-6 Sol (previous main)

none

OpenAI

89.8% [87.6–91.7%]

77% (230/297)

3.4% (18/537)

5.4 s

USD 0.161

Azure

8

GPT-6 Sol

medium

OpenAI

89.7% [87.4–91.6%]

83% (247/297)

6.7% (36/537)

8.4 s

USD 0.164

Azure

9

GPT-6.1 Terra

low

OpenAI

89.2% [86.9–91.1%]

82% (244/297)

6.9% (37/537)

4.9 s

USD 0.087

Azure

10

DeepSeek V4.1 Flash

not applicable

DeepSeek

88.8% [86.5–90.8%]

73% (216/297)

2.2% (12/537)

19.2 s

USD 0.017

Together AI (US)

11

Claude Sonnet 5.5

low

Anthropic

88.4% [86.0–90.4%]

69% (205/297)

0.9% (5/537)

4.3 s

USD 0.114

Anthropic (US)

12

Kimi K3

low

Moonshot AI

87.5% [85.1–89.6%]

68% (201/297)

1.5% (8/537)

12.9 s

USD 0.150

Together AI (US)

13

GPT-5.1 (production, Swiss)

medium

OpenAI

87.1% [84.6–89.2%]

71% (211/297)

4.1% (22/537)

43.6 s

USD 0.055

Swiss inference

14

GPT-5.1

low

OpenAI

86.6% [84.1–88.7%]

67% (200/297)

2.8% (15/537)

21.4 s

USD 0.035

Swiss inference

15

GPT-5.6 luna

low

OpenAI

85.1% [82.6–87.4%]

74% (219/297)

8.6% (46/537)

6.2 s

USD 0.018

Azure

16

GPT-6 Luna

none

OpenAI

83.6% [80.9–85.9%]

61% (181/297)

3.9% (21/537)

4.5 s

USD 0.009

Azure

17

Claude Sonnet 5

low

Anthropic

82.6% [79.9–85.0%]

63% (188/297)

6.7% (36/537)

6.8 s

USD 0.117

Anthropic (US)

18

GPT-5.6 luna

none

OpenAI

80.0% [77.1–82.6%]

65% (193/297)

11.7% (63/537)

4.5 s

USD 0.020

Azure

19

GPT-5.1

minimal

OpenAI

78.3% [75.4–81.0%]

49% (146/297)

5.6% (30/537)

15.7 s

USD 0.041

Swiss inference

20

GPT-5.4-mini

none

OpenAI

72.3% [69.2–75.2%]

26% (77/297)

2.0% (11/537)

3.6 s

USD 0.027

Azure

21

GPT-5-mini

minimal

OpenAI

67.7% [64.5–70.8%]

10% (31/297)

0.6% (3/537)

23.0 s

USD 0.005

Azure

22

Mistral Medium 3

not applicable

Mistral

67.3% [64.0–70.4%]

13% (39/297)

2.8% (15/537)

18.0 s

USD 0.012

Google Vertex AI (EU)

Last updated