This page lists every judge configuration we evaluated on the benchmarking suite. It complements the parent page, which reports only the two production configurations: GPT-6 Sol (reasoning effort none) and GPT-5.1 (reasoning effort medium, Swiss inference). All GPT-6 runs used GPT-6 Sol, version from 22nd of September 2026.

All GPT configurations use the same judge flow (contradiction prompt, verbatim quotes, verification call) and were run on the same 566 answers. TypeSafe Jev, an external decision model, is reported separately at the end because it needs a different flow and was run on public cases only.

Summary

• GPT-6 Sol, effort none (production): best balance. It catches 87% of planted direct-answer hallucinations with 2.9% false alarms and is among the fastest.

• GPT-6 Sol, effort medium: similar catch rate on planted errors (89%, not significantly different) but more than twice the false alarms (6.5% vs 2.9%, significant). Not chosen, because a warning that is often wrong erodes the relationship manager's trust in it.

• GPT-5.1, effort medium (production, Swiss inference): 81% of planted direct-answer hallucinations, same false-alarm rate as GPT-6 Sol. Slower (median 29 s).

• GPT-5.1, effort minimal: about five times faster than medium, but catches only 54% of planted direct-answer hallucinations, significantly fewer than both production configurations. Not recommended.

• GPT-5.4-mini, effort none: the fastest and cheapest GPT option (median 2 s, USD 0.02) with the fewest false alarms (1.6%), but catches only 28% of planted direct-answer hallucinations. Not suitable as the judge.

• GPT-4o on the same flow: catches only 13% of planted direct-answer hallucinations, with 3.9% false alarms. Six of the 566 answers exceed its context window. Not suitable as the judge.

• TypeSafe Jev 1.13 (external, best of three flows): catches about 33% of planted direct-answer hallucinations at 2.6% false alarms, against 88% for GPT-6 Sol on the same public cases. Extremely cheap (about USD 0.003 per answer) but not suitable as the judge. See the sub-page Why TypeSafe Jev is not suitable as a hallucination judge.

Results

95% Wilson intervals in brackets. Planted cases: RM benchmark answers with one planted error. False alarms: 310 correct real answers.

For GPT-4o, the six answers without a verdict (context window exceeded) count as "no warning", as they would in production.

Per 1,000 RM questions

Expected: 20 answers per 1,000 contain a hallucination in the direct answer (200 real agent chats, QA check of 9 September 2026). Applying each configuration's catch rate on planted direct-answer errors:

Statistical tests

Paired exact McNemar tests against GPT-6 Sol (effort none), on the same cases:

GPT-5.1 minimal vs GPT-5.1 medium: medium catches 29 planted direct-answer hallucinations that minimal misses, minimal catches 4 that medium misses (p < 0.001). False alarms 13 vs 9 (p = 0.45). GPT-5.4-mini vs GPT-5.1 medium: medium catches 50 that mini misses, mini none that medium misses (p < 0.001).

External model: TypeSafe Jev 1.13

Jev is a decision model: it returns probabilities for typed yes/no, choice and score questions and cannot write text, so it cannot quote the contradicting sentences our flow relies on. We tested three flows; the best one (statement-level) is reported here. For data-protection reasons only public cases were sent (web search, public documentation, Microsoft Learn, Yahoo Finance): 72 planted direct-answer errors, 134 planted errors in total, and 151 correct real answers. Jev's flag threshold was set on one half of the correct answers (about 3% false alarms) and tested on the other half, in both directions; the two results are pooled below.

The other two Jev flows were weaker: whole answer in one request (sources trimmed to fit 32k tokens) 12–15% at about 3% false alarms, AUC 0.78–0.80; one request per source (no trimming) 6–18%, AUC 0.71–0.74.

Configurations