This page lists the judge configurations we evaluated on the benchmarking suite. It complements the parent page, which reports only the two production configurations: GPT-6 Sol (reasoning effort none) and GPT-5.1 (reasoning effort medium, Swiss inference). All GPT-6 runs used GPT-6 Sol, version from 22nd of September 2026. GPT-6.1 Sol used the version from 29th of September 2026, served by OpenAI through LiteLLM (not Swiss inference).

All GPT configurations use the same judge flow (contradiction prompt, verbatim quotes, verification call) and were run on the same 566 answers. TypeSafe Jev, an external decision model, needs a different flow and was run on public cases only, so its column is marked with * and rests on less data.

GPT-6.1 Sol was run on October 3, 2026 with the current judge code. A repeat run of GPT-6 Sol (effort none) on the same harness moved planted direct-answer recall by 3 points and false alarms by 0.3 points, which is the run-to-run noise to expect.

Summary

• GPT-6 Sol, effort none (production): best balance. It catches 87% of planted direct-answer hallucinations with 2.9% false alarms and is among the fastest.

• GPT-6 Sol, effort medium: similar catch rate on planted errors (89%, not significantly different) but more than twice the false alarms (6.5% vs 2.9%, significant). Not chosen, because a warning that is often wrong erodes the relationship manager's trust in it.

• GPT-6.1 Sol, effort low (its lowest effort; it has no effort none): catches the most, 95% of planted direct-answer hallucinations, but has 7.7% false alarms, significantly more than GPT-6 Sol none (2.9%) and similar to GPT-6 Sol medium. Median 7 s, USD 0.07 per answer. It trades more false alarms for more caught errors; not adopted as the default.

• GPT-5.1, effort medium (production, Swiss inference): 81% of planted direct-answer hallucinations, same false-alarm rate as GPT-6 Sol. Slower (median 29 s).

• GPT-5.1, effort minimal: about five times faster than medium, but catches only 54% of planted direct-answer hallucinations, significantly fewer than both production configurations. Not recommended.

• GPT-5.4-mini, effort none: the fastest and cheapest GPT option (median 2 s, USD 0.02) with the fewest false alarms (1.6%), but catches only 28% of planted direct-answer hallucinations. Not suitable as the judge.

• TypeSafe Jev 1.13 (external, best of three flows): catches about 33% of planted direct-answer hallucinations at 2.6% false alarms, against 88% for GPT-6 Sol on the same public cases. Extremely cheap (about USD 0.003 per answer) but not suitable as the judge. See the sub-page Why TypeSafe Jev is not suitable as a hallucination judge.

Results

95% Wilson intervals in brackets. Planted cases: RM benchmark answers with one planted error. False alarms: 310 correct real answers.

* TypeSafe Jev 1.13 (statement-level flow, the best of the three flows we tested) has less data than the other columns: for data-protection reasons it was run on public cases only (web search, public documentation, Microsoft Learn, Yahoo Finance), that is 72 planted direct-answer errors, 134 planted errors in total and 151 correct answers. Its flag threshold was set on one half of the correct answers (about 3% false alarms) and tested on the other half, in both directions, and the two results are pooled. On the same public cases, GPT-6 Sol none catches 88% of planted direct-answer errors with 2.7% false alarms.

Per 1,000 RM questions

Expected: 20 answers per 1,000 contain a hallucination in the direct answer (200 real agent chats, QA check of 9 September 2026). Applying each configuration's catch rate on planted direct-answer errors:

These figures are based on planted hallucinations. Real hallucinations are too rare, and human reviewers miss them too often, to measure the catch rate on them reliably. The planted cases are built to resemble real ones as closely as we could. * TypeSafe Jev: public cases only, see above.

Statistical tests

Paired exact McNemar tests against GPT-6 Sol (effort none), on the same cases:

GPT-5.1 minimal vs GPT-5.1 medium: medium catches 29 planted direct-answer hallucinations that minimal misses, minimal catches 4 that medium misses (p < 0.001). False alarms 13 vs 9 (p = 0.45). GPT-5.4-mini vs GPT-5.1 medium: medium catches 50 that mini misses, mini none that medium misses (p < 0.001).

GPT-6.1 Sol low vs GPT-6 Sol medium: no significant difference on planted direct-answer hallucinations (6 vs 1; p = 0.13) or false alarms (8 vs 4; p = 0.39).

Configurations