Judge Models
19 min read
This page lists the judge configurations we evaluated on the benchmarking suite. It complements the parent page, which reports only the two production configurations: GPT-6.1 Sol (reasoning effort low, main model) and GPT-5.1 (reasoning effort medium, Swiss inference). All GPT-6 runs used GPT-6 Sol, version from 22nd of September 2026. GPT-6.1 Sol used the version from 29th of September 2026, served by OpenAI through LiteLLM (not Swiss inference). The other providers were served through LiteLLM: Claude Opus 5.5 (version from 22nd of September 2026), Claude Sonnet 5.5 (28th of September 2026) and Claude Sonnet 5 (1st of July 2026) by Anthropic, Kimi K3 (27th of July 2026) and DeepSeek V4.1 Flash by Together AI, Mistral Medium 3 by Google Vertex AI in europe-west4, and Gemini 4 Argon by Google Vertex AI on its EU endpoint.
All configurations except TypeSafe Jev use the same judge flow (contradiction prompt, verbatim quotes, verification call) and were run on the same 834 answers. TypeSafe Jev, an external decision model, needs a different flow and was run on public cases only, so its column is marked with * and rests on less data.
The GPT-6.1 Sol efforts, GPT-5.1 low and the Claude, Kimi and Mistral models were run on with the current judge code. The older configurations judged their remaining answers on with the same code. A repeat run of GPT-6 Sol (effort none) on the same harness moved planted direct-answer recall by 3 points and false alarms by 0.3 points, which is the run-to-run noise to expect. DeepSeek V4.1 Flash was run on 28 October 2026, and Gemini 4 Argon and GPT-5.6 luna (version from 9th of July 2026, efforts none and low) on 30 October 2026, all with the same judge code. GPT-6.1 Terra (version from 26th of October 2026, Azure) was run on 3rd November 2026 and again on 12th November 2026 with the same judge code; its figures are from the 12 November run, which matched the first within run-to-run noise. Claude Haiku 5.5 (version from 7th of October 2026, Google Vertex AI on its EU endpoint) was run on 4th November 2026 with the same judge code. GPT-6 Luna (version from 22nd of September 2026, effort none) and GPT-5-mini (version from 7th of August 2025, effort minimal), both at their lowest effort, were run on 11th November 2026 with the same judge code.
Summary
Thinking level matters as much as the model. GPT-5.1 catches 54% at minimal and 81% at medium effort; for GPT-6 Sol, medium effort doubles the false alarms without catching significantly more. For GPT-6.1 Sol, the higher efforts catch no more than low and take up to seven times as long. Every model has to be benchmarked at every effort level.
GPT-6.1 Sol, effort low (production, main model; low is its lowest effort, it has no effort none): catches the most, 96% of planted direct-answer hallucinations, significantly more than GPT-6 Sol none (87%, p = 0.02), with 7.1% false alarms, significantly more than GPT-6 Sol none (3.4%, p = 0.004). Mean check time 7 s, USD 145 per benchmark run. We accept the extra false alarms for the higher catch rate: about 1 in 1,000 answers reaches the RM with an unwarned hallucination, against about 3 with GPT-6 Sol.
GPT-6.1 Sol, efforts medium, high and xhigh: 93%, 91% and 91% of planted direct-answer hallucinations with 7.6%, 6.9% and 6.1% false alarms, no significant difference from effort low on either. Against GPT-6 Sol none they catch no significantly more, with significantly more false alarms. Slower (mean 10 s, 30 s and 54 s) at a similar cost (USD 140, 134 and 146 per benchmark run). Not recommended over effort low.
GPT-6.1 Terra, effort low (what production picks for this model): 90% of planted direct-answer hallucinations with 6.9% false alarms. Against GPT-6.1 Sol low: no significant difference in catches (p = 0.06) and the same false alarms (p = 1.0). Against GPT-6 Sol none: the same catch rate (p = 0.51), significantly more false alarms (p = 0.003). Faster than GPT-6.1 Sol (mean 4.9 s against 7 s) at half the cost (USD 73 against USD 145 per benchmark run).
Claude Opus 5.5, effort low: catches 94% of planted direct-answer hallucinations, the same as GPT-6.1 Sol low (p = 0.69), with only 1.1% false alarms (6 of 537, against 38 for GPT-6.1 Sol low; p < 0.001). Against GPT-6 Sol none: no significant difference in catches (p = 0.11), significantly fewer false alarms (p = 0.03). Mean check time 8 s, USD 196 per benchmark run, about a third more than GPT-6.1 Sol. The best balance of catch rate and false alarms we measured, and a candidate for the main model.
Gemini 4 Argon, effort low: 88% of planted direct-answer hallucinations with only 1.3% false alarms (7 of 537). Against GPT-6 Sol none: the same catch rate (p = 1.0), significantly fewer false alarms (p = 0.035). Against GPT-6.1 Sol low: no significant difference in catches (p = 0.09), significantly fewer false alarms (p < 0.001). Mean check time 13 s (90th percentile 23 s), USD 76 per benchmark run, about half of GPT-6.1 Sol at Google's introductory price, which doubles on 1 December 2026.
Claude Sonnet 5.5, effort low: 86% of planted direct-answer hallucinations with the fewest false alarms (0.9%). Against GPT-6 Sol none: the same catch rate (p = 1.0), significantly fewer false alarms (p = 0.01). The fastest strong judge (mean 4.3 s), USD 94 per benchmark run.
Kimi K3, effort low: 82% of planted direct-answer hallucinations with 1.5% false alarms, no significant difference from GPT-6 Sol none on either (p = 0.39 and p = 0.052). Mean 13 s, USD 125 per benchmark run.
DeepSeek V4.1 Flash (no reasoning effort): 85% of planted direct-answer hallucinations with 2.2% false alarms, no significant difference from GPT-6 Sol none on either (p = 0.79 and p = 0.33), but significantly fewer catches than GPT-6.1 Sol low (p = 0.01). By far the cheapest usable judge (USD 14 per benchmark run, about a tenth of GPT-6.1 Sol), but slow (mean 19 s, 90th percentile 45 s).
Claude Sonnet 5, effort low: 72% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none (p = 0.004), with 6.7% false alarms, significantly more (p = 0.007). Mean 7 s, USD 96 per benchmark run. Not suitable.
Claude Opus 5, effort low: not scorable. 102 of 834 answers (12%) came back without a verdict, so the check never finished on them. To be rerun with structured output.
Claude Haiku 5.5, effort low: not scorable. 109 of 834 answers (13%) did not finish: 98 came back without a verdict and 11 exceeded its context window. The cheapest judge tested (about USD 9 per benchmark run), but not suitable.
GPT-6 Sol, effort none (previous main model): catches 87% of planted direct-answer hallucinations with 3.4% false alarms. Mean check time 5 s, USD 134 per benchmark run.
GPT-6 Sol, effort medium: similar catch rate on planted errors (89%, not significantly different) but about twice the false alarms (6.7% vs 3.4%, significant, p = 0.006). Mean 8 s, USD 137 per benchmark run. Not chosen, because a warning that is often wrong erodes the relationship manager's trust in it.
GPT-5.1, effort medium (production, Swiss inference): 81% of planted direct-answer hallucinations, significantly fewer than GPT-6.1 Sol (p = 0.004), with 4.1% false alarms, not significantly different from GPT-6 Sol none (p = 0.25). Slower (mean 44 s), USD 44 per benchmark run.
GPT-5.1, effort low: 78% of planted direct-answer hallucinations with 2.8% false alarms, not significantly different from GPT-5.1 medium (p = 0.55 and p = 0.12) or GPT-6 Sol none (p = 0.06 and p = 0.85), and about twice as fast as medium (mean 21 s). USD 28 per benchmark run.
GPT-5.1, effort minimal: about three times faster than medium (mean 16 s, USD 34 per benchmark run), but catches only 54% of planted direct-answer hallucinations, significantly fewer than both production configurations. Not recommended.
GPT-5.4-mini, effort none: the fastest and cheapest GPT option (mean 4 s, USD 22 per benchmark run) with 2.0% false alarms, but catches only 28% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none. Not suitable as the judge.
GPT-5.6 luna, efforts low and none (a model clients move to from GPT-4o): at low, what production picks for this model, it catches 85% of planted direct-answer hallucinations, no significant difference from GPT-6 Sol none (p = 0.75), but with 8.6% false alarms, significantly more (p < 0.001). At none it catches only 71%, significantly fewer than at low (p = 0.004) and than GPT-6 Sol none (p = 0.001), with 11.7% false alarms. None is faster (mean 4.5 s against 6 s) but not cheaper (USD 16 against USD 15 per benchmark run, about a tenth of GPT-6.1 Sol). Use low, never none.
GPT-6 Luna, effort none (its lowest effort): catches 73% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none (p = 0.01) and GPT-6.1 Sol low (p < 0.001), with 3.9% false alarms, no significant difference from GPT-6 Sol none (p = 0.73). Mean check time 4.5 s, USD 8 per benchmark run, about a twentieth of GPT-6.1 Sol. Cheap and fast, but misses about one planted hallucination in four.
GPT-5-mini, effort minimal (its lowest effort): catches only 9% of planted direct-answer hallucinations (8 of 94), significantly fewer than GPT-6 Sol none (p < 0.001), with 0.6% false alarms. At minimal effort it almost never claims a contradiction. Mean check time 23 s (90th percentile 66 s), USD 4 per benchmark run. Not suitable as the judge.
Mistral Medium 3 (no reasoning effort): the cheapest judge (USD 9 per benchmark run, mean 18 s) with 2.8% false alarms, but catches only 14% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none (p < 0.001). Not suitable as the judge.
TypeSafe Jev 1.13 (external, best of three flows): catches about 33% of planted direct-answer hallucinations at 2.6% false alarms, against 88% for GPT-6 Sol on the same public cases. Extremely cheap (about USD 0.003 per answer) but not suitable as the judge. See the sub-page Why TypeSafe Jev is not suitable as a hallucination judge.
Results
95% Wilson intervals in brackets. Planted cases: RM benchmark answers with one planted error. False alarms: 537 correct real answers. Check time: mean per answer with a 95% bootstrap interval. Cost per benchmark run: total over all judged answers.
OpenAI models and TypeSafe Jev
GPT-6 Sol, none (previous main) | GPT-6 Sol, medium | GPT-6.1 Sol, low (production, main) | GPT-6.1 Sol, medium | GPT-6.1 Sol, high | GPT-6.1 Sol, xhigh | GPT-6.1 Terra, low | GPT-5.1, medium (production, Swiss) | GPT-5.1, low | GPT-5.1, minimal | GPT-5.4-mini, none | GPT-5.6 luna, low | GPT-5.6 luna, none | GPT-6 Luna, none | GPT-5-mini, minimal | TypeSafe Jev 1.13* | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Planted, hallucination in the direct answer (94) | 87% (82/94) [79–93%] | 89% (84/94) [82–94%] | 96% (90/94) [90–98%] | 93% (87/94) [85–96%] | 91% (86/94) [84–96%] | 91% (86/94) [84–96%] | 90% (85/94) [83–95%] | 81% (76/94) [72–88%] | 78% (73/94) [68–85%] | 54% (51/94) [44–64%] | 28% (26/94) [20–37%] | 85% (80/94) [77–91%] | 71% (67/94) [61–79%] | 73% (69/94) [64–81%] | 9% (8/94) [4–16%] | 33% (47/144) |
Planted, all (199) | 90% (179/199) [85–93%] | 90% (179/199) [85–93%] | 97% (193/199) [94–99%] | 95% (189/199) [91–97%] | 93% (186/199) [89–96%] | 93% (185/199) [89–96%] | 90% (180/199) [86–94%] | 84% (168/199) [79–89%] | 80% (160/199) [74–85%] | 56% (111/199) [49–63%] | 26% (51/199) [20–32%] | 85% (170/199) [80–90%] | 71% (142/199) [65–77%] | 71% (142/199) [65–77%] | 8% (15/199) [5–12%] | 29% |
False alarms (537 correct answers) | 3.4% (18/537) [2.1–5.2%] | 6.7% (36/537) [4.9–9.1%] | 7.1% (38/537) [5.2–9.6%] | 7.6% (41/537) [5.7–10.2%] | 6.9% (37/537) [5.0–9.4%] | 6.1% (33/537) [4.4–8.5%] | 6.9% (37/537) [5.0–9.4%] | 4.1% (22/537) [2.7–6.1%] | 2.8% (15/537) [1.7–4.6%] | 5.6% (30/537) [3.9–7.9%] | 2.0% (11/537) [1.1–3.6%] | 8.6% (46/537) [6.5–11.2%] | 11.7% (63/537) [9.3–14.7%] | 3.9% (21/537) [2.6–5.9%] | 0.6% (3/537) [0.2–1.6%] | 2.6% (4/151) |
Check time, mean [95% interval] | 5 s [5–6 s] | 8 s [8–9 s] | 7 s [7–8 s] | 10.0 s [9.6–10.4 s] | 30 s [28–31 s] | 54 s [51–57 s] | 4.9 s [4.7–5.1 s] | 44 s [41–46 s] | 21 s [20–23 s] | 16 s [14–17 s] | 4 s [3–4 s] | 6 s [6–7 s] | 4.5 s [4.1–4.9 s] | 4.5 s [4.3–4.7 s] | 23 s [21–25 s] | not measured |
Check time, median / 90th percentile | 4 s / 9 s | 7 s / 15 s | 7 s / 12 s | 9 s / 17 s | 23 s / 54 s | 38 s / 125 s | 4 s / 8 s | 34 s / 106 s | 12 s / 53 s | 8 s / 38 s | 2 s / 7 s | 6 s / 9 s | 4 s / 7 s | 3 s / 8 s | 7 s / 66 s | not measured |
Cost per answer (mean) | USD 0.16 | USD 0.16 | USD 0.17 | USD 0.17 | USD 0.16 | USD 0.18 | USD 0.09 | USD 0.06 | USD 0.04 | USD 0.04 | USD 0.03 | USD 0.02 | USD 0.02 | USD 0.01 | USD 0.005 | about USD 0.003 |
Cost per benchmark run | USD 134 | USD 137 | USD 145 | USD 140 | USD 134 | USD 146 | USD 73 | USD 44 | USD 28 | USD 34 | USD 22 | USD 15 | USD 16 | USD 8 | USD 4 | not measured |
* TypeSafe Jev 1.13 (statement-level flow, the best of the three flows we tested) has less data than the other columns: for data-protection reasons it was run on public cases only (web search, public documentation, Microsoft Learn, Yahoo Finance), that is 72 planted direct-answer errors, 134 planted errors in total and 151 correct answers. Its flag threshold was set on one half of the correct answers (about 3% false alarms) and tested on the other half, in both directions, and the two results are pooled. On the same public cases, GPT-6 Sol none catches 88% of planted direct-answer errors with 2.7% false alarms.
Other providers
GPT-6.1 Sol, low (production, main; for comparison) | Claude Opus 5.5, low | Claude Sonnet 5.5, low | Claude Sonnet 5, low | Kimi K3, low | DeepSeek V4.1 Flash | Gemini 4 Argon, low | Mistral Medium 3 | |
|---|---|---|---|---|---|---|---|---|
Planted, hallucination in the direct answer (94) | 96% (90/94) [90–98%] | 94% (88/94) [87–97%] | 86% (81/94) [78–92%] | 72% (68/94) [63–80%] | 82% (77/94) [73–88%] | 85% (80/94) [77–91%] | 88% (83/94) [80–93%] | 14% (13/94) [8–22%] |
Planted, all (199) | 97% (193/199) [94–99%] | 95% (190/199) [92–98%] | 86% (171/199) [80–90%] | 76% (151/199) [69–81%] | 83% (166/199) [78–88%] | 88% (175/199) [83–92%] | 90% (179/199) [85–93%] | 11% (22/199) [7–16%] |
False alarms (537 correct answers) | 7.1% (38/537) [5.2–9.6%] | 1.1% (6/537) [0.5–2.4%] | 0.9% (5/537) [0.4–2.2%] | 6.7% (36/537) [4.9–9.1%] | 1.5% (8/537) [0.8–2.9%] | 2.2% (12/537) [1.3–3.9%] | 1.3% (7/537) [0.6–2.7%] | 2.8% (15/537) [1.7–4.6%] |
Check time, mean [95% interval] | 7 s [7–8 s] | 8 s [7–8 s] | 4.3 s [4.1–4.4 s] | 7 s [6–7 s] | 13 s [12–14 s] | 19 s [18–21 s] | 13 s [12–14 s] | 18 s [17–20 s] |
Check time, median / 90th percentile | 7 s / 12 s | 7 s / 13 s | 4 s / 7 s | 5 s / 13 s | 10 s / 21 s | 12 s / 45 s | 7 s / 23 s | 11 s / 43 s |
Cost per answer (mean) | USD 0.17 | USD 0.24 | USD 0.11 | USD 0.12 | USD 0.15 | USD 0.02 | USD 0.09 | USD 0.01 |
Cost per benchmark run | USD 145 | USD 196 | USD 94 | USD 96 | USD 125 | USD 14 | USD 76 | USD 9 |
Claude Opus 5 is not in the table: 102 of its 834 answers came back without a verdict, too many to score.
Per 1,000 RM questions
Expected: 20 answers per 1,000 contain a hallucination in the direct answer (200 real agent chats, QA check of 9 September 2026). Applying each configuration's catch rate on planted direct-answer errors:
Per 1,000 answers | GPT-6 Sol, none | GPT-6 Sol, medium | GPT-6.1 Sol, low (production, main) | GPT-6.1 Sol, medium | GPT-6.1 Sol, high | GPT-6.1 Sol, xhigh | GPT-6.1 Terra, low | GPT-5.1, medium | GPT-5.1, low | GPT-5.1, minimal | GPT-5.4-mini, none | GPT-5.6 luna, low | GPT-5.6 luna, none | GPT-6 Luna, none | GPT-5-mini, minimal | TypeSafe Jev 1.13* |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
RM warned | about 17 | about 18 | about 19 | about 19 | about 18 | about 18 | about 18 | about 16 | about 16 | about 11 | about 6 | about 17 | about 14 | about 15 | about 2 | about 7 |
Reach the RM without a warning | about 3 | about 2 | about 1 | about 1 | about 2 | about 2 | about 2 | about 4 | about 4 | about 9 | about 14 | about 3 | about 6 | about 5 | about 18 | about 13 |
Correct answers with a warning | about 33 | about 66 | about 69 | about 75 | about 68 | about 60 | about 68 | about 40 | about 27 | about 55 | about 20 | about 84 | about 115 | about 38 | about 5 | about 25 |
Per 1,000 answers | GPT-6.1 Sol, low (for comparison) | Claude Opus 5.5, low | Claude Sonnet 5.5, low | Claude Sonnet 5, low | Kimi K3, low | DeepSeek V4.1 Flash | Gemini 4 Argon, low | Mistral Medium 3 |
|---|---|---|---|---|---|---|---|---|
RM warned | about 19 | about 19 | about 17 | about 14 | about 16 | about 17 | about 18 | about 3 |
Reach the RM without a warning | about 1 | about 1 | about 3 | about 6 | about 4 | about 3 | about 2 | about 17 |
Correct answers with a warning | about 69 | about 11 | about 9 | about 66 | about 15 | about 22 | about 13 | about 27 |
These figures are based on planted hallucinations. Real hallucinations are too rare, and human reviewers miss them too often, to measure the catch rate on them reliably. The planted cases are built to resemble real ones as closely as we could. * TypeSafe Jev: public cases only, see above.
Statistical tests
Paired exact McNemar tests against GPT-6 Sol (effort none), on the same cases:
Compared with GPT-6 Sol, none | Planted direct-answer hallucinations caught | False alarms |
|---|---|---|
GPT-6 Sol, medium | no significant difference (catches 5 that none misses, misses 3; p = 0.73) | significantly more (29 vs 11 discordant; p = 0.006) |
GPT-6.1 Sol, low | significantly more (catches 9 that none misses, misses 1; p = 0.02) | significantly more (32 vs 12 discordant; p = 0.004) |
GPT-6.1 Sol, medium | no significant difference (7 vs 2; p = 0.18) | significantly more (35 vs 12; p = 0.001) |
GPT-6.1 Sol, high | no significant difference (6 vs 2; p = 0.29) | significantly more (30 vs 11; p = 0.004) |
GPT-6.1 Sol, xhigh | no significant difference (8 vs 2; p = 0.11) | significantly more (27 vs 10; p = 0.008) |
GPT-6.1 Terra, low | no significant difference (6 vs 3; p = 0.51) | significantly more (29 vs 10; p = 0.003) |
GPT-5.1, medium | no significant difference (7 vs 13; p = 0.26) | no significant difference (17 vs 10; p = 0.25) |
GPT-5.1, low | no significant difference (5 vs 14; p = 0.06) | no significant difference (12 vs 14; p = 0.85) |
GPT-5.1, minimal | significantly fewer (3 vs 34; p < 0.001) | significantly more (24 vs 11; p = 0.04) |
GPT-5.4-mini, none | significantly fewer (0 vs 56; p < 0.001) | no significant difference (11 vs 17; p = 0.35) |
GPT-5.6 luna, low | no significant difference (4 vs 6; p = 0.75) | significantly more (40 vs 12; p < 0.001) |
GPT-5.6 luna, none | significantly fewer (3 vs 18; p = 0.001) | significantly more (58 vs 13; p < 0.001) |
GPT-6 Luna, none | significantly fewer (5 vs 18; p = 0.01) | no significant difference (18 vs 15; p = 0.73) |
GPT-5-mini, minimal | significantly fewer (0 vs 74; p < 0.001) | significantly fewer (3 vs 17; p = 0.003) |
Claude Opus 5.5, low | no significant difference (8 vs 2; p = 0.11) | significantly fewer (5 vs 16; p = 0.03) |
Claude Sonnet 5.5, low | no significant difference (7 vs 8; p = 1.0) | significantly fewer (4 vs 16; p = 0.01) |
Claude Sonnet 5, low | significantly fewer (4 vs 18; p = 0.004) | significantly more (32 vs 13; p = 0.007) |
Kimi K3, low | no significant difference (4 vs 8; p = 0.39) | no significant difference (6 vs 16; p = 0.052) |
DeepSeek V4.1 Flash | no significant difference (6 vs 8; p = 0.79) | no significant difference (10 vs 16; p = 0.33) |
Gemini 4 Argon, low | no significant difference (5 vs 4; p = 1.0) | significantly fewer (6 vs 17; p = 0.035) |
Mistral Medium 3 | significantly fewer (0 vs 68; p < 0.001) | no significant difference (14 vs 16; p = 0.86) |
GPT-5.1 minimal vs GPT-5.1 medium: medium catches 29 planted direct-answer hallucinations that minimal misses, minimal catches 4 that medium misses (p < 0.001). False alarms 23 vs 17 (p = 0.43). GPT-5.4-mini vs GPT-5.1 medium: medium catches 50 that mini misses, mini none that medium misses (p < 0.001). GPT-5.6 luna none vs GPT-5.6 luna low: low catches 16 planted direct-answer hallucinations that none misses, none catches 3 that low misses (p = 0.004). False alarms 50 vs 33 (p = 0.08). GPT-5.1 low vs GPT-5.1 medium: no significant difference on planted direct-answer hallucinations (low catches 4 that medium misses, misses 7; p = 0.55) or false alarms (9 vs 18; p = 0.12).
GPT-6.1 Sol low vs GPT-6 Sol medium: no significant difference on planted direct-answer hallucinations (7 vs 1; p = 0.07) or false alarms (12 vs 10; p = 0.83).
Against GPT-6.1 Sol low: efforts medium (1 vs 4; p = 0.38), high (0 vs 4; p = 0.13) and xhigh (2 vs 2; p = 1.0) catch no significantly different number of planted direct-answer hallucinations, with no significant difference in false alarms (12 vs 9, 11 vs 12 and 9 vs 10; p ≥ 0.66). Claude Opus 5.5 catches the same (2 vs 4; p = 0.69) with significantly fewer false alarms (1 vs 32; p < 0.001). Claude Sonnet 5.5 catches significantly fewer (2 vs 11; p = 0.02) with significantly fewer false alarms (3 vs 35; p < 0.001). Kimi K3 catches significantly fewer (1 vs 13; p = 0.002) with significantly fewer false alarms (7 vs 37; p < 0.001). DeepSeek V4.1 Flash catches significantly fewer (2 vs 12; p = 0.01) with significantly fewer false alarms (5 vs 31; p < 0.001). Gemini 4 Argon catches no significantly different number (3 vs 10; p = 0.09) with significantly fewer false alarms (3 vs 34; p < 0.001). GPT-5.6 luna low catches significantly fewer (3 vs 13; p = 0.02) with no significant difference in false alarms (31 vs 23; p = 0.34); at none it catches significantly fewer (1 vs 24; p < 0.001) with significantly more false alarms (52 vs 27; p = 0.007). GPT-6.1 Terra catches no significantly different number (0 vs 5; p = 0.06) with the same false alarms (14 vs 15; p = 1.0). GPT-6 Luna none catches significantly fewer (1 vs 22; p < 0.001) with significantly fewer false alarms (17 vs 34; p = 0.02). GPT-5-mini minimal catches significantly fewer (0 vs 81; p < 0.001) with significantly fewer false alarms (3 vs 36; p < 0.001).
Configurations
Configuration | Model deployment | Reasoning effort |
|---|---|---|
GPT-6 Sol, none | GPT-6 Sol (2026-09-22) | none |
GPT-6 Sol, medium | GPT-6 Sol (2026-09-22) | medium |
GPT-6.1 Sol, low | GPT-6.1 Sol (2026-09-29), OpenAI through LiteLLM | low (lowest supported; what production picks for this model) |
GPT-6.1 Sol, medium / high / xhigh | GPT-6.1 Sol (2026-09-29), OpenAI through LiteLLM | medium / high / xhigh |
GPT-6.1 Terra, low | GPT-6.1 Terra (2026-10-26) | low (what production picks for this model) |
GPT-5.1, medium | GPT-5.1 (2025-11-13), Swiss inference | medium |
GPT-5.1, low | GPT-5.1 (2025-11-13), Swiss inference | low |
GPT-5.1, minimal | GPT-5.1 (2025-11-13), Swiss inference | minimal |
GPT-5.4-mini, none | GPT-5.4-mini (2026-03-17) | none |
GPT-5.6 luna, low | GPT-5.6 luna (2026-07-09) | low (what production picks for this model) |
GPT-5.6 luna, none | GPT-5.6 luna (2026-07-09) | none |
GPT-6 Luna, none | GPT-6 Luna (2026-09-22) | none (lowest supported) |
GPT-5-mini, minimal | GPT-5-mini (2025-08-07) | minimal (lowest supported; it has no effort none) |
Claude Opus 5.5, low | Claude Opus 5.5 (2026-09-22), Anthropic through LiteLLM | low |
Claude Sonnet 5.5, low | Claude Sonnet 5.5 (2026-09-28), Anthropic through LiteLLM | low |
Claude Sonnet 5, low | Claude Sonnet 5 (2026-07-01), Anthropic through LiteLLM | low |
Claude Opus 5, low (not scorable) | Claude Opus 5 (2026-07-24), Anthropic through LiteLLM | low |
Kimi K3, low | Kimi K3 (2026-07-27), Together AI through LiteLLM | low |
DeepSeek V4.1 Flash | DeepSeek V4.1 Flash, Together AI through LiteLLM | not applicable (Together AI rejects every reasoning effort) |
Gemini 4 Argon, low | Gemini 4 Argon, Google Vertex AI EU endpoint through LiteLLM | low (what production picks for this model) |
Mistral Medium 3 | Mistral Medium 3, Google Vertex AI europe-west4 through LiteLLM | not applicable (no reasoning effort) |
TypeSafe Jev | jev-1.13.0 (external API, outside Switzerland) | not applicable (decision model) |
Times and costs are measured per judged answer across all 834 answers (Jev: public cases), including the verification call when a judge claims a contradiction. Cost per answer is higher than in earlier versions of this page because the 834 answers include longer real chats with large sources.
Against the previous Unique AI judge (one GPT-4o call, USD 0.06 per answer on the 165 answers where its token counts are still stored): the new flow on those same answers costs about the same with GPT-5.1 at medium thinking (USD 0.06) and about 1.1 times as much with GPT-6 Sol and no thinking (USD 0.07). The second call runs only when the judge claims a contradiction. GPT-6.1 Sol at USD 0.17 on this suite is about three times that, on these long sources. A European deployment can use any GPT model from 5.1 to 6.1, so GPT-5.1 keeps the cost at today's level. On the same 165 answers GPT-6 Luna (none) costs about USD 0.004 and GPT-5-mini (minimal) about USD 0.006, a tenth or less of the old judge, but they catch 73% and 9% of planted direct-answer hallucinations.
Answers a judge could not finish count as no warning, as in production: GPT-6.1 Sol xhigh 44 (upstream timeouts), GPT-5.1 medium 34, minimal 19 and low 16 (upstream timeouts and context window), GPT-5.4-mini 16 (input limit), GPT-5-mini 24 (input limit and upstream timeouts on the longest chats; 2 without a verdict), Mistral Medium 3 33 (context window), each Claude model 11 (context window), Kimi K3 4 (no verdict in the answer). GPT-6 Sol, GPT-6.1 Sol low, medium and high, DeepSeek V4.1 Flash, Gemini 4 Argon, GPT-5.6 luna (none and low), GPT-6.1 Terra and GPT-6 Luna judged all 834.