Judge Models

19 min read

This page lists the judge configurations we evaluated on the benchmarking suite. It complements the parent page, which reports only the two production configurations: GPT-6.1 Sol (reasoning effort low, main model) and GPT-5.1 (reasoning effort medium, Swiss inference). All GPT-6 runs used GPT-6 Sol, version from 22nd of September 2026. GPT-6.1 Sol used the version from 29th of September 2026, served by OpenAI through LiteLLM (not Swiss inference). The other providers were served through LiteLLM: Claude Opus 5.5 (version from 22nd of September 2026), Claude Sonnet 5.5 (28th of September 2026) and Claude Sonnet 5 (1st of July 2026) by Anthropic, Kimi K3 (27th of July 2026) and DeepSeek V4.1 Flash by Together AI, Mistral Medium 3 by Google Vertex AI in europe-west4, and Gemini 4 Argon by Google Vertex AI on its EU endpoint.

All configurations except TypeSafe Jev use the same judge flow (contradiction prompt, verbatim quotes, verification call) and were run on the same 834 answers. TypeSafe Jev, an external decision model, needs a different flow and was run on public cases only, so its column is marked with * and rests on less data.

The GPT-6.1 Sol efforts, GPT-5.1 low and the Claude, Kimi and Mistral models were run on with the current judge code. The older configurations judged their remaining answers on with the same code. A repeat run of GPT-6 Sol (effort none) on the same harness moved planted direct-answer recall by 3 points and false alarms by 0.3 points, which is the run-to-run noise to expect. DeepSeek V4.1 Flash was run on 28 October 2026, and Gemini 4 Argon and GPT-5.6 luna (version from 9th of July 2026, efforts none and low) on 30 October 2026, all with the same judge code. GPT-6.1 Terra (version from 26th of October 2026, Azure) was run on 3rd November 2026 and again on 12th November 2026 with the same judge code; its figures are from the 12 November run, which matched the first within run-to-run noise. Claude Haiku 5.5 (version from 7th of October 2026, Google Vertex AI on its EU endpoint) was run on 4th November 2026 with the same judge code. GPT-6 Luna (version from 22nd of September 2026, effort none) and GPT-5-mini (version from 7th of August 2025, effort minimal), both at their lowest effort, were run on 11th November 2026 with the same judge code.

Summary

  • Thinking level matters as much as the model. GPT-5.1 catches 54% at minimal and 81% at medium effort; for GPT-6 Sol, medium effort doubles the false alarms without catching significantly more. For GPT-6.1 Sol, the higher efforts catch no more than low and take up to seven times as long. Every model has to be benchmarked at every effort level.

  • GPT-6.1 Sol, effort low (production, main model; low is its lowest effort, it has no effort none): catches the most, 96% of planted direct-answer hallucinations, significantly more than GPT-6 Sol none (87%, p = 0.02), with 7.1% false alarms, significantly more than GPT-6 Sol none (3.4%, p = 0.004). Mean check time 7 s, USD 145 per benchmark run. We accept the extra false alarms for the higher catch rate: about 1 in 1,000 answers reaches the RM with an unwarned hallucination, against about 3 with GPT-6 Sol.

  • GPT-6.1 Sol, efforts medium, high and xhigh: 93%, 91% and 91% of planted direct-answer hallucinations with 7.6%, 6.9% and 6.1% false alarms, no significant difference from effort low on either. Against GPT-6 Sol none they catch no significantly more, with significantly more false alarms. Slower (mean 10 s, 30 s and 54 s) at a similar cost (USD 140, 134 and 146 per benchmark run). Not recommended over effort low.

  • GPT-6.1 Terra, effort low (what production picks for this model): 90% of planted direct-answer hallucinations with 6.9% false alarms. Against GPT-6.1 Sol low: no significant difference in catches (p = 0.06) and the same false alarms (p = 1.0). Against GPT-6 Sol none: the same catch rate (p = 0.51), significantly more false alarms (p = 0.003). Faster than GPT-6.1 Sol (mean 4.9 s against 7 s) at half the cost (USD 73 against USD 145 per benchmark run).

  • Claude Opus 5.5, effort low: catches 94% of planted direct-answer hallucinations, the same as GPT-6.1 Sol low (p = 0.69), with only 1.1% false alarms (6 of 537, against 38 for GPT-6.1 Sol low; p < 0.001). Against GPT-6 Sol none: no significant difference in catches (p = 0.11), significantly fewer false alarms (p = 0.03). Mean check time 8 s, USD 196 per benchmark run, about a third more than GPT-6.1 Sol. The best balance of catch rate and false alarms we measured, and a candidate for the main model.

  • Gemini 4 Argon, effort low: 88% of planted direct-answer hallucinations with only 1.3% false alarms (7 of 537). Against GPT-6 Sol none: the same catch rate (p = 1.0), significantly fewer false alarms (p = 0.035). Against GPT-6.1 Sol low: no significant difference in catches (p = 0.09), significantly fewer false alarms (p < 0.001). Mean check time 13 s (90th percentile 23 s), USD 76 per benchmark run, about half of GPT-6.1 Sol at Google's introductory price, which doubles on 1 December 2026.

  • Claude Sonnet 5.5, effort low: 86% of planted direct-answer hallucinations with the fewest false alarms (0.9%). Against GPT-6 Sol none: the same catch rate (p = 1.0), significantly fewer false alarms (p = 0.01). The fastest strong judge (mean 4.3 s), USD 94 per benchmark run.

  • Kimi K3, effort low: 82% of planted direct-answer hallucinations with 1.5% false alarms, no significant difference from GPT-6 Sol none on either (p = 0.39 and p = 0.052). Mean 13 s, USD 125 per benchmark run.

  • DeepSeek V4.1 Flash (no reasoning effort): 85% of planted direct-answer hallucinations with 2.2% false alarms, no significant difference from GPT-6 Sol none on either (p = 0.79 and p = 0.33), but significantly fewer catches than GPT-6.1 Sol low (p = 0.01). By far the cheapest usable judge (USD 14 per benchmark run, about a tenth of GPT-6.1 Sol), but slow (mean 19 s, 90th percentile 45 s).

  • Claude Sonnet 5, effort low: 72% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none (p = 0.004), with 6.7% false alarms, significantly more (p = 0.007). Mean 7 s, USD 96 per benchmark run. Not suitable.

  • Claude Opus 5, effort low: not scorable. 102 of 834 answers (12%) came back without a verdict, so the check never finished on them. To be rerun with structured output.

  • Claude Haiku 5.5, effort low: not scorable. 109 of 834 answers (13%) did not finish: 98 came back without a verdict and 11 exceeded its context window. The cheapest judge tested (about USD 9 per benchmark run), but not suitable.

  • GPT-6 Sol, effort none (previous main model): catches 87% of planted direct-answer hallucinations with 3.4% false alarms. Mean check time 5 s, USD 134 per benchmark run.

  • GPT-6 Sol, effort medium: similar catch rate on planted errors (89%, not significantly different) but about twice the false alarms (6.7% vs 3.4%, significant, p = 0.006). Mean 8 s, USD 137 per benchmark run. Not chosen, because a warning that is often wrong erodes the relationship manager's trust in it.

  • GPT-5.1, effort medium (production, Swiss inference): 81% of planted direct-answer hallucinations, significantly fewer than GPT-6.1 Sol (p = 0.004), with 4.1% false alarms, not significantly different from GPT-6 Sol none (p = 0.25). Slower (mean 44 s), USD 44 per benchmark run.

  • GPT-5.1, effort low: 78% of planted direct-answer hallucinations with 2.8% false alarms, not significantly different from GPT-5.1 medium (p = 0.55 and p = 0.12) or GPT-6 Sol none (p = 0.06 and p = 0.85), and about twice as fast as medium (mean 21 s). USD 28 per benchmark run.

  • GPT-5.1, effort minimal: about three times faster than medium (mean 16 s, USD 34 per benchmark run), but catches only 54% of planted direct-answer hallucinations, significantly fewer than both production configurations. Not recommended.

  • GPT-5.4-mini, effort none: the fastest and cheapest GPT option (mean 4 s, USD 22 per benchmark run) with 2.0% false alarms, but catches only 28% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none. Not suitable as the judge.

  • GPT-5.6 luna, efforts low and none (a model clients move to from GPT-4o): at low, what production picks for this model, it catches 85% of planted direct-answer hallucinations, no significant difference from GPT-6 Sol none (p = 0.75), but with 8.6% false alarms, significantly more (p < 0.001). At none it catches only 71%, significantly fewer than at low (p = 0.004) and than GPT-6 Sol none (p = 0.001), with 11.7% false alarms. None is faster (mean 4.5 s against 6 s) but not cheaper (USD 16 against USD 15 per benchmark run, about a tenth of GPT-6.1 Sol). Use low, never none.

  • GPT-6 Luna, effort none (its lowest effort): catches 73% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none (p = 0.01) and GPT-6.1 Sol low (p < 0.001), with 3.9% false alarms, no significant difference from GPT-6 Sol none (p = 0.73). Mean check time 4.5 s, USD 8 per benchmark run, about a twentieth of GPT-6.1 Sol. Cheap and fast, but misses about one planted hallucination in four.

  • GPT-5-mini, effort minimal (its lowest effort): catches only 9% of planted direct-answer hallucinations (8 of 94), significantly fewer than GPT-6 Sol none (p < 0.001), with 0.6% false alarms. At minimal effort it almost never claims a contradiction. Mean check time 23 s (90th percentile 66 s), USD 4 per benchmark run. Not suitable as the judge.

  • Mistral Medium 3 (no reasoning effort): the cheapest judge (USD 9 per benchmark run, mean 18 s) with 2.8% false alarms, but catches only 14% of planted direct-answer hallucinations, significantly fewer than GPT-6 Sol none (p < 0.001). Not suitable as the judge.

  • TypeSafe Jev 1.13 (external, best of three flows): catches about 33% of planted direct-answer hallucinations at 2.6% false alarms, against 88% for GPT-6 Sol on the same public cases. Extremely cheap (about USD 0.003 per answer) but not suitable as the judge. See the sub-page Why TypeSafe Jev is not suitable as a hallucination judge.

Results

95% Wilson intervals in brackets. Planted cases: RM benchmark answers with one planted error. False alarms: 537 correct real answers. Check time: mean per answer with a 95% bootstrap interval. Cost per benchmark run: total over all judged answers.

OpenAI models and TypeSafe Jev

GPT-6 Sol, none (previous main)

GPT-6 Sol, medium

GPT-6.1 Sol, low (production, main)

GPT-6.1 Sol, medium

GPT-6.1 Sol, high

GPT-6.1 Sol, xhigh

GPT-6.1 Terra, low

GPT-5.1, medium (production, Swiss)

GPT-5.1, low

GPT-5.1, minimal

GPT-5.4-mini, none

GPT-5.6 luna, low

GPT-5.6 luna, none

GPT-6 Luna, none

GPT-5-mini, minimal

TypeSafe Jev 1.13*

Planted, hallucination in the direct answer (94)

87% (82/94) [79–93%]

89% (84/94) [82–94%]

96% (90/94) [90–98%]

93% (87/94) [85–96%]

91% (86/94) [84–96%]

91% (86/94) [84–96%]

90% (85/94) [83–95%]

81% (76/94) [72–88%]

78% (73/94) [68–85%]

54% (51/94) [44–64%]

28% (26/94) [20–37%]

85% (80/94) [77–91%]

71% (67/94) [61–79%]

73% (69/94) [64–81%]

9% (8/94) [4–16%]

33% (47/144)

Planted, all (199)

90% (179/199) [85–93%]

90% (179/199) [85–93%]

97% (193/199) [94–99%]

95% (189/199) [91–97%]

93% (186/199) [89–96%]

93% (185/199) [89–96%]

90% (180/199) [86–94%]

84% (168/199) [79–89%]

80% (160/199) [74–85%]

56% (111/199) [49–63%]

26% (51/199) [20–32%]

85% (170/199) [80–90%]

71% (142/199) [65–77%]

71% (142/199) [65–77%]

8% (15/199) [5–12%]

29%

False alarms (537 correct answers)

3.4% (18/537) [2.1–5.2%]

6.7% (36/537) [4.9–9.1%]

7.1% (38/537) [5.2–9.6%]

7.6% (41/537) [5.7–10.2%]

6.9% (37/537) [5.0–9.4%]

6.1% (33/537) [4.4–8.5%]

6.9% (37/537) [5.0–9.4%]

4.1% (22/537) [2.7–6.1%]

2.8% (15/537) [1.7–4.6%]

5.6% (30/537) [3.9–7.9%]

2.0% (11/537) [1.1–3.6%]

8.6% (46/537) [6.5–11.2%]

11.7% (63/537) [9.3–14.7%]

3.9% (21/537) [2.6–5.9%]

0.6% (3/537) [0.2–1.6%]

2.6% (4/151)

Check time, mean [95% interval]

5 s [5–6 s]

8 s [8–9 s]

7 s [7–8 s]

10.0 s [9.6–10.4 s]

30 s [28–31 s]

54 s [51–57 s]

4.9 s [4.7–5.1 s]

44 s [41–46 s]

21 s [20–23 s]

16 s [14–17 s]

4 s [3–4 s]

6 s [6–7 s]

4.5 s [4.1–4.9 s]

4.5 s [4.3–4.7 s]

23 s [21–25 s]

not measured

Check time, median / 90th percentile

4 s / 9 s

7 s / 15 s

7 s / 12 s

9 s / 17 s

23 s / 54 s

38 s / 125 s

4 s / 8 s

34 s / 106 s

12 s / 53 s

8 s / 38 s

2 s / 7 s

6 s / 9 s

4 s / 7 s

3 s / 8 s

7 s / 66 s

not measured

Cost per answer (mean)

USD 0.16

USD 0.16

USD 0.17

USD 0.17

USD 0.16

USD 0.18

USD 0.09

USD 0.06

USD 0.04

USD 0.04

USD 0.03

USD 0.02

USD 0.02

USD 0.01

USD 0.005

about USD 0.003

Cost per benchmark run

USD 134

USD 137

USD 145

USD 140

USD 134

USD 146

USD 73

USD 44

USD 28

USD 34

USD 22

USD 15

USD 16

USD 8

USD 4

not measured

* TypeSafe Jev 1.13 (statement-level flow, the best of the three flows we tested) has less data than the other columns: for data-protection reasons it was run on public cases only (web search, public documentation, Microsoft Learn, Yahoo Finance), that is 72 planted direct-answer errors, 134 planted errors in total and 151 correct answers. Its flag threshold was set on one half of the correct answers (about 3% false alarms) and tested on the other half, in both directions, and the two results are pooled. On the same public cases, GPT-6 Sol none catches 88% of planted direct-answer errors with 2.7% false alarms.

Other providers

GPT-6.1 Sol, low (production, main; for comparison)

Claude Opus 5.5, low

Claude Sonnet 5.5, low

Claude Sonnet 5, low

Kimi K3, low

DeepSeek V4.1 Flash

Gemini 4 Argon, low

Mistral Medium 3

Planted, hallucination in the direct answer (94)

96% (90/94) [90–98%]

94% (88/94) [87–97%]

86% (81/94) [78–92%]

72% (68/94) [63–80%]

82% (77/94) [73–88%]

85% (80/94) [77–91%]

88% (83/94) [80–93%]

14% (13/94) [8–22%]

Planted, all (199)

97% (193/199) [94–99%]

95% (190/199) [92–98%]

86% (171/199) [80–90%]

76% (151/199) [69–81%]

83% (166/199) [78–88%]

88% (175/199) [83–92%]

90% (179/199) [85–93%]

11% (22/199) [7–16%]

False alarms (537 correct answers)

7.1% (38/537) [5.2–9.6%]

1.1% (6/537) [0.5–2.4%]

0.9% (5/537) [0.4–2.2%]

6.7% (36/537) [4.9–9.1%]

1.5% (8/537) [0.8–2.9%]

2.2% (12/537) [1.3–3.9%]

1.3% (7/537) [0.6–2.7%]

2.8% (15/537) [1.7–4.6%]

Check time, mean [95% interval]

7 s [7–8 s]

8 s [7–8 s]

4.3 s [4.1–4.4 s]

7 s [6–7 s]

13 s [12–14 s]

19 s [18–21 s]

13 s [12–14 s]

18 s [17–20 s]

Check time, median / 90th percentile

7 s / 12 s

7 s / 13 s

4 s / 7 s

5 s / 13 s

10 s / 21 s

12 s / 45 s

7 s / 23 s

11 s / 43 s

Cost per answer (mean)

USD 0.17

USD 0.24

USD 0.11

USD 0.12

USD 0.15

USD 0.02

USD 0.09

USD 0.01

Cost per benchmark run

USD 145

USD 196

USD 94

USD 96

USD 125

USD 14

USD 76

USD 9

Claude Opus 5 is not in the table: 102 of its 834 answers came back without a verdict, too many to score.

Per 1,000 RM questions

Expected: 20 answers per 1,000 contain a hallucination in the direct answer (200 real agent chats, QA check of 9 September 2026). Applying each configuration's catch rate on planted direct-answer errors:

Per 1,000 answers

GPT-6 Sol, none

GPT-6 Sol, medium

GPT-6.1 Sol, low (production, main)

GPT-6.1 Sol, medium

GPT-6.1 Sol, high

GPT-6.1 Sol, xhigh

GPT-6.1 Terra, low

GPT-5.1, medium

GPT-5.1, low

GPT-5.1, minimal

GPT-5.4-mini, none

GPT-5.6 luna, low

GPT-5.6 luna, none

GPT-6 Luna, none

GPT-5-mini, minimal

TypeSafe Jev 1.13*

RM warned

about 17

about 18

about 19

about 19

about 18

about 18

about 18

about 16

about 16

about 11

about 6

about 17

about 14

about 15

about 2

about 7

Reach the RM without a warning

about 3

about 2

about 1

about 1

about 2

about 2

about 2

about 4

about 4

about 9

about 14

about 3

about 6

about 5

about 18

about 13

Correct answers with a warning

about 33

about 66

about 69

about 75

about 68

about 60

about 68

about 40

about 27

about 55

about 20

about 84

about 115

about 38

about 5

about 25

Per 1,000 answers

GPT-6.1 Sol, low (for comparison)

Claude Opus 5.5, low

Claude Sonnet 5.5, low

Claude Sonnet 5, low

Kimi K3, low

DeepSeek V4.1 Flash

Gemini 4 Argon, low

Mistral Medium 3

RM warned

about 19

about 19

about 17

about 14

about 16

about 17

about 18

about 3

Reach the RM without a warning

about 1

about 1

about 3

about 6

about 4

about 3

about 2

about 17

Correct answers with a warning

about 69

about 11

about 9

about 66

about 15

about 22

about 13

about 27

These figures are based on planted hallucinations. Real hallucinations are too rare, and human reviewers miss them too often, to measure the catch rate on them reliably. The planted cases are built to resemble real ones as closely as we could. * TypeSafe Jev: public cases only, see above.

Statistical tests

Paired exact McNemar tests against GPT-6 Sol (effort none), on the same cases:

Compared with GPT-6 Sol, none

Planted direct-answer hallucinations caught

False alarms

GPT-6 Sol, medium

no significant difference (catches 5 that none misses, misses 3; p = 0.73)

significantly more (29 vs 11 discordant; p = 0.006)

GPT-6.1 Sol, low

significantly more (catches 9 that none misses, misses 1; p = 0.02)

significantly more (32 vs 12 discordant; p = 0.004)

GPT-6.1 Sol, medium

no significant difference (7 vs 2; p = 0.18)

significantly more (35 vs 12; p = 0.001)

GPT-6.1 Sol, high

no significant difference (6 vs 2; p = 0.29)

significantly more (30 vs 11; p = 0.004)

GPT-6.1 Sol, xhigh

no significant difference (8 vs 2; p = 0.11)

significantly more (27 vs 10; p = 0.008)

GPT-6.1 Terra, low

no significant difference (6 vs 3; p = 0.51)

significantly more (29 vs 10; p = 0.003)

GPT-5.1, medium

no significant difference (7 vs 13; p = 0.26)

no significant difference (17 vs 10; p = 0.25)

GPT-5.1, low

no significant difference (5 vs 14; p = 0.06)

no significant difference (12 vs 14; p = 0.85)

GPT-5.1, minimal

significantly fewer (3 vs 34; p < 0.001)

significantly more (24 vs 11; p = 0.04)

GPT-5.4-mini, none

significantly fewer (0 vs 56; p < 0.001)

no significant difference (11 vs 17; p = 0.35)

GPT-5.6 luna, low

no significant difference (4 vs 6; p = 0.75)

significantly more (40 vs 12; p < 0.001)

GPT-5.6 luna, none

significantly fewer (3 vs 18; p = 0.001)

significantly more (58 vs 13; p < 0.001)

GPT-6 Luna, none

significantly fewer (5 vs 18; p = 0.01)

no significant difference (18 vs 15; p = 0.73)

GPT-5-mini, minimal

significantly fewer (0 vs 74; p < 0.001)

significantly fewer (3 vs 17; p = 0.003)

Claude Opus 5.5, low

no significant difference (8 vs 2; p = 0.11)

significantly fewer (5 vs 16; p = 0.03)

Claude Sonnet 5.5, low

no significant difference (7 vs 8; p = 1.0)

significantly fewer (4 vs 16; p = 0.01)

Claude Sonnet 5, low

significantly fewer (4 vs 18; p = 0.004)

significantly more (32 vs 13; p = 0.007)

Kimi K3, low

no significant difference (4 vs 8; p = 0.39)

no significant difference (6 vs 16; p = 0.052)

DeepSeek V4.1 Flash

no significant difference (6 vs 8; p = 0.79)

no significant difference (10 vs 16; p = 0.33)

Gemini 4 Argon, low

no significant difference (5 vs 4; p = 1.0)

significantly fewer (6 vs 17; p = 0.035)

Mistral Medium 3

significantly fewer (0 vs 68; p < 0.001)

no significant difference (14 vs 16; p = 0.86)

GPT-5.1 minimal vs GPT-5.1 medium: medium catches 29 planted direct-answer hallucinations that minimal misses, minimal catches 4 that medium misses (p < 0.001). False alarms 23 vs 17 (p = 0.43). GPT-5.4-mini vs GPT-5.1 medium: medium catches 50 that mini misses, mini none that medium misses (p < 0.001). GPT-5.6 luna none vs GPT-5.6 luna low: low catches 16 planted direct-answer hallucinations that none misses, none catches 3 that low misses (p = 0.004). False alarms 50 vs 33 (p = 0.08). GPT-5.1 low vs GPT-5.1 medium: no significant difference on planted direct-answer hallucinations (low catches 4 that medium misses, misses 7; p = 0.55) or false alarms (9 vs 18; p = 0.12).

GPT-6.1 Sol low vs GPT-6 Sol medium: no significant difference on planted direct-answer hallucinations (7 vs 1; p = 0.07) or false alarms (12 vs 10; p = 0.83).

Against GPT-6.1 Sol low: efforts medium (1 vs 4; p = 0.38), high (0 vs 4; p = 0.13) and xhigh (2 vs 2; p = 1.0) catch no significantly different number of planted direct-answer hallucinations, with no significant difference in false alarms (12 vs 9, 11 vs 12 and 9 vs 10; p ≥ 0.66). Claude Opus 5.5 catches the same (2 vs 4; p = 0.69) with significantly fewer false alarms (1 vs 32; p < 0.001). Claude Sonnet 5.5 catches significantly fewer (2 vs 11; p = 0.02) with significantly fewer false alarms (3 vs 35; p < 0.001). Kimi K3 catches significantly fewer (1 vs 13; p = 0.002) with significantly fewer false alarms (7 vs 37; p < 0.001). DeepSeek V4.1 Flash catches significantly fewer (2 vs 12; p = 0.01) with significantly fewer false alarms (5 vs 31; p < 0.001). Gemini 4 Argon catches no significantly different number (3 vs 10; p = 0.09) with significantly fewer false alarms (3 vs 34; p < 0.001). GPT-5.6 luna low catches significantly fewer (3 vs 13; p = 0.02) with no significant difference in false alarms (31 vs 23; p = 0.34); at none it catches significantly fewer (1 vs 24; p < 0.001) with significantly more false alarms (52 vs 27; p = 0.007). GPT-6.1 Terra catches no significantly different number (0 vs 5; p = 0.06) with the same false alarms (14 vs 15; p = 1.0). GPT-6 Luna none catches significantly fewer (1 vs 22; p < 0.001) with significantly fewer false alarms (17 vs 34; p = 0.02). GPT-5-mini minimal catches significantly fewer (0 vs 81; p < 0.001) with significantly fewer false alarms (3 vs 36; p < 0.001).

Configurations

Configuration

Model deployment

Reasoning effort

GPT-6 Sol, none

GPT-6 Sol (2026-09-22)

none

GPT-6 Sol, medium

GPT-6 Sol (2026-09-22)

medium

GPT-6.1 Sol, low

GPT-6.1 Sol (2026-09-29), OpenAI through LiteLLM

low (lowest supported; what production picks for this model)

GPT-6.1 Sol, medium / high / xhigh

GPT-6.1 Sol (2026-09-29), OpenAI through LiteLLM

medium / high / xhigh

GPT-6.1 Terra, low

GPT-6.1 Terra (2026-10-26)

low (what production picks for this model)

GPT-5.1, medium

GPT-5.1 (2025-11-13), Swiss inference

medium

GPT-5.1, low

GPT-5.1 (2025-11-13), Swiss inference

low

GPT-5.1, minimal

GPT-5.1 (2025-11-13), Swiss inference

minimal

GPT-5.4-mini, none

GPT-5.4-mini (2026-03-17)

none

GPT-5.6 luna, low

GPT-5.6 luna (2026-07-09)

low (what production picks for this model)

GPT-5.6 luna, none

GPT-5.6 luna (2026-07-09)

none

GPT-6 Luna, none

GPT-6 Luna (2026-09-22)

none (lowest supported)

GPT-5-mini, minimal

GPT-5-mini (2025-08-07)

minimal (lowest supported; it has no effort none)

Claude Opus 5.5, low

Claude Opus 5.5 (2026-09-22), Anthropic through LiteLLM

low

Claude Sonnet 5.5, low

Claude Sonnet 5.5 (2026-09-28), Anthropic through LiteLLM

low

Claude Sonnet 5, low

Claude Sonnet 5 (2026-07-01), Anthropic through LiteLLM

low

Claude Opus 5, low (not scorable)

Claude Opus 5 (2026-07-24), Anthropic through LiteLLM

low

Kimi K3, low

Kimi K3 (2026-07-27), Together AI through LiteLLM

low

DeepSeek V4.1 Flash

DeepSeek V4.1 Flash, Together AI through LiteLLM

not applicable (Together AI rejects every reasoning effort)

Gemini 4 Argon, low

Gemini 4 Argon, Google Vertex AI EU endpoint through LiteLLM

low (what production picks for this model)

Mistral Medium 3

Mistral Medium 3, Google Vertex AI europe-west4 through LiteLLM

not applicable (no reasoning effort)

TypeSafe Jev

jev-1.13.0 (external API, outside Switzerland)

not applicable (decision model)

Times and costs are measured per judged answer across all 834 answers (Jev: public cases), including the verification call when a judge claims a contradiction. Cost per answer is higher than in earlier versions of this page because the 834 answers include longer real chats with large sources.

Against the previous Unique AI judge (one GPT-4o call, USD 0.06 per answer on the 165 answers where its token counts are still stored): the new flow on those same answers costs about the same with GPT-5.1 at medium thinking (USD 0.06) and about 1.1 times as much with GPT-6 Sol and no thinking (USD 0.07). The second call runs only when the judge claims a contradiction. GPT-6.1 Sol at USD 0.17 on this suite is about three times that, on these long sources. A European deployment can use any GPT model from 5.1 to 6.1, so GPT-5.1 keeps the cost at today's level. On the same 165 answers GPT-6 Luna (none) costs about USD 0.004 and GPT-5-mini (minimal) about USD 0.006, a tenth or less of the old judge, but they catch 73% and 9% of planted direct-answer hallucinations.

Answers a judge could not finish count as no warning, as in production: GPT-6.1 Sol xhigh 44 (upstream timeouts), GPT-5.1 medium 34, minimal 19 and low 16 (upstream timeouts and context window), GPT-5.4-mini 16 (input limit), GPT-5-mini 24 (input limit and upstream timeouts on the longest chats; 2 without a verdict), Mistral Medium 3 33 (context window), each Claude model 11 (context window), Kimi K3 4 (no verdict in the answer). GPT-6 Sol, GPT-6.1 Sol low, medium and high, DeepSeek V4.1 Flash, Gemini 4 Argon, GPT-5.6 luna (none and low), GPT-6.1 Terra and GPT-6 Luna judged all 834.

Last updated