Agentic Table-Questionnaire Question Extraction Benchmark
19 min read
Questionnaire question extraction turns an uploaded questionnaire into the list of questions the Agentic Table answers. This page reports how well it does that with each extraction model we evaluated. The documents and grading are described under How the benchmark works below.
All 20 models are compared on the 18 standard documents, 1,495 questions; a document a model did not complete counts against it. Each model ran once, so run-to-run noise is not measured.
Key findings
On the 18 standard documents, the best models capture 91% of questions (GPT-6.1 Terra, GPT-5.6 Terra and GPT-6 Sol), and the differences between them are not significant. GPT-6.1 Terra, GPT-5.6 Terra and GPT-6 Sol are within 0.1 points of each other. GPT-6 Luna, GPT-5.6 Sol, GPT-6.1 Sol, GPT-6 Astra, Claude Opus 5.5, Claude Opus 5, GPT-5 Mini, GPT-5.2, GPT-5.4, GPT-5.1, GPT-5.6 Luna, Kimi K3 and Claude Sonnet 5.5 are not significantly different from GPT-6.1 Terra either; Claude Sonnet 5, Claude Haiku 5.5, GPT-4o and Gemini 4 Argon are significantly lower. With 18 documents the tests separate only large differences: even Claude Sonnet 5.5, at 58%, is not significantly different from GPT-6.1 Terra.
Most models complete every standard document. 14 of the 20 models complete all 18 standard documents. Claude Haiku 5.5, Claude Sonnet 5 and Claude Sonnet 5.5 do not complete 4, 4 and 2 of them, because their replies are cut off at the platform's 10,000-token output limit; Gemini 4 Argon does not complete 9, because it returns an empty question list on them; GPT-5.1 does not complete 1, because its reply did not match the extraction format; Kimi K3 does not complete 1, because its requests on that document did not return.
Claude models cannot be used for extraction in the product as it is. Anthropic's structured-output mode rejects the production extraction schema, so the Claude models were scored with the schema given in the prompt instead.
Results
On the 18 standard documents (1,495 questions). 95% intervals in brackets, from a bootstrap over documents, because questions in the same document succeed or fail together. Time, cost per document, precision and fully preserved cover the documents each model completed. Costs are list prices for the extraction calls; "at least" means some calls returned no usage data, so the figure is a lower bound. The full run is all 23 documents, including the ones outside the standard set. Models are ranked by questions captured, highest first.
Model | Reasoning effort | Provider | Questions captured (1,495) | Fully preserved: question, guidance and context | Precision: extracted rows that match a reference question | Standard documents completed (of 18) | Time per document, mean / median / 90th percentile | Cost per document (mean) | Cost of the full 23-document run |
|---|---|---|---|---|---|---|---|---|---|
GPT-6.1 Terra | low | OpenAI | 91% (1,363/1,495) [83–97%] | 70% (1,046/1,495) [52–84%] | 96% (60 extra of 1,545) | 18 | 95 s / 42 s / 124 s | USD 0.030 | at least USD 1.05 |
GPT-5.6 Terra | low | OpenAI | 91% (1,362/1,495) [83–96%] | 69% (1,035/1,495) [51–83%] | 97% (40 extra of 1,523) | 18 | 92 s / 20 s / 43 s | at least USD 0.067 | at least USD 2.43 |
GPT-6 Sol | low | OpenAI | 91% (1,361/1,495) [83–96%] | 69% (1,039/1,495) [52–84%] | 98% (30 extra of 1,512) | 18 | 24 s / 19 s / 39 s | USD 0.061 | USD 2.28 |
GPT-6 Luna | low | OpenAI | 89% (1,326/1,495) [80–95%] | 67% (1,005/1,495) [49–82%] | 98% (29 extra of 1,484) | 18 | 53 s / 38 s / 70 s | at least USD 0.003 | at least USD 0.12 |
GPT-5.6 Sol | low | OpenAI | 88% (1,311/1,495) [78–95%] | 69% (1,027/1,495) [50–83%] | 95% (84 extra of 1,550) | 18 | 87 s / 26 s / 216 s | USD 0.172 | USD 6.38 |
GPT-6.1 Sol | low | OpenAI | 86% (1,290/1,495) [76–95%] | 70% (1,046/1,495) [52–85%] | 95% (73 extra of 1,529) | 18 | 42 s / 43 s / 56 s | USD 0.059 | at least USD 2.09 |
GPT-6 Astra | low | OpenAI | 86% (1,286/1,495) [75–95%] | 70% (1,051/1,495) [52–85%] | 95% (74 extra of 1,527) | 18 | 52 s / 41 s / 116 s | USD 0.303 | at least USD 10.74 |
Claude Opus 5.5 | low | Anthropic | 86% (1,285/1,495) [73–96%] | 68% (1,014/1,495) [49–84%] | 97% (43 extra of 1,462) | 18 | 89 s / 35 s / 132 s | USD 0.197 | USD 7.24 |
Claude Opus 5 | low | Anthropic | 85% (1,278/1,495) [71–96%] | 68% (1,013/1,495) [48–84%] | 98% (37 extra of 1,521) | 18 | 88 s / 46 s / 125 s | USD 0.248 | USD 8.89 |
GPT-5 Mini | low | OpenAI | 84% (1,261/1,495) [70–94%] | 66% (994/1,495) [49–81%] | 94% (87 extra of 1,483) | 18 | 72 s / 37 s / 143 s | at least USD 0.014 | at least USD 0.54 |
GPT-5.2 | low | OpenAI | 84% (1,261/1,495) [72–95%] | 69% (1,025/1,495) [50–84%] | 96% (61 extra of 1,547) | 18 | 43 s / 33 s / 48 s | at least USD 0.086 | at least USD 2.75 |
GPT-5.4 | low | OpenAI | 84% (1,254/1,495) [73–93%] | 69% (1,030/1,495) [51–83%] | 94% (94 extra of 1,569) | 18 | 55 s / 31 s / 75 s | at least USD 0.087 | at least USD 3.20 |
GPT-5.1 | low | OpenAI | 81% (1,204/1,495) [62–94%] | 65% (973/1,495) [44–82%] | 96% (63 extra of 1,424) | 17 | 16 s / 15 s / 24 s | USD 0.055 | USD 2.43 |
GPT-5.6 Luna | low | OpenAI | 80% (1,198/1,495) [60–94%] | 63% (935/1,495) [40–81%] | 99% (18 extra of 1,355) | 18 | 20 s / 14 s / 29 s | USD 0.007 | USD 0.25 |
Kimi K3 | low | Moonshot AI | 68% (1,011/1,495) [42–91%] | 51% (767/1,495) [30–75%] | 98% (18 extra of 1,183) | 17 | 54 s / 43 s / 124 s | USD 0.096 | at least USD 4.39 |
Claude Sonnet 5.5 | low | Anthropic | 58% (874/1,495) [37–87%] | 43% (642/1,495) [26–66%] | 94% (59 extra of 1,021) | 16 | 23 s / 22 s / 42 s | USD 0.083 | USD 4.34 |
Claude Sonnet 5 | low | Anthropic | 45% (670/1,495) [24–75%] | 33% (487/1,495) [16–58%] | 97% (21 extra of 810) | 14 | 38 s / 36 s / 49 s | USD 0.080 | USD 5.23 |
Claude Haiku 5.5 | low | Anthropic | 39% (580/1,495) [21–65%] | 23% (350/1,495) [12–43%] | 96% (30 extra of 773) | 14 | 63 s / 19 s / 30 s | USD 0.004 | USD 0.36 |
GPT-4o (production default) | not applicable | OpenAI | 31% (459/1,495) [18–50%] | 22% (322/1,495) [13–36%] | 96% (21 extra of 508) | 18 | 8 s / 7 s / 11 s | USD 0.038 | USD 1.79 |
Gemini 4 Argon | low | 25% (372/1,495) [7–48%] | 18% (275/1,495) [4–39%] | 99% (6 extra of 456) | 9 | 142 s / 113 s / 244 s | at least USD 0.052 | at least USD 3.81 |
Per 1,000 questions
Scaled from the 1,495 questions of the 18 standard documents, rounded so that captured and not captured add up to 1,000:
Model | Reasoning effort | Provider | Captured | of which fully preserved | Not captured | Extra rows with no reference question |
|---|---|---|---|---|---|---|
GPT-6.1 Terra | low | OpenAI | about 912 | about 700 | about 88 | about 40 |
GPT-5.6 Terra | low | OpenAI | about 911 | about 692 | about 89 | about 27 |
GPT-6 Sol | low | OpenAI | about 910 | about 695 | about 90 | about 20 |
GPT-6 Luna | low | OpenAI | about 887 | about 672 | about 113 | about 19 |
GPT-5.6 Sol | low | OpenAI | about 877 | about 687 | about 123 | about 56 |
GPT-6.1 Sol | low | OpenAI | about 863 | about 700 | about 137 | about 49 |
GPT-6 Astra | low | OpenAI | about 860 | about 703 | about 140 | about 49 |
Claude Opus 5.5 | low | Anthropic | about 860 | about 678 | about 140 | about 29 |
Claude Opus 5 | low | Anthropic | about 855 | about 678 | about 145 | about 25 |
GPT-5 Mini | low | OpenAI | about 843 | about 665 | about 157 | about 58 |
GPT-5.2 | low | OpenAI | about 843 | about 686 | about 157 | about 41 |
GPT-5.4 | low | OpenAI | about 839 | about 689 | about 161 | about 63 |
GPT-5.1 | low | OpenAI | about 805 | about 651 | about 195 | about 42 |
GPT-5.6 Luna | low | OpenAI | about 801 | about 625 | about 199 | about 12 |
Kimi K3 | low | Moonshot AI | about 676 | about 513 | about 324 | about 12 |
Claude Sonnet 5.5 | low | Anthropic | about 585 | about 429 | about 415 | about 39 |
Claude Sonnet 5 | low | Anthropic | about 448 | about 326 | about 552 | about 14 |
Claude Haiku 5.5 | low | Anthropic | about 388 | about 234 | about 612 | about 20 |
GPT-4o (production default) | not applicable | OpenAI | about 307 | about 215 | about 693 | about 14 |
Gemini 4 Argon | low | about 249 | about 184 | about 751 | about 4 |
Model notes
GPT-6.1 Terra, low: captures 91% (1,363/1,495) of questions [83–97%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 42 s per document, USD 0.030 per document.
GPT-5.6 Terra, low: captures 91% (1,362/1,495) of questions [83–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 20 s per document, at least USD 0.067 per document.
GPT-6 Sol, low: captures 91% (1,361/1,495) of questions [83–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 19 s per document, USD 0.061 per document.
GPT-6 Luna, low: captures 89% (1,326/1,495) of questions [80–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 38 s per document, at least USD 0.003 per document.
GPT-5.6 Sol, low: captures 88% (1,311/1,495) of questions [78–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 26 s per document, USD 0.172 per document.
GPT-6.1 Sol, low: captures 86% (1,290/1,495) of questions [76–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 43 s per document, USD 0.059 per document.
GPT-6 Astra, low: captures 86% (1,286/1,495) of questions [75–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 41 s per document, USD 0.303 per document. GPT-6 Astra is the most expensive per document of the reasoning models that complete every standard document; for cost, of those with complete cost evidence.
Claude Opus 5.5, low: captures 86% (1,285/1,495) of questions [73–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 35 s per document, USD 0.197 per document. Scored with the schema in the prompt.
Claude Opus 5, low: captures 85% (1,278/1,495) of questions [71–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 46 s per document, USD 0.248 per document. Claude Opus 5 is the slowest by median time of the reasoning models that complete every standard document. Scored with the schema in the prompt.
GPT-5 Mini, low: captures 84% (1,261/1,495) of questions [70–94%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 37 s per document, at least USD 0.014 per document.
GPT-5.2, low: captures 84% (1,261/1,495) of questions [72–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 33 s per document, at least USD 0.086 per document.
GPT-5.4, low: captures 84% (1,254/1,495) of questions [73–93%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 31 s per document, at least USD 0.087 per document.
GPT-5.1, low: captures 81% (1,204/1,495) of questions [62–94%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.002). It completes 17 of 18 standard documents. Median 15 s per document, USD 0.055 per document. GPT-5.1 does not complete one of the 18 standard documents; its reply on it did not match the extraction format.
GPT-5.6 Luna, low: captures 80% (1,198/1,495) of questions [60–94%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.002). It completes every standard document. Median 14 s per document, USD 0.007 per document. GPT-5.6 Luna is the cheapest per document and the fastest by median time of the reasoning models that complete every standard document; for cost, of those with complete cost evidence.
Kimi K3, low: captures 68% (1,011/1,495) of questions [42–91%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.002). It completes 17 of 18 standard documents. Median 43 s per document, USD 0.096 per document. Kimi K3 does not complete one of the 18 standard documents; its requests on it did not return.
Claude Sonnet 5.5, low: captures 58% (874/1,495) of questions [37–87%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.006). It completes 16 of 18 standard documents. Median 22 s per document, USD 0.083 per document. Claude Sonnet 5.5 does not complete two of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Scored with the schema in the prompt.
Claude Sonnet 5, low: captures 45% (670/1,495) of questions [24–75%], no significant difference from GPT-4o (Holm-adjusted p = 0.19). It completes 14 of 18 standard documents. Median 36 s per document, USD 0.080 per document. Claude Sonnet 5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Scored with the schema in the prompt.
Claude Haiku 5.5, low: captures 39% (580/1,495) of questions [21–65%], no significant difference from GPT-4o (Holm-adjusted p = 0.48). It completes 14 of 18 standard documents. Median 19 s per document, USD 0.004 per document. Claude Haiku 5.5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Scored with the schema in the prompt.
GPT-4o, no reasoning (production default): captures 31% (459/1,495) of questions [18–50%]. It completes every standard document. Median 7 s per document, USD 0.038 per document. The production default. Misses about two in three questions. Not suitable.
Gemini 4 Argon, low: captures 25% (372/1,495) of questions [7–48%], no significant difference from GPT-4o (Holm-adjusted p = 0.516). It completes 9 of 18 standard documents. Median 113 s per document, at least USD 0.052 per document. Gemini 4 Argon does not complete nine of the 18 standard documents; it returned a well-formed but empty question list on them. On five of them, reading the document section by section afterwards found questions, which does not count as completing it on the standard route.
Statistical tests
Paired tests on the same documents: the recall difference in points, with a 95% bootstrap interval over the 18 documents and a two-sided p-value, Holm-adjusted across the comparisons in each table. The intervals are not adjusted, so an interval that excludes zero can still be not significant after the adjustment. With 10,000 resamples the smallest p-value one test can show is 0.0001; after adjusting 19 comparisons that floor is 0.002, shown as p < 0.002. Per-question McNemar tests are not used because questions in one document are not independent.
Against GPT-4o, the production default:
Model | Reasoning effort | Recall difference |
|---|---|---|
GPT-6.1 Terra | low | +60.5 points [+35.8 to +76.6]; significantly higher (Holm-adjusted p < 0.002) |
GPT-5.6 Terra | low | +60.4 points [+35.7 to +76.6]; significantly higher (Holm-adjusted p < 0.002) |
GPT-6 Sol | low | +60.3 points [+35.9 to +76.5]; significantly higher (Holm-adjusted p < 0.002) |
GPT-6 Luna | low | +58.0 points [+32.9 to +74.7]; significantly higher (Holm-adjusted p < 0.002) |
GPT-5.6 Sol | low | +57.0 points [+32.5 to +74.2]; significantly higher (Holm-adjusted p < 0.002) |
GPT-6.1 Sol | low | +55.6 points [+32.0 to +73.3]; significantly higher (Holm-adjusted p < 0.002) |
GPT-6 Astra | low | +55.3 points [+31.5 to +73.4]; significantly higher (Holm-adjusted p < 0.002) |
Claude Opus 5.5 | low | +55.3 points [+29.5 to +74.2]; significantly higher (Holm-adjusted p < 0.002) |
Claude Opus 5 | low | +54.8 points [+28.4 to +74.1]; significantly higher (Holm-adjusted p < 0.002) |
GPT-5 Mini | low | +53.6 points [+29.6 to +71.5]; significantly higher (Holm-adjusted p < 0.002) |
GPT-5.2 | low | +53.6 points [+29.2 to +72.8]; significantly higher (Holm-adjusted p < 0.002) |
GPT-5.4 | low | +53.2 points [+29.4 to +71.2]; significantly higher (Holm-adjusted p < 0.002) |
GPT-5.1 | low | +49.8 points [+21.0 to +71.2]; significantly higher (Holm-adjusted p = 0.002) |
GPT-5.6 Luna | low | +49.4 points [+19.6 to +71.1]; significantly higher (Holm-adjusted p = 0.002) |
Kimi K3 | low | +36.9 points [+15.4 to +59.0]; significantly higher (Holm-adjusted p = 0.002) |
Claude Sonnet 5.5 | low | +27.8 points [+8.9 to +48.6]; significantly higher (Holm-adjusted p = 0.006) |
Claude Sonnet 5 | low | +14.1 points [-0.6 to +34.5]; no significant difference (Holm-adjusted p = 0.19) |
Claude Haiku 5.5 | low | +8.1 points [-4.7 to +25.6]; no significant difference (Holm-adjusted p = 0.48) |
Gemini 4 Argon | low | -5.8 points [-25.9 to +15.7]; no significant difference (Holm-adjusted p = 0.516) |
Against GPT-6.1 Terra, low, the highest recall:
Model | Reasoning effort | Recall difference |
|---|---|---|
GPT-5.6 Terra | low | -0.1 points [-1.0 to +0.6]; no significant difference (Holm-adjusted p = 1) |
GPT-6 Sol | low | -0.1 points [-1.4 to +1.2]; no significant difference (Holm-adjusted p = 1) |
GPT-6 Luna | low | -2.5 points [-4.7 to -0.4]; no significant difference (Holm-adjusted p = 0.242) |
GPT-5.6 Sol | low | -3.5 points [-10.0 to +1.0]; no significant difference (Holm-adjusted p = 1) |
GPT-6.1 Sol | low | -4.9 points [-11.8 to +1.0]; no significant difference (Holm-adjusted p = 1) |
GPT-6 Astra | low | -5.2 points [-12.9 to +1.5]; no significant difference (Holm-adjusted p = 1) |
Claude Opus 5.5 | low | -5.2 points [-15.8 to +1.3]; no significant difference (Holm-adjusted p = 1) |
Claude Opus 5 | low | -5.7 points [-17.4 to +1.4]; no significant difference (Holm-adjusted p = 1) |
GPT-5 Mini | low | -6.8 points [-21.5 to +5.7]; no significant difference (Holm-adjusted p = 1) |
GPT-5.2 | low | -6.8 points [-17.1 to +2.0]; no significant difference (Holm-adjusted p = 1) |
GPT-5.4 | low | -7.3 points [-14.8 to -0.7]; no significant difference (Holm-adjusted p = 0.232) |
GPT-5.1 | low | -10.6 points [-25.9 to +0.2]; no significant difference (Holm-adjusted p = 0.735) |
GPT-5.6 Luna | low | -11.0 points [-29.0 to -0.5]; no significant difference (Holm-adjusted p = 0.156) |
Kimi K3 | low | -23.5 points [-52.0 to -0.2]; no significant difference (Holm-adjusted p = 0.54) |
Claude Sonnet 5.5 | low | -32.7 points [-56.2 to +0.6]; no significant difference (Holm-adjusted p = 0.735) |
Claude Sonnet 5 | low | -46.4 points [-69.3 to -14.4]; significantly lower (Holm-adjusted p = 0.048) |
Claude Haiku 5.5 | low | -52.4 points [-73.6 to -21.4]; significantly lower (Holm-adjusted p < 0.002) |
GPT-4o (production default) | not applicable | -60.5 points [-76.6 to -35.8]; significantly lower (Holm-adjusted p < 0.002) |
Gemini 4 Argon | low | -66.3 points [-86.7 to -40.3]; significantly lower (Holm-adjusted p < 0.002) |
How the benchmark works
The benchmark is a frozen set of 23 questionnaire documents (13 PDF or DOCX, 10 XLSX workbooks) with 2,967 expected questions. For each document a hand-curated reference, the golden record, lists every question with its guidance and context.
Part of a golden record | What it represents |
|---|---|
Question | The action or information the respondent must provide |
Guidance | Instructions, examples, options, limits and required details for the answer |
Context | The section and parent chain needed to understand the question |
Fabricated example, not benchmark content: section "Information security › Incident management"; question "Describe how material security incidents are escalated"; guidance "Include severity thresholds and notification timelines."
Extract. Each document takes the route for its format. PDF and DOCX documents use hierarchical extraction; a document over 20,000 tokens is read section by section instead. Workbooks use workbook-aware extraction, one worksheet per request, with the same 20,000-token limit per worksheet. The standard route is the first choice; a document counts only when the model completes it on the standard route, and documents that take another path are listed in the notes.
Align. The extracted questions are matched to the golden record, including split, merged, missing and extra questions, so that each extracted entry is compared with the question it stands for.
Grade. Deterministic rules decide the clear cases: identical text, and named equivalences. The rest go to a semantic judge that needs two agreeing votes, with a tie-break on disagreement. A question counts as captured when its wording keeps the response obligation; it is fully preserved when its guidance and context are kept as well.
Every model ran the same documents with the same extraction prompts, aligner and judge; only the extraction model changes between rows, and the Claude models receive the extraction schema in the prompt (see the notes). Recall counts captured questions over expected questions, so a document the model does not complete counts as zero.
Setup
Every model was given the same 23 frozen documents with the same prompts, and was graded by the same aligner and two-vote semantic judge; only the model changes between rows. Every model runs at low reasoning effort; GPT-4o, the production default, is not a reasoning model. GPT-4o is the reference every model is compared with.
Hosting. The OpenAI models run on Azure OpenAI, except GPT-6.1 Terra, which runs on OpenAI. The Claude models run on Anthropic, Kimi K3 on Together AI and Gemini 4 Argon on Google Vertex AI in the EU. The configurations table lists each deployment.
Configurations
Model | Model deployment | Reasoning effort |
|---|---|---|
GPT-6.1 Terra | GPT-6.1 Terra (2026-10-26), OpenAI through LiteLLM | low |
GPT-5.6 Terra | GPT-5.6 Terra (2026-07-09), Azure OpenAI | low |
GPT-6 Sol | GPT-6 Sol (2026-09-22), Azure OpenAI | low |
GPT-6 Luna | GPT-6 Luna (2026-09-22), Azure OpenAI | low |
GPT-5.6 Sol | GPT-5.6 Sol (2026-07-09), Azure OpenAI | low |
GPT-6.1 Sol | GPT-6.1 Sol (2026-09-29), Azure OpenAI | low |
GPT-6 Astra | GPT-6 Astra (2026-09-03), Azure OpenAI | low |
Claude Opus 5.5 | Claude Opus 5.5, Anthropic through LiteLLM; schema in the prompt | low |
Claude Opus 5 | Claude Opus 5, Anthropic through LiteLLM; schema in the prompt | low |
GPT-5 Mini | GPT-5 Mini (2025-08-07), Azure OpenAI | low |
GPT-5.2 | GPT-5.2 (2025-12-11), Azure OpenAI | low |
GPT-5.4 | GPT-5.4 (2026-03-05), Azure OpenAI | low |
GPT-5.1 | GPT-5.1 (2025-11-13), Azure OpenAI | low |
GPT-5.6 Luna | GPT-5.6 Luna (2026-07-09), Azure OpenAI | low |
Kimi K3 | Kimi K3, Together AI through LiteLLM | low |
Claude Sonnet 5.5 | Claude Sonnet 5.5, Anthropic through LiteLLM; schema in the prompt | low |
Claude Sonnet 5 | Claude Sonnet 5, Anthropic through LiteLLM; schema in the prompt | low |
Claude Haiku 5.5 | Claude Haiku 5.5, Anthropic through LiteLLM; schema in the prompt | low |
GPT-4o (production default) | GPT-4o (2024-11-20), Azure OpenAI; October 2026 run | not applicable (not a reasoning model) |
Gemini 4 Argon | Gemini 4 Argon, Google Vertex AI EU through LiteLLM | low |
Notes
Documents outside the standard set. Five documents (1,472 questions) are larger than the standard route reads in one request: 20,000 tokens for a PDF or Word document, or for one worksheet of a workbook. Their results reflect document size, not the extraction model. The three PDF and Word documents are read section by section for every model, which is how the product handles long documents. The two workbooks each contain a worksheet over the limit, and this benchmark has no section-by-section result for workbooks, so they are not scored.
Documents not completed. Gemini 4 Argon does not complete nine of the 18 standard documents; it returned a well-formed but empty question list on them. On five of them, reading the document section by section afterwards found questions, which does not count as completing it on the standard route. Claude Haiku 5.5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Claude Sonnet 5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Claude Sonnet 5.5 does not complete two of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. GPT-5.1 does not complete one of the 18 standard documents; its reply on it did not match the extraction format. Kimi K3 does not complete one of the 18 standard documents; its requests on it did not return. A document the model does not complete on the standard route counts as zero.
Claude models. Anthropic's structured-output mode rejects the production extraction schema, so the product cannot use Claude for extraction as it is. The Claude models were scored with the same schema given in the prompt instead, and their replies are checked against it in the same way. The platform caps Claude Sonnet and Haiku replies at 10,000 tokens, which cuts off replies on long documents.
Retried documents. The connection this benchmark uses ends a request after 60 seconds, which the product's own connection does not, and some models need longer on the longest documents. Those documents were run again by the same model on a connection without that limit: GPT-6.1 Sol on six, GPT-6 Astra on six, GPT-5.4 on one and GPT-5 Mini on four. GPT-5 Mini on one, Kimi K3 on two, Claude Sonnet 5.5 on one and Claude Haiku 5.5 on two completed documents on a later attempt; their earlier attempts were cut off, did not return, or returned a reply that did not match the extraction format. The documents that Claude Haiku 5.5, Claude Sonnet 5, Claude Sonnet 5.5 and Kimi K3 did not complete failed again on a rerun. The documents that Gemini 4 Argon and GPT-5.1 did not complete were not rerun, because the failure was in the model's reply rather than the connection. For a retried document, time and cost per document cover the completed attempt; the cost of the full run includes every attempt.
Earlier results. The previous version of this page ran each model at its provider's default reasoning and compared nine models on the 11 documents all of them completed. This version runs every model at low reasoning and compares all 20 models on the 18 standard documents, so the figures differ: GPT-6 Sol captured 92% of the 18 documents at its default and captures 91% at low. GPT-4o's figures come from its October 2026 run with the same documents, prompts, aligner and judge.
Cost evidence. Some calls returned no usage data for GPT-5.6 Terra, GPT-6 Luna, GPT-5 Mini, GPT-5.2, GPT-5.4 and Gemini 4 Argon, so their costs are lower bounds and they are left out of the cost chart. For GPT-6.1 Terra, GPT-6.1 Sol, GPT-6 Astra and Kimi K3 the cost per document is complete, but the cost of the full run is a lower bound, because calls that failed returned no usage data. Costs are list prices for the extraction requests only, not alignment or grading, and not invoices, so they are not comparable with Verbatim library answers or Composed answers from library sources costs. GPT-4o's costs use the October 2026 price sheet.
Times are the summed duration of the extraction calls for a document, including calls that failed and were retried within one attempt, so one slow document can dominate a model's mean: GPT-5.6 Terra's slowest document took 1,247 s against a median of 20 s. The time chart and the summary therefore use the median; the results table gives the mean, median and 90th percentile. GPT-4o's times come from its October 2026 run.
Claim boundary. This is a development benchmark: the corpus was used while building extraction, so it is not unseen evidence, not an estimate for all production documents, and not a release gate. Alignment and grading are model-assisted.