Agentic Table-Questionnaire Question Extraction Benchmark

19 min read

Questionnaire question extraction turns an uploaded questionnaire into the list of questions the Agentic Table answers. This page reports how well it does that with each extraction model we evaluated. The documents and grading are described under How the benchmark works below.

All 20 models are compared on the 18 standard documents, 1,495 questions; a document a model did not complete counts against it. Each model ran once, so run-to-run noise is not measured.

Key findings

  • On the 18 standard documents, the best models capture 91% of questions (GPT-6.1 Terra, GPT-5.6 Terra and GPT-6 Sol), and the differences between them are not significant. GPT-6.1 Terra, GPT-5.6 Terra and GPT-6 Sol are within 0.1 points of each other. GPT-6 Luna, GPT-5.6 Sol, GPT-6.1 Sol, GPT-6 Astra, Claude Opus 5.5, Claude Opus 5, GPT-5 Mini, GPT-5.2, GPT-5.4, GPT-5.1, GPT-5.6 Luna, Kimi K3 and Claude Sonnet 5.5 are not significantly different from GPT-6.1 Terra either; Claude Sonnet 5, Claude Haiku 5.5, GPT-4o and Gemini 4 Argon are significantly lower. With 18 documents the tests separate only large differences: even Claude Sonnet 5.5, at 58%, is not significantly different from GPT-6.1 Terra.

  • Most models complete every standard document. 14 of the 20 models complete all 18 standard documents. Claude Haiku 5.5, Claude Sonnet 5 and Claude Sonnet 5.5 do not complete 4, 4 and 2 of them, because their replies are cut off at the platform's 10,000-token output limit; Gemini 4 Argon does not complete 9, because it returns an empty question list on them; GPT-5.1 does not complete 1, because its reply did not match the extraction format; Kimi K3 does not complete 1, because its requests on that document did not return.

  • Claude models cannot be used for extraction in the product as it is. Anthropic's structured-output mode rejects the production extraction schema, so the Claude models were scored with the schema given in the prompt instead.

Results

On the 18 standard documents (1,495 questions). 95% intervals in brackets, from a bootstrap over documents, because questions in the same document succeed or fail together. Time, cost per document, precision and fully preserved cover the documents each model completed. Costs are list prices for the extraction calls; "at least" means some calls returned no usage data, so the figure is a lower bound. The full run is all 23 documents, including the ones outside the standard set. Models are ranked by questions captured, highest first.

Model

Reasoning effort

Provider

Questions captured (1,495)

Fully preserved: question, guidance and context

Precision: extracted rows that match a reference question

Standard documents completed (of 18)

Time per document, mean / median / 90th percentile

Cost per document (mean)

Cost of the full 23-document run

GPT-6.1 Terra

low

OpenAI

91% (1,363/1,495) [83–97%]

70% (1,046/1,495) [52–84%]

96% (60 extra of 1,545)

18

95 s / 42 s / 124 s

USD 0.030

at least USD 1.05

GPT-5.6 Terra

low

OpenAI

91% (1,362/1,495) [83–96%]

69% (1,035/1,495) [51–83%]

97% (40 extra of 1,523)

18

92 s / 20 s / 43 s

at least USD 0.067

at least USD 2.43

GPT-6 Sol

low

OpenAI

91% (1,361/1,495) [83–96%]

69% (1,039/1,495) [52–84%]

98% (30 extra of 1,512)

18

24 s / 19 s / 39 s

USD 0.061

USD 2.28

GPT-6 Luna

low

OpenAI

89% (1,326/1,495) [80–95%]

67% (1,005/1,495) [49–82%]

98% (29 extra of 1,484)

18

53 s / 38 s / 70 s

at least USD 0.003

at least USD 0.12

GPT-5.6 Sol

low

OpenAI

88% (1,311/1,495) [78–95%]

69% (1,027/1,495) [50–83%]

95% (84 extra of 1,550)

18

87 s / 26 s / 216 s

USD 0.172

USD 6.38

GPT-6.1 Sol

low

OpenAI

86% (1,290/1,495) [76–95%]

70% (1,046/1,495) [52–85%]

95% (73 extra of 1,529)

18

42 s / 43 s / 56 s

USD 0.059

at least USD 2.09

GPT-6 Astra

low

OpenAI

86% (1,286/1,495) [75–95%]

70% (1,051/1,495) [52–85%]

95% (74 extra of 1,527)

18

52 s / 41 s / 116 s

USD 0.303

at least USD 10.74

Claude Opus 5.5

low

Anthropic

86% (1,285/1,495) [73–96%]

68% (1,014/1,495) [49–84%]

97% (43 extra of 1,462)

18

89 s / 35 s / 132 s

USD 0.197

USD 7.24

Claude Opus 5

low

Anthropic

85% (1,278/1,495) [71–96%]

68% (1,013/1,495) [48–84%]

98% (37 extra of 1,521)

18

88 s / 46 s / 125 s

USD 0.248

USD 8.89

GPT-5 Mini

low

OpenAI

84% (1,261/1,495) [70–94%]

66% (994/1,495) [49–81%]

94% (87 extra of 1,483)

18

72 s / 37 s / 143 s

at least USD 0.014

at least USD 0.54

GPT-5.2

low

OpenAI

84% (1,261/1,495) [72–95%]

69% (1,025/1,495) [50–84%]

96% (61 extra of 1,547)

18

43 s / 33 s / 48 s

at least USD 0.086

at least USD 2.75

GPT-5.4

low

OpenAI

84% (1,254/1,495) [73–93%]

69% (1,030/1,495) [51–83%]

94% (94 extra of 1,569)

18

55 s / 31 s / 75 s

at least USD 0.087

at least USD 3.20

GPT-5.1

low

OpenAI

81% (1,204/1,495) [62–94%]

65% (973/1,495) [44–82%]

96% (63 extra of 1,424)

17

16 s / 15 s / 24 s

USD 0.055

USD 2.43

GPT-5.6 Luna

low

OpenAI

80% (1,198/1,495) [60–94%]

63% (935/1,495) [40–81%]

99% (18 extra of 1,355)

18

20 s / 14 s / 29 s

USD 0.007

USD 0.25

Kimi K3

low

Moonshot AI

68% (1,011/1,495) [42–91%]

51% (767/1,495) [30–75%]

98% (18 extra of 1,183)

17

54 s / 43 s / 124 s

USD 0.096

at least USD 4.39

Claude Sonnet 5.5

low

Anthropic

58% (874/1,495) [37–87%]

43% (642/1,495) [26–66%]

94% (59 extra of 1,021)

16

23 s / 22 s / 42 s

USD 0.083

USD 4.34

Claude Sonnet 5

low

Anthropic

45% (670/1,495) [24–75%]

33% (487/1,495) [16–58%]

97% (21 extra of 810)

14

38 s / 36 s / 49 s

USD 0.080

USD 5.23

Claude Haiku 5.5

low

Anthropic

39% (580/1,495) [21–65%]

23% (350/1,495) [12–43%]

96% (30 extra of 773)

14

63 s / 19 s / 30 s

USD 0.004

USD 0.36

GPT-4o (production default)

not applicable

OpenAI

31% (459/1,495) [18–50%]

22% (322/1,495) [13–36%]

96% (21 extra of 508)

18

8 s / 7 s / 11 s

USD 0.038

USD 1.79

Gemini 4 Argon

low

Google

25% (372/1,495) [7–48%]

18% (275/1,495) [4–39%]

99% (6 extra of 456)

9

142 s / 113 s / 244 s

at least USD 0.052

at least USD 3.81

Per 1,000 questions

Scaled from the 1,495 questions of the 18 standard documents, rounded so that captured and not captured add up to 1,000:

Model

Reasoning effort

Provider

Captured

of which fully preserved

Not captured

Extra rows with no reference question

GPT-6.1 Terra

low

OpenAI

about 912

about 700

about 88

about 40

GPT-5.6 Terra

low

OpenAI

about 911

about 692

about 89

about 27

GPT-6 Sol

low

OpenAI

about 910

about 695

about 90

about 20

GPT-6 Luna

low

OpenAI

about 887

about 672

about 113

about 19

GPT-5.6 Sol

low

OpenAI

about 877

about 687

about 123

about 56

GPT-6.1 Sol

low

OpenAI

about 863

about 700

about 137

about 49

GPT-6 Astra

low

OpenAI

about 860

about 703

about 140

about 49

Claude Opus 5.5

low

Anthropic

about 860

about 678

about 140

about 29

Claude Opus 5

low

Anthropic

about 855

about 678

about 145

about 25

GPT-5 Mini

low

OpenAI

about 843

about 665

about 157

about 58

GPT-5.2

low

OpenAI

about 843

about 686

about 157

about 41

GPT-5.4

low

OpenAI

about 839

about 689

about 161

about 63

GPT-5.1

low

OpenAI

about 805

about 651

about 195

about 42

GPT-5.6 Luna

low

OpenAI

about 801

about 625

about 199

about 12

Kimi K3

low

Moonshot AI

about 676

about 513

about 324

about 12

Claude Sonnet 5.5

low

Anthropic

about 585

about 429

about 415

about 39

Claude Sonnet 5

low

Anthropic

about 448

about 326

about 552

about 14

Claude Haiku 5.5

low

Anthropic

about 388

about 234

about 612

about 20

GPT-4o (production default)

not applicable

OpenAI

about 307

about 215

about 693

about 14

Gemini 4 Argon

low

Google

about 249

about 184

about 751

about 4

Model notes

  • GPT-6.1 Terra, low: captures 91% (1,363/1,495) of questions [83–97%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 42 s per document, USD 0.030 per document.

  • GPT-5.6 Terra, low: captures 91% (1,362/1,495) of questions [83–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 20 s per document, at least USD 0.067 per document.

  • GPT-6 Sol, low: captures 91% (1,361/1,495) of questions [83–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 19 s per document, USD 0.061 per document.

  • GPT-6 Luna, low: captures 89% (1,326/1,495) of questions [80–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 38 s per document, at least USD 0.003 per document.

  • GPT-5.6 Sol, low: captures 88% (1,311/1,495) of questions [78–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 26 s per document, USD 0.172 per document.

  • GPT-6.1 Sol, low: captures 86% (1,290/1,495) of questions [76–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 43 s per document, USD 0.059 per document.

  • GPT-6 Astra, low: captures 86% (1,286/1,495) of questions [75–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 41 s per document, USD 0.303 per document. GPT-6 Astra is the most expensive per document of the reasoning models that complete every standard document; for cost, of those with complete cost evidence.

  • Claude Opus 5.5, low: captures 86% (1,285/1,495) of questions [73–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 35 s per document, USD 0.197 per document. Scored with the schema in the prompt.

  • Claude Opus 5, low: captures 85% (1,278/1,495) of questions [71–96%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 46 s per document, USD 0.248 per document. Claude Opus 5 is the slowest by median time of the reasoning models that complete every standard document. Scored with the schema in the prompt.

  • GPT-5 Mini, low: captures 84% (1,261/1,495) of questions [70–94%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 37 s per document, at least USD 0.014 per document.

  • GPT-5.2, low: captures 84% (1,261/1,495) of questions [72–95%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 33 s per document, at least USD 0.086 per document.

  • GPT-5.4, low: captures 84% (1,254/1,495) of questions [73–93%], significantly higher than GPT-4o (31%, Holm-adjusted p < 0.002). It completes every standard document. Median 31 s per document, at least USD 0.087 per document.

  • GPT-5.1, low: captures 81% (1,204/1,495) of questions [62–94%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.002). It completes 17 of 18 standard documents. Median 15 s per document, USD 0.055 per document. GPT-5.1 does not complete one of the 18 standard documents; its reply on it did not match the extraction format.

  • GPT-5.6 Luna, low: captures 80% (1,198/1,495) of questions [60–94%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.002). It completes every standard document. Median 14 s per document, USD 0.007 per document. GPT-5.6 Luna is the cheapest per document and the fastest by median time of the reasoning models that complete every standard document; for cost, of those with complete cost evidence.

  • Kimi K3, low: captures 68% (1,011/1,495) of questions [42–91%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.002). It completes 17 of 18 standard documents. Median 43 s per document, USD 0.096 per document. Kimi K3 does not complete one of the 18 standard documents; its requests on it did not return.

  • Claude Sonnet 5.5, low: captures 58% (874/1,495) of questions [37–87%], significantly higher than GPT-4o (31%, Holm-adjusted p = 0.006). It completes 16 of 18 standard documents. Median 22 s per document, USD 0.083 per document. Claude Sonnet 5.5 does not complete two of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Scored with the schema in the prompt.

  • Claude Sonnet 5, low: captures 45% (670/1,495) of questions [24–75%], no significant difference from GPT-4o (Holm-adjusted p = 0.19). It completes 14 of 18 standard documents. Median 36 s per document, USD 0.080 per document. Claude Sonnet 5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Scored with the schema in the prompt.

  • Claude Haiku 5.5, low: captures 39% (580/1,495) of questions [21–65%], no significant difference from GPT-4o (Holm-adjusted p = 0.48). It completes 14 of 18 standard documents. Median 19 s per document, USD 0.004 per document. Claude Haiku 5.5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Scored with the schema in the prompt.

  • GPT-4o, no reasoning (production default): captures 31% (459/1,495) of questions [18–50%]. It completes every standard document. Median 7 s per document, USD 0.038 per document. The production default. Misses about two in three questions. Not suitable.

  • Gemini 4 Argon, low: captures 25% (372/1,495) of questions [7–48%], no significant difference from GPT-4o (Holm-adjusted p = 0.516). It completes 9 of 18 standard documents. Median 113 s per document, at least USD 0.052 per document. Gemini 4 Argon does not complete nine of the 18 standard documents; it returned a well-formed but empty question list on them. On five of them, reading the document section by section afterwards found questions, which does not count as completing it on the standard route.

Statistical tests

Paired tests on the same documents: the recall difference in points, with a 95% bootstrap interval over the 18 documents and a two-sided p-value, Holm-adjusted across the comparisons in each table. The intervals are not adjusted, so an interval that excludes zero can still be not significant after the adjustment. With 10,000 resamples the smallest p-value one test can show is 0.0001; after adjusting 19 comparisons that floor is 0.002, shown as p < 0.002. Per-question McNemar tests are not used because questions in one document are not independent.

Against GPT-4o, the production default:

Model

Reasoning effort

Recall difference

GPT-6.1 Terra

low

+60.5 points [+35.8 to +76.6]; significantly higher (Holm-adjusted p < 0.002)

GPT-5.6 Terra

low

+60.4 points [+35.7 to +76.6]; significantly higher (Holm-adjusted p < 0.002)

GPT-6 Sol

low

+60.3 points [+35.9 to +76.5]; significantly higher (Holm-adjusted p < 0.002)

GPT-6 Luna

low

+58.0 points [+32.9 to +74.7]; significantly higher (Holm-adjusted p < 0.002)

GPT-5.6 Sol

low

+57.0 points [+32.5 to +74.2]; significantly higher (Holm-adjusted p < 0.002)

GPT-6.1 Sol

low

+55.6 points [+32.0 to +73.3]; significantly higher (Holm-adjusted p < 0.002)

GPT-6 Astra

low

+55.3 points [+31.5 to +73.4]; significantly higher (Holm-adjusted p < 0.002)

Claude Opus 5.5

low

+55.3 points [+29.5 to +74.2]; significantly higher (Holm-adjusted p < 0.002)

Claude Opus 5

low

+54.8 points [+28.4 to +74.1]; significantly higher (Holm-adjusted p < 0.002)

GPT-5 Mini

low

+53.6 points [+29.6 to +71.5]; significantly higher (Holm-adjusted p < 0.002)

GPT-5.2

low

+53.6 points [+29.2 to +72.8]; significantly higher (Holm-adjusted p < 0.002)

GPT-5.4

low

+53.2 points [+29.4 to +71.2]; significantly higher (Holm-adjusted p < 0.002)

GPT-5.1

low

+49.8 points [+21.0 to +71.2]; significantly higher (Holm-adjusted p = 0.002)

GPT-5.6 Luna

low

+49.4 points [+19.6 to +71.1]; significantly higher (Holm-adjusted p = 0.002)

Kimi K3

low

+36.9 points [+15.4 to +59.0]; significantly higher (Holm-adjusted p = 0.002)

Claude Sonnet 5.5

low

+27.8 points [+8.9 to +48.6]; significantly higher (Holm-adjusted p = 0.006)

Claude Sonnet 5

low

+14.1 points [-0.6 to +34.5]; no significant difference (Holm-adjusted p = 0.19)

Claude Haiku 5.5

low

+8.1 points [-4.7 to +25.6]; no significant difference (Holm-adjusted p = 0.48)

Gemini 4 Argon

low

-5.8 points [-25.9 to +15.7]; no significant difference (Holm-adjusted p = 0.516)

Against GPT-6.1 Terra, low, the highest recall:

Model

Reasoning effort

Recall difference

GPT-5.6 Terra

low

-0.1 points [-1.0 to +0.6]; no significant difference (Holm-adjusted p = 1)

GPT-6 Sol

low

-0.1 points [-1.4 to +1.2]; no significant difference (Holm-adjusted p = 1)

GPT-6 Luna

low

-2.5 points [-4.7 to -0.4]; no significant difference (Holm-adjusted p = 0.242)

GPT-5.6 Sol

low

-3.5 points [-10.0 to +1.0]; no significant difference (Holm-adjusted p = 1)

GPT-6.1 Sol

low

-4.9 points [-11.8 to +1.0]; no significant difference (Holm-adjusted p = 1)

GPT-6 Astra

low

-5.2 points [-12.9 to +1.5]; no significant difference (Holm-adjusted p = 1)

Claude Opus 5.5

low

-5.2 points [-15.8 to +1.3]; no significant difference (Holm-adjusted p = 1)

Claude Opus 5

low

-5.7 points [-17.4 to +1.4]; no significant difference (Holm-adjusted p = 1)

GPT-5 Mini

low

-6.8 points [-21.5 to +5.7]; no significant difference (Holm-adjusted p = 1)

GPT-5.2

low

-6.8 points [-17.1 to +2.0]; no significant difference (Holm-adjusted p = 1)

GPT-5.4

low

-7.3 points [-14.8 to -0.7]; no significant difference (Holm-adjusted p = 0.232)

GPT-5.1

low

-10.6 points [-25.9 to +0.2]; no significant difference (Holm-adjusted p = 0.735)

GPT-5.6 Luna

low

-11.0 points [-29.0 to -0.5]; no significant difference (Holm-adjusted p = 0.156)

Kimi K3

low

-23.5 points [-52.0 to -0.2]; no significant difference (Holm-adjusted p = 0.54)

Claude Sonnet 5.5

low

-32.7 points [-56.2 to +0.6]; no significant difference (Holm-adjusted p = 0.735)

Claude Sonnet 5

low

-46.4 points [-69.3 to -14.4]; significantly lower (Holm-adjusted p = 0.048)

Claude Haiku 5.5

low

-52.4 points [-73.6 to -21.4]; significantly lower (Holm-adjusted p < 0.002)

GPT-4o (production default)

not applicable

-60.5 points [-76.6 to -35.8]; significantly lower (Holm-adjusted p < 0.002)

Gemini 4 Argon

low

-66.3 points [-86.7 to -40.3]; significantly lower (Holm-adjusted p < 0.002)

How the benchmark works

The benchmark is a frozen set of 23 questionnaire documents (13 PDF or DOCX, 10 XLSX workbooks) with 2,967 expected questions. For each document a hand-curated reference, the golden record, lists every question with its guidance and context.

Part of a golden record

What it represents

Question

The action or information the respondent must provide

Guidance

Instructions, examples, options, limits and required details for the answer

Context

The section and parent chain needed to understand the question

Fabricated example, not benchmark content: section "Information security › Incident management"; question "Describe how material security incidents are escalated"; guidance "Include severity thresholds and notification timelines."

  • Extract. Each document takes the route for its format. PDF and DOCX documents use hierarchical extraction; a document over 20,000 tokens is read section by section instead. Workbooks use workbook-aware extraction, one worksheet per request, with the same 20,000-token limit per worksheet. The standard route is the first choice; a document counts only when the model completes it on the standard route, and documents that take another path are listed in the notes.

  • Align. The extracted questions are matched to the golden record, including split, merged, missing and extra questions, so that each extracted entry is compared with the question it stands for.

  • Grade. Deterministic rules decide the clear cases: identical text, and named equivalences. The rest go to a semantic judge that needs two agreeing votes, with a tie-break on disagreement. A question counts as captured when its wording keeps the response obligation; it is fully preserved when its guidance and context are kept as well.

Every model ran the same documents with the same extraction prompts, aligner and judge; only the extraction model changes between rows, and the Claude models receive the extraction schema in the prompt (see the notes). Recall counts captured questions over expected questions, so a document the model does not complete counts as zero.

Setup

Every model was given the same 23 frozen documents with the same prompts, and was graded by the same aligner and two-vote semantic judge; only the model changes between rows. Every model runs at low reasoning effort; GPT-4o, the production default, is not a reasoning model. GPT-4o is the reference every model is compared with.

Hosting. The OpenAI models run on Azure OpenAI, except GPT-6.1 Terra, which runs on OpenAI. The Claude models run on Anthropic, Kimi K3 on Together AI and Gemini 4 Argon on Google Vertex AI in the EU. The configurations table lists each deployment.

Configurations

Model

Model deployment

Reasoning effort

GPT-6.1 Terra

GPT-6.1 Terra (2026-10-26), OpenAI through LiteLLM

low

GPT-5.6 Terra

GPT-5.6 Terra (2026-07-09), Azure OpenAI

low

GPT-6 Sol

GPT-6 Sol (2026-09-22), Azure OpenAI

low

GPT-6 Luna

GPT-6 Luna (2026-09-22), Azure OpenAI

low

GPT-5.6 Sol

GPT-5.6 Sol (2026-07-09), Azure OpenAI

low

GPT-6.1 Sol

GPT-6.1 Sol (2026-09-29), Azure OpenAI

low

GPT-6 Astra

GPT-6 Astra (2026-09-03), Azure OpenAI

low

Claude Opus 5.5

Claude Opus 5.5, Anthropic through LiteLLM; schema in the prompt

low

Claude Opus 5

Claude Opus 5, Anthropic through LiteLLM; schema in the prompt

low

GPT-5 Mini

GPT-5 Mini (2025-08-07), Azure OpenAI

low

GPT-5.2

GPT-5.2 (2025-12-11), Azure OpenAI

low

GPT-5.4

GPT-5.4 (2026-03-05), Azure OpenAI

low

GPT-5.1

GPT-5.1 (2025-11-13), Azure OpenAI

low

GPT-5.6 Luna

GPT-5.6 Luna (2026-07-09), Azure OpenAI

low

Kimi K3

Kimi K3, Together AI through LiteLLM

low

Claude Sonnet 5.5

Claude Sonnet 5.5, Anthropic through LiteLLM; schema in the prompt

low

Claude Sonnet 5

Claude Sonnet 5, Anthropic through LiteLLM; schema in the prompt

low

Claude Haiku 5.5

Claude Haiku 5.5, Anthropic through LiteLLM; schema in the prompt

low

GPT-4o (production default)

GPT-4o (2024-11-20), Azure OpenAI; October 2026 run

not applicable (not a reasoning model)

Gemini 4 Argon

Gemini 4 Argon, Google Vertex AI EU through LiteLLM

low

Notes

  • Documents outside the standard set. Five documents (1,472 questions) are larger than the standard route reads in one request: 20,000 tokens for a PDF or Word document, or for one worksheet of a workbook. Their results reflect document size, not the extraction model. The three PDF and Word documents are read section by section for every model, which is how the product handles long documents. The two workbooks each contain a worksheet over the limit, and this benchmark has no section-by-section result for workbooks, so they are not scored.

  • Documents not completed. Gemini 4 Argon does not complete nine of the 18 standard documents; it returned a well-formed but empty question list on them. On five of them, reading the document section by section afterwards found questions, which does not count as completing it on the standard route. Claude Haiku 5.5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Claude Sonnet 5 does not complete four of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. Claude Sonnet 5.5 does not complete two of the 18 standard documents; its replies on them are cut off at the platform's 10,000-token output limit. GPT-5.1 does not complete one of the 18 standard documents; its reply on it did not match the extraction format. Kimi K3 does not complete one of the 18 standard documents; its requests on it did not return. A document the model does not complete on the standard route counts as zero.

  • Claude models. Anthropic's structured-output mode rejects the production extraction schema, so the product cannot use Claude for extraction as it is. The Claude models were scored with the same schema given in the prompt instead, and their replies are checked against it in the same way. The platform caps Claude Sonnet and Haiku replies at 10,000 tokens, which cuts off replies on long documents.

  • Retried documents. The connection this benchmark uses ends a request after 60 seconds, which the product's own connection does not, and some models need longer on the longest documents. Those documents were run again by the same model on a connection without that limit: GPT-6.1 Sol on six, GPT-6 Astra on six, GPT-5.4 on one and GPT-5 Mini on four. GPT-5 Mini on one, Kimi K3 on two, Claude Sonnet 5.5 on one and Claude Haiku 5.5 on two completed documents on a later attempt; their earlier attempts were cut off, did not return, or returned a reply that did not match the extraction format. The documents that Claude Haiku 5.5, Claude Sonnet 5, Claude Sonnet 5.5 and Kimi K3 did not complete failed again on a rerun. The documents that Gemini 4 Argon and GPT-5.1 did not complete were not rerun, because the failure was in the model's reply rather than the connection. For a retried document, time and cost per document cover the completed attempt; the cost of the full run includes every attempt.

  • Earlier results. The previous version of this page ran each model at its provider's default reasoning and compared nine models on the 11 documents all of them completed. This version runs every model at low reasoning and compares all 20 models on the 18 standard documents, so the figures differ: GPT-6 Sol captured 92% of the 18 documents at its default and captures 91% at low. GPT-4o's figures come from its October 2026 run with the same documents, prompts, aligner and judge.

  • Cost evidence. Some calls returned no usage data for GPT-5.6 Terra, GPT-6 Luna, GPT-5 Mini, GPT-5.2, GPT-5.4 and Gemini 4 Argon, so their costs are lower bounds and they are left out of the cost chart. For GPT-6.1 Terra, GPT-6.1 Sol, GPT-6 Astra and Kimi K3 the cost per document is complete, but the cost of the full run is a lower bound, because calls that failed returned no usage data. Costs are list prices for the extraction requests only, not alignment or grading, and not invoices, so they are not comparable with Verbatim library answers or Composed answers from library sources costs. GPT-4o's costs use the October 2026 price sheet.

  • Times are the summed duration of the extraction calls for a document, including calls that failed and were retried within one attempt, so one slow document can dominate a model's mean: GPT-5.6 Terra's slowest document took 1,247 s against a median of 20 s. The time chart and the summary therefore use the median; the results table gives the mean, median and 90th percentile. GPT-4o's times come from its October 2026 run.

  • Claim boundary. This is a development benchmark: the corpus was used while building extraction, so it is not unseen evidence, not an estimate for all production documents, and not a release gate. Alignment and grading are model-assisted.

Last updated