Agentic Table-Question Extraction Benchmark

Question Extraction turns an uploaded questionnaire into the list of questions the Agentic Table answers. This page reports how well it does that with each extraction model we evaluated. Every model was given the same 23 frozen documents with the same prompts, and was graded by the same aligner and two-vote semantic judge; only the model changes between columns. GPT-5.6 Luna and GPT-5 Mini were scored on the PDF/DOCX documents only. No reasoning setting is sent, so each model ran at its provider's default, which is how the product calls extraction. GPT-4o, the production default, is the reference every model is compared with. The documents and grading are described under How the benchmark works below.

Hosting. Every model runs on Azure OpenAI. Extraction uses an Azure OpenAI deployment of GPT-6.1 Sol; the Library Answers and Subagent Answer benchmarks reach it through OpenAI instead.

Headline set. The tables compare the 9 models scored on both routes, on the 11 documents all of them completed on the standard route: 435 questions, 3 PDF/DOCX documents and 8 workbooks, about 15% of the corpus's questions. This keeps every model on the same questions and shows what each one extracts when the route works. The 18 standard documents are reported beside it, with documents a model did not complete counted against it. Each model ran once, so run-to-run noise is not measured.

Summary

• The top models capture 93–94% of questions, and the differences between them are not significant. On the 11 documents every model completes, GPT-6 Astra, GPT-6.1 Sol, GPT-6 Sol and GPT-5.4 are within 0.9 points of each other. GPT-5.6 Terra and GPT-6 Luna are not significantly different from GPT-6 Astra either; GPT-5.2 and GPT-5.6 Sol are significantly lower. On workbooks the models are nearly identical: GPT-6.1 Sol, GPT-6 Sol and GPT-6 Astra each capture 331 of 351 questions, and every model scored on workbooks except GPT-4o is between 89.7% and 94.3%.

• What separates them is completing the document. On the 18 standard documents, where documents a model does not complete count against it, recall ranges from 92% to 27%: GPT-6.1 Sol and GPT-6 Astra do not complete 7 and 6 of them.

• GPT-6 Astra, default: captures 94% of questions (409/435) [90–97%], significantly higher than GPT-4o (61%, p < 0.001). On the 18 standard documents 31% (12 of 18 completed). Mean 30 s per document, USD 0.215 per document. The highest recall here, but it does not complete 6 of the 18 standard documents and costs the most per document of the models with complete cost evidence. Not recommended until those are explained.

• GPT-6.1 Sol, default: captures 94% of questions (407/435) [89–97%], significantly higher than GPT-4o (61%, p < 0.001). On the 18 standard documents 27% (11 of 18 completed). Mean 120 s per document, at least USD 0.040 per document. In the top group on the documents it completes, but it does not complete 7 of the 18 standard documents and is the slowest model. Not recommended.

• GPT-6 Sol, default: captures 93% of questions (406/435) [90–97%], significantly higher than GPT-4o (61%, p < 0.001). On the 18 standard documents 92% (every one completed). Mean 22 s per document, USD 0.047 per document. Recommended: in the top group, completes every standard document, and has complete cost evidence on every document.

• GPT-5.4, default: captures 93% of questions (405/435) [90–96%], significantly higher than GPT-4o (61%, p < 0.001). On the 18 standard documents 72% (17 of 18 completed). Mean 23 s per document, USD 0.055 per document. The reference of the earlier published report. In the top group, but it does not complete 1 of the 18 standard documents in this run.

• GPT-5.6 Terra, default: captures 92% of questions (399/435) [87–95%], significantly higher than GPT-4o (61%, p < 0.001). On the 18 standard documents 84% (every one completed). Mean 16 s per document, USD 0.047 per document. Completes every standard document and is among the fastest models.

• GPT-5.2, default: captures 91% of questions (397/435) [86–95%], significantly higher than GPT-4o (61%, p < 0.001). On the 18 standard documents 86% (every one completed). Mean 14 s per document, USD 0.048 per document. Completes every standard document and is the fastest of the models above 90%.

• GPT-6 Luna, default: captures 91% of questions (394/435) [84–96%], significantly higher than GPT-4o (61%, p = 0.001). On the 18 standard documents 78% (17 of 18 completed). Mean 28 s per document, USD 0.003 per document. The cheapest model per document, but it does not complete 1 of the 18 standard documents and its interval is the widest above 90%.

• GPT-5.6 Sol, default: captures 90% of questions (392/435) [85–94%], significantly higher than GPT-4o (61%, p = 0.001). On the 18 standard documents 91% (every one completed). Mean 18 s per document, USD 0.115 per document. Completes every standard document and is second on the 18-document set, but is below the top group here and costs more than GPT-6 Sol.

• GPT-4o, no reasoning (production default): captures 61% of questions (264/435) [39–80%]. On the 18 standard documents 31% (every one completed). Mean 8 s per document, USD 0.036 per document. The production default. Misses about two in five questions. Not suitable.

Results

On the 11 headline documents (435 questions). 95% intervals in brackets, from a bootstrap over documents, because questions in the same document succeed or fail together. Costs are list prices for the extraction calls; "at least" means some calls returned no usage data, so the figure is a lower bound. The full run is all 23 documents, including the ones outside the headline set.

GPT-5.6 Luna and GPT-5 Mini were scored on PDF/DOCX documents only: their workbook runs did not start, because they failed the pre-run check (a short test extraction before the full run). On the 3 PDF/DOCX documents of the headline set, with every model for comparison:

Per 1,000 questions

Scaled from the 435 headline questions:

The headline set is mostly workbooks. Only 3 of the 10 standard PDF/DOCX documents are in it, because some models did not complete the others. The largest PDF/DOCX documents are outside the standard set; see the notes.

Statistical tests

Paired tests on the same documents: the recall difference in points, with a 95% bootstrap interval over the 11 documents and a two-sided p-value. Per-question McNemar tests are not used because questions in one document are not independent.

How the benchmark works

The benchmark is a frozen set of 23 questionnaire documents (13 PDF or DOCX, 10 XLSX workbooks) with 2,967 expected questions. For each document a hand-curated reference, the golden record, lists every question with its guidance and context.

Fabricated example, not benchmark content: section "Information security › Incident management"; question "Describe how material security incidents are escalated"; guidance "Include severity thresholds and notification timelines."

• Extract. Each document takes the route for its format. PDF and DOCX documents use hierarchical extraction; a document over 20,000 tokens is read section by section instead. Workbooks use workbook-aware extraction, one worksheet per request, with the same 20,000-token limit per worksheet. The standard route is the first choice; documents that take another path are listed in the notes.

• Align. The extracted questions are matched to the golden record, including split, merged, missing and extra questions, so that each extracted entry is compared with the question it stands for.

• Grade. Deterministic rules decide the clear cases: identical text, and named equivalences. The rest go to a semantic judge that needs two agreeing votes, with a tie-break on disagreement. A question counts as captured when its wording keeps the response obligation; it is fully preserved when its guidance and context are kept as well.

Every model ran the same documents (GPT-5.6 Luna and GPT-5 Mini: the PDF/DOCX documents only) with the same extraction prompts, aligner, judge and price sheet; only the extraction model changes between columns. Recall counts captured questions over expected questions, so a document the model does not complete counts as zero.

Configurations

Notes

• Documents outside the headline set. 5 documents (1,472 questions) are larger than the standard route reads in one request: 20,000 tokens for a PDF or Word document, or for one worksheet of a workbook. Their results reflect document size, not the extraction model. The 3 PDF and Word documents were read section by section for every model, which is how the product handles long documents. The 2 workbooks each contain a worksheet over the limit, and in one of them another worksheet's request stopped after about three minutes for most models. The standard route stops on them for every model, and this benchmark has no section-by-section result for workbooks, so these two are not scored. A further 7 documents were not completed by some models only: GPT-6.1 Sol 7, GPT-6 Astra 6, GPT-6 Luna 1, GPT-5.4 1, and GPT-5 Mini 5 on PDF/DOCX. Each stopped after one retry with no recorded cause. They are left out of the headline set and count against the model in the 18-document row.

• Run-to-run. In the earlier published report, a separate run of GPT-5.4 completed all 18 standard documents and captured 90.0% of their questions; in this run it did not complete one of them. Document failures can vary between runs, so the completion figures are not a fixed property of a model.

• Cost evidence. On the headline set, every call has a known list price except for GPT-6.1 Sol, so its cost is a lower bound and it is left out of the cost chart. Over the full 23-document run, cost is complete for GPT-6 Sol, GPT-5.6 Terra and GPT-4o; for the others some calls that did not complete returned no usage data. Costs are list prices for the extraction requests only, not alignment or grading, and not invoices, so they are not comparable with Library Answers or Subagent Answer costs.

• Times are the summed duration of the extraction calls for a document.

• Claim boundary. This is a development benchmark: the corpus was used while building extraction, so it is not unseen evidence, not an estimate for all production documents, and not a release gate. Alignment and grading are model-assisted.