Agentic Table-Question Extraction Benchmark — Published Report
5 min read
Published result
Benchmark completed | Corpus | 23 documents · 2,967 expected questions | |
|---|---|---|---|
Extraction model | GPT-5.4, Azure deployment dated | Versions | Extraction prompts r5 · Semantic judge v5 · Benchmark set v1 · Reference corpus v5 |
90.0% headline result
The first-choice route accepted 1,345 of 1,495 expected questions across 18 documents after exact, rule-based, and semantic grading. Three additional documents required fallback, and two produced no usable extraction.
Results at a glance
Measure | Result | Business meaning |
|---|---|---|
Expected questions accepted | 90.0% · 1,345/1,495 | Accepted questions on the standard first-choice extraction route. |
Exact matches | 1,034 | Question text matched the golden reference exactly after normalization. |
Rule-confirmed equivalents | 267 | Non-exact relationships were objectively equivalent under named deterministic rules. |
Semantic judge passes | 44 | Inexact matches were confirmed as preserving the response obligation. |
Document consistency | 12/18 at least 90% | Distribution across standard-route documents rather than one aggregate alone. |
Methodology
Benchmark flow
1. Select | → | 2. Extract | → | 3. Structure | → | 4. Align | → | 5. Grade |
Each document follows the extraction route appropriate to its format. The resulting question tree is aligned to a hand-curated reference and graded for capture, correctness, and preservation of supporting information.
What one golden record contains
Component | What it represents | Why it is measured separately |
|---|---|---|
Question | The action or information the respondent must provide. | Determines whether the response obligation was captured. |
Helper guidance | Instructions, examples, options, limits, and required details. | Shows whether answer requirements remained usable. |
Context | The section and parent chain needed to interpret the question. | Shows whether the question can still be understood in its hierarchy. |
Fabricated example—not benchmark content
Section “Information security › Incident management”; question “Describe how material security incidents are escalated”; guidance “Include severity thresholds and notification timelines.”
1. Execute the format-aware extraction ladder
Input | First-choice route | Fallback behavior | Recorded output |
|---|---|---|---|
PDF / DOCX | Hierarchical extraction | One chunked fallback if the first route safely declines | Candidate tree, flat rows, manifest, and effective-input evidence |
XLSX | Workbook-native extraction | Full-workbook or per-sheet handling; no silent partial success | Candidate tree, flat rows, manifest, and workbook input capture |
The complete candidate is compared with the complete reference. A document that fails remains visible in the result instead of disappearing from the denominator.
2. Align obligations before judging quality
The evaluation matches expected and extracted trees, including one-to-one, split, merged, missing, and extra relationships. Alignment identifies which entries represent the same response obligation before their content is graded. Schema and cardinality checks reject invalid alignment output rather than silently converting it into a missing-question result.
3. Grade with deterministic locks and semantic consensus
Deterministic rules do not rewrite or “clean” extracted text. They lock grades for objectively clear patterns and send only unresolved dimensions to the model-based judge.
Normalize and lock clear cases | → | Route unresolved dimensions | → | Collect two independent votes | → | Tie-break disagreement or uncertainty | → | Require two supporting votes |
In this sitting, deterministic rules locked 5,776 dimension grades and fully decided 1,514 of 2,156 aligned relationships. The model judged 692 dimensions, and 47 relationships required a tie-break.
Semantic judge decision levers
Decision lever | What is checked | Deterministic handling |
|---|---|---|
Question wording | Actions, requested data, conditions, polarity, quantities, scope, time periods, units, and material qualifiers. | Normalized equality and harmless OCR/politeness changes can pass; a clearly omitted obligation or conjunction can fail. |
Helper guidance | Required answer instructions versus optional examples or presentation guidance. | Unchanged or harmlessly relocated guidance can pass; adding or losing a mandatory instruction can fail. |
Context | Whether the candidate is self-contained or its actual parent chain supplies one clear antecedent. | A self-contained root or matching governing parent can pass; an unresolved reference or root fragment can fail. |
Unresolved cases | Meaning that cannot be decided safely by a named rule. | The semantic judge returns pass, partial, fail, or uncertain. Similarity scores remain diagnostic only. |
Results by extraction path
Route | Documents | Expected questions | Accepted after grading | Interpretation |
|---|---|---|---|---|
Standard | 18 | 1,495 | 1,345 · 90.0% | 3 documents captured every question; 5 captured at least 95%; 12 captured at least 90%. |
Chunked fallback | 3 | 912 | 645 | Lower and less consistent capture than the standard route. |
No usable extraction | 2 | 560 | 0 | Document-level failures remain visible. |
Standard-route outcome accounting
Outcome | Expected questions | How it contributes |
|---|---|---|
Exact matches | 1,034 | Accepted |
Rule-confirmed equivalents | 267 | Accepted |
Semantic judge passes | 44 | Accepted |
Missing or unsuccessful | 150 | Not accepted |
Question-wording outcomes determine acceptance. Helper and context grades diagnose preservation quality but do not rescue a missing or incorrect question.
How to interpret the result
Claim boundary
This is a descriptive development benchmark. The corpus has been used during development, so it is not unseen evidence, an unbiased estimate of all production documents, or a release gate. Use it to diagnose improvements and compare like-for-like runs.
Living publication: updated measurements replace previous values on this stable page. Use Confluence version history to inspect earlier editions.