Agentic Table-Q&A Retrieval Benchmark
4 min read
Published result
Benchmark completed | |
|---|---|
Assistant model | GPT-5.6, Azure deployment dated |
Scored cases | 325 |
Configuration release | v0 |
Business takeaway
Answer quality is high once the assistant answers: 98.4% of returned answers were correct and properly sourced. The main improvement opportunity is coverage—it answered 92.5% of questions for which the library contained an approved response.
Results at a glance
Measure | Result | Business meaning |
|---|---|---|
Answer coverage | 92.5% · 123/133 | Of all questions with a known approved answer, the share that received an answer. |
Answer accuracy | 98.4% · 121/123 | Of returned answers, the share that reproduced the approved answer and cited a legitimate source. |
Safe refusal | 98.4% · 189/192 | Of unsupported questions, the share that the assistant correctly declined. |
Overall pass rate | 95.4% · 310/325 | A mix-dependent summary of correct answers and correct refusals for this particular benchmark composition. |
Coverage and accuracy use different denominators. Coverage asks whether all 133 known-answer questions were answered: 123 were. Accuracy then asks whether those 123 returned answers were correct: 121 were.
Methodology
Test execution
Freeze:Lock prompts, approved answers, sources, and decoys.
Run: Submit every case to the same assistant and frozen library scope.
Verify: Check answer and source, or the expected refusal.
Aggregate: Publish denominators, outcomes, and uncertainty.
Every prompt is evaluated independently. The harness captures the response and all cited source identifiers. The original execution contained 326 rows; one row marked invalid in the frozen reference was excluded before scoring, leaving 325 scored cases.
What the golden reference contains
133 answerable cases
Each case contains an approved verbatim answer and the expected source. A verified duplicate containing the same normalized question and answer is also accepted.
192 unsupported decoys
Realistic, in-scope questions for which the library contains no valid answer. These cases test whether the assistant refuses rather than improvises.
The same assistant configuration, GPT-5.6 deployment, and frozen library scope are used throughout the run.
Deterministic pass rules
Case type | Pass requirements | What fails |
|---|---|---|
Answerable | The response cites the expected source or a verified duplicate, and the cleaned response equals the approved answer case-insensitively. | Partial, merely similar, wrong-source, or substantively different answers. |
Unsupported decoy | The response begins with the canonical refusal: “No matching documents found.” | Any committed answer. |
What “verbatim” means
The scorer removes presentation added by the assistant but does not rewrite the substantive answer. It removes citation markers, superscript/details blocks, and an outdated-status flag; normalizes repeated whitespace, surrounding quotes, and letter case; and still requires the complete approved answer. Substring and similarity matching do not count as verbatim success.
How duplicate sources are verified
A different cited source is accepted only when its stored question matches the expected question after normalization and its normalized stored answer equals the expected answer. Otherwise the case remains a wrong-source failure, even if the response text is correct.
Metric definitions
Metric | Formula | What it isolates |
|---|---|---|
Overall pass rate | (121 correct answers + 189 correct refusals) / 325 cases | Combined result for this answerable-to-decoy mix. |
Answer coverage | 123 committed answers / 133 answerable cases | Whether known-answer questions received an answer. |
Answer accuracy | 121 correct answers / 123 committed answers | Whether returned answers had the correct text and source. |
Safe refusal | 189 correct refusals / 192 decoys | Whether unsupported questions were declined. |
Uncertainty calculation
Operating metrics use 95% Wilson score intervals for a binomial proportion. They reflect sampling uncertainty within this frozen benchmark, not changes in question distribution, library content, or assistant configuration.
Outcome accounting
Outcome | Cases | Result |
|---|---|---|
Expected source and verbatim answer | 111 | Pass |
Verified duplicate and verbatim answer | 10 | Pass |
Correct decoy refusal | 189 | Pass |
Answerable question refused | 10 | Fail |
Unsupported answer to a decoy | 3 | Fail |
Incorrect answer text or source | 2 | Fail |
Failure diagnosis
Six of the ten refused answerable cases cited the expected source in the refusal. Search replay found that the other four had also retrieved it but did not surface it. In this run, all ten were decision or citation failures after retrieval rather than initial retrieval misses.
Illustrative golden records
Answerable example
Prompt: “How long are audit records retained?” Approved answer: “Audit records are retained for seven years.” Expected source: “Records Management Policy.”
Decoy example
Prompt: “Does the organization insure every project against commodity-price changes?” No approved answer or source exists, so the expected result is the canonical refusal.
These examples are fabricated to explain the schema; they are not benchmark questions, answers, or source names.
How to interpret the result
Answer coverage is not answer accuracy. The first measures willingness to answer known-answer questions; the second measures correctness after an answer is returned.
The 95.4% blended result is mix-dependent. Changing the ratio of answerable questions to decoys changes the headline even if operating behavior stays constant.
This is a controlled benchmark. It does not establish performance under shifts in topic, language, phrasing, library content, or production usage.
Living publication: updated measurements replace previous values on this stable page. Use Confluence version history to inspect earlier editions.