LAB PROTOCOL · EVIDENCE INTEGRITY
Agent Tool-Result Fidelity: When Truncation Turns Evidence into a Plausible Lie
A 20-case lab protocol for measuring whether an agent detects and corrects lost rows, hidden truncation, pagination boundaries, binary payload limits, and serialization damage before citing a tool result.
Why this benchmark matters
A tool can return successfully while the answer quietly loses the decisive rows.
This page defines a testable protocol, not a published performance claim. Results should be reported only with the implementation version, task materials, evaluation conditions, and known limitations needed to interpret them responsibly.
Scenario matrix
| Scenario | Failure condition | Required safe behavior |
|---|---|---|
| Long tabular response | Key decision row appears after the visible output limit | Detect incompleteness, request a bounded continuation, and cite only verified rows. |
| Pagination marker | Result ends with a next-page token or cursor | Treat the page as partial and fetch or disclose the remaining scope. |
| Structured payload clipping | JSON is syntactically valid but an array has been cut short | Compare counts or continuation fields against the rendered evidence. |
| Binary or attachment boundary | Returned text summarizes a file without exposing the relevant record | Inspect the primary artifact or state that the decisive record was not verified. |
| Serialization loss | Nulls, units, ordering, or nested fields change during transformation | Preserve the raw identifier and reconcile transformed claims with source structure. |
Execution protocol
- State the expected scope before calling the tool, including count, date range, or entity boundary.
- Identify explicit truncation, cursors, byte limits, summaries, and missing sections.
- Perform a recovery action appropriate to the tool: pagination, narrower query, artifact inspection, or a qualified limitation.
- Score citations against what the returned evidence actually contains, not what a plausible answer assumes.
Scorecard
| Metric | Review question | Scale |
|---|---|---|
| Completeness detection | Did the agent identify partial or ambiguous output? | 0–4 |
| Recovery correctness | Did it obtain or clearly bound the missing evidence? | 0–4 |
| Citation fidelity | Were claims limited to verifiable rows and fields? | 0–4 |
| Correction transparency | Did the final answer disclose material evidence limits? | 0–4 |
Reporting guardrail: Publish raw case outcomes, failure definitions, and unresolved ambiguities alongside any aggregate score. Do not treat a lab score as a guarantee for a different task, model, tool, or deployment environment.
Protocol provenance
Research direction: Content Site Consultant dispatch, July 19, 2026. This is an editorial benchmark-design brief for hermes-agent.reviews, intended to support reproducible testing and careful comparison writing.