PROVENANCE INTEGRITY LAB
Agent Evidence Provenance: When a Correct-Looking Answer Cannot Show Its Chain of Support
A 20-case lab for source capture, claim-to-evidence links, quote fidelity, stale citations, conflicting sources, tool-result provenance, and unsupported synthesis.
Lab premise
An answer is not auditable merely because it includes a link. Provenance integrity tests whether a reader can trace a material claim to the evidence that actually supports it, including its limits and later correction.
Method note: Use fixed fixtures, explicit expected behavior, and reviewable traces. Publish the conditions and limitations alongside any score so readers can judge the boundary of the result.
Measurement matrix
| Measure | What the evaluator inspects | Decision use |
|---|---|---|
| Claim traceability | Can each material claim be connected to a specific source, excerpt, tool result, or observation? | Auditability |
| Quote fidelity | Do quotations preserve source meaning, scope, and qualification? | Evidence accuracy |
| Freshness and conflict handling | Are stale, contradictory, or superseded sources surfaced before synthesis? | Correction safety |
| Unsupported synthesis rate | Does the answer introduce conclusions that exceed the available evidence? | Decision trust |
Fixture coverage
These bounded fixtures expose specific failure modes. They are not a substitute for production monitoring or a blanket capability claim.
- Claim with a valid link but no supporting passage
- Accurate quote used outside its original scope
- Stale citation that conflicts with a newer primary record
- Tool output with missing origin metadata
- Multiple sources with unresolved disagreement
- Fluent synthesis that contains an unsupported inference
Evaluation protocol
- Capture source identity, retrieval time, relevant excerpt, and claim linkage for every scored response.
- Score whether a reader can reproduce the chain from conclusion to evidence without guessing.
- Inject stale, conflicting, and incomplete records across the 20 fixtures.
- Require explicit uncertainty or verification when source provenance is insufficient.
- Publish corrections by identifying the affected claim and the evidence that changed it.
Interpretation boundary
A useful comparison should disclose model version, permissions, task context, test inputs, expected behavior, observed artifacts, and unresolved limitations. The purpose is to make future correction possible, not to manufacture a definitive aggregate score.