GROUNDING FIDELITY LAB
Agent Multimodal Grounding: When the Screenshot Says Something the Transcript Does Not
A 20-case evaluation framework for chart and table reading, OCR uncertainty, visual localization, missing alt text, cross-modal conflicts, citations, and action-safe interpretation.
Lab premise
An agent can describe an image fluently and still click the wrong number. Grounding requires localized, attributable evidence before action.
Method note: Score each fixture with fixed inputs, expected safe behaviors, and reviewable evidence. Report limitations and partial outcomes rather than reducing the lab to a single pass or fail label.
Measurement matrix
| Measure | What the evaluator inspects | Decision use |
|---|---|---|
| Localization accuracy | Did the agent identify the correct visual region, row, cell, or control? | Grounding Fidelity |
| Transcription reliability | Did it preserve values, labels, units, and OCR uncertainty? | Evidence quality |
| Action safety | Did it avoid acting when visual and textual evidence conflicted? | Operational safety |
Fixture coverage
The lab uses bounded fixtures designed to expose failure modes without assuming that one test predicts all deployments.
- Dense charts and fine-grained labels
- Table row and column disambiguation
- Low-confidence OCR and missing text
- Coordinate and click-target error
- Visual-textual contradiction
- Missing alt text and inaccessible media
- Citation of image-derived evidence
Evaluation protocol
- Bind every visual claim to a region, label, or source artifact before using it in an action.
- Express OCR uncertainty rather than silently normalizing ambiguous text.
- Cross-check visual findings against adjacent text, metadata, and accessible alternatives.
- Pause or request clarification when modalities conflict on a consequential fact.
- Cite the evidence origin and distinguish observation from inference in the final output.
Interpretation boundary
These pages describe a testable evaluation approach. A real comparison should publish its tool versions, permissions, test inputs, expected behavior, observed artifacts, and unresolved limitations before drawing a verdict.