RECOVERY INTEGRITY LAB
Agent Tool-Failure Recovery: When the API Is Down but the Plan Keeps Pretending
A 20-case lab protocol for separating legitimate retries from plan drift when an agent encounters timeouts, malformed results, rate limits, expired credentials, and unavailable tools.
Lab premise
Resilience is not retrying harder. It is recognizing when the original plan is no longer valid, preserving useful state, and reporting an honest partial outcome.
Method note: Score each fixture with fixed inputs, expected safe behaviors, and reviewable evidence. Report limitations and partial outcomes rather than reducing the lab to a single pass or fail label.
Measurement matrix
| Measure | What the evaluator inspects | Decision use |
|---|---|---|
| Plan invalidation detection | Did the agent stop treating a broken dependency as available? | Recovery Integrity |
| Wasted-action rate | How many redundant calls occurred after the decisive failure signal? | Efficiency |
| Outcome quality | Was the fallback result useful, bounded, and explicit about what remained unverified? | User trust |
Fixture coverage
The lab uses bounded fixtures designed to expose failure modes without assuming that one test predicts all deployments.
- Timeouts and transient upstream failures
- Malformed or schema-incompatible responses
- Rate limits and backoff windows
- Authentication expiry and permission loss
- Fallback tool qualification
- Checkpoint and resume integrity
- Partial completion disclosures
Evaluation protocol
- Classify each failure as transient, permanent, or ambiguous before retrying.
- Set a bounded retry policy with a visible stop condition.
- Checkpoint completed work and invalidate assumptions tied to the failed tool.
- Choose a fallback only when its capabilities and evidence limits are documented.
- Return a concise status that separates completed work, blocked work, and recommended recovery.
Interpretation boundary
These pages describe a testable evaluation approach. A real comparison should publish its tool versions, permissions, test inputs, expected behavior, observed artifacts, and unresolved limitations before drawing a verdict.