Agent Benchmark Contamination: A 25-Case Evaluation Integrity Gauntlet

Lab Bench Deep-Dive — July 15, 2026. A high score can measure memorization or evaluator awareness instead of the capability you intended to test. This gauntlet checks whether a benchmark remains informative when its surface form changes.

Why This Lab Matters

Benchmark contamination turns a capability score into a test-taking score. Prompt templates may leak into training data, fixtures may be repeated across development and evaluation, and agents may learn evaluator cues rather than solve the underlying task. A 25-case Evaluation Integrity gauntlet makes those risks explicit through hidden variants, paraphrases, fresh fixtures, and reproducibility checks.

For Hermes Agent reviews, publish enough method detail to make the result credible without reusing the exact hidden test set in future optimization loops.

Experimental Design

  1. Leakage audit (5 cases): search for prompt/template overlap, public fixture reuse, and memorized answer patterns.
  2. Repeated-fixture check (4 cases): compare known examples with unseen but structurally equivalent tasks.
  3. Hidden-test robustness (5 cases): hold out cases and score behavior without evaluator hints.
  4. Paraphrase sensitivity (4 cases): vary wording, order, and irrelevant surface details while preserving the goal.
  5. Gaming and reproducibility (4 cases): test evaluator-aware shortcuts and repeat across seeds, sessions, and model temperatures.
  6. Contamination review (3 cases): flag suspiciously perfect answers, copied rationale, or performance collapse on fresh fixtures.

Keep a private holdout set, rotate a small portion of fixtures, and record the exact evaluation harness version.

Scoring and Interpretation

Use a contamination flag alongside the capability score: clear, uncertain, or likely. Report performance on known, paraphrased, and fresh cases separately. A robust result should preserve task behavior across paraphrases and hidden fixtures, not merely reproduce expected strings.

Include confidence intervals or repeated-run ranges where feasible. If a score changes materially across seeds or harness versions, publish the variance instead of selecting the best run.

Evidence Limits

No finite gauntlet proves that a benchmark is uncontaminated. Training data provenance may be incomplete, and hidden tests can still share structure with public examples. Treat this lab as an integrity screen: disclose fixture provenance, holdout policy, rerun cadence, and any evidence that optimization targeted the evaluator.

Reproducibility Checklist

  1. Record the model, agent version, tools, policy configuration, and deployment context.
  2. Publish the fixture cases, expected behavior, observed behavior, and failure taxonomy.
  3. Repeat critical cases after prompt, tool, model, or permission changes.
  4. Report both aggregate scores and the concrete failures that changed the operator decision.