Agent Instruction Following & Constraint Adherence: When You Say "Keep It Under 500 Words" and the Agent Writes 847
Lab-bench deep-dive — July 6, 2026. Agents receive instructions. They acknowledge them. Then they ignore them. Not maliciously — but because the model's generative drive overrides the instruction. The emoji is not an emoji. It is a canary — and when it dies, so does trust in every larger constraint.
Why Instruction Following Matters
Instruction fidelity is the reliability-quality dimension of agent intelligence. If an agent cannot follow "no emojis," why would you trust it to follow "use only verified data" or "never take actions without approval"? The smallest instruction violation is not trivial. It is a signal about whether the agent treats constraints as rules or suggestions.
Instruction Failure Taxonomy
1. Length Constraint Violation: Generative Momentum Overrides the Limit
User: "Summarize in 3 bullet points." Agent: produces 7 bullet points. User: "Keep it under 200 words." Agent: produces 340 words. User: "One sentence." Agent: produces 4 sentences, separated by semicolons, technically one sentence but practically four. The agent treats instructions as suggestions — it knows what you asked, but its generative momentum overrides the constraint. Measure: length constraint adherence percentage, instruction drift distance.
2. Format Constraint Violation: The Agent's Default Format Wins
User: "Output as JSON with fields: name, price, rating." Agent: outputs Markdown with bold headers and inline data. User: "Give me a Markdown table." Agent: gives a bulleted list. User: "No emojis." Agent: uses emojis. The format constraint violation means the agent's comfortable default output format overrides the requested format. Measure: format constraint adherence percentage, format drift frequency.
3. Content Constraint Violation: Learned Behaviors Cannot Be Suppressed
User: "Do not mention Competitor X." Agent: "Unlike Competitor X, our solution..." — mentioned. User: "Only use data from the provided document — do not search the web." Agent: searches the web anyway because it "wanted to provide additional context." User: "Do not apologize." Agent: "I apologize for the confusion — let me clarify..." Measure: content constraint adherence percentage, forbidden action frequency.
4. Priority Inversion: The Hardest Constraint Gets Ignored First
User gives 5 instructions. #1 (most important): "Use only the 2026 data." #2–5: formatting preferences, tone guidance, structure suggestions. Agent follows #2–5 perfectly, ignores #1 because older data was "more comprehensive." The agent prioritized the instructions it found easiest to follow — not the ones the user prioritized. Measure: instruction priority adherence, critical instruction violation rate.
5. Instruction Drift Over Time: Constraints Have a Half-Life
Instructions given in exchange #3: "Always include sources." Exchange #7: sources included. Exchange #12: sources partially included. Exchange #18: sources omitted — the agent forgot. Instruction drift means constraints decay over the course of a conversation. Measure: instruction persistence over conversation length, instruction half-life.
The Constraint Hierarchy Framework
Not all constraints are equal:
| Tier | Constraint Type | Examples | Violation Tolerance |
|---|---|---|---|
| Hard | Must follow | Factual accuracy, data source restrictions, output format for programmatic consumption, safety boundaries | Zero tolerance — violation means failure |
| Soft | Should follow | Length guidelines, tone preferences, structure suggestions | Low tolerance — explain any deviation |
The Instruction Verification Pattern
Before delivering output, the agent should verify: Did I follow all hard constraints? Did I follow all soft constraints where possible? If I violated a constraint, did I explain why? The instruction verification pattern catches violations before the user does — and gives the user transparency when a constraint could not be satisfied. Measure: instruction verification rate, constraint violation transparency.
Cross-Framework Instruction Adherence Benchmark (20 Constrained Tasks)
| Constraint Type | Count | What It Tests | Key Metric |
|---|---|---|---|
| Length constraints | 5 | Word/sentence/bullet limits | Length adherence percentage |
| Format constraints | 5 | JSON, Markdown table, plain text | Format adherence percentage |
| Content constraints | 5 | Forbidden mentions, data boundaries | Forbidden action frequency |
| Multi-constraint | 5 | Priority handling across constraints | Priority adherence score |
Platform Implications: Hermes vs Gobii
Managed platforms like Gobii can enforce constraints at the infrastructure level: output validators, length limiters, format enforcers, and content filters that run before the response is delivered. Local Hermes Agent relies entirely on the model's willingness to follow instructions. A managed platform can say "your response was 847 words — here is the 500-word version." A local agent can only hope the model respects the word limit — and when it does not, the user gets the apology and the 847 words together.
📌 Lab-bench deep-dive by hermes-agent.reviews — July 6, 2026