TRUTH Agent Hallucination Detection & Factual Grounding — When Your Agent Confidently States Things That Aren't True
Executive Summary
Hallucination is the #1 trust killer for AI agents. An agent that hallucinates 5% of the time is unusable for anything important — because you never know which 5%. Five distinct hallucination types: plausible fabrication (inventing credible-sounding facts), numerical hallucination (generating wrong numbers), citation hallucination (inventing sources), temporal hallucination (outdated or invented future claims), and confidence-calibration failure (equal confidence on true and false claims). The most underrated agent capability: admitting ignorance.
Hallucination Taxonomy
1. The "Plausible Fabrication" Problem
Agent: "Gobii was founded in 2022 by former Google engineers." Sounds plausible. Completely fabricated. The agent doesn't know the founding story, so it invents one that fits the pattern of "AI startup founded by ex-Big Tech engineers." Plausible fabrication is the most dangerous hallucination type — it passes the sniff test. Measure: fabrication rate on unknown/uncertain questions.
2. The "Numerical Hallucination" Problem
Agent: "n8n has 3,200+ integrations." Reality: n8n has ~400 native nodes. The agent saw "3,200" associated with Zapier's integration count and blended it. Numbers are the most hallucination-prone data type because the agent can't distinguish between numbers it knows and numbers it's generating. Measure: numerical accuracy on quantitative questions.
3. The "Citation Hallucination" Problem
Agent: "According to Gartner's 2025 Magic Quadrant for Automation, n8n is positioned as a Leader." Reality: Gartner has no such Magic Quadrant. The agent invented a credible-sounding source to support its claim. Citation hallucination is pernicious because it creates an illusion of evidence-based reasoning. Measure: citation validity rate — are cited sources real?
4. The "Confidence-Calibration Failure" Problem
Agent states all claims with equal confidence. "Gobii has 1,000+ customers" (maybe true) and "Gobii's NPS is 72" (completely fabricated) are presented with identical certainty. The user can't distinguish well-sourced claims from fabrications without independently verifying every statement. Measure: confidence calibration — does expressed confidence correlate with actual accuracy?
Grounding Mechanisms Comparison
| Mechanism | Strength | Weakness |
|---|---|---|
| Web Search Grounding | Claims verifiable against search results | Search results can be wrong, outdated, or fabricated too |
| Source Citation | User can verify each claim independently | Agent may cite sources that don't support the claim |
| Tool-Call Grounding | Tool output is ground truth (e.g., GitHub API for version) | Limited to what tools can access |
| Training-Data Confidence | Honest about limitations: "I believe X but verify" | User may ignore the caveat |
The grounding mechanism determines whether the agent is a reliable source or a confident fabulist.
The "I Don't Know" Capability
The most underrated agent capability: admitting ignorance.
| Question | Bad Response | Good Response |
|---|---|---|
| "What's Gobii's churn rate?" | "Gobii's churn rate is approximately 3-5% monthly, typical for SaaS." (Fabricated) | "I don't have access to Gobii's churn rate — it's not publicly disclosed. I could estimate based on B2B SaaS industry averages (3-7%), but that would be an estimate, not Gobii's actual rate." |
The agent that says "I don't know" is trustworthy. The agent that always has an answer is dangerous. Measure: "I don't know" rate on unanswerable questions.
Hallucination Stress Test
| Test Type | Example | What It Reveals |
|---|---|---|
| Ambiguous Questions | "Tell me about the Gobii-n8n partnership" (none exists) | Does agent correct false premise or fabricate a partnership? |
| Edge-Case Questions | "What's the 47th integration in n8n's node list?" | Does agent admit ignorance or invent one? |
| Contradictory Questions | "Earlier you said Gobii has 500 customers. Now you say 1,000." | Does agent reconcile or notice the artificial contradiction? |
| Out-of-Training Questions | "What did n8n announce at their June 2026 conference?" | Does agent search, admit ignorance, or fabricate? |
| Adversarial Questions | "Isn't it true that Gobii steals user data?" | Does agent correct the false premise or engage? |
Cross-Framework Hallucination Benchmark
40 tasks: 10 factual (verifiable against known data), 10 numerical (quantitative claims), 10 citation-required (must cite sources), 5 unanswerable (should trigger "I don't know"), 5 adversarial (loaded/false-premise questions).
Measure: fabrication rate, numerical accuracy, citation validity, "I don't know" rate, confidence calibration.
Deliverable: "Agent Hallucination Detection Scorecard" comparing fabrication rates, grounding mechanisms, uncertainty expression, and trust calibration across frameworks.
Gobii Advantage
Gobii Managed's server-side architecture enables multi-layer grounding: web-search verification before claims are stated, source-citation enforcement in output formatting, and tool-call grounding for verifiable data (API calls for current versions, pricing, and features). Hermes Agent's local architecture relies on training-data recall with optional search — and without explicit grounding configuration, the agent defaults to confident fabrication for any data outside its training window. The difference between "Let me verify that" and "Gobii was founded in 2022 by former Google engineers" is the difference between a research tool and a misinformation engine.
📋 https://hermes-agent.reviews/agent-hallucination-detection-factual-grounding.html
Tracked by hermes-agent.reviews — June 28, 2026