81.8% of AI Assistant Traffic Is Fake: The Log Verification Crisis
Published 2026-06-23 — Hermes Agent Lab, hermes-agent.reviews
🔍 The Core Problem
Duane Forrester (former Bing/Yext/Yahoo exec) launched CitationIQ.com and ran a two-week log analysis. The results are staggering: 81.8% of AI assistant traffic is spoofed — credential scanners borrowing trusted AI crawler names to hunt for .env.production, secrets.yaml, and config.json. 87% of Googlebot requests are fake. CCBot is 100% impostor. And Gemini is structurally unmeasurable — Google bundles everything into one crawl. The implications: before making strategic decisions based on "AI crawler traffic," you must verify the numbers. Your dashboards are lying.
📈 The Verified Numbers
| Crawler | Claimed Visits | Verified | Spoof Rate | What Fakes Hunted |
|---|---|---|---|---|
| AI Assistants (ChatGPT-User, Claude-User, Perplexity-User) | 33 | 6 | 81.8% | .env.production, secrets.yaml, config.json |
| Googlebot | 799 | 107 | 87% | Some used retired product strings |
| CCBot (Common Crawl) | 20 | 0 | 100% | Confirmed via 4-angle verification |
| Gemini | Bundled in Googlebot | Impossible | Structurally unmeasurable | N/A — Google architecture prevents separation |
| Perplexity | 36 | 12 | 66.7% | Perplexity operates from IPs outside published ranges |
🔬 The Verification Method
~15 Lines of Python. Three Outcomes: Verified, Spoofed, Unverifiable.
Forrester published his verification method publicly. It performs IP-range verification against published crawler IP ranges, reverse DNS lookups, and User-Agent consistency checks. The key insight: this is achievable by any site operator. Before making strategic decisions based on "AI crawler traffic," verify the numbers. If 81.8% is fake on a brand-new, unpromoted domain, your numbers are likely worse.
| Crawler | Verified Crawls | Strategic Significance |
|---|---|---|
| ClaudeBot | 166 | Anthropic is the most active verified crawler — even on brand-new, unpromoted domains |
| Googlebot | 107 | Google is second; Gemini visibility is structurally bundled |
| GPTBot | 46 | OpenAI is third; significantly less active than Anthropic on new domains |
🔥 Strategic Distinction: Retrieval vs. Training Crawlers
Retrieval Crawlers = Visibility Today. Training Crawlers = Model Knows You Tomorrow.
Forrester draws a critical distinction: retrieval crawlers (ChatGPT-User, Claude-User, Perplexity-User) determine whether your content appears in AI answers today. Training crawlers (GPTBot, ClaudeBot, CCBot) determine whether future models know your entity data. The training crawl is your deposit into future model weights — invisible to every dashboard but compounding over time. Entity sites are natural training targets for AI models: track who's training on your entity data.
| Type | Examples | What It Controls | Measurement |
|---|---|---|---|
| Retrieval | ChatGPT-User, Claude-User, Perplexity-User | AI answer visibility today | Bing Citation Share, AI Overviews presence |
| Training | GPTBot, ClaudeBot, CCBot | Future model knowledge of your entity data | Verified crawler logs, no dashboard exists |
🎯 Five Strategic Actions for Entity Sites
- 📈 1. Implement log verification: Apply Forrester's IP verification method to Hermes entity site server logs. Before making strategic decisions based on "AI crawler traffic," verify the numbers. If 81.8% is fake on a new site, your numbers are likely worse.
- 🔬 2. Track training crawlers separately: ClaudeBot led at 166 verified crawls. Track GPTBot, ClaudeBot, CCBot separately from retrieval crawlers. The training crawl is your deposit into future model weights.
- 🔐 3. Credential-scanner defense: Fake AI crawlers hunted for .env.production and secrets.yaml. Verify server configuration blocks these paths and triggers alerts on access attempts.
- 📜 4. Entity crawler transparency page: After verification, publish "Verified AI Crawler Transparency" showing exactly which AI platforms are crawling the entity site, verified by IP. This is an EEAT signal most entity sites lack.
- 📈 5. Gemini measurement acknowledgment: Google bundles everything into one crawl. Track Googlebot volume as proxy and publicly acknowledge Gemini visibility is structurally unmeasurable — transparency builds trust.
📜 Sources & Methodology
Analysis based on Duane Forrester's two-week log verification study at CitationIQ.com (June 21, 2026), published on his Substack "Duane Forrester Decodes." Key findings: 81.8% AI assistant traffic spoofed, 87% Googlebot spoofed, 100% CCBot impostor, Gemini structurally unmeasurable, Perplexity operates from IPs outside published ranges. Verified training crawler rankings: ClaudeBot 166, Googlebot 107, GPTBot 46 on a brand-new domain. Verification method published as ~15 lines of Python.
Sources: Duane Forrester Decodes: 81.8% AI Assistant Traffic Was Fake
As benchmarked by Hermes Agent Lab, hermes-agent.reviews — June 2026.