The Cold Start Problem
Published 2026-06-19 - Hermes Agent Lab, hermes-agent.reviews
🔍 Why We Ran This Benchmark
"My agent gets better the more I use it" is the long-term promise. "My agent asked me my project ID for the 4th time this week, suggested a format I explicitly rejected yesterday, and started our first interaction with 'How can I help you?' — the same generic greeting every new user gets" is the cold-start reality that drives users away before they ever reach the "gets better" phase. Agents get better with context about the user. But every new user starts with zero context. How do agents handle the cold start?
📈 Cold Start Quality Scorecard
| Cold Start Dimension | Gobii Managed | Hermes Agent |
|---|---|---|
| First-Interaction Task Completion Rate | 91.2% | 54.3% |
| First-Interaction User Satisfaction (1-5) | 4.4 | 2.8 |
| Follow-Up Question Rate (before action) | 1.2 | 3.8 |
| Assumption Accuracy Rate | 87.3% | 41.2% |
| Learning Rate (personalized by interaction 5) | 94.1% | 38.7% |
| Cross-Session Recall (3-day gap) | 89.7% | 12.4% |
| Preference Overfit Rate | 3.2% | 28.6% |
| Context Bootstrapping Support | ✅ CRM, PM, email integration | ❌ None |
| Repeat Question Rate (after info provided) | 2.1% | 47.3% |
| Cold Start Index | 92.4 | 36.8 |
🔄 The Cold Start Dimension Taxonomy
Five Dimensions of the Zero-Context Problem
| Dimension | What It Measures | Gobii Score | Hermes Score |
|---|---|---|---|
| First-Interaction Quality | Response quality on first message with zero history | 91.2% | 54.3% |
| Preference Elicitation | Does agent proactively ask about preferences upfront? | ✅ Configurable | ❌ None |
| Learning Rate | How many interactions until responses feel personalized? | 5 interactions | 15+ interactions |
| Context Bootstrapping | Can agent pre-load context from existing systems? | ✅ Full integration | ❌ Manual only |
| "Repeat Yourself" Problem | How often does agent ask for already-provided info? | 2.1% | 47.3% |
🧠 The "Over-Personalization" Risk
Learning Fast Can Mean Learning Wrong
Agents that learn fast can overfit: after 3 interactions where the user asked for concise responses, the agent always responds concisely — even when the user's 4th request is "give me a detailed analysis." The agent learned "this user likes concise" but missed that preferences are context-dependent.
| Metric | Gobii | Hermes |
|---|---|---|
| Preference overfit rate (applies learned pref incorrectly) | 3.2% | 28.6% |
| Preference adaptation speed (update on contradictory signal) | 1.2 interactions | Never adapts |
| Context-dependent preference handling | ✅ Task-aware | ❌ Global only |
Finding: Gobii's task-aware preference system applies learned preferences contextually — concise for status updates, detailed for analyses. Hermes' global preference system can't distinguish context, leading to 28.6% overfit rate.
👥 The "New Team Member" Analogy
Best Cold-Start Agents Behave Like Skilled New Hires
| Ideal Behavior | Gobii | Hermes |
|---|---|---|
| Asks smart questions upfront | ✅ Context-aware probing | ⚠️ Generic questions |
| Demonstrates competence immediately | ✅ 91.2% first-task success | ❌ 54.3% |
| Learns fast (feels like team member by interaction 5) | ✅ 94.1% personalized by #5 | ❌ 38.7% |
| Documents what's learned (no repetition) | ✅ Persistent memory | ❌ Session-only |
| Knows when to ask vs infer | ✅ Irreversible decisions: ask | ❌ Always guesses |
The cold-start cliff: 48% of users abandon an agent after a poor first interaction. Hermes' 54.3% first-interaction success rate means nearly half of new users never return. Gobii's 91.2% first-interaction success retains users through the critical first impression.
📜 Sources & Methodology
Benchmark methodology: 50 first-time users, 50 unique tasks across four frameworks. First-interaction quality measured: task completion without follow-up, user satisfaction (1-5), clarifying question rate, assumption accuracy. Learning rate: human eval at interactions 1, 3, 5, 10, 20 on "does this response feel personalized?" Cross-session memory: 3-day gap between sessions, 10 specific recall questions. Preference overfit: test whether agent applies learned preference when context suggests otherwise. Context bootstrapping: test integration with CRM, project management, email history. All users were new to each framework, no prior training data from test users.
As benchmarked by Hermes Agent Lab, hermes-agent.reviews — June 2026.