Agent Speed & Latency: Benchmarks That Actually Matter
Published 2026-06-20 — Hermes Agent Lab, hermes-agent.reviews
🔍 Why We Ran This Benchmark
“My agent responds in 2 seconds” is the demo. “My agent responds in 2 seconds for simple tasks but 45 seconds for complex ones, and there’s no progress indicator during those 45 seconds so users assume it’s broken and re-submit the request, doubling the queue” is the production latency reality. Token speed is the vanity metric. User-perceived latency is what determines whether an agent feels responsive or broken. This benchmark measures the full latency stack across frameworks.
📈 Agent Speed Scorecard
| Latency Metric | Gobii Managed | Hermes Agent | Δ |
|---|---|---|---|
| TTFT (Time-to-First-Token) P50 | 0.18s | 0.42s | 2.3× faster |
| TTFT P95 | 0.34s | 1.87s | 5.5× faster |
| TTFA (Time-to-First-Action) P50 | 0.31s | 2.84s | 9.2× faster |
| TTFA P95 | 0.67s | 11.23s | 16.8× faster |
| TTC (Time-to-Completion) P50 | 3.82s | 18.41s | 4.8× faster |
| TTC P95 | 8.14s | 47.62s | 5.8× faster |
| TTV (Time-to-Value) P50 | 1.12s | 6.38s | 5.7× faster |
| Latency Consistency (P95/P50 ratio) | 2.13 | 11.34 | 5.3× more consistent |
| Streaming Quality Impact | +2.1% errors | +18.7% errors | 8.9× fewer streaming errors |
| Speed Index | 94.7 | 38.2 | — |
⏲ Latency Layer Taxonomy
Five Layers of Agent Latency
| Layer | Definition | Target | Impact When Exceeded |
|---|---|---|---|
| TTFT: Time-to-First-Token | User hits send → first character appears | <500ms | >2s: user questions if it’s working. >5s: user assumes broken |
| TTFA: Time-to-First-Action | User message → agent does something (tool call, clarifying question, acknowledgement) | <1s | >5s: user’s mental model shifts from “thinking” to “stuck.” If agent thinks 10s before acting, user re-submits |
| TTC: Time-to-Completion | Total time from request to finished response | <5s (simple), <30s (medium) | >30s without progress indicator: user abandons. Long tasks need visible progress |
| TTV: Time-to-Value | When does the user get something useful? Intermediate result at 10s while full task takes 60s | <2s (first useful output) | User perception is driven by TTV, not TTC. A 60s task with a 10s intermediate result feels faster than a 10s task with a spinning wheel |
| Perceived Latency | UX layer: streaming output, progress indicators, status messages. “I’m working on it” beats silence | Continuous visible progress | A 30s task with visible progress feels faster than a 10s task with nothing. Silence is the enemy |
🔌 Latency Decomposition
Where the Time Goes: 50-Task Average
| Component | Gobii Managed | Hermes Agent | Notes |
|---|---|---|---|
| Model Inference | 1.42s (37%) | 7.83s (43%) | Prompt processing (input tokens) adds 0.3-1.2s. Hermes’ long system prompts inflate this |
| Tool Execution | 1.18s (31%) | 4.92s (27%) | API call latency, tool processing, output parsing. Search tools dominate: 2-4s per call |
| Orchestration Overhead | 0.48s (13%) | 2.76s (15%) | Framework routing, state management, context assembly, prompt construction. Gobii’s managed runtime eliminates 1-3s of overhead |
| Queue / Waiting | 0.22s (6%) | 1.84s (10%) | Queued behind other tasks, rate-limited APIs, model availability. Not visible in single-agent benchmarks but dominates production |
| Network / I/O | 0.52s (14%) | 1.06s (6%) | Gobii’s edge infrastructure reduces network latency vs local Hermes hitting remote APIs |
⚡ Streaming vs Batch: The Quality Tradeoff
Streaming tokens appear as generated — but at what cost?
| Metric | Batch (Gobii) | Streaming (Hermes) |
|---|---|---|
| Perceived Latency (user feel) | Moderate (waits for completion) | Low (tokens appear instantly) |
| Tool Call Correctness | 94.1% (can revise before sending) | 76.3% (commits early, can’t revise) |
| Structured Output Validity | 99.2% (JSON complete before parsing) | 82.7% (partial JSON parsing errors) |
| Partial Output Errors | 0.8% | 17.3% |
| Reasoning Chain Completeness | 97.8% (full chain before action) | 71.4% (premature action before full reasoning) |
| User Abandonment Rate | 4.2% | 2.1% (streaming keeps users engaged) |
⚠️ Tradeoff: Streaming reduces perceived latency and user abandonment but increases error rate by 8.9×. Gobii’s batch-then-stream approach (complete reasoning, then stream output) achieves the best of both: low perceived latency with high correctness.
🕛 The Slow Tool Problem
When a tool takes 30 seconds, does your agent handle it gracefully?
| Capability | Gobii Managed | Hermes Agent |
|---|---|---|
| Progress Updates During Wait | ✅ “Database query running — usually 20-30s for this dataset” | ❌ Silent wait, no status |
| Parallel Subtask Execution | ✅ Works on other subtasks while waiting | ❌ Blocks on tool call |
| Timeout Handling | ✅ Configurable per-tool timeout with fallback | ⚠️ Default timeout, often hangs |
| Large File Streaming | ✅ Streams large files without loading into memory | ❌ Loads entire file, OOM on large inputs |
| User Re-Submit Protection | ✅ Detects duplicate requests, shows progress | ❌ Each re-submit spawns new task, doubling queue |
📊 Latency Consistency: The P99 Problem
Users remember the slowest experience, not the average
💼 The Speed Psychology: Why This Matters
“Token speed” is the benchmark everyone shows. “200 tokens/second” sounds impressive. But users don’t experience tokens — they experience waiting. The metrics that actually drive satisfaction are:
- ⏰ TTFT <500ms: The “is it working?” moment. If the first character appears in under half a second, the agent feels instantaneous. At 2 seconds, users start questioning. At 5 seconds, they assume it’s broken.
- 🚀 TTFA <1s: The agent must act quickly. If it thinks for 10 seconds before doing anything, the user’s mental model shifts from “thinking” to “stuck.”
- 🎯 TTV <2s: First useful output within 2 seconds. Users don’t need the complete answer immediately — they need something that shows progress.
- 📊 P95/P50 ratio <3: Consistency matters more than speed. An agent that’s sometimes fast and sometimes slow is worse than one that’s consistently moderate. Users remember the worst experience.
⚠️ Methodology Rationale
These benchmarks were conducted on identical hardware (RTX 4090, 64GB RAM, Ryzen 7950X) running 50 standardized tasks spanning code generation, data analysis, research, and multi-step workflows. Each task was run 3 times; values are the median of 3 runs. Streaming benchmarks use a 10ms token arrival simulation. Latency decomposition uses wall-clock instrumentation at each layer. The Speed Index is a composite of TTFT, TTFA, TTC, TTV, and consistency (P95/P50 ratio) weighted equally.