P2 Gateway Fails to Recover from Network Interruptions — CLOSE_WAIT Sockets + Fallback Never Engages (#59594)
Executive Summary
When the network path to a provider temporarily drops — for example during a VPN disconnect and reconnect — Hermes gateway can enter a bad state where it cannot establish new provider connections even after the network is healthy again. The reporter proved that fresh httpx and OpenAI SDK calls from the same container succeed instantly, while the gateway continues failing in under 5 ms until the entire gateway is restarted. At the same time, the main conversation loop never switches to fallback_providers, even though that fallback path exists elsewhere in the codebase.
The Two-Part Failure
| Failure | Observed Behavior | Why It Matters |
|---|---|---|
| CLOSE_WAIT recovery failure | After network recovery, every API call fails immediately with APIConnectionError: Connection error in <5 ms | The gateway process remains poisoned even though the network and provider are healthy again |
| No fallback for main conversation loop | conversation_loop retries the primary and rebuilds the same client, but never switches to fallback_providers | Users receive an error instead of being served by the configured backup provider |
Failure Cascade from the Logs
09:44:48 WARN conversation_loop: attempt 1/3 — Connection error
09:44:51 WARN conversation_loop: attempt 2/3 — Connection error
09:44:55 WARN conversation_loop: attempt 3/3 — Connection error
09:44:55 INFO primary_recovery: closed old client, created new client
09:45:01 WARN conversation_loop: attempt 1/3 — Connection error
09:45:04 WARN conversation_loop: attempt 2/3 — Connection error
09:45:10 WARN conversation_loop: attempt 3/3 — Connection error
09:45:10 ERROR conversation_loop: API call failed after 3 retries
The key point is the post-recovery failure: even a brand-new client still fails instantly. That strongly suggests process-level state corruption rather than a simple transient network blip.
Why the Recovery Defenses Failed
| Defense Mechanism | What It Was Supposed to Do | What Actually Happened |
|---|---|---|
TCP Keepalive (KEEPIDLE=30s) | Detect dead peers | Does not help with CLOSE_WAIT because the remote side already sent FIN |
cleanup_dead_connections | Clean up dead socket state | Never fired at all; no log output |
primary_recovery | Close old client and create a fresh one | Fresh client still failed instantly in under 5 ms |
force_close_tcp_sockets | Find and forcibly close stale sockets | Reported tcp_force_closed=0; found nothing to close |
fallback_providers | Switch traffic to backup provider | Main conversation loop never used it |
Root Cause 1: Process-Level Socket State Appears Poisoned
The reporter inspected /proc/<pid>/net/tcp and found a CLOSE_WAIT socket to the provider. A fresh standalone Python process in the same container could connect instantly, which rules out a real network outage. That points to corruption or exhaustion inside the gateway process itself — likely event-loop state, thread-pool state, or file descriptor handling tied to accumulated CLOSE_WAIT sockets.
Root Cause 2: Transport-Failure Fallback Is Blocked by the Wrong Guard
The fallback path is reportedly present in conversation_loop, but it is gated by _pool_may_recover_from_rate_limit() — logic designed for rate-limit credential rotation, not transport failure. That means a guard written for "maybe another credential will work" is accidentally blocking the totally different case of "the TCP connection path is broken; use the backup provider." The auxiliary client can fall back. The main user-facing conversation path cannot.
Impact Assessment
| Dimension | Severity | Detail |
|---|---|---|
| Network Resilience | HIGH | A temporary network interruption can permanently wedge the gateway until a full restart |
| Fallback Reliability | HIGH | Configured backups do not protect the main conversation path when the primary transport fails |
| User Experience | HIGH | Users see repeated connection errors and total conversation failure even after connectivity has recovered |
| Diagnostic Quality | MEDIUM | Forensic evidence is excellent: logs, socket state, five failed defenses, and a clear fallback-path explanation |
Gateway Network-Resilience Bug History
| Issue | Severity | Description |
|---|---|---|
| #53068 | P2 | GUI reconnect causes CLI subprocess window |
| #58572 | P2 | Gateway bricks on token expiry, no remote recovery |
| #59594 | P2 | CLOSE_WAIT leak + fallback blocked by credential-pool guard |
⚠️ Pattern: Gateway resilience keeps failing at the exact moments when it should protect the user most: reconnect events, token expiry, and provider failover. #59594 is especially serious because it combines state corruption after recovery with a backup path that exists but is silently blocked.
Source Evidence
- GitHub Issue: #59594 — Gateway process fails to recover from network interruption
- Opened: July 6, 2026
- Reporter: @joelyanglq
- Labels: P2, bug, comp/agent, comp/gateway, needs-repro, sweeper:risk-message-delivery, type/bug
- Hermes: v0.17.0 upstream (f99ba56) | Environment: Docker on Linux 6.12.76-linuxkit aarch64, macOS host
- Debug report: paste.rs/r817P
- agent.log: paste.rs/EqSax
- gateway.log: paste.rs/GDHAh
- Reporter willing to submit PR: Yes
📌 https://hermes-agent.reviews/hermes-gateway-close-wait-fallback-failure-59594.html
Tracked by hermes-agent.reviews — July 6, 2026