RELIABILITY TRACKER · P3 OPEN
Home Assistant Gateway Silent Reconnect Failure · Issue #67470
A source-linked review of a reported Home Assistant gateway failure where transient network loss can leave the adapter active-looking but unable to deliver events after reconnect attempts stall.
What the report describes
The reported failure starts after a transient Home Assistant network interruption. The WebSocket reconnect ladder can stall while the Hermes gateway process remains alive, so systemd Restart=always does not restart it. The report also describes leaked aiohttp ClientSession objects on failed connects, unbounded cleanup awaits that can wedge reconnect, and no watchdog for a dead listener. A fresh process can reach Home Assistant while the existing adapter stays silent.
Scope note: This page summarizes the linked open issue. It is not an independent reproduction, and the P3 label should not be read as a guarantee of low operational impact for event-driven Home Assistant workflows.
Failure-chain matrix
| Signal | What to inspect | Safe review response |
|---|---|---|
| Trigger | Transient Home Assistant or network failure interrupts the WebSocket session | Reproduce with a controlled disconnect and retain timestamps for disconnect, retry, and listener recovery. |
| Silent failure mode | Gateway process remains active even though event delivery has stopped | Monitor delivery or listener liveness, not only process existence and systemd health. |
| Reconnect risk | Cleanup awaits can remain unbounded and prevent the reconnect ladder from completing | Bound cleanup and reconnect waits, then record an explicit terminal or retry state. |
| Resource hygiene | Failed connection attempts may leak aiohttp ClientSession objects | Close failed-session resources deterministically and track session counts during repeated retries. |
| Recovery gap | systemd Restart=always does not fire when the process never exits | Add a watchdog or health signal that reflects event-delivery liveness and can trigger recovery. |
Reproduction and monitoring protocol
- Record the issue as an open P3 reliability signal with a potentially higher operational impact when Home Assistant delivery is business-critical.
- Test transient disconnect, stalled cleanup, repeated reconnect, and fresh-process recovery separately.
- Assert both process liveness and event-delivery liveness; a running PID is not sufficient evidence of a healthy adapter.
- Measure client-session creation and closure across retries to detect resource leakage.
- Document manual recovery steps and the evidence required before treating the incident as resolved.
Source evidence
NousResearch/hermes-agent · Issue #67470
Reported labels include P3, comp/plugins, needs-repro, sweeper:risk-message-delivery, and type/bug. Verify current issue state, logs, and remediation before making a release or deployment decision.