_probe_remote_gateway held _remote_probe_lock only around the cache
read and cache write; the network probe itself (_http_probe per path,
each up to ~2s) ran unlocked. On a cold cache a dashboard that fans out
to N panels fired N concurrent probe sets at the (possibly dead)
gateway, each blocking up to ~6s (#5455, #2476).
Wrap the existing lock in a Condition and add an in-flight set: the
first 'leader' thread marks the base_url in-flight and runs the probe
(via extracted _run_remote_probe), while latecomers wait on the
Condition for the leader's cached result instead of probing themselves.
The leader clears the marker and notify_all()s in a finally block, so
waiters are woken even if the probe raises (no deadlock); a bounded wait
budget lets a waiter fall back to probing itself if a leader hangs.
_reset_remote_probe_cache_for_tests also clears the in-flight set.
Expire the cache from the probe's COMPLETION time, not the leader's
entry time: walking every path of a hung gateway takes
len(paths) * timeout (~6s), which exceeds the 5s TTL, so
`entry_time + TTL` would store an already-expired cache line. Waiters
woken right after would then miss the cache and each re-probe the dead
gateway — collapsing single-flight and regressing latency to worse-than-
serial. A regression test drives a walk longer than the TTL and asserts
still exactly one probe set (the shipped happy-path test kept the walk
well under the TTL and so never exercised this).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In cross-container Docker setups the agent-health check resolves the gateway's
runtime status from a shared gateway_state.json. After the keyword-form
read_runtime_status(pid_path=...) call fails against an older agent signature,
_read_gateway_runtime_status fell back to read_runtime_status(pid_path) passing
the sibling gateway.pid path positionally — so the reader parsed PID metadata as
runtime metadata and gateway_state/updated_at never resolved, breaking the
freshness check (issue #5030, #1879).
The positional fallback now passes the correct gateway_state.json sibling path
(pid_path.with_name(runtime_status_file), where runtime_status_file comes from
the agent's _RUNTIME_STATUS_FILE attr, defaulting to gateway_state.json). The
path computation is hoisted so both fallback branches use the correct file. The
keyword happy-path is unchanged for agents that accept pid_path=..., and normal
PID-based health is unaffected.
Verified: the new test test_cross_container_runtime_status_reads_sibling_runtime_file
fails on master without the fix and passes with it; #716 PID-health tests still pass.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* fix(health): probe /health/detailed first and unify gateway env vars (#3355)
* docs(changelog): v0.51.225 — remote gateway health probe gateway_state fix (#3355)
* fix(health): normalize gateway URL health-suffix + cap remote body read (#3355 Codex follow-up)
Codex review of #3355 found two issues:
(1) SILENT — a gateway env var already pointing at a health path (e.g.
GATEWAY_HEALTH_URL=http://host/health) produced doubled paths like
/health/health/detailed once probe paths were appended. Now strip a trailing
/health/detailed, /health, /v1/health, /status suffix before appending
(mirrors api/updates.py).
(2) CORE — the new resp.read() on a 2xx body was unbounded; a large/trickled
remote response could hang /api/health/agent or balloon memory. Cap the read
at _REMOTE_PROBE_BODY_LIMIT_BYTES (64KB)+1 and skip JSON parse when over cap
(still report the gateway alive, just without parsed gateway_state).
Adds regression tests for both (no doubled /health/health path; oversized body
is capped + does not hang). Also updated _FakeResp.read to accept the size arg.
---------
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Co-authored-by: nesquena-hermes <[email protected]>
_gateway_root_pid_path() unconditionally returned <hermes_root>/gateway.pid.
Profile-scoped gateways (started with --profile <name> or via active_profile)
write their runtime files under <hermes_root>/profiles/<name>/ instead of the
root, so the root-level path never existed.
build_agent_health_payload() therefore always received a non-existent pid_path,
fell through to the stale root-level gateway_state.json, and returned alive=None.
This caused the cron/scheduled-jobs page to display "Gateway not configured" even
when a gateway was actively running.
Fix: after failing to find a root-level gateway.pid, fall back to the active
profile directory via get_active_hermes_home(). Root-level wins when it exists,
so deployments that do write there are unaffected. Errors from profile lookup are
swallowed and the root path is returned, preserving the previous safe default.
Adds five focused unit tests covering the new fallback, the priority rule, and
the error-handling path.
The dashboard banner 'Hermes agent is not responding' fires on every
multi-container deployment that doesn't set 'pid: "service:hermes-agent"'
in compose, because get_running_pid() relies on fcntl.flock and
os.kill(pid, 0) — both PID-namespace-scoped and invisible across container
boundaries.
Fix: when get_running_pid() returns None, fall back to a freshness check on
gateway_state.json. The gateway already writes that file on every tick with
gateway_state == 'running' and an aware ISO-8601 updated_at timestamp, so a
recent (<= 120s) timestamp is an equivalent live-process signal that needs
only a shared volume — no PID namespace, no compose workaround, no extra
HTTP probe URL.
Behavior preserved:
- In-namespace deployments still hit the PID-based path first; payload shape
unchanged (no 'reason' key) so #716 contract holds.
- Cross-container alive path adds reason='cross_container_freshness' so
support diagnostics can tell which signal succeeded.
- Stale updated_at, non-running gateway_state, malformed/naive/missing
timestamps, and timestamps far in the future all still report 'down' — the
fallback never produces a false positive.
- Same redaction rules: argv/command/executable/env/raw pid never leak.
Tests: 15 new cases in test_issue1879_cross_container_gateway_liveness.py
covering the cross-container alive path, every refusal case, clock-skew
tolerance, and backward compat with the #716 PID path. Existing #716
heartbeat tests (8) continue to pass.
