A gateway-backed turn on the runs API executes in the Hermes Gateway, but the
run id only lived in process memory, so a WebUI restart orphaned the turn and
stale-pending repair appended 'Response interrupted' while the run finished
on the gateway.
- persist session.gateway_run {run_id, stream_id} as soon as POST /v1/runs returns
- send Idempotency-Key webui-<stream_id> so the gateway keeps a durable run record
- on startup (after bind, before serving) re-register the stream and follow
GET /v1/runs/{id} until terminal; write back the real output, surface a
parked approval once, keep Stop working, clear the record on every settlement
Gate fixes on top of @rodrigogs's #6658, both independently reproduced first.
do_GET called bare check_auth while every write path used check_auth_or_close,
so a body-bearing GET failing top-level auth left its bytes queued on a
keep-alive socket. check_auth_or_close is already body-pending-aware, so the
bodyless login-redirect flow keeps its keep-alive. Removing the now-unused
check_auth import (ruff F401) broke 9 tests that monkeypatch that symbol
through a module alias; rewritten to the new name.
_declared_content_lengths ran isdigit() across the whole raw header value, so
a comma-combined 'Content-Length: 0, 0' -- the same bodyless request as two
repeated headers under RFC 9110 5.3 -- read as unreadable and over-closed a
healthy connection. Now splits on commas before validating each member;
disagreeing lists still fail closed.
Verified: 16/16 adversarial CL vectors still fail closed (no fail-open
reopened), 13/13 over-close cases fixed, live-socket drive shows close armed
on a body-bearing reject and keep-alive preserved on a bodyless one, and a
chunked request behaves byte-identically to master (master never supported
chunked either, so no proxy deployment regresses).
Co-authored-by: rodrigogs <rodrigogs@users.noreply.github.com>
The first cut of the close advertisement had two defects, both caught by CI
(9 red jobs) and by the deep-review gate's own MUST-FIX 1 wording.
1. Dedup was keyed off a private `_close_advertised` flag that end_headers()
set itself. Every SSE endpoint already sends its own `Connection: close`,
and BaseHTTPRequestHandler.send_header flips close_connection when it does
— so end_headers() appended a SECOND copy to every stream response.
Dedup now derives from the pending header buffer, which sees those
site-level headers too.
2. `self.close_connection` was read unguarded. The attribute only exists once
a request line has been parsed, so the many suite handlers built with
`Handler.__new__(Handler)` raised AttributeError instead of emitting
headers — and in _handle_write that AttributeError became a spurious 500.
Both reads are now defensive, and check_auth_or_close() no longer invents
the attribute on a handler that never had it (which would have left a bogus
keep-alive verdict behind).
The mechanism moves to api/helpers.advertise_connection_close(), which also
puts server.py back under the original 750-line guard, so the test's ceiling
bump to 760 is reverted rather than shipped.
Deep-review gate (2026-08-14) findings, all reproduced:
1. close_connection was never serialized: BaseHTTPRequestHandler does not
emit Connection: close from the flag, so pooled clients kept reusing the
socket and died with BrokenPipeError. Handler.end_headers() now emits
the header (deduped) whenever close_connection is armed.
2. check_auth_or_close armed the flag too late (after the 401/302 was
already flushed); it now arms before check_auth() and restores the
prior flag on success.
3. The sidecar provenance rejection closed body-less GETs too; the close
is now gated on read_request_body=True.
4. Remaining reject-before-read sites now close-and-advertise through the
same mechanism: rate-limited CSP reports (204), the deprecated
/api/process-complete-ack (410), /api/health/restart (all outcomes),
and the four upload handlers (413 + pre-read Content-Length/boundary
validation via _reject_before_read).
Tests: pooled-client regression against the production handler (reject
with a body, then follow-up on the same pooled connection succeeds and the
rejection carries Connection: close); unit coverage for the sidecar GET
keep-alive, CSP rate-limit, health/restart and upload oversize paths. The
server.py line guard is bumped to 760 with rationale: the deep-review
requires the emission in Handler.end_headers(), which cannot live outside
server.py.
Move the close-on-reject decision into api.auth.check_auth_or_close so the
added lines land outside server.py, restoring it to 749 lines. Trim the
inline comment to a one-line docstring on the helper, per review feedback.
The rejected-write connection-close test now patches api.auth.check_auth
(the dependency the helper calls) instead of server.check_auth.
Own a private stdout descriptor before agent imports and serialize request log writes. Keep logging failures isolated from HTTP responses. Add successful chat/start subprocess delivery regressions and document header-latency semantics and cron ownership.
Two tests pin opposite constraints on server.py: test_sprint10 requires <750 lines,
and test_issue5543 AST-asserts a signal.signal() call whose first arg literal is
signal.SIGTERM. The loop refactor satisfied the size budget but the loop var _sig
failed the AST literal check. Register both signals explicitly in one shared
try/except: keeps the SIGTERM literal, adds SIGINT (#7078), stays at 749 lines.
The contributor's SIGINT addition (#7315) pushed server.py to 754 lines, tripping
test_sprint10::test_server_py_under_750_lines (master was 749). Fold the two
signal.signal() try/except blocks into a single loop over (SIGTERM, SIGINT) so
SIGINT graceful shutdown lands (#7078) without growing the file past the enforced
split budget. Same _request_shutdown handler, same fail-safe on non-main-thread.
ctl.sh start spawns the server from a non-interactive bash background job,
which causes the child to inherit SIGINT=SIG_IGN. The Stop server buttonsends SIGINT (os.kill(pid, signal.SIGINT)), which is silently ignored.
Register _request_shutdown for SIGINT in addition to SIGTERM so the
shutdown button works for ctl.sh-managed daemons.
* feat(#3351): add trusted-header auth core
* fix(#3351): reconcile trusted sessions per request
* fix(#3351): preserve valid trusted group mappings
* fix(#3351): defer trusted logout reconciliation until after csrf
* fix(auth): clear _pending_set_cookies in reset_trusted_auth_request_state (keep-alive cookie cross-request leak)
Codex gate CORE finding: a Set-Cookie queued in request N but not flushed
survived onto the reused HTTP/1.1 handler and could be emitted by request N+1
— after trusted-identity rotation on logout this could overwrite a valid login
cookie and 401 the user. Reset the queue at the per-request boundary + add a
keep-alive regression test.
* docs(.env.example): document trusted-header auth / reverse-proxy SSO env vars
HERMES_WEBUI_TRUSTED_AUTH_HEADER + TRUSTED_PROXY_CIDRS (with exact-proxy-IP
security warning) + optional groups-header/JSON group-profile-map/logout-url.
* Release exp-v0.52.43: trusted-header auth / reverse-proxy SSO (#5568, #3351) + keep-alive cookie fix + docs
---------
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Phase 1 low-risk backend batch (on top of Batch A). Both byte-verified vs PR head,
Codex SAFE, 64 own + 749 regression tests pass, coexists with Batch A's #5833.
- #5828 (ai-ag2026): frame CORS OPTIONS preflight with Content-Length: 0 (keep-alive hang fix)
- #5834 (ai-ag2026): parse session _index.json from bytes not decoded str (all 10 sites)
Co-authored-by: t <a@b>
Co-authored-by: ai-ag2026 <ai-ag2026@users.noreply.github.com>
The initial fix added ~16 lines to server.py, tripping test_server_py_under_750_lines
(server.py must stay a thin dispatcher; logic belongs in api/). Moved the whole
preflight-response into apply_cors_preflight_headers(handler) in api/routes.py;
server.py's do_OPTIONS is now a 4-line dispatcher. server.py 764 -> 746 lines.
Same security behavior (same-origin/allowlist echo, Vary: Origin, no wildcard,
header-less denial); test rewritten to the header-emitting API + added Vary and
no-header-on-denial assertions and a routes.py wildcard regression guard.
do_OPTIONS answered every CORS preflight with Access-Control-Allow-Origin: *,
advertising broader cross-origin access than a real request is granted — on a
password-less deployment that lets any site read authenticated responses. It now
echoes the request Origin only when _check_same_origin_browser_request approves
it (the exact same-origin / HERMES_WEBUI_ALLOWED_ORIGINS policy the CSRF gate
enforces for real requests), adds Vary: Origin, and never emits *. A disallowed
origin gets a 200 with no CORS headers (browser treats as preflight denial).
Default deployments (no allowlist) are unaffected.
Contributor stage; gate-pass. Co-authored-by: mo7al876any <mo7al876any@users.noreply.github.com>
Move the 1-5s commit_session_memory() extraction off the POST /api/session/new
request thread (was blocking + New Chat) into a fire-and-forget daemon thread,
guarded by the existing per-session in-flight serialization.
Data-loss prevention on managed stop:
- server.py installs a SIGTERM handler that requests an orderly stop so
serve_forever()'s finally block runs drain_all_on_shutdown() (the default
SIGTERM disposition terminated without unwinding, losing in-flight commits).
- session_lifecycle: bounded self-unregistering background-commit registry;
a _draining flag so no NEW worker starts once the drain begins (the inline
generation-drain commits any late arrival instead); a 30s overall drain
deadline so a stuck worker cannot hang the stop path forever.
Co-authored-by: luperrypf <luperrypf@users.noreply.github.com>
server.py exited SILENTLY after 9-16h: no traceback, no shutdown-audit line,
no core dump — the log just stopped mid-request. It runs a ThreadingHTTPServer
with daemon_threads=True, so an unhandled exception in a request/SSE/long-poll
handler thread could terminate work with nothing recorded, and faulthandler was
not enabled so a native crash left nothing at all. The WebUI also configures no
logging handlers, so INFO/ERROR records are dropped by logging's lastResort
filter (WARNING+ only) — meaning even the existing shutdown audit never reached
the log.
Add api/crash_visibility.py (stdlib-only, hooks never raise) and wire
install_crash_visibility() into server.main() before any heavy startup:
* faulthandler.enable(all_threads=True) — native crash dumps a C-level
traceback; SIGUSR1 registered for on-demand hang diagnosis.
* threading.excepthook — logs uncaught daemon/handler-thread exceptions
(thread name, ident, traceback) instead of losing them silently.
* sys.excepthook — logs uncaught main-thread exceptions (KeyboardInterrupt
preserved).
* atexit exit-audit breadcrumb — a clean/unwound exit is recorded; its
absence narrows a silent death to an un-unwound kill (OOM/SIGKILL/abort).
All diagnostics are written directly to the fault stream (stderr, which the
bootstrap redirects into the WebUI log) AND mirrored through logging, so the
line is guaranteed to land regardless of logging config.
No request-handling behavior change, no new deps. Paired memory root-cause: #4765.
Fixes#4633.
Adds a configurable frame-src directive so an extension can embed a
self-hosted web app in an iframe (e.g. an 'external app tab' pinning
Grafana / Vaultwarden / a dashboard). Mirrors the existing
HERMES_WEBUI_CSP_CONNECT_EXTRA knob: space-separated http(s) origins with
optional *.subdomain wildcard + port, validated, ignored if malformed
(directive-injection / paths / ws scheme / bad ports all rejected).
- Default frame-src is 'self' only, so existing same-origin iframes are
unchanged; default-off when the env var is unset.
- frame-ancestors stays 'none' -- this only governs what THIS page may embed,
never who may embed the WebUI.
- Threaded through both the enforced and report-only CSP builders + the
server.py Handler wrapper, so both headers stay identical.
Docs in docs/EXTENSIONS.md. 8 new regression tests
(tests/test_csp_frame_src_extra.py) cover default, valid widening, enforced
parity, directive-injection rejection, path/ws/port rejection, and
connect/frame independence. Existing CSP test suite still green.
Note: full suite shows 2 pre-existing order-dependent flakes
(test_issue1574 spawn-context, test_issue3283 import-order) that pass in
isolation and are untouched by this CSP-only diff.
start_watcher() ran synchronously in main() at boot; if
_resolve_watcher_target() blocked (locked state.db, slow profile
resolution, I/O stall) the server never bound its port — a startup
brick. Run it on a daemon thread with a 5s join so boot proceeds even
if watcher init stalls; the poll loop already runs on its own daemon
thread so liveness is unchanged.
Co-authored-by: minidarkmimi <minidarkmimi@users.noreply.github.com>
* stage bg_task trio combined (#2979 superset) on master for deep review
* fix(bg_task): unsubscribe SessionChannel on header-write failure (Codex deep-review catch) + regression test
* test: realign on-subscribe-recovery anchor to subscribe_to_session_channel after leak fix
* CHANGELOG: bg_task trio as v0.51.340 LD (HELD pending independent review)
* bg_task trio: apply 3 independent-review (greptile) fixes
1. start_session_turn now threads the session PROFILE model defaults
(_read_profile_model_config) into the wakeup model-resolve, so a brand-new
session with an empty model falls back to the profile default not global
DEFAULT_MODEL. Updated the white-box spy test signature accordingly.
2. /api/session/stream omits the Connection header (HTTP/1.1 keep-alive
default) to match the #3103 long-lived-SSE pattern.
3. Reaper now prunes _LAST_EMIT_TS for collected sessions so the coalesce
timestamp map can't grow one permanent entry per session forever.
nesquena APPROVED the PR; these are the 3 non-blocking greptile suggestions.
* test: realign _start_session_turn adapter stub lambda to new profile-defaults signature
Phase-3-LOW backend security fix. #3727 (@rodboev, #1909): enforced CSP now honors the same connect-src as report-only (built from one shared template). Full suite 8205 passed, Codex SAFE (live-verified headers), Opus SHIP. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
## Release v0.51.239 — Release HG (stage-q10)
Phase 3 MEDIUM-ring **salvage** from #3407. The source PR bundled a universal reliability fix with debug scaffolding + Android-specific work; this release ships only the clean, universal nugget.
### Fixed
| Salvaged from | Author | Fix |
|---|---|---|
| #3407 | @PatrickNoFilter | `server.py` ignores `SIGPIPE` (`SIG_IGN`) at import time so a client dropping the connection mid-response (tab close mid-stream, network drop, mobile backgrounding, dropped long-poll, `/api/updates/check` timeout) can't silently `Term` the whole process. The broken write now surfaces as a catchable `BrokenPipeError`; the server keeps serving. |
### Why salvage, not merge whole
#3407 (585L, 11 commits) bundles three groups: (1) the SIGPIPE fix + a 271-line `diag_shim.py` debug module, (2) an Android-cgroup-specific `os.fork`/`setsid` restart rewrite in `updates.py`, (3) personal deploy scripts (`start-webui.sh`/`watchdog-loop.sh`, which the author notes are "user-side infra, not in the server tree"). Only the SIGPIPE fix is universal, low-risk, and ship-ready — the rest is investigation tooling for a now-solved mystery or platform-specific. The source PR is held with a detailed split explanation.
### Added safety over the source PR
The original used a bare `signal.signal(signal.SIGPIPE, ...)` which would `AttributeError` on Windows (no `SIGPIPE`). The salvaged version is `getattr`-guarded so it's a no-op on Windows, preserving the native-Windows support shipped in #1952 (HD).
### Gate results
- **Full pytest suite**: 7498 passed, 9 skipped, 3 xpassed, **0 failed**
- **ruff**: CLEAN · **browser-smoke**: CLEAN
- **Codex (regression)**: SAFE TO SHIP — verified the getattr Windows-guard, that the ignore disposition lands correctly across the `os.execv` self-restart, and that subprocess children use `restore_signals=True` so the ignore doesn't leak to git/shell/editor children.
Regression test `tests/test_issue3407_sigpipe_ignore.py` pins SIG_IGN on POSIX, no-raise import, and the getattr guard.
Co-authored-by: PatrickNoFilter <PatrickNoFilter@users.noreply.github.com>
## Release v0.51.234 — Release HB (stage-q4)
Two medium-risk backend/infra fixes. All gates green.
### Fixes
| PR | Author | Fix |
|----|--------|-----|
| #3289 | @rodboev | Refuse server startup when a live instance already serves the port (Windows/macOS silent port-sharing hazard). Live-listener probe (`GET /health`, 2s timeout) + Windows `SO_EXCLUSIVEADDRUSE` — **preserves fast restart** (POSIX keeps `allow_reuse_address=True`; a dying socket in the kernel backlog times out → startup proceeds). |
| #3486 | @dso2ng | Allow remote/SSH terminal profiles to use target-side workspace paths under `terminal.cwd` without a server-local `stat()`. Local profiles unchanged — bypass only fires for remote backends and only for paths contained within `terminal.cwd`. |
### History note on #3289
This PR was **held earlier this sweep** — its original form globally disabled `SO_REUSEADDR`, which a Codex gate flagged as breaking fast restart (TIME_WAIT bricks rebind for ~60s). The contributor reworked it along the suggested lines (live-listener probe instead of the global disable). This release ships the reworked version. Unheld → full pickup → full gate.
### Gate results
- **Full pytest suite**: 7458 passed, 8 skipped, 3 xpassed, **0 failed**
- **ruff forward gate**: CLEAN
- **browser-smoke gate**: CLEAN (real server boots fine with the new startup probe)
- **Codex (regression)**: SAFE TO SHIP (verified fast-rebind preserved + remote bypass gated on backend+containment, local validation unchanged)
- **Opus (correctness + security)**: SAFE TO SHIP (probe false-positive, `_is_within` containment, local-profile bypass all hold up; applied its one minor double-call cleanup note)
Closes#3289.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Co-authored-by: dso2ng <dso2ng@users.noreply.github.com>
- api/helpers.py: _safe_write() now logs disconnects at debug level
instead of silently passing. No more invisible errors.
- server.py: Restructure exception handlers to catch
_CLIENT_DISCONNECT_ERRORS first, then Exception. Remove the
isinstance() filter inside except Exception (LBYL anti-pattern).
The 500-response fallback now catches _CLIENT_DISCONNECT_ERRORS
separately (expected) and logs unexpected failures via
traceback.print_exc() instead of bare except Exception: pass.
- tests/test_broken_pipe_cascade.py: Add coverage for SSL/Timeout
disconnect routing and 500-response safety (both disconnect
survival and unexpected error logging).
MUST-FIX:
- tests/test_2735_open_in_vscode.py: bump expected open_in_vscode locale
counter from 10 to 11 (Turkish locale added in #2772). The bump fell
out of an in-rebase test edit but never got committed; tagging without
this would have shipped a failing test in the release commit.
SHOULD-FIX inline:
- api/updates.py: case-D drift in _select_apply_compare_ref. The original
#2855 fix used latest_tag in the past-tag predicate; the check side
uses current_tag (HEAD's nearest reachable tag) plus a 'behind == 0'
gate. They drift when HEAD is on an OLDER release tag with commits on
top AND a NEWER tag exists ('case D'): check correctly suggests
advancing to the newer tag, but apply fell through to origin/<branch>.
Mirror the check-side predicate exactly. Adds regression test
test_select_apply_compare_ref_case_d_older_tag_with_commits_and_newer_tag_exists.
- static/messages.js: post-await race guard in _restoreSettledSession.
stream_end without preceding 'done' enters the settlement path, awaits
/api/session, then sets _streamFinalized=true. If a late 'done' event
arrives during that await, it sees _streamFinalized still false and
double-runs the finalize. The guard returns early when done won the
race, avoiding double renderMessages() + double notification.
- server.py: CORS preflight Access-Control-Allow-Methods now includes PUT.
#2776 wired PUT into the router for /api/mcp/servers/{name} but didn't
update the OPTIONS response. Same-origin only in practice, but cosmetic
completeness for CORS-aware deployments.
Opus advisor verdict: all 5 risk areas reviewed, 1 MUST-FIX + 3 SHOULD-FIX
all addressed inline. Net: +69/-9, no new architecture, no behavior risk.
Add `PATCH /api/mcp/servers/{name}` endpoint that accepts `{"enabled": bool}`,
updates `mcp_servers.<name>.enabled` in config.yaml, and calls `reload_config()`.
Mirrors the existing DELETE pattern.
Also wire the previously-defined-but-unrouted `_handle_mcp_server_delete` into
`handle_delete`, and `_handle_mcp_server_update` into a new `handle_put` +
`do_PUT` in server.py — fixing a pre-existing bug where those handlers existed
but were never reachable over HTTP.
UI: add a toggle button in each MCP server row in the system settings panel
(panels.js). Clicking it calls PATCH and reloads the list. Toggle button is
styled with `.mcp-toggle-enabled` / `.mcp-toggle-disabled` CSS classes. The
`toggle_supported` flag in the list response is now `True`.
i18n: add 5 new keys (`mcp_enable_server`, `mcp_disable_server`,
`mcp_enabled_toast`, `mcp_disabled_toast`, `mcp_toggle_failed`) to all 9
non-English locales (English values as placeholder translations).
Tests: add `TestMcpToggle` class with 7 tests covering disable, enable,
404-not-found, empty name, missing field, response payload, and URL-encoded name.
Update `test_empty_config` and visibility panel assertions to reflect
`toggle_supported: True` and the new toggle button in panels.js.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Opus advisor flagged that PR #2160's CSP-report auth carve-out covered all
write methods on the path, not just POST. Currently harmless (PATCH/DELETE
fall through to CSRF 403 or routing 404), but defense-in-depth — scope the
bypass to its actual use case.
CSP report regression suite (6 tests) still passes.
Opus stage-339 review SHOULD-FIX items:
1. server.py: drop 'unsafe-eval' from CSP report-only policy.
Verified by grepping all production JS — zero matches for eval(),
new Function(), or string-form setTimeout/setInterval. Keeping it
was a gratuitous privilege.
2. server.py: add https://cdn.jsdelivr.net to script-src + style-src.
index.html loads Prism/xterm/katex from this CDN with SRI hashes —
without the allowance every page load fires known-good CSP violations
that drown out real signal once a collector is wired.
3. api/commands.py: sanitize plugin command error. Previously returned
f'Plugin command error: {exc}' which would leak paths/env from
FileNotFoundError('/etc/something/secret.key') etc. Now returns only
the exception type name; full traceback goes to server log.
Test asserts updated to match the new policy shape.
Co-authored-by: Opus advisor <opus-advisor@hermes.local>
carlotestor