fix(webui): crash visibility — faulthandler + thread excepthook + exit audit (#4633)

server.py exited SILENTLY after 9-16h: no traceback, no shutdown-audit line,
no core dump — the log just stopped mid-request. It runs a ThreadingHTTPServer
with daemon_threads=True, so an unhandled exception in a request/SSE/long-poll
handler thread could terminate work with nothing recorded, and faulthandler was
not enabled so a native crash left nothing at all. The WebUI also configures no
logging handlers, so INFO/ERROR records are dropped by logging's lastResort
filter (WARNING+ only) — meaning even the existing shutdown audit never reached
the log.

Add api/crash_visibility.py (stdlib-only, hooks never raise) and wire
install_crash_visibility() into server.main() before any heavy startup:
  * faulthandler.enable(all_threads=True) — native crash dumps a C-level
    traceback; SIGUSR1 registered for on-demand hang diagnosis.
  * threading.excepthook — logs uncaught daemon/handler-thread exceptions
    (thread name, ident, traceback) instead of losing them silently.
  * sys.excepthook — logs uncaught main-thread exceptions (KeyboardInterrupt
    preserved).
  * atexit exit-audit breadcrumb — a clean/unwound exit is recorded; its
    absence narrows a silent death to an un-unwound kill (OOM/SIGKILL/abort).

All diagnostics are written directly to the fault stream (stderr, which the
bootstrap redirects into the WebUI log) AND mirrored through logging, so the
line is guaranteed to land regardless of logging config.

No request-handling behavior change, no new deps. Paired memory root-cause: #4765.

Fixes #4633.
This commit is contained in:
hermes-agent committed 2026-07-01 18:01:39 +00:00
1 parent 412cf01ac2
commit 5bca5db900
3 files changed
+559

No files matched your search

+7
View File
@@ -112,6 +112,7 @@ from api.profiles import set_request_profile, clear_request_profile
from api.routes import handle_delete, handle_get, handle_patch, handle_post, handle_put
from api.startup import auto_install_agent_deps, fix_credential_permissions
from api.updates import WEBUI_VERSION
from api.crash_visibility import install_crash_visibility
class QuietHTTPServer(ThreadingHTTPServer):
@@ -546,6 +547,12 @@ def _abort_if_already_serving(host: str, port: int) -> None:
def main() -> None:
from api.config import print_startup_config, verify_hermes_imports, _HERMES_FOUND
# Crash visibility FIRST (issue #4633): enable faulthandler + excepthooks +
# exit audit before any heavy startup work so a native crash or a daemon /
# handler-thread exception during startup or serving produces a diagnostic
# instead of a silent death. The paired memory root-cause is #4765.
install_crash_visibility()
print_startup_config()
fd_limit = _raise_fd_soft_limit()