fix(webui): crash visibility — faulthandler + thread excepthook + exit audit (#4633)
server.py exited SILENTLY after 9-16h: no traceback, no shutdown-audit line,
no core dump — the log just stopped mid-request. It runs a ThreadingHTTPServer
with daemon_threads=True, so an unhandled exception in a request/SSE/long-poll
handler thread could terminate work with nothing recorded, and faulthandler was
not enabled so a native crash left nothing at all. The WebUI also configures no
logging handlers, so INFO/ERROR records are dropped by logging's lastResort
filter (WARNING+ only) — meaning even the existing shutdown audit never reached
the log.
Add api/crash_visibility.py (stdlib-only, hooks never raise) and wire
install_crash_visibility() into server.main() before any heavy startup:
* faulthandler.enable(all_threads=True) — native crash dumps a C-level
traceback; SIGUSR1 registered for on-demand hang diagnosis.
* threading.excepthook — logs uncaught daemon/handler-thread exceptions
(thread name, ident, traceback) instead of losing them silently.
* sys.excepthook — logs uncaught main-thread exceptions (KeyboardInterrupt
preserved).
* atexit exit-audit breadcrumb — a clean/unwound exit is recorded; its
absence narrows a silent death to an un-unwound kill (OOM/SIGKILL/abort).
All diagnostics are written directly to the fault stream (stderr, which the
bootstrap redirects into the WebUI log) AND mirrored through logging, so the
line is guaranteed to land regardless of logging config.
No request-handling behavior change, no new deps. Paired memory root-cause: #4765.
Fixes #4633.
This commit is contained in:
1 parent
412cf01ac2
commit
5bca5db900
3 files changed
+559
No files matched your search
@@ -112,6 +112,7 @@ from api.profiles import set_request_profile, clear_request_profile
|
||||
from api.routes import handle_delete, handle_get, handle_patch, handle_post, handle_put
|
||||
from api.startup import auto_install_agent_deps, fix_credential_permissions
|
||||
from api.updates import WEBUI_VERSION
|
||||
from api.crash_visibility import install_crash_visibility
|
||||
|
||||
|
||||
class QuietHTTPServer(ThreadingHTTPServer):
|
||||
@@ -546,6 +547,12 @@ def _abort_if_already_serving(host: str, port: int) -> None:
|
||||
def main() -> None:
|
||||
from api.config import print_startup_config, verify_hermes_imports, _HERMES_FOUND
|
||||
|
||||
# Crash visibility FIRST (issue #4633): enable faulthandler + excepthooks +
|
||||
# exit audit before any heavy startup work so a native crash or a daemon /
|
||||
# handler-thread exception during startup or serving produces a diagnostic
|
||||
# instead of a silent death. The paired memory root-cause is #4765.
|
||||
install_crash_visibility()
|
||||
|
||||
print_startup_config()
|
||||
|
||||
fd_limit = _raise_fd_soft_limit()
|
||||
|
||||
Reference in new issue
Block a user