Adversarial review of the previous commit found a leak it introduced.
`attach_terminal()` subscribes and only then are the response headers written.
A client that dropped in that instant makes `end_headers()` raise
BrokenPipeError/ConnectionResetError, and because the `try/finally` started
*after* those writes, the exception escaped `_handle_terminal_output` with the
queue still in `_subscribers`. That pins `unwatched_since` at None — only
`unsubscribe()` on an empty subscriber list resets it — so the terminal became
permanently unreapable: the exact fd/thread leak (#4633) this reaper exists to
prevent, and strictly worse than the TOCTOU the previous commit fixed, since
that at least still allowed cleanup.
The try now starts immediately after the attach, so every exit path
unsubscribes. Regression test drives a handler whose `end_headers()` raises and
asserts the subscriber list is empty, `unwatched_since` is restored, and the
terminal is reapable again once the grace elapses. It fails against the
previous ordering.
`_terminals_to_reap()` snapshots victims under `_LOCK` and then releases it,
but `subscribe()` clears `unwatched_since` and appends to `_subscribers` under
the terminal's own `_sub_lock`, which the selection pass never held. A viewer
reconnecting in that gap left object identity — all `close_terminal(expected=…)`
ever checked — still true, so the reaper killed a terminal that now had a live
viewer.
- `_claim_reap_victim()` re-establishes the *whole* selection predicate while
holding `_LOCK` and the terminal's `_sub_lock`, and removes the registry
entry in that same critical section: same object, no subscribers, and
`unwatched_since` still set and still expired. A dead process is still
reaped unconditionally — it cannot come back, and a viewer watching a corpse
must not pin the entry forever.
- `attach_terminal()` does the registry lookup and the subscribe atomically
under `_LOCK`, replacing `get_terminal()` + `term.subscribe()` in the SSE
route. It takes the same two locks in the same order as the claim, so
exactly one side wins: the viewer attaches (and the reap is refused), or the
entry is already gone (and the route answers "terminal not running" before
committing a 200, instead of streaming from a corpse). Registration is the
authority — a registered-but-`closed` terminal still attaches so the viewer
receives `terminal_closed`.
- Teardown moved to `_teardown_terminal()` and runs after both locks are
released; `killpg`+`wait` can block ~2.5s and must not pin the registry.
- Dropped the duplicated `self.last_activity = time.time()` the rebase left in
`put_output()`.
Tests: a barrier regression that pauses the reaper after selection, attaches a
viewer through the production path, resumes, and asserts the terminal stays
registered/live and still receives output; the inverse where the reap claims
first and the attach reports not-running; refreshed-grace and replaced-entry
claims; dead-with-viewer; and a probe that `_LOCK` is free during teardown.
Three of them fail against an identity-only claim.
The fd-leak fix retires a terminal whose shell EXITS, and _MAX_TERMINALS bounds
the worst case, but an interactive shell abandoned by its client (tab closed
without POSTing /api/terminal/close, browser crash, network drop) keeps running
forever — there is deliberately no PDEATHSIG. A background reaper now closes any
terminal that has had zero attached output streams for longer than a 15-minute
idle grace, plus any dead-process terminal as a belt-and-suspenders for the
reader-loop retire.
Attachment is tracked on the subscriber model from the output-broadcast change:
subscribe() clears unwatched_since, the last unsubscribe() sets it. A terminal
is born unwatched, so a spawn nobody ever attaches to is reaped too; the grace
spans a tab refresh / brief drop so a real reconnect keeps the session. Eviction
uses close_terminal(sid, expected=term) so a restart replacement is never torn
down. The reaper thread is a lazily-started daemon singleton like the spawn
supervisor.
Stacked on the terminal fd-leak + output-broadcast changes (same file).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ai-ag2026