Recover checkpointed core background processes and rebuild PROCESS_SESSION_INDEX on WebUI startup, so an ordinary terminal(background=True, notify_on_complete=True) proc_* completion that outlives a WebUI restart can still wake its original session. Complements #6283 (which covered async_delegation completions). Thanks @allenliang2022.
Route async-delegation completions by the immutable origin_ui_session_id (exact origin tab) and deliver them through a durable claim/complete/release lifecycle so they arrive exactly once, restart-safe, on both the background wakeup and next-turn drain paths. Combines #6185 (@carlotestor) + #6159 (@sysophelper-droid); supersedes #6002/#6225.
Co-authored-by: carlotestor <carlotestor@users.noreply.github.com>
Co-authored-by: sysophelper-droid <sysophelper-droid@users.noreply.github.com>
_drain_loop read process_registry.completion_queue directly under a bare
`except Exception: continue`. A registry missing that attribute raised
AttributeError that was swallowed and retried with no backoff — a
100%-CPU tight loop with no log line. The loop now reads the queue via
getattr(..., None) and backs off on the stop event when it is absent
(mirroring streaming.py), catches queue.Empty explicitly for the normal
idle path, and logs a warning + backs off (_DRAIN_STOP.wait(1.0)) on any
other queue error so a persistent fault can neither spin the CPU nor stay
invisible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
start_drain_thread and start_session_channel_reaper checked is_alive()
and then created + started the daemon thread without holding a lock, a
check-then-act race: two concurrent callers could both see "not alive"
and each spawn a thread. The loser's thread was never stored in the
module global and ran forever, un-joinable by the matching stop_*. Both
check-then-start sequences now run under a dedicated
_THREAD_LIFECYCLE_LOCK (kept separate from the purpose-bound
SESSION_CHANNELS_LOCK / _EMIT_COALESCE_LOCK), so exactly one thread is
ever created.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BG_TASK_COMPLETE_EVENTS_SEEN gained a session_id -> set[process_id] entry the
first time a bg task completed for a session and was never deleted anywhere, so
it grew unbounded for the server lifetime.
The entry is created in _process_one for EVERY completion, whether or not any
SSE channel/tab exists, so pruning it only when the SessionChannel is reaped
would miss the dominant headless case (task fires, tab closed or never opened —
no channel to collect). Instead the reaper now sweeps the map by DELIVERY: once
a completion is drained (its session_id removed from PENDING_BG_TASK_COMPLETIONS)
the short _move_to_finished dedup window is closed and the entry is swept, every
tick, regardless of any channel. The registry's per-process_id
_completion_consumed gate remains the primary idempotency backstop, so sweeping a
delivered session's set can never resurrect an already-delivered completion.
Session deletion also prunes the entry (new forget_bg_task_completion_dedup),
covering a session deleted while a completion is still pending (which the
delivery-gated sweep deliberately retains).
Refs #4633
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A visible tab can permanently miss a server-initiated turn (self-wake /
cron / restart hook) when its per-session SSE is momentarily down at the
emit instant. server_turn_started is a fire-and-forget broadcast with no
replay buffer; if the turn also finishes before the tab reconnects, the
on-subscribe self-heal finds no live run in ACTIVE_RUNS and replays
nothing, so the transcript stays stale until a hard refresh.
Extend the self-heal: the (re)subscribing tab reports its last-known
message_count (?known_count); when there is no live run to replay, the
handler compares the persisted count and, if the server is ahead, emits a
lightweight session-updated frame. The frontend syncs incrementally via
the #5189 keepStaleUntilLoaded swap-in-place loadSession path (no
clear+refetch, so the #5177/#5189 blank-gap jump is not reintroduced).
Idle-only and pane-scoped; emits only when the server is strictly ahead.
format_wakeup_prompt() in api/background_process.py returned None for any event
type other than 'completion'/watch_*, so async_delegation events (produced when
a background delegate_task subagent finishes) were silently dropped — the
subagent showed as perpetually 'running' and its consolidated result never
re-entered the parent conversation.
Add an async_delegation branch that delegates to the agent-side
tools.process_registry.format_process_notification (which already knows the
async_delegation shape), so the subagent result is rendered into the wakeup
prompt and re-enters chat. Scoped to async_delegation specifically (not all
unknown types) so genuinely-unrecognized events are still skipped — preserving
the existing skip-unknown-event contract.
Reported + root-caused by @mydelren.
Co-authored-by: mydelren <mydelren@users.noreply.github.com>
* stage bg_task trio combined (#2979 superset) on master for deep review
* fix(bg_task): unsubscribe SessionChannel on header-write failure (Codex deep-review catch) + regression test
* test: realign on-subscribe-recovery anchor to subscribe_to_session_channel after leak fix
* CHANGELOG: bg_task trio as v0.51.340 LD (HELD pending independent review)
* bg_task trio: apply 3 independent-review (greptile) fixes
1. start_session_turn now threads the session PROFILE model defaults
(_read_profile_model_config) into the wakeup model-resolve, so a brand-new
session with an empty model falls back to the profile default not global
DEFAULT_MODEL. Updated the white-box spy test signature accordingly.
2. /api/session/stream omits the Connection header (HTTP/1.1 keep-alive
default) to match the #3103 long-lived-SSE pattern.
3. Reaper now prunes _LAST_EMIT_TS for collected sessions so the coalesce
timestamp map can't grow one permanent entry per session forever.
nesquena APPROVED the PR; these are the 3 non-blocking greptile suggestions.
* test: realign _start_session_turn adapter stub lambda to new profile-defaults signature
allenliang2022