start_session_turn() answers 409 retryable=True while the Agent runtime is
stale, 409 process_wakeup_paused while wake-ups are paused, and 409 with
active_stream_id while the session is busy with another turn. The bridge
released the durable claim as a plain failure each time, so a burst of
refusals within seconds terminally dropped a completion whose origin session
was alive and waiting.
release_async_delegation_delivery(..., retryable=True) now forwards the
refund to an Agent core whose release_event_delivery() accepts the
`retryable` keyword (NousResearch/hermes-agent#120028), detected through
inspect.signature, and falls back to the historical plain release on an
older core. Hard rejections keep the bounded budget.
* fix(tests): make mtime_invalidation and glm_5_3 tests order-independent (#7100)
* fix: expose max reasoning for GPT-5.6 models
* fix(recovery): do not reattach cancelling runs
ACTIVE_RUNS tracks worker lifecycle, which is deliberately broader than
"a turn a browser may attach to". cancel_stream() keeps the row as
phase="cancelling" while the worker unwinds so a successor turn cannot
start on top of it, but the client has already reached a terminal state
for that stream: its run journal ends in a terminal event.
The recovery lookups treated every same-session ACTIVE_RUNS row as
attachable. An idle session holding a cancelling row therefore received a
recovered server_turn_started on every /api/session/stream subscription:
the client attached, replayed the terminal event, tore the renderer down,
resubscribed, and the server replayed the same frame again. The result is
an endless attach/replay loop that rebuilds the transcript repeatedly.
Separate the two meanings instead of narrowing one call site:
- api/config.py gains active_run_is_attachable() and
active_run_cancel_is_stale() as the shared predicates.
- active_stream_id_for_session() (browser recovery) returns attachable
rows only; _session_has_active_turn() (busy check) keeps counting a
fresh cancellation so a successor cannot overlap the unwinding worker.
- _live_active_stream_id() applies the same rule to the hidden-tab
status poller, on both the STREAMS and ACTIVE_RUNS paths.
- routes._cancelled_run_is_stale() now delegates to the shared predicate
rather than keeping a parallel copy of the staleness rule.
- A cancelling row past a bounded unwind window with no live STREAMS
channel is reclaimed from ACTIVE_RUNS and its stream owner released, so
a wedged worker cannot suppress background wakeups forever. Age alone
does not reclaim a row that still owns a live channel.
Staleness anchors on cancelled_at, falling back to started_at, so a
long-running turn cancelled moments ago is never treated as an orphan.
tests/test_cancelling_run_not_attachable.py covers both directions of
each rule. The six behavioral tests fail on the unpatched tree and pass
with the fix; two targeted mutations (forcing the attachability predicate
true, and disabling the staleness reaper) each turn the suite red.
* docs(rfc): document cancellation attach/admission split
Records the runtime contract introduced by the recovery fix in the WebUI
run-state consistency RFC, so the distinction is discoverable instead of
living only in code comments.
- Adds ACTIVE_RUNS to the State Layers table as the worker-lifecycle
registry, explicitly not the set of runs a browser may attach to.
- Adds invariant 9: lifecycle-busy is not client-attachable. Cancellation
splits the two meanings, recovery paths must exclude cancelling rows,
and admission checks must keep counting them.
- Documents the bounded cancellation-unwind window: reclamation needs
both age and the absence of a live STREAMS channel, and staleness is
anchored on the cancellation timestamp.
- Extends the review checklist with the admission-vs-attachment question
and the evidence required when changing a reclamation window.
* docs(changelog): note #7096 cancelling-run reattach, #7083 GPT-5.6 max reasoning, #7101 test order-independence
---------
Co-authored-by: webtecnica <webtecnica@gmail.com>
Co-authored-by: Abdulrahman Elkenany <boudy.elkenany123@gmail.com>
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
Co-authored-by: n <a@n>
Recover checkpointed core background processes and rebuild PROCESS_SESSION_INDEX on WebUI startup, so an ordinary terminal(background=True, notify_on_complete=True) proc_* completion that outlives a WebUI restart can still wake its original session. Complements #6283 (which covered async_delegation completions). Thanks @allenliang2022.
Route async-delegation completions by the immutable origin_ui_session_id (exact origin tab) and deliver them through a durable claim/complete/release lifecycle so they arrive exactly once, restart-safe, on both the background wakeup and next-turn drain paths. Combines #6185 (@carlotestor) + #6159 (@sysophelper-droid); supersedes #6002/#6225.
Co-authored-by: carlotestor <carlotestor@users.noreply.github.com>
Co-authored-by: sysophelper-droid <sysophelper-droid@users.noreply.github.com>
_drain_loop read process_registry.completion_queue directly under a bare
`except Exception: continue`. A registry missing that attribute raised
AttributeError that was swallowed and retried with no backoff — a
100%-CPU tight loop with no log line. The loop now reads the queue via
getattr(..., None) and backs off on the stop event when it is absent
(mirroring streaming.py), catches queue.Empty explicitly for the normal
idle path, and logs a warning + backs off (_DRAIN_STOP.wait(1.0)) on any
other queue error so a persistent fault can neither spin the CPU nor stay
invisible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
start_drain_thread and start_session_channel_reaper checked is_alive()
and then created + started the daemon thread without holding a lock, a
check-then-act race: two concurrent callers could both see "not alive"
and each spawn a thread. The loser's thread was never stored in the
module global and ran forever, un-joinable by the matching stop_*. Both
check-then-start sequences now run under a dedicated
_THREAD_LIFECYCLE_LOCK (kept separate from the purpose-bound
SESSION_CHANNELS_LOCK / _EMIT_COALESCE_LOCK), so exactly one thread is
ever created.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BG_TASK_COMPLETE_EVENTS_SEEN gained a session_id -> set[process_id] entry the
first time a bg task completed for a session and was never deleted anywhere, so
it grew unbounded for the server lifetime.
The entry is created in _process_one for EVERY completion, whether or not any
SSE channel/tab exists, so pruning it only when the SessionChannel is reaped
would miss the dominant headless case (task fires, tab closed or never opened —
no channel to collect). Instead the reaper now sweeps the map by DELIVERY: once
a completion is drained (its session_id removed from PENDING_BG_TASK_COMPLETIONS)
the short _move_to_finished dedup window is closed and the entry is swept, every
tick, regardless of any channel. The registry's per-process_id
_completion_consumed gate remains the primary idempotency backstop, so sweeping a
delivered session's set can never resurrect an already-delivered completion.
Session deletion also prunes the entry (new forget_bg_task_completion_dedup),
covering a session deleted while a completion is still pending (which the
delivery-gated sweep deliberately retains).
Refs #4633
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A visible tab can permanently miss a server-initiated turn (self-wake /
cron / restart hook) when its per-session SSE is momentarily down at the
emit instant. server_turn_started is a fire-and-forget broadcast with no
replay buffer; if the turn also finishes before the tab reconnects, the
on-subscribe self-heal finds no live run in ACTIVE_RUNS and replays
nothing, so the transcript stays stale until a hard refresh.
Extend the self-heal: the (re)subscribing tab reports its last-known
message_count (?known_count); when there is no live run to replay, the
handler compares the persisted count and, if the server is ahead, emits a
lightweight session-updated frame. The frontend syncs incrementally via
the #5189 keepStaleUntilLoaded swap-in-place loadSession path (no
clear+refetch, so the #5177/#5189 blank-gap jump is not reintroduced).
Idle-only and pane-scoped; emits only when the server is strictly ahead.
format_wakeup_prompt() in api/background_process.py returned None for any event
type other than 'completion'/watch_*, so async_delegation events (produced when
a background delegate_task subagent finishes) were silently dropped — the
subagent showed as perpetually 'running' and its consolidated result never
re-entered the parent conversation.
Add an async_delegation branch that delegates to the agent-side
tools.process_registry.format_process_notification (which already knows the
async_delegation shape), so the subagent result is rendered into the wakeup
prompt and re-enters chat. Scoped to async_delegation specifically (not all
unknown types) so genuinely-unrecognized events are still skipped — preserving
the existing skip-unknown-event contract.
Reported + root-caused by @mydelren.
Co-authored-by: mydelren <mydelren@users.noreply.github.com>
* stage bg_task trio combined (#2979 superset) on master for deep review
* fix(bg_task): unsubscribe SessionChannel on header-write failure (Codex deep-review catch) + regression test
* test: realign on-subscribe-recovery anchor to subscribe_to_session_channel after leak fix
* CHANGELOG: bg_task trio as v0.51.340 LD (HELD pending independent review)
* bg_task trio: apply 3 independent-review (greptile) fixes
1. start_session_turn now threads the session PROFILE model defaults
(_read_profile_model_config) into the wakeup model-resolve, so a brand-new
session with an empty model falls back to the profile default not global
DEFAULT_MODEL. Updated the white-box spy test signature accordingly.
2. /api/session/stream omits the Connection header (HTTP/1.1 keep-alive
default) to match the #3103 long-lived-SSE pattern.
3. Reaper now prunes _LAST_EMIT_TS for collected sessions so the coalesce
timestamp map can't grow one permanent entry per session forever.
nesquena APPROVED the PR; these are the 3 non-blocking greptile suggestions.
* test: realign _start_session_turn adapter stub lambda to new profile-defaults signature
happy5318