Metering telemetry is journaled at ~10 Hz during active turns, so marathon
reasoning runs balloon the durable run journal (12.4 MB of a single 18 MB run
was metering). On a mid-run refresh the reconnect path replays the entire
journal as one unpaced SSE burst, OOM-crashing the browser tab in a
crash-refresh loop while the server stays healthy.
- Write time: RunJournalWriter.append_sse_event() skips metering rows via
REPLAY_SKIPPED_SSE_EVENTS, returning None so caller journal-id plumbing is
untouched and remaining journaled seqs stay contiguous.
- Replay time: _replay_run_journal() and emit_replay() filter metering rows
through journal_replay_visible(), so legacy journals never stream the
backlog either.
- Readers (read_run_events / read_session_run_events) deliberately keep every
row: cursor math and the offline-gap coverage check count seqs.
Refs #7272
`_run_journal_live_snapshot` rebuilds the live UI state for an active run
by replaying the whole durable journal. On a long run (22.7k events /
15MB) that rebuild took ~4.4s per session load and blocked the HTTP
thread - the dominant cost of "switching to a session with a task
running takes forever". Four output-preserving fixes:
1. Locate the journal without parsing it (`find_run_file`) and derive the
durable summary from the single `read_run_events` parse instead of
parsing the file twice (`find_run_summary` + `read_run_events`). The
historical `find_run_summary` lookup is kept as the fallback when no
journal file exists on disk.
2. Skip `metering` rows in the projection loop (~60% of a long live
journal's bytes; nothing in the snapshot is built from them), while
preserving the last_ts watermark they carry.
3. Collect reasoning deltas into a list and materialize lazily instead of
`reasoning_text += chunk` (quadratic inside these closures: ~2.3s of
the replay on a 9.7k-chunk run).
4. Fold only a bounded tail of the reasoning transcript for the
interim-echo suffix check instead of `re.sub` over the whole growing
text (~1.5s over 100+ probes).
Measured on a 15MB / 22.7k-row live journal: 4.34s -> 0.49s (8.9x) with
byte-identical snapshot output (sha256 of the canonical JSON unchanged).
Related work: #7310 (slow /api/session under load), #7534 (incremental
journal read cache - complementary: this removes one of the two parses
entirely and fixes the replay-loop hotspots), #7273/#7291 (metering
write-side skip - the read-side loop skip here complements them for
legacy journals that still carry metering rows).
Fable gate found a seam Codex's 3 boundary findings didn't cover: an idle no-cursor
subscriber to /api/sessions/{id}/events misses a run that starts AND finishes inside a
single keepalive tick — the wait loop only looked for a live in-memory STREAMS entry to
attach, never noticing the journal advanced, so the run's transcript was silently missing
until manual refresh.
Add a cheap bounded session_journal_fingerprint() (file-count/max-mtime/total-size from
stat only, never parses bodies) and, inside the idle attach-wait loop, emit a session_snapshot
re-sync whenever the fingerprint advances with no live stream to attach — the same honest
recovery contract used for a failed reconciliation. Re-baselines after each re-sync so a quiet
idle connection never spams snapshots. 2 regression tests; 22/22 SSE tests green.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
append_run_event() without an explicit seq called _next_seq(path), which
re-read and re-parsed the entire journal file on every append — O(n) per
append, O(n^2) over a run. Direct callers therefore paid a full file
reparse per event; only RunJournalWriter avoided it via its own in-memory
counter.
Introduce a module-level _SEQ_CACHE keyed by run-journal file path. The
per-path _lock_for(path) serializes same-path reserve->append so seqs
stay monotonic and file order matches; both RunJournalWriter and the free
append_run_event now draw from this one cache (_reserve_next_seq), so
seqs stay gapless even when both write the same path — the writer's
private counter is removed to keep a single source of truth. An explicit
caller-supplied seq advances the cache (_note_assigned_seq) so it can
never be re-issued. delete_run_journal() now also evicts cache entries
for the removed session so a run re-created at the same path restarts
numbering at 1.
Guard every structural _SEQ_CACHE access (reserve/note/evict) with a
dedicated _SEQ_CACHE_LOCK. The eviction sweep in delete_run_journal
iterates the whole dict; the per-path locks the append path holds do NOT
serialize against it, so a concurrent append on another session inserted
a fresh key mid-iteration and raised "dictionary changed size during
iteration" (the caller swallows it, but the eviction then aborts and
leaks stale seqs, so a re-created run resumed a stale number instead of
restarting at 1). A stress test races deletes against cross-session
appends and asserts no such error.
Rollback: revert this commit; _next_seq(path) re-read behaviour returns.
No on-disk format change, so existing journals stay compatible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`_lock_for` caches one threading.Lock per (dir, file, pid) in the module-global
`_WRITER_LOCKS`, but nothing evicted those entries: `delete_run_journal` rmtree'd
the on-disk `_run_journal/{sid}/` directory yet left the cached lock objects
behind, so a long-lived gateway leaked one entry per deleted run forever.
Delete now evicts every cached lock whose parent directory matches the removed
session (pid-independent) under `_WRITER_LOCKS_GUARD`, leaving unrelated
sessions' locks intact.
Refs #4633, #2097.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Harden interrupted recovery control filtering
* Redesign live-to-final assistant replies
* Fix live activity anchor test fixture
* Fix CI lint issues for live reply tests
* Strengthen live progress prompt contract
* Recover PR #3401 refresh on origin/master
* Repair live-to-final refresh regressions
* Fix live worklog refresh regressions
* Show live footer timer on initial stream start
* Restore live stream shell after reload
* Preserve per-frame live SSE replay cursors
* Preserve reasoning as Worklog Thinking cards
* Quiet Worklog Thinking card styling
* Align Worklog Thinking card styling
* Scope live Worklog Thinking cards by segment
* Suppress exact duplicate settled Thinking
* Close#3401 merge review test gaps
* fix(#3401): resolve 4 deep-review regressions (inline-think, reconnect-dup, neon skin, busy-gate worklog)
Deep review (Codex diff-vs-master + live-browser drive) of the live-to-final refactor
surfaced 4 regressions vs master that the rewritten suite no longer guarded:
1. Inline <think>…</think>answer reasoning vanished — _assistantReasoningPayloadText
used $-anchored regexes so a leading think block + visible answer extracted nothing
and the Thinking card never rendered. Removed the 3 $ anchors to match the
(non-anchored) display stripper. Live: inline-think thinking-only turn now renders.
2. (CORE) reconnect/reload duplicated the live reply — _rememberRunJournalCursor advanced
a closure-local seq but never wrote INFLIGHT[activeSid].lastRunJournalSeq, so a reload
replayed the journal from after_seq=0 over restored lastAssistantText. Now mirrors the
cursor onto INFLIGHT + schedules a throttled persist.
3. Neon skin silently broke — PR deleted the :root[data-skin="neon"] CSS but left Neon in
the picker. Restored the neon CSS block from master.
4. Settled tool-worklog rebuild gated purely on !S.busy — dropped every prior settled
turn's worklog when renderMessages re-ran during an active stream (switch-back to an
in-progress session). Restored master's !S.busy || (S.toolCalls && S.toolCalls.length).
Live: busy re-render now preserves tool cards (4→4, was 4→0).
Live-verified all 4 + confirmed #3709/#3592 invariants still hold (1 thinking card, none
below footer; distinct siblings preserved). + tests/test_issue3401_deep_review_fixes.py (7).
* test(#3401): realign 3 stale source-shape assertions to the deep-review fixes
Fix commit changed two source literals that existing stage tests scanned for:
- test_live_activity_timeline.py (x2): split anchor 'if(!S.busy){' → the restored
'if(!S.busy || (S.toolCalls&&S.toolCalls.length)){' guard (fix 4).
- test_run_journal_frontend_static.py: 'after_seq=0' not in source — fix 2's comment
contained that literal; rephrased the comment to 'the zero floor (after_seq of 0)'.
Intent of all three assertions unchanged; only the matched string updated. No code
behavior change.
* docs(changelog): v0.51.294 — Release JJ (stage-3401, #3401 live-to-final redesign)
---------
Co-authored-by: Frank Song <franksong2702@gmail.com>
Co-authored-by: Nathan-Hermes <nesquena-hermes@users.noreply.github.com>
Co-authored-by: nesquena-hermes <[email protected]>
Frank Song