Opus SHIP-WITH-FIXES (no code change required): the comment/docstring overstated
that the session-list-change listener covers ALL structural mutations. In-app
mutations fire it, but externally-driven changes (scheduled cron completion,
external CLI writes) don't — for those the 30s streaming TTL is the load-bearing
backstop (≤30s bounded, self-healing lag). Comment-only accuracy fix; logic
unchanged (both gates: perf fix correct, freeze effective, staleness bounded).
/api/sessions re-ran the expensive CLI/cron session projection on every
poll while a turn was streaming — a major cause of the multi-second sidebar
latency + 100% CPU on cron-heavy installs (continuing #4672/#4808/#4889).
Root cause: the CLI/cron projection is gated by _CLI_SESSIONS_CACHE, whose
key folds in _sqlite_file_stat_cache_key -> _sqlite_content_fingerprint
(MAX(rowid) FROM messages). During a live turn the gateway writes a message
row per streamed delta, so that fingerprint advances on essentially every
~5s poll — busting the cache and re-running the full candidate-join +
projection (and the lineage-metadata pass) on every poll, contending for the
same SQLite/global lock the streaming worker holds.
The route-level session-list cache already froze its key during streaming
(#4808 _session_list_cache_streaming_freeze_marker), but that freeze never
reached this inner CLI-sessions cache, so the heavy CLI/cron query still
re-ran whenever the outer cache validated.
Fix: while any stream is active, the CLI-sessions cache key folds in the same
stable streaming-freeze marker (keyed only on the set of active stream ids)
instead of the volatile content fingerprint, and the cache TTL widens to a
streaming window. The projection is reused across polls mid-stream and
rebuilt at most once per window. The instant a stream starts/stops the marker
changes, so freshly-finished rows surface promptly. Structural mutations
(cron completion, new/renamed/archived sessions, attention) now also clear
the CLI cache via the existing session-list-changed listener — those signals
never fire per streamed token, which is what makes the freeze safe.
Idle behavior is byte-identical (freeze only engages with active streams).
Applies to both the active-profile and all-profiles cache paths.
Adds tests/test_issue4842_cli_sessions_streaming_freeze.py (6 tests): marker
idle/stable/changes semantics, the core 'key stable across message writes
while streaming' guarantee, idle key still advances with the fingerprint,
streaming TTL > idle TTL, and the structural-mutation listener clears the
CLI cache.
Opus gate finding (BLOCKING): the two #4385 archive-reappears regression tests
mocked Session.load_metadata_only to return an archived stub WITHOUT writing a
real sidecar file, and never set SESSION_DIR. The new stat-gate in
_state_projection_sidecar_metadata short-circuits on SESSION_DIR/{sid}.json
before consulting metadata, so those mocks were never reached -> archived=False.
This is NOT a production regression (in prod, load_metadata_only reads exactly
SESSION_DIR/{sid}.json and returns None when absent, so archived metadata only
exists when the file does — in which case the stat() succeeds and behavior is
identical). The divergence existed only at the tests' mock boundary.
Fix: both tests now write a real archived sidecar JSON under a monkeypatched
SESSION_DIR, exercising the actual production read path (strictly better
coverage than mocking load_metadata_only). Also documented the
SESSION_DIR-as-sole-source assumption on the cache function per Opus's
forward-looking note.
GET /api/sessions pinned CPU to 100% and took multiple seconds on profiles with
many cron sessions. The state.db -> sidebar projection did three pieces of
redundant per-row file I/O, in BOTH the visible pass and the higher-capped
(CRON_PROJECT_CHIP_LIMIT=200) cron-only second pass:
1. _state_projection_sidecar_metadata(sid): uncached open() + 64KB prefix read +
JSON-key scan, once per row.
2. get_last_workspace(): up to two file reads + an is_dir()/remote probe,
returning the SAME workspace for every row.
3. cron/jobs.json: full read + parse per untitled cron row.
On a cron-heavy install that is hundreds of file reads per build, and because
the enclosing _CLI_SESSIONS_CACHE is keyed on a state.db content fingerprint
that advances on every streamed message row, the whole scan was re-paid on
essentially every 5s poll during a live turn (the recurring 100% CPU / slow
/api/sessions from #4672 -> #4808 -> #4842).
Fix:
- Memoize _state_projection_sidecar_metadata by the sidecar's
(path, mtime_ns, size, ctime_ns) stat signature in a bounded module cache, so
a warm build re-stats instead of re-reads. A rename/archive/edit bumps the
signature and invalidates just that one entry (no staleness). This survives
the 5s CLI-cache busting during streaming, which is the actual trigger.
Mirrors the Claude Code parse cache (#4718). Folded into
clear_cli_sessions_cache() for explicit-reset/test parity.
- Build-scoped memos for cron jobs.json (parse once -> job_id->name map) and
get_last_workspace() (resolve once), matching the existing _cron_pid() pattern.
Output is byte-identical. Adds tests/test_issue4842_cron_projection_perf.py
(7 tests: warm-rebuild no re-read, stat-change invalidation, dict-copy isolation,
missing-sidecar default, jobs.json once, workspace once, lifecycle clear).
Dual-gate (Codex + Opus) findings on the Claude Code transcript parse cache:
1. Cache-key staleness hardening: add st_ctime_ns to the (path, mtime_ns, size,
max_messages) key so a rare same-size, same-mtime in-place edit can't serve a
stale parse. Size is already the strong discriminator for append-only JSONL;
ctime_ns is cheap defense-in-depth (a spurious bump costs one harmless re-parse,
never a stale read).
2. Timestamp-fallback bug-for-bug fix: the row-builder guard used identity
(first_ts is None and last_ts is None) but the field assignment uses truthiness
(first_ts or last_ts or _mtime). They disagreed on a falsy 0.0 timestamp
(epoch-0/1970 transcripts), leaving created_at/updated_at/last_message_at = None
instead of the file mtime the pre-cache inline code fell back to. Guard now uses
the same 'not first_ts and not last_ts' test.
Adds 3 regression tests: epoch-0 mtime fallback, the shared-dict read-only
contract (pins the load-bearing invariant), and same-size+mtime ctime invalidation.
The /api/sessions cold build (which a profile switch triggers) was
dominated by get_claude_code_sessions() re-parsing every Claude Code
JSONL transcript line-by-line on each call — 650-1000ms on a 200-file
/ ~130MB tree, repeated on every switch, every 5s CLI-cache expiry, and
every sidebar poll, because the directory is global but the higher
session cache is keyed per active profile.
Memoize the per-file parse by (path, mtime_ns, size) so a warm build
re-stats files instead of re-parsing them; any genuine edit/append
changes the stat signature and invalidates just that file. Also hoist
the loop-invariant get_last_workspace() out of the Claude Code row loop
(it stats config + probes terminal cwd, ~200 redundant calls per build).
Measured /api/sessions cold-rebuild on a 250-session profile: ~670-880ms
-> ~100-200ms (verified in-browser). Output byte-identical.
Credit @rodboev for the browser-level profiling that isolated the real
bottleneck (the prior pre-warm hypothesis could not move it).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex reproduced a SILENT regression: the PR's 4096-BYTE read could miss the
continuation marker the old read_text()[:4096] (4096 CHARS) found. Session.save()
writes compression_anchor_summary (ensure_ascii=False) BEFORE parent_session_id,
so a multi-byte emoji summary puts the marker at char offset ~1230 but byte
offset ~4900 — within the old 4096-char window, past a 4096-byte one → continuation
child no longer detected. Fix: read 16384 bytes (4096 chars worst-case for 4-byte
UTF-8) then slice [:4096] chars, keeping the same needle + bounded I/O. Multibyte
regression test added + proven non-vacuous; updated the read-size assertion to <=16384.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
Codex final-gate found the boundary<watermark advanced signal made
sidecar_advanced_past_watermark globally true, so a stale ts>watermark suffix
row appearing in state.db BEFORE the edited checkpoint bypassed the skip and
resurrected. Gate the bypass on state_replay_idx >= len(sidecar_visible_sequence)
(checkpoint consumed); apply the same gate to the same-second CORE-B exception.
Regression test added + proven non-vacuous. All legit cases (state-only reply
after checkpoint, CORE-B same-second) still pass.
Co-authored-by: AlexeyDsov <alexeydsov@gmail.com>
Final-gate Codex repro found a non-empty-sidecar drop: when the post-edit USER
turn is checkpointed into the sidecar (ts == advanced watermark) but its
ASSISTANT reply exists only in state.db (ts > watermark), the
sidecar_advanced_past_watermark guard was False (max_sidecar_timestamp ==
watermark, not >), so the 'timestamp > watermark' skip dropped the legitimate
reply. GET /api/session returned ['first','first reply','edited prompt'] and
omitted the assistant.
Fix: treat truncation_boundary < watermark (genuinely advanced) as advanced too,
so post-watermark state-only rows merge through even when the sidecar tail only
EQUALS the watermark. Conservative skip preserved for boundary None / == watermark.
Hoisted the loop-invariant boundary check out of the per-message loop. Regression
test added + proven non-vacuous.
Co-authored-by: AlexeyDsov <alexeydsov@gmail.com>
Opus deep-gate found a latent data-loss the Codex harnesses missed: the
empty-sidecar branch kept at_or_after (ts>=watermark) UNCONDITIONALLY, which is
only valid once the watermark was ADVANCED strictly past the original cutoff
(a new turn committed -> boundary < watermark). In the not-advanced states the
content above the watermark is the deleted suffix, so it got resurrected — the
same data-loss class this fix exists to kill. Reachable via crash/cold-load
metadata-vs-sidecar divergence.
- Keep at_or_after ONLY when boundary < watermark (genuinely advanced).
- boundary == watermark (just truncated) OR boundary is None (legacy: a positive
watermark always meant 'frozen at cutoff' pre-#4767) -> conservative ts<=watermark
filter; never resurrects.
- Replaced the backward-scan heuristic with the provable boundary guard.
- Removed the now-dead 'or (not sidecar_messages and watermark>0)' clause in the
non-empty-sidecar path (unreachable; empty sidecar returns earlier).
- Added 2 regression tests (boundary==watermark, legacy None-boundary), both proven
non-vacuous; updated 2 contributor tests that assumed advanced-watermark-without-boundary
(an unreachable state) to pass the boundary the real persistence path always writes.
Co-authored-by: AlexeyDsov <alexeydsov@gmail.com>
Converged after deep bounce cycles: Finding A (production regression — skipped the
state.db source correction before the CLI filter, dropping WebUI sessions with stale-CLI
sidecar flags) FIXED via _apply_sidebar_state_db_overrides(result) before the routes.py
filter on both index + full-scan paths; Finding B (flaky lineage test) FIXED via hermetic
_persisted_session_ids_snapshot fixture. 16 own tests deterministic. Code byte-faithful
from PR head e1a3ecf22eca.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: santastabber <santastabber@users.noreply.github.com>
Fixes ~5s GET /api/sessions on power users (2400+ sessions) — _enrich_sidebar_lineage_metadata
probed state.db for every row's compression lineage. Caps to top-N (default 300, env
HERMES_WEBUI_LINEAGE_TOP_N, 0=disable). Display-safe: both call sites pass an already
pinned-first/newest-first sorted list, so sessions[:N] is exactly the paint-priority window.
Fail-open preserved. Added regression test (RED on master, GREEN here). Code byte-faithful
from PR head 016978578008.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: maksym-mishchenko <maksym-mishchenko@users.noreply.github.com>
allenliang2022: recovered assistant rows were added to the visible transcript but not context_messages,
so the model lost recovered output. Adds _append_recovered_turn_to_context with a role-dedup guard.
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
franksong2702's activity_scene_v1 anchor-scene model: live-stream/settled/refresh/re-entry
show the same ordered turn. Rebased clean (no conflicts) onto current master.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Codex CORE findings on the fingerprint: (1) default busy timeout could stall
/api/sessions ~10s on a locked state.db; (2) COUNT(*) forced a full messages
table SCAN (~16-22ms/refresh on 200k rows). Fix: open file:...?mode=ro with
timeout=0.05 + PRAGMA busy_timeout=50 (returns fast, falls back to stat stamp on
lock), and use index-only MAX(rowid) per table (no COUNT, no scan). Verified:
EXPLAIN shows SEARCH not SCAN, 0.36ms on 200k msgs, 0.16ms while EXCLUSIVE-locked.
Fingerprint shape is now (sessions_max_rowid, messages_max_rowid); regression
tests updated.
Codex root-cause consult confirmed: the CLI-session cache (_CLI_SESSIONS_CACHE,
5s TTL) and session-list cache were keyed on (st_mtime_ns, st_size) of state.db
+ WAL sidecars. Under WAL-mode writes those stamps collide (same mtime-ns bucket
+ WAL frame size after checkpoint), serving a stale cache -> freshly-committed
gateway/CLI session intermittently absent from /api/sessions (the recurring
test_gateway_sync flake AND a real up-to-5s sidebar delay for gateway writes).
PRAGMA data_version is NOT usable here (fresh per-request connection always
reports its own initial value, never advances — verified). Fix = a cheap
content fingerprint (COUNT/MAX(rowid) over sessions+messages) read on a fresh
connection, which DOES advance on every commit and is immune to mtime
granularity. Wired into _sqlite_file_stat_cache_key (models) + the routes-layer
source stamp. + 3 regression tests + corrected the earlier false root-cause
comments.
allenliang2022