Codex gate CORE: the exact-empty sentinel only suppressed restore for a bit-empty
cleared sidecar. If the user cleared, then sent ONE post-clear message, and the
stale pre-clear .json.bak still existed (unlink failed), recovery saw
bak_count>live_count and RESTORED the pre-clear transcript on top of the new
message. Added _live_supersedes_backup_by_clear_generation: suppress restore when
the live sidecar carries a clear_generation the backup lacks AND has a non-empty
transcript AND still shows the clear boundary reset (watermark==boundary==0.0).
Scoped to non-empty live so empty clear-shaped sidecars keep their existing
recovery semantics (active/pending/partial variants still restore). Fails open on
unreadable/partial reads. +2 regression tests (post-clear-message not restored;
post-clear-after-real-compaction still restores).
Closes the crash-window resurrection corner in the #5532 clear-conversation
data-loss fix. /api/session/clear writes a .json.bak before truncating, and
because the cleared sidecar's truncation_watermark is 0.0, startup recovery saw
bak_count > live_count and could restore the backup — resurrecting the cleared
transcript after a crash/restart in the clear window.
Fix: stamp a unique per-clear `clear_generation` UUID on the sidecar when
clearing a session that had messages (persisted via Session model). Recovery's
_session_records_clear_sentinel() returns no_action when the live sidecar carries
a clear_generation the backup doesn't share AND the live sidecar matches the
exact cleared-empty shape (messages/context empty, watermark/boundary 0.0,
pending fields cleared). Same-generation backups stay recoverable; unreadable or
partial matches FAIL OPEN (normal recovery), so a real crash-loss is never
suppressed.
Contributor stage; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex v4 SILENT finding: my audit Finding-2 fix made the orphan-.bak branch emit
a state_db_deleted_webui_tombstone item, but the state_db_missing_rows loop still
emitted the same sid/kind again when both a surviving .json.bak AND a state.db row
exist for one deleted session — double-counting unsafe_to_repair, violating the
no-double-classification invariant. Fix: track _bak_tombstoned_ids from the .bak
branch and skip them in the state.db loop. Regression test
test_audit_no_double_count_when_bak_and_state_db_row_both_survive (verified fails
without the dedup).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
@rodboev independently pushed fixes for messaging-tombstone (matches ours) and an
audit orphan-.bak skip — but the audit skip used _marks_deleted_webui_session
(index heuristic), which re-introduces the crash-suppression bug, and the other 4
resurrection/recovery paths were still open. Rebuilt the stage on rodboev's new
head (fc21c3f3, attribution preserved) and applied the complete superset:
- routes.py claim path: tombstone 404 gated on _state_db_session_source in
('','webui','fork') so foreign-source rows always materialize (finding B)
- models.py sidebar projection: skip tombstoned source='webui' rows w/ no live
sidecar so a deleted session can't resurface in /api/sessions (finding C)
- session_recovery.py orphan-.bak startup: skip restore when durable tombstone
present, durable-only (finding D)
- session_recovery.py audit orphan-.bak: durable-only classify as
deleted_session_skipped (corrects rodboev's index-heuristic version)
- session_recovery.py _read_state_db_missing_sidecar_rows: durable-tombstone-only
so a genuine crash (index intact, no durable tombstone) still repairs (finding F)
Regression tests (all verified fail-without-fix): sidebar-projection, orphan-.bak
startup skip, messaging-not-tombstoned, index-only-crash-repairable. Both advisors
(Codex SAFE TO SHIP + Opus ship-with-fixes) cleared the prior stage; this rebuild
carries the same fixes onto the contributor's latest work.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
When a settled assistant turn carries _anchor_activity_scene and has mixed
content[] (text interleaved with tool_use), promote that ordering into anchor
scene rows (prose / tool / thinking rows in sequence) instead of only the raw
content[] fallback, and mirror the same scene model in the backend hydration
path so a cold reload reconstructs an identical transcript.
Also fixes: post-tool_use text/thinking in a NON-FINAL assistant message was
silently dropped — only the turn-final assistant's post-last-tool text is the
"final answer" tail; earlier assistant messages keep post-tool content as
activity rows. Tail thinking is excluded from the final-answer text but still
emitted as a thinking row (verified on both JS and Python paths).
Tool-row de-duplication is conservative — ID-equal always merges; name/
invocation matches are cardinality-gated and same-message; body-prefix is only
a secondary confirmation inside the id-flexible branch (also requires matching
started_at + compatible name + invocation). It biases toward an extra visible
tool card over ever silently merging two distinct calls.
Gate: Codex SAFE (tail-thinking edge verified already-handled both sides),
Opus SHIP (dedup fails toward duplicate-not-loss; non-final preservation, XSS,
non-mixed regression all clean), full suite 10778 passed (only the 4 known
pre-existing flakes). One non-blocking follow-up noted: bring the prose/thinking
peer-dedup into JS↔Python parity (cosmetic cold-reload extra-narration row).
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Manual /compress shrinks the model-facing context_messages but keeps the
visible messages[] transcript. The old context returned via two paths (#4836):
(1) append-only state.db reconciliation re-appending pre-compression rows, and
(2) startup .bak recovery treating the intentional shrink as data loss.
Half A (routes.py _handle_session_compress): persist truncation_watermark +
truncation_boundary (= watermark of the compressed context), set
compression_anchor_mode="manual", refresh last_prompt_tokens, stamp missing
timestamps on the compressed context, and delete the now-stale .bak. The
boundary==watermark stamp drives the #4772 reconciliation logic onto its
conservative path (block replay of pre-compression rows) while post-compression
turns still merge once sidecar timestamps advance past the watermark.
Half B (session_recovery.py): the .bak recovery guard. MAINTAINER FIX over the
original PR — the contributor suppressed recovery whenever
compression_anchor_mode=="manual", a flag set once at compress and never
cleared, which PERMANENTLY disabled #1558 crash-recovery for any compressed
session (real data loss). Recovery is now suppressed only when the session was
intentionally compressed AND the .bak is genuinely the pre-compression backup,
discriminated by the compaction marker (the same _context_messages_include_
compression_marker signal reconciliation uses): a marked .bak post-dates the
compression -> recover; an unmarked .bak with a larger context is the
shrink-undoing pre-compression one -> suppress. Fail-open on any error.
Resolves an Opus-gate edge in the first cut (length-only heuristic wrongly
suppressed a loss that shrank BOTH messages and context). Two new non-vacuous
regression tests: post-compression real loss recovers, and the both-shrunk
marked-backup case recovers.
Co-authored-by: hyl-ailab <hyl-ailab@users.noreply.github.com>
#3814: O(n) index-count map for /api/sessions. #3815: skip backup-less sidecars on startup recovery. Full suite 8275, Codex SAFE, Opus SHIP, CI 11/11. Co-authored-by: ai-ag2026 <ai-ag2026@users.noreply.github.com>
PR #2053 added worktree-backed session creation. PR #2041 (shipped in
v0.51.42) added state.db sidecar reconciliation that rebuilds a missing
<sid>.json sidecar from the canonical state.db row when the JSON file is
gone (failed save, manual rm, restore-from-backup with mismatched dirs).
The two interact silently. `_state_db_row_to_sidecar()` was hard-coding
`'workspace': ''` and never propagating the four worktree_* fields from
the row to the rebuilt sidecar dict. So a worktree-backed session that
loses its sidecar and gets rebuilt from state.db:
- loses `worktree_path` → matches the empty-session sidebar filter at
`api/models.py:1067/1107` (which spares worktree-backed empty sessions
via `not s.get('worktree_path')`) → session disappears from the
sidebar even though the worktree directory still exists on disk.
- loses `workspace` → downstream tools (terminal panels, file pickers
that use `s.workspace`) operate on empty string instead of the original
worktree path.
- always reports `message_count == 0` → contributes to the empty-session
filter even for sessions that have messages in `state.db.messages`.
Fix:
1. `_read_state_db_missing_sidecar_rows()` SELECT now includes
`workspace, worktree_path, worktree_branch, worktree_repo_root,
worktree_created_at, message_count` (each gated by
`_sql_optional_col()` so older state.db schemas without those columns
continue to work — recovery degrades gracefully rather than 500ing).
2. `_state_db_row_to_sidecar()` propagates each field. workspace comes
from the row if it's a string, otherwise '' (matching pre-fix behavior
for non-worktree sessions). message_count comes from the row if
it's an int, otherwise falls back to `len(messages)` so the rebuilt
sidecar always has a coherent count.
3 new regression tests in tests/test_state_db_worktree_recovery.py
exercise:
- worktree session with messages → all four worktree_* fields preserved.
- non-worktree session → worktree_* fields all None (no spurious
propagation), workspace=''.
- empty worktree session (the worst case) → confirms the rebuilt sidecar
does NOT match the empty-session-exempt filter, so it stays visible
in the sidebar.
Caught by Opus advisor during stage-337 review (the cross-PR interaction
between #2053 and the previously-shipped #2041 wasn't exercised by either
PR's individual test suite).
Two concrete data-corruption vectors flagged in Opus review of PR #2041,
both fixed atomically so the new repair-safe endpoint is safe for production:
1. Shared tmp filename under concurrent calls
`tmp = target.with_suffix('.json.reconcile.tmp')` produced a fixed path
per session ID. Two simultaneous repair-safe POSTs would interleave bytes
in the same tmp file, then both rename → corrupted JSON. Now matches the
`Session.save()` convention at api/models.py:484 with a pid+tid suffix.
2. TOCTOU between target.exists() check and tmp.replace(target)
`os.replace()` overwrites unconditionally. If a concurrent Session.save()
for the same SID materialized the live sidecar in the microsecond window
between the existence check and the rename, the reconciliation would
silently overwrite a live sidecar with a (lossier) state.db reconstruction.
Switched to `os.link()` + `unlink(tmp)` which is atomic create-or-fail —
on FileExistsError we record `skipped: sidecar_appeared_during_reconcile`
and keep the live sidecar untouched.
Plus a round-trip schema-parity test: materialize a sidecar from state.db,
then load it back through `Session.load()` and assert the messages survive.
Catches future schema drift between `_state_db_row_to_sidecar()` and
`Session.__init__()`. Also adds a guard test confirming the .reconcile.tmp
suffix includes pid+tid (regression guard for hazard #1).
Tests: 23 passing across the recovery suite (was 21; +2 new in this commit).
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
(1) api/session_recovery.py: removed misleading dated-format comment claim.
YYYYMMDD_HHMMSS_*.json files don't start with '_' so the underscore-
skip wouldn't apply to them anyway. Replaced with the truthful general
statement: any future non-session JSON marked with the '_' convention
is skipped automatically.
(2) CHANGELOG.md: fixed self-referential typo. v0.50.284 obviously couldn't
have said 'v0.50.285' inside its release notes — the quoted text was
'after deploying v0.50.284'.
Pure documentation. No behavior change. Tests still pass (8/8 in
tests/test_metadata_save_wipe_1558.py).
v0.50.284 shipped startup self-heal in api/session_recovery.py that
crashed on the very first JSON file it scanned in the production
session directory. Verified live on the prod server immediately after
the v0.50.284 deploy:
[recovery] startup recovery failed: 'list' object has no attribute 'get'
Root cause: the production session dir contains _index.json — a
top-level LIST of session metadata dicts (not a dict). _msg_count()
did data.get('messages') which raises AttributeError on a list.
The broad except Exception in server.py's startup hook swallowed the
error and the recovery silently no-op'd for every user — defeating
the entire purpose of the v0.50.284 release.
Fix is three small defensive changes:
1. _msg_count() — added isinstance(data, dict) guard. Non-dict-shaped
JSON files now return -1 (the harmless 'unknown count' sentinel)
instead of raising AttributeError.
2. recover_all_sessions_on_startup() — skips any file whose name starts
with '_' (the existing project convention for non-session metadata
files like _index.json). These are convention-marked as system
files, not session payloads.
3. recover_all_sessions_on_startup() — wraps recover_session(path) in
try/except Exception so a single malformed file can't break recovery
for the rest. Logs and continues.
2 new regression tests:
- test_recover_all_sessions_on_startup_skips_non_session_index_json
- test_msg_count_returns_neg1_for_non_dict_top_level
4026 → 4028 tests passing (+2).
Net effect: any user wiped between v0.50.279 and v0.50.284 deploys
whose session has a .bak shadow will now get auto-recovered on first
launch of v0.50.285, as v0.50.284's release notes promised.
Closes#1558 (follow-up — the original P0 was closed by v0.50.284 but
the recovery half didn't actually run in production).
The PR title and body correctly say 'Closes #1558' but every code comment,
the test file name, error-message strings, docstrings, and the original
commit body referenced #1557 instead. Independent reviewer flagged this:
> The 17 wrong references won't auto-close issue #1558 from the commit
> message — and the test file name will be misleading for future archeology.
> Worth a one-pass s/#1557/#1558/g (and rename test file →
> test_metadata_save_wipe_1558.py) before merge so the artifacts agree
> with reality.
This commit:
- Renames tests/test_metadata_save_wipe_1557.py → test_metadata_save_wipe_1558.py
- Replaces 17 #1557 references with #1558 across:
- tests/test_metadata_save_wipe_1558.py (7 refs)
- api/models.py (5 refs in Session.save guard + backup safeguard comments)
- api/routes.py (2 refs in _clear_stale_stream_state docstring + log)
- api/session_recovery.py (3 refs)
- server.py (3 refs in startup self-heal block)
Verified: 6/6 tests in tests/test_metadata_save_wipe_1558.py pass
with the renamed file + updated references.
v0.50.279 introduced api.routes._clear_stale_stream_state() (#1525) which
calls session.save() to clear stale active_stream_id/pending_* fields. The
helper is called from /api/session and /api/session/status — both of which
load the session with metadata_only=True. Session.load_metadata_only()
synthesizes a stub with messages=[] (its whole purpose: fast metadata read
without parsing the 400KB+ messages array). Session.save() unconditionally
writes self.messages to disk via os.replace(), so saving a metadata-only
stub atomically overwrites the on-disk JSON with messages=[], wiping the
entire conversation.
Production trigger: every SSE reconnect cycle after a server restart polls
/api/session/status, which fans out to _clear_stale_stream_state, which
saves the metadata-only stub. The user reported losing 1000+ message
conversations and seeing 'Reconnecting…' loops on every prompt — the
reconnect loop kept the cycle running until the conversation was empty.
Fix: three layers, defense in depth.
(1) api/models.py: load_metadata_only() now sets _loaded_metadata_only=True
on the returned stub. Session.save() raises RuntimeError if that flag
is set — a hard guard so any future caller making the same mistake
cannot wipe data, only crash visibly.
(2) api/routes.py: _clear_stale_stream_state() now detects the metadata-only
flag and re-loads the full session with metadata_only=False before
mutating persisted state. The full-load path also runs
_repair_stale_pending() which independently clears the stream flags,
so the explicit clear becomes a no-op in most cases — but messages
stay intact.
(3) api/models.py + api/session_recovery.py: every save() that would
SHRINK the messages array (the precise failure shape of #1557) first
snapshots the previous file to <sid>.json.bak. Server.py runs
recover_all_sessions_on_startup() at boot — any session whose live
JSON has fewer messages than its .bak is restored automatically.
Idempotent on clean state. Backup overhead is zero on the normal
grow-the-conversation path.
Reproducer (master): test_metadata_only_save_does_not_wipe_messages goes
from 1000 messages to 0 in a single save() call. After the fix, 1000
messages survive.
Tests: 6 new regression tests in tests/test_metadata_save_wipe_1557.py
covering all three layers. Full pytest: 4019 → 4025 (+6, all green).
Live verified on port 8789: write 1000-msg session with stale active_stream_id,
hit /api/session/status, /api/session — file ends with 1002 messages
(_repair_stale_pending injects an error-marker pair on full reload, harmless
existing behavior), active_stream_id cleared, pending cleared, no Reconnecting
loop.
Closes#1557.
Reported by AvidFuturist via user feedback on v0.50.282.
