release #5532: clear-conversation P0 data-loss (combined, rebuilt on de-brittled base)

Rebuilt the combined #5556+#5553 fix on v0.51.859 (now has the flake fix + the
#5542 RFC de-brittle, so no anchor-test collateral). Nathan's call: rodboev's
#5556 primary + fold self-built #5553 improvements, credit both.

Clear handler now:
- routes through shared truncate_session_at_keep(s,0) (single source of truth,
  sets watermark=_truncation_watermark_for([])==0.0, the #2914 sentinel that
  blocks state.db append-merge replay)
- detaches compression lineage ONLY when the parent is a pre_compression_snapshot
  (Codex-caught: preserve genuine fork parent links for nesting + "Forked from")
- rodboev's persisted-clear read-back verification + stale .bak removal

RFC conflict resolved in favor of the de-brittled (symbol-anchor) master version.
Ships both PRs' test files (state_db_replay + clear_truncation_watermark incl the
fork-preservation regression). Opus (on the amended tree): both resurrection paths
closed, detach lineage-safe, verification sound — SHIP. 3 non-blocking follow-ups
filed (#5570 .bak crash-window, #5571 fork-stitch corner, #5572 messaging clear).

Co-authored-by: rodboev <rodboev@users.noreply.github.com>

CHANGELOG.md

@@ -14,6 +14,7 @@
 
 ### Fixed
 
+- **Clearing a conversation now actually clears it — history no longer comes back after a refresh (P0 data-loss).** `Clear conversation` wiped the on-screen transcript but never recorded a truncation watermark, so the messages resurrected from the agent's state database on the next load: the cleared history reappeared after a refresh, and continuing the conversation still fed the full pre-clear context to the model. Clear now routes through the same truncate-to-empty path as the truncate action (setting the watermark that blocks state-db replay), verifies the empty state actually persisted, removes the stale backup, and — for a compressed-continuation session — detaches the compression-snapshot parent link so the archived transcript can't be stitched back in either (ordinary fork links are preserved). Thanks @rodboev and the maintainer fix. (#5556, #5553, #5532)
 - **Transparent Stream now shows a running tool as running, not already-done.** The live anchor-scene renderer hardcoded the tool-status as "settled" even while the tool was still executing, so a tool call flashed straight to its finished state in Transparent Stream instead of showing its in-progress status. It now uses the row's actual settled state, so a running tool reads as running until it genuinely completes. Thanks @rodboev. (#5547, #5523)
 - **The composer no longer repopulates the tail of a message you already sent.** On mobile/slow connections, a just-sent message could momentarily reappear in the composer because a stale server-side `composer_draft` restored before the empty-draft write caught up. A short per-session suppression window now blocks that stale restore right after send — but scoped to the specific sent payload, so a genuinely new draft you (or another tab) start for the same conversation still restores immediately instead of being swallowed for the window. Drafts are also cleared on the busy send paths (queue / interrupt / steer). Thanks @ruizanthony. (#5471)
 - **Windows: after an upgrade, browsers no longer keep serving stale JS/CSS when git isn't on the server's PATH.** When the WebUI server was launched on Windows from an environment without git on `PATH` (a venv, a service wrapper, a non-interactive launcher), version detection silently degraded to `unknown`, which froze the `?v=<stamp>` static-asset cache key — so browsers kept serving cached `ui.js` / `messages.js` / CSS even after the server restarted with fixed code, and the only workaround was a manual hard-refresh on every device. Version detection now resolves `git.exe` from the Git-for-Windows registry key (and common install paths) when it isn't on `PATH`, so the cache stamp updates and clients pick up frontend fixes automatically. Non-Windows behavior is unchanged. Thanks @allenliang2022. (#5522)

api/routes.py

@@ -13695,11 +13695,42 @@ def handle_post(handler, parsed) -> bool:
         sid = body["session_id"]
         with _get_session_agent_lock(sid):
             had_sidecar_messages = bool(s.messages or [])
-            s.messages = []
+            # Clear is a full truncate-to-empty: route through the SAME helper the
+            # /api/session/truncate handler uses (single source of truth) so the
+            # display + context arrays are emptied AND the truncation watermark is
+            # set via _truncation_watermark_for([]) == 0.0 — the #2914
+            # truncate-to-empty sentinel that blocks state.db replay. Before this,
+            # /clear wiped s.messages but left the watermark unset, so the
+            # append-only state.db merge treated it as "keep everything" and the
+            # cleared history resurrected on the next /api/session read (#5532).
+            from api.session_ops import truncate_session_at_keep
+            truncate_session_at_keep(s, 0)
             s.tool_calls = []
-            s.context_messages = []
-            s.truncation_watermark = 0.0
-            s.truncation_boundary = 0.0
+            # A compressed-continuation child keeps its archived transcript in a
+            # parent sidecar marked pre_compression_snapshot;
+            # _webui_sidecar_lineage_messages_for_display() stitches that parent
+            # back with truncation_watermark=None, so the 0.0 sentinel on the
+            # CHILD does NOT stop the parent from resurrecting the cleared history
+            # on refresh. Detach the compression lineage (#5532/#5553) — but ONLY
+            # when the parent is actually a pre_compression_snapshot; a genuine
+            # fork parent (session_source="fork" from /api/session/branch) must
+            # keep its link so the child still nests + shows "Forked from"
+            # (sessions.js:5720/5964/7105). Dropping every parent broke that
+            # (#5532 Codex gate).
+            _parent_sid = getattr(s, "parent_session_id", None)
+            if _parent_sid:
+                _parent_is_compression_snapshot = False
+                try:
+                    _parent = get_session(_parent_sid, metadata_only=True)
+                    _parent_is_compression_snapshot = bool(
+                        getattr(_parent, "pre_compression_snapshot", False)
+                    )
+                except Exception:
+                    _parent_is_compression_snapshot = False
+                if _parent_is_compression_snapshot:
+                    s.parent_session_id = None
+                    s.compression_anchor_visible_idx = None
+                    s.compression_anchor_message_key = None
             s.active_stream_id = None
             s.pending_user_message = None
             s.pending_attachments = []

tests/test_issue5532_clear_truncation_watermark.py

@@ -0,0 +1,348 @@
+"""Regression tests for #5532 — /api/session/clear P0 data loss.
+
+Root cause: the ``/api/session/clear`` handler wiped ``s.messages`` and
+``s.tool_calls`` but NEVER set ``truncation_watermark``. The append-only
+state.db merge (``merge_session_messages_append_only`` via
+``reconciled_state_db_messages_for_session``) treats an unset / None watermark
+as "keep everything and just dedup", so the next ``/api/session`` read
+resurrected every cleared turn from state.db:
+
+  * history reappeared after clear + refresh, and
+  * because ``context_messages`` also survived the clear, a continued turn still
+    carried the full pre-clear context into the model.
+
+The sibling ``/api/session/truncate`` handler does NOT have this bug: it calls
+``truncate_session_at_keep(session, keep)`` which sets
+``truncation_watermark = truncation_boundary = _truncation_watermark_for(kept)``.
+
+Fix: route ``/clear`` through the SAME helper with ``keep=0``. A full clear is a
+truncate that keeps zero messages, so the watermark becomes
+``_truncation_watermark_for([]) == 0.0`` — the #2914 "truncate-to-empty"
+sentinel that blocks ALL state.db replay — and ``context_messages`` is emptied
+in lockstep. The merge contract is not forked; ``/clear`` and ``/truncate`` now
+set the marker identically.
+
+These tests fail on origin/master (watermark stays None → merge resurrects the
+cleared transcript) and pass with the fix.
+"""
+from __future__ import annotations
+
+import json
+from io import BytesIO
+from types import SimpleNamespace
+
+
+def _msg(role: str, content: str, ts: float, mid: str) -> dict:
+    return {"id": mid, "role": role, "content": content, "timestamp": ts}
+
+
+def _seed_session_dir(monkeypatch, tmp_path):
+    """Point the session store at an isolated tmp dir (mirror #2914 harness)."""
+    import api.models as models
+
+    session_dir = tmp_path / "sessions"
+    session_dir.mkdir(parents=True)
+    monkeypatch.setattr(models, "SESSION_DIR", session_dir)
+    monkeypatch.setattr(models, "SESSION_INDEX_FILE", session_dir / "_index.json")
+    models.SESSIONS.clear()
+    return models
+
+
+def _call_clear(monkeypatch, session_id):
+    """Invoke the real POST /api/session/clear route and return its payload."""
+    import api.routes as routes
+
+    body = {"session_id": session_id}
+    body_bytes = json.dumps(body).encode()
+    monkeypatch.setattr(routes, "_check_csrf", lambda handler: True)
+    # Eviction runs provider I/O; stub it so the test stays hermetic.
+    monkeypatch.setattr("api.config._evict_session_agent", lambda _sid: None)
+
+    captured = {}
+
+    def fake_j(handler, payload, status=200, extra_headers=None):
+        captured["payload"] = payload
+        captured["status"] = status
+
+    monkeypatch.setattr(routes, "j", fake_j)
+
+    handler = SimpleNamespace(
+        headers={"Content-Length": str(len(body_bytes))},
+        rfile=BytesIO(body_bytes),
+    )
+    routes.handle_post(handler, SimpleNamespace(path="/api/session/clear"))
+    return captured
+
+
+def _four_turn_messages():
+    return [
+        _msg("user", "first", 1.0, "u1"),
+        _msg("assistant", "reply first", 2.0, "a1"),
+        _msg("user", "second", 3.0, "u2"),
+        _msg("assistant", "reply second", 4.0, "a2"),
+    ]
+
+
+def test_clear_endpoint_sets_truncation_watermark(monkeypatch, tmp_path):
+    """POST /api/session/clear must set truncation_watermark (0.0 sentinel).
+
+    This is the fail-without-fix assertion: on origin/master the watermark
+    stays None because the handler never set it.
+    """
+    models = _seed_session_dir(monkeypatch, tmp_path)
+    from api.models import Session
+
+    session = Session(
+        session_id="issue5532clear",
+        messages=_four_turn_messages(),
+        context_messages=_four_turn_messages(),
+    )
+    session.save()
+
+    captured = _call_clear(monkeypatch, "issue5532clear")
+    assert captured["payload"].get("ok") is True
+
+    loaded = Session.load("issue5532clear")
+    assert loaded is not None
+    # Full clear = truncate-to-empty: watermark is the #2914 sentinel 0.0
+    # (== _truncation_watermark_for([]) == the last KEPT message timestamp for
+    # keep=0). On master this is None and the assertion fails.
+    assert loaded.truncation_watermark == 0.0
+    assert loaded.truncation_boundary == 0.0
+    assert loaded.messages == []
+
+
+def test_clear_empties_context_messages(monkeypatch, tmp_path):
+    """Second-half fix (#5532): context_messages must also be cleared so a
+    continued turn does not carry pre-clear context into the model. On master
+    the /clear handler left context_messages untouched."""
+    _seed_session_dir(monkeypatch, tmp_path)
+    from api.models import Session
+
+    session = Session(
+        session_id="issue5532ctx",
+        messages=_four_turn_messages(),
+        context_messages=_four_turn_messages(),
+    )
+    session.save()
+
+    _call_clear(monkeypatch, "issue5532ctx")
+
+    loaded = Session.load("issue5532ctx")
+    assert loaded is not None
+    assert loaded.context_messages == []
+
+
+def test_clear_then_read_does_not_resurrect_state_db_messages(monkeypatch, tmp_path):
+    """After /clear, a subsequent /api/session read (the append-only state.db
+    merge) must return EMPTY — the cleared turns must NOT resurrect from
+    state.db.
+
+    On origin/master the watermark is None, so the merge keeps every state.db
+    row and the transcript reappears (the P0 data-loss symptom).
+    """
+    _seed_session_dir(monkeypatch, tmp_path)
+    from api.models import Session, reconciled_state_db_messages_for_session
+
+    session = Session(
+        session_id="issue5532resurrect",
+        messages=_four_turn_messages(),
+        context_messages=_four_turn_messages(),
+    )
+    session.save()
+
+    _call_clear(monkeypatch, "issue5532resurrect")
+    cleared = Session.load("issue5532resurrect")
+    assert cleared is not None
+
+    # state.db still holds the full pre-clear transcript (append-only backing).
+    state_db = [
+        _msg("user", "first", 1.0, "state-u1"),
+        _msg("assistant", "reply first", 2.0, "state-a1"),
+        _msg("user", "second", 3.0, "state-u2"),
+        _msg("assistant", "reply second", 4.0, "state-a2"),
+    ]
+    merged = reconciled_state_db_messages_for_session(
+        cleared, state_messages=state_db
+    )
+    assert merged == [], (
+        "cleared session must not resurrect state.db messages; "
+        f"got {[m.get('content') for m in merged]}"
+    )
+
+
+def test_clear_matches_truncate_to_empty_marker(monkeypatch, tmp_path):
+    """/clear and /truncate(keep=0) must set the SAME watermark/boundary marker
+    so the merge contract is not forked. Guards against a future divergence
+    where /clear grows its own bespoke watermark logic."""
+    _seed_session_dir(monkeypatch, tmp_path)
+    from api.models import Session
+    from api.session_ops import truncate_session_at_keep
+
+    # Reference: what truncate-to-empty produces on the same transcript.
+    ref = Session(
+        session_id="issue5532ref",
+        messages=_four_turn_messages(),
+        context_messages=_four_turn_messages(),
+    )
+    truncate_session_at_keep(ref, 0)
+
+    # Actual: what /clear produces.
+    session = Session(
+        session_id="issue5532match",
+        messages=_four_turn_messages(),
+        context_messages=_four_turn_messages(),
+    )
+    session.save()
+    _call_clear(monkeypatch, "issue5532match")
+    loaded = Session.load("issue5532match")
+
+    assert loaded is not None
+    assert loaded.truncation_watermark == ref.truncation_watermark == 0.0
+    assert loaded.truncation_boundary == ref.truncation_boundary == 0.0
+    assert loaded.messages == ref.messages == []
+    assert loaded.context_messages == ref.context_messages == []
+
+
+def test_clear_detaches_compression_snapshot_parent(monkeypatch, tmp_path):
+    """Codex gate (#5532): clearing a COMPRESSED-CONTINUATION child must not let
+    the pre-clear transcript resurrect via the compression-snapshot parent.
+
+    A compressed continuation keeps its archived transcript in a parent sidecar
+    marked pre_compression_snapshot; _webui_sidecar_lineage_messages_for_display()
+    stitches that parent back for display, merging the child with
+    truncation_watermark=None — so the 0.0 sentinel on the CHILD does NOT stop
+    the parent from resurrecting the cleared history on refresh. The fix detaches
+    the lineage (parent_session_id + compression anchor fields) on clear.
+
+    Fail-without-fix: on the pre-fix handler the child keeps parent_session_id,
+    so the display stitch re-adds the parent's archived messages after clear.
+    """
+    _seed_session_dir(monkeypatch, tmp_path)
+    from api.models import Session
+    import api.routes as routes
+
+    parent = Session(
+        session_id="issue5532parent",
+        messages=_four_turn_messages(),
+        pre_compression_snapshot=True,
+    )
+    parent.save()
+
+    child = Session(
+        session_id="issue5532child",
+        messages=[
+            _msg("user", "post-compression turn", 5.0, "u3"),
+            _msg("assistant", "post-compression reply", 6.0, "a3"),
+        ],
+        context_messages=[],
+        parent_session_id="issue5532parent",
+    )
+    child.save()
+
+    # Sanity: before clear, the display stitch DOES surface the parent snapshot.
+    pre = routes._webui_sidecar_lineage_messages_for_display(Session.load("issue5532child"))
+    assert any(m.get("content") == "first" for m in pre), (
+        "precondition: compressed continuation should stitch the parent snapshot"
+    )
+
+    _call_clear(monkeypatch, "issue5532child")
+
+    loaded = Session.load("issue5532child")
+    assert loaded is not None
+    assert loaded.messages == []
+    # Lineage detached so the stitch can't resurrect the parent transcript.
+    assert getattr(loaded, "parent_session_id", None) in (None, "")
+    assert getattr(loaded, "compression_anchor_visible_idx", None) is None
+    assert getattr(loaded, "compression_anchor_message_key", None) is None
+    assert getattr(loaded, "compression_anchor_summary", None) is None
+    # The load-time display path must return EMPTY — no resurrected parent rows.
+    display = routes._webui_sidecar_lineage_messages_for_display(loaded)
+    assert display == [], f"cleared compressed continuation must not resurrect parent; got {display}"
+
+
+def test_clear_preserves_ordinary_fork_parent_link(monkeypatch, tmp_path):
+    """#5532 Codex gate: clearing a genuine FORK child must NOT drop its
+    parent_session_id — only compression-snapshot parents are detached.
+
+    /api/session/branch stamps session_source="fork" and a parent_session_id so
+    the child nests under its parent and shows the "Forked from" indicator
+    (sessions.js:5720/5964/7105). The clear fix detaches the lineage ONLY when the
+    parent is a pre_compression_snapshot; an ordinary fork parent must survive.
+
+    Fail-without-fix: the unconditional parent detach dropped every parent link,
+    silently un-nesting forked sessions.
+    """
+    _seed_session_dir(monkeypatch, tmp_path)
+    from api.models import Session
+
+    parent = Session(
+        session_id="issue5532forkparent",
+        messages=_four_turn_messages(),
+        # NOT a compression snapshot — an ordinary fork source.
+    )
+    parent.save()
+
+    child = Session(
+        session_id="issue5532forkchild",
+        messages=[
+            _msg("user", "forked turn", 5.0, "u3"),
+            _msg("assistant", "forked reply", 6.0, "a3"),
+        ],
+        context_messages=[],
+        parent_session_id="issue5532forkparent",
+    )
+    child.session_source = "fork"
+    child.save()
+
+    _call_clear(monkeypatch, "issue5532forkchild")
+
+    loaded = Session.load("issue5532forkchild")
+    assert loaded is not None
+    assert loaded.messages == []
+    # The fork parent link MUST survive so the child still nests + shows "Forked from".
+    assert getattr(loaded, "parent_session_id", None) == "issue5532forkparent", (
+        "clearing an ordinary fork child must preserve its parent_session_id"
+    )
+
+
+def test_clear_survives_startup_recovery(monkeypatch, tmp_path):
+    """Codex gate #2 (#5532): a cleared session must stay cleared across a WebUI
+    restart. s.save() writes a pre-clear .json.bak (messages shrank to []), and
+    recover_all_sessions_on_startup restores any session whose .bak has MORE
+    messages than the live file — ignoring the live truncation_watermark==0.0.
+    Without dropping the stale .bak on clear, startup recovery resurrects the
+    cleared transcript (with a None watermark).
+
+    Fail-without-fix: on the pre-fix handler the .bak survives, recovery restores
+    it, and the reloaded session has messages again + watermark None.
+    """
+    models = _seed_session_dir(monkeypatch, tmp_path)
+    from api.models import Session
+    from api import session_recovery
+
+    session = Session(
+        session_id="issue5532recover",
+        messages=_four_turn_messages(),
+        context_messages=_four_turn_messages(),
+    )
+    session.save()
+    # Second save shrinking to nothing is what /clear does; but drive the REAL
+    # route so the .bak-drop path is exercised end to end.
+    _call_clear(monkeypatch, "issue5532recover")
+
+    session_dir = models.SESSION_DIR
+    live_path = session_dir / "issue5532recover.json"
+    # The stale pre-clear backup must be gone (so recovery can't undo the clear).
+    assert not live_path.with_suffix(".json.bak").exists(), (
+        "clear must drop the pre-clear .json.bak so startup recovery can't resurrect it"
+    )
+
+    result = session_recovery.recover_all_sessions_on_startup(session_dir)
+    assert result["restored"] == 0, f"startup recovery must not restore a cleared session; got {result}"
+
+    loaded = Session.load("issue5532recover")
+    assert loaded is not None
+    assert loaded.messages == []
+    assert (loaded.context_messages or []) == []
+    assert loaded.truncation_watermark == 0.0