Apply the existing truncation and checkpoint-order eligibility to marked
state.db image rows before insertion. Keep removed turns out of full,
limited, and model-context reads without changing no-watermark replay.
Preserve conflicting provider payloads for a uniquely proven retained
image row across Undo without restoring distinct removed rows.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Keep image-turn context tagging and upstream's typed steer-wrapper cleanup
in the same settlement path. Verify the combined recovery and streaming
behavior against current master.
Assisted-by: Hermes Agent:gpt-6-sol
Count a proved WebUI-owned pending row in the display baseline before
accepting a state.db self-heal. Without assistant output past that turn,
leave pending state for the run journal to restore partial text and the
interruption marker instead of clearing it prematurely.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
When a dead stream is reconciled from state.db, retain the submitted
prompt and attachment in the sidecar while keeping the Agent-enriched
row in model context. Use one parent snapshot for full-session conflict
projection so the sidecar owner is stable across the display merge.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
Record pending-turn proof under caller-owned session locks during normal
runs and credential recovery. Preserve both conflicting provider payloads
for replay, but display one sidecar-owned bubble when row identity and
visible content are unambiguous.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Persist trusted pending Agent timestamp evidence before the model call so
session reads, including metadata polls and cold reloads, keep only the
submitted prompt and attachment visible. Preserve conflicting provider
payloads without guessing which copy is newer.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
A unique durable row remains one visible row when its sidecar provider payload is stale. Use the state.db payload for replay, cover repeated recovery, and document the image-turn ownership exception.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
AGENTS.md asks for the documentation to move with a runtime-behaviour change, and docs/CONTRACTS.md points at the stable-assistant-turn-anchors RFC for anything that reconciles a live turn with the settled transcript. That is what this change adds, so the rule belongs there.
A new part of the Settlement section says why a preserved live node is not evidence of a running turn, the four facts that have to hold before one is dropped, that ownership is proved from state rather than by comparing two rendered trees, and that only the branches which ADD a turn are gated. It also records the settled-scene half: a prose row that duplicates the final answer, exactly or as a prefix within ten percent of its length, is dropped where both settled renderers take their rows, and live rendering is untouched.
Review follow-up for #7569: change 4 of the live-snapshot perf PR replaced
one quadratic with another. The window-free backward walk is correct, but it
re-walks the raw reasoning transcript's whitespace once per interim event —
on a whitespace-heavy transcript that span is megabytes, and the replay went
~55x slower than master on the production shape the gate measured
(1MB / 20-interim journal: 0.035s master → 1.922s branch).
## Thinking Path
The gate's finding was that the cost is structural, not a tweak: the matcher
consults the *whole* accumulated transcript on every interim event, so any
raw-text rescan of it is O(interim_count x whitespace_span). The fix is the
data-structure change the review asked for — maintain the folded view and the
raw cut offsets incrementally, and match/strip from that indexed state
instead of rescanning the raw buffer.
## What Changed
- `api/streaming.py`: new `_CompactEchoIndex` — an append-only folded index
that records, per non-whitespace character, its raw index. `append` folds a
chunk once (cost proportional to the chunk, never the buffer),
`matches_tail`/`cut_to` compare only the candidate against the folded tail
(O(len(candidate)), no raw walk), `reset` drops the state after the raw
text is truncated. The old raw helpers (`_find_compact_echo_suffix_start`,
`_strip_compact_echo_suffix`) stay as the equivalence oracle.
- `api/routes.py::_run_journal_live_snapshot`: reasoning chunks feed the
index as they are appended; `reasoning_echo_tail_matches` and
`strip_reasoning_echo_tail` consult the index. The strip re-indexes the
truncated transcript so later probes match against it.
- `api/streaming.py` live path: the same swap for the live-stream echo strip.
The three live reasoning buffers (STREAM_REASONING_TEXT, the coalescing
buffer, the per-message segments) each carry a mirrored index, fed on every
append and re-indexed after a strip. The buffer flush resets the buffer's
index with the text it drops, so the index never describes text that is no
longer there.
- `docs/rfcs/webui-run-state-consistency-contract.md`: the echo-match clause
now names the indexed representation and the scaling regression that pins
it.
## Why It Matters
Removing one quadratic by introducing another in the same hot path is the
exact risk the gate flagged — the headline benchmark used a journal shape
where the window-free walk's cost does not dominate, so the aggregate number
still looked like a win. The index keeps the original 8.9x and adds the
scaling guarantee on the shape that was regressing.
## Verification
- Measured on the reproduced shape (1MB whitespace span, 20 interim events):
0.972s → 0.049s. Doubling both span and interim count scales ~2x per step
(x1 0.041s / x2 0.090s / x4 0.175s) — linear, not quadratic.
- Behaviour equivalence: 300 randomized journals (whitespace-heavy,
Unicode, chunked, flagged/unflagged interims) rebuilt with the indexed
implementation and with the raw walk swapped in — sha256 of the canonical
snapshot JSON matched on all 300.
- `tests/test_compact_echo_index.py` (new, 6 tests): oracle equivalence over
2000+ randomized cases, chunked-append equivalence, matches_tail/cut_to
agreement, reset-after-truncation, and two cost probes that fail the raw
walk.
- `tests/test_live_snapshot_echo_scan_scaling.py` (new, 4 tests): the
proportional-scaling regression the review asked for — verified to FAIL on
the pre-fix matcher (14.6x time for 4x rows, the quadratic signature) and
pass on the indexed one.
- Existing echo/journal suites unchanged and green: 19 items across
`test_compact_echo_suffix_linear_scan` / `test_live_snapshot_echo_dedup` /
`test_live_snapshot_single_parse`, plus 66 neighbouring items
(`test_issue1361_cancel_data_loss`, `test_issue2565_reasoning_accumulation`,
`test_issue2713_streaming_segment_flush`, `test_cancel_interrupt`,
`test_run_journal`).
## Contract Change
None. The matcher's observable behaviour is unchanged (same cut points, same
whitespace-insensitive semantics); only its cost profile changes. No config
knobs, no API surface.
## Risks
- Index drift: if a future writer mutates one of the reasoning buffers
without feeding (or resetting) its index, the matcher would silently miss
echoes. Mitigated by keeping the index feed adjacent to every existing
write site and by the reset-on-flush/reset-on-strip symmetry; the
equivalence test would catch a dropped feed as a missed echo.
- Memory: the index stores one char plus one int per non-whitespace
character. On a whitespace-heavy transcript this is bounded by the folded
length, which is what the transcript would cost folded anyway; the raw
text is still the larger object.
## Model Used
Hermes Agent (step-5-preview) with direct file edits and pytest verification.
Cost properties were measured with a purpose-built scaling probe (proportional
span x interim count), not a fixed-size fixture; the probe is kept as a
regression test rather than a script.
## Related
#7569 (this PR), #7310 (slow /api/session under load), #7534 (incremental
journal read cache — complementary), #7273/#7291 (metering write-side skip).
Keep an older persisted unread-clear ordering record when quota rejects its
higher replacement, so reload cannot lose the last durable tombstone. Replace
unread source-shape assertions with Node-driven behavior checks against the real
sessions.js functions.
`_strip_oob_markers_from_messages()` matched on marker presence alone and passed
the whole row to `_strip_oob_blocks()`, so a user row that legitimately quotes a
complete `[OUT-OF-BAND USER MESSAGE]…[/OUT-OF-BAND USER MESSAGE]` example lost
that text on the next settlement — and the shortened pre-match row broke
active-turn identity matching, which duplicated the turn.
The transport only ever appends a consumed `/steer` wrapper to the turn's last
tool result, so the scrub now considers `role == "tool"` rows only and rebuilds
just their `content` (shallow copy): user/assistant rows, stable ids, reasoning
metadata, and turn bookkeeping are never touched.
Measured on the focused file, helper restored to the pre-fix body as sabotage:
- pre-fix body: 1 failed, 7 passed — the new regression prints the deleted
example plus 3 user rows instead of 2 (the duplicated turn)
- with the fix: 8 passed
Neighbors: test_context_history_sanitization.py 20 passed; writeback/settle
family (12 files) 120 passed, 5 skipped; media slice (7 files) 197 passed,
18 subtests.
Docs: docs/rfcs/webui-pending-intent-controls.md now records that the transport
wrapper is dropped from the settled display and the persisted context state, is
scoped to the carrier row, and that a transcript stored before this behavior
heals on its next settlement.
When both the store and the cache held a marker for one session, the merge
chose between them by completed_at while the clear comparison used unread_order.
Two completions in the same millisecond therefore kept the earlier marker and
its message count and cron profile metadata.
Compare unread_order on both sides, matching the ordering the rest of the unread
state uses, and record in the RFC that a deletion record's expiry can let a very
long-lived tab re-add a count for a session deleted more than seven days ago —
bounded and invisible, since that session is no longer listed.
Two defects share one cause: a fact was being discarded because a session was
missing from the visible list. The sidebar filters by profile, project, and
source, so an absent row may simply be hidden.
Clear records were pruned on session existence, which deleted the ordering
fact for any session outside the current filter and re-lit a dot the user had
cleared. They are now pruned by age and supersession only.
The same reasoning rules out pruning stored viewed counts by list membership,
so a deletion records its own key instead. A client that still caches the
acknowledgement now prunes it rather than writing it back.
The previous clear representation is imported once and then removed. Keeping
it would preserve both of its defects: concurrent clears could replace one
another in the shared blob, and it grew with every session ever cleared.
The run state consistency contract is where sidebar metadata work routes, and its
state-layer table already lists the sidebar layer, but nothing recorded the rules
the two client-side unread stores actually follow: viewed counts converge by a max
merge, completion markers are ordered by per-session tombstones with a bounded
retention window, and the storage listener repairs the store instead of only
dropping the local cache.
Write those rules down where they belong, naming the three localStorage keys and
the retention cap, and state the limitation explicitly: each save is a
read-modify-write across separate localStorage operations with no lock between
them, so the contract is convergence rather than atomicity, and two concurrent
clears can still lose a tombstone.
tests/test_webui_run_state_consistency_contract_rfc.py pins the documented keys
and the documented retention window to the constants in static/sessions.js, so
changing one without the other fails. It fails on the previous revision and passes
with this change.
* fix(tests): make mtime_invalidation and glm_5_3 tests order-independent (#7100)
* fix: expose max reasoning for GPT-5.6 models
* fix(recovery): do not reattach cancelling runs
ACTIVE_RUNS tracks worker lifecycle, which is deliberately broader than
"a turn a browser may attach to". cancel_stream() keeps the row as
phase="cancelling" while the worker unwinds so a successor turn cannot
start on top of it, but the client has already reached a terminal state
for that stream: its run journal ends in a terminal event.
The recovery lookups treated every same-session ACTIVE_RUNS row as
attachable. An idle session holding a cancelling row therefore received a
recovered server_turn_started on every /api/session/stream subscription:
the client attached, replayed the terminal event, tore the renderer down,
resubscribed, and the server replayed the same frame again. The result is
an endless attach/replay loop that rebuilds the transcript repeatedly.
Separate the two meanings instead of narrowing one call site:
- api/config.py gains active_run_is_attachable() and
active_run_cancel_is_stale() as the shared predicates.
- active_stream_id_for_session() (browser recovery) returns attachable
rows only; _session_has_active_turn() (busy check) keeps counting a
fresh cancellation so a successor cannot overlap the unwinding worker.
- _live_active_stream_id() applies the same rule to the hidden-tab
status poller, on both the STREAMS and ACTIVE_RUNS paths.
- routes._cancelled_run_is_stale() now delegates to the shared predicate
rather than keeping a parallel copy of the staleness rule.
- A cancelling row past a bounded unwind window with no live STREAMS
channel is reclaimed from ACTIVE_RUNS and its stream owner released, so
a wedged worker cannot suppress background wakeups forever. Age alone
does not reclaim a row that still owns a live channel.
Staleness anchors on cancelled_at, falling back to started_at, so a
long-running turn cancelled moments ago is never treated as an orphan.
tests/test_cancelling_run_not_attachable.py covers both directions of
each rule. The six behavioral tests fail on the unpatched tree and pass
with the fix; two targeted mutations (forcing the attachability predicate
true, and disabling the staleness reaper) each turn the suite red.
* docs(rfc): document cancellation attach/admission split
Records the runtime contract introduced by the recovery fix in the WebUI
run-state consistency RFC, so the distinction is discoverable instead of
living only in code comments.
- Adds ACTIVE_RUNS to the State Layers table as the worker-lifecycle
registry, explicitly not the set of runs a browser may attach to.
- Adds invariant 9: lifecycle-busy is not client-attachable. Cancellation
splits the two meanings, recovery paths must exclude cancelling rows,
and admission checks must keep counting them.
- Documents the bounded cancellation-unwind window: reclamation needs
both age and the absence of a live STREAMS channel, and staleness is
anchored on the cancellation timestamp.
- Extends the review checklist with the admission-vs-attachment question
and the evidence required when changing a reclamation window.
* docs(changelog): note #7096 cancelling-run reattach, #7083 GPT-5.6 max reasoning, #7101 test order-independence
---------
Co-authored-by: webtecnica <webtecnica@gmail.com>
Co-authored-by: Abdulrahman Elkenany <boudy.elkenany123@gmail.com>
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
Co-authored-by: n <a@n>
The RFC (docs/rfcs/session-sse-contract-v1.md) cited hardcoded absolute
line numbers like `api/routes.py:12345-12346` and `api/routes.py:16177`,
and tests/test_issue4812_session_sse_contract_rfc.py validated those exact
lines against source. Any PR that shifted routes.py lines (e.g. #5543,
#5534) broke the test — a chronic brittle-failure class (#5542).
Root fix: anchor by SYMBOL, never by line number.
RFC:
- Strip every `api/routes.py:<NNNN>` / `<NNNN>-<NNNN>` and
`api/streaming.py:<NNNN>` line-number suffix; keep the symbol name
(route string, function/constant names) + the file, e.g.
"_handle_session_events_stream() in api/routes.py".
- Update the inventory note to state names are the stable anchors and the
RFC deliberately avoids line numbers so a source-layout shift can't
invalidate the doc or its contract test.
Test:
- Rewrite the two line-anchor tests to assert (a) each cited symbol still
exists in api/routes.py and (b) the RFC names that symbol — dropping the
`api/routes.py:<line>` regex bounds-checks entirely. Invariant preserved:
"the RFC's cited symbols are real and named in the doc", NOT "the cited
line numbers are exact".
- Add test_rfc_uses_no_hardcoded_routes_line_numbers as a regression guard
that fails if any `*.py:<line>` anchor is reintroduced into the RFC.
Proven de-brittled: prepending 31 lines to api/routes.py (scratch, reverted)
leaves all 34 tests green.
Refs #5542, #5513
nesquena-hermes