Cancelling a running turn briefly collapsed the already-streamed reasoning,
tool cards, and text to a bare "Task cancelled" marker before a follow-up
GET /api/session repopulated them — a visible race between the terminal
cancel SSE event and the refetch (#4076, #1361).
The terminal cancel event now carries a canonical, redacted session snapshot
captured under the per-session agent lock right after the cancelled turn is
persisted (including the recovered partial-assistant message). The frontend
applies that embedded snapshot immediately via _applyCancelSessionPayload and
skips the GET round-trip. The snapshot is a fully detached deep copy (redaction
rebuilds every container) so it cannot alias or be staled by concurrent writes;
it is only applied when its session_id matches the active view; and a
missing/failed snapshot degrades cleanly to the existing GET fallback.
The seven other cancel call sites are standardized via _cancel_event_payload()
(message + type + status), backward-compatible (no session key when none).
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
A user prompt that exists only in Hermes state.db (no sidecar copy) and falls
chronologically BEFORE a newer sidecar message tail was being appended after the
tail (or skipped) by merge_session_messages_append_only — losing or mis-ordering
it on load. The merge now inserts such a state.db user message at its correct
chronological position via _insert_state_message_chronologically, with a
role-aware tie-break for equal timestamps. Append-only invariant + dedup
(seen_content_keys) preserved.
Deterministic-ordering guard (maintainer add, addresses Codex review): on an
equal-timestamp collision the insert no longer lands at a slot whose left
neighbour shares the message's role — it advances past the same-role
equal-timestamp run, so an already-answered earlier turn can't be re-ordered
into user,user,assistant. (The agent core merges adjacent users before send, so
this was benign in practice, but the guard keeps the merged transcript correctly
ordered regardless.) +1 same-second multi-turn regression test, toothless-checked.
Co-authored-by: Dennis Soong <dso2ng@gmail.com>
OSError is too broad — it masks real errors like file-not-found.
ssl.SSLError specifically catches SSL-level disconnects without
swallowing unrelated OSError subtypes.
Closes the test_excludes_broad_oserror CI failure.
Two non-functional cleanups from the second Copilot pass:
1. The inline comment in `test_error_marker_no_preserved_as_draft`
said the legacy "user message above was preserved" wording was used
for the post-retry-give-up case. The actual implementation demotes
give-up markers to a different neutral wording ("Partial output may
have been lost."). Comment rewritten to match the contract.
2. The regression test `test_lost_response_recovered_on_second_read`
declared a `monkeypatch` parameter it never used. Dropped.
Four code-review comments from the automated Copilot reviewer on this PR:
1. `_journal_tool_already_present` dedupe was session-wide, so a
legitimately-repeated tool (e.g. a second `terminal: ls` in an
earlier turn) could cause the retry path to falsely skip
materializing the recovered tool card. The helper now takes a
keyword `stream_id` argument; when supplied, a tool card whose
`_recovered_stream_id` is set AND differs from the candidate is no
longer treated as a duplicate. Untagged tool cards (live tools, or
tool cards carried over from a pre-tagging core transcript) still
match, preserving the existing 'core transcript already has this
tool, don't duplicate' invariant. Two new tests in
`TestJournalToolDedupeScoping` cover both legs of the rule.
2./3. The troubleshooting FAQ pointed at `~/.hermes/webui/sessions/session_<sid>.json`
and `~/.hermes/_run_journal/...`. The actual sidecar filename has
no `session_` prefix and the run-journal lives under the WebUI
sessions dir (`~/.hermes/webui/sessions/_run_journal/<sid>/<stream>.jsonl`,
default). Both paths fixed and an explicit note added about
`HERMES_WEBUI_STATE_DIR` overriding the state root.
4. Drop unused `json` / `queue` / `Path` imports from
`tests/test_session_lost_response_regression.py` so the file stops
carrying noise that future linting would flag.
Reproduces the production failure mode:
1. Stage 1 — sidecar repair runs while the run-journal for the dead
stream is empty on disk. Assert the marker arms the lazy-retry
hook (`_pending_journal_recovery=True`,
`_journal_retry_stream_id`, `_journal_retry_attempts=0`,
`_journal_retry_first_seen_ts`) and does NOT carry the legacy
"no agent output was recovered" wording. Pending sidecar fields
are cleared regardless.
2. Stage 2 — journaled token / tool / tool_complete / token events
appear on disk. Call `get_session(sid)` and assert the marker
self-heals: wording promotes to "recovered from the run journal",
journaled assistant rows + tool card land above the marker in
chronological order, all retry meta is stripped.
Without the lazy-retry path this test fails at the very first
assertion (marker still carries the legacy no-output wording).
