Both gateway SSE read loops used urlopen(timeout=600) and re-checked cancel only
BETWEEN lines, so a half-open gateway (TCP open, zero bytes) pinned the worker
for the full 10 minutes and ignored Stop.
Route raw-line iteration through _iter_sse_lines_cancellable and bound the socket
to a byte-silence budget (_gateway_read_timeout_secs, default 120s, env
HERMES_WEBUI_GATEWAY_READ_TIMEOUT). A read timeout is TERMINAL — CPython's
socket.makefile latches _timeout_occurred on the first socket.timeout, after
which every read raises a bare OSError, so the connection can't be resumed. On a
read timeout (or that poisoned-socket OSError) the generator surfaces the user's
Stop (yield b"" -> caller emits cancel) or re-raises so the caller reports the
stall. A stream that keeps emitting bytes within the window never trips it, so a
genuinely alive (if slow) turn is untouched; only >budget of total byte-silence
is treated as a dead gateway. Both loops (runs-API + legacy) changed identically.
The test fake now models the timeout-poison latch (a bare OSError after the first
timeout) so it reflects real socket semantics; the previous stateless re-raise
had made a multi-window "recovery" path look green that cannot occur in
production.
Refs #2476
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WebUI keeps two parallel arrays per session: messages (display transcript) and
context_messages (what's sent to the model). In large/compacted sessions they
diverge. Forking / in-place truncation copies a prefix of both and relies on
truncate_context_for_display_keep() to translate a display keep-index into the
matching context index — an aligner that prefers a stable per-row `id`. But the
model-context rows carried neither id nor timestamp (a 989-session live scan
found 0 with any id), so alignment fell back to fragile content-signature
matching that goes ambiguous on repeated tool calls / empty assistant turns.
This closes the DATA gap (the alignment logic itself already prefers `id` and
handles the compacted case via the matcher, shipped in #5563): mint a monotonic,
session-unique integer `id` on the per-turn result rows AFTER the context restore
and BEFORE both arrays are built, so the display and model-context copies of a
logical row share the same id.
- api/streaming.py: new _assign_stable_message_ids(); _restore_reasoning_metadata
carries `id` forward across turns (as it already does timestamp); wired into all
three streaming commit sites (main turn, retry, self-heal).
- api/routes.py: same mint on the runs/MoA _handle_chat_sync commit path.
- api/gateway_chat.py: mints ids on the two new rows of a gateway turn.
- The `id` is stripped before the provider API call (not in _API_SAFE_MSG_KEYS),
same treatment as timestamp — nothing new reaches the provider.
session_ops.py alignment was already delivered by #5563 (id-preferring matcher +
compacted-case fall-through), so no change is needed there now; tests updated to
assert the post-#5563 behavior (id-bearing = exact cut; id-less = errs toward
under-keeping, never the old raw-index mis-cut).
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Gate hardening for #5198: the new stream-owner map leaked entries on three
paths that bypass the teardown finally — worker early-return when the stream
was cancelled before startup (streaming.py, gateway_chat.py) and the direct
ACTIVE_RUNS stale-zombie prune (routes.py). Each now calls unregister_stream_owner.
Adds a regression test proving the early-return path no longer leaks.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
When a turn is steered, the agent embeds an
[OUT-OF-BAND USER MESSAGE]...[/OUT-OF-BAND USER MESSAGE] control block, and that
block gets persisted into the session's messages/context_messages (appended onto
a tool-result entry — verified present in real prod session JSON, in both
messages[] and context_messages[]). WebUI then replays context_messages into the
provider-facing conversation history every turn, so the model kept re-seeing the
internal control marker: wasted tokens + stale control data.
Adds a non-mutating, recursive _strip_oob_blocks() (regex, DOTALL+IGNORECASE,
non-greedy) wired into every model-facing chokepoint:
_sanitize_messages_for_api, _api_safe_message_positions,
_restore_reasoning_metadata projection, and the gateway runs-API
conversation_history (context_messages + prefill). The compression path
(/session/compress) inherits the strip via _sanitize_messages_for_api.
The marker stays intact in the persisted transcript (display/reload/audit
fidelity); it's removed only on the way to the provider. Idempotent, only strips
a complete open+close pair, and does not mutate persisted session objects (the
content key is reassigned to a freshly-built value).
Maintainer hardening (Opus gate): relaxed the opening pattern from a mandatory
`]\r?\n` to `]\s*?` so a single-line OOB block is also stripped (DOTALL .*?
already spans newlines) — robustness against a format that omits the newline.
Co-authored-by: Stacey2911 <Stacey2911@users.noreply.github.com>
The legacy /v1/chat/completions approval handler (added for #4549) rendered
the approval card but never populated _STREAM_RUN_IDS, so /api/approval/respond
could not relay the user's choice back to the gateway to resume the parked run
— it fell through to the local approval path and returned {ok: false}. Record
the run_id from the approval payload (no-op when absent), mirroring the runs-API
path. Adds a behavioral regression test that fails on the pre-fix head.
Codex ship-gate finding (CORE). Co-authored-by: rodboev <rodboev@users.noreply.github.com>
The WebUI gateway chat backend routed every browser chat turn through the
gateway's /v1/runs API whenever the gateway advertised approval/runs
support (gateway_chat.py capability gate). The runs API creates an isolated
Api_Server session per turn, so mid-conversation context was lost between
messages. Routing through /v1/runs is now gated behind an explicit opt-in
(_gateway_use_runs_api_enabled: HERMES_WEBUI_GATEWAY_USE_RUNS_API env var or
webui_gateway_use_runs_api config key, env-over-config precedence, default
off). The capability gate now requires BOTH the opt-in flag AND gateway
approval support before using /v1/runs; otherwise normal chat stays on
/v1/chat/completions, preserving the session via X-Hermes-Session-Id.
Operators who want the in-gateway-chat approval-events flow re-enable it
with the flag.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Approve/deny on a remote-gateway backend now forwards to the gateway runs API
(run_id server-derived, existing auth+CSRF preserved, capability-probed + 60s
cached). Non-gateway/local approval path unchanged — relay only fires when a
session has an active gateway run (_STREAM_RUN_IDS). Old gateways get a clear
upgrade message instead of a silent no-op.
Co-authored-by: rodboev
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
* Harden interrupted recovery control filtering
* Redesign live-to-final assistant replies
* Fix live activity anchor test fixture
* Fix CI lint issues for live reply tests
* Strengthen live progress prompt contract
* Recover PR #3401 refresh on origin/master
* Repair live-to-final refresh regressions
* Fix live worklog refresh regressions
* Show live footer timer on initial stream start
* Restore live stream shell after reload
* Preserve per-frame live SSE replay cursors
* Preserve reasoning as Worklog Thinking cards
* Quiet Worklog Thinking card styling
* Align Worklog Thinking card styling
* Scope live Worklog Thinking cards by segment
* Suppress exact duplicate settled Thinking
* Close#3401 merge review test gaps
* fix(#3401): resolve 4 deep-review regressions (inline-think, reconnect-dup, neon skin, busy-gate worklog)
Deep review (Codex diff-vs-master + live-browser drive) of the live-to-final refactor
surfaced 4 regressions vs master that the rewritten suite no longer guarded:
1. Inline <think>…</think>answer reasoning vanished — _assistantReasoningPayloadText
used $-anchored regexes so a leading think block + visible answer extracted nothing
and the Thinking card never rendered. Removed the 3 $ anchors to match the
(non-anchored) display stripper. Live: inline-think thinking-only turn now renders.
2. (CORE) reconnect/reload duplicated the live reply — _rememberRunJournalCursor advanced
a closure-local seq but never wrote INFLIGHT[activeSid].lastRunJournalSeq, so a reload
replayed the journal from after_seq=0 over restored lastAssistantText. Now mirrors the
cursor onto INFLIGHT + schedules a throttled persist.
3. Neon skin silently broke — PR deleted the :root[data-skin="neon"] CSS but left Neon in
the picker. Restored the neon CSS block from master.
4. Settled tool-worklog rebuild gated purely on !S.busy — dropped every prior settled
turn's worklog when renderMessages re-ran during an active stream (switch-back to an
in-progress session). Restored master's !S.busy || (S.toolCalls && S.toolCalls.length).
Live: busy re-render now preserves tool cards (4→4, was 4→0).
Live-verified all 4 + confirmed #3709/#3592 invariants still hold (1 thinking card, none
below footer; distinct siblings preserved). + tests/test_issue3401_deep_review_fixes.py (7).
* test(#3401): realign 3 stale source-shape assertions to the deep-review fixes
Fix commit changed two source literals that existing stage tests scanned for:
- test_live_activity_timeline.py (x2): split anchor 'if(!S.busy){' → the restored
'if(!S.busy || (S.toolCalls&&S.toolCalls.length)){' guard (fix 4).
- test_run_journal_frontend_static.py: 'after_seq=0' not in source — fix 2's comment
contained that literal; rephrased the comment to 'the zero floor (after_seq of 0)'.
Intent of all three assertions unchanged; only the matched string updated. No code
behavior change.
* docs(changelog): v0.51.294 — Release JJ (stage-3401, #3401 live-to-final redesign)
---------
Co-authored-by: Frank Song <franksong2702@gmail.com>
Co-authored-by: Nathan-Hermes <nesquena-hermes@users.noreply.github.com>
Co-authored-by: nesquena-hermes <[email protected]>
## Release v0.51.262 — Release ID (stage-r12)
Phase-3 light slice (no-screenshot items) — 3 PRs.
### Fixed
| Issue/PR | Author | Fix |
|----------|--------|-----|
| #3432 (#3532) | @franksong2702 | Normalize the recall-prefill terminal `user` turn so WebUI doesn't send adjacent `user` roles to strict chat templates (Mistral/Gemma/Jinja). `_normalize_prefill_messages_before_user_turn()` in both `streaming.py` + `gateway_chat.py`; drops only the terminal user tail, preserves assistant/system/mid-list context. **Rebased onto master** (was CONFLICTING). |
| #2558 (#3516) | @rodboev | "Reveal in file manager" now translates container workspace paths (`/workspace`) back to the host mount path for Docker deployments (traversal-safe via `safe_resolve` + sibling-prefix guard). |
### Changed
| Issue/PR | Author | Change |
|----------|--------|--------|
| (#3539) | @Lyr-GW | Completed the Chinese (Simplified) `zh` localization (MCP controls, tool-list pagination) with all interpolations preserved, and the language dropdown now applies the locale **instantly** on change. |
### Review fix absorbed (Codex)
#3539 also added an `allowed=['en','zh']` filter to the Settings language dropdown, which dropped the 9 other shipped locales (it/ja/ru/es/de/pt/ko/fr/tr) — and since save falls back to `en` when the select has no matching option, an existing user of those locales would be **silently reset to English** on a Settings save. Removed the filter (dropdown enumerates all `LOCALES` again, matching master); partially-translated locales fall back per-key to English at render. + regression test `test_issue3539_language_dropdown_all_locales.py`.
### Gate
- Full pytest suite: **7701 passed, 0 failed**
- ESLint: CLEAN · ruff: CLEAN · browser-smoke: CLEAN
- Codex (regression): SHIP-ONLY-WITH-FIXES (dropdown drops-locales) → fixed → **SAFE TO SHIP** (verified prefill drops only terminal user tail in both paths, Docker path-translation traversal-safe, zh interpolations preserved)
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Co-authored-by: Lyr-GW <Lyr-GW@users.noreply.github.com>
#3324 moved the WebUI session/delivery context from a prefill user message into
_webui_ephemeral_system_prompt, but the gateway-backed WebUI chat path built its
own system message from the bare _WEBUI_PROGRESS_PROMPT + prefill helper — which
now returns only recall prefill. That silently dropped connected platforms, home
channels, delivery hints, and session framing on Gateway-routed WebUI chats
(same #3278 data-drop class, on the path the author missed). Build the gateway
request's system message with _webui_ephemeral_system_prompt(surface_context,
config_data=cfg) so the context is preserved. Regression test asserts the
delivery context is present in the gateway system message.
Backfill model-context turns into the visible transcript before saving the
latest reply while keeping hidden [context compaction] markers out of the
visible transcript, so sidebar/header message counts no longer collapse a
compacted gateway session to a 2-message conversation. Resolves the
transcript-loss regression the v0.51.197 Codex gate flagged on the original.
Co-authored-by: AJV20 <24819659+AJV20@users.noreply.github.com>
Frank Song