Keep image-turn context tagging and upstream's typed steer-wrapper cleanup
in the same settlement path. Verify the combined recovery and streaming
behavior against current master.
Assisted-by: Hermes Agent:gpt-6-sol
The suppression predicate _transcript_already_advanced_past_pending()
required a run-journal terminal 'completed' state, but its caller in
_apply_core_sync_or_error_marker already returns early on that same
condition. The predicate branch was therefore unreachable dead code: it
could never suppress anything, and stale-cancel recovery could still
append a duplicate recovered user row, a journal clone and an
interruption marker for a turn that actually finished.
Derive the completion evidence from the crash-safe turn journal instead:
_turn_journal_records_completion() reports True when the exact-stream
'completed' event is the latest terminal state for the pending stream id.
The run journal's terminal write is not durable evidence for this path
(it is consumed by the caller's early return), so the turn journal is the
only still-observable source. A cancelled/crashed turn (interrupted)
keeps recovery active, and completion for another stream of the same
session never settles the pending one.
Also record the same completion event on the Gateway success writeback
path, so Gateway-backed runs have the equivalent completion persistence
instead of leaving the run journal's lost terminal write unrecoverable.
Regression coverage:
- test_turn_journal_helper_reports_completed_for_matching_stream
- test_turn_journal_helper_rejects_interrupted_turn
- test_advances_past_pending_from_turn_journal_without_run_journal_terminal
- test_full_recovery_suppresses_duplicates_on_turn_journal_completion
- test_full_recovery_appends_rows_on_turn_journal_completion_loss
- test_gateway_chat_worker_records_turn_journal_completion
Reverting the predicate gate turns the two new CORE regressions red;
reverting the Gateway write turns the Gateway regression red.
Keep whitespace-normalized text matches for plain Agent rows, retain float timestamps for Gateway turn checkpoints, and test optional attachment suffix behavior rather than source spelling.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Preserve the submitted image turn as the display owner while retaining rich Agent context for model replay and suppressing its durable scalar mirror from recovered WebUI transcripts.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
Settle removed approval producers before fallible projection work, and fail-close every producer owned by a terminal gateway run across completion, failure, cancellation, and teardown while preserving unrelated runs.
The cache-only Steer path only special-cased backend == "gateway" and sent
every other value — including a missing, empty, or unknown backend — to the
cached local agent. Another runtime could own the run while guidance was
enqueued on a stale local agent.
Fail closed instead: the in-process worker now registers its active run with
the explicit local backend tag (WEBUI_LOCAL_CHAT_BACKEND, the same "legacy"
value webui_chat_backend_mode() returns), and cache-only Steer only enqueues
agent.steer() on that tag. Gateway keeps its gateway_steer_queued outcome; a
missing, empty, or foreign backend returns stream_dead without steering.
Extends the positive-ownership regression with backend cases and adds a
worker-side check that the local backend tag is actually registered.
Closes the remote POSIX workspace path preservation gap across Session and streaming. Profile-aware runtime path resolution.
Co-authored-by: alfred-rootson <alfred.wojtasordzik@gmail.com>
* fix(streaming): lock-disciplined STREAMS reads via peek_stream()
Writers pop STREAMS under STREAMS_LOCK, but four hot reads used bare
STREAMS.get(): the streaming worker start, the gateway run start, and the two
SSE attach paths in routes.py. Each is individually saved today by GIL-atomic
dict.get plus a None-guard, but the check-then-act windows (get → register,
get → subscribe) race teardown pops, and the bare pattern invites unprotected
copies. peek_stream() centralizes the locked read; all four sites migrate with
no behavioral change.
* test: dual-patch config.STREAMS alongside routes.STREAMS in SSE seam tests
peek_stream() reads the canonical api.config.STREAMS registry; tests that
inject a fake registry by patching only the routes-level re-export left the
helper reading the real (empty) registry and hung the SSE loop to the
pytest-timeout ceiling. Mirror the test_issue6623 pattern: patch both
bindings.
* docs(changelog): stamp #7339 lock-disciplined STREAMS reads
---------
Co-authored-by: Sebastian B <sb@wmc.sh>
Co-authored-by: n <a@n>
* fix(streaming): route image input mode by the session's actual provider/model
Image routing (native embed vs vision_analyze text pipeline) previously
resolved the provider/model from the global config default only. A session
switched to a text-only model (e.g. deepseek-v4-flash) still had uploaded
images forwarded natively to a model that cannot see them.
Thread the session's resolved provider/model (active_provider/active_model)
and its pre-canonicalization identity (requested_provider) through
_resolve_image_input_mode, _build_native_multimodal_message,
_sanitize_messages_for_api, and both gateway chat paths. When no session
identity is present the behaviour is unchanged (global default).
Tests: upstream parity, session-level native/text routing, requested_provider
capability lookup, and multimodal/sanitize forwarding.
* test: use generic provider/model names in fixtures and docstrings
Replace real provider/model names (custom:newapi, jd-kimi-k2.6,
stepfunplan, step-3.7-flash, deepseek-v4-flash) with neutral placeholders
(mygateway, my-vision-model, my-text-model, my-vision) so the test suite
does not reference any particular deployment's configuration.
test_active_vision_model_routes_native now passes requested_provider to
keep the assertion independent of models.dev capability data.
* fix(streaming): preserve session provider identity across auth-heal retry
Both credential-healing retry branches unconditionally overwrote
_session_requested_provider with the already-canonicalized resolved_provider,
so a named custom:slug session lost its exact vision-capability entry after an
auth-heal retry -- a text-only custom model would then wrongly RESTORE the
historical image_url parts it correctly stripped on the first attempt.
Make the heal-path assignment conditional (initialize only when empty),
preserving the pre-canonicalization identity captured at first resolve.
Regression test asserts both arms: identity preserved -> images stripped;
identity lost -> images kept (the bug).
Addresses Codex gate CORE finding + greptile P1 on #6882.
Co-authored-by: happy5318 <happy5318@users.noreply.github.com>
* test(vision-routing): skip capability-based tests when hermes-agent absent
The capability-derived 'text' verdict delegates to agent.image_routing
(the single source of truth), which is NOT installed in the WebUI standalone
CI environment -- there _resolve_image_input_mode falls back to native, so the
7 tests asserting a capability-based text verdict failed on CI (they only ever
passed locally where the agent package is on PYTHONPATH).
Guard them with a module-level agent-availability probe + skipif (the same
pytest.importorskip pattern used across the suite for agent-dependent tests):
they run fully where the agent is present and skip honestly where it is not.
Agent-free tests (native carve-out, explicit-signal fallback) still run on CI.
* test(vision-routing): drop unused imports (base64, os, TemporaryDirectory)
Pre-existing F401 unused-imports that ruff's diff-vs-master flagged once the
import block entered the changed-line scope. Path is retained (used in the
_make_png / _normalized_attachments type annotations).
---------
Co-authored-by: happy5318 <happy5318@users.noreply.github.com>
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
Prevents a delayed cancelled streaming worker from writing back stale output over a completed successor's result, and clears the Gateway writeback-owner entry on teardown/pre-start-cancel to avoid a slow SESSION_WRITEBACK_OWNERS leak.
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
A provider-qualified model string (@provider:model) is deliberately kept
intact for internal routing (#1253), but three places on the way to the
Hermes gateway mishandled it:
* _clean_session_model_provider returned the whole "provider:model"
value as the provider, so the gateway saw a provider named
"ollama-cloud:deepseek-v4-flash" -> "Unknown provider".
* _session_model_state_from_request extracted the provider correctly
but left the full "@provider:model" string as the model value.
* both gateway request builders sent that string verbatim as
body["model"], which the upstream provider API 404'd on.
Rather than add positional colon-split heuristics, route all three
through config._parse_provider_qualified_model_id() -- the existing
shared grammar that already distinguishes a multi-segment custom
provider ID (custom:<slug>, custom:<host>:<port>) from a model segment
carrying its own colon tag (:free, :0731). A positional split cannot
tell those two shapes apart, so any single split direction breaks the
other; the shared parser is the only thing that gets both right.
_split_provider_qualified_model now delegates to that parser too, which
fixes the same class of truncation at all of its call sites in routes.py
instead of only the two reported ones, and guarantees the provider used
for the routing decision matches the model sent in the request.
Fixes#6722
* fix(#6389): remove clear_offline_buffer() teardown blocks that cause data loss on late reconnects
Keep the real leak fixes:
- unregister_stream_owner() in both streaming.py and gateway_chat.py
- AGENT_INSTANCES.pop() in gateway_chat.py
The clear_offline_buffer() calls could clear a channel's offline buffer
after a late reconnect already fetched the channel but before it subscribed,
causing the reconnecting client to lose all buffered events (including
stream_end) and hang indefinitely on heartbeats.
STREAMS.pop() already releases registry ownership; GC reclaims the buffer
once the last outstanding channel reference is dropped, correctly preserving
replay for in-flight reconnects.
* docs(changelog): stamp #6389 stream-teardown leak fix
---------
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: nesquena-hermes <agent@webui>
* fix: catch unhandled exception in POST /api/updates/check (defensive hardening) (#6180)
* fix: generate non-empty approval_id when gateway approval.request omits it (#6008) (#6168)
* chore: mark update-check try/except as defensive-only guard, drop #6086 linkage
Per maintainer review, the try/except wrapper is defense-in-depth only —
it does NOT fix#6086 (root cause is signal/process-group reaping).
Updated log message and added inline comment to make this explicit.
Leave #6086 open.
* test: isolate state-dir probes from user state
* docs(changelog): stamp #6168 approval_id, #6180 update-check guard, #6305 test isolation
---------
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
Co-authored-by: pxxD1998 <214340659+pxxD1998@users.noreply.github.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Both gateway SSE read loops used urlopen(timeout=600) and re-checked cancel only
BETWEEN lines, so a half-open gateway (TCP open, zero bytes) pinned the worker
for the full 10 minutes and ignored Stop.
Route raw-line iteration through _iter_sse_lines_cancellable and bound the socket
to a byte-silence budget (_gateway_read_timeout_secs, default 120s, env
HERMES_WEBUI_GATEWAY_READ_TIMEOUT). A read timeout is TERMINAL — CPython's
socket.makefile latches _timeout_occurred on the first socket.timeout, after
which every read raises a bare OSError, so the connection can't be resumed. On a
read timeout (or that poisoned-socket OSError) the generator surfaces the user's
Stop (yield b"" -> caller emits cancel) or re-raises so the caller reports the
stall. A stream that keeps emitting bytes within the window never trips it, so a
genuinely alive (if slow) turn is untouched; only >budget of total byte-silence
is treated as a dead gateway. Both loops (runs-API + legacy) changed identically.
The test fake now models the timeout-poison latch (a bare OSError after the first
timeout) so it reflects real socket semantics; the previous stateless re-raise
had made a multi-window "recovery" path look green that cannot occur in
production.
Refs #2476
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WebUI keeps two parallel arrays per session: messages (display transcript) and
context_messages (what's sent to the model). In large/compacted sessions they
diverge. Forking / in-place truncation copies a prefix of both and relies on
truncate_context_for_display_keep() to translate a display keep-index into the
matching context index — an aligner that prefers a stable per-row `id`. But the
model-context rows carried neither id nor timestamp (a 989-session live scan
found 0 with any id), so alignment fell back to fragile content-signature
matching that goes ambiguous on repeated tool calls / empty assistant turns.
This closes the DATA gap (the alignment logic itself already prefers `id` and
handles the compacted case via the matcher, shipped in #5563): mint a monotonic,
session-unique integer `id` on the per-turn result rows AFTER the context restore
and BEFORE both arrays are built, so the display and model-context copies of a
logical row share the same id.
- api/streaming.py: new _assign_stable_message_ids(); _restore_reasoning_metadata
carries `id` forward across turns (as it already does timestamp); wired into all
three streaming commit sites (main turn, retry, self-heal).
- api/routes.py: same mint on the runs/MoA _handle_chat_sync commit path.
- api/gateway_chat.py: mints ids on the two new rows of a gateway turn.
- The `id` is stripped before the provider API call (not in _API_SAFE_MSG_KEYS),
same treatment as timestamp — nothing new reaches the provider.
session_ops.py alignment was already delivered by #5563 (id-preferring matcher +
compacted-case fall-through), so no change is needed there now; tests updated to
assert the post-#5563 behavior (id-bearing = exact cut; id-less = errs toward
under-keeping, never the old raw-index mis-cut).
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Gate hardening for #5198: the new stream-owner map leaked entries on three
paths that bypass the teardown finally — worker early-return when the stream
was cancelled before startup (streaming.py, gateway_chat.py) and the direct
ACTIVE_RUNS stale-zombie prune (routes.py). Each now calls unregister_stream_owner.
Adds a regression test proving the early-return path no longer leaks.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
starship-s