The streaming settlement path could show a misleading "No response from provider"
error card AFTER a completed answer, because a stale terminal/partial result flag
still forced the generic no_response error path even when the merged current turn
already had a final assistant answer. Now the generic no_response terminal error
is suppressed ONLY when the already-merged current turn has a completed assistant
reply (scoped to soft `partial` results per the issue-thread recommendation),
letting normal `done` settlement continue. Real silent failures, partial
failures, auth/quota errors, cancellation, replayed rows, and terminal
tool/compression cases all keep their existing error behavior.
Self-built stage of contributor PR; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
'All 0 credential(s) exhausted for <provider>' did not match _is_quota_error_text
so it fell through to a generic 'Error' with no hint (the exception path's else
branch discarded the hint entirely). Adds a distinct credential_pool_empty
classification (ordered before the quota check) + an explicit exception-path
branch + an actionable pool-specific hint, so the user sees why the turn failed
and what to do. Does not over-match (doesn't swallow real quota errors) and
doesn't block partial-harvest; downstream tolerates the new type.
NOTE: this is the CLASSIFICATION half of #3929 only — the data-loss half
(reasoning/partial-work restore on reload-reconcile) is a scoped follow-up that
needs the reporter's server logs.
Self-built (nesquena-hermes); nesquena independent review APPROVED.
Codex round-3 gate finding: _dedupe_replayed_context_messages deep-copies the
stale-user repaired boundary row (streaming.py:4501) into the context array. When
the mint ran AFTER dedupe, that deep-copied context row was no longer the shared
result dict, so it stayed id-less while the display copy got the minted id —
re-opening the cross-array id divergence for the stale-repair path. Fix: move
_assign_stable_message_ids() to run immediately after _restore_reasoning_metadata
and BEFORE _dedupe_replayed_context_messages at all four commit sites (streaming
main / in-band self-heal / except-path self-heal + routes runs/MoA), so the shared
result rows are stamped before any deep-copy — both arrays inherit the id.
Codex/Opus gate finding: in eager session-save mode the current user turn is
checkpointed into s.messages before the agent runs (no id yet); after
_assign_stable_message_ids stamps the returned result row, the display merge
keeps the durable eager checkpoint and skips the stamped result — leaving the
display row id-less while its context twin carries the minted id, silently
defeating id-based fork/truncate alignment for eager-mode users. The merge now
copies the minted id onto the kept checkpoint when it lacks one (guarded so it
can't overwrite or duplicate). Opus verified cross-array id pairing holds across
the real 3-turn transform pipeline.
WebUI keeps two parallel arrays per session: messages (display transcript) and
context_messages (what's sent to the model). In large/compacted sessions they
diverge. Forking / in-place truncation copies a prefix of both and relies on
truncate_context_for_display_keep() to translate a display keep-index into the
matching context index — an aligner that prefers a stable per-row `id`. But the
model-context rows carried neither id nor timestamp (a 989-session live scan
found 0 with any id), so alignment fell back to fragile content-signature
matching that goes ambiguous on repeated tool calls / empty assistant turns.
This closes the DATA gap (the alignment logic itself already prefers `id` and
handles the compacted case via the matcher, shipped in #5563): mint a monotonic,
session-unique integer `id` on the per-turn result rows AFTER the context restore
and BEFORE both arrays are built, so the display and model-context copies of a
logical row share the same id.
- api/streaming.py: new _assign_stable_message_ids(); _restore_reasoning_metadata
carries `id` forward across turns (as it already does timestamp); wired into all
three streaming commit sites (main turn, retry, self-heal).
- api/routes.py: same mint on the runs/MoA _handle_chat_sync commit path.
- api/gateway_chat.py: mints ids on the two new rows of a gateway turn.
- The `id` is stripped before the provider API call (not in _API_SAFE_MSG_KEYS),
same treatment as timestamp — nothing new reaches the provider.
session_ops.py alignment was already delivered by #5563 (id-preferring matcher +
compacted-case fall-through), so no change is needed there now; tests updated to
assert the post-#5563 behavior (id-bearing = exact cut; id-less = errs toward
under-keeping, never the old raw-index mis-cut).
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
Extract _turn_transcript_lacks_final_assistant_answer() so settlement
logic can inspect an already-merged transcript. The merge wrapper now
delegates to the pure evaluator without changing behavior.
Fixes#5141
Gate hardening for #5198: the new stream-owner map leaked entries on three
paths that bypass the teardown finally — worker early-return when the stream
was cancelled before startup (streaming.py, gateway_chat.py) and the direct
ACTIVE_RUNS stale-zombie prune (routes.py). Each now calls unregister_stream_owner.
Adds a regression test proving the early-return path no longer leaks.
Co-authored-by: starship-s <starship-s@users.noreply.github.com>
Prevent historical assistant reasoning_content from being replayed to
generic/local providers by default while preserving current behavior for
known reasoning-capable provider protocols.
Adds a webui.reasoning_content_replay policy read by the sanitizer:
auto, strip, and preserve. Auto strips for local/generic backends such as
LM Studio/Ollama/llama.cpp and preserves for explicitly known provider
protocols that may require reasoning replay.
This only affects provider-facing sanitized history. It does not modify
stored session/display reasoning metadata.
WebUI's api/streaming.py:_resolve_image_input_mode was a LOCAL
re-implementation of agent/image_routing.py:decide_image_input_mode that
had DIVERGED from the canonical function. The local copy never consulted
the active model's vision capability — it only checked a handful of config
heuristics — so it returned "native" (embedding the image as an image_url
part) even for models KNOWN to lack vision. That silently sends pixels to a
blind model (#21160-class failure) instead of routing through the text
(vision_analyze) pipeline.
Fix: delegate to the canonical agent.image_routing.decide_image_input_mode
(single source of truth), with a try/except fallback that preserves the
historical WebUI behaviour when the agent package is not importable (e.g.
the WebUI standalone test environment).
Carve-out preserved: the canonical router conservatively returns "text" for
UNKNOWN/custom models (no models.dev capability data). WebUI historically
forwards those NATIVELY and relies on the agent's strip-and-retry guard
(_try_shrink_image_parts_in_messages / _strip_images_from_messages) to
downgrade on a provider rejection. We keep that: a canonical "text" verdict
is only honoured when there's a real signal — an explicit user choice
(image_input_mode: text or a configured auxiliary.vision backend) or a model
KNOWN to lack vision. Otherwise we forward native.
Tests:
- Existing gateway test (forwards_image_attachments_as_multimodal_parts)
stays green — unknown "test-model" still forwards native.
- New non-vacuous regression: a model KNOWN to be text-only
(supports_vision == False) now routes "text" — this FAILS against master,
where the old local copy returned "native" for the same config.
- Added coverage for the unknown-model carve-out, known-vision native path,
explicit-text-signal precedence, and the agent-unavailable fallback.
Salvage of #4113 by @gkd2323c. The original PR delegated fully to canonical
but did not add the unknown-model carve-out, which would have regressed the
WebUI native-forwarding behaviour the gateway test encodes.
Co-authored-by: gkd2323c <gkd2323c@users.noreply.github.com>
Adds a dedicated GET /api/commands/moa/resolve endpoint + resolve_moa_config()
that reads the MoA preset/usage server-side. The /moa send-path passes only a
moa_config:true boolean to /api/chat/start; the server re-resolves the real
config itself (never trusts a browser-supplied dict) and threads it per-turn into
the live agent run. Fails closed (409) on gateway-backed sessions.
Replaces the originally-proposed broad 43-command subprocess approach (bounced)
with this narrow, security-reviewed /moa route. Maintainer fixes on top of the
contributor's converged design: moa_config is added to run_conversation only
when not None (older-agent TypeError safety) at the main call AND both auth
self-heal retry sites, and forwarded through the legacy-journal runtime adapter.
Gate: Codex SAFE (after the maintainer fixes) + full suite 10784 pass + the
activity-stream regression gate clean.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
When a turn is steered, the agent embeds an
[OUT-OF-BAND USER MESSAGE]...[/OUT-OF-BAND USER MESSAGE] control block, and that
block gets persisted into the session's messages/context_messages (appended onto
a tool-result entry — verified present in real prod session JSON, in both
messages[] and context_messages[]). WebUI then replays context_messages into the
provider-facing conversation history every turn, so the model kept re-seeing the
internal control marker: wasted tokens + stale control data.
Adds a non-mutating, recursive _strip_oob_blocks() (regex, DOTALL+IGNORECASE,
non-greedy) wired into every model-facing chokepoint:
_sanitize_messages_for_api, _api_safe_message_positions,
_restore_reasoning_metadata projection, and the gateway runs-API
conversation_history (context_messages + prefill). The compression path
(/session/compress) inherits the strip via _sanitize_messages_for_api.
The marker stays intact in the persisted transcript (display/reload/audit
fidelity); it's removed only on the way to the provider. Idempotent, only strips
a complete open+close pair, and does not mutate persisted session objects (the
content key is reassigned to a freshly-built value).
Maintainer hardening (Opus gate): relaxed the opening pattern from a mandatory
`]\r?\n` to `]\s*?` so a single-line OOB block is also stripped (DOTALL .*?
already spans newlines) — robustness against a format that omits the newline.
Co-authored-by: Stacey2911 <Stacey2911@users.noreply.github.com>
Cancelling a running turn briefly collapsed the already-streamed reasoning,
tool cards, and text to a bare "Task cancelled" marker before a follow-up
GET /api/session repopulated them — a visible race between the terminal
cancel SSE event and the refetch (#4076, #1361).
The terminal cancel event now carries a canonical, redacted session snapshot
captured under the per-session agent lock right after the cancelled turn is
persisted (including the recovered partial-assistant message). The frontend
applies that embedded snapshot immediately via _applyCancelSessionPayload and
skips the GET round-trip. The snapshot is a fully detached deep copy (redaction
rebuilds every container) so it cannot alias or be staled by concurrent writes;
it is only applied when its session_id matches the active view; and a
missing/failed snapshot degrades cleanly to the existing GET fallback.
The seven other cancel call sites are standardized via _cancel_event_payload()
(message + type + status), backward-compatible (no session key when none).
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Tool-call args were truncated to 120 chars at three sites (_truncate_tool_args,
the live _tool_args_snapshot in api/streaming.py, and _toolArgsSnapshot in
static/ui.js). For incidental args that's fine, but content/diff-bearing keys
(command/cmd/script/code/patch/diff/old_string/new_string/content/path/
file_path) feed the rendered tool card AND the diff reconstruction in
recovery-rebuilt sessions, so the cap silently corrupted long commands/paths and
produced broken/empty diffs. Exempt those keys with a much larger cap
(_TOOL_ARG_CONTENT_CAP = _TOOL_RESULT_SNIPPET_MAX = 4000) at every site, keyed by
key name; incidental args keep the 120 cap; storage stays bounded. Part of #4925.
Codex re-gate 2: post-run flush is bypassed on exception (jumps to outer except) + retry
paths, so a buffered reasoning tail was lost before apperror there. Added a guarded
_flush_reasoning_buffer() at the top of the inner finally (guaranteed exit for
normal/exception/retry) so the tail always reaches the live Thinking view before teardown.
+ regression asserting the finally flush.
Co-authored-by: wlknight <wlknight@users.noreply.github.com>
Codex re-gate caught a SILENT gap: the tail-flush only ran on on_reasoning(text=None),
but the agent never calls reasoning_callback(None) — so the last sub-100ms coalesced
buffer was silently lost, and a reasoning→tool/output transition stranded the tail.
Added a shared _flush_reasoning_buffer() helper called at every boundary that closes or
reorders the live reasoning stream: on_token (visible output starting), on_tool (tool
boundary), the defensive on_reasoning(None), and right after agent.run_conversation()
returns (terminal catch-all for a turn that ends on reasoning with no trailing token/tool).
Each flush emits the buffered chunk then clears it (since-last-flush delta semantics, so
the frontend's reasoningText += text stays correct). + regression test for all sites.
Co-authored-by: wlknight <wlknight@users.noreply.github.com>
Module-level _compact_for_echo_compare (strips ALL whitespace, \s+ -> '') so _is_visible_output_echo's
endswith() suffix-match detects interim_assistant prose that only differs from visible token prose by
line-break/paragraph formatting. Hoisted to module scope (importable) + behavior-based test (real
token vs interim text equality). Codex SAFE (exact suffix match, not substring). 103 echo/journal tests.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
A failed /steer (or busy-mode steer) no longer silently falls back to interrupt+queue
(which cancelled the active stream). _trySteer now returns delivered:bool, restores the
draft + leaves the stream running on failure; send() only clears staged files when
_steerDelivered===true. Backend hunk comment-only (failure paths already returned
{accepted:False}, never touched CANCEL_FLAGS). i18n value-only on pre-existing keys
(locale-parity preserved, 173 locale tests pass). 48 own tests pass.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: dso2ng <dso2ng@users.noreply.github.com>
In-chat cronjob tool reached cron.jobs whose path constants snapshot at import, so a
chat under a named profile hit the DEFAULT profile's cron dirs. Adds a per-turn
contextvar (_STREAMING_CRON_PROFILE_HOME, reset on every exit) + a tool-call-boundary
wrapper that enters cron_profile_context_for_home() only while the cronjob handler runs
(cron lock, restored immediately — no whole-turn global mutation). Verified: agent
executor's propagate_context_to_thread uses unfiltered contextvars.copy_context() so the
contextvar reaches the sync handler on the ThreadPool worker; reset-token symmetry intact.
Code byte-faithful from PR head c3fd6ab22f8c. 12 own tests pass.
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: roian6 <roian6@users.noreply.github.com>
* stage #4618 (allenliang2022): broaden streaming stale-compressor guard to any model-window mismatch + #4618 regression tests + CHANGELOG
Broadens #3256's default-only live-usage guard: the streaming SSE snapshot now
always resolves the real per-model window via the same helper GET /api/session
hydration uses (_context_length_lookup_inputs_for_model + get_model_context_length)
and corrects whenever it differs from the compressor's cached value, with a
TypeError fallback to the legacy 2-arg form. Fixes 'refresh shows 1M, send reverts
to stale 168k + early auto-compress' on model-switched sessions. Per-stream cache
preserved (one lookup/stream). Code byte-identical to PR head 3beb18e92d73.
Adds 4 source-structure regression tests (RED-proven on master).
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
* fix#4618 gate findings: profile-scoped config (Codex cross-profile) + #4248 256k acceptance gate (Opus downward-clobber)
Codex SHIP-WITH-FIXES: live-snapshot used ambient get_config() which in the
detached streaming worker resolves the process-global/default profile (#3294) ->
for a non-default profile pinning a different per-model context_length it would
surface the WRONG profile's window. Now resolves via get_config_for_profile_home
on the session's own profile home (mirrors the worker's _cfg resolution).
Opus SHIP-WITH-FIXES: broadened guard aligned resolution w/ hydration but not its
#4248 acceptance gate -> a transient low-confidence 256k metadata probe could
clobber a LARGER cached window mid-stream. Now reuses the exact hydration helper
_should_accept_session_context_length_refresh on both modern + legacy paths.
+ regression tests for both. Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
* fix#4618 Codex re-gate findings: broaden save + SSE-done stale-compressor guards too
Codex re-gate found the broadened live-snapshot guard fixed metering but the two
SIBLING paths still used the old default-only exact-cap test:
- api/streaming.py final session-save: persisted stale other-model window (168k)
to s.context_length -> wrong window on reload.
- api/streaming.py terminal SSE: emitted stale window -> indicator REVERTS
on stream end (messages.js overwrites S.lastUsage) = the exact 'send reverts to
168k' symptom.
Both now resolve the real per-model window via the same hydration helper and honor
the #4248 acceptance gate (no 256k downward-clobber), with legacy 2-arg fallback.
This is the root-cause completion across all 3 paths (live/save/SSE-done).
+ 2 regression tests. Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
* docs(#4618): note model_changed-omission rationale (Opus) + broaden CHANGELOG to all-3-paths
* Release v0.51.561 — Release TT (context-window indicator stays correct after model switch; #4618)
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: allenliang2022 <allenliang2022@users.noreply.github.com>
