Commit Graph
2479 Commits
Author SHA1 Message Date
Ruby Hartono bbda885aa6 fix(stream): cap StreamChannel per-subscriber queues (drop-oldest on full)
Memory: each connected tab's subscriber queue was queue.Queue() with no
maxsize. A slow/backpressured/backgrounded tab accumulated every coalesced
token frame for the WHOLE turn (producer = agent token stream) — unbounded
per-tab growth, OOM risk with many tabs x long agentic turns. The offline
buffer was already capped (#4633), but once a subscriber attached the
live-broadcast path flowed into an unbounded per-subscriber queue.

Cap per-subscriber queues at _SUBSCRIBER_QUEUE_MAXSIZE (== _OFFLINE_BUFFER_MAXLEN)
and drop the OLDEST frame on queue.Full in both the broadcast (put_nowait) and
offline-buffer replay (subscribe_with_snapshot) paths, so a slow tab keeps the
most recent tail (older frames stay recoverable via the run journal by
last_event_id). Mirrors SessionChannel.emit's drop-on-full contract.

The cap intentionally equals the offline buffer bound, not SessionChannel's
smaller 16: StreamChannel carries the live token stream and has a #4633
reconnect-replay contract (a resubscribing tab must receive the full retained
offline tail), so capping below the buffer would force every flaky-network
reconnect through the run journal. Matching the buffer bound preserves that
contract while bounding live-broadcast memory to the same worst-case #4633
already accepts. The SSE write deadline (SSE_WRITE_DEADLINE_SECONDS) breaks a
stuck socket within ~20s, so the overflow window is short; this also bounds it
by frame count.

Adds a cumulative subscriber_dropped_events diagnostic counter (ops visibility).

Test: tests/test_stream_subscriber_queue_cap.py — bounded queue never exceeds
cap, drop-oldest keeps newest tail, draining vs non-draining subscribers, and
the #4633 reconnect-replay contract still delivers the full offline tail.
Existing offline-buffer/gap-recovery/multitab tests unchanged.

Contract Routing
- Task type: memory-bound fix (unbounded per-subscriber queue).
- Touched areas: live chat SSE broadcast (StreamChannel).
- Relevant public docs: AGENTS.md, CONTRIBUTING.md, docs/CONTRACTS.md,
  docs/rfcs/webui-run-state-consistency-contract.md (Live stream/SSE layer),
  docs/rfcs/session-sse-contract-v1.md (distinct: that RFC governs the PROPOSED
  per-session endpoint /api/sessions/{id}/events, NOT StreamChannel, which is
  the /api/chat/stream live token broadcast; no text in that RFC is changed).
- State layer mutated: Live stream / SSE layer — specifically the per-subscriber
  queue inside StreamChannel (api/config.py). NOT the durable run journal, NOT
  the transcript, NOT replay.
- Invariant preserved: the #4633 reconnect-replay contract — a resubscribing
  tab must receive the full retained offline tail. The cap is set EQUAL to
  _OFFLINE_BUFFER_MAXLEN so replay delivery is unchanged; only the previously-
  unbounded live-broadcast-to-slow-tab growth is bounded. Regression:
  test_reconnect_replay_still_delivers_full_offline_tail (and the pre-existing
  test_stream_offline_buffer_cap.py suite) pass unchanged.
- Scope boundaries: does not touch replay idempotency (Invariant 5), the turn
  journal, compression, or sidebar metadata. Slow-tab frames dropped at the cap
  stay recoverable via the run journal by last_event_id.

Release note: StreamChannel per-subscriber SSE queues are now bounded
(drop-oldest), capping memory for slow/backgrounded tabs during long turns.

Model Used: builtin:zai-coding-plan/GLM-5.2 (ZCode agent). No AI-specific tool
use mattered beyond the repo's standard read/edit/test workflow.
2026-07-16 20:18:39 +07:00
6ed474ca54 fix(models): route list-shaped custom provider models (#6127) (#6128)
Centralize custom-provider model-ID extraction into _configured_model_ids()
+ _configured_model_options(), used by both catalog construction and runtime
routing so the picker and resolver can't drift. Fixes list-shaped
custom_providers[].models allowlists silently staying on the default provider
instead of routing to their own (#6121). Existing mapping / list-of-strings
configs unaffected.

Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
2026-07-16 00:22:50 -07:00
c39e8edaf1 fix(#6100): clear stale gateway approval attention badge (#6117) (#6125)
Reconcile the gateway pending-mirror in _session_attention_summary()
before counting sidebar approval attention, so a parent session's red
approval dot clears once the underlying gateway approval has drained.
Live approvals stay visible; clarify fallback unchanged. WebUI read-path
mitigation; child-session context-propagation root cause stays in #6100.

Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
2026-07-15 22:34:19 -07:00
nesquena-hermes 500fa485d2 Merge commit 'refs/pull/6013/head' of github.com:nesquena/hermes-webui into stage/6013 2026-07-16 02:28:15 +00:00
nesquena-hermes b877c9aa40 Merge commit 'refs/pull/6102/head' of github.com:nesquena/hermes-webui into stage/6102-cachettl 2026-07-15 22:40:21 +00:00
nesquena-hermes 38119fac25 fix(#6056): extend sidebar title-field redaction to /api/sessions/search (Codex class-fix)
Codex re-gate found the same derived-title leak on the sibling /api/sessions/search
endpoint (3 branches redacted only 'title'). Extract _redact_sidebar_title_fields()
shared helper, apply it in both _sidebar_session_response_item and all 3 search
branches, read api_redact_enabled once per search response (mirrors #4662). +helper
test + source-guard that every search branch calls the redactor.
2026-07-15 21:06:36 +00:00
starship-s f5ef3688c6 fix(cache): keep streaming TTLs above poll cadence 2026-07-15 20:59:59 +00:00
nesquena-hermes ffbc91343a fix(#6056): redact display_title/_state_db_title/parent_title on the sidebar API
Codex gate found the #6056 derived display_title (from raw user-message content)
bypassed sidebar redaction — the serializer redacted only 'title', so a credential
in a delegated goal leaked to the sidebar even with api_redact_enabled=True.
Redact display_title + _state_db_title + parent_title on the same path; +regression
test (proven to fail on the unfixed serializer).
2026-07-15 20:57:31 +00:00
nesquena-hermes b96405ef29 Merge commit 'refs/pull/6056/head' of github.com:nesquena/hermes-webui into stage/6056-delegtitles 2026-07-15 20:46:34 +00:00
nesquena-hermes 754dcca048 Merge commit 'refs/pull/6088/head' of github.com:nesquena/hermes-webui into stage/6088-manualupd 2026-07-15 20:02:02 +00:00
chsong 139c76535b fix tool-limit final answer classification 2026-07-15 19:38:25 +00:00
nesquena-hermes 30059085c2 fix(#6083): only grace-evict CONFIRMED never-persisted shells (Codex finding #2)
Codex re-gate asked to distinguish a confirmed-absent sidecar from an
indeterminate/unreadable one before entering the abandoned-shell grace path.
Add _session_sidecar_exists() (True/False/None) and only grace-evict when the
sidecar is confirmed absent (False); a present-or-indeterminate sidecar keeps
the session resident. Age-window bound retained (fully closes the leak, no
data-loss edge); an SSE-mounted-lease refinement is noted as optional follow-up.
2026-07-15 19:06:10 +00:00
nesquena-hermes 02e5acb890 fix(#6083 follow-up): bound never-persisted empty session shells
Codex gate found the eviction-reorder made never-saved empty shells immortal
(abandoned /api/session/new tabs accumulate unbounded past sessions_cache_max).
Add a grace-window bound: protect a never-persisted empty shell only while fresh
(<30min old) OR while it carries a composer draft; a stale, empty, draftless,
never-saved shell is treated as abandoned and becomes evictable again. A real
compose persists a draft on the first keystroke (disk-backed + reloadable), so
the brick fix is preserved while the leak is closed. +2 regression tests.
2026-07-15 18:44:21 +00:00
Sean 2a191007f1 Merge branch 'master' into fix/6010-cache-providers 2026-07-16 00:54:24 +08:00
webtecnica 40d5e17006 fix: allow manual force-check even when auto-update is disabled (#6082) 2026-07-15 12:18:04 -03:00
ai-ag2026andClaude Opus 4.8 7707ea1d2a fix(webui): stop password managers autofilling the search box, and keep unsaved sessions startable
Two independent defects that chain into "new conversations cannot be started":
every POST /api/session/draft and /api/chat/start 404s a few seconds after
POST /api/session/new returned 200.

1. Search inputs are parsed as username fields.

index.html contains no <form> at all, but does contain password inputs (the
Settings "Access Password" panel). Chromium therefore groups the document's
unowned fields into one synthetic password form and picks the first text input
in the DOM — #sessionSearch, the sidebar conversation filter — as that form's
username field, then autofills the saved account name into it on load.
autocomplete="off" was already present; it is ignored for credential
heuristics.

The autofill fires oninput -> filterSessions() -> GET /api/sessions/search
?content=1&depth=5: a full content search the user never asked for, repeated
on every page load.

Fix: give the four search inputs type="search" plus the per-manager opt-outs
(data-1p-ignore / data-lpignore / data-bwignore / data-form-type), and give the
Settings password inputs a real <form> owner with proper autocomplete tokens so
Chromium scopes the credential form to them instead of to the whole document.
The form uses display:contents, so layout is unchanged. CSS suppresses WebKit's
native search clear button, since these fields ship their own
(#sessionSearchClear).

2. A never-persisted session is treated as evictable.

new_session() intentionally keeps a session in RAM until its first message
(#1171), so the SESSIONS cache is its only copy. _session_is_evictable()
(#4765) short-circuited on zero messages, reasoning that an empty shell "is
recreated on next access". It is not: get_session() has no recreate path and
raises KeyError, so both routes 404 and the session can never be started.

The content search from (1) pulls every hit through get_session(), which blows
past sessions_cache_max (default 300) and evicts the session being composed.
That is why this reproduces within seconds on an install with ~1700 sessions,
and why it grew worse as the session count rose.

Fix: require proof of persistence before evicting, for empty sessions too —
which is what the function's own docstring already promised ("Its full state is
already persisted to the JSON sidecar"). The zero-message branch was the single
path that bypassed it. #4765's memory bound is unaffected: sessions loaded from
disk remain evictable, and its existing tests still pass.

Trade-off: an unsaved empty session now stays resident for the process
lifetime (one per "New Conversation" click, an empty object each). That is a
deliberate application of the function's stated invariant — "slightly more RAM
for a session we are unsure about is strictly better than evicting an unsaved
session". Bounding it by age instead would also work if maintainers prefer.

Validation:
  pytest tests/test_issue4765_sessions_lru_eviction.py  ->  9 passed
  The added regression test fails on the unpatched predicate (verified by
  reverting api/models.py alone) and passes with the fix.

Note: (1) fixes what triggers the churn, (2) fixes what turns that churn into
an unstartable session. Either fix alone leaves a real defect in place, so
they are submitted together.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 10:06:41 +02:00
pxxD1998 73b6c5d604 fix(sessions): cascade delegated subagents on parent deletion 2026-07-14 18:10:03 +08:00
Frank Song 59e92461fc fix: fail closed on wrapped gateway pid helpers 2026-07-14 16:36:23 +08:00
Sean 186fef2895 Merge branch 'master' into fix/6010-cache-providers 2026-07-14 16:09:38 +08:00
Frank Song d08116b4d2 fix: harden gateway restart recovery targeting 2026-07-14 16:01:56 +08:00
Rod Boev ab48096de1 feat(#6033): show delegated goal titles for child sessions 2026-07-14 04:01:06 -04:00
Frank Song ba1a058f87 fix: fail closed on implicit gateway pid recovery 2026-07-14 15:25:36 +08:00
Frank Song 14adbb61a4 fix: pin agent update gateway restart profile 2026-07-14 14:46:00 +08:00
Frank Song 885e267213 fix: retry transient agent gateway restart failures 2026-07-14 14:13:19 +08:00
Isla-Liu 2e8915baf8 perf(webui): skip impossible state-db prefix normalization 2026-07-14 05:38:34 +00:00
Rui Xue b2c7578653 fix(sessions): guard all auto-mint paths + strict config/body parsing (gate review)
Gate review follow-ups for the #6022 no-leak rule: every non-deliberate
newSession/session-new call site passes worktree:false; only deliberate
New Chat inherits the config default.

- Strict boolean config read: only YAML true enables the default;
  quoted strings/ints/lists/dicts/null fall to the safe no-worktree
  default instead of truthiness-coercing.
- Presence-based explicit detection: 'worktree' in body — an explicit
  worktree:null is an opt-out, never a fall-through to config.
- Guarded remaining auto-mint call sites with explicit worktree:false:
  profile switch (panels.js), promptWorkspacePath + switchToWorkspace
  blank-page binds (panels.js), promptNewFile + promptNewFolder
  (ui.js), terminal auto-session (commands.js).
- Tests: worktree:null vs config-true (factory must not be called),
  strict-boolean malformed-shape matrix, static regression gates for
  every auto-mint site + a bare-body sweep so future auto-mint paths
  can't silently omit the key; updated the cmdTerminal preflight
  assertion for the new call shape.
2026-07-14 04:35:06 +00:00
Rui Xue 5f638a1694 feat(sessions): honor the agent's worktree config default in /api/session/new
/api/session/new treated an absent 'worktree' key as false, so the
agent's config-level 'worktree: true' default was never consulted for
WebUI sessions even though the WebUI reuses the agent's own worktree
helper. Both callers should agree on same-agent/same-repo isolation.

Three-value model:
- body 'worktree' absent -> agent config default applies (profile-aware
  via get_config_for_profile_home, so a user with worktree:true in one
  profile but not another gets per-profile behavior)
- explicit true/false -> honored verbatim; explicit always wins

Non-git workspaces: a config-default request degrades to a plain
session and surfaces 'worktree_skipped' in the payload; an explicit
worktree:true keeps the hard 400.

Frontend (same change, load-bearing): newSession() forwards explicit
true/false and omits the key otherwise; the boot-time auto-bind
(_maybeBindFreshDefaultWorkspaceSession) and the onboarding session
send worktree:false explicitly so a config default can never mint a
worktree + branch from merely opening the UI. Deliberate 'New Chat'
sends nothing and inherits the default.

Tests: route matrix (config on/off x body absent/true/false), non-git
fallback vs explicit 400, explicit-false-beats-config-true, config
helper (ambient/named-profile/fail-soft), static UI assertions that
auto-bind and onboarding opt out and New Chat paths don't pin a value.

Fixes #6022
2026-07-14 04:35:06 +00:00
Rod Boev 001d7985bc feat(#6006): instrument send-to-first-token latency 2026-07-13 17:56:44 -04:00
rudidev08 34cf2de57a Fix: "Approve once" no longer persists as a session approval (#6017)
In the local (non-gateway) approval path, _resolve_approval_legacy grouped
"once" together with "session" and called approve_session() for both.
approve_session() records the pattern in the session-wide allowlist that the
guard later consults, so a single "Approve once" click silently approved every
later matching tool call for the whole session -- a destructive tool could then
run with no approval card.

Only "session"/"always" persist now. "once" falls through: the
resolve_gateway_approval() call below still unblocks the parked agent thread for
every choice, so the current tool call proceeds, but nothing is written to
_session_approved and the next matching guarded call re-prompts.

Adds tests/test_issue6017_approve_once_no_persist.py, asserting is_approved is
False after "once" and True after "session".
2026-07-13 13:05:52 -07:00
Rui Xue c833f290a9 fix(worktrees): unlock before remove so WebUI-created worktrees can be reaped
WebUI-created worktrees are locked at creation by the agent
(cli._setup_worktree locks with --reason 'hermes pid=...'), but
remove_worktree_for_session never unlocked, so git refused every removal
with "use 'remove -f -f' to override or unlock first" — force=true
included, since a single --force cannot override a lock.

Mirror the agent's own _cleanup_worktree ordering: fail-soft
'git worktree unlock' right before the remove, after all
dirty/untracked/unpushed/stream/terminal guards have passed. Those
guards remain the real safety layer; force keeps meaning 'override
dirty working state', never 'override a foreign lock'.

Tests: locked worktree now removes cleanly without force; unlock is
fail-soft when the worktree was never locked; existing mocked
call-order assertions updated for the leading unlock call.

Fixes #6023
2026-07-13 17:47:58 +00:00
seanxgao ab4d64f157 perf(providers): invalidate cache on oauth credential changes 2026-07-13 19:58:41 +08:00
seanxgao d3720fa724 perf(providers): cache provider list responses 2026-07-13 17:12:07 +08:00
Rod Boev 064ad82ee6 fix(#5940): key pending-turn recovery to exact checkpoints 2026-07-13 03:41:39 +00:00
Rod Boev 0c8f647e9c fix(#5940): drop unused gateway error adapter 2026-07-13 03:41:39 +00:00
Rod Boev a87977da33 fix(#5940): settle gateway terminal errors before emitting apperror 2026-07-13 03:41:39 +00:00
Rod Boev e6dea2a32e fix(#5940): attach session identity to gateway apperrors 2026-07-13 03:41:39 +00:00
Rod Boev 60ec437393 fix(#5940): preserve gateway terminal errors after stream failures 2026-07-13 03:41:39 +00:00
Rod Boev bc692cc8ac fix(#5940): classify gateway terminal provider errors instead of generic empty turns 2026-07-13 03:41:39 +00:00
nesquena-hermes 53a4b35555 harden #5979 round 3 (Codex): signature-based explicit-pick provenance
Codex found a verified lifecycle hole: the bare boolean model_explicitly_picked
was not cleared on /api/session/update (api/routes.py:14438-14460 changes+persists
the model/provider without touching it), so after an update+reload the flag stayed
True and a cold #433 stale leftover (openai/gpt-5.4) was wrongly preserved instead
of stripped to gpt-5.4.

Fix (Codex's prescription): replace the unscoped boolean with a SIGNATURE bound to
the routing context.
- api/models.py: new model_explicit_pick_signature(model, provider) ->
  '<model>\x1f<provider>'; Session.model_explicit_pick_signature persisted field
  (replaces the boolean; in METADATA_FIELDS, restored via kwargs, survives restart).
- api/routes.py /api/chat/start: on a fresh explicit_model_pick, stamp the signature
  of the resolved model+provider.
- api/streaming.py: treat the send as a deliberate pick ONLY when the persisted
  signature equals the signature recomputed from the session's CURRENT persisted
  model+provider. Any model/provider change (chat-start, session-update,
  normalization, provider repair) yields a different signature and auto-invalidates
  the stale pick — no per-mutation-site clearing needed.

So: deliberate pick on unchanged context -> preserve (b3nw); any switch away ->
signature mismatch -> unmarked -> legacy strip (#433 stays correct even after a
session-update+cold-restart). Adds signature match/invalidate/persist tests.
75 focused tests pass (resolver + sprint40 + the 2 streaming-mock suites).
2026-07-13 01:58:20 +00:00
nesquena-hermes 45cb5bd3e6 feat(#5979): persisted explicit-pick provenance for cold custom-proxy resolution
Nathan's call: adopt Codex's complete mechanism instead of the unconditional
cold-preserve. The cold-catalog custom-proxy decision is now gated on whether the
user DELIBERATELY selected the model this session, closing the residual where a
stale #433 leftover would preserve+400 on a cold+no-disk send.

Mechanism:
- resolve_model_provider(model_id, *, explicitly_picked=False): in the custom-arm
  cold branch (no warm provenance, not config-declared), explicitly_picked=True
  preserves the vendor namespace verbatim (deliberate pick, proxy routes on it);
  otherwise the legacy redundant-prefix strip runs (stale cross-provider leftover,
  #433). Warm endpoint-advertised provenance still wins over the flag.
- Session.model_explicitly_picked: new persisted field (default False, restored
  from metadata via **kwargs, listed in METADATA_FIELDS so it survives a restart
  — b3nw's exact cold scenario).
- /api/chat/start stamps it: True on a fresh explicit_model_pick; cleared only
  when the resolved model actually CHANGED without a pick (real switch away);
  preserved across plain follow-up sends of the same model (the onchange marker
  is one-shot, so per-send explicit_model_pick alone would drop the intent).
- streaming.py reads s.model_explicitly_picked and passes explicitly_picked= into
  resolve_model_provider (inside the profile scope).

Independent end-to-end matrix (all pass): ben default cold, ben picked non-default
cold+warm, ben stale cold (strips), #433 warm+cold, #3872 bedrock, #548 zai-org.
67 resolver+sprint40 tests pass incl. new explicit-pick + stale-companion +
persistence round-trip.
2026-07-13 01:43:17 +00:00
nesquena-hermes f42b4d14f5 harden #5979 follow-up round 2 (Codex): profile-fingerprint-gated warm no-op
Codex finding: the warm helper's fast no-op (if _models_cache_provenance is not
None: return) ignored WHICH profile the resident provenance belongs to. The
catalog globals are process-wide, so profile B's resident provenance could block
profile A from loading A's own valid disk cache — A would then resolve against
B's advertised ids (verified: profile A provenance resident + profile B bare-only
disk cache -> A's openai/gpt-5.4 wrongly preserved).

Fix: gate BOTH the fast no-op and the post-lock recheck on a current-profile
fingerprint match (_provenance_is_current: prov[1] == _models_cache_source_fingerprint()).
On mismatch the helper falls through, loads THIS profile's disk snapshot, and
republishes provenance stamped with the current fingerprint.

Tests: rewrote noop-when-warm to use the CURRENT runtime fingerprint (genuinely
hits the fast path) and added ignores-foreign-profile-resident (foreign fp -> falls
through -> loads this profile's disk snapshot). 66 resolver+sprint40 tests pass.

NOTE: Codex also re-raised the cold+no-disk #433 case (a stale openai/gpt-5.4 on a
bare-only relay preserves instead of strips when no catalog exists at all). Fable
weighed this exact residual and judged cold-preserve strictly better than master
(loud-fail-and-heal vs silent-recurring-truncation). The two advisors diverge;
surfacing to Nathan rather than auto-adopting the heavier persisted-explicit-pick
mechanism.
2026-07-13 01:43:17 +00:00
nesquena-hermes e3152572aa harden #5979 follow-up per Codex gate: non-blocking disk-only warm + profile scope
Codex gate found two real issues in the send-path warm:

1. Blocking lock wait (CORE): get_available_models(prefer_cache=True) still
   acquires _available_models_cache_lock and can wait up to ~60s (unbounded in
   synchronous rebuild mode) on an in-flight rebuild — unacceptable on the send
   hot path. Rewrote warm_models_catalog_provenance_if_cold as a genuinely
   non-blocking, disk-only publish: try the cache lock NON-BLOCKING (return
   immediately if busy — a concurrent build will publish provenance itself),
   read ONLY the on-disk cache via _load_models_cache_from_disk (no network, no
   rebuild), publish snapshot+fingerprint + _sync_models_cache_provenance().
   Verified: returns in ~3.5ms while the lock is held (was a 60s block).

2. Profile scope (CORE): the streaming worker is a separate thread that does NOT
   inherit the HTTP handler's request-profile TLS. Both the warm and the resolve
   read profile-keyed config (cache path + source fingerprint via
   get_active_profile_name), so a cold send from a NAMED profile could resolve
   against the DEFAULT profile's config and route to the wrong provider/base_url
   (get_available_models reloads config on path mismatch). Wrapped warm + resolve
   in profile_scope_for_detached_worker(_resolved_profile_name) so both see the
   owning session's profile (no-op for default/root).

Tests: rewrote never-live-rebuilds (now asserts warm NEVER calls
get_available_models — reads disk directly) and added a non-blocking regression
(warm returns <2s while the cache lock is held). 65 resolver+sprint40 tests pass;
b3nw end-to-end + non-blocking + disk-warm all verified.
2026-07-13 01:43:17 +00:00
nesquena-hermes 6745827eb5 fix(#5979 follow-up): preserve custom-proxy vendor namespace on cold-catalog sends
The earlier #5979 fix (PR #5980, exp-v0.52.53) resolved the WARM-catalog and
config-declared cases, but b3nw confirmed truncation persists on exp-v0.52.55/56.

Root cause (reproduced on master, confirmed by Codex + Fable):
b3nw's config is provider: custom:llm-proxy, model.default: x-ai/grok-4.5, no
models[] allowlist. His DEFAULT (x-ai/grok-4.5) preserves via config-declaration,
but a model he SELECTS in the picker that isn't the default (x-ai/grok-composer-2.5-fast)
truncates on every COLD send. The send path (api/streaming.py:7857) calls
resolve_model_provider without warming the catalog, and the #1855 chat/start fast
path deliberately skips the catalog when a session already has a persisted
model+provider -- so 'cold at send' is the designed state after any restart /
settings-save / cache-expiry, not a rare race. With provenance None and no config
declaration, the #5980 tri-state fell through to the LEGACY family strip
(if prefix in _PROVIDER_MODELS and _is_first_party_model: return bare), and
grok-composer-2.5-fast had graduated into the x-ai first-party catalog -> stripped.

Fix (two changes, both advisor-verified):
1. Flip the custom-arm cold default from family-strip to PRESERVE-verbatim
   (api/config.py). For an explicit custom / custom:<slug> proxy the strip now
   requires POSITIVE provenance (endpoint advertises only the bare id, #433). A
   wrong strip destroys routing info the user can't recover; a wrong preserve
   fails loudly and self-heals on warm. This also honours _endpoint_advertised_model_ids's
   own documented 'None -> preserve verbatim' contract and removes the data-driven
   flaw where a model graduating into the static catalog silently flips routing.
2. Network-free provenance warm on the send path
   (warm_models_catalog_provenance_if_cold, called in streaming.py before the
   resolve). prefer_cache=True republishes provenance from the durable disk cache
   (no live probe, no _cfg_lock held on the worker thread -> no lock-order
   deadlock), so #433's bare-only-strip stays exact whenever a disk cache exists.

Residual (accepted, strictly better than master): a #433-shaped wrong-preserve
now requires memory cold AND disk cache absent/invalid (first run, post-config-save
window, schema/version bump). It fails loudly and heals on the next /api/models or
session-visit rebuild. Under master b3nw's failure was the mirror image but silent,
recurring every send, and unfixable short of declaring every model in config.

Supersedes rodboev's #5996, which targeted the providers.<slug> arm and does not
cover b3nw's actual custom:llm-proxy config (confirmed by Codex).

Tests: rewrite the cold-fallback test (strip -> preserve), sharpen the
foreign-profile test so trusted-strip vs ignored-preserve genuinely diverge, add
b3nw's exact cold non-declared custom:slug case, and add three warm-helper tests
(publishes-from-disk, noop-when-warm, never-live-rebuilds). 64 resolver+sprint40
tests pass; b3nw end-to-end verified; #433 positive-provenance strip verified.

Thanks @b3nw for the CLI-vs-WebUI comparison that localized it.
2026-07-13 01:43:17 +00:00
Jay Stothard c8bf3a1058 fix(streaming): preserve terminal failure precedence 2026-07-12 22:39:15 +00:00
Jay Stothard fe222d4ba2 fix(streaming): avoid false no-response after final tool answer 2026-07-12 22:39:15 +00:00
nesquena-hermes 1443288dee harden #5979 round 3 (Codex): lock-free atomic (snapshot,fingerprint) provenance
The round-2 fix acquired _available_models_cache_lock in the resolver to read
the snapshot+fingerprint consistently. Codex found that introduces a DEADLOCK:
config-save takes _cfg_lock -> _available_models_cache_lock while catalog-refresh
takes _available_models_cache_lock -> _cfg_lock (lock-order inversion), and it
also makes every send wait behind a catalog rebuild.

Fix: publish an immutable (snapshot, publisher_fingerprint) tuple atomically via
_sync_models_cache_provenance() at every cache publish/invalidate site (9 sites),
and have _endpoint_advertised_model_ids read that single global with ONE lock-free
load. Reading one tuple can't tear (GIL), acquires no lock (no ordering edge, no
deadlock), and never blocks behind a rebuild. A reader racing a publish sees the
previous consistent pair, never a torn snapshot/fingerprint.

Verified: two-thread probe — resolver completes in 0.0001s while another thread
holds _available_models_cache_lock (was the deadlock). Real catalog build publishes
the provenance tuple and closes the P0 end-to-end. Adds a deadlock-regression test.
2026-07-12 05:41:08 +00:00
nesquena-hermes 036c671f14 harden #5979 round 2 (Codex): atomic snapshot+fingerprint read + tri-state cold fallback
Two more real issues from the second Codex gate:

1. Atomic provenance read: _endpoint_advertised_model_ids read the global
   snapshot and its source fingerprint as two unlocked steps, while publishers
   assign the new snapshot BEFORE computing the fingerprint (streaming.py
   publish sites). In that window profile B's freshly-published catalog could
   validate against profile A's still-current fingerprint and strip against the
   wrong catalog. Fix: acquire _available_models_cache_lock (RLock) across the
   snapshot read + fingerprint check + memo build, so the pair is always
   consistent. The lock is held only for the two-assignment publish (the network
   rebuild runs before it), so the per-send acquisition never blocks on a probe.

2. Tri-state cold fallback (was: unconditional cold->preserve): an unconditional
   verbatim fallback newly broke the #433 relay cold path (openai/gpt-5.4 sent
   to a relay that only accepts gpt-5.4; master stripped it). Now:
     (1) config declares the full id verbatim (model.default / model.models /
         custom_providers[].models) -> preserve (network-free, survives a cold
         restart -> #5979 works cold via b3nw's model.default)
     (2) endpoint catalog advertised it -> full=preserve / bare-only=strip
     (3) genuinely unknown (cold + not declared) -> LEGACY family heuristic, so
         the edge is never worse than pre-fix: first-party redundant prefix
         still strips (#433 cold), intrinsic/unknown preserved (#3872/#548).
   The #5979 active-user path can't reach (3): a selected id is either
   config-declared or in the catalog the dropdown was built from.

New _model_id_declared_in_config() provides the config-provenance signal.
Tests updated: cold-fallback-to-legacy, config-declared-cold-preserve,
foreign-profile-catalog-ignored (proves untrusted snapshot is not used).
2026-07-12 05:41:08 +00:00
nesquena-hermes 0bdf555779 harden #5979 per Codex gate: union extra_models bucket + profile-isolation fail-safe
Codex regression gate found two real issues on the first cut:

1. extra_models bucket: a provider's advertised ids can be split across the
   catalog's 'models' (visible) and 'extra_models' (picker overflow) buckets.
   Reading only 'models' could miss a bare id that overflowed, leaving the
   #433 stale prefix unstripped. _endpoint_advertised_model_ids now unions
   both buckets.

2. Profile isolation (profiles are islands): _available_models_cache is a
   process global, so a concurrently-active foreign profile could publish the
   snapshot a different profile then reads for provenance — potentially
   stripping an advertised full id against another profile's catalog. The
   accessor now rejects the snapshot unless its published source fingerprint
   (whose config_yaml axis is the profile-specific config path) matches the
   current runtime fingerprint; any mismatch fails safe to verbatim preserve.

Adds tests: extra_models-counts-as-advertised (#433 via overflow bucket) and
foreign-profile-catalog-preserves-verbatim (isolation fail-safe). Test helpers
now stamp the source fingerprint exactly as the real publish sites do.
2026-07-12 05:41:08 +00:00
nesquena-hermes 4349ff4302 fix(config): custom/proxy providers keep the full vendor namespace (#5979)
Resolve the model id for custom/proxy providers by ENDPOINT PROVENANCE
instead of a catalog-family guess. The old gate stripped a vendor prefix
whenever the bare id was first-party of the prefix's home vendor, which
could not distinguish a namespace the proxy actually routes on
(x-ai/grok-4.5 -> keep) from a stale cross-provider leftover on a relay
that only serves bare ids (openai/gpt-5.4 -> strip). A model graduating
into the first-party catalog (agent commit adding grok-4.5) silently
flipped a working custom-proxy id from preserved to stripped -> HTTP 400.

New _endpoint_advertised_model_ids() reads the already-published catalog
snapshot (never builds/probes/touches disk; O(1) per send via a
snapshot-identity memo) and answers what the endpoint advertised:
  - full vendor/model advertised -> preserve (#5979, #3872, #548)
  - only the bare id advertised   -> strip the redundant prefix (#433)
  - cold/unbuilt catalog          -> preserve verbatim (fail-safe toward
                                     the id the user actively selected)
The non-custom first-party-proxy strip arm is unchanged.

Rewrites the two pinned #433 tests to model provenance (bare-only
advertised) and adds #5979 coverage: advertised-full preserve (bare +
named custom), cold-catalog preserve-verbatim, and both-advertised
prefers-full.

Thanks @b3nw for the report and precise root-cause analysis.
2026-07-12 05:41:08 +00:00
b2ae71e01a Release exp-v0.52.48: post-compression context metering (#5872, #4685) (#5977)
* fix(#4685): meter post-compression model context

* Release exp-v0.52.48: meter post-compression model context (#5872, #4685)

---------

Co-authored-by: Rod Boev <rod.boev@gmail.com>
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
2026-07-11 20:41:43 -07:00