Rebase PR #5162 (rename busy_input_mode -> default_message_mode; flip the
default from 'queue' to 'steer') onto current origin/master WITHOUT dropping
the shipped #5170 localStorage persistence mirror.
Rename the mirror machinery to the new setting name for consistency:
_BUSY_INPUT_MODES -> _DEFAULT_MESSAGE_MODES (values unchanged)
_normalizeBusyInputMode -> _normalizeDefaultMessageMode (fallback now 'steer')
_persistBusyInputMode -> _persistDefaultMessageMode
_readPersistedBusyInputMode -> _readPersistedDefaultMessageMode
window._busyInputMode -> window._defaultMessageMode (+ renamed exports)
localStorage: write the new 'hermes-default-message-mode' key; read it with a
fallback to the legacy 'hermes-busy-input-mode' key so an existing user's
persisted preference survives the rename.
Preserve #5170 behavior at every mirror site under the new names:
- boot success -> window._defaultMessageMode=_persistDefaultMessageMode(...)
- boot FAILURE -> window._defaultMessageMode=_readPersistedDefaultMessageMode()
(NOT a hardcoded 'steer' — a saved 'interrupt'/'queue' must still apply when
the server is unreachable; do not regress #5167/#5132)
- preferences autosave, settings-panel load, and _applySavedSettingsUi all
persist through _persistDefaultMessageMode(...)
Tests updated for the rename while keeping the persistence-behavior assertions
(test_1062, test_5145, test_5167); test_5167 gains explicit guards that the
load-failure path reads the persisted pref and never hardcodes a literal mode,
plus autosave/panel-load mirror-write coverage.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
The sidebar session-list GET had 502/503/504 retry logic, but it was gated
to cold boot only. Once `_sessionListHasLoadedOnce` flipped true, every later
refresh (profile switch, focus/visible/reconnect) shipped no retryStatuses, so
a transient 502 during an nginx->backend restart window failed on the first
attempt and left the sidebar stale until a hard reload (Ctrl+F5).
The session-list GET is idempotent, so retrying it is safe unconditionally.
This moves `retries:1` + `retryStatuses:[502,503,504]` into the base request
options so they apply to every refresh, while keeping the larger boot timeout
(`_SESSION_LIST_BOOT_TIMEOUT_MS`) and `retryTimeouts` boot-only. The api()
wrapper in static/workspace.js already retries when the error status is in
retryStatuses, so no other change is needed.
Extends the existing source-string regression test to assert the retry
options are now always present (declared before the boot-only gate) while the
boot path still carries the timeout + timeout retry.
Reported and root-caused by @weidzhou, who traced the boot-only retry gate.
Co-authored-by: weidzhou <weidzhou@users.noreply.github.com>
The regression test re-imports api.profiles in isolation by stubbing
flask/yaml/agent in sys.modules and deleting+reloading the real api /
api.profiles modules. Teardown only popped api/api.* (never restoring the
real modules) and left the flask/yaml stubs behind, so the manipulation
LEAKED: subsequent tests re-imported api.config/api.routes against the stub
yaml (safe_load->None) and a half-populated api package, silently breaking
~120 unrelated tests in the full serial suite (e.g. MCP/provider/config
tests whose get_config patch no longer saw real config). Snapshot every
sys.modules key we touch and restore it exactly (real modules back, injected
stubs removed) in a finally block. Full suite now matches master baseline
(11537 passed, only the 2 known cron-isolation artifacts).
- _repair_bare_custom_provider_model matches display-named providers via
_custom_provider_slug_from_name (custom:my-proxy matches 'My Proxy').
- _ordered_custom_provider_model_ids now handles dict keys, list strings,
and list dicts with id/model/name, aligned with api/config.py catalog.
- config_obj=None fallback uses get_config() instead of raw cfg.
- _read_profile_model_config returns profile config dict too, avoiding a
second YAML parse on hot display paths.
- Tests added for slug matching, list-form models, and get_config fallback.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
The two-tier mtime cache from #4783 fixed the per-request SKILL.md rescan
but left two concurrency holes that only bite at container cold start,
when the frontend fires several profile-data requests at once and the
caches are empty:
1. `_get_profile_skills_stats()` had no lock, so concurrent misses on the
same profile each ran `os.walk(followlinks=True)` + parsed every
SKILL.md simultaneously.
2. `_build_profile_rows_fast()` ran outside `_LIST_PROFILES_CACHE_LOCK`
in `list_profiles_api()`, so every concurrent request rebuilt all rows
(each walking every profile's skill tree) at once.
With ThreadingHTTPServer (one OS thread per request) and Docker overlay2,
this stacked thousands of concurrent stat() calls and stalled workers
57-70s (per the report's thread dumps).
Fix:
- Add a per-profile compute lock (registry guarded by a meta-lock) and
use double-checked locking in `_get_profile_skills_stats()`: concurrent
misses on one profile collapse to a single compute, while independent
profiles still compute in parallel.
- Single-flight the row build in `list_profiles_api()` by holding
`_LIST_PROFILES_CACHE_LOCK` across the build + cache write. Lock order
is strictly list-lock -> per-profile skills-lock, so no deadlock.
The report's third suggestion (debounce the mtime probe) is deliberately
NOT taken: the every-call cheap probe is the #4783 out-of-band
change-detection contract (test_issue4783 asserts it MUST run on every
call). Serializing the misses removes the herd without weakening that
contract, since only the expensive compute is guarded, not the probe.
Adds tests/test_issue5364_skills_stats_thundering_herd.py proving the
herd collapses (single compute / single build under a concurrent burst),
independent profiles still parallelize, and the every-call probe contract
is preserved. All existing #4783 contract tests still pass.
Co-authored-by: claw-io <claw-io@users.noreply.github.com>
_session_message_content_key (the state.db reconciliation key in
api/models.py) normalized whitespace only, while the streaming-side
identity _message_identity strips the workspace prefix for user turns.
WebUI sends the model a workspace-prefixed user_message
([Workspace::v1: /path]\n<text>) while the visible/optimistic bubble and
sidecar row carry the bare <text>. The mismatch made a prefixed state.db
row and a bare sidecar row key differently, so state_db_delta_after_context
failed to align them, treated the state.db copy as new, and appended a
duplicate user turn. The agent-side merge then concatenated the two
adjacent user rows into a permanent composite -- the post-restart
stale-user-prepend bug.
Fix: strip the workspace prefix for role=='user' in the reconciliation
key, reusing the same _strip_workspace_prefix helper the streaming side
uses (lazy import to avoid the api.streaming -> api.models cycle) so the
two dedup layers can't drift again. Assistant/tool keys are unchanged and
prefix-free user messages key identically (idempotent).
Fixes#5339.
Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
nesquena-hermes