Layer three additive pieces on top of the #7021 / #6931 continuation
predicate without changing its shape or CONTINUATION_STARTED_AT_TOLERANCE_SECONDS:
- _reset_from is a boundary marker alongside _branched_from/_delegate_from
(Hermes Agent stamps it in gateway/session_recovery.py and filters it in
_NON_CONTINUATION_CHILD_FILTER_SQL). The same predicate drives the sidebar
listing, lineage metadata/report and the transcript stitcher.
- _branch_markers() returns ('none' | 'markers' | 'unknown', markers);
unparsable, non-object, too-deep, or non-string marker payloads are
'unknown' and fail closed as a lineage boundary instead of degrading to
"no markers" (fail open).
- gateway_watcher._snapshot_hash() hashes the complete canonicalized published
payload, and the cheap fingerprint covers model_config, so a projection change
to title/created_at/metadata with fixed (id, updated_at, message_count) emits
sessions_changed.
_poll_once() resets the parity timestamp when it runs with no subscribers so
the next subscription gets a fresh projection, but since the idle-park change
(#7694) the poll loop parks before calling _poll_once() once the last
subscriber leaves. A tab that reconnects inside the parity interval then
resumes with the stale cheap fingerprint and skips the projection.
Reset _last_cheap_fp and _last_full_projection_at in unsubscribe() when the
last subscriber is removed (not for an unknown queue, not while others
remain).
Stop the permanent ~7% core burn from re-fingerprinting state.db every 5s
and spinning on 0.1s stop-checks while nobody is listening. Block on
Event.wait for the poll interval, and sleep on _idle_wake until the first
subscriber arrives (maint #3035).
Co-authored-by: Cursor <cursoragent@cursor.com>
Readers never upgrade to writers. open_state_db_readonly() opens
file:...?mode=ro only; a read-only open failure propagates instead of
silently retrying with a write-capable handle on the multi-GiB WAL
database the agent is streaming into. The session listing no longer
self-heals idx_messages_session with CREATE INDEX (a writer lock held for
minutes on a large messages table) — a missing index degrades to the
pre-aggregated path. Cron sidebar info, insights and deep health checks in
api/routes.py use the same read-only opener.
Watcher (api/gateway_watcher.py): _poll_once returns without touching the
database when nobody is subscribed, and forces a fresh full projection on
the next subscription.
Maintenance (scripts/ensure_state_db_read_indexes.py): the covering read
indexes (idx_messages_session, idx_messages_session_role with NOCASE,
idx_sessions_webui_fingerprint) are created only under --confirm-drained,
optionally holding an exclusive flock on --lock-file for deployments that
serialise agent turns on a lock file; existing indexes are verified for
shape/collation and coverage (EXPLAIN QUERY PLAN), never silently
replaced.
Tests: tests/test_state_db_readonly_readers.py (read-only failure never
retries writable; no poll without subscribers; listing creates no index;
index tool requires drain, is idempotent, works without a lock file and
refuses a held lock; cron sidebar read is mode=ro). Neighbouring tests
that asserted the retired writable fallback / self-heal writer
(test_issue3887_index_prime, test_issue5455_listing_readonly_connection,
test_issue5455_lineage_readonly_reads, test_issue2628_cli_sessions_perf,
test_issue1494_state_db_fd_leak) now assert the read-only contract;
test_issue3506 subscribes before polling.
The session-listing path already opens the live agent state.db read-only
(file:...?mode=ro) so a write-capable handle doesn't add checkpoint/lock
surface while the agent streams into the same WAL DB (#5455). Three pure-read
projections were missed and still opened a read-WRITE connection:
- read_session_lineage_report
- read_session_lineage_metadata
- the gateway-watcher fingerprint projection (a 5s poll)
Route all three through a shared open_state_db_readonly() helper that mirrors
the listing path: read-only file: URI with a writable fallback that warns. The
self-heal write path (missing idx_messages_session) is intentionally left
writable. No behavior change beyond open mode; reads perform zero writes.
Refs #5455
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* stage-3629: scope gateway watcher to active profile (profile-keyed registry), rebased onto master
* stage-3629: + stop-race-safe subscribe() (Codex CORE) + regression test
Codex caught a TOCTOU between get_watcher() and subscribe(): a watcher reaped as
subscriber-free during a concurrent profile switch left a late subscriber's queue
without the None sentinel -> SSE loop hangs open (keepalives, no events). Opus
independently found the same race (rated follow-up). subscribe() now checks
_stop_event under _sub_lock after appending and self-enqueues the sentinel.
test_subscribe_after_stop_gets_sentinel_immediately pins it.
* stage-3629: + reconnect gateway SSE to new profile on switch (Codex CORE #2)
Second Codex finding: after a profile switch this tab's _gatewaySSE stayed
subscribed to the PREVIOUS profile's watcher (switchToProfile never restarted it;
the probe reattach is gated on !_gatewaySSE which is false while the old ES is
open). switchToProfile() now calls startGatewaySSE() after S.activeProfile is set
— it closes the old ES + reconnects with the new profile cookie, self-gating on
_showCliSessions.
* Release PF: scope gateway watcher to active profile (#2848)
rodboev's #3629, re-pushed to fix both bounce findings (client SSE reconnect +
singleton→profile-keyed registry). Two Codex CORE races caught + fixed inline:
(1) stop/subscribe sentinel race, (2) tab's own SSE staying on the old profile
after switch. Codex SAFE + Opus SHIP (architecture) + gateway-watcher tests green.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
## Release v0.51.256 — Release HX (stage-r4)
Performance — bound WebUI memory growth & idle CPU on large installs.
### Fixed
| Issue | Author | Fix |
|-------|--------|-----|
| #3506 | @nesquena-hermes (reported w/ profiling by @djenttleman) | On a large install (~615 sessions / 40k messages / 454 MB state.db) the WebUI process climbed ~100 MB → ~1.5 GB RSS over days and held high idle CPU. Three root causes fixed: (1) `session_lifecycle._sessions` grew unbounded → new `discard_session()` drops the entry at agent-eviction boundaries, only when no in-flight commit / no uncommitted memory work (retry invariant preserved); (2) cache caps now operator-tunable (`HERMES_WEBUI_AGENT_CACHE_MAX` default 50→25, `HERMES_WEBUI_SESSIONS_MAX`); (3) GatewayWatcher computes a cheap fingerprint before the expensive per-session `MAX(messages.timestamp)` projection and only re-projects on change. |
### Rebase + review notes
- Rebased onto current master; the code diff was verified **byte-identical to the nesquena-APPROVED head** at rebase time (only CHANGELOG re-resolved).
- The Codex regression gate then surfaced **two correctness gaps** the approval didn't catch, both fixed here with regression tests:
1. **Watcher fingerprint missed same-count transcript rewrites.** `/retry`,`/undo`,`/compress` (`SessionDB.replace_messages`) rewrite messages with new timestamps but can leave `message_count` unchanged → stale sidebar `last_activity`. Fixed with a **per-session** grouped message aggregate (`id, count, user_count, MAX(timestamp)`) over the same non-excluded sessions (a global MAX would miss a rewrite of an older, non-newest session); cron/webui stay excluded so idle churn still doesn't re-project.
2. **LRU agent-cache eviction could close a live worker's agent** (`popitem(last=False)`, liveness-blind — pre-existing, but the lower 50→25 cap made it more likely). Eviction now snapshots `ACTIVE_RUNS` session_ids (before the cache lock — no nested lock) and skips live sessions, deferring (temporarily exceeding cap) rather than closing a live agent.
### Gate
- Full pytest suite: **7612 passed, 0 failed** (one boot-cascade flake re-run; clean on re-run)
- ruff: CLEAN · Codex (regression): 4 rounds → both gaps + a stale test fixed → **SAFE TO SHIP**
Co-authored-by: nesquena <nesquena@users.noreply.github.com>
* fix: remove orphaned i18n keys from top-level LOCALES object
Three Traditional Chinese translation keys (cmd_status, memory_saved,
profile_delete_title) were placed outside any locale block between the
en and ru blocks in static/i18n.js. They became top-level properties
of the LOCALES object, causing them to appear as invalid language
options in the Settings > Preferences dropdown.
The correct translations already exist in the zh-Hant locale block.
Fixes#1008
* fix: block stale SSE events from polluting new session's DOM
- appendThinking(): guard with !S.session||!S.activeStreamId to drop
events from a previous session's SSE stream during a session switch
- appendLiveToolCard(): same guard for consistency
- finalizeThinkingCard(): scroll thinking-card-body to top when
scroll is pinned, so completed response is immediately visible
- appendThinking(): auto-scroll thinking card body to bottom while
streaming if user is watching (scroll pinned)
* Fix empty agent sessions in sidebar
* fix: resolve cron UI UX issues — icon ambiguity, toast overlap, running status
Fixes#995 — three sub-issues in the Cron Jobs UI:
1. Dual play icons ambiguous: Resume button now shows a distinct
play+bar icon (play triangle + vertical line) instead of the
identical triangle used by Run now.
2. Toast notification overlapping header buttons: Added
position:relative; z-index:10 to .main-view-header so it
stacks above the fixed toast (z-index:100 within its layer).
3. No running status after trigger: After triggering a job, the
status badge immediately shows 'running…' with a CSS spinner
animation, and polls the cron list every 3s (up to 30s) to
refresh when the job completes.
- Added cron_status_running i18n key in all 5 locales (en, es, de, ru, zh, zh-Hant)
- Added .detail-badge.running CSS class with spinner animation
- New functions: _setCronDetailStatus(), _startCronRunningPoll()
* fix(#1011): address review feedback — poll cleanup, badge persistence, 30s fallback
- _clearCronDetail() now clears _cronRunningPoll interval on navigation
- Poll re-applies 'running' badge after loadCrons() re-render (prevents flicker)
- When poll ends (30s max), detail re-renders with actual status as fallback
* feat: create folder and add space directly from UI (#782)
- After creating a folder via the file tree New folder button, offer to add it as a space via confirm dialog
- Add Create folder if it doesnt exist checkbox in the New Space form
- Backend: support create flag in /api/workspaces/add to mkdir before validation
- i18n: 4 new keys (folder_add_as_space_title/msg/btn, workspace_auto_create_folder) in all 6 locales
* fix: validate workspace path before mkdir to prevent orphan directories
Review feedback (critical): the previous code called mkdir() before
validate_workspace_to_add(), which meant a rejected path (e.g. system dir)
would leave an orphan directory on disk.
New flow:
1. Resolve path and check against blocked system roots BEFORE any mutation
2. mkdir() only if path passes the blocklist check
3. Full validation (exists, is_dir) after mkdir
Also imports _workspace_blocked_roots for the pre-mutation blocklist check.
* fix(#1014): classify model-not-found errors with helpful message
- Add model_not_found error type to streaming.py exception classifier
- Detect 404, 'not found', 'does not exist', 'invalid model' patterns
- Strip HTML tags from provider error messages (nginx 404 pages, etc.)
- Add model_not_found branch to apperror handler in messages.js
- Add i18n key model_not_found_label in all 6 locales
- 15 tests covering detection, sanitization, frontend, and i18n
* feat(ui): add live TPS stat to header
Adds a TPS (Tokens Per Second) chip to the right of the header title bar
that updates live while AI output is streaming.
Metering (api/metering.py)
- Tracks per-session output + reasoning tokens via GlobalMeter singleton
- Per-session TPS = total_tokens / elapsed_time
- Global TPS = average of active sessions' TPS values
- HIGH/LOW are max/min of global_tps snapshots over a 60-minute rolling
window (only recorded when > 0, so idle periods are excluded)
- Thread-safe with a single lock
Metering events emitted from streaming.py
- Throttled at 100ms from token/reasoning/tool callbacks so the display
updates rapidly during fast token streams
- 1Hz ticker as fallback for slow streams (exits when no active sessions)
- Final stats emitted on stream end
Routes (api/routes.py)
- Removed POST /api/metering/interval endpoint (dynamic interval via
focus/blur was replaced with simple always-1s-when-active approach)
UI (static/messages.js, index.html, style.css)
- TPS chip in titlebar: shows 'N.N t/s . N.N high . N.N low'
- Default: '0.0 t/s . 0.0 high' when idle
- Display updates on every metering SSE event (throttled to 100ms)
* feat: session restore speed + title gen reasoning hardening (#1025, #1026)
PR #1025 (@franksong2702): Speed up large session restore paths
- GET /api/session?messages=0 now parses only metadata before the messages array
- Metadata-only loads no longer populate the full-session LRU cache
- Frontend lazy fetch uses resolve_model=0 to avoid cold model-catalog lookup
- Hard reload no longer waits for populateModelDropdown() before restoring session
PR #1026 (@franksong2702): Harden auto title generation for reasoning models
- Raises title-gen completion budget to 512 tokens (reasoning-safe)
- Retries once with 1024 tokens on empty content / finish_reason:length
- Applies retry to both auxiliary and active-agent fallback routes
- Preserves underlying failure reason in title_status on local fallback
Co-authored-by: Frank Song <franksong2702@gmail.com>
* feat: session attention indicators in right slot + last_message_at timestamps (#1024)
PR #1024 (@franksong2702): Polish session attention indicators
- Streaming spinners and unread dots now reuse the right-side actions slot
- Running/unread rows hide timestamps; idle/read rows keep right-aligned timestamps
- Date group carets point down when expanded, right when collapsed
- Pinned group no longer repeats pinned-star icon per row
- Running indicators appear immediately after send (local busy state while /api/sessions catches up)
- Sidebar sorting/grouping/timestamps now prefer last_message_at (derived from last real message)
so metadata-only saves don't make old sessions appear under Today
Co-authored-by: Frank Song <franksong2702@gmail.com>
* docs: v0.50.207 release notes — 10 PRs, 2169 tests (+36)
---------
Co-authored-by: bergeouss <bergeouss@users.noreply.github.com>
Co-authored-by: Josh <josh@fyul.link>
Co-authored-by: Frank Song <franksong2702@gmail.com>
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
* fix(sessions): surface gateway SSE failures and add polling fallback
- add a JSON probe mode for the gateway SSE endpoint
- detect watcher-unavailable 503s from the browser
- fall back to periodic session refresh with a toast
- add probe payload tests and endpoint coverage
Fixes#635
* fix(sessions): surface gateway SSE failures and add polling fallback (#826)
Absorbed from PR #826 by @cloudyun888 (fixes#635).
When the gateway watcher thread is not running, the browser now shows a
toast notification and falls back to 30-second periodic polling for session
sync. Previously the SSE failure was completely silent with no user feedback.
Changes from original PR:
- Deleted misplaced test_gateway_sse_probe_unit.py (was at repo root, not
discovered by `pytest tests/`); unit tests moved into tests/test_gateway_sync.py
- _gateway_sse_probe_payload now checks watcher._thread.is_alive() rather
than just watcher is not None — a watcher instance with a dead poll thread
now correctly reports unavailable and activates the polling fallback
- probeGatewaySSEStatus catch(e) now starts the polling fallback on network
error rather than silently swallowing the failure
- Added 5 unit tests covering all watcher-alive/dead/missing/disabled branches
Co-authored-by: cloudyun888 <269269188+86cloudyun-afk@users.noreply.github.com>
* cleanup(gateway): public is_alive() + dedup probe/live watcher-alive check + changelog
Three small cleanups on top of @cloudyun888's PR #826 absorption:
1. Add GatewayWatcher.is_alive() public accessor so routes.py doesn't
reach into the private _thread attribute. The existing private-
attribute check stays as a defensive fallback for any older in-
memory instance or test double that doesn't implement the full API.
2. Dedupe the watcher_alive computation in _handle_gateway_sse_stream:
the live-SSE path now calls _gateway_sse_probe_payload(...) and reads
its watcher_running field instead of re-deriving the same logic
inline. Keeps probe and SSE in sync automatically.
3. CHANGELOG trailer was (#826, fixes#635, @cloudyun888) — this PR is
#828, so updated to (#828, absorbs PR #826 by @cloudyun888, fixes
#635) matching the repo convention for absorbed PRs (see #805).
Added two regression tests:
- test_gateway_watcher_is_alive_public_method — covers the three
lifecycle states (before start, while running, after stop).
- test_probe_payload_prefers_public_is_alive — asserts the probe
uses watcher.is_alive() rather than poking _thread when the
public method exists.
Full suite: 1735 passed, 0 new failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: cloudyun888 <269269188+86cloudyun-afk@users.noreply.github.com>
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
Co-authored-by: Nathan Esquenazi <nesquena@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add real-time gateway session sync (Phase 1)
- Add gateway_watcher.py: background daemon polling state.db every 5s
for gateway session changes (telegram, discord, slack, etc.)
- Extend get_cli_sessions() to include all non-webui sources
- Add SSE endpoint /api/sessions/gateway/stream for real-time push
- Add dynamic source badges (telegram=blue, discord=purple, slack=dark purple)
- Rename 'Show CLI sessions' to 'Show agent sessions'
- Wire watcher lifecycle into server start/stop
- 10 tests covering metadata, filtering, SSE, and watcher lifecycle
- Activated via the same checkbox as CLI session import
Addresses GitHub issue #272
* fix: SSE event name mismatch, TLS attribute, remove PLAN.md
- Fix critical SSE bug: frontend listened for 'gateway_session_update'
but backend sends 'sessions_changed' -- events were silently dropped
- Fix frontend field check: data.changed -> data.sessions (matches
the actual payload structure from gateway_watcher)
- Fix TLS: ssl.TLSv1_2 -> ssl.TLSVersion.TLSv1_2 (the bare attribute
does not exist, would crash TLS setup and silently fall back to HTTP)
- Remove PLAN.md: implementation plan should not be committed to repo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: test isolation and slow-consumer sentinel in gateway sync
tests/test_gateway_sync.py:
- Fix _get_test_state_dir() path mismatch: the function was computing
HERMES_HOME/webui-mvp-test but conftest.py sets HERMES_HOME=TEST_STATE_DIR,
so state.db was written to a double-nested path the server never read.
Now uses HERMES_WEBUI_STATE_DIR first (which conftest sets directly to
TEST_STATE_DIR), fixing the 7/10 test failures in full-suite ordering.
- Fix conn cleanup: removed conn.close() from inside try blocks so the
connection stays valid for _remove_test_sessions() in the finally block.
Previously the closed conn caused ProgrammingError in finally (swallowed
by bare except), leaving ghost sessions in state.db on test failure.
api/gateway_watcher.py:
- Fix slow-consumer queue eviction: when a subscriber queue fills (>10 events)
and is removed from _subscribers, now puts a None sentinel into it so the
SSE handler unblocks and closes the connection, letting EventSource
auto-reconnect. Without this the connection stayed open but received no
further events.
* fix: test isolation — set HERMES_WEBUI_TEST_STATE_DIR in conftest
The gateway sync tests write directly to state.db and must use the same
path the test server reads from. Previously they computed the path
independently, which broke when test_auth_sessions.py set a different
HERMES_WEBUI_STATE_DIR in the test-process environment at import time.
tests/conftest.py:
- Set HERMES_WEBUI_TEST_STATE_DIR=TEST_STATE_DIR in the test process's
os.environ (via setdefault) so gateway tests can read it reliably.
Using setdefault preserves any explicit override the caller may pass.
tests/test_gateway_sync.py:
- Simplify _get_test_state_dir(): check HERMES_WEBUI_TEST_STATE_DIR first
(now reliably set by conftest), fall back to HERMES_HOME/webui-mvp-test.
Remove the workaround that tried to snapshot HERMES_HOME at import time.
Result: 658/658 tests pass in full-suite ordering (was 651 pass / 7 fail).
---------
Co-authored-by: bergeouss <bergeouss@users.noreply.github.com>
Co-authored-by: Nathan Esquenazi <nesquena@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Anthony Ruiz