Greptile's P2 was right: 'upgrade past the sync' had no installable
target. Version audit across core tags: every RELEASED Agent (0.20.5 ->
0.21.1, tag v2026.9.7) still carries ox-alpha-free in the curated floor;
the removal exists only on core main, which self-reports the same
0.21.1 version string as the stale tag — a version number alone cannot
distinguish fixed from unfixed. State that plainly and present the
config allowlist as the only verified workaround until a release notes
the 2026-09-09 catalog sync.
Verified against a real install (core v0.21.0 + live key + endpoint):
- on Agent releases between v0.20.5 and the 2026-09-09 catalog sync,
the delisted ox-alpha-free can STILL be offered — via core's own
live/curated merge AND via _seed_provider_models_from_core() injecting
the installed core's stale curated ids into WebUI's static list.
'Upgrade the Agent past the sync' is the correct guidance, not
'upgrade the WebUI'.
- allowlist precedence re-verified with a real config.yaml on disk
(group == exactly the allowlisted ids).
- note the 4s foreground rebuild budget: first picker open after a cold
start can serve last-known catalog while the live rebuild finishes
out-of-band.
User-visible runtime behavior changed in this PR (live catalog first for
Agent >= 0.20.5, resynced static fallback): document both failure
directions users can now hit — listed-but-failing ids (tier/region gates
on the live endpoint, the #7507 follow-up) and missing new ids on
older-AGENT installs — with the diagnostic (hermes --version + authed
curl of /zen/go/v1/models) and fixes (allowlist precedence, Agent
upgrade). Follows the file's Symptom/Why/Diagnostic/Fix/File-a-bug
format.
Own a private stdout descriptor before agent imports and serialize request log writes. Keep logging failures isolated from HTTP responses. Add successful chat/start subprocess delivery regressions and document header-latency semantics and cron ownership.
On the msvcrt branch of scripts/ensure_state_db_read_indexes.py the tool
locked byte 0 of --lock-file right after open(..., "a+"). When that open
had just created the file (or the file was empty) the lock range lay beyond
EOF, which Windows refuses, so the documented locked-maintenance workflow
raised instead of acquiring the lock (greptile P1).
Write one byte when the lock file is empty, then seek(0) and lock as
before. Only the Windows branch changes; POSIX flock is untouched and an
existing (deployment-owned) lock file is never rewritten.
Tests (tests/test_pr7445_readonly_review_followups.py): a strict msvcrt
stand-in that refuses to lock past EOF proves the lock is taken on byte 0
of a fresh empty file, that an existing lock file keeps its content, and
that a held lock still propagates before the database is touched.
Review follow-ups on #7445 (nesquena-hermes CHANGES_REQUESTED + Greptile).
api/agent_sessions.py
- The read-only pre-aggregated `latest_messages` candidate ordering now
applies to every missing-index projection (default sidebar, gateway
watcher, cron/webhook/kanban views), not only `include_sources ==
("cron",)`. Since the listing no longer self-heals the index, the
correlated `MAX(mx.timestamp)` scan re-ran on every load: 9.86 s on a
5,000-session / 20,000-message DB vs 0.044 s after this change, with
identical selected rows and order. The indexed seek is unchanged.
- `state_db_file_uri()` / `state_db_readonly_uri()` replace
`Path.as_uri()`: a UNC `HERMES_HOME` produced `file://server/share/...`
and CPython's bundled SQLite (3.11-3.13 Windows builds, no
SQLITE_ALLOW_URI_AUTHORITY) rejected the authority, so every agent-backed
session vanished. The builder emits the empty-authority
`file:////server/share/state.db?mode=ro`, keeps `file:///C:/...` for
drive letters, strips `\\?\` / `\\?\UNC\` prefixes, quotes `?`/`#`, and
leaves POSIX output byte-identical. Strict `mode=ro`, no writable
fallback.
scripts/ensure_state_db_read_indexes.py
- `fcntl` / `msvcrt` are imported conditionally; the module imports and
the no-`--lock-file` path runs on native Windows. `--lock-file` uses
`flock` on POSIX, `msvcrt.locking(LK_NBLCK)` on Windows, and fails with
a clear RuntimeError when neither exists. The rw open reuses the
UNC-aware URI builder.
docs/troubleshooting.md
- Operator section: read-only contract, degraded missing-index listing,
drained maintenance prerequisites, lock behaviour, failure modes, UNC.
Tests: tests/test_pr7445_readonly_review_followups.py (19 tests, red
before this change: correlated-subquery plan, missing URI builder,
ImportError on fcntl). Query-plan contract via EXPLAIN QUERY PLAN — no
CORRELATED SCALAR SUBQUERY on missing-index listings, identical rows to
the indexed listing; platform-parametrised URI cases; maintenance module
imported with `sys.modules["fcntl"] = None`. test_issue2628 no longer
pins the cron-only gate; the flock-held test skips without fcntl.
Serve the SPA shell for /sessions so the post-login redirect (when password auth is enabled) no longer 404s.
Co-authored-by: Skawsk59 <Skawsk59@users.noreply.github.com>
* rebase #7160 onto master (report unverified Agent updates, no auto-restart)
Co-authored-by: hrdwdmrbl <hrdwdmrbl@users.noreply.github.com>
* harden agent-update marker read: no-follow, fstat regular-file, bounded (fixes gate CORE)
The Agent update marker is attacker-adjacent shared state. _read_live_agent_update()
used Path.read_text(), which follows symlinks, blocks forever on a FIFO, and reads an
unbounded regular file — a rejected/stale-runtime request could hang or OOM. Now opens
O_RDONLY|O_NONBLOCK|O_NOFOLLOW, fstat-verifies a small regular file, and reads a bounded
max, classifying anything else as 'unknown' (fail-closed). Adds FIFO/oversized/symlink/
happy-path regression tests.
Co-authored-by: hrdwdmrbl <hrdwdmrbl@users.noreply.github.com>
* fix marker-read portability: gate os.open fast-path on O_NONBLOCK+O_NOFOLLOW
os.O_NONBLOCK is Unix-only; accessing it unconditionally raised AttributeError
on native Windows, breaking every stale-revision barrier (chat/compression/etc).
Now both flags resolve via getattr and the os.open() fast path is taken only
when BOTH are available (_MARKER_SAFE_OPEN_AVAILABLE); otherwise fall back to an
lstat-only classification (absent vs unknown) so a fallback platform can never
symlink-traverse or crash. Adds a Windows-fallback regression test.
Co-authored-by: hrdwdmrbl <hrdwdmrbl@users.noreply.github.com>
* docs(changelog): stamp #7160 stale-runtime report + hardened marker read
---------
Co-authored-by: n <a@n>
Co-authored-by: hrdwdmrbl <hrdwdmrbl@users.noreply.github.com>
A fresh root install on Linux (e.g. a new Proxmox/Ubuntu VM) places the Hermes
Agent at /usr/local/lib/hermes-agent and links the CLI into /usr/local/bin,
while the modern `hermes` launcher is a `#!/usr/bin/env bash` wrapper that
exec's the venv entrypoint. The old discovery only checked
~/.hermes/hermes-agent-style paths and parsed the launcher's shebang as a Python
interpreter, so on a root VM discover_agent_dir() returned None, bootstrap built
a WebUI-only .venv, and chat aborted at launch with "Python environment cannot
import both WebUI dependencies and Hermes Agent."
Two complementary fixes:
- Add /usr/local/lib/hermes-agent to discover_agent_dir()'s candidate list.
- Harden _agent_dir_from_hermes_cli() to follow the launcher's exec target (any
absolute path quoted in the wrapper) up to run_agent.py, not just a Python
shebang. Skips /usr/bin/env indirection.
Now `python3 bootstrap.py` works out of the box for root/container installs with
no HERMES_WEBUI_PYTHON override needed. Adds tests for the bash-wrapper shape,
the env-indirection no-false-positive case, and the FHS candidate probe; updates
docs/troubleshooting.md.
Four code-review comments from the automated Copilot reviewer on this PR:
1. `_journal_tool_already_present` dedupe was session-wide, so a
legitimately-repeated tool (e.g. a second `terminal: ls` in an
earlier turn) could cause the retry path to falsely skip
materializing the recovered tool card. The helper now takes a
keyword `stream_id` argument; when supplied, a tool card whose
`_recovered_stream_id` is set AND differs from the candidate is no
longer treated as a duplicate. Untagged tool cards (live tools, or
tool cards carried over from a pre-tagging core transcript) still
match, preserving the existing 'core transcript already has this
tool, don't duplicate' invariant. Two new tests in
`TestJournalToolDedupeScoping` cover both legs of the rule.
2./3. The troubleshooting FAQ pointed at `~/.hermes/webui/sessions/session_<sid>.json`
and `~/.hermes/_run_journal/...`. The actual sidecar filename has
no `session_` prefix and the run-journal lives under the WebUI
sessions dir (`~/.hermes/webui/sessions/_run_journal/<sid>/<stream>.jsonl`,
default). Both paths fixed and an explicit note added about
`HERMES_WEBUI_STATE_DIR` overriding the state root.
4. Drop unused `json` / `queue` / `Path` imports from
`tests/test_session_lost_response_regression.py` so the file stops
carrying noise that future linting would flag.
CHANGELOG: append an Unreleased / Fixed entry describing the user-visible
behaviour change (interrupted-turn marker now self-heals on the next
session read; gives up gracefully after 12 retries or 24h).
docs/troubleshooting.md: add a 'Symptom → Why → Diagnostic → Fix →
Caps → When to file a bug' entry for the
'no agent output was recovered' marker so users who hit the lost-response
shape on WSL2 / network FS can recognise it, verify the run-journal on
disk, and know that reloading the session is enough.
Closes#1695.
@Patrick-81 reported the bare "AIAgent not available -- check that
hermes-agent is on sys.path" error on a symlinked install (~/Programmes/hermes-agent
linked to ~/hermes-agent). The maintainer's response — three diagnostic
commands plus `pip install -e .` in the agent dir — fixed it for them.
This PR captures both halves of that learning so the next user with the
same shape doesn't have to file a new issue:
1. **Error message diagnostic block.** New helper
`_aiagent_import_error_detail()` in api/streaming.py builds a multi-line
diagnostic when the import fails, including:
- the running Python interpreter
- HERMES_WEBUI_AGENT_DIR (set value, or "(not set)")
- sys.path entries that mention hermes/agent (or "no entries mention..."
— itself a strong diagnostic signal)
- the most-common fix (`pip install -e .` in the agent dir)
- a pointer to docs/troubleshooting.md
The original error message string is preserved as the FIRST line so
existing log scrapers and docs-search keep matching.
Helper is kept as a separate function so it stays out of the hot path
until we actually need to raise — building it on every successful import
would be wasted work.
2. **New docs/troubleshooting.md.** Symptom → Why → Diagnostic commands →
Fix → When-to-file-a-bug template, with one entry to start: the
"AIAgent not available" flow Patrick-81 walked through. Future
recurring failure modes follow the same template. Required a one-line
addition to .gitignore — docs/* is gitignored with an allowlist, and
the new file needed `!docs/troubleshooting.md` to be tracked.
3. **README link.** docs/troubleshooting.md added to the `## Docs` section
so users know where to look first.
13 regression tests in tests/test_1695_aiagent_import_error_detail.py:
9 for the helper output shape (preserves original message line, includes
running python, shows HERMES_WEBUI_AGENT_DIR set/unset both ways, includes
pip-install-e hint, points at troubleshooting doc, lists relevant sys.path
entries when present, says "no entries..." when absent, output is multi-line)
plus 4 for the docs-presence regression (file exists, has the AIAgent
section, includes pip install -e ., describes the diagnostic chain with
readlink + agent/__init__.py verification).
190 streaming/aiagent tests pass after the change. ast.parse on
api/streaming.py clean.
CI failure on prior push was due to the docs/* gitignore swallowing the
new troubleshooting.md file silently — this commit adds the allowlist
entry so the file is tracked.
nesquena-hermes