The Docker smoke gate added in this same PR caught a real production
regression on its very first CI run. v0.51.84 (PR #2470) mounted
hermes-agent-src read-only on the WebUI side and widened the chown
prune to keep the read-only walk happy, but missed that the WebUI's
startup also runs:
uv pip install "$_agent_src[all]"
against the same now-read-only mount. setuptools' egg_info step writes
hermes_agent.egg-info/ inside the source tree even under PEP 517 build
isolation (this is by design -- PEP 517 isolates the BUILD environment,
not the source tree's metadata directory). On a :ro mount this returns
EROFS, the install fails, error_exit fires, and every multi-container
deploy dies at startup. The smoke gate flagged it on both the
two-container and three-container variants.
The fix
-------
Stage the agent source into a writable build dir under /tmp BEFORE
invoking pip install, then point pip at the staged copy.
_stage_src="/tmp/hermes-agent-build"
rm -rf "$_stage_src" && mkdir -p "$_stage_src"
rsync -a --exclude='*.egg-info' --exclude='build' --exclude='dist' \
--exclude='__pycache__' --exclude='.git' \
"$_agent_src"/ "$_stage_src"/
uv pip install "$_stage_src[all]" ...
rm -rf "$_stage_src"
The exclusion list matters: when setuptools sees a pre-baked *.egg-info,
build, or dist directory, it takes a timestamp-update code path that
also reads/writes inside that directory -- which itself fails on a :ro
source. Excluding them keeps the build on the fresh-build path
unconditionally.
rsync is in the production image (Dockerfile line 41-44). For users
running custom WebUI images without rsync, the script falls back to
cp -a + post-copy rm -rf of the same artifacts.
Tests
-----
Two new source-level invariants in tests/test_docker_docs_and_readonly.py:
test_docker_init_stages_agent_source_for_writable_install
-- asserts _stage_src=... is declared
-- asserts every `uv pip install ...[all]` line uses _stage_src,
NOT raw $_agent_src
test_docker_init_excludes_egg_info_during_staging
-- asserts the staging path excludes *.egg-info (rsync exclude
form or cp-fallback's explicit rm -rf both pass)
These would have caught the v0.51.84 regression at the source level
(once written; they're new). The Docker runtime smoke gate is the
durable defence for the broader class of :ro x init-script
interactions, since source-level invariants only catch what they're
written to catch.
Verification
------------
- pytest tests/test_docker_docs_and_readonly.py: 11 passed (9 existing
+ 2 new)
- pytest tests/ -q --timeout=60: 5891 passed, 6 skipped (was 5889;
delta is exactly the 2 new tests)
- bash -n docker_init.bash: clean
Once this lands, the Docker smoke gate's two/three-container variants
should go green, completing the self-validating loop.
Closes the source-only-test gap that let v0.51.84's :ro-mount x chown -h
{} + startup regression reach review with 5800+ green pytests. Adds a
new GitHub Actions workflow .github/workflows/docker-smoke.yml that
actually runs 'docker compose up' against each compose variant.
Triggers
--------
Path-filtered on pull_request + push to master:
Dockerfile, docker_init.bash, docker-compose*.yml, .dockerignore,
.env.docker.example, .github/workflows/docker-smoke.yml itself.
Also workflow_dispatch for manual runs.
Jobs
----
1. compose-config -- preflight that 'docker compose config' parses each
of the three compose files. Cheap, fast, catches schema/interpolation
drift in parallel before any container starts.
2. smoke (matrix: single / two-container / three-container) -- for each
variant:
a. Reap any leftover hermes-smoke-* containers/volumes/networks from
prior runs (defence-in-depth on self-hosted runners; hosted runners
are fresh).
b. docker build -t ghcr.io/nesquena/hermes-webui:latest .
Critical: the multi-container compose files reference the GHCR
image. Without this retag, multi-container smoke would test the
previously-released image, NOT the PR's docker_init.bash / Dockerfile
changes. With the retag, Compose's default pull_policy=missing keeps
the local build in place and the PR is genuinely exercised.
c. mktemp -d for ephemeral HERMES_HOME + HERMES_WORKSPACE so the
runner's host filesystem is never touched.
d. docker compose up -d --wait --wait-timeout 120 (Dockerfile carries a
HEALTHCHECK so --wait blocks on 'healthy', not just 'running').
e. curl /health probe with a 30-attempt x 2s poll loop as headroom for
the multi-container variants' Python dep install phase.
f. grep startup logs for known-bad signatures:
EROFS | Read-only file system | Traceback | PermissionError |
error_exit | groupmod: cannot | usermod: cannot |
Failed to set (UID|GID|owner|permissions|ownership)
These are the exact patterns that would have flagged #2470 in real
time. Failed-to-set is anchored to specific objects to avoid false
positives on benign locale/library bootstrap warnings.
g. trap on EXIT: docker compose down -v --remove-orphans + rm -rf the
ephemeral host paths, regardless of how the job exited.
Safety
------
- permissions: contents: read only -- no GITHUB_TOKEN write scope.
- Fork PRs run with no secrets (standard pull_request, not
pull_request_target).
- No host bind mounts; no ~/.hermes exposure; no network egress beyond
what compose itself needs to pull the agent image.
- timeout-minutes: 15 on the smoke job as a hard ceiling against a
hung docker build.
- Per-run COMPOSE_PROJECT name (hermes-smoke-VARIANT-RUNID-ATTEMPT)
so concurrent runs or reruns can't clobber each other.
Out of scope for v1 (per design review)
---------------------------------------
- HERMES_WEBUI_SMOKE_TEST env flag in docker_init.bash -- production-code
footgun that would let any leaked env var silently exit before
serving traffic.
- --user 60000:60000 -- incompatible with the image's root-init phase
and would skip the very chown branch we are guarding against.
- Local-runnable scripts/docker-smoke-test.sh -- defer until CI gating
ships and we see what contributors actually trip over.
- Hadolint / yamllint -- separate lint workflow, follow-up PR.
- Podman runtime smoke -- defer until a podman-specific bug ships.
Pre-merge verification
----------------------
- actionlint: clean
- YAML parse: clean (3 triggers, 2 jobs, 3-variant matrix)
- bash -n on all 6 run-blocks: clean
- pytest tests/ -q --timeout=60: 5889 passed, 6 skipped (no test impact;
workflow-only change)
- Opus design review on the brief (REVISE -> minimum scope adopted)
- Opus implementation review on this workflow (APPROVE)
The hermes-agent-src named volume in the two- and three-container compose
files is initialised from the agent image's /opt/hermes on first `up` and
Docker reuses it verbatim on every subsequent `up` — even after a fresh
`docker pull` of the agent image. This was the root cause of #1416 (the
'missing entrypoint' symptom was a stale cached volume hiding the new
image's source tree).
Changes:
- Add an 'Upgrading the agent container' section to docs/docker.md with
the canonical `down → docker volume rm → pull → up -d` recipe, plus the
same pointer as a comment block in both multi-container compose files
near the volume declarations.
- Switch the WebUI's hermes-agent-src mount to `:ro` in both multi-container
compose files. The WebUI only reads this volume to install the agent's
Python deps at startup; mounting it read-only enforces that at the kernel
layer and brings the actual mount mode in line with the existing
docs/docker.md architecture diagram (which already labelled this edge as
read-only).
- Align the workspace bind default in both multi-container compose files
with the single-container convention — `${HERMES_WORKSPACE:-${HOME}/workspace}`
instead of `${HERMES_WORKSPACE:-~/workspace}` — so the default resolves
the same way across Linux, macOS, WSL2, and Docker Desktop on Windows.
- Add a 'What the multi-container setup isolates (and what it doesn't)'
section to docs/docker.md to frame the two/three-container setups as
process/network/resource isolation, not filesystem isolation, so users
don't reach for multi-container expecting a trust boundary it doesn't
provide.
- Cross-link #1416 from the Related issues section.
Adds 9 regression tests in tests/test_docker_docs_and_readonly.py covering:
- :ro on the WebUI side of hermes-agent-src in both files
- agent side stays read-write (still needs to populate /opt/hermes on first run)
- ${HOME} (not ~) in workspace bind defaults in both files
- single-container file already uses ${HOME} (pin to prevent drift)
- docs/docker.md has the 'Upgrading the agent container' section + recipe
- compose files reference docs/docker.md + show the upgrade step inline
- docs/docker.md frames the isolation model honestly
Test suite: 42 passed (33 existing Docker tests + 9 new). No behaviour
change for users who set HERMES_WORKSPACE explicitly, and no migration is
required for existing deployments — Docker rebinds the existing volume
read-only on next `up`. Users upgrading the agent image should now follow
the documented `docker volume rm hermes-agent-src` recipe.
Closes#1416 (documented upgrade procedure) and addresses the read-only
half of the multi-container coupling concern raised on #2453.
Opus advisor caught this on stage-369 review — PR #2347 left a 'PR TBD'
placeholder in CHANGELOG that should reference its own number. One-line
attribution fix, no behavior change.
PR #2347 hoisted the inline state object to a `state` variable so the
auto-compression handler could share it with appendLiveCompressionCard.
Behavior is identical — same setCompressionUi() dispatch, same calm
compression-card path — but tests/test_run_journal_frontend_static.py
pinned the literal substring `setCompressionUi({` to verify the call
site. Relax the assertion to accept either inline (`{...}`) or hoisted
(`state`) argument form. Both forms route through the same compression
card path; the over-specific substring was the bug.
api.updates._schedule_restart() spawns a daemon thread that calls
os.execv() after a short sleep. Tests in test_update_banner_fixes.py
monkeypatch os.execv to a no-op, but monkeypatch teardown can win the
race against the daemon thread — when the thread wakes up after
teardown, the real os.execv is back, and it re-execs pytest with the
original argv. From the outside this looked like pytest hanging at 99%
and then restarting the entire suite from 0% in a loop.
The fix shadows os.execv with a permanent no-op wrapper at conftest
module-import time, so late-firing daemon threads can't escape. Tests
that need to verify execv was called still patch it themselves; their
patches sit on top of the wrapper for their lifetime.
Also adds tests/test_pytest_execv_guard.py to pin the guard against
future conftest refactors.
v0.51.74 — Release AX:
- PR #2362 (fixes#2360) — Markdown table cell paragraph spacing
- PR #2363 (refs #2361 + #1925) — Run state consistency RFC docs
- PR #2365 (fixes#1106) — custom_providers.models YAML list format
- PR #2367 — Settings sidebar i18n (with maintainer locale-parity fix)
None of the PRs touched CHANGELOG.md, so adding the canonical entries
manually with proper PR # attribution and issue refs.
Stage-367 maintainer fix on #2367: CI failed on 5 locale-parity tests
because the PR added settings_tab_plugins to English only. Added
translations to all 10 non-English locales (commit ddf8eb5d on the
PR's integration branch).
PR #2367 added settings_tab_plugins to English only. The locale-parity
tests (test_chinese_locale.py, test_japanese_locale.py, etc.) require
every English key to exist in all 10 other locales. CI failed on 5 of them.
Adds the key to all 10 non-English locales with translations:
- it: Plugin, ja: プラグイン, ru: Плагины, es/de/pt/fr: Plugins (loanword),
zh: 插件, zh-TW: 外掛, ko: 플러그인
Co-authored-by: mccxj <mccxj@users.noreply.github.com>
v0.51.73 — Release AW:
- PR #2357 (fixes#2355) — Compression reference card anchoring fix
The contributor's CHANGELOG entry was wiped during the --ours rebase
conflict resolution (because v0.51.72 stamping moved Unreleased entries
elsewhere). Added the canonical PR-numbered entry manually.
Stage-366 carries only #2357. Two other recent PRs were deferred:
- #2347 (live timeline restore, 380 LOC) — real semantic conflicts with
v0.51.72's #2348 (_stripLiveVisibleAssistantEchoFromThinking +
visibleInterimSnippets logic). Needs contributor rebase against current
master.
- #2356 (mobile panel touch targets, 98 LOC) — incompatible architectural
conflicts in static/style.css (.sidebar-nav vs .rail layout choice).
Needs contributor redesign against current sidebar-nav layout.
Replace the earlier frontend-reset approach with a backend side-channel
approach that preserves the queue (event, data) tuple shape.
Problem (Opus catch):
- Live SSE frames emitted by _sse() in api/streaming.py:2296 carried no
'id:' field. Only journal-replay frames (via _sse_with_id) emitted IDs.
- Frontend's _lastRunJournalSeq cursor stayed at 0 during live streaming.
- Mid-stream error → reconnect-to-replay arrived with after_seq=0.
- Server replayed every journaled event from seq 1.
- assistantText (closure-scoped) had accumulated all live tokens already
→ double-rendered output.
Fix:
- api/config.py: STREAM_LAST_EVENT_ID: dict = {} module-level dict.
- api/streaming.py put(): capture journal event_id, write to
STREAM_LAST_EVENT_ID[stream_id]. Keep queue tuple as (event, data).
- api/routes.py _handle_sse_stream: read STREAM_LAST_EVENT_ID[stream_id]
at emit time, use _sse_with_id when set.
- api/streaming.py finally block: pop STREAM_LAST_EVENT_ID for cleanup.
Why side-channel instead of 3-tuple:
- Earlier attempt (queue tuple → (event, data, event_id)) broke 4 existing
tests: test_cancel_interrupt, test_sprint42, test_sprint51,
test_issue1857_usage_overwrite. These all unpack 'event, data = q.get()'.
- Frontend-reset approach (reset assistantText before replay) broke 3
other tests: test_smooth_text_fade, test_streaming_markdown,
test_streaming_race_fix. _wireSSE must NOT reset accumulators because
legacy reconnect doesn't replay events; only journal-replay does.
Side-channel preserves both invariants:
- Queue contract stays (event, data) — legacy consumers unbroken.
- Frontend accumulators stay alive on _wireSSE — legacy reconnect unbroken.
- Live SSE emits 'id:' so the journal cursor advances correctly.
6 regression tests added in test_stage364_opus_live_sse_event_id.py.
1 existing test (test_run_journal_streaming_static.test_streaming_journals_sse_events_before_queue_delivery) updated to be tuple-shape-agnostic.
Test results:
- Full pytest: 5713 passed, 10 skipped, 1 xfailed, 2 xpassed, 0 failed
- Previously-failing 5 tests: ALL PASS
- 6 new regression tests: ALL PASS
Opus advisor caught that the new run-journal replay path could double-render
when the live stream errors mid-stream:
- Live SSE frames emitted by _sse() in api/streaming.py:2296 carry no 'id:'
field. Only _sse_with_id() (used in _replay_run_journal at routes.py:5853)
emits IDs.
- During live streaming, EventSource.lastEventId stays empty, so the frontend's
_lastRunJournalSeq stays at 0.
- If the server dies mid-stream, the error reconnect handler opens replay with
after_seq=0 — server replays every journaled event from seq 1.
- assistantText accumulator (closure scope in messages.js) carries over from
the live phase. The token handler unconditionally appends d.text. Double-
rendered text.
Fix: reset assistantText, reasoningText, liveReasoningText, segmentStart, and
set _smdReconnect=true before opening the replay EventSource. Next live token
clears assistantBody.innerHTML to match the reset accumulator.
4 regression tests added in test_stage364_opus_replay_doublerender_fix.py.
Revert-fix verification confirms 3/4 tests fail against reverted code.
This is the TWO-LAYER catch in action: agent self-verified the producer→
consumer chain works end-to-end (Step 3 in agent-side-empirical-verification.md
PASSED for #2283), and Opus independently caught a separate frontend coupling
issue. Both checks required and both fire.
v0.51.71 — Release AU:
- PR #2349 (fixes#2345) — Stale-stream cleanup non-touching of updated_at
- PR #2343 (refs #2147) — Profiles vs workspaces help card
- PR #2283 (refs #1925) — WebUI run event journal replay (RFC slice 1)
Also relabeled #2283's CHANGELOG entry to add proper PR #2283 attribution
(it had been added without the PR number prefix during the contributor PR),
and #2349's 'PR TBD' placeholder filled in.
Hermes Agent