diff --git a/CHANGELOG.md b/CHANGELOG.md index 5862bb7b..76439d20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,9 +2,13 @@ ## [Unreleased] +### Fixed + +- **PR TBD** — Multi-container Docker startup is no longer broken by the v0.51.84 `:ro` mount on `hermes-agent-src`. `docker_init.bash` was calling `uv pip install "$_agent_src[all]"` against the mounted source tree directly. setuptools' `egg_info` build step touches `hermes_agent.egg-info/` inside the source tree even under PEP 517 build isolation, which `EROFS`-failed on the now-read-only mount and (under `set -e`) killed startup of every multi-container deploy. The init script now stages the agent source into `/tmp/hermes-agent-build` via `rsync` (with a `cp -a` fallback for images without rsync, both excluding any pre-baked `*.egg-info`, `build`, `dist`, and `__pycache__` artifacts) and runs the install against that writable copy, leaving the underlying `:ro` mount untouched. Stage dir is removed after the install completes. This regression was caught by the new Docker runtime smoke gate (below) on its very first CI run against its own PR — 5800+ source-level pytests + the independent reviewer's eyeball had all missed it on PR #2470. + ### Infrastructure -- **PR TBD** — Add a Docker runtime smoke gate (`.github/workflows/docker-smoke.yml`) triggered on PRs and pushes to `master` that modify `Dockerfile`, `docker_init.bash`, `docker-compose*.yml`, `.dockerignore`, or `.env.docker.example`. Validates every compose file parses (`docker compose config`), then matrix-runs the single, two-container, and three-container variants end-to-end: rebuilds the local `Dockerfile` and re-tags it as `ghcr.io/nesquena/hermes-webui:latest` so the multi-container variants exercise PR-level changes rather than the previously-released registry image, `docker compose up -d --wait`s with a 120s health window, probes `/health`, and greps startup logs for known-bad signatures (`EROFS`, `Traceback`, `PermissionError`, `error_exit`, `groupmod: cannot`, `usermod: cannot`, `Failed to set`). Closes the source-only-test gap that let v0.51.84's `:ro`-mount × `chown -h ... {} +` startup regression reach review with 5800+ green pytests. Workflow runs with `permissions: contents: read`, uses per-run project names and a pre-flight orphan reaper for safe concurrency, and unconditionally tears down all volumes/networks in an `EXIT` trap. +- **PR TBD** — Add a Docker runtime smoke gate (`.github/workflows/docker-smoke.yml`) triggered on PRs and pushes to `master` that modify `Dockerfile`, `docker_init.bash`, `docker-compose*.yml`, `.dockerignore`, or `.env.docker.example`. Validates every compose file parses (`docker compose config`), then matrix-runs the single, two-container, and three-container variants end-to-end: rebuilds the local `Dockerfile` and re-tags it as `ghcr.io/nesquena/hermes-webui:latest` so the multi-container variants exercise PR-level changes rather than the previously-released registry image, `docker compose up -d --wait`s with a 120s health window, probes `/health`, and greps startup logs for known-bad signatures (`EROFS`, `Traceback`, `PermissionError`, `error_exit`, `groupmod: cannot`, `usermod: cannot`, `Failed to set`). Closes the source-only-test gap that let v0.51.84's `:ro`-mount × `chown -h ... {} +` startup regression reach review with 5800+ green pytests. Workflow runs with `permissions: contents: read`, uses per-run project names and a pre-flight orphan reaper for safe concurrency, and unconditionally tears down all volumes/networks in an `EXIT` trap. Two new source-level invariants in `tests/test_docker_docs_and_readonly.py` pin the staging path so the underlying `:ro`-incompatible call doesn't regress. ## [v0.51.86] — 2026-05-17 — Release BJ (stage-379 — 4-PR review-bypass batch — memory-provider session lifecycle + cross-provider /model alias + RuntimeAdapter cancel seam + Fork-from-here messaging coord) diff --git a/docker_init.bash b/docker_init.bash index 4cbc4467..8d74fea1 100644 --- a/docker_init.bash +++ b/docker_init.bash @@ -391,7 +391,40 @@ else fi done if [ -n "$_agent_src" ]; then - uv pip install "$_agent_src[all]" --trusted-host pypi.org --trusted-host files.pythonhosted.org || error_exit "Failed to install hermes-agent's requirements" + # The agent source can be mounted read-only (see docker-compose.two-container.yml + # / docker-compose.three-container.yml — the WebUI only reads this volume to + # install the agent's Python dependencies and never writes to it). setuptools' + # `egg_info` build step, however, touches `hermes_agent.egg-info/` inside the + # source tree even under PEP 517 build isolation, which `EROFS`-fails on a + # `:ro` mount and (under `set -e`) kills startup of every multi-container + # deploy. Stage the source into a writable tmpfs copy so the build can write + # its metadata side-by-side without touching the underlying mount. + # + # The copy excludes any pre-baked `*.egg-info` / `build` / `dist` artifacts + # to avoid the timestamp-update path setuptools takes when one is present, + # and `--reflink=auto` makes the copy near-free on overlay2/btrfs where + # supported. We rebuild on every container start (the agent source can + # change across volume re-init); cost is one rsync of ~10MB of Python source. + _stage_src="/tmp/hermes-agent-build" + rm -rf "$_stage_src" + mkdir -p "$_stage_src" + if command -v rsync >/dev/null 2>&1; then + rsync -a \ + --exclude='*.egg-info' --exclude='build' --exclude='dist' \ + --exclude='__pycache__' --exclude='.git' \ + "$_agent_src"/ "$_stage_src"/ \ + || error_exit "Failed to stage hermes-agent source to writable build dir" + else + # Fallback when rsync isn't in the image — straight cp -a, then drop + # the build artifacts that would trip setuptools. + cp -a "$_agent_src"/. "$_stage_src"/ \ + || error_exit "Failed to copy hermes-agent source to writable build dir" + rm -rf "$_stage_src"/*.egg-info "$_stage_src"/build "$_stage_src"/dist 2>/dev/null || true + find "$_stage_src" -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true + fi + uv pip install "$_stage_src[all]" --trusted-host pypi.org --trusted-host files.pythonhosted.org \ + || error_exit "Failed to install hermes-agent's requirements" + rm -rf "$_stage_src" else echo "" echo "!! WARNING: hermes-agent source not found." diff --git a/tests/test_docker_docs_and_readonly.py b/tests/test_docker_docs_and_readonly.py index 87c2dbf6..af2ec3b1 100644 --- a/tests/test_docker_docs_and_readonly.py +++ b/tests/test_docker_docs_and_readonly.py @@ -152,3 +152,72 @@ def test_docker_md_documents_isolation_model(): "isolation expectations — process/network/resource isolation, NOT " "filesystem isolation." ) + + +# ── 5: docker_init.bash stages agent source to a writable build dir ───────── +# +# The :ro mount fixed in PR #2470 broke a second, less obvious surface: +# `uv pip install "$_agent_src[all]"` invokes setuptools' egg_info build step, +# which touches `hermes_agent.egg-info/` *inside the source tree* even under +# PEP 517 build isolation. On a `:ro` mount this returns `EROFS` and (under +# `set -e`) kills container startup. The fix: copy the source tree into a +# writable tmpfs build dir, run the install against THAT, then clean up. +# +# This was caught the first time the Docker smoke gate ran on its own PR — a +# real regression that 5800+ source-level pytests had no way to surface +# because none of them invoked `docker_init.bash` against a real :ro mount. + + +def test_docker_init_stages_agent_source_for_writable_install(): + """docker_init.bash must NOT pass the raw _agent_src path to `uv pip + install` — that hits the :ro mount and fails. It must stage the source + into a writable build dir first (the staged path is used in the install + invocation).""" + src = (REPO / "docker_init.bash").read_text(encoding="utf-8") + + # The fix uses a /tmp staging path that's clearly distinct from the + # mounted source path. Pin the staging marker. + assert "_stage_src=" in src, ( + "docker_init.bash must declare a _stage_src writable build dir " + "before invoking `uv pip install` against the (potentially :ro) " + "hermes-agent source." + ) + + # The install line must reference the staged path, NOT the raw _agent_src + # path. The pre-fix code was: + # uv pip install "$_agent_src[all]" ... + # The fixed code is: + # uv pip install "$_stage_src[all]" ... + install_lines = [ + line for line in src.splitlines() + if "uv pip install" in line and "[all]" in line + ] + assert install_lines, "expected an `uv pip install ...[all]` line in docker_init.bash" + for line in install_lines: + assert '"$_agent_src[all]"' not in line, ( + "docker_init.bash invokes `uv pip install $_agent_src[all]` " + "directly — this fails with EROFS when the hermes-agent volume " + "is mounted :ro (the production multi-container default). " + "Use the writable $_stage_src path instead. " + f"Offending line: {line!r}" + ) + assert "_stage_src" in line, ( + "the `uv pip install ...[all]` line must use the staged writable " + f"path. Offending line: {line!r}" + ) + + +def test_docker_init_excludes_egg_info_during_staging(): + """The staging copy must exclude pre-baked *.egg-info / build / dist + directories. setuptools takes a different (timestamp-update) code path + when one is already present in the source tree, which itself hits the + :ro mount through stat/utime calls. Excluding them keeps the build + happily on the fresh-build code path.""" + src = (REPO / "docker_init.bash").read_text(encoding="utf-8") + # Either rsync --exclude= form or the cp-fallback's explicit rm -rf — + # accept either provided the egg-info exclusion exists. + assert "egg-info" in src, ( + "docker_init.bash staging must exclude *.egg-info from the copy " + "to avoid setuptools' timestamp-update code path." + ) +