fix(docker): stage agent source to writable build dir before pip install

The Docker smoke gate added in this same PR caught a real production
regression on its very first CI run. v0.51.84 (PR #2470) mounted
hermes-agent-src read-only on the WebUI side and widened the chown
prune to keep the read-only walk happy, but missed that the WebUI's
startup also runs:

    uv pip install "$_agent_src[all]"

against the same now-read-only mount. setuptools' egg_info step writes
hermes_agent.egg-info/ inside the source tree even under PEP 517 build
isolation (this is by design -- PEP 517 isolates the BUILD environment,
not the source tree's metadata directory). On a :ro mount this returns
EROFS, the install fails, error_exit fires, and every multi-container
deploy dies at startup. The smoke gate flagged it on both the
two-container and three-container variants.

The fix
-------
Stage the agent source into a writable build dir under /tmp BEFORE
invoking pip install, then point pip at the staged copy.

  _stage_src="/tmp/hermes-agent-build"
  rm -rf "$_stage_src" && mkdir -p "$_stage_src"
  rsync -a --exclude='*.egg-info' --exclude='build' --exclude='dist' \
        --exclude='__pycache__' --exclude='.git' \
        "$_agent_src"/ "$_stage_src"/
  uv pip install "$_stage_src[all]" ...
  rm -rf "$_stage_src"

The exclusion list matters: when setuptools sees a pre-baked *.egg-info,
build, or dist directory, it takes a timestamp-update code path that
also reads/writes inside that directory -- which itself fails on a :ro
source. Excluding them keeps the build on the fresh-build path
unconditionally.

rsync is in the production image (Dockerfile line 41-44). For users
running custom WebUI images without rsync, the script falls back to
cp -a + post-copy rm -rf of the same artifacts.

Tests
-----
Two new source-level invariants in tests/test_docker_docs_and_readonly.py:

  test_docker_init_stages_agent_source_for_writable_install
    -- asserts _stage_src=... is declared
    -- asserts every `uv pip install ...[all]` line uses _stage_src,
       NOT raw $_agent_src

  test_docker_init_excludes_egg_info_during_staging
    -- asserts the staging path excludes *.egg-info (rsync exclude
       form or cp-fallback's explicit rm -rf both pass)

These would have caught the v0.51.84 regression at the source level
(once written; they're new). The Docker runtime smoke gate is the
durable defence for the broader class of :ro x init-script
interactions, since source-level invariants only catch what they're
written to catch.

Verification
------------
- pytest tests/test_docker_docs_and_readonly.py: 11 passed (9 existing
  + 2 new)
- pytest tests/ -q --timeout=60: 5891 passed, 6 skipped (was 5889;
  delta is exactly the 2 new tests)
- bash -n docker_init.bash: clean

Once this lands, the Docker smoke gate's two/three-container variants
should go green, completing the self-validating loop.

CHANGELOG.md

@@ -2,9 +2,13 @@
 
 ## [Unreleased]
 
+### Fixed
+
+- **PR TBD** — Multi-container Docker startup is no longer broken by the v0.51.84 `:ro` mount on `hermes-agent-src`. `docker_init.bash` was calling `uv pip install "$_agent_src[all]"` against the mounted source tree directly. setuptools' `egg_info` build step touches `hermes_agent.egg-info/` inside the source tree even under PEP 517 build isolation, which `EROFS`-failed on the now-read-only mount and (under `set -e`) killed startup of every multi-container deploy. The init script now stages the agent source into `/tmp/hermes-agent-build` via `rsync` (with a `cp -a` fallback for images without rsync, both excluding any pre-baked `*.egg-info`, `build`, `dist`, and `__pycache__` artifacts) and runs the install against that writable copy, leaving the underlying `:ro` mount untouched. Stage dir is removed after the install completes. This regression was caught by the new Docker runtime smoke gate (below) on its very first CI run against its own PR — 5800+ source-level pytests + the independent reviewer's eyeball had all missed it on PR #2470.
+
 ### Infrastructure
 
-- **PR TBD** — Add a Docker runtime smoke gate (`.github/workflows/docker-smoke.yml`) triggered on PRs and pushes to `master` that modify `Dockerfile`, `docker_init.bash`, `docker-compose*.yml`, `.dockerignore`, or `.env.docker.example`. Validates every compose file parses (`docker compose config`), then matrix-runs the single, two-container, and three-container variants end-to-end: rebuilds the local `Dockerfile` and re-tags it as `ghcr.io/nesquena/hermes-webui:latest` so the multi-container variants exercise PR-level changes rather than the previously-released registry image, `docker compose up -d --wait`s with a 120s health window, probes `/health`, and greps startup logs for known-bad signatures (`EROFS`, `Traceback`, `PermissionError`, `error_exit`, `groupmod: cannot`, `usermod: cannot`, `Failed to set`). Closes the source-only-test gap that let v0.51.84's `:ro`-mount × `chown -h ... {} +` startup regression reach review with 5800+ green pytests. Workflow runs with `permissions: contents: read`, uses per-run project names and a pre-flight orphan reaper for safe concurrency, and unconditionally tears down all volumes/networks in an `EXIT` trap.
+- **PR TBD** — Add a Docker runtime smoke gate (`.github/workflows/docker-smoke.yml`) triggered on PRs and pushes to `master` that modify `Dockerfile`, `docker_init.bash`, `docker-compose*.yml`, `.dockerignore`, or `.env.docker.example`. Validates every compose file parses (`docker compose config`), then matrix-runs the single, two-container, and three-container variants end-to-end: rebuilds the local `Dockerfile` and re-tags it as `ghcr.io/nesquena/hermes-webui:latest` so the multi-container variants exercise PR-level changes rather than the previously-released registry image, `docker compose up -d --wait`s with a 120s health window, probes `/health`, and greps startup logs for known-bad signatures (`EROFS`, `Traceback`, `PermissionError`, `error_exit`, `groupmod: cannot`, `usermod: cannot`, `Failed to set`). Closes the source-only-test gap that let v0.51.84's `:ro`-mount × `chown -h ... {} +` startup regression reach review with 5800+ green pytests. Workflow runs with `permissions: contents: read`, uses per-run project names and a pre-flight orphan reaper for safe concurrency, and unconditionally tears down all volumes/networks in an `EXIT` trap. Two new source-level invariants in `tests/test_docker_docs_and_readonly.py` pin the staging path so the underlying `:ro`-incompatible call doesn't regress.
 
 ## [v0.51.86] — 2026-05-17 — Release BJ (stage-379 — 4-PR review-bypass batch — memory-provider session lifecycle + cross-provider /model alias + RuntimeAdapter cancel seam + Fork-from-here messaging coord)
 

docker_init.bash

@@ -391,7 +391,40 @@ else
     fi
   done
   if [ -n "$_agent_src" ]; then
-    uv pip install "$_agent_src[all]" --trusted-host pypi.org --trusted-host files.pythonhosted.org || error_exit "Failed to install hermes-agent's requirements"
+    # The agent source can be mounted read-only (see docker-compose.two-container.yml
+    # / docker-compose.three-container.yml — the WebUI only reads this volume to
+    # install the agent's Python dependencies and never writes to it). setuptools'
+    # `egg_info` build step, however, touches `hermes_agent.egg-info/` inside the
+    # source tree even under PEP 517 build isolation, which `EROFS`-fails on a
+    # `:ro` mount and (under `set -e`) kills startup of every multi-container
+    # deploy. Stage the source into a writable tmpfs copy so the build can write
+    # its metadata side-by-side without touching the underlying mount.
+    #
+    # The copy excludes any pre-baked `*.egg-info` / `build` / `dist` artifacts
+    # to avoid the timestamp-update path setuptools takes when one is present,
+    # and `--reflink=auto` makes the copy near-free on overlay2/btrfs where
+    # supported. We rebuild on every container start (the agent source can
+    # change across volume re-init); cost is one rsync of ~10MB of Python source.
+    _stage_src="/tmp/hermes-agent-build"
+    rm -rf "$_stage_src"
+    mkdir -p "$_stage_src"
+    if command -v rsync >/dev/null 2>&1; then
+      rsync -a \
+        --exclude='*.egg-info' --exclude='build' --exclude='dist' \
+        --exclude='__pycache__' --exclude='.git' \
+        "$_agent_src"/ "$_stage_src"/ \
+        || error_exit "Failed to stage hermes-agent source to writable build dir"
+    else
+      # Fallback when rsync isn't in the image — straight cp -a, then drop
+      # the build artifacts that would trip setuptools.
+      cp -a "$_agent_src"/. "$_stage_src"/ \
+        || error_exit "Failed to copy hermes-agent source to writable build dir"
+      rm -rf "$_stage_src"/*.egg-info "$_stage_src"/build "$_stage_src"/dist 2>/dev/null || true
+      find "$_stage_src" -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
+    fi
+    uv pip install "$_stage_src[all]" --trusted-host pypi.org --trusted-host files.pythonhosted.org \
+      || error_exit "Failed to install hermes-agent's requirements"
+    rm -rf "$_stage_src"
   else
     echo ""
     echo "!! WARNING: hermes-agent source not found."

tests/test_docker_docs_and_readonly.py

@@ -152,3 +152,72 @@ def test_docker_md_documents_isolation_model():
         "isolation expectations — process/network/resource isolation, NOT "
         "filesystem isolation."
     )
+
+
+# ── 5: docker_init.bash stages agent source to a writable build dir ─────────
+#
+# The :ro mount fixed in PR #2470 broke a second, less obvious surface:
+# `uv pip install "$_agent_src[all]"` invokes setuptools' egg_info build step,
+# which touches `hermes_agent.egg-info/` *inside the source tree* even under
+# PEP 517 build isolation. On a `:ro` mount this returns `EROFS` and (under
+# `set -e`) kills container startup. The fix: copy the source tree into a
+# writable tmpfs build dir, run the install against THAT, then clean up.
+#
+# This was caught the first time the Docker smoke gate ran on its own PR — a
+# real regression that 5800+ source-level pytests had no way to surface
+# because none of them invoked `docker_init.bash` against a real :ro mount.
+
+
+def test_docker_init_stages_agent_source_for_writable_install():
+    """docker_init.bash must NOT pass the raw _agent_src path to `uv pip
+    install` — that hits the :ro mount and fails. It must stage the source
+    into a writable build dir first (the staged path is used in the install
+    invocation)."""
+    src = (REPO / "docker_init.bash").read_text(encoding="utf-8")
+
+    # The fix uses a /tmp staging path that's clearly distinct from the
+    # mounted source path. Pin the staging marker.
+    assert "_stage_src=" in src, (
+        "docker_init.bash must declare a _stage_src writable build dir "
+        "before invoking `uv pip install` against the (potentially :ro) "
+        "hermes-agent source."
+    )
+
+    # The install line must reference the staged path, NOT the raw _agent_src
+    # path. The pre-fix code was:
+    #   uv pip install "$_agent_src[all]" ...
+    # The fixed code is:
+    #   uv pip install "$_stage_src[all]" ...
+    install_lines = [
+        line for line in src.splitlines()
+        if "uv pip install" in line and "[all]" in line
+    ]
+    assert install_lines, "expected an `uv pip install ...[all]` line in docker_init.bash"
+    for line in install_lines:
+        assert '"$_agent_src[all]"' not in line, (
+            "docker_init.bash invokes `uv pip install $_agent_src[all]` "
+            "directly — this fails with EROFS when the hermes-agent volume "
+            "is mounted :ro (the production multi-container default). "
+            "Use the writable $_stage_src path instead. "
+            f"Offending line: {line!r}"
+        )
+        assert "_stage_src" in line, (
+            "the `uv pip install ...[all]` line must use the staged writable "
+            f"path. Offending line: {line!r}"
+        )
+
+
+def test_docker_init_excludes_egg_info_during_staging():
+    """The staging copy must exclude pre-baked *.egg-info / build / dist
+    directories. setuptools takes a different (timestamp-update) code path
+    when one is already present in the source tree, which itself hits the
+    :ro mount through stat/utime calls. Excluding them keeps the build
+    happily on the fresh-build code path."""
+    src = (REPO / "docker_init.bash").read_text(encoding="utf-8")
+    # Either rsync --exclude= form or the cp-fallback's explicit rm -rf —
+    # accept either provided the egg-info exclusion exists.
+    assert "egg-info" in src, (
+        "docker_init.bash staging must exclude *.egg-info from the copy "
+        "to avoid setuptools' timestamp-update code path."
+    )
+