Release OV (v0.51.435): supported local pytest runner (#3908) (#4252)

* stage-3908: supported local pytest runner + symlink-guard on venv create/clear (Codex CORE fix)

Rebased rodboev/franksong2702 #3908 onto v0.51.434. Adds scripts/test.sh dev
runner + requirements-dev.txt + test_ci_hygiene.py + conftest version guard.
Defensive fix on the authoritative gate: create_or_rebuild_venv refuses a
symlinked .venv so 'python -m venv --clear' can't wipe the symlink target
(same symlink-hardening class as #4217/#4234/#4240). Locked by a hygiene test.

* Release OV: supported local pytest runner (#3908)

scripts/test.sh + requirements-dev.txt + conftest version guard + hygiene tests.
Self-rebased franksong2702's #3908 onto v0.51.434; full Codex+Opus+suite gate.
Codex CORE fix applied: refuse to create/--clear a virtualenv through a symlinked
.venv (symlink-hardening class, #4217/#4234/#4240), locked by a hygiene assertion.

Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>

---------

Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>

AGENTS.md

@@ -47,6 +47,14 @@ Follow that checklist's safety rules:
 - Keep one logical change per PR; split unrelated refactors or cleanup.
 - Read `docs/CONTRACTS.md` and the linked contract/RFC for the touched
   subsystem before editing.
+- For local pytest runs, use `./scripts/test.sh` instead of bare `python3`,
+  `python -m pytest`, or `pytest`. The script creates/uses the repo `.venv`,
+  pins execution to Python 3.11-3.13, and installs missing dev test dependencies.
+  `HERMES_WEBUI_TEST_PYTHON` selects the supported base interpreter used to
+  create or rebuild `.venv`; it must not install test dependencies into a
+  system/Homebrew interpreter directly.
+  If a direct pytest invocation reports an unsupported interpreter, rerun through
+  `./scripts/test.sh` before debugging product code.
 - Prefer the existing Python + vanilla JavaScript structure. Do not add
   dependencies, build tools, frameworks, or long-lived processes without clear
   justification and a rollback story.

CHANGELOG.md

@@ -3,7 +3,11 @@
 
 ## [Unreleased]
 
-## [v0.51.434] — 2026-06-15 — Release OU (reject symlinked skill files on save)
+## [v0.51.435] — 2026-06-15 — Release OV (supported local pytest runner)
+
+### Added
+
+- **`./scripts/test.sh` runs the suite in a repo-local, version-correct virtualenv.** The runner finds a supported interpreter (Python 3.11–3.13), creates or rebuilds a repo-local `.venv` when needed, installs the pinned dev dependencies from the new `requirements-dev.txt`, and then runs pytest — so contributors get a one-command, correctly-versioned test path instead of a bare `pytest` that may collect against an unsupported system Python. `tests/conftest.py` also fails fast with a clear message if pytest is launched on an unsupported interpreter, and the runner refuses to create or `--clear` a virtualenv through a symlinked `.venv` (which would otherwise empty the symlink's target). Dev-tooling only — no runtime or app behavior changes. (#3908)
 
 ### Fixed
 

CONTRIBUTING.md

@@ -71,10 +71,12 @@ Keep each PR focused. A small related group of fixes is fine. A bug fix plus a C
 
 ### 2. Local Verification
 
-Run the test suite locally:
+Run the test suite locally through the repo runner. It creates/uses a supported
+Python 3.11-3.13 `.venv` and installs missing dev test dependencies, avoiding
+unsupported system interpreters during collection:
 
 ```bash
-pytest tests/ -v --timeout=60
+./scripts/test.sh
 ```
 
 CI also runs this suite on Python `3.11`, `3.12`, and `3.13`.

README.md

@@ -473,18 +473,31 @@ For the deep dive on each of these, see [`docs/docker.md`](docs/docker.md).
 ## Running tests
 
 Tests discover the repo and the Hermes agent dynamically -- no hardcoded paths.
+Use the repo test runner so local runs do not accidentally use an unsupported
+system Python. It creates/uses `.venv` with Python 3.11, 3.12, or 3.13 and
+installs the dev test dependencies from `requirements-dev.txt` when missing.
 
 ```bash
 cd hermes-webui
-pytest tests/ -v --timeout=60
+./scripts/test.sh
 ```
 
-Or using the agent venv explicitly:
+Pass normal pytest arguments after the script for focused runs:
 
 ```bash
-/path/to/hermes-agent/venv/bin/python -m pytest tests/ -v
+./scripts/test.sh tests/test_regressions.py -v
 ```
 
+Or seed the repo `.venv` from an explicit supported base interpreter:
+
+```bash
+HERMES_WEBUI_TEST_PYTHON=/path/to/python3.12 ./scripts/test.sh tests/ -v
+```
+
+The override selects the Python used to create or rebuild `.venv`; dependencies
+are still installed into the repo-local virtual environment, not into the
+system/Homebrew interpreter.
+
 Tests run against an isolated server with a separate state directory.
 Production data and real cron jobs are never touched. Current snapshot:
 **~7,150 tests collected** across **~700 test files**, run in CI on Python 3.11,

TESTING.md

@@ -8,8 +8,8 @@
 > Prerequisites: SSH tunnel is active on port 8787. Open http://localhost:8787 in browser.
 > Server health check: curl http://127.0.0.1:8787/health should return {"status":"ok"}.
 >
-> Automated coverage: ~7,150 tests collected via `pytest tests/ --collect-only -q`. Tests run on every PR via GitHub Actions on Python 3.11, 3.12, and 3.13 (3 parallel shards each), alongside a ruff lint gate, a headless browser smoke test, and a Docker smoke test. The suite covers the bootstrap/static wizard, real provider config persistence (`config.yaml` + `.env`), the `/api/onboarding/*` backend, the onboarding skip/existing-config guard, CSS regression coverage for thinking/tool card animation, streaming session persistence, mobile layout breakpoints, locale parity across 11 languages, and hundreds of issue/PR-pinned regression tests.
+> Automated coverage: ~7,150 tests collected via `./scripts/test.sh tests/ --collect-only -q`. Tests run on every PR via GitHub Actions on Python 3.11, 3.12, and 3.13 (3 parallel shards each), alongside a ruff lint gate, a headless browser smoke test, and a Docker smoke test. The suite covers the bootstrap/static wizard, real provider config persistence (`config.yaml` + `.env`), the `/api/onboarding/*` backend, the onboarding skip/existing-config guard, CSS regression coverage for thinking/tool card animation, streaming session persistence, mobile layout breakpoints, locale parity across 11 languages, and hundreds of issue/PR-pinned regression tests.
-> Run: `pytest tests/ -v --timeout=60`
+> Run: `./scripts/test.sh`
 >
 > Local regression focus: verify that a previously closed workspace panel stays visually closed from first paint through boot completion on desktop refresh; there should be no brief open-then-close flash.
 
@@ -667,7 +667,7 @@ FAIL: Browser freezes, crash, or security issue.
 
 ## Automated Test Coverage Reference
 
-These behaviors are verified by pytest (run: venv/bin/python -m pytest tests/ -v):
+These behaviors are verified by pytest (run: ./scripts/test.sh tests/ -v):
 
 Sprint 1 tests (test_sprint1.py):
   - Server health, session CRUD (create/load/update/delete/sort)
@@ -719,7 +719,7 @@ If you are a Claude agent with browser access, follow these instructions:
 
 *Last updated: Sprint 2, March 30, 2026*
 *Server version: v0.4 (server.py in webui-mvp/)*
-*Run automated tests: python -m pytest tests/ -v*
+*Run automated tests: ./scripts/test.sh tests/ -v*
 
 ---
 
@@ -900,7 +900,7 @@ Manual-only (not covered by automation):
 
 *Last updated: Sprint 3, March 30, 2026*
 *Total automated tests: 48/48*
-*Run: python -m pytest tests/ -v*
+*Run: ./scripts/test.sh tests/ -v*
 
 ---
 
@@ -1230,7 +1230,7 @@ Manual-only for Sprint 5:
 
 *Last updated: Sprint 5, March 30, 2026*
 *Total automated tests: 86/86*
-*Run: python -m pytest tests/ -v*
+*Run: ./scripts/test.sh tests/ -v*
 *Source: <repo>/ | Static: static/style.css + static/app.js*
 
 ---
@@ -1707,7 +1707,7 @@ Manual-only for Sprint 8:
 *Last updated: Sprint 10 complete, March 31, 2026*
 *Total automated tests: 177/177*
 *Regression gate: tests/test_regressions.py (10 tests, one per introduced bug)*
-*Run: python -m pytest tests/ -v*
+*Run: ./scripts/test.sh tests/ -v*
 *Source: <repo>/*
 *Modules: ui.js, workspace.js, sessions.js, messages.js, panels.js, boot.js (app.js deleted)*
 
@@ -1926,7 +1926,7 @@ Bridged CLI sessions:
 ---
 
 *Last updated: v0.51.192, May 31, 2026*
-*Total automated tests collected: ~7,150 (run `pytest tests/ --collect-only -q` for the exact current count)*
+*Total automated tests collected: ~7,150 (run `./scripts/test.sh tests/ --collect-only -q` for the exact current count)*
 *Regression gate: tests/test_regressions.py*
-*Run: pytest tests/ -v --timeout=60*
+*Run: ./scripts/test.sh*
 *Source: <repo>/*

requirements-dev.txt

@@ -0,0 +1,11 @@
+# Hermes WebUI local test/lint dependencies.
+#
+# Runtime dependencies stay in requirements.txt. This file mirrors the Python
+# tools CI installs for contributor validation.
+-r requirements.txt
+pytest
+pytest-timeout
+pytest-asyncio
+pytest-shard
+ruff
+mcp

scripts/test.sh

@@ -0,0 +1,218 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+VENV_DIR="$ROOT/.venv"
+VENV_PY="$VENV_DIR/bin/python"
+REQ_FILE="$ROOT/requirements-dev.txt"
+
+is_supported_python() {
+  "$1" - <<'PY' >/dev/null 2>&1
+import sys
+raise SystemExit(0 if (3, 11) <= sys.version_info[:2] <= (3, 13) else 1)
+PY
+}
+
+python_version() {
+  "$1" - <<'PY'
+import sys
+print(".".join(map(str, sys.version_info[:3])))
+PY
+}
+
+python_major_minor() {
+  "$1" - <<'PY'
+import sys
+print(".".join(map(str, sys.version_info[:2])))
+PY
+}
+
+resolve_executable() {
+  local candidate="$1"
+  if [[ "$candidate" == */* ]]; then
+    [[ -x "$candidate" ]] && printf '%s\n' "$candidate" || true
+  else
+    command -v "$candidate" 2>/dev/null || true
+  fi
+}
+
+find_supported_base_python() {
+  local candidate path
+  for candidate in python3.13 python3.12 python3.11 python3; do
+    path="$(resolve_executable "$candidate")"
+    if [[ -n "$path" ]] && is_supported_python "$path"; then
+      printf '%s\n' "$path"
+      return 0
+    fi
+  done
+  return 1
+}
+
+has_pip() {
+  "$1" -m pip --version >/dev/null 2>&1
+}
+
+ensure_pip() {
+  local py="$1"
+  if ! has_pip "$py"; then
+    "$py" -m ensurepip --upgrade
+  fi
+}
+
+missing_dev_deps() {
+  "$1" - <<'PY'
+import importlib.util
+
+modules = [
+    "cryptography",
+    "mcp",
+    "pytest",
+    "pytest_asyncio",
+    "pytest_shard",
+    "pytest_timeout",
+    "ruff",
+    "yaml",
+]
+missing = [name for name in modules if importlib.util.find_spec(name) is None]
+if missing:
+    print(", ".join(missing))
+    raise SystemExit(1)
+PY
+}
+
+venv_guidance() {
+  local base_py="${1:-}"
+  if [[ -n "$base_py" ]]; then
+    echo "Could not create a working .venv with $base_py ($(python_version "$base_py"))." >&2
+  else
+    echo "Could not create a working .venv for Hermes WebUI tests." >&2
+  fi
+  echo "Install the matching Python venv/ensurepip package (for example python3.x-venv on Debian/Ubuntu)" >&2
+  echo "or set HERMES_WEBUI_TEST_PYTHON to a supported Python 3.11, 3.12, or 3.13 interpreter." >&2
+}
+
+create_or_rebuild_venv() {
+  local base_py="$1"
+  local mode="${2:-create}"
+  local action="Creating"
+  local venv_args=("$VENV_DIR")
+  # Never create/clear a virtualenv *through* a symlink: `python -m venv --clear`
+  # (and a dangling-symlink create) would empty/write the symlink's target, not a
+  # repo-local .venv. A symlinked .venv that already works is used directly above,
+  # before this function is reached, so refusing here only blocks the destructive path.
+  if [[ -L "$VENV_DIR" ]]; then
+    echo "$VENV_DIR is a symlink; refusing to create or clear a virtualenv through it." >&2
+    echo "Remove the .venv symlink (or set HERMES_WEBUI_TEST_PYTHON to a supported Python 3.11-3.13) and rerun." >&2
+    return 2
+  fi
+  if [[ "$mode" == "rebuild" ]]; then
+    action="Rebuilding"
+    venv_args=(--clear "$VENV_DIR")
+  fi
+  echo "$action .venv with $base_py ($(python_version "$base_py"))." >&2
+  if ! "$base_py" -m venv "${venv_args[@]}"; then
+    rm -rf "$VENV_DIR"
+    venv_guidance "$base_py"
+    return 2
+  fi
+  if [[ ! -x "$VENV_PY" ]]; then
+    rm -rf "$VENV_DIR"
+    echo "$VENV_DIR was created but does not contain bin/python." >&2
+    venv_guidance "$base_py"
+    return 2
+  fi
+  if ! has_pip "$VENV_PY"; then
+    rm -rf "$VENV_DIR"
+    echo "$VENV_DIR was created but its Python cannot run pip." >&2
+    venv_guidance "$base_py"
+    return 2
+  fi
+}
+
+select_python() {
+  local requested="${HERMES_WEBUI_TEST_PYTHON:-}"
+  local requested_path base_py
+  local desired_major_minor current_major_minor
+
+  if [[ -n "$requested" ]]; then
+    requested_path="$(resolve_executable "$requested")"
+    if [[ -z "$requested_path" || ! -x "$requested_path" ]]; then
+      echo "HERMES_WEBUI_TEST_PYTHON does not point to an executable: $requested" >&2
+      return 2
+    fi
+    if ! is_supported_python "$requested_path"; then
+      echo "Unsupported Python for Hermes WebUI tests: $requested_path ($(python_version "$requested_path"))" >&2
+      echo "Use Python 3.11, 3.12, or 3.13." >&2
+      return 2
+    fi
+    base_py="$requested_path"
+    desired_major_minor="$(python_major_minor "$base_py")"
+  else
+    base_py="$(find_supported_base_python || true)"
+    if [[ -z "$base_py" ]]; then
+      echo "No supported Python found for Hermes WebUI tests." >&2
+      echo "Install Python 3.11, 3.12, or 3.13, then rerun ./scripts/test.sh." >&2
+      return 2
+    fi
+    desired_major_minor=""
+  fi
+
+  if [[ -x "$VENV_PY" ]]; then
+    if is_supported_python "$VENV_PY"; then
+      current_major_minor="$(python_major_minor "$VENV_PY")"
+      if [[ -z "$desired_major_minor" || "$current_major_minor" == "$desired_major_minor" ]]; then
+        if has_pip "$VENV_PY"; then
+          printf '%s\n' "$VENV_PY"
+          return 0
+        fi
+        echo "Existing .venv uses supported Python $(python_version "$VENV_PY") but cannot run pip; rebuilding." >&2
+        create_or_rebuild_venv "$base_py" rebuild || return $?
+        printf '%s\n' "$VENV_PY"
+        return 0
+      fi
+      echo "Rebuilding .venv from Python $current_major_minor to requested $desired_major_minor." >&2
+      create_or_rebuild_venv "$base_py" rebuild || return $?
+      printf '%s\n' "$VENV_PY"
+      return 0
+    fi
+    echo "Rebuilding unsupported .venv ($(python_version "$VENV_PY")) with $base_py ($(python_version "$base_py"))." >&2
+    create_or_rebuild_venv "$base_py" rebuild || return $?
+    printf '%s\n' "$VENV_PY"
+    return 0
+  fi
+
+  if [[ -e "$VENV_DIR" && ! -x "$VENV_PY" ]]; then
+    echo "$VENV_DIR exists but does not contain bin/python; rebuilding." >&2
+    create_or_rebuild_venv "$base_py" rebuild || return $?
+    printf '%s\n' "$VENV_PY"
+    return 0
+  fi
+
+  create_or_rebuild_venv "$base_py" create || return $?
+  printf '%s\n' "$VENV_PY"
+}
+
+PYTHON_BIN="$(select_python)" || exit $?
+
+if [[ ! -f "$REQ_FILE" ]]; then
+  echo "Missing $REQ_FILE" >&2
+  exit 2
+fi
+
+if missing="$(missing_dev_deps "$PYTHON_BIN" 2>/dev/null)"; then
+  :
+else
+  echo "Installing missing Hermes WebUI test dependencies in $PYTHON_BIN ($(python_version "$PYTHON_BIN"))." >&2
+  if [[ -n "${missing:-}" ]]; then
+    echo "Missing modules: $missing" >&2
+  fi
+  ensure_pip "$PYTHON_BIN"
+  "$PYTHON_BIN" -m pip install --upgrade pip
+  "$PYTHON_BIN" -m pip install -r "$REQ_FILE"
+fi
+
+if [[ $# -eq 0 ]]; then
+  set -- tests/ -v --timeout=60
+fi
+
+exec "$PYTHON_BIN" -m pytest "$@"

tests/conftest.py

@@ -24,6 +24,14 @@ import urllib.request
 import urllib.error
 import pytest
 
+if not (3, 11) <= sys.version_info[:2] <= (3, 13):
+    pytest.exit(
+        "Hermes WebUI tests require Python 3.11, 3.12, or 3.13. "
+        "Run ./scripts/test.sh so the repo-local supported .venv is used "
+        "instead of an unsupported system python.",
+        returncode=3,
+    )
+
 # ── Repo root discovery ────────────────────────────────────────────────────
 # conftest.py lives at <repo>/tests/conftest.py
 TESTS_DIR  = pathlib.Path(__file__).parent.resolve()

tests/test_ci_hygiene.py

@@ -22,6 +22,40 @@ def test_pytest_integration_marker_is_registered():
     assert "integration:" in text
 
 
+def test_local_test_runner_uses_supported_venv_before_pytest_collection():
+    runner = (ROOT / "scripts" / "test.sh").read_text(encoding="utf-8")
+    conftest = (ROOT / "tests" / "conftest.py").read_text(encoding="utf-8")
+
+    assert "python3.13 python3.12 python3.11 python3" in runner
+    assert "requirements-dev.txt" in runner
+    assert 'HERMES_WEBUI_TEST_PYTHON' in runner
+    assert '[[ -x "$candidate" ]] && printf' in runner
+    assert "exec \"$PYTHON_BIN\" -m pytest" in runner
+    # Destructive-fs guard: never create/clear a virtualenv through a symlinked .venv
+    # (`python -m venv --clear` would empty the symlink's target).
+    assert '-L "$VENV_DIR"' in runner
+    assert "Hermes WebUI tests require Python 3.11, 3.12, or 3.13" in conftest
+    assert "Run ./scripts/test.sh" in conftest
+
+
+def test_local_test_runner_bootstrap_handles_broken_venvs_safely():
+    runner = (ROOT / "scripts" / "test.sh").read_text(encoding="utf-8")
+    select_body = runner.split("select_python() {", 1)[1].split("\n}\n\nPYTHON_BIN", 1)[0]
+    create_body = runner.split("create_or_rebuild_venv() {", 1)[1].split("\n}\n\nselect_python", 1)[0]
+
+    assert "has_pip()" in runner
+    assert 'if ! "$base_py" -m venv "${venv_args[@]}"; then' in create_body
+    assert 'rm -rf "$VENV_DIR"' in create_body
+    assert '[[ ! -x "$VENV_PY" ]]' in create_body
+    assert 'if ! has_pip "$VENV_PY"; then' in create_body
+    assert 'venv_guidance "$base_py"' in create_body
+    assert 'printf \'%s\\n\' "$requested_path"' not in select_body
+    assert 'base_py="$requested_path"' in select_body
+    assert 'desired_major_minor="$(python_major_minor "$base_py")"' in select_body
+    assert 'if has_pip "$VENV_PY"; then' in select_body
+    assert 'create_or_rebuild_venv "$base_py" rebuild' in select_body
+
+
 def test_live_model_success_log_is_debug_not_default_console_log():
     ui = (ROOT / "static" / "ui.js").read_text(encoding="utf-8")
 

tests/test_css_tooltips.py

@@ -6,7 +6,7 @@ across index.html, style.css, and i18n.js — replacing native title="" attribut
 with a faster, CSS-driven tooltip system.
 
 Run:
-    /root/hermes-agent/venv/bin/python -m pytest tests/test_css_tooltips.py -v
+    ./scripts/test.sh tests/test_css_tooltips.py -v
 """
 
 import os

tests/test_mobile_layout.py

@@ -13,7 +13,7 @@ They are static checks (no server needed) that catch common regressions:
   - No full-viewport overflow that would break scroll
 
 Run as part of the standard test suite:
-    pytest tests/test_mobile_layout.py -v
+    ./scripts/test.sh tests/test_mobile_layout.py -v
 """
 
 import pathlib

tests/test_security_redaction.py

@@ -365,7 +365,7 @@ def test_redact_session_data_non_sensitive_unchanged():
 
 
 # ── API-level tests (require running test server started by conftest.py) ─────
-# Run via `start.sh && pytest tests/test_security_redaction.py -v`
+# Run via `./scripts/test.sh tests/test_security_redaction.py -v`
 
 def _create_session_with_credentials() -> str:
     """Write a session file with credential-containing messages directly to disk.

tests/test_sidebar_collapse_toggle.py

@@ -8,7 +8,7 @@ the CSS rules (collapse states, transition, flash-prevention), and the JS
 (toggleSidebar / expandSidebar / _isSidebarCollapsed / Cmd+B handler).
 
 Run:
-    pytest tests/test_sidebar_collapse_toggle.py -v
+    ./scripts/test.sh tests/test_sidebar_collapse_toggle.py -v
 """
 
 import pathlib

tests/test_sprint9.py

@@ -1,6 +1,6 @@
 """
 Sprint 9 Tests: app.js module split verification, tool cards, todo panel.
-Run: python -m pytest tests/test_sprint9.py -v
+Run: ./scripts/test.sh tests/test_sprint9.py -v
 """
 import json, pathlib, urllib.error, urllib.request
 

tests/test_workspace_context_menu_and_rename.py

@@ -19,7 +19,7 @@ a user dogfooded the workspace panel:
     The `defaultValue` parameter was silently dropped; only the placeholder
     showed (the "ghost" name the user described).
 
-Run: /root/hermes-agent/venv/bin/python -m pytest tests/test_workspace_context_menu_and_rename.py -v
+Run: ./scripts/test.sh tests/test_workspace_context_menu_and_rename.py -v
 """
 
 from __future__ import annotations