test-infra: hard-isolate test state from production + resilient teardown (#4289)

* test-infra: hard-isolate test state from production + robust teardown

Two production-safety fixes to tests/conftest.py + tests/_pytest_port.py:

1. TEST_STATE_DIR now defaults under the OS temp dir (/tmp/hermes-webui-tests/)
   instead of HERMES_HOME/webui-test-<hash> — which had been writing test state
   INTO the real ~/.hermes/profiles/webui/ (144 leaked webui-test-* dirs found).
   Added a hard-stop guard: refuse to run if the resolved test state dir is
   inside the production Hermes home, so a misconfigured env can never let tests
   wipe/clobber production profiles/sessions/credentials.

2. _rmtree_retry is now resilient on Linux (was a bare shutil.rmtree): retries
   transient OSError [Errno 39] Directory-not-empty races (a background daemon
   writing into the tree during teardown) + a final ignore_errors sweep, and
   warns instead of raising so a benign cleanup race can't spuriously ERROR a
   test (the failure mode seen near #4283 context-reconciliation tests).

* test-infra: anchor production-proximity guard on literal ~/.hermes (not $HERMES_HOME)

The first guard compared TEST_STATE_DIR against $HERMES_HOME, which is overridden
to TEST_STATE_DIR itself during a run -> false-tripped collection (6 modules) with
'test dir X is inside production home X'. Now anchor on the literal user-home
~/.hermes and explicitly allow any dir under the OS temp root, so the temp default
never trips while a real ~/.hermes target still hard-stops.

* test-infra: close TMPDIR-under-~/.hermes guard hole (Codex)

Codex flagged: the 'temp is always allowed' exception could be exploited if
TMPDIR itself is set under ~/.hermes — then a 'temp' path is a production path and
_under_temp would wrongly suppress the guard. Now the temp exception only holds
when the temp root is itself OUTSIDE the production home. Verified: normal /tmp
default allowed; TMPDIR=~/.hermes/tmp now trips the guard.

* test-infra: mirror production-proximity guard into _pytest_port.py (Opus defense-in-depth)

* test-infra: remove ~/.hermes production-path fallbacks from 4 test files (Codex)

Codex's kickback found the real gap: test_sprint5, test_sprint45,
test_onboarding_existing_config, test_regressions each hardcoded a
~/.hermes/webui-mvp-test fallback for their state dir (used when running
standalone or if the env var is unset). Repointed all to the shared isolated
TEST_STATE_DIR from _pytest_port (temp-rooted). Also refreshed stale
~/.hermes/webui-mvp-test references in test_gateway_sync comment + ARCHITECTURE.md.
Now ZERO production-path fallbacks remain in tests.

---------

Co-authored-by: nesquena-hermes <agent@nesquena-hermes>

ARCHITECTURE.md

@@ -145,7 +145,7 @@ Environment variables controlling behavior:
 Test isolation environment variables (set by conftest.py):
 
     HERMES_WEBUI_TEST_PORT=...                         Optional pinned test port
-    HERMES_WEBUI_TEST_STATE_DIR=~/.hermes/webui-test-* Optional pinned test state
+    HERMES_WEBUI_TEST_STATE_DIR=/tmp/hermes-webui-tests/* Optional pinned test state (default: OS temp dir; must be outside ~/.hermes)
     HERMES_WEBUI_DEFAULT_WORKSPACE=.../test-workspace  Isolated test workspace
 
 Tests NEVER talk to the production server (port 8787).