config.yaml and profile config.yaml were persisted with a plain
Path.write_text(), which truncates the target before writing. A crash
(or exception) after the truncate but before the full payload was
flushed left the live config truncated/corrupt, so the next agent/WebUI
start failed to parse it (availability regression).
Extract a shared api.paths._atomic_write_text() helper (tempfile in the
same dir -> write -> flush + os.fsync -> os.replace; unlink tmp on
error), mirroring the existing .env / cost-snapshot atomic pattern in
api.providers, and apply it to _save_yaml_config_file (api.config) and
the two profile model-config writers (api.profiles). On any mid-write
failure os.replace never runs, so the original file stays byte-for-byte
intact.
Preserve the target's permissions: tempfile.mkstemp() hard-codes 0600
and os.replace carries the temp file's mode onto the target, so without
an explicit chmod every save would silently tighten a group/other-
readable config.yaml (the homelab install ships 0644, profiles 0664)
down to owner-only. Copy the existing file's mode before the replace,
falling back to the umask-adjusted 0666 for a new file. config.yaml
holds no secrets, so that tightening would be a regression, not
hardening (unlike .env, which stays 0600 in api.providers).
settings.json is intentionally left untouched here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
profile_env_for_background_worker mirrored the profile home into process-global
os.environ["HERMES_HOME"] and yielded the worker body OUTSIDE the setup lock, so a
concurrent cross-profile worker (e.g. background title generation on another
profile) could clobber os.environ mid-body — the agent config reader then resolved
the WRONG profile's config (intermittent turn-init failures citing an unused
provider; b3nw's v0.51.849 repro). Now installs hermes-agent v0.18.0's
context-local home override (set_hermes_home_override, a ContextVar that
get_hermes_home() consults before os.environ) for the resolved non-default profile
home — set before yield, reset in finally — so a config read in the worker body
resolves THIS profile's home from task-local state, immune to a concurrent
os.environ clobber. No worker serialization, no os.environ mutation, no detection
heuristic (both earlier WebUI-only mitigations were rejected as regression-unsafe).
Graceful degradation: the override module is resolved lazily+optionally
(_resolve_hermes_home_override, mirroring _resolve_secret_scope_module). On agents
< v0.18.0 the symbol is absent -> returns None -> pre-existing os.environ-mirror
behavior unchanged (no regression, no new failure mode).
Self-built (nesquena-hermes); nesquena independent review APPROVED.
The two-tier mtime cache from #4783 fixed the per-request SKILL.md rescan
but left two concurrency holes that only bite at container cold start,
when the frontend fires several profile-data requests at once and the
caches are empty:
1. `_get_profile_skills_stats()` had no lock, so concurrent misses on the
same profile each ran `os.walk(followlinks=True)` + parsed every
SKILL.md simultaneously.
2. `_build_profile_rows_fast()` ran outside `_LIST_PROFILES_CACHE_LOCK`
in `list_profiles_api()`, so every concurrent request rebuilt all rows
(each walking every profile's skill tree) at once.
With ThreadingHTTPServer (one OS thread per request) and Docker overlay2,
this stacked thousands of concurrent stat() calls and stalled workers
57-70s (per the report's thread dumps).
Fix:
- Add a per-profile compute lock (registry guarded by a meta-lock) and
use double-checked locking in `_get_profile_skills_stats()`: concurrent
misses on one profile collapse to a single compute, while independent
profiles still compute in parallel.
- Single-flight the row build in `list_profiles_api()` by holding
`_LIST_PROFILES_CACHE_LOCK` across the build + cache write. Lock order
is strictly list-lock -> per-profile skills-lock, so no deadlock.
The report's third suggestion (debounce the mtime probe) is deliberately
NOT taken: the every-call cheap probe is the #4783 out-of-band
change-detection contract (test_issue4783 asserts it MUST run on every
call). Serializing the misses removes the herd without weakening that
contract, since only the expensive compute is guarded, not the probe.
Adds tests/test_issue5364_skills_stats_thundering_herd.py proving the
herd collapses (single compute / single build under a concurrent burst),
independent profiles still parallelize, and the every-call probe contract
is preserved. All existing #4783 contract tests still pass.
Co-authored-by: claw-io <claw-io@users.noreply.github.com>
Codex round-4 found my followlinks=True change introduced a perf regression: the
probe os.walk lacked the .git/.venv/node_modules/site-packages + skill-support-dir
pruning that iter_skill_index_files applies, so a skill vendoring a dependency
tree made this EVERY-CALL probe walk thousands of irrelevant files — defeating the
cache's whole purpose. The probe now prunes dirnames[:] with the same
EXCLUDED_SKILL_DIRS + SKILL_SUPPORT_DIRS sets as the compute path. Regression test
(change deep inside node_modules/ must not move the probe) added + non-vacuous.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Codex round-3 reproduced: _skill_tree_max_mtime_ns used os.walk(followlinks=False)
while the compute path iter_skill_index_files uses followlinks=True, so an edit to
a symlinked skill's target SKILL.md was invisible to the probe (stale up to TTL).
Added followlinks=True to the probe walk so it descends symlinked skill dirs and
sees target mtime changes. Regression test added + proven non-vacuous.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Codex re-gate: my prior 'TTL-expired + unchanged-mtime -> refresh TTL, no recompute'
branch could leave counts stale INDEFINITELY for an mtime-PRESERVING out-of-band
change (e.g. a git checkout restoring the old mtime) — worse than master's 8s
recompute. Codex proved it (cached (2,2) stayed (2,2) while a fresh compute returned
(1,2)). Now: within TTL + unchanged mtime serves cached (perf win kept); mtime
changed OR TTL expired both force a full recompute (the TTL is a genuine safety
net). Updated the row-2 test to assert the safety recompute.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Codex gate found a SILENT regression: the cache returned on a zero-I/O fast path
while inside the 300s TTL, so _skill_tree_max_mtime_ns never ran during the TTL
window and an out-of-band (CLI/git) skill create/edit/delete was invisible for up
to 5 minutes — WORSE than master's ~8s stale window. Restructured so the cheap
stat-only mtime probe runs on EVERY call (prompt change detection); the TTL now
only bounds the expensive SKILL.md recompute. Unchanged mtime still serves cached
+ refreshes TTL with no recompute (the perf win is preserved). Updated 2 tests
that encoded the flawed zero-I/O design + added a within-TTL-change regression
(proven non-vacuous).
Co-authored-by: Rod Boev <rod.boev@gmail.com>
The registry-derived scrub union is best-effort and returns empty when
hermes_cli.auth can't be imported (WebUI-only CI/test env). That made
ANTHROPIC_TOKEN / CLAUDE_CODE_OAUTH_TOKEN scrubbing depend on the agent being
installed, so the quota-subprocess regression failed on CI (agent absent) while
passing locally (agent present). Add both token names to the fail-closed floor
set so the scrub covers them regardless of the registry import — strictly more
correct (defense-in-depth) and deterministic across environments.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Final credential-surface item: agent/azure_identity_adapter.py treats
IDENTITY_ENDPOINT / MSI_ENDPOINT as ManagedIdentityCredential sources, so an
empty named profile could inherit the host's managed identity on the quota +
detached-worker paths. Add IDENTITY_ENDPOINT / IDENTITY_HEADER / MSI_ENDPOINT /
MSI_SECRET to the non-registry scrub set (config var AZURE_FOUNDRY_BASE_URL kept).
Detached-worker regression extended.
This completes the agent's enumerable credential-env read surface (verified from
source). Decision (with maintainer): stop the Codex deep-gate here — the
remaining theoretical surface is cloud-SDK-internal credential-source endpoints
on pre-existing infra, not vectors introduced by this PR; validated via Opus +
full suite. The PR's #3961 scope (os.environ mirror race on /api/providers +
/api/models) is fully closed plus the broader same-class hardening.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Round-3 gate found two more credential env vars the agent resolves cross-profile
that the scrub set missed, verified against the installed agent source:
- AWS_CONTAINER_CREDENTIALS_FULL_URI — read by model_switch._has_aws_creds:1360
(botocore container-credentials provider; RELATIVE_URI was present, FULL_URI
was not).
- Azure identity family — AZURE_CLIENT_ID / AZURE_CLIENT_SECRET / AZURE_TENANT_ID
/ AZURE_FEDERATED_TOKEN_FILE (agent/azure_identity_adapter.py service-principal
+ workload-identity model auth) and AZURE_FOUNDRY_API_KEY
(runtime_provider.py).
Enumerated the agent's full credential-env read surface from source to close the
whole class in one pass rather than one var per gate round. Deliberately EXCLUDES
config/region vars (AWS_REGION, AWS_DEFAULT_REGION, AZURE_FOUNDRY_BASE_URL) — the
child probe may legitimately need those. Regressions extended to pin FULL_URI +
the Azure family.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex re-gate found the registry-derived scrub still missed credential env vars
the agent resolves via raw os.getenv() that aren't in PROVIDER_REGISTRY:
CUSTOM_API_KEY (hermes_cli/models.py), AWS_BEARER_TOKEN_BEDROCK
(hermes_cli/model_switch.py), AZURE_ANTHROPIC_KEY (runtime_provider.py), and the
boto3 AWS_* credential family (agent/bedrock_adapter.py). Same cross-profile leak
class on custom/AWS-backed provider quota + detached-worker paths.
- Add _NON_REGISTRY_AGENT_CREDENTIAL_ENV_NAMES fail-closed set, union into
_profile_secret_env_names so BOTH leak sites pick it up.
- _account_usage_subprocess_env now relies on the centralized scrub set (WebUI-
only fallback only if the api.profiles import fails), dropping the partial
local AWS pair.
- 2 more regressions (quota subprocess + detached worker) covering CUSTOM_API_KEY
+ AWS family, fail without the fix.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex re-gate of #4544 reproduced a residual cross-profile leak: the PR closed
OPENROUTER_API_KEY but ANTHROPIC_TOKEN / CLAUDE_CODE_OAUTH_TOKEN still leaked the
server-process credential to an empty named profile on /api/provider/quota
(anthropic) and the detached-worker model-rebuild path, because the scrub set was
built only from the WebUI _PROVIDER_ENV_VAR map (which omits OAuth/token-flow
provider env vars).
Widen the shared _profile_secret_env_names() scrub source to union in the agent
registry's api_key_env_vars (hermes_cli.auth.PROVIDER_REGISTRY) so every
credential env var the agent runtime can resolve via raw os.getenv() is scrubbed
in profile-scoped reads. Fixes both leak sites at once (both route through
_profile_secret_env_names). + 2 regressions that fail without the fix.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* stage #4620 (franksong2702): snapshot isolated-profile opt-in at import (fixes#4590)
Security/deployment-hardening follow-up to #4586/#4589. Snapshots
HERMES_WEBUI_ISOLATED_PROFILE at module import (_INITIAL_ISOLATED_PROFILE_OPT_IN)
so _isolated_profile_opt_in() reads the startup snapshot, not live os.environ — a
pinned profile .env loaded later can't flip the isolation posture. _reload_dotenv
key filter kept as defense-in-depth. Adds one-time warning when HERMES_HOME has a
profile-dir shape but isolated wasn't opted in at startup. Code byte-faithful from
PR head 5ef14264.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
* Release v0.51.566 — Release TY (isolated-profile opt-in hardening; #4620)
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Codex re-gate (CORE, reproduced): _reload_dotenv() was guarded, but get_profile_runtime_env() also
copies a profile .env into runtime env, which profile_env_for_background_worker()/streaming apply to
os.environ -> HERMES_WEBUI_ISOLATED_PROFILE=0 in a profile .env still escaped isolation on that path.
Fix: filter _PROTECTED_ENV_KEYS in get_profile_runtime_env's .env loop + add the key to
_BLOCKED_RUNTIME_ENV_KEYS (gateway-parity filter). + runtime-path regression test.
Codex gate (CORE, reproduced): _reload_dotenv() copies a profile's .env into os.environ, so a
contained user could set HERMES_WEBUI_ISOLATED_PROFILE=0 in their own profile .env and silently
escape isolation. Fix: _PROTECTED_ENV_KEYS — _reload_dotenv refuses to override operator/deployment
keys from a profile .env (logs a warning). + regression test proving .env can't disable isolation.
Profile-isolation regression: isolated mode was inferred from HERMES_HOME path shape (*/profiles/<name>),
which the launcher exports for ANY normal named profile -> single-users on a named profile lost profile
switching + saw only 1 profile. Fix: HERMES_WEBUI_ISOLATED_PROFILE explicit opt-in is now the PRIMARY gate.
nesquena-hermes