profile_env_for_background_worker mirrored the profile home into process-global
os.environ["HERMES_HOME"] and yielded the worker body OUTSIDE the setup lock, so a
concurrent cross-profile worker (e.g. background title generation on another
profile) could clobber os.environ mid-body — the agent config reader then resolved
the WRONG profile's config (intermittent turn-init failures citing an unused
provider; b3nw's v0.51.849 repro). Now installs hermes-agent v0.18.0's
context-local home override (set_hermes_home_override, a ContextVar that
get_hermes_home() consults before os.environ) for the resolved non-default profile
home — set before yield, reset in finally — so a config read in the worker body
resolves THIS profile's home from task-local state, immune to a concurrent
os.environ clobber. No worker serialization, no os.environ mutation, no detection
heuristic (both earlier WebUI-only mitigations were rejected as regression-unsafe).
Graceful degradation: the override module is resolved lazily+optionally
(_resolve_hermes_home_override, mirroring _resolve_secret_scope_module). On agents
< v0.18.0 the symbol is absent -> returns None -> pre-existing os.environ-mirror
behavior unchanged (no regression, no new failure mode).
Self-built (nesquena-hermes); nesquena independent review APPROVED.
Codex round-3 gate finding: _dedupe_replayed_context_messages deep-copies the
stale-user repaired boundary row (streaming.py:4501) into the context array. When
the mint ran AFTER dedupe, that deep-copied context row was no longer the shared
result dict, so it stayed id-less while the display copy got the minted id —
re-opening the cross-array id divergence for the stale-repair path. Fix: move
_assign_stable_message_ids() to run immediately after _restore_reasoning_metadata
and BEFORE _dedupe_replayed_context_messages at all four commit sites (streaming
main / in-band self-heal / except-path self-heal + routes runs/MoA), so the shared
result rows are stamped before any deep-copy — both arrays inherit the id.
Codex/Opus gate finding: in eager session-save mode the current user turn is
checkpointed into s.messages before the agent runs (no id yet); after
_assign_stable_message_ids stamps the returned result row, the display merge
keeps the durable eager checkpoint and skips the stamped result — leaving the
display row id-less while its context twin carries the minted id, silently
defeating id-based fork/truncate alignment for eager-mode users. The merge now
copies the minted id onto the kept checkpoint when it lacks one (guarded so it
can't overwrite or duplicate). Opus verified cross-array id pairing holds across
the real 3-turn transform pipeline.
WebUI keeps two parallel arrays per session: messages (display transcript) and
context_messages (what's sent to the model). In large/compacted sessions they
diverge. Forking / in-place truncation copies a prefix of both and relies on
truncate_context_for_display_keep() to translate a display keep-index into the
matching context index — an aligner that prefers a stable per-row `id`. But the
model-context rows carried neither id nor timestamp (a 989-session live scan
found 0 with any id), so alignment fell back to fragile content-signature
matching that goes ambiguous on repeated tool calls / empty assistant turns.
This closes the DATA gap (the alignment logic itself already prefers `id` and
handles the compacted case via the matcher, shipped in #5563): mint a monotonic,
session-unique integer `id` on the per-turn result rows AFTER the context restore
and BEFORE both arrays are built, so the display and model-context copies of a
logical row share the same id.
- api/streaming.py: new _assign_stable_message_ids(); _restore_reasoning_metadata
carries `id` forward across turns (as it already does timestamp); wired into all
three streaming commit sites (main turn, retry, self-heal).
- api/routes.py: same mint on the runs/MoA _handle_chat_sync commit path.
- api/gateway_chat.py: mints ids on the two new rows of a gateway turn.
- The `id` is stripped before the provider API call (not in _API_SAFE_MSG_KEYS),
same treatment as timestamp — nothing new reaches the provider.
session_ops.py alignment was already delivered by #5563 (id-preferring matcher +
compacted-case fall-through), so no change is needed there now; tests updated to
assert the post-#5563 behavior (id-bearing = exact cut; id-less = errs toward
under-keeping, never the old raw-index mis-cut).
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Move the 1-5s commit_session_memory() extraction off the POST /api/session/new
request thread (was blocking + New Chat) into a fire-and-forget daemon thread,
guarded by the existing per-session in-flight serialization.
Data-loss prevention on managed stop:
- server.py installs a SIGTERM handler that requests an orderly stop so
serve_forever()'s finally block runs drain_all_on_shutdown() (the default
SIGTERM disposition terminated without unwinding, losing in-flight commits).
- session_lifecycle: bounded self-unregistering background-commit registry;
a _draining flag so no NEW worker starts once the drain begins (the inline
generation-drain commits any late arrival instead); a 30s overall drain
deadline so a stuck worker cannot hang the stop path forever.
Co-authored-by: luperrypf <luperrypf@users.noreply.github.com>
Fork ("Fork from here") on a large/compacted session could cut context off a
real turn boundary — slicing mid-turn or leaving a dangling tool call, feeding
a malformed context to the model. Align the fork cut to a resolved turn boundary
via a signature matcher handling both compacted and non-compacted display
coordinate spaces; errs toward under-keeping with send-time sanitization backstop.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Rebuilt the combined #5556+#5553 fix on v0.51.859 (now has the flake fix + the
#5542 RFC de-brittle, so no anchor-test collateral). Nathan's call: rodboev's
#5556 primary + fold self-built #5553 improvements, credit both.
Clear handler now:
- routes through shared truncate_session_at_keep(s,0) (single source of truth,
sets watermark=_truncation_watermark_for([])==0.0, the #2914 sentinel that
blocks state.db append-merge replay)
- detaches compression lineage ONLY when the parent is a pre_compression_snapshot
(Codex-caught: preserve genuine fork parent links for nesting + "Forked from")
- rodboev's persisted-clear read-back verification + stale .bak removal
RFC conflict resolved in favor of the de-brittled (symbol-anchor) master version.
Ships both PRs' test files (state_db_replay + clear_truncation_watermark incl the
fork-preservation regression). Opus (on the amended tree): both resurrection paths
closed, detach lineage-safe, verification sound — SHIP. 3 non-blocking follow-ups
filed (#5570 .bak crash-window, #5571 fork-stitch corner, #5572 messaging clear).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
When the WebUI server is launched from a venv python whose PATH does not
include git (e.g. hermes-agent venv on Windows), shutil.which('git')
returns None and WEBUI_VERSION falls back to 'unknown'. That freezes the
?v=<stamp> static-asset cache key, so browsers keep serving stale cached
ui.js/messages.js even after the server is restarted with fixed code —
frontend fixes silently never reach clients (recurring scroll jump-back).
Add a Windows fallback to _resolve_git_executable(): read the
Git-for-Windows registry InstallPath (HKLM + WOW6432Node + HKCU), then
probe common install dirs. Mirrors the existing darwin /usr/bin/git
fallback. No behavior change on posix or when git is already on PATH.
Codex 5th CORE: the ownership set only covered static _PROVIDER_MODELS, not the
active provider's own config.yaml providers.<active>.models allowlist. So an
active provider defined purely via providers: (no static catalog) could be
hijacked by another slug's overlapping entry earlier in config order. Fix folds
the active provider's own providers: allowlist (list + dict shapes, copilot
excluded) into _provider_models_set — completing the ownership model so the
active provider owns its declared models from EITHER the static catalog OR its
own providers: entry. Regression test (fail-without-fix verified). 40 model_resolver
tests green.
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex 4th CORE (root latent bug in the shared canonicalizer): _canonicalise_provider_id
only accepted an alias resolution if the target was in _PROVIDER_DISPLAY. But some
canonical targets (e.g. 'gemini') are indexed in _PROVIDER_MODELS and NOT
_PROVIDER_DISPLAY, so 'google-gemini'->'gemini' was rejected and returned raw
'google-gemini' — leaving the ownership guard empty and letting providers.openai.models
hijack an active Gemini-owned model. Fix mirrors the direct-hit check (line 1445):
accept alias targets in EITHER _PROVIDER_DISPLAY OR _PROVIDER_MODELS. This is the
root fix for the whole alias-canonicalization class (gemini/google-gemini/
google-ai-studio etc.), not just the reported site. Blast-radius verified: 81
canonicalise/provider-alias tests across 5 files still green; x-ai still stays x-ai,
opencode_go still folds. Gemini regression test added (fail-without-fix verified).
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex 3rd CORE (pre-existing latent bug the providers: scan exposes): the
_provider_models_set ownership lookup used the RAW config_provider (e.g. 'z-ai')
but _PROVIDER_MODELS is keyed by canonical slug ('zai'), so an aliased active
provider got an empty ownership set -> _skip_custom_providers guard-2 silently
failed -> another providers.<slug>.models entry could hijack an active-owned
model (verified: active z-ai + glm-5 + providers.openai.models.glm-5 -> openai).
Fix: canonicalise config_provider before the _PROVIDER_MODELS lookup (and reuse
the canonical slug for the scan's ownership guard). Regression test with an
aliased active provider (verified fail-without-fix).
This is the root-cause fix for the alias class — the ownership guard now holds
for every aliased provider (zai/kilo/etc.), not just the reported site.
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex 2nd CORE: the new providers: scan didn't honor the _skip_custom_providers
active/default ownership guard (config.py:2535) that the custom_providers scan
uses. So with an active provider (e.g. ai-gateway) + model.default=gpt-5 AND a
providers.openai.models.gpt-5 entry, the scan hijacked gpt-5 to openai instead of
the active ai-gateway path. Fix: when _skip_custom_providers is true, the
providers: scan only considers the active provider's own canonical slug (skip all
other slugs). Regression tests: active-provider-owns-default not hijacked (fails
without fix); active provider's OWN providers: entry still matches.
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Gate-cert (nesquena-hermes) + Codex found 1 CORE: akay64's new providers: scan in
resolve_model_provider() iterated ALL providers.<slug>.models INCLUDING
providers.copilot.models — but that key is a documented per-model SETTINGS map
(reasoning_effort/limits), NOT a routing allowlist (config.py exception). So a
Copilot per-model settings entry hijacked that model's routing to Copilot
(Codex reproduced gpt-5 -> copilot). Fix (Codex-exact): skip copilot in the scan.
Added the regression tests the PR shipped 0 of: user-defined allowlist routes;
active-openai + providers.copilot.models (dict + list shapes) stays openai
(both verified to fail without the exclusion).
Co-authored-by: akay64 <akay64@users.noreply.github.com>
Codex v4 SILENT finding: my audit Finding-2 fix made the orphan-.bak branch emit
a state_db_deleted_webui_tombstone item, but the state_db_missing_rows loop still
emitted the same sid/kind again when both a surviving .json.bak AND a state.db row
exist for one deleted session — double-counting unsafe_to_repair, violating the
no-double-classification invariant. Fix: track _bak_tombstoned_ids from the .bak
branch and skip them in the state.db loop. Regression test
test_audit_no_double_count_when_bak_and_state_db_row_both_survive (verified fails
without the dedup).
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
@rodboev independently pushed fixes for messaging-tombstone (matches ours) and an
audit orphan-.bak skip — but the audit skip used _marks_deleted_webui_session
(index heuristic), which re-introduces the crash-suppression bug, and the other 4
resurrection/recovery paths were still open. Rebuilt the stage on rodboev's new
head (fc21c3f3, attribution preserved) and applied the complete superset:
- routes.py claim path: tombstone 404 gated on _state_db_session_source in
('','webui','fork') so foreign-source rows always materialize (finding B)
- models.py sidebar projection: skip tombstoned source='webui' rows w/ no live
sidecar so a deleted session can't resurface in /api/sessions (finding C)
- session_recovery.py orphan-.bak startup: skip restore when durable tombstone
present, durable-only (finding D)
- session_recovery.py audit orphan-.bak: durable-only classify as
deleted_session_skipped (corrects rodboev's index-heuristic version)
- session_recovery.py _read_state_db_missing_sidecar_rows: durable-tombstone-only
so a genuine crash (index intact, no durable tombstone) still repairs (finding F)
Regression tests (all verified fail-without-fix): sidebar-projection, orphan-.bak
startup skip, messaging-not-tombstoned, index-only-crash-repairable. Both advisors
(Codex SAFE TO SHIP + Opus ship-with-fixes) cleared the prior stage; this rebuild
carries the same fixes onto the contributor's latest work.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Mirrors the existing custom_providers: scan immediately above. User-defined
providers declared under config.yaml's providers: section with explicit
models: lists were invisible to model-to-provider resolution — bare model
IDs fell through to the stale config_provider, routing through the wrong
endpoint.
The insertion site sits after custom_providers: (unchanged) and before
all heuristics (@provider:model, slash, fallback — all unchanged).
- Add test_profile_model_selection_accepts_model_in_overflow_extra_models
(the extra_models overflow-bucket regression test the warm dossier flagged
as the #5453 ship-pass note).
- Drop the stray dead pseudo-code comment left in handle_post branch route
(#5449 cosmetic nit).
read_importable_agent_session_rows() is a pure read, but it opened a
read-write sqlite connection on the live multi-GB WAL state.db and re-ran a
defensive CREATE INDEX self-heal on every sidebar build. Holding a
write-capable handle while the agent streams into the same DB adds needless
checkpoint/lock surface.
Open the DB read-only (file:...?mode=ro) for the projection, and self-heal a
missing idx_messages_session through a separate short-lived writable
connection only. With the index present (the normal case) the read path
performs zero writes; when it is missing the self-heal still runs and rows
still come back.
Behavior-neutral: identical rows when the index exists; self-heal preserved
when missing; read-only-fs degradation to the pre-aggregated path unchanged.
Tests: tests/test_issue5455_listing_readonly_connection.py (read-only open,
no writable connection when index present, self-heal when missing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nesquena-hermes