The project-scoped recovery pass asked for the global window's exhaustion
signal but never for its own. A scoped query carries a raw candidate window
of its own (limit * 8, re-widened to limit * 32), and compression segments
plus the post-projection visibility filters are spent from that window
BEFORE the logical slice. A lineage-heavy project therefore came back short
with its window fully consumed while older assigned conversations of the same
project still waited behind it, and the loop advanced to the next project —
leaving that project under-delivered on every rebuild.
The scoped query now asks for `return_window_exhaustion`, and a binding
window buys exactly one re-query at the widest limit this pass is allowed to
scan (`query_limit`). The retry is bounded to one extra query per starved
project and cannot out-read the pass's own scan budget.
The regression test builds a starved project whose scoped raw window is
filled by its own compression segments, with an older conversation of the
same project just behind the window. Against the previous head that
conversation was lost — the project recovered only its chain tip — while the
new head recovers it and pays exactly two scoped queries (limits 4 then 8:
one widening step, inside the ceiling). Verified red without the fix and
green with it; the full file is 73 passed, and the sibling suites this branch
touches (test_issue2628_cli_sessions_perf, test_issue4842_cron_projection_perf)
bring the run to 86 passed.
The unassigned refill's proportional widening degenerated in two shapes, and
both spent the forced ceiling query -- 200 conversations read, sidecar-stat()ed
and merged into the payload -- to deliver a 20-row window.
* A refill query that finds ZERO still-unassigned conversations has no density
to extrapolate from, but max(unassigned_seen, 1) evaluated the estimate as
target * width anyway (20 * 40 = 800, clamped to the ceiling). Any profile
that had moved >= 21 of its newest conversations from the WebUI went straight
to a full 200-row read where the 60-row exact-width floor fills the window on
the nose. The proportional term now applies only when the window in hand
actually saw an unassigned conversation.
* When the moved conversations cluster just below the window edge, every
re-count returns the same unassigned count, so the estimate advanced ~2 rows
a time (21, 23, 25): all four queries spent and the last reading the whole
ceiling band. The widening is now also geometric, paced by the queries still
left (width // queries_left, so width/3 then width/2) -- bolder as the forced
ceiling read gets closer, because a too-narrow last estimate costs that whole
read while a too-wide one costs only its own surplus.
Measured on a 5000-conversation state.db (warm cache), unassigned query widths
/ rows the model carries:
* newest 40 moved: [40, 200] 200 rows -> [40, 60] 60 rows, 400 -> 160 stat()
* 8 moves clustered at the bottom edge of 200: [21, 23, 25, 200] 200 rows /
192 unassigned -> [21, 28] 28 rows / 20 unassigned
* every clustered size up to 22 moves is off the ceiling path (42 rows, three
queries); 23+ still ends on the ceiling read, as before
* no WebUI-side moves: unchanged, no refill query at all
Unchanged by construction: the widening is still strictly increasing so the
loop cannot stall, the query count is still bounded by
UNASSIGNED_CLI_REFILL_MAX_QUERIES, and the last query still reads the full
ceiling band -- so the window still fills whenever at least 20 of the newest
200 state.db-unassigned conversations are unassigned once sidecars are read.
The one deliberate cost: a profile whose newest ceiling band is ENTIRELY moved
now climbs (40, 60, 90, 200) instead of (40, 200), paying two extra narrow SQL
queries for the same 200-row payload.
The trade-off note is rewritten with it. "A few spare conversations" was the
best case (24 rows for 20); the worst case inside the same bound is the whole
ceiling band, so the note now states that, plus the two things the old bound
omitted: CANDIDATE_WINDOW_MULTIPLIERS = (8, 32) means a ceiling-width query
touches 1600 raw rows (6400 re-widened), and every bound here is per profile
context, not per sidebar build -- all_profiles=1 runs the loader once each.
The refill's allowance for sidecar-carried assignments was computed once,
from the rows fetched BEFORE it ran, so every moved conversation the refill
itself reached spent an unassigned slot again -- the failure the allowance
was added to remove:
* 30 conversations, 3 moved from the WebUI at the bottom edge of the 20-row
window (cli-08..cli-10): 19 unassigned rows in the payload, and cli-07 --
genuinely unassigned, present in state.db -- never fetched at all;
* 40 conversations, 11 moved straddling the boundary (cli-15..cli-25): 15
unassigned rows, with cli-09..cli-13 never fetched.
Both deliver 20 now. The refill re-classifies its OWN result and widens
again while the window is short. It widens proportionally (by the moved
share observed so far), not one row at a time, so it converges for moves
that cluster as well as for moves spread through the history.
Bounds per sidebar build: at most UNASSIGNED_CLI_REFILL_MAX_QUERIES (4)
unassigned queries, none reading deeper than
UNASSIGNED_CLI_REFILL_SCAN_CEILING (200) conversations, stopping as soon as
the window is full or the filter demonstrably has nothing deeper to give.
The last of the four reads the whole ceiling band, so the window fills
whenever at least 20 of the newest 200 state.db-unassigned conversations
are still unassigned once their sidecars are read. Past that (>180 of the
newest 200 moved from the WebUI) the sidebar is honestly short instead of
paying unbounded I/O -- that boundary is stated in the code and pinned by
a test.
No I/O per row is added: the re-count reuses the scan's memoized sidecar
and project-catalog caches (#4842).
The review asked for "confirming the project still exists for the
session's profile". profile_scoped_project_ids() read
get_active_profile_name(), but _load_cli_sessions_uncached runs once per
profile context in the ?all_profiles=1 view, so every context resolved
against the ACTIVE profile's catalog.
A LIVE assignment owned by another profile was therefore misread as
unresolvable: coerced to None, skipped by the assigned recovery pass, and
unreachable by the unassigned refill too (its state.db row does carry a
project_id, which the "unassigned" SQL filter excludes), so the row left
the payload entirely — while its chip stayed selectable in that same view,
because /api/projects returns every project when all_profiles=1.
Thread the ctx_profile already available at the call site through to the
catalog read. The safe direction is unchanged: an id that belongs to a
different profile, or to no project at all, still coerces to None.
/api/session/move writes project_id onto the WebUI sidecar only; nothing
writes state.db.sessions.project_id. The model's unassigned counter read
state.db exclusively, so it believed all 23 conversations were unassigned
and stopped at its 20-row window — while the route, which sees the
sidecar's project_id through all_sessions(), classified the 3 moved
conversations as ASSIGNED. Net result: 17 unassigned CLI rows in the
payload and cli-00/01/02 never fetched at all, the reviewer's exact
3 + 20 -> 17 number.
Resolve the assignment from the source of truth the route ends up using:
_state_projection_sidecar_metadata (already the UI-owned title/archived
reader, already stat-cached) now also returns project_id, and
_interactive_row_project_id falls back to it when state.db has none. The
unassigned refill query is widened by the number of sidecar-only
assignments, because the SQL 'unassigned' filter cannot see them and would
otherwise spend those slots re-fetching rows already represented.
The GROUP BY probe that finds the projects a global assigned query starved
only fired when that query came back logically FULL. A query can instead
stay short because its bounded raw candidate window was entirely consumed
by compression segments and hidden rows, while older assigned
conversations still exist below it. Read as "the database has nothing more
to give", that shortfall skipped the probe and left the quiet projects
unreachable.
Have read_importable_agent_session_rows report whether the final raw
candidate window was exhausted while its logical projection stayed short
(opt-in via return_window_exhaustion, so the list API is unchanged for
every other caller), and gate the probe on saturation OR that exhaustion.
A short result from an unfilled raw window has genuinely seen every
candidate, so the common one-global-query fast path still pays no probe.
The refill after a saturated global assigned window was cut to the first
32 starved projects in deterministic neediest-first order. The recovery
is re-derived from state.db on every sidebar build, so the same 32 won
every time and every later project stayed unreachable on each rebuild —
the exact starvation the pass exists to repair (greptile P1 on #6659).
Drop the project-count cap. The follow-up's work is bounded by
construction instead: it only fires when the global window came back
saturated, each scoped query is limited to one project's remaining
budget, and the sum of those budgets across every starved project is at
most effective_limit * len(projects) <= PROJECT_ASSIGNED_CLI_SCAN_CEILING.
A fixed cap bounded nothing that the per-project limits did not already
bound; it only decided which projects were permanently left out.
The per-project budget was applied to the rows of ONE global recency-ordered
query, so it was not a per-project bound: when a single project owns the whole
`limit * len(projects)` window, every other project's assigned conversations are
never considered and their chips show nothing (greptile P1 on #6659, reproduced
at the real 200 as well as scaled down).
Two tiers now:
* the global query still runs first and is the only cost a profile whose assigned
history fits in the window pays;
* when it comes back SATURATED, one GROUP BY over sessions.project_id says which
projects still own undelivered rows, and each of those gets its own
project-scoped query with its own remaining budget. `project_ids` narrows the
lineage CTE's seed, so a per-project query is cheaper than the global one and
still returns whole compression lineages.
Cost stays bounded: worst case per build is 1 global query + 1 GROUP BY probe +
PROJECT_ASSIGNED_CLI_REFILL_PROJECTS (32) project-scoped queries, each bounded to
one project's remaining budget. The scan ceiling is now shared equally between
projects (ceiling // len(projects)) rather than first-come, so the recovered
payload stays under the ceiling without letting the loudest project take it all.
profile_scoped_project_ids() duplicated the body of the canonical
api/profiles.py::_profiles_match, whose docstring says it was relocated there
precisely so callers stop duplicating it. Call it instead.
Behaviour-identical: both spellings agree on all twelve None / '' / 'default' /
renamed-root / foreign-profile combinations, which the new parametrised test
asserts cell for cell against _profiles_match itself. It is a drift guard, not
a regression test — swapping the call back to a raw
`p.get('profile') == active` comparison reddens seven of its twelve cells.
_merge_state_row overwrote an already-represented lineage unconditionally, and
recovery pass 2 (the unassigned refill) merges into the same lineage-keyed
table pass 1 fills. A row already carrying a RESOLVED project_id could
therefore be replaced by a bare unassigned projection — deleting the chip this
PR exists to restore and swapping the row's identity from the lineage tip back
to its root.
Reachable wherever the SQL project_assignment filter and the Python lineage
projection disagree about where a compression lineage starts. They differ today
on a blank `source`: _is_continuation_session treats an empty/whitespace source
as "unknown, same conversation", while the lineage CTE compares
LOWER(TRIM(parent.source)) = LOWER(TRIM(child.source)) and so will not walk
from an assigned tip up to a blank-source root. The mixed first pass collapses
the pair into one assigned row; pass 2 then hands the root back as its own
"unassigned" conversation and clobbers it.
The merge is now one-way: overwrite only when the incoming row ADDS a resolved
assignment, or when the row on file has none to lose. Fixing the merge rather
than each possible divergence — it is the one place where "already resolved" is
knowable locally, and pass 2 by construction has less assignment context than
pass 1.
tests/test_issue2841_show_cron_sessions_toggle.py reads this call site as
SOURCE TEXT (`'("cron", "webhook", "kanban") if source_filter is None'`), so
routing it through the new BACKGROUND_CLI_SOURCES constant turned a
pre-existing test in the maintainer's own shard gate red for a
behaviour-neutral refactor.
Put the literal back at that one call site. BACKGROUND_CLI_SOURCES stays where
the kanban-chip fix actually needs it (the background-source check in
_interactive_row_project_id, and interactive_excluded for both recovery
passes). A new test builds the expected literal FROM the constant and asserts
it appears verbatim in api/models.py, so the two spellings cannot drift
without failing here rather than in an unrelated file.
Recovery pass 2 was gated on `first_pass_count >= unassigned_target`, reading a
short first pass as "the database has nothing left". It can also mean the mixed
pass ran out of RAW candidate window: compression segments are spent from it
before the logical slice, so a lineage-heavy profile gets fewer conversations
than it asked for and the unassigned quota stays short. Also run the refill when
assigned conversations inside the window displaced unassigned candidates, which
is the case a narrower unassigned-only query can actually repair. A profile with
no assignments and a small state.db still issues no extra query.
While here, spend the per-project assigned budget through `_merge_state_row`'s
return value instead of pre-incrementing, so the budget counts logical
conversations the way PROJECT_ASSIGNED_CLI_LIMIT documents, and seed it from the
assigned rows already inside the window so the bound stays a total rather than
one bound stacked on another. That accounting is behaviour-neutral today (the
assigned query is newest-first, so window rows are always the first rows it
returns) — it makes the documented contract hold by construction rather than by
that ordering accident, and removes an unused return value.
`_interactive_row_project_id` resolved state.db assignments for anything that
was not cron or webhook, but the interactive projection loop is also what a
`source_filter='kanban'` scan walks. The same kanban row therefore reported two
different chips depending on the caller: None in the default sidebar (its own
bounded second pass, upstream behaviour) and a resolved project_id under
?source=kanban.
Route every source that owns a bounded project-chip pass back through
`_state_row_project_id`, and name that set once (BACKGROUND_CLI_SOURCES) so the
exclusion list and the chip decision cannot drift apart.
`?all_profiles=1` passed `project_assigned_limit=False`, on the premise that
its `visible_session_limit=None` first pass already returns every conversation.
It does not: `_load_cli_sessions_uncached` maps `visible_session_limit is None`
to CLI_VISIBLE_SESSION_LIMIT for the interactive pass, so that view truncates
by recency exactly like the single-profile one. Only cron/webhook/kanban limits
reach the reader as `limit=` directly, where None does mean unbounded.
Net effect of the old value: in the "All profiles" sidebar a project-assigned
CLI session older than the newest 20 interactive conversations was missing from
/api/sessions while its project chip still claimed it — the #6659 bug itself,
in the one view that shows every profile at once. Recovery pass 2 (the
unassigned refill) still ran there, so the two passes disagreed as well.
Pass PROJECT_ASSIGNED_CLI_LIMIT instead of None so the per-project bound and
the PROJECT_ASSIGNED_CLI_SCAN_CEILING guard apply per profile, rather than
issuing an unbounded assigned scan for every profile on the box.
Letting assigned sessions past the *recent* cap was right; letting them past
*all* bounds was not. Three linked defects from review of #6659:
1. The assigned list was effectively unbounded. models.py's 200 cap only applied
to state.db's own pass; imported CLI sidecars from all_sessions() never pass
through it and _cap_recent_cli_sessions() retained every assigned row (1,000
assigned rows -> 1,000 returned). The FINAL MERGED payload is the only place
that sees every source of assigned rows, so bound it there - per project, so
one busy project cannot evict another's history (the >200-assigned case
greptile flagged), and counting logical conversations, not raw rows.
2. Assigned rows spent the unassigned quota: 3 assigned + 20 unassigned yielded
17 unassigned, and the model fetched 20 rows total up front so bumping only
the route counter could not refill. Count the two separately in the route cap
and add a bounded unassigned refill pass, keyed on the lineage, so 20 unique
unassigned logical conversations survive sidecar/lineage dedup. The refill
only runs when the window was actually saturated AND came up short.
3. A deleted or cross-profile project_id hid the session outright: it was
surfaced unchecked, so the row left "Unassigned" and its overflow became
default_hidden with no chip left to reveal it. Resolve non-system ids against
profile-scoped load_projects() and coerce a miss to None BEFORE either cap -
the mirror of the bug this PR fixes.
read_importable_agent_session_rows() grows a `project_assignment` filter
('assigned'/'unassigned') replacing the require_project_id boolean. Both arms
key on the whole compression lineage and are exact complements, so neither
budget can double-count a lineage.
_state_row_project_id() keeps its (sid, source) signature: master's kanban pass
calls it that way, and giving it a row-dict signature turned that whole pass
into a swallowed TypeError that dropped every kanban row.
Two independent 20-session caps hid older project-assigned CLI sessions:
models.py only queried the 20 most recent, and routes.py capped the API
payload again. Fixing the query alone was not enough.
- agent_sessions.py: optional project_id filter; propagate the freshest
project_id along a compression lineage (compression_tip now returns a
3-tuple) so a Project set only on the tip is not lost
- models.py: second bounded query for older project-assigned sessions;
skipped when the schema has no project_id; dedupe by id/lineage
- routes.py: keep the 20 recent, drop unassigned overflow, preserve
assigned overflow as default_hidden (revealed by the project chip)
Adds regression tests for the limit, the root/tip lineage cases and the
route cap.
The row-id fast path for a marked native-image mirror row continued
without advancing state_replay_idx, unlike the ordinary and multimodal-
mirror replay paths. The edited checkpoint then never read as consumed,
so a rerun reply that exists only in state.db was truncated. Master (no
marker branch) keeps that reply.
Advance the replay index when the resolved sidecar row is the next
expected visible row, the same guard the mirror branch uses. Adds a
regression that fails without the fix: sidecar [u1,a1,img#42,a_img,uE],
state [.., marked img#42, a_img, u_old, a_old, uE, aE], watermark 900,
boundary 861 -> aE survives, u_old/a_old stay out.
Found by the release-manager advisor gate on #7754.
Apply the existing truncation and checkpoint-order eligibility to marked
state.db image rows before insertion. Keep removed turns out of full,
limited, and model-context reads without changing no-watermark replay.
Preserve conflicting provider payloads for a uniquely proven retained
image row across Undo without restoring distinct removed rows.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Keep image-turn context tagging and upstream's typed steer-wrapper cleanup
in the same settlement path. Verify the combined recovery and streaming
behavior against current master.
Assisted-by: Hermes Agent:gpt-6-sol
Count a proved WebUI-owned pending row in the display baseline before
accepting a state.db self-heal. Without assistant output past that turn,
leave pending state for the run journal to restore partial text and the
interruption marker instead of clearing it prematurely.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
When a dead stream is reconciled from state.db, retain the submitted
prompt and attachment in the sidecar while keeping the Agent-enriched
row in model context. Use one parent snapshot for full-session conflict
projection so the sidecar owner is stable across the display merge.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
Apply the same display-only conflict projection to full and paginated
session reads before cache and window selection. Keep both provider payloads
for model replay while retaining a single visible sidecar bubble.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Record pending-turn proof under caller-owned session locks during normal
runs and credential recovery. Preserve both conflicting provider payloads
for replay, but display one sidecar-owned bubble when row identity and
visible content are unambiguous.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Persist trusted pending Agent timestamp evidence before the model call so
session reads, including metadata polls and cold reloads, keep only the
submitted prompt and attachment visible. Preserve conflicting provider
payloads without guessing which copy is newer.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
The suppression predicate _transcript_already_advanced_past_pending()
required a run-journal terminal 'completed' state, but its caller in
_apply_core_sync_or_error_marker already returns early on that same
condition. The predicate branch was therefore unreachable dead code: it
could never suppress anything, and stale-cancel recovery could still
append a duplicate recovered user row, a journal clone and an
interruption marker for a turn that actually finished.
Derive the completion evidence from the crash-safe turn journal instead:
_turn_journal_records_completion() reports True when the exact-stream
'completed' event is the latest terminal state for the pending stream id.
The run journal's terminal write is not durable evidence for this path
(it is consumed by the caller's early return), so the turn journal is the
only still-observable source. A cancelled/crashed turn (interrupted)
keeps recovery active, and completion for another stream of the same
session never settles the pending one.
Also record the same completion event on the Gateway success writeback
path, so Gateway-backed runs have the equivalent completion persistence
instead of leaving the run journal's lost terminal write unrecoverable.
Regression coverage:
- test_turn_journal_helper_reports_completed_for_matching_stream
- test_turn_journal_helper_rejects_interrupted_turn
- test_advances_past_pending_from_turn_journal_without_run_journal_terminal
- test_full_recovery_suppresses_duplicates_on_turn_journal_completion
- test_full_recovery_appends_rows_on_turn_journal_completion_loss
- test_gateway_chat_worker_records_turn_journal_completion
Reverting the predicate gate turns the two new CORE regressions red;
reverting the Gateway write turns the Gateway regression red.
A persisted message_count is only authoritative when the write that
produced the messages array also vouched for it. Sidecars written by an
older writer (recovery materialization copied the denormalized state.db
count over real rows) could carry a count stale against the array beside
it, and the prefix shortcut then reads a real shrink as a growth and
skips the #1558 backup.
save() now stamps _mc_v alongside message_count, and _prefix_message_count
refuses any count without the current marker: those files get the full
parse, exactly as on the legacy path. One save re-marks the file, so the
fast path resumes for steady state.
Regression covers the unmarked stale-count shape and the re-marking
transition; mutation-checked by removing the gate (the unmarked fixture
then skips the backup, as the review's reproduction predicted).
The #1558 backup safeguard in Session.save() compares the on-disk message
count with the incoming one and writes <sid>.json.bak only when the save would
shrink the array. Sound -- but it obtained the on-disk count by
json.loads(self.path.read_text()), the entire file, on EVERY save, including
the 99.9% of saves that grow the conversation and never back anything up.
Measured in production (Mac docker stack), in the running container:
sessions/fa3bca34a0c6.json 203,439,398 bytes / 266,940 messages
read_text 17,453 ms
json.loads 2,924 ms
TOTAL to obtain one integer: 20,377 ms <- paid on every save
That was the 39,357 ms /api/session request stuck at t6_after_json_write,
after the read-side re-parse had already been fixed. It scales linearly with
the file, and the file grows for as long as the conversation does.
The decision below needs ONE integer. save() already writes that integer into
the metadata prefix, before `messages`, as `message_count` -- exactly so
readers can have it without parsing the body, which is how
load_metadata_only() and the sidebar freshness check already work. So save()
now reads THAT, through the same bounded 64 KiB prefix reader, and falls back
to the full read + parse whenever the prefix does not carry a usable count: a
legacy pre-#5854 sidecar whose count serialized after the scenes, a file with
no top-level `messages` key at all, a corrupt or truncated prefix, metadata
alone that overflows the budget, or an unreadable path. Fail-open is the
contract -- never "assume no shrink". The full text is still read for one
thing only: the .bak body, when a shrink is actually detected.
An in-memory stat identity was tried first and is NOT sufficient, so it is
gone. The first revision of this branch stamped the (inode, size, mtime_ns) of
the bytes it wrote and trusted the remembered count while that tuple matched,
on the reasoning that "stat says nothing touched it, so the array is the one I
wrote". Review (nesquena-hermes, 2026-09-21) reproduced the hole: a stat tuple
is not a content identity. A same-length in-place rewrite inside one mtime tick
keeps inode, size AND mtime_ns -- ext4 stamps mtime from a coarse clock, so two
writes in the same tick share one mtime_ns -- and the next shrinking save then
reads a stale count, concludes it is growing, overwrites the longer transcript
and writes no .bak. That is the #1558 data-loss shape the safeguard exists to
prevent, reachable on this branch. A count read from the file's own bytes
cannot be fooled that way: any rewrite of the bytes rewrites the count.
So _disk_identity() is removed along with the identity fields on Session, the
stamp after the write, and the seeding in load(). _prefix_message_count() is
the replacement: bounded prefix -> the number the writer itself recorded.
One trap survives the redesign and is pinned. load() de-duplicates adjacent
partials and immediately saves the shorter transcript (#2592), and that save
must produce a .bak because it shrinks the array on purpose. The count that
decides it is the one the writer stamped on the file -- the ON-DISK length it
wrote -- not the post-collapse length of the object. Read the object's own
length instead and the #2592 backup disappears silently.
session_recovery._msg_count() pays the same cost at boot and is deliberately
left alone. Its "torn file -> -1" contract is what makes recovery restore a
.bak over a truncated live file, and no bounded read can prove a file is not
truncated. A boot pays it once; a save paid it every time.
Tests: 9 in tests/test_save_count_without_full_parse.py. The review's
reproduction is the last one -- a same-inode, same-byte-length, same-mtime_ns
in-place rewrite of 2 -> 5 messages, then a save of 3, which must leave the
5-message array recoverable. It fails against the removed stat-identity design
(re-verified by restoring that revision: no .bak is written and the
5-message transcript is lost) and passes here. Mutation-checked the same way:
recording the post-collapse length fails exactly the #2592 test. 783 tests
pass across the 41 files that touch this surface (save/load, metadata-only,
collapse, .bak recovery, sidecar reconciliation); ruff is clean on the diff.
Two findings from the round-1 maintainer review were not addressed
in round-2: the helper clamped an out-of-range value *up* to the
ceiling, and the env var was exposed to per-profile ``.env``
overrides, which contradicts the "process-wide resource limit"
intent and creates a first-load-wins asymmetry across profile
switches. This round fixes both.
``_read_max_session_resolve_concurrent()`` previously treated
out-of-range values like ``HERMES_WEBUI_MAX_SESSION_RESOLVE=999999``
as "the operator wants the maximum" and returned ``64`` — admitting
64 simultaneous unbounded transcript loads. The round-1 reviewer
measured ~9.6 MB per parse on a synthetic 4.27 MB transcript, so the
old ceiling permitted ~614 MB at 64-way concurrency before any
other application memory.
The cap is a *safety bound*, not a user preference. A typo must
fall back to the safe default, never to the permissive extreme —
matching the behavior of every other bad input in this helper
(missing/empty/zero/negative/non-numeric all already fall back to
2). The 64 ceiling is reachable only via an explicit in-range
operator value.
server-wide key, not a per-profile .env override (SILENT)
The cap controls a single process-wide ``BoundedSemaphore`` that
gates full-transcript resolves. A profile's ``.env`` is per-profile
by design; letting it set a server-wide limit lets a single
profile drag the rest into a tighter or looser posture than the
operator intended, with a first-load-wins asymmetry (a later
profile switch cannot resize the already-constructed semaphore).
The same shape as the isolated-profile key added in #4589: only
the operator/launcher env at startup can set it. The cap is added
to both ``_PROTECTED_ENV_KEYS`` (blocks the profile env at the
load step) and ``_BLOCKED_RUNTIME_ENV_KEYS`` (filters the profile
env in the runtime/gateway-parity path), so a profile cannot
inject the env var at any layer.
1. ``api/models.py`` — ``_read_max_session_resolve_concurrent()``:
- ``if n > 64: return 64`` → ``if n > 64: return 2``
- Docstring updated to spell out the "safety bound, not a user
preference" rule and to point at ``_PROTECTED_ENV_KEYS`` as
the place the 64 ceiling is enforced for profile env.
2. ``api/profiles.py`` — two frozensets:
- ``_PROTECTED_ENV_KEYS`` adds
``'HERMES_WEBUI_MAX_SESSION_RESOLVE'`` (next to
``'HERMES_WEBUI_ISOLATED_PROFILE'``).
- ``_BLOCKED_RUNTIME_ENV_KEYS`` adds the same key, with a
comment explaining the first-load-wins asymmetry.
3. ``tests/test_issue7421_resolve_cap_config.py``:
- The round-1 ``test_env_above_upper_bound_is_clamped_to_64``
is renamed to ``test_env_above_upper_bound_falls_back_to_default``
and the assertion + docstring reflect the new fallback
contract. The maintainer's review said explicitly:
"Update the clamp test accordingly."
- Three new tests pin Finding 2:
- ``test_protected_env_keys_includes_resolve_cap``
- ``test_blocked_runtime_env_keys_includes_resolve_cap``
- ``test_filter_runtime_env_strips_resolve_cap`` (end-to-end
exercise: a profile env containing the cap is filtered out
of the gateway-parity view, and an unrelated, non-blocked
key like ``LANG`` still passes through as a sanity check)
- ``tests/test_issue7421_resolve_cap_config.py``: 15/15 pass.
- ``-k "profile or protect or isolated or HERMES_WEBUI_ISOLATED"``:
866 passed, 5 skipped, 1 xfailed, 2 xpassed — no regression.
- The 5 ruff findings on the worktree are all pre-existing
(3× F401 ``sqlite3`` + 2× B904 ``raise-with-from`` in
``api/profiles.py``); the curated gate is gated on
newly-introduced findings only.
Two. Both are the explicit fixes the maintainer requested.
- ``_read_max_session_resolve_concurrent(999999)`` now returns
``2`` instead of ``64``. The 64 ceiling is still reachable
via an explicit in-range operator value.
- ``HERMES_WEBUI_MAX_SESSION_RESOLVE`` set in a profile's
``.env`` is now ignored: the operator/launcher env at
startup is the only place the cap is configurable.
Both changes are backward compatible at the operator level
(an explicit ``=8`` still returns ``8``) and break the
"typo → ceiling" failure mode and the
"first-loaded-profile-wins" asymmetry the maintainer flagged.
- An operator who set ``HERMES_WEBUI_MAX_SESSION_RESOLVE=64``
in their deployment env is unaffected (still returns 64,
as before).
- An operator who set ``HERMES_WEBUI_MAX_SESSION_RESOLVE=128``
in their deployment env now silently falls back to 2 instead
of getting 64. This is the round-1 reviewer's intent:
"Update the clamp test accordingly" + "the safe default,
never the permissive extreme." A startup log warning could
be added in a follow-up if a louder signal is wanted; this
PR keeps the silent-fallback contract consistent with the
other bad-input paths in the same helper.
- A profile that *had* ``HERMES_WEBUI_MAX_SESSION_RESOLVE=64``
in its ``.env`` and was relying on it to set the cap now
silently uses the operator-set value. This is the intended
behavior: the cap is a process-wide resource limit, not a
per-profile toggle. A profile-side log warning is a
follow-up if needed; the runtime env path now drops the
key, so the failure mode is "operator value wins" rather
than "first profile to load wins".
- #7656 — the original PR; this is round-3 of the same branch.
- #7421 — the original env-override feature.
- #4589 — the same shape ("operator-only, not per-profile")
for the isolated-profile key; the comment block on the new
protected key references the same pattern.
[rebase onto current master — same content as 5a7f7d8, no functional
changes beyond a single context-line merge from master]
Squash-rebase of #7656 onto current master so the 3-way merge CI
test no longer pulls in the unmerged stage/7618-7619-renderer-i18n
dirty diff that 5a7f7d8 was based on. The actual change set is
unchanged: 2 files, +231/-1.
What changed (from #7656 / 5a7f7d8):
- api/models.py::_read_max_session_resolve_concurrent — new
module-level helper that reads the cap from the env var
HERMES_WEBUI_MAX_SESSION_RESOLVE with strict validation
(positive int in [1, 64], malformed → default 2).
- tests/test_issue7421_resolve_cap_config.py — 12 regression
tests covering unset/empty/whitespace, headline cases (4/8/64),
safety paths (0/-1/999999/non-numeric), whitespace tolerance,
BoundedSemaphore capacity check, and a static wiring assertion.
Model: MiniMax-M3
A unique durable row remains one visible row when its sidecar provider payload is stale. Use the state.db payload for replay, cover repeated recovery, and document the image-turn ownership exception.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Use exact-key dedup by default and distinguish only rows with two valid different durable IDs.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
Match durable row identity as well as content before skipping a recovered image row, preserving separate same-text turns.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
Reuse the existing unique state-row identity guard before preserving a distinct image projection, so repeated reconciliation does not duplicate the same row.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Preserve the submitted image turn as the display owner while retaining rich Agent context for model replay and suppressing its durable scalar mirror from recovered WebUI transcripts.
Assisted-by: Hermes Agent:gpt-6-sol
Assisted-by: Codex:gpt-6-luna
Assisted-by: Claude Code:claude-opus-5-5
Re-review #7682 (nesquena-hermes 2026-09-22T19:18:00Z, CHANGES_REQUESTED).
`_transcript_already_advanced_past_pending` suppressed recovery on a
transcript-only heuristic: the first assistant row with final visible text
inside the pending turn's boundary. An interrupted multi-segment turn shaped
like
user -> assistant "Let me check the logs first." (unflagged interim prose)
-> tool -> tool ... (stream dies here)
passed the predicate as "finished" even though the interim sentence is not a
final answer and the stream was cut off mid-tool. `_apply_core_sync_or_error_marker`
then cleared pending state and returned early: the run journal was never
replayed and no interruption marker was added — a half-finished turn rendered
as complete, journaled tool output never appearing.
Fix: a genuine final answer inside the turn is now NECESSARY BUT NOT
SUFFICIENT. The suppression also requires durable same-stream terminal
evidence — `_run_journal_terminal_state(session, active_stream_id) == 'completed'`
(a `done`/`stream_end` event). Without it, fall through to the normal
journal-recovery path so the partial output is replayed and the interruption
is marked.
Regression tests (red/green verified — both fail with the gate removed):
- test_does_not_advance_when_unflagged_interim_prose_precedes_tools
- test_full_recovery_runs_journal_replay_for_unflagged_interim_prose
(drives `_apply_core_sync_or_error_marker`, asserts journal replay +
interruption marker present)
- test_does_advance_when_journal_marked_done_after_interim_prose (positive)
Existing `_FakeSession` fixtures updated to carry a session_id and write a
`done` journal event so the durable-evidence gate is satisfied for the
settled-turn tests.
Verification: test_issue6366 35 passed; anchor_scene + cancelled_turn
neighbours 91 passed. Red/green proven on both regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The env override never reached /api/sessions because the route kept a
separate hard-coded cap of 20. Resolve the window in api.config before
profile init, clamp it at 200, and refuse a profile .env override.
A user prompt could render at the very bottom of a conversation, below the
assistant turn that had already answered it.
merge_session_messages_append_only reconciles the sidecar with state.db. When a
state.db row carries a typed row id that is unique to state.db and absent from
the sidecar (the two stores do not share an id space),
row_id_preserves_source_multiplicity is true and the row bypasses the
sidecar-timestamp-range block -- including the
_insert_state_message_chronologically call that block owns. It then fell through
to the terminal merged_messages.append and landed at the tail regardless of its
timestamp.
The full display path re-sorts by timestamp afterwards
(_merged_webui_lineage_messages_for_display), which hid the defect. The
paginated path used for the initial render (msg_limit) does not, so the user saw
their own message below its own answer until a full reload reshuffled it.
Place such a row in its chronological slot when its timestamp predates the
sidecar tail, falling back to append when no safe slot exists. Rows that are
genuinely newest still append at the tail, so the append-only contract (never
drop a local row) is preserved.
Verified on the reported transcript: the merged tail now ends on the assistant
conclusion instead of the orphaned prompt, with the same message count.
The #6366 suppression was too broad and could destroy data in two ways,
both verified with full `_apply_core_sync_or_error_marker()` probes:
1. A repeated prompt could be silently discarded. Checkpoint matching
truncates timestamps to integer seconds and ignored
`_active_turn_token`, so a prior identical prompt from a *different*
stream could match the new pending turn — and the suppression branch
then cleared the only durable copy of the prompt, with no recovered
row and no error marker.
Identity is now bound to the pending stream's exact active-turn token
(`_transcript_user_row_is_pending_turn`). Integer-second timestamp
equality never overrides a token mismatch, and a row that carries no
token cannot be proven to be this turn. Legacy sessions with no token
on either side fall back to the strict content + timestamp + source +
attachments checkpoint.
2. Interrupted tool-running turns were treated as completed. Any
unflagged assistant row counted as settled, including empty rows,
tool-call-only rows, interim-progress rows and compaction rows, so
`:3600` bypassed journal replay and the interruption marker.
Completion now reuses the established final-answer semantics
(`_pending_turn_has_final_assistant_answer` ->
`_assistant_message_has_final_visible_text`) inside the matched
user-turn boundary, rejecting tool-only, empty, interim, journal-
recovered and compaction tails.
The suppression is deliberately asymmetric: a false negative leaves a
visible cosmetic duplicate the user can ignore, while a false positive
silently destroys a prompt or a response. Whenever identity or
completion cannot be positively proven, recovery runs instead of
suppression.
Tests: the 11 predicate unit cases are re-derived against the new
identity/completion helpers, plus full-recovery-path cases that exercise
`_apply_core_sync_or_error_marker()` itself — distinct same-second
repeated turns (token-bearing and legacy) proving the prompt survives,
and empty / tool-call-only / interim / compaction tails proving journal
replay and the interruption marker still run — plus a positive control
for the suppression branch itself.
Stale-cancel recovery in _apply_core_sync_or_error_marker
(line ~3497+) only inspects the transcript tail
(`session.messages[-1]`) when checking whether the durable
transcript already contains the pending turn. When the pending
turn's user row is older than the tail, the tail is a *newer*
settled assistant row that can never match the pending **user**
checkpoint, and the recovery path falls through into the append
branch. The cancel/recovery tail then appends, after the valid
final answer, a recovered user row, a `_partial` clone of the
old journal, and a generic no-response error that survives reload
in the durable transcript.
The fix adds `_transcript_already_advanced_past_pending()`
which scans the messages for any strictly-matching user row
(content + timestamp + source + attachments) and, when one is
found, looks for a later settled assistant row (no
`_partial` / `_error` / `_recovered`). When that is the
case, the helper returns True and the non-completed-journal
branch clears only the stale pending fields and returns
without appending any rows.
Strict match is required so that a *new* user turn that happens
to repeat the same prompt text (e.g. user re-sends the same
question) is not misclassified as a stale advance. The
regression guard is the existing
`test_repeated_pending_prompt_uses_current_checkpoint_identity`
which now passes alongside the new helper unit cases.
The helper is a pure read; running recovery twice therefore
produces the same outcome, satisfying the maintainer's
idempotence requirement.
Validation: 11/11 new tests pass (covers the strict-match
distinction, the partial/error/recovered skip, and idempotence);
124/124 related stale-pending / sidecar-repair / recovery tests
pass; py_compile clean.
Rodrigo Gomes da Silva