A user prompt could render at the very bottom of a conversation, below the
assistant turn that had already answered it.
merge_session_messages_append_only reconciles the sidecar with state.db. When a
state.db row carries a typed row id that is unique to state.db and absent from
the sidecar (the two stores do not share an id space),
row_id_preserves_source_multiplicity is true and the row bypasses the
sidecar-timestamp-range block -- including the
_insert_state_message_chronologically call that block owns. It then fell through
to the terminal merged_messages.append and landed at the tail regardless of its
timestamp.
The full display path re-sorts by timestamp afterwards
(_merged_webui_lineage_messages_for_display), which hid the defect. The
paginated path used for the initial render (msg_limit) does not, so the user saw
their own message below its own answer until a full reload reshuffled it.
Place such a row in its chronological slot when its timestamp predates the
sidecar tail, falling back to append when no safe slot exists. Rows that are
genuinely newest still append at the tail, so the append-only contract (never
drop a local row) is preserved.
Verified on the reported transcript: the merged tail now ends on the assistant
conclusion instead of the orphaned prompt, with the same message count.
The cache-only Steer path only special-cased backend == "gateway" and sent
every other value — including a missing, empty, or unknown backend — to the
cached local agent. Another runtime could own the run while guidance was
enqueued on a stale local agent.
Fail closed instead: the in-process worker now registers its active run with
the explicit local backend tag (WEBUI_LOCAL_CHAT_BACKEND, the same "legacy"
value webui_chat_backend_mode() returns), and cache-only Steer only enqueues
agent.steer() on that tag. Gateway keeps its gateway_steer_queued outcome; a
missing, empty, or foreign backend returns stream_dead without steering.
Extends the positive-ownership regression with backend cases and adds a
worker-side check that the local backend tag is actually registered.
The cache-only Steer branch (no AGENT_INSTANCES worker) only rejected
*conflicting* ownership, so a missing stream owner or a missing
active-run session_id was treated as non-conflicting and let
agent.steer() run and report accepted without proving the requesting
session owns the run.
Fail closed instead: both the stream owner and the active-run session
must be present and strictly equal to the requesting session (backend
and phase are still revalidated) under the same STREAMS_LOCK ->
ACTIVE_RUNS_LOCK edge; any absent identity returns stream_dead without
calling agent.steer(). No cache eviction, no agent closing, HTTP write
still outside the locks.
Regressions: missing owner, missing/empty run session, both missing,
missing run entry -> stream_dead + steer never called; both present and
equal -> accepted, cache untouched. Legacy happy-path fixtures now
register the ownership they rely on.
The cache-only compatibility path selected the matching cached agent,
checked stream liveness under STREAMS_LOCK, released the lock, and only
then called agent.steer(). A Stop running in that gap published
phase='cancelling' and detached STREAMS/AGENT_INSTANCES while the
earlier Steer still enqueued into the cached worker, so the HTTP
response reported accepted guidance that cancellation already owned.
Revalidate stream membership, owner, and active-run
session/backend/phase and call agent.steer() under the same
STREAMS_LOCK -> ACTIVE_RUNS_LOCK edge Stop uses; interrupt,
persistence, and HTTP writes stay outside the lock. Cache eviction and
agent closing remain absent.
Extend tests/test_steer_cancel_ordering.py with a cache-only reverse-
order test (AGENT_INSTANCES absent, matching cached worker, decoy cache
entry) gating both unlocked edges around the stream-lock reacquire.
Stop-first must return stream_dead without calling agent.steer(); the
selected cache object is asserted to be the intended worker. Proven
red on the pre-fix code (order ['cancel','steer']) and green after.
The settled current-summary card was only created when no compaction
marker was loaded at all. A transcript with older (stale) markers plus a
newer `compression_anchor_summary` that no loaded marker references
therefore lost the current summary, which master still rendered.
Decide the fallback from `_latestCompressionReferenceMessage()`
(`referenceMessageRawIdx < 0`) instead of marker presence, and feed that
decision into `_selectCompactionCardPlacements()` before any card node is
built so the fallback is the single preserved-task owner and marker cards
never attach the task list a second time.
Add a render-level regression that executes the real `renderMessages()`
over 220 messages, three stale markers, one preserved task-list message
and a newer summary: the current summary is visible and exactly one task
owner is rendered, in full, virtualized and server-truncated windows.
The renderMessages node harness added in origin/master (#7125 lineage)
evals renderMessages in isolation; the new loaded-compaction gate calls
_loadedCompactionMarkerRawIdxs, which must be stubbed like the other
render helpers (_postProcessWithAnchorSuppression, _usedModelTurnChipLabel).
Replace the remaining source-text assertions in test_issue2628 (the Greptile
P2 thread): the pre-aggregation shape, index detection, correlated seek and
pure-read contracts are already pinned behaviourally by EXPLAIN QUERY PLAN,
result equivalence and PRAGMA index_list in test_pr7445_readonly_review_followups.py,
test_state_db_readonly_readers.py and test_issue3887_index_prime.py.
The one contract only source text covered — the SQL candidate window is
capped and oversampled (limit * 8) so post-projection filtering cannot
starve the visible page — is now asserted behaviourally instead:
* the traced statement SQLite actually runs carries a single LIMIT whose
bound is the oversampled cap (limit * 8 = 160), applied inside the
candidate subquery the outer projection joins;
* a page still fills to limit when the 30 newest candidates are invisible
CLI rows (ended, default title): they consume candidates, not result
slots. Verified to fail (0 rows) against the old non-oversampled shape.
On native Windows the text-mode ``handle.write("\n")`` that seeds an empty
``--lock-file`` is translated to ``\r\n``, so a fresh lock file is two bytes
and the msvcrt regression test fails on the platform it protects. Seed the
file with ``os.write(fd, b"\0")`` on the descriptor instead: exactly one byte
whatever the text-mode newline translation. The Windows-branch tests now run
under a ``\n`` -> ``\r\n`` translating ``open`` so this failure reproduces
on any host.
Remove the two ``... not in src`` assertions added for the previous review
from test_issue2628: the behaviours they hinted at (every missing-index
projection uses the pre-aggregated ordering; listing never creates an
index) are already asserted through query plans, result equivalence and
``PRAGMA index_list`` in test_pr7445_readonly_review_followups.py,
test_state_db_readonly_readers.py and test_issue3887_index_prime.py.
On the msvcrt branch of scripts/ensure_state_db_read_indexes.py the tool
locked byte 0 of --lock-file right after open(..., "a+"). When that open
had just created the file (or the file was empty) the lock range lay beyond
EOF, which Windows refuses, so the documented locked-maintenance workflow
raised instead of acquiring the lock (greptile P1).
Write one byte when the lock file is empty, then seek(0) and lock as
before. Only the Windows branch changes; POSIX flock is untouched and an
existing (deployment-owned) lock file is never rewritten.
Tests (tests/test_pr7445_readonly_review_followups.py): a strict msvcrt
stand-in that refuses to lock past EOF proves the lock is taken on byte 0
of a fresh empty file, that an existing lock file keeps its content, and
that a held lock still propagates before the database is touched.
Review follow-ups on #7445 (nesquena-hermes CHANGES_REQUESTED + Greptile).
api/agent_sessions.py
- The read-only pre-aggregated `latest_messages` candidate ordering now
applies to every missing-index projection (default sidebar, gateway
watcher, cron/webhook/kanban views), not only `include_sources ==
("cron",)`. Since the listing no longer self-heals the index, the
correlated `MAX(mx.timestamp)` scan re-ran on every load: 9.86 s on a
5,000-session / 20,000-message DB vs 0.044 s after this change, with
identical selected rows and order. The indexed seek is unchanged.
- `state_db_file_uri()` / `state_db_readonly_uri()` replace
`Path.as_uri()`: a UNC `HERMES_HOME` produced `file://server/share/...`
and CPython's bundled SQLite (3.11-3.13 Windows builds, no
SQLITE_ALLOW_URI_AUTHORITY) rejected the authority, so every agent-backed
session vanished. The builder emits the empty-authority
`file:////server/share/state.db?mode=ro`, keeps `file:///C:/...` for
drive letters, strips `\\?\` / `\\?\UNC\` prefixes, quotes `?`/`#`, and
leaves POSIX output byte-identical. Strict `mode=ro`, no writable
fallback.
scripts/ensure_state_db_read_indexes.py
- `fcntl` / `msvcrt` are imported conditionally; the module imports and
the no-`--lock-file` path runs on native Windows. `--lock-file` uses
`flock` on POSIX, `msvcrt.locking(LK_NBLCK)` on Windows, and fails with
a clear RuntimeError when neither exists. The rw open reuses the
UNC-aware URI builder.
docs/troubleshooting.md
- Operator section: read-only contract, degraded missing-index listing,
drained maintenance prerequisites, lock behaviour, failure modes, UNC.
Tests: tests/test_pr7445_readonly_review_followups.py (19 tests, red
before this change: correlated-subquery plan, missing URI builder,
ImportError on fcntl). Query-plan contract via EXPLAIN QUERY PLAN — no
CORRELATED SCALAR SUBQUERY on missing-index listings, identical rows to
the indexed listing; platform-parametrised URI cases; maintenance module
imported with `sys.modules["fcntl"] = None`. test_issue2628 no longer
pins the cron-only gate; the flock-held test skips without fcntl.
Readers never upgrade to writers. open_state_db_readonly() opens
file:...?mode=ro only; a read-only open failure propagates instead of
silently retrying with a write-capable handle on the multi-GiB WAL
database the agent is streaming into. The session listing no longer
self-heals idx_messages_session with CREATE INDEX (a writer lock held for
minutes on a large messages table) — a missing index degrades to the
pre-aggregated path. Cron sidebar info, insights and deep health checks in
api/routes.py use the same read-only opener.
Watcher (api/gateway_watcher.py): _poll_once returns without touching the
database when nobody is subscribed, and forces a fresh full projection on
the next subscription.
Maintenance (scripts/ensure_state_db_read_indexes.py): the covering read
indexes (idx_messages_session, idx_messages_session_role with NOCASE,
idx_sessions_webui_fingerprint) are created only under --confirm-drained,
optionally holding an exclusive flock on --lock-file for deployments that
serialise agent turns on a lock file; existing indexes are verified for
shape/collation and coverage (EXPLAIN QUERY PLAN), never silently
replaced.
Tests: tests/test_state_db_readonly_readers.py (read-only failure never
retries writable; no poll without subscribers; listing creates no index;
index tool requires drain, is idempotent, works without a lock file and
refuses a held lock; cron sidebar read is mode=ro). Neighbouring tests
that asserted the retired writable fallback / self-heal writer
(test_issue3887_index_prime, test_issue5455_listing_readonly_connection,
test_issue5455_lineage_readonly_reads, test_issue2628_cli_sessions_perf,
test_issue1494_state_db_fd_leak) now assert the read-only contract;
test_issue3506 subscribes before polling.
Gate found two blocking defects in the new caches:
1. The large-string LRU was keyed on input text only. The installed
agent.redact registry can extend the secret matcher at runtime via
register_redaction_patterns() and exposes no policy generation or
invalidation hook, so a >16k string primed before a registration was
served with the new secret intact (reproduced against the installed
registry). Drop the large-string cache: strings above 16,384 chars run
the redactor uncached again. Regression primes a large string, registers
a runtime pattern on the real registry and proves the next
redact_session_data() response is redacted.
2. The prefilter LRU was count-bounded (8192) but its per-entry gate was a
character cap, which is not a byte bound: 16k four-byte code points is
~64 KiB per key, ~512 MiB at capacity, and the workload died with
MemoryError under 96 MiB of headroom. Gate each entry on
sys.getsizeof(text) <= 2048 so the retained-key ceiling is a hard
16 MiB. Regressions cover max-width Unicode, sensitive verdict pairs,
eviction past the cap, and a low-memory RLIMIT_AS subprocess.
The case labelled as an echo further back than the window still ended the
buffer with the suffix, so it matched regardless of the window. Replace it
with a buffer whose whitespace-padded echo spans more than the default
window, and add a dedicated test proving the same echo is rejected with the
default window and found once the window covers it, for both the optimised
scan and the reference oracle.
`_strip_compact_echo_suffix` probed every candidate cut index and re-folded
the whole remaining tail for each probe, so the cost grew with
`window x suffix` rather than with the data actually inspected. It runs up to
four times per interim assistant message, on the reasoning buffer, the shared
stream mirror and up to two per-message segments.
The practical effect is worst on the longest message of a turn: 297ms for a
200-character message but 5.6s for a 6000-character one, measured on the live
process. That time is spent holding the GIL, so a single long final message
also stalls every other stream served by the same worker. Profiling a live
turn attributed ~5% of total CPU to this one helper.
Fold the window once, then walk backwards across exactly as many non-space
characters as the folded suffix holds. `str.isspace` and the `\s` pattern used
by `_compact_for_echo_compare` agree on all 1114112 Unicode code points, so
the folded view and the walk stay consistent.
Behaviour is unchanged, including the leftmost-cut tie-break on repeated text
and the bounded search window. The new test pins equivalence against a naive
reference oracle over ~2020 cases (exact, partial and absent echoes, mixed
whitespace, unicode, empty and None inputs, repetitions, out-of-window
echoes), asserts the fold no longer runs once per cut index, and caps the
runtime of a long-conclusion call.
Measured: 200 chars 297ms -> 0.149ms, 6000 chars 5645ms -> 0.720ms.
After rewind/cache-miss the keyed visible row stayed distinct from the
persistent parser candidate, so every later growth frame deep-cloned the
parser DOM into the old row. That cut off the previous tail word's
~620ms fade and broke word-node identity.
Promote the parser-owned candidate into the existing row's DOM position
once, mute the already-rendered prefix on the real parser body, and
return the adopted node so later keyed renders hit existing === node.
Regression: two post-transition growth frames keep the first new tail
span as the same .is-new node until an explicit animationend.
Cursorless markdown rebuilds cloned the live parser tree, then later
source growth appended raw bytes through _appendTransparentFadeText.
Incomplete emphasis (Hello ** / Hello **world**) became literal
**world** while the candidate already had <strong>world</strong>.
Parser-owned candidates now keep reconciling from the parsed DOM.
Pending and MEDIA tails stay on the parser. Rendered-prefix mute is
unchanged.
Greptile P1 follow-up on the fade-row rebuild: when a cursorless live
markdown row adopts the incremental node's cloned DOM, that DOM lags the
source text by the streaming parser's held-back tail (pending/text
buffers). Keeping the full source-length resume cursor made the next
reconciliation append nothing, so the pending characters stayed missing
from the live row until settlement.
Read the held-back length from the parser bound on the candidate body
(_smdBindParserIdentity) and trim the source-space cursor by exactly
that length, so the cursor matches the adopted content and the next
reconciliation re-appends the pending tail.
Regression: rebuild while the parser holds an unrendered tail, then
reconcile again — the pending text must appear in the live row before
settlement.
Review follow-ups for #7082:
- Maintainer should-fix: the no-cursor rebuild branch of
_refreshTransparentFadeProseRow cleared the body and re-wrapped every
word as .is-new, so the whole visible row dipped to opacity 0 and
faded back (~620ms), and wholesale node replacement invited the
one-time scroll-anchor bounce (#6257). Now the rendered text is
snapshotted before the clear and the messages.js
_streamFadeMuteRenderedPrefix idiom is re-applied after the rebuild,
so only genuinely-new tail words animate. The helper is exported on
window (__streamFadeMuteRenderedPrefix) the same way
__anchorProseIncrementalNode is.
- Greptile P1: the same branch passed raw source text to
_appendTransparentFadeText, flattening live markdown rows to literal
syntax (links/emphasis/headings lost until settlement). The rebuild
now adopts a deep clone of the candidate's parsed .msg-body (the
incremental streaming-markdown output) when one exists, so the parsed
DOM survives; the plain-text rebuild remains only as fallback for
candidates without a parsed body. The resume cursor stays in source
space when rendered text diverges from source (markdown), and shrinks
to the rendered prefix for plain prose so the parser's held-back
character is re-appended on the next delta instead of dropped.
Regression tests: rebuild must not re-animate the already-rendered
prefix (tail-only .is-new), and rebuild must keep parsed <a>/<strong>
DOM with no literal markdown in the visible text. The two existing
tests of this PR are unchanged and still pass.
Transparent-stream progress rows are built incrementally by
`_anchorProseIncrementalNode`, which streams source text into the vendored
streaming-markdown parser. That parser always holds the most recently written
character in its pending buffer, so a live row's rendered `.msg-body` text
trails its source text by one character.
`_refreshTransparentFadeProseRow` resumed appending from the
`data-stream-fade-text` cursor but fell back to `body.textContent` when the
attribute was absent — exactly the case for incrementally built rows, which
never set it. That fallback mixed two coordinate spaces (rendered text vs
source text), so the computed delta started one character early, in the middle
of a word. The delta was then appended as a sibling of the parser's open `<p>`,
and since `<p>` is a block box the tail rendered on its own line: a single word
split across two lines ("Ces deu" / "x fichiers").
Two independent fixes:
1. Resume strictly from the source-space cursor. When no cursor exists we
cannot know how much source is already rendered, so rebuild the body
instead of guessing a delta from rendered text.
2. Append into the trailing block element rather than the `.msg-body` root, so
a continuation of the current sentence stays on the same line even if a
cursor is ever stale.
Regression tests drive the real vendored parser to build a live row exactly as
production does, assert the parser really does lag its source, and fail if any
word is torn across a block boundary. The existing reconcile harness gains the
new helper dependency.
Opt live assistant streaming nodes out of the global theme color/background transitions so token-by-token markdown updates do not flash or fade on light themes.
Adds a targeted regression test in test_smooth_text_fade.py while keeping the opt-in smooth text fade feature intact.
Anthony Ruiz