Merge branch 'master' into feat/gateway-run-reattach-after-restart
This commit is contained in:
86 files changed
+14695
-480
No files matched your search
@@ -59,6 +59,41 @@ proposes. It is routed in `api/routes.py` and implemented by
|
||||
is a global invalidation signal, not a per-session lifecycle stream. The proposed
|
||||
`GET /api/sessions/{session_id}/events` is per-session and path-distinct.
|
||||
|
||||
### Hidden-tab observation and recovery (implemented client behavior)
|
||||
|
||||
The browser closes its persistent per-session SSE while hidden and uses
|
||||
`_startHiddenActiveStreamPoll()` in `static/messages.js` to poll
|
||||
`GET /api/session/status?session_id=...` immediately and then every six seconds,
|
||||
subject to browser timer throttling. An active stream can be attached through
|
||||
the existing replay path; successful attachment stops the poll.
|
||||
|
||||
HTTP `404` is ambiguous: older profile-visibility guards and the legacy
|
||||
unknown-profile path can return it for a live session. Current master returns
|
||||
`409 session_profile_mismatch` for a known foreign profile; that response
|
||||
remains retryable when another tab changes the browser-wide profile cookie.
|
||||
A single `404` therefore keeps polling and retains the hidden-resume owner.
|
||||
After three consecutive `404` responses, the poll pauses to bound repeated
|
||||
missing-session requests, but retains that owner so returning to the visible
|
||||
tab can reopen SSE. Any other response or network error resets this budget;
|
||||
a newly started poll also starts with a fresh budget.
|
||||
|
||||
HTTP `410` is terminal: it stops the interval and clears the matching
|
||||
hidden-resume session ID. Returning to the visible tab then does not reopen
|
||||
SSE through that owner. Responses and queued ticks belong to one poll timer,
|
||||
not merely a session ID: they cannot stop or attach a replacement poll, even
|
||||
when the replacement observes the same session. The budget counts responses
|
||||
received by the current poll. Cleanup affects browser observation state only;
|
||||
it does not delete a session or cancel an agent run.
|
||||
|
||||
Successful idle responses with no `active_stream_id`, network failures, and
|
||||
other non-success HTTP responses (including `401`, `403`, `429`, and `5xx`)
|
||||
remain retryable. Visibility return normally restores per-session SSE when a
|
||||
resume owner still exists, including after repeated `404` responses. Only a
|
||||
terminal `410` removes that automatic recovery path. A profile restored before
|
||||
the three-miss limit can recover through the next hidden poll; after the limit,
|
||||
recovery waits for visibility return or explicit session loading. Behavior coverage lives in
|
||||
`tests/test_hidden_tab_server_initiated_turn.py`.
|
||||
|
||||
### Heartbeat
|
||||
|
||||
`_SSE_HEARTBEAT_INTERVAL_SECONDS = 5` (defined in `api/routes.py`) is the current
|
||||
@@ -162,7 +197,7 @@ gate rather than inventing values without source support.
|
||||
## Authoritative emitted events (`/api/chat/stream`)
|
||||
|
||||
These are the **real wire `event:` names** emitted by `api/streaming.py` today
|
||||
(23 names). Clients and docs must use this table — not the semantic draft above —
|
||||
(24 names). Clients and docs must use this table — not the semantic draft above —
|
||||
when integrating with the live chat SSE relay.
|
||||
|
||||
| Wire name | Role |
|
||||
@@ -179,6 +214,7 @@ when integrating with the live chat SSE relay.
|
||||
| `title` | Session title update (often after `done`) |
|
||||
| `title_status` | Title generation status / skip reason |
|
||||
| `warning` | Non-fatal provider/fallback warning |
|
||||
| `runtime_model` | Local Agent serving identity observed at output or successful completion |
|
||||
| `apperror` | Terminal application error (no trailing `stream_end`) |
|
||||
| `cancel` | Run cancelled |
|
||||
| `done` | Turn finalized (session payload); title/`stream_end` may follow |
|
||||
@@ -199,6 +235,31 @@ The semantic taxonomy table remains a draft for the proposed per-session
|
||||
endpoint vocabulary and must be confirmed during maintainer review before that
|
||||
endpoint claims parity.
|
||||
|
||||
### Observed local runtime model
|
||||
|
||||
The local worker emits `runtime_model` with
|
||||
`{session_id, stream_id, model, provider?, fallback_active, phase}`. The IDs
|
||||
refer to the original run-journal owner even if compression rotates the Agent's
|
||||
session. `phase="observed_output"` means the Agent's own model was read at a
|
||||
nonempty token/reasoning callback or after successful completion (including a
|
||||
non-streaming reply or credential self-heal). It does not promise that a turn
|
||||
which emitted partial output will finish successfully. Missing Agent model
|
||||
sends no observation; the configured selection is not used as serving proof.
|
||||
`fallback_active` is true only when the Agent explicitly reports its fallback
|
||||
flag. Consecutive identical observations are deduplicated within the turn.
|
||||
|
||||
Fallback lifecycle warnings invalidate older serving evidence and reset the
|
||||
deduplication; status text alone never establishes the replacement model. A
|
||||
fresh observation after the warning reestablishes it, including when a buffered
|
||||
success notice arrives after output. Durable replay retains the event IDs.
|
||||
The journal summary and HTTP `runtime_journal_snapshot.runtime_model` project
|
||||
the latest valid observation from the same session/stream event window, or null
|
||||
if a warning or malformed latest observation invalidated it. No previous run's
|
||||
footer or selected route fills an unknown current run. This is a local-worker
|
||||
producer; gateway attribution and frontend presentation are separate concerns
|
||||
(see #6272 and #7181). `route_observed` remains reserved for route-only data,
|
||||
not successful output.
|
||||
|
||||
## Cursor and resume semantics
|
||||
|
||||
`Last-Event-ID` is the standard SSE reconnect header. Clients send the last
|
||||
|
||||
@@ -215,6 +215,14 @@ settle.
|
||||
Steer must not exist only as a toast or transient DOM state. It must survive
|
||||
refresh, session switch, replay, and settled render.
|
||||
|
||||
**The transport wrapper is not the Steer row.** A Steer is emitted as a standalone
|
||||
typed user row (`role: user`, `display_kind: "steer"`). The surrounding
|
||||
`[OUT-OF-BAND USER MESSAGE …] … [/OUT-OF-BAND USER MESSAGE]` block is transport-level
|
||||
control data: settlement unwraps the single validated frame in place, extracting
|
||||
and preserving the user's authored steer instruction while dropping the wrapper.
|
||||
Legacy tool rows and rows where markers are multiple, nested, incomplete, or
|
||||
contain ambiguous delimiters are preserved byte-for-byte.
|
||||
|
||||
### Relationship to system control events
|
||||
|
||||
Steer is similar to system-delivered control events such as tool-iteration-limit
|
||||
|
||||
@@ -171,7 +171,41 @@ and 5; it does not mark every run-state boundary implemented.
|
||||
do not conflict, and the pairing is unambiguous. Keep the rich sidecar row;
|
||||
if any requirement is missing or contradictory, preserve both rows rather
|
||||
than deduplicating. Literal scalar `[screenshot]` text alone is not identity
|
||||
evidence.
|
||||
evidence. A WebUI-submitted native-image turn has a separate display owner:
|
||||
keep its exact submitted text and attachment in the visible session row,
|
||||
while the Agent's expanded multipart row remains in `context_messages` for
|
||||
model replay. While the turn is active, the WebUI may hide an Agent user row
|
||||
from display only after its worker confirms that the active stream's exact
|
||||
`pending_started_at` value was passed to the Agent as
|
||||
`persist_user_timestamp`; persist that private proof with the session and
|
||||
validate it against the pending stream, source, and timestamp after reload.
|
||||
Never include the proof in public session payloads. This applies to
|
||||
native-image and scalar text-attachment rows, and never changes model
|
||||
context. If multiple user rows share that timestamp, omit the whole
|
||||
ambiguous display bucket until the turn settles; keep all rows in model
|
||||
context. Do not identify the row by its text. If a stream dies before
|
||||
settlement, state.db self-heal must save the submitted prompt and attachments
|
||||
as a visible sidecar row before clearing pending metadata only when there is
|
||||
genuine state.db output beyond that submitted turn. Otherwise, leave pending
|
||||
state intact for journaled partial-output and interruption-marker recovery.
|
||||
The Agent row remains available in model context. A partial continuation
|
||||
must use one consistent parent snapshot when projecting a conflicting
|
||||
provider payload onto its sidecar-owned display row.
|
||||
Match settled native-image scalar projections only with trusted turn and
|
||||
durable-row identity, never the marker alone. A durable row ID proves row
|
||||
identity, not provider-payload freshness: when the sidecar and state.db have
|
||||
conflicting nonempty `api_content`, preserve both versions for model-context
|
||||
replay without mutating either. For visible display, a marked mirror may
|
||||
share the existing sidecar bubble only when its valid durable row ID, exact
|
||||
timestamp, and exact visible user content match; keep the sidecar-owned row
|
||||
and its display metadata. Distinct row IDs, ambiguous or invalid identities,
|
||||
and different visible user text remain separate only while eligible under
|
||||
the existing edit/undo truncation watermark and checkpoint-order rules;
|
||||
removed rows must not reappear in display or model replay. Fill a missing
|
||||
payload from the other copy; repeated reconciliation must remain bounded
|
||||
and idempotent.
|
||||
Agent state.db alone cannot restore the original attachment if the WebUI
|
||||
sidecar is lost.
|
||||
Visible interim assistant progress must remain visible timeline content; a
|
||||
compact Activity disclosure may summarize adjacent tool/debug detail, but it
|
||||
must not be the only place where the user can see emitted progress text.
|
||||
|
||||
@@ -309,6 +309,27 @@ Interpret the two together:
|
||||
|
||||
---
|
||||
|
||||
## MCP panel shows another profile's servers, or "Live status for this profile is unavailable"
|
||||
|
||||
**Symptom.** With several profiles, the MCP settings panel of profile A shows a server as *Active* with a tool count while the tool inventory is empty (or lists profile B's tools); `/reload-mcp` on one profile stops the other profile's servers; or the MCP panel and the external Notes drawer show the notice *"Live status for this profile is unavailable right now"* and `/reload-mcp` answers *"MCP runtime scope could not be confirmed"*.
|
||||
|
||||
**Why.** Hermes Agent keeps one in-process MCP ledger per WebUI process and keys a connection by profile only when it can tell the request serves a profile other than the process's own. The WebUI binds every MCP status read and `/reload-mcp` to the request profile (`ARCHITECTURE.md` §4.10). While a chat turn is streaming, the WebUI mirrors that turn's profile into `HERMES_HOME`; Agents that predate `hermes_constants.pin_process_hermes_home` cannot distinguish that mirror from the process profile, so the WebUI withholds runtime data and refuses the reload instead of showing or resetting another profile's connection.
|
||||
|
||||
**Diagnostic commands.**
|
||||
|
||||
```bash
|
||||
# runtime_scope: "profile" (bound), "legacy_process" (Agent without profile-scoped MCP),
|
||||
# "unavailable" (scope could not be confirmed right now)
|
||||
curl -s -b "hermes_profile=<profile>" http://127.0.0.1:8787/api/mcp/servers | python3 -m json.tool | grep -E '"(name|status|tool_count|runtime_scope)"'
|
||||
python3 -c "import hermes_constants; print(hasattr(hermes_constants, 'pin_process_hermes_home'))"
|
||||
```
|
||||
|
||||
**Fix.** `unavailable` while a turn is running is expected: refresh once the turn finishes. If it persists with no turn running, the Agent predates the process-home pin; upgrade Hermes Agent. `legacy_process` means the Agent has no profile-scoped MCP ledger at all; its `/reload-mcp` stays process-wide by design. After changing a profile's `mcp_servers`, run `/reload-mcp` **from that profile**: it only resets that profile's own connections and retries its failed servers.
|
||||
|
||||
**When to file a bug.** File a WebUI bug if `runtime_scope` is `"profile"` and a server is still reported *Active* with tools you cannot see in the inventory, or if `/reload-mcp` from one profile changes another profile's `tool_count`.
|
||||
|
||||
---
|
||||
|
||||
## Other troubleshooting
|
||||
|
||||
This document grows over time. If a recurring failure mode isn't covered here yet, add it via PR. The format for each entry: **Symptom → Why → Diagnostic commands → Fix → When to file a bug**.
|
||||
|
||||
Reference in new issue
Block a user