Add an update_channel setting (stable|experimental) that selects which git-tag
stream the self-updater tracks on the single linear master line:
stable -> 'v*' promoted, soaked releases (default; unchanged glob)
experimental -> 'exp-v*' every release batch (opt-in testers)
A channel is only a tag glob — no branches, no divergence — so every ff-only
guarantee (#2653/#2846/#3140) is preserved. Channel governs the WebUI repo only;
the Agent repo keeps its historical branch fall-through.
Correctness (advisor-reviewed, Codex + Fable):
- describe uses --match <glob> so a commit tagged both v* and exp-v* resolves
to the channel-correct tag
- stable never falls through to origin/master when HEAD contains the latest
stable tag (the firehose-suppression that makes channels work)
- apply_force_update refuses to reset --hard onto an ancestor ref (rewind guard)
- update cache + in-progress guard keyed by (channel, include_agent)
- channel display badge is a SEPARATE field; WEBUI_VERSION stays channel-neutral
(asset cache-busting / SW cache / stale-client skew all do exact-string equality)
Settings UX: channel dropdown with risk-forward copy + the ff-only asymmetry
helper ('switching back to Stable keeps your current version until Stable catches
up'), channel chip on the version badge, i18n keys, re-check on switch.
Tests: new tests/test_update_channels.py (12 real-git-fixture tests) + existing
update suite updated for the channel-aware signatures.
Address round-2 gate cert (issuecomment-4896321817) on sha:dcec232d:
1. BRICK (Codex strace-verified): v2's fcntl-flock holder check on
.git/index.lock was wrong. Codex straced git 2.43.0 and proved
that git uses O_CREAT|O_EXCL + rename(2), NOT advisory locking,
so an fcntl-flock probe returns False for a lock file a live git
process is holding open. The 'fail-closed' claim was wrong:
auto-delete could still race and corrupt the index.
2. CORE (Codex): if .git/index.lock vanished between _is_lock_held
and os.remove, the FileNotFoundError surfaced as a 'remove-failed'
diagnostic instead of 'already gone' and prevented the apply retry.
Fix: stop deleting lock files from the server entirely. The only thing
that removes a lock is the user, on the host, via the manual command
surfaced in the response. Once the lock is gone, the user re-clicks
Update Now and the normal non-destructive apply path runs.
Implementation:
api/updates.py:
- Removed _is_lock_held (fcntl-flock probe) -- was provably unsafe.
- Removed _try_remove_lock (os.remove path) -- was provably unsafe.
- Removed _GIT_LOCK_FILES_REMOVABLE (no longer enumerable by name).
- Added _inventory_locks(path) -> dict: pure inventory of well-known
+ other lock files under .git/. Never touches anything.
- Rewrote apply_clear_lock to inventory-only + manual-instruction.
When .git/index.lock is absent, runs the normal non-destructive
apply path; when present, returns ok=False with the exact 'rm -f'
command + an explanation of why the server cannot do this safely
(O_CREAT|O_EXCL cannot be detected with advisory probes).
CORE-1 race evaporates: there is no os.remove to race against.
api/routes.py:
- Updated the /api/updates/clear_lock comment to reflect v2.2.
static/ui.js:
- applyClearUpdateLock: when res.lock_held, call new
_renderLockManualInstruction(target, res) which surfaces the exact
'rm -f <path>' command in a copyable code block plus an
'I've removed the lock -- retry update' button that POSTs the
same endpoint again; the second call takes the success branch
and re-runs the normal apply path. Also lists any other lock
files present so the operator can investigate.
tests/test_updates.py:
- Removed all 4 tests for the deleted v2 helpers (_is_lock_held
Returns*, _try_remove_lock*).
- Added test_v2_probe_helpers_removed: regression guard that
fails loud if either helper is ever reintroduced.
- Rewrote test_apply_clear_lock_removes_unheld_lock_and_runs_normal_update
and test_apply_clear_lock_refuses_when_lock_held for the v2.2
contract. The 'refuses' test monkeypatches os.remove to record
any deletion attempt and asserts the call list is empty -- a
strict runtime guard against future regressions to auto-delete.
- Added 7 new tests: test_inventory_locks_* (4), test_apply_clear
lock_with_no_lock_runs_normal_update, test_apply_clear_lock_with
lock_present_returns_manual_instruction, test_apply_clear_lock
listing_includes_other_locks, test_apply_clear_lock_rejects
unknown_target, test_apply_clear_lock_rejects_not_git_repo.
Verification:
./scripts/test.sh tests/test_updates.py -> 73 passed
./scripts/test.sh tests/test_update_banner_fixes.py -> 102 passed
./scripts/test.sh tests/test_api_timeout.py -> 8 passed
python3 scripts/ruff_lint.py --diff upstream/master -> 0 new violations
Full ./scripts/test.sh -> 12137 passed, 18 failed (all 18 pre-existing
flakes in unrelated subsystems; verified not v2.2 regressions)
Address the RED gate cert on PR #5688 (2 BRICK + 1 CORE) and the
incoming Greptile review (1 P1 + 3 P2):
- BRICK-1 (race-safety): drop the mtime heuristic and any age-based
lock removal. Add /api/updates/clear_lock with a fail-closed
fcntl.flock holder probe (refuses if any process still holds the
file, on POSIX). On non-POSIX, fails closed. Touches only
.git/index.lock (the only well-known short-lived lock git creates).
- BRICK-2 (destructive path coupling): remove lock_conflict from the
force-button condition. Add a separate btnClearUpdateLock that calls
/api/updates/clear_lock -- never apply_force_update. The recovery
path never runs checkout/clean/reset --hard.
- CORE-1 (unconditional pre-cleanup): apply_force_update no longer
iterates .git/**/*.lock. Lock cleanup is exclusively the clear_lock
endpoint's job.
- Greptile P1: when a pull-lock error fires after a stash was pushed,
run git stash pop (with apply+drop fallback) so the user's local
modifications aren't stranded in the stash silently. v2 tests
cover the stashed and the clean cases.
- Greptile P2 (broad 'lock file' substring): tightened _GIT_LOCK_SIGNATURES
to specific git error strings ('index.lock': file exists, '.lock':
file exists, 'another git process seems to be running',
'unable to create .git/index.lock') so unrelated ref-transaction
lock-loss messages no longer trigger a lock-conflict response.
- Greptile P2 (redundant git_dir.exists / magic number): both gone as a
side effect of removing the apply_force_update cleanup loop entirely.
Refactor: rebased onto current upstream/master so PR is no longer
'behind' 130 commits.
Tests:
- 14 new v2 tests covering holder probe (true/false/NOGIL probe),
_try_remove_lock (refuse-on-held/success-on-unheld),
apply_clear_lock (success/refused), signature set parameter table
including false-positive class, apply_force_update no-touch
contract, pull-lock stash restore, pull-lock no-stash-when-clean.
- 2 v1 tests removed (they encoded the unsafe behavior).
- 2 v1 parametrize rows dropped (their positive cases relied on the
broad 'lock file' substring).
Closes#5687
- Detect lock file errors (lock_conflict flag) on fetch/status/pull
failures in _apply_update_inner (#1)
- Expose Force Update button on frontend when lock_conflict is
received (#3)
- Proactively clean stale lock files (>30s old) in apply_force_update
using rglob discovery instead of a hardcoded path list (#2)
- Add 10 tests: _is_git_lock_error parametrized unit tests,
_apply_update_inner lock path coverage, and apply_force_update
stale/recent lock behavior
Codex gate: the apply path now surfaces sanitized non-network fetch stderr, but
_QUERY_SECRET_RE only redacted access_token/token/password/auth/key — leaving
client_secret, private_token, oauth_token, api_key in a remote URL exposed in
the user-facing error. Broadened the pattern to cover those + app_secret/secret
+ api[_-]?key. Regression test added + proven non-vacuous.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
The update-check payload now carries a `dirty: bool` reflecting the
working-tree state vs HEAD, computed via the same `git diff-index
--quiet HEAD --` primitive already used by `_dirty_suffix()`. A dirty
install at-or-past the latest release tag used to silently report
"Up to date" with no remediation affordance; the Settings panel
can now read this flag to offer `apply_force_update` (which already
does `git checkout .` + `git reset --hard` to reset to a clean
latest).
The probe is conservatively clean on real errors (timeout, missing
git, fatal) so a transient probe failure never produces a false-
positive "local changes" alert.
Closes#4085
When agent checkouts track main past an older tag but the newest published
tag is on a divergent side branch, stop advertising tag-based updates and
route check/apply through the upstream branch instead.
MUST-FIX:
- tests/test_2735_open_in_vscode.py: bump expected open_in_vscode locale
counter from 10 to 11 (Turkish locale added in #2772). The bump fell
out of an in-rebase test edit but never got committed; tagging without
this would have shipped a failing test in the release commit.
SHOULD-FIX inline:
- api/updates.py: case-D drift in _select_apply_compare_ref. The original
#2855 fix used latest_tag in the past-tag predicate; the check side
uses current_tag (HEAD's nearest reachable tag) plus a 'behind == 0'
gate. They drift when HEAD is on an OLDER release tag with commits on
top AND a NEWER tag exists ('case D'): check correctly suggests
advancing to the newer tag, but apply fell through to origin/<branch>.
Mirror the check-side predicate exactly. Adds regression test
test_select_apply_compare_ref_case_d_older_tag_with_commits_and_newer_tag_exists.
- static/messages.js: post-await race guard in _restoreSettledSession.
stream_end without preceding 'done' enters the settlement path, awaits
/api/session, then sets _streamFinalized=true. If a late 'done' event
arrives during that await, it sees _streamFinalized still false and
double-runs the finalize. The guard returns early when done won the
race, avoiding double renderMessages() + double notification.
- server.py: CORS preflight Access-Control-Allow-Methods now includes PUT.
#2776 wired PUT into the router for /api/mcp/servers/{name} but didn't
update the OPTIONS response. Same-origin only in practice, but cosmetic
completeness for CORS-aware deployments.
Opus advisor verdict: all 5 risk areas reviewed, 1 MUST-FIX + 3 SHOULD-FIX
all addressed inline. Net: +69/-9, no new architecture, no behavior risk.
Fixes#2846. After PR #2758 (the #2653 fix) the update check correctly
falls through to the branch comparison when HEAD has moved past the
latest `v*` tag — so the banner reports the real commit count against
`origin/<branch>`. But `_select_apply_compare_ref` was never updated to
mirror that decision: as long as any `v*` tag exists, it returns
`tags[0]`, even when HEAD is far past it.
Result for everyone running hermes-agent past `v2026.5.16` (i.e. anyone
on agent master between tagged releases):
1. Banner: `Agent (origin/main): 254 updates available` ← correct
2. User clicks Update Now
3. `_select_apply_compare_ref` picks `v2026.5.16` because tags exist
4. `git pull --ff-only origin v2026.5.16` — no-op (HEAD is already past it)
5. `_schedule_restart()` fires anyway, server bounces
6. Next check still reports 254 behind — banner reappears unchanged
`apply_force_update` had the same bug, except worse: `git reset --hard
v2026.5.16` would have actively rewound the user's checkout 254 commits.
The root cause is the same bug class as #2653 — two parallel paths
(`_check_repo_release` and `_select_apply_compare_ref`) that should make
the same decision but didn't. Pre-fix, the "is HEAD past the latest
tag?" predicate lived inline inside `_check_repo_release` only.
Fix
---
Extract `_head_is_past_latest_tag(path, current_tag)` and have both
paths consult it. When HEAD is past the latest tag:
- check path: release check returns None → branch check runs (#2653,
unchanged behaviour, just refactored)
- apply path: falls through to upstream / `origin/<branch>`, never the
stale tag (#2846, new behaviour)
Tests
-----
- `test_select_apply_compare_ref_uses_tag_when_head_is_on_tag` —
unchanged behaviour pinned: HEAD exactly on tag → advance to tag.
- `test_select_apply_compare_ref_falls_through_when_head_is_past_tag` —
the #2846 repro: HEAD = v2026.5.16 + 608 commits → advance to
`origin/main`, not the tag.
- `test_select_apply_compare_ref_no_tags_uses_upstream` — unchanged.
- `test_select_apply_compare_ref_no_tags_no_upstream_uses_default_branch`
— unchanged.
- `test_check_and_apply_paths_agree_when_head_is_past_tag` — symmetry
test, ensures the two paths can't drift apart again.
All 21 tests in `tests/test_updates.py` pass locally (16 existing + 5
new).
Refs #2846, #2653.
When current_tag == latest_tag, _check_repo_release returned behind=0
and reported 'Up to date' even if master had moved hundreds of commits
past the tag. This was visible as Agent: v2026.5.16-593-gedb2d9105
alongside a green 'Up to date' pill in Settings.
Run 'git describe --tags --always' after computing behind==0. If the
output includes a -N-gSHA suffix the tag is not at HEAD; return None so
_check_repo_branch runs and counts the real commit gap via rev-list.
When HEAD is exactly on the latest tag the new branch is never taken and
behaviour is unchanged.
Fixes#2653.
Without --force, git fetch origin --tags refuses to overwrite divergent
local tags and returns 'would clobber existing tag', jamming the entire
WebUI update path indefinitely. The WebUI is a release-tracking consumer
that never pushes tags, so it should always defer to whatever the remote
says a release tag points to. Add --force to all three fetch-tag call
sites:
- _check_repo (the 'Check now' button + periodic check)
- apply_force_update (force-reset to remote HEAD)
- apply_update (stash + pull --ff-only)
Tests:
- Updated 3 existing tests in test_updates.py whose fake_git mocks
asserted the exact ['fetch', 'origin', '--tags'] args list.
- Updated 1 existing test in test_update_banner_fixes.py that asserted
the same shape for apply_update.
- Added 4 new regression tests:
- test_check_repo_fetches_tags_with_force
- test_apply_force_update_fetches_tags_with_force
- test_apply_update_fetches_tags_with_force
- test_check_repo_recovers_from_remote_retag (end-to-end,
proves the bare --tags fetch shape is no longer used)
Closes#2756.
Rebased and enhanced version of PR #287 by @ccqqlo:
- _run_git() now returns stderr on failure instead of empty string,
so the UI can surface actionable git error messages
- Added _split_remote_ref() to split tracking refs like origin/master
into separate remote + branch args for git pull
- Ignore untracked files in stash decision (--untracked-files=no) to
prevent misleading stash-pop failures
- Fail early with clear message on unresolved merge conflicts
- 4 unit tests covering stderr, stdout fallback, exit code, and ref splitting
Based on work by @ccqqlo in PR #287.
Co-authored-by: Nathan Esquenazi <nesquena@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rod Boev