Merge pull request #5101 from nesquena/feat/extension-tts-engine-registration

feat(extensions): TTS-engine registration capability (registerHermesTtsEngine)

CHANGELOG.md

@@ -5,6 +5,8 @@
 
 ### Added
 
+- **Extensions can now register a custom text-to-speech engine** via a new `window.registerHermesTtsEngine({ id, label, synthesize })` API. A registered engine appears in the Settings → TTS Engine dropdown alongside the built-ins (Browser / Edge / ElevenLabs) and is used by **both** playback paths — the hands-free voice-mode auto-read and the per-message "Listen" button. The extension only provides an async `synthesize(text, opts)` that returns audio bytes (ArrayBuffer / Blob); core owns selection, the dropdown option (label inserted via `textContent`, never HTML), and playback through the same `<audio>` lifecycle as the Edge engine. Built-in engine ids (`browser`/`edge`/`elevenlabs`) are reserved and can't be shadowed; invalid descriptors are rejected; a failed/empty synth result degrades gracefully. This unblocks TTS-engine extensions (e.g. a local VOICEVOX engine). Documented in `docs/EXTENSIONS.md`. (extension TTS-engine registration capability)
+
 - **Settings → Extensions gallery cards now link back to their source and show permissions in readable groups.** Gallery and Installed cards now surface a safe "Source" link from registry metadata (falling back to the extension registry path when needed), and the permissions disclosure renders WebUI API access, storage, DOM, sidecar/native-host, filesystem, and external-network declarations as plain rows instead of raw JSON. Thanks @franksong2702. (#5093, #5095)
 
 - **Opt-in CSP `frame-src` allowlist so an extension can embed a self-hosted web app in an iframe.** A new `HERMES_WEBUI_CSP_FRAME_EXTRA` env var widens what the WebUI page may embed in an `<iframe>` (default is same-origin only), mirroring the existing `HERMES_WEBUI_CSP_CONNECT_EXTRA` knob — space-separated `http(s)` origins with optional wildcard subdomain and port, validated and ignored if malformed. This unblocks an "external app tab" extension (pin Grafana / Vaultwarden / your own dashboard as a tab). It only governs what *this* page can embed; `frame-ancestors` stays `'none'`, so it does not let anyone embed the WebUI itself. Default-off; with the var unset, behavior is unchanged. Documented in `docs/EXTENSIONS.md`.

docs/EXTENSIONS.md

@@ -370,6 +370,46 @@ extension inherits the guard:
 This is the supported, forward-looking way for theme-pack and theme-creator
 extensions to integrate with the built-in appearance system.
 
+## Registering a custom TTS engine
+
+Extensions can contribute a **text-to-speech engine** that appears in the
+**Settings → TTS Engine** dropdown alongside the built-ins (Browser / Edge /
+ElevenLabs) and is used by **both** playback paths — the hands-free voice-mode
+auto-read and the per-message "Listen" button. Call
+`window.registerHermesTtsEngine(descriptor)`:
+
+```javascript
+window.registerHermesTtsEngine({
+  id: 'voicevox',                 // [a-z0-9_-], not a built-in
+  label: 'VOICEVOX (local)',      // shown in the dropdown (textContent — escaped)
+  // synthesize(text, opts) -> Promise<ArrayBuffer | Blob | TypedArray> of audio.
+  // opts carries the user's saved { voice, rate, pitch } (engine may ignore).
+  synthesize(text, opts) {
+    return fetch('http://127.0.0.1:50021/...', { /* ... */ })
+      .then(r => r.arrayBuffer());
+  }
+});  // -> true on success, false if rejected
+```
+
+Rules and guarantees:
+
+- **id** must be slug-safe (`[a-z0-9][a-z0-9_-]{0,31}`) and may **not** shadow a
+  built-in engine (`browser`, `edge`, `elevenlabs`) — those are reserved.
+- **label** is inserted with `textContent`, never `innerHTML` (no markup
+  injection into the dropdown).
+- `synthesize` must return audio bytes (`ArrayBuffer`, `Blob`, or a typed array);
+  core coerces to an `ArrayBuffer` and plays it through the same `<audio>`
+  lifecycle as the Edge engine (including stop/rearm in voice mode). A rejected
+  promise or empty/invalid result fails gracefully (toast on the Listen button;
+  re-listen in voice mode).
+- Core owns selection, the dropdown option, and playback; the extension only
+  produces audio. Re-registering the same id updates it in place.
+- **Network note:** if your engine calls a local server (e.g. VOICEVOX on
+  `http://127.0.0.1:50021`), that request is a same-origin-policy / CSP
+  `connect-src` concern like any extension network call — loopback is already in
+  the default `connect-src`. Declare `permissions.network_external` honestly
+  based on where it calls.
+
 ## Extension authoring guidance
 
 Extensions share the page with the WebUI app, so they should be additive and

static/boot.js

@@ -885,6 +885,68 @@ function _micToastKeyForRecognitionError(error){
 window._micActive=window._micActive||false;
 window._micPendingSend=window._micPendingSend||false;
 
+// ── Extension TTS-engine registry (registerHermesTtsEngine) ──────────────────
+// Defined at MODULE scope (not inside the voice-mode IIFE below) so the public
+// API exists even on browsers without SpeechRecognition / speechSynthesis — an
+// extension can register a TTS engine regardless of STT/browser-TTS support.
+// Lets a trusted local extension contribute a TTS engine that appears in the
+// Settings -> TTS Engine dropdown and is used by BOTH playback paths (voice-mode
+// auto-read and the per-message Listen button). The extension provides an async
+// synthesize(text, opts) that returns audio bytes (ArrayBuffer or Blob); core
+// handles selection, the dropdown option, and playback. Mirrors registerHermesSkin.
+//
+//   window.registerHermesTtsEngine({
+//     id: 'voicevox',            // [a-z0-9_-], not a built-in (browser/edge/elevenlabs)
+//     label: 'VOICEVOX (local)',
+//     synthesize(text, opts) { return Promise<ArrayBuffer|Blob>; }
+//   }) -> true on success, false if rejected
+var _HERMES_TTS_ENGINES = Object.create(null);
+var _HERMES_TTS_RESERVED = { browser:1, edge:1, elevenlabs:1 };
+function _hermesTtsValidId(id){ return typeof id==='string' && /^[a-z0-9][a-z0-9_-]{0,31}$/.test(id); }
+function _hermesAddTtsOption(id, label){
+  var sel=document.getElementById('settingsTtsEngine');
+  if(!sel) return;
+  if(sel.querySelector('option[value="'+id+'"]')) return;
+  var opt=document.createElement('option');
+  opt.value=id;
+  opt.textContent=label;   // textContent — never innerHTML (no injection)
+  sel.appendChild(opt);
+}
+window.registerHermesTtsEngine=function(desc){
+  try{
+    if(!desc||typeof desc!=='object') return false;
+    var id=String(desc.id||'').toLowerCase();
+    if(!_hermesTtsValidId(id)) return false;
+    if(_HERMES_TTS_RESERVED[id]) return false;          // can't shadow a built-in
+    if(typeof desc.synthesize!=='function') return false;
+    var label=(typeof desc.label==='string' && desc.label.trim()) ? desc.label.trim().slice(0,48) : id;
+    _HERMES_TTS_ENGINES[id]={ id:id, label:label, synthesize:desc.synthesize };
+    _hermesAddTtsOption(id, label);
+    return true;
+  }catch(_){ return false; }
+};
+window._hermesTtsIsRegistered=function(id){ return !!_HERMES_TTS_ENGINES[id]; };
+// List registered engines (for the settings panel to re-add options on render).
+window._hermesTtsEngineOptions=function(){
+  return Object.keys(_HERMES_TTS_ENGINES).map(function(k){
+    return { id:_HERMES_TTS_ENGINES[k].id, label:_HERMES_TTS_ENGINES[k].label };
+  });
+};
+// Returns a Promise<ArrayBuffer> or null if the engine isn't registered.
+window._hermesTtsSynth=function(id, text, opts){
+  var eng=_HERMES_TTS_ENGINES[id];
+  if(!eng) return null;
+  return Promise.resolve()
+    .then(function(){ return eng.synthesize(text, opts||{}); })
+    .then(function(out){
+      if(!out) throw new Error('empty TTS result');
+      if(out instanceof ArrayBuffer) return out;
+      if(typeof Blob!=='undefined' && out instanceof Blob) return out.arrayBuffer();
+      if(out.buffer instanceof ArrayBuffer) return out.buffer;   // typed array
+      throw new Error('TTS engine returned an unsupported type');
+    });
+};
+
 // ── Turn-based voice mode (#1333) ────────────────────────────────────────
 // Chained flow: listen → send → (agent processes) → TTS response → listen again
 (function(){
@@ -1118,6 +1180,46 @@ window._micPendingSend=window._micPendingSend||false;
     }
     if(!clean){ _startListening(); return; }
     const engine=localStorage.getItem("hermes-tts-engine")||"browser";
+    // Extension-registered TTS engine (window.registerHermesTtsEngine): synth
+    // via the extension, then play through the same Audio lifecycle as edge.
+    if(typeof window._hermesTtsIsRegistered==='function' && window._hermesTtsIsRegistered(engine)){
+      _ttsSpeaking=true;
+      const _opts={
+        voice: localStorage.getItem("hermes-tts-voice")||'',
+        rate: parseFloat(localStorage.getItem("hermes-tts-rate")),
+        pitch: parseFloat(localStorage.getItem("hermes-tts-pitch")),
+      };
+      Promise.resolve(window._hermesTtsSynth(engine, clean, _opts))
+        .then(function(buf){
+          const blob=new Blob([buf]);
+          const url=URL.createObjectURL(blob);
+          const audio=new Audio(url);
+          _playingEdgeAudio=audio;
+          audio.onended=function(){
+            _ttsSpeaking=false;
+            if(_playingEdgeAudio===audio) _playingEdgeAudio=null;
+            URL.revokeObjectURL(url);
+            if(_voiceModeActive) setTimeout(function(){_startListening();},500);
+          };
+          audio.onerror=function(){
+            _ttsSpeaking=false;
+            if(_playingEdgeAudio===audio) _playingEdgeAudio=null;
+            URL.revokeObjectURL(url);
+            if(_voiceModeActive) setTimeout(function(){_startListening();},1000);
+          };
+          audio.play().catch(function(){
+            _ttsSpeaking=false;
+            if(_playingEdgeAudio===audio) _playingEdgeAudio=null;
+            URL.revokeObjectURL(url);
+            if(_voiceModeActive) setTimeout(function(){_startListening();},1000);
+          });
+        })
+        .catch(function(){
+          _ttsSpeaking=false;
+          if(_voiceModeActive) setTimeout(function(){_startListening();},1000);
+        });
+      return;
+    }
     if(engine==="elevenlabs"){
       _ttsSpeaking=true;
       fetch(new URL('api/tts', document.baseURI || location.href).href, {

static/panels.js

@@ -7938,6 +7938,18 @@ async function loadSettingsPanel(){
     // TTS engine selector
     const ttsEngineSel=$('settingsTtsEngine');
     if(ttsEngineSel){
+      // Re-add any extension-registered TTS engines (window.registerHermesTtsEngine)
+      // as options — the <select> markup only hardcodes the built-ins, and this
+      // settings panel can render after an extension registered its engine.
+      if(typeof window._hermesTtsEngineOptions==='function'){
+        window._hermesTtsEngineOptions().forEach(function(e){
+          if(!ttsEngineSel.querySelector('option[value="'+e.id+'"]')){
+            var opt=document.createElement('option');
+            opt.value=e.id; opt.textContent=e.label;
+            ttsEngineSel.appendChild(opt);
+          }
+        });
+      }
       const saved=localStorage.getItem('hermes-tts-engine')||'browser';
       ttsEngineSel.value=saved;
       ttsEngineSel.onchange=function(){

static/ui.js

@@ -6727,6 +6727,26 @@ function speakMessage(btn){
     _playEdgeTtsChunked(clean, btn);
     return;
   }
+  // Extension-registered TTS engine (window.registerHermesTtsEngine). Synthesize
+  // via the extension, then play through the shared audio-buffer path.
+  if(typeof window._hermesTtsIsRegistered==='function' && window._hermesTtsIsRegistered(engine)){
+    if(btn) btn.dataset.speaking='1';
+    _ttsSpeaking=true;
+    const _failReg=function(msg){
+      _ttsSpeaking=false;_playingEdgeAudio=null;
+      if(btn)btn.dataset.speaking='0';
+      if(msg&&typeof showToast==='function') showToast(msg,4000,'error');
+    };
+    const _opts={
+      voice: localStorage.getItem('hermes-tts-voice')||'',
+      rate: parseFloat(localStorage.getItem('hermes-tts-rate')),
+      pitch: parseFloat(localStorage.getItem('hermes-tts-pitch')),
+    };
+    Promise.resolve(window._hermesTtsSynth(engine, clean, _opts))
+      .then(function(buf){ return _playAudioBuf(buf, btn, 'TTS'); })
+      .catch(function(e){ _failReg((e&&e.message)||'TTS engine failed'); });
+    return;
+  }
 
   if(!('speechSynthesis' in window)){
     showToast(t('tts_not_supported')||'Speech synthesis not supported in this browser.');
@@ -6858,6 +6878,24 @@ function autoReadLastAssistant(){
     _playEdgeTtsChunked(clean, null);
     return;
   }
+  // Extension-registered TTS engine (window.registerHermesTtsEngine): synth via
+  // the extension, then play through the shared audio-buffer path. Mirrors the
+  // registered-engine branch in speakMessage() so auto-read honors the selection.
+  if(typeof window._hermesTtsIsRegistered==='function' && window._hermesTtsIsRegistered(engine)){
+    _ttsSpeaking=true;
+    const _opts={
+      voice: localStorage.getItem('hermes-tts-voice')||'',
+      rate: parseFloat(localStorage.getItem('hermes-tts-rate')),
+      pitch: parseFloat(localStorage.getItem('hermes-tts-pitch')),
+    };
+    Promise.resolve(window._hermesTtsSynth(engine, clean, _opts))
+      .then(function(buf){ return _playAudioBuf(buf, null, 'TTS'); })
+      .catch(function(){ _ttsSpeaking=false; _playingEdgeAudio=null; });
+    return;
+  }
+  // Unknown/unregistered engine (e.g. an extension engine that's no longer
+  // registered) — fall back to browser TTS only if it's available.
+  if(!('speechSynthesis' in window)) return;
   // Use chunked playback for browser TTS
   _ttsChunkQueue=_splitForTTS(clean);
   _ttsChunkIndex=0;

tests/test_extension_tts_engine_registration.py

@@ -0,0 +1,108 @@
+"""Extension TTS-engine registration capability (window.registerHermesTtsEngine).
+
+Two layers:
+  1. Structural — the public API + the two playback paths + the settings re-add
+     hook exist.
+  2. Behavioral — a Node harness extracts the registry from boot.js and exercises
+     register / validation / reserved-key guard / synth type-coercion.
+"""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+import textwrap
+from pathlib import Path
+
+import pytest
+
+REPO = Path(__file__).resolve().parent.parent
+BOOT_JS = (REPO / "static" / "boot.js").read_text(encoding="utf-8")
+UI_JS = (REPO / "static" / "ui.js").read_text(encoding="utf-8")
+PANELS_JS = (REPO / "static" / "panels.js").read_text(encoding="utf-8")
+
+
+def test_public_api_present():
+    assert "window.registerHermesTtsEngine=function" in BOOT_JS
+    assert "window._hermesTtsSynth=function" in BOOT_JS
+    assert "window._hermesTtsIsRegistered=function" in BOOT_JS
+    assert "window._hermesTtsEngineOptions=function" in BOOT_JS
+
+
+def test_reserved_builtins_guarded():
+    assert "_HERMES_TTS_RESERVED" in BOOT_JS
+    # browser/edge/elevenlabs must be reserved so an extension can't shadow them
+    assert "browser:1" in BOOT_JS and "edge:1" in BOOT_JS and "elevenlabs:1" in BOOT_JS
+
+
+def test_both_playback_paths_check_registry():
+    # voice-mode auto-read (boot.js _speakResponse) and the per-message Listen
+    # button (ui.js speakMessage) must both route registered engines.
+    assert "_hermesTtsIsRegistered(engine)" in BOOT_JS, "voice-mode path must check the registry"
+    assert "_hermesTtsIsRegistered(engine)" in UI_JS, "per-message path must check the registry"
+
+
+def test_settings_panel_readds_registered_options():
+    assert "_hermesTtsEngineOptions" in PANELS_JS, (
+        "settings panel must re-add registered engine options on render"
+    )
+
+
+def test_option_label_uses_textcontent_not_innerhtml():
+    # the engine label goes into the dropdown via textContent (no HTML injection)
+    assert "opt.textContent=label" in BOOT_JS
+
+
+@pytest.mark.skipif(shutil.which("node") is None, reason="node not available")
+def test_registration_behavior():
+    """Drive the real registry logic from boot.js in a Node harness."""
+    start = BOOT_JS.index("var _HERMES_TTS_ENGINES")
+    end = BOOT_JS.index("window._hermesTtsSynth=function")
+    end = BOOT_JS.index("};", BOOT_JS.index("throw new Error('TTS engine returned", end)) + 2
+    region = BOOT_JS[start:end]
+
+    harness = textwrap.dedent(
+        """
+        const window = {};
+        const document = { getElementById: () => null };  // no <select> in harness
+        %s
+        const results = {};
+        // valid registration
+        results.validOk = window.registerHermesTtsEngine({
+          id: 'voicevox', label: 'VOICEVOX', synthesize: () => new ArrayBuffer(4)
+        });
+        results.isRegistered = window._hermesTtsIsRegistered('voicevox');
+        // reserved key rejected
+        results.reservedRejected = (window.registerHermesTtsEngine({
+          id: 'edge', label: 'x', synthesize: () => new ArrayBuffer(1) }) === false);
+        // bad id rejected
+        results.badIdRejected = (window.registerHermesTtsEngine({
+          id: 'Bad Id!', label: 'x', synthesize: () => new ArrayBuffer(1) }) === false);
+        // missing synthesize rejected
+        results.noSynthRejected = (window.registerHermesTtsEngine({ id: 'nosynth', label: 'x' }) === false);
+        // options list reflects the registered engine
+        results.optionListed = window._hermesTtsEngineOptions().some(e => e.id === 'voicevox');
+        // synth coerces ArrayBuffer through
+        window._hermesTtsSynth('voicevox', 'hi', {}).then(buf => {
+          results.synthReturnsArrayBuffer = (buf instanceof ArrayBuffer);
+          // unregistered engine returns null
+          results.unregisteredNull = (window._hermesTtsSynth('ghost', 'hi', {}) === null);
+          console.log(JSON.stringify(results));
+        });
+        """
+    ) % region
+
+    out = subprocess.run(
+        ["node", "-e", harness], capture_output=True, text=True, timeout=30
+    )
+    assert out.returncode == 0, f"harness failed: {out.stderr}"
+    import json
+    r = json.loads(out.stdout.strip().splitlines()[-1])
+    assert r["validOk"] is True
+    assert r["isRegistered"] is True
+    assert r["reservedRejected"] is True
+    assert r["badIdRejected"] is True
+    assert r["noSynthRejected"] is True
+    assert r["optionListed"] is True
+    assert r["synthReturnsArrayBuffer"] is True
+    assert r["unregisteredNull"] is True