Release: CJK full-width punctuation autolink fix (#6792) (#6991)

* fix(renderer): exclude CJK full-width punctuation from autolink URLs

* fix(test): widen autolink block search window for CJK punctuation fix (#6792)

The CJK full-width punctuation fix expanded the autolink regex and
comment text, pushing target='_blank' and esc(clean) beyond the 600/
700-char search windows in tests.  Widen to 1200 chars.

* fix(test): widen autolink window in test_issue470 to match 1200-char block

The CJK punctuation fix added comment lines that pushed href="${clean}"
beyond the 600-char search window in test_issue470.py::test_autolink_no_esc_on_href,
causing the same failure across 3.11/3.12/3.13 shard-4. test_issue342.py was
already widened in cc52202f; apply the same 1200-char window here.

* fix(test): drive real renderMd() for CJK full-width autolink behaviour

Replace the test-owned FIXED_REGEX copy in test_fullwidth_autolink.js with
a pytest + node harness that extracts and runs the ACTUAL renderMd() from
static/ui.js (same driver pattern as test_renderer_js_behaviour.py).

Coverage per review:
- every CJK trailing mark (），。；：！？、) through both the outer
  paragraph pass and the inline pass (list items + blockquotes)
- ASCII trailing marks (regression guard)
- normal URLs with query params (href keeps raw &, text is escaped)
- Markdown links and pre-existing anchors are stashed, never re-wrapped

Reverse-verified: reverting the \uFF09 exclusion + CJK strip makes 27 of
38 assertions fail on the real production code.

* fix(test): drop stray f-prefix from placeholder-free asserts (F541)

* docs(changelog): stamp #6792 (CJK autolink punctuation)

---------

Co-authored-by: happy5318 <happy5318@users.noreply.github.com>
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>

CHANGELOG.md

@@ -5,6 +5,8 @@
 
 ### Fixed
 
+- **Links ending in Chinese/Japanese full-width punctuation no longer swallow the punctuation into the URL.** When an assistant wrote a URL followed immediately by a CJK full-width mark (`）。，；：！？、`) — common in Chinese/Japanese prose — the autolink pass treated the mark as part of the link, so the rendered href included the trailing punctuation (and often 404'd). Both autolink passes now strip a single trailing CJK full-width mark the same way they already strip ASCII `.,;:!?)`, and a full-width close-paren also terminates the URL. ASCII behavior is unchanged. Thanks @happy5318. (#6792)
+
 - **Mobile: create forms opened from the sidebar drawer are no longer hidden behind it.** On phones the sidebar is a full-screen drawer over the main view; opening a new cron/skill/workspace/profile form from inside it left the drawer covering the form, so the tap looked like a no-op. The panel re-open on a rail tap now happens synchronously (before the panel's async data load, so a form that closes the drawer isn't re-covered), and the four create-form openers now close the drawer the same way their matching detail views already did. Desktop is unaffected. Thanks @silent-reader-cn. (#6921)
 
 - **Mobile: creating a conversation from a project chip now shows it instead of leaving the drawer open over it.** The project-chip quick-create ("+") created the new project conversation but never closed the full-screen mobile sidebar drawer, so on phones the new session was created but hidden underneath. The drawer now closes after the session is created and the sidebar repaints (the failure path keeps it open so the error toast stays visible for retry). Desktop is unaffected. Thanks @silent-reader-cn. (#6933)

static/ui.js

@@ -7455,7 +7455,7 @@ function renderMd(raw){
     // Stash [label](url) links before autolink so the URL in href= is not re-linked
     const _link_stash=[];
     t=t.replace(/\[([^\]]+)\]\(((?:https?:\/\/|file:\/\/|workspace:\/\/|session:\/\/|mailto:|tel:|message:)[^\s\)]+)\)/g,(_,lb,u)=>{_link_stash.push(_markdownAnchor(lb,u));return `\x00L${_link_stash.length-1}\x00`;});
-    t=t.replace(/(https?:\/\/[^\s<>"')\]]+)/g,(url)=>{const trail=url.match(/[.,;:!?)]$/)?url.slice(-1):'';const clean=trail?url.slice(0,-1):url;return `<a href="${clean}" target="_blank" rel="noopener">${esc(clean)}</a>${trail}`;});
+    t=t.replace(/(https?:\/\/[^\s<>"')\]\uFF09]+)/g,(url)=>{const trail=url.match(/[.,;:!?)\uFF09\uFF0C\uFF1B\uFF1A\uFF01\uFF1F\u3001\u3002]$/)?url.slice(-1):'';const clean=trail?url.slice(0,-1):url;return `<a href="${clean}" target="_blank" rel="noopener">${esc(clean)}</a>${trail}`;});
     t=t.replace(/\x00L(\d+)\x00/g,(_,i)=>_link_stash[+i]);
     t=t.replace(/\x00G(\d+)\x00/g,(_,i)=>_img_stash[+i]);
     // Escape any plain text that isn't already wrapped in a tag we produced
@@ -7757,9 +7757,11 @@ function renderMd(raw){
   // Stash <a>, <img> and <pre> blocks so autolink never runs inside them.
   const _al_stash=[];
   s=s.replace(/(<a\b[^>]*>[\s\S]*?<\/a>|<img\b[^>]*>|<pre\b[^>]*>[\s\S]*?<\/pre>)/g,m=>{_al_stash.push(m);return `\x00B${_al_stash.length-1}\x00`;});
-  s=s.replace(/(https?:\/\/[^\s<>"'\)\]]+)/g,(url)=>{
-    // Strip trailing punctuation that was likely not part of the URL
-    const trail=url.match(/[.,;:!?)]$/)?url.slice(-1):'';
+  s=s.replace(/(https?:\/\/[^\s<>"')\]\uFF09]+)/g,(url)=>{
+    // Strip trailing punctuation that was likely not part of the URL.
+    // CJK full-width punctuation (）。，；：！？、) is included because LLMs
+    // frequently use full-width delimiters in Chinese/Japanese text.
+    const trail=url.match(/[.,;:!?)]$/)||url.match(/[\uFF09\uFF0C\uFF1B\uFF1A\uFF01\uFF1F\u3001\u3002]$/)?url.slice(-1):'';
     const clean=trail?url.slice(0,-1):url;
     return `<a href="${clean}" target="_blank" rel="noopener">${esc(clean)}</a>${trail}`;
   });

tests/test_fullwidth_autolink_behaviour.py

@@ -0,0 +1,203 @@
+"""Behavioural tests for the CJK full-width punctuation autolink fix (#6792).
+
+These tests drive the ACTUAL renderMd() in static/ui.js via node — the same
+harness used by test_renderer_js_behaviour.py — so they exercise the real
+production autolink passes (inlineMd + outer paragraph pass) instead of a
+test-owned copy of the regex.
+
+Bug: the autolink regex only excluded ASCII ')' (U+0029) from URL matches.
+When a URL appeared before a CJK full-width right parenthesis ）(U+FF09) —
+extremely common in Chinese LLM output like "（或 https://lmarena.ai）" —
+the ）was swallowed into the <a href="..."> link.
+
+Fix: add \uFF09 to the regex exclusion set AND to the trailing-punctuation
+strip. Also covers full-width comma/period/colon/semicolon/exclamation/
+question mark/ideographic comma for completeness.
+
+Every case asserts BOTH directions:
+- the punctuation never enters href
+- the punctuation remains visible after the closing </a>
+"""
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).parent.parent.resolve()
+UI_JS_PATH = REPO_ROOT / "static" / "ui.js"
+
+NODE = shutil.which("node")
+
+pytestmark = pytest.mark.skipif(NODE is None, reason="node not on PATH")
+
+
+_DRIVER_SRC = r"""
+const fs = require('fs');
+const src = fs.readFileSync(process.argv[2], 'utf8');
+global.window = {};
+global.document = { createElement: () => ({ innerHTML: '', textContent: '' }), baseURI: 'http://localhost/app/' };
+function _sessionUrlForSid(sid) { return '/app/session/' + encodeURIComponent(String(sid || '')); }
+const esc = s => String(s ?? '').replace(/[&<>"']/g, c => (
+  {'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
+const _IMAGE_EXTS=/\.(png|jpg|jpeg|gif|webp|bmp|ico|avif)$/i;
+const _SVG_EXTS=/\.svg$/i;
+const _AUDIO_EXTS=/\.(mp3|ogg|wav|m4a|aac|flac|wma|opus|webm)$/i;
+const _VIDEO_EXTS=/\.(mp4|webm|mkv|mov|avi|ogv|m4v)$/i;
+// Minimal stand-in for ui.js' _inlineMediaHtmlForRef used when the driver
+// extracts only renderMd(). Mirrors the live UI for the cases the existing
+// tests assert against (https image, bare file:// image).
+function _inlineMediaHtmlForRef(ref){
+  const r = String(ref || '');
+  if (/^https?:\/\//.test(r)) return `<img class="msg-media-img" src="${esc(r)}" alt="image" loading="lazy">`;
+  if (/^file:\/\//.test(r)){
+    const m = r.replace(/^file:\/\//i, '');
+    return `<img class="msg-media-img" src="api/media?path=${encodeURIComponent(m)}" alt="image" loading="lazy">`;
+  }
+  return `<img class="msg-media-img" src="api/media?path=${encodeURIComponent(r)}" alt="image" loading="lazy">`;
+}
+
+function extractFunc(name) {
+  const re = new RegExp('function\\s+' + name + '\\s*\\(');
+  const start = src.search(re);
+  if (start < 0) throw new Error(name + ' not found');
+  let i = src.indexOf('{', start);
+  let depth = 1; i++;
+  while (depth > 0 && i < src.length) {
+    if (src[i] === '{') depth++;
+    else if (src[i] === '}') depth--;
+    i++;
+  }
+  return src.slice(start, i);
+}
+eval(extractFunc('_matchBacktickFenceLine'));
+eval(extractFunc('_isBacktickFenceClose'));
+eval(extractFunc('renderMd'));
+
+let buf = '';
+process.stdin.on('data', c => { buf += c; });
+process.stdin.on('end', () => { process.stdout.write(renderMd(buf)); });
+"""
+
+
+@pytest.fixture(scope="module")
+def driver_path(tmp_path_factory):
+    """Write the node driver to a tmp file (works around `node -e` arg quirks)."""
+    p = tmp_path_factory.mktemp("autolink_driver") / "driver.js"
+    p.write_text(_DRIVER_SRC, encoding="utf-8")
+    return str(p)
+
+
+def _render(driver_path, markdown: str) -> str:
+    """Run renderMd against the actual ui.js and return the rendered HTML."""
+    result = subprocess.run(
+        [NODE, driver_path, str(UI_JS_PATH)],
+        input=markdown,
+        capture_output=True,
+        text=True,
+        timeout=30,
+    )
+    if result.returncode != 0:
+        raise RuntimeError(f"node driver failed: {result.stderr}")
+    return result.stdout
+
+
+# Every CJK full-width mark the fix adds to the trailing-punctuation strip:
+# ）(U+FF09) ，(U+FF0C) 。(U+3002) ；(U+FF1B) ：(U+FF1A) ！(U+FF01) ？(U+FF1F) 、(U+3001)
+CJK_TRAILING_MARKS = ["）", "，", "。", "；", "：", "！", "？", "、"]
+# ASCII punctuation the existing autolink strip already handled (regression guard).
+ASCII_TRAILING_MARKS = [")", ",", ".", ";", ":", "!", "?"]
+
+
+class TestAutolinkOuterParagraphPass:
+    """Plain-paragraph URLs go through the OUTER autolink pass in renderMd()."""
+
+    @pytest.mark.parametrize("mark", CJK_TRAILING_MARKS)
+    def test_cjk_trailing_mark_stripped_and_visible_after_anchor(self, driver_path, mark):
+        out = _render(driver_path, f"See https://example.com{mark}")
+        assert 'href="https://example.com"' in out, (
+            f"CJK mark {mark!r} must not enter href. Got: {out!r}"
+        )
+        assert f'href="https://example.com{mark}"' not in out, (
+            f"CJK mark {mark!r} leaked into href. Got: {out!r}"
+        )
+        assert f"</a>{mark}" in out, (
+            f"CJK mark {mark!r} must stay visible after </a>. Got: {out!r}"
+        )
+
+    @pytest.mark.parametrize("mark", ASCII_TRAILING_MARKS)
+    def test_ascii_trailing_mark_stripped_and_visible_after_anchor(self, driver_path, mark):
+        out = _render(driver_path, f"See https://example.com{mark}")
+        assert 'href="https://example.com"' in out, (
+            f"ASCII mark {mark!r} must not enter href. Got: {out!r}"
+        )
+        assert f'href="https://example.com{mark}"' not in out
+        assert f"</a>{mark}" in out
+
+    def test_fullwidth_paren_wrapped_url(self, driver_path):
+        """The original bug shape: （或 https://lmarena.ai）."""
+        out = _render(driver_path, "（或 https://lmarena.ai）")
+        assert 'href="https://lmarena.ai"' in out
+        assert 'href="https://lmarena.ai）"' not in out
+        assert "</a>）" in out
+
+    def test_normal_url_with_query_params_untouched(self, driver_path):
+        out = _render(driver_path, "https://example.com/path?q=1&x=2")
+        # href keeps the raw & (esc() is NOT applied to href — that would
+        # corrupt query strings)…
+        assert 'href="https://example.com/path?q=1&x=2"' in out
+        # …while the visible link text IS escaped (XSS safety).
+        assert ">https://example.com/path?q=1&amp;x=2</a>" in out
+
+    def test_markdown_link_not_double_autolinked(self, driver_path):
+        """[label](url) must be stashed before autolink so its URL is not re-linked."""
+        out = _render(driver_path, "[see docs](https://example.com/doc)")
+        assert 'href="https://example.com/doc"' in out
+        # The link text must be the label, not a second autolinked URL.
+        assert "<a href=\"https://example.com/doc\" target=\"_blank\" rel=\"noopener\">see docs</a>" in out
+        assert out.count("<a ") == 1, f"Markdown link must not be double-wrapped. Got: {out!r}"
+
+    def test_preexisting_anchor_not_autolinked(self, driver_path):
+        """<a href=...> blocks must be stashed so autolink never runs inside them."""
+        out = _render(driver_path, '<a href="https://example.com/pre">pre</a>')
+        assert 'href="https://example.com/pre"' in out
+        assert out.count("<a ") == 1, f"Existing anchor must not be re-wrapped. Got: {out!r}"
+
+
+class TestAutolinkInlinePass:
+    """List items and blockquotes render through inlineMd() — the INLINE autolink pass."""
+
+    @pytest.mark.parametrize("mark", CJK_TRAILING_MARKS)
+    def test_list_item_cjk_mark_stripped(self, driver_path, mark):
+        out = _render(driver_path, f"- See https://example.com{mark}")
+        assert 'href="https://example.com"' in out, (
+            f"Inline pass leaked CJK mark {mark!r} into href. Got: {out!r}"
+        )
+        assert f'href="https://example.com{mark}"' not in out
+        assert f"</a>{mark}" in out
+
+    @pytest.mark.parametrize("mark", CJK_TRAILING_MARKS)
+    def test_blockquote_cjk_mark_stripped(self, driver_path, mark):
+        out = _render(driver_path, f"> See https://example.com{mark}")
+        assert 'href="https://example.com"' in out, (
+            f"Inline pass leaked CJK mark {mark!r} into href. Got: {out!r}"
+        )
+        assert f'href="https://example.com{mark}"' not in out
+        assert f"</a>{mark}" in out
+
+    def test_ordered_list_item_cjk_mark_stripped(self, driver_path):
+        out = _render(driver_path, "1. See https://example.com！")
+        assert 'href="https://example.com"' in out
+        assert 'href="https://example.com！"' not in out
+        assert "</a>！" in out
+
+    def test_inline_fullwidth_paren_wrapped_url(self, driver_path):
+        out = _render(driver_path, "- （或 https://lmarena.ai）")
+        assert 'href="https://lmarena.ai"' in out
+        assert 'href="https://lmarena.ai）"' not in out
+        assert "</a>）" in out
+
+    def test_inline_markdown_link_not_double_autolinked(self, driver_path):
+        out = _render(driver_path, "> [link](https://example.com/x)")
+        assert 'href="https://example.com/x"' in out
+        assert out.count("<a ") == 1, f"Inline markdown link double-wrapped. Got: {out!r}"

tests/test_issue342.py

@@ -45,8 +45,8 @@ def test_autolink_uses_esc_for_xss_safety():
     # Find the autolink section (between the SAFE_TAGS pass and paragraph wrap)
     autolink_idx = content.find('// Autolink: convert plain URLs')
     assert autolink_idx != -1, "Autolink comment not found in ui.js"
-    # Extract the autolink block (next ~600 chars after the comment)
-    autolink_block = content[autolink_idx:autolink_idx + 600]
+    # Extract the autolink block (next ~1200 chars after the comment)
+    autolink_block = content[autolink_idx:autolink_idx + 1200]
     # esc() must be used on the visible link text to prevent XSS
     assert 'esc(clean)' in autolink_block, (
         "Autolink block should use esc(clean) for the link display text (XSS safety), "
@@ -105,7 +105,7 @@ def test_autolink_target_blank_and_rel():
     autolink_idx = content.find('// Autolink: convert plain URLs')
     assert autolink_idx != -1, "Autolink comment not found"
     # Use a larger window to account for the stash preamble added by the fix
-    autolink_block = content[autolink_idx:autolink_idx + 700]
+    autolink_block = content[autolink_idx:autolink_idx + 1200]
     assert 'target="_blank"' in autolink_block, (
         'Autolinked URLs should have target="_blank"'
     )

tests/test_issue470.py

@@ -150,7 +150,7 @@ def test_autolink_no_esc_on_href():
     """Fixed autolink pass must not call esc() on href URL."""
     idx = UI_JS.find('// Autolink: convert plain URLs to clickable links.')
     assert idx != -1, "New autolink comment not found"
-    autolink_section = UI_JS[idx:idx+600]
+    autolink_section = UI_JS[idx:idx+1200]
     # The return line should have href="${clean}" (JS template literal, no esc call)
     assert 'href="${clean}"' in autolink_section, (
         'Autolink should use href="${clean}" not href="${esc(clean)}"'