fix: apply Chinese-aware title matching in the four index resolvers (#798) (#799)

* fix: apply Chinese-aware title matching in the four index resolvers (#798)

`chinese_literature_client.py` already carried a Chinese-aware
`normalize_cn_title` / `has_cjk`, but the four index resolvers (Semantic
Scholar / OpenAlex / Crossref / arXiv) read the ASCII-centric helpers in
`_text_similarity.py`, where `.lower()` folds case but never width (Ｐ
U+FF30 never reaches P U+0050) and `string.punctuation` contains none of
`。`, `《》`, or U+3000.

A real Chinese paper served by an index in a different-but-legitimate
typesetting therefore missed on two paths: the DOI-keyed cross-check,
which gates on the fuzzy ratio alone and scored a fullwidth spelling of
the identical title at 0.625 (under the 0.70 floor) reporting a correct
DOI as DOI_MISMATCH; and the title-fallback search, which requires ratio
AND exact equality and so fell to `unresolvable`. Both feed the
`*_unmatched` contamination signals, so a genuine paper could render as
CONTAMINATED-TRIANGULATION-UNMATCHED.

Promotes `has_cjk` / `normalize_cn_title` into `_text_similarity.py`
byte-identical (the CJK client now re-imports rather than keeping a
private copy, per the #128 anti-drift goal), adds the Chinese-aware form
to `exact_normalized_title` as an additive third branch, and folds it
into `_similarity` through the existing `max`. Both gated on BOTH sides
carrying a Han ideograph, so every non-CJK verdict is provably unchanged
— pinned by an oracle test restating the pre-fix formula in full.

31 new tests, each verified to fail against the pre-fix module.
Full suite: 9255 passed, 3 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qexY5ysaqaAyPp97byf4w

* test: force the ratio-independence and non-destructiveness proofs (#798 review)

Addresses the three requested changes on PR #799.

1. `_cn_titles_match` ratio-independence is now forced, not inferred, in the
   test that claims it. `test_legitimate_variants_match_despite_a_sub_threshold_
   fuzzy_ratio` asserted the match on a pair the repaired `_similarity` scores
   1.0, so a regression that ANDed the ratio back in as a necessary condition
   would still have passed. Its match assertions now run inside a
   `monkeypatch.context()` with `_similarity` replaced by a detonator, scoped so
   the ratio measurements above it still see the real function. A new
   `test_cn_titles_match_never_consults_the_fuzzy_ratio` adds the negative half
   under the same forced conditions, so the invariant cannot be satisfied by a
   helper that has stopped discriminating.

   The shared `_forbid_similarity` helper patches BOTH binding paths: the
   `_text_similarity` module attribute (a qualified call or lazy in-function
   import) and the client's own namespace (a module-level `from ... import
   _similarity`, already bound and blind to the first patch). Both styles were
   mutation-verified to trip it; before this change the named test passed the
   regression that the new one caught.

2. `test_ratio_never_lowered_off_the_cjk_path` asserted `>=` against the base
   ratio alone, so it passed a *raised* non-CJK score and never exercised the
   dotted-acronym branch. Renamed to `test_ratio_unchanged_off_the_cjk_path`
   and rewritten against a full `_pre_fix_similarity` oracle — the companion to
   the existing `_pre_fix_exact_normalized_title`, written out in full for the
   same anti-drift reason — asserting exact equality. Mutation-verified twice:
   one raising a base-branch score, one confined to the acronym branch; the old
   assertion caught neither.

3. "Byte-identical" corrected to "behaviorally equivalent" in the
   `normalize_cn_title` docstring and the CHANGELOG. The promotion hoists the
   wrapper/terminal-mark sets to module constants, precompiles the regex, and
   rewrites comments; behavioral equivalence is what the tests actually pin.
   CHANGELOG test count corrected 31 -> 32 and its oracle sentence updated to
   describe both oracles.

Full suite: 9256 passed, 3 skipped (+1 test). The pre-existing
`test_evidence_rows.py::test_gfm_bare_urls_emails_and_schemes_cannot_autolink`
failure is unchanged and also fails on clean main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qexY5ysaqaAyPp97byf4w

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+### Fixed
+
+- **CJK titles no longer fail the shared exact-title gate in the four index resolvers (#798).** `chinese_literature_client.py` already carried a Chinese-aware `normalize_cn_title` / `has_cjk` (#431 §"Chinese title matching"), but the four index resolvers (Semantic Scholar / OpenAlex / Crossref / arXiv) never saw it — they read the ASCII-centric `_text_similarity` helpers, where `.lower()` folds case but never width (Ｐ U+FF30 never reaches P U+0050) and `string.punctuation` contains none of `。`, `《》`, or U+3000. A Chinese paper an index served in a different-but-legitimate typesetting therefore missed on **two** paths: the DOI-keyed cross-check, which gates on the fuzzy ratio *alone* and scored a fullwidth spelling of the identical title at **0.625 — under the 0.70 floor — reporting a correct DOI as `DOI_MISMATCH`**; and the title-fallback search, which requires ratio **and** exact-normalized equality and so fell to `unresolvable`. Both feed the `*_unmatched` contamination signals, so a genuine paper could accumulate *k* across indexes and be rendered `CONTAMINATED-TRIANGULATION-UNMATCHED` — the protocol doc's own "P0, next to the word 'fabricated'". The failure was invisible to the English test corpus for a measurable reason: a Han character is a whole word, so the same six-codepoint corruption is 37.5% of a 16-character Chinese title but 8% of its 68-character English equivalent (measured 0.625 vs 0.912). Fix promotes `has_cjk` / `normalize_cn_title` into `scripts/_text_similarity.py` (behaviorally equivalent — the promotion hoists locals to module constants, precompiles the regex, and rewrites comments; the CJK client now re-imports rather than keeping a private copy, per the #128 anti-drift goal), adds the Chinese-aware form to `exact_normalized_title` as an additive third branch, and folds it into `_similarity` through the existing `max`. Both are gated on **both** sides carrying a Han ideograph, so every non-CJK verdict and every non-CJK *ratio* is provably unchanged — pinned by two oracle tests that re-state the pre-fix `exact_normalized_title` and `_similarity` formulas in full (the latter including the dotted-acronym branch) and assert exact agreement in both directions, so a raised score is caught as loudly as a lowered one. Cross-script and romanized pairs still cannot match (no translation oracle), Simplified/Traditional is still not folded, and an empty normalized key still never matches. Under the Chinese-aware form the ratio also regains discriminative power on the motivating pair: 1.000 for the identical title against an unchanged 0.510 for a genuinely different paper (the base form separated 0.566 from 0.510 — almost nothing). Also measured and pinned: on the motivating pair the pre-fix ratio was actively *anti-correlated* — the identical title scored **0.606 while a genuinely different paper scored 0.645**, so the wrong paper ranked higher. The repair leaves the unrelated pair's score byte-identical (0.6452 before and after; it is not equal under the CJK form, so nothing is folded in) and lifts only the true match, restoring the ordering title ranking depends on. Not in scope, and unchanged: the base ASCII normalization still collapses `ER+`/`ER-` and `p53`/`P53` because it maps ASCII punctuation to whitespace and lowercases, and `exact_normalized_title` ORs that form in; the fuzzy floor also remains a weak separator for CJK generally (that 0.645 near-miss is pre-existing and untouched here). 32 new tests, including 5 integration tests through the real Crossref client covering both broken paths, each verified to fail against the pre-fix module, plus a monkeypatched detonator pinning that the Chinese DOI-path matcher never consults the fuzzy ratio in either direction.
+
 ## [3.21.1] - 2026-08-24 — Bounded workflow substrates, sealed bakeoffs, and transport hardening
 
 ### Fixed

deep-research/references/chinese_literature_api_protocol.md

@@ -156,7 +156,7 @@ Pacing is per-instance and uses `time.monotonic` (NTP/manual clock adjustments c
 
 ## Chinese title matching
 
-The shared `_text_similarity.exact_normalized_title` (#431 exact-title-or-bust) is ASCII-centric. Measured 2026-07-27 against the real ISTIC title above, it **rejects three legitimate spellings of the identical title**:
+The shared `_text_similarity.exact_normalized_title` (#431 exact-title-or-bust) **was** ASCII-centric. Measured 2026-07-27 against the real ISTIC title above, it **rejected three legitimate spellings of the identical title**:
 
 | variant | shared `_similarity` | shared `exact_normalized_title` |
 |---|---|---|
@@ -166,6 +166,8 @@ The shared `_text_similarity.exact_normalized_title` (#431 exact-title-or-bust)
 | interior spaces around latin runs | 0.929 | ❌ false |
 | a genuinely **different** paper on a related topic | **0.510** | ❌ false |
 
+> **Repaired.** `normalize_cn_title` / `has_cjk` were promoted from this client into `scripts/_text_similarity.py`, so the four index resolvers (Semantic Scholar / OpenAlex / Crossref / arXiv) now apply the same Chinese-aware rule. `exact_normalized_title` gains the CJK form as an additive third branch, and `_similarity` folds it in through the existing `max`. Both changes are gated on **both** sides carrying a Han ideograph, so behaviour off that path is unchanged; a cross-script or romanized pair still cannot match. Two resolver paths were affected: the DOI-keyed cross-check (which gates on the ratio *alone*, so a legitimate fullwidth variant at 0.625 became `DOI_MISMATCH`) and the title-fallback search (which requires ratio **and** exact equality, so the same variant fell to `unresolvable`). Under the Chinese-aware form the ratio also regains its discriminative power on this pair — 1.000 for the identical title against an unchanged 0.510 for the unrelated one. The exclusion of the fuzzy ratio from `_cn_titles_match` itself (conclusion 2 below) is unchanged.
+
 Two conclusions drive `normalize_cn_title` / `_cn_titles_match`:
 
 1. **Normalization must be Chinese-aware and conservative**: explicitly fold only fullwidth ASCII forms, collapse whitespace, remove whitespace only where it touches a Han character, and remove only conventional inert outer title wrappers (`《》`, `「」`, `『』`, `【】`, curly quotation pairs) plus terminal `。`/`.` (fullwidth `．` is folded to `.` first). Whitespace between non-CJK tokens and letter case are retained: deleting or case-folding them can collapse scientific names such as `PD L1` versus `PDL 1`, or `P53` versus `p53`. Whole-string NFKC/casefold is forbidden because it can also collapse `2²` with `22` and `Straße` with `Strasse`; deleting broad punctuation/symbol categories likewise collapses scientific titles such as `ER+` versus `ER-`, `CD4+` versus `CD4−`, and `4.5%` versus `45%`. Question/exclamation marks are retained because they can distinguish otherwise identical titles.

scripts/_text_similarity.py

@@ -67,6 +67,112 @@ def _normalize_title_acronym(s: str) -> str:
     return _normalize_title(collapsed)
 
 
+# CJK Unified Ideographs (U+4E00-U+9FFF). Extension blocks are deliberately not
+# scanned: the base block is sufficient for the applicability gate, and a
+# narrower gate errs toward the pre-repair behavior, which is the safe
+# direction. Fullwidth Latin (U+FF01-U+FF5E) is deliberately NOT in this range
+# either: a title of only fullwidth Latin is not a Chinese title and must not
+# enter the CJK comparison path.
+_CJK_LO = "一"
+_CJK_HI = "鿿"
+
+_CN_WRAPPERS = {
+    ("《", "》"), ("「", "」"), ("『", "』"), ("【", "】"),
+    ("“", "”"), ("‘", "’"),
+}
+# The fullwidth-ASCII fold maps `．` to `.` first. `?`/`？` are kept: they can
+# distinguish an interrogative title from an otherwise identical one.
+_CN_TERMINAL_MARKS = "。."
+_CN_HAN_ADJACENT_SPACE = re.compile(rf"(?<=[{_CJK_LO}-{_CJK_HI}]) +| +(?=[{_CJK_LO}-{_CJK_HI}])")
+
+
+def has_cjk(text: str | None) -> bool:
+    """True iff the string contains a CJK Unified Ideograph."""
+    return any(_CJK_LO <= ch <= _CJK_HI for ch in text or "")
+
+
+def normalize_cn_title(title: str | None) -> str:
+    """Chinese-aware title normalization (#431 CJK repair).
+
+    The base `_normalize_title` is ASCII-centric and, measured on real ISTIC
+    metadata 2026-07-27, rejects four legitimate variants of one identical
+    Chinese title:
+
+      - fullwidth latin/digits (ＰｒｏＥＸＣ vs ProEXC): `.lower()` folds case
+        but never width, so U+FF30 never reaches U+0050 (measured similarity
+        0.625, exact=False)
+      - a trailing CJK full stop (。) and outer wrappers (《》): not members of
+        `string.punctuation`, so the base form keeps them (0.970, False)
+      - spaces touching Han characters: Chinese carries no word breaks, so these
+        are typesetting noise; whitespace between non-CJK tokens stays
+        significant (0.941, False)
+      - the ideographic space U+3000, likewise absent from `string.punctuation`
+
+    Simplified/Traditional folding is deliberately NOT done: it is lossy for
+    proper nouns, and a wrong fold would manufacture a false match. The pair is
+    surfaced to the human instead.
+
+    Behaviorally equivalent to the implementation this was promoted from in
+    `chinese_literature_client.py`, which re-imports it from here rather than
+    keeping a second copy (#128 anti-drift). Not byte-identical: the promotion
+    hoists the wrapper/terminal-mark sets to module constants, precompiles the
+    Han-adjacent-space regex, and rewrites the comments. Equivalence of
+    *behavior* is what the tests pin, on both the CJK path and — through the
+    pre-fix oracles in `test_text_similarity.py` — every non-CJK verdict and
+    ratio.
+    """
+    # Fold only the fullwidth ASCII compatibility block that was observed in the
+    # motivating metadata. Whole-string NFKC/casefold is too broad for an exact
+    # scientific-title key: it collapses e.g. 2² with 22 and Straße with
+    # Strasse. U+3000 is the fullwidth/ideographic space and is removed below.
+    text = "".join(
+        chr(ord(ch) - 0xFEE0) if "！" <= ch <= "～" else " " if ch == "　" else ch
+        for ch in (title or "")
+    ).strip()
+
+    # Only remove wrappers and terminal marks that are demonstrated typesetting
+    # noise. Scientific operators and measurements remain byte-significant:
+    # ER+ != ER-, CD4+ != CD4−, and 4.5% != 45%. In particular, do not return to
+    # a Unicode-category-wide P*/S* deletion rule.
+    while text:
+        previous = text
+        text = text.rstrip(_CN_TERMINAL_MARKS).strip()
+        if len(text) >= 2 and (text[0], text[-1]) in _CN_WRAPPERS:
+            text = text[1:-1].strip()
+        if text == previous:
+            break
+
+    # Whitespace touching a Han character is ordinary Chinese typesetting noise,
+    # including spaces around an embedded Latin abbreviation. Preserve
+    # whitespace *between* non-CJK tokens, where deleting it can collapse
+    # scientifically distinct names (for example `PD L1` versus `PDL 1`). Case is
+    # likewise retained: gene/protein symbols can be case-sensitive.
+    text = re.sub(r"\s+", " ", text).strip()
+    return _CN_HAN_ADJACENT_SPACE.sub("", text)
+
+
+def _cjk_titles_match(a: str, b: str) -> bool:
+    """Exact equality after Chinese-aware normalization, for two titles that
+    BOTH carry a Han ideograph.
+
+    The fuzzy ratio is excluded from this rule in both directions. It is not
+    *sufficient*: Han characters give unrelated papers a high baseline overlap
+    (two genuinely different cervical-cancer papers measured 0.510). And it is
+    not safe as an extra *necessary* condition either — the fullwidth spelling
+    of an identical title measured 0.625, below the 0.70 floor, so ANDing the
+    ratio in would veto a match exact normalization had correctly established
+    and file a real paper at P0 next to the word "fabricated".
+
+    An empty normalized key never matches, mirroring `_cn_titles_match`'s
+    non-empty guard: `《》` and `「」` both normalize to "" and are not the
+    same work.
+    """
+    if not (has_cjk(a) and has_cjk(b)):
+        return False
+    left, right = normalize_cn_title(a), normalize_cn_title(b)
+    return bool(left) and left == right
+
+
 def _similarity(a: str, b: str) -> float:
     """`max` over the base and dotted-acronym normalizations (#431 §0.1,
     F4 non-destructive): the acronym pre-pass can only ever *raise* the score,
@@ -75,6 +181,13 @@ def _similarity(a: str, b: str) -> float:
     the second pass is skipped and the result is the pre-#431 single-form ratio."""
     a_base, b_base = _normalize_title(a), _normalize_title(b)
     base = SequenceMatcher(None, a_base, b_base).ratio()
+    # CJK repair: the DOI-keyed cross-check in every resolver gates on this
+    # ratio ALONE, and a legitimate fullwidth spelling of an identical Chinese
+    # title measures 0.625 — under the 0.70 floor — turning a correct DOI into
+    # DOI_MISMATCH. Folded in via `max` (same F4 non-destructive shape as the
+    # acronym pre-pass), so this can only ever raise a score, never lower one.
+    if _cjk_titles_match(a, b):
+        return 1.0
     a_acr, b_acr = _normalize_title_acronym(a), _normalize_title_acronym(b)
     if a_acr == a_base and b_acr == b_base:  # no dotted run in either title
         return base
@@ -98,6 +211,11 @@ def exact_normalized_title(a: str, b: str) -> bool:
     return (
         _normalize_title(a) == _normalize_title(b)
         or _normalize_title_acronym(a) == _normalize_title_acronym(b)
+        # CJK repair: additive third form, gated on BOTH sides carrying a Han
+        # ideograph. A Latin-only or romanized shadow title is never comparable
+        # this way — there is no translation oracle here, so a cross-script
+        # difference must stay a non-match rather than become evidence.
+        or _cjk_titles_match(a, b)
     )
 
 

scripts/chinese_literature_client.py

@@ -72,10 +72,13 @@ from enum import Enum
 from typing import Any
 
 # Dual-path import: see openalex_client.py comment.
+# `has_cjk` / `normalize_cn_title` were promoted into the shared module so the
+# four index resolvers can apply the same Chinese-aware rule; they are re-
+# imported here so the two call sites cannot drift (the #128 extraction goal).
 try:
-    from _text_similarity import _MAX_RETRIES
+    from _text_similarity import _MAX_RETRIES, has_cjk, normalize_cn_title
 except ImportError:  # pragma: no cover - exercised by the package-import path
-    from scripts._text_similarity import _MAX_RETRIES
+    from scripts._text_similarity import _MAX_RETRIES, has_cjk, normalize_cn_title
 
 
 _DOI_RA_BASE = "https://doi.org/doiRA/"
@@ -107,12 +110,6 @@ _EUTILS_MIN_INTERVAL = 0.34
 # anonymous pacing the sibling index clients use.
 _DOI_MIN_INTERVAL = 0.2
 
-# CJK Unified Ideographs (U+4E00-U+9FFF). Extension blocks are deliberately not
-# scanned: the base block is sufficient for the applicability gate, and a
-# narrower gate errs toward `skipped`, which is the safe direction.
-_CJK_LO = "一"
-_CJK_HI = "鿿"
-
 # Registration agencies whose DOIs this resolver claims. A Crossref-registered
 # Chinese DOI is left to the existing crossref resolver: re-querying it here
 # would burn quota and amplify the Chinese fuzzy-match false positives the
@@ -455,76 +452,6 @@ class DoiTitleLookupOutcome:
     record: dict[str, Any] | None = None
 
 
-def has_cjk(text: str | None) -> bool:
-    """True iff the string contains a CJK Unified Ideograph."""
-    return any(_CJK_LO <= ch <= _CJK_HI for ch in text or "")
-
-
-def normalize_cn_title(title: str | None) -> str:
-    """Chinese-aware title normalization.
-
-    The shared `_text_similarity.exact_normalized_title` (#431) is ASCII-centric
-    and, measured on real ISTIC metadata 2026-07-27, rejects three legitimate
-    variants of one identical Chinese title:
-
-      - fullwidth latin/digits (ＰｒｏＥＸＣ vs ProEXC): an explicit fullwidth-
-        ASCII fold handles these; the shared normalizer does not (measured
-        similarity 0.577, exact=False)
-      - a trailing CJK full stop (。) and outer title wrappers (《》): not in
-        `string.punctuation`, so the shared normalizer keeps them (0.981, False)
-      - spaces touching Han characters: Chinese carries no word breaks, so
-        these are typesetting noise; whitespace between non-CJK tokens remains
-        significant (the measured Chinese/Latin variant scored 0.929, False)
-
-    Simplified/Traditional folding is deliberately NOT done: it is lossy for
-    proper nouns, and a wrong fold would manufacture a false match. The pair is
-    surfaced to the human instead.
-    """
-    # Fold only the fullwidth ASCII compatibility block that was observed in
-    # the motivating metadata. Whole-string NFKC/casefold is too broad for an
-    # exact scientific-title key: it collapses e.g. 2² with 22 and Straße with
-    # Strasse. U+3000 is the fullwidth/ideographic space and is removed below.
-    text = "".join(
-        chr(ord(ch) - 0xFEE0) if "！" <= ch <= "～" else " " if ch == "　" else ch
-        for ch in (title or "")
-    ).strip()
-
-    # Only remove wrappers and terminal marks that are demonstrated typesetting
-    # noise. Scientific operators and measurements remain byte-significant:
-    # ER+ != ER-, CD4+ != CD4−, and 4.5% != 45%. In particular, do not return to
-    # a Unicode-category-wide P*/S* deletion rule.
-    wrappers = {
-        ("《", "》"), ("「", "」"), ("『", "』"), ("【", "】"),
-        ("“", "”"), ("‘", "’"),
-    }
-    # The fullwidth-ASCII fold above has already mapped `．` to `.`. A final
-    # full stop is ordinary title punctuation; keep `?`/`？` because it can
-    # distinguish an interrogative title from an otherwise identical one.
-    terminal_marks = "。."
-    while text:
-        previous = text
-        text = text.rstrip(terminal_marks).strip()
-        if len(text) >= 2 and (text[0], text[-1]) in wrappers:
-            text = text[1:-1].strip()
-        if text == previous:
-            break
-
-    # Whitespace touching a Han character is ordinary Chinese typesetting
-    # noise, including spaces around an embedded Latin abbreviation. Preserve
-    # whitespace *between* non-CJK tokens, where deleting it can collapse
-    # scientifically distinct names (for example `PD L1` versus `PDL 1`).
-    # Case is likewise retained: gene/protein symbols can be case-sensitive,
-    # and the measured fullwidth variant already folds to the same case without
-    # requiring a broad lowercase transform.
-    text = re.sub(r"\s+", " ", text).strip()
-    text = re.sub(
-        rf"(?<=[{_CJK_LO}-{_CJK_HI}]) +| +(?=[{_CJK_LO}-{_CJK_HI}])",
-        "",
-        text,
-    )
-    return text
-
-
 def _cn_titles_match(candidate: str | None, expected: str | None) -> bool:
     """Chinese-aware exact-title-or-bust (#431 discipline).
 

scripts/test_chinese_literature_client.py

@@ -178,23 +178,112 @@ def test_fuzzy_similarity_alone_never_promotes_a_different_paper():
     assert _cn_titles_match(b, a) is False
 
 
-def test_legitimate_variants_match_despite_a_sub_threshold_fuzzy_ratio():
+def _forbid_similarity(patcher):
+    """Install a detonator over `_similarity` so any read of the fuzzy ratio
+    raises instead of quietly returning a passing verdict.
+
+    Patched on BOTH binding paths, because they fail differently: the
+    `_text_similarity` module attribute catches a qualified `ts._similarity(...)`
+    call or a lazy in-function import, while `chinese_literature_client`'s own
+    namespace catches a module-level `from _text_similarity import _similarity`
+    — already bound at import time, so it would never see the first patch.
+    Both styles are verified to trip this.
+    """
+    import _text_similarity
+    import chinese_literature_client
+
+    def _detonate(*args, **kwargs):
+        raise AssertionError(
+            "_cn_titles_match consulted the fuzzy ratio; #431 excludes it in "
+            "both directions (not sufficient, and not safe as a necessary "
+            "condition either)"
+        )
+
+    patcher.setattr(_text_similarity, "_similarity", _detonate)
+    patcher.setattr(
+        chinese_literature_client, "_similarity", _detonate, raising=False
+    )
+
+
+def test_legitimate_variants_match_despite_a_sub_threshold_fuzzy_ratio(monkeypatch):
     """Regression for a live-smoke near-miss: the shared 0.70 ratio scores a
     legitimate FULLWIDTH spelling of the identical title at 0.577, so ANDing it
     in as an extra necessary condition would veto a correct match and file a
     real paper at P0 next to the word 'fabricated'. On CJK the ratio separates
     almost nothing (0.510 unrelated vs 0.577 identical), so it is excluded."""
-    from _text_similarity import _TITLE_SIMILARITY_THRESHOLD, _similarity
+    from difflib import SequenceMatcher
+
+    from _text_similarity import (
+        _TITLE_SIMILARITY_THRESHOLD,
+        _normalize_title,
+        _similarity,
+    )
     from chinese_literature_client import _cn_titles_match
 
     canonical = "宫颈腺癌中ProEXC和PRMT5的表达及其临床意义"
     fullwidth = "宫颈腺癌中ＰｒｏＥＸＣ和ＰＲＭＴ５的表达及其临床意义。"
+    unrelated = "宫颈癌及癌前病变组织hTERC基因表达及其临床意义"
 
-    # The premise: the shared ratio really is below the floor for this pair.
-    assert _similarity(canonical, fullwidth) < _TITLE_SIMILARITY_THRESHOLD
-    # ...and the Chinese-aware rule still matches it.
+    # The premise, measured on the ASCII-centric BASE normalization that
+    # motivated excluding the ratio: an identical title scores 0.566 while an
+    # unrelated paper scores 0.510 — the floor separates almost nothing, and
+    # ANDing it in would veto a correct match.
+    base_ratio = lambda x, y: SequenceMatcher(  # noqa: E731
+        None, _normalize_title(x), _normalize_title(y)
+    ).ratio()
+    assert base_ratio(canonical, fullwidth) < _TITLE_SIMILARITY_THRESHOLD
+    assert base_ratio(canonical, fullwidth) - base_ratio(canonical, unrelated) < 0.10
+
+    # Since the CJK repair, the shared helper no longer CONTRADICTS that rule:
+    # the Chinese-aware normalization is folded into `_similarity` as a third
+    # form, so an identical title reaches 1.0 while the unrelated pair stays at
+    # its 0.510 baseline. It simply stopped being a landmine for the four index
+    # resolvers, whose DOI cross-check gates on it alone.
+    assert _similarity(canonical, fullwidth) == 1.0
+    assert _similarity(canonical, unrelated) < _TITLE_SIMILARITY_THRESHOLD
+
+    # ...and the Chinese-aware rule still matches the pair — proven with the
+    # ratio FORCED to fail, not merely observed while it happened to agree.
+    # That distinction is the whole point here: the repair lifts this pair to
+    # 1.0, so a regression that ANDed the ratio back in as a necessary
+    # condition would sail past an assertion that just called
+    # `_cn_titles_match` and looked at the answer. Scoped to a context so the
+    # measurements above still see the real `_similarity`.
+    with monkeypatch.context() as patcher:
+        _forbid_similarity(patcher)
+        assert _cn_titles_match(fullwidth, canonical) is True
+        assert _cn_titles_match(canonical, fullwidth) is True
+
+
+def test_cn_titles_match_never_consults_the_fuzzy_ratio(monkeypatch):
+    """The ratio-independence invariant in both directions, on one helper.
+
+    The sibling test above proves the POSITIVE half on the pair that motivated
+    the rule. This one adds the negative half under the same forced conditions,
+    so the invariant cannot be satisfied by a helper that has simply stopped
+    discriminating — `_cn_titles_match` must still reject a genuinely different
+    paper without the ratio existing at all.
+
+    #431 excludes the ratio in both directions: it is not sufficient (two
+    unrelated Chinese papers measure 0.510 on Han overlap alone) and not safe
+    as an extra necessary condition either (a legitimate fullwidth spelling of
+    an identical title measures below the 0.70 floor, so ANDing it in would
+    veto a correct match and file a real paper at P0 next to the word
+    "fabricated")."""
+    from chinese_literature_client import _cn_titles_match
+
+    _forbid_similarity(monkeypatch)
+
+    canonical = "宫颈腺癌中ProEXC和PRMT5的表达及其临床意义"
+    fullwidth = "宫颈腺癌中ＰｒｏＥＸＣ和ＰＲＭＴ５的表达及其临床意义。"
+    unrelated = "宫颈癌及癌前病变组织hTERC基因表达及其临床意义"
+
+    # The positive verdict must survive without the ratio existing at all...
     assert _cn_titles_match(fullwidth, canonical) is True
     assert _cn_titles_match(canonical, fullwidth) is True
+    # ...and so must the negative one, so this cannot be satisfied by a helper
+    # that has stopped discriminating.
+    assert _cn_titles_match(unrelated, canonical) is False
 
 
 def test_doi_keyed_exact_generic_title_can_match():

scripts/test_crossref_client.py

@@ -391,3 +391,95 @@ def test_refusal_message_never_carries_mailto(monkeypatch):
             client.title_search("anything")
 
     assert "secret@example.com" not in str(excinfo.value)
+
+
+# ---------------------------------------------------------------------------
+# CJK title matching (#431 repair) — integration through the real client.
+#
+# The shared helpers were ASCII-centric, so a Chinese title that an index serves
+# in a different-but-legitimate typesetting reduced to DOI_MISMATCH (DOI path)
+# or unresolvable (title path) — the same state a fabricated citation produces.
+# ---------------------------------------------------------------------------
+
+_CN_CITED = "基于ProEXC的宫颈癌筛查研究"
+_CN_AS_INDEXED = "基于ＰｒｏＥＸＣ的宫颈癌筛查研究。"  # fullwidth latin + terminal 。
+_CN_UNRELATED = "基于液基细胞学的宫颈癌筛查研究"
+
+
+def _crossref_response(payload):
+    mock_response = MagicMock()
+    mock_response.read.return_value = json.dumps(payload).encode("utf-8")
+    mock_response.__enter__ = MagicMock(return_value=mock_response)
+    mock_response.__exit__ = MagicMock(return_value=None)
+    return mock_response
+
+
+def test_doi_lookup_accepts_legitimate_cjk_typesetting_variant():
+    """A correct DOI whose indexed title uses fullwidth Latin must NOT reduce to
+    DOI_MISMATCH. This path gates on the ratio alone, which measured 0.625 for
+    this pair before the repair."""
+    from crossref_client import CrossrefClient
+
+    response = _crossref_response({"message": {"title": [_CN_AS_INDEXED], "DOI": "10.1000/cn.1"}})
+    with patch("urllib.request.urlopen", return_value=response):
+        result = CrossrefClient().doi_lookup_with_title_check("10.1000/cn.1", _CN_CITED)
+
+    assert result is not None, "correct DOI for a real Chinese paper reported as DOI_MISMATCH"
+    assert result["DOI"] == "10.1000/cn.1"
+
+
+def test_doi_lookup_still_rejects_a_different_cjk_paper():
+    """The guard: Han overlap alone (0.510 measured) must never promote a
+    genuinely different paper."""
+    from crossref_client import CrossrefClient
+
+    response = _crossref_response({"message": {"title": [_CN_UNRELATED], "DOI": "10.1000/cn.2"}})
+    with patch("urllib.request.urlopen", return_value=response):
+        result = CrossrefClient().doi_lookup_with_title_check("10.1000/cn.2", _CN_CITED)
+
+    assert result is None
+
+
+def test_title_search_matches_legitimate_cjk_typesetting_variant():
+    """The title-fallback path requires ratio AND exact-normalized equality;
+    before the repair a legitimate variant failed both and fell to
+    `unresolvable`."""
+    from crossref_client import CrossrefClient
+
+    response = _crossref_response(
+        {"message": {"items": [{"title": [_CN_AS_INDEXED], "DOI": "10.1000/cn.1"}]}}
+    )
+    with patch("urllib.request.urlopen", return_value=response):
+        result = CrossrefClient().title_search(_CN_CITED)
+
+    assert result is not None, "legitimate CJK variant fell through to unresolvable"
+    assert result["DOI"] == "10.1000/cn.1"
+
+
+def test_title_search_still_rejects_a_different_cjk_paper():
+    from crossref_client import CrossrefClient
+
+    response = _crossref_response(
+        {"message": {"items": [{"title": [_CN_UNRELATED], "DOI": "10.1000/cn.2"}]}}
+    )
+    with patch("urllib.request.urlopen", return_value=response):
+        result = CrossrefClient().title_search(_CN_CITED)
+
+    assert result is None
+
+
+def test_title_search_does_not_match_a_cross_script_shadow_title():
+    """An English shadow title is not chimeric-citation evidence and is not a
+    match either: there is no translation oracle in this client."""
+    from crossref_client import CrossrefClient
+
+    response = _crossref_response(
+        {"message": {"items": [{
+            "title": ["ProEXC-based cervical cancer screening"],
+            "DOI": "10.1000/cn.3",
+        }]}}
+    )
+    with patch("urllib.request.urlopen", return_value=response):
+        result = CrossrefClient().title_search(_CN_CITED)
+
+    assert result is None

scripts/test_text_similarity.py

@@ -14,6 +14,7 @@ from __future__ import annotations
 import os
 import sys
 import unittest
+from difflib import SequenceMatcher
 
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 
@@ -67,6 +68,296 @@ class SimilarityTest(unittest.TestCase):
         self.assertLess(ts._similarity("alpha beta gamma", "xyz qrs uvw"), 0.3)
 
 
+#: The motivating ISTIC title, measured 2026-07-27 (see
+#: `deep-research/references/chinese_literature_api_protocol.md` §"Chinese title
+#: matching"). Every entry in `_CN_LEGITIMATE_VARIANTS` is the SAME work; the
+#: shared ASCII-centric normalizer rejected all four before this fix.
+_CN_TITLE = "基于ProEXC的宫颈癌筛查研究"
+_CN_LEGITIMATE_VARIANTS = {
+    "fullwidth_latin": "基于ＰｒｏＥＸＣ的宫颈癌筛查研究",
+    "terminal_cjk_stop": "基于ProEXC的宫颈癌筛查研究。",
+    "spaces_touching_han": "基于 ProEXC 的宫颈癌筛查研究",
+    "book_title_wrapper": "《基于ProEXC的宫颈癌筛查研究》",
+    "ideographic_space": "基于ProEXC的宫颈癌筛查研究　",
+}
+
+
+class CjkExactTitleTest(unittest.TestCase):
+    """A CJK title must survive the four legitimate typesetting variants that
+    Chinese indexes actually serve.
+
+    Before this fix the shared helpers were ASCII-centric: `.lower()` folds case
+    but not width (Ｐ U+FF30 never reaches P U+0050), and `string.punctuation`
+    does not contain `。`, `《》`, or U+3000. A real paper therefore reduced to
+    `unresolvable` / `DOI_MISMATCH` — the same state a fabricated citation
+    produces.
+    """
+
+    def test_legitimate_variants_are_exact_matches(self) -> None:
+        for name, variant in _CN_LEGITIMATE_VARIANTS.items():
+            with self.subTest(variant=name):
+                self.assertTrue(
+                    ts.exact_normalized_title(_CN_TITLE, variant),
+                    f"{name}: legitimate CJK variant rejected as a non-match",
+                )
+
+    def test_exact_match_is_symmetric(self) -> None:
+        for name, variant in _CN_LEGITIMATE_VARIANTS.items():
+            with self.subTest(variant=name):
+                self.assertEqual(
+                    ts.exact_normalized_title(_CN_TITLE, variant),
+                    ts.exact_normalized_title(variant, _CN_TITLE),
+                )
+
+    def test_legitimate_variants_clear_the_ratio_threshold(self) -> None:
+        """The DOI-keyed path (e.g. `crossref_client.doi_lookup`) gates on the
+        ratio ALONE, so exact-match repair is not enough on its own. The
+        fullwidth variant measured 0.625 — below the 0.70 floor — which turned a
+        correct DOI into `DOI_MISMATCH`."""
+        for name, variant in _CN_LEGITIMATE_VARIANTS.items():
+            with self.subTest(variant=name):
+                self.assertGreaterEqual(
+                    ts._similarity(_CN_TITLE, variant),
+                    ts._TITLE_SIMILARITY_THRESHOLD,
+                    f"{name}: legitimate CJK variant fails the DOI cross-check",
+                )
+
+    def test_distinct_cjk_papers_still_rejected(self) -> None:
+        """The measured guard: two genuinely different papers on a related topic
+        scored 0.510 on Han overlap alone. Han characters give unrelated CJK
+        titles a high baseline, so the fix must not convert overlap into a
+        match."""
+        other = "基于液基细胞学的宫颈癌筛查研究"
+        self.assertFalse(ts.exact_normalized_title(_CN_TITLE, other))
+
+    def test_true_match_outranks_an_unrelated_paper(self) -> None:
+        """Before the repair the ratio was actively ANTI-correlated on this
+        pair: the identical title scored 0.606 while a genuinely different paper
+        scored 0.645 — the wrong paper ranked higher. Han overlap dominates the
+        ratio once fullwidth codepoints break the true match apart.
+
+        The repair does not touch the unrelated pair's score (it is not equal
+        under the CJK form, so nothing is folded in); it lifts the true match to
+        1.0, restoring the ordering that title ranking depends on."""
+        unrelated = "基于液基细胞学的宫颈癌筛查研究"
+        true_match = ts._similarity(_CN_TITLE, _CN_LEGITIMATE_VARIANTS["fullwidth_latin"])
+        self.assertGreater(true_match, ts._similarity(_CN_TITLE, unrelated))
+
+    def test_simplified_traditional_not_folded(self) -> None:
+        """Deliberately NOT folded: the fold is lossy for proper nouns and a
+        wrong fold would manufacture a false match. The variant pair is
+        surfaced to the human instead."""
+        self.assertFalse(
+            ts.exact_normalized_title("宫颈癌筛查研究", "宮頸癌篩查研究")
+        )
+
+
+def _pre_fix_exact_normalized_title(a: str, b: str) -> bool:
+    """The #431 formula exactly as it stood before the CJK repair.
+
+    Used as an oracle: the repair is only allowed to change the verdict for a
+    pair where BOTH sides carry a Han ideograph. Everywhere else it must agree
+    with this function byte-for-byte. Written out in full (rather than captured
+    from the module) so the oracle cannot drift with the code it checks."""
+    return (
+        ts._normalize_title(a) == ts._normalize_title(b)
+        or ts._normalize_title_acronym(a) == ts._normalize_title_acronym(b)
+    )
+
+
+def _pre_fix_similarity(a: str, b: str) -> float:
+    """`_similarity` exactly as it stood before the CJK repair.
+
+    The companion oracle to `_pre_fix_exact_normalized_title`, and written out
+    in full for the same reason: captured from the module it checks, it would
+    drift with it. Includes the dotted-acronym branch, so this is the whole
+    pre-fix formula rather than the base ratio alone — off the CJK path the
+    repair must reproduce it exactly, in BOTH directions."""
+    a_base, b_base = ts._normalize_title(a), ts._normalize_title(b)
+    base = SequenceMatcher(None, a_base, b_base).ratio()
+    a_acr, b_acr = ts._normalize_title_acronym(a), ts._normalize_title_acronym(b)
+    if a_acr == a_base and b_acr == b_base:  # no dotted run in either title
+        return base
+    return max(base, SequenceMatcher(None, a_acr, b_acr).ratio())
+
+
+#: Non-CJK pairs spanning every branch of the pre-fix formula: case, ASCII
+#: punctuation, dotted acronyms, the `D. H.` base-form carve-out, distinct
+#: related works, and the empty/whitespace degenerate cases.
+_NON_CJK_PAIRS = [
+    ("Attention Is All You Need", "attention is all you need"),
+    ("D.H. Lawrence and the Novel", "D. H. Lawrence and the Novel"),
+    ("R.A.G. for Question Answering", "RAG for Question Answering"),
+    ("Foo: A Study", "Foo — A Study"),
+    ("Deep Learning, Part I", "Deep Learning, Part II"),
+    ("A Study of Foo", "A Study of Bar"),
+    ("Correction to: A Study of Foo", "A Study of Foo"),
+    ("A/B testing", "A B testing"),
+    ("R&D strategy", "R D strategy"),
+    ("", ""),
+    ("   ", ""),
+    ("　", ""),
+    ("。", "《》"),
+    ("ＰｒｏＥＸＣ assay", "ProEXC assay"),
+]
+
+
+class CjkNonDestructiveTest(unittest.TestCase):
+    """The safety property: the repair may only change the verdict for a pair
+    where BOTH sides carry a Han ideograph. Every other pair must agree with
+    the pre-fix formula exactly."""
+
+    def test_agrees_with_pre_fix_formula_off_the_cjk_path(self) -> None:
+        for left, right in _NON_CJK_PAIRS:
+            with self.subTest(pair=(left, right)):
+                self.assertEqual(
+                    ts.exact_normalized_title(left, right),
+                    _pre_fix_exact_normalized_title(left, right),
+                    "repair changed a verdict outside the both-sides-CJK path",
+                )
+
+    def test_ratio_unchanged_off_the_cjk_path(self) -> None:
+        """Off the CJK path `_similarity` must not move AT ALL — asserted as
+        exact equality against the full pre-fix formula, not as a lower bound.
+
+        A `>=` assertion against the base ratio would pass a regression that
+        *raised* a non-CJK score (0.6 → 1.0 is still `>= base`), and would also
+        miss the dotted-acronym branch entirely. Both directions are pinned:
+        the repair may neither lower nor raise a score outside the
+        both-sides-CJK gate."""
+        for left, right in _NON_CJK_PAIRS:
+            with self.subTest(pair=(left, right)):
+                self.assertEqual(
+                    ts._similarity(left, right),
+                    _pre_fix_similarity(left, right),
+                    "repair moved a ratio outside the both-sides-CJK path",
+                )
+
+    def test_mixed_script_pair_is_not_a_match(self) -> None:
+        """A Latin-only shadow title is not comparable to a Chinese title: there
+        is no translation oracle, so the difference must stay a non-match rather
+        than become chimeric-citation evidence."""
+        self.assertFalse(
+            ts.exact_normalized_title(_CN_TITLE, "ProEXC-based cervical cancer screening")
+        )
+
+    def test_scientific_names_not_collapsed(self) -> None:
+        """The CJK path must not collapse scientifically distinct titles.
+
+        Only pairs the pre-fix formula already separated are asserted here.
+        `ER+`/`ER-` and `p53`/`P53` are NOT listed: the base ASCII normalization
+        maps ASCII punctuation to whitespace and lowercases, so it already
+        collapses those two pairs, and `exact_normalized_title` ORs that form
+        in. That is a pre-existing behavior this additive repair neither causes
+        nor removes — see `test_preserves_case` /
+        `test_preserves_scientific_operators` for the CJK normalizer's own
+        (stricter) behavior in isolation."""
+        for left, right in [
+            ("基于PD L1的研究", "基于PDL 1的研究"),
+            ("基于4.5%的研究", "基于45%的研究"),
+        ]:
+            with self.subTest(pair=(left, right)):
+                self.assertFalse(ts.exact_normalized_title(left, right))
+
+    def test_cjk_repair_adds_no_empty_string_match(self) -> None:
+        """Two blank-normalizing CJK-free titles must not become a match via the
+        new path. (`("", "")` is True under the pre-fix formula and stays True —
+        that is pinned by the oracle test above, not changed here.)"""
+        for left, right in [("。", "《》"), ("　", " ")]:
+            with self.subTest(pair=(left, right)):
+                self.assertFalse(ts.has_cjk(left) and ts.has_cjk(right))
+
+    def test_cjk_titles_normalizing_to_empty_never_match(self) -> None:
+        """The CJK path itself refuses an empty normalized key, mirroring
+        `_cn_titles_match`'s non-empty guard."""
+        self.assertEqual(ts.normalize_cn_title("《》"), "")
+        self.assertFalse(ts.exact_normalized_title("《》。", "「」。"))
+
+
+class NormalizeCnTitleTest(unittest.TestCase):
+    """Unit-level behavior of the CJK-aware normalizer itself."""
+
+    def test_folds_fullwidth_ascii_to_halfwidth(self) -> None:
+        self.assertEqual(ts.normalize_cn_title("ＰｒｏＥＸＣ"), "ProEXC")
+
+    def test_strips_terminal_cjk_full_stop(self) -> None:
+        self.assertEqual(ts.normalize_cn_title("研究。"), "研究")
+
+    def test_strips_outer_title_wrappers(self) -> None:
+        for wrapped in ("《研究》", "「研究」", "『研究』", "【研究】"):
+            with self.subTest(wrapped=wrapped):
+                self.assertEqual(ts.normalize_cn_title(wrapped), "研究")
+
+    def test_removes_whitespace_touching_han(self) -> None:
+        self.assertEqual(ts.normalize_cn_title("基于 ProEXC 的研究"), "基于ProEXC的研究")
+
+    def test_preserves_whitespace_between_latin_tokens(self) -> None:
+        self.assertEqual(ts.normalize_cn_title("PD L1"), "PD L1")
+
+    def test_preserves_case(self) -> None:
+        self.assertEqual(ts.normalize_cn_title("p53"), "p53")
+
+    def test_preserves_scientific_operators(self) -> None:
+        """Unlike the base ASCII normalization, the CJK normalizer keeps `+`,
+        `-`, `.` and `%` byte-significant: ER+ != ER-, 4.5% != 45%."""
+        self.assertNotEqual(ts.normalize_cn_title("ER+"), ts.normalize_cn_title("ER-"))
+        self.assertNotEqual(
+            ts.normalize_cn_title("4.5%"), ts.normalize_cn_title("45%")
+        )
+
+    def test_handles_none(self) -> None:
+        self.assertEqual(ts.normalize_cn_title(None), "")
+
+
+class HasCjkTest(unittest.TestCase):
+    def test_detects_han_ideograph(self) -> None:
+        self.assertTrue(ts.has_cjk("基于ProEXC"))
+
+    def test_latin_only_is_not_cjk(self) -> None:
+        self.assertFalse(ts.has_cjk("ProEXC"))
+
+    def test_fullwidth_latin_alone_is_not_cjk(self) -> None:
+        """Fullwidth Latin is not a Han ideograph — a title of only fullwidth
+        Latin must not enter the CJK path."""
+        self.assertFalse(ts.has_cjk("ＰｒｏＥＸＣ"))
+
+    def test_none_and_empty(self) -> None:
+        self.assertFalse(ts.has_cjk(None))
+        self.assertFalse(ts.has_cjk(""))
+
+
+class SharedHelperReachesEveryResolverTest(unittest.TestCase):
+    """The repair is only worth anything if all four index resolvers pick it up.
+
+    Preventing sibling drift is the stated reason `_text_similarity` exists
+    (#128), so pin that each client resolves the SAME function object rather
+    than a re-implementation that could silently diverge again."""
+
+    CLIENTS = (
+        "semantic_scholar_client",
+        "openalex_client",
+        "crossref_client",
+        "arxiv_client",
+    )
+
+    def test_every_resolver_shares_the_repaired_helpers(self) -> None:
+        import importlib
+
+        for name in self.CLIENTS:
+            module = importlib.import_module(name)
+            with self.subTest(client=name):
+                self.assertIs(module._similarity, ts._similarity)
+                self.assertIs(module.exact_normalized_title, ts.exact_normalized_title)
+
+    def test_cjk_client_shares_the_promoted_normalizer(self) -> None:
+        """`normalize_cn_title` was promoted out of `chinese_literature_client`;
+        it must now re-import the shared one, not keep a private copy."""
+        import chinese_literature_client as cn
+
+        self.assertIs(cn.normalize_cn_title, ts.normalize_cn_title)
+        self.assertIs(cn.has_cjk, ts.has_cjk)
+
+
 class ConstantsTest(unittest.TestCase):
     """Lock the magic numbers — these are protocol-level invariants, not
     arbitrary tuning."""