fix: strip CJK outer wrapper marks only when they enclose one balanced unit (#800) (#804)

* fix: strip CJK outer wrapper marks only when they enclose one balanced unit (#800)

The wrapper strip inherited from #431 was positional: it removed the first
and last characters whenever they matched as a wrapper pair TYPE, without
checking they belonged to the same bracket pair. 《红楼梦》与《金瓶梅》 —
two titles joined in one string — normalized to 红楼梦》与《金瓶梅, leaving
an orphaned 》 mid-key.

Matching correctness was unaffected (both sides mangle identically; no
exploitable asymmetry found in the #799 security pass), but the mangled key
is a semantic anomaly for any future single-sided consumer.

Fix adds _outer_pair_encloses: the outer marks are stripped only when the
interior between them is itself balanced under all six wrapper pairs.
《围城》 still strips to 围城, nested balanced interiors still unwrap
(《基于「ProEXC」的研究》 → 基于「ProEXC」的研究), while 《红楼梦》与《金瓶梅》,
“研究”与“实践”, and “研究与“实践” keep their marks.

Both consumers change together — the CJK client re-imports the shared
function (#799), pinned behaviorally as well as by identity. The two new
discriminating tests are mutation-verified to fail against the pre-fix
module. Full suite: 9262 passed, 3 skipped.

* fix: scope the CJK interior balance scan to the outer pair's own family (#804 review)

Addresses the P1 and three advisories on PR #804.

P1: `’` is the closer of `‘` AND the English apostrophe; `”` likewise appears
unpaired in mixed typesetting. The family-blind interior scan read the lone `’`
in `《Alzheimer’s病中ProEXC表达》` as an unbalanced quote and refused to
strip a genuine `《…》` wrap. Verified against main: that pair went from
exact=True / ratio 1.0 to exact=False / ratio 0.6818 — below the 0.70 floor, so
the DOI-keyed ratio gate and the title-fallback exact gate failed together and a
correct DOI could be reported as a mismatch. That is the failure class #798
repaired, so it is not covered by #800's conservative-direction blessing: that
blessing is for titles whose outer marks are not one pair, and here they are.

Fix scopes the scan to the outer pair's own family — a `《…》` wrap tracks only
`《`/`》` and is blind to quote marks. This costs the check nothing it was
buying: any mark that can orphan the OUTER pair is by definition of that pair's
own family. The whole #800 behavior table survives (`《红楼梦》与《金瓶梅》`
still trips on its stray `》`, `“研究”与“实践”` on its stray `”`, nested
`《基于「ProEXC」的研究》` still unwraps, `《》` → ""), and the flagged
interaction `《「研究』》` → `「研究』` now keeps its mismatched inner quotes as
content, which is what distinguishes it from `《研究》`. Regressions added for
both apostrophe shapes (`’s`, `’98`).

The two lookup maps the family-blind scan needed are now dead and removed.

Advisory 1: the depth-0 stray-closer branch is pinned. The first attempt did
not discriminate — clamping absorbs one closer, so any title with equal
opener/closer counts (every natural case, including `《红楼梦》与《金瓶梅》`)
is refused by the trailing depth check anyway. Discriminating requires interior
closers to outnumber openers by exactly the clamp count, which no natural title
shape produces, so the test uses a documented synthetic asserted against the
helper directly. All three branches are now mutation-verified: clamp-instead-of-
refuse, drop-the-trailing-check, and family-blind (the P1 regression itself).

Advisory 2: the module docstring's "behaviorally equivalent to the
implementation this was promoted from" is qualified as historical — true of the
#798/#799 promotion, false since #800 deliberately changed this behavior.

Advisory 3: the changelog's non-CJK invariance claim is narrowed to what the
code supports. The claim holds on the two `has_cjk`-gated paths; the client's
`_cn_titles_match` is ungated, so a mark-carrying Han-free title can change
verdict there (`《Hamlet》and《Macbeth》` vs its pre-mangled form — verified to
flip). The empty-wrapper guarantee is likewise a `_cjk_titles_match` property:
`exact_normalized_title("《》", "《》")` is still True through the base branch.
Both narrowings are now pinned by tests so the prose cannot drift from the code.

Full suite: 9270 passed, 4 skipped, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Akshath Rajkumar
2026-08-27 15:25:24 +05:30
committed by GitHub
parent 9469fc4d07
commit 5debcd2efb
3 changed files with 212 additions and 9 deletions
+2
View File
@@ -8,6 +8,8 @@ All notable changes to this project will be documented in this file.
- **`check_surface_form_parity` names the broken environment instead of blaming the manifest (#801 follow-up).** With the manifest file present but pyyaml unimportable, `_load_manifest` returned `None`, so the lint failed with "manifest … is present but empty / null / non-mapping" — a misdiagnosis pointing the reader at the wrong file (pyyaml is declared in `requirements-dev.txt`, so only a broken local environment can hit this). The missing-import case is now a distinct `_YamlUnavailableError` and the lint fails naming pyyaml and the `pip install -r requirements-dev.txt` remedy; regression test pins that the message names the environment, not the manifest shape. Also de-enumerated the hand-maintained dependency parenthetical in `docs/SETUP.md` / `docs/SETUP.zh-TW.md` Method line ("PyYAML + jsonschema" was already stale against the manifest it mirrors two lines above); both language files move together.
- **`normalize_cn_title` strips outer wrapper marks only when they enclose the whole title as one balanced unit (#800).** The wrapper strip inherited from #431 (and promoted unchanged by #798/#799) was positional: it removed the first and last characters whenever they matched as a wrapper pair *type*, without checking they belonged to the same bracket pair. `《红楼梦》与《金瓶梅》` — two titles joined in one string — therefore normalized to `红楼梦》与《金瓶梅`, leaving an orphaned `》` mid-key. Matching correctness was never affected (both sides of every comparison pass through the same normalization, and no exploitable asymmetry was found in the #799 security pass), but the mangled key is a semantic anomaly for any future single-sided consumer (display, logging, cache keys, or comparison against an externally-normalized key). Fix adds `_outer_pair_encloses`: the outer marks are stripped only when the interior between them is itself balanced under all six wrapper pairs, so `《围城》` still strips to `围城` and nested balanced interiors (`《基于「ProEXC」的研究》``基于「ProEXC」的研究`) still unwrap, while `《红楼梦》与《金瓶梅》` and `“研究”与“实践”` keep their marks. The interior scan is scoped to the outer pair's **own family**: `` is also the English apostrophe and `”` also appears unpaired, so a family-blind scan read the lone `` in `《Alzheimers病中表达》` as an unbalanced quote and refused to strip a genuine `《…》` wrap — dropping a pair that matched before to exact=False and ratio 0.6818, below the 0.70 floor, which fails the DOI-keyed ratio gate and the title-fallback exact gate at once and is the failure class #798 repaired. Scoping costs the check nothing, since any mark that can orphan the outer pair is by definition of that pair's own family. Both consumers change together — the CJK client re-imports the shared function (#799), pinned behaviorally as well as by identity. Scope of the invariance claim, stated precisely: on the two `has_cjk`-gated paths (`exact_normalized_title`'s third branch and `_similarity`'s CJK fold) every verdict and ratio for titles without Han ideographs is unchanged, pinned by the #799 pre-fix oracles. The client's `_cn_titles_match` is **not** `has_cjk`-gated — it calls `normalize_cn_title` directly — so a mark-carrying title with no Han ideograph can change verdict there (`《Hamlet》and《Macbeth》` no longer matches a pre-mangled `Hamlet》and《Macbeth`); that path is DOI-keyed and Chinese-corpus-only in practice, so the narrowing is accepted rather than gated. Likewise the empty-wrapper guard is a property of the CJK branch specifically (`_cjk_titles_match` requires a non-empty key): `exact_normalized_title("《》", "《》")` remains True through the ungated base-normalization branch, as it did before this change.
- **CJK titles no longer fail the shared exact-title gate in the four index resolvers (#798).** `chinese_literature_client.py` already carried a Chinese-aware `normalize_cn_title` / `has_cjk` (#431 §"Chinese title matching"), but the four index resolvers (Semantic Scholar / OpenAlex / Crossref / arXiv) never saw it — they read the ASCII-centric `_text_similarity` helpers, where `.lower()` folds case but never width ( U+FF30 never reaches P U+0050) and `string.punctuation` contains none of `。`, `《》`, or U+3000. A Chinese paper an index served in a different-but-legitimate typesetting therefore missed on **two** paths: the DOI-keyed cross-check, which gates on the fuzzy ratio *alone* and scored a fullwidth spelling of the identical title at **0.625 — under the 0.70 floor — reporting a correct DOI as `DOI_MISMATCH`**; and the title-fallback search, which requires ratio **and** exact-normalized equality and so fell to `unresolvable`. Both feed the `*_unmatched` contamination signals, so a genuine paper could accumulate *k* across indexes and be rendered `CONTAMINATED-TRIANGULATION-UNMATCHED` — the protocol doc's own "P0, next to the word 'fabricated'". The failure was invisible to the English test corpus for a measurable reason: a Han character is a whole word, so the same six-codepoint corruption is 37.5% of a 16-character Chinese title but 8% of its 68-character English equivalent (measured 0.625 vs 0.912). Fix promotes `has_cjk` / `normalize_cn_title` into `scripts/_text_similarity.py` (behaviorally equivalent — the promotion hoists locals to module constants, precompiles the regex, and rewrites comments; the CJK client now re-imports rather than keeping a private copy, per the #128 anti-drift goal), adds the Chinese-aware form to `exact_normalized_title` as an additive third branch, and folds it into `_similarity` through the existing `max`. Both are gated on **both** sides carrying a Han ideograph, so every non-CJK verdict and every non-CJK *ratio* is provably unchanged — pinned by two oracle tests that re-state the pre-fix `exact_normalized_title` and `_similarity` formulas in full (the latter including the dotted-acronym branch) and assert exact agreement in both directions, so a raised score is caught as loudly as a lowered one. Cross-script and romanized pairs still cannot match (no translation oracle), Simplified/Traditional is still not folded, and an empty normalized key still never matches. Under the Chinese-aware form the ratio also regains discriminative power on the motivating pair: 1.000 for the identical title against an unchanged 0.510 for a genuinely different paper (the base form separated 0.566 from 0.510 — almost nothing). Also measured and pinned: on the motivating pair the pre-fix ratio was actively *anti-correlated* — the identical title scored **0.606 while a genuinely different paper scored 0.645**, so the wrong paper ranked higher. The repair leaves the unrelated pair's score byte-identical (0.6452 before and after; it is not equal under the CJK form, so nothing is folded in) and lifts only the true match, restoring the ordering title ranking depends on. Not in scope, and unchanged: the base ASCII normalization still collapses `ER+`/`ER-` and `p53`/`P53` because it maps ASCII punctuation to whitespace and lowercases, and `exact_normalized_title` ORs that form in; the fuzzy floor also remains a weak separator for CJK generally (that 0.645 near-miss is pre-existing and untouched here). 32 new tests, including 5 integration tests through the real Crossref client covering both broken paths, each verified to fail against the pre-fix module, plus a monkeypatched detonator pinning that the Chinese DOI-path matcher never consults the fuzzy ratio in either direction.
- **Autolink round-trip test no longer rides on an undeclared, version-sensitive dependency (#801).** `test_gfm_bare_urls_emails_and_schemes_cannot_autolink` soft-imported markdown-it-py — not listed in `requirements-dev.txt` — and silently returned when it was absent, so its no-`link_open` tail had never actually run in CI, while an ambient markdown-it-py 2.x (whose linkify pass still tokenizes the escaped forms as links; reproduced under 2.2.0 + linkify-it-py 2.0.3, reported by @akshath-raj in #799) failed the test on clean `main`. Verified dividing line: 2.2.0 fails, 3.0.0 and 4.0.0 pass with linkify-it-py 2.0.3 held constant. The tail is split into its own test (`test_escaped_markdown_yields_no_linkify_tokens_on_round_trip`) gated by `pytest.importorskip("markdown_it", minversion="3.0.0")` + `pytest.importorskip("linkify_it", minversion="2.0.3")` — ambient-old environments now skip visibly instead of failing or passing vacuously — and `requirements-dev.txt` declares `markdown-it-py>=3.0` + `linkify-it-py>=2.0.3` with a reverse pointer at the consuming test, so CI exercises the round-trip for the first time. The same declaration also newly activates the identical soft-import tail in `test_renderer_neutralizes_markdown_active_inventory_path` (`scripts/test_build_submission_packet_manifest.py`), which moves to the same `importorskip` idiom (no version floor: it parses default CommonMark without linkify) and gains the inline-children flatten its top-level-only token scan was missing — `link_open` / `image` / `html_inline` nest under `inline` tokens, so the pre-existing assertion could only ever catch `html_block`; its strengthened assertions verified under markdown-it-py 2.2.0, 3.0.0, and 4.0.0. The triplicated hostile-row construction in `test_evidence_rows.py` is consolidated into one `_hostile_row` helper. Renderer behavior and every renderer-side assertion are unchanged.
+61 -9
View File
@@ -80,6 +80,48 @@ _CN_WRAPPERS = {
("", ""), ("", ""), ("", ""), ("", ""),
("", ""), ("", ""),
}
def _outer_pair_encloses(text: str) -> bool:
"""#800: True iff `text` starts and ends with marks that form ONE matching
wrapper pair AND the interior is balanced *within that pair's own family* —
i.e. the outer pair encloses the whole title as a single unit.
A first/last-character pair check alone is not enough: in
`《红楼梦》与《金瓶梅》` the leading `《` and trailing `》` match as
*character types* but belong to two different bracket pairs, and stripping
them positionally leaves an orphaned `》` mid-string. Scanning the interior
for an unbalanced `》` catches exactly that case, so such titles keep their
marks. An empty interior counts as balanced, preserving the `《》` → ""
normalization.
The scan is deliberately scoped to the outer pair's OWN family (#804 review
P1). A family-blind scan is unsound because two of these marks are not
exclusively wrappers: `` is also the English apostrophe and `”` also
appears unpaired in mixed typesetting. Reading the lone `` in
`《Alzheimers病中表达》` as an unbalanced quote refused to strip
a genuine `《…》` wrap, dropping a pair that matched on main to exact=False
and ratio 0.6818 — below the 0.70 floor, so both resolver paths failed at
once and a correct DOI could be reported as a mismatch. That is the failure
class #798 repaired, so the apostrophe must be invisible to a `《》` scan.
Scoping costs nothing the check was buying: every mark that can orphan the
OUTER pair is by definition a mark of that pair's own family. Interior marks
of other families are content and survive the strip — `《「研究』》` →
`「研究』` keeps the mismatched inner quotes, which is right, since they are
what distinguishes it from `《研究》`."""
if len(text) < 2 or (text[0], text[-1]) not in _CN_WRAPPERS:
return False
opener, closer = text[0], text[-1]
depth = 0
for ch in text[1:-1]:
if ch == opener:
depth += 1
elif ch == closer:
if depth == 0: # closes the OUTER opener: not one unit
return False
depth -= 1
return depth == 0
# The fullwidth-ASCII fold maps `` to `.` first. `?`/`` are kept: they can
# distinguish an interrogative title from an otherwise identical one.
_CN_TERMINAL_MARKS = "。."
@@ -112,14 +154,21 @@ def normalize_cn_title(title: str | None) -> str:
proper nouns, and a wrong fold would manufacture a false match. The pair is
surfaced to the human instead.
Behaviorally equivalent to the implementation this was promoted from in
`chinese_literature_client.py`, which re-imports it from here rather than
keeping a second copy (#128 anti-drift). Not byte-identical: the promotion
hoists the wrapper/terminal-mark sets to module constants, precompiles the
Han-adjacent-space regex, and rewrites the comments. Equivalence of
*behavior* is what the tests pin, on both the CJK path and — through the
pre-fix oracles in `test_text_similarity.py` — every non-CJK verdict and
ratio.
Outer wrappers are stripped only when they enclose the whole title as one
balanced unit (#800): `《围城》` loses its marks, but
`《红楼梦》与《金瓶梅》` keeps them, because its first and last marks belong
to two different bracket pairs and stripping them positionally would leave
an orphaned `》` mid-string.
Promoted here from `chinese_literature_client.py`, which re-imports it
rather than keeping a second copy (#128 anti-drift). The promotion itself
(#798/#799) was behaviorally equivalent — it only hoisted the wrapper and
terminal-mark sets to module constants, precompiled the Han-adjacent-space
regex, and rewrote the comments. That equivalence is now **historical**:
#800 deliberately changed the wrapper-strip behavior described above, so
this function no longer matches the pre-#800 implementation. What the tests
pin today is the #800 behavior on the CJK path plus — through the pre-fix
oracles in `test_text_similarity.py` — every non-CJK verdict and ratio.
"""
# Fold only the fullwidth ASCII compatibility block that was observed in the
# motivating metadata. Whole-string NFKC/casefold is too broad for an exact
@@ -137,7 +186,10 @@ def normalize_cn_title(title: str | None) -> str:
while text:
previous = text
text = text.rstrip(_CN_TERMINAL_MARKS).strip()
if len(text) >= 2 and (text[0], text[-1]) in _CN_WRAPPERS:
# #800: strip the outer wrapper only when it genuinely encloses the
# whole title as one balanced unit — not merely when the first and last
# characters happen to match as pair types.
if _outer_pair_encloses(text):
text = text[1:-1].strip()
if text == previous:
break
+149
View File
@@ -199,6 +199,13 @@ _NON_CJK_PAIRS = [
(" ", ""),
("", "《》"),
(" assay", "ProEXC assay"),
# #800: wrapper marks present but no Han ideograph — the pairedness check
# must not leak through the has_cjk gate and accidentally strip wrappers
# on non-CJK titles. (If normalize_cn_title were called on these, the
# outer marks would strip and the pairs would match — exactly the
# regression this entry is pinned against.)
("《Attention Is All You Need》", "Attention Is All You Need"),
("《A Study of Foo》", "A Study of Foo"),
]
@@ -288,6 +295,148 @@ class NormalizeCnTitleTest(unittest.TestCase):
with self.subTest(wrapped=wrapped):
self.assertEqual(ts.normalize_cn_title(wrapped), "研究")
def test_strips_outer_quotation_wrappers(self) -> None:
for wrapped in ("“围城”", "‘围城’"):
with self.subTest(wrapped=wrapped):
self.assertEqual(ts.normalize_cn_title(wrapped), "围城")
def test_keeps_marks_when_outer_marks_belong_to_two_different_pairs(self) -> None:
"""#800: the first/last marks of `《红楼梦》与《金瓶梅》` match as pair
TYPES (`《`/`》`) but belong to two different brackets — one opening
each title. Positional stripping left an orphaned `》` mid-string; the
pairedness check keeps the title intact instead."""
for title in ("《红楼梦》与《金瓶梅》", "“研究”与“实践”"):
with self.subTest(title=title):
self.assertEqual(ts.normalize_cn_title(title), title)
def test_nested_interior_wrapper_survives_outer_strip(self) -> None:
"""A genuine outer pair enclosing a balanced interior still strips, and
the balanced interior marks survive as content."""
self.assertEqual(
ts.normalize_cn_title("《基于「ProEXC」的研究》"),
"基于「ProEXC」的研究",
)
def test_unclosed_opener_in_interior_keeps_marks(self) -> None:
"""An opener inside the interior that never closes proves the outer
marks are not one unit — a distinct failure mode from the stray-closer
case above (stack non-empty at the end vs. pop from empty), and also
verified to fail against the pre-fix positional strip."""
self.assertEqual(ts.normalize_cn_title("“研究与“实践”"), "“研究与“实践”")
def test_apostrophe_in_embedded_english_does_not_veto_the_outer_strip(self) -> None:
"""#804 review P1: `` is the closer of ``, but it is ALSO the
apostrophe in embedded English — the same codepoint in a non-wrapper
role. A family-blind interior scan read the lone `` in
`《Alzheimers病中…》` as an unbalanced quote and refused to strip a
genuine `《…》` wrap, so a title that matched on main stopped matching:
exact False and ratio 0.6818, below the 0.70 floor. That takes out the
DOI-keyed ratio gate and the title-fallback exact gate at once, which
is the exact failure class #798 repaired — not something #800's
conservative-direction blessing covers, since here the outer pair DOES
enclose the title.
Scoping the balance scan to the outer pair's own family fixes it: a
`《…》` wrap tracks only `《`/`》` and is blind to quote marks."""
for wrapped, bare in (
("《Alzheimers病中表达》", "Alzheimers病中ProEXC表达"),
("98年香港回归研究》", "98年香港回归研究"),
):
with self.subTest(wrapped=wrapped):
self.assertEqual(ts.normalize_cn_title(wrapped), ts.normalize_cn_title(bare))
self.assertTrue(ts.exact_normalized_title(wrapped, bare))
self.assertEqual(ts._similarity(wrapped, bare), 1.0)
def test_interior_scan_ignores_other_families_but_not_its_own(self) -> None:
"""The family scoping is a narrowing, not a blanket weakening: a stray
closer of the OUTER pair's own family still blocks the strip, while a
stray mark of any other family is content.
`《「研究』》` is the interaction the review flagged: the interior
`「研究』` is mismatched, but not in the `《》` family, so the outer pair
still encloses one unit and strips. The mismatched interior marks
survive as content — which is correct, since they are exactly what
distinguishes this title from `《研究》`."""
self.assertEqual(ts.normalize_cn_title("《「研究』》"), "「研究』")
# Same-family stray closer still blocks, at any interior depth.
self.assertEqual(ts.normalize_cn_title("《红楼梦》与《金瓶梅》"), "《红楼梦》与《金瓶梅》")
def test_unclosed_interior_opener_blocks_the_strip(self) -> None:
"""The trailing `depth == 0` check, pinned on its own: `《基于《研究」的分析》`
has an interior `《` that nothing closes, so the outer `》` is closing the
INNER opener and the outer pair is not one unit."""
self.assertEqual(
ts.normalize_cn_title("《基于《研究」的分析》"), "《基于《研究」的分析》"
)
def test_stray_interior_closer_blocks_the_strip_independently(self) -> None:
"""#804 review advisory: the depth-0 stray-closer branch had no
discriminating test — a mutant clamping it (`depth = max(0, depth - 1)`)
instead of refusing passed the whole suite, including the two natural
titles above.
The reason is arithmetic, and worth recording so this test is not
"simplified" back later: clamping absorbs one closer, so a title whose
opener/closer counts are equal — every natural case here, including
`《红楼梦》与《金瓶梅》` — ends at depth 1 under the mutant and is refused
anyway, by the OTHER branch. Discriminating requires interior closers to
outnumber openers by exactly the clamp count, which no natural title
shape produces. Hence a deliberately synthetic string, asserted against
the helper directly: `《》研究《实践》》` opens at depth 0 with a `》`,
which is precisely what the branch exists to refuse."""
self.assertFalse(ts._outer_pair_encloses("《》研究《实践》》"))
# The natural stray-closer title stays refused too — belt and braces,
# since this is the branch that keeps `《红楼梦》与《金瓶梅》` intact.
self.assertFalse(ts._outer_pair_encloses("《红楼梦》与《金瓶梅》"))
def test_mangled_key_still_matches_its_identically_mangled_counterpart(self) -> None:
"""Matching correctness was never broken by the positional strip (both
sides mangled identically), and the pairedness fix must not change that
either way: equal normalizations still match on both paths."""
a = "《红楼梦》与《金瓶梅》"
self.assertTrue(ts.exact_normalized_title(a, a))
# The paired form now keeps its marks while an old-style bare variant
# differs — conservative non-matching is acceptable per #800.
self.assertFalse(ts.exact_normalized_title(a, "红楼梦》与《金瓶梅"))
def test_ungated_client_path_narrowing_is_explicit(self) -> None:
"""#804 review advisory: the invariance claim must not be stated more
broadly than the code supports.
`exact_normalized_title` and `_similarity` reach the CJK form only
behind `has_cjk`, so a Han-free title is untouched there. The client's
`_cn_titles_match` has NO such gate — it calls `normalize_cn_title`
directly — so a mark-carrying Han-free title can change verdict on that
path. Pinned here as a known, accepted narrowing rather than left to be
rediscovered: the path is DOI-keyed and Chinese-corpus-only in
practice."""
from chinese_literature_client import _cn_titles_match
wrapped, mangled = "《Hamlet》and《Macbeth》", "Hamlet》and《Macbeth"
self.assertFalse(_cn_titles_match(wrapped, mangled))
# The gated shared helpers never saw this pair as CJK, before or after.
self.assertFalse(ts.exact_normalized_title(wrapped, mangled))
def test_empty_wrapper_guard_is_a_cjk_branch_property(self) -> None:
"""The "`《》` never matches" guarantee belongs to `_cjk_titles_match`,
which requires a non-empty normalized key. `exact_normalized_title`
still returns True for `《》` vs itself through the ungated BASE
normalization branch — unchanged by #800, and pinned so the changelog's
narrowed wording stays honest."""
self.assertEqual(ts.normalize_cn_title("《》"), "")
self.assertFalse(ts._cjk_titles_match("《》", "「」"))
self.assertTrue(ts.exact_normalized_title("《》", "《》"))
def test_pairedness_behavior_is_shared_by_the_cjk_client(self) -> None:
"""#800 acceptance: both consumers change together. The client
re-imports the shared function (#799), so identity covers it — pinned
behaviorally here so a future private copy cannot hide the drift."""
import chinese_literature_client as cn
title = "《红楼梦》与《金瓶梅》"
self.assertEqual(cn.normalize_cn_title(title), ts.normalize_cn_title(title))
self.assertEqual(cn.normalize_cn_title(title), title)
def test_removes_whitespace_touching_han(self) -> None:
self.assertEqual(ts.normalize_cn_title("基于 ProEXC 的研究"), "基于ProEXC的研究")