mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
* fix: declare markdown-it-py floor and make the autolink round-trip tail run visibly (#801) The no-link_open round-trip tail of test_gfm_bare_urls_emails_and_schemes_ cannot_autolink soft-imported markdown-it-py (undeclared in requirements- dev.txt) and silently returned when absent, so it had never run in CI, while ambient markdown-it-py 2.x failed it on clean main (2.2.0 + linkify-it-py 2.0.3, reported in #799). Verified dividing line: 2.2.0 fails, 3.0.0 and 4.0.0 pass with linkify-it-py held at 2.0.3. - Split the tail into test_escaped_markdown_yields_no_linkify_tokens_on_ round_trip, gated by pytest.importorskip minversions (markdown_it 3.0.0, linkify_it 2.0.3): ambient-old environments skip visibly. - Declare markdown-it-py>=3.0 + linkify-it-py>=2.0.3 in requirements-dev.txt with a reverse pointer at the consuming test, so CI exercises the round trip for the first time. - Move the identical soft-import tail in test_renderer_neutralizes_markdown_ active_inventory_path (newly activated in CI by the same declaration) to the same importorskip idiom; no floor needed (default CommonMark, no linkify) — verified passing under 2.2.0, 3.0.0, and 4.0.0. - Consolidate the triplicated hostile-row construction in test_evidence_rows.py into one _hostile_row helper. Renderer behavior and every renderer-side assertion are unchanged. Verification: both full files 414 passed under markdown-it-py 4.0.0; affected tests re-run under 2.2.0 (pass + visible skip) and 3.0.0 (pass). Closes #801 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013R81d1YwGvJAznkPKk9gNw * fix: flatten inline token children in the newly activated manifest markdown scan (#801) Cross-model review (codex, xhigh) on PR #802 flagged that the twin test's token scan iterated only top-level tokens, but markdown-it nests link_open / image / html_inline under inline tokens' children — so the assertion could only ever catch html_block. Verified empirically, then flattened children into the scan (same idiom as the evidence-rows round-trip test). Strengthened assertion passes under markdown-it-py 2.2.0, 3.0.0, and 4.0.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013R81d1YwGvJAznkPKk9gNw --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
committed by
GitHub
GitHub
parent
e5718cbf58
commit
30ad279cdf
@@ -8,6 +8,8 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
- **CJK titles no longer fail the shared exact-title gate in the four index resolvers (#798).** `chinese_literature_client.py` already carried a Chinese-aware `normalize_cn_title` / `has_cjk` (#431 §"Chinese title matching"), but the four index resolvers (Semantic Scholar / OpenAlex / Crossref / arXiv) never saw it — they read the ASCII-centric `_text_similarity` helpers, where `.lower()` folds case but never width (Ｐ U+FF30 never reaches P U+0050) and `string.punctuation` contains none of `。`, `《》`, or U+3000. A Chinese paper an index served in a different-but-legitimate typesetting therefore missed on **two** paths: the DOI-keyed cross-check, which gates on the fuzzy ratio *alone* and scored a fullwidth spelling of the identical title at **0.625 — under the 0.70 floor — reporting a correct DOI as `DOI_MISMATCH`**; and the title-fallback search, which requires ratio **and** exact-normalized equality and so fell to `unresolvable`. Both feed the `*_unmatched` contamination signals, so a genuine paper could accumulate *k* across indexes and be rendered `CONTAMINATED-TRIANGULATION-UNMATCHED` — the protocol doc's own "P0, next to the word 'fabricated'". The failure was invisible to the English test corpus for a measurable reason: a Han character is a whole word, so the same six-codepoint corruption is 37.5% of a 16-character Chinese title but 8% of its 68-character English equivalent (measured 0.625 vs 0.912). Fix promotes `has_cjk` / `normalize_cn_title` into `scripts/_text_similarity.py` (behaviorally equivalent — the promotion hoists locals to module constants, precompiles the regex, and rewrites comments; the CJK client now re-imports rather than keeping a private copy, per the #128 anti-drift goal), adds the Chinese-aware form to `exact_normalized_title` as an additive third branch, and folds it into `_similarity` through the existing `max`. Both are gated on **both** sides carrying a Han ideograph, so every non-CJK verdict and every non-CJK *ratio* is provably unchanged — pinned by two oracle tests that re-state the pre-fix `exact_normalized_title` and `_similarity` formulas in full (the latter including the dotted-acronym branch) and assert exact agreement in both directions, so a raised score is caught as loudly as a lowered one. Cross-script and romanized pairs still cannot match (no translation oracle), Simplified/Traditional is still not folded, and an empty normalized key still never matches. Under the Chinese-aware form the ratio also regains discriminative power on the motivating pair: 1.000 for the identical title against an unchanged 0.510 for a genuinely different paper (the base form separated 0.566 from 0.510 — almost nothing). Also measured and pinned: on the motivating pair the pre-fix ratio was actively *anti-correlated* — the identical title scored **0.606 while a genuinely different paper scored 0.645**, so the wrong paper ranked higher. The repair leaves the unrelated pair's score byte-identical (0.6452 before and after; it is not equal under the CJK form, so nothing is folded in) and lifts only the true match, restoring the ordering title ranking depends on. Not in scope, and unchanged: the base ASCII normalization still collapses `ER+`/`ER-` and `p53`/`P53` because it maps ASCII punctuation to whitespace and lowercases, and `exact_normalized_title` ORs that form in; the fuzzy floor also remains a weak separator for CJK generally (that 0.645 near-miss is pre-existing and untouched here). 32 new tests, including 5 integration tests through the real Crossref client covering both broken paths, each verified to fail against the pre-fix module, plus a monkeypatched detonator pinning that the Chinese DOI-path matcher never consults the fuzzy ratio in either direction.
|
||||
|
||||
- **Autolink round-trip test no longer rides on an undeclared, version-sensitive dependency (#801).** `test_gfm_bare_urls_emails_and_schemes_cannot_autolink` soft-imported markdown-it-py — not listed in `requirements-dev.txt` — and silently returned when it was absent, so its no-`link_open` tail had never actually run in CI, while an ambient markdown-it-py 2.x (whose linkify pass still tokenizes the escaped forms as links; reproduced under 2.2.0 + linkify-it-py 2.0.3, reported by @akshath-raj in #799) failed the test on clean `main`. Verified dividing line: 2.2.0 fails, 3.0.0 and 4.0.0 pass with linkify-it-py 2.0.3 held constant. The tail is split into its own test (`test_escaped_markdown_yields_no_linkify_tokens_on_round_trip`) gated by `pytest.importorskip("markdown_it", minversion="3.0.0")` + `pytest.importorskip("linkify_it", minversion="2.0.3")` — ambient-old environments now skip visibly instead of failing or passing vacuously — and `requirements-dev.txt` declares `markdown-it-py>=3.0` + `linkify-it-py>=2.0.3` with a reverse pointer at the consuming test, so CI exercises the round-trip for the first time. The same declaration also newly activates the identical soft-import tail in `test_renderer_neutralizes_markdown_active_inventory_path` (`scripts/test_build_submission_packet_manifest.py`), which moves to the same `importorskip` idiom (no version floor: it parses default CommonMark without linkify) and gains the inline-children flatten its top-level-only token scan was missing — `link_open` / `image` / `html_inline` nest under `inline` tokens, so the pre-existing assertion could only ever catch `html_block`; its strengthened assertions verified under markdown-it-py 2.2.0, 3.0.0, and 4.0.0. The triplicated hostile-row construction in `test_evidence_rows.py` is consolidated into one `_hostile_row` helper. Renderer behavior and every renderer-side assertion are unchanged.
|
||||
|
||||
## [3.21.1] - 2026-08-24 — Bounded workflow substrates, sealed bakeoffs, and transport hardening
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -3,3 +3,7 @@ ruamel.yaml>=0.17
|
||||
jsonschema[format]>=4.17
|
||||
pypdf
|
||||
defusedxml
|
||||
# Floors consumed by scripts/test_evidence_rows.py::test_escaped_markdown_yields_no_linkify_tokens_on_round_trip (#801).
|
||||
# markdown-it-py < 3 linkifies escaped bare URLs; dropping these lines makes that test silently skip in CI.
|
||||
markdown-it-py>=3.0
|
||||
linkify-it-py>=2.0.3
|
||||
|
||||
@@ -2416,14 +2416,14 @@ def test_renderer_neutralizes_markdown_active_inventory_path(
|
||||
for line in observation_lines.splitlines()
|
||||
) == 1
|
||||
|
||||
try:
|
||||
from markdown_it import MarkdownIt
|
||||
except ImportError:
|
||||
return
|
||||
tokens = MarkdownIt().parse(rendered)
|
||||
markdown_it = pytest.importorskip("markdown_it")
|
||||
tokens = markdown_it.MarkdownIt().parse(rendered)
|
||||
# link_open / image / html_inline nest under inline tokens' children;
|
||||
# only html_block appears at the top level.
|
||||
flattened = [child for token in tokens for child in (token.children or [])]
|
||||
assert not any(
|
||||
token.type in {"link_open", "image", "html_inline", "html_block"}
|
||||
for token in tokens
|
||||
for token in [*tokens, *flattened]
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1460,21 +1460,25 @@ def test_source_replay_checks_hidden_off_page_rows(
|
||||
assert "EVR-PAGE-0025" in rendered or r"EVR\-PAGE\-0025" in rendered
|
||||
|
||||
|
||||
def _hostile_row(input_fixture: dict[str, Any], text: str) -> dict[str, Any]:
|
||||
return er.build(
|
||||
_raw_row(
|
||||
input_fixture,
|
||||
anchor_value="missing",
|
||||
claim__text=text,
|
||||
detail=text,
|
||||
source__display_label=text,
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def test_line_separators_newlines_tabs_and_bidi_are_visible_data_not_structure(
|
||||
input_fixture: dict[str, Any]
|
||||
) -> None:
|
||||
_runtime_required()
|
||||
hostile = "first\n## forged\tcell\u202eRTL\u2028line\u2029paragraph"
|
||||
row = er.build(
|
||||
_raw_row(
|
||||
input_fixture,
|
||||
anchor_value="missing",
|
||||
claim__text=hostile,
|
||||
detail=hostile,
|
||||
source__display_label=hostile,
|
||||
),
|
||||
None,
|
||||
)
|
||||
row = _hostile_row(input_fixture, hostile)
|
||||
markdown = er.render_markdown([row])
|
||||
html_output = er.render_html([row])
|
||||
for rendered in (markdown, html_output):
|
||||
@@ -1497,40 +1501,23 @@ def test_c1_nel_csi_and_osc_are_visible_ascii_not_raw_controls(
|
||||
input_fixture: dict[str, Any],
|
||||
) -> None:
|
||||
_runtime_required()
|
||||
hostile = f"before{codepoint}after"
|
||||
row = er.build(
|
||||
_raw_row(
|
||||
input_fixture,
|
||||
anchor_value="missing",
|
||||
claim__text=hostile,
|
||||
detail=hostile,
|
||||
source__display_label=hostile,
|
||||
),
|
||||
None,
|
||||
)
|
||||
row = _hostile_row(input_fixture, f"before{codepoint}after")
|
||||
for rendered in (er.render_markdown([row]), er.render_html([row])):
|
||||
assert codepoint not in rendered
|
||||
assert escape in rendered
|
||||
|
||||
|
||||
_AUTOLINK_PROBE_PAYLOAD = (
|
||||
"https://evil.example/path user@example.com ftp://evil.example/file "
|
||||
"javascript:alert(1)"
|
||||
)
|
||||
|
||||
|
||||
def test_gfm_bare_urls_emails_and_schemes_cannot_autolink(
|
||||
input_fixture: dict[str, Any]
|
||||
) -> None:
|
||||
_runtime_required()
|
||||
payload = (
|
||||
"https://evil.example/path user@example.com ftp://evil.example/file "
|
||||
"javascript:alert(1)"
|
||||
)
|
||||
row = er.build(
|
||||
_raw_row(
|
||||
input_fixture,
|
||||
anchor_value="missing",
|
||||
claim__text=payload,
|
||||
detail=payload,
|
||||
source__display_label=payload,
|
||||
),
|
||||
None,
|
||||
)
|
||||
row = _hostile_row(input_fixture, _AUTOLINK_PROBE_PAYLOAD)
|
||||
markdown = er.render_markdown([row])
|
||||
html_output = er.render_html([row])
|
||||
assert r"https\:\/\/evil\.example\/path" in markdown
|
||||
@@ -1540,14 +1527,17 @@ def test_gfm_bare_urls_emails_and_schemes_cannot_autolink(
|
||||
assert "<a" not in html_output.lower()
|
||||
assert "href=" not in html_output.lower()
|
||||
|
||||
try:
|
||||
from markdown_it import MarkdownIt
|
||||
except ImportError:
|
||||
return
|
||||
try:
|
||||
tokens = MarkdownIt("commonmark", {"linkify": True}).enable("linkify").parse(markdown)
|
||||
except ModuleNotFoundError: # markdown-it can be installed without linkify-it-py.
|
||||
return
|
||||
|
||||
def test_escaped_markdown_yields_no_linkify_tokens_on_round_trip(
|
||||
input_fixture: dict[str, Any]
|
||||
) -> None:
|
||||
_runtime_required()
|
||||
# markdown-it-py < 3 linkifies these escaped forms; the floors are declared in
|
||||
# requirements-dev.txt so CI always exercises this round-trip (#801).
|
||||
markdown_it = pytest.importorskip("markdown_it", minversion="3.0.0")
|
||||
pytest.importorskip("linkify_it", minversion="2.0.3")
|
||||
markdown = er.render_markdown([_hostile_row(input_fixture, _AUTOLINK_PROBE_PAYLOAD)])
|
||||
tokens = markdown_it.MarkdownIt("commonmark", {"linkify": True}).enable("linkify").parse(markdown)
|
||||
flattened = [child for token in tokens for child in (token.children or [])]
|
||||
assert all(token.type != "link_open" for token in [*tokens, *flattened])
|
||||
|
||||
|
||||
Reference in New Issue
Block a user