mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
* fix(citation-gate): take ref_slug from the prose join, not the corpus entry (#332) verify_citation wrote summary["ref_slug"] = entry.get("ref_slug"), but literature_corpus_entry.schema.json is additionalProperties:False with no ref_slug property — a schema-valid corpus entry never carries one, so the normal passport path emitted ref_slug: None and violated the summary contract (ref_slug required string). The 133 tests missed it because both fixtures (the _entry helper and the CLI fixture) illegally stuffed ref_slug INTO the corpus entry, masking the production shape. ref_slug is now an explicit prose-sourced param, parallel to the existing `anchor` param (both joined upstream, never read off the entry). verify_passport takes a {citation_key: ref_slug} join map and raises ValueError on a missing join rather than emitting a contract-invalid summary. The standalone CLI has no prose document, so it refuses by default with a clear error and offers an explicit --synthetic-ref-slug citation_key escape hatch (stderr-warned, diagnostic-only) instead of silently fabricating a slug. P2: check_evals_gold_set's flat queried_by enum check under-enforced the status<->queried_by coherence the summary schema requires (ran -> {id,title}; skipped/unreachable -> null; queried_by required present). Replaced it by validating each resolver_outcome against the shipped summary-schema $defs.resolver_outcome (single source of truth, matching the I9b reducer-recompute philosophy), and removed the now-dead STATUS_ENUM/QUERIED_BY_ENUM constants. Fixtures are now schema-valid against literature_corpus_entry.schema.json (added authors/year/source_pointer, dropped ref_slug, fixed an invalid DOI) with a guard test in each module so a future schema change can't silently re-mask. Tests: ref_slug-not-read-off-entry anti-regression, missing-join raises, end-to-end passport summaries validate against the schema, CLI refuse + synthetic modes, 3 queried_by-coherence mutation tests. 112 passing across the citation-gate + eval suites. Closes #332 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(citation-gate): reject non-string / empty ref_slug at the emission point (#332) verify_citation declared ref_slug as a required str but did not enforce it at runtime: a caller passing an empty string or a non-string would stamp a summary the schema rejects (type:string) or that joins to no <!--ref:slug--> prose marker (empty). verify_passport only guarded `is None`, so a present-but-empty slug slipped through. Adds a shared _is_valid_ref_slug (non-empty string) so verify_citation (the single emission point) and verify_passport (the join layer) agree on "bad slug" and can't drift. verify_citation now raises ValueError on any invalid ref_slug; verify_passport re-checks only to name the offending citation_key (context the per-citation layer lacks). Closes the codex-flagged hardening gap on the #332 fix. Tests: parametrized rejection over {None, "", 0, 123, list, dict}; mutation- verified (accept-all stub fails all 7 guard tests). Full suite 2109 pass / 3 skip / 0 fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
committed by
GitHub
parent
3ab2b2247a
commit
6286fd0b66
@@ -12,6 +12,11 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
- **Parallelize the OpenAlex + Crossref backfill lookups per entry in `migrate_literature_corpus_to_v3_9_0.py` (#138).** When both `openalex_unmatched` and `crossref_unmatched` are missing for an entry, the two independent resolver calls (different hosts, per-instance throttle state, monotonic timing) now run concurrently via a 2-worker `ThreadPoolExecutor` instead of one-after-the-other, roughly halving per-entry network wait on a full backfill. Scope is deliberately bounded: only the two calls within one entry overlap — the corpus loop stays sequential (cross-entry parallelism is out of scope; the clients' per-instance throttle assumes serial use), all passport mutation / report bookkeeping / degradation logging stays single-threaded on the orchestrator thread, and an already-set field still never consults its client. A single missing field skips the pool and calls directly. Behavior is otherwise byte-equivalent to the sequential version, including the omit-on-`Unavailable` partial-degradation contract (now surfaced via `Future.result()`). Adds 2 tests (barrier-verified parallel dispatch + the previously-untested API-down degradation path); the 6 existing migration tests pass unchanged.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`verification_gate` reads `ref_slug` from the prose join, not the corpus entry (#332).** `verify_citation`/`verify_passport` previously wrote `summary.ref_slug = entry.get("ref_slug")`, but `literature_corpus_entry.schema.json` is `additionalProperties: false` with no `ref_slug` property — so the normal (schema-valid) passport path emitted `ref_slug: null` and violated the summary contract (a required string). Two non-schema-conformant test fixtures masked it. `ref_slug` is now an explicit prose-sourced parameter parallel to `anchor`: `verify_citation(entry, clients, *, ref_slug, anchor=None, …)` and `verify_passport(passport, clients, *, ref_slug_by_key, anchors=None, …)`, with a `ValueError` on any invalid join — a missing key, or a present-but-empty/non-string slug (validated once at the `verify_citation` emission point via a shared `_is_valid_ref_slug` so the per-citation and passport layers can't drift; the passport layer re-checks only to name the offending `citation_key`) — rather than a contract-invalid summary. The standalone `verify_passport.py` CLI (which has no prose document) now refuses by default with a clear error and offers an explicit `--synthetic-ref-slug citation_key` diagnostic escape hatch instead of silently fabricating a slug. **API-stability note (C-V4):** these are new *required* keyword-only parameters. The spec's C-V4 freeze names v3.10.0, but #182 was specced-but-not-implemented in v3.10 (spec §0 amendment) and first shipped in the v3.11.0 minor release — so no v3.10.0 caller can depend on the old signature, and C-V4 itself permits a minor release to add required fields. The only in-repo callers (the CLI + the internal `verify_passport`→`verify_citation` call) are updated in lockstep.
|
||||
- **`check_evals_gold_set` enforces `status`↔`queried_by` coherence via the shipped schema (#332).** The gold validator's flat `queried_by ∈ {id, title, null}` enum check under-enforced the conditional coherence the summary schema requires (a ran resolver must carry `id`/`title`, a skipped/unreachable one must carry `null`, and `queried_by` must be present). It now validates each `resolver_outcome` against `citation_verification_summary.schema.json`'s `$defs.resolver_outcome` — single source of truth, matching the existing I9b reduce-and-compare philosophy — and the now-dead `STATUS_ENUM`/`QUERIED_BY_ENUM` constants are removed. The shipped gold set already satisfies the stricter check.
|
||||
|
||||
## [3.11.0] - 2026-06-04 — Deterministic citation verification gate (#182)
|
||||
|
||||
The v3.11.0 minor release ships **#182 — a deterministic citation-existence verification gate**
|
||||
|
||||
@@ -29,11 +29,12 @@ from citation_verification_summary import ( # noqa: E402
|
||||
)
|
||||
|
||||
LABEL_ENUM = {"true", "false", "unresolvable"}
|
||||
QUERIED_BY_ENUM = {"id", "title", None}
|
||||
KIND_ENUM = {"valid_doi", "valid_arxiv", "manual_exempt", "fabricated",
|
||||
"fabricated_title_only", "valid_unindexed"}
|
||||
RESOLVER_NAMES = ("crossref", "openalex", "semantic_scholar", "arxiv")
|
||||
STATUS_ENUM = {"matched", "unmatched", "unreachable", "skipped"}
|
||||
# status + queried_by enums (and their status↔queried_by coherence) are now
|
||||
# enforced by _RESOLVER_OUTCOME_VALIDATOR against the shipped summary-schema $def,
|
||||
# not local constants (#332 P2).
|
||||
|
||||
_CORPUS_ENTRY_SCHEMA_PATH = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
@@ -44,6 +45,23 @@ _CORPUS_ENTRY_VALIDATOR = Draft202012Validator(
|
||||
format_checker=Draft202012Validator.FORMAT_CHECKER,
|
||||
)
|
||||
|
||||
# Validate each resolver_outcome against the SHIPPED summary-schema $def rather
|
||||
# than a hand-rolled coherence check, so the gold validator can't drift from the
|
||||
# contract it pins (the I9b single-source-of-truth philosophy, applied to shape).
|
||||
# The $def carries the status↔queried_by allOf coherence (ran → {id,title};
|
||||
# skipped/unreachable → null) plus the required-present rule that a flat enum
|
||||
# check silently under-enforced (#332 P2).
|
||||
_SUMMARY_SCHEMA_PATH = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "shared" / "contracts" / "passport" / "citation_verification_summary.schema.json"
|
||||
)
|
||||
# Safe to validate the extracted $def in isolation because resolver_outcome is
|
||||
# $ref-less today. If it ever gains a $ref into a sibling $def, build the validator
|
||||
# from the full schema document (with a registry) instead of the slice.
|
||||
_RESOLVER_OUTCOME_VALIDATOR = Draft202012Validator(
|
||||
json.loads(_SUMMARY_SCHEMA_PATH.read_text(encoding="utf-8"))["$defs"]["resolver_outcome"]
|
||||
)
|
||||
|
||||
|
||||
def _load_json_strict(path: Path) -> Any:
|
||||
"""Load JSON; raise on duplicate keys (I4)."""
|
||||
@@ -206,17 +224,16 @@ def validate(root: Path) -> list[str]:
|
||||
if entry is None:
|
||||
errors.append(f"I9: {tid} resolver_outcomes missing resolver {resolver!r}")
|
||||
continue
|
||||
status = entry.get("status")
|
||||
if status not in STATUS_ENUM:
|
||||
# Validate the resolver_outcome against the shipped summary-schema
|
||||
# $def: status enum, queried_by enum, queried_by required-present, AND
|
||||
# the status↔queried_by coherence allOf (ran → {id,title};
|
||||
# skipped/unreachable → null). A flat enum check under-enforced the
|
||||
# last two (#332 P2).
|
||||
for verr in _RESOLVER_OUTCOME_VALIDATOR.iter_errors(entry):
|
||||
errors.append(
|
||||
f"I9: {tid} resolver_outcomes.{resolver}.status={status!r} "
|
||||
f"not in {sorted(STATUS_ENUM)}"
|
||||
)
|
||||
queried_by = entry.get("queried_by")
|
||||
if queried_by not in QUERIED_BY_ENUM:
|
||||
errors.append(
|
||||
f"I9: {tid} resolver_outcomes.{resolver}.queried_by={queried_by!r} "
|
||||
f"not in {{'id', 'title', null}}"
|
||||
f"I9: {tid} resolver_outcomes.{resolver} violates "
|
||||
f"citation_verification_summary $defs.resolver_outcome: "
|
||||
f"{verr.message}"
|
||||
)
|
||||
|
||||
recomputed = _reduce_lookup_verified(ros)
|
||||
|
||||
@@ -194,6 +194,48 @@ def test_i9_invalid_queried_by_enum_caught(tmp_path):
|
||||
assert any("I9" in e for e in errors), f"I9 not caught; errors: {errors}"
|
||||
|
||||
|
||||
def test_i9_queried_by_missing_caught(tmp_path):
|
||||
"""I9 coherence (#332 P2): queried_by is REQUIRED present. An entry that omits
|
||||
it must be caught — the summary schema marks queried_by required, and a missing
|
||||
key is load-bearing (an absent key could let an ambiguous unmatched silently
|
||||
reduce to unresolvable instead of false)."""
|
||||
target = _copy_clean(tmp_path)
|
||||
exp = json.loads((target / "expected_outcomes.json").read_text(encoding="utf-8"))
|
||||
del exp["001-valid-doi-test"]["resolver_outcomes"]["crossref"]["queried_by"]
|
||||
(target / "expected_outcomes.json").write_text(json.dumps(exp))
|
||||
errors = check_evals_gold_set.validate(target)
|
||||
assert any("I9" in e for e in errors), f"I9 not caught; errors: {errors}"
|
||||
|
||||
|
||||
def test_i9_ran_resolver_with_null_queried_by_caught(tmp_path):
|
||||
"""I9 coherence (#332 P2): a ran resolver (matched/unmatched) must carry
|
||||
queried_by ∈ {id, title}, never null — the summary schema's allOf forces it.
|
||||
A matched row claiming queried_by=null is incoherent and must be caught."""
|
||||
target = _copy_clean(tmp_path)
|
||||
exp = json.loads((target / "expected_outcomes.json").read_text(encoding="utf-8"))
|
||||
ros = exp["001-valid-doi-test"]["resolver_outcomes"]
|
||||
# crossref is matched in the clean fixture; null queried_by is incoherent.
|
||||
ros["crossref"]["queried_by"] = None
|
||||
(target / "expected_outcomes.json").write_text(json.dumps(exp))
|
||||
errors = check_evals_gold_set.validate(target)
|
||||
assert any("I9" in e for e in errors), f"I9 not caught; errors: {errors}"
|
||||
|
||||
|
||||
def test_i9_skipped_resolver_with_nonnull_queried_by_caught(tmp_path):
|
||||
"""I9 coherence (#332 P2): a skipped/unreachable resolver must carry
|
||||
queried_by=null — there was no query to attribute. A skipped row claiming
|
||||
queried_by='id' is incoherent (the summary schema's allOf forces null)."""
|
||||
target = _copy_clean(tmp_path)
|
||||
exp = json.loads((target / "expected_outcomes.json").read_text(encoding="utf-8"))
|
||||
ros = exp["001-valid-doi-test"]["resolver_outcomes"]
|
||||
# arxiv is skipped on a non-arXiv citation in the clean fixture.
|
||||
assert ros["arxiv"]["status"] == "skipped", "fixture assumption changed"
|
||||
ros["arxiv"]["queried_by"] = "id" # incoherent: skipped must be null
|
||||
(target / "expected_outcomes.json").write_text(json.dumps(exp))
|
||||
errors = check_evals_gold_set.validate(target)
|
||||
assert any("I9" in e for e in errors), f"I9 not caught; errors: {errors}"
|
||||
|
||||
|
||||
def test_i9b_false_with_only_title_only_unmatched_caught(tmp_path):
|
||||
"""I9b (C-V6(a)): a `false`-labeled tuple whose unmatched are all title-only
|
||||
(queried_by != id) is a mislabel — narrowed-false requires an ID-keyed
|
||||
|
||||
@@ -21,14 +21,22 @@ if str(REPO_ROOT / "scripts") not in sys.path:
|
||||
sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||
|
||||
|
||||
_DEFAULT_REF_SLUG = "vaswani-2017-attention"
|
||||
|
||||
|
||||
def _entry(**overrides):
|
||||
# Production-shaped corpus entry: NO anchor field (the v3.7.3 anchor lives in
|
||||
# writer prose, joined by ref_slug — passed to verify_citation as an explicit
|
||||
# `anchor` param, never read off the corpus entry).
|
||||
# Production-shaped corpus entry: schema-valid against
|
||||
# literature_corpus_entry.schema.json (additionalProperties:False). It carries
|
||||
# NEITHER ref_slug NOR anchor — both live in writer prose and are passed to
|
||||
# verify_citation as explicit params, never read off the corpus entry. Carries
|
||||
# all five required corpus fields (citation_key/title/authors/year/source_pointer)
|
||||
# so the fixture matches the real shape the gate sees in production.
|
||||
base = {
|
||||
"citation_key": "vaswani2017",
|
||||
"ref_slug": "vaswani-2017-attention",
|
||||
"title": "Attention Is All You Need",
|
||||
"authors": [{"family": "Vaswani", "given": "Ashish"}],
|
||||
"year": 2017,
|
||||
"source_pointer": "kb://refs/vaswani2017",
|
||||
"doi": "10.5555/abc",
|
||||
"obtained_via": "folder-scan",
|
||||
}
|
||||
@@ -65,7 +73,7 @@ def test_matched_yields_true_and_id_queried():
|
||||
|
||||
cr = MagicMock()
|
||||
cr.doi_lookup_with_title_check.return_value = {"title": ["X"]} # match
|
||||
outcome = verify_citation(_entry(), _clients(crossref=cr))
|
||||
outcome = verify_citation(_entry(), _clients(crossref=cr), ref_slug=_DEFAULT_REF_SLUG)
|
||||
|
||||
assert outcome["lookup_verified"] == "true"
|
||||
assert outcome["resolver_outcomes"]["crossref"]["status"] == "matched"
|
||||
@@ -75,7 +83,7 @@ def test_matched_yields_true_and_id_queried():
|
||||
def test_id_keyed_unmatched_yields_false():
|
||||
from verification_gate import verify_citation
|
||||
# All resolvers miss; entry has a DOI → ID-keyed unmatched → false.
|
||||
outcome = verify_citation(_entry(), _clients())
|
||||
outcome = verify_citation(_entry(), _clients(), ref_slug=_DEFAULT_REF_SLUG)
|
||||
assert outcome["lookup_verified"] == "false"
|
||||
assert outcome["resolver_outcomes"]["crossref"]["queried_by"] == "id"
|
||||
|
||||
@@ -83,7 +91,7 @@ def test_id_keyed_unmatched_yields_false():
|
||||
def test_title_only_unmatched_yields_unresolvable():
|
||||
from verification_gate import verify_citation
|
||||
# No DOI → title-only unmatched everywhere → unresolvable (C-V6(a)).
|
||||
outcome = verify_citation(_entry(doi=None), _clients())
|
||||
outcome = verify_citation(_entry(doi=None), _clients(), ref_slug=_DEFAULT_REF_SLUG)
|
||||
assert outcome["lookup_verified"] == "unresolvable"
|
||||
assert outcome["resolver_outcomes"]["crossref"]["queried_by"] == "title"
|
||||
|
||||
@@ -95,7 +103,7 @@ def test_resolver_outage_is_unreachable():
|
||||
cr = MagicMock()
|
||||
cr.doi_lookup_with_title_check.side_effect = CrossrefUnavailable("down")
|
||||
# other three also miss (id-keyed) → false stands (anti-fabrication bias).
|
||||
outcome = verify_citation(_entry(), _clients(crossref=cr))
|
||||
outcome = verify_citation(_entry(), _clients(crossref=cr), ref_slug=_DEFAULT_REF_SLUG)
|
||||
assert outcome["resolver_outcomes"]["crossref"]["status"] == "unreachable"
|
||||
assert outcome["resolver_outcomes"]["crossref"]["queried_by"] is None
|
||||
assert outcome["lookup_verified"] == "false"
|
||||
@@ -115,6 +123,7 @@ def test_all_unreachable_is_unresolvable():
|
||||
outcome = verify_citation(
|
||||
_entry(arxiv_id="1706.03762"),
|
||||
_clients(crossref=cr, openalex=oa, semantic_scholar=s2, arxiv=ax),
|
||||
ref_slug=_DEFAULT_REF_SLUG,
|
||||
)
|
||||
assert outcome["lookup_verified"] == "unresolvable"
|
||||
for r in ("crossref", "openalex", "semantic_scholar", "arxiv"):
|
||||
@@ -123,7 +132,7 @@ def test_all_unreachable_is_unresolvable():
|
||||
|
||||
def test_manual_entry_all_skipped_unresolvable():
|
||||
from verification_gate import verify_citation
|
||||
outcome = verify_citation(_entry(obtained_via="manual"), _clients())
|
||||
outcome = verify_citation(_entry(obtained_via="manual"), _clients(), ref_slug=_DEFAULT_REF_SLUG)
|
||||
assert outcome["lookup_verified"] == "unresolvable"
|
||||
for r in ("crossref", "openalex", "semantic_scholar", "arxiv"):
|
||||
assert outcome["resolver_outcomes"][r]["status"] == "skipped"
|
||||
@@ -132,28 +141,29 @@ def test_manual_entry_all_skipped_unresolvable():
|
||||
def test_arxiv_skipped_on_non_arxiv_citation():
|
||||
from verification_gate import verify_citation
|
||||
cr = MagicMock(); cr.doi_lookup_with_title_check.return_value = {"title": ["X"]}
|
||||
outcome = verify_citation(_entry(), _clients(crossref=cr)) # no arxiv_id
|
||||
outcome = verify_citation(_entry(), _clients(crossref=cr), ref_slug=_DEFAULT_REF_SLUG) # no arxiv_id
|
||||
assert outcome["resolver_outcomes"]["arxiv"]["status"] == "skipped"
|
||||
|
||||
|
||||
def test_anchor_present_true_for_page_kind():
|
||||
from verification_gate import verify_citation
|
||||
# anchor is an EXPLICIT param (prose-sourced, joined by ref_slug upstream).
|
||||
outcome = verify_citation(_entry(), _clients(), anchor=_PAGE_ANCHOR)
|
||||
outcome = verify_citation(_entry(), _clients(), ref_slug=_DEFAULT_REF_SLUG, anchor=_PAGE_ANCHOR)
|
||||
assert outcome["anchor_present"] is True
|
||||
|
||||
|
||||
def test_anchor_present_false_for_none_kind():
|
||||
from verification_gate import verify_citation
|
||||
outcome = verify_citation(
|
||||
_entry(), _clients(), anchor={"kind": "none", "value": None})
|
||||
_entry(), _clients(), ref_slug=_DEFAULT_REF_SLUG,
|
||||
anchor={"kind": "none", "value": None})
|
||||
assert outcome["anchor_present"] is False
|
||||
|
||||
|
||||
def test_anchor_present_false_when_anchor_omitted():
|
||||
from verification_gate import verify_citation
|
||||
# No anchor param (the prose join found no anchor for this ref_slug) → False.
|
||||
outcome = verify_citation(_entry(), _clients())
|
||||
outcome = verify_citation(_entry(), _clients(), ref_slug=_DEFAULT_REF_SLUG)
|
||||
assert outcome["anchor_present"] is False
|
||||
|
||||
|
||||
@@ -165,20 +175,26 @@ def test_anchor_is_not_read_off_the_corpus_entry():
|
||||
not in literature_corpus)."""
|
||||
from verification_gate import verify_citation
|
||||
e = _entry(anchor={"kind": "page", "value": "99"}) # decoy on the entry
|
||||
outcome = verify_citation(e, _clients()) # no anchor param
|
||||
outcome = verify_citation(e, _clients(), ref_slug=_DEFAULT_REF_SLUG) # no anchor param
|
||||
assert outcome["anchor_present"] is False
|
||||
|
||||
|
||||
def test_outcome_carries_keys_and_timestamp():
|
||||
from verification_gate import verify_citation
|
||||
outcome = verify_citation(_entry(), _clients())
|
||||
outcome = verify_citation(_entry(), _clients(), ref_slug=_DEFAULT_REF_SLUG)
|
||||
assert outcome["citation_key"] == "vaswani2017"
|
||||
assert outcome["ref_slug"] == "vaswani-2017-attention"
|
||||
assert outcome["verification_timestamp"] # never null/empty
|
||||
|
||||
|
||||
def test_outcome_validates_against_summary_schema():
|
||||
"""The outcome must validate against citation_verification_summary.schema."""
|
||||
"""The outcome must validate against citation_verification_summary.schema.
|
||||
|
||||
The fixture (_entry) is schema-valid against literature_corpus_entry.schema
|
||||
(no ref_slug — additionalProperties:False), so ref_slug comes ONLY from the
|
||||
explicit param. Before #332 this passed spuriously because the fixture stuffed
|
||||
an illegal ref_slug into the entry; with a production-shaped entry it now
|
||||
actually exercises the contract (ref_slug must be a non-null string)."""
|
||||
import json
|
||||
from jsonschema import Draft202012Validator
|
||||
from verification_gate import verify_citation
|
||||
@@ -187,11 +203,39 @@ def test_outcome_validates_against_summary_schema():
|
||||
REPO_ROOT / "shared" / "contracts" / "passport"
|
||||
/ "citation_verification_summary.schema.json"
|
||||
).read_text(encoding="utf-8"))
|
||||
outcome = verify_citation(_entry(), _clients())
|
||||
outcome = verify_citation(_entry(), _clients(), ref_slug=_DEFAULT_REF_SLUG)
|
||||
errors = list(Draft202012Validator(schema).iter_errors(outcome))
|
||||
assert errors == [], f"outcome must validate: {errors}"
|
||||
|
||||
|
||||
def test_ref_slug_is_not_read_off_the_corpus_entry():
|
||||
"""Anti-regression for #332: even if a corpus entry erroneously carries a
|
||||
'ref_slug' key, it MUST be ignored — the emitted ref_slug derives ONLY from
|
||||
the explicit param (the ref_slug lives in writer prose, not in
|
||||
literature_corpus, whose schema forbids the field). Pins that we don't
|
||||
resurrect the entry.get('ref_slug') path that emitted ref_slug: None on every
|
||||
schema-valid passport."""
|
||||
from verification_gate import verify_citation
|
||||
e = _entry(ref_slug="decoy-off-the-entry") # illegal decoy on the entry
|
||||
outcome = verify_citation(e, _clients(), ref_slug="from-prose")
|
||||
assert outcome["ref_slug"] == "from-prose"
|
||||
|
||||
|
||||
def test_corpus_fixture_is_schema_valid():
|
||||
"""Guard the guard: the _entry fixture must itself validate against
|
||||
literature_corpus_entry.schema.json (the masking bug in #332 was a fixture
|
||||
that carried a field the corpus schema forbids). If this fails, the schema
|
||||
tests above are testing a shape production never sees."""
|
||||
import json
|
||||
from jsonschema import Draft202012Validator
|
||||
schema = json.loads((
|
||||
REPO_ROOT / "shared" / "contracts" / "passport"
|
||||
/ "literature_corpus_entry.schema.json"
|
||||
).read_text(encoding="utf-8"))
|
||||
errors = list(Draft202012Validator(schema).iter_errors(_entry()))
|
||||
assert errors == [], f"_entry fixture must be a valid corpus entry: {errors}"
|
||||
|
||||
|
||||
# ---------- verify_passport ----------
|
||||
|
||||
|
||||
@@ -201,13 +245,16 @@ def test_verify_passport_runs_each_entry():
|
||||
cr = MagicMock(); cr.doi_lookup_with_title_check.return_value = {"title": ["X"]}
|
||||
passport = {
|
||||
"literature_corpus": [
|
||||
_entry(citation_key="a", ref_slug="slug-a"),
|
||||
_entry(citation_key="b", ref_slug="slug-b", doi=None),
|
||||
_entry(citation_key="a"),
|
||||
_entry(citation_key="b", doi=None),
|
||||
]
|
||||
}
|
||||
outcomes = verify_passport(passport, clients=_clients(crossref=cr))
|
||||
outcomes = verify_passport(
|
||||
passport, clients=_clients(crossref=cr),
|
||||
ref_slug_by_key={"a": "slug-a", "b": "slug-b"})
|
||||
assert len(outcomes) == 2
|
||||
assert {o["citation_key"] for o in outcomes} == {"a", "b"}
|
||||
assert {o["ref_slug"] for o in outcomes} == {"slug-a", "slug-b"}
|
||||
|
||||
|
||||
def test_verify_passport_joins_anchors_by_ref_slug():
|
||||
@@ -217,12 +264,15 @@ def test_verify_passport_joins_anchors_by_ref_slug():
|
||||
from verification_gate import verify_passport
|
||||
passport = {
|
||||
"literature_corpus": [
|
||||
_entry(citation_key="a", ref_slug="slug-a"),
|
||||
_entry(citation_key="b", ref_slug="slug-b"),
|
||||
_entry(citation_key="a"),
|
||||
_entry(citation_key="b"),
|
||||
]
|
||||
}
|
||||
ref_slug_by_key = {"a": "slug-a", "b": "slug-b"}
|
||||
anchors = {"slug-a": _PAGE_ANCHOR} # slug-b absent → anchor_present False
|
||||
outcomes = verify_passport(passport, clients=_clients(), anchors=anchors)
|
||||
outcomes = verify_passport(
|
||||
passport, clients=_clients(),
|
||||
ref_slug_by_key=ref_slug_by_key, anchors=anchors)
|
||||
by_key = {o["citation_key"]: o for o in outcomes}
|
||||
assert by_key["a"]["anchor_present"] is True
|
||||
assert by_key["b"]["anchor_present"] is False
|
||||
@@ -230,8 +280,42 @@ def test_verify_passport_joins_anchors_by_ref_slug():
|
||||
|
||||
def test_verify_passport_empty_corpus():
|
||||
from verification_gate import verify_passport
|
||||
assert verify_passport({"literature_corpus": []}, clients=_clients()) == []
|
||||
assert verify_passport({}, clients=_clients()) == []
|
||||
assert verify_passport(
|
||||
{"literature_corpus": []}, clients=_clients(), ref_slug_by_key={}) == []
|
||||
assert verify_passport({}, clients=_clients(), ref_slug_by_key={}) == []
|
||||
|
||||
|
||||
def test_verify_passport_raises_on_missing_join():
|
||||
"""An entry whose citation_key has no joined ref_slug is a caller error: the
|
||||
summary contract requires a non-null string ref_slug, so verify_passport
|
||||
raises rather than emitting a contract-invalid summary (#332). No silent
|
||||
default to citation_key or empty string."""
|
||||
from verification_gate import verify_passport
|
||||
passport = {"literature_corpus": [_entry(citation_key="orphan")]}
|
||||
with pytest.raises(ValueError, match="orphan"):
|
||||
verify_passport(passport, clients=_clients(), ref_slug_by_key={})
|
||||
|
||||
|
||||
def test_verify_passport_outputs_are_schema_valid():
|
||||
"""End-to-end: a schema-valid passport + a proper join map produces summaries
|
||||
that ALL validate against citation_verification_summary.schema (the #332 P1
|
||||
was that the normal passport path produced schema-invalid ref_slug: None)."""
|
||||
import json
|
||||
from jsonschema import Draft202012Validator
|
||||
from verification_gate import verify_passport
|
||||
schema = json.loads((
|
||||
REPO_ROOT / "shared" / "contracts" / "passport"
|
||||
/ "citation_verification_summary.schema.json"
|
||||
).read_text(encoding="utf-8"))
|
||||
passport = {"literature_corpus": [
|
||||
_entry(citation_key="a"), _entry(citation_key="b", doi=None)]}
|
||||
outcomes = verify_passport(
|
||||
passport, clients=_clients(),
|
||||
ref_slug_by_key={"a": "slug-a", "b": "slug-b"})
|
||||
validator = Draft202012Validator(schema)
|
||||
for o in outcomes:
|
||||
errors = list(validator.iter_errors(o))
|
||||
assert errors == [], f"summary must validate: {errors}"
|
||||
|
||||
|
||||
def test_cache_argument_is_honest_not_silent_noop():
|
||||
@@ -240,4 +324,19 @@ def test_cache_argument_is_honest_not_silent_noop():
|
||||
thinking caching took effect."""
|
||||
from verification_gate import verify_citation
|
||||
with pytest.raises(NotImplementedError):
|
||||
verify_citation(_entry(), _clients(), cache=object())
|
||||
verify_citation(_entry(), _clients(), ref_slug=_DEFAULT_REF_SLUG, cache=object())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", [None, "", 0, 123, ["slug"], {"slug": 1}])
|
||||
def test_verify_citation_rejects_non_string_or_empty_ref_slug(bad):
|
||||
"""#332 hardening: ref_slug is the prose-join key that the summary schema
|
||||
requires as a non-empty string. verify_citation is the single emission point
|
||||
(verify_passport routes through it), so the contract is enforced here once:
|
||||
a non-string or empty ref_slug would stamp a summary that the caller cannot
|
||||
join to any <!--ref:slug--> marker (empty) or that fails the schema's
|
||||
type:string (non-string). Both are caller errors — raise rather than emit a
|
||||
join-broken / contract-invalid summary. Guarding at the entry also closes the
|
||||
verify_passport `is None`-only gap (it now rejects "" and non-str too)."""
|
||||
from verification_gate import verify_citation
|
||||
with pytest.raises((ValueError, TypeError)):
|
||||
verify_citation(_entry(), _clients(), ref_slug=bad)
|
||||
|
||||
@@ -35,29 +35,73 @@ def _no_network_clients():
|
||||
("crossref", "openalex", "semantic_scholar", "arxiv")}
|
||||
|
||||
|
||||
def _entry(**overrides):
|
||||
# Schema-valid corpus entry: all five required corpus fields, NO ref_slug
|
||||
# (literature_corpus_entry.schema is additionalProperties:False — the ref_slug
|
||||
# lives in writer prose, not the corpus).
|
||||
base = {
|
||||
"citation_key": "a", "title": "T",
|
||||
"authors": [{"family": "Doe", "given": "J"}], "year": 2020,
|
||||
"source_pointer": "kb://refs/a",
|
||||
"doi": "10.5555/x", "obtained_via": "folder-scan",
|
||||
}
|
||||
base.update(overrides)
|
||||
return base
|
||||
|
||||
|
||||
def _write_passport(tmp_path, corpus):
|
||||
p = tmp_path / "passport.yaml"
|
||||
p.write_text(yaml.safe_dump({"literature_corpus": corpus}), encoding="utf-8")
|
||||
return p
|
||||
|
||||
|
||||
def test_cli_emits_json_summary(tmp_path, capsys):
|
||||
from verify_passport import run
|
||||
def test_cli_fixture_is_schema_valid():
|
||||
"""Guard the guard: the _entry fixture must validate against
|
||||
literature_corpus_entry.schema.json, so the CLI tests exercise the real
|
||||
production shape (the #332 masking bug was a fixture carrying a forbidden
|
||||
ref_slug field)."""
|
||||
import json
|
||||
from jsonschema import Draft202012Validator
|
||||
schema = json.loads((
|
||||
REPO_ROOT / "shared" / "contracts" / "passport"
|
||||
/ "literature_corpus_entry.schema.json"
|
||||
).read_text(encoding="utf-8"))
|
||||
errors = list(Draft202012Validator(schema).iter_errors(_entry()))
|
||||
assert errors == [], f"_entry fixture must be a valid corpus entry: {errors}"
|
||||
|
||||
# Production-shaped corpus entry: no anchor field (the anchor lives in writer
|
||||
# prose, not in literature_corpus). The ad-hoc CLI has no prose document, so
|
||||
# anchor_present is honestly False — a prose-join is a later-batch concern.
|
||||
passport = _write_passport(tmp_path, [
|
||||
{"citation_key": "a", "ref_slug": "slug-a", "title": "T",
|
||||
"doi": "10.5/x", "obtained_via": "folder-scan"},
|
||||
])
|
||||
|
||||
def test_cli_refuses_without_prose_join(tmp_path, capsys):
|
||||
"""Default behavior: a passport-only CLI cannot honestly emit a
|
||||
citation_verification_summary, because the summary contract requires the
|
||||
prose-sourced ref_slug join that a passport alone does not carry. The CLI
|
||||
refuses (nonzero) with a clear error rather than fabricating ref_slug (#332)."""
|
||||
from verify_passport import run
|
||||
passport = _write_passport(tmp_path, [_entry()])
|
||||
rc = run([str(passport)], clients_factory=_no_network_clients)
|
||||
assert rc != 0
|
||||
err = capsys.readouterr().err
|
||||
assert "ref_slug" in err or "prose" in err # explains why it refused
|
||||
|
||||
|
||||
def test_cli_synthetic_ref_slug_uses_citation_key(tmp_path, capsys):
|
||||
"""Explicit escape hatch: --synthetic-ref-slug citation_key synthesizes
|
||||
ref_slug from citation_key so the ad-hoc tool can still produce output. It
|
||||
warns on stderr that the result is diagnostic (not prose-join-safe) so no
|
||||
downstream consumer mistakes it for a real prose join (#332)."""
|
||||
from verify_passport import run
|
||||
passport = _write_passport(tmp_path, [_entry(citation_key="a")])
|
||||
rc = run([str(passport), "--synthetic-ref-slug", "citation_key"],
|
||||
clients_factory=_no_network_clients)
|
||||
assert rc == 0
|
||||
out = json.loads(capsys.readouterr().out)
|
||||
captured = capsys.readouterr()
|
||||
out = json.loads(captured.out)
|
||||
assert len(out) == 1
|
||||
assert out[0]["citation_key"] == "a"
|
||||
assert out[0]["ref_slug"] == "a" # synthesized from citation_key
|
||||
assert out[0]["lookup_verified"] == "false" # id-keyed unmatched
|
||||
assert out[0]["anchor_present"] is False # no prose anchor available
|
||||
# the synthetic mode must say so on stderr (diagnostic, not prose-join-safe)
|
||||
assert "synthetic" in captured.err.lower() or "diagnostic" in captured.err.lower()
|
||||
|
||||
|
||||
def test_cli_missing_file_errors(tmp_path):
|
||||
|
||||
@@ -2,14 +2,18 @@
|
||||
"""verification_gate — citation existence verification API (Delta 5).
|
||||
|
||||
Public functions:
|
||||
- verify_citation(entry, clients, cache=None) -> CitationVerificationOutcome
|
||||
- verify_passport(passport, clients, cache=None) -> list[outcome]
|
||||
- verify_citation(entry, clients, *, ref_slug, anchor=None, cache=None)
|
||||
-> CitationVerificationOutcome
|
||||
- verify_passport(passport, clients, *, ref_slug_by_key, anchors=None,
|
||||
cache=None) -> list[outcome]
|
||||
|
||||
Composes the four resolvers (crossref / openalex / semantic_scholar / arxiv),
|
||||
maps each resolver's execution to a {status, queried_by} outcome, derives the
|
||||
3-class lookup_verified via the Delta 4 reducer (narrowed-false, C-V6(a)),
|
||||
reads anchor_present from the v3.7.3 anchor marker, and stamps
|
||||
verification_timestamp. Does NOT duplicate the v3.8 audit pipeline — it composes
|
||||
verification_timestamp. Both ref_slug and anchor are prose-sourced inputs joined
|
||||
upstream by citation_key — NEVER read off the corpus entry, whose schema forbids
|
||||
them (#332). Does NOT duplicate the v3.8 audit pipeline — it composes
|
||||
the same lower-layer resolvers and writes the unified summary schema (Delta 4).
|
||||
|
||||
The returned dict validates against
|
||||
@@ -60,6 +64,15 @@ except ImportError: # pragma: no cover - dual-path import
|
||||
_ANCHOR_PRESENT_KINDS = frozenset({"quote", "page", "section", "paragraph"})
|
||||
|
||||
|
||||
def _is_valid_ref_slug(ref_slug: Any) -> bool:
|
||||
"""A ref_slug is valid iff it is a non-empty string: the summary schema
|
||||
requires ref_slug as a string, and an empty slug joins to no
|
||||
<!--ref:slug--> prose marker. Single definition so verify_citation (the
|
||||
emission point) and verify_passport (the join layer) agree on "bad slug"
|
||||
(#332)."""
|
||||
return isinstance(ref_slug, str) and bool(ref_slug)
|
||||
|
||||
|
||||
def _outcome(status: str, queried_by: str | None,
|
||||
response_summary: str | None = None) -> dict[str, Any]:
|
||||
return {"status": status, "queried_by": queried_by,
|
||||
@@ -122,14 +135,21 @@ def verify_citation(
|
||||
entry: Mapping[str, Any],
|
||||
clients: Mapping[str, Any],
|
||||
*,
|
||||
ref_slug: str,
|
||||
anchor: Mapping[str, Any] | None = None,
|
||||
cache=None,
|
||||
) -> dict[str, Any]:
|
||||
"""Verify one citation's existence across the four resolvers.
|
||||
|
||||
`entry` carries citation_key, ref_slug, title, optional doi / arxiv_id,
|
||||
obtained_via. `clients` is a mapping {crossref, openalex, semantic_scholar,
|
||||
arxiv} of resolver clients (injected so callers control network / cache).
|
||||
`entry` carries citation_key, title, authors, year, source_pointer, optional
|
||||
doi / arxiv_id, obtained_via. `clients` is a mapping {crossref, openalex,
|
||||
semantic_scholar, arxiv} of resolver clients (injected so callers control
|
||||
network / cache).
|
||||
|
||||
`ref_slug` is the writer-prose `<!--ref:slug-->` marker this citation renders
|
||||
under, supplied by the caller — never read off the corpus entry (same
|
||||
provenance rule as `anchor`; the entry schema forbids the field, #332).
|
||||
|
||||
`anchor` is the v3.7.3 anchor marker ({kind, value}) for this citation's
|
||||
ref_slug, already parsed from writer prose and joined upstream (None when no
|
||||
anchor marker exists for the ref_slug). It is a SEPARATE input, not an entry
|
||||
@@ -151,6 +171,17 @@ def verify_citation(
|
||||
"cache-through at the verification_gate layer is not yet wired "
|
||||
"(#182 Delta-2 follow-up); pass cache=None"
|
||||
)
|
||||
if not _is_valid_ref_slug(ref_slug):
|
||||
# ref_slug is the prose-join key stamped verbatim into the summary, which
|
||||
# the schema requires as a non-empty string. This is the single emission
|
||||
# point (verify_passport routes through here), so the contract is enforced
|
||||
# once: a non-string fails the schema's type:string, and an empty string
|
||||
# joins to no <!--ref:slug--> marker. Either is a caller (prose-join) error
|
||||
# — refuse rather than emit a contract-invalid / join-broken summary (#332).
|
||||
raise ValueError(
|
||||
f"ref_slug must be a non-empty string (the writer-prose join key), "
|
||||
f"got {ref_slug!r}; corpus entries do not carry ref_slug (#332)"
|
||||
)
|
||||
if entry.get("obtained_via") == "manual":
|
||||
# v3.7.3 manual exemption: no resolver runs — all four skipped (checked
|
||||
# once here rather than re-checked inside each resolver helper).
|
||||
@@ -170,7 +201,7 @@ def verify_citation(
|
||||
}
|
||||
return {
|
||||
"citation_key": entry.get("citation_key"),
|
||||
"ref_slug": entry.get("ref_slug"),
|
||||
"ref_slug": ref_slug,
|
||||
"lookup_verified": reduce_lookup_verified(resolver_outcomes),
|
||||
"anchor_present": _anchor_present(anchor),
|
||||
"verification_timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
@@ -182,19 +213,48 @@ def verify_passport(
|
||||
passport: Mapping[str, Any],
|
||||
clients: Mapping[str, Any],
|
||||
*,
|
||||
ref_slug_by_key: Mapping[str, str],
|
||||
anchors: Mapping[str, Mapping[str, Any]] | None = None,
|
||||
cache=None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Batch helper: run verify_citation over every entry in the passport's
|
||||
literature_corpus[]. `anchors` is the {ref_slug: anchor-marker} join map
|
||||
parsed from writer prose (the v3.7.3 <!--anchor:kind:value--> markers); each
|
||||
entry's anchor is looked up by its ref_slug (absent → anchor_present False).
|
||||
Threading the join here keeps verify_citation a pure per-citation unit."""
|
||||
literature_corpus[].
|
||||
|
||||
`ref_slug_by_key` is the {citation_key: ref_slug} join map: corpus entries are
|
||||
keyed by citation_key, writer prose is keyed by ref_slug, and the join across
|
||||
them is the caller's (Stage 4→5 pipeline's) responsibility — the corpus entry
|
||||
never carries ref_slug itself (#332). An entry whose citation_key has no joined
|
||||
ref_slug raises ValueError rather than emitting a contract-invalid summary;
|
||||
the per-entry summary contract requires a non-null string ref_slug, so a
|
||||
missing join is a caller error, not a silently-defaulted field.
|
||||
|
||||
`anchors` is the {ref_slug: anchor-marker} join map parsed from writer prose
|
||||
(the v3.7.3 <!--anchor:kind:value--> markers); each entry's anchor is looked
|
||||
up by its (joined) ref_slug (absent → anchor_present False). Threading both
|
||||
joins here keeps verify_citation a pure per-citation unit.
|
||||
|
||||
ref_slug_by_key is 1:1 (one summary row per corpus entry); per-prose-occurrence
|
||||
verification (one entry under several ref slugs) is a different API shape, out
|
||||
of scope here.
|
||||
"""
|
||||
corpus = passport.get("literature_corpus") or []
|
||||
anchors = anchors or {}
|
||||
return [
|
||||
verify_citation(
|
||||
entry, clients, anchor=anchors.get(entry.get("ref_slug")),
|
||||
cache=cache)
|
||||
for entry in corpus
|
||||
]
|
||||
outcomes: list[dict[str, Any]] = []
|
||||
for entry in corpus:
|
||||
citation_key = entry.get("citation_key")
|
||||
ref_slug = ref_slug_by_key.get(citation_key)
|
||||
if not _is_valid_ref_slug(ref_slug):
|
||||
# Missing join (None) OR a present-but-empty/non-string slug — both
|
||||
# fail the per-summary contract. Caught here (not just in
|
||||
# verify_citation) so the error names the offending citation_key,
|
||||
# which the per-citation layer doesn't have (#332).
|
||||
raise ValueError(
|
||||
f"no valid ref_slug joined for citation_key {citation_key!r} "
|
||||
f"(got {ref_slug!r}): the citation_key→ref_slug prose join must "
|
||||
"cover every corpus entry with a non-empty string "
|
||||
"(corpus entries do not carry ref_slug; #332)"
|
||||
)
|
||||
outcomes.append(verify_citation(
|
||||
entry, clients, ref_slug=ref_slug,
|
||||
anchor=anchors.get(ref_slug), cache=cache))
|
||||
return outcomes
|
||||
|
||||
@@ -8,10 +8,13 @@ literature_corpus[], and prints the list of per-citation summaries as JSON. A
|
||||
standalone entry point for ad-hoc verification, separate from the Stage 4->5
|
||||
audit pipeline.
|
||||
|
||||
Anchor note: the v3.7.3 anchor lives in writer prose (the <!--anchor:...-->
|
||||
markers), not in literature_corpus. This ad-hoc CLI has no prose document to
|
||||
join against, so every `anchor_present` is False. The prose-marker join (passing
|
||||
an {ref_slug: anchor} map into verify_passport) is wired by the Stage 4->5
|
||||
ref_slug note (#332): both the ref_slug AND the anchor live in writer prose (the
|
||||
<!--ref:slug--> / <!--anchor:...--> markers), not in literature_corpus. The
|
||||
summary contract REQUIRES a non-null string ref_slug, so a passport-only CLI
|
||||
cannot honestly emit a summary — by default it REFUSES (nonzero exit). Pass
|
||||
`--synthetic-ref-slug citation_key` to synthesize ref_slug from citation_key for
|
||||
DIAGNOSTIC output (warned on stderr, not a real prose join). The real
|
||||
{citation_key: ref_slug} + {ref_slug: anchor} joins are wired by the Stage 4->5
|
||||
pipeline / formatter batch, not by this standalone tool.
|
||||
|
||||
Spec: docs/design/2026-05-21-v3.10-182-promote-citation-gate-spec.md §2 Delta 5.
|
||||
@@ -60,6 +63,11 @@ def run(argv: list[str] | None = None, *, clients_factory=_real_clients) -> int:
|
||||
description="Verify citation existence across a Material Passport.",
|
||||
)
|
||||
parser.add_argument("passport", help="Path to the passport YAML file.")
|
||||
parser.add_argument(
|
||||
"--synthetic-ref-slug", choices=["citation_key"], default=None,
|
||||
help="Synthesize ref_slug from each entry's citation_key for DIAGNOSTIC "
|
||||
"output (the tool refuses by default; this is not a real prose "
|
||||
"join, #332).")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
path = Path(args.passport)
|
||||
@@ -74,7 +82,31 @@ def run(argv: list[str] | None = None, *, clients_factory=_real_clients) -> int:
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
|
||||
outcomes = verify_passport(passport, clients=clients_factory())
|
||||
# Refuse by default — a passport alone carries no prose <!--ref:slug--> join,
|
||||
# so it cannot produce a contract-valid summary (see module docstring, #332).
|
||||
corpus = passport.get("literature_corpus") or []
|
||||
if args.synthetic_ref_slug is None:
|
||||
print(
|
||||
"[verify_passport ERROR] cannot emit citation_verification_summary "
|
||||
"from a passport alone: ref_slug is a prose-sourced join "
|
||||
"(<!--ref:slug--> markers) that a passport does not carry. Run the "
|
||||
"Stage 4->5 pipeline (which supplies the prose join), or pass "
|
||||
"--synthetic-ref-slug citation_key for diagnostic output.",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
|
||||
# synthetic mode: ref_slug := citation_key. Diagnostic only.
|
||||
ref_slug_by_key = {
|
||||
e.get("citation_key"): e.get("citation_key") for e in corpus
|
||||
}
|
||||
print(
|
||||
"[verify_passport WARNING] --synthetic-ref-slug citation_key: ref_slug "
|
||||
"synthesized from citation_key. Output is DIAGNOSTIC, not a real prose "
|
||||
"join — do NOT feed it to a consumer that expects prose-joined ref_slugs.",
|
||||
file=sys.stderr)
|
||||
|
||||
outcomes = verify_passport(
|
||||
passport, clients=clients_factory(), ref_slug_by_key=ref_slug_by_key)
|
||||
print(json.dumps(outcomes, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
Reference in New Issue
Block a user