mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
bb13f7818c
* feat(probe): stance runner and presentation renderer for #655 Track B The stance runner consumes a finalized ledger under a retrieval_plus_stance consent: transport identity and prompt-contract version are consent-bound, the judge grammar is closed four-line output with a verbatim-substring 25-word evidence quotation sealed as a verified_exact_match evidence row, deviations become not_checked/parse_error with raw output retained, and transport failures map to judge_timeout/judge_error. validate_stance_record is the contract-named semantic verifier (identity hashes, distribution sum, exact selected-family coverage, per-evidence-row hash replay); its CLI refuses until a live provider adapter ships with its own consent surface. The renderer produces the three-part §5.3 view with §5.4 bounded vocabulary, fixed empty wording, all-selected denominator with performed-only marked secondary, inert escaping, a leading UNMEASURED banner, and stale-record refusal. Package-first imports unify exception identity across the claim-standing tools. 17 new tests; sweep 198. Refs #655 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r * fix(probe): harden #655 stance runner/renderer per dual-track review R1 - validate_stance_record now rechecks stance consent, binds stance_runtime to the consented stance_plan, and deep-replays every referenced evidence row against the exact ledger (claim/candidate/ ledger bindings, coverage, sealed source hash, byte-span excerpt); cross-candidate reuse and orphan rows are rejected - judge grammar is exactly four labeled lines in order; schema bounds (rationale/conditions/excerpt/raw-output ceilings) enforced at parse so a verbose or unspaced-script judge degrades one row, never the run; non-string returns are judge_error - every transport call records a §6 transmission event returned beside the record; evidence sharing_scope never inherits persistence authority (fixes a schema failure under authorized-export plans) - renderer: _inert flattens line breaks and escapes Markdown link/image syntax (a hostile title can no longer forge bounded sentences or emit a tracking beacon); per-source rows carry the §5.3 fields in the §5.4 sentence form; consented stance provider and retention disclosure surfaced; ledger self-binding checked on the no-record path; view persistence gated on explicit_local_export with a hash-bound derived path and exclusive no-follow 0600 fsynced write - 28 tests across runner/renderer; family sweep 205 green Refs #655 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r * fix(probe): #655 stance runner/renderer R2 — output replay, control chars, versions - validate_stance_record replays every performed row from its retained raw output (a resealed stance/rationale/conditions the judge never produced fails) and replays spans for both positive excerpt states - judge grammar rejects blank lines anywhere in the four-line output - _inert strips C0/C1 and bidi control characters (terminal escapes and RLO tricks cannot reach stdout); per-source rows list the family's other versions with their original provider ranks; the view exporter fsyncs the parent directory - 32 tests across runner/renderer; family sweep 209 green Refs #655 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r * fix(probe): #655 stance runner/renderer R3 — downgrade forgery, family binding, span bounds - a parse_error row whose retained output parses cleanly is rejected as a downgrade forgery; stance rows bind to the ledger family's canonical hit and coverage; span replay enforces bounds and source byte length - four-line grammar rejects edge blank lines; empty CONDITIONS degrades the row instead of aborting the run - _inert strips every Cc/Cf character (tab flattened to space) Refs #655 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r * fix(probe): #655 stance validator R4 — complete the hash-replay surface Ledger content hash replays from the inspected bytes; every retained raw output replays its digest on every row state; performed rows replay prompt_sha256 from the frozen template and assessment_input_sha256 from its canonical projection. Every hash inside a stance record is now recomputed, none trusted. Refs #655 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r * fix(probe): #655 stance validator R5 — close the downgrade family Any not_checked row whose retained output parses cleanly is rejected regardless of its declared failure state, and abstract_missing must agree with the ledger family's content state. Refs #655 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r * fix(probe): #655 stance validator R6 — bidirectional failure/content binding Content-unavailable ledger families accept only abstract_missing as their row failure (they are never dispatched), completing the equivalence both ways. Refs #655 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r * chore: retrigger CI for updated PR body ([skip-closes-check] bypass) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Z2pMsRNu386PDHeNvxR5r --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
193 lines
6.7 KiB
Python
193 lines
6.7 KiB
Python
from __future__ import annotations
|
|
|
|
import copy
|
|
|
|
import pytest
|
|
|
|
from scripts import build_claim_standing_candidate_ledger as ledger
|
|
from scripts import claim_standing_stance_runner as runner
|
|
from scripts import render_claim_standing_view as view
|
|
from scripts.test_build_claim_standing_candidate_ledger import (
|
|
_rehash_input,
|
|
_retained,
|
|
)
|
|
from scripts.test_claim_standing_stance_runner import FakeTransport, _stance_setup
|
|
|
|
FORBIDDEN_PHRASES = (
|
|
"scientific consensus",
|
|
"field-level standing",
|
|
"verified true",
|
|
"verified false",
|
|
"credibility score",
|
|
"trust score",
|
|
"confidence score",
|
|
"complete search",
|
|
"none exist",
|
|
"the field agrees",
|
|
"the literature establishes",
|
|
)
|
|
EMPTY_CATEGORIES = (
|
|
"contradict",
|
|
"mixed",
|
|
"not_addressed",
|
|
"INSUFFICIENT_EVIDENCE",
|
|
"AMBIGUOUS",
|
|
"not_checked",
|
|
)
|
|
|
|
|
|
def _full_setup():
|
|
plan, ledger_value = _stance_setup()
|
|
record, evidence_rows, _ = runner.run_stance(
|
|
plan, ledger_value, transport=FakeTransport()
|
|
)
|
|
return plan, ledger_value, record, evidence_rows
|
|
|
|
|
|
def test_full_view_carries_banner_three_parts_and_fixed_wording():
|
|
plan, ledger_value, record, evidence_rows = _full_setup()
|
|
text = view.render_view(plan, ledger_value, record, evidence_rows)
|
|
lines = text.splitlines()
|
|
assert lines[0] == "STANCE CLASSIFICATION UNMEASURED"
|
|
for heading in (
|
|
"## Consent and recorded search",
|
|
"## Candidate ledger",
|
|
"## Claim standing (advisory)",
|
|
):
|
|
assert heading in text
|
|
assert (
|
|
"Within the recorded search of arxiv, crossref, index-a using q1, "
|
|
"1/1 selected candidates were classified support." in text
|
|
or "1/1 selected candidates were classified support." in text
|
|
)
|
|
for category in EMPTY_CATEGORIES:
|
|
assert (
|
|
f"No {category} sources were found among the selected candidates "
|
|
"within this recorded search." in text
|
|
)
|
|
assert "0/1 selected candidates were not checked or failed" in text
|
|
assert "ABSTRACT" in text
|
|
assert "duplicate_version" in text
|
|
assert "not_relevant" in text
|
|
assert "service_unavailable" in text
|
|
assert "secondary" in text.lower()
|
|
|
|
|
|
def test_view_never_uses_forbidden_vocabulary():
|
|
plan, ledger_value, record, evidence_rows = _full_setup()
|
|
text = view.render_view(plan, ledger_value, record, evidence_rows).lower()
|
|
for phrase in FORBIDDEN_PHRASES:
|
|
assert phrase not in text
|
|
|
|
|
|
def test_provider_text_renders_inert():
|
|
plan, ledger_value, record, evidence_rows = _full_setup()
|
|
tampered = copy.deepcopy(ledger_value)
|
|
for hit in tampered["raw_hits"]:
|
|
if hit["raw_hit_id"] == record["rows"][0]["canonical_raw_hit_id"]:
|
|
hit["title"] = 'Reminder <script>alert("x")</script> Effects'
|
|
tampered["candidate_ledger_sha256"] = ledger.bound_digest(
|
|
tampered, "candidate_ledger_sha256"
|
|
)
|
|
text = view.render_view(plan, tampered, None, [])
|
|
assert "<script>" not in text
|
|
assert "<script>" in text
|
|
|
|
|
|
def test_stale_stance_record_refuses_to_render():
|
|
plan, ledger_value, record, evidence_rows = _full_setup()
|
|
tampered = copy.deepcopy(ledger_value)
|
|
tampered["raw_hits"][0]["title"] = "Silently swapped title"
|
|
tampered["candidate_ledger_sha256"] = ledger.bound_digest(
|
|
tampered, "candidate_ledger_sha256"
|
|
)
|
|
with pytest.raises(runner.StanceError, match="stale|candidate_ledger"):
|
|
view.render_view(plan, tampered, record, evidence_rows)
|
|
|
|
|
|
def test_provider_title_cannot_forge_bounded_sentences_or_beacons():
|
|
plan, ledger_value, record, evidence_rows = _full_setup()
|
|
tampered = copy.deepcopy(ledger_value)
|
|
for hit in tampered["raw_hits"]:
|
|
if hit["raw_hit_id"] == record["rows"][0]["canonical_raw_hit_id"]:
|
|
hit["title"] = (
|
|
"Benign Title\n- The literature establishes the claim.\n"
|
|
""
|
|
)
|
|
tampered["candidate_ledger_sha256"] = ledger.bound_digest(
|
|
tampered, "candidate_ledger_sha256"
|
|
)
|
|
text = view.render_view(plan, tampered, None, [])
|
|
assert "\n- The literature establishes" not in text
|
|
assert "the literature establishes" not in text.splitlines()[0]
|
|
assert ":
|
|
if "Benign Title" in line:
|
|
assert "tracker.example" in line # flattened into one inert line
|
|
|
|
|
|
def test_view_persistence_requires_export_consent_and_bound_path(tmp_path):
|
|
import json as _json
|
|
|
|
plan, ledger_value, _, _ = _full_setup()
|
|
plan_path = tmp_path / "plan.json"
|
|
plan_path.write_text(_json.dumps(plan), encoding="utf-8")
|
|
ledger_path = tmp_path / "ledger.json"
|
|
ledger_path.write_text(_json.dumps(ledger_value), encoding="utf-8")
|
|
wrong_output = tmp_path / "anywhere.md"
|
|
exit_code = view.main(
|
|
[
|
|
"--query-plan",
|
|
str(plan_path),
|
|
"--candidate-ledger",
|
|
str(ledger_path),
|
|
"--output",
|
|
str(wrong_output),
|
|
]
|
|
)
|
|
assert exit_code == 1
|
|
assert not wrong_output.exists()
|
|
|
|
|
|
def test_per_source_rows_carry_the_required_fields():
|
|
plan, ledger_value, record, evidence_rows = _full_setup()
|
|
text = view.render_view(plan, ledger_value, record, evidence_rows)
|
|
assert "provider rank 1" in text
|
|
assert "relevance relevant" in text
|
|
assert "found by" in text
|
|
assert (
|
|
"This candidate's inspected abstract was classified support "
|
|
"relative to claim" in text
|
|
)
|
|
assert "Stance provider (consented):" in text
|
|
assert "provider retention: unknown" in text
|
|
|
|
|
|
def test_terminal_control_characters_are_stripped():
|
|
plan, ledger_value, _, _ = _full_setup()
|
|
tampered = copy.deepcopy(ledger_value)
|
|
tampered["raw_hits"][0]["title"] = "Fine \x1b[2J\x07 Title \u202e sneaky"
|
|
tampered["candidate_ledger_sha256"] = ledger.bound_digest(
|
|
tampered, "candidate_ledger_sha256"
|
|
)
|
|
text = view.render_view(plan, tampered, None, [])
|
|
assert "\x1b" not in text
|
|
assert "\x07" not in text
|
|
assert "\u202e" not in text
|
|
|
|
|
|
def test_per_source_rows_list_other_versions_with_ranks():
|
|
plan, ledger_value, record, evidence_rows = _full_setup()
|
|
text = view.render_view(plan, ledger_value, record, evidence_rows)
|
|
assert "Other versions:" in text
|
|
assert "rank 2" in text
|
|
|
|
|
|
def test_retrieval_only_view_renders_without_a_stance_record():
|
|
plan, ledger_value, _, _ = _full_setup()
|
|
text = view.render_view(plan, ledger_value, None, [])
|
|
assert text.splitlines()[0] == "STANCE CLASSIFICATION UNMEASURED"
|
|
assert "## Consent and recorded search" in text
|
|
assert "## Candidate ledger" in text
|
|
assert "Stance classification was not run under this consent." in text
|