mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
Edward Cheng-I Wu
d0663404ff
## #260 — Experiment Provenance Intake + claim→experiment alignment (schema-first) Adds the **intake + alignment** layer for experiment-backed claims (Kong et al. 2026, arXiv:2605.18661, §3.3 + §7.4.3). ARS deliberately keeps experiment **execution** outside the pipeline — the scholar runs experiments externally and brings results back; this change records that provenance and audits manuscript claims against it. Explicit non-goals (carried verbatim into the gate wording): does not run experiments, does not judge whether one was correctly designed/run/statistically-adequate/reproducible, does not auto-fill provenance, does not require provenance for literature-only pipelines. ### What ships **Block A — `experiment_provenance[]` aggregate** (`experiment_provenance_entry.schema.json`): each scholar-entered entry carries a nested `repro_lock` (same inline-object shape as the passport-level lock, re-declared — not `$ref`'d — because the source is inline prose, not a schema file), a `planned_vs_executed[]` record (each `executed:false` unit carries a gate-checked `skip_reason`), and `negative_results[]` / `known_limitations[]` whose **key must be present** (empty `[]` = well-formed advisory; absent key = malformed → gate FAIL, the absent-key rule ported from #261 C3). **Block B — claim→experiment alignment**: the claim manifest gains an optional per-claim `planned_experiment_ids[]` join field, plus a **fourth ref_slug-less aggregate** `experiment_alignment_results[]` (`experiment_alignment_result.schema.json`) with a MECE verdict enum `{ALIGNED, OVERSTATED, NOT_SUPPORTED_BY_PROVENANCE, PROVENANCE_INSUFFICIENT}`. The verdict is **produced by the integrity verification agent AT the gate** (Stage 2.5 sampling / 4.5 full), mirroring #261 C3 — so the row is emitted and gated in the same pass, avoiding the stage-ordering race. A **mixed-evidence claim** (both `planned_refs` and `planned_experiment_ids`) is audited by both paths; the gate decision is **worst-verdict-wins**. `PROVENANCE_MISSING` is deliberately NOT a verdict — a dangling `experiment_id` is a structural lint FAIL, never a fake judge row. ### Invariants + enforcement layering Seven cross-array lint invariants in `check_claim_audit_consistency.py`: EP-INV-1 (experiment_id unique), EP-INV-2 (planned_experiment_ids resolve — rename + forward-reference guard), EP-INV-3 (experiment ids ⟹ empirical; mixed allowed), EP-INV-4 (declaration↔provenance symmetry), EP-INV-5 (declaration well-formedness when present), EA-INV-1 (finding_id unique), EA-INV-2 (alignment-row references resolve). The persisted passport-level `experiment_intake_declaration` gives a **fail-closed legacy boundary, split by enforcement layer (stated precisely)**: the lint deterministically enforces declaration↔provenance *symmetry* (EP-INV-4) and *well-formedness* (EP-INV-5); the **integrity gate — not the lint — owns the `ars_version` numeric legacy decision and the declaration-presence FAIL** (a passport is `legacy_unknown` only with positive `ars_version < #260-constant` proof, everything else is treated-as-post-#260 so the declaration is REQUIRED). The `ars_version` numeric half is kept at the gate layer by design: the release constant it compares against is frozen at ship time, not at intake. ### Three documented departures from the issue's literal text Each corrected after a first-party read of the **tracked** repo: 1. `repro_lock` is an inline-prose object, not a schema file — so "inherit repro_lock" means nesting the shape, not `$ref`'ing a non-existent file. A shared `repro_lock_validation.py` single-sources the field set (imported by both checkers) with a drift test. 2. The claim manifest had **no** experiment pathway (`additionalProperties:false`, `planned_refs` is literature-only) — the join field is **added**, not assumed. 3. "Path X / Tier-1 required / writer-binding" are not named conventions in the tracked repo — the discipline is **described**, not cited by a name a reader cannot find. Two spec-table typos found during implementation were also corrected (the `report_compiler_agent` path and the pytest invocation path). ### Producers (taught in lockstep) Three manifest writers (`synthesis_agent` / `draft_writer_agent` / `report_compiler_agent`) emit `planned_experiment_ids` when an experiment backs a claim; the integrity agent gains a disclosure-only Phase carrying the POSITIONING non-goal verbatim; the orchestrator carries the aggregate + declaration forward; README intake detection sets the declaration. ### Tests `examples/passport_with_experiment_provenance.yaml` (2 experiments, a mixed-evidence claim, an OVERSTATED row) + full TDD suite: schema ±, fail-closed symmetry, declaration well-formedness, mixed-evidence two-row, verdict-derivation, mutation-verified non-vacuous invariants (each fixture flips fully clean when its invariant is neutralised), reverse-invariant producer pins, repro_lock drift, literature-only regression, and a documented D4-c carve-out boundary (the experiment carve-out is the LLM-caller's job; the deterministic detector is manifest-unaware — pinned explicitly so a future deterministic caller is on notice). Full suite green (2347 passed, 0 regression). ### Not a release CHANGELOG `[Unreleased]`. All schemas, the manifest field, and all seven invariants are additive and backward-compatible. Closes #260 🤖 Generated with [Claude Code](https://claude.com/claude-code)