2 Commits

Author	SHA1	Message	Date
Edward Cheng-I Wu	d28c2974df	fix: migrate human-subjects authority references (#680) (#693)	2026-08-09 10:24:02 +08:00
Edward Cheng-I Wu	a3b95fd56b	feat(v3.6.7 Step 6 Phase 6.8): pattern-eval fixtures + harness + manifest validator + wrapper E2E (11 codex rounds, 0 P1) (#67) ... * feat(v3.6.7 Step 6 Phase 6.8): pattern-eval fixtures + harness + manifest validator + wrapper E2E test Spec: docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md §7 (Step 8 eval) + Phase 6.1 deferred wrapper E2E. - 17 micro-fixtures under tests/fixtures/v3_6_7_pattern_eval/ (A1-A5 narrative-side, B1-B5 instrument-side, C1-C3 publication-side, D1-D4 cross-cutting). Each: manifest.json + upstream_context/{passport_snippet.yaml,prior_artifacts/} + bad_run/{deliverable.md,expected_audit_findings.yaml,expected_orchestrator_action.yaml} + good_run/{...}. D2 special case: convergence-theatre PASS verdict + expected_d2_convergence_assertion field per §7.4 criterion 1. - 1 chapter-level integration fixture under integration/chapter_level_run/ exercising A3+C2+D4+C1 across 3-round MATERIAL → ship_with_known_residue escalation per §7.3. round_{1,2,3}/<agent>/<files> + escalation/{user_response,expected_passport_state,expected_pipeline_outcome}.yaml. - scripts/check_pattern_eval_manifest.py — fixture_kind discriminator routes between micro (§7.2) and integration (§7.3) JSON Schema 2020-12 manifest schemas; coverage cross-check enforces 17/17 numbered pattern IDs covered. 23-test suite at scripts/test_check_pattern_eval_manifest.py. - scripts/test_pattern_eval_runtime.py — 69-test pytest harness reading expected_audit_findings.yaml as synthesized verdict and asserting against expected_orchestrator_action.yaml. Per-pattern parametrized: BAD signal matches, GOOD passes, run_id F1 regex compliance, BAD/GOOD run_id uniqueness, integration manifest structural checks, §7.5 17-of-17 coverage. - scripts/test_run_codex_audit_e2e.py — Phase 6.1 deferred verification gate (Linux Bash 4+ only; macOS stock Bash 3.2 self-skips). Mocks codex CLI via PATH-prefix shim emitting canonical Phase 2 JSONL stream + canonical Section 6 verdict text. Validates wrapper produces 4 contract files (jsonl/sidecar/verdict/proposal entry) + 3 diagnostic files; proposal entry validates against audit_artifact_entry.schema.json --mode proposal (verified_at/verified_by absent per Pattern C3 defense); --dry-run writes nothing; --round=2 without --previous-findings rejected with EX_USAGE. - .github/workflows/spec-consistency.yml — added 4 CI steps: Phase 6.8 manifest validation, pattern-eval-unit (micro fixtures), pattern-eval-integration (integration fixture), and Phase 6.1 wrapper E2E (Linux runner only via runner.os == 'Linux' guard). - docs/design/TODO-l-doc-1-18-patterns-prose-retirement.md — files L-doc-1 follow-up ticket per §9.2 enumerating 8 retirement locations (4 in upstream main spec + 4 in this spec) for "18 patterns" → "17 patterns" docs-only PR. - spec §9.2 L-doc-1 row updated to point at the new TODO file. Verification: - python3 scripts/check_pattern_eval_manifest.py: 17/17 micro-fixtures + 1 integration valid; coverage 17/17. - pytest scripts/test_pattern_eval_runtime.py + test_check_pattern_eval_manifest.py: 97/97 passed. - Full repo regression: 704 pytest passed + 3 skipped (Bash 3.2 E2E gate) + 251 unittest passed = 955 green. Phase 6.1 wrapper E2E is the deferred §10 verification gate originally noted in PR #55 as "macOS Bash 3.2 blocking inside session." CI now exercises full dispatch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 1 — 8 findings closed (4 P1 + 4 P2) Round 1 trajectory: P1×4 + P2×4 + P3×1 → P3×1 remaining (deferred — see below). P1 closures: - F-001 schema fidelity: D4 BAD had rogue `pattern:` key + missing required fields; integration verdicts used `i101..i303` non-hex run_id suffixes; integration finding IDs `A3-r1-finding-1` etc. didn't match `^F-[0-9]{3,}$`. Fix: rewrote D4 BAD to schema-conformant shape; sed-replaced integration suffixes `iNNN` → `fNNN` (hex); renamed integration finding IDs to F-101/F-102/F-103/F-104 (round 1) / F-201/F-202 (round 2) / F-301/F-302 (round 3) — encoding round + agent index for traceability while satisfying schema. Augmented manifest validator to validate every fixture's expected_audit_findings.yaml against audit_verdict.schema.json (closes the gap that allowed the brief-driven drift to ship). - F-002 pattern_to_dimension_mapping: brief table was wrong against shared/templates/codex_audit_multifile_template.md §3 "Patterns surfaced" lines. Fix: rebuilt the harness's PATTERN_TO_DIMENSION map and swept all 17 micro-fixture BAD verdicts: A1 3.1→3.4 (cross-section coherence), A2 3.3→3.2 (hallucination), A3 3.2→3.1 (cross-reference), A4 3.1→3.3 (primary-source integrity), A5 3.4→3.2 (hallucination), C2 4(f)→3.7 (COI/disclosure), C3 3.6→3.2 (output metadata claim / hallucination), D1 3.1→3.4 (cross-file coherence), D3 3.7→3.6 (round framing). C1 stays 4(f), D4 stays 4(f), B1-B5 stay 3.5. Integration: round 1 C2 finding 4(f)→3.7; synthesis A3 across all rounds 3.2→3.1. - F-003 lifecycle_rule_not_mechanical: PHASE_TO_PASSPORT_MUTATION was partial (omitted P-PB-dup-* / P-PB-consume-fail / P-PB-crash). Fix: extended to all 24 §5.6 inventory rows + 2 happy-path phases = 26 entries. Duplicate / consume / crash rows marked "conditional" since their final passport state depends on prior session outcome (per §5.6 prose). Added EXPECTED_PHASE_COUNT assertion + test_every_fixture_phase_in_inventory coverage check that fails if any fixture references a non-inventoried phase. - F-004 integration_harness_gap: prior tests only checked manifest fields / directories. Fix: added 4 new tests: * test_integration_round_per_agent_verdicts_validate (parametrized over rounds 1/2/3) — drives each per-round per-agent expected_audit_findings.yaml through schema + status/count consistency + finding-tally checks; asserts at least one agent emits the manifest's declared expected_verdict per round. * test_integration_round_pipeline_state_consistent (parametrized) — asserts expected_pipeline_state.yaml's audit_artifact_appended[].run_id matches per-agent verdict run_ids for that round. * test_integration_escalation_passport_consistent — asserts user_response.acknowledged_finding_ids == manifest.expected_acknowledgement_finding_ids == passport.audit_artifact[].acknowledgement[].finding_ids set; user_choice == ship_with_known_residue. * test_integration_acknowledged_findings_exist_in_round_3 — closes the §3.7 family A row A4 hand-edit attack surface where ack ids reference non-existent findings. P2 closures: - F-005 integration_ack_ids: manifest expected `C2-r3-finding-1` but round 3 has no C2 finding (C2 closed in round 2 fix). Manifest realigned to F-301 (synthesis A3 round-3 residue) + F-302 (compiler D4 round-3 residue), matching what user_response.yaml and expected_passport_state.yaml acknowledge. - F-006 expected_action_not_checked: D4 BAD had impossible expected_path=A + expected_phase=B10 (Path A doesn't append, B10 is Path B). Harness never asserted expected_path or block_message. Fix: D4 BAD expected_path A→B; harness now asserts expected_path matches phase semantics (P-PA-* / A7 → A; P-PB-* / B10 → B); BLOCKING verdicts must carry "[AUDIT GATE" substring in block_message; PASS verdicts must have empty block_message. - F-007 manifest_path_escape: validator did `base / rel` + `exists()` only, allowing absolute paths or `..` escaping the fixture dir. Fix: added _validate_fixture_path_safety rejecting absolute paths, `..` segments, and paths resolving outside the fixture base via `target.relative_to(base.resolve())`. - F-008 ci_wiring_contract: spec §7.6 said "two separate test jobs" but Phase 6.8 ships them as two consecutive steps within the existing spec-consistency job (consistent with the 22 sibling lint steps already in spec-consistency.yml). Fix: added §7.6 deployment note explaining named-step deployment satisfies the unit-vs-integration legibility intent and pointing forward to a v3.6.8+ workflow refactor as the right boundary for promoting these to top-level jobs. P3 deferred: - F-009 audit_note_convention: codex flagged that the requested `<!-- AUDIT NOTE: <pattern-id> trigger/protection -->` convention isn't applied across all 34 micro deliverables. Brief never mandated this convention (only Team C/D added it organically). Treating as "convention is desirable but optional" — not adding 34 comments mechanically; would surface again as P3 in round 2 if codex still considers it load-bearing. Test counts: scripts/ pytest 714 passed + 3 skipped (Bash 3.2 E2E gate); scripts/ unittest 145 passed. Total 859 green vs round-1's 955 (the integration harness expansion absorbed some test counts; net new logic with new tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 2 — 3 findings closed (1 P1 + 2 P2) Round 2 trajectory: P1×1 + P2×2 → 0 P1 + 0 P2 (target). Closures: - F-201 P1 integration_harness_coverage: prior tests were static-consistency only. Implemented test_integration_state_runner_drives_full_pipeline that drives the §7.3 5-step procedure end-to-end: Step 1+2: load each round's per-agent verdicts + drive §5.6 Path B B10 decision per agent. Step 3: assert expected_pipeline_state.yaml matches accumulated passport per round. Step 4: at round-3 MATERIAL, emit escalation_prompt + feed escalation/user_response.yaml. Step 5: append acknowledgement entries per §5.4 + assert expected_passport_state.yaml matches. Plus _simulate_round helper accumulating passport state across rounds. Added B11 (round-cap escalation) to PHASE_TO_PASSPORT_MUTATION (now 27 phases). Round 3 synthesis_agent + report_compiler_agent fixtures' expected_phase B10 → B11 to reflect §5.4 escalation semantics; research_architect_agent stays B10 (PASS). - F-202 P2 finding_id_lineage: audit-template Section 6 line 157 contracts cumulative numbered findings carry forward IDs from round 1; new findings get next available ID. Round-1 fix renumbered these as F-201/F-301 etc. (round-prefixed) which broke the lineage. Reverted: A3 stays F-101 across rounds 1+2+3 (same partial-fix pattern); D4 stays F-103 across rounds 1+2+3. C2/C1 close in round 2 and never recur; F-102/F-104 stay round-1 only. Manifest acks + user_response acks + expected_passport ack finding_ids + expected_pipeline_outcome all updated to F-101 + F-103. New test_integration_finding_id_lineage_carry_forward enforces this. - F-203 P2 dimension_fidelity: B5 was mapped to §3.5 (instrument quality aggregate) but audit template §3.5 line 100 enumerates B1-B4 only; B5 surfaces in §3.1 (primary-source list) line 76. Fix: PATTERN_TO_DIMENSION B5 3.5→3.1; tests/fixtures/v3_6_7_pattern_eval/B5/bad_run/expected_audit_findings.yaml dimension 3.5→3.1. Round-2 retired: codex agreed F-009 audit-note convention should not be enforced — brief did not mandate it. Test counts: scripts/ pytest 716 passed + 3 skipped; scripts/ unittest 145 passed. Total 861 green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 3 — 2 findings closed (2 P2) Round 3 trajectory: P2×2 → 0 P1 + 0 P2 + 0 P3 (target). Closures: - F-301 P2 b11_phase_assertion: prior `-k` filter ('micro or run_id or coverage' / 'integration') in spec-consistency.yml skipped test_phase_inventory_complete and test_every_fixture_phase_in_inventory which carry no matching keyword. Plus _simulate_round didn't enforce B10 vs B11 split — round-3 MATERIAL fixture could regress to B10 silently. Fix: extended unit -k filter to include 'phase_inventory or fixture_phase'; _simulate_round now asserts that round-N MATERIAL agent at round_n == target_rounds emits B11 (escalation) and round_n < target_rounds emits B10. - F-302 P2 outcome_lineage: expected_pipeline_outcome.yaml had F-103 listed in BOTH closed_findings and acknowledged_residue (impossible — closed and acknowledged are exclusive lifecycle terminals). The harness only checked stage_outcome / audit_gate_outcome strings, so the contradiction was untested. Fix: removed F-103 from closed_findings (D4 was carried across rounds 1→2→3 and acknowledged, never closed); added closed/acked disjointness assertion + acked-set equality check against user_response in test_integration_state_runner_drives_full_pipeline. Test counts: 109 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 4 — 1 finding closed (1 P2) Round 4 trajectory: P2×1 → 0. Closure: - F-401 P2 integration_state_subset: per §7.3 lines 2089/2092, harness MUST compare expected pipeline/passport state to actual state. Prior implementation only compared run_id sets + ship_or_block label + ack pairs. Fix: extended Step 3 + Step 5 of test_integration_state_runner_drives_full_pipeline: Step 3 now compares per-agent {run_id, agent, verdict_status} dicts (round implicit from directory) + ship_or_block + overall_verdict + findings_summary{p1,p2,p3} + escalation_prompt_emitted flag. Step 5 now compares full passport {run_id, agent, verdict_status, round} sequence + outcome stage_outcome + audit_gate_outcome + proceed_to_next_stage + final_verdict_summary{rounds_used, target_rounds, unaddressed} fields. Test counts: 109 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 5 — 3 findings closed (1 P1 + 1 P2 + 1 P3) Round 5 trajectory: P1×1 + P2×1 + P3×1 → 0. Closures: - F-501 P1 integration_path_a_reset_axis_missing: spec §7.3 line 1990 lists Path A vs Path B fall-through as a structural axis but every integration step declared expected_path: B. Fix: added _drive_path_a_reverification() helper to test_integration_state_runner_drives_full_pipeline. Before each subsequent round (round 2, round 3) the harness simulates Path A re-verification of every prior-round persisted entry via A7 happy path (NO new passport append per §5.6 A7 invariant). Asserts ≥6 Path A legs fired (3 agents × rounds 2+3) and all reach A7. Path A_reverification_log now serves as the §7.3 axis evidence. - F-502 P2 integration_state_subset_not_closed: dead-code expected_passport_proceed assigned but never asserted; expected_user_options + proceed_to never compared. Fix: added Step 3e per-round expected_user_options assertion (round-cap escalation_prompt → §5.4 trio of ship_with_known_residue/another_round/abort_stage; non-final block → revise/re-audit option present). Step 5 now asserts passport.stage_outcome + passport.proceed_to (must be stage_3/4/5). - F-503 P3 stale_concrete_manifest_snippet: spec §7.3 example payload still listed `["A3-finding-1", "C2-finding-1"]` but actual fixture uses F-101/F-103 (carry-forward IDs from round 1, not pattern-name-encoded). Fix: spec line 2079 example updated to `["F-101", "F-103"]` matching the fixture. Test counts: 109 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 6 — 3 findings closed (2 P2 + 1 P3) Round 6 trajectory: P2×2 + P3×1 → 0. Closures: - F-601 P2 integration_state_equality_still_subset: tightened expected_user_options + proceed_to assertions to exact equality. Round-3 escalation must be EXACTLY {ship_with_known_residue, another_round, abort_stage} (not "any superset"); non-final block must include EXACTLY one revise/re-audit option AND abort_stage. proceed_to must equal "stage_3" exactly (not range stage_3/4/5). - F-602 P2 integration_expected_action_fields_unasserted: per-agent expected_orchestrator_action fields previously checked only at micro-fixture tests. Now _simulate_round() asserts: (a) expected_path ∈ {A, B}; (b) phase→path consistency (P-PA-* / A7 → A; P-PB-* / B10 / B11 → B); (c) phase→passport_mutation per PHASE_TO_PASSPORT_MUTATION; (d) BLOCKING verdicts carry "[AUDIT GATE" in block_message; non-blocking carry "". An impossible integration action like expected_path: A + expected_phase: B11 now fails. - F-603 P3 stale_phase_name_in_success_criterion: spec §7.4 line 2101 example listed "escalation" as the round-3 MATERIAL phase but the implemented + fixture-backed phase is B11. Fix: spec line 2101 example "escalation" → "B11". Test counts: 109 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 7 — 2 findings closed (2 P2) Round 7 trajectory: P2×2 → 0. Closures: - F-701 P2 integration_user_options_still_not_exact: tightened non-final block options to EXACTLY 2 entries (one revise/re-audit + abort_stage). Extras now rejected. - F-702 P2 integration_phase_contract_self_referential_for_nonblocking: extended _simulate_round phase contract to PASS + MINOR verdicts. PASS may only be B10 (fresh merge) or A7 (Path A re-verify); MINOR is B10. B11 is reserved for round-cap MATERIAL escalation. A PASS step declaring B11 or P-PB-* is now rejected. Test counts: 109 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 8 — 3 findings closed (3 P2) Round 8 trajectory: P2×3 → 0. Closures: - F-801 P2 micro_phase_contract_self_referential: extended R7 PASS/MINOR phase contract to micro BAD + GOOD test functions. Micro fixtures don't exercise round-cap, so B11 is rejected at the micro level. PASS micro fixtures (D2 special) require phase ∈ {B10, A7}; MINOR/MATERIAL/AUDIT_FAILED require phase=B10; GOOD always {B10, A7}. - F-802 P2 b11_escalation_message_accepts_b10_block_prefix: round-3 B11 fixtures and harness conflated B10 [AUDIT GATE: proposal] with B11 escalation. Fix: round_3 synthesis + report_compiler expected_block_message updated to "[ESCALATION] round 3/3 MATERIAL — choices: ship_with_known_residue / another_round / abort_stage"; harness now branches per-phase (B11 → "[ESCALATION]" + three §5.4 choice tokens; B10 → "[AUDIT GATE"). - F-803 P2 duplicate_micro_fixture_directory_not_rejected: validator's directory-name match was conditional on `expected in PATTERN_IDS`, allowing an `A1_copy` directory carrying `pattern_id: A1` to pass while overwriting the legitimate dict entry. Fix: validator now hard-fails when directory name is not a known pattern ID, AND when pattern_id mismatches directory name. Test counts: 109 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 9 — 3 findings closed (2 P2 + 1 P3) Round 9 trajectory: P2×2 + P3×1 → 0. Closures: - F-901 P2 failure_phase_synthetic_injections_missing: spec §7.3 line 2093 promised "Phase 6.8 harness's per-phase synthetic injections" but harness only checked phase map size. Fix: added parametrized synthetic injection tests covering all 26 §5.6 inventory phases. test_synthetic_inject_none_mutation_phase parametrized over 19 "Passport mutation: none" phases (must NOT append). test_synthetic_inject_append_mutation_phase parametrized over 3 "appended" phases (must append exactly one entry). test_synthetic_inject_conditional_phases_documented covers the 4 conditional dup/crash rows. - F-902 P2 integration_supersession_axis_not_exercised: spec §7.3 line 2093 claims "representative Path B-supersession happy-path" but harness had no A1.5 / supersession_required logic. Fix: added test_synthetic_supersession_preflight_path_b_filters_higher_round (F-070 closure regression — B2 supersession-mode filter selects round-3 user-dispatched over round-1/round-2 leftover) + test_synthetic_supersession_empty_after_filter_blocks (F-072 closure — empty after filter MUST emit P-PB-supersede-missing BLOCK, not silently fall back). - F-903 P3 c2_minor_stale_material_prose: spec §7.4 line 2099 said "16 of 17 micro-fixtures the failure signal is MATERIAL" but C2 BAD ships as MINOR by design (P3 reflexivity finding) and harness allows MATERIAL OR MINOR. Fix: spec line 2099 prose now reads "15 of 17 ... MATERIAL; for C2 ... MINOR with one P3 reflexivity-temporal-ambiguity finding; for D2 ... non-finding convergence-policy assertion". Test counts: 109 → 135 (26 new tests: 19 none-mutation + 3 appended-mutation + 1 conditional-doc + 2 supersession + 1 D2 lineage). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): codex review round 10 — 1 finding closed (1 P2) Round 10 trajectory: P2×1 → 0. Closure: - F-1001 P2 round9_synthetic_tests_not_wired_to_ci: R9 added 26 test_synthetic_* tests but CI workflow's -k filters omitted them. Codex verified empirically: -k "synthetic" collected 26 tests not running on CI. Fix: extended unit step -k filter to include "synthetic and not supersession" (24 phase injection tests); extended integration step -k filter to include "synthetic and supersession" (2 A1.5 tests). Both legs verified locally collect 95+17=112, full 135 with manifest validator. Test counts: 135 passed, all wired to CI now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v3.6.7 Phase 6.8): CHANGELOG [Unreleased] entry 11 codex review rounds converged to 0 findings; cumulative 24 findings closed (4 P1 + 18 P2 + 2 P3) across rounds 1-10. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): wrapper E2E test uses repo-relative --output-dir CI failure: wrapper rejects absolute --output-dir paths with EX_USAGE. Fix: pass "audit_artifacts" (repo-relative) instead of str(repo / "audit_artifacts"). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v3.6.7 Phase 6.8): wrapper E2E mock emits parse_audit_verdict-compatible Section 6 text CI failure: wrapper exit 70 (AUDIT_FAILED) — parse_audit_verdict.py rejected mock's "Verdict: PASS / Findings — P1: 0..." format. Parser expects "Round N: P1×n / P2×n / P3×n (N total)" or "Round N: 0 findings of any severity. Convergence reached." as the LAST non-empty line of verdict text. Fix: mock now emits the zero-findings convergence form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 15:37:15 +08:00