diff --git a/CHANGELOG.md b/CHANGELOG.md index 940b4b89..13ffeace 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,8 @@ All notable changes to this project will be documented in this file. ### Fixed +- **Conformance machine lines tolerate Markdown decoration; Severity self-correction is escalation-only (#637).** The 2026-08-03 #610 baseline attempt aborted 3/6 panels on decoration alone: every synthesis first attempt (6/6) fenced the four mechanical audit lines, one retry re-emitted them as whole-line inline code spans, and one domain seat self-superseded its Severity declaration mid-card — all three shapes render the content verbatim and visibly, so by the #609 reader-visibility criterion these are false aborts, not leak guards. `check_panel_synthesis.py` gains `audit_candidate_lines`: the four mechanical audit-line grammars now also read fenced content (fence-marker lines dropped, the same open/close state machine as `strip_fences`) and the unwrapped bodies of whole-line inline code spans, normalized to their CommonMark-DISPLAYED form (padded-span one-space strip, indented-fence dedent); rejection rationales and the DA marker keep the plain-line source as a declared, test-pinned boundary. `_one_body` collapses same-value re-statements while two distinct parsed values stay a loud abort, so a disagreeing decoy can never be silently absorbed. `check_phase_conformance.py` accepts a multi-declaration Severity chain only when it strictly escalates (Minor < Major < Critical — the observed self-correction direction, ms01_quant r1's Major -> Critical) with a `[SEVERITY-SUPERSEDED]` advisory trail in the gate log; de-escalation (which could waive the Critical/Major Evidence-Anchor hard gate with one appended weaker line), non-monotone or repeated-value chains, and same-line pipe pairs all keep the abort. Dual-track review (security opus + codex gpt-5.6-sol xhigh) surfaced 8 advisory findings, all first-party verified and closed; round 2 converged clean on both tracks, including a differential proof that the post-fix abort set is a strict superset of the pre-fix one. 16 new pinning tests cover fence-state decoys after malformed/short/mismatched/Unicode-separator closers, CRLF, the operative-value anchor pin, and the padded-span conflict abort. + - **Monthly harness-retirement scope and debt wording no longer drift (#617).** The scheduled generator now checks out the repository and renders its issue body from `scripts/ars_phase_scope_manifest.json`, so the Bucket A inventory, per-skill counts, and agent names stay aligned with the authoritative manifest (currently 23 agents, including `timeline_extraction_agent`). The generated debt categories are model-agnostic, and regression tests cover manifest growth, deterministic CLI output, and removal of the stale inline workflow list. - **E4 dispatch harness: the evidence contract becomes structural (#608).** `reviewer-e4/2026-07-27` requires every checker-rejected response and its checker output to survive a retry, and the 2026-07-27 Spec-A fleet lost that on both launched panels (`observed_provenance_invalid_rate: 1.00`, gate verdict `NOT_COMPUTABLE`) because a retry wrote over the response it was retrying. That is not an operator mistake to correct by trying harder: the preservation step sat at the moment the operator was trying to get the run to proceed. New `scripts/dispatch_e4_panel.py` inverts the order — a response is written to an attempt-numbered path with `O_EXCL` **before** any checker may judge it, so preservation precedes the decision to retry instead of depending on it, and each checker invocation's own bytes are stored beside the response it judged. Checkers run from inside the bundle with relative paths, so nothing has to be stripped and every stored diagnostic is `verbatim`. Paper-blind and paper-visible calls now get separate whitelisted sandboxes and the blind one does not contain the manuscript at all, making blindness a filesystem fact rather than the seat's restraint (hand dispatch put every artifact in one directory); `evals/` is outside both, and the contamination fence is a PATH allowlist rather than a word denylist: the harness may read only the contract, the seven agent files, and the three manuscripts, so a manifest is unreadable and a future held-out artifact is unreadable by default. A word denylist was written first and measured to be worse than the failure it guarded against — `manifest` and `seeded` are ordinary review vocabulary and 5 of the 18 committed real panels of this set contain one, so gating assembled prompts would have aborted roughly a quarter of panels after all five cards existed, with no replacement draw permitted, which is a false-abort channel of the exact kind #609 was raised to remove. Ground-truth tokens in model output are now an advisory `leak_canary_hits` field, never a gate. The seat set is derived from the contract with `panel_size` asserted rather than hard-coded a third time; only a reviewer-conformance exit is retried, because §11 routes every exit-2 class to abort-no-retry and retrying one would file a `phase1_retries` event the evidence contract does not classify as a retry; and the one permitted Phase 2 recovery is recognised from the checker's own `[PROTOCOL-VIOLATION: multi_dissent=true]` line, now pinned by the checker's tests so a reword fails CI instead of silently killing a fleet. The four closed status fields are derived, with `provenance_status` computed by re-checking that each named location still resolves rather than by trusting the write path. The work directory mirrors the set's own tree — `runs/.json` beside `runs/raw//`, or the blocked namespace for an aborted panel — with every `*_location` already record-relative, so promoting a run is a copy and nothing is rewritten at commit time; that rewrite is what previously turned a verbatim diagnostic into a paraphrase. A completed panel carries `adjudication.status: "pending"` because the harness cannot fill `per_defect` without the held-out manifest and does not guess. A work directory inside the repository is refused with nothing written, since writing a record there is the act being refused, and an internal preservation fault produces a blocked record rather than a traceback: losing the record is the one failure mode this mechanism cannot afford. Cross-model review then found four more places where the harness was not yet dispatching the frozen shape or not yet durable, all fixed here: the field-analysis call sent a one-line placeholder instead of `field_analyst_agent.md`; the synthesizer received only its sprint-contract arithmetic subsection, which stops above its Editorial Decision Letter and Revision Roadmap instructions, so a panel could be score-eligible with neither artifact while the arithmetic checker still passed; a synthesis-layer failure (checker exit 1) aborted instead of being voided and re-run once with the diagnostics as delimited data, as §8.1 requires, so ordinary stochastic formatting blocked valid panels; and a transport timeout or missing binary escaped as a traceback with no record at all, while a nonzero exit truncated stderr to 400 characters inside a field stamped `verbatim`. A no-response event now writes its exact bytes and blocks the run without filing a retry event, which is what the contract says a re-dispatch after no response is. `--date` and `--fixture` are validated before they become path components (one separator relocated the evidence bundle, filed a blocked run under the scored namespace, or lost the record entirely); `read_prompt_material` refuses a path that reaches its target through a link, since an allowlist over names would otherwise authorize whatever the name points at; a record's `diagnostic` field is stripped of absolute local paths before a blocked record is committed to a public repo; and `seats_for` runs inside the handler so contract drift leaves a blocked record rather than a traceback. Two further review rounds found nine more, in the same two classes. What the panel was asked: the five Phase 2 calls never received the Phase 0 reviewer configuration cards, so a generic seat was dispatched where full mode defines a configured identity (the cards go to Phase 2 only, being paper-derived); the one permitted Phase 1 retry re-sent the identical prompt, and since each `claude -p` is a fresh conversation it could not learn the lint gap §4 requires be hinted; and the CLI was launched without `--bare`, so it auto-discovered the maintainer's user-level CLAUDE.md, hooks, plugins and auto-memory -- context the allowlist never authorized, reaching a paper-blind prompt -- while the record already attested "no session persistence" (`--bare` requires `ANTHROPIC_API_KEY`, an operational cost taken so the fence is true). What the record attests: `suite_commit` was read AFTER a tens-of-minutes panel and ignored a dirty tree, so it could name a commit the dispatched bytes never came from; it is now read before dispatch and paired with `suite_commit_reproducible`. Durability: repeating a blocked panel into the same work directory made `emit`'s bundle rename raise `ENOTEMPTY`, losing the record entirely while the traceback's exit 1 reads as this harness's `EXIT_BLOCKED` -- a fleet driver would file a blocked panel that has no record; relocation is now non-destructive and non-fatal and both `emit` call sites are wrapped. Honesty of attestation: `diagnostic_form` is derived from whether the text was rewritten instead of being stamped `verbatim` unconditionally, and the path scrubber's general strip-anything-path-shaped pass was removed after it was measured to corrupt ordinary diagnostics (`title/field/word_count` became `titleword_count`, a DOI URL lost its host). And the symlink refusal added one round earlier was itself a false-abort: comparing `resolve()` against `absolute()` refused every set root under a symlinked ancestor, which on darwin means every `/tmp` path and every pytest tmpdir, with a message blaming the manuscript; it is now scoped to components at or below the declared root, and its pin was rebuilt because the original passed on darwin with the fix removed. A third round then caught the previous round's own fix converting a loud crash into a silent one: making the bundle relocation non-fatal meant a repeated panel identity overwrote the first attempt's record and left it stamped `provenance_status: valid` while pointing at the first attempt's bundle -- a retry destroying the account of what it replaced, which is the failure this harness exists to eliminate, one layer above where it was fixed. A repeat is now refused before anything moves or is written, the location prefix is computed from where the bundle actually is, the record is written `O_EXCL` like the artifacts, and `locations_resolve_from` is enforced in `emit` rather than only in tests so a prefix mistake downgrades the attestation. The indirection refusal was extended from the set root to every allowlist family from its own base's parent (agent files are now sent whole, so a redirected one would deliver its entire target into a prompt, and a `manuscripts/ -> manifests/` directory link would otherwise BE the base and never be walked). Absolute paths are removed at the source rather than by a scrubber, since dropping the over-broad pattern had re-opened exposure for work directories under neither `REPO` nor `$HOME` -- the README's own `/tmp/...` shape. And a fail-closed preflight refuses before dispatch when neither `ANTHROPIC_API_KEY` nor an `apiKeyHelper` is visible, since `--bare` reads neither OAuth nor the keychain and discovering that as a transport abort costs the first call of a fleet. A fourth round found the two deliverables no checker looks at. `check_panel_synthesis.py` validates the audit lines and the arithmetic and exits 0 on a synthesis carrying no Editorial Decision Letter and no Revision Roadmap, and nothing validates the field analyst at all -- the harness's own clean-path fixture was the placeholder string `field analysis` and stayed score-eligible, which is a demonstration that an unconfigured panel could be measured. Both are now required by the harness, since it is the only place an absent deliverable can be caught, and the fixtures carry the real sections. The synthesis call is paper-visible again, as the committed 2026-07-25 artifact records it: a blind synthesizer cannot check a disputed reviewer claim against the paper, which is most of what arbitration is. A nonzero exit or a timeout that nonetheless produced output now preserves that output as a `partial-response` artifact and says so, because the contract's no-response carve-out applies only when there is no response. The auth preflight passes `--settings` when an `apiKeyHelper` is the credential path, which it previously accepted without ever supplying. And the run's own work directory and set root join `REPO` and `$HOME` as scrubbed prefixes, exact rather than heuristic, since the README's own `/tmp/...` shape is under neither. One more, found by walking §2 step by step rather than by waiting for a reviewer: step 1 requires the contract be prepared and validated before dispatch, and the harness was sending the on-disk template verbatim. The template carries no `generated_at`; the committed 2026-07-27 bundle's contract does, because the real dispatch populated it, so the seats were receiving a different contract from the one the frozen shape sends -- and step 1's abort-on-error gate was absent entirely, so a malformed contract would have reached all five seats unchecked. `prepare_contract` stamps `generated_at` (`--generated-at`, defaulting to now in UTC) and `validate_contract` runs `check_sprint_contract.py` on the prepared object, refusing before the first call. A fifth round found five more, three of them created by the fixes above. Every seat was receiving all five reviewer configuration cards rather than its own, which SKILL.md Iron Rule #2 forbids and whose anti-pattern table names overlap suppression as "unexecutable under blindness" -- shipping the panel's angles to each seat is what would make it executable, and a suppressed finding is a MISSED in strict recall, so the fix for one instance would have depressed the metric the fleet exists to produce; each seat now gets only its own card, and a seat with none is told so rather than handed its peers'. The contract validator staged its file inside the work directory and ran before the auth preflight, so a refusal printing "nothing was written" wrote anyway, poisoned the emptiness precondition, and turned the documented fix-and-retry into a blocked record for a panel that never dispatched, consuming the panel identity; it now stages in a scratch directory after the preflight. The synthesis call passed `diagnostics` into the `manuscript` parameter, so every first attempt received `None` and every retry received the checker transcript as the paper -- caught only because a reviewer read the call site, since the existing test exercised the builder rather than the wiring. Instructions and data are now sent as the system and user halves §2 names rather than concatenated into one user message; a malformed contract leaves a blocked record instead of an uncaught exit; the `` block carries the Iron Rule #7 data-boundary sentence its two siblings already had; and a filesystem-root work directory can no longer reduce a scrub prefix to `/` and delete every slash in a diagnostic. A sixth round found six more. A seat that follows its own instruction -- on two or more dissents, abort with `[PROTOCOL-VIOLATION: multi_dissent=true]` INSTEAD of drafting a card -- failed the checker at parse_report, whose output then never carries the token, so a compliant seat lost the one §5 recovery; the token-only shape (strictly one non-blank line, so it cannot be smuggled inside a real card) is now recognised from the response itself. Card #4's slice ran to end-of-analysis and swallowed `## Review Strategy Recommendations`, whose reviewer-complementarity notes are panel-wide Iron Rule #2 material; a card now ends at the next card or the next section heading. A field analysis missing any of the four required cards (the analyst's own quality gate) dispatched a generic seat while staying score-eligible; it now aborts at field analysis, and only the DA -- seat five, cardless by design -- gets the no-card notice. The field-analysis and synthesis calls gained the Iron Rule #7 data-boundary sentence their agent files do not carry (under `--bare --system-prompt` nothing else would say it). A `[DELIVERABLE-MISSING]` diagnostic is now written to the artifact the record names as authoritative rather than pointing at a model response or an already-PASS gate log. And per §6's independent cycles, one seat's conformance failure no longer stops the remaining seats: every seat runs, every attempt is preserved, and the panel then aborts with the first failure -- a transport fault stays panel-fatal. `check_phase_conformance.py` gains `--phase1-only` (mutually exclusive with `--phase2`, mirroring the existing `--synthesis` / `--layer1-only` split) because the retry decision is taken while Phase 2 has not been requested yet, and a reimplemented gate would not be the checker's own output. A seventh round — codex r6 plus a closing security pass that drove the whole contract on the scripted transport and swept every assembled prompt — found six more. The deliverable gate was a literal `## `-substring test and failed in both directions: the committed synthesizer output varies its heading level (`# Part 1: Editorial Decision Letter` in two acceptance-cohort panels) and sometimes drops the "Part N: " wrapper, so 3 of the 18 committed real panels would have aborted AFTER the full panel had burned with no replacement draw permitted — the #609 false-abort channel reopened at the synthesis step — while a synthesis that merely mentioned the heading in prose, or emitted it empty, or inside a fence, passed. Sections are now resolved as real nonblank heading sections via the repo's shared `heading_section` (H1-H3, "Part N: " wrapper optional); the one 2026-07-24 panel titling its letter `# Editorial Decision` stays outside the gate as a documented accepted miss, because accepting that bare name would also accept a seat's own `## Editorial Decision` section pasted into a synthesis. The Phase 1 retry hint was delivered as user content where §4 says "hinted in the system prompt" — role placement is part of the registered condition — and now rides the system half. A precondition refusal into a work directory holding an interrupted attempt's bundle reopened that bundle, appended the new abort to its journal, and renamed the whole directory — stale responses included — under the new blocked stem with the record attesting `provenance_status: valid` over evidence it never produced; an existing bundle is now refused untouched, with no record written, because a duplicate refusal record costs nothing and the earlier evidence has no other copy. A mid-edit contract template escaped `main` as a `JSONDecodeError` traceback with no record (its exit 1 reads as `EXIT_BLOCKED` to a fleet driver); it is now a precondition failure with a blocked record. Agent files were read lazily per call, so a checkout change mid-panel could deliver different bytes to later seats while the record still named the pre-dispatch `suite_commit` as reproducible; the seven files are snapshotted at builder construction. And `card_for` scanned the whole analysis for the first `Card #N` mention, so a card number named in an earlier inconsistency note would hand that seat the surrounding prose as its configuration with the missing-cards gate none the wiser; only the cards section is searched now, found by the same H1-H3 variants the deliverable gate accepts, so a variant analysis cannot pass the gate and then dispatch a cardless panel. A seat that decorates the multi-dissent token with prose or a fence still loses the §5 recovery — pinned as a documented accepted miss, since the strictly-one-line shape is what keeps the token from being smuggled out of a real card. The set README gains the harness bundle-directory layout in §6 and a comparability rule: harness cohorts change the dispatched condition (`--bare`, per-seat cards, split system/user halves, separate sandboxes, stamped contract), so they are never compared against the hand-dispatched rows and both conditions re-run under the harness. An eighth round — codex r7 in parallel with a second closing security pass — then caught the seventh round's own deliverable fix repeating the class it fixed, plus five more. The heading-variant rule judged emptiness by `heading_section`'s same-or-higher-level body, and the committed synthesizer output organises a letter's content as sibling H2 sections as often as child H3 ones, so two real panels (one in the operative 2026-07-25 acceptance cohort) read as deliverable-less — the net false-abort count was unchanged at 3/18 with two panels swapped, and no test guarded them. The gate now measures the deliverable's INTERVAL — from its heading to the next required-deliverable heading — via a position-aware `_heading_lines` whose fence rules are copied from `heading_section` and pinned to it by test; measured against all 18 committed panels the one abort left is the documented accepted miss, and the interval rule also closes two shapes the seventh-round spelling passed (an empty letter borrowing a contentful roadmap's body, in either heading-level direction). Builder construction had ALSO moved the agent-file read outside `dispatch_panel`'s handler, so a symlinked or missing agent file escaped as a traceback after the bundle was on disk — no record, with exit 1 reading as `EXIT_BLOCKED`; construction is back inside the handler and an unreadable prompt file is a `PreconditionFailure`. The synthesis boundary sentence now covers every delimited block, because a manuscript directive can be echoed into a reviewer card or the field analysis and the synthesizer's agent file carries no untrusted-material rule; the Phase 1 metadata envelope — whose title is quoted from the manuscript's H1 — gains the same data-only sentence. A `claude` exit 0 with empty stdout is a transport event rather than a conformance failure that consumes the one permitted Phase 1 retry. And git provenance failure (outside a worktree both commands fail while `status` prints nothing) no longer reads as a clean tree at commit "unknown": unknown provenance is declared dirty, so `suite_commit_reproducible` cannot be true for a commit that does not exist. Two shapes are documented as accepted misses rather than gated: a field analysis whose four cards are bare heading shells (validating card substance across the committed card shapes — inline single-line cards and heading cards — risks the false-abort class this round just removed twice, the analyst's own quality gate owns card content, and all 19 committed analyses carry substantive cards), and a deliverable heading hidden inside an HTML comment (`_heading_lines` mirrors `heading_section` in tracking fences but not comments; the comment-visibility channel is #613's scope). A ninth round — codex r8 in parallel with a third closing security pass (which confirmed the interval rule at 1/18 aborts, both prior regressions closed, and found nothing blocking) — added five more. The CLI's own built-in tools are now denied per call (`--disallowedTools`): `--bare` cuts customization and `--strict-mcp-config` cuts MCP, but neither disables Read/Bash/WebSearch, and the checkout is public, so a paper-blind seat could otherwise retrieve a manuscript's held-out siblings mid-call with no tool-use audit trail in a text response — the seats' task is pure text and needs no tool; a NEW tool name shipping in a later CLI is a declared residual. §6's independent cycles are confined to reviewer-conformance failures: a checker infra exit (2) or crash (1) inside the seat loop now aborts the round immediately instead of re-running the same global fault once per seat. A mid-panel checkout change downgrades `suite_commit_reproducible` (the checkers and their imports load from REPO afresh at each gate, so the pre-dispatch snapshot alone could attest a reproducible commit for a run that mixed checker versions). A regular file passed as `--work-dir` is a stated precondition refusal instead of a `NotADirectoryError` traceback with no record. And `--date` must be a real calendar day in canonical ISO form — the shape regex alone passed `2026-02-31`, which would have consumed a full panel and been committed as invalid provenance. One more shape is documented as an accepted miss: heading lines count as interval content (excluding them would reject the committed pattern that writes the decision in a heading line, a real-corpus shape, to close one with zero corpus occurrences). A tenth round — codex r9 in parallel with a fourth closing security pass — caught the ninth round's own tool fix being the wrong polarity, plus five more. The security pass measured the `--disallowedTools` deny list incomplete against the installed CLI the day it was written (13 built-in names beyond its 14) and found its completeness sentence to be a false attestation; the CLI's own whitelist spelling `--tools ""` now does the shutoff — under an emptied whitelist a tool added by a later CLI is closed by default, the property a deny list can never have, and the argument that replaced this harness's word denylist with a path allowlist — with the deny list riding behind as depth only. The auth path stopped smuggling context past `--bare`: `--settings` loads the whole file it names, and the user's own settings may carry `env`, hooks and plugin configuration, so the `apiKeyHelper` now travels alone in a staged one-key file, and nothing is passed at all when `ANTHROPIC_API_KEY` is set. A synthesis whose checker passes but whose Editorial Decision Letter or Revision Roadmap is absent is voided and re-run once per §8.1 instead of aborting a completed panel on an ordinary stochastic omission after all twelve calls had burned. An external `--set-root` no longer reads as reproducible provenance (its manuscript bytes are not covered by the repo commit). A sandbox-creation or initial-write failure — read-only parent, full storage, two processes racing on one work directory — is a stated setup result instead of a traceback whose exit 1 reads as `EXIT_BLOCKED` with no record. And `stem_for` applies the same calendar-day validation as the CLI, so the library-level guard is not weaker than the entry point's. An eleventh round — codex r10 in parallel with the fifth closing security pass — added seven more, three behavioral and four hygiene. The hard-coded Phase 1 metadata field mislabeled the MS02 quality-assurance manuscript as educational technology for every paper-blind call, biasing that fixture's measurement before any seat saw its card; the field now follows the fixture (the ms00 value matches the committed 2026-07-27 bundle's metadata). `card_for`'s catch-all line regex minted a marker from prose like "covered by Card #3", truncating the current card and letting the named card resolve to the mention line itself; only heading lines open a card now, measured against all 26 committed field analyses (every card in the corpus opens with a heading, every cross-mention sits in prose). A transport timeout's summary no longer serializes `str(TimeoutExpired)` — the whole argv, system prompt and staged paths included — into a log meant for public commit. Hygiene from the security pass: the `apiKeyHelper` staging happens once per transport instead of once per call (a six-panel fleet would have strewn ~96 copies of the helper command across the temp tree); an EMPTY leftover `bundle/` no longer blocks the refusal record (it holds no evidence to protect, and the message sent the operator to move aside evidence that did not exist); `_bare_auth_available` returns an actual bool; and the README names the pre-fleet single-panel smoke test for the never-live-called `--tools ""` and `--bare`/xhigh/thinking interactions. A twelfth round — codex r11 in parallel with the sixth closing security pass (which measured the heading-anchored card slicing against the full corpus: 110 slices, zero deltas, confirming the old regex had never actually truncated a committed card) — added two corrections and pinned one rebuttal. A shrunk panel now aborts with §6's own `[PANEL-SHRUNK: usable=, panel_size=]` cardinality marker ahead of the first seat's diagnostic, so the operational monitor can count these rounds. The one console refusal assembled from a raw exception is scrubbed (an OSError spells out absolute paths; console-only, never a committed record). A claimed missing §8.1 terminal marker was REFUTED by measurement and pinned: the synthesis checker's exit-1 output opens with its own `[PANEL-SYNTHESIS-MISMATCH...]` line and the abort diagnostic is exactly that line. The corpus-size figure in the card-slicing rationale is corrected from 30 to 26 (three independent counts agree; the conclusion it supported was re-verified on all 26). A thirteenth round — codex r12 in parallel with the seventh closing security pass — added six more. The security pass caught a stray `.coverage` build artifact (a SQLite file carrying the operator's absolute paths) that a bulk `git add` had swept into the amended commit; it is removed and ignored (`.coverage`, `.coverage.*`, `htmlcov/`). It also proved the path scrubber's resolved-only prefixes never matched an OSError's caller-spelled paths — on darwin `/tmp` resolves to `/private/tmp`, so a setup failure under the README's own `--work-dir /tmp/...` example put an absolute path into a committed record, refuting the README's no-absolute-path sentence; `RUN_ROOTS` now registers both spellings (absolute only, since a short relative spelling would corrupt ordinary prose as a substring). Sandbox inputs are exclusive-create, so two panels accidentally given the same empty work directory can no longer silently cross-contaminate — the loser's writes previously overwrote the winner's contract, metadata and manuscript before the bundle's O_EXCL collision, letting gates check Phase 1 against the other fixture's paper. A mid-panel OSError (a gate-log write, a checker launch) leaves a blocked `[IO-FAULT: ...]` record instead of escaping as a traceback. An abort diagnostic now exists byte-for-byte in the artifact its record names authoritative: the panel-shrunk composite goes to its own `panel-shrunk.log`, and the transport response-status suffix is composed before the log is written. And `--replicate` is validated up front (1-99, in `stem_for` too): a nonpositive value minted normal-looking run ids, and a long one raised ENAMETOOLONG only in `emit` after the full panel had burned. A fourteenth round — codex r13 in parallel with the eighth closing security pass (which verified the diagnostic-artifact equivalence across all twelve abort shapes and the `/tmp` scrub end-to-end) — added four more. A `[PHASE1-MANUSCRIPT-LEAK]` verdict is never granted the Phase 1 retry: the leak shares exit 3 with the structural lints, so the harness retried it and a clean second attempt made the contaminated panel score-eligible, although the checker's own comment names blindness "the half a retry must not be granted in spite of". The Phase 2 card wrapper no longer forbids what the card exists to do: it said the card "may not alter your identity" while the card IS where full mode's reviewer identity comes from, so an obedient seat would refuse its own configuration and the harness would measure generic reviewers — identity adoption is now authorized explicitly while Phase 1 commitments, scoring procedure and output format stay fenced. Allowlist keys bind to the declared LEXICAL paths, never resolved ones: resolving a manuscript name that is itself a symlink to a held-out manifest inserted the manifest's real path as an allowed key, and a read via the target's own spelling then passed both membership and the symlink walk. And `main`'s precondition stage catches OSError too — `validate_contract` stages in a temp directory, and a full TMPDIR escaped as a traceback with no record while the work directory could be on a different, writable filesystem (`_git_state` is guarded the same way, and the temp root joins the scrub prefixes). A fifteenth round — codex r14 in parallel with the ninth closing security pass — hardened the last second-order failure paths. The post-panel provenance re-probe is guarded (a git spawn failure AFTER a completed, expensive panel raised before `_emit_or_explain` and cost the record; the initial probe already tolerated the same OSError). Every abort handler's own journal and artifact writes are best-effort — an exception raised inside an `except` block is not routed to a later sibling, so a journal device failing mid-abort escaped WITH the blocked result it was recording; a missing artifact is caught by `locations_resolve_from`, which downgrades the attestation instead of losing the record. A setup failure after the bundle was created returns that bundle rather than None, so the refusal path no longer mistakes this invocation's own half-written bundle for an earlier attempt's evidence. `emit` refuses a repeated identity across BOTH record namespaces — a successful run repeated produced a blocked record beside the normal one under the identical identity. And the two OSError-derived diagnostics declare their rewrite honestly: they were scrubbed before composition, so `scrub` found nothing left to change and stamped `verbatim` on strings whose paths were in fact removed; `PanelAborted` now carries an explicit form set at composition. A sixteenth round — codex r15 in parallel with the tenth closing security pass — closed the race and custody residue. The work directory is claimed atomically (`.claimed`, O_EXCL) before anything else: two processes racing past the emptiness check both built state in one directory, and the loser could consume the run identity with a blocked record, leaving the winner's finished panel unable to emit — the loser of the claim now dies having written nothing. A setup failure journals its diagnostic so the record's authoritative artifact actually holds the line, and a bundle that merely OPENED a directory already holding content is never handed back by the setup handler (the stale branch keeps custody — the same F3 class, reopened by the previous round's own fix and caught by the security pass). The staged `apiKeyHelper` copy is removed at process exit. One provenance bound is declared rather than detected, in the README: an edit made during a gate and reverted before the end probe is not caught — prompt material is snapshotted and immune, the checkers load per gate, so the checkout must not move while a panel runs. A seventeenth round — codex r16 in parallel with the eleventh closing security pass — polished the remaining edges. A claim-time OSError (unwritable parent, full disk) is a stated precondition refusal instead of an uncaught exit 1. Record locations are serialized with POSIX separators regardless of platform, so a record produced on Windows still resolves when committed and read on POSIX. A settings file whose valid JSON has a non-object top level carries no helper instead of crashing the auth preflight. The README's `normalized` definition now describes the scrubber's actual (and intended) semantics — registered run roots are removed wherever they appear in a harness-assembled diagnostic, since an OSError spells its path mid-sentence, not only leading. And the claim-refusal wording credits an earlier or concurrent invocation rather than always another one. An eighteenth round — codex r17 in parallel with the twelfth closing security pass (which delivered the final residual inventory) — closed the last escape hatches. The blindness check runs BEFORE structural parsing under `--phase1-only`: a response both malformed and carrying a manuscript shingle reported only the grammar failure, so the dispatcher granted the retry a proven leak must never receive. `locations_resolve_from` covers `synthesis_retries` (the bundle-side check already covered every retry event; the record-side predicate now matches). Undecodable UTF-8 in prompt material or in the settings file is a stated refusal, not a `UnicodeDecodeError` traceback — it is a ValueError, caught by neither precondition handler on its own. And transport construction (which re-reads settings and stages the helper file, after `.claimed` exists) routes its failures through the record path. A nineteenth round — codex r18 in parallel with the thirteenth closing security pass (which proved the checker-order asymmetry unreachable: a leaking seat's Phase 2 is never dispatched) — rejected the structurally ambiguous shapes. A synthesis carrying TWO letters or two roadmaps was score-eligible on the first interval; a duplicate is now rejected like an absence, riding the same §8.1 rerun. A duplicated configuration-card number let `card_for` hand the seat the first copy while synthesis received the whole conflicting analysis; the panel now aborts at field analysis naming the duplicated card. And the transport-construction handler declares its diagnostic rewrite (`normalized`) like its two sibling handlers instead of stamping `verbatim` on a pre-scrubbed string. A twentieth round — codex r19 in parallel with the fourteenth closing security pass — held the measured condition steady on three more edges and pinned one rebuttal. An unexpected Card #5 (six superseded-namespace analyses emit one spontaneously; none of the 18 scored panels do) never reaches the DA, whose seat is cardless by design. The work directory is made absolute at entry: a relative `--work-dir` left the sandbox relative, and the subprocess cd's INTO the sandbox before the CLI re-resolves the same relative `--add-dir` from its new cwd. The helper value is read in one defensive snapshot, since a settings change between the availability probe and the second read raised KeyError past the transport-construction handler. And the claimed late-provenance routing gap was REFUTED by measurement and pinned: the reachable shape (an artifact vanishing before emission) is routed to the blocked namespaces by the bundle-side re-check that runs before the destination is chosen — the record-relative predicate downstream is a defensive invariant over emit's own arithmetic. A twenty-first round — codex r20 in parallel with the fifteenth closing security pass — tightened the last preflight edge and pinned two boundaries. A truthy non-string or whitespace-only `apiKeyHelper` no longer passes the preflight and stages an unusable credential (which would have converted an operator precondition into a dispatched blocked run — the exact cost the preflight exists to prevent). A claimed final-artifact completeness gap was resolved AGAINST widening `provenance_status`: the README scopes that closed two-value field to retry-evidence completeness alone and states verbatim that `valid` does not attest panel completeness — an accepted artifact deleted by an outside force after dispatch is the maintainer's commit-time concern, with the raw bundle beside the record; the scoping is pinned by a test quoting the contract. And the Card #5 guard's rationale comment counts the corpus correctly (six superseded-namespace analyses, none scored). A twenty-second round — codex r21 in parallel with the sixteenth closing security pass — sealed the delimiter boundary. Every untrusted block travels through one `_delimited` helper that refuses text carrying its own closing delimiter — a manuscript containing `` (or a card containing ``, and so on for every block) would close the declared data early and let the remainder read as instructions; refusal keeps the dispatched bytes untouched, where an escape scheme would change what the model receives. The retry hint's checker transcript — which rides the system half per §4 and can echo model-controlled text — is now explicitly fenced as checker-output DATA and delimiter-checked like every other block. A symlink-loop work or set root is a stated precondition refusal instead of a `RuntimeError` traceback. And a whitespace-only `ANTHROPIC_API_KEY` fails the preflight instead of suppressing a valid helper and failing the first live call. A twenty-third round — codex r22 (down to a single finding) in parallel with the seventeenth closing security pass (which byte-audited the delimiter refactor: exactly one intended line changed across all 32 prompt halves, zero corpus hits for any closer token) — worked the token-only audit classification to its honest resting place — in two steps, the second correcting the first. The first spelling replaced the retry event's diagnostic with the response's token line; the next codex round refuted it against the contract (`verbatim` is byte-for-byte CHECKER output, and the named gate log did not hold the replacement). The retry event therefore keeps the checker's own line — its eligibility was already machine-readable as `stage: phase2_multi_dissent`, with the bare token verbatim in the named rejected response — and §11's exhausted marker rides the TERMINAL record only, in a harness-written artifact that is byte-equal to the recorded diagnostic, with the checker's gate log untouched beside it. A twenty-fifth round — codex r24 in parallel with the nineteenth closing security pass — hardened the delimiter refusal to whitespace-variant closers (`` is as valid an end tag as the exact spelling) and made an operator interrupt durable: Ctrl-C mid-call escaped everything, stranding a `.claimed` directory with a partial bundle and no record, which a rerun then refused — the interrupt now leaves a blocked record with its own byte-equal artifact, every completed attempt already preserved by write-once. A twenty-sixth round — codex r25, again a single finding — restored the restarted Phase 1's own §4 retry: the §5 multi-dissent recovery restarts at Phase 1, and every Phase 1 pass carries one structural retry, but the restart was dispatched with a single-attempt budget, so an ordinary formatting slip on the replacement blocked an otherwise recoverable panel. A twenty-seventh round — codex r26 in parallel with the twenty-first closing security pass — finished the fence equivalences. Closer detection is case-insensitive and covers the self-closing spelling (HTML reads tag names case-insensitively and a model may too). Card discovery and duplicate counting are fence-aware like the deliverable gate, so a fenced template inside the cards section cannot configure a seat from example text. The bare-root scrub pass requires a path boundary (`/tmp` registered no longer eats the `/tmp` inside a sibling `/tmp2/file`). And a helper that dies between preflight and staging is a loud precondition instead of an empty flag list that would launch `--bare` uncredentialed and burn the first live call. A twenty-eighth round — the twenty-second closing security pass, with codex r27 returning ZERO findings for the first time — caught the previous round's helper guard sitting behind the tolerant probe's short-circuit, unreachable for the very scenario it named; `auth_flags` now classifies the settings file itself in one read, and reaching it without any usable credential — never configured, deleted, rewritten or broken since the preflight — refuses loudly instead of launching `--bare` uncredentialed. A twenty-ninth round — codex r28 in parallel with the twenty-third closing security pass — carried the §4 grammar into the checker itself and stopped mutating refused directories. `parse_phase1` validated only the Scoring Plan, so a response missing its Contract Paraphrase and terminal `[CONTRACT-ACKNOWLEDGED]` passed `--phase1-only` and the dispatcher could retry — or accept — a protocol-invalid Phase 1; the checker now enforces all three §4 requirements, on both its modes. And the work-directory claim looks before it plants: an accidental `--work-dir /tmp` gets its refusal record without a stray `.claimed` first, while the post-claim emptiness check stays for the race. A thirtieth round — codex r29 in parallel with the twenty-fourth closing security pass (which replayed the four real dispatch Phase 1 outputs against the stricter grammar: all PASS, zero false aborts) — gave the paraphrase check its floor: a bare heading over one line passed although the contract's `paraphrase_minimum_dimensions` is "all", so the paragraph count is now the machine-checkable lower bound (the real outputs carry exactly one paragraph per dimension, and the test fixture was raised to that valid shape); whether each paragraph is TIED to its dimension stays with the seat's own §4 preflight. A second interrupt window (after dispatch, before the record) was resolved AGAINST further handling, with the security pass's standing measurement: the window is sub-second, the bundle already sits preserved on disk, and re-entering a half-finished emission would add real complexity for a recovery the write-once layout already provides by hand. A thirty-first round — codex r30 in parallel with the twenty-fifth closing security pass — made the transport abort path tell the whole truth. A partial-response write that itself fails is no longer claimed "preserved" (the summary now says it could NOT be), and the transport summary and stderr are scrubbed at the source so the committed log matches the record byte-for-byte — the partial response itself is evidence and is never rewritten. A thirty-second round — codex r31, a single finding — routed SIGTERM into the same durable abort path as Ctrl-C: a fleet runner cancels with SIGTERM, which does not raise KeyboardInterrupt on its own, so the process exited with no blocked record and a stranded `.claimed` marker. A thirty-third round — codex r32 — installed the record atomically (staged `.tmp` plus `os.replace`; an ENOSPC mid-write left a truncated JSON at the final path that later runs refused, unrecoverable normally) and rerouted a late record-side predicate failure to the blocked namespaces — the third reviewer angle on this seam; the second rename cannot collide because the identity check already covered both namespaces. A thirty-fourth round — codex r33 — completed the §4 grammar (the H2 sequence must be exactly `## Contract Paraphrase` then `## Scoring Plan`, in order and nothing else at H2; presence alone let a reordered or extra-sectioned precommitment pass, and all four real dispatch outputs carry exactly the required sequence) and wrapped the whole claim-to-record span against interrupts: an interrupt outside the dispatch loop — preflight, contract staging, transport setup, emission — now exits with a stated refusal instead of a traceback, everything preserved in place. A thirty-fifth round — codex r33's successor — aligned the Phase 1 envelope with the seat files' own promise (all five state the contract arrives "under `## Contract`" and the metadata "under `## Paper Metadata`"; the plain labels were a different envelope from the registered instructions) and made the terminal acknowledgement check fence-aware, so an acknowledgement that exists only inside an unclosed code fence no longer passes. A thirty-sixth round — codex r34's successor — carried the §11 cardinality marker onto the synthesis-stage seat failure (§8.1 classifies that exit 3 as an unusable reviewer; the count is honestly `unknown`, the checker's own finding rides in the diagnostic) and scrubbed checker crash tracebacks at the `run_checker` outlet — a crashing checker's stderr spells absolute script and module paths straight into committed gate logs, while normal checker output is relative-path-only so the scrub is a no-op there and `verbatim` stays honest. A thirty-seventh round — codex r35's successor — closed the two second-order effects of the previous fixes. The acknowledgement check now requires BOTH tails: the raw tail catches output trailing after the marker (a fenced block vanished from the structural view before the tail was computed), the fence-aware tail catches a marker that exists only as fenced code. And the checker-output normalization state travels with the output — `run_checker` returns a third element, carried through the attempt chain into `PanelAborted` and `RetryEvent`, so a scrubbed crash diagnostic is stamped `normalized`, never `verbatim`. A thirty-eighth round — codex r36's successor — closed the last race seam and the last acknowledgement spelling. `.claimed` existing refuses OUTRIGHT, before the occupied pre-check: an occupied-looking directory may be the owner's live run (sandboxes built, bundle still empty), and skipping the claim check there let a loser claim the owner's empty bundle through the stale branch and consume the identity. And the acknowledgement is compared unstripped bar the line ending — an indented marker renders as a code block, and stripping before comparison let it pass the exact terminal-line requirement. A thirty-ninth round — codex r37's successor — widened the closer refusal to attributed end tags: `` is invalid HTML, but tolerant parsers accept it, and the boundary must not depend on the model being a strict parser — while the word boundary keeps a longer tag name (``) out. A fortieth round — codex r38's successor — bounded the last card at any non-card heading of its own level or higher (a literal `## ` check would let an H3-headed strategy section ride inside an H3-headed Card #4) and gave the exhausted synthesis retry its §11 `[SYNTHESIS-MISMATCH]` terminal marker, riding its own byte-equal artifact with the gate log untouched beside it. A forty-first round — codex r40's successor — closed the last two `verbatim` false attestations in the abort handlers: the transport arm derived the form from the summary alone (a scrubbed stderr left the named log rewritten under a `verbatim` stamp), and the two preflight failure paths dropped the fact of normalization entirely — `validate_contract` discarded the checker's own form and the OSError arm pre-stripped the path so the record's scrub fallback saw clean text; `PreconditionFailure` now carries the form to the record. A forty-second round — codex r41's successor — closed three install-side gaps: the §4 paragraph floor counted six bare `### Dn` headings as six paragraphs (a heading now separates paragraphs and never counts as one; all four committed Phase 1 outputs still PASS), the staged record file became the identity lock — one path per stem across BOTH namespaces, re-checked inside the lock — so a concurrent emission of the same identity can no longer leave a scored record beside a blocked one, and a staged-write failure after the raw bundle moved into `runs/raw/` now rolls the bundle back so the identity is not consumed by a failure that wrote no record. A forty-third round — codex r42's successor — extended the §4 separator set to the other zero-content lines (thematic breaks, single-line HTML comments); the list is closed by design, not a CommonMark block classifier — a bulleted paraphrase still counts as content, because refusing real prose over formatting is the false-abort channel #609 exists to remove. A forty-fourth round — codex r43's successor — kept the abort classifiers honest on three edges: multi-line HTML comment blocks join the zero-content set via a conservative line-leading-opener state (prose mentioning `` satisfying the paragraph floor — is declined under the declared closed-separator-list bound: recognizing comment openers behind arbitrary CommonMark container prefixes is the renderer-rewrite slope both this round and #609 exist to refuse, and §4's TIED judgment stays with the seat's preflight. A forty-seventh round — codex r46's successor — admitted lone list markers (`-`/`*`/`+` with no item text) into the closed separator list (zero parse slope, zero false-abort surface — a marker WITH text still counts), and honored the record contract's MUST at emission: a terminal abort whose named artifact fails to resolve is rewritten once from its own diagnostic (the bytes ARE the diagnostic, so equality holds by construction) before the downgrade fires, and the README now states plainly that the downgrade is the terminal fallback for any unresolvable named location, not an attestation that a retry occurred. A forty-eighth round — codex r47's successor — closed the zero-content axis with an explicit termination bound written into the counter itself: variants beyond the closed list (container-prefixed comments, malformed comments like ``, entity/whitespace tricks) are declared out of scope — the variant space is unbounded, the observed base rate in committed panels is zero, and §4's substantive judgment sits with the seat's preflight, not this floor. 259 new cases (239 harness + 20 conformance) run the whole contract on a scripted transport with no model call, reusing the two checkers' own valid fixtures so a card that stops satisfying a real gate stops satisfying the harness tests too. Measurement-side only: no reviewer prompt, contract, template, or fixture changed. diff --git a/scripts/check_panel_synthesis.py b/scripts/check_panel_synthesis.py index 7ac645d5..4a3df89f 100644 --- a/scripts/check_panel_synthesis.py +++ b/scripts/check_panel_synthesis.py @@ -164,6 +164,82 @@ def strip_fences( return out +_INLINE_CODE_LINE_RE = re.compile( + r"^ {0,3}(?P`+)(?P[^`]+?)(?P=ticks)[ \t]*$" +) + + +def audit_candidate_lines(text: str) -> list[str]: + """Candidate lines for the synthesis mechanical audit-line grammars. + + The four audit lines (``dimension_verdicts`` / ``fired_conditions`` / + ``da_critical_adjudications`` / bare ``editorial_decision=``) are + routinely emitted inside a CommonMark code fence, or wrapped whole in an + inline code span — both render the line verbatim and visibly to a + reader, so rejecting them is a false abort, not a leak guard (#637: + every synthesis first attempt of the 2026-08-03 baseline cohort, 6/6, + fenced the audit block; one retry re-emitted it as inline code spans). + This walker therefore keeps fenced content (dropping only the + fence-marker lines themselves, with the same open/close state machine as + ``strip_fences``) and additionally yields the unwrapped body of any + non-fenced line that is entirely one inline code span. Candidates are + normalized to what a CommonMark renderer would DISPLAY, since rendered + visibility is the acceptance criterion: an inline span whose body has + both a leading and a trailing space renders with one space stripped + from each side, and lines inside an indented fence render dedented by + up to the opener's indent — both normalizations are applied here so + the start-anchored grammars see the displayed text. + + Declared boundaries (ACCEPTED misses — change deliberately, with + tests): blockquoted or emphasis-wrapped audit lines never match the + line-anchored grammars and stay non-candidates; an inline code span + inside a fence renders its backticks literally and is not unwrapped; + the rejection-rationale and DA-consistency-marker grammars keep the + plain-line source in ``parse_synthesis`` because both are quotable + diagnostics whose fenced or wrapped occurrences read as quotation. + One asymmetry of that boundary is accepted deliberately: a fenced + marker in a state where the marker is forbidden stays invisible to + ``check_da_terminal_gate`` and no longer aborts upstream (pre-#637 the + whole fenced block aborted at ``fired_conditions``); the editorial + decision itself is independently recomputed from the reviewer cards by + the layer-2 check, so the slip is an accounting cosmetic, not a + decision channel. + """ + out: list[str] = [] + fence_char: str | None = None + fence_len = 0 + fence_indent = 0 + for line in _COMMONMARK_LINE_END_RE.split(text): + if fence_char is not None: + match = _FENCE_CLOSE_RE.fullmatch(line) + if (match and match.group("fence")[0] == fence_char + and len(match.group("fence")) >= fence_len): + fence_char, fence_len, fence_indent = None, 0, 0 + else: + dedent = 0 + while (dedent < fence_indent and dedent < len(line) + and line[dedent] == " "): + dedent += 1 + out.append(line[dedent:]) + continue + match = _FENCE_OPEN_RE.fullmatch(line) + if match: + token = match.group("fence") + info = match.group("info") + if token[0] != "`" or "`" not in info: + fence_char, fence_len = token[0], len(token) + fence_indent = match.start("fence") + continue + out.append(line) + if span := _INLINE_CODE_LINE_RE.fullmatch(line): + body = span.group("body") + if (len(body) >= 2 and body[0] == " " and body[-1] == " " + and body.strip(" ")): + body = body[1:-1] + out.append(body) + return out + + def _split_by(lines: list[str], heading_re: re.Pattern[str]): sections: dict[str, list[str]] = {} dupes: set[str] = set() @@ -960,14 +1036,25 @@ _MARKER_RE = re.compile( def _one_body( lines: list[str], pattern: re.Pattern[str], label: str, path: str ) -> str: - bodies = [match.group("body") for line in lines + """Exactly one DISTINCT body for an audit-line grammar. + + Re-statements of an audit line that parse to the SAME stripped body (a + plain line plus its fenced copy, or a padded inline-span variant of it) + collapse to one value; the collapse is value-level, not byte-level, by + design — the candidate list already mixes raw lines with their + rendering-normalized forms. Two candidates that parse to different + bodies remain a loud abort, so a decoy that disagrees with the + operative line can never be silently absorbed (#637). + """ + bodies = [match.group("body").strip() for line in lines if (match := pattern.fullmatch(line))] - if len(bodies) != 1: + distinct = list(dict.fromkeys(bodies)) + if len(distinct) != 1: raise SynthesisError( f"[SYNTHESIS-PARSE: {path}: expected exactly one {label} line, " - f"found {len(bodies)}]" + f"found {len(distinct)}]" ) - return bodies[0].strip() + return distinct[0] def _comma_tokens(body: str) -> list[str]: @@ -975,9 +1062,13 @@ def _comma_tokens(body: str) -> list[str]: def parse_synthesis(path: str, text: str, contract: dict) -> Synthesis: + # The four mechanical audit lines read from the decoration-tolerant + # candidate source; rationales and the DA marker stay plain-line-only + # (see audit_candidate_lines for the declared boundary). lines = strip_fences(text) + audit_lines = audit_candidate_lines(text) fired = _comma_tokens(_one_body( - lines, _FIRED_LIST_RE, "fired_conditions", path + audit_lines, _FIRED_LIST_RE, "fired_conditions", path )) condition_ids = {c["condition_id"] for c in contract["failure_conditions"]} if len(fired) != len(set(fired)) or set(fired) - condition_ids: @@ -986,7 +1077,7 @@ def parse_synthesis(path: str, text: str, contract: dict) -> Synthesis: ) verdict_tokens = _comma_tokens(_one_body( - lines, _VERDICTS_RE, "dimension_verdicts", path + audit_lines, _VERDICTS_RE, "dimension_verdicts", path )) verdicts: dict[str, str] = {} for token in verdict_tokens: @@ -1001,7 +1092,7 @@ def parse_synthesis(path: str, text: str, contract: dict) -> Synthesis: verdicts[match.group("dim")] = match.group("value") adjudication_tokens = _comma_tokens(_one_body( - lines, _ADJUDICATIONS_RE, "da_critical_adjudications", path + audit_lines, _ADJUDICATIONS_RE, "da_critical_adjudications", path )) adjudications: dict[str, str] = {} for token in adjudication_tokens: @@ -1016,8 +1107,10 @@ def parse_synthesis(path: str, text: str, contract: dict) -> Synthesis: ) adjudications[match.group("id")] = match.group("value") - decisions = [match.group("action") for line in lines - if (match := _DECISION_RE.fullmatch(line))] + decisions = list(dict.fromkeys( + match.group("action") for line in audit_lines + if (match := _DECISION_RE.fullmatch(line)) + )) if len(decisions) != 1 or decisions[0] not in ACTION_ENUM: raise SynthesisError( f"[SYNTHESIS-PARSE: {path}: expected exactly one valid decision]" diff --git a/scripts/check_phase_conformance.py b/scripts/check_phase_conformance.py index 2eb8f952..f97a896e 100644 --- a/scripts/check_phase_conformance.py +++ b/scripts/check_phase_conformance.py @@ -853,11 +853,46 @@ def check_scoring_seat_anchors(report: panel.ReviewerReport) -> None: ) if not is_finding: continue - if len(severities) != 1 or severity_declarations != 1: + # A finding needs at least one parseable Severity declaration, and + # every declaration must parse. When a card declares more than one + # ACROSS lines and the chain strictly ESCALATES (Minor < Major < + # Critical), the LAST in reading order is operative — the current + # model generation self-corrects mid-card with explicit supersession + # prose ("See the Severity line below, which supersedes the line + # above", #637 ms01_quant r1: Major -> Critical), and Phase 2 permits + # no retry for this class, so a strict exactly-one rule turns a + # visible, reader-unambiguous correction into a whole-panel abort. + # Every other multi-declaration shape keeps the loud abort: + # de-escalation could waive the Critical/Major Evidence-Anchor + # requirement by appending one weaker line; a non-monotone or + # repeated-value chain signals several findings bundled under one W + # heading (the one-finding-per-heading accounting feeds the severity + # ladder); and two parseable declarations on ONE line are not a + # reading-order correction at all. The advisory line below keeps the + # full declaration trail in the gate log for adjudication. + parseable_per_line = [ + sum(1 for _ in _SEVERITY_RE.finditer(line)) for line in block + ] + if (not severities or len(severities) != severity_declarations + or any(count > 1 for count in parseable_per_line)): raise ConformanceError( f"[FINDING-GRAMMAR: {report.path}: {title} must contain " "exactly one parseable Severity declaration]" ) + severity_rank = {"Minor": 0, "Major": 1, "Critical": 2} + if any(severity_rank[later] <= severity_rank[earlier] + for earlier, later in zip(severities, severities[1:])): + raise ConformanceError( + f"[FINDING-GRAMMAR: {report.path}: {title}: multiple " + "Severity declarations must form a strictly escalating " + "self-correction chain]" + ) + if len(severities) > 1: + print( + f"[SEVERITY-SUPERSEDED: {report.path}: {title}: " + + " -> ".join(severities) + "]" + ) + operative_severity = severities[-1] anchor_declarations = sum( len(_ANCHOR_DECL_RE.findall(line)) for line in block ) @@ -865,7 +900,7 @@ def check_scoring_seat_anchors(report: panel.ReviewerReport) -> None: match.group("value") for line in block for match in _ANCHOR_RE.finditer(line) ] - if severities[0] not in {"Critical", "Major"}: + if operative_severity not in {"Critical", "Major"}: if anchor_declarations > 1 or len(anchors) != anchor_declarations: raise ConformanceError( f"[FINDING-GRAMMAR: {report.path}: {title} may contain " @@ -877,7 +912,8 @@ def check_scoring_seat_anchors(report: panel.ReviewerReport) -> None: if len(anchors) != 1 or anchor_declarations != 1: raise ConformanceError( f"[ANCHOR-MISSING: {report.path}: {title} " - f"{severities[0]} finding needs exactly one Evidence Anchor]" + f"{operative_severity} finding needs exactly one " + "Evidence Anchor]" ) _validate_anchor(anchors[0], f"{report.path}:{title}") diff --git a/scripts/test_check_panel_synthesis.py b/scripts/test_check_panel_synthesis.py index 735b4d8f..74a7dcbb 100644 --- a/scripts/test_check_panel_synthesis.py +++ b/scripts/test_check_panel_synthesis.py @@ -327,11 +327,15 @@ def test_malformed_fence_closer_keeps_reviewer_report_hidden(): cps.parse_report("eic.md", text, FULL) -def test_malformed_fence_closer_keeps_synthesis_hidden(): +def test_malformed_fence_closer_synthesis_parses_as_fenced_content(): + """The fence never closes (`~~~not-a-close` is content, not a closer), + so the whole synthesis stays fenced — and fenced audit lines are + candidates since #637. Fence-STATE correctness stays pinned by the + parse_report twins above, whose grammars remain fence-blind.""" synthesis, _ = synthesis_for(reports()) text = "~~~text\n~~~not-a-close\n" + synthesis + "\n~~~\n" - with pytest.raises(cps.SynthesisError, match="fired_conditions"): - cps.parse_synthesis("s.md", text, FULL) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.decision == "editorial_decision=accept" @pytest.mark.parametrize("separator", ("\x85", "\u2028", "\u2029")) @@ -347,11 +351,225 @@ def test_unicode_separator_cannot_close_commonmark_fence(separator): @pytest.mark.parametrize("separator", ("\x85", "\u2028", "\u2029")) -def test_unicode_separator_keeps_synthesis_fenced(separator): +def test_unicode_separator_synthesis_parses_as_fenced_content(separator): + """A Unicode separator is not a CommonMark line end: it can neither + close the fence nor split the line, so the glued `~~~note` stays + one content line. The rest of the synthesis is ordinary fenced content \u2014 + a candidate since #637 \u2014 and parses.""" synthesis, _ = synthesis_for(reports()) - text = "~~~text\n~~~" + separator + synthesis + "\n~~~\n" + text = "~~~text\n~~~" + separator + "note\n" + synthesis + "\n~~~\n" + parsed = cps.parse_synthesis("fenced-synthesis.md", text, FULL) + assert parsed.decision == "editorial_decision=accept" + + +def test_fenced_audit_block_parses_as_observed_in_637_cohort(): + """Every synthesis first attempt of the 2026-08-03 baseline cohort (6/6) + emitted the four audit lines inside a plain code fence.""" + synthesis, _ = synthesis_for(reports()) + parsed = cps.parse_synthesis( + "s.md", "```\n" + synthesis + "\n```\n", FULL + ) + assert parsed.fired == ["F0"] + assert parsed.decision == "editorial_decision=accept" + + +def test_inline_code_wrapped_audit_lines_parse(): + """ms01_quant baseline r2's a2 retry wrapped each audit line whole in an + inline code span, defeating the line-anchored grammar (#637).""" + synthesis, _ = synthesis_for(reports()) + text = "\n".join(f"`{line}`" for line in synthesis.split("\n")) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.decision == "editorial_decision=accept" + + +def test_identical_duplicate_audit_lines_collapse(): + """A byte-identical re-statement (plain line plus its fenced copy) is + one candidate, not a duplicate abort.""" + synthesis, _ = synthesis_for(reports()) + fired_line = next(line for line in synthesis.split("\n") + if line.startswith("fired_conditions:")) + text = synthesis + "\n```\n" + fired_line + "\n```\n" + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.fired == ["F0"] + + +def test_conflicting_fenced_audit_line_still_aborts(): + """A fenced decoy that disagrees with the operative line stays a loud + abort \u2014 tolerance collapses identical values only.""" + synthesis, _ = synthesis_for(reports()) + text = synthesis + "\n```\nfired_conditions: [F1]\n```\n" + with pytest.raises(cps.SynthesisError, match="found 2"): + cps.parse_synthesis("s.md", text, FULL) + + +def test_prose_quoted_checker_diagnostic_is_not_a_candidate(): + """The a2 remediation notes of the #637 cohort quote the checker error + inside longer prose lines; anchored grammars must not see those.""" + synthesis, _ = synthesis_for(reports()) + note = ( + "**Checker remediation:** the prior emission failed " + "`[SYNTHESIS-PARSE: expected exactly one fired_conditions line, " + "found 0]` and the canonical block is re-emitted plainly above." + ) + parsed = cps.parse_synthesis( + "s.md", synthesis + "\n" + note + "\n", FULL + ) + assert parsed.fired == ["F0"] + + +def test_blockquoted_audit_line_documents_an_accepted_miss(): + """Documents an ACCEPTED miss: a blockquoted audit line renders as + quotation and is deliberately not a candidate. Change deliberately.""" + synthesis, _ = synthesis_for(reports()) + text = synthesis.replace("fired_conditions:", "> fired_conditions:", 1) with pytest.raises(cps.SynthesisError, match="fired_conditions"): - cps.parse_synthesis("hidden-synthesis.md", text, FULL) + cps.parse_synthesis("s.md", text, FULL) + + +def test_inline_span_inside_fence_documents_an_accepted_miss(): + """Documents an ACCEPTED miss: backticks inside a fence are literal + rendered content, so a wrapped line there is not unwrapped. Change + deliberately.""" + synthesis, _ = synthesis_for(reports()) + fenced = "\n".join(f"`{line}`" for line in synthesis.split("\n")) + with pytest.raises(cps.SynthesisError, match="fired_conditions"): + cps.parse_synthesis("s.md", "```\n" + fenced + "\n```\n", FULL) + + +def test_fenced_rationale_and_marker_document_an_accepted_miss(): + """Documents an ACCEPTED miss: rationales and the DA marker keep the + plain-line source \u2014 their fenced occurrences read as quotation. Change + deliberately.""" + synthesis, _ = synthesis_for(reports()) + text = ( + synthesis + + "\n```\nC9 rejection rationale: quoted example\n" + "[DA-CRITICAL-VS-ACCEPT: 2 validated/unresolved]\n```\n" + ) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert "C9" not in parsed.rejection_rationales + assert parsed.marker_count is None + # The forbidden-marker asymmetry is part of the same accepted boundary: + # with no active DA-critical adjudications a stated marker is forbidden, + # and this fenced one stays invisible rather than aborting — the + # decision channel is pinned independently by the layer-2 recompute. + assert not any( + "marker forbidden" in diag + for diag in cps.check_da_terminal_gate(reports(), parsed) + ) + + +def test_padded_inline_code_wrapped_audit_lines_parse(): + """CommonMark renders `` ` body ` `` with one padding space stripped + from each side, so a padded span displays the audit line verbatim and + must be a candidate in its DISPLAYED form.""" + synthesis, _ = synthesis_for(reports()) + text = "\n".join(f"` {line} `" for line in synthesis.split("\n")) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.decision == "editorial_decision=accept" + + +def test_padded_span_conflicting_value_still_aborts(): + """Rendering normalization must not weaken the distinct-value guard: a + padded-span decoy that disagrees with the operative line still aborts.""" + synthesis, _ = synthesis_for(reports()) + text = synthesis + "\n` fired_conditions: [F1] `\n" + with pytest.raises(cps.SynthesisError, match="found 2"): + cps.parse_synthesis("s.md", text, FULL) + + +def test_all_space_span_body_is_not_stripped(): + """CommonMark strips padding only when the body is not all spaces; an + all-space span stays as-is and is simply a non-candidate.""" + synthesis, _ = synthesis_for(reports()) + parsed = cps.parse_synthesis( + "s.md", synthesis + "\n` `\n", FULL + ) + assert parsed.decision == "editorial_decision=accept" + + +def test_indented_fence_content_parses_dedented(): + """A fence opened with up to 3 leading spaces renders its content + dedented by the opener's indent, so indented fenced audit lines are + candidates in their displayed (dedented) form.""" + synthesis, _ = synthesis_for(reports()) + indented = "\n".join(" " + line for line in synthesis.split("\n")) + text = " ```\n" + indented + "\n ```\n" + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.decision == "editorial_decision=accept" + + +def test_crlf_fenced_audit_block_parses(): + """CRLF line endings are CommonMark line ends; the fenced audit block + parses identically under them.""" + synthesis, _ = synthesis_for(reports()) + text = ("```\n" + synthesis + "\n```\n").replace("\n", "\r\n") + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.decision == "editorial_decision=accept" + + +def test_malformed_closer_fence_state_pins_wrapped_decoy_hidden(): + """Pins fence STATE through the candidate walker: `~~~not-a-close` does + not close the fence, so a wrapped decoy after it is fenced — backticks + render literally, the decoy is never unwrapped, and the plain synthesis + before the fence parses. A mutation that accepts the malformed closer + unwraps the decoy and flips this to a found-2 abort.""" + synthesis, _ = synthesis_for(reports()) + text = ( + synthesis + + "\n~~~text\n~~~not-a-close\n`fired_conditions: [F1]`\n~~~\n" + ) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.fired == ["F0"] + + +def test_short_closer_fence_state_pins_wrapped_decoy_hidden(): + """A prospective closer shorter than the opener does not close the + fence (CommonMark length rule), so the wrapped decoy after it stays + fenced and hidden.""" + synthesis, _ = synthesis_for(reports()) + text = ( + synthesis + + "\n````\n```\n`fired_conditions: [F1]`\n````\n" + ) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.fired == ["F0"] + + +def test_mismatched_closer_fence_state_pins_wrapped_decoy_hidden(): + """A backtick closer cannot close a tilde fence; the wrapped decoy + after it stays fenced and hidden.""" + synthesis, _ = synthesis_for(reports()) + text = ( + synthesis + + "\n~~~\n```\n`fired_conditions: [F1]`\n~~~\n" + ) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.fired == ["F0"] + + +@pytest.mark.parametrize("separator", ("\x85", "\u2028", "\u2029")) +def test_unicode_separator_fence_state_pins_wrapped_decoy_hidden(separator): + """A Unicode separator is not a CommonMark line end: `~~~note` + stays one content line and cannot close the fence, so the wrapped + decoy after it stays fenced and hidden. A mutation that splits on the + separator closes the fence and flips this to a found-2 abort.""" + synthesis, _ = synthesis_for(reports()) + text = ( + synthesis + + "\n~~~text\n~~~" + separator + "note\n" + "`fired_conditions: [F1]`\n~~~\n" + ) + parsed = cps.parse_synthesis("s.md", text, FULL) + assert parsed.fired == ["F0"] + + +def test_unterminated_fence_content_stays_candidate(): + """An unterminated fence retains (not grows) its content, and fenced + audit lines remain candidates to end-of-input.""" + synthesis, _ = synthesis_for(reports()) + parsed = cps.parse_synthesis("s.md", "```\n" + synthesis, FULL) + assert parsed.decision == "editorial_decision=accept" @pytest.mark.parametrize("score", ("warn", "block")) diff --git a/scripts/test_check_phase_conformance.py b/scripts/test_check_phase_conformance.py index 89b04a2a..278b6732 100644 --- a/scripts/test_check_phase_conformance.py +++ b/scripts/test_check_phase_conformance.py @@ -1592,6 +1592,10 @@ second""" def test_same_line_duplicate_severity_declarations_fail(): + """Unchanged by #637: a mid-line second declaration is a declaration + (`_SEVERITY_DECL_RE`) whose value cannot parse mid-prose, so the + declared-but-unparseable guard still aborts. Only cross-line + supersession takes last-wins.""" body = ( "### W1: hidden critical\n" "**Severity**: Minor and **Severity**: Critical" @@ -1601,6 +1605,126 @@ def test_same_line_duplicate_severity_declarations_fail(): phase.check_scoring_seat_anchors(report) +def test_same_line_pipe_separated_severity_pair_fails(): + """`_SEVERITY_RE` also parses after a table-cell pipe, so both halves + of `Minor | Critical` on ONE line are parseable — but two parseable + declarations on one line are not a reading-order self-correction and + must not enter the supersession path (which could otherwise waive the + anchor requirement via `Critical | Minor`).""" + for pair in ("Minor | **Severity**: Critical", + "Critical | **Severity**: Minor"): + body = ( + "### W1: table smuggle\n" + f"**Severity**: {pair}" + ) + report, _ = parse_report("eic", body=body) + with pytest.raises( + phase.ConformanceError, match="exactly one parseable Severity" + ): + phase.check_scoring_seat_anchors(report) + + +def test_cross_line_severity_supersession_takes_last(capsys): + """#637 ms01_quant baseline r1: the domain seat declared Major, then + self-corrected to Critical with explicit supersession prose. The card + passes with the last value operative and the trail in the gate log.""" + body = ( + "### W1: construct mismatch\n" + "**Severity**: Major\n" + "Correction: recording this as Critical; the Severity line below " + "supersedes the line above.\n" + "**Severity**: Critical\n" + '**Evidence Anchor**: text: "quote" p. 1' + ) + report, _ = parse_report("eic", body=body) + phase.check_scoring_seat_anchors(report) + assert ( + "[SEVERITY-SUPERSEDED: p2.md: W1: construct mismatch: " + "Major -> Critical]" + ) in capsys.readouterr().out + + +def test_unparseable_severity_declaration_still_fails(): + body = ( + "### W1: bad value\n" + "**Severity**: High" + ) + report, _ = parse_report("eic", body=body) + with pytest.raises( + phase.ConformanceError, match="exactly one parseable Severity" + ): + phase.check_scoring_seat_anchors(report) + + +def test_revisited_severity_value_still_fails(): + """A chain that revisits a value (Minor -> Major -> Minor) is not a + supersession — a non-escalating chain keeps the anti-bundling abort.""" + body = ( + "### W1: bundled pair\n" + "**Severity**: Minor\n" + "first\n" + "**Severity**: Major\n" + "second\n" + "**Severity**: Minor" + ) + report, _ = parse_report("eic", body=body) + with pytest.raises( + phase.ConformanceError, match="strictly escalating" + ): + phase.check_scoring_seat_anchors(report) + + +def test_deescalating_severity_pair_still_fails(): + """Critical -> Minor must abort: last-wins de-escalation would waive + the Critical Evidence-Anchor hard gate with one appended line. Only + the observed self-correction direction (escalation) is tolerated.""" + body = ( + "### W1: fabricated denominators\n" + "**Severity**: Critical\n" + "The paper invents denominators.\n" + "**Severity**: Minor" + ) + report, _ = parse_report("eic", body=body) + with pytest.raises( + phase.ConformanceError, match="strictly escalating" + ): + phase.check_scoring_seat_anchors(report) + + +def test_distinct_severity_bundle_still_fails(): + """Three findings bundled under one W heading with distinct descending + severities are not a supersession chain and keep the loud abort.""" + body = ( + "### W1: bundled triple\n" + "**Severity**: Critical\n" + "first\n" + "**Severity**: Major\n" + "second\n" + "**Severity**: Minor" + ) + report, _ = parse_report("eic", body=body) + with pytest.raises( + phase.ConformanceError, match="strictly escalating" + ): + phase.check_scoring_seat_anchors(report) + + +def test_escalating_supersession_operative_value_needs_anchor(): + """Minor -> Major with NO anchor must abort at ANCHOR-MISSING: the + LAST value (Major) is operative. Pins last-wins — if the first value + (Minor) were operative the card would take the no-anchor branch and + pass.""" + body = ( + "### W1: upgraded finding\n" + "**Severity**: Minor\n" + "on reflection this forecloses the design claim\n" + "**Severity**: Major" + ) + report, _ = parse_report("eic", body=body) + with pytest.raises(phase.ConformanceError, match="ANCHOR-MISSING"): + phase.check_scoring_seat_anchors(report) + + def test_noncanonical_heading_and_severity_label_fail_together(): body = ( "### Weakness 1: fabricated denominators\n"