mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
Edward Cheng-I Wu
b6062c1401
* feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#787) Probe set: 30 refs (10 easy DOI-keyed journal articles; 10 hard: 3 arXiv, 2 DOI-less NeurIPS, 5 non-English; 10 fabrications), every real row resolver-confirmed same-day, every fabrication negative-checked. 180 same-day paired calls (30 x 3 repeats x 2 models), majority verdicts. Result: all five measures PASS with superiority — recall 1.00 vs 0.80, grounded completion 0.933 vs 0.900, p95 latency 26.5s vs 58.7s, zero guard misfires, false disagreement 0.00 = 0.00. Transport-qualified: gpt-5.6-sol stays provisional on the first-party API route (jq guards unexercised; allowlist unchanged). Report + probe-set sha256 under audits/; per-call index committed beside the probe set. Campaign side-product (transport): page-open webSearch items (action.type != "search") are skipped for binding instead of failing the stream (opened-page URLs still can never become bound sources), and DEVELOPER_INSTRUCTIONS requires an empty sources array for NOT_FOUND/NOT_SEARCHED. 52 transport tests green. Defective-tool run 1 archived unscored; three probe-row transcription errors were flagged MISMATCH by both models, independently re-verified, corrected, re-run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: narrow the page-open exemption to the observed action.type == "other" shape (#788 codex P2) An empty action object, unknown action type, or non-dict action on a webSearch item is stream-fatal again; only the observed page-open shape is skipped. Mutation test sweeps four bad shapes (52 -> 53 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: anchor the page-open exemption to the first-party closed WebSearchAction set Run-3 surfaced a third real shape ({"type": "openPage", "url": ...}) that the single-observation exemption rejected, tool-suppressing the baseline's measures (13 EVENT_STREAM_INVALID cells). The exempt set is now exactly the non-search members of the app-server protocol's closed WebSearchAction oneOf — {other, openPage, findInPage} plus the Responses-API spellings — verified against `codex app-server generate-json-schema` on 0.147.0. Unknown shapes stay stream-fatal (mutation sweep unchanged); 54 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * docs: score preregistered run 4 as the gate result; runs 1-3 recorded as exploratory Run 4 (frozen fixture @3fc6ddb, frozen parser @c9c865d, both pushed pre-run): all five measures PASS, zero misfires on BOTH models, recall 1.00 vs 0.80, grounded completion 0.933 vs 0.867, p95 27.5s vs 51.1s. Report rewritten with the preregistration statement and the full exploratory-round accounting; call index replaced with run-4 data; claim surfaces and CHANGELOG updated to run-4 numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * docs(code): pin the bare-discriminator decision against the first-party schema (#788 codex round-2 P2 rejected with evidence) The round-2 finding claimed openPage/findInPage require url/pattern; the protocol schema (generate-json-schema, 0.147.0) marks every non-search variant required:["type"] with url/pattern nullable optionals. Demanding optional fields is the exact false-fatality class that invalidated bakeoff runs 1 and 3. Decision recorded in the comment and pinned by bare-discriminator test rows (54 tests, +2 param rows). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix+docs: close codex round-3 findings — ordering, replayability, exposure analysis P2 (ordering): webSearch action-shape validation now runs BEFORE the MODEL_RETURNED_NOT_SEARCHED early return, so a model NOT_SEARCHED verdict can never mask response-shape drift; mutation test added (55 tests). P2 (replayability): the 180 full receipt rows, the offline scorer (verified to reproduce the gate byte-for-byte from committed artifacts alone), and the parameterized fleet runner are committed beside the probe set; the report states the replayability boundary plainly (raw event streams are digest-only by transport design). P1 (answer-key exposure): empirical scan across all 540 retained receipts finds zero repo-referencing bound queries/sources; report gains an exposure-analysis section with scope caveats and corroboration; the structural fix (sealed hash-commit preregistration, fresh fabrication pool per run) is filed as #789 for future bakeoffs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-4 P2s — fleet gate, timeout margin, fresh-probe rule, nearest-rank p95 Scorer refuses truncated/duplicated/partial fleets (exactly one row per (ref_id, repeat) across 30x3) before computing any measure; p95 moves to the nearest-rank order statistic (51.13/27.46 -> 51.20/28.09, matching the review's own recomputation; gate unchanged) and the method is named on every surface. Runner outer timeout raised above the transport's inner 300s deadline so its finally-block cleanup always fires first. The canonical recorded-run note and report outcome now require a FRESH probe set for the API-route run per #789 (this set's labels are public), resolving the self-contradiction with the exposure analysis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-5 P2s — scorer consumes runner output + binds receipt identity score_run.py now scores either the committed run-4 JSONLs (default) or a fleet-runner output directory (argv[1]), so a reproduced fleet can never silently re-report the old result; every scored row must pass identity binding (outer model/ref/repeat, receipt.model, receipt.request_id, and a request_digest recomputed from the probe set via the transport's canonical form), refusing mis-associated or edited fleets. Verified: committed data reproduces the gate unchanged, results-dir mode scores the live run-4 cells, and a cross-model receipt swap is refused. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-6 P2s — dual-fleet hard-zero + unambiguous shape code Measure 4 now requires zero guard misfires in BOTH fleets (a baseline suppressed by tool misfires cannot anchor a fair comparison — the run-3 lesson, now enforced by the scorer). The transport emits EVENT_STREAM_INVALID for a non-null non-list search `results` value instead of silently skipping into NO_BOUND_SEARCH_RESULTS (wrong_search_shape fixture expectation updated in lockstep), and the scorer's shape family is trimmed to exactly the emitted shape codes with the behavior-family classification documented. Provably no effect on the scored run: run-4 contains zero rows in any affected code family (only SOURCE_NOT_IN_SEARCH_RESULTS 12/6, a behavior code) and both fleets already sit at zero misfires; all gate numbers unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-7 P2s — results-shape check before verdict return + same-day fleet enforcement The results-shape validation joins the pre-verdict scan loop so a model NOT_SEARCHED verdict can never mask dict-shaped search results (mutation test added: wrong_search_shape + NOT_SEARCHED -> EVENT_STREAM_INVALID; 56 tests). The runner refuses to resume over cells from an earlier date, and the scorer refuses mixed-date fleets across both models (run-4 is single-date; gate numbers unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-8 P2s — complete pre-verdict search validation + runner failure semantics The pre-verdict scan now performs the COMPLETE search-item strict validation (query type/length/control chars + results shape), covering legacy action-less items — the round-7 placement validated only action-typed items, which also made the round-7 mutation test fail (a red test my verification pipeline masked via tail; committed here only with PYTEST_EXIT=0 verified directly). Runner: a fleet with any failed call now exits nonzero instead of printing ALL DONE, and an outer- timeout kill sweeps the adapter's orphaned temp dirs (the detached app-server exits on stdin EOF; the ephemeral auth copy is what the verifier's skipped finally-block would have removed). 56 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-9 P2s — NOT_SEARCHED source contract, retry-not-skip, quiescent sweep, dated rows Transport: NOT_SEARCHED with a populated sources array fails closed as FINAL_OUTPUT_INVALID before the early return (mutation test; 57 tests). Runner: a same-day cell that recorded a failure is discarded and retried on resume instead of silently counting as complete; the orphan-tempdir sweep runs only after the executor drains so it can never delete a live worker's ephemeral CODEX_HOME. Scorer: every row must carry a real ISO date — an undated fleet cannot satisfy the same-day gate on empty strings. Committed run-4 data re-verified green under all new gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-10 findings — single-path validation, scorer gate exit, probe-hash pin P1 root treatment: the entire search-processing pipeline (cap, strict per-item validation, reference-bound filter, URL binding incl. the result-entry object-shape check) now runs BEFORE any verdict branch, so every shape-fatal path fires identically regardless of the model's answer — the verdict-masking bug class (rounds 3/7/8/9/10) is closed by construction, not by another patch. Emptiness outcomes stay verdict- conditional (an honest NOT_SEARCHED with no bound search remains model behavior). Mutation test: bound search with a non-object result entry + NOT_SEARCHED verdict -> EVENT_STREAM_INVALID (58 tests). Scorer: refuses a probe set whose whole-file sha256 differs from the frozen hash (labels now inside the scoring identity), and exits nonzero when any gate fails. The P1's rerun demand is accepted: a run-5 fleet under this frozen parser follows as the scored gate run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * docs: run 5 under the frozen final instrument is the gate result (#788 codex round-10 P1 accepted) Fleet rerun 2026-08-20 under parser+scorerdb6ed67(pushed pre-run): all five measures PASS with zero misfires on both fleets — recall 0.90 vs 0.80, grounded completion 0.911 vs 0.889, p95 26.1s vs 47.6s. Run 4 reclassified as a prior-instrument exploratory round; committed receipts/index/scorer default swapped to run-5 data (committed scorer replays the gate from repo artifacts alone, exit 0); all claim surfaces carry run-5 numbers and the cross-fleet consistency note (candidate led measures 1/2/5 in every full paired fleet). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-11 P1s — no failed-trial erasure + full-fleet entry validation (instrument fixpoint) Runner: a recorded failed trial is never deleted on resume; it is carried into the failure count and forces a nonzero exit, so the only path past a failure is rerunning the ENTIRE fleet fresh — selective retry-until-green is structurally impossible. (Provably no scored fleet was affected: runs 4 and 5 each completed in a single invocation with zero failures and no retry/carried lines in their logs.) Transport: the strict pre-verdict loop now validates every consumed field of EVERY search item — id, query, results-list shape, and each entry's object shape, bound or unbound — reaching the instrument fixpoint: no field the pipeline reads is unvalidated, so no future verdict-masking variant of this class exists. Mutation tests for unbound-malformed-entries and id shapes (60 tests). A run-6 fleet under this frozen instrument follows as the gate run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * docs: run 6 under the fixpoint instrument is the gate result Fleet rerun 2026-08-20 underadf18f9(parser fixpoint + no-erasure runner + full-gate scorer, all pushed pre-run): 5/5 PASS, zero misfires both fleets — recall 1.00 vs 0.70, grounded completion 0.911 vs 0.867, p95 28.8s vs 43.3s. Run 5 reclassified prior-instrument; artifacts and scorer default swapped to run-6; leak scan clean across all 900 retained receipts (runs 2-6); candidate led measures 1/2/5 in all four full paired fleets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-12 P2s — pinned effort, fleet-private temp root, split disclosure Runner pins ARS_CROSS_MODEL_REASONING_EFFORT to the provider default (explicitly unset per call, recorded per row) and routes all transport temp dirs into a fleet-private mkdtemp root so the timeout sweep can never touch another invocation's dirs. The audit now discloses the one run-6 1-1-1 split (baseline fab-05: MISMATCH/NOT_SEARCHED/NOT_FOUND -> INDETERMINATE, conservative miss) and names the actual baseline misses (fab-01, fab-08 majority NOT_SEARCHED; fab-05 split) — verified against the committed receipts, correcting a stale carried-over sentence. The effort variable was verified unset for every fleet (shell env + profile carry no export); gate numbers unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-13 P2s — receipt-contract validation, effort-marker gates, always-sweep Scorer validates every non-null receipt against the closed contract (required keys, verdict/searched types, positive-verdict grounding with fully-bound sources, empty sources on NOT_FOUND/NOT_SEARCHED, queries present when searched) before any metric trusts it, and results-dir scoring requires the pinned-effort marker on every row (committed gate-run rows predate the marker; the audit attests their configuration). Runner refuses carried cells without the marker and sweeps the fleet-private temp root in a finally-block on every outcome — a signal/OOM-killed verifier no longer leaves its ephemeral auth copy. Committed gate scoring still exits 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-14 P2s — per-verdict receipt invariants + runner rejects malformed parsed receipts Scorer: grounded verdicts (VERIFIED/MISMATCH/NOT_FOUND) require searched=true and a null reason_code; NOT_SEARCHED requires searched=false and a reason from the transport's closed emitted set — a fabricated NOT_FOUND-without-search or NOT_SEARCHED-with-search row can no longer contribute to recall or completion. Runner: a verifier exiting 0 with parsed-but-malformed output records RECEIPT_INVALID, counts as a failure, and forces nonzero exit. Committed run-6 gate scoring re-verified: exit 0, numbers unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-15 P2s — uniform item-field validation + shared receipt contract (axis terminal) Transport: every webSearch item — page-opens included — now has its action payload validated against the closed WebSearchAction variant types (url/pattern/query string-or-null, queries string-array), plus uniform id and results/entry shape checks; a recognized discriminator with a wrong-typed payload is stream-fatal (mutation sweep; 60 tests). Tooling: the full closed receipt contract (exact key set, transport/ auth_mode/containment, digest formats, per-verdict cross-field invariants) moves into a shared receipt_contract.py imported by BOTH run_fleet.py and score_run.py — one implementation, applied to fresh cells, resumed cells, and every scored row, so the two consumers cannot diverge. This terminates the validation axis: every field of every webSearch item and every key of every receipt is now checked; committed run-6 gate scoring re-verified exit 0 with numbers unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix+docs: close codex round-16 — strict source bindings; instrument-freeze boundary pinned (P1 declined with recorded rationale) receipt_contract.py enforces the full canonical source-binding shape (closed 4-key object, non-trivial https URL, non-empty item id, result_index 0-127 with bool exclusion) — committed run-6 data passes unchanged. The round-16 rerun demand is DECLINED under a pinned maintainer boundary, recorded in the report and the canonical recorded-run note: runs 4/5 were discarded because consumed-data gaps could alter scored outcomes; post-run-6 hardening validates only surfaces outside every consumed path and cannot change any verdict, binding, latency, or measure of a past fleet — such hardening applies from the next fleet. The disagreement is recorded, not hidden. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-17 P2s — complete schema mirror + latency-sample validation receipt_contract.py is now a COMPLETE stdlib mirror of the canonical receipt schema: identifier/event-id/https-url patterns and length bounds, array caps (queries<=32, sources<=16), closed entry objects, auth_mode const, NOT_SEARCHED => empty queries+sources with a mandatory reason, unknown reason codes refused globally. Scorer refuses boolean, negative, non-numeric, or absurd wall_seconds before the percentile gate. Committed run-6 gate scoring re-verified: exit 0, numbers unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-18 P2s — hashability, boolean identity, SIGTERM cleanup Validator: array/object verdict or reason_code becomes a contract failure instead of an uncaught TypeError (which would have escaped the runner's SystemExit handling and re-opened the no-reroll gap); containment flags are checked by identity (`is True`) so integer 1 cannot satisfy the schema's boolean constants. Runner: SIGTERM/SIGINT raise SystemExit so the finally-block sweep of the fleet-private auth copies also runs on cancellation. Mutation-verified (3/3 caught); committed run-6 scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-19 P2s — discriminator hashability, integer repeats, real ISO timestamps Transport: the WebSearchAction discriminator is type-checked before set membership in both _is_page_open and the uniform loop — an array/object type fails closed as EVENT_STREAM_INVALID instead of crashing the verifier past shape accounting. Scorer: repeat must be an exact int in 1..3 (1.0 satisfied the completeness Counter while minting ...-r1.0), and ts must parse as a full ISO timestamp with offset instead of a digit-shaped prefix. Committed run-6 scoring exit 0 unchanged; 60 transport tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-20 — source-query binding, midnight guard; null-action P2 declined with schema evidence receipt_contract.py rejects sources whose search_item_id has no retained entry in search_queries (unretained evidence never counts as grounding). run_fleet.py fails visibly BEFORE reporting success when a fresh fleet's cells span two calendar dates. The explicit-null-action P2 is declined on first-party evidence: ThreadItem types action as anyOf[WebSearchAction, null] (generate-json-schema, 0.147.0), so null is protocol-legal and follows the legacy path where the item still faces the complete strict validation — fatal-izing it is the run-1/run-3 false-fatality class; decision pinned in the code comment. Run-6 scoring exit 0 unchanged; 60 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-21 — cancellation stops queued quota burn, worker failures persist; open-variant P2 declined with schema evidence P1: SIGTERM/SIGINT set a STOP event making every queued job a no-op (marked [CANCELLED], counted as failure), so shutdown waits only for the at-most-3 in-flight calls instead of burning the rest of a paid 180-call fleet. P2: a worker exception after the paid call persists a failed cell with the job identity, so a resume can never treat the consumed trial as missing and re-roll it. The closed-variant P2 is declined on first-party evidence: no WebSearchAction variant sets additionalProperties, so extra fields are schema-legal and rejecting them would make any future informational field fleet-fatal; decision pinned in the code comment, known fields stay type-checked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-22 P1 — stray STOP=None placeholder no longer nullifies the cancellation event The round-21 placeholder assignment landed AFTER the Event creation in module order, resetting STOP to None and disabling queued-call cancellation exactly as the review read it. The placeholder is removed; the Event created before worker start is the one the signal handler sets. Static check pins that no STOP=None assignment remains. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-23 P2 — refuse contradictory receipt+error rows A row carrying both a valid receipt and a truthy error is structurally impossible from the runner and is refused as corrupted/external instead of being scored as grounded evidence; error rows with a null receipt stay counted as misfires. Committed run-6 scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-24 P2s — output-contract precedence, portable probe digest, runner preflight Transport: NOT_FOUND carrying sources is FINAL_OUTPUT_INVALID even when the stream also lacks a bound search — output-contract violations now outrank emptiness outcomes so the shape event cannot be misfiled as a behavior code. Probe digest verification moves into the shared module with CRLF->LF normalization (a Windows autocrlf checkout is not probe drift) and the runner runs the same preflight BEFORE any paid call, so 180 subscription calls can never be spent on a fixture the scorer will refuse. Run-6 scoring exit 0 unchanged; 60 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-25 — scorer-equivalent resume preflight + exclusive fleet lock validate_row (shared receipt_contract) now carries the COMPLETE row validation — contradiction, outer identity, exact-integer repeat, full ISO timestamp, sane latency, receipt identity binding to the probe row, closed receipt contract — and is the single implementation used by both the scorer and the runner's resume preflight, so a misnamed or copied cell fails before any further quota is spent. The runner takes an exclusive flock on the output dir, refusing a second concurrent invocation that would duplicate paid calls and race cell writes. Committed run-6 scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-26 — URL-binding drift hits measure 4; cross-platform fleet lock Transport: a bound search whose non-empty result entries yield no extractable URL is EVENT_STREAM_INVALID (provider moved/renamed the URL key = response-shape drift), no longer the behavioral NO_BOUND_SEARCH_RESULTS; the pre-existing pin of the old classification is updated in lockstep and a canonical_url regression test added (62 tests). Runner: the fleet lock falls back to msvcrt.locking on Windows, keeping the documented reproduction path viable. Note: run-6's receipts contain zero NO_BOUND_SEARCH_RESULTS / SOURCE_NOT-with-empty-binding rows of the reclassified kind (reason distribution: only SOURCE_NOT_IN_SEARCH_RESULTS with non-empty bindings and clean rows), so the gate numbers are provably unaffected; the change also falls under the pinned instrument-freeze boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-27 P2s — key-drift/value split, full resume validation, Windows invoke, UTF-8 I/O Transport: EVENT_STREAM_INVALID for empty bindings now requires that NO recognized URL key exists across the bound entries (true key drift); a recognized key with an unusable value stays behavioral — both sides test-pinned (62 tests). Runner: every resumed cell, failed ones included, faces validate_row + the effort check before further quota is spent; the transport is invoked via sys.executable (Windows honors no shebang); all subprocess/artifact text I/O pinned to strict UTF-8 in runner and scorer. Run-6 scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-28 P2 — URL-key drift determined pre-verdict The key-drift determination (bound entries present, no recognized URL key anywhere) moves before the NOT_SEARCHED early return, so a model NOT_SEARCHED answer can no longer mask renamed-URL-key response drift; the post-verdict emptiness branch keeps only behavioral outcomes. Masking regression test added (63 tests); run-6 scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-29 P1 — fleet runner gated to POSIX The #630 transport's process-group containment (start_new_session + os.killpg in _stop_process) is POSIX-only, so a native-Windows fleet would consume paid calls while every cell fails during cleanup — the rounds-26/27 surface accommodations implied support the deeper stack never had. The runner now refuses non-POSIX up front with a WSL pointer; the dead msvcrt lock branch is removed (the scorer, which is genuinely offline and portable, keeps its CRLF-tolerant digest and UTF-8 reads). Run-6 scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix+docs: close codex round-30 — receipt-level invariance proof replaces live-validation claim; session isolation; stray-file preflight P1 resolved by PROOF rather than a third rerun: run 6's 180 receipts carry exactly two reason states (null; SOURCE_NOT_IN_SEARCH_RESULTS 12/8) with zero receipt-less, error, NO_BOUND, NO_REFERENCE, FINAL_OUTPUT_INVALID, MODEL_RETURNED_NOT_SEARCHED, or EVENT_STREAM_INVALID rows — each post-run-6 transport change either touches unconsumed surfaces or only relabels cells in code families that provably never occurred, so no run-6 cell can differ under the shipped parser. The stale "run 6 live-validates the shipped parser" and "instrument FIXPOINT / no masking path remains" sentences are replaced with the precise provable statements; the freeze policy now REQUIRES this proof standard (no proof on a consumed path = rerun, as runs 4/5 were). P2s: verifier subprocesses start in their own session so an interactive Ctrl-C cannot turn in-flight calls into resume-poisoning EXIT failures; the runner refuses unexpected result files before spending quota. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-31 P2s — orphaned .tmp cells refused, fresh cells identity-bound The preflight refuses orphaned atomic-write temp files (a crash between write and rename must not silently re-roll a completed paid trial), and fresh cells face the same validate_row identity binding as resumed cells and the scorer before being persisted as success — a receipt with the wrong model/request_id/digest becomes a recorded RECEIPT_INVALID failure. Run-6 scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * feat: counterbalanced interleaved scheduling for the bakeoff fleet (#788 round-32 P1 accepted) The two models' calls for each (reference, repeat) cell are adjacent in the queue with deterministic parity-alternating pair order, so model identity is decorrelated from execution time — provider load or web-search drift during the fleet can no longer masquerade as a model effect. A counterbalanced run-7 follows as the scored gate run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * docs: counterbalanced run 7 is the gate result; run 6 superseded for order confound Run 7 under frozen instrument69cd04a(interleaved parity-alternating pair scheduling): all five measures PASS — recall 0.90 vs 0.80, p95 25.0s vs 49.6s (median 14.8 vs 17.5), grounded completion tied at 0.900, zero misfires both fleets, two 1-1-1 splits disclosed and scored as conservative misses. Honesty note carried on every claim surface: the sequential fleets' grounded-completion edge did NOT survive counterbalancing and is not claimed; recall and latency led in all five paired fleets. Artifacts and scorer default swapped to run-7 (committed scorer replays the gate, exit 0); leak scan clean across 1,080 retained receipts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * fix: close codex round-33 P1 — resume refuses half-complete counterbalanced pairs Every (reference, repeat) pair must be wholly present or wholly missing on resume: a one-sided pair would run the counterpart far from its partner and silently reintroduce the model-vs-time confound. Run 7 is unaffected (single uninterrupted invocation); scoring exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j * docs: close codex round-34 P2 — recorded-run note limited to the measured superiority (2 and 5, tie on 1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
evals/ — gold-set corpora for ARS measurement targets
This directory holds the v3.10 #184 generalized gold sets. Each subdirectory under gold/ is a self-contained gold set for one measurement target. The structure mirrors the v3.8 scripts/fixtures/claim_audit_calibration/ pattern but generalizes to multiple targets.
Layout
evals/
├── README.md # this file
├── gold/
│ ├── citation_extraction/ # Phase 1a/1b — baseline for #182
│ │ ├── README.md
│ │ ├── manifest.yaml
│ │ ├── tuples/
│ │ │ └── NNN-{kind-slug}-{discriminator}.json
│ │ └── expected_outcomes.json
│ ├── rq_framing_patterns/ # #257 Socratic wording advisory calibration
│ ├── status_classification/ # Phase 2 (lands post-#183)
│ └── summarization_adequacy/ # Phase 2 (lands post-#183)
Running the harness (#184 Phase 1b)
The multi-task harness scripts/run_evals.py discovers every gold/<task>/manifest.yaml, measures each task, and emits a report shaped by shared/evals_lift_report.schema.json:
PYTHONPATH=. python -m scripts.run_evals # all tasks
PYTHONPATH=. python -m scripts.run_evals --task citation_extraction --output report.json
PYTHONPATH=. python -m scripts.run_evals --baseline before.json --compare after.json
--baseline + --compare produce a side-by-side report (lift_pre / lift_post). The ranking-lift gate scripts/check_ranking_lift.py reads those reports and blocks on un-acknowledged regressions; CI wires both via .github/workflows/eval-harness.yml (Delta 3 path filter). Tasks whose entrypoint module or gold set is not yet present are reported as pending/skipped rather than failing.
Authoring conventions
See each task's README.md for task-specific conventions (tuple naming, kind distributions, expected outcomes shape).
Validator
Run python -m scripts.check_evals_gold_set evals/gold/<task> to validate any gold set against its manifest. The same validator runs in CI on every PR that touches evals/gold/**.
Provenance
- Phase 1a (citation-extraction gold set) + Phase 1b (
run_evals.pyharness + lift gate): v3.10 #184, specdocs/design/2026-05-21-v3.10-184-extend-eval-harness-spec.md - RQ framing patterns: Kong #257 idea-diversity advisory, spec
docs/design/2026-05-28-kong-257-idea-diversity-coverage-gap-advisory.md - Phase 2 (status + summarization): scheduled post-#183 ship