mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
Edward Cheng-I Wu
75070eec84
* feat(evals): #653 reviewer-calibration suite scaffolding — corpus assembler, isolated dispatcher, deterministic scorer, pre-registered rubric/RUN_PLAN (corpus freeze pending PDF access) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2iNYa6YYYaPUwD2Z2Jr5e * feat(evals): #653 freeze the ICLR 2026 calibration corpus manifest (12 papers) + shared PDF-text normalization Corpus freeze (PR-A of #653): `corpus/papers.json` (label-free, 6+6 ICLR 2026 papers by the pre-registered seed; pypdf 6.11.0; pool hashes unchanged from the 2026-08-07 selection) and `manifests/gold_labels.json` (public Decision note ids + strings). No page-cap exclusion fired; `verify` PASS. First real-PDF contact found a hashing defect: pypdf emits lone UTF-16 surrogates from math fonts (61 in one sampled manuscript) and strict UTF-8 encoding raised, so `extracted_text_sha256` was uncomputable. The normalization now lives in one shared module (`scripts/_calibration_pdf_text.py`: NFC + lone-surrogate -> U+FFFD), imported by both the assembler and the dispatcher so freeze/verify/dispatch hash identical bytes; the rule is recorded in the manifest's `extraction.text_normalization` and `verify` fails hard on rule drift (a rule, not a version). Two tests added (41 total). `scripts/fetch_calibration_corpus.py` is the authenticated OpenReview operator tool that produces the freeze input, so the "third-party reconstruction" claim in the README is backed by a runnable path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EehvYnmG8Xym5tXhTLfVm1 * refactor(evals): #653 simplify pass — shared hashing/fence/git-state, contract 1.1 docs /simplify findings applied (reuse, simplification, efficiency, altitude): - `_calibration_pdf_text.py` owns `sha256_hex` + `pdf_facts` (bytes hashed and parsed from one read via BytesIO; `extract_text=False` lets `verify` skip extraction when the pypdf version cannot be compared); surrogate replacement is one `re.sub` pass. Both the assembler and the dispatcher import it. - dispatcher reuses E4's closed data-fence grammar (`_delimited`), `_git_state` (declares unknown provenance dirty instead of raising), and the evidence path guard (`assert_plain_file`: rejects symlinked parent components, not just the leaf); one `_prepare` preamble for both stages; a text-hash mismatch now names its cause (installed vs manifest pypdf version). - assembler: exclusion rows stay dicts, `pool_list_mismatches` shared by freeze/verify, exclusion set built once. - scorer: `confusion`/`bootstrap_ci` take (predicted, gold) pairs (same RNG stream as before), `Counter` for the exact-mode vote, dead `_path` dropped. - RUN_PLAN/README: measurement contract 1.0 is closed to new rows (#664); the run publishes under 1.1 with its pre-registration record + write-once execution manifest (dispatcher/scorer support lands with the scored run). Re-freeze after the refactor reproduces papers[] and gold_labels byte-for-byte. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EehvYnmG8Xym5tXhTLfVm1 * fix(evals): #653 Iron Rule #7 at the two whole-file call boundaries + paper-id shape check Security review round 1 (first-party) found two below-threshold gaps and both are verified real: - The calibration dispatcher omitted E4's `DATA_BOUNDARY` sentence on the field-analyst call (the one E4 call that carries it, because `field_analyst_agent.md` states no untrusted-material rule of its own). Restored, and a fitted `REPORT_BOUNDARY` added on the synthesizer call, whose agent file is likewise dispatched whole with no such rule. Pinned by a transport-capture test that checks both sentences precede their fence. - Paper ids are spliced into file names (`<id>.pdf`, `cards/<id>/`) but `load_pool` accepted any non-empty string. Ids now must match `^[A-Za-z0-9_-]+$` (OpenReview's forum-id shape) in the assembler and the fetch tool; test pins the refusal. 43 tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EehvYnmG8Xym5tXhTLfVm1 * fix(evals): #653 codex round 1 — dispatch/verify invariant parity, card-path guard, scorer completeness Codex round 1 (gpt-6-astra xhigh) findings 2-7, 10, 11 and the cheap half of 9, each re-verified first-party before the change: - dispatcher: frozen cards go through the same plain-file guard as PDFs and agent files (a symlinked card1.md -> gold_labels.json was readable); the manifest's text_normalization rule and page_count are checked before dispatch, so dispatch and verify enforce the same manuscript invariants; transport-failure artifacts keep the partial stdout and stderr verbatim; every call attempt records RFC-3339 start/complete and prompt/output hashes into the panel record and cards freeze (the per-call evidence the heldout-measurement/1.1 execution manifest is built from). - verify: label must match decision_raw under the label transform; paper count and per-class label counts must equal the recorded quotas (synchronized paper+label removal no longer passes). - scorer: a second record for the same paper/replicate is a hard error, not a silent overwrite; a gold paper with no complete ensemble blocks the full tier; an A1 override needs its verbatim `raw` excerpt present in synthesis.md. Nine regression tests added (52 total). Real-corpus verify still PASS. Not addressed here (need a decision): finding 1 (camera-ready format leaks the accept label) and finding 8 (numeric seat scores vs categorical seat contract); finding 9's manifest/row builders land with the scored run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EehvYnmG8Xym5tXhTLfVm1 * fix(evals): #653 drop the numeric score axis — protocol Phase 2 forbids AUC, seats are categorical Codex round 1 finding 8, verified against the source: the seat contract (eic/methodology/... agents) emits criterion-bound categorical judgements and states "Do not total, weight, average"; `calibration_mode_protocol.md` Phase 2 says "Do not report AUC: there is no continuous rubric score." The scorer nevertheless extracted a `Weighted Average` figure (a retired field) and RUN_PLAN promised AUC + score variance, so a conforming run would have published null numerics against a plan that promised them. The scorer now reports only what the protocol's full-tier table names: confusion matrix, balanced accuracy, FNR, FPR (bootstrap CIs), exact-label agreement (count/share/target-set size, with the binary-gold caveat), and replicate stability as categorical agreement (on side, on exact label). AUC is emitted as an explicit NOT REPORTED line. RUN_PLAN and the test fixtures follow. 52 tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EehvYnmG8Xym5tXhTLfVm1 * docs(evals): #653 mark the 2026-09-06 corpus SUPERSEDED (layout leaks the label, #828); RUN_PLAN model currency - README/RUN_PLAN: the frozen ICLR 2026 corpus is a harness-rehearsal corpus only — camera-ready replacement makes accepted PDFs visibly different from rejected submission PDFs (6/6 + 6/6; 30/30 in a fresh accepted-pool sample). No profile or measurement row may be published from it; the gold corpus becomes an ICLR 2027 submission-time capture. The "Why ICLR 2026" rationale is kept as pre-registered and annotated with the two facts that now cut against it (layout leak; Fable 5.1's 2026-06 cutoff covers the decisions). - RUN_PLAN + dispatcher default: subject `claude-fable-5` -> `claude-fable-5-1`, judge `gpt-5.6-sol` -> `gpt-6-astra` (provisional, #783 policy). Pre-dispatch edits, not amendments. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EehvYnmG8Xym5tXhTLfVm1 * feat(evals): #828 layout-tell guard at corpus freeze — refuse a corpus whose page-1 layout is not constant `assemble_calibration_corpus.py freeze` now reads page 1 of every cached PDF and evaluates four venue-template signals (published-as header, under-review header, "Anonymous authors", >=10 bare three-digit line numbers). Any signal that is not constant across the whole corpus refuses the freeze with the per-class counts; a uniform corpus records `layout_tell_check` in papers.json. `verify` recomputes the same check (skipped with a warning when a PDF is not cached; a manifest without the block warns). On the superseded 2026-09-06 ICLR 2026 corpus every signal is 6/0, so `verify` now FAILs on it by design. Shared `_open_reader` + `first_page_text` in the PDF helper. Six tests (signal detection, full and partial separation refused, uniform freeze + verify round-trip, missing-PDF skip, pre-check manifest warning). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bJSoFkGeRaWTMJodk4VGh * feat(evals): #653/#828 rehearsal fixes + heldout-measurement/1.1 manifest and row builders Rehearsal 2026-09-06 (2 papers x 1 replicate, blocked at the first call by a rejected API key) exposed three dispatcher gaps, all fixed with tests: - credential preflight: zero-cost `GET /v1/models` before the first billed call; a definitive 401/403 refuses (key never echoed), network trouble is `inconclusive` and proceeds; outcome recorded in every record - credential rejection mid-run (`Failed to authenticate` / `API Error: 401` / `Not logged in`) is never retried (`CredentialRejected`); other transport failures keep the single retry - an aborted cards stage writes `runs/blocked-cards-<paper>.json` with its per-call rows instead of losing them; both stages share one record writer 1.1 contract substrate (RUN_PLAN "pre-registration record + execution manifest" item): - `dispatch_calibration_panel.py --stage manifest` folds the completed call rows of one attempt (frozen cards + panel records; `load_attempt` refuses mixed attempt identities) into a write-once, schema-validated `execution-manifest.json` - `build_calibration_measurement_row.py` composes the 1.1 row: plan and rubric hashed and compared against `frozen_commit` (drift refuses; dirty commit refuses), manifest re-derived from the records and compared field-for-field, judge rows required (no judges, no row), agreement recomputed by the checker's own `judge_divergence` (extracted from `check_heldout_measurement_report.py`, behaviour unchanged), validated by the checker before a write-once write - adjudication rubric gains `## Resolution direction` (flags_only, I13 lower-bound labelling); README tooling section; RUN_PLAN names the row builder; DATA_FLOWS names the dispatcher's preflight touchpoint; scorer docstring de-staled (no score axis); pytest manifest +1 No calibration number is recorded anywhere in the repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bJSoFkGeRaWTMJodk4VGh * fix(evals): #653/#828 codex round 2 — bind every row input to its attempt, harden the guards 12 of 13 findings applied (gpt-6-astra xhigh, read-only exec): - P1 foreign metrics: scorer output is bound to the attempt (per_panel keys == the complete panel records here, attempt ids match, n_papers matches) - P1 raw drift: record admission re-hashes every completed call's raw output against output_sha256 (manifest stage and row builder alike); prompts are not retained (they embed the manuscript) - P1 preflight redirects: the probe uses a no-redirect opener (a 3xx is `inconclusive`) and skips a non-https ANTHROPIC_BASE_URL - P2 estimand: class-A adjudication is now pre-registered as bidirectional (every synthesis decision transcribed blind and compared with the grammar), so the row publishes a point_estimate instead of an I13 "lower bound" that only meant audit coverage - P2 pre-write parity with R5: manifest timestamps parsed and ordered before the write; declared claims checked against the local manifest - P2 strict JSON: inputs parsed with the checker's strict loader, outputs serialized with allow_nan=False and round-tripped - P2 judge failures: `--blocked-run` ledger entries merge into attempts.blocked_runs (I11) - P2 admission by content: suite/stage/status/provenance from the record body, never the filename; blocked records are identity-checked too - P2 cards re-run: a reused evidence dir refuses (write-once stage records) - P2 auth signature: anchored at the start of stdout/stderr and limited to exit-code failures; a timeout's partial prose is never a credential error - P2 layout signals: phrase tests run on whitespace-folded text - P2 partial PDF cache: verify checks every cached PDF (can refuse, cannot clear) instead of skipping the guard - P3 real `git show` test for sha256_at_commit on a temporary repository Partially applied: "distinguish unobservable signals from absence" (not built; the constancy rule is pre-registered as stricter by design). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bJSoFkGeRaWTMJodk4VGh * fix(evals): #653/#828 shared transport — capture every assistant message, fence the subject's config Rehearsal take 2 (2026-09-06/07, 8 billed calls on the first paper) found two transport defects in `ClaudeCliTransport`, shared by the E4 and the calibration dispatchers: - text-mode `claude -p` prints only the LAST assistant message: the first paper's synthesis (long enough to be continued) came back starting mid-table, with the Editorial Decision Letter and its `### Decision:` line in the missing head. The transport now runs `--output-format stream-json --verbose` and concatenates the text blocks of every assistant message; an error result or an unreadable stream is a TransportFailure that keeps the raw bytes. - `--bare` does not fence the subject: a two-call probe on 2.1.260 showed the operator's whole global CLAUDE.md arriving as a system-reminder, plus `settings.json` `language` and the output style (the seats appended Traditional-Chinese "plain-language summary" sections). The subject now runs with an allowlisted environment (PATH/HOME/LANG/TMPDIR/TERM/USER/ SHELL + ANTHROPIC_*; no CLAUDE_* inherited from a parent session) and a per-transport empty `CLAUDE_CONFIG_DIR`; the same probe then reported no instruction beyond the SDK identity line and the date. E4 tests: one fake updated to emit stream-json; five new tests (message joining, error/junk results, unreadable-stream failure with bytes, environment allowlist, argv/env of a live call). Calibration docs and the panel record's `dispatch` field describe the new recipe (pre-dispatch change, no amendment). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bJSoFkGeRaWTMJodk4VGh * fix(evals): #653/#828 codex round 3 on the shared transport — eviction signals, LF framing, network env, failure evidence Five P2 findings (gpt-6-astra xhigh, read-only exec), all applied: - refusal-fallback eviction: assistant `supersedes` and system `model_refusal_fallback.retracted_message_uuids` (wire fields verified in the installed CLI 2.1.260) drop retracted partials before concatenation - NDJSON split on LF only (`str.splitlines` also splits on U+0085 / U+2028 / U+2029 inside a JSON string); CRLF tolerated - environment allowlist keeps documented network/TLS inputs (proxies, NODE_EXTRA_CA_CERTS, SSL_CERT_*, CLAUDE_CODE_CLIENT_*); an apiKeyHelper that needs more is documented as unsupported behind the fence - transport failures carry assistant TEXT in `stdout` and the raw stream in `raw_stdout`; a framing-only stream is "no model response" (E4 no longer writes stream metadata as a partial response); both dispatchers preserve the raw stream as `*.transport-stream.jsonl` - a structured error result (`[TRANSPORT: result <subtype>]`, diagnostic in stdout) is classified by the calibration retry loop like the plain-text startup failure: a credential rejection is never retried E4 tests +6 (256), calibration +1. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bJSoFkGeRaWTMJodk4VGh * feat(evals): #653/#828 keep the raw stream of successful calls as evidence `ClaudeCliTransport.last_raw_stdout` exposes the stream-json framing of the most recent successful call; the calibration dispatcher writes it next to the text as `<label>.transport-stream.jsonl`, so the next rehearsal shows how many assistant messages a deliverable spanned (the 2026-09-06 synthesis lost its head to exactly that). Probe 2026-09-07: a 12,000-line reply at effort low arrived as ONE text message after a thinking-only message, so the head loss is attributed to multiple text messages in one turn (likely interleaved thinking at xhigh), not to an output-length continuation; the parser covers both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014bJSoFkGeRaWTMJodk4VGh * fix(evals): #653/#828 allow requiring a successful credential preflight * fix(calibration): bind audited decisions and preserve failed dispatch evidence * fix(transport): retain truncated UTF-8 output as byte evidence --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
evals/ — gold-set corpora for ARS measurement targets
This directory holds the v3.10 #184 generalized gold sets. Each subdirectory under gold/ is a self-contained gold set for one measurement target. The structure mirrors the v3.8 scripts/fixtures/claim_audit_calibration/ pattern but generalizes to multiple targets.
Layout
evals/
├── README.md # this file
├── gold/
│ ├── citation_extraction/ # Phase 1a/1b — baseline for #182
│ │ ├── README.md
│ │ ├── manifest.yaml
│ │ ├── tuples/
│ │ │ └── NNN-{kind-slug}-{discriminator}.json
│ │ └── expected_outcomes.json
│ ├── rq_framing_patterns/ # #257 Socratic wording advisory calibration
│ ├── status_classification/ # Phase 2 (lands post-#183)
│ └── summarization_adequacy/ # Phase 2 (lands post-#183)
Running the harness (#184 Phase 1b)
The multi-task harness scripts/run_evals.py discovers every gold/<task>/manifest.yaml, measures each task, and emits a report shaped by shared/evals_lift_report.schema.json:
PYTHONPATH=. python -m scripts.run_evals # all tasks
PYTHONPATH=. python -m scripts.run_evals --task citation_extraction --output report.json
PYTHONPATH=. python -m scripts.run_evals --baseline before.json --compare after.json
--baseline + --compare produce a side-by-side report (lift_pre / lift_post). The ranking-lift gate scripts/check_ranking_lift.py reads those reports and blocks on un-acknowledged regressions; CI wires both via .github/workflows/eval-harness.yml (Delta 3 path filter). Tasks whose entrypoint module or gold set is not yet present are reported as pending/skipped rather than failing.
Authoring conventions
See each task's README.md for task-specific conventions (tuple naming, kind distributions, expected outcomes shape).
Validator
Run python -m scripts.check_evals_gold_set evals/gold/<task> to validate any gold set against its manifest. The same validator runs in CI on every PR that touches evals/gold/**.
Provenance
- Phase 1a (citation-extraction gold set) + Phase 1b (
run_evals.pyharness + lift gate): v3.10 #184, specdocs/design/2026-05-21-v3.10-184-extend-eval-harness-spec.md - RQ framing patterns: Kong #257 idea-diversity advisory, spec
docs/design/2026-05-28-kong-257-idea-diversity-coverage-gap-advisory.md - Phase 2 (status + summarization): scheduled post-#183 ship