* evals: add `claude plugin eval` suite for the academic-paper citation-check flow
Eight cases (six fire, two negative) under plugin-evals-citation-check/,
graded as a with/without-plugin ablation. Fire cases carry a synthetic
source pack so the four author-defined citation failures (no source on
file, wrong authors, hedged finding cited as established, retracted or
concern-flagged paper cited as live) are detectable offline. Styles: APA 7
(en / zh-TW mixed / es), IEEE, Vancouver (style unnamed), Chicago NB.
Cases pin model: sonnet; run with --judge-model opus.
Calibration (two pilots, 2026-09-13) and caveats are in the suite README.
The with-plugin arm cannot load the mode prompt in the eval sandbox
because the command stub references plugin files by relative path
(#857), and plain-language prompts fired the skill in 3 of 6 cases (#858;
the Spanish case is one data point for #850). No uplift figure is claimed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH
* evals(citation-check): apply cross-model review findings and re-calibrate
Twelve of thirteen review findings applied: replace the disputable
four-author "et al." planting in 04 with a year mismatch; make the clean
citations in 02 and 06 supported by their abstracts; replace real Taiwan
journal names in 02 with fictional ones; tie the 08 presence regexes to an
"unused" statement; require metadata preservation and reject audit content
on the 07 conversion negative; add no-overreach to 05; turn the 02 language
check into an llm grader; exempt unchanged entries in a complete corrected
list from no-false-positive and drop its DOI claim; drop the 04
style-name regex (both arms fixed the year without naming the style).
The one rejected finding (08 skill check "display-only") was wrong: it
carries arm: both and is scored; README says so.
Pilot 3 on the revised suite: $4.65, max 142 s / 7 turns / $0.45 per run;
04 and 08 re-run clean after the last two grader fixes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Edward Cheng-I Wu