Edward Cheng-I Wu
90f2176cd5
* evals: add `claude plugin eval` suite for the academic-paper citation-check flow Eight cases (six fire, two negative) under plugin-evals-citation-check/, graded as a with/without-plugin ablation. Fire cases carry a synthetic source pack so the four author-defined citation failures (no source on file, wrong authors, hedged finding cited as established, retracted or concern-flagged paper cited as live) are detectable offline. Styles: APA 7 (en / zh-TW mixed / es), IEEE, Vancouver (style unnamed), Chicago NB. Cases pin model: sonnet; run with --judge-model opus. Calibration (two pilots, 2026-09-13) and caveats are in the suite README. The with-plugin arm cannot load the mode prompt in the eval sandbox because the command stub references plugin files by relative path (#857), and plain-language prompts fired the skill in 3 of 6 cases (#858; the Spanish case is one data point for #850). No uplift figure is claimed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH * evals(citation-check): apply cross-model review findings and re-calibrate Twelve of thirteen review findings applied: replace the disputable four-author "et al." planting in 04 with a year mismatch; make the clean citations in 02 and 06 supported by their abstracts; replace real Taiwan journal names in 02 with fictional ones; tie the 08 presence regexes to an "unused" statement; require metadata preservation and reject audit content on the 07 conversion negative; add no-overreach to 05; turn the 02 language check into an llm grader; exempt unchanged entries in a complete corrected list from no-false-positive and drop its DOI claim; drop the 04 style-name regex (both arms fixed the year without naming the style). The one rejected finding (08 skill check "display-only") was wrong: it carries arm: both and is scored; README says so. Pilot 3 on the revised suite: $4.65, max 142 s / 7 turns / $0.45 per run; 04 and 08 re-run clean after the last two grader fixes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1.0 KiB
type, focus, weight
| type | focus | weight |
|---|---|---|
| llm | last_message | 1.5 |
Two problems are visible only against the source pack. Work through them one at a time and quote the sentence or row in the response you rely on.
Check 1 — Halvorsen, Salt Roads and Ledgers (note 2, bibliography). This work does not appear in the source pack. The response must identify it and say it could not be matched to any source the user has (unverifiable, not in pack, possibly fabricated, author must supply it, or equivalent). Treating it as verified, or not mentioning it, fails.
Check 2 — Marlowe (note 5). The text says Marlowe "establishes that toll rates were fixed by a central authority". The source pack abstract says the article proposes this only as a hypothesis for further testing and notes the evidence is also compatible with regional coordination. The response must identify this sentence and say the claim overstates the source (hypothesis presented as established, claim strength exceeds source, or equivalent).
Pass only if both checks are satisfied.