Edward Cheng-I Wu
90f2176cd5
* evals: add `claude plugin eval` suite for the academic-paper citation-check flow Eight cases (six fire, two negative) under plugin-evals-citation-check/, graded as a with/without-plugin ablation. Fire cases carry a synthetic source pack so the four author-defined citation failures (no source on file, wrong authors, hedged finding cited as established, retracted or concern-flagged paper cited as live) are detectable offline. Styles: APA 7 (en / zh-TW mixed / es), IEEE, Vancouver (style unnamed), Chicago NB. Cases pin model: sonnet; run with --judge-model opus. Calibration (two pilots, 2026-09-13) and caveats are in the suite README. The with-plugin arm cannot load the mode prompt in the eval sandbox because the command stub references plugin files by relative path (#857), and plain-language prompts fired the skill in 3 of 6 cases (#858; the Spanish case is one data point for #850). No uplift figure is claimed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH * evals(citation-check): apply cross-model review findings and re-calibrate Twelve of thirteen review findings applied: replace the disputable four-author "et al." planting in 04 with a year mismatch; make the clean citations in 02 and 06 supported by their abstracts; replace real Taiwan journal names in 02 with fictional ones; tie the 08 presence regexes to an "unused" statement; require metadata preservation and reject audit content on the 07 conversion negative; add no-overreach to 05; turn the 02 language check into an llm grader; exempt unchanged entries in a complete corrected list from no-false-positive and drop its DOI claim; drop the 04 style-name regex (both arms fixed the year without naming the style). The one rejected finding (08 skill check "display-only") was wrong: it carries arm: both and is scored; README says so. Pilot 3 on the revised suite: $4.65, max 142 s / 7 turns / $0.45 per run; 04 and 08 re-run clean after the last two grader fixes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1.1 KiB
type, focus, weight
| type | focus | weight |
|---|---|---|
| llm | last_message | 1.5 |
The user supplied a source pack and said it is complete. Two references have problems that only the source pack reveals. Work through the two checks one at a time and quote the sentence or table row in the response that you rely on.
Check 1 — Brandt et al. (2019). The reference list attributes "Belonging as a mediator of peer-mentoring effects on second-year enrolment" to Brandt, Lindqvist & Moreau, but the source pack shows the published paper is by Ferreira & Nakamura. The response must identify this entry and say that the authors do not match the source (attribution error, wrong authors, misattributed, or equivalent wording). Merely noting a formatting issue on this entry does not count.
Check 2 — Ellison (2023). This reference does not appear in the source pack at all. The response must identify it and say that it could not be matched to any source the user has (unverifiable, not in source pack, possibly fabricated, needs the author to supply the source, or equivalent). Treating it as verified, or not mentioning it, fails this check.
Pass only if both checks are satisfied.