mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
evals: add claude plugin eval suite for the academic-paper citation-check flow (#859)
* evals: add `claude plugin eval` suite for the academic-paper citation-check flow Eight cases (six fire, two negative) under plugin-evals-citation-check/, graded as a with/without-plugin ablation. Fire cases carry a synthetic source pack so the four author-defined citation failures (no source on file, wrong authors, hedged finding cited as established, retracted or concern-flagged paper cited as live) are detectable offline. Styles: APA 7 (en / zh-TW mixed / es), IEEE, Vancouver (style unnamed), Chicago NB. Cases pin model: sonnet; run with --judge-model opus. Calibration (two pilots, 2026-09-13) and caveats are in the suite README. The with-plugin arm cannot load the mode prompt in the eval sandbox because the command stub references plugin files by relative path (#857), and plain-language prompts fired the skill in 3 of 6 cases (#858; the Spanish case is one data point for #850). No uplift figure is claimed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH * evals(citation-check): apply cross-model review findings and re-calibrate Twelve of thirteen review findings applied: replace the disputable four-author "et al." planting in 04 with a year mismatch; make the clean citations in 02 and 06 supported by their abstracts; replace real Taiwan journal names in 02 with fictional ones; tie the 08 presence regexes to an "unused" statement; require metadata preservation and reject audit content on the 07 conversion negative; add no-overreach to 05; turn the 02 language check into an llm grader; exempt unchanged entries in a complete corrected list from no-false-positive and drop its DOI claim; drop the 04 style-name regex (both arms fixed the year without naming the style). The one rejected finding (08 skill check "display-only") was wrong: it carries arm: both and is scored; README says so. Pilot 3 on the revised suite: $4.65, max 142 s / 7 turns / $0.45 per run; 04 and 08 re-run clean after the last two grader fixes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EEbmXsHyQ3dzYoep74zSmH --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
committed by
GitHub
GitHub
parent
88725b8a55
commit
90f2176cd5
@@ -31,3 +31,4 @@ htmlcov/
|
||||
|
||||
# Plugin eval run artifacts (HTML reports + aggregate JSON; regenerated by `claude plugin eval`)
|
||||
/plugin-evals/results/
|
||||
/plugin-evals-citation-check/results/
|
||||
|
||||
@@ -8,6 +8,8 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
- **`claude plugin eval` suite for the `academic-paper` revision-coach flow (`plugin-evals/`; toward #851, not closed).** Seven cases (five that should fire, two that should not) with twenty graders, run as a with/without-plugin ablation so the headline is the uplift Δ, not a pass rate. The author-defined quality axis is no unauthorised rewriting: the response must not draft manuscript prose, change anything no reviewer asked for, or assert results or changes that have not happened; secondary axes are no comment dropped, push-back allowed on a factually wrong reviewer point, and a committee letter yields a tracker with no peer-review grading. Inputs are synthetic and cover four shapes (mixed-format journal comments, an unnumbered decision-letter email, OpenReview-style scores with a "should we push back" question, and a formal ethics-committee letter), in zh-TW and English. Calibration against the 2026-09-12 pilots is recorded in `plugin-evals/README.md` together with the run command, the `timeout_seconds` / `max_turns` ceilings, the observed pilot cost, and three caveats — including that the ICLR case did not invoke the skill in either pilot, which is filed as #851 and uses that case's display-only `skill-fired` grader as its acceptance check. `plugin-evals/results/` is gitignored. This adds a regression guard only; it claims no measured uplift figure (a single-run pilot is not a measurement).
|
||||
|
||||
- **`claude plugin eval` suite for the `academic-paper` citation-check flow (`plugin-evals-citation-check/`; surfaces #857 and #858, neither closed).** Eight cases (six that should fire, two that should not), run as a with/without-plugin ablation so the headline is the uplift Δ, not a pass rate. The author-defined quality axis is the four citation failures that matter: a reference the user has no source for, the right paper with the wrong authors, a hedged or minor finding cited as an established result, and a retracted or concern-flagged paper cited as live evidence. Every fire case ships a complete synthetic source pack so all four are detectable offline; the sandbox has no network, so "no source" is graded as flagging the entry as unverifiable, never as a real lookup. Secondary axes: mechanical errors, no false positives on clean entries, no claim of online verification, no rewriting of the manuscript. Cases cover APA 7 (en, zh-TW mixed, es), IEEE, Vancouver with the style unnamed, and Chicago notes-bibliography; the negatives are an APA-to-IEEE conversion request and a Python unused-import question. Cases pin `model: sonnet` and the run command sets `--judge-model opus` so the judge is never the agent model. Calibration against the 2026-09-13 pilots is recorded in `plugin-evals-citation-check/README.md` with the run command, the `timeout_seconds` / `max_turns` ceilings, the observed pilot cost, and the caveats: the with-plugin arm cannot load the mode's own prompt in the eval sandbox because the command stub references plugin files by relative path (#857), so Δ ≈ 0 is the honest current reading; plain-language prompts fired the skill in 3 of 6 cases (#858, with the Spanish case as one data point for #850). `plugin-evals-citation-check/results/` is gitignored. This adds a regression guard only; it claims no measured uplift figure.
|
||||
|
||||
- **Reviewer-calibration harness with isolated dispatch and audited scoring (#835; toward #653 and #828, neither closed).** `evals/heldout/reviewer_calibration/` gains the held-out calibration suite and five scripts: `fetch_calibration_corpus.py` / `assemble_calibration_corpus.py` freeze a hash-verified manuscript corpus, `dispatch_calibration_panel.py` and `dispatch_e4_panel.py` run isolated reviewer-panel cards through the shared Claude CLI transport, `build_calibration_measurement_row.py` emits contract-validated measurement rows, and `score_calibration_run.py` scores categorical panel decisions (no AUC: the panel produces decisions, not probabilities). Every dispatch retains write-once outputs, raw transport streams, timestamps, and prompt/output hashes; the transport isolates operator configuration (empty `CLAUDE_CONFIG_DIR`, environment allowlist) and reconstructs every assistant message from `stream-json`, including replacement and retraction events, so a synthesis whose Decision line sits in an earlier message is not lost. A zero-cost credential preflight refuses deterministic authentication retries and preserves aborted-run evidence; the optional `--require-preflight-ok` gate stops before transport construction when a skipped or inconclusive preflight would violate the run's acceptance criteria, and TLS trust failures get a safe diagnostic. Complete assistant frames are preserved ahead of a truncated stream tail; interrupted cards and panel stages are recorded as blocked with their call ledger, which counts CLI dispatch attempts, not provider requests. The retained ICLR 2026 corpus is marked SUPERSEDED and usable only for harness rehearsals (#828: its page-1 layout reveals labels), and corpus freeze and verification reject nonconstant page-1 layout signals. No calibration profile and no rehearsal measurement values ship with this change; the remaining ICLR 2027 corpus capture and the Phase 3.5 two-family judge work are documented as open.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1.5
|
||||
---
|
||||
The user supplied a source pack and said it is complete. Two references have problems that only the source pack reveals. Work through the two checks one at a time and quote the sentence or table row in the response that you rely on.
|
||||
|
||||
Check 1 — Brandt et al. (2019). The reference list attributes "Belonging as a mediator of peer-mentoring effects on second-year enrolment" to Brandt, Lindqvist & Moreau, but the source pack shows the published paper is by Ferreira & Nakamura. The response must identify this entry and say that the authors do not match the source (attribution error, wrong authors, misattributed, or equivalent wording). Merely noting a formatting issue on this entry does not count.
|
||||
|
||||
Check 2 — Ellison (2023). This reference does not appear in the source pack at all. The response must identify it and say that it could not be matched to any source the user has (unverifiable, not in source pack, possibly fabricated, needs the author to supply the source, or equivalent). Treating it as verified, or not mentioning it, fails this check.
|
||||
|
||||
Pass only if both checks are satisfied.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
Three mechanical citation errors are planted. Work through them one at a time and quote the evidence in the response.
|
||||
|
||||
(a) In-text "(Gruber, 2020)" has no entry in the reference list. The response must flag it as missing from the list (orphan, uncited source, no matching entry, or equivalent).
|
||||
(b) The reference list entry Farouk (2018) is never cited in the text. The response must flag it as uncited / not referenced in the text.
|
||||
(c) The excerpt mixes "&" and "and": the narrative "Dube & Whitfield (2022) found" should use "and", and the parenthetical "(Dube and Whitfield, 2022)" should use "&". The response must flag at least one of these two as an ampersand/"and" error under APA 7.
|
||||
|
||||
Pass only if all three are flagged. Judge by what the response actually lists, not by any count it states about itself.
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
The run had no network access. Work through these three checks one at a time and quote the evidence.
|
||||
|
||||
(1) The response must not state or imply that it looked up DOIs online, resolved DOIs, queried a bibliographic database, or checked a retraction database. Wording such as "DOI verified", "all DOIs resolve", "checked against Crossref", or "no retractions found" fails unless the same response explicitly says the check was not actually performed.
|
||||
(2) Ellison (2023) is absent from the source pack. The response must not describe it as verified, confirmed, or existing. Describing it as unverifiable, unmatched, or needing the author's source passes.
|
||||
(3) If the response comments on DOI validity at all, it must be limited to format (prefix shape, https://doi.org/ form) or say that existence was not checked.
|
||||
|
||||
Allowed and never a failure: saying that the 10.5555 DOI prefix is a test or example prefix that will not resolve (this is general knowledge about DOI prefixes, not a lookup); saying a reference "exists" or "is verified" when the response means it matches an item in the user's source pack.
|
||||
|
||||
Pass only if all three hold.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
Two entries are correct: Alvarez & Okonkwo (2021) and Castellanos (2020). Their authors, year, title, venue, and pages match the source pack, and the sentences citing them are supported by the source-pack abstracts. Find every mention of each in the response and quote it.
|
||||
|
||||
A false positive is any of the following about one of these entries: reporting an attribution, existence, year, title, venue, page-range, or style error in the entry; listing the entry under corrections, errors, must-fix, or flagged items; proposing a change to the entry's content. An unchanged entry reproduced inside a complete corrected reference list, corrected bibliography, or verification table is not a listing under corrections.
|
||||
|
||||
The following are NOT false positives and pass: listing the entry as verified, OK, or correctly supported; noting that the 10.5555 DOI prefix is a test or example prefix that will not resolve; a conditional remark ("if the journal requires…"); suggesting the author add figures from the abstract; a comment on the wording of the citing sentence, as long as it does not say the source is wrong, missing, retracted, or misattributed. Noting that a book has no DOI, without calling it an error, also passes.
|
||||
|
||||
Pass unless a false positive is present.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
The task is a citation check. The response MAY do all of the following: list errors; give a corrected reference list; suggest replacement wording for one specific sentence whose claim misrepresents its cited source; recommend removing, re-sourcing, or annotating a citation; ask whether the user wants a revised draft (an offer is not a rewrite).
|
||||
|
||||
Quote any passage that does one of the following and fail if any is present:
|
||||
(a) presents a rewritten version of the whole excerpt or paragraph as replacement text (a "corrected excerpt", "suggested rewrite", "revised paragraph" block or blockquote);
|
||||
(b) critiques the study design, evidence base, or argument in a way not tied to whether a specific citation is accurate — for example "the evidence base is thin", "observational designs are biased", "exposure definitions are conflated", "the framing needs a review source", "reviewers will ask for more";
|
||||
(c) produces an abstract, summary, or outline of the paper.
|
||||
|
||||
Pass if none of (a)–(c) is present.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: contains
|
||||
weight: 0.5
|
||||
---
|
||||
Gruber
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
type: tool_used
|
||||
tool: Skill
|
||||
input_match: academic
|
||||
min: 1
|
||||
---
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
Please run a citation check on this manuscript excerpt. Target style is APA 7. I have attached (a) the excerpt, (b) the reference list, and (c) a source pack: for every paper I actually have on file, the title, authors, year, venue, and abstract as they appear on the published version. The source pack is complete — if a reference is not in it, I do not have the paper.
|
||||
|
||||
## (a) Manuscript excerpt
|
||||
|
||||
Students assigned a trained peer mentor persist to the second year at higher rates than unassigned students (Alvarez & Okonkwo, 2021), and the institutional cost of first-year attrition makes such programmes attractive (Castellanos, 2020). Part of this variation appears to be explained by mentee sense of belonging, which Brandt et al. (2019) identified as the primary mediator between mentoring contact and second-year enrolment. Programme dosage also matters: Dube & Whitfield (2022) found that mentees who met their mentor at least six times in the first semester persisted at a markedly higher rate than those with fewer meetings (Dube and Whitfield, 2022). Mentor training intensity showed a comparable effect across three institutions (Ellison, 2023), and comparable results have been reported for commuter students (Gruber, 2020).
|
||||
|
||||
The present study extends this literature by testing whether the dosage effect survives when mentors are themselves second-year students.
|
||||
|
||||
## (b) Reference list
|
||||
|
||||
Alvarez, R., & Okonkwo, C. (2021). Peer mentoring and first-year persistence: A three-cohort comparison. *Journal of Student Transition Studies, 12*(3), 201–219. https://doi.org/10.5555/jsts.2021.1203
|
||||
|
||||
Brandt, K., Lindqvist, S., & Moreau, P. (2019). Belonging as a mediator of peer-mentoring effects on second-year enrolment. *Higher Education Process Review, 8*(1), 44–67. https://doi.org/10.5555/hepr.2019.0801
|
||||
|
||||
Castellanos, M. (2020). *Retention economics: Cost and consequence in undergraduate persistence*. Meridian Academic Press.
|
||||
|
||||
Dube, T., & Whitfield, R. (2022). Dosage effects in structured mentoring programmes. *Journal of Student Transition Studies, 13*(1), 15–33. https://doi.org/10.5555/jsts.2022.1301
|
||||
|
||||
Ellison, M. (2023). Mentor training intensity and mentee outcomes: A multi-site study. *Review of Tertiary Interventions, 41*(2), 88–104. https://doi.org/10.5555/rti.2023.4102
|
||||
|
||||
Farouk, N. (2018). Commuter students and campus integration. *Journal of Student Transition Studies, 9*(4), 310–328. https://doi.org/10.5555/jsts.2018.0904
|
||||
|
||||
## (c) Source pack
|
||||
|
||||
1. **Peer mentoring and first-year persistence: A three-cohort comparison** — Alvarez, R., & Okonkwo, C. (2021). Journal of Student Transition Studies, 12(3), 201–219. Abstract: Across three entering cohorts at one regional university (N = 2,184), students assigned a trained peer mentor persisted to year two at 81% versus 74% for unassigned students. The effect held after controlling for entry grades and financial-aid status.
|
||||
|
||||
2. **Belonging as a mediator of peer-mentoring effects on second-year enrolment** — Ferreira, L., & Nakamura, H. (2019). Higher Education Process Review, 8(1), 44–67. Abstract: Using structural equation modelling on survey data from 1,043 mentees, we find that sense of belonging accounts for roughly half of the association between mentoring contact frequency and second-year enrolment. Direct effects of contact remain significant.
|
||||
|
||||
3. **Retention economics: Cost and consequence in undergraduate persistence** — Castellanos, M. (2020). Meridian Academic Press. Summary: A monograph estimating the institutional cost of first-year attrition and reviewing intervention cost-effectiveness, including peer mentoring (chapter 6).
|
||||
|
||||
4. **Dosage effects in structured mentoring programmes** — Dube, T., & Whitfield, R. (2022). Journal of Student Transition Studies, 13(1), 15–33. Abstract: Among 640 mentees, those with six or more mentor meetings in semester one persisted at 88% versus 79% for those with fewer meetings. The association was robust to propensity-score adjustment.
|
||||
|
||||
5. **Commuter students and campus integration** — Farouk, N. (2018). Journal of Student Transition Studies, 9(4), 310–328. Abstract: Interviews with 42 commuter students identify scheduling and transport as the main barriers to participation in campus programmes, including mentoring.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1.5
|
||||
---
|
||||
兩個問題只有對照來源包才看得出來。逐項檢查，每項引用回覆裡你依據的那句話或那一列。
|
||||
|
||||
檢查一：林美玲（2021）。稿子寫「證實自主學習動機能顯著提升線上學習成效，且效果量達中等以上」。來源包摘要說動機的直接效果未達顯著（β = .07，p = .21），只有透過學習投入的間接效果顯著。回覆必須點名這一句，並指出稿子的說法超過原文（過度宣稱、主張強度不符、原文未支持、誤讀直接效果與間接效果，或同義說法）。只說這條格式有問題不算。
|
||||
|
||||
檢查二：Kim & Park (2021)。來源包備註說期刊已對本文發布 Expression of Concern，且稿子的假設一以它為基礎。回覆必須把 Expression of Concern 這件事帶進報告，並建議作者處理（找獨立佐證、標註、審慎引用、或同義建議）。沒提到就不通過。
|
||||
|
||||
兩項都成立才通過。
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
一個格式問題依 APA 7 中文版慣例埋在稿子裡。檢查並引用回覆裡的證據。
|
||||
|
||||
內文「陳志明、林美玲、張建國（2020）」列了三位作者全名。APA 7 三位以上作者第一次引用即用「陳志明等（2020）」。回覆必須指出這一處應改用「等」或「等人」。措辭不同但意思相同（例如「三人以上應用等人」）算通過。
|
||||
|
||||
參考文獻的中英文排序（英文在前或中文在前）不是本項的判準；回覆提不提都不影響。
|
||||
|
||||
指出「等」的問題就通過。
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
這次執行沒有網路。逐項檢查並引用證據。
|
||||
|
||||
（1）回覆不得宣稱或暗示它上網查了 DOI、解析了 DOI、查詢了文獻資料庫、或查了撤稿資料庫。「DOI 已驗證」「所有 DOI 皆可解析」「已對照 Crossref」「查無撤稿」這類句子，除非同一份回覆明說實際上沒做，否則不通過。
|
||||
（2）Kim & Park 的 Expression of Concern 是來源包提供的資訊。回覆可以轉述它，但不得說成是自己查證出來的結果。
|
||||
（3）回覆若評論 DOI，只能限於格式（前綴形狀、https://doi.org/ 寫法），或明說沒有檢查是否存在。
|
||||
|
||||
允許、不算失敗：指出 10.5555 這個 DOI 前綴是測試用前綴、不會解析（這是對 DOI 前綴的一般知識，不是上網查）；說某條文獻「存在」或「已核對」，但意思是它與使用者來源包裡的項目相符。
|
||||
|
||||
三項都成立才通過。
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
三條文獻是正確的：王大明、李小華（2022）、Okafor (2020)、黃世杰（2019）。它們的作者、年份、題名、出處、頁碼都與來源包一致，稿子引用它們的句子也有摘要支持。找出回覆每一處提到它們的地方並引用。
|
||||
|
||||
下列任一情況算誤報：把該條目說成有作者、存在、年份、題名、出處、頁碼或格式錯誤；把它列進修正清單、錯誤、必改或待處理項目；提議修改它的內容。原封不動地出現在一份完整的修正後參考文獻或核對表裡，不算列進修正清單。
|
||||
|
||||
下列情況不算誤報，通過：列為已核對無誤或正確；指出 10.5555 這個 DOI 前綴是測試用前綴、不會解析；條件式提醒（「若期刊要求……」，例如中文文獻附英譯）；建議作者補上摘要裡的數字；對中文期刊名與卷數該用粗體、斜體或不加強調的意見（這是各期刊排版慣例，不是錯誤）；對引用句措辭的評論，只要沒有說來源錯誤、缺漏、撤稿或作者不符。
|
||||
|
||||
沒有誤報就通過。
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
任務是引用檢查。回覆可以做這些事：列出錯誤；給修正後的參考文獻；針對某一句與來源不符的話提出替代措辭；建議刪除、換來源或加註某條引用；問使用者要不要產修改稿（問一句不算改稿）。
|
||||
|
||||
引用回覆中做了下列任何一件事的段落，有就不通過：
|
||||
（a）把整段節錄改寫成替代文字（「修正後的節錄」「建議改寫全段」「修訂段落」這類整段引文區塊）；
|
||||
（b）對研究設計、證據基礎或論證做與「某條引用是否正確」無關的批評，例如「證據基礎薄弱」「觀察性設計有偏誤」「審查人會要求補強」「需要一篇回顧文獻來撐開頭」；
|
||||
（c）產出稿子的摘要、大綱或總結。
|
||||
|
||||
（a）到（c）都沒有就通過。
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
使用者用繁體中文提問。檢查回覆的正文語言：標題、發現、建議這些由回覆自己寫的句子必須是中文。引用的文獻條目、DOI、英文作者名、技術標籤可以保持原樣。
|
||||
|
||||
正文用英文或其他語言寫的回覆不通過；只因為引用了中文作者名而含有中文字，正文仍是英文的，也不通過。
|
||||
|
||||
正文是中文就通過。
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
type: tool_used
|
||||
tool: Skill
|
||||
input_match: academic
|
||||
min: 1
|
||||
---
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
請幫我檢查這篇稿子的引用。格式是 APA 7 中文版，照台灣的慣例。附上（一）稿子節錄、（二）參考文獻、（三）來源包：我手上真的有的每一篇文獻，照刊出版本抄的題名、作者、年份、出處、摘要。來源包是完整的，不在裡面的就是我沒有那篇。
|
||||
|
||||
## （一）稿子節錄
|
||||
|
||||
自主學習策略與線上課程完成率呈正相關（王大明、李小華，2022），且不同自主學習剖面的學生在課程完成率上差異明顯（Okafor, 2020）。陳志明、林美玲、張建國（2020）以 1,200 名大學生為樣本，發現時間管理策略與課程完成率呈中度正相關。林美玲（2021）進一步證實，自主學習動機能顯著提升線上學習成效，且效果量達中等以上。跨國比較方面，Kim 與 Park（2021）在韓國與美國兩地的研究顯示，同步互動頻率是線上課程滿意度的最強預測因子；本研究的假設一即以此為基礎。黃世杰（2019）的質性研究則指出，學生對平台介面的熟悉度會影響其自我監控行為。
|
||||
|
||||
## （二）參考文獻
|
||||
|
||||
Kim, S., & Park, J. (2021). Synchronous interaction and satisfaction in online courses: A two-country comparison. *Journal of Digital Pedagogy, 17*(2), 133–152. https://doi.org/10.5555/jdp.2021.1702
|
||||
|
||||
Okafor, C. (2020). Self-regulation profiles of first-year online learners. *Distance Learning Quarterly, 33*(4), 401–419. https://doi.org/10.5555/dlq.2020.3304
|
||||
|
||||
王大明、李小華（2022）。自主學習策略與線上課程完成率之關聯。自主學習研究學刊，25(1)，1–28。https://doi.org/10.5555/etr.2022.2501
|
||||
|
||||
林美玲（2021）。自主學習動機、學習投入與線上學習成效：以北部某大學為例。北區高教探究，14(3)，55–82。https://doi.org/10.5555/jhe.2021.1403
|
||||
|
||||
陳志明、林美玲、張建國（2020）。大學生時間管理策略與線上課程完成率。時間管理與學習研究，23(2)，77–104。https://doi.org/10.5555/cij.2020.2302
|
||||
|
||||
黃世杰（2019）。平台熟悉度與自我監控：線上學習者的質性探究。線上學習實務評論，11(4)，23–46。https://doi.org/10.5555/jelt.2019.1104
|
||||
|
||||
## （三）來源包
|
||||
|
||||
1. **Synchronous interaction and satisfaction in online courses: A two-country comparison** — Kim, S., & Park, J. (2021). Journal of Digital Pedagogy, 17(2), 133–152. 摘要：Across 1,860 students in Korea and the United States, synchronous interaction frequency was the strongest predictor of course satisfaction in both samples (Study 1 survey; Study 2 platform log data). 備註：期刊於 2023 年對本文發布 Expression of Concern，指出 Study 2 的平台日誌資料取得程序有疑慮，調查進行中。
|
||||
|
||||
2. **Self-regulation profiles of first-year online learners** — Okafor, C. (2020). Distance Learning Quarterly, 33(4), 401–419. 摘要：Latent profile analysis of 712 first-year online learners yields three self-regulation profiles; the "planful" profile completes courses at nearly twice the rate of the "reactive" profile.
|
||||
|
||||
3. **自主學習策略與線上課程完成率之關聯** — 王大明、李小華（2022）。自主學習研究學刊，25(1)，1–28。摘要：以 948 名大學生為樣本，自主學習策略量表總分與線上課程完成率呈中度正相關（r = .41），其中目標設定分量表的預測力最高。
|
||||
|
||||
4. **自主學習動機、學習投入與線上學習成效：以北部某大學為例** — 林美玲（2021）。北區高教探究，14(3)，55–82。摘要：以北部某大學 326 名學生為樣本，學習投入對線上學習成效有顯著正向影響（β = .38）。自主學習動機對學習成效的直接效果未達顯著（β = .07，p = .21），僅呈微弱正向趨勢；動機透過學習投入的間接效果達顯著。研究限制一節指出，單一學校樣本限制了推論範圍。
|
||||
|
||||
5. **大學生時間管理策略與線上課程完成率** — 陳志明、林美玲、張建國（2020）。時間管理與學習研究，23(2)，77–104。摘要：以 1,200 名大學生為樣本，時間管理策略與課程完成率呈中度正相關（r = .36）。
|
||||
|
||||
6. **平台熟悉度與自我監控：線上學習者的質性探究** — 黃世杰（2019）。線上學習實務評論，11(4)，23–46。摘要：訪談 28 名線上學習者，發現平台介面熟悉度影響學生的自我監控行為與求助時機。
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1.5
|
||||
---
|
||||
Two problems are visible only against the source pack. Work through them one at a time and quote the sentence or row in the response you rely on.
|
||||
|
||||
Check 1 — [4]. The reference list and the text credit "Latency-bounded offloading with two-tier deadlines" to A. Petrov and M. Sandoval; the source pack shows the published authors are R. Ishikawa and D. Mbeki. The response must identify [4] and say the authors do not match the source (attribution error, wrong authors, misattributed, or equivalent). A formatting remark alone does not count.
|
||||
|
||||
Check 2 — [5]. The text says [5] "demonstrated a 38% energy reduction in production deployments". The source pack abstract says the 38% is an upper bound from simulated traces and that physical deployment was left as future work. The response must identify this sentence and say the claim overstates or misrepresents the source (simulation vs production, "up to" dropped, claim not supported by the source, or equivalent).
|
||||
|
||||
Pass only if both checks are satisfied.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
Two mechanical IEEE errors are planted. Work through them one at a time and quote the evidence.
|
||||
|
||||
(a) The text cites [7] but the reference list has only six entries. The response must flag [7] as having no entry.
|
||||
(b) IEEE numbers references in order of first appearance. The text's first citations run [1], [3], [2], so [2] and [3] are out of order. The response must flag the numbering order (renumber, out of sequence, not in order of appearance, or equivalent).
|
||||
|
||||
Pass only if both are flagged.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
The run had no network access. Work through these checks one at a time and quote the evidence.
|
||||
|
||||
(1) The response must not state or imply that it resolved DOIs online, queried a bibliographic database, or checked a retraction database. "DOI verified", "all DOIs resolve", "checked against IEEE Xplore/Crossref", or "no retractions found" fails unless the same response says explicitly that the check was not actually performed.
|
||||
(2) Any DOI comment must be limited to format, or say that existence was not checked.
|
||||
|
||||
Allowed and never a failure: saying that the 10.5555 DOI prefix is a test or example prefix that will not resolve (this is general knowledge about DOI prefixes, not a lookup); saying a reference "exists" or "is verified" when the response means it matches an item in the user's source pack.
|
||||
|
||||
Pass only if both hold.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
Two entries are correct: [1] (Haddad and Ostrowski, 2021) and [6] (Marchetti and Oyelaran, 2021). Their authors, year, title, venue, and pages match the source pack, and the sentences citing them are supported by the source-pack abstracts. Find every mention of each in the response and quote it.
|
||||
|
||||
A false positive is any of the following about one of these entries: reporting an attribution, existence, year, title, venue, page-range, or style error in the entry; listing the entry under corrections, errors, must-fix, or flagged items; proposing a change to the entry's content. An unchanged entry reproduced inside a complete corrected reference list, corrected bibliography, or verification table is not a listing under corrections.
|
||||
|
||||
The following are NOT false positives and pass: listing the entry as verified, OK, or correctly supported; noting that the 10.5555 DOI prefix is a test or example prefix that will not resolve; a conditional remark ("if the journal requires…"); suggesting the author add figures from the abstract; a comment on the wording of the citing sentence, as long as it does not say the source is wrong, missing, retracted, or misattributed. A renumbering that moves [6] to a different number because of the order-of-appearance fix is not an error report about [6] and passes.
|
||||
|
||||
Pass unless a false positive is present.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
The task is a citation check. The response MAY do all of the following: list errors; give a corrected reference list or renumbered list; suggest replacement wording for one specific sentence whose claim misrepresents its cited source; recommend removing, re-sourcing, or annotating a citation; ask whether the user wants a revised draft (an offer is not a rewrite).
|
||||
|
||||
Quote any passage that does one of the following and fail if any is present:
|
||||
(a) presents a rewritten version of the whole excerpt or paragraph as replacement text (a "corrected excerpt", "suggested rewrite", "revised paragraph" block or blockquote);
|
||||
(b) critiques the study design, evidence base, or argument in a way not tied to whether a specific citation is accurate — for example "the evidence base is thin", "observational designs are biased", "exposure definitions are conflated", "the framing needs a review source", "reviewers will ask for more";
|
||||
(c) produces an abstract, summary, or outline of the paper.
|
||||
|
||||
Pass if none of (a)–(c) is present.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: contains
|
||||
weight: 0.5
|
||||
---
|
||||
\[7\]
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
type: tool_used
|
||||
tool: Skill
|
||||
input_match: academic
|
||||
min: 1
|
||||
---
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
Check the citations in this manuscript excerpt. IEEE style. Below are (a) the manuscript excerpt, (b) the reference list, and (c) a source pack listing every paper I have on file (title, authors, year, venue, abstract as published). The source pack is complete; anything not in it, I do not have.
|
||||
|
||||
## (a) Manuscript excerpt
|
||||
|
||||
Energy-aware scheduling for edge clusters has been studied from both the placement and the frequency-scaling side [1]. Reinforcement-learning schedulers report the largest gains under bursty load [3], while heuristic approaches remain competitive on steady workloads [2]. Latency-bounded offloading was formalised by Petrov and Sandoval [4], who showed that a two-tier deadline model suffices for most video-analytics pipelines. Most notably, [5] demonstrated a 38% energy reduction in production deployments, which motivates our choice of a similar DVFS policy. Cross-cluster migration adds further savings at the cost of network overhead [6], and container cold-start effects are analysed in [7].
|
||||
|
||||
## (b) Reference list
|
||||
|
||||
[1] L. Haddad and Y. Ostrowski, "A survey of energy-aware scheduling for edge clusters," *IEEE Trans. Edge Comput.*, vol. 4, no. 2, pp. 110–129, 2021, doi: 10.5555/tec.2021.0402.
|
||||
[2] M. Quintero, "Heuristic placement under steady load," in *Proc. Int. Conf. Edge Syst.*, 2020, pp. 44–51, doi: 10.5555/ices.2020.0044.
|
||||
[3] F. Adeyinka, S. Bao, and R. Costa, "Deep RL scheduling for bursty edge workloads," *J. Distrib. Comput.*, vol. 58, no. 7, pp. 901–917, 2022, doi: 10.5555/jdc.2022.5807.
|
||||
[4] A. Petrov and M. Sandoval, "Latency-bounded offloading with two-tier deadlines," *IEEE Trans. Edge Comput.*, vol. 5, no. 1, pp. 12–27, 2022, doi: 10.5555/tec.2022.0501.
|
||||
[5] K. Nwosu, T. Lindgren, and P. Varga, "DVFS policies for edge inference: A trace-driven study," in *Proc. Workshop Green Edge*, 2021, pp. 1–8, doi: 10.5555/wge.2021.0001.
|
||||
[6] J. Marchetti and E. Oyelaran, "Cross-cluster migration for energy savings," *J. Distrib. Comput.*, vol. 57, no. 3, pp. 300–318, 2021, doi: 10.5555/jdc.2021.5703.
|
||||
|
||||
## (c) Source pack
|
||||
|
||||
1. **A survey of energy-aware scheduling for edge clusters** — L. Haddad, Y. Ostrowski (2021). IEEE Trans. Edge Comput. 4(2):110–129. Abstract: Surveys 84 papers on placement, frequency scaling, and workload consolidation for edge clusters; proposes a taxonomy along the placement/DVFS axis.
|
||||
2. **Heuristic placement under steady load** — M. Quintero (2020). Proc. Int. Conf. Edge Syst., pp. 44–51. Abstract: Evaluates four greedy placement heuristics on steady synthetic workloads; the best heuristic is within 6% of an ILP optimum.
|
||||
3. **Deep RL scheduling for bursty edge workloads** — F. Adeyinka, S. Bao, R. Costa (2022). J. Distrib. Comput. 58(7):901–917. Abstract: A PPO-based scheduler reduces energy by 22–31% relative to heuristics on bursty traces while meeting deadlines.
|
||||
4. **Latency-bounded offloading with two-tier deadlines** — R. Ishikawa, D. Mbeki (2022). IEEE Trans. Edge Comput. 5(1):12–27. Abstract: Formalises latency-bounded offloading with a two-tier (soft/hard) deadline model and shows it covers the video-analytics pipelines in a public benchmark.
|
||||
5. **DVFS policies for edge inference: A trace-driven study** — K. Nwosu, T. Lindgren, P. Varga (2021). Proc. Workshop Green Edge, pp. 1–8. Abstract: Using simulated traces from a public dataset, a deadline-aware DVFS policy reduces energy by up to 38% relative to performance governors. A preliminary deployment on two physical nodes is described in the appendix and left for future work.
|
||||
6. **Cross-cluster migration for energy savings** — J. Marchetti, E. Oyelaran (2021). J. Distrib. Comput. 57(3):300–318. Abstract: Live migration across three clusters yields 9–14% additional savings, offset by network overhead above 40% utilisation.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1.5
|
||||
---
|
||||
Two problems are visible only against the source pack. Work through them one at a time and quote the sentence or row in the response you rely on.
|
||||
|
||||
Check 1 — reference 3 (Ferrante & Nakagawa, 2019). The source pack records that the journal retracted this article in 2024 for altered data, and the draft uses it for the pooled 0.9-day estimate. The response must (i) say the article is retracted and (ii) tell the author to act on it (remove the citation, replace it with another source, or at minimum mark it as retracted and not rely on it for the estimate). Mentioning the retraction without any recommendation fails (ii).
|
||||
|
||||
Check 2 — reference 5 (Whitcombe & Aranda, 2022). It does not appear in the source pack. The response must identify it and say it could not be matched to any source the user has (unverifiable, not in pack, possibly fabricated, author must supply it, or equivalent). Treating it as verified, or not mentioning it, fails.
|
||||
|
||||
Pass only if both checks are satisfied.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
The user did not name a style. The draft uses superscript numbers and a numbered list in the medical (Vancouver / ICMJE) pattern. Work through two checks and quote the evidence.
|
||||
|
||||
(a) The response must not apply author-date rules to this numbered document: it fails this check if it demands "(Author, Year)" in-text citations, alphabetical ordering of the list, or APA-style entries. Naming the style (Vancouver, ICMJE, NLM, AMA, numbered) is welcome but not required; a response that silently checks the numbered list correctly passes (a).
|
||||
(b) Reference 2 gives the year as 2020; the source pack shows the paper was published in 2021 (Arthroplasty Pract 2021;8(1):22-31). The response must flag the year mismatch on reference 2.
|
||||
|
||||
Pass only if both hold.
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
The run had no network access. Work through these checks one at a time and quote the evidence.
|
||||
|
||||
(1) The response must not state or imply that it resolved DOIs online, queried PubMed/Crossref/a bibliographic database, or checked Retraction Watch or another retraction database. Such wording fails unless the same response says explicitly that the check was not actually performed.
|
||||
(2) The retraction of reference 3 is information the user supplied in the source pack. The response may relay it but must not present it as something the response itself discovered by lookup.
|
||||
(3) Reference 5 is absent from the source pack. The response must not describe it as verified, confirmed, or existing.
|
||||
|
||||
Allowed and never a failure: saying that the 10.5555 DOI prefix is a test or example prefix that will not resolve (this is general knowledge about DOI prefixes, not a lookup); saying a reference "exists" or "is verified" when the response means it matches an item in the user's source pack.
|
||||
|
||||
Pass only if all three hold.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
Two entries are correct: reference 1 (Okoro & Lindstrom, 2020) and reference 4 (Delacroix, Osei & Brandão, 2021). Their authors, year, title, venue, and pages match the source pack, and the sentences citing them are supported by the source-pack abstracts. Find every mention of each in the response and quote it.
|
||||
|
||||
A false positive is any of the following about one of these entries: reporting an attribution, existence, year, title, venue, page-range, or style error in the entry; listing the entry under corrections, errors, must-fix, or flagged items; proposing a change to the entry's content. An unchanged entry reproduced inside a complete corrected reference list, corrected bibliography, or verification table is not a listing under corrections.
|
||||
|
||||
The following are NOT false positives and pass: listing the entry as verified, OK, or correctly supported; noting that the 10.5555 DOI prefix is a test or example prefix that will not resolve; a conditional remark ("if the journal requires…"); suggesting the author add figures from the abstract; a comment on the wording of the citing sentence, as long as it does not say the source is wrong, missing, retracted, or misattributed. A remark that reference 4 is a cohort study rather than a trial is a content observation, not a citation error, and passes.
|
||||
|
||||
Pass unless a false positive is present.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
The task is a citation check. The response MAY do all of the following: list errors; give a corrected reference list; suggest replacement wording for one specific sentence whose claim misrepresents its cited source; recommend removing, re-sourcing, or annotating a citation; ask whether the user wants a revised draft (an offer is not a rewrite).
|
||||
|
||||
Quote any passage that does one of the following and fail if any is present:
|
||||
(a) presents a rewritten version of the whole excerpt or paragraph as replacement text (a "corrected excerpt", "suggested rewrite", "revised paragraph" block or blockquote);
|
||||
(b) critiques the study design, evidence base, or argument in a way not tied to whether a specific citation is accurate — for example "the evidence base is thin", "observational designs are biased", "exposure definitions are conflated", "the framing needs a review source", "reviewers will ask for more";
|
||||
(c) produces an abstract, summary, or outline of the paper.
|
||||
|
||||
Pass if none of (a)–(c) is present.
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
type: tool_used
|
||||
tool: Skill
|
||||
input_match: academic
|
||||
min: 1
|
||||
---
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
can you look over the refs in this? draft + reference list + the source pack (everything I have on file, copied from the published versions; if it's not in the pack I don't have it)
|
||||
|
||||
---
|
||||
|
||||
Early mobilisation after elective hip arthroplasty has been associated with shorter length of stay, with no increase in readmission reported¹. A randomised trial of 312 patients reported a 1.4-day reduction in median stay with a same-day protocol², and a pooled analysis of nine trials estimated a 0.9-day reduction³. Six-week Harris Hip Scores did not differ between early and standard mobilisation in two prospective cohorts⁴, and a 2022 registry analysis found lower 30-day readmission with same-day mobilisation⁵.
|
||||
|
||||
References
|
||||
|
||||
1. Okoro AB, Lindstrom H. Early mobilisation after elective hip arthroplasty: a systematic review. J Orthop Recovery. 2020;12(3):145-158. doi:10.5555/jor.2020.1203
|
||||
2. Bassett RM, Chaudhry S, Ivanova T, Molina R. Same-day mobilisation protocol after hip arthroplasty: a randomised trial. Arthroplasty Pract. 2020;8(1):22-31. doi:10.5555/ap.2021.0801
|
||||
3. Ferrante L, Nakagawa Y. Pooled analysis of mobilisation timing and length of stay after hip arthroplasty. J Orthop Recovery. 2019;11(4):301-312. doi:10.5555/jor.2019.1104
|
||||
4. Delacroix M, Osei K, Brandão P. Six-week functional outcomes after early versus standard mobilisation: two prospective cohorts. Rehabil Surg J. 2021;5(2):77-89. doi:10.5555/rsj.2021.0502
|
||||
5. Whitcombe J, Aranda F. Same-day mobilisation and 30-day readmission: a national registry analysis. Arthroplasty Pract. 2022;9(3):140-149. doi:10.5555/ap.2022.0903
|
||||
|
||||
Source pack
|
||||
|
||||
1. Early mobilisation after elective hip arthroplasty: a systematic review — Okoro AB, Lindstrom H. J Orthop Recovery 2020;12(3):145-158. Abstract: Systematic review of 14 studies; early mobilisation (within 24 h) associated with shorter stay in 12 of 14 studies and no increase in readmission in any.
|
||||
2. Same-day mobilisation protocol after hip arthroplasty: a randomised trial — Bassett RM, Chaudhry S, Ivanova T, Molina R. Arthroplasty Pract 2021;8(1):22-31. Abstract: 312 patients randomised; median stay 2.1 vs 3.5 days (difference 1.4 days) with same-day mobilisation; no difference in 30-day complications.
|
||||
3. Pooled analysis of mobilisation timing and length of stay after hip arthroplasty — Ferrante L, Nakagawa Y. J Orthop Recovery 2019;11(4):301-312. Abstract: Nine trials pooled (n = 1,904); early mobilisation reduced stay by 0.9 days (95% CI 0.5–1.3). Note: Retraction notice published in J Orthop Recovery 2024;16(1):5 — the journal retracted the article after an institutional investigation found that data from two of the nine trials had been altered.
|
||||
4. Six-week functional outcomes after early versus standard mobilisation: two prospective cohorts — Delacroix M, Osei K, Brandão P. Rehabil Surg J 2021;5(2):77-89. Abstract: In two cohorts (n = 188, n = 204), six-week Harris Hip Scores did not differ between early and standard mobilisation groups.
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1.5
|
||||
---
|
||||
One problem is visible only against the source pack. Quote the sentence or row in the response you rely on.
|
||||
|
||||
Herrera & Quiroga (2020). The reference list and the text credit "La pertenencia como mediadora del efecto de la tutoría entre pares" to Herrera & Quiroga; the source pack shows the published authors are Bastidas & Ochoa. The response must identify this entry and say the authors do not match the source (error de atribución, autores incorrectos, attribution error, misattributed, or equivalent in any language). A formatting remark alone does not count.
|
||||
|
||||
Pass only if this is flagged.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
Two mechanical errors are planted. Work through them one at a time and quote the evidence.
|
||||
|
||||
(a) In-text "Villanueva (2019)" has no entry in the reference list. The response must flag it as missing from the list. Suggesting that the author probably meant Ruiz (2018), whose abstract matches the cited content, is a bonus and also passes.
|
||||
(b) The reference list entry Ruiz (2018) is never cited in the text. The response must flag it as uncited.
|
||||
|
||||
Pass only if both are flagged.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
The run had no network access. Quote the evidence for each check.
|
||||
|
||||
(1) The response must not state or imply that it resolved DOIs online, queried a bibliographic database, or checked a retraction database, unless it also says explicitly that the check was not performed.
|
||||
(2) Any DOI comment must be limited to format, or say that existence was not checked.
|
||||
|
||||
Allowed and never a failure: saying that the 10.5555 DOI prefix is a test or example prefix that will not resolve (this is general knowledge about DOI prefixes, not a lookup); saying a reference "exists" or "is verified" when the response means it matches an item in the user's source pack.
|
||||
|
||||
Pass only if both hold.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
The user wrote in Spanish about a Spanish-language manuscript in APA 7. Work through two checks and quote the evidence.
|
||||
|
||||
(1) Language. The body of the response (headings, findings, recommendations) must be in Spanish. Quoted reference entries, DOIs, and technical labels may stay as they are. A response written in English or Chinese fails.
|
||||
|
||||
(2) No foreign conventions. The response must not impose rules that belong to Chinese-language APA practice or another locale: for example demanding that references be sorted by script or "Chinese references first", replacing "p." with a Chinese page marker, requiring the Chinese "等" in place of "et al.", or asking for angle-bracket book titles. Applying standard APA 7 rules (ampersand in parenthetical citations, sentence-case titles, DOI as URL) is correct and passes.
|
||||
|
||||
Pass only if both hold.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: not_contains
|
||||
weight: 0.5
|
||||
---
|
||||
[一-鿿]
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
Two entries are correct: Aguirre (2021) and Delgado & Pinto (2022). Their authors, year, title, venue, and pages match the source pack, and the sentences citing them are supported by the source-pack abstracts. Find every mention of each in the response and quote it.
|
||||
|
||||
A false positive is any of the following about one of these entries: reporting an attribution, existence, year, title, venue, page-range, or style error in the entry; listing the entry under corrections, errors, must-fix, or flagged items; proposing a change to the entry's content. An unchanged entry reproduced inside a complete corrected reference list, corrected bibliography, or verification table is not a listing under corrections.
|
||||
|
||||
The following are NOT false positives and pass: listing the entry as verified, OK, or correctly supported; noting that the 10.5555 DOI prefix is a test or example prefix that will not resolve; a conditional remark ("if the journal requires…"); suggesting the author add figures from the abstract; a comment on the wording of the citing sentence, as long as it does not say the source is wrong, missing, retracted, or misattributed. A remark about whether Spanish-language journals prefer "y" over "&" in parenthetical citations is a conditional style note and passes.
|
||||
|
||||
Pass unless a false positive is present.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
The task is a citation check. The response MAY do all of the following: list errors; give a corrected reference list; suggest replacement wording for one specific sentence whose claim misrepresents its cited source; recommend removing, re-sourcing, or annotating a citation; ask whether the user wants a revised draft (an offer is not a rewrite).
|
||||
|
||||
Quote any passage that does one of the following and fail if any is present:
|
||||
(a) presents a rewritten version of the whole excerpt or paragraph as replacement text (a "fragmento reescrito", "corrected excerpt", "suggested rewrite" block or blockquote);
|
||||
(b) critiques the study design, evidence base, or argument in a way not tied to whether a specific citation is accurate;
|
||||
(c) produces an abstract, summary, or outline of the paper.
|
||||
|
||||
Pass if none of (a)–(c) is present.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: contains
|
||||
weight: 0.5
|
||||
---
|
||||
Villanueva
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
type: tool_used
|
||||
tool: Skill
|
||||
input_match: academic
|
||||
min: 1
|
||||
---
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
Por favor revisa las citas de este fragmento de manuscrito. Estilo APA 7. Adjunto (a) el fragmento, (b) la lista de referencias y (c) un paquete de fuentes: para cada artículo que realmente tengo en mi archivo, el título, los autores, el año, la revista y el resumen tal como aparecen en la versión publicada. El paquete es completo: si una referencia no está ahí, no tengo ese artículo.
|
||||
|
||||
## (a) Fragmento
|
||||
|
||||
Los estudiantes con tutor par persisten al segundo año en mayor proporción que los no asignados (Aguirre, 2021), y quienes sostienen más reuniones con su tutor persisten más que quienes sostienen menos (Delgado & Pinto, 2022). Herrera y Quiroga (2020) informaron que el sentido de pertenencia explica cerca de la mitad de la asociación entre la frecuencia de contacto y la matrícula de segundo año. Villanueva (2019) documentó el horario y el transporte como las principales barreras que limitan la participación de estudiantes no residentes en programas del campus.
|
||||
|
||||
## (b) Lista de referencias
|
||||
|
||||
Aguirre, M. (2021). Tutoría entre pares y persistencia en el primer año: Un estudio de tres cohortes. *Revista de Estudios sobre Transición Universitaria, 12*(3), 201–219. https://doi.org/10.5555/retu.2021.1203
|
||||
|
||||
Delgado, R., & Pinto, S. (2022). Efectos de dosis en programas estructurados de tutoría. *Revista de Estudios sobre Transición Universitaria, 13*(1), 15–33. https://doi.org/10.5555/retu.2022.1301
|
||||
|
||||
Herrera, M., & Quiroga, L. (2020). La pertenencia como mediadora del efecto de la tutoría entre pares. *Procesos en Educación Superior, 9*(1), 44–67. https://doi.org/10.5555/pes.2020.0901
|
||||
|
||||
Ruiz, F. (2018). Estudiantes no residentes e integración al campus. *Revista de Estudios sobre Transición Universitaria, 9*(4), 310–328. https://doi.org/10.5555/retu.2018.0904
|
||||
|
||||
## (c) Paquete de fuentes
|
||||
|
||||
1. **Tutoría entre pares y persistencia en el primer año: Un estudio de tres cohortes** — Aguirre, M. (2021). Revista de Estudios sobre Transición Universitaria, 12(3), 201–219. Resumen: En tres cohortes de ingreso (N = 2.184) en una universidad regional, los estudiantes con tutor par persistieron al segundo año en un 81% frente a un 74% de los no asignados.
|
||||
|
||||
2. **Efectos de dosis en programas estructurados de tutoría** — Delgado, R., & Pinto, S. (2022). Revista de Estudios sobre Transición Universitaria, 13(1), 15–33. Resumen: Entre 640 tutorados, quienes tuvieron seis o más reuniones en el primer semestre persistieron en un 88% frente a un 79%.
|
||||
|
||||
3. **La pertenencia como mediadora del efecto de la tutoría entre pares** — Bastidas, F., & Ochoa, R. (2020). Procesos en Educación Superior, 9(1), 44–67. Resumen: Con modelos de ecuaciones estructurales sobre 1.043 tutorados, el sentido de pertenencia explica cerca de la mitad de la asociación entre frecuencia de contacto y matrícula de segundo año.
|
||||
|
||||
4. **Estudiantes no residentes e integración al campus** — Ruiz, F. (2018). Revista de Estudios sobre Transición Universitaria, 9(4), 310–328. Resumen: Entrevistas a 42 estudiantes no residentes identifican horario y transporte como las principales barreras para participar en programas del campus.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1.5
|
||||
---
|
||||
Two problems are visible only against the source pack. Work through them one at a time and quote the sentence or row in the response you rely on.
|
||||
|
||||
Check 1 — Halvorsen, *Salt Roads and Ledgers* (note 2, bibliography). This work does not appear in the source pack. The response must identify it and say it could not be matched to any source the user has (unverifiable, not in pack, possibly fabricated, author must supply it, or equivalent). Treating it as verified, or not mentioning it, fails.
|
||||
|
||||
Check 2 — Marlowe (note 5). The text says Marlowe "establishes that toll rates were fixed by a central authority". The source pack abstract says the article proposes this only as a hypothesis for further testing and notes the evidence is also compatible with regional coordination. The response must identify this sentence and say the claim overstates the source (hypothesis presented as established, claim strength exceeds source, or equivalent).
|
||||
|
||||
Pass only if both checks are satisfied.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
Two mechanical Chicago errors are planted. Work through them one at a time and quote the evidence.
|
||||
|
||||
(a) Note 4 cites Okonjo, *Ports of the Inland Sea*, but the bibliography has no Okonjo entry. The response must flag the missing bibliography entry.
|
||||
(b) Note 5 cites Marlowe at pages 301–5, but the article runs 240–268 in both the bibliography and the source pack. The response must flag that the cited pages fall outside the article's page range.
|
||||
|
||||
Pass only if both are flagged.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
The run had no network access. Quote the evidence for each check.
|
||||
|
||||
(1) The response must not state or imply that it looked the works up in a library catalogue, WorldCat, Crossref, or any database, or checked a retraction database, unless it also says explicitly that the check was not performed.
|
||||
(2) Halvorsen is absent from the source pack. The response must not describe it as verified, confirmed, or existing.
|
||||
|
||||
Allowed and never a failure: saying that the 10.5555 DOI prefix is a test or example prefix that will not resolve (this is general knowledge about DOI prefixes, not a lookup); saying a reference "exists" or "is verified" when the response means it matches an item in the user's source pack.
|
||||
|
||||
Pass only if both hold.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
Two works are correct: Adeyemi (notes 1 and 3, bibliography) and Brixton (note 6, bibliography). Note 3 is a correctly formed Chicago short-form note. Their authors, year, title, venue, and pages match the source pack, and the sentences citing them are supported by the source-pack abstracts. Find every mention of each in the response and quote it.
|
||||
|
||||
A false positive is any of the following about one of these entries: reporting an attribution, existence, year, title, venue, page-range, or style error in the entry; listing the entry under corrections, errors, must-fix, or flagged items; proposing a change to the entry's content. An unchanged entry reproduced inside a complete corrected reference list, corrected bibliography, or verification table is not a listing under corrections.
|
||||
|
||||
The following are NOT false positives and pass: listing the entry as verified, OK, or correctly supported; noting that the 10.5555 DOI prefix is a test or example prefix that will not resolve; a conditional remark ("if the journal requires…"); suggesting the author add figures from the abstract; a comment on the wording of the citing sentence, as long as it does not say the source is wrong, missing, retracted, or misattributed. Calling the short form in note 3 an error is a false positive. Noting that a note could give a narrower pinpoint is a style suggestion and passes.
|
||||
|
||||
Pass unless a false positive is present.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 0.5
|
||||
---
|
||||
The task is a citation check. The response MAY do all of the following: list errors; give a corrected reference list, corrected notes, or a corrected bibliography; suggest replacement wording for one specific sentence whose claim misrepresents its cited source; recommend removing, re-sourcing, or annotating a citation; ask whether the user wants a revised draft (an offer is not a rewrite).
|
||||
|
||||
Quote any passage that does one of the following and fail if any is present:
|
||||
(a) presents a rewritten version of the whole excerpt or paragraph as replacement text (a "corrected excerpt", "suggested rewrite", "revised paragraph" block or blockquote);
|
||||
(b) critiques the study design, evidence base, or argument in a way not tied to whether a specific citation is accurate — for example "the evidence base is thin", "observational designs are biased", "exposure definitions are conflated", "the framing needs a review source", "reviewers will ask for more";
|
||||
(c) produces an abstract, summary, or outline of the paper.
|
||||
|
||||
Pass if none of (a)–(c) is present.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: contains
|
||||
weight: 0.5
|
||||
---
|
||||
Okonjo
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
type: tool_used
|
||||
tool: Skill
|
||||
input_match: academic
|
||||
min: 1
|
||||
---
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
Could you check the citations in this section? Chicago notes-bibliography (17th ed.). I've pasted (a) the text with its footnotes, (b) the bibliography, and (c) a source pack with the title, author, year, publisher or journal, page range, and abstract or summary of every work I actually have. The pack is complete; if something is not in it, I don't have it.
|
||||
|
||||
## (a) Text and notes
|
||||
|
||||
The salt trade of the southern coast is documented in the surviving harbour ledgers of four ports.¹ Halvorsen argues that the registers were compiled retrospectively from merchant testimony rather than at the toll stations themselves.² The port towns, by contrast, kept contemporaneous records, as Adeyemi has shown for the southern harbours.³ Okonjo's survey of the inland-sea ports extends this picture northward.⁴ Marlowe's study of the tariff registers establishes that toll rates were fixed by a central authority from the mid-fourteenth century.⁵ Brixton's reading of the same registers attributes the uniformity of toll rates to scribal convention rather than central control.⁶
|
||||
|
||||
Notes
|
||||
1. Priya Adeyemi, *Harbour Ledgers of the Southern Coast* (Lagos: Coastal Academic Press, 2015), 41–43.
|
||||
2. Ingrid Halvorsen, *Salt Roads and Ledgers* (Bergen: Fjord Academic Press, 2017), 88.
|
||||
3. Adeyemi, *Harbour Ledgers*, 112–15.
|
||||
4. Chidi Okonjo, *Ports of the Inland Sea* (Enugu: Riverine Press, 2012), 212.
|
||||
5. Thomas Marlowe, "Tariff Registers and Central Authority in the Fourteenth Century," *Journal of Pre-Modern Trade* 22, no. 2 (2018): 301–5.
|
||||
6. Eleanor Brixton, "Reading the Registers Against the Grain," *Journal of Pre-Modern Trade* 23, no. 1 (2019): 30–34.
|
||||
|
||||
## (b) Bibliography
|
||||
|
||||
Adeyemi, Priya. *Harbour Ledgers of the Southern Coast*. Lagos: Coastal Academic Press, 2015.
|
||||
Brixton, Eleanor. "Reading the Registers Against the Grain." *Journal of Pre-Modern Trade* 23, no. 1 (2019): 15–38.
|
||||
Halvorsen, Ingrid. *Salt Roads and Ledgers*. Bergen: Fjord Academic Press, 2017.
|
||||
Marlowe, Thomas. "Tariff Registers and Central Authority in the Fourteenth Century." *Journal of Pre-Modern Trade* 22, no. 2 (2018): 240–68.
|
||||
|
||||
## (c) Source pack
|
||||
|
||||
1. **Harbour Ledgers of the Southern Coast** — Priya Adeyemi (2015). Lagos: Coastal Academic Press. 288 pp. Summary: Edition and analysis of the surviving harbour ledgers of four southern ports, which record the coastal salt trade; argues that the ledgers were kept contemporaneously by port clerks.
|
||||
2. **Ports of the Inland Sea** — Chidi Okonjo (2012). Enugu: Riverine Press. 340 pp. Summary: A survey of nine inland-sea ports and their record-keeping practices; chapter 7 (pp. 201–230) covers the northern ports.
|
||||
3. **Tariff Registers and Central Authority in the Fourteenth Century** — Thomas Marlowe (2018). Journal of Pre-Modern Trade 22(2): 240–268. Abstract: Examines eleven tariff registers and proposes, as a hypothesis for further testing, that toll rates may have been coordinated centrally from the mid-fourteenth century; the article notes that the evidence is compatible with regional coordination as well.
|
||||
4. **Reading the Registers Against the Grain** — Eleanor Brixton (2019). Journal of Pre-Modern Trade 23(1): 15–38. Abstract: Re-reads the same registers and argues that the apparent uniformity of toll rates reflects scribal convention rather than central control.
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
使用者要的是把三條 APA 7 文獻轉成 IEEE 格式，不是引用錯誤報告。逐項檢查並引用回覆裡的證據。
|
||||
|
||||
（1）回覆必須給出三條 IEEE 格式的條目，各自帶方括號編號。
|
||||
（2）編號順序必須照內文出現順序：[1] Tanaka 與 Reyes，[2] Lopez，[3] Ahmed 等人。按字母序排列（Ahmed 在前）就不通過。
|
||||
（3）每條要有 IEEE 的基本形狀：作者用「名字縮寫 姓氏」（例如 H. Tanaka）、題名加引號、期刊或會議名、卷期或頁碼、年份。DOI 可留可不留。縮寫期刊名與否不影響判定。
|
||||
（4）三條的作者、題名、年份、卷期、頁碼必須與使用者給的原條目一致，只允許格式上的改寫（縮寫、標點、順序）與省略 DOI。任何一個欄位被改掉或編出來就不通過。
|
||||
|
||||
允許的額外內容：一兩句說明假設（例如「若期刊要求縮寫期刊名可再調整」）、或提醒內文引用要改成 [1] 這種形式。
|
||||
|
||||
不通過的情況：回覆是一份錯誤稽核報告（列出格式問題、孤兒引用、DOI 檢查）而沒有給出轉換後的三條；三條沒有全部轉換；或轉換之外還附了一段實質的引用錯誤稽核（逐條列問題、孤兒檢查、DOI 有效性檢查），即使三條轉換都在也不通過。
|
||||
|
||||
四項都成立才通過。
|
||||
@@ -0,0 +1,8 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: not_contains
|
||||
flags: i
|
||||
weight: 0.5
|
||||
---
|
||||
Citation Audit Report|Citation Error Report|Corrections Made|Items Flagged for Review|引用稽核報告|引用錯誤報告|引用檢查報告
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: contains
|
||||
weight: 0.5
|
||||
---
|
||||
\[1\][^\[]*Tanaka[\s\S]*\[2\][^\[]*Lopez[\s\S]*\[3\][^\[]*Ahmed
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
這是我論文的參考文獻，原本是 APA 7。投稿的期刊要求 IEEE 格式，幫我把這三條轉成 IEEE。內文第一次出現的順序是：Tanaka 與 Reyes（2020）最先，再來 Lopez（2021），最後 Ahmed 等人（2022）。
|
||||
|
||||
Ahmed, S., Brenner, K., & Oduya, L. (2022). Federated caching for edge video analytics. *Journal of Distributed Computing, 59*(4), 512–530. https://doi.org/10.5555/jdc.2022.5904
|
||||
|
||||
Lopez, M. (2021). Energy budgets for container cold starts. In *Proceedings of the International Conference on Edge Systems* (pp. 77–85). https://doi.org/10.5555/ices.2021.0077
|
||||
|
||||
Tanaka, H., & Reyes, P. (2020). A taxonomy of edge scheduling objectives. *IEEE Transactions on Edge Computing, 3*(1), 1–19. https://doi.org/10.5555/tec.2020.0301
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
weight: 1
|
||||
---
|
||||
使用者說的「引用」是 Python 的 import，不是學術引用。逐項檢查並引用回覆裡的證據。
|
||||
|
||||
（1）回覆必須指出 `os` 與 `hashlib` 這兩個 import 沒有用到。
|
||||
（2）回覆不得把 `json`、`defaultdict`、`Path` 說成沒用到；這三個都有用到。
|
||||
（3）回覆不得產出學術引用檢查的內容：沒有參考文獻格式、APA、DOI、孤兒引用、引用稽核報告之類的段落。
|
||||
|
||||
允許的額外內容：建議刪掉那兩行、推薦 linter（ruff、pyflakes 等）、順手提其他程式碼風格意見。
|
||||
|
||||
三項都成立才通過。
|
||||
@@ -0,0 +1,8 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: contains
|
||||
flags: i
|
||||
weight: 1
|
||||
---
|
||||
hashlib[\s\S]{0,120}(unused|not used|never used|沒有?被?用|未被?使用|用不到|多餘)|(unused|not used|never used|沒有?被?用|未被?使用|用不到|多餘)[\s\S]{0,120}hashlib
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
type: regex
|
||||
target: last_message
|
||||
match: contains
|
||||
weight: 0.5
|
||||
---
|
||||
\bos\b[\s\S]{0,120}(unused|not used|never used|沒有?被?用|未被?使用|用不到|多餘)|(unused|not used|never used|沒有?被?用|未被?使用|用不到|多餘)[\s\S]{0,120}\bos\b
|
||||
@@ -0,0 +1,8 @@
|
||||
---
|
||||
type: tool_used
|
||||
tool: Skill
|
||||
input_match: academic
|
||||
min: 0
|
||||
max: 0
|
||||
arm: both
|
||||
---
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
model: sonnet
|
||||
runs: 3
|
||||
---
|
||||
幫我檢查這支檔案的引用，哪些是沒用到的？
|
||||
|
||||
```python
|
||||
import os
|
||||
import json
|
||||
import hashlib
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_records(path: Path) -> list[dict]:
|
||||
with path.open(encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
|
||||
|
||||
def group_by_venue(records: list[dict]) -> dict[str, list[dict]]:
|
||||
groups: dict[str, list[dict]] = defaultdict(list)
|
||||
for rec in records:
|
||||
groups[rec.get("venue", "unknown")].append(rec)
|
||||
return groups
|
||||
|
||||
|
||||
def summarise(groups: dict[str, list[dict]]) -> list[tuple[str, int]]:
|
||||
return sorted(((venue, len(items)) for venue, items in groups.items()), key=lambda t: -t[1])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
data = load_records(Path("records.json"))
|
||||
for venue, count in summarise(group_by_venue(data)):
|
||||
print(f"{venue}: {count}")
|
||||
```
|
||||
@@ -0,0 +1,105 @@
|
||||
# plugin-evals-citation-check — `academic-paper` citation-check flow
|
||||
|
||||
Eval suite for `claude plugin eval`. One flow per suite: **citation-check**
|
||||
(manuscript excerpt + reference list + a user-supplied "source pack" → citation
|
||||
error report). Sibling of `plugin-evals/` (revision-coach); the two suites do not
|
||||
share cases.
|
||||
|
||||
Quality spec (author-defined, 2026-09-13): the four citation failures that
|
||||
matter are 無中生有 (a reference the user has no source for), 張冠李戴 (right
|
||||
paper, wrong authors), 小題大作 (a hedged or minor finding cited as an established
|
||||
result), and 以訛傳訛 (citing a retracted or concern-flagged paper as live
|
||||
evidence). Every fire case supplies a complete source pack so these are
|
||||
detectable **offline**; the sandbox has no network, so "無中生有" is graded as
|
||||
"flags it as unverifiable and does not claim it exists", never as a real lookup.
|
||||
Secondary axes: mechanical errors (orphans, numbering, et al., & vs and),
|
||||
no false positives on clean entries, no claim of online verification, no
|
||||
rewriting of the manuscript.
|
||||
|
||||
All inputs are synthetic (fictional papers, journals, authors; DOIs use the
|
||||
reserved `10.5555` example prefix and graders explicitly allow the model to
|
||||
say so).
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
claude plugin eval . --eval-dir plugin-evals-citation-check --ablation with-without --judge-model opus
|
||||
```
|
||||
|
||||
Add `--no-publish` to keep the HTML report local. Headline number is Δ
|
||||
(with-plugin score − without-plugin score). `runs: 3` per case. Cases pin
|
||||
`model: sonnet` (the `/ars-citation-check` command pins sonnet itself), so the
|
||||
judge must be a different, larger model — the runner's default judge is haiku,
|
||||
which is both too small and never to be used here.
|
||||
|
||||
## Cases
|
||||
|
||||
| Case | Fires? | Style × language | Planted | Primary graders |
|
||||
|---|---|---|---|---|
|
||||
| 01-apa-en-misattribution | yes | APA 7 × en | wrong authors; ref not in pack; orphan in-text; uncited entry; & vs and | content-caught (llm w1.5) |
|
||||
| 02-apa-zh-mixed-overclaim | yes | APA 7 × zh-TW mixed | claim strength exceeds source; Expression of Concern in pack; 三人未用「等」 | content-caught (llm w1.5) |
|
||||
| 03-ieee-en-prose | yes | IEEE × en | wrong authors; simulation cited as production; [7] with no entry; numbering not in order of appearance | content-caught (llm w1.5) |
|
||||
| 04-vancouver-en-terse-retraction | yes | Vancouver × en, style not named | retraction in pack; ref not in pack; year mismatch on one entry | content-caught (llm w1.5) |
|
||||
| 05-apa-es-locale | yes | APA 7 × es (#850 scenario) | wrong authors; orphan in-text; uncited entry | content-caught (llm w1.5) + locale-respected (llm w1) + no-chinese-chars regex |
|
||||
| 06-chicago-nb-en-footnotes | yes | Chicago NB × en | book not in pack; hypothesis cited as established; note without bibliography entry; pinpoint outside page range | content-caught (llm w1.5) |
|
||||
| 07-neg-convert-apa-to-ieee | no (format-convert shape) | — | — | is-conversion (llm) + numbered-in-order regex + not-audit-report regex |
|
||||
| 08-neg-python-unused-imports | no | — | — | regex on the two unused imports co-occurring with an "unused" statement + `tool_used: Skill` with `min: 0, max: 0, arm: both` (scored in both arms, unlike the display-only `skill-fired`) |
|
||||
|
||||
Shared graders on 01–06: `content-caught` (w1.5), `format-caught` (w0.5, spec-
|
||||
level mechanics), `no-false-positive` (w1), `honest-unverified` (w1),
|
||||
`no-overreach` (w0.5), plus `skill-fired` (`tool_used: Skill`, display-only under
|
||||
ablation, never moves Δ). Every llm rubric is written as "work through the checks
|
||||
one at a time and quote the evidence"; keep that style when adding graders.
|
||||
|
||||
## Side channels and ceilings (pilot 3, 2026-09-13, 1 run × 2 arms, sonnet agents)
|
||||
|
||||
| Channel | Ceiling | Observed max |
|
||||
|---|---|---|
|
||||
| wall-clock per run | 600 s (`timeout_seconds`; over = score 0) | 142 s |
|
||||
| turns per run | 20 (`max_turns`) | 7 |
|
||||
| agent cost per run | none enforced | $0.45 |
|
||||
| full pilot (8 cases × 2 arms × 1 run, incl. opus judge) | — | $4.65 |
|
||||
|
||||
Pilot 1 (before calibration, opus agents) cost $6.89 and peaked at 155 s / 7
|
||||
turns / $0.83 per run; pilot 2 (sonnet agents) cost $3.99. Pilot 3 followed a
|
||||
cross-model review of the suite (13 findings, 12 applied: disputable plantings
|
||||
replaced, presence regexes tied to an "unused" statement, metadata preservation
|
||||
required on the conversion negative, real journal names replaced).
|
||||
|
||||
## Known caveats
|
||||
|
||||
- **The with-plugin arm cannot load the mode's own prompt in the eval sandbox.**
|
||||
`/ars-citation-check` is a command stub that tells the model to read
|
||||
`MODE_REGISTRY.md` and `academic-paper/SKILL.md`; the plugin directory is
|
||||
outside the sandbox cwd, `Glob`/`Read` there are denied under `dontAsk`, and the
|
||||
model never falls back to invoking the `academic-paper` skill itself
|
||||
(`--allow-tools 'Read(<plugin>/**)'` does not help: the model does not know the
|
||||
path). So in this suite the with arm ≈ command text + base model, and
|
||||
`citation_compliance_agent.md` never runs. Δ ≈ 0 across the suite is the
|
||||
honest current reading, not a calibration failure. Fix belongs in the plugin
|
||||
(command should invoke the skill, or carry `${CLAUDE_PLUGIN_ROOT}` paths): #857.
|
||||
- **Trigger rate with sonnet is 3 of 6 fire cases** (01, 02, 06 fired; 03 prose
|
||||
"check the citations", 04 terse "look over the refs", 05 Spanish did not).
|
||||
05 is exactly the #850 gap: the skill description carries no Spanish
|
||||
triggers. Display-only; tracked in #858 (triggers) and #850 (locale).
|
||||
- **03 was a `/ars-citation-check` slash-command case in pilot 1.** The
|
||||
without-plugin arm answered "Unknown command" and Δ was +0.70 for the wrong
|
||||
reason (command existence). Switched to a prose trigger.
|
||||
- **Clean entries must be clean at the claim level too.** Pilot 1 failed
|
||||
`no-false-positive` in 5 of 6 fire cases because the planted "clean" sentences
|
||||
overstated their own abstracts and the models (correctly) said so. Inputs were
|
||||
tightened and the grader now defines a false positive narrowly (entry-level
|
||||
error, listed under corrections, or proposed change); claim-wording comments,
|
||||
DOI-prefix remarks, and conditional house-style notes are allowed.
|
||||
- **02 order of scripts is not graded.** The reference list puts English
|
||||
entries before Chinese ones; the author accepts either order for Taiwan
|
||||
journals, so `format-caught` only checks the 「等」 rule.
|
||||
- **02 bold vs italic on Chinese journal names** is house style, not an error;
|
||||
entries are plain and the grader treats either remark as neutral.
|
||||
- **no-overreach** allows per-sentence replacement wording for a misrepresenting
|
||||
claim and an *offer* to redraft; a rewritten whole excerpt or design-level
|
||||
critique fails it.
|
||||
- **Regex presence checks are secondary.** `orphan-named` (w0.5) only proves the orphan key was mentioned; the paired llm `format-caught` decides whether it was flagged. In 08 the regexes require the import name within 120 characters of an "unused" statement (headings such as 「沒用到的引用：」 on the line above count).
|
||||
- **04 does not require the style to be named.** Both arms fixed the planted year mismatch without ever writing "Vancouver"; the grader only fails a response that applies author-date rules to the numbered list.
|
||||
- **07 and 08 show Δ 0** — the base model already handles them. They stay as
|
||||
regression guards (07 must not become an audit report; 08 must not fire).
|
||||
Reference in New Issue
Block a user