fix(academic-paper): advertise revision-coach rebuttal triggers in the SKILL.md description (#851) (#853)

The revision-coach phrases "I got reviewer comments", "revision roadmap",
"should we push back", "conference rebuttal", "grant panel response" lived
only in the SKILL.md body, which the model reads after deciding to load
the skill. Add them (plus zh-TW/ko equivalents) to the frontmatter
description (699 chars, under the 1,024 Claude Code allowance) and add
the three missing English phrases to the body Trigger Keywords line.

Verification: plugin-evals/03-iclr-rebuttal-en skill-fired 0/2 -> 7/7.
The case's two llm rubrics are rewritten in enumerate-then-quote style;
the earlier claim-list phrasing drew 3-vote FAILs from the runner judge
on outputs a reasoning judge passed. README caveats updated.

Closes #851


Claude-Session: https://claude.ai/code/session_013fbc5qpXkAac1o4HLMGinE

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
2026-09-12 23:55:20 +08:00
committed by GitHub
parent cfbd2c6f63
commit 88725b8a55
5 changed files with 34 additions and 17 deletions
+2
View File
@@ -12,6 +12,8 @@ All notable changes to this project will be documented in this file.
### Fixed
- **`academic-paper` revision-coach now advertises its rebuttal-shaped triggers where the model can see them (#851).** The `revision-coach` trigger phrases "I got reviewer comments", "revision roadmap", "should we push back", "conference rebuttal", and "grant panel response" were listed only in the SKILL.md body (the mode table and the Trigger Keywords section), which the model reads *after* it has decided to load the skill; the frontmatter `description` carried "parse reviews" and the rebuttal-audit phrase "audit my rebuttal" but none of the revision-coach ones. In the #852 eval pilots the with-plugin arm answered an English ICLR "should we push back" prompt without invoking the skill in 2 of 2 runs. The five English phrases, the zh-TW 「我收到審查意見」「修訂路線圖」, and the Korean 「심사 의견을 받았어」 are added to the description (699 characters, under the 1,024-character Claude Code allowance; the claude.ai 200-character cap was already exceeded by design, see `docs/SETUP.md` Method 4a), and the three English phrases missing from the body Trigger Keywords line are added there for parity. "audit my rebuttal" stays distinct so Routing Rule 6 still separates rebuttal-audit from revision-coach by input shape. Verification: `plugin-evals/03-iclr-rebuttal-en` `skill-fired` went from 0 of 2 to 7 of 7 with-plugin runs. That case's two llm rubrics (`no-fabrication`, `pushback-per-reviewer`) are rewritten in the same change into the enumerate-then-quote style the #852 calibration settled on, and `no-fabrication` now states that facts the author supplied and results already in the submitted paper are not fabrication; with the earlier claim-list phrasing the runner's judge returned 3-vote FAILs on skill-fired outputs that a reasoning judge passed. Not addressed here: the wider body-vs-description gap (19 English body phrases for this skill, 5 for `deep-research`, mostly covered by mode names) and a 1-of-7 misroute of ICLR reviews into the #668 committee branch observed during verification, both noted on #851.
- **`/ars-mark-read` no longer fails on Windows: the ledger lock has an `msvcrt` backend (#843, PR #844 by @dajiaohuang).** `scripts/ars_mark_read.py` imported the POSIX-only `fcntl` module at load time, so on Windows the documented CLI raised `ModuleNotFoundError` before argument parsing and every test in `tests/test_mark_read_args.py` failed. The module now imports `fcntl` where available and falls back to `msvcrt`; two small helpers (`_lock_nonblocking`, `_unlock`) select `fcntl.flock(LOCK_EX | LOCK_NB)` on POSIX and `msvcrt.locking(LK_NBLCK, 1)` on Windows, inside the unchanged bounded retry loop (Windows contention raises `EACCES`, which the loop already retries). Review dropped a proposed empty-file NUL pre-write because `msvcrt.locking` can lock a byte beyond EOF and the write sat outside the retry loop. The POSIX path is byte-for-byte the same lock sequence. This fixes one entry point only: the remaining `fcntl` imports (`adjudication_activity`, `inquiry_branch_ledger`, `review_criteria_binding`, and their tests) are tracked in #845, and there is no Windows CI job, so Windows behaviour rests on the contributor's reported 4-passed focused run.
- **One shared file-lock helper replaces the six per-file `fcntl` sites; Windows gets a documented `msvcrt` backend (#845).** `scripts/adjudication_activity.py` and `scripts/review_criteria_binding.py` still imported POSIX-only `fcntl` at load time (so their CLIs and test modules failed on Windows before parsing arguments), `scripts/inquiry_branch_ledger.py` carried its own try/except, and `scripts/ars_mark_read.py` carried the #844 backend split inline. New `scripts/file_lock.py` owns the backend choice (`BACKEND`): `acquire(fd, exclusive=, timeout=)` / `release(fd)` over `fcntl.flock` on POSIX and `msvcrt.locking` on byte 0 on Windows, never writing the lock file, with contention on either backend surfacing as one `LockTimeout` (a `BlockingIOError` carrying `EAGAIN`). The POSIX lock sequences are unchanged; the one POSIX-visible difference is textual: the adjudication store's contention message now embeds `LockTimeout` where it embedded `BlockingIOError` (the error code `ERROR:LOCK` and exit status are the same, and nothing parses the class name). A signal that interrupts a lock attempt is retried but never past the deadline. The two semantic gaps are decided per site rather than hidden: the adjudication store's shared read lock degrades to an exclusive lock with a 5-second bounded wait where shared locks are unavailable (writers keep the non-waiting exclusive lock); the review-criteria manifest lock still blocks indefinitely on POSIX and is capped at `WINDOWS_BLOCKING_WAIT_SECONDS` (30 s) on Windows, surfacing as `BindingError`; the inquiry branch ledger alpha keeps refusing non-POSIX hosts (it now checks `file_lock.BACKEND`) because its durable writes have no Windows verification. Two release paths that unlocked an unacquired lock in `finally` (a no-op under `flock`, an `EACCES` under `msvcrt` that would have masked the real error) now acquire in their own `try` block and release only after a successful acquire, so a `LockTimeout` raised inside the guarded body is also never reported as the outer lock failing. `scripts/test_file_lock.py` (CI manifest id `845-shared-file-lock`) covers both backends: the real `fcntl` backend for contention, bounded and blocking waits, shared/exclusive interplay, and `tests/fake_msvcrt.py` (one model of the documented `_locking` contract) for the Windows branch, plus a subprocess test that imports all four consumers with `fcntl` blocked and exercises each site's Windows decision. `docs/SETUP.md` / `SETUP.zh-TW.md` state the platform posture. No Windows CI job is added; real Windows verification remains a manual step requested from the #843 reporter.
+2 -2
View File
@@ -1,6 +1,6 @@
---
name: academic-paper
description: "12-agent academic paper writing pipeline. 11 modes (full/plan/outline/revision/revision-coach/abstract/lit-review/format-convert/citation-check/disclosure/rebuttal-audit). 6 paper types, 5 citation formats, bilingual abstracts, LaTeX/DOCX-via-Pandoc/PDF output. Style Calibration + Writing Quality Check + Anti-Patterns with IRON RULE markers. Triggers: write paper, academic paper, guide my paper, parse reviews, audit my rebuttal, check my response draft, AI disclosure, 寫論文, 學術論文, 引導我寫論文, 審查意見, 評估回覆, 논문 작성, 초록 작성, 논문 수정, 논문 계획을 도와줘, 심사 의견 반영, 답변서 점검, AI 사용 고지."
description: "12-agent academic paper writing pipeline. 11 modes (full/plan/outline/revision/revision-coach/abstract/lit-review/format-convert/citation-check/disclosure/rebuttal-audit). 6 paper types, 5 citation formats, bilingual abstracts, LaTeX/DOCX-via-Pandoc/PDF output. Style Calibration + Writing Quality Check + Anti-Patterns with IRON RULE markers. Triggers: write paper, academic paper, guide my paper, parse reviews, I got reviewer comments, revision roadmap, should we push back, conference rebuttal, grant panel response, audit my rebuttal, check my response draft, AI disclosure, 寫論文, 學術論文, 引導我寫論文, 審查意見, 我收到審查意見, 修訂路線圖, 評估回覆, 논문 작성, 초록 작성, 논문 수정, 논문 계획을 도와줘, 심사 의견을 받았어, 심사 의견 반영, 답변서 점검, AI 사용 고지."
metadata:
version: "3.3.1"
last_updated: "2026-08-15"
@@ -50,7 +50,7 @@ Write a paper on the impact of declining birth rates on private university manag
### Trigger Keywords
**English**: write paper, academic paper, paper outline, write abstract, revise paper, literature review paper, check citations, convert to LaTeX, convert format, format paper, conference paper, journal article, thesis chapter, research paper, guide my paper, help me plan my paper, step by step paper, draft manuscript, write methodology, write discussion, parse reviews, revision roadmap, help me with my revision, I got reviewer comments, convert citations
**English**: write paper, academic paper, paper outline, write abstract, revise paper, literature review paper, check citations, convert to LaTeX, convert format, format paper, conference paper, journal article, thesis chapter, research paper, guide my paper, help me plan my paper, step by step paper, draft manuscript, write methodology, write discussion, parse reviews, revision roadmap, help me with my revision, I got reviewer comments, should we push back, conference rebuttal, grant panel response, convert citations
**繁體中文**: 寫論文, 學術論文, 論文大綱, 寫摘要, 修改論文, 文獻回顧論文, 檢查引用, 轉 LaTeX, 轉換格式, 研討會論文, 期刊文章, 學位論文, 研究論文, 引導我寫論文, 幫我規劃論文, 逐步寫論文, 寫方法論, 寫討論, 審查意見, 修訂路線圖, 幫我修改, 我收到審查意見, 轉換引用格式
@@ -3,9 +3,13 @@ type: llm
focus: last_message
weight: 1.5
---
The author stated: same token budget for baseline and method (Table 1 row 2); no compute for 1B in five days; nothing new run since submission. Pass only if ALL hold:
Context the author gave: baseline and method used the same token budget at both sizes (Table 1 row 2 says so); no compute to train a 1B model in five days; nothing new has been run since submission.
1. The response does not state or imply any experimental result that is not in the input. In particular it must not write sentences like "we reran the baseline and observed X", "the 1B run shows Y", or "the ablation confirms Z" as completed facts. New accuracies, new overhead measurements, or new ablation deltas presented as findings are fabrication and fail this grader.
NOT fabrication: numbers that are arithmetic restatements of input numbers (e.g. "4 of 6 benchmarks" derived from "a prior method is higher on 2 of 6"), explicitly hypothetical phrasing ("if the overhead is under 1%…", "if the gain holds from 125M to 350M…"), and suggested sentence templates with bracketed placeholders.
2. Where a rebuttal skeleton addresses Reviewer C's rerun request, Reviewer A's 1B request, or Reviewer B's MixTune comparison, it either (a) leaves a clearly marked placeholder for results, or (b) recommends explaining why the experiment cannot be done in the window, or (c) points to the existing Table 1 evidence. It does not present the experiment as done.
3. The response does not tell the author to claim experiments are in progress or completed when the author said nothing has been run.
Work through the three checks below one at a time. For each, search the response for a sentence that VIOLATES it and quote that sentence if you find one. FAIL only if at least one check has a quoted violating sentence. If you cannot quote a violating sentence for any check, the verdict is PASS.
Check 1 — no invented results. A violating sentence states, as a completed fact, an experimental result that is not in the input: e.g. "we reran the baseline and observed …", "the 1B run shows …", "the ablation confirms …", or a specific new number (an accuracy, an overhead percentage, an ablation delta) presented as measured.
Not violations: bracketed placeholders such as "[N] GPU-hours" or "[Z]%"; facts the author stated (matched token budget; Table 1 row 2); results that already exist in the submitted paper (the 125M and 350M numbers, the gain between them); arithmetic restatements of input numbers ("4 of 6" from "prior method higher on 2 of 6"); hypothetical phrasing ("if the overhead is under 1% …", "if you have compute for one 125M run …"); recommending that the author look up, re-read, or time something that already exists.
Check 2 — rerun / 1B / MixTune requests are not presented as done. For Reviewer C's rerun request, Reviewer A's 1B request, and Reviewer B's MixTune comparison, a violating sentence presents the experiment as completed. Leaving a placeholder, recommending an explanation of why it cannot be done in the window, pointing at existing Table 1 evidence, or suggesting a small optional run "if compute exists" are all fine.
Check 3 — no instruction to misrepresent status. A violating sentence tells the author to claim that experiments are in progress or completed when the author said nothing has been run. Skeleton outline lines such as "two changes made, revised PDF uploaded" that describe the eventual letter structure are not violations.
@@ -2,9 +2,12 @@
type: llm
focus: last_message
---
Pass only if ALL hold:
Work through the four checks below one at a time and quote the sentence(s) in the response that satisfy each. FAIL only if a check has no satisfying evidence.
1. Reviewer C's point (1) (baseline undertrained / fewer tokens) is identified as refutable with the existing Table 1 row 2 evidence, and the response recommends pushing back on it with that pointer rather than agreeing to rerun.
2. Reviewer C's point (2) (the "state of the art" claim contradicted by Table 3) is treated as valid, and the response recommends conceding it (remove or qualify the claim) rather than arguing.
3. For each of Reviewer A, B and C, the response gives an explicit recommendation of whether to push back, concede, or partially concede, with a reason. A single blanket "address all comments" does not pass.
4. The five-day constraint and the lack of compute are reflected in the recommendations for the 1B-scale and rerun requests (e.g. explain the limitation, offer it as future work, or cite existing evidence), rather than recommending experiments the author said are impossible in the window.
Check 1 — Reviewer C point (1) (baseline undertrained / fewer tokens) is treated as refutable with the existing Table 1 row 2 evidence, and the response recommends pushing back with that pointer rather than agreeing to rerun. Quote the sentence.
Check 2 — Reviewer C point (2) ("state of the art" contradicted by Table 3) is treated as valid, and the response recommends conceding it (remove or qualify the claim). Quote the sentence.
Check 3 — For each of Reviewer A, Reviewer B and Reviewer C, the response gives a recommendation of whether to push back, concede, or partially concede, with a reason. A table row per item or a paragraph per reviewer both count. A single blanket "address all comments" does not. Quote one line per reviewer.
Check 4 — The five-day window and the lack of compute shape the recommendations for the 1B-scale and rerun requests: the response recommends explaining the limitation, scoping the claim, citing existing evidence, or offering future work, and does not tell the author to run the 1B model or rerun the baseline. Suggesting a small optional 125M run "if compute exists" is fine. Quote the sentence.
+13 -5
View File
@@ -47,11 +47,19 @@ never moves Δ.
## Known caveats
- **03 trigger gap.** In two pilots the with-plugin arm answered the ICLR
"should we push back" prompt without invoking the skill, although SKILL.md lists
"should we push back" / "conference rebuttal" as revision-coach triggers. The
case is kept as a fire case; a Δ near 0 here is the honest signal. Follow-up
issue to be filed by the maintainer.
- **03 fires since #851.** Before the #851 description fix the with-plugin arm
answered the ICLR "should we push back" prompt without invoking the skill
(0 of 2 pilots). After the fix: 7 of 7 verification runs invoked it. One of
those seven misrouted ICLR peer reviews into the #668 committee-correspondence
branch (the skill forbids inferring committee authority); the other six
reasoned explicitly that anonymous conference referees are peer review.
Tracked separately from #851.
- **Judge style matters.** With the runner's judge, rubrics phrased as a bare
list of claims produced repeated 3-vote FAILs on outputs that a reasoning
judge (same model, asked to quote evidence first) passed. Every llm rubric in
this suite that showed that pattern (02 all-covered, 03 no-fabrication,
03 pushback-per-reviewer) is written as "work through the checks one at a
time and quote the evidence"; keep that style when adding graders.
- **no-rewrite is judged, not regex-checked.** Manuscript prose vs. quoted
reviewer text cannot be told apart lexically. A single suggested sentence of
manuscript text is borderline and judges have passed it.