mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
feat: submission-package verifier Slice 1 — CLI skeleton + Family C reference integrity (#394) (#407)
* feat(scripts): submission-package verifier slice 1 — Family C joined marker path (#394) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(scripts): Family C fallback extraction (\cite + author-year) + fingerprint pin (#394) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(ci+docs): wire #394 slice-1 tests into pytest manifest; CHANGELOG; spec §5.3 warn + §10.3 fingerprint adjudication Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(scripts): adopt codex review — no identity guess on partial joins; C2/warn never strict-eligible; --report-out fingerprint exclusion; page-locator tails (#394) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(scripts): apply /simplify review — single bib grammar, check registry + roster guard, one-walk reads, unjoined channel for fallback, schema id↔family binding (#394) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
committed by
GitHub
GitHub
parent
2e487cefd3
commit
55ef51dcb0
@@ -10,6 +10,8 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
### Added
|
||||
|
||||
- **Submission-package verifier Slice 1: CLI skeleton + Family C reference integrity (#394).** First implementation slice of the 2026-06-10 #394 spec (slices are dependency-ordered; Family C ships first because it needs zero new parser dependencies). New `scripts/verify_submission_package.py` standalone CLI: point it at any output package directory and it runs the two-way reference-integrity set check (orphan in-text citation = `fail`, uncited reference entry = `warn`) and writes `submission_verification_report.json` validating against the new `shared/contracts/submission/submission_verification_report.schema.json`. The deterministic **joined marker path** consumes `<!--ref:slug-->` markers plus a real prose-reference join — the passport's `citation_verification_summary[]` (`--passport`), an explicit scholar-supplied map (`--join-map`), or a package `.bib` via the documented slug==citation_key identity relation — and markers with NO join source report `NOT-CHECKED(missing prose-reference join)`, never a guessed comparison (§3.3). Non-ARS / post-converted packages fall back to format-aware **best-effort extraction** (`\cite{}` for LaTeX, author-year regex for Markdown), heuristic-classed: the schema STRUCTURALLY forbids `signal_class: heuristic` + `strict_eligible: true`, so the fallback path can never be promoted to blocking by any later policy slice. Report header carries `extraction_path`, `not_checked_count` (incompleteness is never folded into pass, §1.4), `package_fingerprint` (spec §10 open item 3 adjudicated: the audit-snapshot manifest convention — byte-sorted `path:sha256` lines, fingerprint = SHA-256 of the manifest, report file excluded), and `policy_slug` (always null until the slice-4 orchestrator hook; the script never reads `terminal_policies`, §5.3). Exit codes separate "all checked, pass" (0) from "passed what was checkable" (3) per §8. 19 tests + 7 fixture packages with mutation discipline (orphan / uncited / no-join failures proven to fire); CI-wired via the pytest manifest. Advisory-only: no manuscript byte changes, no pipeline hook yet (slice 4). #394 stays open until all slices land.
|
||||
|
||||
- **Design doc: deterministic submission-package verifier (#394, blindspot-audit F-5, design-first — no implementation yet).** `docs/design/2026-06-10-394-submission-package-verifier-spec.md` designs `scripts/verify_submission_package.py`, the script-layer backstop for the mechanical subset of the formatter's prompt-layer submission checklists (the #182 promotion pattern: LLM self-check → deterministic gate). Three check families in adjudicated priority order — blind-review residue (raw-structure scan: PDF/DOCX metadata authors, tracked-changes/comment authors, self-citation phrasing; **artifact ≠ rendered view** is a stated premise), venue-declared limits vs actuals (scholar-declared `venue_profile` schema, never inferred from the journal name — R-L3-2-D mirror), reference integrity (two-way set check; the deterministic path requires an explicit slug↔key join source or reports `NOT-CHECKED`) — plus a stretch *assessment* of `repro_lock` presence/shape checking that leaves the recorded gates-don't-read-it boundary untouched. Two load-bearing rules: `signal_class` and `strict_eligible` are separate axes (heuristic checks are structurally excluded from strict; A4's deterministic signal still isn't block-worthy because the de-anonymization judgment is the scholar's), and **strict fails closed on incompleteness** (`VERIFICATION-INCOMPLETE` when a strict-eligible check can't run — a missing parser must not silently waive the one class the user opted into blocking on). Terminality via a new additive `terminal_policies.submission_package` key, evaluated by the orchestrator against a fingerprint+policy-slug-stamped report (package-level freshness guard — explicitly a new post-format gate, not the ref-marker stamp path). Cross-model reviewed (codex): 2 P1 (Family C join requirement; strict fail-open on NOT-CHECKED) + 4 P2 + 2 P3, all adopted. 4 dependency-ordered slices; advisory-only until slice 4.
|
||||
|
||||
- **`Real-use findings` release-notes convention documented; showcase refresh tracked (#395, blindspot-audit F-9).** CONTRIBUTING.md gains a Release checklist section documenting the convention: a release's CHANGELOG entry includes a `Real-use findings` subsection when issues were discovered through actual use on a real paper — one line per issue naming the run — so lived-experience provenance has a fixed, greppable home instead of being buried in spec prose (the v3.6.7 production chapter run surfaced 17 drift patterns and nothing structural recorded that provenance class; release motivation since v3.8 has been almost entirely external papers, which is itself a per-release signal worth seeing). Omitted when empty, never padded. The showcase refresh half of F-9 (no post-v2.7 end-to-end artifact set exists) is split to tracking issue #404, blocked on the next real paper with no artificial deadline per the adjudication.
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
# ARS 資深研究者視角盲點盤點
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| 日期 | 2026-06-10 |
|
||||
| 基準 | main `6252b1b`（v3.12.0 + Item 7 spec） |
|
||||
| 性質 | 產品級審查：這套工具拿來產真論文，缺什麼、擋什麼、騙了自己什麼 |
|
||||
| 研究者畫像 | 高教品保領域、systematic review／政策研究、中英雙語投稿、單人為主偶有共作 |
|
||||
| 外部輸入 | AACSB Global Research Impact Task Force, *A Framework for Research Impact* (May 2026) |
|
||||
| 審查軌跡 | 初稿 → 主 context 自查（3 修正）→ codex 0.137 high + gemini-3.1-pro dual-track（11+7 findings）→ 本版 |
|
||||
| 狀態 | 待使用者裁定；裁定前不開 issues |
|
||||
|
||||
分類標記：**【盲點】**初衷內該有而沒有／**【增補】**初衷外但高價值，附說服理由／**【non-goal】**誠實列出為什麼不做。撞已拍板 negative scope 的提案，明寫要求推翻哪條決策。
|
||||
|
||||
---
|
||||
|
||||
## 0. 一句話結論
|
||||
|
||||
ARS 是一台「文獻→寫作→審查→修訂」中段防錯密度極高的單篇論文品管機；它最大的結構性問題不在中段（中段是強項），而在三處：**(1) 跨篇 research program 層級的 state 為零**，每篇論文都從失憶開始（是盲點還是 scope 外，送裁定）；**(2) 品質機制全部回答「這篇有沒有錯」，沒有任何機制讓「這篇無懈可擊但平庸」變得可見**；**(3) 工具的旗艦防線（deterministic citation verification）在使用者自己的研究類型（政策研究、灰色文獻為主）上覆蓋率系統性最低**。
|
||||
|
||||
Dual-track 補了第四條：本盤點初稿自己也犯了「AI 對自寫物 anchoring」的錯——兩個提案（F-4、F-5 之半）提的是 repo 已存在的東西，一個提案（F-2 novelty mirror）原始形狀踩了 hidden-ranking 紅線。修訂記錄保留在各節，因為這些誤判本身就是「為什麼這個 repo 需要 cross-model 審」的現場證據。
|
||||
|
||||
---
|
||||
|
||||
## 1. 研究生命週期走查
|
||||
|
||||
生命週期：**grant/計畫申請** → RQ 孵化 → 文獻 → 方法設計 → 實驗/資料 → 分析 → 寫作 → 內部審查/修改 → 投稿 → rebuttal → camera-ready → 發表後 → 下一篇。（grant 段是 gemini 審查補上的，初稿漏列，見 F-13。）
|
||||
|
||||
| 階段 | 覆蓋 | 深度 | 判定 |
|
||||
|---|---|---|---|
|
||||
| grant/計畫申請 | funding_statement_guide（formatter reference）為止 | 近零 | **未拍板留白**：F-13 |
|
||||
| RQ 孵化 | deep-research `socratic`（5-layer）+ FINER guidance-tool 對話 + #257 wording advisory | 完整對話工作流；Layer 1/5 已含 impact 探問（dual-track 修正，見 F-4 撤回紀錄） | 蓋到 |
|
||||
| 文獻搜尋/篩選 | `lit-review` / `systematic-review`（PRISMA）+ literature_corpus + 四索引 citation gate | 全 suite 最厚 | 蓋到；灰色文獻低覆蓋（F-3） |
|
||||
| 方法設計 | research_architect（Methodology Blueprint）；preregistration 僅 reference | Blueprint 為止 | 蓋到 |
|
||||
| 實驗/資料蒐集 | 無。#260 只做 scholar 宣告的 provenance intake | 僅 intake/audit | **non-goal（正確）**：Kong §3.3 rejected；companion experiment-agent 接手 |
|
||||
| 分析 | synthesis（文獻層）+ meta_analysis（SR）+ figure fidelity gate #261 | 文獻層完整；primary data 分析無 | non-goal（experiment-agent 領地），邊界清楚 |
|
||||
| 寫作 | academic-paper 10 modes、12 agents、style calibration、anti-leakage | 完整 | 蓋到 |
|
||||
| 內部審查/修改 | reviewer 6 modes + in-pair evaluator + R&R traceability + revision/revision-coach | 全 suite 最厚 | 蓋到；「建設 vs 防禦」失衡（F-2） |
|
||||
| 投稿 | formatter：cover letter、CRediT 14 角色模板、DAS 四模板、COI/funding/ethics、雙盲版去作者資訊、Pre-Output Final Checklist（皆 prompt 層）；disclosure mode；journal_submission_guide（含 TSSCI 節） | **prompt 層 checklist 完整**（初稿低估，codex 修正）；deterministic 驗證零 | F-5 改寫後仍立案：gap 是 deterministic 化不是從無到有 |
|
||||
| rebuttal/response | `revision`（point-by-point R&R）+ `revision-coach`（Response Letter Skeleton）吃外部審稿意見 | 完整工作流 | 蓋到 |
|
||||
| camera-ready | format-convert（LaTeX/DOCX/PDF）為止 | 排版機械層 | 蓋到 |
|
||||
| 發表後 | monitoring_agent（optional：retraction/correction alert、contradictory findings、author tracking）+ literature_monitoring_strategies（citation alert 指南） | advisory 指南層，非主流程（初稿寫「全空白」過度，codex 修正） | 殘餘 gap 窄而明確：F-7 |
|
||||
| 下一篇（research program） | 無任何機制 | 零 | **F-1**：事實成立，分類送裁 |
|
||||
|
||||
走查結論：中段密度世界級。前端薄是 deliberate（實驗外包、idea generation 是 Kong L2 紅線）。**grant 段與「發表後+下一篇」段是未經裁定的留白**——沒有任何 design lesson 記錄過為什麼不做，跟五條 Rejected mechanisms 的待遇不對等。這是本盤點送裁定的核心。
|
||||
|
||||
---
|
||||
|
||||
## 2. 發現總表（dual-track 修訂後）
|
||||
|
||||
| # | 發現 | 分類 | 狀態 |
|
||||
|---|---|---|---|
|
||||
| F-1 | 跨篇 research program state 為零 | 增補（scope 裁定）；初稿標盲點，codex 論證 POSITIONING 寫的是 research-to-publication 不含 program 管理，降級 | 送裁 §3 |
|
||||
| F-2 | 品質機制無「平庸可見性」 | 盲點 | 立案 §4；形狀 2 經 codex 抓出紅線問題後重設計 |
|
||||
| F-3 | 灰色文獻在四索引 citation gate 下系統性低覆蓋（初稿「全部查無」過度，已軟化） | 盲點 | 立案 §6.2 |
|
||||
| F-4 | ~~RQ 孵化缺 impact-pathway 探問~~ | **撤回** | §5.1 保留撤回紀錄：提案內容已存在於 socratic Layer 1/5 |
|
||||
| F-5 | submission package 檢查的 deterministic 化（初稿「無此功能」錯；prompt 層 checklist 已完整） | 增補 | 改寫後立案 §5.2 |
|
||||
| F-6 | venue selection 無支援 | 增補（撞線） | §5.3；dual-track 後條件收窄為 scholar-supplied candidate universe |
|
||||
| F-7 | 發表後段殘餘 gap：自我論文的 citation-context audit、errata workflow、OA self-archiving 合規 | 增補（要動 POSITIONING scope） | 改寫後立案 §5.4 |
|
||||
| F-8 | env flag 無單一總表（7 個 user flag，SETUP.md env 表列 5）；安全功能全 opt-in 預設關 | 盲點 | 立案 §7.1（數字經自查修正，codex 同向確認） |
|
||||
| F-9 | Dogfooding 半斷流：worked example 停在 v2.7；v3.8 之後版本全為外部論文驅動（初稿「v3.4 之後全論文驅動、真實使用零痕跡」錯——v3.6.7 即 chapter-run 驅動且有 spec 記錄） | 盲點（窗口收窄後仍成立） | 立案 §7.2 |
|
||||
| F-10 | 中文線止於排版：TSSCI 檢索/引用/字體有，中文審查慣例與投稿 workflow 無 | 增補 | 立案 §6.1 |
|
||||
| F-11 | Multi-author：team_collaboration_protocol.md 已有 human-convention 層（初稿「零支援」錯）；passport 層 multi-author state 為零 | non-goal 維持 | §8 |
|
||||
| F-12 | plain-language summary / 衍生物 | non-goal 維持（撞 Paper2X），理由 §8 | — |
|
||||
| F-13 | grant/funding lifecycle 整段不在走查與產品內（gemini 補抓） | 增補（scope 裁定） | §5.5 |
|
||||
| F-14 | bonus hygiene：Originality 權重 repo 內部不一致（quality_rubrics.md 20% vs review_criteria_framework.md 15%，codex 抓） | 盲點（小） | 順手修，可直接開 issue |
|
||||
|
||||
---
|
||||
|
||||
## 3. F-1 跨篇知識累積：research program 層級失憶【增補，scope 裁定】
|
||||
|
||||
### 事實（未被任何審查挑戰）
|
||||
|
||||
所有跨 session 機制都是**單篇內**的：Material Passport 是 per-run state；`resume_from_passport` 續同一 run；`compliance_history[]`、`reset_boundary[]`、`experiment_provenance[]` 都掛在單篇 passport 上。唯二跨篇的東西：verification cache（引用查核的效能快取）、`literature_corpus[]`（使用者外部維護重餵，No corpus mutation Iron Rule 明文不回寫）。
|
||||
|
||||
真實研究者的工作單位是研究線不是論文：同批文獻寫三篇、上一篇的 limitations 是下一篇的 RQ 種子、上一篇被 reviewer 打過的弱點下一篇先補、paper A 主張過 X 則 paper B 不能無意識主張 ¬X。ARS 自己的機制已經在單篇內生產這些資產（Stage 6 AI Self-Reflection、Acknowledged Limitations、R&R Traceability Matrix），但 run 結束即死。
|
||||
|
||||
### 分類修訂（codex P1）
|
||||
|
||||
初稿標【盲點】，論證是「copilot 服務 scholar，scholar 的存在形式是研究線」。codex 反駁：POSITIONING 的自我定義是 "full research-to-publication pipeline"——publication 是終點，research program 管理是擴張解釋。我接受：**這是 scope 決定不是落在既有 scope 內的漏洞**，改標【增補】，裁定權在使用者。AACSB spiral model（outcome 回饋下一輪 inquiry）是擴的理由，不是已承諾的依據。
|
||||
|
||||
### 紅線檢查（dual-track 後收緊）
|
||||
|
||||
「上一篇 limitation → 下一篇 RQ」做成 ARS 主動提案 = 撞 Kong L2。codex 進一步指出：L2 的 verb test 允許的是「對 scholar-supplied RQ 的 wording advisory」，**pre-RQ 階段 surface 上篇 limitations 並不在 L2 的明文允許清單內**，屬於 L2 沒有預想過的新 seam。合規形狀因此要滿足三個條件（codex 修訂）：
|
||||
|
||||
1. **Scholar-initiated**：scholar 明示「載入我的研究線」才啟動，不是新 run 自動跳出。
|
||||
2. **All-artifacts-visible**：呈現該研究線的全部 prior limitations / 未解意見，不做「最相關的三條」這種隱性篩選（同 hidden-ranking 原則）。
|
||||
3. **不 derive**：呈現原文 + Socratic 問句為止，不從 limitation 推導、改寫或排序候選 RQ。
|
||||
|
||||
跨篇 claim 一致性檢查（自我前作的 claim registry 對照）是 audit 不是 generation，#262 機制形狀可複用，紅線風險低。
|
||||
|
||||
### 與 data layer boundary 的關係（送裁）
|
||||
|
||||
2026-04-22 的禁令擋的是「整合外部 corpus」；research-line ledger 承接的是 **ARS 自產 artifacts**（passport、reflection、R&R matrix——格式 ARS 定義、內容 ARS 產出）。我判斷不在禁令射程內，但邊界是使用者拍的板。
|
||||
|
||||
### 裁定問題
|
||||
|
||||
要不要開「research line passport」設計線？最小切片：①scholar-initiated 的 prior-limitations advisory surface（上述三條件）；②自我 claim registry 的 cross-paper audit。若裁定不做，建議把「research program 層」明文寫進 POSITIONING non-goal——現在它是留白，跟五條 Rejected mechanisms 的記錄紀律不對等。
|
||||
|
||||
---
|
||||
|
||||
## 4. F-2 防禦性品質 vs 建設性品質：「無懈可擊但平庸」會全綠通過【盲點】
|
||||
|
||||
### 事實
|
||||
|
||||
防錯側：DA 3 mandatory checkpoints、Stage 2.5/4.5 integrity（zero issues 才放行）、7-mode failure checklist（no escape hatch）、generator-evaluator contract、sprint contract、citation gate、claim audit、49 條 lint。防的全是 hallucination / drift / corruption / sycophancy。
|
||||
|
||||
提質側：理論貢獻與論證銳度只出現在**評分側**（EIC/peer review 的 Originality 加權——主 rubric 20%，舊 reference 殘留 15%，見 F-14）與**防守側**（DA So-what test、CER stress test）。寫作期/修訂期的 coaching 全面降維到流程與排序（「選三件事改」「排優先序」）。Socratic 的深度集中在 RQ 期（5-layer 含 assumption probing 與 significance），**不延伸進寫作期**：沒有任何 mode 在 drafting/revision 階段陪 scholar 磨「這篇對哪條理論線的推進點、跟哪個學派對話、delta 在哪」。
|
||||
|
||||
結構性後果：一篇文獻完備、引用全驗證、統計無誤、論證結構完整、回應了所有模擬審稿意見的論文，可以全閘綠燈通過——而它可能是 AACSB p.13 受訪者批評的 "micro extensions of what we already know"。reviewer panel 忠實模擬現行 journal review 體制，等於把該體制推向保守增量的力場也內建了。25 modes 的 spectrum 分布（14 Fidelity / 7 Balanced / 4 Originality）顯示系統重心本來就在 fidelity。
|
||||
|
||||
### 與 S0 洞見的鏡像關係
|
||||
|
||||
HEEACT 評鑑工具加 S0 的原因是「AI 把報告寫到無懈可擊 = 品質訊號失效」。ARS 站在生產端，正是那台把論文寫到無懈可擊的機器。評鑑端已知 compliance ≠ perfection；生產端還沒有對應自覺。解法不是讓 ARS 學會判斷論文價值（LLM 判 novelty 不可靠，v3.0 自承 DA "attacks arguments, never premises"），是讓**平庸這個屬性變得可見**，判斷留給 scholar。
|
||||
|
||||
### 修補形狀（dual-track 後修訂）
|
||||
|
||||
1. **Contribution sharpening 對話層**【維持】：把 socratic Layer 5（SIGNIFICANCE & CONTRIBUTION，已存在於 RQ 期）的問句結構**延伸進 plan mode 與 revision coaching**（"十年後引用本文的人會說它證明了什麼？" "拿掉本文，這條文獻線少了哪塊？"）。改動小：plan_mode_protocol 與 Phase 2.5 coaching 各加一節，問句直接從 socratic_mentor_agent Layer 5 移植。過 L2（問不給）。
|
||||
2. **Novelty delta 鏡子**【初稿形狀踩線，重設計】：初稿提「本文 claim ↔ 三篇最近鄰文獻」對照表。**codex 抓出紅線問題：「選哪三篇」本身就是 hidden selection pressure，Co-Scientist L1 管的是候選集隱性建構與 anchoring，不是最後有沒有寫「你決定」**。重設計後的合規形狀：對照集由 scholar 指定（"跟這五篇比"），或 enumerate 全部 bibliography 中標記為 same-RQ 的 entries（全集可見、不選樣）。成本變高，價值降低；**優先級降到形狀 1 之後**，甚至可以不做。
|
||||
3. **Practitioner/policy-reader persona**【維持，誠實標注】：field_analyst 動態配 reviewer 機制現成，加一個非學術讀者 persona。LLM 模擬 practitioner 仍是 LLM，價值是視角多樣性不是真 stakeholder。
|
||||
|
||||
教訓記錄：「advisory ≠ 自動紅線安全」。本盤點初稿把 advisory 當免死金牌用了一次，被兩個模型從不同條目（F-2、F-6）獨立抓到同一原則。這條應該寫進未來所有 ARS 提案的 review 慣例：**檢查紅線時，先檢查候選集是怎麼建構的，再檢查最後誰決定**。
|
||||
|
||||
### 裁定問題
|
||||
|
||||
推薦先做形狀 1（最便宜、問句現成、最貼 Socratic DNA）。形狀 2 重設計後還值不值得做，送裁。
|
||||
|
||||
---
|
||||
|
||||
## 5. 生命週期斷點的修補提案
|
||||
|
||||
### 5.1 F-4 撤回紀錄：impact-pathway 探問已存在
|
||||
|
||||
初稿提案「socratic 加 impact-pathway 探問維度」。**Dual-track 兩模型一致打掉前提**：`socratic_mentor_agent.md` Layer 1 已有 "If your research succeeds, how would the world be different?" / "Important to whom?"，Layer 5（SIGNIFICANCE & CONTRIBUTION）已有 "If your research succeeds, who would make different decisions as a result?" / "Who benefits once it's filled?"；`research_question_agent.md` 明寫 FINER 是 "guidance tool (not a scoring tool): Designs 2-3 guiding questions for each FINER dimension"。我提案要加的問題逐字級地已經存在。
|
||||
|
||||
誤判根因：subagent 地圖只讀了 SKILL.md 層（拿到 5-layer 的名字沒拿到 agent prompt 的問句），主 context 自查的 grep 關鍵詞（stakeholder / impact pathway / who will use）漏掉了實際表述（how would the world be different / who would make different decisions）。關鍵詞錨定偏誤 + 對自寫提案的 anchoring，雙重失效；同 model 的獨立 context（codex prompt 內含維度提示）反而抓到。
|
||||
|
||||
殘餘 gap 只剩一條小的：Layer 5 的探問停在「誰會不同」，沒有「發表載體通路」的具體化（學術期刊 vs 評鑑準則 vs 政策白皮書——對政策研究者這是 RQ 期就該想的 dissemination 路徑）。價值低，不獨立立案，併入 F-2 形狀 1 的問句清單即可。
|
||||
|
||||
### 5.2 F-5 submission package 的 deterministic 化【增補，改寫後立案】
|
||||
|
||||
初稿宣稱「submission package 完整性檢查無」。**codex 修正：formatter_agent prompt 層已有完整 checklist**——雙盲版去作者資訊（:673）、CRediT 14 角色模板、DAS 四模板、COI/funding/ethics statement、Pre-Output Final Checklist（內容完整性+格式合規+必要元素+投稿包四節，any FAIL → fix and re-check）。
|
||||
|
||||
改寫後的真 gap：**這些全是 prompt 層 checklist（LLM 自我核對），零 deterministic 驗證**。對照 ARS 自己的演進邏輯——citation 從 prompt 層防線走到 #182 deterministic gate 花了 8 輪——submission package 正站在同一條演進線的起點。最值得 deterministic 化的三項：①雙盲去識別化殘留掃描（PDF metadata 作者欄、acknowledgments、自引措辭 "in our previous work"、補充檔檔名）——纯 script 可驗、失敗成本高（desk reject）、對單人研究者（沒有第二雙眼睛）價值最大；②字數/結構限制 vs venue 宣告的機械比對；③reference list ↔ 正文引用的雙向 set 比對。另 codex 指出 `repro_lock` 明文不被 integrity gate 讀（artifact_reproducibility_pattern.md:120-128）——transparency 鏈最後一哩斷在這裡，可併入同一個 verifier。
|
||||
|
||||
### 5.3 F-6 venue selection【增補，撞線，條件收窄】
|
||||
|
||||
現況：無 mode、無 agent；top_journals_by_field.md 是 EIC calibration 內部 reference。
|
||||
|
||||
**紅線分析（dual-track 後收窄）**：gemini 維持我的撞線判定並補刀（「AI 內部 retrieval 哪些 venue 顯示，本身就是 Top-K filter」）；codex 給出唯一可行形狀的精確條件：**candidate universe 必須 scholar-supplied 或 exhaustively disclosed**。亦即 ARS 不產生候選清單，只對 scholar 自己列出的 venues 填多軸事實表（scope 宣告、turnaround、OA 政策、字數限制、AI disclosure 政策——最後這項 ARS 的 venue_disclosure_policies 已有種子）。「幫我找適合的期刊」這個原始需求本身做不了（候選建構=隱性排名），能做的是「我在這四本之間猶豫」的事實比較器。價值縮水後還值不值得，送裁；優先級低於 F-1/F-2/F-5。
|
||||
|
||||
### 5.4 F-7 發表後段【增補，要求擴 POSITIONING scope，措辭修正】
|
||||
|
||||
初稿寫「citation tracking、errata、OA self-archiving 全空白」。**codex 修正：monitoring_agent 不是空白**——retraction/correction alert（含對自己研究的 impact assessment）、contradictory findings 偵測、author tracking、citation alert 設定指南都在，定位是 optional 的 post-research advisory。
|
||||
|
||||
準確的殘餘 gap 三項（皆是「對自己論文」的視角，monitoring_agent 是「對引用的別人論文」的視角）：①**自我 citation-context audit**：誰引用了我、把我的 claim 引成什麼樣——技術上是 L3 claim-faithfulness 的鏡像（同一套 anchor/judge 機制反向用），gemini 也獨立指出「總結 verifiable post-publication metrics 供機構 review 用」是不違反 Paper2X 的正當行政用途；②**自我 errata workflow**：發表後發現錯誤的更正流程支援；③**OA self-archiving 合規**：Sherpa Romeo 查詢、postprint 版本管理。
|
||||
|
||||
**此提案要求修改 POSITIONING 的 scope 敘述**（"research-to-publication"——publication 是終點站）。若裁定不擴，建議至少把「發表後是 deliberate non-goal」寫進 POSITIONING，理由同 F-1：留白與五條 Rejected mechanisms 的記錄紀律不對等。
|
||||
|
||||
### 5.5 F-13 grant/funding lifecycle【增補，scope 裁定；gemini 補抓】
|
||||
|
||||
本盤點初稿的生命週期走查從 RQ 孵化起跳，**整段漏掉 grant/計畫申請**——gemini 點出這是資深研究者實際行政負擔的大宗，且是 research program 的真正起點（先有計畫核定才有研究線）。repo 現況：funding_statement_guide 只處理「論文裡的 funding 聲明」，計畫書寫作（研究目的、文獻、方法、預期成果、預算敘述）零支援。
|
||||
|
||||
對使用者畫像（國科會申請）這是真實年度事件。但本盤點對它的立場有保留，跟 §9 的警告同源：**計畫書的「預期影響/預期成果」段是 impact-washing 的最高危文類**（承諾未發生的事），ARS 若進這個文類，最自然的滑坡就是「把預期影響寫到無懈可擊」。可辯護的切法：grant 的文獻段與方法段跟 ARS 現有能力（lit-review、research_architect）高度重疊，重用即可；預期成果段只做 advisory 探問不做生成。是否值得為此開 grant-mode（vs 使用者自己拿現有 modes 拼裝），送裁。我的傾向：**不開專屬 mode**，在 docs 補一頁「用現有 modes 寫計畫書的組裝指南」即可，把生成式支援明文排除。
|
||||
|
||||
---
|
||||
|
||||
## 6. 使用者畫像對位
|
||||
|
||||
### 6.1 F-10 中文線：比 SKILL.md 表面深，但止於排版【增補】
|
||||
|
||||
比預期好的部分（agent prompt 層實查）：literature_strategist 有 TSSCI/Airiti/台灣碩博士論文網檢索策略；apa7_chinese_citation_guide 有 TSSCI 期刊引用專節；formatter 有 xeCJK + TSSCI 期刊格式路由；pipeline Stage 5 指定 Source Han Serif TC。中文研究的「找文獻→寫→排版」鏈是通的。
|
||||
|
||||
缺的部分：①中文審查慣例——reviewer 5 persona 與 rubric 全以國際英文期刊為框架，台灣學報審查文化無對應；②TSSCI 投稿 workflow 無；③術語強制英文（"Academic terminology is kept in English"）對純中文社科論文是反向摩擦——台灣教育學界多數場合要求中文術語為主、英文夾注，現行規則方向相反。
|
||||
|
||||
裁定問題：投入量取決於你未來兩年的 TSSCI 投稿篇數，這是只有你知道的事實。若投，①③是寫作期就會痛的；②可以人肉。
|
||||
|
||||
### 6.2 F-3 灰色文獻：citation gate 對政策研究系統性低覆蓋【盲點】
|
||||
|
||||
四索引（S2 / OpenAlex / Crossref / arXiv）對政策研究的核心證據型態——政府報告、評鑑手冊、白皮書、法規、無 DOI 的國際組織文件——覆蓋率系統性偏低（初稿「全部查無」過度：部分 OECD/UNESCO 出版品有 DOI 可解析；codex 修正，gemini 對本條整體判 "factual and accurately assessed"）。C-V6 的精度優先設計（title-only unmatched → `unresolvable`，不 block）讓這些引用不被誤殺，這是對的；但後果是政策研究的 bibliography 大量落在 `unresolvable`，deterministic gate 的有效覆蓋率對這類研究大幅下降。**旗艦防線在維護者本人的研究類型上效力最低。**#250（gold set 缺 real-but-unindexed tuples）是同一個洞的工程面，本條是產品面：#250 說「量不到」，本條說「防不到」。
|
||||
|
||||
增補方向（成本遞增）：①`obtained_via: manual` 的灰色文獻 entry 加 structured provenance（URL + accessed_date + archive snapshot），manual 路線從「豁免」升級成「另一種可驗」；②URL 活性 + Wayback snapshot 存在性的 deterministic 檢查（script 層、無 LLM）；③接政府出版品 API（每轄區不同，維護地獄）。我的判斷：①②值得，③不值得。
|
||||
|
||||
---
|
||||
|
||||
## 7. 維護者品質 vs 使用者門檻
|
||||
|
||||
### 7.1 F-8 Opt-in 文化的暗面【盲點】
|
||||
|
||||
說公道話：49 條 lint、191 個 script、INV-* 全跑在 CI/maintainer 層，end user 零阻力。「lint 多 = 門檻高」不成立。
|
||||
|
||||
真正的門檻（數字經主 context 自查修正，codex 同向確認；gemini 在此條接受了未驗證的初稿數字，是 gemini 本輪唯一的 fact-check 失手，分歧記錄於 §11）：
|
||||
|
||||
1. **Flag 文件碎片化**：user-facing runtime flag 共 7 個，SETUP.md env 表列 5 個；`ARS_CLAIM_AUDIT`（claim audit 總開關）散在 README 行文三處、不在任何總表；`ARS_CACHE_DIR` 只在 design spec。問題本質不是數量是**沒有單一 flag 總表**。
|
||||
2. **安全功能全部 opt-in 預設關**：strict citation policy、claim audit（default OFF）、cross-model verification。新研究者照 QUICKSTART 三步裝完，拿到 advisory 後綴（detection unconditional，標記會出現，這點誠實），但**沒有任何 block**。v2.7 那次 31% 引用錯誤率換來的 deterministic gate，預設不擋任何東西。Backward-compat 紀律（byte-equivalent 升級）與安全預設在此對撞，目前一律犧牲後者。
|
||||
3. **配置疲勞**：paper full Phase 0 九項訪談；SETUP.md 408 行、5 種安裝法；新手路徑與 power-user 全貌之間無中間階梯。
|
||||
|
||||
增補形狀：①`ARS_PROFILE=strict|standard|minimal` 一鍵 profile（已確認 repo 無此名）；②單一 flag 總表進 SETUP.md（純文件工）；③裁定問題：新 user 預設要不要 citation_existence=strict？打破 byte-equivalent 慣例的取捨，只有你能裁。
|
||||
|
||||
### 7.2 F-9 Dogfooding 半斷流【盲點，時間線經 codex 修正】
|
||||
|
||||
初稿宣稱「v3.4 之後演進全為外部論文驅動、真實使用零痕跡」。**codex 打掉**：v3.6.7（04-30）的 spec 明寫源自 "v3.6.5/v3.6.6 production academic chapter run" 的 17 個 drift patterns——這正是 chapter 真實使用的 repo 痕跡（也吻合維護者 memory 中的 Springer chapter 時段）；v3.9.4.1 等版本是 codex post-ship 修正非論文驅動。
|
||||
|
||||
修正後仍成立的事實：①唯一完整 end-to-end worked example（showcase/）停在 v2.7 時代（2026-03-09），其後 **9 個 minor 版的新機制（triangulation、terminal policy、claim audit、Kong track、experiment provenance）沒有任何一個出現在完整真實 run 的 artifacts 裡**；②v3.8 之後（05-16 起）的 motivation 全是外部論文（Zhao/Co-Scientist/Kong/Kim）；③eval gold sets 幾乎全合成（citation_extraction 51 tuples 是 fabricated DOI，#250 自知；surface_form_parity 7 項中 3 項 maintainer 自寫）。
|
||||
|
||||
論文驅動買到結構化 threat model（可審計、可 lint 化），但量測的是「合成威脅下的防線」不是「真實使用中的價值」。風險走向：工具變成失敗模式文獻的博物館，而不是自己寫論文時痛處的解藥。
|
||||
|
||||
增補形狀（成本近零，改慣例不改程式）：①下一篇真論文走一次 full pipeline，產出第二個 showcase；考慮立「每 minor 至少一次 real-run smoke」的 release 慣例；②CHANGELOG 加 `Real-use findings` 慣例節，讓 lived experience 在 repo 留痕（v3.6.7 的 chapter-run 來源寫在 spec 內文深處，初稿盤點都沒挖到——這正說明回流痕跡需要一個固定位置）。
|
||||
|
||||
---
|
||||
|
||||
## 8. 誠實的 non-goal 清單
|
||||
|
||||
| 項目 | 為什麼不做 | 狀態 |
|
||||
|---|---|---|
|
||||
| 實驗執行/autonomous coding | Kong §3.3 rejected；scholar 跑、ARS 驗 provenance | 已拍板，維持 |
|
||||
| Idea generation（RQ 提案/排序/改寫） | Kong L2，cognitive ownership 論證堅實。本盤點所有 RQ 期提案都設計成問不給；F-1 經 codex 收緊為三條件 | 已拍板，維持 |
|
||||
| Paper2X auto-generation（含 plain-language summary 自動轉製） | POSITIONING rejected。**評估後不提案鬆動**：scholar-led 變體理論上可分，但它是 impact-washing 高危文類（S0 鏡像），dissemination design 已劃給 repo 外。不為它鬆動乾淨的紅線 | 維持 |
|
||||
| Authenticated crawl / paywall bypass | 2026-04-22 拍板，集體風險論證 | 維持 |
|
||||
| Data layer（外部 corpus 整合） | Passport 是唯一 input port。F-1 的 ledger 是 ARS 自產 artifacts 承接，我判斷不在射程內，邊界送裁 | 維持（F-1 邊界送裁） |
|
||||
| Primary data 統計分析 | experiment-agent 領地 | 維持 |
|
||||
| Multi-author 協作 | 修正：team_collaboration_protocol.md 已有 human-convention 層（roles/handoffs/approval 規則）+ intake 收 co-author data；零的是 passport 層 multi-author state（permissions、merge、co-author 確認）。**維持 non-goal**（單人工具的複雜度紅利就在單人），protocol 文件已足 | 維持，無需動作 |
|
||||
| Grant 預期成果段的生成式支援 | F-13 評估後明文排除（impact-washing 最高危文類）；文獻/方法段重用現有 modes | 新增，建議記錄 |
|
||||
| 發表後段、research program 層 | 若 F-7/F-1 裁定不做 → 寫進 POSITIONING non-goal，消除留白 | 待裁 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 對 AACSB 報告的立場（含對使用者的挑戰）
|
||||
|
||||
**同意且已轉化為提案的**：impact-by-design——但 dual-track 證明 ARS 的 RQ 期已經做了（F-4 撤回是好消息：理念已內建），殘餘只在寫作期延伸（F-2 形狀 1）；spiral model 的跨輪迴路（F-1 的擴 scope 理由）；external engagement 視角（F-2 形狀 3）。
|
||||
|
||||
**挑戰報告本身的**：①它是學校層級 advocacy 文件，分析單位是 institution（p.10 明說 impact 不該落在個別 faculty 肩上），直接搬到單人 copilot 是 category error，本盤點只取「個人研究線」中間層；②它引了 Campbell's Law（p.19 n.15）卻對自家 Assessment Tool 毫無防 gaming 設計——一張自填 narrative worksheet，正是「把 impact 敘事寫到無懈可擊」的邀請函。
|
||||
|
||||
**挑戰使用者的（真反駁，不是迂迴同意）**：你同意 "research that reaches"，但你的機構角色和 S0 洞見都在告訴你 reach 的敘事面有多容易造假。這份報告若被工具化，最自然的產品化路徑就是「impact statement writer」——而那正是你在評鑑端要抓的東西。本盤點刻意把 reach 的接入點全部放在源頭（RQ 期，已存在）與事實層（citation-context audit、scholar-supplied 對照集），一個都不放在敘事層；F-13 的 grant 預期成果段同理排除。gemini 對這個 refusal 的評語是 "exceptionally well-argued" 但提醒「彙整 verifiable 既成 metrics 供機構表單」是正當用途——我接受這個區分：**回顧既成事實可以，前瞻承諾不行**。如果你不同意這個切法，分歧值得開 issue 吵。
|
||||
|
||||
---
|
||||
|
||||
## 10. 只有使用者能裁的判斷（按優先序）
|
||||
|
||||
1. **F-1 research-line ledger**：①算不算 data layer 禁令射程（我判斷不算）；②要不要開設計線（最小切片：scholar-initiated limitations surface + 自我 claim audit）；③不做的話要不要寫進 POSITIONING non-goal。
|
||||
2. **F-7 發表後段要不要進 scope**：動 POSITIONING 的 identity 級決定。不做也請寫下 non-goal，消除留白。
|
||||
3. **F-8 新 user 預設要不要 strict**：打破 byte-equivalent 慣例的取捨。
|
||||
4. **F-2 做哪個形狀**：推薦先做形狀 1（socratic Layer 5 問句移植進 plan/revision coaching，最便宜）；形狀 2 重設計後價值縮水，可不做；形狀 3 便宜但價值虛。
|
||||
5. **F-13 grant 段**：開組裝指南頁（我的傾向）還是完全不碰。
|
||||
6. **F-10 中文線投入量**：取決於你的 TSSCI 投稿計畫。
|
||||
7. **F-5/F-9/F-14**：低爭議（deterministic 化方向、real-run 慣例、rubric 權重 hygiene），可直接轉 scoped issues。
|
||||
|
||||
---
|
||||
|
||||
## 11. Dual-track 紀錄（codex 0.137 high-reasoning + gemini-3.1-pro-preview，2026-06-10）
|
||||
|
||||
**兩模型獨立收斂的（最高可信）**：①F-4 前提錯誤（兩邊都引 socratic_mentor_agent Layer 1/5 原文）；②「advisory ≠ 自動紅線安全，候選集建構本身是 ranking」（codex 打 F-2 形狀 2、gemini 打 F-6，不同條目同一原則）。
|
||||
|
||||
**codex 獨有的真 catch**：F-9 時間線（v3.6.7 chapter-run 證據）、F-5 低估（formatter checklist 原文）、F-7 過度宣稱（monitoring_agent 原文）、F-1 分類與 L2 三條件、F-14 rubric 不一致、F-3 措辭軟化。本輪 codex 表現顯著優於 gemini（11 條中 9 條成立）。
|
||||
|
||||
**gemini 獨有的真 catch**：F-13 grant lifecycle 整段遺漏（codex 沒抓到）；「回顧既成 metrics vs 前瞻承諾」的正當用途區分。
|
||||
|
||||
**分歧點（cross-model 的真學習）**：F-8 數字——codex 判 factual error（與主 context 自查一致），gemini 判 factual（接受了未驗證的 14/9 數字）。同一條宣稱、兩個 verdict，誰做了 first-party grep 誰就對。又一次印證 [[feedback_ai_only_chains_fail_at_fluent_wrongness]]：cross-model 不是保險，first-party deterministic 驗證才是。
|
||||
|
||||
**初稿五個被打掉/修正的宣稱全數源自同一根因**：subagent 地圖讀到 SKILL.md/README 層、沒讀 agent prompt 層（formatter/socratic_mentor/monitoring 的細節全在 agent .md 內文），主 context 又對自寫提案 anchoring。教訓：**對 prompt-架構 repo 做產品盤點，「功能存不存在」必須查到 agent prompt 層才算數**。
|
||||
|
||||
---
|
||||
|
||||
## 12. 裁定結果（2026-06-10，使用者拍板）
|
||||
|
||||
| Finding | 裁定 | 落點 |
|
||||
|---|---|---|
|
||||
| F-1 跨篇記憶 | **B：不做機制，出文件**。使用者問「靠 Claude Code memory 行不行」→ 答：個人層可（重餵 passport + assistant memory 提醒），產品層不可（public skill 不能依賴使用者環境；ARS anti-leakage 哲學本來就不信 LLM 記憶；passport 是唯一 state of record） | **#397**（POSITIONING non-goal + cross-paper workflow guide） |
|
||||
| F-2 平庸可見性 | 做形狀 1（Layer 5 問句移植進 plan/revision coaching）；形狀 2/3 不做 | **#393**（p1） |
|
||||
| F-3 灰色文獻 | 本輪未單獨立案（①manual provenance 強化可併入未來 citation-gate 線） | 報告留檔 |
|
||||
| F-5 投稿包 | 做，design-first | **#394** |
|
||||
| F-7 發表後 | **不做**：「使用者自己會查，戰線不宜拉過長」→ 寫成 recorded non-goal | **#397** |
|
||||
| F-8 引用防線預設 | **預設不動，加配置訪談提問讓使用者選** | **#392** |
|
||||
| F-9 real-run 慣例 | 做 | **#395** |
|
||||
| F-10 TSSCI / F-13 計畫書 | **不做**：「給國際通用的 skill，用現有功能就好」 | 結案，報告留檔 |
|
||||
| F-14 rubric 權重 | 修 | **#396** |
|
||||
|
||||
---
|
||||
|
||||
## 附錄：方法與證據
|
||||
|
||||
- 錨點文本全讀：POSITIONING.md（Rejected mechanisms 五條）、Kong L1/L2 design lessons（state-authority + verb test）、Co-Scientist L1（hidden ranking）、README、docs/PERFORMANCE.md、MODE_REGISTRY.md、data layer boundary memory（2026-04-22）。
|
||||
- AACSB *A Framework for Research Impact*（May 2026）46 頁全讀：spiral model（App. B）、Assessment Tool（pp.20-23）、impact indicators（App. D）、Campbell's Law 自引（p.19 n.15）。
|
||||
- 兩路 subagent 事實提取（SKILL.md 六軸地圖 + repo 八點掃描）→ 主 context 自查修正 3 處 → codex + gemini dual-track → 本版修訂 9 處。
|
||||
- 本報告不動工程線：#330/#272/#250/#219/#89/#387 各有歸屬。
|
||||
@@ -0,0 +1,215 @@
|
||||
# Harness Retirement Audit — `academic-research-skills`
|
||||
|
||||
| | |
|
||||
|-|-|
|
||||
| Repo path | `~/Projects/academic-research-skills` |
|
||||
| Branch / commit | `main @ e48d2c2` (2026-06-09) |
|
||||
| Date | 2026-06-10 |
|
||||
| Target model | **Fable 5** (`claude-fable-5`), migrated from Opus 4.8 |
|
||||
| Files scanned | 73 prompt files (`SKILL.md` ×4, `agents/*.md` ×40, `commands/*.md` ×14, `shared/*.md` ×11+, references, hooks, `.claude/CLAUDE.md`) |
|
||||
| Auditor | `/harness-retirement` skill v0.1.0 |
|
||||
|
||||
## Executive summary
|
||||
|
||||
- **Total findings**: 9 (5 recommended accept, 3 defer, 1 cosmetic)
|
||||
- **By category**:
|
||||
|
||||
| Cat | Count | Priority |
|
||||
|-----|------:|----------|
|
||||
| 1 — Hardcoded model IDs | 4 (F-001, F-002, F-003, F-009) | high (mechanical) |
|
||||
| 2 — Anti-hallucination patches | 1 (F-008) | low — almost all candidates passed the iron rule (contract-bound, not generic) |
|
||||
| 3 — Model/sampling overrides | 1 (F-004) | **high — the single biggest Fable 5 win** |
|
||||
| 4 — Few-shot redundancy | 0 | — (no file has 3+ examples; already lean) |
|
||||
| 5 — Defensive scaffolding | 0 | — (all retry logic is HTTP-429 backoff protecting external API contracts) |
|
||||
| 6 — Negative framing | 1 category-level (F-007) | medium, next minor release |
|
||||
| — — Model-premise re-baseline | 2 (F-005, F-006) | medium (measurement, not prompt edits) |
|
||||
|
||||
- **Suggested batch order**: F-004 → F-001/F-002 → F-003 (verify OpenAI lineup first) → F-005 → F-006 (open issue) → F-007 (next minor) → F-008/F-009 (opportunistic)
|
||||
- **Overall verdict**: the repo is in unusually good harness shape. PR #346/#347 already established the "inherited session model, don't assert an id string" pattern; anti-hallucination text is contract-bound (R-CIM-D / EP-INV-3 / MATERIAL GAP protocol) rather than generic patching; few-shot and defensive-scaffolding debt is zero. The remaining debt is concentrated in **display-name drift** and **the `model: opus` floor that Fable 5 turned into a ceiling**.
|
||||
|
||||
⚠️ **Before applying ANY finding**: run the full lint suite (`python3 scripts/run_ci_pytest_manifest.py`). Several lints assert literal prose strings (e.g. `check_cross_model_verification_sync.py`, `check_version_consistency.py`). A prose edit that forgets its lint mirror will fail CI — and worse, a lint that silently stops matching is a fake-green.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### [F-001] `shared/cross_model_verification.md:28,33` — Category 1 (hardcoded model ID)
|
||||
|
||||
**Excerpt**
|
||||
```
|
||||
| Claude Opus 4.8 | _(inherited Claude Code session model)_ | Anthropic | Primary model (default for all ARS skills) |
|
||||
...
|
||||
**Recommended cross-verification pair:** Claude Opus 4.8 (primary) + GPT-5.4 Pro or Gemini 3.1 Pro (verifier).
|
||||
```
|
||||
|
||||
**Why this is debt.** The API-ID column already says *inherited session model* (the #346/#347 fix), but the display name still pins a generation. It was hand-bumped 4.7→4.8 in #347 and is now stale again under Fable 5 — proof that the display name is on a manual treadmill.
|
||||
|
||||
**Proposed change.** Make the row generation-agnostic so it never needs a bump again:
|
||||
```
|
||||
| Claude (session model) | _(inherited Claude Code session model — e.g., Fable 5)_ | Anthropic | Primary model (default for all ARS skills) |
|
||||
...
|
||||
**Recommended cross-verification pair:** the inherited Claude session model (primary) + GPT-5.x Pro or Gemini 3.x Pro (verifier).
|
||||
```
|
||||
Check `check_cross_model_verification_sync.py` for literal-string asserts before editing.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
### [F-002] `scripts/announce-ars-loaded.sh:84` — Category 1 (hardcoded model ID, injected every session)
|
||||
|
||||
**Excerpt**
|
||||
```
|
||||
Token budget reference: docs/PERFORMANCE.md (a single full pipeline run ≈ $4–6 on Opus 4.7)."
|
||||
```
|
||||
|
||||
**Why this is debt.** This is the SessionStart hook — the line lands in **every** ARS session's context. "Opus 4.7" is two generations stale, and the $4–6 estimate was measured on that model. A user on Fable 5 reads a wrong model name and a wrong cost anchor at session start, every time.
|
||||
|
||||
**Proposed change.** `≈ $4–6 per full pipeline run (order-of-magnitude; measured on Opus 4.7 — see docs/PERFORMANCE.md)` — or drop the model name entirely and let PERFORMANCE.md own the measurement provenance. Sync `docs/PERFORMANCE.md` + `.zh-TW.md` in the same PR.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
### [F-003] cross-verifier lineup drift: `gpt-5.4` vs `gpt-5.5` — Category 1
|
||||
|
||||
**Locations**
|
||||
- `shared/cross_model_verification.md` (entire file: supported-models table, env examples, the `gpt-5.4*)` case glob at :276)
|
||||
- `academic-pipeline/agents/claim_ref_alignment_audit_agent.md:66` — judge default `gpt-5.5-xhigh`
|
||||
- `shared/templates/codex_audit_multifile_template.md:263` — `gpt-5.5 + xhigh`
|
||||
- `.claude/CLAUDE.md:202` — "GPT-5.4 Pro"
|
||||
|
||||
**Why this is debt.** Internal inconsistency: the citation judge defaults to **gpt-5.5**-xhigh while the cross-model verification doc teaches **gpt-5.4** everywhere — and its `gpt-5.4*)` shell glob will *not* match a `gpt-5.5` value, so a user following the judge's default into `ARS_CROSS_MODEL` hits the unsupported-model warning. Local codex CLI already defaults to gpt-5.5.
|
||||
|
||||
**Proposed change.** Unify on the current OpenAI lineup (verify before editing: does the Responses API `web_search` tool support the 5.5 family?). Update table, env examples, case glob (`gpt-5.4*|gpt-5.5*)` or regenerate), and `.claude/CLAUDE.md:202`.
|
||||
|
||||
**Iron-rule check.** Not a blind bump — the web-search-grounding path (#346) is load-bearing. Confirm tool support on the new id first.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
### [F-004] `commands/ars-full.md`, `ars-reviewer.md`, `ars-revision-coach.md` — Category 3 (model override; floor became ceiling)
|
||||
|
||||
**Excerpt** (frontmatter, all three files)
|
||||
```yaml
|
||||
model: opus
|
||||
```
|
||||
|
||||
**Why this is debt.** v3.7.0 pinned the heavy modes to `opus` as a *floor*: "never run full/reviewer/revision-coach on something weaker than the session might be." Under Fable 5 the same pin is a **ceiling**: a Fable 5 session invoking `/ars-full` gets silently *downgraded* to Opus 4.8 for the most quality-critical paths in the suite. The three plugin agents already use `model: inherit` — the commands never got the same treatment because at v3.7.0 "opus" and "best available" were the same thing. That equivalence is what expired.
|
||||
|
||||
**Proposed change.** Two options, pick one after verifying command-frontmatter semantics:
|
||||
1. **Delete the `model:` line** in the three opus-pinned commands → command inherits the session model (safe, no syntax risk).
|
||||
2. `model: inherit` if the command frontmatter accepts it (agents do; verify for commands).
|
||||
|
||||
Leave the 11 `model: sonnet` light-mode commands alone — that's cost routing (a product decision), not a capability scaffold, and sonnet remains the right tier for abstract/outline/format-convert.
|
||||
|
||||
**Iron-rule check.** The original opus pin is documented (v3.7.0 CHANGELOG) but its premise — "opus = strongest available" — is what changed. No measurement pins the heavy modes to Opus 4.8 specifically.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
### [F-005] `tests/fixtures/issue_133_routing/README.md:21` + smoke suite — measurement gap (not a prompt edit)
|
||||
|
||||
**Excerpt**
|
||||
```
|
||||
- **100% pass on Opus 4.7** (primary model — most ARS users)
|
||||
```
|
||||
|
||||
**Why this matters.** Tests are out of edit-scope for this skill, but the acceptance *definition* ("primary model — most ARS users") has drifted two generations, and the 8 routing smoke tests have never been run on Fable 5. The README itself warns: "A test that passes on Opus 4.7 today can regress on Opus 4.8 tomorrow."
|
||||
|
||||
**Proposed action.** (a) Re-run the issue-133 routing smoke suite on a Fable 5 session; (b) reword the threshold to "100% on the current primary model (tracked in cross_model_verification.md)" so the definition stops drifting.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
### [F-006] Opus 4.8 system-card premises behind #272 / #273 / #274 — re-baseline, keep the rules
|
||||
|
||||
**Locations.** The shipped guidance (`shared/ground_truth_isolation_pattern.md` §2A, editorial-synthesizer concise-output discipline, same-family calibration note) — none of it hardcodes "4.8" in prompt text (good design; the premise lives in issues/CHANGELOG).
|
||||
|
||||
**Why this matters.** All three rules were written against **Opus 4.8 system-card findings** (indirect-prompt-injection regression §4.1.4 over-caveat / multi-turn concession, rubric-aware judging §6.3.7). Fable 5's behavioral profile is different. The rules themselves are keep-worthy regardless (security depth, concision, epistemic honesty are model-agnostic virtues), but the **priority arguments** built on 4.8 regressions need re-baselining — in particular #272's runtime-enforcement urgency, which should be re-scored against the Fable 5 system card when available.
|
||||
|
||||
**Proposed action.** Open a small tracking issue: "Re-baseline model-behavior premises (#272/#273/#274) against the Fable 5 system card." No prompt edits.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
### [F-007] Negative-framing density — Category 6 (category-level, next minor release)
|
||||
|
||||
**Top files** (count of `don't / do not / never / avoid / must not`):
|
||||
```
|
||||
41 academic-pipeline/agents/pipeline_orchestrator_agent.md
|
||||
33 deep-research/agents/socratic_mentor_agent.md
|
||||
28 academic-pipeline/agents/integrity_verification_agent.md
|
||||
26 academic-paper/agents/literature_strategist_agent.md
|
||||
26 academic-paper-reviewer/agents/editorial_synthesizer_agent.md
|
||||
```
|
||||
|
||||
**Why this is (partial) debt.** Most hits are hard gate/contract boundaries ("do not abort Phase 1, do not attempt schema repair") where the negative form is correct per the iron rule. But at 41 negatives in one agent, a fraction will be reframeable as shorter positive directives, which current models follow more reliably.
|
||||
|
||||
**Proposed action.** Not a bulk rewrite. At the next minor release, run a positive-reframe pass over the top 3 files only, one small PR each, with the full lint suite green per file (several lints pattern-match exact prose). Keep every negative that is a hard boundary.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
### [F-008] `shared/compliance_checkpoint_protocol.md:74` + `shared/agents/compliance_agent.md:113` — Category 2 (bare anti-hallucination tail)
|
||||
|
||||
**Excerpt**
|
||||
```
|
||||
... Apply [Anti-Leakage Protocol](...) — do not hallucinate.
|
||||
... mark `[MATERIAL GAP]`, item auto-FAILs, tier dictates block/warn. Never hallucinate.
|
||||
```
|
||||
|
||||
**Why this is (marginal) debt.** Both sentences sit at the end of a fully-specified MATERIAL GAP protocol that already defines the positive behavior (mark the gap, auto-FAIL, tier decides). The bare "never hallucinate" tail adds no instruction the protocol doesn't.
|
||||
|
||||
**Iron-rule check.** Academic compliance is high-stakes and the failure is silent — the textbook keep case. **Recommend defer** (annotate as known debt) rather than delete; revisit only with calibration evidence that the tail is inert.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [x] defer (annotate: high-stakes domain, silent failure; delete only with measurement)
|
||||
|
||||
---
|
||||
|
||||
### [F-009] `academic-paper/references/disclosure_mode_protocol.md:114` — Category 1 (cosmetic)
|
||||
|
||||
**Excerpt**
|
||||
```
|
||||
Replace `[MODEL_VERSION]` with the actual model used in this run (e.g., `Opus 4.7`, `Sonnet 4.6`).
|
||||
```
|
||||
|
||||
**Why this is (cosmetic) debt.** The file already implements the correct placeholder pattern — the *e.g.* list is just stale and may anchor users to copy old names. Refresh to `(e.g., Fable 5, Sonnet 4.6)` opportunistically when the file is next touched.
|
||||
|
||||
**Decision** — [ ] accept [ ] reject [ ] defer
|
||||
|
||||
---
|
||||
|
||||
## Kept as debt (iron rule filtered out — do not resurface next audit)
|
||||
|
||||
- `shared/cross_model_verification.md:196,235` — `temperature: 0.1` on GPT/Gemini verifier calls. **Kept**: documented rationale at :261 (deterministic factual task; variance reduction), and it parameterizes non-Claude models.
|
||||
- `deep-research/references/{openalex,semantic_scholar,arxiv,crossref}_api_protocol.md` — HTTP-429 backoff/retry. **Kept**: protects external API contracts, not a model scaffold.
|
||||
- R-CIM-D "do NOT invent ids or rename" (draft_writer :591, synthesis :354, report_compiler :341). **Kept**: schema contract (EP-INV-3), definitional not defensive.
|
||||
- `visualization_agent.md:441`, `risk_of_bias_agent.md:33`, `pipeline_orchestrator_agent.md:555`, `collaboration_depth_agent.md:75`, `synthesis_agent.md:162`, `literature_corpus_consumers.md:107` — "do not invent X" with positive counterpart in the same clause. **Kept**: negative + positive reinforce each other (Cat 6 counter-example).
|
||||
- `commands/*.md` `model: sonnet` ×11 — cost routing, product decision.
|
||||
- `deep-research/agents/*` `model: inherit` ×3 — already the target pattern.
|
||||
- `tests/fixtures/issue_133_routing/` Opus 4.7 references *inside test fixtures* — records/pins, out of scope (the README threshold definition is F-005, separate).
|
||||
|
||||
## Apply log (2026-06-10 apply turn)
|
||||
|
||||
| Finding | Action | Notes | Verified |
|
||||
|---------|--------|-------|----------|
|
||||
| F-001 | accepted → applied | primary row generation-agnostic; gpt-5.4* kept accepted | lint suite 43/43 green |
|
||||
| F-002 | accepted → applied | announce + PERFORMANCE ×2 provenance-labelled | lint green |
|
||||
| F-003 | accepted → applied | gpt-5.5 ($5/$30) / gpt-5.5-pro ($30/$180) verified first-party (developers.openai.com, 2026-06-10); web_search-on-Responses confirmed; case glob `gpt-5.5*\|gpt-5.4*` | lint green |
|
||||
| F-004 | accepted → applied | `model:` line deleted on 3 heavy commands; body "uses opus" sentences synced | grep: zero `model: opus` residual |
|
||||
| F-005 | accepted → applied + run | threshold reworded; **Fable 5 smoke: 8/8 routing-class PASS** (01–08, incl. all 3 escape-hatch behaviors); 02/04 destination picks needed Routing-Rules/MODE_REGISTRY context (harness approximation, resolved on re-run with full rules) | subagent transcripts 2026-06-10 |
|
||||
| F-006 | accepted → issue drafted | `audits/issues-to-file-2026-06-10.md` (gh issue create blocked by session permission; command ready) | — |
|
||||
| F-007 | deferred | next minor; top-3 files sample-reframe, one PR each | — |
|
||||
| F-008 | deferred (annotated) | in-file `harness-retirement` annotations added at both sites | lint green |
|
||||
| F-009 | accepted → applied | e.g. list refreshed | — |
|
||||
|
||||
## Next audit
|
||||
|
||||
- Suggested: next minor release, or when the Fable 5 system card publishes (re-check F-006)
|
||||
- Carry forward: F-008 annotation; re-verify `model: sonnet` cost routing still matches the lineup
|
||||
@@ -1,6 +1,6 @@
|
||||
# ARS #394 — Deterministic submission-package verifier (design-first)
|
||||
|
||||
**Status**: DESIGN — no implementation in this PR; slices follow per §9.
|
||||
**Status**: DESIGN — slices ship per §9. Slice 1 (CLI skeleton + Family C + report schema + fixtures) SHIPPED; slices 2–4 follow.
|
||||
**Issue**: #394 (blindspot-audit F-5, adjudicated design-first).
|
||||
**Decision trail**: 2026-06-10 researcher-blindspot audit; cross-model review corrected the initial claim — the formatter's prompt-layer checklist already exists; the gap is deterministic enforcement, not absence.
|
||||
|
||||
@@ -97,7 +97,7 @@ All fields nullable: a partially-declared profile runs the checks it can and `NO
|
||||
|
||||
1. **Standalone CLI (always available):** `python scripts/verify_submission_package.py <package_dir> [--venue-profile profile.yaml] [--passport passport.yaml]` → human-readable report + JSON (`submission_verification_report.json`). Works with zero pipeline context — a scholar can point it at any folder before any submission.
|
||||
2. **Pipeline hook (Stage 5 FINALIZE, post-formatter) — explicitly a NEW package-level gate, not the ref-marker stamp path:** the v3.10 terminality machinery is finalizer-stamped ref markers + the formatter's stamp-only rule 11; this verifier runs *after* the formatter on the whole package, so that carrier cannot serve it. Instead the report itself is the evaluated carrier: it embeds a `package_fingerprint` (manifest of file hashes) and the policy slug in force at evaluation, and the orchestrator MUST NOT reuse a report whose fingerprint or slug no longer matches (the freshness guard, package-level analog of the `policy_hash` stamp). Advisory results append to `provenance_summary.md` (`Submission Package Advisories`); under `terminal_policies.submission_package: strict`, a strict-eligible `fail` returns the package to the formatter fix loop (bounded: 2 fix rounds, then surface to the scholar — mirroring the revision-loop cap philosophy) instead of emitting. **Strict fails closed on incompleteness:** a strict-eligible check that reports `NOT-CHECKED` under strict is `VERIFICATION-INCOMPLETE` and blocks emission exactly like a `fail` — otherwise a missing parser silently waives the one check class the scholar opted into blocking on (the fail-open hole). Advisory default is unaffected (`NOT-CHECKED` is surfaced, never blocking).
|
||||
3. **Policy evaluation stays single-homed:** the orchestrator (finalizer side) decides terminality by reading `terminal_policies`; the script only reports per-check `{pass, fail, NOT-CHECKED}` + class `{deterministic, heuristic}`. The script never reads `terminal_policies` — same division as #182's gate (detection unconditional, terminality decided by the policy evaluator).
|
||||
3. **Policy evaluation stays single-homed:** the orchestrator (finalizer side) decides terminality by reading `terminal_policies`; the script only reports per-check `{pass, fail, warn, NOT-CHECKED}` + class `{deterministic, heuristic}` (slice-1 reconciliation: §3.3 already assigns `warn` to the uncited-reference case, so the status set here carries it too — `warn` is advisory-only and never policy-promotable). The script never reads `terminal_policies` — same division as #182's gate (detection unconditional, terminality decided by the policy evaluator).
|
||||
4. **No ref-marker change:** nothing in this design touches the v3.7.3 marker grammar; the carrier is the report file + `provenance_summary.md` section (the #333 precedent for "advisory needs a home but the marker slot is taken").
|
||||
|
||||
## 6. Boundary
|
||||
@@ -136,7 +136,7 @@ Advisory-only through slice 3; nothing blocks until slice 4 lands the policy key
|
||||
|
||||
1. zh-TW self-citation phrasing list (A5) needs first-party curation — no anglophone-only pattern list.
|
||||
2. Family D adjudication (§3.4): presence/shape check vs leaving `repro_lock` fully out — maintainer call at slice 3.
|
||||
3. Whether the report's `package_fingerprint` should reuse the audit-snapshot hashing convention or a plain file manifest — decide at slice 1.
|
||||
3. Whether the report's `package_fingerprint` should reuse the audit-snapshot hashing convention or a plain file manifest — decide at slice 1. **ADJUDICATED (slice 1):** reuse the audit-snapshot manifest convention (`scripts/audit_snapshot.py` `write_manifest`), adapted to package level: one `<package-relative-path>:<sha256>` line per file, LC_ALL=C byte-sorted, newline-joined with a trailing newline; the fingerprint is the SHA-256 of that manifest text. The report file itself is excluded (it cannot fingerprint its own bytes). Pinned by an independent reimplementation in `scripts/test_verify_submission_package.py`.
|
||||
4. LaTeX word counting (`texcount` vs detex-and-count) — declare the method, don't promise venue-exact numbers.
|
||||
|
||||
## 11. Ship gate + definition of done (per slice)
|
||||
|
||||
@@ -207,3 +207,7 @@ path = "scripts/test_check_rubric_weight_consistency.py"
|
||||
[[pytest]]
|
||||
id = "392-citation-verification-intake"
|
||||
path = "scripts/test_check_392_citation_verification_intake.py"
|
||||
|
||||
[[pytest]]
|
||||
id = "394-submission-package-verifier"
|
||||
path = "scripts/test_verify_submission_package.py"
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
# Quality Assurance Feedback Loops in Higher Education
|
||||
|
||||
## Introduction
|
||||
|
||||
Prior work established the feedback-loop framing (Smith, 2024) <!--ref:smith2024--><!--anchor:page:12-->.
|
||||
A complementary measurement lens appears in Chen and Lee (2023) <!--ref:chenlee2023 ok--><!--anchor:section:3.2-->.
|
||||
|
||||
## Method
|
||||
|
||||
We follow the protocol of Smith (2024) <!--ref:smith2024--><!--anchor:page:14-->.
|
||||
|
||||
## References
|
||||
|
||||
Chen, W., & Lee, M. (2023). Measuring assurance of learning. *Journal of Quality Assurance*, 12(3), 45–67.
|
||||
|
||||
Smith, J. (2024). Feedback loops in institutional accreditation. *Higher Education Policy*, 37(1), 1–22.
|
||||
@@ -0,0 +1,13 @@
|
||||
@article{smith2024,
|
||||
author = {Smith, Jordan},
|
||||
year = {2024},
|
||||
title = {Feedback loops in institutional accreditation},
|
||||
journal = {Higher Education Policy},
|
||||
}
|
||||
|
||||
@article{chenlee2023,
|
||||
author = {Chen, Wei and Lee, Mina},
|
||||
year = {2023},
|
||||
title = {Measuring assurance of learning},
|
||||
journal = {Journal of Quality Assurance},
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
# Post-converted manuscript without ARS markers
|
||||
|
||||
Smith (2024) established the feedback-loop framing. The measurement lens
|
||||
follows (Chen & Lee, 2023). One claim cites a source absent from the
|
||||
reference list (Nowhere, 2020).
|
||||
|
||||
## References
|
||||
|
||||
Chen, W., & Lee, M. (2023). Measuring assurance of learning. *Journal of Quality Assurance*, 12(3), 45–67.
|
||||
|
||||
Smith, J. (2024). Feedback loops in institutional accreditation. *Higher Education Policy*, 37(1), 1–22.
|
||||
@@ -0,0 +1,13 @@
|
||||
@article{smith2024,
|
||||
author = {Smith, Jordan},
|
||||
year = {2024},
|
||||
title = {Feedback loops in institutional accreditation},
|
||||
journal = {Higher Education Policy},
|
||||
}
|
||||
|
||||
@article{chenlee2023,
|
||||
author = {Chen, Wei and Lee, Mina},
|
||||
year = {2023},
|
||||
title = {Measuring assurance of learning},
|
||||
journal = {Journal of Quality Assurance},
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
\documentclass{article}
|
||||
\begin{document}
|
||||
|
||||
The framing follows \citet{smith2024}, with measurement detail in
|
||||
\citep[see][p.~45]{chenlee2023}. One claim cites a key that never made the
|
||||
bibliography~\cite{ghost2024}.
|
||||
|
||||
\bibliography{references}
|
||||
\end{document}
|
||||
@@ -0,0 +1,13 @@
|
||||
@article{smith2024,
|
||||
author = {Smith, Jordan},
|
||||
year = {2024},
|
||||
title = {Feedback loops in institutional accreditation},
|
||||
journal = {Higher Education Policy},
|
||||
}
|
||||
|
||||
@article{chenlee2023,
|
||||
author = {Chen, Wei and Lee, Mina},
|
||||
year = {2023},
|
||||
title = {Measuring assurance of learning},
|
||||
journal = {Journal of Quality Assurance},
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
# Manuscript with markers but no join source
|
||||
|
||||
The framing follows Smith (2024) <!--ref:smith-feedback-2024--><!--anchor:page:12-->.
|
||||
|
||||
## References
|
||||
|
||||
Smith, J. (2024). Feedback loops in institutional accreditation. *Higher Education Policy*, 37(1), 1–22.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Manuscript with an orphaned in-text citation
|
||||
|
||||
The framing follows Smith (2024) <!--ref:smith2024--><!--anchor:page:12-->.
|
||||
A claim cites a reference that never made the list (Ghost, 2024) <!--ref:ghost2024--><!--anchor:none:-->.
|
||||
|
||||
## References
|
||||
|
||||
Smith, J. (2024). Feedback loops in institutional accreditation. *Higher Education Policy*, 37(1), 1–22.
|
||||
@@ -0,0 +1,6 @@
|
||||
@article{smith2024,
|
||||
author = {Smith, Jordan},
|
||||
year = {2024},
|
||||
title = {Feedback loops in institutional accreditation},
|
||||
journal = {Higher Education Policy},
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
# A passport carrying a declared literature_corpus[] but NO
|
||||
# citation_verification_summary[] — supplies a reference list, not a join.
|
||||
literature_corpus:
|
||||
- citation_key: smith2024
|
||||
title: Feedback loops in institutional accreditation
|
||||
year: 2024
|
||||
authors:
|
||||
- family: Smith
|
||||
given: Jordan
|
||||
@@ -0,0 +1,16 @@
|
||||
# A passport whose citation_verification_summary[] carries the REAL prose join
|
||||
# (ref_slug differs from citation_key, proving the join is consumed rather than
|
||||
# an identity guess) plus the literature_corpus[] reference list.
|
||||
citation_verification_summary:
|
||||
- citation_key: smith2024
|
||||
ref_slug: smith-feedback-2024
|
||||
lookup_verified: "true"
|
||||
anchor_present: true
|
||||
verification_timestamp: "2026-06-10T00:00:00+00:00"
|
||||
literature_corpus:
|
||||
- citation_key: smith2024
|
||||
title: Feedback loops in institutional accreditation
|
||||
year: 2024
|
||||
authors:
|
||||
- family: Smith
|
||||
given: Jordan
|
||||
@@ -0,0 +1,7 @@
|
||||
# Manuscript whose prose slugs differ from the corpus citation_keys
|
||||
|
||||
The framing follows Smith (2024) <!--ref:smith-feedback-2024 ok--><!--anchor:page:12-->.
|
||||
|
||||
## References
|
||||
|
||||
Smith, J. (2024). Feedback loops in institutional accreditation. *Higher Education Policy*, 37(1), 1–22.
|
||||
@@ -0,0 +1,9 @@
|
||||
# Manuscript with an uncited reference entry
|
||||
|
||||
Only one source is actually cited (Smith, 2024) <!--ref:smith2024--><!--anchor:page:12-->.
|
||||
|
||||
## References
|
||||
|
||||
Chen, W., & Lee, M. (2023). Measuring assurance of learning. *Journal of Quality Assurance*, 12(3), 45–67.
|
||||
|
||||
Smith, J. (2024). Feedback loops in institutional accreditation. *Higher Education Policy*, 37(1), 1–22.
|
||||
@@ -0,0 +1,13 @@
|
||||
@article{smith2024,
|
||||
author = {Smith, Jordan},
|
||||
year = {2024},
|
||||
title = {Feedback loops in institutional accreditation},
|
||||
journal = {Higher Education Policy},
|
||||
}
|
||||
|
||||
@article{chenlee2023,
|
||||
author = {Chen, Wei and Lee, Mina},
|
||||
year = {2023},
|
||||
title = {Measuring assurance of learning},
|
||||
journal = {Journal of Quality Assurance},
|
||||
}
|
||||
@@ -0,0 +1,394 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for verify_submission_package — #394 Slice 1 (CLI skeleton + Family C).
|
||||
|
||||
Spec: docs/design/2026-06-10-394-submission-package-verifier-spec.md §3.3 / §5.1
|
||||
/ §7.3 / §8. Mutation discipline per repo convention: every check has a fixture
|
||||
that fails it and a test proving the failure fires.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import jsonschema
|
||||
import pytest
|
||||
|
||||
from verify_submission_package import run
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
FIXTURES = REPO_ROOT / "scripts" / "fixtures" / "submission_package"
|
||||
SCHEMA_PATH = (
|
||||
REPO_ROOT / "shared" / "contracts" / "submission"
|
||||
/ "submission_verification_report.schema.json"
|
||||
)
|
||||
REPORT_BASENAME = "submission_verification_report.json"
|
||||
|
||||
|
||||
def load_schema():
|
||||
return json.loads(SCHEMA_PATH.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def run_dir(package_dir, extra_args=()):
|
||||
"""Run the CLI on a package dir; returns (exit_code, report_dict)."""
|
||||
rc = run([str(package_dir), *extra_args])
|
||||
report_path = package_dir / REPORT_BASENAME
|
||||
report = (
|
||||
json.loads(report_path.read_text(encoding="utf-8"))
|
||||
if report_path.is_file() else None
|
||||
)
|
||||
return rc, report
|
||||
|
||||
|
||||
def run_on(fixture_name, tmp_path, extra_args=()):
|
||||
"""Copy a fixture package into tmp and run the CLI on the copy.
|
||||
|
||||
Returns (exit_code, report_dict, package_dir). The copy keeps the repo
|
||||
fixture pristine (the CLI writes its report into the package dir).
|
||||
"""
|
||||
package_dir = tmp_path / fixture_name
|
||||
shutil.copytree(FIXTURES / fixture_name, package_dir)
|
||||
rc, report = run_dir(package_dir, extra_args)
|
||||
return rc, report, package_dir
|
||||
|
||||
|
||||
def checks_by_id(report):
|
||||
return {c["id"]: c for c in report["checks"]}
|
||||
|
||||
|
||||
# --- Round 1: clean package, joined marker path -----------------------------
|
||||
|
||||
def test_clean_package_all_pass_exit_0(tmp_path):
|
||||
rc, report, _ = run_on("clean", tmp_path)
|
||||
assert rc == 0
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["status"] == "pass"
|
||||
assert by_id["C2"]["status"] == "pass"
|
||||
assert report["header"]["not_checked_count"] == 0
|
||||
|
||||
|
||||
def test_clean_package_is_deterministic_joined_marker(tmp_path):
|
||||
_, report, _ = run_on("clean", tmp_path)
|
||||
assert report["header"]["extraction_path"] == "joined_marker"
|
||||
for c in report["checks"]:
|
||||
assert c["family"] == "reference_integrity"
|
||||
assert c["signal_class"] == "deterministic"
|
||||
# strict_eligible is class-level: C1 promotable, C2 (warn-only) never —
|
||||
# asserted in test_C2_is_never_strict_eligible.
|
||||
|
||||
|
||||
def test_clean_report_validates_against_schema(tmp_path):
|
||||
_, report, _ = run_on("clean", tmp_path)
|
||||
jsonschema.validate(report, load_schema())
|
||||
|
||||
|
||||
def test_policy_slug_is_null_in_standalone_runs(tmp_path):
|
||||
# §5.2/§5.3: the script never reads terminal_policies; the slug is stamped
|
||||
# by the slice-4 orchestrator hook. A standalone run always emits null.
|
||||
_, report, _ = run_on("clean", tmp_path)
|
||||
assert report["header"]["policy_slug"] is None
|
||||
|
||||
|
||||
def test_report_written_into_package_dir(tmp_path):
|
||||
_, _, package_dir = run_on("clean", tmp_path)
|
||||
assert (package_dir / REPORT_BASENAME).is_file()
|
||||
|
||||
|
||||
# --- Round 2: fail / warn / NOT-CHECKED paths + exit codes -------------------
|
||||
|
||||
def test_orphan_intext_citation_fails_C1_exit_1(tmp_path):
|
||||
rc, report, _ = run_on("orphan_intext", tmp_path)
|
||||
assert rc == 1
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["status"] == "fail"
|
||||
assert "ghost2024" in by_id["C1"]["detail"]
|
||||
assert by_id["C1"]["location"] == "paper.md"
|
||||
# The orphan is deterministic-classed on the joined marker path.
|
||||
assert by_id["C1"]["signal_class"] == "deterministic"
|
||||
assert by_id["C1"]["strict_eligible"] is True
|
||||
jsonschema.validate(report, load_schema())
|
||||
|
||||
|
||||
def test_uncited_reference_entry_warns_C2_exit_0(tmp_path):
|
||||
# §3.3: uncited reference entry = warn (some venues allow further-reading
|
||||
# entries) — advisory, so the exit code stays 0.
|
||||
rc, report, _ = run_on("uncited_reference", tmp_path)
|
||||
assert rc == 0
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["status"] == "pass"
|
||||
assert by_id["C2"]["status"] == "warn"
|
||||
assert "chenlee2023" in by_id["C2"]["detail"]
|
||||
|
||||
|
||||
def test_markers_without_join_source_not_checked_exit_3(tmp_path):
|
||||
# §3.3 + §8 join test: markers present, passport supplies a corpus (a
|
||||
# reference list) but NO citation_verification_summary — never a guessed
|
||||
# comparison.
|
||||
passport = FIXTURES / "passports" / "corpus_only.yaml"
|
||||
rc, report, _ = run_on("marker_no_join", tmp_path,
|
||||
extra_args=["--passport", str(passport)])
|
||||
assert rc == 3
|
||||
by_id = checks_by_id(report)
|
||||
for cid in ("C1", "C2"):
|
||||
assert by_id[cid]["status"] == "not_checked"
|
||||
assert "missing prose-reference join" in by_id[cid]["detail"]
|
||||
assert report["header"]["not_checked_count"] == 2
|
||||
assert report["header"]["extraction_path"] == "none"
|
||||
jsonschema.validate(report, load_schema())
|
||||
|
||||
|
||||
def test_join_map_resolves_the_no_join_case(tmp_path):
|
||||
# The explicit scholar-supplied join map is a valid join source (§3.3) and
|
||||
# joins the prose slug to the corpus citation_key.
|
||||
passport = FIXTURES / "passports" / "corpus_only.yaml"
|
||||
join = tmp_path / "join.yaml"
|
||||
join.write_text("smith-feedback-2024: smith2024\n", encoding="utf-8")
|
||||
rc, report, _ = run_on(
|
||||
"marker_no_join", tmp_path,
|
||||
extra_args=["--passport", str(passport), "--join-map", str(join)])
|
||||
assert rc == 0
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["status"] == "pass"
|
||||
assert by_id["C2"]["status"] == "pass"
|
||||
assert report["header"]["extraction_path"] == "joined_marker"
|
||||
|
||||
|
||||
def test_missing_package_dir_is_usage_error(tmp_path):
|
||||
assert run([str(tmp_path / "does-not-exist")]) == 2
|
||||
|
||||
|
||||
def test_unparseable_passport_is_usage_error(tmp_path):
|
||||
|
||||
bad = tmp_path / "bad.yaml"
|
||||
bad.write_text("just a string\n", encoding="utf-8")
|
||||
package = tmp_path / "pkg"
|
||||
package.mkdir()
|
||||
(package / "paper.md").write_text("# x\n", encoding="utf-8")
|
||||
assert run([str(package), "--passport", str(bad)]) == 2
|
||||
|
||||
|
||||
# --- Round 3: fallback extraction, summary join, fingerprint -----------------
|
||||
|
||||
def test_fallback_latex_cite_extraction_is_heuristic_best_effort(tmp_path):
|
||||
# §3.3: post-converted sources fall back to \cite{} extraction; the header
|
||||
# downgrades to best-effort and the whole path is heuristic-classed
|
||||
# (advisory-only) — even a true orphan fail is NOT strict-eligible.
|
||||
rc, report, _ = run_on("fallback_latex", tmp_path)
|
||||
assert rc == 1
|
||||
assert report["header"]["extraction_path"] == "best_effort"
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["status"] == "fail"
|
||||
assert "ghost2024" in by_id["C1"]["detail"]
|
||||
assert "smith2024" not in by_id["C1"]["detail"]
|
||||
for cid in ("C1", "C2"):
|
||||
assert by_id[cid]["signal_class"] == "heuristic"
|
||||
assert by_id[cid]["strict_eligible"] is False
|
||||
assert by_id["C2"]["status"] == "pass"
|
||||
jsonschema.validate(report, load_schema())
|
||||
|
||||
|
||||
def test_fallback_authoryear_extraction_matches_bib_metadata(tmp_path):
|
||||
rc, report, _ = run_on("fallback_authoryear", tmp_path)
|
||||
assert rc == 1
|
||||
assert report["header"]["extraction_path"] == "best_effort"
|
||||
by_id = checks_by_id(report)
|
||||
# Only the unmatched (Nowhere, 2020) is an orphan; Smith (2024) narrative
|
||||
# and (Chen & Lee, 2023) parenthetical both join to bib metadata.
|
||||
assert by_id["C1"]["status"] == "fail"
|
||||
assert "nowhere" in by_id["C1"]["detail"].lower()
|
||||
assert "smith" not in by_id["C1"]["detail"].lower()
|
||||
assert "chen" not in by_id["C1"]["detail"].lower()
|
||||
# Both bib entries were cited, so C2 passes — and the references section
|
||||
# itself was not scanned as in-text prose.
|
||||
assert by_id["C2"]["status"] == "pass"
|
||||
assert by_id["C1"]["signal_class"] == "heuristic"
|
||||
|
||||
|
||||
def test_summary_join_consumes_real_prose_join(tmp_path):
|
||||
# The prose slug (smith-feedback-2024) differs from the citation_key
|
||||
# (smith2024): a pass proves the citation_verification_summary join was
|
||||
# consumed, not an identity guess (§3.3).
|
||||
passport = FIXTURES / "passports" / "summary_join.yaml"
|
||||
rc, report, _ = run_on("summary_join", tmp_path,
|
||||
extra_args=["--passport", str(passport)])
|
||||
assert rc == 0
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["status"] == "pass"
|
||||
assert by_id["C2"]["status"] == "pass"
|
||||
assert report["header"]["extraction_path"] == "joined_marker"
|
||||
for cid in ("C1", "C2"):
|
||||
assert by_id[cid]["signal_class"] == "deterministic"
|
||||
|
||||
|
||||
def test_no_machine_readable_reference_list_not_checked(tmp_path):
|
||||
package = tmp_path / "pkg"
|
||||
package.mkdir()
|
||||
(package / "paper.md").write_text(
|
||||
"Smith (2024) said things.\n", encoding="utf-8")
|
||||
rc = run([str(package)])
|
||||
report = json.loads(
|
||||
(package / REPORT_BASENAME).read_text(encoding="utf-8"))
|
||||
assert rc == 3
|
||||
for c in report["checks"]:
|
||||
assert c["status"] == "not_checked"
|
||||
assert "no machine-readable reference list" in c["detail"]
|
||||
|
||||
|
||||
def test_fingerprint_follows_audit_snapshot_convention_excluding_report(tmp_path):
|
||||
# §10 open item 3 (adjudicated at slice 1): `<relative-path>:<sha256>`
|
||||
# lines, byte-sorted, trailing newline, fingerprint = sha256 of the
|
||||
# manifest text; the report file itself is excluded. Pinned here by an
|
||||
# independent reimplementation.
|
||||
import hashlib
|
||||
|
||||
_, report, package_dir = run_on("clean", tmp_path)
|
||||
lines = []
|
||||
for p in sorted(package_dir.rglob("*")):
|
||||
if not p.is_file() or p.name == REPORT_BASENAME:
|
||||
continue
|
||||
digest = hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
lines.append(f"{p.relative_to(package_dir).as_posix()}:{digest}")
|
||||
lines.sort()
|
||||
expected = hashlib.sha256(
|
||||
("\n".join(lines) + "\n").encode("utf-8")).hexdigest()
|
||||
assert report["header"]["package_fingerprint"] == expected
|
||||
|
||||
|
||||
def test_fingerprint_stable_across_reruns_with_report_present(tmp_path):
|
||||
# Second run sees the first run's report inside the package dir; the
|
||||
# exclusion keeps the fingerprint stable (freshness guard usable, §5.2).
|
||||
|
||||
_, first, package_dir = run_on("clean", tmp_path)
|
||||
run([str(package_dir)])
|
||||
second = json.loads(
|
||||
(package_dir / REPORT_BASENAME).read_text(encoding="utf-8"))
|
||||
assert (second["header"]["package_fingerprint"]
|
||||
== first["header"]["package_fingerprint"])
|
||||
|
||||
|
||||
# --- Codex review round: P1 partial-join identity guess + P2s ----------------
|
||||
|
||||
def test_partial_summary_join_never_falls_back_to_identity(tmp_path):
|
||||
# P1: a marker slug ABSENT from the join source must never be compared via
|
||||
# an identity guess — even (especially) when the slug coincidentally equals
|
||||
# a citation_key in the reference list (§3.3 "never a guessed comparison").
|
||||
package = tmp_path / "pkg"
|
||||
package.mkdir()
|
||||
(package / "paper.md").write_text(
|
||||
"Joined (Smith, 2024) <!--ref:smith-feedback-2024-->.\n"
|
||||
"Unjoined but key-shaped (Smith, 2024) <!--ref:smith2024-->.\n",
|
||||
encoding="utf-8")
|
||||
passport = FIXTURES / "passports" / "summary_join.yaml"
|
||||
rc = run([str(package), "--passport", str(passport)])
|
||||
report = json.loads(
|
||||
(package / REPORT_BASENAME).read_text(encoding="utf-8"))
|
||||
assert rc == 1
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["status"] == "fail"
|
||||
assert "no join entry" in by_id["C1"]["detail"]
|
||||
assert "smith2024" in by_id["C1"]["detail"]
|
||||
|
||||
|
||||
def test_C2_is_never_strict_eligible(tmp_path):
|
||||
# P2: C2's worst outcome is warn, which is advisory-only and never
|
||||
# policy-promotable (§5.3) — so the check itself is not strict-eligible,
|
||||
# even on the deterministic path.
|
||||
_, report, _ = run_on("clean", tmp_path)
|
||||
by_id = checks_by_id(report)
|
||||
assert by_id["C1"]["strict_eligible"] is True
|
||||
assert by_id["C2"]["strict_eligible"] is False
|
||||
|
||||
|
||||
def test_custom_report_out_inside_package_excluded_from_fingerprint(tmp_path):
|
||||
# P2: a --report-out path inside the package must be excluded from the
|
||||
# fingerprint like the default basename, or reruns self-reference.
|
||||
|
||||
package = tmp_path / "clean"
|
||||
shutil.copytree(FIXTURES / "clean", package)
|
||||
out = package / "custom_report.json"
|
||||
run([str(package), "--report-out", str(out)])
|
||||
first = json.loads(out.read_text(encoding="utf-8"))
|
||||
run([str(package), "--report-out", str(out)])
|
||||
second = json.loads(out.read_text(encoding="utf-8"))
|
||||
assert (first["header"]["package_fingerprint"]
|
||||
== second["header"]["package_fingerprint"])
|
||||
|
||||
|
||||
def test_authoryear_fallback_tolerates_page_locators(tmp_path):
|
||||
# P3: `Smith (2024, p. 12)` / `(Chen & Lee, 2023, pp. 45–67)` are common
|
||||
# locator forms; missing them creates avoidable fallback false orphans.
|
||||
package = tmp_path / "pkg"
|
||||
package.mkdir()
|
||||
(package / "paper.md").write_text(
|
||||
"Smith (2024, p. 12) framed it; details follow "
|
||||
"(Chen & Lee, 2023, pp. 45–67).\n", encoding="utf-8")
|
||||
shutil.copy(FIXTURES / "fallback_authoryear" / "references.bib",
|
||||
package / "references.bib")
|
||||
rc = run([str(package)])
|
||||
report = json.loads(
|
||||
(package / REPORT_BASENAME).read_text(encoding="utf-8"))
|
||||
by_id = checks_by_id(report)
|
||||
assert rc == 0
|
||||
assert by_id["C1"]["status"] == "pass"
|
||||
assert by_id["C2"]["status"] == "pass"
|
||||
|
||||
|
||||
def test_schema_rejects_warn_with_strict_eligible():
|
||||
# P2: warn is advisory-only and never policy-promotable — tightened
|
||||
# structurally like the heuristic exclusion.
|
||||
bad = _minimal_report(status="warn", strict_eligible=True)
|
||||
with pytest.raises(jsonschema.ValidationError):
|
||||
jsonschema.validate(bad, load_schema())
|
||||
ok = _minimal_report(status="warn", strict_eligible=False)
|
||||
jsonschema.validate(ok, load_schema())
|
||||
|
||||
|
||||
# --- Report schema structural contract --------------------------------------
|
||||
|
||||
def _minimal_report(**check_overrides):
|
||||
check = {
|
||||
"id": "C1",
|
||||
"family": "reference_integrity",
|
||||
"signal_class": "deterministic",
|
||||
"strict_eligible": True,
|
||||
"status": "pass",
|
||||
"detail": "ok",
|
||||
"location": None,
|
||||
}
|
||||
check.update(check_overrides)
|
||||
return {
|
||||
"header": {
|
||||
"extraction_path": "joined_marker",
|
||||
"not_checked_count": 0,
|
||||
"package_fingerprint": "0" * 64,
|
||||
"policy_slug": None,
|
||||
},
|
||||
"checks": [check],
|
||||
}
|
||||
|
||||
|
||||
def test_schema_rejects_heuristic_strict_eligible():
|
||||
# §3.1/§6: heuristic checks are advisory-only STRUCTURALLY — the schema
|
||||
# itself forbids the promotion, not just the emitter.
|
||||
bad = _minimal_report(signal_class="heuristic", strict_eligible=True)
|
||||
with pytest.raises(jsonschema.ValidationError):
|
||||
jsonschema.validate(bad, load_schema())
|
||||
ok = _minimal_report(signal_class="heuristic", strict_eligible=False)
|
||||
jsonschema.validate(ok, load_schema())
|
||||
|
||||
|
||||
def test_schema_binds_check_id_prefix_to_family():
|
||||
# The id prefix encodes the family (spec §3 tables); the contract binds
|
||||
# them so a later-slice emitter cannot ship mismatched pairs.
|
||||
bad = _minimal_report(id="C1", family="blind_review_residue")
|
||||
with pytest.raises(jsonschema.ValidationError):
|
||||
jsonschema.validate(bad, load_schema())
|
||||
bad2 = _minimal_report(id="A1", family="reference_integrity")
|
||||
with pytest.raises(jsonschema.ValidationError):
|
||||
jsonschema.validate(bad2, load_schema())
|
||||
|
||||
|
||||
def test_schema_rejects_unknown_status():
|
||||
bad = _minimal_report(status="skipped")
|
||||
with pytest.raises(jsonschema.ValidationError):
|
||||
jsonschema.validate(bad, load_schema())
|
||||
@@ -0,0 +1,582 @@
|
||||
#!/usr/bin/env python3
|
||||
"""verify_submission_package CLI — deterministic submission-package verifier
|
||||
(#394 Slice 1: CLI skeleton + Family C reference integrity).
|
||||
|
||||
python scripts/verify_submission_package.py <package_dir> \
|
||||
[--passport passport.yaml] [--join-map map.yaml] [--report-out path]
|
||||
|
||||
Reads the files in an output package and runs the Family C two-way reference
|
||||
integrity check (in-text citation keys <-> reference-list entries), writing
|
||||
`submission_verification_report.json` (validating against
|
||||
shared/contracts/submission/submission_verification_report.schema.json) plus a
|
||||
human-readable summary to stdout.
|
||||
|
||||
Design contract (spec docs/design/2026-06-10-394-submission-package-verifier-spec.md):
|
||||
|
||||
- Detection is unconditional; terminality is the policy evaluator's job. This
|
||||
script NEVER reads `terminal_policies` (§5.3) — `policy_slug` is emitted null.
|
||||
- The joined marker path is deterministic; it needs a real prose-reference join
|
||||
(§3.3): the run's `citation_verification_summary[]` (via --passport), an
|
||||
explicit scholar-supplied join map (--join-map), or a package `.bib` whose
|
||||
keys map to slugs by the documented identity relation (draft_writer_agent.md:
|
||||
the slug IS the corpus `citation_key`). Markers with NO join source report
|
||||
`not_checked(missing prose-reference join)` — never a guessed comparison,
|
||||
and a slug an explicit join source does not cover is reported as unjoined,
|
||||
never identity-guessed.
|
||||
- Fallback extraction (`\\cite{}` for LaTeX, author-year regex for Markdown
|
||||
text) is heuristic-classed: advisory-only, `strict_eligible: false`, header
|
||||
`extraction_path: best_effort` (§3.3).
|
||||
- Every check reports pass | fail | warn | not_checked; `not_checked` is
|
||||
surfaced in the header count, never folded into pass (§1.4).
|
||||
- `package_fingerprint` reuses the audit-snapshot manifest convention
|
||||
(scripts/audit_snapshot.py; spec §10 open item 3, adjudicated at slice 1):
|
||||
`<relative-path>:<sha256>` lines, byte-sorted, trailing newline, fingerprint
|
||||
= SHA-256 of the manifest text. The report file itself is excluded.
|
||||
|
||||
Exit codes: 0 = no fail (warns allowed) and everything checked; 1 = >=1 fail;
|
||||
2 = usage/IO error; 3 = no fail but >=1 not_checked ("passed what was
|
||||
checkable", §8).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterator, Optional
|
||||
|
||||
import yaml
|
||||
|
||||
try:
|
||||
from audit_snapshot import sha256_hex
|
||||
except ImportError: # pragma: no cover - dual-path import
|
||||
from scripts.audit_snapshot import sha256_hex
|
||||
|
||||
REPORT_BASENAME = "submission_verification_report.json"
|
||||
|
||||
# Files scanned for in-text citations. provenance_summary.md is an advisory
|
||||
# carrier that legitimately repeats ref_slugs / citation_keys (#333) — scanning
|
||||
# it would manufacture false in-text hits.
|
||||
_MANUSCRIPT_SUFFIXES = {".md", ".tex", ".txt"}
|
||||
_SCAN_EXCLUDED_NAMES = {"provenance_summary.md", REPORT_BASENAME}
|
||||
|
||||
# v3.7.1+ marker grammar with the canonical slug charset (the lint-side
|
||||
# REF_PATTERN in check_v3_7_3_three_layer_citation.py). The suffix handling is
|
||||
# deliberately broader than REF_PATTERN's `[^-]*?` status group: a finalized
|
||||
# package carries `LOW-WARN` / `CONTAMINATED-*` suffix tokens (formatter
|
||||
# pass-through allowlist) that REF_PATTERN does not match, and missing those
|
||||
# markers here would fabricate orphans. Anchor markers (`<!--anchor:...-->`)
|
||||
# are a different grammar and never match.
|
||||
_REF_MARKER_RE = re.compile(r"<!--ref:([A-Za-z][A-Za-z0-9_:-]*)(?:\s[^>]*)?-->")
|
||||
|
||||
# BibTeX entry heads after an `^@` split: `article{key,`. @comment/@preamble/
|
||||
# @string carry no citation key and are excluded.
|
||||
_BIB_ENTRY_HEAD_RE = re.compile(
|
||||
r"(?!comment|preamble|string)[A-Za-z]+\s*\{\s*([^,\s}]+)\s*,",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
_LOCATION_CAP = 5 # findings listed per check detail before truncation
|
||||
|
||||
# Check registry mirroring the spec §3 family tables: id -> (family,
|
||||
# fail_capable). strict_eligible = fail_capable AND deterministic signal
|
||||
# (§3.1 separate axes; a warn-only check is never policy-promotable, §5.3).
|
||||
# Later slices extend this table — and may attach per-check conditional
|
||||
# eligibility (e.g. A4's venue-profile condition) — instead of special-casing
|
||||
# call sites. build_report enforces the roster: a runner that silently omits
|
||||
# a registered check cannot emit a report (the §1.4/#349 fail-open guard).
|
||||
_CHECK_REGISTRY = {
|
||||
"C1": ("reference_integrity", True),
|
||||
"C2": ("reference_integrity", False),
|
||||
}
|
||||
|
||||
# --- Fallback (best-effort) extraction grammar (§3.3, heuristic-classed) -----
|
||||
|
||||
# \cite / \citep / \citet / \citealp / starred forms, up to two optional args.
|
||||
_LATEX_CITE_RE = re.compile(r"\\cite[a-zA-Z]*\*?(?:\[[^\]]*\]){0,2}\{([^}]*)\}")
|
||||
|
||||
# Reference-list headings: fallback prose scanning stops here so rendered
|
||||
# reference entries are not mistaken for in-text citations.
|
||||
_REFS_HEADING_RE = re.compile(
|
||||
r"^#{0,6}\s*(?:references|bibliography|參考文獻)\s*$",
|
||||
re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
|
||||
_NAME = r"[A-Z][\w'’-]+"
|
||||
# Narrative: `Smith (2024)`, `Smith et al. (2024)`, `Smith and Chen (2024)`,
|
||||
# with an optional page-locator tail: `Smith (2024, p. 12)`.
|
||||
_NARRATIVE_CITE_RE = re.compile(
|
||||
r"(" + _NAME + r")(?:\s+et al\.?|\s+(?:and|&)\s+" + _NAME + r")?"
|
||||
r"\s+\((\d{4})[a-z]?(?:\s*,\s*pp?\.?[^)]*)?\)")
|
||||
# Parenthetical group content is split on `;` and each segment matched:
|
||||
# `(Smith, 2024)`, `(Chen & Lee, 2023)`, `(Smith et al., 2024a)`,
|
||||
# `(Chen & Lee, 2023, pp. 45–67)`.
|
||||
_PAREN_GROUP_RE = re.compile(r"\(([^()]+)\)")
|
||||
_PAREN_SEGMENT_RE = re.compile(
|
||||
r"^\s*(" + _NAME + r")[^\d]*?(\d{4})[a-z]?(?:\s*,\s*pp?\.?[^;]*)?\s*$")
|
||||
|
||||
|
||||
def compute_package_fingerprint(package_dir: Path,
|
||||
report_relpath: Optional[str] = None) -> str:
|
||||
"""Audit-snapshot manifest convention over the package files (§10 item 3):
|
||||
one `<package-relative-path>:<sha256>` line per file, LC_ALL=C byte-sorted,
|
||||
trailing newline; fingerprint = SHA-256 of the manifest text. The report
|
||||
file is excluded — the report cannot fingerprint its own bytes — including
|
||||
a custom --report-out path inside the package (report_relpath, as a
|
||||
package-relative posix path), or reruns would self-reference."""
|
||||
excluded = {REPORT_BASENAME, report_relpath}
|
||||
lines = []
|
||||
for path in package_dir.rglob("*"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
rel = path.relative_to(package_dir).as_posix()
|
||||
if rel in excluded:
|
||||
continue
|
||||
lines.append(f"{rel}:{sha256_hex(path.read_bytes())}")
|
||||
lines.sort() # byte sort over the composed line, matching audit_snapshot
|
||||
manifest_text = "\n".join(lines) + "\n"
|
||||
return sha256_hex(manifest_text.encode("utf-8"))
|
||||
|
||||
|
||||
def _collect_package_texts(package_dir: Path
|
||||
) -> tuple[dict[str, str], dict[str, str]]:
|
||||
"""One walk, one read per file: ({manuscript rel: text}, {bib rel: text})."""
|
||||
manuscripts: dict[str, str] = {}
|
||||
bibs: dict[str, str] = {}
|
||||
for path in sorted(package_dir.rglob("*")):
|
||||
if not path.is_file() or path.name in _SCAN_EXCLUDED_NAMES:
|
||||
continue
|
||||
suffix = path.suffix.lower()
|
||||
if suffix not in _MANUSCRIPT_SUFFIXES and suffix != ".bib":
|
||||
continue
|
||||
rel = path.relative_to(package_dir).as_posix()
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
(bibs if suffix == ".bib" else manuscripts)[rel] = text
|
||||
return manuscripts, bibs
|
||||
|
||||
|
||||
def extract_ref_markers(manuscripts: dict[str, str]) -> dict[str, str]:
|
||||
"""{slug: first-seen package-relative location} from <!--ref:slug--> markers."""
|
||||
found: dict[str, str] = {}
|
||||
for rel in sorted(manuscripts):
|
||||
for m in _REF_MARKER_RE.finditer(manuscripts[rel]):
|
||||
found.setdefault(m.group(1), rel)
|
||||
return found
|
||||
|
||||
|
||||
def _iter_bib_entries(bibs: dict[str, str]) -> Iterator[tuple[str, str]]:
|
||||
"""Yield (citation_key, raw entry body) per BibTeX entry across the
|
||||
package's .bib files — the single entry-head grammar both the key set and
|
||||
the author-year metadata derive from, so they cannot drift."""
|
||||
for rel in sorted(bibs):
|
||||
for chunk in re.split(r"(?m)^\s*@", bibs[rel])[1:]:
|
||||
head = _BIB_ENTRY_HEAD_RE.match(chunk)
|
||||
if head:
|
||||
yield head.group(1), chunk
|
||||
|
||||
|
||||
def parse_bib_keys(bibs: dict[str, str]) -> set[str]:
|
||||
return {key for key, _body in _iter_bib_entries(bibs)}
|
||||
|
||||
|
||||
def _parse_bib_metadata(bibs: dict[str, str]) -> dict[tuple, set]:
|
||||
"""{(first-author-surname-lower, year): {citation_key, ...}} from package
|
||||
.bib entries, for author-year fallback matching. Best-effort field parsing
|
||||
— the whole fallback path is heuristic-classed anyway (§3.3)."""
|
||||
metadata: dict[tuple, set] = {}
|
||||
for key, body in _iter_bib_entries(bibs):
|
||||
author = re.search(r"author\s*=\s*[{\"]([^}\"]+)", body, re.IGNORECASE)
|
||||
year = re.search(r"year\s*=\s*[{\"]?(\d{4})", body, re.IGNORECASE)
|
||||
if not (author and year):
|
||||
continue
|
||||
surname = _first_author_surname(author.group(1))
|
||||
if surname:
|
||||
metadata.setdefault(
|
||||
(surname.lower(), year.group(1)), set()).add(key)
|
||||
return metadata
|
||||
|
||||
|
||||
def _first_author_surname(author_field: str) -> str:
|
||||
first = author_field.split(" and ")[0].strip()
|
||||
if "," in first:
|
||||
return first.split(",")[0].strip()
|
||||
parts = first.split()
|
||||
return parts[-1] if parts else ""
|
||||
|
||||
|
||||
def _corpus_metadata(passport: dict[str, Any]) -> dict[tuple, set]:
|
||||
metadata: dict[tuple, set] = {}
|
||||
for e in passport.get("literature_corpus") or []:
|
||||
key = e.get("citation_key")
|
||||
year = e.get("year")
|
||||
authors = e.get("authors") or []
|
||||
family = authors[0].get("family") if (
|
||||
authors and isinstance(authors[0], dict)) else None
|
||||
if isinstance(key, str) and family and year is not None:
|
||||
metadata.setdefault((str(family).lower(), str(year)), set()).add(key)
|
||||
return metadata
|
||||
|
||||
|
||||
def _strip_reference_section(text: str) -> str:
|
||||
m = _REFS_HEADING_RE.search(text)
|
||||
return text[: m.start()] if m else text
|
||||
|
||||
|
||||
def _extract_fallback(manuscripts: dict[str, str],
|
||||
metadata: dict[tuple, set]
|
||||
) -> tuple[dict[str, str], dict[str, str]]:
|
||||
"""Best-effort in-text extraction (§3.3 fallback path): \\cite{} keys from
|
||||
.tex, author-year hits from .md/.txt matched against reference metadata.
|
||||
Returns (in_text {citation_key: location}, unresolved {display token:
|
||||
location}) — unresolved hits stay out of the citation-key namespace so a
|
||||
key that textually equals the token never silently merges."""
|
||||
in_text: dict[str, str] = {}
|
||||
unresolved: dict[str, str] = {}
|
||||
for rel in sorted(manuscripts):
|
||||
text = manuscripts[rel]
|
||||
if rel.lower().endswith(".tex"):
|
||||
for m in _LATEX_CITE_RE.finditer(text):
|
||||
for key in m.group(1).split(","):
|
||||
key = key.strip()
|
||||
if key:
|
||||
in_text.setdefault(key, rel)
|
||||
continue
|
||||
prose = _strip_reference_section(text)
|
||||
hits = [(m.group(1), m.group(2))
|
||||
for m in _NARRATIVE_CITE_RE.finditer(prose)]
|
||||
for g in _PAREN_GROUP_RE.finditer(prose):
|
||||
for segment in g.group(1).split(";"):
|
||||
m = _PAREN_SEGMENT_RE.match(segment)
|
||||
if m:
|
||||
hits.append((m.group(1), m.group(2)))
|
||||
for surname, year in hits:
|
||||
keys = metadata.get((surname.lower(), year))
|
||||
if keys:
|
||||
for key in keys:
|
||||
in_text.setdefault(key, rel)
|
||||
else:
|
||||
unresolved.setdefault(f"{surname} ({year})", rel)
|
||||
return in_text, unresolved
|
||||
|
||||
|
||||
def _check(check_id: str, status: str, detail: str, *,
|
||||
signal_class: str = "deterministic",
|
||||
location: Optional[str] = None) -> dict[str, Any]:
|
||||
family, fail_capable = _CHECK_REGISTRY[check_id]
|
||||
return {
|
||||
"id": check_id,
|
||||
"family": family,
|
||||
"signal_class": signal_class,
|
||||
"strict_eligible": fail_capable and signal_class == "deterministic",
|
||||
"status": status,
|
||||
"detail": detail,
|
||||
"location": location,
|
||||
}
|
||||
|
||||
|
||||
def _not_checked_pair(reason: str) -> list[dict[str, Any]]:
|
||||
return [
|
||||
_check("C1", "not_checked", reason),
|
||||
_check("C2", "not_checked", reason),
|
||||
]
|
||||
|
||||
|
||||
def _listed(keys: set[str]) -> str:
|
||||
shown = sorted(keys)[:_LOCATION_CAP]
|
||||
extra = len(keys) - len(shown)
|
||||
listing = ", ".join(shown)
|
||||
if extra > 0:
|
||||
listing += f", … (+{extra} more)"
|
||||
return listing
|
||||
|
||||
|
||||
def _compare_sets(in_text: dict[str, str], reference_keys: set[str],
|
||||
*, signal_class: str, in_text_label: str,
|
||||
reference_label: str,
|
||||
unjoined: Optional[dict[str, str]] = None,
|
||||
unjoined_label: str = ("with no join entry in the supplied "
|
||||
"join source")
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Two-way set check (§3.3): orphan in-text citation = fail (C1); uncited
|
||||
reference entry = warn (C2 — some venues allow further-reading entries).
|
||||
`unjoined` carries in-text hits that cannot be placed in the citation-key
|
||||
namespace (marker slugs the join source does not cover; fallback hits with
|
||||
no metadata match): they are a C1 fail in their own right — NEVER compared
|
||||
via an identity guess (§3.3), which would silently pass a slug that
|
||||
coincidentally equals a citation_key."""
|
||||
unjoined = unjoined or {}
|
||||
orphans = {k for k in in_text if k not in reference_keys}
|
||||
uncited = reference_keys - set(in_text)
|
||||
checks = []
|
||||
if orphans or unjoined:
|
||||
parts = []
|
||||
if orphans:
|
||||
parts.append(
|
||||
f"{len(orphans)} in-text citation(s) absent from "
|
||||
f"{reference_label}: {_listed(orphans)}")
|
||||
if unjoined:
|
||||
parts.append(
|
||||
f"{len(unjoined)} in-text citation(s) {unjoined_label}: "
|
||||
f"{_listed(set(unjoined))}")
|
||||
first_loc = min(
|
||||
[in_text[k] for k in orphans] + list(unjoined.values()))
|
||||
checks.append(_check(
|
||||
"C1", "fail",
|
||||
"; ".join(parts) + f" [{in_text_label}]",
|
||||
signal_class=signal_class, location=first_loc))
|
||||
else:
|
||||
checks.append(_check(
|
||||
"C1", "pass",
|
||||
f"all {len(in_text)} in-text citation(s) present in "
|
||||
f"{reference_label} [{in_text_label}]",
|
||||
signal_class=signal_class))
|
||||
if uncited:
|
||||
checks.append(_check(
|
||||
"C2", "warn",
|
||||
f"{len(uncited)} reference entr(ies) never cited in text: "
|
||||
f"{_listed(uncited)} [{in_text_label}]",
|
||||
signal_class=signal_class))
|
||||
else:
|
||||
checks.append(_check(
|
||||
"C2", "pass",
|
||||
f"all {len(reference_keys)} reference entr(ies) cited in text "
|
||||
f"[{in_text_label}]",
|
||||
signal_class=signal_class))
|
||||
return checks
|
||||
|
||||
|
||||
def _load_yaml(path: Path) -> dict[str, Any]:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"expected a YAML mapping in {path}")
|
||||
return data
|
||||
|
||||
|
||||
def _join_from_passport(passport: dict[str, Any]) -> dict[str, str]:
|
||||
"""{ref_slug: citation_key} from the passport's
|
||||
citation_verification_summary[] rows (the per-citation prose join the
|
||||
Stage 4->5 run already established, §3.3)."""
|
||||
join: dict[str, str] = {}
|
||||
for row in passport.get("citation_verification_summary") or []:
|
||||
slug = row.get("ref_slug")
|
||||
key = row.get("citation_key")
|
||||
if isinstance(slug, str) and slug and isinstance(key, str) and key:
|
||||
join[slug] = key
|
||||
return join
|
||||
|
||||
|
||||
def _corpus_keys(passport: dict[str, Any]) -> set[str]:
|
||||
return {
|
||||
e.get("citation_key")
|
||||
for e in passport.get("literature_corpus") or []
|
||||
if isinstance(e.get("citation_key"), str)
|
||||
}
|
||||
|
||||
|
||||
def run_family_c(package_dir: Path,
|
||||
passport: Optional[dict[str, Any]] = None,
|
||||
join_map: Optional[dict[str, str]] = None
|
||||
) -> tuple[list[dict[str, Any]], str]:
|
||||
"""Run Family C over the package. Returns (checks, extraction_path)."""
|
||||
manuscripts, bibs = _collect_package_texts(package_dir)
|
||||
if not manuscripts:
|
||||
return _not_checked_pair(
|
||||
"no manuscript found (no .md/.tex/.txt file in the package)"), "none"
|
||||
|
||||
markers = extract_ref_markers(manuscripts)
|
||||
bib_keys = parse_bib_keys(bibs)
|
||||
corpus_keys = _corpus_keys(passport) if passport else set()
|
||||
summary_join = _join_from_passport(passport) if passport else {}
|
||||
|
||||
# Reference-list side: a machine-readable source — package .bib keys, or
|
||||
# the passport's declared literature_corpus[] keys. Without one, neither
|
||||
# path has anything to compare against.
|
||||
if bib_keys:
|
||||
reference_keys, reference_label = bib_keys, "the package .bib reference list"
|
||||
elif corpus_keys:
|
||||
reference_keys, reference_label = (
|
||||
corpus_keys, "the passport literature_corpus reference list")
|
||||
else:
|
||||
return _not_checked_pair(
|
||||
"no machine-readable reference list (no package .bib and no "
|
||||
"passport literature_corpus[])"), "none"
|
||||
|
||||
if markers:
|
||||
# Joined marker path (deterministic). Join precedence: explicit
|
||||
# scholar-supplied map > the run's citation_verification_summary[] >
|
||||
# .bib identity relation.
|
||||
if join_map is not None:
|
||||
join: Optional[dict[str, str]] = dict(join_map)
|
||||
elif summary_join:
|
||||
join = summary_join
|
||||
elif bib_keys:
|
||||
# Documented identity relation (draft_writer_agent.md: the slug IS
|
||||
# the corpus citation_key): every marker slug joins to itself, so
|
||||
# a slug that is not a .bib key is simply an orphan.
|
||||
join = None
|
||||
else:
|
||||
return _not_checked_pair(
|
||||
"missing prose-reference join: <!--ref:slug--> markers found "
|
||||
"but no citation_verification_summary, --join-map, or package "
|
||||
".bib supplies the slug->citation_key join (§3.3 — never a "
|
||||
"guessed comparison)"), "none"
|
||||
if join is None: # .bib identity relation
|
||||
in_text, unjoined = dict(markers), {}
|
||||
else:
|
||||
# An explicit join source (summary / --join-map) must cover every
|
||||
# cited slug; a slug it does not cover is reported as such — NEVER
|
||||
# compared via an identity guess, which would silently pass a slug
|
||||
# that coincidentally equals a citation_key (§3.3).
|
||||
in_text, unjoined = {}, {}
|
||||
for slug, loc in markers.items():
|
||||
if slug in join:
|
||||
in_text.setdefault(join[slug], loc)
|
||||
else:
|
||||
unjoined.setdefault(slug, loc)
|
||||
return _compare_sets(
|
||||
in_text, reference_keys, signal_class="deterministic",
|
||||
in_text_label="joined marker path",
|
||||
reference_label=reference_label, unjoined=unjoined), "joined_marker"
|
||||
|
||||
# Fallback path (§3.3): no markers — non-ARS or post-converted source.
|
||||
# Format-aware best-effort extraction, heuristic-classed (advisory-only).
|
||||
metadata = _parse_bib_metadata(bibs)
|
||||
if passport:
|
||||
for k, v in _corpus_metadata(passport).items():
|
||||
metadata.setdefault(k, set()).update(v)
|
||||
in_text, unresolved = _extract_fallback(manuscripts, metadata)
|
||||
return _compare_sets(
|
||||
in_text, reference_keys, signal_class="heuristic",
|
||||
in_text_label="best-effort extraction",
|
||||
reference_label=reference_label, unjoined=unresolved,
|
||||
unjoined_label="unmatched against any reference metadata"
|
||||
), "best_effort"
|
||||
|
||||
|
||||
def build_report(package_dir: Path, checks: list[dict[str, Any]],
|
||||
extraction_path: str,
|
||||
report_path: Optional[Path] = None) -> dict[str, Any]:
|
||||
emitted = {c["id"] for c in checks}
|
||||
if emitted != set(_CHECK_REGISTRY):
|
||||
# Roster guard (§1.4/#349): a runner that silently omits a registered
|
||||
# check would read as "covered"; fail loud instead.
|
||||
raise ValueError(
|
||||
f"check roster mismatch: emitted {sorted(emitted)}, "
|
||||
f"registered {sorted(_CHECK_REGISTRY)}")
|
||||
report_relpath = None
|
||||
if report_path is not None:
|
||||
try:
|
||||
report_relpath = report_path.resolve().relative_to(
|
||||
package_dir.resolve()).as_posix()
|
||||
except ValueError:
|
||||
pass # report written outside the package — nothing to exclude
|
||||
return {
|
||||
"header": {
|
||||
"extraction_path": extraction_path,
|
||||
"not_checked_count": sum(
|
||||
1 for c in checks if c["status"] == "not_checked"),
|
||||
"package_fingerprint": compute_package_fingerprint(
|
||||
package_dir, report_relpath),
|
||||
# §5.2/§5.3: stamped by the slice-4 policy evaluator, never here.
|
||||
"policy_slug": None,
|
||||
},
|
||||
"checks": checks,
|
||||
}
|
||||
|
||||
|
||||
def render_human(report: dict[str, Any]) -> str:
|
||||
h = report["header"]
|
||||
lines = [
|
||||
"submission package verification "
|
||||
f"(extraction: {h['extraction_path']}, "
|
||||
f"not-checked: {h['not_checked_count']}, "
|
||||
f"fingerprint: {h['package_fingerprint'][:12]}…)",
|
||||
]
|
||||
for c in report["checks"]:
|
||||
status = c["status"].upper().replace("NOT_CHECKED", "NOT-CHECKED")
|
||||
loc = f" @ {c['location']}" if c["location"] else ""
|
||||
lines.append(
|
||||
f" [{status}] {c['id']} ({c['family']}, {c['signal_class']})"
|
||||
f"{loc}: {c['detail']}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def exit_code_for(report: dict[str, Any]) -> int:
|
||||
statuses = {c["status"] for c in report["checks"]}
|
||||
if "fail" in statuses:
|
||||
return 1
|
||||
if "not_checked" in statuses:
|
||||
return 3 # "passed what was checkable" (§8) — distinct from a full pass
|
||||
return 0
|
||||
|
||||
|
||||
def run(argv: Optional[list[str]] = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="verify_submission_package",
|
||||
description="Deterministic submission-package verifier (#394 Slice 1: "
|
||||
"Family C reference integrity).",
|
||||
epilog="Exit codes: 0 all-checked no-fail; 1 at least one fail; "
|
||||
"2 usage/IO error; 3 no fail but at least one NOT-CHECKED.")
|
||||
parser.add_argument("package_dir", help="Output package directory to verify.")
|
||||
parser.add_argument(
|
||||
"--passport", default=None,
|
||||
help="Material Passport YAML supplying citation_verification_summary[] "
|
||||
"(the prose-reference join) and/or literature_corpus[] (the "
|
||||
"declared reference list).")
|
||||
parser.add_argument(
|
||||
"--join-map", default=None,
|
||||
help="Explicit scholar-supplied {ref_slug: citation_key} YAML/JSON "
|
||||
"mapping (overrides every other join source).")
|
||||
parser.add_argument(
|
||||
"--report-out", default=None,
|
||||
help=f"Report path (default: <package_dir>/{REPORT_BASENAME}).")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
package_dir = Path(args.package_dir)
|
||||
if not package_dir.is_dir():
|
||||
print(f"[verify_submission_package ERROR] not a directory: "
|
||||
f"{package_dir}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
passport = None
|
||||
if args.passport is not None:
|
||||
try:
|
||||
passport = _load_yaml(Path(args.passport))
|
||||
except (OSError, ValueError, yaml.YAMLError) as e:
|
||||
print(f"[verify_submission_package ERROR] could not load passport: "
|
||||
f"{e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
join_map = None
|
||||
if args.join_map is not None:
|
||||
try:
|
||||
raw = _load_yaml(Path(args.join_map))
|
||||
except (OSError, ValueError, yaml.YAMLError) as e:
|
||||
print(f"[verify_submission_package ERROR] could not load join map: "
|
||||
f"{e}", file=sys.stderr)
|
||||
return 2
|
||||
join_map = {str(slug): str(key) for slug, key in raw.items()}
|
||||
|
||||
checks, extraction_path = run_family_c(
|
||||
package_dir, passport=passport, join_map=join_map)
|
||||
report_path = (Path(args.report_out) if args.report_out
|
||||
else package_dir / REPORT_BASENAME)
|
||||
report = build_report(package_dir, checks, extraction_path,
|
||||
report_path=report_path)
|
||||
|
||||
try:
|
||||
report_path.write_text(
|
||||
json.dumps(report, indent=2, ensure_ascii=False) + "\n",
|
||||
encoding="utf-8")
|
||||
except OSError as e:
|
||||
print(f"[verify_submission_package ERROR] could not write report: {e}",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
|
||||
print(render_human(report))
|
||||
return exit_code_for(report)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(run())
|
||||
@@ -0,0 +1,225 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://github.com/Imbad0202/academic-research-skills/shared/contracts/submission/submission_verification_report.schema.json",
|
||||
"title": "Submission Verification Report",
|
||||
"description": "#394 deterministic submission-package verifier report (spec docs/design/2026-06-10-394-submission-package-verifier-spec.md §7.3). Emitted by scripts/verify_submission_package.py as submission_verification_report.json inside the package directory. Detection is unconditional; terminality is decided by the policy evaluator (orchestrator) reading terminal_policies — the script never reads terminal_policies (§5.3). Slice 1 ships Family C (reference_integrity) only; the check-id/family vocabulary already covers Families A/B/D so later slices are additive, not schema-breaking.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"header",
|
||||
"checks"
|
||||
],
|
||||
"properties": {
|
||||
"header": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"extraction_path",
|
||||
"not_checked_count",
|
||||
"package_fingerprint",
|
||||
"policy_slug"
|
||||
],
|
||||
"properties": {
|
||||
"extraction_path": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"joined_marker",
|
||||
"best_effort",
|
||||
"none"
|
||||
],
|
||||
"description": "How in-text citations were extracted (§3.3). joined_marker = <!--ref:slug--> markers joined to citation_keys via a real join source (deterministic-eligible). best_effort = format-aware fallback extraction (\\cite{} / author-year regex) — the whole Family C run downgrades to heuristic class (advisory-only). none = no extraction ran (every extraction-dependent check is not_checked)."
|
||||
},
|
||||
"not_checked_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"description": "Number of checks with status=not_checked. Surfaced in the header so incompleteness is never folded into pass (§1.4 honesty premise / #349 lesson)."
|
||||
},
|
||||
"package_fingerprint": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9a-f]{64}$",
|
||||
"description": "SHA-256 over the package manifest, reusing the audit-snapshot hashing convention (scripts/audit_snapshot.py write_manifest, spec §10 open item 3 — adjudicated at slice 1): one line per file `<package-relative-path>:<sha256-hex>`, LC_ALL=C byte-sorted, newline-joined with a trailing newline; the fingerprint is the SHA-256 of that manifest text. The report file itself (submission_verification_report.json) is excluded from the manifest — the report cannot fingerprint its own bytes. Freshness-guard carrier (§5.2): the orchestrator MUST NOT reuse a report whose fingerprint no longer matches the package."
|
||||
},
|
||||
"policy_slug": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
],
|
||||
"description": "The terminal_policies.submission_package value in force when the policy evaluator consumed this report (§5.2 freshness guard). Stamped by the slice-4 orchestrator hook, NEVER by the script (the script never reads terminal_policies, §5.3). Null until slice 4 lands the policy key — and always null in a standalone CLI run."
|
||||
}
|
||||
}
|
||||
},
|
||||
"checks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/check"
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"check": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"family",
|
||||
"signal_class",
|
||||
"strict_eligible",
|
||||
"status",
|
||||
"detail",
|
||||
"location"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"pattern": "^[ABCD][1-9][0-9]*$",
|
||||
"description": "Check id per the spec §3 family tables (A1–A6, B*, C1 orphan in-text citation, C2 uncited reference entry, D*)."
|
||||
},
|
||||
"family": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"blind_review_residue",
|
||||
"venue_limits",
|
||||
"reference_integrity",
|
||||
"repro_lock_linkage"
|
||||
],
|
||||
"description": "Spec §3 check family. Slice 1 emits reference_integrity only."
|
||||
},
|
||||
"signal_class": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"deterministic",
|
||||
"heuristic"
|
||||
],
|
||||
"description": "Whether the signal is a fact about the file (deterministic) or a pattern match that can false-positive (heuristic). A SEPARATE axis from strict_eligible (§3.1 load-bearing): deterministic signal ≠ block-worthy."
|
||||
},
|
||||
"strict_eligible": {
|
||||
"type": "boolean",
|
||||
"description": "Whether a fail on this check is promotable by terminal_policies.submission_package: strict (slice 4). Heuristic checks are NEVER strict-eligible — enforced structurally below, not by default."
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"pass",
|
||||
"fail",
|
||||
"warn",
|
||||
"not_checked"
|
||||
],
|
||||
"description": "pass / fail / warn (advisory finding that is never block-worthy, e.g. C2 uncited reference entry — some venues allow further-reading entries, §3.3) / not_checked (the check could not run; reason in detail; NEVER folded into pass, §1.4)."
|
||||
},
|
||||
"detail": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Human-readable finding. For status=not_checked this carries the reason (e.g. 'missing prose-reference join', 'no machine-readable reference list')."
|
||||
},
|
||||
"location": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
],
|
||||
"description": "Package-relative file path (optionally :line) where the finding sits; null when the finding is package-level."
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"description": "Heuristic checks are advisory-only STRUCTURALLY (§3.1 / §6): a heuristic signal_class forbids strict_eligible=true. This is a class exclusion, not a default — no emitter and no later policy slice can promote a heuristic check without a schema break.",
|
||||
"if": {
|
||||
"properties": {
|
||||
"signal_class": {
|
||||
"const": "heuristic"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"strict_eligible": {
|
||||
"const": false
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"description": "warn is advisory-only and never policy-promotable (§5.3): a warn outcome forbids strict_eligible=true. Checks whose worst outcome is warn (C2 uncited reference entry) are not strict-eligible by class.",
|
||||
"if": {
|
||||
"properties": {
|
||||
"status": {
|
||||
"const": "warn"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"strict_eligible": {
|
||||
"const": false
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"description": "The id prefix encodes the family (spec §3 tables: A=blind_review_residue, B=venue_limits, C=reference_integrity, D=repro_lock_linkage); the contract binds the redundant pair so an emitter cannot ship a mismatch downstream code keys on.",
|
||||
"if": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"pattern": "^A"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"family": {
|
||||
"const": "blind_review_residue"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"pattern": "^B"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"family": {
|
||||
"const": "venue_limits"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"pattern": "^C"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"family": {
|
||||
"const": "reference_integrity"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"pattern": "^D"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"family": {
|
||||
"const": "repro_lock_linkage"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user