mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
* fix(claims): align distribution-surface claims with evidence ceilings (#753) - plugin.json / marketplace.json: drop 'Production-grade' / '39-agent ensemble' for matrix-licensed wording ('contract-audited', '39 prompt roles (3 plugin-exposed agents; the rest run inline by default)') - academic-pipeline/SKILL.md: no-bypass prose rewritten to the actual mechanism (mandatory checkpoints; overrides require recorded user reasoning); #528 content-lock hash updated in the same commit - shared/cross_model_verification.md: 31%->5-10% relabeled as an unvalidated working hypothesis - shared/ground_truth_isolation_pattern.md: gold-labels rule rewritten to the intended boundary (no unconditional loading into operational agent context) - version-consistency invariant 8: binds 'N prompt roles' spelling too, checks every count token (finditer) - new scripts/check_distribution_surface_claims.py (D1-D5, 20 mutation tests, CI-wired): fail-closed manifest load, shared claim vocabulary imported from check_stage_capability_matrix, percentage refusal, mandatory bindable count token, plugin-exposed count bound to MIRRORS Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki * fix(claims): codex R1 — integrity-family must-PASS sweep + lint case/boundary fixes (#753) - integrity 'must PASS with zero issues' absolutes now name the recorded 3-round FAIL-loop exit (integrity_review_protocol, reinforcement_content, team_collaboration_protocol, integrity_verification_agent, SKILL.md flow row); 'recorded with reasoning' weakened to 'recorded user decision' (rationale escalates per compliance override ladder) - D3 percent check lowercases input (matrix caller parity) - D5 plugin-exposed regex case-insensitive - AGENT_CLAIM_RE gains trailing boundaries (39-agentic / singular 'prompt role' no longer count as bound); 4 new mutation tests (20 -> 24) - SKILL.md #528 content-lock hash rebumped Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki * fix(claims): codex R2 — Stage 2.5 routing parity, passport-state honesty, gold-set scope, strict JSON (#753) - Stage 2.5 flow row + both state-machine checkpoint triggers name the recorded FAIL-loop exit (SKILL.md + pipeline_state_machine.md, both content-lock hashes rebumped) - team protocol handoff checklist: FAIL-loop continuation keeps passport verification_status UNVERIFIED; VERIFIED only on zero-issue PASS - ground-truth gold exception scoped to synthetic/public-safe content; live-reviewer calibration sets stay runtime-supplied - D1 rejects non-standard JSON constants (NaN/Infinity) via parse_constant; 2 new tests (24 -> 26) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki * fix(claims): codex R3 — prerequisite checker + handoff materials accept the recorded FAIL-loop route (#753) - state_tracker_agent prerequisite table: Stage 3 / Stage 5 entry rows accept a recorded Integrity Check FAIL Loop resolution (previously the documented continuation route was unreachable at the checker) - SKILL.md handoff lines 2.5->3 and 4.5->5 no longer mislabel a FAIL-loop continuation draft as verified; team protocol Materials/Approval rows aligned the same way - SKILL.md + state_tracker_agent content-lock hashes rebumped Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki * fix(claims): codex R4 — orchestrator transfer rows + advisory dispatch accept the recorded FAIL-loop route (#753) - orchestrator 2.5->3 and 4.5->5 transfer rows no longer require a 'Verified'-labeled draft on a recorded FAIL-loop continuation - #660/#672 advisory dispatch anchors to the Stage 4.5 terminal resolution (PASS, or recorded FAIL-loop continuation) instead of exact PASS only - orchestrator content-lock hash rebumped Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
committed by
GitHub
GitHub
parent
704b46d247
commit
e9759dc4f4
@@ -4,7 +4,7 @@
|
||||
"name": "Cheng-I Wu",
|
||||
"url": "https://github.com/Imbad0202"
|
||||
},
|
||||
"description": "Academic Research Skills — production-grade research, writing, peer review, and pipeline orchestration for Claude Code.",
|
||||
"description": "Academic Research Skills — contract-audited research, writing, peer review, and pipeline orchestration for Claude Code.",
|
||||
"plugins": [
|
||||
{
|
||||
"name": "academic-research-skills",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "academic-research-skills",
|
||||
"version": "3.20.1",
|
||||
"description": "Production-grade academic research pipeline for Claude Code: research → write → review → revise → finalize. 4 skills, 27 modes, 39-agent ensemble, v3.7.3 + v3.8 L3 claim-faithfulness gate, v3.9.0 cross-index triangulation, v3.10 triangulation policy layer, v3.11 deterministic citation verification gate (#182).",
|
||||
"description": "Contract-audited academic research pipeline for Claude Code: research → write → review → revise → finalize. 4 skills, 27 modes, 39 prompt roles (3 plugin-exposed agents; the rest run inline by default), v3.7.3 + v3.8 L3 claim-faithfulness gate, v3.9.0 cross-index triangulation, v3.10 triangulation policy layer, v3.11 deterministic citation verification gate (#182). Capability ceilings: docs/STAGE_CAPABILITY_MATRIX.md.",
|
||||
"author": {
|
||||
"name": "Cheng-I Wu",
|
||||
"url": "https://github.com/Imbad0202"
|
||||
|
||||
@@ -166,6 +166,11 @@ jobs:
|
||||
PYTHONPATH: scripts
|
||||
run: python3 scripts/check_version_consistency.py
|
||||
|
||||
- name: Check distribution-surface claim discipline (#753)
|
||||
# The pytest companion `test_check_distribution_surface_claims.py`
|
||||
# runs via the unified manifest.
|
||||
run: python3 scripts/check_distribution_surface_claims.py
|
||||
|
||||
- name: Check SETUP cross-model example parity (#491)
|
||||
# The pytest companion `test_check_setup_cross_model_parity.py`
|
||||
# runs via the unified manifest (and also under pytest.yml's
|
||||
|
||||
@@ -12,6 +12,8 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Distribution-surface claims aligned with evidence ceilings (#753).** The 2026-08-17 ISO 42001-spirit audit (T-1 through T-4, #761) found four claim surfaces sitting outside every existing lock, their language outrunning the evidence record. All four are rewritten to matrix-licensed language: (1) `.claude-plugin/plugin.json` / `.claude-plugin/marketplace.json` descriptions drop "Production-grade" and "39-agent ensemble" (8 task families are `NOT_RUN` in `docs/STAGE_CAPABILITY_MATRIX.md`; "ensemble" implied execution/error independence the default inline path does not provide) in favor of "contract-audited … 39 prompt roles (3 plugin-exposed agents; the rest run inline by default)" with an explicit capability-ceiling pointer. (2) The `academic-pipeline/SKILL.md` no-bypass prose ("no `--no-block` escape hatch" / "cannot be skipped, must PASS" / "must PASS with zero issues") is rewritten to what the mechanism actually is — mandatory checkpoints whose only path past a block is an explicit, recorded user decision (rationale requirements escalate on repeated overrides per `shared/compliance_checkpoint_protocol.md`), a trust-based control with an audit trail; the same "must PASS with zero issues" absolutes across the integrity-protocol family (`references/integrity_review_protocol.md`, `references/reinforcement_content.md`, `references/team_collaboration_protocol.md`, `agents/integrity_verification_agent.md`, the SKILL.md stage-flow rows for both Stage 2.5 and 4.5, and the state-machine checkpoint-trigger rows) now name the recorded 3-round FAIL-loop exit instead of contradicting it — including the team-protocol handoff rows, the `state_tracker_agent` prerequisite checker, and the `pipeline_orchestrator_agent` transfer rows + #660/#672 advisory dispatch (Stage 3 / Stage 5 entry and both advisory reruns now accept the recorded FAIL-loop route instead of making it unreachable or relabeling the draft as verified), with a FAIL-loop continuation keeping the passport `UNVERIFIED` — and the #528 content-lock hashes (SKILL.md, `pipeline_state_machine.md`, `state_tracker_agent.md`, `pipeline_orchestrator_agent.md`) are updated in the same commits. (3) The `shared/cross_model_verification.md` "31% → ~5-10%" error-reduction estimate is relabeled as an unvalidated working hypothesis — no local study has measured a post-verification rate. (4) The `shared/ground_truth_isolation_pattern.md` "never bundle gold labels into the repository" sentence is rewritten to its intended boundary (no unconditional loading into operational agent context), resolving the literal contradiction with `evals/gold/` existing in-repo for explicitly-invoked harnesses; the exception is scoped to synthetic/public-safe content, with live-reviewer calibration sets staying runtime-supplied and never committed. Two lint layers keep the distribution manifests from re-drifting (the three rewritten prose surfaces carry no dedicated claim-language lint; `academic-pipeline/SKILL.md` sits under the byte-level #528 content lock only): version-consistency invariant 8 now binds the agent-count number in both the legacy "N-agent" and the new "N prompt roles" spellings and checks every count token, not just the first; and a new `scripts/check_distribution_surface_claims.py` (26 mutation tests, CI-wired via spec-consistency.yml + the pytest manifest) fails closed on missing/unparseable distribution manifests (non-standard `NaN`/`Infinity` constants included), refuses unlicensed claim language in any plugin/marketplace description field (the capability matrix's effectiveness vocabulary imported from `check_stage_capability_matrix.py` so a matrix-side stem addition binds this surface in the same commit, any percentage figure, "production-grade", "agent ensemble"), requires the plugin description to carry a count token in a spelling invariant 8 can bind (a reworded count cannot silently detach the number from the tree), and binds any "N plugin-exposed" claim to the `check_agents_mirror_sync` MIRRORS roster size.
|
||||
|
||||
- **Citation-surface version drift + version-consistency invariant 12 (#754).** `CITATION.cff` and the `POSITIONING.md` citation prose had silently sat at 3.14.0 for six minor releases while the suite moved to 3.20.1 (the Zenodo archive itself was current — v3.20.1 deposit published 2026-08-16 — so this was pure metadata drift; root cause: the file was added 2026-06-15, after the version lint existed, and nothing forced new version-bearing surfaces into the lint's coverage). Both surfaces now read 3.20.1, and `scripts/check_version_consistency.py` gains invariant 12: CITATION.cff is parsed as YAML (a regex scrape would misread the legitimate quoted spelling as drift) with absence an error like README.md, its `version` gated through the broad-capture + strict-semver idiom, and its `date-released` compared against the latest CHANGELOG entry date with the invariant-10 ±7-day window (the second half of the same drift — the date had sat 6 weeks stale); every `(Version X.Y.Z)` token in POSITIONING.md must be canonical and equal the suite version, so a `v`-prefixed or truncated human edit errors instead of being silently filtered (the pre-#169 lesson). Both surfaces are wired into the aligned test fixture, so every pass-case test now exercises the invariant; 16 targeted mutation tests (53 → 69). Hardened through a 4-angle cleanup review plus a 5-round cross-model review under an explicit threat-model ceiling (crash-on-malformed-date, silent-skip, and silent-capture-dropout paths closed; the duplicate-YAML-key construction adjudicated as a documented known limitation — last-wins matches every CFF consumer). Filed from the 2026-08-17 ISO 42001-spirit audit (#761).
|
||||
|
||||
## [3.20.1] - 2026-08-15 — Contract-honesty hardening and bounded evaluation substrates
|
||||
|
||||
@@ -126,12 +126,12 @@ This mirrors PaperOrchestra's parallel execution of Plot Generation (Step 2) and
|
||||
|
||||
1. **Stage 1 RESEARCH** -> user confirmation -> Stage 2
|
||||
2. **Stage 2 WRITE** -> user confirmation -> Stage 2.5
|
||||
3. **Stage 2.5 INTEGRITY** -> PASS -> Stage 3 (FAIL -> fix and re-verify, max 3 rounds)
|
||||
3. **Stage 2.5 INTEGRITY** -> PASS -> Stage 3 (FAIL -> fix and re-verify, max 3 rounds; then Integrity Check FAIL Loop -> recorded user decision)
|
||||
4. **Stage 3 REVIEW** -> Accept -> Stage 4.5 / Minor|Major -> Stage 4 / Reject -> Stage 2 or end
|
||||
5. **Stage 4 REVISE** -> user confirmation -> Stage 3'
|
||||
6. **Stage 3' RE-REVIEW** -> Accept|Minor -> Stage 4.5 / Major -> Stage 4'
|
||||
7. **Stage 4' RE-REVISE** -> user confirmation -> Stage 4.5 (no return to review)
|
||||
8. **Stage 4.5 FINAL INTEGRITY** -> PASS (zero issues) -> Stage 5 (FAIL -> fix and re-verify)
|
||||
8. **Stage 4.5 FINAL INTEGRITY** -> PASS (zero issues) -> Stage 5 (FAIL -> fix and re-verify; after 3 unresolved rounds -> Integrity Check FAIL Loop -> recorded user decision)
|
||||
9. **Stage 5 FINALIZE** -> MD -> DOCX via Pandoc when available (otherwise instructions) -> ask about LaTeX -> confirm -> PDF -> completion checkpoint (FULL) -> Stage 6 (user may decline Stage 6: marked `skipped`, pipeline goes directly to `completed`)
|
||||
10. **Stage 6 PROCESS SUMMARY** -> ask language version -> generate process record MD -> LaTeX -> PDF -> terminal acknowledgement (`finish` / `end` / `done` / `confirm`, or an unambiguous natural-language equivalent) -> pipeline global state `completed`
|
||||
|
||||
@@ -280,13 +280,13 @@ After user confirmation:
|
||||
- Stage 1 --> 2: deep-research handoff (RQ Brief + Methodology Blueprint + Bibliography + Synthesis)
|
||||
- #672 cargo on every transition: exact builder-produced `preregistration-artifact/1.0` receipt and its named companion when provided; validate and carry byte-for-byte
|
||||
- Stage 2 --> 2.5: Pass complete paper to integrity_verification_agent
|
||||
- Stage 2.5 --> 3: Pass verified paper to reviewer
|
||||
- Stage 2.5 --> 3: Pass the Stage 2.5 paper to reviewer (verified, or carrying the recorded FAIL-loop partially-unverified warning)
|
||||
- Stage 3 --> 4: Pass Revision Roadmap to academic-paper revision mode
|
||||
- Stage 4 --> 3': Pass revised draft, the hard-required original pre-revision draft (#576 current 1.1 §3.1 Phase 2A comparison base), exact author-adjudication sidecar, fully replayed Revision-Evidence Bundle, Response to Reviewers, Editorial Decision Letter, Round-1 findings, the immutable Roadmap, the exact ordered patch/report pairs projected by the bundle, and Round-1 Reviewer Configuration Cards. Missing original/roadmap/author/bundle is `manifest_incomplete`; this is the default contract re-review transfer. A user-requested fresh full review at 3' remains a separate full-mode branch.
|
||||
- Stage 3' --> 4': Pass new Revision Roadmap + R&R Traceability Matrix (Schema 11) to academic-paper revision mode; the traceability sidecar (frozen `previously_missed`/`indeterminate` records, #576 §8) rides through 4' toward Stage 4.5
|
||||
- Stage 3' --> 4.5 (Accept/Minor direct path): Pass verified revised draft + the traceability sidecar's frozen records to integrity_verification_agent as gate input
|
||||
- Stage 4/4' --> 4.5: Pass revision-completed paper to integrity_verification_agent (final verification); on the Major-via-4' path the Stage 3' traceability sidecar travels along as gate input
|
||||
- Stage 4.5 --> 5: Pass verified final draft to the one mandatory Stage-5 entry checkpoint; run #660 then #672 against that same accepted artifact ID/SHA-256 before format-convert dispatch
|
||||
- Stage 4.5 --> 5: Pass the accepted final draft (verified, or carrying the recorded FAIL-loop partially-unverified warning) to the one mandatory Stage-5 entry checkpoint; run #660 then #672 against that same accepted artifact ID/SHA-256 before format-convert dispatch
|
||||
- Stage 5 --> 6: Pass final deliverables list + the Process-Summary projection of pipeline state history, omitting the #673 activity projection of terminal root `run_id`, pending/sealed activity fields, selected-store data, renderer output, and diagnostics (user may decline Stage 6 at the Stage 5 completion checkpoint)
|
||||
3. Begin next stage
|
||||
```
|
||||
@@ -339,9 +339,9 @@ Routing into Mode B requires explicit user signal — `/ars-<mode>` slash comman
|
||||
|
||||
Stage 2.5 (pre-review) and Stage 4.5 (post-revision) verification. 5-phase protocol: references → citation context → statistical data → originality → claims.
|
||||
|
||||
⚠️ **IRON RULE**: Stage 4.5 must PASS with zero issues to proceed to Stage 5. Stage 4.5 performs a fresh from-scratch pass without relying on Stage 2.5 conclusions; this is not a claim of independent error processes.
|
||||
⚠️ **IRON RULE**: Stage 4.5 must reach a recorded terminal resolution before Stage 5: PASS, or — after the 3-round integrity FAIL loop is exhausted — an explicit, recorded user decision on the listed unresolved items (rationale requirements escalate on repeated overrides; see `shared/compliance_checkpoint_protocol.md`). Unresolved items are never silently dropped. Stage 4.5 performs a fresh from-scratch pass without relying on Stage 2.5 conclusions; this is not a claim of independent error processes.
|
||||
|
||||
⚠️ **IRON RULE (v3.2)**: Both Stage 2.5 and Stage 4.5 must also run the **AI Research Failure Mode Checklist** — a 7-mode taxonomy extending the citation hallucination checks into implementation bugs, hallucinated results, shortcut reliance, bug-as-insight, methodology fabrication, and pipeline-level frame-lock. If any of the 7 modes is `SUSPECTED`, or if Modes 1/3/5/6 are `INSUFFICIENT EVIDENCE`, the pipeline **blocks** and the user must acknowledge (confirm / override with reasoning / revise) before the pipeline proceeds. There is no `--no-block` escape hatch. Stage 6 PROCESS SUMMARY then reports the full failure-mode audit log as part of the AI Self-Reflection Report.
|
||||
⚠️ **IRON RULE (v3.2)**: Both Stage 2.5 and Stage 4.5 must also run the **AI Research Failure Mode Checklist** — a 7-mode taxonomy extending the citation hallucination checks into implementation bugs, hallucinated results, shortcut reliance, bug-as-insight, methodology fabrication, and pipeline-level frame-lock. If any of the 7 modes is `SUSPECTED`, or if Modes 1/3/5/6 are `INSUFFICIENT EVIDENCE`, the pipeline **blocks** and the user must acknowledge (confirm / override with reasoning / revise) before the pipeline proceeds. No configuration flag silences this block; the only path past it is the recorded user acknowledgment above — a trust-based control with an audit trail. Stage 6 PROCESS SUMMARY then reports the full failure-mode audit log as part of the AI Self-Reflection Report.
|
||||
|
||||
> See `references/integrity_review_protocol.md` for the 5-phase citation/claim verification procedures.
|
||||
> See `references/ai_research_failure_modes.md` for the 7-mode AI research failure checklist and block/override logic.
|
||||
@@ -535,7 +535,7 @@ Explicit prohibitions to prevent common failure modes:
|
||||
| 5 | **Silently dropping reviewer concerns** | Revision addresses 8 of 10 concerns and hopes nobody notices | The R&R tracking table must account for every concern with explicit status |
|
||||
| 6 | **Re-verifying only known issues at Stage 4.5** | Final integrity check only re-checks Stage 2.5 findings | Stage 4.5 must run a fresh from-scratch pass; revision may introduce new issues |
|
||||
| 7 | **Inflating Collaboration Quality scores** | Giving 90/100 to avoid awkward self-criticism | Honesty first: no inflation, no pleasantries; cite specific evidence for every score |
|
||||
| 8 | **Bypassing the Failure Mode Checklist block** (v3.2) | "The 7-mode checklist is new, let's skip it this run" | Stage 2.5/4.5 Failure Mode Checklist is MANDATORY and BLOCKING; no `--no-block` flag exists; overrides require user reasoning recorded for Stage 6 |
|
||||
| 8 | **Bypassing the Failure Mode Checklist block** (v3.2) | "The 7-mode checklist is new, let's skip it this run" | Stage 2.5/4.5 Failure Mode Checklist is MANDATORY and BLOCKING; there is no unrecorded bypass — every override requires user reasoning recorded for Stage 6 |
|
||||
|
||||
---
|
||||
|
||||
@@ -548,7 +548,7 @@ Explicit prohibitions to prevent common failure modes:
|
||||
| Material handoff | Stage-to-stage handoff materials are complete and correctly formatted |
|
||||
| State tracking | Pipeline state updated in real time; Progress Dashboard accurate |
|
||||
| **Mandatory checkpoint** | **User confirmation required after each stage completion** |
|
||||
| **Mandatory integrity check** | **Stage 2.5 and 4.5 cannot be skipped, must PASS** |
|
||||
| **Mandatory integrity check** | **Stage 2.5 and 4.5 always run; continuation past a non-PASS result requires an explicit, recorded user decision** |
|
||||
| **Mandatory failure mode checklist** (v3.2) | **Stage 2.5 and 4.5 must run the 7-mode AI research failure checklist; suspected failures block; overrides require user reasoning** |
|
||||
| No overstepping | ⚠️ IRON RULE: Orchestrator does not perform substantive research/writing/reviewing, only dispatching |
|
||||
| No forcing | ⚠️ IRON RULE: User can pause or exit pipeline at any time (but cannot skip integrity checks) |
|
||||
|
||||
@@ -606,7 +606,7 @@ E6 detection is semantic and may be model-mediated. The disposition runtime reco
|
||||
- Phase E executes semantic/model-mediated E1 extraction to create the registered population, then E1.1 reports bounded candidate gaps with semantic completeness unknown. E2 (source tracing) + E3 (cross-referencing) run on the #549 risk-stratified registry selection: 100% of registered HIGH-IMPACT claims + a 10% RANDOM sentinel of the registered remainder, topped up to min(10, registered total) — fewer than 10 registered claims total → audit the whole registry
|
||||
- **Phase C4 (#260): the D7 declaration-anchored anti-skip runs on the passport (not sampled — it is a single passport-level check); experiment_alignment_results[] rows are produced for the sampled experiment-backed claims (>= 30% — C4's own rate; the general claim check is #549 risk-stratified, no longer a flat 30%).**
|
||||
- Issues found -> produce correction list -> fix -> re-verify corrected items
|
||||
- **Must PASS to proceed to Stage 3 (REVIEW)**
|
||||
- **Stage 3 (REVIEW) entry requires PASS, or — after the 3-round Integrity Check FAIL Loop — an explicit, recorded user decision on the listed unresolved items**
|
||||
|
||||
### Mode 2: Final Verification (Stage 4.5 — Post-Revision Final Check)
|
||||
|
||||
@@ -620,7 +620,7 @@ E6 detection is semantic and may be model-mediated. The disposition runtime reco
|
||||
- Special focus: Citations, data, and claims added or modified during the revision process
|
||||
- ADDITIONALLY: Compare with Stage 2.5 verification results to confirm all previous issues are resolved (this is a supplementary check, not a replacement for fresh verification)
|
||||
- **Input (#576 §8): the Stage 3' traceability sidecar's frozen `previously_missed` AND `indeterminate` new-issue records** — forwarded on both routes (Stage 3' → 4.5 direct on Accept/Minor; through 4' with the roadmap on Major). Consume both attributions as integrity-check input, not just cargo. Current #576 1.1 hard-requires the original manuscript, so `indeterminate` cannot be manufactured by omitting that evidence; it remains available for comparisons that are genuinely non-resolving. Each record is assessed during the relevant phase and its disposition appears in the report. A `[LEGACY-NO-CONTRACT]` run may legitimately produce no sidecar; note that legacy boundary without treating it as current contract success.
|
||||
- **Must PASS with zero issues to proceed to Stage 5 (FINALIZE)**
|
||||
- **Stage 5 (FINALIZE) entry requires PASS with zero issues, or — after the 3-round Integrity Check FAIL Loop — an explicit, recorded user decision on the listed unresolved items**
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -639,13 +639,13 @@ consumer.
|
||||
|-----------|----------------------|-----------------|----------------|
|
||||
| Stage 1 -> 2 | RQ Brief, Methodology Blueprint, Annotated Bibliography, Synthesis Report | Schema 1 (RQ Brief), Schema 2 (Bibliography), Schema 3 (Synthesis) | deep-research handoff protocol; when active, separately carry the #683 context/#684 binding pointer named by the preceding lifecycle |
|
||||
| Stage 2 -> 2.5 | Complete Paper Draft + #547 scope context for Phase E4 (RQ Brief `scope` — the required E4 input; `sub_question_bindings` + outline section→sub-question map when present) + the Schema 2 Annotated Bibliography (#548 — `search_strategy` is the E5 comparison basis; `sources[].relevance` + `relevance_score` ground the nearest-prior-work check), when one exists + unchanged #684 binding pointer/receipts when active | Schema 4 (Paper Draft) + Schema 1 scope fields + Schema 2 (search_strategy + source relevance metadata) + review-target contracts | Pass to integrity_verification_agent; integrity does not consume criteria as a verdict input |
|
||||
| Stage 2.5 -> 3 | Verified Paper Draft + Integrity Report + E6 finding-set companion and, when findings exist, `authorized_to_continue` disposition sidecar + unchanged #684 manifest/context/brief when active | Schema 4 + Schema 5 + `claim-strength-drift-findings/1.0` + conditional `claim-strength-drift-disposition/1.0` + review-target contracts | Pass only after E6 has no findings or every reported row has explicit authorization; restoration/pause does not transfer the current draft. Carry forward `experiment_provenance[]` + `experiment_alignment_results[]` + `experiment_intake_declaration` (#260); the integrity verdict never consumes criteria binding |
|
||||
| Stage 2.5 -> 3 | Stage 2.5 Paper Draft (verified, or carrying the recorded Integrity Check FAIL Loop partially-unverified warning) + Integrity Report + E6 finding-set companion and, when findings exist, `authorized_to_continue` disposition sidecar + unchanged #684 manifest/context/brief when active | Schema 4 + Schema 5 + `claim-strength-drift-findings/1.0` + conditional `claim-strength-drift-disposition/1.0` + review-target contracts | Pass only after E6 has no findings or every reported row has explicit authorization; restoration/pause does not transfer the current draft. Carry forward `experiment_provenance[]` + `experiment_alignment_results[]` + `experiment_intake_declaration` (#260); the integrity verdict never consumes criteria binding |
|
||||
| Stage 3 -> **coaching** -> 4 | Editorial Decision, immutable Revision Roadmap, exact claim surfaces, 5 Review Reports, and the Schema 6 closed `review_panel_provenance` carrier; coaching adds the complete explicit author sidecar without mutating the Roadmap | Schema 6 + `revision-roadmap/1.0` + `claim-surface-manifest/1.0` + `author-adjudication/1.0` | For `reviewer_full`, verify the provenance artifact raw digest and deterministic replay before transfer; preserve its valid/invalid carrier byte-for-byte. Source-ordered dialogue records one explicit author choice per item, exact targets, and any exact claim/collateral authority -> revision mode |
|
||||
| Stage 4 -> 3' | Revised Draft, hard-required Original (pre-revision) Draft (the #576 1.1 §3.1 Phase 2A input; the required bundle already carries the exact matched round's pre draft, so declaring it absent is `manifest_incomplete`, never a `first_link_not_run` degradation), Response to Reviewers + Editorial Decision Letter (display only) + the Round-1 Schema 6 `review_panel_provenance` carrier and exact artifact bytes + the Round-1 review findings (the Schema 6 review reports the roadmap items trace to — the #576 §4 level-3 criterion layer; absent → transported Schema 7 fields alone, `[ROUND1-FINDINGS-ABSENT]`) + the Round-1 Revision Roadmap being verified + every ordered apply report and paired revision patch/diff file (`<output>.apply-report.json`, the sidecar beside each revised draft, #390; the manifest pair list must exactly equal the fully replayed bundle's ordered write-round projection, the FIRST report's `base_draft_hash` must equal the Original Draft hash prefix, every inner link must join, and only the LAST output hash may equal the Revised Draft hash prefix; any omission, substitution, reorder, or broken link → `manifest_hash_mismatch`) + the Round-1 Reviewer Configuration Cards (yardstick continuity — field_analyst is NOT re-run at Stage 3'; `re_review_mode_protocol.md` § Yardstick Continuity) + unchanged #684 target-review authority when active | Schema 4 (revised + original) + Schema 8 (Response to Reviewers) + Schema 6 (letter + Round-1 review reports + provenance carrier) + raw provenance artifact + Schema 7 (Roadmap, machine-form JSON — § Stage 3' Re-Review Contract Dispatch producer obligations) + apply-report sidecar JSON + revision patch JSON + configuration cards (no numbered schema) + review-target contracts | Before re-review, verify the carrier's raw artifact digest and deterministic replay; on any absent/unreachable/digest/schema/replay failure use the closed invalid state with six unknown axes, never letter reconstruction. Pass to reviewer (marked as verification round) under § Stage 3' Re-Review Contract Dispatch. This row is the re-review-mode transfer — the default Stage 3'. When the user explicitly requests a fresh full review at 3' instead (mid-entry quick→full path: no Schema 7 Roadmap or Round-1 cards exist), transfer the Revised Draft + available context only, dispatch full mode (field_analyst runs by definition), and do NOT mark it a verification round. A changed target requires a new non-comparable target review id |
|
||||
| Stage 3' -> **coaching** -> 4' | New Revision Roadmap (if Major) | #670 authority family + `shared/contracts/re_review/traceability.schema.json` | Pass the immutable roadmap, exact claim surfaces, traceability sidecar, and new complete author sidecar to revision mode; coaching uses a source-ordered explicit author checkpoint, and prior-round choices are never inferred or carried forward |
|
||||
| Stage 3' -> 4.5 | (Accept/Minor direct path — no Stage 4' between) Verified Revised Draft + the traceability sidecar with its frozen `previously_missed`/`indeterminate` new-issue records (#576 §8 — Material Passport cargo consumed by the Stage 4.5 gate) | Schema 4 (revised) + traceability sidecar | Pass to integrity_verification_agent (final verification); the frozen records are gate INPUT, not just cargo |
|
||||
| Stage 4/4' -> 4.5 | Revised/Re-Revised Draft + #547/#548 context + complete validated `revision-evidence-bundle/1.0` from exact integrity PASS through every review write/no-op/integrity round + (Major-via-4' path) the Stage 3' traceability sidecar with its frozen `previously_missed`/`indeterminate` new-issue records | Schema 4 + #670 bundle + traceability sidecar | Pass to integrity_verification_agent; registered surfaces are replayed, while the explicit unregistered-claim boundary remains mandatory E6 review input |
|
||||
| Stage 4.5 -> 5 | Final Verified Draft + Final Integrity Report + E6 finding-set companion and, when findings exist, `authorized_to_continue` disposition sidecar + exact preregistration sidecar/companion + independent #660/#672 results | Schema 4 + Schema 5 + E6 finding/disposition contracts + `preregistration-artifact/1.0`; independent advisory schemas | Refuse transfer while E6 derives `restore_required` or `paused`. After E6 closure, at the one mandatory entry checkpoint run #660 then #672 on identical accepted-draft ID/SHA and surface both without changing routing. On confirmation: Produce MD -> DOCX via Pandoc when available (otherwise instructions) -> ask about LaTeX -> confirm -> PDF. Carry forward `experiment_alignment_results[]` + `experiment_intake_declaration` (#260) to formatter surface + Stage 6 histogram |
|
||||
| Stage 4.5 -> 5 | Final Accepted Draft (verified, or carrying the recorded Integrity Check FAIL Loop partially-unverified warning) + Final Integrity Report + E6 finding-set companion and, when findings exist, `authorized_to_continue` disposition sidecar + exact preregistration sidecar/companion + independent #660/#672 results | Schema 4 + Schema 5 + E6 finding/disposition contracts + `preregistration-artifact/1.0`; independent advisory schemas | Refuse transfer while E6 derives `restore_required` or `paused`. After E6 closure, at the one mandatory entry checkpoint run #660 then #672 on identical accepted-draft ID/SHA and surface both without changing routing. On confirmation: Produce MD -> DOCX via Pandoc when available (otherwise instructions) -> ask about LaTeX -> confirm -> PDF. Carry forward `experiment_alignment_results[]` + `experiment_intake_declaration` (#260) to formatter surface + Stage 6 histogram |
|
||||
| Stage 5 -> 6 | Final deliverables list + Process-Summary projection of pipeline state history and agent logs, explicitly omitting the #673 activity projection of terminal root `run_id`, pending/sealed activity fields, selected-store data, renderer output, and diagnostics | — (Process Record; no numbered schema) | Dispatched only after the user confirms the Stage 5 completion checkpoint (FULL). User may decline Stage 6 there: mark it `skipped`, set pipeline state `completed`. Protocol: `../references/process_summary_protocol.md`; terminal semantics: `../references/pipeline_state_machine.md` § Stage 6 terminal semantics |
|
||||
|
||||
**#672 sidecar continuity:** At Stage 1, this shell-capable orchestrator alone
|
||||
@@ -877,7 +877,7 @@ AI/author origin, paper-mill production, misconduct, contextual validity,
|
||||
precision/recall, false-positive/false-negative rate, list coverage, or
|
||||
publisher acceptance.
|
||||
|
||||
This advisory does not alter Stage 4.5 PASS, the mandatory Stage-5 checkpoint,
|
||||
This advisory does not alter the Stage 4.5 verdict, the mandatory Stage-5 checkpoint,
|
||||
or any terminal-policy state. It never edits the draft or suggests replacement
|
||||
text. Surface the validated report and let the user preserve, revise, or
|
||||
proceed. A revision changes the input bytes, invalidates the current advisory,
|
||||
@@ -901,7 +901,8 @@ and never authorizes this read-only orchestrator to mutate a passport in place.
|
||||
## Cross-Document Consistency Advisory Dispatch (#672)
|
||||
|
||||
At the same single mandatory Stage-5 entry checkpoint, after the same exact
|
||||
Stage 4.5 PASS, run #660 first and #672 second. Both use the identical designated
|
||||
Stage 4.5 terminal resolution (PASS, or a recorded Integrity Check FAIL Loop
|
||||
continuation), run #660 first and #672 second. Both use the identical designated
|
||||
accepted draft. Enforce this exact machine join before either carrier is shown:
|
||||
|
||||
```text
|
||||
@@ -941,8 +942,9 @@ redacted `ADVISORY_UNAVAILABLE:<CODE>` diagnostic. Neither result blocks or
|
||||
delays the checkpoint or requires remediation before Stage 5.
|
||||
|
||||
Any manuscript revision stales both carriers. Return through existing integrity
|
||||
review to a fresh exact Stage 4.5 PASS, then rerun #660 followed by #672 on the
|
||||
new accepted bytes. Reusing either old carrier or rerunning only one is invalid
|
||||
review to a fresh exact Stage 4.5 terminal resolution (PASS, or a recorded
|
||||
FAIL-loop continuation), then rerun #660 followed by #672 on the new accepted
|
||||
bytes. Reusing either old carrier or rerunning only one is invalid
|
||||
handoff cargo. Rendering is replay-first, one explicit page of at most 25, with
|
||||
no `--all`. See
|
||||
`shared/references/cross_document_consistency_advisory_protocol.md`.
|
||||
|
||||
@@ -485,12 +485,12 @@ Check whether prerequisite materials for entering the specified stage are availa
|
||||
| Stage 1 | None (can start from scratch) | User-provided topic/direction |
|
||||
| Stage 2 | None (but Stage 1 output recommended) | RQ Brief, Methodology Blueprint, Bibliography, Synthesis |
|
||||
| Stage 2.5 | Paper Draft | -- |
|
||||
| Stage 3 | **Verified Paper Draft + Integrity Report (Pre)** | -- |
|
||||
| Stage 3 | **Verified Paper Draft + Integrity Report (Pre)** — or, on a recorded Integrity Check FAIL Loop continuation, the Stage 2.5 draft + Integrity Report (Pre) carrying the partially-unverified warning | -- |
|
||||
| Stage 4 | Review Reports + Revision Roadmap | Paper Draft |
|
||||
| Stage 3' | Revised Draft + hard-required Original pre-revision Draft + Round-1 Revision Roadmap + exact author-adjudication sidecar + fully replayed Revision-Evidence Bundle (current contract re-review; not required for an explicitly requested fresh full review) | Response to Reviewers; Editorial Decision Letter; Round-1 findings; exact ordered apply report/patch pairs matching the bundle projection; Round-1 Reviewer Configuration Cards. Missing any current hard-required artifact or a mismatched pair is `manifest_incomplete`, not a warning-only degradation. |
|
||||
| Stage 4' | Re-Review Report (Decision: Major) | Revised Draft |
|
||||
| Stage 4.5 | Revised Draft or Re-Revised Draft | -- |
|
||||
| Stage 5 | **Integrity Report (Final) — verdict: PASS** | -- |
|
||||
| Stage 5 | **Integrity Report (Final) — verdict: PASS, or FAIL with a recorded Integrity Check FAIL Loop resolution** | -- |
|
||||
| Stage 6 | None (Final Paper already delivered at Stage 5) | Pipeline state history + dialogue_log_ref ranges |
|
||||
|
||||
**Return format:**
|
||||
|
||||
@@ -41,7 +41,8 @@ Execution steps:
|
||||
4. Result handling:
|
||||
- PASS (zero issues) -> checkpoint -> Stage 5
|
||||
- FAIL -> fix -> re-verify -> PASS -> Stage 5
|
||||
5. ⚠️ **IRON RULE**: Must PASS with zero issues to proceed to Stage 5
|
||||
- FAIL after 3 correction rounds -> Integrity Check FAIL Loop (`pipeline_state_machine.md`): unresolved items listed, user decision recorded
|
||||
5. ⚠️ **IRON RULE**: Stage 5 entry requires PASS with zero issues, or — only after the 3-round FAIL loop is exhausted — an explicit, recorded user decision on the listed unresolved items; unresolved items are never silently dropped
|
||||
```
|
||||
|
||||
## Tortured-Phrase Advisory Boundary (#660)
|
||||
|
||||
@@ -159,7 +159,7 @@ This document defines all legal states, transition conditions, transition action
|
||||
| checkpoint | Stage 2 | User confirms | handoff RQ Brief + Methodology Blueprint + Bibliography + Synthesis |
|
||||
| Stage 2 | **checkpoint** | Stage 2 completed, Paper Draft produced | Wait for user confirmation |
|
||||
| checkpoint | Stage 2.5 | User confirms | Pass Paper Draft to integrity agent |
|
||||
| Stage 2.5 | **checkpoint** | PASS | Wait for user confirmation |
|
||||
| Stage 2.5 | **checkpoint** | PASS, or recorded Integrity Check FAIL Loop resolution (§ below) | Wait for user confirmation |
|
||||
| Stage 2.5 | Stage 2.5 (retry) | FAIL | Fix issues, re-verify (max 3 rounds) |
|
||||
| checkpoint | Stage 3 | User confirms | Pass verified paper to reviewer |
|
||||
| Stage 3 | **checkpoint** | Decision produced | Wait for user confirmation |
|
||||
@@ -172,7 +172,7 @@ This document defines all legal states, transition conditions, transition action
|
||||
| checkpoint | Stage 4' | Decision = Major, user confirms | Pass new Revision Roadmap |
|
||||
| Stage 4' | **checkpoint** | Stage 4' completed | Wait for user confirmation |
|
||||
| checkpoint | Stage 4.5 | User confirms | Pass revised draft to final verification |
|
||||
| Stage 4.5 | **checkpoint** | PASS (zero issues) | Wait for user confirmation |
|
||||
| Stage 4.5 | **checkpoint** | PASS (zero issues), or recorded Integrity Check FAIL Loop resolution (§ below) | Wait for user confirmation |
|
||||
| Stage 4.5 | Stage 4.5 (retry) | FAIL | Fix issues, re-verify (max 3 rounds) |
|
||||
| checkpoint | Stage 5 | User confirms (MANDATORY — the Stage 5 entry gate; see § Stage 5 boundary semantics) | Pass final accepted draft; record the finalization-format decision (citation style) |
|
||||
| Stage 5 | **checkpoint** | Stage 5 completed, Final Paper delivered | Wait for user confirmation (FULL — never SLIM; see § Stage 5 boundary semantics) |
|
||||
|
||||
@@ -10,6 +10,6 @@ Stage-specific reinforcement content for the Mid-Conversation Reinforcement Prot
|
||||
| Stage 3→4 | IRON RULE: Max 2 revision loops. Anti-Pattern: Sycophantic revision. |
|
||||
| Stage 4→3' | IRON RULE: Each concern independently verified. Anti-Pattern: Rubber-stamp re-review. |
|
||||
| Stage 3'→4' | IRON RULE: Max 2 revision loops. Anti-Pattern: Silently dropping reviewer concerns. |
|
||||
| Stage 4/4'→4.5 | IRON RULE: Must PASS with zero issues. Anti-Pattern: Re-verifying only known issues. |
|
||||
| Stage 4/4'→4.5 | IRON RULE: PASS with zero issues, or a recorded user decision via the Integrity Check FAIL Loop. Anti-Pattern: Re-verifying only known issues. |
|
||||
| Stage 4.5→5 | IRON RULE: PDF from LaTeX only. Anti-Pattern: Orchestrator doing substantive work. |
|
||||
| Any FULL/SLIM checkpoint | IRON RULE: `collaboration_depth_agent` output is **advisory only** and never blocks progression. Anti-Pattern: treating the observer's Zone/scores as a gate or a leaderboard. |
|
||||
|
||||
@@ -56,9 +56,9 @@ For each stage transition, the following handoff procedure applies:
|
||||
|------|--------|
|
||||
| **Who hands off** | Integration Lead |
|
||||
| **Who receives** | Review Coordinator |
|
||||
| **Materials** | Verified Paper Draft + Integrity Report (Schema 5) |
|
||||
| **Approval needed** | Integrity verdict is PASS; any PASS_WITH_CONDITIONS items acknowledged |
|
||||
| **Handoff checklist** | Integrity Report attached; all SERIOUS/MEDIUM issues resolved |
|
||||
| **Materials** | Stage 2.5 Paper Draft (verified, or carrying the recorded FAIL-loop warning) + Integrity Report (Schema 5) |
|
||||
| **Approval needed** | Integrity verdict is PASS (any PASS_WITH_CONDITIONS items acknowledged), or the recorded Integrity Check FAIL Loop user decision |
|
||||
| **Handoff checklist** | Integrity Report attached; all SERIOUS/MEDIUM issues resolved, or covered by the recorded FAIL-loop decision |
|
||||
|
||||
### Stage 3 -> Stage 4 (Review -> Revise)
|
||||
|
||||
@@ -86,9 +86,9 @@ For each stage transition, the following handoff procedure applies:
|
||||
|------|--------|
|
||||
| **Who hands off** | Integration Lead |
|
||||
| **Who receives** | Lead Author |
|
||||
| **Materials** | Final Verified Draft + Final Integrity Report |
|
||||
| **Approval needed** | Integrity verdict PASS with zero issues |
|
||||
| **Handoff checklist** | All previous integrity issues confirmed resolved; Material Passport updated to VERIFIED |
|
||||
| **Materials** | Final Draft (verified, or carrying the recorded partially-unverified warning) + Final Integrity Report |
|
||||
| **Approval needed** | Integrity verdict PASS with zero issues (or the recorded Integrity Check FAIL Loop user decision) |
|
||||
| **Handoff checklist** | All previous integrity issues resolved, or covered by the recorded FAIL-loop decision with the partially-unverified warning carried forward; Material Passport `verification_status` set to `VERIFIED` only on a zero-issue PASS (a FAIL-loop continuation keeps `UNVERIFIED`) |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -619,3 +619,7 @@ path = "scripts/test_pdf_read_preflight.py"
|
||||
[[pytest]]
|
||||
id = "745-stage-capability-matrix"
|
||||
path = "scripts/test_check_stage_capability_matrix.py"
|
||||
|
||||
[[pytest]]
|
||||
id = "753-distribution-surface-claims"
|
||||
path = "scripts/test_check_distribution_surface_claims.py"
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Distribution-surface claim discipline lint (#753).
|
||||
|
||||
The #745 capability matrix pins README claim anchors, and version-consistency
|
||||
invariant 8 binds the plugin.json agent-count number to the tree — but the
|
||||
2026-08-17 ISO 42001-spirit audit (T-1) found the marketing surfaces
|
||||
themselves (`.claude-plugin/plugin.json` / `.claude-plugin/marketplace.json`
|
||||
description fields) sat outside every lock, so their language had silently
|
||||
outrun the evidence record ("Production-grade", "39-agent ensemble" against 8
|
||||
`NOT_RUN` task families and 3 plugin-exposed agents). This lint closes that
|
||||
surface so the same drift class fails CI instead of shipping. It lives apart
|
||||
from check_version_consistency.py on purpose: claim language is not
|
||||
version-scoped, and that script's manifest checks deliberately fail OPEN on
|
||||
these conditions (invariant 8 defers absence to invariant 4) where this lint
|
||||
must fail CLOSED.
|
||||
|
||||
D1. Both distribution manifests exist and parse as JSON (fail-closed: a
|
||||
missing or unparseable manifest is an error, never a silent pass —
|
||||
the invariant-4 lesson: deleting or renaming a claim surface must not
|
||||
disable its check).
|
||||
D2. Every description field (plugin.json top-level; marketplace.json
|
||||
top-level and each plugins[] entry) is a non-empty string.
|
||||
D3. No description carries claim language the evidence record does not
|
||||
license. Descriptions are marketing surfaces, never evidence-carrying
|
||||
matrix rows, so the capability matrix's effectiveness vocabulary
|
||||
(imported from check_stage_capability_matrix so a matrix-side stem
|
||||
addition binds this surface in the same commit) is refused
|
||||
unconditionally — a description cannot cite the MEASURED-row
|
||||
licensing the matrix's M6 check honors — as is any percentage figure
|
||||
(the matrix's unmeasured-percentage rule, `_PERCENT_RE`; the T-3
|
||||
"31% -> ~5-10%" class), plus the two surface-specific phrases the
|
||||
audit removed: "production-grade" and "agent ensemble"
|
||||
(execution/error independence the default inline path does not
|
||||
provide). Word-boundary matching, not semantic parsing (the M6
|
||||
convention; see the semantics note at FORBIDDEN_PATTERNS).
|
||||
D4. The plugin.json description carries at least one agent-count token
|
||||
in a spelling invariant 8 can bind ("N-agent" / "N prompt roles",
|
||||
canonical casing). Invariant 8 is opt-in by spelling — a reworded
|
||||
or dropped count silently detaches the number from the tree (#414's
|
||||
drift class, nearly reintroduced by the #753 reword itself) — so
|
||||
the presence half fails closed here while the value half stays
|
||||
invariant 8's job.
|
||||
D5. Any "N plugin-exposed" count in a description equals the
|
||||
check_agents_mirror_sync MIRRORS roster size — the same #414 drift
|
||||
class for the second number the #753 description introduced.
|
||||
|
||||
The manifests are also read by check_version_consistency.py (invariants 4
|
||||
and 8); the double-report on a missing/malformed manifest is intentional —
|
||||
per-script fail-closed loaders are the repo convention, and this lint must
|
||||
not inherit invariant 8's silent-pass posture.
|
||||
|
||||
Usage:
|
||||
python3 scripts/check_distribution_surface_claims.py [--root PATH]
|
||||
|
||||
Exit 0 = all invariants hold; exit 1 = violations (listed on stderr).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_SCRIPTS_DIR = Path(__file__).resolve().parent
|
||||
if str(_SCRIPTS_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(_SCRIPTS_DIR))
|
||||
|
||||
from check_agents_mirror_sync import MIRRORS # noqa: E402
|
||||
from check_stage_capability_matrix import ( # noqa: E402
|
||||
_MEASURED_LICENSED_STEMS,
|
||||
_NEVER_LICENSED_STEMS,
|
||||
_PERCENT_RE,
|
||||
)
|
||||
from check_version_consistency import AGENT_CLAIM_RE # noqa: E402
|
||||
|
||||
REPO_ROOT = _SCRIPTS_DIR.parent
|
||||
|
||||
# Matching-semantics note: the matrix's M6 check scans row prose with plain
|
||||
# case-insensitive SUBSTRING matching; descriptions are short marketing
|
||||
# strings, so this surface uses word-boundary regexes instead — same
|
||||
# vocabulary, deliberately stricter about false fires ("proven" must not
|
||||
# fire inside "provenance", pinned by test). Stems that are complete words
|
||||
# take a trailing boundary; the rest are prefix stems (improv*, guarantee*).
|
||||
_WHOLE_WORD_STEMS = frozenset({"proven"})
|
||||
|
||||
# Surface-specific phrases the 2026-08-17 audit removed; not part of the
|
||||
# matrix vocabulary because they are only over-claims on a distribution
|
||||
# surface (a matrix row could legitimately discuss an "ensemble" design).
|
||||
_SURFACE_ONLY_PHRASES = (
|
||||
("production-grade", r"\bproduction[- ]grade\b"),
|
||||
("agent ensemble", r"\bagent ensembles?\b"),
|
||||
)
|
||||
|
||||
FORBIDDEN_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = tuple(
|
||||
(
|
||||
stem,
|
||||
re.compile(
|
||||
r"\b" + re.escape(stem)
|
||||
+ (r"\b" if stem in _WHOLE_WORD_STEMS else ""),
|
||||
re.IGNORECASE,
|
||||
),
|
||||
)
|
||||
for stem in _NEVER_LICENSED_STEMS + _MEASURED_LICENSED_STEMS
|
||||
) + tuple(
|
||||
(label, re.compile(pattern, re.IGNORECASE))
|
||||
for label, pattern in _SURFACE_ONLY_PHRASES
|
||||
)
|
||||
|
||||
_PLUGIN_EXPOSED_RE = re.compile(r"(\d+)\s+plugin-exposed", re.IGNORECASE)
|
||||
|
||||
|
||||
def _reject_json_constant(value: str) -> None:
|
||||
"""`json.loads` accepts non-standard NaN/Infinity by default; strict
|
||||
distribution consumers reject them, so D1 must too (fail-closed)."""
|
||||
raise ValueError(f"non-standard JSON constant {value!r}")
|
||||
|
||||
|
||||
def _load_manifest(path: Path, errors: list[str]) -> dict | None:
|
||||
"""D1: fail-closed manifest load."""
|
||||
if not path.is_file():
|
||||
errors.append(f"D1: {path} is missing — the claim surface cannot "
|
||||
f"silently leave this lint's coverage")
|
||||
return None
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"),
|
||||
parse_constant=_reject_json_constant)
|
||||
except (json.JSONDecodeError, UnicodeDecodeError, ValueError) as exc:
|
||||
errors.append(f"D1: {path} failed to parse as JSON ({exc})")
|
||||
return None
|
||||
if not isinstance(data, dict):
|
||||
errors.append(f"D1: {path} top level is not a JSON object")
|
||||
return None
|
||||
return data
|
||||
|
||||
|
||||
def _check_description(surface: str, value, errors: list[str]) -> None:
|
||||
"""D2 shape + D3 claim-language discipline + D5 plugin-exposed binding
|
||||
for one description field."""
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
errors.append(f"D2: {surface} description is missing, empty, or "
|
||||
f"not a string")
|
||||
return
|
||||
for label, pattern in FORBIDDEN_PATTERNS:
|
||||
if pattern.search(value):
|
||||
errors.append(
|
||||
f"D3: {surface} description contains unlicensed claim "
|
||||
f"language ({label!r}) — distribution surfaces must stay "
|
||||
f"within the evidence record "
|
||||
f"(docs/STAGE_CAPABILITY_MATRIX.md); see #753"
|
||||
)
|
||||
# _PERCENT_RE spells "percent" lowercase and its matrix caller lowercases
|
||||
# its input first — mirror that here so "31 Percent" cannot slip through.
|
||||
if _PERCENT_RE.search(value.lower()):
|
||||
errors.append(
|
||||
f"D3: {surface} description carries a percentage figure — "
|
||||
f"descriptions can never carry the measurement provenance the "
|
||||
f"capability matrix requires for numeric claims; see #753"
|
||||
)
|
||||
for m in _PLUGIN_EXPOSED_RE.finditer(value):
|
||||
if int(m.group(1)) != len(MIRRORS):
|
||||
errors.append(
|
||||
f"D5: {surface} description claims {m.group(1)} "
|
||||
f"plugin-exposed agents but the check_agents_mirror_sync "
|
||||
f"MIRRORS roster has {len(MIRRORS)}"
|
||||
)
|
||||
|
||||
|
||||
def run(root: Path) -> list[str]:
|
||||
errors: list[str] = []
|
||||
plugin_dir = root / ".claude-plugin"
|
||||
|
||||
plugin = _load_manifest(plugin_dir / "plugin.json", errors)
|
||||
if plugin is not None:
|
||||
description = plugin.get("description")
|
||||
_check_description("plugin.json", description, errors)
|
||||
if isinstance(description, str) and not AGENT_CLAIM_RE.search(
|
||||
description
|
||||
):
|
||||
errors.append(
|
||||
"D4: plugin.json description carries no agent-count token "
|
||||
"invariant 8 can bind ('N-agent' / 'N prompt roles', "
|
||||
"canonical casing) — a reworded or dropped count detaches "
|
||||
"the advertised number from the tree (#414); see #753"
|
||||
)
|
||||
|
||||
marketplace = _load_manifest(plugin_dir / "marketplace.json", errors)
|
||||
if marketplace is not None:
|
||||
_check_description(
|
||||
"marketplace.json", marketplace.get("description"), errors
|
||||
)
|
||||
plugins = marketplace.get("plugins")
|
||||
if not isinstance(plugins, list) or not plugins:
|
||||
errors.append("D2: marketplace.json plugins[] is missing or "
|
||||
"empty — the per-plugin description surface cannot "
|
||||
"silently leave this lint's coverage")
|
||||
else:
|
||||
for i, entry in enumerate(plugins):
|
||||
if not isinstance(entry, dict):
|
||||
errors.append(f"D2: marketplace.json plugins[{i}] is "
|
||||
f"not an object")
|
||||
continue
|
||||
_check_description(
|
||||
f"marketplace.json plugins[{i}]",
|
||||
entry.get("description"),
|
||||
errors,
|
||||
)
|
||||
return errors
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=REPO_ROOT)
|
||||
args = parser.parse_args(argv)
|
||||
errors = run(args.root)
|
||||
if errors:
|
||||
for e in errors:
|
||||
print(f"check_distribution_surface_claims: {e}", file=sys.stderr)
|
||||
return 1
|
||||
print("PASSED: check_distribution_surface_claims — distribution-surface "
|
||||
"descriptions stay within the evidence record")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -70,10 +70,10 @@ REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
# reviewed against the #528 resolutions.
|
||||
# ---------------------------------------------------------------------------
|
||||
CONTENT_LOCKS = {
|
||||
"academic-pipeline/SKILL.md": "4b69f08e4786feb530465a282c83e7cd7f39435a95ed4758efa68b902ebdfdc5",
|
||||
"academic-pipeline/agents/pipeline_orchestrator_agent.md": "05846f7ed70cb2f7e8828affd6ccc5f611f2285b539b57a91f693ccf234d2edb",
|
||||
"academic-pipeline/agents/state_tracker_agent.md": "59455856e8186192d5f07dbd5aaf4e71db427158856eae7b6658544cb488fd3c",
|
||||
"academic-pipeline/references/pipeline_state_machine.md": "a9bfad44e24f6553cb42b77190eea64e2db796d9caeae08bd3df233cb4ef93e6",
|
||||
"academic-pipeline/SKILL.md": "5754d20341aa4aa8f80a836981b7ada9d08eceb42ba9f5a2b8e9fb82dcaec040",
|
||||
"academic-pipeline/agents/pipeline_orchestrator_agent.md": "b93427c8ab033d0cb8b396ef86100afdedfd2fe96abfe4522fbd16d311241488",
|
||||
"academic-pipeline/agents/state_tracker_agent.md": "2716bab5686a6129777f595ad86bf1e1cc01fa5d8d1ec192fa8880018dfe968a",
|
||||
"academic-pipeline/references/pipeline_state_machine.md": "70872764d04a1cabdc1a342cbaf66ebed98a3764d506136d541c1880759510e7",
|
||||
"academic-pipeline/references/process_summary_protocol.md": "1052d8cb8ee00c1cd0fcc70a18aee5a0f92db2ebe0a74930b04d4b05d888cfdf",
|
||||
}
|
||||
|
||||
|
||||
@@ -20,7 +20,8 @@ Invariants enforced:
|
||||
7. Every version-bearing H2 heading in docs/<name>.md has a matching
|
||||
version-bearing H2 (same version) in docs/<name>.zh-TW.md and vice versa.
|
||||
Plain headings may differ — only version TAGS must stay in lockstep.
|
||||
8. The plugin.json description's "N-agent" claim (when present) equals the
|
||||
8. The plugin.json description's agent-count claim (when present; both the
|
||||
legacy "N-agent" and the #753 "N prompt roles" spellings) equals the
|
||||
number of unique *_agent.md files in the tree (#414: the advertised
|
||||
number had silently drifted from the tree). The plugin-root agents/
|
||||
mirror dir is excluded from the count — real byte-identical copies of
|
||||
@@ -99,8 +100,14 @@ H2_VERSION_RE = re.compile(r"(?<![\w.])v" + _VSEG + r"(?![\d.\-A-Za-z])")
|
||||
|
||||
NON_VERSION_CHANGELOG_TOKENS = frozenset({"Unreleased"})
|
||||
|
||||
# Invariant 8: the outward-facing agent-count claim, e.g. "38-agent ensemble".
|
||||
AGENT_CLAIM_RE = re.compile(r"(\d+)-agent")
|
||||
# Invariant 8: the outward-facing agent-count claim. Two licensed spellings:
|
||||
# the legacy "38-agent" form and the #753 evidence-aligned "39 prompt roles"
|
||||
# form (exact plural). Both bind the stated number to the tree's *_agent.md
|
||||
# count, so a reworded claim surface cannot detach the number from the
|
||||
# inventory. Trailing \b keeps near-miss spellings ("39-agentic",
|
||||
# "39 prompt role") from counting as bound — check_distribution_surface_claims
|
||||
# D4 then flags them as unbindable.
|
||||
AGENT_CLAIM_RE = re.compile(r"(\d+)(?:-agent\b|\s+prompt\s+roles\b)")
|
||||
|
||||
# Invariant 9: minimum body length (chars, after strip) for the latest entry.
|
||||
CHANGELOG_BODY_MIN_CHARS = 100
|
||||
@@ -642,15 +649,17 @@ def _check_key_additions(
|
||||
|
||||
def _check_agent_count_claim(root: Path) -> list[str]:
|
||||
"""Invariant 8 (#414): when plugin.json's description advertises an
|
||||
"N-agent" count, N must equal the number of unique *_agent.md files in
|
||||
the tree. The plugin-root agents/ mirror dir is excluded — its files are
|
||||
agent count ("N-agent" or, since #753, "N prompt roles"), every such N
|
||||
must equal the number of unique *_agent.md files in the tree. The
|
||||
plugin-root agents/ mirror dir is excluded — its files are
|
||||
byte-identical aliases of deep-research agents (real copies since #413,
|
||||
symlinks before; check_agents_mirror_sync.py pins the byte-equality), so
|
||||
counting them would double-count. resolve() additionally dedups any
|
||||
remaining symlink alias. Missing/malformed manifest or a description
|
||||
without a count claim is NOT an invariant-8 error — the manifest problems
|
||||
are invariant 4's to report, and the claim is optional (only a stated
|
||||
number must be true)."""
|
||||
without a count claim is NOT an invariant-8 error — the manifest
|
||||
problems are invariant 4's to report, and here the claim is optional
|
||||
(only a stated number must be true; check_distribution_surface_claims
|
||||
D4 is where the presence of a bindable count fails closed)."""
|
||||
plugin_json = root / ".claude-plugin" / "plugin.json"
|
||||
if not plugin_json.is_file():
|
||||
return []
|
||||
@@ -661,23 +670,22 @@ def _check_agent_count_claim(root: Path) -> list[str]:
|
||||
description = data.get("description")
|
||||
if not isinstance(description, str):
|
||||
return []
|
||||
m = AGENT_CLAIM_RE.search(description)
|
||||
if m is None:
|
||||
claims = [int(m.group(1)) for m in AGENT_CLAIM_RE.finditer(description)]
|
||||
if not claims:
|
||||
return []
|
||||
claimed = int(m.group(1))
|
||||
actual = len({
|
||||
p.resolve()
|
||||
for p in root.rglob("*_agent.md")
|
||||
if ".git" not in p.parts
|
||||
and p.relative_to(root).parts[0] != "agents" # plugin-root mirror = aliases (#413)
|
||||
})
|
||||
if claimed != actual:
|
||||
return [
|
||||
f"{plugin_json}: description claims {claimed}-agent but the tree "
|
||||
f"has {actual} unique *_agent.md files (agents/ mirror aliases "
|
||||
f"excluded, symlinks deduplicated)"
|
||||
]
|
||||
return []
|
||||
return [
|
||||
f"{plugin_json}: description claims an agent count of {claimed} "
|
||||
f"but the tree has {actual} unique *_agent.md files (agents/ "
|
||||
f"mirror aliases excluded, symlinks deduplicated)"
|
||||
for claimed in claims
|
||||
if claimed != actual
|
||||
]
|
||||
|
||||
|
||||
def _check_citation_surfaces(
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
"""Mutation tests for check_distribution_surface_claims.py (#753)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(REPO_ROOT / "scripts") not in sys.path:
|
||||
sys.path.insert(0, str(REPO_ROOT / "scripts"))
|
||||
|
||||
from check_distribution_surface_claims import run # noqa: E402
|
||||
|
||||
CLEAN_PLUGIN = {
|
||||
"name": "fixture",
|
||||
"description": "Contract-audited academic research pipeline. "
|
||||
"4 skills, 27 modes, 39 prompt roles.",
|
||||
}
|
||||
CLEAN_MARKETPLACE = {
|
||||
"name": "fixture",
|
||||
"description": "Contract-audited research and writing skills.",
|
||||
"plugins": [
|
||||
{"name": "fixture", "description": "4 skills + 27 modes."},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _write(root: Path, plugin=CLEAN_PLUGIN, marketplace=CLEAN_MARKETPLACE,
|
||||
plugin_raw: str | None = None) -> Path:
|
||||
d = root / ".claude-plugin"
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
if plugin is not None or plugin_raw is not None:
|
||||
(d / "plugin.json").write_text(
|
||||
plugin_raw if plugin_raw is not None else json.dumps(plugin),
|
||||
encoding="utf-8",
|
||||
)
|
||||
if marketplace is not None:
|
||||
(d / "marketplace.json").write_text(
|
||||
json.dumps(marketplace), encoding="utf-8"
|
||||
)
|
||||
return root
|
||||
|
||||
|
||||
def _fires(errors: list[str], fragment: str) -> None:
|
||||
assert any(fragment in e for e in errors), (
|
||||
f"expected an error containing {fragment!r}, got: {errors}"
|
||||
)
|
||||
|
||||
|
||||
def test_shipped_manifests_pass():
|
||||
assert run(REPO_ROOT) == []
|
||||
|
||||
|
||||
def test_clean_synthetic_manifests_pass(tmp_path):
|
||||
assert run(_write(tmp_path)) == []
|
||||
|
||||
|
||||
def test_missing_plugin_json_fails_closed(tmp_path):
|
||||
_write(tmp_path, plugin=None)
|
||||
_fires(run(tmp_path), "D1")
|
||||
|
||||
|
||||
def test_missing_marketplace_json_fails_closed(tmp_path):
|
||||
_write(tmp_path, marketplace=None)
|
||||
_fires(run(tmp_path), "D1")
|
||||
|
||||
|
||||
def test_unparseable_plugin_json_fails_closed(tmp_path):
|
||||
_write(tmp_path, plugin_raw="{not json", plugin=None)
|
||||
_fires(run(tmp_path), "D1")
|
||||
|
||||
|
||||
def test_non_object_top_level_fails_closed(tmp_path):
|
||||
_write(tmp_path, plugin_raw='["a list"]', plugin=None)
|
||||
_fires(run(tmp_path), "D1")
|
||||
|
||||
|
||||
def test_nonstandard_json_nan_fails_closed(tmp_path):
|
||||
# json.loads accepts NaN by default; strict consumers reject it
|
||||
_write(tmp_path, plugin=None,
|
||||
plugin_raw='{"description": "39 prompt roles", "score": NaN}')
|
||||
_fires(run(tmp_path), "D1")
|
||||
|
||||
|
||||
def test_nonstandard_json_infinity_fails_closed(tmp_path):
|
||||
_write(tmp_path, plugin=None,
|
||||
plugin_raw='{"description": "39 prompt roles", "x": Infinity}')
|
||||
_fires(run(tmp_path), "D1")
|
||||
|
||||
|
||||
def test_missing_description_fires_d2(tmp_path):
|
||||
_write(tmp_path, plugin={"name": "fixture"})
|
||||
_fires(run(tmp_path), "D2")
|
||||
|
||||
|
||||
def test_empty_description_fires_d2(tmp_path):
|
||||
_write(tmp_path, plugin={"name": "fixture", "description": " "})
|
||||
_fires(run(tmp_path), "D2")
|
||||
|
||||
|
||||
def test_empty_plugins_list_fires_d2(tmp_path):
|
||||
_write(tmp_path, marketplace={"name": "f", "description": "ok",
|
||||
"plugins": []})
|
||||
_fires(run(tmp_path), "D2")
|
||||
|
||||
|
||||
def test_production_grade_fires_d3(tmp_path):
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f", "description": "Production-grade research pipeline."
|
||||
})
|
||||
_fires(run(tmp_path), "production-grade")
|
||||
|
||||
|
||||
def test_production_grade_space_spelling_fires_d3(tmp_path):
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f", "description": "A production grade pipeline."
|
||||
})
|
||||
_fires(run(tmp_path), "production-grade")
|
||||
|
||||
|
||||
def test_agent_ensemble_fires_d3(tmp_path):
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f", "description": "A 39-agent ensemble for research."
|
||||
})
|
||||
_fires(run(tmp_path), "agent ensemble")
|
||||
|
||||
|
||||
def test_effectiveness_stems_fire_d3(tmp_path):
|
||||
for phrase in ("guaranteed accuracy", "proven results",
|
||||
"state-of-the-art pipeline", "improves error rates",
|
||||
"outperforms manual review"):
|
||||
errors = run(_write(tmp_path, plugin={
|
||||
"name": "f", "description": f"A pipeline with {phrase}."
|
||||
}))
|
||||
_fires(errors, "D3")
|
||||
|
||||
|
||||
def test_provenance_does_not_false_fire(tmp_path):
|
||||
# word boundary: "proven" must not fire inside "provenance"
|
||||
assert run(_write(tmp_path, plugin={
|
||||
"name": "f",
|
||||
"description": "Citation provenance tracking — 39 prompt roles.",
|
||||
})) == []
|
||||
|
||||
|
||||
def test_marketplace_plugin_entry_description_covered(tmp_path):
|
||||
_write(tmp_path, marketplace={
|
||||
"name": "f", "description": "ok",
|
||||
"plugins": [{"name": "f",
|
||||
"description": "Production-grade skills."}],
|
||||
})
|
||||
_fires(run(tmp_path), "plugins[0]")
|
||||
|
||||
|
||||
def test_percentage_figure_fires_d3(tmp_path):
|
||||
# the T-3 class: "31% -> ~5-10%" style numbers on a marketing surface
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f",
|
||||
"description": "Cuts citation errors by 31% — 39 prompt roles.",
|
||||
})
|
||||
_fires(run(tmp_path), "percentage")
|
||||
|
||||
|
||||
def test_missing_count_token_fires_d4(tmp_path):
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f", "description": "Contract-audited research pipeline."
|
||||
})
|
||||
_fires(run(tmp_path), "D4")
|
||||
|
||||
|
||||
def test_unbindable_count_spelling_fires_d4(tmp_path):
|
||||
# "39 agents" is a spelling invariant 8 cannot bind — the number would
|
||||
# silently detach from the tree count
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f", "description": "A pipeline with 39 agents."
|
||||
})
|
||||
_fires(run(tmp_path), "D4")
|
||||
|
||||
|
||||
def test_plugin_exposed_count_drift_fires_d5(tmp_path):
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f",
|
||||
"description": "39 prompt roles (4 plugin-exposed agents).",
|
||||
})
|
||||
_fires(run(tmp_path), "D5")
|
||||
|
||||
|
||||
def test_plugin_exposed_count_matching_passes(tmp_path):
|
||||
assert run(_write(tmp_path, plugin={
|
||||
"name": "f",
|
||||
"description": "39 prompt roles (3 plugin-exposed agents).",
|
||||
})) == []
|
||||
|
||||
|
||||
def test_percentage_case_variant_fires_d3(tmp_path):
|
||||
# "_PERCENT_RE" is lowercase-only; the caller must lowercase first
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f",
|
||||
"description": "A 31 Percent reduction — 39 prompt roles.",
|
||||
})
|
||||
_fires(run(tmp_path), "percentage")
|
||||
|
||||
|
||||
def test_plugin_exposed_case_variant_fires_d5(tmp_path):
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f",
|
||||
"description": "39 prompt roles (4 Plugin-Exposed agents).",
|
||||
})
|
||||
_fires(run(tmp_path), "D5")
|
||||
|
||||
|
||||
def test_near_miss_spelling_agentic_fires_d4(tmp_path):
|
||||
# "39-agentic" must not count as a bound "N-agent" token
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f", "description": "A 39-agentic workflow suite."
|
||||
})
|
||||
_fires(run(tmp_path), "D4")
|
||||
|
||||
|
||||
def test_near_miss_spelling_singular_role_fires_d4(tmp_path):
|
||||
# "39 prompt role" (singular) is not the licensed plural spelling
|
||||
_write(tmp_path, plugin={
|
||||
"name": "f", "description": "A suite with 39 prompt role."
|
||||
})
|
||||
_fires(run(tmp_path), "D4")
|
||||
@@ -833,32 +833,49 @@ class TestAgentCountClaim(unittest.TestCase):
|
||||
f"# {name}\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
def test_agent_claim_drift_fails(self) -> None:
|
||||
def _claim_case(self, description: str) -> "subprocess.CompletedProcess":
|
||||
"""Run the lint against a two-agent fixture tree whose plugin.json
|
||||
carries the given description."""
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
_write_aligned_fixture(root)
|
||||
_write_plugin_manifests(
|
||||
root, "3.5.0", description="fixture, 3-agent ensemble, more"
|
||||
)
|
||||
_write_plugin_manifests(root, "3.5.0", description=description)
|
||||
self._write_agents(root, ["alpha", "beta"])
|
||||
result = _run(root)
|
||||
self.assertEqual(result.returncode, 1, msg=f"stdout={result.stdout!r}")
|
||||
self.assertIn("3-agent", result.stdout)
|
||||
self.assertIn("2", result.stdout)
|
||||
return _run(root)
|
||||
|
||||
def test_agent_claim_drift_fails(self) -> None:
|
||||
result = self._claim_case("fixture, 3-agent ensemble, more")
|
||||
self.assertEqual(result.returncode, 1, msg=f"stdout={result.stdout!r}")
|
||||
self.assertIn("agent count of 3", result.stdout)
|
||||
self.assertIn("2", result.stdout)
|
||||
|
||||
def test_prompt_roles_claim_drift_fails(self) -> None:
|
||||
"""#753: the "N prompt roles" spelling binds to the same tree count
|
||||
as the legacy "N-agent" spelling."""
|
||||
result = self._claim_case("fixture, 3 prompt roles, more")
|
||||
self.assertEqual(result.returncode, 1, msg=f"stdout={result.stdout!r}")
|
||||
self.assertIn("agent count of 3", result.stdout)
|
||||
|
||||
def test_prompt_roles_claim_matching_passes(self) -> None:
|
||||
result = self._claim_case("fixture, 2 prompt roles, more")
|
||||
self.assertEqual(
|
||||
result.returncode, 0,
|
||||
msg=f"stdout={result.stdout!r} stderr={result.stderr!r}",
|
||||
)
|
||||
|
||||
def test_agent_claim_matching_passes(self) -> None:
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
_write_aligned_fixture(root)
|
||||
_write_plugin_manifests(
|
||||
root, "3.5.0", description="fixture, 2-agent ensemble, more"
|
||||
)
|
||||
self._write_agents(root, ["alpha", "beta"])
|
||||
result = _run(root)
|
||||
self.assertEqual(
|
||||
result.returncode, 0,
|
||||
msg=f"stdout={result.stdout!r} stderr={result.stderr!r}",
|
||||
)
|
||||
result = self._claim_case("fixture, 2-agent ensemble, more")
|
||||
self.assertEqual(
|
||||
result.returncode, 0,
|
||||
msg=f"stdout={result.stdout!r} stderr={result.stderr!r}",
|
||||
)
|
||||
|
||||
def test_every_count_token_checked_not_just_first(self) -> None:
|
||||
"""#753: finditer semantics — a correct first token cannot shadow a
|
||||
drifted second one."""
|
||||
result = self._claim_case("fixture, 2-agent core, 5 prompt roles")
|
||||
self.assertEqual(result.returncode, 1, msg=f"stdout={result.stdout!r}")
|
||||
self.assertIn("agent count of 5", result.stdout)
|
||||
|
||||
def test_agent_claim_symlink_alias_not_double_counted(self) -> None:
|
||||
"""Legacy/transition pin: a symlink alias in the plugin-root agents/
|
||||
|
||||
@@ -33,7 +33,7 @@ reviewer/DA handoff and must not call this citation adapter with reviewer data.
|
||||
|
||||
A stress test of 68 AI-generated citations found 31% had problems — and all passed three rounds of same-model integrity checks. The root cause: the verifying AI and the generating AI share the same training data distribution, so they share the same blind spots. A different model (trained on overlapping but not identical data, with different RLHF tuning) can catch errors that the primary model systematically misses.
|
||||
|
||||
**What it improves:** Error rate reduction (estimated 31% → ~5-10%). Different models catch different types of hallucination patterns.
|
||||
**What it improves:** Different models catch different types of hallucination patterns. The post-verification error rate has never been measured — the residual-rate hypothesis (that cross-model checks cut the 31% above to single digits) is unvalidated.
|
||||
|
||||
**What it doesn't solve:** Frame-lock (all LLMs share most training data), sycophancy (all RLHF models have this tendency). These are degree improvements, not kind improvements.
|
||||
|
||||
|
||||
@@ -165,8 +165,13 @@ can reason about data-flow safety without reading every agent definition file.
|
||||
repo-tracked rubric files that describe output format or structural
|
||||
requirements — not answer keys, not expected content. For calibration gold
|
||||
sets, require the human researcher to supply a session file at runtime. Never
|
||||
bundle gold labels into the repository or reference them from `SKILL.md` in a
|
||||
way that loads them unconditionally.
|
||||
wire gold labels into operational agent context: no reference from `SKILL.md`
|
||||
or any always-loaded file may load them unconditionally. Repo-tracked gold
|
||||
sets under `evals/` are consistent with this boundary only when their content
|
||||
is synthetic or public-safe — evaluation harnesses (`scripts/run_evals.py`)
|
||||
read them on explicit invocation only. Calibration gold sets for live
|
||||
reviewer scoring remain runtime-supplied and are never committed (they may
|
||||
contain human or private data).
|
||||
|
||||
**DO: Pass scores back through a reviewer agent that holds the rubric
|
||||
privately.** The review workflow is: reviewer reads paper + rubric → reviewer
|
||||
|
||||
Reference in New Issue
Block a user