mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
fix(validate): make judgment contract executable
This commit is contained in:
boshu
+4
-10
@@ -346,7 +346,7 @@
|
||||
"path": "skills/validate/",
|
||||
"has_skill_md": true,
|
||||
"has_references": true,
|
||||
"reference_count": 23
|
||||
"reference_count": 24
|
||||
},
|
||||
{
|
||||
"name": "domain",
|
||||
@@ -666,9 +666,7 @@
|
||||
"path": "cli/cmd/ao/",
|
||||
"purpose": "Check AgentOps health",
|
||||
"bounded_context": "BC4",
|
||||
"driven_by_skills": [
|
||||
"validate"
|
||||
]
|
||||
"driven_by_skills": []
|
||||
},
|
||||
{
|
||||
"name": "ao done",
|
||||
@@ -2861,12 +2859,10 @@
|
||||
],
|
||||
"drives_commands": [
|
||||
"ao beads exec",
|
||||
"ao doctor",
|
||||
"ao goals measure",
|
||||
"ao pawl"
|
||||
],
|
||||
"path": "skills/validate/",
|
||||
"references": 23
|
||||
"references": 24
|
||||
},
|
||||
{
|
||||
"sku": "skill:workflow-builder",
|
||||
@@ -3009,9 +3005,7 @@
|
||||
"bounded_context": "BC4",
|
||||
"purpose": "Check AgentOps health",
|
||||
"status": "active",
|
||||
"driven_by_skills": [
|
||||
"validate"
|
||||
],
|
||||
"driven_by_skills": [],
|
||||
"flags": [
|
||||
"--before",
|
||||
"--config",
|
||||
|
||||
@@ -933,8 +933,8 @@
|
||||
{
|
||||
"name": "validate",
|
||||
"source_skill": "skills/validate",
|
||||
"source_hash": "80f8ce29d25e19886b727cc3ee8039fa9766d4366b4633c2b9b4644e51f3f8d6",
|
||||
"generated_hash": "faa16ceda4eeaa7c85ab39e4a572e562e697fce4d1a8c39f68a797584a4adf38"
|
||||
"source_hash": "365a0640f005f086b211669d0bb4eca3aa9947bd43d2733f3d4098deafd18550",
|
||||
"generated_hash": "407e3bdfa5aa67be1d3ba89ae60a20bb01e234dad9b0be1d4c8ab135dc01503f"
|
||||
},
|
||||
{
|
||||
"name": "workflow-builder",
|
||||
|
||||
@@ -2,6 +2,6 @@
|
||||
"generator": "codex-sync",
|
||||
"source_skill": "skills/validate",
|
||||
"layout": "modular",
|
||||
"source_hash": "80f8ce29d25e19886b727cc3ee8039fa9766d4366b4633c2b9b4644e51f3f8d6",
|
||||
"generated_hash": "faa16ceda4eeaa7c85ab39e4a572e562e697fce4d1a8c39f68a797584a4adf38"
|
||||
"source_hash": "365a0640f005f086b211669d0bb4eca3aa9947bd43d2733f3d4098deafd18550",
|
||||
"generated_hash": "407e3bdfa5aa67be1d3ba89ae60a20bb01e234dad9b0be1d4c8ab135dc01503f"
|
||||
}
|
||||
|
||||
+120
-326
@@ -2,371 +2,170 @@
|
||||
name: validate
|
||||
description: Produce PASS/WARN/FAIL verdicts for
|
||||
---
|
||||
# $validate — Canonical Validator Skill
|
||||
# $validate — Canonical Validator
|
||||
|
||||
> **Loop position:** move 6 (prove acceptance) of the [operating loop](../../docs/architecture/operating-loop.md) — the driving adapter for the `validate_acceptance` port: every Given/When/Then must map to a passing test before a bead closes, and the verdict binds to the SLICE's own acceptance test (its ATDD contract), not only the parent intent issue. This is S5 of the [narrow-waist micro-cycle](../../docs/architecture/operating-loop.md#the-narrow-waist-micro-cycle-canonical--every-loop-skill-cites-this).
|
||||
> **Purpose:** Move 6 of the operating loop: independently remeasure a bounded artifact or slice, prove its acceptance, and emit one schema-valid PASS/WARN/FAIL verdict with an executable next action.
|
||||
|
||||
> **Role:** validator. Input = artifact (plan, spec, code, PR, fitness gate). Output = `verdict.v1` (PASS / WARN / FAIL with rationale + findings).
|
||||
## Critical Constraints
|
||||
|
||||
`$validate` is a driving adapter for the `validate_acceptance` port in the
|
||||
[Intent-to-Loop Hexagon](../../docs/architecture/intent-to-loop-hexagon.md).
|
||||
When the artifact contains a `hexagon:` block, preserve the bounded context,
|
||||
context packet, guard adapters, and done state in the verdict.
|
||||
When the artifact claims DONE/closed/green, apply the
|
||||
[Completion-Claim Kernel](../shared/validation-contract.md#completion-claim-kernel)
|
||||
before returning PASS.
|
||||
- **One role: validator.** Never edit the artifact, commit, push, close tracker work, or operate infrastructure. **Why:** a judge that mutates the subject destroys independence.
|
||||
- Rerun cited commands on the actual artifact; never accept author evidence, conversational memory, or uncited figures as proof. **Why:** validation remeasures rather than agrees.
|
||||
- PASS requires every mandatory check green, all blockers resolved, disclosed `not_checked` coverage, and an independent judge when the mode claims independence. **Why:** missing evidence cannot be laundered into confidence.
|
||||
- `judge_id == author_id` cannot independently PASS; inline `--quick` fallback is stamped waived. **Why:** self-grading is guidance, not assurance.
|
||||
- Every judge brief says: **"READ-ONLY except writing your single verdict file at `<path>`. Do NOT commit, push, or run tracker/infra ops (git push, br/bd, dolt)."** **Why:** role scope is model-independent.
|
||||
- This skill validates work; `ao pawl` certifies landing with a commit-bound verdict. **Why:** a pre-work PASS never pre-authorizes a main write.
|
||||
- Use runtime-native judges for a requested multi-judge mode; never start NTM, Agent Mail, managed agents, Gas City, or another runtime unless explicitly selected. **Why:** judgment does not silently broaden orchestration.
|
||||
- `WARN|FAIL|REFUTED -> AUTO-REDO`: consult the pawl, return findings as re-plan evidence, repair through the owning producer, and rerun the same checks. **Why:** ordinary negative verdicts are self-correction, not human andons.
|
||||
- `BREAKER -> HOLD -> ONE-HELPER`; `HELPER-UNSTUCK -> AUTO-REDO`. Hold judgment and use one bounded local-shell helper to inspect contradictory evidence or a broken validator. **Why:** one recovery pass distinguishes tooling failure from a real stop.
|
||||
- `HELPER-ESCALATE -> HUMAN`; `REFUSAL-LANE|EXPLICIT-JUDGMENT|EXHAUSTED-BUDGET -> HUMAN`. **Why:** waivers, risk acceptance, unavailable authority, refusal, or exhausted recovery require the operator.
|
||||
|
||||
**A verdict is re-plan evidence, not just a retry trigger.** A FAIL/WARN (and its
|
||||
findings) surfaces UP to the [operating loop](../../docs/architecture/operating-loop.md):
|
||||
the *remaining* moves may be refactored, inserted, dropped, or reordered in response
|
||||
— not only the failed objective re-run. `$rpi` is one turn's executor over that loop,
|
||||
not an autonomous `--auto` re-planner (the old rpi CLI command surface was removed in f61c5f0e7 — historical).
|
||||
Looping a failed objective forever without asking whether the plan should change is
|
||||
the waterfall anti-pattern.
|
||||
## Modes
|
||||
|
||||
## Modes (≤8 per Fix-F mode-flag budget)
|
||||
|
||||
| Mode | Purpose | Replaces / absorbs |
|
||||
| Mode | Judge shape | Purpose |
|
||||
|---|---|---|
|
||||
| (default) | 2-judge multi-judge consensus on any artifact | `$council` default |
|
||||
| `--quick` | Inline single-agent structured review | `$council --quick` |
|
||||
| `--deep` | 4-judge thorough review | `$council --deep` |
|
||||
| `--mixed` | Cross-vendor (Claude + Codex), N×2 judges | `$council --mixed` |
|
||||
| `--debate` | Adversarial 2-round refinement | `$council --debate`; `/red-team` (absorbed 2026-07-07) |
|
||||
| `--mode=post-impl` | Code-readiness pipeline (complexity → bug-hunt → council) | `vibe` (absorbed) |
|
||||
| `--mode=pre-impl [--target=X]` | Plan/spec validation; target ∈ {scenario,fitness,ratchet,scope,skill,health} | `$pre-mortem`, `$goals measure`, `$scope`, `$heal-skill` (deep audit), `ao doctor`; `/eval-outcomes` + `/flywheel` (absorbed 2026-07-07) |
|
||||
| `--mode=pr` | PR-shape verdict (diff review + acceptance check) | `/review` (absorbed 2026-07-07) |
|
||||
| default | 2 independent | general artifact consensus |
|
||||
| `--quick` | inline, waived | bounded sanity/readiness check |
|
||||
| `--deep` | 4 perspectives | thorough requirement/feasibility/scope review |
|
||||
| `--mixed` | explicitly selected families | cross-family review |
|
||||
| `--debate` | 2+ judges, 2 rounds | adversarial critique/rebuttal |
|
||||
| `--mode=post-impl` | pipeline + isolated judges | acceptance and completion audit |
|
||||
| `--mode=pre-impl [--target=X]` | 2–4 judges | plan/spec/fitness/scope/skill/health audit |
|
||||
| `--mode=pr` | diff + acceptance judges | submission readiness |
|
||||
|
||||
**Mode-budget assertion:** 8 modes. Adding a 9th requires demoting an existing one OR refusing the addition (per Fix-F § continuous CI gate).
|
||||
**Mode-budget assertion:** 8 modes. Adding a ninth requires merging or removing an existing mode.
|
||||
|
||||
### Folded skills (cp-ki8): `validation` + `pr-validate` retired into these modes
|
||||
|
||||
The retired validation and pr-validate lanes were the Phase-1 placeholders for `--mode=post-impl`
|
||||
and `--mode=pr`; both are now retired (cp-ki8) and their load-bearing contract folded
|
||||
here so no capability is lost:
|
||||
|
||||
- **`--mode=post-impl` (was the validation lane) — full close-out + no-self-grading invariant.**
|
||||
Beyond the inline `complexity → bug-hunt → council` pipeline, this mode owns the
|
||||
`validate_acceptance` port: every Given/When/Then from the intent issue must map to a
|
||||
passing test (criterion→test roll-up; activity logs do not close beads), and the
|
||||
acceptance verdict **must be produced by a blind, context-isolated sub-agent judge that
|
||||
did not author the code** (author ≠ validator — `ag-9jle.5` / `ag-lmdx.4`). Refuse to
|
||||
certify acceptance when `judge_id == author_id`; the only escape is an inline-fallback
|
||||
self-grade that is stamped as *waived, not independently validated*. Apply the
|
||||
[Completion-Claim Kernel](../shared/validation-contract.md#completion-claim-kernel)
|
||||
before accepting any DONE/closed/green claim. For epic-scope close-out this mode may
|
||||
delegate to `vibe`, `$post-mortem`, and `/curate --mode=forge` rather than inlining them.
|
||||
- **`--mode=pr` (was the pr-validate lane) — submission-readiness checks.** In addition to the
|
||||
diff/acceptance verdict, run, in order: (1) **upstream alignment FIRST** (BLOCKING —
|
||||
`git rev-list --count HEAD..origin/main`; fail if many commits behind or merge would
|
||||
conflict), (2) CONTRIBUTING.md compliance (BLOCKING), (3) **isolation** — single commit
|
||||
type + thematic files + atomic scope, (4) scope-creep containment, (5) quality gate
|
||||
(tests/lint, non-blocking). On FAIL, emit remediation steps (split-by-type cherry-pick,
|
||||
rebase-on-upstream) so the verdict is actionable.
|
||||
|
||||
### Folded triggers (ag-s43tg wave 1): `vibe` + `bead-completion-audit` route here
|
||||
|
||||
- **`vibe` → `--mode=post-impl`.** Use when doing a quick readiness or sanity check
|
||||
that code is ready to commit or ship, short of a full review — the post-impl
|
||||
pipeline (complexity → bug-hunt → council) is the vibe check.
|
||||
- **`bead-completion-audit` → `--mode=post-impl` close-out.** Use when
|
||||
auditing closed beads for real shipped evidence, acceptance proof, and truthful
|
||||
closeout — the Completion-Claim Kernel and the no-self-grading invariant above
|
||||
own this audit.
|
||||
Folded triggers remain load-bearing: **`vibe` → `--mode=post-impl`** for code
|
||||
readiness, and `bead-completion-audit` routes to post-implementation closeout.
|
||||
Detailed target and evidence rules live in
|
||||
[canonical-validation-protocol](references/canonical-validation-protocol.md).
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
$validate path/to/plan.md # default 2-judge consensus
|
||||
$validate --quick path/to/plan.md # inline single-agent
|
||||
$validate --deep path/to/spec.md # 4-judge thorough
|
||||
$validate --mode=pre-impl path/to/plan.md # pre-mortem mode
|
||||
$validate --mode=post-impl recent # vibe mode (post-implement)
|
||||
$validate --mode=pr 123 # PR review by PR number
|
||||
$validate --mode=pre-impl --target=fitness # fitness gate against GOALS.md
|
||||
$validate path/to/artifact
|
||||
$validate --quick path/to/artifact
|
||||
$validate --deep path/to/spec.md
|
||||
$validate --mode=pre-impl --target=skill skills/example
|
||||
$validate --mode=post-impl recent
|
||||
$validate --mode=pr 123
|
||||
```
|
||||
|
||||
Default uses runtime-native subagent spawning. Falls back to `--quick` (inline) when no multi-agent capability detected.
|
||||
## Execution Workflow
|
||||
|
||||
## Execution
|
||||
### 1. Resolve mode, artifact, and authority
|
||||
|
||||
### Step 1: Resolve mode + target
|
||||
|
||||
Parse `--mode` and `--target`. Default mode is multi-judge. Validate combinations:
|
||||
|
||||
| Mode | Allowed `--target` |
|
||||
|---|---|
|
||||
| default, --quick, --deep, --mixed, --debate | n/a |
|
||||
| --mode=post-impl | n/a (pipeline scope is recent code changes) |
|
||||
| --mode=pre-impl | scenario, fitness, ratchet, scope, skill, health (default: pre-mortem on plan) |
|
||||
| --mode=pr | n/a (PR ID/path is positional) |
|
||||
|
||||
Reject invalid combinations (e.g., `--mode=pr --target=fitness`).
|
||||
|
||||
### Step 2: Load artifact + context
|
||||
Reject invalid mode/target combinations. Confirm the artifact exists, is within
|
||||
scope, and has explicit acceptance. For a goal-design packet, run:
|
||||
|
||||
```bash
|
||||
# resolve artifact:
|
||||
ARTIFACT="${1:-recent}" # path, PR ID, or "recent"
|
||||
|
||||
# load FAIL patterns:
|
||||
# (folded into skill body; not a separate hook)
|
||||
scripts/check-goal-design-packet.sh <packet-dir>
|
||||
```
|
||||
|
||||
If `ARTIFACT` is a goal-design packet directory containing `intent.md` and
|
||||
`driver.md`, run the deterministic packet checker before spawning or judging:
|
||||
Nonzero is deterministic FAIL evidence. Load only the selected section of
|
||||
[canonical-validation-protocol](references/canonical-validation-protocol.md)
|
||||
plus the relevant language/risk standards.
|
||||
|
||||
```bash
|
||||
scripts/check-goal-design-packet.sh "$ARTIFACT"
|
||||
```
|
||||
**Checkpoint:** record artifact identity, commit/digest, mode, targets, author id,
|
||||
required checks, and `not_checked` before dispatch.
|
||||
|
||||
Non-zero exit is a FAIL verdict. Include the command and output in
|
||||
`COMMANDS RUN:`. This is a pre-check inside the existing validator role, not a
|
||||
new mode.
|
||||
### 2. Run deterministic prechecks
|
||||
|
||||
For `--mode=pre-impl`, also load:
|
||||
- `.agents/planning-rules/*.md` (compiled planning rules)
|
||||
- `.agents/findings/registry.jsonl` (active findings)
|
||||
- `.agents/pre-mortem-checks/*.md` (compiled prevention)
|
||||
- pre-implementation: target rubric, temporal/error-rescue checks, test pyramid;
|
||||
- post-implementation: complexity, bug sweep, slice acceptance, completion kernel;
|
||||
- PR: upstream alignment first, contribution rules, atomicity, scope, tests/lint;
|
||||
- all modes: source precedence, artifact freshness, and output-schema availability.
|
||||
|
||||
For `--mode=post-impl`, run pre-checks:
|
||||
- complexity audit (radon for python, gocyclo for go)
|
||||
- bug-hunt sweep (skill-body convention; no `/review` skill needed)
|
||||
A deterministic red result is a FAIL verdict; do not spend judge tokens to
|
||||
rediscover it.
|
||||
|
||||
For `--mode=pr`, fetch the PR diff (`gh pr diff <id>` or path).
|
||||
### 3. Dispatch isolated judges
|
||||
|
||||
> **LIVE-PATH — where this judge orchestration is operative, and where the pawl is.**
|
||||
> Steps 3–7 (spawn backend → run judges → write the `.agents/council/` verdict) are the live path
|
||||
> for **PRE-work and non-merge validation**: plans, specs, artifacts, fitness gates, mid-arc
|
||||
> completion checks, and the `--mode=pre-impl` / `post-impl` / `pr` verdicts. **At the merge-to-main
|
||||
> door this orchestration is NOT the operative machinery** — that door is driven by the pawl scripts:
|
||||
> `scripts/pawl-review.sh` runs the cross-family refuter and, on CONFIRMED, `scripts/pawl-verdict.sh`
|
||||
> writes the commit-bound verdict the pre-push gate enforces (see [`$push`](../push/SKILL.md) and
|
||||
> [`pawl-review`](../pawl-review/SKILL.md)). Use this skill to validate the WORK; use
|
||||
> `ao pawl` to certify the LANDING — they compose, not compete. The judge machinery below is retained for
|
||||
> that pre-work / non-merge scope, not deleted.
|
||||
Register dispatch intent before spawning. Use the current runtime's native
|
||||
subagents for multi-judge modes; if unavailable, degrade only to explicit
|
||||
`--quick` and stamp the independence waiver.
|
||||
|
||||
### Step 3: Determine spawn backend
|
||||
Each judge receives the artifact path/digest, bounded context, mandatory checks,
|
||||
verdict path, and exact read-only clamp. It independently reruns commands.
|
||||
|
||||
1. `spawn_agent` available → Codex sub-agent
|
||||
2. `TeamCreate` available → Claude native team
|
||||
3. `task` (read-only skill tool, OpenCode) → opencode subagent
|
||||
4. None → fall back to `--quick` (inline single-agent)
|
||||
**Checkpoint:** author and judge identities differ, dispatch scopes do not
|
||||
collide, and every claimed judge produced a nonempty verdict artifact.
|
||||
|
||||
Log selected backend in the verdict frontmatter.
|
||||
### 4. Consolidate fail-closed
|
||||
|
||||
### Step 4: Run judges
|
||||
- PASS: all required judges/checks PASS (or the declared deep-mode majority), no blocker.
|
||||
- WARN: nonblocking concern or explicit coverage gap remains.
|
||||
- FAIL: any deterministic failure, blocker, counterfeit/self judge, stale artifact, or malformed evidence.
|
||||
|
||||
| Mode | Judges | Perspectives |
|
||||
|---|---|---|
|
||||
| default | 2 | independent (no labeled perspectives) |
|
||||
| --deep | 4 | missing-requirements, feasibility, scope, spec-completeness |
|
||||
| --mixed | 2N (default N=3) | same N perspectives across Claude + Codex |
|
||||
| --debate | 2+ rounds | adversarial; 2 rounds with critique-rebuttal |
|
||||
| --quick | 0 (inline self) | structured review |
|
||||
| --mode=post-impl | 2 + pipeline | complexity → bug-hunt → 2-judge council |
|
||||
| --mode=pre-impl | 2-4 | per target preset |
|
||||
| --mode=pr | 2 | diff-review + acceptance-check |
|
||||
Contradictory verdicts are FAIL until reconciled; never average away a blocker.
|
||||
|
||||
Each judge gets:
|
||||
- artifact path
|
||||
- relevant context (planning rules, findings)
|
||||
- council FAIL pattern check prompt (top 8)
|
||||
- temporal interrogation prompt (--deep + --target=plan)
|
||||
### 5. Write and validate outputs
|
||||
|
||||
### Step 5: Mandatory checks (auto-trigger)
|
||||
Write `.agents/council/YYYY-MM-DD-validate-<slug>.md` with exactly one anchored
|
||||
`## Council Verdict: PASS|WARN|FAIL`, plus `result.json` matching
|
||||
`schemas/verdict.v1.schema.json`. The report contains exactly one anchored
|
||||
`VERDICT:`, a nonempty `COMMANDS RUN:`, `REASONS:`, findings, and `not_checked`.
|
||||
|
||||
For `--mode=pre-impl --target=plan`:
|
||||
- temporal interrogation (auto for plans with 5+ files or 3+ deps)
|
||||
- error & rescue map
|
||||
- council FAIL pattern check (top 8)
|
||||
- test pyramid coverage check
|
||||
- input validation check (enum-like fields)
|
||||
**Checkpoint:** run both validators in Output Specification; PASS is illegal if
|
||||
either output is missing, malformed, stale, self-graded, or unsupported by commands.
|
||||
|
||||
For `--mode=post-impl`:
|
||||
- L0/L1/L2 coverage check on changed files
|
||||
- **refactor-under-green check:** if the diff is a refactor, assert no acceptance/unit test text changed (`git diff` of test files is empty) — a refactor that edits a test changed behavior and is a new slice, not a refactor (S4). FAIL if a refactor diff also edits a test.
|
||||
### 6. Route the verdict
|
||||
|
||||
For `--mode=pre-impl --target=fitness`:
|
||||
- read GOALS.md
|
||||
- evaluate each gate against current state
|
||||
- report PASS/WARN/FAIL per gate + aggregate
|
||||
|
||||
### Step 6: Consolidate to verdict
|
||||
|
||||
Each judge returns a per-judge result. Consolidate:
|
||||
- PASS only if all judges PASS (or majority for --deep)
|
||||
- WARN if any judge raises a warning the others don't dispute
|
||||
- FAIL if any judge raises a blocker the others don't override
|
||||
|
||||
### Step 7: Write verdict
|
||||
|
||||
Output path: `.agents/council/YYYY-MM-DD-validate-<topic-slug>.md`
|
||||
|
||||
```markdown
|
||||
---
|
||||
id: validate-YYYY-MM-DD-<slug>
|
||||
type: verdict
|
||||
date: YYYY-MM-DD
|
||||
mode: <mode>
|
||||
target: <target or n/a>
|
||||
artifact: <path>
|
||||
backend: <codex-subagents | claude-teams | opencode | inline>
|
||||
---
|
||||
|
||||
# Validate Verdict — <topic>
|
||||
|
||||
## Council Verdict: PASS / WARN / FAIL
|
||||
|
||||
| Failure mode | Risk | Severity | Addressed? |
|
||||
|---|---|---|---|
|
||||
| ... | ... | ... | ... |
|
||||
|
||||
## Pseudocode Fixes (when WARN/FAIL)
|
||||
(copy-pastable into affected issues per pre-mortem 4.6 contract)
|
||||
|
||||
## FAIL Pattern Check
|
||||
(top 8 patterns — status per pattern)
|
||||
|
||||
## Verdict
|
||||
PASS — proceed
|
||||
WARN — review concerns, accept risk, or apply fixes
|
||||
FAIL — block; revise artifact and rerun
|
||||
```
|
||||
|
||||
The exact heading `## Council Verdict: PASS / WARN / FAIL` is mandatory — downstream validators and ledger readers parse it with anchored regex.
|
||||
|
||||
### Step 8: Persist findings (when applicable)
|
||||
|
||||
For `--mode=pre-impl` reusable findings: append to `.agents/findings/registry.jsonl` (atomic temp+rename). Every FAIL finding must be emitted as a next-loop producer — a `$pre-mortem` check or `$plan` planning-rule keyed by dedup_key (the escape→check ratchet, S6), not just a logged finding; a defect the membrane caught that produces no next-loop check will recur. See [operating-loop.md → move 7](../../docs/architecture/operating-loop.md).
|
||||
|
||||
### Step 9: Report
|
||||
|
||||
1. Verdict (PASS/WARN/FAIL).
|
||||
2. Key concerns (when not PASS).
|
||||
3. Output path.
|
||||
4. Recommended next action.
|
||||
|
||||
## --target taxonomy (pre-impl)
|
||||
|
||||
| `--target` | What gets graded | Replaces |
|
||||
|---|---|---|
|
||||
| (default) | Plan/spec for an upcoming `$implement` | `$pre-mortem` |
|
||||
| scenario | Holdout scenario gate | `/eval-outcomes` |
|
||||
| fitness | GOALS.md fitness gates | `$goals measure`, `ao goals measure` |
|
||||
| ratchet | Brownian Ratchet checkpoint | `/flywheel`, `ao ratchet status` |
|
||||
| scope | Frozen-dirs declaration | `$scope` |
|
||||
| skill | SKILL.md hygiene + audit | `$heal-skill` (heal.sh hygiene + audit.sh deep audit) |
|
||||
| health | Repo health probe | `ao doctor` |
|
||||
|
||||
Each target has its own inline check rubric until Phase 2 extraction.
|
||||
|
||||
## Validation discipline (2026-06-09, cards 6–10, cp-hhd7)
|
||||
|
||||
### Verdict form — the gate parses these lines anchored
|
||||
|
||||
```
|
||||
VERDICT: PASS
|
||||
(blank line)
|
||||
COMMANDS RUN:
|
||||
<actual commands + verbatim output snippets>
|
||||
REASONS:
|
||||
- bullet citing a COMMANDS RUN line
|
||||
```
|
||||
|
||||
A verdict with no `COMMANDS RUN:` section is **unverified** — reject it and
|
||||
dispatch a fresh validator. A verdict whose `COMMANDS RUN:` lists only commands
|
||||
the **author** ran (not the judge) is a **counterfeit judge** — treat as FAIL and
|
||||
re-route to a genuinely independent validator. No `##` headings or parentheticals
|
||||
on `VERDICT:` or `COMMANDS RUN:` lines; the gate parses them anchored.
|
||||
|
||||
### Cross-family floor (POLICY → gate icb6 enforces; this skill supports)
|
||||
|
||||
For **assurance closes** (the control-plane verdict-gate, cp-icb6), the floor is
|
||||
≥2 verdicts from ≥2 distinct model families, author family excluded, fail-closed.
|
||||
**This skill supports that policy via `--mixed` mode and the verdict form above;
|
||||
the policy itself lives in the gate, not here.** A same-model council is valid for
|
||||
non-assurance decisions (design brainstorms, quick checks) — do not refuse those.
|
||||
Tier mapping:
|
||||
- **STRICT** (irreversible, security, production close): Codex + strong-Gemini (A1); or Fable + Codex (A2).
|
||||
- **ROUTINE** (everyday close): Codex + non-author-Claude (A3); or Fable + non-author (A4).
|
||||
|
||||
The A7 ruling (2026-06-09, memory `validation-family-policy-risk-tiered`):
|
||||
Gemini is currently **benched** for STRICT validation — use Codex + Fable for A1/A2
|
||||
tiers. Gemini may return for STRICT when Bo graduates it from the bench. Do not
|
||||
present Gemini paths as live for tier A1/A2 until then.
|
||||
|
||||
### Judge empirically on a differentiating fixture (card 9, cp-8720)
|
||||
|
||||
When two implementations of the same intent exist, do NOT award based on
|
||||
authorship or surface aesthetics. Run both on a **differentiating fixture** (an
|
||||
input that exposes their behavioral difference), record the outputs verbatim, and
|
||||
graft the loser's unique assets onto the winner. "My worker wrote it" is not
|
||||
evidence.
|
||||
|
||||
### Dispatch record first (card 3, cp-hhtu)
|
||||
|
||||
Before dispatching a validator, register intent on the bead graph (update status,
|
||||
assign actor). **Two parallel validators on the same bead produce a dedup incident,
|
||||
not a cross-family quorum.** Check for an existing actor before spawning.
|
||||
|
||||
### Judge lanes carry an explicit write-scope clamp (2026-07-02, showcase kernel R2)
|
||||
|
||||
Every judge brief states, verbatim: **"READ-ONLY except writing your single verdict
|
||||
file at `<path>`. Do NOT commit, push, or run tracker/infra ops (git push, br/bd,
|
||||
dolt)."** The clamp is role-scoped, not model-scoped — crank workers legitimately
|
||||
hold broad write scopes; judges re-measure, they never mutate. Proven live: an
|
||||
unclamped codex acceptance judge pushed the feature branch unprompted and attempted
|
||||
`bd dolt push` twice mid-judgment. A judge that mutates while judging can corrupt
|
||||
the artifact under judgment or preempt the pawl.
|
||||
|
||||
### Judges re-measure; they do not read (card 8)
|
||||
|
||||
A judge re-runs the cited commands on the actual artifacts. It does not read the
|
||||
author's evidence file and agree. Attest `judge_source: <model>` inside `COMMANDS RUN`
|
||||
so the gate can confirm the judge identity. A judge that ran nothing is a reader, not
|
||||
a verifier — discard its verdict.
|
||||
|
||||
## Evidence figures are measured, never inferred (the cp-801l lesson)
|
||||
|
||||
A worker's evidence file may only contain numbers and outputs that were **captured** —
|
||||
pasted verbatim from a command's output — never reconstructed from memory. The
|
||||
canonical failure: "36 checks — 35 pass" stated with confidence was inference; the
|
||||
measured reality was 36 run / 34 pass / 1 fail / 1 skip, on a different commit.
|
||||
|
||||
- **Validators:** treat any uncited figure (a count, a pass-rate, a timing, a commit
|
||||
hash) as **unverifiable → FAIL** until a log is produced or the claim is corrected.
|
||||
- **Corrections happen by APPENDED erratum** — a dated erratum block crediting the
|
||||
source measurement — never by silently editing the original figure. A silently
|
||||
edited evidence file is indistinguishable from a fabricated one.
|
||||
|
||||
## Constraints (one-role-per-skill)
|
||||
|
||||
- **One role: validator.** Output is always a verdict. Never mutates code (delegates to `$implement` for fixes).
|
||||
- **No new modes** without dropping/merging an existing one (Fix-F mode-budget cap = 8).
|
||||
- **Verdict heading is regex-anchored** — do not alter the `## Council Verdict: ...` text format.
|
||||
Report verdict, key findings, artifact paths, and one executable next action.
|
||||
WARN/FAIL re-enters the operating loop through the owning producer. Reusable
|
||||
failures compile into a planning/pre-mortem check instead of becoming a dead log.
|
||||
|
||||
## Output Specification
|
||||
|
||||
**Format:** a PASS/WARN/FAIL validation verdict plus a markdown council summary to stdout; machine-readable `result.json`.
|
||||
**Files:** writes `.agents/council/YYYY-MM-DD-validate-<slug>.md` and `result.json`; appends reusable findings to `.agents/findings/registry.jsonl`; may refresh `.agents/planning-rules/` and `.agents/pre-mortem-checks/`.
|
||||
**Exit signal:** FAIL re-cranks on the same objective (up to 3 attempts). Exhaustion enters HOLD and permits exactly one bounded fresh-context or cross-family helper pass; HELPER-UNSTUCK resumes with the new approach, while only HELPER-ESCALATE (or a refusal/explicit-judgment or spent hard-ceiling class) routes to the human. DONE requires a fully-green acceptance roll-up.
|
||||
**Artifact directory:** markdown verdicts under `.agents/council/`; machine-readable `result.json` at the invocation output root.
|
||||
|
||||
## See Also
|
||||
**Filename convention:** `.agents/council/YYYY-MM-DD-validate-<topic-slug>.md` plus `result.json`.
|
||||
|
||||
- `skills/rpi/SKILL.md` — orchestrator that fires `$validate --mode=pre-impl` after `$plan`
|
||||
- `skills/curate/SKILL.md` — miner role (paired canonical skill)
|
||||
- `schemas/verdict.v1.schema.json` — output contract
|
||||
- [`pawl-review`](../pawl-review/SKILL.md) — executes immutable fresh reviewer lanes and hands evidence to `ao pawl`; the membrane alone binds the commit verdict
|
||||
**Serialization/schema format:** `result.json` follows closed `schemas/verdict.v1.schema.json`; markdown has one `## Council Verdict: PASS|WARN|FAIL` and the anchored evidence form in the protocol reference.
|
||||
|
||||
**Validator command:** validate `result.json` with `Draft202012Validator(schema, format_checker=FormatChecker())`, an explicit timezone-aware RFC3339 parse of `validated_at`, and the pinned author identity so a PASS requires `validator_session != author_session`; then verify the markdown has exactly one council verdict, exactly one matching `VERDICT:`, and a `judge=<validator_session> command=<command>` evidence line under `COMMANDS RUN:` before `REASONS:`. The executable fixture suite is `bash skills/validate/scripts/validate.sh`.
|
||||
|
||||
**Downstream handoff:** pass mode/target, artifact path/digest/commit, author and judge identities/families, deterministic commands and exit codes, verdict/report paths, findings, `not_checked`, waiver/breaker state, and the exact next action; landing still requires `ao pawl`.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- [ ] Artifact identity and acceptance are pinned before checks run.
|
||||
- [ ] Judges are read-only, independent when claimed, and rerun commands themselves.
|
||||
- [ ] Every number, timing, and commit is captured from cited output.
|
||||
- [ ] PASS has no blockers, counterfeit judges, stale evidence, or undisclosed gaps.
|
||||
- [ ] Markdown and JSON agree and pass their deterministic validators.
|
||||
- [ ] WARN/FAIL/REFUTED returns to AUTO-REDO; only real breakers reach a human.
|
||||
- [ ] Validation PASS is never substituted for the commit-bound landing verdict.
|
||||
|
||||
## Examples
|
||||
|
||||
### Reject a counterfeit completion claim
|
||||
|
||||
**User says:** `$validate --mode=post-impl recent`
|
||||
|
||||
**What happens:** the isolated judge reruns the slice acceptance command and
|
||||
finds the author's reported count was inferred and the test exits nonzero.
|
||||
|
||||
**Result:** FAIL with captured output, corrected count, owning producer, and the
|
||||
same validation command as next action; no human andon and no landing claim.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Response |
|
||||
|---|---|
|
||||
| Judge unavailable | Use explicit waived `--quick` or HOLD only if independence is required |
|
||||
| Artifact changed mid-review | Mark stale, pin the new digest, rerun all checks |
|
||||
| Judges disagree | FAIL until the blocker is reproduced and reconciled |
|
||||
| Empty `COMMANDS RUN:` | Discard the verdict and dispatch a real verifier |
|
||||
|
||||
## Reference Documents
|
||||
|
||||
- [references/validate.feature](references/validate.feature) — Executable spec: verdict.v1 PASS/WARN/FAIL for any artifact, --mode selects shape, 8-mode budget (soc-qk4b)
|
||||
|
||||
## Reference library (incl. rescued vibe references, ag-s43tg)
|
||||
|
||||
- [references/canonical-validation-protocol.md](references/canonical-validation-protocol.md) — modes, targets, evidence, isolation, landing boundary
|
||||
- [references/report-format.md](references/report-format.md) — markdown detail
|
||||
- [references/deep-audit-protocol.md](references/deep-audit-protocol.md) — deep-mode perspectives
|
||||
- [references/test-pyramid-inventory.md](references/test-pyramid-inventory.md) — coverage selection
|
||||
- [references/post-verdict-actions.md](references/post-verdict-actions.md) — bounded follow-through
|
||||
- [references/validate.feature](references/validate.feature) — executable behavior
|
||||
- [references/complexity-analysis.md](references/complexity-analysis.md)
|
||||
- [references/deep-audit-protocol.md](references/deep-audit-protocol.md)
|
||||
- [references/deep-checks.md](references/deep-checks.md)
|
||||
- [references/examples.md](references/examples.md)
|
||||
- [references/go-patterns.md](references/go-patterns.md)
|
||||
@@ -374,13 +173,10 @@ measured reality was 36 run / 34 pass / 1 fail / 1 skip, on a different commit.
|
||||
- [references/json-standards.md](references/json-standards.md)
|
||||
- [references/markdown-standards.md](references/markdown-standards.md)
|
||||
- [references/patterns.md](references/patterns.md)
|
||||
- [references/post-verdict-actions.md](references/post-verdict-actions.md)
|
||||
- [references/python-standards.md](references/python-standards.md)
|
||||
- [references/quick-mode-vibe.md](references/quick-mode-vibe.md)
|
||||
- [references/report-format.md](references/report-format.md)
|
||||
- [references/rust-standards.md](references/rust-standards.md)
|
||||
- [references/shell-standards.md](references/shell-standards.md)
|
||||
- [references/test-pyramid-inventory.md](references/test-pyramid-inventory.md)
|
||||
- [references/test-pyramid-weighting.md](references/test-pyramid-weighting.md)
|
||||
- [references/typescript-standards.md](references/typescript-standards.md)
|
||||
- [references/verification-report.md](references/verification-report.md)
|
||||
@@ -388,6 +184,4 @@ measured reality was 36 run / 34 pass / 1 fail / 1 skip, on a different commit.
|
||||
- [references/vibe-suppressions.md](references/vibe-suppressions.md)
|
||||
- [references/write-time-quality.md](references/write-time-quality.md)
|
||||
- [references/yaml-standards.md](references/yaml-standards.md)
|
||||
- [references/validate.feature](references/validate.feature)
|
||||
- [references/vibe.feature](references/vibe.feature) — rescued vibe executable spec
|
||||
- [scripts/prescan.sh](scripts/prescan.sh) — rescued vibe pre-scan helper
|
||||
- [references/vibe.feature](references/vibe.feature)
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Canonical Validation Protocol
|
||||
|
||||
This reference carries the detailed mode, target, and evidence rules for the
|
||||
`validate` kernel. Load only the sections selected by the current mode.
|
||||
|
||||
## Mode details
|
||||
|
||||
- Default: two independent judges; PASS requires both PASS.
|
||||
- `--quick`: inline structured review. It may guide work but is stamped
|
||||
`waived`, never independently validated.
|
||||
- `--deep`: four perspectives—missing requirements, feasibility, scope, and
|
||||
specification completeness. PASS uses the declared majority rule, with any
|
||||
unresolved blocker still failing closed.
|
||||
- `--mixed`: the same perspectives across explicitly selected model families.
|
||||
- `--debate`: two adversarial rounds with critique and rebuttal.
|
||||
- `--mode=post-impl`: complexity, bug sweep, acceptance roll-up, then isolated
|
||||
judges. A refactor that changes acceptance/unit test text is a new behavior
|
||||
slice, not a refactor.
|
||||
- `--mode=pre-impl`: plan/spec checks, optionally specialized by `--target`.
|
||||
- `--mode=pr`: upstream alignment first, contribution rules, atomic isolation,
|
||||
scope containment, then tests/lint.
|
||||
|
||||
The eight-mode budget is fixed. A ninth mode must replace or merge an existing
|
||||
mode rather than growing the public surface.
|
||||
|
||||
## Pre-implementation targets
|
||||
|
||||
| Target | Required checks |
|
||||
|---|---|
|
||||
| default plan | temporal interrogation, error/rescue map, FAIL patterns, test pyramid, enum/input validation |
|
||||
| scenario | holdout scenario and falsifying edge |
|
||||
| fitness | each GOALS.md gate against current measured state |
|
||||
| ratchet | current checkpoint, evidence, and legal next transition |
|
||||
| scope | frozen paths, authority boundary, and escape prevention |
|
||||
| skill | strict hygiene plus profile-aware deep audit |
|
||||
| health | executable repository health probes and disclosed gaps |
|
||||
|
||||
A goal-design packet containing `intent.md` and `driver.md` first runs
|
||||
`scripts/check-goal-design-packet.sh <packet-dir>`; nonzero is FAIL evidence.
|
||||
|
||||
## Post-implementation acceptance
|
||||
|
||||
Every Given/When/Then maps to a passing acceptance test for its own vertical
|
||||
slice. Activity logs and parent-issue summaries do not close a bead. Apply the
|
||||
Completion-Claim Kernel in `skills/shared/validation-contract.md` to every
|
||||
DONE/closed/green claim.
|
||||
|
||||
The acceptance judge is blind and context-isolated. `judge_id == author_id`
|
||||
cannot independently PASS. Inline fallback is marked waived and cannot satisfy
|
||||
an assurance close.
|
||||
|
||||
## PR checks
|
||||
|
||||
Run in order:
|
||||
|
||||
1. upstream alignment (`git rev-list --count HEAD..origin/main`) and conflict risk;
|
||||
2. repository contribution rules;
|
||||
3. one thematic/atomic change shape;
|
||||
4. scope-creep containment;
|
||||
5. relevant tests and lint.
|
||||
|
||||
FAIL includes executable remediation such as rebase or split-by-type; it never
|
||||
mutates the branch from the judge lane.
|
||||
|
||||
## Evidence discipline
|
||||
|
||||
The validator reruns cited commands on the actual artifact. It does not accept
|
||||
the author's evidence file as proof. Every count, timing, commit, and pass rate
|
||||
is pasted from captured output. Uncited figures fail until measured.
|
||||
|
||||
Corrections are appended as dated errata crediting the source measurement;
|
||||
silently rewriting an evidence claim is forbidden.
|
||||
|
||||
Verdict text uses anchored lines:
|
||||
|
||||
```text
|
||||
VERDICT: PASS
|
||||
|
||||
COMMANDS RUN:
|
||||
judge=<validator_session> command=<command rerun by this judge>
|
||||
<bounded output excerpts produced by this judge>
|
||||
REASONS:
|
||||
- each reason cites a command or artifact
|
||||
```
|
||||
|
||||
Exactly one verdict is allowed. `COMMANDS RUN:` must include at least one
|
||||
`judge=<validator_session> command=<command>` line, and the pinned author identity
|
||||
must differ from that validator session for PASS. A judge that ran nothing is a
|
||||
reader, not a verifier.
|
||||
|
||||
## Judge isolation
|
||||
|
||||
Every judge brief includes this exact clamp:
|
||||
|
||||
> READ-ONLY except writing your single verdict file at `<path>`. Do NOT commit,
|
||||
> push, or run tracker/infra ops (git push, br/bd, dolt).
|
||||
|
||||
Register dispatch intent before spawning. Two validators accidentally assigned
|
||||
to the same lane/bead are a dedup incident, not an independent quorum.
|
||||
|
||||
## Assurance and landing
|
||||
|
||||
Non-merge validation may use runtime-native judges. Assurance close requires the
|
||||
policy-selected independent-family floor. Benched families are not presented as
|
||||
live strict lanes.
|
||||
|
||||
Landing is a different door: `ao pawl` produces the commit-bound verdict and the
|
||||
pre-push gate authorizes the remote-main write. A validation PASS never replaces
|
||||
or pre-authorizes that landing proof.
|
||||
|
||||
## Failure routing
|
||||
|
||||
WARN/FAIL/REFUTED findings return to the operating loop as re-plan evidence.
|
||||
Repair and remeasure automatically while the bounded budget has progress.
|
||||
HOLD/ESCALATE occurs only on a breaker: exhausted attempts/time/cost, oscillation,
|
||||
refusal, unavailable authority, or explicit risk/waiver judgment.
|
||||
Executable
+122
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
REPO_ROOT="$(cd "$SKILL_DIR/../.." && pwd)"
|
||||
SKILL_MD="$SKILL_DIR/SKILL.md"
|
||||
SCHEMA="$REPO_ROOT/schemas/verdict.v1.schema.json"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
record() {
|
||||
local label="$1"
|
||||
shift
|
||||
if "$@"; then echo "PASS: $label"; PASS=$((PASS + 1));
|
||||
else echo "FAIL: $label"; FAIL=$((FAIL + 1)); fi
|
||||
}
|
||||
|
||||
validate_contract() {
|
||||
local skill_md="$1"
|
||||
[[ "$(awk '/^---$/{n++;next} n==2 && /^## /{print;exit}' "$skill_md")" == "## Critical Constraints" ]] &&
|
||||
grep -Fq 'WARN|FAIL|REFUTED -> AUTO-REDO' "$skill_md" &&
|
||||
grep -Fq 'BREAKER -> HOLD -> ONE-HELPER' "$skill_md" &&
|
||||
grep -Fq 'HELPER-UNSTUCK -> AUTO-REDO' "$skill_md" &&
|
||||
grep -Fq 'HELPER-ESCALATE -> HUMAN' "$skill_md" &&
|
||||
grep -Fq 'REFUSAL-LANE|EXPLICIT-JUDGMENT|EXHAUSTED-BUDGET -> HUMAN' "$skill_md" &&
|
||||
grep -Fq '**Mode-budget assertion:** 8 modes.' "$skill_md" &&
|
||||
grep -Fq 'vibe` → `--mode=post-impl`' "$skill_md" &&
|
||||
grep -Fq 'READ-ONLY except writing your single verdict file' "$skill_md" &&
|
||||
grep -Fq 'This skill validates work; `ao pawl` certifies landing' "$skill_md" &&
|
||||
grep -Fq '**Checkpoint:**' "$skill_md" &&
|
||||
grep -Fq '**Artifact directory:**' "$skill_md" &&
|
||||
grep -Fq '**Filename convention:**' "$skill_md" &&
|
||||
grep -Fq '**Serialization/schema format:**' "$skill_md" &&
|
||||
grep -Fq '**Validator command:**' "$skill_md" &&
|
||||
grep -Fq '**Downstream handoff:**' "$skill_md" &&
|
||||
grep -Fq '## Quality Checklist' "$skill_md"
|
||||
}
|
||||
|
||||
validate_result() {
|
||||
local result="$1"
|
||||
local author_session="$2"
|
||||
python3 -c 'import json,re,sys; from datetime import datetime; from jsonschema import Draft202012Validator,FormatChecker; schema=json.load(open(sys.argv[1])); instance=json.load(open(sys.argv[2])); author=sys.argv[3]; Draft202012Validator.check_schema(schema); Draft202012Validator(schema,format_checker=FormatChecker()).validate(instance); ts=instance["validated_at"]; assert re.fullmatch(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})",ts); assert datetime.fromisoformat(ts.replace("Z","+00:00")).tzinfo is not None; assert instance["verdict"] != "PASS" or (instance["validator_session"] and instance["validator_session"] != author and not any(f["severity"] in {"critical","significant"} for f in instance["findings"]) and not instance["not_checked"])' "$SCHEMA" "$result" "$author_session" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
validate_report() {
|
||||
local report="$1"
|
||||
local validator_session="$2"
|
||||
local author_session="$3"
|
||||
local council_verdict report_verdict
|
||||
council_verdict="$(sed -nE 's/^## Council Verdict: (PASS|WARN|FAIL)$/\1/p' "$report")"
|
||||
report_verdict="$(sed -nE 's/^VERDICT: (PASS|WARN|FAIL)$/\1/p' "$report")"
|
||||
[[ "$(grep -Ec '^## Council Verdict: (PASS|WARN|FAIL)$' "$report")" -eq 1 ]] &&
|
||||
[[ "$(grep -Ec '^VERDICT: (PASS|WARN|FAIL)$' "$report")" -eq 1 ]] &&
|
||||
[[ "$council_verdict" == "$report_verdict" ]] &&
|
||||
[[ -n "$validator_session" ]] &&
|
||||
[[ "$report_verdict" != "PASS" || "$validator_session" != "$author_session" ]] &&
|
||||
awk -v judge="$validator_session" 'BEGIN{state=0;commands=0;reasons=0;seen=0;sourced=0;invalid=0;prefix="judge=" judge " command="} /^COMMANDS RUN:$/{commands++;if(state!=0)invalid=1;state=1;next} /^REASONS:$/{reasons++;if(state!=1||!seen)invalid=1;state=2;next} state==1&&NF{seen=1;if(index($0,prefix)==1&&length($0)>length(prefix))sourced=1} END{exit !(commands==1&&reasons==1&&seen&&sourced&&state==2&&!invalid)}' "$report"
|
||||
}
|
||||
|
||||
record "validate contract is complete" validate_contract "$SKILL_MD"
|
||||
record "verdict schema exists" test -s "$SCHEMA"
|
||||
record "protocol reference exists" test -s "$SKILL_DIR/references/canonical-validation-protocol.md"
|
||||
record "kernel stays within 250 lines" test "$(wc -l < "$SKILL_MD")" -le 250
|
||||
|
||||
pawl_fixture="$(mktemp)"
|
||||
output_fixture="$(mktemp)"
|
||||
valid_result="$(mktemp)"
|
||||
invalid_result="$(mktemp)"
|
||||
invalid_date_result="$(mktemp)"
|
||||
self_judge_result="$(mktemp)"
|
||||
blocking_finding_result="$(mktemp)"
|
||||
significant_finding_result="$(mktemp)"
|
||||
unchecked_result="$(mktemp)"
|
||||
valid_report="$(mktemp)"
|
||||
invalid_report="$(mktemp)"
|
||||
wrong_order_report="$(mktemp)"
|
||||
unsupported_report="$(mktemp)"
|
||||
trap 'rm -f "$pawl_fixture" "$output_fixture" "$valid_result" "$invalid_result" "$invalid_date_result" "$self_judge_result" "$blocking_finding_result" "$significant_finding_result" "$unchecked_result" "$valid_report" "$invalid_report" "$wrong_order_report" "$unsupported_report"' EXIT
|
||||
|
||||
sed 's/HELPER-UNSTUCK -> AUTO-REDO/HELPER-UNSTUCK -> MANUAL/' "$SKILL_MD" >"$pawl_fixture"
|
||||
awk '!/\*\*Validator command:\*\*/' "$SKILL_MD" >"$output_fixture"
|
||||
if validate_contract "$pawl_fixture"; then echo "FAIL: missing helper transition rejected"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: missing helper transition rejected"; PASS=$((PASS + 1)); fi
|
||||
if validate_contract "$output_fixture"; then echo "FAIL: incomplete output handoff rejected"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: incomplete output handoff rejected"; PASS=$((PASS + 1)); fi
|
||||
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAV","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$valid_result"
|
||||
printf '%s\n' '{"verdict_id":"short","bead_id":"bad","verdict":"MAYBE","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"not-a-date","validator_session":"author","schema_version":1}' >"$invalid_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAV","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"not-a-date","validator_session":"judge-1","schema_version":1}' >"$invalid_date_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAX","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"author","schema_version":1}' >"$self_judge_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAY","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[{"severity":"critical","description":"release gate is bypassed","location":"scripts/release.sh:10"}],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$blocking_finding_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FB0","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[{"severity":"significant","description":"required regression is missing","location":"tests/release.bats"}],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$significant_finding_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAZ","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":["release gate"],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$unchecked_result"
|
||||
record "result validator accepts complete verdict.v1" validate_result "$valid_result" "author"
|
||||
if validate_result "$invalid_result" "author"; then echo "FAIL: result validator rejects malformed verdict"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: result validator rejects malformed verdict"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$invalid_date_result" "author"; then echo "FAIL: result validator rejects invalid RFC3339 timestamp"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: result validator rejects invalid RFC3339 timestamp"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$self_judge_result" "author"; then echo "FAIL: PASS result rejects author as validator"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects author as validator"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$blocking_finding_result" "author"; then echo "FAIL: PASS result rejects critical or significant findings"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects critical or significant findings"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$significant_finding_result" "author"; then echo "FAIL: PASS result rejects significant findings"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects significant findings"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$unchecked_result" "author"; then echo "FAIL: PASS result rejects nonempty not_checked"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects nonempty not_checked"; PASS=$((PASS + 1)); fi
|
||||
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: PASS' 'COMMANDS RUN:' 'judge=judge-1 command=go test ./...' 'REASONS:' '- command exited zero' >"$valid_report"
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: FAIL' 'COMMANDS RUN:' 'judge=judge-1 command=go test ./...' 'REASONS:' '- contradictory headings' >"$invalid_report"
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: PASS' 'REASONS:' '- wrong order' 'COMMANDS RUN:' 'judge=judge-1 command=go test ./...' >"$wrong_order_report"
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: PASS' 'COMMANDS RUN:' 'tests looked good' 'REASONS:' '- unsupported prose' >"$unsupported_report"
|
||||
record "report validator accepts measured single verdict" validate_report "$valid_report" "judge-1" "author"
|
||||
if validate_report "$invalid_report" "judge-1" "author"; then echo "FAIL: report validator rejects contradictory verdict headings"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: report validator rejects contradictory verdict headings"; PASS=$((PASS + 1)); fi
|
||||
if validate_report "$wrong_order_report" "judge-1" "author"; then echo "FAIL: report validator rejects REASONS before COMMANDS RUN"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: report validator rejects REASONS before COMMANDS RUN"; PASS=$((PASS + 1)); fi
|
||||
if validate_report "$unsupported_report" "judge-1" "author"; then echo "FAIL: report validator rejects commands without judge source"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: report validator rejects commands without judge source"; PASS=$((PASS + 1)); fi
|
||||
|
||||
echo
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
(( FAIL == 0 ))
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema_version": "2",
|
||||
"generated_at": "2026-07-12T19:00:13Z",
|
||||
"generated_at": "2026-07-12T20:26:54Z",
|
||||
"skill_count": 62,
|
||||
"skills": [
|
||||
{
|
||||
@@ -1702,7 +1702,7 @@
|
||||
"llm-eval-harness"
|
||||
],
|
||||
"context_rel": [],
|
||||
"references_count": 23,
|
||||
"references_count": 24,
|
||||
"codex_override_present": true
|
||||
},
|
||||
{
|
||||
|
||||
+120
-327
@@ -26,372 +26,170 @@ metadata:
|
||||
dependencies: [pawl-review]
|
||||
output_contract: schemas/verdict.v1.schema.json
|
||||
---
|
||||
# /validate — Canonical Validator
|
||||
|
||||
# /validate — Canonical Validator Skill
|
||||
> **Purpose:** Move 6 of the operating loop: independently remeasure a bounded artifact or slice, prove its acceptance, and emit one schema-valid PASS/WARN/FAIL verdict with an executable next action.
|
||||
|
||||
> **Loop position:** move 6 (prove acceptance) of the [operating loop](../../docs/architecture/operating-loop.md) — the driving adapter for the `validate_acceptance` port: every Given/When/Then must map to a passing test before a bead closes, and the verdict binds to the SLICE's own acceptance test (its ATDD contract), not only the parent intent issue. This is S5 of the [narrow-waist micro-cycle](../../docs/architecture/operating-loop.md#the-narrow-waist-micro-cycle-canonical--every-loop-skill-cites-this).
|
||||
## Critical Constraints
|
||||
|
||||
> **Role:** validator. Input = artifact (plan, spec, code, PR, fitness gate). Output = `verdict.v1` (PASS / WARN / FAIL with rationale + findings).
|
||||
- **One role: validator.** Never edit the artifact, commit, push, close tracker work, or operate infrastructure. **Why:** a judge that mutates the subject destroys independence.
|
||||
- Rerun cited commands on the actual artifact; never accept author evidence, conversational memory, or uncited figures as proof. **Why:** validation remeasures rather than agrees.
|
||||
- PASS requires every mandatory check green, all blockers resolved, disclosed `not_checked` coverage, and an independent judge when the mode claims independence. **Why:** missing evidence cannot be laundered into confidence.
|
||||
- `judge_id == author_id` cannot independently PASS; inline `--quick` fallback is stamped waived. **Why:** self-grading is guidance, not assurance.
|
||||
- Every judge brief says: **"READ-ONLY except writing your single verdict file at `<path>`. Do NOT commit, push, or run tracker/infra ops (git push, br/bd, dolt)."** **Why:** role scope is model-independent.
|
||||
- This skill validates work; `ao pawl` certifies landing with a commit-bound verdict. **Why:** a pre-work PASS never pre-authorizes a main write.
|
||||
- Use runtime-native judges for a requested multi-judge mode; never start NTM, Agent Mail, managed agents, Gas City, or another runtime unless explicitly selected. **Why:** judgment does not silently broaden orchestration.
|
||||
- `WARN|FAIL|REFUTED -> AUTO-REDO`: consult the pawl, return findings as re-plan evidence, repair through the owning producer, and rerun the same checks. **Why:** ordinary negative verdicts are self-correction, not human andons.
|
||||
- `BREAKER -> HOLD -> ONE-HELPER`; `HELPER-UNSTUCK -> AUTO-REDO`. Hold judgment and use one bounded local-shell helper to inspect contradictory evidence or a broken validator. **Why:** one recovery pass distinguishes tooling failure from a real stop.
|
||||
- `HELPER-ESCALATE -> HUMAN`; `REFUSAL-LANE|EXPLICIT-JUDGMENT|EXHAUSTED-BUDGET -> HUMAN`. **Why:** waivers, risk acceptance, unavailable authority, refusal, or exhausted recovery require the operator.
|
||||
|
||||
`/validate` is a driving adapter for the `validate_acceptance` port in the
|
||||
[Intent-to-Loop Hexagon](../../docs/architecture/intent-to-loop-hexagon.md).
|
||||
When the artifact contains a `hexagon:` block, preserve the bounded context,
|
||||
context packet, guard adapters, and done state in the verdict.
|
||||
When the artifact claims DONE/closed/green, apply the
|
||||
[Completion-Claim Kernel](../shared/validation-contract.md#completion-claim-kernel)
|
||||
before returning PASS.
|
||||
## Modes
|
||||
|
||||
**A verdict is re-plan evidence, not just a retry trigger.** A FAIL/WARN (and its
|
||||
findings) surfaces UP to the [operating loop](../../docs/architecture/operating-loop.md):
|
||||
the *remaining* moves may be refactored, inserted, dropped, or reordered in response
|
||||
— not only the failed objective re-run. `/rpi` is one turn's executor over that loop,
|
||||
not an autonomous `--auto` re-planner (the old rpi CLI command surface was removed in f61c5f0e7 — historical).
|
||||
Looping a failed objective forever without asking whether the plan should change is
|
||||
the waterfall anti-pattern.
|
||||
|
||||
## Modes (≤8 per Fix-F mode-flag budget)
|
||||
|
||||
| Mode | Purpose | Replaces / absorbs |
|
||||
| Mode | Judge shape | Purpose |
|
||||
|---|---|---|
|
||||
| (default) | 2-judge multi-judge consensus on any artifact | `/council` default |
|
||||
| `--quick` | Inline single-agent structured review | `/council --quick` |
|
||||
| `--deep` | 4-judge thorough review | `/council --deep` |
|
||||
| `--mixed` | Cross-vendor (Claude + Codex), N×2 judges | `/council --mixed` |
|
||||
| `--debate` | Adversarial 2-round refinement | `/council --debate`; `/red-team` (absorbed 2026-07-07) |
|
||||
| `--mode=post-impl` | Code-readiness pipeline (complexity → bug-hunt → council) | `vibe` (absorbed) |
|
||||
| `--mode=pre-impl [--target=X]` | Plan/spec validation; target ∈ {scenario,fitness,ratchet,scope,skill,health} | `/pre-mortem`, `/goals measure`, `/scope`, `/heal-skill` (deep audit), `ao doctor`; `/eval-outcomes` + `/flywheel` (absorbed 2026-07-07) |
|
||||
| `--mode=pr` | PR-shape verdict (diff review + acceptance check) | `/review` (absorbed 2026-07-07) |
|
||||
| default | 2 independent | general artifact consensus |
|
||||
| `--quick` | inline, waived | bounded sanity/readiness check |
|
||||
| `--deep` | 4 perspectives | thorough requirement/feasibility/scope review |
|
||||
| `--mixed` | explicitly selected families | cross-family review |
|
||||
| `--debate` | 2+ judges, 2 rounds | adversarial critique/rebuttal |
|
||||
| `--mode=post-impl` | pipeline + isolated judges | acceptance and completion audit |
|
||||
| `--mode=pre-impl [--target=X]` | 2–4 judges | plan/spec/fitness/scope/skill/health audit |
|
||||
| `--mode=pr` | diff + acceptance judges | submission readiness |
|
||||
|
||||
**Mode-budget assertion:** 8 modes. Adding a 9th requires demoting an existing one OR refusing the addition (per Fix-F § continuous CI gate).
|
||||
**Mode-budget assertion:** 8 modes. Adding a ninth requires merging or removing an existing mode.
|
||||
|
||||
### Folded skills (cp-ki8): `validation` + `pr-validate` retired into these modes
|
||||
|
||||
The retired validation and pr-validate lanes were the Phase-1 placeholders for `--mode=post-impl`
|
||||
and `--mode=pr`; both are now retired (cp-ki8) and their load-bearing contract folded
|
||||
here so no capability is lost:
|
||||
|
||||
- **`--mode=post-impl` (was the validation lane) — full close-out + no-self-grading invariant.**
|
||||
Beyond the inline `complexity → bug-hunt → council` pipeline, this mode owns the
|
||||
`validate_acceptance` port: every Given/When/Then from the intent issue must map to a
|
||||
passing test (criterion→test roll-up; activity logs do not close beads), and the
|
||||
acceptance verdict **must be produced by a blind, context-isolated sub-agent judge that
|
||||
did not author the code** (author ≠ validator — `ag-9jle.5` / `ag-lmdx.4`). Refuse to
|
||||
certify acceptance when `judge_id == author_id`; the only escape is an inline-fallback
|
||||
self-grade that is stamped as *waived, not independently validated*. Apply the
|
||||
[Completion-Claim Kernel](../shared/validation-contract.md#completion-claim-kernel)
|
||||
before accepting any DONE/closed/green claim. For epic-scope close-out this mode may
|
||||
delegate to `vibe`, `/post-mortem`, and `/curate --mode=forge` rather than inlining them.
|
||||
- **`--mode=pr` (was the pr-validate lane) — submission-readiness checks.** In addition to the
|
||||
diff/acceptance verdict, run, in order: (1) **upstream alignment FIRST** (BLOCKING —
|
||||
`git rev-list --count HEAD..origin/main`; fail if many commits behind or merge would
|
||||
conflict), (2) CONTRIBUTING.md compliance (BLOCKING), (3) **isolation** — single commit
|
||||
type + thematic files + atomic scope, (4) scope-creep containment, (5) quality gate
|
||||
(tests/lint, non-blocking). On FAIL, emit remediation steps (split-by-type cherry-pick,
|
||||
rebase-on-upstream) so the verdict is actionable.
|
||||
|
||||
### Folded triggers (ag-s43tg wave 1): `vibe` + `bead-completion-audit` route here
|
||||
|
||||
- **`vibe` → `--mode=post-impl`.** Use when doing a quick readiness or sanity check
|
||||
that code is ready to commit or ship, short of a full review — the post-impl
|
||||
pipeline (complexity → bug-hunt → council) is the vibe check.
|
||||
- **`bead-completion-audit` → `--mode=post-impl` close-out.** Use when
|
||||
auditing closed beads for real shipped evidence, acceptance proof, and truthful
|
||||
closeout — the Completion-Claim Kernel and the no-self-grading invariant above
|
||||
own this audit.
|
||||
Folded triggers remain load-bearing: **`vibe` → `--mode=post-impl`** for code
|
||||
readiness, and `bead-completion-audit` routes to post-implementation closeout.
|
||||
Detailed target and evidence rules live in
|
||||
[canonical-validation-protocol](references/canonical-validation-protocol.md).
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
/validate path/to/plan.md # default 2-judge consensus
|
||||
/validate --quick path/to/plan.md # inline single-agent
|
||||
/validate --deep path/to/spec.md # 4-judge thorough
|
||||
/validate --mode=pre-impl path/to/plan.md # pre-mortem mode
|
||||
/validate --mode=post-impl recent # vibe mode (post-implement)
|
||||
/validate --mode=pr 123 # PR review by PR number
|
||||
/validate --mode=pre-impl --target=fitness # fitness gate against GOALS.md
|
||||
/validate path/to/artifact
|
||||
/validate --quick path/to/artifact
|
||||
/validate --deep path/to/spec.md
|
||||
/validate --mode=pre-impl --target=skill skills/example
|
||||
/validate --mode=post-impl recent
|
||||
/validate --mode=pr 123
|
||||
```
|
||||
|
||||
Default uses runtime-native subagent spawning. Falls back to `--quick` (inline) when no multi-agent capability detected.
|
||||
## Execution Workflow
|
||||
|
||||
## Execution
|
||||
### 1. Resolve mode, artifact, and authority
|
||||
|
||||
### Step 1: Resolve mode + target
|
||||
|
||||
Parse `--mode` and `--target`. Default mode is multi-judge. Validate combinations:
|
||||
|
||||
| Mode | Allowed `--target` |
|
||||
|---|---|
|
||||
| default, --quick, --deep, --mixed, --debate | n/a |
|
||||
| --mode=post-impl | n/a (pipeline scope is recent code changes) |
|
||||
| --mode=pre-impl | scenario, fitness, ratchet, scope, skill, health (default: pre-mortem on plan) |
|
||||
| --mode=pr | n/a (PR ID/path is positional) |
|
||||
|
||||
Reject invalid combinations (e.g., `--mode=pr --target=fitness`).
|
||||
|
||||
### Step 2: Load artifact + context
|
||||
Reject invalid mode/target combinations. Confirm the artifact exists, is within
|
||||
scope, and has explicit acceptance. For a goal-design packet, run:
|
||||
|
||||
```bash
|
||||
# resolve artifact:
|
||||
ARTIFACT="${1:-recent}" # path, PR ID, or "recent"
|
||||
|
||||
# load FAIL patterns:
|
||||
# (folded into skill body; not a separate hook)
|
||||
scripts/check-goal-design-packet.sh <packet-dir>
|
||||
```
|
||||
|
||||
If `ARTIFACT` is a goal-design packet directory containing `intent.md` and
|
||||
`driver.md`, run the deterministic packet checker before spawning or judging:
|
||||
Nonzero is deterministic FAIL evidence. Load only the selected section of
|
||||
[canonical-validation-protocol](references/canonical-validation-protocol.md)
|
||||
plus the relevant language/risk standards.
|
||||
|
||||
```bash
|
||||
scripts/check-goal-design-packet.sh "$ARTIFACT"
|
||||
```
|
||||
**Checkpoint:** record artifact identity, commit/digest, mode, targets, author id,
|
||||
required checks, and `not_checked` before dispatch.
|
||||
|
||||
Non-zero exit is a FAIL verdict. Include the command and output in
|
||||
`COMMANDS RUN:`. This is a pre-check inside the existing validator role, not a
|
||||
new mode.
|
||||
### 2. Run deterministic prechecks
|
||||
|
||||
For `--mode=pre-impl`, also load:
|
||||
- `.agents/planning-rules/*.md` (compiled planning rules)
|
||||
- `.agents/findings/registry.jsonl` (active findings)
|
||||
- `.agents/pre-mortem-checks/*.md` (compiled prevention)
|
||||
- pre-implementation: target rubric, temporal/error-rescue checks, test pyramid;
|
||||
- post-implementation: complexity, bug sweep, slice acceptance, completion kernel;
|
||||
- PR: upstream alignment first, contribution rules, atomicity, scope, tests/lint;
|
||||
- all modes: source precedence, artifact freshness, and output-schema availability.
|
||||
|
||||
For `--mode=post-impl`, run pre-checks:
|
||||
- complexity audit (radon for python, gocyclo for go)
|
||||
- bug-hunt sweep (skill-body convention; no `/review` skill needed)
|
||||
A deterministic red result is a FAIL verdict; do not spend judge tokens to
|
||||
rediscover it.
|
||||
|
||||
For `--mode=pr`, fetch the PR diff (`gh pr diff <id>` or path).
|
||||
### 3. Dispatch isolated judges
|
||||
|
||||
> **LIVE-PATH — where this judge orchestration is operative, and where the pawl is.**
|
||||
> Steps 3–7 (spawn backend → run judges → write the `.agents/council/` verdict) are the live path
|
||||
> for **PRE-work and non-merge validation**: plans, specs, artifacts, fitness gates, mid-arc
|
||||
> completion checks, and the `--mode=pre-impl` / `post-impl` / `pr` verdicts. **At the merge-to-main
|
||||
> door this orchestration is NOT the operative machinery** — that door is driven by the pawl scripts:
|
||||
> `scripts/pawl-review.sh` runs the cross-family refuter and, on CONFIRMED, `scripts/pawl-verdict.sh`
|
||||
> writes the commit-bound verdict the pre-push gate enforces (see [`/push`](../push/SKILL.md) and
|
||||
> [`pawl-review`](../pawl-review/SKILL.md)). Use this skill to validate the WORK; use
|
||||
> `ao pawl` to certify the LANDING — they compose, not compete. The judge machinery below is retained for
|
||||
> that pre-work / non-merge scope, not deleted.
|
||||
Register dispatch intent before spawning. Use the current runtime's native
|
||||
subagents for multi-judge modes; if unavailable, degrade only to explicit
|
||||
`--quick` and stamp the independence waiver.
|
||||
|
||||
### Step 3: Determine spawn backend
|
||||
Each judge receives the artifact path/digest, bounded context, mandatory checks,
|
||||
verdict path, and exact read-only clamp. It independently reruns commands.
|
||||
|
||||
1. `spawn_agent` available → Codex sub-agent
|
||||
2. `TeamCreate` available → Claude native team
|
||||
3. `task` (read-only skill tool, OpenCode) → opencode subagent
|
||||
4. None → fall back to `--quick` (inline single-agent)
|
||||
**Checkpoint:** author and judge identities differ, dispatch scopes do not
|
||||
collide, and every claimed judge produced a nonempty verdict artifact.
|
||||
|
||||
Log selected backend in the verdict frontmatter.
|
||||
### 4. Consolidate fail-closed
|
||||
|
||||
### Step 4: Run judges
|
||||
- PASS: all required judges/checks PASS (or the declared deep-mode majority), no blocker.
|
||||
- WARN: nonblocking concern or explicit coverage gap remains.
|
||||
- FAIL: any deterministic failure, blocker, counterfeit/self judge, stale artifact, or malformed evidence.
|
||||
|
||||
| Mode | Judges | Perspectives |
|
||||
|---|---|---|
|
||||
| default | 2 | independent (no labeled perspectives) |
|
||||
| --deep | 4 | missing-requirements, feasibility, scope, spec-completeness |
|
||||
| --mixed | 2N (default N=3) | same N perspectives across Claude + Codex |
|
||||
| --debate | 2+ rounds | adversarial; 2 rounds with critique-rebuttal |
|
||||
| --quick | 0 (inline self) | structured review |
|
||||
| --mode=post-impl | 2 + pipeline | complexity → bug-hunt → 2-judge council |
|
||||
| --mode=pre-impl | 2-4 | per target preset |
|
||||
| --mode=pr | 2 | diff-review + acceptance-check |
|
||||
Contradictory verdicts are FAIL until reconciled; never average away a blocker.
|
||||
|
||||
Each judge gets:
|
||||
- artifact path
|
||||
- relevant context (planning rules, findings)
|
||||
- council FAIL pattern check prompt (top 8)
|
||||
- temporal interrogation prompt (--deep + --target=plan)
|
||||
### 5. Write and validate outputs
|
||||
|
||||
### Step 5: Mandatory checks (auto-trigger)
|
||||
Write `.agents/council/YYYY-MM-DD-validate-<slug>.md` with exactly one anchored
|
||||
`## Council Verdict: PASS|WARN|FAIL`, plus `result.json` matching
|
||||
`schemas/verdict.v1.schema.json`. The report contains exactly one anchored
|
||||
`VERDICT:`, a nonempty `COMMANDS RUN:`, `REASONS:`, findings, and `not_checked`.
|
||||
|
||||
For `--mode=pre-impl --target=plan`:
|
||||
- temporal interrogation (auto for plans with 5+ files or 3+ deps)
|
||||
- error & rescue map
|
||||
- council FAIL pattern check (top 8)
|
||||
- test pyramid coverage check
|
||||
- input validation check (enum-like fields)
|
||||
**Checkpoint:** run both validators in Output Specification; PASS is illegal if
|
||||
either output is missing, malformed, stale, self-graded, or unsupported by commands.
|
||||
|
||||
For `--mode=post-impl`:
|
||||
- L0/L1/L2 coverage check on changed files
|
||||
- **refactor-under-green check:** if the diff is a refactor, assert no acceptance/unit test text changed (`git diff` of test files is empty) — a refactor that edits a test changed behavior and is a new slice, not a refactor (S4). FAIL if a refactor diff also edits a test.
|
||||
### 6. Route the verdict
|
||||
|
||||
For `--mode=pre-impl --target=fitness`:
|
||||
- read GOALS.md
|
||||
- evaluate each gate against current state
|
||||
- report PASS/WARN/FAIL per gate + aggregate
|
||||
|
||||
### Step 6: Consolidate to verdict
|
||||
|
||||
Each judge returns a per-judge result. Consolidate:
|
||||
- PASS only if all judges PASS (or majority for --deep)
|
||||
- WARN if any judge raises a warning the others don't dispute
|
||||
- FAIL if any judge raises a blocker the others don't override
|
||||
|
||||
### Step 7: Write verdict
|
||||
|
||||
Output path: `.agents/council/YYYY-MM-DD-validate-<topic-slug>.md`
|
||||
|
||||
```markdown
|
||||
---
|
||||
id: validate-YYYY-MM-DD-<slug>
|
||||
type: verdict
|
||||
date: YYYY-MM-DD
|
||||
mode: <mode>
|
||||
target: <target or n/a>
|
||||
artifact: <path>
|
||||
backend: <codex-subagents | claude-teams | opencode | inline>
|
||||
---
|
||||
|
||||
# Validate Verdict — <topic>
|
||||
|
||||
## Council Verdict: PASS / WARN / FAIL
|
||||
|
||||
| Failure mode | Risk | Severity | Addressed? |
|
||||
|---|---|---|---|
|
||||
| ... | ... | ... | ... |
|
||||
|
||||
## Pseudocode Fixes (when WARN/FAIL)
|
||||
(copy-pastable into affected issues per pre-mortem 4.6 contract)
|
||||
|
||||
## FAIL Pattern Check
|
||||
(top 8 patterns — status per pattern)
|
||||
|
||||
## Verdict
|
||||
PASS — proceed
|
||||
WARN — review concerns, accept risk, or apply fixes
|
||||
FAIL — block; revise artifact and rerun
|
||||
```
|
||||
|
||||
The exact heading `## Council Verdict: PASS / WARN / FAIL` is mandatory — downstream validators and ledger readers parse it with anchored regex.
|
||||
|
||||
### Step 8: Persist findings (when applicable)
|
||||
|
||||
For `--mode=pre-impl` reusable findings: append to `.agents/findings/registry.jsonl` (atomic temp+rename). Every FAIL finding must be emitted as a next-loop producer — a `/pre-mortem` check or `/plan` planning-rule keyed by dedup_key (the escape→check ratchet, S6), not just a logged finding; a defect the membrane caught that produces no next-loop check will recur. See [operating-loop.md → move 7](../../docs/architecture/operating-loop.md).
|
||||
|
||||
### Step 9: Report
|
||||
|
||||
1. Verdict (PASS/WARN/FAIL).
|
||||
2. Key concerns (when not PASS).
|
||||
3. Output path.
|
||||
4. Recommended next action.
|
||||
|
||||
## --target taxonomy (pre-impl)
|
||||
|
||||
| `--target` | What gets graded | Replaces |
|
||||
|---|---|---|
|
||||
| (default) | Plan/spec for an upcoming `/implement` | `/pre-mortem` |
|
||||
| scenario | Holdout scenario gate | `/eval-outcomes` |
|
||||
| fitness | GOALS.md fitness gates | `/goals measure`, `ao goals measure` |
|
||||
| ratchet | Brownian Ratchet checkpoint | `/flywheel`, `ao ratchet status` |
|
||||
| scope | Frozen-dirs declaration | `/scope` |
|
||||
| skill | SKILL.md hygiene + audit | `/heal-skill` (heal.sh hygiene + audit.sh deep audit) |
|
||||
| health | Repo health probe | `ao doctor` |
|
||||
|
||||
Each target has its own inline check rubric until Phase 2 extraction.
|
||||
|
||||
## Validation discipline (2026-06-09, cards 6–10, cp-hhd7)
|
||||
|
||||
### Verdict form — the gate parses these lines anchored
|
||||
|
||||
```
|
||||
VERDICT: PASS
|
||||
(blank line)
|
||||
COMMANDS RUN:
|
||||
<actual commands + verbatim output snippets>
|
||||
REASONS:
|
||||
- bullet citing a COMMANDS RUN line
|
||||
```
|
||||
|
||||
A verdict with no `COMMANDS RUN:` section is **unverified** — reject it and
|
||||
dispatch a fresh validator. A verdict whose `COMMANDS RUN:` lists only commands
|
||||
the **author** ran (not the judge) is a **counterfeit judge** — treat as FAIL and
|
||||
re-route to a genuinely independent validator. No `##` headings or parentheticals
|
||||
on `VERDICT:` or `COMMANDS RUN:` lines; the gate parses them anchored.
|
||||
|
||||
### Cross-family floor (POLICY → gate icb6 enforces; this skill supports)
|
||||
|
||||
For **assurance closes** (the control-plane verdict-gate, cp-icb6), the floor is
|
||||
≥2 verdicts from ≥2 distinct model families, author family excluded, fail-closed.
|
||||
**This skill supports that policy via `--mixed` mode and the verdict form above;
|
||||
the policy itself lives in the gate, not here.** A same-model council is valid for
|
||||
non-assurance decisions (design brainstorms, quick checks) — do not refuse those.
|
||||
Tier mapping:
|
||||
- **STRICT** (irreversible, security, production close): Codex + strong-Gemini (A1); or Fable + Codex (A2).
|
||||
- **ROUTINE** (everyday close): Codex + non-author-Claude (A3); or Fable + non-author (A4).
|
||||
|
||||
The A7 ruling (2026-06-09, memory `validation-family-policy-risk-tiered`):
|
||||
Gemini is currently **benched** for STRICT validation — use Codex + Fable for A1/A2
|
||||
tiers. Gemini may return for STRICT when Bo graduates it from the bench. Do not
|
||||
present Gemini paths as live for tier A1/A2 until then.
|
||||
|
||||
### Judge empirically on a differentiating fixture (card 9, cp-8720)
|
||||
|
||||
When two implementations of the same intent exist, do NOT award based on
|
||||
authorship or surface aesthetics. Run both on a **differentiating fixture** (an
|
||||
input that exposes their behavioral difference), record the outputs verbatim, and
|
||||
graft the loser's unique assets onto the winner. "My worker wrote it" is not
|
||||
evidence.
|
||||
|
||||
### Dispatch record first (card 3, cp-hhtu)
|
||||
|
||||
Before dispatching a validator, register intent on the bead graph (update status,
|
||||
assign actor). **Two parallel validators on the same bead produce a dedup incident,
|
||||
not a cross-family quorum.** Check for an existing actor before spawning.
|
||||
|
||||
### Judge lanes carry an explicit write-scope clamp (2026-07-02, showcase kernel R2)
|
||||
|
||||
Every judge brief states, verbatim: **"READ-ONLY except writing your single verdict
|
||||
file at `<path>`. Do NOT commit, push, or run tracker/infra ops (git push, br/bd,
|
||||
dolt)."** The clamp is role-scoped, not model-scoped — crank workers legitimately
|
||||
hold broad write scopes; judges re-measure, they never mutate. Proven live: an
|
||||
unclamped codex acceptance judge pushed the feature branch unprompted and attempted
|
||||
`bd dolt push` twice mid-judgment. A judge that mutates while judging can corrupt
|
||||
the artifact under judgment or preempt the pawl.
|
||||
|
||||
### Judges re-measure; they do not read (card 8)
|
||||
|
||||
A judge re-runs the cited commands on the actual artifacts. It does not read the
|
||||
author's evidence file and agree. Attest `judge_source: <model>` inside `COMMANDS RUN`
|
||||
so the gate can confirm the judge identity. A judge that ran nothing is a reader, not
|
||||
a verifier — discard its verdict.
|
||||
|
||||
## Evidence figures are measured, never inferred (the cp-801l lesson)
|
||||
|
||||
A worker's evidence file may only contain numbers and outputs that were **captured** —
|
||||
pasted verbatim from a command's output — never reconstructed from memory. The
|
||||
canonical failure: "36 checks — 35 pass" stated with confidence was inference; the
|
||||
measured reality was 36 run / 34 pass / 1 fail / 1 skip, on a different commit.
|
||||
|
||||
- **Validators:** treat any uncited figure (a count, a pass-rate, a timing, a commit
|
||||
hash) as **unverifiable → FAIL** until a log is produced or the claim is corrected.
|
||||
- **Corrections happen by APPENDED erratum** — a dated erratum block crediting the
|
||||
source measurement — never by silently editing the original figure. A silently
|
||||
edited evidence file is indistinguishable from a fabricated one.
|
||||
|
||||
## Constraints (one-role-per-skill)
|
||||
|
||||
- **One role: validator.** Output is always a verdict. Never mutates code (delegates to `/implement` for fixes).
|
||||
- **No new modes** without dropping/merging an existing one (Fix-F mode-budget cap = 8).
|
||||
- **Verdict heading is regex-anchored** — do not alter the `## Council Verdict: ...` text format.
|
||||
Report verdict, key findings, artifact paths, and one executable next action.
|
||||
WARN/FAIL re-enters the operating loop through the owning producer. Reusable
|
||||
failures compile into a planning/pre-mortem check instead of becoming a dead log.
|
||||
|
||||
## Output Specification
|
||||
|
||||
**Format:** a PASS/WARN/FAIL validation verdict plus a markdown council summary to stdout; machine-readable `result.json`.
|
||||
**Files:** writes `.agents/council/YYYY-MM-DD-validate-<slug>.md` and `result.json`; appends reusable findings to `.agents/findings/registry.jsonl`; may refresh `.agents/planning-rules/` and `.agents/pre-mortem-checks/`.
|
||||
**Exit signal:** FAIL re-cranks on the same objective (up to 3 attempts). Exhaustion enters HOLD and permits exactly one bounded fresh-context or cross-family helper pass; HELPER-UNSTUCK resumes with the new approach, while only HELPER-ESCALATE (or a refusal/explicit-judgment or spent hard-ceiling class) routes to the human. DONE requires a fully-green acceptance roll-up.
|
||||
**Artifact directory:** markdown verdicts under `.agents/council/`; machine-readable `result.json` at the invocation output root.
|
||||
|
||||
## See Also
|
||||
**Filename convention:** `.agents/council/YYYY-MM-DD-validate-<topic-slug>.md` plus `result.json`.
|
||||
|
||||
- `skills/rpi/SKILL.md` — orchestrator that fires `/validate --mode=pre-impl` after `/plan`
|
||||
- `skills/curate/SKILL.md` — miner role (paired canonical skill)
|
||||
- `schemas/verdict.v1.schema.json` — output contract
|
||||
- [`pawl-review`](../pawl-review/SKILL.md) — executes immutable fresh reviewer lanes and hands evidence to `ao pawl`; the membrane alone binds the commit verdict
|
||||
**Serialization/schema format:** `result.json` follows closed `schemas/verdict.v1.schema.json`; markdown has one `## Council Verdict: PASS|WARN|FAIL` and the anchored evidence form in the protocol reference.
|
||||
|
||||
**Validator command:** validate `result.json` with `Draft202012Validator(schema, format_checker=FormatChecker())`, an explicit timezone-aware RFC3339 parse of `validated_at`, and the pinned author identity so a PASS requires `validator_session != author_session`; then verify the markdown has exactly one council verdict, exactly one matching `VERDICT:`, and a `judge=<validator_session> command=<command>` evidence line under `COMMANDS RUN:` before `REASONS:`. The executable fixture suite is `bash skills/validate/scripts/validate.sh`.
|
||||
|
||||
**Downstream handoff:** pass mode/target, artifact path/digest/commit, author and judge identities/families, deterministic commands and exit codes, verdict/report paths, findings, `not_checked`, waiver/breaker state, and the exact next action; landing still requires `ao pawl`.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- [ ] Artifact identity and acceptance are pinned before checks run.
|
||||
- [ ] Judges are read-only, independent when claimed, and rerun commands themselves.
|
||||
- [ ] Every number, timing, and commit is captured from cited output.
|
||||
- [ ] PASS has no blockers, counterfeit judges, stale evidence, or undisclosed gaps.
|
||||
- [ ] Markdown and JSON agree and pass their deterministic validators.
|
||||
- [ ] WARN/FAIL/REFUTED returns to AUTO-REDO; only real breakers reach a human.
|
||||
- [ ] Validation PASS is never substituted for the commit-bound landing verdict.
|
||||
|
||||
## Examples
|
||||
|
||||
### Reject a counterfeit completion claim
|
||||
|
||||
**User says:** `/validate --mode=post-impl recent`
|
||||
|
||||
**What happens:** the isolated judge reruns the slice acceptance command and
|
||||
finds the author's reported count was inferred and the test exits nonzero.
|
||||
|
||||
**Result:** FAIL with captured output, corrected count, owning producer, and the
|
||||
same validation command as next action; no human andon and no landing claim.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Response |
|
||||
|---|---|
|
||||
| Judge unavailable | Use explicit waived `--quick` or HOLD only if independence is required |
|
||||
| Artifact changed mid-review | Mark stale, pin the new digest, rerun all checks |
|
||||
| Judges disagree | FAIL until the blocker is reproduced and reconciled |
|
||||
| Empty `COMMANDS RUN:` | Discard the verdict and dispatch a real verifier |
|
||||
|
||||
## Reference Documents
|
||||
|
||||
- [references/validate.feature](references/validate.feature) — Executable spec: verdict.v1 PASS/WARN/FAIL for any artifact, --mode selects shape, 8-mode budget (soc-qk4b)
|
||||
|
||||
## Reference library (incl. rescued vibe references, ag-s43tg)
|
||||
|
||||
- [references/canonical-validation-protocol.md](references/canonical-validation-protocol.md) — modes, targets, evidence, isolation, landing boundary
|
||||
- [references/report-format.md](references/report-format.md) — markdown detail
|
||||
- [references/deep-audit-protocol.md](references/deep-audit-protocol.md) — deep-mode perspectives
|
||||
- [references/test-pyramid-inventory.md](references/test-pyramid-inventory.md) — coverage selection
|
||||
- [references/post-verdict-actions.md](references/post-verdict-actions.md) — bounded follow-through
|
||||
- [references/validate.feature](references/validate.feature) — executable behavior
|
||||
- [references/complexity-analysis.md](references/complexity-analysis.md)
|
||||
- [references/deep-audit-protocol.md](references/deep-audit-protocol.md)
|
||||
- [references/deep-checks.md](references/deep-checks.md)
|
||||
- [references/examples.md](references/examples.md)
|
||||
- [references/go-patterns.md](references/go-patterns.md)
|
||||
@@ -399,13 +197,10 @@ measured reality was 36 run / 34 pass / 1 fail / 1 skip, on a different commit.
|
||||
- [references/json-standards.md](references/json-standards.md)
|
||||
- [references/markdown-standards.md](references/markdown-standards.md)
|
||||
- [references/patterns.md](references/patterns.md)
|
||||
- [references/post-verdict-actions.md](references/post-verdict-actions.md)
|
||||
- [references/python-standards.md](references/python-standards.md)
|
||||
- [references/quick-mode-vibe.md](references/quick-mode-vibe.md)
|
||||
- [references/report-format.md](references/report-format.md)
|
||||
- [references/rust-standards.md](references/rust-standards.md)
|
||||
- [references/shell-standards.md](references/shell-standards.md)
|
||||
- [references/test-pyramid-inventory.md](references/test-pyramid-inventory.md)
|
||||
- [references/test-pyramid-weighting.md](references/test-pyramid-weighting.md)
|
||||
- [references/typescript-standards.md](references/typescript-standards.md)
|
||||
- [references/verification-report.md](references/verification-report.md)
|
||||
@@ -413,6 +208,4 @@ measured reality was 36 run / 34 pass / 1 fail / 1 skip, on a different commit.
|
||||
- [references/vibe-suppressions.md](references/vibe-suppressions.md)
|
||||
- [references/write-time-quality.md](references/write-time-quality.md)
|
||||
- [references/yaml-standards.md](references/yaml-standards.md)
|
||||
- [references/validate.feature](references/validate.feature)
|
||||
- [references/vibe.feature](references/vibe.feature) — rescued vibe executable spec
|
||||
- [scripts/prescan.sh](scripts/prescan.sh) — rescued vibe pre-scan helper
|
||||
- [references/vibe.feature](references/vibe.feature)
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Canonical Validation Protocol
|
||||
|
||||
This reference carries the detailed mode, target, and evidence rules for the
|
||||
`validate` kernel. Load only the sections selected by the current mode.
|
||||
|
||||
## Mode details
|
||||
|
||||
- Default: two independent judges; PASS requires both PASS.
|
||||
- `--quick`: inline structured review. It may guide work but is stamped
|
||||
`waived`, never independently validated.
|
||||
- `--deep`: four perspectives—missing requirements, feasibility, scope, and
|
||||
specification completeness. PASS uses the declared majority rule, with any
|
||||
unresolved blocker still failing closed.
|
||||
- `--mixed`: the same perspectives across explicitly selected model families.
|
||||
- `--debate`: two adversarial rounds with critique and rebuttal.
|
||||
- `--mode=post-impl`: complexity, bug sweep, acceptance roll-up, then isolated
|
||||
judges. A refactor that changes acceptance/unit test text is a new behavior
|
||||
slice, not a refactor.
|
||||
- `--mode=pre-impl`: plan/spec checks, optionally specialized by `--target`.
|
||||
- `--mode=pr`: upstream alignment first, contribution rules, atomic isolation,
|
||||
scope containment, then tests/lint.
|
||||
|
||||
The eight-mode budget is fixed. A ninth mode must replace or merge an existing
|
||||
mode rather than growing the public surface.
|
||||
|
||||
## Pre-implementation targets
|
||||
|
||||
| Target | Required checks |
|
||||
|---|---|
|
||||
| default plan | temporal interrogation, error/rescue map, FAIL patterns, test pyramid, enum/input validation |
|
||||
| scenario | holdout scenario and falsifying edge |
|
||||
| fitness | each GOALS.md gate against current measured state |
|
||||
| ratchet | current checkpoint, evidence, and legal next transition |
|
||||
| scope | frozen paths, authority boundary, and escape prevention |
|
||||
| skill | strict hygiene plus profile-aware deep audit |
|
||||
| health | executable repository health probes and disclosed gaps |
|
||||
|
||||
A goal-design packet containing `intent.md` and `driver.md` first runs
|
||||
`scripts/check-goal-design-packet.sh <packet-dir>`; nonzero is FAIL evidence.
|
||||
|
||||
## Post-implementation acceptance
|
||||
|
||||
Every Given/When/Then maps to a passing acceptance test for its own vertical
|
||||
slice. Activity logs and parent-issue summaries do not close a bead. Apply the
|
||||
Completion-Claim Kernel in `skills/shared/validation-contract.md` to every
|
||||
DONE/closed/green claim.
|
||||
|
||||
The acceptance judge is blind and context-isolated. `judge_id == author_id`
|
||||
cannot independently PASS. Inline fallback is marked waived and cannot satisfy
|
||||
an assurance close.
|
||||
|
||||
## PR checks
|
||||
|
||||
Run in order:
|
||||
|
||||
1. upstream alignment (`git rev-list --count HEAD..origin/main`) and conflict risk;
|
||||
2. repository contribution rules;
|
||||
3. one thematic/atomic change shape;
|
||||
4. scope-creep containment;
|
||||
5. relevant tests and lint.
|
||||
|
||||
FAIL includes executable remediation such as rebase or split-by-type; it never
|
||||
mutates the branch from the judge lane.
|
||||
|
||||
## Evidence discipline
|
||||
|
||||
The validator reruns cited commands on the actual artifact. It does not accept
|
||||
the author's evidence file as proof. Every count, timing, commit, and pass rate
|
||||
is pasted from captured output. Uncited figures fail until measured.
|
||||
|
||||
Corrections are appended as dated errata crediting the source measurement;
|
||||
silently rewriting an evidence claim is forbidden.
|
||||
|
||||
Verdict text uses anchored lines:
|
||||
|
||||
```text
|
||||
VERDICT: PASS
|
||||
|
||||
COMMANDS RUN:
|
||||
judge=<validator_session> command=<command rerun by this judge>
|
||||
<bounded output excerpts produced by this judge>
|
||||
REASONS:
|
||||
- each reason cites a command or artifact
|
||||
```
|
||||
|
||||
Exactly one verdict is allowed. `COMMANDS RUN:` must include at least one
|
||||
`judge=<validator_session> command=<command>` line, and the pinned author identity
|
||||
must differ from that validator session for PASS. A judge that ran nothing is a
|
||||
reader, not a verifier.
|
||||
|
||||
## Judge isolation
|
||||
|
||||
Every judge brief includes this exact clamp:
|
||||
|
||||
> READ-ONLY except writing your single verdict file at `<path>`. Do NOT commit,
|
||||
> push, or run tracker/infra ops (git push, br/bd, dolt).
|
||||
|
||||
Register dispatch intent before spawning. Two validators accidentally assigned
|
||||
to the same lane/bead are a dedup incident, not an independent quorum.
|
||||
|
||||
## Assurance and landing
|
||||
|
||||
Non-merge validation may use runtime-native judges. Assurance close requires the
|
||||
policy-selected independent-family floor. Benched families are not presented as
|
||||
live strict lanes.
|
||||
|
||||
Landing is a different door: `ao pawl` produces the commit-bound verdict and the
|
||||
pre-push gate authorizes the remote-main write. A validation PASS never replaces
|
||||
or pre-authorizes that landing proof.
|
||||
|
||||
## Failure routing
|
||||
|
||||
WARN/FAIL/REFUTED findings return to the operating loop as re-plan evidence.
|
||||
Repair and remeasure automatically while the bounded budget has progress.
|
||||
HOLD/ESCALATE occurs only on a breaker: exhausted attempts/time/cost, oscillation,
|
||||
refusal, unavailable authority, or explicit risk/waiver judgment.
|
||||
Executable
+122
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
REPO_ROOT="$(cd "$SKILL_DIR/../.." && pwd)"
|
||||
SKILL_MD="$SKILL_DIR/SKILL.md"
|
||||
SCHEMA="$REPO_ROOT/schemas/verdict.v1.schema.json"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
record() {
|
||||
local label="$1"
|
||||
shift
|
||||
if "$@"; then echo "PASS: $label"; PASS=$((PASS + 1));
|
||||
else echo "FAIL: $label"; FAIL=$((FAIL + 1)); fi
|
||||
}
|
||||
|
||||
validate_contract() {
|
||||
local skill_md="$1"
|
||||
[[ "$(awk '/^---$/{n++;next} n==2 && /^## /{print;exit}' "$skill_md")" == "## Critical Constraints" ]] &&
|
||||
grep -Fq 'WARN|FAIL|REFUTED -> AUTO-REDO' "$skill_md" &&
|
||||
grep -Fq 'BREAKER -> HOLD -> ONE-HELPER' "$skill_md" &&
|
||||
grep -Fq 'HELPER-UNSTUCK -> AUTO-REDO' "$skill_md" &&
|
||||
grep -Fq 'HELPER-ESCALATE -> HUMAN' "$skill_md" &&
|
||||
grep -Fq 'REFUSAL-LANE|EXPLICIT-JUDGMENT|EXHAUSTED-BUDGET -> HUMAN' "$skill_md" &&
|
||||
grep -Fq '**Mode-budget assertion:** 8 modes.' "$skill_md" &&
|
||||
grep -Fq 'vibe` → `--mode=post-impl`' "$skill_md" &&
|
||||
grep -Fq 'READ-ONLY except writing your single verdict file' "$skill_md" &&
|
||||
grep -Fq 'This skill validates work; `ao pawl` certifies landing' "$skill_md" &&
|
||||
grep -Fq '**Checkpoint:**' "$skill_md" &&
|
||||
grep -Fq '**Artifact directory:**' "$skill_md" &&
|
||||
grep -Fq '**Filename convention:**' "$skill_md" &&
|
||||
grep -Fq '**Serialization/schema format:**' "$skill_md" &&
|
||||
grep -Fq '**Validator command:**' "$skill_md" &&
|
||||
grep -Fq '**Downstream handoff:**' "$skill_md" &&
|
||||
grep -Fq '## Quality Checklist' "$skill_md"
|
||||
}
|
||||
|
||||
validate_result() {
|
||||
local result="$1"
|
||||
local author_session="$2"
|
||||
python3 -c 'import json,re,sys; from datetime import datetime; from jsonschema import Draft202012Validator,FormatChecker; schema=json.load(open(sys.argv[1])); instance=json.load(open(sys.argv[2])); author=sys.argv[3]; Draft202012Validator.check_schema(schema); Draft202012Validator(schema,format_checker=FormatChecker()).validate(instance); ts=instance["validated_at"]; assert re.fullmatch(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})",ts); assert datetime.fromisoformat(ts.replace("Z","+00:00")).tzinfo is not None; assert instance["verdict"] != "PASS" or (instance["validator_session"] and instance["validator_session"] != author and not any(f["severity"] in {"critical","significant"} for f in instance["findings"]) and not instance["not_checked"])' "$SCHEMA" "$result" "$author_session" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
validate_report() {
|
||||
local report="$1"
|
||||
local validator_session="$2"
|
||||
local author_session="$3"
|
||||
local council_verdict report_verdict
|
||||
council_verdict="$(sed -nE 's/^## Council Verdict: (PASS|WARN|FAIL)$/\1/p' "$report")"
|
||||
report_verdict="$(sed -nE 's/^VERDICT: (PASS|WARN|FAIL)$/\1/p' "$report")"
|
||||
[[ "$(grep -Ec '^## Council Verdict: (PASS|WARN|FAIL)$' "$report")" -eq 1 ]] &&
|
||||
[[ "$(grep -Ec '^VERDICT: (PASS|WARN|FAIL)$' "$report")" -eq 1 ]] &&
|
||||
[[ "$council_verdict" == "$report_verdict" ]] &&
|
||||
[[ -n "$validator_session" ]] &&
|
||||
[[ "$report_verdict" != "PASS" || "$validator_session" != "$author_session" ]] &&
|
||||
awk -v judge="$validator_session" 'BEGIN{state=0;commands=0;reasons=0;seen=0;sourced=0;invalid=0;prefix="judge=" judge " command="} /^COMMANDS RUN:$/{commands++;if(state!=0)invalid=1;state=1;next} /^REASONS:$/{reasons++;if(state!=1||!seen)invalid=1;state=2;next} state==1&&NF{seen=1;if(index($0,prefix)==1&&length($0)>length(prefix))sourced=1} END{exit !(commands==1&&reasons==1&&seen&&sourced&&state==2&&!invalid)}' "$report"
|
||||
}
|
||||
|
||||
record "validate contract is complete" validate_contract "$SKILL_MD"
|
||||
record "verdict schema exists" test -s "$SCHEMA"
|
||||
record "protocol reference exists" test -s "$SKILL_DIR/references/canonical-validation-protocol.md"
|
||||
record "kernel stays within 250 lines" test "$(wc -l < "$SKILL_MD")" -le 250
|
||||
|
||||
pawl_fixture="$(mktemp)"
|
||||
output_fixture="$(mktemp)"
|
||||
valid_result="$(mktemp)"
|
||||
invalid_result="$(mktemp)"
|
||||
invalid_date_result="$(mktemp)"
|
||||
self_judge_result="$(mktemp)"
|
||||
blocking_finding_result="$(mktemp)"
|
||||
significant_finding_result="$(mktemp)"
|
||||
unchecked_result="$(mktemp)"
|
||||
valid_report="$(mktemp)"
|
||||
invalid_report="$(mktemp)"
|
||||
wrong_order_report="$(mktemp)"
|
||||
unsupported_report="$(mktemp)"
|
||||
trap 'rm -f "$pawl_fixture" "$output_fixture" "$valid_result" "$invalid_result" "$invalid_date_result" "$self_judge_result" "$blocking_finding_result" "$significant_finding_result" "$unchecked_result" "$valid_report" "$invalid_report" "$wrong_order_report" "$unsupported_report"' EXIT
|
||||
|
||||
sed 's/HELPER-UNSTUCK -> AUTO-REDO/HELPER-UNSTUCK -> MANUAL/' "$SKILL_MD" >"$pawl_fixture"
|
||||
awk '!/\*\*Validator command:\*\*/' "$SKILL_MD" >"$output_fixture"
|
||||
if validate_contract "$pawl_fixture"; then echo "FAIL: missing helper transition rejected"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: missing helper transition rejected"; PASS=$((PASS + 1)); fi
|
||||
if validate_contract "$output_fixture"; then echo "FAIL: incomplete output handoff rejected"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: incomplete output handoff rejected"; PASS=$((PASS + 1)); fi
|
||||
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAV","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$valid_result"
|
||||
printf '%s\n' '{"verdict_id":"short","bead_id":"bad","verdict":"MAYBE","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"not-a-date","validator_session":"author","schema_version":1}' >"$invalid_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAV","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"not-a-date","validator_session":"judge-1","schema_version":1}' >"$invalid_date_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAX","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"author","schema_version":1}' >"$self_judge_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAY","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[{"severity":"critical","description":"release gate is bypassed","location":"scripts/release.sh:10"}],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$blocking_finding_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FB0","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[{"severity":"significant","description":"required regression is missing","location":"tests/release.bats"}],"not_checked":[],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$significant_finding_result"
|
||||
printf '%s\n' '{"verdict_id":"01ARZ3NDEKTSV4RRFFQ69G5FAZ","bead_id":"01ARZ3NDEKTSV4RRFFQ69G5FAW","verdict":"PASS","confidence":"HIGH","briefing_learnings":[],"findings":[],"not_checked":["release gate"],"validated_at":"2026-07-12T13:00:00Z","validator_session":"judge-1","schema_version":1}' >"$unchecked_result"
|
||||
record "result validator accepts complete verdict.v1" validate_result "$valid_result" "author"
|
||||
if validate_result "$invalid_result" "author"; then echo "FAIL: result validator rejects malformed verdict"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: result validator rejects malformed verdict"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$invalid_date_result" "author"; then echo "FAIL: result validator rejects invalid RFC3339 timestamp"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: result validator rejects invalid RFC3339 timestamp"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$self_judge_result" "author"; then echo "FAIL: PASS result rejects author as validator"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects author as validator"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$blocking_finding_result" "author"; then echo "FAIL: PASS result rejects critical or significant findings"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects critical or significant findings"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$significant_finding_result" "author"; then echo "FAIL: PASS result rejects significant findings"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects significant findings"; PASS=$((PASS + 1)); fi
|
||||
if validate_result "$unchecked_result" "author"; then echo "FAIL: PASS result rejects nonempty not_checked"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: PASS result rejects nonempty not_checked"; PASS=$((PASS + 1)); fi
|
||||
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: PASS' 'COMMANDS RUN:' 'judge=judge-1 command=go test ./...' 'REASONS:' '- command exited zero' >"$valid_report"
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: FAIL' 'COMMANDS RUN:' 'judge=judge-1 command=go test ./...' 'REASONS:' '- contradictory headings' >"$invalid_report"
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: PASS' 'REASONS:' '- wrong order' 'COMMANDS RUN:' 'judge=judge-1 command=go test ./...' >"$wrong_order_report"
|
||||
printf '%s\n' '## Council Verdict: PASS' 'VERDICT: PASS' 'COMMANDS RUN:' 'tests looked good' 'REASONS:' '- unsupported prose' >"$unsupported_report"
|
||||
record "report validator accepts measured single verdict" validate_report "$valid_report" "judge-1" "author"
|
||||
if validate_report "$invalid_report" "judge-1" "author"; then echo "FAIL: report validator rejects contradictory verdict headings"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: report validator rejects contradictory verdict headings"; PASS=$((PASS + 1)); fi
|
||||
if validate_report "$wrong_order_report" "judge-1" "author"; then echo "FAIL: report validator rejects REASONS before COMMANDS RUN"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: report validator rejects REASONS before COMMANDS RUN"; PASS=$((PASS + 1)); fi
|
||||
if validate_report "$unsupported_report" "judge-1" "author"; then echo "FAIL: report validator rejects commands without judge source"; FAIL=$((FAIL + 1));
|
||||
else echo "PASS: report validator rejects commands without judge source"; PASS=$((PASS + 1)); fi
|
||||
|
||||
echo
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
(( FAIL == 0 ))
|
||||
Reference in New Issue
Block a user