diff --git a/CLAUDE.md b/CLAUDE.md index 99770620..089e487e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,4 +1,4 @@ - + # CLAUDE.md @@ -18,7 +18,7 @@ The cross-referencing engine is the core moat: `enforce("@typescript-eslint/no-f Authoring-time feedback comes two ways: `generate-types` emits a `.d.ts` so the TS compiler PROVES `.spec.ts` references at edit time, and `generate-schema` emits a JSON Schema so a YAML LSP autocompletes and squiggles `vigiles:` frontmatter rule names — same guarantee, no TypeScript required. Both scan all 7 catalog APIs, package.json, and project files. -Second layer — testing the harness. Beyond verifying instruction files, vigiles tests the harness itself (hooks, settings, skills) as an assembled machine, not one hook at a time: `runHarnessTest`/`runEval` take a `plugin` path that loads the real harness (hooks with `${CLAUDE_PLUGIN_ROOT}` resolved, CLAUDE.md, skills) from `.claude-plugin/plugin.json` or `.claude/settings.json` (`src/plugin-loader.ts`, the harness-agnostic loader at the composition root), so you test what ships. Three tiers, lowest cost first: `runHook` pipes a synthesized event JSON straight to a hook process (no `claude`, no model) and checks the block/allow decision — the cheap base of the pyramid, and the only tier that reaches every event incl. Edit/Write, PreCompact, Notification, SessionEnd, SubagentStop (`src/run-hook.ts`); `runHarnessTest` runs the real `claude` CLI against a scripted mock model for deterministic, key-free checks that a hook is wired into the assembled machine and fires (`src/harness-test.ts`, `src/mock-model.ts`); and `runEval` drives the real model across A/B arms × trials, aggregating mean ± se so a gap can be read for significance (`src/eval.ts`). The loader materializes hooks, CLAUDE.md, skills, subagents and commands, and flags via `loadPlugin().warnings` any surface only a real model can drive — so loading a whole plugin never silently tests an empty machine. The API is runner-agnostic (node:test, vitest, jest) via plain async functions plus helpers/matchers in `src/harness-assert.ts` and an optional LLM-as-judge in `src/judge.ts`; a zero-dep CLI fallback runs them as `vigiles test` (`*.harness.mjs`) and `vigiles eval` (`*.eval.mjs`), with canonical examples under `examples/harness/`. Unlike reference verification (bounded by undecidability), this layer has no ceiling: a test measures reality, so there is nothing to game. A harness eval is NOT a model/prompt eval: the unit under test is the harness loaded as it SHIPS (the real Claude Code system prompt + the real CLAUDE.md + real hooks/settings), which is exactly what a generic eval runner (promptfoo et al., which configure an agent from YAML) cannot reproduce — so vigiles owns this, it does not rebuild the eval stack. The discipline is to keep the costly real-model surface THIN: push every question that can be answered deterministically into runHook/runHarnessTest, and let only the two irreducibly-real-model questions (does a description FIRE, does behaviour MOVE) touch a real model. The highest-value question this layer exists to answer is BEHAVIORAL and side-effecting — does the assembled harness, run end-to-end, actually DO the task AND not do the dangerous thing? — which a completion-grader (promptfoo et al.) structurally cannot reach. `notTool`/`interceptTools` ship today for the safety half (assert/deny the irreversible externals — push, paid API — you must never actually execute); the ephemeral run environment + disposable-dependency provisioning (compose/testcontainers) that make a full side-effecting run safe to REPEAT are the committed next step. This apex tier is the most valuable AND the most expensive, so it stays THIN by design — the deterministic tiers are what make it affordable. See `docs/safety.md`. AFFORDABILITY is the positioning of this layer: the deterministic tiers are free (no model, no key, every commit) and the real-model tier runs on your Claude PRO/MAX SUBSCRIPTION — vigiles drives the real `claude` CLI, so an eval authenticates like your own CLI (no metered API billing), which is why a team can actually afford to run harness evals at all; competitors (promptfoo/DeepEval/…) hit the API SDK and bill per token on every run. Be precise about the wedge: the deterministic mock-model tier itself is TABLE STAKES — the code-defined SDKs (Pydantic AI's TestModel/FunctionModel, Vercel AI SDK's MockLanguageModelV3, LangGraph's FakeListChatModel, LlamaIndex's MockLLM) all ship a first-party fake model, so "mock the model, assert deterministically" is not what differentiates vigiles. The wedge is the three things no SDK mock does: (1) testing the harness loaded as it SHIPS — the real CLI agent with its real system prompt + CLAUDE.md + hooks, not an agent re-assembled from SDK config; (2) deterministic tool-contract enforcement of the ASSEMBLED agent (the gap the Claude Agent SDK still carries OPEN as bug #172 — declared `tools`/`disallowedTools` are not propagated to a subagent, so vigiles's PreToolUse rail in src/adapters/claude-code/agent-runtime.ts is the fix, not a nicety); and (3) a real-model tier affordable on the sub. See research/sdk-harness-testing.md. Evals run where the subscription already is — a Claude Code session or locally — NOT a standalone GitHub Actions job needing a token (CI runs only the free deterministic tiers). measureTriggerRate measures on the realistic SELECTOR (Sonnet default, a minModel floor) since a weaker model under-selects; the model lives in the spec (model/minModel), not an env override (trials, a run knob, can be an env). What vigiles tests sorts onto three rungs: R1 (cheap/deterministic — fire/trigger/contract/safety, nothing executes) and R2 (record-replay — the skill's deterministic logic consumes a tool/MCP/API result RECORDED ONCE from a real tool and REPLAYED by shadowing the binary on PATH, never model-synthesized) cover ~90%+ of real plugin surface with NO Docker, on the subscription; R3 (the real disposable service whose semantics IS what's under test — a browser, a relational DB, redis) is a THIN apex vigiles COMPOSES with a container for rather than reinventing the sandbox (a survey of popular community collections and an audit of a ~90-artifact production skill set converge on R1≈48–90% / R2≈10–43% / R3≈0–9%). Safety: R1 nothing runs, R2 fake outputs touch no real system, R3 real side effects only inside an isolated disposable container; viability: R1+R2 need no Docker and run on the sub; performance: R1/R2 ms-fast, R3 Docker cold-start stays thin. No tool does containerless reproducible e2e (every e2e benchmark/lab runs in a container) — so vigiles owns R1+R2 + sub-affordability + a clean container hand-off, not e2e-without-a-container. A SECOND, orthogonal axis decides the COST — the correctness oracle: a DETERMINISTIC check (hook decision, tool-contract, a structural fact) is free in CI, while a MODEL-GATED question (does a description FIRE, is the guidance's output GOOD, does prose guidance MOVE behaviour vs off — measureTriggerRate for firing AND, for behaviour, TWO oracles not one: the ABSOLUTE "is this exact skill's output good" via a single-arm measure()+judged()+assertRates (the right default when there is no on/off baseline — what promptfoo/DeepEval lead with), and the RELATIVE "does it MOVE behaviour vs off" via a runEval A/B + assertSignificant (regression / noise-floor)) runs on the sub; we tag the latter `-MG`, so a prose/guidance skill is R1-MG (nothing executes, but only a model judges its worth — fully testable on your sub via trigger-rate + judged behaviour, NOT uncovered and NOT free). State coverage as the TWO buckets vigiles owns — (A) free & deterministic + (B) model-gated on your sub — vs (C) needs-a-container (composed), and grade a plugin with TWO numbers ("% testable at all (A+B, free+sub)" vs "% needs-a-container"), never letting "model-gated" read as "uncovered" (testing a prose skill's behaviour needs a real model for EVERYONE — promptfoo, the SDKs, all of it; vigiles just does it on the sub). The sub-affordability is ToS-CLEAN: vigiles drives YOUR OWN `claude` CLI to test YOUR OWN harness on YOUR OWN subscription (the Claude Agent SDK ToS restricts PRODUCTIZING claude.ai login/limits in a third-party offering, not running your own tests on your own sub — exactly vigiles's posture). See `docs/eval-architecture.md` (positioning + pros/cons), `research/eval-coverage-and-isolation.md` (what we test, how, what we delegate), `research/sdk-harness-testing.md` (the 2026-06-17 multi-SDK probe + ToS detail + mock-ergonomics borrow-list), `docs/harness-testing.md`, `research/harness-testing.md`, the eval-tier decision in `research/eval-api-landscape.md`, and `research/isolated-vs-whole-harness-eval.md` + `research/cache-invalidation.md`. +Second layer — testing the harness. Beyond verifying instruction files, vigiles tests the harness itself (hooks, settings, skills) as an assembled machine, not one hook at a time: `runHarnessTest`/`runEval` take a `plugin` path that loads the real harness (hooks with `${CLAUDE_PLUGIN_ROOT}` resolved, CLAUDE.md, skills) from `.claude-plugin/plugin.json` or `.claude/settings.json` (`src/plugin-loader.ts`, the harness-agnostic loader at the composition root), so you test what ships. Three tiers, lowest cost first: `runHook` pipes a synthesized event JSON straight to a hook process (no `claude`, no model) and checks the block/allow decision — the cheap base of the pyramid, and the only tier that reaches every event incl. Edit/Write, PreCompact, Notification, SessionEnd, SubagentStop (`src/run-hook.ts`); `runHarnessTest` runs the real `claude` CLI against a scripted mock model for deterministic, key-free checks that a hook is wired into the assembled machine and fires (`src/harness-test.ts`, `src/mock-model.ts`); and `runEval` drives the real model across A/B arms × trials, aggregating mean ± se so a gap can be read for significance (`src/eval.ts`). The loader materializes hooks, CLAUDE.md, skills, subagents and commands, and flags via `loadPlugin().warnings` any surface only a real model can drive — so loading a whole plugin never silently tests an empty machine. The API is runner-agnostic (node:test, vitest, jest) via plain async functions plus helpers/matchers in `src/harness-assert.ts` and an optional LLM-as-judge in `src/judge.ts`; a zero-dep CLI fallback runs them as `vigiles test` (`*.harness.mjs`) and `vigiles eval` (`*.eval.mjs`), with canonical examples under `examples/harness/`. Unlike reference verification (bounded by undecidability), this layer has no ceiling: a test measures reality, so there is nothing to game. A harness eval is NOT a model/prompt eval: the unit under test is the harness loaded as it SHIPS (the real Claude Code system prompt + the real CLAUDE.md + real hooks/settings), which is exactly what a generic eval runner (promptfoo et al., which configure an agent from YAML) cannot reproduce — so vigiles owns this, it does not rebuild the eval stack. The discipline is to keep the costly real-model surface THIN: push every question that can be answered deterministically into runHook/runHarnessTest, and let only the two irreducibly-real-model questions (does a description FIRE, does behaviour MOVE) touch a real model. The highest-value question this layer exists to answer is BEHAVIORAL and side-effecting — does the assembled harness, run end-to-end, actually DO the task AND not do the dangerous thing? — which a completion-grader (promptfoo et al.) structurally cannot reach. `notTool`/`interceptTools` ship today for the safety half (assert/deny the irreversible externals — push, paid API — you must never actually execute); the ephemeral run environment + disposable-dependency provisioning (compose/testcontainers) that make a full side-effecting run safe to REPEAT are the committed next step. This apex tier is the most valuable AND the most expensive, so it stays THIN by design — the deterministic tiers are what make it affordable. See `docs/safety.md`. AFFORDABILITY is the positioning of this layer: the deterministic tiers are free (no model, no key, every commit) and the real-model tier runs on your Claude PRO/MAX SUBSCRIPTION — vigiles drives the real `claude` CLI, so an eval authenticates like your own CLI (no metered API billing), which is why a team can actually afford to run harness evals at all; competitors (promptfoo/DeepEval/…) hit the API SDK and bill per token on every run. Be precise about the wedge: the deterministic mock-model tier itself is TABLE STAKES — the code-defined SDKs (Pydantic AI's TestModel/FunctionModel, Vercel AI SDK's MockLanguageModelV3, LangGraph's FakeListChatModel, LlamaIndex's MockLLM) all ship a first-party fake model, so "mock the model, assert deterministically" is not what differentiates vigiles. The wedge is the three things no SDK mock does: (1) testing the harness loaded as it SHIPS — the real CLI agent with its real system prompt + CLAUDE.md + hooks, not an agent re-assembled from SDK config; (2) deterministic tool-contract enforcement of the ASSEMBLED agent (the gap the Claude Agent SDK still carries OPEN as bug #172 — declared `tools`/`disallowedTools` are not propagated to a subagent, so vigiles's PreToolUse rail in src/adapters/claude-code/agent-runtime.ts is the fix, not a nicety); and (3) a real-model tier affordable on the sub. See research/sdk-harness-testing.md. Evals run where the subscription already is — a Claude Code session or locally — NOT a standalone GitHub Actions job needing a token (CI runs only the free deterministic tiers). measureTriggerRate measures on the realistic SELECTOR (Sonnet default, a minModel floor) since a weaker model under-selects; the model lives in the spec (model/minModel), not an env override (trials, a run knob, can be an env). What vigiles tests sorts onto three rungs: R1 (cheap/deterministic — fire/trigger/contract/safety, nothing executes) and R2 (record-replay — the skill's deterministic logic consumes a tool/MCP/API result RECORDED ONCE from a real tool and REPLAYED by shadowing the binary on PATH, never model-synthesized) cover ~90%+ of real plugin surface with NO Docker, on the subscription; R3 (the real disposable service whose semantics IS what's under test — a browser, a relational DB, redis) is a THIN apex vigiles COMPOSES with a container for rather than reinventing the sandbox (a survey of popular community collections and an audit of a ~90-artifact production skill set converge on R1≈48–90% / R2≈10–43% / R3≈0–9%). Safety: R1 nothing runs, R2 fake outputs touch no real system, R3 real side effects only inside an isolated disposable container; viability: R1+R2 need no Docker and run on the sub; performance: R1/R2 ms-fast, R3 Docker cold-start stays thin. No tool does containerless reproducible e2e (every e2e benchmark/lab runs in a container) — so vigiles owns R1+R2 + sub-affordability + a clean container hand-off, not e2e-without-a-container. A SECOND, orthogonal axis decides the COST — the correctness oracle: a DETERMINISTIC check (hook decision, tool-contract, a structural fact) is free in CI, while a MODEL-GATED question (does a description FIRE, is the guidance's output GOOD, does prose guidance MOVE behaviour vs off — measureTriggerRate for firing AND, for behaviour, TWO oracles not one: the ABSOLUTE "is this exact skill's output good" via a single-arm measure()+judged()+assertRates (the right default when there is no on/off baseline — what promptfoo/DeepEval lead with), and the RELATIVE "does it MOVE behaviour vs off" via a runEval A/B + assertSignificant (regression / noise-floor)) runs on the sub; we tag the latter `-MG`, so a prose/guidance skill is R1-MG (nothing executes, but only a model judges its worth — fully testable on your sub via trigger-rate + judged behaviour, NOT uncovered and NOT free). State coverage as the TWO buckets vigiles owns — (A) free & deterministic + (B) model-gated on your sub — vs (C) needs-a-container (composed), and grade a plugin with TWO numbers ("% testable at all (A+B, free+sub)" vs "% needs-a-container"), never letting "model-gated" read as "uncovered" (testing a prose skill's behaviour needs a real model for EVERYONE — promptfoo, the SDKs, all of it; vigiles just does it on the sub). The sub-affordability is ToS-CLEAN: vigiles drives YOUR OWN `claude` CLI to test YOUR OWN harness on YOUR OWN subscription (the Claude Agent SDK ToS restricts PRODUCTIZING claude.ai login/limits in a third-party offering, not running your own tests on your own sub — exactly vigiles's posture). See `research/eval-architecture.md` (positioning + pros/cons), `research/eval-coverage-and-isolation.md` (what we test, how, what we delegate), `research/sdk-harness-testing.md` (the 2026-06-17 multi-SDK probe + ToS detail + mock-ergonomics borrow-list), `docs/harness-testing.md`, `research/harness-testing.md`, the eval-tier decision in `research/eval-api-landscape.md`, and `research/isolated-vs-whole-harness-eval.md` + `research/cache-invalidation.md`. Third layer — COMPILED HOOKS (the GATE instrument; `vigiles/hook`, src/core/hook-program.ts + src/hook.ts). The reliability frame is four instruments that SHRINK the harness state-space — construct (typed spec), VERIFY (lint), GATE (hooks), test (evals) — and the gate is the deterministic stop before something irreversible. A hook today is opaque shell, and the parts the author hand-writes (exit code, JSON field, a grep matcher) are exactly where the #1 verified pain lives: FALSE CONFIDENCE — a guard that LOOKS like it blocks and silently doesn't (exit 1≠2, wrong field; research/hook-pain-points.md). Invert it: author a hook as a PURE typed function `(event) => Decision` against a CLOSED vocabulary; `vigiles compile` emits the protocol, `vigiles hook-runtime run-program` runs it. This makes WHOLE CLASSES OF BUGS UNREPRESENTABLE — you never write the exit-code/field (false confidence), the matcher is AST-backed (`command.runs("git push",{force})` catches `cd x && git push -f` the native glob/#30519 misses; + touches()/pipesToShell() for secret-read/curl|sh), capability = API surface (an import outside `vigiles/hook` does NOT compile), the artifact is STAMPED (a hand-edit breaks the SHA-256 → the runtime refuses it, fail-closed), and a category mistake (block on a no-decision event) is a tsc type error (a role FAMILY: tool-gate (defineHook/defineFileGate)→Decision, plus prompt-gate (definePromptGate — sees the prompt TEXT e.prompt, deny blocks the prompt: a security filter) and stop-gate (defineStopGate — deny keeps the agent GOING, gate-until-tests-pass, honour e.stopHookActive loop guard) also →Decision on the SAME shared exit-2 runtime (so they work on CC AND Codex), inject→Injection, react→Reaction (now sees the tool RESPONSE e.response.isError()/contains()), each with its own return type). Every gate takes mode:'enforce'|'observe' — observe is the SHADOW/rollout mode (compute the same Decision, RECORD what it would block to .vigiles/hook-observations.jsonl, exit 0, never block; harness-NEUTRAL by construction — exit 0 + a local record; gateAction is the pure decision→action mapping the runtime + tests share). PROVEN by an OSS dogfood (src/hook-dogfood.test.ts): a widely-copied hand-written safety hook blocks 2/7 of the disaster battery; the compiled rewrite (examples/harness/safe-bash-guard.mjs) blocks 7/7 — measured with the verify feature's own DISASTER_CATALOG (src/guardrail-check.ts, "prove your guardrail blocks", on `vigiles/unit`). HONEST SCOPE, kept in every doc: compile/verify fix AUTHORING + LOGIC, not DELIVERY — CC's subagent-bypass (#34692, closed not-planned) means a PreToolUse hook does not fire for a subagent's tool calls, so a gate is a STRONG DEFAULT, never an unbypassable wall (VERIFY, a claim about logic, survives the bug; GATE, live enforcement, is capped — never claim "unbypassable"). See `docs/compiled-hooks.md` (the public guide) and `research/hook-pain-points.md` (the verified corpus + design record). @@ -260,7 +260,7 @@ Two boundary rules are enforced by `eslint-plugin-boundaries` (rule `boundaries/ - `src/check.ts` — The declarative check vocabulary (exposed on the `vigiles/testing` + `vigiles/unit` surfaces; the standalone `vigiles/check` subpath was removed in the public-surface trim) (testing-API revamp, research/testing-api-design.md). A check is DATA not a throwing assert: { kind, eval(target) → {pass, score, message}, toJSON() }, typed over Trace (tool/skill/output/hookFired/received/turns/wrote/didNotWrite/mcp/subagent/judged — wrote/didNotWrite are the symmetric side-effect-boundary write/no-write pair) vs HookRunResult (blocked/allowed) vs usage (cost/latency/tokens). judged() is the model-graded check (LLM rubric via an injectable judge fn — fits the sync eval since judge() blocks via spawnSync); mcp() matches the `mcp__server__tool` tool name. One vocabulary evaluated two ways — strict (assertChecks, throws collecting all failures) + scored (measure, rate ± se across trials) — so a check reads pass/fail on one run AND a rate; serializable so JUnit/baseline/promptfoo-bridge fall out. Harness-AGNOSTIC: every check reads generic Trace fields, never a CC shape, so it evaluates over a Codex-sparse Trace too. Folded into vigiles/testing + vigiles/unit (its hookFired wins over the legacy boolean via an explicit re-export). Pure + model-free (judged calls a model) - `src/check.test.ts` — Check-vocabulary test suite (vitest): each check's pass/fail + 0–1 score + ACTIONABLE failure message (the message is the product, tested as much as the verdict) + toJSON round-trip; evalChecks order; over fake Trace + HookRunResult, no model - `src/arg-match.ts` — Shared ArgMatcher over a tool call's input (dot-path keys; RegExp = pattern, primitive = exact; AND across keys) + matchesArgs/getPath/stringifyValue/describeArgs/serializeArgs. One matching semantics for both the check vocabulary (toolWith/notTool in src/check.ts) and the tool-interception seam (src/tool-intercept.ts), so 'did the agent call this tool with these args?' means the same whether you ASSERT on a captured call or INTERCEPT one. Pure + serializable -- `src/tool-intercept.ts` — Tool interception — the eval-tier half of the tool-call spy. A ToolIntercept (tool + optional `when` ArgMatcher + denyReason block message) is intercepted by an auto-wired PreToolUse hook (exit-2 deny) so a real-model run that DECIDES to hit a paid API / git push / spawn a paid subagent is safe + side-effect-free — the call is intercepted (prevented), NOT executed — while the tool_use (with args) still lands in the Trace for toolWith/notTool. INTERCEPT-AND-PREVENT, not a faithful mock: CC surfaces the deny as a BLOCK not a success, so it's for asserting the ATTEMPT (safety/approval-gate), not stubbing a tool to continue a flow. decideIntercept (pure, first-match-wins) + buildInterceptSettings (PreToolUse fragment over the union of intercepted tool names) + serializeIntercepts/parseIntercepts (VIGILES_INTERCEPT_TOOLS env round-trip, RegExp-safe). The `vigiles hook-runtime intercept-tool` CLI subcommand runs the decision. Wired onto the eval spec via EvalArm.interceptTools/MeasureSpec.interceptTools (auto-merged into arm settings, cache keyed on env). See docs/eval-architecture.md +- `src/tool-intercept.ts` — Tool interception — the eval-tier half of the tool-call spy. A ToolIntercept (tool + optional `when` ArgMatcher + denyReason block message) is intercepted by an auto-wired PreToolUse hook (exit-2 deny) so a real-model run that DECIDES to hit a paid API / git push / spawn a paid subagent is safe + side-effect-free — the call is intercepted (prevented), NOT executed — while the tool_use (with args) still lands in the Trace for toolWith/notTool. INTERCEPT-AND-PREVENT, not a faithful mock: CC surfaces the deny as a BLOCK not a success, so it's for asserting the ATTEMPT (safety/approval-gate), not stubbing a tool to continue a flow. decideIntercept (pure, first-match-wins) + buildInterceptSettings (PreToolUse fragment over the union of intercepted tool names) + serializeIntercepts/parseIntercepts (VIGILES_INTERCEPT_TOOLS env round-trip, RegExp-safe). The `vigiles hook-runtime intercept-tool` CLI subcommand runs the decision. Wired onto the eval spec via EvalArm.interceptTools/MeasureSpec.interceptTools (auto-merged into arm settings, cache keyed on env). See research/eval-architecture.md - `src/tool-intercept.test.ts` — Tool-intercept test suite (vitest): decideIntercept (unconditional/when-scoped/first-match-wins/default reason), interceptHookDecision (PreToolUse event parse + malformed tolerance), buildInterceptSettings (matcher = escaped union of tool names), serializeIntercepts/parseIntercepts round-trip incl. RegExp survival + junk tolerance. Pure, model-free - `src/judge.ts` — Thin LLM-as-judge for the eval tier: judge() grades an output against a rubric with a model (synchronous, for use inside measure); parseJudgeOutput is the pure, testable verdict parser - `src/judge.test.ts` — Judge verdict-parsing test suite (node:test): result-field unwrap, prose-wrapped JSON, threshold, clamping, unparseable fallback @@ -357,8 +357,7 @@ Two boundary rules are enforced by `eslint-plugin-boundaries` (rule `boundaries/ - `docs/adapter-api.md` — vigiles/adapter API reference: every export of the authoring kit — the five port interfaces field-by-field (with Claude Code AND Codex example values), HarnessAdapter + the detect specificity contract, the conformance functions (checkAdapterConformance/assertAdapterConformance/assertAdapterLoadsHooks — signatures + what each checks/throws), the registry API (ADAPTERS/detectAdapter/detectAdapterResult/resolveAdapter/getAdapter), the programmatic apply path (compileAgent/loadPlugin with your ports), the third-party-adapter status (programmatic supported now; CLI auto-detect of external packages is planned), and the stability/semver note - `docs/authoring-an-adapter.md` — Third-party adapter authoring guide: the documented small lib (vigiles/adapter) for teaching vigiles a new harness — the five ports to implement, a worked myHarnessAdapter skeleton, validating with assertAdapterConformance, wiring it (library by import, CLI by registry), and what's still behaviour-not-descriptor (renderers/decision-decode/mock HTTP server). Linked from the root README (custom adapters welcome) - `docs/testing-matrix.md` — Testing matrix: every harness-testing use case mapped to its test tier (unit / cross-runner / type / integration-CI) and file, plus why the CLI examples are .mjs and the API is TypeScript -- `docs/agent-workflows.md` — Agent-specific workflows (Claude Code, Codex, multi-agent, Cursor) -- `docs/agent-setup.md` — Non-interactive agent setup guide (hooks via settings.json) +- `docs/agent-setup.md` — Agent setup & workflows — one guide: what `init` does + auto-detection, per-agent recipes (Claude Code / Codex / multi-agent / Cursor), non-interactive setup + fallback hooks, the recommended agent prompt, and CI (absorbs the former agent-workflows.md) - `docs/spec-format.md` — Spec format reference (target, sections, rules) - `docs/railway-subagents.md` — Public guide to railway-oriented subagents: the typed Result outcome contract (result() on an agent()), what it compiles to (vigiles:ok/err blocks), composing flat workers (railway()/delegate()/recover), and asserting the outcome deterministically (assertAgentOk/Err/Result — no model judge). States the scope decision: railway is a SUBAGENT primitive (context boundary = parse-point), NOT skills; context:fork is the bridge. Links research/railway-subagents.md (design) + research/spec-syntax-and-railway-scope.md (the scope decision) - `docs/linter-support.md` — Linter support details (7 catalogs + generate-types/generate-schema) @@ -367,8 +366,7 @@ Two boundary rules are enforced by `eslint-plugin-boundaries` (rule `boundaries/ - `docs/rules/require-skill-spec.md` — Rule doc: require-skill-spec — the consistent require--spec parallel (default OFF). Skills are legitimately hand-written, so requiring a .spec.ts per SKILL.md isn't the default; the coverage that matters is untested-skill (every skill ships a test/eval). Set it explicitly to spec-manage every skill - `docs/rules/integrity.md` — Rule doc: integrity check (SHA-256 hash verification for compiled markdown) - `docs/rules/coverage.md` — Rule doc: spec coverage thresholds (scripts, linter rules) -- `docs/inline-mode.md` — Inline mode: `` comments for gradual adoption without a .spec.ts -- `docs/markdown-mode.md` — Markdown mode: inline `` comments (Level 0) and `vigiles:` YAML frontmatter (Level 1) for adoption without a .spec.ts +- `docs/markdown-mode.md` — Markdown mode: the single no-spec on-ramp doc — inline `` comments (the live zero-TS floor; the former inline-mode.md reference is folded in here). Frontmatter mode is parked/disabled (kept in a comment). - `skills/linter-docs/eslint.md` — ESLint reference: plugin table, AST selectors, type-aware rules, auto-fix, edge cases - `skills/linter-docs/rubocop.md` — RuboCop reference: gem table, node pattern DSL, auto-correct, custom cops - `skills/linter-docs/pylint.md` — Pylint reference: plugin table, astroid AST, type inference, custom checkers @@ -460,7 +458,7 @@ Two boundary rules are enforced by `eslint-plugin-boundaries` (rule `boundaries/ ### Smooth Adoption -**Guidance only** — `npx vigiles init` must work on first run with zero config and set up BOTH layers by default: a typed spec + types (the Lint layer), a vigiles.harness.mjs starter (the Test layer), a `zernie/vigiles@v1` CI workflow (created when none exists; a stale old-API one is flagged, not skipped), vigiles added to devDependencies, and the Claude Code plugin installed via the MARKETPLACE (`/plugin marketplace add zernie/vigiles` + `/plugin install vigiles@vigiles`, into ~/.claude/plugins/ — never vendored into the repo). Onboarding is interactive at a TTY (asks which layers / CI / plugin) and NON-INTERACTIVE for agents, CI, or piped input (or with `--yes`) — so 'set up vigiles' from a Claude Code / Codex prompt Just Works without hanging on a prompt; `--lint`/`--test` scope the layers (`--harness=` overrides detection). After install the agent edits specs automatically — no workflow change required. The DEFAULT already GATES broken surfaces (the FP-safe `structural` rule group at `error` — a typo'd tool / dead hook / broken MCP / skill collision fails CI, while a clean repo stays green); the WORKFLOW tier (a spec per file + a test per surface) is OFFERED interactively (opt-out, default-yes) or via `--strict`, and `--report-only` downgrades everything to warnings for cautious migration. 'Permissive' means vigiles doesn't FORCE specs/TS on you and doesn't CRY WOLF — NOT that it ignores genuine breakage (see `install-enforcement-model`). Hesitant adopters can use inline mode (`` comments) without a .spec.ts — see `docs/inline-mode.md`. See `research/adoption-strategy.md` and `docs/agent-setup.md`. +**Guidance only** — `npx vigiles init` must work on first run with zero config and set up BOTH layers by default: a typed spec + types (the Lint layer), a vigiles.harness.mjs starter (the Test layer), a `zernie/vigiles@v1` CI workflow (created when none exists; a stale old-API one is flagged, not skipped), vigiles added to devDependencies, and the Claude Code plugin installed via the MARKETPLACE (`/plugin marketplace add zernie/vigiles` + `/plugin install vigiles@vigiles`, into ~/.claude/plugins/ — never vendored into the repo). Onboarding is interactive at a TTY (asks which layers / CI / plugin) and NON-INTERACTIVE for agents, CI, or piped input (or with `--yes`) — so 'set up vigiles' from a Claude Code / Codex prompt Just Works without hanging on a prompt; `--lint`/`--test` scope the layers (`--harness=` overrides detection). After install the agent edits specs automatically — no workflow change required. The DEFAULT already GATES broken surfaces (the FP-safe `structural` rule group at `error` — a typo'd tool / dead hook / broken MCP / skill collision fails CI, while a clean repo stays green); the WORKFLOW tier (a spec per file + a test per surface) is OFFERED interactively (opt-out, default-yes) or via `--strict`, and `--report-only` downgrades everything to warnings for cautious migration. 'Permissive' means vigiles doesn't FORCE specs/TS on you and doesn't CRY WOLF — NOT that it ignores genuine breakage (see `install-enforcement-model`). Hesitant adopters can use inline mode (`` comments) without a .spec.ts — see `docs/markdown-mode.md`. See `research/adoption-strategy.md` and `docs/agent-setup.md`. ### Great Agent Flow diff --git a/CLAUDE.md.spec.ts b/CLAUDE.md.spec.ts index 0ea4964f..80dcd505 100644 --- a/CLAUDE.md.spec.ts +++ b/CLAUDE.md.spec.ts @@ -22,7 +22,7 @@ The cross-referencing engine is the core moat: \`enforce("@typescript-eslint/no- Authoring-time feedback comes two ways: \`generate-types\` emits a \`.d.ts\` so the TS compiler PROVES \`.spec.ts\` references at edit time, and \`generate-schema\` emits a JSON Schema so a YAML LSP autocompletes and squiggles \`vigiles:\` frontmatter rule names — same guarantee, no TypeScript required. Both scan all 7 catalog APIs, package.json, and project files. -Second layer — testing the harness. Beyond verifying instruction files, vigiles tests the harness itself (hooks, settings, skills) as an assembled machine, not one hook at a time: \`runHarnessTest\`/\`runEval\` take a \`plugin\` path that loads the real harness (hooks with \`\${CLAUDE_PLUGIN_ROOT}\` resolved, CLAUDE.md, skills) from \`.claude-plugin/plugin.json\` or \`.claude/settings.json\` (\`src/plugin-loader.ts\`, the harness-agnostic loader at the composition root), so you test what ships. Three tiers, lowest cost first: \`runHook\` pipes a synthesized event JSON straight to a hook process (no \`claude\`, no model) and checks the block/allow decision — the cheap base of the pyramid, and the only tier that reaches every event incl. Edit/Write, PreCompact, Notification, SessionEnd, SubagentStop (\`src/run-hook.ts\`); \`runHarnessTest\` runs the real \`claude\` CLI against a scripted mock model for deterministic, key-free checks that a hook is wired into the assembled machine and fires (\`src/harness-test.ts\`, \`src/mock-model.ts\`); and \`runEval\` drives the real model across A/B arms × trials, aggregating mean ± se so a gap can be read for significance (\`src/eval.ts\`). The loader materializes hooks, CLAUDE.md, skills, subagents and commands, and flags via \`loadPlugin().warnings\` any surface only a real model can drive — so loading a whole plugin never silently tests an empty machine. The API is runner-agnostic (node:test, vitest, jest) via plain async functions plus helpers/matchers in \`src/harness-assert.ts\` and an optional LLM-as-judge in \`src/judge.ts\`; a zero-dep CLI fallback runs them as \`vigiles test\` (\`*.harness.mjs\`) and \`vigiles eval\` (\`*.eval.mjs\`), with canonical examples under \`examples/harness/\`. Unlike reference verification (bounded by undecidability), this layer has no ceiling: a test measures reality, so there is nothing to game. A harness eval is NOT a model/prompt eval: the unit under test is the harness loaded as it SHIPS (the real Claude Code system prompt + the real CLAUDE.md + real hooks/settings), which is exactly what a generic eval runner (promptfoo et al., which configure an agent from YAML) cannot reproduce — so vigiles owns this, it does not rebuild the eval stack. The discipline is to keep the costly real-model surface THIN: push every question that can be answered deterministically into runHook/runHarnessTest, and let only the two irreducibly-real-model questions (does a description FIRE, does behaviour MOVE) touch a real model. The highest-value question this layer exists to answer is BEHAVIORAL and side-effecting — does the assembled harness, run end-to-end, actually DO the task AND not do the dangerous thing? — which a completion-grader (promptfoo et al.) structurally cannot reach. \`notTool\`/\`interceptTools\` ship today for the safety half (assert/deny the irreversible externals — push, paid API — you must never actually execute); the ephemeral run environment + disposable-dependency provisioning (compose/testcontainers) that make a full side-effecting run safe to REPEAT are the committed next step. This apex tier is the most valuable AND the most expensive, so it stays THIN by design — the deterministic tiers are what make it affordable. See \`docs/safety.md\`. AFFORDABILITY is the positioning of this layer: the deterministic tiers are free (no model, no key, every commit) and the real-model tier runs on your Claude PRO/MAX SUBSCRIPTION — vigiles drives the real \`claude\` CLI, so an eval authenticates like your own CLI (no metered API billing), which is why a team can actually afford to run harness evals at all; competitors (promptfoo/DeepEval/…) hit the API SDK and bill per token on every run. Be precise about the wedge: the deterministic mock-model tier itself is TABLE STAKES — the code-defined SDKs (Pydantic AI's TestModel/FunctionModel, Vercel AI SDK's MockLanguageModelV3, LangGraph's FakeListChatModel, LlamaIndex's MockLLM) all ship a first-party fake model, so "mock the model, assert deterministically" is not what differentiates vigiles. The wedge is the three things no SDK mock does: (1) testing the harness loaded as it SHIPS — the real CLI agent with its real system prompt + CLAUDE.md + hooks, not an agent re-assembled from SDK config; (2) deterministic tool-contract enforcement of the ASSEMBLED agent (the gap the Claude Agent SDK still carries OPEN as bug #172 — declared \`tools\`/\`disallowedTools\` are not propagated to a subagent, so vigiles's PreToolUse rail in src/adapters/claude-code/agent-runtime.ts is the fix, not a nicety); and (3) a real-model tier affordable on the sub. See research/sdk-harness-testing.md. Evals run where the subscription already is — a Claude Code session or locally — NOT a standalone GitHub Actions job needing a token (CI runs only the free deterministic tiers). measureTriggerRate measures on the realistic SELECTOR (Sonnet default, a minModel floor) since a weaker model under-selects; the model lives in the spec (model/minModel), not an env override (trials, a run knob, can be an env). What vigiles tests sorts onto three rungs: R1 (cheap/deterministic — fire/trigger/contract/safety, nothing executes) and R2 (record-replay — the skill's deterministic logic consumes a tool/MCP/API result RECORDED ONCE from a real tool and REPLAYED by shadowing the binary on PATH, never model-synthesized) cover ~90%+ of real plugin surface with NO Docker, on the subscription; R3 (the real disposable service whose semantics IS what's under test — a browser, a relational DB, redis) is a THIN apex vigiles COMPOSES with a container for rather than reinventing the sandbox (a survey of popular community collections and an audit of a ~90-artifact production skill set converge on R1≈48–90% / R2≈10–43% / R3≈0–9%). Safety: R1 nothing runs, R2 fake outputs touch no real system, R3 real side effects only inside an isolated disposable container; viability: R1+R2 need no Docker and run on the sub; performance: R1/R2 ms-fast, R3 Docker cold-start stays thin. No tool does containerless reproducible e2e (every e2e benchmark/lab runs in a container) — so vigiles owns R1+R2 + sub-affordability + a clean container hand-off, not e2e-without-a-container. A SECOND, orthogonal axis decides the COST — the correctness oracle: a DETERMINISTIC check (hook decision, tool-contract, a structural fact) is free in CI, while a MODEL-GATED question (does a description FIRE, is the guidance's output GOOD, does prose guidance MOVE behaviour vs off — measureTriggerRate for firing AND, for behaviour, TWO oracles not one: the ABSOLUTE "is this exact skill's output good" via a single-arm measure()+judged()+assertRates (the right default when there is no on/off baseline — what promptfoo/DeepEval lead with), and the RELATIVE "does it MOVE behaviour vs off" via a runEval A/B + assertSignificant (regression / noise-floor)) runs on the sub; we tag the latter \`-MG\`, so a prose/guidance skill is R1-MG (nothing executes, but only a model judges its worth — fully testable on your sub via trigger-rate + judged behaviour, NOT uncovered and NOT free). State coverage as the TWO buckets vigiles owns — (A) free & deterministic + (B) model-gated on your sub — vs (C) needs-a-container (composed), and grade a plugin with TWO numbers ("% testable at all (A+B, free+sub)" vs "% needs-a-container"), never letting "model-gated" read as "uncovered" (testing a prose skill's behaviour needs a real model for EVERYONE — promptfoo, the SDKs, all of it; vigiles just does it on the sub). The sub-affordability is ToS-CLEAN: vigiles drives YOUR OWN \`claude\` CLI to test YOUR OWN harness on YOUR OWN subscription (the Claude Agent SDK ToS restricts PRODUCTIZING claude.ai login/limits in a third-party offering, not running your own tests on your own sub — exactly vigiles's posture). See \`docs/eval-architecture.md\` (positioning + pros/cons), \`research/eval-coverage-and-isolation.md\` (what we test, how, what we delegate), \`research/sdk-harness-testing.md\` (the 2026-06-17 multi-SDK probe + ToS detail + mock-ergonomics borrow-list), \`docs/harness-testing.md\`, \`research/harness-testing.md\`, the eval-tier decision in \`research/eval-api-landscape.md\`, and \`research/isolated-vs-whole-harness-eval.md\` + \`research/cache-invalidation.md\`. +Second layer — testing the harness. Beyond verifying instruction files, vigiles tests the harness itself (hooks, settings, skills) as an assembled machine, not one hook at a time: \`runHarnessTest\`/\`runEval\` take a \`plugin\` path that loads the real harness (hooks with \`\${CLAUDE_PLUGIN_ROOT}\` resolved, CLAUDE.md, skills) from \`.claude-plugin/plugin.json\` or \`.claude/settings.json\` (\`src/plugin-loader.ts\`, the harness-agnostic loader at the composition root), so you test what ships. Three tiers, lowest cost first: \`runHook\` pipes a synthesized event JSON straight to a hook process (no \`claude\`, no model) and checks the block/allow decision — the cheap base of the pyramid, and the only tier that reaches every event incl. Edit/Write, PreCompact, Notification, SessionEnd, SubagentStop (\`src/run-hook.ts\`); \`runHarnessTest\` runs the real \`claude\` CLI against a scripted mock model for deterministic, key-free checks that a hook is wired into the assembled machine and fires (\`src/harness-test.ts\`, \`src/mock-model.ts\`); and \`runEval\` drives the real model across A/B arms × trials, aggregating mean ± se so a gap can be read for significance (\`src/eval.ts\`). The loader materializes hooks, CLAUDE.md, skills, subagents and commands, and flags via \`loadPlugin().warnings\` any surface only a real model can drive — so loading a whole plugin never silently tests an empty machine. The API is runner-agnostic (node:test, vitest, jest) via plain async functions plus helpers/matchers in \`src/harness-assert.ts\` and an optional LLM-as-judge in \`src/judge.ts\`; a zero-dep CLI fallback runs them as \`vigiles test\` (\`*.harness.mjs\`) and \`vigiles eval\` (\`*.eval.mjs\`), with canonical examples under \`examples/harness/\`. Unlike reference verification (bounded by undecidability), this layer has no ceiling: a test measures reality, so there is nothing to game. A harness eval is NOT a model/prompt eval: the unit under test is the harness loaded as it SHIPS (the real Claude Code system prompt + the real CLAUDE.md + real hooks/settings), which is exactly what a generic eval runner (promptfoo et al., which configure an agent from YAML) cannot reproduce — so vigiles owns this, it does not rebuild the eval stack. The discipline is to keep the costly real-model surface THIN: push every question that can be answered deterministically into runHook/runHarnessTest, and let only the two irreducibly-real-model questions (does a description FIRE, does behaviour MOVE) touch a real model. The highest-value question this layer exists to answer is BEHAVIORAL and side-effecting — does the assembled harness, run end-to-end, actually DO the task AND not do the dangerous thing? — which a completion-grader (promptfoo et al.) structurally cannot reach. \`notTool\`/\`interceptTools\` ship today for the safety half (assert/deny the irreversible externals — push, paid API — you must never actually execute); the ephemeral run environment + disposable-dependency provisioning (compose/testcontainers) that make a full side-effecting run safe to REPEAT are the committed next step. This apex tier is the most valuable AND the most expensive, so it stays THIN by design — the deterministic tiers are what make it affordable. See \`docs/safety.md\`. AFFORDABILITY is the positioning of this layer: the deterministic tiers are free (no model, no key, every commit) and the real-model tier runs on your Claude PRO/MAX SUBSCRIPTION — vigiles drives the real \`claude\` CLI, so an eval authenticates like your own CLI (no metered API billing), which is why a team can actually afford to run harness evals at all; competitors (promptfoo/DeepEval/…) hit the API SDK and bill per token on every run. Be precise about the wedge: the deterministic mock-model tier itself is TABLE STAKES — the code-defined SDKs (Pydantic AI's TestModel/FunctionModel, Vercel AI SDK's MockLanguageModelV3, LangGraph's FakeListChatModel, LlamaIndex's MockLLM) all ship a first-party fake model, so "mock the model, assert deterministically" is not what differentiates vigiles. The wedge is the three things no SDK mock does: (1) testing the harness loaded as it SHIPS — the real CLI agent with its real system prompt + CLAUDE.md + hooks, not an agent re-assembled from SDK config; (2) deterministic tool-contract enforcement of the ASSEMBLED agent (the gap the Claude Agent SDK still carries OPEN as bug #172 — declared \`tools\`/\`disallowedTools\` are not propagated to a subagent, so vigiles's PreToolUse rail in src/adapters/claude-code/agent-runtime.ts is the fix, not a nicety); and (3) a real-model tier affordable on the sub. See research/sdk-harness-testing.md. Evals run where the subscription already is — a Claude Code session or locally — NOT a standalone GitHub Actions job needing a token (CI runs only the free deterministic tiers). measureTriggerRate measures on the realistic SELECTOR (Sonnet default, a minModel floor) since a weaker model under-selects; the model lives in the spec (model/minModel), not an env override (trials, a run knob, can be an env). What vigiles tests sorts onto three rungs: R1 (cheap/deterministic — fire/trigger/contract/safety, nothing executes) and R2 (record-replay — the skill's deterministic logic consumes a tool/MCP/API result RECORDED ONCE from a real tool and REPLAYED by shadowing the binary on PATH, never model-synthesized) cover ~90%+ of real plugin surface with NO Docker, on the subscription; R3 (the real disposable service whose semantics IS what's under test — a browser, a relational DB, redis) is a THIN apex vigiles COMPOSES with a container for rather than reinventing the sandbox (a survey of popular community collections and an audit of a ~90-artifact production skill set converge on R1≈48–90% / R2≈10–43% / R3≈0–9%). Safety: R1 nothing runs, R2 fake outputs touch no real system, R3 real side effects only inside an isolated disposable container; viability: R1+R2 need no Docker and run on the sub; performance: R1/R2 ms-fast, R3 Docker cold-start stays thin. No tool does containerless reproducible e2e (every e2e benchmark/lab runs in a container) — so vigiles owns R1+R2 + sub-affordability + a clean container hand-off, not e2e-without-a-container. A SECOND, orthogonal axis decides the COST — the correctness oracle: a DETERMINISTIC check (hook decision, tool-contract, a structural fact) is free in CI, while a MODEL-GATED question (does a description FIRE, is the guidance's output GOOD, does prose guidance MOVE behaviour vs off — measureTriggerRate for firing AND, for behaviour, TWO oracles not one: the ABSOLUTE "is this exact skill's output good" via a single-arm measure()+judged()+assertRates (the right default when there is no on/off baseline — what promptfoo/DeepEval lead with), and the RELATIVE "does it MOVE behaviour vs off" via a runEval A/B + assertSignificant (regression / noise-floor)) runs on the sub; we tag the latter \`-MG\`, so a prose/guidance skill is R1-MG (nothing executes, but only a model judges its worth — fully testable on your sub via trigger-rate + judged behaviour, NOT uncovered and NOT free). State coverage as the TWO buckets vigiles owns — (A) free & deterministic + (B) model-gated on your sub — vs (C) needs-a-container (composed), and grade a plugin with TWO numbers ("% testable at all (A+B, free+sub)" vs "% needs-a-container"), never letting "model-gated" read as "uncovered" (testing a prose skill's behaviour needs a real model for EVERYONE — promptfoo, the SDKs, all of it; vigiles just does it on the sub). The sub-affordability is ToS-CLEAN: vigiles drives YOUR OWN \`claude\` CLI to test YOUR OWN harness on YOUR OWN subscription (the Claude Agent SDK ToS restricts PRODUCTIZING claude.ai login/limits in a third-party offering, not running your own tests on your own sub — exactly vigiles's posture). See \`research/eval-architecture.md\` (positioning + pros/cons), \`research/eval-coverage-and-isolation.md\` (what we test, how, what we delegate), \`research/sdk-harness-testing.md\` (the 2026-06-17 multi-SDK probe + ToS detail + mock-ergonomics borrow-list), \`docs/harness-testing.md\`, \`research/harness-testing.md\`, the eval-tier decision in \`research/eval-api-landscape.md\`, and \`research/isolated-vs-whole-harness-eval.md\` + \`research/cache-invalidation.md\`. Third layer — COMPILED HOOKS (the GATE instrument; \`vigiles/hook\`, src/core/hook-program.ts + src/hook.ts). The reliability frame is four instruments that SHRINK the harness state-space — construct (typed spec), VERIFY (lint), GATE (hooks), test (evals) — and the gate is the deterministic stop before something irreversible. A hook today is opaque shell, and the parts the author hand-writes (exit code, JSON field, a grep matcher) are exactly where the #1 verified pain lives: FALSE CONFIDENCE — a guard that LOOKS like it blocks and silently doesn't (exit 1≠2, wrong field; research/hook-pain-points.md). Invert it: author a hook as a PURE typed function \`(event) => Decision\` against a CLOSED vocabulary; \`vigiles compile\` emits the protocol, \`vigiles hook-runtime run-program\` runs it. This makes WHOLE CLASSES OF BUGS UNREPRESENTABLE — you never write the exit-code/field (false confidence), the matcher is AST-backed (\`command.runs("git push",{force})\` catches \`cd x && git push -f\` the native glob/#30519 misses; + touches()/pipesToShell() for secret-read/curl|sh), capability = API surface (an import outside \`vigiles/hook\` does NOT compile), the artifact is STAMPED (a hand-edit breaks the SHA-256 → the runtime refuses it, fail-closed), and a category mistake (block on a no-decision event) is a tsc type error (a role FAMILY: tool-gate (defineHook/defineFileGate)→Decision, plus prompt-gate (definePromptGate — sees the prompt TEXT e.prompt, deny blocks the prompt: a security filter) and stop-gate (defineStopGate — deny keeps the agent GOING, gate-until-tests-pass, honour e.stopHookActive loop guard) also →Decision on the SAME shared exit-2 runtime (so they work on CC AND Codex), inject→Injection, react→Reaction (now sees the tool RESPONSE e.response.isError()/contains()), each with its own return type). Every gate takes mode:'enforce'|'observe' — observe is the SHADOW/rollout mode (compute the same Decision, RECORD what it would block to .vigiles/hook-observations.jsonl, exit 0, never block; harness-NEUTRAL by construction — exit 0 + a local record; gateAction is the pure decision→action mapping the runtime + tests share). PROVEN by an OSS dogfood (src/hook-dogfood.test.ts): a widely-copied hand-written safety hook blocks 2/7 of the disaster battery; the compiled rewrite (examples/harness/safe-bash-guard.mjs) blocks 7/7 — measured with the verify feature's own DISASTER_CATALOG (src/guardrail-check.ts, "prove your guardrail blocks", on \`vigiles/unit\`). HONEST SCOPE, kept in every doc: compile/verify fix AUTHORING + LOGIC, not DELIVERY — CC's subagent-bypass (#34692, closed not-planned) means a PreToolUse hook does not fire for a subagent's tool calls, so a gate is a STRONG DEFAULT, never an unbypassable wall (VERIFY, a claim about logic, survives the bug; GATE, live enforcement, is capped — never claim "unbypassable"). See \`docs/compiled-hooks.md\` (the public guide) and \`research/hook-pain-points.md\` (the verified corpus + design record). @@ -444,7 +444,7 @@ Two boundary rules are enforced by \`eslint-plugin-boundaries\` (rule \`boundari "src/arg-match.ts": "Shared ArgMatcher over a tool call's input (dot-path keys; RegExp = pattern, primitive = exact; AND across keys) + matchesArgs/getPath/stringifyValue/describeArgs/serializeArgs. One matching semantics for both the check vocabulary (toolWith/notTool in src/check.ts) and the tool-interception seam (src/tool-intercept.ts), so 'did the agent call this tool with these args?' means the same whether you ASSERT on a captured call or INTERCEPT one. Pure + serializable", "src/tool-intercept.ts": - "Tool interception — the eval-tier half of the tool-call spy. A ToolIntercept (tool + optional `when` ArgMatcher + denyReason block message) is intercepted by an auto-wired PreToolUse hook (exit-2 deny) so a real-model run that DECIDES to hit a paid API / git push / spawn a paid subagent is safe + side-effect-free — the call is intercepted (prevented), NOT executed — while the tool_use (with args) still lands in the Trace for toolWith/notTool. INTERCEPT-AND-PREVENT, not a faithful mock: CC surfaces the deny as a BLOCK not a success, so it's for asserting the ATTEMPT (safety/approval-gate), not stubbing a tool to continue a flow. decideIntercept (pure, first-match-wins) + buildInterceptSettings (PreToolUse fragment over the union of intercepted tool names) + serializeIntercepts/parseIntercepts (VIGILES_INTERCEPT_TOOLS env round-trip, RegExp-safe). The `vigiles hook-runtime intercept-tool` CLI subcommand runs the decision. Wired onto the eval spec via EvalArm.interceptTools/MeasureSpec.interceptTools (auto-merged into arm settings, cache keyed on env). See docs/eval-architecture.md", + "Tool interception — the eval-tier half of the tool-call spy. A ToolIntercept (tool + optional `when` ArgMatcher + denyReason block message) is intercepted by an auto-wired PreToolUse hook (exit-2 deny) so a real-model run that DECIDES to hit a paid API / git push / spawn a paid subagent is safe + side-effect-free — the call is intercepted (prevented), NOT executed — while the tool_use (with args) still lands in the Trace for toolWith/notTool. INTERCEPT-AND-PREVENT, not a faithful mock: CC surfaces the deny as a BLOCK not a success, so it's for asserting the ATTEMPT (safety/approval-gate), not stubbing a tool to continue a flow. decideIntercept (pure, first-match-wins) + buildInterceptSettings (PreToolUse fragment over the union of intercepted tool names) + serializeIntercepts/parseIntercepts (VIGILES_INTERCEPT_TOOLS env round-trip, RegExp-safe). The `vigiles hook-runtime intercept-tool` CLI subcommand runs the decision. Wired onto the eval spec via EvalArm.interceptTools/MeasureSpec.interceptTools (auto-merged into arm settings, cache keyed on env). See research/eval-architecture.md", "src/tool-intercept.test.ts": "Tool-intercept test suite (vitest): decideIntercept (unconditional/when-scoped/first-match-wins/default reason), interceptHookDecision (PreToolUse event parse + malformed tolerance), buildInterceptSettings (matcher = escaped union of tool names), serializeIntercepts/parseIntercepts round-trip incl. RegExp survival + junk tolerance. Pure, model-free", "src/judge.ts": @@ -633,10 +633,8 @@ Two boundary rules are enforced by \`eslint-plugin-boundaries\` (rule \`boundari "Third-party adapter authoring guide: the documented small lib (vigiles/adapter) for teaching vigiles a new harness — the five ports to implement, a worked myHarnessAdapter skeleton, validating with assertAdapterConformance, wiring it (library by import, CLI by registry), and what's still behaviour-not-descriptor (renderers/decision-decode/mock HTTP server). Linked from the root README (custom adapters welcome)", "docs/testing-matrix.md": "Testing matrix: every harness-testing use case mapped to its test tier (unit / cross-runner / type / integration-CI) and file, plus why the CLI examples are .mjs and the API is TypeScript", - "docs/agent-workflows.md": - "Agent-specific workflows (Claude Code, Codex, multi-agent, Cursor)", "docs/agent-setup.md": - "Non-interactive agent setup guide (hooks via settings.json)", + "Agent setup & workflows — one guide: what `init` does + auto-detection, per-agent recipes (Claude Code / Codex / multi-agent / Cursor), non-interactive setup + fallback hooks, the recommended agent prompt, and CI (absorbs the former agent-workflows.md)", "docs/spec-format.md": "Spec format reference (target, sections, rules)", "docs/railway-subagents.md": "Public guide to railway-oriented subagents: the typed Result outcome contract (result() on an agent()), what it compiles to (vigiles:ok/err blocks), composing flat workers (railway()/delegate()/recover), and asserting the outcome deterministically (assertAgentOk/Err/Result — no model judge). States the scope decision: railway is a SUBAGENT primitive (context boundary = parse-point), NOT skills; context:fork is the bridge. Links research/railway-subagents.md (design) + research/spec-syntax-and-railway-scope.md (the scope decision)", @@ -652,10 +650,8 @@ Two boundary rules are enforced by \`eslint-plugin-boundaries\` (rule \`boundari "Rule doc: integrity check (SHA-256 hash verification for compiled markdown)", "docs/rules/coverage.md": "Rule doc: spec coverage thresholds (scripts, linter rules)", - "docs/inline-mode.md": - "Inline mode: `` comments for gradual adoption without a .spec.ts", "docs/markdown-mode.md": - "Markdown mode: inline `` comments (Level 0) and `vigiles:` YAML frontmatter (Level 1) for adoption without a .spec.ts", + "Markdown mode: the single no-spec on-ramp doc — inline `` comments (the live zero-TS floor; the former inline-mode.md reference is folded in here). Frontmatter mode is parked/disabled (kept in a comment).", "skills/linter-docs/eslint.md": "ESLint reference: plugin table, AST selectors, type-aware rules, auto-fix, edge cases", "skills/linter-docs/rubocop.md": @@ -760,7 +756,7 @@ Two boundary rules are enforced by \`eslint-plugin-boundaries\` (rule \`boundari ), "smooth-adoption": guidance( - "`npx vigiles init` must work on first run with zero config and set up BOTH layers by default: a typed spec + types (the Lint layer), a vigiles.harness.mjs starter (the Test layer), a `zernie/vigiles@v1` CI workflow (created when none exists; a stale old-API one is flagged, not skipped), vigiles added to devDependencies, and the Claude Code plugin installed via the MARKETPLACE (`/plugin marketplace add zernie/vigiles` + `/plugin install vigiles@vigiles`, into ~/.claude/plugins/ — never vendored into the repo). Onboarding is interactive at a TTY (asks which layers / CI / plugin) and NON-INTERACTIVE for agents, CI, or piped input (or with `--yes`) — so 'set up vigiles' from a Claude Code / Codex prompt Just Works without hanging on a prompt; `--lint`/`--test` scope the layers (`--harness=` overrides detection). After install the agent edits specs automatically — no workflow change required. The DEFAULT already GATES broken surfaces (the FP-safe `structural` rule group at `error` — a typo'd tool / dead hook / broken MCP / skill collision fails CI, while a clean repo stays green); the WORKFLOW tier (a spec per file + a test per surface) is OFFERED interactively (opt-out, default-yes) or via `--strict`, and `--report-only` downgrades everything to warnings for cautious migration. 'Permissive' means vigiles doesn't FORCE specs/TS on you and doesn't CRY WOLF — NOT that it ignores genuine breakage (see `install-enforcement-model`). Hesitant adopters can use inline mode (`` comments) without a .spec.ts — see `docs/inline-mode.md`. See `research/adoption-strategy.md` and `docs/agent-setup.md`.", + "`npx vigiles init` must work on first run with zero config and set up BOTH layers by default: a typed spec + types (the Lint layer), a vigiles.harness.mjs starter (the Test layer), a `zernie/vigiles@v1` CI workflow (created when none exists; a stale old-API one is flagged, not skipped), vigiles added to devDependencies, and the Claude Code plugin installed via the MARKETPLACE (`/plugin marketplace add zernie/vigiles` + `/plugin install vigiles@vigiles`, into ~/.claude/plugins/ — never vendored into the repo). Onboarding is interactive at a TTY (asks which layers / CI / plugin) and NON-INTERACTIVE for agents, CI, or piped input (or with `--yes`) — so 'set up vigiles' from a Claude Code / Codex prompt Just Works without hanging on a prompt; `--lint`/`--test` scope the layers (`--harness=` overrides detection). After install the agent edits specs automatically — no workflow change required. The DEFAULT already GATES broken surfaces (the FP-safe `structural` rule group at `error` — a typo'd tool / dead hook / broken MCP / skill collision fails CI, while a clean repo stays green); the WORKFLOW tier (a spec per file + a test per surface) is OFFERED interactively (opt-out, default-yes) or via `--strict`, and `--report-only` downgrades everything to warnings for cautious migration. 'Permissive' means vigiles doesn't FORCE specs/TS on you and doesn't CRY WOLF — NOT that it ignores genuine breakage (see `install-enforcement-model`). Hesitant adopters can use inline mode (`` comments) without a .spec.ts — see `docs/markdown-mode.md`. See `research/adoption-strategy.md` and `docs/agent-setup.md`.", ), "great-agent-flow": guidance( diff --git a/HANDOFF.md b/HANDOFF.md index 6dab5c06..b51a446c 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -14,58 +14,50 @@ ## RESUME HERE -**Branch `claude/vigiles-cost-analysis-9ko9mv`** (name is misleading — the work is -ADOPTION FIXES, not cost analysis) → **PR #66 OPEN, merging-when-green.** This session acted -on a skills-monorepo FIELD REPORT (a team ran vigiles on a 46-skill CI library with no -`plugin.json` and hit blockers) — all 7 feedback points fixed. +**Branch `claude/skill-eval-cost-benefit-q4ivfp`** (name misleading — the work is a +PUBLIC-DOCS REVAMP, not cost analysis) → **PR #67 OPEN, merging-when-green.** The user flagged the +docs as over-claiming maturity + hard to scan; this session overhauled the README + docs. -**MERGE STATE (resume here first):** PR #66 open, **merging-when-green**; **subscribed to its -activity**; a `send_later` check-in (trigger `trig_01HgpKfkov7BQAJNzHtMNEoq`, ~14:44Z) re-checks CI +**MERGE STATE (resume here first):** PR #67 open, **merging-when-green**; **subscribed to its +activity**; a `send_later` check-in (trigger `trig_01LX727g2VMn1qpe7k31eJ1o`, ~21:18Z) re-checks CI and **squash-merges into main with a CLEAN message (NO session link / model-id) when all 6 jobs -green**, else re-arms. If resuming: `get_check_runs` for #66 → merge if green, then **unsubscribe**. -Latest SHA `aa5526d` (a HANDOFF-refresh commit sits on top). CI jobs: validate/describe/check/test/ -e2e/harness; `test` runs ~5-7 min and is always last. The lone allowed failure anywhere is env-only -`dialect-drift` (CI pins CC, so it passes in CI). +green**, else re-arms. If resuming: `get_check_runs` for #67 → merge if green, then **unsubscribe**. +Latest SHA `f1d0a2d`. CI jobs: validate/describe/check/test/e2e/harness (`test` ~5-7 min, last). +Lone allowed failure: env-only `dialect-drift` (CI pins CC → passes in CI). -**CODE-REVIEW LOOP (done):** Codex-bot reviewed every pushed commit and found **13 real P2 bugs -across 5 rounds — ALL fixed + tested** (api-extractor surface, eslint void-expr, single-skill bundled -resources, root-SKILL.md coverage + colocation, sharedDirs-from-repo-root, scoped harness detection, -per-surface→repo-level fallback, hook-only + hooks-convention plugin shape, loadable-only surface -count, foreign-repo sharedDirs root, query-suffix vs glob-skip). Codex then **hit its usage quota** -(no more reviews incoming) — the loop ends by quota, NOT by proof of correctness. WATCH-OUT: the -single-skill-dir targeting + `.claude`-fallback subsystem generated most siblings; classes are now -closed + tested, but if a NEW real bug there surfaces, prefer a redesign or NARROWING the PR (drop -single-skill-dir) over another patch. +**Shipped (branch — all `docs:`):** -**Shipped (branch — `feat(scan)` + `fix(lint)` + docs):** +- **Version honesty** — STABILITY.md + README FAQ claimed "0.x"; real npm version is **12.7.0** + (semantic-release cuts a major per breaking change). Fixed to v12 + honest framing. Root cause: the + `package.json` `0.0.0-semantically-released` placeholder misread as "0.x". Added `vigiles/linting` + to the stable entry-points list. +- **README scannability** — `More` link-farm → 3 Diátaxis buckets + index pointer; collapsed the dense + audit/lint/test/eval reconciliation paragraph; trimmed Proof 2/3. Eval heading → "the only way to put + a real number on cost." +- **docs/README.md** reorganized (Guides / Reference / Explanation) + 6 docs added that were missing + from the index (harnesses, adapter-api, authoring-an-adapter, railway-subagents, faq, what-vigiles-catches). +- **Merges** — `related-tools`→`comparison`; `inline-mode`→`markdown-mode` (one no-spec on-ramp doc; + NB inline mode is LIVE — only FRONTMATTER mode is disabled); `agent-setup`+`agent-workflows`→one guide. +- **`eval-architecture.md` (54KB design-of-record ADR) relocated `docs/`→`research/`** per doc-tiers: + 5 public links repointed (testing-api / measuring-skills), research/src relative paths fixed, added to + research index + `status:`/`topic:` frontmatter, both `CLAUDE.md` + `research/CLAUDE.md` recompiled. +- **Codex-bot review** caught 1 real bug (the compiler surface is `vigiles/linting`, not `vigiles/spec`) — fixed. +- Kept standalone by JUDGMENT (merging would bloat, not help): `testing-matrix`, `migrating-from-promptfoo`. -- **P0-1** `loadPlugin` recognizes THREE repo shapes — published plugin / bare `skills/*` library / - plain `.claude/skills` user repo — via a new optional `PluginLayout.userSurfaceRoot` (`.claude` - in the CC adapter; core stays agnostic, `.claude` literal only in the adapter). Root `skills/` - WINS over `.claude/skills`. A single skill dir works. `LoadedPlugin.sources` maps each - materialized key → its real on-disk path. Reads the PROJECT `.claude` only, never `~/.claude`. -- **P0-2** `vigiles lint` now SCOPES to an explicit dir arg (was: ignored the path, scanned the - whole repo → reported foreign surfaces). `runLint` resolves ONE `scanRoot` (single existing dir → - narrow; file / several / none → cwd, so **bare `lint` is byte-identical**) threaded into all 21 - surface appliers (replaced `process.cwd()`). Bugfix — only `lint ` changes. -- **P1-3** skill-resources skips glob / placeholder refs (`* ? { } < >`) + `~/` home paths. -- **P1-4** OPT-IN `sharedDirs` config — a ref whose first segment is a declared shared dir also - resolves at the repo root; scoped so nothing outside it is masked; default byte-identical. -- **P2-6** lethal-trifecta advisory collapsed to one line per unit. **P2-7** `docs/skills-monorepo.md`. +Deleted: docs/{related-tools,inline-mode,agent-workflows,eval-architecture}.md (last one moved to research/). -Files: `src/plugin-loader.ts`, `src/core/layout.ts`, `src/adapters/claude-code/layout.ts`, -`src/scan.ts`, `src/core/skill-resources.ts`, `src/core/types.ts` (`sharedDirs`), `src/cli.ts` -(scanRoot). Tests: `src/scan.test.ts`, `src/core/skill-resources.test.ts`, `src/scan-cli.test.ts` -(lint-scoping e2e), `src/adapters/claude-code/plugin-loader.test.ts`. +**REPO ABOUT (USER ACTION — no tool/API access to set it):** paste into Settings→About — +desc "Like Lighthouse for your agent harness — verify your CLAUDE.md/AGENTS.md, skills & hooks are real, +then test and measure they actually work. Claude Code + Codex.", website https://zernie.github.io/vigiles/, +topics: claude-code codex agentic-coding ai-agents llm claude anthropic developer-tools cli linter testing +evals typescript mcp skills. -**NEXT (not blocking):** the feedback is fully addressed. The field report's exact "global -`~/.claude` skills appeared" symptom couldn't be reproduced from code (their env) — P0-2's -correct rooting closes it either way. Possible follow-up: the `init` ADOPT/untested discovery -also walks cwd; P0-2 scopes the lint appliers, adopt-discovery is a separate pass. +**NEXT (not blocking):** none required. Candidate follow-up: a deterministic `no-internal-links-in-public-docs` +lint rule (P1 roadmap) — currently hand-enforced (this session verified it by grep). -**TEST STATUS:** full vitest **2074 passed** locally; the only failure is env-only -`dialect-drift.test.ts` (installed vs pinned CC — CI pins it). eslint/tsc/test:types/fmt/api all -green locally. +**TEST STATUS:** touched-gate dogfoods pass locally (research-index, self-command-refs, doc-command-coverage, +orphans, inline) + build / integrity / orphan-docs / fmt / no-internal-links green. Full vitest not re-run +(docs-only change); env-only `dialect-drift` still fails locally (CI pins CC). ## Don't re-read unless the task needs it diff --git a/README.md b/README.md index 25c72c0c..b1198258 100644 --- a/README.md +++ b/README.md @@ -132,7 +132,7 @@ This subagent — a helper your main agent hands work to — lists a tool that d apart, so the wrong one fires (e.g. "agent-coder" ↔ "agent-tester", 83% alike) ``` -One popular plugin ships **45 pairs of skills** with near-identical descriptions. Your agent picks which skill to run by _reading_ those descriptions, so when two match it fires the wrong one. Still perfectly valid markdown. +One popular plugin ships **45 pairs** of near-identical skill descriptions. Your agent picks a skill by _reading_ them — so when two match, it fires the wrong one. Still perfectly valid markdown. **[How triggering works →](docs/measuring-skills.md)** ## Proof 3 — it can quietly read your secrets and send them out @@ -144,7 +144,7 @@ One popular plugin ships **45 pairs of skills** with near-identical descriptions · can send data out (Bash, WebFetch) ``` -Hand one subagent all three powers and a poisoned web page can tell it to read your `.env` and POST it anywhere — no exploit code, just the tools it was given. The **80 still looks like a B** — that's the point: a healthy-looking grade can hide a single subagent that's a data-leak waiting to happen. vigiles spots it from the tool list alone, free, no model. +Hand one subagent all three powers and a poisoned web page can make it read your `.env` and POST it anywhere — no exploit code, just the tools it was given. The **80 looks like a B** — and that's the trap: a healthy grade hiding a subagent that's a data-leak waiting to happen. vigiles spots it from the tool list alone, free, no model. That's the whole idea: it checks your harness against **reality, not style**. Every tool, hook, file, script, and skill you reference is verified to actually resolve — and where you name a linter rule, it's checked to exist _and_ be enabled (ESLint, Ruff, Clippy, and more). **[Everything it catches →](docs/what-vigiles-catches.md)** · point `audit` at a whole marketplace and it ranks every plugin the same way. @@ -160,7 +160,7 @@ That's the whole idea: it checks your harness against **reality, not style**. Ev | `test` | Does the harness behave? | No — a scripted stand-in | Every commit | | `eval` | Does a skill actually help? | Yes — your subscription | On demand | -`audit` and `lint` share one engine. **`lint` is the CI gate** — it fails the build on broken references, bad tool contracts, dead hooks, and skill collisions (Proofs 1 and 2). **`audit`** runs those same checks, adds the Safety ring, renders the graded report, and can also run two opt-in _live_ checks (does your MCP server connect, do your skills fire). `test` and `eval` go past _does it exist_ to _does it work_. (`init` / `compile` / `eject` manage the spec layer underneath; you rarely run them by hand.) +**One engine, two doors.** `audit` is the local report; **`lint` is the CI gate** that fails the build on the same deterministic checks — broken refs, bad tool contracts, dead hooks, skill collisions (Proofs 1–2). `test` and `eval` go further: past _does it exist_ to _does it work_. (`init` / `compile` / `eject` manage the spec layer underneath — you rarely run them by hand.) ### 🔎 Lint — your instructions stop lying @@ -172,7 +172,7 @@ Every path, script, symbol, and rule verified against reality — plus tool cont A hook that blocks nothing, a skill that hijacks unrelated prompts, context that never reaches the model — each passes a naive "did it run?" check. That gap is **false confidence**: a guard that looks like it works and silently doesn't. vigiles tests the real thing — hooks block, skills fire, subagents finish what they promised, a stray `git push` is caught before it happens. It drives a scripted stand-in for the model, not a live call, so it needs no key and runs on every commit. **[How testing works →](docs/harness-testing.md)** -### 📊 Eval — does a skill help, or just cost more? +### 📊 Eval — the only way to put a real number on cost _"Caveman Mode cuts 65% of your tokens." Says who?_ vigiles A/Bs the claim on real coding tasks and hands you three numbers: the **token bill**, whether it hit its **target**, and whether your code still **works**. @@ -237,18 +237,22 @@ Targets Claude Code and Codex out of the box, or [your own harness](docs/authori - **Is this a framework I have to build around?** No. It's a tool you run — like ESLint, Lighthouse, or `npm audit`. One command, a report, an optional CI gate. There's a library API for automation, but you never touch it to get value. - **Isn't this just a markdown linter?** No — it checks whether your instruction file is _true_ (every path/script/symbol/rule exists and is enabled), then tests and measures your harness. A style linter can't do any of that. - **Do I have to write TypeScript?** No — your agent writes the spec (`init` adopts your CLAUDE.md or AGENTS.md into one), or plain markdown lints with zero new files. Compiler-grade guarantees are opt-in, like TS's `strict` ([why?](docs/faq.md#why-are-the-strongest-guarantees-opt-in-not-the-default)). -- **Is it stable enough to adopt?** Yes — the CLI is stable; only the library API is still evolving ([details](STABILITY.md)). +- **Is it stable enough to adopt?** The CLI you run is small and rarely changes; the library API still moves between releases. The high version number is release automation (a new major per breaking change), not age — see [Stability](STABILITY.md). - **Non-JS repo?** `npx vigiles lint` verifies your CLAUDE.md or AGENTS.md with no install (Ruff/Clippy/Pylint/… too). **[Full FAQ →](docs/faq.md)** **Not for you if** you want a model/capability benchmark or runtime guardrails in the request path — vigiles is build-/CI-time. -## More +## Docs -**Docs** — **[What it catches and prevents →](docs/what-vigiles-catches.md)** · **[Verifying instruction files →](docs/verifying-instruction-files.md)** ([rules matrix](docs/verifying-instruction-files.md#the-validation-rules--the-full-matrix)) · **[Harness testing →](docs/harness-testing.md)** · **[Measuring skills →](docs/measuring-skills.md)** · **[CLI →](docs/cli.md)** · **[GitHub Action →](docs/github-action.md)** · **[Skills →](docs/skills.md)** · **[Plugin-author guide →](docs/for-plugin-authors.md)** · **[Docs index →](docs/README.md)** · **[API reference →](https://zernie.github.io/vigiles/)** +The **[docs index](docs/README.md)** is the full map, grouped by what you're doing: -**Project** — **[Stability →](STABILITY.md)** · **[Related tools →](docs/related-tools.md)** · companion to [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need). +- **Guides** — [verify instruction files](docs/verifying-instruction-files.md) · [test your harness](docs/harness-testing.md) · [measure a skill](docs/measuring-skills.md) · [ship a plugin](docs/for-plugin-authors.md) · [Codex & other harnesses](docs/harnesses.md) +- **Reference** — [CLI](docs/cli.md) · [rules matrix](docs/verifying-instruction-files.md#the-validation-rules--the-full-matrix) · [testing API](docs/testing-api.md) · [full API](https://zernie.github.io/vigiles/) +- **Explanation** — [what it catches](docs/what-vigiles-catches.md) · [how it compares](docs/comparison.md) · [FAQ](docs/faq.md) + +**Project** — [Stability](STABILITY.md) · [Related tools](docs/comparison.md#what-vigiles-composes-with) · companion to [Feedback Loop Is All You Need](https://zernie.com/blog/feedback-loop-is-all-you-need). ## License diff --git a/STABILITY.md b/STABILITY.md index e158151e..cdb51fdc 100644 --- a/STABILITY.md +++ b/STABILITY.md @@ -1,19 +1,23 @@ # Stability -> vigiles is **0.x**. This page states exactly what you can depend on today and -> what may still move, so you can adopt the stable parts now without getting -> surprised by a change to the parts that are still evolving. +> vigiles is at **v12** — but read that as _"still moving fast,"_ not +> _"battle-hardened."_ `semantic-release` cuts a **new major on every breaking +> API change**, and there have been a lot of them. The number is an artifact of +> how it ships, not a claim of maturity. This page says what I try hardest not +> to break, and what's still in motion. -Honest beats a fake 1.0: pre-1.0 semver keeps the deeper, still-moving surfaces -free to improve while the parts most people actually use stay put. +The steadiest contract is the **CLI** — the commands you run, their flags, and +their exit codes. Most of the churn is in the library API underneath it. ## What's stable — depend on it - **The CLI** — the verbs (`init`, `compile`, `lint`, `test`, `eval`, `scan`, `generate`), their flags, and their **exit codes** - (`0` clean / `1` warn / `2` error). This is the narrowest, stablest contract - and what ~90% of users touch — including the GitHub Action, which wraps it. + (`0` clean / `1` warn / `2` error). This is the narrowest, steadiest contract + and the surface almost everyone touches — including the GitHub Action, which wraps it. - **The authoring + testing library entry points:** + - `vigiles/linting` — the compiler + reference verification + (`compileClaude`, `compileSkill`, …). - `vigiles/spec` — the core builders (`enforce`, `guidance`, `claude`, `skill`, `agent`, `file`, `cmd`, `ref`, `dir`, `glob`, `result`, `delegate`, `railway`). diff --git a/docs/README.md b/docs/README.md index f7dc32ab..75052bf0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,60 +1,65 @@ # Documentation — index -How-to and reference docs for using vigiles. New here? Start with the -[README](../README.md). +How-to and reference docs for using vigiles. **New here? Start with the +[README](../README.md)** for the pitch and a 5-minute quick start. -## Shipping a plugin? +The docs are grouped by what you're trying to do: -- [`for-plugin-authors.md`](for-plugin-authors.md) — the plugin-author journey end to end: scan a draft for structural health, fix what it flags, make your skills actually fire for users, rank against a marketplace, and gate it in CI. +- **[Guides](#guides--help-me-do-x)** — step-by-step, task-first ("help me do X"). +- **[Reference](#reference--the-exact-flag-symbol-or-rule)** — exact flags, symbols, rules. +- **[Explanation](#explanation--why-its-built-this-way)** — the reasoning and trade-offs. -## Verify your instruction files (layer 1) +--- -- [`verifying-instruction-files.md`](verifying-instruction-files.md) — the full guide: the markdown→typed-spec ladder, the three rule types (`enforce` / `guidance` / `guard`), verified references + marks, and the before/after tables. -- [`skills-monorepo.md`](skills-monorepo.md) — adopt vigiles in a CI-tested skill library or a plain `.claude/` repo (no `plugin.json`): the three repo shapes it loads, the `sharedDirs` opt-in, and what a `SKILL.md` body ref resolves. +## Guides — "help me do X" -## Guard the harness — compiled hooks +### Verify your instruction files (the Lint layer) -- [`compiled-hooks.md`](compiled-hooks.md) — author a hook as a pure typed function against the closed `vigiles/hook` vocabulary and compile it, making whole classes of hook bugs unrepresentable (false confidence, matcher bypass, capability creep). The deterministic gate instrument beside verify + test. +- [`verifying-instruction-files.md`](verifying-instruction-files.md) — the master guide: the markdown → typed-spec ladder, the three rule types (`enforce` / `guidance` / `guard`), verified references, and the before/after tables. Holds the [full validation-rules matrix](verifying-instruction-files.md#the-validation-rules--the-full-matrix). +- [`markdown-mode.md`](markdown-mode.md) — the no-spec on-ramp: verify rules in plain markdown with inline `` comments, no TypeScript. +- [`skills-monorepo.md`](skills-monorepo.md) — adopt vigiles in a CI-tested skill library or a plain `.claude/` repo (no `plugin.json`). -## Two on-ramps — plain markdown → typed spec +### Test & measure your harness (the Test + Eval layers) -- [`markdown-mode.md`](markdown-mode.md) — verify rules in plain markdown with inline `` comments, no TypeScript (frontmatter is a kept, demoted advanced option). -- [`inline-mode.md`](inline-mode.md) — inline-comment mode in depth. -- [`spec-format.md`](spec-format.md) — the typed `.spec.ts` format (target, sections, rules, verified references) — the source of truth. +- [`harness-testing.md`](harness-testing.md) — task-first how-to: pick what you want to test (hook / wiring / skill firing / behaviour) and the tier that answers it, with a copy-paste first test and CI. + - [`harness-testing-claude-code.md`](harness-testing-claude-code.md) — Claude Code specifics: `scriptModel`, `${CLAUDE_PLUGIN_ROOT}` / `pluginDir` / the `Skill` tool, the bubblewrap sandbox. + - [`harness-testing-codex.md`](harness-testing-codex.md) — Codex specifics: `runHarnessTest({ adapter: codexAdapter })` against real `codex exec`, the Responses mock, what maps and what doesn't. +- [`measuring-skills.md`](measuring-skills.md) — A/B a skill, plugin, model, or rule change on real coding tasks: the metric triple (bill / target / blast-radius), the worked example, and why it's affordable on your subscription. +- [`migrating-from-promptfoo.md`](migrating-from-promptfoo.md) — move existing skill evals onto the subscription: the concept + assertion mapping, a worked side-by-side, and the honest gaps. -## Reference - -- **Library entry points** (grouped by concern, so a future non-Claude-Code harness can sit beside the current one): - - `vigiles/linting` — Layer 1: the spec builders + compiler (`claude`, `enforce`, `guidance`, `file`, `cmd`, `symbol`, …). - - `vigiles/testing` — Layer 2: the three tiers (`runHook`, `runHarnessTest`, `runEval`) + the runner-agnostic assertions. - - `vigiles/claude-code` — the Claude Code-specific adapter (`loadPlugin`, `scriptModel`, the mock). - - `vigiles/spec` — the authoring surface (the spec builders; also the module-augmentation target for generated types). - - Per-tier barrels `vigiles/unit` / `vigiles/integration` / `vigiles/e2e` make a test's capability legible from its import. -- **[API reference (generated) →](https://zernie.github.io/vigiles/)** — every exported symbol across all entry points, generated from the source by API Documenter and published to GitHub Pages. The hand-written guides here are the human-facing layer; this is the exhaustive symbol-level reference. -- [`cli.md`](cli.md) — the full CLI, the Claude Code plugin, and the `lint` validation rules. -- [`github-action.md`](github-action.md) — run vigiles in CI: the composite Action, every input, the sticky PR comment, versioning. -- [`linter-support.md`](linter-support.md) — the 7 linter catalogs + `generate-types` / `generate-schema`. -- [`comparison.md`](comparison.md) — before/after tables, the determinism breakdown, the flow diagram. -- [`related-tools.md`](related-tools.md) — what vigiles composes with rather than replaces. -- **Validation rules:** [`require-instructions-spec`](rules/require-instructions-spec.md) · [`require-skill-spec`](rules/require-skill-spec.md) · [`integrity`](rules/integrity.md) · [`coverage`](rules/coverage.md) · [`untested-skill`](rules/untested-skill.md) · [`untested-subagent`](rules/untested-subagent.md) · [`untested-hook`](rules/untested-hook.md) · [`unmarked-refs`](rules/unmarked-refs.md). - -## Test your harness (layer 2) - -- [`harness-testing.md`](harness-testing.md) — the task-first how-to guide: pick what you want to test (hook / wiring / skill firing / behaviour) and the tier that answers it, with a copy-paste first test, CI, and the coverage table. - - [`testing-api.md`](testing-api.md) — the full API reference: every predicate, assertion, `check`, matcher, and option (`measureTriggerRate` / `runEval` / significance), plus imports & harness selection. - - [`harness-testing-claude-code.md`](harness-testing-claude-code.md) — Claude Code specifics: the oh-my-claudecode walkthrough, `${CLAUDE_PLUGIN_ROOT}` / `pluginDir` / the `Skill` tool, `scriptModel`, the bubblewrap sandbox. - - [`harness-testing-codex.md`](harness-testing-codex.md) — Codex specifics: `runHarnessTest({ adapter: codexAdapter })` against real `codex exec`, the OpenAI Responses mock, what maps and what doesn't. -- [`testing-matrix.md`](testing-matrix.md) — every use case mapped to its test tier and file. -- [`sandboxing.md`](sandboxing.md) — what the sandbox isolates vs records (honestly): IO / `rm -rf`, the three network modes (deny-all / `recordEgress` / allowlisted `egress: { allow }`), tiers and limits. - -## Measure what works (layer 3) - -- [`measuring-skills.md`](measuring-skills.md) — A/B a skill, plugin, model, or rule change on real coding tasks: the metric triple (bill / target / blast-radius correctness), the worked `measureArms` example, the ecosystem benchmark, and why it's affordable on your subscription. -- [`migrating-from-promptfoo.md`](migrating-from-promptfoo.md) — move existing skill evals onto the subscription: the concept + assertion mapping, a side-by-side worked example, and the honest gaps (redteam). -- [`eval-architecture.md`](eval-architecture.md) — the cost model + the two testing verbs reconciled with what ships. - -## Skills & agents +### Author & ship - [`skills.md`](skills.md) — authoring a SKILL.md across the three on-ramps; the prose-vs-gates split. -- [`agent-setup.md`](agent-setup.md) — non-interactive setup for agents (hooks via settings.json). -- [`agent-workflows.md`](agent-workflows.md) — workflows for Claude Code, Codex, Cursor, multi-agent. +- [`compiled-hooks.md`](compiled-hooks.md) — author a hook as a pure typed function against the closed `vigiles/hook` vocabulary and compile it, making whole classes of hook bugs unrepresentable (false confidence, matcher bypass, capability creep). +- [`railway-subagents.md`](railway-subagents.md) — the typed `Result` subagent contract: declare a typed outcome, compose flat workers, and assert the outcome deterministically (no model judge). +- [`for-plugin-authors.md`](for-plugin-authors.md) — the plugin-author journey end to end: scan a draft, fix what it flags, make your skills fire, rank against a marketplace, gate it in CI. +- [`github-action.md`](github-action.md) — run vigiles in CI: the composite Action, every input, the sticky PR comment, versioning. + +### Harnesses, adapters & agents + +- [`harnesses.md`](harnesses.md) — which harness vigiles targets and how you pick one (by import), plus the capability matrix. +- [`authoring-an-adapter.md`](authoring-an-adapter.md) — teach vigiles a new harness: the five ports, a worked skeleton, validating with the conformance kit. +- [`agent-setup.md`](agent-setup.md) — agent setup & workflows in one guide: what `init` does, per-agent recipes (Claude Code / Codex / multi-agent / Cursor), non-interactive setup + fallback hooks, and CI. + +## Reference — "the exact flag, symbol, or rule" + +- [`cli.md`](cli.md) — the full CLI: every verb and flag, the Claude Code plugin, `lint` vs `audit`. +- [`testing-api.md`](testing-api.md) — the full harness-testing API: every predicate, assertion, `check`, matcher, and option (`measureTriggerRate` / `runEval` / significance). +- [`spec-format.md`](spec-format.md) — the typed `.spec.ts` format (target, sections, rules, verified references) — the source of truth. +- [`linter-support.md`](linter-support.md) — the 7 linter catalogs + `generate-types` / `generate-schema`. +- [`adapter-api.md`](adapter-api.md) — the adapter API reference: every port field, the conformance functions, the registry API. +- **Validation rules:** the [full matrix](verifying-instruction-files.md#the-validation-rules--the-full-matrix) lives in the linting guide; each rule has a doc under [`rules/`](rules/). +- **Library entry points** (grouped by concern so a future harness can sit beside the current one): + - `vigiles/linting` — the compiler + reference verification (`compileClaude`, `compileSkill`, …). + - `vigiles/spec` — the spec builders (`claude`, `enforce`, `guidance`, `file`, `cmd`, `symbol`, …) and the module-augmentation target for generated types. + - `vigiles/testing`, `vigiles/unit` — the harness-test tiers + the `check` vocabulary + runner-agnostic assertions. + - `vigiles/claude-code`, `vigiles/codex` — the per-harness adapters. + - `vigiles/adapter` — the adapter-authoring kit. +- **[API reference (generated) →](https://zernie.github.io/vigiles/)** — every exported symbol across all entry points, generated from the source. The hand-written guides here are the human-facing layer; this is the exhaustive symbol-level reference. + +## Explanation — "why it's built this way" + +- [`what-vigiles-catches.md`](what-vigiles-catches.md) — the taxonomy of problems vigiles handles: the prevented / caught / measured model, biggest-problem-first. +- [`comparison.md`](comparison.md) — before/after tables, the determinism breakdown, the flow diagram, and what vigiles composes with rather than replaces. +- [`sandboxing.md`](sandboxing.md) — what the sandbox isolates vs records (honestly): IO / `rm -rf`, the three network modes, tiers and limits. +- [`faq.md`](faq.md) — the front-door FAQ across all four layers. diff --git a/docs/agent-setup.md b/docs/agent-setup.md index 26dfd95e..1d1b95f8 100644 --- a/docs/agent-setup.md +++ b/docs/agent-setup.md @@ -1,58 +1,122 @@ -# Agent Setup Guide +# Agent setup & workflows -**One command is all you need.** `npx vigiles init` handles everything non-interactively — the agent runs it, skills and hooks take over, and there are no manual chores afterward. This guide shows what happens and what fallbacks exist. +**One command is all you need.** `npx vigiles init` handles everything +non-interactively — the agent runs it, the installed skills and hooks take over +(auto-compiling specs, blocking stray edits, nudging when something needs +attention), and there are no manual chores afterward. This guide covers what +`init` does, the per-agent specifics, and the fallbacks. → Back to [README](../README.md) ## Contents -- [What an Agent Can Do](#what-an-agent-can-do) -- [Non-Interactive Setup](#non-interactive-setup) - - [Step 1: Run the wizard](#step-1-run-the-wizard) - - [Step 2: Install hooks directly (fallback)](#step-2-install-hooks-directly-fallback) - - [Step 3: Edit the spec](#step-3-edit-the-spec) - - [Step 4: Compile and verify](#step-4-compile-and-verify) -- [Recommended Agent Prompt](#recommended-agent-prompt) -- [What the Agent Gets Wrong](#what-the-agent-gets-wrong) +- [What `init` does](#what-init-does) +- [Per-agent](#per-agent) — [Claude Code](#claude-code) · [Codex / Copilot](#codex--github-copilot) · [Multi-agent](#multi-agent-claude--codex) · [Cursor / Windsurf](#cursor--windsurf--other-formats) +- [Non-interactive setup (agents & CI)](#non-interactive-setup-agents--ci) +- [CI pipeline](#ci-pipeline) +- [What the agent gets wrong](#what-the-agent-gets-wrong) - [See also](#see-also) -## What an Agent Can Do +## What `init` does -| Action | Agent can do it? | How | -| ------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| Create spec file | Yes | `npx vigiles init` (non-interactive wizard) | -| Generate types | Yes | `npx vigiles generate types` | -| Compile specs | Yes | `npx vigiles compile` | -| Add the dev dependency | Yes | `npx vigiles init` adds `vigiles` to devDeps | -| Add CI step | Yes | Edit `.github/workflows/*.yml` directly | -| Install the plugin | Maybe | `claude plugin install vigiles@vigiles` if the `claude` CLI is on PATH; else the user runs the two `/plugin` commands in-session | -| Install hooks (fallback) | Yes | Write to `.claude/settings.json` directly | +`vigiles init` scans your project, **auto-detects which agents you already use**, +and sets both layers up. No `--target` flag needed unless you want to override. -**The plugin installs globally** — into `~/.claude/plugins/`, never vendored into your repo. `init` calls the `claude plugin` CLI when available. If the `claude` CLI is not on PATH, it prints the two in-session slash commands for the user to run. An agent that can't reach `claude` at all can still get hook behaviour by writing directly to `.claude/settings.json` (see Step 2 below). +| Signal | What it means | +| ------------------------------------------ | ------------------------------------------------- | +| `CLAUDE.md` exists | Claude Code in use — suggest migration if no spec | +| `AGENTS.md` exists | Codex / GitHub Copilot in use | +| `.claude/` directory | Claude Code project config | +| `.cursorrules` | Cursor in use — suggest rule-porter | +| `.github/copilot-instructions.md` | GitHub Copilot custom instructions | +| `.windsurfrules` | Windsurf in use | +| `rule-porter` / `rulesync` in package.json | Sync tool already installed | -## Non-Interactive Setup - -### Step 1: Run the wizard - -```bash -npx vigiles init # or `npx vigiles init --yes` to be explicit -``` - -`init` **auto-detects a non-TTY** and runs without prompts. A user prompt as simple as _"set up vigiles in this repo"_ is enough — the agent runs this command and gets sensible defaults. - -**What `init` sets up by default:** +**What it sets up by default:** - **Lint layer** — a typed `.spec.ts` + generated types - **Test layer** — a starter `vigiles.harness.mjs` - **CI** — a `zernie/vigiles@v1` workflow at `.github/workflows/vigiles.yml` - **Dependency** — `vigiles` added to `devDependencies` -- **Plugin** — Claude Code plugin installed via the marketplace +- **Plugin** — the Claude Code plugin, installed **globally** via the marketplace (into `~/.claude/plugins/`, never vendored into your repo) -Scope it with flags when needed: `--lint`, `--test` (one layer or both), `--harness=claude,codex`, `--no-gha`, `--no-plugin`, `--strict`. A human running it in a terminal gets interactive prompts instead. +Scope with flags: `--lint` / `--test` (one layer or both), `--harness=claude,codex`, +`--no-gha`, `--no-plugin`, `--strict`. -### Step 2: Install hooks directly (fallback) +## Per-agent -The plugin already brings the hooks. If you'd rather commit project-level hooks instead of (or alongside) the plugin, write them to `.claude/settings.json`: +### Claude Code + +Instruction file: `CLAUDE.md`. Once the plugin is installed, the agent no longer +has to remember to compile: + +| Hook | Trigger | Action | +| ----------- | ----------------------------------------------- | ---------------------------------------- | +| PreToolUse | Agent tries to Edit/Write a compiled `.md` file | Blocks the edit, redirects to `.spec.ts` | +| PostToolUse | Agent edits a `.spec.ts` file | Auto-runs `vigiles compile` | +| PostToolUse | Agent edits linter config or `package.json` | Auto-runs `vigiles generate types` | + +`init` installs the plugin via the marketplace; by hand in a Claude Code session: + +``` +/plugin marketplace add zernie/vigiles +/plugin install vigiles@vigiles +``` + +⚠️ **Without the plugin**, run `vigiles compile` manually after editing specs. CI still catches stale files. + +### Codex / GitHub Copilot + +Instruction file: `AGENTS.md`, read directly — there is no plugin or hook system. +The enforcement path is: + +```bash +npx vigiles init --harness=codex # full setup: scaffolds AGENTS.md.spec.ts + types + CI + Codex skills +# 1. edit AGENTS.md.spec.ts (source of truth) +# 2. npx vigiles compile → regenerates AGENTS.md +# 3. CI: npx vigiles lint && npx vigiles generate types --check +``` + +Use the full `init --harness=codex` (not `init --target=AGENTS.md`, which only +scaffolds the spec) — it's what generates `.vigiles/generated.d.ts` and the CI +config that step 3's `generate types --check` depends on. Authoring skills install +**globally** via the cross-agent `skills` CLI (no repo vendoring): +`npx skills add zernie/vigiles -a codex -g -y` → `~/.agents/skills/`, which that +same command handles. Codex hooks (`.codex/config.toml [hooks]`) aren't auto-wired yet. + +### Multi-agent (Claude + Codex) + +Use a **single spec with multiple targets** — one source of truth, two outputs: + +```typescript +export default claude({ + target: ["CLAUDE.md", "AGENTS.md"], + rules: { ... }, +}); +``` + +Both compile from the same spec with the same linter verification. + +### Cursor / Windsurf / other formats + +vigiles compiles to **markdown only** (CLAUDE.md, AGENTS.md). For non-markdown +formats (`.cursorrules`, `.github/copilot-instructions.md`, Windsurf), use a sync +tool to convert from the compiled markdown — [rule-porter](https://github.com/nichochar/rule-porter) +or [rulesync](https://github.com/dyoshikawa/rulesync). vigiles is the source-of-truth +compiler; sync tools handle the last mile. + +## Non-interactive setup (agents & CI) + +`init` **auto-detects a non-TTY** and runs without prompts — a prompt as simple as +_"set up vigiles in this repo"_ is enough: + +```bash +npx vigiles init # or `npx vigiles init --yes` to be explicit +``` + +**Fallback — install hooks directly.** The plugin already brings the hooks. To +commit project-level hooks instead of (or alongside) the plugin, write them to +`.claude/settings.json`: ```json { @@ -73,44 +137,38 @@ The plugin already brings the hooks. If you'd rather commit project-level hooks } ``` -This is equivalent to what the plugin installs, but written directly without the skills system. - -### Step 3: Edit the spec - -**The agent reads the generated `.spec.ts`** and fills in the project's actual conventions — sections, key files, commands, and rules. Use the `edit-spec` skill instructions as a guide for the spec format. - -### Step 4: Compile and verify - -```bash -npx vigiles compile -npx vigiles lint -``` - -## Recommended Agent Prompt - -If you want an agent to set up vigiles in a project, use this prompt: +**Recommended agent prompt** — if you want an agent to set up vigiles: ``` Set up vigiles for this project: -1. Run `npx vigiles init` (it adds vigiles to devDependencies and installs the +1. Run `npx vigiles init` (adds vigiles to devDependencies and installs the Claude Code plugin via the marketplace — nothing is vendored into the repo) -2. Read the generated .spec.ts file -3. Fill in the project's actual conventions based on the codebase -4. Run `npm install`, then `npx vigiles compile` to verify everything works -5. Commit the .spec.ts, compiled .md, .vigiles/generated.d.ts, and package.json +2. Read the generated .spec.ts, fill in the project's actual conventions +3. Run `npm install`, then `npx vigiles compile` to verify +4. Commit the .spec.ts, compiled .md, .vigiles/generated.d.ts, and package.json ``` -## What the Agent Gets Wrong +## CI pipeline -Common issues when agents set up vigiles: +All agents share the same CI step: -- **Editing CLAUDE.md directly** — the PreToolUse hook prevents this if installed -- **Using wrong rule names** — `enforce("no-console")` instead of `enforce("eslint/no-console")`. The compiler catches this. -- **Forgetting to compile** — the PostToolUse hook handles this automatically -- **Adding headers inside sections** — the compiler catches `#`/`##` headers in section content +```yaml +- name: Verify specs + run: npx vigiles lint && npx vigiles generate types --check +``` + +It catches hash mismatches (someone edited the compiled `.md`), missing specs +(`require-instructions-spec`), and stale generated types. + +## What the agent gets wrong + +- **Editing CLAUDE.md directly** — the PreToolUse hook prevents this if installed. +- **Wrong rule names** — `enforce("no-console")` instead of `enforce("eslint/no-console")`. The compiler catches it. +- **Forgetting to compile** — the PostToolUse hook handles it automatically. +- **Headers inside sections** — the compiler catches `#`/`##` headers in section content. ## See also -- [Agent Workflows](agent-workflows.md) — per-agent setup (Claude Code, Codex, Cursor, CI) -- [Markdown mode](markdown-mode.md) — inline comments and frontmatter (no `.spec.ts` required) -- [CLI reference](cli.md) +- [Markdown mode](markdown-mode.md) — the no-spec on-ramp (inline `` comments). +- [CLI reference](cli.md) — every verb and flag. +- [Harnesses](harnesses.md) — how vigiles targets Claude Code, Codex, and beyond. diff --git a/docs/agent-workflows.md b/docs/agent-workflows.md deleted file mode 100644 index 1db3a519..00000000 --- a/docs/agent-workflows.md +++ /dev/null @@ -1,127 +0,0 @@ -# Agent Workflows - -**vigiles is low-friction by design.** Run `npx vigiles init` and the installed skills and hooks handle the rest — auto-compiling specs, blocking stray edits, and nudging the agent when something needs attention. This guide shows the per-agent setup. - -→ Back to [README](../README.md) - -vigiles verifies the rule references in agent instruction files — declared as inline comments, `vigiles:` YAML frontmatter, or a typed spec compiled to markdown ([markdown mode](markdown-mode.md)). Different AI agents read different files, but the validation pipeline is the same. The workflows below use spec mode, the deepest level; the inline and frontmatter levels need no build step. - -## Contents - -- [Auto-Detection](#auto-detection) -- [Claude Code](#claude-code) -- [Codex / GitHub Copilot](#codex--github-copilot) -- [Multi-Agent (Claude + Codex)](#multi-agent-claude--codex) -- [Cursor / Windsurf / Other Formats](#cursor--windsurf--other-formats) -- [CI Pipeline](#ci-pipeline) -- [See also](#see-also) - -## Auto-Detection - -`vigiles init` scans your project and **auto-detects which agents you're already using** — no `--target` flag needed unless you want to override. - -| Signal | What it means | -| ------------------------------------------ | ------------------------------------------------- | -| `CLAUDE.md` exists | Claude Code in use — suggest migration if no spec | -| `AGENTS.md` exists | Codex / GitHub Copilot in use | -| `.claude/` directory | Claude Code project config | -| `.cursorrules` | Cursor in use — suggest rule-porter | -| `.github/copilot-instructions.md` | GitHub Copilot custom instructions | -| `.windsurfrules` | Windsurf in use | -| `rule-porter` / `rulesync` in package.json | Sync tool already installed | -| Symlinked instruction files | Notes them in output | - -The wizard creates specs for detected targets, generates types, compiles, and adds a CI step. - -## Claude Code - -**Instruction file:** `CLAUDE.md` - -**Setup:** - -```bash -npx vigiles init -# init installs the plugin via the marketplace; to do it by hand in Claude Code: -# /plugin marketplace add zernie/vigiles -# /plugin install vigiles@vigiles -``` - -**What the plugin does once installed** — the agent no longer needs to remember to compile: - -| Hook | Trigger | Action | -| ----------- | ----------------------------------------------- | ---------------------------------------- | -| PreToolUse | Agent tries to Edit/Write a compiled `.md` file | Blocks the edit, redirects to `.spec.ts` | -| PostToolUse | Agent edits a `.spec.ts` file | Auto-runs `vigiles compile` | -| PostToolUse | Agent edits linter config or `package.json` | Auto-runs `vigiles generate types` | - -⚠️ **Without the plugin**, you must run `vigiles compile` manually after editing specs. CI still catches stale files. - -## Codex / GitHub Copilot - -**Instruction file:** `AGENTS.md` - -**Setup:** - -```bash -npx vigiles init --target=AGENTS.md -``` - -Codex and GitHub Copilot read `AGENTS.md` directly. There is no plugin or hook system — these agents don't support it. The enforcement path is: - -1. Edit `AGENTS.md.spec.ts` (the source of truth) -2. Run `npx vigiles compile` to regenerate `AGENTS.md` -3. CI verifies freshness: `npx vigiles lint && npx vigiles generate types --check` - -**Authoring skills for Codex** install globally via the cross-agent `skills` CLI — no repo vendoring: `npx skills add zernie/vigiles -a codex -g -y` installs into `~/.agents/skills/`. `vigiles init --harness=codex` runs this automatically. Codex hooks (`.codex/config.toml [hooks]`) aren't auto-wired yet. - -ℹ️ **If you also use Claude Code**, install the plugin (`/plugin marketplace add zernie/vigiles` then `/plugin install vigiles@vigiles`, or `vigiles init`) for auto-recompilation. - -## Multi-Agent (Claude + Codex) - -Use a **single spec with multiple targets** — one source of truth, two outputs: - -```typescript -export default claude({ - target: ["CLAUDE.md", "AGENTS.md"], - rules: { ... }, -}); -``` - -Both files compile from the same spec with the same linter verification. - -```bash -npx vigiles init # for CLAUDE.md (primary) -npx vigiles init --target=AGENTS.md # adds AGENTS.md target -``` - -Or just set `target: ["CLAUDE.md", "AGENTS.md"]` in your spec directly. - -## Cursor / Windsurf / Other Formats - -vigiles compiles to **markdown only** (CLAUDE.md, AGENTS.md). For non-markdown formats (`.cursorrules`, `.github/copilot-instructions.md`, Windsurf), use a sync tool to convert from the compiled markdown: - -- [rule-porter](https://github.com/nichochar/rule-porter) — bidirectional conversion between agent formats -- [rulesync](https://github.com/dyoshikawa/rulesync) — unified rule management across 10+ tools - -vigiles is the source of truth compiler. Sync tools handle the last mile. - -## CI Pipeline - -All agents share the same CI step: - -```yaml -- name: Verify specs - run: npx vigiles lint && npx vigiles generate types --check -``` - -This catches: - -- **Hash mismatches** — someone edited the compiled `.md` directly -- **Missing specs** — `require-instructions-spec` rule requires a `.spec.ts` behind every `.md` -- **Stale generated types** — linter config changed but types weren't regenerated - -## See also - -- [Agent Setup](agent-setup.md) — non-interactive installation and recommended agent prompt -- [Markdown mode](markdown-mode.md) — inline comments and frontmatter (no `.spec.ts` required) -- [CLI reference](cli.md) diff --git a/docs/cli.md b/docs/cli.md index 3bf58b8f..86e2b4e6 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -148,8 +148,7 @@ See the [rules matrix](verifying-instruction-files.md#the-validation-rules--the- `vigiles lint` accepts files **or a directory** (`vigiles lint .` discovers the instruction files under it); with no argument it discovers them from the repo root. -See the [agent setup guide](agent-setup.md) and -[agent workflows](agent-workflows.md). +See the [agent setup & workflows guide](agent-setup.md). ### `compile [files...]` — harness selection diff --git a/docs/comparison.md b/docs/comparison.md index 86b2257e..e756ad03 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -63,6 +63,23 @@ Out of scope — use other tools: Illustrative code blocks (typo demos, template placeholders, speculative refs in design docs) opt out via `` immediately before the fence, or `` anywhere in a file that's entirely illustrative. Placeholders containing `<` or `>` are auto-skipped. Refs that can't be verified because the underlying tool isn't installed (e.g. `pylint/X` on a machine without pylint) are reported separately from real errors. +## What vigiles composes with + +vigiles owns one thing: compile-time verification of typed specs against real +linter configs, filesystems, and `package.json`, plus testing the harness those +specs describe. Everything else, compose: + +- **Architectural linting** — [ast-grep](https://ast-grep.github.io/), [Dependency Cruiser](https://github.com/sverweij/dependency-cruiser), [Steiger](https://github.com/feature-sliced/steiger). Reference their rules via `enforce()`. +- **File sync across agents** — [Ruler](https://github.com/intellectronica/ruler), [rulesync](https://github.com/dyoshikawa/rulesync), [block/ai-rules](https://github.com/block/ai-rules). vigiles compiles the source; sync tools distribute. For non-markdown formats (`.cursorrules`, Copilot), [rule-porter](https://github.com/nichochar/rule-porter) or rulesync convert the compiled output. +- **Markdown linting** — [markdownlint](https://github.com/DavidAnson/markdownlint). vigiles generates markdown; structure is correct by construction. +- **Code-block linting in docs** — [eslint-plugin-markdown](https://github.com/eslint/eslint-plugin-markdown) for syntax, [twoslash](https://shikijs.github.io/twoslash/) for TS type-checking. +- **Prose quality** — [Vale](https://vale.sh). Different concern. +- **Runtime LLM rule checking** — opposite paradigm: those tools send your code to a model on every check (tokens, non-reproducible verdicts); vigiles compiles once and checks deterministically forever after with `eslint`, `ruff`, `tsc`, Cedar evaluation. + +Specs compile to `CLAUDE.md` by default; set `target: "AGENTS.md"` or +`target: ["CLAUDE.md", "AGENTS.md"]` for multiple outputs from one spec. See the +[spec format reference](spec-format.md). + ## Flow ``` diff --git a/docs/eval-architecture.md b/docs/eval-architecture.md deleted file mode 100644 index 2640f3e1..00000000 --- a/docs/eval-architecture.md +++ /dev/null @@ -1,718 +0,0 @@ -# Eval architecture — how vigiles tests Claude Code harness features - -> Status: design of record (2026-06-16). Captures the conceptual model behind -> the two testing verbs (`vigiles test` / `vigiles eval`), reconciles it with -> what the codebase **already** ships, and ranks the genuine remaining gaps into -> a build roadmap. Companion to [`harness-testing.md`](harness-testing.md) (the -> user guide). - -## The kicker - -Real-model evals are invoked manually (`npm run test:eval`), and their results get -frozen as `FINDING:` comments baked into the `*.eval.mjs` files. **A snapshot of a -past run is documentation, not protection** — edit a skill description and break -its trigger rate, and nothing re-ran the classifier. This doc is the reevaluation. - -> **Now shipped — the eval lock.** That exact gap is closed by a committed -> integrity stamp: `vigiles eval --update` (local, on your subscription) records -> each named eval's result; `vigiles eval --check` (CI) fails "stale" when an -> input changed without a re-run — **without a model call**. See -> [The eval lock](#the-eval-lock-the-ci-staleness-gate). The rest of this doc is -> the reasoning that led there. - -The honest scope correction up front: the gap is **narrower than "build an eval -runner,"** and the fix is **not** "add a GitHub Actions eval job." vigiles already -has `vigiles eval` (discovers + runs `*.eval.mjs`), a record/replay cache, a -significance-gated baseline, a check vocabulary scored across trials, and -trigger-rate with recall **and** precision. The real moves are (1) **run the -real-model eval where the subscription already is** — a Claude Code session (the -agent loop / web / a scheduled session) or locally, since vigiles drives the -`claude` CLI; NOT a metered GitHub Actions workflow (CI runs only the free -deterministic tiers); (2) make `vigiles eval` **fail honestly** so a session run -can't false-green (`--min`, `--no-skip`, corrupt-cache throw, the Sonnet model -floor); (3) **honest model pinning** for cached/baselined results; and (4) the -**tool-call spy/fake** for side-effecting skills. See -[What already exists](#what-already-exists) before building anything. - -## Positioning & pros/cons (the approach, decided 2026-06-17) - -> The canonical positioning **statement** lives in `CLAUDE.md` (`## Positioning`, -> layer 2). This section is the **detailed** pros/cons behind it. - -**The thesis: the harness eval you can actually afford to run.** Almost nobody -evals their harness because the usual tools (promptfoo, DeepEval, Braintrust, -Inspect) hit the model **API SDK** and bill **per token on every run** → real -money on every CI run → so it doesn't get run. vigiles inverts the cost curve two -ways: (a) **most harness questions need no model at all** — `runHook` + mock-model -`runHarnessTest` answer "does the hook fire/block/inject?" deterministically, -free, every commit; (b) when a question **is** irreducibly real-model -(does a description _fire_, does behaviour _move_), vigiles drives the **`claude` -CLI**, so the eval runs on the **Pro/Max subscription** the user already pays for — -in a Claude Code session or locally — not a metered API key in CI. (Confirmed this -session: a real eval ran with `apiKeySource:"none"`, i.e. on the OAuth sub.) - -### Pros (why this is defensible) - -- **Cost** — the structural advantage. Free deterministic tiers + sub-priced real-model - tier vs competitors' per-token-every-run. This is the only reason a small team - will _actually_ eval their harness. -- **Fidelity** — the unit under test is the harness **loaded as it ships** - (`plugin-loader`: real `plugin.json`/`hooks`/`settings`/`CLAUDE.md`). A - YAML-config eval runner reconstructs an agent; it can't host this. -- **Honesty** — measures in-plugin with real sibling competition (vs others' - optimistic one-skill isolation), on the realistic selector (Sonnet, not haiku), - with significance + `pass^k`; `interceptTools` intercepts-and-prevents a - side-effecting tool in the **real** hook layer (a safety assertion others can't - make). - -### Cons / limits (state them honestly) - -- **The sub is rate-limited.** This works _because_ the real-model surface is thin - by design — it is **not** a license for huge trial counts; heavy volume still - wants metered API or a higher tier. -- **Real-model evals stay non-deterministic** — a statistical rate ± se across - trials, never a single-run gate. (The deterministic tiers are the per-commit - gate.) -- **The tool-call spy is intercept-and-prevent, not a faithful mock** — CC - surfaces the deny as a _block_, so it asserts the ATTEMPT, not a continued flow. -- **Trigger-rate must run on the realistic model** — a cheap haiku run - under-measures selection (dogfooded: 0.50 haiku vs 0.90 Sonnet). The `minModel` - floor enforces this. -- **Evals aren't a zero-effort CI checkbox** — you run them deliberately in a - session, which is a workflow change vs "add a GitHub Action." -- **No dataset / red-team / scorer-library / web UI** — that's promptfoo's lane; - we bridge or skip, not chase. - -## Coverage & scope — what we test, what we delegate - -What a test needs from the _outside world_ sorts onto **three rungs**, and you -**pick the lowest rung that faithfully measures the thing**: - -- **R1 — cheap / deterministic (nothing executes):** hook-fires (`runHook`), - trigger-rate recall+precision (`measureTriggerRate`), tool-contract / `notTool`. - No tool, no service, no Docker. _"calls a tool" → R1._ -- **R2 — stub / record-replay:** the deterministic logic consumes a tool/MCP/API - **result** that is **recorded ONCE** from a real tool and **replayed** by - shadowing the binary on PATH / stubbing the MCP — no live service. **Never** - model-synthesized stubs (drift → false confidence); reuse the eval cache's - record/replay machinery. _"needs the result" → R2._ -- **R3 — real disposable service/container:** the real system's **semantics** is - what's under test (real SQL vs a real schema, a real browser, a DB/redis/ - analytics engine). _"real semantics under test" → R3._ - -**A second, orthogonal axis decides the cost: the oracle.** The rungs say _what -executes_; they do **not** say _who decides pass/fail_. A **deterministic** oracle -(hook block/allow, tool-contract, a structural fact) is **free, no model, in CI**; -a **model-gated** oracle (does a description **fire**? does prose guidance **change -behavior**? is the output good, judged?) needs a **real model — on your -subscription**, not metered API. We tag the latter `-MG` (e.g. **`R1-MG`** = -nothing executes but only a model can judge — _the case for any prose skill_). -Don't read "R1" as "free": a prose guidance skill is **R1-MG** — it's fully -testable (trigger-rate + a judged behavioral eval), just on the sub, not in CI. -That's the boundary, not a coverage hole. - -**What vigiles can and can't test — three buckets.** Folding both axes: -**(A) Free & deterministic** (R1/R2 + deterministic oracle — every commit); -**(B) Model-gated on your sub** (`-MG` + model oracle — no metered API); -**(C) Needs a real service** (R3 — vigiles **composes** with a container, doesn't -run it). **A + B is "testable by vigiles"; only C is delegated.** So grade a plugin -with **two numbers, not three**: **"% testable at all (free + sub)"** vs **"% that -needs a container"** — and always say which bucket, so "testable" never hides -whether it's free or sub-priced. - -**Distribution (blended, scrubbed).** A survey of popular community plugin -collections **and** an audit of a ~90-artifact real-world production skill set -**converge**: **R1 ≈ 48–90%, R2 ≈ 10–43%, R3 ≈ 0–9%.** Net — **R1+R2 covers -~90%+** of real plugin surface with **no Docker, on the subscription**; the R3 -apex is **thin** and collapses to a handful of real services. Every common -SaaS/CLI integration (GitHub / issue-tracker / chat / CI / linters / test-runners) -is faithfully **replayable at R2**. - -**The e2e landscape (honest).** Real side-effecting e2e is mature -(SWE-bench/Verified, Terminal-Bench, OSWorld, WebArena; the labs' per-task cloud -sandboxes; AISI Inspect's Docker sandbox) — but **every one runs inside a -container/VM/cloud sandbox.** There is no "safe reproducible e2e without a -container," so at R3 vigiles **composes with a container, does not reinvent the -sandbox, and does not claim containerless e2e.** - -**Across the axes.** SAFETY: R1 nothing executes; R2 fake outputs, no real system; -R3 real side effects only inside an isolated disposable container — layered with -provenance confinement + the ephemeral run env + `interceptTools`. VIABILITY: R1+R2 -need no Docker, run on the sub (affordable + cross-platform); R3 needs Docker. -PERFORMANCE: R1/R2 ms-fast deterministic; R3 Docker cold-start is seconds — keep -thin. **Non-goals:** containerless reproducible e2e; per-host egress on macOS; -verifying vendor MCP connectors' live semantics (vendor's job); becoming a -sandbox/orchestrator (compose instead). - -**Competitor comparison.** Completion-graders (promptfoo / DeepEval / Braintrust) -— metered API every run, no real-harness load, no cheap no-model tiers. -Containerized e2e (SWE-bench / Inspect / Codex) — faithful but heavy / metered / -infra. vigiles — owns **R1+R2 + sub-affordability + a clean container hand-off at -R3**. The unclaimed seam is R1+R2 + sub-pricing + compose-with-container, **NOT** -e2e-without-a-container. - -**Build verdict.** A **PATH-shim / record-replay helper (fake-on-PATH)** is -**higher leverage** than a testcontainers integration — it unlocks the ~43% R2 -with no Docker and covers far more real plugins. Real-service provisioning stays a -thin, composed apex. - -## Core model: every harness feature = a deterministic part + a behavioral part - -This is the load-bearing idea. Decompose every harness feature (a skill, a hook, -a `CLAUDE.md` rule, a subagent, an MCP server) into two parts: - -| Part | Becomes a | Mechanism | Cost | Runs | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------- | ---------------------- | ----------------- | -| **Deterministic** — does the hook fire/block? does the file parse? do permission rules match? does an extracted prompt-builder/checker produce the right string? does an MCP tool return the right shape? | **TEST** — exact binary assertion | `runHook`, `runHarnessTest`, plain `node:test` over an extracted pure core | free, no model | every PR | -| **Behavioral** — does a description _trigger_? does `CLAUDE.md` _change behavior_? does an agent reach the right _outcome_? | **EVAL** — statistical, scored, threshold-gated | `measureTriggerRate`, `runEval`/`measure` A/B | real model → real cost | gated (see knobs) | - -**Most of a feature is the deterministic part, and that's where -protection-per-dollar is highest.** Only the irreducibly-stochastic slice runs a -model. This mirrors the repo's existing "keep the real-model surface THIN" -discipline: of all harness questions, only two -are _irreducibly_ real-model — _does a description fire_ and _does behavior move_. - -Practical corollary, and a prerequisite for the dogfood work below: **most -testable skill logic is inline guidance prose, not code.** Where a deterministic -core is _embedded in a prompt_, you can't test it for free. So the highest-leverage -move is often to **extract the deterministic core into a script** (the -`prune-illustrate` `generate.sh` with its `STYLE_SUFFIX` constant is the template -to copy), then test that script at the free subprocess tier. Extraction converts -a behavioral question into a deterministic one — the cheapest possible win. - -## trigger-rate is a classifier eval, not a unit test - -Name it right, because the name dictates how you run it. The "unit under test" is -the **model's routing decision** (stochastic); the artifact being tested is the -**description string**. Each `(prompt, skill)` pair is a labeled example. The -metrics are information-retrieval metrics: - -- **recall** — fires when it should (`TriggerRateReport.rate`), -- **precision** — stays quiet when it shouldn't, including on sibling-skill prompts - (`TriggerRateReport.precision` / `falsePositiveRate`, driven by - `irrelevantPrompts`). - -Run it like an ML eval: a labeled set, a threshold (`recall ≥ 0.9`), tolerant of -noise, tracked for drift — **not** like jest with an exact assertion. This is -exactly why snapshotting its score is wrong: a frozen number protects nothing; -only re-running the classifier does. - -> Already shipped: `measureTriggerRate` + `assertTriggerRate` (min recall, -> maxFalsePositive, minPrecision) + the deterministic `checkPromptDiversity` -> pre-flight (NCD-based near-duplicate + min-size gate, so you can't measure a -> rate over three copy-pasted prompts). The framing here is the _justification_ -> for that API, and the argument for **running it in CI** rather than -> snapshotting it. - -## Two orthogonal knobs on every behavioral eval - -Every behavioral eval is configured along two independent axes. Keeping them -orthogonal is what stops the snapshot/hash machinery from metastasizing into -every test. - -1. **Reproducibility** — how you make a stochastic run repeatable: - `exact-assert` | `record/replay cassette` | `hermetic fixture` | - `live + threshold`. -2. **When you run it** — the deterministic tiers run **every commit in CI** (free, - no model); a real-model eval is **run deliberately** on the subscription, not in - CI: `on-demand (a Claude Code session / local)` | `hash-lockfile (replay)` | - `nightly/manual`. - -The snapshot/hash machinery is **just the `hash-lockfile` value of knob 2** — one -option most features never pick. Concretely: - -- a hook is `(exact-assert, every-commit CI)` — `runHook`, free, no model; -- trigger-rate is `(live + threshold, on-demand)` _if cheap_ (**Sonnet** — the - realistic selector — with bodies stubbed); run it in a session, not per-PR; -- an expensive agent eval is `(cassette, hash-lockfile)` plus a nightly live run. - -## Match the mechanism to the eval's cost - -The single rule that drives every gating decision: - -- **Cheap eval** (Sonnet, body stubbed via `stubSkillBodies`, ~pennies on the - sub): **run it deliberately with a threshold gate** — in a Claude Code session - or locally, when it's worth it, not on every PR. No snapshot machinery at all. - vigiles has the significance-gated baseline (`eval-baseline.ts`) that makes "did - this change move the number beyond the noise floor?" a real gate, not a bare - pass-rate. - -- **Expensive eval** (opus, multi-turn, N trials, spawns subagents, clones repos — - $10s–$100s/run): _pay as few times as possible and amortize._ - - **Record/replay cassette = amortization.** Pay the trajectory once at record - time; every CI replay is $0. The expensive eval becomes a deterministic fixture - until inputs change. (vigiles' `eval-cache.ts` already does input-keyed - record/replay incl. post-run filesystem restore.) - - **hash-lockfile = invalidation.** Input unchanged → replay free; input changed - → re-record (pay once). You spend the full amount _only when the definition - actually changes_ — exactly when you want to. - - **Trials are the cost multiplier** (confidence = N trials × dataset size). - N=1 smoke per PR; high-N nightly. Subset-sample per PR, full suite nightly. - `maxCostUsd` is the hard cap (already implemented). - - **The nightly live tier is the one thing you cannot amortize.** Detecting "did - the model get worse" requires hitting the live model with nothing cached. - Schedule it, cap it, budget for it. Everything else drives per-PR cost to ~0. - -## What already exists - -Read this before proposing to build anything — much of the design is shipped. - -| Capability | Module | Notes | -| ------------------------------------------------------- | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Run behavioral scripts (`*.eval.mjs`) | `vigiles eval` (`cli.ts` → `run-scripts.ts`) | aggregates pass/skip/fail by exit code; `--trials=N`. Run locally on the sub. A bare (no-target) `eval` spends quota over the whole tree, so it asks first (`decideRunScripts`): name the eval(s), `--all`, or answer the prompt; headless → refuses (exit 2). | -| Committed staleness gate (CI, no model) | `eval-lock.ts` (`eval --check`/`--update`) | integrity hash of inputs; the CI half of evals you produce locally | -| A/B harness arms + Welch significance | `eval.ts` (`runEval`, `measureArms`), `stats.ts` | the differentiator — harness loaded _as it ships_ | -| Declarative check vocabulary (data, not asserts) | `check.ts` | `tool`/`skill`/`output`/`hookFired`/`received`/`turns`/`wrote`/`subagent`/`mcp`/`judged`/`cost`/`latency`/`tokens` — one vocab, strict + scored | -| Scored eval + threshold gate | `eval.ts` (`measure`, `assertRates`), `harness-assert.ts` | rate ± se, pass^k | -| Record/replay cache | `eval-cache.ts` | input-keyed (excludes `measure`), restores post-run filesystem | -| Committed baseline + regression gate | `eval-baseline.ts` | Welch current-vs-baseline; `lowerIsBetter`; JUnit | -| Trigger-rate (recall **and** precision) | `eval.ts` (`measureTriggerRate`) | `irrelevantPrompts` → `falsePositiveRate`/`precision` | -| Cheap-firing path | `eval.ts` (`stubSkillBodies`) | strip skill body, stop at selection — ~18× cheaper | -| Prompt-set diversity pre-flight | `eval.ts` (`checkPromptDiversity`) | deterministic NCD gate, no model | -| Cost / latency / token capture, concurrency, budget cap | `eval.ts` | `maxCostUsd`, `runPool`, 429 backoff | -| JUnit output | `eval.ts`, `eval-baseline.ts` | CI-consumable | -| Deterministic hook tier | `run-hook.ts` | event-JSON → hook, every event, no model | -| Sandbox + allowlisted egress | `sandbox.ts`, `egress.ts` | confine untrusted harness code | - -What is **genuinely missing** is in [Capability gaps](#capability-gaps-ranked). - -## The eval lock (the CI staleness gate) - -Real-model evals run on your **subscription**, so they only run **locally** — -never in CI. The lock lets CI verify the committed numbers still match the current -inputs **without running the model**. It is the snapshot/lockfile pattern -(`Cargo.lock` + `npm ci`; `jest --ci` / `cargo-insta`), and it is an **integrity -hash, not a cache** — the local [record/replay cache](#what-already-exists) is a -separate, gitignored speed optimization. - -| | the **cache** | the **lock** | -| ----------- | --------------------- | ----------------------------------- | -| purpose | local iteration speed | CI staleness detection | -| lifecycle | gitignored, throwaway | **committed**, reviewed in the diff | -| runs in CI? | no | **yes** (`eval --check`, no model) | - -How you use it: - -- **`vigiles eval --update`** (local, on your subscription): records each **named** - eval's report to a committed `.vigiles/eval-locks/.lock.json`, and prints - the per-number delta vs the prior lock. -- **`vigiles eval --check`** (CI): recompute the input hash, compare. Match → - pass, **no model call**. Mismatch → fail "stale, run `--update`." The committed - diff of `recall: 0.90 → 0.65` **is the quality gate** a human reviews. -- In a workflow: `uses: zernie/vigiles@v1` with `command: eval-check` (`vigiles -init` scaffolds this job). It is a green no-op until you commit your first lock. - -**The split that makes it sound.** The lock stores only the model's _observed -behavior_ (the recorded numbers). Your script's assertions re-run live against -those numbers on every `--check`. So: - -- ❌ change an **input** (skill / prompt / model) → stale → re-run `--update`. -- ✅ change only a **threshold** in the test → valid replay, no model — the - assertion just re-judges the saved numbers. - -The hash covers the model-affecting inputs: task, files, settings, tools, -plugin-dir contents, model, and `evalApiVersion`. - -**Honest scope.** The lock proves _"your saved numbers match your current inputs,"_ -not _"they reflect today's model."_ There is no automated live run — model/harness -drift is caught when you re-run `--update` and review the moved numbers. What it -_does_ catch is the common bug: edit a skill, forget to re-eval, ship stale numbers. - -## `evalApiVersion` — a hand-bumped behavior epoch (not the CC version) - -A monotonic integer **you** own (in `.vigilesrc.json` under `eval.apiVersion`), -bumped only when a _harness change on your side_ would shift eval outputs (a -CLAUDE.md edit, a hook change) but isn't otherwise in the lock's inputs. Like a -migration number / `CACHE_VERSION`. Bumping it makes `eval --check` report the -committed results stale, forcing a local re-run. - -Why the Claude Code version is **not** hashed into the lock: - -- `--check` runs in CI where `claude` is **pinned**, while a dev's local `claude` - is whatever they have — hashing the version would false-trip `--check` on every - PR where those differ. -- It is the honest-scope line above: the gate is about author-controlled inputs. - Keeping the version out is what lets `--check` stay **binary-free** in CI. - -The version is recorded on the lock as provenance. (The local **cache** _does_ key -on it — via `HarnessRuntime.versionKey`, which is `major.minor` for Claude Code -but `""` for Codex, since Codex's minor is patch-cadence. That's local replay -soundness, a separate axis.) - -## Model strategy — measure on what users run; compare models as arms (decided 2026-06-17) - -Which model an eval uses is **not** cosmetic. Dogfooding the shipped `test-harness` -skill found a 0.50 trigger-rate on `claude-haiku-4-5` vs **0.90 on -`claude-sonnet-4-6`** — same skill, same prompts. Trigger-rate is a _selection_ -measurement and haiku is a much weaker selector, so a haiku eval gives -false-negative recall and would fail skills that are fine on the model users -actually run. Conclusions: - -1. **Default to the realistic selector — Sonnet.** `measureTriggerRate` now - defaults to `"sonnet"` (was haiku), and the `minModel` floor (also Sonnet) - fails a run that resolves below it. Haiku stays available as a deliberate, - _pessimistic_ override (a lower bound), never the default for a selection - measurement. The model lives in the **spec** (`model`/`minModel`), not a CLI/env - override — it's part of the measurement definition, not a run knob like - `--trials`. -2. **No multi-model matrix runner by default.** Running every eval across - `[haiku, sonnet, opus]` multiplies cost on every run — promptfoo's "providers" - lane, against our keep-the-real-model-surface-thin discipline. -3. **A model comparison is a harness A/B → model-as-an-arm.** When you _do_ want - "does my harness hold on the cheaper tier / after a model upgrade?", set - `model` per **arm** (`EvalArm.model`) and let the existing significance - machinery read the gap — no separate matrix DSL. `measureTriggerRate` stays - single-model (loop it for a matrix). This is the one model feature we built. -4. **(Considered, not yet built) A model FLOOR.** A configurable `minModel` - (default Sonnet) that fails/warns when an eval resolves below it — the runtime - guard (post-env) that a static lint can't give, since the haiku footgun entered - via an env var. Deferred pending a decision on warn-vs-fail + config source. - -### Honest pinning (the orthogonal axis) - -Picking the right model (above) is separate from **pinning** it for a -cached/baselined result. The defaults are floating aliases (`runEval` → `"haiku"`, -`measure`/`measureTriggerRate` → `"sonnet"`); for a lockfiled/baselined result a -floating alias is **dishonest** (it can re-point while the hash says "unchanged"). -Pin a **dated** id (e.g. `claude-haiku-4-5-20251001`) so the hash is honest; a -dated id 404ing on deprecation is a **feature** (forces a re-eval onto a current -model) as long as the failure is surfaced. `isDatedModel` + the floating-alias -cache warning already nudge this. A cache-off run (a one-shot session eval) can -use the plain `sonnet` alias without churn; pin a dated id only when you turn on -the record/replay cache or a committed baseline. - -## Deferred (YAGNI): canary / ETag scaling optimization - -Only worth it with _many_ expensive evals **and** frequent CC bumps. On a version -bump, run **one** cheap fingerprint prompt: matches → trust all snapshots, skip the -rebuild; moved → invalidate + rebuild. Plus auto-rebless-within-tolerance: when -only the version changed, auto-rerun and auto-accept if metrics are within -tolerance, page a human only when a number actually moves. **Document it, don't -build it yet.** - -## Isolation lies — bound which interactions matter (closure-scoped hashing) - -A skill's behavior depends on context (`CLAUDE.md`, sibling skills, hooks), so pure -isolation gives false confidence. But the interactions that _matter_ are a short -finite list, not a cross product: - -(a) **triggering collisions** — descriptions compete (inherently whole-set); -(b) **guidance conflicts** — skill rules vs `CLAUDE.md` vs another skill; -(c) **hook/tool interception**. - -"Add everything to the hash" globally causes (1) a rebuild storm (edit any skill → -every snapshot dies) and (2) combinatorial state explosion (testing every config -combo). Don't. - -The fix is what Bazel/Nix/Turborepo do: hash each eval's **observed dependency -closure** — the specific skills/`CLAUDE.md`/hooks that _actually loaded_ during the -run — not the global everything. Editing `wrap-up` doesn't invalidate the -`illustrate` eval because it isn't in `illustrate`'s closure. Prefer **observed** -(snapshot what the harness loaded; the `plugin-loader` already materializes this) -over **declared** (no dep lists to hand-maintain and drift). - -Granularity matched to cost: - -- cheap trigger eval → a coarse "all frontmatter" key is fine (re-runs the whole - matrix on any description edit — cheap, bodies stubbed); -- expensive behavioral → a fine closure key so an unrelated edit doesn't detonate a - $50 rerun. - -The combinatorial tail you can't enumerate: curate **named integration cases** for -(a)(b)(c), and let the **nightly full-config live run** (real harness, everything -loaded) catch the unanticipated interaction as a metric drop. Prune, don't -enumerate. - -## Behavioral / side-effecting eval is the umbrella — the adversarial-gate is one member (note 2026-06-17) - -The apex of this layer is **behavioral, side-effecting** eval — the assembled -harness run end-to-end against real-but-**disposable** dependencies (the "ephemeral -integration eval"): does the harness actually DO the task AND not do the dangerous -thing? The **adversarial-gate test** (ask the agent to skip an enforcement gate; -assert it refuses — a `notTool`-shaped check) is a high-value **member** of that -family, **not a replacement** for it. It's cheap and it sets up the **eval→enforce -bridge** (when the prose gate caves under pressure, vigiles's deterministic -hook/rail is the fix — layer 2 hands off to layer 1), but the broad behavioral -tier — multiple surfaces firing together, real side effects against ephemeral -deps, graded by the `Trace`/check vocabulary — is the flagship. Don't let the -narrow check stand in for the umbrella. - -### The adversarial-gate test — worked example and the eval→enforce bridge - -The pattern is concrete and reusable. A worked dogfood lives at -[`examples/harness/dogfood/adversarial-gate.eval.mjs`](../examples/harness/dogfood/adversarial-gate.eval.mjs): -an inline `SKILL.md` that states a measurement gate ("never run a benchmark -without a baseline; refuse if asked to skip it"), an adversarial task prompt -that explicitly asks to skip the gate ("I don't need a baseline, just scaffold -it"), and two checks: - -```js -checks: [ - notTool("Bash", { command: /bench\.sh|hyperfine|time\s/ }), // didn't do the forbidden thing - output(/baseline|refus|can't|won't/i), // pushed back in prose -]; -``` - -**The eval→enforce bridge** is the key takeaway. If this eval passes at ≥ 0.9, -the prose gate is robust enough on its own. If it passes at only ≥ 0.7, the -prose alone is fragile under adversarial pressure — and the fix is NOT a better -SKILL.md description. Prose gates can always be talked out of. The fix is a -deterministic `PreToolUse` hook that checks the forbidden condition and blocks -the call regardless of what the user says (see -`src/adapters/claude-code/agent-runtime.ts` for the hook skeleton). The eval -told you _where_ the soft boundary is; the hook is the hard wall. A rate below -the acceptable floor is an automatic referral from layer 2 (test) to layer 1 -(deterministic constraint) — that is the bridge. - -## Token & cost as a first-class measurement — input / output / cache (decided 2026-06-17) - -A harness change moves tokens on **both** sides and usually **trades them off**: a -skill or CLAUDE.md injection ADDS input every turn; a "compression" skill cuts -OUTPUT. Net cost = f(fresh-input, cached-input, output). So a single total -token/cost number can **bless a change that's net-negative** — the dogfood proof is -SkillBenchmark's Caveman run (cut output yet **2–4×'d total cost** via system-prompt -injection). Honest cost verification therefore requires the classes **separated**. - -State today: `UsageTrace` carries `inputTokens`/`outputTokens`/`costUsd`/`durationMs` -(from claude's `total_cost_usd` + `usage.input_tokens`/`output_tokens`), but (a) the -`tokens()` check **collapses** input+output into one number, (b) **cache tokens** -(`cache_creation_input_tokens`/`cache_read_input_tokens`) aren't captured at all — -and a large CLAUDE.md/skill is cached (~0.1× input), so omitting them makes the cost -of exactly the harness changes you'd test misleading, and (c) there's no first-class -A/B **delta per class**. - -Native support (decided): - -1. Extend `UsageTrace` to all token classes — `inputTokens` (fresh), - `cacheCreationTokens`, `cacheReadTokens`, `outputTokens`, `costUsd` — captured - from the CLI usage block. -2. First-class checks `inputTokens({max})` / `outputTokens({max})` / - `cacheTokens({…})` beside `cost`/`tokens` (keep `tokens` as the convenience - total). -3. A/B token/cost **delta per class** in `measureArms`, gated by the existing Welch - significance — so "verbose vs caveman" reports input↑ / output↓ / net-cost± with - a **p-value**, not an eyeballed CI overlap. This is the cost/ROI optimizer made - native and input/output-separated, and the honest-measurement differentiator - (competitors report a single total or eyeball CIs). - -## Capability gaps, ranked - -The genuinely missing primitives (everything above is shipped). Ranked by -protection-per-dollar unlocked. - -1. **KEYSTONE — tool-call spy/fake.** Assert on the **arguments** a skill causes - the agent to pass to a tool, **without executing the tool** (no real image-API - call, no real `git push`, no real subagent spawn). This is precisely what - promptfoo-style tools _can't_ do — they grade a completion; they can't see "the - agent decided to push to `main`." - - **Correction to the original framing:** a tool-spy does **not** "unlock the - cheap (free, no-model) tier." Asserting on args the _model_ chose still needs - the real model to make the routing/argument decision — you can't get it from a - scripted mock. The spy is an **eval-tier** capability: real model, **faked - tools**. The saving is **eliminating the expensive side effect**, not - eliminating the model. (Today the existing `tool()`/`skill()` checks already - _read_ `ToolCall.input`, so argument _inspection_ exists in the `Trace`; what's - missing is **interception** — preventing the call and returning a canned - result so the real-model run is cheap and side-effect-free.) - - Where the logic _can_ be lifted out of the prompt into a script, prefer that - (gap #5) — it's the free deterministic test, strictly cheaper than any - model-driven spy. - - **Shipped (inspection half):** `toolWith(name, args)` and `notTool(name, -args?)` in `src/check.ts` over a shared, serializable `ArgMatcher` - (`src/arg-match.ts`; dot-path keys, RegExp = pattern, primitive = exact) — - assert _how_ a tool was called, and the negative/safety form (#2). These read - the `Trace` the harness/eval tier already captures. - - **Shipped (interception, end-to-end):** declare `interceptTools: [{ tool, -when?, denyReason? }]` on a `measure` / `runEval` arm. `src/tool-intercept.ts` + - the `vigiles hook-runtime intercept-tool` PreToolUse subcommand deny the real execution - (exit 2), so a - real-model run that _decides_ to hit a paid API / `git push` / spawn a paid - subagent is **safe and side-effect-free** — yet its arguments still land in - the `Trace` for `toolWith` / `notTool`. The eval tier auto-merges the hook - into the arm's settings (appending, never clobbering), carries the intercept - list (RegExp matchers intact) in `VIGILES_INTERCEPT_TOOLS`, and keys the cache - on it so two intercept configs sharing tool names don't collide. Pure core - fully unit-tested; the wiring sits under the eval tier's 100% gate. - - **Honest assessment (2026-06-17) — keep, with scope.** Three caveats the - "keystone" label shouldn't paper over: - 1. **Intercept-and-prevent, not a faithful mock.** CC surfaces the exit-2 deny - as a _blocked_ call, not a success, so this is sound for "did the agent - ATTEMPT X" (safety / approval-gate / first-attempt) and unsound for "stub - the tool and let a multi-step flow continue as if it returned" — the call - is intercepted (prevented), NOT executed. There is no CC primitive for - "skip execution, return this as success." - 2. **Mostly ergonomic on the inspection side.** `toolWith` overlaps the - existing `toolUsedWith` predicate (`harness-assert.ts`); the genuinely new - bit is the serializable _negative_ check and the interception. For many - safety cases the simplest protection — **don't allowlist the tool, then - assert the attempt** — needs no new primitive; `interceptTools` earns its - keep at the margins (args-scoped interception, intercepting a tool you - otherwise want allowed, and capturing an intercepted `Task` spawn's args). - 3. **One unverified assumption.** Arg-capture-under-deny (the `tool_use` lands - in the stream _before_ the hook denies) is asserted from CC semantics but - not yet proven against a live model. `examples/harness/intercept-tools.eval.mjs` - is the end-to-end validation (skips without `claude`); run it with a key - before relying on the spy. Cost is **not** reduced — the model call remains; - only the side effect is removed. - - **vs competitors:** the _assertions_ are at parity with promptfoo `trajectory:*`; - the differentiator is intercepting in the **real shipped harness** (promptfoo - reconstructs an agent from YAML/SDK and can't), but that edge is narrow - (attempt/safety, not faithful mocking). -2. **Negative / safety assertions** (a mode of #1 — highest value, most - overlooked). Did **not** call the paid API before approval; did **not** push to - the wrong branch; did **not** file a security advisory for a model-only repro. - **Shipped:** `notTool(name, args?)` in `check.ts` + the `interceptTools` - interception from #1 — together they assert the agent _didn't_ take a dangerous action, - cheaply and for real. -3. **Outbound HTTP/curl fake + request-body assertion** (the network case of #1). - Distinct from `egress.ts` (which records/allows at the packet layer) — this - _fakes_ the endpoint and asserts the request **body** (e.g. the image prompt = - `CONCEPT + STYLE_SUFFIX`). -4. **Hermetic fixtures + seam-ability.** Committed fixture repos; skills refactored - to point at a local fixture instead of cloning/pushing for real. Partly a - _skill-side_ refactor, not a vigiles primitive — but vigiles should make the - fixture wiring ergonomic. -5. **Subprocess golden harness.** Generalize `runHook` to "run _this script_ - against a fixture, assert stdout/exit" — for extracted deterministic cores (a - miner/checker, a `generate.sh` prompt-builder). This is the tier that makes the - "extract the core" prerequisite pay off. Highest protection-per-dollar where the - logic is extractable. -6. **hash-lockfile + cassette cost machinery** (knob 2) — only for genuinely - expensive behavioral evals. `eval --check` / `--update`, `.snapshot.json`, - `evalApiVersion`, dated-model pin. -7. **Closure-scoped (observed) invalidation** — the dependency-closure hashing from - the section above; layers on top of #6 once there are enough expensive evals to - warrant it. - - **Shipped (cache-key hardening, 2026-06-17):** the record/replay key now - content-hashes a native `--plugin-dir` (`hashDir` — editing a skill in it - invalidates, where a path-only key false-replayed), treats the tool list as a - set, and salts a `CACHE_FORMAT_VERSION`; floating-alias model drift is warned. - Full best-practice survey + the shipped/deferred decisions (eviction deferred - as disk hygiene) are captured in the design record. - -## Dogfood targets - -These six skills live in a **separate portfolio repo**, used as worked examples to -validate the vigiles API — they are **not** in this repo. Mapped to tier + the gap -each needs: - -| Skill | Deterministic part (TEST) | Behavioral part (EVAL) | Gap it needs | -| ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **writing-quality** (pure guidance) | regex linter for mechanical tropes + a trigger test (currently **missing**) | A/B lift — existing eval doesn't discriminate (both arms 100% on blatant tropes) | a discriminating fixture (subtler structural-trope draft) | -| **illustrate** (paid image API) | prompt-builder asserts recipe + style-suffix; approval-gate "no API call before confirm" | one live image smoke nightly | **fake outbound curl + call-spy** (#1, #3) | -| **prune-illustrate** (paid image API; has `generate.sh` w/ `STYLE_SUFFIX` const — the **only clean unit seam**, the template to copy) | `generate.sh` assembly via faked curl (body = CONCEPT+STYLE_SUFFIX, env overrides, out path) | live smoke | **fake-curl** + subprocess golden (#3, #5). _Security:_ it commits `api-key.txt` and the trigger eval `cpSync`s `.claude/skills` to `/tmp` — leaks the key to CI; rotate → env var → gitignore → add a secret-scan guard (itself a Layer-2 dogfood) | -| **wrap-up** (git commit+push) | ToC gen, ≤3-commit skip, zero-commit stop, "no push to wrong branch / no unasked PR"; assert 4 sections + ToC, links-not-duplicates `STATE.md` | — | **hermetic git fixture + fake push + spy** (#1, #4) | -| **audience-test** (spawns 3–5 paid agents + screenshots) | panel-composition invariants (≥1 non-expert, 3–5, not all-expert) asserted from spawn **args** with agents faked; reader's-cut transform | full-run structure (7 deliverables, in character) | **fake the Agent tool + assert on its call args** (#1) | -| **cross-field-bug-hunt** (clones repos, spawns paid subagents, boots Rails+PG) | miner+checker **golden** test on frozen Lago/Solidus model fixtures → re-flags the known pair + trace, **zero model** — code already exists, needs ~no new primitive | planted-bug fixture repo → agent finds it, bucket A, right `file:line` | hermetic fixture repo + faked clone + subprocess golden (#5) | - -Two structural notes carried from the analysis: (1) most testable skill logic is -**inline guidance, not code**, so extracting deterministic cores into scripts is a -prerequisite (`generate.sh` is the model); (2) the **cross-field golden test needs -almost no new vigiles capability** — do it first. - -## Ranked build roadmap - -Ordered by protection-per-dollar, with the dogfood that validates each step. - -1. **Run the behavioral tier where the subscription already is — a Claude Code - session, NOT GitHub Actions.** The original "wire evals into CI" framing was - wrong: real-model evals don't belong in a standalone GitHub Actions workflow - that needs a metered (or sub-token-as-secret) credential. CI runs the **free - deterministic tiers** (`ci.yml` — `runHook` + mock-model `runHarnessTest`, no - token); the **real-model eval** runs on your **subscription** in a Claude Code - session (the agent loop / web / a scheduled session) or locally — `vigiles -drives the `claude`CLI, so it authenticates like your own CLI does (no metered -API). _Validates on:_ the **missing`writing-quality` trigger case\*\*. - - **Shipped (eval robustness, applies wherever `vigiles eval` runs):** - `--min=N` (fail if fewer than N evals actually ran — no silent zero), - `--no-skip` (a skipped tier fails), a **corrupt-cache throw** (a broken - cassette surfaces, not a silent re-run), a **model floor** (`minModel`, - default Sonnet — a too-weak selector fails before spending a token), and the - floating-alias cache warning. Measure trigger-rate on **Sonnet** (dogfooded: - 0.50 haiku vs 0.90 Sonnet — haiku under-selects). **Removed:** the speculative - `evals.yml` GitHub Actions workflow + the `--model`/`VIGILES_MODEL` env knob - (model belongs in the spec, not a hidden override). **Remaining:** the - `writing-quality` trigger case lives in the separate portfolio repo. -2. **Cross-field miner/checker golden fixture test.** Near-zero new primitive — the - code exists; freeze the Lago/Solidus model fixtures and assert it re-flags the - known pair + trace, zero model. Cheapest real protection available. _Needs:_ a - thin generalization toward the subprocess golden harness (#5). -3. **KEYSTONE: tool-call spy/fake + negative/safety checks** (#1, #2). Build tool - interception (capture args, return canned result, prevent side effect) at the - eval tier, and add `notTool`/arg-matcher checks to `check.ts`. _Validates on:_ - `illustrate` approval-gate, `audience-test` panel composition. -4. **Outbound curl fake + body assertion** (#3). _Validates on:_ `prune-illustrate` - `generate.sh` (body = CONCEPT+STYLE_SUFFIX) + `illustrate`. Pairs with the - secret-scan guard dogfood. -5. **Hermetic fixtures + the hash-lockfile cost machinery** (#4, #6). `eval --check` - / `--update`, `.snapshot.json`, `evalApiVersion`, dated-model pin; nightly live - tier scheduled + capped. _Validates on:_ `wrap-up` + `cross-field` against - fixture repos, gated by hash-lockfile + nightly live. -6. **Closure-scoped (observed) invalidation** (#7) — only once there are enough - expensive evals that an unrelated edit detonating a rebuild is a real pain. -7. **Native input/output/cache token + cost measurement + A/B delta** — split - `tokens()` into `inputTokens`/`outputTokens`, capture cache tokens - (`cache_creation`/`cache_read`), and report a per-class A/B delta gated by Welch - significance. The honest cost-claim verifier (the Caveman gap: output↓ but net - cost↑). _Validates on:_ the `skill-compression` (Caveman) eval — assert output↓ - AND input/net honestly, with a p-value. **HIGH** (a money story; cheap to build — - the data model is half there). -8. **Adversarial-gate check + the eval→enforce bridge** — a first-class "ask the - agent to skip the enforcement gate, assert it refuses" check (the `notTool` - shape); when it fails, point at the deterministic rail (layer 2 → layer 1). - _Validates on:_ an OMC enforcement-skill dogfood. -9. **Whole-harness trigger-rate tier** — `measureTriggerRate` is isolated today - (cheap, but it _overstates recall and understates false-positives_ because skill - selection is competitive and Claude Code evicts least-used skill descriptions - under a context budget). Add an `installSet`/`withHarness` arm that co-installs - the skill alongside the user's real set as a **release gate**, plus a - near-neighbor middle tier built on the existing `ncd`/`findSimilarRules` engine. - This is a genuine wedge — **no existing eval tool populates the install set**. - -## Where this design is wrong / open questions - -Consolidated pushback, for the record: - -1. **The biggest correction: most of the "machinery" is already built.** The cache, - the significance-gated baseline, the check vocabulary, trigger-rate - recall+precision, cost/budget/concurrency, JUnit — all shipped. Framing this as - "design the eval system" overstates the work. The real deliverables are a **CI - job**, a **gating policy**, **dated-model honesty**, and a **handful of - assertion primitives**. Don't rebuild what `eval.ts` / `eval-cache.ts` / - `eval-baseline.ts` / `check.ts` already do. -2. **The tool-spy does not move work to the free tier.** It needs the real model - (the routing decision is what you're testing); it only removes the _side effect_. - Treat it as a cheaper/safer **eval**, not a deterministic test. The genuinely - free win is **extracting the deterministic core into a script** and testing it at - the subprocess golden tier — so #5 is arguably co-equal with the keystone where - the logic is extractable. -3. **hash-lockfile vs the existing baseline must be reconciled, not duplicated.** - For cheap evals the existing `(live + threshold, on-demand)` baseline gives - _strictly more_ drift protection than a lockfile. The lockfile is a cost - concession for expensive evals **only**, and only safe with the nightly backstop. - Retrofitting cheap evals onto a lockfile would _remove_ protection. -4. **An HTTP cassette does not escape the snapshot problem.** Replaying one recorded - trajectory is the same false-green as a frozen comment. Only the nightly live run - detects model drift. This is a property of _replay_, not of the recording format. -5. **Everything else in the original thinking holds and is good:** the - feature = test + eval decomposition, the two orthogonal knobs, cost-matched - mechanism, trigger-rate-as-classifier, `evalApiVersion` as a behavior epoch - distinct from the CC version, dated-model honesty, the deferred canary, and - closure-scoped (observed) invalidation. These are the spine of the doc. - -## Where to start - -**Step 1 + Step 2 in parallel**, because they're cheap and prove the model end to -end: - -- Wire the cheap behavioral tier into CI as a gate, pin a dated model, delete the - comment-snapshots — and add the missing `writing-quality` trigger case as the - first thing the new gate protects. -- Land the `cross-field` miner/checker **golden** test (zero model, code already - exists) as the first subprocess-golden dogfood. - -Then build the **keystone tool-call spy/fake (#1) + negative checks (#2)**, since -every remaining expensive dogfood (`illustrate`, `audience-test`, `wrap-up`) is -blocked on it. - - diff --git a/docs/faq.md b/docs/faq.md index e082700a..e635a197 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -49,7 +49,7 @@ Test layer drives the real `claude` / `codex` CLI. You can even **Most of vigiles needs no model and no key.** Lint and the deterministic Test tiers run in milliseconds on every commit, free. -The only thing that needs a model is a real-model **eval**. That runs on **your own Claude Pro/Max subscription** via the `claude` CLI — **$0 of metered API tokens**. Tools like promptfoo / DeepEval hit a metered API and bill per token on every run. See [the eval architecture](eval-architecture.md). +The only thing that needs a model is a real-model **eval**. That runs on **your own Claude Pro/Max subscription** via the `claude` CLI — **$0 of metered API tokens**. Tools like promptfoo / DeepEval hit a metered API and bill per token on every run. See [measuring skills](measuring-skills.md). ## What does `vigiles audit` actually run — and why did it "find nothing"? diff --git a/docs/inline-mode.md b/docs/inline-mode.md deleted file mode 100644 index c2a621aa..00000000 --- a/docs/inline-mode.md +++ /dev/null @@ -1,124 +0,0 @@ -# Inline mode - -Inline mode lets you adopt vigiles **one rule at a time**, without committing -to a `.spec.ts` compile step. You add `` HTML -comments directly to your existing `CLAUDE.md` / `AGENTS.md`, and -`vigiles lint` verifies them the same way it verifies spec-declared rules: -linter-backed existence check, closest-match typo suggestions, disabled-rule -detection, and GitHub Actions annotations. - -It's the vigiles equivalent of `// eslint-disable-next-line` — minimum -commitment, maximum incrementalism. - -## When to use it - -- You already have a `CLAUDE.md` and don't want to port it to `.spec.ts` -- You want to experiment with a single rule before committing to the full - vigiles workflow -- Your project isn't a TypeScript project at all and the build step feels - like dead weight -- Hesitant teammates want to see the verification work before accepting a - new file type in the repo - -If you already have a real TypeScript project and you want the strongest -guarantees (editor-time type safety, programmatic rule composition, the -`generate-types` type generation), use spec mode instead — see the main README. - -## Format - -A single HTML comment per rule: - -```md - -``` - -Three required pieces: - -1. `vigiles:enforce` — only `enforce` is supported inline. Guidance rules - are just paragraphs in the surrounding prose, so a `guidance` comment - would be a tautology. -2. `/` — the same reference format as `enforce()` in spec - mode. Supports all seven catalogs (ESLint, Stylelint, Ruff, Clippy, - Pylint, RuboCop, Cedar), scoped plugin names - (`eslint/@typescript-eslint/...`), and the vigiles-internal namespace - (`vigiles/` for built-in checks like `vigiles/orphan-docs`). -3. `""` — a simple double-quoted string. No newlines, no embedded - quotes. If you need either, move to spec mode. - -## Example - -```md -# My Project - - - - - -## Logging - -All application output must go through the shared logger module. -Do not use `console.log` directly in src/. - -## Async - -Every promise must be awaited or explicitly voided. The ESLint rule -enforces this automatically. -``` - -## What lint catches - -Running `vigiles lint CLAUDE.md` on the above file will: - -- Verify each `eslint/…`, `ruff/…` reference against your actual linter - config -- Emit closest-match suggestions on typos: `"no-consol"` → - `did you mean "eslint/no-console"?` -- Emit `::error` annotations when running inside GitHub Actions -- Exit with code 2 (hard error) on any failed rule, so CI fails fast - -## What lint does NOT do in inline mode - -- **No type safety at edit time.** The `.spec.ts` path gets TypeScript - squiggles in the editor because `StrictLinterRule` is a type union of - every rule in your linters. Inline mode is strings-in-markdown, so - typos only surface at `vigiles lint` time. Still catches them before - CI, just not in the editor. -- **No programmatic composition.** You can't reuse a batch of rules from - a helper. Each comment is its own line. -- **No rule deduplication via NCD.** Duplicate-rule detection runs on - spec-mode files; inline rules are ungrouped. - -All of this is fine for the adoption-onramp use case. When you outgrow it, -port to spec mode. - -## Mixing inline and spec mode - -Spec mode wins. If a file has both `CLAUDE.md.spec.ts` and inline -comments inside `CLAUDE.md`, the spec compiler will overwrite the markdown -on the next compile, and your inline comments will be gone. Pick one per -file. - -## `require-instructions-spec` - -The built-in `require-instructions-spec` validation rule demands a `.spec.ts` -sibling for every `CLAUDE.md` / `AGENTS.md`. It is **narrow**: only a -`.spec.ts` satisfies it. Inline mode is a valid plain-markdown on-ramp, but -it does **not** satisfy `require-instructions-spec` — so an inline-mode user -simply keeps the rule off (it is off by default; turning it on is a -workflow-tier opt-in for teams that want to require a typed spec). You don't -need a `vigiles-disable require-instructions-spec` comment unless you have -enabled the rule. - -## Graduating to spec mode - -When you've accumulated a dozen or so inline rules and the prose is -starting to feel crowded, run: - -```bash -npx vigiles init --target=CLAUDE.md -``` - -That scaffolds a `CLAUDE.md.spec.ts` next to your existing `CLAUDE.md`. -Copy the inline enforce rules into the `rules:` block, delete the inline -comments, and run `vigiles compile`. The markdown output will be rebuilt -with a `sha256` hash header, and future edits flow through the spec. diff --git a/docs/markdown-mode.md b/docs/markdown-mode.md index 140c921e..bae7c8a3 100644 --- a/docs/markdown-mode.md +++ b/docs/markdown-mode.md @@ -29,30 +29,62 @@ to plain markdown anytime — so graduating to a spec is never a one-way door. ## Inline `enforce` comments -The minimum-commitment path: add a single HTML comment per rule, anywhere -in your existing markdown. +The minimum-commitment path: add a single HTML comment per rule, anywhere in +your existing markdown. It's the vigiles equivalent of +`// eslint-disable-next-line` — maximum incrementalism, zero new files. ```md ``` -Only `enforce` is supported inline — the prose around the comment _is_ the -guidance, so a `guidance` comment would be a tautology. The reference uses -the same `/` format as everywhere else in vigiles. +Three pieces, all required: -This is the vigiles equivalent of `// eslint-disable-next-line`: maximum -incrementalism, zero new files. For the full reference — fenced-block -handling, scoped plugin names, graduating to a typed spec — see -[docs/inline-mode.md](inline-mode.md). +1. **`vigiles:enforce`** — only `enforce` is supported inline. The prose around the comment _is_ the guidance, so a `guidance` comment would be a tautology. +2. **`/`** — the same reference format as `enforce()` in spec mode. All seven catalogs (ESLint, Stylelint, Ruff, Clippy, Pylint, RuboCop, Cedar), scoped plugin names (`eslint/@typescript-eslint/...`), and the vigiles-internal namespace (`vigiles/orphan-docs`) work here. +3. **`""`** — a double-quoted string shown to the agent as context. No newlines or embedded quotes; if you need either, move to a spec. + +A fuller example: + +```md +# My Project + + + + + +## Logging + +All application output must go through the shared logger module. +``` ### What `vigiles lint` catches - Verifies each rule reference against your real linter config. -- Emits closest-match suggestions on typos: - `"no-consol"` → `did you mean "eslint/no-console"?` +- Emits closest-match suggestions on typos: `"no-consol"` → `did you mean "eslint/no-console"?` - Flags rules that exist but are disabled in your linter config. -- Emits `::error` annotations under GitHub Actions. -- Exits with code 2 (hard error) on any failed rule, so CI fails fast. +- Emits `::error` annotations under GitHub Actions; exits code 2 on any failed rule, so CI fails fast. + +### What it does _not_ do (vs a typed spec) + +- **No edit-time type safety.** A `.spec.ts` gets editor squiggles because rules are a type union; inline strings surface typos only at `vigiles lint` time (still before CI). +- **No programmatic composition** — each comment stands alone. +- **No NCD duplicate detection** — that runs on spec-mode files. + +That's all fine for the adoption on-ramp. When you outgrow it, graduate to a spec. + +### Mixing with a spec — don't + +A file is checked for inline rules **only when it isn't managed by a spec** (no sibling `.spec.ts`, no `vigiles:sha256 … compiled from …` header). If both exist, the compiler overwrites the markdown on the next compile and your inline comments vanish. Pick one per file. + +### Graduating to a spec + +When a dozen inline rules start crowding the prose: + +```bash +npx vigiles init --target=CLAUDE.md +``` + +That scaffolds a `CLAUDE.md.spec.ts` beside your `CLAUDE.md`. Copy the enforce rules into the `rules:` block, delete the inline comments, and run `vigiles compile` — the markdown is rebuilt with a `sha256` header, and future edits flow through the spec. `vigiles eject` reverses it anytime. --- @@ -75,7 +107,6 @@ file. ## See also -- [`inline-mode.md`](inline-mode.md) — the full inline-comment reference. - [`spec-format.md`](spec-format.md) — the typed `.spec.ts` source of truth. - [`verifying-instruction-files.md`](verifying-instruction-files.md) — the lint guide. diff --git a/docs/measuring-skills.md b/docs/measuring-skills.md index 0a460ad5..2f2805cb 100644 --- a/docs/measuring-skills.md +++ b/docs/measuring-skills.md @@ -1,7 +1,7 @@ # Measuring skills & plugins — does it actually help? -> The README has the pitch ("Measure — does it actually help, or just cost -> more?"); this is the full guide. vigiles is the only harness tool that can A/B a +> The README has the pitch (the only way to put a real number on cost); this is +> the full guide. vigiles is the only harness tool that can A/B a > skill, plugin, model, or rule change on **real coding tasks** and tell you > whether it moved the needle — on your **Claude subscription**, not metered API. @@ -86,7 +86,7 @@ Two ways to specify an arm: | promptfoo, DeepEval, … | metered API SDK | billed **per token, every run** | | **vigiles** | your Claude Pro/Max sub | **$0 extra** beyond your sub | -That's why vigiles can measure continuously — on every change, not once — while a per-token competitor cannot. Most of vigiles needs no model at all. Only this measurement tier does, and it runs where your subscription already is. See [`docs/eval-architecture.md`](eval-architecture.md) for the cost model. +That's why vigiles can measure continuously — on every change, not once — while a per-token competitor cannot. Most of vigiles needs no model at all. Only this measurement tier does, and it runs where your subscription already is. ### What a run reports — and the metered-API warning diff --git a/docs/related-tools.md b/docs/related-tools.md deleted file mode 100644 index 59e593a2..00000000 --- a/docs/related-tools.md +++ /dev/null @@ -1,22 +0,0 @@ -# Related tools - -vigiles doesn't try to do everything. It owns one thing: compile-time -verification of typed specs against real linter configs, filesystems, and -package.json, plus testing the harness those specs describe. Everything else, -compose: - -- **Architectural linting** — [ast-grep](https://ast-grep.github.io/), [Dependency Cruiser](https://github.com/sverweij/dependency-cruiser), [Steiger](https://github.com/feature-sliced/steiger). Reference their rules via `enforce()`. -- **File sync** across agents — [Ruler](https://github.com/intellectronica/ruler), [rulesync](https://github.com/dyoshikawa/rulesync), [block/ai-rules](https://github.com/block/ai-rules). vigiles compiles the source; sync tools distribute. -- **Markdown linting** — [markdownlint](https://github.com/DavidAnson/markdownlint). vigiles generates markdown; structure is correct by construction. -- **Code-block linting in docs** — [eslint-plugin-markdown](https://github.com/eslint/eslint-plugin-markdown) for syntax, [twoslash](https://shikijs.github.io/twoslash/) for TS type-checking. -- **Prose quality** — [Vale](https://vale.sh). Different concern. -- **Runtime LLM rule checking** (e.g. ai-rulez `"AI-Powered Rule Enforcement"`) — opposite paradigm. Those tools send your code to a model on every check, costing tokens and giving non-reproducible verdicts. vigiles compiles once and checks deterministically forever after with `eslint`, `ruff`, `tsc`, Cedar evaluation — tools as deterministic as their inputs. - -## Output targets - -Specs compile to `CLAUDE.md` by default. Set `target: "AGENTS.md"` or -`target: ["CLAUDE.md", "AGENTS.md"]` for multiple outputs from one spec. For -non-markdown formats (`.cursorrules`, Copilot), use -[rule-porter](https://github.com/nichochar/rule-porter) or -[rulesync](https://github.com/dyoshikawa/rulesync) to convert. See the -[spec format reference](spec-format.md). diff --git a/docs/safety.md b/docs/safety.md index 263e7c4b..3af050c3 100644 --- a/docs/safety.md +++ b/docs/safety.md @@ -102,7 +102,7 @@ const report = measure(spec, { - ✅ **Sound for:** "did the agent _attempt_ X?" — safety gates, approval checks, "no paid call", "no push to the wrong branch". `notTool` is the **negative safety assertion** a completion-grading eval structurally can't make: it sees the agent's _decision to act_, not just its final text. - ⚠️ **Not for:** stubbing a tool to return a fake success and letting a multi-step flow continue. The model sees a block, so a sequence that needs the real result breaks. (Claude Code has no "skip-but-return-success" primitive for arbitrary tools; deny is the closest safe thing.) -**Testing that an enforcement gate actually holds** — including under an adversarial prompt that asks the agent to skip it — is done with `notTool` + `output` checks in `measure`. The worked dogfood is [`examples/harness/dogfood/adversarial-gate.eval.mjs`](../examples/harness/dogfood/adversarial-gate.eval.mjs). When that eval shows a prose gate can be talked out of, the deterministic `PreToolUse` hook is the fix — see the eval→enforce bridge note in [`eval-architecture.md`](eval-architecture.md#the-adversarial-gate-test--worked-example-and-the-evalenforce-bridge). +**Testing that an enforcement gate actually holds** — including under an adversarial prompt that asks the agent to skip it — is done with `notTool` + `output` checks in `measure`. The worked dogfood is [`examples/harness/dogfood/adversarial-gate.eval.mjs`](../examples/harness/dogfood/adversarial-gate.eval.mjs). When that eval shows a prose gate can be talked out of, the deterministic `PreToolUse` hook is the fix. ## At a glance — what's confined, per tier @@ -139,7 +139,7 @@ Yes — `sandbox: false` is the explicit, greppable opt-out for code you trust, opt-in-not-always-on argument, `recordEgress`, `egress: { allow }`, dogfood findings. - [`harness-testing.md`](harness-testing.md) — the three tiers and where the boundary sits. -- [`eval-architecture.md`](eval-architecture.md) — `interceptTools`/`notTool` in the - eval design, with the intercept-≠-mock trade-off. +- [`testing-api.md`](testing-api.md) — `interceptTools`/`notTool` in the + testing API, with the intercept-≠-mock trade-off. - [`src/sandbox.ts`](../src/sandbox.ts) · [`src/egress.ts`](../src/egress.ts) · [`src/tool-intercept.ts`](../src/tool-intercept.ts) — the pure, tested seams. diff --git a/docs/sandboxing.md b/docs/sandboxing.md index 1d2e4ac3..167218de 100644 --- a/docs/sandboxing.md +++ b/docs/sandboxing.md @@ -125,7 +125,7 @@ The same `session-start` hook is dogfooded a second time under `egress: { allow: ## See also - [Testing your harness](harness-testing.md) — the three tiers + the sandbox boundary. -- [Eval architecture](eval-architecture.md) — `interceptTools`/`notTool`: +- [Testing API](testing-api.md) — `interceptTools`/`notTool`: preventing a real model's tool side effects at the eval tier. - [`src/sandbox.ts`](../src/sandbox.ts) — `decideSandbox` (the pure policy), `bwrapArgs`, `parseEgressLog`. - [`src/egress.ts`](../src/egress.ts) — the `egress: { allow }` allowlist: ruleset builder, counter parser, the pure seams. diff --git a/docs/spec-format.md b/docs/spec-format.md index 5a406062..4fd901f0 100644 --- a/docs/spec-format.md +++ b/docs/spec-format.md @@ -7,7 +7,7 @@ vigiles specs are TypeScript files (`*.spec.ts`) that compile to markdown instru Be honest about what a spec is **not** for. The reference checks — does this `file()` exist, is this linter rule enabled, is this `cmd()` a real script — do **not** need a spec. vigiles runs them on a plain CLAUDE.md via inline -[`` comments](inline-mode.md), on purpose, as the +[`` comments](markdown-mode.md), on purpose, as the on-ramp. If verification is all you want, **stay in markdown**. A spec earns its place when you cross from **declaring** your harness to diff --git a/docs/verifying-instruction-files.md b/docs/verifying-instruction-files.md index a78e42f6..e743516a 100644 --- a/docs/verifying-instruction-files.md +++ b/docs/verifying-instruction-files.md @@ -270,7 +270,7 @@ Everything vigiles compiles and lints is **deterministic** — same input, same ## See also -- [Markdown mode](markdown-mode.md) · [Inline mode](inline-mode.md) — the no-spec on-ramps. +- [Markdown mode](markdown-mode.md) — the no-spec on-ramp (inline `` comments). - [Spec format reference](spec-format.md) — every section and rule kind. - [Linter support](linter-support.md) — the 7 catalogs + `generate-types` / `generate-schema`. - [CLI & CI reference](cli.md) · [Agent setup](agent-setup.md). diff --git a/src/core/inline.test.ts b/src/core/inline.test.ts index e1cb1f92..39b92ddb 100644 --- a/src/core/inline.test.ts +++ b/src/core/inline.test.ts @@ -91,7 +91,7 @@ text }); it("ignores vigiles:enforce markers inside fenced code blocks", () => { - // Illustrative example in docs/inline-mode.md would otherwise get + // Illustrative example in docs/markdown-mode.md would otherwise get // picked up as a live rule. const { rules, errors } = parseInlineRules( `# Docs diff --git a/src/eval-cost.ts b/src/eval-cost.ts index 3da64bee..1268801d 100644 --- a/src/eval-cost.ts +++ b/src/eval-cost.ts @@ -9,7 +9,7 @@ * the `claude` CLI) + a running session tally. We deliberately do NOT show a * "% of your subscription" — Anthropic does not expose a subscription's quota or * limit programmatically (and the real limits are rolling rate windows, not a - * dollar bucket), so any percentage would be fiction. See docs/eval-architecture.md. + * dollar bucket), so any percentage would be fiction. See research/eval-architecture.md. * * Pure + injectable (env + an output sink), so the whole thing is unit-tested * without a model or a real key. diff --git a/src/eval.ts b/src/eval.ts index 4a0f1038..8055c262 100644 --- a/src/eval.ts +++ b/src/eval.ts @@ -124,7 +124,7 @@ export interface EvalArm { * opus: { model: "claude-opus-4-8" } }` — so model-as-an-arm answers "does my * harness still hold on the cheaper tier / after a model upgrade?" through the * same significance machinery, with no separate model-matrix runner. Omit to - * use the eval-level model. See `docs/eval-architecture.md` (model strategy). + * use the eval-level model. See `research/eval-architecture.md` (model strategy). */ readonly model?: string; } @@ -1186,7 +1186,7 @@ function isRecord(v: unknown): v is Record { * e.g. `claude-haiku-4-5-20251001`. A floating alias (`haiku`, `sonnet`, or even * `claude-sonnet-4-6` with no date) can change underneath you — so a cached or * baselined result pinned to it can silently hide model drift. See - * `docs/eval-architecture.md` (honest model pinning). + * `research/eval-architecture.md` (honest model pinning). */ export function isDatedModel(model: string): boolean { return /\d{8}$/.test(model); diff --git a/src/scaffold-test.ts b/src/scaffold-test.ts index bb40305e..17c626c4 100644 --- a/src/scaffold-test.ts +++ b/src/scaffold-test.ts @@ -297,7 +297,7 @@ function safetySection( // --- Safety (deterministic) — generated from ${input.name}'s side-effecting tools: ${sideEffecting.join(", ")} --- // In a real run, replace this constructed Trace with a real \`runHarness\` / // \`measure\` turn (use interceptTools so a real model's attempt is DENIED, never -// executed — see docs/eval-architecture.md). The checks below are derived from the +// executed — see research/eval-architecture.md). The checks below are derived from the // declared tools contract — the agent's "hole" asserted to stay in its lane. { const trace = {