* feat!: eval lock (CI staleness gate) + per-adapter harness versionKey Real-model evals run on your subscription, so they only run locally — never in CI. The eval LOCK closes the resulting gap: a committed integrity stamp that lets CI verify the committed numbers still match the current inputs WITHOUT a model call (distinct from the gitignored, local-speed eval cache). - `vigiles eval --update` (local): records each NAMED eval's report to a committed `.vigiles/eval-locks/<name>.lock.json` + prints the per-number delta. - `vigiles eval --check` (CI): recompute the input hash, fail "stale" on a mismatch — binary-free, deterministic. Green no-op until the first lock is committed (smooth adoption). - The clean split: the lock stores only the model's observed behavior; the script's own assertions re-run live against the replayed report, so a threshold-only edit is a valid replay while an input change is stale. - Honest scope: the harness version is provenance, NOT a hash input (CI's pinned claude ≠ a dev's local claude would false-trip), and there is no nightly run — model/harness drift is a local `--update` concern. Also extracts the harness version reduction to a per-adapter port method (`HarnessRuntime.versionKey`): Claude Code keys major.minor (~quarterly), Codex opts out with "" (its minor is patch-cadence, ~weekly). This fixes the local cache churning weekly for Codex and is where the cadence decision belongs. Whole-package coherence: the GHA gains `command: eval-check`, `vigiles init` scaffolds the eval-check CI job, and this repo dogfoods it. Wired at both eval seams (runEvalWith + measureTriggerRateWith), 100%-covered, docs updated (internal research + public guides). BREAKING CHANGE: `HarnessRuntime` (exported on `vigiles/adapter`) gains a required `versionKey(raw: string): string` method. Third-party adapters must add it; return "" to opt out of version partitioning. First-party adapters (claude-code, codex) are updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * feat: eval-lock agent awareness (skill + nudge hook) + cohesive-delivery rule Close the loop on the eval lock so the AGENT knows it exists and maintains it — without the invasive option of editing the user's CLAUDE.md. - test-harness skill documents the lock lifecycle (--update/--check, commit the lock, re-run after editing a skill). - A PostToolUse nudge hook (hooks/eval-lock-nudge.sh → hook-runtime eval-lock-nudge): after editing a SKILL.md / *.eval.* with committed locks present, inject a NON-BLOCKING reminder to re-run `vigiles eval --update`. Self-gated (silent until a lock is committed), never blocks (the gate is CI's `eval --check`). Wired into the vigiles plugin; runHook + unit + .sh tests. - NO init edit to the user's CLAUDE.md — verified that's not how plugins deliver context (no plugin-level instructions file exists; the channel is a hook/skill). Docs + dedup: - research/agent-context-delivery.md captures the VERIFIED CC/Codex context-delivery mechanics ONCE (SessionStart additionalContext is the plugin channel; CLAUDE.md is the user's file; skill descriptions always-in-context; Codex AGENTS.md + hooks but inject deferred) so we stop re-researching it. - Per-adapter guides (harness-testing-claude-code/-codex.md) explain how the reminder reaches the agent per harness (CC injects; Codex inject is a documented follow-on), with the agnostic guide linking DOWN to them (no duplication). Cohesive-view rule: - New CLAUDE.md `cohesive-feature-delivery` rule — a definition-of-done CHECKLIST (CLI × GHA × init × agent-awareness × both adapters × per-adapter deduped docs × tests) so a feature's INTEGRATION across surfaces is checked as one thing, generalizing cohesive-cli-surface + harness-parity + great-agent-flow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * test: CLI e2e for the eval-lock invalidation flow (update → check → stale) Closes the gap flagged in review: the joined CLI flow wasn't tested as one thing. A real *.eval.cjs fixture (model-free — drives runEvalWith from dist with a fake runner) run through the built CLI proves the whole loop: `eval --update` writes a committed lock, `eval --check` with the same inputs replays green (no model), and an input change makes `eval --check` go red ("stale"). Binary-free, no subscription — exactly how it runs in CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * docs: scannable eval-lock docs + prose-clarity & easy-adoption rules Make the eval-lock docs readable at a glance and frame upkeep as agent-driven, not a manual chore. Docs (scannable rewrite — ASCII flow diagram, comparison tables, ✅/❌/⚠️ signposts, short sentences, no walls of text): - docs/harness-testing.md — the lock how-to now leads with "you mostly won't touch this by hand; the test-harness skill records it + a hook reminds you," then a diagram + tables. - docs/harness-testing-claude-code.md / -codex.md — the per-harness nudge delivery as tables; the Codex inject caveat is one clear ⚠️ row. - docs/eval-architecture.md — the "what's stale vs a free replay" split as ✅/❌ bullets. - skills/test-harness/SKILL.md — Step 5 now tells the agent to run `eval --update` + commit the lock as part of its job (so it IS done for you). - README — the Quick-start reassurance leads with "Adoption is smooth: one command, then your agent does the rest," names skills AND hooks, no chores. Two new CLAUDE.md rules: - prose-clarity — public prose must be clear + scannable (lead with the point, one idea per sentence, define/drop jargon, tables/bullets/ASCII/Mermaid over walls, emoji as signposts, one bold per point). Prefer ASCII/Mermaid; never commit generated raster images into docs. - lead-with-easy-adoption — docs (README first screen especially) must make it clear up front that adoption is low-friction because skills + hooks do the work; frame every feature's upkeep as agent/skill/hook-driven, not homework. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * docs: prose-clarity sweep across 13 user-facing guides Apply the new prose-clarity rule to the prose-heavy public guides: lead with the point, one idea per sentence, tables/bullets/ASCII over walls of text, ✅/❌/⚠️/ℹ️ signposts, one bold phrase per point, jargon glossed or dropped. Net −578 lines — shorter and far more scannable, with meaning unchanged. Swept: for-plugin-authors, what-vigiles-catches, measuring-skills, skills, harnesses, comparison, faq, sandboxing, safety, agent-setup, agent-workflows, verifying-instruction-files, compiled-hooks. The onboarding pair leads with the low-friction "one command, the agent does the rest" framing. Reference docs (cli, testing-api, spec-format, …) and the per-rule docs were deliberately left alone — they're already table-shaped, and churning them is high-risk / low-value. Verified: no heading text changed (anchors intact), no `research/` links added to public docs, the validation-rules matrix is complete (28 rule links → 28), safety claims retained, command refs resolve (self-command-refs), no orphan docs, fmt clean. Produced by parallel rewrite agents under strict preserve-anchors/links/ code constraints, then verified + spot-read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9 * feat!: encode hook inject support into HookProtocol.injectableEvents Codex inject was documented as "deferred" but the official Codex hooks docs confirm `additionalContext` is honored on SessionStart / UserPromptSubmit / PreToolUse / PostToolUse / SubagentStart — the same shape as Claude Code. The gap existed because inject support was a prose deferral, not a tested contract: nothing in the adapter conformance kit asserted a shellHooks harness could actually deliver an inject hook. Encode it as a required port field so the gap can't recur silently: - HookProtocol gains `injectableEvents: readonly string[]` — the events that honor `additionalContext` injection, per-adapter. - claudeCodeHookProtocol: [SessionStart, UserPromptSubmit, PostToolUse]; codexHookProtocol: + [PreToolUse, SubagentStart]. - Conformance rejects a shellHooks adapter with an empty list; the adapter-contract suite asserts every shellHooks adapter declares the events vigiles' shipped hooks use (PostToolUse, SessionStart) — the test that would have caught the original miss. - `compile` warns on an inject hook only when its event isn't injectable on the target harness (react output stays CC-confirmed-only). - Correct every "Codex inject deferred" reference across docs, research, and CLAUDE.md; document injectableEvents in the adapter API + authoring guides. BREAKING CHANGE: HookProtocol (exported on vigiles/adapter) gains a required `injectableEvents` field — a third-party adapter must declare it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF for the eval-lock branch + inject-port encoding Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: fan out hook install + the CI workflow to every declared harness A repo can enable BOTH harnesses (`harness: ["claude-code","codex"]`), and instruction files already mirror to both — but `compile` installed a hook into only ONE harness (`installHooks` ignored `config.harness` and auto-detected a single adapter), and the scaffolded CI workflow hardcoded the Claude Code binary. So an inject/gate hook that works on both could land in `.claude/settings.json` while silently missing from `.codex/config.toml` — the "works on CC, absent on Codex" failure the harness-parity rule exists to prevent. - `resolveHarnessAdapters` (adapter-registry.ts) returns the FULL declared set: `--harness=` flag → that one; else config `harness` list → all of them; else auto-detect → the detected one (de-duped by name). - `installHooks` loops over it, merging the SAME compiled hook into every enabled harness's native config (JSON / TOML), with per-harness warnings. - The `init` CI workflow is harness-aware: the deterministic harness job installs `@anthropic-ai/claude-code` and/or `@openai/codex` per declared harness (`harnessTestBinaries`). Skills compile byte-identical across harnesses and subagents are CC-only by design, so hooks + the CI binary were the real fan-out gaps. Tests: resolveHarnessAdapters units (fan-out / flag-singular / detect / alias-dedup), a both-harness hook-install e2e (the same hook in both configs), and harness-aware workflow e2es (codex-only, both). Documented as §4 in research/multi-harness-compile.md + the compiled-hooks guide. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: wire vigiles's nudge hooks into .codex/config.toml during init Closes the last Codex-parity gap in agent-awareness: Claude Code gets the eval-lock + refs nudge hooks from the global marketplace plugin, but Codex has no global plugin store, so it got skills only — the proactive nudges never reached a Codex user (the "works on CC, silently absent on Codex" failure). Codex config is repo-committed (the idiomatic place), so `init` now wires the two PostToolUse nudges into `.codex/config.toml`: - `codexPluginHooks` (pure) declares them as DIRECT `npx vigiles hook-runtime …` commands — no plugin root, no vendored bash script. The runtime entrypoints read the event JSON on stdin and emit the `additionalContext` shape Codex honors on PostToolUse (confirmed + encoded in HookProtocol.injectableEvents). - `applyCodexPluginHooks` (pure, unit-tested) merges them idempotently: re-runs replace in place, and the user's own Codex hooks + every other config key are preserved. `wireCodexHooks` in cli.ts does only the IO. - Safety: only an INTENTIONAL exit 2 blocks an edit (the refs nudge under `unmarked-refs: error`); an npx-resolution failure exits non-2, so a missing dep never blocks. Loud, documented deferral (no-silent-skips): the SessionStart lint summary (plain stdout, unconfirmed on Codex) and the compile-on-edit / pre-edit guards (filename-gated bash, no harness-neutral entrypoint yet) stay manual on Codex. Also: the remaining pre-release Codex item — running the eval tier against the real `codex` binary once — is added to the launch-readiness P0 list in the roadmap. Tests: codexPluginHooks shape + applyCodexPluginHooks (fresh / idempotent / preserves user config); the planPluginInstall codex notes/vendors updated. Verified end-to-end that `init --harness=codex` writes the config and a pre-existing `.codex/config.toml` (model + a user hook) is preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: update codex install e2e for the new vendors:true (repo hooks) The codex install plan now writes nudge hooks into .codex/config.toml, so `vendors` flipped to true. Updated the unit assertion last commit but missed this e2e (run by both the `e2e` and `test` CI jobs). The skills-are-global checks below are unchanged and still stand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: cover the non-Actions lock message branch + address Codex review The `test` job's coverage gate failed: `emitLockMessage`'s console.warn/ console.log branches (the non-GITHUB_ACTIONS path) are unreachable in CI, where GITHUB_ACTIONS is globally set. Added a wiring test that DELETES the var to exercise both branches regardless of environment. Also addresses two Codex review findings on the eval-lock work: - P1 (soundness): `evalArmsInputs` omitted `spec.stubs` from the hash, though tool stubs are written onto PATH and are model-facing input — so a changed canned CLI output after `--update` left the same `inputsHash` and `--check` replayed stale results. Now folds a canonical (name-sorted) view of the stubs into the hash; a new wiring test proves a stub change → stale. - P2 (collision): two eval names normalizing to the same slug (`foo/bar` vs `foo bar` → `foo-bar.lock.json`) could let `--check` replay the WRONG eval's report. `readLock` now rejects a lock whose stored `name` differs from the requested one, degrading a collision to a safe miss (→ stale/re-run), never wrong data. Unit-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: two more eval-lock soundness gaps from Codex review - An unnamed eval in `--check` mode called `produce()` (ran the model), breaking the no-model-in-CI contract: in CI there's no subscription, so it would hit the model or fail for missing auth. Now `check` mode throws a clear, actionable error for an unnamed eval (never reaching the model); `off`/`update` keep producing (the eval just isn't gated locally). - `spec.ephemeralEnv` toggles the trial's run environment (scrubbed env + throwaway HOME vs the inherited process env) — a model-facing input that was omitted from the lock hash, so a report recorded under one mode could be replayed under the other with the same inputsHash. Now folded into the hashed inputs. Both proven by new wiring tests (unnamed-check throws without calling the runner; an ephemeralEnv toggle → stale). Coverage holds at 100% lines/ statements/functions (verified with GITHUB_ACTIONS=true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fold the harness identity into the trigger-rate lock hash Codex review (P2): a trigger-rate eval recorded with one harness driver and later switched via `measureTriggerRate(spec, { evalDriver: codexEvalDriver })` hashed identically (prompts/model/plugin unchanged), so `--check` could replay a Claude-recorded report for a Codex run — defeating the gate for the Codex eval path this PR adds. `EvalDriver` gains an optional `harness` identity (defaults to "claude-code", so existing single-harness locks are unaffected); `claudeEvalDriver` and `codexEvalDriver` declare theirs, and `measureTriggerRate`/`...With` thread it into the hashed trigger inputs. A new wiring test proves a harness switch → stale. API surface regenerated (additive `harness?` on EvalDriver). This closes the input-hash bug class on the trigger-rate seam (the runEval seam's stubs + ephemeralEnv were fixed in the prior two commits). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — PR #53 merge loop + the 5 review fixes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: three more Codex-review findings (matchers + plugin-root path hash) - Codex nudge hooks used Claude tool names (`Edit|Write`) in their matcher, but Codex's edit tool is `apply_patch` — so the eval-lock + refs nudges wired into `.codex/config.toml` would never fire. Matcher → `^apply_patch$`. - The compiled-hook FAN-OUT to Codex regex-wraps the raw matcher without translating tool names, so a CC-authored `tools("Edit","Write")` gate emits `^(Edit|Write)$` into Codex config and silently never fires though compile reports success. vigiles has no cross-dialect tool-name map yet, so install now WARNS LOUDLY when a tool-matched hook is fanned to a non-CC harness (no silent non-firing success); a real mapping is the follow-up. - For eval arms using `plugin`, `resolveHarness` expands `${PLUGIN_ROOT}` to the checkout's ABSOLUTE path before the settings are hashed, so a lock recorded at `/home/dev/...` was falsely STALE when `--check` recomputed it at `/home/runner/...` in CI. `stripPluginRoot` normalizes the prefix back to a token → location-independent hash. New wiring test: the same plugin at two different abs paths replays (not stale). Coverage holds at 100% lines/statements/functions (verified GITHUB_ACTIONS=true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: guard stripPluginRoot against a hookless plugin's undefined settings A plugin that ships only instructions/skills (no hooks) resolves to settings:undefined; JSON.stringify(undefined) is undefined, so the prior .split() threw before the eval could run. Guard the non-string case (pass it through). Regression I introduced in the plugin-root normalization last commit; new test covers a hookless-plugin arm (update + check, no throw). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: add a lock-hash COMPLETENESS guard for both seams The Codex review kept finding the same class of bug: a model-facing input omitted from the lock hash, so `--check` replays stale results (stubs, ephemeralEnv, driver-harness, plugin-root all slipped through). Our tests proved the MECHANISM (record→replay, task-change→stale) but never asserted the input set was COMPLETE. This adds a structural guard: for each input that steers the run, mutate ONLY it and assert the lock inputsHash CHANGES — one parametrized test per seam (runEval's evalArmsInputs + trigger-rate's triggerInputs). A new input added to EvalSpec/TriggerRateSpec without hashing it now fails a row HERE in CI, instead of a reviewer finding it later. Covers task/model/tools/fixture/ arm.files/arm.settings/arm.model/interceptTools/stubs/ephemeralEnv and prompts/irrelevantPrompts/model/harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
5.8 KiB
Agent Workflows
vigiles is low-friction by design. Run npx vigiles init and the installed skills and hooks handle the rest — auto-compiling specs, blocking stray edits, and nudging the agent when something needs attention. This guide shows the per-agent setup.
→ Back to README
vigiles verifies the rule references in agent instruction files — declared as inline comments, vigiles: YAML frontmatter, or a typed spec compiled to markdown (markdown mode). Different AI agents read different files, but the validation pipeline is the same. The workflows below use spec mode, the deepest level; the inline and frontmatter levels need no build step.
Contents
- Auto-Detection
- Claude Code
- Codex / GitHub Copilot
- Multi-Agent (Claude + Codex)
- Cursor / Windsurf / Other Formats
- CI Pipeline
- See also
Auto-Detection
vigiles init scans your project and auto-detects which agents you're already using — no --target flag needed unless you want to override.
| Signal | What it means |
|---|---|
CLAUDE.md exists |
Claude Code in use — suggest migration if no spec |
AGENTS.md exists |
Codex / GitHub Copilot in use |
.claude/ directory |
Claude Code project config |
.cursorrules |
Cursor in use — suggest rule-porter |
.github/copilot-instructions.md |
GitHub Copilot custom instructions |
.windsurfrules |
Windsurf in use |
rule-porter / rulesync in package.json |
Sync tool already installed |
| Symlinked instruction files | Notes them in output |
The wizard creates specs for detected targets, generates types, compiles, and adds a CI step.
Claude Code
Instruction file: CLAUDE.md
Setup:
npx vigiles init
# init installs the plugin via the marketplace; to do it by hand in Claude Code:
# /plugin marketplace add zernie/vigiles
# /plugin install vigiles@vigiles
What the plugin does once installed — the agent no longer needs to remember to compile:
| Hook | Trigger | Action |
|---|---|---|
| PreToolUse | Agent tries to Edit/Write a compiled .md file |
Blocks the edit, redirects to .spec.ts |
| PostToolUse | Agent edits a .spec.ts file |
Auto-runs vigiles compile |
| PostToolUse | Agent edits linter config or package.json |
Auto-runs vigiles generate types |
⚠️ Without the plugin, you must run vigiles compile manually after editing specs. CI still catches stale files.
Codex / GitHub Copilot
Instruction file: AGENTS.md
Setup:
npx vigiles init --target=AGENTS.md
Codex and GitHub Copilot read AGENTS.md directly. There is no plugin or hook system — these agents don't support it. The enforcement path is:
- Edit
AGENTS.md.spec.ts(the source of truth) - Run
npx vigiles compileto regenerateAGENTS.md - CI verifies freshness:
npx vigiles lint && npx vigiles generate types --check
Authoring skills for Codex install globally via the cross-agent skills CLI — no repo vendoring: npx skills add zernie/vigiles -a codex -g -y installs into ~/.agents/skills/. vigiles init --harness=codex runs this automatically. Codex hooks (.codex/config.toml [hooks]) aren't auto-wired yet.
ℹ️ If you also use Claude Code, install the plugin (/plugin marketplace add zernie/vigiles then /plugin install vigiles@vigiles, or vigiles init) for auto-recompilation.
Multi-Agent (Claude + Codex)
Use a single spec with multiple targets — one source of truth, two outputs:
export default claude({
target: ["CLAUDE.md", "AGENTS.md"],
rules: { ... },
});
Both files compile from the same spec with the same linter verification.
npx vigiles init # for CLAUDE.md (primary)
npx vigiles init --target=AGENTS.md # adds AGENTS.md target
Or just set target: ["CLAUDE.md", "AGENTS.md"] in your spec directly.
Cursor / Windsurf / Other Formats
vigiles compiles to markdown only (CLAUDE.md, AGENTS.md). For non-markdown formats (.cursorrules, .github/copilot-instructions.md, Windsurf), use a sync tool to convert from the compiled markdown:
- rule-porter — bidirectional conversion between agent formats
- rulesync — unified rule management across 10+ tools
vigiles is the source of truth compiler. Sync tools handle the last mile.
CI Pipeline
All agents share the same CI step:
- name: Verify specs
run: npx vigiles lint && npx vigiles generate types --check
This catches:
- Hash mismatches — someone edited the compiled
.mddirectly - Missing specs —
require-instructions-specrule requires a.spec.tsbehind every.md - Stale generated types — linter config changed but types weren't regenerated
See also
- Agent Setup — non-interactive installation and recommended agent prompt
- Markdown mode — inline comments and frontmatter (no
.spec.tsrequired) - CLI reference