Files
zernie__vigiles/docs/agent-workflows.md
T
zernie 15c0607a5b feat!: eval lock (CI staleness gate) + full Codex parity for hooks & inject (#53)
* feat!: eval lock (CI staleness gate) + per-adapter harness versionKey

Real-model evals run on your subscription, so they only run locally — never
in CI. The eval LOCK closes the resulting gap: a committed integrity stamp
that lets CI verify the committed numbers still match the current inputs
WITHOUT a model call (distinct from the gitignored, local-speed eval cache).

- `vigiles eval --update` (local): records each NAMED eval's report to a
  committed `.vigiles/eval-locks/<name>.lock.json` + prints the per-number delta.
- `vigiles eval --check` (CI): recompute the input hash, fail "stale" on a
  mismatch — binary-free, deterministic. Green no-op until the first lock is
  committed (smooth adoption).
- The clean split: the lock stores only the model's observed behavior; the
  script's own assertions re-run live against the replayed report, so a
  threshold-only edit is a valid replay while an input change is stale.
- Honest scope: the harness version is provenance, NOT a hash input (CI's
  pinned claude ≠ a dev's local claude would false-trip), and there is no
  nightly run — model/harness drift is a local `--update` concern.

Also extracts the harness version reduction to a per-adapter port method
(`HarnessRuntime.versionKey`): Claude Code keys major.minor (~quarterly),
Codex opts out with "" (its minor is patch-cadence, ~weekly). This fixes the
local cache churning weekly for Codex and is where the cadence decision belongs.

Whole-package coherence: the GHA gains `command: eval-check`, `vigiles init`
scaffolds the eval-check CI job, and this repo dogfoods it. Wired at both eval
seams (runEvalWith + measureTriggerRateWith), 100%-covered, docs updated
(internal research + public guides).

BREAKING CHANGE: `HarnessRuntime` (exported on `vigiles/adapter`) gains a
required `versionKey(raw: string): string` method. Third-party adapters must
add it; return "" to opt out of version partitioning. First-party adapters
(claude-code, codex) are updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* feat: eval-lock agent awareness (skill + nudge hook) + cohesive-delivery rule

Close the loop on the eval lock so the AGENT knows it exists and maintains it —
without the invasive option of editing the user's CLAUDE.md.

- test-harness skill documents the lock lifecycle (--update/--check, commit the
  lock, re-run after editing a skill).
- A PostToolUse nudge hook (hooks/eval-lock-nudge.sh → hook-runtime
  eval-lock-nudge): after editing a SKILL.md / *.eval.* with committed locks
  present, inject a NON-BLOCKING reminder to re-run `vigiles eval --update`.
  Self-gated (silent until a lock is committed), never blocks (the gate is CI's
  `eval --check`). Wired into the vigiles plugin; runHook + unit + .sh tests.
- NO init edit to the user's CLAUDE.md — verified that's not how plugins deliver
  context (no plugin-level instructions file exists; the channel is a hook/skill).

Docs + dedup:
- research/agent-context-delivery.md captures the VERIFIED CC/Codex
  context-delivery mechanics ONCE (SessionStart additionalContext is the plugin
  channel; CLAUDE.md is the user's file; skill descriptions always-in-context;
  Codex AGENTS.md + hooks but inject deferred) so we stop re-researching it.
- Per-adapter guides (harness-testing-claude-code/-codex.md) explain how the
  reminder reaches the agent per harness (CC injects; Codex inject is a documented
  follow-on), with the agnostic guide linking DOWN to them (no duplication).

Cohesive-view rule:
- New CLAUDE.md `cohesive-feature-delivery` rule — a definition-of-done CHECKLIST
  (CLI × GHA × init × agent-awareness × both adapters × per-adapter deduped docs ×
  tests) so a feature's INTEGRATION across surfaces is checked as one thing,
  generalizing cohesive-cli-surface + harness-parity + great-agent-flow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* test: CLI e2e for the eval-lock invalidation flow (update → check → stale)

Closes the gap flagged in review: the joined CLI flow wasn't tested as one
thing. A real *.eval.cjs fixture (model-free — drives runEvalWith from dist with
a fake runner) run through the built CLI proves the whole loop: `eval --update`
writes a committed lock, `eval --check` with the same inputs replays green (no
model), and an input change makes `eval --check` go red ("stale"). Binary-free,
no subscription — exactly how it runs in CI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* docs: scannable eval-lock docs + prose-clarity & easy-adoption rules

Make the eval-lock docs readable at a glance and frame upkeep as agent-driven,
not a manual chore.

Docs (scannable rewrite — ASCII flow diagram, comparison tables, //⚠️
signposts, short sentences, no walls of text):
- docs/harness-testing.md — the lock how-to now leads with "you mostly won't
  touch this by hand; the test-harness skill records it + a hook reminds you,"
  then a diagram + tables.
- docs/harness-testing-claude-code.md / -codex.md — the per-harness nudge
  delivery as tables; the Codex inject caveat is one clear ⚠️ row.
- docs/eval-architecture.md — the "what's stale vs a free replay" split as
  / bullets.
- skills/test-harness/SKILL.md — Step 5 now tells the agent to run
  `eval --update` + commit the lock as part of its job (so it IS done for you).
- README — the Quick-start reassurance leads with "Adoption is smooth: one
  command, then your agent does the rest," names skills AND hooks, no chores.

Two new CLAUDE.md rules:
- prose-clarity — public prose must be clear + scannable (lead with the point,
  one idea per sentence, define/drop jargon, tables/bullets/ASCII/Mermaid over
  walls, emoji as signposts, one bold per point). Prefer ASCII/Mermaid; never
  commit generated raster images into docs.
- lead-with-easy-adoption — docs (README first screen especially) must make it
  clear up front that adoption is low-friction because skills + hooks do the
  work; frame every feature's upkeep as agent/skill/hook-driven, not homework.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* docs: prose-clarity sweep across 13 user-facing guides

Apply the new prose-clarity rule to the prose-heavy public guides: lead with the
point, one idea per sentence, tables/bullets/ASCII over walls of text, //⚠️/ℹ️
signposts, one bold phrase per point, jargon glossed or dropped. Net −578 lines —
shorter and far more scannable, with meaning unchanged.

Swept: for-plugin-authors, what-vigiles-catches, measuring-skills, skills,
harnesses, comparison, faq, sandboxing, safety, agent-setup, agent-workflows,
verifying-instruction-files, compiled-hooks. The onboarding pair leads with the
low-friction "one command, the agent does the rest" framing.

Reference docs (cli, testing-api, spec-format, …) and the per-rule docs were
deliberately left alone — they're already table-shaped, and churning them is
high-risk / low-value.

Verified: no heading text changed (anchors intact), no `research/` links added to
public docs, the validation-rules matrix is complete (28 rule links → 28), safety
claims retained, command refs resolve (self-command-refs), no orphan docs, fmt
clean. Produced by parallel rewrite agents under strict preserve-anchors/links/
code constraints, then verified + spot-read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WVmD8n7P62fuJ91aQh9yY9

* feat!: encode hook inject support into HookProtocol.injectableEvents

Codex inject was documented as "deferred" but the official Codex hooks
docs confirm `additionalContext` is honored on SessionStart /
UserPromptSubmit / PreToolUse / PostToolUse / SubagentStart — the same
shape as Claude Code. The gap existed because inject support was a prose
deferral, not a tested contract: nothing in the adapter conformance kit
asserted a shellHooks harness could actually deliver an inject hook.

Encode it as a required port field so the gap can't recur silently:

- HookProtocol gains `injectableEvents: readonly string[]` — the events
  that honor `additionalContext` injection, per-adapter.
- claudeCodeHookProtocol: [SessionStart, UserPromptSubmit, PostToolUse];
  codexHookProtocol: + [PreToolUse, SubagentStart].
- Conformance rejects a shellHooks adapter with an empty list; the
  adapter-contract suite asserts every shellHooks adapter declares the
  events vigiles' shipped hooks use (PostToolUse, SessionStart) — the
  test that would have caught the original miss.
- `compile` warns on an inject hook only when its event isn't injectable
  on the target harness (react output stays CC-confirmed-only).
- Correct every "Codex inject deferred" reference across docs, research,
  and CLAUDE.md; document injectableEvents in the adapter API + authoring
  guides.

BREAKING CHANGE: HookProtocol (exported on vigiles/adapter) gains a
required `injectableEvents` field — a third-party adapter must declare it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF for the eval-lock branch + inject-port encoding

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: fan out hook install + the CI workflow to every declared harness

A repo can enable BOTH harnesses (`harness: ["claude-code","codex"]`),
and instruction files already mirror to both — but `compile` installed a
hook into only ONE harness (`installHooks` ignored `config.harness` and
auto-detected a single adapter), and the scaffolded CI workflow hardcoded
the Claude Code binary. So an inject/gate hook that works on both could
land in `.claude/settings.json` while silently missing from
`.codex/config.toml` — the "works on CC, absent on Codex" failure the
harness-parity rule exists to prevent.

- `resolveHarnessAdapters` (adapter-registry.ts) returns the FULL declared
  set: `--harness=` flag → that one; else config `harness` list → all of
  them; else auto-detect → the detected one (de-duped by name).
- `installHooks` loops over it, merging the SAME compiled hook into every
  enabled harness's native config (JSON / TOML), with per-harness warnings.
- The `init` CI workflow is harness-aware: the deterministic harness job
  installs `@anthropic-ai/claude-code` and/or `@openai/codex` per declared
  harness (`harnessTestBinaries`).

Skills compile byte-identical across harnesses and subagents are CC-only
by design, so hooks + the CI binary were the real fan-out gaps.

Tests: resolveHarnessAdapters units (fan-out / flag-singular / detect /
alias-dedup), a both-harness hook-install e2e (the same hook in both
configs), and harness-aware workflow e2es (codex-only, both). Documented
as §4 in research/multi-harness-compile.md + the compiled-hooks guide.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: wire vigiles's nudge hooks into .codex/config.toml during init

Closes the last Codex-parity gap in agent-awareness: Claude Code gets the
eval-lock + refs nudge hooks from the global marketplace plugin, but Codex
has no global plugin store, so it got skills only — the proactive nudges
never reached a Codex user (the "works on CC, silently absent on Codex"
failure).

Codex config is repo-committed (the idiomatic place), so `init` now wires
the two PostToolUse nudges into `.codex/config.toml`:

- `codexPluginHooks` (pure) declares them as DIRECT `npx vigiles
  hook-runtime …` commands — no plugin root, no vendored bash script. The
  runtime entrypoints read the event JSON on stdin and emit the
  `additionalContext` shape Codex honors on PostToolUse (confirmed +
  encoded in HookProtocol.injectableEvents).
- `applyCodexPluginHooks` (pure, unit-tested) merges them idempotently:
  re-runs replace in place, and the user's own Codex hooks + every other
  config key are preserved. `wireCodexHooks` in cli.ts does only the IO.
- Safety: only an INTENTIONAL exit 2 blocks an edit (the refs nudge under
  `unmarked-refs: error`); an npx-resolution failure exits non-2, so a
  missing dep never blocks.

Loud, documented deferral (no-silent-skips): the SessionStart lint summary
(plain stdout, unconfirmed on Codex) and the compile-on-edit / pre-edit
guards (filename-gated bash, no harness-neutral entrypoint yet) stay manual
on Codex.

Also: the remaining pre-release Codex item — running the eval tier against
the real `codex` binary once — is added to the launch-readiness P0 list in
the roadmap.

Tests: codexPluginHooks shape + applyCodexPluginHooks (fresh / idempotent /
preserves user config); the planPluginInstall codex notes/vendors updated.
Verified end-to-end that `init --harness=codex` writes the config and a
pre-existing `.codex/config.toml` (model + a user hook) is preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: update codex install e2e for the new vendors:true (repo hooks)

The codex install plan now writes nudge hooks into .codex/config.toml, so
`vendors` flipped to true. Updated the unit assertion last commit but missed
this e2e (run by both the `e2e` and `test` CI jobs). The skills-are-global
checks below are unchanged and still stand.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: cover the non-Actions lock message branch + address Codex review

The `test` job's coverage gate failed: `emitLockMessage`'s console.warn/
console.log branches (the non-GITHUB_ACTIONS path) are unreachable in CI,
where GITHUB_ACTIONS is globally set. Added a wiring test that DELETES the
var to exercise both branches regardless of environment.

Also addresses two Codex review findings on the eval-lock work:

- P1 (soundness): `evalArmsInputs` omitted `spec.stubs` from the hash, though
  tool stubs are written onto PATH and are model-facing input — so a changed
  canned CLI output after `--update` left the same `inputsHash` and `--check`
  replayed stale results. Now folds a canonical (name-sorted) view of the
  stubs into the hash; a new wiring test proves a stub change → stale.

- P2 (collision): two eval names normalizing to the same slug (`foo/bar` vs
  `foo bar` → `foo-bar.lock.json`) could let `--check` replay the WRONG eval's
  report. `readLock` now rejects a lock whose stored `name` differs from the
  requested one, degrading a collision to a safe miss (→ stale/re-run), never
  wrong data. Unit-tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: two more eval-lock soundness gaps from Codex review

- An unnamed eval in `--check` mode called `produce()` (ran the model),
  breaking the no-model-in-CI contract: in CI there's no subscription, so it
  would hit the model or fail for missing auth. Now `check` mode throws a
  clear, actionable error for an unnamed eval (never reaching the model);
  `off`/`update` keep producing (the eval just isn't gated locally).

- `spec.ephemeralEnv` toggles the trial's run environment (scrubbed env +
  throwaway HOME vs the inherited process env) — a model-facing input that
  was omitted from the lock hash, so a report recorded under one mode could
  be replayed under the other with the same inputsHash. Now folded into the
  hashed inputs.

Both proven by new wiring tests (unnamed-check throws without calling the
runner; an ephemeralEnv toggle → stale). Coverage holds at 100% lines/
statements/functions (verified with GITHUB_ACTIONS=true).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: fold the harness identity into the trigger-rate lock hash

Codex review (P2): a trigger-rate eval recorded with one harness driver and
later switched via `measureTriggerRate(spec, { evalDriver: codexEvalDriver })`
hashed identically (prompts/model/plugin unchanged), so `--check` could
replay a Claude-recorded report for a Codex run — defeating the gate for the
Codex eval path this PR adds.

`EvalDriver` gains an optional `harness` identity (defaults to "claude-code",
so existing single-harness locks are unaffected); `claudeEvalDriver` and
`codexEvalDriver` declare theirs, and `measureTriggerRate`/`...With` thread it
into the hashed trigger inputs. A new wiring test proves a harness switch →
stale. API surface regenerated (additive `harness?` on EvalDriver).

This closes the input-hash bug class on the trigger-rate seam (the runEval
seam's stubs + ephemeralEnv were fixed in the prior two commits).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF — PR #53 merge loop + the 5 review fixes

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: three more Codex-review findings (matchers + plugin-root path hash)

- Codex nudge hooks used Claude tool names (`Edit|Write`) in their matcher,
  but Codex's edit tool is `apply_patch` — so the eval-lock + refs nudges
  wired into `.codex/config.toml` would never fire. Matcher → `^apply_patch$`.

- The compiled-hook FAN-OUT to Codex regex-wraps the raw matcher without
  translating tool names, so a CC-authored `tools("Edit","Write")` gate emits
  `^(Edit|Write)$` into Codex config and silently never fires though compile
  reports success. vigiles has no cross-dialect tool-name map yet, so install
  now WARNS LOUDLY when a tool-matched hook is fanned to a non-CC harness
  (no silent non-firing success); a real mapping is the follow-up.

- For eval arms using `plugin`, `resolveHarness` expands `${PLUGIN_ROOT}` to
  the checkout's ABSOLUTE path before the settings are hashed, so a lock
  recorded at `/home/dev/...` was falsely STALE when `--check` recomputed it
  at `/home/runner/...` in CI. `stripPluginRoot` normalizes the prefix back
  to a token → location-independent hash. New wiring test: the same plugin at
  two different abs paths replays (not stale).

Coverage holds at 100% lines/statements/functions (verified GITHUB_ACTIONS=true).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: guard stripPluginRoot against a hookless plugin's undefined settings

A plugin that ships only instructions/skills (no hooks) resolves to
settings:undefined; JSON.stringify(undefined) is undefined, so the prior
.split() threw before the eval could run. Guard the non-string case (pass it
through). Regression I introduced in the plugin-root normalization last commit;
new test covers a hookless-plugin arm (update + check, no throw).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: add a lock-hash COMPLETENESS guard for both seams

The Codex review kept finding the same class of bug: a model-facing input
omitted from the lock hash, so `--check` replays stale results (stubs,
ephemeralEnv, driver-harness, plugin-root all slipped through). Our tests
proved the MECHANISM (record→replay, task-change→stale) but never asserted
the input set was COMPLETE.

This adds a structural guard: for each input that steers the run, mutate
ONLY it and assert the lock inputsHash CHANGES — one parametrized test per
seam (runEval's evalArmsInputs + trigger-rate's triggerInputs). A new input
added to EvalSpec/TriggerRateSpec without hashing it now fails a row HERE in
CI, instead of a reviewer finding it later. Covers task/model/tools/fixture/
arm.files/arm.settings/arm.model/interceptTools/stubs/ephemeralEnv and
prompts/irrelevantPrompts/model/harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-01 01:55:49 +05:00

5.8 KiB
Raw Blame History

Agent Workflows

vigiles is low-friction by design. Run npx vigiles init and the installed skills and hooks handle the rest — auto-compiling specs, blocking stray edits, and nudging the agent when something needs attention. This guide shows the per-agent setup.

→ Back to README

vigiles verifies the rule references in agent instruction files — declared as inline comments, vigiles: YAML frontmatter, or a typed spec compiled to markdown (markdown mode). Different AI agents read different files, but the validation pipeline is the same. The workflows below use spec mode, the deepest level; the inline and frontmatter levels need no build step.

Contents

Auto-Detection

vigiles init scans your project and auto-detects which agents you're already using — no --target flag needed unless you want to override.

Signal What it means
CLAUDE.md exists Claude Code in use — suggest migration if no spec
AGENTS.md exists Codex / GitHub Copilot in use
.claude/ directory Claude Code project config
.cursorrules Cursor in use — suggest rule-porter
.github/copilot-instructions.md GitHub Copilot custom instructions
.windsurfrules Windsurf in use
rule-porter / rulesync in package.json Sync tool already installed
Symlinked instruction files Notes them in output

The wizard creates specs for detected targets, generates types, compiles, and adds a CI step.

Claude Code

Instruction file: CLAUDE.md

Setup:

npx vigiles init
# init installs the plugin via the marketplace; to do it by hand in Claude Code:
#   /plugin marketplace add zernie/vigiles
#   /plugin install vigiles@vigiles

What the plugin does once installed — the agent no longer needs to remember to compile:

Hook Trigger Action
PreToolUse Agent tries to Edit/Write a compiled .md file Blocks the edit, redirects to .spec.ts
PostToolUse Agent edits a .spec.ts file Auto-runs vigiles compile
PostToolUse Agent edits linter config or package.json Auto-runs vigiles generate types

⚠️ Without the plugin, you must run vigiles compile manually after editing specs. CI still catches stale files.

Codex / GitHub Copilot

Instruction file: AGENTS.md

Setup:

npx vigiles init --target=AGENTS.md

Codex and GitHub Copilot read AGENTS.md directly. There is no plugin or hook system — these agents don't support it. The enforcement path is:

  1. Edit AGENTS.md.spec.ts (the source of truth)
  2. Run npx vigiles compile to regenerate AGENTS.md
  3. CI verifies freshness: npx vigiles lint && npx vigiles generate types --check

Authoring skills for Codex install globally via the cross-agent skills CLI — no repo vendoring: npx skills add zernie/vigiles -a codex -g -y installs into ~/.agents/skills/. vigiles init --harness=codex runs this automatically. Codex hooks (.codex/config.toml [hooks]) aren't auto-wired yet.

If you also use Claude Code, install the plugin (/plugin marketplace add zernie/vigiles then /plugin install vigiles@vigiles, or vigiles init) for auto-recompilation.

Multi-Agent (Claude + Codex)

Use a single spec with multiple targets — one source of truth, two outputs:

export default claude({
  target: ["CLAUDE.md", "AGENTS.md"],
  rules: { ... },
});

Both files compile from the same spec with the same linter verification.

npx vigiles init                      # for CLAUDE.md (primary)
npx vigiles init --target=AGENTS.md   # adds AGENTS.md target

Or just set target: ["CLAUDE.md", "AGENTS.md"] in your spec directly.

Cursor / Windsurf / Other Formats

vigiles compiles to markdown only (CLAUDE.md, AGENTS.md). For non-markdown formats (.cursorrules, .github/copilot-instructions.md, Windsurf), use a sync tool to convert from the compiled markdown:

  • rule-porter — bidirectional conversion between agent formats
  • rulesync — unified rule management across 10+ tools

vigiles is the source of truth compiler. Sync tools handle the last mile.

CI Pipeline

All agents share the same CI step:

- name: Verify specs
  run: npx vigiles lint && npx vigiles generate types --check

This catches:

  • Hash mismatches — someone edited the compiled .md directly
  • Missing specsrequire-instructions-spec rule requires a .spec.ts behind every .md
  • Stale generated types — linter config changed but types weren't regenerated

See also