vigiles

You have a rule your agent follows half the time — and no way to know which one.

Verify your CLAUDE.md or AGENTS.md, skills, and hooks are real — and prove they actually work.

---

You review every PR. Nothing reviews your CLAUDE.md.

Your harness — the CLAUDE.md or AGENTS.md rules, skills, subagents, and hooks steering your agent — is the one part nobody checks. Nobody verified it's real. Nobody tested it works. That's not a system. That's vibes.

And vibes break silently mid-task: a subagent wired to a tool that doesn't exist, two skills your agent can't tell apart, one helper quietly able to read your secrets and send them out.

vigiles¹ checks your harness is real, not just well-formed — Claude Code and Codex alike. One command, no key, no config, safe on any repo:

npx vigiles audit

It's free and open-source, runs entirely on your machine, and never bills per token. (eval is the only step that calls a model — on your own Claude subscription.) Here's what it caught on plugins people actually ship. ↓

What it caught

Like Google's Lighthouse, but for your agent harness. One command grades it A–F across five categories, every fix shown inline:

• Truthfulness — do the references resolve?
• Triggering — do skills fire, without colliding?
• Structure — are tool contracts and configs valid?
• Safety — any way for the agent to leak your data?
• Tested — does the harness ship tests?

These are real scans of public plugins — run npx vigiles audit <any-repo> for your own. The examples below use Claude Code subagents; the same checks run on Codex AGENTS.md, skills, and hooks. ↓

Proof 1 — a tool your agent thinks it has and doesn't

✗ tester — Tool "AskUserQuestion" is never available to a subagent.
    → remove or correct it — it's silently dropped from the contract.

This subagent — a helper your main agent hands work to — lists a tool that doesn't exist for it. The harness drops it without a word, so the agent quietly loses a capability it thinks it has. The markdown is perfectly valid. vigiles catches it and hands you the one-line fix.

Proof 2 — two skills your agent can't tell apart

✗ Triggering   0  (0/100)
    └ 45 pairs of near-identical skill descriptions — the agent can't tell them
      apart, so the wrong one fires  (e.g. "agent-coder" ↔ "agent-tester", 83% alike)

One popular plugin ships 45 pairs of near-identical skill descriptions. Your agent picks a skill by reading them — so when two match, it fires the wrong one. Still perfectly valid markdown. How triggering works →

Proof 3 — it can quietly read your secrets and send them out

◑ Safety   80  (80/100)
    └ subagent "tester" holds all three lethal-trifecta legs:
        reads private data (Bash, Read) · takes in untrusted web content (WebFetch)
        · can send data out (Bash, WebFetch)

Hand one subagent all three powers and a poisoned web page can make it read your .env and POST it anywhere — no exploit code, just the tools it was given. The 80 looks like a B — and that's the trap: a healthy grade hiding a subagent that's a data-leak waiting to happen. vigiles spots it from the tool list alone, free, no model.

That's the whole idea: it checks your harness against reality, not style. Every tool, hook, file, script, and skill you reference is verified to actually resolve — and where you name a linter rule, it's checked to exist and be enabled (ESLint, Ruff, Clippy, and more). Everything it catches → · point audit at a whole marketplace and it ranks every plugin the same way.

How it works — vibes → verified

audit shows you where your setup is still vibes. Turning that into verified is four commands over one engine — and almost none of it needs a model or a key.

Command	Answers	Needs a model?	When to run
audit	Everything, graded A–F	No — read-only2	Anytime; it's the report
lint	Do the structural checks pass?	No	CI gate, every push
test	Does the harness behave?	No — a scripted stand-in	Every commit
eval	Does a skill actually help?	Yes — your subscription	On demand

One engine, two doors. audit is the local report; lint is the CI gate that fails the build on the same deterministic checks — broken refs, bad tool contracts, dead hooks, skill collisions (Proofs 1–2). test and eval go further: past does it exist to does it work. (init / compile / eject manage the spec layer underneath — you rarely run them by hand.)

🔎 Lint — your instructions stop lying

Every path, script, symbol, and rule verified against reality — plus tool contracts, skill collisions, and dead hooks (the catches above). You don't write the checks. npx vigiles init writes a CLAUDE.md.spec.ts beside your file: the same rules, each reference now wrapped so vigiles can confirm it exists. compile turns that back into the CLAUDE.md (or AGENTS.md) your agent already reads. Your agent edits the spec in plain English; eject deletes it and leaves your original untouched. How →

🧪 Test — does the harness actually do its job?

A hook that blocks nothing, a skill that hijacks unrelated prompts, context that never reaches the model — each passes a naive "did it run?" check. That gap is false confidence: a guard that looks like it works and silently doesn't. vigiles tests the real thing — hooks block, skills fire, subagents finish what they promised, a stray git push is caught before it happens. It drives a scripted stand-in for the model, not a live call, so it needs no key and runs on every commit. How testing works →

📊 Eval — the only way to put a real number on cost

"Caveman Mode cuts 65% of your tokens." Says who? vigiles A/Bs the claim on real coding tasks and hands you three numbers: the token bill, whether it hit its target, and whether your code still works.

caveman vs verbose · haiku · $0 on your subscription
  output tokens   762 → 842   (+11% — the "saving" reversed)
  correctness     1.0 → 1.0   (the fact survived)

Point it at any harness change that claims a number — does a compression skill pay for itself, is a subagent worth its cost, which model is cheapest here. promptfoo and DeepEval bill per token, every run; vigiles runs on your own Claude Pro/Max subscription, so you measure on every change, not once. A committed lock file (like package-lock) keeps CI honest without re-calling the model. (Claude Code today; Codex landing.) Measure a skill →

Quick start

1. See what's broken — read-only, no setup:

npx vigiles audit

2. Set it up when you like what you see. Paste into Claude Code or Codex:

Set up vigiles in this repo: run `npx vigiles init` and accept the defaults. If I
already have a CLAUDE.md or AGENTS.md, adopt it into a spec and show me which
references are stale. Then compile and write + run one harness test for a hook or
skill of mine. Don't run a real-model eval without asking me first.

Or run it yourself:

npx vigiles init   # adopts your files (non-destructive — eject reverses), adds CI,
                   # installs vigiles's skills + hooks as a Claude Code plugin (in
                   # ~/.claude/, not your repo). On Codex, skills install globally too.

Interactive in a terminal, non-interactive for agents/CI (or --yes). Works with Claude Code and Codex — vigiles verifies CLAUDE.md and AGENTS.md the same way. Codex setup →

Adoption is smooth: one command, then your agent does the rest. init installs the skills and hooks, so a plain-English ask does the work — no specs to hand-write, no hooks to wire:

• "test my skills" → scaffolds and runs a trigger/behaviour test, then commits its result so CI can check it (test-harness)
• "harden my rules" → upgrades prose guidance into enforced linter rules (strengthen)
• "add a rule to my CLAUDE.md or AGENTS.md" → edits the source and recompiles (edit-spec)

The hooks keep it honest in-loop — nudging the agent to tag a linter-rule mention so vigiles can verify it, or to re-run a test whose result just went stale — so there are no chores to remember.

What init sets up

• Both lint and test by default; scope with --lint / --test.
• Already have a CLAUDE.md / AGENTS.md, skills, or subagents? init adopts them all into specs faithfully and non-destructively — untouched until you compile (and eject undoes it).
• Adds vigiles to devDependencies; installs the Claude Code plugin (skills + hooks) via the marketplace — globally, never vendored.
• Wires CI as a zernie/vigiles@v1 workflow (needs only read + PR-comment permissions) that posts a sticky PR comment + a valid output.

Targets Claude Code and Codex out of the box, or your own harness. Prefer to write tests yourself? JS or TS (*.harness.{mjs,ts}) — run with npx vigiles test.

FAQ

• Is this a framework I have to build around? No. It's a tool you run — like ESLint, Lighthouse, or npm audit. One command, a report, an optional CI gate. There's a library API for automation, but you never touch it to get value.
• Isn't this just a markdown linter? No — it checks whether your instruction file is true (every path/script/symbol/rule exists and is enabled), then tests and measures your harness. A style linter can't do any of that.
• Do I have to write TypeScript? No — your agent writes the spec (init adopts your CLAUDE.md or AGENTS.md into one), or plain markdown lints with zero new files. Compiler-grade guarantees are opt-in, like TS's strict (why?).
• Is it stable enough to adopt? The CLI you run is small and rarely changes; the library API still moves between releases. The high version number is release automation (a new major per breaking change), not age — see Stability.
• Non-JS repo? npx vigiles lint verifies your CLAUDE.md or AGENTS.md with no install (Ruff/Clippy/Pylint/… too).

Full FAQ →

Not for you if you want a model/capability benchmark or runtime guardrails in the request path — vigiles is build-/CI-time.

Docs

The docs index is the full map, grouped by what you're doing:

• Guides — verify instruction files · test your harness · measure a skill · ship a plugin · Codex & other harnesses
• Reference — CLI · rules matrix · testing API · full API
• Explanation — what it catches · how it compares · FAQ

Project — Stability · Related tools · companion to Feedback Loop Is All You Need.

License

MIT

---

1. vigiles — the watchmen of ancient Rome, who guarded the city (and fought its fires) by night. Quis custodiet ipsos custodes? — "who watches the watchmen?" (Juvenal, Satire VI). ↩︎

2. audit reads only by default. Two deeper checks — live MCP connections and skill-firing — are opt-in and ask before they run. ↩︎