mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
feat: unified Trace + predicates, model-request capture, safe-by-default sandbox (#27)
* feat: unified Trace + predicate vocabulary
Both testing tiers now produce one Trace (toolCalls + output + turns + file):
a runHarnessTest result is a Trace, and runEval's measure ctx (RunContext
extends Trace) is too. Over it sits one set of bare predicates — usedTool,
toolCount, skillResolved, toolUsedWith — exported from harness-assert.ts; each
assertTool* is now a thin throw-wrapper over a predicate, and an eval measure
reuses the same bare predicates as metrics. Testing asserts, eval measures —
same vocabulary, separate consumers, never one dual-purpose function.
- harness-test.ts: Trace interface; parseResultEvent/parseOutput capture the
final answer from the stream both tiers already produce; result.output added.
- eval.ts: ctx gains toolCalls + output (eval switched to stream-json so the
per-turn tool_use events survive); pass^k (MetricStat.passK = succeeded every
trial) alongside mean ± se in the report and formatEvalReport.
- harness-assert.ts: bare predicates + assertToolUsedWith (tool-argument
assertion over toolCalls[].input); assert* re-typed to accept any Trace.
Deferred: trace.hooks (which hook fired + decision) — needs mock instrumentation
to record invocations rather than infer them from marker files.
* feat: output + hook + reliability verbs over Trace, native coverage
Completes the Trace vocabulary the unification opened and records hook firing.
- trace.hooks: parseHooks records which hook fired + its decision from the CLI's
hook_response stream events (hook_name/hook_event/exit_code/outcome) — honest,
not inferred from marker files. HookFire on Trace, populated by both tiers.
- predicates/asserts: hookFired/hookBlocked/assertHookFired (recorded hook
firing), outputContains/assertOutputContains (the agent's final answer),
reliable/assertReliable (pass^k gate — succeeded every trial).
- The two Edit/Write hook regression tests now also assert via trace.hooks
(transcript:true), so firing is checked from the stream, not just markers.
- coverage: npm run coverage uses node 22 native V8 coverage (no dep); ~86%
lines with claude present. npm test switched to glob discovery
(node --test "dist/**/*.test.js") so a new *.test.js can't be left out.
Docs + research matrix updated: trace.hooks and coverage tooling marked shipped.
* test: migrate the suite to Vitest (primary runner)
node:test was the zero-dep dogfood, but its DX (noisy TAP, weak watch/filter, no
coverage thresholds or inline ignores) doesn't pay off. Vitest runs the TS
sources directly (no pre-build for the unit bits), gives a readable reporter,
watch/filter, and V8 coverage with thresholds.
- vitest.config.mjs: two projects — `unit` (src/**/*.test.ts, with .js→.ts
extensionAlias so NodeNext imports resolve; 60s timeout to match node:test's
no-timeout default for catalog/CLI-spawning suites) and `runners` (the existing
cross-runner matcher constraint, loaded from dist as a user would).
- Mechanical per-file change: import { test|describe|it } from "node:test" →
"vitest"; before/after → beforeAll/afterAll via import alias (call sites
unchanged). node:assert/strict kept as-is (works under vitest).
- The runner-agnostic claim is still proven: harness-assert keeps its
node:test/jest matchers, and test/runners/ exercises them under vitest + jest.
- scripts: test → vitest run; test:vitest → the runners project; coverage →
vitest run --coverage (V8). Coverage is scoped/threshold-gated to the
harness-testing pillar (driven to 100% next).
* test: 100% coverage gate on the harness-testing pillar
`npm run coverage` now enforces 100% lines/functions/statements (branches ≥ 90)
on the seven harness-testing modules, via Vitest V8 coverage.
- eval.ts: split the orchestration into a testable `runEvalWith(spec, runner)`
seam (injectable agent runner); `runEval` is the thin real-spawn wrapper
(v8-ignored). Lets the loop / measure-context / aggregation be tested with
canned stream-json — no model.
- mock-model.test.ts (new): drives the scripted server directly (SSE + JSON,
tool + text, count_tokens / HEAD / health, onTurn, repeat / empty-script).
- plugin-loader: a malformed plugin.json no longer crashes the loader
(safeReadJson); covered by new malformed-manifest + manifest-mcpServers tests.
- Added tests for the bare predicates' output/hook verbs, the run-hook-tier
assertions + matcher message closures, parse* defensive branches, judge's
firstJsonObject catch, and parseHookOutput's catch.
- Subprocess wrappers that can't be unit-tested (real claude spawn in
runHarnessTest/spawnClaude/judge) are v8-ignored with justification; they stay
covered by the claude-backed suite.
* feat: capture model requests into the Trace (trace.modelRequests)
The scripted mock now records every /v1/messages request it receives —
system prompt + messages, flattened to text — exposed as handle.requests
and surfaced on the unified Trace as modelRequests. This closes the gap
where a SessionStart hook's injected additionalContext (or a slash
command's expansion) could be proven to *fire* but not to *reach the
model*.
- mock-model.ts: extractRequest (pure, exported) + per-request capture,
MockHandle.requests
- harness-test.ts: Trace.modelRequests, populated by runHarnessTest
(harness tier only — with or without transcript)
- eval.ts: modelRequests is [] (the eval tier drives the real API, no
mock between claude and the model to capture requests)
- harness-assert.ts: requestContains predicate + assertRequestContains
- tests for extractRequest shapes, request capture, and the predicate;
100% coverage gate still green
* feat: safe-by-default sandbox for executing untrusted plugin hooks
runHarnessTest runs the real claude CLI, which runs the real hooks of
whatever plugin you load — for an external plugin/pluginDir that means
executing third-party code with your privileges. Make that safe by
default.
Policy (src/sandbox.ts, pure + exhaustively unit-tested):
- specTrusted: inline settings/files = trusted; any external
plugin/pluginDir = untrusted (provenance-based).
- decideSandbox: "auto" (default) runs trusted code directly and confines
untrusted code; if no sandbox is available it REFUSES (throws) rather
than running unconfined. sandbox:false is the explicit dangerous
opt-out; "strict" forces confinement even for trusted code.
Confinement (runSandboxed, bubblewrap): the mock and claude are
co-launched inside ONE network namespace (--unshare-all). Loopback is
auto-up so the in-sandbox mock is reachable, but there is no external
route, so a malicious hook can't phone home. Filesystem is --ro-bind
read-only except the throwaway work dir, a fresh empty $HOME, and an IO
dir; captured requests stream out through the IO dir into
trace.modelRequests. mock-entry.ts is the in-sandbox mock runner.
The end-to-end test proves a sandboxed run blocks egress while the mock
stays reachable (gated on bwrap+claude, skips otherwise). Existing
pluginDir/plugin tests are marked sandbox:false (we trust the pinned
vendored fixtures), keeping them on the direct path.
Also: enforce the 100% coverage gate in CI (was never run — npm test !=
coverage) with bubblewrap + claude installed so the sandbox + claude-
gated tests actually execute; stop tracking coverage/ build artifacts.
* feat: harden sandbox (clearenv + Linux-only), dogfood it, document testing
Hardening (src/sandbox.ts):
- --clearenv: drop ALL inherited env so untrusted code can't even read host
secrets (ANTHROPIC_API_KEY, cloud creds); only PATH/HOME/TMPDIR are set back.
Verified the confined claude still resolves via the restored PATH.
- Linux-only: sandboxAvailable() short-circuits on non-Linux (bubblewrap is a
Linux tool); decideSandbox messages and the spec JSDoc say so, so macOS/Windows
get a clear "needs Linux + bwrap, or pass sandbox:false" instead of a confusing
install hint.
Dogfood (src/sandbox.test.ts), both gated on bwrap+claude:
- injection test: a SessionStart hook (Claude Code's nested additionalContext
form) runs confined and its context is found in trace.modelRequests — proving
the capture chain end-to-end ("fired AND landed").
- superpowers dogfood: the REAL pinned obra/superpowers SessionStart hook runs
CONFINED (untrusted → sandbox, no egress) and its genuine output is captured.
It emits a TOP-LEVEL additionalContext, which Claude Code (nested form) never
injects — so modelRequests shows the context did NOT reach the model. That
"fired ≠ landed" gap is exactly what modelRequests surfaces, on real code.
Docs:
- README "Test your Claude Code harness": new subsections for trace.modelRequests
("did the injected context reach the model?") and the safe-by-default,
Linux-only sandbox, plus two coverage-matrix rows.
- real-superpowers.harness.mjs: drop the stale "future bwrap/docker boundary"
note — the boundary exists now; point at the confined dogfood.
* feat: close out testing pillar — measureTriggerRate + dangling-ref warning
Two remaining gaps from the roadmap, both small, both close real holes.
1. measureTriggerRate (src/eval.ts) — does a skill's description actually FIRE
across varied prompts? Wiring (does the Skill tool resolve) is the
deterministic tier's job; real activation is a property only the model can
answer, and the #1 documented skill-authoring pain. Install a plugin natively
(pluginDir), give varied prompts + a `fired` predicate over the run's Trace
(reuse the bare predicates, e.g. skillResolved), get an overall + per-prompt
rate. measureTriggerRateWith takes an injected runner so the loop is
unit-tested with no model; assertTriggerRate gates on a minimum; canonical
example examples/harness/skill-trigger-rate.eval.mjs.
2. loadPlugin dangling-intra-plugin-ref warning (src/plugin-loader.ts) — scans
the plugin's own text files (hook scripts included — those aren't materialized
into files) for root-relative refs to skills/hooks/commands/agents files that
don't exist on disk. Catches the partial-vendor / broken-path class the
dogfood hit twice: verified it flags real obra/superpowers'
skills/using-superpowers/SKILL.md (omitted from the vendored slice).
100% coverage gate still green (603 tests). Spec Key Files updated + CLAUDE.md
recompiled.
* docs: research — eval-API landscape, summarized and scored against ours
Durable record of the eval-ecosystem comparison so it doesn't live only in chat.
Two halves, as requested: (1) the field summarized — per-tool capability profiles
for promptfoo, DeepEval, Braintrust, Inspect, LangSmith, OpenAI Evals, plus the
RAG/observability/academic tools we're orthogonal to; (2) scored against our eval
API on the dimensions that decide a *harness-eval* API — strengths (harness A/B
arms, pass^k, se/std, unified Trace predicates, runner-agnostic/zero-dep), gaps
(cost/concurrency/caching, significance testing, persisted reports + regression
gating, thin judge / no dataset primitive).
Records the strategic options A–D and the decision: pursue B→A→C, defer D, first
reviewable unit B1 (cost capture) + B2 (record/replay cache).
Wired into the link graph (Key Files + research/harness-testing.md See-also) so
the no-orphan-docs rule holds; CLAUDE.md recompiled.
* feat: eval Phase B1+B2 — cost/latency capture + record/replay cache
The first reviewable unit of the B→A→C roadmap to make the eval API world-class
(research/eval-api-landscape.md). Both pieces are pure + behind the injectable
AgentRunner seam, so the 100% statements/lines/functions gate holds.
B1 — cost/latency/token capture. The stream-json result event already carries
total_cost_usd / usage / duration_ms; we were dropping it. parseUsage extracts it,
RunContext.usage exposes it to measure (so `{ cost: ctx.usage.costUsd }` is a
metric), ArmReport.usage + EvalReport.totalCostUsd aggregate it, and
formatEvalReport prints `$… · …s/run · …k tok` when present (silent under the
mock, so existing output is unchanged).
B2 — record/replay cache (src/eval-cache.ts). cacheKey hashes everything that
determines the model's output — task, resolved fixture files + settings, model,
tools, trialIndex — but DELIBERATELY NOT measure, so editing your metric and
re-running re-scores the recorded runs for free; the model is re-called only when
a model-affecting input changes (or cache:"off"). Crucially it snapshots and
restores the post-run filesystem, so measure's ctx.file()/ctx.sh() stay sound on
replay — a stdout-only cache would silently mis-score. Opt in via cache /
cacheDir on EvalSpec; wired into runEvalWith through runWithCache + executeTrial.
Tests: eval-cache key stability/sensitivity, record round-trip + malformed
tolerance, fs snapshot/restore; parseUsage, aggregateUsage, and an end-to-end
record→replay through runEvalWith proving the model isn't re-called on a hit and
restored files re-score correctly. 614 tests, gate green. Cache dir gitignored.
* feat: eval Phase B3+B4 — bounded concurrency, rate-limit backoff, budget cap
Completes Phase B of the eval-API roadmap (research/eval-api-landscape.md).
B3 — concurrency + rate-limit backoff. runEvalWith now flattens arms × trials
into a flat unit list run through a generic bounded-concurrency pool (runPool,
order-preserving); `concurrency` defaults to 1 (the safe, no-surprise default —
raise it to cut wall-clock time). Each model call is wrapped in runWithRetry:
while isRateLimited matches the captured streams (rate limit / 429 / overloaded),
it backs off exponentially and retries, up to rateLimitRetries (default 3) with
retryBackoffMs base (default 1000). spawnAgent now captures stderr so the signal
is visible; RunOut gained an optional stderr.
B4 — budget cap. maxCostUsd stops launching new trials once measured cost crosses
the cap (in-flight trials finish, the rest are skipped); EvalReport.aborted flags
it. Uses the per-run cost from B1.
All pure/orchestration behind the injectable runner seam (runPool, isRateLimited
exported and unit-tested; retry/concurrency/budget driven through runEvalWith with
fake runners — pool concurrency bound, retry-then-succeed, give-up at retries 0,
and early-abort all asserted). 620 tests, gate green (100% stmts/lines/funcs).
* test: model-free conformance suite over real vendored plugins
Grounds the loader against reality instead of synthetic fixtures. A table-driven
suite runs loadPlugin over each pinned vendored plugin (obra/superpowers,
wshobson/accessibility) and asserts invariants that must hold for any well-formed
plugin: a real surface loads (never a silent empty machine), ${CLAUDE_PLUGIN_ROOT}
fully resolves, ≥1 skill materializes, and the surface + dangling-ref warnings are
accurate (the known superpowers partial-vendor ref is flagged, nothing spurious).
This is the committed form of the check that caught the dangling-ref bug. It's
model-free, offline, and pinned by SHA — deterministic, in the coverage gate, no
claude CLI or API key (tier 1 only; the real-model tier stays opt-in in bench/).
Assertions are invariants, not version trivia (">= 1 skill", not "exactly N"), so
a harmless re-pin won't break them while a real loader/detector regression will.
622 tests, gate green (100% stmts/lines/funcs).
* fix: recompile CLAUDE.md so its integrity hash matches (audit clean)
The previous commit formatted CLAUDE.md after compiling it, and a bare `*` in the
new vendor.test.ts Key Files description got prettier-escaped to `\*` — desyncing
the embedded hash (audit exit 2). Dropped the glob char from the prose and
recompiled last so the hash covers the final, prettier-clean bytes. audit exit 0.
* feat: eval Phase A1+A5 — significance testing for A/B arms
Pays down the debt that assertImproves(..., { by: se }) booked: that API admits
the "is this gap real or noise?" question but punts the noise floor to the user.
src/stats.ts answers it — a Welch's t-test over the per-arm summary stats
(mean/se/n, no raw rows, no EvalReport change) yielding a two-sided p-value and a
verdict, via a self-contained Numerical-Recipes incomplete beta. Pure + model-free,
validated against known t-table critical values (t_.975 at df 4/10/15) and the
arcsin closed form for the incomplete beta — grounded numerics, not just
internally consistent.
Surfaced as assertSignificant / significantlyBeats, and assertImproves now takes
{ significant: true } to demand a computed noise floor instead of a hand-fed `by`.
Deliberately scoped DOWN from the original Phase A: A2 (power analysis), A3
(pass@k at arbitrary k), and A4 (paired/blocked design) are deferred — each is
correct but not yet tied to an observed pain, and building rigor ahead of need is
the trap. They come back when a real eval demands them. The empirical cross-check
(running a real bench/ comparison through compareArms) is also deferred: no API
key here, so A1 is validated on known-answer distributions for now.
629 tests, gate green (100% stmts/lines/funcs, lint clean, audit 0).
* docs: README — reflect the eval tier's new capabilities
Level 3 (runEval) predated the eval-API work; it said "reports the gap … run it
now and then" with no mention of what shipped. Surgical update (no new sections,
per the readme-brevity rule):
- gap is reported as mean ± se; assertSignificant turns it into a CI gate via a
computed (not hand-fed) Welch t-test noise floor
- the example shows the cache (cache:"readwrite") and the cost/latency/token line
- notes concurrency + maxCostUsd budget cap
- points at measureTriggerRate as the same-tier answer to skill activation
* docs: sync harness-testing guide with the eval tier's shipped capabilities
The README now advertises features the "full guide" it links to didn't cover.
Brought docs/harness-testing.md up to date:
- Significance: assertSignificant / significantlyBeats / compareArms — a Welch
t-test with a computed (not hand-fed) noise floor; added as the third gating
option alongside assertImproves / assertReliable, replacing the thin "raise
trials" note. Re-exported compareArms + Comparison through vigiles/harness-assert
so the whole eval-analysis surface lives behind one entry point (no ./stats).
- Cost / caching / concurrency: ctx.usage + report.usage + totalCostUsd, plus the
concurrency / maxCostUsd / cache:"readwrite" record-replay knobs.
- measureTriggerRate: the trigger-rate (skill-activation) tier + its example.
- Fixed two stale claims: the suite runs under vitest (not node --test), and the
eval orchestration is now fully unit-tested via an injected runner (only the
real-claude spawnAgent is out of the gate) — the 100% gate holds.
No src behaviour change beyond the re-export. 629 tests, gate green, audit clean.
* docs: testing-matrix — add the eval-tier surfaces shipped in this PR
The matrix predated cache / significance / trigger-rate / usage / vendored
conformance. Added unit rows for: usage capture+aggregation, record/replay cache,
concurrency + rate-limit retry + maxCostUsd abort, measureTriggerRateWith,
significance stats (welchTTest/compareArms/tPValueTwoSided), vendored real-plugin
conformance, and the dangling-ref warning; broadened the assert-helpers row to
cover the significance/triggerRate/tool-sequence asserts. Added the
measureTriggerRate end-to-end integration row + skill-trigger-rate example.
Corrected the "not unit-tested" note: only the real-claude spawnAgent is out of
the gate — the eval orchestration is pinned via the injected runner.
* feat: spec compilation for subagents (agent() → agents/<name>.md)
A skill is reference material the model reads on activation; a subagent is a
delegated worker with a *contract* — a dispatch description, an allowed-tools
rail, a model, a system prompt, and the rules it follows. That contract is the
"railway" and it's exactly the compile-time-verifiable surface vigiles owns, so
subagents (more than skills) want spec compilation.
- spec.ts: `agent()` builder + AgentSpec (name, description, model?, tools?,
body, rules?).
- compile.ts: `compileAgent` — YAML frontmatter (name/description/model/tools),
system-prompt body with the same verified file()/cmd()/symbol()/ref() marks as
any instruction file, an optional ## Rules section, and a SHA-256 integrity
hash. Verifies the tool contract: built-in set + mcp__server__tool pattern,
"did you mean" on a near-miss, and an error for tools that are *never* available
to a subagent (Agent/AskUserQuestion/EnterPlanMode/ExitPlanMode/ScheduleWakeup/
WaitForMcpServers). adoptDiff handles the new target. editDistance exported from
linters.ts for reuse.
- agent.test.ts: frontmatter, tool verification (built-in/MCP/never-available/
did-you-mean), body-ref validation, Rules section, spec-name check, adoptDiff
round-trip.
Grounded by a background research pass (research/subagent-compilation.md): the
real Claude Code frontmatter, and the key finding that `tools:` is documentation,
not runtime enforcement (issue #54898) — the real rail is a generated PreToolUse
hook. That hook-emission layer (compile the contract into BOTH the allowlist and
an enforcing hook, and assert they agree) is the documented next step and the
differentiator. This commit is the compile+verify foundation.
638 tests, gate green (100% stmts/lines/funcs), lint/fmt/audit clean.
* feat: agent spec sections + dogfood on a real OSS subagent
Dogfooding the agent API against wshobson's real ui-visual-validator settled the
design: a subagent is a flat, multi-section role contract (10 `##` sections,
prose + `###` subs), NOT a step/gate pipeline — and it ships with NO tools line,
inheriting every tool (the #1 footgun, in the wild).
- spec.ts: AgentSpec gains `sections` (named `##` blocks) + a `body` intro,
mirroring claude() rather than skill() — the shape real agents take. Steps/gates
stay a skill concept; a subagent's "railway" is the tool-contract envelope, not
a state machine.
- compile.ts: renderAgentSections verifies each section's refs and rejects nested
`##` headers / a `rules` key clash, same rules as a CLAUDE.md spec's sections.
- agent.test.ts: a dogfood test reproduces the real agent's shape and ADDS the
least-privilege rail it omits (Read/Grep/Glob/Bash, no Edit/Write), asserting it
compiles clean — the value-add over the hand-written original. Plus section
ref-verification + reserved-key tests.
- research/subagent-compilation.md: records the dogfood finding and the
"flat flow yes / flow-generator no / tool-catalog generator optional" answer.
641 tests, gate green (100% stmts/lines/funcs), lint/fmt/audit clean.
* docs: empirical subagent survey — no iterator needed (saturated, ~100 agents)
Surveyed ~90–100 real subagents across 9+ repos and several ecosystems (haiku
agents) — wshobson, undeadlist, VoltAgent, 0xfurai, lst97, rshah515 — plus the
official Anthropic plugins (Ralph loop, PR-review-toolkit) and orchestration
plugins (barkain). The result saturated: every additional batch returned zero.
Findings recorded in research/subagent-compilation.md:
- Iterative SYNTAX in an agent file: 0 / ~100. (Iterative LANGUAGE in prose is
common; structural loops/state machines: none.) No typed Iterator/Step/Phase/
Flow primitive exists anywhere in the ecosystem — community or the official SDK.
- Where looping actually lives: a Stop hook + state file (Ralph re-feeds the
prompt), or external scripts/commands — never the subagent .md. Architectural
reason: markdown is declarative+stateless; a loop is imperative+stateful.
- Delegation graphs are linear/star and dynamic → even a typed handoff field is
only weakly supported; prose + the Task/Agent allowlist covers them.
- Verdict: ship the flat model; do NOT add an iterator (solution seeking a
problem). Reinforces the compile-to-hooks thesis — emit PreToolUse (tool rail)
/ Stop (loop) hooks, don't invent iterator types. handoffs/outputSchema = maybe.
No code change — research only. Confidence ~90%; gate untouched.
* docs: note in-repo prior art for the agent enforcement hook
A final survey straggler surfaced that vigiles already emits an enforcement hook:
the vigiles:result mark compiles to a Stop hook (skill-hook) that gates skill
completion (test/e2e/run.sh, research/runtime-enforcement.md). The planned agent
PreToolUse tool-rail is the same emit-a-hook pattern pointed at a different event,
so roadmap item #1 records it as prior art to mirror — not greenfield.
* docs: design — railway-style subagents (verified plan-as-code; Temporal analogy)
New research/railway-subagents.md exploring how to express a FLOW over flat
subagents without reinventing a workflow engine, prompted by two inputs:
- Plan-as-code ("ultra plan" / dynamic workflows): Anthropic's orchestrator
generates an ephemeral, unverified JS orchestration script on the fly. vigiles'
angle = the typed, verified, compiled counterpart (spec is source of truth →
compile to a command .md + enforcing hooks + state schema; refs checked at
compile time; regression-testable via runEval).
- Nested subagents (Claude Code, experimental, user-reported): a deeper delegation
tree makes a typed depth/cycle/dangling-target graph check load-bearing.
Frames the Temporal analogy the user raised (workflow=deterministic railway;
delegate=activity; gate=signal; state-file+Stop-hook+eval-replay=durable replay;
"thinking happens in the activity, not the railway") and lays out three options:
(1) manual marks, (2) a workflow() TS-spec compiled via a limited vigiles API
(recommended core), (3) a thin Temporal-like deterministic driver over the
harness's native Task+hooks+state (north star; explicit boundary: don't build an
engine, delegate execution to the harness).
Honest verification note: plan-as-code is confirmed for Claude Code
(code.claude.com/docs/en/workflows, found earlier this session); the nested-
subagent cap is NOT yet verified — an automated check examined the wrong product
(the Developer Platform "Managed Agents" API, depth-1), so it's recorded as
user-reported/to-verify and the design is made robust to the exact cap.
Wired into Key Files + cross-linked with subagent-compilation.md. Docs only.
* docs: surface ultraplan as inspiration in discoverable index + ideas backlog
ultraplan / dynamic-workflows (plan-as-code) is the closest external thing to the
direction we're building, so make it easy to trip over:
- research/README.md: new "Subagents & orchestration" group indexing
subagent-compilation.md + railway-subagents.md, with ultraplan called out by
name and flagged "closest external thing to our direction — read for inspiration."
- research/feature-ideas.md: an "Inspiration to watch" callout framing ultraplan
and pointing at railway-subagents.md for the full design (Temporal analogy +
the marks / workflow()-spec / driver options).
Both spots are one click from README → research/. Docs only.
* feat(agent): PreToolUse tool-contract rail — enforce the declared tools allowlist
Closes the declared-vs-enforced gap for subagents (Claude Code #54898):
`tools:` frontmatter is documentation, not a runtime boundary. A subagent
inherits the session's grants, so the list only filters what's offered. The
deterministic layer that actually constrains it is a PreToolUse hook.
src/agent-runtime.ts (vigiles agent-hook): parse the active agent's compiled
.md frontmatter `tools:` (the single source of truth), and block (exit 2 + the
contract fed back to the model) any tool outside it. decidePreToolUse is the
pure allow/deny; .vigiles/active-agent.json records the dispatched agent —
mirroring the skill Stop-hook (src/skill-runtime.ts). No `tools:` line means
inherit-all, so the rail imposes no restriction.
The declared list and the enforced rail compile from the same source, so they
agree by construction — proven by a round-trip test (compile → parse the
frontmatter the hook reads → equals declared tools → allows exactly those).
Proven deterministically at the runHook unit tier against the real built CLI
process: a tool-event hook is reached cheaply there, where driving a live tool
call against a scripted mock is flaky (the e2e defers to it, with a note).
+22 tests; full suite 663 passing; 100% stmts/lines/funcs gate holds.
* test(agent): ground the PreToolUse rail on a real vendored subagent
The runtime suite proved the rail's logic with synthetic specs; add two tests
against the REAL pinned wshobson ui-visual-validator under examples/harness/vendor/.
It's the documented footgun in the wild: a rigorous visual validator that bases
judgments "solely on visual evidence" yet ships with NO tools: line, so it
inherits every tool incl. Edit/Write. parseAgentTools on the actual file returns
null (the rail honestly reports "no contract yet" rather than inventing one);
the spec form ADDS the least-privilege rail the original omits, compiles, and the
same rail the hook reads then blocks Write/Edit — the differentiator on a real
subagent, not a hand-built fixture.
Mirrors src/vendor.test.ts (pinned by SHA, offline, model-free).
* docs: refresh testing matrices for the subagent PreToolUse rail
Both matrices were stale on the Subagents surface. The new tool-contract rail
(agent-hook) is a runHook-testable hook, so the agents/ Hook-unit cell flips from
— to ✅; add a Coverage row for src/agent-runtime.test.ts. Kept honest about the
end-to-end gap: the deterministic/integration cell stays 🟡 ("rail not
live-armed") because Claude Code doesn't surface the active subagent to hooks, so
arming .vigiles/active-agent.json in a live session is still unsolved.
* feat(railway): railway-oriented subagents — typed Result contracts + finite composition
Flat subagents that return either success or error with rich detail on BOTH
tracks, composed on a sub-Turing railway. Railway-oriented programming (Wlaschin)
with a subagent as the step — uses the flat-worker finding, doesn't contradict it.
- result(ok, err) (spec.ts): a subagent's typed Result contract; compileAgent
renders it as an ## Output contract section (the vigiles:ok / vigiles:err block
the worker must emit).
- parseAgentResult (agent-result.ts): pure text -> Result<S,E> (ok | err |
malformed), validated against the contract shape. The shared primitive.
- railway({ steps, onError, recover }) + delegate() (spec.ts): the composer is
deliberately sub-Turing — a finite step list + bounded recovery, NO loop
combinator, so termination is structural and every reference is statically
checkable (the thing ultraplan's generated script can't be). compileRailway /
validateRailway emit the orchestrator command and resolve every delegate target
(stale-ref), reject an empty railway, and require recover.max >= 1.
- assertAgentOk / assertAgentErr / assertAgentResult (harness-assert.ts): test a
subagent's outcome the way you assert a hook decision, reusing parseAgentResult
— the testing-framework payoff of the result contract.
Deferred (the runtime half): vigiles emits + verifies, it does not drive the
railway (that's the engine, Option 3); runtime enforcement of "did the worker
emit a valid result block?" awaits what a SubagentStop hook can see.
+27 tests; full suite 690 passing; 100% stmts/lines/funcs gate holds.
* feat(cli): compile agent + railway specs; dogfood a ship-pr railway
Wire the railway surface end-to-end so it's reachable from `vigiles compile`,
then dogfood it on this repo.
CLI:
- loadSpec now returns agent/railway specs too; `vigiles compile` dispatches
_specType "agent" → compileAgentToFile and "railway" → compileRailwayToFile.
- A railway's delegate() targets are resolved against every agent spec in the
project (collectAgentNames), so an unknown target fails compile (stale-ref,
exit 1) — verified by a CLI test.
Dogfood (examples/railway/): five flat agent() workers, each with a result()
contract — planner → implementer → reviewer on the success track, bounded fixer
recovery (max 2), reporter on the error track. Compiled THROUGH the real CLI to
one .md per agent (with vigiles:ok/err Output contracts) + ship-pr.md (the
orchestrator command). The artifacts are committed alongside their specs.
Also: compileRailway now emits a trailing newline (prettier-clean, matching the
other compilers).
+2 CLI tests; full suite 692 passing; 100% stmts/lines/funcs gate holds;
fmt + audit clean.
---------
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
zernie
@@ -45,11 +45,19 @@ jobs:
|
||||
gem install rubocop
|
||||
rustup component add clippy
|
||||
|
||||
- name: Install claude CLI + bubblewrap (deterministic harness + sandbox tests)
|
||||
# claude (no API key needed — the mock tier) lets the claude-gated tests
|
||||
# run; bubblewrap lets the sandbox confinement test actually execute and
|
||||
# prove egress is blocked, instead of skipping.
|
||||
run: |
|
||||
npm i -g @anthropic-ai/claude-code
|
||||
sudo apt-get update && sudo apt-get install -y bubblewrap
|
||||
|
||||
- name: Build
|
||||
run: npm run build
|
||||
|
||||
- name: Run tests
|
||||
run: npm test
|
||||
- name: Run tests + 100% coverage gate (harness-testing pillar)
|
||||
run: npm run coverage
|
||||
|
||||
- name: Cross-runner matcher tests (vitest + jest)
|
||||
run: npx vitest run && npx jest
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
node_modules/
|
||||
dist/
|
||||
coverage/
|
||||
.DS_Store
|
||||
logo-v*.png
|
||||
.idea
|
||||
@@ -7,3 +8,5 @@ logo-v*.png
|
||||
# vigiles transient skill-runtime state
|
||||
.vigiles/active-skill.json
|
||||
examples/demo/.vigiles/
|
||||
# eval record/replay cache (recorded model runs)
|
||||
.vigiles/eval-cache/
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
.vigiles/generated.d.ts
|
||||
|
||||
# Coverage artifacts (v8 / lcov output)
|
||||
coverage/
|
||||
|
||||
# Vendored third-party plugin snapshots (dogfood fixtures) — formatted by their
|
||||
# upstreams, not us. See examples/harness/vendor/*/SOURCE.
|
||||
examples/harness/vendor/
|
||||
|
||||
Vendored
+5
-1
@@ -75,13 +75,15 @@ declare module "vigiles/generated" {
|
||||
/** All enabled linter rules across all detected linters. */
|
||||
export type LinterRule = EslintRule;
|
||||
|
||||
/** 11 npm scripts from package.json. */
|
||||
/** 13 npm scripts from package.json. */
|
||||
export type NpmScript =
|
||||
| "build"
|
||||
| "test"
|
||||
| "coverage"
|
||||
| "lint"
|
||||
| "fmt"
|
||||
| "fmt:check"
|
||||
| "demo"
|
||||
| "test:e2e"
|
||||
| "test:harness"
|
||||
| "test:eval"
|
||||
@@ -306,9 +308,11 @@ declare module "vigiles/spec" {
|
||||
scripts:
|
||||
| "build"
|
||||
| "test"
|
||||
| "coverage"
|
||||
| "lint"
|
||||
| "fmt"
|
||||
| "fmt:check"
|
||||
| "demo"
|
||||
| "test:e2e"
|
||||
| "test:harness"
|
||||
| "test:eval"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!-- vigiles:sha256:50302f93695fb02f compiled from CLAUDE.md.spec.ts -->
|
||||
<!-- vigiles:sha256:0190c408ba68003b compiled from CLAUDE.md.spec.ts -->
|
||||
|
||||
# CLAUDE.md
|
||||
|
||||
@@ -34,8 +34,8 @@ Core modules: `src/spec.ts` (types + builders), `src/compile.ts` (compiler), `sr
|
||||
|
||||
## Key Files
|
||||
|
||||
- `src/spec.ts` — Type system and builder functions (enforce, guidance, claude, skill, file, cmd, ref)
|
||||
- `src/compile.ts` — Compiler: spec → markdown with SHA-256 hash, linter verification, reference validation
|
||||
- `src/spec.ts` — Type system and builder functions (enforce, guidance, claude, skill, agent, file, cmd, ref; result/railway/delegate for railway-oriented subagents)
|
||||
- `src/compile.ts` — Compiler: spec → markdown with SHA-256 hash, linter verification, reference validation; compileClaude/compileSkill/compileAgent (subagents: frontmatter + verified tool contract + body marks + result-contract Output section) + compileRailway/validateRailway (orchestrator command over flat workers; delegate-target resolution + bounded recovery)
|
||||
- `src/linters.ts` — Cross-referencing engine (ESLint, Stylelint, Ruff, Clippy, Pylint, RuboCop, Cedar)
|
||||
- `src/cedar.test.ts` — Cedar policy resolution tests — filesystem-based @id() lookup with filename fallback
|
||||
- `src/generate-types.ts` — Type generator: scans linters/package.json/filesystem → emits .d.ts
|
||||
@@ -48,6 +48,12 @@ Core modules: `src/spec.ts` (types + builders), `src/compile.ts` (compiler), `sr
|
||||
- `src/frontmatter.test.ts` — Frontmatter parser test suite (node:test)
|
||||
- `src/action.ts` — GitHub Action wrapper
|
||||
- `src/spec.test.ts` — Spec + compiler test suite (node:test)
|
||||
- `src/agent.test.ts` — Subagent compilation test suite (node:test): agent() builder + compileAgent — frontmatter, tool-contract verification (built-in/MCP/never-available/did-you-mean), body-ref validation, Rules section, hash, adoptDiff round-trip
|
||||
- `src/agent-runtime.ts` — Agent PreToolUse tool-contract rail — the differentiator that closes the declared-vs-enforced gap (#54898): tools: is documentation, so a PreToolUse hook (vigiles agent-hook) blocks any tool outside the active subagent's contract. parseAgentTools reads the compiled .md frontmatter (the single source of truth the hook enforces), decidePreToolUse is the pure allow/deny, and .vigiles/active-agent.json tracks the dispatched agent — mirrors the skill Stop-hook (src/skill-runtime.ts)
|
||||
- `src/agent-runtime.test.ts` — Agent-runtime test suite: pure parse/decide logic, active-agent round-trip, hook ⇄ allowlist agree (the declared contract IS the enforced rail), the real built CLI hook driven deterministically via runHook (the unit tier reaches PreToolUse where a live tool call is flaky), and grounding on the REAL vendored wshobson ui-visual-validator (ships no tools: line → inherits all; the spec adds the rail it omits)
|
||||
- `src/agent-result.ts` — Railway result parser: a subagent with a result() contract ends its turn with a vigiles:ok/err block; parseAgentResult turns that text into a discriminated outcome (ok | err | malformed) and validates it against the contract shape. Pure text→Result<S,E>, the primitive the orchestrator + the assertAgentOk/Err/Result test helpers both reuse
|
||||
- `src/agent-result.test.ts` — Railway result-parser test suite: ok/err/malformed tracks, last-block-wins, JSON + shape validation across every field type (string/number/boolean/string[]), both success and error tracks
|
||||
- `src/railway.test.ts` — Railway surface test suite: result() Output-contract rendering in compileAgent, delegate()/railway() builders, compileRailway orchestrator output, validateRailway static checks (unknown delegate target, empty railway, bounded recovery — the sub-Turing guarantees)
|
||||
- `src/validate.test.ts` — Validation test suite (node:test)
|
||||
- `src/cli.test.ts` — CLI integration + E2E test suite (node:test)
|
||||
- `src/integrity.ts` — Integrity check: SHA-256 hash verification for compiled markdown (detects hand-edits)
|
||||
@@ -66,16 +72,24 @@ Core modules: `src/spec.ts` (types + builders), `src/compile.ts` (compiler), `sr
|
||||
- `src/symbols.test.ts` — Symbol extractor test suite (node:test)
|
||||
- `src/refs.ts` — Symbol reference verification: the `vigiles:symbol path#name` mark (verify the named file defines the symbol) + unmarkedCodeRefs enforcement for the refs-hook
|
||||
- `src/refs.test.ts` — Symbol reference verification test suite (node:test)
|
||||
- `src/mock-model.ts` — Scriptable, dependency-free Anthropic Messages SSE mock (startMock/scriptModel) — point real claude at it via ANTHROPIC_BASE_URL for deterministic harness tests
|
||||
- `src/harness-test.ts` — Deterministic Claude Code harness testing: runHarnessTest runs real claude + real hooks/settings against a scripted mock model (Stop-hooks reliable; tool-event hooks via the eval tier)
|
||||
- `src/mock-model.ts` — Scriptable, dependency-free Anthropic Messages SSE mock (startMock/scriptModel) — point real claude at it via ANTHROPIC_BASE_URL for deterministic harness tests; extractRequest + onRequest capture each request into trace.modelRequests
|
||||
- `src/harness-test.ts` — Deterministic Claude Code harness testing: runHarnessTest runs real claude + real hooks/settings against a scripted mock model (Stop-hooks reliable; tool-event hooks via the eval tier); safe-by-default — an external plugin/pluginDir is confined per src/sandbox.ts
|
||||
- `src/harness-test.test.ts` — Harness-test suite (node:test, skips without claude)
|
||||
- `src/sandbox.ts` — Safe-by-default confinement: decideSandbox is the pure policy (untrusted plugin code never runs unconfined unless sandbox:false), runSandboxed co-launches the mock + claude inside one bubblewrap network namespace (loopback-only — mock reachable, egress blocked); specTrusted/bwrapArgs/parseRequestLog are the pure, tested seams
|
||||
- `src/sandbox.test.ts` — Sandbox test suite: pure policy/trust/args/log-parse coverage + a gated end-to-end test proving a sandboxed run blocks network egress while the in-sandbox mock stays reachable (skips without bwrap/claude)
|
||||
- `src/mock-entry.ts` — In-sandbox mock entry: run as a subprocess inside the bwrap netns so the scripted mock lives on the isolated loopback; streams captured requests to a file the parent reads back for trace.modelRequests
|
||||
- `src/run-hook.ts` — Hook unit tier: runHook pipes a synthesized event JSON to a hook process (no claude, no model) and reports exit code + normalized block/allow decision — the cheap base of the pyramid, and the only tier that reaches every event (Edit/Write, PreCompact, Notification, SessionEnd, SubagentStop); parseHookOutput/decideHook are the pure, testable decision logic
|
||||
- `src/run-hook.test.ts` — Hook unit-tier test suite (node:test): pure decision logic + real shell hooks across exit codes, stdin event passthrough, env injection, JSON permission decisions
|
||||
- `src/eval.ts` — Harness eval API: runEval drives the real claude CLI across arms x trials and aggregates mean ± se (variance) — the empirical half of testing your harness (generalizes bench/)
|
||||
- `src/eval.test.ts` — Eval aggregation/formatting + variance test suite (node:test)
|
||||
- `src/plugin-loader.ts` — Plugin/repo harness loader: loadPlugin reads real hooks (inline plugin.json, a hooks string path, the hooks/hooks.json convention e.g. obra/superpowers, or .claude/settings.json) with ${CLAUDE_PLUGIN_ROOT} resolved, plus CLAUDE.md + skills + agents + commands materialized; .warnings flags surfaces the deterministic tier can't drive (subagents/commands/MCP, or an empty machine) so a plugin load never silently tests nothing; resolveHarness layers inline settings/files on top so a test/eval runs the assembled machine
|
||||
- `src/eval.ts` — Harness eval API: runEval drives the real claude CLI across arms x trials and aggregates mean ± se (variance) + cost/latency/token usage; bounded concurrency (runPool) + rate-limit backoff + maxCostUsd budget cap; record/replay cache (cache:readwrite) replays runs so editing measure re-scores for free; measureTriggerRate measures how reliably a skill's description FIRES across varied prompts (the #1 skill-authoring pain) — the empirical half of testing your harness (generalizes bench/)
|
||||
- `src/eval.test.ts` — Eval aggregation/formatting + variance + usage/cache test suite (node:test)
|
||||
- `src/eval-cache.ts` — Eval record/replay cache: cacheKey hashes the model-affecting inputs (task, resolved files+settings, model, tools, trialIndex) but NOT measure, so re-scoring replays for free; snapshotDir/restoreDir round-trip the post-run filesystem so ctx.file()/ctx.sh() stay sound on replay
|
||||
- `src/eval-cache.test.ts` — Eval-cache test suite (node:test): key stability/sensitivity, record round-trip + malformed-record tolerance, filesystem snapshot/restore
|
||||
- `src/stats.ts` — Significance testing for eval A/B arms: Welch's t-test over the per-arm summary stats (mean/se/n, no raw rows) → two-sided p-value + verdict (Numerical-Recipes incomplete beta); compareArms computes the noise floor instead of the assertImproves({by}) hand-fed gap — behind assertSignificant/significantlyBeats
|
||||
- `src/stats.test.ts` — Stats test suite (node:test): incomplete-beta vs known closed forms, p-values vs t-table critical values, Welch significant/noise/deterministic cases
|
||||
- `src/plugin-loader.ts` — Plugin/repo harness loader: loadPlugin reads real hooks (inline plugin.json, a hooks string path, the hooks/hooks.json convention e.g. obra/superpowers, or .claude/settings.json) with ${CLAUDE_PLUGIN_ROOT} resolved, plus CLAUDE.md + skills + agents + commands materialized; .warnings flags surfaces the deterministic tier can't drive (subagents/commands/MCP, an empty machine, or dangling intra-plugin file refs e.g. a partial vendor) so a plugin load never silently tests nothing; resolveHarness layers inline settings/files on top so a test/eval runs the assembled machine
|
||||
- `src/plugin-loader.test.ts` — Plugin-loader test suite (node:test): CLAUDE_PLUGIN_ROOT resolution, CLAUDE.md/skills/agents/commands materialization, surface + empty-machine + MCP warnings, settings merge, in-repo dogfood
|
||||
- `src/harness-assert.ts` — Runner-agnostic harness helpers: withHarness (auto-cleanup), throwing `assert*` helpers incl. assertHookBlocked/assertHookAllowed (node:test/any runner), and vigilesMatchers (toHaveCreated/toBlock/toBeatBaseline) for vitest/jest expect.extend
|
||||
- `src/vendor.test.ts` — Conformance suite over REAL vendored plugins under examples/harness/vendor/: model-free, in-gate, table-driven loadPlugin invariants (loads a surface, ${CLAUDE_PLUGIN_ROOT} resolves, skills materialize, surface + dangling-ref warnings accurate) — grounded in reality (pinned by SHA, offline, no API key), the shape that caught the superpowers partial-vendor
|
||||
- `src/harness-assert.ts` — Runner-agnostic harness helpers: withHarness (auto-cleanup), throwing `assert*` helpers incl. assertHookBlocked/assertHookAllowed and assertAgentOk/Err/Result (test a subagent's railway outcome via parseAgentResult — the testing-framework payoff of the result contract), and vigilesMatchers (toHaveCreated/toBlock/toBeatBaseline) for vitest/jest expect.extend
|
||||
- `src/harness-assert.test.ts` — Harness-assert test suite (node:test): eval delta helpers + matcher pass/fail logic
|
||||
- `src/judge.ts` — Thin LLM-as-judge for the eval tier: judge() grades an output against a rubric with a model (synchronous, for use inside measure); parseJudgeOutput is the pure, testable verdict parser
|
||||
- `src/judge.test.ts` — Judge verdict-parsing test suite (node:test): result-field unwrap, prose-wrapped JSON, threshold, clamping, unparseable fallback
|
||||
@@ -92,9 +106,11 @@ Core modules: `src/spec.ts` (types + builders), `src/compile.ts` (compiler), `sr
|
||||
- `src/proofs.test.ts` — Proof system + evolution engine tests (node:test)
|
||||
- `CLAUDE.md.spec.ts` — This file — the source of truth for CLAUDE.md
|
||||
- `examples/SKILL.md.spec.ts` — Example SKILL.md spec
|
||||
- `examples/railway/ship-pr.md.spec.ts` — Dogfood: a railway() over five flat agent() workers (planner→implementer→reviewer, bounded fixer recovery, reporter error track), each with a result() contract. Compiles via the real `vigiles compile` to ship-pr.md (orchestrator command) + one .md per agent (with vigiles:ok/err Output contracts); every delegate() target is resolved against the sibling agent specs at compile time
|
||||
- `examples/harness/hook-unit.harness.mjs` — Canonical hook unit-tier example (runHook): test a hook's logic in isolation with no claude CLI — the cheap base of the pyramid; runs in CI for free
|
||||
- `examples/harness/policy-gate.harness.mjs` — Canonical deterministic harness test (runHarnessTest): a PreToolUse Bash policy gate (block-no-verify shape) + a SessionStart setup hook (obra/superpowers shape)
|
||||
- `examples/harness/skill-outcome.eval.mjs` — Canonical skill-outcome eval (runEval): does a skill change the agent's output? — the question you ask of any SKILL.md
|
||||
- `examples/harness/skill-trigger-rate.eval.mjs` — Canonical trigger-rate eval (measureTriggerRate): does a skill's description actually FIRE across varied prompts? — installs a real pinned plugin via pluginDir and reuses the skillResolved predicate
|
||||
- `examples/harness/plugin-cohesion.harness.mjs` — Canonical cohesion test (runHarnessTest with plugin:): load a whole plugin (.claude-plugin/plugin.json + CLAUDE.md) and assert multiple hooks fire together
|
||||
- `bench/evals/refs-hook.eval.mjs` — Worked eval reproducing benchmark #4 (forcing symbol marks → verifiable references?) as a runEval library call
|
||||
- `research/adoption-strategy.md` — Adoption strategy: zero-config setup, progressive enforcement, agent workflows
|
||||
@@ -103,6 +119,8 @@ Core modules: `src/spec.ts` (types + builders), `src/compile.ts` (compiler), `sr
|
||||
- `research/feature-ideas.md` — Feature ideas: plugin API, custom rules, exhaustive coverage
|
||||
- `research/ai-code-quality.md` — Research: AI code quality patterns
|
||||
- `research/self-evolving-specs.md` — Design doc: self-evolving spec system (proofs, Merkle history, evolution engine)
|
||||
- `research/subagent-compilation.md` — Research + roadmap: compiling typed subagent definitions (agent() → agents/<name>.md) — the real Claude Code frontmatter, the declared-vs-enforced gap (tools: is documentation; PreToolUse hook is the rail, issue #54898), the empirical no-iterator survey (~100 subagents), prior-art agent contracts, and the prioritized next layers (generated enforcement hook, handoff resolution, trigger-rate for dispatch)
|
||||
- `research/railway-subagents.md` — Design exploration: railway-style orchestration over flat subagents — verified plan-as-code as the counterpart to ultra-plan/dynamic-workflows, the Temporal analogy (workflow/activity/gate/durable-state), and three options (manual marks / workflow() TS-spec compilation / a thin Temporal-like deterministic driver over the harness's Task+hooks+state)
|
||||
- `research/code-search-for-agents.md` — Research: code search approaches (grep vs embeddings vs AST-grep)
|
||||
- `research/runtime-enforcement.md` — Research: spec-derived runtime enforcement via hooks, skill contracts, session audit
|
||||
- `research/agent-integration.md` — Research: deterministic backstop for AI agents — hooks, proofs, static checks anchored at the spec
|
||||
@@ -116,6 +134,7 @@ Core modules: `src/spec.ts` (types + builders), `src/compile.ts` (compiler), `sr
|
||||
- `research/distribution-strategy.md` — Why nobody uses vigiles yet: funnel diagnosis + scan demo proposal as highest-leverage intervention
|
||||
- `research/reference-verification-limits.md` — Synthesis: the conceptual boundary of reference verification — proxy-vs-judgment gap, prose undecidability (active mark vs passive symbol-table sweep), the doc-format landscape (explicit-link = marking; identity-based = the real fix), and the delegate/ignore/own rule for existing tools (Sphinx etc.)
|
||||
- `research/harness-testing.md` — Testing the Claude Code harness — the three-tier design (unit runHook + deterministic runHarnessTest + real-model runEval), the assembled-machine plugin loader, and a coverage assessment against real plugins (protect-mcp, obra/superpowers, block-no-verify, wshobson agents/skills)
|
||||
- `research/eval-api-landscape.md` — Eval-API landscape: the LLM/agent eval field (promptfoo, DeepEval, Braintrust, Inspect, LangSmith, OpenAI Evals) summarized then scored against our eval API — strengths (harness A/B arms, pass^k, se/std, unified Trace predicates), gaps (cost/concurrency/caching, significance testing, regression gating), and the B→A→C roadmap (defer D)
|
||||
- `research/skill-authoring-pains.md` — Research: pains authoring agent skills (triggering, drift, testing, distribution) + strategic note on documentation-vs-procedure split and verifying SKILL.md references
|
||||
- `docs/harness-testing.md` — Harness-testing guide: three layers (verify refs / deterministic / eval), test the whole machine via plugin:, runner-agnostic usage (node:test/vitest/jest) + matchers, variance, LLM-judge, CLI fallback
|
||||
- `docs/testing-matrix.md` — Testing matrix: every harness-testing use case mapped to its test tier (unit / cross-runner / type / integration-CI) and file, plus why the CLI examples are .mjs and the API is TypeScript
|
||||
|
||||
+46
-8
@@ -37,9 +37,9 @@ Core modules: \`src/spec.ts\` (types + builders), \`src/compile.ts\` (compiler),
|
||||
|
||||
keyFiles: {
|
||||
"src/spec.ts":
|
||||
"Type system and builder functions (enforce, guidance, claude, skill, file, cmd, ref)",
|
||||
"Type system and builder functions (enforce, guidance, claude, skill, agent, file, cmd, ref; result/railway/delegate for railway-oriented subagents)",
|
||||
"src/compile.ts":
|
||||
"Compiler: spec → markdown with SHA-256 hash, linter verification, reference validation",
|
||||
"Compiler: spec → markdown with SHA-256 hash, linter verification, reference validation; compileClaude/compileSkill/compileAgent (subagents: frontmatter + verified tool contract + body marks + result-contract Output section) + compileRailway/validateRailway (orchestrator command over flat workers; delegate-target resolution + bounded recovery)",
|
||||
"src/linters.ts":
|
||||
"Cross-referencing engine (ESLint, Stylelint, Ruff, Clippy, Pylint, RuboCop, Cedar)",
|
||||
"src/cedar.test.ts":
|
||||
@@ -61,6 +61,18 @@ Core modules: \`src/spec.ts\` (types + builders), \`src/compile.ts\` (compiler),
|
||||
"src/frontmatter.test.ts": "Frontmatter parser test suite (node:test)",
|
||||
"src/action.ts": "GitHub Action wrapper",
|
||||
"src/spec.test.ts": "Spec + compiler test suite (node:test)",
|
||||
"src/agent.test.ts":
|
||||
"Subagent compilation test suite (node:test): agent() builder + compileAgent — frontmatter, tool-contract verification (built-in/MCP/never-available/did-you-mean), body-ref validation, Rules section, hash, adoptDiff round-trip",
|
||||
"src/agent-runtime.ts":
|
||||
"Agent PreToolUse tool-contract rail — the differentiator that closes the declared-vs-enforced gap (#54898): tools: is documentation, so a PreToolUse hook (vigiles agent-hook) blocks any tool outside the active subagent's contract. parseAgentTools reads the compiled .md frontmatter (the single source of truth the hook enforces), decidePreToolUse is the pure allow/deny, and .vigiles/active-agent.json tracks the dispatched agent — mirrors the skill Stop-hook (src/skill-runtime.ts)",
|
||||
"src/agent-runtime.test.ts":
|
||||
"Agent-runtime test suite: pure parse/decide logic, active-agent round-trip, hook ⇄ allowlist agree (the declared contract IS the enforced rail), the real built CLI hook driven deterministically via runHook (the unit tier reaches PreToolUse where a live tool call is flaky), and grounding on the REAL vendored wshobson ui-visual-validator (ships no tools: line → inherits all; the spec adds the rail it omits)",
|
||||
"src/agent-result.ts":
|
||||
"Railway result parser: a subagent with a result() contract ends its turn with a vigiles:ok/err block; parseAgentResult turns that text into a discriminated outcome (ok | err | malformed) and validates it against the contract shape. Pure text→Result<S,E>, the primitive the orchestrator + the assertAgentOk/Err/Result test helpers both reuse",
|
||||
"src/agent-result.test.ts":
|
||||
"Railway result-parser test suite: ok/err/malformed tracks, last-block-wins, JSON + shape validation across every field type (string/number/boolean/string[]), both success and error tracks",
|
||||
"src/railway.test.ts":
|
||||
"Railway surface test suite: result() Output-contract rendering in compileAgent, delegate()/railway() builders, compileRailway orchestrator output, validateRailway static checks (unknown delegate target, empty railway, bounded recovery — the sub-Turing guarantees)",
|
||||
"src/validate.test.ts": "Validation test suite (node:test)",
|
||||
"src/cli.test.ts": "CLI integration + E2E test suite (node:test)",
|
||||
"src/integrity.ts":
|
||||
@@ -90,25 +102,41 @@ Core modules: \`src/spec.ts\` (types + builders), \`src/compile.ts\` (compiler),
|
||||
"Symbol reference verification: the `vigiles:symbol path#name` mark (verify the named file defines the symbol) + unmarkedCodeRefs enforcement for the refs-hook",
|
||||
"src/refs.test.ts": "Symbol reference verification test suite (node:test)",
|
||||
"src/mock-model.ts":
|
||||
"Scriptable, dependency-free Anthropic Messages SSE mock (startMock/scriptModel) — point real claude at it via ANTHROPIC_BASE_URL for deterministic harness tests",
|
||||
"Scriptable, dependency-free Anthropic Messages SSE mock (startMock/scriptModel) — point real claude at it via ANTHROPIC_BASE_URL for deterministic harness tests; extractRequest + onRequest capture each request into trace.modelRequests",
|
||||
"src/harness-test.ts":
|
||||
"Deterministic Claude Code harness testing: runHarnessTest runs real claude + real hooks/settings against a scripted mock model (Stop-hooks reliable; tool-event hooks via the eval tier)",
|
||||
"Deterministic Claude Code harness testing: runHarnessTest runs real claude + real hooks/settings against a scripted mock model (Stop-hooks reliable; tool-event hooks via the eval tier); safe-by-default — an external plugin/pluginDir is confined per src/sandbox.ts",
|
||||
"src/harness-test.test.ts":
|
||||
"Harness-test suite (node:test, skips without claude)",
|
||||
"src/sandbox.ts":
|
||||
"Safe-by-default confinement: decideSandbox is the pure policy (untrusted plugin code never runs unconfined unless sandbox:false), runSandboxed co-launches the mock + claude inside one bubblewrap network namespace (loopback-only — mock reachable, egress blocked); specTrusted/bwrapArgs/parseRequestLog are the pure, tested seams",
|
||||
"src/sandbox.test.ts":
|
||||
"Sandbox test suite: pure policy/trust/args/log-parse coverage + a gated end-to-end test proving a sandboxed run blocks network egress while the in-sandbox mock stays reachable (skips without bwrap/claude)",
|
||||
"src/mock-entry.ts":
|
||||
"In-sandbox mock entry: run as a subprocess inside the bwrap netns so the scripted mock lives on the isolated loopback; streams captured requests to a file the parent reads back for trace.modelRequests",
|
||||
"src/run-hook.ts":
|
||||
"Hook unit tier: runHook pipes a synthesized event JSON to a hook process (no claude, no model) and reports exit code + normalized block/allow decision — the cheap base of the pyramid, and the only tier that reaches every event (Edit/Write, PreCompact, Notification, SessionEnd, SubagentStop); parseHookOutput/decideHook are the pure, testable decision logic",
|
||||
"src/run-hook.test.ts":
|
||||
"Hook unit-tier test suite (node:test): pure decision logic + real shell hooks across exit codes, stdin event passthrough, env injection, JSON permission decisions",
|
||||
"src/eval.ts":
|
||||
"Harness eval API: runEval drives the real claude CLI across arms x trials and aggregates mean ± se (variance) — the empirical half of testing your harness (generalizes bench/)",
|
||||
"Harness eval API: runEval drives the real claude CLI across arms x trials and aggregates mean ± se (variance) + cost/latency/token usage; bounded concurrency (runPool) + rate-limit backoff + maxCostUsd budget cap; record/replay cache (cache:readwrite) replays runs so editing measure re-scores for free; measureTriggerRate measures how reliably a skill's description FIRES across varied prompts (the #1 skill-authoring pain) — the empirical half of testing your harness (generalizes bench/)",
|
||||
"src/eval.test.ts":
|
||||
"Eval aggregation/formatting + variance test suite (node:test)",
|
||||
"Eval aggregation/formatting + variance + usage/cache test suite (node:test)",
|
||||
"src/eval-cache.ts":
|
||||
"Eval record/replay cache: cacheKey hashes the model-affecting inputs (task, resolved files+settings, model, tools, trialIndex) but NOT measure, so re-scoring replays for free; snapshotDir/restoreDir round-trip the post-run filesystem so ctx.file()/ctx.sh() stay sound on replay",
|
||||
"src/eval-cache.test.ts":
|
||||
"Eval-cache test suite (node:test): key stability/sensitivity, record round-trip + malformed-record tolerance, filesystem snapshot/restore",
|
||||
"src/stats.ts":
|
||||
"Significance testing for eval A/B arms: Welch's t-test over the per-arm summary stats (mean/se/n, no raw rows) → two-sided p-value + verdict (Numerical-Recipes incomplete beta); compareArms computes the noise floor instead of the assertImproves({by}) hand-fed gap — behind assertSignificant/significantlyBeats",
|
||||
"src/stats.test.ts":
|
||||
"Stats test suite (node:test): incomplete-beta vs known closed forms, p-values vs t-table critical values, Welch significant/noise/deterministic cases",
|
||||
"src/plugin-loader.ts":
|
||||
"Plugin/repo harness loader: loadPlugin reads real hooks (inline plugin.json, a hooks string path, the hooks/hooks.json convention e.g. obra/superpowers, or .claude/settings.json) with ${CLAUDE_PLUGIN_ROOT} resolved, plus CLAUDE.md + skills + agents + commands materialized; .warnings flags surfaces the deterministic tier can't drive (subagents/commands/MCP, or an empty machine) so a plugin load never silently tests nothing; resolveHarness layers inline settings/files on top so a test/eval runs the assembled machine",
|
||||
"Plugin/repo harness loader: loadPlugin reads real hooks (inline plugin.json, a hooks string path, the hooks/hooks.json convention e.g. obra/superpowers, or .claude/settings.json) with ${CLAUDE_PLUGIN_ROOT} resolved, plus CLAUDE.md + skills + agents + commands materialized; .warnings flags surfaces the deterministic tier can't drive (subagents/commands/MCP, an empty machine, or dangling intra-plugin file refs e.g. a partial vendor) so a plugin load never silently tests nothing; resolveHarness layers inline settings/files on top so a test/eval runs the assembled machine",
|
||||
"src/plugin-loader.test.ts":
|
||||
"Plugin-loader test suite (node:test): CLAUDE_PLUGIN_ROOT resolution, CLAUDE.md/skills/agents/commands materialization, surface + empty-machine + MCP warnings, settings merge, in-repo dogfood",
|
||||
"src/vendor.test.ts":
|
||||
"Conformance suite over REAL vendored plugins under examples/harness/vendor/: model-free, in-gate, table-driven loadPlugin invariants (loads a surface, ${CLAUDE_PLUGIN_ROOT} resolves, skills materialize, surface + dangling-ref warnings accurate) — grounded in reality (pinned by SHA, offline, no API key), the shape that caught the superpowers partial-vendor",
|
||||
"src/harness-assert.ts":
|
||||
"Runner-agnostic harness helpers: withHarness (auto-cleanup), throwing `assert*` helpers incl. assertHookBlocked/assertHookAllowed (node:test/any runner), and vigilesMatchers (toHaveCreated/toBlock/toBeatBaseline) for vitest/jest expect.extend",
|
||||
"Runner-agnostic harness helpers: withHarness (auto-cleanup), throwing `assert*` helpers incl. assertHookBlocked/assertHookAllowed and assertAgentOk/Err/Result (test a subagent's railway outcome via parseAgentResult — the testing-framework payoff of the result contract), and vigilesMatchers (toHaveCreated/toBlock/toBeatBaseline) for vitest/jest expect.extend",
|
||||
"src/harness-assert.test.ts":
|
||||
"Harness-assert test suite (node:test): eval delta helpers + matcher pass/fail logic",
|
||||
"src/judge.ts":
|
||||
@@ -138,12 +166,16 @@ Core modules: \`src/spec.ts\` (types + builders), \`src/compile.ts\` (compiler),
|
||||
"src/proofs.test.ts": "Proof system + evolution engine tests (node:test)",
|
||||
"CLAUDE.md.spec.ts": "This file — the source of truth for CLAUDE.md",
|
||||
"examples/SKILL.md.spec.ts": "Example SKILL.md spec",
|
||||
"examples/railway/ship-pr.md.spec.ts":
|
||||
"Dogfood: a railway() over five flat agent() workers (planner→implementer→reviewer, bounded fixer recovery, reporter error track), each with a result() contract. Compiles via the real `vigiles compile` to ship-pr.md (orchestrator command) + one .md per agent (with vigiles:ok/err Output contracts); every delegate() target is resolved against the sibling agent specs at compile time",
|
||||
"examples/harness/hook-unit.harness.mjs":
|
||||
"Canonical hook unit-tier example (runHook): test a hook's logic in isolation with no claude CLI — the cheap base of the pyramid; runs in CI for free",
|
||||
"examples/harness/policy-gate.harness.mjs":
|
||||
"Canonical deterministic harness test (runHarnessTest): a PreToolUse Bash policy gate (block-no-verify shape) + a SessionStart setup hook (obra/superpowers shape)",
|
||||
"examples/harness/skill-outcome.eval.mjs":
|
||||
"Canonical skill-outcome eval (runEval): does a skill change the agent's output? — the question you ask of any SKILL.md",
|
||||
"examples/harness/skill-trigger-rate.eval.mjs":
|
||||
"Canonical trigger-rate eval (measureTriggerRate): does a skill's description actually FIRE across varied prompts? — installs a real pinned plugin via pluginDir and reuses the skillResolved predicate",
|
||||
"examples/harness/plugin-cohesion.harness.mjs":
|
||||
"Canonical cohesion test (runHarnessTest with plugin:): load a whole plugin (.claude-plugin/plugin.json + CLAUDE.md) and assert multiple hooks fire together",
|
||||
"bench/evals/refs-hook.eval.mjs":
|
||||
@@ -158,6 +190,10 @@ Core modules: \`src/spec.ts\` (types + builders), \`src/compile.ts\` (compiler),
|
||||
"research/ai-code-quality.md": "Research: AI code quality patterns",
|
||||
"research/self-evolving-specs.md":
|
||||
"Design doc: self-evolving spec system (proofs, Merkle history, evolution engine)",
|
||||
"research/subagent-compilation.md":
|
||||
"Research + roadmap: compiling typed subagent definitions (agent() → agents/<name>.md) — the real Claude Code frontmatter, the declared-vs-enforced gap (tools: is documentation; PreToolUse hook is the rail, issue #54898), the empirical no-iterator survey (~100 subagents), prior-art agent contracts, and the prioritized next layers (generated enforcement hook, handoff resolution, trigger-rate for dispatch)",
|
||||
"research/railway-subagents.md":
|
||||
"Design exploration: railway-style orchestration over flat subagents — verified plan-as-code as the counterpart to ultra-plan/dynamic-workflows, the Temporal analogy (workflow/activity/gate/durable-state), and three options (manual marks / workflow() TS-spec compilation / a thin Temporal-like deterministic driver over the harness's Task+hooks+state)",
|
||||
"research/code-search-for-agents.md":
|
||||
"Research: code search approaches (grep vs embeddings vs AST-grep)",
|
||||
"research/runtime-enforcement.md":
|
||||
@@ -184,6 +220,8 @@ Core modules: \`src/spec.ts\` (types + builders), \`src/compile.ts\` (compiler),
|
||||
"Synthesis: the conceptual boundary of reference verification — proxy-vs-judgment gap, prose undecidability (active mark vs passive symbol-table sweep), the doc-format landscape (explicit-link = marking; identity-based = the real fix), and the delegate/ignore/own rule for existing tools (Sphinx etc.)",
|
||||
"research/harness-testing.md":
|
||||
"Testing the Claude Code harness — the three-tier design (unit runHook + deterministic runHarnessTest + real-model runEval), the assembled-machine plugin loader, and a coverage assessment against real plugins (protect-mcp, obra/superpowers, block-no-verify, wshobson agents/skills)",
|
||||
"research/eval-api-landscape.md":
|
||||
"Eval-API landscape: the LLM/agent eval field (promptfoo, DeepEval, Braintrust, Inspect, LangSmith, OpenAI Evals) summarized then scored against our eval API — strengths (harness A/B arms, pass^k, se/std, unified Trace predicates), gaps (cost/concurrency/caching, significance testing, regression gating), and the B→A→C roadmap (defer D)",
|
||||
"research/skill-authoring-pains.md":
|
||||
"Research: pains authoring agent skills (triggering, drift, testing, distribution) + strategic note on documentation-vs-procedure split and verifying SKILL.md references",
|
||||
"docs/harness-testing.md":
|
||||
|
||||
@@ -313,7 +313,8 @@ assert(JSON.parse(r.stdout).num_turns > 1); // the Stop hook forced more work
|
||||
### Level 3 — does it change what Claude does? (real AI, occasional)
|
||||
|
||||
`runEval` runs the **real** model N times with your change **on vs off** and
|
||||
reports the gap. Costs tokens, so you run it now and then — not on every save:
|
||||
reports the gap as **mean ± se** — so you can tell signal from noise instead of
|
||||
eyeballing two averages:
|
||||
|
||||
```typescript
|
||||
import { runEval, formatEvalReport } from "vigiles/eval";
|
||||
@@ -325,10 +326,21 @@ const report = await runEval({
|
||||
marked: ctx.sh("grep -c vigiles:symbol SKILL.md") !== "0",
|
||||
}),
|
||||
trials: 6,
|
||||
cache: "readwrite", // replay past runs — editing `measure` re-scores for free
|
||||
});
|
||||
console.log(formatEvalReport(report)); // off marked=0.00 on marked=0.50
|
||||
console.log(formatEvalReport(report));
|
||||
// off marked=0.00 on marked=0.50±0.20 pass^k=0 ($0.07 · 1.2s/run · 4.1k tok)
|
||||
```
|
||||
|
||||
`assertSignificant(report, { baseline: "off", arm: "on", metric: "marked" })`
|
||||
turns the gap into a CI gate — a Welch t-test decides whether it cleared the
|
||||
noise floor, **computed** from the arms' spread, not hand-fed. Runs go
|
||||
**concurrently**, track **cost / latency / tokens** (cap them with `maxCostUsd`),
|
||||
and the **record/replay cache** makes re-scoring after a `measure` edit free.
|
||||
|
||||
Same tier, different question: **`measureTriggerRate`** measures how reliably a
|
||||
skill's _description fires_ across varied prompts — the #1 skill-authoring pain.
|
||||
|
||||
### Test your skills for real — and assert on what Claude _did_
|
||||
|
||||
Install a plugin the way Claude actually does (`pluginDir` → `--plugin-dir`) so
|
||||
@@ -356,6 +368,43 @@ dangerous tool was never used, which "the file looks unchanged" can't. It works
|
||||
on **real third-party plugins** too: the suite confirms real `obra/superpowers`
|
||||
and `wshobson/agents` skills resolve this way, with no markers injected.
|
||||
|
||||
### Did the injected context actually reach the model?
|
||||
|
||||
A SessionStart hook or a slash command can _fire_ and still inject **nothing** —
|
||||
wrong output shape, wrong platform. `trace.modelRequests` records what the model
|
||||
actually received (system + messages), so you assert it landed, not just that the
|
||||
hook ran — **"fired ≠ landed"**:
|
||||
|
||||
```typescript
|
||||
import { assertRequestContains } from "vigiles/harness-assert";
|
||||
|
||||
assertRequestContains(r, "You have superpowers"); // the additionalContext reached the model
|
||||
```
|
||||
|
||||
(Dogfood: this is exactly how vigiles found that real `obra/superpowers` emits a
|
||||
_top-level_ `additionalContext`, which Claude Code — reading the _nested_ form —
|
||||
never injects. The hook fired; the context never landed.)
|
||||
|
||||
### Running an untrusted plugin? It's confined by default
|
||||
|
||||
Testing a third-party plugin means executing **its** hooks. `runHarnessTest` is
|
||||
safe by default: code you wrote (inline `settings`/`files`) runs directly, but an
|
||||
external `plugin` / `pluginDir` is **confined under bubblewrap** — a network
|
||||
namespace with **no egress** (a malicious hook can't phone home), a read-only
|
||||
filesystem, and a **cleared environment** (your `ANTHROPIC_API_KEY` and other
|
||||
secrets aren't even visible). If no sandbox is available the run **refuses**
|
||||
rather than executing unconfined:
|
||||
|
||||
```typescript
|
||||
runHarnessTest({ pluginDir: "./vendor/some-plugin", model }); // confined, or refuses
|
||||
runHarnessTest({ pluginDir: "./audited", model, sandbox: false }); // you vouch for it → direct
|
||||
```
|
||||
|
||||
Confinement is **Linux-only** (bubblewrap); on macOS / Windows an untrusted run
|
||||
refuses unless you pass `sandbox: false`. The suite dogfoods it on real
|
||||
`obra/superpowers` — its `SessionStart` hook runs in a no-egress sandbox, and the
|
||||
test proves egress is blocked while the scripted mock stays reachable.
|
||||
|
||||
### Run them in CI
|
||||
|
||||
`vigiles test` runs `*.harness.mjs` files (free, no key); `vigiles eval` runs
|
||||
@@ -379,10 +428,12 @@ npx vigiles eval --trials=6 examples/harness/skill-outcome.eval.mjs
|
||||
| Hooks — PreCompact / Notification / SessionEnd / SubagentStop | ✅ logic | — (mock can't trigger) | 🟡 |
|
||||
| CLAUDE.md / instructions | ✅ refs | 🟡 present, not behaviour | ✅ behaviour |
|
||||
| Skills | 🟡 refs | ✅ resolves via `pluginDir` | ✅ activation |
|
||||
| Subagents (`agents/`) | 🟡 refs | 🔴 hard | ✅ via Task |
|
||||
| Subagents (`agents/`) | ✅ tool rail · 🟡 refs | 🟡 rail not live-armed | ✅ via Task |
|
||||
| Slash commands (`commands/`) | 🟡 refs | 🟡 needs prompt capture | ✅ via `/cmd` |
|
||||
| MCP servers | ✅ tool refs (`vigiles:mcp`) | 🔴 | 🔴 |
|
||||
| settings.json | 🟡 assert merged | ✅ applied | ✅ |
|
||||
| Hook context injection (does it _land_?) | — n/a | ✅ `trace.modelRequests` | ✅ |
|
||||
| Untrusted plugin execution | — n/a | ✅ confined (bwrap, Linux) | 🟡 outer sandbox |
|
||||
|
||||
✅ shipped · 🟡 partial · 🔴 gap · — n/a. Full detail + roadmap: [`research/harness-testing-coverage-matrix.md`](research/harness-testing-coverage-matrix.md).
|
||||
|
||||
|
||||
+208
-23
@@ -143,6 +143,80 @@ assertToolCalls(r, (calls) => /* any custom rule over the list */ true);
|
||||
`assertToolCount` takes `{ min, max, exactly }`; `assertToolCalls` is the escape
|
||||
hatch for a custom invariant like _"every Edit was preceded by a Read"_.
|
||||
|
||||
You can also assert on a tool's **arguments**, not just its name (DeepEval-style)
|
||||
— e.g. the `Edit` targeted the right file, not just that _an_ Edit ran:
|
||||
|
||||
```ts
|
||||
import { assertToolUsedWith } from "vigiles/harness-assert";
|
||||
|
||||
assertToolUsedWith(
|
||||
r,
|
||||
"Edit",
|
||||
(input) => (input as { file_path?: string }).file_path === "src/billing.ts",
|
||||
);
|
||||
```
|
||||
|
||||
## One Trace, two consumers — predicates and assertions
|
||||
|
||||
Both tiers produce one **`Trace`**: the observable record of a run —
|
||||
`toolCalls`, `hooks` (which fired + its decision), `output` (the final answer),
|
||||
`turns`, and `file(p)`. A `runHarnessTest` result _is_ a `Trace`, and so is the
|
||||
`ctx` handed to a `runEval` `measure`. Over that one shape there is one set of
|
||||
**bare predicates** — pure functions returning a value, with **no `assert`
|
||||
prefix and no throw**:
|
||||
|
||||
```ts
|
||||
import {
|
||||
usedTool,
|
||||
toolCount,
|
||||
skillResolved,
|
||||
toolUsedWith,
|
||||
outputContains,
|
||||
hookFired,
|
||||
hookBlocked,
|
||||
} from "vigiles/harness-assert";
|
||||
|
||||
usedTool(trace, "Skill"); // boolean
|
||||
usedTool(trace, /^mcp__github__merge/); // boolean (regex)
|
||||
toolCount(trace, "Write"); // number
|
||||
skillResolved(trace, "demo:greet"); // boolean
|
||||
toolUsedWith(trace, "Edit", (i) => isRightFile(i)); // boolean (tool argument)
|
||||
outputContains(trace, /done/i); // boolean (the agent's final answer)
|
||||
hookFired(trace, "PreToolUse:Edit"); // boolean (recorded from the stream)
|
||||
hookBlocked(trace, "PreToolUse"); // boolean (fired AND exit ≠ 0)
|
||||
```
|
||||
|
||||
`trace.hooks` is **recorded**, not inferred: each `HookFire` (`name`, `event`,
|
||||
`exitCode`, `blocked`, `output`) comes from the CLI's `hook_response` stream
|
||||
events, so a test asserts a hook _actually_ fired and blocked — no marker file
|
||||
the hook had to write. Capture it the same way as `toolCalls` (`transcript:
|
||||
true` on the harness tier; always on at the eval tier). The throwing form is
|
||||
`assertHookFired(trace, name, { blocked: true })`; `assertOutputContains(trace,
|
||||
needle)` does the same for the final answer.
|
||||
|
||||
The two consumers stay **separate** — same vocabulary, never one dual-purpose
|
||||
function:
|
||||
|
||||
- **Testing** asserts (pass/fail, every commit, free). Each `assert*` is just a
|
||||
predicate wrapped in a throw: `assertToolUsed` is `usedTool` + throw,
|
||||
`assertSkillResolved` is `skillResolved` + throw.
|
||||
- **Eval** measures (mean ± se / pass^k, occasional, paid). A `measure` reuses
|
||||
the **bare** predicates directly as metrics:
|
||||
|
||||
```ts
|
||||
measure: (trace) => ({
|
||||
usedSkill: skillResolved(trace, "demo:greet"), // bool → fraction-true + pass^k
|
||||
safe: !usedTool(trace, /merge|delete/), // bool → fraction-true + pass^k
|
||||
});
|
||||
```
|
||||
|
||||
A test can then gate on the result three ways: `assertImproves` (the mean gap
|
||||
beats a threshold), `assertSignificant(report, { baseline, arm, metric })` — a
|
||||
Welch t-test decides whether the gap clears the noise floor (computed from the
|
||||
arms' spread, not hand-fed) — or `assertReliable(report, { arm, metric })`, the
|
||||
metric succeeded on **every** trial (pass^k = 1), the reliability bar for a
|
||||
non-deterministic harness.
|
||||
|
||||
`runEval` arms take `pluginDir` too, so an A/B can be "skill installed" vs "off"
|
||||
and measure **real** activation (the model triggering the skill by its
|
||||
description), superseding the older "tell the agent to read a SKILL.md" trick:
|
||||
@@ -173,21 +247,34 @@ upstream `LICENSE` and a `SOURCE` file recording repo and commit. There is no
|
||||
clone at test time, so they run **offline and deterministically**. Refresh
|
||||
deliberately with [`tools/refresh-vendor.sh`](../tools/refresh-vendor.sh).
|
||||
|
||||
**The safety line: `loadPlugin` parses, it never executes.** A hook is a real
|
||||
child process with full `env` — `runHook`/`runHarnessTest` run the _actual_ hook,
|
||||
not a reimplementation, and there is **no sandbox** beyond a temp cwd and a
|
||||
timeout. That is fine for hooks _you_ wrote and for read-only governance hooks
|
||||
(inspect the event → decide), but a third-party **setup** hook (superpowers'
|
||||
`SessionStart`, say) can install, write, or call out. So the dogfood asserts such
|
||||
a hook is correctly **wired** and stops there; it does not run it.
|
||||
**Safe by default — untrusted hooks are confined, not trusted.** A hook is a
|
||||
real child process: `runHarnessTest` runs the _actual_ hook, not a
|
||||
reimplementation. Code _you_ authored (inline `settings`/`files`) is trusted and
|
||||
runs directly. But an external `plugin` / `pluginDir` brings in **third-party
|
||||
hooks**, and those are confined by default (`sandbox: "auto"`):
|
||||
|
||||
To actually _execute_ untrusted third-party hooks, put a real boundary around it:
|
||||
the cheapest correct one is the **ephemeral CI container** (the runner is the
|
||||
sandbox) — run that job only there, never in a plain local `npm test`. A
|
||||
heavier-weight local sandbox (bubblewrap/`bwrap`, `sandbox-exec`, or Docker) is a
|
||||
reasonable opt-in if you need it, but it is not built into the library today —
|
||||
it's tracked as a potential improvement (an opt-in `sandbox:` option on the
|
||||
execution tiers) in [`research/feature-ideas.md`](../research/feature-ideas.md) §13.
|
||||
- **bubblewrap available** → the run is confined. The mock and `claude` are
|
||||
co-launched inside **one network namespace** (`--unshare-all`): loopback is up
|
||||
so the in-sandbox mock is reachable, but there is **no external route**, so a
|
||||
malicious hook cannot phone home. The filesystem is read-only except the
|
||||
throwaway work dir, a fresh empty `$HOME`, and an IO dir.
|
||||
- **no bubblewrap** → the run **refuses** (throws) rather than executing an
|
||||
untrusted hook unconfined. Install `bwrap`, or pass `sandbox: false` to opt out
|
||||
if you trust the code / the outer container.
|
||||
|
||||
```ts
|
||||
runHarnessTest({ pluginDir: "./vendor/some-plugin", model }); // auto: confined, or refuses
|
||||
runHarnessTest({ pluginDir: "./vendor/audited", model, sandbox: false }); // you vouch for it → direct
|
||||
runHarnessTest({ settings, model, sandbox: "strict" }); // force confinement even for inline
|
||||
```
|
||||
|
||||
The policy lives in [`src/sandbox.ts`](../src/sandbox.ts) (`decideSandbox` is a
|
||||
pure function — untrusted code never runs unconfined unless you typed
|
||||
`sandbox: false`), and the end-to-end test proves egress is blocked while the
|
||||
mock stays reachable. Network egress confinement on a bare laptop comes from the
|
||||
netns; in CI the ephemeral container is an additional boundary. Subtlety: bwrap
|
||||
confines filesystem + network here, not a kernel-exploit boundary — for that you
|
||||
still want the outer container / a microVM.
|
||||
|
||||
## Deterministic tests in your runner
|
||||
|
||||
@@ -260,17 +347,27 @@ and the type augmentation is compile-checked in
|
||||
jest uses the CommonJS dist natively (no ESM flags); the `vigiles/vitest` entry
|
||||
is ESM because vitest is ESM-only.
|
||||
|
||||
**Reliable for:** SessionStart, Stop, UserPromptSubmit, and Bash
|
||||
PreToolUse/PostToolUse — the governance/policy shapes most real plugins use.
|
||||
**Not** for Edit/Write tool-event hooks (headless-gated — drive file actions via
|
||||
Bash, or test those at the eval tier).
|
||||
**Reliable for:** SessionStart, Stop, UserPromptSubmit, and Bash **and
|
||||
Edit/Write** PreToolUse/PostToolUse — the governance/policy shapes most real
|
||||
plugins use (`--allowedTools` allowlists the edit tools past the permission
|
||||
prompt; verified on claude 2.1.169). The events the mock can't trigger —
|
||||
PreCompact, Notification, SessionEnd, SubagentStop — belong to the `runHook`
|
||||
unit tier.
|
||||
|
||||
## Evals — does the change move behaviour?
|
||||
|
||||
`runEval` drives the real model N trials × arm and aggregates: **mean** for
|
||||
numbers, **fraction-true** for booleans, with **std / se** so you can tell a
|
||||
real gap from noise (`formatEvalReport` prints `metric=mean±se`). An arm is a
|
||||
fixture + settings, or a whole `plugin`.
|
||||
real gap from noise, plus **pass^k** (τ-bench) — _did the metric succeed on
|
||||
every trial?_ — the reliability question a non-deterministic harness needs
|
||||
("worked every time" ≠ "worked on average"). `formatEvalReport` prints
|
||||
`metric=mean±se pass^k=…`; each `stat` carries `passK`. An arm is a fixture +
|
||||
settings, or a whole `plugin`.
|
||||
|
||||
The `measure` ctx is a full `Trace`, so a metric can read the agent's
|
||||
**actions** (`ctx.toolCalls`) and its **final answer** (`ctx.output`), not just
|
||||
end-state files — reuse the bare predicates above (`usedTool`, `skillResolved`,
|
||||
…) to compute them.
|
||||
|
||||
```ts
|
||||
import { runEval, formatEvalReport } from "vigiles/eval";
|
||||
@@ -288,11 +385,81 @@ const report = await runEval({
|
||||
}),
|
||||
trials: 6,
|
||||
});
|
||||
console.log(formatEvalReport(report)); // vanilla marked=0.00 gated marked=0.50±0.20
|
||||
console.log(formatEvalReport(report));
|
||||
// vanilla marked=0.00 pass^k=0 gated marked=0.50±0.20 pass^k=0 ($0.07 · 1.2s/run · 4.1k tok)
|
||||
```
|
||||
|
||||
A difference smaller than the combined `se` of the two arms is not yet
|
||||
significant — raise `trials`.
|
||||
(The cost/latency/token suffix and a `— $… total` header appear when the run
|
||||
reports usage; they're silent under the scripted mock.)
|
||||
|
||||
### Significance — is the gap real?
|
||||
|
||||
`se` gives you the spread; **significance** tells you whether the gap clears it.
|
||||
`assertSignificant` runs a Welch's t-test over the two arms' summary stats and
|
||||
throws unless the arm beats the baseline at `alpha` (default 0.05) — the noise
|
||||
floor is **computed**, not hand-fed via `assertImproves(..., { by })`:
|
||||
|
||||
```ts
|
||||
import {
|
||||
assertSignificant,
|
||||
significantlyBeats,
|
||||
compareArms,
|
||||
} from "vigiles/harness-assert";
|
||||
|
||||
assertSignificant(report, {
|
||||
baseline: "vanilla",
|
||||
arm: "gated",
|
||||
metric: "marked",
|
||||
});
|
||||
significantlyBeats(report, "vanilla", "gated", "marked"); // the bare predicate
|
||||
// or: assertImproves(report, { baseline, arm, metric, significant: true });
|
||||
|
||||
const c = compareArms(report, "vanilla", "gated", "marked");
|
||||
// → { delta, seDelta, t, df, pValue, significant } (reads mean/se/n, no raw rows)
|
||||
```
|
||||
|
||||
For 0/1 metrics this is the t approximation to the two-proportion test — close at
|
||||
eval trial counts. An insignificant gap means **raise `trials`** until the noise
|
||||
floor drops below it.
|
||||
|
||||
### Cost, caching, concurrency
|
||||
|
||||
Every run captures **cost / latency / tokens** from the result event: `ctx.usage`
|
||||
(`{ costUsd, durationMs, inputTokens, outputTokens }`) is on the `measure` ctx,
|
||||
`report.arms[a].usage` aggregates per arm, and `report.totalCostUsd` sums the run.
|
||||
Three knobs make a real-model eval cheap enough to run often:
|
||||
|
||||
- `concurrency: N` — run N trials at once (default 1). Rate-limit / overload
|
||||
responses back off and retry automatically (`rateLimitRetries`, `retryBackoffMs`).
|
||||
- `maxCostUsd: N` — stop launching trials once measured cost crosses the cap;
|
||||
in-flight trials finish and `report.aborted` is set.
|
||||
- `cache: "readwrite"` — **record/replay**. Each trial's output _and_ post-run
|
||||
filesystem are recorded under `cacheDir`; a matching re-run replays without
|
||||
calling the model. The key excludes `measure`, so **editing your metric and
|
||||
re-running re-scores for free** — the model is re-called only when a
|
||||
model-affecting input (task, files, settings, model, tools) changes.
|
||||
|
||||
### Trigger rate — does the skill _fire_?
|
||||
|
||||
A skill's value is its description activating on the right task — the #1
|
||||
skill-authoring pain, and a property only the real model decides (the
|
||||
deterministic tier proves the _wiring_; this proves the _activation_).
|
||||
`measureTriggerRate` installs a plugin natively and runs the model over a set of
|
||||
varied prompts, reporting how often a `Trace` predicate holds:
|
||||
|
||||
```ts
|
||||
import { measureTriggerRate, formatTriggerRateReport } from "vigiles/eval";
|
||||
import { skillResolved, assertTriggerRate } from "vigiles/harness-assert";
|
||||
|
||||
const report = await measureTriggerRate({
|
||||
pluginDir: "./my-plugin",
|
||||
prompts: ["…varied tasks the skill should handle…"],
|
||||
fired: (t) => skillResolved(t, "my-plugin:greet"),
|
||||
trials: 2,
|
||||
});
|
||||
console.log(formatTriggerRateReport(report)); // trigger-rate: 80% (10 runs)
|
||||
assertTriggerRate(report, { min: 0.6 }); // gate in CI
|
||||
```
|
||||
|
||||
### LLM-as-judge for subjective outcomes
|
||||
|
||||
@@ -330,6 +497,23 @@ vigiles eval --trials=6 # discover & run *.eval.mjs (forwards VIGILES_TRIAL
|
||||
`vigiles test` needs only the `claude` CLI (no API key) — so it runs the
|
||||
deterministic tier in CI at zero cost. See the repo's `harness` CI job.
|
||||
|
||||
## Coverage
|
||||
|
||||
The suite runs under **vitest** (`npm test` → `vitest run`); `npm run coverage`
|
||||
adds V8 coverage and prints per-file line/branch/function %:
|
||||
|
||||
```bash
|
||||
npm run coverage # vitest run --coverage
|
||||
```
|
||||
|
||||
The deterministic tiers (`runHook`, `runHarnessTest`) and **all the pure eval
|
||||
orchestration** — the loop (`runEvalWith`), the record/replay cache, usage
|
||||
aggregation, and the significance stats — are fully unit-tested via an
|
||||
**injected runner** (canned stream-json, no model). Only the real-`claude`
|
||||
subprocess (`spawnAgent`) is excluded from the gate (exercised by `bench/`);
|
||||
everything around it is covered, so the statement/line/function gate holds at
|
||||
100%.
|
||||
|
||||
## Canonical examples
|
||||
|
||||
- [`examples/harness/hook-unit.harness.mjs`](../examples/harness/hook-unit.harness.mjs) — unit-test a hook's logic with `runHook`, no `claude` CLI (the cheap base of the pyramid).
|
||||
@@ -338,6 +522,7 @@ deterministic tier in CI at zero cost. See the repo's `harness` CI job.
|
||||
- [`examples/harness/real-superpowers.harness.mjs`](../examples/harness/real-superpowers.harness.mjs) — dogfood `loadPlugin` on a real, pinned obra/superpowers snapshot (key-free, offline).
|
||||
- [`examples/harness/real-wshobson.harness.mjs`](../examples/harness/real-wshobson.harness.mjs) — dogfood `loadPlugin` on a real wshobson/agents sub-plugin (the no-hooks marketplace shape).
|
||||
- [`examples/harness/skill-outcome.eval.mjs`](../examples/harness/skill-outcome.eval.mjs) — does a skill change the agent's output?
|
||||
- [`examples/harness/skill-trigger-rate.eval.mjs`](../examples/harness/skill-trigger-rate.eval.mjs) — does a skill's description _fire_ across varied prompts? (`measureTriggerRate`)
|
||||
- [`bench/evals/refs-hook.eval.mjs`](../bench/evals/refs-hook.eval.mjs) — the refs-hook A/B (benchmark #4).
|
||||
|
||||
## See also
|
||||
|
||||
+65
-44
@@ -11,64 +11,85 @@ the file that proves it. Two kinds of coverage:
|
||||
|
||||
## Coverage
|
||||
|
||||
| Use case | Tier | Where |
|
||||
| ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------- |
|
||||
| Hook unit tier — `runHook` (exit codes / stdin event / env / JSON decision) | unit | `src/run-hook.test.ts` |
|
||||
| Hook decision logic (`parseHookOutput` / `decideHook`) | unit | `src/run-hook.test.ts` |
|
||||
| Plugin loader — inline `plugin.json` hooks | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — `hooks` string-path | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — `hooks/hooks.json` convention | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — repo `.claude/settings.json` | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — manifest-wins precedence | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — bare dir (no hooks) | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — `agents/` + `commands/` materialized + surface warnings | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — empty-machine + MCP warnings | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — in-repo dogfood | unit | `src/plugin-loader.test.ts` |
|
||||
| `resolveHarness` — merge / passthrough / undefined | unit | `src/plugin-loader.test.ts` |
|
||||
| Eval aggregation (mean, std, se, n) + report formatting | unit | `src/eval.test.ts` |
|
||||
| Assert helpers (`assertCreated`/`assertNotCreated`/`assertServedTurns`/`assertHookBlocked`/`assertHookAllowed`/`improvement`/`assertImproves`) | unit | `src/harness-assert.test.ts` |
|
||||
| Matchers register + pass under **vitest** (via the `vigiles/vitest` entry) | unit | `test/runners/matchers.vitest.mjs` |
|
||||
| Matchers register + pass under **jest** (via the `vigiles/jest` entry) | unit | `test/runners/matchers.jest.cjs` |
|
||||
| Matcher **types** augment vitest/jest `expect` | unit | `test/types/smoke.vitest.ts`, `smoke.jest.ts` |
|
||||
| CLI runner (`discoverScripts`/`runScripts`/summary) | unit | `src/run-scripts.test.ts` |
|
||||
| Judge verdict parsing (`parseJudgeOutput`) | unit | `src/judge.test.ts` |
|
||||
| `runHarnessTest` end-to-end, incl. `plugin:` | integration (CI) | `examples/harness/policy-gate.harness.mjs`, `plugin-cohesion.harness.mjs` |
|
||||
| `withHarness` (auto-cleanup wrapper) | integration (CI) | `examples/harness/plugin-cohesion.harness.mjs` |
|
||||
| `runEval` end-to-end, incl. `plugin` arm | integration (real model) | `bench/evals/refs-hook.eval.mjs`, `examples/harness/skill-outcome.eval.mjs` |
|
||||
| `judge()` model call | integration (real model) | (parsing is unit-tested; the spawn is not) |
|
||||
| Use case | Tier | Where |
|
||||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------- |
|
||||
| Hook unit tier — `runHook` (exit codes / stdin event / env / JSON decision) | unit | `src/run-hook.test.ts` |
|
||||
| Hook decision logic (`parseHookOutput` / `decideHook`) | unit | `src/run-hook.test.ts` |
|
||||
| Subagent PreToolUse tool-contract rail — `agent-hook` (parse contract / allow-deny / hook ⇄ allowlist agree / real built-CLI hook via `runHook`; grounded on the vendored `ui-visual-validator`) | unit | `src/agent-runtime.test.ts` |
|
||||
| Plugin loader — inline `plugin.json` hooks | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — `hooks` string-path | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — `hooks/hooks.json` convention | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — repo `.claude/settings.json` | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — manifest-wins precedence | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — bare dir (no hooks) | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — `agents/` + `commands/` materialized + surface warnings | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — empty-machine + MCP + dangling-intra-plugin-ref warnings | unit | `src/plugin-loader.test.ts` |
|
||||
| Plugin loader — in-repo dogfood | unit | `src/plugin-loader.test.ts` |
|
||||
| Vendored **real-plugin** conformance (`loadPlugin` invariants, pinned + offline) | unit | `src/vendor.test.ts` |
|
||||
| `resolveHarness` — merge / passthrough / undefined | unit | `src/plugin-loader.test.ts` |
|
||||
| Eval aggregation (mean / std / se / n / pass^k) + report formatting | unit | `src/eval.test.ts` |
|
||||
| Eval usage capture + aggregation (`parseUsage` / `aggregateUsage`, cost/latency/tokens) | unit | `src/eval.test.ts` |
|
||||
| Eval record/replay cache (`cacheKey` / snapshot+restore / replay skips the model) | unit | `src/eval-cache.test.ts`, `src/eval.test.ts` |
|
||||
| Eval concurrency + rate-limit retry + `maxCostUsd` abort (`runPool` / `isRateLimited`) | unit | `src/eval.test.ts` |
|
||||
| Trigger-rate orchestration (`measureTriggerRateWith`) | unit | `src/eval.test.ts` |
|
||||
| Significance stats (`welchTTest` / `compareArms` / `tPValueTwoSided` / incomplete-beta vs t-table) | unit | `src/stats.test.ts` |
|
||||
| Assert/predicate helpers (create/turns/hook-block; tool used/notUsed/count/sequence/with; skillResolved; output/request-contains; hookFired; improvement/improves; reliable; **significant**; **triggerRate**) | unit | `src/harness-assert.test.ts` |
|
||||
| Matchers register + pass under **vitest** (via the `vigiles/vitest` entry) | unit | `test/runners/matchers.vitest.mjs` |
|
||||
| Matchers register + pass under **jest** (via the `vigiles/jest` entry) | unit | `test/runners/matchers.jest.cjs` |
|
||||
| Matcher **types** augment vitest/jest `expect` | unit | `test/types/smoke.vitest.ts`, `smoke.jest.ts` |
|
||||
| CLI runner (`discoverScripts`/`runScripts`/summary) | unit | `src/run-scripts.test.ts` |
|
||||
| Judge verdict parsing (`parseJudgeOutput`) | unit | `src/judge.test.ts` |
|
||||
| `runHarnessTest` end-to-end, incl. `plugin:` | integration (CI) | `examples/harness/policy-gate.harness.mjs`, `plugin-cohesion.harness.mjs` |
|
||||
| `withHarness` (auto-cleanup wrapper) | integration (CI) | `examples/harness/plugin-cohesion.harness.mjs` |
|
||||
| `runEval` end-to-end, incl. `plugin` arm | integration (real model) | `bench/evals/refs-hook.eval.mjs`, `examples/harness/skill-outcome.eval.mjs` |
|
||||
| `measureTriggerRate` end-to-end (skill activation via `pluginDir`) | integration (real model) | `examples/harness/skill-trigger-rate.eval.mjs` |
|
||||
| `judge()` model call | integration (real model) | (parsing is unit-tested; the spawn is not) |
|
||||
|
||||
## Surface coverage — which plugin surface is reachable at which tier
|
||||
|
||||
A Claude Code plugin/repo has several surfaces. They are reachable at different
|
||||
tiers, because some only do anything under a real model:
|
||||
|
||||
| Surface | Hook unit (`runHook`) | Deterministic (`runHarnessTest`) | Eval (`runEval`) |
|
||||
| ------------------------------------------------------------- | --------------------- | --------------------------------- | ------------------ |
|
||||
| Hooks — SessionStart / Stop / UserPromptSubmit | ✅ logic | ✅ fires in machine | ✅ |
|
||||
| Hooks — Bash PreToolUse / PostToolUse | ✅ logic | ✅ fires in machine | ✅ |
|
||||
| Hooks — Edit/Write PreToolUse / PostToolUse | ✅ logic | ⚠ headless-gated (drive via Bash) | ✅ |
|
||||
| Hooks — PreCompact / Notification / SessionEnd / SubagentStop | ✅ logic | — (mock can't trigger) | partial |
|
||||
| CLAUDE.md / instruction files | — | ✅ present in context | ✅ moves behaviour |
|
||||
| Skills | — | ✅ body present (activation n/g) | ✅ |
|
||||
| Subagents (`agents/`) | — | ⚠ materialized, not invoked | ✅ (Task) |
|
||||
| Slash commands (`commands/`) | — | ⚠ materialized, not invoked | ✅ |
|
||||
| MCP servers | — | — (not wired; warned) | bring-your-own |
|
||||
| Surface | Hook unit (`runHook`) | Deterministic (`runHarnessTest`) | Eval (`runEval`) |
|
||||
| ------------------------------------------------------------- | --------------------- | ----------------------------------- | ------------------------------------ |
|
||||
| Hooks — SessionStart / Stop / UserPromptSubmit | ✅ logic | ✅ fires in machine | ✅ |
|
||||
| Hooks — Bash PreToolUse / PostToolUse | ✅ logic | ✅ fires in machine | ✅ |
|
||||
| Hooks — Edit/Write PreToolUse / PostToolUse | ✅ logic | ⚠ headless-gated (drive via Bash) | ✅ |
|
||||
| Hooks — PreCompact / Notification / SessionEnd / SubagentStop | ✅ logic | — (mock can't trigger) | partial |
|
||||
| CLAUDE.md / instruction files | — | ✅ present in context | ✅ moves behaviour |
|
||||
| Skills | — | ✅ resolves via `pluginDir` | ✅ activation (`measureTriggerRate`) |
|
||||
| Subagents (`agents/`) | ✅ tool-contract rail | ⚠ materialized; rail not live-armed | ✅ (Task) |
|
||||
| Slash commands (`commands/`) | — | ⚠ materialized, not invoked | ✅ |
|
||||
| MCP servers | — | — (not wired; warned) | bring-your-own |
|
||||
|
||||
`loadPlugin(...).warnings` surfaces the ⚠ rows for a given plugin, so a "load the
|
||||
whole plugin" test never silently runs an empty machine (e.g. a subagents-only
|
||||
plugin like wshobson/agents `tdd-workflows`, which ships 2 agents + 4 commands
|
||||
and **no** hooks).
|
||||
|
||||
The **subagent tool-contract rail** (`agent-hook`) makes the `agents/` Hook-unit
|
||||
cell `✅`: the generated `PreToolUse` hook that enforces an agent's declared
|
||||
`tools:` is `runHook`-testable like any other hook, and the declared list and the
|
||||
enforced rail are compiled from one source (proven by a round-trip test). The
|
||||
deterministic cell stays ⚠ for an honest reason — Claude Code doesn't surface the
|
||||
active subagent to hooks, so arming the rail in a live session
|
||||
(`.vigiles/active-agent.json`) is still unsolved; the logic is proven at the unit
|
||||
tier, not yet end-to-end in a real dispatch. See `research/subagent-compilation.md`.
|
||||
|
||||
## What is intentionally _not_ unit-tested
|
||||
|
||||
`runHarnessTest`, `runEval`, `withHarness`, and `judge()`'s model call all spawn
|
||||
the real `claude` CLI (and, for evals, a real model). A unit test can't drive
|
||||
that deterministically, so the seam we _can_ pin — the mock model
|
||||
(`src/mock-model.ts`), the loader, the aggregation, the parsing, the matchers —
|
||||
is unit-tested, and the end-to-end behaviour is exercised by the example suite
|
||||
in CI. The deterministic examples (`*.harness.mjs`) run with **no API key** (real
|
||||
`claude` + scripted mock model), so they are CI-affordable; the evals
|
||||
(`*.eval.mjs`) cost real model calls and are run manually / in a keyed job.
|
||||
Only the real-`claude` subprocess is out of the gate: `runHarnessTest`,
|
||||
`withHarness`, `judge()`'s model call, and `runEval`/`measureTriggerRate`'s
|
||||
default `spawnAgent` runner spawn the CLI (and, for evals, a real model). A unit
|
||||
test can't drive that deterministically — but everything _around_ it is pinned.
|
||||
The eval **orchestration** (`runEvalWith` / `measureTriggerRateWith`) takes an
|
||||
**injected runner**, so the loop, cache, concurrency, budget, usage aggregation,
|
||||
and significance run against canned stream-json with no model; the mock model
|
||||
(`src/mock-model.ts`), the loader, the parsing, and the matchers are unit-tested
|
||||
too. End-to-end behaviour is exercised by the example suite in CI: the
|
||||
deterministic examples (`*.harness.mjs`) run with **no API key** (real `claude` +
|
||||
scripted mock model), so they're CI-affordable; the evals (`*.eval.mjs`) cost
|
||||
real model calls and run manually / in a keyed job.
|
||||
|
||||
## Why are the CLI examples `.mjs` (JavaScript), not TypeScript?
|
||||
|
||||
|
||||
@@ -39,6 +39,7 @@ const plugin = fileURLToPath(new URL("./fixture-plugin", import.meta.url));
|
||||
await withHarness(
|
||||
{
|
||||
plugin, // fixture-plugin's real hooks (SessionStart + PreToolUse) + CLAUDE.md
|
||||
sandbox: false, // in-repo fixture we authored → trusted, run direct
|
||||
model: scriptModel([
|
||||
{ tool: "Bash", input: { command: "rm -rf /tmp/should-be-blocked" } },
|
||||
{ tool: "Bash", input: { command: "echo ok > RESULT" } },
|
||||
|
||||
@@ -8,11 +8,13 @@
|
||||
* commit, with upstream LICENSE + SOURCE). There is no clone at test time, so
|
||||
* this runs offline and deterministically — see ./vendor/<plugin>/SOURCE.
|
||||
*
|
||||
* Safety: `loadPlugin` only PARSES the harness — it never runs a hook.
|
||||
* superpowers' hook is a side-effectful SessionStart *setup* script, so this
|
||||
* test asserts it is correctly WIRED and deliberately does NOT execute it.
|
||||
* Executing third-party hooks belongs to a sandboxed tier (the ephemeral CI
|
||||
* container, or a future bwrap/docker boundary), never to a bare `loadPlugin`.
|
||||
* Safety: `loadPlugin` only PARSES the harness — it never runs a hook. This
|
||||
* example verifies superpowers' SessionStart hook is correctly WIRED, key-free.
|
||||
* To actually EXECUTE that untrusted third-party hook, `runHarnessTest` now runs
|
||||
* it CONFINED by default under bubblewrap (`src/sandbox.ts`) — see the dogfood in
|
||||
* `src/sandbox.test.ts`, which runs this same hook in a no-egress sandbox and
|
||||
* checks its real output (and shows, via `trace.modelRequests`, that its
|
||||
* top-level `additionalContext` does NOT reach Claude Code — "fired ≠ landed").
|
||||
*
|
||||
* Pure + key-free: needs neither the `claude` CLI nor an API key, so it runs in
|
||||
* CI for free. Run: `node examples/harness/real-superpowers.harness.mjs`.
|
||||
@@ -59,7 +61,7 @@ console.log(
|
||||
for (const w of loaded.warnings) console.log(` ⚠ ${w}`);
|
||||
console.log(
|
||||
"\nNote: loadPlugin parsed the harness; it did NOT execute the SessionStart\n" +
|
||||
"setup hook (side-effectful). Wiring is verified here; execution is deferred\n" +
|
||||
"to a sandboxed tier. No third-party code ran.",
|
||||
"setup hook. Wiring is verified here; CONFINED execution of this same hook is\n" +
|
||||
"dogfooded in src/sandbox.test.ts (bubblewrap, no egress). No code ran here.",
|
||||
);
|
||||
console.log("\n1 passed.");
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Canonical example — a skill *trigger-rate* eval (`measureTriggerRate`).
|
||||
*
|
||||
* A skill's value is its description firing on the right task — the #1 documented
|
||||
* skill-authoring pain. Wiring (does the Skill tool resolve it) is the
|
||||
* deterministic tier's job; whether the *real model chooses* the skill across
|
||||
* varied phrasings is a property only the model can answer. `measureTriggerRate`
|
||||
* installs a plugin natively (`pluginDir`), runs a set of prompts, and reports
|
||||
* how reliably your `fired` predicate holds.
|
||||
*
|
||||
* npx vigiles eval examples/harness/skill-trigger-rate.eval.mjs
|
||||
* node examples/harness/skill-trigger-rate.eval.mjs 2 # trials per prompt
|
||||
*
|
||||
* Real model → real cost. Needs the `claude` CLI + model auth and a built dist/.
|
||||
* External users import from the package: `from "vigiles/eval"`.
|
||||
*/
|
||||
import {
|
||||
measureTriggerRate,
|
||||
formatTriggerRateReport,
|
||||
} from "../../dist/eval.js";
|
||||
import { skillResolved } from "../../dist/harness-assert.js";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const trials = Number(process.env.VIGILES_TRIALS || process.argv[2] || 1);
|
||||
|
||||
// A real, pinned vendored plugin (no clone at test time). Its TDD skill should
|
||||
// activate when a task is about writing/changing code with tests.
|
||||
const pluginDir = fileURLToPath(
|
||||
new URL("./vendor/superpowers@6fd4507", import.meta.url),
|
||||
);
|
||||
const skill = "superpowers:test-driven-development";
|
||||
|
||||
const report = await measureTriggerRate({
|
||||
pluginDir,
|
||||
prompts: [
|
||||
"Add an `isEven(n)` function to utils.js — write it test-first.",
|
||||
"Implement a stack class in stack.js. Use TDD.",
|
||||
"Fix the off-by-one in paginate(); add a regression test first.",
|
||||
],
|
||||
// reuse a bare predicate: did the model activate the skill (no error)?
|
||||
fired: (t) => skillResolved(t, skill),
|
||||
trials,
|
||||
});
|
||||
|
||||
console.log(formatTriggerRateReport(report));
|
||||
|
||||
// A trigger-rate eval is a measurement, not a hard gate by default — but you can
|
||||
// gate in CI with assertTriggerRate(report, { min: 0.6 }) from vigiles/harness-assert.
|
||||
if (report.n === 0) {
|
||||
throw new Error("no runs executed");
|
||||
}
|
||||
console.log(`\n✓ measured ${report.n} run(s).`);
|
||||
@@ -0,0 +1,30 @@
|
||||
<!-- vigiles:sha256:4e15c156920f8c31 compiled from examples/railway/fixer.md.spec.ts -->
|
||||
|
||||
---
|
||||
|
||||
name: fixer
|
||||
description: Address a failing step's findings, then re-verify. Dispatched by the railway's bounded recovery.
|
||||
model: sonnet
|
||||
tools: Read, Edit, Bash, Grep
|
||||
|
||||
---
|
||||
|
||||
You receive a failing step's error payload (findings or logs)
|
||||
and fix the underlying issue. Re-run `npm test` before reporting success.
|
||||
If you cannot fix it, return a reason so the railway falls to the error track.
|
||||
|
||||
## Output contract
|
||||
|
||||
Finish your turn with exactly one fenced block — success or error — matching one of these shapes.
|
||||
|
||||
On success:
|
||||
|
||||
```vigiles:ok
|
||||
{ "files": string[], "summary": string }
|
||||
```
|
||||
|
||||
On error:
|
||||
|
||||
```vigiles:err
|
||||
{ "reason": string, "retryable": boolean }
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
/**
|
||||
* Example subagent (recovery step): fixer.
|
||||
*
|
||||
* Source of truth for `agents/fixer.md`. The bounded-recovery worker: it takes a
|
||||
* failing step's error payload and tries to address it. The railway caps how
|
||||
* many times it runs (recover.max) — the finite, sub-Turing guarantee.
|
||||
*/
|
||||
import { agent, result, instructions, cmd } from "../../src/spec.js";
|
||||
|
||||
export default agent({
|
||||
name: "fixer",
|
||||
description:
|
||||
"Address a failing step's findings, then re-verify. Dispatched by the railway's bounded recovery.",
|
||||
model: "sonnet",
|
||||
tools: ["Read", "Edit", "Bash", "Grep"],
|
||||
body: instructions`You receive a failing step's error payload (findings or logs)
|
||||
and fix the underlying issue. Re-run ${cmd("npm test")} before reporting success.
|
||||
If you cannot fix it, return a reason so the railway falls to the error track.`,
|
||||
output: result(
|
||||
{ files: "string[]", summary: "string" },
|
||||
{ reason: "string", retryable: "boolean" },
|
||||
),
|
||||
});
|
||||
@@ -0,0 +1,30 @@
|
||||
<!-- vigiles:sha256:df16b875ba61702a compiled from examples/railway/implementer.md.spec.ts -->
|
||||
|
||||
---
|
||||
|
||||
name: implementer
|
||||
description: Implement an approved plan: make the edits and prove the build passes. Dispatch after the planner.
|
||||
model: sonnet
|
||||
tools: Read, Edit, Write, Bash, Grep, Glob
|
||||
|
||||
---
|
||||
|
||||
You implement the plan handed to you, one step at a time. After
|
||||
the edits, run `npm run build` and `npm test`; only report success
|
||||
once both pass. On failure, report where you stopped so the fixer can recover.
|
||||
|
||||
## Output contract
|
||||
|
||||
Finish your turn with exactly one fenced block — success or error — matching one of these shapes.
|
||||
|
||||
On success:
|
||||
|
||||
```vigiles:ok
|
||||
{ "files": string[], "summary": string }
|
||||
```
|
||||
|
||||
On error:
|
||||
|
||||
```vigiles:err
|
||||
{ "failedAt": string, "logs": string, "retryable": boolean }
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
/**
|
||||
* Example subagent (railway step): implementer.
|
||||
*
|
||||
* Source of truth for `agents/implementer.md`. Receives the planner's success
|
||||
* payload, makes the edits, and returns the files it changed — or a structured
|
||||
* failure (where it stopped + whether a retry could help).
|
||||
*/
|
||||
import { agent, result, instructions, cmd } from "../../src/spec.js";
|
||||
|
||||
export default agent({
|
||||
name: "implementer",
|
||||
description:
|
||||
"Implement an approved plan: make the edits and prove the build passes. Dispatch after the planner.",
|
||||
model: "sonnet",
|
||||
tools: ["Read", "Edit", "Write", "Bash", "Grep", "Glob"],
|
||||
body: instructions`You implement the plan handed to you, one step at a time. After
|
||||
the edits, run ${cmd("npm run build")} and ${cmd("npm test")}; only report success
|
||||
once both pass. On failure, report where you stopped so the fixer can recover.`,
|
||||
output: result(
|
||||
{ files: "string[]", summary: "string" },
|
||||
{ failedAt: "string", logs: "string", retryable: "boolean" },
|
||||
),
|
||||
});
|
||||
@@ -0,0 +1,30 @@
|
||||
<!-- vigiles:sha256:29a509e62504dc31 compiled from examples/railway/planner.md.spec.ts -->
|
||||
|
||||
---
|
||||
|
||||
name: planner
|
||||
description: Break a change request into an ordered, reviewable plan. Dispatch FIRST in the ship-pr railway.
|
||||
model: sonnet
|
||||
tools: Read, Grep, Glob
|
||||
|
||||
---
|
||||
|
||||
You turn a change request into a concrete, ordered plan. Read the
|
||||
relevant code first; do not write any. Verify the build is green with `npm run build`
|
||||
before planning around it.
|
||||
|
||||
## Output contract
|
||||
|
||||
Finish your turn with exactly one fenced block — success or error — matching one of these shapes.
|
||||
|
||||
On success:
|
||||
|
||||
```vigiles:ok
|
||||
{ "steps": string[], "summary": string }
|
||||
```
|
||||
|
||||
On error:
|
||||
|
||||
```vigiles:err
|
||||
{ "reason": string, "retryable": boolean }
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
/**
|
||||
* Example subagent (railway step): planner.
|
||||
*
|
||||
* The source of truth — `agents/planner.md` is a compiled build artifact. A flat
|
||||
* worker that returns a typed Result: a plan on success, a reason on failure.
|
||||
* Run `vigiles compile` to regenerate the markdown.
|
||||
*/
|
||||
import { agent, result, instructions, cmd } from "../../src/spec.js";
|
||||
|
||||
export default agent({
|
||||
name: "planner",
|
||||
description:
|
||||
"Break a change request into an ordered, reviewable plan. Dispatch FIRST in the ship-pr railway.",
|
||||
model: "sonnet",
|
||||
tools: ["Read", "Grep", "Glob"],
|
||||
body: instructions`You turn a change request into a concrete, ordered plan. Read the
|
||||
relevant code first; do not write any. Verify the build is green with ${cmd("npm run build")}
|
||||
before planning around it.`,
|
||||
output: result(
|
||||
{ steps: "string[]", summary: "string" },
|
||||
{ reason: "string", retryable: "boolean" },
|
||||
),
|
||||
});
|
||||
@@ -0,0 +1,30 @@
|
||||
<!-- vigiles:sha256:2dba1e6b88ef08b6 compiled from examples/railway/reporter.md.spec.ts -->
|
||||
|
||||
---
|
||||
|
||||
name: reporter
|
||||
description: Summarize a railway failure for a human. Dispatched on the error track when recovery is exhausted.
|
||||
model: haiku
|
||||
tools: Read
|
||||
|
||||
---
|
||||
|
||||
You receive the error payload of the step that failed and the
|
||||
recovery attempts that were exhausted. Write a concise, factual report: what was
|
||||
attempted, where it failed, and what a human should look at next.
|
||||
|
||||
## Output contract
|
||||
|
||||
Finish your turn with exactly one fenced block — success or error — matching one of these shapes.
|
||||
|
||||
On success:
|
||||
|
||||
```vigiles:ok
|
||||
{ "reported": boolean, "summary": string }
|
||||
```
|
||||
|
||||
On error:
|
||||
|
||||
```vigiles:err
|
||||
{ "reason": string, "retryable": boolean }
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
/**
|
||||
* Example subagent (error track): reporter.
|
||||
*
|
||||
* Source of truth for `agents/reporter.md`. The railway's onError handler: it
|
||||
* runs with the failing step's error payload and records the failure clearly so
|
||||
* a human can pick it up.
|
||||
*/
|
||||
import { agent, result, instructions } from "../../src/spec.js";
|
||||
|
||||
export default agent({
|
||||
name: "reporter",
|
||||
description:
|
||||
"Summarize a railway failure for a human. Dispatched on the error track when recovery is exhausted.",
|
||||
model: "haiku",
|
||||
tools: ["Read"],
|
||||
body: instructions`You receive the error payload of the step that failed and the
|
||||
recovery attempts that were exhausted. Write a concise, factual report: what was
|
||||
attempted, where it failed, and what a human should look at next.`,
|
||||
output: result(
|
||||
{ reported: "boolean", summary: "string" },
|
||||
{ reason: "string", retryable: "boolean" },
|
||||
),
|
||||
});
|
||||
@@ -0,0 +1,30 @@
|
||||
<!-- vigiles:sha256:a055143eb4a78162 compiled from examples/railway/reviewer.md.spec.ts -->
|
||||
|
||||
---
|
||||
|
||||
name: reviewer
|
||||
description: Review the implemented diff for correctness. Dispatch LAST on the success track.
|
||||
model: opus
|
||||
tools: Read, Grep, Bash
|
||||
|
||||
---
|
||||
|
||||
You review the diff for correctness and regressions. Re-run
|
||||
`npm test` yourself — do not trust the report. Approve only when the change
|
||||
is correct; otherwise return concrete, actionable findings.
|
||||
|
||||
## Output contract
|
||||
|
||||
Finish your turn with exactly one fenced block — success or error — matching one of these shapes.
|
||||
|
||||
On success:
|
||||
|
||||
```vigiles:ok
|
||||
{ "summary": string }
|
||||
```
|
||||
|
||||
On error:
|
||||
|
||||
```vigiles:err
|
||||
{ "findings": string[], "blocking": boolean }
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
/**
|
||||
* Example subagent (railway step): reviewer.
|
||||
*
|
||||
* Source of truth for `agents/reviewer.md`. The last success-track step: it
|
||||
* either approves (ok) or returns blocking findings (err) that route to the
|
||||
* recovery / error track.
|
||||
*/
|
||||
import { agent, result, instructions, cmd } from "../../src/spec.js";
|
||||
|
||||
export default agent({
|
||||
name: "reviewer",
|
||||
description:
|
||||
"Review the implemented diff for correctness. Dispatch LAST on the success track.",
|
||||
model: "opus",
|
||||
tools: ["Read", "Grep", "Bash"],
|
||||
body: instructions`You review the diff for correctness and regressions. Re-run
|
||||
${cmd("npm test")} yourself — do not trust the report. Approve only when the change
|
||||
is correct; otherwise return concrete, actionable findings.`,
|
||||
output: result(
|
||||
{ summary: "string" },
|
||||
{ findings: "string[]", blocking: "boolean" },
|
||||
),
|
||||
});
|
||||
@@ -0,0 +1,19 @@
|
||||
<!-- vigiles:sha256:aacbda14c3ac1dd1 compiled from examples/railway/ship-pr.md.spec.ts -->
|
||||
|
||||
# Railway: ship-pr
|
||||
|
||||
Dispatch these subagents on the **success track**, in order. Each returns a result block (`vigiles:ok` / `vigiles:err`). If a step returns an error, stop the success track and run the error handler with that error payload.
|
||||
|
||||
## Success track
|
||||
|
||||
1. **planner** — break the request into an ordered plan
|
||||
2. **implementer** — implement the plan; prove build + tests pass
|
||||
3. **reviewer** — review the diff for correctness
|
||||
|
||||
## Recovery
|
||||
|
||||
If a step errors, retry it via **fixer** up to 2× before falling to the error track.
|
||||
|
||||
## On error
|
||||
|
||||
Run **reporter** with the failing step's error payload.
|
||||
@@ -0,0 +1,29 @@
|
||||
/**
|
||||
* Example railway: ship-pr — railway-oriented orchestration over flat subagents.
|
||||
*
|
||||
* Source of truth for `ship-pr.md` (the orchestrator command the lead agent
|
||||
* reads). Each step is a flat worker that returns a typed Result (vigiles:ok /
|
||||
* vigiles:err); the success track flows planner → implementer → reviewer, the
|
||||
* first error short-circuits, bounded recovery retries via the fixer, and an
|
||||
* exhausted failure routes to the reporter. No loop combinator — the value is a
|
||||
* finite tree, so it always terminates and every delegate() target is resolved
|
||||
* against the real agent specs in this directory at compile time.
|
||||
*/
|
||||
import { railway, delegate } from "../../src/spec.js";
|
||||
|
||||
export default railway({
|
||||
name: "ship-pr",
|
||||
steps: [
|
||||
delegate("planner", "break the request into an ordered plan"),
|
||||
delegate("implementer", "implement the plan; prove build + tests pass"),
|
||||
delegate("reviewer", "review the diff for correctness"),
|
||||
],
|
||||
recover: {
|
||||
step: delegate(
|
||||
"fixer",
|
||||
"address the failing step's findings, then re-verify",
|
||||
),
|
||||
max: 2,
|
||||
},
|
||||
onError: delegate("reporter", "report the exhausted failure for a human"),
|
||||
});
|
||||
Generated
+73
@@ -29,6 +29,7 @@
|
||||
"@types/node": "^20.19.39",
|
||||
"@typescript-eslint/eslint-plugin": "^8.58.0",
|
||||
"@typescript-eslint/parser": "^8.58.0",
|
||||
"@vitest/coverage-v8": "^4.1.8",
|
||||
"eslint": "^10.1.0",
|
||||
"eslint-plugin-sonarjs": "^4.0.2",
|
||||
"globals": "^17.4.0",
|
||||
@@ -3333,6 +3334,47 @@
|
||||
"win32"
|
||||
]
|
||||
},
|
||||
"node_modules/@vitest/coverage-v8": {
|
||||
"version": "4.1.8",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-4.1.8.tgz",
|
||||
"integrity": "sha512-lt3kovsyHwYe00wq4D1ti0Z974fWj4NLp6siqiyEufUpyFwK9Yhi7rBhac9JL5aA0zoMrJqc4vYPZRUnI7l7nw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@bcoe/v8-coverage": "^1.0.2",
|
||||
"@vitest/utils": "4.1.8",
|
||||
"ast-v8-to-istanbul": "^1.0.0",
|
||||
"istanbul-lib-coverage": "^3.2.2",
|
||||
"istanbul-lib-report": "^3.0.1",
|
||||
"istanbul-reports": "^3.2.0",
|
||||
"magicast": "^0.5.2",
|
||||
"obug": "^2.1.1",
|
||||
"std-env": "^4.0.0-rc.1",
|
||||
"tinyrainbow": "^3.1.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://opencollective.com/vitest"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@vitest/browser": "4.1.8",
|
||||
"vitest": "4.1.8"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@vitest/browser": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/coverage-v8/node_modules/@bcoe/v8-coverage": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/@bcoe/v8-coverage/-/v8-coverage-1.0.2.tgz",
|
||||
"integrity": "sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/expect": {
|
||||
"version": "4.1.8",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.8.tgz",
|
||||
@@ -3571,6 +3613,25 @@
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/ast-v8-to-istanbul": {
|
||||
"version": "1.0.3",
|
||||
"resolved": "https://registry.npmjs.org/ast-v8-to-istanbul/-/ast-v8-to-istanbul-1.0.3.tgz",
|
||||
"integrity": "sha512-jCMQ6ZylLPudp0CDfBmQBZUsrh1/8psbmu9ibeVWKuHWD0YrH9YABwlKu5kVEFoT0GCQQW9Z/SxfuEbbkGQCRg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@jridgewell/trace-mapping": "^0.3.31",
|
||||
"estree-walker": "^3.0.3",
|
||||
"js-tokens": "^10.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ast-v8-to-istanbul/node_modules/js-tokens": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-10.0.0.tgz",
|
||||
"integrity": "sha512-lM/UBzQmfJRo9ABXbPWemivdCW8V2G8FHaHdypQaIy523snUjog0W71ayWXTjiR+ixeMyVHN2XcpnTd/liPg/Q==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/babel-jest": {
|
||||
"version": "30.4.1",
|
||||
"resolved": "https://registry.npmjs.org/babel-jest/-/babel-jest-30.4.1.tgz",
|
||||
@@ -6127,6 +6188,18 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.5.5"
|
||||
}
|
||||
},
|
||||
"node_modules/magicast": {
|
||||
"version": "0.5.3",
|
||||
"resolved": "https://registry.npmjs.org/magicast/-/magicast-0.5.3.tgz",
|
||||
"integrity": "sha512-pVKE4UdSQ7DvHzivsCIFx2BJn1mHG6KsyrFcaxFx6tONdneEuThrDx0Cj3AMg58KyN4pzYT+LHOotxDQDjNvkw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/parser": "^7.29.3",
|
||||
"@babel/types": "^7.29.0",
|
||||
"source-map-js": "^1.2.1"
|
||||
}
|
||||
},
|
||||
"node_modules/make-dir": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz",
|
||||
|
||||
+4
-2
@@ -40,7 +40,8 @@
|
||||
],
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"test": "npm run build && node --test dist/spec.test.js dist/validate.test.js dist/cli.test.js dist/proofs.test.js dist/inline.test.js dist/sidecar.test.js dist/coverage.test.js dist/session.test.js dist/orphans.test.js dist/cedar.test.js dist/doc-refs.test.js dist/frontmatter.test.js dist/skill-pipeline.test.js dist/skill-runtime.test.js dist/skill-driver.test.js dist/skill-test.test.js dist/compile-generator.test.js dist/community-skills.test.js dist/action-gate.test.js dist/symbols.test.js dist/refs.test.js dist/harness-test.test.js dist/eval.test.js dist/run-scripts.test.js dist/plugin-loader.test.js dist/harness-assert.test.js dist/judge.test.js dist/run-hook.test.js dist/mcp.test.js",
|
||||
"test": "npm run build && vitest run",
|
||||
"coverage": "npm run build && vitest run --coverage",
|
||||
"lint": "eslint src/",
|
||||
"fmt": "prettier --write .",
|
||||
"fmt:check": "prettier --check .",
|
||||
@@ -48,7 +49,7 @@
|
||||
"test:e2e": "bash test/e2e/run.sh",
|
||||
"test:harness": "npm run build && node dist/cli.js test",
|
||||
"test:eval": "npm run build && node dist/cli.js eval",
|
||||
"test:vitest": "npm run build && vitest run",
|
||||
"test:vitest": "npm run build && vitest run --project runners",
|
||||
"test:jest": "npm run build && jest",
|
||||
"test:types": "npm run build && tsc --noEmit -p test/types/tsconfig.json"
|
||||
},
|
||||
@@ -59,6 +60,7 @@
|
||||
"@types/node": "^20.19.39",
|
||||
"@typescript-eslint/eslint-plugin": "^8.58.0",
|
||||
"@typescript-eslint/parser": "^8.58.0",
|
||||
"@vitest/coverage-v8": "^4.1.8",
|
||||
"eslint": "^10.1.0",
|
||||
"eslint-plugin-sonarjs": "^4.0.2",
|
||||
"globals": "^17.4.0",
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* Tests for action gates: deterministic checks bound to a tool action type,
|
||||
* fired regardless of plan order (the dynamic-workflow reframe).
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { evaluateAction, type ActionGate } from "./action-gate.js";
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
/**
|
||||
* Tests for parseAgentResult (src/agent-result.ts) — the railway result parser.
|
||||
* A subagent with a result() contract ends its turn with a vigiles:ok/err block;
|
||||
* this turns that text into a discriminated outcome (ok | err | malformed). Pure,
|
||||
* model-free — the primitive both the orchestrator and the assert helpers reuse.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { parseAgentResult } from "./agent-result.js";
|
||||
import { result } from "./spec.js";
|
||||
|
||||
const okBlock = (json: string) => "Done.\n\n```vigiles:ok\n" + json + "\n```\n";
|
||||
const errBlock = (json: string) =>
|
||||
"Failed.\n\n```vigiles:err\n" + json + "\n```\n";
|
||||
|
||||
test("parses a success block (no contract)", () => {
|
||||
const r = parseAgentResult(
|
||||
okBlock('{ "files": ["a.ts"], "summary": "done" }'),
|
||||
);
|
||||
assert.equal(r.kind, "ok");
|
||||
assert.deepEqual(r.kind === "ok" && r.value, {
|
||||
files: ["a.ts"],
|
||||
summary: "done",
|
||||
});
|
||||
});
|
||||
|
||||
test("parses an error block (no contract)", () => {
|
||||
const r = parseAgentResult(
|
||||
errBlock('{ "reason": "boom", "retryable": true }'),
|
||||
);
|
||||
assert.equal(r.kind, "err");
|
||||
assert.deepEqual(r.kind === "err" && r.error, {
|
||||
reason: "boom",
|
||||
retryable: true,
|
||||
});
|
||||
});
|
||||
|
||||
test("malformed when no result block is present", () => {
|
||||
const r = parseAgentResult("I finished the task, all good!");
|
||||
assert.equal(r.kind, "malformed");
|
||||
assert.match(
|
||||
r.kind === "malformed" ? r.reason : "",
|
||||
/no vigiles:ok\/vigiles:err/,
|
||||
);
|
||||
});
|
||||
|
||||
test("malformed on invalid JSON", () => {
|
||||
const r = parseAgentResult(okBlock("{ not json"));
|
||||
assert.equal(r.kind, "malformed");
|
||||
assert.match(r.kind === "malformed" ? r.reason : "", /invalid JSON/);
|
||||
});
|
||||
|
||||
test("malformed when the block is JSON but not an object", () => {
|
||||
for (const body of ["[1, 2, 3]", "42", '"a string"']) {
|
||||
const r = parseAgentResult(okBlock(body));
|
||||
assert.equal(r.kind, "malformed", `body ${body}`);
|
||||
assert.match(
|
||||
r.kind === "malformed" ? r.reason : "",
|
||||
/must be a JSON object/,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("the LAST block wins (earlier illustrative blocks ignored)", () => {
|
||||
const text =
|
||||
'First I might do:\n```vigiles:ok\n{ "draft": true }\n```\n' +
|
||||
'but actually:\n```vigiles:err\n{ "reason": "nope" }\n```\n';
|
||||
const r = parseAgentResult(text);
|
||||
assert.equal(r.kind, "err");
|
||||
});
|
||||
|
||||
test("contract validation: missing required field → malformed", () => {
|
||||
const c = result(
|
||||
{ files: "string[]", summary: "string" },
|
||||
{ reason: "string" },
|
||||
);
|
||||
const r = parseAgentResult(okBlock('{ "files": ["a.ts"] }'), c);
|
||||
assert.equal(r.kind, "malformed");
|
||||
assert.match(
|
||||
r.kind === "malformed" ? r.reason : "",
|
||||
/missing field "summary"/,
|
||||
);
|
||||
});
|
||||
|
||||
test("contract validation: wrong field type → malformed", () => {
|
||||
const c = result({ summary: "string" }, { reason: "string" });
|
||||
const r = parseAgentResult(okBlock('{ "summary": 123 }'), c);
|
||||
assert.equal(r.kind, "malformed");
|
||||
assert.match(
|
||||
r.kind === "malformed" ? r.reason : "",
|
||||
/"summary" should be string/,
|
||||
);
|
||||
});
|
||||
|
||||
test("contract validation: every field type matches → ok", () => {
|
||||
const c = result(
|
||||
{ s: "string", n: "number", b: "boolean", arr: "string[]" },
|
||||
{ reason: "string" },
|
||||
);
|
||||
const r = parseAgentResult(
|
||||
okBlock('{ "s": "x", "n": 1, "b": false, "arr": ["a", "b"] }'),
|
||||
c,
|
||||
);
|
||||
assert.equal(r.kind, "ok");
|
||||
});
|
||||
|
||||
test("contract validation: string[] rejects a non-array and an array with non-strings", () => {
|
||||
const c = result({ arr: "string[]" }, { reason: "string" });
|
||||
assert.equal(
|
||||
parseAgentResult(okBlock('{ "arr": "x" }'), c).kind,
|
||||
"malformed",
|
||||
);
|
||||
assert.equal(
|
||||
parseAgentResult(okBlock('{ "arr": ["a", 2] }'), c).kind,
|
||||
"malformed",
|
||||
);
|
||||
});
|
||||
|
||||
test("contract validation runs against the err track too", () => {
|
||||
const c = result(
|
||||
{ summary: "string" },
|
||||
{ reason: "string", retryable: "boolean" },
|
||||
);
|
||||
const bad = parseAgentResult(errBlock('{ "reason": "x" }'), c); // missing retryable
|
||||
assert.equal(bad.kind, "malformed");
|
||||
const good = parseAgentResult(
|
||||
errBlock('{ "reason": "x", "retryable": true }'),
|
||||
c,
|
||||
);
|
||||
assert.equal(good.kind, "err");
|
||||
});
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* vigiles — parse a subagent's railway result.
|
||||
*
|
||||
* A subagent with a `result()` contract is told (in its compiled system prompt)
|
||||
* to end its turn with exactly one fenced block:
|
||||
*
|
||||
* ```vigiles:ok
|
||||
* { "files": ["a.ts"], "summary": "done" }
|
||||
* ```
|
||||
*
|
||||
* or `vigiles:err` for the error track. This module extracts and validates that
|
||||
* block — the single primitive the railway orchestrator and the harness-test
|
||||
* assertions (`assertAgentOk`/`assertAgentErr`) both build on. Pure and
|
||||
* model-free: hand it the worker's text, get back a discriminated outcome.
|
||||
*
|
||||
* "Railway-oriented" is literal here: the parse is `text -> Result<S, E>` with a
|
||||
* third `malformed` track for a worker that didn't honor its contract (no block,
|
||||
* bad JSON, or a shape that doesn't match the declared schema).
|
||||
*/
|
||||
|
||||
import type { OutputContract, OutputFieldType } from "./spec.js";
|
||||
|
||||
/** The outcome of parsing a worker's result block. */
|
||||
export type ParsedAgentResult<
|
||||
S = Record<string, unknown>,
|
||||
E = Record<string, unknown>,
|
||||
> =
|
||||
| { readonly kind: "ok"; readonly value: S }
|
||||
| { readonly kind: "err"; readonly error: E }
|
||||
| { readonly kind: "malformed"; readonly reason: string };
|
||||
|
||||
// Capture every vigiles:ok / vigiles:err fenced block; the LAST one is the
|
||||
// worker's final answer (earlier ones may be illustrative in its reasoning).
|
||||
const BLOCK_RE = /```vigiles:(ok|err)[ \t]*\r?\n([\s\S]*?)```/g;
|
||||
|
||||
/** Does a runtime value match a declared field type? */
|
||||
function fieldMatches(value: unknown, type: OutputFieldType): boolean {
|
||||
switch (type) {
|
||||
case "string":
|
||||
return typeof value === "string";
|
||||
case "number":
|
||||
return typeof value === "number";
|
||||
case "boolean":
|
||||
return typeof value === "boolean";
|
||||
case "string[]":
|
||||
return Array.isArray(value) && value.every((v) => typeof v === "string");
|
||||
}
|
||||
}
|
||||
|
||||
/** Validate a parsed object against a contract track; null when it conforms. */
|
||||
function shapeError(
|
||||
obj: Record<string, unknown>,
|
||||
shape: Readonly<Record<string, OutputFieldType>>,
|
||||
): string | null {
|
||||
for (const [field, type] of Object.entries(shape)) {
|
||||
if (!(field in obj)) return `missing field "${field}"`;
|
||||
if (!fieldMatches(obj[field], type)) {
|
||||
return `field "${field}" should be ${type}`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the last `vigiles:ok` / `vigiles:err` block from a worker's output.
|
||||
*
|
||||
* With a `contract`, the parsed object is validated against the matching track's
|
||||
* shape — a worker that emits the wrong shape is `malformed`, not a silent pass.
|
||||
* Without one, any well-formed JSON block is accepted.
|
||||
*/
|
||||
export function parseAgentResult(
|
||||
text: string,
|
||||
contract?: OutputContract,
|
||||
): ParsedAgentResult {
|
||||
BLOCK_RE.lastIndex = 0;
|
||||
let last: { track: "ok" | "err"; body: string } | null = null;
|
||||
for (let m = BLOCK_RE.exec(text); m !== null; m = BLOCK_RE.exec(text)) {
|
||||
last = { track: m[1] as "ok" | "err", body: m[2] };
|
||||
}
|
||||
if (!last) {
|
||||
return {
|
||||
kind: "malformed",
|
||||
reason: "no vigiles:ok/vigiles:err block found",
|
||||
};
|
||||
}
|
||||
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(last.body);
|
||||
} catch {
|
||||
return {
|
||||
kind: "malformed",
|
||||
reason: `invalid JSON in vigiles:${last.track} block`,
|
||||
};
|
||||
}
|
||||
if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
|
||||
return {
|
||||
kind: "malformed",
|
||||
reason: `vigiles:${last.track} block must be a JSON object`,
|
||||
};
|
||||
}
|
||||
const obj = parsed as Record<string, unknown>;
|
||||
|
||||
if (contract) {
|
||||
const shape = last.track === "ok" ? contract.ok : contract.err;
|
||||
const err = shapeError(obj, shape);
|
||||
if (err) {
|
||||
return { kind: "malformed", reason: `${last.track} block: ${err}` };
|
||||
}
|
||||
}
|
||||
|
||||
return last.track === "ok"
|
||||
? { kind: "ok", value: obj }
|
||||
: { kind: "err", error: obj };
|
||||
}
|
||||
@@ -0,0 +1,415 @@
|
||||
/**
|
||||
* Tests for the agent runtime (src/agent-runtime.ts) — the PreToolUse
|
||||
* tool-contract rail. A subagent's `tools:` frontmatter is documentation, not a
|
||||
* runtime boundary (Claude Code #54898); this hook turns it into enforcement by
|
||||
* blocking any tool outside the active agent's compiled allowlist. Model-free:
|
||||
* the decision logic is pure and the runtime ops are plain filesystem.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { agent } from "./spec.js";
|
||||
import { compileAgent } from "./compile.js";
|
||||
import {
|
||||
parseAgentTools,
|
||||
decidePreToolUse,
|
||||
setActiveAgent,
|
||||
readActiveAgent,
|
||||
clearActiveAgent,
|
||||
evaluatePreToolUse,
|
||||
} from "./agent-runtime.js";
|
||||
import { makeTmpDir, cleanupTmpDir } from "./test-utils.js";
|
||||
import { runHook } from "./run-hook.js";
|
||||
import { writeFileSync, readFileSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// parseAgentTools — read the contract back out of compiled markdown
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("parseAgentTools reads the tools list from compiled frontmatter", () => {
|
||||
const { markdown } = compileAgent(
|
||||
agent({
|
||||
name: "reviewer",
|
||||
description: "Review a diff.",
|
||||
tools: ["Read", "Grep", "Bash"],
|
||||
body: "b",
|
||||
}),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(parseAgentTools(markdown), ["Read", "Grep", "Bash"]);
|
||||
});
|
||||
|
||||
test("parseAgentTools returns null when no tools: line (inherit-all)", () => {
|
||||
const { markdown } = compileAgent(
|
||||
agent({ name: "a", description: "d", body: "b" }),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.equal(parseAgentTools(markdown), null);
|
||||
});
|
||||
|
||||
test("parseAgentTools returns [] for an empty tools list", () => {
|
||||
// Hand-built frontmatter: a `tools:` line with nothing after it.
|
||||
const md = [
|
||||
"---",
|
||||
"",
|
||||
"name: a",
|
||||
"description: d",
|
||||
"tools:",
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
"body",
|
||||
].join("\n");
|
||||
assert.deepEqual(parseAgentTools(md), []);
|
||||
});
|
||||
|
||||
test("parseAgentTools returns null when there is no frontmatter", () => {
|
||||
assert.equal(parseAgentTools("# just a heading\n\ntools: Read\n"), null);
|
||||
});
|
||||
|
||||
test("parseAgentTools ignores a `tools:` line in the body, only reads frontmatter", () => {
|
||||
const md = [
|
||||
"---",
|
||||
"name: a",
|
||||
"description: d",
|
||||
"tools: Read",
|
||||
"---",
|
||||
"",
|
||||
"Here the prose mentions tools: Write, Edit which must be ignored.",
|
||||
].join("\n");
|
||||
assert.deepEqual(parseAgentTools(md), ["Read"]);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// decidePreToolUse — the pure rail
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("decidePreToolUse allows a tool inside the contract", () => {
|
||||
const d = decidePreToolUse(["Read", "Grep"], "Read");
|
||||
assert.equal(d.allow, true);
|
||||
assert.equal(d.message, "");
|
||||
});
|
||||
|
||||
test("decidePreToolUse blocks a tool outside the contract, naming the allowlist", () => {
|
||||
const d = decidePreToolUse(["Read", "Grep"], "Write");
|
||||
assert.equal(d.allow, false);
|
||||
assert.match(d.message, /"Write" is not in this subagent's allowed-tools/);
|
||||
assert.match(d.message, /Read, Grep/);
|
||||
});
|
||||
|
||||
test("decidePreToolUse with null allowlist imposes no restriction (inherit-all)", () => {
|
||||
assert.equal(decidePreToolUse(null, "Bash").allow, true);
|
||||
});
|
||||
|
||||
test("decidePreToolUse with an empty allowlist denies everything", () => {
|
||||
const d = decidePreToolUse([], "Read");
|
||||
assert.equal(d.allow, false);
|
||||
assert.match(d.message, /\(none\)/);
|
||||
});
|
||||
|
||||
test("decidePreToolUse matches MCP tools exactly", () => {
|
||||
const allowed = ["Read", "mcp__github__issue_write"];
|
||||
assert.equal(
|
||||
decidePreToolUse(allowed, "mcp__github__issue_write").allow,
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
decidePreToolUse(allowed, "mcp__github__delete_file").allow,
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// active-agent tracking
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("setActiveAgent / readActiveAgent / clearActiveAgent round-trip", () => {
|
||||
const dir = makeTmpDir("agent-active");
|
||||
try {
|
||||
assert.equal(readActiveAgent(dir), null);
|
||||
setActiveAgent(dir, "agents/reviewer.md");
|
||||
assert.equal(readActiveAgent(dir), "agents/reviewer.md");
|
||||
clearActiveAgent(dir);
|
||||
assert.equal(readActiveAgent(dir), null);
|
||||
// clearing again is a no-op, not an error
|
||||
clearActiveAgent(dir);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("readActiveAgent tolerates a malformed marker file", () => {
|
||||
const dir = makeTmpDir("agent-active-bad");
|
||||
try {
|
||||
setActiveAgent(dir, "x"); // creates .vigiles/active-agent.json
|
||||
writeFileSync(join(dir, ".vigiles", "active-agent.json"), "{ not json");
|
||||
assert.equal(readActiveAgent(dir), null);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("readActiveAgent returns null when the marker lacks a string agent field", () => {
|
||||
const dir = makeTmpDir("agent-active-shape");
|
||||
try {
|
||||
setActiveAgent(dir, "x");
|
||||
writeFileSync(
|
||||
join(dir, ".vigiles", "active-agent.json"),
|
||||
JSON.stringify({ agent: 42 }),
|
||||
);
|
||||
assert.equal(readActiveAgent(dir), null);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// evaluatePreToolUse — the wired hook decision against the compiled .md
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("evaluatePreToolUse blocks an out-of-contract tool for the active agent", () => {
|
||||
const dir = makeTmpDir("agent-eval");
|
||||
try {
|
||||
const { markdown } = compileAgent(
|
||||
agent({
|
||||
name: "reviewer",
|
||||
description: "Review a diff.",
|
||||
tools: ["Read", "Grep"], // no Write/Edit/Bash
|
||||
body: "b",
|
||||
}),
|
||||
{ basePath: dir, specFile: "agents/reviewer.md.spec.ts" },
|
||||
);
|
||||
writeFileSync(join(dir, "reviewer.md"), markdown);
|
||||
setActiveAgent(dir, "reviewer.md");
|
||||
|
||||
assert.equal(evaluatePreToolUse(dir, "Read").allow, true);
|
||||
const blocked = evaluatePreToolUse(dir, "Write");
|
||||
assert.equal(blocked.allow, false);
|
||||
assert.match(blocked.message, /allowed-tools contract/);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("evaluatePreToolUse allows anything when no agent is active", () => {
|
||||
const dir = makeTmpDir("agent-eval-none");
|
||||
try {
|
||||
assert.equal(evaluatePreToolUse(dir, "Bash").allow, true);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("evaluatePreToolUse allows when the active agent's .md is missing", () => {
|
||||
const dir = makeTmpDir("agent-eval-missing");
|
||||
try {
|
||||
setActiveAgent(dir, "agents/ghost.md");
|
||||
assert.equal(evaluatePreToolUse(dir, "Write").allow, true);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("evaluatePreToolUse allows everything for an inherit-all agent", () => {
|
||||
const dir = makeTmpDir("agent-eval-inherit");
|
||||
try {
|
||||
const { markdown } = compileAgent(
|
||||
agent({ name: "open", description: "d", body: "b" }), // no tools: line
|
||||
{ basePath: dir, specFile: "agents/open.md.spec.ts" },
|
||||
);
|
||||
writeFileSync(join(dir, "open.md"), markdown);
|
||||
setActiveAgent(dir, "open.md");
|
||||
assert.equal(evaluatePreToolUse(dir, "Bash").allow, true);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// The differentiator's invariant: hook ⇄ allowlist agree
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("the rail the hook enforces is exactly the declared contract (round-trip)", () => {
|
||||
// The whole point of #54898: the `tools:` field documents intent but doesn't
|
||||
// enforce it. vigiles compiles ONE source (spec.tools) into BOTH the
|
||||
// frontmatter (intent) AND the list the PreToolUse hook reads (enforcement),
|
||||
// so the two cannot drift. Prove it: compile → parse the frontmatter the hook
|
||||
// will read → it equals the declared tools, and the hook allows exactly those.
|
||||
const declared = ["Read", "Grep", "Glob", "Bash"];
|
||||
const { markdown } = compileAgent(
|
||||
agent({
|
||||
name: "ui-visual-validator",
|
||||
description: "Validate UI visually.",
|
||||
tools: declared,
|
||||
body: "b",
|
||||
}),
|
||||
{ specFile: "agents/ui-visual-validator.md.spec.ts" },
|
||||
);
|
||||
|
||||
const enforced = parseAgentTools(markdown);
|
||||
assert.deepEqual(enforced, declared); // hook reads exactly what was declared
|
||||
|
||||
for (const t of declared) {
|
||||
assert.equal(decidePreToolUse(enforced, t).allow, true);
|
||||
}
|
||||
for (const t of ["Write", "Edit", "Task", "WebFetch"]) {
|
||||
assert.equal(decidePreToolUse(enforced, t).allow, false); // the least-privilege rail
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// agent-hook (CLI): the real PreToolUse rail process
|
||||
//
|
||||
// Tool-event hooks are best proven at the cheap unit tier (CLAUDE.md: runHook
|
||||
// is "the only tier that reaches every event incl. ... PreToolUse"): pipe a
|
||||
// real synthesized PreToolUse event to the BUILT CLI hook and assert the
|
||||
// block/allow decision — deterministic, no model, no flaky live tool call.
|
||||
// Requires the build (npm test / coverage build it first), like src/cli.test.ts.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const CLI = resolve(__dirname, "..", "dist", "cli.js");
|
||||
|
||||
/** Set up a temp project with a compiled agent and mark it active. */
|
||||
function projectWithActiveAgent(tools: string[]): string {
|
||||
const dir = makeTmpDir("agent-hook-cli");
|
||||
const { markdown } = compileAgent(
|
||||
agent({
|
||||
name: "reader",
|
||||
description: "read-only worker",
|
||||
tools,
|
||||
body: "b",
|
||||
}),
|
||||
{ basePath: dir, specFile: "agents/reader.md.spec.ts" },
|
||||
);
|
||||
writeFileSync(join(dir, "reader.md"), markdown);
|
||||
setActiveAgent(dir, "reader.md");
|
||||
return dir;
|
||||
}
|
||||
|
||||
test("agent-hook CLI blocks (exit 2) an out-of-contract tool", () => {
|
||||
const dir = projectWithActiveAgent(["Read", "Grep"]);
|
||||
try {
|
||||
const r = runHook(
|
||||
`node ${CLI} agent-hook`,
|
||||
{
|
||||
hook_event_name: "PreToolUse",
|
||||
tool_name: "Write",
|
||||
tool_input: { file_path: "x.ts" },
|
||||
},
|
||||
{ cwd: dir },
|
||||
);
|
||||
assert.equal(r.blocked, true);
|
||||
assert.equal(r.exitCode, 2);
|
||||
assert.match(r.stderr, /allowed-tools contract/);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("agent-hook CLI allows (exit 0) an in-contract tool", () => {
|
||||
const dir = projectWithActiveAgent(["Read", "Grep"]);
|
||||
try {
|
||||
const r = runHook(
|
||||
`node ${CLI} agent-hook`,
|
||||
{
|
||||
hook_event_name: "PreToolUse",
|
||||
tool_name: "Read",
|
||||
tool_input: { file_path: "x.ts" },
|
||||
},
|
||||
{ cwd: dir },
|
||||
);
|
||||
assert.equal(r.blocked, false);
|
||||
assert.equal(r.exitCode, 0);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("agent-hook CLI allows when no agent is active", () => {
|
||||
const dir = makeTmpDir("agent-hook-none");
|
||||
try {
|
||||
const r = runHook(
|
||||
`node ${CLI} agent-hook`,
|
||||
{
|
||||
hook_event_name: "PreToolUse",
|
||||
tool_name: "Bash",
|
||||
tool_input: { command: "rm -rf /" },
|
||||
},
|
||||
{ cwd: dir },
|
||||
);
|
||||
assert.equal(r.blocked, false);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("agent-hook CLI allows on a malformed/empty event (no tool name)", () => {
|
||||
const dir = projectWithActiveAgent(["Read"]);
|
||||
try {
|
||||
const r = runHook(`node ${CLI} agent-hook`, {}, { cwd: dir });
|
||||
assert.equal(r.blocked, false);
|
||||
assert.equal(r.exitCode, 0);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Grounded in a REAL vendored subagent (not a synthetic fixture)
|
||||
//
|
||||
// wshobson's ui-visual-validator is pinned under examples/harness/vendor/. It
|
||||
// is the documented footgun in the wild: a "rigorous visual validator" that
|
||||
// "bases judgments solely on visual evidence" yet ships with NO `tools:` line —
|
||||
// so it inherits EVERY tool, including Edit/Write it has no business holding.
|
||||
// These assertions check the rail against that actual file, not a hand-built
|
||||
// one. __dirname is src/ (unit run) or dist/ (built) — both one level under the
|
||||
// repo root, so the relative vendor path resolves either way (matches
|
||||
// src/vendor.test.ts).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const REAL_AGENT = join(
|
||||
__dirname,
|
||||
"../examples/harness/vendor/wshobson-accessibility@cf6059d/agents/ui-visual-validator.md",
|
||||
);
|
||||
|
||||
test("real vendored subagent ships no tools: line — the rail correctly reports it inherits all", () => {
|
||||
const md = readFileSync(REAL_AGENT, "utf-8");
|
||||
// The wild footgun: no contract at all → parseAgentTools returns null →
|
||||
// decidePreToolUse imposes no restriction. The rail honestly reports "there
|
||||
// is no rail here yet" rather than inventing one — which is exactly why the
|
||||
// omitted-tools authoring warning is the next roadmap item.
|
||||
assert.equal(parseAgentTools(md), null);
|
||||
assert.equal(decidePreToolUse(parseAgentTools(md), "Write").allow, true);
|
||||
});
|
||||
|
||||
test("the spec form ADDS the rail the real subagent omits, and it parses + enforces", () => {
|
||||
// Reconstruct the real agent AS a spec with the least-privilege contract its
|
||||
// hand-written original lacks (read + run visual tests; never Edit/Write),
|
||||
// compile it, then prove the SAME PreToolUse rail the hook reads now blocks
|
||||
// the tools the original silently held. This is the differentiator on a real
|
||||
// subagent: compile turns the missing contract into an enforced one.
|
||||
const md = readFileSync(REAL_AGENT, "utf-8");
|
||||
const nameLine = /^name:\s*(.+)$/m.exec(md);
|
||||
const descLine = /^description:\s*(.+)$/m.exec(md);
|
||||
assert.ok(nameLine && descLine); // sanity: we're reading the real frontmatter
|
||||
|
||||
const { markdown, errors } = compileAgent(
|
||||
agent({
|
||||
name: nameLine[1].trim(),
|
||||
description: descLine[1].trim(),
|
||||
model: "sonnet",
|
||||
tools: ["Read", "Grep", "Glob", "Bash"], // the rail the original omits
|
||||
body: "You are an experienced UI visual validation expert.",
|
||||
}),
|
||||
{ specFile: "agents/ui-visual-validator.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(errors, []);
|
||||
|
||||
const enforced = parseAgentTools(markdown);
|
||||
assert.deepEqual(enforced, ["Read", "Grep", "Glob", "Bash"]);
|
||||
assert.equal(decidePreToolUse(enforced, "Bash").allow, true);
|
||||
// the tools the wild original inherited but a visual validator must not hold:
|
||||
assert.equal(decidePreToolUse(enforced, "Write").allow, false);
|
||||
assert.equal(decidePreToolUse(enforced, "Edit").allow, false);
|
||||
});
|
||||
@@ -0,0 +1,163 @@
|
||||
/**
|
||||
* vigiles — Agent runtime: the PreToolUse tool-contract rail.
|
||||
*
|
||||
* A subagent declares an allowed-tools contract in its frontmatter (`tools:`).
|
||||
* But that field is documentation, not a hard runtime boundary (Claude Code
|
||||
* issue #54898): permissions are session-wide, a subagent inherits the parent
|
||||
* session's grants, and `tools:` only filters what's *offered* — it can't deny
|
||||
* what the session allows. The deterministic layer that actually closes the gap
|
||||
* is a **PreToolUse hook** that blocks any tool the active agent's contract
|
||||
* doesn't list.
|
||||
*
|
||||
* This is the same emit-a-hook pattern the skill runtime already ships
|
||||
* (`src/skill-runtime.ts`): there a `Stop` hook reads the active skill's
|
||||
* compiled SKILL.md and runs its result gate; here a `PreToolUse` hook reads
|
||||
* the active agent's compiled `.md`, parses its `tools:` allowlist, and
|
||||
* allows/denies the tool call. The compiled markdown's frontmatter is the
|
||||
* single source of truth — the same list that documents intent IS the list the
|
||||
* hook enforces, so the two agree by construction (see `enforcedTools`).
|
||||
*
|
||||
* Which agent is active is recorded in `.vigiles/active-agent.json` — Claude
|
||||
* Code hooks don't surface the dispatched subagent, so vigiles records it
|
||||
* (mirrors `.vigiles/active-skill.json`). The decision logic below is
|
||||
* harness-agnostic and fully testable.
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
readFileSync,
|
||||
writeFileSync,
|
||||
mkdirSync,
|
||||
rmSync,
|
||||
} from "node:fs";
|
||||
import { resolve, dirname } from "node:path";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Parse the tool contract from a compiled agent .md
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Extract the YAML frontmatter block (between the first pair of `---` fences). */
|
||||
function extractFrontmatter(markdown: string): string | null {
|
||||
const lines = markdown.split("\n");
|
||||
let start = -1;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim() === "---") {
|
||||
start = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (start === -1) return null;
|
||||
for (let i = start + 1; i < lines.length; i++) {
|
||||
if (lines[i].trim() === "---") {
|
||||
return lines.slice(start + 1, i).join("\n");
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse an agent's allowed-tools contract from its compiled markdown.
|
||||
*
|
||||
* Returns the list of allowed tool names, or `null` when the agent declares no
|
||||
* `tools:` line at all — which in Claude Code means it inherits EVERY tool (the
|
||||
* #1 footgun). `null` is the "no restriction" signal the decision logic honors;
|
||||
* an empty list (`tools:` with nothing after it) means "no tools allowed".
|
||||
*/
|
||||
export function parseAgentTools(markdown: string): string[] | null {
|
||||
const fm = extractFrontmatter(markdown);
|
||||
if (fm === null) return null;
|
||||
const match = /^tools:[ \t]*(.*)$/m.exec(fm);
|
||||
if (!match) return null;
|
||||
return match[1]
|
||||
.split(",")
|
||||
.map((t) => t.trim())
|
||||
.filter((t) => t.length > 0);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// The pure decision: is this tool inside the contract?
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface PreToolDecision {
|
||||
/** Whether the tool call is allowed (true) or blocked (false). */
|
||||
readonly allow: boolean;
|
||||
/** Message fed back to the model on a block; empty on allow. */
|
||||
readonly message: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether `tool` is allowed under an agent's tool contract. Pure, so the
|
||||
* rail is unit-testable without spawning anything.
|
||||
*
|
||||
* - `allowed === null` → the agent declared no `tools:` line, so it inherits
|
||||
* everything and the rail imposes no restriction (allow).
|
||||
* - otherwise → allow iff the tool is in the allowlist; deny anything else,
|
||||
* feeding the contract back to the model so it self-corrects.
|
||||
*/
|
||||
export function decidePreToolUse(
|
||||
allowed: readonly string[] | null,
|
||||
tool: string,
|
||||
): PreToolDecision {
|
||||
if (allowed === null) return { allow: true, message: "" };
|
||||
if (allowed.includes(tool)) return { allow: true, message: "" };
|
||||
const list = allowed.length > 0 ? allowed.join(", ") : "(none)";
|
||||
return {
|
||||
allow: false,
|
||||
message:
|
||||
`Tool "${tool}" is not in this subagent's allowed-tools contract ` +
|
||||
`(${list}). Use only the listed tools, or widen the agent's \`tools\`.`,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Active-agent tracking (mirrors .vigiles/active-skill.json)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const ACTIVE_PATH = ".vigiles/active-agent.json";
|
||||
|
||||
/** Record the subagent currently dispatched, so PreToolUse enforces its contract. */
|
||||
export function setActiveAgent(cwd: string, agentPath: string): void {
|
||||
const p = resolve(cwd, ACTIVE_PATH);
|
||||
mkdirSync(dirname(p), { recursive: true });
|
||||
writeFileSync(p, JSON.stringify({ agent: agentPath }) + "\n");
|
||||
}
|
||||
|
||||
/** Clear the active-agent marker (the subagent finished). */
|
||||
export function clearActiveAgent(cwd: string): void {
|
||||
const p = resolve(cwd, ACTIVE_PATH);
|
||||
if (existsSync(p)) rmSync(p);
|
||||
}
|
||||
|
||||
/** The path of the active agent's compiled `.md`, or null when none is active. */
|
||||
export function readActiveAgent(cwd: string): string | null {
|
||||
const p = resolve(cwd, ACTIVE_PATH);
|
||||
if (!existsSync(p)) return null;
|
||||
try {
|
||||
const parsed = JSON.parse(readFileSync(p, "utf-8")) as { agent?: unknown };
|
||||
return typeof parsed.agent === "string" ? parsed.agent : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// PreToolUse-hook decision
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* PreToolUse-hook decision. If an agent is active, parse its compiled `.md`
|
||||
* tool contract and allow the call only when the tool is in the allowlist;
|
||||
* otherwise block and tell the model which tools it may use. With no active
|
||||
* agent (or an agent that inherits all tools), always allow — the rail only
|
||||
* constrains agents that declared a contract.
|
||||
*/
|
||||
export function evaluatePreToolUse(cwd: string, tool: string): PreToolDecision {
|
||||
const agentPath = readActiveAgent(cwd);
|
||||
if (!agentPath) return { allow: true, message: "" };
|
||||
|
||||
const full = resolve(cwd, agentPath);
|
||||
if (!existsSync(full)) return { allow: true, message: "" };
|
||||
|
||||
const allowed = parseAgentTools(readFileSync(full, "utf-8"));
|
||||
return decidePreToolUse(allowed, tool);
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
/**
|
||||
* Tests for subagent spec compilation (src/spec.ts `agent()` + src/compile.ts
|
||||
* `compileAgent`). A subagent is a delegated worker with a contract — a tool
|
||||
* "rail" and rules — so compilation verifies the tool list and the body's
|
||||
* references, and emits frontmatter + an integrity hash. Model-free.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { agent, instructions, file, cmd, enforce, guidance } from "./spec.js";
|
||||
import { compileAgent, adoptDiff } from "./compile.js";
|
||||
import { makeTmpDir, cleanupTmpDir } from "./test-utils.js";
|
||||
|
||||
test("agent() sets the spec type", () => {
|
||||
const a = agent({ name: "reviewer", description: "Review a diff." });
|
||||
assert.equal(a._specType, "agent");
|
||||
assert.equal(a.name, "reviewer");
|
||||
});
|
||||
|
||||
test("compileAgent renders frontmatter (name/description/model/tools) + hash", () => {
|
||||
const { markdown, errors } = compileAgent(
|
||||
agent({
|
||||
name: "reviewer",
|
||||
description: "Review a diff for correctness.",
|
||||
model: "sonnet",
|
||||
tools: ["Read", "Grep", "Bash"],
|
||||
body: "You are a careful code reviewer.",
|
||||
}),
|
||||
{ specFile: "agents/reviewer.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(errors, []);
|
||||
assert.match(markdown, /^<!-- vigiles:sha256:[a-f0-9]+ compiled from/);
|
||||
assert.match(markdown, /\nname: reviewer\n/);
|
||||
assert.match(markdown, /\ndescription: Review a diff for correctness\.\n/);
|
||||
assert.match(markdown, /\nmodel: sonnet\n/);
|
||||
assert.match(markdown, /\ntools: Read, Grep, Bash\n/);
|
||||
assert.match(markdown, /You are a careful code reviewer\./);
|
||||
});
|
||||
|
||||
test("compileAgent: minimal agent omits model/tools and has no rules section", () => {
|
||||
const { markdown, errors } = compileAgent(
|
||||
agent({ name: "echo", description: "Echo things.", body: "Just echo." }),
|
||||
{ specFile: "agents/echo.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(errors, []);
|
||||
assert.doesNotMatch(markdown, /\nmodel:/);
|
||||
assert.doesNotMatch(markdown, /\ntools:/);
|
||||
assert.doesNotMatch(markdown, /## Rules/);
|
||||
});
|
||||
|
||||
test("compileAgent accepts built-in and MCP tools, flags unknown with a hint", () => {
|
||||
const ok = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
tools: ["Read", "Task", "Skill", "mcp__github__issue_write"],
|
||||
body: "b",
|
||||
}),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(ok.errors, []);
|
||||
|
||||
// a near-miss → "did you mean", and a far token → no hint
|
||||
const bad = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
tools: ["Reed", "xyzzy123"],
|
||||
body: "b",
|
||||
}),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.equal(bad.errors.length, 2);
|
||||
const reed = bad.errors.find((e) => e.message.includes('"Reed"'));
|
||||
assert.ok(reed && reed.type === "unknown-tool");
|
||||
assert.match(reed.message, /Did you mean "Read"\?/);
|
||||
const far = bad.errors.find((e) => e.message.includes('"xyzzy123"'));
|
||||
assert.ok(far && !/Did you mean/.test(far.message)); // no close match → no hint
|
||||
});
|
||||
|
||||
test("compileAgent flags tools that are never available to a subagent", () => {
|
||||
const { errors } = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
tools: ["Read", "Agent", "ExitPlanMode"], // last two never reach a subagent
|
||||
body: "b",
|
||||
}),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.equal(errors.length, 2);
|
||||
assert.ok(
|
||||
errors.every((e) => /never available to a subagent/.test(e.message)),
|
||||
);
|
||||
});
|
||||
|
||||
test("compileAgent verifies body references against the filesystem", () => {
|
||||
const dir = makeTmpDir("agent");
|
||||
try {
|
||||
writeFileSync(join(dir, "real.ts"), "export const x = 1;\n");
|
||||
const ok = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
body: instructions`Read ${file("real.ts")}.`,
|
||||
}),
|
||||
{ basePath: dir, specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(ok.errors, []);
|
||||
|
||||
const stale = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
body: instructions`Read ${file("missing.ts")} and run ${cmd("npm run nope")}.`,
|
||||
}),
|
||||
{ basePath: dir, specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.ok(stale.errors.some((e) => e.type === "stale-file"));
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("compileAgent renders a Rules section the worker must follow", () => {
|
||||
const { markdown, errors } = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
rules: {
|
||||
"no-floating": enforce(
|
||||
"@typescript-eslint/no-floating-promises",
|
||||
"Await promises.",
|
||||
),
|
||||
"research-first": guidance("Check the docs before guessing."),
|
||||
},
|
||||
}),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(errors, []);
|
||||
assert.match(markdown, /## Rules/);
|
||||
assert.match(
|
||||
markdown,
|
||||
/\*\*Enforced by:\*\* `@typescript-eslint\/no-floating-promises`/,
|
||||
);
|
||||
assert.match(
|
||||
markdown,
|
||||
/\*\*Guidance only\*\* — Check the docs before guessing\./,
|
||||
);
|
||||
});
|
||||
|
||||
test("compileAgent flags a bad spec filename", () => {
|
||||
const notSpec = compileAgent(agent({ name: "a", description: "d" }), {
|
||||
specFile: "agents/reviewer.md",
|
||||
});
|
||||
assert.ok(notSpec.errors.some((e) => e.type === "spec-name-mismatch"));
|
||||
|
||||
const notMd = compileAgent(agent({ name: "a", description: "d" }), {
|
||||
specFile: "reviewer.spec.ts",
|
||||
});
|
||||
assert.ok(notMd.errors.some((e) => e.type === "spec-name-mismatch"));
|
||||
});
|
||||
|
||||
test("dogfood: a real OSS subagent as a spec, with the tool rail it shipped WITHOUT", () => {
|
||||
// Reproduces the shape of wshobson's real `ui-visual-validator` subagent
|
||||
// (examples/harness/vendor/wshobson-accessibility@.../agents/ui-visual-validator.md):
|
||||
// model: sonnet, a multi-`##`-section role contract, and — critically — it
|
||||
// ships with NO `tools:` line, so it inherits EVERY tool (the #1 footgun). A
|
||||
// spec ADDS the least-privilege rail (read + run visual tests; never Edit/Write),
|
||||
// which compile verifies. This is the value-add over the hand-written original.
|
||||
const reviewer = agent({
|
||||
name: "ui-visual-validator",
|
||||
description:
|
||||
"Rigorous visual validation expert. Use PROACTIVELY to verify UI modifications achieved their goals.",
|
||||
model: "sonnet",
|
||||
tools: ["Read", "Grep", "Glob", "Bash"], // the rail the original omits
|
||||
body: "You are an experienced UI visual validation expert.",
|
||||
sections: {
|
||||
Purpose:
|
||||
"Verify UI modifications, design-system compliance, and accessibility through systematic visual analysis.",
|
||||
"Core Principles": [
|
||||
"- Default assumption: the goal has NOT been achieved until proven.\n",
|
||||
"- Base judgments solely on visual evidence, never code hints.",
|
||||
],
|
||||
"Forbidden Behaviors":
|
||||
"- Assuming code changes automatically produce visual results.\n- Accepting 'looks different' as 'looks correct'.",
|
||||
},
|
||||
});
|
||||
|
||||
const { markdown, errors } = compileAgent(reviewer, {
|
||||
specFile: "agents/ui-visual-validator.md.spec.ts",
|
||||
});
|
||||
|
||||
assert.deepEqual(errors, []); // real content compiles clean; tools verified
|
||||
assert.match(markdown, /\nname: ui-visual-validator\n/);
|
||||
assert.match(markdown, /\nmodel: sonnet\n/);
|
||||
assert.match(markdown, /\ntools: Read, Grep, Glob, Bash\n/); // the added rail
|
||||
assert.doesNotMatch(markdown, /\btools:.*Edit/); // least-privilege: no Edit/Write
|
||||
assert.match(markdown, /## Purpose/);
|
||||
assert.match(markdown, /## Forbidden Behaviors/);
|
||||
assert.match(
|
||||
markdown,
|
||||
/You are an experienced UI visual validation expert\./,
|
||||
);
|
||||
});
|
||||
|
||||
test("compileAgent rejects a section that clashes with the rules field", () => {
|
||||
const { errors } = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
sections: { rules: "this should be the rules field" },
|
||||
}),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.ok(errors.some((e) => e.type === "reserved-section-key"));
|
||||
});
|
||||
|
||||
test("compileAgent verifies refs inside sections", () => {
|
||||
const dir = makeTmpDir("agent-sections");
|
||||
try {
|
||||
const { errors } = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
sections: {
|
||||
Workflow: instructions`First read ${file("missing.ts")}.`, // stale file
|
||||
},
|
||||
}),
|
||||
{ basePath: dir, specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.ok(errors.some((e) => e.type === "stale-file"));
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("adoptDiff round-trips a compiled agent (valid hash, no changes)", () => {
|
||||
const dir = makeTmpDir("agent-adopt");
|
||||
try {
|
||||
const spec = agent({
|
||||
name: "reviewer",
|
||||
description: "Review a diff.",
|
||||
tools: ["Read", "Grep"],
|
||||
body: "Review carefully.",
|
||||
});
|
||||
const { markdown } = compileAgent(spec, {
|
||||
basePath: dir,
|
||||
specFile: "agents/reviewer.md.spec.ts",
|
||||
});
|
||||
writeFileSync(join(dir, "agents-reviewer.md"), markdown);
|
||||
const res = adoptDiff("agents-reviewer.md", spec, dir);
|
||||
assert.equal(res.changed, false);
|
||||
assert.equal(res.hasHash, true);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
import { describe, it } from "node:test";
|
||||
import { describe, it } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
+50
-1
@@ -3,7 +3,7 @@
|
||||
*
|
||||
* These test the full flow: CLI → init/compile/audit → filesystem output.
|
||||
*/
|
||||
import { describe, it, before, after } from "node:test";
|
||||
import { describe, it, beforeAll as before, afterAll as after } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import {
|
||||
mkdtempSync,
|
||||
@@ -126,6 +126,55 @@ describe("CLI: vigiles compile", () => {
|
||||
rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("should compile agent + railway specs and resolve delegate targets", () => {
|
||||
const tmpDir = mkdtempSync(join(tmpdir(), "vigiles-compile-railway-"));
|
||||
writeFileSync(
|
||||
join(tmpDir, "package.json"),
|
||||
JSON.stringify({ name: "test", scripts: { test: "echo ok" } }),
|
||||
);
|
||||
const specSrc = resolve(process.cwd(), "dist", "spec.js");
|
||||
writeFileSync(
|
||||
join(tmpDir, "worker.md.spec.ts"),
|
||||
`import { agent, result } from "${specSrc}";\n` +
|
||||
`export default agent({ name: "worker", description: "d", tools: ["Read"], body: "b", output: result({ summary: "string" }, { reason: "string" }) });\n`,
|
||||
);
|
||||
writeFileSync(
|
||||
join(tmpDir, "flow.md.spec.ts"),
|
||||
`import { railway, delegate } from "${specSrc}";\n` +
|
||||
`export default railway({ name: "flow", steps: [delegate("worker")] });\n`,
|
||||
);
|
||||
const { stdout, exitCode } = run(
|
||||
"compile worker.md.spec.ts flow.md.spec.ts",
|
||||
tmpDir,
|
||||
);
|
||||
assert.equal(exitCode, 0, stdout);
|
||||
const agentMd = readFileSync(join(tmpDir, "worker.md"), "utf8");
|
||||
assert.ok(agentMd.includes("## Output contract"), agentMd);
|
||||
assert.ok(agentMd.includes("```vigiles:ok"), agentMd);
|
||||
const flowMd = readFileSync(join(tmpDir, "flow.md"), "utf8");
|
||||
assert.ok(flowMd.includes("# Railway: flow"), flowMd);
|
||||
assert.ok(flowMd.includes("**worker**"), flowMd);
|
||||
rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("should fail when a railway delegates to an unknown agent", () => {
|
||||
const tmpDir = mkdtempSync(join(tmpdir(), "vigiles-compile-railway-bad-"));
|
||||
writeFileSync(
|
||||
join(tmpDir, "package.json"),
|
||||
JSON.stringify({ name: "test", scripts: { test: "echo ok" } }),
|
||||
);
|
||||
const specSrc = resolve(process.cwd(), "dist", "spec.js");
|
||||
writeFileSync(
|
||||
join(tmpDir, "flow.md.spec.ts"),
|
||||
`import { railway, delegate } from "${specSrc}";\n` +
|
||||
`export default railway({ name: "flow", steps: [delegate("ghost")] });\n`,
|
||||
);
|
||||
const { stdout, exitCode } = run("compile flow.md.spec.ts", tmpDir);
|
||||
assert.equal(exitCode, 1, stdout);
|
||||
assert.ok(stdout.includes("unknown agent"), stdout);
|
||||
rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("should compile subdirectory spec to same directory", () => {
|
||||
const tmpDir = mkdtempSync(join(tmpdir(), "vigiles-compile-subdir-"));
|
||||
writeFileSync(
|
||||
|
||||
+120
-7
@@ -27,19 +27,26 @@ import { ruleSeverity, ruleOptions } from "./types.js";
|
||||
import {
|
||||
compileClaude,
|
||||
compileSkill,
|
||||
compileAgent,
|
||||
compileRailway,
|
||||
checkFileHash,
|
||||
addHash,
|
||||
validateFileRef,
|
||||
validateCommandRef,
|
||||
} from "./compile.js";
|
||||
import type { CompileError } from "./compile.js";
|
||||
import type { ClaudeSpec, SkillSpec } from "./spec.js";
|
||||
import type { ClaudeSpec, SkillSpec, AgentSpec, Railway } from "./spec.js";
|
||||
import { findSimilarRules } from "./proofs.js";
|
||||
import { parseInlineRules } from "./inline.js";
|
||||
import { parseFrontmatterRules } from "./frontmatter.js";
|
||||
import { generateSchema } from "./generate-schema.js";
|
||||
import { compileGeneratorSkill } from "./compile-generator.js";
|
||||
import { evaluateAction, loadActionGates } from "./action-gate.js";
|
||||
import {
|
||||
evaluatePreToolUse,
|
||||
setActiveAgent,
|
||||
clearActiveAgent,
|
||||
} from "./agent-runtime.js";
|
||||
import { verifySymbolRefs, unmarkedCodeRefs } from "./refs.js";
|
||||
import { verifyMcpRefs, loadMcpServers, mcpRefMessage } from "./mcp.js";
|
||||
import {
|
||||
@@ -80,9 +87,9 @@ function findSpecs(pattern?: string): string[] {
|
||||
});
|
||||
}
|
||||
|
||||
async function loadSpec(
|
||||
specPath: string,
|
||||
): Promise<ClaudeSpec | SkillSpec | null> {
|
||||
type AnySpec = ClaudeSpec | SkillSpec | AgentSpec | Railway;
|
||||
|
||||
async function loadSpec(specPath: string): Promise<AnySpec | null> {
|
||||
const fullPath = resolve(process.cwd(), specPath);
|
||||
|
||||
// Try multiple dist/ path strategies
|
||||
@@ -111,12 +118,12 @@ async function loadSpec(
|
||||
if (existsSync(distPath)) {
|
||||
try {
|
||||
const mod = (await import(distPath)) as {
|
||||
default: ClaudeSpec | SkillSpec | { default: ClaudeSpec | SkillSpec };
|
||||
default: AnySpec | { default: AnySpec };
|
||||
};
|
||||
// CJS double-default: `{ default: { default: spec } }`.
|
||||
const raw = mod.default;
|
||||
if (raw && typeof raw === "object" && "default" in raw) {
|
||||
return (raw as { default: ClaudeSpec | SkillSpec }).default;
|
||||
return (raw as { default: AnySpec }).default;
|
||||
}
|
||||
return raw;
|
||||
} catch {
|
||||
@@ -137,7 +144,7 @@ async function loadSpec(
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
timeout: 15000,
|
||||
});
|
||||
return JSON.parse(output.trim()) as ClaudeSpec | SkillSpec;
|
||||
return JSON.parse(output.trim()) as AnySpec;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
@@ -238,11 +245,66 @@ function compileSkillToFile(spec: SkillSpec, specPath: string): boolean {
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Compile a subagent spec → agents/<name>.md (with its result-contract section). */
|
||||
function compileAgentToFile(spec: AgentSpec, specPath: string): boolean {
|
||||
const outputPath = specPath.replace(/\.spec\.ts$/, "");
|
||||
const { markdown, errors } = compileAgent(spec, {
|
||||
basePath: process.cwd(),
|
||||
specFile: specPath,
|
||||
});
|
||||
writeFileSync(resolve(process.cwd(), outputPath), markdown);
|
||||
if (errors.length === 0) {
|
||||
console.log(`\n✓ ${specPath} → ${outputPath}`);
|
||||
return true;
|
||||
}
|
||||
console.log(`\n✗ ${specPath} — ${String(errors.length)} error(s)`);
|
||||
printErrors(specPath, errors);
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile a railway spec → the orchestrator command markdown. `knownAgents` is
|
||||
* the set of compiled agent names in the project, so every `delegate()` target
|
||||
* is resolved at compile time (an unknown target is a stale-ref error).
|
||||
*/
|
||||
function compileRailwayToFile(
|
||||
spec: Railway,
|
||||
specPath: string,
|
||||
knownAgents: readonly string[],
|
||||
): boolean {
|
||||
const outputPath = specPath.replace(/\.spec\.ts$/, "");
|
||||
const { markdown, errors } = compileRailway(spec, {
|
||||
specFile: specPath,
|
||||
knownAgents,
|
||||
});
|
||||
writeFileSync(resolve(process.cwd(), outputPath), markdown);
|
||||
if (errors.length === 0) {
|
||||
console.log(`\n✓ ${specPath} → ${outputPath}`);
|
||||
return true;
|
||||
}
|
||||
console.log(`\n✗ ${specPath} — ${String(errors.length)} error(s)`);
|
||||
printErrors(specPath, errors);
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Names of every compiled agent spec in the project — resolves delegate() targets. */
|
||||
async function collectAgentNames(): Promise<string[]> {
|
||||
const names: string[] = [];
|
||||
for (const p of findSpecs()) {
|
||||
const s = await loadSpec(p);
|
||||
if (s && s._specType === "agent") names.push(s.name);
|
||||
}
|
||||
return names;
|
||||
}
|
||||
|
||||
async function compile(
|
||||
specPaths: string[],
|
||||
config: VigilesConfig,
|
||||
): Promise<boolean> {
|
||||
let allValid = true;
|
||||
// Resolved lazily on the first railway spec — every delegate() target is
|
||||
// checked against the agents defined anywhere in the project.
|
||||
let knownAgents: string[] | null = null;
|
||||
for (const specPath of specPaths) {
|
||||
// Generator skills can't be executed to markdown — compile from source.
|
||||
const source = readFileSync(resolve(process.cwd(), specPath), "utf-8");
|
||||
@@ -263,6 +325,11 @@ async function compile(
|
||||
if (!compileClaudeToFile(spec, specPath, config)) allValid = false;
|
||||
} else if (spec._specType === "skill") {
|
||||
if (!compileSkillToFile(spec, specPath)) allValid = false;
|
||||
} else if (spec._specType === "agent") {
|
||||
if (!compileAgentToFile(spec, specPath)) allValid = false;
|
||||
} else if (spec._specType === "railway") {
|
||||
knownAgents ??= await collectAgentNames();
|
||||
if (!compileRailwayToFile(spec, specPath, knownAgents)) allValid = false;
|
||||
}
|
||||
}
|
||||
return allValid;
|
||||
@@ -2076,6 +2143,43 @@ function skillStartCommand(target: string | undefined): void {
|
||||
console.log(`Active skill: ${target}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* PreToolUse-hook entrypoint: enforce the active subagent's allowed-tools
|
||||
* contract. Reads the tool event on stdin, parses the active agent's compiled
|
||||
* `.md` tool rail, and blocks (exit 2 + reason on stderr) any tool outside it —
|
||||
* the deterministic boundary `tools:` alone can't provide (Claude Code #54898).
|
||||
*/
|
||||
function agentHookCommand(): void {
|
||||
let raw = "";
|
||||
try {
|
||||
raw = readFileSync(0, "utf-8");
|
||||
} catch {
|
||||
/* no stdin */
|
||||
}
|
||||
let tool = "";
|
||||
try {
|
||||
tool = (JSON.parse(raw) as { tool_name?: string }).tool_name ?? "";
|
||||
} catch {
|
||||
/* malformed input → no tool, allow */
|
||||
}
|
||||
if (!tool) return;
|
||||
const decision = evaluatePreToolUse(process.cwd(), tool);
|
||||
if (!decision.allow) {
|
||||
console.error(decision.message);
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
/** Mark a subagent active so the PreToolUse hook enforces its tool contract. */
|
||||
function agentStartCommand(target: string | undefined): void {
|
||||
if (!target) {
|
||||
console.error("Usage: vigiles agent-start <agents/<name>.md>");
|
||||
process.exit(2);
|
||||
}
|
||||
setActiveAgent(process.cwd(), target);
|
||||
console.log(`Active agent: ${target}`);
|
||||
}
|
||||
|
||||
/** Dispatch the skill-runtime subcommands. Returns false if unrecognized. */
|
||||
function handleSkillCommand(command: string, restArgs: string[]): boolean {
|
||||
switch (command) {
|
||||
@@ -2091,6 +2195,15 @@ function handleSkillCommand(command: string, restArgs: string[]): boolean {
|
||||
case "skill-hook":
|
||||
skillHookCommand();
|
||||
return true;
|
||||
case "agent-start":
|
||||
agentStartCommand(restArgs[0]);
|
||||
return true;
|
||||
case "agent-done":
|
||||
clearActiveAgent(process.cwd());
|
||||
return true;
|
||||
case "agent-hook":
|
||||
agentHookCommand();
|
||||
return true;
|
||||
case "action-hook":
|
||||
actionHookCommand();
|
||||
return true;
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
* scripted model and assert their control flow. If these (esp. pr-review-loop)
|
||||
* run and assert cleanly, the generator form + skill-test cover the deep tail.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
* rendering steps / gates / branches / loops to markdown, plus verifying the
|
||||
* gate references it carries.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
|
||||
+332
-3
@@ -14,14 +14,19 @@ import { fileDefinesSymbol, langForFile } from "./symbols.js";
|
||||
import type {
|
||||
ClaudeSpec,
|
||||
SkillSpec,
|
||||
AgentSpec,
|
||||
SkillInput,
|
||||
SkillStep,
|
||||
Gate,
|
||||
Rule,
|
||||
InstructionFragment,
|
||||
OutputContract,
|
||||
OutputFieldType,
|
||||
Railway,
|
||||
RailwayStep,
|
||||
} from "./spec.js";
|
||||
|
||||
import { checkLinterRule, extractLinterName } from "./linters.js";
|
||||
import { checkLinterRule, extractLinterName, editDistance } from "./linters.js";
|
||||
import type { LinterCheckResult } from "./linters.js";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -85,7 +90,9 @@ export interface CompileError {
|
||||
| "section-too-long"
|
||||
| "section-has-header"
|
||||
| "reserved-section-key"
|
||||
| "spec-name-mismatch";
|
||||
| "spec-name-mismatch"
|
||||
| "unknown-tool"
|
||||
| "invalid-railway";
|
||||
message: string;
|
||||
path?: string;
|
||||
}
|
||||
@@ -805,6 +812,325 @@ export function compileSkill(
|
||||
return { markdown: addHash(content, specFile), errors };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Compile a subagent spec → agents/<name>.md
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// The tool contract a subagent may declare — the rails it runs on. Anything
|
||||
// else must be an MCP tool (mcp__server__tool), else it's a typo / nonexistent
|
||||
// tool the dispatched worker could never call.
|
||||
const KNOWN_AGENT_TOOLS = [
|
||||
"Read",
|
||||
"Write",
|
||||
"Edit",
|
||||
"Bash",
|
||||
"Grep",
|
||||
"Glob",
|
||||
"WebSearch",
|
||||
"WebFetch",
|
||||
"NotebookEdit",
|
||||
"TodoWrite",
|
||||
"Task",
|
||||
"Skill",
|
||||
] as const;
|
||||
|
||||
const MCP_TOOL_RE = /^mcp__[a-z0-9_-]+__[a-z0-9_-]+$/i;
|
||||
|
||||
// Tools the platform never exposes to a subagent, whatever the list says — so a
|
||||
// subagent listing one is a guaranteed-dead reference only a compiler catches.
|
||||
const NEVER_AVAILABLE_TOOLS = new Set([
|
||||
"Agent",
|
||||
"AskUserQuestion",
|
||||
"EnterPlanMode",
|
||||
"ExitPlanMode",
|
||||
"ScheduleWakeup",
|
||||
"WaitForMcpServers",
|
||||
]);
|
||||
|
||||
/** Closest known tool by edit distance (≤ 3), for a "did you mean" hint. */
|
||||
function closestTool(tool: string): string | null {
|
||||
let best: string | null = null;
|
||||
let bestDistance = Infinity;
|
||||
for (const known of KNOWN_AGENT_TOOLS) {
|
||||
const d = editDistance(tool.toLowerCase(), known.toLowerCase());
|
||||
if (d < bestDistance) {
|
||||
bestDistance = d;
|
||||
best = known;
|
||||
}
|
||||
}
|
||||
return bestDistance <= 3 ? best : null;
|
||||
}
|
||||
|
||||
/** Verify a subagent's allowed-tools contract — the rails are real tools. */
|
||||
function validateAgentTools(tools: readonly string[]): CompileError[] {
|
||||
const errors: CompileError[] = [];
|
||||
for (const tool of tools) {
|
||||
if (NEVER_AVAILABLE_TOOLS.has(tool)) {
|
||||
errors.push({
|
||||
type: "unknown-tool",
|
||||
message: `Tool "${tool}" is never available to a subagent — remove it from the tools list.`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
if ((KNOWN_AGENT_TOOLS as readonly string[]).includes(tool)) continue;
|
||||
if (MCP_TOOL_RE.test(tool)) continue;
|
||||
const near = closestTool(tool);
|
||||
const hint = near ? ` Did you mean "${near}"?` : "";
|
||||
errors.push({
|
||||
type: "unknown-tool",
|
||||
message: `Unknown tool "${tool}" in agent tools — use a built-in tool (${KNOWN_AGENT_TOOLS.join(", ")}) or an MCP tool (mcp__server__tool).${hint}`,
|
||||
});
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
/** Build the subagent YAML frontmatter (name / description / model / tools). */
|
||||
function renderAgentFrontmatter(spec: AgentSpec): string {
|
||||
const fm = [
|
||||
"---",
|
||||
"",
|
||||
`name: ${spec.name}`,
|
||||
`description: ${spec.description}`,
|
||||
];
|
||||
if (spec.model !== undefined) fm.push(`model: ${spec.model}`);
|
||||
if (spec.tools && spec.tools.length > 0) {
|
||||
fm.push(`tools: ${spec.tools.join(", ")}`);
|
||||
}
|
||||
fm.push("", "---");
|
||||
return fm.join("\n");
|
||||
}
|
||||
|
||||
/** Render the subagent's named `##` system-prompt sections (verified like CLAUDE.md). */
|
||||
function renderAgentSections(
|
||||
sections: Record<string, string | InstructionFragment[]>,
|
||||
basePath: string,
|
||||
): SectionResult {
|
||||
const lines: string[] = [];
|
||||
const errors: CompileError[] = [];
|
||||
for (const [name, content] of Object.entries(sections)) {
|
||||
if (name.toLowerCase() === "rules") {
|
||||
errors.push({
|
||||
type: "reserved-section-key",
|
||||
message: `Section key "${name}" is reserved — use the \`rules\` field instead.`,
|
||||
});
|
||||
}
|
||||
const heading = name.charAt(0).toUpperCase() + name.slice(1);
|
||||
if (typeof content === "string") {
|
||||
errors.push(...validateSectionContent(name, content));
|
||||
lines.push(`## ${heading}\n\n${content.trim()}`);
|
||||
} else {
|
||||
errors.push(...validateRefs(content, basePath));
|
||||
const rendered = content.map(renderFragment).join("");
|
||||
errors.push(...validateSectionContent(name, rendered));
|
||||
lines.push(`## ${heading}\n\n${rendered.trim()}`);
|
||||
}
|
||||
}
|
||||
return { lines, errors };
|
||||
}
|
||||
|
||||
/** Render a result-contract track shape as a compact `{ "f": type, … }` line. */
|
||||
function renderShape(shape: Readonly<Record<string, OutputFieldType>>): string {
|
||||
const fields = Object.entries(shape)
|
||||
.map(([k, t]) => `"${k}": ${t}`)
|
||||
.join(", ");
|
||||
return fields ? `{ ${fields} }` : "{}";
|
||||
}
|
||||
|
||||
/**
|
||||
* Render the subagent's typed result contract — the `## Output contract` section
|
||||
* that turns a flat worker into a railway step: it must end its turn with a
|
||||
* `vigiles:ok` / `vigiles:err` block matching one of these shapes, so its
|
||||
* outcome is parseable (`parseAgentResult`) and testable (`assertAgentOk`).
|
||||
*/
|
||||
function renderOutputContract(contract: OutputContract): string {
|
||||
return [
|
||||
"## Output contract",
|
||||
"",
|
||||
"Finish your turn with exactly one fenced block — success or error — matching one of these shapes.",
|
||||
"",
|
||||
"On success:",
|
||||
"",
|
||||
"```vigiles:ok",
|
||||
renderShape(contract.ok),
|
||||
"```",
|
||||
"",
|
||||
"On error:",
|
||||
"",
|
||||
"```vigiles:err",
|
||||
renderShape(contract.err),
|
||||
"```",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
/** Render the rules a subagent must follow as a `## Rules` section. */
|
||||
function renderAgentRules(rules: Record<string, Rule>): string {
|
||||
const parts = ["## Rules", ""];
|
||||
for (const [id, rule] of Object.entries(rules)) {
|
||||
parts.push(compileRule(id, rule), "");
|
||||
}
|
||||
return parts.join("\n").trim();
|
||||
}
|
||||
|
||||
export interface CompileAgentResult {
|
||||
markdown: string;
|
||||
errors: CompileError[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile an AgentSpec into a subagent markdown file with YAML frontmatter.
|
||||
* Verifies the tool contract and the body's references; the marks the body
|
||||
* carries (`vigiles:symbol`, file/cmd refs) are the same ones `audit` re-checks.
|
||||
*/
|
||||
export function compileAgent(
|
||||
spec: AgentSpec,
|
||||
options: { basePath?: string; specFile?: string } = {},
|
||||
): CompileAgentResult {
|
||||
const basePath = options.basePath ?? process.cwd();
|
||||
const specFile = options.specFile ?? "agent.md.spec.ts";
|
||||
const errors: CompileError[] = [];
|
||||
|
||||
if (!specFile.endsWith(".spec.ts")) {
|
||||
errors.push({
|
||||
type: "spec-name-mismatch",
|
||||
message: `Spec file "${specFile}" must end with .spec.ts`,
|
||||
});
|
||||
} else if (!/\.md$/i.test(basename(specFile, ".spec.ts"))) {
|
||||
errors.push({
|
||||
type: "spec-name-mismatch",
|
||||
message: `Spec file "${specFile}" should be named <output>.spec.ts (e.g., agents/reviewer.md.spec.ts)`,
|
||||
});
|
||||
}
|
||||
|
||||
if (spec.tools) errors.push(...validateAgentTools(spec.tools));
|
||||
if (Array.isArray(spec.body)) {
|
||||
errors.push(...validateRefs(spec.body, basePath));
|
||||
}
|
||||
|
||||
const sections: string[] = [];
|
||||
if (spec.body !== undefined) sections.push(renderBody(spec.body).trim());
|
||||
if (spec.sections) {
|
||||
const result = renderAgentSections(spec.sections, basePath);
|
||||
sections.push(...result.lines);
|
||||
errors.push(...result.errors);
|
||||
}
|
||||
if (spec.rules && Object.keys(spec.rules).length > 0) {
|
||||
sections.push(renderAgentRules(spec.rules));
|
||||
}
|
||||
if (spec.output) sections.push(renderOutputContract(spec.output));
|
||||
const body = sections.join("\n\n");
|
||||
errors.push(...checkInlineCode(body, DEFAULT_MAX_INLINE_CODE_LINES));
|
||||
|
||||
const content = renderAgentFrontmatter(spec) + "\n\n" + body.trim() + "\n";
|
||||
return { markdown: addHash(content, specFile), errors };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Compile a railway → an orchestrator command
|
||||
//
|
||||
// A railway composes flat subagents on a success/error track. It compiles to an
|
||||
// orchestrator command the lead agent reads — NOT a runtime engine (vigiles
|
||||
// verifies + emits; the agent executes; the per-step rails enforce). Every
|
||||
// delegate target is resolved against the known agent set (stale-ref), the
|
||||
// step list must be non-empty, and recovery must be bounded — the finite,
|
||||
// sub-Turing guarantees that make the whole flow checkable.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface CompileRailwayOptions {
|
||||
/** Names of compiled agents, to resolve `delegate` targets. Skipped if omitted. */
|
||||
knownAgents?: readonly string[];
|
||||
specFile?: string;
|
||||
}
|
||||
|
||||
export interface CompileRailwayResult {
|
||||
markdown: string;
|
||||
errors: CompileError[];
|
||||
}
|
||||
|
||||
/** Verify a railway: non-empty, bounded recovery, every delegate target real. */
|
||||
export function validateRailway(
|
||||
rw: Railway,
|
||||
knownAgents?: readonly string[],
|
||||
): CompileError[] {
|
||||
const errors: CompileError[] = [];
|
||||
if (rw.steps.length === 0) {
|
||||
errors.push({
|
||||
type: "invalid-railway",
|
||||
message: `Railway "${rw.name}" has no steps.`,
|
||||
});
|
||||
}
|
||||
if (rw.recover && rw.recover.max < 1) {
|
||||
errors.push({
|
||||
type: "invalid-railway",
|
||||
message: `Railway "${rw.name}" recover.max must be ≥ 1 (got ${String(rw.recover.max)}).`,
|
||||
});
|
||||
}
|
||||
if (knownAgents) {
|
||||
const known = new Set(knownAgents);
|
||||
const refs: RailwayStep[] = [...rw.steps];
|
||||
if (rw.onError) refs.push(rw.onError);
|
||||
if (rw.recover) refs.push(rw.recover.step);
|
||||
for (const s of refs) {
|
||||
if (!known.has(s.agent)) {
|
||||
errors.push({
|
||||
type: "stale-ref",
|
||||
message: `Railway "${rw.name}" delegates to unknown agent "${s.agent}".`,
|
||||
path: s.agent,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
/** Render the orchestrator command markdown for a railway. */
|
||||
function renderRailwayMarkdown(rw: Railway): string {
|
||||
const lines = [
|
||||
`# Railway: ${rw.name}`,
|
||||
"",
|
||||
"Dispatch these subagents on the **success track**, in order. Each returns a " +
|
||||
"result block (`vigiles:ok` / `vigiles:err`). If a step returns an error, " +
|
||||
"stop the success track and run the error handler with that error payload.",
|
||||
"",
|
||||
"## Success track",
|
||||
"",
|
||||
];
|
||||
rw.steps.forEach((s, i) => {
|
||||
const task = s.task ? ` — ${s.task}` : "";
|
||||
lines.push(`${String(i + 1)}. **${s.agent}**${task}`);
|
||||
});
|
||||
if (rw.recover) {
|
||||
lines.push(
|
||||
"",
|
||||
"## Recovery",
|
||||
"",
|
||||
`If a step errors, retry it via **${rw.recover.step.agent}** up to ${String(rw.recover.max)}× before falling to the error track.`,
|
||||
);
|
||||
}
|
||||
if (rw.onError) {
|
||||
lines.push(
|
||||
"",
|
||||
"## On error",
|
||||
"",
|
||||
`Run **${rw.onError.agent}** with the failing step's error payload.`,
|
||||
);
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile a railway into an orchestrator command markdown (with integrity hash),
|
||||
* resolving every delegate target against `knownAgents` when provided.
|
||||
*/
|
||||
export function compileRailway(
|
||||
rw: Railway,
|
||||
options: CompileRailwayOptions = {},
|
||||
): CompileRailwayResult {
|
||||
const errors = validateRailway(rw, options.knownAgents);
|
||||
const specFile = options.specFile ?? `${rw.name}.railway.spec.ts`;
|
||||
const content = renderRailwayMarkdown(rw) + "\n";
|
||||
return { markdown: addHash(content, specFile), errors };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Hash check for existing files
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -850,7 +1176,7 @@ export interface AdoptResult {
|
||||
*/
|
||||
export function adoptDiff(
|
||||
filePath: string,
|
||||
spec: ClaudeSpec | SkillSpec,
|
||||
spec: ClaudeSpec | SkillSpec | AgentSpec,
|
||||
basePath: string,
|
||||
): AdoptResult {
|
||||
const fullPath = resolve(basePath, filePath);
|
||||
@@ -868,6 +1194,9 @@ export function adoptDiff(
|
||||
} else if (spec._specType === "skill") {
|
||||
const { markdown } = compileSkill(spec, { basePath, specFile: filePath });
|
||||
compiledContent = markdown;
|
||||
} else if (spec._specType === "agent") {
|
||||
const { markdown } = compileAgent(spec, { basePath, specFile: filePath });
|
||||
compiledContent = markdown;
|
||||
}
|
||||
|
||||
// Simple line-based diff
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { describe, it, before, after } from "node:test";
|
||||
import { describe, it, beforeAll as before, afterAll as after } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { describe, it } from "node:test";
|
||||
import { describe, it } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* Tests for the eval record/replay cache (src/eval-cache.ts) — the pure key,
|
||||
* record I/O, and filesystem snapshot/restore. Model-free.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync, mkdirSync, existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
cacheKey,
|
||||
readCache,
|
||||
writeCache,
|
||||
snapshotDir,
|
||||
restoreDir,
|
||||
type CacheKeyInput,
|
||||
} from "./eval-cache.js";
|
||||
import { makeTmpDir, cleanupTmpDir } from "./test-utils.js";
|
||||
|
||||
const baseKey: CacheKeyInput = {
|
||||
task: "do it",
|
||||
model: "haiku",
|
||||
tools: ["Read", "Edit"],
|
||||
files: { "a.txt": "x" },
|
||||
settings: { hooks: {} },
|
||||
trialIndex: 0,
|
||||
};
|
||||
|
||||
test("cacheKey is stable and order-independent for object inputs", () => {
|
||||
const k1 = cacheKey(baseKey);
|
||||
// same logical input, object keys in a different order → same key
|
||||
const k2 = cacheKey({
|
||||
trialIndex: 0,
|
||||
settings: { hooks: {} },
|
||||
files: { "a.txt": "x" },
|
||||
tools: ["Read", "Edit"],
|
||||
model: "haiku",
|
||||
task: "do it",
|
||||
});
|
||||
assert.equal(k1, k2);
|
||||
});
|
||||
|
||||
test("cacheKey changes when any model-affecting input changes", () => {
|
||||
const base = cacheKey(baseKey);
|
||||
assert.notEqual(base, cacheKey({ ...baseKey, trialIndex: 1 }));
|
||||
assert.notEqual(base, cacheKey({ ...baseKey, task: "other" }));
|
||||
assert.notEqual(base, cacheKey({ ...baseKey, files: { "a.txt": "y" } }));
|
||||
// tool order is significant (arrays keep order)
|
||||
assert.notEqual(base, cacheKey({ ...baseKey, tools: ["Edit", "Read"] }));
|
||||
});
|
||||
|
||||
test("readCache returns null on miss and on malformed records", () => {
|
||||
const dir = makeTmpDir("cache");
|
||||
try {
|
||||
const key = cacheKey(baseKey);
|
||||
assert.equal(readCache(dir, key), null); // miss
|
||||
mkdirSync(dir, { recursive: true });
|
||||
writeFileSync(join(dir, `${key}.json`), "{ not json");
|
||||
assert.equal(readCache(dir, key), null); // malformed
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("writeCache then readCache round-trips a record", () => {
|
||||
const dir = makeTmpDir("cache");
|
||||
try {
|
||||
const key = cacheKey(baseKey);
|
||||
const record = {
|
||||
out: { code: 0, stdout: "stream" },
|
||||
files: { "OUT.txt": "result" },
|
||||
};
|
||||
writeCache(dir, key, record);
|
||||
const got = readCache(dir, key);
|
||||
assert.deepEqual(got, record);
|
||||
} finally {
|
||||
cleanupTmpDir(dir);
|
||||
}
|
||||
});
|
||||
|
||||
test("snapshotDir captures nested text files; restoreDir rebuilds them", () => {
|
||||
const src = makeTmpDir("snap-src");
|
||||
const dst = makeTmpDir("snap-dst");
|
||||
try {
|
||||
mkdirSync(join(src, "sub"), { recursive: true });
|
||||
writeFileSync(join(src, "top.txt"), "top");
|
||||
writeFileSync(join(src, "sub", "deep.txt"), "deep");
|
||||
// node_modules is skipped
|
||||
mkdirSync(join(src, "node_modules"), { recursive: true });
|
||||
writeFileSync(join(src, "node_modules", "skip.txt"), "skip");
|
||||
|
||||
const snap = snapshotDir(src);
|
||||
assert.equal(snap["top.txt"], "top");
|
||||
assert.equal(snap[join("sub", "deep.txt")], "deep");
|
||||
assert.ok(
|
||||
!(join("node_modules", "skip.txt") in snap),
|
||||
"node_modules skipped",
|
||||
);
|
||||
|
||||
restoreDir(dst, snap);
|
||||
assert.ok(existsSync(join(dst, "top.txt")));
|
||||
assert.ok(existsSync(join(dst, "sub", "deep.txt")));
|
||||
} finally {
|
||||
cleanupTmpDir(src);
|
||||
cleanupTmpDir(dst);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* vigiles — record/replay cache for the eval tier.
|
||||
*
|
||||
* A real-model eval is slow and costs money, yet most iteration is on the
|
||||
* `measure` function, not the model call. This cache records each trial's raw
|
||||
* output AND its post-run filesystem, keyed on everything that determines the
|
||||
* model's behaviour — `task`, the resolved fixture files + settings, model,
|
||||
* tools, and the trial index — but DELIBERATELY NOT the `measure` function. So
|
||||
* editing your metric and re-running re-scores the captured runs for free; the
|
||||
* model is only re-called when a model-affecting input changes (or `cache:"off"`,
|
||||
* which always re-samples for a fresh statistic).
|
||||
*
|
||||
* Restoring the post-run filesystem is what makes replay *sound*: `measure`
|
||||
* routinely reads agent-produced files via `ctx.file()` / `ctx.sh("grep …")`, so
|
||||
* a stdout-only cache would silently mis-score on replay. We snapshot the cwd's
|
||||
* text files after the run and restore them into a fresh dir before re-scoring.
|
||||
*/
|
||||
import {
|
||||
readFileSync,
|
||||
writeFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
statSync,
|
||||
} from "node:fs";
|
||||
import { join, relative, resolve, dirname } from "node:path";
|
||||
|
||||
import { sha256short, type SHA256Hash } from "./hash.js";
|
||||
import type { RunOut } from "./eval.js";
|
||||
|
||||
/** Cache behaviour: never touch the cache / read-only / read-and-write. */
|
||||
export type CacheMode = "off" | "read" | "readwrite";
|
||||
|
||||
/** Everything that determines a trial's model output (the cache key inputs). */
|
||||
export interface CacheKeyInput {
|
||||
readonly task: string;
|
||||
readonly model: string;
|
||||
readonly tools: readonly string[];
|
||||
/** The resolved fixture + arm + plugin files written before the run. */
|
||||
readonly files: Record<string, string>;
|
||||
/** The resolved `.claude/settings.json` for the arm (or undefined). */
|
||||
readonly settings: unknown;
|
||||
/** Which trial this is — distinct trials are distinct samples, cached apart. */
|
||||
readonly trialIndex: number;
|
||||
}
|
||||
|
||||
/** A recorded trial: its raw output plus the post-run cwd snapshot. */
|
||||
export interface CacheRecord {
|
||||
readonly out: RunOut;
|
||||
/** Text files present in the cwd after the run (relative path → contents). */
|
||||
readonly files: Record<string, string>;
|
||||
}
|
||||
|
||||
const MAX_SNAPSHOT_FILE_BYTES = 1024 * 1024;
|
||||
const SKIP_DIRS = new Set(["node_modules", ".git"]);
|
||||
|
||||
/**
|
||||
* Canonicalize a value so the key is stable regardless of object key order —
|
||||
* recursively sorts object keys. Arrays keep order (it's significant for tools).
|
||||
*/
|
||||
function canonical(value: unknown): unknown {
|
||||
if (Array.isArray(value)) return value.map(canonical);
|
||||
if (value !== null && typeof value === "object") {
|
||||
const obj = value as Record<string, unknown>;
|
||||
const out: Record<string, unknown> = {};
|
||||
for (const k of Object.keys(obj).sort()) out[k] = canonical(obj[k]);
|
||||
return out;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/** Deterministic content hash of the key inputs (order-independent). */
|
||||
export function cacheKey(input: CacheKeyInput): SHA256Hash {
|
||||
return sha256short(JSON.stringify(canonical(input)));
|
||||
}
|
||||
|
||||
/** Read a cached record by key, or null on miss / unreadable / malformed. */
|
||||
export function readCache(dir: string, key: SHA256Hash): CacheRecord | null {
|
||||
const path = join(dir, `${key}.json`);
|
||||
if (!existsSync(path)) return null;
|
||||
try {
|
||||
return JSON.parse(readFileSync(path, "utf-8")) as CacheRecord;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Write a cached record by key (creating the cache dir as needed). */
|
||||
export function writeCache(
|
||||
dir: string,
|
||||
key: SHA256Hash,
|
||||
record: CacheRecord,
|
||||
): void {
|
||||
mkdirSync(dir, { recursive: true });
|
||||
writeFileSync(join(dir, `${key}.json`), JSON.stringify(record));
|
||||
}
|
||||
|
||||
/** Snapshot the text files under `cwd` as `relativePath → contents` (bounded). */
|
||||
export function snapshotDir(cwd: string): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
const walk = (dir: string): void => {
|
||||
for (const entry of readdirSync(dir)) {
|
||||
if (SKIP_DIRS.has(entry)) continue;
|
||||
const full = join(dir, entry);
|
||||
const st = statSync(full);
|
||||
if (st.isDirectory()) walk(full);
|
||||
else if (st.isFile() && st.size <= MAX_SNAPSHOT_FILE_BYTES) {
|
||||
out[relative(cwd, full)] = readFileSync(full, "utf-8");
|
||||
}
|
||||
}
|
||||
};
|
||||
walk(resolve(cwd));
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Restore a snapshot into `cwd`, recreating directories as needed. */
|
||||
export function restoreDir(cwd: string, files: Record<string, string>): void {
|
||||
for (const [rel, content] of Object.entries(files)) {
|
||||
const full = resolve(cwd, rel);
|
||||
mkdirSync(dirname(full), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
}
|
||||
+467
-9
@@ -1,12 +1,34 @@
|
||||
/**
|
||||
* Tests for the eval aggregation/formatting (deterministic, no model). The full
|
||||
* `runEval` drives the real `claude` CLI and is exercised by the `bench/`
|
||||
* harness rather than the unit suite.
|
||||
* Tests for the eval aggregation/formatting + orchestration (deterministic, no
|
||||
* model). `runEval` itself spawns the real `claude` CLI (bench/ exercises that);
|
||||
* `runEvalWith` takes an injected runner, so the loop / `measure` context /
|
||||
* aggregation are tested here against canned stream-json — no model.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { aggregate, aggregateStats, formatEvalReport } from "./eval.js";
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
aggregate,
|
||||
aggregateStats,
|
||||
aggregateUsage,
|
||||
parseUsage,
|
||||
formatEvalReport,
|
||||
runEvalWith,
|
||||
runPool,
|
||||
isRateLimited,
|
||||
measureTriggerRateWith,
|
||||
formatTriggerRateReport,
|
||||
type AgentRunArgs,
|
||||
} from "./eval.js";
|
||||
import {
|
||||
usedTool,
|
||||
outputContains,
|
||||
assertTriggerRate,
|
||||
} from "./harness-assert.js";
|
||||
import { makeTmpDir, cleanupTmpDir } from "./test-utils.js";
|
||||
|
||||
test("aggregateStats reports mean, sample std, se, and n", () => {
|
||||
const s = aggregateStats([{ x: 2 }, { x: 4 }, { x: 6 }]);
|
||||
@@ -23,6 +45,18 @@ test("aggregateStats gives std 0 for a single observation", () => {
|
||||
assert.equal(s.x.se, 0);
|
||||
});
|
||||
|
||||
test("aggregateStats reports pass^k: 1 only when every trial succeeds", () => {
|
||||
// booleans: all true → passK 1; any false → 0
|
||||
const all = aggregateStats([{ ok: true }, { ok: true }]);
|
||||
assert.equal(all.ok.passK, 1);
|
||||
const some = aggregateStats([{ ok: true }, { ok: false }]);
|
||||
assert.equal(some.ok.passK, 0);
|
||||
// counts: a trial succeeds when > 0
|
||||
const counts = aggregateStats([{ marks: 2 }, { marks: 0 }]);
|
||||
assert.equal(counts.marks.passK, 0);
|
||||
assert.equal(aggregateStats([{ marks: 1 }, { marks: 3 }]).marks.passK, 1);
|
||||
});
|
||||
|
||||
test("aggregate averages numbers and takes the true-fraction of booleans", () => {
|
||||
const agg = aggregate([
|
||||
{ marks: 2, caught: true },
|
||||
@@ -39,13 +73,408 @@ test("aggregate tolerates missing keys across rows", () => {
|
||||
assert.equal(agg.b, 1);
|
||||
});
|
||||
|
||||
test("runEvalWith drives arms × trials via an injected runner (no model)", async () => {
|
||||
// Canned stream-json: the `on` arm reports a Skill tool_use, a hook firing,
|
||||
// and a result with num_turns + answer; the `off` arm reports a bare result
|
||||
// (no tool / hook / num_turns / answer) — exercising both makeContext branches.
|
||||
const onStream = [
|
||||
JSON.stringify({
|
||||
type: "assistant",
|
||||
message: {
|
||||
content: [{ type: "tool_use", id: "t1", name: "Skill", input: {} }],
|
||||
},
|
||||
}),
|
||||
JSON.stringify({
|
||||
type: "system",
|
||||
subtype: "hook_response",
|
||||
hook_name: "Stop",
|
||||
hook_event: "Stop",
|
||||
exit_code: 0,
|
||||
outcome: "success",
|
||||
output: "",
|
||||
}),
|
||||
JSON.stringify({ type: "result", result: "answer is on", num_turns: 2 }),
|
||||
].join("\n");
|
||||
const offStream = JSON.stringify({ type: "result" }); // no num_turns/result
|
||||
|
||||
const seen: AgentRunArgs[] = [];
|
||||
const fakeRunner = (
|
||||
a: AgentRunArgs,
|
||||
): Promise<{ code: number; stdout: string }> => {
|
||||
seen.push(a);
|
||||
return Promise.resolve({
|
||||
code: 0,
|
||||
stdout: a.hasSettings ? onStream : offStream,
|
||||
});
|
||||
};
|
||||
|
||||
const report = await runEvalWith(
|
||||
{
|
||||
fixture: { "a.txt": "hi" },
|
||||
arms: {
|
||||
off: {},
|
||||
on: {
|
||||
settings: {
|
||||
hooks: {
|
||||
Stop: [{ hooks: [{ type: "command", command: "true" }] }],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
task: "do it",
|
||||
trials: 2,
|
||||
spacingSec: 0,
|
||||
measure: (ctx) => ({
|
||||
used: usedTool(ctx, "Skill"),
|
||||
turns: ctx.turns,
|
||||
sawFile: ctx.file("a.txt") !== null,
|
||||
missing: ctx.file("nope.txt") === null,
|
||||
shOk: ctx.sh("echo hi") === "hi",
|
||||
// failing command WITH stdout → catch returns the partial stdout
|
||||
shPartial: ctx.sh("echo part; exit 1") === "part",
|
||||
// failing command WITHOUT stdout → catch returns ""
|
||||
shEmpty: ctx.sh("exit 7") === "",
|
||||
onArm: outputContains(ctx, "answer is on"),
|
||||
}),
|
||||
},
|
||||
fakeRunner,
|
||||
);
|
||||
|
||||
assert.equal(seen.length, 4); // 2 arms × 2 trials
|
||||
assert.equal(report.arms.off?.runs, 2);
|
||||
assert.equal(report.arms.on?.runs, 2);
|
||||
// `on` arm: Skill used, 2 turns, answer present → all true (pass^k = 1)
|
||||
assert.equal(report.arms.on?.metrics.used, 1);
|
||||
assert.equal(report.arms.on?.metrics.turns, 2);
|
||||
assert.equal(report.arms.on?.stats.used?.passK, 1);
|
||||
// `off` arm: no Skill, 0 turns, no answer
|
||||
assert.equal(report.arms.off?.metrics.used, 0);
|
||||
assert.equal(report.arms.off?.metrics.turns, 0);
|
||||
// both arms: the fixture file is present, sh try/catch all hold
|
||||
assert.equal(report.arms.off?.metrics.sawFile, 1);
|
||||
assert.equal(report.arms.off?.metrics.shOk, 1);
|
||||
assert.equal(report.arms.off?.metrics.shPartial, 1);
|
||||
assert.equal(report.arms.off?.metrics.shEmpty, 1);
|
||||
});
|
||||
|
||||
test("runEvalWith honors provided optionals (name/model/tools/timeout, arm.files)", async () => {
|
||||
const runner = (): Promise<{ code: number; stdout: string }> =>
|
||||
Promise.resolve({
|
||||
code: 0,
|
||||
stdout: JSON.stringify({ type: "result", result: "r", num_turns: 1 }),
|
||||
});
|
||||
const report = await runEvalWith(
|
||||
{
|
||||
name: "custom",
|
||||
fixture: { "base.txt": "b" },
|
||||
arms: { a: { files: { "extra.txt": "e" } } }, // arm.files spread branch
|
||||
task: "t",
|
||||
trials: 1,
|
||||
model: "sonnet",
|
||||
allowedTools: ["Read"],
|
||||
timeoutMs: 1000,
|
||||
spacingSec: 0,
|
||||
measure: (ctx) => ({
|
||||
both: ctx.file("base.txt") !== null && ctx.file("extra.txt") !== null,
|
||||
}),
|
||||
},
|
||||
runner,
|
||||
);
|
||||
assert.equal(report.name, "custom"); // spec.name provided
|
||||
assert.equal(report.arms.a?.metrics.both, 1);
|
||||
});
|
||||
|
||||
test("measureTriggerRateWith aggregates per-prompt and overall trigger rate", async () => {
|
||||
const skillStream =
|
||||
JSON.stringify({
|
||||
type: "assistant",
|
||||
message: {
|
||||
content: [{ type: "tool_use", id: "t1", name: "Skill", input: {} }],
|
||||
},
|
||||
}) +
|
||||
"\n" +
|
||||
JSON.stringify({ type: "result", num_turns: 1 });
|
||||
const plainStream = JSON.stringify({ type: "result", num_turns: 1 });
|
||||
|
||||
const seen: AgentRunArgs[] = [];
|
||||
const runner = (
|
||||
a: AgentRunArgs,
|
||||
): Promise<{ code: number; stdout: string }> => {
|
||||
seen.push(a);
|
||||
// prompts containing "fire" trigger the Skill; the rest don't
|
||||
return Promise.resolve({
|
||||
code: 0,
|
||||
stdout: a.task.includes("fire") ? skillStream : plainStream,
|
||||
});
|
||||
};
|
||||
|
||||
const report = await measureTriggerRateWith(
|
||||
{
|
||||
pluginDir: "/some/plugin",
|
||||
prompts: ["fire one", "ignore this", "fire two"],
|
||||
fired: (t) => usedTool(t, "Skill"),
|
||||
trials: 2,
|
||||
spacingSec: 0,
|
||||
},
|
||||
runner,
|
||||
);
|
||||
|
||||
assert.equal(report.n, 6); // 3 prompts × 2 trials
|
||||
assert.equal(seen.length, 6);
|
||||
assert.equal(seen[0]?.pluginDir, "/some/plugin"); // pluginDir forwarded
|
||||
assert.ok(Math.abs(report.rate - 4 / 6) < 1e-9); // 2 firing prompts × 2 trials
|
||||
const fireOne = report.perPrompt.find((p) => p.prompt === "fire one");
|
||||
assert.equal(fireOne?.fired, 2);
|
||||
assert.equal(fireOne?.rate, 1);
|
||||
const ignore = report.perPrompt.find((p) => p.prompt === "ignore this");
|
||||
assert.equal(ignore?.fired, 0);
|
||||
assert.equal(ignore?.rate, 0);
|
||||
assert.ok(formatTriggerRateReport(report).includes("trigger-rate: 67%"));
|
||||
});
|
||||
|
||||
test("parseUsage pulls cost/latency/tokens from the result event", () => {
|
||||
const stdout = JSON.stringify({
|
||||
type: "result",
|
||||
total_cost_usd: 0.02,
|
||||
duration_ms: 900,
|
||||
usage: { input_tokens: 120, output_tokens: 30 },
|
||||
});
|
||||
const u = parseUsage(stdout);
|
||||
assert.equal(u.costUsd, 0.02);
|
||||
assert.equal(u.durationMs, 900);
|
||||
assert.equal(u.inputTokens, 120);
|
||||
assert.equal(u.outputTokens, 30);
|
||||
});
|
||||
|
||||
test("parseUsage is all-zero when no result/usage is present", () => {
|
||||
const u = parseUsage("");
|
||||
assert.equal(u.costUsd, 0);
|
||||
assert.equal(u.durationMs, 0);
|
||||
assert.equal(u.inputTokens, 0);
|
||||
assert.equal(u.outputTokens, 0);
|
||||
});
|
||||
|
||||
test("aggregateUsage totals and averages cost/latency/tokens", () => {
|
||||
const u = aggregateUsage([
|
||||
{ costUsd: 0.01, durationMs: 1000, inputTokens: 100, outputTokens: 50 },
|
||||
{ costUsd: 0.03, durationMs: 2000, inputTokens: 200, outputTokens: 150 },
|
||||
]);
|
||||
assert.ok(Math.abs(u.totalCostUsd - 0.04) < 1e-9);
|
||||
assert.ok(Math.abs(u.meanCostUsd - 0.02) < 1e-9);
|
||||
assert.equal(u.meanDurationMs, 1500);
|
||||
assert.equal(u.totalInputTokens, 300);
|
||||
assert.equal(u.totalOutputTokens, 200);
|
||||
});
|
||||
|
||||
test("aggregateUsage is all-zero for no runs", () => {
|
||||
const u = aggregateUsage([]);
|
||||
assert.equal(u.totalCostUsd, 0);
|
||||
assert.equal(u.meanCostUsd, 0);
|
||||
assert.equal(u.meanDurationMs, 0);
|
||||
});
|
||||
|
||||
test("runEvalWith record/replay cache: replays without re-calling the model", async () => {
|
||||
const cacheDir = makeTmpDir("eval-cache");
|
||||
const resultStream = JSON.stringify({
|
||||
type: "result",
|
||||
num_turns: 1,
|
||||
result: "done",
|
||||
total_cost_usd: 0.01,
|
||||
duration_ms: 1200,
|
||||
usage: { input_tokens: 100, output_tokens: 50 },
|
||||
});
|
||||
let calls = 0;
|
||||
const recordingRunner = (
|
||||
a: AgentRunArgs,
|
||||
): Promise<{ code: number; stdout: string }> => {
|
||||
calls++;
|
||||
writeFileSync(join(a.cwd, "OUT.txt"), "agent output"); // a side-effect to snapshot
|
||||
return Promise.resolve({ code: 0, stdout: resultStream });
|
||||
};
|
||||
const spec = {
|
||||
fixture: { "in.txt": "x" },
|
||||
arms: { only: {} },
|
||||
task: "do it",
|
||||
trials: 2,
|
||||
spacingSec: 0,
|
||||
cacheDir,
|
||||
measure: (ctx: { file: (p: string) => string | null }) => ({
|
||||
created: ctx.file("OUT.txt") !== null,
|
||||
}),
|
||||
};
|
||||
try {
|
||||
const r1 = await runEvalWith(
|
||||
{ ...spec, cache: "readwrite" as const },
|
||||
recordingRunner,
|
||||
);
|
||||
assert.equal(calls, 2); // model called for both trials
|
||||
assert.equal(r1.arms.only?.metrics.created, 1);
|
||||
assert.ok(Math.abs(r1.totalCostUsd - 0.02) < 1e-9); // 2 × $0.01
|
||||
assert.equal(r1.arms.only?.usage.totalInputTokens, 200);
|
||||
|
||||
// second run, read-only, with a runner that throws if called → must replay
|
||||
const boom = (): Promise<{ code: number; stdout: string }> => {
|
||||
throw new Error("runner should not be called on a cache hit");
|
||||
};
|
||||
const r2 = await runEvalWith({ ...spec, cache: "read" as const }, boom);
|
||||
assert.equal(r2.arms.only?.metrics.created, 1); // OUT.txt restored → measure sees it
|
||||
assert.ok(Math.abs(r2.totalCostUsd - 0.02) < 1e-9); // replayed usage
|
||||
} finally {
|
||||
cleanupTmpDir(cacheDir);
|
||||
}
|
||||
});
|
||||
|
||||
test("runPool maps with bounded concurrency, preserving order", async () => {
|
||||
let inFlight = 0;
|
||||
let maxInFlight = 0;
|
||||
const worker = async (n: number): Promise<number> => {
|
||||
inFlight++;
|
||||
maxInFlight = Math.max(maxInFlight, inFlight);
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
inFlight--;
|
||||
return n * 2;
|
||||
};
|
||||
const out = await runPool([1, 2, 3, 4, 5], 2, worker);
|
||||
assert.deepEqual(out, [2, 4, 6, 8, 10]); // order preserved
|
||||
assert.equal(maxInFlight, 2); // never exceeded, did reach the limit
|
||||
});
|
||||
|
||||
test("isRateLimited detects rate-limit / overload in either stream", () => {
|
||||
assert.ok(
|
||||
isRateLimited({ code: 1, stdout: "", stderr: "Error: 429 happened" }),
|
||||
);
|
||||
assert.ok(isRateLimited({ code: 1, stdout: "overloaded_error" }));
|
||||
assert.ok(isRateLimited({ code: 1, stdout: "rate limit exceeded" }));
|
||||
assert.ok(!isRateLimited({ code: 0, stdout: "all good" }));
|
||||
});
|
||||
|
||||
test("runEvalWith retries a rate-limited run, then succeeds", async () => {
|
||||
let calls = 0;
|
||||
const runner = (): Promise<{ code: number; stdout: string }> => {
|
||||
calls++;
|
||||
const stdout =
|
||||
calls === 1
|
||||
? "rate limit exceeded"
|
||||
: JSON.stringify({ type: "result", num_turns: 1, result: "ok" });
|
||||
return Promise.resolve({ code: 0, stdout });
|
||||
};
|
||||
const report = await runEvalWith(
|
||||
{
|
||||
arms: { only: {} },
|
||||
task: "t",
|
||||
trials: 1,
|
||||
spacingSec: 0,
|
||||
retryBackoffMs: 0,
|
||||
measure: (ctx) => ({ turns: ctx.turns }),
|
||||
},
|
||||
runner,
|
||||
);
|
||||
assert.equal(calls, 2); // retried once
|
||||
assert.equal(report.arms.only?.metrics.turns, 1);
|
||||
});
|
||||
|
||||
test("runEvalWith gives up after rateLimitRetries=0 (no retry)", async () => {
|
||||
let calls = 0;
|
||||
const runner = (): Promise<{ code: number; stdout: string }> => {
|
||||
calls++;
|
||||
return Promise.resolve({ code: 0, stdout: "rate limit" });
|
||||
};
|
||||
await runEvalWith(
|
||||
{
|
||||
arms: { only: {} },
|
||||
task: "t",
|
||||
trials: 1,
|
||||
spacingSec: 0,
|
||||
rateLimitRetries: 0,
|
||||
retryBackoffMs: 0,
|
||||
measure: () => ({ ok: true }),
|
||||
},
|
||||
runner,
|
||||
);
|
||||
assert.equal(calls, 1); // gave up immediately
|
||||
});
|
||||
|
||||
test("runEvalWith honors concurrency and inter-run spacing", async () => {
|
||||
let inFlight = 0;
|
||||
let maxInFlight = 0;
|
||||
const stream = JSON.stringify({ type: "result", num_turns: 1, result: "ok" });
|
||||
const runner = async (): Promise<{ code: number; stdout: string }> => {
|
||||
inFlight++;
|
||||
maxInFlight = Math.max(maxInFlight, inFlight);
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
inFlight--;
|
||||
return { code: 0, stdout: stream };
|
||||
};
|
||||
await runEvalWith(
|
||||
{
|
||||
arms: { a: {}, b: {} },
|
||||
task: "t",
|
||||
trials: 3,
|
||||
spacingSec: 0.001, // > 0 → exercises the spacing path
|
||||
concurrency: 3,
|
||||
measure: () => ({ ok: true }),
|
||||
},
|
||||
runner,
|
||||
);
|
||||
assert.equal(maxInFlight, 3); // 6 units, 3 in flight at once
|
||||
});
|
||||
|
||||
test("runEvalWith aborts when maxCostUsd is exceeded", async () => {
|
||||
const stream = JSON.stringify({
|
||||
type: "result",
|
||||
num_turns: 1,
|
||||
result: "ok",
|
||||
total_cost_usd: 0.1,
|
||||
});
|
||||
let calls = 0;
|
||||
const runner = (): Promise<{ code: number; stdout: string }> => {
|
||||
calls++;
|
||||
return Promise.resolve({ code: 0, stdout: stream });
|
||||
};
|
||||
const report = await runEvalWith(
|
||||
{
|
||||
arms: { only: {} },
|
||||
task: "t",
|
||||
trials: 5,
|
||||
spacingSec: 0,
|
||||
maxCostUsd: 0.15, // exceeded after 2 trials ($0.20)
|
||||
measure: () => ({ ok: true }),
|
||||
},
|
||||
runner,
|
||||
);
|
||||
assert.equal(report.aborted, true);
|
||||
assert.equal(calls, 2); // stopped early
|
||||
assert.equal(report.arms.only?.runs, 2); // only completed trials counted
|
||||
assert.ok(Math.abs(report.totalCostUsd - 0.2) < 1e-9);
|
||||
});
|
||||
|
||||
test("assertTriggerRate gates on the minimum rate", () => {
|
||||
const report = { rate: 0.5, n: 4, perPrompt: [] };
|
||||
assert.doesNotThrow(() => {
|
||||
assertTriggerRate(report, { min: 0.5 });
|
||||
});
|
||||
assert.throws(() => {
|
||||
assertTriggerRate(report, { min: 0.8 });
|
||||
});
|
||||
});
|
||||
|
||||
const NO_USAGE = {
|
||||
totalCostUsd: 0,
|
||||
meanCostUsd: 0,
|
||||
meanDurationMs: 0,
|
||||
totalInputTokens: 0,
|
||||
totalOutputTokens: 0,
|
||||
} as const;
|
||||
|
||||
test("formatEvalReport renders one line per arm", () => {
|
||||
const out = formatEvalReport({
|
||||
name: "demo",
|
||||
trials: 6,
|
||||
totalCostUsd: 0,
|
||||
aborted: false,
|
||||
arms: {
|
||||
vanilla: { runs: 6, metrics: { caught: 0 }, stats: {} },
|
||||
gated: { runs: 6, metrics: { caught: 0.5 }, stats: {} },
|
||||
vanilla: { runs: 6, metrics: { caught: 0 }, stats: {}, usage: NO_USAGE },
|
||||
gated: { runs: 6, metrics: { caught: 0.5 }, stats: {}, usage: NO_USAGE },
|
||||
},
|
||||
});
|
||||
assert.match(out, /demo \(6 trials\/arm\)/);
|
||||
@@ -53,17 +482,46 @@ test("formatEvalReport renders one line per arm", () => {
|
||||
assert.match(out, /gated\s+caught=0\.50/);
|
||||
});
|
||||
|
||||
test("formatEvalReport shows ± se when stats are present", () => {
|
||||
test("formatEvalReport shows ± se and pass^k when stats are present", () => {
|
||||
const out = formatEvalReport({
|
||||
name: "demo",
|
||||
trials: 3,
|
||||
totalCostUsd: 0,
|
||||
aborted: false,
|
||||
arms: {
|
||||
gated: {
|
||||
runs: 3,
|
||||
metrics: { caught: 0.5 },
|
||||
stats: { caught: { mean: 0.5, std: 0.5, se: 0.25, n: 3 } },
|
||||
stats: { caught: { mean: 0.5, std: 0.5, se: 0.25, n: 3, passK: 0 } },
|
||||
usage: NO_USAGE,
|
||||
},
|
||||
},
|
||||
});
|
||||
assert.match(out, /caught=0\.50±0\.25/);
|
||||
assert.match(out, /pass\^k=0/);
|
||||
});
|
||||
|
||||
test("formatEvalReport surfaces cost/latency/tokens when usage is present", () => {
|
||||
const out = formatEvalReport({
|
||||
name: "demo",
|
||||
trials: 2,
|
||||
totalCostUsd: 0.05,
|
||||
aborted: false,
|
||||
arms: {
|
||||
gated: {
|
||||
runs: 2,
|
||||
metrics: { caught: 1 },
|
||||
stats: {},
|
||||
usage: {
|
||||
totalCostUsd: 0.05,
|
||||
meanCostUsd: 0.025,
|
||||
meanDurationMs: 1500,
|
||||
totalInputTokens: 2000,
|
||||
totalOutputTokens: 1400,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
assert.match(out, /\$0\.0500 total/);
|
||||
assert.match(out, /\$0\.0500 · 1\.5s\/run · 3\.4k tok/);
|
||||
});
|
||||
|
||||
+559
-77
@@ -34,6 +34,21 @@ import { tmpdir } from "node:os";
|
||||
import { resolve, join, dirname } from "node:path";
|
||||
|
||||
import { resolveHarness } from "./plugin-loader.js";
|
||||
import {
|
||||
parseToolCalls,
|
||||
parseResultEvent,
|
||||
parseHooks,
|
||||
type ToolCall,
|
||||
type Trace,
|
||||
} from "./harness-test.js";
|
||||
import {
|
||||
cacheKey,
|
||||
readCache,
|
||||
writeCache,
|
||||
snapshotDir,
|
||||
restoreDir,
|
||||
type CacheMode,
|
||||
} from "./eval-cache.js";
|
||||
|
||||
/** One arm of the comparison: fixture overrides + settings (hooks) for this arm. */
|
||||
export interface EvalArm {
|
||||
@@ -57,15 +72,32 @@ export interface EvalArm {
|
||||
readonly pluginDir?: string;
|
||||
}
|
||||
|
||||
/** Context handed to `measure` after a run, to compute that run's metrics. */
|
||||
export interface RunContext {
|
||||
/** Per-run resource use, parsed from the terminal `result` event (0 when absent). */
|
||||
export interface EvalUsage {
|
||||
/** `total_cost_usd` reported by claude. */
|
||||
readonly costUsd: number;
|
||||
/** Wall-clock `duration_ms` of the run. */
|
||||
readonly durationMs: number;
|
||||
readonly inputTokens: number;
|
||||
readonly outputTokens: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Context handed to `measure` after a run, to compute that run's metrics. It is
|
||||
* a `Trace` (so the bare predicates `usedTool` / `skillResolved` / `toolCount` /
|
||||
* `toolUsedWith` from `harness-assert.ts` run over it, the same as over a
|
||||
* `runHarnessTest` result) plus the eval-only `sh` end-state probe and `usage`.
|
||||
*/
|
||||
export interface RunContext extends Trace {
|
||||
readonly cwd: string;
|
||||
readonly exitCode: number;
|
||||
readonly stdout: string;
|
||||
/** `num_turns` reported by claude, or 0. */
|
||||
readonly turns: number;
|
||||
/** Contents of a file under the working dir, or null if absent. */
|
||||
file(path: string): string | null;
|
||||
/** The tools the agent invoked, each paired with its result (parsed from the stream). */
|
||||
readonly toolCalls: readonly ToolCall[];
|
||||
/** Cost / latency / tokens for this run (use as metrics, e.g. `{ cost: ctx.usage.costUsd }`). */
|
||||
readonly usage: EvalUsage;
|
||||
/** Run a shell command in the working dir; returns trimmed stdout ("" on error). */
|
||||
sh(command: string): string;
|
||||
}
|
||||
@@ -92,6 +124,32 @@ export interface EvalSpec<M extends Metrics> {
|
||||
readonly timeoutMs?: number;
|
||||
/** Seconds to wait between runs (avoid rate-limit bursts). Default 4. */
|
||||
readonly spacingSec?: number;
|
||||
/**
|
||||
* Record/replay cache mode. Default `"off"` (always re-sample). `"readwrite"`
|
||||
* records each trial (output + post-run files) and replays it on a matching
|
||||
* re-run — so editing `measure` re-scores for free; the model is re-called only
|
||||
* when a model-affecting input changes. `"read"` replays but never records.
|
||||
* The cache key excludes `measure`, so changing your metric still hits.
|
||||
*/
|
||||
readonly cache?: CacheMode;
|
||||
/** Where cache records live. Default `.vigiles/eval-cache` under cwd. */
|
||||
readonly cacheDir?: string;
|
||||
/**
|
||||
* How many trials to run at once (across all arms × trials). Default 1 (fully
|
||||
* sequential — the safe, no-surprise default). Raise it to cut wall-clock time;
|
||||
* rate-limit bursts are absorbed by the retry/backoff below.
|
||||
*/
|
||||
readonly concurrency?: number;
|
||||
/**
|
||||
* Abort the run once measured cost reaches this many USD. In-flight trials
|
||||
* finish; remaining ones are skipped and `report.aborted` is set. Needs the
|
||||
* model to report `total_cost_usd` (the eval tier does).
|
||||
*/
|
||||
readonly maxCostUsd?: number;
|
||||
/** Retries on a detected rate-limit/overload before giving up. Default 3. */
|
||||
readonly rateLimitRetries?: number;
|
||||
/** Base backoff ms (doubled each retry). Default 1000. */
|
||||
readonly retryBackoffMs?: number;
|
||||
}
|
||||
|
||||
/** Per-metric summary statistics across an arm's runs. */
|
||||
@@ -104,6 +162,22 @@ export interface MetricStat {
|
||||
readonly se: number;
|
||||
/** Number of runs the metric was observed in. */
|
||||
readonly n: number;
|
||||
/**
|
||||
* pass^k (τ-bench): 1 if the metric succeeded on EVERY trial, else 0. The
|
||||
* reliability question a non-deterministic harness needs — "worked every time"
|
||||
* is not "worked on average". A trial counts as a success when its value is
|
||||
* truthy (booleans true, counts > 0), so model your metric as success/fail.
|
||||
*/
|
||||
readonly passK: number;
|
||||
}
|
||||
|
||||
/** Aggregated cost / latency / tokens across an arm's runs. */
|
||||
export interface ArmUsage {
|
||||
readonly totalCostUsd: number;
|
||||
readonly meanCostUsd: number;
|
||||
readonly meanDurationMs: number;
|
||||
readonly totalInputTokens: number;
|
||||
readonly totalOutputTokens: number;
|
||||
}
|
||||
|
||||
export interface ArmReport {
|
||||
@@ -112,12 +186,18 @@ export interface ArmReport {
|
||||
readonly metrics: Record<string, number>;
|
||||
/** Per-metric mean / std / se / n, so an A/B gap can be read for significance. */
|
||||
readonly stats: Record<string, MetricStat>;
|
||||
/** Cost / latency / token totals + means for this arm. */
|
||||
readonly usage: ArmUsage;
|
||||
}
|
||||
|
||||
export interface EvalReport {
|
||||
readonly name: string;
|
||||
readonly trials: number;
|
||||
readonly arms: Record<string, ArmReport>;
|
||||
/** Total measured cost across every arm × trial (0 when usage wasn't reported). */
|
||||
readonly totalCostUsd: number;
|
||||
/** True if a `maxCostUsd` budget cap stopped the run before all trials ran. */
|
||||
readonly aborted: boolean;
|
||||
}
|
||||
|
||||
function writeFiles(cwd: string, files: Record<string, string>): void {
|
||||
@@ -128,65 +208,122 @@ function writeFiles(cwd: string, files: Record<string, string>): void {
|
||||
}
|
||||
}
|
||||
|
||||
interface RunOut {
|
||||
/** The raw output of one trial: the agent's exit code + captured streams. */
|
||||
export interface RunOut {
|
||||
code: number;
|
||||
stdout: string;
|
||||
/** Captured stderr, when the runner provides it (used for rate-limit detection). */
|
||||
stderr?: string;
|
||||
}
|
||||
|
||||
function spawnAgent(
|
||||
task: string,
|
||||
cwd: string,
|
||||
model: string,
|
||||
tools: readonly string[],
|
||||
hasSettings: boolean,
|
||||
pluginDir: string | undefined,
|
||||
timeoutMs: number,
|
||||
): Promise<RunOut> {
|
||||
/** The per-trial arguments handed to an {@link AgentRunner}. */
|
||||
export interface AgentRunArgs {
|
||||
readonly task: string;
|
||||
readonly cwd: string;
|
||||
readonly model: string;
|
||||
readonly tools: readonly string[];
|
||||
readonly hasSettings: boolean;
|
||||
readonly pluginDir: string | undefined;
|
||||
readonly timeoutMs: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one trial and returns its raw output. The default ({@link spawnAgent})
|
||||
* drives the real `claude` CLI; `runEvalWith` takes one explicitly, so the eval
|
||||
* orchestration is testable without a model (pass a fake returning canned
|
||||
* stream-json) and a custom runtime can be plugged in.
|
||||
*/
|
||||
export type AgentRunner = (args: AgentRunArgs) => Promise<RunOut>;
|
||||
|
||||
/* v8 ignore start -- real claude subprocess; exercised by bench/, not the unit gate */
|
||||
function spawnAgent(a: AgentRunArgs): Promise<RunOut> {
|
||||
return new Promise((resolvePromise) => {
|
||||
const args = [
|
||||
"-p",
|
||||
task,
|
||||
a.task,
|
||||
// stream-json (+ --verbose, required with -p) so the per-turn tool_use
|
||||
// events survive into `ctx.toolCalls` — the unified Trace, same as the
|
||||
// harness tier. The terminal `result` event still carries num_turns/output.
|
||||
"--output-format",
|
||||
"json",
|
||||
"stream-json",
|
||||
"--verbose",
|
||||
"--model",
|
||||
model,
|
||||
a.model,
|
||||
"--permission-mode",
|
||||
"acceptEdits",
|
||||
...(pluginDir !== undefined ? ["--plugin-dir", resolve(pluginDir)] : []),
|
||||
...(hasSettings ? ["--settings", "settings.json"] : []),
|
||||
...(a.pluginDir !== undefined
|
||||
? ["--plugin-dir", resolve(a.pluginDir)]
|
||||
: []),
|
||||
...(a.hasSettings ? ["--settings", "settings.json"] : []),
|
||||
"--allowedTools",
|
||||
...tools,
|
||||
...a.tools,
|
||||
];
|
||||
const child = spawn("claude", args, {
|
||||
cwd,
|
||||
cwd: a.cwd,
|
||||
env: process.env,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.on("data", (d: Buffer) => (stdout += d.toString()));
|
||||
const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs);
|
||||
child.stderr.on("data", (d: Buffer) => (stderr += d.toString()));
|
||||
const timer = setTimeout(() => child.kill("SIGKILL"), a.timeoutMs);
|
||||
child.on("close", (code) => {
|
||||
clearTimeout(timer);
|
||||
resolvePromise({ code: code ?? 0, stdout });
|
||||
resolvePromise({ code: code ?? 0, stdout, stderr });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the eval: every arm × every trial against the real `claude` CLI, with the
|
||||
* metric computed per run and aggregated per arm. Requires `claude` on PATH and
|
||||
* working model auth (e.g. `ANTHROPIC_API_KEY`). Thin wrapper over
|
||||
* {@link runEvalWith} with the real agent runner.
|
||||
*/
|
||||
export async function runEval<M extends Metrics>(
|
||||
spec: EvalSpec<M>,
|
||||
): Promise<EvalReport> {
|
||||
return runEvalWith(spec, spawnAgent);
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
const sleep = (ms: number): Promise<void> =>
|
||||
new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
/** Pull cost / latency / tokens out of a parsed `result` event (0 when absent). */
|
||||
function usageFrom(result: Record<string, unknown> | null): EvalUsage {
|
||||
const num = (v: unknown): number => (typeof v === "number" ? v : 0);
|
||||
const usage = (result?.usage ?? {}) as Record<string, unknown>;
|
||||
return {
|
||||
costUsd: num(result?.total_cost_usd),
|
||||
durationMs: num(result?.duration_ms),
|
||||
inputTokens: num(usage.input_tokens),
|
||||
outputTokens: num(usage.output_tokens),
|
||||
};
|
||||
}
|
||||
|
||||
/** Parse per-run cost/latency/tokens from a stream — pure, model-free. */
|
||||
export function parseUsage(stdout: string): EvalUsage {
|
||||
return usageFrom(parseResultEvent(stdout));
|
||||
}
|
||||
|
||||
function makeContext(cwd: string, out: RunOut): RunContext {
|
||||
let turns = 0;
|
||||
try {
|
||||
turns = (JSON.parse(out.stdout) as { num_turns?: number }).num_turns ?? 0;
|
||||
} catch {
|
||||
/* non-JSON output */
|
||||
}
|
||||
const result = parseResultEvent(out.stdout);
|
||||
const turns = typeof result?.num_turns === "number" ? result.num_turns : 0;
|
||||
const output = typeof result?.result === "string" ? result.result : "";
|
||||
return {
|
||||
cwd,
|
||||
exitCode: out.code,
|
||||
stdout: out.stdout,
|
||||
turns,
|
||||
toolCalls: parseToolCalls(out.stdout),
|
||||
hooks: parseHooks(out.stdout),
|
||||
output,
|
||||
usage: usageFrom(result),
|
||||
// The eval tier drives the real API (no mock between claude and the model),
|
||||
// so the requests can't be captured here — modelRequests is harness-tier only.
|
||||
modelRequests: [],
|
||||
file: (p) => {
|
||||
const f = resolve(cwd, p);
|
||||
return existsSync(f) ? readFileSync(f, "utf-8") : null;
|
||||
@@ -246,81 +383,426 @@ export function aggregateStats(
|
||||
n > 1
|
||||
? Math.sqrt(values.reduce((a, b) => a + (b - mean) ** 2, 0) / (n - 1))
|
||||
: 0;
|
||||
out[k] = { mean, std, se: n > 0 ? std / Math.sqrt(n) : 0, n };
|
||||
const passK = n > 0 && values.every((v) => v > 0) ? 1 : 0;
|
||||
out[k] = { mean, std, se: n > 0 ? std / Math.sqrt(n) : 0, n, passK };
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Aggregate per-run usage into an arm's cost / latency / token totals + means. */
|
||||
export function aggregateUsage(usages: readonly EvalUsage[]): ArmUsage {
|
||||
const n = usages.length;
|
||||
const sum = (f: (u: EvalUsage) => number): number =>
|
||||
usages.reduce((a, u) => a + f(u), 0);
|
||||
const totalCostUsd = sum((u) => u.costUsd);
|
||||
return {
|
||||
totalCostUsd,
|
||||
meanCostUsd: n > 0 ? totalCostUsd / n : 0,
|
||||
meanDurationMs: n > 0 ? sum((u) => u.durationMs) / n : 0,
|
||||
totalInputTokens: sum((u) => u.inputTokens),
|
||||
totalOutputTokens: sum((u) => u.outputTokens),
|
||||
};
|
||||
}
|
||||
|
||||
/** Resolved per-run settings shared across an eval's trials. */
|
||||
interface RunConfig {
|
||||
readonly model: string;
|
||||
readonly tools: readonly string[];
|
||||
readonly timeoutMs: number;
|
||||
readonly cache: CacheMode;
|
||||
readonly cacheDir: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the eval: every arm × every trial against the real `claude` CLI, with the
|
||||
* metric computed per run and aggregated per arm. Requires `claude` on PATH and
|
||||
* working model auth (e.g. `ANTHROPIC_API_KEY`).
|
||||
* Run one trial through the cache: on a hit, restore the recorded post-run
|
||||
* filesystem into `cwd` and return the recorded output (no model call); on a
|
||||
* miss, run the agent and (in `readwrite`) record output + cwd snapshot. The
|
||||
* cache key excludes `measure`, so editing the metric still replays.
|
||||
*/
|
||||
export async function runEval<M extends Metrics>(
|
||||
async function runWithCache(
|
||||
runArgs: AgentRunArgs,
|
||||
keyParts: {
|
||||
files: Record<string, string>;
|
||||
settings: unknown;
|
||||
trialIndex: number;
|
||||
},
|
||||
runner: AgentRunner,
|
||||
cfg: RunConfig,
|
||||
): Promise<RunOut> {
|
||||
if (cfg.cache === "off") return runner(runArgs);
|
||||
const key = cacheKey({
|
||||
task: runArgs.task,
|
||||
model: runArgs.model,
|
||||
tools: runArgs.tools,
|
||||
files: keyParts.files,
|
||||
settings: keyParts.settings,
|
||||
trialIndex: keyParts.trialIndex,
|
||||
});
|
||||
const hit = readCache(cfg.cacheDir, key);
|
||||
if (hit) {
|
||||
restoreDir(runArgs.cwd, hit.files);
|
||||
return hit.out;
|
||||
}
|
||||
const out = await runner(runArgs);
|
||||
if (cfg.cache === "readwrite") {
|
||||
writeCache(cfg.cacheDir, key, { out, files: snapshotDir(runArgs.cwd) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Execute one trial in a fresh sandbox; returns its metric row + usage. */
|
||||
async function executeTrial<M extends Metrics>(
|
||||
spec: EvalSpec<M>,
|
||||
arm: EvalArm,
|
||||
trialIndex: number,
|
||||
runner: AgentRunner,
|
||||
cfg: RunConfig,
|
||||
): Promise<{ row: M; usage: EvalUsage }> {
|
||||
const cwd = mkdtempSync(join(tmpdir(), "vigiles-eval-"));
|
||||
try {
|
||||
const { files, settings } = resolveHarness({
|
||||
plugin: arm.plugin,
|
||||
settings: arm.settings,
|
||||
files: { ...spec.fixture, ...arm.files },
|
||||
});
|
||||
writeFiles(cwd, files);
|
||||
const hasSettings = settings !== undefined;
|
||||
if (hasSettings) {
|
||||
writeFileSync(
|
||||
join(cwd, "settings.json"),
|
||||
JSON.stringify(settings, null, 2).replaceAll("{cwd}", cwd),
|
||||
);
|
||||
}
|
||||
const out = await runWithCache(
|
||||
{
|
||||
task: spec.task,
|
||||
cwd,
|
||||
model: cfg.model,
|
||||
tools: cfg.tools,
|
||||
hasSettings,
|
||||
pluginDir: arm.pluginDir,
|
||||
timeoutMs: cfg.timeoutMs,
|
||||
},
|
||||
{ files, settings, trialIndex },
|
||||
runner,
|
||||
cfg,
|
||||
);
|
||||
const ctx = makeContext(cwd, out);
|
||||
return { row: spec.measure(ctx), usage: ctx.usage };
|
||||
} finally {
|
||||
rmSync(cwd, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
// A signal in the captured streams that the model call was rate-limited /
|
||||
// overloaded — worth a backoff + retry rather than counting as a real sample.
|
||||
const RATE_LIMIT_RE = /rate.?limit|\b429\b|overloaded|too many requests/i;
|
||||
|
||||
/** Whether a run's captured output looks like a rate-limit / overload. Pure. */
|
||||
export function isRateLimited(out: RunOut): boolean {
|
||||
return RATE_LIMIT_RE.test(`${out.stderr ?? ""}\n${out.stdout}`);
|
||||
}
|
||||
|
||||
/** Call `runner`, retrying with exponential backoff while it looks rate-limited. */
|
||||
async function runWithRetry(
|
||||
runArgs: AgentRunArgs,
|
||||
runner: AgentRunner,
|
||||
retries: number,
|
||||
baseMs: number,
|
||||
): Promise<RunOut> {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
const out = await runner(runArgs);
|
||||
if (!isRateLimited(out) || attempt >= retries) return out;
|
||||
await sleep(baseMs * 2 ** attempt);
|
||||
}
|
||||
}
|
||||
|
||||
/** Map `worker` over `items` with at most `concurrency` in flight, order preserved. */
|
||||
export async function runPool<T, R>(
|
||||
items: readonly T[],
|
||||
concurrency: number,
|
||||
worker: (item: T) => Promise<R>,
|
||||
): Promise<R[]> {
|
||||
const results = new Array<R>(items.length);
|
||||
let next = 0;
|
||||
const drain = async (): Promise<void> => {
|
||||
for (;;) {
|
||||
const i = next++;
|
||||
const item = items[i];
|
||||
if (i >= items.length || item === undefined) return;
|
||||
results[i] = await worker(item);
|
||||
}
|
||||
};
|
||||
const workers = Math.max(1, Math.min(concurrency, items.length || 1));
|
||||
await Promise.all(Array.from({ length: workers }, drain));
|
||||
return results;
|
||||
}
|
||||
|
||||
/** One unit of work: a single trial of a single arm. */
|
||||
interface Unit {
|
||||
readonly armName: string;
|
||||
readonly arm: EvalArm;
|
||||
readonly trialIndex: number;
|
||||
}
|
||||
|
||||
/** Flatten arms × trials into a single work list (so concurrency spans both). */
|
||||
function buildUnits(arms: Record<string, EvalArm>, trials: number): Unit[] {
|
||||
const units: Unit[] = [];
|
||||
for (const [armName, arm] of Object.entries(arms)) {
|
||||
for (let t = 0; t < trials; t++)
|
||||
units.push({ armName, arm, trialIndex: t });
|
||||
}
|
||||
return units;
|
||||
}
|
||||
|
||||
type DoneResult<M extends Metrics> = {
|
||||
readonly armName: string;
|
||||
readonly skipped: false;
|
||||
readonly row: M;
|
||||
readonly usage: EvalUsage;
|
||||
};
|
||||
type UnitResult<M extends Metrics> =
|
||||
| DoneResult<M>
|
||||
| { readonly armName: string; readonly skipped: true };
|
||||
|
||||
/** Group completed (non-skipped) unit results by arm and aggregate each. */
|
||||
function aggregateArms<M extends Metrics>(
|
||||
armNames: readonly string[],
|
||||
results: readonly UnitResult<M>[],
|
||||
): { arms: Record<string, ArmReport>; totalCostUsd: number } {
|
||||
const arms: Record<string, ArmReport> = {};
|
||||
let totalCostUsd = 0;
|
||||
for (const armName of armNames) {
|
||||
const done = results.filter(
|
||||
(r): r is DoneResult<M> => !r.skipped && r.armName === armName,
|
||||
);
|
||||
const rows = done.map((d) => d.row);
|
||||
const usage = aggregateUsage(done.map((d) => d.usage));
|
||||
totalCostUsd += usage.totalCostUsd;
|
||||
arms[armName] = {
|
||||
runs: rows.length,
|
||||
metrics: aggregate(rows),
|
||||
stats: aggregateStats(rows),
|
||||
usage,
|
||||
};
|
||||
}
|
||||
return { arms, totalCostUsd };
|
||||
}
|
||||
|
||||
/**
|
||||
* The eval orchestration — every arm × trial via `runner`, run through the cache
|
||||
* and a rate-limit retry, with at most `concurrency` in flight and an optional
|
||||
* `maxCostUsd` budget cap; metric + usage computed per run and aggregated per
|
||||
* arm. Exported with an injectable `runner` so the loop, `measure` context,
|
||||
* caching, pooling, and aggregation are unit-testable without spawning a model
|
||||
* (pass a fake returning canned stream-json). `runEval` is this with the real
|
||||
* agent runner.
|
||||
*/
|
||||
export async function runEvalWith<M extends Metrics>(
|
||||
spec: EvalSpec<M>,
|
||||
runner: AgentRunner,
|
||||
): Promise<EvalReport> {
|
||||
const trials = spec.trials ?? 5;
|
||||
const spacing = (spec.spacingSec ?? 4) * 1000;
|
||||
const concurrency = spec.concurrency ?? 1;
|
||||
const retries = spec.rateLimitRetries ?? 3;
|
||||
const backoffMs = spec.retryBackoffMs ?? 1000;
|
||||
const cfg: RunConfig = {
|
||||
model: spec.model ?? "haiku",
|
||||
tools: spec.allowedTools ?? ["Read", "Edit", "Write", "Bash"],
|
||||
timeoutMs: spec.timeoutMs ?? 240000,
|
||||
cache: spec.cache ?? "off",
|
||||
cacheDir: spec.cacheDir ?? resolve(process.cwd(), ".vigiles", "eval-cache"),
|
||||
};
|
||||
const retrying: AgentRunner = (a) =>
|
||||
runWithRetry(a, runner, retries, backoffMs);
|
||||
|
||||
const units = buildUnits(spec.arms, trials);
|
||||
let spent = 0;
|
||||
let aborted = false;
|
||||
const worker = async (unit: Unit): Promise<UnitResult<M>> => {
|
||||
if (aborted) return { armName: unit.armName, skipped: true };
|
||||
const { row, usage } = await executeTrial(
|
||||
spec,
|
||||
unit.arm,
|
||||
unit.trialIndex,
|
||||
retrying,
|
||||
cfg,
|
||||
);
|
||||
spent += usage.costUsd;
|
||||
if (spec.maxCostUsd !== undefined && spent >= spec.maxCostUsd) {
|
||||
aborted = true;
|
||||
}
|
||||
if (spacing > 0) await sleep(spacing);
|
||||
return { armName: unit.armName, skipped: false, row, usage };
|
||||
};
|
||||
|
||||
const results = await runPool(units, concurrency, worker);
|
||||
const { arms, totalCostUsd } = aggregateArms<M>(
|
||||
Object.keys(spec.arms),
|
||||
results,
|
||||
);
|
||||
return { name: spec.name ?? "eval", trials, arms, totalCostUsd, aborted };
|
||||
}
|
||||
|
||||
/** Render one metric: `name=mean±se pass^k=…` (se/pass^k shown when measured). */
|
||||
function formatMetric(
|
||||
name: string,
|
||||
mean: number,
|
||||
stat: MetricStat | undefined,
|
||||
): string {
|
||||
const base =
|
||||
stat && stat.se > 0
|
||||
? `${name}=${mean.toFixed(2)}±${stat.se.toFixed(2)}`
|
||||
: `${name}=${mean.toFixed(2)}`;
|
||||
return stat && stat.n > 0 ? `${base} pass^k=${String(stat.passK)}` : base;
|
||||
}
|
||||
|
||||
/** Compact tokens like `3.4k`; whole numbers under 1000 stay as-is. */
|
||||
function fmtTokens(n: number): string {
|
||||
return n >= 1000 ? `${(n / 1000).toFixed(1)}k` : String(n);
|
||||
}
|
||||
|
||||
/** A `($0.0123 · 1.2s/run · 3.4k tok)` suffix, or "" when no usage was reported. */
|
||||
function formatUsage(u: ArmUsage): string {
|
||||
if (u.totalCostUsd === 0 && u.totalInputTokens + u.totalOutputTokens === 0) {
|
||||
return "";
|
||||
}
|
||||
const tok = fmtTokens(u.totalInputTokens + u.totalOutputTokens);
|
||||
return ` ($${u.totalCostUsd.toFixed(4)} · ${(u.meanDurationMs / 1000).toFixed(1)}s/run · ${tok} tok)`;
|
||||
}
|
||||
|
||||
/** Format an eval report as a compact table for the console (mean ± se, pass^k). */
|
||||
export function formatEvalReport(report: EvalReport): string {
|
||||
const header =
|
||||
report.totalCostUsd > 0
|
||||
? `${report.name} (${String(report.trials)} trials/arm) — $${report.totalCostUsd.toFixed(4)} total`
|
||||
: `${report.name} (${String(report.trials)} trials/arm)`;
|
||||
const lines = [header];
|
||||
for (const [arm, r] of Object.entries(report.arms)) {
|
||||
const parts = Object.entries(r.metrics)
|
||||
.map(([k, v]) => formatMetric(k, v, r.stats[k]))
|
||||
.join(" ");
|
||||
lines.push(` ${arm.padEnd(10)} ${parts}${formatUsage(r.usage)}`);
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
// --- trigger-rate: does a skill/behaviour actually FIRE across varied prompts ---
|
||||
|
||||
/**
|
||||
* Measure how reliably a skill/behaviour *triggers*. A skill's value is its
|
||||
* description firing on the right task — the #1 documented skill-authoring pain —
|
||||
* and that's a property of the real model, not the wiring (which the
|
||||
* deterministic tier already proves). Install the plugin natively (`pluginDir`),
|
||||
* give a set of varied `prompts`, and a `fired` predicate over the run's `Trace`
|
||||
* (reuse the bare predicates, e.g. `(t) => skillResolved(t, "x:y")`).
|
||||
*/
|
||||
export interface TriggerRateSpec {
|
||||
/** Plugin dir installed natively (`--plugin-dir`) so its skills/commands activate. */
|
||||
readonly pluginDir: string;
|
||||
/** The varied prompts to test the trigger against. */
|
||||
readonly prompts: readonly string[];
|
||||
/** Did the behaviour fire on this run? e.g. `(t) => skillResolved(t, "x:y")`. */
|
||||
readonly fired: (trace: Trace) => boolean;
|
||||
/** Trials per prompt. Default 1. */
|
||||
readonly trials?: number;
|
||||
/** Model alias. Default "haiku". */
|
||||
readonly model?: string;
|
||||
/** Tools the agent may use. Default: Read Edit Write Bash Skill. */
|
||||
readonly allowedTools?: readonly string[];
|
||||
/** Per-run timeout ms. Default 240000. */
|
||||
readonly timeoutMs?: number;
|
||||
/** Seconds to wait between runs (avoid rate-limit bursts). Default 4. */
|
||||
readonly spacingSec?: number;
|
||||
}
|
||||
|
||||
/** Per-prompt trigger result: how many of its trials fired. */
|
||||
export interface PromptTriggerStat {
|
||||
readonly prompt: string;
|
||||
readonly fired: number;
|
||||
readonly trials: number;
|
||||
/** `fired / trials` (0 when no trials). */
|
||||
readonly rate: number;
|
||||
}
|
||||
|
||||
export interface TriggerRateReport {
|
||||
/** Overall fraction of runs in which the behaviour fired (0..1). */
|
||||
readonly rate: number;
|
||||
/** Total runs (prompts × trials). */
|
||||
readonly n: number;
|
||||
readonly perPrompt: readonly PromptTriggerStat[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger-rate orchestration — every prompt × trial via `runner`, the `fired`
|
||||
* predicate evaluated per run and aggregated into an overall + per-prompt rate.
|
||||
* Exported with an injectable `runner` so the loop is unit-testable without a
|
||||
* model; `measureTriggerRate` is this with the real agent runner.
|
||||
*/
|
||||
export async function measureTriggerRateWith(
|
||||
spec: TriggerRateSpec,
|
||||
runner: AgentRunner,
|
||||
): Promise<TriggerRateReport> {
|
||||
const trials = spec.trials ?? 1;
|
||||
const model = spec.model ?? "haiku";
|
||||
const tools = spec.allowedTools ?? ["Read", "Edit", "Write", "Bash"];
|
||||
const tools = spec.allowedTools ?? ["Read", "Edit", "Write", "Bash", "Skill"];
|
||||
const timeoutMs = spec.timeoutMs ?? 240000;
|
||||
const spacing = (spec.spacingSec ?? 4) * 1000;
|
||||
|
||||
const arms: Record<string, ArmReport> = {};
|
||||
for (const [armName, arm] of Object.entries(spec.arms)) {
|
||||
const rows: Metrics[] = [];
|
||||
const perPrompt: PromptTriggerStat[] = [];
|
||||
let firedTotal = 0;
|
||||
let n = 0;
|
||||
for (const prompt of spec.prompts) {
|
||||
let fired = 0;
|
||||
for (let t = 0; t < trials; t++) {
|
||||
const cwd = mkdtempSync(join(tmpdir(), "vigiles-eval-"));
|
||||
const cwd = mkdtempSync(join(tmpdir(), "vigiles-trigger-"));
|
||||
try {
|
||||
const { files, settings } = resolveHarness({
|
||||
plugin: arm.plugin,
|
||||
settings: arm.settings,
|
||||
files: { ...spec.fixture, ...arm.files },
|
||||
});
|
||||
writeFiles(cwd, files);
|
||||
const hasSettings = settings !== undefined;
|
||||
if (hasSettings) {
|
||||
writeFileSync(
|
||||
join(cwd, "settings.json"),
|
||||
JSON.stringify(settings, null, 2).replaceAll("{cwd}", cwd),
|
||||
);
|
||||
}
|
||||
const out = await spawnAgent(
|
||||
spec.task,
|
||||
const out = await runner({
|
||||
task: prompt,
|
||||
cwd,
|
||||
model,
|
||||
tools,
|
||||
hasSettings,
|
||||
arm.pluginDir,
|
||||
hasSettings: false,
|
||||
pluginDir: spec.pluginDir,
|
||||
timeoutMs,
|
||||
);
|
||||
rows.push(spec.measure(makeContext(cwd, out)));
|
||||
});
|
||||
if (spec.fired(makeContext(cwd, out))) fired++;
|
||||
} finally {
|
||||
rmSync(cwd, { recursive: true, force: true });
|
||||
await sleep(spacing);
|
||||
}
|
||||
}
|
||||
arms[armName] = {
|
||||
runs: rows.length,
|
||||
metrics: aggregate(rows),
|
||||
stats: aggregateStats(rows),
|
||||
};
|
||||
perPrompt.push({
|
||||
prompt,
|
||||
fired,
|
||||
trials,
|
||||
rate: trials > 0 ? fired / trials : 0,
|
||||
});
|
||||
firedTotal += fired;
|
||||
n += trials;
|
||||
}
|
||||
return { name: spec.name ?? "eval", trials, arms };
|
||||
return { rate: n > 0 ? firedTotal / n : 0, n, perPrompt };
|
||||
}
|
||||
|
||||
/** Format an eval report as a compact table for the console (mean ± se). */
|
||||
export function formatEvalReport(report: EvalReport): string {
|
||||
const lines = [`${report.name} (${String(report.trials)} trials/arm)`];
|
||||
for (const [arm, r] of Object.entries(report.arms)) {
|
||||
const parts = Object.entries(r.metrics)
|
||||
.map(([k, v]) => {
|
||||
const se = r.stats[k]?.se ?? 0;
|
||||
return se > 0
|
||||
? `${k}=${v.toFixed(2)}±${se.toFixed(2)}`
|
||||
: `${k}=${v.toFixed(2)}`;
|
||||
})
|
||||
.join(" ");
|
||||
lines.push(` ${arm.padEnd(10)} ${parts}`);
|
||||
/* v8 ignore start -- real claude subprocess; thin wrapper over measureTriggerRateWith */
|
||||
/**
|
||||
* Measure a skill/behaviour's real trigger rate across prompts × trials against
|
||||
* the real `claude` CLI. Requires `claude` + model auth.
|
||||
*/
|
||||
export async function measureTriggerRate(
|
||||
spec: TriggerRateSpec,
|
||||
): Promise<TriggerRateReport> {
|
||||
return measureTriggerRateWith(spec, spawnAgent);
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
/** Format a trigger-rate report: overall %, then each prompt's rate. */
|
||||
export function formatTriggerRateReport(report: TriggerRateReport): string {
|
||||
const pct = (report.rate * 100).toFixed(0);
|
||||
const lines = [`trigger-rate: ${pct}% (${String(report.n)} runs)`];
|
||||
for (const p of report.perPrompt) {
|
||||
lines.push(` ${p.rate.toFixed(2)} ${p.prompt.slice(0, 60)}`);
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* Tests for the YAML frontmatter rule parser.
|
||||
*/
|
||||
|
||||
import { describe, it } from "node:test";
|
||||
import { describe, it } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { parseFrontmatterRules, hasFrontmatterRules } from "./frontmatter.js";
|
||||
|
||||
+384
-4
@@ -3,32 +3,77 @@
|
||||
* eval delta helpers and the jest/vitest matchers. The pure logic is exercised
|
||||
* here with fake result/report objects (no model, no claude).
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
improvement,
|
||||
assertImproves,
|
||||
significantlyBeats,
|
||||
assertSignificant,
|
||||
reliable,
|
||||
assertReliable,
|
||||
assertCreated,
|
||||
assertNotCreated,
|
||||
assertServedTurns,
|
||||
assertHookBlocked,
|
||||
assertHookAllowed,
|
||||
assertAgentOk,
|
||||
assertAgentErr,
|
||||
assertAgentResult,
|
||||
usedTool,
|
||||
toolCount,
|
||||
skillResolved,
|
||||
toolUsedWith,
|
||||
outputContains,
|
||||
requestContains,
|
||||
assertRequestContains,
|
||||
hookFired,
|
||||
hookBlocked,
|
||||
assertToolUsed,
|
||||
assertToolNotUsed,
|
||||
assertSkillResolved,
|
||||
assertToolUsedWith,
|
||||
assertOutputContains,
|
||||
assertHookFired,
|
||||
assertToolCount,
|
||||
assertToolSequence,
|
||||
assertToolCalls,
|
||||
vigilesMatchers,
|
||||
} from "./harness-assert.js";
|
||||
import { result } from "./spec.js";
|
||||
import type { EvalReport } from "./eval.js";
|
||||
import type { HarnessTestResult } from "./harness-test.js";
|
||||
import type { HarnessTestResult, HookFire } from "./harness-test.js";
|
||||
import type { HookRunResult } from "./run-hook.js";
|
||||
|
||||
/** Minimal HookRunResult stand-in for the run-hook-tier assertions/matcher. */
|
||||
function fakeHook(blocked: boolean): HookRunResult {
|
||||
return {
|
||||
exitCode: blocked ? 2 : 0,
|
||||
stdout: "",
|
||||
stderr: "",
|
||||
json: null,
|
||||
blocked,
|
||||
decision: blocked ? "deny" : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
const NO_USAGE = {
|
||||
totalCostUsd: 0,
|
||||
meanCostUsd: 0,
|
||||
meanDurationMs: 0,
|
||||
totalInputTokens: 0,
|
||||
totalOutputTokens: 0,
|
||||
} as const;
|
||||
|
||||
const report: EvalReport = {
|
||||
name: "demo",
|
||||
trials: 6,
|
||||
totalCostUsd: 0,
|
||||
aborted: false,
|
||||
arms: {
|
||||
vanilla: { runs: 6, metrics: { caught: 0 }, stats: {} },
|
||||
gated: { runs: 6, metrics: { caught: 0.5 }, stats: {} },
|
||||
vanilla: { runs: 6, metrics: { caught: 0 }, stats: {}, usage: NO_USAGE },
|
||||
gated: { runs: 6, metrics: { caught: 0.5 }, stats: {}, usage: NO_USAGE },
|
||||
},
|
||||
};
|
||||
|
||||
@@ -36,6 +81,11 @@ const report: EvalReport = {
|
||||
function fakeResult(
|
||||
present: string[],
|
||||
toolCalls: HarnessTestResult["toolCalls"] = [],
|
||||
extra: {
|
||||
output?: string;
|
||||
hooks?: readonly HookFire[];
|
||||
modelRequests?: HarnessTestResult["modelRequests"];
|
||||
} = {},
|
||||
): HarnessTestResult {
|
||||
return {
|
||||
exitCode: 0,
|
||||
@@ -44,6 +94,9 @@ function fakeResult(
|
||||
cwd: "/tmp/x",
|
||||
turns: 2,
|
||||
toolCalls,
|
||||
hooks: extra.hooks ?? [],
|
||||
output: extra.output ?? "",
|
||||
modelRequests: extra.modelRequests ?? [],
|
||||
file: (p: string) => (present.includes(p) ? "content" : null),
|
||||
cleanup: () => undefined,
|
||||
};
|
||||
@@ -72,6 +125,120 @@ test("assertImproves passes on a positive gap and throws otherwise", () => {
|
||||
});
|
||||
});
|
||||
|
||||
test("significantlyBeats / assertSignificant use a computed noise floor", () => {
|
||||
const stat = (mean: number, se: number, n: number) => ({
|
||||
mean,
|
||||
se,
|
||||
n,
|
||||
std: se * Math.sqrt(n),
|
||||
passK: 0,
|
||||
});
|
||||
const sigReport: EvalReport = {
|
||||
name: "sig",
|
||||
trials: 20,
|
||||
totalCostUsd: 0,
|
||||
aborted: false,
|
||||
arms: {
|
||||
base: {
|
||||
runs: 20,
|
||||
metrics: { caught: 0.1 },
|
||||
stats: { caught: stat(0.1, 0.05, 20) },
|
||||
usage: NO_USAGE,
|
||||
},
|
||||
// a real, tight separation → significant
|
||||
good: {
|
||||
runs: 20,
|
||||
metrics: { caught: 0.6 },
|
||||
stats: { caught: stat(0.6, 0.05, 20) },
|
||||
usage: NO_USAGE,
|
||||
},
|
||||
// a small gap drowned in wide se → not significant
|
||||
noisy: {
|
||||
runs: 5,
|
||||
metrics: { caught: 0.2 },
|
||||
stats: { caught: stat(0.2, 0.2, 5) },
|
||||
usage: NO_USAGE,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
assert.equal(significantlyBeats(sigReport, "base", "good", "caught"), true);
|
||||
assert.equal(significantlyBeats(sigReport, "base", "noisy", "caught"), false);
|
||||
assert.equal(significantlyBeats(sigReport, "base", "good", "missing"), false);
|
||||
|
||||
assert.doesNotThrow(() => {
|
||||
assertSignificant(sigReport, {
|
||||
baseline: "base",
|
||||
arm: "good",
|
||||
metric: "caught",
|
||||
});
|
||||
});
|
||||
// a real but not-significant gap throws
|
||||
assert.throws(() => {
|
||||
assertSignificant(sigReport, {
|
||||
baseline: "base",
|
||||
arm: "noisy",
|
||||
metric: "caught",
|
||||
});
|
||||
});
|
||||
// missing data throws with the no-data message
|
||||
assert.throws(() => {
|
||||
assertSignificant(sigReport, {
|
||||
baseline: "base",
|
||||
arm: "good",
|
||||
metric: "missing",
|
||||
});
|
||||
}, /no data to compare/);
|
||||
// assertImproves delegates to the significance test when asked
|
||||
assert.doesNotThrow(() => {
|
||||
assertImproves(sigReport, {
|
||||
baseline: "base",
|
||||
arm: "good",
|
||||
metric: "caught",
|
||||
significant: true,
|
||||
});
|
||||
});
|
||||
assert.throws(() => {
|
||||
assertImproves(sigReport, {
|
||||
baseline: "base",
|
||||
arm: "noisy",
|
||||
metric: "caught",
|
||||
significant: true,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
test("reliable / assertReliable gate on pass^k (succeeded every trial)", () => {
|
||||
const rep: EvalReport = {
|
||||
name: "rel",
|
||||
trials: 4,
|
||||
totalCostUsd: 0,
|
||||
aborted: false,
|
||||
arms: {
|
||||
flaky: {
|
||||
runs: 4,
|
||||
metrics: { safe: 0.75 },
|
||||
stats: { safe: { mean: 0.75, std: 0.5, se: 0.25, n: 4, passK: 0 } },
|
||||
usage: NO_USAGE,
|
||||
},
|
||||
solid: {
|
||||
runs: 4,
|
||||
metrics: { safe: 1 },
|
||||
stats: { safe: { mean: 1, std: 0, se: 0, n: 4, passK: 1 } },
|
||||
usage: NO_USAGE,
|
||||
},
|
||||
},
|
||||
};
|
||||
assert.equal(reliable(rep, "solid", "safe"), true);
|
||||
assert.equal(reliable(rep, "flaky", "safe"), false);
|
||||
assert.doesNotThrow(() => {
|
||||
assertReliable(rep, { arm: "solid", metric: "safe" });
|
||||
});
|
||||
assert.throws(() => {
|
||||
assertReliable(rep, { arm: "flaky", metric: "safe" });
|
||||
});
|
||||
});
|
||||
|
||||
test("assertCreated / assertNotCreated check the sandbox", () => {
|
||||
const r = fakeResult(["RESULT"]);
|
||||
assert.doesNotThrow(() => {
|
||||
@@ -98,12 +265,84 @@ test("assertServedTurns checks the mock turn count", () => {
|
||||
});
|
||||
});
|
||||
|
||||
test("assertHookBlocked / assertHookAllowed (run-hook results)", () => {
|
||||
assert.doesNotThrow(() => {
|
||||
assertHookBlocked(fakeHook(true));
|
||||
});
|
||||
assert.throws(() => {
|
||||
assertHookBlocked(fakeHook(false));
|
||||
});
|
||||
assert.doesNotThrow(() => {
|
||||
assertHookAllowed(fakeHook(false));
|
||||
});
|
||||
assert.throws(() => {
|
||||
assertHookAllowed(fakeHook(true));
|
||||
});
|
||||
});
|
||||
|
||||
test("assertAgentOk / assertAgentErr / assertAgentResult (subagent outcomes)", () => {
|
||||
const ok = '```vigiles:ok\n{ "summary": "done" }\n```';
|
||||
const err = '```vigiles:err\n{ "reason": "boom" }\n```';
|
||||
const c = result({ summary: "string" }, { reason: "string" });
|
||||
|
||||
// assertAgentOk: returns the value on success; throws on err / malformed
|
||||
assert.deepEqual(assertAgentOk(ok), { summary: "done" });
|
||||
assert.deepEqual(assertAgentOk(ok, c), { summary: "done" });
|
||||
assert.throws(() => assertAgentOk(err), /returned an error result/);
|
||||
assert.throws(() => assertAgentOk("no block here"), /no vigiles/);
|
||||
|
||||
// assertAgentErr: returns the error on failure; throws on ok / malformed
|
||||
assert.deepEqual(assertAgentErr(err), { reason: "boom" });
|
||||
assert.throws(() => assertAgentErr(ok), /returned a success result/);
|
||||
assert.throws(() => assertAgentErr("nope"), /expected an error result/);
|
||||
|
||||
// assertAgentResult: general predicate
|
||||
assert.doesNotThrow(() => {
|
||||
assertAgentResult(ok, (r) => r.kind === "ok" && r.value.summary === "done");
|
||||
});
|
||||
assert.throws(() => {
|
||||
assertAgentResult(ok, (r) => r.kind === "err");
|
||||
}, /did not satisfy the predicate: ok/);
|
||||
// malformed path includes the reason in the message
|
||||
assert.throws(() => {
|
||||
assertAgentResult("plain", (r) => r.kind === "ok");
|
||||
}, /malformed \(no vigiles/);
|
||||
});
|
||||
|
||||
test("vigilesMatchers.toHaveCreated reports pass/fail", () => {
|
||||
const r = fakeResult(["BLOCKED"]);
|
||||
assert.equal(vigilesMatchers.toHaveCreated(r, "BLOCKED").pass, true);
|
||||
assert.equal(vigilesMatchers.toHaveCreated(r, "nope").pass, false);
|
||||
});
|
||||
|
||||
test("vigilesMatchers.toBlock + every matcher's message() render both states", () => {
|
||||
assert.equal(vigilesMatchers.toBlock(fakeHook(true)).pass, true);
|
||||
assert.equal(vigilesMatchers.toBlock(fakeHook(false)).pass, false);
|
||||
// invoke .message() in pass and fail states to cover the message closures
|
||||
assert.match(vigilesMatchers.toBlock(fakeHook(true)).message(), /to block/);
|
||||
assert.match(vigilesMatchers.toBlock(fakeHook(false)).message(), /to block/);
|
||||
assert.match(
|
||||
vigilesMatchers.toHaveCreated(fakeResult(["X"]), "X").message(),
|
||||
/to create/,
|
||||
);
|
||||
assert.match(
|
||||
vigilesMatchers.toHaveCreated(fakeResult([]), "X").message(),
|
||||
/to create/,
|
||||
);
|
||||
assert.match(
|
||||
vigilesMatchers
|
||||
.toBeatBaseline(report, "vanilla", "gated", "caught")
|
||||
.message(),
|
||||
/to beat/,
|
||||
);
|
||||
assert.match(
|
||||
vigilesMatchers
|
||||
.toBeatBaseline(report, "gated", "vanilla", "caught")
|
||||
.message(),
|
||||
/to beat/,
|
||||
);
|
||||
});
|
||||
|
||||
test("vigilesMatchers.toBeatBaseline respects the `by` threshold", () => {
|
||||
assert.equal(
|
||||
vigilesMatchers.toBeatBaseline(report, "vanilla", "gated", "caught").pass,
|
||||
@@ -162,6 +401,147 @@ test("assertSkillResolved: needs a non-error Skill tool_use with that name", ()
|
||||
});
|
||||
});
|
||||
|
||||
// --- bare predicates (the shared vocabulary) -------------------------------
|
||||
|
||||
test("usedTool / toolCount: bare predicates return values, don't throw", () => {
|
||||
const r = fakeResult([], [skillCall, bashCall, bashCall]);
|
||||
assert.equal(usedTool(r, "Skill"), true);
|
||||
assert.equal(usedTool(r, /^Bash$/), true);
|
||||
assert.equal(usedTool(r, "Task"), false);
|
||||
assert.equal(toolCount(r, "Bash"), 2);
|
||||
assert.equal(toolCount(r, /^mcp__/), 0);
|
||||
});
|
||||
|
||||
test("skillResolved: true only for a non-error Skill call by that name", () => {
|
||||
assert.equal(skillResolved(fakeResult([], [skillCall]), "demo:greet"), true);
|
||||
assert.equal(skillResolved(fakeResult([], [skillCall]), "demo:other"), false);
|
||||
const errored = { ...skillCall, isError: true, resultText: "No such skill" };
|
||||
assert.equal(skillResolved(fakeResult([], [errored]), "demo:greet"), false);
|
||||
});
|
||||
|
||||
test("toolUsedWith: matches a tool by name AND its input", () => {
|
||||
const editCall = {
|
||||
name: "Edit",
|
||||
input: { file_path: "src/x.ts", old_string: "a", new_string: "b" },
|
||||
resultText: "",
|
||||
isError: false,
|
||||
};
|
||||
const r = fakeResult([], [editCall]);
|
||||
const targets = (p: string) => (i: unknown) =>
|
||||
(i as { file_path?: string }).file_path === p;
|
||||
assert.equal(toolUsedWith(r, "Edit", targets("src/x.ts")), true);
|
||||
assert.equal(toolUsedWith(r, "Edit", targets("src/other.ts")), false);
|
||||
assert.equal(toolUsedWith(r, "Write", targets("src/x.ts")), false);
|
||||
});
|
||||
|
||||
test("assertToolUsedWith: tool-argument assertion (asserts on input, not name)", () => {
|
||||
const editCall = {
|
||||
name: "Edit",
|
||||
input: { file_path: "note.txt" },
|
||||
resultText: "",
|
||||
isError: false,
|
||||
};
|
||||
const r = fakeResult([], [editCall]);
|
||||
const targets = (p: string) => (i: unknown) =>
|
||||
(i as { file_path?: string }).file_path === p;
|
||||
assertToolUsedWith(r, "Edit", targets("note.txt"));
|
||||
assert.throws(() => {
|
||||
assertToolUsedWith(r, "Edit", targets("WRONG.txt"));
|
||||
});
|
||||
});
|
||||
|
||||
// --- output predicate (DeepEval-style "what did the agent say") ------------
|
||||
|
||||
test("outputContains / assertOutputContains check trace.output", () => {
|
||||
const r = fakeResult([], [], { output: "All done: created RESULT.md" });
|
||||
assert.equal(outputContains(r, "RESULT.md"), true);
|
||||
assert.equal(outputContains(r, /created \w+/), true);
|
||||
assert.equal(outputContains(r, "missing"), false);
|
||||
assertOutputContains(r, "All done");
|
||||
assert.throws(() => {
|
||||
assertOutputContains(r, "nope");
|
||||
});
|
||||
});
|
||||
|
||||
// --- model-request predicate (did the injected context reach the model) ----
|
||||
|
||||
test("requestContains / assertRequestContains search system + messages", () => {
|
||||
const r = fakeResult([], [], {
|
||||
modelRequests: [
|
||||
{
|
||||
system: "You have superpowers. Use the using-superpowers skill.",
|
||||
messages: [{ role: "user", text: "go" }],
|
||||
},
|
||||
{
|
||||
system: "",
|
||||
messages: [
|
||||
{ role: "user", text: "/audit the repo" },
|
||||
{ role: "assistant", text: "on it" },
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
// hits in the system prompt (SessionStart additionalContext shape)
|
||||
assert.equal(requestContains(r, "You have superpowers"), true);
|
||||
assert.equal(requestContains(r, /super\w+/), true);
|
||||
// hits in a message (slash-command expansion shape)
|
||||
assert.equal(requestContains(r, "/audit the repo"), true);
|
||||
assert.equal(requestContains(r, "never sent"), false);
|
||||
assertRequestContains(r, "superpowers");
|
||||
assert.throws(() => {
|
||||
assertRequestContains(r, "never sent");
|
||||
});
|
||||
});
|
||||
|
||||
test("assertRequestContains hints when no requests were captured (eval tier)", () => {
|
||||
const r = fakeResult([], [], { modelRequests: [] });
|
||||
assert.equal(requestContains(r, "anything"), false);
|
||||
assert.throws(() => {
|
||||
assertRequestContains(r, "anything");
|
||||
}, /harness-tier only/);
|
||||
});
|
||||
|
||||
// --- hook predicates (recorded from the stream, not marker files) ----------
|
||||
|
||||
const hookFires: readonly HookFire[] = [
|
||||
{
|
||||
name: "PreToolUse:Edit",
|
||||
event: "PreToolUse",
|
||||
exitCode: 2,
|
||||
blocked: true,
|
||||
output: "BLOCKED",
|
||||
},
|
||||
{
|
||||
name: "PostToolUse:Bash",
|
||||
event: "PostToolUse",
|
||||
exitCode: 0,
|
||||
blocked: false,
|
||||
output: "ok",
|
||||
},
|
||||
];
|
||||
|
||||
test("hookFired / hookBlocked: match by label and by bare event", () => {
|
||||
const r = fakeResult([], [], { hooks: hookFires });
|
||||
assert.equal(hookFired(r, "PreToolUse:Edit"), true); // full label
|
||||
assert.equal(hookFired(r, "PostToolUse"), true); // bare event
|
||||
assert.equal(hookFired(r, /^PreToolUse/), true); // regex
|
||||
assert.equal(hookFired(r, "SessionStart"), false);
|
||||
assert.equal(hookBlocked(r, "PreToolUse"), true); // the Edit hook blocked
|
||||
assert.equal(hookBlocked(r, "PostToolUse"), false); // the Bash hook didn't
|
||||
});
|
||||
|
||||
test("assertHookFired: fires, and { blocked: true } demands a block", () => {
|
||||
const r = fakeResult([], [], { hooks: hookFires });
|
||||
assertHookFired(r, "PreToolUse:Edit");
|
||||
assertHookFired(r, "PreToolUse", { blocked: true });
|
||||
assert.throws(() => {
|
||||
assertHookFired(r, "SessionStart"); // never fired
|
||||
});
|
||||
assert.throws(() => {
|
||||
assertHookFired(r, "PostToolUse", { blocked: true }); // fired but didn't block
|
||||
});
|
||||
});
|
||||
|
||||
// --- sequence / budget invariants (idea 1) ---------------------------------
|
||||
|
||||
const call = (name: string) => ({
|
||||
|
||||
+381
-32
@@ -18,15 +18,25 @@ import {
|
||||
type HarnessTestSpec,
|
||||
type HarnessTestResult,
|
||||
type ToolCall,
|
||||
type Trace,
|
||||
} from "./harness-test.js";
|
||||
import type { EvalReport } from "./eval.js";
|
||||
import type { EvalReport, TriggerRateReport } from "./eval.js";
|
||||
import type { HookRunResult } from "./run-hook.js";
|
||||
import type { OutputContract } from "./spec.js";
|
||||
import { parseAgentResult, type ParsedAgentResult } from "./agent-result.js";
|
||||
import { compareArms } from "./stats.js";
|
||||
|
||||
// Re-export the significance primitives so the whole eval-analysis surface lives
|
||||
// behind `vigiles/harness-assert` (no separate entry point).
|
||||
export { compareArms } from "./stats.js";
|
||||
export type { Comparison } from "./stats.js";
|
||||
|
||||
/**
|
||||
* Run a harness test, hand the result to `fn`, and always clean up the sandbox.
|
||||
* Returns whatever `fn` returns. Use this instead of calling `cleanup()` by
|
||||
* hand — it survives assertion failures.
|
||||
*/
|
||||
/* v8 ignore start -- thin wrapper over runHarnessTest (spawns the real CLI) */
|
||||
export async function withHarness<T>(
|
||||
spec: HarnessTestSpec,
|
||||
fn: (r: HarnessTestResult) => T | Promise<T>,
|
||||
@@ -38,6 +48,7 @@ export async function withHarness<T>(
|
||||
r.cleanup();
|
||||
}
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
// --- Plain throwing assertions (any runner) --------------------------------
|
||||
|
||||
@@ -80,12 +91,164 @@ export function assertHookAllowed(r: HookRunResult): void {
|
||||
}
|
||||
}
|
||||
|
||||
// --- subagent railway outcome (parse the worker's result block) ------------
|
||||
//
|
||||
// A subagent with a result() contract ends its turn with a vigiles:ok/err block.
|
||||
// These wrap parseAgentResult so a test can assert the worker's *outcome* the
|
||||
// same way it asserts a hook decision — the testing-framework payoff of the
|
||||
// railway contract: `assertAgentOk(r.output)` instead of substring-matching prose.
|
||||
|
||||
/**
|
||||
* Assert the worker's output is a success result, and return its `value`. With a
|
||||
* `contract`, the value is validated against the success shape (a wrong/missing
|
||||
* field fails the assertion). A malformed or error result throws.
|
||||
*/
|
||||
export function assertAgentOk(
|
||||
output: string,
|
||||
contract?: OutputContract,
|
||||
): Record<string, unknown> {
|
||||
const r = parseAgentResult(output, contract);
|
||||
if (r.kind === "ok") return r.value;
|
||||
const why = r.kind === "err" ? "returned an error result" : r.reason;
|
||||
return fail(`expected a success result from the subagent, but ${why}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert the worker's output is an error result, and return its `error`. The
|
||||
* railway's error track — proves the worker reported failure with rich detail
|
||||
* (not that it crashed or returned prose). A malformed or success result throws.
|
||||
*/
|
||||
export function assertAgentErr(
|
||||
output: string,
|
||||
contract?: OutputContract,
|
||||
): Record<string, unknown> {
|
||||
const r = parseAgentResult(output, contract);
|
||||
if (r.kind === "err") return r.error;
|
||||
const why = r.kind === "ok" ? "returned a success result" : r.reason;
|
||||
return fail(`expected an error result from the subagent, but ${why}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert the parsed result satisfies `predicate` — the general form, for
|
||||
* checking rich detail (e.g. `(r) => r.kind === "ok" && r.value.files.length > 0`).
|
||||
*/
|
||||
export function assertAgentResult(
|
||||
output: string,
|
||||
predicate: (r: ParsedAgentResult) => boolean,
|
||||
contract?: OutputContract,
|
||||
): void {
|
||||
const r = parseAgentResult(output, contract);
|
||||
if (!predicate(r)) {
|
||||
const detail = r.kind === "malformed" ? ` (${r.reason})` : "";
|
||||
fail(`subagent result did not satisfy the predicate: ${r.kind}${detail}`);
|
||||
}
|
||||
}
|
||||
|
||||
function nameMatches(name: string, pat: string | RegExp): boolean {
|
||||
return typeof pat === "string" ? name === pat : pat.test(name);
|
||||
}
|
||||
|
||||
function toolNames(r: HarnessTestResult): string {
|
||||
return r.toolCalls.map((c) => c.name).join(", ") || "none";
|
||||
function toolNames(trace: Trace): string {
|
||||
return trace.toolCalls.map((c) => c.name).join(", ") || "none";
|
||||
}
|
||||
|
||||
// --- bare predicates over a Trace (the shared vocabulary, no throw) ---------
|
||||
//
|
||||
// Pure fns returning a value, so the SAME vocabulary runs in both consumers:
|
||||
// the throwing `assert*` helpers below wrap them for the testing tier, and an
|
||||
// eval `measure` reuses them directly as metrics (`measure: (t) => ({ safe:
|
||||
// !usedTool(t, /merge|delete/) })`). Never one dual-purpose function.
|
||||
|
||||
/**
|
||||
* Did the agent invoke a tool whose name matches `name` (string = exact,
|
||||
* RegExp = test)? The predicate behind `assertToolUsed` / `assertToolNotUsed`.
|
||||
*/
|
||||
export function usedTool(trace: Trace, name: string | RegExp): boolean {
|
||||
return trace.toolCalls.some((c) => nameMatches(c.name, name));
|
||||
}
|
||||
|
||||
/** How many tools matching `name` the agent invoked. Behind `assertToolCount`. */
|
||||
export function toolCount(trace: Trace, name: string | RegExp): number {
|
||||
return trace.toolCalls.filter((c) => nameMatches(c.name, name)).length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Did the `Skill` tool resolve `skill` (e.g. `"superpowers:test-driven-development"`)
|
||||
* without error? The skill-activation predicate behind `assertSkillResolved`.
|
||||
*/
|
||||
export function skillResolved(trace: Trace, skill: string): boolean {
|
||||
const call = trace.toolCalls.find(
|
||||
(c) =>
|
||||
c.name === "Skill" && (c.input as { skill?: string })?.skill === skill,
|
||||
);
|
||||
return call !== undefined && !call.isError;
|
||||
}
|
||||
|
||||
/**
|
||||
* Did the agent invoke a tool matching `name` whose INPUT satisfies
|
||||
* `inputMatcher` — a tool-ARGUMENT predicate (DeepEval-style), e.g. an `Edit`
|
||||
* that targeted the right file. The predicate behind `assertToolUsedWith`.
|
||||
*/
|
||||
export function toolUsedWith(
|
||||
trace: Trace,
|
||||
name: string | RegExp,
|
||||
inputMatcher: (input: unknown) => boolean,
|
||||
): boolean {
|
||||
return trace.toolCalls.some(
|
||||
(c) => nameMatches(c.name, name) && inputMatcher(c.input),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Does the agent's final answer (`trace.output`) contain `needle` (string =
|
||||
* substring, RegExp = test)? The output predicate behind `assertOutputContains`
|
||||
* — the DeepEval-style "what did the agent actually say" check.
|
||||
*/
|
||||
export function outputContains(trace: Trace, needle: string | RegExp): boolean {
|
||||
return typeof needle === "string"
|
||||
? trace.output.includes(needle)
|
||||
: needle.test(trace.output);
|
||||
}
|
||||
|
||||
/** All text the model received across every request (system + every message). */
|
||||
function requestText(trace: Trace): string {
|
||||
return trace.modelRequests
|
||||
.map((r) => [r.system, ...r.messages.map((m) => m.text)].join("\n"))
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Did ANY request the model received contain `needle` — searching the system
|
||||
* prompt and every message across all requests? The predicate that proves
|
||||
* injected context *reached the model*: a SessionStart hook's `additionalContext`
|
||||
* or a slash command's expansion. Harness tier only — the eval tier drives the
|
||||
* real API, so its `modelRequests` (and this) is empty. Behind `assertRequestContains`.
|
||||
*/
|
||||
export function requestContains(
|
||||
trace: Trace,
|
||||
needle: string | RegExp,
|
||||
): boolean {
|
||||
const text = requestText(trace);
|
||||
return typeof needle === "string" ? text.includes(needle) : needle.test(text);
|
||||
}
|
||||
|
||||
/**
|
||||
* Did a hook matching `name` fire? Matches against both the hook label
|
||||
* (`"PreToolUse:Edit"`) and the bare event (`"PreToolUse"`), so `/PreToolUse/`
|
||||
* or `"PreToolUse:Edit"` both work. The predicate behind `assertHookFired`.
|
||||
*/
|
||||
export function hookFired(trace: Trace, name: string | RegExp): boolean {
|
||||
return trace.hooks.some(
|
||||
(h) => nameMatches(h.name, name) || nameMatches(h.event, name),
|
||||
);
|
||||
}
|
||||
|
||||
/** Did a hook matching `name` fire AND block (exit ≠ 0 / outcome "error")? */
|
||||
export function hookBlocked(trace: Trace, name: string | RegExp): boolean {
|
||||
return trace.hooks.some(
|
||||
(h) =>
|
||||
(nameMatches(h.name, name) || nameMatches(h.event, name)) && h.blocked,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -94,13 +257,10 @@ function toolNames(r: HarnessTestResult): string {
|
||||
* a subagent (`"Task"`). Needs `transcript: true`. The action invariant the
|
||||
* skill/MCP/command surfaces are really about.
|
||||
*/
|
||||
export function assertToolUsed(
|
||||
r: HarnessTestResult,
|
||||
name: string | RegExp,
|
||||
): void {
|
||||
if (!r.toolCalls.some((c) => nameMatches(c.name, name))) {
|
||||
export function assertToolUsed(trace: Trace, name: string | RegExp): void {
|
||||
if (!usedTool(trace, name)) {
|
||||
fail(
|
||||
`expected a tool matching ${String(name)} to be used; tools used: [${toolNames(r)}] (did you set transcript:true?)`,
|
||||
`expected a tool matching ${String(name)} to be used; tools used: [${toolNames(trace)}] (did you set transcript:true?)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -110,11 +270,9 @@ export function assertToolUsed(
|
||||
* (e.g. a destructive MCP tool was never called). "File unchanged" can pass by
|
||||
* accident; "the tool was never used" is the real invariant. Needs `transcript`.
|
||||
*/
|
||||
export function assertToolNotUsed(
|
||||
r: HarnessTestResult,
|
||||
name: string | RegExp,
|
||||
): void {
|
||||
const hit = r.toolCalls.find((c) => nameMatches(c.name, name));
|
||||
export function assertToolNotUsed(trace: Trace, name: string | RegExp): void {
|
||||
// `find` is the negative of `usedTool` and narrows the hit for the message.
|
||||
const hit = trace.toolCalls.find((c) => nameMatches(c.name, name));
|
||||
if (hit) {
|
||||
fail(
|
||||
`expected no tool matching ${String(name)} to be used, but ${hit.name} was`,
|
||||
@@ -126,13 +284,16 @@ export function assertToolNotUsed(
|
||||
* Assert the `Skill` tool resolved `skill` (e.g. `"superpowers:test-driven-development"`)
|
||||
* without error — the correct skill-activation invariant, vs. grepping the body.
|
||||
*/
|
||||
export function assertSkillResolved(r: HarnessTestResult, skill: string): void {
|
||||
const call = r.toolCalls.find(
|
||||
export function assertSkillResolved(trace: Trace, skill: string): void {
|
||||
if (skillResolved(trace, skill)) return;
|
||||
// skillResolved is false → either no matching Skill call, or it errored.
|
||||
// Reconstruct which, for a useful message.
|
||||
const call = trace.toolCalls.find(
|
||||
(c) =>
|
||||
c.name === "Skill" && (c.input as { skill?: string })?.skill === skill,
|
||||
);
|
||||
if (!call) {
|
||||
const seen = r.toolCalls
|
||||
const seen = trace.toolCalls
|
||||
.filter((c) => c.name === "Skill")
|
||||
.map((c) => (c.input as { skill?: string })?.skill ?? "?")
|
||||
.join(", ");
|
||||
@@ -140,9 +301,93 @@ export function assertSkillResolved(r: HarnessTestResult, skill: string): void {
|
||||
`expected the Skill tool to resolve "${skill}"; Skill calls: [${seen || "none"}]`,
|
||||
);
|
||||
}
|
||||
if (call.isError) {
|
||||
fail(
|
||||
`the Skill "${skill}" was invoked but errored: ${call.resultText.slice(0, 200)}`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert the agent invoked a tool matching `name` whose INPUT satisfies
|
||||
* `inputMatcher` — a tool-ARGUMENT invariant (DeepEval-style). Asserts not just
|
||||
* *that* a tool ran but *with what args*, e.g. an `Edit` that targeted the right
|
||||
* file: `assertToolUsedWith(r, "Edit", (i) => (i as { file_path?: string })
|
||||
* .file_path === "src/x.ts")`. Needs `transcript`.
|
||||
*/
|
||||
export function assertToolUsedWith(
|
||||
trace: Trace,
|
||||
name: string | RegExp,
|
||||
inputMatcher: (input: unknown) => boolean,
|
||||
message?: string,
|
||||
): void {
|
||||
if (!toolUsedWith(trace, name, inputMatcher)) {
|
||||
const seen = trace.toolCalls
|
||||
.filter((c) => nameMatches(c.name, name))
|
||||
.map((c) => JSON.stringify(c.input))
|
||||
.join(", ");
|
||||
fail(
|
||||
`the Skill "${skill}" was invoked but errored: ${call.resultText.slice(0, 200)}`,
|
||||
message ??
|
||||
`expected a ${String(name)} call whose input matches; ${String(name)} inputs: [${seen || "none"}]`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Assert the agent's final answer contains `needle` (string substring / RegExp). */
|
||||
export function assertOutputContains(
|
||||
trace: Trace,
|
||||
needle: string | RegExp,
|
||||
): void {
|
||||
if (!outputContains(trace, needle)) {
|
||||
const shown = trace.output.slice(0, 200) || "(empty)";
|
||||
fail(
|
||||
`expected the agent's final answer to contain ${String(needle)}; got: ${shown}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert some request the model received contained `needle` — the "did the
|
||||
* injected context land" invariant (SessionStart `additionalContext`, slash
|
||||
* command expansion). Harness tier only; a zero-request trace fails with a hint
|
||||
* that the eval tier can't capture requests.
|
||||
*/
|
||||
export function assertRequestContains(
|
||||
trace: Trace,
|
||||
needle: string | RegExp,
|
||||
): void {
|
||||
if (!requestContains(trace, needle)) {
|
||||
const n = trace.modelRequests.length;
|
||||
const hint =
|
||||
n === 0
|
||||
? " (no requests captured — modelRequests is harness-tier only)"
|
||||
: "";
|
||||
fail(
|
||||
`expected a model request to contain ${String(needle)}; ${String(n)} request(s) captured${hint}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function hookNames(trace: Trace): string {
|
||||
return trace.hooks.map((h) => h.name).join(", ") || "none";
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert a hook matching `name` fired (and, with `{ blocked: true }`, that it
|
||||
* blocked) — the honest hook-firing check, recorded from the run's stream rather
|
||||
* than inferred from a marker file the hook had to write. Needs `transcript`.
|
||||
*/
|
||||
export function assertHookFired(
|
||||
trace: Trace,
|
||||
name: string | RegExp,
|
||||
opts: { blocked?: boolean } = {},
|
||||
): void {
|
||||
if (!hookFired(trace, name)) {
|
||||
fail(
|
||||
`expected a hook matching ${String(name)} to fire; hooks fired: [${hookNames(trace)}] (did you set transcript:true?)`,
|
||||
);
|
||||
}
|
||||
if (opts.blocked === true && !hookBlocked(trace, name)) {
|
||||
fail(
|
||||
`expected a hook matching ${String(name)} to block, but none did; hooks fired: [${hookNames(trace)}]`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -155,18 +400,18 @@ export function assertSkillResolved(r: HarnessTestResult, skill: string): void {
|
||||
* "never touched it"). Catches runaway loops and wasted work. Needs `transcript`.
|
||||
*/
|
||||
export function assertToolCount(
|
||||
r: HarnessTestResult,
|
||||
trace: Trace,
|
||||
name: string | RegExp,
|
||||
bounds: { min?: number; max?: number; exactly?: number },
|
||||
): void {
|
||||
const n = r.toolCalls.filter((c) => nameMatches(c.name, name)).length;
|
||||
const n = toolCount(trace, name);
|
||||
const ok =
|
||||
(bounds.exactly === undefined || n === bounds.exactly) &&
|
||||
(bounds.min === undefined || n >= bounds.min) &&
|
||||
(bounds.max === undefined || n <= bounds.max);
|
||||
if (!ok) {
|
||||
fail(
|
||||
`expected count of ${String(name)} to satisfy ${JSON.stringify(bounds)}, got ${String(n)} (tools: [${toolNames(r)}])`,
|
||||
`expected count of ${String(name)} to satisfy ${JSON.stringify(bounds)}, got ${String(n)} (tools: [${toolNames(trace)}])`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -178,17 +423,17 @@ export function assertToolCount(
|
||||
* Needs `transcript`.
|
||||
*/
|
||||
export function assertToolSequence(
|
||||
r: HarnessTestResult,
|
||||
trace: Trace,
|
||||
names: ReadonlyArray<string | RegExp>,
|
||||
): void {
|
||||
let i = 0;
|
||||
for (const c of r.toolCalls) {
|
||||
for (const c of trace.toolCalls) {
|
||||
const want = names[i];
|
||||
if (want !== undefined && nameMatches(c.name, want)) i++;
|
||||
}
|
||||
if (i < names.length) {
|
||||
fail(
|
||||
`expected tools in order [${names.map((n) => String(n)).join(" → ")}]; got [${toolNames(r)}]`,
|
||||
`expected tools in order [${names.map((n) => String(n)).join(" → ")}]; got [${toolNames(trace)}]`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -199,12 +444,41 @@ export function assertToolSequence(
|
||||
* was preceded by a Read of that file". Needs `transcript`.
|
||||
*/
|
||||
export function assertToolCalls(
|
||||
r: HarnessTestResult,
|
||||
trace: Trace,
|
||||
predicate: (calls: readonly ToolCall[]) => boolean,
|
||||
message = "tool-call invariant failed",
|
||||
): void {
|
||||
if (!predicate(r.toolCalls)) {
|
||||
fail(`${message}; tools used: [${toolNames(r)}]`);
|
||||
if (!predicate(trace.toolCalls)) {
|
||||
fail(`${message}; tools used: [${toolNames(trace)}]`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Did `arm` succeed on EVERY trial for `metric` — τ-bench pass^k = 1? The
|
||||
* reliability predicate over an eval report (vs. `improvement`, which reads the
|
||||
* mean gap). Reads `report.arms[arm].stats[metric].passK`.
|
||||
*/
|
||||
export function reliable(
|
||||
report: EvalReport,
|
||||
arm: string,
|
||||
metric: string,
|
||||
): boolean {
|
||||
return report.arms[arm]?.stats[metric]?.passK === 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert `arm` passed `metric` on every trial (pass^k = 1) — the reliability
|
||||
* gate for a non-deterministic harness ("worked every time", not "on average").
|
||||
*/
|
||||
export function assertReliable(
|
||||
report: EvalReport,
|
||||
opts: { arm: string; metric: string },
|
||||
): void {
|
||||
if (!reliable(report, opts.arm, opts.metric)) {
|
||||
const pk = report.arms[opts.arm]?.stats[opts.metric]?.passK;
|
||||
fail(
|
||||
`expected ${opts.arm} to pass ${opts.metric} on every trial (pass^k=1), got pass^k=${String(pk ?? "n/a")}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -221,14 +495,73 @@ export function improvement(
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert `arm` beats `baseline` on `metric` by more than `by`. With `by` left at
|
||||
* 0 this just asserts a positive gap; pass the combined se to demand the gap
|
||||
* clear the noise floor.
|
||||
* Did `arm` *significantly* beat `baseline` on `metric` — a positive gap whose
|
||||
* two-sided Welch t-test p-value is below `alpha` (default 0.05)? The grounded
|
||||
* upgrade over `improvement`: the noise floor is computed from the arms' spread,
|
||||
* not hand-fed. False when either arm/metric is missing. See `src/stats.ts`.
|
||||
*/
|
||||
// eslint-disable-next-line max-params -- positional predicate mirrors `improvement` + alpha
|
||||
export function significantlyBeats(
|
||||
report: EvalReport,
|
||||
baseline: string,
|
||||
arm: string,
|
||||
metric: string,
|
||||
alpha = 0.05,
|
||||
): boolean {
|
||||
const c = compareArms(report, baseline, arm, metric, alpha);
|
||||
return c !== null && c.delta > 0 && c.significant;
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert `arm` significantly beats `baseline` on `metric` (positive gap, p < α).
|
||||
* The statistical gate for a non-deterministic A/B — "the gap clears the noise",
|
||||
* with the noise floor computed, not supplied. The honest version of
|
||||
* `assertImproves(..., { by: se })`.
|
||||
*/
|
||||
export function assertSignificant(
|
||||
report: EvalReport,
|
||||
opts: { baseline: string; arm: string; metric: string; alpha?: number },
|
||||
): void {
|
||||
const c = compareArms(
|
||||
report,
|
||||
opts.baseline,
|
||||
opts.arm,
|
||||
opts.metric,
|
||||
opts.alpha,
|
||||
);
|
||||
if (c === null) {
|
||||
fail(
|
||||
`no data to compare ${opts.arm} vs ${opts.baseline} on ${opts.metric}`,
|
||||
);
|
||||
}
|
||||
const alpha = opts.alpha ?? 0.05;
|
||||
if (!(c.delta > 0 && c.significant)) {
|
||||
fail(
|
||||
`expected ${opts.arm} to significantly beat ${opts.baseline} on ${opts.metric} (α=${String(alpha)}); Δ=${c.delta.toFixed(3)}, p=${c.pValue.toFixed(3)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert `arm` beats `baseline` on `metric`. By default just a positive gap > `by`
|
||||
* (pass the combined se to clear the noise floor by hand). Pass `{ significant:
|
||||
* true }` to demand a Welch t-test at `alpha` instead — the computed noise floor.
|
||||
*/
|
||||
export function assertImproves(
|
||||
report: EvalReport,
|
||||
opts: { baseline: string; arm: string; metric: string; by?: number },
|
||||
opts: {
|
||||
baseline: string;
|
||||
arm: string;
|
||||
metric: string;
|
||||
by?: number;
|
||||
significant?: boolean;
|
||||
alpha?: number;
|
||||
},
|
||||
): void {
|
||||
if (opts.significant === true) {
|
||||
assertSignificant(report, opts);
|
||||
return;
|
||||
}
|
||||
const by = opts.by ?? 0;
|
||||
const delta = improvement(report, opts.baseline, opts.arm, opts.metric);
|
||||
if (delta <= by) {
|
||||
@@ -238,6 +571,22 @@ export function assertImproves(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Assert a skill/behaviour triggered on at least `min` (0..1) of its runs — the
|
||||
* reliability gate for a skill's *activation* (does its description fire on the
|
||||
* task), over a {@link TriggerRateReport} from `measureTriggerRate`.
|
||||
*/
|
||||
export function assertTriggerRate(
|
||||
report: TriggerRateReport,
|
||||
opts: { min: number },
|
||||
): void {
|
||||
if (report.rate < opts.min) {
|
||||
fail(
|
||||
`expected a trigger rate ≥ ${String(opts.min)}, got ${report.rate.toFixed(2)} (${String(report.n)} runs)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// --- jest / vitest matchers (expect.extend) --------------------------------
|
||||
|
||||
interface MatcherOutput {
|
||||
|
||||
+187
-1
@@ -8,7 +8,7 @@
|
||||
* tests below. An earlier claude version gated them headlessly; the tests lock in
|
||||
* that they work on current CLIs (verified on 2.1.169) and catch a re-gate.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -17,18 +17,55 @@ import {
|
||||
scriptModel,
|
||||
claudeAvailable,
|
||||
parseToolCalls,
|
||||
parseOutput,
|
||||
parseHooks,
|
||||
buildClaudeArgs,
|
||||
} from "./harness-test.js";
|
||||
import {
|
||||
assertToolUsed,
|
||||
assertToolNotUsed,
|
||||
assertSkillResolved,
|
||||
assertToolUsedWith,
|
||||
assertToolSequence,
|
||||
assertToolCount,
|
||||
assertToolCalls,
|
||||
assertHookFired,
|
||||
} from "./harness-assert.js";
|
||||
|
||||
const maybe = claudeAvailable() ? test : test.skip;
|
||||
|
||||
// Pure: the shared claude argv (no model, no claude). Covers the transcript /
|
||||
// pluginDir / settings / default-tools branches.
|
||||
test("buildClaudeArgs: transcript, pluginDir, settings, and tool defaults", () => {
|
||||
const base = buildClaudeArgs({ model: scriptModel([]) }, false);
|
||||
assert.deepEqual(base.slice(0, 2), ["-p", "go"]); // default prompt
|
||||
assert.ok(base.includes("json") && !base.includes("stream-json"));
|
||||
assert.ok(!base.includes("--plugin-dir") && !base.includes("--settings"));
|
||||
// default allowed tools come last
|
||||
assert.deepEqual(base.slice(-5), [
|
||||
"--allowedTools",
|
||||
"Read",
|
||||
"Edit",
|
||||
"Write",
|
||||
"Bash",
|
||||
]);
|
||||
|
||||
const full = buildClaudeArgs(
|
||||
{
|
||||
model: scriptModel([]),
|
||||
prompt: "do it",
|
||||
transcript: true,
|
||||
pluginDir: "examples/harness/fixture-skill-plugin",
|
||||
allowedTools: ["Bash"],
|
||||
},
|
||||
true,
|
||||
);
|
||||
assert.deepEqual(full.slice(0, 2), ["-p", "do it"]);
|
||||
assert.ok(full.includes("stream-json") && full.includes("--verbose"));
|
||||
assert.ok(full.includes("--plugin-dir") && full.includes("--settings"));
|
||||
assert.deepEqual(full.slice(-2), ["--allowedTools", "Bash"]);
|
||||
});
|
||||
|
||||
// Regression: a PostToolUse hook fires on an Edit/Write tool use. This is the
|
||||
// deterministic-tier capability a stale comment once said was impossible; the
|
||||
// 2026-06-09 spike showed it firing 3/3, so we lock it in.
|
||||
@@ -51,6 +88,7 @@ maybe("a PostToolUse hook fires on an Edit/Write tool use", async () => {
|
||||
],
|
||||
},
|
||||
},
|
||||
transcript: true, // capture the stream so r.hooks records the firing
|
||||
model: scriptModel([
|
||||
{ tool: "Write", input: { file_path: "hello.txt", content: "banana" } },
|
||||
{ text: "done" },
|
||||
@@ -66,6 +104,9 @@ maybe("a PostToolUse hook fires on an Edit/Write tool use", async () => {
|
||||
/FIRED/,
|
||||
"the Write|Edit PostToolUse hook fired",
|
||||
);
|
||||
// Honest check: the hook firing is recorded in the stream, not just inferred
|
||||
// from the marker file above.
|
||||
assertHookFired(r, "PostToolUse");
|
||||
} finally {
|
||||
r.cleanup();
|
||||
}
|
||||
@@ -94,6 +135,7 @@ maybe("a PreToolUse hook blocks an Edit tool use", async () => {
|
||||
],
|
||||
},
|
||||
},
|
||||
transcript: true, // capture the stream so r.hooks records the block decision
|
||||
model: scriptModel([
|
||||
{ tool: "Read", input: { file_path: "note.txt" } },
|
||||
{
|
||||
@@ -115,6 +157,9 @@ maybe("a PreToolUse hook blocks an Edit tool use", async () => {
|
||||
"old",
|
||||
"the PreToolUse hook blocked the edit (file unchanged)",
|
||||
);
|
||||
// Honest check: the hook fired AND its decision was a block — recorded from
|
||||
// the stream, not inferred from the marker file + "file unchanged" pair.
|
||||
assertHookFired(r, "PreToolUse:Edit", { blocked: true });
|
||||
} finally {
|
||||
r.cleanup();
|
||||
}
|
||||
@@ -179,6 +224,7 @@ maybe(
|
||||
);
|
||||
const r = await runHarnessTest({
|
||||
pluginDir,
|
||||
sandbox: false, // in-repo fixture we authored → trusted, run direct
|
||||
allowedTools: ["Read", "Edit", "Write", "Bash", "Skill"],
|
||||
transcript: true, // populate r.toolCalls
|
||||
model: scriptModel([
|
||||
@@ -217,6 +263,7 @@ for (const [label, dir, skill] of [
|
||||
maybe(`a real ${label} skill resolves via --plugin-dir`, async () => {
|
||||
const r = await runHarnessTest({
|
||||
pluginDir: join(__dirname, dir),
|
||||
sandbox: false, // pinned vendored plugin we audited → trusted, run direct
|
||||
allowedTools: ["Read", "Edit", "Write", "Bash", "Skill"],
|
||||
transcript: true,
|
||||
model: scriptModel([{ tool: "Skill", input: { skill } }, { text: "ok" }]),
|
||||
@@ -252,6 +299,13 @@ maybe("tool-call sequence + budget invariants hold on a real run", async () => {
|
||||
assertToolSequence(r, ["Read", "Edit"]); // ordering
|
||||
assertToolCount(r, "Edit", { max: 1 }); // budget
|
||||
assertToolCount(r, "Write", { exactly: 0 });
|
||||
// tool-ARGUMENT invariant: the Edit targeted the right file (not just "an Edit ran")
|
||||
assertToolUsedWith(
|
||||
r,
|
||||
"Edit",
|
||||
(i) => (i as { file_path?: string }).file_path === "note.txt",
|
||||
);
|
||||
assert.equal(typeof r.output, "string"); // unified Trace: final answer captured
|
||||
assertToolCalls(
|
||||
r,
|
||||
(calls) => {
|
||||
@@ -301,3 +355,135 @@ test("parseToolCalls: pairs tool_use with tool_result from a stream-json transcr
|
||||
assert.equal(calls[0]?.isError, false);
|
||||
assert.equal(parseToolCalls("{not stream json}").length, 0);
|
||||
});
|
||||
|
||||
test("parseToolCalls: covers id / content-shape / error / no-result branches", () => {
|
||||
const stream = [
|
||||
"", // blank line skipped
|
||||
"not json — ignored", // parse error skipped
|
||||
JSON.stringify({ type: "x", message: { content: "notarray" } }), // content not an array
|
||||
JSON.stringify({
|
||||
type: "a",
|
||||
message: {
|
||||
content: [
|
||||
{ type: "text", text: "prose" }, // neither tool_use nor tool_result
|
||||
{ type: "tool_use", id: "u1", name: "A", input: {} },
|
||||
{ type: "tool_use", input: {} }, // tool_use with no name → skipped
|
||||
{ type: "tool_use", name: "NoId", input: {} }, // tool_use with no id
|
||||
{ type: "tool_use", id: "u3", name: "C", input: {} },
|
||||
],
|
||||
},
|
||||
}),
|
||||
JSON.stringify({
|
||||
type: "u",
|
||||
message: {
|
||||
content: [
|
||||
{
|
||||
type: "tool_result",
|
||||
tool_use_id: "u1",
|
||||
content: "plain",
|
||||
is_error: true,
|
||||
},
|
||||
{ type: "tool_result", content: ["x", { text: "y" }, { z: 1 }] }, // no id; array w/ string + text + no-text
|
||||
{ type: "tool_result", tool_use_id: "u3", content: 42 }, // non-string, non-array content
|
||||
],
|
||||
},
|
||||
}),
|
||||
].join("\n");
|
||||
const calls = parseToolCalls(stream);
|
||||
assert.equal(calls.length, 3); // A, NoId, C (the no-name tool_use is skipped)
|
||||
const a = calls.find((c) => c.name === "A");
|
||||
assert.equal(a?.resultText, "plain"); // string content
|
||||
assert.equal(a?.isError, true);
|
||||
// The no-id tool_use and the no-id tool_result both key on "" and so pair up;
|
||||
// the array content ["x", {text:"y"}, {no text}] joins to "xy".
|
||||
const noid = calls.find((c) => c.name === "NoId");
|
||||
assert.equal(noid?.resultText, "xy");
|
||||
assert.equal(noid?.isError, false);
|
||||
const c = calls.find((c) => c.name === "C");
|
||||
assert.equal(c?.resultText, ""); // non-string/array content (42) → ""
|
||||
});
|
||||
|
||||
test("parseOutput: returns the final answer from the terminal result event", () => {
|
||||
const stream = [
|
||||
"", // blank line → skipped
|
||||
JSON.stringify({ type: "assistant", message: { content: [] } }),
|
||||
"not json — ignored",
|
||||
JSON.stringify({
|
||||
type: "result",
|
||||
subtype: "success",
|
||||
result: "the answer",
|
||||
}),
|
||||
].join("\n");
|
||||
assert.equal(parseOutput(stream), "the answer");
|
||||
// a result event whose `result` is non-string → ""
|
||||
assert.equal(parseOutput(JSON.stringify({ type: "result", result: 42 })), "");
|
||||
// single-object `--output-format json` carries the same {type:"result"} shape
|
||||
assert.equal(
|
||||
parseOutput(JSON.stringify({ type: "result", result: "x" })),
|
||||
"x",
|
||||
);
|
||||
assert.equal(parseOutput("no result event here"), "");
|
||||
});
|
||||
|
||||
test("parseHooks: records hook firing + block decision from stream events", () => {
|
||||
const stream = [
|
||||
JSON.stringify({
|
||||
type: "system",
|
||||
subtype: "hook_response",
|
||||
hook_name: "PostToolUse:Bash",
|
||||
hook_event: "PostToolUse",
|
||||
exit_code: 0,
|
||||
outcome: "success",
|
||||
output: "POST_OK\n",
|
||||
}),
|
||||
JSON.stringify({
|
||||
type: "system",
|
||||
subtype: "hook_response",
|
||||
hook_name: "PreToolUse:Edit",
|
||||
hook_event: "PreToolUse",
|
||||
exit_code: 2,
|
||||
outcome: "error",
|
||||
output: "BLOCKED\n",
|
||||
}),
|
||||
JSON.stringify({ type: "assistant", message: { content: [] } }), // ignored
|
||||
].join("\n");
|
||||
const hooks = parseHooks(stream);
|
||||
assert.equal(hooks.length, 2);
|
||||
assert.equal(hooks[0]?.name, "PostToolUse:Bash");
|
||||
assert.equal(hooks[0]?.blocked, false);
|
||||
assert.equal(hooks[1]?.event, "PreToolUse");
|
||||
assert.equal(hooks[1]?.exitCode, 2);
|
||||
assert.equal(hooks[1]?.blocked, true);
|
||||
assert.equal(parseHooks("{not stream json}").length, 0);
|
||||
});
|
||||
|
||||
test("parseHooks: defensive field coercion + the block decision branches", () => {
|
||||
const stream = [
|
||||
// outcome success but a non-zero exit → blocked via the exit-code arm
|
||||
JSON.stringify({
|
||||
type: "system",
|
||||
subtype: "hook_response",
|
||||
hook_name: "Stop",
|
||||
hook_event: "Stop",
|
||||
exit_code: 1,
|
||||
outcome: "success",
|
||||
output: "x",
|
||||
}),
|
||||
// malformed: non-number exit_code, missing name/event/output → coerced
|
||||
JSON.stringify({
|
||||
type: "system",
|
||||
subtype: "hook_response",
|
||||
exit_code: "nope",
|
||||
}),
|
||||
// a non-hook_response system event → skipped
|
||||
JSON.stringify({ type: "system", subtype: "init" }),
|
||||
].join("\n");
|
||||
const hooks = parseHooks(stream);
|
||||
assert.equal(hooks.length, 2);
|
||||
assert.equal(hooks[0]?.blocked, true); // success + exit 1 → blocked
|
||||
assert.equal(hooks[1]?.exitCode, undefined); // non-number → undefined
|
||||
assert.equal(hooks[1]?.name, ""); // missing → ""
|
||||
assert.equal(hooks[1]?.event, ""); // missing → ""
|
||||
assert.equal(hooks[1]?.output, ""); // missing → ""
|
||||
assert.equal(hooks[1]?.blocked, false); // not error, no numeric exit
|
||||
});
|
||||
|
||||
+244
-47
@@ -22,10 +22,11 @@
|
||||
* The "steps" are the scripted model turns — their real home is deterministic
|
||||
* harness testing, not production enforcement.
|
||||
*
|
||||
* Note: the simple mock drives the Bash tool and Stop hooks reliably; the
|
||||
* Edit/Write tools are gated in headless mode and don't fire via the mock —
|
||||
* drive file actions through Bash, or use the real-model eval tier (`eval.ts`)
|
||||
* for Edit/Write hooks.
|
||||
* Note: the mock drives Bash and Stop hooks, and — verified on claude 2.1.169 —
|
||||
* the Edit/Write tools too (allowlisted past the permission prompt), so their
|
||||
* PreToolUse/PostToolUse hooks fire in this tier. The events the mock can't
|
||||
* trigger (PreCompact / Notification / SessionEnd / SubagentStop) belong to the
|
||||
* `runHook` unit tier.
|
||||
*/
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import {
|
||||
@@ -39,11 +40,28 @@ import {
|
||||
import { tmpdir } from "node:os";
|
||||
import { resolve, join, dirname } from "node:path";
|
||||
|
||||
import { startMock, type ModelTurn } from "./mock-model.js";
|
||||
import { startMock, type ModelTurn, type ModelRequest } from "./mock-model.js";
|
||||
import { resolveHarness } from "./plugin-loader.js";
|
||||
import {
|
||||
decideSandbox,
|
||||
specTrusted,
|
||||
sandboxAvailable,
|
||||
runSandboxed,
|
||||
type SandboxMode,
|
||||
} from "./sandbox.js";
|
||||
|
||||
export { scriptModel, type ModelTurn } from "./mock-model.js";
|
||||
export {
|
||||
scriptModel,
|
||||
type ModelTurn,
|
||||
type ModelRequest,
|
||||
} from "./mock-model.js";
|
||||
export { loadPlugin, resolveHarness } from "./plugin-loader.js";
|
||||
export {
|
||||
decideSandbox,
|
||||
specTrusted,
|
||||
sandboxAvailable,
|
||||
type SandboxMode,
|
||||
} from "./sandbox.js";
|
||||
|
||||
export interface HarnessTestSpec {
|
||||
/** Fixture files to write in a fresh temp working dir (path → contents). */
|
||||
@@ -81,9 +99,83 @@ export interface HarnessTestSpec {
|
||||
readonly transcript?: boolean;
|
||||
/** Per-run wall-clock timeout in ms. Default 60000. */
|
||||
readonly timeoutMs?: number;
|
||||
/**
|
||||
* Confinement policy for the code this run executes (`src/sandbox.ts`).
|
||||
* Default `"auto"` is safe-by-default: an inline-only spec (you authored it)
|
||||
* runs directly, but an external `plugin` / `pluginDir` brings in untrusted
|
||||
* third-party hooks and is run under bubblewrap — or, if no sandbox is
|
||||
* available, the run REFUSES rather than executing unconfined. Pass `false` to
|
||||
* opt out and run unconfined (you audited the code, or trust the outer
|
||||
* container); `"strict"` to force confinement even for trusted code.
|
||||
*
|
||||
* NOTE: confined execution is **Linux only** (bubblewrap is a Linux tool). On
|
||||
* macOS / Windows no sandbox is available, so an untrusted run will REFUSE
|
||||
* under `"auto"`/`"strict"` — use `sandbox: false` there if you trust the code.
|
||||
*/
|
||||
readonly sandbox?: SandboxMode;
|
||||
}
|
||||
|
||||
export interface HarnessTestResult {
|
||||
/**
|
||||
* A hook invocation observed during the run, recorded (not inferred) from the
|
||||
* `hook_response` system events the CLI emits in the stream — so a test can
|
||||
* assert which hook fired and whether it blocked, instead of inferring it from a
|
||||
* marker file the hook had to write.
|
||||
*/
|
||||
export interface HookFire {
|
||||
/** The hook label, e.g. `"PreToolUse:Edit"` (`Event:Matcher`). */
|
||||
readonly name: string;
|
||||
/** The hook event, e.g. `"PreToolUse"`, `"PostToolUse"`, `"Stop"`. */
|
||||
readonly event: string;
|
||||
/** The hook process exit code (2 = block), or undefined if not reported. */
|
||||
readonly exitCode: number | undefined;
|
||||
/** Whether the hook blocked / errored (exit ≠ 0 or outcome "error"). */
|
||||
readonly blocked: boolean;
|
||||
/** What the hook printed (its block reason / diagnostic), or "". */
|
||||
readonly output: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The observable record of ONE run — the unified shape produced by BOTH testing
|
||||
* tiers: `runHarnessTest`'s result and `runEval`'s `measure` ctx (`eval.ts`)
|
||||
* both satisfy it. That's what lets the bare predicates in `harness-assert.ts`
|
||||
* (`usedTool` / `skillResolved` / `toolCount` / `toolUsedWith` / `hookFired` /
|
||||
* `outputContains`) run over either, with the testing helpers asserting and eval
|
||||
* measuring over the same vocabulary.
|
||||
*/
|
||||
export interface Trace {
|
||||
/**
|
||||
* The tools the agent invoked, each paired with its result — parsed from the
|
||||
* transcript. Empty unless the run captured the stream (`transcript: true` on
|
||||
* the harness tier; always on the eval tier). Lets a test assert on the
|
||||
* agent's *actions* (skills, MCP tools, subagents) instead of grepping stdout.
|
||||
*/
|
||||
readonly toolCalls: readonly ToolCall[];
|
||||
/**
|
||||
* The hooks that fired during the run, each with its decision — parsed from
|
||||
* the CLI's `hook_response` stream events. Same capture requirement as
|
||||
* `toolCalls` (empty without the stream). Lets a test assert hook firing
|
||||
* honestly instead of via a marker file.
|
||||
*/
|
||||
readonly hooks: readonly HookFire[];
|
||||
/** The agent's final answer text (the terminal `result` event), or "". */
|
||||
readonly output: string;
|
||||
/**
|
||||
* The requests the model received, captured by the scripted mock — each with
|
||||
* its `system` prompt and `messages`, flattened to text. Lets a test assert
|
||||
* what actually reached the model (a SessionStart hook's injected context, a
|
||||
* slash command's expansion), not just that a hook fired. **Harness tier
|
||||
* only**: the mock sees the requests, so this is populated by `runHarnessTest`
|
||||
* (with or without `transcript`); the eval tier drives the real API, so its
|
||||
* `modelRequests` is always empty.
|
||||
*/
|
||||
readonly modelRequests: readonly ModelRequest[];
|
||||
/** Number of model turns. */
|
||||
readonly turns: number;
|
||||
/** Final contents of a file under the working dir, or null if absent. */
|
||||
file(path: string): string | null;
|
||||
}
|
||||
|
||||
export interface HarnessTestResult extends Trace {
|
||||
readonly exitCode: number;
|
||||
readonly stdout: string;
|
||||
/** Hook block messages and diagnostics land here. */
|
||||
@@ -92,14 +184,6 @@ export interface HarnessTestResult {
|
||||
readonly cwd: string;
|
||||
/** Number of model turns the agent took (mock turns served). */
|
||||
readonly turns: number;
|
||||
/**
|
||||
* The tools the agent invoked, each paired with its result — parsed from the
|
||||
* transcript. Empty unless `transcript: true`. Lets a test assert on the
|
||||
* agent's *actions* (skills, MCP tools, subagents) instead of grepping stdout.
|
||||
*/
|
||||
readonly toolCalls: readonly ToolCall[];
|
||||
/** Final contents of a file under the working dir, or null if absent. */
|
||||
file(path: string): string | null;
|
||||
/** Remove the temp working dir. */
|
||||
cleanup(): void;
|
||||
}
|
||||
@@ -166,6 +250,99 @@ export function parseToolCalls(streamJson: string): ToolCall[] {
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* The terminal `result` event — present in BOTH `--output-format` shapes (a
|
||||
* `{type:"result", …}` line in stream-json, the single object in `json`), or
|
||||
* null. The seam for the final answer + turn count without parsing twice.
|
||||
*/
|
||||
export function parseResultEvent(
|
||||
stdout: string,
|
||||
): Record<string, unknown> | null {
|
||||
for (const line of stdout.split("\n")) {
|
||||
if (!line.trim()) continue;
|
||||
let evt: Record<string, unknown>;
|
||||
try {
|
||||
evt = JSON.parse(line) as Record<string, unknown>;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (evt.type === "result") return evt;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** The agent's final answer text from a transcript / result object, or "". */
|
||||
export function parseOutput(stdout: string): string {
|
||||
const result = parseResultEvent(stdout)?.result;
|
||||
return typeof result === "string" ? result : "";
|
||||
}
|
||||
|
||||
/**
|
||||
* The hooks that fired, recorded from the CLI's `hook_response` stream events
|
||||
* (`--output-format stream-json`). Each carries the hook name/event, its exit
|
||||
* code, and whether it blocked — the honest record vs. inferring from marker
|
||||
* files. Returns [] for the non-stream `json` output (no per-hook events).
|
||||
*/
|
||||
function toHookFire(evt: Record<string, unknown>): HookFire {
|
||||
const exitCode =
|
||||
typeof evt.exit_code === "number" ? evt.exit_code : undefined;
|
||||
return {
|
||||
name: typeof evt.hook_name === "string" ? evt.hook_name : "",
|
||||
event: typeof evt.hook_event === "string" ? evt.hook_event : "",
|
||||
exitCode,
|
||||
blocked:
|
||||
evt.outcome === "error" || (exitCode !== undefined && exitCode !== 0),
|
||||
output: typeof evt.output === "string" ? evt.output : "",
|
||||
};
|
||||
}
|
||||
|
||||
export function parseHooks(stdout: string): HookFire[] {
|
||||
const hooks: HookFire[] = [];
|
||||
for (const line of stdout.split("\n")) {
|
||||
if (!line.trim()) continue;
|
||||
let evt: Record<string, unknown>;
|
||||
try {
|
||||
evt = JSON.parse(line) as Record<string, unknown>;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (evt.type === "system" && evt.subtype === "hook_response") {
|
||||
hooks.push(toHookFire(evt));
|
||||
}
|
||||
}
|
||||
return hooks;
|
||||
}
|
||||
|
||||
/**
|
||||
* The `claude` CLI argv for a harness run (shared by the direct and sandboxed
|
||||
* paths). `ANTHROPIC_BASE_URL` is set by the caller's environment / wrapper, not
|
||||
* here. Pure, so the arg shape is unit-tested.
|
||||
*/
|
||||
export function buildClaudeArgs(
|
||||
spec: HarnessTestSpec,
|
||||
hasSettings: boolean,
|
||||
): string[] {
|
||||
const tools = spec.allowedTools ?? ["Read", "Edit", "Write", "Bash"];
|
||||
return [
|
||||
"-p",
|
||||
spec.prompt ?? "go",
|
||||
...(spec.transcript
|
||||
? ["--output-format", "stream-json", "--verbose"]
|
||||
: ["--output-format", "json"]),
|
||||
"--model",
|
||||
"claude-sonnet-4-5",
|
||||
...(spec.pluginDir !== undefined
|
||||
? ["--plugin-dir", resolve(spec.pluginDir)]
|
||||
: []),
|
||||
...(hasSettings ? ["--settings", "settings.json"] : []),
|
||||
"--allowedTools",
|
||||
...tools,
|
||||
];
|
||||
}
|
||||
|
||||
/* v8 ignore start -- spawns the real claude CLI + filesystem; exercised by the
|
||||
claude-backed suite, excluded from the deterministic coverage gate (the parse
|
||||
helpers above carry the testable logic). */
|
||||
/** Whether the `claude` CLI is available — harness tests need it. */
|
||||
export function claudeAvailable(): boolean {
|
||||
try {
|
||||
@@ -232,10 +409,22 @@ function spawnClaude(
|
||||
/**
|
||||
* Run the real `claude` CLI against a scripted mock model, with the given
|
||||
* fixture and settings (hooks). Deterministic — same script, same result.
|
||||
*
|
||||
* Safe by default: an external `plugin` / `pluginDir` brings in untrusted
|
||||
* third-party hooks and is confined under bubblewrap (`spec.sandbox`, default
|
||||
* `"auto"`); if no sandbox is available the run REFUSES rather than executing
|
||||
* unconfined. See `src/sandbox.ts`.
|
||||
*/
|
||||
export async function runHarnessTest(
|
||||
spec: HarnessTestSpec,
|
||||
): Promise<HarnessTestResult> {
|
||||
const decision = decideSandbox({
|
||||
trusted: specTrusted(spec),
|
||||
mode: spec.sandbox ?? "auto",
|
||||
available: sandboxAvailable(),
|
||||
});
|
||||
if (decision.action === "throw") throw new Error(decision.reason);
|
||||
|
||||
const cwd = mkdtempSync(join(tmpdir(), "vigiles-harness-"));
|
||||
const { files, settings } = resolveHarness({
|
||||
plugin: spec.plugin,
|
||||
@@ -243,42 +432,50 @@ export async function runHarnessTest(
|
||||
files: spec.files,
|
||||
});
|
||||
writeFixture(cwd, files, settings);
|
||||
const args = buildClaudeArgs(spec, settings !== undefined);
|
||||
const timeoutMs = spec.timeoutMs ?? 60000;
|
||||
|
||||
const build = (
|
||||
out: { code: number; stdout: string; stderr?: string },
|
||||
turns: number,
|
||||
modelRequests: readonly ModelRequest[],
|
||||
): HarnessTestResult => ({
|
||||
exitCode: out.code,
|
||||
stdout: out.stdout,
|
||||
stderr: out.stderr ?? "",
|
||||
cwd,
|
||||
turns,
|
||||
toolCalls: parseToolCalls(out.stdout),
|
||||
hooks: parseHooks(out.stdout),
|
||||
output: parseOutput(out.stdout),
|
||||
modelRequests,
|
||||
file: (p: string): string | null => {
|
||||
const f = resolve(cwd, p);
|
||||
return existsSync(f) ? readFileSync(f, "utf-8") : null;
|
||||
},
|
||||
cleanup: (): void => {
|
||||
rmSync(cwd, { recursive: true, force: true });
|
||||
},
|
||||
});
|
||||
|
||||
// Confined path: the mock is co-launched inside the sandbox's netns.
|
||||
if (decision.action === "sandbox") {
|
||||
const out = await runSandboxed({
|
||||
cwd,
|
||||
claudeArgs: args,
|
||||
script: spec.model,
|
||||
timeoutMs,
|
||||
});
|
||||
return build(out, out.requests.length, out.requests);
|
||||
}
|
||||
|
||||
// Direct path: mock runs in this process; claude reaches it over localhost.
|
||||
const mock = await startMock(spec.model);
|
||||
try {
|
||||
const tools = spec.allowedTools ?? ["Read", "Edit", "Write", "Bash"];
|
||||
const args = [
|
||||
"-p",
|
||||
spec.prompt ?? "go",
|
||||
...(spec.transcript
|
||||
? ["--output-format", "stream-json", "--verbose"]
|
||||
: ["--output-format", "json"]),
|
||||
"--model",
|
||||
"claude-sonnet-4-5",
|
||||
...(spec.pluginDir !== undefined
|
||||
? ["--plugin-dir", resolve(spec.pluginDir)]
|
||||
: []),
|
||||
...(settings !== undefined ? ["--settings", "settings.json"] : []),
|
||||
"--allowedTools",
|
||||
...tools,
|
||||
];
|
||||
const out = await spawnClaude(args, cwd, mock.url, spec.timeoutMs ?? 60000);
|
||||
return {
|
||||
exitCode: out.code,
|
||||
stdout: out.stdout,
|
||||
stderr: out.stderr,
|
||||
cwd,
|
||||
turns: mock.count,
|
||||
toolCalls: parseToolCalls(out.stdout),
|
||||
file: (p: string): string | null => {
|
||||
const f = resolve(cwd, p);
|
||||
return existsSync(f) ? readFileSync(f, "utf-8") : null;
|
||||
},
|
||||
cleanup: (): void => {
|
||||
rmSync(cwd, { recursive: true, force: true });
|
||||
},
|
||||
};
|
||||
const out = await spawnClaude(args, cwd, mock.url, timeoutMs);
|
||||
return build(out, mock.count, [...mock.requests]);
|
||||
} finally {
|
||||
mock.close();
|
||||
}
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
* Tests for the inline-rule parser.
|
||||
*/
|
||||
|
||||
import { describe, it } from "node:test";
|
||||
import { describe, it } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { parseInlineRules, hasInlineRules } from "./inline.js";
|
||||
|
||||
+8
-1
@@ -3,7 +3,7 @@
|
||||
* itself needs claude + auth, but the verdict parsing is pure — that's what we
|
||||
* pin here (the part most likely to silently break).
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { parseJudgeOutput } from "./judge.js";
|
||||
@@ -50,3 +50,10 @@ test("returns a safe verdict on unparseable / empty output", () => {
|
||||
assert.equal(noScore.score, 0);
|
||||
assert.match(noScore.reason, /unparseable/);
|
||||
});
|
||||
|
||||
test("tolerates malformed JSON between braces (firstJsonObject catch)", () => {
|
||||
// Has `{` and `}` but invalid JSON between → JSON.parse throws → null.
|
||||
const v = parseJudgeOutput("prefix { score: nope, } suffix");
|
||||
assert.equal(v.score, 0);
|
||||
assert.match(v.reason, /unparseable/);
|
||||
});
|
||||
|
||||
@@ -55,6 +55,7 @@ function firstJsonObject(s: string): unknown {
|
||||
}
|
||||
}
|
||||
|
||||
/* v8 ignore start -- spawns the real claude CLI; parseJudgeOutput holds the logic */
|
||||
/** Grade `output` against `rubric` with a model. Synchronous (for `measure`). */
|
||||
export function judge(opts: JudgeOptions): JudgeResult {
|
||||
const threshold = opts.threshold ?? 0.5;
|
||||
@@ -89,6 +90,7 @@ export function judge(opts: JudgeOptions): JudgeResult {
|
||||
}
|
||||
return parseJudgeOutput(res.stdout ?? "", threshold);
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
/**
|
||||
* Parse a verdict out of the grader's stdout — pure, so the parsing is testable
|
||||
|
||||
+1
-1
@@ -462,7 +462,7 @@ function makeResult(
|
||||
* short so edit distance is more appropriate than NCD (which is tuned
|
||||
* for longer texts).
|
||||
*/
|
||||
function editDistance(a: string, b: string): number {
|
||||
export function editDistance(a: string, b: string): number {
|
||||
if (a === b) return 0;
|
||||
const m = a.length;
|
||||
const n = b.length;
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
* REAL minimal MCP server (examples/harness/fixture-mcp-server.mjs) — it speaks
|
||||
* the actual stdio JSON-RPC protocol, so these are deterministic and offline.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { join } from "node:path";
|
||||
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* vigiles — the in-sandbox mock entry.
|
||||
*
|
||||
* Run as a subprocess INSIDE the bubblewrap network namespace (see
|
||||
* `src/sandbox.ts`), so the scripted mock lives on the sandbox's isolated
|
||||
* loopback — reachable by the confined `claude`, unreachable from outside.
|
||||
* Reads the model script from a file, streams each captured request to an ndjson
|
||||
* file the parent reads back (for `trace.modelRequests`), and writes its chosen
|
||||
* port so the wrapper can point `ANTHROPIC_BASE_URL` at it.
|
||||
*
|
||||
* node mock-entry.js <scriptFile> <requestsFile> <portFile>
|
||||
*
|
||||
* Not unit-tested directly (it's a daemon driven only through a live sandbox);
|
||||
* exercised end-to-end by the bwrap-backed integration test.
|
||||
*/
|
||||
import { readFileSync, appendFileSync, writeFileSync } from "node:fs";
|
||||
|
||||
import { startMock, type ModelTurn } from "./mock-model.js";
|
||||
|
||||
void (async (): Promise<void> => {
|
||||
const [scriptFile, requestsFile, portFile] = process.argv.slice(2);
|
||||
if (!scriptFile || !requestsFile || !portFile) {
|
||||
process.stderr.write("mock-entry: scriptFile requestsFile portFile\n");
|
||||
process.exit(2);
|
||||
}
|
||||
const turns = JSON.parse(readFileSync(scriptFile, "utf-8")) as ModelTurn[];
|
||||
const handle = await startMock(turns, {
|
||||
onRequest: (req) => {
|
||||
appendFileSync(requestsFile, JSON.stringify(req) + "\n");
|
||||
},
|
||||
});
|
||||
// Signal readiness last: the wrapper waits for a non-empty port file.
|
||||
writeFileSync(portFile, new URL(handle.url).port);
|
||||
// Stay alive until the wrapper kills us once `claude` has finished.
|
||||
})();
|
||||
@@ -0,0 +1,229 @@
|
||||
/**
|
||||
* Tests for the scripted Anthropic mock (src/mock-model.ts). The mock is an
|
||||
* in-process HTTP server, so its full behaviour — SSE vs JSON turns, tool vs
|
||||
* text turns, count_tokens / HEAD / health tolerance, the onTurn probe, and the
|
||||
* last-turn-repeat / empty-script defaults — is testable directly, no claude.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
startMock,
|
||||
scriptModel,
|
||||
extractRequest,
|
||||
type TurnInfo,
|
||||
} from "./mock-model.js";
|
||||
|
||||
const post = (url: string, body: unknown): Promise<Response> =>
|
||||
fetch(`${url}/v1/messages`, { method: "POST", body: JSON.stringify(body) });
|
||||
|
||||
interface MsgBlock {
|
||||
type?: string;
|
||||
text?: string;
|
||||
input?: unknown;
|
||||
}
|
||||
interface MsgResponse {
|
||||
content?: MsgBlock[];
|
||||
stop_reason?: string;
|
||||
model?: string;
|
||||
}
|
||||
const readMsg = async (r: Response): Promise<MsgResponse> =>
|
||||
(await r.json()) as MsgResponse;
|
||||
|
||||
test("startMock: SSE and JSON turns, tool and text, count/HEAD/health, onTurn", async () => {
|
||||
const seen: TurnInfo[] = [];
|
||||
// Two mocks so all four (stream|json) × (tool|text) combinations are exercised.
|
||||
const streamMock = await startMock(
|
||||
scriptModel([
|
||||
{ tool: "Bash", input: { command: "ls" } },
|
||||
{ text: "all done" },
|
||||
{ tool: "NoInput" }, // tool turn with no input → `turn.input ?? {}`
|
||||
{}, // turn with neither tool nor text → `turn.text ?? ""`
|
||||
]),
|
||||
{ onTurn: (info) => seen.push(info) },
|
||||
);
|
||||
try {
|
||||
// turn 0: streaming tool turn → streamTurn (tool branch)
|
||||
const sse = await (
|
||||
await post(streamMock.url, {
|
||||
stream: true,
|
||||
model: "m",
|
||||
messages: [{ content: "go" }],
|
||||
})
|
||||
).text();
|
||||
assert.match(sse, /"type":"tool_use"/);
|
||||
assert.match(sse, /"name":"Bash"/);
|
||||
// turn 1: streaming text turn → streamTurn (text branch); a tool_result in
|
||||
// the request exercises the onTurn hasToolResult probe.
|
||||
const sse2 = await (
|
||||
await post(streamMock.url, {
|
||||
stream: true,
|
||||
messages: [{ content: [{ type: "tool_result", content: "x" }] }],
|
||||
})
|
||||
).text();
|
||||
assert.match(sse2, /"text_delta"/);
|
||||
assert.match(sse2, /all done/);
|
||||
// turns 2 & 3: streaming tool turn with no input, then a no-text turn —
|
||||
// exercising the `turn.input ?? {}` / `turn.text ?? ""` defaults.
|
||||
assert.match(
|
||||
await (
|
||||
await post(streamMock.url, {
|
||||
stream: true,
|
||||
messages: [{ content: "x" }],
|
||||
})
|
||||
).text(),
|
||||
/"name":"NoInput"/,
|
||||
);
|
||||
assert.match(
|
||||
await (
|
||||
await post(streamMock.url, {
|
||||
stream: true,
|
||||
messages: [{ content: "x" }],
|
||||
})
|
||||
).text(),
|
||||
/message_stop/,
|
||||
);
|
||||
|
||||
// count_tokens (else Claude Code hangs)
|
||||
const counted = (await (
|
||||
await fetch(`${streamMock.url}/v1/messages/count_tokens`, {
|
||||
method: "POST",
|
||||
body: "{}",
|
||||
})
|
||||
).json()) as unknown;
|
||||
assert.deepEqual(counted, { input_tokens: 10 });
|
||||
|
||||
// HEAD + a non-messages health GET both return {}
|
||||
assert.equal(
|
||||
(await fetch(`${streamMock.url}/v1/messages`, { method: "HEAD" })).status,
|
||||
200,
|
||||
);
|
||||
assert.deepEqual(
|
||||
(await (await fetch(`${streamMock.url}/health`)).json()) as unknown,
|
||||
{},
|
||||
);
|
||||
|
||||
assert.equal(seen[0]?.stream, true);
|
||||
assert.equal(seen[1]?.hasToolResult, true);
|
||||
assert.ok(streamMock.count >= 2);
|
||||
} finally {
|
||||
streamMock.close();
|
||||
}
|
||||
|
||||
const jsonMock = await startMock(
|
||||
scriptModel([
|
||||
{ tool: "Bash", input: { command: "ls" } },
|
||||
{ text: "ok" },
|
||||
{ tool: "NoInput" }, // tool turn with no input → `turn.input ?? {}`
|
||||
]),
|
||||
);
|
||||
try {
|
||||
// turn 0: non-streaming tool turn → jsonTurn (tool branch)
|
||||
const tool = await readMsg(
|
||||
await post(jsonMock.url, { messages: [{ content: "go" }] }),
|
||||
);
|
||||
assert.equal(tool.content?.[0]?.type, "tool_use");
|
||||
assert.equal(tool.stop_reason, "tool_use");
|
||||
// turn 1: non-streaming text turn (no model → default echo) → jsonTurn (text)
|
||||
const txt = await readMsg(
|
||||
await post(jsonMock.url, { messages: [{ content: "go" }] }),
|
||||
);
|
||||
assert.equal(txt.content?.[0]?.text, "ok");
|
||||
assert.equal(txt.model, "claude-mock");
|
||||
// turn 2: non-streaming tool turn with no input → `turn.input ?? {}`
|
||||
const noInput = await readMsg(
|
||||
await post(jsonMock.url, { messages: [{ content: "go" }] }),
|
||||
);
|
||||
assert.deepEqual(noInput.content?.[0]?.input, {});
|
||||
} finally {
|
||||
jsonMock.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("extractRequest: flattens system + messages, tolerates odd shapes", () => {
|
||||
// system as a string; message content as a string
|
||||
assert.deepEqual(
|
||||
extractRequest({
|
||||
system: "be brief",
|
||||
messages: [{ role: "user", content: "go" }],
|
||||
}),
|
||||
{ system: "be brief", messages: [{ role: "user", text: "go" }] },
|
||||
);
|
||||
// system as a text-block array; content as a block array (text + non-text)
|
||||
assert.deepEqual(
|
||||
extractRequest({
|
||||
system: [
|
||||
{ type: "text", text: "A" },
|
||||
{ type: "text", text: "B" },
|
||||
],
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
"raw",
|
||||
{ type: "text", text: "C" },
|
||||
{ type: "tool_result", content: "ignored" }, // no `text` → ""
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
{ system: "AB", messages: [{ role: "user", text: "rawC" }] },
|
||||
);
|
||||
// missing system → ""; missing role → ""; non-array messages → []
|
||||
assert.deepEqual(extractRequest({ messages: [{ content: "x" }] }), {
|
||||
system: "",
|
||||
messages: [{ role: "", text: "x" }],
|
||||
});
|
||||
assert.deepEqual(extractRequest({}), { system: "", messages: [] });
|
||||
assert.deepEqual(extractRequest({ messages: "nope" as unknown }), {
|
||||
system: "",
|
||||
messages: [],
|
||||
});
|
||||
});
|
||||
|
||||
test("startMock: captures each request via handle.requests", async () => {
|
||||
const mock = await startMock(scriptModel([{ text: "ok" }]));
|
||||
try {
|
||||
await post(mock.url, {
|
||||
system: "You have superpowers",
|
||||
messages: [{ role: "user", content: "go" }],
|
||||
});
|
||||
// count_tokens / HEAD must NOT be recorded as model requests
|
||||
await fetch(`${mock.url}/v1/messages/count_tokens`, {
|
||||
method: "POST",
|
||||
body: "{}",
|
||||
});
|
||||
await fetch(`${mock.url}/v1/messages`, { method: "HEAD" });
|
||||
await post(mock.url, { messages: [{ role: "user", content: "again" }] });
|
||||
|
||||
assert.equal(mock.requests.length, 2);
|
||||
assert.equal(mock.requests[0]?.system, "You have superpowers");
|
||||
assert.equal(mock.requests[0]?.messages[0]?.text, "go");
|
||||
assert.equal(mock.requests[1]?.messages[0]?.text, "again");
|
||||
} finally {
|
||||
mock.close();
|
||||
}
|
||||
});
|
||||
|
||||
test("startMock: repeats the last turn and defaults an empty script", async () => {
|
||||
const repeat = await startMock(scriptModel([{ text: "only" }]));
|
||||
try {
|
||||
for (let i = 0; i < 2; i++) {
|
||||
const j = await readMsg(
|
||||
await post(repeat.url, { messages: [{ content: "x" }] }),
|
||||
);
|
||||
assert.equal(j.content?.[0]?.text, "only"); // 2nd call repeats the last turn
|
||||
}
|
||||
} finally {
|
||||
repeat.close();
|
||||
}
|
||||
|
||||
const empty = await startMock(scriptModel([]));
|
||||
try {
|
||||
// empty script + no messages field → default { text: "" }
|
||||
const j = await readMsg(await post(empty.url, {}));
|
||||
assert.equal(j.content?.[0]?.text, "");
|
||||
} finally {
|
||||
empty.close();
|
||||
}
|
||||
});
|
||||
+67
-2
@@ -41,11 +41,29 @@ export interface TurnInfo {
|
||||
readonly hasToolResult: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* One `/v1/messages` request the mock received, flattened to text for
|
||||
* assertions. This is the seam that lets a harness test prove what reached the
|
||||
* model — a SessionStart hook's injected `additionalContext`, or a slash
|
||||
* command's expansion — not just that a hook fired.
|
||||
*/
|
||||
export interface ModelRequest {
|
||||
/** The system prompt, flattened to text (string or text-block array). */
|
||||
readonly system: string;
|
||||
/** The conversation messages, each flattened to `{ role, text }`. */
|
||||
readonly messages: readonly {
|
||||
readonly role: string;
|
||||
readonly text: string;
|
||||
}[];
|
||||
}
|
||||
|
||||
export interface MockHandle {
|
||||
readonly url: string;
|
||||
close(): void;
|
||||
/** Number of model turns served so far. */
|
||||
readonly count: number;
|
||||
/** Every `/v1/messages` request the mock received, in order. */
|
||||
readonly requests: readonly ModelRequest[];
|
||||
}
|
||||
|
||||
function writeEvent(res: ServerResponse, event: string, data: unknown): void {
|
||||
@@ -156,7 +174,42 @@ function jsonTurn(res: ServerResponse, turn: ModelTurn, model: string): void {
|
||||
interface ReqBody {
|
||||
stream?: boolean;
|
||||
model?: string;
|
||||
messages?: { content?: unknown }[];
|
||||
system?: unknown;
|
||||
messages?: { role?: unknown; content?: unknown }[];
|
||||
}
|
||||
|
||||
/** Flatten Anthropic content (string, or an array of text/other blocks) to text. */
|
||||
function flattenContent(content: unknown): string {
|
||||
if (typeof content === "string") return content;
|
||||
if (!Array.isArray(content)) return "";
|
||||
return content
|
||||
.map((b) => {
|
||||
if (typeof b === "string") return b;
|
||||
const t = (b as { text?: unknown }).text;
|
||||
return typeof t === "string" ? t : "";
|
||||
})
|
||||
.join("");
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract a {@link ModelRequest} from a request body — the `system` prompt and
|
||||
* `messages`, each flattened to text. Pure and exported so the capture logic is
|
||||
* testable without the HTTP server.
|
||||
*/
|
||||
export function extractRequest(body: {
|
||||
system?: unknown;
|
||||
messages?: unknown;
|
||||
}): ModelRequest {
|
||||
const messages = Array.isArray(body.messages)
|
||||
? body.messages.map((m) => {
|
||||
const msg = m as { role?: unknown; content?: unknown };
|
||||
return {
|
||||
role: typeof msg.role === "string" ? msg.role : "",
|
||||
text: flattenContent(msg.content),
|
||||
};
|
||||
})
|
||||
: [];
|
||||
return { system: flattenContent(body.system), messages };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -166,9 +219,15 @@ interface ReqBody {
|
||||
*/
|
||||
export function startMock(
|
||||
script: readonly ModelTurn[],
|
||||
opts: { onTurn?: (info: TurnInfo) => void } = {},
|
||||
opts: {
|
||||
onTurn?: (info: TurnInfo) => void;
|
||||
/** Called with each `/v1/messages` request as it arrives — used by the
|
||||
* in-sandbox mock entry to stream requests to a file for the parent. */
|
||||
onRequest?: (req: ModelRequest) => void;
|
||||
} = {},
|
||||
): Promise<MockHandle> {
|
||||
let i = 0;
|
||||
const requests: ModelRequest[] = [];
|
||||
const server = http.createServer((req, res) => {
|
||||
let body = "";
|
||||
req.on("data", (c) => (body += c as string));
|
||||
@@ -192,6 +251,9 @@ export function startMock(
|
||||
res.end(JSON.stringify({ input_tokens: 10 }));
|
||||
return;
|
||||
}
|
||||
const request = extractRequest(reqBody);
|
||||
requests.push(request);
|
||||
opts.onRequest?.(request);
|
||||
const last = JSON.stringify(reqBody.messages?.at(-1)?.content ?? "");
|
||||
opts.onTurn?.({
|
||||
n: i,
|
||||
@@ -214,6 +276,9 @@ export function startMock(
|
||||
get count() {
|
||||
return i;
|
||||
},
|
||||
get requests() {
|
||||
return requests;
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
import { describe, it } from "node:test";
|
||||
import { describe, it } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
* real assembled machine (hooks + CLAUDE.md + skills) so a test/eval runs
|
||||
* against what ships, not a retyped subset. Model-free.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
@@ -54,6 +54,45 @@ test("loadPlugin resolves CLAUDE_PLUGIN_ROOT to the absolute plugin path", () =>
|
||||
}
|
||||
});
|
||||
|
||||
test("loadPlugin warns on dangling intra-plugin file references", () => {
|
||||
const root = makeTmpDir("plugin-dangling");
|
||||
try {
|
||||
// a hook script that reads one skill that exists and one that doesn't
|
||||
mkdirSync(join(root, "hooks"), { recursive: true });
|
||||
writeFileSync(
|
||||
join(root, "hooks", "session-start"),
|
||||
// the missing ref appears twice → exercises the dedup (seen) path
|
||||
'cat "$ROOT/skills/present/SKILL.md"\ncat "$ROOT/skills/missing/SKILL.md"\necho "$ROOT/skills/missing/SKILL.md"\n',
|
||||
);
|
||||
mkdirSync(join(root, "skills", "present"), { recursive: true });
|
||||
writeFileSync(join(root, "skills", "present", "SKILL.md"), "# present\n");
|
||||
|
||||
const { warnings } = loadPlugin(root);
|
||||
const dangling = warnings.find((w) => w.includes("intra-plugin"));
|
||||
assert.ok(dangling, "expected a dangling-ref warning");
|
||||
assert.ok(
|
||||
dangling.includes("skills/missing/SKILL.md"),
|
||||
"names the missing ref",
|
||||
);
|
||||
assert.ok(
|
||||
!dangling.includes("skills/present/SKILL.md"),
|
||||
"ignores the present ref",
|
||||
);
|
||||
} finally {
|
||||
cleanupTmpDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test("loadPlugin: a complete plugin has no dangling-ref warning", () => {
|
||||
const root = makePlugin(); // skills/foo/SKILL.md exists, no broken refs
|
||||
try {
|
||||
const { warnings } = loadPlugin(root);
|
||||
assert.ok(!warnings.some((w) => w.includes("intra-plugin")));
|
||||
} finally {
|
||||
cleanupTmpDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test("loadPlugin materializes CLAUDE.md and skills into the sandbox", () => {
|
||||
const root = makePlugin();
|
||||
try {
|
||||
@@ -292,6 +331,37 @@ test("loadPlugin warns when a plugin declares MCP servers", () => {
|
||||
}
|
||||
});
|
||||
|
||||
test("loadPlugin warns when the manifest declares mcpServers (no .mcp.json)", () => {
|
||||
const root = makeTmpDir("mcpmanifest");
|
||||
try {
|
||||
mkdirSync(join(root, ".claude-plugin"), { recursive: true });
|
||||
writeFileSync(join(root, "CLAUDE.md"), "# x\n");
|
||||
writeFileSync(
|
||||
join(root, ".claude-plugin", "plugin.json"),
|
||||
JSON.stringify({ name: "m", mcpServers: { demo: { command: "x" } } }),
|
||||
);
|
||||
assert.ok(loadPlugin(root).warnings.some((w) => w.includes("MCP")));
|
||||
} finally {
|
||||
cleanupTmpDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test("loadPlugin tolerates a malformed plugin.json (does not crash)", () => {
|
||||
const root = makeTmpDir("badmanifest");
|
||||
try {
|
||||
mkdirSync(join(root, ".claude-plugin"), { recursive: true });
|
||||
writeFileSync(join(root, "CLAUDE.md"), "# x\n");
|
||||
writeFileSync(join(root, ".claude-plugin", "plugin.json"), "{ not json");
|
||||
const loaded = loadPlugin(root);
|
||||
// malformed manifest → no hooks, no MCP warning; CLAUDE.md still loads.
|
||||
assert.deepEqual(loaded.settings, {});
|
||||
assert.ok(!loaded.warnings.some((w) => w.includes("MCP")));
|
||||
assert.equal(loaded.files["CLAUDE.md"], "# x\n");
|
||||
} finally {
|
||||
cleanupTmpDir(root);
|
||||
}
|
||||
});
|
||||
|
||||
test("a fully-covered plugin (hooks + CLAUDE.md + skills) has no warnings", () => {
|
||||
const root = makePlugin();
|
||||
try {
|
||||
|
||||
+58
-22
@@ -37,22 +37,20 @@ export interface LoadedPlugin {
|
||||
readonly warnings: readonly string[];
|
||||
}
|
||||
|
||||
interface PluginManifest {
|
||||
hooks?: unknown;
|
||||
skills?: string;
|
||||
mcpServers?: unknown;
|
||||
}
|
||||
|
||||
const MAX_SKILL_FILE_BYTES = 256 * 1024;
|
||||
|
||||
/** Parse a JSON file, or null on any error (missing / malformed). */
|
||||
function safeReadJson(path: string): Record<string, unknown> | null {
|
||||
try {
|
||||
return JSON.parse(readFileSync(path, "utf-8")) as Record<string, unknown>;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Read and return the `.hooks` field of a JSON file, or undefined on any error. */
|
||||
function readHooksFile(path: string): unknown {
|
||||
try {
|
||||
return (JSON.parse(readFileSync(path, "utf-8")) as { hooks?: unknown })
|
||||
.hooks;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
return safeReadJson(path)?.hooks;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -63,9 +61,10 @@ function readHooksFile(path: string): unknown {
|
||||
* 4. a plain repo's `.claude/settings.json`.
|
||||
*/
|
||||
function readHooks(root: string): unknown {
|
||||
const manifestPath = join(root, ".claude-plugin", "plugin.json");
|
||||
if (existsSync(manifestPath)) {
|
||||
const m = JSON.parse(readFileSync(manifestPath, "utf-8")) as PluginManifest;
|
||||
// A malformed plugin.json must not crash the loader — fall through to the
|
||||
// other layouts (safeReadJson returns null on a parse error).
|
||||
const m = safeReadJson(join(root, ".claude-plugin", "plugin.json"));
|
||||
if (m) {
|
||||
if (typeof m.hooks === "string") return readHooksFile(join(root, m.hooks));
|
||||
if (m.hooks !== undefined) return m.hooks;
|
||||
}
|
||||
@@ -167,6 +166,15 @@ function pluginWarnings(
|
||||
`plugin declares MCP server(s) (mcpServers / .mcp.json) — the loader does not wire MCP; bring the server up yourself if your test needs it.`,
|
||||
);
|
||||
}
|
||||
const dangling = danglingRefs(root);
|
||||
if (dangling.length) {
|
||||
const shown = dangling.slice(0, 5).join(", ");
|
||||
const more =
|
||||
dangling.length > 5 ? `, … (+${String(dangling.length - 5)})` : "";
|
||||
warnings.push(
|
||||
`plugin references ${String(dangling.length)} intra-plugin file(s) that don't exist (broken path / partial vendor): ${shown}${more}`,
|
||||
);
|
||||
}
|
||||
if (!hooks && Object.keys(files).length === 0) {
|
||||
warnings.push(
|
||||
`nothing was loaded (no hooks, CLAUDE.md, skills, agents, or commands) — the deterministic harness would run an effectively empty machine.`,
|
||||
@@ -178,14 +186,42 @@ function pluginWarnings(
|
||||
/** Whether the plugin declares any MCP servers (manifest field or .mcp.json). */
|
||||
function hasMcp(root: string): boolean {
|
||||
if (existsSync(join(root, ".mcp.json"))) return true;
|
||||
const manifestPath = join(root, ".claude-plugin", "plugin.json");
|
||||
if (!existsSync(manifestPath)) return false;
|
||||
try {
|
||||
const m = JSON.parse(readFileSync(manifestPath, "utf-8")) as PluginManifest;
|
||||
return m.mcpServers !== undefined;
|
||||
} catch {
|
||||
return false;
|
||||
return (
|
||||
safeReadJson(join(root, ".claude-plugin", "plugin.json"))?.mcpServers !==
|
||||
undefined
|
||||
);
|
||||
}
|
||||
|
||||
// A plugin-relative path reference to a file under a standard surface dir, with a
|
||||
// known extension — e.g. a hook script that `cat`s `skills/using-superpowers/SKILL.md`.
|
||||
const INTRA_REF_RE =
|
||||
/(?:skills|hooks|commands|agents)\/[A-Za-z0-9._/-]+\.(?:md|sh|cmd|mjs|cjs|js|ts|py|rb|txt|json)/g;
|
||||
|
||||
/**
|
||||
* Intra-plugin file references that don't resolve — the partial-vendor / broken-
|
||||
* path class (e.g. obra/superpowers' `SessionStart` reads
|
||||
* `skills/using-superpowers/SKILL.md`, which a sliced vendor snapshot omits). We
|
||||
* scan the plugin's own text files under the surface dirs (hooks scripts
|
||||
* included — those aren't materialized into `files`) for root-relative path refs
|
||||
* and report the ones missing on disk. A static check that would have caught a
|
||||
* bug the dogfood hit twice. Best-effort: a warning, not an error.
|
||||
*/
|
||||
function danglingRefs(root: string): string[] {
|
||||
const missing = new Set<string>();
|
||||
const seen = new Set<string>();
|
||||
for (const surface of ["hooks", "skills", "agents", "commands"]) {
|
||||
const dir = join(root, surface);
|
||||
if (!existsSync(dir) || !statSync(dir).isDirectory()) continue;
|
||||
for (const content of Object.values(readTree(dir, root))) {
|
||||
for (const m of content.matchAll(INTRA_REF_RE)) {
|
||||
const ref = m[0];
|
||||
if (seen.has(ref)) continue;
|
||||
seen.add(ref);
|
||||
if (!existsSync(join(root, ref))) missing.add(ref);
|
||||
}
|
||||
}
|
||||
}
|
||||
return [...missing];
|
||||
}
|
||||
|
||||
type HooksObj = { hooks?: Record<string, unknown[]> };
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@
|
||||
* evolution engine.
|
||||
*/
|
||||
|
||||
import { describe, it } from "node:test";
|
||||
import { describe, it } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
/**
|
||||
* Tests for the railway-oriented subagent surface: the result() contract on an
|
||||
* agent (compiled into a vigiles:ok/err output section) and railway()/delegate()
|
||||
* composition over flat workers (compiled to an orchestrator command, with
|
||||
* compile-time verification of delegate targets + bounded recovery). Model-free.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { agent, result, railway, delegate } from "./spec.js";
|
||||
import { compileAgent, compileRailway, validateRailway } from "./compile.js";
|
||||
|
||||
// --- result() contract on an agent -----------------------------------------
|
||||
|
||||
test("compileAgent renders the result contract as an Output contract section", () => {
|
||||
const { markdown, errors } = compileAgent(
|
||||
agent({
|
||||
name: "coder",
|
||||
description: "Write code.",
|
||||
tools: ["Read", "Edit", "Bash"],
|
||||
body: "You write code.",
|
||||
output: result(
|
||||
{ files: "string[]", summary: "string" },
|
||||
{ reason: "string", retryable: "boolean" },
|
||||
),
|
||||
}),
|
||||
{ specFile: "agents/coder.md.spec.ts" },
|
||||
);
|
||||
assert.deepEqual(errors, []);
|
||||
assert.match(markdown, /## Output contract/);
|
||||
assert.match(markdown, /```vigiles:ok/);
|
||||
assert.match(markdown, /"files": string\[\], "summary": string/);
|
||||
assert.match(markdown, /```vigiles:err/);
|
||||
assert.match(markdown, /"reason": string, "retryable": boolean/);
|
||||
});
|
||||
|
||||
test("an agent without a result contract has no Output contract section", () => {
|
||||
const { markdown } = compileAgent(
|
||||
agent({ name: "a", description: "d", body: "b" }),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.doesNotMatch(markdown, /## Output contract/);
|
||||
});
|
||||
|
||||
test("an empty contract track renders as {}", () => {
|
||||
const { markdown } = compileAgent(
|
||||
agent({
|
||||
name: "a",
|
||||
description: "d",
|
||||
body: "b",
|
||||
output: result({}, { reason: "string" }),
|
||||
}),
|
||||
{ specFile: "a.md.spec.ts" },
|
||||
);
|
||||
assert.match(markdown, /```vigiles:ok\n\{\}\n```/);
|
||||
});
|
||||
|
||||
// --- delegate() / railway() builders ---------------------------------------
|
||||
|
||||
test("delegate() carries an optional task hint", () => {
|
||||
assert.deepEqual(delegate("planner"), {
|
||||
_step: "delegate",
|
||||
agent: "planner",
|
||||
});
|
||||
assert.deepEqual(delegate("coder", "write it"), {
|
||||
_step: "delegate",
|
||||
agent: "coder",
|
||||
task: "write it",
|
||||
});
|
||||
});
|
||||
|
||||
test("railway() sets the spec type and fields", () => {
|
||||
const rw = railway({
|
||||
name: "ship",
|
||||
steps: [delegate("planner"), delegate("coder")],
|
||||
onError: delegate("reporter"),
|
||||
recover: { step: delegate("fixer"), max: 2 },
|
||||
});
|
||||
assert.equal(rw._specType, "railway");
|
||||
assert.equal(rw.steps.length, 2);
|
||||
});
|
||||
|
||||
// --- compileRailway --------------------------------------------------------
|
||||
|
||||
test("compileRailway renders an orchestrator command with hash + tracks", () => {
|
||||
const { markdown, errors } = compileRailway(
|
||||
railway({
|
||||
name: "ship",
|
||||
steps: [delegate("planner", "draft a plan"), delegate("coder")],
|
||||
onError: delegate("reporter"),
|
||||
recover: { step: delegate("fixer"), max: 2 },
|
||||
}),
|
||||
{ knownAgents: ["planner", "coder", "reporter", "fixer"] },
|
||||
);
|
||||
assert.deepEqual(errors, []);
|
||||
assert.match(markdown, /^<!-- vigiles:sha256:[a-f0-9]+ compiled from/);
|
||||
assert.match(markdown, /# Railway: ship/);
|
||||
assert.match(markdown, /## Success track/);
|
||||
assert.match(markdown, /1\. \*\*planner\*\* — draft a plan/);
|
||||
assert.match(markdown, /2\. \*\*coder\*\*/);
|
||||
assert.match(markdown, /## Recovery[\s\S]*\*\*fixer\*\* up to 2×/);
|
||||
assert.match(markdown, /## On error[\s\S]*\*\*reporter\*\*/);
|
||||
});
|
||||
|
||||
test("compileRailway omits Recovery / On error when not declared", () => {
|
||||
const { markdown } = compileRailway(
|
||||
railway({ name: "min", steps: [delegate("solo")] }),
|
||||
{ knownAgents: ["solo"] },
|
||||
);
|
||||
assert.doesNotMatch(markdown, /## Recovery/);
|
||||
assert.doesNotMatch(markdown, /## On error/);
|
||||
});
|
||||
|
||||
// --- validateRailway (the static, sub-Turing guarantees) -------------------
|
||||
|
||||
test("flags a delegate to an unknown agent (stale-ref)", () => {
|
||||
const errs = validateRailway(
|
||||
railway({ name: "ship", steps: [delegate("planr")] }), // typo
|
||||
["planner", "coder"],
|
||||
);
|
||||
assert.equal(errs.length, 1);
|
||||
assert.equal(errs[0].type, "stale-ref");
|
||||
assert.match(errs[0].message, /unknown agent "planr"/);
|
||||
});
|
||||
|
||||
test("checks the onError and recover targets too", () => {
|
||||
const errs = validateRailway(
|
||||
railway({
|
||||
name: "ship",
|
||||
steps: [delegate("planner")],
|
||||
onError: delegate("ghost"),
|
||||
recover: { step: delegate("phantom"), max: 1 },
|
||||
}),
|
||||
["planner"],
|
||||
);
|
||||
assert.equal(errs.length, 2);
|
||||
assert.ok(errs.every((e) => e.type === "stale-ref"));
|
||||
});
|
||||
|
||||
test("flags an empty railway", () => {
|
||||
const errs = validateRailway(railway({ name: "empty", steps: [] }));
|
||||
assert.ok(
|
||||
errs.some(
|
||||
(e) => e.type === "invalid-railway" && /no steps/.test(e.message),
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
test("flags unbounded/zero recovery (must be ≥ 1 — the finite guarantee)", () => {
|
||||
const errs = validateRailway(
|
||||
railway({
|
||||
name: "ship",
|
||||
steps: [delegate("planner")],
|
||||
recover: { step: delegate("fixer"), max: 0 },
|
||||
}),
|
||||
["planner", "fixer"],
|
||||
);
|
||||
assert.ok(
|
||||
errs.some(
|
||||
(e) => e.type === "invalid-railway" && /max must be ≥ 1/.test(e.message),
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
test("skips agent resolution when knownAgents is omitted", () => {
|
||||
// no registry → don't flag delegate targets (mirrors linter verify modes)
|
||||
const errs = validateRailway(
|
||||
railway({ name: "ship", steps: [delegate("anything")] }),
|
||||
);
|
||||
assert.deepEqual(errs, []);
|
||||
});
|
||||
|
||||
test("compileRailway defaults the spec filename from the railway name", () => {
|
||||
const { markdown } = compileRailway(
|
||||
railway({ name: "ship", steps: [delegate("a")] }),
|
||||
);
|
||||
assert.match(markdown, /compiled from ship\.railway\.spec\.ts/);
|
||||
});
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
* (fenced blocks skipped), the `vigiles:symbol path#symbol` mark, verifying the
|
||||
* named file defines the symbol, and the unmarked-code enforcement.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
* decision logic and real (tiny shell) hooks across exit codes / JSON output /
|
||||
* stdin passthrough / env injection.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
@@ -21,6 +21,8 @@ test("parseHookOutput parses a JSON decision and ignores plain text", () => {
|
||||
});
|
||||
assert.equal(parseHookOutput("just a log line"), null);
|
||||
assert.equal(parseHookOutput(" not json {x"), null);
|
||||
// starts with "{" but invalid JSON → JSON.parse throws → caught → null
|
||||
assert.equal(parseHookOutput('{"unterminated": '), null);
|
||||
});
|
||||
|
||||
test("decideHook: exit 2 blocks regardless of stdout", () => {
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
* Discovery and formatting are pure-ish; `runScripts` spawns trivial node
|
||||
* scripts in a temp dir, so the whole suite stays fast and model-free.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
/**
|
||||
* Tests for the safe-by-default confinement layer (src/sandbox.ts). The policy
|
||||
* (`decideSandbox`), trust test (`specTrusted`), bwrap argv (`bwrapArgs`), and
|
||||
* request-log parser are pure — exercised here with no bwrap. The end-to-end
|
||||
* confinement test (a sandboxed run can't reach the network) is gated on a real
|
||||
* bwrap + claude and skips otherwise, the same pattern as the claude-backed suite.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
decideSandbox,
|
||||
specTrusted,
|
||||
sandboxAvailable,
|
||||
bwrapArgs,
|
||||
parseRequestLog,
|
||||
} from "./sandbox.js";
|
||||
import {
|
||||
runHarnessTest,
|
||||
claudeAvailable,
|
||||
scriptModel,
|
||||
} from "./harness-test.js";
|
||||
import { assertRequestContains, assertHookFired } from "./harness-assert.js";
|
||||
|
||||
test("specTrusted: inline-only is trusted, any external plugin is not", () => {
|
||||
assert.equal(specTrusted({}), true);
|
||||
assert.equal(specTrusted({ plugin: "./repo" }), false);
|
||||
assert.equal(specTrusted({ pluginDir: "/some/plugin" }), false);
|
||||
});
|
||||
|
||||
test("decideSandbox: false is the dangerous opt-out — always runs direct", () => {
|
||||
for (const trusted of [true, false]) {
|
||||
for (const available of [true, false]) {
|
||||
assert.deepEqual(decideSandbox({ trusted, mode: false, available }), {
|
||||
action: "direct",
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test("decideSandbox: strict forces confinement, throws without a sandbox", () => {
|
||||
assert.deepEqual(
|
||||
decideSandbox({ trusted: true, mode: "strict", available: true }),
|
||||
{ action: "sandbox" },
|
||||
);
|
||||
const refused = decideSandbox({
|
||||
trusted: true,
|
||||
mode: "strict",
|
||||
available: false,
|
||||
});
|
||||
assert.equal(refused.action, "throw");
|
||||
assert.match(refused.action === "throw" ? refused.reason : "", /bwrap/);
|
||||
});
|
||||
|
||||
test("decideSandbox auto: trusted runs direct regardless of availability", () => {
|
||||
assert.deepEqual(
|
||||
decideSandbox({ trusted: true, mode: "auto", available: false }),
|
||||
{ action: "direct" },
|
||||
);
|
||||
assert.deepEqual(
|
||||
decideSandbox({ trusted: true, mode: "auto", available: true }),
|
||||
{ action: "direct" },
|
||||
);
|
||||
});
|
||||
|
||||
test("decideSandbox auto: untrusted is sandboxed, or REFUSES if it can't be", () => {
|
||||
assert.deepEqual(
|
||||
decideSandbox({ trusted: false, mode: "auto", available: true }),
|
||||
{ action: "sandbox" },
|
||||
);
|
||||
const refused = decideSandbox({
|
||||
trusted: false,
|
||||
mode: "auto",
|
||||
available: false,
|
||||
});
|
||||
assert.equal(refused.action, "throw");
|
||||
// safe-by-default: never silently runs untrusted code unconfined
|
||||
assert.match(
|
||||
refused.action === "throw" ? refused.reason : "",
|
||||
/refusing to execute|sandbox: false/,
|
||||
);
|
||||
});
|
||||
|
||||
test("sandboxAvailable returns a boolean (covers the probe)", () => {
|
||||
assert.equal(typeof sandboxAvailable(), "boolean");
|
||||
});
|
||||
|
||||
test("bwrapArgs: isolated net, cleared env, ro root, writable work + io + home", () => {
|
||||
const args = bwrapArgs({
|
||||
cwd: "/work",
|
||||
ioDir: "/io",
|
||||
home: "/io/home",
|
||||
path: "/usr/bin:/bin",
|
||||
});
|
||||
const joined = args.join(" ");
|
||||
assert.ok(args.includes("--unshare-all")); // fresh net namespace, no egress
|
||||
assert.ok(args.includes("--clearenv")); // drop host secrets from env
|
||||
assert.ok(joined.includes("--ro-bind / /")); // system read-only
|
||||
assert.ok(joined.includes("--bind /work /work")); // writable work dir
|
||||
assert.ok(joined.includes("--bind /io /io")); // writable IO relay dir
|
||||
assert.ok(joined.includes("--setenv HOME /io/home")); // fresh empty HOME
|
||||
assert.ok(joined.includes("--setenv PATH /usr/bin:/bin")); // PATH set back
|
||||
assert.ok(joined.includes("--chdir /work"));
|
||||
assert.ok(args.includes("--die-with-parent"));
|
||||
});
|
||||
|
||||
test("parseRequestLog: parses ndjson, skips blank and partial lines", () => {
|
||||
const ndjson =
|
||||
JSON.stringify({ system: "s1", messages: [{ role: "user", text: "a" }] }) +
|
||||
"\n\n" +
|
||||
JSON.stringify({ system: "s2", messages: [] }) +
|
||||
"\n" +
|
||||
'{"system":"partial' + // a half-written final line → skipped
|
||||
"\n";
|
||||
const reqs = parseRequestLog(ndjson);
|
||||
assert.equal(reqs.length, 2);
|
||||
assert.equal(reqs[0]?.system, "s1");
|
||||
assert.equal(reqs[1]?.system, "s2");
|
||||
assert.deepEqual(parseRequestLog(""), []);
|
||||
});
|
||||
|
||||
// --- end-to-end confinement (needs a real bwrap + claude) ------------------
|
||||
|
||||
const sandboxRunnable = sandboxAvailable() && claudeAvailable();
|
||||
|
||||
// The security property, proven through the real stack: a sandboxed run's Bash
|
||||
// can reach the in-sandbox mock (so turns are served + requests captured) but
|
||||
// CANNOT reach the external network — egress is blocked by the netns.
|
||||
test.skipIf(!sandboxRunnable)(
|
||||
"a sandboxed run blocks network egress while the mock stays reachable",
|
||||
async () => {
|
||||
const probe =
|
||||
"node -e \"fetch('https://example.com',{signal:AbortSignal.timeout(4000)})" +
|
||||
".then(r=>require('fs').writeFileSync('NET','open:'+r.status))" +
|
||||
".catch(()=>require('fs').writeFileSync('NET','blocked'))\"";
|
||||
const r = await runHarnessTest({
|
||||
sandbox: "strict", // force the sandbox path on a trusted inline spec
|
||||
allowedTools: ["Bash", "Write"],
|
||||
model: scriptModel([
|
||||
{ tool: "Bash", input: { command: probe } },
|
||||
{ text: "done" },
|
||||
]),
|
||||
timeoutMs: 120000,
|
||||
});
|
||||
try {
|
||||
// mock was reachable inside the netns → turns served + requests relayed out
|
||||
assert.ok(r.turns >= 1, "expected the in-sandbox mock to serve a turn");
|
||||
assert.ok(
|
||||
r.modelRequests.length >= 1,
|
||||
"expected captured requests relayed out of the sandbox",
|
||||
);
|
||||
// the real payoff: external egress was blocked
|
||||
assert.equal(
|
||||
r.file("NET"),
|
||||
"blocked",
|
||||
"expected external network to be unreachable from the sandbox",
|
||||
);
|
||||
} finally {
|
||||
r.cleanup();
|
||||
}
|
||||
},
|
||||
130000,
|
||||
);
|
||||
|
||||
// trace.modelRequests proves injected context actually REACHES the model — a
|
||||
// SessionStart hook (emitting Claude Code's nested form) under the sandbox, and
|
||||
// we find its additionalContext in the model's request. "fired" AND "landed".
|
||||
test.skipIf(!sandboxRunnable)(
|
||||
"a SessionStart hook's injected context reaches the model (trace.modelRequests)",
|
||||
async () => {
|
||||
const marker = "VIGILES_CTX_MARKER_42";
|
||||
const hookCmd = `node -e "console.log(JSON.stringify({hookSpecificOutput:{hookEventName:'SessionStart',additionalContext:'${marker}'}}))"`;
|
||||
const r = await runHarnessTest({
|
||||
settings: {
|
||||
hooks: {
|
||||
SessionStart: [
|
||||
{
|
||||
matcher: "startup",
|
||||
hooks: [{ type: "command", command: hookCmd }],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
sandbox: "strict", // force the sandbox path even though this is trusted
|
||||
transcript: true,
|
||||
model: scriptModel([{ text: "ok" }]),
|
||||
timeoutMs: 120000,
|
||||
});
|
||||
try {
|
||||
assertHookFired(r, "SessionStart");
|
||||
assertRequestContains(r, marker); // the injected context landed in the model's request
|
||||
} finally {
|
||||
r.cleanup();
|
||||
}
|
||||
},
|
||||
130000,
|
||||
);
|
||||
|
||||
// Dogfood — the execute-and-verify payoff on a REAL pinned third-party plugin:
|
||||
// obra/superpowers' SessionStart hook is UNTRUSTED, so it runs CONFINED (no
|
||||
// sandbox:false). We assert the real hook FIRED inside the sandbox and produced
|
||||
// its genuine output. (It emits a *top-level* additionalContext, which Claude
|
||||
// Code — reading the *nested* form — does NOT inject; so trace.modelRequests
|
||||
// shows the context did NOT reach the model. That "fired ≠ landed" gap is exactly
|
||||
// what modelRequests exists to surface, proven against real third-party code.)
|
||||
test.skipIf(!sandboxRunnable)(
|
||||
"dogfood: superpowers' SessionStart runs confined and its real output is captured",
|
||||
async () => {
|
||||
const superpowers = join(
|
||||
__dirname,
|
||||
"../examples/harness/vendor/superpowers@6fd4507",
|
||||
);
|
||||
const r = await runHarnessTest({
|
||||
plugin: superpowers, // external → untrusted → confined (no sandbox:false)
|
||||
transcript: true,
|
||||
model: scriptModel([{ text: "ok" }]),
|
||||
timeoutMs: 120000,
|
||||
});
|
||||
try {
|
||||
assertHookFired(r, "SessionStart"); // the real third-party hook ran, confined
|
||||
const ctx = r.hooks.find((h) => h.event === "SessionStart")?.output ?? "";
|
||||
assert.ok(
|
||||
ctx.includes("You have superpowers"),
|
||||
"expected superpowers' SessionStart to produce its real injected-context output",
|
||||
);
|
||||
} finally {
|
||||
r.cleanup();
|
||||
}
|
||||
},
|
||||
130000,
|
||||
);
|
||||
+286
@@ -0,0 +1,286 @@
|
||||
/**
|
||||
* vigiles — safe-by-default confinement for executing untrusted harness code.
|
||||
*
|
||||
* `runHarnessTest` runs the real `claude` CLI, which runs the real hooks of
|
||||
* whatever plugin you load. For code YOU authored (inline `settings`/`files`)
|
||||
* that's fine — trust is implicit. But pointing it at someone else's `plugin` /
|
||||
* `pluginDir` executes THEIR hooks with your privileges. This module makes that
|
||||
* safe by default: untrusted code is confined under bubblewrap, or — if no
|
||||
* sandbox is available — the run refuses rather than executing unconfined.
|
||||
*
|
||||
* Confinement (proven on bwrap 0.9): `--unshare-all` gives a fresh network
|
||||
* namespace whose loopback is auto-up but has NO external route — so the
|
||||
* scripted mock, co-launched INSIDE the namespace, is reachable over 127.0.0.1
|
||||
* while a malicious hook cannot phone home. The filesystem is `--ro-bind`
|
||||
* read-only except the throwaway work dir, a fresh empty `$HOME`, and an IO dir
|
||||
* used to hand the script in and stream captured requests back out.
|
||||
*
|
||||
* The policy (`decideSandbox`), trust test (`specTrusted`), and bwrap argv
|
||||
* (`bwrapArgs`) are pure and unit-tested; the executor (`runSandboxed`) needs a
|
||||
* real bwrap and is covered by the integration test, which skips where bwrap is
|
||||
* absent — the same pattern as the real-`claude` paths.
|
||||
*/
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import {
|
||||
mkdtempSync,
|
||||
mkdirSync,
|
||||
writeFileSync,
|
||||
readFileSync,
|
||||
existsSync,
|
||||
rmSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { type ModelTurn, type ModelRequest } from "./mock-model.js";
|
||||
|
||||
/**
|
||||
* How to treat code execution. `"auto"` (default) is safe-by-default: trusted
|
||||
* code runs directly, untrusted code is sandboxed if possible and otherwise
|
||||
* refuses. `false` is the dangerous opt-out — run unconfined (you audited it, or
|
||||
* you trust the outer container). `"strict"` forces confinement even for trusted
|
||||
* code and throws if no sandbox is available.
|
||||
*/
|
||||
export type SandboxMode = "auto" | "strict" | false;
|
||||
|
||||
/**
|
||||
* Whether bubblewrap is available to confine untrusted code. **Linux only** —
|
||||
* bubblewrap is a Linux tool, so this is always `false` on macOS / Windows,
|
||||
* where confined execution isn't supported and untrusted code must instead be
|
||||
* run via `sandbox: false` (trusting the outer container) or skipped.
|
||||
*/
|
||||
export function sandboxAvailable(): boolean {
|
||||
/* v8 ignore next -- non-Linux has no bwrap; CI/coverage runs on Linux */
|
||||
if (process.platform !== "linux") return false;
|
||||
try {
|
||||
return spawnSync("bwrap", ["--version"], { stdio: "ignore" }).status === 0;
|
||||
} catch {
|
||||
/* v8 ignore next -- defensive: spawnSync only throws on a fork failure */
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Is this spec's executed code trusted? Inline `settings`/`files` you authored
|
||||
* are trusted; any external `plugin` / `pluginDir` brings in third-party hooks
|
||||
* and is NOT — committing it to your repo is the same trust decision as a
|
||||
* dependency, so the trust boundary follows provenance: foreign = confined.
|
||||
*/
|
||||
export function specTrusted(spec: {
|
||||
plugin?: string;
|
||||
pluginDir?: string;
|
||||
}): boolean {
|
||||
return spec.plugin === undefined && spec.pluginDir === undefined;
|
||||
}
|
||||
|
||||
/** The chosen action for a run: execute directly, confine it, or refuse. */
|
||||
export type SandboxDecision =
|
||||
| { readonly action: "direct" }
|
||||
| { readonly action: "sandbox" }
|
||||
| { readonly action: "throw"; readonly reason: string };
|
||||
|
||||
/**
|
||||
* The pure safe-by-default policy. Untrusted code NEVER runs unconfined unless
|
||||
* the caller explicitly opted out (`mode: false`). This is the whole security
|
||||
* contract, isolated as a pure function so it is exhaustively unit-tested.
|
||||
*/
|
||||
export function decideSandbox(opts: {
|
||||
trusted: boolean;
|
||||
mode: SandboxMode;
|
||||
available: boolean;
|
||||
}): SandboxDecision {
|
||||
// Explicit dangerous opt-out: run unconfined, trusted or not.
|
||||
if (opts.mode === false) return { action: "direct" };
|
||||
// Force confinement regardless of trust; refuse if we can't.
|
||||
if (opts.mode === "strict") {
|
||||
return opts.available
|
||||
? { action: "sandbox" }
|
||||
: {
|
||||
action: "throw",
|
||||
reason:
|
||||
"sandbox: 'strict' requires Linux + bubblewrap (bwrap), which was not available",
|
||||
};
|
||||
}
|
||||
// auto: trusted code runs directly; untrusted must be confined or refused.
|
||||
if (opts.trusted) return { action: "direct" };
|
||||
return opts.available
|
||||
? { action: "sandbox" }
|
||||
: {
|
||||
action: "throw",
|
||||
reason:
|
||||
"refusing to execute an untrusted plugin's hooks without a sandbox: " +
|
||||
"the sandbox needs Linux + bubblewrap (bwrap) — install it to run " +
|
||||
"confined, or pass sandbox: false to run unconfined if you trust this " +
|
||||
"code / the outer container",
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The bubblewrap confinement argv (everything before the command): a fresh
|
||||
* network namespace (`--unshare-all`, loopback-only, no egress), a read-only
|
||||
* system, writable mounts limited to the work dir, the IO dir, and a fresh empty
|
||||
* HOME (inside the IO dir so it needs no mountpoint on the read-only root, and so
|
||||
* no host credentials/config leak in), and a **cleared environment** —
|
||||
* `--clearenv` drops every host variable (API keys, cloud creds) and only PATH /
|
||||
* HOME / TMPDIR are set back, so untrusted code can't even read your secrets.
|
||||
* Pure, so the confinement shape is asserted in a unit test.
|
||||
*/
|
||||
export function bwrapArgs(opts: {
|
||||
cwd: string;
|
||||
ioDir: string;
|
||||
home: string;
|
||||
path: string;
|
||||
}): string[] {
|
||||
return [
|
||||
// New user/net/pid/ipc/uts/cgroup namespaces. The net namespace has only a
|
||||
// loopback route, so the in-sandbox mock is reachable but egress is blocked.
|
||||
"--unshare-all",
|
||||
// Drop ALL inherited env (host secrets); only the essentials are set back.
|
||||
"--clearenv",
|
||||
"--ro-bind",
|
||||
"/",
|
||||
"/",
|
||||
"--dev",
|
||||
"/dev",
|
||||
"--proc",
|
||||
"/proc",
|
||||
// Writable: the work dir and the IO dir (later binds override the ro-bind).
|
||||
"--bind",
|
||||
opts.cwd,
|
||||
opts.cwd,
|
||||
"--bind",
|
||||
opts.ioDir,
|
||||
opts.ioDir,
|
||||
// A fresh empty HOME so no host credentials/config are visible.
|
||||
"--setenv",
|
||||
"HOME",
|
||||
opts.home,
|
||||
"--setenv",
|
||||
"TMPDIR",
|
||||
opts.ioDir,
|
||||
// PATH must be set back explicitly (cleared above) so node/claude resolve.
|
||||
"--setenv",
|
||||
"PATH",
|
||||
opts.path,
|
||||
"--chdir",
|
||||
opts.cwd,
|
||||
"--die-with-parent",
|
||||
"--new-session",
|
||||
];
|
||||
}
|
||||
|
||||
/** Parse the in-sandbox mock's ndjson request log into {@link ModelRequest}s. */
|
||||
export function parseRequestLog(ndjson: string): ModelRequest[] {
|
||||
const out: ModelRequest[] = [];
|
||||
for (const line of ndjson.split("\n")) {
|
||||
if (!line.trim()) continue;
|
||||
try {
|
||||
out.push(JSON.parse(line) as ModelRequest);
|
||||
} catch {
|
||||
/* a partially-written final line — skip */
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The raw output of a sandboxed run: exit code, captured stdout, and requests. */
|
||||
export interface SandboxRunOut {
|
||||
readonly code: number;
|
||||
readonly stdout: string;
|
||||
readonly requests: readonly ModelRequest[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Co-launch the scripted mock and `claude` inside ONE bubblewrap network
|
||||
* namespace: the mock serves on the sandbox's loopback (reachable), egress is
|
||||
* blocked, and captured requests stream out through the bound IO dir. Paths come
|
||||
* in via env so the wrapper needs no escaping; `claude`'s args are the wrapper's
|
||||
* positional params (`"$@"`).
|
||||
*/
|
||||
const WRAPPER = [
|
||||
// start the in-sandbox mock; it writes its port to $VIG_PORT when ready
|
||||
'node "$VIG_MOCKENTRY" "$VIG_SCRIPT" "$VIG_REQS" "$VIG_PORT" &',
|
||||
"MOCKPID=$!",
|
||||
"i=0",
|
||||
'while [ ! -s "$VIG_PORT" ] && [ "$i" -lt 200 ]; do sleep 0.05; i=$((i+1)); done',
|
||||
'export ANTHROPIC_BASE_URL="http://127.0.0.1:$(cat "$VIG_PORT")"',
|
||||
"export ANTHROPIC_API_KEY=sk-vigiles-mock",
|
||||
'claude "$@"',
|
||||
"code=$?",
|
||||
'kill "$MOCKPID" 2>/dev/null',
|
||||
'exit "$code"',
|
||||
].join("\n");
|
||||
|
||||
/* v8 ignore start -- spawns bwrap + the real claude CLI; exercised by the
|
||||
bwrap-backed integration test (skipped without bwrap), not the unit gate —
|
||||
the pure policy/args/parse helpers above carry the testable logic. */
|
||||
export function runSandboxed(opts: {
|
||||
cwd: string;
|
||||
claudeArgs: readonly string[];
|
||||
script: readonly ModelTurn[];
|
||||
timeoutMs: number;
|
||||
}): Promise<SandboxRunOut> {
|
||||
const ioDir = mkdtempSync(join(tmpdir(), "vigiles-sbx-"));
|
||||
const home = join(ioDir, "home");
|
||||
mkdirSync(home);
|
||||
const scriptF = join(ioDir, "script.json");
|
||||
const reqsF = join(ioDir, "requests.ndjson");
|
||||
const portF = join(ioDir, "port");
|
||||
writeFileSync(scriptF, JSON.stringify(opts.script));
|
||||
writeFileSync(reqsF, "");
|
||||
// The mock entry is only runnable as built JS. In production __dirname is
|
||||
// dist/ (sibling); under vitest the source runs from src/, so fall back to
|
||||
// the built dist/ copy.
|
||||
const mockEntry =
|
||||
[
|
||||
join(__dirname, "mock-entry.js"),
|
||||
join(__dirname, "..", "dist", "mock-entry.js"),
|
||||
].find((p) => existsSync(p)) ?? join(__dirname, "mock-entry.js");
|
||||
const args = [
|
||||
...bwrapArgs({
|
||||
cwd: opts.cwd,
|
||||
ioDir,
|
||||
home,
|
||||
path: process.env.PATH ?? "/usr/local/bin:/usr/bin:/bin",
|
||||
}),
|
||||
"--setenv",
|
||||
"VIG_MOCKENTRY",
|
||||
mockEntry,
|
||||
"--setenv",
|
||||
"VIG_SCRIPT",
|
||||
scriptF,
|
||||
"--setenv",
|
||||
"VIG_REQS",
|
||||
reqsF,
|
||||
"--setenv",
|
||||
"VIG_PORT",
|
||||
portF,
|
||||
"sh",
|
||||
"-c",
|
||||
WRAPPER,
|
||||
"sh",
|
||||
...opts.claudeArgs,
|
||||
];
|
||||
return new Promise((resolvePromise) => {
|
||||
const child = spawn("bwrap", args, {
|
||||
cwd: opts.cwd,
|
||||
env: process.env,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let stdout = "";
|
||||
child.stdout.on("data", (d: Buffer) => (stdout += d.toString()));
|
||||
child.stderr.on("data", () => {
|
||||
/* hook diagnostics — not needed for the captured result */
|
||||
});
|
||||
const timer = setTimeout(() => child.kill("SIGKILL"), opts.timeoutMs);
|
||||
child.on("close", (code) => {
|
||||
clearTimeout(timer);
|
||||
const requests = parseRequestLog(
|
||||
existsSync(reqsF) ? readFileSync(reqsF, "utf-8") : "",
|
||||
);
|
||||
rmSync(ioDir, { recursive: true, force: true });
|
||||
resolvePromise({ code: code ?? 0, stdout, requests });
|
||||
});
|
||||
});
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
import { describe, it, before, after } from "node:test";
|
||||
import { describe, it, beforeAll as before, afterAll as after } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
import { describe, it, before, after } from "node:test";
|
||||
import { describe, it, beforeAll as before, afterAll as after } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
* short-circuit — the control flow the declarative steps form can't express.
|
||||
* The model is a scripted mock; gates use `true`/`false` for determinism.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
* steps → vigiles:gate markers, and the result postcondition gate. Gate
|
||||
* references are verified against the project at compile time.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { skill, step, input, cmd, file } from "./spec.js";
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* Tests for the skill runtime: parsing vigiles:gate / vigiles:result markers
|
||||
* out of a compiled SKILL.md and executing the gate ladder with short-circuit.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, writeFileSync, rmSync, existsSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
/**
|
||||
* Tests for the skill-testing wrapper: scripting the model and asserting the
|
||||
* deterministic action/gate sequence with plain node:test assertions.
|
||||
* deterministic action/gate sequence with plain assertions.
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { cmd } from "./spec.js";
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
import { describe, it } from "node:test";
|
||||
import { describe, it } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
|
||||
+167
@@ -490,6 +490,173 @@ export function skill(spec: Omit<SkillSpec, "_specType">): SkillSpec {
|
||||
return { _specType: "skill", ...spec };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Subagent specs
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* A subagent definition (compiles to `agents/<name>.md`). Unlike a skill —
|
||||
* reference material the model reads on activation — a subagent is a *delegated
|
||||
* worker with a contract*: a dispatch `description`, an allowed-`tools` rail, an
|
||||
* optional `model`, a system-prompt `body`, and the `rules` it must follow. That
|
||||
* tool contract + those rules are the "railway" a subagent runs on, and they're
|
||||
* exactly the compile-time-verifiable surface vigiles owns: the body's
|
||||
* `file()`/`cmd()`/`symbol()` marks are checked like any instruction file, and
|
||||
* the tools list is verified against the real tool set.
|
||||
*/
|
||||
export interface AgentSpec {
|
||||
readonly _specType: "agent";
|
||||
/** Subagent name (frontmatter + dispatch handle). */
|
||||
readonly name: string;
|
||||
/** When to dispatch this subagent — the trigger (frontmatter). */
|
||||
readonly description: string;
|
||||
/** Model alias (e.g. "sonnet", "opus", "haiku", "inherit"). Optional. */
|
||||
readonly model?: string;
|
||||
/**
|
||||
* The allowed-tools contract — the rails the worker runs on. Each entry must be
|
||||
* a known built-in tool (Read/Write/Edit/Bash/Grep/Glob/WebSearch/WebFetch/
|
||||
* NotebookEdit/TodoWrite/Task/Skill) or an MCP tool (`mcp__server__tool`).
|
||||
* Omit to inherit all tools. Verified at compile time.
|
||||
*/
|
||||
readonly tools?: readonly string[];
|
||||
/**
|
||||
* The lead/intro prose of the system prompt (the "You are…" opener), before any
|
||||
* sections. Carries verified `file()`/`cmd()`/`symbol()`/`ref()` marks. No
|
||||
* markdown headers — use `sections` for those.
|
||||
*/
|
||||
readonly body?: string | InstructionFragment[];
|
||||
/**
|
||||
* Named `##` sections of the system prompt (e.g. Purpose, Core Principles,
|
||||
* Capabilities) — the shape real subagents actually take. Same verified-ref +
|
||||
* no-nested-`##` rules as a CLAUDE.md spec's sections. Use `body` for the intro
|
||||
* and `sections` for the structured rest.
|
||||
*/
|
||||
readonly sections?: Record<string, string | InstructionFragment[]>;
|
||||
/** Rules the worker must follow — rendered as a `## Rules` section. */
|
||||
readonly rules?: Record<string, Rule>;
|
||||
/**
|
||||
* The typed result contract — what this worker returns on success/error. When
|
||||
* set, compiles to an `## Output contract` section instructing the worker to
|
||||
* end with a `vigiles:ok` / `vigiles:err` block, so its outcome is parseable
|
||||
* and testable (see `result()`, `parseAgentResult`, `assertAgentOk`).
|
||||
*/
|
||||
readonly output?: OutputContract;
|
||||
}
|
||||
|
||||
/**
|
||||
* Define a subagent specification (compiles to `agents/<name>.md`).
|
||||
*
|
||||
* // agents/reviewer.md.spec.ts
|
||||
* export default agent({
|
||||
* name: "reviewer",
|
||||
* description: "Review a diff for correctness. Dispatch PROACTIVELY after edits.",
|
||||
* model: "sonnet",
|
||||
* tools: ["Read", "Grep", "Bash"],
|
||||
* body: instructions`Review the diff. Run ${cmd("npm test")} first.`,
|
||||
* rules: {
|
||||
* "no-floating": enforce("@typescript-eslint/no-floating-promises", "Await promises."),
|
||||
* },
|
||||
* });
|
||||
*/
|
||||
export function agent(spec: Omit<AgentSpec, "_specType">): AgentSpec {
|
||||
return { _specType: "agent", ...spec };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Subagent result contract + railway composition (railway-oriented subagents)
|
||||
//
|
||||
// A subagent is a flat worker, but instead of returning prose it returns a
|
||||
// typed Result — either success or error, with rich detail on BOTH tracks. A
|
||||
// `railway()` then composes flat workers: the success track flows worker→worker,
|
||||
// and the first error short-circuits to an error handler. This is Wlaschin's
|
||||
// railway-oriented programming with a subagent as the step. It is deliberately
|
||||
// sub-Turing — a finite list of steps + a bounded recovery, no loop/iterator
|
||||
// combinator — so termination is readable off the value and every reference is
|
||||
// statically checkable (the thing ultraplan's generated script can't be). See
|
||||
// research/railway-subagents.md and research/subagent-compilation.md.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** The field types a result contract can declare (kept tiny + dependency-free). */
|
||||
export type OutputFieldType = "string" | "number" | "boolean" | "string[]";
|
||||
|
||||
/**
|
||||
* A subagent's typed result contract: the shape it must return on success
|
||||
* (`ok`) and on failure (`err`). Rich on both tracks — an error is structured
|
||||
* detail, not a bare pass/fail bit. Compiles into the worker's system prompt
|
||||
* (the `vigiles:ok` / `vigiles:err` block it must emit) and is the schema the
|
||||
* `parseAgentResult` parser + the `assertAgentOk/Err` test helpers validate.
|
||||
*/
|
||||
export interface OutputContract {
|
||||
readonly _ref: "output";
|
||||
readonly ok: Readonly<Record<string, OutputFieldType>>;
|
||||
readonly err: Readonly<Record<string, OutputFieldType>>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Declare a subagent's success/error result contract.
|
||||
*
|
||||
* result(
|
||||
* { files: "string[]", summary: "string" }, // rich success
|
||||
* { reason: "string", retryable: "boolean" }, // rich error
|
||||
* )
|
||||
*
|
||||
* (Distinct from a skill's `result:` postcondition gate — this types a
|
||||
* subagent's *return value*, the success/error tracks of the railway.)
|
||||
*/
|
||||
export function result(
|
||||
ok: Record<string, OutputFieldType>,
|
||||
err: Record<string, OutputFieldType>,
|
||||
): OutputContract {
|
||||
return { _ref: "output", ok, err };
|
||||
}
|
||||
|
||||
/** One step on a railway: dispatch a flat subagent (the "activity"). */
|
||||
export interface RailwayStep {
|
||||
readonly _step: "delegate";
|
||||
/** The subagent to dispatch — resolved against compiled agent names. */
|
||||
readonly agent: string;
|
||||
/** Optional task hint passed to the worker. */
|
||||
readonly task?: string;
|
||||
}
|
||||
|
||||
/** Build a railway step that dispatches `agent` (optionally with a task hint). */
|
||||
export function delegate(agent: string, task?: string): RailwayStep {
|
||||
return task === undefined
|
||||
? { _step: "delegate", agent }
|
||||
: { _step: "delegate", agent, task };
|
||||
}
|
||||
|
||||
/**
|
||||
* A railway over flat subagents. `steps` run in order on the success track; the
|
||||
* first step that returns an error short-circuits to `onError`. `recover`
|
||||
* optionally retries the failing step a *bounded* number of times before the
|
||||
* error track. There is intentionally no loop combinator — the value is a finite
|
||||
* tree, so it always terminates and is fully verifiable at compile time.
|
||||
*/
|
||||
export interface Railway {
|
||||
readonly _specType: "railway";
|
||||
readonly name: string;
|
||||
readonly steps: readonly RailwayStep[];
|
||||
/** Error track — runs with the failing step's error payload. */
|
||||
readonly onError?: RailwayStep;
|
||||
/** Bounded recovery: retry the failing step up to `max` times (finite). */
|
||||
readonly recover?: { readonly step: RailwayStep; readonly max: number };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compose flat subagents into a railway (compiles to an orchestrator command).
|
||||
*
|
||||
* railway({
|
||||
* name: "ship",
|
||||
* steps: [delegate("planner"), delegate("coder"), delegate("reviewer")],
|
||||
* onError: delegate("reporter"),
|
||||
* recover: { step: delegate("fixer"), max: 2 },
|
||||
* })
|
||||
*/
|
||||
export function railway(spec: Omit<Railway, "_specType">): Railway {
|
||||
return { _specType: "railway", ...spec };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Spec file naming convention (#11)
|
||||
//
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
/**
|
||||
* Tests for the significance stats (src/stats.ts) — pure, model-free. The
|
||||
* numerics are checked against known closed forms and t-table values so the
|
||||
* p-values are grounded, not just internally consistent.
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
regularizedIncompleteBeta,
|
||||
tPValueTwoSided,
|
||||
welchTTest,
|
||||
compareArms,
|
||||
} from "./stats.js";
|
||||
import type { EvalReport, MetricStat } from "./eval.js";
|
||||
|
||||
const close = (a: number, b: number, tol = 1e-3): boolean =>
|
||||
Math.abs(a - b) < tol;
|
||||
|
||||
test("regularizedIncompleteBeta: endpoints and a known closed form", () => {
|
||||
assert.equal(regularizedIncompleteBeta(2, 3, 0), 0);
|
||||
assert.equal(regularizedIncompleteBeta(2, 3, 1), 1);
|
||||
// I_0.5(0.5, 0.5) = (2/π)·arcsin(√0.5) = 0.5 (the arcsin distribution)
|
||||
assert.ok(close(regularizedIncompleteBeta(0.5, 0.5, 0.5), 0.5));
|
||||
// both code branches: x below and above (a+1)/(a+b+2)
|
||||
assert.ok(close(regularizedIncompleteBeta(0.5, 0.5, 0.25), 1 / 3, 5e-3)); // arcsin: 2/π·asin(0.5)=1/3
|
||||
assert.ok(close(regularizedIncompleteBeta(0.5, 0.5, 0.75), 2 / 3, 5e-3));
|
||||
});
|
||||
|
||||
test("tPValueTwoSided matches t-table critical values (p≈0.05)", () => {
|
||||
assert.equal(tPValueTwoSided(0, 10), 1); // no difference
|
||||
assert.ok(close(tPValueTwoSided(2.131, 15), 0.05, 5e-3)); // t_.975,15 = 2.131
|
||||
assert.ok(close(tPValueTwoSided(2.776, 4), 0.05, 5e-3)); // t_.975,4 = 2.776
|
||||
assert.ok(close(tPValueTwoSided(2.228, 10), 0.05, 5e-3)); // t_.975,10 = 2.228
|
||||
assert.ok(tPValueTwoSided(8, 30) < 1e-6); // huge t → ~0
|
||||
assert.equal(tPValueTwoSided(2, 0), 1); // df ≤ 0 guard
|
||||
});
|
||||
|
||||
test("welchTTest: clearly-significant vs clearly-noise gaps", () => {
|
||||
// big separation, tight se → significant
|
||||
const sig = welchTTest(
|
||||
{ mean: 0.6, se: 0.05, n: 20 },
|
||||
{ mean: 0.1, se: 0.05, n: 20 },
|
||||
);
|
||||
assert.ok(sig.delta > 0 && sig.significant && sig.pValue < 0.01);
|
||||
|
||||
// small gap, wide se → not significant
|
||||
const noise = welchTTest(
|
||||
{ mean: 0.5, se: 0.2, n: 5 },
|
||||
{ mean: 0.4, se: 0.2, n: 5 },
|
||||
);
|
||||
assert.ok(!noise.significant && noise.pValue > 0.1);
|
||||
});
|
||||
|
||||
test("welchTTest: deterministic arms (se = 0) decide by exact difference", () => {
|
||||
// perfect separation, no variance → significant (p = 0)
|
||||
const sep = welchTTest({ mean: 1, se: 0, n: 5 }, { mean: 0, se: 0, n: 5 });
|
||||
assert.ok(sep.significant && sep.pValue === 0 && sep.df === 0);
|
||||
// identical deterministic arms → not significant (p = 1)
|
||||
const same = welchTTest({ mean: 1, se: 0, n: 5 }, { mean: 1, se: 0, n: 5 });
|
||||
assert.ok(!same.significant && same.pValue === 1);
|
||||
});
|
||||
|
||||
test("welchTTest: one deterministic arm, one varying (mixed df terms)", () => {
|
||||
const c = welchTTest(
|
||||
{ mean: 0.9, se: 0.1, n: 10 }, // varying arm contributes to df
|
||||
{ mean: 0.2, se: 0, n: 10 }, // deterministic baseline contributes 0
|
||||
);
|
||||
assert.ok(c.delta > 0 && c.significant && c.df > 0);
|
||||
});
|
||||
|
||||
function makeReport(
|
||||
stats: Record<string, Record<string, MetricStat>>,
|
||||
): EvalReport {
|
||||
const arms: EvalReport["arms"] = {};
|
||||
for (const [name, s] of Object.entries(stats)) {
|
||||
arms[name] = {
|
||||
runs: 0,
|
||||
metrics: {},
|
||||
stats: s,
|
||||
usage: {
|
||||
totalCostUsd: 0,
|
||||
meanCostUsd: 0,
|
||||
meanDurationMs: 0,
|
||||
totalInputTokens: 0,
|
||||
totalOutputTokens: 0,
|
||||
},
|
||||
};
|
||||
}
|
||||
return { name: "r", trials: 0, totalCostUsd: 0, aborted: false, arms };
|
||||
}
|
||||
|
||||
test("compareArms reads arm stats from a report, or null if absent", () => {
|
||||
const stat = (mean: number, se: number, n: number): MetricStat => ({
|
||||
mean,
|
||||
se,
|
||||
n,
|
||||
std: se * Math.sqrt(n),
|
||||
passK: 0,
|
||||
});
|
||||
const report = makeReport({
|
||||
base: { caught: stat(0.1, 0.05, 20) },
|
||||
arm: { caught: stat(0.6, 0.05, 20) },
|
||||
});
|
||||
const c = compareArms(report, "base", "arm", "caught");
|
||||
assert.ok(c && c.delta > 0 && c.significant);
|
||||
assert.equal(compareArms(report, "base", "arm", "missing"), null); // metric absent
|
||||
assert.equal(compareArms(report, "nope", "arm", "caught"), null); // arm absent
|
||||
});
|
||||
+164
@@ -0,0 +1,164 @@
|
||||
/**
|
||||
* vigiles — significance testing for eval A/B arms.
|
||||
*
|
||||
* The eval tier already reports mean ± se per arm; this answers the question that
|
||||
* `assertImproves(..., { by: se })` punted to the user: is the gap between two
|
||||
* arms real, or noise? A Welch's t-test over the per-arm summary stats (mean, se,
|
||||
* n) — no raw rows needed — yields a two-sided p-value and a significance verdict.
|
||||
* Pure + model-free, so it's fully unit-tested against known t-table values.
|
||||
*
|
||||
* For 0/1 (proportion) metrics this is the t approximation to the two-proportion
|
||||
* test — close at the trial counts evals use, and one code path for any metric.
|
||||
* The numerics (log-gamma, incomplete beta) are specialized to the argument range
|
||||
* these tests produce (a, b ≥ 0.5; x ∈ (0,1)); they are not a general library.
|
||||
*/
|
||||
import type { EvalReport } from "./eval.js";
|
||||
|
||||
// Lanczos coefficients (g = 7) for log-gamma; sufficient for the beta args here.
|
||||
const LANCZOS = [
|
||||
676.5203681218851, -1259.1392167224028, 771.32342877765313,
|
||||
-176.61502916214059, 12.507343278686905, -0.13857109526572012,
|
||||
9.9843695780195716e-6, 1.5056327351493116e-7,
|
||||
];
|
||||
|
||||
/** Log-gamma via Lanczos. Valid for x ≥ 0.5 (all args used below satisfy this). */
|
||||
function lgamma(x: number): number {
|
||||
const g = 7;
|
||||
const xm1 = x - 1;
|
||||
const base = LANCZOS.reduce(
|
||||
(acc, c, i) => acc + c / (xm1 + i + 1),
|
||||
0.99999999999980993,
|
||||
);
|
||||
const tt = xm1 + g + 0.5;
|
||||
return (
|
||||
0.5 * Math.log(2 * Math.PI) +
|
||||
(xm1 + 0.5) * Math.log(tt) -
|
||||
tt +
|
||||
Math.log(base)
|
||||
);
|
||||
}
|
||||
|
||||
/** Continued fraction for the incomplete beta (Numerical Recipes betacf). */
|
||||
function betacf(a: number, b: number, x: number): number {
|
||||
const MAXIT = 200;
|
||||
const EPS = 1e-12;
|
||||
const qab = a + b;
|
||||
const qap = a + 1;
|
||||
const qam = a - 1;
|
||||
let c = 1;
|
||||
let d = 1 / (1 - (qab * x) / qap);
|
||||
let h = d;
|
||||
for (let m = 1; m <= MAXIT; m++) {
|
||||
const m2 = 2 * m;
|
||||
let aa = (m * (b - m) * x) / ((qam + m2) * (a + m2));
|
||||
d = 1 / (1 + aa * d);
|
||||
c = 1 + aa / c;
|
||||
h *= d * c;
|
||||
aa = (-(a + m) * (qab + m) * x) / ((a + m2) * (qap + m2));
|
||||
d = 1 / (1 + aa * d);
|
||||
c = 1 + aa / c;
|
||||
const del = d * c;
|
||||
h *= del;
|
||||
if (Math.abs(del - 1) < EPS) break;
|
||||
}
|
||||
return h;
|
||||
}
|
||||
|
||||
/** Regularized incomplete beta I_x(a, b) ∈ [0, 1]. */
|
||||
export function regularizedIncompleteBeta(
|
||||
a: number,
|
||||
b: number,
|
||||
x: number,
|
||||
): number {
|
||||
if (x <= 0) return 0;
|
||||
if (x >= 1) return 1;
|
||||
const front = Math.exp(
|
||||
lgamma(a + b) -
|
||||
lgamma(a) -
|
||||
lgamma(b) +
|
||||
a * Math.log(x) +
|
||||
b * Math.log(1 - x),
|
||||
);
|
||||
return x < (a + 1) / (a + b + 2)
|
||||
? (front * betacf(a, b, x)) / a
|
||||
: 1 - (front * betacf(b, a, 1 - x)) / b;
|
||||
}
|
||||
|
||||
/** Two-sided p-value for Student's t with `df` degrees of freedom. */
|
||||
export function tPValueTwoSided(t: number, df: number): number {
|
||||
if (df <= 0) return 1;
|
||||
return regularizedIncompleteBeta(df / 2, 0.5, df / (df + t * t));
|
||||
}
|
||||
|
||||
/** The verdict on one arm-vs-baseline comparison for a single metric. */
|
||||
export interface Comparison {
|
||||
/** mean(arm) − mean(baseline). */
|
||||
readonly delta: number;
|
||||
/** Combined standard error of the difference. */
|
||||
readonly seDelta: number;
|
||||
/** Welch t statistic (delta / seDelta). */
|
||||
readonly t: number;
|
||||
/** Welch–Satterthwaite degrees of freedom. */
|
||||
readonly df: number;
|
||||
/** Two-sided p-value for the difference. */
|
||||
readonly pValue: number;
|
||||
/** p < alpha — the difference is unlikely to be noise. */
|
||||
readonly significant: boolean;
|
||||
}
|
||||
|
||||
type Summary = {
|
||||
readonly mean: number;
|
||||
readonly se: number;
|
||||
readonly n: number;
|
||||
};
|
||||
|
||||
// Variance contribution of one arm to the Welch df denominator. Guarded by v > 0
|
||||
// (se > 0 ⇒ n ≥ 2, so n − 1 ≥ 1); a deterministic arm (se = 0) contributes 0.
|
||||
const dfTerm = (v: number, n: number): number =>
|
||||
v > 0 ? (v * v) / (n - 1) : 0;
|
||||
|
||||
/** Welch's unequal-variance t-test between two arms' summary stats. */
|
||||
export function welchTTest(
|
||||
arm: Summary,
|
||||
baseline: Summary,
|
||||
alpha = 0.05,
|
||||
): Comparison {
|
||||
const delta = arm.mean - baseline.mean;
|
||||
const va = arm.se ** 2;
|
||||
const vb = baseline.se ** 2;
|
||||
const seDelta = Math.sqrt(va + vb);
|
||||
if (seDelta === 0) {
|
||||
// Both arms are deterministic: significant iff they differ at all.
|
||||
const significant = delta !== 0;
|
||||
return {
|
||||
delta,
|
||||
seDelta,
|
||||
t: 0,
|
||||
df: 0,
|
||||
pValue: significant ? 0 : 1,
|
||||
significant,
|
||||
};
|
||||
}
|
||||
const t = delta / seDelta;
|
||||
const df = (va + vb) ** 2 / (dfTerm(va, arm.n) + dfTerm(vb, baseline.n));
|
||||
const pValue = tPValueTwoSided(t, df);
|
||||
return { delta, seDelta, t, df, pValue, significant: pValue < alpha };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare two arms on a metric using their reported summary stats, or null if
|
||||
* either arm/metric is absent. The grounded form of `assertImproves`'s `by`: it
|
||||
* computes the noise floor instead of asking the caller to supply it.
|
||||
*/
|
||||
export function compareArms(
|
||||
report: EvalReport,
|
||||
baseline: string,
|
||||
arm: string,
|
||||
metric: string,
|
||||
alpha = 0.05,
|
||||
): Comparison | null {
|
||||
const a = report.arms[arm]?.stats[metric];
|
||||
const b = report.arms[baseline]?.stats[metric];
|
||||
if (!a || !b) return null;
|
||||
return welchTTest(a, b, alpha);
|
||||
}
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
* Tests for the cross-language symbol index (ast-grep): per-file extraction,
|
||||
* the project index, and bare/scoped resolution (unique/ambiguous/missing).
|
||||
*/
|
||||
import { test } from "node:test";
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { describe, it, before, after } from "node:test";
|
||||
import { describe, it, beforeAll as before, afterAll as after } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import {
|
||||
mkdtempSync,
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
/**
|
||||
* Conformance tests over REAL, vendored plugins (examples/harness/vendor/*).
|
||||
*
|
||||
* Model-free and in-gate: loadPlugin parses + materializes each plugin's ACTUAL
|
||||
* shipped layout, and we assert invariants that must hold for any well-formed
|
||||
* plugin — it loads, `${CLAUDE_PLUGIN_ROOT}` resolves, skills materialize, and the
|
||||
* warnings (surface + dangling-ref) are accurate. Grounded in reality rather than
|
||||
* synthetic fixtures: this is the shape that caught the superpowers partial-vendor
|
||||
* dangling ref. Each plugin is pinned by commit SHA, so the suite is deterministic
|
||||
* and offline (no network, no model, no API key).
|
||||
*
|
||||
* Assertions are INVARIANTS, never version trivia — we check "≥ 1 skill loaded"
|
||||
* and "the known dangling ref is flagged, nothing spurious", not "exactly N
|
||||
* skills" (which would break on a harmless re-pin and test nothing real).
|
||||
*/
|
||||
import { test } from "vitest";
|
||||
import assert from "node:assert/strict";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { loadPlugin } from "./plugin-loader.js";
|
||||
|
||||
// __dirname is dist/ at runtime; the vendored plugins live at the repo root.
|
||||
const VENDOR = "../examples/harness/vendor";
|
||||
|
||||
interface PluginCase {
|
||||
readonly label: string;
|
||||
readonly dir: string;
|
||||
readonly expectHooks: boolean;
|
||||
readonly minSkills: number;
|
||||
readonly expectAgents: boolean;
|
||||
readonly expectCommands: boolean;
|
||||
/** Intra-plugin file refs knowingly absent in the snapshot (e.g. a partial vendor). */
|
||||
readonly knownDangling: readonly string[];
|
||||
}
|
||||
|
||||
const PLUGINS: readonly PluginCase[] = [
|
||||
{
|
||||
label: "obra/superpowers",
|
||||
dir: "superpowers@6fd4507",
|
||||
expectHooks: true,
|
||||
minSkills: 1,
|
||||
expectAgents: false,
|
||||
expectCommands: false,
|
||||
// the vendored slice omits skills/using-superpowers/, which SessionStart reads
|
||||
knownDangling: ["skills/using-superpowers/SKILL.md"],
|
||||
},
|
||||
{
|
||||
label: "wshobson/accessibility",
|
||||
dir: "wshobson-accessibility@cf6059d",
|
||||
expectHooks: false,
|
||||
minSkills: 1,
|
||||
expectAgents: true,
|
||||
expectCommands: true,
|
||||
knownDangling: [],
|
||||
},
|
||||
];
|
||||
|
||||
const hasWarning = (ws: readonly string[], sub: string): boolean =>
|
||||
ws.some((w) => w.includes(sub));
|
||||
|
||||
/** How many intra-plugin dangling refs the loader flagged (parsed from the warning). */
|
||||
const danglingCount = (ws: readonly string[]): number => {
|
||||
const w = ws.find((x) => x.includes("intra-plugin"));
|
||||
const m = w?.match(/references (\d+) intra-plugin/);
|
||||
return m?.[1] ? Number(m[1]) : 0;
|
||||
};
|
||||
|
||||
const skillCount = (files: Record<string, string>): number =>
|
||||
Object.keys(files).filter((f) => /skills\/.*SKILL\.md$/.test(f)).length;
|
||||
|
||||
for (const p of PLUGINS) {
|
||||
test(`${p.label}: loadPlugin parses the real shipped layout`, () => {
|
||||
const loaded = loadPlugin(join(__dirname, VENDOR, p.dir));
|
||||
|
||||
// 1. a real surface loaded — never a silent empty machine
|
||||
assert.ok(
|
||||
loaded.settings.hooks || Object.keys(loaded.files).length > 0,
|
||||
"expected hooks or files to load",
|
||||
);
|
||||
assert.ok(
|
||||
!hasWarning(loaded.warnings, "nothing was loaded"),
|
||||
"should not be an empty machine",
|
||||
);
|
||||
|
||||
// 2. hooks presence matches, and ${CLAUDE_PLUGIN_ROOT} fully resolved
|
||||
assert.equal(Boolean(loaded.settings.hooks), p.expectHooks);
|
||||
assert.ok(
|
||||
!JSON.stringify(loaded.settings).includes("CLAUDE_PLUGIN_ROOT"),
|
||||
"no unresolved ${CLAUDE_PLUGIN_ROOT}",
|
||||
);
|
||||
|
||||
// 3. skills materialized into the sandbox
|
||||
assert.ok(
|
||||
skillCount(loaded.files) >= p.minSkills,
|
||||
`expected ≥ ${String(p.minSkills)} skill(s)`,
|
||||
);
|
||||
|
||||
// 4. surface warnings are accurate (agents/commands belong to the eval tier)
|
||||
assert.equal(hasWarning(loaded.warnings, "subagent file"), p.expectAgents);
|
||||
assert.equal(
|
||||
hasWarning(loaded.warnings, "slash-command file"),
|
||||
p.expectCommands,
|
||||
);
|
||||
|
||||
// 5. dangling-ref detector is accurate: exactly the known set, nothing spurious
|
||||
assert.equal(danglingCount(loaded.warnings), p.knownDangling.length);
|
||||
for (const ref of p.knownDangling) {
|
||||
assert.ok(hasWarning(loaded.warnings, ref), `should flag ${ref}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -82,6 +82,15 @@ echo " turns: passing-gate=$TP failing-gate=$TF"
|
||||
if [ "$TP" = "1" ]; then ok "passing result gate → claude stops (1 turn)"; else bad "passing gate should stop in 1 turn (got $TP)"; fi
|
||||
if [ "$TF" -gt "$TP" ] 2>/dev/null; then ok "failing result gate → Stop blocked, claude forced to continue ($TF turns)"; else bad "failing gate should block (turns=$TF)"; fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Note: the subagent PreToolUse tool-contract rail (`vigiles agent-hook`) is a
|
||||
# *tool-event* hook. Driving a tool call deterministically needs the model to
|
||||
# actually invoke the tool, which is flaky against a scripted mock — so the rail
|
||||
# is proven at the cheap, deterministic unit tier instead: a real synthesized
|
||||
# PreToolUse event piped straight to the built CLI hook process. See the
|
||||
# "agent-hook (CLI): the real PreToolUse rail process" tests in
|
||||
# src/agent-runtime.test.ts (runHook against `dist/cli.js agent-hook`).
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "E2E: $PASS passed, $FAIL failed"
|
||||
|
||||
+55
-6
@@ -1,12 +1,61 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
// Scope vitest to the cross-runner integration tests only — the src/*.test.ts
|
||||
// suites are node:test and must not be picked up here.
|
||||
// Vitest is the primary runner. Two projects:
|
||||
// - `unit` — the TypeScript source suites (`src/**/*.test.ts`), run directly
|
||||
// (esbuild), with `.js` import specifiers resolved to their `.ts`
|
||||
// source so NodeNext-style imports work without a build step.
|
||||
// - `runners` — the cross-runner constraint: the same `vigilesMatchers` register
|
||||
// and pass under vitest, loaded from the built `dist` the way a
|
||||
// user would (`npm run test:vitest`). Proves runner-agnosticism.
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ["test/runners/**/*.vitest.mjs"],
|
||||
// Load the opt-in entry the way a user would — this also tests that
|
||||
// `vigiles/vitest` registers the matchers (auto-register).
|
||||
setupFiles: ["./dist/vitest.mjs"],
|
||||
projects: [
|
||||
{
|
||||
test: {
|
||||
name: "unit",
|
||||
include: ["src/**/*.test.ts"],
|
||||
// node:test had no per-test timeout; some suites scan all 7 linter
|
||||
// catalogs or spawn the built CLI and legitimately take several seconds.
|
||||
testTimeout: 60000,
|
||||
hookTimeout: 60000,
|
||||
},
|
||||
resolve: { extensionAlias: { ".js": [".ts", ".js"] } },
|
||||
},
|
||||
{
|
||||
test: {
|
||||
name: "runners",
|
||||
include: ["test/runners/**/*.vitest.mjs"],
|
||||
setupFiles: ["./dist/vitest.mjs"],
|
||||
},
|
||||
},
|
||||
],
|
||||
// 100% gate, scoped to the harness-testing pillar (the deterministic library
|
||||
// this suite owns end-to-end). The eval path's real `spawn` boundary is the
|
||||
// one thing a unit test can't reach — it carries a `v8 ignore` marker.
|
||||
coverage: {
|
||||
provider: "v8",
|
||||
include: [
|
||||
"src/harness-test.ts",
|
||||
"src/harness-assert.ts",
|
||||
"src/eval.ts",
|
||||
"src/run-hook.ts",
|
||||
"src/mock-model.ts",
|
||||
"src/plugin-loader.ts",
|
||||
"src/judge.ts",
|
||||
"src/sandbox.ts",
|
||||
],
|
||||
// 100% lines/functions/statements. Branches floor at 90: the remainder
|
||||
// are defensive fallbacks that can't be hit deterministically — `?? ""` on
|
||||
// already-typed CLI output, `n > 0 ? … : 0` on non-empty arrays, a
|
||||
// signal-kill exit code (`res.status ?? (res.signal ? 1 : 0)`). Gaming
|
||||
// those with ignores would only lower the signal.
|
||||
thresholds: {
|
||||
lines: 100,
|
||||
functions: 100,
|
||||
statements: 100,
|
||||
branches: 90,
|
||||
},
|
||||
reporter: ["text", "lcov"],
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user