mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
9413061d9b
* refactor(exports)!: split the testing surface on COST, and name the cost The exports map split the testing API by TEST TIER — `vigiles/testing`, `vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had a home: the vitest projects, keyed on test-file NAMES. The exports map was a second, competing implementation of it, and zero of 19 comparable packages publish `./unit` / `./integration` / `./e2e` subpaths. It splits on COST now, on two axes at once — the PATH and the NAME. "." -> ./dist/test.js (was ./dist/core/spec.js) "./spec" -> ./dist/core/spec.js unchanged "./eval" -> ./dist/eval-surface.js NEW ./linting ./hook ./claude-code ./codex ./adapter ./experimental ./vitest ./jest unchanged DELETED: ./testing, ./unit, ./integration, ./e2e. 14 subpaths -> 11. No `./test` subpath: the bare package name IS the testing surface. Two names for one door is the duplication being removed (`.` + `./spec` used to be exactly that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with `vigiles eval`, the free half answers to the plain package name. Accepted; `@playwright/test` publishes no subpath naming a test type at all. THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`: paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate, paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top of a file; the name warns EVERY time, at the call site. Reading `await judged(trace, "did it refuse?")` on line 140, the import is long out of view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined path with an `experimental_` prefix; this applies the device to a second axis. TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something, and a type is never called. `paid_EvalReport` would be noise — and since the 29 report types are re-exported from BOTH barrels, prefixing them would put `paid_` names on the FREE surface. ⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN. `paid_judged(rubric, { judge: myFn })` calls your function and spends nothing; the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path bills (`paid_judge` is the one where `paid_` is exact — no injection seam). `metered_` would be precise but reads a beat slower, and a warning not absorbed at a glance is not a warning. Clarity won; the imprecision is written into the module docstring, each symbol's JSDoc, and docs/testing-api.md. ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime` sat unprefixed among six `experimental_` siblings on `./experimental`. One missing prefix is worse than none: the six that comply teach the reader that an unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`. BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the TypeScript compiler (so it can tell a value from a type instead of grepping) and FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check` and a CI step beside `docs:check`. ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts` (12 cases, colocated) fires on planted violations AND is silent on the legitimate surface next door. Four mutations, each failing on its OWN assertion: drop alias resolution -> 2 fail (the vacuous-pass: `./eval` is built entirely from `export { x as paid_x }`, and an alias reports Alias, not Value — a check that skipped resolution would read every export as a type and pass on a fully unprefixed barrel) exit(1) -> exit(0) -> 1 fail (gate degrades to nudge) delete the ./experimental entry from SURFACES -> 1 fail (see below) never push a violation -> 4 fail 🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check. An entire axis could stop being policed with nothing turning red: the axis test passes its own surface list, the "real barrels" test is satisfied by a SHORTER list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the same way (its fixture lacked `./experimental`, so it exited 1 either way) and now runs a clean twin that must exit 0. NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols. New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus decideHook / parseHookOutput) existed only on `unit` and are on the root now. TWO HONEST COSTS, in the docstrings rather than hidden: 1. The type coupling crosses the boundary and cannot be removed. ~27 helpers (assertImproves, compareArms, diffReports, cost/latency/tokens, formatEvalReport, …) read an eval RESULT and spend nothing, so they are free and unprefixed — but their argument types are defined on the paid side. Hence the 29 duplicated types, so nobody imports `./eval` for a type alone. 2. Free is not fast. The old "real CLI, no API key" tier collapses into the root, so the import no longer signals that `runHarnessTest` spawns a real claude under bubblewrap for ~40s. Duration lives in the runner config now. `.` moving off `core/spec` means every doc example importing a spec builder from bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught those. Entry points are enumerated in four places independently of package.json; all updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs (AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation to still fire on the new filenames), vitest.config.mjs comment. Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files) · npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier --check . · npm run api:check (11 entries, no drift) · npm run build · plus npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also `node dist/cli.js lint` exits 0 with every compiled-markdown hash valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd * ci: закрыть класс «сгенерированный файл протух, а CI зелёный» Шаг «Check generated types compile» перегенерирует файл и компилирует результат. Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на 314 строк / ~147 записей, и CI всё это время была зелёной. Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает. Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine 08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift in a generated file is invisible unless something regenerates it with --check. Upstream vigiles tracks its own copy and it IS stale on main for exactly that reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает строка в CI. Проверено обеими половинами: на актуальном файле проходит, на подложенном дрейфе (одна строка в конец) падает. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd --------- Co-authored-by: Claude <noreply@anthropic.com>
144 lines
6.1 KiB
TypeScript
144 lines
6.1 KiB
TypeScript
/**
|
||
* audit → the ONE read-vs-run decision. `audit` is a Lighthouse-style LOCAL
|
||
* report: a deterministic READ by default — safe + identical on every OS, nothing
|
||
* executes — NOT a CI step (CI uses `vigiles lint`, the deterministic gate). The
|
||
* executing checks (safety battery, live MCP resolution, skill-firing trigger-rate)
|
||
* run ONLY when there's a human to consent: at a TTY `audit` ASKS once (and
|
||
* remembers in `.vigilesrc.json` `audit.measure`); headless (an agent / `--json` /
|
||
* `--no-interactive` / a pipe) it stays a read + a one-line nudge — never hangs,
|
||
* never silently executes. There is deliberately NO execution flag: automation
|
||
* tests the harness through the `vigiles` testing API + skills (the layered tiers),
|
||
* not through the report verb. The IO (prompt / run / remember) lives in the CLI;
|
||
* this is the pure decision + helpers.
|
||
*/
|
||
|
||
/** Only the env vars that signal a reachable model (parse, don't validate). */
|
||
export interface ModelEnv {
|
||
readonly ANTHROPIC_API_KEY?: string;
|
||
readonly CLAUDECODE?: string;
|
||
readonly CLAUDE_CODE_ENTRYPOINT?: string;
|
||
}
|
||
|
||
/**
|
||
* Is a real model reachable for the trigger tier? Either a metered API key
|
||
* (`ANTHROPIC_API_KEY`), OR an authenticated Claude Code session (`CLAUDECODE=1`
|
||
* / `CLAUDE_CODE_ENTRYPOINT`, web/desktop/CLI) — the latter drives the `claude`
|
||
* CLI on the user's subscription, no key needed and $0 metered. A tiny env-only
|
||
* predicate (not a live probe), so it never spends a token just to decide.
|
||
*/
|
||
export function hasModelAccess(env: ModelEnv): boolean {
|
||
return Boolean(
|
||
env.ANTHROPIC_API_KEY ||
|
||
env.CLAUDECODE === "1" ||
|
||
env.CLAUDE_CODE_ENTRYPOINT,
|
||
);
|
||
}
|
||
|
||
/**
|
||
* Is the reachable model METERED (a paid API key) rather than a subscription?
|
||
* Only affects the consent DISCLOSURE wording (a metered key bills per token; a
|
||
* subscription is $0 metered) — the run/skip decision itself is consent-driven,
|
||
* not metered-driven.
|
||
*/
|
||
export function isMeteredAccess(env: ModelEnv): boolean {
|
||
return Boolean(env.ANTHROPIC_API_KEY);
|
||
}
|
||
|
||
/** Why the executing checks were skipped (drives the "not run" nudge). */
|
||
export type ExecuteSkipReason =
|
||
| "nothing" // no executable surface at all (no hooks / MCP / skills) — no nudge
|
||
| "headless" // an agent / --json / non-interactive / a pipe — no human to consent
|
||
| "remembered-no"; // a sticky .vigilesrc choice said no
|
||
|
||
/**
|
||
* What `audit` should do about the EXECUTING checks (battery + live MCP +
|
||
* trigger-rate), as ONE bundle:
|
||
* - `run` — run them now (a remembered yes).
|
||
* - `ask` — interactive human + something to run + no sticky choice: ask once,
|
||
* then remember.
|
||
* - `skip` — stay a deterministic read; the `reason` drives a one-line nudge.
|
||
*/
|
||
export type ExecuteDecision =
|
||
| { readonly kind: "run" }
|
||
| { readonly kind: "ask" }
|
||
| { readonly kind: "skip"; readonly reason: ExecuteSkipReason };
|
||
|
||
export interface ExecuteEnv {
|
||
/** Is there ANY executable surface — runnable hooks, an own-repo MCP server, or
|
||
* a model-invocable skill? Nothing to run → never ask, never nudge. */
|
||
readonly hasExecutable: boolean;
|
||
/** Both stdin AND stdout are a terminal (a human who can answer + wait). */
|
||
readonly isTTY: boolean;
|
||
/** `--json` — machine output; stays a read even at a TTY (never prompt). */
|
||
readonly json: boolean;
|
||
/** `--no-interactive` / `--yes` — explicit agent/CI mode (never prompt). */
|
||
readonly noInteractive: boolean;
|
||
/** Sticky remembered choice from `.vigilesrc.json` (`audit.measure`), or undefined. */
|
||
readonly remembered?: boolean;
|
||
}
|
||
|
||
/**
|
||
* Decide what `audit` does with the executing checks. Total + pure; the first
|
||
* matching rule wins. There is NO execution flag — `audit` is a local report, so
|
||
* the executing checks need a human to consent:
|
||
* 1. nothing executable → skip "nothing" (a clean read; no nudge)
|
||
* 2. headless (`--json` / `--no-interactive` / non-TTY — an agent, a pipe, CI) →
|
||
* skip "headless" (no one to ask; automation uses the `vigiles` testing API)
|
||
* 3. sticky no → skip "remembered-no"
|
||
* 4. sticky yes → run
|
||
* 5. interactive human, no sticky choice → ask (then remember)
|
||
*/
|
||
export function decideExecute(o: ExecuteEnv): ExecuteDecision {
|
||
if (!o.hasExecutable) return { kind: "skip", reason: "nothing" };
|
||
if (o.json || o.noInteractive || !o.isTTY)
|
||
return { kind: "skip", reason: "headless" };
|
||
if (o.remembered === false) return { kind: "skip", reason: "remembered-no" };
|
||
if (o.remembered === true) return { kind: "run" };
|
||
return { kind: "ask" };
|
||
}
|
||
|
||
/**
|
||
* The one-line "executing checks not run" nudge for a skipped read (the
|
||
* no-silent-skips corollary). Returns null for `nothing` (nothing to run — not a
|
||
* gap). There is no flag to point at — `audit` runs them only interactively, and
|
||
* automation uses the `vigiles` testing API.
|
||
*/
|
||
export function formatExecuteSkip(reason: ExecuteSkipReason): string | null {
|
||
switch (reason) {
|
||
case "nothing":
|
||
return null;
|
||
case "headless":
|
||
return (
|
||
"\nℹ Executing checks (safety battery · live MCP · skill firing) skipped — " +
|
||
"`audit` runs them only interactively (a terminal). For automation, test the " +
|
||
"harness with the `vigiles` testing API."
|
||
);
|
||
case "remembered-no":
|
||
return (
|
||
"\nℹ Executing checks not run (you disabled them — edit .vigilesrc.json " +
|
||
"`audit.measure` to re-enable)."
|
||
);
|
||
}
|
||
}
|
||
|
||
/**
|
||
* A starter `--prompts` file (the real `TriggerPromptSet` shape: bare skill name
|
||
* → `{ prompts, irrelevant }`). One entry per triggerable skill, with TODO
|
||
* placeholders the user replaces with real requests.
|
||
*/
|
||
export function scaffoldTriggerPrompts(skillNames: readonly string[]): string {
|
||
const obj: Record<string, { prompts: string[]; irrelevant: string[] }> = {};
|
||
for (const name of skillNames) {
|
||
obj[name] = {
|
||
prompts: [
|
||
`TODO: a request that SHOULD trigger "${name}"`,
|
||
`TODO: a differently-phrased request that should also trigger it`,
|
||
],
|
||
irrelevant: [
|
||
`TODO: an unrelated request that should NOT trigger "${name}"`,
|
||
],
|
||
};
|
||
}
|
||
return JSON.stringify(obj, null, 2) + "\n";
|
||
}
|