Files
zernie__vigiles/src/scan-trigger-suggest.ts
zernie 9413061d9b feat!: split the public exports by COST — vigiles/test (free) vs vigiles/eval (spends money) (#149)
* refactor(exports)!: split the testing surface on COST, and name the cost

The exports map split the testing API by TEST TIER — `vigiles/testing`,
`vigiles/unit`, `vigiles/integration`, `vigiles/e2e`. That taxonomy already had
a home: the vitest projects, keyed on test-file NAMES. The exports map was a
second, competing implementation of it, and zero of 19 comparable packages
publish `./unit` / `./integration` / `./e2e` subpaths.

It splits on COST now, on two axes at once — the PATH and the NAME.

  "."             -> ./dist/test.js          (was ./dist/core/spec.js)
  "./spec"        -> ./dist/core/spec.js     unchanged
  "./eval"        -> ./dist/eval-surface.js  NEW
  ./linting ./hook ./claude-code ./codex ./adapter ./experimental
  ./vitest ./jest                            unchanged

DELETED: ./testing, ./unit, ./integration, ./e2e.  14 subpaths -> 11.

No `./test` subpath: the bare package name IS the testing surface. Two names for
one door is the duplication being removed (`.` + `./spec` used to be exactly
that). The cost: CLI-verb symmetry is now one-sided — `vigiles/eval` rhymes with
`vigiles eval`, the free half answers to the plain package name. Accepted;
`@playwright/test` publishes no subpath naming a test type at all.

THE NAME AXIS. Every runtime export on `./eval` is prefixed `paid_`:
paid_runEval, paid_measure, paid_measureArms, paid_measureTriggerRate,
paid_judge, paid_judged, paid_claudeEvalDriver. The path warns ONCE at the top
of a file; the name warns EVERY time, at the call site. Reading
`await judged(trace, "did it refuse?")` on line 140, the import is long out of
view. Not a new idiom here — `vigiles/experimental` already pairs a quarantined
path with an `experimental_` prefix; this applies the device to a second axis.

TYPES ARE NOT PREFIXED, deliberately: the prefix warns about CALLING something,
and a type is never called. `paid_EvalReport` would be noise — and since the 29
report types are re-exported from BOTH barrels, prefixing them would put `paid_`
names on the FREE surface.

⚠️ THE PREFIX OVERSTATES SLIGHTLY, AND THIS IS RECORDED, NOT HIDDEN.
`paid_judged(rubric, { judge: myFn })` calls your function and spends nothing;
the `measure*` family takes an injectable `evalDriver`. Only the DEFAULT path
bills (`paid_judge` is the one where `paid_` is exact — no injection seam).
`metered_` would be precise but reads a beat slower, and a warning not absorbed
at a glance is not a warning. Clarity won; the imprecision is written into the
module docstring, each symbol's JSDoc, and docs/testing-api.md.

ALSO FIXED: a PRE-EXISTING violation of the older convention. `makeDockerRuntime`
sat unprefixed among six `experimental_` siblings on `./experimental`. One
missing prefix is worse than none: the six that comply teach the reader that an
unprefixed name is safe. Renamed at source to `experimental_makeDockerRuntime`.

BOTH CONVENTIONS NOW HAVE A GATE, because that violation is exactly what prose
buys you. `scripts/check-export-prefixes.mjs` reads the built `.d.ts` with the
TypeScript compiler (so it can tell a value from a type instead of grepping) and
FAILS with exit 1 — it is a gate, not a nudge. Wired as `npm run exports:check`
and a CI step beside `docs:check`.

ITS TEST HAS BOTH HALVES, and mutation says so. `scripts/check-export-prefixes.test.ts`
(12 cases, colocated) fires on planted violations AND is silent on the legitimate
surface next door. Four mutations, each failing on its OWN assertion:

  drop alias resolution        -> 2 fail  (the vacuous-pass: `./eval` is built
                                  entirely from `export { x as paid_x }`, and an
                                  alias reports Alias, not Value — a check that
                                  skipped resolution would read every export as a
                                  type and pass on a fully unprefixed barrel)
  exit(1) -> exit(0)           -> 1 fail  (gate degrades to nudge)
  delete the ./experimental
    entry from SURFACES        -> 1 fail  (see below)
  never push a violation       -> 4 fail

🔴 THE THIRD MUTATION PASSED GREEN AT FIRST — a hole in the TEST, not the check.
An entire axis could stop being policed with nothing turning red: the axis test
passes its own surface list, the "real barrels" test is satisfied by a SHORTER
list, and the CLI test still exits 1 on its `./eval` defect. Fixed by asserting
SURFACES covers exactly the quarantined subpaths. The CLI test was vacuous the
same way (its fixture lacked `./experimental`, so it exited 1 either way) and now
runs a clean twin that must exit 0.

NOTHING WAS LOST, AND IT IS MEASURED. Union of the four old barrels: 190 symbols.
New: 183 on the root + 36 on ./eval, overlapping in 29 report types re-exported
from both on purpose. 183 + 36 - 29 = 190; set difference empty in BOTH
directions (comparing `paid_x` to `x` by identity). `vigiles/testing` was NOT a
superset of `vigiles/unit`: 11 symbols (all of guardrail-check.ts, plus
decideHook / parseHookOutput) existed only on `unit` and are on the root now.

TWO HONEST COSTS, in the docstrings rather than hidden:
1. The type coupling crosses the boundary and cannot be removed. ~27 helpers
   (assertImproves, compareArms, diffReports, cost/latency/tokens,
   formatEvalReport, …) read an eval RESULT and spend nothing, so they are free
   and unprefixed — but their argument types are defined on the paid side. Hence
   the 29 duplicated types, so nobody imports `./eval` for a type alone.
2. Free is not fast. The old "real CLI, no API key" tier collapses into the root,
   so the import no longer signals that `runHarnessTest` spawns a real claude
   under bubblewrap for ~40s. Duration lives in the runner config now.

`.` moving off `core/spec` means every doc example importing a spec builder from
bare "vigiles" now says "vigiles/spec" (11 blocks). `docs:check` is what caught
those.

Entry points are enumerated in four places independently of package.json; all
updated: scripts/api-extractor.mjs (1:1 with exports now — `vigiles.api.md` is
the test surface, `vigiles-spec.api.md` is new), typedoc.json, eslint.config.mjs
(AGNOSTIC_SURFACE + the no-barrel-imports list; both rules verified by mutation
to still fire on the new filenames), vitest.config.mjs comment.

Gates, by name and in order: npx vitest run (2939 passed, 25 skipped, 174 files)
· npm run lint (0 errors, 206 warnings — unchanged baseline) · npx prettier
--check . · npm run api:check (11 entries, no drift) · npm run build · plus
npm run docs:check (91 blocks, 0 findings) and npm run exports:check. Also
`node dist/cli.js lint` exits 0 with every compiled-markdown hash valid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd

* ci: закрыть класс «сгенерированный файл протух, а CI зелёный»

Шаг «Check generated types compile» перегенерирует файл и компилирует результат.
Свежесгенерированный компилируется всегда — поэтому закоммиченная копия в проверке
НЕ УЧАСТВУЕТ. Замер 16.08: отслеживаемый `.vigiles/generated.d.ts` отставал на
314 строк / ~147 записей, и CI всё это время была зелёной.

Название шага честное — он проверяет, что типы КОМПИЛИРУЮТСЯ, а не что они
актуальны. Именно поэтому дыра пережила ревью: шаг делает ровно то, что обещает.

Предсказание записано за восемь дней до: в .gitignore репозитория zernie/mine
08.08 этим обосновано, почему тот репозиторий свою копию не отслеживает — «drift
in a generated file is invisible unless something regenerates it with --check.
Upstream vigiles tracks its own copy and it IS stale on main for exactly that
reason». Через восемь дней стало хуже. Запись предсказания не помогла; помогает
строка в CI.

Проверено обеими половинами: на актуальном файле проходит, на подложенном
дрейфе (одна строка в конец) падает.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uvk2Zx66BAuxuLGLFL2umd

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-16 22:46:26 +05:00

144 lines
6.1 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* audit → the ONE read-vs-run decision. `audit` is a Lighthouse-style LOCAL
* report: a deterministic READ by default — safe + identical on every OS, nothing
* executes — NOT a CI step (CI uses `vigiles lint`, the deterministic gate). The
* executing checks (safety battery, live MCP resolution, skill-firing trigger-rate)
* run ONLY when there's a human to consent: at a TTY `audit` ASKS once (and
* remembers in `.vigilesrc.json` `audit.measure`); headless (an agent / `--json` /
* `--no-interactive` / a pipe) it stays a read + a one-line nudge — never hangs,
* never silently executes. There is deliberately NO execution flag: automation
* tests the harness through the `vigiles` testing API + skills (the layered tiers),
* not through the report verb. The IO (prompt / run / remember) lives in the CLI;
* this is the pure decision + helpers.
*/
/** Only the env vars that signal a reachable model (parse, don't validate). */
export interface ModelEnv {
readonly ANTHROPIC_API_KEY?: string;
readonly CLAUDECODE?: string;
readonly CLAUDE_CODE_ENTRYPOINT?: string;
}
/**
* Is a real model reachable for the trigger tier? Either a metered API key
* (`ANTHROPIC_API_KEY`), OR an authenticated Claude Code session (`CLAUDECODE=1`
* / `CLAUDE_CODE_ENTRYPOINT`, web/desktop/CLI) — the latter drives the `claude`
* CLI on the user's subscription, no key needed and $0 metered. A tiny env-only
* predicate (not a live probe), so it never spends a token just to decide.
*/
export function hasModelAccess(env: ModelEnv): boolean {
return Boolean(
env.ANTHROPIC_API_KEY ||
env.CLAUDECODE === "1" ||
env.CLAUDE_CODE_ENTRYPOINT,
);
}
/**
* Is the reachable model METERED (a paid API key) rather than a subscription?
* Only affects the consent DISCLOSURE wording (a metered key bills per token; a
* subscription is $0 metered) — the run/skip decision itself is consent-driven,
* not metered-driven.
*/
export function isMeteredAccess(env: ModelEnv): boolean {
return Boolean(env.ANTHROPIC_API_KEY);
}
/** Why the executing checks were skipped (drives the "not run" nudge). */
export type ExecuteSkipReason =
| "nothing" // no executable surface at all (no hooks / MCP / skills) — no nudge
| "headless" // an agent / --json / non-interactive / a pipe — no human to consent
| "remembered-no"; // a sticky .vigilesrc choice said no
/**
* What `audit` should do about the EXECUTING checks (battery + live MCP +
* trigger-rate), as ONE bundle:
* - `run` — run them now (a remembered yes).
* - `ask` — interactive human + something to run + no sticky choice: ask once,
* then remember.
* - `skip` — stay a deterministic read; the `reason` drives a one-line nudge.
*/
export type ExecuteDecision =
| { readonly kind: "run" }
| { readonly kind: "ask" }
| { readonly kind: "skip"; readonly reason: ExecuteSkipReason };
export interface ExecuteEnv {
/** Is there ANY executable surface — runnable hooks, an own-repo MCP server, or
* a model-invocable skill? Nothing to run → never ask, never nudge. */
readonly hasExecutable: boolean;
/** Both stdin AND stdout are a terminal (a human who can answer + wait). */
readonly isTTY: boolean;
/** `--json` — machine output; stays a read even at a TTY (never prompt). */
readonly json: boolean;
/** `--no-interactive` / `--yes` — explicit agent/CI mode (never prompt). */
readonly noInteractive: boolean;
/** Sticky remembered choice from `.vigilesrc.json` (`audit.measure`), or undefined. */
readonly remembered?: boolean;
}
/**
* Decide what `audit` does with the executing checks. Total + pure; the first
* matching rule wins. There is NO execution flag — `audit` is a local report, so
* the executing checks need a human to consent:
* 1. nothing executable → skip "nothing" (a clean read; no nudge)
* 2. headless (`--json` / `--no-interactive` / non-TTY — an agent, a pipe, CI) →
* skip "headless" (no one to ask; automation uses the `vigiles` testing API)
* 3. sticky no → skip "remembered-no"
* 4. sticky yes → run
* 5. interactive human, no sticky choice → ask (then remember)
*/
export function decideExecute(o: ExecuteEnv): ExecuteDecision {
if (!o.hasExecutable) return { kind: "skip", reason: "nothing" };
if (o.json || o.noInteractive || !o.isTTY)
return { kind: "skip", reason: "headless" };
if (o.remembered === false) return { kind: "skip", reason: "remembered-no" };
if (o.remembered === true) return { kind: "run" };
return { kind: "ask" };
}
/**
* The one-line "executing checks not run" nudge for a skipped read (the
* no-silent-skips corollary). Returns null for `nothing` (nothing to run — not a
* gap). There is no flag to point at — `audit` runs them only interactively, and
* automation uses the `vigiles` testing API.
*/
export function formatExecuteSkip(reason: ExecuteSkipReason): string | null {
switch (reason) {
case "nothing":
return null;
case "headless":
return (
"\n Executing checks (safety battery · live MCP · skill firing) skipped — " +
"`audit` runs them only interactively (a terminal). For automation, test the " +
"harness with the `vigiles` testing API."
);
case "remembered-no":
return (
"\n Executing checks not run (you disabled them — edit .vigilesrc.json " +
"`audit.measure` to re-enable)."
);
}
}
/**
* A starter `--prompts` file (the real `TriggerPromptSet` shape: bare skill name
* → `{ prompts, irrelevant }`). One entry per triggerable skill, with TODO
* placeholders the user replaces with real requests.
*/
export function scaffoldTriggerPrompts(skillNames: readonly string[]): string {
const obj: Record<string, { prompts: string[]; irrelevant: string[] }> = {};
for (const name of skillNames) {
obj[name] = {
prompts: [
`TODO: a request that SHOULD trigger "${name}"`,
`TODO: a differently-phrased request that should also trigger it`,
],
irrelevant: [
`TODO: an unrelated request that should NOT trigger "${name}"`,
],
};
}
return JSON.stringify(obj, null, 2) + "\n";
}