mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
848538d888
Render the test and eval tiers as measured findings rather than feature bullets: a real vendored OSS guard's own verdicts through runHook, and measureTriggerRate on superpowers' TDD skill at 83% over 30 runs. Both render from committed fixtures, one regenerated from the engine and pinned by a browser test, the other copied from the lock so CI spends no model quota. Adds a /comparison page rendered from a probe snapshot, retires four unmeasured claims plus two shipped-but-"planned" rows, and type-checks rendered snippets against the real .d.ts instead of scanning them with a regex. Committing this repo's first eval lock also revealed that the CI eval-check step had never once executed — it short-circuited on anyLocksCommitted and exited 0. Two defects behind it are fixed: a quota consent gate that refused a model-free verify, and a dogfood step that now names the one eval holding a lock.
35 lines
1.4 KiB
JavaScript
35 lines
1.4 KiB
JavaScript
/**
|
|
* Copy the committed eval LOCK into the site's fixtures.
|
|
*
|
|
* The eval tier costs real model quota, so unlike the guard fixture this CANNOT be
|
|
* regenerated by re-running the measurement in CI — that would spend money on every
|
|
* push. The lock (`.vigiles/eval-locks/<name>.lock.json`, written by
|
|
* `vigiles eval --update`) IS the committed measurement; this only copies it where
|
|
* the site can import it, so the page still renders from a machine-written artifact
|
|
* rather than a number somebody retyped.
|
|
*
|
|
* Refresh the measurement itself with:
|
|
* node dist/cli.js eval examples/harness/skill-trigger-rate.eval.mjs --trials=3 --update
|
|
*/
|
|
import { copyFileSync, existsSync, mkdirSync } from "node:fs";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
const here = (p) => fileURLToPath(new URL(p, import.meta.url));
|
|
const LOCK = here(
|
|
"../../.vigiles/eval-locks/superpowers-tdd-trigger-rate.lock.json",
|
|
);
|
|
const OUT = here(
|
|
"../src/components/sections/__fixtures__/tdd-trigger-rate.json",
|
|
);
|
|
|
|
if (!existsSync(LOCK)) {
|
|
console.error(
|
|
`no eval lock at ${LOCK} — it is committed, so this means it was deleted. ` +
|
|
`Re-measure: node dist/cli.js eval examples/harness/skill-trigger-rate.eval.mjs --trials=3 --update`,
|
|
);
|
|
process.exit(1);
|
|
}
|
|
mkdirSync(here("../src/components/sections/__fixtures__"), { recursive: true });
|
|
copyFileSync(LOCK, OUT);
|
|
console.log(`[gen-eval-fixture] ${LOCK} → ${OUT}`);
|