mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
81be2fd07e
* docs(vault): harness-quality tech camps + telemetry/default-install decision Two-camps tech map (runtime/observability vs pre-deploy simulation vs app-frameworks), the field direction, and the live fork: local-first deterministic measurement over hosted runtime telemetry. Cross-linked from the vault index and funded-adjacency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(vault): add pseudocode DX + fundability/exit thesis to tech-direction Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(research): tech direction — four-instrument loop + local observability ledger The architecture decision of record for the next build phase: typed spec as declared ground truth; verify/gate/measure/observe as facets of one loop; the deterministic-vs-behavioral precision principle; the local agent-readable runs.jsonl flight-recorder ledger; promote-prose; per-surface map. Monetization kept out (vault only). Indexed + recompiled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: add Direction section to root CLAUDE.md positioning Elevate the next-phase direction to the north-star doc (read first every session): the four-instrument loop (verify/gate/measure/observe) with the typed spec as declared ground truth, the deterministic-vs-behavioral precision principle, the local agent-readable runs.jsonl flight recorder, promote-prose, and the per-surface map. Tech only; monetization stays in the vault. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): local agent-readable flight-recorder ledger (runs.jsonl) The connective layer of the four-instrument loop: a versioned, append-only, tolerant-to-torn-lines JSONL ledger every instrument (verify/gate/measure/observe) appends to, read off one schema by audit, by the agent debugging its own harness, and later by an aggregation surface. Harness-agnostic; lives at the composition root (not core, not an adapter). Best-effort append never breaks a session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF for direction + observe-ledger session * feat(observe): emit compiled-hook + agent-rail decisions to the ledger First emit points into .vigiles/runs.jsonl: the compiled-hook gate (emitGate — enforce deny/ask + observe would-be) and the subagent tool-contract rail (agentHookCommand deny). The legacy hook-observations.jsonl still written for back-compat; the unified ledger is additive. E2E asserts the observe-mode run populates runs.jsonl with a decision-shaped hook record. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): render the flight recorder in vigiles audit formatLedgerSummary (pure, tested) — total + counts by kind + recent high-signal denials (blocked gates, out-of-contract tool calls) — and audit prints it (terminal, non-json) off the local ledger. Empty/skipped until something is recorded. JSON AuditReport unchanged (a versioned schema bump is the deliberate follow-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: gitignore the observability ledger (.vigiles/runs.jsonl) Runtime flight-recorder artifact — local record, never committed (sibling of active-skill.json + eval-cache/). * feat(observe): emit skill-fire + trigger-rate metrics to the ledger skill-start records a SkillObservation{fired:true} (name = the skill dir); a completed measureTriggerRate records recall (+ precision when measured) as eval records. Extends ledger coverage across the behavioral surfaces alongside the deterministic gate/rail emits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(skills): add model-invocable debug-my-harness skill Reads the .vigiles/runs.jsonl flight recorder to diagnose why the harness misbehaved — skill fires/collisions, hook decisions (blocked vs wrongly-allowed), subagent contract violations, trigger-rate drift — and recommends a fix, handing off to strengthen/edit-spec/test-harness. The agent-readable payoff of the observe layer. Ships a dogfood trigger eval (recall + precision); loads clean (skills-dogfood). Skill enumerations + keyFiles synced (3→4 model-invocable). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(observe): emit capability-diff to the ledger A computed capability-diff (--capability-diff) now records a capability-diff record (added/removed tools + widened verdict). Completes the ledger's emit coverage — all five record kinds (hook/agent/skill/eval/capability-diff) now have live emitters. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF — observe layer built (ledger + 5 emits + audit render + skill) * feat(observe): carry the flight-recorder summary in the AuditReport JSON + HTML summarizeObservations (pure, shares denial logic with the terminal formatter) → an additive/optional observations field on AuditReport (no schema bump, per the adoptability/adoptable precedent). The CLI reads the ledger once, feeding both the JSON (the product boundary — audit --json + the HTML) and the terminal render. New report app: schema mirror + a presentational Observations component + a Flight recorder section. Covered by observe + audit-report tests; end-to-end --json + HTML verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: refresh HANDOFF — AuditReport JSON/HTML ledger integration done * docs(readme): verify-first front door (catch the silent breakage) Retune the tagline + hook to lead with the verify moat (references are real, not just well-formed) over the prior test-first framing; add a coding-scoped subhead naming CLAUDE.md/skills/hooks + Claude Code & Codex. audit stays the proof-first hero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): lock verify-first tagline in the editor-guide (prevent drift) * docs: encode research-depth discipline (vault) + link directionality (doc-tiers) Vault operating rule 6: competitive/strategy research saved in full detail, depth-tagged (deep/medium/thin), coverage-mapped, cross-linked, indexed — shallow never masquerades as deep. Root doc-tiers: link directionality invariant — links point outward only (vault may cite research; research/public must never cite the vault). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): demote the 7-catalog breadth from the headline Lead the reference-verification pitch with the harness-native cross-ref (a referenced tool/hook/file/skill that doesn't resolve). The 7-linter-catalog breadth becomes a supporting mention, not the headline number. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: delete hook-spec spike and demote scaffold-test to skill-internal Trim the launch surface of two dead-weight items flagged by a strategy review: - Delete the src/core/hook-spec.ts + hook-spec.test.ts spike — imported nowhere and superseded by the shipped compiled hooks (src/core/hook-program.ts). Drops its two keyFiles entries and recompiles CLAUDE.md. - Demote the scaffold-test VERB to skill-internal: remove it from VERBS + its dispatch/handler in the CLI (the engine src/scaffold-test.ts + its test stay, used by the test-harness skill). Prune the now-orphaned helpers/imports, the scan-cli e2e block for the removed verb, and the scaffold-demo example, and update the docs (cli.md, spec-format.md, STABILITY.md, CLAUDE.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF with observe layer + verify-first pivot state Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: format harness-observability-direction.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(action): capability-diff PR comment (blast-radius diff vs PR base) Fold the agent capability-surface diff into the sticky PR comment: on a pull_request the Action materializes the PR base with `git archive` (no worktree/smudge), runs `audit --capability-diff`, and appends a short section only when the surface changed. `fail-on-widen` gates a widening. Dogfooded in this repo's own CI (fetch-depth: 0 + capability-diff: true). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: salvage deep agent-eval-testing competitor teardown (partial workflow) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh HANDOFF — capability-diff shipped + teardown salvaged Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(vault): post-mortem the teardown throttle+interrupt; log the 2x2 re-run Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: fold in the 4 re-run teardown clusters (all 5 now deep, 41 companies) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: HANDOFF — teardown re-run complete (5 clusters, 41 companies) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: add raw structured teardown appendix (41 companies, durable source) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: add the Opus cross-corpus strategic synthesis (conformance-kernel thesis) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: fold in Fable contrarian pass + Opus reconciliation (registry-first bet) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * vault: record the distro thesis (leading direction) + index it * docs: fix doc-tiers — repo is PUBLIC, strategy is vault-only (stop leaks) The doc-tiering rules wrongly treated research/ + CLAUDE.md as a 'private record' holding the moat/positioning narrative. The repo is open-source, so those tiers are public. Corrected: research/CLAUDE.md hold the TECHNICAL record (how the tool works, public-safe); all strategy/moat/monetization/pivot theses live ONLY in the git-crypt vault. Adds the competitor-playbook test. * docs: de-strategize + refresh HANDOFF (it's public — strategy → vault) * docs: move 6 strategic docs to the git-crypt vault (repo is public) measurement-authority, divergent-bets, distribution-strategy, typed-spec-moat, strategic-synthesis-2026-06, eval-startups-positioning move from research/ (public) to startup/ (encrypted). Inbound references neutralized (no vault inward-links), index/keyFiles entries removed, both specs recompiled, vault index updated. * fix: encrypt the 6 migrated vault docs (git mv bypassed git-crypt) * docs: record the git-mv-into-vault-doesn't-encrypt gotcha + fix rule 3 (all public) * docs: pivot-decision (vault) + roadmap reshape + finish moved-doc ref cleanup Adds startup/pivot-decision-2026-07.md (decision + validation gate). Adds two pivot-agnostic technical items to research/roadmap.md 'Now' (step-enforcement A/B, observe→promote) + de-strategizes its top direction banner. Neutralizes the relative-path refs to the 6 vault-moved docs the migration missed (README + plugin-structural fully clean; roadmap partial — 37 scattered refs flagged as follow-up). * docs: HANDOFF — migration done (targeted-6) + flag roadmap ref-sweep follow-up * vault: save the pivot exploration log (all options considered + why) * docs: finish roadmap de-strategization (neutralize moved-doc refs + gut GTM section) * fix(cli): write capability-diff observation to the audited root's ledger appendObservation defaulted to process.cwd(), so `audit ./after --capability-diff=./before` from a parent dir recorded the blast-radius event into the caller's .vigiles/runs.jsonl instead of ./after's — where the audit reads it back via readObservations(root). Pass root. (Codex review #55.) --------- Co-authored-by: Claude <noreply@anthropic.com>
246 lines
11 KiB
YAML
246 lines
11 KiB
YAML
name: "vigiles"
|
|
description: "Verify the references in your agent instruction files (CLAUDE.md/AGENTS.md) and compile typed specs — in CI."
|
|
author: "zernie"
|
|
branding:
|
|
icon: "check-circle"
|
|
color: "orange"
|
|
|
|
inputs:
|
|
command:
|
|
description: >
|
|
Which vigiles command to run: 'lint' (verify references + integrity +
|
|
coverage), 'compile' (specs → markdown), or 'eval-check' (verify committed
|
|
eval locks against current inputs — the staleness gate; runs NO model, so
|
|
it is the CI-safe half of evals you produce locally with `vigiles eval --update`).
|
|
required: false
|
|
default: "lint"
|
|
paths:
|
|
description: "Comma- or space-separated paths to operate on (.md files for lint, .spec.ts files for compile). Auto-discovers when empty."
|
|
required: false
|
|
default: ""
|
|
version:
|
|
description: "Version of the vigiles npm package to run (e.g. '1', '1.2.3', 'latest'). Use 'local' to run the action's own checkout — for dogfooding via `uses: ./` in this repo's CI."
|
|
required: false
|
|
default: "latest"
|
|
max-rules:
|
|
description: "Maximum number of rules per spec file (maps to the CLI's --max-rules)."
|
|
required: false
|
|
default: ""
|
|
catalog-only:
|
|
description: "Only check that linter rules exist; skip config-enabled checks (maps to the CLI's --catalog-only). 'true' or 'false'."
|
|
required: false
|
|
default: "false"
|
|
working-directory:
|
|
description: "Directory to run vigiles in."
|
|
required: false
|
|
default: "."
|
|
comment:
|
|
description: "On pull_request events, post/update a sticky PR comment with the result ('true' or 'false'). Requires `pull-requests: write`."
|
|
required: false
|
|
default: "true"
|
|
capability-diff:
|
|
description: >
|
|
On pull_request events, compare the agent's capability surface (the
|
|
subagents' tool/effect blast radius) against the PR base and fold the diff
|
|
into the sticky comment (maps to `audit --capability-diff`). Requires
|
|
`fetch-depth: 0` on checkout so the base commit is present. 'true' or 'false'.
|
|
required: false
|
|
default: "false"
|
|
fail-on-widen:
|
|
description: "With capability-diff, fail the run when the PR WIDENS the blast radius (maps to the CLI's --fail-on-widen). 'true' or 'false'."
|
|
required: false
|
|
default: "false"
|
|
github-token:
|
|
description: "Token used to post the PR comment. Defaults to the workflow token."
|
|
required: false
|
|
default: ${{ github.token }}
|
|
|
|
outputs:
|
|
valid:
|
|
description: "'true' if vigiles passed (exit 0), 'false' otherwise."
|
|
value: ${{ steps.vigiles.outputs.valid }}
|
|
|
|
runs:
|
|
using: "composite"
|
|
steps:
|
|
- id: vigiles
|
|
shell: bash
|
|
working-directory: ${{ inputs.working-directory }}
|
|
env:
|
|
# Inputs are passed through the environment (never interpolated into
|
|
# the script body) so a value can't break out of the command line.
|
|
VIGILES_COMMAND: ${{ inputs.command }}
|
|
VIGILES_PATHS: ${{ inputs.paths }}
|
|
VIGILES_VERSION: ${{ inputs.version }}
|
|
VIGILES_MAX_RULES: ${{ inputs.max-rules }}
|
|
VIGILES_CATALOG_ONLY: ${{ inputs.catalog-only }}
|
|
VIGILES_ACTION_PATH: ${{ github.action_path }}
|
|
VIGILES_COMMENT: ${{ inputs.comment }}
|
|
VIGILES_CAPABILITY_DIFF: ${{ inputs.capability-diff }}
|
|
VIGILES_FAIL_ON_WIDEN: ${{ inputs.fail-on-widen }}
|
|
GH_TOKEN: ${{ inputs.github-token }}
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
cmd="${VIGILES_COMMAND:-lint}"
|
|
# 'eval-check' is the CI staleness gate — it maps to the real verb
|
|
# `eval --check` (verify committed locks vs current inputs, NO model).
|
|
# Real evals run locally on a subscription (`eval --update`), never in CI.
|
|
if [[ "$cmd" == "eval-check" ]]; then
|
|
args=("eval" "--check")
|
|
else
|
|
args=("$cmd")
|
|
fi
|
|
|
|
# paths: split on commas and whitespace into positional args.
|
|
paths="${VIGILES_PATHS:-}"
|
|
if [[ -n "${paths// /}" ]]; then
|
|
IFS=', ' read -r -a _paths <<< "$paths"
|
|
args+=("${_paths[@]}")
|
|
fi
|
|
|
|
max_rules="${VIGILES_MAX_RULES:-}"
|
|
if [[ -n "${max_rules// /}" ]]; then
|
|
args+=("--max-rules=${max_rules}")
|
|
fi
|
|
|
|
if [[ "${VIGILES_CATALOG_ONLY:-false}" == "true" ]]; then
|
|
args+=("--catalog-only")
|
|
fi
|
|
|
|
# Resolve the runner. 'local' runs the action's own build (CI
|
|
# dogfood via `uses: ./`); any other value runs the published npm
|
|
# package at that version, so the Action reuses the same tested
|
|
# artifact users get from `npx vigiles`.
|
|
version="${VIGILES_VERSION:-latest}"
|
|
if [[ "$version" == "local" ]]; then
|
|
runner=(node "${VIGILES_ACTION_PATH}/dist/cli.js")
|
|
else
|
|
runner=(npx --yes "vigiles@${version}")
|
|
fi
|
|
|
|
echo "vigiles: ${runner[*]} ${args[*]}"
|
|
|
|
# Run, streaming to the log AND capturing for the summary/comment.
|
|
out="$(mktemp)"
|
|
set +e
|
|
GITHUB_ACTIONS=true "${runner[@]}" "${args[@]}" 2>&1 | tee "$out"
|
|
status="${PIPESTATUS[0]}"
|
|
set -e
|
|
|
|
# Capability diff (moat #2): did this PR WIDEN the agent's blast radius?
|
|
# Runs orthogonally to the main command — it's an additive PR-comment
|
|
# section, so it works whatever `command` gated above. The BEFORE tree is
|
|
# a detached worktree of the PR base; the AFTER tree is this checkout.
|
|
capdiff="" # markdown appended to the sticky comment
|
|
capstatus=0
|
|
if [[ "${VIGILES_CAPABILITY_DIFF:-false}" == "true" \
|
|
&& "${GITHUB_EVENT_NAME:-}" == "pull_request" ]]; then
|
|
base_sha="$(jq -r '.pull_request.base.sha // empty' "${GITHUB_EVENT_PATH:-/dev/null}")"
|
|
if [[ -z "$base_sha" ]]; then
|
|
echo "::warning::capability-diff: no PR base sha in the event payload"
|
|
else
|
|
# A shallow checkout may not carry the base commit — fetch it first.
|
|
git fetch --no-tags --depth=1 origin "$base_sha" >/dev/null 2>&1 \
|
|
|| git fetch --no-tags origin "$base_sha" >/dev/null 2>&1 || true
|
|
base_dir="$(mktemp -d)"
|
|
# Materialize the base tree with `git archive` (NOT a worktree): it
|
|
# exports stored blobs WITHOUT running checkout/smudge filters, so it's
|
|
# light and can't trip a required filter (e.g. git-crypt) the runner
|
|
# lacks. The agent surface (unencrypted) lands as plaintext to diff.
|
|
if git archive "$base_sha" 2>/dev/null | tar -x -C "$base_dir" 2>/dev/null; then
|
|
# Diff the SAME relative dir in both trees (support a subdir working-directory).
|
|
rel="$(git rev-parse --show-prefix 2>/dev/null || echo '')"
|
|
before="${base_dir%/}/${rel}"; before="${before%/}"
|
|
capout="$(mktemp)"
|
|
fail_flag=(); [[ "${VIGILES_FAIL_ON_WIDEN:-false}" == "true" ]] && fail_flag=(--fail-on-widen)
|
|
# --no-html --no-json: this run is for the diff only, so skip the
|
|
# report artifacts. The full audit still streams to the CI log; only
|
|
# the capability-diff paragraph is extracted for the sticky comment.
|
|
set +e
|
|
"${runner[@]}" audit . --capability-diff="$before" --no-html --no-json "${fail_flag[@]}" 2>&1 | tee "$capout"
|
|
capstatus="${PIPESTATUS[0]}"
|
|
set -e
|
|
# Pull out just the "Capability surface …" paragraph (a self-contained
|
|
# block with no internal blank lines), then drop the no-change case
|
|
# so a PR that touches nothing capability-relevant gets no section.
|
|
capdiff="$(awk '/Capability surface/{f=1} f&&NF{print} f&&!NF{exit}' "$capout")"
|
|
case "$capdiff" in *"unchanged"*) capdiff="" ;; esac
|
|
else
|
|
echo "::warning::capability-diff: could not materialize PR base ${base_sha} (need fetch-depth: 0?)"
|
|
fi
|
|
rm -rf "$base_dir" 2>/dev/null || true
|
|
fi
|
|
fi
|
|
|
|
# A capability WIDENING (fail-on-widen → exit 1) flips an otherwise-green
|
|
# run to failed, so the gate is visible in the output + PR comment.
|
|
if [[ "$capstatus" -ne 0 && "$status" -eq 0 ]]; then
|
|
status="$capstatus"
|
|
fi
|
|
|
|
if [[ "$status" -eq 0 ]]; then
|
|
echo "valid=true" >> "$GITHUB_OUTPUT"
|
|
headline="✅ \`vigiles ${cmd}\` passed"
|
|
else
|
|
echo "valid=false" >> "$GITHUB_OUTPUT"
|
|
headline="❌ \`vigiles ${cmd}\` failed (exit ${status})"
|
|
echo "::error::vigiles ${cmd} failed (exit ${status})"
|
|
fi
|
|
|
|
# Build a markdown report (cap the log tail so a huge run stays sane).
|
|
report="$(mktemp)"
|
|
{
|
|
echo "## vigiles"
|
|
echo
|
|
echo "$headline"
|
|
echo
|
|
if [[ -n "$capdiff" ]]; then
|
|
echo "### Capability diff"
|
|
echo
|
|
echo '```'
|
|
echo "$capdiff"
|
|
echo '```'
|
|
echo
|
|
fi
|
|
echo '<details><summary>Output</summary>'
|
|
echo
|
|
echo '```'
|
|
tail -c 60000 "$out"
|
|
echo '```'
|
|
echo
|
|
echo '</details>'
|
|
} > "$report"
|
|
|
|
# Job summary (always, when available) — the modern run-page output.
|
|
if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then
|
|
cat "$report" >> "$GITHUB_STEP_SUMMARY"
|
|
fi
|
|
|
|
# Sticky PR comment (pull_request events only, best-effort): find the
|
|
# prior vigiles comment by marker and update it in place, else create.
|
|
marker="<!-- vigiles-action -->"
|
|
if [[ "${VIGILES_COMMENT:-true}" == "true" \
|
|
&& "${GITHUB_EVENT_NAME:-}" == "pull_request" \
|
|
&& -n "${GH_TOKEN:-}" ]] && command -v gh >/dev/null 2>&1; then
|
|
pr="$(jq -r '.pull_request.number // empty' "${GITHUB_EVENT_PATH:-/dev/null}")"
|
|
if [[ -n "$pr" ]]; then
|
|
body="$(printf '%s\n%s\n' "$marker" "$(cat "$report")")"
|
|
repo="${GITHUB_REPOSITORY}"
|
|
cid="$(gh api "repos/${repo}/issues/${pr}/comments" --paginate \
|
|
--jq "map(select(.body | startswith(\"${marker}\"))) | .[0].id // empty" \
|
|
2>/dev/null || true)"
|
|
if [[ -n "$cid" ]]; then
|
|
gh api -X PATCH "repos/${repo}/issues/comments/${cid}" -f body="$body" >/dev/null \
|
|
&& echo "Updated PR comment #${cid}" \
|
|
|| echo "::warning::could not update PR comment (need pull-requests: write?)"
|
|
else
|
|
gh api -X POST "repos/${repo}/issues/${pr}/comments" -f body="$body" >/dev/null \
|
|
&& echo "Posted PR comment" \
|
|
|| echo "::warning::could not post PR comment (need pull-requests: write?)"
|
|
fi
|
|
fi
|
|
fi
|
|
|
|
exit "$status"
|