Files
zernie__vigiles/action.yml
T
zernie 81be2fd07e feat: capability-diff PR comment + local observe ledger (#55)
* docs(vault): harness-quality tech camps + telemetry/default-install decision

Two-camps tech map (runtime/observability vs pre-deploy simulation vs
app-frameworks), the field direction, and the live fork: local-first
deterministic measurement over hosted runtime telemetry. Cross-linked
from the vault index and funded-adjacency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(vault): add pseudocode DX + fundability/exit thesis to tech-direction

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(research): tech direction — four-instrument loop + local observability ledger

The architecture decision of record for the next build phase: typed spec as
declared ground truth; verify/gate/measure/observe as facets of one loop; the
deterministic-vs-behavioral precision principle; the local agent-readable
runs.jsonl flight-recorder ledger; promote-prose; per-surface map. Monetization
kept out (vault only). Indexed + recompiled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: add Direction section to root CLAUDE.md positioning

Elevate the next-phase direction to the north-star doc (read first every
session): the four-instrument loop (verify/gate/measure/observe) with the typed
spec as declared ground truth, the deterministic-vs-behavioral precision
principle, the local agent-readable runs.jsonl flight recorder, promote-prose,
and the per-surface map. Tech only; monetization stays in the vault.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(observe): local agent-readable flight-recorder ledger (runs.jsonl)

The connective layer of the four-instrument loop: a versioned, append-only,
tolerant-to-torn-lines JSONL ledger every instrument (verify/gate/measure/observe)
appends to, read off one schema by audit, by the agent debugging its own harness,
and later by an aggregation surface. Harness-agnostic; lives at the composition
root (not core, not an adapter). Best-effort append never breaks a session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: refresh HANDOFF for direction + observe-ledger session

* feat(observe): emit compiled-hook + agent-rail decisions to the ledger

First emit points into .vigiles/runs.jsonl: the compiled-hook gate (emitGate —
enforce deny/ask + observe would-be) and the subagent tool-contract rail
(agentHookCommand deny). The legacy hook-observations.jsonl still written for
back-compat; the unified ledger is additive. E2E asserts the observe-mode run
populates runs.jsonl with a decision-shaped hook record.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(observe): render the flight recorder in vigiles audit

formatLedgerSummary (pure, tested) — total + counts by kind + recent
high-signal denials (blocked gates, out-of-contract tool calls) — and audit
prints it (terminal, non-json) off the local ledger. Empty/skipped until
something is recorded. JSON AuditReport unchanged (a versioned schema bump is
the deliberate follow-up).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: gitignore the observability ledger (.vigiles/runs.jsonl)

Runtime flight-recorder artifact — local record, never committed (sibling of
active-skill.json + eval-cache/).

* feat(observe): emit skill-fire + trigger-rate metrics to the ledger

skill-start records a SkillObservation{fired:true} (name = the skill dir); a
completed measureTriggerRate records recall (+ precision when measured) as eval
records. Extends ledger coverage across the behavioral surfaces alongside the
deterministic gate/rail emits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(skills): add model-invocable debug-my-harness skill

Reads the .vigiles/runs.jsonl flight recorder to diagnose why the harness
misbehaved — skill fires/collisions, hook decisions (blocked vs wrongly-allowed),
subagent contract violations, trigger-rate drift — and recommends a fix, handing
off to strengthen/edit-spec/test-harness. The agent-readable payoff of the
observe layer. Ships a dogfood trigger eval (recall + precision); loads clean
(skills-dogfood). Skill enumerations + keyFiles synced (3→4 model-invocable).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(observe): emit capability-diff to the ledger

A computed capability-diff (--capability-diff) now records a capability-diff
record (added/removed tools + widened verdict). Completes the ledger's emit
coverage — all five record kinds (hook/agent/skill/eval/capability-diff) now
have live emitters.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: refresh HANDOFF — observe layer built (ledger + 5 emits + audit render + skill)

* feat(observe): carry the flight-recorder summary in the AuditReport JSON + HTML

summarizeObservations (pure, shares denial logic with the terminal formatter)
→ an additive/optional observations field on AuditReport (no schema bump, per the
adoptability/adoptable precedent). The CLI reads the ledger once, feeding both the
JSON (the product boundary — audit --json + the HTML) and the terminal render. New
report app: schema mirror + a presentational Observations component + a Flight
recorder section. Covered by observe + audit-report tests; end-to-end --json + HTML
verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: refresh HANDOFF — AuditReport JSON/HTML ledger integration done

* docs(readme): verify-first front door (catch the silent breakage)

Retune the tagline + hook to lead with the verify moat (references are real,
not just well-formed) over the prior test-first framing; add a coding-scoped
subhead naming CLAUDE.md/skills/hooks + Claude Code & Codex. audit stays the
proof-first hero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): lock verify-first tagline in the editor-guide (prevent drift)

* docs: encode research-depth discipline (vault) + link directionality (doc-tiers)

Vault operating rule 6: competitive/strategy research saved in full detail,
depth-tagged (deep/medium/thin), coverage-mapped, cross-linked, indexed — shallow
never masquerades as deep. Root doc-tiers: link directionality invariant — links
point outward only (vault may cite research; research/public must never cite the vault).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(readme): demote the 7-catalog breadth from the headline

Lead the reference-verification pitch with the harness-native cross-ref (a
referenced tool/hook/file/skill that doesn't resolve). The 7-linter-catalog
breadth becomes a supporting mention, not the headline number.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor: delete hook-spec spike and demote scaffold-test to skill-internal

Trim the launch surface of two dead-weight items flagged by a strategy review:

- Delete the src/core/hook-spec.ts + hook-spec.test.ts spike — imported
  nowhere and superseded by the shipped compiled hooks (src/core/hook-program.ts).
  Drops its two keyFiles entries and recompiles CLAUDE.md.

- Demote the scaffold-test VERB to skill-internal: remove it from VERBS +
  its dispatch/handler in the CLI (the engine src/scaffold-test.ts + its test
  stay, used by the test-harness skill). Prune the now-orphaned helpers/imports,
  the scan-cli e2e block for the removed verb, and the scaffold-demo example,
  and update the docs (cli.md, spec-format.md, STABILITY.md, CLAUDE.md).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF with observe layer + verify-first pivot state

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* style: format harness-observability-direction.md

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(action): capability-diff PR comment (blast-radius diff vs PR base)

Fold the agent capability-surface diff into the sticky PR comment: on a
pull_request the Action materializes the PR base with `git archive` (no
worktree/smudge), runs `audit --capability-diff`, and appends a short
section only when the surface changed. `fail-on-widen` gates a widening.
Dogfooded in this repo's own CI (fetch-depth: 0 + capability-diff: true).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: salvage deep agent-eval-testing competitor teardown (partial workflow)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: refresh HANDOFF — capability-diff shipped + teardown salvaged

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(vault): post-mortem the teardown throttle+interrupt; log the 2x2 re-run

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: fold in the 4 re-run teardown clusters (all 5 now deep, 41 companies)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: HANDOFF — teardown re-run complete (5 clusters, 41 companies)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: add raw structured teardown appendix (41 companies, durable source)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: add the Opus cross-corpus strategic synthesis (conformance-kernel thesis)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: fold in Fable contrarian pass + Opus reconciliation (registry-first bet)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* vault: record the distro thesis (leading direction) + index it

* docs: fix doc-tiers — repo is PUBLIC, strategy is vault-only (stop leaks)

The doc-tiering rules wrongly treated research/ + CLAUDE.md as a 'private
record' holding the moat/positioning narrative. The repo is open-source, so
those tiers are public. Corrected: research/CLAUDE.md hold the TECHNICAL record
(how the tool works, public-safe); all strategy/moat/monetization/pivot theses
live ONLY in the git-crypt vault. Adds the competitor-playbook test.

* docs: de-strategize + refresh HANDOFF (it's public — strategy → vault)

* docs: move 6 strategic docs to the git-crypt vault (repo is public)

measurement-authority, divergent-bets, distribution-strategy, typed-spec-moat,
strategic-synthesis-2026-06, eval-startups-positioning move from research/ (public)
to startup/ (encrypted). Inbound references neutralized (no vault inward-links),
index/keyFiles entries removed, both specs recompiled, vault index updated.

* fix: encrypt the 6 migrated vault docs (git mv bypassed git-crypt)

* docs: record the git-mv-into-vault-doesn't-encrypt gotcha + fix rule 3 (all public)

* docs: pivot-decision (vault) + roadmap reshape + finish moved-doc ref cleanup

Adds startup/pivot-decision-2026-07.md (decision + validation gate). Adds two
pivot-agnostic technical items to research/roadmap.md 'Now' (step-enforcement A/B,
observe→promote) + de-strategizes its top direction banner. Neutralizes the
relative-path refs to the 6 vault-moved docs the migration missed (README +
plugin-structural fully clean; roadmap partial — 37 scattered refs flagged as
follow-up).

* docs: HANDOFF — migration done (targeted-6) + flag roadmap ref-sweep follow-up

* vault: save the pivot exploration log (all options considered + why)

* docs: finish roadmap de-strategization (neutralize moved-doc refs + gut GTM section)

* fix(cli): write capability-diff observation to the audited root's ledger

appendObservation defaulted to process.cwd(), so `audit ./after
--capability-diff=./before` from a parent dir recorded the blast-radius event
into the caller's .vigiles/runs.jsonl instead of ./after's — where the audit
reads it back via readObservations(root). Pass root. (Codex review #55.)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-02 23:18:27 +05:00

246 lines
11 KiB
YAML

name: "vigiles"
description: "Verify the references in your agent instruction files (CLAUDE.md/AGENTS.md) and compile typed specs — in CI."
author: "zernie"
branding:
icon: "check-circle"
color: "orange"
inputs:
command:
description: >
Which vigiles command to run: 'lint' (verify references + integrity +
coverage), 'compile' (specs → markdown), or 'eval-check' (verify committed
eval locks against current inputs — the staleness gate; runs NO model, so
it is the CI-safe half of evals you produce locally with `vigiles eval --update`).
required: false
default: "lint"
paths:
description: "Comma- or space-separated paths to operate on (.md files for lint, .spec.ts files for compile). Auto-discovers when empty."
required: false
default: ""
version:
description: "Version of the vigiles npm package to run (e.g. '1', '1.2.3', 'latest'). Use 'local' to run the action's own checkout — for dogfooding via `uses: ./` in this repo's CI."
required: false
default: "latest"
max-rules:
description: "Maximum number of rules per spec file (maps to the CLI's --max-rules)."
required: false
default: ""
catalog-only:
description: "Only check that linter rules exist; skip config-enabled checks (maps to the CLI's --catalog-only). 'true' or 'false'."
required: false
default: "false"
working-directory:
description: "Directory to run vigiles in."
required: false
default: "."
comment:
description: "On pull_request events, post/update a sticky PR comment with the result ('true' or 'false'). Requires `pull-requests: write`."
required: false
default: "true"
capability-diff:
description: >
On pull_request events, compare the agent's capability surface (the
subagents' tool/effect blast radius) against the PR base and fold the diff
into the sticky comment (maps to `audit --capability-diff`). Requires
`fetch-depth: 0` on checkout so the base commit is present. 'true' or 'false'.
required: false
default: "false"
fail-on-widen:
description: "With capability-diff, fail the run when the PR WIDENS the blast radius (maps to the CLI's --fail-on-widen). 'true' or 'false'."
required: false
default: "false"
github-token:
description: "Token used to post the PR comment. Defaults to the workflow token."
required: false
default: ${{ github.token }}
outputs:
valid:
description: "'true' if vigiles passed (exit 0), 'false' otherwise."
value: ${{ steps.vigiles.outputs.valid }}
runs:
using: "composite"
steps:
- id: vigiles
shell: bash
working-directory: ${{ inputs.working-directory }}
env:
# Inputs are passed through the environment (never interpolated into
# the script body) so a value can't break out of the command line.
VIGILES_COMMAND: ${{ inputs.command }}
VIGILES_PATHS: ${{ inputs.paths }}
VIGILES_VERSION: ${{ inputs.version }}
VIGILES_MAX_RULES: ${{ inputs.max-rules }}
VIGILES_CATALOG_ONLY: ${{ inputs.catalog-only }}
VIGILES_ACTION_PATH: ${{ github.action_path }}
VIGILES_COMMENT: ${{ inputs.comment }}
VIGILES_CAPABILITY_DIFF: ${{ inputs.capability-diff }}
VIGILES_FAIL_ON_WIDEN: ${{ inputs.fail-on-widen }}
GH_TOKEN: ${{ inputs.github-token }}
run: |
set -euo pipefail
cmd="${VIGILES_COMMAND:-lint}"
# 'eval-check' is the CI staleness gate — it maps to the real verb
# `eval --check` (verify committed locks vs current inputs, NO model).
# Real evals run locally on a subscription (`eval --update`), never in CI.
if [[ "$cmd" == "eval-check" ]]; then
args=("eval" "--check")
else
args=("$cmd")
fi
# paths: split on commas and whitespace into positional args.
paths="${VIGILES_PATHS:-}"
if [[ -n "${paths// /}" ]]; then
IFS=', ' read -r -a _paths <<< "$paths"
args+=("${_paths[@]}")
fi
max_rules="${VIGILES_MAX_RULES:-}"
if [[ -n "${max_rules// /}" ]]; then
args+=("--max-rules=${max_rules}")
fi
if [[ "${VIGILES_CATALOG_ONLY:-false}" == "true" ]]; then
args+=("--catalog-only")
fi
# Resolve the runner. 'local' runs the action's own build (CI
# dogfood via `uses: ./`); any other value runs the published npm
# package at that version, so the Action reuses the same tested
# artifact users get from `npx vigiles`.
version="${VIGILES_VERSION:-latest}"
if [[ "$version" == "local" ]]; then
runner=(node "${VIGILES_ACTION_PATH}/dist/cli.js")
else
runner=(npx --yes "vigiles@${version}")
fi
echo "vigiles: ${runner[*]} ${args[*]}"
# Run, streaming to the log AND capturing for the summary/comment.
out="$(mktemp)"
set +e
GITHUB_ACTIONS=true "${runner[@]}" "${args[@]}" 2>&1 | tee "$out"
status="${PIPESTATUS[0]}"
set -e
# Capability diff (moat #2): did this PR WIDEN the agent's blast radius?
# Runs orthogonally to the main command — it's an additive PR-comment
# section, so it works whatever `command` gated above. The BEFORE tree is
# a detached worktree of the PR base; the AFTER tree is this checkout.
capdiff="" # markdown appended to the sticky comment
capstatus=0
if [[ "${VIGILES_CAPABILITY_DIFF:-false}" == "true" \
&& "${GITHUB_EVENT_NAME:-}" == "pull_request" ]]; then
base_sha="$(jq -r '.pull_request.base.sha // empty' "${GITHUB_EVENT_PATH:-/dev/null}")"
if [[ -z "$base_sha" ]]; then
echo "::warning::capability-diff: no PR base sha in the event payload"
else
# A shallow checkout may not carry the base commit — fetch it first.
git fetch --no-tags --depth=1 origin "$base_sha" >/dev/null 2>&1 \
|| git fetch --no-tags origin "$base_sha" >/dev/null 2>&1 || true
base_dir="$(mktemp -d)"
# Materialize the base tree with `git archive` (NOT a worktree): it
# exports stored blobs WITHOUT running checkout/smudge filters, so it's
# light and can't trip a required filter (e.g. git-crypt) the runner
# lacks. The agent surface (unencrypted) lands as plaintext to diff.
if git archive "$base_sha" 2>/dev/null | tar -x -C "$base_dir" 2>/dev/null; then
# Diff the SAME relative dir in both trees (support a subdir working-directory).
rel="$(git rev-parse --show-prefix 2>/dev/null || echo '')"
before="${base_dir%/}/${rel}"; before="${before%/}"
capout="$(mktemp)"
fail_flag=(); [[ "${VIGILES_FAIL_ON_WIDEN:-false}" == "true" ]] && fail_flag=(--fail-on-widen)
# --no-html --no-json: this run is for the diff only, so skip the
# report artifacts. The full audit still streams to the CI log; only
# the capability-diff paragraph is extracted for the sticky comment.
set +e
"${runner[@]}" audit . --capability-diff="$before" --no-html --no-json "${fail_flag[@]}" 2>&1 | tee "$capout"
capstatus="${PIPESTATUS[0]}"
set -e
# Pull out just the "Capability surface …" paragraph (a self-contained
# block with no internal blank lines), then drop the no-change case
# so a PR that touches nothing capability-relevant gets no section.
capdiff="$(awk '/Capability surface/{f=1} f&&NF{print} f&&!NF{exit}' "$capout")"
case "$capdiff" in *"unchanged"*) capdiff="" ;; esac
else
echo "::warning::capability-diff: could not materialize PR base ${base_sha} (need fetch-depth: 0?)"
fi
rm -rf "$base_dir" 2>/dev/null || true
fi
fi
# A capability WIDENING (fail-on-widen → exit 1) flips an otherwise-green
# run to failed, so the gate is visible in the output + PR comment.
if [[ "$capstatus" -ne 0 && "$status" -eq 0 ]]; then
status="$capstatus"
fi
if [[ "$status" -eq 0 ]]; then
echo "valid=true" >> "$GITHUB_OUTPUT"
headline="✅ \`vigiles ${cmd}\` passed"
else
echo "valid=false" >> "$GITHUB_OUTPUT"
headline="❌ \`vigiles ${cmd}\` failed (exit ${status})"
echo "::error::vigiles ${cmd} failed (exit ${status})"
fi
# Build a markdown report (cap the log tail so a huge run stays sane).
report="$(mktemp)"
{
echo "## vigiles"
echo
echo "$headline"
echo
if [[ -n "$capdiff" ]]; then
echo "### Capability diff"
echo
echo '```'
echo "$capdiff"
echo '```'
echo
fi
echo '<details><summary>Output</summary>'
echo
echo '```'
tail -c 60000 "$out"
echo '```'
echo
echo '</details>'
} > "$report"
# Job summary (always, when available) — the modern run-page output.
if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then
cat "$report" >> "$GITHUB_STEP_SUMMARY"
fi
# Sticky PR comment (pull_request events only, best-effort): find the
# prior vigiles comment by marker and update it in place, else create.
marker="<!-- vigiles-action -->"
if [[ "${VIGILES_COMMENT:-true}" == "true" \
&& "${GITHUB_EVENT_NAME:-}" == "pull_request" \
&& -n "${GH_TOKEN:-}" ]] && command -v gh >/dev/null 2>&1; then
pr="$(jq -r '.pull_request.number // empty' "${GITHUB_EVENT_PATH:-/dev/null}")"
if [[ -n "$pr" ]]; then
body="$(printf '%s\n%s\n' "$marker" "$(cat "$report")")"
repo="${GITHUB_REPOSITORY}"
cid="$(gh api "repos/${repo}/issues/${pr}/comments" --paginate \
--jq "map(select(.body | startswith(\"${marker}\"))) | .[0].id // empty" \
2>/dev/null || true)"
if [[ -n "$cid" ]]; then
gh api -X PATCH "repos/${repo}/issues/comments/${cid}" -f body="$body" >/dev/null \
&& echo "Updated PR comment #${cid}" \
|| echo "::warning::could not update PR comment (need pull-requests: write?)"
else
gh api -X POST "repos/${repo}/issues/${pr}/comments" -f body="$body" >/dev/null \
&& echo "Posted PR comment" \
|| echo "::warning::could not post PR comment (need pull-requests: write?)"
fi
fi
fi
exit "$status"