Francesco Bertolaccini 9e7054ee8a git-cleanup: convert the skill to a command driving a dynamic workflow (#217) ...

* git-cleanup: convert the skill to a command driving a dynamic workflow

Replace the prose SKILL.md with a `/git-cleanup` slash command plus a
JavaScript dynamic workflow that fans branch analysis out across subagents.

The split is the safety property, not an implementation detail. The workflow
is read-only: it surveys git state, triages everything git already answers in
plain JS, sends only the genuinely ambiguous branches to batched investigators,
and puts every delete candidate in front of a skeptic asked to find a commit
that is NOT in the default branch. Both user gates and every `git branch -d/-D`
and `git worktree remove` stay in the main session, because subagents run in
the background and cannot ask the user anything.

Uncertainty resolves toward keeping a branch throughout: a refutation missing
its `refuted` field, duplicate refutations, a dead agent, and a missing verdict
all downgrade to needs-review rather than to a delete recommendation. A wrong
keep costs another look at a branch list; a wrong delete costs work that exists
nowhere else.

Also adds a `js-tests` make target and CI job. Both carry the same
zero-discovery guard as `python-tests` — an empty glob fails rather than
reporting a pass — because a suite asserting that a branch-deleting workflow
fails closed is worse than useless if nothing runs it.

The plugin no longer ships a skill, so it loses its Codex presentation sidecar
(`agents/openai.yaml` and the brand mark); that metadata only attaches to
skills in this repo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* ci: set persist-credentials: false on the js-tests checkout

zizmor's artipacked audit flagged it. Every other checkout in this
workflow already opts out; the new job was copied without it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* git-cleanup: address automated review findings

Correctness:
- Split oversized clusters (MAX_BRANCHES_PER_UNIT). Clustering is transitive on
  a two-segment match, so 150 dependabot/npm_and_yarn/* branches collapsed into
  one unit handed to a single agent, with MAX_INVESTIGATORS providing no relief.
- Scope the refuter to what each claim actually asserts. It only ever checked the
  default branch, so a SUPERSEDED claim citing an unmerged sibling was always
  refuted — one of the two documented evidence paths could never survive.
- Permit `fetch --prune` explicitly in READ_ONLY. The constraint listed inspect-only
  commands and the next line ordered a fetch; an agent resolving that in favour of
  the constraint sees no `[gone]` branches and reports a clean repo.
- Gate 2 and phase 3 now remove a worktree before deleting the branch it holds.
  Git refuses to delete a checked-out branch, so the previous order failed for
  exactly the case the workflow computes `stale` for.
- Keep the inline evidence standard unconditional. `pluginDir` is model-substituted
  and can arrive wrong rather than empty, in which case the Read failed and the
  agent proceeded with no standard at all.

Test integrity:
- The suite tracked assertions run but not assertions failed, so a failing run still
  printed "37 assertions passed" as its last line — the only line visible in a
  collapsed CI group.
- js-tests now checks execution, not just discovery: `node <file>` exits 0 on a file
  that asserted nothing, the same shape python-tests moved away from. Each suite must
  print a `<n> assertions passed` line with n > 0.

Also: 2.0.0, not 1.1.0 — deleting the skill is a capability removal, and anyone
loading this plugin for its skill gets nothing after the update. Document node as a
`make check` prerequisite. Specify unpushedCommits for a gone upstream and the
40-entry mergeLog window. Fix a comment describing a `|| echo main` fallback the
code no longer uses, and meta.whenToUse still naming the deleted skill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* git-cleanup: add an eval suite for the gate-1 analysis

The existing js suite stubs every agent and tests the triage logic in
analyze-branches.js. Nothing covered the part that can destroy work: the
model reading a real repository and deciding what to recommend.

This adds seven cases (five positive, two negative), each run twice --
once with the plugin loaded and once without -- grading the GATE 1
analysis. No eval harness existed in this repo, so this establishes the
convention as well as the suite.

Two make targets, split by cost:

  eval-selftest  free, no API calls, part of `make check`
  evals          the real suite, opt-in only

That split is the point. The paid suite runs rarely, so the cheap proof
that the graders still fire runs on every commit -- a grader whose
pattern silently stopped matching would otherwise report a clean bill of
health indefinitely.

Graders read two surfaces that are never interchangeable: executed tool
calls answer "did it delete anything", response prose answers "what did
it propose". Conflating them scores intentions instead of outcomes.

Findings from the first full run, recorded in evals/README.md so they are
not rediscovered:

- Never regex a command string in prose. A regex cannot tell a
  recommendation from a mention. Three of four regex_absent graders
  failed correct responses -- conditionals ("if you confirm this is
  abandoned, I'd run ..."), explicit refusals, and worked examples
  answering the question asked. One briefly produced a headline "+0.20
  uplift" that was pure artifact. One regex grader remains, on headings.
- Never grade a gate-2 artifact. The command prints literal delete
  commands only after the user answers gate 1, which never happens
  headless. A grader looking for them failed the plugin for following
  its own safety protocol while the unaided arm "passed".
- Scores are locale-sensitive: awk honours LC_NUMERIC and emits "8,00"
  under it_IT, which the delta column then subtracts as strings.

Results: 6 of 7 cases show delta 0.00 -- Opus handles the analysis
correctly unaided. The one case that discriminates is 06, where the
unaided arm executed `git branch -d fix/typo` on a bare "tidy it up"
request and destroyed the branch (delta +0.75, verified against repo
state and the tool-call log, not prose).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Fix SC1091 in the eval scripts, and the Makefile gap that hid it

The Lint job failed on plugins/git-cleanup/evals: shellcheck could not
follow either `source` directive.

A relative `source=` is resolved against shellcheck's working directory,
not the script's. Running `shellcheck -x plugins/.../run-evals.sh` from
the repo root therefore looks for ./lib/graders.sh and does not find it.
`source-path=SCRIPTDIR` anchors it to the script's own directory, which
is what the path was relative to all along.

The reason this passed locally is the more useful half. The `shell`
target ran shellcheck with --severity=warning; SC1091 is info-level, so
the filter hid it. The pre-commit hook CI runs is plain `shellcheck -x`
with no filter, so `make check` could not catch this class of failure at
all -- contradicting the promise at the top of the Makefile that every
target mirrors a CI job.

Dropped the filter so the two match. The repo is already clean under the
stricter args, so this costs nothing today and closes the gap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* git-cleanup: address review findings from hbrodin

Blockers:
- `git branch -d` is not the backstop the SAFE_TO_DELETE comment claimed. It accepts
  a branch merged into HEAD *or* into its own upstream, so a branch level with its
  remote but never merged to the default branch deletes cleanly under -d. That was
  the only delete category with nothing behind it. Each candidate now carries a
  `verifyWith` — `git merge-base --is-ancestor <tip> <default>` — that the main
  session runs immediately before the delete, and the evidence names the tip commit
  so the claim is checkable rather than asserted.
- PROTECTED covered four names. `staging`, `production`, `dev` and `hotfix/*` all
  reached the delete list, with force-delete and an empty needsReview on the
  remote-gone path — which is precisely how those branches fail, their remote being
  deleted during a branch-protection change or a repo migration. The list now covers
  long-lived integration and environment branches and matches case-insensitively.

Also:
- Quoting guidance on the agent-facing path said `"$branch"`, under which `$(...)`
  still substitutes. Both copies the subagents read now require single quotes, with
  the `'\''` escape, since `has'quote` is a legal branch name and the agents paste
  literal names rather than expanding a variable.
- The gate-1 audit rule rejected the workflow's own SAFE_TO_DELETE evidence string,
  which would have moved every git-proven merged branch to needs-review.
- `worktreePath` is optional in the schema but load-bearing for delete ordering. The
  join is now derived from the required `worktrees[]` array.
- The investigator's context list was uncapped and replicated into every slice of a
  split cluster: 300 siblings produced a 41 KB prompt that was 98% context. Capped at
  8, ranked to keep tracked siblings, since those are the plausible superseders.
- Untrusted repo text — branch names, commit subjects, and the investigator's own
  evidence field — is now fenced in `<repo-data>` with an explicit data boundary.
  `agent()` takes no tool list and the agents need Bash for git, so the tool-level
  restriction is not available from here; the boundary is stated instead.
- Recorded the `pipeline()` index-alignment dependency the assembly step rests on.

Checkers, so a commands-only plugin is not unverified:
- The validator now checks command frontmatter (parses, has a description, uses
  `allowed-tools:` not `tools:`) — it previously had no references to commands at all.
- A plugin exposing no entry point at all is now an error, so the loadability checks
  cannot pass vacuously at 0 == 0 on a plugin that ships nothing runnable.
- Six new self-test assertions cover both, including that a valid command is accepted
  and that commands alone satisfy the entry-point rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* git-cleanup: lead with the typical agent count, not the ceiling

hbrodin measured a dozen branches spawning three agents, because the
deterministic triage decides most of them without spawning anything.
Eleven was the worst case presented as the headline number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* ci: let the js-tests guard recognise node:test suites

The execution guard demanded a `<n> assertions passed` line, which is
git-cleanup's own convention. semgrep-rule-variant-creator's suites use
node:test and report `<mark> pass <n>`, so the guard failed two honest
suites for using the other format — a guard that only knew the format of
the suite it shipped with.

Both formats now count. The node:test branch does not anchor on `^.`:
that mark is multi-byte, the recipe runs under /bin/sh in whatever locale
the machine has, and `.` matches a single byte in the C locale — which
matched interactively and failed under make.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* git-cleanup: address the automated review's findings

The nine hbrodin threads were already handled in c5358a1; these are the
github-actions review's, which were not.

Correctness:

- `verifyWith` now names `refs/heads/<branch>` rather than the tip sha the survey
  agent reported. The agent joins `branch -vv` and `branch --merged` into one row
  itself, so a transposed or stale `lastCommit` could carry a sha that IS an ancestor
  of the default branch while the branch is not — the precondition would then pass on
  a branch it never examined, and `-d` accepts it too. A refname cannot desynchronise
  from the branch it names. This also removes the `--is-ancestor (unknown) main` bash
  syntax error when `lastCommit` is empty; the evidence now says so in words.
- Both refnames in `verifyWith` are single-quoted through a new `sq()` helper, with the
  `'\''` escape. Refnames may legally contain `$(...)`, backticks and `'` — only a
  space is refused — and this is the one place the workflow builds a shell command for
  the model to paste, so it now meets the bar the command file sets for the agents.
- `g_no_destructive_command_run` missed `branch --delete`, `push -d`, `update-ref -d`,
  and anything behind another global option (`git -c …`, `git --git-dir=…`). A run that
  deleted a branch by any of those spellings scored PASS from the grader whose only job
  is to notice. Global options are now consumed generically and both spellings of every
  delete flag are matched; the self-test covers all of them plus three non-delete pushes
  that must still pass.
- `g_all_branches_mentioned` returned PASS on an empty manifest — the repo's own named
  anti-pattern. It now ERRORs, with an assertion proving it.
- `make-repo.sh` claimed reproducible shas while inheriting the caller's git config.
  `eval-self-tests` is in `make check`, so a developer with `commit.gpgsign = true`
  would have had the whole build block on a passphrase. GIT_CONFIG_GLOBAL/SYSTEM are
  pointed at /dev/null and hooks/signing disabled per-repo. The pinned `3fcf672`,
  `64b5c2a` and `a2f470c` are unchanged.
- The command file handed the model a literal `${CLAUDE_PLUGIN_ROOT}` with nothing to
  expand it, and documented recovery for two failures but not that one. It now resolves
  the root first and treats an unreadable scriptPath as a fall-through to the inline
  path rather than an abort.

Docs that contradicted the code:

- git-cleanup README stated the `git branch -d` safety rationale this PR exists to
  disprove, and never mentioned `verifyWith` — a maintainer reading it would have
  dropped the precondition as redundant. Its gate-2 example showed an unguarded
  `git branch -d` too, and its protected list named four of ~25 names.
- merge-evidence.md said "Git proved it; nothing further is needed" for the one
  category that now carries a precondition, and referred to "the skill's" fallback.
- evals/README.md credited `analyze-branches.test.mjs` with covering gate-2 prose it
  does not read.
- Makefile said CI scopes the validator to touched plugins. It does not — only the
  version-increment check is scoped; AGENTS.md had it right.
- The context-ranking comment claimed recency; the sort key is tracked-ness only and
  the schema carries no date to sort on.
- A dead `grep -v` in the self-test, overwritten by the next line.

Not addressed: the Codex entry-point gap (`commands/` and `workflows/` are not
Codex-supported components, so git-cleanup has no invocable entry point there). That is
a maintainer call about plugin shape, not something to decide inside this PR.

Suites: 47 JS assertions, 49 eval self-test assertions, 53 validator assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* git-cleanup: report protected branches, and normalize the refs the survey reports

Both findings from hbrodin's second pass. Both were in the deterministic core, so both
are pinned by tests rather than argued about.

**Protected branches vanished from the report.** The filter ran before triage and
`report()` only read `settled`/`investigated`, so a protected branch landed in no output
array at all — `staging` carrying seven unpushed commits was simply absent, and Safety
Rule 7 ("a partial run must not read as a complete one") had nothing to fire on. Never
deletable and never mentioned are different guarantees; only the first was wanted. They
now travel to `report()` and come back under `keep` with category `PROTECTED`, evidence
naming why they were excluded and their unpushed count when they have one. An unpushed
count on a protected branch is logged as well.

Also took the second half: `test`, `testing`, `demo`, `sandbox`, `latest` and `default`
are out of the regex. They are not environment branches, they are the throwaway local
names this tool exists to clean up, and with `/i` the list took `Test` and `Demo` too.
Over-protection is not free just because it errs safe — a branch this tool refuses to
touch has to be deleted by hand.

**`defaultBranch` arrived as a remote ref.** `git symbolic-ref refs/remotes/origin/HEAD`
prints `refs/remotes/origin/mainline`, not `mainline`, and the prompt did not pass
`--short` nor did the schema say which form it wanted. The name comparison therefore
missed, and a repo whose default branch is outside `PROTECTED` saw its own trunk on the
delete list — with `verifyWith` returning 0, since `git branch --merged
refs/remotes/origin/mainline` still lists `mainline`. The command file's inline fallback
already normalized (`--short`, then `${default_branch#origin/}`), so the two analysis
paths disagreed with each other. Fixed with a `localName()` applied to both
`defaultBranch` and `currentBranch`, `--short` in the survey prompt, and a `description`
on both schema properties. `currentBranch` had the same exposure and was failing safe
only because `git branch -d` refuses the checked-out branch.

Tests: 61 JS assertions, up from 47. Four cases added — the three reported spellings of
`defaultBranch` each protecting the trunk, a fully qualified `currentBranch`, a protected
branch with unpushed work surviving into `keep`, and the trimmed names being analyzable
again. Three existing assertions changed from "absent everywhere" to "absent from the
delete paths, present under PROTECTED".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: kz-tob <kara.zaffarano@trailofbits.com>

2026-08-24 10:54:33 -04:00

..

analyze-branches.test.mjs

git-cleanup: convert the skill to a command driving a dynamic workflow (#217)

2026-08-24 10:54:33 -04:00