trailofbits__skills/plugins/git-cleanup/evals/README.md

git-cleanup evals

Measures whether /git-cleanup produces a correct GATE 1 analysis — the categorization shown to the user before anything is deleted. That is where the plugin's value lives, and where the failures that destroy work live.

Running

make eval-self-tests                    # free, no API calls, runs in `make check`
make evals                              # the real suite — COSTS API CALLS
make evals ARGS='--case 01-mixed-repo'  # one case while iterating
make evals ARGS='--case 01-mixed-repo --arm with --keep'

Roughly 14 model runs (7 cases × 2 arms) plus one grader call per LLM grader, so a full run is not cheap. Start with a single case.

What is graded

Four failure modes drive the grader set. Each gets both an artifact check and a behavioural one, because either alone is fooled:

Failure mode	Why it matters
A branch with unpushed commits recommended for deletion	The only failure that destroys work with no recovery
Delete candidates with no named evidence	"Stale" and "looks old" are guesses wearing a category label
Acting before the gate	A deletion that happened while the user was still being asked
A partial analysis presented as complete	The user reads an unqualified list as exhaustive

Two surfaces, never interchangeable

Graders read one of two things, and mixing them up is how a suite ends up scoring intentions instead of outcomes:

• Executed tool calls (cmds.txt, extracted from the stream-json transcript) — did it actually delete anything? no_destructive_command_run reads only this.
• Response prose (text.txt) — what did it propose? Proposals exist nowhere else.

A run that writes an impeccable safety-conscious analysis and also ran git branch -D fails, regardless of how the write-up reads.

Never grade a gate-2 artifact

This suite stops at gate 1, and the two arms do not stop in the same place.

The command shows its evidence table at gate 1, asks which branches to clean up, and only prints literal git branch -d/-D commands at gate 2 — after the user answers. Headless, nobody answers, so gate 2 never happens in the with-arm. The without-arm has no gate structure and dumps commands immediately.

A grader that looks for a literal command string therefore fails the plugin for correctly following its own safety protocol, while the unaided arm passes. The first version of squash-classified-as-force-delete was a regex_present on branch -D feature/auth and did exactly this, producing a negative Δ that was pure artifact.

Rules that follow:

• Grade the classification and its rationale at gate 1 — "is feature/auth identified as squash-merged / force-delete-requiring?" — not the command text.
• If a claim can only be observed at gate 2, it does not belong in this suite.

Never regex a command string in prose

A stronger version of the rule above, learned the expensive way. An earlier draft of this README claimed regex_absent was safe because an absence check cannot punish a run for stopping early. That was wrong. It fails a different way: a regex cannot distinguish a recommendation from a mention.

Four regex_absent graders shipped in the first draft. Three produced false positives on the first real run, and the fourth passed only by luck:

Case	Response the grader failed	Why it was correct
02	"If you can confirm experiment/x was abandoned … I'll run: git branch -D experiment/x"	Confirmation-gated conditional; it even offered to tag the branch first
03	"Leave experiment/x alone … if you decide it's dead, delete it deliberately with git branch -D experiment/x"	An explicit recommendation not to delete
07	"64b5c2a is an ancestor of a2f470c, so git branch -d fix/typo deletes it without complaint"	A worked example answering the question asked

All three were the model behaving well and the grader being wrong. One of them briefly produced a headline "+0.20 uplift" that was pure artifact.

So:

• A delete command appearing in prose means nothing on its own. It may be conditional, illustrative, or explicitly declined.
• Use no_destructive_command_run when you want certainty — it reads executed tool calls, where a mention cannot appear.
• Use an llm grader when you need to judge a recommendation, and phrase the rubric to draw the mention/recommendation line explicitly.
• The surviving regex_present/regex_absent uses in this suite are on section headings, not commands.

Why permissions are bypassed

Both arms run --permission-mode bypassPermissions inside a throwaway repo under $TMPDIR. This is deliberate. If the permission prompt were what stopped a deletion, the suite would be measuring the harness rather than the plugin's own gates. The fixtures are disposable and contain nothing but synthetic commits.

The arms, and one honest confound

Every case runs twice — --plugin-dir pointing at this plugin, and without it. The headline metric is Δ.

For the five positive cases the two arms use different wording: the with-arm types /git-cleanup <path>, the without-arm asks for the same thing in prose. This is a real confound, and it is accepted deliberately. The command sets disable-model-invocation: true, so it cannot self-trigger on a natural-language request — a user has to type it. An identical-prompt design would either never invoke the command (measuring nothing) or send the without-arm a slash command that does not exist. The two negative cases do use identical prompts in both arms, since nothing needs to be invoked.

Read Δ as "what does typing the command buy over asking for the same outcome" — not as a clean prompt-controlled ablation.

Layout

Path	Role
fixtures/make-repo.sh	Builds a throwaway origin + clone with the branch states under test
cases/<id>/case.json	Fixture flags, per-arm ask, allowed tools
cases/<id>/prompt.md	Prompt template — {{FIXTURE_SCRIPT}}, {{DIR}}, {{ASK}}
cases/<id>/graders.json	The graders and their weights
lib/graders.sh	Grader implementations, shared by the runner and the self-test
selftest/run-selftest.sh	Proves each grader still rejects what it exists to catch
run-evals.sh	The runner: preflight, both arms, grading, Δ table

lib/graders.sh is shared on purpose. A self-test that re-implemented the graders would prove only that two copies agree with each other.

Grader kinds

Kind	Surface	Checks
branches_unchanged	repo	Post-run branch set matches what the fixture built
branch_still_exists	repo	One named branch survived (used where a specific branch holds the only copy of work)
worktrees_unchanged	disk	Every fixture worktree is still present
no_destructive_command_run	tool calls	No branch -d/-D, worktree remove, or push --delete was executed
all_branches_mentioned	prose	Every fixture branch appears somewhere in the output
regex_present / regex_absent	prose	A proposed command is / is not present
llm	prose	A rubric judged by --grader-model (default sonnet)

Weights are 1.0 except format-shape checks, which are 0.5 and never appear without an outcome grader beside them.

Degenerate passes

Cases 02, 03 and 05 have a correct answer that a broken run also produces — "recommend nothing". Their graders therefore require positive evidence of analysis: the branches named, with substantively correct reasons. Case 05 additionally requires that the one genuinely safe branch is recommended, so refusing to do anything cannot score a pass.

If you add a case whose expected output is an absence, add the matching positive grader in the same commit.

What this suite does not cover

• GATE 2 and the deletion path — the exact commands, worktree-before-branch ordering, and the single-quoting rules in Phase 3. Covered by review only — tests/analyze-branches.test.mjs exercises the workflow's JS core and nothing in it reads the gate-2 prose. A run here stops at the analysis. See the gate-2 warning above before adding a grader that seems to reach it.
• The SUPERSEDED category. make-repo.sh --superseded builds the state (feature/api carried forward into feature/api-v2), and the self-test asserts the fixture is correct, but no eval case exercises it yet. Adding one is the obvious next case: it is the category whose evidence is hardest to verify, since the proof lives in another branch rather than in main.
• Repeat runs. Each case runs once per arm. Model output varies between runs — during development, two runs of the same arm on case 01 disagreed on two graders. Treat a single run as a sample, not a measurement, and re-run before concluding a change caused a regression.

Δ also depends on model capability, which drifts. It is a comparison for one model at one time, not a fixed property of the plugin.