Files
Dan Guido 488d37c71c Collapse the review into one xhigh opus tier, and stop invented evidence (#256)
* Pin the review model, raise effort to xhigh, stop invented evidence

Three problems with the automated review, found while auditing what was
actually reviewing PRs #253-#255.

**The model was never specified.** `claude_review.sh` passed `--effort`
but no `--model`, so the reviewer was whatever the CLI happened to
default to at run time, and nothing in the logs said which. The CLI
version was pinned to stop CI changing without a commit while the larger
lever on review quality was left floating. Both tiers now pin `opus`, and
the run line records model, effort and resolved CLI version.

**The CLI is now deliberately unpinned**, in both this workflow and the
loadability check, so they track releases as they ship. For loadability
that is the point: the version worth proving plugins load against is the
one users run. The logged version is what makes a surprising result
attributable after the fact.

**The review claimed to run things it cannot run.** Its allowlist is
`gh pr` reads plus Read, Grep and Glob — no interpreter, no test runner.
Across three PRs it reported a Python snippet it had "verified directly"
whose regex cannot compile, a pytest suite it had "run locally" with a
pass count that does not match reality, and a shell script three
"independent verifiers" had supposedly executed. Every conclusion was
correct and every proof was fabricated. The deep prompt already said "you
cannot execute anything"; the prompt that actually runs never did. That
statement moves into the shared prompt, extended to forbid reporting
output no tool produced.

Effort goes low -> xhigh on the tier that runs on every push, so the
strongest review is the default rather than something to remember to ask
for. The shared prompt also now says to rank on consequence rather than
diff size, after a one-character fault that made a checker miss its own
target was filed as a nit.

The `fast` name is kept: it is the check name branch protection matches
on, and it describes the trigger rather than the effort.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Collapse the two review tiers into one

The split was a cheap pass on every push plus an xhigh adversarial pass
behind a `deep-review` label. It did not survive contact: the label was
never created, so the deep tier skipped 9 times and ran zero, and every
review this repository has ever received came from the cheap tier.

With both tiers now on opus at xhigh, the only thing left separating them
was scope, and there was no reason to gate the better scope behind a
label somebody has to remember to apply. So there is one job. It takes
the deep tier's prompt and tools — reads beyond the diff, gets git
history, checks the PR's claims against the files — and the cheap tier's
`--edit-last` posting, which matters now that it runs on every push.

`fast` is gone from the check name, the script's tier argument, and the
concurrency key. The name was already inaccurate once effort went to
xhigh, and the ruleset on main requires license/cla, Validate, Pre-commit
and bats — not this check — so nothing depended on it. The elaborate
label-keyed concurrency group goes too; it existed only to stop a push
cancelling an in-flight deep review, and there is no second tier to
collide with. Timeout follows the deep tier at 30 minutes, since xhigh on
a large diff needs the room.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Address the review of this PR

The new xhigh reviewer found real problems in its own configuration.

**The API key was in the unpinned install step's environment.** It was
there only to satisfy `if: env.ANTHROPIC_API_KEY != ''`, since `secrets`
is not available in a step-level `if:`. Combined with `@latest`, that
runs npm lifecycle scripts from a version nobody chose beside an
org-wide credential. Presence is now reduced to a boolean in its own
step, and the install step holds no secret at all — which is what makes
@latest acceptable there.

**The posting guard counted comments it did not write.** It matched any
issue comment in the window, so a maintainer replying to the previous
review could stand in for a review this run never posted: green check,
no review. Measured on this PR's siblings — the old query counts 2 on
#253 and #255, the new bot-only query counts 1. Also paginated, since
the unfiltered call would miss a review past the first 30 comments.

**`AGENTS.md` reaches the reviewer and tells it to run things.** CLAUDE.md
is just `@AGENTS.md`, so it loads as project instructions telling the
model to run `make check`, run `prek run -a`, and consult a
`claude-code-guide` subagent — none of which it can do. That is the
fabrication channel this PR exists to close, arriving by a route the
prompt did not address. The prompt now names those files and says they
are addressed to someone else.

**validate.yml goes back to a pinned CLI.** Unpinning it was my
extension, not what was asked, and the risk is misplaced: that job is a
required check, so an upstream release renaming a field in
`plugin list --json` reddens every open PR at once and blocks merges
with no commit to explain it. The review job can go red harmlessly. This
also re-aligns Codex and Claude, both pinned there again, and keeps
dependabot.yml's note about "the pinned npm CLI versions" accurate.

Smaller: the prompt described its allowlist as "only `gh pr` reads" while
mandating `gh pr comment`, a write, and granting `git log`/`git diff`.
And the comment over `MODEL` claimed an attributability the `opus` alias
does not provide, since it tracks new Opus releases by design.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Unpin the Claude Code CLI on the loadability check too

Reverses the revert two commits back, deliberately and with the argument
against it on the record.

The review made the case for pinning here: `Validate plugins and skills`
is a required check, so a Claude Code release that breaks plugin loading
reddens every open PR at once with no commit to explain it. That reading
is correct about the mechanics and wrong about which failure costs more.
A pin nobody remembers to bump drifts until CI is proving loadability
against a version no user runs, which is the one thing this check exists
to establish — and it fails silently, by passing. The loud break is the
preferable failure, and it is now the chosen one rather than an oversight.

Dependabot does not track npm CLIs installed this way, so the real choice
was a live version or a stale one, never a maintained one.

The Codex CLI beside it stays pinned at 0.146.0. Same class of manual pin
and arguably the same argument applies, but unpinning another vendor's
CLI was not asked for and would widen this change past its subject.
dependabot.yml's note is corrected to match: singular, and naming which
one is pinned and why the other is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 21:29:51 -04:00

109 lines
5.1 KiB
YAML

# Automated review on pull requests.
#
# An organization-level ANTHROPIC_API_KEY is visible to this repository, so this
# workflow is LIVE the moment it merges — it will start reviewing PRs immediately.
# (It is not listed by `gh secret list` at the repo level, which is what made this
# easy to get wrong.)
#
# Every review step is still gated on the key being present, so if the org secret is
# ever removed these jobs go quiet rather than red. A review workflow that fails on
# every PR for want of a credential teaches people to ignore red checks.
#
# One tier, deliberately. This was two — a cheap pass on every push and an xhigh
# adversarial pass behind a `deep-review` label — and the split did not survive
# contact: the label was never created, so the deep tier skipped 9 times and ran
# zero, while every real review came from the cheap tier. Rather than wire up a
# label to remember to apply, the default is now the strong review.
#
# Fork PRs get NO automated review. A `pull_request_target` variant was written and
# then deleted: it checked out the fork tree and ran this repo's review script from
# it, so the script itself — and any CLAUDE.md or .claude/hooks beside it — was
# fork-authored and executed with the org API key in the environment. The SHA pin
# and the tool allowlist did not touch that path. Doing it safely means checking
# out base into the workspace and the fork commit into a subdirectory, running
# everything from base; that is a deliberate piece of work, not a footnote to this
# change.
name: Claude Review
on:
pull_request:
# ready_for_review is load-bearing: the job skips drafts, so without it a
# draft-then-ready PR gets no event and no review at all.
types: [opened, synchronize, reopened, ready_for_review]
# Granted per job rather than here. A workflow-level write grant applies to every job
# including ones that never need it, which zizmor flags as excessive-permissions.
permissions: {}
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
review:
name: Claude review
# Skips drafts, bots, and forks. Forks are excluded because under `pull_request`
# they get a read-only token and could not post a comment anyway, and the
# privileged alternative is unsafe (above).
if: >-
github.event.pull_request.draft == false &&
github.event.pull_request.user.type != 'Bot' &&
github.event.pull_request.head.repo.fork == false
runs-on: ubuntu-latest
# xhigh on a large diff needs room. The old cheap tier ran in 20.
timeout-minutes: 30
permissions:
# read: check out the diff under review.
contents: read
# write: post the review as a PR comment. This is the whole point of the job —
# a run that cannot comment produces a green check and no review, which reads
# as "reviewed and clean".
pull-requests: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# Needs history to diff against origin/main. At depth 1 the diff comes back
# empty, which is indistinguishable from "nothing to find".
fetch-depth: 0
persist-credentials: false
# The key is reduced to a boolean here so that no later step needs it in scope
# merely to test for it. `secrets` is not available in a step-level `if:`, so
# gating on presence otherwise means putting the secret in that step's `env:` —
# which for the install step below would run npm lifecycle scripts, from an
# unpinned version nobody chose, beside an org-wide credential.
- name: Check for API key
id: key
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
if [ -n "$ANTHROPIC_API_KEY" ]; then
echo "present=true" >>"$GITHUB_OUTPUT"
else
echo "present=false" >>"$GITHUB_OUTPUT"
echo "ANTHROPIC_API_KEY is not visible here; skipping the review." >&2
fi
# Deliberately unpinned: the reviewer should get CLI fixes and features as they
# ship. The trade is that a bad release can change review behaviour with no
# commit here, so `claude_review.sh` logs the resolved version and model on every
# run — that is what makes a surprising review attributable afterwards. The model
# itself IS pinned, in that script; it is the larger lever on review quality.
# Note this step holds no secret, which is what makes @latest acceptable here.
- name: Install Claude Code CLI # zizmor: ignore[adhoc-packages]
if: steps.key.outputs.present == 'true'
run: |
npm install --global --prefix "$RUNNER_TEMP/claude-cli" \
"@anthropic-ai/claude-code@latest"
- name: Review
if: steps.key.outputs.present == 'true'
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO: ${{ github.repository }}
run: |
export PATH="$RUNNER_TEMP/claude-cli/bin:$PATH"
.github/scripts/claude_review.sh