nehal-a2z
929b0ec6da
* fix: align review skills with CLI 0.7 scopes * docs: rely on review auto-auth * fix(skills): match current CLI review and configuration contracts * test(skills): add offline CLI behavior evaluations * test(skills): assert saved PR prompt output format * fix(skills): clarify untracked inclusion with default review scope
CLI behavior evaluations
Six offline cases exercise public scope flags, default and uncommitted untracked-file inclusion, local versus PR prompt retrieval, EU browser authentication, and incomplete/skipped review output. No shell, writes, network tools, or production reviews are granted.
With Claude Code 2.1.269+ and an authenticated account, run from the plugin root:
claude plugin eval . --tag cli-parity --runs 1 --ablation with-without --no-publish --max-cost-usd 10 --keep-temp
Pin --model for comparisons. Positive skill activation is diagnostic and does
not contribute to the outcome score. Deterministic graders check specific command
contracts; advisory LLM graders are with-only and excluded from the ablation
score. Inspect the actual answers and retained transcripts: regex checks and LLM
judges do not establish complete semantic correctness. One run per arm is a smoke
evaluation, not a reliable effect-size estimate. Results stay under ignored
evals/results/; do not commit account metadata or private source provenance.
See the official evaluator documentation.