kz-tob
4b1b74b181
* Give differential-review a trigger, and name every component in its README differential-review's description listed what it does and never named a situation, so it competed on capability wording alone. It now closes with the triggers its own README already documents — reviewing a PR, commit, or diff; checking whether a change re-introduces a fixed bug; asking what else a change could break; finding modified code with no test. The same plugin's README never mentioned adversarial-modeler, which is what Phase 5 dispatches for HIGH RISK changes. Checking whether that was isolated turned up more of it, and the sweep found three kinds of gap: zeroize-audit's agent table was missing three of its eleven agents — 0-preflight, which gates the entire run, plus 5b-poc-validator and 5c-poc-verifier. All three appear in the phase diagram directly above the table, which is why they read as present. constant-time-analysis documents the ct-analyzer CLI end to end and never says the plugin also ships a skill and a command. entry-point-analyzer lists phrases that trigger its skill but never names the skill or its command. Three more READMEs describe their skill without naming it. That matters most where the skill name is not the plugin name and a user cannot guess it: chrome-mcp-troubleshooting and interpreting-culture-index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix review findings and make the README sweep a gate The two PoC rows I added to zeroize-audit said Phase 4. The diagram three lines above them, SKILL.md, and workflows/phase-5-poc-validation.md all say Phase 5, steps 5a and 5b. "Wave 5a" is a label that exists nowhere. A debugger consulting the table — the artifact this branch designates as what runs when — would have opened phase-4-poc-generation.md and found no validation in it. Also corrected the sentence introducing that table, which still said 10 agents across 8 phases against 11 across 9, and the Phase 0 diagram line, which still credited the orchestrator for a gate the new row credits to 0-preflight. differential-review's README claimed the agent is "dispatched", and named it bare in a column whose other rows are namespaced. Nothing dispatches it: the only instruction is prose in SKILL.md, and a bare subagent_type fails at runtime. Namespaced both, and corrected the five stale line counts in the same file — reporting.md is 369 lines, not the ~120 the token-efficiency section budgets for. Drop the dead `name: trailofbits:<cmd>` key from five command files. The three newest command files carry no name: at all, #275 namespaced 22 bare invocations, and this branch documents the `/<plugin>:<cmd>` form — so the key contradicts the docs it sits next to. Then make the sweep repeatable. Doing this by hand three times found eight gaps and missed two more, both of the same shape: a workflow ships under meta.name, not its filename, so a README citing the filename never writes the name a reader types. The validator now checks that a README names every skill, agent, command, and workflow its plugin ships, reading meta.name for workflows. It refuses a run that inspected zero components, and six self-test assertions hold it to known-bad fixtures. It found git-cleanup on its first run: ships as /git-cleanup:git-cleanup-analysis, README cites workflows/analyze-branches.js four times and that name never. static-analysis had the same gap for codeql-build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix both P2s: the gate was a substring test, and the dispatch was still bare The README gate ran `name not in text`. That reads as thorough and could not fail for a large share of what it counted: `draw` was satisfied by "(draw cards instead)", `semgrep-rule` by the plugin's own name in the install line, `burp-search` by a `scripts/burp-search.sh` path that is a different thing, and `audit` by the prose "shared-state struct audit". Match by kind instead. Commands and workflows are reachable only as `/<plugin>:<name>`, so require that literal — it is the only string a user can type. Agents are dispatched by identifier and never typed as prose, so require an identifier-shaped mention. Skills are genuinely referred to by bare name, so require only a delimited occurrence, which is what stops "draws" counting as `draw`. That surfaced seven real gaps, the four above plus insecure-defaults' audit-pipeline workflow, mutation-testing's skill, and trailmark's code-slice-worker. All seven fixed. adversarial-modeler was still bare at SKILL.md:96. Line 77 was the decision-tree mention; line 96 is the "Delegate to this agent" instruction a model actually acts on, so the runtime failure the last commit claimed to fix survived it. Namespaced, and it now says why. Also from the review: a per-kind floor, since a single total stays healthy while skill_files() — 63% of coverage — silently stops matching; workflow_names anchored to the meta block, because a bare search takes any earlier `name:` in a comment, and .mjs was invisible; and AGENTS.md documents the new hard failure. Self-test 88 -> 96, each new rule with a negative control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.6 KiB
Insecure Defaults Detection
Audits a codebase for insecure default configuration, tracing each candidate before reporting it.
Install
/plugin install insecure-defaults # marketplace
/plugin install ./plugins/insecure-defaults # local checkout
Use
/insecure-defaults:audit # whole repo
/insecure-defaults:audit src/ # subtree
/insecure-defaults:audit src/config/app.py # one file
Argument is a file or a directory. Optional; defaults to ..
Whatever you point at is the target, however test-like it looks: a run scoped to tests/ audits the tests. Exclusions (fixtures, docs, vendored code) apply only outside the scope you named.
Use the command, not the workflow. Invoking
insecure-defaults:audit-pipeline directly stops immediately. No fallback: if the
corpus can't be read, the run errors rather than guessing.
What it finds
| Category | Example | Corpus |
|---|---|---|
| Fallback secrets | SECRET = env.get('KEY') or 'dev' |
fallback-secrets.md |
| Default credentials | seeded admin / admin123 |
default-credentials.md |
| Fail-open switches | getenv('REQUIRE_AUTH', 'false') |
fail-open-security.md |
| Weak crypto | hashlib.md5(password) |
weak-crypto.md |
| Permissive access | ACL='public-read', 0o666, CORS * |
permissive-access.md |
| Debug leakage | traceback.format_exc() in a response |
debug-features.md |
Each category is three files that must agree:
workflows/audit.js |
an { id, title } row, all the script knows. Ships as /insecure-defaults:audit-pipeline |
references/<id>.json |
title + seed patterns |
references/<id>.md |
Report when / Skip when, plus worked vulnerable/secure pairs |
The sweep agent loads both files for its own category and no others. It has to be the agent, not the script: a workflow has no filesystem access. tests/seed-coverage.js checks the three correspond, since nothing at runtime can.
Candidates come in two shapes, judged differently:
- Configurable: a lookup with a fallback. Only a bug if the app runs with it.
env.get('K', 'x')does;env['K']crashes instead, so it's fine. - Unconditional: no configuration anywhere, insecure as written. About half of all findings. A missing env var is not grounds to refute one.
How it runs
| Phase | Agents | Model | Does |
|---|---|---|---|
| Recon | 1 | Sonnet | Classify scope, profile stack + deploy manifests |
| Discover | 6 | Sonnet | One sweep per category, in parallel |
| Verify | N | Opus | Refuting agents batched by category, ≤16 findings each |
| Report | 1 | Sonnet | Severity, remediation, coverage |
Between Discover and Verify: dedup keyed on category:file:line, so the rule id prefixes the path. Two patterns in one category hitting the same line collapse; the same line flagged by two different categories stays as two candidates. hashlib.md5(k) can be a real weak-crypto finding and a false permissive-access match at once, and one merged verdict would have to cover both readings.
Verify then batches by category, ≤16 findings per agent. Each agent reads exactly one corpus and applies one discriminator. A category with more than 16 findings is split across several agents, so no single agent can run past the tool-call cap and return a partial verdict list. Coverage is uncapped; only per-agent size is.
Sweeps collect and don't judge: a sweep only greps, so it files candidates without classifying them and the verifier decides with the file in context. Each verifier starts at refuted: true and stops at the first step that kills a candidate:
- Is the file reachable in production?
- Is the insecure value the one that runs? Configurable → does it fail-secure instead? Unconditional → this step can't refute it.
- Is the value actually insecure?
- Does it reach a security decision? Cite the sink.
- Does deployment always supply the var? Configurable only, and no answer refutes: every manifest setting it lowers severity, none is the CRITICAL case, and a partial or undetermined answer counts as reachable.
Incomplete trace = refuted. If the corpus can't be found, the run aborts rather than continuing without it.
Each sweep reports whether it could actually read its corpus, and one failure aborts the run.
Patterns
Seed patterns are a floor, not the search.
Recon reports the project's own config wrappers, flagged if they can return a default. Each sweep then derives patterns for the detected stack: framework keys, language idioms (ENV.fetch, System.getProperty(k, d), ${VAR:-default}), and manifest formats (default = in HCL, ENV in a Dockerfile). A codebase reading everything through get_setting("X", "default") barely matches the generic seeds.
Sweeps report seed patterns and derived patterns separately. Any sweep whose derived list is empty only looked for generic idioms, and is named in the report as a coverage gap.
Seeds are POSIX ERE: [[:space:]] and [0-9], never \s, \d or \b. Some grep builds silently fail to match those, and a pattern that matches nothing is indistinguishable from a clean result. tests/seed-coverage.js rejects them.
To add a category: a row in CATEGORIES in workflows/audit.js, plus <id>.json and <id>.md in references/.
When not to use it
- Semgrep or a linter fits better: fixed pattern, no reachability question.
- You want extensive secret detection: use gitleaks/trufflehog for committed credentials.
Tests
node tests/harness.js workflows/audit.js
node tests/harness.js workflows/audit.js --self-test
node tests/seed-coverage.js .
Details in tests/README.md. Whether the prompts work on a real model needs a live run.