mirror of
https://github.com/trailofbits/skills.git
synced 2026-09-14 14:28:48 +08:00
Dan Guido
8ea3b6a700
* Add validator self-test, structural checks, and make check The repo documented ~53 rules in AGENTS.md and machine-enforced 6 of them. This closes the gap for the ones a machine can decide, and adds the guard that keeps the checkers honest. New error-level checks (all currently pass, so none of this blocks anyone today): agent files must use `tools:` while skills use `allowed-tools:` (the loader silently ignores the wrong key, so the restriction just does not apply); subagent_type must be namespaced or the dispatch fails at runtime; plugin dir names kebab-case and <=64 chars; plugin README present, listed rather than stat'd so `Readme.md` fails on Linux CI the way it should; semver format; the forbidden runtime sidecars AGENTS.md already banned but nothing checked; and version-increment against the base branch, which is the gap that letdebfb29ship an allowed-tools fix across 25 plugins that no installed user received. New warning-level checks, reported but not blocking: the two required SKILL.md sections, the 500-line limit, and unresolved relative references. 55 warnings across 40 plugins today, concentrated in testing-handbook-skills and building-secure-contracts. The point of the exercise is the three anti-vacuity guards. A checker that has silently stopped matching reports a clean repo forever, and that failure mode has shipped repeatedly: `--self-test` builds fixtures and asserts every checker rejects a known-bad one; a full scan that resolves zero references exits 1 rather than declaring everything clean; and SELF_TEST_MINIMUM fails the self-test if it runs fewer than 20 assertions, because the self-test is itself a checker. It currently runs 26. The reference extractor skips fenced and inline code so that skill-authoring docs citing example paths do not generate warnings nobody reads. Makefile mirrors CI as one `make check`. Its RUFF_VERSION must match the ruff-pre-commit rev, and the self-test asserts that — verified by breaking it. * Fix CI checks that could pass or fail without inspecting anything Four of these are the same bug in different places: a check whose empty case is indistinguishable from success. - python-tests ran `python3 <file>` per file. A test file with no `if __name__ == "__main__"` block exits 0 having run nothing, which reads as a pass. All 10 current files happen to comply; nothing enforced it. Now pytest per directory, run via `python3 -m` from inside each one to preserve the sibling imports these suites rely on, with --import-mode=importlib because c-review and rust-review both ship scripts/test_split.py and the default import mode collides on the basename. 278 tests now run where the count was previously unknown. - bats used --no-run-if-empty, so a broken glob was a silent pass. This repo ships bats suites; finding none is a failure. - The SKILL.md frontmatter walk printed "All 0 SKILL.md files have valid frontmatter" if discovery broke. Now fails on zero. - zeroize-audit's shell regression suites matched no CI glob and had never run. Also: the validator self-test runs first, before validation. The hardcoded-path grep now covers .sh, .bats, .yml and .toml — test fixtures and install scripts are where absolute paths hide. The personal-email exclusion is anchored; an unanchored '.git' also dropped any line containing '/github'. The two npm CLI installs are pinned rather than @latest, which zizmor flagged and which made CI able to break with no commit. check_codex_loadability.py: tempdir cleanup raced the codex app-server and threw `OSError: [Errno 39] Directory not empty: '.git'`, failing the job after every loadability check had passed (seen on PR #148 today). A teardown race must not be reported as a validation failure. * Wire validators into pre-commit; fix two dead Dependabot entries pre-commit: the three .github/scripts validators were CI-only, so the first signal a contributor got was a red check after pushing — AGENTS.md asked people to remember to run them by hand. Now they run locally, with the validator self-test scoped to fire only when the validator itself changes. Adds actionlint, zizmor, check-toml, check-merge-conflict, and detect-private-key. detect-private-key earns its place: there is no secret scanning here at all, which is how a live API key sat in an untracked config file in a working tree this morning without anything noticing. Markdown linting is deliberately absent, with the measurement recorded in the config so the next person does not have to redo it: markdownlint reports ~12,400 violations across this repo (7,282 MD013 line-length alone). It would land either permanently red or with so many rules disabled that it checks nothing. Dependabot had two defects that made it quieter than it looked. The `pip` entry at / had no manifest to resolve — there is no root pyproject.toml and uv.lock is gitignored — so it reported nothing, indefinitely. And two script directories with real dependencies were uncovered: trailmark's slicing-code-context (trailmark>=0.5,<0.6) and yara-authoring's rule scripts (yara-x>=0.10.0). Switched to the uv ecosystem per the house standard, grouped minor/patch so majors still get their own CI run. * Add automated PR review, inert until a key is configured Two tiers: fast (effort low, every push, sticky comment) and deep (effort xhigh, on a deep-review label). Per-job permissions with an empty workflow-level grant; concurrency keyed per tier so a routine push cannot cancel an in-flight deep review — GitHub will not re-fire `labeled` for a label already present, so that would leave a PR sitting labeled with no review, looking reviewed. The prompt is the substance. It forbids pre-filtering: current models follow "only report high-severity issues" literally, investigating fully and then declining to report what they judge below the bar, which reads as a capability regression but is a prompt bug. It asks for everything ranked P1-P4 with a concrete failure scenario each, and filters downstream. It also names the five defect classes that actually reach main in a repo of markdown that instructs a model, rather than asking for generic code review. There is no ANTHROPIC_API_KEY secret on this repo, so every review step is gated on the secret being present. Until someone adds it these workflows check out the code and do nothing — they do not fail. A review workflow that goes red on every PR for want of a credential teaches people to ignore red checks. Fork review is a separate file with a separate decision attached. It needs pull_request_target, because under `pull_request` a fork PR gets a read-only token and the job could not post its comment at all. The usual exploit path is closed by a maintainer-only label gate, a checkout pinned to the SHA as of the labeling event (so a later force-push does not change what is reviewed), and a tool allowlist with no general Bash, so fork code is read and never executed. The residual prompt-injection risk is documented in the file header. Deleting that one file costs nothing but fork review. * Replace the PR checklist with make check Deletes the 20-item PR checklist. Current model guidance is explicit that verification scaffolding of that shape degrades output rather than improving it, and that removing it costs no capability — so the response is to move each check into machinery, not to restate it louder. Every item that could be mechanized landed in the validator first, with a fixture, before this prose was allowed to go. What replaces the checklist is two lists: what the validator enforces so you do not have to, and what no tool can check. The second list is the one that matters — whether the description actually triggers, whether the version bump is the right size, whether the README row is in the right section (the validator only checks the plugin appears somewhere in that file). Also corrects the frontmatter example, which showed `allowed-tools` as a YAML list. Every SKILL.md in the repo uses the space-delimited form;debfb29converted them deliberately. The doc was the outlier, and it caused a false finding during review of #192. Agent files genuinely do use a YAML list, under the `tools` key — both forms are now shown side by side with the reason the distinction matters. Two new sections. "Scripts a plugin ships" leads with the rule this whole change is built around: a checker that inspects zero items must fail, not pass, with the three worked examples. "Working effectively in this repo" covers effort sweeps, a subagent cap (current models over-delegate, which is a reversal from the previous generation), not adding verification scaffolding to prompts, and not telling a reviewer to pre-filter — that last one reads as a capability regression and is a prompt bug. Scopes the claude-code-guide line to questions that cannot be answered by reading this repo, and adds .opencode/ to the banned sidecar list now that the validator enforces it. * Fix three bugs CI caught in the guardrails themselves All three were mine, and the first is the one that mattered. 1. The version-increment check ran against every plugin, not just the ones the branch touched — so it demanded a version bump from all 42 plugins on a PR that changed no plugin at all. Now scoped to plugins with file changes between the base ref and HEAD. Added three self-test assertions covering it (touched plugin errors, untouched plugin does not), and verified by reverting the fix and confirming the self-test goes red. The check had no coverage before, which is exactly why it shipped broken. 2. The new pre-commit hooks invoked `uv run`, which is not installed in the Pre-commit CI job. The validator declares no dependencies, so plain python3 is correct and one less thing to install. 3. The review workflows called `claude` without installing it. Added a pinned install step, matching how validate.yml installs the same CLI. Also corrects a factual claim in both review workflow headers. I wrote that they were inert for want of an ANTHROPIC_API_KEY, based on `gh secret list` returning empty — but that only lists repository secrets. An organization-level key is visible here, so the fast tier is live on merge. The headers now say so. The deep and fork tiers still cannot fire until their labels are created. * Fix the findings from this PR's own automated review The review posted on #206 found eight issues. Seven were real. Working through them in severity order: P1 — deleted claude-review-fork.yml. It checked out the fork tree and then ran this repo's review script *from that tree*, so the script itself, and any CLAUDE.md or .claude/hooks sitting beside it, was fork-authored and executed with the org API key in the environment. My header claimed "fork code is read, never executed" and that was simply wrong; the SHA pin and the tool allowlist do not touch that path. Doing it safely means checking out base into the workspace and the fork commit into a subdirectory, which is deliberate work rather than a footnote to this change. Fork PRs get no automated review for now, and claude-review.yml explains why. P2 — the prompts instructed `gh pr comment --body-file <file>` while the tool allowlist has no Write and no general Bash, so there was nowhere to put a file. The reviewer hit this and fell back to `--body-file -`; since a give-up does not fail the step, the failure mode was a green check with no review. Now uses a heredoc on stdin. The deep tier also told the model to run the scripts a PR adds, which its allowlist cannot do; it now says so explicitly. While fixing that: both tier heredocs were unquoted so `${PR_NUMBER}` would interpolate, which means backticks in the prompt body were command substitution. Quoted them and moved substitution out to parameter expansion afterwards. shellcheck caught this. P2 — `no-version-bump` was a phantom feature: named in the validator's error message and in AGENTS.md, read by nothing. Implemented via --allow-no-bump, wired to the label through validate.yml. P2 — AGENTS.md claimed both loadability checks run in `make check`. They do not and cannot; they need two CLIs installed. It now says what runs locally, what is CI-only, and how to run the loadability checks by hand. P3 — the new shell-suites discovery reintroduced the exact bug this PR exists to remove: it printed "No shell regression suites found" and exited 0, and used `**` without globstar so it only ever matched one directory level. Now find-based and fails on empty. It is a Makefile target but not part of `make check`, because zeroize-audit's suite pipes to `python3 -` which the modern-python shim rejects — filed as #207. P3 — a failed `git diff` returned an empty changed-plugin set, silently disarming the version check for every plugin. Now raises. P3 — the `/home/user/` exclusion was a content filter applied across all of plugins/, so a skill legitimately documenting that path would be dropped from the results. Scoped to the shim bats fixtures that need it. Not fixed, deliberately: the reviewer noted `_check_ruff_parity` covers one pinned version. True, and it is the only version both files pin. * Fix the second review pass: the review job could not detect its own silence The marquee one: claude_review.sh ended with `claude --print` and nothing else, so if the model finished without calling `gh pr comment` — a denied tool, a hit timeout, or it simply summarising instead of posting — the script exited 0 and the job went green with no review attached. That is defect class 1 from the prompt this very script ships, in the script that ships it. It now timestamps before the run and fails if no comment was created or updated since. _check_ruff_parity returned None (= pass) when the Makefile or pre-commit config was missing, while still counting toward SELF_TEST_MINIMUM. A vacuous pass inside the anti-vacuity harness. Now returns an error string. The scoping claim in AGENTS.md and the Makefile was backwards. I wrote that CI scopes the validator to changed plugins while local scans everything, making local a strict superset. Both scan everything; only the version-increment check is scoped, and it is the one check CI runs that local cannot — so local is not a superset at all. Corrected in both places. README said `make check` "runs everything CI runs" and the Makefile echoed "this is what CI will run". Neither is true: it omits both loadability checks, the version-increment check, and every pre-commit hook except ruff/shellcheck/shfmt. Both now say what they actually cover. AGENTS.md already had this right. Stale count in a pre-commit comment ("these three" preceding two hooks). Accepted without change, with reasoning: the reference resolver matches any file in the plugin, so a link to a file that exists under a different skill resolves when it should not — pinning a base directory produces a flood of false positives, and the looser check is still worth having. The subagent-dispatch check only knows its own plugin's agents, so a bare cross-plugin reference is missed; catching that needs a repo-wide agent map, which is a larger change than belongs here. * Fix the third review pass, including two more silent-pass checks The hardcoded-path step had the defect this PR exists to remove. It was written as `if grep A | grep -v B | grep -v C`, so the exit status came from the last `grep -v`. A first-stage failure — plugins/ renamed, a grep built without -P — produced no output, exited 2, and the step printed "No hardcoded user paths found" while inspecting nothing. Now counts the files it scanned, fails if that is zero, and checks grep's own status via PIPESTATUS rather than the pipe's. The self-test fixture named "empty scan returns non-zero" was passing for the wrong reason: it deleted the plugin's files but left the directory, so scan_plugins_directory still returned {"demo"} and the non-zero came from the missing-README error. The guard it claimed to cover was never exercised. It now rmtree's the directory and asserts the fixture actually emptied plugins/ before relying on it. Same class of bug as everything else here, one level up. Concurrency: every `labeled` event that was not `deep-review` resolved to the `fast` key with cancel-in-progress. Adding `dependencies` mid-review cancelled the running review and then skipped both jobs, leaving a cancelled check, no review, and no event that would re-trigger one until the next push. Each label now gets its own key. The version-increment check read the old version at `base.sha` while changed_plugins diffed `base...HEAD`. Those disagree once anything lands on main after a branch forks, so a PR could fail for not out-bumping a sibling it never saw. Both halves now use the merge base, and the error message says to rebase. Smaller: the sidecar ban only matched `.agents/plugins/marketplace.json` while AGENTS.md bans `.agents/` — widened, with fixtures for it and for `plugins/*/.codex-plugin`, neither of which the harness covered. `make shell` globbed only plugins/, so the one new shell script in this PR went unchecked locally. The deep job's fork exclusion is now explicit rather than relying on secrets being unavailable to fork events. changed_plugins' docstring said working tree; it compares commits.
Trail of Bits Skills Marketplace
A Claude Code plugin marketplace from Trail of Bits providing skills to enhance AI-assisted security analysis, testing, and development workflows. Codex can load this marketplace through its Claude marketplace compatibility.
Also see: claude-code-config · skills-curated · claude-code-devcontainer · dropkit
Installation
Claude Code Marketplace
/plugin marketplace add trailofbits/skills
Browse and Install Plugins
/plugin menu
Codex
Codex supports Claude plugin marketplaces directly, so this repository does not need Codex-specific sidecar metadata.
Install the marketplace with:
codex plugin marketplace add trailofbits/skills
codex plugin list
codex plugin add <plugin-name>@trailofbits
Local Development
To add the marketplace locally (e.g., for testing or development), navigate to the parent directory of this repository:
cd /path/to/parent # e.g., if repo is at ~/projects/skills, be in ~/projects
/plugins marketplace add ./skills
Available Plugins
Smart Contract Security
| Plugin | Description |
|---|---|
| building-secure-contracts | Smart contract security toolkit with vulnerability scanners for 6 blockchains |
| entry-point-analyzer | Identify state-changing entry points in smart contracts for security auditing |
Code Auditing
| Plugin | Description |
|---|---|
| agentic-actions-auditor | Audit GitHub Actions workflows for AI agent security vulnerabilities |
| audit-context-building | Build deep architectural context through ultra-granular code analysis |
| burpsuite-project-parser | Search and extract data from Burp Suite project files |
| c-review | Comprehensive C/C++ security review with clustered parallel workers and SARIF output |
| differential-review | Security-focused differential review of code changes with git history analysis |
| dimensional-analysis | Annotate codebases with dimensional analysis comments to detect unit mismatches and formula bugs |
| fp-check | Systematic false positive verification for security bug analysis with mandatory gate reviews |
| insecure-defaults | Detect insecure default configurations, hardcoded credentials, and fail-open security patterns |
| rust-review | Comprehensive Rust security review covering safe/unsafe boundary, memory safety, concurrency, panic-DoS, FFI, and async runtime with SARIF output |
| semgrep-rule-creator | Create and refine Semgrep rules for custom vulnerability detection |
| semgrep-rule-variant-creator | Port existing Semgrep rules to new target languages with test-driven validation |
| sharp-edges | Identify error-prone APIs, dangerous configurations, and footgun designs |
| static-analysis | Static analysis toolkit with CodeQL, Semgrep, and SARIF parsing |
| supply-chain-risk-auditor | Audit supply-chain threat landscape of project dependencies |
| testing-handbook-skills | Skills from the Testing Handbook: fuzzers, static analysis, sanitizers, coverage |
| trailmark | Code graph analysis, bounded subagent context slicing, Mermaid diagrams, mutation testing triage, and protocol verification |
| variant-analysis | Find similar vulnerabilities across codebases using pattern-based analysis |
| vulnerability-triage-brocards | Triage vulnerability reports using 7 brocards to accept, dismiss, or request more info before deeper analysis |
Malware Analysis
| Plugin | Description |
|---|---|
| yara-authoring | YARA detection rule authoring with linting, atom analysis, and best practices |
Verification
| Plugin | Description |
|---|---|
| constant-time-analysis | Detect compiler-induced timing side-channels in cryptographic code |
| mutation-testing | Configure mewt/muton mutation testing campaigns — scope targets, tune timeouts, optimize long runs |
| property-based-testing | Property-based testing guidance for multiple languages and smart contracts |
| spec-to-code-compliance | Specification-to-code compliance checker for blockchain audits |
| zeroize-audit | Detect missing or compiler-eliminated zeroization of secrets in C/C++ and Rust |
Reverse Engineering
| Plugin | Description |
|---|---|
| dwarf-expert | Interact with and understand the DWARF debugging format |
Mobile Security
| Plugin | Description |
|---|---|
| firebase-apk-scanner | Scan Android APKs for Firebase security misconfigurations |
Development
| Plugin | Description |
|---|---|
| ask-questions-if-underspecified | Clarify requirements before implementing |
| devcontainer-setup | Create pre-configured devcontainers with Claude Code and language-specific tooling |
| gh-cli | Intercept GitHub URL fetches and redirect to the authenticated gh CLI |
| git-cleanup | Safely clean up git worktrees and local branches with gated confirmation workflow |
| github-triage | Triage open GitHub issues and PRs: merge ready bot/approved PRs, review unreviewed ones via subagents, close resolved issues with cited comments, cross-link pending fixes, and score the rest with local-only priority and change-size estimates |
| let-fate-decide | Draw Tarot cards using cryptographic randomness to add entropy to vague planning |
| modern-python | Modern Python tooling and best practices with uv, ruff, and pytest |
| open-sourcing | Prepare a repository for public release: secrets hygiene, licensing, CI readiness, and release automation |
| seatbelt-sandboxer | Generate minimal macOS Seatbelt sandbox configurations |
| second-opinion | Run code reviews using external LLM CLIs (OpenAI Codex, Google Gemini) on changes, diffs, or commits. Bundles Codex's built-in MCP server. |
| skill-improver | Iterative skill refinement loop using automated fix-review cycles |
| workflow-skill-design | Design patterns for workflow-based Claude Code skills with review agent |
Team Management
| Plugin | Description |
|---|---|
| culture-index | Interpret Culture Index survey results for individuals and teams |
Tooling
| Plugin | Description |
|---|---|
| claude-in-chrome-troubleshooting | Diagnose and fix Claude in Chrome MCP extension connectivity issues |
Infrastructure
| Plugin | Description |
|---|---|
| debug-buttercup | Debug Buttercup Kubernetes deployments |
Trophy Case
Bugs discovered using Trail of Bits Skills. Found something? Let us know!
When reporting bugs you've found, feel free to mention:
Found using Trail of Bits Skills
| Skill | Bug |
|---|---|
| constant-time-analysis | Timing side-channel in ML-DSA signing |
Contributing
We welcome contributions! See AGENTS.md for skill authoring guidelines, and
run make check before you push — it runs most of CI locally (see AGENTS.md for
what it does not cover).
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Made by Trail of Bits.