mirror of
https://github.com/trailofbits/skills.git
synced 2026-09-14 14:28:48 +08:00
kz-tob
3deb39e7b3
* testing-handbook-skills: make 15 descriptions routable Every description was a tool-encyclopedia blurb averaging 125 chars — the first half defined the tool, the second half restated it as a trigger. "Coverage-guided fuzzer built into LLVM for C/C++ projects. Use for fuzzing C/C++ code that can be compiled with Clang." Fifteen skills competing on wording like that lose to each other and to siblings elsewhere in the marketplace. Each is now three parts: what it does for the reader, task first, since the name field already carries the tool name; what it covers, in concrete flags and symbols; then two to four situations in the words a user would type. The anchors are the point — LLVMFuzzerTestOneInput, fuzz_target!, FuzzedDataProvider, afl-clang-fast, ASAN_OPTIONS, project.yaml, an ASan stack trace, a campaign that finds nothing. Fix the generator too, or the next skill it emits is thin again. All four templates prescribed the shape being removed, and their worked examples were these same descriptions. agent-prompt.md now says why the existing rule is not enough: "MUST include Use when" is satisfied by "Use for fuzzing C/C++ code", which is how these got written. Put descriptions on one quoted line rather than a folded block. Eight of these skills already fail the plugin's own 500-line limit, and folded blocks added 4-5 lines to each; one quoted line removes 1-2 instead. It is also what the rest of the repo uses at this length and is exempt from the validator's plain-scalar rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * testing-handbook-skills: ground three descriptions, gate placeholders Three descriptions advertised anchors that appear nowhere in the skill they route to — the worst failure for a change about routing, since the description wins the query and the skill then has nothing to say. harness-writing claimed "C/C++, Rust, Python, and Ruby". Grepping it for ruby, gem, or .rb returns only that line; Python appears once, as a Related Skills row. It would have taken "harness for my Ruby gem" from ruzzy and delivered a file with no Ruby in it. Now C/C++ and Rust, which is what the 12 LLVMFuzzerTestOneInput and 16 fuzz_target! sites cover. constant-time-testing named ctgrind, whose only occurrence in the whole plugin was that description. Replaced with Timecop and Valgrind, at 20 and 10 hits. Also leads with measuring a running implementation and adds a "Not for" line, restoring the boundary constant-time-analysis already documents in its own When NOT to Use. cargo-fuzz claimed "cargo fuzz init and add"; only init, run, coverage, and crash exist. Dropped add, added the nightly requirement and cargo fuzz coverage, both of which the body does cover. Gate the class rather than just these three. A description shipped with a {placeholder} still in it passed every check, because the shortcode pattern needs double braces — and this branch widened the templates' slots, so the surface grew. validate-skills.py now rejects it, and test_validate_skills.py holds each description check to a known-bad fixture plus a positive control. Stdlib only, since CI runs these with --no-project --with pytest, an environment without pyyaml. Fix the pointer to a section that does not exist, drop the two-part "what AND when" bar from testing.md's checklist since the old thin descriptions satisfied it, correct the README's skill inventory, and take the version to MINOR — this changes what the generator emits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop semgrep and codeql from the cross-reference graph The prose and the summary table were updated to 14 skills, but the graph still declared a Tools subgraph with semgrep and codeql and drew both edges between them. Neither skill exists under skills/, so the graph rendered 16 nodes beneath a sentence claiming 14 and promising that only generated skills are shown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fail validation on frontmatter that carries no fields extract_frontmatter returned (None, None) for an empty block, because yaml.safe_load("") is None and that is not a parse error. validate_skill branches on the error, so every frontmatter check was skipped and a skill with no name and no description printed a clean tick. A bare scalar took the same path and died on .get with an uncaught AttributeError. Extraction now pairs both cases with an error, and validate_frontmatter reports rather than returning silently when handed a non-mapping. Four tests cover it, stubbing the parser so they run in CI's pyyaml-free environment; they fail against the previous code and nothing else does. Ground the atheris description's two API anchors in the body: rename the harness entry point to TestOneInput, matching upstream Atheris and its error messages, and add a FuzzedDataProvider section covering the typed draws and the fixed-order rule. Both were advertised in the description and appeared nowhere else in the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Split atheris, and fix the FuzzedDataProvider method table The structured-input section pushed atheris from 519 lines to 552, making a file already over the plugin's 500-line error limit worse — the opposite of what the PR body claimed. Both it and the two worked harnesses move to sibling files, the split this plugin's own agent-prompt.md prescribes for the band. SKILL.md is now 482 lines and passes the line-count check it has failed since it was generated. Three errors in the method table, all mine, corrected in the moved copy: - remaining_bytes() returns a count and consumes nothing; it was listed as the way to get the remaining input. Following it hands the target an int where bytes is expected. The idiom is ConsumeBytes(fdp.remaining_bytes()). - ConsumeIntList takes (count, bytes) and was shown with no arguments. - ConsumeUnicode permits lone surrogates, not surrogate pairs. The pairs gloss suggests valid text; unpaired surrogates raise UnicodeEncodeError the moment a target encodes them, so the campaign reports its own input handling rather than the target's. Every method is checked against the pybind registration in atheris.cc, which exposes remaining_bytes despite the upstream README omitting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>