mirror of
https://github.com/trailofbits/skills.git
synced 2026-09-14 14:28:48 +08:00
testing-handbook-skills: make 15 descriptions routable (#276)
* testing-handbook-skills: make 15 descriptions routable Every description was a tool-encyclopedia blurb averaging 125 chars — the first half defined the tool, the second half restated it as a trigger. "Coverage-guided fuzzer built into LLVM for C/C++ projects. Use for fuzzing C/C++ code that can be compiled with Clang." Fifteen skills competing on wording like that lose to each other and to siblings elsewhere in the marketplace. Each is now three parts: what it does for the reader, task first, since the name field already carries the tool name; what it covers, in concrete flags and symbols; then two to four situations in the words a user would type. The anchors are the point — LLVMFuzzerTestOneInput, fuzz_target!, FuzzedDataProvider, afl-clang-fast, ASAN_OPTIONS, project.yaml, an ASan stack trace, a campaign that finds nothing. Fix the generator too, or the next skill it emits is thin again. All four templates prescribed the shape being removed, and their worked examples were these same descriptions. agent-prompt.md now says why the existing rule is not enough: "MUST include Use when" is satisfied by "Use for fuzzing C/C++ code", which is how these got written. Put descriptions on one quoted line rather than a folded block. Eight of these skills already fail the plugin's own 500-line limit, and folded blocks added 4-5 lines to each; one quoted line removes 1-2 instead. It is also what the rest of the repo uses at this length and is exempt from the validator's plain-scalar rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * testing-handbook-skills: ground three descriptions, gate placeholders Three descriptions advertised anchors that appear nowhere in the skill they route to — the worst failure for a change about routing, since the description wins the query and the skill then has nothing to say. harness-writing claimed "C/C++, Rust, Python, and Ruby". Grepping it for ruby, gem, or .rb returns only that line; Python appears once, as a Related Skills row. It would have taken "harness for my Ruby gem" from ruzzy and delivered a file with no Ruby in it. Now C/C++ and Rust, which is what the 12 LLVMFuzzerTestOneInput and 16 fuzz_target! sites cover. constant-time-testing named ctgrind, whose only occurrence in the whole plugin was that description. Replaced with Timecop and Valgrind, at 20 and 10 hits. Also leads with measuring a running implementation and adds a "Not for" line, restoring the boundary constant-time-analysis already documents in its own When NOT to Use. cargo-fuzz claimed "cargo fuzz init and add"; only init, run, coverage, and crash exist. Dropped add, added the nightly requirement and cargo fuzz coverage, both of which the body does cover. Gate the class rather than just these three. A description shipped with a {placeholder} still in it passed every check, because the shortcode pattern needs double braces — and this branch widened the templates' slots, so the surface grew. validate-skills.py now rejects it, and test_validate_skills.py holds each description check to a known-bad fixture plus a positive control. Stdlib only, since CI runs these with --no-project --with pytest, an environment without pyyaml. Fix the pointer to a section that does not exist, drop the two-part "what AND when" bar from testing.md's checklist since the old thin descriptions satisfied it, correct the README's skill inventory, and take the version to MINOR — this changes what the generator emits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop semgrep and codeql from the cross-reference graph The prose and the summary table were updated to 14 skills, but the graph still declared a Tools subgraph with semgrep and codeql and drew both edges between them. Neither skill exists under skills/, so the graph rendered 16 nodes beneath a sentence claiming 14 and promising that only generated skills are shown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fail validation on frontmatter that carries no fields extract_frontmatter returned (None, None) for an empty block, because yaml.safe_load("") is None and that is not a parse error. validate_skill branches on the error, so every frontmatter check was skipped and a skill with no name and no description printed a clean tick. A bare scalar took the same path and died on .get with an uncaught AttributeError. Extraction now pairs both cases with an error, and validate_frontmatter reports rather than returning silently when handed a non-mapping. Four tests cover it, stubbing the parser so they run in CI's pyyaml-free environment; they fail against the previous code and nothing else does. Ground the atheris description's two API anchors in the body: rename the harness entry point to TestOneInput, matching upstream Atheris and its error messages, and add a FuzzedDataProvider section covering the typed draws and the fixed-order rule. Both were advertised in the description and appeared nowhere else in the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Split atheris, and fix the FuzzedDataProvider method table The structured-input section pushed atheris from 519 lines to 552, making a file already over the plugin's 500-line error limit worse — the opposite of what the PR body claimed. Both it and the two worked harnesses move to sibling files, the split this plugin's own agent-prompt.md prescribes for the band. SKILL.md is now 482 lines and passes the line-count check it has failed since it was generated. Three errors in the method table, all mine, corrected in the moved copy: - remaining_bytes() returns a count and consumes nothing; it was listed as the way to get the remaining input. Following it hands the target an int where bytes is expected. The idiom is ConsumeBytes(fdp.remaining_bytes()). - ConsumeIntList takes (count, bytes) and was shown with no arguments. - ConsumeUnicode permits lone surrogates, not surrogate pairs. The pairs gloss suggests valid text; unpaired surrogates raise UnicodeEncodeError the moment a target encodes them, so the campaign reports its own input handling rather than the target's. Every method is checked against the pybind registration in atheris.cc, which exposes remaining_bytes despite the upstream README omitting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kz-tob
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "testing-handbook-skills",
|
||||
"version": "1.1.1",
|
||||
"version": "1.2.0",
|
||||
"description": "Skills from the Trail of Bits Application Security Testing Handbook (appsec.guide)",
|
||||
"author": {
|
||||
"name": "Paweł Płatek"
|
||||
|
||||
@@ -113,7 +113,7 @@ Each generated skill:
|
||||
|
||||
## Skills Cross-Reference
|
||||
|
||||
This graph shows the 16 generated skills and their cross-references (from the Related Skills section of each skill). Only links between actually generated skills are shown.
|
||||
This graph shows the 14 generated skills and their cross-references (from the Related Skills section of each skill). Only links between actually generated skills are shown.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
@@ -135,11 +135,6 @@ graph TB
|
||||
ossfuzz[ossfuzz]
|
||||
end
|
||||
|
||||
subgraph Tools
|
||||
semgrep[semgrep]
|
||||
codeql[codeql]
|
||||
end
|
||||
|
||||
subgraph Domain
|
||||
wycheproof[wycheproof]
|
||||
constant-time-testing[constant-time-testing]
|
||||
@@ -171,10 +166,6 @@ graph TB
|
||||
ruzzy -.-> libfuzzer
|
||||
ruzzy -.-> aflpp
|
||||
|
||||
%% Tool ↔ Tool alternatives
|
||||
semgrep -.-> codeql
|
||||
codeql -.-> semgrep
|
||||
|
||||
%% Technique → Fuzzer references
|
||||
harness-writing --> libfuzzer
|
||||
harness-writing --> aflpp
|
||||
@@ -218,7 +209,6 @@ graph TB
|
||||
|------|--------|
|
||||
| Fuzzers (6) | libfuzzer, aflpp, libafl, cargo-fuzz, atheris, ruzzy |
|
||||
| Techniques (6) | harness-writing, address-sanitizer, coverage-analysis, fuzzing-dictionary, fuzzing-obstacles, ossfuzz |
|
||||
| Tools (2) | semgrep, codeql |
|
||||
| Domain (2) | wycheproof, constant-time-testing |
|
||||
|
||||
**Note:** Some skills reference planned/external skills not yet generated (e.g., `honggfuzz`, `fuzzing-corpus`, `sarif-parsing`). Run `validate-skills.py` to see the full list.
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prove validate-skills.py still detects what it exists to detect.
|
||||
|
||||
A checker that has silently stopped matching passes every run and looks clean
|
||||
doing it. Each test here builds a known-bad frontmatter block and asserts the
|
||||
matching check rejects it, then asserts a good one is accepted — so a check
|
||||
that starts matching nothing, or everything, fails this suite.
|
||||
|
||||
Stdlib only, like every other suite in this repo: CI runs these with
|
||||
`uv run --no-project --with pytest`, an environment that has pytest and
|
||||
nothing else. validate-skills.py imports pyyaml at module scope purely to
|
||||
parse frontmatter, and none of the checks under test here call it, so a
|
||||
stand-in satisfies the import when pyyaml is absent. It raises rather than
|
||||
returning a value, so no test can quietly come to depend on a fake parser.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import re
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
try: # pragma: no cover - depends on the environment, both branches are fine
|
||||
import yaml # noqa: F401
|
||||
except ModuleNotFoundError: # pragma: no cover
|
||||
_stub = types.ModuleType("yaml")
|
||||
_stub.YAMLError = type("YAMLError", (Exception,), {})
|
||||
|
||||
def _unavailable(*_args, **_kwargs):
|
||||
raise AssertionError(
|
||||
"validate-skills.py's YAML parsing is not under test here and pyyaml "
|
||||
"is not installed — a test reaching this needs the real dependency"
|
||||
)
|
||||
|
||||
_stub.safe_load = _unavailable
|
||||
sys.modules["yaml"] = _stub
|
||||
|
||||
_SPEC = importlib.util.spec_from_file_location(
|
||||
"validate_skills", Path(__file__).parent / "validate-skills.py"
|
||||
)
|
||||
assert _SPEC and _SPEC.loader
|
||||
validate_skills = importlib.util.module_from_spec(_SPEC)
|
||||
sys.modules["validate_skills"] = validate_skills
|
||||
_SPEC.loader.exec_module(validate_skills)
|
||||
|
||||
# Frontmatter descriptions are single-quoted single lines by convention, so the
|
||||
# shipped-description sweep below reads them without needing a YAML parser.
|
||||
DESCRIPTION_LINE = re.compile(r'^description:\s*"(.*)"\s*$', re.M)
|
||||
|
||||
GOOD_DESCRIPTION = (
|
||||
"Sets up and runs libFuzzer, the coverage-guided fuzzer built into LLVM, on "
|
||||
"C/C++ code that compiles with Clang. Covers harness structure and campaign "
|
||||
"triage. Use when writing an LLVMFuzzerTestOneInput harness, or working out "
|
||||
"why a libFuzzer run finds nothing."
|
||||
)
|
||||
|
||||
|
||||
def check(frontmatter) -> list[str]:
|
||||
"""Run frontmatter validation and return the errors it reported."""
|
||||
result = validate_skills.ValidationResult(skill_name="fixture", skill_path=Path("SKILL.md"))
|
||||
validate_skills.validate_frontmatter(frontmatter, result)
|
||||
return result.errors
|
||||
|
||||
|
||||
def frontmatter(**overrides) -> dict:
|
||||
base = {"name": "libfuzzer", "type": "fuzzer", "description": GOOD_DESCRIPTION}
|
||||
base.update(overrides)
|
||||
return base
|
||||
|
||||
|
||||
def test_good_frontmatter_is_accepted() -> None:
|
||||
"""The positive control. If this fails, every rejection below proves nothing."""
|
||||
assert check(frontmatter()) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"parsed",
|
||||
[pytest.param(None, id="empty-block"), pytest.param("just a string", id="bare-scalar")],
|
||||
)
|
||||
def test_non_mapping_frontmatter_is_rejected(parsed) -> None:
|
||||
"""A block yaml accepts but that carries no fields must not validate silently.
|
||||
|
||||
This is the whole-check version of the zero-guard: every field check below
|
||||
reads `frontmatter.get(...)`, so anything that is not a mapping used to skip
|
||||
all of them and report the skill clean.
|
||||
"""
|
||||
assert check(parsed) != []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("parsed", "expected"),
|
||||
[
|
||||
pytest.param(None, "empty", id="empty-block"),
|
||||
pytest.param("just a string", "not a mapping", id="bare-scalar"),
|
||||
],
|
||||
)
|
||||
def test_extraction_reports_non_mapping_frontmatter(monkeypatch, parsed, expected) -> None:
|
||||
"""`extract_frontmatter` must pair a non-mapping result with an error.
|
||||
|
||||
`validate_skill` runs the field checks only when extraction reported no
|
||||
error, so returning `(None, None)` for an empty block skipped them all.
|
||||
The parser is stubbed rather than fed YAML so this runs in CI's
|
||||
pyyaml-free environment, where it matters most.
|
||||
"""
|
||||
monkeypatch.setattr(validate_skills.yaml, "safe_load", lambda _text: parsed)
|
||||
|
||||
frontmatter_result, error = validate_skills.extract_frontmatter("---\n\n---\n\n# Skill\n")
|
||||
|
||||
assert frontmatter_result is None
|
||||
assert error and expected in error
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("description", "expected"),
|
||||
[
|
||||
pytest.param(
|
||||
"Coverage-guided fuzzer built into LLVM for C/C++ projects.",
|
||||
"trigger phrase",
|
||||
id="no-trigger-phrase",
|
||||
),
|
||||
pytest.param(
|
||||
"Sets up {fuzzer}. Use when fuzzing {language} projects.",
|
||||
"template placeholders",
|
||||
id="template-placeholder",
|
||||
),
|
||||
pytest.param(
|
||||
"Sets up a fuzzer for {language} projects. Use when fuzzing Rust.",
|
||||
"template placeholders",
|
||||
id="single-placeholder-among-real-text",
|
||||
),
|
||||
pytest.param(
|
||||
"Sets up <b>libFuzzer</b>. Use when fuzzing C/C++.",
|
||||
"HTML/XML tags",
|
||||
id="html-tag",
|
||||
),
|
||||
pytest.param(
|
||||
"Sets up libFuzzer. {{< hint >}} Use when fuzzing C/C++.",
|
||||
"Hugo shortcodes",
|
||||
id="hugo-shortcode",
|
||||
),
|
||||
pytest.param("x" * 1025 + " Use when fuzzing.", "too long", id="over-length"),
|
||||
pytest.param("", "Missing required field", id="empty"),
|
||||
],
|
||||
)
|
||||
def test_bad_description_is_rejected(description: str, expected: str) -> None:
|
||||
errors = check(frontmatter(description=description))
|
||||
assert any(expected in e for e in errors), f"expected {expected!r} in {errors}"
|
||||
|
||||
|
||||
def test_placeholder_check_does_not_fire_on_real_descriptions() -> None:
|
||||
"""Every shipped description must survive the placeholder check.
|
||||
|
||||
This is the guard against the check being too greedy — a pattern that also
|
||||
matched ordinary prose would make the whole plugin unshippable.
|
||||
"""
|
||||
skills_dir = Path(__file__).parent.parent / "skills"
|
||||
shipped = sorted(skills_dir.glob("*/SKILL.md"))
|
||||
assert len(shipped) >= 15, f"expected the plugin's skills, found {len(shipped)}"
|
||||
|
||||
for skill in shipped:
|
||||
match = DESCRIPTION_LINE.search(skill.read_text(encoding="utf-8"))
|
||||
assert match, f"{skill.parent.name}: no single-line quoted description found"
|
||||
assert not validate_skills.PLACEHOLDER_PATTERN.search(match.group(1)), (
|
||||
f"{skill.parent.name}: description matches the placeholder pattern"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("field", "value", "expected"),
|
||||
[
|
||||
pytest.param("name", "LibFuzzer", "Invalid name", id="uppercase-name"),
|
||||
pytest.param("name", "claude-fuzzer", "reserved word", id="reserved-word"),
|
||||
pytest.param("type", "wildly-invalid", "Invalid type", id="bad-type"),
|
||||
],
|
||||
)
|
||||
def test_bad_field_is_rejected(field: str, value: str, expected: str) -> None:
|
||||
errors = check(frontmatter(**{field: value}))
|
||||
assert any(expected in e for e in errors), f"expected {expected!r} in {errors}"
|
||||
@@ -48,6 +48,10 @@ RESERVED_WORDS = frozenset({"anthropic", "claude"})
|
||||
VALID_SKILL_TYPES = frozenset({"tool", "fuzzer", "technique", "domain"})
|
||||
NAME_PATTERN = re.compile(r"^[a-z0-9-]{1,64}$")
|
||||
SHORTCODE_PATTERN = re.compile(r"\{\{[<%]")
|
||||
# The templates express their fill-in slots as {like this}. The shortcode pattern
|
||||
# needs *double* braces, so a description shipped with a slot still in it passed
|
||||
# every check and printed a clean tick.
|
||||
PLACEHOLDER_PATTERN = re.compile(r"\{[^{}]*\}")
|
||||
ESCAPED_BACKTICKS_PATTERN = re.compile(r"\\`{3}")
|
||||
HTML_TAG_PATTERN = re.compile(r"<[^>]+>")
|
||||
|
||||
@@ -158,10 +162,24 @@ def extract_frontmatter(content: str) -> tuple[dict | None, str | None]:
|
||||
|
||||
frontmatter_text = "\n".join(lines[1:end_idx])
|
||||
try:
|
||||
return yaml.safe_load(frontmatter_text), None
|
||||
parsed = yaml.safe_load(frontmatter_text)
|
||||
except yaml.YAMLError as e:
|
||||
return None, f"YAML parse error: {e}"
|
||||
|
||||
# An empty block parses to None and a bare scalar to a str, neither of which
|
||||
# is an error to yaml. Returning either with no error made every frontmatter
|
||||
# check below skip silently, so a nameless, descriptionless skill printed a
|
||||
# clean tick.
|
||||
if parsed is None:
|
||||
return None, "Frontmatter block is empty (no name or description to validate)"
|
||||
if not isinstance(parsed, dict):
|
||||
return None, (
|
||||
f"Frontmatter is not a mapping: parsed as {type(parsed).__name__}, "
|
||||
f"expected 'key: value' fields"
|
||||
)
|
||||
|
||||
return parsed, None
|
||||
|
||||
|
||||
def detect_skill_type(
|
||||
content: str,
|
||||
@@ -218,8 +236,11 @@ def validate_frontmatter(
|
||||
frontmatter: Parsed frontmatter dict.
|
||||
result: ValidationResult to update.
|
||||
"""
|
||||
if frontmatter is None:
|
||||
return # Error already added during extraction
|
||||
if not isinstance(frontmatter, dict):
|
||||
result.add_error(
|
||||
"No frontmatter fields to validate: the block is missing, empty, or not a mapping"
|
||||
)
|
||||
return
|
||||
|
||||
# Validate name field
|
||||
name = frontmatter.get("name")
|
||||
@@ -263,6 +284,13 @@ def validate_frontmatter(
|
||||
if SHORTCODE_PATTERN.search(desc_str):
|
||||
result.add_error("Description contains Hugo shortcodes")
|
||||
|
||||
leftover = PLACEHOLDER_PATTERN.findall(desc_str)
|
||||
if leftover:
|
||||
result.add_error(
|
||||
f"Description still contains template placeholders: {leftover} — "
|
||||
f"replace every {{...}} slot with real content"
|
||||
)
|
||||
|
||||
# Validate type field (recommended but not strictly required for backwards compat)
|
||||
skill_type = frontmatter.get("type")
|
||||
if not skill_type:
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: address-sanitizer
|
||||
type: technique
|
||||
description: >
|
||||
AddressSanitizer detects memory errors during fuzzing.
|
||||
Use when fuzzing C/C++ code to find buffer overflows and use-after-free bugs.
|
||||
description: "Builds and runs code under AddressSanitizer to catch buffer overflows, use-after-free, and other memory errors during fuzzing or tests. Covers -fsanitize=address builds, ASAN_OPTIONS, reading the crash report, LeakSanitizer, and the overhead and platform trade-offs. Use when fuzzing C/C++ or Rust that has unsafe blocks or FFI, when debugging a memory corruption crash, or when reading an ASan stack trace."
|
||||
---
|
||||
|
||||
# AddressSanitizer (ASan)
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: aflpp
|
||||
type: fuzzer
|
||||
description: >
|
||||
AFL++ is a fork of AFL with better fuzzing performance and advanced features.
|
||||
Use for multi-core fuzzing of C/C++ projects.
|
||||
description: "Sets up and runs AFL++ for multi-core fuzzing of C/C++ projects built with afl-clang-fast or afl-gcc-fast. Covers instrumentation modes, parallel main and secondary campaigns, persistent mode, corpus minimization, and crash triage. Use when scaling fuzzing across cores, fuzzing a mature C/C++ codebase, reading the afl-fuzz status screen, or moving on after libFuzzer has plateaued."
|
||||
---
|
||||
|
||||
# AFL++
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: atheris
|
||||
type: fuzzer
|
||||
description: >
|
||||
Atheris is a coverage-guided Python fuzzer based on libFuzzer.
|
||||
Use for fuzzing pure Python code and Python C extensions.
|
||||
description: "Sets up and runs Atheris, the coverage-guided Python fuzzer built on libFuzzer. Covers TestOneInput harnesses, FuzzedDataProvider, instrumenting both pure Python and native C extensions, and running under AddressSanitizer. Use when fuzzing a Python package, hunting memory corruption in a Python C extension, or choosing between Atheris and Hypothesis for a Python target."
|
||||
---
|
||||
|
||||
# Atheris
|
||||
@@ -31,7 +29,7 @@ import sys
|
||||
import atheris
|
||||
|
||||
@atheris.instrument_func
|
||||
def test_one_input(data: bytes):
|
||||
def TestOneInput(data: bytes):
|
||||
if len(data) == 4:
|
||||
if data[0] == 0x46: # "F"
|
||||
if data[1] == 0x55: # "U"
|
||||
@@ -40,7 +38,7 @@ def test_one_input(data: bytes):
|
||||
raise RuntimeError("You caught me")
|
||||
|
||||
def main():
|
||||
atheris.Setup(sys.argv, test_one_input)
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
if __name__ == "__main__":
|
||||
@@ -152,7 +150,7 @@ import sys
|
||||
import atheris
|
||||
|
||||
@atheris.instrument_func
|
||||
def test_one_input(data: bytes):
|
||||
def TestOneInput(data: bytes):
|
||||
"""
|
||||
Fuzzing entry point. Called with random byte sequences.
|
||||
|
||||
@@ -172,13 +170,28 @@ def test_one_input(data: bytes):
|
||||
# Let unexpected exceptions crash (that's what we're looking for!)
|
||||
|
||||
def main():
|
||||
atheris.Setup(sys.argv, test_one_input)
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
### Structured Input with FuzzedDataProvider
|
||||
|
||||
A target taking several typed arguments wastes most of the fuzzer's inputs if the harness
|
||||
slices `data` by hand, because every mutation shifts the byte offsets of everything after it.
|
||||
`atheris.FuzzedDataProvider` splits one `bytes` input into typed values instead:
|
||||
|
||||
```python
|
||||
fdp = atheris.FuzzedDataProvider(data)
|
||||
name = fdp.ConsumeUnicodeNoSurrogates(fdp.ConsumeIntInRange(0, 64))
|
||||
strict = fdp.ConsumeBool()
|
||||
```
|
||||
|
||||
See [structured-input.md](structured-input.md) for the full method reference, the fixed-draw-
|
||||
order rule, and what each method returns once the buffer runs dry.
|
||||
|
||||
### Harness Rules
|
||||
|
||||
| Do | Don't |
|
||||
@@ -201,10 +214,10 @@ with atheris.instrument_imports():
|
||||
import your_module
|
||||
from another_module import target_function
|
||||
|
||||
def test_one_input(data: bytes):
|
||||
def TestOneInput(data: bytes):
|
||||
target_function(data)
|
||||
|
||||
atheris.Setup(sys.argv, test_one_input)
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
```
|
||||
|
||||
@@ -249,7 +262,7 @@ import atheris
|
||||
# _cbor2 ensures the C library is imported
|
||||
from _cbor2 import loads
|
||||
|
||||
def test_one_input(data: bytes):
|
||||
def TestOneInput(data: bytes):
|
||||
try:
|
||||
loads(data)
|
||||
except Exception:
|
||||
@@ -257,7 +270,7 @@ def test_one_input(data: bytes):
|
||||
pass
|
||||
|
||||
def main():
|
||||
atheris.Setup(sys.argv, test_one_input)
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
if __name__ == "__main__":
|
||||
@@ -392,7 +405,7 @@ with atheris.instrument_imports():
|
||||
import target_module
|
||||
# Don't instrument test harness code
|
||||
|
||||
def test_one_input(data: bytes):
|
||||
def TestOneInput(data: bytes):
|
||||
target_module.parse(data)
|
||||
```
|
||||
|
||||
@@ -416,58 +429,8 @@ Note: Modify flags in Dockerfile if using containerized setup.
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### Example: Pure Python Parser
|
||||
|
||||
```python
|
||||
import sys
|
||||
import atheris
|
||||
import json
|
||||
|
||||
@atheris.instrument_func
|
||||
def test_one_input(data: bytes):
|
||||
try:
|
||||
# Fuzz Python's JSON parser
|
||||
json.loads(data.decode('utf-8', errors='ignore'))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
pass
|
||||
|
||||
def main():
|
||||
atheris.Setup(sys.argv, test_one_input)
|
||||
atheris.Fuzz()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
### Example: HTTP Request Parsing
|
||||
|
||||
```python
|
||||
import sys
|
||||
import atheris
|
||||
|
||||
with atheris.instrument_imports():
|
||||
from urllib3 import HTTPResponse
|
||||
from io import BytesIO
|
||||
|
||||
def test_one_input(data: bytes):
|
||||
try:
|
||||
# Fuzz HTTP response parsing
|
||||
fake_response = HTTPResponse(
|
||||
body=BytesIO(data),
|
||||
headers={},
|
||||
preload_content=False
|
||||
)
|
||||
fake_response.read()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def main():
|
||||
atheris.Setup(sys.argv, test_one_input)
|
||||
atheris.Fuzz()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
Two complete harnesses — a pure-Python parser and an HTTP response parser — are in
|
||||
[examples.md](examples.md).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Atheris Examples
|
||||
|
||||
Two complete harnesses, each runnable as written.
|
||||
|
||||
## Example: Pure Python Parser
|
||||
|
||||
```python
|
||||
import sys
|
||||
import atheris
|
||||
import json
|
||||
|
||||
@atheris.instrument_func
|
||||
def TestOneInput(data: bytes):
|
||||
try:
|
||||
# Fuzz Python's JSON parser
|
||||
json.loads(data.decode('utf-8', errors='ignore'))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
pass
|
||||
|
||||
def main():
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
## Example: HTTP Request Parsing
|
||||
|
||||
```python
|
||||
import sys
|
||||
import atheris
|
||||
|
||||
with atheris.instrument_imports():
|
||||
from urllib3 import HTTPResponse
|
||||
from io import BytesIO
|
||||
|
||||
def TestOneInput(data: bytes):
|
||||
try:
|
||||
# Fuzz HTTP response parsing
|
||||
fake_response = HTTPResponse(
|
||||
body=BytesIO(data),
|
||||
headers={},
|
||||
preload_content=False
|
||||
)
|
||||
fake_response.read()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def main():
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
Both catch `Exception` broadly to get a campaign started. Narrow that to the exceptions the
|
||||
target is documented to raise before you trust the results — a bare `except Exception` also
|
||||
swallows the bugs you are fuzzing for.
|
||||
@@ -0,0 +1,67 @@
|
||||
# Structured Input with FuzzedDataProvider
|
||||
|
||||
A target that takes several typed arguments — a string, a length, a flag — wastes most of
|
||||
the fuzzer's inputs if the harness slices `data` by hand, because every mutation shifts the
|
||||
byte offsets of everything after it. `atheris.FuzzedDataProvider` splits one `bytes` input
|
||||
into typed values while keeping each draw stable under mutation.
|
||||
|
||||
## Basic Usage
|
||||
|
||||
```python
|
||||
@atheris.instrument_func
|
||||
def TestOneInput(data: bytes):
|
||||
fdp = atheris.FuzzedDataProvider(data)
|
||||
name = fdp.ConsumeUnicodeNoSurrogates(fdp.ConsumeIntInRange(0, 64))
|
||||
count = fdp.ConsumeIntInRange(1, 1000)
|
||||
strict = fdp.ConsumeBool()
|
||||
your_target_function(name, count, strict=strict)
|
||||
```
|
||||
|
||||
Draw in a fixed order. Each call consumes from where the last one left off, so inserting or
|
||||
reordering a call reinterprets every byte after it and devalues the corpus already built.
|
||||
|
||||
Ask for a size before the content it bounds, as above: an unbounded string lets the fuzzer
|
||||
spend the whole buffer on one field and starve every draw after it.
|
||||
|
||||
## Method Reference
|
||||
|
||||
| Need | Call |
|
||||
|------|------|
|
||||
| Raw bytes | `ConsumeBytes(count)` |
|
||||
| Text | `ConsumeUnicodeNoSurrogates(count)` |
|
||||
| Text, including unpaired surrogates | `ConsumeUnicode(count)` |
|
||||
| Bounded integer | `ConsumeIntInRange(min, max)` |
|
||||
| Sized integer | `ConsumeInt(size)` (signed), `ConsumeUInt(size)` |
|
||||
| Float | `ConsumeFloat()`, `ConsumeRegularFloat()` (no `NaN`/`Inf`), `ConsumeProbability()` |
|
||||
| Flag | `ConsumeBool()` |
|
||||
| Choice from a fixed set | `PickValueInList(values)` |
|
||||
| Everything not yet consumed | `ConsumeBytes(fdp.remaining_bytes())` |
|
||||
|
||||
`remaining_bytes()` is an accessor, not a draw — it returns the count of unconsumed bytes and
|
||||
consumes nothing. It is the one method here that is not named `Consume*`, and the only one
|
||||
whose return value is a length rather than a value. Pass it to `ConsumeBytes` to drain the
|
||||
buffer; using it directly hands your target an `int` where it expects `bytes`.
|
||||
|
||||
Prefer `ConsumeUnicodeNoSurrogates` unless you are specifically testing surrogate handling.
|
||||
`ConsumeUnicode` may emit unpaired surrogates (U+D800–U+DFFF), which are legal in a Python
|
||||
`str` but raise `UnicodeEncodeError` the moment the target encodes them — so a target that
|
||||
encodes anywhere reports a crash on its own input handling rather than on your target's logic.
|
||||
|
||||
List variants take the element count first:
|
||||
|
||||
| Call | Produces |
|
||||
|------|----------|
|
||||
| `ConsumeIntList(count, bytes)` | `count` integers of `bytes` size each |
|
||||
| `ConsumeIntListInRange(count, min, max)` | `count` integers in `[min, max]` |
|
||||
| `ConsumeFloatList(count)` | `count` arbitrary floats, `NaN` and `Inf` included |
|
||||
| `ConsumeRegularFloatList(count)` | `count` floats, never `NaN` or `Inf` |
|
||||
| `ConsumeProbabilityList(count)` | `count` floats in `[0, 1]` |
|
||||
| `ConsumeFloatListInRange(count, min, max)` | `count` floats in `[min, max]` |
|
||||
|
||||
## Running Out of Input
|
||||
|
||||
Every method degrades rather than raising when the buffer empties: consumers return empty
|
||||
values, and `ConsumeIntInRange` returns `min`. A harness that draws more than the fuzzer
|
||||
supplies will not error — it will quietly test the same degenerate case over and over, which
|
||||
looks like a healthy campaign that has stopped finding anything. Check `remaining_bytes()`
|
||||
and return early if your harness needs a minimum amount of input.
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: cargo-fuzz
|
||||
type: fuzzer
|
||||
description: >
|
||||
cargo-fuzz is the de facto fuzzing tool for Rust projects using Cargo.
|
||||
Use for fuzzing Rust code with libFuzzer backend.
|
||||
description: "Sets up and runs cargo-fuzz, the standard fuzzing tool for Cargo-based Rust projects. Covers cargo fuzz init, the nightly toolchain requirement, fuzz_target! harnesses, Arbitrary-derived structured inputs, sanitizer options, cargo fuzz coverage, and reproducing a crash artifact. Use when fuzzing a Rust crate, writing a fuzz_target!, exercising unsafe blocks or FFI in Rust, or triaging a cargo fuzz crash."
|
||||
---
|
||||
|
||||
# cargo-fuzz
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: constant-time-testing
|
||||
type: domain
|
||||
description: >
|
||||
Constant-time testing detects timing side channels in cryptographic code.
|
||||
Use when auditing crypto implementations for timing vulnerabilities.
|
||||
description: "Measures timing side channels in cryptographic implementations by running them, using dudect for statistical analysis and Timecop over Valgrind for dynamic tracing. Covers the formal, symbolic, dynamic, and statistical tool categories and how to read a result. Use when testing whether a running implementation is constant-time, measuring timing variance on a compiled binary, or investigating a suspected timing attack. Not for statically inspecting compiler output — the constant-time-analysis plugin covers that."
|
||||
---
|
||||
|
||||
# Constant-Time Testing
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: coverage-analysis
|
||||
type: technique
|
||||
description: >
|
||||
Coverage analysis measures code exercised during fuzzing.
|
||||
Use when assessing harness effectiveness or identifying fuzzing blockers.
|
||||
description: "Measures and interprets what a fuzzing campaign actually reaches, using llvm-cov, lcov, or a fuzzer's own coverage output. Covers baselining a new campaign, reading coverage reports, and turning uncovered regions into harness, seed, or dictionary work. Use when a fuzzer plateaus, when judging whether a harness is effective, after changing a harness, or when asking why some code is never reached."
|
||||
---
|
||||
|
||||
# Coverage Analysis
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: fuzzing-dictionary
|
||||
type: technique
|
||||
description: >
|
||||
Fuzzing dictionaries guide fuzzers with domain-specific tokens.
|
||||
Use when fuzzing parsers, protocols, or format-specific code.
|
||||
description: "Builds and applies fuzzing dictionaries so a fuzzer can produce the keywords, magic bytes, and tokens a target expects. Covers extracting tokens from source, headers, binaries, and specifications, dictionary syntax, and wiring one into libFuzzer or AFL++. Use when fuzzing a parser, protocol, or file format, when coverage stalls at input validation, or when a target compares against fixed strings."
|
||||
---
|
||||
|
||||
# Fuzzing Dictionary
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: fuzzing-obstacles
|
||||
type: technique
|
||||
description: >
|
||||
Techniques for patching code to overcome fuzzing obstacles.
|
||||
Use when checksums, global state, or other barriers block fuzzer progress.
|
||||
description: "Patches past the barriers that stop a fuzzer making progress — checksum and hash verification, magic-value validation, time-based seeds, and other non-deterministic global state. Covers locating the blocking check, neutering it behind a fuzzing build flag, and avoiding the false positives a patch can introduce. Use when a fuzzer is stuck at validation, when coverage shows large regions behind a checksum, or when valid inputs are impractical to generate."
|
||||
---
|
||||
|
||||
# Overcoming Fuzzing Obstacles
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: harness-writing
|
||||
type: technique
|
||||
description: >
|
||||
Techniques for writing effective fuzzing harnesses across languages.
|
||||
Use when creating new fuzz targets or improving existing harness code.
|
||||
description: "Designs and improves fuzzing harnesses for C/C++ and Rust. Covers mapping raw bytes onto a target API, generating structured inputs, avoiding non-determinism and false crashes, and deciding what to fuzz together. Use when writing a first LLVMFuzzerTestOneInput or fuzz_target! harness, when a campaign finds nothing or reports crashes that will not reproduce, or when the target API needs structured rather than raw input."
|
||||
---
|
||||
|
||||
# Writing Fuzzing Harnesses
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: libafl
|
||||
type: fuzzer
|
||||
description: >
|
||||
LibAFL is a modular fuzzing library for building custom fuzzers. Use for
|
||||
advanced fuzzing needs, custom mutators, or non-standard fuzzing targets.
|
||||
description: "Builds custom fuzzers with LibAFL, the modular Rust fuzzing library. Covers composing observers, feedbacks, mutators, schedulers, and executors into a fuzzer for targets the standard tools do not fit. Use when writing a bespoke fuzzer or mutator, fuzzing a non-standard target or architecture, implementing a fuzzing research idea, or when libFuzzer and AFL++ lack the control you need."
|
||||
---
|
||||
|
||||
# LibAFL
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: libfuzzer
|
||||
type: fuzzer
|
||||
description: >
|
||||
Coverage-guided fuzzer built into LLVM for C/C++ projects. Use for fuzzing
|
||||
C/C++ code that can be compiled with Clang.
|
||||
description: "Sets up and runs libFuzzer, the coverage-guided fuzzer built into LLVM, on C/C++ code that compiles with Clang. Covers harness structure, -fsanitize=fuzzer builds, corpus and dictionary management, sanitizer integration, and campaign triage. Use when writing or debugging an LLVMFuzzerTestOneInput harness, starting fuzzing on a C/C++ library, choosing between libFuzzer and AFL++, or working out why a libFuzzer run finds nothing."
|
||||
---
|
||||
|
||||
# libFuzzer
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: ossfuzz
|
||||
type: technique
|
||||
description: >
|
||||
OSS-Fuzz provides free continuous fuzzing for open source projects.
|
||||
Use when setting up continuous fuzzing infrastructure or enrolling projects.
|
||||
description: "Enrolls a project in OSS-Fuzz, Google's free continuous fuzzing service for open source, and drives it locally. Covers project.yaml, Dockerfile and build.sh setup, the helper scripts, reproducing OSS-Fuzz crash reports, and the acceptance criteria. Use when setting up continuous fuzzing for an open-source project, reproducing an OSS-Fuzz bug report, or testing an OSS-Fuzz build before submitting it."
|
||||
---
|
||||
|
||||
# OSS-Fuzz
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: ruzzy
|
||||
type: fuzzer
|
||||
description: >
|
||||
Ruzzy is a coverage-guided Ruby fuzzer by Trail of Bits.
|
||||
Use for fuzzing pure Ruby code and Ruby C extensions.
|
||||
description: "Sets up and runs Ruzzy, Trail of Bits' coverage-guided Ruby fuzzer and the only production-ready one for the language. Covers harness structure, fuzzing pure Ruby and the native C extensions in gems, and sanitizer builds. Use when fuzzing a Ruby library or gem, testing a Ruby C extension for memory safety, or asking how to fuzz Ruby at all."
|
||||
---
|
||||
|
||||
# Ruzzy
|
||||
|
||||
@@ -1,9 +1,6 @@
|
||||
---
|
||||
name: testing-handbook-generator
|
||||
description: >
|
||||
Meta-skill that analyzes the Trail of Bits Testing Handbook (appsec.guide)
|
||||
and generates Claude Code skills for security testing tools and techniques.
|
||||
Use when creating new skills based on handbook content.
|
||||
description: "Generates Claude Code skills from the Trail of Bits Testing Handbook (appsec.guide), analyzing handbook pages and emitting SKILL.md files with the structure each skill type requires. Use when creating or refreshing a skill from handbook content, or when the user names the testing handbook or appsec.guide. Not for answering security testing questions — the generated skills cover those."
|
||||
---
|
||||
|
||||
# Testing Handbook Skill Generator
|
||||
|
||||
@@ -73,7 +73,7 @@ Before writing SKILL.md, verify ALL items:
|
||||
- [ ] **No shortcodes**: No `{{<` or `{{% ` patterns remain in output
|
||||
- [ ] **No escaped backticks**: No `\``` ` patterns remain (should be unescaped to ` ``` `)
|
||||
- [ ] **Required section**: Has `## When to Use` heading
|
||||
- [ ] **Trigger phrase**: Description contains "Use when" or "Use for"
|
||||
- [ ] **Trigger phrase**: Description contains "Use when" or "Use for", naming two or more concrete situations rather than restating the tool's purpose (see "Description quality" under Critical Rules)
|
||||
- [ ] **Code preserved**: All code blocks have language specifier and exact content
|
||||
- [ ] **Related Skills placeholder**: Has `## Related Skills` with `<!-- PASS2: ... -->` comment
|
||||
|
||||
@@ -125,6 +125,24 @@ Templates use `\``` ` (backslash-escaped backticks) to show code block examples
|
||||
- `type`: one of `tool`, `fuzzer`, `technique`, `domain` (determines required sections)
|
||||
- `description`: max 1024 chars, MUST include "Use when {trigger}" or "Use for {purpose}"
|
||||
|
||||
**Description quality.** The trigger phrase is necessary, not sufficient — "Use for
|
||||
fuzzing C/C++ code" satisfies the check and still loses every routing contest to a
|
||||
sibling skill. This is the highest-leverage line in a skill: one that never triggers
|
||||
may as well not exist. Write it in three parts, on a single quoted line:
|
||||
|
||||
1. **What it does for the reader**, leading with the task, not a definition of the
|
||||
tool. "Sets up and runs libFuzzer …", not "libFuzzer is a coverage-guided fuzzer."
|
||||
The `name` field already carries the tool name.
|
||||
2. **What it covers** — the concrete commands, flags, file names, and API symbols.
|
||||
These are what a user's own words get matched against.
|
||||
3. **Use when …**, naming situations in the words a user would actually type: the
|
||||
symptom ("a campaign that finds nothing"), the artifact in front of them
|
||||
(`LLVMFuzzerTestOneInput`, `project.yaml`, an ASan stack trace), the decision
|
||||
("choosing between libFuzzer and AFL++"). Two to four situations, not one.
|
||||
|
||||
Anchor the domain if the wording would otherwise match unrelated repositories, and
|
||||
add a closing "Not for …" when a sibling skill owns the adjacent case.
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Situation | Action |
|
||||
|
||||
+3
-8
@@ -8,8 +8,7 @@ Use this template for domain-specific security testing (cryptographic testing, w
|
||||
---
|
||||
name: {domain-name-lowercase}
|
||||
type: domain
|
||||
description: >
|
||||
{Summary of domain and testing approach}. Use when {trigger conditions}.
|
||||
description: "{What this skill does for the reader, leading with the task rather than defining the domain}. Covers {the specific tools, formats, and checks}. Use when {the situations a user describes, in the words they would type — the symptom, the file, the error, the question}."
|
||||
---
|
||||
|
||||
# {Domain Name}
|
||||
@@ -313,9 +312,7 @@ When generating a domain skill, map to relevant tool and technique skills:
|
||||
---
|
||||
name: crypto-testing
|
||||
type: domain
|
||||
description: >
|
||||
Methodology for testing cryptographic implementations.
|
||||
Use when auditing crypto code, validating implementations, or testing for timing attacks.
|
||||
description: "Tests cryptographic implementations against known attacks and edge cases using standard test vectors and timing analysis. Covers vector formats, result flags, and reading a failure. Use when auditing crypto code for correctness or timing leaks, checking a library against standard test vectors, or investigating why two implementations disagree on the same input."
|
||||
---
|
||||
|
||||
# Cryptographic Testing
|
||||
@@ -453,9 +450,7 @@ Essential for code handling secrets.
|
||||
---
|
||||
name: web-security-testing
|
||||
type: domain
|
||||
description: >
|
||||
Methodology for web application security testing.
|
||||
Use when auditing web apps, APIs, or web-based services.
|
||||
description: "Tests web applications and APIs for the vulnerability classes that matter, from injection and access control to session handling. Covers proxy setup, request tampering, and triaging what a scanner reports. Use when auditing a web app or HTTP API, testing an authentication or session flow, or working through a scanner finding on a live target."
|
||||
---
|
||||
|
||||
# Web Security Testing
|
||||
|
||||
+2
-5
@@ -8,8 +8,7 @@ Use this template for language-specific fuzzers (libFuzzer, AFL++, cargo-fuzz, e
|
||||
---
|
||||
name: {fuzzer-name-lowercase}
|
||||
type: fuzzer
|
||||
description: >
|
||||
{Summary from handbook}. Use for fuzzing {language} projects.
|
||||
description: "Sets up and runs {fuzzer} on {language} projects{, plus what makes it the right choice}. Covers harness structure, {build flags and instrumentation}, corpus and dictionary management, and crash triage. Use when {the situations a user describes — writing or debugging a specific harness API, starting on a particular kind of target, choosing between this fuzzer and its alternatives, or diagnosing a campaign that finds nothing}."
|
||||
---
|
||||
|
||||
# {Fuzzer Name}
|
||||
@@ -331,9 +330,7 @@ When generating a fuzzer skill, identify and link to these technique skills:
|
||||
---
|
||||
name: libfuzzer
|
||||
type: fuzzer
|
||||
description: >
|
||||
Coverage-guided fuzzer built into LLVM. Use for fuzzing C/C++ projects
|
||||
that can be compiled with Clang.
|
||||
description: "Sets up and runs libFuzzer, the coverage-guided fuzzer built into LLVM, on C/C++ code that compiles with Clang. Covers harness structure, -fsanitize=fuzzer builds, corpus and dictionary management, sanitizer integration, and campaign triage. Use when writing or debugging an LLVMFuzzerTestOneInput harness, starting fuzzing on a C/C++ library, choosing between libFuzzer and AFL++, or working out why a libFuzzer run finds nothing."
|
||||
---
|
||||
|
||||
# libFuzzer
|
||||
|
||||
+3
-8
@@ -8,8 +8,7 @@ Use this template for cross-cutting techniques that apply to multiple tools (har
|
||||
---
|
||||
name: {technique-name-lowercase}
|
||||
type: technique
|
||||
description: >
|
||||
{Summary of what this technique does}. Use when {trigger conditions}.
|
||||
description: "{What applying this technique achieves, leading with the outcome rather than naming the technique}. Covers {the concrete steps, tools, and flags}. Use when {the situations a user describes, in the words they would type — the symptom, the stalled campaign, the error, the question}."
|
||||
---
|
||||
|
||||
# {Technique Name}
|
||||
@@ -265,9 +264,7 @@ When generating a technique skill, create bidirectional links:
|
||||
---
|
||||
name: fuzz-harness-writing
|
||||
type: technique
|
||||
description: >
|
||||
Techniques for writing effective fuzzing harnesses. Use when creating
|
||||
new fuzz targets or improving existing harness code.
|
||||
description: "Designs and improves fuzzing harnesses for C/C++ and Rust. Covers mapping raw bytes onto a target API, generating structured inputs, avoiding non-determinism and false crashes, and deciding what to fuzz together. Use when writing a first fuzz target, when a campaign finds nothing or reports crashes that will not reproduce, or when the target API needs structured rather than raw input."
|
||||
---
|
||||
|
||||
# Writing Fuzzing Harnesses
|
||||
@@ -453,9 +450,7 @@ fuzz_target!(|data: &[u8]| {
|
||||
---
|
||||
name: address-sanitizer
|
||||
type: technique
|
||||
description: >
|
||||
Memory error detection for C/C++ fuzzing. Use when fuzzing C/C++ code
|
||||
to detect memory corruption bugs like buffer overflows and use-after-free.
|
||||
description: "Builds and runs code under AddressSanitizer to catch buffer overflows, use-after-free, and other memory errors during fuzzing or tests. Covers -fsanitize=address builds, ASAN_OPTIONS, reading the crash report, LeakSanitizer, and the overhead and platform trade-offs. Use when fuzzing C/C++ or Rust that has unsafe blocks or FFI, when debugging a memory corruption crash, or when reading an ASan stack trace."
|
||||
---
|
||||
|
||||
# AddressSanitizer (ASan)
|
||||
|
||||
+2
-5
@@ -8,8 +8,7 @@ Use this template for static analysis tools (Semgrep, CodeQL) and similar standa
|
||||
---
|
||||
name: {tool-name-lowercase}
|
||||
type: tool
|
||||
description: >
|
||||
{Summary from handbook}. Use when {trigger conditions based on tool purpose}.
|
||||
description: "{What the tool does for the reader, leading with the task rather than defining the tool}. Covers {the concrete commands, config files, and output formats}. Use when {the situations a user describes, in the words they would type — the file in front of them, the error, the question}."
|
||||
---
|
||||
|
||||
# {Tool Name}
|
||||
@@ -259,9 +258,7 @@ Not all sections apply to every tool. Use this guide:
|
||||
---
|
||||
name: semgrep
|
||||
type: tool
|
||||
description: >
|
||||
Fast static analysis for finding bugs, detecting vulnerabilities, and enforcing code standards.
|
||||
Use when scanning code for security issues, enforcing patterns, or integrating into CI/CD pipelines.
|
||||
description: "Scans code with Semgrep to find bugs, vulnerabilities, and pattern violations across a repository. Covers rule syntax, running the registry rulesets, writing custom rules, and wiring results into CI. Use when scanning a codebase for security issues, writing or debugging a Semgrep rule, triaging semgrep output, or adding a static analysis gate to a pipeline."
|
||||
---
|
||||
|
||||
# Semgrep
|
||||
|
||||
@@ -124,7 +124,11 @@ DESC=$(yq '.description' "$SKILL")
|
||||
- No XML/HTML tags in name or description (pattern: `<[^>]+>`)
|
||||
- No reserved words ("anthropic", "claude") in name
|
||||
- `type` field ensures correct section validation (if missing, type is inferred from content)
|
||||
- Description should include both "what" (tool purpose) and "when" (trigger conditions)
|
||||
- Description leads with what the skill does for the reader, not a definition of the tool,
|
||||
names the concrete commands, flags, and API symbols it covers, and closes with two or
|
||||
more situations in the words a user would type. "Coverage-guided fuzzer built into LLVM.
|
||||
Use for fuzzing C/C++ code" has a what and a when and still routes to nothing — see
|
||||
"Description quality" in `agent-prompt.md` for the bar.
|
||||
- No Hugo shortcodes in frontmatter (pattern: `\{\{[<%]`)
|
||||
|
||||
**Trigger phrase validation:**
|
||||
@@ -346,7 +350,8 @@ Before delivering each generated skill:
|
||||
- [ ] Warnings reviewed and addressed (or documented as acceptable)
|
||||
|
||||
### Content Quality (manual review)
|
||||
- [ ] Description includes what AND when
|
||||
- [ ] Description leads with the task, names concrete anchors, and gives two or more situations
|
||||
- [ ] Every command, flag, file name, and API symbol the description advertises appears in the skill body
|
||||
- [ ] When to Use section has clear triggers
|
||||
- [ ] Quick Reference is actionable
|
||||
- [ ] Code examples are complete and runnable
|
||||
@@ -419,7 +424,7 @@ After validation, document results:
|
||||
|
||||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| YAML parse error | Bad indentation in description | Use `>` for multi-line |
|
||||
| YAML parse error | Unquoted description containing `: ` or ` #` | Put the description on one double-quoted line |
|
||||
| Missing section | Template not fully populated | Fill from handbook |
|
||||
| Over 500 lines | Too much detail in main file | Split to supporting files |
|
||||
| Broken reference | Supporting file not created | Create file or remove link |
|
||||
@@ -446,7 +451,7 @@ After each generation run, systematically review and improve the generator.
|
||||
| Shortcodes | Were there shortcodes or formats not handled? | `discovery.md` (section 3.2) |
|
||||
| Manual fixes | Did any skills require manual fixes after generation? | Templates or agent prompt |
|
||||
| Detection | Are there patterns in the handbook not detected? | `discovery.md` (section 1.3) |
|
||||
| Activation | Did activation testing reveal description issues? | Templates (description guidance) |
|
||||
| Activation | Did activation testing reveal description issues? | `agent-prompt.md` ("Description quality") and the templates |
|
||||
| Validation | Did a bug slip through validation? | `testing.md` (add new check) |
|
||||
|
||||
### Improvement Log Format
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
---
|
||||
name: wycheproof
|
||||
type: domain
|
||||
description: >
|
||||
Wycheproof provides test vectors for validating cryptographic implementations.
|
||||
Use when testing crypto code for known attacks and edge cases.
|
||||
description: "Validates cryptographic implementations against Project Wycheproof's test vectors, which encode known attacks and edge cases across AES, RSA, ECDSA, ECDH, and more. Covers loading test vectors, mapping result flags onto pass and fail expectations, and reading a failure. Use when testing a crypto implementation against known attacks, checking a library against standard test vectors, or investigating why two implementations disagree on the same input."
|
||||
---
|
||||
|
||||
# Wycheproof
|
||||
|
||||
Reference in New Issue
Block a user