mirror of
https://github.com/trailofbits/skills.git
synced 2026-09-14 14:28:48 +08:00
Clinton Thomas
798b591aa4
* Convert spec-to-code-compliance to a dynamic workflow
The skill held a fixed seven-phase plan and drove a subagent from prose, which
is the shape Maker Week asks us to move into a script. Phase 3 required
line-by-line YAML IR for every function in the codebase before Phase 4 could
start, so on any real target the window was exhausted partway through and the
remaining requirements got a plausible check rather than a real one.
workflows/spec-compliance.js inverts that. Requirements are extracted once, then
each one gets its own agent to hunt the code with, so no context ever holds the
whole behavioral model. Divergences go to two agents that did not produce them,
one re-reading the code and one re-reading the document, and what either refutes
is dropped. A separate agent sweeps the reverse direction, which the
requirement-driven pass cannot cover.
The removed resources are replaced by the workflow's schemas: OUTPUT_REQUIREMENTS
set minimum item counts that a spec with fewer requirements can only meet by
inventing them, alongside a "zero speculation" rule in the same file.
COMPLETENESS_CHECKLIST was a self-verification pass. IR_EXAMPLES demonstrated
YAML formats the schemas now enforce. SKILL.md keeps the judgment that stays
judgment: which verdicts matter, and when a gap is a code fix or a docs fix.
Fixes a classification bug in passing. SKILL.md called undocumented behavior
UNDOCUMENTED CODE PATH while the other two resources called it
code_stronger_than_spec, and the former was not one of the six legal match_type
values, so following the skill emitted a verdict outside its own enum.
commands/spec-compliance.md is removed: it forwarded two arguments to the skill,
required naming the spec that Phase 0 exists to discover, and would have collided
with the workflow on the same slash command.
Adds three eval cases. routes-not-inline measures Δ +1.00 for dispatch.
name-is-not-evidence and documents-contradict both measure Δ 0.00 — Opus 5
handles those unaided — and say so in their own descriptions rather than
implying coverage they do not have.
* Restore domain guidance dropped in the workflow conversion
The conversion deleted IR_EXAMPLES.md on the grounds that the workflow's schemas
replace it. The schemas replace the YAML formats it demonstrated; they do not
replace knowing that `unchecked` suspends a guarantee the calling code is written
as though it still has, or that a 0.3% fee is implemented as `amountIn * 997`
and matches nothing you can grep for. A Solidity user brought the same spec and
got a checker with no calibration for it.
audit-context-building kept its worked examples and added DOMAIN_NOTES.md for the
same cross-domain problem. Following that:
- DOMAIN_NOTES.md maps what counts as a specification, what enforcement looks
like, and where it hides across contracts, C and C++, services, and decompiled
firmware, plus scoping a check against an RFC or standard rather than a project
document. Generalizing past contracts was previously done by deleting the
"non-blockchain" exclusion and adding nothing.
- WORKED_EXAMPLE.md carries three requirements to a verdict, one per verdict
that is easy to get wrong: arithmetic that satisfies a requirement it does not
resemble, an absence whose credibility is the search record, and enforcement
present on every path but the one nobody tested.
- ANALYSIS_FORMAT.md holds the on-disk format in one place. The agent and the
workflow prompt were both describing it, which is the duplication this
conversion was meant to remove.
README gains a migration note: the 1.x command is gone, arguments inverted, PDFs
work and DOCX never did, and the report is no longer a fixed 16 sections.
* Survive an unresolvable checker agent instead of failing the phase
Running the workflow end to end for the first time failed all six requirement
checks with "agent type 'spec-to-code-compliance:spec-compliance-checker' not
found" and returned nothing salvageable. The proximate cause was a stale session
— the plugin had been installed after the session started, so its agents were not
in the registry — but a hard dependency on a namespaced agentType with no
fallback turns any resolution failure into total loss of the run.
checkRequirement now retries on the default agent with the checker's load-bearing
rules inlined, and logs once that it did. The fan-out is concurrent, so every
item in the first batch attempts the specialized agent before the flag is set;
those attempts fail at spawn without consuming tokens, and later batches skip
straight to the fallback.
The prompt no longer says "in the format your instructions define", which was
only true on the specialized path and left the fallback with no format at all.
Verified end to end against evals/documents-contradict/fixture: 26 requirements
extracted from two documents, 6 checked, 3 divergences found and none refuted —
the operator zeroing balances via reassign (critical), and the Senior-tier
collateral bypass from both directions (high). The SPEC/README fee contradiction
was reported as a documentation fix with the note that README frames it as
deliberate. The report named all 20 unchecked requirements as unknown rather than
compliant, and declined to give the unchecked fee requirements a verdict while
still reporting the contradiction as a direct observation.
Confirms the id-collision fix in bf96722 was a real bug, not a hypothetical: both
documents numbered from REQ-01 and the second series landed as REQ-01-2 onward.
* Address PR review: honesty gaps and a fragile fallback
allowed-tools omitted Workflow, the tool the skill's only instruction requires.
audit-context-building has the same omission, so its conversion needs the same
fix.
Requirement checks that died were dropped by `checked.filter(Boolean)` with no
record anywhere: the report could present a short alignment matrix with no
indication that four of ten requirements were never checked, and
`requirementsChecked` was indistinguishable from a smaller selection. pipeline()
preserves input order, so the nulls name them — now logged, passed to the report
as unknown-rather-than-compliant, and returned as `checksFailed`.
The fallback's error matching was verified against the message an unresolvable
agent type actually emits, not guessed, but gating recovery on any phrasing is
the fragility the review identifies: a reworded runtime message restores the
outage. It now falls back on any failure of the typed dispatch and uses the
message only to decide whether to latch, so a transient error retries one
requirement instead of downgrading the rest.
Refuted divergences left no trace in the report. A single refuter drops a
finding, so one over-confident refutation could lose a real divergence with
nothing for a reader to notice. The report now carries a "considered and dropped"
list with the reasoning.
The report path was defaulted to the expected location whether or not the write
happened, so a schema-valid summary with no Write looked like a successful run.
The script cannot check the filesystem, so it no longer invents the path:
`report` is null with a warning when the agent named none, and `analysisFile` is
now required in ALIGNMENT_SCHEMA so an unwritten analysis fails the schema rather
than being silently absent from a file the report tells the reader to open.
`limit: 0` passed the nullish check, selected nothing, and reported it as every
check failing. Clamped below as well as above.
Both READMEs and SKILL.md documented `{path, spec, limit}` as slash-command
input; a slash command passes a string, so the object literal became the path.
The root README table still described the plugin as being for blockchain audits.
---------
Co-authored-by: kz-tob <kara.zaffarano@trailofbits.com>
Spec-to-Code Compliance
Check code against the documentation that specifies it. Every gap is either a bug or a documentation fix, and which one it is is the finding.
Author: Omar Inuwa
Install
/plugin install trailofbits/skills/plugins/spec-to-code-compliance
Use
/spec-to-code-compliance:spec-compliance ./contracts
The slash command takes a path. To name the specification directly or widen the fan-out, ask for the run in
words — "run spec-compliance on ./contracts against SPEC.md, checking 20 requirements" — and the values reach
the script as {path, spec, limit}. Typing that object after the slash command does not work: it arrives as a
string and is treated as the path.
Writes spec-compliance/REPORT.md and one analysis per requirement under spec-compliance/requirements/. The
session gets the alignment matrix and the surviving divergences, not the analysis.
How it works
- Extract — find the documents describing intended behavior, and split them into individually checkable requirements, quoted verbatim. Compound claims are split: a sentence requiring two things is two requirements, because the code can get one right and the other wrong.
- Align — one agent per requirement hunts the code for that requirement alone, reading the enforcement, its callees, and its callers. A separate agent sweeps the reverse direction for behavior no document mentions.
- Verify — each divergence goes to two agents that did not produce it, one re-reading the code and one re-reading the document, both trying to refute it. What either knocks down is dropped.
- Report — alignment matrix, surviving divergences worst first, undocumented behavior, and the problems in the documentation itself.
Per-requirement fan-out is what makes the check honest. Judging a requirement means reading a call chain; thirty requirements is thirty call chains, which does not fit one context window. Done inline, the first few requirements get a real check and the rest get a plausible one — and a verdict resting on a promising function name reads exactly like one resting on having read the function.
Verdicts
| Verdict | Meaning |
|---|---|
implemented |
The enforcement was found and read |
partial |
Holds on some paths, not others — usually the most serious verdict in a report |
contradicted |
The code does something incompatible with the requirement |
absent |
Looked and it is not there; rests entirely on the recorded searches |
stronger-than-spec |
The code enforces more than the document asks, so nothing records the dependency |
undecidable |
The requirement is too vague to check — a finding against the document |
Components
workflows/spec-compliance.js— the orchestrationagents/spec-compliance-checker.md— the per-requirement worker; dispatch it directly for a single requirementresources/ANALYSIS_FORMAT.md— the on-disk format for a per-requirement analysisresources/DOMAIN_NOTES.md— what counts as a specification and where enforcement hides, across contracts, C and C++, services, and firmware, plus scoping a check against an RFC or standardresources/WORKED_EXAMPLE.md— three requirements chased to a verdict, for calibrationresources/DIVERGENCE_RUBRIC.md— severity, and the two directions of a gap
Resources live under skills/spec-to-code-compliance/.
Migrating from 1.x
/trailofbits:spec-compliance <spec> <codebase> is removed. The entry point is
/spec-to-code-compliance:spec-compliance <path>, and the arguments inverted: the path comes first and the
specification is discovered rather than named. Pass {path, spec} when you want to point at it directly.
PDF whitepapers work. DOCX and Notion exports do not — 1.x claimed to normalize them and had no converter; the discovery step now names an unreadable document instead of guessing at its contents.
The report is no longer a fixed 16 sections. Sections with nothing in them are left out rather than filled.
Related
audit-context-building— build the system model first when the code is unfamiliarissue-writer— turn surviving divergences into client-facing findings