skills/trailofbits__skills
1
0
Fork 0
mirror of https://github.com/trailofbits/skills.git synced 2026-09-14 14:28:48 +08:00
Code Issues Packages Projects Releases Wiki Activity

static-analysis: convert the semgrep scan fan-out to a dynamic workflow (#231)

Browse Source 
* feat(static-analysis): ship /static-analysis:semgrep-scan and a scan runner

Two entry points over one implementation. SKILL.md keeps its gated five-step
path, where the user reviews and edits the ruleset list before anything runs.
workflows/semgrep-scan.js runs the same scan end to end without stopping. Both
read the same references/, so a ruleset added to rulesets.md reaches both at
once. This is the shape variant-analysis uses.

The workflow sits at the plugin root beside the four already on main and ships
as /static-analysis:semgrep-scan. Four phases: Detect resolves the output
directory and profiles languages and Pro, Select reads references/rulesets.md
and writes rulesets.json, Scan runs the script, Report post-filters, merges and
summarizes.

Step 4 is skills/semgrep/scripts/run-scans.sh, not a fan-out of subagents.
Nothing in the scan phase needs judgement: the agents would run fixed commands
and report $? and a jq count. Exit codes now stay with the processes that
produced them and finding counts come from the JSON they wrote, so no phase
reports on work a later phase has to go behind and re-verify. --metrics=off, the
--include scoping, the output-directory --exclude and the severity flags are
properties of the script. Cross-language rulesets run once rather than once per
language and never take --include; a ruleset already in baseline is dropped from
its language; language keys fold onto a canonical name, so js and javascript are
one unit; two spellings of one repository collapse to one clone. Parallelism is
the script's --jobs.

The workflow does not stop for ruleset approval. None of main's four ask the
user anything, and invoking one with a target is the opt-in. The scan is
read-only over the target -- no --autofix or --fix is ever passed, every write
lands inside the output directory, and the script refuses to run when the output
directory is the target. Semgrep rules are declarative YAML, so pointing --config
at a cloned rule repository executes nothing from it. The gate was scope
confirmation rather than protection from a dangerous action, and what ran is
recorded in rulesets.json and scans.json either way. A deliberate change to a
security skill's stated policy, not an oversight; the gated path remains for when
the ruleset selection is the thing that matters.

Fixes four defects the old prose path carried: allowed-tools omitted the tool
Step 4 needed; --severity MEDIUM/HIGH/CRITICAL is rejected by semgrep, so
important-only mode never ran; each scanner deleted the shared repos/ clone while
others were still reading it; and the scanner agent declared its tools as a comma
string where the loader expects a list.

Two bugs the new suites found while being written. `eval "$cmd" &` reports exit
status 1 whatever the command exited with, which marked every failed scan as a
success; commands are built as an argv array and executed directly, which also
leaves no quoting surface. And the target was resolved with `cd && pwd` while the
output directory was not, so on any path crossing a symlink -- every path under
/var on macOS -- both the equality check and the inside-the-target test missed,
and the run scanned its own cloned rule repositories.

tests/run_scan_tests.sh covers the script: command generation via --dry-run, and
execution, exit codes and clone failures against stub semgrep and git binaries.
tests/workflow-harness.js compiles the workflow with stubbed globals and asserts
that a relative target, an output directory equal to the target, a dead phase and
a failed scan each stop the run rather than reach the report as an empty result;
--self-test mutates the workflow six ways and requires every mutation to turn a
scenario red. Both are hermetic, reach no network, and CI's existing shell-suite
discovery runs them, so no workflow file changes.

Removes references/scanner-task-prompt.md and the no-Workflow fallback it served.
There is no hand-rolled path and no second implementation in prose.

* fix(static-analysis): resolve the skill dir at runtime, filter the merged SARIF, clear stale raw output

Three defects flagged by kz-tob on #231.

The workflow hardcoded SKILL_DIR as a repo-relative path, so every scripted
command only resolved inside a checkout of this repo. A marketplace install
runs with the user's own project as cwd and the scan phase would have found
no run-scans.sh at all. Resolved at runtime instead, folded into the Detect
phase, following the cascade variants.js already uses. Each candidate ends at
scripts/run-scans.sh, which makes a stale install self-excluding: verified
against the 1.2.2 install on disk, which ships merge_sarif.py and nothing
else, and the glob correctly declines to bind to it. An unresolved directory
throws rather than leaving an agent to compose semgrep commands by hand.

In important-only mode both the workflow and scan-workflow.md told the agent
to apply the scan-modes.md jq filter to the merged SARIF. That filter reads
.results[].extra.metadata, which SARIF does not have, so it exits with
"Cannot iterate over null" and results.sarif stayed unfiltered while the JSON
side was filtered. The metadata is not recoverable from SARIF, but finding
identity is: (check_id, path, start.line) and (ruleId, uri, region.startLine)
match field-for-field, confirmed against real semgrep output, and it is the
same triple the merge already dedups on. merge_sarif.py --important keeps the
findings the JSON filter kept and fails rather than filtering if any scan has
no *-important.json beside it, since a partial key set would drop real
findings from the deliverable. The merge command blocks in SKILL.md and
scan-workflow.md show both modes, so copying the block without reading the
paragraph under it cannot produce an unfiltered deliverable.

run-scans.sh never cleared raw/. merge_sarif.py globs every *.sarif there, so
a rerun into a reused output directory that dropped a ruleset still merged
the previous run's output for it.

Tests: 58 shell assertions (+3), 46 workflow assertions (+13) with three new
mutations, and 16 pytest cases for merge_sarif.py. Each fix is mutation-tested;
reverting any one of them turns the suite red.

* fix(static-analysis): honor a bare-path arg and guard the important-only merge

* fix(static-analysis): fail the important-only post-filter loudly on a jq error

* fix(static-analysis): report merge failures and exclude failed scans

* fix(static-analysis): log the approved plan once and flag zero-coverage rulesets

* fix(static-analysis): drop the SARIF Multitool merge path

* fix(static-analysis): report SARIF files the merge could not read

* fix(static-analysis): record the exclude pattern applied to every scan

* fix(static-analysis): move the exclude-pattern assertions after their fixtures

* fix(static-analysis): stop SIGPIPE marking a healthy rule repo as empty

* test(static-analysis): pin exclusion for a target holding glob metacharacters

* docs(static-analysis): describe the semgrep skill as it now runs

---------

Co-authored-by: kz-tob <kara.zaffarano@trailofbits.com>
This commit is contained in:
Akshay K
2026-08-10 16:03:13 -04:00
committed by GitHub
parent 7b9bd5f950
commit 94abac91ca
15 changed files with 3162 additions and 430 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/static-analysis/.claude-plugin/plugin.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "static-analysis",
"version": "1.2.2",
"version": "1.3.0",
"description": "Static analysis toolkit with CodeQL, Semgrep, and SARIF parsing for security vulnerability detection",
"author": {
"name": "Axel Mierczuk & Paweł Płatek"
plugins/static-analysis/README.md
+45 -5
View File
@@ -47,12 +47,52 @@ Use this plugin when you need to:
- Aggregate and deduplicate results from multiple files
- CI/CD integration patterns
## Agents Included
## Running a Semgrep scan
| Agent | Tools | Purpose |
|--------------------|------------------------|----------------------------------------------------------------|
| `semgrep-scanner` | Bash | Executes parallel semgrep scans for a language category |
| `semgrep-triager` | Read, Grep, Glob, Write | Classifies findings as true/false positives by reading source |
Two entry points over one implementation.
```
/static-analysis:semgrep-scan {"target": "/abs/path", "mode": "run-all"}
```
runs the scan end to end: detect languages and Pro, select rulesets, scan, merge, report. It
does not stop to have the ruleset list approved — invoking it with a target is the opt-in. That
is safe because the scan is read-only over the target: no `--autofix`, every write inside the
output directory, and it refuses to run when the output directory is the target.
Ask for the `semgrep` skill instead when the ruleset selection is the thing that matters. Its
five-step path presents the list and waits for approval before anything runs. Both paths read
the same `references/`, so a ruleset added to `rulesets.md` reaches both.
## Workflows Included
| Workflow | Purpose |
|----------|---------|
| `workflows/semgrep-scan.js` | Ships as `/static-analysis:semgrep-scan`. Four phases: Detect, Select, Scan, Report |
## Scripts Included
| Script | Purpose |
|--------|---------|
| `skills/semgrep/scripts/run-scans.sh` | Builds every semgrep command from the selected rulesets, clones the third-party rule repos, runs the scans in batches and writes `scans.json` |
| `skills/semgrep/scripts/merge_sarif.py` | Merges the per-scan SARIF into one `results.sarif` |
Generating the commands in one place is what makes `--metrics=off`, the `--include` scoping
rule, and the output-directory `--exclude` properties of the code rather than instructions a
model can drop.
No subagent runs any part of the scan — the workflow's Scan phase calls `run-scans.sh`. Exit
codes come from the semgrep processes and finding counts from the JSON they wrote, so nothing in
the result is a self-report that a later step has to go behind and verify.
## Tests
Both suites are hermetic, reach no network, and are discovered by CI's existing shell-suite step.
| Suite | Covers |
|-------|--------|
| `tests/run_scan_tests.sh` | `run-scans.sh`: command generation via `--dry-run`, plus execution, exit codes and clone failures against stub `semgrep` and `git` binaries |
| `tests/run_workflow_tests.sh` | `semgrep-scan.js`, via `tests/workflow-harness.js`, which compiles it with stubbed globals. `--self-test` mutates the workflow and requires every mutation to turn a scenario red |
## Installation
plugins/static-analysis/agents/semgrep-scanner.md
-90
View File
@@ -1,90 +0,0 @@
---
name: semgrep-scanner
description: "Executes Semgrep CLI scans for a specific language category and produces SARIF output. Spawned by the semgrep skill as a parallel worker — one agent per detected language."
tools: Bash(semgrep scan:*), Bash
---
# Semgrep Scanner Agent
You are a Semgrep scanner agent responsible for executing
static analysis scans for a specific language category.
## Core Rules
1. **Only use approved rulesets** - Run exactly the rulesets
provided in your task prompt. Never add or remove rulesets.
2. **Always use `--metrics=off`** - Prevents sending telemetry
to Semgrep servers. No exceptions.
3. **Use `--pro` when available** - If the task indicates Pro
engine is available, always include the `--pro` flag for
cross-file taint tracking.
4. **Parallel execution** - Run all rulesets simultaneously
using `&` and `wait`. Never run rulesets sequentially.
## Scan Command Pattern
For each approved ruleset, generate and run:
```bash
semgrep [--pro if available] \
--metrics=off \
--config [RULESET] \
--json -o [OUTPUT_DIR]/[lang]-[ruleset-name].json \
--sarif-output=[OUTPUT_DIR]/[lang]-[ruleset-name].sarif \
[TARGET] &
```
After launching all rulesets:
```bash
wait
```
## Language Scoping
For language-specific rulesets (e.g., `p/python`, `p/java`),
add `--include` to restrict parsing to relevant files:
```bash
--include="*.java" --include="*.jsp" # for Java
--include="*.py" # for Python
--include="*.js" --include="*.jsx" # for JavaScript
```
Do NOT add `--include` to cross-language rulesets like
`p/security-audit`, `p/secrets`, or third-party repos that
contain rules for multiple languages.
## GitHub URL Rulesets
For rulesets specified as GitHub URLs (e.g.,
`https://github.com/trailofbits/semgrep-rules`):
- Clone into `[OUTPUT_DIR]/repos/[repo-name]` so cloned
repos stay inside the results directory
- Use the local path as the `--config` value (do NOT pass
the URL directly — semgrep's URL handling is unreliable
for repos with non-standard YAML)
- After all scans complete, delete the cloned repos:
`[ -n "[OUTPUT_DIR]" ] && rm -rf [OUTPUT_DIR]/repos`
## Output Requirements
After all scans complete, report:
- Number of findings per ruleset
- Any scan errors or warnings
- File paths of all generated JSON and SARIF results
- If Pro was used, note any cross-file findings detected
## Error Handling
- If a ruleset fails to download, report the error but
continue with remaining rulesets
- If semgrep exits non-zero for a scan, capture stderr and
include in report
- Never silently skip a failed ruleset
## Full Reference
For the complete scanner task prompt template with variable
substitutions and examples, see:
`{baseDir}/skills/semgrep/references/scanner-task-prompt.md`
plugins/static-analysis/skills/semgrep/SKILL.md
+107 -41
View File
@@ -1,27 +1,31 @@
---
name: semgrep
description: >-
Run Semgrep static analysis scan on a codebase using parallel subagents.
Supports two scan modes — "run all" (full ruleset coverage) and "important
only" (high-confidence security vulnerabilities). Automatically detects and
uses Semgrep Pro for cross-file taint analysis when available. Use when asked
to scan code for vulnerabilities, run a security audit with Semgrep, find
bugs, or perform static analysis. Spawns parallel workers for multi-language
codebases.
allowed-tools: Bash Read Glob Task AskUserQuestion TaskCreate TaskList TaskUpdate
Runs a Semgrep security scan over a codebase: detects languages, selects
rulesets, presents the plan for explicit approval, then runs every approved
ruleset through scripts/run-scans.sh, which batches the semgrep processes and
writes scans.json, and merges the output to SARIF. Supports two scan modes,
"run all" for full ruleset coverage and "important only" for security
findings at medium-to-high confidence and impact. Uses Semgrep Pro for
cross-file taint analysis when it is available. Use when asked to scan code
for vulnerabilities, run a security audit with Semgrep, find bugs, or perform
static analysis. For the same scan without the approval gate, use the
/static-analysis:semgrep-scan workflow.
allowed-tools: Bash Read Glob AskUserQuestion TaskCreate TaskList TaskUpdate
---
# Semgrep Security Scan
Run a Semgrep scan with automatic language detection, parallel execution via Task subagents, and merged SARIF output.
Run a Semgrep scan with automatic language detection, parallel execution, and merged SARIF output.
## Essential Principles
1. **Always use `--metrics=off`** — Semgrep sends telemetry by default; `--config auto` also phones home. Every `semgrep` command must include `--metrics=off` to prevent data leakage during security audits.
2. **User must approve the scan plan (Step 3 is a hard gate)** — The original "scan this codebase" request is NOT approval. Present exact rulesets, target, engine, and mode; wait for explicit "yes"/"proceed" before spawning scanners.
3. **Third-party rulesets are required, not optional** — Trail of Bits, 0xdea, and Decurity rules catch vulnerabilities absent from the official registry. Include them whenever the detected language matches.
4. **Spawn all scan Tasks in a single message** — Parallel execution is the core performance advantage. Never spawn Tasks sequentially; always emit all Task tool calls in one response.
4. **`scripts/run-scans.sh` generates the commands; do not write them yourself** — it builds every `semgrep` line from the approved list. That is what makes `--metrics=off`, the `--include` scoping rule, and the parallel dispatch properties of the code rather than instructions. Give it the approved rulesets and let it run.
5. **Always check for Semgrep Pro before scanning** — Pro enables cross-file taint tracking and catches ~250% more true positives. Skipping the check means silently missing critical inter-file vulnerabilities.
6. **Report what did not run** — `scans.json` carries `failed` and `skipped` alongside `scans`. A ruleset whose repo would not clone, or whose scan exited non-zero, must appear in the report. A partial scan presented as a complete one is worse than no scan.
## When to Use
@@ -66,12 +70,15 @@ The output directory is resolved **once** at the start of Step 1 and used throug
```
$OUTPUT_DIR/
├── rulesets.txt # Approved rulesets (logged after Step 3)
├── rulesets.json # The approved plan (Step 3), read by run-scans.sh (Step 4)
├── scans.json # What ran, failed, skipped, and covered nothing (Step 4)
├── raw/ # Per-scan raw output (unfiltered)
│ ├── python-python.json
│ ├── python-python.json # <language>-<ruleset> for language-scoped rules
│ ├── python-python.sarif
│ ├── python-django.json
│ ├── python-django.sarif
│ ├── all-security-audit.json # all-<ruleset> for cross-language rules, run once
│ ├── all-security-audit.sarif
│ └── ...
└── results/ # Final merged output
└── results.sarif
@@ -84,14 +91,22 @@ $OUTPUT_DIR/
**Optional:** Semgrep Pro — enables cross-file taint tracking, inter-procedural analysis, and additional languages (Apex, C#, Elixir). Check with:
```bash
semgrep --pro --validate --config p/default 2>/dev/null && echo "Pro available" || echo "OSS only"
# --metrics=off because Principle 1 has no exceptions, and this is the first semgrep command
# of a run. stderr is kept because "OSS only" has several causes (logged out, no subscription,
# registry blocked) and the run downgrades silently for all of them.
if PRO_ERR=$(semgrep --pro --validate --metrics=off --config p/default 2>&1); then
echo "Pro available"
else
echo "OSS only"
echo " reason: $(printf '%s' "$PRO_ERR" | tail -n 3)"
fi
```
**Limitations:** OSS mode cannot track data flow across files. Pro mode uses `-j 1` for cross-file analysis (slower per ruleset, but parallel rulesets compensate).
## Scan Modes
Select mode in Step 2 of the workflow. Mode affects both scanner flags and post-processing.
Select mode in Step 2. Mode affects both the scan flags and post-processing.
| Mode | Coverage | Findings Reported |
|------|----------|-------------------|
@@ -99,7 +114,7 @@ Select mode in Step 2 of the workflow. Mode affects both scanner flags and post-
| **Important only** | All rulesets, pre- and post-filtered | Security vulns only, medium-high confidence/impact |
**Important only** applies two filter layers:
1. **Pre-filter**: `--severity MEDIUM --severity HIGH --severity CRITICAL` (CLI flag)
1. **Pre-filter**: `--severity WARNING --severity ERROR` (CLI flag)
2. **Post-filter**: JSON metadata — keeps only `category=security`, `confidence∈{MEDIUM,HIGH}`, `impact∈{MEDIUM,HIGH}`
See [scan-modes.md](references/scan-modes.md) for metadata criteria and jq filter commands.
@@ -108,26 +123,56 @@ See [scan-modes.md](references/scan-modes.md) for metadata criteria and jq filte
```
┌──────────────────────────────────────────────────────────────────┐
│ MAIN AGENT (this skill) │
│ MAIN SESSION (this skill) │
│ Step 1: Detect languages + check Pro availability │
│ Step 2: Select scan mode + rulesets (ref: rulesets.md) │
│ Step 3: Present plan + rulesets, get approval [⛔ HARD GATE] │
│ Step 4: Spawn parallel scan Tasks (approved rulesets + mode) │
│ Step 5: Merge results and report │
│ Step 4: Run scripts/run-scans.sh with the approved rulesets │
│ Step 5: Post-filter, merge, report, delete repos/ │
└──────────────────────────────────────────────────────────────────┘
│ Step 4
│ Step 4: Bash
▼
┌─────────────────┐
│ Scan Tasks │
│ (parallel) │
├─────────────────┤
│ Python scanner │
│ JS/TS scanner │
│ Go scanner │
│ Docker scanner │
└─────────────────┘
┌──────────────────────────────────────────────────────────────────┐
│ scripts/run-scans.sh │
│ clone each third-party repo once, into repos/ │
│ generate one semgrep command per ruleset │
│ ├── python p/python, p/django --include=*.py│
│ ├── javascript p/javascript --include=*.js│
│ ├── docker p/dockerfile │
│ └── cross-language p/security-audit, p/secrets, │
│ the cloned repos (no filter) │
│ run in batches of --jobs, exit code read per process │
│ write scans.json — scans, failed, skipped │
└──────────────────────────────────────────────────────────────────┘
```
The approval gate stays in the session; the script is execution only and asks nothing. The
approved list reaches it as a JSON file, so the scan cannot reach a ruleset the user declined.
Cross-language rulesets go in one shared unit rather than being repeated per language.
`p/security-audit`, `p/secrets`, and the third-party repos scan the whole target unscoped,
so running them once per language ran the identical command N times and left the SARIF
merge to dedup the copies.
## Running it as a Workflow
This plugin ships `/static-analysis:semgrep-scan`, which runs the whole scan end to end:
detect languages and Pro, select rulesets from [rulesets.md](references/rulesets.md), run
`scripts/run-scans.sh`, merge and report. Pass it a JSON object, not prose:
```
/static-analysis:semgrep-scan {"target": "/abs/path", "mode": "run-all"}
```
**It does not stop for ruleset approval.** Invoking it with a target is the opt-in, the same
way `/variant-analysis:variants` works. That is safe to do because the scan is read-only over
the target — no `--autofix`, every write inside the output directory — so the approval gate
below is a scope confirmation rather than a safety one. What ran is recorded in
`rulesets.json` and `scans.json` either way.
Use the workflow when you want the scan run; work the five steps below when the ruleset
selection itself matters and you want to see and edit the list first.
## Workflow
**Follow the detailed workflow in [scan-workflow.md](workflows/scan-workflow.md).** Summary:
@@ -137,24 +182,40 @@ See [scan-modes.md](references/scan-modes.md) for metadata criteria and jq filte
| 1 | Resolve output dir, detect languages + Pro availability | — | Use Glob, not Bash |
| 2 | Select scan mode + rulesets | — | [rulesets.md](references/rulesets.md) |
| 3 | Present plan, get explicit approval | ⛔ HARD | AskUserQuestion |
| 4 | Spawn parallel scan Tasks | — | [scanner-task-prompt.md](references/scanner-task-prompt.md) |
| 5 | Merge results and report | — | Merge script (below) |
| 4 | Run the scans | — | `scripts/run-scans.sh` |
| 5 | Post-filter, merge, report, clean up | — | Merge script (below) |
**Task enforcement:** On invocation, create 5 tasks with blockedBy dependencies (each step blocks the previous). Step 3 is a HARD GATE — mark complete ONLY after user explicitly approves.
**Merge command (Step 5):**
```bash
uv run {baseDir}/scripts/merge_sarif.py $OUTPUT_DIR/raw $OUTPUT_DIR/results/results.sarif
# run-all
uv run {baseDir}/scripts/merge_sarif.py "$OUTPUT_DIR/raw" "$OUTPUT_DIR/results/results.sarif" \
--scans "$OUTPUT_DIR/scans.json"
# important-only, once the JSON post-filter has run over every file in raw/
uv run {baseDir}/scripts/merge_sarif.py "$OUTPUT_DIR/raw" "$OUTPUT_DIR/results/results.sarif" \
--important --scans "$OUTPUT_DIR/scans.json"
```
## Agents
`--scans` drops the output of scans listed under `.failed`. A scan that died part-way may still
have written a `.sarif`, and under `--important` that file has no post-filter beside it, which is
an error rather than an empty filter. Without the flag one dead scan denies every healthy scan a
merged result. The excluded files are named on stdout, so they can go in the report.
| Agent | Tools | Purpose |
|-------|-------|---------|
| `static-analysis:semgrep-scanner` | Bash | Executes parallel semgrep scans for a language category |
The post-filter reads metadata SARIF does not carry, so it cannot be re-run against the merged
file; `--important` instead keeps the findings the JSON filter kept, matched on
`(rule, file, line)`. Without it `results.sarif` is unfiltered while the JSON side is not.
Use `subagent_type: static-analysis:semgrep-scanner` in Step 4 when spawning Task subagents.
## Workflow and agents
| Component | Purpose |
|-----------|---------|
| `scripts/run-scans.sh` | Builds every scan command from the approved rulesets, runs them in batches, and writes `scans.json` |
Step 4 is a Bash call. No subagent runs any part of the scan: exit codes and finding counts are
read from the processes and the JSON they wrote.
## Rationalizations to Reject
@@ -167,7 +228,9 @@ Use `subagent_type: static-analysis:semgrep-scanner` in Step 4 when spawning Tas
| "Add extra rulesets without asking" | Modifying approved list without consent breaks trust |
| "Third-party rulesets are optional" | Trail of Bits, 0xdea, Decurity catch vulnerabilities not in official registry — REQUIRED |
| "Use --config auto" | Sends metrics; less control over rulesets |
| "One Task at a time" | Defeats parallelism; spawn all Tasks together |
| "I'll just run the semgrep commands myself" | `run-scans.sh` is what enforces `--metrics=off`, the `--include` rule and the output-directory `--exclude`. Hand-written commands drop them silently |
| "The script failed, I'll run semgrep directly to get something" | A non-zero exit means no scan succeeded. Report that and stop; a hand-run subset reads as a full scan |
| "Some scans failed, the run still finished" | `failed` and `skipped` are part of `scans.json`. Report them or the user reads a partial scan as a clean one |
| "Pro is too slow, skip --pro" | Cross-file analysis catches 250% more true positives; worth the time |
| "Semgrep handles GitHub URLs natively" | URL handling fails on repos with non-standard YAML; always clone first |
| "Cleanup is optional" | Cloned repos pollute the user's workspace and accumulate across runs |
@@ -180,11 +243,11 @@ Use `subagent_type: static-analysis:semgrep-scanner` in Step 4 when spawning Tas
|------|---------|
| [rulesets.md](references/rulesets.md) | Complete ruleset catalog and selection algorithm |
| [scan-modes.md](references/scan-modes.md) | Pre/post-filter criteria and jq commands |
| [scanner-task-prompt.md](references/scanner-task-prompt.md) | Template for spawning scanner subagents |
| Workflow | Purpose |
|----------|---------|
| [scan-workflow.md](workflows/scan-workflow.md) | Complete 5-step scan execution process |
| `scripts/run-scans.sh` | The scan runner Step 4 calls |
## Success Criteria
@@ -194,11 +257,14 @@ Use `subagent_type: static-analysis:semgrep-scanner` in Step 4 when spawning Tas
- [ ] Scan mode selected by user (run all / important only)
- [ ] Rulesets include third-party rules for all detected languages
- [ ] User explicitly approved the scan plan (Step 3 gate passed)
- [ ] All scan Tasks spawned in a single message and completed
- [ ] `run-scans.sh` exited 0 and wrote `$OUTPUT_DIR/scans.json`
- [ ] `failed` and `skipped` from `scans.json` are empty, or listed in the report
- [ ] Every `semgrep` command used `--metrics=off`
- [ ] Approved rulesets logged to `$OUTPUT_DIR/rulesets.txt`
- [ ] Approved plan written to `$OUTPUT_DIR/rulesets.json` at the Step 3 gate, and passed to
the scanner unchanged
- [ ] `coveredNothing` from `scans.json` is empty, or listed in the report
- [ ] Raw per-scan outputs stored in `$OUTPUT_DIR/raw/`
- [ ] `results.sarif` exists in `$OUTPUT_DIR/results/` and is valid JSON
- [ ] Important-only mode: post-filter applied before merge; unfiltered results preserved in `raw/`
- [ ] Important-only mode: post-filter applied before merge, merge run with `--important`, unfiltered results preserved in `raw/`
- [ ] Results summary reported with severity and category breakdown
- [ ] Cloned repos (if any) cleaned up from `$OUTPUT_DIR/repos/`
plugins/static-analysis/skills/semgrep/references/scan-modes.md
+51 -15
View File
@@ -13,10 +13,20 @@ Focused on high-confidence security vulnerabilities. Excludes code quality, best
Add these flags to every `semgrep` command:
```bash
--severity MEDIUM --severity HIGH --severity CRITICAL
--severity WARNING --severity ERROR
```
This excludes LOW/INFO severity findings at scan time, reducing output volume before post-filtering.
This excludes INFO findings at scan time, reducing output volume before post-filtering.
`--severity` takes `INFO`, `WARNING`, or `ERROR`, and nothing else. Anything else exits 2 before
scanning, with no output written. The `LOW`/`MEDIUM`/`HIGH`/`CRITICAL` scale in the table below
belongs to the rule metadata, which the post-filter reads. The two are not interchangeable.
The two scales do not nest. A registry rule can carry CLI severity `INFO` and metadata
`impact: HIGH`, and this flag drops it at scan time before the post-filter sees it. The volume
reduction is why the pre-filter runs at scan time, but it makes important-only "WARNING and
above, then filtered on metadata" rather than "everything the metadata filter would keep".
Check a missing finding against a run-all scan before concluding the rule did not fire.
### Post-Filter: Metadata Criteria
@@ -28,13 +38,15 @@ After scanning, filter each JSON result file to keep only findings matching ALL
| `extra.metadata.confidence` | `"MEDIUM"`, `"HIGH"` | Excludes low-precision rules (high false positive rate) |
| `extra.metadata.impact` | `"MEDIUM"`, `"HIGH"` | Excludes low-impact informational findings |
**Third-party rules** (Trail of Bits, 0xdea, Decurity, etc.) may not have `confidence`/`impact`/`category` metadata. Findings **without** these metadata fields are **kept** — we cannot filter what is not annotated, and third-party rules are typically security-focused.
**Third-party rules** (Trail of Bits, 0xdea, Decurity, etc.) may not have `confidence`/`impact`/`category` metadata. Findings **without** these metadata fields are **kept by the post-filter** — we cannot filter what is not annotated, and third-party rules are typically security-focused.
This exemption applies only to findings that reach the post-filter. The pre-filter above runs first and on every command, including the cross-language unit that carries the cloned third-party repos, so an unannotated rule with CLI `severity: INFO` is dropped at scan time and never becomes a finding the exemption can keep. A third-party rule is exempt from the metadata filter, not from `--severity`.
### Semgrep Metadata Background
Semgrep security rules have these metadata fields (required for `category: security` in the official registry):
| Field | Purpose | Values |
| Field | Purpose | Metadata values (never CLI `--severity` values) |
|---|---|---|
| `severity` (top-level) | Overall rule severity, derived from likelihood × impact | `LOW`, `MEDIUM`, `HIGH`, `CRITICAL` |
| `category` | Rule category | `security`, `correctness`, `best-practice`, `maintainability`, `performance` |
@@ -75,9 +87,14 @@ Raw scan output lives in `$OUTPUT_DIR/raw/`. The filter creates `*-important.jso
```bash
# Apply important-only filter to all scan result JSON files in raw/
filter_failed=0
for f in "$OUTPUT_DIR/raw"/*-*.json; do
[[ "$f" == *-triage.json || "$f" == *-important.json ]] && continue
jq '{
out="${f%.json}-important.json"
# The redirect creates $out before jq runs, so a jq failure leaves a zero-byte file sitting
# there. merge_sarif.py --important reads that as a corrupt filter and aborts the whole
# merge, losing every other scan's findings to one bad file. Delete it and name the file.
if ! jq '{
results: [.results[] |
((.extra.metadata.category // "security") | ascii_downcase) as $cat |
((.extra.metadata.confidence // "HIGH") | ascii_upcase) as $conf |
@@ -90,21 +107,40 @@ for f in "$OUTPUT_DIR/raw"/*-*.json; do
],
errors: .errors,
paths: .paths
}' "$f" > "${f%.json}-important.json"
}' "$f" >"$out"; then
rm -f "$out"
filter_failed=$((filter_failed + 1))
echo "post-filter failed on $f" >&2
continue
fi
BEFORE=$(jq '.results | length' "$f")
AFTER=$(jq '.results | length' "${f%.json}-important.json")
AFTER=$(jq '.results | length' "$out")
echo "$f: $BEFORE → $AFTER findings (filtered $(( BEFORE - AFTER )))"
done
[ "$filter_failed" -eq 0 ] ||
echo "$filter_failed file(s) failed to filter; fix them before merging" >&2
```
### Scanner Task Modifications
### The Filter Does Not Apply to SARIF
In important-only mode, add `[SEVERITY_FLAGS]` to the scanner template:
Both filters above are written for semgrep's JSON shape and cannot be pointed at a `.sarif` file.
SARIF has no top-level `.results` and no `extra.metadata` — `category`, `confidence` and `impact`
are simply not in the format — so the filter exits with `Cannot iterate over null`, and
redirecting it over its own input truncates the file first.
```bash
semgrep [--pro if available] --metrics=off [SEVERITY_FLAGS] --config [RULESET] --json -o [OUTPUT_DIR]/raw/[lang]-[ruleset].json --sarif-output=[OUTPUT_DIR]/raw/[lang]-[ruleset].sarif [TARGET] &
```
The merged SARIF is filtered by `scripts/merge_sarif.py --important` instead, which keeps the
findings these filters kept by matching `(check_id, path, start.line)` against SARIF's
`(ruleId, uri, region.startLine)`. Run the JSON filter over every file in `raw/` first: the merge
fails rather than filtering if any scan has no `*-important.json` beside it, since a partial key
set would drop real findings from the primary deliverable with nothing downstream to notice.
Where `[SEVERITY_FLAGS]` is:
- **Run all**: *(empty)*
- **Important only**: `--severity MEDIUM --severity HIGH --severity CRITICAL`
### Scanner Command Modifications
`scripts/run-scans.sh` puts these on every command it generates; they are not yours to add.
- **Run all**: no severity flags
- **Important only**: `--severity WARNING --severity ERROR`
That pre-filter is applied by semgrep at scan time, before the metadata post-filter above. A
rule shipping with CLI severity INFO is dropped by the flag and never reaches the filter that
would have kept it.
plugins/static-analysis/skills/semgrep/references/scanner-task-prompt.md
-140
View File
@@ -1,140 +0,0 @@
# Scanner Subagent Task Prompt
Use this prompt template when spawning scanner Tasks in Step 4. Use `subagent_type: static-analysis:semgrep-scanner`.
## Template
```
You are a Semgrep scanner for [LANGUAGE_CATEGORY].
## Task
Run Semgrep scans for [LANGUAGE] files and save results to [OUTPUT_DIR]/raw.
## Pro Engine Status: [PRO_AVAILABLE: true/false]
## Scan Mode: [SCAN_MODE: run-all/important-only]
## APPROVED RULESETS (from user-confirmed plan)
[LIST EXACT RULESETS USER APPROVED - DO NOT SUBSTITUTE]
Example:
- p/python
- p/django
- p/security-audit
- p/secrets
- https://github.com/trailofbits/semgrep-rules
## Commands to Run (in parallel)
### Clone GitHub URL rulesets first:
```bash
mkdir -p [OUTPUT_DIR]/repos
# For each GitHub URL ruleset, clone into [OUTPUT_DIR]/repos/[name]:
git clone --depth 1 https://github.com/org/repo [OUTPUT_DIR]/repos/repo-name
```
### Generate commands for EACH approved ruleset:
```bash
semgrep [--pro if available] --metrics=off [SEVERITY_FLAGS] [INCLUDE_FLAGS] --config [RULESET] --json -o [OUTPUT_DIR]/raw/[lang]-[ruleset].json --sarif-output=[OUTPUT_DIR]/raw/[lang]-[ruleset].sarif [TARGET] &
```
Wait for all to complete:
```bash
wait
```
### Clean up cloned repos:
```bash
[ -n "[OUTPUT_DIR]" ] && rm -rf [OUTPUT_DIR]/repos
```
## Critical Rules
- Use ONLY the rulesets listed above - do not add or remove any
- Always use --metrics=off (prevents sending telemetry to Semgrep servers)
- Use --pro when Pro is available (enables cross-file taint tracking)
- If scan mode is **important-only**, add `--severity MEDIUM --severity HIGH --severity CRITICAL` to every command
- If scan mode is **run-all**, do NOT add severity flags
- Run all rulesets in parallel with & and wait
- For GitHub URL rulesets, always clone into [OUTPUT_DIR]/repos/ and use the local path as --config (do NOT pass URLs directly to semgrep — its URL handling is unreliable for repos with non-standard YAML)
- Add `--include` flags for language-specific rulesets (e.g., `--include="*.py"` for p/python). Do NOT add `--include` to cross-language rulesets like p/security-audit, p/secrets, or third-party repos
- After all scans complete, delete [OUTPUT_DIR]/repos/ to avoid leaving cloned repos behind
## Output
Report:
- Number of findings per ruleset
- Any scan errors
- File paths of JSON results (in [OUTPUT_DIR]/raw/)
- [If Pro] Note any cross-file findings detected
```
## Variable Substitutions
| Variable | Description | Example |
|----------|-------------|---------|
| `[LANGUAGE_CATEGORY]` | Language group being scanned | Python, JavaScript, Docker |
| `[LANGUAGE]` | Specific language | Python, TypeScript, Go |
| `[OUTPUT_DIR]` | Output directory (absolute path, resolved in Step 1) | /path/to/static_analysis_semgrep_1 |
| `[PRO_AVAILABLE]` | Whether Pro engine is available | true, false |
| `[SEVERITY_FLAGS]` | Severity pre-filter flags | *(empty)* for run-all, `--severity MEDIUM --severity HIGH --severity CRITICAL` for important-only |
| `[INCLUDE_FLAGS]` | File extension filter for language-specific rulesets | `--include="*.py"` for Python rulesets, *(empty)* for cross-language rulesets like p/security-audit, p/secrets, or third-party repos |
| `[RULESET]` | Semgrep ruleset identifier or local clone path | p/python, [OUTPUT_DIR]/repos/semgrep-rules |
| `[TARGET]` | Absolute path to directory to scan | /path/to/codebase |
## Example: Python Scanner Task
```
You are a Semgrep scanner for Python.
## Task
Run Semgrep scans for Python files and save results to /path/to/static_analysis_semgrep_1/raw.
## Pro Engine Status: true
## Scan Mode: run-all
## APPROVED RULESETS (from user-confirmed plan)
- p/python
- p/django
- p/security-audit
- p/secrets
- https://github.com/trailofbits/semgrep-rules
## Commands to Run (in parallel)
### Clone GitHub URL rulesets first:
```bash
mkdir -p /path/to/static_analysis_semgrep_1/repos
git clone --depth 1 https://github.com/trailofbits/semgrep-rules /path/to/static_analysis_semgrep_1/repos/trailofbits
```
### Run scans:
```bash
semgrep --pro --metrics=off --include="*.py" --config p/python --json -o /path/to/static_analysis_semgrep_1/raw/python-python.json --sarif-output=/path/to/static_analysis_semgrep_1/raw/python-python.sarif /path/to/codebase &
semgrep --pro --metrics=off --include="*.py" --config p/django --json -o /path/to/static_analysis_semgrep_1/raw/python-django.json --sarif-output=/path/to/static_analysis_semgrep_1/raw/python-django.sarif /path/to/codebase &
semgrep --pro --metrics=off --config p/security-audit --json -o /path/to/static_analysis_semgrep_1/raw/python-security-audit.json --sarif-output=/path/to/static_analysis_semgrep_1/raw/python-security-audit.sarif /path/to/codebase &
semgrep --pro --metrics=off --config p/secrets --json -o /path/to/static_analysis_semgrep_1/raw/python-secrets.json --sarif-output=/path/to/static_analysis_semgrep_1/raw/python-secrets.sarif /path/to/codebase &
semgrep --pro --metrics=off --config /path/to/static_analysis_semgrep_1/repos/trailofbits --json -o /path/to/static_analysis_semgrep_1/raw/python-trailofbits.json --sarif-output=/path/to/static_analysis_semgrep_1/raw/python-trailofbits.sarif /path/to/codebase &
wait
```
### Clean up cloned repos:
```bash
rm -rf /path/to/static_analysis_semgrep_1/repos
```
## Critical Rules
- Use ONLY the rulesets listed above - do not add or remove any
- Always use --metrics=off
- Use --pro when Pro is available
- Run all rulesets in parallel with & and wait
- Clone GitHub URL rulesets into the output dir repos/ subfolder, use local path as --config
- Add --include="*.py" to language-specific rulesets (p/python, p/django) but NOT to p/security-audit, p/secrets, or third-party repos
- Delete repos/ after scanning
## Output
Report:
- Number of findings per ruleset
- Any scan errors
- File paths of JSON results (in raw/ subdirectory)
- Note any cross-file findings detected
```
plugins/static-analysis/skills/semgrep/scripts/merge_sarif.py
+232 -96
View File
@@ -5,90 +5,163 @@
"""Merge SARIF files into a single consolidated output.
Usage:
uv run merge_sarif.py RAW_DIR OUTPUT_FILE
uv run merge_sarif.py RAW_DIR OUTPUT_FILE [--important] [--scans scans.json]
Reads *.sarif files from RAW_DIR (e.g., $OUTPUT_DIR/raw), produces
OUTPUT_FILE (e.g., $OUTPUT_DIR/results/results.sarif) containing all
findings merged and deduplicated.
Attempts to use SARIF Multitool for merging if available, falls back to
pure Python implementation.
--scans names the scans.json run-scans.sh wrote, and drops the SARIF files
belonging to scans recorded under .failed. Those files exist because a scan
that died part-way may still have written one, so without this a single dead
scan takes the whole run's deliverable with it: under --important its output
has no post-filter beside it, and that is an error rather than an empty
filter. Pass it whenever scans.json exists.
--important restricts the merged output to the findings that survived the
important-only post-filter. That filter reads semgrep's JSON metadata
(category/confidence/impact), which SARIF does not carry, so it cannot be
re-run against SARIF. Finding identity can be matched across the two formats
though, and that is what this flag does. Without it the merged SARIF in
important-only mode keeps every finding the mode exists to exclude.
The merge is pure Python and shells out to nothing. It used to try
`npx @microsoft/sarif-multitool` first, which made the output depend on
whether that package happened to be in the npx cache: the two backends do not
agree. Only this one dedups results on (ruleId, uri, startLine), which is the
identity --important matches against and the reason the report is told to
count from the merged file rather than sum per-scan totals. Multitool also
normalizes artifactLocation.uri, which would leave --important matching
nothing and blaming a semgrep format change that never happened.
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
Key = tuple[str, str, int]
def has_sarif_multitool() -> bool:
"""Check if SARIF Multitool is pre-installed via npx."""
if not shutil.which("npx"):
return False
try:
result = subprocess.run(
["npx", "--no-install", "@microsoft/sarif-multitool", "--version"],
capture_output=True,
timeout=30,
def sarif_key(result: dict) -> Key:
"""Identity of one SARIF result: rule, file, line.
The same triple in semgrep's JSON is (check_id, path, start.line), verified
field-for-field against semgrep output. Both the merge dedup and the
--important filter read it from here so the two cannot drift apart. The
filter is only correct while its keys are the keys the merge produced.
"""
locations = result.get("locations", [])
uri = ""
start_line = 0
if locations:
phys = locations[0].get("physicalLocation", {})
uri = phys.get("artifactLocation", {}).get("uri", "")
start_line = phys.get("region", {}).get("startLine", 0)
return (result.get("ruleId", ""), uri, start_line)
def json_key(result: dict) -> Key:
"""The same identity read out of a semgrep JSON result."""
return (
result.get("check_id", ""),
result.get("path", ""),
result.get("start", {}).get("line", 0),
)
def failed_sarifs(scans_json: Path) -> set[Path]:
"""Resolved paths of the SARIF files belonging to scans that did not succeed.
run-scans.sh records a failed scan carrying the same paths a success does, because a scan
that crashed part-way may still have written a file. What it wrote is not a scan result:
it is whatever semgrep produced before it stopped. Those files must not be held to the
post-filter requirement, or one dead process denies every healthy scan a deliverable.
"""
data = json.loads(scans_json.read_text())
failed = data.get("failed")
if not isinstance(failed, list):
raise ValueError(
f"{scans_json} has no .failed array; it is not a scans.json written by run-scans.sh"
)
return result.returncode == 0
except subprocess.TimeoutExpired:
print("Warning: SARIF Multitool version check timed out", file=sys.stderr)
return False
except FileNotFoundError:
return False
except OSError as e:
print(f"Warning: Failed to check SARIF Multitool: {e}", file=sys.stderr)
return False
return {
Path(entry["sarif"]).resolve()
for entry in failed
if isinstance(entry, dict) and entry.get("sarif")
}
def merge_with_multitool(sarif_files: list[Path]) -> dict | None:
"""Use SARIF Multitool to merge SARIF files. Returns merged SARIF or None."""
if not sarif_files:
return None
def surviving_keys(sarif_files: list[Path]) -> set[Key]:
"""Keys kept by the important-only post-filter, one *-important.json per SARIF.
with tempfile.NamedTemporaryFile(suffix=".sarif", delete=False) as tmp:
tmp_path = Path(tmp.name)
Derived from the SARIF files going into the merge rather than by globbing
*-important.json, so a post-filter that ran on only some of them is an error
here instead of a merged SARIF quietly missing whole rulesets.
try:
cmd = [
"npx",
"--no-install",
"@microsoft/sarif-multitool",
"merge",
*[str(f) for f in sarif_files],
"--output-file",
str(tmp_path),
"--force",
]
result = subprocess.run(cmd, capture_output=True, timeout=120)
if result.returncode != 0:
print(f"SARIF Multitool merge failed: {result.stderr.decode()}", file=sys.stderr)
return None
Raises ValueError when a filter file is missing or unreadable. Filtering
against a partial key set drops real findings from the primary deliverable
and there is nothing downstream that could notice. That is stricter than
the merge itself, which warns and skips a SARIF file it cannot parse: an
unparseable SARIF contributes no findings either way, while an unparseable
filter file silently removes findings the SARIF does contain.
"""
keys: set[Key] = set()
missing: list[str] = []
for sarif_file in sarif_files:
filtered = sarif_file.with_name(f"{sarif_file.stem}-important.json")
if not filtered.is_file():
missing.append(filtered.name)
continue
try:
data = json.loads(filtered.read_text())
except json.JSONDecodeError as e:
raise ValueError(f"{filtered} is not valid JSON: {e}") from e
results = data.get("results")
if not isinstance(results, list):
raise ValueError(
f"{filtered} has no .results array; it is not a filtered semgrep result file"
)
for result in results:
keys.add(json_key(result))
return json.loads(tmp_path.read_text())
except subprocess.TimeoutExpired as e:
print(f"SARIF Multitool timed out: {e}", file=sys.stderr)
return None
except json.JSONDecodeError as e:
print(f"SARIF Multitool produced invalid JSON: {e}", file=sys.stderr)
return None
except FileNotFoundError as e:
print(f"SARIF Multitool not found: {e}", file=sys.stderr)
return None
except OSError as e:
print(f"SARIF Multitool OS error ({type(e).__name__}): {e}", file=sys.stderr)
return None
finally:
tmp_path.unlink(missing_ok=True)
if missing:
raise ValueError(
f"{len(missing)} of {len(sarif_files)} scans have no post-filtered JSON "
f"({', '.join(sorted(missing)[:5])}"
f"{', ...' if len(missing) > 5 else ''}). Run the important-only post-filter "
"over every file in the raw directory first."
)
return keys
def merge_sarif_pure_python(sarif_files: list[Path]) -> dict:
"""Pure Python SARIF merge (fallback)."""
def filter_to_keys(merged: dict, keys: set[Key]) -> tuple[int, int]:
"""Keep only results whose identity survived the post-filter. Returns (kept, dropped)."""
kept = 0
dropped = 0
for run in merged.get("runs", []):
keeping = []
for result in run.get("results", []):
if sarif_key(result) in keys:
keeping.append(result)
kept += 1
else:
dropped += 1
run["results"] = keeping
return kept, dropped
def merge_sarif_pure_python(sarif_files: list[Path]) -> tuple[dict, list[str]]:
"""Merge every SARIF into one run, deduplicating results by sarif_key.
Returns (merged, unparseable). The unparseable list is returned rather than just
warned about: a scan can exit 0, write a valid .json that puts it in .scans with a
finding count, and still leave a truncated .sarif. Dropping that file silently makes
results.sarif short by exactly those findings with nothing anywhere pointing at it.
The dedup is what makes the merged total meaningful: one finding flagged by two
rulesets is one row here and two in a sum of per-scan counts.
"""
merged = {
"version": "2.1.0",
"$schema": "https://json.schemastore.org/sarif-2.1.0.json",
@@ -97,7 +170,7 @@ def merge_sarif_pure_python(sarif_files: list[Path]) -> dict:
seen_rules: dict[str, dict] = {}
all_results: list[dict] = []
seen_results: set[tuple[str, str, int]] = set()
seen_results: set[Key] = set()
tool_info: dict | None = None
skipped_files: list[str] = []
@@ -120,15 +193,7 @@ def merge_sarif_pure_python(sarif_files: list[Path]) -> dict:
seen_rules[rule_id] = rule
for result in run.get("results", []):
rule_id = result.get("ruleId", "")
uri = ""
start_line = 0
locations = result.get("locations", [])
if locations:
phys = locations[0].get("physicalLocation", {})
uri = phys.get("artifactLocation", {}).get("uri", "")
start_line = phys.get("region", {}).get("startLine", 0)
dedup_key = (rule_id, uri, start_line)
dedup_key = sarif_key(result)
if dedup_key in seen_results:
continue
seen_results.add(dedup_key)
@@ -142,25 +207,32 @@ def merge_sarif_pure_python(sarif_files: list[Path]) -> dict:
merged_run["tool"]["driver"]["rules"] = list(seen_rules.values())
merged["runs"].append(merged_run)
if skipped_files:
print(
f"WARNING: {len(skipped_files)} of {len(sarif_files)} SARIF files "
f"could not be parsed. Results may be incomplete.",
file=sys.stderr,
)
for sf in skipped_files:
print(f" Skipped: {sf}", file=sys.stderr)
return merged
return merged, skipped_files
def main() -> int:
if len(sys.argv) != 3:
print(f"Usage: {sys.argv[0]} RAW_DIR OUTPUT_FILE", file=sys.stderr)
argv = sys.argv[1:]
important = "--important" in argv
argv = [a for a in argv if a != "--important"]
scans_json: Path | None = None
if "--scans" in argv:
i = argv.index("--scans")
if i + 1 >= len(argv):
print("--scans needs the path to scans.json", file=sys.stderr)
return 1
scans_json = Path(argv[i + 1])
del argv[i : i + 2]
if len(argv) != 2:
print(
f"Usage: {sys.argv[0]} RAW_DIR OUTPUT_FILE [--important] [--scans scans.json]",
file=sys.stderr,
)
return 1
raw_dir = Path(sys.argv[1])
output_file = Path(sys.argv[2])
raw_dir = Path(argv[0])
output_file = Path(argv[1])
if not raw_dir.is_dir():
print(f"Error: {raw_dir} is not a directory", file=sys.stderr)
@@ -174,20 +246,84 @@ def main() -> int:
print("No SARIF files found, nothing to merge", file=sys.stderr)
return 1
# Before the post-filter requirement below, so a dead scan is dropped rather than held to it.
if scans_json is not None:
try:
excluded = failed_sarifs(scans_json)
except (OSError, json.JSONDecodeError, ValueError) as e:
print(f"Error: {e}", file=sys.stderr)
return 1
kept = [p for p in sarif_files if p.resolve() not in excluded]
if len(kept) != len(sarif_files):
names = sorted(p.name for p in sarif_files if p.resolve() in excluded)
print(
f"excluding {len(sarif_files) - len(kept)} SARIF file(s) from failed scans: "
f"{', '.join(names)}"
)
sarif_files = kept
# Every scan failed. Writing an empty merge here would report a clean run over nothing.
if not sarif_files:
print(
f"Error: every scan in {scans_json} failed, so there is nothing to merge",
file=sys.stderr,
)
return 1
# Resolved before the merge, not after: a post-filter that did not run is a broken run,
# and finding that out first costs nothing while finding it out afterwards means either
# a wrong results.sarif on disk or a merge thrown away.
keys: set[Key] = set()
if important:
try:
keys = surviving_keys(sarif_files)
except ValueError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
print(f"important-only: {len(keys)} findings survived the post-filter")
# Ensure output directory exists
output_file.parent.mkdir(parents=True, exist_ok=True)
# Try SARIF Multitool first, fall back to pure Python
merged: dict | None = None
if has_sarif_multitool():
print("Using SARIF Multitool for merge...")
merged = merge_with_multitool(sarif_files)
if merged:
print("SARIF Multitool merge successful")
merged, unparseable = merge_sarif_pure_python(sarif_files)
if unparseable:
# stdout, alongside the --scans exclusions, because that is the stream the Report phase
# reads and it must give these their own section. On stderr this was invisible to the
# summary: the scan sits in .scans as a success carrying its finding count, so nothing
# in scans.json or results.sarif shows that those findings went missing.
names = ", ".join(sorted(Path(p).name for p in unparseable))
print(
f"unparseable: {len(unparseable)} of {len(sarif_files)} SARIF files could not be "
f"parsed and are missing from the merge: {names}"
)
# Nothing was read at all, so "0 findings" would be a clean run rather than a broken one.
if unparseable and len(unparseable) == len(sarif_files):
print(
f"Error: none of the {len(sarif_files)} SARIF files could be parsed; "
"there is nothing to merge",
file=sys.stderr,
)
return 1
if merged is None:
print("Using pure Python merge (SARIF Multitool not available or failed)")
merged = merge_sarif_pure_python(sarif_files)
if important:
before = sum(len(run.get("results", [])) for run in merged.get("runs", []))
kept, dropped = filter_to_keys(merged, keys)
print(f"important-only: kept {kept} of {before} merged findings, dropped {dropped}")
# Unreachable while the two formats agree: every key came from the *-important.json
# sibling of a SARIF in this merge, so at least one must match something. Reaching it
# means sarif_key and json_key are no longer reading one identity out of two shapes —
# semgrep changed an output format — and every finding was dropped for that reason
# rather than by the filter. Writing anyway ships an empty results.sarif that reads as
# a clean important-only run, which is the failure this whole flag exists to prevent.
# Guarded on keys: a filter that legitimately kept nothing is a real zero, not drift.
if keys and before and not kept:
print(
f"Error: the post-filter kept {len(keys)} findings and the merge read {before}, "
"but none of them matched. The JSON and SARIF records of a finding no longer "
"reduce to the same (rule, file, line), so refusing to write a zero-finding "
f"{output_file}. Compare a raw *.json against its *.sarif in {raw_dir}.",
file=sys.stderr,
)
return 1
result_count = sum(len(run.get("results", [])) for run in merged.get("runs", []))
print(f"Merged SARIF contains {result_count} findings")
plugins/static-analysis/skills/semgrep/scripts/run-scans.sh
Executable
+604
View File
@@ -0,0 +1,604 @@
#!/usr/bin/env bash
# Step 4 of the semgrep skill: run the selected rulesets and report what each scan produced.
# Generating every command here is what keeps --metrics=off, --include and --exclude out of a
# model's hands. Exit codes and counts come from the processes and the JSON they wrote.
# No `declare -A` and no `wait -n`, so this runs on macOS's bash 3.2.
set -euo pipefail
readonly METRICS_OFF="--metrics=off"
# semgrep --severity accepts INFO, WARNING and ERROR only, and rejects anything else with exit 2
# before scanning. The metadata thresholds LOW/MEDIUM/HIGH are applied by the post-filter in
# references/scan-modes.md, not here.
SEVERITY_FLAGS=(--severity WARNING --severity ERROR)
readonly DEFAULT_JOBS=4
usage() {
cat <<'USAGE'
Usage: run-scans.sh --target DIR --output-dir DIR --mode MODE --rulesets FILE [options]
--target DIR absolute path to the tree to scan
--output-dir DIR absolute path for results; raw/ and repos/ are created under it
--mode MODE run-all | important-only
--rulesets FILE JSON: {"baseline":[...], "<language>":[...], "third_party":["https://..."]}
--pro add --pro to every command (Semgrep Pro engine)
--jobs N concurrent semgrep processes (default 4)
--dry-run print the commands that would run, then exit; clones nothing
Writes OUTPUT_DIR/scans.json:
{scans:[{lang,ruleset,json,sarif,findings}], failed:[...], skipped:[...],
unscoped:[lang], alsoShared:["lang/ruleset"]}
USAGE
}
die() {
echo "run-scans.sh: $*" >&2
exit 1
}
# --include globs per language. Checked against semgrep by scanning one file per extension;
# re-run that when bumping it, because a type omitted here is excluded with no signal and the
# ruleset reads clean rather than incomplete. Absent because semgrep does not parse them:
# .mts, .cts, .C, Containerfile, Dockerfile.prod. javascript carries the TypeScript globs
# because one detection category covers both.
includes_for() {
case "$1" in
python) echo '*.py *.pyi' ;;
javascript) echo '*.js *.jsx *.mjs *.cjs *.ts *.tsx' ;;
typescript) echo '*.ts *.tsx' ;;
go) echo '*.go' ;;
ruby) echo '*.rb' ;;
java) echo '*.java *.jsp' ;;
kotlin) echo '*.kt *.kts' ;;
php) echo '*.php *.phtml' ;;
c) echo '*.c *.h' ;;
cpp) echo '*.c *.cc *.cpp *.cxx *.h *.hh *.hpp *.hxx' ;;
csharp) echo '*.cs' ;;
rust) echo '*.rs' ;;
scala) echo '*.scala' ;;
swift) echo '*.swift' ;;
elixir) echo '*.ex *.exs' ;;
solidity) echo '*.sol' ;;
docker) echo 'Dockerfile *.dockerfile' ;;
terraform) echo '*.tf *.tfvars *.hcl' ;;
json) echo '*.json' ;;
apex) echo '*.cls *.trigger' ;;
cloudformation) echo '*.yaml *.yml *.json' ;;
github-actions) echo '*.yml *.yaml' ;;
kubernetes) echo '*.yaml *.yml' ;;
yaml) echo '*.yaml *.yml' ;;
*) echo '' ;;
esac
}
# Folds plan spellings onto the keys includes_for knows, so `js` and `javascript` do not
# become two units and scan the same ruleset twice.
canonical_lang() {
local k
k=$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')
case "$k" in
js | jsx | node | nodejs | 'js/ts' | 'javascript/typescript') echo javascript ;;
ts | tsx) echo typescript ;;
golang) echo go ;;
'c/c++' | 'c++' | cxx) echo cpp ;;
dockerfile) echo docker ;;
k8s) echo kubernetes ;;
'c#' | dotnet) echo csharp ;;
sol) echo solidity ;;
tf | hcl) echo terraform ;;
cfn) echo cloudformation ;;
'github actions' | githubactions | gha) echo github-actions ;;
salesforce) echo apex ;;
*) echo "$k" ;;
esac
}
# Filenames, not identifiers: a registry id and a clone path must both reduce to something safe
# to concatenate into a shell-quoted path.
slug() {
case "$1" in
*://*) repo_dir_name "$1" ;;
*) printf '%s' "${1##*/}" | sed 's/[^A-Za-z0-9._-]\{1,\}/-/g; s/^-\{1,\}//; s/-\{1,\}$//' ;;
esac
}
# The clone directory carries the owner: several orgs publish a repo named semgrep-rules, and
# keying on the basename alone would collide them into one directory.
repo_dir_name() {
local u=$1 repo owner
u=${u%.git}
u=${u%/}
repo=${u##*/}
u=${u%/*}
owner=${u##*/}
printf '%s' "${owner:-unknown}-${repo:-rules}" | sed 's/[^A-Za-z0-9._-]\{1,\}/-/g'
}
TARGET="" OUTPUT_DIR="" MODE="" RULESETS_FILE="" PRO="" DRY_RUN=""
JOBS=$DEFAULT_JOBS
while [ $# -gt 0 ]; do
case "$1" in
--target)
TARGET=${2:-}
shift 2
;;
--output-dir)
OUTPUT_DIR=${2:-}
shift 2
;;
--mode)
MODE=${2:-}
shift 2
;;
--rulesets)
RULESETS_FILE=${2:-}
shift 2
;;
--jobs)
JOBS=${2:-}
shift 2
;;
--pro)
PRO=1
shift
;;
--dry-run)
DRY_RUN=1
shift
;;
-h | --help)
usage
exit 0
;;
*) die "unknown argument: $1" ;;
esac
done
command -v jq >/dev/null 2>&1 || die "jq is required"
[ -n "$DRY_RUN" ] || command -v semgrep >/dev/null 2>&1 || die "semgrep is required"
# Each check fails the run rather than degrading it: a scan with no rulesets, or against the
# wrong path, produces a clean-looking empty report.
case "$TARGET" in /*) ;; *) die "--target must be an absolute path, got '${TARGET}'" ;; esac
case "$OUTPUT_DIR" in /*) ;; *) die "--output-dir must be an absolute path, got '${OUTPUT_DIR}'" ;; esac
[ -d "$TARGET" ] || die "--target is not a directory: $TARGET"
case "$MODE" in run-all | important-only) ;; *) die "--mode must be run-all or important-only, got '${MODE}'" ;; esac
if [ -z "$RULESETS_FILE" ] || [ ! -r "$RULESETS_FILE" ]; then
die "--rulesets file is missing or unreadable: ${RULESETS_FILE}"
fi
jq -e . "$RULESETS_FILE" >/dev/null 2>&1 || die "--rulesets is not valid JSON: $RULESETS_FILE"
case "$JOBS" in '' | *[!0-9]*) die "--jobs must be a positive integer, got '${JOBS}'" ;; esac
[ "$JOBS" -ge 1 ] || die "--jobs must be at least 1"
# Both paths resolve the same way before being compared. Resolving only one makes the equality
# and inside-the-target checks miss on any symlinked path (every /var path on macOS), and the
# run then scans its own cloned rules. The output dir may not exist yet, so the deepest
# existing ancestor is resolved and the rest appended.
resolve_path() {
local p=$1 tail=""
while [ ! -d "$p" ]; do
tail="/${p##*/}$tail"
p=${p%/*}
[ -n "$p" ] || p=/
done
printf '%s' "$(cd "$p" && pwd)$tail"
}
TARGET=$(resolve_path "$TARGET")
TARGET_ROOT=${TARGET%/}
OUTPUT_ROOT=$(resolve_path "${OUTPUT_DIR%/}")
[ "$OUTPUT_ROOT" != "$TARGET_ROOT" ] ||
die "--output-dir is the scan target; the run would scan its own output and its cloned rules"
# A denylist, not an allowlist: real directories contain spaces and parentheses, which are
# inert. These four stay live inside double quotes and a control character can smuggle in a
# newline. One pattern each, since one bracket expression cannot hold both quote styles.
has_unsafe_char() {
case "$1" in
*'"'*) return 0 ;;
*'$'*) return 0 ;;
*'`'*) return 0 ;;
*[\\]*) return 0 ;;
*[[:cntrl:]]*) return 0 ;;
esac
return 1
}
for p in "$TARGET" "$OUTPUT_ROOT"; do
! has_unsafe_char "$p" || die "path contains a character that is not safe to splice: $p"
done
# jq -e over the whole file, so a bad entry stops the run before anything is fetched.
jq -e '
to_entries
| all(.value | type == "array")
' "$RULESETS_FILE" >/dev/null 2>&1 || die "every value in --rulesets must be an array"
jq -e '
to_entries
| map(select(.key != "third_party") | .value[])
| all(type == "string" and test("^[A-Za-z0-9._-]+(/[A-Za-z0-9._-]+)*$")
and (split("/") | index("..") | not))
' "$RULESETS_FILE" >/dev/null 2>&1 ||
die "ruleset entries must be registry identifiers like p/python; repository URLs go under third_party"
jq -e '
(.third_party // [])
| all(type == "string" and test("^https://[A-Za-z0-9.-]+(:[0-9]+)?(/[A-Za-z0-9._-]+)+(\\.git)?/?$"))
' "$RULESETS_FILE" >/dev/null 2>&1 || die "third_party entries must be https git URLs"
RAW_DIR="$OUTPUT_ROOT/raw"
REPOS_DIR="$OUTPUT_ROOT/repos"
# The default output dir lands inside the target, so without --exclude every scan also reads
# the cloned rule repos (full of literal example secrets) and the raw JSON siblings are still
# writing. --exclude takes a pattern, not a rooted path: `out` also excludes `vendor/out`, and
# there is no anchored form, so it is announced rather than applied quietly.
EXCLUDE_ARG=""
EXCLUDE_PATTERN=""
case "$OUTPUT_ROOT/" in
"$TARGET_ROOT"/*)
rel=${OUTPUT_ROOT#"$TARGET_ROOT"/}
EXCLUDE_ARG="--exclude=$rel"
# Recorded in scans.json, not just announced here. This drops files from every scan, and an
# unanchored pattern drops more than the output directory: --exclude=out also skips
# src/out/. On stderr alone the report has nothing to show, so the gap looks like clean
# coverage — the same silent truncation `coveredNothing` and the unparseable list exist to
# prevent.
EXCLUDE_PATTERN="$rel"
echo "note: output directory is inside the target; excluding '$rel' from every scan." >&2
echo " semgrep matches that pattern anywhere in the tree." >&2
;;
esac
WORK=$(mktemp -d "${TMPDIR:-/tmp}/run-scans.XXXXXX")
cleanup() { rm -rf "$WORK"; }
trap cleanup EXIT
SCAN_LIST="$WORK/scans.tsv"
SKIPPED="$WORK/skipped.tsv"
ALSO_SHARED="$WORK/also-shared.txt"
UNSCOPED="$WORK/unscoped.txt"
# A ruleset whose --include globs matched no file exits 0 with an empty result, which is
# indistinguishable in scans.json from a ruleset that ran and found nothing. That is how a plan
# naming the wrong languages reads as a clean audit: p/python on a Go tree reports 0 findings
# exactly like p/gosec would have. semgrep counts what it opened in .paths.scanned, so the two
# can be told apart and the ones that covered nothing named.
COVERED_NOTHING="$WORK/covered-nothing.txt"
: >"$SCAN_LIST"
: >"$SKIPPED"
: >"$ALSO_SHARED"
: >"$UNSCOPED"
: >"$COVERED_NOTHING"
# Stems are the filename half of every output path and the key results are matched on, so a
# collision would let one scan's result be read as another's.
USED_STEMS="$WORK/stems.txt"
: >"$USED_STEMS"
unique_stem() {
local base=$1 n=2 candidate=$1
while grep -Fxq "$candidate" "$USED_STEMS" 2>/dev/null; do
candidate="$base-$n"
n=$((n + 1))
done
printf '%s\n' "$candidate" >>"$USED_STEMS"
printf '%s' "$candidate"
}
add_scan() {
local lang=$1 ruleset=$2 config=$3 includes=$4 stem
stem=$(unique_stem "$(printf '%s' "$lang" | sed 's/[^A-Za-z0-9._-]\{1,\}/-/g')-$(slug "$ruleset")")
printf '%s\t%s\t%s\t%s\t%s\n' "$stem" "$lang" "$ruleset" "$config" "$includes" >>"$SCAN_LIST"
}
mapfile_compat() { # read newline-separated stdin into a named array, bash 3.2 safe
local __name=$1 __line
eval "$__name=()"
while IFS= read -r __line; do
[ -n "$__line" ] || continue
eval "$__name+=(\"\$__line\")"
done
}
BASELINE=()
THIRD_PARTY=()
LANG_KEYS=()
mapfile_compat BASELINE < <(jq -r '(.baseline // []) | unique[]' "$RULESETS_FILE")
mapfile_compat THIRD_PARTY < <(jq -r '(.third_party // [])[]' "$RULESETS_FILE")
mapfile_compat LANG_KEYS < <(jq -r 'to_entries[] | select(.key != "baseline" and .key != "third_party" and (.value | length) > 0) | .key' "$RULESETS_FILE")
# Deduplicated by clone directory rather than by exact string: two spellings of one repository
# (with and without .git) survive a string comparison but collide on one destination, and the
# second clone would fail into a non-empty tree.
CLONE_URLS=()
CLONE_NAMES=()
for url in ${THIRD_PARTY[@]+"${THIRD_PARTY[@]}"}; do
name=$(repo_dir_name "$url")
seen=""
for existing in ${CLONE_NAMES[@]+"${CLONE_NAMES[@]}"}; do
[ "$existing" = "$name" ] && seen=1 && break
done
[ -n "$seen" ] && continue
CLONE_URLS+=("$url")
CLONE_NAMES+=("$name")
done
if [ -z "$DRY_RUN" ]; then
# Cleared for the same reason each clone destination is. merge_sarif.py globs every *.sarif
# in here, so into a reused output directory a run that drops a ruleset still merges the
# previous run's output for it: results.sarif and its total then cover a ruleset this run's
# scans.json never mentions. Only this script's own output lives here.
rm -rf "$RAW_DIR"
mkdir -p "$RAW_DIR"
[ ${#CLONE_URLS[@]} -eq 0 ] || mkdir -p "$REPOS_DIR"
fi
# ------------------------------------------------------------------ clone phase
CLONED_URLS=()
CLONED_PATHS=()
i=0
while [ $i -lt ${#CLONE_URLS[@]} ]; do
url=${CLONE_URLS[$i]}
name=${CLONE_NAMES[$i]}
i=$((i + 1))
dest="$REPOS_DIR/$name"
if [ -n "$DRY_RUN" ]; then
CLONED_URLS+=("$url")
CLONED_PATHS+=("$dest")
continue
fi
# Cleared first: git clone refuses a non-empty directory, so a reused output directory would
# drop an approved ruleset while usable rules sat on disk.
rm -rf "$dest"
if ! err=$(git clone --depth 1 "$url" "$dest" 2>&1); then
printf '%s\t%s\n' "$url" "$(printf '%s' "$err" | tail -n 3 | tr '\n' ' ')" >>"$SKIPPED"
continue
fi
# A repository that cloned but carries no rules scans nothing, and reporting it as fine would
# show a completed scan against a ruleset that never ran.
#
# -print -quit rather than `find … | head -1`: under pipefail, head exits after the first line
# and find dies on SIGPIPE with 141 once its output passes the pipe buffer. pipefail makes the
# pipeline non-zero, so a repository with enough rules to fill 64 KiB of pathnames — which is
# every real rule repo, trailofbits/semgrep-rules included — was recorded as carrying none.
# The required third-party rules then silently did not run. -quit stops at the first match
# with no reader to race.
if [ -z "$(find "$dest" \( -name '*.yaml' -o -name '*.yml' \) -type f -print -quit)" ]; then
printf '%s\t%s\n' "$url" "cloned but contains no rule files" >>"$SKIPPED"
continue
fi
CLONED_URLS+=("$url")
CLONED_PATHS+=("$dest")
done
# ------------------------------------------------------------- build the scan list
# Cross-language rulesets scan the whole target unscoped, so they run once for the run rather
# than once per language, and they never take --include: they carry rules for every language
# and a filter would drop findings in the files it does not match.
for ruleset in ${BASELINE[@]+"${BASELINE[@]}"}; do
add_scan "all" "$ruleset" "$ruleset" ""
done
i=0
while [ $i -lt ${#CLONED_URLS[@]} ]; do
add_scan "all" "${CLONED_URLS[$i]}" "${CLONED_PATHS[$i]}" ""
i=$((i + 1))
done
is_baseline() {
for b in ${BASELINE[@]+"${BASELINE[@]}"}; do
[ "$b" = "$1" ] && return 0
done
return 1
}
# Language keys are folded onto their canonical name before any unit exists, so `js` and
# `javascript` in one plan produce one unit rather than two with identical globs.
CANON_SEEN="$WORK/canon.txt"
: >"$CANON_SEEN"
for key in ${LANG_KEYS[@]+"${LANG_KEYS[@]}"}; do
# The key reaches the scan command as the language half of the output filename, and it
# arrives as free-form generated text, so it gets the same treatment the paths do.
! has_unsafe_char "$key" || die "ruleset key reaches the scan command as a filename; '$key' is not a language"
lang=$(canonical_lang "$key")
[ "$lang" != "all" ] || die "ruleset key '$key' names the reserved language 'all' used by the cross-language unit"
printf '%s\n' "$lang" >>"$CANON_SEEN"
done
# Redirected rather than piped: a `for … in $(…)` splits on every whitespace character, and a
# pipeline would put the loop in a subshell.
sort -u "$CANON_SEEN" >"$WORK/langs.txt"
while IFS= read -r lang; do
[ -n "$lang" ] || continue
# jq cannot express canonical_lang, so the union happens here: every key folding onto this
# language contributes its rulesets, deduplicated. A ruleset listed twice for one language
# would otherwise scan twice, and the second copy's failure would read as the first's success.
own=""
for key in ${LANG_KEYS[@]+"${LANG_KEYS[@]}"}; do
[ "$(canonical_lang "$key")" = "$lang" ] || continue
own="$own$(jq -r --arg k "$key" '.[$k][]' "$RULESETS_FILE")
"
done
own=$(printf '%s' "$own" | grep -v '^$' | sort -u || true)
[ -n "$own" ] || continue
globs=$(includes_for "$lang")
had_own=""
while IFS= read -r ruleset; do
[ -n "$ruleset" ] || continue
# A ruleset already running unscoped over the whole target does not need a narrower second
# run. The merged SARIF dedups the copies; a per-scan sum of findings does not.
if is_baseline "$ruleset"; then
printf '%s/%s\n' "$lang" "$ruleset" >>"$ALSO_SHARED"
continue
fi
add_scan "$lang" "$ruleset" "$ruleset" "$globs"
had_own=1
done <<EOF
$own
EOF
# Reported only when the language actually got a unit. An unrecognized name costs the
# --include optimization, not coverage: the rules run against every file and fail to match
# what they do not understand. A language emptied by the baseline dedup ran nothing at all,
# so naming it here would point at a scan that never happened.
if [ -n "$had_own" ] && [ -z "$globs" ]; then
printf '%s\n' "$lang" >>"$UNSCOPED"
fi
done <"$WORK/langs.txt"
[ -s "$SCAN_LIST" ] || die "the ruleset plan produced no scans; there is nothing to run"
# ---------------------------------------------------------------------- commands
# Populates the global ARGV with the command for one scan. An array executed directly rather
# than a string run through eval: `eval "$cmd" &` reports exit status 1 whatever the command
# actually exited with, which would mark every failed scan as a success, and building argv also
# leaves no shell-quoting surface for a ruleset or path to escape through.
ARGV=()
build_argv() {
local config=$1 includes=$2 json=$3 sarif=$4 g
ARGV=(semgrep)
[ -z "$PRO" ] || ARGV+=(--pro)
ARGV+=("$METRICS_OFF")
[ "$MODE" != "important-only" ] || ARGV+=("${SEVERITY_FLAGS[@]}")
# Unquoted on purpose: includes is a space-separated glob list and must word-split here.
# shellcheck disable=SC2086
for g in $includes; do ARGV+=("--include=$g"); done
# On every command including the unscoped cross-language ones: those are precisely the
# rulesets that would otherwise read the cloned rule repositories.
[ -z "$EXCLUDE_ARG" ] || ARGV+=("$EXCLUDE_ARG")
ARGV+=(--config "$config" --json -o "$json" "--sarif-output=$sarif" "$TARGET")
}
# A pasteable rendering of ARGV for --dry-run. Anything outside a plainly safe set is wrapped
# whole, so a glob reaches semgrep rather than being expanded by the shell it is pasted into.
render_argv() {
local out="" a
for a in "${ARGV[@]}"; do
case "$a" in
*[!A-Za-z0-9._=/:-]*) out="$out \"$a\"" ;;
*) out="$out $a" ;;
esac
done
printf '%s' "${out# }"
}
if [ -n "$DRY_RUN" ]; then
while IFS=$'\t' read -r stem lang ruleset config includes; do
build_argv "$config" "$includes" "$RAW_DIR/$stem.json" "$RAW_DIR/$stem.sarif"
render_argv
printf '\n'
done <"$SCAN_LIST"
exit 0
fi
# ------------------------------------------------------------------- run the scans
# Batched rather than a rolling slot count, because `wait -n` needs bash 4.3 and this has to run
# on the bash 3.2 that ships with macOS. semgrep holds the rules and the scanned ASTs in memory,
# so an unbounded fan-out gets processes OOM-killed and those come back as ordinary scan
# failures with nothing pointing at memory as the cause.
total=$(wc -l <"$SCAN_LIST" | tr -d ' ')
echo "running $total scan(s), $JOBS at a time" >&2
pids=()
stems=()
run_batch() {
local idx=0
for idx in "${!pids[@]}"; do
if wait "${pids[$idx]}"; then
echo 0 >"$WORK/rc.${stems[$idx]}"
else
echo $? >"$WORK/rc.${stems[$idx]}"
fi
done
pids=()
stems=()
}
while IFS=$'\t' read -r stem lang ruleset config includes; do
json="$RAW_DIR/$stem.json"
sarif="$RAW_DIR/$stem.sarif"
build_argv "$config" "$includes" "$json" "$sarif"
"${ARGV[@]}" >"$WORK/out.$stem" 2>"$WORK/err.$stem" &
pids+=("$!")
stems+=("$stem")
[ ${#pids[@]} -lt "$JOBS" ] || run_batch
done <"$SCAN_LIST"
[ ${#pids[@]} -eq 0 ] || run_batch
# ---------------------------------------------------------------------- assemble
: >"$WORK/scans.jsonl"
: >"$WORK/failed.jsonl"
while IFS=$'\t' read -r stem lang ruleset config includes; do
json="$RAW_DIR/$stem.json"
sarif="$RAW_DIR/$stem.sarif"
rc=$(cat "$WORK/rc.$stem" 2>/dev/null || echo 127)
findings=-1
scanned=-1
ok=""
# Exit 0 covers both "found nothing" and "found plenty", so it says nothing about findings.
# Exit 1 is a successful scan on older versions. 7 means the config would not load and 2 a
# bad argument; in both cases no scan happened.
if { [ "$rc" -eq 0 ] || [ "$rc" -eq 1 ]; } && [ -s "$json" ] && [ -s "$sarif" ]; then
if findings=$(jq -e '.results | length' "$json" 2>/dev/null); then
ok=1
# null and empty are different answers: a semgrep that does not report .paths gives -1,
# "not known", while a present-but-empty list is a scan that genuinely opened no file.
# Collapsing them with `// []` would report every scan as covering nothing.
scanned=$(jq 'if .paths.scanned == null then -1 else (.paths.scanned | length) end' \
"$json" 2>/dev/null || echo -1)
else
findings=-1
fi
fi
if [ -n "$ok" ]; then
[ "$scanned" -ne 0 ] || printf '%s/%s\n' "$lang" "$ruleset" >>"$COVERED_NOTHING"
jq -nc --arg lang "$lang" --arg ruleset "$ruleset" --arg json "$json" \
--arg sarif "$sarif" --argjson findings "$findings" --argjson scanned "$scanned" \
'{lang:$lang, ruleset:$ruleset, json:$json, sarif:$sarif, findings:$findings, filesScanned:$scanned}' \
>>"$WORK/scans.jsonl"
else
# Carries the same paths a success does: a scan that crashed part-way may still have
# written a partial file, and the report needs to be able to name it.
err=$(tail -c 800 "$WORK/err.$stem" 2>/dev/null || true)
[ -n "$err" ] || err="semgrep exited $rc"
jq -nc --arg lang "$lang" --arg ruleset "$ruleset" --arg json "$json" \
--arg sarif "$sarif" --arg error "$err" \
'{lang:$lang, ruleset:$ruleset, json:$json, sarif:$sarif, error:$error}' \
>>"$WORK/failed.jsonl"
fi
done <"$SCAN_LIST"
PRO_JSON=false
[ -z "$PRO" ] || PRO_JSON=true
jq -n \
--arg outputDir "$OUTPUT_ROOT" \
--arg rawDir "$RAW_DIR" \
--arg reposPath "$REPOS_DIR" \
--arg mode "$MODE" \
--arg excludePattern "$EXCLUDE_PATTERN" \
--argjson pro "$PRO_JSON" \
--slurpfile scans "$WORK/scans.jsonl" \
--slurpfile failed "$WORK/failed.jsonl" \
--rawfile skippedRaw "$SKIPPED" \
--rawfile alsoSharedRaw "$ALSO_SHARED" \
--rawfile unscopedRaw "$UNSCOPED" \
--rawfile coveredNothingRaw "$COVERED_NOTHING" \
'{
outputDir: $outputDir, rawDir: $rawDir, reposPath: $reposPath, mode: $mode, pro: $pro,
excludePattern: $excludePattern,
scans: $scans, failed: $failed,
skipped: ($skippedRaw | split("\n") | map(select(length > 0)) | map(split("\t"))
| map({ruleset: .[0], reason: (.[1] // "clone failed")})),
alsoShared: ($alsoSharedRaw | split("\n") | map(select(length > 0)) | unique),
unscoped: ($unscopedRaw | split("\n") | map(select(length > 0)) | unique),
coveredNothing: ($coveredNothingRaw | split("\n") | map(select(length > 0)) | unique)
}' >"$OUTPUT_ROOT/scans.json"
n_ok=$(jq '.scans | length' "$OUTPUT_ROOT/scans.json")
n_failed=$(jq '.failed | length' "$OUTPUT_ROOT/scans.json")
n_skipped=$(jq '.skipped | length' "$OUTPUT_ROOT/scans.json")
echo "$n_ok scan(s) succeeded, $n_failed failed, $n_skipped skipped" >&2
echo "$OUTPUT_ROOT/scans.json"
# A run where nothing succeeded is a failed run, not a clean one. Reporting zero findings from
# it is the failure this whole script is shaped to avoid.
[ "$n_ok" -gt 0 ] || exit 1
plugins/static-analysis/skills/semgrep/scripts/test_merge_sarif.py
+581
View File
@@ -0,0 +1,581 @@
# /// script
# requires-python = ">=3.11"
# dependencies = ["pytest>=8"]
# ///
"""Tests for merge_sarif.py, with the weight on --important.
The important-only merge rests on one claim: a finding's identity in semgrep's JSON output
(check_id, path, start.line) is the same triple SARIF carries as (ruleId, uri,
region.startLine). test_key_contract pins the field names this script reads, but builds both
halves itself, so only test_key_contract_against_real_semgrep can notice semgrep changing
either shape. That one runs semgrep and compares the two records of one real finding.
The negatives matter as much: a post-filter that ran over only some scans, or wrote a file
that will not parse, must fail the merge. Filtering against a partial key set drops real
findings from the primary deliverable and nothing downstream could notice. The exception is a
scan recorded under .failed in scans.json, whose output is whatever a dying process wrote:
--scans drops those, so one crashed scan cannot deny every healthy scan a deliverable.
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
from pathlib import Path
import pytest
from merge_sarif import (
filter_to_keys,
json_key,
merge_sarif_pure_python,
sarif_key,
surviving_keys,
)
SCRIPT = Path(__file__).with_name("merge_sarif.py")
RULE = "python.lang.security.insecure-hash-algorithms-md5.insecure-hash-algorithm-md5"
OTHER = "python.lang.security.audit.subprocess-shell-true.subprocess-shell-true"
def sarif_result(rule: str, uri: str, line: int) -> dict:
return {
"ruleId": rule,
"message": {"text": "finding"},
"locations": [
{
"physicalLocation": {
"artifactLocation": {"uri": uri},
"region": {"startLine": line, "startColumn": 1},
}
}
],
}
def sarif_doc(*results: dict) -> dict:
return {
"version": "2.1.0",
"runs": [{"tool": {"driver": {"name": "semgrep", "rules": []}}, "results": list(results)}],
}
def json_result(rule: str, path: str, line: int) -> dict:
return {"check_id": rule, "path": path, "start": {"line": line, "col": 1}, "extra": {}}
def write_scan(raw: Path, stem: str, sarif: list[dict], filtered: list[dict] | None) -> None:
"""One scan's output: the SARIF the merge reads, and optionally its post-filtered JSON."""
(raw / f"{stem}.sarif").write_text(json.dumps(sarif_doc(*sarif)))
if filtered is not None:
(raw / f"{stem}-important.json").write_text(
json.dumps({"results": filtered, "errors": [], "paths": {}})
)
def run_merge(raw: Path, out: Path, *flags: str) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(SCRIPT), str(raw), str(out), *flags],
capture_output=True,
text=True,
)
def count(sarif_file: Path) -> int:
data = json.loads(sarif_file.read_text())
return sum(len(run.get("results", [])) for run in data.get("runs", []))
# --------------------------------------------------------------- the cross-format contract
def test_key_contract():
"""The shapes this script expects, pinned.
Both halves are built by this file from one literal path, so this fixes the field names
`sarif_key` and `json_key` read and nothing more. It cannot notice semgrep changing either
output — test_key_contract_against_real_semgrep below is the one that can.
"""
from_json = json_result(RULE, "src/app.py", 5)
from_sarif = sarif_result(RULE, "src/app.py", 5)
assert json_key(from_json) == sarif_key(from_sarif) == (RULE, "src/app.py", 5)
MD5_RULE = """\
rules:
- id: insecure-md5
pattern: hashlib.md5(...)
message: md5 is insecure
languages: [python]
severity: WARNING
"""
def semgrep_bin() -> str:
"""Fail rather than skip, the same reason run_workflow_tests.sh fails without node.
A skip here reads as a clean run while the only check that could catch cross-format drift
silently did not execute. semgrep is this plugin's own dependency and CI installs it.
"""
found = shutil.which("semgrep")
if not found:
raise AssertionError(
"semgrep is not installed, so the JSON/SARIF contract went unverified. "
"Install it (pip install semgrep) — this suite must not pass without it."
)
return found
@pytest.mark.parametrize("absolute", [True, False])
def test_key_contract_against_real_semgrep(tmp_path, absolute):
"""The contract read out of semgrep itself, over one real finding.
--important rests entirely on the claim that (check_id, path, start.line) in the JSON is
(ruleId, uri, region.startLine) in the SARIF. Only this test can see that claim break: if
semgrep changes either shape, the keys stop matching, every finding is dropped and the
deliverable goes empty. run-scans.sh always passes an absolute target; the relative case is
parametrized because a path shape is exactly where the two formats would diverge first.
"""
src = tmp_path / "src"
src.mkdir()
(src / "app.py").write_text(
"import hashlib\ndef f(x):\n return hashlib.md5(x).hexdigest()\n"
)
rule = tmp_path / "md5.yaml"
rule.write_text(MD5_RULE)
json_out = tmp_path / "out.json"
sarif_out = tmp_path / "out.sarif"
proc = subprocess.run(
[
semgrep_bin(),
"--metrics=off",
"--config",
str(rule),
"--json",
"-o",
str(json_out),
f"--sarif-output={sarif_out}",
str(src) if absolute else "src",
],
cwd=tmp_path,
capture_output=True,
text=True,
)
assert json_out.is_file(), f"semgrep wrote no JSON: {proc.stderr}"
assert sarif_out.is_file(), f"semgrep wrote no SARIF: {proc.stderr}"
json_results = json.loads(json_out.read_text())["results"]
sarif_results = json.loads(sarif_out.read_text())["runs"][0]["results"]
assert len(json_results) == 1, f"expected one JSON finding, got {len(json_results)}"
assert len(sarif_results) == 1, f"expected one SARIF finding, got {len(sarif_results)}"
# The whole contract in one line. A mismatch here is the empty-deliverable bug, found at
# its source rather than as a zero-finding results.sarif nobody can explain.
assert json_key(json_results[0]) == sarif_key(sarif_results[0])
def test_keys_differ_on_line():
assert json_key(json_result(RULE, "src/app.py", 5)) != sarif_key(
sarif_result(RULE, "src/app.py", 6)
)
def test_sarif_key_tolerates_a_result_with_no_location():
assert sarif_key({"ruleId": RULE}) == (RULE, "", 0)
# ------------------------------------------------------------------------- surviving_keys
def test_surviving_keys_reads_every_scan(tmp_path):
raw = tmp_path
write_scan(raw, "py", [sarif_result(RULE, "a.py", 5)], [json_result(RULE, "a.py", 5)])
write_scan(raw, "secrets", [sarif_result(OTHER, "b.py", 9)], [json_result(OTHER, "b.py", 9)])
keys = surviving_keys(sorted(raw.glob("*.sarif")))
assert keys == {(RULE, "a.py", 5), (OTHER, "b.py", 9)}
def test_surviving_keys_is_empty_when_the_filter_kept_nothing(tmp_path):
"""A real outcome, distinct from a filter that never ran: the files exist and are empty."""
write_scan(tmp_path, "python-python", [sarif_result(RULE, "a.py", 5)], [])
assert surviving_keys(sorted(tmp_path.glob("*.sarif"))) == set()
def test_a_scan_with_no_filtered_json_fails_the_merge(tmp_path):
"""The silent-omission case: without this, that scan's findings vanish from results.sarif."""
raw = tmp_path
write_scan(raw, "py", [sarif_result(RULE, "a.py", 5)], [json_result(RULE, "a.py", 5)])
write_scan(raw, "all-secrets", [sarif_result(OTHER, "b.py", 9)], None)
with pytest.raises(ValueError, match="all-secrets-important.json"):
surviving_keys(sorted(raw.glob("*.sarif")))
def test_an_unparseable_filter_file_fails_the_merge(tmp_path):
write_scan(tmp_path, "python-python", [sarif_result(RULE, "a.py", 5)], [])
(tmp_path / "python-python-important.json").write_text("not json")
with pytest.raises(ValueError, match="not valid JSON"):
surviving_keys(sorted(tmp_path.glob("*.sarif")))
def test_a_filter_file_with_no_results_array_fails_the_merge(tmp_path):
"""Catches a SARIF file handed in where a filtered JSON belongs; it would filter to nothing."""
write_scan(tmp_path, "python-python", [sarif_result(RULE, "a.py", 5)], [])
(tmp_path / "python-python-important.json").write_text(json.dumps(sarif_doc()))
with pytest.raises(ValueError, match="no .results array"):
surviving_keys(sorted(tmp_path.glob("*.sarif")))
# -------------------------------------------------------------------------- filter_to_keys
def test_filter_keeps_only_surviving_findings():
merged = sarif_doc(sarif_result(RULE, "a.py", 5), sarif_result(OTHER, "a.py", 3))
kept, dropped = filter_to_keys(merged, {(RULE, "a.py", 5)})
assert (kept, dropped) == (1, 1)
assert [r["ruleId"] for r in merged["runs"][0]["results"]] == [RULE]
def test_filter_against_an_empty_key_set_empties_the_results():
merged = sarif_doc(sarif_result(RULE, "a.py", 5))
assert filter_to_keys(merged, set()) == (0, 1)
assert merged["runs"][0]["results"] == []
# ------------------------------------------------------------------------------ end to end
def test_important_merge_filters_the_deliverable(tmp_path):
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw,
"python-python",
[sarif_result(RULE, "a.py", 5), sarif_result(OTHER, "a.py", 3)],
[json_result(RULE, "a.py", 5)],
)
out = tmp_path / "results" / "results.sarif"
proc = run_merge(raw, out, "--important")
assert proc.returncode == 0, proc.stderr
assert count(out) == 1
assert json.loads(out.read_text())["runs"][0]["results"][0]["ruleId"] == RULE
def test_run_all_merge_keeps_everything(tmp_path):
"""The default path must be unchanged by the flag's existence."""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw,
"python-python",
[sarif_result(RULE, "a.py", 5), sarif_result(OTHER, "a.py", 3)],
[json_result(RULE, "a.py", 5)],
)
out = tmp_path / "results" / "results.sarif"
assert run_merge(raw, out).returncode == 0
assert count(out) == 2
def test_important_without_a_post_filter_fails_and_writes_nothing(tmp_path):
"""The whole point of resolving keys before the merge: no half-right file on disk."""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(raw, "python-python", [sarif_result(RULE, "a.py", 5)], None)
out = tmp_path / "results" / "results.sarif"
proc = run_merge(raw, out, "--important")
assert proc.returncode == 1
assert "post-filtered JSON" in proc.stderr
assert not out.exists()
def test_a_total_key_mismatch_fails_the_merge(tmp_path):
"""Cross-format drift, forced: the filter kept a finding no merged result matches.
Without the guard this writes a zero-finding results.sarif and exits 0, so important-only
reports a clean run that found nothing while the JSON side has findings.
"""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw,
"python-python",
[sarif_result(RULE, "src/app.py", 5)],
[json_result(RULE, "/abs/proj/src/app.py", 5)],
)
out = tmp_path / "results" / "results.sarif"
proc = run_merge(raw, out, "--important")
assert proc.returncode == 1, proc.stdout
assert "none of them matched" in proc.stderr
assert not out.exists()
def test_a_filter_that_kept_nothing_is_not_a_mismatch(tmp_path):
"""The guard is conditioned on the key set, not on the kept count.
A post-filter that legitimately excluded every finding is a real zero. If this goes red,
important-only can no longer report an honest empty result.
"""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(raw, "python-python", [sarif_result(RULE, "a.py", 5)], [])
out = tmp_path / "results" / "results.sarif"
proc = run_merge(raw, out, "--important")
assert proc.returncode == 0, proc.stderr
assert count(out) == 0
def test_a_partial_key_mismatch_still_merges(tmp_path):
"""One matching key is enough to prove the formats still agree; the rest is the filter."""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw,
"python-python",
[sarif_result(RULE, "a.py", 5), sarif_result(OTHER, "a.py", 3)],
[json_result(RULE, "a.py", 5), json_result(OTHER, "/elsewhere/a.py", 3)],
)
out = tmp_path / "results" / "results.sarif"
proc = run_merge(raw, out, "--important")
assert proc.returncode == 0, proc.stderr
assert count(out) == 1
def test_important_leaves_an_existing_deliverable_alone_when_it_fails(tmp_path):
raw = tmp_path / "raw"
raw.mkdir()
write_scan(raw, "python-python", [sarif_result(RULE, "a.py", 5)], None)
out = tmp_path / "results.sarif"
out.write_text(json.dumps(sarif_doc(sarif_result(RULE, "a.py", 5))))
before = out.read_text()
assert run_merge(raw, out, "--important").returncode == 1
assert out.read_text() == before
# --------------------------------------------------------------------- unparseable SARIF
def test_an_unparseable_sarif_is_named_on_stdout(tmp_path):
"""The silent-omission case this whole flag set exists to prevent.
A scan can exit 0, write a valid .json — so it lands in .scans with a finding count — and
still leave a truncated .sarif. The merge drops it and the total is short by exactly those
findings. On stderr that was invisible to the report; it has to be in the same stream the
Report phase reads, and named, or the run presents as clean.
"""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(raw, "python-python", [sarif_result(RULE, "a.py", 5)], None)
(raw / "python-broken.sarif").write_text('{"runs":[{"results":[')
out = tmp_path / "results.sarif"
proc = run_merge(raw, out)
assert proc.returncode == 0, proc.stderr
assert count(out) == 1, "the healthy scan must still merge"
assert "unparseable" in proc.stdout
assert "python-broken.sarif" in proc.stdout, "the file must be named, not just counted"
def test_every_sarif_unparseable_is_an_error(tmp_path):
"""Zero findings from zero readable files is a broken run, not a clean one."""
raw = tmp_path / "raw"
raw.mkdir()
(raw / "python-broken.sarif").write_text('{"runs":[{"results":[')
out = tmp_path / "results.sarif"
proc = run_merge(raw, out)
assert proc.returncode == 1
assert "nothing to merge" in proc.stderr
assert not out.exists()
def test_the_merge_returns_what_it_could_not_read(tmp_path):
"""The list is a return value, so a caller cannot forget to look at it."""
write_scan(tmp_path, "ok", [sarif_result(RULE, "a.py", 5)], None)
(tmp_path / "bad.sarif").write_text("{{{")
merged, unparseable = merge_sarif_pure_python(sorted(tmp_path.glob("*.sarif")))
assert [Path(p).name for p in unparseable] == ["bad.sarif"]
assert sum(len(r["results"]) for r in merged["runs"]) == 1
# ------------------------------------------------------------------------- one merge backend
def test_the_merge_shells_out_to_nothing(tmp_path):
"""Run with an empty PATH, so any external merge tool is unreachable.
The merge used to try `npx @microsoft/sarif-multitool` first and fall back to Python, which
made the result depend on whether that package sat in the npx cache. Only the Python merge
dedups on sarif_key, and multitool rewrites artifactLocation.uri, so on a machine that had
it cached --important would match nothing and blame a semgrep format change. One backend,
same answer everywhere.
"""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw, "python-python", [sarif_result(RULE, "a.py", 5)], [json_result(RULE, "a.py", 5)]
)
out = tmp_path / "results.sarif"
proc = subprocess.run(
[sys.executable, str(SCRIPT), str(raw), str(out), "--important"],
capture_output=True,
text=True,
env={"PATH": "", "HOME": str(tmp_path)},
)
assert proc.returncode == 0, proc.stderr
assert count(out) == 1
assert "multitool" not in proc.stdout.lower(), "no external merge tool may be consulted"
# The empty PATH above proves the merge survives without a backend, not that it stopped
# looking for one: a reintroduced optional branch would just fall back and pass. This is
# the assertion that fails if one comes back.
assert "import subprocess" not in SCRIPT.read_text(), (
"merge_sarif.py must not shell out; a second merge backend disagrees with this one "
"on dedup and on artifactLocation.uri, and which one runs would depend on the machine"
)
# ------------------------------------------------------------------------------------ --scans
def write_scans_json(path: Path, succeeded: list[Path], failed: list[Path]) -> Path:
"""A scans.json in run-scans.sh's shape. Both lists carry the same `sarif` key."""
path.write_text(
json.dumps(
{
"scans": [
{"lang": "python", "ruleset": "p/python", "sarif": str(p)} for p in succeeded
],
"failed": [
{"lang": "python", "ruleset": "p/x", "sarif": str(p), "error": "exited 7"}
for p in failed
],
"skipped": [],
}
)
)
return path
def dead_scan(raw: Path, stem: str) -> Path:
"""A scan recorded under .failed: its SARIF is on disk with no post-filter beside it."""
sarif = raw / f"{stem}.sarif"
sarif.write_text(json.dumps(sarif_doc(sarif_result(OTHER, "b.py", 9))))
return sarif
def test_a_failed_scan_no_longer_denies_every_other_scan_a_deliverable(tmp_path):
"""The point of the flag: one dead scan must not take the whole important-only merge."""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw, "python-python", [sarif_result(RULE, "a.py", 5)], [json_result(RULE, "a.py", 5)]
)
dead = dead_scan(raw, "python-broken")
scans = write_scans_json(tmp_path / "scans.json", [raw / "python-python.sarif"], [dead])
out = tmp_path / "results.sarif"
proc = run_merge(raw, out, "--important", "--scans", str(scans))
assert proc.returncode == 0, proc.stderr
assert count(out) == 1
assert "python-broken.sarif" in proc.stdout, "the excluded file must be named for the report"
def test_the_same_run_without_scans_json_still_fails(tmp_path):
"""Pins that the flag is what makes it survivable, not a change in the merge's strictness."""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw, "python-python", [sarif_result(RULE, "a.py", 5)], [json_result(RULE, "a.py", 5)]
)
dead_scan(raw, "python-broken")
out = tmp_path / "results.sarif"
assert run_merge(raw, out, "--important").returncode == 1
assert not out.exists()
def test_a_succeeded_scan_missing_its_filter_still_fails(tmp_path):
"""Only failed scans are exempt.
A healthy scan with no post-filter beside it still aborts the merge: its findings are real,
and filtering against a key set that never saw them drops them from the deliverable with
nothing downstream able to notice.
"""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw, "python-python", [sarif_result(RULE, "a.py", 5)], [json_result(RULE, "a.py", 5)]
)
healthy = dead_scan(raw, "python-other") # same shape, but recorded as a success below
scans = write_scans_json(tmp_path / "scans.json", [raw / "python-python.sarif", healthy], [])
out = tmp_path / "results.sarif"
proc = run_merge(raw, out, "--important", "--scans", str(scans))
assert proc.returncode == 1
assert "python-other-important.json" in proc.stderr
assert not out.exists()
def test_run_all_also_drops_a_failed_scan_output(tmp_path):
"""A dead process's file is not a scan result in either mode."""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(raw, "python-python", [sarif_result(RULE, "a.py", 5)], None)
dead = dead_scan(raw, "python-broken")
scans = write_scans_json(tmp_path / "scans.json", [raw / "python-python.sarif"], [dead])
out = tmp_path / "results.sarif"
proc = run_merge(raw, out, "--scans", str(scans))
assert proc.returncode == 0, proc.stderr
assert count(out) == 1, "the failed scan's finding must not reach the merge"
def test_every_scan_failed_is_an_error_not_an_empty_merge(tmp_path):
"""Excluding everything would otherwise write an empty SARIF and report a clean run."""
raw = tmp_path / "raw"
raw.mkdir()
dead = dead_scan(raw, "python-broken")
scans = write_scans_json(tmp_path / "scans.json", [], [dead])
out = tmp_path / "results.sarif"
proc = run_merge(raw, out, "--scans", str(scans))
assert proc.returncode == 1
assert "nothing to merge" in proc.stderr
assert not out.exists()
def test_a_scans_json_with_no_failed_array_is_rejected(tmp_path):
"""Catches the wrong file being passed; treating it as 'nothing failed' would be silent."""
raw = tmp_path / "raw"
raw.mkdir()
write_scan(
raw, "python-python", [sarif_result(RULE, "a.py", 5)], [json_result(RULE, "a.py", 5)]
)
bad = tmp_path / "scans.json"
bad.write_text(json.dumps({"scans": []}))
out = tmp_path / "results.sarif"
proc = run_merge(raw, out, "--important", "--scans", str(bad))
assert proc.returncode == 1
assert "not a scans.json" in proc.stderr
assert not out.exists()
def test_scans_flag_without_a_path_is_rejected(tmp_path):
raw = tmp_path / "raw"
raw.mkdir()
write_scan(raw, "python-python", [sarif_result(RULE, "a.py", 5)], None)
proc = run_merge(raw, tmp_path / "results.sarif", "--scans")
assert proc.returncode == 1
assert "--scans needs" in proc.stderr
def test_an_empty_raw_directory_is_an_error(tmp_path):
raw = tmp_path / "raw"
raw.mkdir()
assert run_merge(raw, tmp_path / "o.sarif").returncode == 1
# ------------------------------------------------------------------------------ merge dedup
def test_merge_dedups_one_finding_flagged_by_two_rulesets(tmp_path):
"""The reason the report counts from the merge and never sums per-scan counts."""
write_scan(tmp_path, "python-python", [sarif_result(RULE, "a.py", 5)], None)
write_scan(tmp_path, "all-audit", [sarif_result(RULE, "a.py", 5)], None)
merged, _ = merge_sarif_pure_python(sorted(tmp_path.glob("*.sarif")))
assert sum(len(run["results"]) for run in merged["runs"]) == 1
plugins/static-analysis/skills/semgrep/workflows/scan-workflow.md
+151 -41
View File
@@ -45,10 +45,22 @@ else
OUTPUT_DIR="${BASE}_${N}"
fi
mkdir -p "$OUTPUT_DIR/raw" "$OUTPUT_DIR/results"
# Absolute from here on. run-scans.sh rejects a relative path, and that rejection lands
# *after* the user has passed the hard gate, so a path this skill generated itself would send
# them back through approval.
OUTPUT_DIR=$(cd "$OUTPUT_DIR" && pwd)
# The -d test first: `cd ""` returns 0, so a TARGET that was never bound would pass a bare
# `cd || exit` and silently resolve to the session's CWD, scanning whatever happens to be there.
[ -n "$TARGET" ] && [ -d "$TARGET" ] || { echo "ERROR: TARGET is unset or not a directory"; exit 1; }
TARGET=$(cd "$TARGET" && pwd)
echo "Output directory: $OUTPUT_DIR"
echo "Target: $TARGET"
```
`$OUTPUT_DIR` is used by all subsequent steps. Pass its **absolute path** to scanner subagents. Scanners write raw output to `$OUTPUT_DIR/raw/`; merged/filtered results go to `$OUTPUT_DIR/results/`.
Pass `$TARGET` and `$OUTPUT_DIR` to Step 4 exactly as resolved here. Do not re-derive either.
`$OUTPUT_DIR` is used by all subsequent steps. Raw per-scan output goes to `$OUTPUT_DIR/raw/`; merged and filtered results go to `$OUTPUT_DIR/results/`.
**Detect Pro availability** (requires Bash):
@@ -58,7 +70,10 @@ if ! command -v semgrep >/dev/null 2>&1; then
exit 1
fi
semgrep --version
semgrep --pro --validate --config p/default 2>/dev/null && echo "Pro: AVAILABLE" || echo "Pro: NOT AVAILABLE"
# --metrics=off applies here too. This is the first semgrep invocation of the run and it
# resolves p/default against the registry, so without the flag an audit phones home before
# the user has approved anything. Principle 1 has no exceptions.
semgrep --pro --validate --metrics=off --config p/default 2>/dev/null && echo "Pro: AVAILABLE" || echo "Pro: NOT AVAILABLE"
```
**Detect languages** using Glob (not Bash). Run these patterns against the target directory and count matches:
@@ -148,6 +163,10 @@ Present plan to user with **explicit ruleset listing**:
**Output directory:** $OUTPUT_DIR
**Engine:** Semgrep Pro (cross-file analysis) | Semgrep OSS (single-file)
**Scan mode:** Run all | Important only (security vulns, medium-high confidence/impact)
[in important-only mode, add:] Note: important-only passes --severity WARNING --severity ERROR
to every command, including the third-party repos. Trail of Bits / 0xdea / Decurity rules that
ship with CLI severity INFO are dropped at scan time, before the metadata filter that would
otherwise keep them. Choose "Run all" if you want those.
### Detected Languages/Technologies:
- Python (1,234 files) - Django framework detected
@@ -200,78 +219,127 @@ Before marking Step 3 complete:
- [ ] User given opportunity to modify rulesets
- [ ] User explicitly approved (quote their confirmation)
- [ ] **Final ruleset list captured for Step 4**
- [ ] Agent type listed: `static-analysis:semgrep-scanner`
### Log Approved Rulesets
After approval, write the approved rulesets to `$OUTPUT_DIR/rulesets.txt`:
After approval, write the approved plan to `$OUTPUT_DIR/rulesets.json`. This is the same file
Step 4 hands to the scanner: what the user approved and what runs are one artifact, so there is
no second copy to transcribe and no way for the two to disagree.
Fill in the plan that was just approved. Every value is an array, even a single ruleset:
```bash
cat > "$OUTPUT_DIR/rulesets.txt" << RULESETS
# Semgrep Scan — Approved Rulesets
# Generated: $(date -Iseconds)
# Scan mode: <run-all|important-only>
## Rulesets:
<one ruleset per line, e.g.:>
p/security-audit
p/secrets
p/python
p/django
https://github.com/trailofbits/semgrep-rules
cat > "$OUTPUT_DIR/rulesets.json" << 'RULESETS'
{
"baseline": [<the always-on rulesets from Step 2>],
"<each detected language>": [<its approved rulesets>],
"third_party": [<approved repository URLs>]
}
RULESETS
```
One key per language *detected in Step 1*, using the lowercase names from that step. A language
key for a language the target does not contain scans nothing: its `--include` globs match no
file, semgrep exits 0 with an empty result, and the report shows the ruleset with 0 findings
exactly as it would for a ruleset that ran and found nothing. The script counts the files each
scan opened and lists any that covered nothing under `coveredNothing` in `scans.json`, but
getting the languages right here is what stops it happening.
Repository URLs go under `third_party` and nowhere else. Registry identifiers like `p/python`
go under a language key; a `https://…` there fails the identifier check and the script exits
without scanning.
---
## Step 4: Spawn Parallel Scan Tasks
## Step 4: Run the Scans
> **Entry:** Step 3 approved — user explicitly confirmed the plan.
> **Exit:** All scan Tasks completed; result files exist in `$OUTPUT_DIR/raw/`.
> **Exit:** `$OUTPUT_DIR/scans.json` exists; result files exist in `$OUTPUT_DIR/raw/`.
**Use `$OUTPUT_DIR` resolved in Step 1.** It already exists; no need to create it again. Scanners write all output to `$OUTPUT_DIR/raw/`.
Run the script against the plan Step 3 already wrote. One Bash call; there is no subagent in
this step, and no second copy of the ruleset list to compose here.
**Spawn N Tasks in a SINGLE message** (one per language category) using `subagent_type: static-analysis:semgrep-scanner`.
```bash
{baseDir}/scripts/run-scans.sh \
--target "$TARGET" \
--output-dir "$OUTPUT_DIR" \
--mode run-all \
--rulesets "$OUTPUT_DIR/rulesets.json"
```
Use the scanner task prompt template from [scanner-task-prompt.md](../references/scanner-task-prompt.md).
Do not rewrite `rulesets.json` here. It is the plan the user approved at the Step 3 gate, and
regenerating it at this point is how a ruleset nobody agreed to reaches the scanner. If it needs
to change, go back to Step 3 and get the change approved.
**Mode-dependent scanner flags:**
- **Run all**: No additional flags
- **Important only**: Add `--severity MEDIUM --severity HIGH --severity CRITICAL` to every `semgrep` command
`--mode` is `run-all` or `important-only`. Add `--pro` only when Step 1
printed `Pro: AVAILABLE`; it puts `--pro` on every command, so passing it without a licence
fails every scan in the run. `--jobs N` sets how many semgrep processes run at once (default 4);
semgrep holds the rules and the scanned ASTs in memory, so raising it on a large tree trades
memory for wall-clock.
**Example — 3 Language Scan (with approved rulesets):**
Repository URLs go under `third_party` and nowhere else. A `https://…` under a language key
fails the registry-identifier check and the script exits without scanning.
Spawn these 3 Tasks in a SINGLE message:
The script clones each third-party repo once, generates every `semgrep` command, and runs them
in batches. `--metrics=off`, the `--include` scoping rule, `--exclude` for the output directory
and the severity flags are all its job, not yours. It writes `$OUTPUT_DIR/scans.json`:
1. **Task: Python Scanner** — Rulesets: p/python, p/django, p/security-audit, p/secrets, trailofbits → `$OUTPUT_DIR/raw/python-*.json`
2. **Task: JavaScript Scanner** — Rulesets: p/javascript, p/react, p/nodejs, p/security-audit, p/secrets, trailofbits → `$OUTPUT_DIR/raw/js-*.json`
3. **Task: Docker Scanner** — Rulesets: p/dockerfile → `$OUTPUT_DIR/raw/docker-*.json`
| Field | Meaning |
|-------|---------|
| `scans` | Rulesets that ran, with `json`, `sarif`, `findings` and `filesScanned` for each. `findings` is counted from the JSON the scan wrote; `filesScanned` is how many files semgrep opened, or `-1` when it did not say |
| `coveredNothing` | Rulesets that ran against zero files, because their `--include` globs matched nothing in the target. They report 0 findings exactly like a ruleset that ran and found nothing, so a plan naming a language the target does not contain reads as a clean audit. **Must be shown.** |
| `failed` | Rulesets that ran and did not produce usable output, with the `json` and `sarif` paths they may have partly written, and the stderr excerpt. **Must be shown to the user.** |
| `skipped` | Rulesets dropped before scanning, mostly repos that would not clone. **Must be shown.** |
| `unscoped` | Languages with no `--include` globs, which ran against every file |
| `alsoShared` | Rulesets dropped from a language because the same ruleset is already running unscoped over the whole target. Coverage is unaffected; report them so a per-ruleset accounting adds up |
| `excludePattern` | Set when the output directory sits inside the target: the pattern passed as `--exclude` to every scan, or `""`. semgrep matches it anywhere in the tree, so `out` also drops `src/out/`. **Must be shown when non-empty.** |
| `reposPath` | The clone directory Step 5 deletes |
### Operational Notes
**A non-zero exit means no scan succeeded.** The script exits 1 when `scans` is empty, so a run
that produced nothing fails loudly rather than handing Step 5 an empty result to report as zero
findings. Read the message, say that no scan ran, and stop; do not retry with adjusted
arguments, because the approved plan is what produced them.
- Always use **absolute paths** for `[TARGET]` — subagents can't resolve relative paths
- Clone GitHub URL rulesets into `$OUTPUT_DIR/repos/` — never pass URLs directly to `--config` (semgrep's URL handling fails on repos with non-standard YAML)
- Delete `$OUTPUT_DIR/repos/` after all scans complete
- Run rulesets in parallel with `&` and `wait`, not sequentially
- Use `--include="*.py"` for language-specific rulesets, but NOT for cross-language rulesets (p/security-audit, p/secrets, third-party repos)
**If `failed` or `skipped` is non-empty**, carry both into the Step 5 report. A run that covered
four of nine rulesets reads exactly like one that covered four of four unless you say otherwise.
---
## Step 5: Merge Results and Report
> **Entry:** Step 4 complete — all scan Tasks finished.
> **Exit:** `results.sarif` exists in `$OUTPUT_DIR/results/` and is valid JSON.
> **Entry:** Step 4 complete — the workflow returned.
> **Exit:** `results.sarif` exists in `$OUTPUT_DIR/results/` and is valid JSON; `repos/` deleted.
Read the result with `jq` from `$OUTPUT_DIR/scans.json`. Every entry there was written after
the script checked the exit code and confirmed both output files were non-empty, so the entries
do not need re-verifying.
**Important-only mode: Post-filter before merge.** Apply the filter from [scan-modes.md](../references/scan-modes.md) ("Filter All Result Files in a Directory" section) to each result JSON in `$OUTPUT_DIR/raw/`. The filter creates `*-important.json` files alongside the originals — the originals are preserved unmodified.
**Generate merged SARIF** using the merge script. The resolved path is in SKILL.md's "Merge command" section — use that exact path:
```bash
uv run {baseDir}/scripts/merge_sarif.py $OUTPUT_DIR/raw $OUTPUT_DIR/results/results.sarif
# run-all
uv run {baseDir}/scripts/merge_sarif.py "$OUTPUT_DIR/raw" "$OUTPUT_DIR/results/results.sarif" \
--scans "$OUTPUT_DIR/scans.json"
# important-only, once the post-filter above has run over every file in raw/
uv run {baseDir}/scripts/merge_sarif.py "$OUTPUT_DIR/raw" "$OUTPUT_DIR/results/results.sarif" \
--important --scans "$OUTPUT_DIR/scans.json"
```
- **Run-all mode:** The script merges all `*.sarif` files from `$OUTPUT_DIR/raw/`.
- **Important-only mode:** Run the post-filter first (creates `*-important.json` in `raw/`), then run the merge script. Raw SARIF files are unaffected by the JSON post-filter, so the merge operates on the unfiltered SARIF. For SARIF-level filtering, apply the jq post-filter from scan-modes.md to `$OUTPUT_DIR/results/results.sarif` after merge.
- **Important-only mode:** `--important` is not optional. The JSON post-filter does not touch the
SARIF files the merge reads, so without that flag `results.sarif` keeps every finding the mode
exists to exclude while the JSON side is correctly filtered, and the Total findings counted from
it is the run-all total.
Do **not** try to run the jq filter from scan-modes.md against a `.sarif` file. It reads
`.results[].extra.metadata`, which SARIF does not have — there is no top-level `.results` at
all — so it exits with `Cannot iterate over null` and, if redirected over its own input,
truncates the merged SARIF to nothing. `--important` matches findings across the two formats on
`(rule, file, line)`, the same key the merge dedups on, and fails rather than filtering if any
scan in `raw/` has no `*-important.json` beside it.
**Verify merged SARIF is valid:**
@@ -281,6 +349,14 @@ python -c "import json; d=json.load(open('$OUTPUT_DIR/results/results.sarif'));
If verification fails, the merge script produced invalid output — investigate before reporting.
**Delete the cloned rulesets** once the merge has succeeded. The workflow clones each
third-party repo into `repos/` and leaves it there for the scanners; this is the only place
the deletion happens, and nothing that reads it is still running by now.
```bash
[ -n "$OUTPUT_DIR" ] && rm -rf "$OUTPUT_DIR/repos"
```
**Report to user:**
```
@@ -288,7 +364,8 @@ If verification fails, the merge script produced invalid output — investigate
**Scanned:** 1,804 files
**Rulesets used:** 9 (including Trail of Bits)
**Total findings:** 156
**Total findings:** 156 [count this from results.sarif, never by summing scans[].findings:
one finding flagged by two rulesets is one row in the merge and two in that sum]
### By Severity:
- ERROR: 5
@@ -302,10 +379,43 @@ If verification fails, the merge script produced invalid output — investigate
- Insecure configuration: 12
- Code quality: 8
### Did Not Run:
[omit this section only when failed and skipped are both empty]
- Skipped: <ruleset> — <reason from the workflow>
- Failed: <ruleset> — <error from the workflow>
### Also Covered Unscoped:
[omit when alsoShared is empty]
- <ruleset> — already running over the whole target from the baseline, so it was not scanned
again under <language>. Coverage is unaffected; this is why the ruleset count and the scan
count differ
### Ran Unscoped:
[omit when unscoped is empty]
- <language> — no --include map, so its rulesets ran against every file
### Covered Nothing:
[omit when coveredNothing is empty]
- <language>/<ruleset> — matched no file in the target, so it reports 0 findings without having
looked at anything. Check the plan against the languages Step 1 detected: this is what a
ruleset for a language the target does not contain looks like
### Missing From The Merge:
[omit when the merge printed no "unparseable:" line]
- <file> — the scan succeeded and is counted in scans.json, but its SARIF could not be parsed,
so its findings are not in results.sarif. The total below is short by that scan's `findings`
count from scans.json
### Excluded From Every Scan:
[omit when excludePattern is empty]
- <excludePattern> — the output directory sits inside the target, so this pattern was excluded
from every scan. semgrep matches it anywhere in the tree, so any other directory with that
name was skipped too. Move the output directory outside the target to scan those files
Results written to:
- $OUTPUT_DIR/results/results.sarif (merged SARIF)
- $OUTPUT_DIR/raw/ (per-scan raw results, unfiltered)
- $OUTPUT_DIR/rulesets.txt (approved rulesets)
- $OUTPUT_DIR/rulesets.json (the approved plan, as passed to the scanner)
```
**Verify** before reporting: confirm `results.sarif` exists and is valid JSON.
plugins/static-analysis/tests/run_scan_tests.sh
Executable
+512
View File
@@ -0,0 +1,512 @@
#!/usr/bin/env bash
# Regression suite for scripts/run-scans.sh, discovered by CI's run_*.sh glob.
#
# Command generation is checked with --dry-run; execution, exit codes and clone failures run
# against stub semgrep and git binaries on PATH, so the suite is hermetic.
#
# The negative assertions matter most: a repo that would not clone, and a scan that exited
# non-zero, must not reach `scans`. The counter at the bottom fails the run when fewer
# assertions execute than are written.
set -uo pipefail
PLUGIN_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
SCRIPT="$PLUGIN_ROOT/skills/semgrep/scripts/run-scans.sh"
readonly EXPECTED_ASSERTIONS=78
command -v jq >/dev/null 2>&1 || {
echo "run_scan_tests.sh: jq not found — required" >&2
exit 1
}
[ -x "$SCRIPT" ] || [ -r "$SCRIPT" ] || {
echo "run_scan_tests.sh: cannot read $SCRIPT" >&2
exit 1
}
PASS=0
FAIL=0
ok() {
if [ "$1" = "0" ]; then
PASS=$((PASS + 1))
else
FAIL=$((FAIL + 1))
echo " FAIL: $2" >&2
fi
}
# Asserts on a value rather than a status, so the failure message can show what was actually got.
eq() {
if [ "$1" = "$2" ]; then
PASS=$((PASS + 1))
else
FAIL=$((FAIL + 1))
echo " FAIL: $3 (expected '$2', got '$1')" >&2
fi
}
contains() {
case "$1" in
*"$2"*) PASS=$((PASS + 1)) ;;
*)
FAIL=$((FAIL + 1))
echo " FAIL: $3" >&2
;;
esac
}
lacks() {
case "$1" in
*"$2"*)
FAIL=$((FAIL + 1))
echo " FAIL: $3" >&2
;;
*) PASS=$((PASS + 1)) ;;
esac
}
WORK=$(mktemp -d "${TMPDIR:-/tmp}/run-scan-tests.XXXXXX")
trap 'rm -rf "$WORK"' EXIT
TARGET="$WORK/proj"
mkdir -p "$TARGET/src"
printf 'x = 1\n' >"$TARGET/src/a.py"
plan() { # plan <name> <json>
printf '%s\n' "$2" >"$WORK/$1.json"
printf '%s' "$WORK/$1.json"
}
dry() { # dry <plan-file> [extra args...] -> generated commands on stdout
local p=$1
shift
bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all \
--rulesets "$p" --dry-run "$@" 2>/dev/null
}
# Returns the die() message; used to assert a bad plan is rejected rather than degraded.
fails() { # fails <args...> -> prints stderr, returns 0 if the script exited non-zero
local out
if out=$("$@" 2>&1); then
printf '%s' "$out"
return 1
fi
printf '%s' "$out"
return 0
}
echo "→ argument validation"
BASIC=$(plan basic '{"baseline":["p/security-audit"],"python":["p/python"],"third_party":[]}')
msg=$(fails bash "$SCRIPT" --target relative/path --output-dir "$WORK/out" --mode run-all --rulesets "$BASIC" --dry-run)
ok $? "a relative --target must be rejected"
contains "$msg" "absolute path" "the message must name the absolute-path requirement"
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir rel --mode run-all --rulesets "$BASIC" --dry-run)
ok $? "a relative --output-dir must be rejected"
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode wrong --rulesets "$BASIC" --dry-run)
ok $? "an unknown --mode must be rejected"
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$TARGET" --mode run-all --rulesets "$BASIC" --dry-run)
ok $? "an output directory equal to the target must be rejected"
contains "$msg" "scan target" "the message must explain the run would scan its own output"
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$WORK/nope.json" --dry-run)
ok $? "a missing rulesets file must be rejected"
printf 'not json' >"$WORK/bad.json"
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$WORK/bad.json" --dry-run)
ok $? "a rulesets file that is not JSON must be rejected"
# A single ruleset written without the brackets would otherwise iterate the string.
SCALAR=$(plan scalar '{"docker":"p/dockerfile","third_party":[]}')
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$SCALAR" --dry-run)
ok $? "a scalar where an array is expected must be rejected"
contains "$msg" "must be an array" "the message must name the array requirement"
URLKEY=$(plan urlkey '{"python":["https://github.com/x/y"],"third_party":[]}')
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$URLKEY" --dry-run)
ok $? "a repository URL filed under a language key must be rejected"
contains "$msg" "third_party" "the message must point at third_party"
TRAVERSE=$(plan traverse '{"baseline":["p/python/../../.."],"third_party":[]}')
ok "$(
fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$TRAVERSE" --dry-run >/dev/null
echo $?
)" "a traversing ruleset id must be rejected"
ABS=$(plan abs '{"baseline":["/etc"],"third_party":[]}')
ok "$(
fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$ABS" --dry-run >/dev/null
echo $?
)" "an absolute path is not a registry identifier"
BADURL=$(plan badurl '{"baseline":[],"third_party":["http://insecure/x/y"]}')
ok "$(
fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$BADURL" --dry-run >/dev/null
echo $?
)" "a non-https third_party URL must be rejected"
EMPTY=$(plan empty '{"baseline":[],"third_party":[]}')
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$EMPTY" --dry-run)
ok $? "a plan with no rulesets at all must be rejected, not run as an empty scan"
contains "$msg" "nothing to run" "the message must say there is nothing to run"
RESERVED=$(plan reserved '{"all":["p/python"],"third_party":[]}')
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$RESERVED" --dry-run)
ok $? "the reserved language name 'all' must be rejected"
# shellcheck disable=SC2016 # the literal $(id) is the payload under test
INJECT=$(plan inject '{"py$(id)":["p/python"],"third_party":[]}')
msg=$(fails bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/out" --mode run-all --rulesets "$INJECT" --dry-run)
ok $? "a language key holding a command substitution must be rejected"
echo "→ command generation"
out=$(dry "$BASIC")
n=$(printf '%s\n' "$out" | grep -c 'semgrep')
eq "$n" "2" "one command per ruleset"
n=$(printf '%s\n' "$out" | grep -c -- '--metrics=off')
eq "$n" "2" "--metrics=off must be on every command"
lacks "$out" "--severity" "run-all mode must add no severity flags"
lacks "$out" "--pro" "--pro must be absent unless asked for"
# The cross-language ruleset scans the whole target: a filter would drop findings in the files
# it does not match.
baseline_cmd=$(printf '%s\n' "$out" | grep 'p/security-audit')
lacks "$baseline_cmd" "--include" "cross-language rulesets must never take --include"
py_cmd=$(printf '%s\n' "$out" | grep 'p/python')
contains "$py_cmd" '"--include=*.py"' "language rulesets must be scoped with --include"
contains "$py_cmd" '"--include=*.pyi"' "every glob for the language must be present"
contains "$baseline_cmd" 'raw/all-security-audit.json' "cross-language output stems start with all-"
out=$(dry "$BASIC" --mode important-only)
n=$(printf '%s\n' "$out" | grep -c -- '--severity WARNING --severity ERROR')
eq "$n" "2" "important-only must add the severity pre-filter to every command"
out=$(dry "$BASIC" --pro)
n=$(printf '%s\n' "$out" | grep -c -- '--pro')
eq "$n" "2" "--pro must reach every command when requested"
echo "→ output directory exclusion"
# The default output directory sits inside the target, where the cloned rule repos and the raw
# JSON of sibling scans would otherwise be scanned as if they were the user's code.
out=$(bash "$SCRIPT" --target "$TARGET" --output-dir "$TARGET/out" --mode run-all --rulesets "$BASIC" --dry-run 2>/dev/null)
n=$(printf '%s\n' "$out" | grep -c -- '--exclude=out')
eq "$n" "2" "--exclude must be on every command, including the cross-language ones"
out=$(dry "$BASIC")
lacks "$out" "--exclude" "no --exclude when the output directory is outside the target"
# The containment test is `case "$OUTPUT_ROOT/" in "$TARGET_ROOT"/*)`. The quotes are what make
# it a string compare: unquoted, a target holding [ ] * or ? is read as a glob and a path like
# proj[1] stops matching itself, so --exclude is never added and every scan then reads raw/ and
# the cloned rule repos, which carry literal example secrets.
GLOBDIR="$WORK/glob[1]/src"
mkdir -p "$GLOBDIR"
printf 'x = 1\n' >"$GLOBDIR/a.py"
out=$(bash "$SCRIPT" --target "$WORK/glob[1]" --output-dir "$WORK/glob[1]/out" \
--mode run-all --rulesets "$BASIC" --dry-run 2>/dev/null)
contains "$out" "--exclude=out" "a target holding glob metacharacters must still exclude its output directory"
echo "→ cross-language hoisting and dedup"
# p/secrets is in baseline and in the python list. Running it twice would put two findings
# counts in the report for one scan; the merged SARIF dedups the results but a sum does not.
SHARED=$(plan shared '{"baseline":["p/secrets"],"python":["p/python","p/secrets"],"third_party":[]}')
out=$(dry "$SHARED")
n=$(printf '%s\n' "$out" | grep -c 'p/secrets')
eq "$n" "1" "a ruleset already running unscoped must not be scanned again per language"
n=$(printf '%s\n' "$out" | grep -c 'p/python')
eq "$n" "1" "the language keeps the rulesets that are its own"
# js and javascript are the same language. Two units would carry identical --include sets, and
# a ruleset named under both would be scanned and counted twice.
ALIAS=$(plan alias '{"baseline":[],"js":["p/javascript"],"javascript":["p/nodejs"],"third_party":[]}')
out=$(dry "$ALIAS")
n=$(printf '%s\n' "$out" | grep -c 'raw/javascript-')
eq "$n" "2" "aliased keys must fold onto one language, one command per distinct ruleset"
DUPE=$(plan dupe '{"baseline":[],"js":["p/javascript"],"javascript":["p/javascript"],"third_party":[]}')
out=$(dry "$DUPE")
n=$(printf '%s\n' "$out" | grep -c 'p/javascript')
eq "$n" "1" "the same ruleset under two aliases of one language must be scanned once"
# Two spellings of one repository collapse to one clone directory, so the second clone would
# fail into a non-empty tree and both would then read the same result.
GITDUP=$(plan gitdup '{"baseline":[],"third_party":["https://github.com/trailofbits/semgrep-rules","https://github.com/trailofbits/semgrep-rules.git"]}')
out=$(dry "$GITDUP")
n=$(printf '%s\n' "$out" | grep -c 'trailofbits-semgrep-rules')
eq "$n" "1" "two spellings of one repository must clone and scan once"
# Different owners publishing a repo of the same name must stay distinct.
TWOORG=$(plan twoorg '{"baseline":[],"third_party":["https://github.com/trailofbits/semgrep-rules","https://github.com/elttam/semgrep-rules"]}')
out=$(dry "$TWOORG")
contains "$out" "trailofbits-semgrep-rules" "the clone directory must carry the owner"
contains "$out" "elttam-semgrep-rules" "a second org's identically named repo must not collide"
UNKNOWN=$(plan unknown '{"baseline":[],"cobol":["p/cobol"],"third_party":[]}')
out=$(dry "$UNKNOWN")
lacks "$out" "--include" "an unrecognised language must run unscoped rather than guess a glob"
echo "→ execution, exit codes and finding counts"
# Stub semgrep: writes the output files it was told to and exits with $STUB_RC. The real binary
# is not needed to prove how this script reads a result, and stubbing keeps the suite hermetic.
mkdir -p "$WORK/bin"
cat >"$WORK/bin/semgrep" <<'STUB'
#!/usr/bin/env bash
json=""; sarif=""; prev=""
for a in "$@"; do
case "$prev" in -o) json=$a ;; esac
case "$a" in --sarif-output=*) sarif=${a#--sarif-output=} ;; esac
prev=$a
done
if [ "${STUB_WRITE:-1}" = "1" ]; then
# STUB_PATHS unset means no .paths key at all, which is the "semgrep did not say" case the
# script must report as -1 rather than as a scan that opened no file.
if [ -n "${STUB_PATHS:-}" ]; then
printf '{"results":[%s],"paths":{"scanned":[%s]}}' "${STUB_RESULTS:-}" "$STUB_PATHS" > "$json"
else
printf '{"results":[%s]}' "${STUB_RESULTS:-}" > "$json"
fi
printf '{"runs":[]}' > "$sarif"
fi
[ -z "${STUB_STDERR:-}" ] || echo "$STUB_STDERR" >&2
exit "${STUB_RC:-0}"
STUB
chmod +x "$WORK/bin/semgrep"
run_real() { # run_real <plan> [env assignments already exported] -> scans.json path
local p=$1 outdir=$2
rm -rf "$outdir"
PATH="$WORK/bin:$PATH" bash "$SCRIPT" --target "$TARGET" --output-dir "$outdir" \
--mode run-all --rulesets "$p" --jobs 2 >/dev/null 2>&1
echo $?
}
ONE=$(plan one '{"baseline":[],"python":["p/python"],"third_party":[]}')
export STUB_RC=0 STUB_RESULTS='{"a":1},{"b":2}'
rc=$(run_real "$ONE" "$WORK/o1")
eq "$rc" "0" "a successful scan must exit 0"
eq "$(jq '.scans | length' "$WORK/o1/scans.json")" "1" "the successful scan must appear in scans"
eq "$(jq -r '.scans[0].findings' "$WORK/o1/scans.json")" "2" "findings must be counted from the JSON, not reported"
eq "$(jq '.failed | length' "$WORK/o1/scans.json")" "0" "a successful scan must not appear in failed"
# Exit 1 means "findings present" on older semgrep, so it is a successful scan.
export STUB_RC=1 STUB_RESULTS='{"a":1}'
rc=$(run_real "$ONE" "$WORK/o2")
eq "$(jq '.scans | length' "$WORK/o2/scans.json")" "1" "exit 1 must count as a successful scan"
# Exit 7 is a config that would not load: no scan happened.
export STUB_RC=7 STUB_RESULTS=''
rc=$(run_real "$ONE" "$WORK/o3")
eq "$rc" "1" "a run where every scan failed must exit non-zero"
eq "$(jq '.scans | length' "$WORK/o3/scans.json")" "0" "a scan that exited 7 must not reach scans"
eq "$(jq '.failed | length' "$WORK/o3/scans.json")" "1" "a scan that exited 7 must be reported as failed"
contains "$(jq -r '.failed[0].json' "$WORK/o3/scans.json")" "raw/python-python.json" \
"a failed entry must carry the paths it may have partly written"
# Exit 0 with no output file is the case the exit code alone cannot catch.
export STUB_RC=0 STUB_WRITE=0
rc=$(run_real "$ONE" "$WORK/o4")
eq "$(jq '.failed | length' "$WORK/o4/scans.json")" "1" "exit 0 with no output file must be a failure, not a clean scan"
unset STUB_WRITE
echo "→ reused output directory"
# merge_sarif.py globs every *.sarif in raw/ and cannot tell one run's output from another's, so
# a ruleset dropped between two runs into the same output directory would still reach
# results.sarif and its total, under a ruleset the current scans.json never mentions.
export STUB_RC=0 STUB_RESULTS='{"a":1}'
rerun() {
PATH="$WORK/bin:$PATH" bash "$SCRIPT" --target "$TARGET" --output-dir "$WORK/o5" \
--mode run-all --rulesets "$ONE" --jobs 1 >/dev/null 2>&1
}
rm -rf "$WORK/o5"
rerun
printf '{"runs":[]}' >"$WORK/o5/raw/docker-dockerfile.sarif"
printf '{"results":[]}' >"$WORK/o5/raw/docker-dockerfile.json"
rerun
if [ -e "$WORK/o5/raw/docker-dockerfile.sarif" ]; then stale=present; else stale=gone; fi
eq "$stale" "gone" "a previous run's SARIF must not survive into a rerun's raw/"
eq "$(find "$WORK/o5/raw" -type f | wc -l | tr -d ' ')" "2" "raw/ must hold only the current run's output"
eq "$(jq '.scans | length' "$WORK/o5/scans.json")" "1" "clearing raw/ must not break the rerun itself"
echo "→ clone failures"
export STUB_RC=0 STUB_RESULTS=''
mkdir -p "$WORK/gitbin"
cat >"$WORK/gitbin/git" <<'GITSTUB'
#!/usr/bin/env bash
# clone <flags> <url> <dest>
dest=${*: -1}
case "${GIT_STUB_MODE:-fail}" in
fail) echo "fatal: repository not found" >&2; exit 128 ;;
empty) mkdir -p "$dest"; exit 0 ;;
# Enough rule files that `find` output passes the 64 KiB pipe buffer. A single rules.yaml
# fits well inside it, so the old `find | head -1` pipeline never got SIGPIPE and this case
# passed against a repo shape no real rule repository has.
ok)
mkdir -p "$dest"
i=1
while [ "$i" -le 2000 ]; do
printf 'rules: []\n' >"$dest/rule-with-a-fairly-long-filename-to-fill-the-buffer-$i.yaml"
i=$((i + 1))
done
exit 0
;;
esac
GITSTUB
chmod +x "$WORK/gitbin/git"
REPO=$(plan repo '{"baseline":["p/security-audit"],"third_party":["https://github.com/x/rules"]}')
run_clone() {
rm -rf "$2"
PATH="$WORK/gitbin:$WORK/bin:$PATH" GIT_STUB_MODE="$1" bash "$SCRIPT" --target "$TARGET" \
--output-dir "$2" --mode run-all --rulesets "$REPO" --jobs 1 >/dev/null 2>&1
echo $?
}
run_clone fail "$WORK/c1" >/dev/null
eq "$(jq '.skipped | length' "$WORK/c1/scans.json")" "1" "a repo that will not clone must be reported as skipped"
eq "$(jq -r '.skipped[0].ruleset' "$WORK/c1/scans.json")" "https://github.com/x/rules" \
"the skipped entry must name the repository the user approved"
eq "$(jq '[.scans[].ruleset | select(test("github"))] | length' "$WORK/c1/scans.json")" "0" \
"a repo that failed to clone must not appear as a scanned ruleset"
# A clone that succeeds but carries no rules scans nothing; reporting it as fine would show a
# completed scan against a ruleset that never ran.
run_clone empty "$WORK/c2" >/dev/null
eq "$(jq '.skipped | length' "$WORK/c2/scans.json")" "1" "a clone with no rule files must be skipped"
run_clone ok "$WORK/c3" >/dev/null
eq "$(jq '.skipped | length' "$WORK/c3/scans.json")" "0" "a healthy clone must not be skipped"
eq "$(jq '.scans | length' "$WORK/c3/scans.json")" "2" "a healthy clone must be scanned alongside the baseline"
# ------------------------------------------------------------------------- coverage reporting
# A ruleset whose --include globs match nothing exits 0 with an empty result, which reads in the
# report exactly like a ruleset that ran and found nothing. That is how a plan naming the wrong
# languages — p/python on a Go tree — comes back as a clean audit.
echo "→ rulesets that covered nothing"
unset STUB_PATHS
export STUB_RC=0 STUB_RESULTS=''
# semgrep opened files: a real scan that happened to find nothing.
export STUB_PATHS='"a.py","b.py"'
run_real "$ONE" "$WORK/cn1" >/dev/null
eq "$(jq -r '.scans[0].filesScanned' "$WORK/cn1/scans.json")" "2" "filesScanned must come from .paths.scanned"
eq "$(jq '.coveredNothing | length' "$WORK/cn1/scans.json")" "0" \
"a scan that opened files must not be reported as covering nothing"
# semgrep opened nothing: the ruleset does not apply to this tree at all.
export STUB_PATHS=' '
run_real "$ONE" "$WORK/cn2" >/dev/null
eq "$(jq -r '.scans[0].filesScanned' "$WORK/cn2/scans.json")" "0" "an empty scanned list must be 0, not unknown"
eq "$(jq -r '.coveredNothing | join(",")' "$WORK/cn2/scans.json")" "python/p/python" \
"a ruleset that opened no file must be named in coveredNothing"
eq "$(jq '.scans | length' "$WORK/cn2/scans.json")" "1" \
"it still ran, so it stays in scans rather than moving to failed"
# No .paths key at all. Unknown is not the same answer as zero, and reporting every scan as
# covering nothing would make the field worthless on any semgrep that does not emit it.
unset STUB_PATHS
run_real "$ONE" "$WORK/cn3" >/dev/null
eq "$(jq -r '.scans[0].filesScanned' "$WORK/cn3/scans.json")" "-1" "a missing .paths must report -1, not 0"
eq "$(jq '.coveredNothing | length' "$WORK/cn3/scans.json")" "0" \
"an unknown file count must not be reported as covering nothing"
# --------------------------------------------------------------- the recorded exclude pattern
# semgrep matches --exclude anywhere in the tree, so an output directory named `out` inside the
# target also drops the target's own src/out/. Announced only on stderr that is invisible to the
# report and the missing files read as clean coverage.
echo "→ the recorded exclude pattern"
unset STUB_PATHS
export STUB_RC=0 STUB_RESULTS=''
rm -rf "$TARGET/out"
PATH="$WORK/bin:$PATH" bash "$SCRIPT" --target "$TARGET" --output-dir "$TARGET/out" \
--mode run-all --rulesets "$BASIC" --jobs 2 >/dev/null 2>&1
# Asserted before the value is read: jq on a missing file prints nothing, which compares equal
# to the empty string the second case expects, so both would pass having checked nothing.
ok "$([ -f "$TARGET/out/scans.json" ] && echo 0 || echo 1)" "the excluded run must write scans.json"
eq "$(jq -r '.excludePattern' "$TARGET/out/scans.json" 2>/dev/null || echo NOFILE)" "out" \
"scans.json must record the pattern excluded from every scan"
rm -rf "$TARGET/out"
run_real "$BASIC" "$WORK/ex2" >/dev/null
ok "$([ -f "$WORK/ex2/scans.json" ] && echo 0 || echo 1)" "the unexcluded run must write scans.json"
eq "$(jq -r '.excludePattern' "$WORK/ex2/scans.json" 2>/dev/null || echo NOFILE)" "" \
"excludePattern must be empty when nothing was excluded"
# ------------------------------------------------------------------ the documented post-filter
# The important-only loop lives in scan-modes.md rather than in a script, so it is extracted and
# run here. Redirection creates the output before jq does, and merge_sarif.py --important treats
# a zero-byte filter file as corrupt and refuses the whole merge — one unfilterable scan would
# take every other scan's findings with it.
echo "→ documented post-filter loop"
MODES="$PLUGIN_ROOT/skills/semgrep/references/scan-modes.md"
FILTER_LOOP=$(awk '
/^### Filter All Result Files in a Directory$/ { found = 1 }
found && /^```bash$/ { inblock = 1; next }
inblock && /^```$/ { exit }
inblock { print }
' "$MODES")
# Extraction is itself a checker: a renamed heading or fence would otherwise run an empty
# string through bash and pass every assertion below.
contains "$FILTER_LOOP" "jq" "the post-filter loop must be extractable from scan-modes.md"
contains "$FILTER_LOOP" "important.json" "the extracted block must be the important-only filter"
FRAW="$WORK/filter/raw"
mkdir -p "$FRAW"
# One finding passes the filter, one is excluded by confidence.
cat >"$FRAW/python-good.json" <<'JSON'
{"results":[
{"check_id":"r.keep","path":"a.py","start":{"line":5},
"extra":{"metadata":{"category":"security","confidence":"HIGH","impact":"HIGH"}}},
{"check_id":"r.drop","path":"a.py","start":{"line":9},
"extra":{"metadata":{"category":"security","confidence":"LOW","impact":"HIGH"}}}
],"errors":[],"paths":{}}
JSON
# Whatever the cause (a truncated write, a disk that filled), jq cannot read this one.
printf '{"results":[{"check_id":"r.x"' >"$FRAW/python-broken.json"
FILTER_ERR=$(OUTPUT_DIR="$WORK/filter" bash -c "$FILTER_LOOP" 2>&1 >/dev/null)
eq "$(jq '.results | length' "$FRAW/python-good-important.json" 2>/dev/null)" "1" \
"a readable scan must still be filtered when a sibling fails"
eq "$(find "$FRAW" -name '*-important.json' -size 0 | wc -l | tr -d ' ')" "0" \
"a failed filter must not leave a zero-byte -important.json for the merge to choke on"
ok "$([ -e "$FRAW/python-broken-important.json" ] && echo 1 || echo 0)" \
"the unfilterable scan must have no -important.json at all"
contains "$FILTER_ERR" "python-broken.json" "the failure must name the file it happened on"
# The merge is the gate that must still refuse: the scan is unfiltered, so its findings cannot
# be included or excluded honestly. It now fails naming the missing file rather than reporting
# a corrupt one.
MERGE="$PLUGIN_ROOT/skills/semgrep/scripts/merge_sarif.py"
printf '{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"semgrep","rules":[]}},"results":[]}]}\n' \
>"$FRAW/python-good.sarif"
cp "$FRAW/python-good.sarif" "$FRAW/python-broken.sarif"
MERGE_ERR=$(python3 "$MERGE" "$FRAW" "$WORK/filter/results.sarif" --important 2>&1 >/dev/null)
ok "$([ -e "$WORK/filter/results.sarif" ] && echo 1 || echo 0)" \
"the merge must write nothing while a scan is unfiltered"
contains "$MERGE_ERR" "python-broken-important.json" \
"the merge error must name the scan that has no filtered JSON"
TOTAL=$((PASS + FAIL))
echo
echo "$PASS passed, $FAIL failed, $TOTAL run"
if [ "$FAIL" -ne 0 ]; then
exit 1
fi
# A suite that stopped early would otherwise exit 0 having proved nothing.
if [ "$TOTAL" -ne "$EXPECTED_ASSERTIONS" ]; then
echo "ran $TOTAL assertions, expected $EXPECTED_ASSERTIONS, so the suite did not run in full" >&2
exit 1
fi
echo "$TOTAL assertions passed"
plugins/static-analysis/tests/run_workflow_tests.sh
Executable
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
# Wrapper so CI's run_*.sh discovery picks up the workflow harness; without it a guard removed
# from semgrep-scan.js reaches main with CI green. --self-test is a separate invocation because
# it mutates the workflow and requires every mutation to turn a scenario red.
set -euo pipefail
PLUGIN_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
# Fail, do not skip: a missing interpreter must not read as a clean run.
if ! command -v node >/dev/null 2>&1; then
echo "run_workflow_tests.sh: node not found — required to run this suite" >&2
exit 1
fi
node "$PLUGIN_ROOT/tests/workflow-harness.js" "$PLUGIN_ROOT/workflows/semgrep-scan.js"
node "$PLUGIN_ROOT/tests/workflow-harness.js" "$PLUGIN_ROOT/workflows/semgrep-scan.js" --self-test
plugins/static-analysis/tests/workflow-harness.js
+386
View File
@@ -0,0 +1,386 @@
#!/usr/bin/env node
// Exercises workflows/semgrep-scan.js with every agent stubbed. The runtime injects globals
// and wraps the body in an async function; that is reproduced here by stripping the `export`
// and wrapping, so each phase guard is testable offline.
//
// node workflow-harness.js <path-to-semgrep-scan.js> [--self-test]
//
// --self-test mutates the workflow and requires each mutation to turn a scenario red, so a
// harness that stopped checking anything cannot still report success.
"use strict";
const fs = require("fs");
const workflowPath = process.argv[2];
const selfTest = process.argv.includes("--self-test");
if (!workflowPath) {
console.error("usage: node workflow-harness.js <path-to-semgrep-scan.js> [--self-test]");
process.exit(2);
}
const SOURCE = fs.readFileSync(workflowPath, "utf8");
function compile(src) {
const body = src.replace(/^export const meta/m, "const meta");
return new Function(
"agent",
"parallel",
"phase",
"log",
"args",
`return (async () => {\n${body}\n})()`,
);
}
// An installed-plugin path on purpose. Every later phase must splice in the value the detect
// phase resolved, and a repo-relative constant would still look right against a repo checkout.
const SKILL_DIR = "/home/u/.claude/plugins/cache/trailofbits/static-analysis/1.3.0/skills/semgrep";
const DETECT = {
target: "/proj",
outputDir: "/proj/static_analysis_semgrep_1",
skillDir: SKILL_DIR,
pro: false,
proReason: "",
languages: [{ name: "python", files: 12 }],
frameworks: ["django"],
};
const SELECT = { rulesetsPath: "/proj/static_analysis_semgrep_1/rulesets.json", counts: { baseline: 2, language: 2, thirdParty: 1 } };
const SCAN = { ok: true, scansJson: "/proj/static_analysis_semgrep_1/scans.json", succeeded: 4, failed: 0, skipped: 0, error: "" };
const REPORT = { ok: true, resultsSarif: "/proj/static_analysis_semgrep_1/results/results.sarif", total: 7, report: "# Semgrep Scan Complete", error: "" };
// Each phase's reply is overridable; `null` stands for an agent that returned nothing.
async function run(src, { args, detect = DETECT, select = SELECT, scan = SCAN, report = REPORT } = {}) {
const logs = [];
const prompts = {};
const replies = { detect, select, scan, report };
const agent = async (prompt, opts = {}) => {
const label = opts.label || "?";
prompts[label] = prompt;
if (!(label in replies)) throw new Error(`unexpected agent label: ${label}`);
return replies[label];
};
const parallel = async (thunks) =>
Promise.all(thunks.map((t) => Promise.resolve().then(t).catch(() => null)));
const out = await compile(src)(agent, parallel, () => {}, (m) => logs.push(m), args);
return { out, logs, prompts };
}
async function throws(src, opts) {
try {
await run(src, opts);
return null;
} catch (e) {
return e.message;
}
}
let PASS = 0;
const FAILURES = [];
const ok = (cond, msg) => {
if (cond) PASS++;
else FAILURES.push(msg);
};
// ---------------------------------------------------------------- scenarios
// Each returns a list of [condition, message]. --self-test re-runs them against mutated
// sources and requires that at least one goes red, which is what proves they bite.
const SCENARIOS = {
"happy path returns the assembled result": async (src) => {
const { out, prompts } = await run(src, { args: { target: "/proj" } });
return [
[out && out.total === 7, "the merged finding total must be returned"],
[out && out.outputDir === DETECT.outputDir, "the resolved output directory must be returned"],
[out && out.succeeded === 4, "the scan counts must be carried through"],
[Object.keys(prompts).length === 4, "all four phases must run"],
];
},
"args parse from a prose string": async (src) => {
const { out } = await run(src, { args: "target: /proj; mode: important-only" });
return [[out && out.mode === "important-only", "a prose args string must still set the mode"]];
},
// A bare path carries no `key:` for the prose parser to find, and it is the natural way to
// invoke this. Parsing it to {} left the target empty and the run scanned cwd, reporting a
// full result set for a tree the caller never named.
"a bare path is taken as the target rather than falling back to cwd": async (src) => {
const plain = await run(src, { args: "/some/other/proj" });
const spaced = await run(src, { args: "/Users/me/My Project" });
const relative = await run(src, { args: "~/code/app" });
return [
[/Target: \/some\/other\/proj/.test(plain.prompts.detect), "a bare path must become the target"],
[!/current working directory/.test(plain.prompts.detect), "cwd must not be the target when a path was given"],
[/Target: \/Users\/me\/My Project/.test(spaced.prompts.detect), "a path containing a space must survive"],
[/Target: ~\/code\/app/.test(relative.prompts.detect), "a ~ path must become the target"],
];
},
"an args string that names no path stops the run rather than scanning cwd": async (src) => {
const prose = await throws(src, { args: "please scan the repo for bugs" });
const brokenJson = await throws(src, { args: '{"target": "/proj"' });
return [
[prose && /could not parse args/.test(prose), "prose that names no path must throw"],
[brokenJson && /could not parse args/.test(brokenJson), "malformed JSON must throw, not become a target"],
];
},
"an unknown mode is rejected before any agent runs": async (src) => {
const msg = await throws(src, { args: { mode: "everything" } });
return [[msg && /mode must be one of/.test(msg), "an unknown mode must throw"]];
},
"a non-integer jobs is rejected": async (src) => {
const msg = await throws(src, { args: { jobs: "lots" } });
return [[msg && /jobs must be a positive integer/.test(msg), "a non-integer --jobs must throw"]];
},
// The guards below all protect the same thing: a later phase acting on a path an earlier
// phase did not actually resolve, which silently scans or deletes the wrong tree.
"a relative target from detect is rejected": async (src) => {
const msg = await throws(src, { detect: { ...DETECT, target: "proj" } });
return [[msg && /non-absolute target/.test(msg), "a relative target must throw rather than reach the scan"]];
},
"an output directory equal to the target is rejected": async (src) => {
const msg = await throws(src, { detect: { ...DETECT, outputDir: "/proj" } });
return [[msg && /scan target/.test(msg), "an output directory equal to the target must throw"]];
},
"a dead detect phase stops the run": async (src) => {
const msg = await throws(src, { detect: null });
return [[msg && /no scan ran/.test(msg), "a detect agent that returned nothing must stop the run"]];
},
"a select phase with no ruleset file stops the run": async (src) => {
const msg = await throws(src, { select: { rulesetsPath: "", counts: {} } });
return [[msg && /no scan ran/.test(msg), "an empty ruleset path must stop the run"]];
},
// The most important negative: a failed scan must not reach the report as an empty result.
"a failed scan stops the run rather than reporting zero findings": async (src) => {
const msg = await throws(src, { scan: { ...SCAN, ok: false, succeeded: 0, error: "semgrep exited 7" } });
return [
[msg && /no scan succeeded/.test(msg), "a failed scan must throw"],
[msg && /semgrep exited 7/.test(msg), "the script's own error must be carried into the message"],
];
},
"a dead report phase names where the raw output is": async (src) => {
const msg = await throws(src, { report: null });
return [[msg && /raw/.test(msg), "a dead merge phase must still point at the raw scan output"]];
},
// The mirror of the failed-scan guard. Without a failure channel in REPORT_SCHEMA the agent
// has to fill in a total, and total 0 with an unwritten results.sarif path is what a scan
// that genuinely found nothing looks like.
"a failed merge stops the run rather than returning zero findings": async (src) => {
const msg = await throws(src, {
report: { ok: false, resultsSarif: "", total: -1, report: "", error: "merge_sarif.py exited 1" },
});
return [
[msg && /merge failed/.test(msg), "a failed merge must throw"],
[msg && /merge_sarif\.py exited 1/.test(msg), "the merge's own error must reach the message"],
[msg && /raw/.test(msg), "the message must still point at the raw scan output"],
];
},
// Dropping a dead scan's output is what keeps one crashed scan from denying every healthy
// scan a merged result, so the flag has to actually reach the command.
"the merge is told which scans failed": async (src) => {
const { prompts } = await run(src, {});
const merge = prompts.report.split("\n").find((l) => l.includes("merge_sarif.py")) || "";
return [
[/--scans/.test(prompts.report), "the merge must be passed the scans.json it should read"],
[prompts.report.includes(`--scans "${SCAN.scansJson}"`), "the path must be the one the scan phase reported"],
[/merge_sarif\.py/.test(merge), "the merge command must still be one pasteable line"],
];
},
"pro reaches the scan command only when detected": async (src) => {
const off = await run(src, {});
const on = await run(src, { detect: { ...DETECT, pro: true } });
return [
[!/--pro/.test(off.prompts.scan), "--pro must be absent when Pro was not available"],
[/--pro/.test(on.prompts.scan), "--pro must be passed when Pro was detected"],
];
},
"jobs reaches the scan command only when given": async (src) => {
const without = await run(src, {});
const with_ = await run(src, { args: { jobs: 8 } });
return [
[!/--jobs/.test(without.prompts.scan), "--jobs must be absent unless asked for"],
[/--jobs 8/.test(with_.prompts.scan), "--jobs must reach the command when given"],
];
},
"the scan phase calls the script and nothing else": async (src) => {
const { prompts } = await run(src, {});
return [
[/run-scans\.sh/.test(prompts.scan), "the scan phase must invoke run-scans.sh"],
[/--rulesets "\/proj\/static_analysis_semgrep_1\/rulesets\.json"/.test(prompts.scan), "the ruleset file from the select phase must be passed through"],
[/Do not add rulesets/.test(prompts.scan), "the agent must be told not to compose its own commands"],
];
},
// The skill directory is wherever the plugin was installed. A constant only resolves inside a
// checkout of this repo, so for a marketplace install every scripted command would run against
// a path that does not exist.
"the detect phase resolves the skill directory rather than assuming one": async (src) => {
const { prompts } = await run(src, {});
return [
[/CLAUDE_PLUGIN_ROOT/.test(prompts.detect), "the search must try $CLAUDE_PLUGIN_ROOT first"],
[/plugins\/cache/.test(prompts.detect), "the search must cover a marketplace install"],
[/run-scans\.sh/.test(prompts.detect), "the search must anchor on the script, so a stale install cannot match"],
];
},
"the resolved skill directory is what later phases use": async (src) => {
const { prompts } = await run(src, {});
const repoRelative = /(^|[^/\w])plugins\/static-analysis\/skills\/semgrep/;
return [
[prompts.scan.includes(`${SKILL_DIR}/scripts/run-scans.sh`), "the scan must run the resolved script path"],
[prompts.select.includes(`${SKILL_DIR}/references/rulesets.md`), "the catalogue must be read from the resolved path"],
[prompts.report.includes(`${SKILL_DIR}/scripts/merge_sarif.py`), "the merge must use the resolved script path"],
[!repoRelative.test(prompts.scan), "no repo-relative path may survive into the scan prompt"],
[!repoRelative.test(prompts.select), "no repo-relative path may survive into the select prompt"],
[!repoRelative.test(prompts.report), "no repo-relative path may survive into the report prompt"],
];
},
"an unresolved skill directory stops the run": async (src) => {
const empty = await throws(src, { detect: { ...DETECT, skillDir: "" } });
const relative = await throws(src, { detect: { ...DETECT, skillDir: "plugins/static-analysis/skills/semgrep" } });
return [
[empty && /skill directory/.test(empty), "a skill directory that could not be found must throw"],
[empty && /no scan ran/.test(empty), "the message must say no scan ran"],
[relative && /skill directory/.test(relative), "a relative skill directory must throw rather than reach the scan"],
];
},
"an explicit skill argument replaces the search": async (src) => {
const { prompts } = await run(src, { args: { skill: "/opt/sa/skills/semgrep" } });
return [
[/\/opt\/sa\/skills\/semgrep\/scripts\/run-scans\.sh/.test(prompts.detect), "the given directory must be confirmed, not searched for"],
[!/CLAUDE_PLUGIN_ROOT/.test(prompts.detect), "the four-step search must be skipped when the path was given"],
];
},
"the select phase is pointed at the shared catalogue": async (src) => {
const { prompts } = await run(src, {});
return [
[/references\/rulesets\.md/.test(prompts.select), "ruleset selection must read rulesets.md rather than memory"],
[/third_party/.test(prompts.select), "the required third-party rules must be named"],
];
},
// The JSON post-filter reads .results[].extra.metadata, which SARIF does not have, so the
// merged SARIF cannot be filtered by re-running it — results.sarif is the primary deliverable
// and would keep everything the mode exists to exclude while the JSON side was filtered.
"important-only filters the merged SARIF through the merge script": async (src) => {
const { prompts } = await run(src, { args: { mode: "important-only" } });
const merge = prompts.report.split("\n").find((l) => l.includes("merge_sarif.py")) || "";
return [
[/--mode important-only/.test(prompts.scan), "the mode must reach the scan command"],
[/scan-modes\.md/.test(prompts.report), "important-only must post-filter before the merge"],
[/--important/.test(merge), "the merge command itself must carry --important"],
[/Cannot iterate over null/.test(prompts.report), "the report must say why jq cannot filter the SARIF"],
];
},
"run-all does not filter the merge": async (src) => {
const { prompts } = await run(src, {});
return [
[!/--important/.test(prompts.report), "--important must be absent in run-all mode"],
[!/scan-modes\.md/.test(prompts.report), "run-all must not post-filter"],
];
},
"the report is told to count from the merged SARIF": async (src) => {
const { prompts } = await run(src, {});
return [
[/Never sum the per-scan findings counts/.test(prompts.report), "the report must not sum per-scan counts"],
[/Did Not Run/.test(prompts.report), "failed and skipped rulesets must get their own section"],
[/rm -rf/.test(prompts.report), "the cloned rule repos must be deleted after the merge"],
// A ruleset that opened no file reports 0 findings like any other. Left out of the
// report, a plan aimed at the wrong languages is indistinguishable from a clean scan.
[/coveredNothing/.test(prompts.report), "rulesets that matched no file must be reported"],
// The merge drops an unparseable SARIF and exits 0. That scan is a success in scans.json,
// so unless the report carries the line, the shortfall has nothing pointing at it.
[/unparseable/.test(prompts.report), "SARIF files missing from the merge must be reported"],
[/excludePattern/.test(prompts.report), "the pattern excluded from every scan must be reported"],
];
},
};
// Mutations the scenarios must notice. Each removes one guard or one flag.
const MUTATIONS = [
["drop the absolute-target guard", (s) => s.replace(/if \(!target \|\| !target\.startsWith\('\/'\)\).*\n/, "")],
["drop the output-dir-equals-target guard", (s) => s.replace(/if \(outputDir\.replace[\s\S]*?\n\}\n/, "")],
["treat a failed scan as success", (s) => s.replace("if (!scanned || !scanned.ok) {", "if (false) {")],
["always pass --pro", (s) => s.replace("${detected.pro ? ' \\\\\\n --pro' : ''}", "' \\\\\\n --pro'")],
["drop the mode validation", (s) => s.replace(/if \(!MODES\.has\(mode\)\) \{[\s\S]*?\n\}\n/, "")],
["drop the dead-detect guard", (s) => s.replace("if (!detected) throw new Error('the detect phase returned nothing; no scan ran')", "if (!detected) return {}")],
["hardcode the skill directory back to a repo-relative path", (s) => s.replace(/^const SKILL_DIR = .*$/m, "const SKILL_DIR = 'plugins/static-analysis/skills/semgrep'")],
["accept an unresolved skill directory", (s) => s.replace("if (!SKILL_DIR || !SKILL_DIR.startsWith('/')) {", "if (false) {")],
["drop --important from the important-only merge", (s) => s.replace("mode === 'important-only' ? ' --important' : ''", "''")],
["ignore a bare path and scan cwd", (s) => s.replace("if (bare) return { target: text }", "if (false) return { target: text }")],
["accept unparseable args instead of refusing", (s) => s.replace(" throw new Error(`could not parse args", " return {}\n throw new Error(`could not parse args")],
["treat a failed merge as success", (s) => s.replace("if (!reported.ok) {", "if (false) {")],
["drop --scans from the merge", (s) => s.replace(/\n\s*`\s*--scans "\$\{scanned\.scansJson[\s\S]*?`,/, "")],
["stop reporting rulesets that covered nothing", (s) => s.replace(/\n\s*'\s*\.coveredNothing gets its own section[\s\S]*?clean audit\.',/, "")],
["stop reporting unparseable SARIF", (s) => s.replace(/\n\s*'6\. Read the merge command[\s\S]*?not in the merge\.',/, "")],
["stop reporting the exclude pattern", (s) => s.replace(/\n\s*'7\. If \.excludePattern[\s\S]*?clean coverage\.',/, "")],
];
(async () => {
for (const [name, fn] of Object.entries(SCENARIOS)) {
let results;
try {
results = await fn(SOURCE);
} catch (e) {
FAILURES.push(`${name}: threw unexpectedly: ${e.message}`);
continue;
}
for (const [cond, msg] of results) ok(cond, `${name}: ${msg}`);
}
if (selfTest) {
let bitten = 0;
for (const [label, mutate] of MUTATIONS) {
const mutated = mutate(SOURCE);
if (mutated === SOURCE) {
FAILURES.push(`self-test: mutation "${label}" changed nothing — it no longer matches the source`);
continue;
}
let red = false;
for (const fn of Object.values(SCENARIOS)) {
try {
const results = await fn(mutated);
if (results.some(([cond]) => !cond)) {
red = true;
break;
}
} catch {
red = true;
break;
}
}
if (red) bitten++;
else FAILURES.push(`self-test: no scenario caught the mutation "${label}"`);
}
ok(bitten === MUTATIONS.length, `self-test: ${bitten}/${MUTATIONS.length} mutations caught`);
}
if (FAILURES.length) {
for (const f of FAILURES) console.error(` FAIL: ${f}`);
console.error(`${FAILURES.length} failed, ${PASS} passed`);
process.exit(1);
}
if (PASS === 0) {
console.error("no assertions ran — discovery is broken");
process.exit(1);
}
console.log(`${PASS} assertions passed`);
})();
plugins/static-analysis/workflows/semgrep-scan.js
+475
View File
@@ -0,0 +1,475 @@
// Ships as /static-analysis:semgrep-scan. Plugin workflows are namespaced by the plugin's
// `name` field, which cannot be overridden per component, so the prefix is always
// `static-analysis:`. meta.name below supplies the rest, not the filename.
export const meta = {
name: 'semgrep-scan',
description:
'Scan a codebase with Semgrep: detect languages, select rulesets, run every approved ruleset in parallel, merge to SARIF and report',
whenToUse:
'When the user wants a Semgrep scan run end to end without being asked to approve the ruleset list. Pass args as a JSON OBJECT, not a prose string: {"target": "/abs/path", "mode": "run-all", "out": "/abs/path"}. target defaults to cwd; mode is run-all or important-only; out defaults to an auto-incremented static_analysis_semgrep_N beside the target. The gated path, where the user reviews and edits the rulesets before anything runs, is the semgrep SKILL.md itself — use that when the ruleset selection matters.',
phases: [
{ title: 'Detect', detail: 'resolve the output directory, check semgrep and Pro, glob languages and framework markers' },
{ title: 'Select', detail: 'choose rulesets for the detected stack from references/rulesets.md' },
{ title: 'Scan', detail: 'run scripts/run-scans.sh over the selected rulesets' },
{ title: 'Report', detail: 'post-filter, merge to SARIF, summarize, remove the cloned rule repos' },
],
}
// args = { target, out, mode, jobs, skill }, all optional. Prose is parsed too (`target: /x;
// mode: run-all`), since a caller passing a string would otherwise kill the run on the first line.
const ARGS_HELP =
'Pass a JSON object like {"target": "/abs/path", "mode": "run-all"}, a bare path, or ' +
'`target: /abs/path; mode: run-all`.'
const parseArgs = (raw) => {
if (!raw) return {}
if (typeof raw === 'object') return raw
if (typeof raw !== 'string') return {}
const text = raw.trim()
if (!text) return {}
if (text.startsWith('{')) {
try {
return JSON.parse(text)
} catch {
// Fall through to key: value parsing rather than dying on a malformed brace.
}
}
const KEYS = ['target', 'out', 'mode', 'jobs', 'skill']
const out = {}
let key = null
for (const part of text.split(/;\s*|\n/)) {
const m = part.match(/^\s*(\w+)\s*:\s*([\s\S]*)$/)
if (m && KEYS.includes(m[1].toLowerCase())) {
key = m[1].toLowerCase()
out[key] = m[2].trim()
} else if (key && part.trim()) {
out[key] = `${out[key]} ${part.trim()}`.trim()
}
}
if (Object.keys(out).length) return out
// Nothing matched a key. `/static-analysis:semgrep-scan ~/proj` is the other shape a caller
// reaches for, and the regex above cannot see it: a path opens with / ~ or . rather than a
// word character and a colon. That fell through as {}, leaving targetHint empty, and the run
// scanned cwd instead — a complete, clean-looking report over a tree nobody asked about,
// which is worse than not running. A leading { is excluded so malformed JSON lands on the
// throw below rather than becoming a target named after its own opening brace.
const bare = !text.startsWith('{') && (/^[/~.]/.test(text) || !/\s/.test(text))
if (bare) return { target: text }
// Not a path and not a key: pair. Defaulting to cwd here is the same silent wrong-target
// scan, so this is the one place the parser refuses rather than guesses.
throw new Error(`could not parse args: ${JSON.stringify(raw)}. ${ARGS_HELP}`)
}
const input = parseArgs(args)
const MODES = new Set(['run-all', 'important-only'])
const mode = input.mode || 'run-all'
if (!MODES.has(mode)) {
throw new Error(`mode must be one of ${[...MODES].join(', ')}, got ${JSON.stringify(input.mode)}`)
}
// A string either way, so "4" and 4 behave the same; the script validates it too.
const jobs = input.jobs === undefined || input.jobs === null ? '' : String(input.jobs).trim()
if (jobs && !/^[1-9][0-9]*$/.test(jobs)) {
throw new Error(`jobs must be a positive integer, got ${JSON.stringify(input.jobs)}`)
}
// No approval gate: the scan is read-only over the target and every write lands inside the
// output directory. Invoking this with a target is the opt-in; SKILL.md's five steps are the
// gated path for when the ruleset list matters.
const targetHint = (input.target || '').trim()
const outHint = (input.out || '').trim()
const skillHint = (input.skill || '').trim()
const DETECT_SCHEMA = {
type: 'object',
required: ['target', 'outputDir', 'skillDir', 'pro', 'languages'],
additionalProperties: false,
properties: {
target: { type: 'string', description: 'absolute path that was scanned for languages' },
outputDir: { type: 'string', description: 'absolute path of the created output directory' },
skillDir: {
type: 'string',
description: 'absolute path of the semgrep skill directory (the parent of scripts/), or "" if it could not be found',
},
pro: { type: 'boolean', description: 'true only when `semgrep --pro --validate` succeeded' },
proReason: { type: 'string', description: 'when pro is false, the last lines of stderr explaining why; else ""' },
languages: {
type: 'array',
description: 'one entry per detected language category, with the file count that justified it',
items: {
type: 'object',
required: ['name', 'files'],
additionalProperties: false,
properties: {
name: { type: 'string', description: 'lowercase category: python, javascript, go, docker, terraform, …' },
files: { type: 'integer', description: 'how many files matched' },
},
},
},
frameworks: {
type: 'array',
description: 'frameworks read out of package.json / pyproject.toml / go.mod and the like, e.g. django, react',
items: { type: 'string' },
},
},
}
const SELECT_SCHEMA = {
type: 'object',
required: ['rulesetsPath', 'counts'],
additionalProperties: false,
properties: {
rulesetsPath: { type: 'string', description: 'absolute path of the rulesets JSON that was written' },
counts: {
type: 'object',
required: ['baseline', 'language', 'thirdParty'],
additionalProperties: false,
properties: {
baseline: { type: 'integer' },
language: { type: 'integer' },
thirdParty: { type: 'integer' },
},
},
},
}
const SCAN_SCHEMA = {
type: 'object',
required: ['ok', 'scansJson', 'succeeded', 'failed', 'skipped'],
additionalProperties: false,
properties: {
ok: {
type: 'boolean',
description:
'true only when run-scans.sh exited 0. It exits non-zero when no scan succeeded, which is a failed run rather than a run that found nothing.',
},
scansJson: { type: 'string', description: 'absolute path of scans.json, or "" when the script did not get that far' },
succeeded: { type: 'integer', description: 'length of .scans in scans.json, or -1 if unreadable' },
failed: { type: 'integer', description: 'length of .failed' },
skipped: { type: 'integer', description: 'length of .skipped' },
error: { type: 'string', description: 'the script stderr when ok is false, else ""' },
},
}
// ok exists for the same reason SCAN_SCHEMA's does. Without it the only fields are required
// ones, so an agent whose merge command exited non-zero has nowhere to say so and fills in
// total: 0 and a results.sarif path that was never written. The workflow would then return a
// populated success object that reads exactly like a scan which found nothing.
const REPORT_SCHEMA = {
type: 'object',
required: ['ok', 'resultsSarif', 'total', 'report'],
additionalProperties: false,
properties: {
ok: {
type: 'boolean',
description:
'true only when the merge command exited 0 and wrote the merged SARIF. A merge that failed is not a scan that found nothing.',
},
resultsSarif: { type: 'string', description: 'absolute path of the merged SARIF, or "" when the merge wrote none' },
total: {
type: 'integer',
description: 'finding count read from the merged SARIF, never summed from per-scan counts; -1 when the merge failed',
},
report: { type: 'string', description: 'the markdown summary to show the user' },
error: { type: 'string', description: 'the merge stderr when ok is false, else ""' },
},
}
// Every later phase runs a script or reads a reference out of the semgrep skill directory, and
// where that directory is depends on how the plugin was loaded. It cannot be a constant:
// ${CLAUDE_PLUGIN_ROOT} is exported to hook, MCP and LSP subprocesses and substituted into
// skill and agent content, but a workflow script is none of those. A repo-relative path only
// resolves inside a checkout of trailofbits/skills, and a marketplace install — which is every
// real user — runs with their own project as cwd.
//
// So it is resolved at runtime, folded into the Detect phase rather than costing its own agent
// turn. The agent's Bash subprocess can read $CLAUDE_PLUGIN_ROOT even though this script cannot.
// Each candidate ends at scripts/run-scans.sh rather than the skill directory, which makes a
// stale install self-excluding: versions before that script have nothing for the search to bind
// to, and binding to one would leave the scan phase with no command to run.
const RESOLVE_SKILL_DIR = skillHint
? [
`The semgrep skill directory is \`${skillHint}\`. Confirm \`${skillHint}/scripts/run-scans.sh\``,
'exists and report it as skillDir; report "" if it does not.',
].join('\n')
: [
'Locate the semgrep skill directory. Run these in order and stop at the first that',
' prints a path:',
'',
' ls "$CLAUDE_PLUGIN_ROOT/skills/semgrep/scripts/run-scans.sh" 2>/dev/null',
' ls ~/.claude/plugins/cache/*/static-analysis/*/skills/semgrep/scripts/run-scans.sh 2>/dev/null | sort -V | tail -1',
" find . -maxdepth 6 -type f -path '*static-analysis/skills/semgrep/scripts/run-scans.sh' 2>/dev/null | head -1",
" find \"$HOME\" -maxdepth 9 -type f -path '*static-analysis/skills/semgrep/scripts/run-scans.sh' 2>/dev/null | head -1",
'',
' The last one takes ~15s and is the fallback for a plugin loaded with --plugin-dir from',
' outside the current tree, so run it only if the first three print nothing.',
'',
' Report the directory two levels above the matched file (the one containing scripts/) as',
' skillDir. If all four print nothing, report skillDir as "" rather than guessing a path.',
].join('\n')
phase('Detect')
const detected = await agent(
[
'Resolve where this Semgrep run will write, confirm the tool is usable, and profile the codebase.',
'',
`Target: ${targetHint || 'the current working directory'}`,
outHint ? `Output directory: ${outHint}` : 'Output directory: choose it as described below.',
'',
'1. Resolve the target to an absolute path with `cd … && pwd`. Fail if it is not a directory.',
outHint
? '2. Use the output directory given above. `mkdir -p "$OUT/raw" "$OUT/results"`, then resolve it with `cd … && pwd`.'
: [
'2. Pick the output directory: `static_analysis_semgrep_1` beside the target, incrementing',
' the suffix while the name exists. `mkdir -p "$OUT/raw" "$OUT/results"`, then resolve it',
' with `cd … && pwd`.',
].join('\n'),
' It must be an absolute path and must not be the target itself.',
'',
`3. ${RESOLVE_SKILL_DIR}`,
'',
'4. Confirm semgrep is installed (`semgrep --version`). If it is not, stop and say so —',
' there is no point profiling a codebase you cannot scan.',
'',
'5. Check Pro. Keep stderr: "OSS only" has several causes (logged out, no subscription,',
' registry blocked) and the run downgrades silently for all of them.',
' if PRO_ERR=$(semgrep --pro --validate --metrics=off --config p/default 2>&1); then',
' echo "Pro available"; else echo "OSS only"; printf \'%s\' "$PRO_ERR" | tail -n 3; fi',
' --metrics=off matters here too: this is the first semgrep call of the run and it resolves',
' p/default against the registry, so without it an audit phones home before scanning.',
'',
'6. Detect languages by counting files. Report the count that justified each category, since',
' a category with one file is worth knowing about before its rulesets run. Cover at least:',
' python, javascript, typescript, go, ruby, java, php, c, cpp, rust, docker, terraform,',
' kubernetes. Use lowercase category names.',
'',
'7. Read the framework markers that exist — package.json, pyproject.toml, Gemfile, go.mod,',
' Cargo.toml, pom.xml — and name the frameworks you find (django, flask, react, express,',
' nextjs, spring, …). These select extra rulesets in the next phase.',
'',
'Report a language only when files actually matched. An invented category costs a ruleset',
'that scans nothing and reads in the final report as coverage that happened.',
].join('\n'),
{ label: 'detect', schema: DETECT_SCHEMA },
)
if (!detected) throw new Error('the detect phase returned nothing; no scan ran')
const { target, outputDir } = detected
if (!target || !target.startsWith('/')) throw new Error(`detect returned a non-absolute target: ${JSON.stringify(target)}`)
if (!outputDir || !outputDir.startsWith('/')) throw new Error(`detect returned a non-absolute output directory: ${JSON.stringify(outputDir)}`)
if (outputDir.replace(/\/+$/, '') === target.replace(/\/+$/, '')) {
throw new Error(`output directory is the scan target (${outputDir}); the run would scan its own output`)
}
// Throwing here rather than degrading. Without the skill directory the scan phase has no
// run-scans.sh to invoke, and an agent handed the flag list and a Bash tool would compose the
// semgrep commands by hand — dropping --metrics=off, the --include scoping and the
// output-directory --exclude, which is the exact failure the script exists to prevent.
const SKILL_DIR = (detected.skillDir || '').replace(/\/+$/, '')
if (!SKILL_DIR || !SKILL_DIR.startsWith('/')) {
throw new Error(
'could not locate the semgrep skill directory, so run-scans.sh cannot be invoked and no scan ran. ' +
'Pass it explicitly as {"skill": "/abs/path/to/skills/semgrep"}.',
)
}
const langNames = (detected.languages || []).map((l) => l.name).filter(Boolean)
log(
`target ${target}, output ${outputDir}, ${detected.pro ? 'Pro' : 'OSS'}, ` +
`${langNames.length ? langNames.join(', ') : 'no languages detected'}` +
`${(detected.frameworks || []).length ? ` (${detected.frameworks.join(', ')})` : ''}`,
)
if (!detected.pro && detected.proReason) log(`Pro unavailable: ${detected.proReason}`)
log(`skill directory ${SKILL_DIR}`)
phase('Select')
const selected = await agent(
[
'Choose the Semgrep rulesets for this codebase and write them to a file.',
'',
`Output directory: ${outputDir}`,
`Target: ${target}`,
`Detected languages: ${langNames.length ? langNames.join(', ') : '(none)'}`,
`Detected frameworks: ${(detected.frameworks || []).join(', ') || '(none)'}`,
'',
`Follow the Ruleset Selection Algorithm in ${SKILL_DIR}/references/rulesets.md. Read that file;`,
'do not select from memory. It is the same catalogue the manual path uses, so a ruleset added',
'there has to reach this run too.',
'',
'It covers, in order: the security baseline that is always included, language rulesets,',
'framework rulesets for what was detected, infrastructure rulesets, and the third-party',
'repositories. The third-party rules from Trail of Bits, 0xdea and Decurity are required',
'rather than optional wherever the detected language matches — they catch vulnerabilities',
'absent from the official registry, and dropping them is the most common way this scan',
'comes back quieter than it should.',
'',
`Write the result to ${outputDir}/rulesets.json in exactly this shape:`,
'',
' {',
' "baseline": ["p/security-audit", "p/secrets"],',
' "python": ["p/python", "p/django"],',
' "javascript": ["p/javascript"],',
' "third_party": ["https://github.com/trailofbits/semgrep-rules"]',
' }',
'',
'Rules for that file, all enforced by the script that reads it, which exits without scanning',
'rather than guessing:',
' - every value is an ARRAY, even a single ruleset',
' - language keys hold registry identifiers like p/python; never a URL',
' - repository URLs go under third_party and nowhere else, as https:// URLs',
' - "all" is reserved and cannot be a language key',
' - one key per language, using the lowercase names from the detected list',
'',
'Include a language key only for languages that were actually detected. A ruleset for a',
'language that is not present scans nothing and pads the report with coverage that did not',
'happen.',
].join('\n'),
{ label: 'select', schema: SELECT_SCHEMA },
)
if (!selected || !selected.rulesetsPath) throw new Error('the select phase produced no ruleset file; no scan ran')
const c = selected.counts || {}
log(`rulesets: ${c.baseline || 0} baseline, ${c.language || 0} language, ${c.thirdParty || 0} third-party`)
phase('Scan')
// One agent, one command. Parallelism is the script's --jobs, and each exit code comes from
// the process that produced it; fanning out agents would put an LLM between semgrep and its
// own exit status.
const scanned = await agent(
[
'Run the Semgrep scans. One command, exactly as written:',
'',
` ${SKILL_DIR}/scripts/run-scans.sh \\`,
` --target "${target}" \\`,
` --output-dir "${outputDir}" \\`,
` --mode ${mode} \\`,
` --rulesets "${selected.rulesetsPath}"${detected.pro ? ' \\\n --pro' : ''}${jobs ? ` \\\n --jobs ${jobs}` : ''}`,
'',
'Do not add rulesets, change the flags, or run semgrep yourself. The script generates every',
'command, and --metrics=off, the --include scoping and the output-directory --exclude are',
'its job. A scan you compose by hand drops those silently.',
'',
'It writes scans.json in the output directory and prints its path. Read the counts from that',
'file with jq:',
` jq '.scans | length' "${outputDir}/scans.json"`,
` jq '.failed | length' "${outputDir}/scans.json"`,
` jq '.skipped | length' "${outputDir}/scans.json"`,
'',
'A non-zero exit means no scan succeeded. Report ok=false with the stderr, and do not retry',
'with different arguments: the ruleset file is what produced them.',
].join('\n'),
{ label: 'scan', schema: SCAN_SCHEMA },
)
if (!scanned || !scanned.ok) {
const why = (scanned && scanned.error) || 'the scan agent returned nothing'
throw new Error(`no scan succeeded: ${why}`)
}
log(`${scanned.succeeded} scans succeeded, ${scanned.failed} failed, ${scanned.skipped} skipped`)
phase('Report')
const reported = await agent(
[
'Merge the scan output and write the summary.',
'',
`Output directory: ${outputDir}`,
`Scan results: ${scanned.scansJson || `${outputDir}/scans.json`}`,
`Mode: ${mode}`,
'',
mode === 'important-only'
? [
'1. Post-filter first. Apply the "Filter All Result Files in a Directory" jq filter from',
` ${SKILL_DIR}/references/scan-modes.md to every result JSON in ${outputDir}/raw/.`,
' It writes *-important.json alongside the originals and leaves them untouched.',
' Every .sarif in raw/ must end up with one, because step 2 requires it.',
].join('\n')
: '1. Run-all mode: no post-filter. Merge the raw output as it is.',
'',
'2. Merge:',
` uv run ${SKILL_DIR}/scripts/merge_sarif.py "${outputDir}/raw" "${outputDir}/results/results.sarif"${
mode === 'important-only' ? ' --important' : ''
} \\`,
` --scans "${scanned.scansJson || `${outputDir}/scans.json`}"`,
'',
' --scans drops the output of scans recorded under .failed. A scan that died part-way may',
' still have written a .sarif, and without this one dead scan denies every healthy scan a',
' merged result: its output has no post-filter beside it, which is an error rather than an',
' empty filter. The excluded files are named on stdout; carry them into the report.',
...(mode === 'important-only'
? [
'',
' --important is what applies the mode to the merged SARIF. Do not try to run the jq',
' filter from step 1 against a .sarif file: that filter reads .results[].extra.metadata,',
' which SARIF does not have at all, so it exits with "Cannot iterate over null" and',
' leaves the deliverable unfiltered. The flag matches findings across the two formats',
' on (rule, file, line) instead, and fails the merge if any scan has no filtered JSON.',
]
: []),
'',
'',
' A non-zero exit means the merge wrote no results.sarif. Report ok=false with its stderr,',
' total=-1 and resultsSarif="". Do not retry it with different arguments and do not write a',
' summary reporting zero findings: the scans succeeded and the merge did not, and those two',
' read identically once a total of 0 is written down.',
'',
'3. Confirm the merged file is valid JSON and count the findings FROM IT:',
` jq '[.runs[].results[]] | length' "${outputDir}/results/results.sarif"`,
' Never sum the per-scan findings counts instead. One finding flagged by two rulesets is',
' one row in the merge and two in that sum.',
'',
'4. Delete the cloned rule repositories, now that nothing is reading them:',
` [ -n "${outputDir}" ] && rm -rf "${outputDir}/repos"`,
'',
'5. Write the summary as markdown. Include the finding total, a breakdown by severity and by',
' rule category, and where the results were written.',
'',
' Read .failed and .skipped from scans.json and give them their own "Did Not Run" section',
' whenever either is non-empty, naming the ruleset and the reason. A run that covered four',
' of nine rulesets reads exactly like one that covered four of four unless you say so.',
' Do the same for .unscoped (languages with no --include, which ran against every file) and',
' .alsoShared (rulesets not repeated per language because they already ran over the whole',
' target — coverage is unaffected, but it explains why the ruleset and scan counts differ).',
'',
' .coveredNothing gets its own section too, and matters more than the other two: those are',
' rulesets that opened zero files because their --include globs matched nothing. They',
' report 0 findings exactly like a ruleset that ran and found nothing, so leaving them out',
' is how a plan aimed at the wrong languages reads as a clean audit.',
'',
'6. Read the merge command\'s own stdout and carry two lines into the report if present:',
' "excluding N SARIF file(s) from failed scans" and "unparseable: N of M SARIF files".',
' The unparseable one is the important one. That scan is in .scans as a success with a',
' finding count, so its findings are missing from results.sarif and nothing in scans.json',
' says so — the total simply comes out lower. Name those files and say their findings are',
' not in the merge.',
'',
'7. If .excludePattern in scans.json is non-empty, give it a line too. Every scan skipped',
' that pattern, and semgrep matches it anywhere in the tree, so a target with its own',
' directory of the same name lost those files as well. It reads as clean coverage.',
].join('\n'),
{ label: 'report', schema: REPORT_SCHEMA },
)
if (!reported) throw new Error(`the merge phase returned nothing; raw scan output is in ${outputDir}/raw`)
// Symmetric with the scan phase. A merge that exited non-zero wrote no results.sarif, and
// returning that as a run with zero findings hides a completed scan behind a failed merge.
if (!reported.ok) {
const why = reported.error || 'the merge agent did not say why'
throw new Error(`the merge failed, so there is no results.sarif: ${why}. Raw scan output is in ${outputDir}/raw`)
}
log(`${reported.total} findings in ${reported.resultsSarif}`)
return {
target,
outputDir,
mode,
pro: detected.pro,
languages: detected.languages,
rulesetsPath: selected.rulesetsPath,
scansJson: scanned.scansJson,
succeeded: scanned.succeeded,
failed: scanned.failed,
skipped: scanned.skipped,
resultsSarif: reported.resultsSarif,
total: reported.total,
report: reported.report,
}