init mutation-testing skill (#140)

* init mutation-testing skill

* fix metadata

* init codex symlink

* polish according to review feedback

---------

Co-authored-by: Dan Guido <dan@trailofbits.com>
This commit is contained in:
bohendo
2026-03-31 18:12:57 -04:00
committed by GitHub
parent 6b79b5e879
commit af35cdeac9
9 changed files with 794 additions and 0 deletions
@@ -0,0 +1,10 @@
{
"name": "mutation-testing",
"version": "1.0.0",
"description": "Configures mewt or muton mutation testing campaigns — scopes targets, tunes timeouts, and optimizes long-running runs. Use when the user mentions mewt, muton, mutation testing, or wants to configure or optimize a mutation testing campaign.",
"author": {
"name": "Trail of Bits",
"email": "opensource@trailofbits.com",
"url": "https://github.com/trailofbits"
}
}
+43
View File
@@ -0,0 +1,43 @@
# Mutation Testing — Campaign Configuration (mewt/muton)
Helps configure [mewt](https://github.com/trailofbits/mewt) or [muton](https://github.com/trailofbits/muton) mutation testing campaigns — scoping targets, tuning timeouts, and optimizing long-running runs so you can execute `mewt run` or `muton run` with confidence.
> **Note**: muton and mewt share identical interfaces but target different languages — mewt for general-purpose languages, muton for TON smart contracts (Tact, Tolk, FunC). All commands and configuration patterns in this plugin apply to both tools. File names change accordingly: `mewt.toml` → `muton.toml`, `mewt.sqlite` → `muton.sqlite`.
## What It Does
Walks through a 5-phase configuration workflow:
1. **Initialize and validate targets** — run `mewt init`, review auto-generated config, fix include/ignore patterns
2. **Generate mutants and assess scope** — count mutants, time the test command, estimate campaign duration
3. **Decide on optimization** — choose between full run, targeted components, high/medium severity only, or two-phase campaign
4. **Validate test command and timeout** — verify the command works; set manual timeout for recompilation-heavy languages (Solidity/Foundry, heavy C++)
5. **Final validation checklist** — confirm config, mutant count, target selection, and timeout before running
## When to Use
- Setting up a new mutation testing campaign
- Optimizing a campaign that would take too long to run
- Diagnosing why no mutants are generated or why the test command fails
## Prerequisites
- [mewt](https://github.com/trailofbits/mewt) v3.0.0+ or [muton](https://github.com/trailofbits/muton) v3.0.0+ installed
- A test suite runnable from the command line
- Source code in a supported language:
- **mewt**: Rust, Solidity, Go, TypeScript, JavaScript
- **muton**: Tact, Tolk, FunC (TON smart contract languages)
## Example Usage
```
User: "Help me set up mewt for this Solidity project"
User: "Configure muton for this FunC codebase"
User: "My mewt campaign would take 30 hours — how do I optimize it?"
→ Guides through configuration, scope assessment, and optimization
```
## References
- [mewt GitHub Repository](https://github.com/trailofbits/mewt)
- [Use mutation testing to find the bugs your tests don't catch](https://blog.trailofbits.com/2025/09/18/use-mutation-testing-to-find-the-bugs-your-tests-dont-catch/) — Trail of Bits blog post
@@ -0,0 +1,76 @@
---
name: mutation-testing
description: "Configures mewt or muton mutation testing campaigns — scopes targets, tunes timeouts, and optimizes long-running runs. Use when the user mentions mewt, muton, mutation testing, or wants to configure or optimize a mutation testing campaign."
allowed-tools:
- Read
- Write
- Bash
- Grep
---
# Mutation Testing — Campaign Configuration (mewt/muton)
> **Note**: muton and mewt share identical interfaces but target different languages — mewt for general-purpose languages (Rust, Solidity, Go, TypeScript, JavaScript), muton for TON smart contracts (Tact, Tolk, FunC). All examples use `mewt` commands, but they work exactly the same with `muton`. File names change accordingly: `mewt.toml` → `muton.toml`, `mewt.sqlite` → `muton.sqlite`.
## When to Use
Use this skill when the user:
- Mentions "mewt", "muton", or "mutation testing"
- Needs to configure or optimize a mutation testing campaign
- Wants to run `mewt run` and needs help getting set up first
## When NOT to Use
Do not use this skill when the user:
- Wants to analyze or report on completed campaign results
- Asks about tests or coverage without mentioning mutation testing
---
## Quick Start
Load [workflows/configuration.md](workflows/configuration.md) — a 5-phase guide from `mewt init` to a validated, ready-to-run campaign.
**General question or unfamiliar command?**
Run `mewt --help` or `mewt <subcommand> --help`, then assist.
---
## Reference Index
| File | Content |
|------|---------|
| [workflows/configuration.md](workflows/configuration.md) | 5-phase guide: init, scope, optimize, validate, run |
| [references/optimization-strategies.md](references/optimization-strategies.md) | Per-file targeting, two-phase campaigns, mutation type filtering |
---
## Essential Commands
```bash
# Initialize and mutate
mewt init # Create mewt.toml and mewt.sqlite
mewt mutate [paths] # Generate mutants without running tests
mewt run [paths] # Run the full campaign
# Inspect configuration and scope
mewt print config # View effective configuration
mewt print targets # Table of all targeted files
mewt print mutations --language [lang] # Available mutation types
mewt status # Mutant count and per-file breakdown
# Investigate specific mutants
mewt print mutants --target [path] # All mutants for a file
mewt print mutants --severity high # Filter by severity
mewt print mutant --id [id] # View mutated code diff
mewt test --ids [ids] # Re-test specific mutants
```
---
## What Results Mean
- **Caught/TestFail**: Tests detected the mutation (good)
- **Uncaught**: Mutation survived — indicates untested logic
- **Timeout**: Tests took too long, inconclusive
- **Skipped**: A more severe mutant already failed on the same line
@@ -0,0 +1,323 @@
# Optimization Strategies
Apply these strategies **before** running a campaign when Phase 3 of the configuration workflow requires optimization (estimated >16 hours or user requests).
---
## Priority 1: Verify Target Selection
**Most common issue:** Mutating non-source code.
**Diagnostic:**
```bash
mewt print config # Check [targets] include/ignore
mewt print targets # Check what was actually mutated
```
**Look for unintended files:**
- Mocks: `src/mocks/`, `__mocks__/`
- Tests: `*_test.rs`, `*.test.js`, `tests/`
- Dependencies: `vendor/`, `node_modules/`
- Generated: `proto/`, `generated/`
**Fix:** Update `[targets]` in `mewt.toml` to be more specific:
```toml
# Before (too broad)
[targets]
include = ["**/*.rs"]
# After (specific)
[targets]
include = ["src/**/*.rs", "lib/**/*.rs"]
ignore = ["test", "mock", "generated"]
```
Re-run `mewt mutate` and check new count.
---
## Priority 2: Analyze Project Structure
**Goal:** Understand mutant distribution and test organization to choose the right optimization.
**1. Get mutant counts per component:**
```bash
# Use single quotes to prevent shell glob expansion
mewt print mutants --target 'src/auth/**/*.rs' | wc -l
mewt print mutants --target 'src/core/**/*.rs' | wc -l
mewt print mutants --target 'src/utils/**/*.rs' | wc -l
```
Present breakdown to user:
```
Component breakdown:
- src/auth/: 200 mutants × 5s = ~17 min
- src/core/: 800 mutants × 8s = ~1.8 hrs
- src/utils/: 150 mutants × 3s = ~8 min
Total: 1150 mutants, ~2.3 hrs worst-case
```
**2. Count mutations by severity:**
```bash
# Check enabled mutation types
mewt print config | grep mutations
# Count by severity level
mewt print mutants --severity high | wc -l
mewt print mutants --severity medium | wc -l
mewt print mutants --severity low | wc -l
# Or count specific mutation types
mewt print mutants --mutation-types ER | wc -l
mewt print mutants --mutation-types CR | wc -l
# Compare to total
mewt print mutants | wc -l
```
Example output:
```
High/Medium severity: 450 mutants
Total mutants: 1200
Percentage: 37.5%
```
**Note:** The percentage varies drastically between codebases (15% to 50+ % is common).
---
## Priority 3: Choose Optimization Approach
Based on project structure analysis, present options to user with concrete time estimates:
### Option A: Run Full Campaign
- "Estimated ~X hours worst-case (likely faster in practice)"
- "Recommend starting Friday evening for weekend completion"
- **When to suggest:** Duration acceptable, comprehensive coverage desired
### Option B: Target Critical Components
- "Focus on specific components: src/auth/ (~17 min), src/crypto/ (~45 min)"
- "Start with one component and expand scope after review?"
- **When to suggest:** Clear component boundaries, user wants rapid iteration
**Implementation:**
```toml
[targets]
# Start with critical component
include = ["src/auth/**/*.rs"]
# After review, expand scope
# include = ["src/auth/**/*.rs", "src/core/**/*.rs"]
```
After editing `mewt.toml`, purge removed targets then mutate any newly included files:
```bash
mewt purge # removes targets no longer matching [targets].include/ignore
mewt mutate src/ # adds mutants for any newly included files
mewt status # confirm reduced mutant count
```
### Option C: High/Medium Severity Only
- "Limit to high/medium severity mutations (X mutants, ~Y hours)"
- "Low severity (operator shuffles) tests edge cases, less critical"
- **When to suggest:** Time-constrained, need actionable findings quickly
**Implementation (by severity level):**
```toml
[run]
mutations = ["ER", "CR", "IF", "IT"] # Specific types (high/medium)
```
After editing `mewt.toml`, full regeneration is required since existing mutants may no longer be valid under the new filter:
```bash
mewt purge --all # clear all existing mutants
mewt mutate src/ # regenerate with restricted mutation types
mewt status # confirm reduced mutant count
```
Or use severity filtering during analysis instead (no database changes needed):
```bash
# Run all mutants but filter results by severity
mewt results --severity high,medium
mewt print mutants --severity high
```
**Trade-offs to explain:**
- High/med severity: ~30-40% of mutants (varies by codebase)
- Low severity: ~60-70% of mutants (operator shuffles, edge cases)
- Low severity still provides value, just lower priority
- Using severity filters during analysis allows flexibility without re-running campaign
### Option D: Two-Phase Campaign (Integration-Heavy Only)
- "Phase 1: Targeted tests (estimable upfront), Phase 2: Re-test uncaught with full suite (duration depends on Phase 1 survivor count)"
- "Total: Phase 1 estimate + (survivors × full-suite time) vs naive total"
- **When to suggest:** Integration tests dominate, unit tests don't map cleanly to files
See Two-Phase Campaigns section below for detailed setup.
---
## Two-Phase Campaigns
**Use ONLY for integration-heavy test suites.** Not recommended for well-organized unit tests.
### When to Use
**Good fit:**
- Integration tests dominate runtime
- Unit tests provide broad coverage but don't map cleanly to specific files
- Targeted test commands significantly faster than full suite
**Not recommended:**
- Well-organized unit tests with clear file mappings
- Tests already fast and targeted
### Setup
**Phase 1 config (targeted tests):**
```toml
# TWO-PHASE CAMPAIGN
# Phase 1: Targeted tests (duration estimable upfront)
# Phase 2: Re-test uncaught mutants (duration depends on Phase 1 survivor count)
[test]
# PHASE 2: Uncomment after phase 1 completes
# cmd = "cargo test"
# timeout = 60
# PHASE 1: Targeted tests
[[test.per_target]]
glob = "src/auth/*.rs"
cmd = "cargo test auth::unit"
timeout = 10
[[test.per_target]]
glob = "src/core/*.rs"
cmd = "cargo test core::unit"
timeout = 15
# Catch-all: full suite for any file not matched above.
# Required unless [targets] is scoped to exactly the globs listed above.
[[test.per_target]]
glob = "**/*.rs"
cmd = "cargo test"
timeout = 60
```
**Rationale:** Phase 1 uses fast targeted tests. Phase 2 re-tests only the survivors with the comprehensive suite.
### Execution
**Phase 1:**
```bash
mewt run
```
Wait for completion.
**Phase 2 (after phase 1 completes):**
1. **Extract uncaught mutants:**
```bash
mewt results --status Uncaught --format ids > uncaught_ids.txt
```
2. **Update mewt.toml:**
- Comment out all `[[test.per_target]]` sections (including the catch-all)
- Uncomment Phase 2 `[test]` section
3. **Re-test with full suite:**
```bash
mewt test --ids-file uncaught_ids.txt
```
4. **Review final results:**
```bash
mewt results # Remaining uncaught are true coverage gaps
```
**Example speedup:**
```
Naive approach:
2,000 mutants × 45s = 25 hours
Two-phase approach:
Phase 1: 2,000 mutants × 8s = 4.4 hours → 450 uncaught (example outcome)
Phase 2: 450 uncaught × 45s = 5.6 hours → 180 truly uncaught
Total: ~10 hours (2.5× speedup)
Note: Phase 2 duration is unknowable before Phase 1 completes — it depends entirely
on how many mutants survive. The figures above illustrate one possible outcome.
Present Phase 1 as a firm estimate; present Phase 2 as (survivors × full-suite time)
once Phase 1 results are available.
```
---
## Per-Target Test Configuration
**Use when:** Tests are well-organized by module/file, and running targeted tests is significantly faster than the full suite.
### Setup Pattern
```toml
# Test full suite for every mutant (slow but comprehensive)
[test]
cmd = "go test ./..."
timeout = 45
# ALTERNATIVE: Targeted tests per file (fast, may miss cross-module failures)
[[test.per_target]]
glob = "auth/*.go"
cmd = "go test ./auth"
timeout = 10
[[test.per_target]]
glob = "core/*.go"
cmd = "go test ./core"
timeout = 15
[[test.per_target]]
glob = "utils/*.go"
cmd = "go test ./utils"
timeout = 8
# Catch-all for unmatched files
[[test.per_target]]
glob = "*.go"
cmd = "go test ./..."
timeout = 45
```
**Ordering matters:** First match wins. Place most specific patterns first, catch-all last.
### Verify Speedup
```bash
time go test ./... # Full suite: 45s
time go test ./auth # Targeted: 8s
```
If targeted tests aren't significantly faster, this optimization won't help.
### Trade-offs
**Benefits:**
- Faster campaign execution
- Scales linearly with codebase size
**Risks:**
- May miss cross-module integration bugs
- Requires correct glob-to-test mapping
**Mitigation:**
- Use this for initial passes
- Consider two-phase approach for comprehensive validation
@@ -0,0 +1,328 @@
# Configuration and Optimization Guide
Guide for configuring mewt and optimizing mutation testing performance **before** running a campaign.
## Goal
Configure mewt so the user can run `mewt run` with optimal settings that balance thoroughness and execution time.
---
## Configuration Workflow
### Phase 1: Initialize and Validate Targets
**Entry:** User has a codebase and wants to configure mutation testing.
**Actions:**
1. **Initialize mewt:**
```bash
mewt init # Creates mewt.toml and mewt.sqlite
```
Note: If working with a config in a non-standard location, use `--config path/to/mewt.toml`. The parent directory of the config file becomes the working directory, and relative paths in the config resolve from there.
2. **Review auto-generated configuration:**
```bash
mewt print config
```
3. **Verify target patterns:**
- **Include patterns** should match only source code: `src/`, `lib/`, `contracts/`
- **Ignore patterns** should exclude tests, dependencies, generated code
- Note: Ignore patterns use substring matching (e.g., `"test"` matches `tests/`, `test_utils.rs`)
4. **Edit `mewt.toml` if needed** to fix target patterns:
```toml
[targets]
include = ["src/**/*.rs"] # Specific source directories only
ignore = ["test", "mock"] # Exclude test/mock files within src/
```
**Exit:** `mewt.toml` contains valid target patterns that match intended source files.
---
### Phase 2: Generate Mutants and Assess Scope
**Entry:** Phase 1 exit criteria met (valid `mewt.toml` exists).
**Actions:**
1. **Generate mutants:**
```bash
mewt mutate src/
```
Note: Output shows per-target summaries with severity breakdown (high/medium/low). Use `--verbose` to see individual mutants.
2. **Check mutant count and distribution:**
```bash
mewt status # View total mutant count
mewt print targets # Pretty table showing which files were mutated
```
3. **Time the test command:**
```bash
time <test-command-from-config> # e.g., time cargo test
```
Note the baseline test duration.
4. **Calculate worst-case campaign duration:**
- Formula: `mutant_count × test_duration`
- Example: 500 mutants × 10s = ~1.4 hours
- Actual runtime typically faster (tests catch mutants quickly, skipping reduces load)
**Exit:** Know the mutant count, test duration, and estimated campaign time.
---
### Phase 3: Decide on Optimization Strategy
**Entry:** Phase 2 exit criteria met (mutant count and time estimate known).
**Decision Tree:**
```
Estimated campaign duration?
|
+-- < 1 hour
| └─> Proceed to Phase 4 (no optimization needed)
|
+-- 1-16 hours
| └─> Consult user: Acceptable? Run overnight/end-of-day?
| +-- User accepts --> Proceed to Phase 4
| +-- User declines --> Apply optimization (see Optimization Strategies below)
|
+-- > 16 hours
└─> Explore optimization options (see Optimization Strategies below)
```
**Actions (if optimization needed):**
Read `references/optimization-strategies.md` for detailed strategies and examples. Then:
1. Verify target selection (most common issue — check `mewt print targets` for unintended files)
2. Analyze project structure (`mewt print mutants --target 'src/component/**'` per component)
3. Present options to user with time estimates (full campaign / target critical components / high-severity only / two-phase)
4. Apply chosen optimization to `mewt.toml`
5. **If `[targets]` or `[run].mutations` changed**, update the database and recalculate duration:
- **Target scope narrowed** (Option B): purge removed targets, then mutate any newly included files:
```bash
mewt purge # removes targets no longer in [targets].include/ignore
mewt mutate src/ # adds mutants for any newly included files
mewt status # verify reduced mutant count
```
- **Mutation types restricted** (Option C): full regeneration required since existing mutants may no longer be valid:
```bash
mewt purge --all
mewt mutate src/
mewt status # verify reduced mutant count
```
Update the duration estimate before proceeding to Phase 4.
**Exit:** Either campaign duration is acceptable, or `mewt.toml` has been optimized, the database updated, and the new duration estimate confirmed.
---
### Phase 4: Validate Test Command and Timeout
**Entry:** Phase 3 exit criteria met (optimization applied if needed).
**Actions:**
1. **If test configuration was modified in Phase 3,** verify it works:
```bash
<test-command-from-config> # Should succeed without errors
```
Skip this step if Phase 2's timing already validated the unmodified command.
2. **Check if timeout adjustment needed:**
**Default:** Mewt auto-calculates timeout (baseline test time × 2), which accounts for incremental recompilation in most cases.
**Exception:** For compiled languages where recompilation of dependents dominates test time (Solidity/Foundry):
```bash
# Test with warm cache
time forge test # e.g., 0.8s
# Simulate mutation: touch source file to trigger dependent recompilation
touch src/Contract.sol
# Test again (includes recompilation)
time forge test # e.g., 5.2s
# If drastically different, set manual timeout in mewt.toml
```
If recompilation time >> test time:
```toml
[test]
cmd = "forge test"
timeout = 11 # Based on: 5.2s × 2 = 10.4s, round up
```
Otherwise, omit `timeout` and let mewt auto-calculate.
**Exit:** Test command verified working (if modified), timeout appropriately set (auto or manual).
---
### Phase 5: Final Validation
**Entry:** Phase 4 exit criteria met (test command works if modified, timeout set).
**Actions:**
Run through the validation checklist to verify all prior phases completed successfully:
- [ ] `mewt print config` — Configuration syntax valid, no errors
- [ ] `mewt status` — Mutant count matches expected count (Phase 2 count if no optimization applied; lower post-optimization count if `[targets]` or `[run].mutations` was narrowed)
- [ ] `mewt print targets` — Only intended files mutated (no tests, mocks, dependencies)
- [ ] Test command verified — Already validated in Phase 2 (and Phase 4 if modified)
- [ ] Timeout set — Auto-calculated or manually set for recompilation-heavy languages
- [ ] Scope acceptable — Duration estimate from Phase 2 acceptable to user
**Exit:** Ready to run `mewt run`.
---
## Configuration Reference
### File Structure
```toml
db = "mewt.sqlite"
[log]
level = "info" # trace, debug, info, warn, error
[targets]
# BE SPECIFIC: Source code only, never tests/dependencies
include = ["src/**/*.js", "lib/**/*.js"]
ignore = ["test", "mock"] # substring matches, not globs
[run]
# Optional: Restrict mutation types (omit to test all)
# mutations = ["ER", "CR", "IF", "IT"]
[test]
cmd = "npm test"
# timeout = 30 # Optional: auto-calculated if omitted (2× baseline)
# Per-target rules (first match wins)
[[test.per_target]]
glob = "src/core/*.js"
cmd = "npm test -- core"
timeout = 20
```
### Target Configuration Examples
**Important:** Restrictive `include` patterns exclude most unwanted files. Only add `ignore` patterns for items within included paths.
```toml
# Rust project
[targets]
include = ["src/**/*.rs"]
ignore = ["test", "mock", "generated"]
# Solidity project
[targets]
include = ["contracts/**/*.sol"]
ignore = ["test", "interfaces", "mocks"]
# Go project
[targets]
include = ["**/*.go"]
ignore = ["test", "mock", "generated"]
# JavaScript/TypeScript
[targets]
include = ["src/**/*.ts", "lib/**/*.ts"]
ignore = ["test", "spec", "mock"]
```
### Test Configuration
**Timeout Calculation:**
Mutants trigger incremental recompilation (only mutated file + dependents). Mewt's auto-calculated timeout (2× baseline) usually accounts for this.
**Edge case:** In some compiled languages (Solidity/Foundry), recompiling dependent files takes much longer than running tests. Verify by timing tests, touching a file, and timing again. If drastically different, set manual timeout based on the slower measurement.
```toml
# Option 1: Auto-calculate (recommended for most languages)
[test]
cmd = "cargo test"
# Omit timeout — mewt measures baseline and applies 2× multiplier
# Option 2: Explicit timeout (for recompilation-heavy languages)
[test]
cmd = "forge test"
timeout = 11 # Based on: touch file, time test (5.2s), × 2
```
---
## Troubleshooting
### No Mutants Generated
**Check language support:**
```bash
mewt print mutations --language rust
```
**Verify patterns:**
```bash
mewt print config
ls src/**/*.rs # Do files exist and match include patterns?
```
**Common causes:**
- Include pattern doesn't match files
- Ignore pattern too broad (e.g., `"test"` matches `test_utils.rs`)
- Unsupported language
---
### Test Command Fails
**Run command manually:**
```bash
pytest # Should work from project directory without errors
```
**Find correct command:**
- Check: `Makefile`, `justfile`, `package.json`, `README.md`
- In monorepos, may need to run from workspace subdirectory
---
### Configuration Validation
Before running `mewt run`, complete Phase 5's validation checklist above. If any item fails, return to the relevant phase to fix it.
---
## Campaign Execution Timing
Recommend timing based on estimated duration:
- **< 1 hour:** Run anytime
- **1-16 hours:** Start end-of-day, results by morning
- **16-48 hours:** Start Friday evening, results Monday
- **Two-phase:** Phase 1 overnight, Phase 2 next day
---
## Configuration Principles
- **Configure via `mewt.toml`** — Not CLI flags (version control the config)
- **Target source code specifically** — Exclude tests, dependencies, generated code
- **Prefer limiting files over mutation types** — Better to assess critical code thoroughly
- **Verify test commands** — Run manually before campaign
- **Trust auto-calculated timeouts** — 2× baseline accounts for incremental recompilation in most cases
- **Measure before optimizing** — Profile actual test times before applying per-target config
- **Document decisions** — Commit `mewt.toml` with comments explaining configuration choices