skills/callstack__agent-device
1
0
Fork 0
mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00
Code Issues Packages Projects Releases Wiki Activity

test: ratchet mutation score over enumerated decision kernels (#1441)

Browse Source 
* test: ratchet mutation score over enumerated decision kernels

Adds a Stryker (vitest runner) mutation lane scoped to the decision kernels,
a per-module baseline with tool/config provenance, and a ratchet that only
lets scores rise. Non-gating until two consecutive stable weekly sweeps.

Refs #1415

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore: declare the mutation test-scope seam for production-export analysis

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: own kernel tests in the mutation registry and ship the #1430 lane envelope

- restore bench:help-conformance, broken by a formatting-path edit
- kernel test files select their module on PRs (registry `tests` + workflow paths),
  asserted to reach the kernel through the import graph
- every mutation run writes the standard scheduled-lane artifact envelope
- move src/utils/__tests__/errors.test.ts beside its source per the mirror rule

Refs #1415, #1430

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: derive kernel test ownership and land the scheduled-lane health monitor

Ownership of a kernel's tests is now computed from the static import graph
(scripts/mutation/ownership.ts) instead of a hand-listed set, so a test that
reaches a kernel indirectly -- src/__tests__/daemon-error.test.ts through
src/daemon.ts -- selects that kernel on a PR. The PR lane triggers on every src
test and shards the derived modules, keeping wall clock at one module.

The lane envelope (#1430) is now written on every exit path with the stage it
reached, so a crash before any mutant runs is distinguishable from a lane that
never ran. Adds the derived cadence monitor (scripts/lane-health, daily
workflow): scheduled lanes are enumerated from .github/workflows/ and reported
dark, failing, or never-run against their own cron cadence.

* fix: merge only Stryker reports from a shard directory

The shard artifacts now carry the lane envelope beside mutation.json, and the
merge globbed every .json under the download path, so the ratchet job fed the
envelope to the report parser and died after the mutants had already run.

* fix(mutation): fail on incomplete shard sets and envelope pre-run failures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mutation): downgrade a passing envelope when a later lane step fails

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(mutation): shard by registry, defer the PR lane, drop the bundled watcher

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs: describe registry sharding and the deferred PR mutation lane

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mutation): make the pre-graduation tooling exception select real mutants

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mutation): give the worktree fixture commits their own identity

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
devin-ai-integration[bot]
2026-07-28 10:07:14 +02:00
committed by GitHub
parent d747ef6230
commit 8cce0ef6b8
29 changed files with 4315 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.fallowrc.json
+15 -1
View File
@@ -66,7 +66,7 @@
},
{
"comment": "Tool config default exports, loaded by the tool rather than imported.",
"file": "{tsdown.config.ts,website/rspress.config.ts,test/skillgym/skillgym.config.ts,test/skillgym/suites/*.ts}",
"file": "{tsdown.config.ts,vitest.mutation.config.ts,website/rspress.config.ts,test/skillgym/skillgym.config.ts,test/skillgym/suites/*.ts}",
"exports": [
"default"
]
@@ -81,6 +81,20 @@
"GatedKeysAreResolverKeys"
]
},
{
"comment": "Mutation-lane seam: readTestScope is consumed by vitest.mutation.config.ts, a tool config outside --production analysis.",
"file": "scripts/mutation/test-scope.ts",
"exports": [
"readTestScope"
]
},
{
"comment": "Mutation-lane completeness gate: ownedTestFiles enumerates the whole derived ownership map, which only the assertion in ownership.test.ts consumes — a production consumer would defeat the point.",
"file": "scripts/mutation/ownership.ts",
"exports": [
"ownedTestFiles"
]
},
{
"comment": "Help-benchmark conformance seams: helpTopicIds feeds the topic-coverage gate and PRIVATE_AX_RECOVERY_SAMPLE feeds the skillgym suite; both consumers are test-tree files outside --production analysis.",
"file": "{src/cli/parser/cli-help.ts,scripts/help-conformance-sample-outputs.mjs}",
.github/workflows/mutation-affected.yml
+178
View File
@@ -0,0 +1,178 @@
name: Mutation Affected
# PR-side half of the decision-kernel mutation lane (issue #1415).
#
# Graduation, not a flag day: this job runs on every PR that touches a kernel
# module, but `scripts/mutation/run.ts` only exits non-zero once the committed
# baseline reports `gating: true` — earned by two consecutive stable weekly
# sweeps (mutation-weekly.yml). Until then it is a report with the same numbers,
# so the gate's first failing day is not also its first running day.
#
# Cost control: before graduation an affected run is a report nobody acts on, so
# `select` returns an empty matrix and no mutants run — except when the diff
# touches the lane's own tooling, the one case where a pre-graduation run buys
# something (the gate has to be proven before it can bite).
#
# Scope is the AFFECTED modules only, and affectedness is DERIVED from the import
# graph (scripts/mutation/ownership.ts): a kernel source, or any test that reaches
# one. Reaching a kernel is a superset of killing its mutants, so the `select` job
# frequently returns several modules — sharded like the weekly sweep so the PR's
# wall clock is one module, not their sum. The weekly run stays the full sweep.
on:
pull_request:
paths:
# Kernel sources, every src test (ownership is derived, so any test may own
# a kernel — `select` decides, not this filter), and the lane's own tooling.
# scripts/mutation/workflow.test.ts asserts this covers the registry.
- "src/kernel/errors.ts"
- "src/daemon/ref-frame.ts"
- "src/commands/interaction/runtime/settle.ts"
- "src/utils/scroll-edge-state.ts"
- "src/selectors/**"
- "src/**/*.test.ts"
- "scripts/mutation/**"
- "scripts/lib/**"
- "stryker.config.json"
- "mutation-baselines/**"
- ".github/workflows/mutation-affected.yml"
permissions:
contents: read
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
select:
name: Select affected kernels
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
modules: ${{ steps.select.outputs.modules }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Ratchet self-test
run: pnpm mutation:test
- id: select
name: Derive the affected shard matrix
run: |
modules=$(pnpm --silent mutation:affected --list-affected \
--base "origin/${{ github.event.pull_request.base.ref }}" | tail -n1)
echo "modules=$modules" >> "$GITHUB_OUTPUT"
echo "Affected mutation shards: $modules" >> "$GITHUB_STEP_SUMMARY"
# The self-test and the derivation run before any mutant, so a failure here
# would leave the lane with no envelope at all (#1430).
- name: Record a failed lane envelope
if: failure()
run: |
pnpm mutation:run --affected --fail-envelope \
"affected selection failed before any mutant ran (run ${{ github.run_id }})" || true
- name: Upload selection envelope
if: failure()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-affected-select
path: .tmp/mutation/lane-envelope.json
if-no-files-found: warn
mutants:
name: Mutants (${{ matrix.name }})
needs: select
if: needs.select.outputs.modules != '[]'
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.select.outputs.modules) }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Run mutants for ${{ matrix.name }}
run: |
pnpm mutation:run --modules ${{ matrix.module }} \
${{ matrix.shard && format('--shard {0}', matrix.shard) || '' }}
- name: Record a failed shard envelope
if: failure()
run: |
pnpm mutation:run --affected --modules ${{ matrix.module }} --fail-envelope \
"shard ${{ matrix.name }} failed before producing a report (run ${{ github.run_id }})" || true
- name: Upload shard report
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-affected-shard-${{ matrix.name }}
path: |
.tmp/mutation/mutation.json
.tmp/mutation/mutation.html
.tmp/mutation/lane-envelope.json
if-no-files-found: warn
ratchet:
name: Affected decision-kernel mutants
needs: [select, mutants]
if: always() && needs.select.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Download shard reports
if: needs.select.outputs.modules != '[]'
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
pattern: mutation-affected-shard-*
path: .tmp/mutation/shards
# No shards means no affected kernel: run.ts reports "nothing to mutate"
# and still writes the envelope, so the lane is never silently absent.
- name: Ratchet the affected modules
run: |
if [ -d .tmp/mutation/shards ]; then
expected=$(echo '${{ needs.select.outputs.modules }}' | jq length)
pnpm mutation:check --report-dir .tmp/mutation/shards --affected \
--expect-shards "$expected" \
--base "origin/${{ github.event.pull_request.base.ref }}"
else
pnpm mutation:affected --base "origin/${{ github.event.pull_request.base.ref }}"
fi
- name: Record a failed lane envelope
if: failure()
run: |
pnpm mutation:run --affected --fail-envelope \
"affected ratchet failed before producing a verdict (run ${{ github.run_id }})" || true
- name: Upload mutation report
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-affected
path: |
.tmp/mutation/shards
.tmp/mutation/lane-envelope.json
if-no-files-found: ignore
.github/workflows/mutation-weekly.yml
+153
View File
@@ -0,0 +1,153 @@
name: Mutation Weekly
# Weekly mutation sweep over the enumerated decision kernels (issue #1415).
# Mutation score is the mechanical answer to "is this test load-bearing or
# decorative"; a full-suite sweep is unaffordable, so the scope is the kernel
# registry in scripts/mutation/modules.ts and nothing else.
#
# Sharded one job per module: the whole sweep is ~2,150 mutants, and the selector
# module alone is ~1,280 of them, so a single job would sit near the 30-minute
# acceptance budget on an ubuntu runner. The shards' JSON reports are merged into
# one verdict by the ratchet job (`--report-dir`), so the ratchet still sees a
# full sweep.
#
# The lane reports; it does not commit. The proposed baseline rides in the
# artifact and applying it is a reviewed `pnpm mutation:baseline` commit — a score
# can never lower itself. Gating (after two consecutive stable weekly runs) is
# enforced on PRs by mutation-affected.yml.
on:
schedule:
# Sundays 05:00 UTC — after the nightly lanes, before the working week.
- cron: "0 5 * * 0"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
shard:
name: Mutants (${{ matrix.name }})
runs-on: ubuntu-latest
# The acceptance budget is 30 minutes of wall clock for the lane, and shards
# run in parallel, so the budget is per shard. The observed rate on a 2-core
# runner is ~3s/mutant, which is why selectors (~1,280 mutants) is sliced.
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
# Kept in step with KERNEL_MODULES by the shard-coverage assertion in
# scripts/mutation/workflow.test.ts, which also pins --expect-shards to
# the number of entries here.
include:
- { name: kernel-errors, module: kernel-errors }
- { name: daemon-ref-frame, module: daemon-ref-frame }
- { name: interaction-settle, module: interaction-settle }
- { name: scroll-edge-state, module: scroll-edge-state }
- { name: selectors-1, module: selectors, shard: 1/4 }
- { name: selectors-2, module: selectors, shard: 2/4 }
- { name: selectors-3, module: selectors, shard: 3/4 }
- { name: selectors-4, module: selectors, shard: 4/4 }
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Run mutants for ${{ matrix.name }}
run: |
pnpm mutation:run --modules ${{ matrix.module }} \
${{ matrix.shard && format('--shard {0}', matrix.shard) || '' }}
# A shard can die before Stryker writes anything (install, config, crash);
# without this the artifact is absent and "shard failed" is indistinguishable
# from "lane never ran". --fail-envelope leaves a real verdict untouched.
- name: Record a failed shard envelope
if: failure()
run: |
pnpm mutation:run --modules ${{ matrix.module }} --fail-envelope \
"shard ${{ matrix.name }} failed before producing a report (run ${{ github.run_id }})" || true
- name: Upload shard report
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-shard-${{ matrix.name }}
path: |
.tmp/mutation/mutation.json
.tmp/mutation/lane-envelope.json
if-no-files-found: warn
ratchet:
name: Mutation ratchet
runs-on: ubuntu-latest
needs: shard
if: always()
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Ratchet self-test
run: pnpm mutation:test
- name: Download shard reports
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
pattern: mutation-shard-*
path: .tmp/mutation/shards
# run.ts writes the markdown verdict to $GITHUB_STEP_SUMMARY when the
# runner exports it, so the summary and the artifact carry the same numbers.
- name: Ratchet the merged sweep and propose the next baseline
run: |
pnpm mutation:check --report-dir .tmp/mutation/shards --expect-shards 8 --update
cp mutation-baselines/decision-kernels.json .tmp/mutation/proposed-baseline.json
git checkout -- mutation-baselines/decision-kernels.json
# The self-test and the artifact download both run before the ratchet, so a
# failure there would otherwise leave the aggregate lane with no envelope.
- name: Record a failed lane envelope
if: failure()
run: |
pnpm mutation:run --fail-envelope \
"weekly ratchet job failed before producing a verdict (run ${{ github.run_id }})" || true
# Freshness/drift telemetry (#1430): the envelope states commit, Stryker
# version, config hash, duration and result, so a lane going dark or a tool
# bump is visible without reading these logs.
- name: Lane envelope
if: always()
run: |
if [ ! -f .tmp/mutation/lane-envelope.json ]; then
echo 'No lane envelope was written — the job died before it could run node.' \
>> "$GITHUB_STEP_SUMMARY"
exit 0
fi
{
echo '<details><summary>Lane envelope (schema #1430)</summary>'
echo
echo '```json'
cat .tmp/mutation/lane-envelope.json
echo '```'
echo '</details>'
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload mutation report
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mutation-decision-kernels
path: |
.tmp/mutation/shards
.tmp/mutation/lane-envelope.json
.tmp/mutation/proposed-baseline.json
if-no-files-found: warn
docs/agents/testing.md
+59
View File
@@ -21,6 +21,7 @@ The mapping it encodes, for when you need to run a gate directly or reason about
| SkillGym prompts/assertions | `pnpm test:skillgym:case <case-id>` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) — agentic routing + local-help-consumption proof only; command-planning knowledge checks belong in the help bench |
| `.ad` grammar (`src/replay/script.ts`, gesture arity, replay vars) | `pnpm exec vitest run --project unit-core test/replay-compat` — the frozen replay-compat corpus asserts which released script surfaces still parse; a flipped verdict is edited in `test/replay-compat/manifest.ts`, never in the script. Adding or re-pinning a corpus entry also runs `pnpm check:replay-compat`, which re-derives each entry from its release tag in git history |
| Anything in `src/`, `test/`, `skills/` | `pnpm format` |
| A decision kernel or its tests (`src/kernel/errors.ts`, `src/daemon/ref-frame.ts`, `src/commands/interaction/runtime/settle.ts`, `src/utils/scroll-edge-state.ts`, `src/selectors/`) | `pnpm mutation:affected --base origin/main` (minutes; GitHub runs it per PR — see the mutation ratchet section) |
Two traps worth naming:
@@ -120,6 +121,64 @@ The plan documents the rule and changed path behind every selected check.
Model and catalog live under `scripts/check-affected/`; the derivation is guarded
by `pnpm check:affected:test` (the `Affected-check Selector` CI job).
## Mutation ratchet over decision kernels
Mutation score is the mechanical answer to "is this test load-bearing or decorative". A full-suite
sweep is unaffordable, so the scope is an enumerated list of pure decision kernels — modules where a
surviving mutant means a silently wrong agent-facing decision. The registry
(`scripts/mutation/modules.ts`) is the single source of truth: `stryker.config.json`'s `mutate` globs
are asserted against it, and PR-affected selection maps changed files through it. Modules that spawn
subprocesses or wait real time stay out by construction.
```sh
pnpm mutation:test # ratchet self-test (fast, no Stryker)
pnpm mutation:run --modules selectors # one module locally (~7 min for selectors)
pnpm mutation:check # ratchet an existing .tmp/mutation/mutation.json
pnpm mutation:baseline # full sweep, then record it (reviewed commit)
```
- **Weekly full sweep** (`.github/workflows/mutation-weekly.yml`) runs `shardMatrix()` from the
registry: one job per module, except modules that declare a `shards` count and are sliced with
`--shard i/n` (selectors is ~1,280 mutants, well past the 30-minute budget in one job). The ratchet
merges the shard reports (`--report-dir`) for one verdict and requires the full set
(`--expect-shards`), so a dead shard fails the lane instead of scoring its module as 0. Results are
reported as a job summary plus an artifact. It never commits: the proposed baseline rides in the
artifact, and applying it is a reviewed `pnpm mutation:baseline` commit, so a score cannot lower
itself.
- **PR lane** (`.github/workflows/mutation-affected.yml`) derives the affected shard matrix
(`--list-affected`) and merges the shards into one verdict. Before graduation the matrix is empty —
a report nobody acts on is not worth the runner minutes — unless the diff touches the lane's own
tooling, the one pre-graduation run that buys something: the gate has to be proven before it bites.
Lane sources own no kernel, so that exception adds `LANE_CANARY` (`kernel-errors`, the registry's
cheapest real sweep) to whatever the diff derives; otherwise it would select zero mutants and prove
nothing. `scripts/mutation/selection.test.ts` drives both halves of the rule through the real CLI
against a throwaway worktree commit.
- **Ratchet**: scores may only rise. `mutation-baselines/decision-kernels.json` records the
high-water score per module plus the Stryker version and config content hash that produced it, so a
score change caused by a tool/config change is reported as provenance drift, never as a
test-strength regression.
- **Graduation, not a flag day**: gating is off until two consecutive comparable weekly sweeps pass
(`stableRuns`/`requiredStableRuns` in the baseline); the PR job starts selecting modules — and
failing on them — once the committed baseline says `gating: true`. A regression or provenance drift
resets the counter.
- **Test scope** is derived from Vitest's module graph (`vitest related` over the mutated files), the
same delegation `pnpm check:affected` uses; see `scripts/mutation/test-scope.ts` for the three
groups it drops and why dropping them cannot hide a surviving mutant.
- **Test ownership is derived, never listed** (`scripts/mutation/ownership.ts`): a test owns every
kernel its imports reach, so `src/__tests__/daemon-error.test.ts` selects `kernel-errors` through
`src/daemon.ts` without naming it. A listed set of test files would silently omit exactly those
indirect tests and rot as tests are added — weakening one would skip the ratchet. Reaching a kernel
is a superset of killing its mutants, so the PR lane over-selects on purpose and shards the
selected modules; a false positive costs runner minutes, a false negative costs the gate. Non-kernel
*sources* are not owned: they can only move a score through those tests, and the weekly sweep
re-measures the whole surface.
- **Lane envelope** (`scripts/lib/lane-envelope.ts`, issue #1430): every run writes
`.tmp/mutation/lane-envelope.json` — schema version, commit, Stryker version, config hash, seed
(`null`; the input is enumerated, not randomized), duration, result, stage, per-module scores — and
both workflows upload it, so lane freshness and tool drift are readable without parsing logs. It is
written on every exit path, including a crash before any mutant runs: an absent envelope would be
indistinguishable from a lane that never ran.
## Live web smoke
The live web platform smoke runs the public built CLI against a local fixture page through the managed web backend:
mutation-baselines/decision-kernels.json
+53
View File
@@ -0,0 +1,53 @@
{
"schemaVersion": 1,
"requiredStableRuns": 2,
"stableRuns": 0,
"gating": false,
"modules": {
"kernel-errors": {
"score": 55.19,
"killed": 101,
"survived": 82,
"total": 183,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"daemon-ref-frame": {
"score": 100,
"killed": 55,
"survived": 0,
"total": 55,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"interaction-settle": {
"score": 68.95,
"killed": 151,
"survived": 68,
"total": 219,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"scroll-edge-state": {
"score": 28.75,
"killed": 92,
"survived": 228,
"total": 320,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"selectors": {
"score": 70.56,
"killed": 901,
"survived": 376,
"total": 1277,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
}
}
}
package.json
+9 -2
View File
@@ -108,9 +108,14 @@
"perf": "node --experimental-strip-types scripts/perf/run.ts",
"perf:ios": "node --experimental-strip-types scripts/perf/run.ts --platform ios",
"perf:android": "node --experimental-strip-types scripts/perf/run.ts --platform android",
"mutation:run": "node --experimental-strip-types scripts/mutation/run.ts",
"mutation:baseline": "node --experimental-strip-types scripts/mutation/run.ts --update",
"mutation:check": "node --experimental-strip-types scripts/mutation/run.ts --no-run",
"mutation:affected": "node --experimental-strip-types scripts/mutation/run.ts --affected",
"mutation:test": "node --experimental-strip-types --test scripts/mutation/*.test.ts",
"lint": "oxlint . --deny-warnings",
"format": "node ./node_modules/oxfmt/bin/oxfmt --write src test skills scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
"format:check": "node ./node_modules/oxfmt/bin/oxfmt --check src test skills scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
"format": "node ./node_modules/oxfmt/bin/oxfmt --write src test skills scripts/mutation scripts/lib scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts vitest.mutation.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
"format:check": "node ./node_modules/oxfmt/bin/oxfmt --check src test skills scripts/mutation scripts/lib scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts vitest.mutation.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
"fallow": "fallow audit --base origin/main",
"fallow:all": "fallow --summary",
"fallow:baseline": "(fallow dead-code --save-baseline fallow-baselines/dead-code.json --summary || true) && (fallow health --save-baseline fallow-baselines/health.json --summary || true)",
@@ -229,6 +234,8 @@
},
"devDependencies": {
"@chenglou/freerange": "^0.0.1",
"@stryker-mutator/core": "9.6.1",
"@stryker-mutator/vitest-runner": "9.6.1",
"@types/node": "^22.19.21",
"@vitest/coverage-v8": "4.1.8",
"fallow": "^2.95.0",
pnpm-lock.yaml
Generated
+1373 -1
View File
File diff suppressed because it is too large Load Diff
scripts/check-affected/run.ts
+6 -15
View File
@@ -8,8 +8,8 @@
import fs from 'node:fs';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { parseArgs as parseNodeArgs } from 'node:util';
import { runCmdStreaming, runCmdSync } from '../../src/utils/exec.ts';
import { parseScriptArgs } from '../lib/cli-args.ts';
import {
assertCatalogComplete,
CHECK_CATALOG,
@@ -26,21 +26,12 @@ const repoRoot = runCmdSync('git', ['rev-parse', '--show-toplevel']).stdout.trim
const USAGE = 'Usage: pnpm check:affected [--base <ref>] [--head <ref>] [--json] [--run]\n';
function parseArgs(argv: readonly string[]): Args {
const { values } = parseNodeArgs({
args: [...argv],
options: {
base: { type: 'string', default: 'origin/main' },
head: { type: 'string', default: 'HEAD' },
json: { type: 'boolean', default: false },
run: { type: 'boolean', default: false },
help: { type: 'boolean', short: 'h', default: false },
},
allowPositionals: false,
const values = parseScriptArgs(argv, USAGE, {
base: { type: 'string', default: 'origin/main' },
head: { type: 'string', default: 'HEAD' },
json: { type: 'boolean', default: false },
run: { type: 'boolean', default: false },
});
if (values.help) {
process.stdout.write(USAGE);
process.exit(0);
}
return {
base: values.base ?? 'origin/main',
head: values.head ?? 'HEAD',
scripts/lib/cli-args.ts
+26
View File
@@ -0,0 +1,26 @@
import { parseArgs, type ParseArgsConfig } from 'node:util';
type Options = NonNullable<ParseArgsConfig['options']>;
/**
* `parseArgs` plus the shared script `--help` contract: `-h`/`--help` prints the
* usage the caller owns and exits 0, so no script re-implements the flag.
*/
export function parseScriptArgs<T extends Options>(
argv: readonly string[],
usage: string,
options: T,
): ReturnType<typeof parseArgs<{ options: T & { help: { type: 'boolean' } } }>>['values'] {
const { values } = parseArgs({
args: [...argv],
options: { ...options, help: { type: 'boolean', short: 'h', default: false } },
allowPositionals: false,
});
if (values.help) {
process.stdout.write(usage);
process.exit(0);
}
return values as ReturnType<
typeof parseArgs<{ options: T & { help: { type: 'boolean' } } }>
>['values'];
}
scripts/lib/lane-envelope.ts
+69
View File
@@ -0,0 +1,69 @@
// Standard artifact envelope for scheduled lanes (issue #1430).
//
// A scheduled lane can go dark or drift for weeks while PR CI stays green, so
// every artifact it uploads must say — without a human reading logs — which
// commit produced it, which tool/config version measured it, how long it took,
// and whether it passed. Freshness monitoring reads `finishedAt`/`result`; drift
// analysis reads `tool`/`configHash`.
//
// Lanes that adopt the envelope later should import this module rather than
// re-deriving the field names.
export const LANE_ENVELOPE_SCHEMA_VERSION = 1;
export type LaneResult = 'pass' | 'fail';
export type LaneEnvelope<T> = {
schemaVersion: number;
/** Stable lane id — the metric key freshness/health jobs group by. */
lane: string;
commit: string;
ref: string | undefined;
runUrl: string | undefined;
/** Tool versions, or content hashes where a version is not enough. */
tool: Record<string, string>;
configHash: string;
/** Only lanes with randomized input record a seed; `null` states "not applicable". */
seed: string | null;
startedAt: string;
finishedAt: string;
durationMs: number;
result: LaneResult;
/** Lane-specific payload; kept separate so the envelope shape stays stable. */
data: T;
};
export type LaneEnvelopeInput<T> = {
lane: string;
commit: string;
tool: Record<string, string>;
configHash: string;
startedAtMs: number;
result: LaneResult;
data: T;
seed?: string | null;
now?: number;
};
export function laneEnvelope<T>(input: LaneEnvelopeInput<T>): LaneEnvelope<T> {
const finished = input.now ?? Date.now();
const { GITHUB_SERVER_URL, GITHUB_REPOSITORY, GITHUB_RUN_ID, GITHUB_REF_NAME } = process.env;
return {
schemaVersion: LANE_ENVELOPE_SCHEMA_VERSION,
lane: input.lane,
commit: input.commit,
ref: GITHUB_REF_NAME,
runUrl:
GITHUB_SERVER_URL && GITHUB_REPOSITORY && GITHUB_RUN_ID
? `${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}`
: undefined,
tool: input.tool,
configHash: input.configHash,
seed: input.seed ?? null,
startedAt: new Date(input.startedAtMs).toISOString(),
finishedAt: new Date(finished).toISOString(),
durationMs: Math.max(0, finished - input.startedAtMs),
result: input.result,
data: input.data,
};
}
scripts/mutation/config.test.ts
+41
View File
@@ -0,0 +1,41 @@
// The Stryker config and the module registry are two views of one scope. If
// they drift, the weekly sweep silently measures something other than the
// enumerated decision kernels — the exact failure this lane exists to catch.
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { test } from 'node:test';
import { fileURLToPath } from 'node:url';
import { mutateGlobs } from './modules.ts';
import { configHash } from './run.ts';
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..');
function readConfig(): Record<string, unknown> {
const text = fs.readFileSync(path.join(repoRoot, 'stryker.config.json'), 'utf8');
return JSON.parse(text) as Record<string, unknown>;
}
test('stryker mutate globs mirror the kernel-module registry', () => {
assert.deepEqual(
readConfig().mutate,
mutateGlobs(),
'stryker.config.json `mutate` drifted from KERNEL_MODULES — update scripts/mutation/modules.ts and the config together.',
);
});
test('stryker owns no pass/fail threshold — the ratchet does', () => {
const thresholds = readConfig().thresholds as { break?: number | null } | undefined;
assert.equal(
thresholds?.break ?? null,
null,
"A Stryker `break` threshold would fail runs on an absolute score; gating is the ratchet's job (scripts/mutation/ratchet.ts).",
);
});
test('the config content hash is stable and content-addressed', () => {
assert.equal(configHash('a'), configHash('a'));
assert.notEqual(configHash('a'), configHash('b'));
assert.match(configHash('a'), /^sha256:[0-9a-f]{12}$/);
});
scripts/mutation/envelope.test.ts
+276
View File
@@ -0,0 +1,276 @@
// The envelope is the only thing a downloaded scheduled-lane artifact can be
// interpreted from months later (#1430), so its required fields are asserted
// rather than assumed: a lane that stops emitting one of them makes freshness
// and tool-drift monitoring silently useless.
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { test } from 'node:test';
import { runCmdSync } from '../../src/utils/exec.ts';
import { laneEnvelope, LANE_ENVELOPE_SCHEMA_VERSION } from '../lib/lane-envelope.ts';
const repoRoot = path.resolve(import.meta.dirname, '../..');
function workflow(name: string): string {
return fs.readFileSync(path.join(repoRoot, '.github/workflows', name), 'utf8');
}
test('the envelope carries schema, commit, tool/config provenance, duration and result', () => {
const envelope = laneEnvelope({
lane: 'mutation-decision-kernels',
commit: 'a'.repeat(40),
tool: { stryker: '9.6.1' },
configHash: 'sha256:abcdef123456',
startedAtMs: 1_000,
now: 61_000,
result: 'pass',
data: { scope: 'full-sweep' },
});
assert.equal(envelope.schemaVersion, LANE_ENVELOPE_SCHEMA_VERSION);
assert.equal(envelope.lane, 'mutation-decision-kernels');
assert.equal(envelope.commit, 'a'.repeat(40));
assert.deepEqual(envelope.tool, { stryker: '9.6.1' });
assert.equal(envelope.configHash, 'sha256:abcdef123456');
// Mutation input is enumerated, not randomized: `null` is an explicit
// "not applicable", not a forgotten field.
assert.equal(envelope.seed, null);
assert.equal(envelope.durationMs, 60_000);
assert.equal(envelope.finishedAt, '1970-01-01T00:01:01.000Z');
assert.equal(envelope.result, 'pass');
assert.deepEqual(envelope.data, { scope: 'full-sweep' });
});
test('a failed ratchet is recorded as a failed lane run', () => {
const envelope = laneEnvelope({
lane: 'mutation-decision-kernels',
commit: 'b'.repeat(40),
tool: { stryker: '9.6.1' },
configHash: 'sha256:abcdef123456',
startedAtMs: 0,
now: 0,
result: 'fail',
data: {},
});
assert.equal(envelope.result, 'fail');
assert.equal(envelope.durationMs, 0);
});
// A lane that crashes before it can measure anything is the dark-lane case: an
// absent envelope is indistinguishable from a lane that never ran, so the run
// script must emit one from its failure path too.
test('a crashed run still writes an envelope naming the stage it died in', () => {
const envelopePath = path.join(repoRoot, '.tmp/mutation/lane-envelope.json');
fs.rmSync(envelopePath, { force: true });
const result = runCmdSync(
'node',
[
'--experimental-strip-types',
'scripts/mutation/run.ts',
'--report',
'.tmp/mutation/absent-report.json',
'--modules',
'kernel-errors',
],
{ cwd: repoRoot, allowFailure: true },
);
assert.notEqual(result.exitCode, 0, 'a missing report must fail the run');
assert.ok(fs.existsSync(envelopePath), 'no envelope written for a crashed run');
const envelope = JSON.parse(fs.readFileSync(envelopePath, 'utf8')) as {
result: string;
tool: Record<string, string>;
configHash: string;
data: { stage: string; error: string | null };
};
assert.equal(envelope.result, 'fail');
assert.equal(envelope.data.stage, 'report');
assert.match(envelope.data.error ?? '', /absent-report\.json/);
// Provenance is read before the work, so a crash still reports its tool/config.
assert.ok(envelope.tool.stryker);
assert.match(envelope.configHash, /^sha256:/);
});
// Shard artifacts carry the envelope next to the report, so merging "every JSON
// under the shard directory" fed the envelope to the report parser and crashed
// the ratchet job after the mutants had already run.
test('merging shard reports ignores the envelope sitting beside them', () => {
const shards = path.join(repoRoot, '.tmp/mutation/envelope-test-shards/shard-a');
fs.mkdirSync(shards, { recursive: true });
fs.writeFileSync(
path.join(shards, 'mutation.json'),
JSON.stringify({
files: {
'src/kernel/errors.ts': { mutants: [{ status: 'Killed' }, { status: 'Survived' }] },
},
}),
);
fs.writeFileSync(
path.join(shards, 'lane-envelope.json'),
JSON.stringify(
laneEnvelope({
lane: 'mutation-decision-kernels',
commit: 'c'.repeat(40),
tool: { stryker: '9.6.1' },
configHash: 'sha256:abcdef123456',
startedAtMs: 0,
now: 0,
result: 'pass',
data: {},
}),
),
);
const result = runCmdSync(
'node',
[
'--experimental-strip-types',
'scripts/mutation/run.ts',
'--report-dir',
'.tmp/mutation/envelope-test-shards',
'--modules',
'kernel-errors',
],
{ cwd: repoRoot, allowFailure: true },
);
assert.match(result.stdout, /merging 1 shard report\(s\)/);
assert.match(result.stdout, /kernel-errors/);
assert.doesNotMatch(result.stderr, /Cannot convert undefined or null to object/);
fs.rmSync(path.join(repoRoot, '.tmp/mutation/envelope-test-shards'), {
recursive: true,
force: true,
});
});
function runMutation(args: readonly string[]): {
exitCode: number;
stdout: string;
stderr: string;
} {
const result = runCmdSync(
'node',
['--experimental-strip-types', 'scripts/mutation/run.ts', ...args],
{
cwd: repoRoot,
allowFailure: true,
},
);
return { exitCode: result.exitCode ?? 1, stdout: result.stdout, stderr: result.stderr };
}
type Envelope = {
result: string;
data: { stage: string; error: string | null; modules: readonly { id: string }[] };
};
function readEnvelope(): Envelope {
return JSON.parse(
fs.readFileSync(path.join(repoRoot, '.tmp/mutation/lane-envelope.json'), 'utf8'),
) as Envelope;
}
// A merged shard set is only a sweep if every requested module actually reported.
// summarizeReport scores an absent module as 0, and while the lane is non-gating a
// 0 only *reports* a regression — so a dead matrix shard would otherwise be
// aggregated into a passing "complete" envelope claiming the sweep happened.
test('an incomplete shard set fails instead of scoring the missing module as zero', () => {
const shards = path.join(repoRoot, '.tmp/mutation/partial-shards/shard-kernel-errors');
fs.mkdirSync(shards, { recursive: true });
fs.writeFileSync(
path.join(shards, 'mutation.json'),
JSON.stringify({
files: { 'src/kernel/errors.ts': { mutants: [{ status: 'Killed' }] } },
}),
);
const result = runMutation([
'--report-dir',
'.tmp/mutation/partial-shards',
'--modules',
'kernel-errors,daemon-ref-frame',
]);
assert.notEqual(result.exitCode, 0, 'a missing shard must fail the aggregate');
assert.match(result.stderr, /Incomplete shard set/);
assert.match(result.stderr, /daemon-ref-frame/);
const envelope = readEnvelope();
assert.equal(envelope.result, 'fail');
assert.equal(envelope.data.stage, 'ratchet');
fs.rmSync(path.join(repoRoot, '.tmp/mutation/partial-shards'), { recursive: true, force: true });
});
// Argument parsing and the provenance/baseline reads used to sit outside the
// envelope boundary, so the lane could exit without declaring itself at all.
test('a malformed invocation still writes an envelope', () => {
fs.rmSync(path.join(repoRoot, '.tmp/mutation/lane-envelope.json'), { force: true });
const result = runMutation(['--modules', 'not-a-kernel']);
assert.notEqual(result.exitCode, 0);
const envelope = readEnvelope();
assert.equal(envelope.result, 'fail');
assert.equal(envelope.data.stage, 'setup');
assert.match(envelope.data.error ?? '', /Unknown mutation module/);
});
// The weekly self-test and the affected-selection job run before any mutant, so
// their failure has to be declared by the lane rather than only by the job log.
test('--fail-envelope declares a step that failed before the sweep', () => {
fs.rmSync(path.join(repoRoot, '.tmp/mutation/lane-envelope.json'), { force: true });
const result = runMutation(['--fail-envelope', 'self-test failed']);
assert.notEqual(result.exitCode, 0, 'a pre-run failure must not report success');
const envelope = readEnvelope();
assert.equal(envelope.result, 'fail');
assert.equal(envelope.data.stage, 'setup');
assert.equal(envelope.data.error, 'self-test failed');
});
// The workflows run it from `if: failure()`, which also fires when the ratchet
// itself failed — a generic reason must never displace the specific one.
test('--fail-envelope keeps a failure the run already reported', () => {
fs.rmSync(path.join(repoRoot, '.tmp/mutation/lane-envelope.json'), { force: true });
runMutation(['--fail-envelope', 'the real failure']);
runMutation(['--fail-envelope', 'a later generic failure']);
assert.equal(readEnvelope().data.error, 'the real failure');
});
// A pass is not a verdict worth preserving: the weekly job copies the proposed
// baseline and restores the committed one *after* the ratchet passed, so a failure
// there would otherwise publish the failed scheduled job as a passing lane.
test('--fail-envelope downgrades a passing envelope when a later step fails', () => {
const shards = path.join(repoRoot, '.tmp/mutation/pass-then-fail/shard-kernel-errors');
fs.mkdirSync(shards, { recursive: true });
// A perfect shard so the ratchet passes: the score can only rise from the
// committed kernel-errors baseline.
fs.writeFileSync(
path.join(shards, 'mutation.json'),
JSON.stringify({
files: { 'src/kernel/errors.ts': { mutants: [{ status: 'Killed' }, { status: 'Killed' }] } },
}),
);
const passing = runMutation([
'--report-dir',
'.tmp/mutation/pass-then-fail',
'--modules',
'kernel-errors',
]);
assert.equal(passing.exitCode, 0, passing.stderr);
assert.equal(readEnvelope().result, 'pass');
runMutation(['--fail-envelope', 'the baseline copy step failed']);
const envelope = readEnvelope();
assert.equal(envelope.result, 'fail', 'a failed job must not publish a passing envelope');
assert.equal(envelope.data.error, 'the baseline copy step failed');
fs.rmSync(path.join(repoRoot, '.tmp/mutation/pass-then-fail'), { recursive: true, force: true });
});
test('both mutation lanes record an envelope for failures before the sweep', () => {
for (const name of ['mutation-weekly.yml', 'mutation-affected.yml']) {
assert.match(workflow(name), /--fail-envelope/, `${name} can fail without an envelope`);
}
});
test('both mutation lanes publish the envelope', () => {
for (const name of ['mutation-weekly.yml', 'mutation-affected.yml']) {
assert.match(
workflow(name),
/\.tmp\/mutation\/lane-envelope\.json/,
`${name} does not upload the lane envelope`,
);
}
assert.match(workflow('mutation-weekly.yml'), /Lane envelope/);
});
scripts/mutation/modules.test.ts
+77
View File
@@ -0,0 +1,77 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { test } from 'node:test';
import { fileURLToPath } from 'node:url';
import {
affectedModules,
ALL_MODULE_IDS,
KERNEL_MODULES,
moduleForFile,
mutateGlobs,
shardMatrix,
} from './modules.ts';
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..');
test('every enumerated kernel path exists', () => {
for (const module of KERNEL_MODULES) {
for (const owned of module.owns) {
assert.ok(
fs.existsSync(path.join(repoRoot, owned)),
`${module.id} owns a path that no longer exists: ${owned}`,
);
}
}
});
test('changed sources map onto the module that owns them', () => {
assert.equal(moduleForFile('src/kernel/errors.ts'), 'kernel-errors');
assert.equal(moduleForFile('./src/daemon/ref-frame.ts'), 'daemon-ref-frame');
assert.equal(moduleForFile('src/selectors/parse.ts'), 'selectors');
// Selector tests live under the owned prefix; every other kernel's tests are
// attributed by ownership.ts, not by this path match.
assert.equal(moduleForFile('src/selectors/__tests__/resolve.test.ts'), 'selectors');
assert.equal(moduleForFile('src/kernel/rect.ts'), undefined);
assert.equal(moduleForFile('README.md'), undefined);
});
test('affected selection is deduplicated and registry-ordered', () => {
assert.deepEqual(
affectedModules([
'src/selectors/parse.ts',
'src/selectors/match.ts',
'src/kernel/errors.ts',
'docs/agents/testing.md',
]),
['kernel-errors', 'selectors'],
);
assert.deepEqual(affectedModules(['docs/agents/testing.md']), []);
});
test('mutate globs default to every module and narrow on request', () => {
assert.deepEqual(mutateGlobs(), mutateGlobs(ALL_MODULE_IDS));
assert.deepEqual(mutateGlobs(['kernel-errors']), ['src/kernel/errors.ts']);
});
// One job per module is only affordable while a module fits the lane's budget;
// the shard count is registry data so the workflows and the runner agree on it.
test('the shard matrix slices only the modules that declare shards', () => {
assert.deepEqual(shardMatrix(['kernel-errors']), [
{ name: 'kernel-errors', module: 'kernel-errors' },
]);
assert.deepEqual(shardMatrix(['selectors']), [
{ name: 'selectors-1', module: 'selectors', shard: '1/4' },
{ name: 'selectors-2', module: 'selectors', shard: '2/4' },
{ name: 'selectors-3', module: 'selectors', shard: '3/4' },
{ name: 'selectors-4', module: 'selectors', shard: '4/4' },
]);
assert.equal(shardMatrix().length, ALL_MODULE_IDS.length + 3);
// Every registry module reaches the matrix: an unsharded sweep is not a sweep.
for (const id of ALL_MODULE_IDS) {
assert.ok(
shardMatrix().some((spec) => spec.module === id),
`${id} has no shard`,
);
}
});
scripts/mutation/modules.ts
+153
View File
@@ -0,0 +1,153 @@
// Enumerated decision kernels the mutation lane measures (issue #1415).
//
// Mutation score is the only mechanical answer to "is this test load-bearing or
// decorative", but a full-suite sweep is unaffordable. This registry is the
// single source of truth for what Stryker mutates: `stryker.config.json`'s
// `mutate` globs are asserted against it, and PR-affected gating maps changed
// files onto modules through it.
//
// Membership rule: pure decision kernels only — a surviving mutant here means a
// silently wrong agent-facing decision. Anything that spawns subprocesses or
// waits real time is out of scope by construction (its mutants would be timeout
// noise, not test-strength signal).
export type ModuleId =
| 'kernel-errors'
| 'daemon-ref-frame'
| 'interaction-settle'
| 'scroll-edge-state'
| 'selectors';
export type KernelModule = {
readonly id: ModuleId;
readonly label: string;
/** Globs handed to Stryker's `mutate`. */
readonly mutate: readonly string[];
/**
* Paths this module owns; a trailing `/` marks a directory prefix. Sources
* only — the tests whose strength the score measures are derived from the
* import graph in `ownership.ts`, never listed here.
*/
readonly owns: readonly string[];
/**
* How many parallel jobs the module's mutants are sliced across. One job per
* module is the default; a module big enough to outrun the lane's 30-minute
* budget declares more (`--shard i/n` picks the slice).
*/
readonly shards?: number;
};
export const KERNEL_MODULES: readonly KernelModule[] = [
{
id: 'kernel-errors',
label: 'Error retriability + hints',
mutate: ['src/kernel/errors.ts'],
owns: ['src/kernel/errors.ts'],
},
{
id: 'daemon-ref-frame',
label: 'Ref-frame admission matrix (ADR 0014)',
mutate: ['src/daemon/ref-frame.ts'],
owns: ['src/daemon/ref-frame.ts'],
},
{
id: 'interaction-settle',
label: 'Interaction settle decisions',
mutate: ['src/commands/interaction/runtime/settle.ts'],
owns: ['src/commands/interaction/runtime/settle.ts'],
},
{
id: 'scroll-edge-state',
label: 'Scroll edge-state detection',
mutate: ['src/utils/scroll-edge-state.ts'],
owns: ['src/utils/scroll-edge-state.ts'],
},
{
id: 'selectors',
label: 'Selector parsing + matching',
mutate: ['src/selectors/**/*.ts', '!src/selectors/**/*.test.ts', '!src/selectors/__tests__/**'],
// Selector tests live under the owned directory, so the prefix covers them.
owns: ['src/selectors/'],
// ~1,280 mutants at the observed ~3s/mutant on a 2-core runner is ~64
// minutes in one job — past the acceptance budget and its own timeout.
shards: 4,
},
];
export const ALL_MODULE_IDS: readonly ModuleId[] = KERNEL_MODULES.map((module) => module.id);
export function moduleById(id: ModuleId): KernelModule {
const found = KERNEL_MODULES.find((module) => module.id === id);
if (!found) throw new Error(`Unknown mutation module: ${id}`);
return found;
}
export function isModuleId(value: string): value is ModuleId {
return ALL_MODULE_IDS.includes(value as ModuleId);
}
/** Mutate globs for a module subset, in registry order. */
export function mutateGlobs(ids: readonly ModuleId[] = ALL_MODULE_IDS): string[] {
return KERNEL_MODULES.filter((module) => ids.includes(module.id)).flatMap((module) => [
...module.mutate,
]);
}
/**
* The module a lane-tooling change proves itself against before graduation.
* `kernel-errors` is the cheapest real sweep in the registry (one file, ~183
* mutants), so a change to the ratchet, the config, or the baseline runs actual
* mutants end to end without paying for the full sweep.
*/
export const LANE_CANARY: ModuleId = 'kernel-errors';
/** One mutation job: a module, optionally one slice of it. */
export type ShardSpec = { name: string; module: ModuleId; shard?: string };
/**
* The jobs a module set expands into. Both workflows' matrices are this list, so
* a registry module (or a change to its shard count) can never leave the sweep
* without the workflow assertions noticing.
*/
export function shardMatrix(ids: readonly ModuleId[] = ALL_MODULE_IDS): ShardSpec[] {
return KERNEL_MODULES.filter((module) => ids.includes(module.id)).flatMap((module) => {
const count = module.shards ?? 1;
if (count === 1) return [{ name: module.id, module: module.id }];
return Array.from({ length: count }, (_unused, index) => ({
name: `${module.id}-${index + 1}`,
module: module.id,
shard: `${index + 1}/${count}`,
}));
});
}
export function normalizePath(filePath: string): string {
return filePath.replaceAll('\\', '/').replace(/^\.\//, '');
}
/**
* Which kernel module owns a repository-relative *source* path, if any.
*
* Test-file attribution is derived from the import graph — see
* `derivedAffectedModules` in `ownership.ts`, which is what the PR lane calls.
*/
export function moduleForFile(filePath: string): ModuleId | undefined {
const normalized = normalizePath(filePath);
for (const module of KERNEL_MODULES) {
for (const owned of module.owns) {
const match = owned.endsWith('/') ? normalized.startsWith(owned) : normalized === owned;
if (match) return module.id;
}
}
return undefined;
}
/** Kernel modules whose registry-owned paths a diff touches. */
export function affectedModules(changedFiles: readonly string[]): ModuleId[] {
const ids = new Set<ModuleId>();
for (const file of changedFiles) {
const id = moduleForFile(file);
if (id) ids.add(id);
}
return KERNEL_MODULES.filter((module) => ids.has(module.id)).map((module) => module.id);
}
scripts/mutation/ownership.test.ts
+90
View File
@@ -0,0 +1,90 @@
import assert from 'node:assert/strict';
import path from 'node:path';
import { test } from 'node:test';
import { fileURLToPath } from 'node:url';
import { KERNEL_MODULES } from './modules.ts';
import {
derivedAffectedModules,
isTestFile,
mutatedSources,
ownedTestFiles,
ownershipDeriver,
reachableFrom,
} from './ownership.ts';
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..');
test('a kernel is owned by the mirrored test that imports it directly', () => {
const deriver = ownershipDeriver(repoRoot);
assert.deepEqual(deriver.ownersOf('src/kernel/__tests__/errors.test.ts'), ['kernel-errors']);
assert.ok(
deriver.ownersOf('src/daemon/__tests__/ref-frame.test.ts').includes('daemon-ref-frame'),
);
assert.ok(
deriver
.ownersOf('src/commands/interaction/runtime/settle.test.ts')
.includes('interaction-settle'),
);
});
// The omission that hand-listed ownership could not see: this test asserts over
// normalizeError without importing src/kernel/errors.ts itself.
test('a kernel is owned by tests that reach it indirectly', () => {
const deriver = ownershipDeriver(repoRoot);
assert.ok(
deriver.ownersOf('src/__tests__/daemon-error.test.ts').includes('kernel-errors'),
'daemon-error.test.ts exercises normalizeError but does not own kernel-errors',
);
assert.ok(
deriver
.ownersOf('src/commands/interaction/runtime/gestures.test.ts')
.includes('scroll-edge-state'),
);
});
test('ownership is complete: every test reaching a kernel owns it', () => {
const owned = ownedTestFiles(repoRoot);
const cache = new Map<string, string[]>();
for (const module of KERNEL_MODULES) {
const sources = mutatedSources(module, repoRoot);
assert.ok(sources.length > 0, `${module.id} mutates nothing`);
const files = owned.get(module.id) ?? [];
assert.ok(
files.length > 0,
`${module.id} has no owning tests — its score cannot be attributed`,
);
for (const testFile of files) {
const reachable = reachableFrom(testFile, repoRoot, cache);
assert.ok(
sources.some((source) => reachable.has(source)),
`${testFile} owns ${module.id} without reaching it`,
);
}
// The converse: a diff to an owning test selects the module on a PR.
assert.ok(
derivedAffectedModules([files[0]!], repoRoot).includes(module.id),
`changing ${files[0]} does not select ${module.id}`,
);
}
});
test('non-kernel sources and non-tests are not owned', () => {
assert.deepEqual(derivedAffectedModules(['README.md', 'src/kernel/rect.ts'], repoRoot), []);
assert.ok(!isTestFile('src/kernel/errors.ts'));
assert.ok(!isTestFile('scripts/mutation/ownership.test.ts'));
assert.ok(isTestFile('src/kernel/__tests__/errors.test.ts'));
});
test('derived selection unions source ownership with test reachability', () => {
assert.deepEqual(
derivedAffectedModules(
[
'src/selectors/parse.ts',
'src/daemon/__tests__/ref-frame.test.ts',
'docs/agents/testing.md',
],
repoRoot,
).filter((id) => id === 'selectors' || id === 'daemon-ref-frame'),
['daemon-ref-frame', 'selectors'],
);
});
scripts/mutation/ownership.ts
+136
View File
@@ -0,0 +1,136 @@
// Which kernel module a changed file belongs to, DERIVED — never hand-listed.
//
// A mutation score is a statement about the tests that kill the mutants, so the
// PR lane must re-measure a kernel whenever one of *those* tests changes. An
// enumerated list of test files cannot state that: it silently omits tests that
// exercise a kernel indirectly (`src/__tests__/daemon-error.test.ts` reaches
// `normalizeError` through `src/daemon.ts`), and nothing fails when a new test
// is added. So ownership is computed from the static import graph instead: a test
// file owns every kernel module whose mutated sources it can reach.
//
// The derivation is deliberately a superset — reaching a kernel is cheaper to
// prove than killing its mutants, so an unrelated diff can select a module and
// pay for a report. False positives cost runner minutes; a false negative would
// let a weakened test slip past the ratchet, which is the thing the lane exists
// to catch.
//
// Non-test source changes outside the registry are NOT owned: they can only move
// a score through the tests that reach the kernel, and the weekly full sweep is
// what re-measures the whole surface. The PR lane's claim is narrower on purpose
// — kernel sources plus the tests that exercise them.
import fs from 'node:fs';
import path from 'node:path';
import { walkFiles } from '../lib/walk-files.ts';
import {
affectedModules,
KERNEL_MODULES,
normalizePath,
type ModuleId,
type KernelModule,
} from './modules.ts';
import { expandMutateFiles } from './test-scope.ts';
/** Test files the mutation lane can attribute to a kernel at all. */
export function isTestFile(filePath: string): boolean {
const normalized = normalizePath(filePath);
return normalized.startsWith('src/') && normalized.endsWith('.test.ts');
}
/** Repository-relative modules a file imports, following relative specifiers only. */
function importsOf(file: string, repoRoot: string, cache: Map<string, string[]>): string[] {
const cached = cache.get(file);
if (cached) return cached;
const absolute = path.join(repoRoot, file);
const text = fs.existsSync(absolute) ? fs.readFileSync(absolute, 'utf8') : '';
const specifiers = [...text.matchAll(/(?:from|import)\s*\(?\s*'(?<spec>\.[^']+)'/g)].map(
(match) => match.groups!.spec,
);
const resolved = [
...new Set(
specifiers.flatMap((specifier) => {
const base = path.posix.normalize(path.posix.join(path.posix.dirname(file), specifier));
return [base, `${base}.ts`, `${base}/index.ts`].filter((candidate) =>
fs.existsSync(path.join(repoRoot, candidate)),
);
}),
),
].filter((candidate) => candidate.endsWith('.ts'));
cache.set(file, resolved);
return resolved;
}
/** Every repository-relative module `file` reaches through the import graph. */
export function reachableFrom(
file: string,
repoRoot: string,
cache: Map<string, string[]> = new Map(),
): Set<string> {
const seen = new Set<string>();
const queue = [normalizePath(file)];
while (queue.length > 0) {
const current = queue.shift()!;
if (seen.has(current)) continue;
seen.add(current);
queue.push(...importsOf(current, repoRoot, cache));
}
return seen;
}
/** The concrete sources Stryker mutates for a module. */
export function mutatedSources(module: KernelModule, repoRoot: string): string[] {
return expandMutateFiles(module.mutate, repoRoot);
}
type Deriver = {
/** Kernel modules a single test file exercises, in registry order. */
ownersOf: (testFile: string) => ModuleId[];
};
/** A deriver with caches shared across files — one graph walk per module, not per query. */
export function ownershipDeriver(repoRoot: string): Deriver {
const importCache = new Map<string, string[]>();
const sources = KERNEL_MODULES.map((module) => ({
id: module.id,
sources: new Set(mutatedSources(module, repoRoot)),
}));
return {
ownersOf(testFile) {
const reachable = reachableFrom(testFile, repoRoot, importCache);
return sources
.filter((entry) => [...entry.sources].some((source) => reachable.has(source)))
.map((entry) => entry.id);
},
};
}
/**
* Kernel modules a diff affects: registry-owned paths plus every module the
* changed tests reach. Registry order, deduplicated.
*/
export function derivedAffectedModules(
changedFiles: readonly string[],
repoRoot: string,
): ModuleId[] {
const ids = new Set<ModuleId>(affectedModules(changedFiles));
const tests = changedFiles.filter(isTestFile).map(normalizePath);
if (tests.length > 0) {
const deriver = ownershipDeriver(repoRoot);
for (const testFile of tests) {
for (const id of deriver.ownersOf(testFile)) ids.add(id);
}
}
return KERNEL_MODULES.filter((module) => ids.has(module.id)).map((module) => module.id);
}
/** Every test file in the repository, per module that owns it — one graph walk. */
export function ownedTestFiles(repoRoot: string): Map<ModuleId, string[]> {
const deriver = ownershipDeriver(repoRoot);
const owned = new Map<ModuleId, string[]>(KERNEL_MODULES.map((module) => [module.id, []]));
for (const file of walkFiles(path.join(repoRoot, 'src'), (file) => file.endsWith('.test.ts'))) {
const relative = normalizePath(path.relative(repoRoot, file));
for (const id of deriver.ownersOf(relative)) owned.get(id)!.push(relative);
}
for (const files of owned.values()) files.sort();
return owned;
}
scripts/mutation/ratchet.test.ts
+240
View File
@@ -0,0 +1,240 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { test } from 'node:test';
import { ALL_MODULE_IDS } from './modules.ts';
import { renderReport } from './report.ts';
import {
applyRun,
BASELINE_SCHEMA_VERSION,
DEFAULT_REQUIRED_STABLE_RUNS,
emptyBaseline,
evaluateRatchet,
type Baseline,
type ModuleBaseline,
type Provenance,
} from './ratchet.ts';
import { summarizeReport, type StrykerReport } from './score.ts';
const PROVENANCE: Provenance = { strykerVersion: '9.6.1', configHash: 'sha256:abcdef123456' };
const NOW = '2026-07-27T00:00:00.000Z';
function mutants(statuses: readonly string[]): StrykerReport {
return {
files: {
'src/kernel/errors.ts': {
mutants: statuses.map((status, index) => ({
status,
mutatorName: 'ConditionalExpression',
location: { start: { line: index + 1 } },
})),
},
},
};
}
function baselineWith(entry: Partial<ModuleBaseline>): Baseline {
return {
...emptyBaseline(),
modules: {
'kernel-errors': {
score: 75,
killed: 3,
survived: 1,
total: 4,
strykerVersion: PROVENANCE.strykerVersion,
configHash: PROVENANCE.configHash,
updatedAt: NOW,
...entry,
},
},
};
}
test('scores count timeouts as killed and uncovered mutants as survived', () => {
const [score] = summarizeReport(mutants(['Killed', 'Timeout', 'Survived', 'NoCoverage']), [
'kernel-errors',
]);
assert.deepEqual(
{ score: score?.score, killed: score?.killed, survived: score?.survived, total: score?.total },
{ score: 50, killed: 2, survived: 2, total: 4 },
);
});
test('statuses outside the score (Ignored, CompileError) leave the denominator', () => {
const [score] = summarizeReport(mutants(['Killed', 'Ignored', 'CompileError']), [
'kernel-errors',
]);
assert.equal(score?.total, 1);
assert.equal(score?.score, 100);
});
test('a lowered score is a regression, and gating makes it fail with its survivors', () => {
const baseline = { ...baselineWith({ score: 100 }), stableRuns: 2, gating: true };
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.failed, true);
assert.deepEqual(
result.regressions.map((verdict) => verdict.module),
['kernel-errors'],
);
assert.match(result.regressions[0]?.detail ?? '', /fell 100% -> 50%/);
const markdown = renderReport(result, baseline, PROVENANCE);
assert.match(markdown, /Surviving mutants:/);
assert.match(markdown, /`src\/kernel\/errors\.ts:2` ConditionalExpression/);
assert.match(markdown, /scores may only rise/);
});
test('a lowered score is reported but does not fail while the lane is non-gating', () => {
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const baseline = baselineWith({ score: 100 });
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.regressions.length, 1);
assert.equal(result.failed, false);
assert.match(renderReport(result, baseline, PROVENANCE), /still non-gating/);
});
test('a regression never rewrites the baseline high-water mark', () => {
const baseline = baselineWith({ score: 100 });
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const next = applyRun(baseline, scores, evaluateRatchet(scores, baseline, PROVENANCE), {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(next.modules['kernel-errors']?.score, 100);
assert.equal(next.stableRuns, 0);
});
test('a risen score is recorded with its provenance', () => {
const baseline = baselineWith({ score: 50 });
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.verdicts[0]?.status, 'improved');
const next = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.deepEqual(next.modules['kernel-errors'], {
score: 100,
killed: 2,
survived: 0,
total: 2,
strykerVersion: PROVENANCE.strykerVersion,
configHash: PROVENANCE.configHash,
updatedAt: NOW,
});
});
test('gating graduates after the required consecutive stable full sweeps', () => {
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
let baseline = baselineWith({ score: 100 });
assert.equal(baseline.gating, false);
for (let run = 1; run <= baseline.requiredStableRuns; run += 1) {
const result = evaluateRatchet(scores, baseline, PROVENANCE);
baseline = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(baseline.stableRuns, run);
}
assert.equal(baseline.gating, true);
// A later regression resets graduation, so gating has to be re-earned.
const dropped = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const regressed = evaluateRatchet(dropped, baseline, PROVENANCE);
assert.equal(regressed.failed, true);
const after = applyRun(baseline, dropped, regressed, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(after.stableRuns, 0);
assert.equal(after.gating, false);
});
test('an affected PR run never advances graduation', () => {
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
const baseline = baselineWith({ score: 100 });
const next = applyRun(baseline, scores, evaluateRatchet(scores, baseline, PROVENANCE), {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: false,
});
assert.equal(next.stableRuns, 0);
});
test('a tool or config change is provenance drift, not a test-strength regression', () => {
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
for (const drift of [{ strykerVersion: '9.5.0' }, { configHash: 'sha256:000000000000' }]) {
const baseline = { ...baselineWith({ score: 100, ...drift }), stableRuns: 2, gating: true };
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.regressions.length, 0);
assert.equal(result.failed, false);
assert.equal(result.comparable, false);
assert.match(result.drifted[0]?.detail ?? '', /not attributable to test strength/);
// Drift rebases onto the new tool/config and costs the graduation streak.
const next = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(next.modules['kernel-errors']?.score, 50);
assert.equal(next.modules['kernel-errors']?.configHash, PROVENANCE.configHash);
assert.equal(next.stableRuns, 0);
}
});
test('a module with no baseline yet is new, recorded, and blocks graduation', () => {
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const baseline = emptyBaseline();
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.verdicts[0]?.status, 'new');
assert.equal(result.comparable, false);
const next = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(next.modules['kernel-errors']?.score, 50);
assert.equal(next.stableRuns, 0);
});
test('the report names the fix command and the graduation state', () => {
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
const baseline = { ...baselineWith({ score: 100 }), stableRuns: 1 };
const markdown = renderReport(
evaluateRatchet(scores, baseline, PROVENANCE),
baseline,
PROVENANCE,
);
assert.match(markdown, /gating \*\*off\*\* \(1\/2 stable weekly runs\)/);
assert.match(markdown, /Stryker `9\.6\.1` · config `sha256:abcdef123456`/);
});
test('the committed baseline is the shape the lane graduates from', () => {
const baseline = JSON.parse(
fs.readFileSync(
path.resolve(import.meta.dirname, '../../mutation-baselines/decision-kernels.json'),
'utf8',
),
) as Baseline;
assert.equal(baseline.schemaVersion, BASELINE_SCHEMA_VERSION);
// Issue #1415's N: gating is earned by two consecutive stable weekly sweeps.
assert.equal(baseline.requiredStableRuns, DEFAULT_REQUIRED_STABLE_RUNS);
assert.deepEqual(Object.keys(baseline.modules), [...ALL_MODULE_IDS]);
for (const [id, module] of Object.entries(baseline.modules)) {
assert.ok(module.total > 0, `${id} has no mutants`);
assert.ok(module.strykerVersion.length > 0, `${id} has no Stryker version`);
assert.match(module.configHash, /^sha256:[0-9a-f]{12}$/);
}
});
scripts/mutation/ratchet.ts
+208
View File
@@ -0,0 +1,208 @@
// The ratchet itself: scores may only rise, gating is earned, and a score that
// moved for tool/config reasons is never mistaken for test-strength change.
//
// Graduation (issue #1415): the lane starts non-gating. Each comparable weekly
// full sweep with no regression increments `stableRuns`; once it reaches
// `requiredStableRuns` the baseline flips to `gating: true`, after which a
// regression fails the run — weekly as a full sweep, PRs for AFFECTED modules
// only. A regression or a non-comparable run resets the counter to zero.
import { roundScore, type ModuleScore, type SurvivingMutant } from './score.ts';
import type { ModuleId } from './modules.ts';
export const BASELINE_SCHEMA_VERSION = 1;
export const DEFAULT_REQUIRED_STABLE_RUNS = 2;
export type ModuleBaseline = {
score: number;
killed: number;
survived: number;
total: number;
// Provenance travels with every module baseline so a tool or config change is
// distinguishable from a test-strength change.
strykerVersion: string;
configHash: string;
updatedAt: string;
};
export type Baseline = {
schemaVersion: number;
requiredStableRuns: number;
stableRuns: number;
gating: boolean;
modules: Record<string, ModuleBaseline>;
};
export type Provenance = { readonly strykerVersion: string; readonly configHash: string };
export type VerdictStatus = 'new' | 'improved' | 'held' | 'regressed' | 'provenance-drift';
export type ModuleVerdict = {
readonly module: ModuleId;
readonly score: number;
readonly killed: number;
readonly total: number;
readonly baselineScore: number | undefined;
readonly delta: number | undefined;
readonly status: VerdictStatus;
readonly surviving: readonly SurvivingMutant[];
readonly detail: string;
};
export type RatchetResult = {
readonly verdicts: readonly ModuleVerdict[];
readonly regressions: readonly ModuleVerdict[];
readonly drifted: readonly ModuleVerdict[];
/** A run is comparable when every module has a same-provenance baseline. */
readonly comparable: boolean;
readonly gating: boolean;
/** True when the caller must exit non-zero. */
readonly failed: boolean;
};
export function emptyBaseline(requiredStableRuns: number = DEFAULT_REQUIRED_STABLE_RUNS): Baseline {
return {
schemaVersion: BASELINE_SCHEMA_VERSION,
requiredStableRuns,
stableRuns: 0,
gating: false,
modules: {},
};
}
function driftDetail(previous: ModuleBaseline, provenance: Provenance): string {
return (
`baseline recorded by stryker ${previous.strykerVersion} / config ${previous.configHash}, ` +
`this run used stryker ${provenance.strykerVersion} / config ${provenance.configHash}; ` +
'the score change is not attributable to test strength — re-record with `pnpm mutation:baseline`'
);
}
function verdictFor(
score: ModuleScore,
previous: ModuleBaseline | undefined,
provenance: Provenance,
): ModuleVerdict {
const base = {
module: score.module,
score: score.score,
killed: score.killed,
total: score.total,
surviving: score.surviving,
};
if (!previous) {
return {
...base,
baselineScore: undefined,
delta: undefined,
status: 'new',
detail: `no baseline recorded yet (measured ${score.score}%)`,
};
}
const delta = roundScore(score.score - previous.score);
if (
previous.strykerVersion !== provenance.strykerVersion ||
previous.configHash !== provenance.configHash
) {
return {
...base,
baselineScore: previous.score,
delta,
status: 'provenance-drift',
detail: driftDetail(previous, provenance),
};
}
if (delta < 0) {
return {
...base,
baselineScore: previous.score,
delta,
status: 'regressed',
detail:
`mutation score fell ${previous.score}% -> ${score.score}% ` +
`(${score.survived} surviving mutants)`,
};
}
return {
...base,
baselineScore: previous.score,
delta,
status: delta > 0 ? 'improved' : 'held',
detail:
delta > 0
? `mutation score rose ${previous.score}% -> ${score.score}%`
: `mutation score held at ${score.score}%`,
};
}
export function evaluateRatchet(
scores: readonly ModuleScore[],
baseline: Baseline,
provenance: Provenance,
): RatchetResult {
const verdicts = scores.map((score) =>
verdictFor(score, baseline.modules[score.module], provenance),
);
const regressions = verdicts.filter((verdict) => verdict.status === 'regressed');
const drifted = verdicts.filter((verdict) => verdict.status === 'provenance-drift');
const comparable = drifted.length === 0 && verdicts.every((v) => v.status !== 'new');
return {
verdicts,
regressions,
drifted,
comparable,
gating: baseline.gating,
failed: baseline.gating && regressions.length > 0,
};
}
export type ApplyOptions = {
readonly provenance: Provenance;
readonly now: string;
/** Only the weekly full sweep may advance graduation; affected runs may not. */
readonly countsTowardGraduation: boolean;
};
/**
* Fold a run into the baseline: keep the high-water score per module (a
* regression never rewrites it, so the ratchet keeps failing until the tests are
* restored), rebase modules whose provenance drifted onto the new tool/config,
* then advance or reset graduation.
*/
export function applyRun(
baseline: Baseline,
scores: readonly ModuleScore[],
result: RatchetResult,
options: ApplyOptions,
): Baseline {
const modules: Record<string, ModuleBaseline> = { ...baseline.modules };
for (const score of scores) {
const previous = modules[score.module];
const status = result.verdicts.find((verdict) => verdict.module === score.module)?.status;
const rebase = !previous || status === 'provenance-drift';
if (!rebase && score.score < previous.score) continue;
modules[score.module] = {
score: rebase ? score.score : Math.max(previous.score, score.score),
killed: score.killed,
survived: score.survived,
total: score.total,
strykerVersion: options.provenance.strykerVersion,
configHash: options.provenance.configHash,
updatedAt: options.now,
};
}
const stable = result.comparable && result.regressions.length === 0;
const stableRuns = !options.countsTowardGraduation
? baseline.stableRuns
: stable
? baseline.stableRuns + 1
: 0;
return {
...baseline,
schemaVersion: BASELINE_SCHEMA_VERSION,
stableRuns,
gating: stableRuns >= baseline.requiredStableRuns,
modules,
};
}
scripts/mutation/report.ts
+81
View File
@@ -0,0 +1,81 @@
// Markdown rendering for the mutation lane: GitHub job summary and terminal
// output share one renderer, so the artifact and the console never disagree.
import { moduleById } from './modules.ts';
import type { Baseline, ModuleVerdict, Provenance, RatchetResult } from './ratchet.ts';
const DEFAULT_MAX_SURVIVING_LISTED = 20;
function formatDelta(delta: number | undefined): string {
if (delta === undefined) return '—';
return delta > 0 ? `+${delta}` : String(delta);
}
function renderRow(verdict: ModuleVerdict): string {
const module = moduleById(verdict.module);
const baseline = verdict.baselineScore === undefined ? '—' : `${verdict.baselineScore}%`;
return (
`| \`${verdict.module}\` — ${module.label} | ${verdict.score}% | ${baseline} | ` +
`${formatDelta(verdict.delta)} | ${verdict.killed}/${verdict.total} | ` +
`${verdict.surviving.length} | ${verdict.status} |`
);
}
function renderDetail(verdict: ModuleVerdict, maxListed: number): string[] {
const lines = ['', `### \`${verdict.module}\` — ${verdict.status}`, '', verdict.detail];
if (verdict.surviving.length > 0) {
lines.push('', 'Surviving mutants:', '');
for (const mutant of verdict.surviving.slice(0, maxListed)) {
lines.push(`- \`${mutant.file}:${mutant.line}\` ${mutant.mutator}`);
}
if (verdict.surviving.length > maxListed) {
lines.push(`- …and ${verdict.surviving.length - maxListed} more`);
}
}
return lines;
}
export type RenderOptions = {
readonly title?: string;
readonly maxSurvivingListed?: number;
};
export function renderReport(
result: RatchetResult,
baseline: Baseline,
provenance: Provenance,
options: RenderOptions = {},
): string {
const maxListed = options.maxSurvivingListed ?? DEFAULT_MAX_SURVIVING_LISTED;
const lines: string[] = [
`## ${options.title ?? 'Mutation score — decision kernels'}`,
'',
`Stryker \`${provenance.strykerVersion}\` · config \`${provenance.configHash}\` · ` +
`gating **${baseline.gating ? 'on' : 'off'}** ` +
`(${baseline.stableRuns}/${baseline.requiredStableRuns} stable weekly runs)`,
'',
'| Module | Score | Baseline | Δ | Killed/Total | Surviving | Status |',
'| --- | --- | --- | --- | --- | --- | --- |',
...result.verdicts.map(renderRow),
];
for (const verdict of [...result.regressions, ...result.drifted]) {
lines.push(...renderDetail(verdict, maxListed));
}
if (result.failed) {
lines.push(
'',
'Mutation ratchet failed: scores may only rise. Kill the surviving mutants listed above, ' +
'then re-run `pnpm mutation:run` (full sweep) or `pnpm mutation:check --report <file>` ' +
'against an existing Stryker report.',
);
} else if (result.regressions.length > 0) {
lines.push(
'',
'Scores regressed while the lane is still non-gating — no failure recorded, but the ' +
'surviving mutants above are the tests to strengthen before gating turns on.',
);
}
return `${lines.join('\n')}\n`;
}
scripts/mutation/run.ts
+559
View File
@@ -0,0 +1,559 @@
// Entrypoint for the decision-kernel mutation lane (issue #1415).
//
// pnpm mutation:run full sweep + ratchet check
// pnpm mutation:baseline full sweep, then record the scores
// pnpm mutation:check --report <file> ratchet an existing Stryker report
// pnpm mutation:affected --base origin/main
// PR lane: mutate only the kernel
// modules the diff touches
//
// The weekly workflow runs the full sweep and writes the rendered report to the
// job summary plus an artifact; the PR lane runs the affected subset and only
// fails once the baseline has graduated to `gating: true`.
import crypto from 'node:crypto';
import fs from 'node:fs';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { runCmdStreaming, runCmdSync } from '../../src/utils/exec.ts';
import { parseScriptArgs } from '../lib/cli-args.ts';
import { laneEnvelope } from '../lib/lane-envelope.ts';
import {
ALL_MODULE_IDS,
isModuleId,
LANE_CANARY,
mutateGlobs,
normalizePath,
shardMatrix,
type ModuleId,
type ShardSpec,
} from './modules.ts';
import { derivedAffectedModules } from './ownership.ts';
import { renderReport } from './report.ts';
import {
applyRun,
emptyBaseline,
evaluateRatchet,
type Baseline,
type Provenance,
type RatchetResult,
} from './ratchet.ts';
import { mergeReports, summarizeReport, type ModuleScore, type StrykerReport } from './score.ts';
import {
expandMutateFiles,
relatedTestFiles,
TEST_SCOPE_ENV,
writeTestScope,
} from './test-scope.ts';
const repoRoot = runCmdSync('git', ['rev-parse', '--show-toplevel']).stdout.trim();
const CONFIG_PATH = 'stryker.config.json';
const BASELINE_PATH = 'mutation-baselines/decision-kernels.json';
const DEFAULT_REPORT_PATH = '.tmp/mutation/mutation.json';
const TEST_SCOPE_PATH = '.tmp/mutation/test-scope.json';
const ENVELOPE_PATH = '.tmp/mutation/lane-envelope.json';
const LANE_ID = 'mutation-decision-kernels';
const USAGE = `Usage: pnpm mutation:run [options]
--modules <a,b> Restrict to specific kernel modules (default: all)
--affected Restrict to the modules touched between --base and HEAD
--base <ref> Base ref for --affected (default: origin/main)
--report <file> Read an existing Stryker JSON report instead of running Stryker
--report-dir <d> Merge every *.json Stryker report under <d> (weekly shards)
--expect-shards <n>
Fail unless --report-dir holds exactly n shard reports
--shard <i/n> Mutate only the i-th of n balanced slices of the module's
sources (the big modules exceed one job's budget)
--update Record the run into the baseline (ratchet + graduation)
--summary <file> Also write the markdown report to <file>
--no-run Alias for --report with the default report path
--list-affected Print the PR lane's shard matrix as JSON and exit (empty
until the baseline graduates, unless the diff touches the
lane's own tooling)
--fail-envelope <reason>
Write a failed lane envelope for a step that ran before (or
instead of) the sweep, e.g. a self-test failure
`;
type Args = {
modules: readonly ModuleId[];
affected: boolean;
base: string;
report: string | undefined;
reportDir: string | undefined;
update: boolean;
summary: string | undefined;
listAffected: boolean;
failEnvelope: string | undefined;
shard: Shard | undefined;
expectShards: number | undefined;
};
/** One-based slice of a module's mutated sources: `--shard 2/4`. */
type Shard = { index: number; count: number };
function parseShard(value: string | undefined): Shard | undefined {
if (!value) return undefined;
const match = /^(\d+)\/(\d+)$/.exec(value.trim());
const index = Number(match?.[1]);
const count = Number(match?.[2]);
if (!match || index < 1 || index > count) {
throw new Error(`--shard expects i/n with 1 <= i <= n, got "${value}"`);
}
return { index, count };
}
function parseModules(value: string | undefined): readonly ModuleId[] {
if (!value) return ALL_MODULE_IDS;
const ids = value
.split(',')
.map((entry) => entry.trim())
.filter(Boolean);
const unknown = ids.filter((id) => !isModuleId(id));
if (unknown.length > 0) {
throw new Error(
`Unknown mutation module(s): ${unknown.join(', ')}. Known: ${ALL_MODULE_IDS.join(', ')}`,
);
}
return ids.filter(isModuleId);
}
function parseMutationArgs(argv: readonly string[]): Args {
const values = parseScriptArgs(argv, USAGE, {
modules: { type: 'string' },
affected: { type: 'boolean', default: false },
base: { type: 'string', default: 'origin/main' },
report: { type: 'string' },
'report-dir': { type: 'string' },
update: { type: 'boolean', default: false },
summary: { type: 'string' },
'no-run': { type: 'boolean', default: false },
'list-affected': { type: 'boolean', default: false },
'fail-envelope': { type: 'string' },
shard: { type: 'string' },
'expect-shards': { type: 'string' },
});
return {
modules: parseModules(values.modules),
affected: Boolean(values.affected),
base: values.base ?? 'origin/main',
report: values.report ?? (values['no-run'] ? DEFAULT_REPORT_PATH : undefined),
reportDir: values['report-dir'],
update: Boolean(values.update),
summary: values.summary,
listAffected: Boolean(values['list-affected']),
failEnvelope: values['fail-envelope'],
shard: parseShard(values.shard),
expectShards: values['expect-shards'] ? Number(values['expect-shards']) : undefined,
};
}
/** Short, stable content hash of the Stryker config — half of a run's provenance. */
export function configHash(configText: string): string {
return `sha256:${crypto.createHash('sha256').update(configText).digest('hex').slice(0, 12)}`;
}
function readProvenance(root: string = repoRoot): Provenance {
const pkgPath = path.join(root, 'node_modules/@stryker-mutator/core/package.json');
const version = fs.existsSync(pkgPath)
? (JSON.parse(fs.readFileSync(pkgPath, 'utf8')) as { version: string }).version
: 'unknown';
return {
strykerVersion: version,
configHash: configHash(fs.readFileSync(path.join(root, CONFIG_PATH), 'utf8')),
};
}
function readBaseline(root: string = repoRoot): Baseline {
const file = path.join(root, BASELINE_PATH);
if (!fs.existsSync(file)) return emptyBaseline();
return JSON.parse(fs.readFileSync(file, 'utf8')) as Baseline;
}
function writeBaseline(baseline: Baseline, root: string = repoRoot): void {
const file = path.join(root, BASELINE_PATH);
fs.mkdirSync(path.dirname(file), { recursive: true });
fs.writeFileSync(file, `${JSON.stringify(baseline, null, 2)}\n`);
}
function changedFiles(base: string): string[] {
const result = runCmdSync('git', ['diff', '--name-only', '--merge-base', base, 'HEAD'], {
cwd: repoRoot,
allowFailure: true,
});
return result.stdout.split('\n').filter(Boolean);
}
// The JSON report path comes from the config (Stryker's CLI takes no nested
// reporter options), so a run always writes DEFAULT_REPORT_PATH.
/**
* Balances a module's sources across `count` jobs. Mutant count tracks file size
* closely enough that greedy longest-first packing keeps the slowest slice near
* the mean — the alternative is one 1,277-mutant selectors job that outruns both
* the 30-minute acceptance budget and its own timeout.
*/
function shardFiles(files: readonly string[], shard: Shard, root: string): string[] {
const bins: { size: number; files: string[] }[] = Array.from({ length: shard.count }, () => ({
size: 0,
files: [],
}));
const weighed = files
.map((file) => ({ file, size: fs.statSync(path.join(root, file)).size }))
.sort((a, b) => b.size - a.size || a.file.localeCompare(b.file));
for (const { file, size } of weighed) {
const bin = bins.reduce((smallest, next) => (next.size < smallest.size ? next : smallest));
bin.files.push(file);
bin.size += size;
}
return bins[shard.index - 1]!.files.sort();
}
async function runStryker(
modules: readonly ModuleId[],
reportPath: string,
shard: Shard | undefined,
): Promise<void> {
const absolute = path.isAbsolute(reportPath) ? reportPath : path.join(repoRoot, reportPath);
fs.mkdirSync(path.dirname(absolute), { recursive: true });
fs.rmSync(absolute, { force: true });
// The suite Stryker replays per mutant is derived from Vitest's module graph
// over the mutated files, not hand-listed; see scripts/mutation/test-scope.ts.
const globs = mutateGlobs(modules);
const all = expandMutateFiles(globs, repoRoot);
// A sharded job mutates concrete files, so the slice is exact rather than a
// glob the next contributor has to keep in step with the registry.
const mutate = shard ? shardFiles(all, shard, repoRoot) : globs;
const scopePath = path.join(repoRoot, TEST_SCOPE_PATH);
const testFiles = relatedTestFiles(shard ? mutate : all, repoRoot);
writeTestScope(testFiles, scopePath);
process.stdout.write(
`mutation: ${modules.join(', ')}${shard ? ` shard ${shard.index}/${shard.count}` : ''} -> ` +
`${shard ? mutate.length : all.length} source(s), ${testFiles.length} related test file(s)\n`,
);
// Stryker's `--mutate` takes one comma-separated value, not repeated args.
const args = ['exec', 'stryker', 'run', CONFIG_PATH, '--mutate', mutate.join(',')];
const result = await runCmdStreaming('pnpm', args, {
cwd: repoRoot,
allowFailure: true,
env: { ...process.env, [TEST_SCOPE_ENV]: scopePath },
onStdoutChunk: (chunk) => void process.stdout.write(chunk),
onStderrChunk: (chunk) => void process.stderr.write(chunk),
});
// Stryker exits non-zero on a low score too; the ratchet — not Stryker's own
// thresholds — owns the verdict, so only a missing report is fatal here.
if (!fs.existsSync(absolute)) {
throw new Error(
`Stryker produced no report at ${reportPath} (exit ${result.exitCode}). See output above.`,
);
}
}
function emit(markdown: string, summaryPath: string | undefined): void {
process.stdout.write(`\n${markdown}`);
const targets = [summaryPath, process.env.GITHUB_STEP_SUMMARY].filter(
(target): target is string => Boolean(target),
);
for (const target of targets) {
fs.mkdirSync(path.dirname(path.resolve(target)), { recursive: true });
fs.appendFileSync(target, markdown);
}
}
/**
* How far the lane got. A run that dies in `stryker` or `report` is exactly the
* failure the freshness monitor must see, so the stage rides in the envelope
* rather than only in the job log.
*/
type Stage = 'setup' | 'select' | 'stryker' | 'report' | 'ratchet' | 'complete';
/**
* Provenance for a lane that died before it could read any: `unknown` is a
* reported fact, whereas skipping the envelope would be silence.
*/
const UNKNOWN_PROVENANCE: Provenance = { strykerVersion: 'unknown', configHash: 'unknown' };
type LaneState = {
stage: Stage;
provenance: Provenance;
baseline: Baseline;
modules: readonly ModuleId[];
affected: boolean;
scores: readonly ModuleScore[];
result: RatchetResult | undefined;
error: string | undefined;
/**
* `--fail-envelope` keeps an existing *failure* (its reason is the specific
* one) but replaces an existing pass: a post-verdict step can fail after the
* ratchet passed, and publishing that job as passing is the bug the envelope
* exists to prevent.
*/
recoveryOnly: boolean;
};
/** The result of an envelope already on disk, if there is a readable one. */
function existingResult(file: string): string | undefined {
if (!fs.existsSync(file)) return undefined;
try {
return (JSON.parse(fs.readFileSync(file, 'utf8')) as { result?: string }).result;
} catch {
return undefined;
}
}
/**
* Scheduled-lane artifact envelope (#1430). Without it a downloaded report cannot
* say which commit or Stryker/config version produced it, how long the sweep took,
* or whether it passed — freshness and tool-drift monitoring would have to parse
* logs.
*
* Written on every exit path, including a crashed setup or a Stryker run that
* produced no report: a lane that fails before it can measure anything is the
* dark-lane case, and an absent envelope is indistinguishable from a lane that
* never ran.
*/
function writeEnvelope(state: LaneState, startedAtMs: number): void {
const target = path.join(repoRoot, ENVELOPE_PATH);
if (state.recoveryOnly && existingResult(target) === 'fail') {
process.stdout.write(`mutation: ${ENVELOPE_PATH} already reports a failure; left as is.\n`);
return;
}
const envelope = laneEnvelope({
lane: LANE_ID,
commit: runCmdSync('git', ['rev-parse', 'HEAD'], {
cwd: repoRoot,
allowFailure: true,
}).stdout.trim(),
tool: { stryker: state.provenance.strykerVersion },
configHash: state.provenance.configHash,
startedAtMs,
result: state.stage === 'complete' && !state.result?.failed ? 'pass' : 'fail',
data: {
scope: state.affected ? 'affected' : 'full-sweep',
stage: state.stage,
error: state.error ?? null,
modules: state.modules.map((id) => {
const score = state.scores.find((entry) => entry.module === id);
return {
id,
score: score?.score ?? null,
killed: score?.killed ?? null,
total: score?.total ?? null,
status: state.result?.verdicts.find((verdict) => verdict.module === id)?.status ?? null,
};
}),
gating: state.baseline.gating,
stableRuns: state.baseline.stableRuns,
requiredStableRuns: state.baseline.requiredStableRuns,
},
});
fs.mkdirSync(path.dirname(target), { recursive: true });
fs.writeFileSync(target, `${JSON.stringify(envelope, null, 2)}\n`);
process.stdout.write(
`\nLane envelope (${ENVELOPE_PATH}): ${envelope.result} at stage ${state.stage} in ` +
`${Math.round(envelope.durationMs / 1000)}s at ${envelope.commit.slice(0, 12)}.\n`,
);
}
function readReport(file: string): StrykerReport {
const absolute = path.isAbsolute(file) ? file : path.join(repoRoot, file);
return JSON.parse(fs.readFileSync(absolute, 'utf8')) as StrykerReport;
}
function readShardedReports(dir: string, expected: number | undefined): StrykerReport {
const root = path.isAbsolute(dir) ? dir : path.join(repoRoot, dir);
// Shard artifacts also carry the lane envelope and the derived test scope, so
// the report is selected by name rather than by "every .json here".
const files = fs
.globSync(`**/${path.basename(DEFAULT_REPORT_PATH)}`, { cwd: root })
.map((file) => path.join(root, file))
.sort();
if (files.length === 0) throw new Error(`No Stryker JSON reports found under ${dir}`);
// Sub-sharded modules make "every module has mutants" too weak on its own: the
// surviving slices would still cover the module, so the expected shard count is
// asserted as well.
if (expected !== undefined && files.length !== expected) {
throw new Error(
`Incomplete shard set from ${dir}: ${files.length} report(s), expected ${expected}. ` +
'A shard job failed or its artifact is absent; the aggregate is not a sweep.',
);
}
process.stdout.write(`mutation: merging ${files.length} shard report(s) from ${dir}\n`);
return mergeReports(files.map(readReport));
}
async function produceReport(
modules: readonly ModuleId[],
reportPath: string | undefined,
shard: Shard | undefined,
): Promise<StrykerReport> {
if (!reportPath) await runStryker(modules, DEFAULT_REPORT_PATH, shard);
return readReport(reportPath ?? DEFAULT_REPORT_PATH);
}
function recordRun(args: Args, state: LaneState, result: RatchetResult): void {
const next = applyRun(state.baseline, state.scores, result, {
provenance: state.provenance,
now: new Date().toISOString(),
// Only the full sweep proves stability; an affected subset says nothing
// about the modules it skipped.
countsTowardGraduation: !args.affected && state.modules.length === ALL_MODULE_IDS.length,
});
writeBaseline(next);
process.stdout.write(
`\nBaseline updated (${BASELINE_PATH}): ${next.stableRuns}/${next.requiredStableRuns} ` +
`stable runs, gating ${next.gating ? 'on' : 'off'}.\n`,
);
}
/**
* A merged shard set must cover every requested module. `summarizeReport` scores
* a module with no mutants as 0, and while the lane is non-gating a 0 is only
* *reported* as a regression — so a matrix shard that died would otherwise be
* aggregated into a `complete`/`pass` envelope claiming the sweep happened.
*/
function assertShardsCoverModules(state: LaneState, dir: string): void {
const missing = state.scores.filter((score) => score.total === 0).map((score) => score.module);
if (missing.length === 0) return;
throw new Error(
`Incomplete shard set from ${dir}: no mutants for ${missing.join(', ')}. ` +
'A shard job failed or its artifact is absent; the aggregate is not a sweep.',
);
}
/** Reads shard reports, an existing report, or runs Stryker — and scores them. */
async function scoreModules(args: Args, state: LaneState): Promise<void> {
state.stage = args.reportDir || args.report ? 'report' : 'stryker';
const report = args.reportDir
? readShardedReports(args.reportDir, args.expectShards)
: await produceReport(state.modules, args.report, args.shard);
state.stage = 'ratchet';
state.scores = summarizeReport(report, state.modules);
if (args.reportDir) assertShardsCoverModules(state, args.reportDir);
}
async function sweep(args: Args, state: LaneState): Promise<number> {
state.stage = 'select';
// Test attribution is derived from the import graph, not a listed set of test
// files: see scripts/mutation/ownership.ts.
// Same selection the PR matrix uses, graduation rule included, so the ratchet
// job can never run mutants the `select` job decided not to spend.
if (args.affected) {
state.modules = [...new Set(affectedMatrix(args.base).map((entry) => entry.module))];
}
if (state.modules.length === 0) {
process.stdout.write('mutation: no decision-kernel modules affected — nothing to mutate.\n');
state.stage = 'complete';
return 0;
}
await scoreModules(args, state);
const result = evaluateRatchet(state.scores, state.baseline, state.provenance);
state.result = result;
const title = args.affected
? 'Mutation score — affected decision kernels'
: 'Mutation score — decision kernels';
emit(renderReport(result, state.baseline, state.provenance, { title }), args.summary);
if (args.update) recordRun(args, state, result);
state.stage = 'complete';
return result.failed ? 1 : 0;
}
/** Sources of the lane itself: a change here must prove itself on real mutants. */
const LANE_TOOLING = ['scripts/mutation/', 'scripts/lib/', 'stryker.config.json', 'mutation-'];
/**
* The PR lane's matrix. Before graduation the affected run is a report nobody
* acts on, so it costs runner minutes for no verdict: it stays empty until the
* baseline reaches `gating: true`. The exception is a diff that changes the lane
* itself — that is the one case where the pre-graduation run buys something,
* because the gate has to be proven before it can bite.
*/
export function affectedMatrixFor(
changed: readonly string[],
gating: boolean,
root: string = repoRoot,
): ShardSpec[] {
const touchesLane = changed.some((file) =>
LANE_TOOLING.some((prefix) => normalizePath(file).startsWith(prefix)),
);
if (!gating && !touchesLane) return [];
const modules = new Set(derivedAffectedModules(changed, root));
// Lane sources own no kernel, so a tooling-only diff derives nothing: without
// the canary the "prove the gate" exception would select zero mutants and
// prove nothing.
if (touchesLane) modules.add(LANE_CANARY);
return shardMatrix(ALL_MODULE_IDS.filter((id) => modules.has(id)));
}
function affectedMatrix(base: string): ShardSpec[] {
return affectedMatrixFor(changedFiles(base), readBaseline().gating);
}
/**
* Everything after `--help` runs inside the envelope boundary, argument parsing
* and provenance reads included: a lane that cannot even read its own config is
* the failure freshness monitoring most needs to see.
*/
async function run(argv: readonly string[], state: LaneState): Promise<number> {
const args = parseMutationArgs(argv);
state.affected = args.affected;
state.provenance = readProvenance();
state.baseline = readBaseline();
state.modules = args.modules;
if (args.failEnvelope) {
// A step that ran before the sweep failed (the weekly self-test, setup): the
// lane produced no measurement, and that is what the envelope must say.
state.error = args.failEnvelope;
state.recoveryOnly = true;
return 1;
}
return await sweep(args, state);
}
async function main(argv = process.argv.slice(2)): Promise<number> {
const startedAtMs = Date.now();
if (argv.includes('--list-affected')) {
// Selection only — no lane run, so no envelope. The shard jobs that consume
// this matrix each write their own.
const args = parseMutationArgs(argv);
process.stdout.write(`${JSON.stringify(affectedMatrix(args.base))}\n`);
return 0;
}
const state: LaneState = {
stage: 'setup',
provenance: UNKNOWN_PROVENANCE,
baseline: emptyBaseline(),
modules: [],
affected: false,
scores: [],
result: undefined,
error: undefined,
recoveryOnly: false,
};
try {
return await run(argv, state);
} catch (error: unknown) {
state.error = error instanceof Error ? error.message : String(error);
process.stderr.write(`mutation: ${state.error}\n`);
return 1;
} finally {
writeEnvelope(state, startedAtMs);
}
}
if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) {
main()
.then((code) => {
process.exitCode = code;
})
.catch((error: unknown) => {
process.stderr.write(`mutation: ${error instanceof Error ? error.message : String(error)}\n`);
process.exitCode = 1;
});
}
scripts/mutation/score.ts
+103
View File
@@ -0,0 +1,103 @@
// Per-module mutation scores derived from a Stryker JSON report.
import { ALL_MODULE_IDS, moduleForFile, normalizePath, type ModuleId } from './modules.ts';
/** The subset of Stryker's JSON report schema this lane reads. */
export type StrykerMutant = {
readonly mutatorName?: string;
readonly status: string;
readonly location?: { readonly start?: { readonly line?: number } };
};
export type StrykerReport = {
readonly files: Record<string, { readonly mutants: readonly StrykerMutant[] }>;
};
export type SurvivingMutant = {
readonly file: string;
readonly line: number;
readonly mutator: string;
};
export type ModuleScore = {
readonly module: ModuleId;
readonly score: number;
readonly killed: number;
readonly survived: number;
readonly total: number;
readonly surviving: readonly SurvivingMutant[];
};
// Stryker counts a timeout as killed. `NoCoverage` counts as survived here: an
// uncovered mutant is precisely the "decorative test" signal this lane exists to
// surface. Every other status (Ignored, CompileError, RuntimeError) leaves the
// denominator, so tool-side noise cannot move the score.
const KILLED_STATUSES = new Set(['Killed', 'Timeout']);
const SURVIVED_STATUSES = new Set(['Survived', 'NoCoverage']);
/**
* Merge sharded Stryker reports into one. The weekly sweep runs one shard per
* kernel module so no single job approaches its time budget; the ratchet still
* evaluates a single full-sweep report.
*/
export function mergeReports(reports: readonly StrykerReport[]): StrykerReport {
const files: Record<string, { mutants: StrykerMutant[] }> = {};
for (const report of reports) {
for (const [file, entry] of Object.entries(report.files)) {
const existing = files[file];
if (existing) existing.mutants.push(...entry.mutants);
else files[file] = { mutants: [...entry.mutants] };
}
}
return { files };
}
export function roundScore(value: number): number {
return Math.round(value * 100) / 100;
}
function compareMutants(a: SurvivingMutant, b: SurvivingMutant): number {
return a.file.localeCompare(b.file) || a.line - b.line || a.mutator.localeCompare(b.mutator);
}
export function summarizeReport(
report: StrykerReport,
ids: readonly ModuleId[] = ALL_MODULE_IDS,
): ModuleScore[] {
const buckets = new Map<
ModuleId,
{ killed: number; survived: number; surviving: SurvivingMutant[] }
>();
for (const id of ids) buckets.set(id, { killed: 0, survived: 0, surviving: [] });
for (const [file, entry] of Object.entries(report.files)) {
const id = moduleForFile(file);
if (!id) continue;
const bucket = buckets.get(id);
if (!bucket) continue;
for (const mutant of entry.mutants) {
if (KILLED_STATUSES.has(mutant.status)) {
bucket.killed += 1;
} else if (SURVIVED_STATUSES.has(mutant.status)) {
bucket.survived += 1;
bucket.surviving.push({
file: normalizePath(file),
line: mutant.location?.start?.line ?? 0,
mutator: mutant.mutatorName ?? 'unknown',
});
}
}
}
return [...buckets].map(([module, bucket]) => {
const total = bucket.killed + bucket.survived;
return {
module,
score: total === 0 ? 0 : roundScore((bucket.killed / total) * 100),
killed: bucket.killed,
survived: bucket.survived,
total,
surviving: bucket.surviving.sort(compareMutants),
};
});
}
scripts/mutation/selection.test.ts
+92
View File
@@ -0,0 +1,92 @@
// Selection is what the PR lane spends money on, so both halves of the rule are
// asserted end to end through the real CLI: nothing runs before graduation, and
// the "prove the gate" exception for a lane-tooling diff selects real mutants
// rather than an empty matrix that proves nothing.
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { after, test } from 'node:test';
import { runCmdSync } from '../../src/utils/exec.ts';
import { LANE_CANARY, shardMatrix, type ShardSpec } from './modules.ts';
import { affectedMatrixFor } from './run.ts';
const repoRoot = path.resolve(import.meta.dirname, '../..');
const worktrees: string[] = [];
after(() => {
for (const dir of worktrees) {
runCmdSync('git', ['worktree', 'remove', '--force', dir], { cwd: repoRoot });
}
});
/**
* A throwaway worktree holding one commit that touches only the given files, so
* `--list-affected` runs against a real `git diff` rather than a stubbed list.
*/
function worktreeWithCommit(name: string, files: readonly string[]): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), `mutation-${name}-`));
fs.rmSync(dir, { recursive: true });
runCmdSync('git', ['worktree', 'add', '--detach', '--quiet', dir, 'HEAD'], { cwd: repoRoot });
worktrees.push(dir);
for (const file of files) {
fs.appendFileSync(path.join(dir, file), '\n');
}
runCmdSync('git', ['add', ...files], { cwd: dir });
// CI runners have no committer identity configured, and this commit is a
// fixture, so it carries its own rather than depending on the environment.
runCmdSync(
'git',
[
'-c',
'user.name=mutation-selection-test',
'-c',
'user.email=mutation-selection-test@invalid',
'commit',
'--quiet',
'--no-verify',
'-m',
`touch ${name}`,
],
{ cwd: dir },
);
return dir;
}
function listAffected(cwd: string): ShardSpec[] {
const result = runCmdSync(
'node',
[
'--experimental-strip-types',
path.join(repoRoot, 'scripts/mutation/run.ts'),
'--list-affected',
'--base',
'HEAD~1',
],
{ cwd },
);
assert.equal(result.exitCode, 0, result.stderr);
return JSON.parse(result.stdout.trim().split('\n').at(-1)!) as ShardSpec[];
}
test('a lane-tooling diff selects real mutants even before graduation', () => {
const dir = worktreeWithCommit('tooling', ['scripts/mutation/ratchet.ts']);
// The lane's own sources own no kernel, so derivation alone yields nothing:
// without the canary this exception would run zero mutants.
assert.deepEqual(listAffected(dir), shardMatrix([LANE_CANARY]));
});
test('a kernel diff selects nothing until the baseline graduates', () => {
const dir = worktreeWithCommit('kernel', ['src/utils/scroll-edge-state.ts']);
assert.deepEqual(listAffected(dir), []);
// …and the same diff selects that module once gating is on.
assert.deepEqual(
affectedMatrixFor(['src/utils/scroll-edge-state.ts'], true),
shardMatrix(['scroll-edge-state']),
);
});
test('a docs-only diff selects nothing even once gating is on', () => {
assert.deepEqual(affectedMatrixFor(['docs/agents/testing.md'], true), []);
});
scripts/mutation/test-scope.test.ts
+28
View File
@@ -0,0 +1,28 @@
import assert from 'node:assert/strict';
import path from 'node:path';
import { test } from 'node:test';
import { expandMutateFiles, threadHostileTestFiles } from './test-scope.ts';
import { mutateGlobs } from './modules.ts';
const repoRoot = path.resolve(import.meta.dirname, '../..');
test('mutate globs expand to real kernel sources and exclude selector tests', () => {
const files = expandMutateFiles(mutateGlobs(), repoRoot);
assert.ok(files.includes('src/kernel/errors.ts'));
assert.ok(files.some((file) => file.startsWith('src/selectors/')));
assert.deepEqual(
files.filter((file) => file.endsWith('.test.ts')),
[],
);
});
test('the thread-hostile exclusion is derived from imports, not listed', () => {
const files = threadHostileTestFiles(repoRoot);
// Both reasons are represented: process.chdir (CLI capture harness) and a
// worker inside a worker (the node:worker_threads PNG pipeline).
assert.ok(files.includes('src/__tests__/cli-help.test.ts'));
assert.ok(files.includes('src/utils/__tests__/png-worker.test.ts'));
// A pure decision-kernel test is never excluded — that would hide survivors.
assert.ok(!files.includes('src/daemon/__tests__/ref-frame.test.ts'));
assert.ok(files.every((file) => file.endsWith('.test.ts')));
});
scripts/mutation/test-scope.ts
+135
View File
@@ -0,0 +1,135 @@
// Which tests the mutation lane runs, derived from Vitest's module graph.
//
// Stryker replays the configured suite for every mutant, so the suite it is
// pointed at decides whether the weekly sweep fits its 30-minute budget. Pointing
// it at the whole unit suite (487 files) makes the initial dry run alone cost
// minutes; hand-listing per-kernel test files would be a second source of truth
// that silently rots. So the scope is derived the same way `pnpm check:affected`
// derives affected tests: `vitest related` over the mutated files, i.e. Vitest's
// own static module graph.
//
// Two files are removed from whatever Vitest returns:
// - the subprocess-stub group (it spawns stubbed binaries and waits real
// subprocess/retry/poll time — out of scope by the issue's constraint, and
// thousands of mutant runs would turn it into timeout noise);
// - tests that cannot run in the thread pool Stryker's vitest runner forces:
// the in-process CLI-capture tests (`process.chdir` throws in a worker
// thread) and the `node:worker_threads` PNG pipeline tests (a worker inside
// a worker raises uncaught MessagePort errors that kill the runner);
// - anything outside `src/`: the unit suite also hosts the help-conformance
// gates from `scripts/__tests__`, which assert over the repo's own registries
// rather than over any decision kernel and own their CI job.
//
// Nothing here weakens the ratchet: a mutant only an excluded test could kill
// shows up as a survivor — visible work, never a silent pass.
import fs from 'node:fs';
import path from 'node:path';
import { runCmdSync } from '../../src/utils/exec.ts';
import { walkFiles } from '../lib/walk-files.ts';
import { normalizePath } from './modules.ts';
/** Env var carrying the resolved scope file to `vitest.mutation.config.ts`. */
export const TEST_SCOPE_ENV = 'AGENT_DEVICE_MUTATION_TEST_FILES';
const CLI_CAPTURE_HARNESS = 'src/__tests__/cli-capture.ts';
/** Source modules that own a `node:worker_threads` worker. */
function workerThreadModules(repoRoot: string): string[] {
return walkFiles(
path.join(repoRoot, 'src'),
(file) => file.endsWith('.ts') && !file.endsWith('.test.ts'),
)
.filter((file) => fs.readFileSync(file, 'utf8').includes('node:worker_threads'))
.map((file) => path.basename(file, '.ts'));
}
/**
* Repo-relative test files that cannot survive Stryker's thread pool, derived
* from what they import rather than listed: the chdir-using CLI capture harness
* and any module that itself starts a worker thread.
*/
export function threadHostileTestFiles(repoRoot: string): string[] {
const modules = [path.basename(CLI_CAPTURE_HARNESS, '.ts'), ...workerThreadModules(repoRoot)];
const importsHostileModule = new RegExp(`from '[^']*/(${modules.join('|')})(\\.ts)?'`);
return walkFiles(path.join(repoRoot, 'src'), (file) => file.endsWith('.test.ts'))
.filter((file) => importsHostileModule.test(fs.readFileSync(file, 'utf8')))
.map((file) => normalizePath(path.relative(repoRoot, file)))
.sort();
}
/**
* Concrete source files behind a module's mutate globs. Stryker's `!`-prefixed
* exclusions are applied here too — `fs.globSync` has no notion of them, and
* dropping them would feed selector *tests* into the scope derivation.
*/
export function expandMutateFiles(globs: readonly string[], repoRoot: string): string[] {
const negated = globs.filter((glob) => glob.startsWith('!')).map((glob) => glob.slice(1));
const excluded = new Set(fs.globSync(negated, { cwd: repoRoot }).map(normalizePath));
return fs
.globSync(
globs.filter((glob) => !glob.startsWith('!')),
{ cwd: repoRoot },
)
.map(normalizePath)
.filter((file) => !excluded.has(file))
.sort();
}
type VitestJsonReport = { testResults?: readonly { name: string }[] };
/**
* Test files Vitest considers related to `sourceFiles`, minus the groups this
* lane cannot run. `vitest related` executes them once (seconds), which also
* proves the scope is green before Stryker's dry run depends on it.
*/
export function relatedTestFiles(
sourceFiles: readonly string[],
repoRoot: string,
excluded: readonly string[] = threadHostileTestFiles(repoRoot),
): string[] {
const reportFile = path.join(repoRoot, '.tmp/mutation/related-tests.json');
fs.mkdirSync(path.dirname(reportFile), { recursive: true });
fs.rmSync(reportFile, { force: true });
runCmdSync(
'pnpm',
[
'exec',
'vitest',
'related',
...sourceFiles,
'--project',
'unit-core',
'--run',
'--reporter=json',
`--outputFile=${reportFile}`,
],
{ cwd: repoRoot, allowFailure: true },
);
if (!fs.existsSync(reportFile)) {
throw new Error(
`vitest related produced no report at ${reportFile} — cannot derive the mutation test scope.`,
);
}
const report = JSON.parse(fs.readFileSync(reportFile, 'utf8')) as VitestJsonReport;
const excludedSet = new Set(excluded);
return [
...new Set(
(report.testResults ?? [])
.map((result) => normalizePath(path.relative(repoRoot, result.name)))
.filter((file) => file.startsWith('src/') && !excludedSet.has(file)),
),
].sort();
}
/** Read the scope Stryker was handed, or `undefined` for "whole unit suite". */
export function readTestScope(): string[] | undefined {
const file = process.env[TEST_SCOPE_ENV];
if (!file || !fs.existsSync(file)) return undefined;
const files = JSON.parse(fs.readFileSync(file, 'utf8')) as string[];
return files.length > 0 ? files : undefined;
}
export function writeTestScope(files: readonly string[], file: string): void {
fs.mkdirSync(path.dirname(file), { recursive: true });
fs.writeFileSync(file, `${JSON.stringify(files, null, 2)}\n`);
}
scripts/mutation/workflow.test.ts
+86
View File
@@ -0,0 +1,86 @@
// The workflows' YAML cannot read the kernel registry, so these assertions keep
// the two in step: a module added to KERNEL_MODULES that no weekly shard runs
// would silently drop out of the sweep, and one no PR path filter selects would
// silently stop gating once the ratchet graduates.
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { test } from 'node:test';
import { KERNEL_MODULES, shardMatrix } from './modules.ts';
const repoRoot = path.resolve(import.meta.dirname, '../..');
function workflow(name: string): string {
return fs.readFileSync(path.join(repoRoot, '.github/workflows', name), 'utf8');
}
test('the weekly sweep shards exactly the registry matrix', () => {
const yaml = workflow('mutation-weekly.yml');
const jobs = [...yaml.matchAll(/^ {10}- \{ (?<entry>[^}]+) \}$/gm)].map((match) =>
Object.fromEntries(
match
.groups!.entry.split(', ')
.map((pair) => pair.split(': ') as [string, string])
.map(([key, value]) => [key, value]),
),
);
assert.deepEqual(
jobs,
shardMatrix().map((spec) =>
spec.shard ? { ...spec } : { name: spec.name, module: spec.module },
),
);
});
test('the weekly sweep merges the shards into one ratcheted verdict', () => {
const yaml = workflow('mutation-weekly.yml');
assert.match(yaml, /pnpm mutation:check --report-dir/);
assert.match(yaml, /GITHUB_STEP_SUMMARY|\$GITHUB_STEP_SUMMARY/);
// A dead shard must not be merged into a verdict that looks like a sweep.
assert.match(
yaml,
new RegExp(`--expect-shards ${shardMatrix().length}\\b`),
'the weekly ratchet does not require the full shard set',
);
});
// A shard that outruns the job timeout reports nothing, so the per-shard budget
// is the acceptance criterion made mechanical.
test('no mutation shard is allowed to exceed the 30-minute budget', () => {
for (const name of ['mutation-weekly.yml', 'mutation-affected.yml']) {
for (const [, minutes] of workflow(name).matchAll(/timeout-minutes: (\d+)/g)) {
assert.ok(Number(minutes) <= 30, `${name} declares a ${minutes}-minute job`);
}
}
});
test('every kernel path a PR can touch selects the affected mutation job', () => {
const paths = [...workflow('mutation-affected.yml').matchAll(/^ {6}- "(?<glob>[^"]+)"$/gm)].map(
(match) => match.groups!.glob,
);
for (const module of KERNEL_MODULES) {
for (const owned of module.owns) {
const selected = paths.some(
(glob) =>
glob === owned ||
glob === `${owned}**` ||
(glob.endsWith('/**') && owned.startsWith(glob.slice(0, -2))),
);
assert.ok(selected, `no path filter selects ${owned} (module ${module.id})`);
}
}
// Ownership is derived, so any test in src/ can own a kernel; the filter must
// let all of them through and leave the decision to the `select` job. A
// narrower filter is exactly the omission the derivation exists to prevent.
assert.ok(
paths.includes('src/**/*.test.ts'),
'the PR lane must trigger on every src test, since test ownership is derived',
);
assert.match(workflow('mutation-affected.yml'), /mutation:affected --list-affected/);
// The lane's own sources fail open into it too: a ratchet or baseline edit must
// prove itself against real mutants, not against a stale report.
for (const own of ['scripts/mutation/**', 'stryker.config.json', 'mutation-baselines/**']) {
assert.ok(paths.includes(own), `missing path filter ${own}`);
}
});
src/utils/__tests__/errors.test.ts → src/kernel/__tests__/errors.test.ts
+1 -1
View File
@@ -1,6 +1,6 @@
import { test } from 'vitest';
import assert from 'node:assert/strict';
import { AppError, normalizeError, toAppErrorCode } from '../../kernel/errors.ts';
import { AppError, normalizeError, toAppErrorCode } from '../errors.ts';
test('normalizeError adds default hint and strips diagnostic metadata from details', () => {
const err = new AppError('COMMAND_FAILED', 'runner failed', {
stryker.config.json
+41
View File
@@ -0,0 +1,41 @@
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"_comment": "Decision-kernel mutation lane (issue #1415). The mutate list below MUST mirror KERNEL_MODULES in scripts/mutation/modules.ts — scripts/mutation/config.test.ts asserts it. Ratchet verdicts come from scripts/mutation/run.ts, never from Stryker's own thresholds. Any edit here changes the config content hash, which invalidates every recorded baseline as non-comparable until re-recorded with `pnpm mutation:baseline`.",
"packageManager": "pnpm",
"tsconfigFile_comment": "Points at a path that does not exist on purpose. Stryker's sandbox rewrites tsconfig `extends`/`references` through the classic TypeScript API, which the repo's TypeScript 7 native package no longer exposes (`ts.parseConfigFileTextToJson is not a function`). Our tsconfig has neither extends nor references, so there is nothing to rewrite — skipping the preprocessor is exact, not a workaround for a real dependency. The behaviour relied on is that the preprocessor no-ops when the file is absent: see packages/core/src/sandbox/ts-config-preprocessor.ts (https://github.com/stryker-mutator/stryker-js/blob/master/packages/core/src/sandbox/ts-config-preprocessor.ts) and https://stryker-mutator.io/docs/stryker-js/configuration/#tsconfigfile-string. Re-check it on every Stryker upgrade: if a future version errors on a missing tsconfig instead, point this at a real, extends-free tsconfig.",
"tsconfigFile": "tsconfig.stryker-absent.json",
"plugins_comment": "Named explicitly: the default `@stryker-mutator/*` glob does not resolve under pnpm's non-flat node_modules, so plugin discovery finds no test runner.",
"plugins": ["@stryker-mutator/vitest-runner"],
"testRunner": "vitest",
"vitest": {
"configFile": "vitest.mutation.config.ts",
"related": true
},
"reporters": ["progress", "clear-text", "json"],
"jsonReporter": {
"fileName": ".tmp/mutation/mutation.json"
},
"htmlReporter": {
"fileName": ".tmp/mutation/mutation.html"
},
"coverageAnalysis": "perTest",
"ignoreStatic": true,
"tempDirName": ".tmp/stryker",
"cleanTempDir": true,
"timeoutMS": 20000,
"timeoutFactor": 2,
"thresholds": {
"high": 90,
"low": 70,
"break": null
},
"mutate": [
"src/kernel/errors.ts",
"src/daemon/ref-frame.ts",
"src/commands/interaction/runtime/settle.ts",
"src/utils/scroll-edge-state.ts",
"src/selectors/**/*.ts",
"!src/selectors/**/*.test.ts",
"!src/selectors/__tests__/**"
]
}
vitest.config.ts
+1 -1
View File
@@ -10,7 +10,7 @@ import slowTestGateReporter from './scripts/vitest-slow-test-reporter.ts';
// docs/agents/testing.md "tests must not wait real time"). Serialized below with
// per-file isolation so only one such file spawns stubs at a time, the same
// execution contract the pre-split android index.test.ts aggregation provided.
const SUBPROCESS_STUB_TESTS = [
export const SUBPROCESS_STUB_TESTS = [
'src/platforms/android/__tests__/{app-lifecycle-install,app-lifecycle-open,device-input-state,input-actions,notifications,settings}.test.ts',
'src/daemon/__tests__/runtime-hints.test.ts',
'src/platforms/apple/core/__tests__/index.test.ts',
vitest.mutation.config.ts
+26
View File
@@ -0,0 +1,26 @@
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { defineConfig } from 'vitest/config';
import { readTestScope, threadHostileTestFiles } from './scripts/mutation/test-scope.ts';
import { SUBPROCESS_STUB_TESTS } from './vitest.config.ts';
const repoRoot = path.dirname(fileURLToPath(import.meta.url));
// Test scope for the decision-kernel mutation lane (issue #1415).
//
// `scripts/mutation/run.ts` derives the per-run scope from Vitest's module graph
// (`vitest related` over the mutated files) and hands it over through
// AGENT_DEVICE_MUTATION_TEST_FILES; the fallback is the deterministic unit suite,
// which keeps `pnpm exec stryker run` usable by hand. Excluded either way: the
// subprocess-stub group and the CLI-capture tests — see
// scripts/mutation/test-scope.ts for why, and why excluding them cannot hide a
// surviving mutant.
const scope = readTestScope();
export default defineConfig({
test: {
include: scope ?? ['src/**/*.test.ts'],
exclude: [...SUBPROCESS_STUB_TESTS, ...threadHostileTestFiles(repoRoot), '**/node_modules/**'],
setupFiles: ['src/__tests__/hermetic-env-setup.ts', 'src/__tests__/process-memo-setup.ts'],
},
});