chore(mutation): shrink to report-only — drop the ratchet, baseline and graduation (#1457, #1781) (#1828)

* chore(mutation): shrink the lane to report-only (#1457, #1781 wave 2)

The mutation harness's two real catches (#1474, #1475) both came from humans
reading the weekly score report. The ratchet half never operated: the baseline
was committed exactly twice (8cce0ef6b, 60400d04b), both times with
`stableRuns: 0, gating: false`, and was never updated after the very fixes it
triggered — the weekly job computed a new baseline and then `git checkout --`d
it, uploading a proposal nobody applied in 3+ weeks. A gate nobody arms is
harness weight; the report is the part that paid.

Deletes ratchet.ts + ratchet.test.ts, mutation-baselines/, and every
baseline/graduation/gating path in run.ts (`--update`, `mutation:baseline`).
run.ts now exits non-zero only on a harness failure, never on a score. The
report renders the per-kernel table (kernel, score, killed, survived, total,
timeouts) plus the surviving mutants a strengthening PR works from.

Kernel scoping stays: stryker.config.json and KERNEL_MODULES are untouched.

* fix(mutation): restore denominator coverage and publish the table before judging the shard set

Review of #1828:
- `report.test.ts` re-asserts that Ignored/CompileError/RuntimeError leave the
  denominator — the one behaviour `ratchet.test.ts` covered and nothing replaced.
  A `tally()` edit that counted tool noise would have deflated every published
  score with a green `mutation:test`.
- `assertShardsCoverModules` now runs after `emit()`, so an incomplete shard set
  still publishes the kernels that completed instead of only an error string.
  This makes the workflow comments' claim about the job summary true rather than
  re-wording them down.

* chore(mutation): trigger the affected lane on exactly the paths that can select mutants

The PR lane returns an empty matrix unless the diff touches the harness, so the
kernel-source and `**/*.test.ts` triggers only bought a 1-4 min no-op job on
~96% of PRs. `on.pull_request.paths` is now exactly `LANE_TOOLING` plus the
workflow file, asserted in both directions by workflow.test.ts against the
exported constant — a missing path would let a harness change merge unproven,
an extra one starts a job that can only answer `[]`.

Also drops the workflow header's contradictory scope paragraph: it claimed the
lane selects on kernel sources and any test reaching one, which has not been
true since the ratchet went.

* fix(mutation): score and publish a short shard set before failing on the count

The expected-count check ran inside readShardedReports, before anything was
summarized, so on the weekly's real `--expect-shards 10` one dead shard threw
away the nine that had reported — the earlier reorder only moved the
zero-mutants check. The merge now returns the shard count, and both verdicts
run after emit() with the same exit code and `score` stage.

Regression uses the weekly argument shape (`--expect-shards 10`, one shard
present) and asserts the reporting kernel's row reaches stdout while the run
still fails.
This commit is contained in:
Michał Pierzchała
2026-08-18 17:47:29 +02:00
committed by GitHub
parent 9d6154eecb
commit 423927fdd8
21 changed files with 402 additions and 880 deletions
+23 -32
View File
@@ -2,43 +2,33 @@ name: Mutation Affected
# PR-side half of the decision-kernel mutation lane (issue #1415).
#
# Graduation, not a flag day: this job runs on every PR that touches a kernel
# module, but `scripts/mutation/run.ts` only exits non-zero once the committed
# baseline reports `gating: true` — earned by two consecutive stable weekly
# sweeps (mutation-weekly.yml). Until then it is a report with the same numbers,
# so the gate's first failing day is not also its first running day.
# Advisory only — the lane reports and never gates (#1457). `scripts/mutation/run.ts`
# exits non-zero on a harness failure, never on a score.
#
# Cost control: before graduation an affected run is a report nobody acts on, so
# `select` returns an empty matrix and no mutants run — except when the diff
# touches the lane's own tooling, the one case where a pre-graduation run buys
# something (the gate has to be proven before it can bite).
# Cost control: the kernel report that pays is the weekly sweep, so this lane
# spends mutants on one thing only — proving the harness still runs end to end
# when the harness itself changes. `select` therefore returns an empty matrix
# unless the diff touches the lane's own sources. Selecting on derived kernel
# ownership instead would run the full ten-shard sweep on 24 of the last 40
# merged PRs.
#
# Scope is the AFFECTED modules only, and affectedness is DERIVED from the import
# graph (scripts/mutation/ownership.ts): a kernel source, or any test that reaches
# one. Reaching a kernel is a superset of killing its mutants, so the `select` job
# frequently returns several modules — sharded like the weekly sweep so the PR's
# wall clock is one module, not their sum. The weekly run stays the full sweep.
# On a harness diff the matrix is the canary (`LANE_CANARY`, the registry's
# cheapest real sweep — lane sources own no kernel, so derivation alone would
# select nothing) plus any kernel that same diff reaches through the import
# graph (scripts/mutation/ownership.ts): a kernel source, or a test that reaches
# one. Selected modules are sharded like the weekly sweep, so the PR's wall
# clock is one module rather than their sum.
on:
pull_request:
paths:
# Kernel sources, every src/package test (ownership is derived, so any
# test may own a kernel — `select` decides, not this filter), and the
# lane's own tooling. scripts/mutation/workflow.test.ts asserts this
# covers the registry.
- 'packages/kernel/src/errors.ts'
- 'src/daemon/ref-frame.ts'
- 'src/commands/interaction/runtime/settle.ts'
- 'src/utils/scroll-edge-state.ts'
- 'packages/selectors/src/**'
- 'packages/ad-script/src/internal/target-annotation-serde.ts'
- 'src/snapshot/snapshot-occlusion.ts'
- 'src/**/*.test.ts'
- 'packages/*/src/**/*.test.ts'
# Exactly `LANE_TOOLING` in scripts/mutation/run.ts plus this file — the
# only diffs that can produce a non-empty matrix, asserted both ways by
# scripts/mutation/workflow.test.ts. A kernel or test path here would only
# start a job that selects nothing.
- 'scripts/mutation/**'
- 'scripts/lib/**'
- 'stryker.config.json'
- 'mutation-baselines/**'
- '.github/workflows/mutation-affected.yml'
permissions:
@@ -64,7 +54,7 @@ jobs:
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Ratchet self-test
- name: Harness self-test
uses: ./.github/actions/run-gate
with: { gate: mutation-model }
@@ -138,7 +128,7 @@ jobs:
.tmp/mutation/lane-envelope.json
if-no-files-found: warn
ratchet:
report:
name: Affected decision-kernel mutants
needs: [select, mutants]
if: always() && needs.select.result == 'success'
@@ -162,7 +152,8 @@ jobs:
# No shards means no affected kernel: run.ts reports "nothing to mutate"
# and still writes the envelope, so the lane is never silently absent.
- name: Ratchet the affected modules
# Either invocation writes the score table to $GITHUB_STEP_SUMMARY.
- name: Score the affected modules
run: |
if [ -d .tmp/mutation/shards ]; then
expected=$(echo '${{ needs.select.outputs.modules }}' | jq length)
@@ -177,7 +168,7 @@ jobs:
if: failure()
run: |
pnpm gate mutation --affected --fail-envelope \
"affected ratchet failed before producing a verdict (run ${{ github.run_id }})" || true
"affected report failed before producing a score table (run ${{ github.run_id }})" || true
- name: Upload mutation report
if: always()
+16 -21
View File
@@ -8,13 +8,11 @@ name: Mutation Weekly
# Sharded one job per module: the whole sweep is ~2,150 mutants, and the selector
# module alone is ~1,280 of them, so a single job would sit near the 30-minute
# acceptance budget on an ubuntu runner. The shards' JSON reports are merged into
# one verdict by the ratchet job (`--report-dir`), so the ratchet still sees a
# full sweep.
# one score table by the report job (`--report-dir`).
#
# The lane reports; it does not commit. The proposed baseline rides in the
# artifact and applying it is a reviewed `pnpm mutation:baseline` commit — a score
# can never lower itself. Gating (after two consecutive stable weekly runs) is
# enforced on PRs by mutation-affected.yml.
# The lane reports and never gates (#1457): the per-kernel table lands in the job
# summary and the artifact, and a low score is an input for a human-authored
# test-strengthening PR (#1474, #1475 were written that way).
on:
schedule:
@@ -90,8 +88,8 @@ jobs:
.tmp/mutation/lane-envelope.json
if-no-files-found: warn
ratchet:
name: Mutation ratchet
report:
name: Mutation score report
runs-on: ubuntu-latest
needs: shard
if: always()
@@ -103,7 +101,7 @@ jobs:
- name: Setup toolchain
uses: ./.github/actions/setup-node-pnpm
- name: Ratchet self-test
- name: Harness self-test
uses: ./.github/actions/run-gate
with: { gate: mutation-model }
@@ -113,9 +111,13 @@ jobs:
pattern: mutation-shard-*
path: .tmp/mutation/shards
# run.ts writes the markdown verdict to $GITHUB_STEP_SUMMARY when the
# run.ts writes the per-kernel score table to $GITHUB_STEP_SUMMARY when the
# runner exports it, so the summary and the artifact carry the same numbers.
- name: Ratchet the merged sweep and propose the next baseline
# It publishes the table before judging the shard set, so even the one
# failure it can raise — an incomplete shard set — still reports the kernels
# that completed. A red lane here always means the sweep did not happen,
# never a low score.
- name: Score the merged sweep
uses: ./.github/actions/run-gate
with:
gate: mutation-check
@@ -124,20 +126,14 @@ jobs:
.tmp/mutation/shards
--expect-shards
10
--update
- name: Stage the proposed baseline artifact
run: |
cp mutation-baselines/decision-kernels.json .tmp/mutation/proposed-baseline.json
git checkout -- mutation-baselines/decision-kernels.json
# The self-test and the artifact download both run before the ratchet, so a
# failure there would otherwise leave the aggregate lane with no envelope.
# The self-test and the artifact download both run before the scoring step,
# so a failure there would otherwise leave the aggregate lane with no envelope.
- name: Record a failed lane envelope
if: failure()
run: |
pnpm gate mutation --fail-envelope \
"weekly ratchet job failed before producing a verdict (run ${{ github.run_id }})" || true
"weekly report job failed before producing a score table (run ${{ github.run_id }})" || true
# Freshness/drift telemetry (#1430): the envelope states commit, Stryker
# version, config hash, duration and result, so a lane going dark or a tool
@@ -167,5 +163,4 @@ jobs:
path: |
.tmp/mutation/shards
.tmp/mutation/lane-envelope.json
.tmp/mutation/proposed-baseline.json
if-no-files-found: warn
+1 -2
View File
@@ -10,7 +10,6 @@
"node_modules/**",
"**/*.md",
"scripts/maestro-conformance/corpus/**",
"fallow-baselines/**",
"mutation-baselines/**"
"fallow-baselines/**"
]
}
+32 -31
View File
@@ -44,7 +44,7 @@ The mapping it encodes, for when you need to run a gate directly or reason about
| Daemon RPC wire surface (the declarations listed in `test/wire-compat/surface.ts` — JSON-RPC envelope, request/response/error/artifact/progress framing, `/health` payload, HTTP auth headers) | `pnpm exec vitest run --project unit-core test/wire-compat` holds the ledger to its source and prints the digest to paste; `pnpm check:daemon-wire-compat` compares it against the last released tag and requires a `DAEMON_RPC_PROTOCOL_VERSION` bump or a `compatibleChanges` ack for the drift. Read ADR 0006 to decide which; `test/wire-compat/README.md` walks both |
| Anything in `src/`, `test/` | `pnpm format` (`skills/` is Markdown-only guidance: oxfmt ignores `**/*.md`, and the affected-check selector classifies it docs-only) |
| Workspace package source (`packages/*/src/**`) | Root format/lint/typecheck plus layering (R11 package-boundaries); Vitest resolves affected tests through the module graph; package manifests/tsconfigs fail open to the full set |
| A decision kernel or its tests (`packages/kernel/src/errors.ts`, `src/daemon/ref-frame.ts`, `src/commands/interaction/runtime/settle.ts`, `src/utils/scroll-edge-state.ts`, `packages/selectors/src/`) | `pnpm mutation:affected --base origin/main` (minutes; GitHub runs it per PR — see the mutation ratchet section) |
| A decision kernel or its tests (`packages/kernel/src/errors.ts`, `src/daemon/ref-frame.ts`, `src/commands/interaction/runtime/settle.ts`, `src/utils/scroll-edge-state.ts`, `packages/selectors/src/`) | `pnpm mutation:run --modules <kernel>` (minutes; optional — the lane reports, it never gates, see the mutation section) |
Two traps worth naming:
@@ -243,7 +243,7 @@ The three facts the manifest cannot derive live together in `scripts/gate/declar
coverage wrapper, one reporting-only `test:*` script, and the Android replay owner hidden inside a
third-party action's `script:` input.
## Mutation ratchet over decision kernels
## Mutation report over decision kernels
Mutation score is the mechanical answer to "is this test load-bearing or decorative". A full-suite
sweep is unaffordable, so the scope is an enumerated list of pure decision kernels — modules where a
@@ -252,46 +252,47 @@ surviving mutant means a silently wrong agent-facing decision. The registry
are asserted against it, and PR-affected selection maps changed files through it. Modules that spawn
subprocesses or wait real time stay out by construction.
Mutation runs **report-only** on the seven decision kernels — a weekly full sweep plus a per-PR
affected sweep. It never gates: `scripts/mutation/run.ts` exits non-zero on a harness failure (a
missing report, an incomplete shard set, a bad argument) and never on a score. A low score is an
input for a human-authored test-strengthening PR, which is exactly how #1474 and #1475 were written.
The ratchet, baseline file and graduation rule this lane used to carry were deleted in #1457: in
three weeks nobody applied a baseline, so the gate half never operated while the report half was
paying.
```sh
pnpm mutation:test # ratchet self-test (fast, no Stryker)
pnpm mutation:test # harness self-test (fast, no Stryker)
pnpm mutation:run --modules selectors # one module locally (~7 min for selectors)
pnpm mutation:check # ratchet an existing .tmp/mutation/mutation.json
pnpm mutation:baseline # full sweep, then record it (reviewed commit)
pnpm mutation:check # score an existing .tmp/mutation/mutation.json
```
- **Weekly full sweep** (`.github/workflows/mutation-weekly.yml`) runs `shardMatrix()` from the
registry: one job per module, except modules that declare a `shards` count and are sliced with
`--shard i/n` (selectors is ~1,280 mutants, well past the 30-minute budget in one job). The ratchet
merges the shard reports (`--report-dir`) for one verdict and requires the full set
(`--expect-shards`), so a dead shard fails the lane instead of scoring its module as 0. Results are
reported as a job summary plus an artifact. It never commits: the proposed baseline rides in the
artifact, and applying it is a reviewed `pnpm mutation:baseline` commit, so a score cannot lower
itself.
- **PR lane** (`.github/workflows/mutation-affected.yml`) derives the affected shard matrix
(`--list-affected`) and merges the shards into one verdict. Before graduation the matrix is empty —
a report nobody acts on is not worth the runner minutes — unless the diff touches the lane's own
tooling, the one pre-graduation run that buys something: the gate has to be proven before it bites.
Lane sources own no kernel, so that exception adds `LANE_CANARY` (`kernel-errors`, the registry's
cheapest real sweep) to whatever the diff derives; otherwise it would select zero mutants and prove
nothing. `scripts/mutation/selection.test.ts` drives both halves of the rule through the real CLI
against a throwaway worktree commit.
- **Ratchet**: scores may only rise. `mutation-baselines/decision-kernels.json` records the
high-water score per module plus the Stryker version and config content hash that produced it, so a
score change caused by a tool/config change is reported as provenance drift, never as a
test-strength regression.
- **Graduation, not a flag day**: gating is off until two consecutive comparable weekly sweeps pass
(`stableRuns`/`requiredStableRuns` in the baseline); the PR job starts selecting modules — and
failing on them — once the committed baseline says `gating: true`. A regression or provenance drift
resets the counter.
`--shard i/n` (selectors is ~1,280 mutants, well past the 30-minute budget in one job). The report
job merges the shard reports (`--report-dir`) into one per-kernel table — kernel, score, killed,
survived, total, timeouts, plus the surviving mutants — and requires the full set
(`--expect-shards`), so a dead shard fails the lane instead of publishing its module as 0%. The
table lands in the job summary and the artifact.
- **PR lane** (`.github/workflows/mutation-affected.yml`) exists to prove the harness still runs end
to end when the harness changes, so its matrix (`--list-affected`) is empty unless the diff touches
the lane's own sources: the weekly sweep is the kernel report, and selecting on derived kernel
ownership would run the full ten-shard sweep on 24 of the last 40 merged PRs for a report nobody
gates on. The workflow triggers on exactly those sources (`LANE_TOOLING` in `run.ts`, asserted both
ways by `workflow.test.ts`), so a PR that could only select `[]` never starts the job. Lane sources
own no kernel, so a harness diff adds `LANE_CANARY` (`kernel-errors`, the registry's cheapest real
sweep) to whatever kernels that same diff derives; otherwise it would select zero mutants and prove
nothing. It reports the same table. `scripts/mutation/selection.test.ts` drives both halves of the
rule through the real CLI against a throwaway worktree commit.
- **Provenance**: every report and lane envelope carries the Stryker version and the config content
hash that produced it, so scores measured across a tool or config change are not read as
test-strength change.
- **Test scope** is derived from Vitest's module graph (`vitest related` over the mutated files), the
same delegation `pnpm check:affected` uses; see `scripts/mutation/test-scope.ts` for the three
groups it drops and why dropping them cannot hide a surviving mutant.
- **Test ownership is derived, never listed** (`scripts/mutation/ownership.ts`): a test owns every
kernel its imports reach, so `src/__tests__/daemon-error.test.ts` selects `kernel-errors` through
`src/daemon.ts` without naming it. A listed set of test files would silently omit exactly those
indirect tests and rot as tests are added — weakening one would skip the ratchet. Reaching a kernel
is a superset of killing its mutants, so the PR lane over-selects on purpose and shards the
selected modules; a false positive costs runner minutes, a false negative costs the gate. Non-kernel
`src/daemon.ts` without naming it. Reaching a kernel is a superset of killing its mutants, so the
derivation over-selects on purpose; it applies to the modules a lane-tooling diff selects. Non-kernel
*sources* are not owned: they can only move a score through those tests, and the weekly sweep
re-measures the whole surface.
- **Lane envelope** (`scripts/lib/lane-envelope.ts`, issue #1430): every run writes
-71
View File
@@ -1,71 +0,0 @@
{
"schemaVersion": 1,
"requiredStableRuns": 2,
"stableRuns": 0,
"gating": false,
"modules": {
"kernel-errors": {
"score": 55.19,
"killed": 101,
"survived": 82,
"total": 183,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"daemon-ref-frame": {
"score": 100,
"killed": 55,
"survived": 0,
"total": 55,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"interaction-settle": {
"score": 68.95,
"killed": 151,
"survived": 68,
"total": 219,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"scroll-edge-state": {
"score": 28.75,
"killed": 92,
"survived": 228,
"total": 320,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"selectors": {
"score": 70.56,
"killed": 901,
"survived": 376,
"total": 1277,
"strykerVersion": "9.6.1",
"configHash": "sha256:806d9f2e657f",
"updatedAt": "2026-07-27T14:29:40.750Z"
},
"target-annotation-serde": {
"score": 94.03,
"killed": 315,
"survived": 20,
"total": 335,
"strykerVersion": "9.6.1",
"configHash": "sha256:8b633048f754",
"updatedAt": "2026-08-02T07:28:56.278Z"
},
"snapshot-occlusion": {
"score": 89.74,
"killed": 175,
"survived": 20,
"total": 195,
"strykerVersion": "9.6.1",
"configHash": "sha256:8b633048f754",
"updatedAt": "2026-08-02T07:28:56.278Z"
}
}
}
-1
View File
@@ -119,7 +119,6 @@
"size": "node scripts/size-report.mjs",
"perf": "node --experimental-strip-types scripts/perf/run.ts",
"mutation:run": "node --experimental-strip-types scripts/mutation/run.ts",
"mutation:baseline": "node --experimental-strip-types scripts/mutation/run.ts --update",
"mutation:check": "node --experimental-strip-types scripts/mutation/run.ts --no-run",
"mutation:affected": "node --experimental-strip-types scripts/mutation/run.ts --affected",
"mutation:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/mutation/*.test.ts",
+2 -2
View File
@@ -128,8 +128,8 @@ export const CHECK_CATALOG: readonly CheckSpec[] = [
gate('fuzz-parsers', 'Parser fuzz invariants', 'fuzz:parsers', false),
gate('mutation', 'Mutation sweep', 'mutation:run', false),
gate('mutation-affected', 'Affected mutation shard selection', 'mutation:affected', false),
gate('mutation-check', 'Mutation ratchet against the baseline', 'mutation:check', false),
gate('mutation-model', 'Mutation ratchet self-test', 'mutation:test'),
gate('mutation-check', 'Mutation score for an existing report', 'mutation:check', false),
gate('mutation-model', 'Mutation harness self-test', 'mutation:test'),
gate(
'concurrency-torture',
'Session/lease/lock torture sweep',
+2 -2
View File
@@ -25,12 +25,12 @@ test('stryker mutate globs mirror the kernel-module registry', () => {
);
});
test('stryker owns no pass/fail threshold — the ratchet does', () => {
test('stryker owns no pass/fail threshold — the lane never gates on a score', () => {
const thresholds = readConfig().thresholds as { break?: number | null } | undefined;
assert.equal(
thresholds?.break ?? null,
null,
"A Stryker `break` threshold would fail runs on an absolute score; gating is the ratchet's job (scripts/mutation/ratchet.ts).",
'A Stryker `break` threshold would fail runs on an absolute score; this lane reports scores and never gates on them (#1457).',
);
});
+50 -20
View File
@@ -41,7 +41,7 @@ test('the envelope carries schema, commit, tool/config provenance, duration and
assert.deepEqual(envelope.data, { scope: 'full-sweep' });
});
test('a failed ratchet is recorded as a failed lane run', () => {
test('a lane that produced no report is recorded as a failed run', () => {
const envelope = laneEnvelope({
lane: 'mutation-decision-kernels',
commit: 'b'.repeat(40),
@@ -92,7 +92,7 @@ test('a crashed run still writes an envelope naming the stage it died in', () =>
// Shard artifacts carry the envelope next to the report, so merging "every JSON
// under the shard directory" fed the envelope to the report parser and crashed
// the ratchet job after the mutants had already run.
// the reporting job after the mutants had already run.
test('merging shard reports ignores the envelope sitting beside them', () => {
const shards = path.join(repoRoot, '.tmp/mutation/envelope-test-shards/shard-a');
fs.mkdirSync(shards, { recursive: true });
@@ -170,9 +170,9 @@ function readEnvelope(): Envelope {
}
// A merged shard set is only a sweep if every requested module actually reported.
// summarizeReport scores an absent module as 0, and while the lane is non-gating a
// 0 only *reports* a regression — so a dead matrix shard would otherwise be
// aggregated into a passing "complete" envelope claiming the sweep happened.
// summarizeReport scores an absent module as 0, so a dead matrix shard would
// otherwise be published as a 0% kernel in a passing "complete" envelope
// claiming the sweep happened.
test('an incomplete shard set fails instead of scoring the missing module as zero', () => {
const shards = path.join(repoRoot, '.tmp/mutation/partial-shards/shard-kernel-errors');
fs.mkdirSync(shards, { recursive: true });
@@ -191,14 +191,46 @@ test('an incomplete shard set fails instead of scoring the missing module as zer
assert.notEqual(result.exitCode, 0, 'a missing shard must fail the aggregate');
assert.match(result.stderr, /Incomplete shard set/);
assert.match(result.stderr, /daemon-ref-frame/);
// The kernels that did complete are still worth reading on a failed-shard day,
// so the table is published before the shard set is judged.
assert.match(result.stdout, /\| `kernel-errors` — [^|]+\| 100% \|/);
const envelope = readEnvelope();
assert.equal(envelope.result, 'fail');
assert.equal(envelope.data.stage, 'ratchet');
assert.equal(envelope.data.stage, 'score');
fs.rmSync(path.join(repoRoot, '.tmp/mutation/partial-shards'), { recursive: true, force: true });
});
// Argument parsing and the provenance/baseline reads used to sit outside the
// envelope boundary, so the lane could exit without declaring itself at all.
// The weekly job's real shape: `--expect-shards 10` over a directory missing
// shards. The count check used to run inside the merge, before anything was
// scored, so one dead shard discarded the nine that had reported.
test('a short shard set publishes the shards that did report, then fails', () => {
const shards = path.join(repoRoot, '.tmp/mutation/short-shards/shard-kernel-errors');
fs.mkdirSync(shards, { recursive: true });
fs.writeFileSync(
path.join(shards, 'mutation.json'),
JSON.stringify({
files: { 'packages/kernel/src/errors.ts': { mutants: [{ status: 'Killed' }] } },
}),
);
const result = runMutation([
'--report-dir',
'.tmp/mutation/short-shards',
'--expect-shards',
'10',
'--modules',
'kernel-errors',
]);
assert.notEqual(result.exitCode, 0, 'a short shard set must fail the aggregate');
assert.match(result.stderr, /1 report\(s\), expected 10/);
assert.match(result.stdout, /\| `kernel-errors` — [^|]+\| 100% \|/);
const envelope = readEnvelope();
assert.equal(envelope.result, 'fail');
assert.equal(envelope.data.stage, 'score');
fs.rmSync(path.join(repoRoot, '.tmp/mutation/short-shards'), { recursive: true, force: true });
});
// Argument parsing and the provenance read used to sit outside the envelope
// boundary, so the lane could exit without declaring itself at all.
test('a malformed invocation still writes an envelope', () => {
fs.rmSync(path.join(repoRoot, '.tmp/mutation/lane-envelope.json'), { force: true });
const result = runMutation(['--modules', 'not-a-kernel']);
@@ -221,7 +253,7 @@ test('--fail-envelope declares a step that failed before the sweep', () => {
assert.equal(envelope.data.error, 'self-test failed');
});
// The workflows run it from `if: failure()`, which also fires when the ratchet
// The workflows run it from `if: failure()`, which also fires when the sweep
// itself failed — a generic reason must never displace the specific one.
test('--fail-envelope keeps a failure the run already reported', () => {
fs.rmSync(path.join(repoRoot, '.tmp/mutation/lane-envelope.json'), { force: true });
@@ -230,20 +262,18 @@ test('--fail-envelope keeps a failure the run already reported', () => {
assert.equal(readEnvelope().data.error, 'the real failure');
});
// A pass is not a verdict worth preserving: the weekly job copies the proposed
// baseline and restores the committed one *after* the ratchet passed, so a failure
// there would otherwise publish the failed scheduled job as a passing lane.
// A pass is not a result worth preserving: the weekly job uploads its artifact
// *after* the report was rendered, so a failure there would otherwise publish
// the failed scheduled job as a passing lane.
test('--fail-envelope downgrades a passing envelope when a later step fails', () => {
const shards = path.join(repoRoot, '.tmp/mutation/pass-then-fail/shard-kernel-errors');
fs.mkdirSync(shards, { recursive: true });
// A perfect shard so the ratchet passes: the score can only rise from the
// committed kernel-errors baseline.
// A shard whose only mutant survived — a 0% score. The lane reports scores and
// never gates on them (#1457), so this run is still a pass: the sweep happened.
fs.writeFileSync(
path.join(shards, 'mutation.json'),
JSON.stringify({
files: {
'packages/kernel/src/errors.ts': { mutants: [{ status: 'Killed' }, { status: 'Killed' }] },
},
files: { 'packages/kernel/src/errors.ts': { mutants: [{ status: 'Survived' }] } },
}),
);
const passing = runMutation([
@@ -252,13 +282,13 @@ test('--fail-envelope downgrades a passing envelope when a later step fails', ()
'--modules',
'kernel-errors',
]);
assert.equal(passing.exitCode, 0, passing.stderr);
assert.equal(passing.exitCode, 0, 'a low score must never fail the lane');
assert.equal(readEnvelope().result, 'pass');
runMutation(['--fail-envelope', 'the baseline copy step failed']);
runMutation(['--fail-envelope', 'the artifact upload step failed']);
const envelope = readEnvelope();
assert.equal(envelope.result, 'fail', 'a failed job must not publish a passing envelope');
assert.equal(envelope.data.error, 'the baseline copy step failed');
assert.equal(envelope.data.error, 'the artifact upload step failed');
fs.rmSync(path.join(repoRoot, '.tmp/mutation/pass-then-fail'), { recursive: true, force: true });
});
+5 -5
View File
@@ -3,7 +3,7 @@
// Mutation score is the only mechanical answer to "is this test load-bearing or
// decorative", but a full-suite sweep is unaffordable. This registry is the
// single source of truth for what Stryker mutates: `stryker.config.json`'s
// `mutate` globs are asserted against it, and PR-affected gating maps changed
// `mutate` globs are asserted against it, and PR-affected selection maps changed
// files onto modules through it.
//
// Membership rule: pure decision kernels only — a surviving mutant here means a
@@ -112,10 +112,10 @@ export function mutateGlobs(ids: readonly ModuleId[] = ALL_MODULE_IDS): string[]
}
/**
* The module a lane-tooling change proves itself against before graduation.
* `kernel-errors` is the cheapest real sweep in the registry (one file, ~183
* mutants), so a change to the ratchet, the config, or the baseline runs actual
* mutants end to end without paying for the full sweep.
* The module a lane-tooling change proves itself against. `kernel-errors` is the
* cheapest real sweep in the registry (one file, ~183 mutants), so a change to
* the harness or the config runs actual mutants end to end without paying for
* the full sweep.
*/
export const LANE_CANARY: ModuleId = 'kernel-errors';
+6 -7
View File
@@ -1,8 +1,8 @@
// Which kernel module a changed file belongs to, DERIVED — never hand-listed.
//
// A mutation score is a statement about the tests that kill the mutants, so the
// PR lane must re-measure a kernel whenever one of *those* tests changes. An
// enumerated list of test files cannot state that: it silently omits tests that
// A mutation score is a statement about the tests that kill the mutants, so a
// selection is only honest if it follows *those* tests. An enumerated list of
// test files cannot state that: it silently omits tests that
// exercise a kernel indirectly (`src/__tests__/daemon-error.test.ts` reaches
// `normalizeError` through `src/daemon.ts`), and nothing fails when a new test
// is added. So ownership is computed from the static import graph instead: a test
@@ -11,13 +11,12 @@
// The derivation is deliberately a superset — reaching a kernel is cheaper to
// prove than killing its mutants, so an unrelated diff can select a module and
// pay for a report. False positives cost runner minutes; a false negative would
// let a weakened test slip past the ratchet, which is the thing the lane exists
// to catch.
// leave a kernel whose tests changed unmeasured.
//
// Non-test source changes outside the registry are NOT owned: they can only move
// a score through the tests that reach the kernel, and the weekly full sweep is
// what re-measures the whole surface. The PR lane's claim is narrower on purpose
// kernel sources plus the tests that exercise them.
// what re-measures the whole surface. The derived claim is narrower on purpose
// kernel sources plus the tests that exercise them.
import fs from 'node:fs';
import path from 'node:path';
-240
View File
@@ -1,240 +0,0 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { test } from 'node:test';
import { ALL_MODULE_IDS } from './modules.ts';
import { renderReport } from './report.ts';
import {
applyRun,
BASELINE_SCHEMA_VERSION,
DEFAULT_REQUIRED_STABLE_RUNS,
emptyBaseline,
evaluateRatchet,
type Baseline,
type ModuleBaseline,
type Provenance,
} from './ratchet.ts';
import { summarizeReport, type StrykerReport } from './score.ts';
const PROVENANCE: Provenance = { strykerVersion: '9.6.1', configHash: 'sha256:abcdef123456' };
const NOW = '2026-07-27T00:00:00.000Z';
function mutants(statuses: readonly string[]): StrykerReport {
return {
files: {
'packages/kernel/src/errors.ts': {
mutants: statuses.map((status, index) => ({
status,
mutatorName: 'ConditionalExpression',
location: { start: { line: index + 1 } },
})),
},
},
};
}
function baselineWith(entry: Partial<ModuleBaseline>): Baseline {
return {
...emptyBaseline(),
modules: {
'kernel-errors': {
score: 75,
killed: 3,
survived: 1,
total: 4,
strykerVersion: PROVENANCE.strykerVersion,
configHash: PROVENANCE.configHash,
updatedAt: NOW,
...entry,
},
},
};
}
test('scores count timeouts as killed and uncovered mutants as survived', () => {
const [score] = summarizeReport(mutants(['Killed', 'Timeout', 'Survived', 'NoCoverage']), [
'kernel-errors',
]);
assert.deepEqual(
{ score: score?.score, killed: score?.killed, survived: score?.survived, total: score?.total },
{ score: 50, killed: 2, survived: 2, total: 4 },
);
});
test('statuses outside the score (Ignored, CompileError) leave the denominator', () => {
const [score] = summarizeReport(mutants(['Killed', 'Ignored', 'CompileError']), [
'kernel-errors',
]);
assert.equal(score?.total, 1);
assert.equal(score?.score, 100);
});
test('a lowered score is a regression, and gating makes it fail with its survivors', () => {
const baseline = { ...baselineWith({ score: 100 }), stableRuns: 2, gating: true };
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.failed, true);
assert.deepEqual(
result.regressions.map((verdict) => verdict.module),
['kernel-errors'],
);
assert.match(result.regressions[0]?.detail ?? '', /fell 100% -> 50%/);
const markdown = renderReport(result, baseline, PROVENANCE);
assert.match(markdown, /Surviving mutants:/);
assert.match(markdown, /`packages\/kernel\/src\/errors\.ts:2` ConditionalExpression/);
assert.match(markdown, /scores may only rise/);
});
test('a lowered score is reported but does not fail while the lane is non-gating', () => {
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const baseline = baselineWith({ score: 100 });
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.regressions.length, 1);
assert.equal(result.failed, false);
assert.match(renderReport(result, baseline, PROVENANCE), /still non-gating/);
});
test('a regression never rewrites the baseline high-water mark', () => {
const baseline = baselineWith({ score: 100 });
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const next = applyRun(baseline, scores, evaluateRatchet(scores, baseline, PROVENANCE), {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(next.modules['kernel-errors']?.score, 100);
assert.equal(next.stableRuns, 0);
});
test('a risen score is recorded with its provenance', () => {
const baseline = baselineWith({ score: 50 });
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.verdicts[0]?.status, 'improved');
const next = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.deepEqual(next.modules['kernel-errors'], {
score: 100,
killed: 2,
survived: 0,
total: 2,
strykerVersion: PROVENANCE.strykerVersion,
configHash: PROVENANCE.configHash,
updatedAt: NOW,
});
});
test('gating graduates after the required consecutive stable full sweeps', () => {
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
let baseline = baselineWith({ score: 100 });
assert.equal(baseline.gating, false);
for (let run = 1; run <= baseline.requiredStableRuns; run += 1) {
const result = evaluateRatchet(scores, baseline, PROVENANCE);
baseline = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(baseline.stableRuns, run);
}
assert.equal(baseline.gating, true);
// A later regression resets graduation, so gating has to be re-earned.
const dropped = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const regressed = evaluateRatchet(dropped, baseline, PROVENANCE);
assert.equal(regressed.failed, true);
const after = applyRun(baseline, dropped, regressed, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(after.stableRuns, 0);
assert.equal(after.gating, false);
});
test('an affected PR run never advances graduation', () => {
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
const baseline = baselineWith({ score: 100 });
const next = applyRun(baseline, scores, evaluateRatchet(scores, baseline, PROVENANCE), {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: false,
});
assert.equal(next.stableRuns, 0);
});
test('a tool or config change is provenance drift, not a test-strength regression', () => {
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
for (const drift of [{ strykerVersion: '9.5.0' }, { configHash: 'sha256:000000000000' }]) {
const baseline = { ...baselineWith({ score: 100, ...drift }), stableRuns: 2, gating: true };
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.regressions.length, 0);
assert.equal(result.failed, false);
assert.equal(result.comparable, false);
assert.match(result.drifted[0]?.detail ?? '', /not attributable to test strength/);
// Drift rebases onto the new tool/config and costs the graduation streak.
const next = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(next.modules['kernel-errors']?.score, 50);
assert.equal(next.modules['kernel-errors']?.configHash, PROVENANCE.configHash);
assert.equal(next.stableRuns, 0);
}
});
test('a module with no baseline yet is new, recorded, and blocks graduation', () => {
const scores = summarizeReport(mutants(['Killed', 'Survived']), ['kernel-errors']);
const baseline = emptyBaseline();
const result = evaluateRatchet(scores, baseline, PROVENANCE);
assert.equal(result.verdicts[0]?.status, 'new');
assert.equal(result.comparable, false);
const next = applyRun(baseline, scores, result, {
provenance: PROVENANCE,
now: NOW,
countsTowardGraduation: true,
});
assert.equal(next.modules['kernel-errors']?.score, 50);
assert.equal(next.stableRuns, 0);
});
test('the report names the fix command and the graduation state', () => {
const scores = summarizeReport(mutants(['Killed', 'Killed']), ['kernel-errors']);
const baseline = { ...baselineWith({ score: 100 }), stableRuns: 1 };
const markdown = renderReport(
evaluateRatchet(scores, baseline, PROVENANCE),
baseline,
PROVENANCE,
);
assert.match(markdown, /gating \*\*off\*\* \(1\/2 stable weekly runs\)/);
assert.match(markdown, /Stryker `9\.6\.1` · config `sha256:abcdef123456`/);
});
test('the committed baseline is the shape the lane graduates from', () => {
const baseline = JSON.parse(
fs.readFileSync(
path.resolve(import.meta.dirname, '../../mutation-baselines/decision-kernels.json'),
'utf8',
),
) as Baseline;
assert.equal(baseline.schemaVersion, BASELINE_SCHEMA_VERSION);
// Issue #1415's N: gating is earned by two consecutive stable weekly sweeps.
assert.equal(baseline.requiredStableRuns, DEFAULT_REQUIRED_STABLE_RUNS);
assert.deepEqual(Object.keys(baseline.modules), [...ALL_MODULE_IDS]);
for (const [id, module] of Object.entries(baseline.modules)) {
assert.ok(module.total > 0, `${id} has no mutants`);
assert.ok(module.strykerVersion.length > 0, `${id} has no Stryker version`);
assert.match(module.configHash, /^sha256:[0-9a-f]{12}$/);
}
});
-208
View File
@@ -1,208 +0,0 @@
// The ratchet itself: scores may only rise, gating is earned, and a score that
// moved for tool/config reasons is never mistaken for test-strength change.
//
// Graduation (issue #1415): the lane starts non-gating. Each comparable weekly
// full sweep with no regression increments `stableRuns`; once it reaches
// `requiredStableRuns` the baseline flips to `gating: true`, after which a
// regression fails the run — weekly as a full sweep, PRs for AFFECTED modules
// only. A regression or a non-comparable run resets the counter to zero.
import { roundScore, type ModuleScore, type SurvivingMutant } from './score.ts';
import type { ModuleId } from './modules.ts';
export const BASELINE_SCHEMA_VERSION = 1;
export const DEFAULT_REQUIRED_STABLE_RUNS = 2;
export type ModuleBaseline = {
score: number;
killed: number;
survived: number;
total: number;
// Provenance travels with every module baseline so a tool or config change is
// distinguishable from a test-strength change.
strykerVersion: string;
configHash: string;
updatedAt: string;
};
export type Baseline = {
schemaVersion: number;
requiredStableRuns: number;
stableRuns: number;
gating: boolean;
modules: Record<string, ModuleBaseline>;
};
export type Provenance = { readonly strykerVersion: string; readonly configHash: string };
export type VerdictStatus = 'new' | 'improved' | 'held' | 'regressed' | 'provenance-drift';
export type ModuleVerdict = {
readonly module: ModuleId;
readonly score: number;
readonly killed: number;
readonly total: number;
readonly baselineScore: number | undefined;
readonly delta: number | undefined;
readonly status: VerdictStatus;
readonly surviving: readonly SurvivingMutant[];
readonly detail: string;
};
export type RatchetResult = {
readonly verdicts: readonly ModuleVerdict[];
readonly regressions: readonly ModuleVerdict[];
readonly drifted: readonly ModuleVerdict[];
/** A run is comparable when every module has a same-provenance baseline. */
readonly comparable: boolean;
readonly gating: boolean;
/** True when the caller must exit non-zero. */
readonly failed: boolean;
};
export function emptyBaseline(requiredStableRuns: number = DEFAULT_REQUIRED_STABLE_RUNS): Baseline {
return {
schemaVersion: BASELINE_SCHEMA_VERSION,
requiredStableRuns,
stableRuns: 0,
gating: false,
modules: {},
};
}
function driftDetail(previous: ModuleBaseline, provenance: Provenance): string {
return (
`baseline recorded by stryker ${previous.strykerVersion} / config ${previous.configHash}, ` +
`this run used stryker ${provenance.strykerVersion} / config ${provenance.configHash}; ` +
'the score change is not attributable to test strength — re-record with `pnpm mutation:baseline`'
);
}
function verdictFor(
score: ModuleScore,
previous: ModuleBaseline | undefined,
provenance: Provenance,
): ModuleVerdict {
const base = {
module: score.module,
score: score.score,
killed: score.killed,
total: score.total,
surviving: score.surviving,
};
if (!previous) {
return {
...base,
baselineScore: undefined,
delta: undefined,
status: 'new',
detail: `no baseline recorded yet (measured ${score.score}%)`,
};
}
const delta = roundScore(score.score - previous.score);
if (
previous.strykerVersion !== provenance.strykerVersion ||
previous.configHash !== provenance.configHash
) {
return {
...base,
baselineScore: previous.score,
delta,
status: 'provenance-drift',
detail: driftDetail(previous, provenance),
};
}
if (delta < 0) {
return {
...base,
baselineScore: previous.score,
delta,
status: 'regressed',
detail:
`mutation score fell ${previous.score}% -> ${score.score}% ` +
`(${score.survived} surviving mutants)`,
};
}
return {
...base,
baselineScore: previous.score,
delta,
status: delta > 0 ? 'improved' : 'held',
detail:
delta > 0
? `mutation score rose ${previous.score}% -> ${score.score}%`
: `mutation score held at ${score.score}%`,
};
}
export function evaluateRatchet(
scores: readonly ModuleScore[],
baseline: Baseline,
provenance: Provenance,
): RatchetResult {
const verdicts = scores.map((score) =>
verdictFor(score, baseline.modules[score.module], provenance),
);
const regressions = verdicts.filter((verdict) => verdict.status === 'regressed');
const drifted = verdicts.filter((verdict) => verdict.status === 'provenance-drift');
const comparable = drifted.length === 0 && verdicts.every((v) => v.status !== 'new');
return {
verdicts,
regressions,
drifted,
comparable,
gating: baseline.gating,
failed: baseline.gating && regressions.length > 0,
};
}
export type ApplyOptions = {
readonly provenance: Provenance;
readonly now: string;
/** Only the weekly full sweep may advance graduation; affected runs may not. */
readonly countsTowardGraduation: boolean;
};
/**
* Fold a run into the baseline: keep the high-water score per module (a
* regression never rewrites it, so the ratchet keeps failing until the tests are
* restored), rebase modules whose provenance drifted onto the new tool/config,
* then advance or reset graduation.
*/
export function applyRun(
baseline: Baseline,
scores: readonly ModuleScore[],
result: RatchetResult,
options: ApplyOptions,
): Baseline {
const modules: Record<string, ModuleBaseline> = { ...baseline.modules };
for (const score of scores) {
const previous = modules[score.module];
const status = result.verdicts.find((verdict) => verdict.module === score.module)?.status;
const rebase = !previous || status === 'provenance-drift';
if (!rebase && score.score < previous.score) continue;
modules[score.module] = {
score: rebase ? score.score : Math.max(previous.score, score.score),
killed: score.killed,
survived: score.survived,
total: score.total,
strykerVersion: options.provenance.strykerVersion,
configHash: options.provenance.configHash,
updatedAt: options.now,
};
}
const stable = result.comparable && result.regressions.length === 0;
const stableRuns = !options.countsTowardGraduation
? baseline.stableRuns
: stable
? baseline.stableRuns + 1
: 0;
return {
...baseline,
schemaVersion: BASELINE_SCHEMA_VERSION,
stableRuns,
gating: stableRuns >= baseline.requiredStableRuns,
modules,
};
}
+80
View File
@@ -0,0 +1,80 @@
// The report is the lane's whole product (#1457): it never gates, so a table
// that stops carrying the numbers is the lane failing silently rather than
// loudly. Both #1474 and #1475 were written from this table.
//
// Only the renderer is exercised here; every test that runs the CLI lives in
// envelope.test.ts, because a second file spawning `run.ts` would race it over
// the repo-relative lane envelope.
import assert from 'node:assert/strict';
import { test } from 'node:test';
import { renderReport } from './report.ts';
import { summarizeReport } from './score.ts';
const PROVENANCE = { strykerVersion: '9.6.1', configHash: 'sha256:abcdef123456' };
test('the table carries score, killed, survived, total and timeouts per kernel', () => {
const scores = summarizeReport(
{
files: {
'packages/kernel/src/errors.ts': {
mutants: [
{ status: 'Killed' },
{ status: 'Timeout' },
{
status: 'Survived',
mutatorName: 'ConditionalExpression',
location: { start: { line: 7 } },
},
{
status: 'NoCoverage',
mutatorName: 'BooleanLiteral',
location: { start: { line: 9 } },
},
],
},
},
},
['kernel-errors'],
);
const markdown = renderReport(scores, PROVENANCE);
// Timeouts count as killed, so a score propped up by slow mutants rather than
// assertions is only visible if the column is rendered.
assert.match(markdown, /\| `kernel-errors` — [^|]+\| 50% \| 2 \| 2 \| 4 \| 1 \|/);
// The surviving mutants are what a test-strengthening PR works from.
assert.match(markdown, /- `packages\/kernel\/src\/errors\.ts:7` ConditionalExpression/);
assert.match(markdown, /- `packages\/kernel\/src\/errors\.ts:9` BooleanLiteral/);
});
// The denominator is the report's arithmetic: counting tool-side noise would
// silently deflate every score the lane publishes. Ported from the deleted
// ratchet.test.ts, which was where this lived.
test('statuses outside the score (Ignored, CompileError, RuntimeError) leave the denominator', () => {
const [score] = summarizeReport(
{
files: {
'packages/kernel/src/errors.ts': {
mutants: [
{ status: 'Killed' },
{ status: 'Ignored' },
{ status: 'CompileError' },
{ status: 'RuntimeError' },
],
},
},
},
['kernel-errors'],
);
assert.equal(score?.total, 1);
assert.equal(score?.score, 100);
});
test('a kernel with no surviving mutants lists no detail section', () => {
const scores = summarizeReport(
{ files: { 'packages/kernel/src/errors.ts': { mutants: [{ status: 'Killed' }] } } },
['kernel-errors'],
);
const markdown = renderReport(scores, PROVENANCE);
assert.match(markdown, /\| `kernel-errors` — [^|]+\| 100% \| 1 \| 0 \| 1 \| 0 \|/);
assert.doesNotMatch(markdown, /^### /m);
});
+33 -44
View File
@@ -1,36 +1,36 @@
// Markdown rendering for the mutation lane: GitHub job summary and terminal
// output share one renderer, so the artifact and the console never disagree.
//
// The lane reports and never gates (#1457), so the report is the whole product:
// a per-kernel score table plus the surviving mutants a test-strengthening PR
// would have to kill.
import { moduleById } from './modules.ts';
import type { Baseline, ModuleVerdict, Provenance, RatchetResult } from './ratchet.ts';
import type { ModuleScore } from './score.ts';
const DEFAULT_MAX_SURVIVING_LISTED = 20;
function formatDelta(delta: number | undefined): string {
if (delta === undefined) return '—';
return delta > 0 ? `+${delta}` : String(delta);
}
export type Provenance = { readonly strykerVersion: string; readonly configHash: string };
function renderRow(verdict: ModuleVerdict): string {
const module = moduleById(verdict.module);
const baseline = verdict.baselineScore === undefined ? '—' : `${verdict.baselineScore}%`;
function renderRow(score: ModuleScore): string {
const module = moduleById(score.module);
return (
`| \`${verdict.module}\`${module.label} | ${verdict.score}% | ${baseline} | ` +
`${formatDelta(verdict.delta)} | ${verdict.killed}/${verdict.total} | ` +
`${verdict.surviving.length} | ${verdict.status} |`
`| \`${score.module}\`${module.label} | ${score.score}% | ${score.killed} | ` +
`${score.survived} | ${score.total} | ${score.timeout} |`
);
}
function renderDetail(verdict: ModuleVerdict, maxListed: number): string[] {
const lines = ['', `### \`${verdict.module}\`${verdict.status}`, '', verdict.detail];
if (verdict.surviving.length > 0) {
lines.push('', 'Surviving mutants:', '');
for (const mutant of verdict.surviving.slice(0, maxListed)) {
lines.push(`- \`${mutant.file}:${mutant.line}\` ${mutant.mutator}`);
}
if (verdict.surviving.length > maxListed) {
lines.push(`- …and ${verdict.surviving.length - maxListed} more`);
}
function renderDetail(score: ModuleScore, maxListed: number): string[] {
const lines = [
'',
`### \`${score.module}\`${score.survived} surviving mutant(s) at ${score.score}%`,
'',
];
for (const mutant of score.surviving.slice(0, maxListed)) {
lines.push(`- \`${mutant.file}:${mutant.line}\` ${mutant.mutator}`);
}
if (score.surviving.length > maxListed) {
lines.push(`- …and ${score.surviving.length - maxListed} more`);
}
return lines;
}
@@ -41,8 +41,7 @@ export type RenderOptions = {
};
export function renderReport(
result: RatchetResult,
baseline: Baseline,
scores: readonly ModuleScore[],
provenance: Provenance,
options: RenderOptions = {},
): string {
@@ -51,31 +50,21 @@ export function renderReport(
`## ${options.title ?? 'Mutation score — decision kernels'}`,
'',
`Stryker \`${provenance.strykerVersion}\` · config \`${provenance.configHash}\` · ` +
`gating **${baseline.gating ? 'on' : 'off'}** ` +
`(${baseline.stableRuns}/${baseline.requiredStableRuns} stable weekly runs)`,
'report only — this lane never fails a build.',
'',
'| Module | Score | Baseline | Δ | Killed/Total | Surviving | Status |',
'| --- | --- | --- | --- | --- | --- | --- |',
...result.verdicts.map(renderRow),
'| Kernel | Score | Killed | Survived | Total | Timeout |',
'| --- | --- | --- | --- | --- | --- |',
...scores.map(renderRow),
];
for (const verdict of [...result.regressions, ...result.drifted]) {
lines.push(...renderDetail(verdict, maxListed));
for (const score of scores) {
if (score.surviving.length > 0) lines.push(...renderDetail(score, maxListed));
}
if (result.failed) {
lines.push(
'',
'Mutation ratchet failed: scores may only rise. Kill the surviving mutants listed above, ' +
'then re-run `pnpm mutation:run` (full sweep) or `pnpm mutation:check --report <file>` ' +
'against an existing Stryker report.',
);
} else if (result.regressions.length > 0) {
lines.push(
'',
'Scores regressed while the lane is still non-gating — no failure recorded, but the ' +
'surviving mutants above are the tests to strengthen before gating turns on.',
);
}
lines.push(
'',
'A low score names tests worth strengthening (#1474, #1475 were written from this table); ' +
'it is an input for a human-authored PR, not a verdict.',
);
return `${lines.join('\n')}\n`;
}
+86 -120
View File
@@ -1,15 +1,15 @@
// Entrypoint for the decision-kernel mutation lane (issue #1415).
//
// pnpm mutation:run full sweep + ratchet check
// pnpm mutation:baseline full sweep, then record the scores
// pnpm mutation:check --report <file> ratchet an existing Stryker report
// pnpm mutation:run full sweep, rendered as a report
// pnpm mutation:check --report <file> score an existing Stryker report
// pnpm mutation:affected --base origin/main
// PR lane: mutate only the kernel
// modules the diff touches
//
// The weekly workflow runs the full sweep and writes the rendered report to the
// job summary plus an artifact; the PR lane runs the affected subset and only
// fails once the baseline has graduated to `gating: true`.
// The lane reports and never gates (#1457): a score can only ever be printed,
// written to the job summary and uploaded as an artifact. The only non-zero exit
// is a harness failure — a missing report, an incomplete shard set, a bad
// argument — never a low score.
import crypto from 'node:crypto';
import fs from 'node:fs';
@@ -29,15 +29,7 @@ import {
type ShardSpec,
} from './modules.ts';
import { derivedAffectedModules } from './ownership.ts';
import { renderReport } from './report.ts';
import {
applyRun,
emptyBaseline,
evaluateRatchet,
type Baseline,
type Provenance,
type RatchetResult,
} from './ratchet.ts';
import { renderReport, type Provenance } from './report.ts';
import { mergeReports, summarizeReport, type ModuleScore, type StrykerReport } from './score.ts';
import {
expandMutateFiles,
@@ -49,7 +41,6 @@ import {
const repoRoot = runCmdSync('git', ['rev-parse', '--show-toplevel']).stdout.trim();
const CONFIG_PATH = 'stryker.config.json';
const BASELINE_PATH = 'mutation-baselines/decision-kernels.json';
const DEFAULT_REPORT_PATH = '.tmp/mutation/mutation.json';
const TEST_SCOPE_PATH = '.tmp/mutation/test-scope.json';
const ENVELOPE_PATH = '.tmp/mutation/lane-envelope.json';
@@ -66,12 +57,10 @@ const USAGE = `Usage: pnpm mutation:run [options]
Fail unless --report-dir holds exactly n shard reports
--shard <i/n> Mutate only the i-th of n balanced slices of the module's
sources (the big modules exceed one job's budget)
--update Record the run into the baseline (ratchet + graduation)
--summary <file> Also write the markdown report to <file>
--no-run Alias for --report with the default report path
--list-affected Print the PR lane's shard matrix as JSON and exit (empty
until the baseline graduates, unless the diff touches the
lane's own tooling)
unless the diff touches the lane's own tooling)
--fail-envelope <reason>
Write a failed lane envelope for a step that ran before (or
instead of) the sweep, e.g. a self-test failure
@@ -83,7 +72,6 @@ type Args = {
base: string;
report: string | undefined;
reportDir: string | undefined;
update: boolean;
summary: string | undefined;
listAffected: boolean;
failEnvelope: string | undefined;
@@ -127,7 +115,6 @@ function parseMutationArgs(argv: readonly string[]): Args {
base: { type: 'string', default: 'origin/main' },
report: { type: 'string' },
'report-dir': { type: 'string' },
update: { type: 'boolean', default: false },
summary: { type: 'string' },
'no-run': { type: 'boolean', default: false },
'list-affected': { type: 'boolean', default: false },
@@ -141,7 +128,6 @@ function parseMutationArgs(argv: readonly string[]): Args {
base: values.base ?? 'origin/main',
report: values.report ?? (values['no-run'] ? DEFAULT_REPORT_PATH : undefined),
reportDir: values['report-dir'],
update: Boolean(values.update),
summary: values.summary,
listAffected: Boolean(values['list-affected']),
failEnvelope: values['fail-envelope'],
@@ -166,18 +152,6 @@ function readProvenance(root: string = repoRoot): Provenance {
};
}
function readBaseline(root: string = repoRoot): Baseline {
const file = path.join(root, BASELINE_PATH);
if (!fs.existsSync(file)) return emptyBaseline();
return JSON.parse(fs.readFileSync(file, 'utf8')) as Baseline;
}
function writeBaseline(baseline: Baseline, root: string = repoRoot): void {
const file = path.join(root, BASELINE_PATH);
fs.mkdirSync(path.dirname(file), { recursive: true });
fs.writeFileSync(file, `${JSON.stringify(baseline, null, 2)}\n`);
}
function changedFiles(base: string): string[] {
const result = runCmdSync('git', ['diff', '--name-only', '--merge-base', base, 'HEAD'], {
cwd: repoRoot,
@@ -242,8 +216,8 @@ async function runStryker(
onStdoutChunk: (chunk) => void process.stdout.write(chunk),
onStderrChunk: (chunk) => void process.stderr.write(chunk),
});
// Stryker exits non-zero on a low score too; the ratchet — not Stryker's own
// thresholds — owns the verdict, so only a missing report is fatal here.
// Stryker exits non-zero on a low score too; this lane reports scores rather
// than judging them, so only a missing report is fatal here.
if (!fs.existsSync(absolute)) {
throw new Error(
`Stryker produced no report at ${reportPath} (exit ${result.exitCode}). See output above.`,
@@ -267,7 +241,7 @@ function emit(markdown: string, summaryPath: string | undefined): void {
* failure the freshness monitor must see, so the stage rides in the envelope
* rather than only in the job log.
*/
type Stage = 'setup' | 'select' | 'stryker' | 'report' | 'ratchet' | 'complete';
type Stage = 'setup' | 'select' | 'stryker' | 'report' | 'score' | 'complete';
/**
* Provenance for a lane that died before it could read any: `unknown` is a
@@ -278,17 +252,15 @@ const UNKNOWN_PROVENANCE: Provenance = { strykerVersion: 'unknown', configHash:
type LaneState = {
stage: Stage;
provenance: Provenance;
baseline: Baseline;
modules: readonly ModuleId[];
affected: boolean;
scores: readonly ModuleScore[];
result: RatchetResult | undefined;
error: string | undefined;
/**
* `--fail-envelope` keeps an existing *failure* (its reason is the specific
* one) but replaces an existing pass: a post-verdict step can fail after the
* ratchet passed, and publishing that job as passing is the bug the envelope
* exists to prevent.
* one) but replaces an existing pass: a step can fail after the report was
* rendered, and publishing that job as passing is the bug the envelope exists
* to prevent.
*/
recoveryOnly: boolean;
};
@@ -303,6 +275,19 @@ function existingResult(file: string): string | undefined {
}
}
/**
* A module's measurement for the envelope, or explicit nulls when the lane died
* before measuring it an absent field would read as "no mutants", a score.
*/
function envelopeModule(id: ModuleId, scores: readonly ModuleScore[]) {
const score = scores.find((entry) => entry.module === id);
if (!score) {
return { id, score: null, killed: null, survived: null, total: null, timeout: null };
}
const { module: _module, surviving: _surviving, ...measured } = score;
return { id, ...measured };
}
/**
* Scheduled-lane artifact envelope (#1430). Without it a downloaded report cannot
* say which commit or Stryker/config version produced it, how long the sweep took,
@@ -329,24 +314,14 @@ function writeEnvelope(state: LaneState, startedAtMs: number): void {
tool: { stryker: state.provenance.strykerVersion },
configHash: state.provenance.configHash,
startedAtMs,
result: state.stage === 'complete' && !state.result?.failed ? 'pass' : 'fail',
// The lane never judges a score, so the result states whether the lane
// produced a report at all.
result: state.stage === 'complete' ? 'pass' : 'fail',
data: {
scope: state.affected ? 'affected' : 'full-sweep',
stage: state.stage,
error: state.error ?? null,
modules: state.modules.map((id) => {
const score = state.scores.find((entry) => entry.module === id);
return {
id,
score: score?.score ?? null,
killed: score?.killed ?? null,
total: score?.total ?? null,
status: state.result?.verdicts.find((verdict) => verdict.module === id)?.status ?? null,
};
}),
gating: state.baseline.gating,
stableRuns: state.baseline.stableRuns,
requiredStableRuns: state.baseline.requiredStableRuns,
modules: state.modules.map((id) => envelopeModule(id, state.scores)),
},
});
fs.mkdirSync(path.dirname(target), { recursive: true });
@@ -362,7 +337,12 @@ function readReport(file: string): StrykerReport {
return JSON.parse(fs.readFileSync(absolute, 'utf8')) as StrykerReport;
}
function readShardedReports(dir: string, expected: number | undefined): StrykerReport {
/**
* Merges whatever shard reports are present and says how many there were. The
* expected-count verdict is the caller's, so a short set is still scored and
* published before it fails.
*/
function readShardedReports(dir: string): { report: StrykerReport; count: number } {
const root = path.isAbsolute(dir) ? dir : path.join(repoRoot, dir);
// Shard artifacts also carry the lane envelope and the derived test scope, so
// the report is selected by name rather than by "every .json here".
@@ -371,17 +351,8 @@ function readShardedReports(dir: string, expected: number | undefined): StrykerR
.map((file) => path.join(root, file))
.sort();
if (files.length === 0) throw new Error(`No Stryker JSON reports found under ${dir}`);
// Sub-sharded modules make "every module has mutants" too weak on its own: the
// surviving slices would still cover the module, so the expected shard count is
// asserted as well.
if (expected !== undefined && files.length !== expected) {
throw new Error(
`Incomplete shard set from ${dir}: ${files.length} report(s), expected ${expected}. ` +
'A shard job failed or its artifact is absent; the aggregate is not a sweep.',
);
}
process.stdout.write(`mutation: merging ${files.length} shard report(s) from ${dir}\n`);
return mergeReports(files.map(readReport));
return { report: mergeReports(files.map(readReport)), count: files.length };
}
async function produceReport(
@@ -393,28 +364,23 @@ async function produceReport(
return readReport(reportPath ?? DEFAULT_REPORT_PATH);
}
function recordRun(args: Args, state: LaneState, result: RatchetResult): void {
const next = applyRun(state.baseline, state.scores, result, {
provenance: state.provenance,
now: new Date().toISOString(),
// Only the full sweep proves stability; an affected subset says nothing
// about the modules it skipped.
countsTowardGraduation: !args.affected && state.modules.length === ALL_MODULE_IDS.length,
});
writeBaseline(next);
process.stdout.write(
`\nBaseline updated (${BASELINE_PATH}): ${next.stableRuns}/${next.requiredStableRuns} ` +
`stable runs, gating ${next.gating ? 'on' : 'off'}.\n`,
);
}
/**
* A merged shard set must cover every requested module. `summarizeReport` scores
* a module with no mutants as 0, and while the lane is non-gating a 0 is only
* *reported* as a regression so a matrix shard that died would otherwise be
* aggregated into a `complete`/`pass` envelope claiming the sweep happened.
* A merged shard set is only a sweep if every expected shard reported and every
* requested module has mutants sub-sharded modules make the second check too
* weak alone, since a surviving slice still covers its module. `summarizeReport`
* scores an absent module as 0, so without this a dead shard would be published
* as a `complete`/`pass` envelope claiming the sweep happened.
*
* Both verdicts run after the table is published: the kernels that did complete
* are the part of the run still worth reading on a failed-shard day.
*/
function assertShardsCoverModules(state: LaneState, dir: string): void {
function assertCompleteSweep(state: LaneState, dir: string, shards: number, args: Args): void {
if (args.expectShards !== undefined && shards !== args.expectShards) {
throw new Error(
`Incomplete shard set from ${dir}: ${shards} report(s), expected ${args.expectShards}. ` +
'A shard job failed or its artifact is absent; the aggregate is not a sweep.',
);
}
const missing = state.scores.filter((score) => score.total === 0).map((score) => score.module);
if (missing.length === 0) return;
throw new Error(
@@ -423,24 +389,24 @@ function assertShardsCoverModules(state: LaneState, dir: string): void {
);
}
/** Reads shard reports, an existing report, or runs Stryker — and scores them. */
async function scoreModules(args: Args, state: LaneState): Promise<void> {
/**
* Reads shard reports, an existing report, or runs Stryker and scores them.
* Returns the shard count when the source was a shard directory.
*/
async function scoreModules(args: Args, state: LaneState): Promise<number | undefined> {
state.stage = args.reportDir || args.report ? 'report' : 'stryker';
const report = args.reportDir
? readShardedReports(args.reportDir, args.expectShards)
: await produceReport(state.modules, args.report, args.shard);
const shards = args.reportDir ? readShardedReports(args.reportDir) : undefined;
const report = shards?.report ?? (await produceReport(state.modules, args.report, args.shard));
state.stage = 'ratchet';
state.stage = 'score';
state.scores = summarizeReport(report, state.modules);
if (args.reportDir) assertShardsCoverModules(state, args.reportDir);
return shards?.count;
}
async function sweep(args: Args, state: LaneState): Promise<number> {
state.stage = 'select';
// Test attribution is derived from the import graph, not a listed set of test
// files: see scripts/mutation/ownership.ts.
// Same selection the PR matrix uses, graduation rule included, so the ratchet
// job can never run mutants the `select` job decided not to spend.
// Same selection the PR matrix uses, so the reporting job can never score
// mutants the `select` job decided not to spend.
if (args.affected) {
state.modules = [...new Set(affectedMatrix(args.base).map((entry) => entry.module))];
}
@@ -450,49 +416,52 @@ async function sweep(args: Args, state: LaneState): Promise<number> {
return 0;
}
await scoreModules(args, state);
const result = evaluateRatchet(state.scores, state.baseline, state.provenance);
state.result = result;
const shards = await scoreModules(args, state);
const title = args.affected
? 'Mutation score — affected decision kernels'
: 'Mutation score — decision kernels';
emit(renderReport(result, state.baseline, state.provenance, { title }), args.summary);
emit(renderReport(state.scores, state.provenance, { title }), args.summary);
if (args.update) recordRun(args, state, result);
if (args.reportDir) assertCompleteSweep(state, args.reportDir, shards ?? 0, args);
state.stage = 'complete';
return result.failed ? 1 : 0;
return 0;
}
/** Sources of the lane itself: a change here must prove itself on real mutants. */
const LANE_TOOLING = ['scripts/mutation/', 'scripts/lib/', 'stryker.config.json', 'mutation-'];
/**
* Sources of the lane itself: a change here must prove itself on real mutants.
* These are also the only paths that can produce a non-empty matrix, so
* `mutation-affected.yml` triggers on exactly them asserted by
* `workflow.test.ts`, since a filter that missed one would let a harness change
* merge unproven, and any wider filter only buys a no-op job.
*/
export const LANE_TOOLING = ['scripts/mutation/', 'scripts/lib/', 'stryker.config.json'];
/**
* The PR lane's matrix. Before graduation the affected run is a report nobody
* acts on, so it costs runner minutes for no verdict: it stays empty until the
* baseline reaches `gating: true`. The exception is a diff that changes the lane
* itself that is the one case where the pre-graduation run buys something,
* because the gate has to be proven before it can bite.
* The PR lane's matrix: empty unless the diff touches the harness, otherwise the
* canary plus whatever kernels that same diff derives. The kernel report that
* pays is the weekly sweep, so the PR lane spends mutants on one thing only
* proving the harness still runs end to end when the harness changes. Selecting
* on derived kernel ownership alone would run the full ten-shard sweep on 24 of
* the last 40 merged PRs, a per-PR full sweep for a report nobody gates on.
*/
export function affectedMatrixFor(
changed: readonly string[],
gating: boolean,
root: string = repoRoot,
): ShardSpec[] {
const touchesLane = changed.some((file) =>
LANE_TOOLING.some((prefix) => normalizePath(file).startsWith(prefix)),
);
if (!gating && !touchesLane) return [];
if (!touchesLane) return [];
const modules = new Set(derivedAffectedModules(changed, root));
// Lane sources own no kernel, so a tooling-only diff derives nothing: without
// the canary the "prove the gate" exception would select zero mutants and
// prove nothing.
if (touchesLane) modules.add(LANE_CANARY);
// the canary the lane would select zero mutants and prove nothing.
modules.add(LANE_CANARY);
return shardMatrix(ALL_MODULE_IDS.filter((id) => modules.has(id)));
}
function affectedMatrix(base: string): ShardSpec[] {
return affectedMatrixFor(changedFiles(base), readBaseline().gating);
return affectedMatrixFor(changedFiles(base));
}
/**
@@ -504,7 +473,6 @@ async function run(argv: readonly string[], state: LaneState): Promise<number> {
const args = parseMutationArgs(argv);
state.affected = args.affected;
state.provenance = readProvenance();
state.baseline = readBaseline();
state.modules = args.modules;
if (args.failEnvelope) {
// A step that ran before the sweep failed (the weekly self-test, setup): the
@@ -528,11 +496,9 @@ async function main(argv = process.argv.slice(2)): Promise<number> {
const state: LaneState = {
stage: 'setup',
provenance: UNKNOWN_PROVENANCE,
baseline: emptyBaseline(),
modules: [],
affected: false,
scores: [],
result: undefined,
error: undefined,
recoveryOnly: false,
};
+28 -22
View File
@@ -25,6 +25,9 @@ export type ModuleScore = {
readonly killed: number;
readonly survived: number;
readonly total: number;
/** Timeouts, already counted in `killed` reported so a score propped up by
* slow mutants rather than assertions is visible. */
readonly timeout: number;
readonly surviving: readonly SurvivingMutant[];
};
@@ -37,8 +40,8 @@ const SURVIVED_STATUSES = new Set(['Survived', 'NoCoverage']);
/**
* Merge sharded Stryker reports into one. The weekly sweep runs one shard per
* kernel module so no single job approaches its time budget; the ratchet still
* evaluates a single full-sweep report.
* kernel module so no single job approaches its time budget; the report is still
* rendered from a single full-sweep view.
*/
export function mergeReports(reports: readonly StrykerReport[]): StrykerReport {
const files: Record<string, { mutants: StrykerMutant[] }> = {};
@@ -52,7 +55,7 @@ export function mergeReports(reports: readonly StrykerReport[]): StrykerReport {
return { files };
}
export function roundScore(value: number): number {
function roundScore(value: number): number {
return Math.round(value * 100) / 100;
}
@@ -60,33 +63,35 @@ function compareMutants(a: SurvivingMutant, b: SurvivingMutant): number {
return a.file.localeCompare(b.file) || a.line - b.line || a.mutator.localeCompare(b.mutator);
}
type Bucket = { killed: number; survived: number; timeout: number; surviving: SurvivingMutant[] };
function tally(bucket: Bucket, file: string, mutant: StrykerMutant): void {
if (KILLED_STATUSES.has(mutant.status)) {
bucket.killed += 1;
if (mutant.status === 'Timeout') bucket.timeout += 1;
return;
}
if (!SURVIVED_STATUSES.has(mutant.status)) return;
bucket.survived += 1;
bucket.surviving.push({
file: normalizePath(file),
line: mutant.location?.start?.line ?? 0,
mutator: mutant.mutatorName ?? 'unknown',
});
}
export function summarizeReport(
report: StrykerReport,
ids: readonly ModuleId[] = ALL_MODULE_IDS,
): ModuleScore[] {
const buckets = new Map<
ModuleId,
{ killed: number; survived: number; surviving: SurvivingMutant[] }
>();
for (const id of ids) buckets.set(id, { killed: 0, survived: 0, surviving: [] });
const buckets = new Map<ModuleId, Bucket>();
for (const id of ids) buckets.set(id, { killed: 0, survived: 0, timeout: 0, surviving: [] });
for (const [file, entry] of Object.entries(report.files)) {
const id = moduleForFile(file);
if (!id) continue;
const bucket = buckets.get(id);
const bucket = id ? buckets.get(id) : undefined;
if (!bucket) continue;
for (const mutant of entry.mutants) {
if (KILLED_STATUSES.has(mutant.status)) {
bucket.killed += 1;
} else if (SURVIVED_STATUSES.has(mutant.status)) {
bucket.survived += 1;
bucket.surviving.push({
file: normalizePath(file),
line: mutant.location?.start?.line ?? 0,
mutator: mutant.mutatorName ?? 'unknown',
});
}
}
for (const mutant of entry.mutants) tally(bucket, file, mutant);
}
return [...buckets].map(([module, bucket]) => {
@@ -97,6 +102,7 @@ export function summarizeReport(
killed: bucket.killed,
survived: bucket.survived,
total,
timeout: bucket.timeout,
surviving: bucket.surviving.sort(compareMutants),
};
});
+12 -14
View File
@@ -1,7 +1,7 @@
// Selection is what the PR lane spends money on, so both halves of the rule are
// asserted end to end through the real CLI: nothing runs before graduation, and
// the "prove the gate" exception for a lane-tooling diff selects real mutants
// rather than an empty matrix that proves nothing.
// asserted end to end through the real CLI: a kernel diff selects nothing (the
// weekly sweep is the kernel report), and a lane-tooling diff selects real
// mutants rather than an empty matrix that proves nothing.
import assert from 'node:assert/strict';
import fs from 'node:fs';
@@ -70,23 +70,21 @@ function listAffected(cwd: string): ShardSpec[] {
return JSON.parse(result.stdout.trim().split('\n').at(-1)!) as ShardSpec[];
}
test('a lane-tooling diff selects real mutants even before graduation', () => {
const dir = worktreeWithCommit('tooling', ['scripts/mutation/ratchet.ts']);
test('a lane-tooling diff selects real mutants', () => {
const dir = worktreeWithCommit('tooling', ['scripts/mutation/run.ts']);
// The lane's own sources own no kernel, so derivation alone yields nothing:
// without the canary this exception would run zero mutants.
// without the canary a harness change would run zero mutants.
assert.deepEqual(listAffected(dir), shardMatrix([LANE_CANARY]));
});
test('a kernel diff selects nothing until the baseline graduates', () => {
// The weekly sweep is the kernel report; selecting on derived ownership would
// run the full ten-shard sweep on most PRs for a report nobody gates on.
test('a kernel diff selects nothing — only a harness diff spends mutants', () => {
const dir = worktreeWithCommit('kernel', ['src/utils/scroll-edge-state.ts']);
assert.deepEqual(listAffected(dir), []);
// …and the same diff selects that module once gating is on.
assert.deepEqual(
affectedMatrixFor(['src/utils/scroll-edge-state.ts'], true),
shardMatrix(['scroll-edge-state']),
);
assert.deepEqual(affectedMatrixFor(['src/utils/scroll-edge-state.ts']), []);
});
test('a docs-only diff selects nothing even once gating is on', () => {
assert.deepEqual(affectedMatrixFor(['docs/agents/testing.md'], true), []);
test('a docs-only diff selects nothing', () => {
assert.deepEqual(affectedMatrixFor(['docs/agents/testing.md']), []);
});
+1 -1
View File
@@ -21,7 +21,7 @@
// assert over the repo's own registries rather than any decision kernel
// and own their CI job.
//
// Nothing here weakens the ratchet: a mutant only an excluded test could kill
// Nothing here inflates the score: a mutant only an excluded test could kill
// shows up as a survivor — visible work, never a silent pass.
import fs from 'node:fs';
+24 -36
View File
@@ -1,13 +1,15 @@
// The workflows' YAML cannot read the kernel registry, so these assertions keep
// the two in step: a module added to KERNEL_MODULES that no weekly shard runs
// would silently drop out of the sweep, and one no PR path filter selects would
// silently stop gating once the ratchet graduates.
// The workflows' YAML cannot read the kernel registry or the lane's own source
// list, so these assertions keep them in step: a module added to KERNEL_MODULES
// that no weekly shard runs would silently drop out of the sweep, and a PR path
// filter out of step with LANE_TOOLING either lets a harness change merge
// unproven or starts a job that selects nothing.
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { test } from 'node:test';
import { KERNEL_MODULES, shardMatrix } from './modules.ts';
import { shardMatrix } from './modules.ts';
import { LANE_TOOLING } from './run.ts';
const repoRoot = path.resolve(import.meta.dirname, '../..');
@@ -33,15 +35,15 @@ test('the weekly sweep shards exactly the registry matrix', () => {
);
});
test('the weekly sweep merges the shards into one ratcheted verdict', () => {
test('the weekly sweep merges the shards into one score table', () => {
const yaml = workflow('mutation-weekly.yml');
assert.match(yaml, /gate: mutation-check[\s\S]*--report-dir/);
assert.match(yaml, /GITHUB_STEP_SUMMARY|\$GITHUB_STEP_SUMMARY/);
// A dead shard must not be merged into a verdict that looks like a sweep.
// A dead shard must not be merged into a table that looks like a sweep.
assert.match(
yaml,
new RegExp(`--expect-shards\\s+${shardMatrix().length}\\b`),
'the weekly ratchet does not require the full shard set',
'the weekly report does not require the full shard set',
);
});
@@ -55,39 +57,25 @@ test('no mutation shard is allowed to exceed the 30-minute budget', () => {
}
});
test('every kernel path a PR can touch selects the affected mutation job', () => {
// Only a harness diff can produce a non-empty matrix, so the trigger is asserted
// in both directions against LANE_TOOLING: a missing path lets a harness change
// merge without ever running a mutant, and an extra one starts a select job that
// can only answer `[]`.
test('the affected lane triggers on exactly the lane sources that can select mutants', () => {
// Quote style is the formatter's business (oxfmt formats the workflow tree), so
// accept either spelling of the same scalar rather than pinning this gate to it.
const paths = [
...workflow('mutation-affected.yml').matchAll(/^ {6}- (?<q>['"])(?<glob>[^'"]+)\k<q>$/gm),
].map((match) => match.groups!.glob);
for (const module of KERNEL_MODULES) {
for (const owned of module.owns) {
const selected = paths.some(
(glob) =>
glob === owned ||
glob === `${owned}**` ||
(glob.endsWith('/**') && owned.startsWith(glob.slice(0, -2))),
);
assert.ok(selected, `no path filter selects ${owned} (module ${module.id})`);
}
}
// Ownership is derived, so any test in src/ or a workspace package's src/
// can own a kernel; the filter must let all of them through and leave the
// decision to the `select` job. A narrower filter is exactly the omission
// the derivation exists to prevent.
assert.ok(
paths.includes('src/**/*.test.ts'),
'the PR lane must trigger on every src test, since test ownership is derived',
);
assert.ok(
paths.includes('packages/*/src/**/*.test.ts'),
'the PR lane must trigger on every packages/*/src test too — target-annotation-serde is owned by one',
const expected = [
...LANE_TOOLING.map((prefix) => (prefix.endsWith('/') ? `${prefix}**` : prefix)),
// The workflow reruns itself so a trigger edit is proven by the lane it edits.
'.github/workflows/mutation-affected.yml',
];
assert.deepEqual(
[...paths].sort(),
[...expected].sort(),
'the PR path filter drifted from LANE_TOOLING in scripts/mutation/run.ts',
);
assert.match(workflow('mutation-affected.yml'), /gate: mutation-affected[\s\S]*--list-affected/);
// The lane's own sources fail open into it too: a ratchet or baseline edit must
// prove itself against real mutants, not against a stale report.
for (const own of ['scripts/mutation/**', 'stryker.config.json', 'mutation-baselines/**']) {
assert.ok(paths.includes(own), `missing path filter ${own}`);
}
});
+1 -1
View File
@@ -1,6 +1,6 @@
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"_comment": "Decision-kernel mutation lane (issue #1415). The mutate list below MUST mirror KERNEL_MODULES in scripts/mutation/modules.ts — scripts/mutation/config.test.ts asserts it. Ratchet verdicts come from scripts/mutation/run.ts, never from Stryker's own thresholds. Any edit here changes the config content hash, which invalidates every recorded baseline as non-comparable until re-recorded with `pnpm mutation:baseline`.",
"_comment": "Decision-kernel mutation lane (issue #1415). The mutate list below MUST mirror KERNEL_MODULES in scripts/mutation/modules.ts — scripts/mutation/config.test.ts asserts it. The lane reports scores and never gates on them (#1457), so no `break` threshold may be set here either. Any edit changes the config content hash the report and lane envelope carry, so scores measured before and after are not comparable.",
"packageManager": "pnpm",
"tsconfigFile_comment": "Points at a path that does not exist on purpose. Stryker's sandbox rewrites tsconfig `extends`/`references` through the classic TypeScript API, which the repo's TypeScript 7 native package no longer exposes (`ts.parseConfigFileTextToJson is not a function`). Our tsconfig has neither extends nor references, so there is nothing to rewrite — skipping the preprocessor is exact, not a workaround for a real dependency. The behaviour relied on is that the preprocessor no-ops when the file is absent: see packages/core/src/sandbox/ts-config-preprocessor.ts (https://github.com/stryker-mutator/stryker-js/blob/master/packages/core/src/sandbox/ts-config-preprocessor.ts) and https://stryker-mutator.io/docs/stryker-js/configuration/#tsconfigfile-string. Re-check it on every Stryker upgrade: if a future version errors on a missing tsconfig instead, point this at a real, extends-free tsconfig.",
"tsconfigFile": "tsconfig.stryker-absent.json",