mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
fix(tests): unzero the preview / component-tests tier, and make a zero-collected run say so (#4531)
* fix(tests): unzero the component tier — one mock factory was aborting the whole run `preview / component-tests` was reporting `failure` on `main` while executing ZERO tests. Reproduced deterministically (3/3 atd353f785c3, and in isolation): the whole `component` project aborts before any reporter prints, with no `Test Files` line, no per-file results, and exit 1. Root cause, one file: `src/tests/pages/apps/review/review-queue-nav.browser.test.tsx:30` mocks `~/providers/FeatureFlagsProvider` with a WHOLESALE factory naming only `useFeatureFlags`. The review queue page's row now renders the review entry point, which reads flags through `useOptionalFeatureFlags`, so the named import has nothing to bind to: SyntaxError: The requested module '/src/providers/FeatureFlagsProvider.tsx' does not provide an export named 'useOptionalFeatureFlags' In the node `unit` project that would fail ONE file. In BROWSER mode it kills the run: vitest resolves a manual mock over the browser-to-node channel inside a Playwright route handler that does not catch (`@vitest/browser-playwright/dist/index.js`, `await module.resolve()` inside `page.route`), so the rejection escapes as an Unhandled Rejection in the orchestrator. The printed error names neither the file nor the real cause. It is wrapped twice -- once by the browser mocker, once again on the node side -- and the innermost `cause` is dropped in transit, so all you see is the generic "[vitest] There was an error when mocking a module ... make sure there are no top level variables inside", which points at hoisting and is wrong. The root cause above was recovered by temporarily patching `createHelpfulError` in the browser tester bundle to inline `cause.stack`; the file was then confirmed by bisecting the 50 candidate files down to one. This is the SECOND time this class has bitten (see the header of `src/components/AppBlocks/__tests__/featureFlagsMockCompleteness.test.ts`, which fixed six sibling suites in the AppBlocks directory and deliberately scoped its guard there). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(ci): make the component tier fail LOUDLY on zero collected, not silently The `preview / component-tests` tier could report `failure` having executed ZERO tests, and nothing anywhere said so. The shared `npm-report-only-suite` Tekton task computes its verdict from the runner's EXIT CODE alone, so an abort that collected nothing and a genuine list of red assertions both render as `component:fail` / "Component suite failed" -- same words, same colour, same place. That is the shape that trains people to click through a tier. `pnpm test:component` now runs through `scripts/test-component-run.mjs`, which asks vitest for a JSON report and hands it to `scripts/ci/assert-component-suite-ran.mjs`. That gate prints a ledger -- `N executed, N skipped, across N files; N failed suites, N failed tests` -- and fails when nothing was collected, or when the executed count falls below a floor (1240, ~55% of the 2254 measured on a full green run of 201 files on 2026-08-31). Deliberate limits, each of which is a way this could have been wrong: - It can only ever ADD a failure. Vitest's own exit code is passed straight through when the ledger is satisfied, so a red suite stays red for its own reason. - EXECUTED, not TOTAL. `numTotalTests` counts skipped tests, so a suite that self-skipped wholesale would satisfy a total-based floor having run nothing. - `failed` counts as executed. A guard that scored a red run as "did not run" would fire on every genuine failure, and the tier would then be red for two different reasons that nobody could tell apart -- the exact confusion this removes. - A signal-killed runner short-circuits the gate entirely. The CI task wraps this in `timeout(1)` and distinguishes "timeout" from "fail" on purpose; a killed run also writes no report, so relabelling it "collected nothing" would be a wrong answer rather than a missing one. - A narrowed run (a file argument, or `-t`) skips the FLOOR but not the zero check: the collected count is then a property of the filter, but a single-file run that collects nothing is the cheapest reproduction of the abort and is precisely when someone is debugging it. The message enumerates every cause it cannot tell apart rather than asserting one -- an absence is the observable the most causes share, and a guard that names the wrong one sends the next reader hunting a bug that is not there. All three named have been observed on this suite; two of them were observed while writing this change, and the browser-crash one arrives wearing the mock error's headline with the real cause on the `Caused by:` line. Verification: - 16 unit tests over JSON fixtures (`scripts/__tests__/`), covering zero, the floor boundary both sides, all-red, all-skipped, narrowed both ways, import-failed suites, and a missing/unparseable report. - Mutation-checked: 9 mutants of the gate, ALL KILLED, each by the specific named test written for it (verified per-test, not "some test failed"), against a green 16/16 baseline. - End-to-end negative control: reverting the one-line fix in the previous commit and running through the wrapper produces the ledger's abort message and exit 1, from the missing-report branch. - End-to-end positive: the full suite through `pnpm run test:component`. Docs corrected while here: CONTRIBUTING said component tests "don't run in CI at all" and put the file count at 106; they do run, report-only, in the preview pipeline, and there are 201 files with ~2,250 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): the zero-collected message overstated a PARTIAL abort The headline said "THE COMPONENT SUITE COLLECTED NOTHING ... nothing executed", and the code contradicts it in the case that actually happens most: the JSON reporter writes at the END of a run, so an abort part-way through leaves no report at all. Measured while writing this: a run that aborted 68 files into 201 had 68 files scrolled past as green, wrote nothing, and got told nothing executed. Both halves of that were wrong to assert. Tests HAD executed, and a reader who scrolls up and sees green lines is entitled to believe the message is confused -- or, worse, to believe the green lines are coverage. So the headline is now "THIS RUN PRODUCED NO ACCOUNTABLE RESULT", which is true of both shapes, and the diagnosis says explicitly that an abort can land part-way and that whatever scrolled past is unaccounted for rather than confirmed. The missing-report branch says why the report is absent (the reporter writes at the end) and that a partial run and a run that never started are indistinguishable from there -- which is the reason neither counts as one that ran. No behaviour change: the same runs fail, with the same exit code. The four test assertions that pinned the old headline move with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): close six findings from the adversarial audit of this PR An adversarial audit of #4531 found two ways the new guard reproduced the very pathology it was written to remove, plus four smaller gaps. All six are fixed here; nothing about the payload fix inc32aa06e8dchanges. 1. A SIGNAL WAS RELABELLED AS A TEST FAILURE. `child.on('exit')` returned a hardcoded 143 for EVERY signal. `report-only-suite-task.yaml` branches on 137 to report `oom-killed` -- "this is an OUT-OF-MEMORY kill, not a timeout; raise the task's memory limit" -- and before this wrapper existed an OOM-killer SIGKILL on vitest reached that task as 137, because pnpm re-raises. With a constant 143 it matched no branch and fell through to `RC=1`, verdict `fail`, rendered "Component suite failed". A memory problem reported as a test failure, on the same tier, in the same words. Now `128 + os.constants.signals[sig]`, so 137/143/130 come out right and the mapping cannot drift from the names node hands back. The comment above the branch claimed it passed the status through; it does now. 2. A CALLER `--outputFile` MADE A GREEN RUN REPORT "COLLECTED NOTHING". Measured by the auditor: `pnpm test:component <file> --outputFile=/tmp/x.json` wrote an 18/18 green report to the caller's path, left the wrapper's path empty, and printed the full "this tier verified NOTHING on this commit" diagnosis naming three causes, none of them real. `.github/workflows/lint.yml` runs the SIBLING unit tier with exactly that flag, so it is a copy-paste away. Now refused up front with the fix in the message. 3. `isNarrowed` WAS WRONG IN BOTH DIRECTIONS ON SPACE-SEPARATED FLAGS. `--max-workers 1` -- the form CONTRIBUTING steers people towards for sizing a run on a shared box -- put `1` in a positional slot, so the run scored "narrowed" and THE FLOOR WAS SILENTLY TURNED OFF; same for `--reporter`, `--retry`, `--bail`, `--project`, `--pool`. In the other direction `--shard=1/4` scored NOT narrowed, so a healthy sharded run would fail the floor while being told "Do NOT lower the floor to make this green" -- misdirection, not merely a false red. Value-taking flags now consume their value, and `--shard`/`--changed`/`--related` narrow explicitly. 4. WINDOWS. `spawn()` of `vitest.cmd` without `shell` has failed since the node 18.20.2/20.12.2 CVE fix; `scripts/test-unit-run.mjs` already sets `shell: process.platform === 'win32'` for this reason. Added. And a spawn failure (rc 127) now short-circuits instead of handing the gate a missing report and getting forty lines about mock factories and dead browsers. 5. THE REPORT IS NOW DELETED BEFORE THE RUN, not only after. Cleanup after the run is skipped by exactly the paths that leave a stale report behind (a signal death, a throw), so a healthy 2254-test report could survive into a later run that aborted before writing one -- the gate reading it, printing a green ledger, and passing: silently inert in precisely its own use case. 6. THE GATE COMPUTED A FILE COUNT AND NEVER CHECKED IT. Replaced with a LEDGER, which is the stronger of the two checks: it walks `src/` for every `*.browser.test.tsx` and fails when one is absent from the report, NAMING it. The test floor sits at ~55%, so ~45% of the suite could stop being collected while the gate stayed green -- and the incident this whole guard descends from is exactly that shape (six files contributing 0 of 438, nothing red). The expectation is re-derived every run, so there is no constant to go stale. Skipped when narrowed, and when the walk finds nothing -- an empty walk is not a measurement, and it SAYS so rather than passing quietly. Also: the fixed factory now names all THREE hooks the flags module exports. `useFeatureFlagsReady` is the third and has four live consumers; none is in this page's graph today, which is the only reason naming two loads at all. The comment said "BOTH hooks", which reads as "the module has exactly two" -- and it is the template the PR proposes copying to fifty files. Verification: - 27 unit tests (was 16), including the on-disk ledger against a fixture tree that contains a `node_modules/` and a non-browser `.test.tsx` neither of which may count. - Mutation-checked: 17 mutants, ALL KILLED, each by its own named test. One SURVIVED on the first sweep -- `out.length > 0 ? out : null` was unreachable from the fixture, which had no `src/` at all, so the walk returned early. A second case with a `src/` that holds no browser tests reaches it; an empty array is TRUTHY, so that mutant would otherwise have armed a ledger over zero expected files and passed everything. - The on-disk ledger run against the REAL tree: 201/201 green, and dropping one real file from the report fails and names it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): close audit round 2 — two regressions the round-1 fixes introduced A delta re-audit of `6ceac37f33..7f005f0482` confirmed 4 of the 7 claims outright (the signal mapping, the pre-run clear, the file ledger, the fixed factory) with its own positive and negative controls, and found that two of the fixes had introduced new defects of their own. Both are here, with the smaller findings. 🔴 THE ON-DISK LEDGER FALSE-FAILED A LEGITIMATE RUN, WITH A MESSAGE FORBIDDING THE FIX. `--exclude`, `--dir` and `--root` were added to VALUE_FLAGS but not to NARROWING_FLAGS. All three take a value AND genuinely shrink the collected file set, so `pnpm test:component --exclude 'src/tests/**'` scored as a FULL run and the ledger failed it naming up to 200 files -- asserting the include broke or the run died, and telling the reader "Do NOT silence this by narrowing the walk", which is the only thing that would have fixed it. `--config` joins them: it can replace the project's `include` outright, which is the assumption the walk is built on. This is precisely the shape the round was convened to prevent, produced by the round's own fix. 🔴 `shell: win32` WENT ON THE SHARED `run()`, SO IT ALSO WRAPPED THE GATE SPAWN. The claim said this matched `scripts/test-unit-run.mjs`; it did not -- that file puts `shell` on its vitest spawn only and deliberately leaves its `process.execPath` spawn alone. Node with `shell: true` concatenates argv UNESCAPED (DEP0190), and the gate is spawned as `process.execPath`, which on Windows is `C:\Program Files\nodejs\node.exe`. So on the one platform the option exists to support, every `pnpm test:component` would have failed at the gate step with a cmd.exe parse error rather than any of this wrapper's messages. `shell` is now per-call. Also: - KEBAB AND CAMEL ARE THE SAME FLAG TO VITEST (cac camelCases every option key), so a hand-enumerated list has a hole wherever it carries one spelling and not the other -- and it did: `--max-workers`/`--maxWorkers` were both listed, `--test-timeout` was not, so the kebab form silently disabled both checks. Spellings are now canonicalised rather than enumerated, and the test asserts them in PAIRS. - `--output-file` (kebab) was NOT refused, so the exact defect the refusal exists for was still reachable -- under a test named "catches every spelling". Meanwhile `--outputFile.junit=` and `--outputFile.html=` WERE refused, though neither touches the `.json` key: object-form output paths are per-reporter, so that was over-strict and the stated reason ("the bare form sets the path for EVERY reporter") is true only of the bare form. Both directions fixed. - The rc-127 short-circuit is keyed on a spawn-failure FLAG, not on the number. 127 is an exit code a runner can produce on its own, and relabelling that as "the binary could not be started" is a second wrong answer that also skips the gate on a run that happened. - `--repo-root <dir>` did not consume its value, so with the flag FIRST the directory was read as the report path: "EISDIR: illegal operation on a directory" plus the whole abort diagnosis. Every test passed it last, which is why nothing caught it; the ledger tests now pass it first. - `--related` removed. It is a vitest SUBCOMMAND, and this wrapper always spawns `vitest run …`, so it cannot arrive -- the entry and its assertion were both inert, the assertion pinning behaviour for an input the runner cannot receive. - The factory now names all FOUR runtime exports of the flags module, including `FeatureFlagsProvider`. The comment said "EVERY runtime hook", which was narrower than the module -- and "not in this graph today" is exactly the reasoning that put this file in the diff. - A comment the previous round made false: it said the positional rule would catch `-t foo` anyway. It stopped being true the moment `-t` was listed as value-taking -- the value would now be CONSUMED -- so the explicit narrowing branch is load-bearing. - CONTRIBUTING documents the refusal, the file ledger, and which flags narrow. Coverage for the two claims that shipped with none: `main` now takes its collaborators as injected defaults, so the ORDER of effects can be asserted rather than inspected. The two `clearReport()` calls are byte-identical statements -- only their position carries the meaning, so a refactor moving the first below the run reopens "a stale report satisfies the next run's ledger" with every other test green. Verification: 35 unit tests (was 27). Mutation-checked at 17 mutants, ALL KILLED, zero survivors -- including re-running round 1's guards, because an audit fix resets the gate. Real-tree controls re-run with the new argument parsing, flag first and flag absent: 201/201 green, and dropping one real file fails and names it. ESLint on the touched files is back to the base count; two `no-empty-function` errors this change introduced are fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): close audit round 3 — the canonicaliser re-opened the class it closed Round 3 verified 8 of round 2's 9 claims against the diff or against vitest's own source (it read cac's `setDotProp` to confirm the `--outputFile.junit=` allowance is genuinely safe, and `cliOptionsConfig` to confirm `-r`/`-c` really are root/config). It found one regression and one coverage hole, both here. 🔴 A BOOLEAN FLAG SWALLOWED THE FILENAME AFTER IT — the same shape round 2 fixed, re-introduced by round 2's own mechanism. `canonicalFlag` collapsed `.subkey` onto the parent, so every dot-subkey inherited its parent's value-consuming behaviour, and the two entries `--coverage.reporter`/`--coverage.provider` became a single `coverage` that also matched the BARE `--coverage` — which takes no value (`argument: ""` in vitest's `cliOptionsConfig`). So `pnpm test:component --coverage <file>` ate the FILE as `--coverage`'s value, scored the run as full, and failed an 18/18 green single-file run naming ~200 files as ABSENT while telling the reader not to narrow the walk. Same for `--coverage.enabled <file>` and `--browser.headless <file>`. Measured base-vs-head, all three flipped `true` to `false`. Matching is now on the full canonical PATH, so a subkey is value-taking only if it is listed as one. 🔴 THREE MUTANTS SURVIVED A GREEN 35-TEST SUITE, AND EACH WAS THIS PR'S OWN HEADLINE FAILURE. The injected fakes took no parameter, so nothing could observe the argv `main` builds — the seam between the two modules this change exists to wire together. Dropping `--narrowed` from the gate argv makes every narrowed run hard-fail both the floor and the on-disk ledger; dropping `--outputFile.json=` makes every run report "does not exist" plus the whole abort diagnosis; dropping `...argv` makes `pnpm test:component <file>` silently run the entire suite behind a healthy-looking ledger. Testing `isNarrowed` in isolation cannot see whether its answer is ever USED. The fakes now capture their argument and three cases assert it; all three mutants die. Also: - `--repo-root` with no value was a silent fall-through to the real repo root, so a fixture report got graded against the 201 real files and failed with a confident diagnosis about the include breaking — produced by a typo. Now a usage error (exit 2). - CONTRIBUTING: the sample ledger omitted `measured <date>`, which the real output carries; and "skips both checks but not the zero check" counted a different pair than the two enumerated four lines above. The checks are now numbered and the sentence names which ones a narrowed run skips. Verification: 39 unit tests (was 35). Mutation-checked at 19 mutants, ALL KILLED, zero survivors — the three that survived round 3, the round-3 fixes themselves, and every guard from rounds 1 and 2 re-run, because an audit fix resets the gate. Real-tree controls re-run: 201/201 green, one missing file still fails. ESLint clean on the changed test file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): close audit round 4 — replace the flag LIST with a shape rule Round 4 measured `isNarrowed` at base vs head across every one of vitest 4.1.11's 72 boolean options and 73 value-taking ones, and found the flag list had traded 22 loud wrong answers for 25 quiet ones. Splitting `coverage` into two subkeys left `--retry.count 2`, `--browser.name chromium` and sixteen more `coverage.*` paths reading their VALUE as a filename, scoring a FULL run as narrowed and switching the file ledger and the floor off with a one-line note. That is the direction this file's own comments repeatedly name as the worse of the two. 🔴 THE LIST WAS THE PROBLEM, AND THREE ROUNDS WERE SPENT ON IT. Enumerating flag NAMES missed `--test-timeout` next to `--testTimeout`. Canonicalising spellings then made `--coverage <file>` swallow the file. Splitting into subkeys produced the 25 above. vitest 4.1.11 has a 164-path option tree; a hand-maintained copy of it is wrong the day it is written, and each fix moved the wrongness rather than removing it. So the rule is now about the ARGUMENT, not the flag: a positional is a file filter if it looks like a path, or if nothing before it could have been expecting a value. A non-path token straight after a flag is that flag's value, WHATEVER the flag is — correct for all 73 value-taking options without naming one of them, and correct for all 72 booleans too. `VALUE_FLAGS` is deleted. What remains is `NARROWING_FLAGS` (flags that genuinely shrink the run and must be named, because omitting one fails LOUDLY) and a five-entry `PATH_VALUE_FLAGS` for the only residual the shape cannot decide: a value that is itself a path. 🔴 AND THE DECISION IS NOW A SENTENCE, NOT A BOOLEAN. No rule over an unknowable flag list is right always; what must never happen is being wrong SILENTLY, because `--narrowed` disables the two checks this whole change exists to add. `narrowingReason` returns why, and the runner prints it: `--retry.count 2` reading `2` as a file filter is obvious on sight and invisible otherwise. Also: - `canonicalFlag` camelCases only the FIRST dot segment, matching cac's own `camelcaseOptionName` (`name.split(".").map((v,i) => i===0 ? camelcase(v) : v)`). Camel-casing all of them made this wrapper accept `--coverage.reports-directory`, a spelling vitest does not — so the two would disagree about the next token. - `--repo-root=<dir>` is parsed. Matching only the space form left the inline spelling falling through BOTH branches to the real repo root — byte-for-byte the failure the missing-value guard was added to close, reachable through one extra character. A value that is itself a flag is rejected too. Verification: 45 unit tests (was 44 after the round's own additions; 39 before). Mutation-checked at 23 mutants, ALL KILLED, zero survivors, each by its OWN named test. 🔴 THE FIRST SWEEP REPORTED FOUR SURVIVORS AND WAS WRONG ABOUT ALL FOUR, WHICH IS WORTH RECORDING BECAUSE THE HARNESS HAD THIS PR'S OWN DEFECT. Two were mislabelled — killed by a different test than the one named, which the sweep scores as SURVIVED. Of the other two, one was a real gap (`--reporter=json AppNameCrumb`, now covered) and one was `canonicalFlag`'s dot handling, which is equivalent for every input that has no dashed subkey (also now covered). Along the way the harness itself was found reusing a JSON report path without clearing it — the same stale-report defect fixed in the product two commits ago — and now clears it and verifies the mutation reached disk before running. A mutant reported SURVIVED is a claim about the instrument until the instrument has been controlled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * wip(tests): stop guessing which vitest flags take values Safety commit of in-progress round-5 rework so it is not lost; the agent was interrupted mid mutation-sweep. Verification is NOT complete — the sweep, the red/green pair and the merged-tree re-run have not been reported. Do not merge on this commit. Round 5 measured both prior approaches against vitest 4.1.11's real option table (170 long options, 74 boolean, 96 value-taking): the hand-maintained list was wrong 73 times, the shape heuristic 74. The error count never moved, only its direction. This removes the question instead of answering it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEa6GDJyTiu2R146ndYsLK * test(ci): finish the verificationbacb8a2cf5was pushed withoutbacb8a2cf5landed as a rescue commit marked `wip` because the session that wrote it died with the change only in a working tree. The change itself is unaltered; this is the verification it was missing, plus the merge of a main that moved twice underneath it. Nothing here modifies the rule. WHY THE RULE CHANGED, WITH THE NUMBER THAT JUSTIFIES IT. Round 5 of the audit enumerated vitest 4.1.11's REAL CLI option table -- by calling `createCLI()` and reading each cac option's `isBoolean` rather than by hand -- and ran `isNarrowed(['--<flag>', 'VALUE'])` against both revisions imported side by side: hand-maintained flag list wrong on 73 options (every value-taking one) QUIET shape heuristic wrong on 74 options (every boolean one) LOUD 73 versus 74. The heuristic did not beat the list; it moved the wrongness off one half of the table onto the other. The direction improved, which is worth something, but the error count did not -- and `pnpm test:component --coverage AppNameCrumb` would run one test and then be failed against the 1240 floor with "the include broke or the run died". That is a mis-posed question, so it is no longer asked: any argument at all means narrowed. Re-measured against the same table on the merged tree, with the same method: 164 long options (72 boolean, 92 value-taking) UNSAFE-LOUD -- rule claims FULL so the floor and ledger fire on a partial run: 0 no-argument invocation (what CI runs): NOT narrowed, so all three checks arm (My enumeration walks the global + `run` commands and sees 164/72/92 where round 5's saw 170/74/96; the traversal differs slightly, the conclusion does not.) 🔴 The comparison is one-directional ON PURPOSE, and the other direction is a real cost rather than a rounding error: all 164 options now score as narrowed, so an arg-ful run does not get the floor or the file ledger even when it was genuinely full. That is affordable for one measured reason -- `pr-preview-pipeline.yaml` invokes `pnpm run test:component` with NO arguments, so CI is the `argv.length === 0` path and always gets all three. What is given up is enforcing a floor on an ad-hoc local run. `VITEST_MAX_WORKERS=4` in the environment sizes a run without giving that up. Verification on the MERGED tree (d77dd394db), all at load <= 16 with no browser-session errors in any run quoted: no-arg full run (the CI path) 202 files / 2257 tests, exit 0, no NARROWED line, floor 1240 and the 202-file ledger both ARMED guard red arm exit 1, "THIS RUN PRODUCED NO ACCOUNTABLE RESULT", failing on the missing-report branch guard green arm exit 0, "2 executed, 0 skipped, across 1 files" gate unit tests 37 passed mutation sweep 20 mutants, ALL KILLED, zero survivors 🔴 The red arm was run NARROWED, deliberately: it proves `--narrowed` disables the floor and the ledger and NOT the zero-collected check, which is the one failure this whole change exists to catch. THE MERGE. main moved twice; both times the only conflict was `package.json`, and both times it was the same semantic hazard -- main editing `test:lint-rules` and this branch editing `test:component`, adjacent lines of one object, where taking either side wholesale silently reverts the other and makes the entire guard inert with every test green. Resolved by parsing the merged JSON, not by reading the diff: `test:lint-rules` now matches origin/main byte-for-byte and `test:component` is the wrapper. The wiring guard added for exactly this was then checked against the bad resolution -- applying main's `package.json` wholesale fails one test, the one written for it. Round 5's two remaining 🟢 items are closed by the rework itself rather than separately: the stale JSDoc described `VALUE_FLAGS`, which no longer exists, and the test whose description claimed "a BOOLEAN flag does not swallow the filename after it" while passing entirely through the path check is gone with the heuristic it tested. That one was load-bearing -- a description asserting coverage its body did not provide is what let the class through -- so its replacement asserts a property with no free parameters instead, across both halves of the option table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style(tests): format the gate test file The rescue commit was pushed before a prettier pass could run over the last edit to this file, and `ESLint + Prettier (changed files)` caught it: the file is ADDED by this PR, so it is covered by the added-files prettier gate. One line. This red was MINE, not inherited — unlike `Unit tests (1)/(2)` (a redis integration test that arrived from main) and `preview / smoke-tests` (#4516 changing `reaction.toggle`), both of which are verified as pre-existing and are left alone. CONTRIBUTING.md also reports unformatted, and that one is NOT actionable here: it is already unformatted on origin/main, and it is modified rather than added, so the gate (which reads `added.txt`) does not cover it. Reformatting it would bury this PR under an unrelated whole-file diff. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zachary Lowden
+63
-3
@@ -30,8 +30,11 @@ consequences:
|
||||
though they pass in isolation. So a red unit job is a signal to look, not proof
|
||||
you broke something, and a green overall check doesn't mean the tests passed.
|
||||
Read the job output.
|
||||
- **Component tests don't run in CI at all.** The 106 `*.browser.test.tsx` files
|
||||
need real Chromium and only execute when someone runs them locally.
|
||||
- **Component tests run in the PR-preview pipeline, report-only.** The 201
|
||||
`*.browser.test.tsx` files need real Chromium; they do not run in GitHub Actions,
|
||||
but the in-cluster preview pipeline runs them and posts a
|
||||
`preview / component-tests` commit status. It never blocks, so read it rather
|
||||
than relying on it.
|
||||
|
||||
A green check on a fork PR therefore means much less than it looks like. Verify
|
||||
locally.
|
||||
@@ -85,7 +88,7 @@ with `nix develop -c`, or use direnv (`cp .envrc.example .envrc && direnv allow`
|
||||
```bash
|
||||
pnpm typecheck # full repo
|
||||
pnpm test:unit:run # ~8,900 unit tests, node env
|
||||
pnpm test:component # ~1,300 component tests in real Chromium, slower
|
||||
pnpm test:component # ~2,250 component tests in real Chromium, slower
|
||||
pnpm exec prettier --check <files you added>
|
||||
pnpm exec eslint <files you added>
|
||||
```
|
||||
@@ -111,6 +114,63 @@ The reliable method for any of these commands is to run it on unmodified `main`
|
||||
first, save the output, then compare. A failure that also happens on `main` isn't
|
||||
yours.
|
||||
|
||||
### `pnpm test:component` fails on ZERO COLLECTED, not only on red tests
|
||||
|
||||
The browser suite has a failure mode that runs **no tests at all** and, until you
|
||||
read the numbers, looks exactly like an ordinary failure. `pnpm test:component`
|
||||
runs through `scripts/test-component-run.mjs`, which asks vitest for a JSON report
|
||||
and hands it to `scripts/ci/assert-component-suite-ran.mjs`. That gate prints a
|
||||
ledger and applies three checks:
|
||||
|
||||
```
|
||||
test:component ledger: 2254 executed, 0 skipped, across 201 files; 0 failed suites,
|
||||
0 failed tests (baseline 2254 tests / 201 files measured 2026-08-31; 201 on disk;
|
||||
floor 1240)
|
||||
```
|
||||
|
||||
1. **Nothing collected fails** — always, on every invocation.
|
||||
2. **A `*.browser.test.tsx` on disk that is absent from the report fails, and is
|
||||
named.** A file that stops being collected reports as *absence*, so no failure
|
||||
count and no per-test list can show it.
|
||||
3. **A floor on executed tests**, as a backstop for a partial collapse that still
|
||||
leaves every file present.
|
||||
|
||||
The gate can only ever *add* a failure — vitest's own exit code is passed straight
|
||||
through otherwise.
|
||||
|
||||
Three things to know before you run it with arguments:
|
||||
|
||||
- **Any argument at all skips checks 2 and 3. Check 1 always applies.** Passing
|
||||
*anything* — a filename, `--shard`, even `--max-workers 4` — means what the run
|
||||
should have collected is not knowable from the arguments, so the floor and the file
|
||||
ledger are not asserted. Run with **no arguments** to get all three; that is what CI
|
||||
does. To size a run without giving up the checks, use the environment instead:
|
||||
`VITEST_MAX_WORKERS=4 pnpm test:component`.
|
||||
(This is deliberately not a parser. Deciding which vitest flags consume the next
|
||||
token was tried twice and lost twice — measured against vitest's real option table,
|
||||
a hand-maintained flag list was wrong on 73 options and the shape heuristic that
|
||||
replaced it was wrong on 74. The rule above cannot be wrong about any of them.)
|
||||
- **`--outputFile` is refused** (exit 2), in every spelling that would collide with
|
||||
the report the gate reads. Run `pnpm exec vitest run --project component` directly
|
||||
if you want your own report.
|
||||
|
||||
Why it exists: a `vi.mock` factory that throws is resolved inside a Playwright
|
||||
route handler that does not catch, so the rejection escapes as an
|
||||
`Unhandled Rejection` in the orchestrator and kills the whole run — no
|
||||
`Test Files` line, no per-file results, exit 1. The `preview / component-tests`
|
||||
tier reads only the exit code, so that abort and a genuine list of red assertions
|
||||
rendered identically. One such file zeroed all 201 suites on `main` at
|
||||
`d353f785c3`. A browser crash under host load produces the same shape, wearing the
|
||||
same misleading headline. The gate's message enumerates the causes it cannot tell
|
||||
apart; read the error printed above it.
|
||||
|
||||
The commonest cause is a **wholesale** factory that stops naming an export
|
||||
something in the file's module graph imports. `local-rules/no-wholesale-module-mock`
|
||||
guards that for a listed set of modules, and
|
||||
`src/components/AppBlocks/__tests__/featureFlagsMockCompleteness.test.ts` guards
|
||||
the feature-flags module — but only under `src/components/AppBlocks`, so neither
|
||||
covers you by default.
|
||||
|
||||
## Where tests go
|
||||
|
||||
**Never put test files under `src/pages`.** Next.js treats every `.ts`/`.tsx`
|
||||
|
||||
+1
-1
@@ -103,7 +103,7 @@
|
||||
"test:apps": "vitest --project 'app:*'",
|
||||
"test:apps:run": "vitest run --project 'app:*'",
|
||||
"test:lint-rules": "vitest run --project 'unit*' src/server/notifications/__tests__/notification-settings-polarity.test.ts src/server/schema/__tests__/track.addView.schema.test.ts src/server/services/__tests__/hub-filter-parity.test.ts src/server/services/__tests__/no-agent-ground-truth-write.test.ts src/server/services/__tests__/no-coerce-boolean-in-api.test.ts src/server/services/__tests__/no-direct-shared-module-mock.test.ts src/server/services/__tests__/no-doubled-free-slot-noun.test.ts src/server/services/__tests__/no-hand-typed-redis-key-constants.test.ts src/server/services/__tests__/no-io-in-transaction.test.ts src/server/services/__tests__/no-lint-rules-script-drift.test.ts src/server/services/__tests__/no-module-scope-cache.test.ts src/server/services/__tests__/no-pk-addressed-engagement-write.test.ts src/server/services/__tests__/no-server-infra-in-app-graph.test.ts src/server/services/__tests__/no-sharp-outside-native-project.test.ts src/server/services/__tests__/no-stale-moderator-route-probe.test.ts src/server/services/__tests__/no-static-html2canvas-import.test.ts src/server/services/__tests__/no-unbounded-paging-fake.test.ts src/server/services/__tests__/no-unguarded-billable-submit.test.ts src/server/services/__tests__/no-unguarded-user-text.test.ts src/server/services/__tests__/no-unloadable-image-fixture.test.ts src/server/services/__tests__/no-unmuteable-comment-processor.test.ts src/server/services/__tests__/no-unpriced-default-model.test.ts src/server/services/__tests__/no-unverified-provenance-write.test.ts src/server/services/__tests__/no-unwrapped-knob-rotation.test.ts src/server/services/__tests__/no-wholesale-module-mock.test.ts src/server/services/__tests__/poi-checks-strip-benign-phrases.test.ts src/server/services/__tests__/video-leaderboard-badge-staging.test.ts",
|
||||
"test:component": "vitest run --project component",
|
||||
"test:component": "node scripts/test-component-run.mjs",
|
||||
"test:component:watch": "vitest --project component",
|
||||
"meilisearch:migrate": "NODE_ENV=development tsx scripts/oneoffs/meilisearch-migration.ts",
|
||||
"tsscript": "NODE_ENV=development tsx",
|
||||
|
||||
@@ -0,0 +1,645 @@
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
canonicalFlag,
|
||||
conflictingOutputFile,
|
||||
exitCodeForSignal,
|
||||
isNarrowed,
|
||||
main,
|
||||
narrowingReason,
|
||||
} from '../test-component-run.mjs';
|
||||
|
||||
/**
|
||||
* Tests for the `preview / component-tests` positive control
|
||||
* (scripts/ci/assert-component-suite-ran.mjs) and the one branch of its caller that can be
|
||||
* wrong invisibly (`isNarrowed`).
|
||||
*
|
||||
* 🔴 WHAT THIS GUARD IS FOR, restated because it decides every case below: the browser
|
||||
* suite can abort having executed ZERO tests, and the CI tier reads only the exit code, so
|
||||
* an abort and a genuine list of red assertions render identically. The guard's whole job
|
||||
* is to make "collected nothing" say so. A test suite for it therefore has to pin the
|
||||
* ZERO case and the boundary, not just the happy path.
|
||||
*
|
||||
* The cases below are chosen so each kills a specific mutation of the guard:
|
||||
*
|
||||
* - dropping 'failed' from the EXECUTED set -> 'a run that is entirely RED still counts'
|
||||
* - `executed === 0` -> `executed < 0` -> 'zero executed fails'
|
||||
* - floor `<` -> `<=` -> 'exactly at the floor passes'
|
||||
* - floor `<` -> `>` -> 'one below the floor fails'
|
||||
* - skipping the narrowed check -> 'a narrowed run skips the floor'
|
||||
* - narrowed short-circuiting the zero check -> 'a narrowed run still fails on zero'
|
||||
* - treating a missing report as nothing-to-do-> 'a missing report is a failure'
|
||||
*
|
||||
* 🔴 The fixture numbers are deliberately NOT round multiples of the floor and NOT equal to
|
||||
* any constant the guard names, except where a case is specifically pinning that boundary.
|
||||
*/
|
||||
|
||||
const SCRIPT = resolve(__dirname, '../ci/assert-component-suite-ran.mjs');
|
||||
|
||||
/** Kept in step with MIN_TESTS in the script under test. */
|
||||
const MIN_TESTS = 1240;
|
||||
|
||||
let dir: string;
|
||||
beforeAll(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), 'assert-component-suite-ran-'));
|
||||
});
|
||||
afterAll(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
type FileSpec = { name: string; passed?: number; failed?: number; skipped?: number };
|
||||
|
||||
/** Build a vitest-shaped JSON report. `files` with no assertions model an import failure. */
|
||||
function writeReport(name: string, files: FileSpec[]): string {
|
||||
const testResults = files.map((f) => {
|
||||
const assertionResults = [
|
||||
...Array.from({ length: f.passed ?? 0 }, () => ({ status: 'passed' })),
|
||||
...Array.from({ length: f.failed ?? 0 }, () => ({ status: 'failed' })),
|
||||
...Array.from({ length: f.skipped ?? 0 }, () => ({ status: 'skipped' })),
|
||||
];
|
||||
return {
|
||||
name: f.name,
|
||||
status: (f.failed ?? 0) > 0 || assertionResults.length === 0 ? 'failed' : 'passed',
|
||||
assertionResults,
|
||||
};
|
||||
});
|
||||
const path = join(dir, name);
|
||||
writeFileSync(
|
||||
path,
|
||||
JSON.stringify({
|
||||
numFailedTestSuites: testResults.filter((r) => r.status === 'failed').length,
|
||||
numFailedTests: files.reduce((n, f) => n + (f.failed ?? 0), 0),
|
||||
numTotalTests: testResults.reduce((n, r) => n + r.assertionResults.length, 0),
|
||||
testResults,
|
||||
})
|
||||
);
|
||||
return path;
|
||||
}
|
||||
|
||||
/** One big healthy file is enough — the guard counts assertions, not files. */
|
||||
const healthy = (n: number, extra: FileSpec[] = []): FileSpec[] => [
|
||||
{ name: 'src/components/X.browser.test.tsx', passed: n },
|
||||
...extra,
|
||||
];
|
||||
|
||||
/**
|
||||
* 🔴 EVERY CASE PASSES `--repo-root`, AND IT MUST.
|
||||
*
|
||||
* The gate also compares the report's file list against every `*.browser.test.tsx` on disk.
|
||||
* Left pointing at the real repo, that check grades a two-file FIXTURE against 201 real files
|
||||
* and every case below fails for a reason none of them is about. Pointing it at a directory
|
||||
* with no `src/` makes the walk return "unavailable", which the gate skips — and prints, so a
|
||||
* reader can see which checks a given run actually applied. The on-disk ledger has its own
|
||||
* fixture tree and its own cases further down.
|
||||
*/
|
||||
function runGate(reportPath: string, ...args: string[]) {
|
||||
const r = spawnSync(process.execPath, [SCRIPT, reportPath, '--repo-root', dir, ...args], {
|
||||
encoding: 'utf8',
|
||||
});
|
||||
return { code: r.status, out: `${r.stdout}${r.stderr}` };
|
||||
}
|
||||
|
||||
/**
|
||||
* Same, but pointed at a caller-supplied tree so the on-disk ledger is armed.
|
||||
*
|
||||
* 🔴 `--repo-root` goes FIRST here on purpose. Picking the report as "the first non-flag
|
||||
* argument" read the DIRECTORY as the report path whenever the flag preceded it — measured:
|
||||
* "EISDIR: illegal operation on a directory" plus the whole abort diagnosis. Every earlier
|
||||
* case passed it last, which is precisely why nothing caught it.
|
||||
*/
|
||||
function runGateAt(reportPath: string, root: string, ...args: string[]) {
|
||||
const r = spawnSync(process.execPath, [SCRIPT, '--repo-root', root, reportPath, ...args], {
|
||||
encoding: 'utf8',
|
||||
});
|
||||
return { code: r.status, out: `${r.stdout}${r.stderr}` };
|
||||
}
|
||||
|
||||
describe('assert-component-suite-ran', () => {
|
||||
it('a healthy full run passes and prints the ledger', () => {
|
||||
const report = writeReport('healthy.json', healthy(2254));
|
||||
const { code, out } = runGate(report);
|
||||
expect(out).toContain('2254 executed');
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
it('zero executed FAILS, and says the run verified nothing', () => {
|
||||
// The whole point: exit code 0 or 1 from the runner is irrelevant, the guard decides on
|
||||
// what was collected. Two files present, neither contributing an assertion — exactly the
|
||||
// shape a collection-time abort leaves behind.
|
||||
const report = writeReport('zero.json', [
|
||||
{ name: 'src/a.browser.test.tsx' },
|
||||
{ name: 'src/b.browser.test.tsx' },
|
||||
]);
|
||||
const { code, out } = runGate(report);
|
||||
expect(out).toContain('NO ACCOUNTABLE RESULT');
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
it('a run that is entirely RED still counts as having run', () => {
|
||||
// 🔴 Kills the mutant that drops 'failed' from EXECUTED. A guard that treated red as
|
||||
// "did not run" would fire on every genuine failure — the tier would then be red for
|
||||
// two different reasons at once and nobody could tell them apart, which is the exact
|
||||
// confusion this whole change exists to remove.
|
||||
const report = writeReport('allred.json', [{ name: 'src/a.browser.test.tsx', failed: 1873 }]);
|
||||
const { code, out } = runGate(report);
|
||||
expect(out).toContain('1873 executed');
|
||||
expect(out).not.toContain('NO ACCOUNTABLE RESULT');
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
it('SKIPPED tests do not count towards the floor', () => {
|
||||
// 🔴 Kills a `numTotalTests`-based floor. 1901 total, of which only 61 executed: a
|
||||
// total-based check waves this through, and a suite that self-skips wholesale is
|
||||
// exactly as unverified as one that aborted.
|
||||
const report = writeReport('skipped.json', [
|
||||
{ name: 'src/a.browser.test.tsx', passed: 61, skipped: 1840 },
|
||||
]);
|
||||
const { code, out } = runGate(report);
|
||||
expect(out).toContain('61 executed, 1840 skipped');
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
it('exactly at the floor passes', () => {
|
||||
const report = writeReport('atfloor.json', healthy(MIN_TESTS));
|
||||
expect(runGate(report).code).toBe(0);
|
||||
});
|
||||
|
||||
it('one below the floor fails, and names the floor', () => {
|
||||
const report = writeReport('belowfloor.json', healthy(MIN_TESTS - 1));
|
||||
const { code, out } = runGate(report);
|
||||
expect(out).toContain(`EXECUTED ONLY ${MIN_TESTS - 1} TESTS`);
|
||||
expect(out).toContain('Do NOT lower the floor');
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
it('a narrowed run skips the floor', () => {
|
||||
// The fast iteration loop runs one file. 7 is far below the floor and must pass here.
|
||||
const report = writeReport('narrow.json', healthy(7));
|
||||
const { code, out } = runGate(report, '--narrowed');
|
||||
expect(out).toContain('floor SKIPPED');
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
it('a narrowed run STILL fails on zero collected', () => {
|
||||
// 🔴 Kills the mutant where `--narrowed` short-circuits the whole gate. A single-file
|
||||
// run that collects nothing is the cheapest reproduction of the abort, and is precisely
|
||||
// when someone is debugging it — the guard must not go quiet there.
|
||||
const report = writeReport('narrowzero.json', [{ name: 'src/a.browser.test.tsx' }]);
|
||||
const { code, out } = runGate(report, '--narrowed');
|
||||
expect(out).toContain('NO ACCOUNTABLE RESULT');
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
it('names the files that failed WITHOUT running an assertion', () => {
|
||||
// The per-file version of the abort: the file failed to IMPORT, so its tests did not
|
||||
// run at all, and both the failure count and the per-test list read as clean.
|
||||
const report = writeReport('importfail.json', [
|
||||
{ name: 'src/components/Good.browser.test.tsx', passed: 1300 },
|
||||
{ name: 'src/components/Broken.browser.test.tsx' },
|
||||
]);
|
||||
const { code, out } = runGate(report);
|
||||
expect(out).toContain('FAILED WITHOUT RUNNING A SINGLE ASSERTION');
|
||||
expect(out).toContain('src/components/Broken.browser.test.tsx');
|
||||
// Still a pass overall — the runner's own exit code owns that verdict, this is a NAME
|
||||
// for the shape, not a second gate.
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
it('a MISSING report is a failure, not a no-op', () => {
|
||||
// 🔴 The case the guard was written for: a run that dies early enough writes no report.
|
||||
// Treating that as "nothing to check" would make the guard silent exactly when it is
|
||||
// needed.
|
||||
const { code, out } = runGate(join(dir, 'does-not-exist.json'));
|
||||
expect(out).toContain('does not exist');
|
||||
expect(out).toContain('NO ACCOUNTABLE RESULT');
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
it('--repo-root with no usable value is a USAGE ERROR, not a silent fall-through', () => {
|
||||
// 🔴 It used to fall back to the script's own repo, so a fixture report was graded against
|
||||
// the 201 REAL files and failed with a confident diagnosis about the include breaking —
|
||||
// produced by a typo. All three shapes are pinned because the first fix covered only the
|
||||
// first: `--repo-root=<dir>` fell through BOTH branches and reached the same wrong answer
|
||||
// through one extra character.
|
||||
const report = writeReport('reporoot-noval.json', healthy(1500));
|
||||
for (const args of [['--repo-root'], ['--repo-root='], ['--repo-root', '--narrowed']]) {
|
||||
const r = spawnSync(process.execPath, [SCRIPT, report, ...args], { encoding: 'utf8' });
|
||||
expect(`${r.stdout}${r.stderr}`, args.join(' ')).toContain(
|
||||
'--repo-root requires a directory'
|
||||
);
|
||||
expect(r.status, args.join(' ')).toBe(2);
|
||||
}
|
||||
});
|
||||
|
||||
it('--repo-root=<dir> is honoured, not ignored', () => {
|
||||
// The inline spelling must actually WORK, not merely be rejected when empty — otherwise
|
||||
// the fix above would be a refusal where a feature belongs.
|
||||
const report = writeReport('reporoot-inline.json', healthy(1500));
|
||||
const r = spawnSync(process.execPath, [SCRIPT, report, `--repo-root=${dir}`], {
|
||||
encoding: 'utf8',
|
||||
});
|
||||
expect(`${r.stdout}${r.stderr}`).toContain('on-disk count UNAVAILABLE');
|
||||
expect(r.status).toBe(0);
|
||||
});
|
||||
|
||||
it('an UNPARSEABLE report is a failure', () => {
|
||||
const path = join(dir, 'garbage.json');
|
||||
writeFileSync(path, '{ this is not json');
|
||||
const { code, out } = runGate(path);
|
||||
expect(out).toContain('not valid JSON');
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
it('the harness itself can produce a red verdict — negative control', () => {
|
||||
// Without this, every green above is equally consistent with a gate wired to nothing.
|
||||
// `runGate` has produced a 1 in the cases above; this pins that the SAME invocation
|
||||
// shape returns 0 and 1 for two inputs that differ only in what was collected.
|
||||
const green = writeReport('control-green.json', healthy(1500));
|
||||
const red = writeReport('control-red.json', healthy(0));
|
||||
expect([runGate(green).code, runGate(red).code]).toEqual([0, 1]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the on-disk ledger — a file that stops being COLLECTED', () => {
|
||||
/**
|
||||
* 🔴 THE STRONGER OF THE TWO CHECKS, and the reason it exists: the test floor sits at ~55%,
|
||||
* so ~45% of the suite can stop being collected while the gate stays green. The incident
|
||||
* this whole guard descends from is exactly that shape — six files contributing 0 of 438
|
||||
* with nothing red. This catches ONE file going missing.
|
||||
*/
|
||||
let tree: string;
|
||||
beforeAll(() => {
|
||||
tree = mkdtempSync(join(tmpdir(), 'component-ondisk-'));
|
||||
mkdirSync(join(tree, 'src/components/Deep'), { recursive: true });
|
||||
mkdirSync(join(tree, 'src/node_modules/pkg'), { recursive: true });
|
||||
for (const p of [
|
||||
'src/components/A.browser.test.tsx',
|
||||
'src/components/Deep/B.browser.test.tsx',
|
||||
'src/components/Deep/C.browser.test.tsx',
|
||||
]) {
|
||||
writeFileSync(join(tree, p), '// fixture\n');
|
||||
}
|
||||
// Neither of these may be counted: the walk must skip `node_modules`, and a `.test.tsx`
|
||||
// that is not a `.browser.test.tsx` belongs to the node project, not this one. If either
|
||||
// leaked in, the ledger would demand a file the component run can never collect.
|
||||
writeFileSync(join(tree, 'src/node_modules/pkg/D.browser.test.tsx'), '// fixture\n');
|
||||
writeFileSync(join(tree, 'src/components/E.test.tsx'), '// fixture\n');
|
||||
});
|
||||
afterAll(() => rmSync(tree, { recursive: true, force: true }));
|
||||
|
||||
const all = [
|
||||
'src/components/A.browser.test.tsx',
|
||||
'src/components/Deep/B.browser.test.tsx',
|
||||
'src/components/Deep/C.browser.test.tsx',
|
||||
];
|
||||
|
||||
it('passes when every file on disk appears in the report', () => {
|
||||
const report = writeReport(
|
||||
'ondisk-all.json',
|
||||
all.map((name, i) => ({ name: join(tree, name), passed: 500 + i }))
|
||||
);
|
||||
const { code, out } = runGateAt(report, tree);
|
||||
expect(out).toContain('3 on disk');
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
it('FAILS when one file is missing, and NAMES it', () => {
|
||||
// 1502 executed is comfortably above the floor, so the floor cannot be what fails this —
|
||||
// if the on-disk check were deleted this case would go green.
|
||||
const report = writeReport(
|
||||
'ondisk-missing.json',
|
||||
all.slice(0, 2).map((name) => ({ name: join(tree, name), passed: 751 }))
|
||||
);
|
||||
const { code, out } = runGateAt(report, tree);
|
||||
expect(out).toContain('ABSENT FROM THE RUN');
|
||||
expect(out).toContain('Deep/C.browser.test.tsx');
|
||||
expect(out).not.toContain('EXECUTED ONLY');
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
it('a NARROWED run does not trip it', () => {
|
||||
const report = writeReport('ondisk-narrow.json', [{ name: join(tree, all[0]), passed: 4 }]);
|
||||
expect(runGateAt(report, tree, '--narrowed').code).toBe(0);
|
||||
});
|
||||
|
||||
it('says so, and skips, when there is no tree at all', () => {
|
||||
// A ZERO from a walk that found nothing is indistinguishable from a suite with no files,
|
||||
// so the gate must not build a ledger on it — but it must SAY it did not, or a reader
|
||||
// cannot tell which checks this run applied.
|
||||
const report = writeReport('ondisk-unavailable.json', healthy(1500));
|
||||
const { code, out } = runGateAt(report, join(dir, 'no-such-tree'));
|
||||
expect(out).toContain('on-disk count UNAVAILABLE');
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
it('says so, and skips, when `src/` EXISTS but holds no browser tests', () => {
|
||||
// 🔴 THIS CASE EXISTS BECAUSE THE OTHER ONE CANNOT REACH THE CODE IT LOOKS LIKE IT
|
||||
// COVERS. With no `src/` at all the walk returns early, so the "an empty result is not a
|
||||
// measurement" line below it never executes — a mutation sweep proved it: turning
|
||||
// `out.length > 0 ? out : null` into a bare `out` SURVIVED a green suite. An empty array
|
||||
// is TRUTHY, so that mutant would arm a ledger over zero expected files, which passes
|
||||
// everything while reading as a check that ran.
|
||||
const emptyTree = mkdtempSync(join(tmpdir(), 'component-ondisk-empty-'));
|
||||
mkdirSync(join(emptyTree, 'src/components'), { recursive: true });
|
||||
writeFileSync(join(emptyTree, 'src/components/NotATest.tsx'), '// fixture\n');
|
||||
try {
|
||||
const report = writeReport('ondisk-empty-src.json', healthy(1500));
|
||||
const { code, out } = runGateAt(report, emptyTree);
|
||||
expect(out).toContain('on-disk count UNAVAILABLE');
|
||||
expect(out).not.toContain('0 on disk');
|
||||
expect(code).toBe(0);
|
||||
} finally {
|
||||
rmSync(emptyTree, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('isNarrowed / narrowingReason', () => {
|
||||
/**
|
||||
* 🔴 THIS BLOCK USED TO PIN A PARSER, AND THE PARSER IS GONE. Read the header of
|
||||
* `narrowingReason` for why: measured against vitest 4.1.11's REAL option table (170 long
|
||||
* options, 74 boolean, 96 value-taking, enumerated by calling `createCLI()` and reading each
|
||||
* cac option's `isBoolean`), the hand-maintained flag list was wrong on 73 and the shape
|
||||
* heuristic that replaced it was wrong on 74. It moved the wrongness rather than removing it.
|
||||
*
|
||||
* So the rule is now "any argument at all means narrowed", and these cases pin THAT — a
|
||||
* property with no free parameters, rather than a heuristic with a long tail.
|
||||
*/
|
||||
it('NO arguments is a full run — the CI invocation, and the only one that gets all three checks', () => {
|
||||
expect(isNarrowed([])).toBe(false);
|
||||
expect(narrowingReason([])).toBeNull();
|
||||
});
|
||||
|
||||
it('ANY argument narrows, whatever it is', () => {
|
||||
// 🔴 Deliberately spans both halves of the table that the two previous rules each got
|
||||
// wrong: BOOLEAN flags (which the shape rule mis-read as consuming their neighbour) and
|
||||
// VALUE-taking flags (which the list mis-read the same way), plus bare filters. Under this
|
||||
// rule every one is the same answer, which is the point — there is no set of options for
|
||||
// which it can be wrong.
|
||||
for (const argv of [
|
||||
['src/components/X.browser.test.tsx'], // a path filter
|
||||
['AppNameCrumb'], // a bare substring filter
|
||||
['--coverage'], // boolean, no value
|
||||
['--coverage', 'AppNameCrumb'], // boolean + filter — the shape rule got this wrong
|
||||
['--run'],
|
||||
['--silent'],
|
||||
['--update'],
|
||||
['--max-workers', '4'], // value-taking + value — the list got this wrong
|
||||
['--reporter', './my-reporter.js'], // value-taking with a PATH value
|
||||
['--retry', '-1', 'AppNameCrumb'], // a negative-number value
|
||||
['--shard=1/4'],
|
||||
['--changed'],
|
||||
['-t', 'renders'],
|
||||
['--'],
|
||||
[''],
|
||||
]) {
|
||||
expect(isNarrowed(argv), JSON.stringify(argv)).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('the reason NAMES every argument, so skipping the checks is never quiet', () => {
|
||||
// The reason is printed by the runner. It must identify what caused the skip, because
|
||||
// `--narrowed` disables the floor and the on-disk ledger.
|
||||
const reason = narrowingReason(['--max-workers', '4']);
|
||||
expect(reason).toContain('--max-workers');
|
||||
expect(reason).toContain('4');
|
||||
expect(reason).toContain('not knowable');
|
||||
});
|
||||
|
||||
it('canonicalFlag leaves dot SUBKEYS verbatim, exactly as cac does', () => {
|
||||
// Still used by `conflictingOutputFile`, which has to recognise ONE flag by name. cac's
|
||||
// `camelcaseOptionName` camelCases only the first segment, so camel-casing the rest would
|
||||
// make this wrapper accept `--coverage.reports-directory`, a spelling vitest does not.
|
||||
expect(canonicalFlag('--coverage.reports-directory')).toBe('coverage.reports-directory');
|
||||
expect(canonicalFlag('--max-workers')).toBe('maxWorkers');
|
||||
expect(canonicalFlag('--output-file.json=/x')).toBe('outputFile.json');
|
||||
expect(canonicalFlag('src/x.browser.test.tsx')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('conflictingOutputFile', () => {
|
||||
it('catches every spelling that would redirect the report', () => {
|
||||
// 🔴 Measured before this existed: an 18/18 GREEN single-file run reported an abort that
|
||||
// "verified NOTHING" and exited 1, because `--outputFile=<p>` clobbered the
|
||||
// `--outputFile.json=` form the wrapper appends. `.github/workflows/lint.yml` runs the
|
||||
// SIBLING unit tier with exactly that flag, so this is a copy-paste away.
|
||||
//
|
||||
// 🔴 The KEBAB form is here because vitest accepts it and the first version of this guard
|
||||
// did not catch it — so the test whose name claimed "every spelling" covered four of five.
|
||||
expect(conflictingOutputFile(['--outputFile=/tmp/x.json'])).toBe('--outputFile=/tmp/x.json');
|
||||
expect(conflictingOutputFile(['--outputFile.json=/tmp/x.json'])).toBe(
|
||||
'--outputFile.json=/tmp/x.json'
|
||||
);
|
||||
expect(conflictingOutputFile(['--outputFile', '/tmp/x.json'])).toBe('--outputFile');
|
||||
expect(conflictingOutputFile(['--output-file=/tmp/x.json'])).toBe('--output-file=/tmp/x.json');
|
||||
expect(conflictingOutputFile(['--output-file', '/tmp/x.json'])).toBe('--output-file');
|
||||
});
|
||||
|
||||
it('does not fire on the ordinary flags, or on a NON-json output key', () => {
|
||||
expect(conflictingOutputFile([])).toBeNull();
|
||||
expect(conflictingOutputFile(['--max-workers=4', '--reporter=verbose'])).toBeNull();
|
||||
// Not a prefix match: a different flag that merely starts the same way must pass.
|
||||
expect(conflictingOutputFile(['--outputFileSomethingElse=1'])).toBeNull();
|
||||
// 🔴 Object-form output paths are PER REPORTER, so a junit or html path does not touch the
|
||||
// `.json` key this wrapper writes. Refusing them was over-strict — a legitimate invocation
|
||||
// rejected by a guard whose stated reason ("the bare form sets the path for EVERY
|
||||
// reporter") is true only of the bare form.
|
||||
expect(conflictingOutputFile(['--outputFile.junit=/tmp/j.xml'])).toBeNull();
|
||||
expect(conflictingOutputFile(['--outputFile.html=/tmp/h'])).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('exitCodeForSignal', () => {
|
||||
it('maps each signal to the shell 128+N, not a constant', () => {
|
||||
// 🔴 This was a hardcoded 143 for EVERY signal, which re-created the exact mislabelling
|
||||
// this whole change exists to remove: the CI task branches on 137 to report `oom-killed`
|
||||
// ("raise the task's memory limit"), and a constant 143 matches no branch and falls
|
||||
// through to `fail` — a memory problem rendered as a test failure, on the same tier, in
|
||||
// the same words. The three are asserted separately because a mutant returning any single
|
||||
// constant must fail at least two of them.
|
||||
expect(exitCodeForSignal('SIGKILL')).toBe(137);
|
||||
expect(exitCodeForSignal('SIGTERM')).toBe(143);
|
||||
expect(exitCodeForSignal('SIGINT')).toBe(130);
|
||||
});
|
||||
|
||||
it('falls back to 143 for a name node does not know', () => {
|
||||
expect(exitCodeForSignal('SIGNOTAREALSIGNAL')).toBe(143);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the WIRING — package.json must actually invoke the wrapper', () => {
|
||||
/**
|
||||
* 🔴 THE SEAM NOBODY OWNS, AND IT WAS ALMOST SEVERED BY A MERGE.
|
||||
*
|
||||
* Every other test in this file exercises the scripts DIRECTLY. Not one of them loads
|
||||
* `package.json`, so if `test:component` stops pointing at the wrapper the entire guard goes
|
||||
* inert — no zero-collected check, no file ledger, no floor — and all 45 of them stay green.
|
||||
* The tier would go back to reporting an abort as "Component suite failed" with nothing to
|
||||
* say it had tested nothing, which is the exact state this change exists to end.
|
||||
*
|
||||
* Not hypothetical. Merging `origin/main` into this branch produced a conflict in
|
||||
* `package.json` where main had edited `test:lint-rules` and this branch had edited
|
||||
* `test:component` — ADJACENT LINES of the same object. Taking either side wholesale silently
|
||||
* reverts one of the two, and the "take theirs" resolution is the one that looks safest and
|
||||
* kills this feature. A textual conflict made it loud that time; the next overlap may not.
|
||||
*/
|
||||
it('`pnpm test:component` runs scripts/test-component-run.mjs, not vitest directly', () => {
|
||||
const pkg = JSON.parse(readFileSync(resolve(__dirname, '../../package.json'), 'utf8'));
|
||||
const script = pkg.scripts['test:component'];
|
||||
// Asserted as the WHOLE normalised string, not a substring: a partial match is satisfied by
|
||||
// `vitest run --project component && node scripts/test-component-run.mjs` and by anything
|
||||
// else that merely mentions the file.
|
||||
expect(script.trim()).toBe('node scripts/test-component-run.mjs');
|
||||
});
|
||||
|
||||
it('the script it names EXISTS and exports the entry point', async () => {
|
||||
// A wiring assertion that only reads package.json can pass over a deleted file.
|
||||
const target = resolve(__dirname, '../test-component-run.mjs');
|
||||
expect(existsSync(target)).toBe(true);
|
||||
expect(typeof main).toBe('function');
|
||||
});
|
||||
});
|
||||
|
||||
describe('main — the ORDER of effects, which no output can show', () => {
|
||||
/**
|
||||
* 🔴 THESE PIN SEQUENCING, WHICH IS WHY THEY EXIST AT ALL. The two `clearReport()` calls are
|
||||
* byte-identical statements; only their POSITION carries the meaning, so a refactor that
|
||||
* moves the first below the run reopens "a stale report satisfies the next run's ledger" —
|
||||
* the gate silently inert in exactly its own use case — with every other test still green.
|
||||
* Both this and the spawn-failure short-circuit shipped with no coverage.
|
||||
*/
|
||||
const ok = { rc: 0, signal: null, spawnFailed: false };
|
||||
|
||||
/**
|
||||
* 🔴 THE FAKES CAPTURE THEIR ARGV, AND THAT IS NOT DECORATION. An earlier version took no
|
||||
* parameter, so nothing in this file could observe the command `main` actually builds — and
|
||||
* a mutation battery found THREE survivors, each of them this PR's own headline failure
|
||||
* mode, all with 35 tests green:
|
||||
* - dropping `--narrowed` from the gate argv → every narrowed run hard-fails the floor
|
||||
* and the on-disk ledger;
|
||||
* - dropping `--outputFile.json=` from vitest → every run reports "does not exist" plus
|
||||
* the whole abort diagnosis;
|
||||
* - dropping `...argv` from vitest → `pnpm test:component <file>` silently
|
||||
* runs the WHOLE suite.
|
||||
* The argv is the seam between the two modules this change exists to wire together, and
|
||||
* testing `isNarrowed` in isolation cannot see whether its answer is ever USED.
|
||||
*/
|
||||
function harness(vitestResult: Record<string, unknown>, gateResult = ok) {
|
||||
const order: string[] = [];
|
||||
const seen: { vitest: string[]; gate: string[] } = { vitest: [], gate: [] };
|
||||
return {
|
||||
order,
|
||||
seen,
|
||||
make: (argv: string[]) => ({
|
||||
argv,
|
||||
clear: () => order.push('clear'),
|
||||
runVitest: async (a: string[]) => {
|
||||
order.push('vitest');
|
||||
seen.vitest = a;
|
||||
return vitestResult;
|
||||
},
|
||||
runGate: async (a: string[]) => {
|
||||
order.push('gate');
|
||||
seen.gate = a;
|
||||
return gateResult;
|
||||
},
|
||||
log: () => undefined,
|
||||
}),
|
||||
get opts() {
|
||||
return this.make([]);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
it('clears the stale report BEFORE starting the runner', async () => {
|
||||
const h = harness(ok);
|
||||
expect(await main(h.opts)).toBe(0);
|
||||
expect(h.order).toEqual(['clear', 'vitest', 'gate', 'clear']);
|
||||
expect(h.order.indexOf('clear')).toBeLessThan(h.order.indexOf('vitest'));
|
||||
});
|
||||
|
||||
it('a SIGNAL death returns the runner status and never consults the gate', async () => {
|
||||
// Consulting the gate here would read a report that a killed run never wrote and call it
|
||||
// "collected nothing" — relabelling a truncation as an abort, which is a wrong answer
|
||||
// rather than a missing one, and would break the CI task's timeout/oom branches.
|
||||
const h = harness({ rc: 137, signal: 'SIGKILL', spawnFailed: false });
|
||||
expect(await main(h.opts)).toBe(137);
|
||||
expect(h.order).toEqual(['clear', 'vitest']);
|
||||
});
|
||||
|
||||
it('a failed SPAWN returns without consulting the gate', async () => {
|
||||
const h = harness({ rc: 127, signal: null, spawnFailed: true });
|
||||
expect(await main(h.opts)).toBe(127);
|
||||
expect(h.order).toEqual(['clear', 'vitest']);
|
||||
});
|
||||
|
||||
it('a runner that legitimately EXITS 127 is still graded', async () => {
|
||||
// 🔴 Keyed on the spawn flag, not on the number. 127 is an exit code a runner can produce
|
||||
// on its own; treating it as "the binary could not be started" is a second wrong answer,
|
||||
// and it skips the gate on a run that did happen.
|
||||
const h = harness({ rc: 127, signal: null, spawnFailed: false });
|
||||
expect(await main(h.opts)).toBe(127);
|
||||
expect(h.order).toEqual(['clear', 'vitest', 'gate', 'clear']);
|
||||
});
|
||||
|
||||
it('the gate can only ADD a failure — it never turns red into green', async () => {
|
||||
const red = harness({ rc: 1, signal: null, spawnFailed: false }, ok);
|
||||
expect(await main(red.opts)).toBe(1);
|
||||
|
||||
const gateRed = harness(ok, { rc: 1, signal: null, spawnFailed: false });
|
||||
expect(await main(gateRed.opts)).toBe(1);
|
||||
});
|
||||
|
||||
it('the gate is told --narrowed exactly when isNarrowed says so — the answer is USED', async () => {
|
||||
// 🔴 Kills the mutation that drops `--narrowed` from the gate argv. Without it, every
|
||||
// narrowed run hard-fails the floor AND the on-disk ledger — the loudest wrong answer this
|
||||
// wrapper can give, and one that 35 tests of `isNarrowed` in isolation cannot see.
|
||||
const narrow = harness(ok);
|
||||
expect(await main(narrow.make(['src/components/X.browser.test.tsx']))).toBe(0);
|
||||
expect(narrow.seen.gate).toContain('--narrowed');
|
||||
|
||||
// 🔴 The full-run case is the EMPTY argv, and only that. It used to be
|
||||
// `['--max-workers', '4']`, which the old parser scored as full; under the current rule any
|
||||
// argument narrows, so the empty invocation is the only one that arms the floor and the
|
||||
// ledger — and it is exactly what `pr-preview-pipeline.yaml` runs.
|
||||
const full = harness(ok);
|
||||
expect(await main(full.make([]))).toBe(0);
|
||||
expect(full.seen.gate).not.toContain('--narrowed');
|
||||
});
|
||||
|
||||
it('the runner is given the JSON report path AND the caller arguments', async () => {
|
||||
// 🔴 Two more mutants that survived a green suite. Drop `--outputFile.json=` and every run
|
||||
// reports "does not exist" plus the abort diagnosis; drop `...argv` and
|
||||
// `pnpm test:component <file>` silently runs the whole suite while printing a ledger that
|
||||
// looks entirely healthy.
|
||||
const h = harness(ok);
|
||||
expect(await main(h.make(['src/components/X.browser.test.tsx', '--bail', '1']))).toBe(0);
|
||||
expect(h.seen.vitest.filter((a) => a.startsWith('--outputFile.json='))).toHaveLength(1);
|
||||
expect(h.seen.vitest.slice(-3)).toEqual(['src/components/X.browser.test.tsx', '--bail', '1']);
|
||||
// …and it is still the component project that runs.
|
||||
expect(h.seen.vitest.slice(0, 3)).toEqual(['run', '--project', 'component']);
|
||||
});
|
||||
|
||||
it('a conflicting --outputFile refuses BEFORE anything runs', async () => {
|
||||
const order: string[] = [];
|
||||
const rc = await main({
|
||||
argv: ['--outputFile=/tmp/x.json'],
|
||||
clear: () => order.push('clear'),
|
||||
runVitest: async () => {
|
||||
order.push('vitest');
|
||||
return ok;
|
||||
},
|
||||
runGate: async () => {
|
||||
order.push('gate');
|
||||
return ok;
|
||||
},
|
||||
log: () => undefined,
|
||||
});
|
||||
expect(rc).toBe(2);
|
||||
expect(order).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,346 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Positive control for the `preview / component-tests` tier.
|
||||
*
|
||||
* 🔴 WHY: THE BROWSER SUITE HAS A FAILURE MODE THAT COLLECTS ZERO TESTS AND IS
|
||||
* INDISTINGUISHABLE, TO EVERY CONSUMER, FROM A SUITE THAT RAN.
|
||||
*
|
||||
* A `vi.mock` factory that throws in browser mode is resolved over the browser<->node
|
||||
* channel inside a Playwright route handler that does not catch
|
||||
* (`@vitest/browser-playwright/dist/index.js`, `await module.resolve()` inside `page.route`).
|
||||
* The rejection escapes as an `Unhandled Rejection` in the ORCHESTRATOR and kills the run
|
||||
* before any reporter prints: no `Test Files` line, no `Tests` line, no per-file results,
|
||||
* exit 1. Measured on `main` at d353f785c3 — one file
|
||||
* (`src/tests/pages/apps/review/review-queue-nav.browser.test.tsx`) zeroed all 201 files.
|
||||
*
|
||||
* The tier that runs this — the shared `npm-report-only-suite` Tekton task in the
|
||||
* datapacket-talos repo — computes its verdict from the runner's EXIT CODE alone. So an
|
||||
* abort that executed nothing and a genuine list of failing assertions both render as
|
||||
* `component:fail` / "Component suite failed": the same words, the same colour, in the same
|
||||
* place. A tier that cannot say "this run tested nothing" is a tier people learn to click
|
||||
* through, which is strictly worse than no tier at all.
|
||||
*
|
||||
* So this asserts a LEDGER over the run's JSON report. It never converts red to green — the
|
||||
* caller keeps vitest's exit code whenever the ledger is satisfied. It only ever ADDS a
|
||||
* failure, and it says which KIND.
|
||||
*
|
||||
* Usage: node scripts/ci/assert-component-suite-ran.mjs <vitest-json-report> [--narrowed]
|
||||
*
|
||||
* `--narrowed` skips the floor (not the zero check) because the caller passed a file or
|
||||
* `-t` filter, which makes the collected count a property of the filter rather than of the
|
||||
* suite. The floor guards the CI invocation, which passes no filters.
|
||||
*/
|
||||
import { existsSync, readdirSync, readFileSync } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
/**
|
||||
* The floor, and the reason it is a floor rather than an exact count.
|
||||
*
|
||||
* Measured 2026-08-31 on a full green run of this project:
|
||||
* `Test Files 201 passed (201) / Tests 2254 passed (2254)`, against 201 `*.browser.test.tsx`
|
||||
* files on disk. Set at ~55% of that so ordinary churn — a file deleted, a describe block
|
||||
* folded away, a legitimately smaller run on a branch — does not trip it, while the failure
|
||||
* this exists for (zero, or a handful of files surviving a collection-time abort) does.
|
||||
*
|
||||
* 🔴 Do NOT lower this to make a run green. If the suite has genuinely shrunk by half,
|
||||
* re-derive the number from a run you have READ, and say so in the commit.
|
||||
*/
|
||||
export const MIN_TESTS = 1240;
|
||||
export const BASELINE = { files: 201, tests: 2254, measuredOn: '2026-08-31' };
|
||||
|
||||
/**
|
||||
* 🔴 THIS MESSAGE NAMES EVERY CAUSE IT CANNOT TELL APART, AND MUST KEEP DOING SO.
|
||||
*
|
||||
* "Collected nothing" is an ABSENCE, and an absence is the observable that the most causes
|
||||
* share — so it identifies none of them. A message that asserts one cause sends the next
|
||||
* reader hunting a bug that is not there, which is worse than saying "here are the three,
|
||||
* read the error above". All three below have been observed on this suite.
|
||||
*/
|
||||
export const ABORT_DIAGNOSIS = `
|
||||
🔴 THIS RUN PRODUCED NO ACCOUNTABLE RESULT. IT IS NOT A TEST FAILURE AND IT IS NOT A PASS —
|
||||
nothing here says what was verified on this commit. Read the error printed ABOVE this
|
||||
line; it is the only thing that separates the causes below.
|
||||
|
||||
🔴 Do not read green lines above as coverage. An abort can land PART-WAY: files can have
|
||||
run and passed and still leave no report, because the run died before it could say what
|
||||
ran. Whatever scrolled past is unaccounted for, not confirmed.
|
||||
|
||||
1. A \`vi.mock\` FACTORY THREW. Vitest resolves manual mocks over the browser<->node
|
||||
channel inside a Playwright route handler that does not catch, so the rejection escapes
|
||||
as an "Unhandled Rejection" in the ORCHESTRATOR and kills the whole run — no summary,
|
||||
no per-file results. The printed error is generic ("[vitest] There was an error when
|
||||
mocking a module ... make sure there are no top level variables inside"): the real cause
|
||||
is wrapped twice and its innermost \`cause\` is dropped in transit, so it names neither
|
||||
the file nor the error, and its advice about hoisting is usually WRONG.
|
||||
Two defects produce it:
|
||||
- a WHOLESALE factory that no longer names an export something in the file's module
|
||||
graph imports ("does not provide an export named X") — add the export to the factory
|
||||
(an \`importOriginal\` spread is the usual cure but is not always available: see
|
||||
src/components/AppBlocks/__tests__/featureFlagsMockCompleteness.test.ts);
|
||||
- a factory reading a module-scope binding still in its temporal dead zone — \`vi.mock\`
|
||||
is hoisted above the imports and the factory fires during import evaluation, BEFORE
|
||||
the file's own \`const\`s initialise. Move the binding into \`vi.hoisted(() => ...)\`.
|
||||
To attribute it to a FILE, bisect by file set — the run names no file on its own:
|
||||
pnpm exec vitest run --project component <a subset of the files>
|
||||
|
||||
2. THE BROWSER DIED, OR NEVER CAME UP. Two signatures, both host CONTENTION rather than a
|
||||
defect in the code under test — check the load average and free memory, and re-run
|
||||
serially (--max-workers=1) before believing anything else this run says:
|
||||
- "Failed to connect to the browser session ... within the timeout" — it never started;
|
||||
- "Browser connection was closed while running tests. Was the page closed
|
||||
unexpectedly?" — it started, ran, and was killed part-way. 🔴 THIS ONE ARRIVES
|
||||
WEARING THE MOCKING ERROR ABOVE: it is wrapped by the same \`createHelpfulError\`, so
|
||||
the headline blames a \`vi.mock\` factory and the real cause is on the "Caused by:"
|
||||
line. Read that line before chasing a mock. Note that ONE crash zeroes the WHOLE
|
||||
run, however many files had already gone green above it.
|
||||
On NixOS the first signature is also what a playwright/Chromium REVISION MISMATCH looks
|
||||
like — it collects files and executes none. See CLAUDE.md, "Browser/component tests on
|
||||
NixOS".
|
||||
|
||||
3. THE PROJECT SELECTED NOTHING. A \`--project\`/glob that matches no file exits without
|
||||
running, and an empty selection is not a pass.
|
||||
`;
|
||||
|
||||
/**
|
||||
* Count what a vitest JSON report says actually happened.
|
||||
*
|
||||
* 🔴 EXECUTED, NOT TOTAL. `numTotalTests` counts SKIPPED tests, so a suite that skipped
|
||||
* itself wholesale would satisfy a total-based floor having run nothing. `failed` counts as
|
||||
* executed on purpose: a red test did the work this script asserts happened, and the caller's
|
||||
* exit code already owns the pass/fail verdict — a guard that treated a red run as "did not
|
||||
* run" would fire on every genuine test failure.
|
||||
*/
|
||||
export function tally(report) {
|
||||
const EXECUTED = new Set(['passed', 'failed']);
|
||||
let executed = 0;
|
||||
let skipped = 0;
|
||||
for (const file of report?.testResults ?? []) {
|
||||
for (const assertion of file?.assertionResults ?? []) {
|
||||
if (EXECUTED.has(assertion.status)) executed += 1;
|
||||
else skipped += 1;
|
||||
}
|
||||
}
|
||||
return {
|
||||
executed,
|
||||
skipped,
|
||||
files: Array.isArray(report?.testResults) ? report.testResults.length : 0,
|
||||
failedSuites: report?.numFailedTestSuites ?? 0,
|
||||
failedTests: report?.numFailedTests ?? 0,
|
||||
// A file that fails to IMPORT is a failed SUITE with zero assertions. That pair is the
|
||||
// per-file version of the whole-run abort: nothing in the file is collected, so a failure
|
||||
// count and a per-test list both read as clean. Named, not merely counted — the count is
|
||||
// the thing people read past.
|
||||
suitesWithNoAssertions: (report?.testResults ?? [])
|
||||
.filter((f) => f?.status === 'failed' && (f?.assertionResults ?? []).length === 0)
|
||||
.map((f) => f?.name)
|
||||
.filter(Boolean),
|
||||
/** Every file the report accounts for, for the on-disk ledger in `verdict`. */
|
||||
names: (report?.testResults ?? []).map((f) => f?.name).filter(Boolean),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Every `*.browser.test.tsx` under `src/`, which is exactly the `component` project's `include`
|
||||
* in `vitest.config.mts`. Walked rather than globbed: `grep -r` and friends honour `.gitignore`
|
||||
* here, and a silently-narrower expectation is a ledger that cannot notice anything.
|
||||
*
|
||||
* Returns `null` when it cannot see the tree at all (wrong cwd, no `src/`), because a ZERO from
|
||||
* a walk that found nothing is indistinguishable from a suite with no files — and a ledger
|
||||
* built on an unproven zero would pass everything.
|
||||
*/
|
||||
export function browserTestFilesOnDisk(root) {
|
||||
const src = join(root, 'src');
|
||||
if (!existsSync(src)) return null;
|
||||
const out = [];
|
||||
const walk = (dir) => {
|
||||
for (const entry of readdirSync(dir, { withFileTypes: true })) {
|
||||
const p = join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
if (entry.name === 'node_modules' || entry.name === '.next') continue;
|
||||
walk(p);
|
||||
} else if (entry.name.endsWith('.browser.test.tsx')) {
|
||||
out.push(p);
|
||||
}
|
||||
}
|
||||
};
|
||||
walk(src);
|
||||
return out.length > 0 ? out : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The verdict, as data. `{ ok, code, lines[] }`; `code` is what the caller should exit with.
|
||||
*
|
||||
* `onDisk` is the list from `browserTestFilesOnDisk`, or `null` when it could not be derived.
|
||||
*/
|
||||
export function verdict(counts, { narrowed = false, onDisk = null } = {}) {
|
||||
const lines = [
|
||||
`test:component ledger: ${counts.executed} executed, ${counts.skipped} skipped, ` +
|
||||
`across ${counts.files} files; ${counts.failedSuites} failed suites, ` +
|
||||
`${counts.failedTests} failed tests (baseline ${BASELINE.tests} tests / ` +
|
||||
`${BASELINE.files} files measured ${BASELINE.measuredOn}` +
|
||||
(onDisk ? `; ${onDisk.length} on disk` : '; on-disk count UNAVAILABLE') +
|
||||
(narrowed ? '; floor SKIPPED — narrowed by a filter' : `; floor ${MIN_TESTS}`) +
|
||||
')',
|
||||
];
|
||||
|
||||
if (counts.suitesWithNoAssertions.length > 0) {
|
||||
lines.push(
|
||||
`\n⚠️ ${counts.suitesWithNoAssertions.length} FILE(S) FAILED WITHOUT RUNNING A SINGLE ` +
|
||||
'ASSERTION — these did not fail a test, they failed to IMPORT, so every test in them\n' +
|
||||
' was silently NOT RUN. A failure count and a per-test list both read as clean here.\n' +
|
||||
counts.suitesWithNoAssertions.map((n) => ` ${n}`).join('\n') +
|
||||
'\n The usual cause is a wholesale `vi.mock` factory that no longer names an export\n' +
|
||||
" something in the file's module graph imports — read that file's error above."
|
||||
);
|
||||
}
|
||||
|
||||
if (counts.executed === 0) {
|
||||
return { ok: false, code: 1, lines: [...lines, ABORT_DIAGNOSIS] };
|
||||
}
|
||||
|
||||
/**
|
||||
* 🔴 A LEDGER OVER FILES, NOT A SECOND FLOOR — and it is the stronger of the two checks.
|
||||
*
|
||||
* The test floor sits at ~55%, so up to ~45% of the suite can stop being COLLECTED while the
|
||||
* gate stays green. The historical incident this guard descends from is exactly that shape:
|
||||
* six files contributed 0 of 438 tests and nothing turned red
|
||||
* (src/components/AppBlocks/__tests__/featureFlagsMockCompleteness.test.ts). Comparing the
|
||||
* report's file list against what is on disk catches ONE file going missing, not 90, and
|
||||
* needs no constant to maintain — the expectation is re-derived every run.
|
||||
*
|
||||
* Skipped when narrowed (a filter legitimately selects fewer files) and when the walk could
|
||||
* not run. Files are matched by SUFFIX because the report carries absolute paths.
|
||||
*/
|
||||
if (!narrowed && onDisk) {
|
||||
const collected = new Set(counts.names.map((n) => n.replace(/\\/g, '/')));
|
||||
const missing = onDisk.filter((p) => {
|
||||
const rel = p.replace(/\\/g, '/');
|
||||
for (const c of collected) if (c === rel || c.endsWith(rel) || rel.endsWith(c)) return false;
|
||||
return true;
|
||||
});
|
||||
if (missing.length > 0) {
|
||||
return {
|
||||
ok: false,
|
||||
code: 1,
|
||||
lines: [
|
||||
...lines,
|
||||
`\n🔴 ${missing.length} \`*.browser.test.tsx\` FILE(S) ON DISK ARE ABSENT FROM THE ` +
|
||||
`RUN'S REPORT.\n` +
|
||||
' They were not collected at all — not run, not skipped, not failed. That reports\n' +
|
||||
' as ABSENCE, so no failure count and no per-test list can show it:\n' +
|
||||
missing.map((p) => ` ${p}`).join('\n') +
|
||||
'\n Either the project include stopped matching them, or the run did not finish\n' +
|
||||
' collecting. Do NOT silence this by narrowing the walk.',
|
||||
],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (!narrowed && counts.executed < MIN_TESTS) {
|
||||
return {
|
||||
ok: false,
|
||||
code: 1,
|
||||
lines: [
|
||||
...lines,
|
||||
`\n🔴 THE COMPONENT SUITE EXECUTED ONLY ${counts.executed} TESTS (floor ${MIN_TESTS}, ` +
|
||||
`baseline ${BASELINE.tests}).\n` +
|
||||
' Too few to be this suite. Either a large number of files stopped being COLLECTED —\n' +
|
||||
' which reports as ABSENCE, not as failure — or the suite genuinely shrank. Compare\n' +
|
||||
' the file count above against what is on disk:\n' +
|
||||
' find src -name "*.browser.test.tsx" | wc -l\n' +
|
||||
' Do NOT lower the floor to make this green.',
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
return { ok: true, code: 0, lines };
|
||||
}
|
||||
|
||||
function main(argv) {
|
||||
const args = argv.slice(2);
|
||||
const narrowed = args.includes('--narrowed');
|
||||
|
||||
// 🔴 `--repo-root <dir>` CONSUMES ITS VALUE, and this loop is why. Picking the report as
|
||||
// "the first argument that is not a flag" read the DIRECTORY as the report path whenever the
|
||||
// flag came first — measured: `--repo-root /tmp/tree report.json` failed with
|
||||
// "EISDIR: illegal operation on a directory" plus the entire abort diagnosis. The tests
|
||||
// happened to pass it last, so nothing caught it.
|
||||
let reportPath = null;
|
||||
let root = null;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
// 🔴 BOTH SPELLINGS. Matching only `--repo-root <dir>` left `--repo-root=<dir>` — the more
|
||||
// conventional GNU form — falling through BOTH branches, so `root` stayed null and the walk
|
||||
// silently graded against the real repo. That is byte-for-byte the failure the missing-value
|
||||
// guard below was added to close, still reachable through one extra character: a guard
|
||||
// spelled rather than structural.
|
||||
if (args[i] === '--repo-root' || args[i].startsWith('--repo-root=')) {
|
||||
const inline = args[i].startsWith('--repo-root=')
|
||||
? args[i].slice('--repo-root='.length)
|
||||
: null;
|
||||
const value = inline !== null ? inline : args[i + 1];
|
||||
// A missing value is a USAGE ERROR, not a silent fall-through to the real repo. With
|
||||
// `?? null` it fell back to this script's own root, so a fixture report was graded
|
||||
// against the 201 real files and failed with a diagnosis about the include breaking —
|
||||
// a confident wrong answer produced by a typo. A value that is itself a flag is the
|
||||
// same mistake wearing a plausible shape.
|
||||
if (value === undefined || value === '' || value.startsWith('--')) {
|
||||
console.error('--repo-root requires a directory');
|
||||
return 2;
|
||||
}
|
||||
root = value;
|
||||
if (inline === null) i += 1;
|
||||
} else if (!args[i].startsWith('--') && reportPath === null) {
|
||||
reportPath = args[i];
|
||||
}
|
||||
}
|
||||
if (!reportPath) {
|
||||
console.error(
|
||||
'usage: assert-component-suite-ran.mjs <vitest-json-report> [--narrowed] [--repo-root <dir>]'
|
||||
);
|
||||
return 2;
|
||||
}
|
||||
|
||||
// 🔴 A MISSING OR UNREADABLE REPORT IS A FINDING, NOT A PASS. It is exactly what the abort
|
||||
// this guard exists for produces when it dies early enough, and treating it as "nothing to
|
||||
// check" would make the guard silent in the one case it was written for.
|
||||
if (!existsSync(reportPath)) {
|
||||
console.error(
|
||||
`\n${reportPath} does not exist — the run wrote no JSON report.\n` +
|
||||
'The reporter writes at the END of a run, so this means vitest never reached that\n' +
|
||||
'point, or --outputFile never reached vitest. Measured: an abort 68 files into a\n' +
|
||||
'201-file run still wrote nothing, so a partial run and a run that never started are\n' +
|
||||
'INDISTINGUISHABLE from here — which is why neither counts as one that ran.' +
|
||||
ABORT_DIAGNOSIS
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
let report;
|
||||
try {
|
||||
report = JSON.parse(readFileSync(reportPath, 'utf8'));
|
||||
} catch (err) {
|
||||
console.error(
|
||||
`\n${reportPath} is not valid JSON (${err.message}) — cannot verify this run collected ` +
|
||||
'anything, so it does not count as one that did.' +
|
||||
ABORT_DIAGNOSIS
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
// `--repo-root <dir>` exists so the unit tests can point the on-disk walk at a fixture tree.
|
||||
// Default is this script's own repo, which is the only thing a real run should ever grade.
|
||||
const repoRoot = root ?? resolve(fileURLToPath(new URL('../..', import.meta.url)));
|
||||
|
||||
const result = verdict(tally(report), { narrowed, onDisk: browserTestFilesOnDisk(repoRoot) });
|
||||
for (const line of result.lines) (result.ok ? console.log : console.error)(line);
|
||||
return result.code;
|
||||
}
|
||||
|
||||
// Same main-guard shape as scripts/test-unit-run.mjs, so importing this from a test does not
|
||||
// execute it.
|
||||
if (
|
||||
import.meta.url === `file://${process.argv[1]}` ||
|
||||
process.argv[1] === fileURLToPath(import.meta.url)
|
||||
) {
|
||||
process.exit(main(process.argv));
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* `pnpm run test:component` — runs the browser-mode `component` project AND asserts that it
|
||||
* collected something.
|
||||
*
|
||||
* The assertion itself lives in `scripts/ci/assert-component-suite-ran.mjs`, which owns the
|
||||
* reason it exists and is unit-tested against fixtures. This file is only the plumbing: run
|
||||
* vitest with a JSON report beside the normal output, then hand that report to the gate.
|
||||
*
|
||||
* 🔴 IT NEVER TURNS RED INTO GREEN. Vitest's own exit code is passed through whenever the
|
||||
* ledger is satisfied; the gate can only ever ADD a failure. Two paths deliberately bypass
|
||||
* it — a signal-killed runner (see below) and a narrowed run (the gate's `--narrowed`).
|
||||
*/
|
||||
import { spawn } from 'node:child_process';
|
||||
import { rmSync } from 'node:fs';
|
||||
import { constants as osConstants } from 'node:os';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
// Under `node_modules/` so it is gitignored with the rest of it, and so it sits beside the
|
||||
// install the run used rather than in a shared tmpdir two agents can collide in.
|
||||
const REPORT = resolve(repoRoot, 'node_modules/.civitai-component-report.json');
|
||||
const GATE = resolve(repoRoot, 'scripts/ci/assert-component-suite-ran.mjs');
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
/**
|
||||
* A vitest flag's CANONICAL PATH: leading dashes off, `=value` off, the FIRST dot-segment kebab
|
||||
* camelCased and later segments left verbatim — which is exactly cac's own
|
||||
* `camelcaseOptionName`, `name.split(".").map((v,i) => i===0 ? camelcase(v) : v).join(".")`.
|
||||
* `--output-file` and `--outputFile` both give `outputFile`; `--outputFile.json=x` gives
|
||||
* `outputFile.json`.
|
||||
*
|
||||
* Kebab and camel are the same flag to vitest, which is why `--max-workers` and `--maxWorkers`
|
||||
* both work — so anything comparing a flag by name has to normalise rather than enumerate
|
||||
* spellings. Used ONLY by `conflictingOutputFile`, which needs to recognise one specific flag.
|
||||
* Nothing here tries to decide whether a flag takes a value; see `narrowingReason` for why.
|
||||
*/
|
||||
export function canonicalFlag(arg) {
|
||||
if (!arg.startsWith('-')) return null;
|
||||
const stripped = arg.replace(/^--?/, '').split('=')[0];
|
||||
const [head, ...rest] = stripped.split('.');
|
||||
return [head.replace(/-+([a-zA-Z0-9])/g, (_, c) => c.toUpperCase()), ...rest].join('.');
|
||||
}
|
||||
|
||||
/**
|
||||
* WHY this run counts as narrowed, or `null` for a full run.
|
||||
*
|
||||
* 🔴 ANY ARGUMENT AT ALL MEANS NARROWED. THIS IS DELIBERATELY NOT A PARSER, AND FIVE ROUNDS OF
|
||||
* AUDIT ARE THE REASON.
|
||||
*
|
||||
* The question "does this flag consume the next token?" was attacked twice and lost twice.
|
||||
* Measured against vitest 4.1.11's REAL option table — enumerated by calling `createCLI()` and
|
||||
* reading each cac option's `isBoolean`: 170 long options, 74 boolean, 96 value-taking:
|
||||
*
|
||||
* - a hand-maintained list of value-taking flags was wrong on 73 of them (every value-taking
|
||||
* option), reading a flag's VALUE as a filename — the QUIET direction, both checks silently
|
||||
* off;
|
||||
* - replacing it with a shape heuristic ("is the token path-like?") was wrong on 74 (every
|
||||
* BOOLEAN option), reading a real filter as a value — the LOUD direction, so
|
||||
* `pnpm test:component --coverage AppNameCrumb` ran one test and then failed it against the
|
||||
* 1240 floor and the on-disk ledger with "the include broke or the run died".
|
||||
*
|
||||
* 73 versus 74. The heuristic did not beat the list; it moved the wrongness off one set of
|
||||
* options and onto the other. That is a mis-posed question, not one needing a better answer, so
|
||||
* it is no longer asked.
|
||||
*
|
||||
* 🔴 WHAT THIS COSTS, STATED PLAINLY: an arg-ful run does not get the floor or the on-disk
|
||||
* ledger, even when the argument was only `--max-workers 4` and the run really was full. That
|
||||
* is affordable for exactly one reason, and it is a measured one rather than an assumption —
|
||||
* `pr-preview-pipeline.yaml` invokes `pnpm run test:component` with NO arguments, so CI is the
|
||||
* `argv.length === 0` path and always gets both checks. What is given up is enforcing a floor
|
||||
* on an ad-hoc local run, which nobody needs.
|
||||
*
|
||||
* 🔴 AND THE ZERO-COLLECTED CHECK IS NOT SKIPPED — not here, not ever. `--narrowed` disables the
|
||||
* floor and the ledger only. The failure this whole change exists for (a run that aborts having
|
||||
* collected nothing) is caught on every invocation, including a single-file one, which is the
|
||||
* cheapest reproduction of it and precisely when someone is debugging it.
|
||||
*/
|
||||
export function narrowingReason(argv) {
|
||||
if (argv.length === 0) return null;
|
||||
return (
|
||||
`arguments were passed (${argv.map((a) => `\`${a}\``).join(' ')}), so what this run SHOULD ` +
|
||||
'have collected is not knowable from here'
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the caller narrowed the run — the boolean the gate needs. `narrowingReason` owns the
|
||||
* rule and the explanation.
|
||||
*/
|
||||
export function isNarrowed(argv) {
|
||||
return narrowingReason(argv) !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* A caller-supplied `--outputFile` silently redirects the JSON report the gate reads.
|
||||
*
|
||||
* 🔴 MEASURED, AND IT IS THE WORST FAILURE THIS SCRIPT CAN PRODUCE: an 18/18 green single-file
|
||||
* run reported "THE COMPONENT SUITE COLLECTED NOTHING … this tier verified NOTHING on this
|
||||
* commit" and exited 1, because `--outputFile=<path>` clobbers the `--outputFile.json=` form
|
||||
* appended below and the wrapper's report was never written. The diagnosis then names three
|
||||
* causes, none of which is the real one. `.github/workflows/lint.yml` runs the SIBLING unit
|
||||
* tier with exactly that flag, so anyone extending this tier by copying that line hits it.
|
||||
*
|
||||
* Refused rather than honoured: the bare `--outputFile=` form sets the path for EVERY reporter,
|
||||
* so honouring it would point the default reporter at the same file. A loud config error with
|
||||
* the fix in it beats either silent wrong answer.
|
||||
*
|
||||
* 🔴 SCOPED TO THE FORMS THAT ACTUALLY COLLIDE — the bare one and `.json`. An earlier revision
|
||||
* refused `--outputFile.junit=` and `--outputFile.html=` too, neither of which touches the
|
||||
* `.json` key: object-form output paths merge per reporter, so those are legitimate and
|
||||
* refusing them was over-strict. It also missed `--output-file` entirely, which vitest accepts
|
||||
* (cac camelCases option keys), leaving the exact defect this exists for reachable through the
|
||||
* kebab spelling.
|
||||
*/
|
||||
export function conflictingOutputFile(argv) {
|
||||
return argv.find((a) => ['outputFile', 'outputFile.json'].includes(canonicalFlag(a))) ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The shell's own convention: a process killed by signal N exits 128+N.
|
||||
*
|
||||
* 🔴 THIS USED TO BE A HARDCODED 143 FOR EVERY SIGNAL, AND THAT RE-CREATED THE EXACT
|
||||
* MISLABELLING THIS WHOLE CHANGE EXISTS TO REMOVE. `report-only-suite-task.yaml` branches on
|
||||
* 137 to report `oom-killed` — "this is an OUT-OF-MEMORY kill, not a timeout; raise the task's
|
||||
* memory limit" — and on 124 for a timeout. Before this wrapper existed, an OOM-killer SIGKILL
|
||||
* on vitest reached that task as 137 (pnpm re-raises). With a constant 143 the wrapper would
|
||||
* hand it 143 instead, which matches no branch and falls through to `RC=1` — verdict `fail`,
|
||||
* rendered as "Component suite failed". A memory problem would have been reported as a test
|
||||
* failure, on the same tier, in the same words.
|
||||
*
|
||||
* `os.constants.signals` is the mapping node itself uses, so this cannot drift from the names
|
||||
* node hands back.
|
||||
*/
|
||||
export function exitCodeForSignal(signal) {
|
||||
const n = osConstants.signals[signal];
|
||||
return typeof n === 'number' ? 128 + n : 143;
|
||||
}
|
||||
|
||||
/**
|
||||
* 🔴 `shell` IS PER-CALL, NOT SHARED, AND THAT DISTINCTION IS THE WHOLE POINT.
|
||||
*
|
||||
* The vitest spawn needs it on Windows: since the node 18.20.2/20.12.2 CVE fix, `spawn()` of a
|
||||
* `.cmd`/`.bat` without a shell fails outright. `scripts/test-unit-run.mjs` does exactly this —
|
||||
* and, deliberately, only for its vitest spawn.
|
||||
*
|
||||
* The GATE spawn must NOT have it. `shell: true` makes node concatenate argv UNESCAPED (it
|
||||
* emits DEP0190 saying so), and the gate is spawned as `process.execPath`, which on Windows
|
||||
* defaults to `C:\\Program Files\\nodejs\\node.exe`. Measured on this box with a spaced path:
|
||||
* `{shell:false}` → status 0, `{shell:true}` → status 1, "Cannot find module". Putting `shell`
|
||||
* on the shared helper therefore broke every `pnpm test:component` on Windows at the gate step,
|
||||
* with a cmd.exe parse error instead of any of this wrapper's messages — on the exact platform
|
||||
* the option was added to support.
|
||||
*/
|
||||
function run(bin, argv, { shell = false } = {}) {
|
||||
return new Promise((done) => {
|
||||
const child = spawn(bin, argv, { cwd: repoRoot, stdio: 'inherit', shell });
|
||||
// The CI task wraps this in `timeout(1)`, which signals only its direct child. Forward
|
||||
// so a budget overrun stops the runner instead of orphaning it.
|
||||
const forward = (sig) => () => {
|
||||
if (!child.killed) child.kill(sig);
|
||||
};
|
||||
process.on('SIGINT', forward('SIGINT'));
|
||||
process.on('SIGTERM', forward('SIGTERM'));
|
||||
child.on('error', (err) => {
|
||||
console.error(`test:component: failed to spawn ${bin}: ${err.message}`);
|
||||
done({ rc: 127, signal: null, spawnFailed: true });
|
||||
});
|
||||
child.on('exit', (code, signal) =>
|
||||
done({ rc: signal ? exitCodeForSignal(signal) : code ?? 1, signal, spawnFailed: false })
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
function vitestBinPath() {
|
||||
return resolve(
|
||||
repoRoot,
|
||||
'node_modules/.bin',
|
||||
process.platform === 'win32' ? 'vitest.cmd' : 'vitest'
|
||||
);
|
||||
}
|
||||
|
||||
function clearReport() {
|
||||
try {
|
||||
rmSync(REPORT, { force: true });
|
||||
} catch {
|
||||
/* not worth failing a run over */
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 🔴 THE COLLABORATORS ARE INJECTED SO THE ORDER OF EFFECTS CAN BE ASSERTED, NOT INSPECTED.
|
||||
*
|
||||
* Two things here are pure sequencing and cannot be seen from any output: the report is cleared
|
||||
* BEFORE the runner starts, and a signal death or a failed spawn returns without consulting the
|
||||
* gate. Both were shipped with no coverage; the two `clearReport()` calls are now identical
|
||||
* statements, so nothing but their position carries the meaning, and a refactor that moves the
|
||||
* first one below the run reopens "a stale report satisfies the next run's ledger" — the gate
|
||||
* silently inert in exactly its own use case — with every test still green.
|
||||
*
|
||||
* Defaults are the real implementations, so the shipped path is the tested path minus the fakes.
|
||||
*/
|
||||
export async function main({
|
||||
argv = args,
|
||||
runVitest = (a) => run(vitestBinPath(), a, { shell: process.platform === 'win32' }),
|
||||
runGate = (a) => run(process.execPath, a),
|
||||
clear = clearReport,
|
||||
log = console.error,
|
||||
} = {}) {
|
||||
const conflict = conflictingOutputFile(argv);
|
||||
if (conflict) {
|
||||
log(
|
||||
`\ntest:component: refusing to run — \`${conflict}\` would redirect the JSON report this\n` +
|
||||
`command reads to decide whether the suite collected anything (it writes\n` +
|
||||
`--outputFile.json=${REPORT}).\n` +
|
||||
'Left alone, that produces the worst output this script has: a fully GREEN run reported\n' +
|
||||
'as an abort that verified nothing, naming causes none of which is real.\n' +
|
||||
'Run vitest directly if you need your own report:\n' +
|
||||
' pnpm exec vitest run --project component --reporter=json --outputFile=<path>'
|
||||
);
|
||||
return 2;
|
||||
}
|
||||
|
||||
// 🔴 Cleared BEFORE the run, not only after. A previous run's report is a fully-formed
|
||||
// healthy ledger; if this run then aborts before writing one, the gate reads the STALE file,
|
||||
// prints a green ledger and passes. Cleanup after the run cannot close that, because the
|
||||
// paths that skip it (a signal death, a throw) are the same ones that leave a stale report.
|
||||
clear();
|
||||
|
||||
// `--reporter=default` is restated because naming a second reporter REPLACES the default
|
||||
// set rather than adding to it — without it the human-readable output disappears and all
|
||||
// that is left is a JSON file nobody reads.
|
||||
const { rc, signal, spawnFailed } = await runVitest([
|
||||
'run',
|
||||
'--project',
|
||||
'component',
|
||||
'--reporter=default',
|
||||
'--reporter=json',
|
||||
`--outputFile.json=${REPORT}`,
|
||||
...argv,
|
||||
]);
|
||||
|
||||
/**
|
||||
* 🔴 A TRUNCATED RUN IS NOT AN ABORTED ONE, AND MUST NOT BE RELABELLED AS ONE.
|
||||
*
|
||||
* The CI task wraps this in `timeout(1)` and reads the exit code to tell "timeout" from
|
||||
* "fail" — it reports a budget overrun as an UNKNOWN verdict, deliberately, because no
|
||||
* suite went red. A killed run also writes no JSON report, which looks exactly like the
|
||||
* abort the gate exists to catch if you only look at the report. So a signal death
|
||||
* short-circuits: pass the runner's own status through and make no claim about what was
|
||||
* collected.
|
||||
*
|
||||
* (`timeout` signals only its direct child, which is the shell, so in the current CI
|
||||
* wiring this fires only if something signals this process directly. It is here because
|
||||
* the alternative — silently converting a kill into a confident "the suite collected
|
||||
* nothing" — is a wrong answer, not a missing one.)
|
||||
*/
|
||||
if (signal) {
|
||||
log(
|
||||
`\ntest:component: the runner was killed by ${signal} — passing that through as ${rc}. ` +
|
||||
'No claim is made about what it collected: a killed run writes no report, and that is ' +
|
||||
'not the same as a run that collected nothing.'
|
||||
);
|
||||
return rc;
|
||||
}
|
||||
|
||||
// 🔴 Same reasoning as the signal branch, for the case that produces no report for a reason
|
||||
// the gate cannot describe: the runner never STARTED. `spawnFailed` is what `run()` reports
|
||||
// when `spawn` itself errors. Handing that to the gate makes it print forty lines about mock
|
||||
// factories, dead browsers and empty selections — none of which happened.
|
||||
//
|
||||
// Keyed on the FLAG, not on `rc === 127`: 127 is a real exit code a runner can produce on its
|
||||
// own, and relabelling that as "the binary could not be started" is a second wrong answer.
|
||||
if (spawnFailed) {
|
||||
log(
|
||||
'\ntest:component: the vitest binary could not be started, so nothing ran and there is ' +
|
||||
'nothing to account for. Check that `pnpm install` has completed in this checkout.'
|
||||
);
|
||||
return rc;
|
||||
}
|
||||
|
||||
// 🔴 SAY WHY, whenever the checks are being turned off. `--narrowed` disables the file
|
||||
// ledger and the floor — the two checks this whole change exists to add — so a wrong
|
||||
// narrowing decision is the QUIET failure, the one direction the comments here repeatedly
|
||||
// name as worse than a loud one. Printing the token that caused it is what stops it being
|
||||
// quiet: `--retry.count 2` reading `2` as a file filter is obviously wrong on sight, and
|
||||
// invisible otherwise.
|
||||
const reason = narrowingReason(argv);
|
||||
if (reason) {
|
||||
log(
|
||||
`\ntest:component: NARROWED — ${reason}, so the file ledger and the floor are skipped. ` +
|
||||
'The zero-collected check still applies. Run with NO arguments to get all three — that ' +
|
||||
'is what CI does.'
|
||||
);
|
||||
}
|
||||
|
||||
const gate = await runGate([GATE, REPORT, ...(reason ? ['--narrowed'] : [])]);
|
||||
|
||||
// Best-effort, and AFTER the gate has read it. The load-bearing clear is the one BEFORE the
|
||||
// run; this one only keeps the tree tidy.
|
||||
clear();
|
||||
|
||||
// The gate can only ADD a failure — a green gate hands the runner's own verdict back.
|
||||
return gate.rc !== 0 ? gate.rc : rc;
|
||||
}
|
||||
|
||||
// Main-guard, so importing this from a test does not launch a browser suite.
|
||||
if (
|
||||
import.meta.url === `file://${process.argv[1]}` ||
|
||||
process.argv[1] === fileURLToPath(import.meta.url)
|
||||
) {
|
||||
process.exit(await main());
|
||||
}
|
||||
@@ -27,8 +27,32 @@ vi.mock('~/server/utils/server-side-helpers', () => ({
|
||||
createServerSideProps: () => async () => ({ props: {} }),
|
||||
}));
|
||||
|
||||
// 🔴 ALL THREE HOOKS, because this factory REPLACES the module. The queue page's row now
|
||||
// renders the review entry point, which reads flags through `useOptionalFeatureFlags`
|
||||
// (the non-throwing variant, correct outside a provider). A factory naming only
|
||||
// `useFeatureFlags` left that named import nothing to bind to:
|
||||
// SyntaxError: The requested module '/src/providers/FeatureFlagsProvider.tsx'
|
||||
// does not provide an export named 'useOptionalFeatureFlags'
|
||||
// and in BROWSER mode that does not fail this file — it takes down the whole run. The
|
||||
// factory is resolved over the browser<->node channel inside a Playwright route handler
|
||||
// that does not catch, so the rejection escapes as an Unhandled Rejection in the
|
||||
// orchestrator: no summary, no per-file results, zero tests collected, exit 1. This one
|
||||
// file zeroed the entire `preview / component-tests` tier.
|
||||
//
|
||||
// 🔴 So the rule is EVERY RUNTIME EXPORT the module has, not "the ones we know about".
|
||||
// `useFeatureFlagsReady` is the third hook (`src/providers/FeatureFlagsProvider.tsx:36`), with
|
||||
// four live consumers — useChatEnabled, useFeatureNotice, NavTidyNotice,
|
||||
// YellowBuzzMigrationNotice — and `FeatureFlagsProvider` itself is the fourth export
|
||||
// (`:37`, imported today only by `src/pages/_app.tsx`). Neither is in this page's graph TODAY,
|
||||
// which is the only reason naming fewer would still load — and "not in this graph today" is
|
||||
// precisely the reasoning that put this file in the diff. So name all four.
|
||||
// The flag hooks return the SAME flags: the gate must be decided by this fixture, not by
|
||||
// which of them a component happens to call.
|
||||
vi.mock('~/providers/FeatureFlagsProvider', () => ({
|
||||
useFeatureFlags: () => state.flags,
|
||||
useOptionalFeatureFlags: () => state.flags,
|
||||
useFeatureFlagsReady: () => true,
|
||||
FeatureFlagsProvider: ({ children }: { children: unknown }) => children,
|
||||
}));
|
||||
|
||||
// Stub the modal component (assert whether a selection opened it) but keep the
|
||||
@@ -73,7 +97,12 @@ const PENDING = {
|
||||
bundleSizeBytes: '2048',
|
||||
bundleSha256: 'abc',
|
||||
manifest: {},
|
||||
fileSummary: { files: [{ path: 'index.js', sha256: 'x', sizeBytes: 10 }], added: [], removed: [], changed: [] },
|
||||
fileSummary: {
|
||||
files: [{ path: 'index.js', sha256: 'x', sizeBytes: 10 }],
|
||||
added: [],
|
||||
removed: [],
|
||||
changed: [],
|
||||
},
|
||||
manifestDiffSummary: { kind: 'first-version', fields: [] },
|
||||
reviewRepoUrl: 'https://forgejo.example/repo',
|
||||
pushCommitUrl: null,
|
||||
@@ -91,8 +120,16 @@ const emptyQuery = () => ({
|
||||
vi.mock('~/utils/trpc', () => ({
|
||||
trpc: {
|
||||
useUtils: () => ({
|
||||
blocks: { listPendingRequests: inert, listApprovedRequests: inert, listRejectedRequests: inert },
|
||||
appListings: { listPendingRequests: inert, listApprovedRequests: inert, listRejectedRequests: inert },
|
||||
blocks: {
|
||||
listPendingRequests: inert,
|
||||
listApprovedRequests: inert,
|
||||
listRejectedRequests: inert,
|
||||
},
|
||||
appListings: {
|
||||
listPendingRequests: inert,
|
||||
listApprovedRequests: inert,
|
||||
listRejectedRequests: inert,
|
||||
},
|
||||
}),
|
||||
blocks: {
|
||||
listPendingRequests: {
|
||||
|
||||
Reference in New Issue
Block a user