mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
perf(dev-server): let the test queue cap each run's vitest pool (#4965)
* perf(dev-server): let the test queue cap each run's vitest pool The queue serialises full-suite runs at concurrency 1, which makes a run wait behind every other agent's. Measured over 23.4h of the daemon's own history (50 runs, 12 worktrees): median run 549s, median wait 186s, mean wait 405s, worst 2247s. Raising concurrency is the only lever that helps a change whose closure reaches the hot services, but it cannot be raised alone: vitest sizes its pool at `cpus - 1`, so two uncapped runs ask for 62 workers on a 32-core box. VITEST_MAX_WORKERS cannot carry the cap here — the daemon spawns the child with the daemon's own environment, so the caller's copy never arrives and the daemon's is fixed at start. The CLI flag is the only channel that reaches a queued run, and it is forwarded through `pnpm run` into vitest. Verified by pool id rather than by argv alone: 8 files at --max-workers=2 ran on workers [1 2]; the same 8 uncapped ran on [1 2 3 4 5 6 7 8]. Adds a runtime setter beside it so the width can be tuned without a second daemon restart, and each key of `test config` is applied only when sent — a concurrency change must not silently drop the cap. Also replaces the "~75s" figure in the full-suite hook, which was off by 7x against the measured median and was what every agent budgeted against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(dev-server): queue full typechecks in their own lane A full `pnpm run typecheck` is one core and up to an 8 GB heap, and several agents starting one at once pegs the box the same way concurrent suites did. With CIVITAI_TEST_QUEUE set it now goes through the dev-server queue. The queue gains run kinds with separate limits rather than one pool, because the loads differ: a suite saturates every core, tsc is effectively single-threaded. One shared limit would either hold a typecheck behind every queued suite or let two suites run at once. Each lane takes only its own head of the queue, and a run's position is reported within its lane. The scalar concurrency every existing caller passes still sets the unit lane only; reading it as "every lane" would raise the typecheck limit on any machine that had only ever tuned the suite. A typecheck stays direct in CI, with any argument (the scripts gate's `-p tsconfig.scripts.json`), with the tsc test seam in use (otherwise the typecheck tests would queue behind real runs and assert on the daemon's REAL tsc), and with a heap override (a queued run gets the daemon's environment, so the override would be silently dropped). typecheck.mjs reuses test-unit-run.mjs's queue client rather than a copy. Also fixes the worker cap missing a caller's camelCase `--maxWorkers`, which vitest treats as the same flag — the queue would have appended a second, conflicting width after it. Nine revert controls, each red on its own named test, restores verified by hash — including the one nothing else catches: a typecheck posted without its kind is accepted as a unit run and spawns a full suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(hooks): send full-program tsc runs to the queued typecheck script A direct `npx tsc --noEmit` skips the typecheck lane, so agents doing it at once are N single-core 8 GB heaps pegging the box. It is also the wrong check: tsc at node's default heap can abort part-way with zero diagnostics and a log that reads clean. scripts/typecheck.mjs raises the heap and names that crash. The hook now denies a full-program tsc (no -p, or -p at the root tsconfig) and points at `pnpm run typecheck`. Narrow runs pass untouched: a sub-project (`-p tsconfig.scripts.json`, which the scripts gate itself recommends), named files, --build, and informational flags. TYPECHECK_DIRECT=1 opts out for diagnosing tsc itself. Selftest: 68 rows green. Controls: disabling the guard fails all 10 block rows; matching `tsc\b` instead of `tsc(?=\s|$)` fails only "tsc-alias is not tsc". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -162,8 +162,13 @@ export function unboundedDevRequest(command) {
|
||||
.filter((seg) => !isBounded(seg));
|
||||
}
|
||||
|
||||
// The full unit suite (~21,500 tests / ~75s, serialised through the dev-server queue) belongs at
|
||||
// the END of a task, once — not between edits. Denied rather than asked, so the redirect reaches
|
||||
// The full unit suite belongs at the END of a task, once — not between edits.
|
||||
//
|
||||
// The numbers below are measured, and the "~75s" they replace was not: a day of the daemon's own
|
||||
// queue history (50 full runs, 12 worktrees, 2026-09-17/18) puts the RUN at a median of 549s and
|
||||
// the QUEUE WAIT in front of it at a median of 186s, mean 405s, worst 2247s. An agent budgeting a
|
||||
// mid-iteration suite run against 75s is off by 7x on the run alone, which is precisely the
|
||||
// miscalculation that fills the queue this hook exists to protect. Denied rather than asked, so the redirect reaches
|
||||
// the agent at the moment of the mistake instead of interrupting the user. FULL_SUITE=1 is the
|
||||
// deliberate opt-in for the single pre-commit run or an explicit user request.
|
||||
const VITEST_INVOCATION =
|
||||
@@ -184,13 +189,55 @@ export function fullUnitSuiteRun(command) {
|
||||
}
|
||||
|
||||
const FULL_SUITE_REASON =
|
||||
'Full unit suite blocked mid-iteration: it is ~21,500 tests / ~75s and serialised through the ' +
|
||||
"dev-server queue, blocking everyone else's runs. Run only the test files covering your change: " +
|
||||
'Full unit suite blocked mid-iteration: it is ~25,000 tests, ~9 minutes to run, and serialised ' +
|
||||
"through the dev-server queue behind a typical 3-minute wait — blocking everyone else's runs. " +
|
||||
'Run only the test files covering your change: ' +
|
||||
"`pnpm exec vitest run --project 'unit*' <files>` — find them with " +
|
||||
'`grep -rln \'<symbol>\' src --include=*.test.ts`. The full suite runs ONCE, right before ' +
|
||||
'committing; for that single run (or when the user explicitly asked for a full run), prefix the ' +
|
||||
'command with FULL_SUITE=1.';
|
||||
|
||||
// A full-program `tsc` run directly, instead of through `pnpm run typecheck`. Two reasons, and the
|
||||
// second would hold even if the first went away:
|
||||
// 1. It skips the typecheck lane of the dev-server queue, so several agents doing it at once is
|
||||
// N single-core 8 GB heaps pegging the box — the condition the lane exists to prevent.
|
||||
// 2. It is the WRONG CHECK. tsc at node's default heap can abort part-way with ZERO diagnostics
|
||||
// and a log that reads as clean; scripts/typecheck.mjs raises the heap and names that crash.
|
||||
// Measured: a file with 8 real errors reported "No errors found" under `npx tsc -p
|
||||
// tsconfig.json` and 8 errors under `pnpm run typecheck`.
|
||||
// Narrow runs are left alone: a sub-project (`-p tsconfig.scripts.json`, which the scripts gate
|
||||
// itself recommends), named files, `--build`, and informational flags. TYPECHECK_DIRECT=1 is the
|
||||
// deliberate opt-out for diagnosing tsc itself.
|
||||
const TSC_INVOCATION = new RegExp(
|
||||
String.raw`^\s*(?:\w+=\S*\s+)*(?:` +
|
||||
String.raw`(?:(?:pnpm|yarn|bun)\s+(?:exec|dlx)\s+|npx\s+|\.?[\\/]?node_modules[\\/]\.bin[\\/])?tsc(?=\s|$)` +
|
||||
String.raw`|node\s+(?:--\S+\s+)*\S*typescript[\\/]lib[\\/]tsc\.js\b)`
|
||||
);
|
||||
const TSC_ROOT_PROJECT = /^(?:\.[\\/]?|(?:\.[\\/])?tsconfig\.json)$/;
|
||||
const TSC_NOT_A_CHECK = /(?:^|\s)(?:-v|--version|-h|--help|--init|--showConfig|--all|-b|--build)\b/;
|
||||
|
||||
export function directRootTypecheck(command) {
|
||||
if (/TYPECHECK_DIRECT\s*=\s*1|\$env:TYPECHECK_DIRECT/.test(command)) return false;
|
||||
return command.split(/[;&|\n]+/).some((seg) => {
|
||||
if (!TSC_INVOCATION.test(seg)) return false;
|
||||
if (TSC_NOT_A_CHECK.test(seg)) return false;
|
||||
const tokens = seg.trim().split(/\s+/).map((t) => t.replace(/^['"]|['"]$/g, ''));
|
||||
// Named source files make tsc ignore tsconfig entirely: a narrow check of those files only.
|
||||
if (tokens.some((t) => !t.startsWith('-') && /\.(?:[cm]?tsx?|d\.ts)$/.test(t))) return false;
|
||||
const at = tokens.findIndex((t) => /^(?:-p|--project)(?:=|$)/.test(t));
|
||||
if (at === -1) return true;
|
||||
const inline = tokens[at].includes('=') ? tokens[at].split('=')[1] : tokens[at + 1];
|
||||
return TSC_ROOT_PROJECT.test(inline ?? '');
|
||||
});
|
||||
}
|
||||
|
||||
const DIRECT_TSC_REASON =
|
||||
'Direct full-program tsc blocked: use `pnpm run typecheck`. That script queues the run in the ' +
|
||||
"dev-server typecheck lane (several agents' 8 GB tsc heaps at once is what pegs the box), and it " +
|
||||
'is also the only form that cannot report a crashed run as clean — plain tsc at the default ' +
|
||||
'heap can abort with zero diagnostics. A sub-project (`-p tsconfig.scripts.json`) or named ' +
|
||||
'files still run directly. To diagnose tsc itself, prefix the command with TYPECHECK_DIRECT=1.';
|
||||
|
||||
// Patterns that would kill Claude Code or critical processes - BLOCK OUTRIGHT
|
||||
const DANGEROUS_PATTERNS = [
|
||||
{ pattern: /taskkill\s+\/\/F\s+\/\/IM\s+node\.exe/i, reason: 'This would kill all Node.js processes including Claude Code itself' },
|
||||
@@ -287,6 +334,17 @@ stdin.on('end', () => {
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (directRootTypecheck(command)) {
|
||||
console.log(JSON.stringify({
|
||||
hookSpecificOutput: {
|
||||
hookEventName: 'PreToolUse',
|
||||
permissionDecision: 'deny',
|
||||
permissionDecisionReason: DIRECT_TSC_REASON,
|
||||
}
|
||||
}));
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
for (const { pattern, check, reason } of GUARDED_PATTERNS) {
|
||||
if (check ? check(command) : pattern.test(command)) {
|
||||
console.log(JSON.stringify({
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
* only mentions a port must all run untouched.
|
||||
*/
|
||||
|
||||
import { fullUnitSuiteRun, unboundedDevRequest } from './check-writable.mjs';
|
||||
import { directRootTypecheck, fullUnitSuiteRun, unboundedDevRequest } from './check-writable.mjs';
|
||||
|
||||
let failures = 0;
|
||||
const check = (name, cmd, expectBlocked) => {
|
||||
@@ -57,6 +57,35 @@ const checkSuite = (name, cmd, expectBlocked) => {
|
||||
console.log(`${pass ? 'PASS' : 'FAIL'} ${name} blocked=${blocked} want=${expectBlocked}`);
|
||||
};
|
||||
|
||||
const checkTsc = (name, cmd, expectBlocked) => {
|
||||
const blocked = directRootTypecheck(cmd);
|
||||
const pass = blocked === expectBlocked;
|
||||
if (!pass) failures++;
|
||||
console.log(`${pass ? 'PASS' : 'FAIL'} ${name} blocked=${blocked} want=${expectBlocked}`);
|
||||
};
|
||||
|
||||
checkTsc('npx tsc --noEmit', 'npx tsc --noEmit', true);
|
||||
checkTsc('pnpm exec tsc', 'pnpm exec tsc --noEmit', true);
|
||||
checkTsc('root project named', 'npx tsc --noEmit -p tsconfig.json', true);
|
||||
checkTsc('root project as dot', 'pnpm exec tsc --noEmit --project .', true);
|
||||
checkTsc('root project, = form', 'npx tsc --noEmit --project=./tsconfig.json', true);
|
||||
checkTsc('.bin path', 'node_modules/.bin/tsc --noEmit', true);
|
||||
checkTsc('windows .bin path', '.\\node_modules\\.bin\\tsc --noEmit', true);
|
||||
checkTsc('node on tsc.js with a heap flag', 'node --max-old-space-size=8192 node_modules/typescript/lib/tsc.js --noEmit', true);
|
||||
checkTsc('env prefix does not hide it', 'NODE_OPTIONS=--max-old-space-size=8192 npx tsc --noEmit', true);
|
||||
checkTsc('chained after something', 'git status && npx tsc --noEmit', true);
|
||||
|
||||
checkTsc('the repo script itself', 'pnpm run typecheck', false);
|
||||
checkTsc('scripts sub-project (the gate recommends it)', 'pnpm exec tsc --noEmit -p tsconfig.scripts.json', false);
|
||||
checkTsc('app sub-project', 'npx tsc --noEmit -p apps/notifications/tsconfig.json', false);
|
||||
checkTsc('named files', 'npx tsc --noEmit src/utils/a.ts src/utils/b.tsx', false);
|
||||
checkTsc('version', 'npx tsc --version', false);
|
||||
checkTsc('build mode', 'npx tsc -b packages/civitai-auth', false);
|
||||
checkTsc('opt-out marker', 'TYPECHECK_DIRECT=1 npx tsc --noEmit', false);
|
||||
checkTsc('opt-out marker, powershell', '$env:TYPECHECK_DIRECT=1; npx tsc --noEmit', false);
|
||||
checkTsc('prose mentioning it', 'echo "never run npx tsc --noEmit here"', false);
|
||||
checkTsc('tsc-alias is not tsc', 'npx tsc-alias -p tsconfig.json', false);
|
||||
|
||||
checkSuite('bare full unit run', 'pnpm run test:unit:run', true);
|
||||
checkSuite('full run with worker cap', 'pnpm run test:unit:run --max-workers=8', true);
|
||||
checkSuite('direct vitest, unit project, no files', "pnpm exec vitest run --project 'unit*'", true);
|
||||
|
||||
@@ -650,14 +650,31 @@ async function cmdTest(sub, rest) {
|
||||
case 'cancel':
|
||||
result = await daemonRequest(`/test-runs/${rest[0]}`, { method: 'DELETE' });
|
||||
break;
|
||||
case 'config':
|
||||
result = rest[0]
|
||||
? await daemonRequest('/test-runs/config', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ concurrency: Number(rest[0]) }),
|
||||
})
|
||||
case 'config': {
|
||||
// `test config 2 --max-workers 15` sets both in one POST. Each key is sent only when it was
|
||||
// typed, because the daemon leaves an absent key alone — sending a default for the one you
|
||||
// did not mean to change is how the cap gets dropped while raising concurrency.
|
||||
const body = {};
|
||||
if (rest[0] !== undefined && !rest[0].startsWith('--')) body.concurrency = Number(rest[0]);
|
||||
// Both spellings, because a caller who types the `=` form and silently gets no cap has no
|
||||
// way to tell that from a cap that was applied — the reply prints maxWorkers either way.
|
||||
const typecheckAt = rest.findIndex((a) => /^--typecheck(=|$)/.test(a));
|
||||
if (typecheckAt !== -1) {
|
||||
const inline = rest[typecheckAt].split('=')[1];
|
||||
body.typecheckConcurrency = Number(inline !== undefined ? inline : rest[typecheckAt + 1]);
|
||||
}
|
||||
const capAt = rest.findIndex((a) => /^--max-workers(=|$)/.test(a));
|
||||
if (capAt !== -1) {
|
||||
const inline = rest[capAt].split('=')[1];
|
||||
const raw = inline !== undefined ? inline : rest[capAt + 1];
|
||||
// `--max-workers none` is the only way back to an uncapped pool without a restart.
|
||||
body.maxWorkers = raw === undefined || raw === 'none' ? null : Number(raw);
|
||||
}
|
||||
result = Object.keys(body).length
|
||||
? await daemonRequest('/test-runs/config', { method: 'POST', body: JSON.stringify(body) })
|
||||
: await daemonRequest('/test-runs/config');
|
||||
break;
|
||||
}
|
||||
default:
|
||||
console.error(`Unknown test subcommand: ${action}`);
|
||||
console.error('Usage: test [run|wait|list|show|logs|cancel|config]');
|
||||
@@ -1043,6 +1060,8 @@ Commands:
|
||||
test list List runs and queue state
|
||||
test cancel <id> Cancel a queued or running run
|
||||
test config [n] Show or set the concurrency limit (0 pauses the queue)
|
||||
[--max-workers <n>|none] also caps each run's vitest pool
|
||||
[--typecheck <n>] sets the typecheck lane's limit
|
||||
wt stale List worktrees whose PR merged (read-only)
|
||||
wt rm <path> Remove a worktree safely (unlinks junctions first)
|
||||
[--stop-server] [--force]
|
||||
|
||||
@@ -87,6 +87,8 @@ function loadSkillConfig() {
|
||||
prewarmRoutes: ['/api/user/settings'],
|
||||
prewarmTimeout: 300000,
|
||||
testConcurrency: 1,
|
||||
typecheckConcurrency: 1,
|
||||
testMaxWorkers: null,
|
||||
prodGroups: [],
|
||||
};
|
||||
|
||||
@@ -168,6 +170,24 @@ function loadSkillConfig() {
|
||||
else if (value) console.error(`Ignoring TEST_CONCURRENCY=${value} (want an integer >= 0)`);
|
||||
break;
|
||||
}
|
||||
case 'TYPECHECK_CONCURRENCY': {
|
||||
// Same guard as TEST_CONCURRENCY, for the same reason: the queue is built at module
|
||||
// scope and its constructor throws, so a typo here must degrade, not stop the daemon.
|
||||
const parsed = parseInt(value, 10);
|
||||
if (Number.isInteger(parsed) && parsed >= 0) config.typecheckConcurrency = parsed;
|
||||
else if (value) console.error(`Ignoring TYPECHECK_CONCURRENCY=${value} (want an integer >= 0)`);
|
||||
break;
|
||||
}
|
||||
case 'TEST_MAX_WORKERS': {
|
||||
// Same reasoning as TEST_CONCURRENCY above — this feeds a constructor that throws, and
|
||||
// the queue is built at module scope, so a typo would stop the daemon binding at all.
|
||||
// An empty value means "no cap", which is the default and not an error.
|
||||
if (!value) break;
|
||||
const parsed = parseInt(value, 10);
|
||||
if (Number.isInteger(parsed) && parsed >= 1) config.testMaxWorkers = parsed;
|
||||
else console.error(`Ignoring TEST_MAX_WORKERS=${value} (want an integer >= 1)`);
|
||||
break;
|
||||
}
|
||||
case 'DEVSERVER_PROD_GROUPS':
|
||||
config.prodGroups = parseGroupList(value);
|
||||
break;
|
||||
@@ -1990,7 +2010,13 @@ async function stopAppSessions() {
|
||||
// Session manager
|
||||
const sessions = new Map();
|
||||
|
||||
const testQueue = new TestQueue({ concurrency: skillConfig.testConcurrency });
|
||||
const testQueue = new TestQueue({
|
||||
concurrency: {
|
||||
unit: skillConfig.testConcurrency,
|
||||
typecheck: skillConfig.typecheckConcurrency,
|
||||
},
|
||||
maxWorkers: skillConfig.testMaxWorkers,
|
||||
});
|
||||
|
||||
// A tracked session owns its port whatever its status says. Status is a report the daemon
|
||||
// writes about a process it cannot see into — it has read `crashed` for a session whose
|
||||
@@ -2570,11 +2596,22 @@ async function main() {
|
||||
}));
|
||||
return;
|
||||
}
|
||||
// `request` throws on an unknown kind, and it does so inside a request handler: unguarded,
|
||||
// that is a malformed body taking the handler down rather than a 400 to the caller.
|
||||
let view;
|
||||
try {
|
||||
view = testQueue.request({
|
||||
worktree: resolve(parsed.worktree),
|
||||
args: Array.isArray(parsed.args) ? parsed.args : [],
|
||||
kind: parsed.kind,
|
||||
});
|
||||
} catch (err) {
|
||||
res.writeHead(400);
|
||||
res.end(JSON.stringify({ error: err.message }));
|
||||
return;
|
||||
}
|
||||
res.writeHead(200);
|
||||
res.end(JSON.stringify(testQueue.request({
|
||||
worktree: resolve(parsed.worktree),
|
||||
args: Array.isArray(parsed.args) ? parsed.args : [],
|
||||
})));
|
||||
res.end(JSON.stringify(view));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -2589,7 +2626,14 @@ async function main() {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
testQueue.setConcurrency(parsed.concurrency);
|
||||
// Each key is applied only when the caller sent it. A POST carrying one field must
|
||||
// not reset the other to its default — that is how a concurrency change would
|
||||
// silently drop the worker cap and hand the box an uncapped pair.
|
||||
if (parsed.concurrency !== undefined) testQueue.setConcurrency(parsed.concurrency);
|
||||
if (parsed.typecheckConcurrency !== undefined) {
|
||||
testQueue.setConcurrency(parsed.typecheckConcurrency, 'typecheck');
|
||||
}
|
||||
if (parsed.maxWorkers !== undefined) testQueue.setMaxWorkers(parsed.maxWorkers);
|
||||
} catch (err) {
|
||||
res.writeHead(400);
|
||||
res.end(JSON.stringify({ error: err.message }));
|
||||
@@ -2599,9 +2643,18 @@ async function main() {
|
||||
res.writeHead(200);
|
||||
res.end(JSON.stringify({
|
||||
concurrency: testQueue.concurrency,
|
||||
typecheckConcurrency: testQueue.concurrencyFor('typecheck'),
|
||||
maxWorkers: testQueue.maxWorkers,
|
||||
paused: testQueue.paused,
|
||||
queued: testQueue.order.length,
|
||||
running: testQueue.running.size,
|
||||
lanes: {
|
||||
unit: { queued: testQueue.queuedFor('unit'), running: testQueue.runningFor('unit') },
|
||||
typecheck: {
|
||||
queued: testQueue.queuedFor('typecheck'),
|
||||
running: testQueue.runningFor('typecheck'),
|
||||
},
|
||||
},
|
||||
}));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -29,7 +29,36 @@ import { StringDecoder } from 'string_decoder';
|
||||
*/
|
||||
export const READ_WINDOW_BYTES = 64 * 1024;
|
||||
|
||||
/**
|
||||
* The kinds of run the queue serialises, and the npm script each one is.
|
||||
*
|
||||
* Separate lanes with separate limits rather than one pool, because they are not the same load: a
|
||||
* unit run saturates every core, while `tsc` is effectively single-threaded and spends its budget
|
||||
* on an 8 GB heap (see scripts/typecheck.mjs). One shared number would either starve the
|
||||
* typechecks behind a suite or let several suites run at once; there is no value right for both.
|
||||
*
|
||||
* `capWorkers` says whether `--max-workers` means anything to that script. tsc has no worker pool,
|
||||
* so handing it the flag would be an unknown argument rather than a smaller run.
|
||||
*/
|
||||
export const RUN_KINDS = {
|
||||
unit: { script: 'test:unit:run', capWorkers: true, defaultConcurrency: 1 },
|
||||
typecheck: { script: 'typecheck', capWorkers: false, defaultConcurrency: 1 },
|
||||
};
|
||||
|
||||
export const DEFAULT_KIND = 'unit';
|
||||
|
||||
export function normalizeKind(kind) {
|
||||
if (kind === undefined || kind === null || kind === '') return DEFAULT_KIND;
|
||||
if (!Object.prototype.hasOwnProperty.call(RUN_KINDS, kind)) {
|
||||
throw new Error(`unknown run kind: ${kind} (want one of ${Object.keys(RUN_KINDS).join(', ')})`);
|
||||
}
|
||||
return kind;
|
||||
}
|
||||
|
||||
export const DEFAULT_CONCURRENCY = 1;
|
||||
// null, not a number: no cap is not the same decision as a cap that happens to equal today's core
|
||||
// count, and only the first one keeps following the box when it changes.
|
||||
export const DEFAULT_MAX_WORKERS = null;
|
||||
const DEFAULT_ABANDON_AFTER_MS = 10 * 60 * 1000;
|
||||
const DEFAULT_RUN_TIMEOUT_MS = 30 * 60 * 1000;
|
||||
const DEFAULT_KILL_GRACE_MS = 30 * 1000;
|
||||
@@ -160,11 +189,41 @@ export function createOutputCapture(onLine) {
|
||||
return { path, writeFd, drain, close };
|
||||
}
|
||||
|
||||
export function defaultStartRun({ worktree, args, onLog, onExit }) {
|
||||
/**
|
||||
* The `--max-workers` argument a queued run should carry, if any.
|
||||
*
|
||||
* Vitest sizes its own pool at `cpus - 1`, which is right for a queue of one and wrong the moment
|
||||
* two runs share the box: at concurrency 2 an uncapped pair asks for 62 workers on 32 cores.
|
||||
* `VITEST_MAX_WORKERS` cannot do this job here — the daemon spawns the child with the DAEMON's
|
||||
* environment, so the caller's copy never arrives and the daemon's own is fixed at whatever it was
|
||||
* started with. The CLI flag is forwarded through `pnpm run` into vitest, and is the only knob that
|
||||
* reaches a queued run.
|
||||
*
|
||||
* A caller who passed their own `--max-workers` keeps it: they asked for a specific width, and a
|
||||
* second copy of the flag would decide the run by argument order rather than by intent.
|
||||
*/
|
||||
export function workerCapArgv(maxWorkers, args) {
|
||||
if (!maxWorkers) return [];
|
||||
// Both spellings: vitest reads kebab and camel as one flag (see canonicalFlag in
|
||||
// scripts/test-component-run.mjs), so matching only `--max-workers` would miss a caller's
|
||||
// `--maxWorkers=3` and append a second, conflicting width after it.
|
||||
if (args.some((a) => /^--max(?:-w|W)orkers(?:=|$)/.test(String(a)))) return [];
|
||||
return [`--max-workers=${maxWorkers}`];
|
||||
}
|
||||
|
||||
export function defaultStartRun({
|
||||
worktree,
|
||||
args,
|
||||
onLog,
|
||||
onExit,
|
||||
maxWorkers = null,
|
||||
kind = DEFAULT_KIND,
|
||||
}) {
|
||||
const emitter = new EventEmitter();
|
||||
const isWindows = process.platform === 'win32';
|
||||
const pnpm = isWindows ? 'pnpm.cmd' : 'pnpm';
|
||||
const argv = ['run', 'test:unit:run', ...args];
|
||||
const { script, capWorkers } = RUN_KINDS[normalizeKind(kind)];
|
||||
const argv = ['run', script, ...args, ...(capWorkers ? workerCapArgv(maxWorkers, args) : [])];
|
||||
|
||||
onLog('info', `> ${pnpm} ${argv.join(' ')}`);
|
||||
|
||||
@@ -294,6 +353,7 @@ export class TestQueue {
|
||||
constructor(options = {}) {
|
||||
const {
|
||||
concurrency = DEFAULT_CONCURRENCY,
|
||||
maxWorkers = DEFAULT_MAX_WORKERS,
|
||||
startRun = defaultStartRun,
|
||||
now = () => Date.now(),
|
||||
abandonAfterMs = DEFAULT_ABANDON_AFTER_MS,
|
||||
@@ -302,7 +362,8 @@ export class TestQueue {
|
||||
waitCommand = DEFAULT_WAIT_COMMAND,
|
||||
} = options;
|
||||
|
||||
this.concurrency = normalizeConcurrency(concurrency);
|
||||
this.limits = normalizeLimits(concurrency);
|
||||
this.maxWorkers = normalizeMaxWorkers(maxWorkers);
|
||||
this.startRun = startRun;
|
||||
this.now = now;
|
||||
this.abandonAfterMs = abandonAfterMs;
|
||||
@@ -315,17 +376,46 @@ export class TestQueue {
|
||||
this.running = new Set();
|
||||
}
|
||||
|
||||
/**
|
||||
* The unit lane's limit. Kept under the bare name `concurrency` because the daemon, the CLI and
|
||||
* the waiter all read it off a run view, and answering "which lane?" to that question would
|
||||
* break every one of them over a setting most callers never touch.
|
||||
*/
|
||||
get concurrency() {
|
||||
return this.limits[DEFAULT_KIND];
|
||||
}
|
||||
|
||||
concurrencyFor(kind) {
|
||||
return this.limits[normalizeKind(kind)];
|
||||
}
|
||||
|
||||
runningFor(kind) {
|
||||
const want = normalizeKind(kind);
|
||||
let n = 0;
|
||||
for (const id of this.running) if (this.runs.get(id)?.kind === want) n += 1;
|
||||
return n;
|
||||
}
|
||||
|
||||
queuedFor(kind) {
|
||||
const want = normalizeKind(kind);
|
||||
return this.order.reduce((n, id) => n + (this.runs.get(id)?.kind === want ? 1 : 0), 0);
|
||||
}
|
||||
|
||||
get paused() {
|
||||
return this.concurrency === 0;
|
||||
}
|
||||
|
||||
request({ worktree, args = [] } = {}) {
|
||||
request({ worktree, args = [], kind = DEFAULT_KIND } = {}) {
|
||||
if (!worktree) throw new Error('worktree is required');
|
||||
// Rejected BEFORE anything is recorded: an unknown kind must not leave a run in the map that
|
||||
// no lane will ever pump, which is an entry that waits forever while reporting position 1.
|
||||
const runKind = normalizeKind(kind);
|
||||
const at = this.now();
|
||||
const run = {
|
||||
id: nextId(),
|
||||
worktree,
|
||||
args,
|
||||
kind: runKind,
|
||||
status: 'queued',
|
||||
enqueuedAt: at,
|
||||
touchedAt: at,
|
||||
@@ -388,10 +478,20 @@ export class TestQueue {
|
||||
return this.view(id);
|
||||
}
|
||||
|
||||
setConcurrency(value) {
|
||||
this.concurrency = normalizeConcurrency(value);
|
||||
setConcurrency(value, kind = DEFAULT_KIND) {
|
||||
const lane = normalizeKind(kind);
|
||||
this.limits[lane] = normalizeConcurrency(value);
|
||||
this.pump();
|
||||
return this.concurrency;
|
||||
return this.limits[lane];
|
||||
}
|
||||
|
||||
/**
|
||||
* Takes effect on the NEXT run to start, never on one already running — the width is fixed when
|
||||
* vitest is spawned. Nothing here kills a run to resize it.
|
||||
*/
|
||||
setMaxWorkers(value) {
|
||||
this.maxWorkers = normalizeMaxWorkers(value);
|
||||
return this.maxWorkers;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -504,8 +604,15 @@ export class TestQueue {
|
||||
// --- internals ---
|
||||
|
||||
pump() {
|
||||
while (this.running.size < this.concurrency && this.order.length > 0) {
|
||||
this.start(this.order.shift());
|
||||
// Per lane, and each lane takes only ITS OWN head of the queue — a typecheck must not wait
|
||||
// behind a suite it shares no budget with, which is the whole reason the limits are separate.
|
||||
for (const kind of Object.keys(RUN_KINDS)) {
|
||||
for (;;) {
|
||||
if (this.runningFor(kind) >= this.limits[kind]) break;
|
||||
const at = this.order.findIndex((id) => this.runs.get(id)?.kind === kind);
|
||||
if (at === -1) break;
|
||||
this.start(this.order.splice(at, 1)[0]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -537,7 +644,14 @@ export class TestQueue {
|
||||
};
|
||||
|
||||
try {
|
||||
handle = this.startRun({ worktree: run.worktree, args: run.args, onLog, onExit });
|
||||
handle = this.startRun({
|
||||
worktree: run.worktree,
|
||||
args: run.args,
|
||||
onLog,
|
||||
onExit,
|
||||
maxWorkers: this.maxWorkers,
|
||||
kind: run.kind,
|
||||
});
|
||||
} catch (err) {
|
||||
// A runner that reported an exit and then threw has already produced a verdict; overwriting
|
||||
// it here would replace a real result with the noise that followed it.
|
||||
@@ -594,9 +708,21 @@ export class TestQueue {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Position within the run's OWN lane, not within `order`. A typecheck sitting behind four queued
|
||||
* suites it will never wait for is at position 1, and reporting 5 there is a number the caller
|
||||
* then budgets against for no reason.
|
||||
*/
|
||||
positionOf(id) {
|
||||
const at = this.order.indexOf(id);
|
||||
return at === -1 ? 0 : at + 1;
|
||||
const run = this.runs.get(id);
|
||||
if (!run) return 0;
|
||||
let n = 0;
|
||||
for (const queuedId of this.order) {
|
||||
if (this.runs.get(queuedId)?.kind !== run.kind) continue;
|
||||
n += 1;
|
||||
if (queuedId === id) return n;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
view(id) {
|
||||
@@ -608,11 +734,13 @@ export class TestQueue {
|
||||
worktree: run.worktree,
|
||||
args: run.args,
|
||||
// Exact, not estimated: the index in one ordered array. 0 means "not waiting behind anyone".
|
||||
kind: run.kind,
|
||||
position: this.positionOf(id),
|
||||
queueLength: this.order.length,
|
||||
running: this.running.size,
|
||||
concurrency: this.concurrency,
|
||||
paused: this.paused,
|
||||
queueLength: this.queuedFor(run.kind),
|
||||
running: this.runningFor(run.kind),
|
||||
concurrency: this.limits[run.kind],
|
||||
maxWorkers: this.maxWorkers,
|
||||
paused: this.limits[run.kind] === 0,
|
||||
enqueuedAt: run.enqueuedAt,
|
||||
startedAt: run.startedAt,
|
||||
finishedAt: run.finishedAt,
|
||||
@@ -627,6 +755,40 @@ export class TestQueue {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Same defensive shape as normalizeConcurrency, with one difference that matters: 0 is REJECTED
|
||||
* rather than treated as a pause. `--max-workers=0` is not a smaller run, it is a run with no
|
||||
* workers, and vitest's own resolution treats a falsy value as "unset" — so a 0 that slipped
|
||||
* through here would silently restore the uncapped pool the setting exists to prevent.
|
||||
*/
|
||||
function normalizeMaxWorkers(value) {
|
||||
if (value === null || value === undefined || value === '') return null;
|
||||
const parsed = typeof value === 'number' ? value : parseInt(value, 10);
|
||||
if (!Number.isInteger(parsed) || parsed < 1) {
|
||||
throw new Error(`maxWorkers must be an integer >= 1, or null for no cap, got: ${value}`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Accepts the old scalar as well as a per-lane object. The scalar sets the UNIT lane only and
|
||||
* leaves the others at their defaults — reinterpreting it as "every lane" would silently raise the
|
||||
* typecheck limit on every machine that had ever set TEST_CONCURRENCY for the suite.
|
||||
*/
|
||||
function normalizeLimits(value) {
|
||||
const limits = {};
|
||||
for (const [kind, spec] of Object.entries(RUN_KINDS)) limits[kind] = spec.defaultConcurrency;
|
||||
if (value === undefined || value === null) return limits;
|
||||
if (typeof value === 'object') {
|
||||
for (const [kind, n] of Object.entries(value)) {
|
||||
limits[normalizeKind(kind)] = normalizeConcurrency(n);
|
||||
}
|
||||
return limits;
|
||||
}
|
||||
limits[DEFAULT_KIND] = normalizeConcurrency(value);
|
||||
return limits;
|
||||
}
|
||||
|
||||
function normalizeConcurrency(value) {
|
||||
const parsed = typeof value === 'number' ? value : parseInt(value, 10);
|
||||
if (!Number.isInteger(parsed) || parsed < 0) {
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
import { EventEmitter } from 'events';
|
||||
import { beforeEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import * as QueueModule from '../../.claude/skills/dev-server/scripts/test-queue.mjs';
|
||||
|
||||
// The module is plain .mjs, so TS infers `request`'s payload from nothing and lands on types no
|
||||
// call here can satisfy. Named once instead of cast at every call site.
|
||||
type Kind = 'unit' | 'typecheck';
|
||||
type View = {
|
||||
id: string;
|
||||
kind: Kind;
|
||||
status: string;
|
||||
position: number;
|
||||
queueLength: number;
|
||||
running: number;
|
||||
concurrency: number;
|
||||
};
|
||||
type Queue = {
|
||||
request: (run: { worktree: string; args?: string[]; kind?: string }) => View;
|
||||
get: (id: string) => View;
|
||||
list: () => View[];
|
||||
setConcurrency: (value: number, kind?: Kind) => number;
|
||||
concurrencyFor: (kind: Kind) => number;
|
||||
};
|
||||
type Runner = EventEmitter & { finish: (code: number) => void; kind: Kind; worktree: string };
|
||||
type RunnerArgs = { worktree: string; kind: Kind };
|
||||
|
||||
const { TestQueue } = QueueModule as unknown as {
|
||||
TestQueue: new (options: Record<string, unknown>) => Queue;
|
||||
};
|
||||
|
||||
let started: Runner[];
|
||||
|
||||
const build = (concurrency: unknown) =>
|
||||
new TestQueue({
|
||||
concurrency,
|
||||
now: () => 1_000,
|
||||
startRun: ({ worktree, kind }: RunnerArgs) => {
|
||||
const handle = new EventEmitter() as Runner;
|
||||
handle.finish = (code) => handle.emit('exit', code);
|
||||
handle.kind = kind;
|
||||
handle.worktree = worktree;
|
||||
started.push(handle);
|
||||
return handle;
|
||||
},
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
started = [];
|
||||
});
|
||||
|
||||
/**
|
||||
* 🔴 The property the lanes exist for. If you are here because this fails after collapsing the
|
||||
* lanes back into one pool: that is the regression, not a stale test. A single pool puts a
|
||||
* one-core typecheck behind every queued 31-worker suite, which is the "everything crawls"
|
||||
* condition this was built to end.
|
||||
*/
|
||||
describe('a typecheck does not wait behind the unit lane', () => {
|
||||
it('starts immediately while the unit lane is full and has runs queued ahead of it', () => {
|
||||
const queue = build({ unit: 1, typecheck: 1 });
|
||||
queue.request({ worktree: '/wt/suite-a' });
|
||||
queue.request({ worktree: '/wt/suite-b' });
|
||||
queue.request({ worktree: '/wt/suite-c' });
|
||||
|
||||
const check = queue.request({ worktree: '/wt/tc', kind: 'typecheck' });
|
||||
|
||||
expect(check.status).toBe('running');
|
||||
expect(started.map((r) => `${r.kind}:${r.worktree}`)).toEqual([
|
||||
'unit:/wt/suite-a',
|
||||
'typecheck:/wt/tc',
|
||||
]);
|
||||
});
|
||||
|
||||
// Position is within the run's own lane. Counting the suites in `order` would tell a caller they
|
||||
// are fourth in line for a slot they will never wait on.
|
||||
it("reports a queued typecheck's position within its own lane only", () => {
|
||||
const queue = build({ unit: 1, typecheck: 1 });
|
||||
queue.request({ worktree: '/wt/suite-a' });
|
||||
queue.request({ worktree: '/wt/suite-b' });
|
||||
queue.request({ worktree: '/wt/suite-c' });
|
||||
queue.request({ worktree: '/wt/tc-running', kind: 'typecheck' });
|
||||
|
||||
const waiting = queue.request({ worktree: '/wt/tc-waiting', kind: 'typecheck' });
|
||||
|
||||
expect(waiting.status).toBe('queued');
|
||||
expect(waiting.position).toBe(1);
|
||||
expect(waiting.queueLength).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('each lane enforces its own limit', () => {
|
||||
it('holds a second typecheck while the first runs, even with the unit lane idle', () => {
|
||||
const queue = build({ unit: 4, typecheck: 1 });
|
||||
queue.request({ worktree: '/wt/tc-1', kind: 'typecheck' });
|
||||
|
||||
const second = queue.request({ worktree: '/wt/tc-2', kind: 'typecheck' });
|
||||
|
||||
expect(second.status).toBe('queued');
|
||||
expect(started).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('holds a second suite while the first runs, even with the typecheck lane idle', () => {
|
||||
const queue = build({ unit: 1, typecheck: 4 });
|
||||
queue.request({ worktree: '/wt/suite-a' });
|
||||
|
||||
const second = queue.request({ worktree: '/wt/suite-b' });
|
||||
|
||||
expect(second.status).toBe('queued');
|
||||
expect(started).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('frees a lane slot only for that lane', () => {
|
||||
const queue = build({ unit: 1, typecheck: 1 });
|
||||
queue.request({ worktree: '/wt/suite-a' });
|
||||
const waitingSuite = queue.request({ worktree: '/wt/suite-b' });
|
||||
queue.request({ worktree: '/wt/tc-1', kind: 'typecheck' });
|
||||
const waitingCheck = queue.request({ worktree: '/wt/tc-2', kind: 'typecheck' });
|
||||
|
||||
started.find((r) => r.kind === 'typecheck')!.finish(0);
|
||||
|
||||
expect(queue.get(waitingCheck.id).status).toBe('running');
|
||||
expect(queue.get(waitingSuite.id).status).toBe('queued');
|
||||
});
|
||||
|
||||
it('starts waiting typechecks when their lane is raised at runtime', () => {
|
||||
const queue = build({ unit: 1, typecheck: 1 });
|
||||
queue.request({ worktree: '/wt/tc-1', kind: 'typecheck' });
|
||||
const second = queue.request({ worktree: '/wt/tc-2', kind: 'typecheck' });
|
||||
|
||||
queue.setConcurrency(2, 'typecheck');
|
||||
|
||||
expect(queue.get(second.id).status).toBe('running');
|
||||
});
|
||||
});
|
||||
|
||||
describe('configuring the lanes', () => {
|
||||
/**
|
||||
* The scalar form is what every existing caller passes, from TEST_CONCURRENCY. Reading it as
|
||||
* "every lane" would quietly raise the typecheck limit on any machine that had only ever tuned
|
||||
* the suite — nobody who set that variable asked for more concurrent tsc heaps.
|
||||
*/
|
||||
it('reads a bare number as the unit lane only', () => {
|
||||
const queue = build(3);
|
||||
|
||||
expect(queue.concurrencyFor('unit')).toBe(3);
|
||||
expect(queue.concurrencyFor('typecheck')).toBe(1);
|
||||
});
|
||||
|
||||
it('refuses an unknown kind without recording a run no lane would ever start', () => {
|
||||
const queue = build({ unit: 1, typecheck: 1 });
|
||||
|
||||
expect(() => queue.request({ worktree: '/wt/x', kind: 'lint' })).toThrow(/unknown run kind/);
|
||||
expect(queue.list()).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -11,7 +11,9 @@ vi.mock('child_process', async (importOriginal) => ({
|
||||
|
||||
// Lives under scripts/ because the daemon is not part of the app's module graph — same arrangement
|
||||
// as the rest of the queue's tests.
|
||||
const { defaultStartRun } = await import('../../.claude/skills/dev-server/scripts/test-queue.mjs');
|
||||
const { defaultStartRun, TestQueue } = await import(
|
||||
'../../.claude/skills/dev-server/scripts/test-queue.mjs'
|
||||
);
|
||||
|
||||
const fakeChild = () => {
|
||||
const child = new EventEmitter() as EventEmitter & Record<string, unknown>;
|
||||
@@ -69,3 +71,93 @@ describe('the queued run does not re-enter the queue', () => {
|
||||
delete process.env.CIVITAI_TEST_QUEUE_PROBE;
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* The worker cap only exists because concurrency > 1 is on the table: vitest sizes its own pool at
|
||||
* `cpus - 1`, so two uncapped runs ask for 62 workers on a 32-core box. `VITEST_MAX_WORKERS` cannot
|
||||
* carry it — the daemon spawns the child with the daemon's own environment — so the CLI flag is the
|
||||
* only channel, and these pin that it is actually on the command line.
|
||||
*
|
||||
* 🔴 If you are here to delete one of these: the failure they protect against is SILENT. A cap that
|
||||
* never reaches vitest leaves the summary, the exit code and the test count all identical; the only
|
||||
* visible difference is the box falling over under an oversubscribed pair. Do not replace these with
|
||||
* an assertion on `queue.maxWorkers`, which passes with the argv line deleted.
|
||||
*/
|
||||
describe("the queue caps each run's vitest pool", () => {
|
||||
const argvOf = (call: number) => spawn.mock.calls[call][1] as string[];
|
||||
|
||||
// Typed here rather than at each call: the runner is a .mjs module, so TS infers a bare
|
||||
// EventEmitter and does not see the `dispose` the handle actually carries.
|
||||
const start = (opts: Record<string, unknown>) => {
|
||||
const handle = defaultStartRun({
|
||||
worktree: '/repo',
|
||||
args: [],
|
||||
onLog: () => undefined,
|
||||
onExit: () => undefined,
|
||||
...opts,
|
||||
}) as EventEmitter & { dispose: () => void };
|
||||
handle.dispose();
|
||||
};
|
||||
|
||||
it('puts --max-workers on the command line when a cap is set', () => {
|
||||
start({ maxWorkers: 15 });
|
||||
expect(argvOf(0)).toEqual(['run', 'test:unit:run', '--max-workers=15']);
|
||||
});
|
||||
|
||||
// The default is no cap, and that has to stay byte-identical to what the daemon ran before this
|
||||
// setting existed — otherwise every run on a machine that never configured it changes width.
|
||||
it('adds nothing when no cap is set', () => {
|
||||
start({});
|
||||
expect(argvOf(0)).toEqual(['run', 'test:unit:run']);
|
||||
});
|
||||
|
||||
// A caller who named a width asked for that width. Appending a second copy would decide the run
|
||||
// by argument order, which is not a thing either of them chose.
|
||||
it('leaves a caller-supplied width alone', () => {
|
||||
start({ args: ['--max-workers=3'], maxWorkers: 15 });
|
||||
expect(argvOf(0)).toEqual(['run', 'test:unit:run', '--max-workers=3']);
|
||||
});
|
||||
|
||||
// vitest reads kebab and camel as one flag, so a caller's `--maxWorkers` is the same request as
|
||||
// `--max-workers` and must suppress the queue's copy just the same.
|
||||
it('honours the camelCase spelling of the caller flag too', () => {
|
||||
start({ args: ['--maxWorkers=3'], maxWorkers: 15 });
|
||||
expect(argvOf(0)).toEqual(['run', 'test:unit:run', '--maxWorkers=3']);
|
||||
});
|
||||
|
||||
// tsc has no worker pool, so the flag would reach it as an unknown argument rather than a smaller
|
||||
// run. Pinned because the cap is configured on the QUEUE, which now serves both lanes.
|
||||
it('runs the typecheck script and never hands it the worker cap', () => {
|
||||
start({ kind: 'typecheck', maxWorkers: 15 });
|
||||
expect(argvOf(0)).toEqual(['run', 'typecheck']);
|
||||
});
|
||||
|
||||
it('honours the space-separated spelling of the caller flag too', () => {
|
||||
start({ args: ['--max-workers', '3'], maxWorkers: 15 });
|
||||
expect(argvOf(0)).toEqual(['run', 'test:unit:run', '--max-workers', '3']);
|
||||
});
|
||||
|
||||
// The cap is configured on the QUEUE and has to survive the hop into the runner. Asserting on
|
||||
// defaultStartRun alone would pass with that hop deleted.
|
||||
it("hands the queue's cap to the runner it starts", () => {
|
||||
const startRun = vi.fn<(opts: { maxWorkers: number | null }) => EventEmitter>(
|
||||
() => new EventEmitter()
|
||||
);
|
||||
// Cast for the same reason as the handle above — TestQueue comes from a .mjs module, so TS
|
||||
// infers `request`'s payload from nothing and lands on `{ args?: never[] }`.
|
||||
const queue = new TestQueue({ concurrency: 1, maxWorkers: 15, startRun }) as unknown as {
|
||||
request: (run: { worktree: string; args: string[] }) => unknown;
|
||||
};
|
||||
|
||||
queue.request({ worktree: '/repo', args: [] });
|
||||
|
||||
expect(startRun).toHaveBeenCalledTimes(1);
|
||||
expect(startRun.mock.calls[0][0]).toMatchObject({ maxWorkers: 15 });
|
||||
});
|
||||
|
||||
// 0 is not a smaller run, it is no run — and vitest reads a falsy width as "unset", so a 0 that
|
||||
// slipped through would silently restore the uncapped pool this setting exists to prevent.
|
||||
it('refuses a width of zero rather than treating it as a pause', () => {
|
||||
expect(() => new TestQueue({ concurrency: 1, maxWorkers: 0 })).toThrow(/integer >= 1/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
import { spawn } from 'child_process';
|
||||
import { createServer } from 'http';
|
||||
import type { AddressInfo } from 'net';
|
||||
import { dirname, resolve } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
|
||||
|
||||
/**
|
||||
* What a script ENQUEUES, observed on the wire. Drives the real script as a child process against a
|
||||
* stub daemon on DEV_DAEMON_PORT, the same arrangement as test-unit-run.test.ts.
|
||||
*
|
||||
* 🔴 The kind is the whole contract here, and nothing else checks it. A typecheck posted WITHOUT
|
||||
* `kind: 'typecheck'` is accepted as a unit run, so the daemon spawns `pnpm run test:unit:run` —
|
||||
* the caller asked for a one-core tsc and gets a 31-worker suite, reported under the right exit
|
||||
* code. Every other test in this area would stay green.
|
||||
*/
|
||||
async function enqueuedBy(script: string) {
|
||||
const posted: Record<string, unknown>[] = [];
|
||||
const server = createServer((req, res) => {
|
||||
let body = '';
|
||||
req.on('data', (chunk) => (body += chunk));
|
||||
req.on('end', () => {
|
||||
if (req.method === 'POST' && req.url === '/test-runs') posted.push(JSON.parse(body || '{}'));
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(
|
||||
req.url?.includes('/logs')
|
||||
? JSON.stringify({ logs: [] })
|
||||
: JSON.stringify({
|
||||
id: 'stub',
|
||||
status: 'completed',
|
||||
exitCode: 0,
|
||||
position: 0,
|
||||
queueLength: 0,
|
||||
logIndex: 0,
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
await new Promise<void>((done) => server.listen(0, '127.0.0.1', done));
|
||||
const { port } = server.address() as AddressInfo;
|
||||
|
||||
try {
|
||||
await new Promise<void>((done) => {
|
||||
const child = spawn(process.execPath, [resolve(repoRoot, script)], {
|
||||
cwd: repoRoot,
|
||||
env: {
|
||||
...process.env,
|
||||
CI: '',
|
||||
CIVITAI_TEST_QUEUE: '1',
|
||||
DEV_DAEMON_PORT: String(port),
|
||||
TYPECHECK_TSC_PATH: '',
|
||||
TYPECHECK_HEAP_MB: '',
|
||||
},
|
||||
stdio: ['ignore', 'ignore', 'ignore'],
|
||||
});
|
||||
// Bounded on its own. If routing is broken the script falls through to a REAL tsc, which
|
||||
// runs for minutes; killing it here turns that into the assertion below naming what was
|
||||
// (not) posted, instead of a 60s test timeout that says nothing.
|
||||
const cap = setTimeout(() => child.kill(), 15_000);
|
||||
child.on('exit', () => {
|
||||
clearTimeout(cap);
|
||||
done();
|
||||
});
|
||||
});
|
||||
} finally {
|
||||
await new Promise<void>((done) => server.close(() => done()));
|
||||
}
|
||||
return posted;
|
||||
}
|
||||
|
||||
describe('what each script enqueues', () => {
|
||||
it('a full typecheck enqueues in the typecheck lane', async () => {
|
||||
const posted = await enqueuedBy('scripts/typecheck.mjs');
|
||||
expect(posted.map((p) => p.kind)).toEqual(['typecheck']);
|
||||
});
|
||||
|
||||
it('a full unit run enqueues in the unit lane', async () => {
|
||||
const posted = await enqueuedBy('scripts/test-unit-run.mjs');
|
||||
expect(posted.map((p) => p.kind)).toEqual(['unit']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,44 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { typecheckQueueDecision } from '../typecheck-queue.mjs';
|
||||
|
||||
const ON = { CIVITAI_TEST_QUEUE: '1' };
|
||||
|
||||
describe('which typechecks go through the queue', () => {
|
||||
it('queues a full typecheck when the flag is on', () => {
|
||||
expect(typecheckQueueDecision([], ON)).toEqual({ queue: true });
|
||||
});
|
||||
|
||||
it('never queues on CI', () => {
|
||||
expect(typecheckQueueDecision([], { ...ON, CI: 'true' }).queue).toBe(false);
|
||||
});
|
||||
|
||||
it.each(['', '0', 'false', 'off', 'no'])('stays direct when the flag is %j', (flag) => {
|
||||
expect(typecheckQueueDecision([], { CIVITAI_TEST_QUEUE: flag }).queue).toBe(false);
|
||||
});
|
||||
|
||||
// `-p tsconfig.scripts.json` is how the scripts gate runs tsc. Queueing that behind a full-repo
|
||||
// check turns a cheap narrow run into a wait, and pushes callers toward batching work into
|
||||
// fewer, bigger runs — the opposite of what the queue is for.
|
||||
it('stays direct when any argument narrows the run', () => {
|
||||
expect(typecheckQueueDecision(['-p', 'tsconfig.scripts.json'], ON).queue).toBe(false);
|
||||
});
|
||||
|
||||
/**
|
||||
* 🔴 Without this the typecheck TESTS break on any machine with the flag set: each one drives
|
||||
* scripts/typecheck.mjs with a stub tsc through this seam, and a queued run is spawned by the
|
||||
* daemon, which runs the REAL tsc — so every case would wait behind real typechecks and then
|
||||
* assert on the wrong process's output.
|
||||
*/
|
||||
it('stays direct when the tsc test seam is in use', () => {
|
||||
expect(typecheckQueueDecision([], { ...ON, TYPECHECK_TSC_PATH: '/stub/tsc.js' }).queue).toBe(
|
||||
false
|
||||
);
|
||||
});
|
||||
|
||||
// A queued run is spawned with the daemon's environment, so the caller's heap override would be
|
||||
// silently dropped — they would ask for a size and get the default.
|
||||
it('stays direct when the caller overrides the heap', () => {
|
||||
expect(typecheckQueueDecision([], { ...ON, TYPECHECK_HEAP_MB: '12288' }).queue).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -31,6 +31,8 @@ let DAEMON = null;
|
||||
// Whether the queue has taken ownership of this run. Once it has, a later failure must NOT be
|
||||
// answered by starting a second, unqueued suite — see the note where this is set.
|
||||
let accepted = false;
|
||||
/** Read by a caller of `runQueued` deciding what a later failure may do. */
|
||||
export const queueAccepted = () => accepted;
|
||||
const POLL_MS = 2000;
|
||||
|
||||
/**
|
||||
@@ -118,7 +120,15 @@ function warnIfLogsDropped(state) {
|
||||
);
|
||||
}
|
||||
|
||||
async function runQueued(args) {
|
||||
/**
|
||||
* Exported so `scripts/typecheck.mjs` can queue through the SAME client rather than a copy of it.
|
||||
* Everything below — the accepted flag, the 404-means-restarted rule, the drain before exit — was
|
||||
* learned the hard way on this path, and a second client would have to relearn each of them.
|
||||
*
|
||||
* `fallback` is what an unusable queue degrades to. It is only ever called BEFORE the daemon has
|
||||
* accepted the run; after acceptance a failure exits 2 instead, for the reason given at `accepted`.
|
||||
*/
|
||||
export async function runQueued(args, { kind = 'unit', fallback = runDirect } = {}) {
|
||||
// Resolved once, up front: this is the module that decides pass from fail, and a waiter that
|
||||
// discovers it cannot load that rule at the moment it must apply it has no verdict to give.
|
||||
const { exitCodeFor } = await import(pathToFileURL(QUEUE).href);
|
||||
@@ -133,20 +143,20 @@ async function runQueued(args) {
|
||||
DAEMON = resolveDaemonUrl();
|
||||
} catch (err) {
|
||||
console.error(`Test queue address unusable (${err.message}); running directly.`);
|
||||
return runDirect(args);
|
||||
return fallback(args);
|
||||
}
|
||||
|
||||
let run;
|
||||
try {
|
||||
run = await post('/test-runs', { worktree: repoRoot, args });
|
||||
run = await post('/test-runs', { worktree: repoRoot, args, kind });
|
||||
} catch {
|
||||
await ensureDaemon();
|
||||
try {
|
||||
run = await post('/test-runs', { worktree: repoRoot, args });
|
||||
run = await post('/test-runs', { worktree: repoRoot, args, kind });
|
||||
} catch (err) {
|
||||
// Never leave a caller unable to run tests because the queue is unavailable.
|
||||
console.error(`Test queue unreachable (${err.message}); running directly.`);
|
||||
return runDirect(args);
|
||||
return fallback(args);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
/**
|
||||
* Whether a `pnpm run typecheck` should go through the dev-server queue.
|
||||
*
|
||||
* Its own module rather than a function inside scripts/typecheck.mjs, because that file runs tsc
|
||||
* at import: nothing could load it to test this rule without starting a multi-minute typecheck.
|
||||
*/
|
||||
|
||||
export function typecheckQueueDecision(args, env) {
|
||||
if (env.CI) return { queue: false, why: 'CI runs the typecheck directly' };
|
||||
if (!env.CIVITAI_TEST_QUEUE || /^(0|false|off|no)$/i.test(env.CIVITAI_TEST_QUEUE)) {
|
||||
return { queue: false, why: 'CIVITAI_TEST_QUEUE is not set' };
|
||||
}
|
||||
// The typecheck tests drive scripts/typecheck.mjs through this seam with a stub tsc. On a machine
|
||||
// with the queue flag set they would otherwise queue behind real typechecks and run the REAL tsc
|
||||
// in the daemon's child, never the stub — each case minutes long and asserting on the wrong run.
|
||||
if (env.TYPECHECK_TSC_PATH) return { queue: false, why: 'the tsc test seam is in use' };
|
||||
// A queued run is spawned with the DAEMON's environment, not the caller's, so an override set
|
||||
// here would silently not apply — the caller asked for a heap size and would get the default.
|
||||
if (env.TYPECHECK_HEAP_MB) return { queue: false, why: 'a heap override does not reach a queued run' };
|
||||
// Any argument at all means a narrowed run (`-p tsconfig.scripts.json`, a single project). Same
|
||||
// rule as scripts/test-component-run.mjs and for the same reason: telling a cheap narrow tsc
|
||||
// invocation from an expensive one by parsing its flags is a guess, and queueing a cheap one
|
||||
// turns a quick check into a wait behind a full run.
|
||||
if (args.length > 0) return { queue: false, why: 'arguments narrow the run' };
|
||||
return { queue: true };
|
||||
}
|
||||
@@ -53,10 +53,37 @@
|
||||
*/
|
||||
|
||||
import { spawn } from 'node:child_process';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { createRequire } from 'node:module';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import { typecheckQueueDecision } from './typecheck-queue.mjs';
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
|
||||
// A full typecheck costs one core and up to an 8 GB heap, and several agents starting one at once
|
||||
// is what pegs the box. With CIVITAI_TEST_QUEUE set this hands the run to the dev-server queue's
|
||||
// typecheck lane, which spawns `pnpm run typecheck` back in this worktree with the flag off — so
|
||||
// the child comes through here again and falls to the direct path below.
|
||||
//
|
||||
// Top-level await on purpose: a queued run never returns (the client exits the process with the
|
||||
// run's verdict), so everything below is reached only when the queue could not take the run.
|
||||
if (typecheckQueueDecision(process.argv.slice(2), process.env).queue) {
|
||||
const client = resolve(dirname(fileURLToPath(import.meta.url)), 'test-unit-run.mjs');
|
||||
if (existsSync(client)) {
|
||||
const { runQueued, queueAccepted } = await import(pathToFileURL(client).href);
|
||||
try {
|
||||
await runQueued([], { kind: 'typecheck', fallback: () => undefined });
|
||||
} catch (err) {
|
||||
if (queueAccepted()) {
|
||||
console.error(`Lost contact with the test queue (${err?.message ?? err}). The run may still be queued.`);
|
||||
process.exit(2);
|
||||
}
|
||||
console.error(`Typecheck queue failed (${err?.message ?? err}); running directly.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const DEFAULT_HEAP_MB = 8192;
|
||||
const heapMb = Number(process.env.TYPECHECK_HEAP_MB || DEFAULT_HEAP_MB);
|
||||
if (!Number.isFinite(heapMb) || heapMb <= 0) {
|
||||
|
||||
Reference in New Issue
Block a user