From 4b17ea7d350b9445ff4c1da01348bbb1ddfcfb21 Mon Sep 17 00:00:00 2001 From: Maxim Date: Thu, 13 Aug 2026 17:24:05 +0200 Subject: [PATCH] fix(scripts): tokenize before hunting loader calls in the purity gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #4893 hard-fail gate's loader-call detector gave WRONG VERDICTS IN BOTH DIRECTIONS. It layered two regexes — a comment/string/template alternation that blanked only the comment branch, and `\b(?:import|require(?:\.resolve)?)\s*\(` over the result — then classified an argument as static from the FIRST CHARACTER after the paren. All nine shapes below were reproduced against the real gate before the rewrite: false FAIL throw new Error("use require(path) instead") false FAIL `import(${x})` inside a template false FAIL o.import(y) / mod.require(x) (member calls, not loaders) false PASS /https:\/\//; …import(n) (the regex's `//` blanked the rest of the line, hiding a real dynamic call) false PASS import(`stream${n}`) (merely STARTS with a quote) false PASS import("zo" + n) (same) false PASS import(`${base}/v2/index.mjs`) (same — the fat entry) false PASS __require(name) (no \b inside `__require`) Replaced with `scanSource`, a single-pass tokenizer that classifies every character as code / comment / string / template / regex and returns a length-preserving masked view plus a literal-span list. The one surviving regex now only ever sees code, so import-shaped TEXT cannot reach it at all; an argument counts as static only when it is one COMPLETE literal with no concatenation or interpolation; `__require` is matched; and a member call is rejected both by lookbehind and by a whitespace-skipping back-scan (so `m\n .import(x)` is not a loader either). Proven in both directions: nine innocent/violation pairs run through the real `assertEntryPurity`, each innocent form CLEAN and each matching real violation FAIL. Re-proved end-to-end by prepending `import "streamdown"` to the real dist/v2/headless.mjs — exit 1 naming all five families — then restoring it byte-identically. On the untouched dist the scan sees 66 loader calls in the `.cjs` graph and classifies all 66 static, so it passes because it LOOKED. Also adds the first `.cjs` fixtures: every existing fixture was `.mjs`, leaving the script's `format: "cjs"` branch and the `require()` shape asserted by nothing. Tests 24 → 47. `stripComments` is renamed `maskNonCode`, since it now blanks literals and regexes too; it had no caller outside this script and its test. The RN guard keeps its own copy, untouched. dev-docs/bundle-size.md: the four holes a sibling agent documented as known limitations this round are closed and removed from that list; what genuinely remains (regex-vs-division heuristic, no JSX/TS, indirect loaders) replaces them. Co-Authored-By: Claude Opus 5 (1M context) --- dev-docs/bundle-size.md | 55 ++- .../__tests__/assert-headless-purity.test.mjs | 274 ++++++++++- .../__tests__/fixtures/purity-chunk.cjs | 8 + .../purity-cjs-unanalyzable-entry.cjs | 16 + .../__tests__/fixtures/purity-entry.cjs | 4 + .../scripts/assert-headless-purity.mjs | 439 ++++++++++++++++-- 6 files changed, 728 insertions(+), 68 deletions(-) create mode 100644 packages/react-core/scripts/__tests__/fixtures/purity-chunk.cjs create mode 100644 packages/react-core/scripts/__tests__/fixtures/purity-cjs-unanalyzable-entry.cjs create mode 100644 packages/react-core/scripts/__tests__/fixtures/purity-entry.cjs diff --git a/dev-docs/bundle-size.md b/dev-docs/bundle-size.md index e050a07e9c..9902857c9b 100644 --- a/dev-docs/bundle-size.md +++ b/dev-docs/bundle-size.md @@ -91,7 +91,6 @@ baseline to maintain, and no conflict with the Phase 2 freeze on `limit` fields: Both entries are guarded because `@copilotkit/react-native` imports both. Runs in `static_bundle_size.yml` — the step there is named after `/v2/headless` only, but the script asserts `/v2/context` as well. Mechanically: - - It bundles each entry with **esbuild** (`bundle: true`, `write: false`, `metafile: true`; `react` / `react-dom` and the JSX runtimes external; CSS and font assets on the `empty` loader, which still records them as graph @@ -117,18 +116,25 @@ baseline to maintain, and no conflict with the Phase 2 freeze on `limit` fields: warnings print but are non-fatal — third-party code warns for reasons that say nothing about #4893. - The one place it still reads **text** is to find `import(…)` / `require(…)` / - `require.resolve(…)` calls whose argument is not a string literal — the one - edge shape a bundler genuinely cannot see through — and only in the graph's - first-party files. Comments are stripped first (a single - comment/string/template alternation, so a `//` inside a string and a quote - inside a comment are each consumed by the other branch), which is why a - documented counter-example naming a banned dep no longer trips it. + `require.resolve(…)` / `__require(…)` calls whose argument is not a **complete** + string literal — the one edge shape a bundler genuinely cannot see through — + and only in the graph's first-party files. That scan runs over the output of + `scanSource`, a small single-pass **tokenizer** that blanks comments, strings, + templates _and_ regex literals while preserving offsets, so the one surviving + regex only ever sees code. A documented counter-example naming a banned dep + cannot trip it, a `//` inside a regex cannot hide a real call, and an argument + counts as static only when it is one whole literal with no concatenation or + interpolation. - Negative tests: `packages/react-core/scripts/__tests__/assert-headless-purity.test.mjs`, run by `pnpm --filter @copilotkit/react-core test:scripts` (chained from that package's `test`). They cover both directions — a forbidden dep reached only - through a relative chunk edge, a forbidden dep left external, an unresolvable - edge, an unanalyzable loader call, and banned tokens present only in comments - and strings, which must **pass**. + through a relative chunk edge (in both an `.mjs` and a `.cjs` entry, so the + `format: "cjs"` branch and the `require()` shape are exercised too), a + forbidden dep left external, an unresolvable edge, an unanalyzable loader call, + and banned tokens present only in comments and strings, which must **pass**. + Each detector shape fixed in the tokenizer rewrite has a **pair**: the innocent + form must pass and the matching real violation must fail. + 2. `packages/react-native/src/__tests__/headless-entry-surface.test.ts` — walks the relative-import graph of this package's own `src/`, from both `src/headless.ts` and `src/index.ts`, and fails if a reached module imports a @@ -136,8 +142,9 @@ baseline to maintain, and no conflict with the Phase 2 freeze on `limit` fields: render stack directly, or (headless entry only) pulls the optional native chat/attachment peer deps. It extracts static `import`/`export … from`, bare side-effect `import "x"`, `import()` and `require()`/`require.resolve()` — - Metro follows the lazy forms too — strips comments with the same - comment/string/template alternation the purity gate uses, reports a + Metro follows the lazy forms too — strips comments with its own + comment/string/template alternation (the purity gate has since moved to the + tokenizer described above), reports a non-literal loader argument as unanalyzable rather than ignoring it, and fails loudly on a local edge it cannot resolve. Runs in the normal test job. @@ -155,18 +162,18 @@ is closed: react-core's own build leaves `@copilotkit/core`, assertion, not a complete one. Documented rather than glossed, because a doc that claims a gate is airtight is how the last round of this went wrong: -- **A non-literal loader argument is only partly detected.** The detector matches - `import(` / `require(` / `require.resolve(` and then inspects only the **first - character** after the paren; anything beginning with `"`, `'` or a backtick is - treated as a static literal and skipped. So `` import(`stream${n}`) `` and - `import("zo" + n)` — a template literal or a concatenation that merely _starts_ - with a quote — read as analyzable while hiding their target. -- **`__require(…)` is not matched.** The pattern is anchored with `\b` before - `require`, and there is no word boundary inside `__require`, so rolldown's - emitted CJS-interop form escapes the unanalyzable-call check entirely. -- **String literals are not stripped before that text scan** (only comments are), - so import-shaped text inside a string — `const doc = "require(x)"` — can - false-positive and fail the gate on code that links nothing. +- **The loader-call scan is a tokenizer, not a parser.** `scanSource` classifies + every character as code / comment / string / template / regex, which closes the + wrong-verdict holes listed in the previous round (a first-character-only literal + test, unmatched `__require`, unstripped string and regex literals, and member + calls read as bare loaders — all now covered by paired tests). What remains: + regex-vs-division is decided from the previous significant token plus a keyword + list, so a regex directly after `)` — `if (x) /re/.test(s)` — is read as + division; a misread recovers at the next newline, so its blast radius is one + line. No JSX or TypeScript syntax is handled (the targets are built `.mjs` / + `.cjs`). And **indirect** loaders are beyond any text scan — aliasing `require` + to another name and calling that, `createRequire(…)`, + `Function("return import('x')")`, or `globalThis["im" + "port"]`. - **Workspace-sibling `dist` counts as first-party.** esbuild resolves pnpm symlinks to real paths, so `@copilotkit/core` enters the graph as `../core/dist/index.mjs`, with no `node_modules/` segment. Two consequences: diff --git a/packages/react-core/scripts/__tests__/assert-headless-purity.test.mjs b/packages/react-core/scripts/__tests__/assert-headless-purity.test.mjs index 03b6f93aae..ccedb3815b 100644 --- a/packages/react-core/scripts/__tests__/assert-headless-purity.test.mjs +++ b/packages/react-core/scripts/__tests__/assert-headless-purity.test.mjs @@ -27,8 +27,9 @@ import { forbiddenHits, isEntrypoint, isForbiddenPackage, + maskNonCode, packageNameFor, - stripComments, + scanSource, unanalyzableLoaderCalls, } from "../assert-headless-purity.mjs"; @@ -115,32 +116,228 @@ describe("forbiddenHits", () => { }); }); -describe("stripComments / unanalyzableLoaderCalls", () => { - it("blanks comments without disturbing string literals or line numbers", () => { +describe("maskNonCode / scanSource", () => { + it("blanks comments, strings, templates and regexes without moving an offset", () => { const code = [ 'const a = "// not a comment";', "// a comment", - "const b = 1;", + "const b = /re\\/gex/;", + "const c = 1;", ].join("\n"); - const stripped = stripComments(code); - assert.ok(stripped.includes('"// not a comment"')); - assert.ok(!stripped.includes("a comment\n")); - assert.equal(stripped.split("\n").length, code.split("\n").length); + const masked = maskNonCode(code); + assert.equal(masked.length, code.length, "offsets must still line up"); + assert.equal(masked.split("\n").length, code.split("\n").length); + // Code survives; every literal and comment is blanked. + assert.ok(masked.includes("const a = ")); + assert.ok(masked.includes("const c = 1;")); + assert.ok(!masked.includes("not a comment")); + assert.ok(!masked.includes("a comment")); + assert.ok(!masked.includes("gex")); }); - it("reports a loader call whose argument is not a string literal", () => { - const calls = unanalyzableLoaderCalls("export const f = (n) => import(n);"); - assert.equal(calls.length, 1); + it("consumes a `//` inside a string as part of the string, not as a comment", () => { + // If the comment branch won, `const b` would be blanked too. + const masked = maskNonCode('const a = "// x";\nconst b = 2;'); + assert.ok(masked.includes("const b = 2;")); }); - it("ignores literal loader calls and commented-out ones", () => { + it("consumes a `/*` inside a string, so no block comment is opened", () => { + // The Round-2 defeat of a sibling guard: a `/*` hidden in a literal made the + // stripper swallow everything to the next `*/`. + const masked = maskNonCode( + 'const a = "/* not a comment";\nconst b = 2;\nconst c = 3;', + ); + assert.ok(masked.includes("const b = 2;")); + assert.ok(masked.includes("const c = 3;")); + }); + + it("does not let a regex literal containing // blank the rest of its line", () => { + // The exact false NEGATIVE this detector shipped with: the `//` at the end of + // `/https:\/\//` read as a comment start and hid the real call after it. + const masked = maskNonCode( + "const re = /https:\\/\\//; const load = (n) => import(n);", + ); + assert.ok(masked.includes("import(")); + assert.ok(!masked.includes("https")); + }); + + it("keeps a template's ${…} interpolation as CODE while blanking its text", () => { + const { masked, literals } = scanSource("const t = `a ${import(n)} b`;"); + assert.ok(masked.includes("import(n)"), "the interpolation is real code"); + assert.ok(!masked.includes("a "), "the literal chunks are blanked"); + assert.equal(literals.length, 1); + assert.equal(literals[0].interpolated, true); + assert.equal(literals[0].terminated, true); + }); + + it("cannot be swallowed by an apostrophe in prose", () => { + const masked = maskNonCode( + "// it doesn't matter\nconst load = (n) => import(n);", + ); + assert.ok(masked.includes("import(n)")); + }); + + it("marks a literal the file never closes as unterminated", () => { + const { literals } = scanSource('const a = "never closed'); + assert.equal(literals.length, 1); + assert.equal(literals[0].terminated, false); + }); +}); + +// Both directions of the loader-call detector. Every case in the two blocks below +// was a WRONG VERDICT the pre-tokenizer version actually gave, reproduced against +// the real gate: it blanked comments but left string, template and regex literals +// intact (so import-shaped TEXT hard-failed CI), matched a bare `\brequire` / +// `\bimport` (so member calls hard-failed and rolldown's `__require` escaped), and +// classified an argument as static from its FIRST CHARACTER alone (so anything +// merely starting with a quote read as clean while hiding its target). +describe("unanalyzableLoaderCalls — must NOT report (false-positive direction)", () => { + it("import-shaped text inside a string literal", () => { assert.deepEqual( unanalyzableLoaderCalls( - ['import "streamdown";', "// const x = await import(name);"].join("\n"), + 'export function f() {\n throw new Error("use require(path) instead");\n}', ), [], ); }); + + it("import-shaped text inside a template literal", () => { + assert.deepEqual( + unanalyzableLoaderCalls("export const doc = (x) => `import(${x})`;"), + [], + ); + }); + + it("import-shaped text inside a regex literal", () => { + assert.deepEqual( + unanalyzableLoaderCalls("export const e = /import\\(x\\)/;"), + [], + ); + }); + + it("member calls, including one split across lines", () => { + assert.deepEqual( + unanalyzableLoaderCalls( + [ + "export const a = (o, y) => o.import(y);", + "export const b = (m, x) => m.require(x);", + "export const c = (m, x) => m?.require(x);", + "export const d = (m, x) => m", + " .import(x);", + ].join("\n"), + ), + [], + ); + }); + + it("an identifier that merely ends with a loader word", () => { + assert.deepEqual( + unanalyzableLoaderCalls("export const h = (x) => myrequire(x);"), + [], + ); + }); + + it("complete static literals, including a non-interpolated template and import attributes", () => { + assert.deepEqual( + unanalyzableLoaderCalls( + [ + 'import "streamdown";', + 'export const a = () => import("./m.mjs");', + "export const b = () => import(`./m.mjs`);", + 'export const c = () => require.resolve("./m.mjs");', + 'export const d = () => import( "./m.mjs" );', + 'export const e = () => import("./m.json", { with: { type: "json" } });', + "// const z = await import(name);", + "/* import(other) */", + ].join("\n"), + ), + [], + ); + }); +}); + +describe("unanalyzableLoaderCalls — MUST report (false-negative direction)", () => { + /** @param {string} code @param {RegExp} pattern */ + const reportsOne = (code, pattern) => { + const calls = unanalyzableLoaderCalls(code); + assert.equal( + calls.length, + 1, + `expected exactly one report, got ${calls.length}: ${JSON.stringify(calls)}`, + ); + assert.match(calls[0], pattern); + }; + + it("a plain variable argument", () => { + reportsOne("export const f = (n) => import(n);", /import\(n\)/); + }); + + it("an INTERPOLATED template argument (starts with a quote, hides its target)", () => { + reportsOne( + "export const load = (n) => import(`stream${n}`);", + /import\(`stream\$\{n\}`\)/, + ); + }); + + it("an interpolated path to the fat entry", () => { + reportsOne( + 'const base = ".";\nexport const load = () => import(`${base}/v2/index.mjs`);', + /v2\/index\.mjs/, + ); + }); + + it("a CONCATENATED argument (starts with a quote, hides its target)", () => { + reportsOne( + 'export const load = (n) => import("zo" + n);', + /import\("zo" \+ n\)/, + ); + }); + + it("rolldown's __require, which a \\brequire pattern cannot match", () => { + reportsOne( + "export const load = (name) => __require(name);", + /__require\(name\)/, + ); + }); + + it("require.resolve with a variable", () => { + reportsOne( + "export const load = (n) => require.resolve(n);", + /require\.resolve\(n\)/, + ); + }); + + it("a real dynamic call on the same line as a regex containing //", () => { + reportsOne( + "export const re = /https:\\/\\//; export const load = (n) => import(n);", + /import\(n\)/, + ); + }); + + it("a call whose parenthesis is never closed", () => { + reportsOne("export const load = (n) => import(n", /import\(n/); + }); + + it("a call whose argument sits inside an unterminated string", () => { + reportsOne( + 'export const load = () => import("./m.mjs', + /import\("\.\/m\.mjs/, + ); + }); + + it("reports each of several calls in one file", () => { + assert.equal( + unanalyzableLoaderCalls( + [ + "export const a = (n) => import(n);", + "export const b = (n) => __require(n);", + 'export const c = (n) => require("zo" + n);', + 'export const ok = () => import("./m.mjs");', + ].join("\n"), + ).length, + 3, + ); + }); }); describe("collectModuleGraph", () => { @@ -163,6 +360,27 @@ describe("collectModuleGraph", () => { ); }); + // Two of the four real targets are `.cjs`, which takes the script's + // `format: "cjs"` branch and carries the `require()` loader shape — and until + // these fixtures landed every fixture was `.mjs`, so that branch and that shape + // were asserted by nothing at all. + it('walks a CommonJS entry through the format:"cjs" branch', async () => { + const graph = await collectModuleGraph({ + entryFile: fixture("purity-entry.cjs"), + pkgRoot, + }); + const entryText = fs.readFileSync(fixture("purity-entry.cjs"), "utf8"); + assert.ok(!entryText.includes("zod")); + assert.ok( + graph.inputs.some((input) => input.endsWith("purity-chunk.cjs")), + "the relative require() edge must be in the graph", + ); + assert.ok( + graph.inputs.some((input) => /node_modules\/zod\//.test(input)), + "the dep behind that require() must be in the graph", + ); + }); + it("fails loudly on an edge it cannot resolve instead of reading as clean", async () => { await assert.rejects( collectModuleGraph({ @@ -224,6 +442,36 @@ describe("assertEntryPurity", () => { assert.equal(report.unanalyzable.length, 1); assert.match(report.unanalyzable[0], /purity-unanalyzable-entry\.mjs/); }); + + it("catches a forbidden dep behind a relative require() edge (cjs entry)", async () => { + const report = await assertEntryPurity({ + entryFile: fixture("purity-entry.cjs"), + pkgRoot, + forbidden: STAND_IN, + }); + assert.deepEqual(report.hits, [{ dep: "zod", via: ["zod"] }]); + assert.deepEqual(report.unanalyzable, []); + }); + + it("reports BOTH __require and a concatenated require() in a cjs entry, and neither counter-example", async () => { + const report = await assertEntryPurity({ + entryFile: fixture("purity-cjs-unanalyzable-entry.cjs"), + pkgRoot, + forbidden: REAL_FORBIDDEN, + }); + assert.deepEqual(report.hits, []); + // The static `require("./purity-chunk.cjs")` and the import-shaped text in + // `message` must contribute nothing; the two hidden loaders must both appear. + assert.equal( + report.unanalyzable.length, + 2, + `expected two reports, got ${JSON.stringify(report.unanalyzable)}`, + ); + assert.ok(report.unanalyzable.some((r) => /__require\(name\)/.test(r))); + assert.ok( + report.unanalyzable.some((r) => /require\("zo" \+ name\)/.test(r)), + ); + }); }); // The gate can only fail CI if its CLI block actually RUNS. The first version of diff --git a/packages/react-core/scripts/__tests__/fixtures/purity-chunk.cjs b/packages/react-core/scripts/__tests__/fixtures/purity-chunk.cjs new file mode 100644 index 0000000000..aa2495879c --- /dev/null +++ b/packages/react-core/scripts/__tests__/fixtures/purity-chunk.cjs @@ -0,0 +1,8 @@ +// The CommonJS half of the graph fixtures. Exists because every other fixture is +// `.mjs`, which left the script's `format: "cjs"` branch — and the `require()` +// loader shape it is meant to cover — completely unexercised. +// `zod` stands in for a forbidden dep (a real dependency of this package, so it +// resolves through node_modules exactly as shiki/streamdown would). +const { z } = require("zod"); + +module.exports = { schema: z.string() }; diff --git a/packages/react-core/scripts/__tests__/fixtures/purity-cjs-unanalyzable-entry.cjs b/packages/react-core/scripts/__tests__/fixtures/purity-cjs-unanalyzable-entry.cjs new file mode 100644 index 0000000000..a4b3626fa0 --- /dev/null +++ b/packages/react-core/scripts/__tests__/fixtures/purity-cjs-unanalyzable-entry.cjs @@ -0,0 +1,16 @@ +// CJS fail-loud fixture. Two shapes in one file, both of which the pre-tokenizer +// detector called clean: +// • `__require(name)` — rolldown's CJS-interop shim, which a `\brequire` +// word-boundary pattern cannot match (there is no boundary inside `__require`). +// • `require("zo" + name)` — a concatenation that merely STARTS with a quote, +// which the old "first character after the paren" test read as a static literal. +// The innocent counter-examples on either side must stay silent. +const staticOk = require("./purity-chunk.cjs"); +const message = "call require(path) with a literal"; + +module.exports = { + staticOk, + message, + rolldown: (name) => __require(name), + concatenated: (name) => require("zo" + name), +}; diff --git a/packages/react-core/scripts/__tests__/fixtures/purity-entry.cjs b/packages/react-core/scripts/__tests__/fixtures/purity-entry.cjs new file mode 100644 index 0000000000..a39a1d4de3 --- /dev/null +++ b/packages/react-core/scripts/__tests__/fixtures/purity-entry.cjs @@ -0,0 +1,4 @@ +// CJS mirror of purity-entry.mjs: an entry that names no dependency and reaches +// one only through a relative `require()` edge, bundled through the script's +// `format: "cjs"` branch. +module.exports = require("./purity-chunk.cjs"); diff --git a/packages/react-core/scripts/assert-headless-purity.mjs b/packages/react-core/scripts/assert-headless-purity.mjs index e5887e1a5a..39da531499 100644 --- a/packages/react-core/scripts/assert-headless-purity.mjs +++ b/packages/react-core/scripts/assert-headless-purity.mjs @@ -53,8 +53,12 @@ // entry hid behind a variable. The one place this script therefore reads text is // to find those calls in the graph's FIRST-PARTY files (our own and our workspace // siblings' built output — third-party dynamic requires are endemic and say -// nothing about #4893), and it strips comments before looking, so a documented -// counter-example cannot trip it. +// nothing about #4893). That scan runs over a TOKENIZED view of the file in which +// comments, strings, templates and regexes are blanked (see `scanSource` below), so +// a documented counter-example — or any other import-shaped text — cannot trip it, +// and a `//` inside a regex cannot hide a real call. See the block comment above +// `scanSource` for the wrong verdicts the earlier regex pair produced in BOTH +// directions, and for what the tokenizer still does not cover. // // esbuild is already this package's devDependency and already runs in the same CI // job (scripts/measure-copilotchat.mjs), and react/react-dom stay external there @@ -111,54 +115,427 @@ const GRAPH_BLINDING_WARNINGS = [ /could not be resolved/i, ]; +// ─── Why the text scan is a tokenizer and not an alternation of regexes ────── +// The previous version layered two regexes: one comment/string/template +// alternation to blank comments, and one `\b(?:import|require)\s*\(` to find +// loader calls. It gave WRONG VERDICTS IN BOTH DIRECTIONS, and every case below +// was reproduced against the real gate before this rewrite: +// +// false FAIL throw new Error("use require(path) instead") ← import-shaped TEXT +// false FAIL `import(${x})` ← inside a template +// false FAIL o.import(y) / mod.require(x) ← member calls, not loaders +// false PASS const re = /https:\/\//; …import(n) ← the regex's `//` blanked +// the rest of the line, +// hiding a real call +// false PASS import(`stream${n}`) ← "starts with a quote" +// false PASS import("zo" + n) ← "starts with a quote" +// false PASS __require(name) ← no \b inside `__require` +// +// The first four are one root cause: a regex cannot know whether the text it +// matched is CODE. So the scan now runs a real (small) tokenizer, `scanSource`, +// which walks the file once and classifies every character as code, comment, +// string, template or regex. Only ONE regex survives, and it is applied +// exclusively to the tokenizer's code-only output, so it can no longer see into a +// literal or a comment at all. +// +// What the tokenizer does NOT do, stated plainly so the next reader does not +// over-trust it (it is a masker, not a parser): +// • Regex-vs-division is a heuristic on the previous significant token +// (see REGEX_AFTER_KEYWORD): a regex directly after `)` — `if (x) /re/.test(s)` +// — is read as division. Regex mode bails at a newline, so the blast radius of +// a misread is the rest of that ONE line, never the file. +// • No JSX and no TypeScript syntax. The targets are built `.mjs` / `.cjs`. +// • Indirect loaders stay invisible to any text scan: `const r = require; r(x)`, +// `createRequire(...)`, `Function("return import('x')")`, `globalThis["im"+"port"]`. +// Those are out of reach without evaluating the module. + +/** Identifier characters, used for both boundary and keyword lookback. */ +const IDENT_CHAR = /[A-Za-z0-9_$]/; + /** - * Comment / string / template alternation, scanned left-to-right in ONE pass. - * - * Ported verbatim from the sibling guard in - * packages/react-native/src/__tests__/headless-entry-surface.test.ts, which hit - * the same class of bug. Matching strings with the SAME alternation is what makes - * comment stripping correct: a `//` inside a string literal is consumed as part of - * the string before the comment branch can see it, and a quote inside a comment is - * consumed as part of the comment. `'` and `"` deliberately do not cross a - * newline, so an unbalanced apostrophe in prose ("doesn't") cannot swallow the - * rest of a file. + * A `/` directly after one of these follows a VALUE, so it is division, not the + * start of a regex literal. (`)` is the deliberate over-approximation noted above.) */ -const COMMENT_OR_LITERAL = - /"(?:[^"\\\n]|\\.)*"|'(?:[^'\\\n]|\\.)*'|`(?:[^`\\]|\\.)*`|\/\/[^\n]*|\/\*[\s\S]*?\*\//g; - -/** Every way a module can pull another one in at runtime. */ -const LOADER_CALL = /\b(?:import|require(?:\.resolve)?)\s*\(\s*/g; +const VALUE_BEFORE_SLASH = new Set([")", "]", '"', "'", "`"]); /** - * Blanks out comments, preserving newlines (and therefore line numbers) and - * leaving real string/template literals untouched. + * Keywords a regex literal may legally follow. Needed because the character + * before the `/` in `return /re/.test(s)` is an identifier character, which would + * otherwise read as division. + */ +const REGEX_AFTER_KEYWORD = new Set([ + "return", + "typeof", + "instanceof", + "in", + "of", + "new", + "delete", + "void", + "throw", + "case", + "do", + "else", + "yield", + "await", +]); + +/** + * Whether the `/` whose previous significant code character sits at `lastIndex` + * starts a regex literal rather than being a division operator. + * + * @param {string} code + * @param {number} lastIndex - Offset of the last significant code character, or -1. + * @returns {boolean} + */ +function regexAllowedAfter(code, lastIndex) { + if (lastIndex < 0) return true; // start of file + const char = code[lastIndex]; + if (VALUE_BEFORE_SLASH.has(char)) return false; + // Operators, `(`, `,`, `;`, `{`, `}`, `:` — all positions where a value starts. + if (!IDENT_CHAR.test(char)) return true; + let start = lastIndex; + while (start >= 0 && IDENT_CHAR.test(code[start])) start -= 1; + return REGEX_AFTER_KEYWORD.has(code.slice(start + 1, lastIndex + 1)); +} + +/** + * Single-pass scanner that classifies every character of a JS source file. + * + * @param {string} code + * @returns {{ masked: string, literals: { start: number, end: number, interpolated: boolean, terminated: boolean }[] }} + * `masked` has the SAME LENGTH as `code`, with every comment, string, template + * and regex character replaced by a space and every newline preserved — so + * offsets and line numbers still line up with the original. `literals` holds one + * span per string/template literal in source order, `end` exclusive and + * including the closing quote; `interpolated` is true for a template containing + * `${…}`, and `terminated` is false for a literal the file never closes. + */ +export function scanSource(code) { + /** @type {[number, number][]} Sorted, non-overlapping ranges to blank. */ + const blanks = []; + const blank = (from, to) => { + const start = Math.max(0, from); + const end = Math.min(code.length, to); + if (end <= start) return; + const last = blanks[blanks.length - 1]; + if (last && last[1] === start) last[1] = end; + else blanks.push([start, end]); + }; + + /** @type {{ start: number, end: number, interpolated: boolean, terminated: boolean }[]} */ + const literals = []; + /** Enclosing templates we are inside via `${…}`, innermost last. */ + const templates = []; + + let mode = "code"; + let braceDepth = 0; + let literalStart = -1; + let interpolated = false; + // Offset of the last significant CODE character; drives regex-vs-division. + let lastCode = -1; + let i = 0; + + while (i < code.length) { + const char = code[i]; + + if (mode === "code") { + if (char === "/" && code[i + 1] === "/") { + blank(i, i + 2); + mode = "line-comment"; + i += 2; + } else if (char === "/" && code[i + 1] === "*") { + blank(i, i + 2); + mode = "block-comment"; + i += 2; + } else if (char === '"' || char === "'") { + mode = char === '"' ? "double" : "single"; + literalStart = i; + interpolated = false; + blank(i, i + 1); + i += 1; + } else if (char === "`") { + mode = "template"; + literalStart = i; + interpolated = false; + blank(i, i + 1); + i += 1; + } else if (char === "/" && regexAllowedAfter(code, lastCode)) { + mode = "regex"; + blank(i, i + 1); + i += 1; + } else if (char === "}" && braceDepth === 0 && templates.length) { + // The `}` closing a `${…}` interpolation: back into the template. + blank(i, i + 1); + const frame = templates.pop(); + braceDepth = frame.braceDepth; + literalStart = frame.literalStart; + interpolated = true; + mode = "template"; + i += 1; + } else { + if (char === "{") braceDepth += 1; + else if (char === "}") braceDepth = Math.max(0, braceDepth - 1); + if (!/\s/.test(char)) lastCode = i; + i += 1; + } + continue; + } + + if (mode === "line-comment") { + if (char === "\n") mode = "code"; + else blank(i, i + 1); + i += 1; + continue; + } + + if (mode === "block-comment") { + if (char === "*" && code[i + 1] === "/") { + blank(i, i + 2); + mode = "code"; + i += 2; + } else { + if (char !== "\n") blank(i, i + 1); + i += 1; + } + continue; + } + + if (mode === "single" || mode === "double") { + const quote = mode === "single" ? "'" : '"'; + if (char === "\\") { + blank(i, i + 2); + i += 2; + } else if (char === quote) { + blank(i, i + 1); + literals.push({ + start: literalStart, + end: i + 1, + interpolated: false, + terminated: true, + }); + lastCode = i; + mode = "code"; + i += 1; + } else if (char === "\n") { + // A quoted string cannot span a raw newline, so either the source is + // invalid or we mis-entered: end the span here rather than let one stray + // apostrophe swallow the rest of the file. + literals.push({ + start: literalStart, + end: i, + interpolated: false, + terminated: false, + }); + mode = "code"; + i += 1; + } else { + blank(i, i + 1); + i += 1; + } + continue; + } + + if (mode === "template") { + if (char === "\\") { + blank(i, i + 2); + i += 2; + } else if (char === "$" && code[i + 1] === "{") { + blank(i, i + 2); + templates.push({ literalStart, braceDepth }); + braceDepth = 0; + mode = "code"; + i += 2; + } else if (char === "`") { + blank(i, i + 1); + literals.push({ + start: literalStart, + end: i + 1, + interpolated, + terminated: true, + }); + lastCode = i; + mode = "code"; + i += 1; + } else { + if (char !== "\n") blank(i, i + 1); + i += 1; + } + continue; + } + + if (mode === "regex" || mode === "regex-class") { + if (char === "\\") { + blank(i, i + 2); + i += 2; + } else if (char === "\n") { + // A regex literal cannot span a newline, so we mis-read a division `/`. + // Recover at the line break: a misread can never reach past one line. + mode = "code"; + i += 1; + } else if (mode === "regex" && char === "[") { + blank(i, i + 1); + mode = "regex-class"; + i += 1; + } else if (mode === "regex-class" && char === "]") { + blank(i, i + 1); + mode = "regex"; + i += 1; + } else if (mode === "regex" && char === "/") { + blank(i, i + 1); + lastCode = i; + mode = "code"; + i += 1; + } else { + blank(i, i + 1); + i += 1; + } + continue; + } + + /* c8 ignore next -- unreachable: every mode above continues the loop. */ + i += 1; + } + + // An unterminated literal at EOF: record it so a loader argument inside it is + // classified as NOT a complete literal, i.e. reported rather than skipped. + if (mode === "single" || mode === "double" || mode === "template") { + literals.push({ + start: literalStart, + end: code.length, + interpolated, + terminated: false, + }); + } + + const parts = []; + let cursor = 0; + for (const [start, end] of blanks) { + parts.push(code.slice(cursor, start)); + parts.push(code.slice(start, end).replace(/[^\n]/g, " ")); + cursor = end; + } + parts.push(code.slice(cursor)); + return { masked: parts.join(""), literals }; +} + +/** + * Blanks comments, string/template literals AND regex literals, preserving both + * length and newlines (and therefore offsets and line numbers). + * + * Named for what it does: the previous `stripComments` blanked comments only and + * left literals intact, which is precisely how import-shaped TEXT reached the + * loader-call matcher and hard-failed CI on code that links nothing. * * @param {string} code * @returns {string} */ -export function stripComments(code) { - return code.replace(COMMENT_OR_LITERAL, (match) => - match.startsWith("//") || match.startsWith("/*") - ? match.replace(/[^\n]/g, " ") - : match, +export function maskNonCode(code) { + return scanSource(code).masked; +} + +/** + * Every way a module can pull another one in at runtime, including rolldown's + * `__require` CJS-interop shim (which `\brequire` misses: there is no word + * boundary inside `__require`). + * + * The lookbehind rejects an identifier that merely ENDS with one of these words + * (`myrequire(x)`) and the common `o.import(` member form; `precededByMemberDot` + * then covers the same member call split across lines, which a single-character + * lookbehind cannot see. + * + * Only ever applied to `scanSource`'s masked output, never to raw source. + */ +const LOADER_CALL = + /(?= 0 && /\s/.test(masked[i])) i -= 1; + return i >= 0 && masked[i] === "." && masked[i - 1] !== "."; +} + +/** + * Offsets of a loader call's FIRST argument, given the offset of its `(`, or null + * when the call is never closed. Depth-aware over the MASKED source, so a paren or + * comma inside a string cannot end it. Stopping at the first top-level comma keeps + * `import("./m", { with: { type: "json" } })` a one-literal argument. + * + * @param {string} masked + * @param {number} openIndex + * @returns {{ start: number, end: number } | null} + */ +function firstArgumentRange(masked, openIndex) { + let depth = 0; + for (let i = openIndex; i < masked.length; i += 1) { + const char = masked[i]; + if (char === "(" || char === "[" || char === "{") depth += 1; + else if (char === ")" || char === "]" || char === "}") { + depth -= 1; + if (depth === 0) return { start: openIndex + 1, end: i }; + } else if (char === "," && depth === 1) { + return { start: openIndex + 1, end: i }; + } + } + return null; +} + +/** + * Whether a loader argument is a COMPLETE single literal — the only shape a + * bundler can resolve. "Starts with a quote" is not enough, and that was the whole + * false-negative class: `import("zo" + n)` and `` import(`stream${n}`) `` both + * start with one and both hide their target. + * + * @param {object} options + * @param {string} options.code - The original source (for the whitespace check). + * @param {{ start: number, end: number, interpolated: boolean, terminated: boolean }[]} options.literals + * @param {{ start: number, end: number } | null} options.range + * @returns {boolean} + */ +function isCompleteLiteralArgument({ code, literals, range }) { + if (!range) return false; + const inside = literals.filter( + (literal) => literal.start >= range.start && literal.end <= range.end, + ); + if (inside.length !== 1) return false; + const [literal] = inside; + if (!literal.terminated || literal.interpolated) return false; + // Nothing but whitespace may surround it: that is what rejects a concatenation. + return ( + code.slice(range.start, literal.start).trim() === "" && + code.slice(literal.end, range.end).trim() === "" ); } /** - * Loader calls whose argument is not a string literal, and which therefore hide - * whatever they load from any static analysis — including a bundler's. + * Loader calls whose argument is not a complete string literal, and which + * therefore hide whatever they load from any static analysis — including a + * bundler's. * * @param {string} code * @returns {string[]} A short excerpt per unanalyzable call. */ export function unanalyzableLoaderCalls(code) { - const stripped = stripComments(code); + const { masked, literals } = scanSource(code); const found = []; - for (const match of stripped.matchAll(LOADER_CALL)) { - const after = stripped.slice(match.index + match[0].length); - if (/^["'`]/.test(after)) continue; + for (const match of masked.matchAll(LOADER_CALL)) { + if (precededByMemberDot(masked, match.index)) continue; + const openIndex = match.index + match[0].length - 1; + const range = firstArgumentRange(masked, openIndex); + if (isCompleteLiteralArgument({ code, literals, range })) continue; found.push( - stripped + // Excerpt from the ORIGINAL source: the masked form would print the + // argument as blanks, which tells a reader nothing. + code .slice(match.index, match.index + match[0].length + 48) .replace(/\s+/g, " ") .trim(),