mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
fix(scripts): tokenize before hunting loader calls in the purity gate
The #4893 hard-fail gate's loader-call detector gave WRONG VERDICTS IN BOTH DIRECTIONS. It layered two regexes — a comment/string/template alternation that blanked only the comment branch, and `\b(?:import|require(?:\.resolve)?)\s*\(` over the result — then classified an argument as static from the FIRST CHARACTER after the paren. All nine shapes below were reproduced against the real gate before the rewrite: false FAIL throw new Error("use require(path) instead") false FAIL `import(${x})` inside a template false FAIL o.import(y) / mod.require(x) (member calls, not loaders) false PASS /https:\/\//; …import(n) (the regex's `//` blanked the rest of the line, hiding a real dynamic call) false PASS import(`stream${n}`) (merely STARTS with a quote) false PASS import("zo" + n) (same) false PASS import(`${base}/v2/index.mjs`) (same — the fat entry) false PASS __require(name) (no \b inside `__require`) Replaced with `scanSource`, a single-pass tokenizer that classifies every character as code / comment / string / template / regex and returns a length-preserving masked view plus a literal-span list. The one surviving regex now only ever sees code, so import-shaped TEXT cannot reach it at all; an argument counts as static only when it is one COMPLETE literal with no concatenation or interpolation; `__require` is matched; and a member call is rejected both by lookbehind and by a whitespace-skipping back-scan (so `m\n .import(x)` is not a loader either). Proven in both directions: nine innocent/violation pairs run through the real `assertEntryPurity`, each innocent form CLEAN and each matching real violation FAIL. Re-proved end-to-end by prepending `import "streamdown"` to the real dist/v2/headless.mjs — exit 1 naming all five families — then restoring it byte-identically. On the untouched dist the scan sees 66 loader calls in the `.cjs` graph and classifies all 66 static, so it passes because it LOOKED. Also adds the first `.cjs` fixtures: every existing fixture was `.mjs`, leaving the script's `format: "cjs"` branch and the `require()` shape asserted by nothing. Tests 24 → 47. `stripComments` is renamed `maskNonCode`, since it now blanks literals and regexes too; it had no caller outside this script and its test. The RN guard keeps its own copy, untouched. dev-docs/bundle-size.md: the four holes a sibling agent documented as known limitations this round are closed and removed from that list; what genuinely remains (regex-vs-division heuristic, no JSX/TS, indirect loaders) replaces them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Maxim
+31
-24
@@ -91,7 +91,6 @@ baseline to maintain, and no conflict with the Phase 2 freeze on `limit` fields:
|
||||
Both entries are guarded because `@copilotkit/react-native` imports both. Runs
|
||||
in `static_bundle_size.yml` — the step there is named after `/v2/headless`
|
||||
only, but the script asserts `/v2/context` as well. Mechanically:
|
||||
|
||||
- It bundles each entry with **esbuild** (`bundle: true`, `write: false`,
|
||||
`metafile: true`; `react` / `react-dom` and the JSX runtimes external;
|
||||
CSS and font assets on the `empty` loader, which still records them as graph
|
||||
@@ -117,18 +116,25 @@ baseline to maintain, and no conflict with the Phase 2 freeze on `limit` fields:
|
||||
warnings print but are non-fatal — third-party code warns for reasons that
|
||||
say nothing about #4893.
|
||||
- The one place it still reads **text** is to find `import(…)` / `require(…)` /
|
||||
`require.resolve(…)` calls whose argument is not a string literal — the one
|
||||
edge shape a bundler genuinely cannot see through — and only in the graph's
|
||||
first-party files. Comments are stripped first (a single
|
||||
comment/string/template alternation, so a `//` inside a string and a quote
|
||||
inside a comment are each consumed by the other branch), which is why a
|
||||
documented counter-example naming a banned dep no longer trips it.
|
||||
`require.resolve(…)` / `__require(…)` calls whose argument is not a **complete**
|
||||
string literal — the one edge shape a bundler genuinely cannot see through —
|
||||
and only in the graph's first-party files. That scan runs over the output of
|
||||
`scanSource`, a small single-pass **tokenizer** that blanks comments, strings,
|
||||
templates _and_ regex literals while preserving offsets, so the one surviving
|
||||
regex only ever sees code. A documented counter-example naming a banned dep
|
||||
cannot trip it, a `//` inside a regex cannot hide a real call, and an argument
|
||||
counts as static only when it is one whole literal with no concatenation or
|
||||
interpolation.
|
||||
- Negative tests: `packages/react-core/scripts/__tests__/assert-headless-purity.test.mjs`,
|
||||
run by `pnpm --filter @copilotkit/react-core test:scripts` (chained from that
|
||||
package's `test`). They cover both directions — a forbidden dep reached only
|
||||
through a relative chunk edge, a forbidden dep left external, an unresolvable
|
||||
edge, an unanalyzable loader call, and banned tokens present only in comments
|
||||
and strings, which must **pass**.
|
||||
through a relative chunk edge (in both an `.mjs` and a `.cjs` entry, so the
|
||||
`format: "cjs"` branch and the `require()` shape are exercised too), a
|
||||
forbidden dep left external, an unresolvable edge, an unanalyzable loader call,
|
||||
and banned tokens present only in comments and strings, which must **pass**.
|
||||
Each detector shape fixed in the tokenizer rewrite has a **pair**: the innocent
|
||||
form must pass and the matching real violation must fail.
|
||||
|
||||
2. `packages/react-native/src/__tests__/headless-entry-surface.test.ts` — walks
|
||||
the relative-import graph of this package's own `src/`, from both
|
||||
`src/headless.ts` and `src/index.ts`, and fails if a reached module imports a
|
||||
@@ -136,8 +142,9 @@ baseline to maintain, and no conflict with the Phase 2 freeze on `limit` fields:
|
||||
render stack directly, or (headless entry only) pulls the optional native
|
||||
chat/attachment peer deps. It extracts static `import`/`export … from`, bare
|
||||
side-effect `import "x"`, `import()` and `require()`/`require.resolve()` —
|
||||
Metro follows the lazy forms too — strips comments with the same
|
||||
comment/string/template alternation the purity gate uses, reports a
|
||||
Metro follows the lazy forms too — strips comments with its own
|
||||
comment/string/template alternation (the purity gate has since moved to the
|
||||
tokenizer described above), reports a
|
||||
non-literal loader argument as unanalyzable rather than ignoring it, and fails
|
||||
loudly on a local edge it cannot resolve. Runs in the normal test job.
|
||||
|
||||
@@ -155,18 +162,18 @@ is closed: react-core's own build leaves `@copilotkit/core`,
|
||||
assertion, not a complete one. Documented rather than glossed, because a doc that
|
||||
claims a gate is airtight is how the last round of this went wrong:
|
||||
|
||||
- **A non-literal loader argument is only partly detected.** The detector matches
|
||||
`import(` / `require(` / `require.resolve(` and then inspects only the **first
|
||||
character** after the paren; anything beginning with `"`, `'` or a backtick is
|
||||
treated as a static literal and skipped. So `` import(`stream${n}`) `` and
|
||||
`import("zo" + n)` — a template literal or a concatenation that merely _starts_
|
||||
with a quote — read as analyzable while hiding their target.
|
||||
- **`__require(…)` is not matched.** The pattern is anchored with `\b` before
|
||||
`require`, and there is no word boundary inside `__require`, so rolldown's
|
||||
emitted CJS-interop form escapes the unanalyzable-call check entirely.
|
||||
- **String literals are not stripped before that text scan** (only comments are),
|
||||
so import-shaped text inside a string — `const doc = "require(x)"` — can
|
||||
false-positive and fail the gate on code that links nothing.
|
||||
- **The loader-call scan is a tokenizer, not a parser.** `scanSource` classifies
|
||||
every character as code / comment / string / template / regex, which closes the
|
||||
wrong-verdict holes listed in the previous round (a first-character-only literal
|
||||
test, unmatched `__require`, unstripped string and regex literals, and member
|
||||
calls read as bare loaders — all now covered by paired tests). What remains:
|
||||
regex-vs-division is decided from the previous significant token plus a keyword
|
||||
list, so a regex directly after `)` — `if (x) /re/.test(s)` — is read as
|
||||
division; a misread recovers at the next newline, so its blast radius is one
|
||||
line. No JSX or TypeScript syntax is handled (the targets are built `.mjs` /
|
||||
`.cjs`). And **indirect** loaders are beyond any text scan — aliasing `require`
|
||||
to another name and calling that, `createRequire(…)`,
|
||||
`Function("return import('x')")`, or `globalThis["im" + "port"]`.
|
||||
- **Workspace-sibling `dist` counts as first-party.** esbuild resolves pnpm
|
||||
symlinks to real paths, so `@copilotkit/core` enters the graph as
|
||||
`../core/dist/index.mjs`, with no `node_modules/` segment. Two consequences:
|
||||
|
||||
@@ -27,8 +27,9 @@ import {
|
||||
forbiddenHits,
|
||||
isEntrypoint,
|
||||
isForbiddenPackage,
|
||||
maskNonCode,
|
||||
packageNameFor,
|
||||
stripComments,
|
||||
scanSource,
|
||||
unanalyzableLoaderCalls,
|
||||
} from "../assert-headless-purity.mjs";
|
||||
|
||||
@@ -115,32 +116,228 @@ describe("forbiddenHits", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("stripComments / unanalyzableLoaderCalls", () => {
|
||||
it("blanks comments without disturbing string literals or line numbers", () => {
|
||||
describe("maskNonCode / scanSource", () => {
|
||||
it("blanks comments, strings, templates and regexes without moving an offset", () => {
|
||||
const code = [
|
||||
'const a = "// not a comment";',
|
||||
"// a comment",
|
||||
"const b = 1;",
|
||||
"const b = /re\\/gex/;",
|
||||
"const c = 1;",
|
||||
].join("\n");
|
||||
const stripped = stripComments(code);
|
||||
assert.ok(stripped.includes('"// not a comment"'));
|
||||
assert.ok(!stripped.includes("a comment\n"));
|
||||
assert.equal(stripped.split("\n").length, code.split("\n").length);
|
||||
const masked = maskNonCode(code);
|
||||
assert.equal(masked.length, code.length, "offsets must still line up");
|
||||
assert.equal(masked.split("\n").length, code.split("\n").length);
|
||||
// Code survives; every literal and comment is blanked.
|
||||
assert.ok(masked.includes("const a = "));
|
||||
assert.ok(masked.includes("const c = 1;"));
|
||||
assert.ok(!masked.includes("not a comment"));
|
||||
assert.ok(!masked.includes("a comment"));
|
||||
assert.ok(!masked.includes("gex"));
|
||||
});
|
||||
|
||||
it("reports a loader call whose argument is not a string literal", () => {
|
||||
const calls = unanalyzableLoaderCalls("export const f = (n) => import(n);");
|
||||
assert.equal(calls.length, 1);
|
||||
it("consumes a `//` inside a string as part of the string, not as a comment", () => {
|
||||
// If the comment branch won, `const b` would be blanked too.
|
||||
const masked = maskNonCode('const a = "// x";\nconst b = 2;');
|
||||
assert.ok(masked.includes("const b = 2;"));
|
||||
});
|
||||
|
||||
it("ignores literal loader calls and commented-out ones", () => {
|
||||
it("consumes a `/*` inside a string, so no block comment is opened", () => {
|
||||
// The Round-2 defeat of a sibling guard: a `/*` hidden in a literal made the
|
||||
// stripper swallow everything to the next `*/`.
|
||||
const masked = maskNonCode(
|
||||
'const a = "/* not a comment";\nconst b = 2;\nconst c = 3;',
|
||||
);
|
||||
assert.ok(masked.includes("const b = 2;"));
|
||||
assert.ok(masked.includes("const c = 3;"));
|
||||
});
|
||||
|
||||
it("does not let a regex literal containing // blank the rest of its line", () => {
|
||||
// The exact false NEGATIVE this detector shipped with: the `//` at the end of
|
||||
// `/https:\/\//` read as a comment start and hid the real call after it.
|
||||
const masked = maskNonCode(
|
||||
"const re = /https:\\/\\//; const load = (n) => import(n);",
|
||||
);
|
||||
assert.ok(masked.includes("import("));
|
||||
assert.ok(!masked.includes("https"));
|
||||
});
|
||||
|
||||
it("keeps a template's ${…} interpolation as CODE while blanking its text", () => {
|
||||
const { masked, literals } = scanSource("const t = `a ${import(n)} b`;");
|
||||
assert.ok(masked.includes("import(n)"), "the interpolation is real code");
|
||||
assert.ok(!masked.includes("a "), "the literal chunks are blanked");
|
||||
assert.equal(literals.length, 1);
|
||||
assert.equal(literals[0].interpolated, true);
|
||||
assert.equal(literals[0].terminated, true);
|
||||
});
|
||||
|
||||
it("cannot be swallowed by an apostrophe in prose", () => {
|
||||
const masked = maskNonCode(
|
||||
"// it doesn't matter\nconst load = (n) => import(n);",
|
||||
);
|
||||
assert.ok(masked.includes("import(n)"));
|
||||
});
|
||||
|
||||
it("marks a literal the file never closes as unterminated", () => {
|
||||
const { literals } = scanSource('const a = "never closed');
|
||||
assert.equal(literals.length, 1);
|
||||
assert.equal(literals[0].terminated, false);
|
||||
});
|
||||
});
|
||||
|
||||
// Both directions of the loader-call detector. Every case in the two blocks below
|
||||
// was a WRONG VERDICT the pre-tokenizer version actually gave, reproduced against
|
||||
// the real gate: it blanked comments but left string, template and regex literals
|
||||
// intact (so import-shaped TEXT hard-failed CI), matched a bare `\brequire` /
|
||||
// `\bimport` (so member calls hard-failed and rolldown's `__require` escaped), and
|
||||
// classified an argument as static from its FIRST CHARACTER alone (so anything
|
||||
// merely starting with a quote read as clean while hiding its target).
|
||||
describe("unanalyzableLoaderCalls — must NOT report (false-positive direction)", () => {
|
||||
it("import-shaped text inside a string literal", () => {
|
||||
assert.deepEqual(
|
||||
unanalyzableLoaderCalls(
|
||||
['import "streamdown";', "// const x = await import(name);"].join("\n"),
|
||||
'export function f() {\n throw new Error("use require(path) instead");\n}',
|
||||
),
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
it("import-shaped text inside a template literal", () => {
|
||||
assert.deepEqual(
|
||||
unanalyzableLoaderCalls("export const doc = (x) => `import(${x})`;"),
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
it("import-shaped text inside a regex literal", () => {
|
||||
assert.deepEqual(
|
||||
unanalyzableLoaderCalls("export const e = /import\\(x\\)/;"),
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
it("member calls, including one split across lines", () => {
|
||||
assert.deepEqual(
|
||||
unanalyzableLoaderCalls(
|
||||
[
|
||||
"export const a = (o, y) => o.import(y);",
|
||||
"export const b = (m, x) => m.require(x);",
|
||||
"export const c = (m, x) => m?.require(x);",
|
||||
"export const d = (m, x) => m",
|
||||
" .import(x);",
|
||||
].join("\n"),
|
||||
),
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
it("an identifier that merely ends with a loader word", () => {
|
||||
assert.deepEqual(
|
||||
unanalyzableLoaderCalls("export const h = (x) => myrequire(x);"),
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
it("complete static literals, including a non-interpolated template and import attributes", () => {
|
||||
assert.deepEqual(
|
||||
unanalyzableLoaderCalls(
|
||||
[
|
||||
'import "streamdown";',
|
||||
'export const a = () => import("./m.mjs");',
|
||||
"export const b = () => import(`./m.mjs`);",
|
||||
'export const c = () => require.resolve("./m.mjs");',
|
||||
'export const d = () => import( "./m.mjs" );',
|
||||
'export const e = () => import("./m.json", { with: { type: "json" } });',
|
||||
"// const z = await import(name);",
|
||||
"/* import(other) */",
|
||||
].join("\n"),
|
||||
),
|
||||
[],
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("unanalyzableLoaderCalls — MUST report (false-negative direction)", () => {
|
||||
/** @param {string} code @param {RegExp} pattern */
|
||||
const reportsOne = (code, pattern) => {
|
||||
const calls = unanalyzableLoaderCalls(code);
|
||||
assert.equal(
|
||||
calls.length,
|
||||
1,
|
||||
`expected exactly one report, got ${calls.length}: ${JSON.stringify(calls)}`,
|
||||
);
|
||||
assert.match(calls[0], pattern);
|
||||
};
|
||||
|
||||
it("a plain variable argument", () => {
|
||||
reportsOne("export const f = (n) => import(n);", /import\(n\)/);
|
||||
});
|
||||
|
||||
it("an INTERPOLATED template argument (starts with a quote, hides its target)", () => {
|
||||
reportsOne(
|
||||
"export const load = (n) => import(`stream${n}`);",
|
||||
/import\(`stream\$\{n\}`\)/,
|
||||
);
|
||||
});
|
||||
|
||||
it("an interpolated path to the fat entry", () => {
|
||||
reportsOne(
|
||||
'const base = ".";\nexport const load = () => import(`${base}/v2/index.mjs`);',
|
||||
/v2\/index\.mjs/,
|
||||
);
|
||||
});
|
||||
|
||||
it("a CONCATENATED argument (starts with a quote, hides its target)", () => {
|
||||
reportsOne(
|
||||
'export const load = (n) => import("zo" + n);',
|
||||
/import\("zo" \+ n\)/,
|
||||
);
|
||||
});
|
||||
|
||||
it("rolldown's __require, which a \\brequire pattern cannot match", () => {
|
||||
reportsOne(
|
||||
"export const load = (name) => __require(name);",
|
||||
/__require\(name\)/,
|
||||
);
|
||||
});
|
||||
|
||||
it("require.resolve with a variable", () => {
|
||||
reportsOne(
|
||||
"export const load = (n) => require.resolve(n);",
|
||||
/require\.resolve\(n\)/,
|
||||
);
|
||||
});
|
||||
|
||||
it("a real dynamic call on the same line as a regex containing //", () => {
|
||||
reportsOne(
|
||||
"export const re = /https:\\/\\//; export const load = (n) => import(n);",
|
||||
/import\(n\)/,
|
||||
);
|
||||
});
|
||||
|
||||
it("a call whose parenthesis is never closed", () => {
|
||||
reportsOne("export const load = (n) => import(n", /import\(n/);
|
||||
});
|
||||
|
||||
it("a call whose argument sits inside an unterminated string", () => {
|
||||
reportsOne(
|
||||
'export const load = () => import("./m.mjs',
|
||||
/import\("\.\/m\.mjs/,
|
||||
);
|
||||
});
|
||||
|
||||
it("reports each of several calls in one file", () => {
|
||||
assert.equal(
|
||||
unanalyzableLoaderCalls(
|
||||
[
|
||||
"export const a = (n) => import(n);",
|
||||
"export const b = (n) => __require(n);",
|
||||
'export const c = (n) => require("zo" + n);',
|
||||
'export const ok = () => import("./m.mjs");',
|
||||
].join("\n"),
|
||||
).length,
|
||||
3,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("collectModuleGraph", () => {
|
||||
@@ -163,6 +360,27 @@ describe("collectModuleGraph", () => {
|
||||
);
|
||||
});
|
||||
|
||||
// Two of the four real targets are `.cjs`, which takes the script's
|
||||
// `format: "cjs"` branch and carries the `require()` loader shape — and until
|
||||
// these fixtures landed every fixture was `.mjs`, so that branch and that shape
|
||||
// were asserted by nothing at all.
|
||||
it('walks a CommonJS entry through the format:"cjs" branch', async () => {
|
||||
const graph = await collectModuleGraph({
|
||||
entryFile: fixture("purity-entry.cjs"),
|
||||
pkgRoot,
|
||||
});
|
||||
const entryText = fs.readFileSync(fixture("purity-entry.cjs"), "utf8");
|
||||
assert.ok(!entryText.includes("zod"));
|
||||
assert.ok(
|
||||
graph.inputs.some((input) => input.endsWith("purity-chunk.cjs")),
|
||||
"the relative require() edge must be in the graph",
|
||||
);
|
||||
assert.ok(
|
||||
graph.inputs.some((input) => /node_modules\/zod\//.test(input)),
|
||||
"the dep behind that require() must be in the graph",
|
||||
);
|
||||
});
|
||||
|
||||
it("fails loudly on an edge it cannot resolve instead of reading as clean", async () => {
|
||||
await assert.rejects(
|
||||
collectModuleGraph({
|
||||
@@ -224,6 +442,36 @@ describe("assertEntryPurity", () => {
|
||||
assert.equal(report.unanalyzable.length, 1);
|
||||
assert.match(report.unanalyzable[0], /purity-unanalyzable-entry\.mjs/);
|
||||
});
|
||||
|
||||
it("catches a forbidden dep behind a relative require() edge (cjs entry)", async () => {
|
||||
const report = await assertEntryPurity({
|
||||
entryFile: fixture("purity-entry.cjs"),
|
||||
pkgRoot,
|
||||
forbidden: STAND_IN,
|
||||
});
|
||||
assert.deepEqual(report.hits, [{ dep: "zod", via: ["zod"] }]);
|
||||
assert.deepEqual(report.unanalyzable, []);
|
||||
});
|
||||
|
||||
it("reports BOTH __require and a concatenated require() in a cjs entry, and neither counter-example", async () => {
|
||||
const report = await assertEntryPurity({
|
||||
entryFile: fixture("purity-cjs-unanalyzable-entry.cjs"),
|
||||
pkgRoot,
|
||||
forbidden: REAL_FORBIDDEN,
|
||||
});
|
||||
assert.deepEqual(report.hits, []);
|
||||
// The static `require("./purity-chunk.cjs")` and the import-shaped text in
|
||||
// `message` must contribute nothing; the two hidden loaders must both appear.
|
||||
assert.equal(
|
||||
report.unanalyzable.length,
|
||||
2,
|
||||
`expected two reports, got ${JSON.stringify(report.unanalyzable)}`,
|
||||
);
|
||||
assert.ok(report.unanalyzable.some((r) => /__require\(name\)/.test(r)));
|
||||
assert.ok(
|
||||
report.unanalyzable.some((r) => /require\("zo" \+ name\)/.test(r)),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// The gate can only fail CI if its CLI block actually RUNS. The first version of
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
// The CommonJS half of the graph fixtures. Exists because every other fixture is
|
||||
// `.mjs`, which left the script's `format: "cjs"` branch — and the `require()`
|
||||
// loader shape it is meant to cover — completely unexercised.
|
||||
// `zod` stands in for a forbidden dep (a real dependency of this package, so it
|
||||
// resolves through node_modules exactly as shiki/streamdown would).
|
||||
const { z } = require("zod");
|
||||
|
||||
module.exports = { schema: z.string() };
|
||||
@@ -0,0 +1,16 @@
|
||||
// CJS fail-loud fixture. Two shapes in one file, both of which the pre-tokenizer
|
||||
// detector called clean:
|
||||
// • `__require(name)` — rolldown's CJS-interop shim, which a `\brequire`
|
||||
// word-boundary pattern cannot match (there is no boundary inside `__require`).
|
||||
// • `require("zo" + name)` — a concatenation that merely STARTS with a quote,
|
||||
// which the old "first character after the paren" test read as a static literal.
|
||||
// The innocent counter-examples on either side must stay silent.
|
||||
const staticOk = require("./purity-chunk.cjs");
|
||||
const message = "call require(path) with a literal";
|
||||
|
||||
module.exports = {
|
||||
staticOk,
|
||||
message,
|
||||
rolldown: (name) => __require(name),
|
||||
concatenated: (name) => require("zo" + name),
|
||||
};
|
||||
@@ -0,0 +1,4 @@
|
||||
// CJS mirror of purity-entry.mjs: an entry that names no dependency and reaches
|
||||
// one only through a relative `require()` edge, bundled through the script's
|
||||
// `format: "cjs"` branch.
|
||||
module.exports = require("./purity-chunk.cjs");
|
||||
@@ -53,8 +53,12 @@
|
||||
// entry hid behind a variable. The one place this script therefore reads text is
|
||||
// to find those calls in the graph's FIRST-PARTY files (our own and our workspace
|
||||
// siblings' built output — third-party dynamic requires are endemic and say
|
||||
// nothing about #4893), and it strips comments before looking, so a documented
|
||||
// counter-example cannot trip it.
|
||||
// nothing about #4893). That scan runs over a TOKENIZED view of the file in which
|
||||
// comments, strings, templates and regexes are blanked (see `scanSource` below), so
|
||||
// a documented counter-example — or any other import-shaped text — cannot trip it,
|
||||
// and a `//` inside a regex cannot hide a real call. See the block comment above
|
||||
// `scanSource` for the wrong verdicts the earlier regex pair produced in BOTH
|
||||
// directions, and for what the tokenizer still does not cover.
|
||||
//
|
||||
// esbuild is already this package's devDependency and already runs in the same CI
|
||||
// job (scripts/measure-copilotchat.mjs), and react/react-dom stay external there
|
||||
@@ -111,54 +115,427 @@ const GRAPH_BLINDING_WARNINGS = [
|
||||
/could not be resolved/i,
|
||||
];
|
||||
|
||||
// ─── Why the text scan is a tokenizer and not an alternation of regexes ──────
|
||||
// The previous version layered two regexes: one comment/string/template
|
||||
// alternation to blank comments, and one `\b(?:import|require)\s*\(` to find
|
||||
// loader calls. It gave WRONG VERDICTS IN BOTH DIRECTIONS, and every case below
|
||||
// was reproduced against the real gate before this rewrite:
|
||||
//
|
||||
// false FAIL throw new Error("use require(path) instead") ← import-shaped TEXT
|
||||
// false FAIL `import(${x})` ← inside a template
|
||||
// false FAIL o.import(y) / mod.require(x) ← member calls, not loaders
|
||||
// false PASS const re = /https:\/\//; …import(n) ← the regex's `//` blanked
|
||||
// the rest of the line,
|
||||
// hiding a real call
|
||||
// false PASS import(`stream${n}`) ← "starts with a quote"
|
||||
// false PASS import("zo" + n) ← "starts with a quote"
|
||||
// false PASS __require(name) ← no \b inside `__require`
|
||||
//
|
||||
// The first four are one root cause: a regex cannot know whether the text it
|
||||
// matched is CODE. So the scan now runs a real (small) tokenizer, `scanSource`,
|
||||
// which walks the file once and classifies every character as code, comment,
|
||||
// string, template or regex. Only ONE regex survives, and it is applied
|
||||
// exclusively to the tokenizer's code-only output, so it can no longer see into a
|
||||
// literal or a comment at all.
|
||||
//
|
||||
// What the tokenizer does NOT do, stated plainly so the next reader does not
|
||||
// over-trust it (it is a masker, not a parser):
|
||||
// • Regex-vs-division is a heuristic on the previous significant token
|
||||
// (see REGEX_AFTER_KEYWORD): a regex directly after `)` — `if (x) /re/.test(s)`
|
||||
// — is read as division. Regex mode bails at a newline, so the blast radius of
|
||||
// a misread is the rest of that ONE line, never the file.
|
||||
// • No JSX and no TypeScript syntax. The targets are built `.mjs` / `.cjs`.
|
||||
// • Indirect loaders stay invisible to any text scan: `const r = require; r(x)`,
|
||||
// `createRequire(...)`, `Function("return import('x')")`, `globalThis["im"+"port"]`.
|
||||
// Those are out of reach without evaluating the module.
|
||||
|
||||
/** Identifier characters, used for both boundary and keyword lookback. */
|
||||
const IDENT_CHAR = /[A-Za-z0-9_$]/;
|
||||
|
||||
/**
|
||||
* Comment / string / template alternation, scanned left-to-right in ONE pass.
|
||||
*
|
||||
* Ported verbatim from the sibling guard in
|
||||
* packages/react-native/src/__tests__/headless-entry-surface.test.ts, which hit
|
||||
* the same class of bug. Matching strings with the SAME alternation is what makes
|
||||
* comment stripping correct: a `//` inside a string literal is consumed as part of
|
||||
* the string before the comment branch can see it, and a quote inside a comment is
|
||||
* consumed as part of the comment. `'` and `"` deliberately do not cross a
|
||||
* newline, so an unbalanced apostrophe in prose ("doesn't") cannot swallow the
|
||||
* rest of a file.
|
||||
* A `/` directly after one of these follows a VALUE, so it is division, not the
|
||||
* start of a regex literal. (`)` is the deliberate over-approximation noted above.)
|
||||
*/
|
||||
const COMMENT_OR_LITERAL =
|
||||
/"(?:[^"\\\n]|\\.)*"|'(?:[^'\\\n]|\\.)*'|`(?:[^`\\]|\\.)*`|\/\/[^\n]*|\/\*[\s\S]*?\*\//g;
|
||||
|
||||
/** Every way a module can pull another one in at runtime. */
|
||||
const LOADER_CALL = /\b(?:import|require(?:\.resolve)?)\s*\(\s*/g;
|
||||
const VALUE_BEFORE_SLASH = new Set([")", "]", '"', "'", "`"]);
|
||||
|
||||
/**
|
||||
* Blanks out comments, preserving newlines (and therefore line numbers) and
|
||||
* leaving real string/template literals untouched.
|
||||
* Keywords a regex literal may legally follow. Needed because the character
|
||||
* before the `/` in `return /re/.test(s)` is an identifier character, which would
|
||||
* otherwise read as division.
|
||||
*/
|
||||
const REGEX_AFTER_KEYWORD = new Set([
|
||||
"return",
|
||||
"typeof",
|
||||
"instanceof",
|
||||
"in",
|
||||
"of",
|
||||
"new",
|
||||
"delete",
|
||||
"void",
|
||||
"throw",
|
||||
"case",
|
||||
"do",
|
||||
"else",
|
||||
"yield",
|
||||
"await",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Whether the `/` whose previous significant code character sits at `lastIndex`
|
||||
* starts a regex literal rather than being a division operator.
|
||||
*
|
||||
* @param {string} code
|
||||
* @param {number} lastIndex - Offset of the last significant code character, or -1.
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function regexAllowedAfter(code, lastIndex) {
|
||||
if (lastIndex < 0) return true; // start of file
|
||||
const char = code[lastIndex];
|
||||
if (VALUE_BEFORE_SLASH.has(char)) return false;
|
||||
// Operators, `(`, `,`, `;`, `{`, `}`, `:` — all positions where a value starts.
|
||||
if (!IDENT_CHAR.test(char)) return true;
|
||||
let start = lastIndex;
|
||||
while (start >= 0 && IDENT_CHAR.test(code[start])) start -= 1;
|
||||
return REGEX_AFTER_KEYWORD.has(code.slice(start + 1, lastIndex + 1));
|
||||
}
|
||||
|
||||
/**
|
||||
* Single-pass scanner that classifies every character of a JS source file.
|
||||
*
|
||||
* @param {string} code
|
||||
* @returns {{ masked: string, literals: { start: number, end: number, interpolated: boolean, terminated: boolean }[] }}
|
||||
* `masked` has the SAME LENGTH as `code`, with every comment, string, template
|
||||
* and regex character replaced by a space and every newline preserved — so
|
||||
* offsets and line numbers still line up with the original. `literals` holds one
|
||||
* span per string/template literal in source order, `end` exclusive and
|
||||
* including the closing quote; `interpolated` is true for a template containing
|
||||
* `${…}`, and `terminated` is false for a literal the file never closes.
|
||||
*/
|
||||
export function scanSource(code) {
|
||||
/** @type {[number, number][]} Sorted, non-overlapping ranges to blank. */
|
||||
const blanks = [];
|
||||
const blank = (from, to) => {
|
||||
const start = Math.max(0, from);
|
||||
const end = Math.min(code.length, to);
|
||||
if (end <= start) return;
|
||||
const last = blanks[blanks.length - 1];
|
||||
if (last && last[1] === start) last[1] = end;
|
||||
else blanks.push([start, end]);
|
||||
};
|
||||
|
||||
/** @type {{ start: number, end: number, interpolated: boolean, terminated: boolean }[]} */
|
||||
const literals = [];
|
||||
/** Enclosing templates we are inside via `${…}`, innermost last. */
|
||||
const templates = [];
|
||||
|
||||
let mode = "code";
|
||||
let braceDepth = 0;
|
||||
let literalStart = -1;
|
||||
let interpolated = false;
|
||||
// Offset of the last significant CODE character; drives regex-vs-division.
|
||||
let lastCode = -1;
|
||||
let i = 0;
|
||||
|
||||
while (i < code.length) {
|
||||
const char = code[i];
|
||||
|
||||
if (mode === "code") {
|
||||
if (char === "/" && code[i + 1] === "/") {
|
||||
blank(i, i + 2);
|
||||
mode = "line-comment";
|
||||
i += 2;
|
||||
} else if (char === "/" && code[i + 1] === "*") {
|
||||
blank(i, i + 2);
|
||||
mode = "block-comment";
|
||||
i += 2;
|
||||
} else if (char === '"' || char === "'") {
|
||||
mode = char === '"' ? "double" : "single";
|
||||
literalStart = i;
|
||||
interpolated = false;
|
||||
blank(i, i + 1);
|
||||
i += 1;
|
||||
} else if (char === "`") {
|
||||
mode = "template";
|
||||
literalStart = i;
|
||||
interpolated = false;
|
||||
blank(i, i + 1);
|
||||
i += 1;
|
||||
} else if (char === "/" && regexAllowedAfter(code, lastCode)) {
|
||||
mode = "regex";
|
||||
blank(i, i + 1);
|
||||
i += 1;
|
||||
} else if (char === "}" && braceDepth === 0 && templates.length) {
|
||||
// The `}` closing a `${…}` interpolation: back into the template.
|
||||
blank(i, i + 1);
|
||||
const frame = templates.pop();
|
||||
braceDepth = frame.braceDepth;
|
||||
literalStart = frame.literalStart;
|
||||
interpolated = true;
|
||||
mode = "template";
|
||||
i += 1;
|
||||
} else {
|
||||
if (char === "{") braceDepth += 1;
|
||||
else if (char === "}") braceDepth = Math.max(0, braceDepth - 1);
|
||||
if (!/\s/.test(char)) lastCode = i;
|
||||
i += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (mode === "line-comment") {
|
||||
if (char === "\n") mode = "code";
|
||||
else blank(i, i + 1);
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (mode === "block-comment") {
|
||||
if (char === "*" && code[i + 1] === "/") {
|
||||
blank(i, i + 2);
|
||||
mode = "code";
|
||||
i += 2;
|
||||
} else {
|
||||
if (char !== "\n") blank(i, i + 1);
|
||||
i += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (mode === "single" || mode === "double") {
|
||||
const quote = mode === "single" ? "'" : '"';
|
||||
if (char === "\\") {
|
||||
blank(i, i + 2);
|
||||
i += 2;
|
||||
} else if (char === quote) {
|
||||
blank(i, i + 1);
|
||||
literals.push({
|
||||
start: literalStart,
|
||||
end: i + 1,
|
||||
interpolated: false,
|
||||
terminated: true,
|
||||
});
|
||||
lastCode = i;
|
||||
mode = "code";
|
||||
i += 1;
|
||||
} else if (char === "\n") {
|
||||
// A quoted string cannot span a raw newline, so either the source is
|
||||
// invalid or we mis-entered: end the span here rather than let one stray
|
||||
// apostrophe swallow the rest of the file.
|
||||
literals.push({
|
||||
start: literalStart,
|
||||
end: i,
|
||||
interpolated: false,
|
||||
terminated: false,
|
||||
});
|
||||
mode = "code";
|
||||
i += 1;
|
||||
} else {
|
||||
blank(i, i + 1);
|
||||
i += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (mode === "template") {
|
||||
if (char === "\\") {
|
||||
blank(i, i + 2);
|
||||
i += 2;
|
||||
} else if (char === "$" && code[i + 1] === "{") {
|
||||
blank(i, i + 2);
|
||||
templates.push({ literalStart, braceDepth });
|
||||
braceDepth = 0;
|
||||
mode = "code";
|
||||
i += 2;
|
||||
} else if (char === "`") {
|
||||
blank(i, i + 1);
|
||||
literals.push({
|
||||
start: literalStart,
|
||||
end: i + 1,
|
||||
interpolated,
|
||||
terminated: true,
|
||||
});
|
||||
lastCode = i;
|
||||
mode = "code";
|
||||
i += 1;
|
||||
} else {
|
||||
if (char !== "\n") blank(i, i + 1);
|
||||
i += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (mode === "regex" || mode === "regex-class") {
|
||||
if (char === "\\") {
|
||||
blank(i, i + 2);
|
||||
i += 2;
|
||||
} else if (char === "\n") {
|
||||
// A regex literal cannot span a newline, so we mis-read a division `/`.
|
||||
// Recover at the line break: a misread can never reach past one line.
|
||||
mode = "code";
|
||||
i += 1;
|
||||
} else if (mode === "regex" && char === "[") {
|
||||
blank(i, i + 1);
|
||||
mode = "regex-class";
|
||||
i += 1;
|
||||
} else if (mode === "regex-class" && char === "]") {
|
||||
blank(i, i + 1);
|
||||
mode = "regex";
|
||||
i += 1;
|
||||
} else if (mode === "regex" && char === "/") {
|
||||
blank(i, i + 1);
|
||||
lastCode = i;
|
||||
mode = "code";
|
||||
i += 1;
|
||||
} else {
|
||||
blank(i, i + 1);
|
||||
i += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
/* c8 ignore next -- unreachable: every mode above continues the loop. */
|
||||
i += 1;
|
||||
}
|
||||
|
||||
// An unterminated literal at EOF: record it so a loader argument inside it is
|
||||
// classified as NOT a complete literal, i.e. reported rather than skipped.
|
||||
if (mode === "single" || mode === "double" || mode === "template") {
|
||||
literals.push({
|
||||
start: literalStart,
|
||||
end: code.length,
|
||||
interpolated,
|
||||
terminated: false,
|
||||
});
|
||||
}
|
||||
|
||||
const parts = [];
|
||||
let cursor = 0;
|
||||
for (const [start, end] of blanks) {
|
||||
parts.push(code.slice(cursor, start));
|
||||
parts.push(code.slice(start, end).replace(/[^\n]/g, " "));
|
||||
cursor = end;
|
||||
}
|
||||
parts.push(code.slice(cursor));
|
||||
return { masked: parts.join(""), literals };
|
||||
}
|
||||
|
||||
/**
|
||||
* Blanks comments, string/template literals AND regex literals, preserving both
|
||||
* length and newlines (and therefore offsets and line numbers).
|
||||
*
|
||||
* Named for what it does: the previous `stripComments` blanked comments only and
|
||||
* left literals intact, which is precisely how import-shaped TEXT reached the
|
||||
* loader-call matcher and hard-failed CI on code that links nothing.
|
||||
*
|
||||
* @param {string} code
|
||||
* @returns {string}
|
||||
*/
|
||||
export function stripComments(code) {
|
||||
return code.replace(COMMENT_OR_LITERAL, (match) =>
|
||||
match.startsWith("//") || match.startsWith("/*")
|
||||
? match.replace(/[^\n]/g, " ")
|
||||
: match,
|
||||
export function maskNonCode(code) {
|
||||
return scanSource(code).masked;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every way a module can pull another one in at runtime, including rolldown's
|
||||
* `__require` CJS-interop shim (which `\brequire` misses: there is no word
|
||||
* boundary inside `__require`).
|
||||
*
|
||||
* The lookbehind rejects an identifier that merely ENDS with one of these words
|
||||
* (`myrequire(x)`) and the common `o.import(` member form; `precededByMemberDot`
|
||||
* then covers the same member call split across lines, which a single-character
|
||||
* lookbehind cannot see.
|
||||
*
|
||||
* Only ever applied to `scanSource`'s masked output, never to raw source.
|
||||
*/
|
||||
const LOADER_CALL =
|
||||
/(?<![.\w$])(?:__require|require(?:\.resolve)?|import)\s*\(/g;
|
||||
|
||||
/**
|
||||
* True when the loader word starting at `index` is a property access — `o.import(x)`,
|
||||
* `mod?.require(x)`, or the same split over a line break — which loads nothing.
|
||||
* `...import(x)` is a spread, not a member access.
|
||||
*
|
||||
* @param {string} masked
|
||||
* @param {number} index
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function precededByMemberDot(masked, index) {
|
||||
let i = index - 1;
|
||||
while (i >= 0 && /\s/.test(masked[i])) i -= 1;
|
||||
return i >= 0 && masked[i] === "." && masked[i - 1] !== ".";
|
||||
}
|
||||
|
||||
/**
|
||||
* Offsets of a loader call's FIRST argument, given the offset of its `(`, or null
|
||||
* when the call is never closed. Depth-aware over the MASKED source, so a paren or
|
||||
* comma inside a string cannot end it. Stopping at the first top-level comma keeps
|
||||
* `import("./m", { with: { type: "json" } })` a one-literal argument.
|
||||
*
|
||||
* @param {string} masked
|
||||
* @param {number} openIndex
|
||||
* @returns {{ start: number, end: number } | null}
|
||||
*/
|
||||
function firstArgumentRange(masked, openIndex) {
|
||||
let depth = 0;
|
||||
for (let i = openIndex; i < masked.length; i += 1) {
|
||||
const char = masked[i];
|
||||
if (char === "(" || char === "[" || char === "{") depth += 1;
|
||||
else if (char === ")" || char === "]" || char === "}") {
|
||||
depth -= 1;
|
||||
if (depth === 0) return { start: openIndex + 1, end: i };
|
||||
} else if (char === "," && depth === 1) {
|
||||
return { start: openIndex + 1, end: i };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a loader argument is a COMPLETE single literal — the only shape a
|
||||
* bundler can resolve. "Starts with a quote" is not enough, and that was the whole
|
||||
* false-negative class: `import("zo" + n)` and `` import(`stream${n}`) `` both
|
||||
* start with one and both hide their target.
|
||||
*
|
||||
* @param {object} options
|
||||
* @param {string} options.code - The original source (for the whitespace check).
|
||||
* @param {{ start: number, end: number, interpolated: boolean, terminated: boolean }[]} options.literals
|
||||
* @param {{ start: number, end: number } | null} options.range
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function isCompleteLiteralArgument({ code, literals, range }) {
|
||||
if (!range) return false;
|
||||
const inside = literals.filter(
|
||||
(literal) => literal.start >= range.start && literal.end <= range.end,
|
||||
);
|
||||
if (inside.length !== 1) return false;
|
||||
const [literal] = inside;
|
||||
if (!literal.terminated || literal.interpolated) return false;
|
||||
// Nothing but whitespace may surround it: that is what rejects a concatenation.
|
||||
return (
|
||||
code.slice(range.start, literal.start).trim() === "" &&
|
||||
code.slice(literal.end, range.end).trim() === ""
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Loader calls whose argument is not a string literal, and which therefore hide
|
||||
* whatever they load from any static analysis — including a bundler's.
|
||||
* Loader calls whose argument is not a complete string literal, and which
|
||||
* therefore hide whatever they load from any static analysis — including a
|
||||
* bundler's.
|
||||
*
|
||||
* @param {string} code
|
||||
* @returns {string[]} A short excerpt per unanalyzable call.
|
||||
*/
|
||||
export function unanalyzableLoaderCalls(code) {
|
||||
const stripped = stripComments(code);
|
||||
const { masked, literals } = scanSource(code);
|
||||
const found = [];
|
||||
for (const match of stripped.matchAll(LOADER_CALL)) {
|
||||
const after = stripped.slice(match.index + match[0].length);
|
||||
if (/^["'`]/.test(after)) continue;
|
||||
for (const match of masked.matchAll(LOADER_CALL)) {
|
||||
if (precededByMemberDot(masked, match.index)) continue;
|
||||
const openIndex = match.index + match[0].length - 1;
|
||||
const range = firstArgumentRange(masked, openIndex);
|
||||
if (isCompleteLiteralArgument({ code, literals, range })) continue;
|
||||
found.push(
|
||||
stripped
|
||||
// Excerpt from the ORIGINAL source: the masked form would print the
|
||||
// argument as blanks, which tells a reader nothing.
|
||||
code
|
||||
.slice(match.index, match.index + match[0].length + 48)
|
||||
.replace(/\s+/g, " ")
|
||||
.trim(),
|
||||
|
||||
Reference in New Issue
Block a user