Files

88 lines
4.6 KiB
Rust
Raw Permalink Normal View History

feat(kernel): R1 scaffold — napi-rs extraction kernel, buffer contract, routing + fallback, grammar-parity CI Phase 0 of the Rust extraction-kernel migration (docs/design/ rust-kernel-migration-plan.md, now checked in with §3a recording the shipped state): - codegraph-kernel/ napi-rs crate: extractFile(path, content, language) → five flat buffers (meta/nodes/edges/refs/arena), one JS boundary crossing per file. Node ids computed Rust-side, byte-identical to generateNodeId (pinned by test vector). Reserved per-node metrics slot for the Arc 3.2 code-metrics work. - Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures, byte-range scope stack → ::-joined qualified names, contains edges, refs attributed to the innermost enclosing symbol). Seed TS/JS queries are smoke-level; R2 replaces them with the full port. - Routing seam in extractFromSource with per-file wasm fallback. DEFAULT_ROUTED is empty — no behavior change until a language passes its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before routing; EDGE_KINDS became a runtime array because kind order is now wire contract. - Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the exact crate revisions (tree-sitter-typescript v0.23.2, tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) — the tree-sitter-wasms builds were 2023-era, which the new kernel-grammar-parity test caught on day one. Production TS/JS parsing gets 2.5 years of grammar fixes; full suite green (2456 tests). - Build/release wiring: scripts/build-kernel.sh + npm run build:kernel; release.yml kernel prebuild matrix (continue-on-error — the kernel is optional everywhere, bundles fall back to the wasm path); bundles stage lib/kernel/codegraph-kernel.node; release job runs the kernel suites with CODEGRAPH_KERNEL_EXPECT=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 20:13:51 -05:00
fn main() {
napi_build::setup();
feat(kernel): R7b Kotlin walker — kotlin module, vendored-grammar-C build, kotlin default-routed (#1382) Sixth R7b port — the T1½ batch finale. Checklist-first recipe (docs/design/kotlin-kernel-port-checklist.md, 1,121 lines, dist-extractor ground truth); parity passed FIRST RUN on all three repos. THE NOVEL MECHANISM — vendored-grammar-C (the §4 tracker's prescription, first use): the crates.io tree-sitter-kotlin 0.3.8 pins `tree-sitter >= 0.21, < 0.23` (the kernel links 0.25) and tree-sitter-kotlin-ng is a DIFFERENT grammar (8 fields vs 0, renamed kinds — extractor-breaking), so no crate dep is possible. The fwcd 0.3.8 tag's sha-matched parser.c + scanner.c are vendored into codegraph-kernel/grammars/kotlin and compiled by build.rs (cc), exposed via tree-sitter-language::LanguageFn. The wasm re-vendor is behavior-NEUTRAL (0 CST/error disagreements across 1,984 gate-repo files; old-vs-new full-init dumps byte-identical ×3) — a reproducibility re-vendor, ABI stays 14. Walker firsts: extension-function receivers (getReceiverType → `WidgetK::extend` QN OVERRIDE with no package prefix, the qualified-receiver `com::qext` first-segment bug, and the owner-contains fallback that excludes `interface` kinds and is source-order dependent) and extractModifiers (expect/actual platform modifiers → the node DECORATORS wire field on every created node — the KMP synthesizer's feed, incl. `actual typealias`). Preserved bug-for-bug: the FIELD_COUNT-0 dead cluster (no signatures, ZERO type-annotation refs), hook-consumed property initializers emitting nothing (incl. `by lazy {}`), the bodiless-vs-bodied class header asymmetry, enum- entry bodies being invisible, KDoc never a docstring AND chain-breaking, comment-gluing into import/package extents, `@Anno(args)` emitting nothing while `@Marker` decorates, zero instantiates refs, the paren-then-lambda `trailing()` garbage callee, text-includes visibility/suspend false positives, and the packaged-file value-ref target drop. The fun-interface misparse-recovery hook is DEFER-SHIELDED (every such file has_error) and deliberately not ported. The swift-sweep lesson pre-applied: the shared `assignment` shadow-prune case is implemented alongside the property_declaration case. Gates: sweeps 0-diff okio 299/322, okhttp 531/580, kotlinx.coroutines 1031/1082 (deferrals exactly the predicted 23/49/51 — both-arm grammar reality incl. PHANTOM hasError files with complete CSTs; the kernel trusts the flag); full-init dumps byte-identical ×3 (46.5k/108.9k/92.3k lines); KMP expect/actual synthesis IDENTICAL across arms (412 edges on kotlinx.coroutines — the tracker's KMP validation); kernel-kotlin-parity suite (torture reflowed off the phantom shapes + .kts script + CRLF variants + fun-interface and phantom defer pins) + kotlin grammar-parity row (the C-build ↔ wasm table identity proof); full suite 2,633 green ×2 under CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += kotlin (15 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 17:33:18 -05:00
// Kotlin grammar — vendored C, compiled here instead of a crate dep: the
// crates.io tree-sitter-kotlin 0.3.8 pins `tree-sitter >= 0.21, < 0.23`
// (the kernel links 0.25) and tree-sitter-kotlin-ng is a DIFFERENT
// grammar (8 fields vs 0, renamed kinds — extractor-breaking). Sources
// are the fwcd 0.3.8 tag's checked-in generated artifacts, sha-matched
// against the crates.io tarball (kotlin checklist §Grammar prep):
// parser.c 54104a7ef1555c265b746c790e0f8bb953cc17806e9df0c3af82f7f62c06a70a
// scanner.c 27f73337ec357fc341fa57538f34c14277b0346980c3405dc30beab6202ec6d0
// Flags crib the tarball's own bindings/rust/build.rs.
let mut c = cc::Build::new();
c.include("grammars/kotlin");
c.file("grammars/kotlin/parser.c");
c.file("grammars/kotlin/scanner.c");
c.flag_if_supported("-Wno-unused-parameter");
c.flag_if_supported("-Wno-unused-but-set-variable");
c.flag_if_supported("-Wno-trigraphs");
c.flag_if_supported("-utf-8"); // msvc
c.compile("tree-sitter-kotlin");
println!("cargo:rerun-if-changed=grammars/kotlin");
feat(kernel): R7b Lua+Luau walker — one lua module, vendored-grammar-C lua v0.4.1, tree-sitter-luau 1.2.0 pin, both default-routed (#1384) R7b batch 4 #2 (docs/design/lua-luau-kernel-port-checklist.md is the authoritative quirk list). ONE walker for both dialects (ccpp precedent) — the differences are exactly four: luau's type_definition aliases, the `export `-slice isExported hook, the return-type signature suffix, and the grammar handle. Grammar prep is kernel-side only, no wasm change: lua is the SECOND vendored-grammar-C language (the vendored wasm is the v0.4.1 tag, a revision not on crates.io — tag artifacts compiled via build.rs, shas pinned); luau is a plain crate pin =1.2.0 whose tarball is sha-identical to the tag (the swift tag≠crate divergence does not recur). Grammar-parity rows replace the bump gate entirely. Preserved bug-for-bug (all probe-pinned): the require/visitNode-hook ASYMMETRIES (top-level requires — including inside top-level if/for/while — mint import nodes while the identical body-level statement emits `calls "require"`; top-level `local x = foo()` initializers are invisible while global `x = foo()` calls emit), the BFS string-win inside require args (`require(script:WaitForChild("Kid"))` → import Kid) and Roblox instance paths, receiver-QN methods (`M.sub.deep::chained`, `_G::installed`, stack-QN nested globals like `render::leakedGlobal`), the raw-text callee world (colon forms with `self` never stripped, bracket callees, newline-glued chains byte-verbatim, the `(handler)` paren-conversion), LUA_SPEC function-as-value capture with the `M.cb = cb` param-storage skip and first-occurrence dedupe, LuaDoc `---` keeping a leading `- ` plus `--!strict` joining docstring chains (block-comment docstrings keep interior CRLF bytes), variable nodes at the IDENTIFIER with positional value pairing, duplicate same-(kind,name,line) ids, and the lua↔luau isExported wire divergence (lua functions: flag absent; luau functions: present-false; methods: absent in both; variables: present-false in both; `export type`: true). Gates: parity sweeps first-run 0-diff on kong/lazy.nvim/lua-resty-core (lua) + lune/Fusion (luau) — 1,734 clean files byte-parity, deferrals 1/0/0/3/8 matching the survey's both-arm predictions exactly (kong's 1 = a deliberately invalid fixture; luau's = grammar-inherent generic type packs and default type params); full-init dumps byte-identical kernel-vs-wasm ×4 (kong 157,650 dump lines); kernel-lua-parity suite (both torture fixtures + in-memory CRLF variants + glue-chain, duplicate-id, and cross-dialect defer pins + kernel-arm wire-flag pins); full suite 2,647 green ×2 with CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += lua, luau (18 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 18:42:14 -05:00
// Lua grammar — vendored C (second vendored-grammar-C language): the
// vendored wasm is tree-sitter-grammars/tree-sitter-lua v0.4.1 (tag
// 816840c592), which is NOT on crates.io (only 0.1/0.2/0.5 exist; 0.5.0
// adds Lua-5.5 `global` — a future bump with its own gate). Sources are
// the v0.4.1 tag's checked-in generated artifacts, sha-recorded in the
// lua-luau checklist §Grammar prep:
// parser.c b34a362e43f0311f405721f3089e94f97f31da403b154d456d093e64609a4081
// scanner.c 35bbd630b5a7421d46d2e91185eeea09bf78565d44cb676b63ca20d0f1b54bbd
let mut lua = cc::Build::new();
lua.include("grammars/lua");
lua.file("grammars/lua/parser.c");
lua.file("grammars/lua/scanner.c");
lua.flag_if_supported("-Wno-unused-parameter");
lua.flag_if_supported("-Wno-unused-but-set-variable");
lua.flag_if_supported("-Wno-trigraphs");
lua.flag_if_supported("-utf-8"); // msvc
lua.compile("tree-sitter-lua");
println!("cargo:rerun-if-changed=grammars/lua");
feat(kernel): R7b Scala walker — scala module, vendored-grammar-C master@0aca5d0a6f, scala default-routed (#1385) R7b batch 4 #3 (docs/design/scala-kernel-port-checklist.md is the authoritative quirk list). The third vendored-grammar-C language and the biggest grammar in the tree (35MB parser.c): the vendored wasm is tree-sitter/tree-sitter-scala master@0aca5d0a6f — a post-v0.26.0 generation sync that is not a release (the 0.26.0 crate is 30 states BEHIND, so a crate pin would be a silent downgrade). NO wasm change: production has parsed with this exact revision since #91 — the kernel-grammar-parity row (ABI 15, 26,650 states, 32 fields, id-by-id tables) is the whole alignment proof. Preserved bug-for-bug (all probe-pinned): the leak-through asymmetries — extension methods mint NO nodes (first def's body calls leak to the enclosing scope, later defs invisible, and the braced form resolves its body field to the `{` TOKEN via first-match-wins field lookup → whole extension invisible); anonymous `new T { … }` template_body members leak to the enclosing scope (findAnonymousClassBody misses template_body); the bodied-vs-bodiless class asymmetry (bodiless headers walk class_parameters → default-value calls emit FROM the class; bodied ones never see them) — plus first-segment import names (`import com.example.C` → `com`), the val/var hook keyed on the enclosing-definition NODE TYPE (object vals → constants/value-ref targets, class/trait/enum/given vals → fields) with consumed initializers, every def routed through extractMethod with the top-level function fallback, nested defs in bodies minting NOTHING (the inverse of kotlin) while body-local classes extract fully, curried signatures keeping only the FIRST parameter list (type params win the `parameters` field), enum cases positioned at the CASE node with invisible params/extends tails, extends with-chains via scalaBaseTypeName, `@deprecated(args)` decorates, the #750 capitalized-chain re-encode (`WidgetS.create().render`), literal-receiver silence, static-member reads AND writes, infix invisibility, `derives` silence, scaladoc retention with the CRLF `\r` pin, full value-reference machinery (shadow prune, last-wins same-name targets, `$X`/`${X}` interpolation reads), and SCALA_SPEC fn-refs (bare ids + postfix eta unwrap + varinit, var-init non-capture). Gates: parity sweeps first-run 0-diff on os-lib/cats/scala3-compiler-src/ scala3-library-src — 1,935 clean files byte-parity, deferrals 0/15/57/116 matching the survey's predictions exactly (scala-3's PHANTOM hasError files — flag-true, zero ERROR nodes, capture-checking `^` — defer on the FLAG); full-init dumps byte-identical ×3 (os-lib, cats, scala3 whole-repo 950,889 dump lines); kernel-scala-parity suite (9 fixtures + 9 in-memory CRLF variants incl. Scala-3 indentation through the external scanner + phantom/real-error defer pins + first-segment/namespace/value-ref pins); full suite 2,669 green ×3 with CODEGRAPH_KERNEL_EXPECT=1 (kernel-scaffold's stays-wasm example moved scala → pascal). DEFAULT_ROUTED += scala (19 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 19:02:52 -05:00
// Scala grammar — vendored C (third vendored-grammar-C language): the
// vendored wasm is tree-sitter/tree-sitter-scala master@0aca5d0a6f (the
// 2026-04-22 generation sync — 30 states PAST the v0.26.0 tag/crate, so a
// crate pin would be a silent downgrade). Sources are that commit's
// checked-in generated artifacts, sha-recorded in the scala checklist
// §Grammar prep:
// parser.c bc3c3c794f19461d99d04de6c31d57fa3e41243509b9ab023a9b88ed3273d102
// scanner.c e4ba242568ee3493015598997bf60f613802616eade62717c21109287ef64752
// parser.c is 35 MB — the biggest grammar in the tree; expect a slow cc
// step on clean builds.
let mut scala = cc::Build::new();
scala.include("grammars/scala");
scala.file("grammars/scala/parser.c");
scala.file("grammars/scala/scanner.c");
scala.flag_if_supported("-Wno-unused-parameter");
scala.flag_if_supported("-Wno-unused-but-set-variable");
scala.flag_if_supported("-Wno-trigraphs");
scala.flag_if_supported("-utf-8"); // msvc
scala.compile("tree-sitter-scala");
println!("cargo:rerun-if-changed=grammars/scala");
feat(kernel): R7b Dart walker — dart module, vendored-grammar-C d4d8f3e + wasm byte-copy vendor, dart default-routed (#1386) R7b batch 4 #4 — the FINAL R7b language (docs/design/dart-kernel-port-checklist.md is the authoritative quirk list). The fourth vendored-grammar-C language, with a twist: production dart resolved its wasm from tree-sitter-wasms, whose dart dependency is an UNPINNED github:UserNobody14/tree-sitter-dart — a routine dependency update would have silently changed dart's grammar. This PR byte-copies the shipping 0.1.13 artifact into src/extraction/wasm/ (VENDORED_WASM_LANGS += dart) and compiles the same-commit (d4d8f3e337d8) parser.c/scanner.c in the kernel — table identity proven by the kernel-grammar-parity row. crates.io tree-sitter-dart is the nielsenko fork (different lineage) — rejected. The center of gravity is THE SIBLING-BODY DOUBLE-WALK, reproduced bug-for-bug: dart attaches every function/method body as a NEXT SIBLING of its signature, and the TS walkers consume each body TWICE — once via resolveBody (attributed to the function/method) and once via the enclosing generic walk (attributed to the file/class). Duplicate local-function nodes with the SAME id under different parents, duplicated calls/instantiates refs, and file/class-attributed fn-ref twins all emit in the exact observed interleave (a dedicated fixture pins the duplicate-id rows; the bloc kind-census spot-check pins the counts). Also preserved (probe-pinned): the extractBareCall selector matrix (the first callTypes=[] language — cascades completely invisible, `?.` encodes like `.`, the `ConfigT.load()` calls+references double emission with no callee-of-call skip, capitalized-chain `Foo.create().run` re-encode, const-object callee names); the constructor hooks (unnamed ctor skipped, named ctors/factories renamed to the CTOR name with the class as returnType, `@override (T) m()` record-misparse rescued by class-name validation); operator methods minting `method "<anonymous>"`; static_final_declaration constants via the visitNode hook while instance fields mint NOTHING; the prefixed-return-type prefix bug (`other.OtherClass f()` → returnType `other`); enum `with` mixins silent vs `implements` working; anonymous extensions named after the ON type; deferred imports invisible; named-argument callbacks NOT fn-ref-captured (the Flutter `onPressed:` idiom — future accuracy PR, TS-side first); `async*`/`sync*` NOT async; value-refs with the LIVE dart sibling-body pull and the `$X`-vs-`${X}` interpolation asymmetry; dartdoc kept in all three comment forms with the annotation-broken chain. Gates: parity sweeps first-run 0-diff on shelf/bloc/flutter — 5,815 clean files byte-parity, deferrals 10/21/1341 ≈ the survey's 10/21/~1340 (both-arm grammar reality: empty object patterns — the sealed-class idiom — and unnamed `library;` dominate; --max-deferral 0.3); full-init dumps byte-identical ×3 (shelf 7,959 / bloc 40,026 / flutter 1,855,319 dump lines); bloc per-kind node census identical across arms (the double-walk duplicate rows survive the store identically); kernel-dart-parity suite (7 fixtures + in-memory CRLF variants + double-walk duplicate-id pin + generated-file skip pin + two defer pins); full suite 2,688 green ×2 with CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += dart (20 langs — R7b COMPLETE). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 19:55:48 -05:00
// Dart grammar — vendored C (fourth vendored-grammar-C language): the
// production wasm is the tree-sitter-wasms 0.1.13 artifact, whose dart
// dependency is an UNPINNED github:UserNobody14/tree-sitter-dart — the
// vendored wasm byte-copy (src/extraction/wasm/) plus these same-commit
// sources kill that hazard. crates.io tree-sitter-dart is the nielsenko
// FORK (different lineage) — rejected. Sources are
// UserNobody14/tree-sitter-dart master@d4d8f3e337d8 (dart checklist
// §Grammar prep):
// parser.c 5a42b47abb4d494f125dbdee9138979248041689b1aa36355550fa3e28dcb8b8
// scanner.c 07a7b7818b175e9460523e705dd88d20f7b5141bac95c593d4426e6d52284996
// scanner.c is load-bearing: string-template char classes and /** */ doc
// comments are external tokens.
let mut dart = cc::Build::new();
dart.include("grammars/dart");
dart.file("grammars/dart/parser.c");
dart.file("grammars/dart/scanner.c");
dart.flag_if_supported("-Wno-unused-parameter");
dart.flag_if_supported("-Wno-unused-but-set-variable");
dart.flag_if_supported("-Wno-trigraphs");
dart.flag_if_supported("-Wno-unused-function");
dart.flag_if_supported("-utf-8"); // msvc
dart.compile("tree-sitter-dart");
println!("cargo:rerun-if-changed=grammars/dart");
feat(kernel): R1 scaffold — napi-rs extraction kernel, buffer contract, routing + fallback, grammar-parity CI Phase 0 of the Rust extraction-kernel migration (docs/design/ rust-kernel-migration-plan.md, now checked in with §3a recording the shipped state): - codegraph-kernel/ napi-rs crate: extractFile(path, content, language) → five flat buffers (meta/nodes/edges/refs/arena), one JS boundary crossing per file. Node ids computed Rust-side, byte-identical to generateNodeId (pinned by test vector). Reserved per-node metrics slot for the Arc 3.2 code-metrics work. - Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures, byte-range scope stack → ::-joined qualified names, contains edges, refs attributed to the innermost enclosing symbol). Seed TS/JS queries are smoke-level; R2 replaces them with the full port. - Routing seam in extractFromSource with per-file wasm fallback. DEFAULT_ROUTED is empty — no behavior change until a language passes its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before routing; EDGE_KINDS became a runtime array because kind order is now wire contract. - Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the exact crate revisions (tree-sitter-typescript v0.23.2, tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) — the tree-sitter-wasms builds were 2023-era, which the new kernel-grammar-parity test caught on day one. Production TS/JS parsing gets 2.5 years of grammar fixes; full suite green (2456 tests). - Build/release wiring: scripts/build-kernel.sh + npm run build:kernel; release.yml kernel prebuild matrix (continue-on-error — the kernel is optional everywhere, bundles fall back to the wasm path); bundles stage lib/kernel/codegraph-kernel.node; release job runs the kernel suites with CODEGRAPH_KERNEL_EXPECT=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 20:13:51 -05:00
}