mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
a7b09fbefea27ea696c8eec0a6ff622ed4d2b787
173 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8a39b7c876 |
build: consume form-graph from npm instead of the local link
Swaps the pnpm link: override for the published package (^0.3.0, currently 0.3.1) and removes the two accommodations the link needed: the turbopack.root widening (Turbopack refused files outside the repo) and the tsconfig react/react-dom paths pin (the linked checkout's own @types/react could split the program). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
60004ec26f |
feat(generation): 14 more image families, reconcile lane, persistence scopes
Graphs + handlers + differentials for flux (5 modes), flux-kontext, flux2, flux2-klein, boogu, krea2, imagen4, pony-v7, reve, mai, ernie, seedream, anima, mage-flow, hi-dream, hi-dream-o1, openai, lens, qwen family and nano-banana — the image hub is complete except grok (spans both output types). Probed workflowVersions semantics differ per family: boogu's model wins the workflow (reconcile registry), mage-flow index-remaps (a correct), qwen substitutes via the lock. reconcile.ts: v1's cross-level checkpoint effects as one pure policy with two adapters — a raw normalizer before parse and a store rule on both hubs. Locked slots beat cross-family models; version siblings re-pick through the lock. Family effectiveEcosystem computeds delegate to it. Persistence: per-family scope buckets via the lib's new graph.scope (familyScope group-or-key, workflowScoped images/video, perModelScope turbo-variant cfg/steps, bare seed/controlNets, ecosystem per output type), persisted under one localStorage record in BaseGenerationForm; layout and rehydration pinned by persistence.test.ts. Review-pass hardening: handler differential pins each case's parsed ecosystem (caught four redirect-vacuous cases), matrix extra-shapes keyed by record with canaries, per-key boundary shape, exact-set video canary (exposed legacy WanVideo as row-less). Reuse hoists: familyResources, perModelSlider, img2imgImages, versionModeOf shared by graph+handler lanes, resourcesToLoras. Types: zero as-never casts. Fixed a mis-transcribed snippets reference schema (now imports the real one), widened PartialResourceValue to the picker's real pre-hydration contract, def factories converted to satisfies so input types flow to Controller onChange, SelectInput/OutputFormatInput genericized over their value unions. Toolchain (remove before PR): link: override + turbopack.root widening + react type-path pins for local lib development. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
a961f68bfc |
feat(generation): form-graph client forms, handler lane, and seedance
Three lanes on top of the ported graphs, all review-driven this session:
CLIENT (`src/components/form-graph/generation/`):
- BaseGenerationForm owns the store over the composed root (FormProvider),
renders the workflow picker and the submit footer, and switches between
the per-output forms on the graph's own `output` computed.
- Image/VideoGenerationForm in the generation_v2 GenerationForm idiom: one
`<Controller graph={imageHub|videoHub} name=...>` per field — typed from
that hub's registry (computeds and branch tags included via the state
type) — wired to the REAL generation_v2 inputs: BaseModelInput,
ResourceSelect* with a ported VersionGroupSelector,
ImageUploadMultipleInput, VideoInput, GenerationTextEditor (snippets +
trigger words from editor meta), ControlNetsInput, AspectRatioInput,
Priority/OutputFormat, ActiveWildcards with the ported add/remove flow,
ResourceAlerts via MultiController, and v2's wan version picker (sets
ecosystem; the tag follows).
- Demo at `/form-graph` (standalone; mounts ResourceDataProvider itself).
`/data-graph-v2` now renders the packaged GenerationFormV2 — its
hand-assembled provider stack was missing GenerationProvider and
ResourceDataProvider and threw for any signed-in visitor.
SERVER (`src/server/services/orchestrator/form-graph/`):
- Per-family handlers (ltx, wan, seedance, stable-diffusion, z-image,
chroma) transcribed from the data-graph handlers but importing only
ported modules, plus a dispatcher (`createFormGraphStepInput`) with v1's
seed normalization and enhanced-compatibility engine rewrite; unported
ecosystems are a loud error.
- The acceptance gate: a differential test feeds BOTH dispatchers the same
parsed data and asserts identical @civitai/client steps — 29 cases over
draft batching, comfy routing, controlnets, every wan version (both
flipt states of 2.2 multi-step), distilled LTX, prompt-enhancer $refs.
GRAPHS:
- video/seedance.graph.ts (no resources, no negative prompt; per-version
resolution/duration ceilings) — unblocked the video matrix's
hidden-ecosystem gate coverage.
- `resolveCompatibleEcosystem`: v1's workflow→ecosystem sync effect as a
pure redirect in both hubs' input transforms (txt2img + WanVideo30
parses as SD1, pinned), with the workflow-group override guard.
- LTX gets the model-wins split (`effectiveEcosystem` emit +
`checkpointDef({ modelWins: true })`); wan/ltx `shift`/`numFrames`
mirror v1's REJECTING inputs; full v1 picker meta everywhere
(controlNet groups, sampler presets, gate payloads, editor meta with
the editor-vs-registers-target split for ZImage Base).
- Family dispatches are switch statements returning member graphs
directly (record-less branches).
Toolchain: the form-graph override is `file:` (Turbopack cannot read
outside the workspace root, which `link:` always violates) and the package
is in `transpilePackages`. Requires local form-graph at 066377d — publish
+ swap the override before any push or PR.
Verified: 4,757 differential cases, typecheck and lint clean; full unit
suite 29,056 passed with only the 10 pre-existing Windows-portability
failures (identical on clean main).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
eaa277b0d4 |
docs(build): close option 1 on the Turbopack chunk collision — nested async chunking does not fit the memory ceiling (#4500)
The flag was already tried and reverted: #3458 enabled it 2026-07-30, #3807 turned it off 08-11 after the release build OOMKilled three times at 37-39 GiB against the enforced 40Gi builder cgroup. The 2026-08-18 doc was written after that revert and still recommended it as the only lever, with the RSS cost listed as unverified. Re-measured on 16.3.1: +43% peak next-build RSS, worse than the +33% on record. Live distribution n=67 builds, worst observed 28.59 GiB; times that multiplier lands at 37.3-39.8 GiB, the band the OOM kills were recorded in. No headroom. The turbopackSourceMaps escape hatch is closed too - it cannot be split client/server and server .js.map has a hard consumer gate. The flag VALUE is unchanged (still false). This is docs plus one comment, so the lever is closed off in the place someone would next reach for it. |
||
|
|
bc74ba06ea |
feat(apps): merge /apps/my-submissions into /apps/mine — one author table, nested history, inactive collapse (#4154)
* feat(apps): merge /apps/my-submissions into /apps/mine — one author table `/apps/mine` becomes THE author surface: one table over every app the caller owns or holds an accepted collaborator seat on, on-site and off-site, with that app's publish history nested in its row and fetched on expand. `/apps/my-submissions` 301s here; its page component is deleted rather than stubbed. The row set stays `appListings.listMine` (→ `resolveAccessibleListingIds`). Re-deriving it from a submissions read is the regression this page exists to prevent: a publish request is scoped to `submittedByUserId`, so a collaborator (who submitted nothing), a transfer recipient and a moderator-claimed owner all see an empty page on apps they can plainly edit. Server: - `listMyAppListings` now carries `iconUrl` / `coverUrl` / `updatedAt`. The media columns are integer FKs to `Image`, so this is a select widening plus a shared URL projection (`listing-media-url.ts`, also adopted by the public store read so there is one copy of the widths). No screenshot fallback here — a missing cover is the fact its author needs to see. - New `appListings.listingHistory`: one listing's merged publish history, authorized through `resolveListingAccess` (ownership ∪ accepted seat), gated on `app-blocks-author` only. UI: - Active/inactive partition on LISTING status only (`rejected`, `removed`). `draft` stays in the main table; a withdrawn SUBMISSION never moves an app. - Inactive collapse: closed by default, count in the header, paginated, with `aria-expanded`/`aria-controls` derived from the same state the panel renders. - Icon + cover per row at fixed dimensions with `loading="lazy"`, and a placeholder for each — all 11 `removed` listings in production have no cover, so that path is the main render path for the collapse. - Mobile degrades to cards; exactly one layout is rendered. - The "My submissions" sub-nav tab is retired and `hasSubmissions` now widens "My apps", so no population loses its route. * test(apps): use the canonical dbMock in the two new block-service tests `no-direct-shared-module-mock` went red on them: a per-file mock of the db-client specifier freezes that file's mock shape into every later file in the same worker under `isolate: false`. Both now take `dbMock.dbRead` / `dbMock.dbWrite` from `src/__tests__/mocks/db.mock`, which the global setup registers and resets per file. No assertion changed — the local `mockDb` / `mockWriteDb` handles keep their names, so the bodies are untouched. The guard is a TEXT scan over the file, so the forbidden call cannot be quoted even in a comment; the note explaining the migration says so rather than spelling it. * fix(apps): surface first-version submissions, restore the completeness advisory Four audit findings on #4154. §1 (blocking) — `app_block_publish_requests.app_block_id` is NULL until approve, and the history query keyed on it alone. Measured on production 2026-08-20: 3 of 3 `rejected` and 27 of 33 `withdrawn` rows carry a NULL FK, so 100% of rejections were unreachable — including from the "your app was rejected" notification this PR repoints at /apps/mine. Two sub-cases: (a) a PENDING first version still has its draft listing, only the FK is null. `blockRequestWhereForListing` now falls back to the listing's slug, scoped to its CANONICAL OWNER (never the viewer, so a collaborator still sees it; never unscoped, so a recycled slug cannot leak a stranger's review) and gated on `kind === 'onsite'`, which is now what keeps off-site listings out of the code stream. (b) a REJECTED/WITHDRAWN first version has its listing deleted to release the slug. New submitter-scoped `listMyOrphanedSubmissions` + `appListings.listMyOrphanedSubmissions` surface those as their own always-visible "Submissions without a listing" group, with the reviewer's reason. Submitter-scoped because it is the only identity the surviving record carries. No schema change, no migration; the deletion is deliberately left as-is. §2 (blocking) — `computeListingProblems` had lost its only renderer when the two my-submissions tables lost their importer, which also falsified `listingCoverUrl`'s "the author must see the gap" rationale. `listMine` now carries `problems` and the row renders `ListingProblemsIndicator`. §3 — Withdraw was offered to everyone. Both procs are submitter-scoped, so a collaborator / transfer recipient / mod-claimed owner got a button that only red-toasts. Entries now carry a server-computed `canWithdraw`. §4 — `blocks.withdrawPublishRequest` carries `enforceAppBlocksFlag` while this page gates on `appBlocksAuthor` only. The UI is gated rather than the mutation loosened: removing an authorization gate is not a change to make in passing. Also fixed here: the empty state was an early return, so an account whose only records were orphans would have seen "you don't own any apps yet" instead of them — the same defect one layer up. Caught by its own test. `app-access.accessible-listings.test.ts`'s fake discriminated its two `findUnique` reads by `select.slug`, which worked only because the access resolver happened not to want that column; it now keys on `connectClientId`, which is asked for by exactly one of them on purpose. * fix(apps): close the seams an independent mutation sweep found open My sweep reported 9/9 killed; an independently-constructed one found 7 survivors out of 13. Two of them reintroduced, undetected, findings this PR had just fixed. That is the standing lesson about a sweep built from its own assertions, and it is why these are behavioural fixes, not extra assertions. (a) The off-site half of `canWithdraw` had no guard. The two streams are mapped by two separate `.map()` calls, so the wiring exists twice and can be wrong in one alone — rebinding the LISTING stream to treat every viewer as the submitter SURVIVED, while the identical block-stream mutation died. Each stream is now pinned on its own, plus a case asserting both answers in ONE payload with the two ids deliberately different. (b) The `problems` wiring was unpinned server-side: `computeListingProblems` is well tested as a pure function and the client test supplies `problems` by hand, so both sides were green in isolation and nothing tested the join. `problems: []`, `coverId` fed from `r.iconId`, and the screenshot default `0 -> 99` all passed. Fixtures now carry pairwise-distinct sibling values (iconId 7, coverId 9, screenshots 3) so an operand swap changes the answer. The screenshot mutant needed one more turn: `0 ?? 99` is `0`, so with the relation PRESENT the default is unreachable by construction. It fires only when `_count` is absent, which now has its own case — and pins the direction, since a default that hides `no-screenshots` silently turns an incomplete listing into a clean one. (d) Two silent-zero paths on the one surface this population has. A failing `listMine` early-returned and took the orphan group with it; and `orphansQuery.error` was read NOWHERE, so a failed orphan read rendered nothing and said nothing — indistinguishable from "you have no rejected submissions", the exact lie the group exists to stop telling. Both now surface, neither blanks the other, and the empty state no longer renders over a broken read. Also, all cheap and all wrong before: - the kind-gate rationale claimed a no-op; it is a real behaviour change for the offsite-listing-that-carries-a-block shape (#3844), where the old `appBlockId ? … : null` DID run the block query. Stated as the tightening it is, with why. - a `describe` title my own delta falsified ("CONDITIONAL on the block, not on the kind" — it is now conditional on the kind). - `take: limit` ran BEFORE the ownership de-dup, so the orphan group could return a short page while more existed. Split into `ORPHAN_SCAN_LIMIT` (the read) and `ORPHANED_SUBMISSIONS_LIMIT` (applied after the filter). The dead `opts.limit` — unreachable from the router — is dropped rather than wired. - softened the null-FK branch's coverage claim: it does NOT restore first-version history for a transferred or moderator-claimed listing, only for an owner who submitted it themselves. * fix(apps): close the two guards that became silent-zero paths themselves F1 — the early return keyed on orphan DATA length but not orphan ERROR, so with both reads failed and zero orphan rows only the listMine alert rendered and the orphan failure reported nothing, permanently. Zero rows is not the same fact as "the read succeeded and found none" — the silent-zero lesson, applied to the guard written for it. F2 — the container never forwarded `orphansQuery.isLoading`, so `hasNothingAtAll` could not tell "no orphans" from "orphan read in flight". The two procedures batch into one request under `httpBatchStreamLink` but stream back independently, so `rowsQuery` resolving empty first is ordinary, not exotic — and the page asserted "You don't own or collaborate on any apps yet" over a pending read. Streaming makes that MORE reachable, not less. The `isLoading && orphanedSubmissions.length === 0` guard was also uncovered (reverting it to plain `isLoading` survived the last battery). Its real effect is not the error case its comment described: once orphans resolve while rows are still loading, falling through renders the group rather than a spinner over data that already arrived. Covered and documented as what it does. Two comments corrected, both over-claiming rather than wrong: - the de-dup ordering test claimed it kills a `take` mutant. It does not — `mockImplementation` ignores `take`, so all 28 rows come back regardless and the case passes. It pins the ORDER of de-dup vs display cap; renamed and re-commented to say so, and the remaining self-referential `take` coverage is recorded in the follow-up issue. - `ORPHAN_SCAN_LIMIT` read as though under-return were closed. It is 4× harder, not closed: still reachable at ≥76 of the newest 100 rows de-duping as owned with non-owned rows past position 100. Stated with the real bound, plus the index reality — neither existing index can serve the ORDER BY under this predicate, so Postgres top-N sorts either way and a wider scan is cheap. |
||
|
|
0f1830eb56 |
Merge remote-tracking branch 'origin/main' into moderator-app-pages
# Conflicts: # src/pages/api/mod/csam/training-data-report.ts # src/pages/api/mod/training-data/resolve.ts # src/server/controllers/csam.controller.ts |
||
|
|
c78f85dd5a |
fix(deps): Next 16.3.0 → 16.3.1, and make the compiled-branch gate hard (#3983) (#4075)
* fix(deps): Next 16.3.0 -> 16.3.1, and make the compiled-branch gate hard (#3983) This is the fix for #3983. The defect was in the bundler, not in our source. Turbopack's value analyzer in 16.3.0 models a bare `return someAsyncFn()` tail call as `Promise<Promise<T>>`. That is always truthy, so a caller that `await`s it is analysed as always-true and every statement after the resulting conditional is eliminated as dead code. `isAppListingsEnabled` ends in exactly that shape, which is why `resolveStoreVisibilityScopeUninstrumented` lost two of its three returns, fell off the end, and produced `undefined` for every non-privileged caller — served as the whole catalog on one read path (`?? 'full'`) and as an empty store on the other (`?? 'none'`). Upstream: vercel/next.js#96601 "[turbopack] Collapse nested promises in the analyzer", backported as #96675, shipped in 16.3.1. MEASURED, not inferred. Two production builds of THIS commit on one machine, same Node 24.19.0, differing only in the pinned Next: 16.3.0 async function S(e){if(await p(e))return"full"} 16.3.1 async function w(e){return await c(e)?"full":await y(e)?"public-external":"none"} Both read out of the emitted `.next/server` chunks by source-map attribution and identified by their source neighbour `STORE_SCOPE_FLAGS`, never by minified name. Note the fixed form is a TERNARY — `grep 'return"public-external"'` returns zero on the FIXED build too, which is why the gate reads source maps. `package.json` already allowed 16.3.1 (`^16.3.0`); only the lockfile pinned 16.3.0, so the substance here is the lockfile. The floor is raised to `^16.3.1` so a fresh resolution cannot land back on the broken compiler. `patches/next@…` is renamed and its `patchedDependencies` key updated — that patch is the unrelated libvips/SVG one-liner (vercel/next.js#96681), it still applies cleanly, and 16.3.1 still does not carry the loader entry upstream, so it stays. `--warn-only` is removed from `scripts/assert-compiled-branches.mjs` in the same commit. It existed only because the 16.3.0 build genuinely violated the gate, and a permanently-red gate trains everyone to click through. Keeping the bump and the strictness atomic means the gate's strictness always matches the toolchain: a revert of the bump turns it red instead of silently passing. Verified on this commit: - gate exit 0 (hard, no --warn-only) against the 16.3.1 build - gate exit 1 against a 16.3.0 build of the same tree — watched red - `scripts/ci/assert-next-svg-patch-applied.mjs` OK on both installed copies - unit suite 1149 files / 18,123 tests passed, 0 failed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(build): ship @swc/helpers' module-sync branch into the standalone image (#3983) The bump built clean, passed every source-level gate — unit suites, typecheck, ESLint + Prettier, schema drift, the event-engine pin, and the now-hard compiled-branch gate — and the container could not boot: Error: Cannot find module '.../@swc/helpers/esm/_interop_require_default.js' at ... next/dist/server/require-hook.js code: 'MODULE_NOT_FOUND' Same shape as the defect this PR exists to fix: a correct source tree producing a broken artefact, invisible to everything that reads source. ROOT CAUSE, measured on the published artefact rather than inferred. `output: 'standalone'` does not ship node_modules; it ships the subset @vercel/nft traced. nft resolves a bare specifier under the `require`/`default` conditions. Node (>= 22.10) additionally honours `module-sync` for a CJS `require`. When a package's `exports` map points those two at different files, the build traces one and the running process asks for the other. next/dist/shared/lib/constants.js does `require('@swc/helpers/_/_interop_require_default')`, reached from the generated server.js via `next` -> config.js -> constants.js, i.e. before any application code. The relevant delta is not next itself but next's own dependency: next 16.3.0 next 16.3.1 @swc/helpers 0.5.15 0.5.23 ./_/_interop_require_default {import,default} {module-sync,webpack,import,default} require.resolve() under CJS cjs/...cjs esm/...js Both resolutions were RUN, not reasoned about. nft still traced the cjs file, so the published image carried that package as exactly cjs/_interop_require_default.cjs, cjs/_interop_require_wildcard.cjs and package.json — no esm/ directory at all. Adding only the missing esm/ directory to that exact image, nothing else changed, boots it: "Next.js 16.3.1 ... Ready". FIX. `outputFileTracingIncludes` force-includes BOTH condition branches of EVERY installed @swc/helpers copy — not the one file missing today, because which helper Next requires and which branch each resolver picks are upstream details that move. Globs are version- and hash-agnostic (`@swc+helpers@*`), plus a flat form for a hoisted layout. ~950 KB per copy. Verified on a local production build of this commit: both copies land in .next/standalone with complete esm/ (108 and 105 files) and cjs/, and next's virtual store links the 0.5.23 copy. Attached to three existing API-route keys rather than a `'**'` key. copyTracedFiles unions every entry's traced set into the single .next/standalone node_modules, so one entry carrying it is enough, while `'**'` would make all 572 entries read/parse/rewrite their .nft.json concurrently — 826 MB of JSON in one Promise.all — on a build already tuned against OOM. GATE, because a glob is a silent no-op once it stops matching. scripts/ci/assert-standalone-boot-graph.mjs runs in the Dockerfile's RUNNER stage: the first gate in that file to run against the runtime filesystem rather than the build tree, and the only one that can see this class of defect. It reads the GENERATED server.js for the specifiers that process requires at module scope and loads them in a child rooted at the shipped tree — no package, version, virtual-store path or patch hash hardcoded, so it keeps covering this after the next bump. Exit 2, never 0, when it cannot observe its input. It must run in the runner and not the builder: /app there is byte-for-byte what ships, whereas the builder's complete node_modules sits above .next/standalone on the resolution path and can satisfy a require the image cannot. Watched red and green on real artefacts, not only fixtures: - exit 1 with this exact MODULE_NOT_FOUND against the published broken image; - exit 0 against the same image with only the esm/ directory added; - exit 0 against the local production build of this commit, isolated from any parent node_modules; - exit 1 again after deleting exactly esm/_interop_require_default.js from that same local build. src/tests/build/standalone-boot-graph.test.ts pins the MECHANISM (a module-sync/default split with only the default branch present) rather than the package, and all 5 of its cases were watched to fail against a neutered gate. NOT VERIFIED. Nothing about production: this is only true of production once it merges, is promoted main -> release, is built and is serving. The gate covers the ENTRYPOINT's require graph; route chunks load lazily, so a condition mismatch reachable only from a route would still surface at request time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: retrigger preview The previous preview run (pr-preview-4075-b2wnw) never scheduled: its build-image and typecheck pods sat Pending with ExceededNodeResources for 82 minutes and the run hit the 1h30m PipelineRunTimeout. No verdict was produced — this was build-pool capacity contention, not a code failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
fa001acecb |
docs(build): diagnose the intermittent Turbopack chunk-hash collision (#4084)
Documents the root cause of the intermittent 'Two or more assets with different content were emitted to the same output path' build failure, and leaves a pointer comment at the one config flag that would mitigate it.
Root cause: a birthday collision in Turbopack's 7-character server chunk-name hash. The build emits ~24.5k server chunks; the first hash character is bounded in practice to {0,1,2}, leaving roughly 5% of nominal hash space, with ~11.7k chunks in the largest namespace.
Ruled out by experiment: the shared BuildKit cache mount. The exact failing tree reproduces the failure locally with no Docker, no cache mount and no concurrency, twice, identically.
Note on the error text: the two hashed files it prints are Turbopack's debug dumps, not the collision; the colliding path is the line above. '2 errors' is one collision reported twice (chunk plus source map).
No code fix here. The only lever that attacks the mechanism (turbopackServerSideNestedAsyncChunking, ~-71% chunks, ~9x fewer collisions) is broken on 16.3.0 and is deliberately disabled for a peak-memory cost that is unverified.
Anti-trap note recorded in the doc: a Next 16.3.1 bump makes the failure go away by re-rolling which chunk pairs collide, not by changing collision probability. Since another PR already lands 16.3.1 for unrelated reasons, that bump will look like the fix and the failures will return.
|
||
|
|
23e706a225 | Merge branch 'main' into moderator-app-pages | ||
|
|
03979077a0 |
perf: sever client→server import bridges in the _app graph (#3802)
Removes 95 modules from _app's client module graph: no Prisma client, no redis client, no server env schema compiled into the browser. Adds a convention guard (test:lint-rules) that fails if _app's graph reaches server infrastructure again, printing the offending import chain. Also fixes a pre-existing durable-logout path: _app no longer deletes the auth cookie on an ambiguous session: null, since getServerAuthSession fail-softs and a hub outage is indistinguishable on the wire from an expired token. Adds the regression test that path never had. Measured negative, stated in the PR: module-graph size is NOT a meaningful lever on peak builder RSS (-2.8% for ~7% of the graph). This does not buy back the server-chunk regression from #3807. Reviewed adversarially in four rounds; ~15 findings addressed. |
||
|
|
c771513011 |
fix(build): bring release-build peak memory back under the 40Gi builder limit (#3807)
* perf(build): stop nested-async chunking the server build
The flag trades peak builder memory for smaller server chunks: ~72% fewer
emitted server chunks and roughly half the server chunk bytes, in exchange for
~+8% CI build time and ~+33% peak builder RSS (numbers from #3458, recorded in
the comment this replaces).
That trade was accepted when nothing measured build RSS. It now has to be paid
against an enforced 40Gi builder cgroup, and the release build OOMKilled three
times at 37-39 GiB. Turning it off is a real regression in emitted chunk size,
not a free win, but a build that cannot complete is worth less than smaller
chunks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(build): pin turbopackFileSystemCacheForBuild to the pre-16.3.0 default
Next 16.3.0 flipped this experimental flag's default to true, so leaving it
unset silently changed the production build. `turbopack-build/impl.js` derives
`dependencyTracking` from it, which changes what turbo-tasks retains in memory,
not only what is written to `.next/cache`.
The 16.3.0 upgrade (
|
||
|
|
1d513423b5 |
Merge origin/main into moderator-app-pages
54 commits. Three conflicts: - `package.json` — both sides added a workspace dependency (`@civitai/moderation` here, `@civitai/flipt` on main). Kept both. - `pnpm-lock.yaml` — took main's and regenerated with `pnpm install --lockfile-only` against the merged package.json rather than hand-merging it. - `src/pages/moderator/reports.tsx` — modify/delete. Our delete stands: the page is migrated and `reports` is already in the MIGRATED_ROUTES redirect in `src/pages/moderator/[...slug].tsx`. Main's schema changes left the generated Prisma client stale (`userCosmeticShopItemResale`, `sfwBio`), so `pnpm run db:generate` was re-run; the regenerated output is included. Root typecheck 0 errors, apps/moderator typecheck 0 errors. ⚠️ The deleted page was not just deleted on main — `159f535f7a` ADDED a capability to it: `RemoveReportedPlacement`, the removal path for the new `StickerPlacement` report reason. The migrated `/reports/[slug]` in apps/moderator knows the reason (it is in `reportReasonLabels`) but has no placement removal action, so that capability is currently absent from the spoke. Recorded rather than ported here — it is a feature, not a merge resolution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
d9ffacb259 | Merge branch 'main' of https://github.com/civitai/civitai | ||
|
|
0f5f28b90d |
feat(packages): add @civitai/flipt and wire creator-studio to it
Extracts the monolith's Flipt client into a shared workspace package so other apps can gate on the same flags. The package carries over the production hardening as-is — init timeout, failure circuit breaker, generational TTL eval cache, dev-only local overrides, fail-closed on every path — and follows the @civitai/axiom / @civitai/redis shape: raw TS entry, package-owned lazy env, injected logging. Flag keys stay in the app. FLIPT_FEATURE_FLAGS and the cache-bypass set remain in src/server/flipt/client.ts and are passed to the factory as `cacheBypass`; which flags are incident kill-switches is app knowledge, not package knowledge. Connection env resolves on the first evaluation rather than at construction, so a missing FLIPT_URL degrades that instance to fail-closed instead of throwing out of the import that built it. Building it eagerly broke 61 unit suites at collection, whose env mock has no Flipt vars. creator-studio gets a getFlipt() shim mirroring its redis.ts/logger.ts (lazy, globalThis-cached, app logger injected), the transpile + dep entries, documented env, and a test pinning the wiring — nothing else in the app imports the shim yet, so without it neither svelte-check nor a build would prove it resolves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f108d51493 |
fix(prisma): externalize @prisma/client so $queryRaw sees one Sql class
`Prisma` imported from `@prisma/client` was bundled into the app layer while
`dbRead`/`dbWrite` reached a second copy through the transpiled
`@civitai/db-schema`. `$queryRaw` identifies its tagged-template argument with
`instanceof Sql`, so a `Prisma.join()` built by the other copy failed that check
and was bound as a plain value, producing:
42883 operator does not exist: integer = jsonb
This 500'd every `homeBlock.getHomeBlock` on the homepage feed, and was latent
for the ~20 other server modules that use `Prisma.join`. `require.resolve`
reports a single physical path from every workspace dir, so the duplication is
only visible at the bundler layer.
`@prisma/client` is already in Next's built-in serverExternalPackages defaults;
listing it explicitly is what stops `transpilePackages` from pulling it back in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
e27f4a72ee |
Merge remote-tracking branch 'origin/main' into moderator-app-pages
# Conflicts: # CLAUDE.md # apps/moderator/CLAUDE.md |
||
|
|
30e32db0a9 |
feat(telemetry): emit structured logs over OTLP alongside Axiom (dark by default) (#3736)
* feat(telemetry): emit structured logs over OTLP alongside Axiom (dark by default)
Adds a third sink to the single `logToAxiom` chokepoint: the same record is
handed to the OpenTelemetry Logs API in addition to the existing stderr write
and the Axiom ingest. Neither existing sink changes behaviour.
Ships INERT: `emitOtelLog` no-ops unless `OTEL_LOGS_ENABLED === 'true'`, which
nothing sets. Merging this changes nothing observable; enabling it is a config
change per workload, not a code deploy.
The bridge (`@civitai/telemetry/otel-logs`)
- Injected into `createAxiomLogger(overrides, deps)` rather than imported by it,
so `@civitai/axiom` keeps no OpenTelemetry dependency and no OpenTelemetry
semantics. Its own env.ts header already prescribed this shape.
- ONE serialization, N sinks: the line is built once and the OTLP body is that
exact string, so a query written against either path matches the other and the
hot path pays for one JSON.stringify. This refactors the three-branch stringify
catch ladder from three console.error calls to assign-once/write-once; a test
pins each branch.
- Severity: `type` is a hybrid field (mostly a severity word, sometimes an event
name, sometimes absent), so only the four clear cases map. Everything else is
UNSPECIFIED(0) with severityText OMITTED — downstream level resolution prefers
text over number, so an invented string would outrank a correct number. The raw
value always survives as the `civitai.type` attribute, so nothing is lost. A Map,
not an object literal: an object lookup with a caller-supplied key fails open on
inherited keys like `toString`.
- Attributes are a CLOSED set. Spreading the payload would reproduce the schema
explosion `safeError` exists to prevent. The payload is already in the body.
- The logger is resolved LAZILY per emit and only once a provider is registered.
`logs.getLogger()` called before registration returns a proxy that is never
delegated and stays silent forever, with no error — a module-scope getLogger is
the bug, not a style choice.
- Counters make a zero legible: a bare "no records arrived" cannot distinguish a
disabled flag from an unregistered provider from a broken emit. Emitted and
skipped{disabled|no_provider|emit_threw} separate them.
- Emission can never throw to the caller: logToAxiom is awaited on hot paths.
SDK correctness
- Batch processor: `exportTimeoutMillis` made explicit (env-tunable) because it is
the one value coupled to something outside the process — a hung exporter must
not hold the shutdown flush past the termination grace period. The default is
sized for the shortest grace period any of these workloads runs with.
- Shutdown was SIGTERM-only and awaited NEITHER promise, so the final batch raced
termination — immaterial at one record per boot, a real loss path once it carries
traffic. Now SIGTERM + SIGINT, awaited, idempotent across a second signal, and
allSettled so one failing exporter cannot block the other flush.
- Provider construction moved into the package so the `traceBased` hazard has one
home and a regression test. `traceBased: true` would drop every record emitted
inside an unsampled span; with head sampling below 1.0 that is most in-request
records, silently. It is inert at SDK defaults and must stay that way.
- `serverExternalPackages` gains the four logs-pipeline packages so the Logs API's
globalThis provider registry is written and read by one module instance.
`@opentelemetry/resources` and `semantic-conventions` are deliberately excluded —
the web tracing SDK reaches them from the client graph.
Tests
41 tests across the two package suites, using the real SDK — no module mock, which
would make the assertions vacuous. Each guard was watched to fail for its own
reason: 14 mutations, each killed by the specifically-named test that owns it.
The bundling failure is structurally invisible to Vitest (it aliases the package to
source and never runs the bundler). The runtime check for it is
skipped{reason="no_provider"} being non-zero in a real process; a build-time
module-identity scan is the follow-up.
* fix(telemetry): drop the serverExternalPackages additions from this PR
The four logs-pipeline entries (@opentelemetry/api, /api-logs, /sdk-logs,
/exporter-logs-otlp-proto) failed the image build. That change needs a full
`next build` as its gate, which makes it its own PR rather than a rider on the
bridge.
The bridge does not depend on it. It resolves the Logs API lazily, per emit, so
a second bundled module copy is survivable rather than fatal, and
skipped{reason="no_provider"} is the runtime signal if one ever shows up. The
list is now byte-identical to main; only a comment recording this is added.
* fix(telemetry): register the OTel sink from the server entry, not the shared shim
`src/server/logging/client.ts` is in the CLIENT bundle. It reads as server-only
and is not: _app.tsx -> server/services/system-cache -> server/redis/fail-open-log
reaches it, so a module-scope import there is bundled for the browser too.
Importing the bridge from it pulled prom-client into the browser graph and failed
the image build with `Can't resolve 'cluster' / 'fs' / 'v8'`. Nothing local caught
it: typecheck, eslint, and both vitest projects were green, because only the
bundler walks that edge.
The sink is now REGISTERED rather than imported — instrumentation.node.ts, which
only ever runs on the server, calls setStructuredLogSink(emitOtelLog) at boot. The
shim keeps a type-only import, which is erased and adds no runtime edge.
This works because deps.emitLog is read per call rather than captured at
construction. That was incidental before and is now load-bearing, so a test pins
it: registering a sink AFTER the logger is built must still fire. Mutating the
factory to capture at construction fails that test and only that test.
|
||
|
|
29ce14c96a |
Merge remote-tracking branch 'origin/main' into moderator-app-pages
# Conflicts: # packages/civitai-db-schema/prisma/schema.full.prisma # packages/civitai-db-schema/src/models.ts # pnpm-lock.yaml # src/pages/moderator/image-rating-review.tsx # src/pages/moderator/image-tags.tsx # src/pages/moderator/reports.tsx # src/server/services/cosmetic.service.ts |
||
|
|
d1b3241814 |
chore(deps): upgrade to Next 16.3.0 and stop paying nested-async chunking in dev (#3742)
* perf(dev): stop paying server nested-async chunking in dev The flag exists to shrink emitted server chunks in a production build. A dev server never emits those chunks, so the walk over dynamic-import paths is pure cost there, on a module graph `_app` already makes large. Keeps the build-time behaviour from #3458 unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(deps): upgrade next to 16.3.0 16.3.0 is the first release carrying vercel/next.js#93788, which fixes a lock-order inversion in turbo-tasks-backend and removes an incorrect `unsafe impl Send` on dash_map_multi::RefMut that let a caller hold a shard write guard across an await. The symptom is a dev server that pegs most of the cores with flat memory and never finishes compiling a route, recoverable only by restart. Verified by reading the file at each tag: the unsound impl is present through 16.2.12, the last 16.2 release, and gone in 16.3.0. It was never backported, so no 16.2.x can pick it up. The `get_multiple_mut` equal-keys assert still exists in 16.3.0, so this is not expected to silence that panic — only to stop a panicking worker from wedging every other worker behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: commit the agent-rules block next dev writes into CLAUDE.md `next dev` on 16.3.0 appends a `nextjs-agent-rules` block to CLAUDE.md and re-adds it every run (next/dist/server/lib/generate-agent-files.js), so the choice is to commit it or carry an uncommitted change forever. It earns its place here: it points at `node_modules/next/dist/docs/`, which is already the authority this repo cites for Turbopack defaults, and the warning is aimed squarely at a pages-router app on Next 16. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a86a39e481 |
chore(dev-server): spoke apps, branch-switch survival, and per-worktree env (#3741)
* feat(dev-server): run the SvelteKit spoke apps as daemon sidecars The apps under apps/ each need `vite dev` in their own directory so vite picks up that app's .env, plus readiness parsing, crash handling and a log buffer. AuthHub already encodes all of that, so a spoke is an AuthHub with a label. Ports are fixed per app rather than auto-assigned so a redirect between two of them (moderator -> auth) always lands in the same place, and --strictPort fails loudly on a collision instead of drifting to the next free port. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(dev-server): keep the dev server alive across branch switches Switching branches in place meant deleting .next and reinstalling, or the server would hang. The daemon now watches HEAD and handles the switch. It does NOT kill the dev server to do it. Measured on this repo, same routes, same machine: cold start after a kill ......... /models 42.6s left running across a switch .... /models 7.7-9.2s, /images 1.0-1.6s The running process keeps its in-memory module graph, so only what actually changed recompiles, and routes it isn't asked for stay warm. A restart is forced only when pnpm-lock.yaml or schema.full.prisma changes, since node_modules and the generated Prisma client can't be swapped under a live process. KILL_ON_BRANCH_SWITCH=true restores the old behaviour. Also adds: - PREWARM_ROUTES, compiled in the background on start and after each switch, so the ~45s first-route compile lands on the daemon rather than on your next click. Detail pages are separate routes from their list pages and need their own entry. - PER_BRANCH_DIST_DIR (default off) via distDir in next.config.mjs, with LRU eviction against both a count and a size budget. Off because a shared .next lets unchanged modules stay valid across branches, and each per-branch dir grows to ~4GB. - TUI session switching (s/Tab) plus a session counter, for running several worktrees side by side. - defender-exclusions.ps1: Windows real-time scanning over ~210k node_modules files is a large chunk of the cost on this machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(dev-server): start a session from its own worktree's .env The env path fell back to the daemon's project root, so every session in every worktree ran on whichever tree happened to launch the daemon. A branch that adds a flag or points at a different database silently ran without it, and a NODE_OPTIONS set in a worktree .env never reached the process at all. Prefer the session worktree's .env, falling back to the project root only when the worktree has none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(dev-server): stop orphaning spokes, and say which .env a session loaded Review findings on this branch. Spoke apps were started but never stopped: /shutdown, SIGINT and SIGTERM each tore down sessions, the rgb proxy and the auth hub, and left the spokes running. Since spokes bind with --strictPort, one survivor makes the next start of that app fail EADDRINUSE against a process the daemon no longer tracks. The env-path change in 04114ab9 picks between two files that are not merged, so it can point a session at a different database with no receipt. Log the chosen path at start and return it as `envPath` in session status. Docs corrected where they described behaviour the code does not have — the worktree/root .env precedence, PREWARM_ROUTES having a default (it is empty, and prewarm-on-start needs HEALTH_CHECK_URL), and skill .env edits reaching a session that a branch switch deliberately does not restart. .env.example gains the ten knobs it was missing, and defender-exclusions.ps1 takes -ReposRoot instead of hardcoding one machine's layout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c451c702f1 |
Merge origin/main into moderator-app-pages (round 2)
Main advanced 11 commits after the previous merge, re-conflicting the PR. Brings it current again. Conflicts, same migration-seam pattern plus the db-queries PR (#3575) main just merged: - cosmetic.schema.ts / cosmetic.router.ts / cosmetic.service.ts: kept the branch's removal of grantCosmeticsToUsers/revokeCosmeticsFromUsers (moderator cosmetics grant -> spoke) and kept main's NEW stickers feature (getStickerCosmetics + getStickerCosmeticsSchema). Reconciled imports to the auto-merged router body, which uses only the sticker path. - package.json: kept all three workspace deps (@civitai/db-queries + @civitai/db-schema from main, @civitai/moderation from the branch). - pnpm-lock.yaml: took main's and reconciled via `pnpm install`. db-schema regeneration: main's db-queries PR restructured @civitai/db-schema. Ran db:generate (slim-schema + prisma generate) to make the generated Kysely types consistent, and dropped a stale `BaseModelLicensingFee` entry from updated-at-tables.ts (the model was removed on main; that generated file is stale on main too and its CI regenerates it). Verified: main app `tsc --noEmit` exit 0; spoke `svelte-check` 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
86c77399e1 |
Merge origin/main into moderator-app-pages
Brings the branch current with main (491 commits). Three code conflicts, all in the moderator-migration seam where main kept developing handlers the branch moved to the spoke: - article.router.ts / article.service.ts: kept the branch's removal of the article rating-review MODERATOR handlers (getArticleRatingReviews, getArticleRatingReviewCounts, resolveArticleRatingReview — now spoke-owned), and kept main's NEW article-image-scan feature (resolveArticleImageScan, rescanArticleImage + their schemas). The auto-merged router body already used only the image-scan procedures, so the imports were reconciled to match. - routers/moderator/index.ts: kept main's new imports (getModelChangeHistory, getImagesModRules) for its added moderator procedures. Two semantic fixes the auto-merge left dangling (main added consumers of things the branch had removed as "orphaned"): - Restored image.service `resolveIngestionError` — main's new resolveArticleImageScan reuses it. getIngestionErrorImages stays removed (still spoke-only, no caller). - Restored the `moderatorProcedure` import in article.router.ts for main's new resolveImageScan procedure. Verified: main app `tsc --noEmit` exit 0; spoke `svelte-check` 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
62a00997ea | Merge branch 'main' into feat/db-queries-kysely | ||
|
|
0801071370 |
perf(build): enable Turbopack server-side nested async chunking (-316.7 MiB of server chunk source) (#3458)
* perf(build): enable Turbopack server-side nested async chunking Next's own default table has `turbopackClientSideNestedAsyncChunking` defaulting to TRUE in build mode, but `turbopackServerSideNestedAsyncChunking` defaulting to FALSE in both dev AND build. The server build therefore never received the async-chunk dedup the client build already has, so every async chunk group re-emits its entire module graph. Measured on the production server build: emitted chunk files 21,247 chunk source bytes 587 MB distinct modules 7,844 module copies emitted 1,176,346 (~150 copies per module) duplicated 535.8 MB (96.7% of chunk bytes) Enabling the flag on the same commit: chunk files 21,247 -> 5,830 (-72.6%) chunk bytes 587.4 MB -> 255.3 MB (-316.7 MiB, -56.5%) page entries 548 -> 548 (unchanged) build wall time 228s -> 312s (+37%) Runtime risk is low: the Turbopack runtime loader installs a module factory only `if (!moduleFactories.has(id))`, so duplicate copies are already registered-then-discarded today. This changes what is emitted, not module identity - no singleton or `instanceof` semantics change. Eager chunk sets are byte-identical before and after, so process startup work is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(build): trim the chunking comment to the mechanism, drop unverified aggregates An adversarial audit could not reproduce three of the figures baked into the config comment (7,844 distinct modules / 1,176,346 module copies / ~150 copies per module), and the "96.7% of chunk bytes duplicated" claim does not cohere with the other numbers in the same block: 535.8 MB is 91.2% of 587.4 MB, and 96.7% only holds against module-source bytes rather than chunk bytes. It also measured the real cost far lower than the comment claimed. The scoping pass's "+37% build wall time" compared against an anomalously fast local baseline. The actual CI build of this commit took 312.8s against a ~290s baseline median (+7.9%); a cold same-commit local A/B gave +13% wall / +19% compile. Keeping the mechanism and the shape of the trade in the config, and moving the measurement table to the PR, where it can be corrected without a code change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
01ad026c66 |
Merge origin/main into moderator-app-pages
Brings the branch up to date with main (321 commits), incl. the canonical @civitai/shared package. Conflict resolutions: - @civitai/shared: took main's canonical package (browsing-levels + flags + index); the spoke keeps its allBrowsingLevelsFlag import (now main's Blocked-excluding semantics), per instruction. - next.config.mjs: serverExternalPackages took main's (adds @pyroscope/nodejs + @datadog/pprof); the auto-merged transpilePackages keeps both sides. - image.controller.ts + routers/moderator/index.ts: kept main's added imports. - pnpm-lock.yaml: regenerated via pnpm install against the merged package.json. Verified: main app `tsc --noEmit` exit 0; spoke `svelte-check` 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
43b75f54d9 |
feat(shared): add @civitai/shared package (Flags + browsing-levels)
New client-safe, dependency-free package for cross-app pure constants + utilities, wired into the main app (transpilePackages + workspace dep). First shared-package home for these primitives; app-level copies can adopt it to dedup later. - flags.ts: the Flags bitwise utility (port of the main app's ~/shared/utils/flags). - browsing-levels.ts: NsfwLevel bitwise levels + the generic browsing-level toolkit ported from ~/shared/constants/browsingLevel.constants — level groupings + flags, parseBitwiseBrowsingLevel/flagifyBrowsingLevel, labels + descriptions, severity/label helpers, and the browsing-level predicates. The App-Blocks/orchestrator/deprecated/UI/ reasons pieces stay app-specific. - allBrowsingLevelsFlag matches the main app (EXCLUDES Blocked); a distinct allBrowsingLevelsWithBlockedFlag covers moderator "show Blocked" contexts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
b96b7c7772 |
feat(db-queries): Phase-0 @civitai/db-queries foundation + reports port
Scaffold the shared Kysely data-access package and wire it into the main app, per docs/db-queries-kysely-plan.md. - @civitai/db-queries: globalThis-backed client vars + connect(), nested-read (jsonArrayFrom/jsonObjectFrom) and toJson write helpers, and the reports queries (setReportStatus / setReportStatusMany) as the reference module. - Main app: kyselyDb.ts builds clients over the existing pgDb/datapacket pools and calls connect() at boot (instrumentation.node.ts); getKyselyWithoutLag twin of getDbWithoutLag; bulkSetReportStatus repointed to setReportStatusMany. - Testing: offline compile-only harness (Kysely DummyDriver) + vitest config, reports compiled-SQL tests (incl. the empty-array IN () guard), and a two-tier testing section in the package README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
08bbfeaca5 |
chore(monorepo): add @civitai/moderation + wire buzz/email/notifications into the spoke
Foundational wiring for the moderator image-moderation slice.
- @civitai/moderation (NEW package): the cross-app moderator-action contract —
action-input zod schemas + MOD_ACTION names (schema.ts) and a no-retry
createModeratorClient (client.ts) the main app uses to call the spoke's
/api/mod/*. Producer (spoke endpoint) and consumer (main-app client) share the
same schemas so they can't drift (same pattern as @civitai/notifications).
- Declare @civitai/{moderation,buzz,email,notifications} for the spoke
(apps/moderator/package.json) + add them to vite ssr.noExternal so the raw-TS
packages transpile under the SvelteKit build.
- Add @civitai/moderation to next.config transpilePackages (+ notifications,
already imported by the spoke's tos-violation notify).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
d0a795c1d2 |
feat(profiling): Grafana Pyroscope continuous CPU profiling SDK (dark by default) (#3112)
* feat(profiling): add Grafana Pyroscope continuous CPU profiling SDK (dark by default) Adds the @pyroscope/nodejs in-process wall/CPU profiler behind a PYROSCOPE_ENABLED env gate (default OFF). Ships as a complete no-op: the SDK is dynamically imported ONLY when the flag is armed, so an unset flag never loads the package or its native @datadog/pprof addon. When armed it runs a 100Hz wall+CPU sampler on a separate thread and pushes pprof to a configurable ingest endpoint (PYROSCOPE_SERVER_ADDRESS), tagged by pod/version/pool for diff-by-version flamegraphs. Purpose: continuous server-CPU profiling to rank and book per-request CPU optimizations (the live lever for our peak-capacity-capped web tier). - src/server/pyroscope.ts (new): flag-gated dynamic init, mirrors the cpu-profiler.ts arm-time-never-crashes discipline. Wall/CPU only (no heap), collectCpuTime on. - src/instrumentation.node.ts: fire-and-forget registration by the existing profiler hooks. - next.config.mjs: externalize the SDK + @datadog/pprof (native addon) from the server bundle. - package.json / pnpm-lock.yaml: add @pyroscope/nodejs@0.6.1 (exact pin — fresh native package). - .nvmrc: 20.13 -> 20.20.2 (the SDK's node engines floor). No behavior change with the flag unset (how it ships). * harden(pyroscope): make the profile flush interval env-tunable The flush (serialize -> protobuf encode -> push) runs on the MAIN thread every flushIntervalMs (default 60s), so on a CPU-ceiling-capped pool it's a periodic on-loop blip at peak. Wire PYROSCOPE_FLUSH_INTERVAL_MS so a Stage-2 canary can lengthen it from env (no code change) if event-loop-lag p99 shows the stall is material. Still dark by default; unset = SDK default 60s. (Addresses the sole arming-time risk from the #3112 adversarial audit.) |
||
|
|
9e27404a9a |
Merge pull request #2892 from civitai/feat/notifications-monorepo-migration
Feat/notifications monorepo migration |
||
|
|
0bea99cf84 |
feat(buzz): extract @civitai/buzz server-side client; refactor buzz.service onto it
- New @civitai/buzz package: server-only buzz-service client (fetch transport +
typed per-endpoint methods + wire response types) plus the browser-safe
account-type name-map. BuzzTypes in the app is now a facade delegating to it.
- Refactor src/server/services/buzz.service.ts onto the client; behavior-preserving
(same request bodies, retries, and tRPC error mapping via mapError).
- responses.ts audited field-by-field against the .NET buzz service (source of
truth); fixes real wire-type errors, incl. CreateTransactionsResponse
({ transactions: string[]; conflicts: string[] }) and CreateTransactionResponse
(+remainingBalance).
- createBuzzTransactionMany: reconcile expected vs succeeded+conflicted and log
buzz-transaction-many-dropped when the batch silently drops transactions (e.g.
insufficientFunds), which the endpoint reports in neither array.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
927e31bfee |
feat(notifications): fold notification domain into apps/notifications + @civitai/notifications
Move the notification domain out of the monolith (and the external notification-server repo) into an in-repo app + shared package, and cut the monolith off the notification DB entirely. - @civitai/notifications (package): the stable producer/reader seam — zod schema contracts, NotificationCategory/signal constants (now the single source of truth; src/server/common/enums re-exports it), and an HTTP client (create/bulk/query/count/markRead/exists/cleanup) that POSTs to the app. - apps/notifications (app): Fastify producer + read API (shared-secret auth, internal ingress) and the fan-out poll worker, on the shared @civitai/* clients. Owns the settings-filter, base-row read, unread-count cache, replication-lag routing, and mark-read retry/queue. - Monolith cutover: notification.service, send-notifications, club.service, reaction.controller, and the admin cleanup endpoint call the package directly; deleted notifDb.ts, the notif prom gauges, getNotifDbWithoutLag, and the dead notification-cache; getClient dropped its notification instances; NOTIFICATION_DB_URL/_REPLICA_URL made optional. - @civitai/db: extracted generic createPool/createClients (getClient now wraps them) and a redis-agnostic createLagTracker (monolith db-lag-helpers + the app both build on it). Verified: monolith + app typecheck, 28 unit tests, two code reviews, and a live smoke of the producer/read path scoped to user 5. NOT a merge-and-deploy — this is a hard cutover; follow docs/plans/notifications-predeploy-checklist.md (app must be deployed with the worker gated off and NOTIFICATIONS_ENDPOINT set before the monolith ships). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f905920d5f |
feat(auth): centralized-hub login cutover — remove in-app login UI
Login moves fully to the centralized auth hub (auth.civitai.com); the main app no longer renders any login UI of its own. - Remove the in-page login UI + social-login components (LoginContent, LoginModal, EmailLogin, SignInError, SocialButton/Social/SocialLabel) and migrate every consumer (AccountsCard, IframeHost, /login, discord/link-role) to the hub popup / full-page redirect. /login is now a pure server-side hub redirect; AccountsCard is self-contained (local provider map). - Hub owns its login analytics: emit LoginRedirect to the shared ClickHouse tracker; thread the login `reason` end-to-end (popup + full-page) and re-home referral attribution off the deleted LoginContent cookie via the post-login URL. - Hub login hardening: Discord role_connections.write scope + granted-scope storage on the Account; email magic-link grace window (not single-use-deleted, so link scanners can't lock out the real click); plus-address signup block; never store provider avatars; prompt=select_account passthrough; default returnUrl; fail (not switch accounts) on a lost-session link callback. - @civitai/auth hygiene: drop single-consumer OAUTH_PROVIDERS/OAuthProvider (keep ProviderId + hubLoginUrl). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
307824d106 |
feat(auth): impersonation wiring + browser-client adoption + read-path revocation
- Impersonation goes through AccountProvider.impersonate/exitImpersonation → authClient → /api/auth/impersonate (POST/DELETE proxy); ImpersonateButton driven by session.impersonatedBy; removed the ogAccount localStorage. - AccountProvider + components call @civitai/auth/client (no inline auth fetch); reload-in-place on impersonate. - sessionClient injects isRevoked (reject revoked tokens on cache hit); sync.ts no longer leaks the raw error. - Readability: collapsed identical logout/logoutAll; flattened swapAccount to guard clauses; centralized the three JWT decoders into token-claims.ts; deduped the CookieWritable interface; shared civ-cookie helper. - next.config: transpile @civitai/auth (needed for the browser client bundle). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
552a53e27c |
Merge origin/main into monorepo-bootstrap
Brings in main's App Blocks feature (+ @civitai/app-sdk, @civitai/blocks-react). Conflicts resolved: - package.json: keep @civitai/auth (workspace) + main's @civitai/app-sdk + @civitai/blocks-react - .github/workflows/pr-check.yml: accept main's deletion - src/shared/utils/prisma/models.ts: keep our @civitai/db-schema re-export shim - db-helpers.ts / redis/client.ts / prom/client.ts: keep our package-extracted shims (ours) - oauth/authorize.ts: keep both imports (OIDC nonce + app-block client check) - pnpm-lock.yaml: reconciled via pnpm install Ported main's App Blocks infra additions INTO our extracted packages so the incoming code compiles against the package architecture: - @civitai/redis: REDIS_KEYS.BLOCKS + REDIS_SYS_KEYS.BLOCKS keys; withRedisCommandTimeout (app shim) - @civitai/db: 'apps' pool instance + APPS_DATABASE_URL/appsUrl - @civitai/telemetry: App Blocks metrics (blockBuzzAttribution, appStorageOps/QuotaExceeded/Latency) Typecheck: 0 errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
b5c7bfb8b3 |
fix(og): ship @vercel/og in standalone so /api/og stops 500ing (#2509)
* fix(og): ship @vercel/og in standalone so /api/og stops 500ing /api/og renders via next/og's `ImageResponse`, which on the nodejs runtime lazily require()s `next/dist/compiled/@vercel/og/index.node.js` (+ its resvg/yoga WASM + fonts). @vercel/nft can't follow that dynamic require, so with output:'standalone' the file is traced OUT of the image — every origin (cache-miss) /api/og render then throws `Cannot find module ... index.node.js` → res.status(500). Impact: the dominant app-emitted 500 source on dp-prod (~1.9/s, ~74% of the app-500 floor) since the Next 16.2.7 upgrade. Cloudflare edge-caching of OG images masked it for popular entities, but cache-bypass returns 500 for every type (verified: model/image/post/article). Degrades social-share/OG previews for any uncached entity. Fix: add `/api/og` to outputFileTracingIncludes with the compiled @vercel/og dir (entry + WASM + fonts). Two version-agnostic globs (no hardcoded next@hash) to be robust to pnpm symlink resolution. Verification: requires a standalone build — confirm on the PR preview that a cache-busted /api/og?type=model&id=<id> returns 200 (currently 500 at origin). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(og): give FallbackCard logo explicit width so the fallback can't 500 Verifying the standalone-packaging fix on a preview exposed a SECOND, pre- existing /api/og 500 that the module-not-found error had masked: the catch-path `FallbackCard` renders the wordmark logo as `<img height={60}>` with NO width. satori then throws "Image size cannot be determined. Please provide the width and height of the image." — so the fallback ImageResponse itself throws and the outer catch returns 500. This means ANY missing entity (e.g. a deleted/invalid id from a social crawler) or any OgCard render error hit FallbackCard and STILL 500'd, even with the module present. Verified on the pr-2509 preview: existing model/image → 200, but missing post/article/bounty/challenge → 500 with this exact error. Fix: logo_dark_mode.png is 142x30 (≈4.733), so set width={284} at height={60} to preserve aspect and satisfy satori. Now the fallback renders instead of throwing. (Other cards already set both dims; only FallbackCard omitted width.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
be0ccbc27b |
Merge origin/main into monorepo-bootstrap
Conflicts resolved:
- package.json: keep @civitai/auth workspace dep + take @civitai/client 0.2.0-beta.71
- src/pages/login/index.tsx: take main's session?.user guard (login-loop fix)
- .github/workflows/pr-check.yml: keep our lint:packages step + main's updated
no-io-in-transaction comment
- src/shared/utils/prisma/{enums,models}.ts: keep our @civitai/db-schema re-export
shims; regenerated the package's generated enums/models from the merged schema
(CsamReportType gains ExternalLink)
- pnpm-lock.yaml: reconciled via pnpm install
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
21df8012cb |
chore(config): add civitai-dev RGB origins to allowedDevOrigins
Allow civitai-dev.green/blue/red as dev origins (RGB proxy) for everyone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
ee89847977 |
Merge remote-tracking branch 'origin/main' into monorepo-bootstrap
# Conflicts: # .gitignore # src/server/prom/client.ts |
||
|
|
96f484e46e |
feat(build): ship server source maps for prod CPU-profile de-minification (#2460)
* feat(build): ship server source maps for prod CPU-profile de-minification
Prod pods capture V8 .cpuprofiles to find event-loop blockers, but the
standalone image shipped zero server .js.map files, so frames were minified
and unnameable (e.g. `p @ src_17njnbr._.js:0`).
Build change (Turbopack / Next 16):
- Under Turbopack the only source-map lever is `turbopackSourceMaps`, whose
build-time default IS `productionBrowserSourceMaps` (already true). So server
chunk maps (.next/server/**/*.js.map) are already emitted at build time;
`experimental.serverSourceMaps` is webpack-only and ignored by Turbopack.
Reworded next.config.mjs comment to state this accurately.
- output:'standalone' traces via @vercel/nft, which follows import/require/fs
and does NOT copy sibling .map files, so the maps were dropped from the image.
Dockerfile now stages just the server-chunk maps (structure-preserving tar)
in the builder and overlays them onto .next/server in the runner.
Maps are inert at runtime (loaded only by an inspector/stack resolver) -> no
request-path perf cost. Cost is build time + ~order-of-tens-of-MB image size.
Resolver tool:
- scripts/resolve-cpuprofile.mjs maps each frame's (chunk.js, line, col) back to
original {source, line, name} via the `source-map` package, ranks hottest
self-time leaves, and reconstructs the longest-synchronous-block stack, named.
- Verified end-to-end against a real esbuild-minified bundle + map (synthetic
profile frames resolved back to original fn names + .ts locations). Prod proof
awaits the next map-enabled deploy + a fresh capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(build): publish server maps as on-demand artifact, not in runtime image
The previous approach overlaid all server .js.map files into the runtime
image, adding ~761 MB to every prod pod (11,305 maps) for a debug aid that
is only needed OFFLINE by the cpuprofile resolver.
Instead:
- Dockerfile: revert the runtime-stage map COPY (runtime image is lean again).
Add a `FROM scratch AS maps` target holding ONLY the staged server maps; it
shares every builder layer (cache hit) and is published separately.
- resolve-cpuprofile.mjs: add `--image <tag-or-ref>` mode that fetches that
build's maps from ghcr.io/civitai/civitai-web-maps:<tag> via `crane export`
(falls back to `oras pull`), then resolves. Keeps the local `--maps <dir>`
mode. Maps are keyed by the exact image tag so a profile from image X
resolves against X's maps.
The Tekton build pipeline (datapacket-talos) publishes the `maps` target to
the sibling repo after the main build+push, reusing the same ghcr creds, as a
non-fatal step.
Verified end-to-end: pushed a synthetic maps image to a local registry, then
`--image` fetched + extracted it and de-minified frames to original src/*.ts
functions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(source-maps): correct stale next.config comment + clean up resolver temp dir on error
Audit follow-ups on the artifact-based source-maps PR:
- next.config.mjs: the comment still described the pre-revision behavior (maps
baked into the runtime image). Reworded to reflect the on-demand maps artifact.
- resolve-cpuprofile.mjs: wrap the post-fetch body in try/finally so the fetched
maps temp dir (hundreds of MB) is always cleaned up, even if resolution throws
(previously leaked on the --image error path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
d572773c31 |
Merge branch 'main' into monorepo-bootstrap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
6aa4d37392 |
Merge origin/main into next-16-migration (2nd)
Conflicts resolved: - next.config.mjs: kept main's `transpilePackages: ['superjson']` (superjson v2 upgrade #2425) alongside our `serverExternalPackages` and top-level `outputFileTracingIncludes`. - pnpm-lock.yaml: regenerated via `pnpm install` against the merged package.json (next ^16.2.7 preserved; superjson v2). - src/pages/api/trpc/[trpc].ts: auto-merged — kept our Next 16 withAxiom typing and main's new TOO_MANY_REQUESTS auth-skip. - src/middleware.ts: main's edits were functionally identical to our renamed src/proxy.ts; rename-aware merge kept proxy.ts (no duplicate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
92eaff16e4 |
perf(trpc): upgrade superjson 1.9.1 → 2.2.6 (#2425)
Replaces the closed devalue migration (#2410) with the lower-risk lever from the best-practice audit. We were 3 major versions behind (1.9.1 → 2.2.6, the latter actively maintained), and every benchmark that motivated the devalue swap compared devalue only against superjson 1.x — the stale version we run. superjson 2.x is the era after its serialization-perf work, so this directly tests "is the serializer the cost?" with none of devalue's downsides. Why this over devalue: - LENIENT, like 1.x — still coerces Prisma Decimal / dayjs / SDK objects / Error / class instances instead of throwing. No per-route 500-risk tax, no SSR full-page-500 surface (the recurring problem with devalue). - Drop-in: identical `{json, meta, v:1}` wire format (verified) → cross-version compatible during the rolling deploy, and the existing transformer wiring (`superjson.serialize`/`.deserialize`, default import at 3 sites) is unchanged. - ESM-only is superjson 2.x's only breaking change (Node ≥16) — handled by adding `superjson` to `transpilePackages` for the standalone server bundle (same mechanism devalue needed). No `require()` of superjson exists in src/. Verified v2: default export intact, serialize/deserialize round-trip Date/Map. Transitive deps copy-anything 3→4, is-what 4→5 are superjson's own. The higher-ceiling fix (epoch dates + no transformer on the feed path) remains a separate follow-up. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9e233a3270 |
Merge origin/main into next-16-migration
Conflicts resolved: - pnpm-lock.yaml: regenerated via `pnpm install` against the merged package.json (kept next ^16.2.7). - src/pages/api/trpc/[trpc].ts: kept main's native `allowMethodOverride: true` handler (+ responseMeta caching, onError Axiom logging) and applied the Next 16 withAxiom typing fix to the export; dropped the branch's now-redundant manual restoreMethodOverride(). - next.config.mjs: auto-merged, then moved main's `outputFileTracingIncludes` from `experimental` to top-level (it moved to top-level in Next 15; under `experimental` on Next 16 it is silently ignored, which would regress the src/static-content standalone tracing fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
3530a85f30 |
Migrate to Next.js 16 + Turbopack; fix Turbopack-dev runtime regressions
Build/deps/config:
- next, eslint-config-next, @next/bundle-analyzer → ^16; Turbopack is the
default build bundler. Added turbopack:{} config, dropped removed config keys.
- Removed all 151 typed-scss-modules *.module.scss.d.ts in favor of ambient
declarations in src/types/css-modules.d.ts (Turbopack panics on the .d.ts).
- CSS modules: rewrote bare :global to function form for Turbopack.
- trpc/[trpc] handler typing; bundler-agnostic applySourceMaps.
Runtime fixes (Turbopack dev/prod):
- CAConsentManager: static-import instead of next/dynamic. A dynamic component
wrapping the whole app async-loads its chunk under Turbopack dev, causing a
whole-tree hydration mismatch that re-mounts the app and orphans the server
DOM (the "double layout"; US:CA-gated). General rule: wrapper dynamics remount,
leaf dynamics are fine.
- ReactQueryDevtools: load client-only (ssr:false) — not SSR-safe under Turbopack.
- SharedWorkers (signals + civitai-link): Turbopack doesn't compile .ts
SharedWorker entries (vercel/next.js#74842), breaking dev AND the prod build.
Pre-bundle each with esbuild to public/workers/*.js (scripts/build-workers.mjs,
wired into predev/prebuild) and instantiate via static path.
See docs/next-16-migration.md for full details and remaining open items.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
|
4d9ec59205 |
fix(content): drop duplicate outputFileTracingIncludes
Briant landed the same standalone static-content fix in |
||
|
|
d48a46b0af |
fix(content): bundle src/static-content into standalone output
The content endpoints (content.get / checkTosUpdate tRPC procedures and the
/content/[[...slug]] and /safety pages) read markdown from src/static-content/*.md
at runtime via fs (resolve('src/static-content/...')). Next output-file-tracing
cannot follow a dynamically-built fs read, so the .md files were only ever
included in standalone builds incidentally.
A lockfile regen (
|
||
|
|
bb605de707 |
@
fix(build): ship src/static-content markdown in standalone output Several entry points read markdown from src/static-content at runtime via fs (dynamic string paths @vercel/nft cannot trace). With output: standalone the build only ships traced files, so the markdown was missing from the deployed image and every read hit ENOENT -> 500/404 (/safety, /content, and the site-wide checkTosUpdate TOS check). Worked locally only because the full source tree is present. Add experimental.outputFileTracingIncludes keyed by every read site so the files are force-included into the standalone bundle. On Next 14 this key lives under experimental (top-level only takes effect in Next 15). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> @ |
||
|
|
c8b8c8202d |
Fix all Sass deprecations surfaced by Next.js 15
Next 15's sass-loader uses Dart Sass's modern API, which surfaces
pre-existing deprecation warnings loudly. Fix them at the source instead
of silencing, with output verified byte-identical (or functionally
identical for mixed-decls):
- CosmeticLights.module.scss: darken() -> color.adjust($lightness),
abs() -> math.abs(), slash division -> math.div(); add @use sass:color
/ sass:math.
- Burger.module.scss: round(calc($x / n)) -> math.round(math.div($x, n));
add @use sass:math.
- gift-cards/index.module.scss: mixed-decls — wrap the trailing
`color: white` (after nested rules) in `& { }`. Same selector, same
computed style.
- next.config.mjs: drop the temporary sassOptions.silenceDeprecations.
Verified by compiling all 179 src/**/*.scss: zero deprecation warnings
remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
4fdb431fba |
Migrate to Next.js 15
Bump next 14.2.28 -> 15.5.19 (Pages Router app). Also bumps eslint-config-next and @next/eslint-plugin-next to 15.5.19, and eslint 8.22.0 -> 8.57.1 (Next 15's plugin rules use context.filename, added in ESLint 8.40). Config / API changes required by Next 15: - next.config.mjs: experimental.serverComponentsExternalPackages -> top-level serverExternalPackages; removed the now-stable experimental.instrumentationHook flag. - .eslintrc.js: added settings.next.rootDir so the Next ESLint plugin can locate the pages dir (otherwise the rules crash on load). - bot-detection.middleware.ts: dropped request.ip (NextRequest.ip removed in 15); cf-connecting-ip / x-forwarded-for fallbacks remain. - api/trpc/[trpc].ts: assert the default export as NextApiHandler. Next 15's route-type validator rejected it because withAxiom's NextConfig overload (declared first) structurally matches an arrow function. - Moved api/auth/oauth/__tests__/token.test.ts -> server/oauth/__tests__/token-endpoint.test.ts. Next 15 type-validates every file under pages/api as a route; the colocated test had no default export. Tests still pass (8/8). - Deleted the unused src/app/layout.tsx stub (whole app is Pages Router); removes the spurious "i18n unsupported in App Router" build warning. Build memory + Windows local-build helpers: - package.json build:dev / build:analyze heap 8192 -> 16384 (the app needs >8GB to compile). - scripts/graceful-fs-patch.cjs: optional NODE_OPTIONS --require preload to mitigate Windows EMFILE during local builds (no-op on Linux/CI). Validated: typecheck clean; lint at parity with main; build compiles, type-checks, passes route validation and page-data collection. The full production build's "Collecting build traces" step is FD-bound on Windows (EMFILE) and should be run on CI/Linux. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |