Files
civitai__civitai/scripts/build-workers.mjs
T
Manuel Emilio Urena 0f62920179 feat(model-files): dedupe model resources — official-resource linking + backfill (#2869)
* docs: design for linking existing files into a model version (hash dedup)

Worked-out design for ClickUp 868k376wr "Dedupe model files by hash in
'Link to existing model'". Extends the existing RecommendedResource
linked-component system to file granularity (own files only), with SHA256
as the dedup identity layer and allow-delete/stale-on-read lifecycle.
Forward-only; retroactive byte reclaim flagged as an open question.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(model-files): dedupe model resources — official-resource linking + backfill

Server linking mechanism (used by the backfill and Subtask B):
- addLinkedComponent accepts an explicit targetFileId (owner-authz'd, FORBIDDEN
  on non-owners) instead of only auto-picking the primary file, dedupes per
  (sourceId, resourceId, fileId), and optionally deletes a validated
  replaceFileId to reclaim its bytes (must be a component file on the edited
  version, never a primary model file).
- add constants.system.officialUserId (12042163); export inferComponentType.

Discoverability (Meili "Link to Existing Model" picker, scoped to modelVersion):
- new "Official" tab surfacing CivitaiOfficial canonical resources.
- relax the base-model constraint on the Official/Mine tabs — in both the Meili
  query and the client-side version filter — so a creator can reuse a component
  across base-model variants.

A.2 backfill (admin, dry-run default): link official checkpoints' bundled
component files to their standalone official model and reclaim the dupe;
matching validated against prod (Z Image -> Flux.1-AE / Qwen3).

The self-serve own-files picker from the original design was cut: it would have
exposed draft/unscanned files, and the linked-component download path gates only
on the host model's status — a moderation bypass. See docs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(model-files): add Subtask B implementation spec (dedupe-by-hash)

Full spec for Subtask B of the Dedupe Model Resources epic: client
prevent-the-upload (B.1a, 5GB worker-hash cap), server post-scan safety
net (B.1b), hourly official-uploads reclaim job (B.2), and the required
read-path staleness fix. Ships on the same branch/PR as Subtask A.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(model-files): fix Subtask B helper contract — host-side weights guard

The canonical official file is frequently type='Model' (a standalone
VAE/encoder's primary file), so it must NOT be type-filtered. Move the
primary-weights guard to the host side and derive componentType from the
host file's type, not the canonical's. Set client hash cap to 5GB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(model-files): add Subtask B implementation plan (7 TDD tasks)

Task-by-task plan for dedupe-by-hash: official-file lookup helpers, tRPC
surface, read-path staleness fix, B.1b post-scan safety net, B.2 hourly
reclaim job, client SHA256 worker/hook, and B.1a upload-prevention wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(model-files): B.1a hashing = WebCrypto + jsSHA (Justin's approach)

Header-metadata dedup ruled out: verified against real S3 bytes that
official safetensors carry only {"format":"pt"} in __metadata__, no
hash (DB headerData is unreliable). Keep B.1a with client full-file
SHA256 = byte identity = stored ModelFileHash.SHA256. Swap the worker
impl from hash-wasm to native WebCrypto (<=1GB) + jsSHA streaming (>1GB);
5GB cap is now a time guard. Reference computeSHA256InChunks hash-chains
and must not be used.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(model-files): official-file lookup helpers (size + hash) for dedupe

* feat(model-files): tRPC queries for official file size/hash lookup

* fix(model-files): drop stale linked components (deleted source) on reads

* feat(model-files): B.1b post-scan dedup of official-matching uploads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(model-files): B.2 hourly job rehoming non-official dupes onto official files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(model-files): client SHA256 worker + hook for official-match dedup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(model-files): B.1a prevent upload when bytes match an official file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(model-files): B.1a intercept falls through to normal upload on error

Wrap resolveOfficialMatch in try/catch so network errors treat match as
null and fall through to upload. Wrap the mutateAsync+enrichment block in
try/catch so a server 500 shows an error notification and falls through
instead of freezing the file in isPending state.

Also fix import order (components → hooks → utils) and add the missing
expect(findByHash).not.toHaveBeenCalled() assertion to the
primary-weights-host test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(model-files): use correct showErrorNotification shape in B.1a intercept

* fix(model-files): correct SHA256 hash case (C1), B.2 loop-until-drained (T5), generation linked-VAE staleness (I2)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(model-files): dedicated linkOfficialFileByHash endpoint for B.1a (C2/I1)

* feat(model-files): show a checking indicator during B.1a official-match

On a size collision, B.1a hashes the file (seconds for large files) and
does a server round-trip before it can skip the upload — with no visual
cue. Add an onHashStart callback fired only when hashing actually begins
(not on the fast size-gate), set an isCheckingOfficial flag on the file,
and render a spinner + 'Checking for an existing copy on Civitai…' in the
FileCard. Cleared on every exit (link, no-match, over-cap, error).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(model-files): match official file by SHA256 alone; componentType from official identity

The host file's declared type no longer gates the match — a byte-identical
file labeled 'Other' (or mislabeled) still matches. componentType is now
derived from the official file's true identity (its standalone model type,
then its own file type), not the user's label, so a VAE isn't stamped
'TextEncoder' just because the uploader mislabeled it. An official checkpoint
match yields no componentType and is never linked (only accessories are).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(model-files): check main-section files for dedup too (close bypass)

The primary-type gate let a user skip the dedup check by dropping an
official file in the main file section (typed as Model). Remove that gate
from the client hash check and the official lookup so the file's bytes —
not its declared type — decide. B.1a now prevents the upload for a
main-section file that matches an official accessory (pointer, no
replaceFileId). A genuine checkpoint match links nothing (no componentType)
and uploads normally. B.1b still skips primary-typed files, since
addLinkedComponent refuses to delete primary weights.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(model-files): render ellipsis in B.1a checking indicator

A lint autofix had rewritten the literal … in the JSX text node to the
escape …, which JSX text does not process — so it rendered literally
as 'Civitai…'. Use the &hellip; HTML entity, which JSX decodes and
the escape autofix leaves alone.

* fix(model-files): address Copilot review — primary-weights consistency + hardening

Use primaryModelFileTypes (Model/Pruned Model/Diffusion Model/UNet) as the
single source of truth for 'never dedup/delete primary weights', which was
applied inconsistently:
- official-file.service: an official Diffusion Model/UNet file was treated as
  a linkable accessory (inferComponentType maps them non-null). After removing
  the host gate, B.1a could have linked a user's Flux/Wan primary weights and
  gutted the model. Now excluded.
- dedupe-official-uploads (B.2): SQL now excludes Diffusion Model/UNet host
  types too, so the job can't delete Flux/Wan primary weights.
- addLinkedComponent replaceFileId guard: rejects all primary-weights types
  (not just Model/Pruned Model) and Training Data.
Also: reject a targetVersionId that mismatches the targetFileId's parent
version (avoids inconsistent denormalized data); fix a stale Mine/Official
tab comment; correct the spec's SHA256 casing (stored UPPERCASE, not lowercase).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(model-files): cleanup from final A+B quality review

- Static import of findOfficialFileByHash (no cycle exists; drop the defensive dynamic import).
- Move findOfficialFilesBySizeHandler router -> controller (match every other handler).
- Derive B.2 job's primary-weights SQL exclusion from primaryModelFileTypes via Prisma.join (no hand-copied list).
- Extract toLinkedComponent() — dedupe the two identical LinkedComponent enrichment sites.
- Extract skipBaseModelForOwnTabs() — shared by the Meili filter + client filterVersions (was copy-pasted).
- Tests: JSON.stringify(where) -> toMatchObject; drop a redundant no-op test; assert the B.1b error path is actually entered.
- Comment hygiene: drop file-path headers, caller-narration, name-restatements, and (I1) tracker refs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(model-files): findOfficialFilesBySizeHandler follows the handler convention

Accept { input, ctx } (typed like the other model-file controller handlers) with
async + throwDbError, and wire it directly on the router query instead of the
inline ({ input }) => handler(input) wrapper.

* fix(ResourceSelectModalContent): reorder tabs array for improved user experience

* docs(model-files): reduce link-existing-files docs to how-it-works + Subtask C

A and B are shipped, so their implementation detail (file:line refs, code,
TDD steps, status logs, file-by-file/testing sections) no longer belongs in
the docs. Rewrite the design doc as a conceptual explanation of how linking
works, describe A/B at a high level, and keep Subtask C (the remaining
stipulated work). Delete the B spec and B plan (pure implementation
artifacts for completed work; retained in git history).

* docs(model-files): rewrite link-existing-files design as conceptual how-it-works

Drop the implementation detail (file:line refs, code, status logs,
file-by-file/testing sections) now that A and B are shipped; explain how
linking works conceptually, summarize A/B, and keep Subtask C (remaining
work).

* refactor(model-files): drop internal subtask labels (B.1a/B.1b/B.2/A.2) from comments

Rewrite comments to describe the actual behavior instead of referencing the
epic's subtask identifiers, which mean nothing to a dev coming to this code
fresh. Also rename the cryptic Axiom log names ('b1b-official-dedup',
'b2-official-dedup') to descriptive ones.

* refactor(model-files): drop hostType from findOfficialFileByHash / linkOfficialFileByHash

componentType now derives solely from the official file's own identity (its
standalone model type, then its own file type), which is authoritative — the
caller's declared type was only a last-resort fallback that never classified a
real accessory and could mislabel a non-accessory match. Removed from the
service, the endpoint schema, the B.1b caller, and the client request.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(model-files): consolidate single-fn files + boolean size lookup

- fold official-file.service into model-file.service (findOfficialFilesBySize,
  findOfficialFileByHash, OfficialFileMatch); repoint importers + tests
- fold linked-component-helpers (selectLiveLinkedComponents) into model-helpers
- rename findOfficialFilesBySize -> hasOfficialFileOfSize, return boolean via
  count() instead of an unused { id }[] payload across service/controller/tRPC/client
- clarify post-scan official-dedup comment to name the S3 delete (replaceFileId
  -> deleteFile) that actually reclaims storage

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dedupe-official-uploads): align host-type filter with processor + gate host version

The SQL host-type filter used a `NOT IN (primaryModelFileTypes)` denylist that
passed types processDedupePairs cannot rehome (inferComponentType returns null):
those rows never get a linked-component pointer, so they requery every run and,
if enough cluster at the front of a full batch, starve processable pairs. Replace
it with an allowlist derived from inferComponentType itself so the two can't drift.

Also gate the host on `hv.status = 'Published'` (not just the model) so a file on
a draft version of a published model isn't silently deleted and rehomed.

Validated on the replica: removes 3 unprocessable rows (Other x2, Negative) and 1
draft-version VAE; the 111 published rehomable candidates are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(link-existing-files): design for 30-day quarantine of replaced files

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(link-existing-files): restore endpoint is moderator-only

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(link-existing-files): implementation plan for quarantine of replaced files

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(model-file): add replacedAt column for quarantine

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(model-file): make replacedAt migration ADD COLUMN idempotent

* feat(link-existing-files): quarantine replaced file instead of deleting

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(test): alias @civitai/notifications in vitest config

Server-side vitest tests import ~/server/common/enums.ts which pulls in
@civitai/notifications; the workspace-package alias list was missing it,
breaking module resolution under vitest. Additive alias fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(link-existing-files): hide quarantined files from version file lists

* fix(link-existing-files): filter quarantined files from loadModelVersion files select

* feat(link-existing-files): daily job purges replaced files after 30d

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(link-existing-files): exclude self-row from refcount so purge job actually frees S3

processReplacedFiles keeps the ModelFile row (only sets dataPurged=true), so
deleteModelFileObject's refcount guard always found the row as a "live"
reference to its own url, silently skipped the S3 delete, and the job still
reported success — leaking bytes forever since dataPurged=true gates it out
of future runs.

Thread an optional excludeId through urlsSafeToDelete/deleteModelFileObject
so the job can exclude its own row id. Existing callers (deleteFile,
updateFile, deleteModelFileObjects) are unaffected — the param is optional.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(link-existing-files): moderator restore endpoint for quarantined files

* test(link-existing-files): restore test uses non-default priorVisibility + default-case

* fix(link-existing-files): make markFileReplaced idempotent; document getVaeFiles invariant

A second markFileReplaced call on an already-quarantined file re-stamped
priorVisibility with the current (Private) value and reset the 30-day
clock, breaking moderator restore back to the true original visibility.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(link-existing-files): remove quarantine plan doc (feature shipped)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(model-files): consolidate linking+dedup into single feature reference

Fold the how-it-works + shipped quarantine behavior into docs/features/
model-file-linking-dedup.md; drop the superseded root design doc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(model-files): remove superseded quarantine implementation plan

Superseded by docs/features/model-file-linking-dedup.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(model-files): rework linking+dedup doc as timeless reference

Drop ClickUp ref, Remaining/Subtask-C section, and A/B/shipped labels;
correct key-code file paths (official-file.service.ts was consolidated).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dedupe-official-uploads): report link failures to Axiom in aggregate

Track linked/failed counts per run and emit an error-level Axiom log when any
pair fails to link, so a systemic break (e.g. a failing addLinkedComponent
path) surfaces instead of looking like a clean no-op behind scattered per-pair
warnings. Return payload now carries found/linked/failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 18:43:43 -04:00

70 lines
2.7 KiB
JavaScript

// Bundles SharedWorker scripts to public/workers/*.js with esbuild.
//
// Why: Turbopack does not compile `.ts` SharedWorker entry scripts referenced
// via `new SharedWorker(new URL('./worker.ts', import.meta.url))` — it emits the
// raw .ts (dev) or nothing (prod build), so the browser fails with
// "Failed to fetch a worker script". See vercel/next.js#74842.
//
// Fix: pre-bundle each worker to a self-contained classic (IIFE) JS file served
// statically from /public, and instantiate via `new SharedWorker('/workers/x.js')`
// (a static path, not new URL(import.meta.url)) so Turbopack is bypassed entirely.
//
// Run via `pnpm build:workers` (also wired into predev/prebuild).
import esbuild from 'esbuild';
import { config as loadEnv } from 'dotenv';
import { existsSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
// Load .env so NEXT_PUBLIC_* values are available to inline (mirrors what Next
// does for the main bundle). Optional in CI where they may come from real env.
for (const f of ['.env.local', '.env']) {
const p = path.join(root, f);
if (existsSync(p)) loadEnv({ path: p });
}
// Replace every `process.env` reference in the bundle with a literal object.
// The workers import `~/env/client` (t3-env), which reads process.env.NEXT_PUBLIC_*
// and validates on import — there is no `process` in a worker, so we inline the
// public env and skip validation.
const publicEnv = { NODE_ENV: process.env.NODE_ENV ?? 'production', SKIP_ENV_VALIDATION: 'true' };
for (const [k, v] of Object.entries(process.env)) {
if (k.startsWith('NEXT_PUBLIC_') && v !== undefined) publicEnv[k] = v;
}
const define = { 'process.env': JSON.stringify(publicEnv) };
const workers = [
{ in: 'src/utils/signals/worker.ts', out: 'public/workers/signals.worker.js' },
{ in: 'src/workers/civitai-link.worker.ts', out: 'public/workers/civitai-link.worker.js' },
{ in: 'src/workers/file-hash.worker.ts', out: 'public/workers/file-hash.worker.js' },
];
const watch = process.argv.includes('--watch');
for (const w of workers) {
const options = {
entryPoints: [path.join(root, w.in)],
outfile: path.join(root, w.out),
bundle: true,
platform: 'browser',
format: 'iife', // classic SharedWorker — broadest support, no `type: module` needed
target: 'es2020',
define,
tsconfig: path.join(root, 'tsconfig.json'), // resolves the `~/*` path alias
legalComments: 'none',
logLevel: 'info',
};
if (watch) {
const ctx = await esbuild.context(options);
await ctx.watch();
console.log(`[build-workers] watching ${w.in}`);
} else {
await esbuild.build(options);
console.log(`[build-workers] built ${w.out}`);
}
}