mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
0f62920179
* docs: design for linking existing files into a model version (hash dedup) Worked-out design for ClickUp 868k376wr "Dedupe model files by hash in 'Link to existing model'". Extends the existing RecommendedResource linked-component system to file granularity (own files only), with SHA256 as the dedup identity layer and allow-delete/stale-on-read lifecycle. Forward-only; retroactive byte reclaim flagged as an open question. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(model-files): dedupe model resources — official-resource linking + backfill Server linking mechanism (used by the backfill and Subtask B): - addLinkedComponent accepts an explicit targetFileId (owner-authz'd, FORBIDDEN on non-owners) instead of only auto-picking the primary file, dedupes per (sourceId, resourceId, fileId), and optionally deletes a validated replaceFileId to reclaim its bytes (must be a component file on the edited version, never a primary model file). - add constants.system.officialUserId (12042163); export inferComponentType. Discoverability (Meili "Link to Existing Model" picker, scoped to modelVersion): - new "Official" tab surfacing CivitaiOfficial canonical resources. - relax the base-model constraint on the Official/Mine tabs — in both the Meili query and the client-side version filter — so a creator can reuse a component across base-model variants. A.2 backfill (admin, dry-run default): link official checkpoints' bundled component files to their standalone official model and reclaim the dupe; matching validated against prod (Z Image -> Flux.1-AE / Qwen3). The self-serve own-files picker from the original design was cut: it would have exposed draft/unscanned files, and the linked-component download path gates only on the host model's status — a moderation bypass. See docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model-files): add Subtask B implementation spec (dedupe-by-hash) Full spec for Subtask B of the Dedupe Model Resources epic: client prevent-the-upload (B.1a, 5GB worker-hash cap), server post-scan safety net (B.1b), hourly official-uploads reclaim job (B.2), and the required read-path staleness fix. Ships on the same branch/PR as Subtask A. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model-files): fix Subtask B helper contract — host-side weights guard The canonical official file is frequently type='Model' (a standalone VAE/encoder's primary file), so it must NOT be type-filtered. Move the primary-weights guard to the host side and derive componentType from the host file's type, not the canonical's. Set client hash cap to 5GB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model-files): add Subtask B implementation plan (7 TDD tasks) Task-by-task plan for dedupe-by-hash: official-file lookup helpers, tRPC surface, read-path staleness fix, B.1b post-scan safety net, B.2 hourly reclaim job, client SHA256 worker/hook, and B.1a upload-prevention wiring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model-files): B.1a hashing = WebCrypto + jsSHA (Justin's approach) Header-metadata dedup ruled out: verified against real S3 bytes that official safetensors carry only {"format":"pt"} in __metadata__, no hash (DB headerData is unreliable). Keep B.1a with client full-file SHA256 = byte identity = stored ModelFileHash.SHA256. Swap the worker impl from hash-wasm to native WebCrypto (<=1GB) + jsSHA streaming (>1GB); 5GB cap is now a time guard. Reference computeSHA256InChunks hash-chains and must not be used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(model-files): official-file lookup helpers (size + hash) for dedupe * feat(model-files): tRPC queries for official file size/hash lookup * fix(model-files): drop stale linked components (deleted source) on reads * feat(model-files): B.1b post-scan dedup of official-matching uploads Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(model-files): B.2 hourly job rehoming non-official dupes onto official files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(model-files): client SHA256 worker + hook for official-match dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(model-files): B.1a prevent upload when bytes match an official file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(model-files): B.1a intercept falls through to normal upload on error Wrap resolveOfficialMatch in try/catch so network errors treat match as null and fall through to upload. Wrap the mutateAsync+enrichment block in try/catch so a server 500 shows an error notification and falls through instead of freezing the file in isPending state. Also fix import order (components → hooks → utils) and add the missing expect(findByHash).not.toHaveBeenCalled() assertion to the primary-weights-host test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(model-files): use correct showErrorNotification shape in B.1a intercept * fix(model-files): correct SHA256 hash case (C1), B.2 loop-until-drained (T5), generation linked-VAE staleness (I2) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(model-files): dedicated linkOfficialFileByHash endpoint for B.1a (C2/I1) * feat(model-files): show a checking indicator during B.1a official-match On a size collision, B.1a hashes the file (seconds for large files) and does a server round-trip before it can skip the upload — with no visual cue. Add an onHashStart callback fired only when hashing actually begins (not on the fast size-gate), set an isCheckingOfficial flag on the file, and render a spinner + 'Checking for an existing copy on Civitai…' in the FileCard. Cleared on every exit (link, no-match, over-cap, error). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model-files): match official file by SHA256 alone; componentType from official identity The host file's declared type no longer gates the match — a byte-identical file labeled 'Other' (or mislabeled) still matches. componentType is now derived from the official file's true identity (its standalone model type, then its own file type), not the user's label, so a VAE isn't stamped 'TextEncoder' just because the uploader mislabeled it. An official checkpoint match yields no componentType and is never linked (only accessories are). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model-files): check main-section files for dedup too (close bypass) The primary-type gate let a user skip the dedup check by dropping an official file in the main file section (typed as Model). Remove that gate from the client hash check and the official lookup so the file's bytes — not its declared type — decide. B.1a now prevents the upload for a main-section file that matches an official accessory (pointer, no replaceFileId). A genuine checkpoint match links nothing (no componentType) and uploads normally. B.1b still skips primary-typed files, since addLinkedComponent refuses to delete primary weights. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model-files): render ellipsis in B.1a checking indicator A lint autofix had rewritten the literal … in the JSX text node to the escape …, which JSX text does not process — so it rendered literally as 'Civitai…'. Use the … HTML entity, which JSX decodes and the escape autofix leaves alone. * fix(model-files): address Copilot review — primary-weights consistency + hardening Use primaryModelFileTypes (Model/Pruned Model/Diffusion Model/UNet) as the single source of truth for 'never dedup/delete primary weights', which was applied inconsistently: - official-file.service: an official Diffusion Model/UNet file was treated as a linkable accessory (inferComponentType maps them non-null). After removing the host gate, B.1a could have linked a user's Flux/Wan primary weights and gutted the model. Now excluded. - dedupe-official-uploads (B.2): SQL now excludes Diffusion Model/UNet host types too, so the job can't delete Flux/Wan primary weights. - addLinkedComponent replaceFileId guard: rejects all primary-weights types (not just Model/Pruned Model) and Training Data. Also: reject a targetVersionId that mismatches the targetFileId's parent version (avoids inconsistent denormalized data); fix a stale Mine/Official tab comment; correct the spec's SHA256 casing (stored UPPERCASE, not lowercase). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(model-files): cleanup from final A+B quality review - Static import of findOfficialFileByHash (no cycle exists; drop the defensive dynamic import). - Move findOfficialFilesBySizeHandler router -> controller (match every other handler). - Derive B.2 job's primary-weights SQL exclusion from primaryModelFileTypes via Prisma.join (no hand-copied list). - Extract toLinkedComponent() — dedupe the two identical LinkedComponent enrichment sites. - Extract skipBaseModelForOwnTabs() — shared by the Meili filter + client filterVersions (was copy-pasted). - Tests: JSON.stringify(where) -> toMatchObject; drop a redundant no-op test; assert the B.1b error path is actually entered. - Comment hygiene: drop file-path headers, caller-narration, name-restatements, and (I1) tracker refs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(model-files): findOfficialFilesBySizeHandler follows the handler convention Accept { input, ctx } (typed like the other model-file controller handlers) with async + throwDbError, and wire it directly on the router query instead of the inline ({ input }) => handler(input) wrapper. * fix(ResourceSelectModalContent): reorder tabs array for improved user experience * docs(model-files): reduce link-existing-files docs to how-it-works + Subtask C A and B are shipped, so their implementation detail (file:line refs, code, TDD steps, status logs, file-by-file/testing sections) no longer belongs in the docs. Rewrite the design doc as a conceptual explanation of how linking works, describe A/B at a high level, and keep Subtask C (the remaining stipulated work). Delete the B spec and B plan (pure implementation artifacts for completed work; retained in git history). * docs(model-files): rewrite link-existing-files design as conceptual how-it-works Drop the implementation detail (file:line refs, code, status logs, file-by-file/testing sections) now that A and B are shipped; explain how linking works conceptually, summarize A/B, and keep Subtask C (remaining work). * refactor(model-files): drop internal subtask labels (B.1a/B.1b/B.2/A.2) from comments Rewrite comments to describe the actual behavior instead of referencing the epic's subtask identifiers, which mean nothing to a dev coming to this code fresh. Also rename the cryptic Axiom log names ('b1b-official-dedup', 'b2-official-dedup') to descriptive ones. * refactor(model-files): drop hostType from findOfficialFileByHash / linkOfficialFileByHash componentType now derives solely from the official file's own identity (its standalone model type, then its own file type), which is authoritative — the caller's declared type was only a last-resort fallback that never classified a real accessory and could mislabel a non-accessory match. Removed from the service, the endpoint schema, the B.1b caller, and the client request. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(model-files): consolidate single-fn files + boolean size lookup - fold official-file.service into model-file.service (findOfficialFilesBySize, findOfficialFileByHash, OfficialFileMatch); repoint importers + tests - fold linked-component-helpers (selectLiveLinkedComponents) into model-helpers - rename findOfficialFilesBySize -> hasOfficialFileOfSize, return boolean via count() instead of an unused { id }[] payload across service/controller/tRPC/client - clarify post-scan official-dedup comment to name the S3 delete (replaceFileId -> deleteFile) that actually reclaims storage Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dedupe-official-uploads): align host-type filter with processor + gate host version The SQL host-type filter used a `NOT IN (primaryModelFileTypes)` denylist that passed types processDedupePairs cannot rehome (inferComponentType returns null): those rows never get a linked-component pointer, so they requery every run and, if enough cluster at the front of a full batch, starve processable pairs. Replace it with an allowlist derived from inferComponentType itself so the two can't drift. Also gate the host on `hv.status = 'Published'` (not just the model) so a file on a draft version of a published model isn't silently deleted and rehomed. Validated on the replica: removes 3 unprocessable rows (Other x2, Negative) and 1 draft-version VAE; the 111 published rehomable candidates are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(link-existing-files): design for 30-day quarantine of replaced files Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(link-existing-files): restore endpoint is moderator-only Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(link-existing-files): implementation plan for quarantine of replaced files Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(model-file): add replacedAt column for quarantine Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model-file): make replacedAt migration ADD COLUMN idempotent * feat(link-existing-files): quarantine replaced file instead of deleting Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test): alias @civitai/notifications in vitest config Server-side vitest tests import ~/server/common/enums.ts which pulls in @civitai/notifications; the workspace-package alias list was missing it, breaking module resolution under vitest. Additive alias fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(link-existing-files): hide quarantined files from version file lists * fix(link-existing-files): filter quarantined files from loadModelVersion files select * feat(link-existing-files): daily job purges replaced files after 30d Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(link-existing-files): exclude self-row from refcount so purge job actually frees S3 processReplacedFiles keeps the ModelFile row (only sets dataPurged=true), so deleteModelFileObject's refcount guard always found the row as a "live" reference to its own url, silently skipped the S3 delete, and the job still reported success — leaking bytes forever since dataPurged=true gates it out of future runs. Thread an optional excludeId through urlsSafeToDelete/deleteModelFileObject so the job can exclude its own row id. Existing callers (deleteFile, updateFile, deleteModelFileObjects) are unaffected — the param is optional. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(link-existing-files): moderator restore endpoint for quarantined files * test(link-existing-files): restore test uses non-default priorVisibility + default-case * fix(link-existing-files): make markFileReplaced idempotent; document getVaeFiles invariant A second markFileReplaced call on an already-quarantined file re-stamped priorVisibility with the current (Private) value and reset the 30-day clock, breaking moderator restore back to the true original visibility. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(link-existing-files): remove quarantine plan doc (feature shipped) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model-files): consolidate linking+dedup into single feature reference Fold the how-it-works + shipped quarantine behavior into docs/features/ model-file-linking-dedup.md; drop the superseded root design doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model-files): remove superseded quarantine implementation plan Superseded by docs/features/model-file-linking-dedup.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model-files): rework linking+dedup doc as timeless reference Drop ClickUp ref, Remaining/Subtask-C section, and A/B/shipped labels; correct key-code file paths (official-file.service.ts was consolidated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dedupe-official-uploads): report link failures to Axiom in aggregate Track linked/failed counts per run and emit an error-level Axiom log when any pair fails to link, so a systemic break (e.g. a failing addLinkedComponent path) surfaces instead of looking like a clean no-op behind scattered per-pair warnings. Return payload now carries found/linked/failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
70 lines
2.7 KiB
JavaScript
70 lines
2.7 KiB
JavaScript
// Bundles SharedWorker scripts to public/workers/*.js with esbuild.
|
|
//
|
|
// Why: Turbopack does not compile `.ts` SharedWorker entry scripts referenced
|
|
// via `new SharedWorker(new URL('./worker.ts', import.meta.url))` — it emits the
|
|
// raw .ts (dev) or nothing (prod build), so the browser fails with
|
|
// "Failed to fetch a worker script". See vercel/next.js#74842.
|
|
//
|
|
// Fix: pre-bundle each worker to a self-contained classic (IIFE) JS file served
|
|
// statically from /public, and instantiate via `new SharedWorker('/workers/x.js')`
|
|
// (a static path, not new URL(import.meta.url)) so Turbopack is bypassed entirely.
|
|
//
|
|
// Run via `pnpm build:workers` (also wired into predev/prebuild).
|
|
import esbuild from 'esbuild';
|
|
import { config as loadEnv } from 'dotenv';
|
|
import { existsSync } from 'node:fs';
|
|
import path from 'node:path';
|
|
import { fileURLToPath } from 'node:url';
|
|
|
|
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
|
|
|
|
// Load .env so NEXT_PUBLIC_* values are available to inline (mirrors what Next
|
|
// does for the main bundle). Optional in CI where they may come from real env.
|
|
for (const f of ['.env.local', '.env']) {
|
|
const p = path.join(root, f);
|
|
if (existsSync(p)) loadEnv({ path: p });
|
|
}
|
|
|
|
// Replace every `process.env` reference in the bundle with a literal object.
|
|
// The workers import `~/env/client` (t3-env), which reads process.env.NEXT_PUBLIC_*
|
|
// and validates on import — there is no `process` in a worker, so we inline the
|
|
// public env and skip validation.
|
|
const publicEnv = { NODE_ENV: process.env.NODE_ENV ?? 'production', SKIP_ENV_VALIDATION: 'true' };
|
|
for (const [k, v] of Object.entries(process.env)) {
|
|
if (k.startsWith('NEXT_PUBLIC_') && v !== undefined) publicEnv[k] = v;
|
|
}
|
|
|
|
const define = { 'process.env': JSON.stringify(publicEnv) };
|
|
|
|
const workers = [
|
|
{ in: 'src/utils/signals/worker.ts', out: 'public/workers/signals.worker.js' },
|
|
{ in: 'src/workers/civitai-link.worker.ts', out: 'public/workers/civitai-link.worker.js' },
|
|
{ in: 'src/workers/file-hash.worker.ts', out: 'public/workers/file-hash.worker.js' },
|
|
];
|
|
|
|
const watch = process.argv.includes('--watch');
|
|
|
|
for (const w of workers) {
|
|
const options = {
|
|
entryPoints: [path.join(root, w.in)],
|
|
outfile: path.join(root, w.out),
|
|
bundle: true,
|
|
platform: 'browser',
|
|
format: 'iife', // classic SharedWorker — broadest support, no `type: module` needed
|
|
target: 'es2020',
|
|
define,
|
|
tsconfig: path.join(root, 'tsconfig.json'), // resolves the `~/*` path alias
|
|
legalComments: 'none',
|
|
logLevel: 'info',
|
|
};
|
|
|
|
if (watch) {
|
|
const ctx = await esbuild.context(options);
|
|
await ctx.watch();
|
|
console.log(`[build-workers] watching ${w.in}`);
|
|
} else {
|
|
await esbuild.build(options);
|
|
console.log(`[build-workers] built ${w.out}`);
|
|
}
|
|
}
|