briant
c60fd7c2e1
A moderator pastes a Hugging Face repo URL at /moderator/huggingface-import, picks files, and our servers fetch the weights into our storage — replacing a human downloading 20GB and re-uploading it through the browser wizard. A finished import attaches to a ModelVersion as a ModelFile, which puts it on the existing scan and hash pipeline. The transfer is resumable by construction. A 20GB file cannot move inside one request or one job run — jobs here hold a lock measured in minutes and a deploy rolls the pod — so it is a sequence of independent parts: a ranged read from HF written as one multipart part. The row stores `uploadId`, `partSize` and the parts written so far, and each cron run moves as many as fit in its budget. A deploy costs one part, not the file. Notes for review: - Parts complete out of order, so the resume point is the SET of missing part numbers, never a count, and a resume addresses the bucket recorded on the row rather than whatever the backend config resolves to now. - Every write inside a claimed run is fenced by `claimedBy`, and completion re-reads status from the PRIMARY first — without that, a cancel arriving during the final part still finalised the upload. - `PART_SIZE_BYTES` is fixed at 16MB rather than `getUploadChunkSize`, whose 1000-part cap is a browser-presigning bound and would make part size (and so pod memory) grow with the file. - Imported objects use the same bucket and the same `buildUploadKey` as a browser upload. Nothing about the import appears in the key; the HuggingFaceImport row is the index, because a key is immutable and a column is not. - `official-model-admin` gains `hf-imports` and `attach-import`, so the upload step it used to hand back to a human is scriptable. 🔴 The migration has already been applied to production by hand; any other environment still needs it. This repo never runs `prisma migrate deploy`. 🔴 Depends on `refactor/shared-upload-key`, which must merge first — this branch carries those four files so it compiles, and they become no-ops on rebase. Verified: typecheck, lint and prettier clean; 37 tests over the transfer engine, several mutation-checked. Full suite 41,224 passed / 28 failed — all 28 pre-existing on main (9 verified against a clean tree, 19 from a package.json/lockfile mismatch on @civitai/generation-metadata). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015cyrXpr87t9Tj3bnzRrUhp
9.4 KiB
Onboarding a model to the generator
How a new model gets from nothing to generatable. That covers a new ecosystem, a new CivitaiOfficial model, or a new version of an existing one. Mods test it on the live generator before anyone else can see it. The onboard-generator-model skill runs this flow end to end. This doc is what that skill follows, and the reference for doing any step by hand.
The flow
| # | Step | Lands in | Tool | When |
|---|---|---|---|---|
| 1 | Ecosystem + base model records | packages/civitai-shared/src/basemodel.constants.ts |
add-ecosystem skill |
new ecosystem or new base model |
| — | Deploy | only if step 1 ran | ||
| 2 | Model, Draft, owned by CivitaiOfficial, with description — or the existing model's description updated for the new version |
model.upsert + moderator.models.transferOwnership |
write-model-description, then official-model-admin (create-model / update-description) once the user has approved the text |
always (review only if nothing changes) |
| 3 | Version, Draft, with its base model and usageControl |
modelVersion.upsert |
official-model-admin (create-version) |
always |
| 4 | Coverage row, then a cache bust | "EcosystemCheckpoints", written through postgres-query --writable |
generation-coverage (add) |
always |
| 5 | Gate rule that hides it from non-mods | Redis generation:gate-rules:by-id |
generation-gate-rules (add) or /moderator/generation-config |
always |
| 6 | Generation support, in both generator lanes | constants, graphs, handlers | add-generation-support |
always |
| 7 | Prompt-enhancement guide | orchestrator prompt-analysis service | add-prompt-enhancement-guide |
new ecosystem (image/video only) |
| — | Deploy, then mods test on the live generator | generator-launch (check) |
||
| 8 | Launch: publish, then remove the gate | a human, following generator-launch (launch) |
Every tool in the last column is a skill. onboard-generator-model runs them in this order.
For a new version of an existing model, step 2 is a review of the live description with write-model-description. In every case, the user approves the exact text before it is written: official-model-admin shows the full HTML (or a diff for an update) and refuses to write unless it gets the approval hash of that text.
Why a new base model takes two deploys
modelVersion.upsert checks baseModel against the base-model names compiled into the server that handles the request (baseModelSchema in src/server/schema/model-version.schema.ts). So a version with a brand-new base model can't be created until the constants that name it are in production.
Step 6 hardcodes the new version's ID: it goes in the ecosystemSettings default model and in each graph's defaultModelId. So step 6 can't be written before step 3. The minimum is step 1, deploy, steps 2–6, deploy. A new model or version on an existing base model needs one deploy.
Step 1 on its own adds nothing to the generator. An ecosystem enters the picker only through its ecosystemSupport generation entry and a workflow list in config/workflows.ts, and both are part of step 6. The base-model record can still show up in upload and filter pickers. Set hidden: true on it if that is too early.
How a mod tests an unpublished version
This needs no special mechanism. Two existing rules combine:
- Coverage. The first branch of the
GenerationCoverageview covers any version listed in"EcosystemCheckpoints", and it does not check status. The view is a plain view, so the row takes effect as soon as it is inserted.resourceDataCachesits in front of it with a one-hour TTL, so bust the cache withmodelVersion.bustCacheafter inserting. - Access.
getResourceCanGenerateinsrc/server/services/generation/generation.service.tstreatsDraftas a generatable status. It lets moderators and the model owner through the private check and refuses everyone else.
The result: a Draft version with an EcosystemCheckpoints row can be generated by moderators and by CivitaiOfficial, and by nobody else. Ideogram 4 was in this state for testing (version 3246186).
Why the gate rule is still needed
Draft stops non-mods from generating with the version, but it doesn't stop them from seeing it. Once step 6 deploys, the ecosystem is in everyone's picker, and a locked default checkpoint or a versions.options entry lists the version. A non-mod who selects it fails at submit. The gate rule removes it from view:
- new ecosystem: gate the ecosystem key (
EcosystemRecord.key, case-sensitive) - new version in an existing ecosystem: gate the model version ID
Use availableTo: moderators with presentation: hidden, so mods keep access and everyone else never sees the item. Add the rule before the deploy. A rule that names an ecosystem or version the running code doesn't know yet has no effect, so adding it early is safe. The rules model is described in generation-gating-rules-model.md.
A hidden rule also blocks canGenerate site-wide for everyone it gates. A version published while its rule is still in place stays ungeneratable for non-mods, including from its model page.
Two version shapes
The production EcosystemCheckpoints rows show two shapes of CivitaiOfficial version:
| Shape | usageControl |
Files | Examples |
|---|---|---|---|
| API-only: the provider hosts the model | ExternalGeneration |
none | Qwen 3, Seedance 2.5, Flux 3 Video, Muse Image |
| Hosted weights: runs on our cluster | Download |
uploaded in the version wizard | Ideogram 4.0, MiniMax H3, LTXV 2.5, Mage Flow |
MiniMax H3 appears in both rows, one version each, so the base model alone doesn't decide the kind. File count doesn't decide it either: a few older ExternalGeneration versions carry files that are never used. The signals that do help:
- The handler's
engine.comfymeans hosted weights. A closed provider's engine, orfal, means API-only. A model-family engine such aswanorltx2settles nothing. - How existing versions of the same base model are set up.
- For a new ecosystem, the
@civitai/orchestration-clientinput type.Comfy*means hosted weights.
The deciding question is whether the provider publishes weights we run, or only an API. official-model-admin evidence collects these signals, and the user confirms the kind.
The upload wizard (/models/<modelId>/model-versions/<versionId>/wizard) skips its files step for ExternalGeneration. For hosted weights there are two routes: weights already on Hugging Face are queued at /moderator/huggingface-import and fetched server-side, then attached with official-model-admin attach-import; anything else is uploaded by hand at ?step=2, or through the Manage files item in the version menu. Either way a human drives it — this is the one step no skill completes on its own.
A hosted-weights version is ready to cover once it has a scanned file of a weight type, and a checkpoint also needs a SafeTensor file. That's the same rule checkLoadable in src/server/services/resource-load.service.ts applies. official-model-admin files checks it.
Launch
These steps are done by a human, after the deploy. The generator-launch skill (launch --version <id>) prints them with the real URLs and commands, and ticks off the steps already done.
- Publish, with the green Publish button in the version panel of
/models/<modelId>?modelVersionId=<versionId>. On a Draft model this publishes the model and the version together; on a published model it publishes only the version. - Remove the gate rule with
onboard.mjs gate remove, or in the mod UI.
Publish before removing the gate. While the rule is in place, the published version stays hidden from non-mods, because a hidden rule also blocks site-wide canGenerate. Removing the gate first would show everyone a Draft model they can't generate with.
After that, only if relevant:
- A
GenerationBaseModelrow makes community resources for downloadable weights generatable (coverage branch 3). Its key is the base model's display name. - An
AuctionBaserow adds a paid featured-resources auction. Whether to add one is a product decision. Its key is the ecosystem key. - Training support: the
add-training-supportskill. - An ecosystem landing page: the
ecosystem-seo-pageskill.
The EcosystemCheckpoints row overrides everything else: the version stays covered even after it is unpublished. An ExternalGeneration version is also covered by branch 2 once published, so deleting its row after launch is what lets a later unpublish remove coverage. So far, every row has been kept.
The SQL for all three coverage branches, GenerationBaseModel and AuctionBase is in the add-generation-support skill under "Post-onboarding".
Mistakes that fail silently
- Wiring only one lane. The generator has two lanes, data-graph (
src/shared/data-graph/generation/) and form-graph (src/shared/form-graph/generation/). Each has its own graph, handler and registration. The parity tests andhandlers.differential.test.tslist ecosystems by hand, so a missing entry isn't flagged anywhere. - Skipping the cache bust. After the coverage insert, the generator reports the version as uncovered for up to an hour.
- Misspelling the gate target. A rule with a wrong ecosystem key has no effect, and nothing distinguishes it from having no rule at all.