mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
feat(generation): require a SafeTensor weight file to load a checkpoint
The cluster serves SafeTensor only, so a checkpoint without one cannot be loaded — and since paid loading is how a non-resident checkpoint generates at all, it cannot generate either. `covered` must be false for it rather than "covered but never offered a load", because the model detail page's Create button reads canGenerate. Scoped to checkpoints, on the view's checkpoint disjunct rather than its shared EXISTS. Loading is a checkpoint-only feature, and applying the rule to every type would drop 3,287 covered textual inversions carrying 1.39 billion lifetime generations — embeddings ship as PickleTensor and are not served by the loader. `checkLoadable` is scoped the same way for the same reason. Measured against the production replica, 2026-09-09: covered checkpoints 33,811 -> 31,569, total view rows 933,851 -> 931,609, every row of the delta a checkpoint. 834 of the 2,242 have generation history (6.5M lifetime, 0.43% of checkpoint generation); none is offered by the live GenerationCoverage and none generated in the last month, so applying the migration ahead of any UI is safe. An allow-list, not the deny-list the shared clause uses: `format` is free text and frequently unset, so `<> ALL (...)` cannot promise SafeTensor while `= 'SafeTensor'` is null-safe in the intended direction. This reverses part of the 2026-09-08 Diffusers ruling for checkpoints only (174 versions); recorded in paid-model-loading-decisions.md as an open question for Justin rather than left in a migration comment. CoveredCheckpoint returns as a disjunct — not the conjunct it was — excusing 6 auction-resident checkpoints from the requirement until the auction is retired. Refusals now say which of the two reasons applies. The old copy told every unloadable resource it "runs through an external provider", which is false for a GGUF checkpoint; the moderator page hardcoded that same wrong string separately. `no-divergent-safetensor-rule` pins the two halves together: nothing executes the view (the suite mocks $queryRaw wholesale), so a text guard over the shared literals and the checkpoint scoping is the enforceable shape. Mutation-probed three ways. Migration is NOT auto-applied — run it manually where you want it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -146,8 +146,17 @@ every measured number are in [paid-model-loading-coverage.md](paid-model-loading
|
||||
model with a loadable file. The LORA/TI/VAE/LoCon/DoRA/Upscaler branch is unchanged.
|
||||
- [x] drop the `CoveredCheckpoint` conjunct, and allow `Diffusers` while keeping Core ML and ONNX
|
||||
excluded — [the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)
|
||||
- [x] **checkpoints require a SafeTensor weight file** —
|
||||
`20260909180000_generation_coverage_next_safetensor_checkpoints`, 2026-09-09. Narrows the
|
||||
2026-09-08 view: Diffusers stays loadable for every type *except* checkpoints, and
|
||||
`CoveredCheckpoint` returns as a disjunct excusing 6 auction-resident versions.
|
||||
**Written, not yet applied to any environment.**
|
||||
- [ ] 🔴 keep `EcosystemCheckpoints` — 62 of 63 checkpoint defaults depend on it
|
||||
- [x] diffed against production 2026-09-08 — nothing loses coverage; [the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)
|
||||
- [x] diffed against production 2026-09-08 — nothing lost coverage *at that point*; [the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)
|
||||
- [ ] 🔴 **2,242 covered checkpoints lose coverage when the SafeTensor migration is applied**
|
||||
(33,811 -> 31,569; 834 with generation history, 6.5M lifetime generations). A narrowing, so
|
||||
there is no safe window — apply it when the readers of `covered` are ready.
|
||||
*Closes when:* applied to production and the covered-checkpoint count reads 31,569.
|
||||
- [ ] **Set `usageControl = 'ExternalGeneration'` on the 36 mislabelled API versions.** All
|
||||
published, none POI, coverage preserved 36/36. Mod-only to set via the app, so it is a direct
|
||||
DB write.
|
||||
@@ -239,9 +248,11 @@ the platform second.
|
||||
- [ ] **C11 — retire auctions.** ([868ktt5b2](https://app.clickup.com/t/868ktt5b2)) Do not scope
|
||||
until 868gtq1kt (splitting featuring out of auctions) has an answer — auctions do two jobs
|
||||
and paid loading replaces one. ~89 files under `src/`.
|
||||
- [x] the `CoveredCheckpoint` conflict is resolved by removing it from coverage (Phase 1.6), so
|
||||
the auction job can no longer un-cover a paid checkpoint. What remains is deciding whether
|
||||
that job should keep writing rows nothing reads.
|
||||
- [x] the `CoveredCheckpoint` conflict is resolved by removing it as a coverage *conjunct*
|
||||
(Phase 1.6), so the auction job can no longer un-cover a paid checkpoint. It survives as a
|
||||
disjunct covering 6 auction-resident versions that lack a SafeTensor file — those would lose
|
||||
coverage on the next auction prune, but none is loadable, so none can have been paid for.
|
||||
What remains is deciding whether that job should keep writing rows nothing else reads.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -55,7 +55,9 @@ This is the thing an earlier reading of these docs got wrong, and it inverted a
|
||||
| `CoveredCheckpoint` | 514 rows | Auction-won community checkpoints — a **residency proxy**. Written and pruned weekly by `handle-auctions.ts`. **This is what paid loading replaces.** (638 versions are covered *as checkpoints* — the rest come from `EcosystemCheckpoints`.) |
|
||||
| `EcosystemCheckpoints` | 125 | The generator's **default model per ecosystem**. **62 of the 63 checkpoint defaults are covered through it — and zero through `CoveredCheckpoint`.** Not a loophole; the registry that keeps the generator working. |
|
||||
|
||||
Dropping `CoveredCheckpoint` is the feature. Dropping `EcosystemCheckpoints` would remove the
|
||||
Dropping `CoveredCheckpoint` **as a conjunct** is the feature — it no longer gates anything. It
|
||||
survives as a *disjunct* excusing 6 auction-resident checkpoints from the SafeTensor requirement, and
|
||||
is deleted when the auction stops writing rows. Dropping `EcosystemCheckpoints` would remove the
|
||||
default model from half the ecosystems the generator supports — see
|
||||
[the defaults audit](#the-defaults-audit).
|
||||
|
||||
@@ -89,8 +91,23 @@ contributes. Earlier drafts of these docs used 514 for both, which is what made
|
||||
fail to add up.
|
||||
|
||||
155 published, licensed, standard checkpoints on supported base models were blocked **only** by file
|
||||
format: 132 Diffusers, 21 Core ML, 2 ONNX. Diffusers is loadable (Justin, 2026-09-08); Core ML and
|
||||
ONNX are inference-runtime formats rather than servable weights and stay excluded.
|
||||
format under `GenerationCoverage`: 132 Diffusers, 21 Core ML, 2 ONNX.
|
||||
|
||||
⚠️ **Diffusers was ruled loadable (Justin, 2026-09-08) and then narrowed back out for CHECKPOINTS on
|
||||
2026-09-09.** The loader serves SafeTensor only, so the checkpoint branch of `GenerationCoverageNext`
|
||||
now requires a SafeTensor weight file (migration
|
||||
`20260909180000_generation_coverage_next_safetensor_checkpoints`) and `checkLoadable` in
|
||||
`resource-load.service.ts` refuses everything else with `unsupported-format`. Diffusers remains
|
||||
accepted for **every other model type** — the shared `EXISTS` is unchanged, so the Core ML / ONNX
|
||||
deny-list still governs LoRA/TI/VAE/LoCon/DoRA/Upscaler.
|
||||
|
||||
🔴 **The SafeTensor narrowing took 2,242 checkpoints back out.** Measured 2026-09-09 against the
|
||||
production replica: covered checkpoints **33,811 -> 31,569**, total view rows **933,851 -> 931,609**
|
||||
(every row of the delta a checkpoint; textual inversions unchanged at 6,299, all 514 auction rows
|
||||
retained). 834 of the 2,242 have any generation history — 6.5M lifetime generations, 0.43% of all
|
||||
checkpoint generation. Scoping matters: putting the rule on the shared `EXISTS` instead would have
|
||||
removed 3,287 textual inversions carrying **1.39 billion** generations, more than every SafeTensor TI
|
||||
combined.
|
||||
|
||||
### The loader population, split by bucket
|
||||
|
||||
|
||||
@@ -169,6 +169,25 @@ is dead code. Removing it widens covered checkpoints by roughly two orders of ma
|
||||
must stay**, because 62 of 63 checkpoint defaults are covered through it and none through
|
||||
`CoveredCheckpoint`. See [coverage](paid-model-loading-coverage.md#the-two-tables-do-opposite-jobs).
|
||||
|
||||
⚠️ **Amended 2026-09-09.** `CoveredCheckpoint` is no longer a conjunct — Justin's condition holds —
|
||||
but it returns as a **disjunct** in `20260909180000_generation_coverage_next_safetensor_checkpoints`,
|
||||
excusing 6 auction-resident checkpoints from the new SafeTensor requirement. Auction membership no
|
||||
longer decides coverage; it stands in for residency until C11 retires the job, and is deleted with it.
|
||||
|
||||
### 2.7 ⚠️ Diffusers is loadable — except for checkpoints
|
||||
|
||||
Justin ruled Diffusers loadable on 2026-09-08, and the first `GenerationCoverageNext` migration
|
||||
removed it from the excluded-format list for every type. The loader serves **SafeTensor only**, so
|
||||
the checkpoint branch was narrowed back on 2026-09-09 — 174 Diffusers checkpoints sit among the
|
||||
2,242 versions that lose coverage. Diffusers is untouched for LoRA/TI/VAE/LoCon/DoRA/Upscaler.
|
||||
|
||||
**Open:** whether Justin accepts the narrowing as an implementation constraint, or wants the loader
|
||||
to grow Diffusers support. *Decides:* Justin. *Closes when:* he answers here, or the migration is
|
||||
applied to production with the narrowing intact.
|
||||
|
||||
@dev: The loader can only serve SafeTensor today. OK to drop Diffusers/GGUF/PickleTensor checkpoints
|
||||
from coverage (2,242 versions, 0.43% of checkpoint generation), or should the loader learn them?
|
||||
|
||||
### 2.4 ✅ The C4 webhook — not now
|
||||
|
||||
**Closed 2026-09-08: no.** Both reasons to build it went away on the same day.
|
||||
|
||||
@@ -240,15 +240,18 @@ The premise of the feature is that the generator stops being restricted to a cur
|
||||
2026-09-08 this is scoped and decided; the full model, the audit and every measured number live in
|
||||
[paid-model-loading-coverage.md](paid-model-loading-coverage.md). In short:
|
||||
|
||||
- **`CoveredCheckpoint` goes away.** It is the auction's residency proxy, it has four uses and all
|
||||
four are generation, and dropping it widens covered checkpoints by roughly two orders of magnitude
|
||||
([the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)).
|
||||
- **`CoveredCheckpoint` stops gating.** It is the auction's residency proxy, it has four uses and all
|
||||
four are generation, and removing it *as a conjunct* widens covered checkpoints by roughly two
|
||||
orders of magnitude ([the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)). It
|
||||
remains as a *disjunct* excusing 6 auction-resident checkpoints from the SafeTensor rule, and is
|
||||
deleted with the auction.
|
||||
- **`EcosystemCheckpoints` stays.** It is the generator's default model per ecosystem — 62 of the 63
|
||||
checkpoint defaults are covered through it and none through `CoveredCheckpoint`. Removing it would
|
||||
strip the default model from half the supported ecosystems.
|
||||
- **`GenerationBaseModel` stays as the gate.** It marks the base models where the orchestrator has
|
||||
extended checkpoint/diffuser support, i.e. where community models can run.
|
||||
- **Diffusers becomes loadable**; Core ML and ONNX stay excluded.
|
||||
- **Checkpoints must carry a SafeTensor weight file** (2026-09-09); GGUF, PickleTensor, Diffusers,
|
||||
Core ML, ONNX and unset are all unloadable. Diffusers stays accepted for every other type.
|
||||
- **File-less models never touch the loader**, and "file-less" means *no loadable file*, not *no file
|
||||
row* — 36 API models carry a `Training Data` archive and would otherwise read as loadable.
|
||||
|
||||
@@ -343,7 +346,9 @@ deciding anything.
|
||||
- **`CoveredCheckpoint` stops gating generation**; `EcosystemCheckpoints` and `GenerationBaseModel`
|
||||
stay. Coverage means *allowed to generate*; residency is the orchestrator's axis.
|
||||
- **Only base models in `GenerationBaseModel` are loadable.** Everything else is out of scope for v1.
|
||||
- A checkpoint needs a **correct model file** to be loadable; file-less API models never are.
|
||||
- A checkpoint needs a **SafeTensor** weight file to be loadable (2026-09-09); file-less API models
|
||||
never are, and a GGUF/PickleTensor/Diffusers checkpoint gets a different refusal —
|
||||
`UNLOADABLE_MESSAGES` in `resource-load.service.ts` is the single source of both.
|
||||
- A load that never finishes is **refunded**.
|
||||
- The purchase path refuses anything **not in `GenerationCoverageNext`** (composed with ecosystem
|
||||
type support by `isGenerationEligible`), which is how the `RentCivit` rule is enforced without
|
||||
|
||||
Reference in New Issue
Block a user