feat(generation): require a SafeTensor weight file to load a checkpoint

The cluster serves SafeTensor only, so a checkpoint without one cannot be loaded —
and since paid loading is how a non-resident checkpoint generates at all, it cannot
generate either. `covered` must be false for it rather than "covered but never
offered a load", because the model detail page's Create button reads canGenerate.

Scoped to checkpoints, on the view's checkpoint disjunct rather than its shared
EXISTS. Loading is a checkpoint-only feature, and applying the rule to every type
would drop 3,287 covered textual inversions carrying 1.39 billion lifetime
generations — embeddings ship as PickleTensor and are not served by the loader.
`checkLoadable` is scoped the same way for the same reason.

Measured against the production replica, 2026-09-09: covered checkpoints
33,811 -> 31,569, total view rows 933,851 -> 931,609, every row of the delta a
checkpoint. 834 of the 2,242 have generation history (6.5M lifetime, 0.43% of
checkpoint generation); none is offered by the live GenerationCoverage and none
generated in the last month, so applying the migration ahead of any UI is safe.

An allow-list, not the deny-list the shared clause uses: `format` is free text and
frequently unset, so `<> ALL (...)` cannot promise SafeTensor while `= 'SafeTensor'`
is null-safe in the intended direction.

This reverses part of the 2026-09-08 Diffusers ruling for checkpoints only (174
versions); recorded in paid-model-loading-decisions.md as an open question for
Justin rather than left in a migration comment. CoveredCheckpoint returns as a
disjunct — not the conjunct it was — excusing 6 auction-resident checkpoints from
the requirement until the auction is retired.

Refusals now say which of the two reasons applies. The old copy told every
unloadable resource it "runs through an external provider", which is false for a
GGUF checkpoint; the moderator page hardcoded that same wrong string separately.

`no-divergent-safetensor-rule` pins the two halves together: nothing executes the
view (the suite mocks $queryRaw wholesale), so a text guard over the shared literals
and the checkpoint scoping is the enforceable shape. Mutation-probed three ways.

Migration is NOT auto-applied — run it manually where you want it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
briant
2026-09-09 14:45:44 -06:00
parent 52f9445245
commit 2c9957e149
12 changed files with 465 additions and 37 deletions
+15 -4
View File
@@ -146,8 +146,17 @@ every measured number are in [paid-model-loading-coverage.md](paid-model-loading
model with a loadable file. The LORA/TI/VAE/LoCon/DoRA/Upscaler branch is unchanged.
- [x] drop the `CoveredCheckpoint` conjunct, and allow `Diffusers` while keeping Core ML and ONNX
excluded — [the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)
- [x] **checkpoints require a SafeTensor weight file**
`20260909180000_generation_coverage_next_safetensor_checkpoints`, 2026-09-09. Narrows the
2026-09-08 view: Diffusers stays loadable for every type *except* checkpoints, and
`CoveredCheckpoint` returns as a disjunct excusing 6 auction-resident versions.
**Written, not yet applied to any environment.**
- [ ] 🔴 keep `EcosystemCheckpoints` — 62 of 63 checkpoint defaults depend on it
- [x] diffed against production 2026-09-08 — nothing loses coverage; [the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)
- [x] diffed against production 2026-09-08 — nothing lost coverage *at that point*; [the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)
- [ ] 🔴 **2,242 covered checkpoints lose coverage when the SafeTensor migration is applied**
(33,811 -> 31,569; 834 with generation history, 6.5M lifetime generations). A narrowing, so
there is no safe window — apply it when the readers of `covered` are ready.
*Closes when:* applied to production and the covered-checkpoint count reads 31,569.
- [ ] **Set `usageControl = 'ExternalGeneration'` on the 36 mislabelled API versions.** All
published, none POI, coverage preserved 36/36. Mod-only to set via the app, so it is a direct
DB write.
@@ -239,9 +248,11 @@ the platform second.
- [ ] **C11 — retire auctions.** ([868ktt5b2](https://app.clickup.com/t/868ktt5b2)) Do not scope
until 868gtq1kt (splitting featuring out of auctions) has an answer — auctions do two jobs
and paid loading replaces one. ~89 files under `src/`.
- [x] the `CoveredCheckpoint` conflict is resolved by removing it from coverage (Phase 1.6), so
the auction job can no longer un-cover a paid checkpoint. What remains is deciding whether
that job should keep writing rows nothing reads.
- [x] the `CoveredCheckpoint` conflict is resolved by removing it as a coverage *conjunct*
(Phase 1.6), so the auction job can no longer un-cover a paid checkpoint. It survives as a
disjunct covering 6 auction-resident versions that lack a SafeTensor file — those would lose
coverage on the next auction prune, but none is loadable, so none can have been paid for.
What remains is deciding whether that job should keep writing rows nothing else reads.
---
+20 -3
View File
@@ -55,7 +55,9 @@ This is the thing an earlier reading of these docs got wrong, and it inverted a
| `CoveredCheckpoint` | 514 rows | Auction-won community checkpoints — a **residency proxy**. Written and pruned weekly by `handle-auctions.ts`. **This is what paid loading replaces.** (638 versions are covered *as checkpoints* — the rest come from `EcosystemCheckpoints`.) |
| `EcosystemCheckpoints` | 125 | The generator's **default model per ecosystem**. **62 of the 63 checkpoint defaults are covered through it — and zero through `CoveredCheckpoint`.** Not a loophole; the registry that keeps the generator working. |
Dropping `CoveredCheckpoint` is the feature. Dropping `EcosystemCheckpoints` would remove the
Dropping `CoveredCheckpoint` **as a conjunct** is the feature — it no longer gates anything. It
survives as a *disjunct* excusing 6 auction-resident checkpoints from the SafeTensor requirement, and
is deleted when the auction stops writing rows. Dropping `EcosystemCheckpoints` would remove the
default model from half the ecosystems the generator supports — see
[the defaults audit](#the-defaults-audit).
@@ -89,8 +91,23 @@ contributes. Earlier drafts of these docs used 514 for both, which is what made
fail to add up.
155 published, licensed, standard checkpoints on supported base models were blocked **only** by file
format: 132 Diffusers, 21 Core ML, 2 ONNX. Diffusers is loadable (Justin, 2026-09-08); Core ML and
ONNX are inference-runtime formats rather than servable weights and stay excluded.
format under `GenerationCoverage`: 132 Diffusers, 21 Core ML, 2 ONNX.
⚠️ **Diffusers was ruled loadable (Justin, 2026-09-08) and then narrowed back out for CHECKPOINTS on
2026-09-09.** The loader serves SafeTensor only, so the checkpoint branch of `GenerationCoverageNext`
now requires a SafeTensor weight file (migration
`20260909180000_generation_coverage_next_safetensor_checkpoints`) and `checkLoadable` in
`resource-load.service.ts` refuses everything else with `unsupported-format`. Diffusers remains
accepted for **every other model type** — the shared `EXISTS` is unchanged, so the Core ML / ONNX
deny-list still governs LoRA/TI/VAE/LoCon/DoRA/Upscaler.
🔴 **The SafeTensor narrowing took 2,242 checkpoints back out.** Measured 2026-09-09 against the
production replica: covered checkpoints **33,811 -> 31,569**, total view rows **933,851 -> 931,609**
(every row of the delta a checkpoint; textual inversions unchanged at 6,299, all 514 auction rows
retained). 834 of the 2,242 have any generation history — 6.5M lifetime generations, 0.43% of all
checkpoint generation. Scoping matters: putting the rule on the shared `EXISTS` instead would have
removed 3,287 textual inversions carrying **1.39 billion** generations, more than every SafeTensor TI
combined.
### The loader population, split by bucket
@@ -169,6 +169,25 @@ is dead code. Removing it widens covered checkpoints by roughly two orders of ma
must stay**, because 62 of 63 checkpoint defaults are covered through it and none through
`CoveredCheckpoint`. See [coverage](paid-model-loading-coverage.md#the-two-tables-do-opposite-jobs).
⚠️ **Amended 2026-09-09.** `CoveredCheckpoint` is no longer a conjunct — Justin's condition holds —
but it returns as a **disjunct** in `20260909180000_generation_coverage_next_safetensor_checkpoints`,
excusing 6 auction-resident checkpoints from the new SafeTensor requirement. Auction membership no
longer decides coverage; it stands in for residency until C11 retires the job, and is deleted with it.
### 2.7 ⚠️ Diffusers is loadable — except for checkpoints
Justin ruled Diffusers loadable on 2026-09-08, and the first `GenerationCoverageNext` migration
removed it from the excluded-format list for every type. The loader serves **SafeTensor only**, so
the checkpoint branch was narrowed back on 2026-09-09 — 174 Diffusers checkpoints sit among the
2,242 versions that lose coverage. Diffusers is untouched for LoRA/TI/VAE/LoCon/DoRA/Upscaler.
**Open:** whether Justin accepts the narrowing as an implementation constraint, or wants the loader
to grow Diffusers support. *Decides:* Justin. *Closes when:* he answers here, or the migration is
applied to production with the narrowing intact.
@dev: The loader can only serve SafeTensor today. OK to drop Diffusers/GGUF/PickleTensor checkpoints
from coverage (2,242 versions, 0.43% of checkpoint generation), or should the loader learn them?
### 2.4 ✅ The C4 webhook — not now
**Closed 2026-09-08: no.** Both reasons to build it went away on the same day.
+10 -5
View File
@@ -240,15 +240,18 @@ The premise of the feature is that the generator stops being restricted to a cur
2026-09-08 this is scoped and decided; the full model, the audit and every measured number live in
[paid-model-loading-coverage.md](paid-model-loading-coverage.md). In short:
- **`CoveredCheckpoint` goes away.** It is the auction's residency proxy, it has four uses and all
four are generation, and dropping it widens covered checkpoints by roughly two orders of magnitude
([the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)).
- **`CoveredCheckpoint` stops gating.** It is the auction's residency proxy, it has four uses and all
four are generation, and removing it *as a conjunct* widens covered checkpoints by roughly two
orders of magnitude ([the numbers](paid-model-loading-coverage.md#what-changes-in-numbers)). It
remains as a *disjunct* excusing 6 auction-resident checkpoints from the SafeTensor rule, and is
deleted with the auction.
- **`EcosystemCheckpoints` stays.** It is the generator's default model per ecosystem — 62 of the 63
checkpoint defaults are covered through it and none through `CoveredCheckpoint`. Removing it would
strip the default model from half the supported ecosystems.
- **`GenerationBaseModel` stays as the gate.** It marks the base models where the orchestrator has
extended checkpoint/diffuser support, i.e. where community models can run.
- **Diffusers becomes loadable**; Core ML and ONNX stay excluded.
- **Checkpoints must carry a SafeTensor weight file** (2026-09-09); GGUF, PickleTensor, Diffusers,
Core ML, ONNX and unset are all unloadable. Diffusers stays accepted for every other type.
- **File-less models never touch the loader**, and "file-less" means *no loadable file*, not *no file
row* — 36 API models carry a `Training Data` archive and would otherwise read as loadable.
@@ -343,7 +346,9 @@ deciding anything.
- **`CoveredCheckpoint` stops gating generation**; `EcosystemCheckpoints` and `GenerationBaseModel`
stay. Coverage means *allowed to generate*; residency is the orchestrator's axis.
- **Only base models in `GenerationBaseModel` are loadable.** Everything else is out of scope for v1.
- A checkpoint needs a **correct model file** to be loadable; file-less API models never are.
- A checkpoint needs a **SafeTensor** weight file to be loadable (2026-09-09); file-less API models
never are, and a GGUF/PickleTensor/Diffusers checkpoint gets a different refusal —
`UNLOADABLE_MESSAGES` in `resource-load.service.ts` is the single source of both.
- A load that never finishes is **refunded**.
- The purchase path refuses anything **not in `GenerationCoverageNext`** (composed with ecosystem
type support by `isGenerationEligible`), which is how the `RentCivit` rule is enforced without