mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
Zachary Lowden
dce428a492
* feat(app-blocks): author-fee accrual ledger + daily settlement rail (slice 2a, dark) The rail that pays an app author the per-generation fee slice 1 (#4922) learned to compute. Adds the ledger, the daily settlement job and the clawback path. NOTHING WRITES TO THE LEDGER YET. The viewer-charge path is deliberately not in this PR — see "What is NOT here" below. So this moves no money: the table stays empty, the settlement job no-ops on an empty scan, and the whole path is behind app-blocks-author-fee-enabled, which is false. TWO HOPS, MIRRORING THE MODEL LICENSING FEE. The licensing rail charges the viewer at generation time and writes a per-resource fee row, then deliver-creator-compensation mints to the creator daily. Same two hops and the same externalTransactionId dedup discipline here, in a civitai-owned table — because orchestration.resourceCompensations is keyed on a modelVersionId and written by the orchestrator, and an app fee has no model version. That fork is recorded rather than left for a reader to rediscover. NO FRACTIONAL ACCRUAL, AND THAT REVERSES THE DESIGN IT INHERITED. The licensing fee is fractional because it is priced per-image at 0.01 buzz and the viewer pays the ceiling of the sum, so the creator's share genuinely has sub-buzz resolution. This fee does not: max(flat, pct x base) is floored to whole Buzz before the viewer is shown or charged it (D7 requires the viewer see the exact number before the run, and Buzz cannot express a fraction). The author is credited exactly what the viewer was debited — the platform is a conduit and takes no cut. The daily batch therefore exists for ledger volume, not rounding. Consequence, stated in three places because slice 3 must surface it: an author who sets a 0 flat leg and a low percentage earns nothing on cheap generations, forever. Same shape as the $0.00 spend bounty this arc replaced; the difference is it is now the author's explicit choice and the platform default avoids it. DECISIONS IMPLEMENTED D6 blue Buzz in, blue Buzz out — settlement groups by buzz type and never coerces. A collapsed bucket would convert non-withdrawable Buzz into withdrawable earnings and no total would change. D10 the percent leg prices off base only (no code change; recorded). Self-dealing is excluded at accrual, and counted rather than dropped. The app owner is snapshotted at WRITE time so an ownership transfer cannot retroactively move earnings already accrued. CLAWBACK. The orchestrator refunds undelivered work after submit, so the fee has to follow or an author earns on a generation the viewer got refunded. Not a new policy — CalculateLicenseFees already weights every fee by delivered fraction. Before settlement the accrual is voided in place; after it, a negative carry-forward row nets against the next run. A bucket whose net goes non-positive is HELD, not forgiven at zero, so the debt stays visible. Mint happens BEFORE the status flip, deliberately: a crash between them settles late rather than paying twice, and the deterministic dedup key is what makes the retry safe. MIGRATION IS MANUAL-APPLY, AND SHIPS BEFORE THE CODE (rule 8, the #4903 precedent). Committed for history only. VERIFICATION pnpm typecheck OK - 0 type errors in 165s settlement suite 17 passed mutation sweep 6/6 killed, EACH by its own named test: self-dealing guard, D6 bucket key, dedup-key composition, non-positive hold, clawback sign, zero-fee guard positive control applied +1000 to the minted amount -> 2 tests red, proving the harness executes the code under test What is NOT here, and why: the viewer-charge path. The fee must be priced from the whatIf base and added to BOTH reservations before submit — otherwise it escapes the viewer's per-app consent budget entirely, which is the one real safety hole this design found. That touches four submit paths on the hot billing path and deserves its own focused audit rather than being folded in behind a new table. It is the next PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(app-blocks): FK targets app_blocks, not AppBlock — and the entry_type CHECK is unreachable Two corrections the dev-first apply caught, both in the migration only. 1. The app_block_id foreign key referenced "AppBlock", which does not exist. The Prisma model is AppBlock but it carries @@map("app_blocks"), so the physical table is app_blocks — confirmed against the existing block_spend_attribution_app_block_id_fkey, which points at app_blocks. Applied as written this would have failed outright on every database. Hand-writing the migration rather than generating it is what lost the mapped name. 2. The entry_type CHECK is SUBSUMED by the amount-sign CHECK and cannot fire. The sign check reads (entry_type='accrual' AND fee>=0) OR (entry_type='clawback' AND fee<=0), so any third value makes both disjuncts false and is rejected there first. Measured, not reasoned: an insert with entry_type='bogus' on dev came back rejected by ..._amount_sign_check, never by ..._entry_type_check. The constraint stays as an explicit statement of the allowed set, but it is now labelled as not-a-reachable-guard so nobody reads it as coverage. If the sign check is ever loosened this becomes live and needs its own negative control. APPLIED AND VERIFIED — dev first, then prod, per rule 8. dev cnpg-cluster-dev-1 / cnpg-database-dev prod cnpg-cluster-nvme0-5 / cnpg-database (primary re-derived live), with SET lock_timeout='5s' — the four FKs take a lock on User, a hot table, so failing fast beats queueing behind it Both databases: 18 columns, 4 foreign keys, 5 checks, 4 indexes, 0 rows, with a positive control (a bogus column name returns 0, so the query discriminates). block_spend_attribution unchanged at 602 rows. DDL replayed to both prod standbys (-4 and -7 report 18 columns). Constraint behaviour was exercised on dev, not merely confirmed present: a valid row inserts (positive control), and 7 negative controls are all rejected — 6 of them by their own named constraint. The seventh is finding 2 above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(app-blocks): round 0 — two comments asserted seams that do not exist, and the new-table rationale was false Round 0 (requirements & deletion) found five things wrong with what this PR ASSERTS rather than with what it does. Four are fixed here; the fifth is a deletion decision for the operator and is left open. 1. THE NEW-TABLE RATIONALE WAS FALSE, in both the migration and the Prisma docblock. It claimed block_spend_attribution is "IMMUTABLE by design". It is not — that model carries status, voidedReason, confirmedAt, voidedAt, paidOutAt and payoutId, the same accrued/settled/voided lifecycle plus a payout key. Those columns are merely dead today because the rail that wrote them was removed, which is a different claim. A reader who checked the schema would have found the lifecycle columns and concluded the reasoning was wrong. The requirement survives on a better reason, verified rather than substituted: recordSpendAttribution runs inside `void (async () => { … })()` so that a failed attribution write can never break a generation — droppable telemetry. An accrual is a money obligation and must be awaited. The seam, not the row shape, is what separates them. 2. A comment named `chargeBlockAuthorFee` "in the router" as the caller that performs the debit. That function has never existed anywhere in this repository; the name appeared only in that sentence. 3. A comment claimed "the charge path reads this same predicate before taking the money" about the self-dealing exclusion. There is no charge path and no shared predicate — slice 1 has no self-dealing check at all. Corrected, and turned into an explicit obligation on slice 2b: call this before the debit or extract it, because the exclusion is NOT enforced upstream today. 4. BlockAuthorFeeAccrualStatus and BlockAuthorFeeEntryType were exported and then never referenced, including inside their own file — every status was written as a bare literal, so a typo'd 'setled' would have compiled and matched nothing. Now bound to named constants used at all ten write and compare sites. 5. The settlement job's getJobDate/setLastRun cursor gated nothing. settleBlockAuthorFees scans `status: 'accrued'` with no date filter, so the cursor was read only to interpolate into a log line and then written back — two DB round-trips and a persisted KeyValue row no branch consulted. The real idempotency mechanism is the deterministic externalTransactionId. Removed; a cursor that reads like a run-once guard while guarding nothing is worse than none. Also: the D<n> labels had no referent inside this repository — they index an internal decision memo that is not here, so a bare "D6" was authority a reader could not resolve. D1, D6 and D7 are now stated in full at the top of the service, and the note records that D8 and D10 implement nothing in this file. And a caveat that was missing entirely: a newly added cron is NOT picked up by a deploy. Jobs are discovered through /api/internal/get-jobs and the external scheduler needs an explicit refresh, so this job will not be dispatched until someone performs that out-of-band step. Merged does not mean running. VERIFICATION after the fixes pnpm typecheck OK - 0 type errors settlement suite 17 passed mutation sweep 6/6 still killed, each by its own named test — re-run after the literal-to-constant refactor, since that touched all ten status/entry comparison sites LEFT OPEN, deliberately — operator decisions, not mine: * clawbackBlockAuthorFee has ZERO production callers, and its negative carry-forward arm is unreachable until something has settled (two PRs away). * The base rate: bulk-payout-block-attributions.ts is the same machine — registered daily job, idempotent mint, clawback carry-forward, net<=0 hold — built 2026-05-31 and still unwired. mintPayoutForOwner has no production caller to this day. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(app-blocks): retire the clawback from slice 2a — it had zero callers and an unreachable arm Round 0's deletion candidate, taken. The clawback reverses a charge this PR deliberately does not make, so it cannot be needed yet: * clawbackBlockAuthorFee had ZERO production callers — only its own tests. * Its negative carry-forward arm is reachable only once a row has status 'settled'. Nothing has settled and nothing can until the charge path lands and the job runs for a day — at least two PRs away. The base rate is what made this decisive rather than tidy. bulk-payout-block-attributions.ts is the same machine — registered daily job, idempotent mint, clawback carry-forward, net<=0 hold — built 2026-05-31 and still unwired; mintPayoutForOwner has no production caller to this day. The way that rail rotted is that it shipped ahead of its consumer. Slice 2b brings the clawback back together with the refund path that drives it. REMOVED clawbackBlockAuthorFee, ClawbackReason, ClawbackBlockAuthorFeeResult the entry_type column, its composite unique, and BlockAuthorFeeEntryType the amount-SIGN check (there are no negative rows now) the entry_type check — already measured unreachable, shadowed by the sign check the 'clawed_back' status the non-positive bucket hold, bucketsSkippedNonPositive, and its test 5 clawback tests REPLACED, so the invariant lives where it can actually be violated CHECK (fee_buzz > 0). The settlement job's old "a bucket can never sum to <= 0" branch was unreachable the moment negative rows went away, and an unreachable guard reads as coverage while providing none. The rule is now enforced at the write (accrueBlockAuthorFee refuses fee <= 0) and in the schema, not in a dead branch. Unique key is now workflow_id alone. VERIFICATION pnpm typecheck OK - 0 type errors (one real error caught first: the job still logged the removed bucketsSkippedNonPositive field) settlement suite 11 passed (was 17; 6 removed with the clawback) mutation sweep 4/4 killed, each by its own named test — self-dealing, D6 bucket key, dedup-key composition, zero-fee guard positive control +1000 on the minted amount -> 2 tests red, so the harness genuinely executes the code SCHEMA RE-APPLIED — dev then prod, table dropped and recreated (it was empty and nothing referenced it). The drop was GUARDED: a DO block re-counts rows and inbound constraints at the moment of the drop and raises rather than destroying anything, so a row that landed in between aborts the transaction. Guard printed "0 rows, 0 inbound references" on both. dev cnpg-cluster-dev-1 17 cols, 4 checks, 0 rows prod cnpg-cluster-nvme0-5 17 cols, 4 checks, 0 rows (lock_timeout 5s) standbys -4 and -7 converged to the same after WAL replay; -7 lagged ~30s and was polled to convergence rather than assumed block_spend_attribution unchanged Constraints exercised on dev, not merely counted: a valid row inserts, and 6 negative controls are each rejected BY THEIR OWN named constraint — no shadowing now that the redundant entry_type check is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(app-blocks): round 1 — the settlement rail could lose money one way and pay twice the other Round 1 returned "needs rework" with four blockers. All four fixed. CI was RED at the previous head and I had not checked it after pushing; that is what surfaced three of these. 1. 🔴 THE MINT RESULT WAS DISCARDED, AND ROWS FLIPPED REGARDLESS. createBuzzTransactionMany does NOT throw on a per-transaction failure — its own comment says an insufficientFunds or otherwise-rejected result "is dropped from BOTH arrays: the money did NOT move and it is otherwise invisible". It also filters out transactions failing `fromAccountId !== toAccountId && amount > 0` before calling the service. The old code awaited it, ignored the return, and flipped every row to settled: the author was never paid while the ledger asserted they were, and status='accrued' was the only handle that could have found it afterwards. Now reconciled by COUNT (successes come back as opaque ids, so the count is the only thing that reconciles); a conflict counts as settled because the money already moved under that key. If the count does not account for every bucket, NOTHING is flipped — we cannot tell which dropped, so the rows stay accrued and retry. The precedent is challenge-funding.ts, which carries the same reconciliation and the same comment. 2. 🔴 THE DEDUP KEY NAMED A DAY BUT THE SCAN WAS UNBOUNDED, so it was wrong in BOTH directions, and the module comment confidently asserted the opposite. (a) SILENT LOSS — a second run on the same day swept up rows accrued since the first, minted them under the SAME key, got a benign-looking conflict, and flipped them settled. Money gone. Compounded by (1), which made it invisible. (b) DOUBLE PAY — if the flip failed after a good mint, rows stayed accrued until the next run 24h later, under a DIFFERENT date key, and were minted again. The comment claimed "the next run re-derives the same key"; on a daily cron it never does. Fixed by bounding the scan at the day boundary: a run for day D settles only rows accrued strictly before the start of D. Every row now belongs to exactly one settlement day, derivable from the row, so (a) cannot sweep fresh rows and (b) re-derives the same key and conflicts instead of paying twice. 🔴 This is also why deleting getJobDate/setLastRun last round was not the whole story. It gated nothing AS WRITTEN — that part was right — but it was standing in for a run-once guard, and removing it without replacing the mechanism left (a) exposed. The boundary is the replacement, and it is stronger: it bounds the ROWS rather than the invocations, so it holds under the concurrent runs createJob's lock expiry can produce. 3. 🔴 CI RED — three checks, all mine: * block-spend-attribution-status-default — and this one was emitting a FALSE claim about a table this PR does not touch. The guard filtered migration files with a file-level `sql.includes(TABLE)`, which reads PROSE: this migration's comment header explains why it is a SEPARATE table from block_spend_attribution, which passed the filter; the `"status" TEXT … DEFAULT` regex then matched THIS table's own column and won on last-wins sort order. It reported `provisions DEFAULT 'accrued' … while the schema says @default("tracked")` about a column nobody had changed. That is worse than a false red: the guard exists because a default the payout read does not select makes rows invisible to payout with no error (#4036), so anyone "fixing" it by following its message ships that defect. Fixed in the GUARD — comments stripped, file split into statements, only statements naming the table scanned — not by rewording this migration, which would leave the next table to trip it. * app-access.call-site-ledger — the guard fails on GROWTH because it forces a collaborator decision. Registered: the author fee accrues to the app OWNER only, deliberately not widened to ACCEPTED collaborators. That is the ledger's existing D4 applied to a new earnings surface, not a new decision — earnings are appOwnerUserId-keyed precisely so an ex-owner keeps what they accrued before transferring an app away, and widening the WRITE would make that inexpressible. Resolving the owner at settlement instead would retroactively re-route earnings on every transfer. * ESLint + Prettier — both new files were unformatted. Formatted. Plus no-direct-shared-module-mock: the test now uses the canonical db/logging mocks via the repo's own codemod. 4. 🟡 buzz_type had no CHECK — the one money-critical column without one, while status and governing_leg both had theirs. BuzzAccountType also contains BANK types (creatorProgramBank, cashPending, cashSettled, club), and settlement cast the column to it unchecked, so a junk or bank value would surface only as a silently-dropped transaction. Now CHECK IN ('blue','green','yellow','red'), verified on dev: 'creatorProgramBank' rejected by that constraint by name, 'blue' accepted as a positive control. Also fixed: the settlement scan read the REPLICA while the flip wrote the primary (replica lag would re-bucket an already-settled row); buzzMinted was incremented even when updateMany matched 0 rows, so the job logged an affirmative "minted" for money it had not moved; and the settlement key was built from two independent spellings, only one of which was pinned. TESTS — the absences round 1 named were structural, not oversight: the fake resolved createBuzzTransactionMany to `undefined`, which can express neither a conflict nor a drop. It now returns `{ transactions, conflicts }` like the real one, and five tests were added: mint-did-not-reconcile flips nothing, a conflict counts as settled, the day boundary is midnight of the settled day, a concurrent flip claims no buzz, and one key serves both the mint and the row stamp. VERIFICATION pnpm typecheck OK - 0 type errors the four red suites 50 passed (4 files) prettier --check clean (positive control: the earlier --write listed all five paths by name, so they do resolve) mutation sweep 6 real mutants, 6 KILLED, each by its own named test: discarded mint result, missing day boundary, replica read, unconditional buzz count, duplicated key spelling positive control +1000 on the minted amount -> 2 tests red 1 SURVIVOR, and it is an EQUIVALENT MUTANT, not a gap: dateStr off `date` vs off `boundary` is the same string for every input (toISOString is always UTC and boundary is midnight of that same UTC day; checked across day edges and a year boundary). Recorded in-code so nobody writes a test asserting a difference that cannot exist. SCHEMA re-applied dev then prod for the buzz_type CHECK, same guarded drop (re-counts rows and inbound references at the moment of the drop, raises rather than destroying). All three prod instances: 17 cols, 5 checks, 0 rows. Standby -4 lagged ~30s and was polled to convergence rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(app-blocks): round 2 — the settlement key named the RUN day, so a deferred retry paid twice Round 2 found that round 1's flagship fix did not do what it said, and that its fail-closed branch made things worse. Both confirmed against the code before acting; both were mine. 🔴 F1 — THE KEY WAS DERIVED FROM THE INVOCATION, NOT FROM THE ROW. `dateStr` came from `args.date` — the day the job ran — while the scan admitted every accrued row below the boundary, i.e. all of history, bucketed by (owner, buzzType) with no accrual day. So a row whose flip failed on 09-18 was re-scanned on 09-19 and minted under `…-2026-09-19-…`: a NEW key, no conflict, owner paid twice. That is exactly the failure mode the module comment claimed the day boundary had fixed — asserted three times, false every time. The sentence "an unflipped row still belongs to its original day, so its key is unchanged" was the worst of them: the day never came from a row. 🔴 F2 — THE FAIL-CLOSED BRANCH FED F1. One dropped bucket meant NOTHING in the batch was flipped, including buckets whose money had already moved — and those then sat `accrued` until the next day, where F1 re-minted them. Round 1 introduced this while fixing an under-payment, and its comment claimed the trade was "money owed, which is recoverable". It was money PAID TWICE, which is not. 🟡 F3 — a same-day second run still lost money via `take` truncation: run 1 mints a partial bucket and flips, run 2 picks up the remainder, builds the same key, gets a conflict, and flips those rows without ever paying for them. A conflict says that KEY minted; it does not say those ROWS were in it. THE REDESIGN — one complete accrual day at a time, keyed on the row. * The loop settles the OLDEST unsettled accrual day, scanning exactly [dayStart, dayEnd) rather than everything below a boundary. * The bucket carries `accrualDay`, and `keyForBucket` builds the key from it. Every component now comes from the rows; nothing comes from the clock. A retry tomorrow, next week, or after a month with the flag off re-derives the SAME key and conflicts. F1 closed. * A day that exceeds `limit` is SKIPPED WHOLE and reported (`daysTruncated`), never cut. `take: limit + 1` makes the overflow detectable instead of silent. A partial bucket is what F3 lost money to, so the rule is that a bucket is always settled whole. F3 closed. * ONE BUCKET PER MINT CALL, so a drop is attributable. createBuzzTransactionMany reports only counts and opaque ids, which is why the batched version could not say which bucket failed and answered by flipping nothing. Per-bucket makes it answerable: this bucket moved, or it did not — and its peers are unaffected. F2 closed. 🟢 F6 — the "EQUIVALENT MUTANT" comment was FALSE as written, and I had checked it with fixtures that could not see the counter-example. Date.UTC maps years 0-99 to 1900+y, so year 0026 gives `0026-…` from one spelling and `1926-…` from the other. Unreachable in production, but the comment forbade writing the test that would have found it. Gone with the rewrite. 🟡 F4 — the job's prose contradicted the service: it still said the scan had "no date filter" (the recorded rationale for deleting the cursor) and "since the last run". Both corrected, and the cursor's real justification stated: idempotency lives in the ROWS, not the invocation, which is what holds when createJob's lock expires and two runs overlap. 🟢 F5 — the status-default guard stripped only `--` comments while Prisma-generated migrations in this repo open with `/* Warnings */` headers, so the prose was wider than the code. Both comment forms are stripped now. VERIFICATION pnpm typecheck OK - 0 type errors affected suites 52 passed (4 files) prettier clean mutation sweep 7 mutants, 7 KILLED, each by its own named test: run-day key, truncated oversized day, flip-on-drop, un-day-scoped scan, D6 bucket collapse, unconditional buzz count positive control +1000 on the minted amount -> 2 tests red No SURVIVORS this round. The previous round reported one and explained it away as equivalent; that explanation was wrong (F6), which is a reason to distrust a survivor-with-a-story rather than to be reassured by one. Schema unchanged — no migration in this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(app-blocks): round 3 — the day loop could never advance past a day it could not finish Round 3's verdict was "merge after fixing 🔴-1". The money path held this time — eleven of thirteen claims landed clean and no double-payment or silent-loss path survived — but the rewrite introduced a LIVENESS defect, and my own mutation sweep could not have caught it. 🔴 A DAY THE LOOP COULD NOT FINISH BLOCKED EVERY LATER DAY. The loop re-derived "the oldest unsettled day" on every iteration with nothing to advance past one that did not complete. Two arms: * an OVERSIZED day hit `break` and stayed the oldest forever, so every later day was blocked permanently — and since the job passes no `limit`, the only recovery was a code change and a deploy; * a persistently REJECTED bucket left its rows `accrued`, so the next iteration re-selected the same day and re-minted it, burning all 30 iterations on one stuck owner while every other author stopped being paid. Neither risks money — the row-derived key still makes a retry conflict — but both halt settlement for everyone else behind a single log line. The previous round's comment even stated the rule for the second arm ("the day is still the oldest, so the next iteration would select it again and spin") sixty lines above the code that did exactly that. FIXED with a monotonic per-run cursor: the day selection asks for the oldest day AT OR AFTER `cursorFrom`, and the cursor advances to `dayEnd` BEFORE any early exit, so every path leaves its day behind. A stuck day now costs one iteration and is retried on the next run instead of wedging the queue. The oversized and emptied-day arms `continue` rather than `break`, which is only safe because the cursor makes a spin impossible. 🟡 THE MINT IS NOW WRAPPED. The buzz client THROWS on any non-2xx and its retry allowlist covers only connection-level errors, so a 5xx throws on the first response. Unwrapped, bucket 1 of N aborted buckets 2..N, the day loop and the completion log — and splitting one batched call into N per-bucket calls multiplied that exposure by N, so this was a hazard the previous round's own fix created. A throw is now handled exactly like a drop: rows stay `accrued`, the same key is re-derived next run, peers still settle. 🟢 Two claims corrected rather than reworded: the module header still described a `Math.floor` the previous commit deleted (the only remaining `Math.floor` in the file was inside the sentence about it), and "a conflict is the same payment" was stated unconditionally when it holds only while no row can JOIN an already-minted (day, owner, currency) group. That precondition is now written down, along with what breaks it — a backfill writing a historical `accrued_at` would turn it into a silent underpayment. 🔴 MY OWN MUTATION SWEEP WAS THE REAL FINDING, and it is the one worth carrying forward. Every settlement test used a single-day fixture, so the loop body ran exactly once in every test. The multi-day walk, the cursor, `maxDays` and the continue-vs-break semantics were structurally unreachable — a mutant over any of them would have killed nothing. Last round's "7 mutants, 7 killed" was therefore a true statement about seven mutants that did not include the control flow that round had just rewritten. A sweep is only as wide as the mutants you imagined. Fixed by a `days()` fixture that drives N days, plus 7 tests: the multi-day walk, an oversized day not blocking later days, a dropped bucket not blocking later days, a thrown mint treated as a drop, the cursor advancing strictly, `maxDays`, and a day that empties under the loop. ⚠️ AND THAT FIXTURE IMMEDIATELY EXPOSED A TEST-ISOLATION LEAK: `vi.clearAllMocks()` clears call history but NOT `mockResolvedValueOnce` queues, so a day queue left by one test was consumed by the next and the suite was order-dependent. The mocks fed with `...Once` are now explicitly reset. VERIFICATION pnpm typecheck OK - 0 type errors affected suites 59 passed (4 files) prettier clean mutation sweep 11 mutants, 11 KILLED, each by its own named test — five of them loop control flow (no cursor advance, cursor absent from the query, oversized-day break, empty-day break, maxDays ignored) and six money path (run-day key, truncate oversized day, uncaught throw, D6 collapse, ...) positive control +1000 on the minted amount -> 2 tests red Two mutants needed a second pass and both are recorded rather than quietly re-run: `empty_day_breaks_run` SURVIVED the first sweep — a genuine gap, closed by the new empty-day test — and `oversized_breaks_run` was SKIPPED because its pattern matched 10 sites, which is a sweep that reports nothing while looking like it ran. Schema unchanged; no migration in this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(app-blocks): round 4 — the wrap stopped one statement short, and a false claim about retries Round 4's verdict was SAFE TO MERGE with no blockers: the three liveness defects round 3 set out to fix are genuinely fixed, verified by watching five mutants go red rather than by reading the claims. What it found was four 🟡s of one shape — the code is right and the comment claims more than the code does — plus two gaps in my own mutation sweep. 🟡 THE RETRY CLAIM WAS FALSE, IN THE REASSURING DIRECTION. The comment said the buzz client's "retry allowlist covers only connection-level errors — a 5xx or a 504 throws on the first response". Traced and wrong: `createTransactions` calls `post` with no options, so `shouldRetry` is undefined, and `withRetries` reads `shouldRetry ? predicate(...) : true` under a comment saying "Absent predicate keeps the historical behaviour: retry everything". `isSafeToRetry` is exported and never applied on this path. So a 5xx IS retried at the client default (3), and a run with N payable buckets can issue up to 4N POSTs during an outage. That paragraph was the file's only statement about mint cost, so it is what someone would have sized a timeout or an alert against. 🟡 THE DIAGNOSTIC COULD NOT TELL PERMANENT FROM TRANSIENT. `buzzService` is built with a `mapError`, so every non-2xx arrives as a TRPCError carrying the fixed string "An unexpected error ocurred, please try again later". A permanent 400 and a transient 503 produced BYTE-IDENTICAL log lines, repeated daily forever, so the permanent one was indistinguishable from noise — and the repo already ships `getBuzzApiStatus` to read the real status back through the wrapper. Now logged. 🟡 THE WRAP STOPPED ONE STATEMENT SHORT. The mint was wrapped; `updateMany` one line below it was not — and the sentence justifying the wrap ("bucket 1 of N throwing aborts buckets 2..N, the day loop and the completion log") stayed true, verbatim, of the unwrapped call. `id: { in: rowIds }` can carry up to `limit` ids, so a statement timeout there is not exotic. Money was safe either way; what it cost was every remaining bucket and every later day in the run. Wrapped, with its own log line — a flip that throws after a successful mint is the one case where money moved and the rows do not say so. 🟡 THE CURSOR BOUNDS BLOCKING WITHIN A RUN, NOT ACROSS RUNS — and round 3's comment claimed the stronger property ("nothing is abandoned permanently"). Both stuck-day arms are permanent: an oversized day is unsettleable until someone raises `limit`, and a persistently-rejected owner's rows stay accrued forever. Each is re-selected on every later run, so 30 accumulated stuck days would consume the whole budget and settlement would stop for everyone — the same end state the cursor was added to prevent, reached 30 days later. `maxDays` now counts only PRODUCTIVE days, with `maxIterations` (default maxDays * 4) as the absolute bound. 🟢 Removed an unkillable clause rather than leaving it to read as a guard: `threw === null &&` in the `moved` expression. `mint` is assigned only inside the `try`, so it is null on every throw path and the expression was already false there — measured, deleting it left the suite green. 🟢 Completed the test reset list: the comment claimed EVERY `...Once` mock was reset, while `create` — fed `mockRejectedValueOnce` twice — was not. 🔴 TWO GAPS IN MY OWN SWEEP, both closed, both worth recording: * `maxIterations`'s DEFAULT was unpinned — the only test exercising the cap passed one explicitly, so a mutant replacing the default SURVIVED. * `threwStatus` was unpinned — a mutant nulling it SURVIVED. And the first fix was initially WORSE than the gap: feeding the same stuck day forever "killed" the mutant only by HANGING the suite — 43s, `tests 0ms`, no named failure, indistinguishable from a CI timeout. It was also a mock artifact: in production the cursor is monotonic and bounded by `boundary`, so an unbounded default could never hang. The day supply is finite now and the mutant fails an assertion in 5ms. ⚠️ AND A REPORTING CORRECTION I OWE: round 3's sweep was reported as "11 killed, each by its own named test". Two of the five loop mutants share one killer, so the independence claim was overstated. This round's sweep prints the FULL killer list per mutant rather than the first. VERIFICATION pnpm typecheck OK - 0 type errors affected suites 64 passed (4 files) prettier clean mutation sweep 9 mutants, 9 KILLED. Killer counts reported rather than assumed: flip-unwrapped 1, stuck-days-toward-budget 1, maxIterations-default 1, threwStatus 1, mint-throw 1, D6 collapse 1, no-cursor-advance 1, run-day key 5. positive control +1000 on the minted amount -> 2 tests red Two mutants needed a precise anchor rather than a shell pattern (18 and 10 incidental matches respectively) — a sweep that SKIPS reports nothing while looking like it ran, which is the same failure shape as a silent zero. Schema unchanged; no migration in this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(app-blocks): round 5 — the productive-day fix covered one of the two arms it named, and a test I reported as closing a gap did not Round 5 returned three 🟡s. Two of them refute claims made in the previous round's commit message, and I verified both against the code before acting. 🟡 F1 — THE PRODUCTIVE-DAY COUNT DELIVERED HALF OF WHAT ITS COMMENT PROMISED. `productiveDays += 1` fired as soon as a day had rows, BEFORE a single mint was attempted. The comment enumerated two permanent stuck arms and said the count now excluded both. Only the oversized arm was excluded — it `continue`s above the increment. The persistently-rejected-owner arm, named verbatim in the same comment, reached the increment and was counted productive even when every bucket dropped and zero rows flipped. So one permanently-rejected owner still accrues one stuck-but-"productive" day per day, and thirty of them exhaust `maxDays` on stuck days alone — settlement stops for everyone, which is the failure the count was added to prevent. The test named for that behaviour only ever exercised the oversized arm (`limit: 1`), so it passed while its own arm was uncovered. Now counted AFTER the bucket loop, conditional on something landing, and pinned by a test that drops every bucket on two days and asserts the third still settles. 🟡 F2 — THE MOTIVATING EXAMPLE WAS THE ONE CASE WHERE THE CLAIM IS FALSE. The new `threwStatus` comment said "a permanent 400 and a transient 503 produce BYTE-IDENTICAL log lines". `mapError` names 400, 404 and 409 explicitly ("Your request is invalid", "Not found", "There is a conflict with the transaction"), so those three were already distinguishable. The fix is still worth having — 401, 403, 408, 429, 500, 502 and 503 all fall to `default` and genuinely are identical, so a permanent auth failure versus a transient outage was the real indistinguishable pair. The example was wrong, not the reason; corrected rather than reworded. 🟡 F3 — A TEST I REPORTED AS CLOSING A MUTATION GAP DID NOT CLOSE IT. The `threwStatus` test used a plain Error carrying a `status` property and asserted `toHaveProperty('threwStatus')` — a single-argument EXISTENCE check. `getBuzzApiStatus` returns undefined for that shape, so the field logged `null`, and the mutant `threwStatus: null` SURVIVED while the test passed. Re-measured here before fixing: mutant applied, 30/30 still green. The previous commit and PR comment both recorded that gap as closed. It now uses a real `BuzzApiError` inside a `TRPCError` cause — the shape production actually produces — and asserts the VALUE is 503. 🟡 F4 — WRAPPING THE FLIP MADE ITS FAILURE SILENT. Before the wrap an `updateMany` throw failed the job and was loud; after it, the run returns success with only an Axiom line. A systemic flip failure would report `rowsSettled: 0` on a job that says it succeeded, every night, while money left on day one of each bucket. Added `flipFailures` to the result and the job log — the one counter that means money moved without a settled row — and put the amount on the log line, which carried `settlementKey` but not the sum. 🟢 F5 — the flip-failure message asserted a state the code cannot observe. A connection drop after the UPDATE commits raises with the rows already flipped, so "money moved, rows still accrued" would be the opposite of the truth on a line an operator acts on. Now "mint landed, flip did not confirm". 🟢 F6 — the `buzzMinted` comment claimed `count > 0` distinguishes "a payment this run did not make". It does not: it cannot tell a fresh mint from a conflict on a key an earlier run already paid, and the wrapped flip makes that path ordinary. The cross-run total stays correct, so this is reporting, not money — stated rather than reworded, because the exact claim is what a reader would rely on. VERIFICATION pnpm typecheck OK - 0 type errors affected suites 65 passed (4 files) prettier clean mutation sweep 3 mutants re-run against the new guards, 3 KILLED, each by its own named test: threwStatus nulled (this one SURVIVED before the fix — re-measured, not assumed), productiveDays counted unconditionally, flipFailures not counted. ⚠️ SCOPE OF THAT SWEEP, STATED BECAUSE THE LAST FOUR ROUNDS OVERSTATED THEIRS: it covers the three guards this round added or repaired. The nine mutants from round 4 were not re-run. Schema unchanged; no migration in this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(app-blocks): split slice 2 — this PR is the ACCRUAL LEDGER only, settlement goes to 2b The settlement rail has no consumer. `accrueBlockAuthorFee` has zero callers on main, so the table it writes accrues nothing, so the daily job it feeds settles nothing — a cron, a Buzz-minting path and 25 tests whose first real exercise would be the day a charge path lands. That is a lot of unexercised money-moving code to carry through review on the strength of a table that is empty by construction. It moves to `zach/app-blocks-author-fee-slice2b` and comes back with the viewer-charge path that gives it rows. What leaves this PR: - `settleBlockAuthorFees`, `SettlementBucket`, `SettleBlockAuthorFeesResult`, `utcDayStart`, `keyForBucket` - `src/server/jobs/settle-block-author-fees.ts` and its registration in the run-jobs webhook (reverted to the main version verbatim) - the 25-test settlement suite What stays — the ledger, which is the part that has to exist first: - the migration, unchanged and NOT re-timestamped; it is already hand-applied to dev and all three prod instances - `accrueBlockAuthorFee` and the six tests covering it - the `bafa_` id helper - the call-site-ledger registration and the migration-comment filtering fix RENAMED `author-fee-settlement.service.ts` to `author-fee-accrual.service.ts`. A file named for settlement that settles nothing is the same defect this PR's own audit ladder found four times in rounds 4 and 5 — a name claiming more than the code does. The call-site ledger keys on the file path and fails on both GROWTH and SHRINK, so its entry moved with the file. `STATUS_SETTLED` is now exported despite nothing in this slice reading or writing it. It names one of the two states of a CHECK-constrained column this slice's migration ships, and 2b is the writer; exporting it keeps one spelling of the literal across both slices rather than letting 2b re-declare it, where a typo would match no row and fail silently. Its comment says so, so its presence is not read as evidence that anything settles. Kept as a commit on top rather than a force-push: the six audit rounds behind the settlement code are the record of how it got correct, and 2b inherits that history. The PR diff is computed against the merge base, so what reviewers see is the ledger-only slice either way. Rail is still DARK: 0 callers of `accrueBlockAuthorFee`, flag `app-blocks-author-fee-enabled` is `enabled: false`. Tests: 4323 -> 4298 in the blocks suite, exactly the 25 settlement tests, moved not deleted. The four-suite run goes 65 -> 40. Typecheck 0 errors; the 43-file lint-rules suite is green at 599. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Table of Contents
About the Project
Our goal with this project is to create a platform where people can share their stable diffusion models (textual inversions, hypernetworks, aesthetic gradients, VAEs, and any other crazy stuff people do to customize their AI generations), collaborate with others to improve them, and learn from each other's work. The platform allows users to create an account, upload their models, and browse models that have been shared by others. Users can also leave comments and feedback on each other's models to facilitate collaboration and knowledge sharing.
Tech Stack
We've built this project using a combination of modern web technologies, including Next.js for the frontend, TRPC for the API, and Prisma + Postgres for the database. By leveraging these tools, we've been able to create a scalable and maintainable platform that is both user-friendly and powerful.
- DB: Prisma + Postgres
- API: tRPC
- Front-end + Back-end: NextJS
- UI Kit: Mantine
- Storage: Cloudflare
Getting Started
To get a local copy up and running, follow these steps.
Prerequisites
- Docker, with Compose v2 (
docker compose, not the retired hyphenateddocker-compose). The database, Redis, MinIO, Meilisearch, ClickHouse and the mail catcher all run as containers. - Node.js
24.19.0. Not "20 or later" —package.jsondeclaresengines.node: ">=24.0.0 <25". The exact version lives in.nvmrc; CI installs that file's version and the production image is built on the same one, sonvm use(or any tool that reads.nvmrc) is the right way to get it. Note that nothing stops you:pnpm installonly printsWARN Unsupported engineand carries on, so the wrong major surfaces later as odd test failures rather than as a refusal at install time. - pnpm. This repo is pnpm-only, and this one is enforced —
npm installexits 1 via thepreinstallonly-allow pnpmhook.corepack enablewill pick up thepackageManagerfield for you. - Make (optional).
Installation
Standard setup
git clone https://github.com/civitai/civitai.git
cd civitai
nvm use # reads .nvmrc -> 24.19.0
corepack enable
git submodule update --init event-engine-common
cp .env-example .env.development
docker compose -f docker-compose.base.yml up -d
pnpm install
pnpm dev
Optional: Nix flake
Optional, and not the supported default. The standard setup above is what the project expects and what CI builds; nothing in the repo requires Nix, and you can ignore this section entirely. It exists because NixOS cannot use Prisma's published engines (there is no
linux-nixosbuild), so a flake is the practical way to work on this repo there. If you are not on NixOS and not already a flakes user, skip it.
The flake owns the toolchain, so you do not install Node or pnpm yourself:
git clone https://github.com/civitai/civitai.git
cd civitai
nix run .#dev
That single command checks Docker is usable, checks out the
event-engine-common submodule, creates .env.development from .env-example
if you do not already have one, starts the container stack, waits for Postgres,
runs pnpm install, and then starts the dev server on
http://localhost:3000. Every step is idempotent — it is
safe to re-run in a checkout that already works, and it will not overwrite your
.env.development or touch your data.
Useful variants:
nix run .#dev -- --no-start # bootstrap only, leave the services running
nix run .#dev -- --full # also start the signals/buzz containers (see below)
nix run .#doctor # check the flake's pins against the repo
nix flake check # the same checks, plus their own self-test
For an interactive shell with the same toolchain, use nix develop, or copy
.envrc.example to .envrc and run direnv allow to get it
automatically on cd.
With devcontainers
⚠️ Known out of step:
.devcontainer/public/docker-compose.ymlpinsmcr.microsoft.com/devcontainers/typescript-node:1-22, i.e. Node 22, which is outside this repo'sengines.noderange.pnpm installwill warn rather than stop, so the container comes up and then misbehaves in ways that look like your branch. There is no1-24tag (the template major moved on);3-24is the closest equivalent. Not changed here because it could not be exercised.
⚠️ Important Warning for Windows Users: Either clone this repo onto a WSL volume, or use the "clone repository in named container volume" command. Otherwise, you will see performance issues.
- Open the directory up in your IDE of choice
- VS Code should prompt you to "Open in container"
- If not, you may need to manually run
Dev Containers: Open Folder in Container
- If not, you may need to manually run
- For other IDEs, you may need to open the
.devcontainer/devcontainer.jsonfile, and click "Create devcontainer and mount sources" - Note: this may take some time to run initially
- VS Code should prompt you to "Open in container"
- Run
make run
The signals and buzz services
docker-compose.base.yml holds everything a contributor needs (and is also what
nix run .#dev starts). The extra services in docker-compose.yml
(signals, buzz) come from private ghcr.io images, so they only work for
internal members:
- create a GitHub personal access token with
read:packages - set it as
CR_PAT echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin- then
docker compose up -d(or, with the flake,nix run .#dev -- --full)
After the first start
- Edit
.env.development. Most defaults work out of the box; these do not:- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
minioadmin) — note it is port 9001, port 9000 is the S3 API itself — go to "Access Keys", click "Create Access Key", and copy the key and secret intoS3_UPLOAD_KEY/S3_UPLOAD_SECRETandS3_IMAGE_UPLOAD_KEY/S3_IMAGE_UPLOAD_SECRET. WEBHOOK_TOKEN— any random string; it authenticates requests to the webhook endpoint.EMAIL_USER,EMAIL_PASS, andEMAIL_FROM(a valid email format) — any values, but they must be set for user registration to work.
- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
- On an empty database, populate it. These are slow and destructive, which is
why no bootstrap runs them for you:
make run-migrations make reseed - Visit http://localhost:3000.
Please report any issues with these commands to us on discord.
* Note that account creation will run emails through maildev, which can be accessed at http://localhost:1080.
Altering your user
- First, create an account for yourself as you normally would through the UI.
- You may wish to set yourself up as a moderator. To do so:
- Use a database editor (like DataGrip) or connect directly to the
DB (
PGPASSWORD=postgres psql -h localhost -p 15432 -U postgres civitai) - Find your user (by email or username), and change
isModeratortotrue
- Use a database editor (like DataGrip) or connect directly to the
DB (
Known limitations
Services that require external input will currently not work locally. These include:
- Orchestration (Generation, Training)
- Signals (Chat, Notifications, other real-time updates)
- Buzz
Contributing
Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the repository to your own GitHub account.
- Create a new branch for your changes.
- Make your changes to the code.
- Commit your changes and push the branch to your forked repository.
- Open a pull request on our repository.
If you would like to be more involved, consider joining the Community Development Team! For more information on the team as well as how to join, see Calling All Developers: Join Civitai's Community Development Team.
Data Migrations
Over the course of development, you may need to change the structure of the database. To do this:
- Make your changes to the
packages/civitai-db-schema/prisma/schema.full.prismafile. Notschema.prisma— that one is gitignored and regenerated fromschema.full.prismabyscripts/generate-slim-schema.json everypnpm run db:generate, so edits to it are silently overwritten. - Run
pnpm run db:migrate:empty "brief description here". This createspackages/civitai-db-schema/prisma/migrations/YYYYMMDDHHmmss_brief_description_here/migration.sqlfor you, in the one directory Prisma reads. To create it by hand instead, use that same path — not theprisma/migrationsdirectory at the repo root, which predates the monorepo layout and is no longer read. - Put your sql changes in the generated
migration.sql- These are usually simple sql commands like
ALTER TABLE ...
- These are usually simple sql commands like
- Run
make run-migrationsandmake gen-prisma - If you are adding/changing a column or table, please try to keep the
gen_seed.tsfile up to date with these changes.
Sponsors
Support this project by becoming a sponsor. Your logo will show up here with a link to your website.
License
Apache License 2.0 - Please have a look at the LICENSE for more details.