mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
Zachary Lowden
c442286109
* fix(models): chunk the sale-badge lookup so a scrolled feed stops 400ing `model.getActiveSales` caps `ids` at 500. `useModelSaleBadges` fed it the whole accumulated list of an infinite feed, so from roughly the fifth page of cards onward EVERY call was rejected by input validation before the resolver ran. The sale badge then disappeared from the entire grid for anyone who scrolled — and because a rejected input is a 400, nothing watching server errors ever saw it. Chunked client-side rather than raising the cap. The cap is protecting real per-id work: `getActiveSalesForModels` resolves each id through the per-id cache, whose `packed.mGet` decomposes into one Redis GET per id on a cluster, and every id that misses lands in a raw `IN (…)` across a five-table join on the read replica. The procedure is public, so the length of that array is the only thing bounding the work — a bigger number would just move the wall. The cap and the chunk size are now one exported constant, so the client cannot drift past what the server accepts. Reuses the existing arrival-order chunker instead of writing a third copy of it. Moved it from `Sticker/sticker.util` to `shared/utils/chunk-ids` and renamed it `chunkIds`: two callers already had nothing to do with stickers, and importing it from there would have pulled the cosmetics/zustand graph into the model feed bundle. Arrival order is load-bearing, not incidental — sorting reshuffles every chunk boundary as a feed appends, changing every key and refetching the whole surface each page. Regression coverage drives the hook and validates each request it builds against the procedure's own schema, so a raised chunk size or a lowered cap both fail. Red at origin/main with `expected [ 1200 ] to deeply equal []` (one request of 1200 ids) and `expected 1 to be 3`; green at HEAD. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(models): keep badges on screen while a chunk loads, and size the chunk to the page Review round on the chunking fix. Four code findings, all verified against the source or the installed package before acting on them. 1. `placeholderData: (prev) => prev` is INERT under `trpc.useQueries`, so the first cut would have blanked every sale badge in the grid on every page of scroll — for feeds under the cap, i.e. exactly the population the original 400 never touched. Measured against the installed @tanstack/query-core, both arms in one run: `QueriesObserver` matches previous observers by `queryHash` alone, so a changed key builds a fresh `QueryObserver` whose `#lastQueryWithDefinedData` is empty and the placeholder resolves to `undefined`; `QueryObserver` (what `useQuery` uses, the positive control) keeps one observer for the component's life and does carry it across. The option is dropped and keep-previous is done in the hook. 2. Chunk size split from the cap and matched to the feed's page size. The trailing partial chunk re-keys on every page, so ids asked per distinct id is (chunk/page + 1)/2 — 3.0x at 500/100, 1.0x at 100/100, for the SAME number of requests per page. It also keeps a request inside both wire budgets in `~/utils/trpc`, so it stays a batchable GET instead of an unbatchable POST. The cap stays 500: it is protecting the per-id Redis fan-out, which the schema now says instead of blaming the SQL — measured on the replica, execution time is flat from 100 to 5000 ids and only planning grows. 3. `endsAt` rode the wire and nothing branched on it — the badge only formatted it. A full chunk's key is now stable, so a mounted feed could keep serving a sale that had ended, advertising a discount the model page and the charge path both refuse. Both hooks re-apply the end edge. 4. Reuse: `discountType` takes `SaleDiscountKind` from the package the server declares the output with rather than a local union restatement, and the fourth open-coded copy of the chunker folds into `chunkIds`. The schema module moves to `src/server/schema/`, which is where this repo's tRPC input contracts live and where the sibling cap constant this mirrors already sits; the app-graph guard passes with it imported client-side. Tests: the chunker's coverage moves with the chunker, and the hook's file now re-renders, so the merge memo key, the partial-load path and keep-previous are observable rather than asserted into a single synchronous render. Added a seam guard pinning the router to the shared schema — the drift that reproduces the outage was previously unguarded. Every guard was watched to fail: nine mutations, each killed by its own assertion with its own message, restored green after. The fixture is a literal above the CAP, not derived from the chunk. Sized off the chunk it sat at 220, under the cap, and the reverted hook was measurably GREEN on the headline assertion — a positive control on the fixture now keeps that honest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(models): gate the sale badge on read, and make the cap guard behavioural Second review round. Two lanes independently found the same defect in the first round's own fix, and the test lane found that the fix shipped a red suite. The end-edge check was applied where the merged map was BUILT, which is a stamp, not a gate. The keep-previous map added in the same commit therefore escaped it and could re-serve a window that had closed since it was stored — measured at 100 of 100 expired entries returned. The gate moves onto the map being handed out, and its clock is re-read on every event that changes which map that is: a chunk arriving, and falling back or recovering. The fallback transition is the load-bearing one, because it hands out the same object and an identity-keyed memo would not re-check it — the first attempt at this fix was keyed that way and the new test caught it. Residual, stated rather than implied: a feed left mounted and idle re-reads nothing, so a window closing with no scroll and no refetch stays badged. That was equally true before this hook chunked — the map was never re-checked at all — and closing it needs the gate at the per-card read in `ModelCard`, whose only tests are browser-mode and cannot run in this environment. Registering the new guard: adding a `no-divergent-*` test without its three companion entries turns `no-lint-rules-script-drift` red (5 failed / 6 passed, verified). Wired into the `test:lint-rules` script and both guard inventories — without which it also never ran under that script at all. The guard itself was spelled rather than structural and was walkable: declaring a private `const getActiveSalesSchema` with a lower cap in the router left every string check green while every request 400d. It now parses real arrays through the procedure's real parser, so a cap wrong in either direction fails wherever it was spelled. Four more mutants that survived the previous round now die: `endsAt` read as a Date only (the string payload the comment describes would throw inside a render), the single-card hook's end-edge check, the chunk size returning to the cap, and the merged map losing referential stability. The fake was also lying in a way that hid two of those: it stamped `dataUpdatedAt` on every call rather than per resolved id-set, so the merge memo never memoized under test. It now stamps per id-set and carries a string `endsAt` arm. Every guard added here was watched to fail: six mutations, each killed by its own assertion, restored green after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(models): say what the id cap actually bounds The schema comment claimed the cap was "the only thing standing between an anonymous caller and that fan-out". That is true of ONE request and wrong about a caller: the cap bounds how WIDE a single call may be, and says nothing about how many calls arrive. Reading it as an abuse control overstates it. Both copies of the claim are reworded to state the property the constant really has — one request's fan-out stays proportional to a page of cards rather than to an accumulated feed — and the ModelCardContext copy drops the same aside. * docs(models): narrow the cap guard to what it checks, and decouple a control Round-2 review found three places where a claim was wider than what backs it. The cap guard's docstring said the router cap "and the size its card surfaces chunk to are one contract across two files", but the body never reads the card surface. Measured: swapping chunkIds(modelIds, MODEL_SALE_IDS_PER_REQUEST) for a literal 400 leaves no-divergent-active-sales-cap at 4 passed (4), while useModelSaleBadges goes 3 failed | 15 passed (18). Narrowed the sentence rather than widening the body -- the only way to read the call site from a node-project guard is a source-text match, and this file already records that a spelled version of this guard was written once and shown to be walkable. It now states its scope first, names the blind spot with both measured arms, and points at the file that does pin the card-surface half, including that that file runs in the full unit suite and not in test:lint-rules. The residual comment in ModelCardContext said "left mounted and IDLE", which reads as an edge case. refetchOnWindowFocus is false app-wide and the only per-query override is staleTime, so the end-edge memo re-reads the clock only on a chunk resolving, a reconnect or a remount -- which means on any surface that has stopped growing (a profile's OnSaleSection, a search page nobody is paging, a feed scrolled to the end) the gate freezes for the life of the mount. Stated as the steady state it is, with the reason a refetchInterval is the wrong lever: it would re-issue the per-id Redis fan-out the cap exists to bound, per tick per mounted feed per user, to fix a display staleness. The GET-budget positive control built its over-budget fixture from MODEL_SALE_IDS_PER_QUERY. At 7-digit ids the serialized input is 8N + 9 chars against MAX_GET_INPUT_LENGTH (2500), so break-even is N = 312 -- lowering the cap below that turns the control's true into a false and reds the test for a reason unrelated to the property under test. Now a fixed literal at 400 (3209 chars). Comments and a test fixture only; no behaviour change. Test Files 4 passed (4) / Tests 40 passed (40). Red arm re-derived against the shipped tree (merge-base ModelCardContext + HEAD's tests): 12 failed | 10 passed (22), headline "expected 1 to be 12". typecheck 0 errors; eslint --no-cache 0 problems on all three files; prettier clean against a negative control. * docs(models): retract the cap-guard coverage claim in the two agent-facing docs Round 2 found the retracted claim surviving verbatim in the two docs this PR adds it to, while the guard's own docstring had already been corrected. Measured: the string is absent at origin/main and present twice at the PR head, so this PR introduced both copies. Both now state the guard's real scope (server side only; it cannot see the call site) and name the file that pins the other half, plus the fact that file is not in test:lint-rules -- so a test:lint-rules run alone does not cover the seam. no-lint-rules-script-drift: 11 passed (it pins names and counts, not these parentheticals, so it was always green either way). --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Table of Contents
About the Project
Our goal with this project is to create a platform where people can share their stable diffusion models (textual inversions, hypernetworks, aesthetic gradients, VAEs, and any other crazy stuff people do to customize their AI generations), collaborate with others to improve them, and learn from each other's work. The platform allows users to create an account, upload their models, and browse models that have been shared by others. Users can also leave comments and feedback on each other's models to facilitate collaboration and knowledge sharing.
Tech Stack
We've built this project using a combination of modern web technologies, including Next.js for the frontend, TRPC for the API, and Prisma + Postgres for the database. By leveraging these tools, we've been able to create a scalable and maintainable platform that is both user-friendly and powerful.
- DB: Prisma + Postgres
- API: tRPC
- Front-end + Back-end: NextJS
- UI Kit: Mantine
- Storage: Cloudflare
Getting Started
To get a local copy up and running, follow these steps.
Prerequisites
- Docker, with Compose v2 (
docker compose, not the retired hyphenateddocker-compose). The database, Redis, MinIO, Meilisearch, ClickHouse and the mail catcher all run as containers. - Node.js
24.19.0. Not "20 or later" —package.jsondeclaresengines.node: ">=24.0.0 <25". The exact version lives in.nvmrc; CI installs that file's version and the production image is built on the same one, sonvm use(or any tool that reads.nvmrc) is the right way to get it. Note that nothing stops you:pnpm installonly printsWARN Unsupported engineand carries on, so the wrong major surfaces later as odd test failures rather than as a refusal at install time. - pnpm. This repo is pnpm-only, and this one is enforced —
npm installexits 1 via thepreinstallonly-allow pnpmhook.corepack enablewill pick up thepackageManagerfield for you. - Make (optional).
Installation
Standard setup
git clone https://github.com/civitai/civitai.git
cd civitai
nvm use # reads .nvmrc -> 24.19.0
corepack enable
git submodule update --init event-engine-common
cp .env-example .env.development
docker compose -f docker-compose.base.yml up -d
pnpm install
pnpm dev
Optional: Nix flake
Optional, and not the supported default. The standard setup above is what the project expects and what CI builds; nothing in the repo requires Nix, and you can ignore this section entirely. It exists because NixOS cannot use Prisma's published engines (there is no
linux-nixosbuild), so a flake is the practical way to work on this repo there. If you are not on NixOS and not already a flakes user, skip it.
The flake owns the toolchain, so you do not install Node or pnpm yourself:
git clone https://github.com/civitai/civitai.git
cd civitai
nix run .#dev
That single command checks Docker is usable, checks out the
event-engine-common submodule, creates .env.development from .env-example
if you do not already have one, starts the container stack, waits for Postgres,
runs pnpm install, and then starts the dev server on
http://localhost:3000. Every step is idempotent — it is
safe to re-run in a checkout that already works, and it will not overwrite your
.env.development or touch your data.
Useful variants:
nix run .#dev -- --no-start # bootstrap only, leave the services running
nix run .#dev -- --full # also start the signals/buzz containers (see below)
nix run .#doctor # check the flake's pins against the repo
nix flake check # the same checks, plus their own self-test
For an interactive shell with the same toolchain, use nix develop, or copy
.envrc.example to .envrc and run direnv allow to get it
automatically on cd.
With devcontainers
⚠️ Known out of step:
.devcontainer/public/docker-compose.ymlpinsmcr.microsoft.com/devcontainers/typescript-node:1-22, i.e. Node 22, which is outside this repo'sengines.noderange.pnpm installwill warn rather than stop, so the container comes up and then misbehaves in ways that look like your branch. There is no1-24tag (the template major moved on);3-24is the closest equivalent. Not changed here because it could not be exercised.
⚠️ Important Warning for Windows Users: Either clone this repo onto a WSL volume, or use the "clone repository in named container volume" command. Otherwise, you will see performance issues.
- Open the directory up in your IDE of choice
- VS Code should prompt you to "Open in container"
- If not, you may need to manually run
Dev Containers: Open Folder in Container
- If not, you may need to manually run
- For other IDEs, you may need to open the
.devcontainer/devcontainer.jsonfile, and click "Create devcontainer and mount sources" - Note: this may take some time to run initially
- VS Code should prompt you to "Open in container"
- Run
make run
The signals and buzz services
docker-compose.base.yml holds everything a contributor needs (and is also what
nix run .#dev starts). The extra services in docker-compose.yml
(signals, buzz) come from private ghcr.io images, so they only work for
internal members:
- create a GitHub personal access token with
read:packages - set it as
CR_PAT echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin- then
docker compose up -d(or, with the flake,nix run .#dev -- --full)
After the first start
- Edit
.env.development. Most defaults work out of the box; these do not:- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
minioadmin) — note it is port 9001, port 9000 is the S3 API itself — go to "Access Keys", click "Create Access Key", and copy the key and secret intoS3_UPLOAD_KEY/S3_UPLOAD_SECRETandS3_IMAGE_UPLOAD_KEY/S3_IMAGE_UPLOAD_SECRET. WEBHOOK_TOKEN— any random string; it authenticates requests to the webhook endpoint.EMAIL_USER,EMAIL_PASS, andEMAIL_FROM(a valid email format) — any values, but they must be set for user registration to work.
- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
- On an empty database, populate it. These are slow and destructive, which is
why no bootstrap runs them for you:
make run-migrations make reseed - Visit http://localhost:3000.
Please report any issues with these commands to us on discord.
* Note that account creation will run emails through maildev, which can be accessed at http://localhost:1080.
Altering your user
- First, create an account for yourself as you normally would through the UI.
- You may wish to set yourself up as a moderator. To do so:
- Use a database editor (like DataGrip) or connect directly to the
DB (
PGPASSWORD=postgres psql -h localhost -p 15432 -U postgres civitai) - Find your user (by email or username), and change
isModeratortotrue
- Use a database editor (like DataGrip) or connect directly to the
DB (
Known limitations
Services that require external input will currently not work locally. These include:
- Orchestration (Generation, Training)
- Signals (Chat, Notifications, other real-time updates)
- Buzz
Contributing
Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the repository to your own GitHub account.
- Create a new branch for your changes.
- Make your changes to the code.
- Commit your changes and push the branch to your forked repository.
- Open a pull request on our repository.
If you would like to be more involved, consider joining the Community Development Team! For more information on the team as well as how to join, see Calling All Developers: Join Civitai's Community Development Team.
Data Migrations
Over the course of development, you may need to change the structure of the database. To do this:
- Make your changes to the
packages/civitai-db-schema/prisma/schema.full.prismafile. Notschema.prisma— that one is gitignored and regenerated fromschema.full.prismabyscripts/generate-slim-schema.json everypnpm run db:generate, so edits to it are silently overwritten. - Run
pnpm run db:migrate:empty "brief description here". This createspackages/civitai-db-schema/prisma/migrations/YYYYMMDDHHmmss_brief_description_here/migration.sqlfor you, in the one directory Prisma reads. To create it by hand instead, use that same path — not theprisma/migrationsdirectory at the repo root, which predates the monorepo layout and is no longer read. - Put your sql changes in the generated
migration.sql- These are usually simple sql commands like
ALTER TABLE ...
- These are usually simple sql commands like
- Run
make run-migrationsandmake gen-prisma - If you are adding/changing a column or table, please try to keep the
gen_seed.tsfile up to date with these changes.
Sponsors
Support this project by becoming a sponsor. Your logo will show up here with a link to your website.
License
Apache License 2.0 - Please have a look at the LICENSE for more details.