Justin Maier
c4c402ef40
* fix(search): page the models delta scan by keyset, not OFFSET prepareBatches walked the set of models updated since the last run with an unordered OFFSET/LIMIT loop. The set is re-evaluated on every page and its membership moves while the scan runs: an edit that unpublishes a model, or flips it to Unsearchable, removes a row from under the cursor and shifts every later page down by one, so a model eligible for the whole scan is silently never indexed. At ~1,659 published edits per day a multi-page scan meets that routinely, and this loop is also what an index repair leans on. Ordering the OFFSET query would not have fixed it -- order was never the problem, membership was. Page by a forward-only id cursor instead, which is unreachable by a membership change because ids are immutable. prepareBatches is hoisted to an exported prepareModelsBatches, mirroring prepareUsersBatches, so the paging can be driven by a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(search): make the keyset paging fake refuse what it cannot read Review found the fake answered queries it had not understood, so three mutations of the production query passed: - ORDER BY id DESC passed all four cases. The fake sorted ascending in both arms whatever the SQL asked, and /ORDER BY id/ matches DESC. Against a real database that mutation walks the cursor backwards from the top of the table and the scan never terminates. - A literal LIMIT 2000 fell through to a members.size default, handing back the whole set on page one -- at which point the headline case passes under an OFFSET implementation too. - id >= instead of id > fell through to an uncursored read and reddened with "the scan is not advancing", which is not what that defect does. The fake now honours ORDER BY direction and throws on a query whose LIMIT or cursor it cannot find. Case 1 also asserts the mid-scan edit actually landed -- keyset is meant to be unmoved by it, so nothing else in that case could tell a dead mutation from a working one -- and pins batchSize against the production constant so page-size drift names itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(search): pin the page query's eligibility predicates The paging fake models membership as an opaque set of ids, so it cannot see which rows the WHERE clause selects. Two review lanes independently demonstrated the consequence: deleting any one of the three predicates left the whole suite green. Dropping the updatedAt bound turns the delta scan into a full scan of every published model every 15 minutes; dropping the availability bound puts Unsearchable models into the public index. Both are the shape of an ordinary WHERE-clause tidy-up, and this is the only test that reads this query. Pinned textually rather than by teaching the fake to carry per-row timestamps: one assertion covers all three predicates where a behavioural fixture would only cover updatedAt, and a smaller fix round is the safer one. The watermark comment now names the symbols it depends on instead of restating base.search-index's semantics, which would rot silently if that file changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(search): pin the page query by whole clause, values, and the empty page Round three of review found three mutations of the query that left the suite green. Substring assertions were the common cause. - Widening. Appending `OR availability = 'Unsearchable'` leaves every asserted fragment present while AND binds tighter than OR, so every Unsearchable model is returned on every page. The sibling test next door already carried a written record of an adversarial round that beat this same assertion shape, so its `norm`/`renderTag`/`whereClausesOf` helpers move to `sql-shape.test-utils` and both files now pin a whole normalised clause with toBe rather than substrings. - Value corruption. `renderTag` renders a bind param as `?`, so the clause is blind to values and `Availability.Unsearchable` -> `Private` was green. The page query's bind values are pinned separately. - The empty-page break was unreachable: every fixture ended on a short page, so deleting `if (!ids.length) break` left the suite green while production reads `ids[ids.length - 1].id` off an empty array and kills the index job on any run where the eligible set is an exact multiple of the page size, or empty. A fixture of exactly READ_BATCH_SIZE members reaches it; the deletion now fails with that same TypeError. No production change in this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(search): pin that the processor runs the function under test Round four of review demonstrated the cheapest possible revert of this PR: leave the exported prepareModelsBatches alone and re-inline the old OFFSET body at the wiring site. Production goes back to row-losing paging and all 17 tests stay green, because every case imports the exported function directly and nothing in the repo read modelsSearchIndex.prepareBatches. The test file's own header claimed such a revert would redden it. That was false. createSearchIndexUpdateProcessor now returns prepareBatches, alongside the updateSyncChunkSize it already exposed for the same reason, and the test asserts the processor runs the function it drives. Measured cost, recorded in the test: a behaviour-preserving wrapper at the wiring site also fails this. That is inherent to an identity assertion, and the fix is to keep the wiring a direct reference. Also drops `toBe` from the sql-shape docstring, which named an assertion neither of its two callers uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(search): pin the bounds query and the id range it returns prepareModelsBatches returns four things; this file pinned two. Nothing read startId/endId, and the fake short-circuited the MIN/MAX statement on text.includes('MIN(id)') and handed back a hardcoded row, so the whole bounds query was invisible. Review demonstrated two mutants that left every case green: - swapping the aliases to MAX(id) as "startId", MIN(id) as "endId" makes endId - startId negative in base.search-index's range fan-out, so newly created models silently stop being indexed on every run; - deleting the bounds query guts the full rebuild, which then indexes zero models while the case named "issues no page query at all on a full rebuild" stays green, because it only asserts no page query fired. The fake now answers the aggregate the statement asked for, rather than returning fixed numbers under fixed names, so an alias swap produces a different id. The bounds statement's WHERE clause and binds are pinned the way the page query's already were. The "createdAt" bound there against "updatedAt" on the page query is pre-existing and deliberate, and is now pinned so a one-word change between the two cannot pass unnoticed. The file header claimed the fake refuses a query it cannot read. The bounds query was the one place it defaulted instead, which is why this survived six rounds; the header now says what the fake actually does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(search): pin the rebuild path's bounds, and both statements' table The rebuild branch was the one place startId/endId are the entire output, and the only case on it asserted that updateIds was empty and no page query fired. Both of those are also true of a rebuild that does nothing: returning { startId: 0, endId: 0, updateIds: [] } early for a missing watermark passed every case, and makes base.search-index's range fan-out zero tasks, so a whole index rebuild creates no batches and indexes nothing. Neither statement's table was pinned either. whereClausesOf captures from WHERE onward, so FROM "Model" -> FROM "ModelVersion" in either query left the file green while the scan paged a different entity. Also corrects the header's frequency claim. It said a multi-page scan meets concurrent edits as a matter of routine, citing ~1,659 edits a day; at a 15-minute cadence that is ~17 rows against a 2,000-row page, which argues the single-page case. The PR description was corrected for this and the source was not, which left the retracted claim in the file the next reader actually opens. The corrected text also states the mechanism that does not need an edit at all: no ORDER BY over a parallel seq scan lets synchronize_seqscans cut successive pages out of different orderings. That paragraph first shipped broken, because a cron string in a block comment closes it. The comment now says so rather than leaving the next person to rediscover it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Table of Contents
About the Project
Our goal with this project is to create a platform where people can share their stable diffusion models (textual inversions, hypernetworks, aesthetic gradients, VAEs, and any other crazy stuff people do to customize their AI generations), collaborate with others to improve them, and learn from each other's work. The platform allows users to create an account, upload their models, and browse models that have been shared by others. Users can also leave comments and feedback on each other's models to facilitate collaboration and knowledge sharing.
Tech Stack
We've built this project using a combination of modern web technologies, including Next.js for the frontend, TRPC for the API, and Prisma + Postgres for the database. By leveraging these tools, we've been able to create a scalable and maintainable platform that is both user-friendly and powerful.
- DB: Prisma + Postgres
- API: tRPC
- Front-end + Back-end: NextJS
- UI Kit: Mantine
- Storage: Cloudflare
Getting Started
To get a local copy up and running, follow these steps.
Prerequisites
- Docker, with Compose v2 (
docker compose, not the retired hyphenateddocker-compose). The database, Redis, MinIO, Meilisearch, ClickHouse and the mail catcher all run as containers. - Node.js
24.19.0. Not "20 or later" —package.jsondeclaresengines.node: ">=24.0.0 <25". The exact version lives in.nvmrc; CI installs that file's version and the production image is built on the same one, sonvm use(or any tool that reads.nvmrc) is the right way to get it. Note that nothing stops you:pnpm installonly printsWARN Unsupported engineand carries on, so the wrong major surfaces later as odd test failures rather than as a refusal at install time. - pnpm. This repo is pnpm-only, and this one is enforced —
npm installexits 1 via thepreinstallonly-allow pnpmhook.corepack enablewill pick up thepackageManagerfield for you. - Make (optional).
Installation
Standard setup
git clone https://github.com/civitai/civitai.git
cd civitai
nvm use # reads .nvmrc -> 24.19.0
corepack enable
git submodule update --init event-engine-common
cp .env-example .env.development
docker compose -f docker-compose.base.yml up -d
pnpm install
pnpm dev
Optional: Nix flake
Optional, and not the supported default. The standard setup above is what the project expects and what CI builds; nothing in the repo requires Nix, and you can ignore this section entirely. It exists because NixOS cannot use Prisma's published engines (there is no
linux-nixosbuild), so a flake is the practical way to work on this repo there. If you are not on NixOS and not already a flakes user, skip it.
The flake owns the toolchain, so you do not install Node or pnpm yourself:
git clone https://github.com/civitai/civitai.git
cd civitai
nix run .#dev
That single command checks Docker is usable, checks out the
event-engine-common submodule, creates .env.development from .env-example
if you do not already have one, starts the container stack, waits for Postgres,
runs pnpm install, and then starts the dev server on
http://localhost:3000. Every step is idempotent — it is
safe to re-run in a checkout that already works, and it will not overwrite your
.env.development or touch your data.
Useful variants:
nix run .#dev -- --no-start # bootstrap only, leave the services running
nix run .#dev -- --full # also start the signals/buzz containers (see below)
nix run .#doctor # check the flake's pins against the repo
nix flake check # the same checks, plus their own self-test
For an interactive shell with the same toolchain, use nix develop, or copy
.envrc.example to .envrc and run direnv allow to get it
automatically on cd.
With devcontainers
⚠️ Known out of step:
.devcontainer/public/docker-compose.ymlpinsmcr.microsoft.com/devcontainers/typescript-node:1-22, i.e. Node 22, which is outside this repo'sengines.noderange.pnpm installwill warn rather than stop, so the container comes up and then misbehaves in ways that look like your branch. There is no1-24tag (the template major moved on);3-24is the closest equivalent. Not changed here because it could not be exercised.
⚠️ Important Warning for Windows Users: Either clone this repo onto a WSL volume, or use the "clone repository in named container volume" command. Otherwise, you will see performance issues.
- Open the directory up in your IDE of choice
- VS Code should prompt you to "Open in container"
- If not, you may need to manually run
Dev Containers: Open Folder in Container
- If not, you may need to manually run
- For other IDEs, you may need to open the
.devcontainer/devcontainer.jsonfile, and click "Create devcontainer and mount sources" - Note: this may take some time to run initially
- VS Code should prompt you to "Open in container"
- Run
make run
The signals and buzz services
docker-compose.base.yml holds everything a contributor needs (and is also what
nix run .#dev starts). The extra services in docker-compose.yml
(signals, buzz) come from private ghcr.io images, so they only work for
internal members:
- create a GitHub personal access token with
read:packages - set it as
CR_PAT echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin- then
docker compose up -d(or, with the flake,nix run .#dev -- --full)
After the first start
- Edit
.env.development. Most defaults work out of the box; these do not:- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
minioadmin) — note it is port 9001, port 9000 is the S3 API itself — go to "Access Keys", click "Create Access Key", and copy the key and secret intoS3_UPLOAD_KEY/S3_UPLOAD_SECRETandS3_IMAGE_UPLOAD_KEY/S3_IMAGE_UPLOAD_SECRET. WEBHOOK_TOKEN— any random string; it authenticates requests to the webhook endpoint.EMAIL_USER,EMAIL_PASS, andEMAIL_FROM(a valid email format) — any values, but they must be set for user registration to work.
- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
- On an empty database, populate it. These are slow and destructive, which is
why no bootstrap runs them for you:
make run-migrations make reseed - Visit http://localhost:3000.
Please report any issues with these commands to us on discord.
* Note that account creation will run emails through maildev, which can be accessed at http://localhost:1080.
Altering your user
- First, create an account for yourself as you normally would through the UI.
- You may wish to set yourself up as a moderator. To do so:
- Use a database editor (like DataGrip) or connect directly to the
DB (
PGPASSWORD=postgres psql -h localhost -p 15432 -U postgres civitai) - Find your user (by email or username), and change
isModeratortotrue
- Use a database editor (like DataGrip) or connect directly to the
DB (
Known limitations
Services that require external input will currently not work locally. These include:
- Orchestration (Generation, Training)
- Signals (Chat, Notifications, other real-time updates)
- Buzz
Contributing
Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the repository to your own GitHub account.
- Create a new branch for your changes.
- Make your changes to the code.
- Commit your changes and push the branch to your forked repository.
- Open a pull request on our repository.
If you would like to be more involved, consider joining the Community Development Team! For more information on the team as well as how to join, see Calling All Developers: Join Civitai's Community Development Team.
Data Migrations
Over the course of development, you may need to change the structure of the database. To do this:
- Make your changes to the
packages/civitai-db-schema/prisma/schema.full.prismafile. Notschema.prisma— that one is gitignored and regenerated fromschema.full.prismabyscripts/generate-slim-schema.json everypnpm run db:generate, so edits to it are silently overwritten. - Run
pnpm run db:migrate:empty "brief description here". This createspackages/civitai-db-schema/prisma/migrations/YYYYMMDDHHmmss_brief_description_here/migration.sqlfor you, in the one directory Prisma reads. To create it by hand instead, use that same path — not theprisma/migrationsdirectory at the repo root, which predates the monorepo layout and is no longer read. - Put your sql changes in the generated
migration.sql- These are usually simple sql commands like
ALTER TABLE ...
- These are usually simple sql commands like
- Run
make run-migrationsandmake gen-prisma - If you are adding/changing a column or table, please try to keep the
gen_seed.tsfile up to date with these changes.
Sponsors
Support this project by becoming a sponsor. Your logo will show up here with a link to your website.
License
Apache License 2.0 - Please have a look at the LICENSE for more details.