Files
Zachary Lowden 481582d969 flake: own the dev toolchain, guard the pins, one command to a running app (#4107)
* feat(flake): make the Nix flake own the dev toolchain, and add one command to start

The flake shipped nodejs_22 while package.json declares engines.node
">=24.0.0 <25", .nvmrc pins 24.19.0 and the Dockerfile builds production on
node:24.19.0-alpine3.24. A NixOS developer was running a major the repo does
not support, and nothing said so.

Toolchain:
- node and pnpm are now DERIVED from .nvmrc and package.json's packageManager
  rather than named twice. .nvmrc is treated as the authority because it is what
  every workflow's actions/setup-node reads and what the Dockerfile tracks.
- flake.lock moved 2026-04-23 -> 2026-08-18 (117 days). At that rev nodejs_24 is
  exactly 24.19.0, which is what made agreeing with .nvmrc possible at all.
- pnpm now comes from `pnpm_10`, not the unversioned `pkgs.pnpm`. At the new
  rev the unversioned attribute resolves to 11.21.0 -- a major bump that
  rewrites pnpm-lock.yaml -- so this bump would otherwise have shipped pnpm 11
  to every dev shell silently.
- postgresql_16 -> postgresql_17, matching the primary `db` container. The
  postgres/redis/clickhouse entries are CLIENTS for the compose-hosted servers;
  that is now stated in the file instead of left to be guessed.
- npm_config_manage_package_manager_versions=false. Measured: without it, pnpm
  downloads and re-execs the exact version from the packageManager field, so the
  flake's pnpm pin was being defeated at runtime (`pnpm --version` returns
  10.28.1 with the var unset, 10.34.5 with it set).

Guards (`nix flake check`, 4 checks):
- toolchain-pins: the flake's node must satisfy engines.node and equal .nvmrc,
  and its pnpm must share a major with packageManager. Deliberately does NOT
  re-check the .nvmrc/Dockerfile/engines triangle -- node-version-consistency.test.ts
  already owns that, and a predicate open-coded twice starts disagreeing.
- prisma-pin: re-derives the resolved @prisma/client AND its engine commit from
  pnpm-lock.yaml and compares them to the values flake.nix hardcodes. These were
  correct but unguarded: package.json declares `^6.3.0`, a caret range, so a
  routine lockfile refresh moves the client while the flake's engines stay put,
  and the failure surfaces at runtime in every dev shell.
- pin-guards-selftest: breaks each pin on purpose and requires the guard that
  owns it to fire while the others stay silent.
- dev-scripts: builds the shell entrypoints, which is what runs their shellcheck.
  (`nix flake check` builds checks.* but only EVALUATES packages.*, measured.)

Entrypoints:
- `nix run .#dev` - docker preflight, submodule, .env.development, compose up,
  wait for postgres, pnpm install, then `next dev`. Every step idempotent and
  non-destructive; migrations and seeding stay opt-in.
- `nix run .#dev-server` - runs the dev-server CLI on the flake's node. The
  daemon re-execs itself with process.execPath, so whichever node starts the CLI
  is the node it runs on until it is restarted.
- `nix run .#doctor` - the same pin checks against the working tree.

Compose project is pinned to `civitai` so every worktree shares the one local
stack instead of each spawning a duplicate that fails on the port binds.

* fix(flake): give `nix run` the same env as the dev shell, not just the shell

Found by running the bootstrap on a genuinely clean worktree rather than
reasoning about it. `mkShell`'s `env` applies to `nix develop` only, so both
values it carried were absent from `nix run .#dev`:

- `pnpm install`'s postinstall runs `prisma generate`. Without
  PRISMA_QUERY_ENGINE_LIBRARY et al, prisma tried to fetch an engine for
  platform `linux-nixos` and the bootstrap died on
  `404 ... /linux-nixos/libquery_engine.so.node.sha256`.
- pnpm re-execed itself as 10.28.1 from the packageManager field even though
  PATH pointed at the flake's 10.34.5, so the app reported a pnpm the flake had
  not pinned.

The env is now one attrset (`devEnv`) rendered two ways: `env` for the shell and
an `export` preamble for the apps, so they cannot drift. `nix run .#dev-server`
gets it too -- the daemon runs `pnpm install` / `db:generate` on its own when it
sees the lockfile move, which would have hit the identical 404.

* docs: describe the toolchain the repo actually has, not the one it used to

Every claim below was checked against the code before rewriting, and the
measurements are quoted where they are load-bearing.

README.md
- "Node.js (version 20 or later)" -> 24.19.0, with .nvmrc named as the authority.
- `make init` was DEAD, not merely awkward: it ran `npm i`, and package.json's
  `preinstall` runs `only-allow pnpm`, which exits 1 under an npm user agent
  (measured, with the pnpm-user-agent control exiting 0). Both bootstrap paths
  the README offered went through it.
- MinIO console is on :9001, not :9000 (:9000 is the S3 API). The instructions
  sent people to the wrong port to mint the keys the next step needs.
- `git submodule update --recursive` -> `--init`; without `--init` it is a no-op
  on a fresh clone, which is precisely when it is being run.
- Data Migrations step 1 pointed at `schema.prisma`, which is gitignored and
  regenerated from `schema.full.prisma` on every `db:generate`, so edits to it
  were silently discarded.
- Adds the Nix path (`nix run .#dev`) and a real non-Nix sequence.
- engines.node is ADVISORY, stated plainly: pnpm 10.34.5 under node 26.7.0
  against ">=24.0.0 <25" prints `WARN Unsupported engine` and exits 0. An
  earlier draft of this very README claimed it refuses. It does not, and that is
  the reason the drift survived so long.

Makefile
- `npm i` -> `pnpm install` (see above). `npm-install` kept as an alias.
- `gen-prisma` ran a bare `prisma generate`, which reads the gitignored slim
  schema that does not exist yet on a fresh clone; now `pnpm run db:generate`,
  which generates it first.
- `dev` ran bare `cross-env`/`next`, requiring the caller to put
  node_modules/.bin on PATH by hand; now via `pnpm exec`.
- `docker-compose` (EOL v1) -> `docker compose`.
- COMPOSE_PROJECT_NAME pinned to `civitai`. Reproduced first: `make start` in a
  worktree died with `Bind for :::15434 failed: port is already allocated`
  because compose named the project after the directory.

.envrc.example (new, tracked) + .gitignore
- `.env*` matched `.envrc` too, so nothing tracked in the repo mentioned the
  flake at all -- the only reference was a line in CLAUDE.md filed under
  worktree hygiene. Placeholders only; the real .envrc stays ignored.

.claude/skills/dev-server/SKILL.md
- The skill said nothing about node. The daemon is spawned with
  `process.execPath` (cli.mjs:66, console.mjs:87) and hands its env to every
  `next dev` it supervises, so the first shell to run a CLI verb decides the
  node for everything, indefinitely. Measured on this box: daemon on 26.7.0,
  with no pnpm on PATH at all. Documents `nix run .#dev-server` and how to check.
- `npm run dev:daemon` -> `pnpm run dev:daemon`, in a repo that bans npm.

src/__tests__/node-version-consistency.test.ts
- Comment-only. It said flake.nix "is on a different major" and could not be
  aligned because the pinned nixpkgs had no Node 24 this new. Both halves are
  now false, and a comment a maintainer might act on is worth correcting.

Also: docs/pnpm-migration.md's "Node.js 18.x or later"; the generated-header
line in scripts/generate-slim-schema.js telling readers to run `npm run
db:generate`; CLAUDE.md's local-dev section (no node version, no services) and
its stale "flake's 22.22.2" figure.

NOT changed, because it could not be exercised here: the devcontainer pins
typescript-node:1-22 (Node 22, outside engines.node). Flagged in README with the
tag to use -- there is no `1-24`, the template major moved on, so `3-24`.

* docs(flake): the four postgres containers are not all one version

prisma-pit and db are postgres 17; notification-db and logical-db are 15. The
comment justifying postgresql_17 read as though they were uniform, which would
have made the next person's version decision from the wrong premise.

* docs: keep the non-Nix path the default, demote the flake to optional

The flake is used by one maintainer. Everyone else uses Docker + nvm, and that
has to stay the path a contributor lands on. The previous revision inverted
that: README's Installation section led with "With Nix (recommended...)" and
titled the standard path "Without Nix" — framing the majority workflow as the
fallback. CLAUDE.md opened "From nothing to a running app, one command:" with
`nix run .#dev`, and the dev-server skill led its fix with "Start it through the
flake and this cannot happen".

None of that made Nix *required* — verified: `.github/` is untouched by this
branch, no workflow references Nix (the apparent hits are substrings of
`eslint-unix.json` and `--format unix`), and `nix flake check` is not wired to
any CI gate. It was purely an ordering-and-emphasis problem, which is the kind
that costs a new contributor twenty minutes before they find the section that
applies to them.

Changes, all editorial:

- README: `#### Standard setup` now precedes `#### Optional: Nix flake`, and the
  Nix section opens with a blockquote saying it is not the supported default,
  that nothing requires it, and why it exists at all (NixOS has no published
  `linux-nixos` Prisma engine, so a flake is the practical way to work there).
  The signals/buzz instructions lead with `docker compose up -d` and mention
  `nix run .#dev -- --full` parenthetically.
- CLAUDE.md: the bootstrap block is now the nvm/docker sequence, labelled as the
  default path, with the flake shown after it as NixOS-only and explicitly
  flagged as something not to assume a contributor has. The dev-server step no
  longer instructs going through `nix run .#dev-server`; it states the
  requirement (a shell whose node matches `.nvmrc`) and notes the flake does that
  for you on NixOS.
- dev-server SKILL.md: the fix is now stated setup-agnostically — start the
  daemon from a shell whose node matches `.nvmrc` with pnpm on PATH, which
  `nvm use` gives you — with the flake wrapper presented as the optional NixOS
  convenience, and an explicit note that nothing in the document depends on Nix.

No behaviour, tooling or gate changes: the Makefile, flake, guards and their
tests are untouched by this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 19:48:54 -05:00

11 KiB

Contributors Forks Stargazers Issues Apache License 2.0 Discord


Table of Contents

About the Project

Our goal with this project is to create a platform where people can share their stable diffusion models (textual inversions, hypernetworks, aesthetic gradients, VAEs, and any other crazy stuff people do to customize their AI generations), collaborate with others to improve them, and learn from each other's work. The platform allows users to create an account, upload their models, and browse models that have been shared by others. Users can also leave comments and feedback on each other's models to facilitate collaboration and knowledge sharing.

Tech Stack

We've built this project using a combination of modern web technologies, including Next.js for the frontend, TRPC for the API, and Prisma + Postgres for the database. By leveraging these tools, we've been able to create a scalable and maintainable platform that is both user-friendly and powerful.

  • DB: Prisma + Postgres
  • API: tRPC
  • Front-end + Back-end: NextJS
  • UI Kit: Mantine
  • Storage: Cloudflare

Getting Started

To get a local copy up and running, follow these steps.

Prerequisites

  • Docker, with Compose v2 (docker compose, not the retired hyphenated docker-compose). The database, Redis, MinIO, Meilisearch, ClickHouse and the mail catcher all run as containers.
  • Node.js 24.19.0. Not "20 or later" — package.json declares engines.node: ">=24.0.0 <25". The exact version lives in .nvmrc; CI installs that file's version and the production image is built on the same one, so nvm use (or any tool that reads .nvmrc) is the right way to get it. Note that nothing stops you: pnpm install only prints WARN Unsupported engine and carries on, so the wrong major surfaces later as odd test failures rather than as a refusal at install time.
  • pnpm. This repo is pnpm-only, and this one is enforced — npm install exits 1 via the preinstall only-allow pnpm hook. corepack enable will pick up the packageManager field for you.
  • Make (optional).

Installation

Standard setup

git clone https://github.com/civitai/civitai.git
cd civitai
nvm use                                              # reads .nvmrc -> 24.19.0
corepack enable
git submodule update --init event-engine-common
cp .env-example .env.development
docker compose -f docker-compose.base.yml up -d
pnpm install
pnpm dev

Optional: Nix flake

Optional, and not the supported default. The standard setup above is what the project expects and what CI builds; nothing in the repo requires Nix, and you can ignore this section entirely. It exists because NixOS cannot use Prisma's published engines (there is no linux-nixos build), so a flake is the practical way to work on this repo there. If you are not on NixOS and not already a flakes user, skip it.

The flake owns the toolchain, so you do not install Node or pnpm yourself:

git clone https://github.com/civitai/civitai.git
cd civitai
nix run .#dev

That single command checks Docker is usable, checks out the event-engine-common submodule, creates .env.development from .env-example if you do not already have one, starts the container stack, waits for Postgres, runs pnpm install, and then starts the dev server on http://localhost:3000. Every step is idempotent — it is safe to re-run in a checkout that already works, and it will not overwrite your .env.development or touch your data.

Useful variants:

nix run .#dev -- --no-start   # bootstrap only, leave the services running
nix run .#dev -- --full       # also start the signals/buzz containers (see below)
nix run .#doctor              # check the flake's pins against the repo
nix flake check               # the same checks, plus their own self-test

For an interactive shell with the same toolchain, use nix develop, or copy .envrc.example to .envrc and run direnv allow to get it automatically on cd.

With devcontainers

⚠️ Known out of step: .devcontainer/public/docker-compose.yml pins mcr.microsoft.com/devcontainers/typescript-node:1-22, i.e. Node 22, which is outside this repo's engines.node range. pnpm install will warn rather than stop, so the container comes up and then misbehaves in ways that look like your branch. There is no 1-24 tag (the template major moved on); 3-24 is the closest equivalent. Not changed here because it could not be exercised.

⚠️ Important Warning for Windows Users: Either clone this repo onto a WSL volume, or use the "clone repository in named container volume" command. Otherwise, you will see performance issues.

  • Open the directory up in your IDE of choice
    • VS Code should prompt you to "Open in container"
      • If not, you may need to manually run Dev Containers: Open Folder in Container
    • For other IDEs, you may need to open the .devcontainer/devcontainer.json file, and click "Create devcontainer and mount sources"
    • Note: this may take some time to run initially
  • Run make run

The signals and buzz services

docker-compose.base.yml holds everything a contributor needs (and is also what nix run .#dev starts). The extra services in docker-compose.yml (signals, buzz) come from private ghcr.io images, so they only work for internal members:

  • create a GitHub personal access token with read:packages
  • set it as CR_PAT
  • echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin
  • then docker compose up -d (or, with the flake, nix run .#dev -- --full)

After the first start

  1. Edit .env.development. Most defaults work out of the box; these do not:
    • S3 upload credentials. Open the MinIO console at http://localhost:9001 (username and password both minioadmin) — note it is port 9001, port 9000 is the S3 API itself — go to "Access Keys", click "Create Access Key", and copy the key and secret into S3_UPLOAD_KEY / S3_UPLOAD_SECRET and S3_IMAGE_UPLOAD_KEY / S3_IMAGE_UPLOAD_SECRET.
    • WEBHOOK_TOKEN — any random string; it authenticates requests to the webhook endpoint.
    • EMAIL_USER, EMAIL_PASS, and EMAIL_FROM (a valid email format) — any values, but they must be set for user registration to work.
  2. On an empty database, populate it. These are slow and destructive, which is why no bootstrap runs them for you:
    make run-migrations
    make reseed
    
  3. Visit http://localhost:3000.

Please report any issues with these commands to us on discord.

* Note that account creation will run emails through maildev, which can be accessed at http://localhost:1080.

Altering your user

  • First, create an account for yourself as you normally would through the UI.
  • You may wish to set yourself up as a moderator. To do so:
    • Use a database editor (like DataGrip) or connect directly to the DB (PGPASSWORD=postgres psql -h localhost -p 15432 -U postgres civitai)
    • Find your user (by email or username), and change isModerator to true

Known limitations

Services that require external input will currently not work locally. These include:

  • Orchestration (Generation, Training)
  • Signals (Chat, Notifications, other real-time updates)
  • Buzz

Contributing

Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the repository to your own GitHub account.
  2. Create a new branch for your changes.
  3. Make your changes to the code.
  4. Commit your changes and push the branch to your forked repository.
  5. Open a pull request on our repository.

If you would like to be more involved, consider joining the Community Development Team! For more information on the team as well as how to join, see Calling All Developers: Join Civitai's Community Development Team.

Data Migrations

Over the course of development, you may need to change the structure of the database. To do this:

  1. Make your changes to the packages/civitai-db-schema/prisma/schema.full.prisma file. Not schema.prisma — that one is gitignored and regenerated from schema.full.prisma by scripts/generate-slim-schema.js on every pnpm run db:generate, so edits to it are silently overwritten.
  2. Run pnpm run db:migrate:empty "brief description here". This creates packages/civitai-db-schema/prisma/migrations/YYYYMMDDHHmmss_brief_description_here/migration.sql for you, in the one directory Prisma reads. To create it by hand instead, use that same path — not the prisma/migrations directory at the repo root, which predates the monorepo layout and is no longer read.
  3. Put your sql changes in the generated migration.sql
    • These are usually simple sql commands like ALTER TABLE ...
  4. Run make run-migrations and make gen-prisma
  5. If you are adding/changing a column or table, please try to keep the gen_seed.ts file up to date with these changes.

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website.

License

Apache License 2.0 - Please have a look at the LICENSE for more details.