Files
civitai__civitai/flake.nix
Zachary Lowden 481582d969 flake: own the dev toolchain, guard the pins, one command to a running app (#4107)
* feat(flake): make the Nix flake own the dev toolchain, and add one command to start

The flake shipped nodejs_22 while package.json declares engines.node
">=24.0.0 <25", .nvmrc pins 24.19.0 and the Dockerfile builds production on
node:24.19.0-alpine3.24. A NixOS developer was running a major the repo does
not support, and nothing said so.

Toolchain:
- node and pnpm are now DERIVED from .nvmrc and package.json's packageManager
  rather than named twice. .nvmrc is treated as the authority because it is what
  every workflow's actions/setup-node reads and what the Dockerfile tracks.
- flake.lock moved 2026-04-23 -> 2026-08-18 (117 days). At that rev nodejs_24 is
  exactly 24.19.0, which is what made agreeing with .nvmrc possible at all.
- pnpm now comes from `pnpm_10`, not the unversioned `pkgs.pnpm`. At the new
  rev the unversioned attribute resolves to 11.21.0 -- a major bump that
  rewrites pnpm-lock.yaml -- so this bump would otherwise have shipped pnpm 11
  to every dev shell silently.
- postgresql_16 -> postgresql_17, matching the primary `db` container. The
  postgres/redis/clickhouse entries are CLIENTS for the compose-hosted servers;
  that is now stated in the file instead of left to be guessed.
- npm_config_manage_package_manager_versions=false. Measured: without it, pnpm
  downloads and re-execs the exact version from the packageManager field, so the
  flake's pnpm pin was being defeated at runtime (`pnpm --version` returns
  10.28.1 with the var unset, 10.34.5 with it set).

Guards (`nix flake check`, 4 checks):
- toolchain-pins: the flake's node must satisfy engines.node and equal .nvmrc,
  and its pnpm must share a major with packageManager. Deliberately does NOT
  re-check the .nvmrc/Dockerfile/engines triangle -- node-version-consistency.test.ts
  already owns that, and a predicate open-coded twice starts disagreeing.
- prisma-pin: re-derives the resolved @prisma/client AND its engine commit from
  pnpm-lock.yaml and compares them to the values flake.nix hardcodes. These were
  correct but unguarded: package.json declares `^6.3.0`, a caret range, so a
  routine lockfile refresh moves the client while the flake's engines stay put,
  and the failure surfaces at runtime in every dev shell.
- pin-guards-selftest: breaks each pin on purpose and requires the guard that
  owns it to fire while the others stay silent.
- dev-scripts: builds the shell entrypoints, which is what runs their shellcheck.
  (`nix flake check` builds checks.* but only EVALUATES packages.*, measured.)

Entrypoints:
- `nix run .#dev` - docker preflight, submodule, .env.development, compose up,
  wait for postgres, pnpm install, then `next dev`. Every step idempotent and
  non-destructive; migrations and seeding stay opt-in.
- `nix run .#dev-server` - runs the dev-server CLI on the flake's node. The
  daemon re-execs itself with process.execPath, so whichever node starts the CLI
  is the node it runs on until it is restarted.
- `nix run .#doctor` - the same pin checks against the working tree.

Compose project is pinned to `civitai` so every worktree shares the one local
stack instead of each spawning a duplicate that fails on the port binds.

* fix(flake): give `nix run` the same env as the dev shell, not just the shell

Found by running the bootstrap on a genuinely clean worktree rather than
reasoning about it. `mkShell`'s `env` applies to `nix develop` only, so both
values it carried were absent from `nix run .#dev`:

- `pnpm install`'s postinstall runs `prisma generate`. Without
  PRISMA_QUERY_ENGINE_LIBRARY et al, prisma tried to fetch an engine for
  platform `linux-nixos` and the bootstrap died on
  `404 ... /linux-nixos/libquery_engine.so.node.sha256`.
- pnpm re-execed itself as 10.28.1 from the packageManager field even though
  PATH pointed at the flake's 10.34.5, so the app reported a pnpm the flake had
  not pinned.

The env is now one attrset (`devEnv`) rendered two ways: `env` for the shell and
an `export` preamble for the apps, so they cannot drift. `nix run .#dev-server`
gets it too -- the daemon runs `pnpm install` / `db:generate` on its own when it
sees the lockfile move, which would have hit the identical 404.

* docs: describe the toolchain the repo actually has, not the one it used to

Every claim below was checked against the code before rewriting, and the
measurements are quoted where they are load-bearing.

README.md
- "Node.js (version 20 or later)" -> 24.19.0, with .nvmrc named as the authority.
- `make init` was DEAD, not merely awkward: it ran `npm i`, and package.json's
  `preinstall` runs `only-allow pnpm`, which exits 1 under an npm user agent
  (measured, with the pnpm-user-agent control exiting 0). Both bootstrap paths
  the README offered went through it.
- MinIO console is on :9001, not :9000 (:9000 is the S3 API). The instructions
  sent people to the wrong port to mint the keys the next step needs.
- `git submodule update --recursive` -> `--init`; without `--init` it is a no-op
  on a fresh clone, which is precisely when it is being run.
- Data Migrations step 1 pointed at `schema.prisma`, which is gitignored and
  regenerated from `schema.full.prisma` on every `db:generate`, so edits to it
  were silently discarded.
- Adds the Nix path (`nix run .#dev`) and a real non-Nix sequence.
- engines.node is ADVISORY, stated plainly: pnpm 10.34.5 under node 26.7.0
  against ">=24.0.0 <25" prints `WARN Unsupported engine` and exits 0. An
  earlier draft of this very README claimed it refuses. It does not, and that is
  the reason the drift survived so long.

Makefile
- `npm i` -> `pnpm install` (see above). `npm-install` kept as an alias.
- `gen-prisma` ran a bare `prisma generate`, which reads the gitignored slim
  schema that does not exist yet on a fresh clone; now `pnpm run db:generate`,
  which generates it first.
- `dev` ran bare `cross-env`/`next`, requiring the caller to put
  node_modules/.bin on PATH by hand; now via `pnpm exec`.
- `docker-compose` (EOL v1) -> `docker compose`.
- COMPOSE_PROJECT_NAME pinned to `civitai`. Reproduced first: `make start` in a
  worktree died with `Bind for :::15434 failed: port is already allocated`
  because compose named the project after the directory.

.envrc.example (new, tracked) + .gitignore
- `.env*` matched `.envrc` too, so nothing tracked in the repo mentioned the
  flake at all -- the only reference was a line in CLAUDE.md filed under
  worktree hygiene. Placeholders only; the real .envrc stays ignored.

.claude/skills/dev-server/SKILL.md
- The skill said nothing about node. The daemon is spawned with
  `process.execPath` (cli.mjs:66, console.mjs:87) and hands its env to every
  `next dev` it supervises, so the first shell to run a CLI verb decides the
  node for everything, indefinitely. Measured on this box: daemon on 26.7.0,
  with no pnpm on PATH at all. Documents `nix run .#dev-server` and how to check.
- `npm run dev:daemon` -> `pnpm run dev:daemon`, in a repo that bans npm.

src/__tests__/node-version-consistency.test.ts
- Comment-only. It said flake.nix "is on a different major" and could not be
  aligned because the pinned nixpkgs had no Node 24 this new. Both halves are
  now false, and a comment a maintainer might act on is worth correcting.

Also: docs/pnpm-migration.md's "Node.js 18.x or later"; the generated-header
line in scripts/generate-slim-schema.js telling readers to run `npm run
db:generate`; CLAUDE.md's local-dev section (no node version, no services) and
its stale "flake's 22.22.2" figure.

NOT changed, because it could not be exercised here: the devcontainer pins
typescript-node:1-22 (Node 22, outside engines.node). Flagged in README with the
tag to use -- there is no `1-24`, the template major moved on, so `3-24`.

* docs(flake): the four postgres containers are not all one version

prisma-pit and db are postgres 17; notification-db and logical-db are 15. The
comment justifying postgresql_17 read as though they were uniform, which would
have made the next person's version decision from the wrong premise.

* docs: keep the non-Nix path the default, demote the flake to optional

The flake is used by one maintainer. Everyone else uses Docker + nvm, and that
has to stay the path a contributor lands on. The previous revision inverted
that: README's Installation section led with "With Nix (recommended...)" and
titled the standard path "Without Nix" — framing the majority workflow as the
fallback. CLAUDE.md opened "From nothing to a running app, one command:" with
`nix run .#dev`, and the dev-server skill led its fix with "Start it through the
flake and this cannot happen".

None of that made Nix *required* — verified: `.github/` is untouched by this
branch, no workflow references Nix (the apparent hits are substrings of
`eslint-unix.json` and `--format unix`), and `nix flake check` is not wired to
any CI gate. It was purely an ordering-and-emphasis problem, which is the kind
that costs a new contributor twenty minutes before they find the section that
applies to them.

Changes, all editorial:

- README: `#### Standard setup` now precedes `#### Optional: Nix flake`, and the
  Nix section opens with a blockquote saying it is not the supported default,
  that nothing requires it, and why it exists at all (NixOS has no published
  `linux-nixos` Prisma engine, so a flake is the practical way to work there).
  The signals/buzz instructions lead with `docker compose up -d` and mention
  `nix run .#dev -- --full` parenthetically.
- CLAUDE.md: the bootstrap block is now the nvm/docker sequence, labelled as the
  default path, with the flake shown after it as NixOS-only and explicitly
  flagged as something not to assume a contributor has. The dev-server step no
  longer instructs going through `nix run .#dev-server`; it states the
  requirement (a shell whose node matches `.nvmrc`) and notes the flake does that
  for you on NixOS.
- dev-server SKILL.md: the fix is now stated setup-agnostically — start the
  daemon from a shell whose node matches `.nvmrc` with pnpm on PATH, which
  `nvm use` gives you — with the flake wrapper presented as the optional NixOS
  convenience, and an explicit note that nothing in the document depends on Nix.

No behaviour, tooling or gate changes: the Makefile, flake, guards and their
tests are untouched by this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 19:48:54 -05:00

322 lines
14 KiB
Nix

{
description = "Civitai local development environment (NixOS, x86_64-linux)";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
};
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = import nixpkgs { inherit system; };
inherit (pkgs) lib;
# =====================================================================
# Toolchain
#
# Nothing here hardcodes a version the repo already declares elsewhere.
# `.nvmrc` and `package.json` are read directly, so the flake cannot
# silently drift from what CI installs and what production runs. The
# residual drift a derivation cannot prevent -- nixpkgs moving the patch
# version under us -- is caught by `checks.toolchain-pins` below, so it
# surfaces as a red `nix flake check` rather than as a mystery at runtime.
#
# Authorities, for the record:
# node .nvmrc (every GitHub workflow: node-version-file: .nvmrc)
# (Dockerfile FROM node:<same>)
# pnpm package.json packageManager field
# prisma pnpm-lock.yaml resolved @prisma/client (see prisma-engines)
# =====================================================================
nodeVersion = lib.trim (builtins.readFile ./.nvmrc);
nodeMajor = lib.head (lib.splitString "." nodeVersion);
nodejs = pkgs."nodejs_${nodeMajor}";
packageJson = builtins.fromJSON (builtins.readFile ./package.json);
# "pnpm@10.28.1" -> "10". nixpkgs will not carry every patch release and
# does not need to: the lockfile format is tied to the major. Pinning the
# major attribute (rather than the unversioned `pkgs.pnpm`) is what stops
# a `nix flake update` silently handing everyone pnpm 11 -- which is
# exactly what the unversioned attribute did between this flake's old lock
# and its new one.
pnpmMajor = lib.head (lib.splitString "."
(lib.last (lib.splitString "@" packageJson.packageManager)));
# `nodejs-slim`, not `nodejs`: pnpm's own launcher only needs a runtime,
# and nixpkgs warns if you override the full package here. This keeps
# pnpm's shebang node on the same major as the shell's node.
pnpm = pkgs."pnpm_${pnpmMajor}".override {
nodejs-slim = pkgs."nodejs-slim_${nodeMajor}";
};
# =====================================================================
# Prisma engines
#
# `@prisma/client` embeds an engine commit and verifies it against the
# engine binary at runtime, so the engines must match the client EXACTLY.
# nixpkgs never packaged 6.13.0 (prisma-engines jumps 6.7 -> 6.18), so we
# fetch Prisma's official prebuilt engines for one commit and patchelf
# them onto NixOS rather than building from source or drifting the repo's
# pinned client.
#
# Both values below are duplicates of information that lives in
# pnpm-lock.yaml, which Nix cannot parse. `checks.prisma-pin` re-derives
# them from the lockfile and fails if they have diverged -- package.json
# declares `@prisma/client: ^6.3.0`, a caret range, so a routine lockfile
# refresh is all it takes.
#
# To bump: change prismaVersion + engineCommit (from the lockfile's
# `@prisma/engines-version@<version>-<n>.<commit>`), set the three sha256s
# to lib.fakeSha256, build once, and paste in what nix reports.
# =====================================================================
prismaVersion = "6.13.0";
engineCommit = "361e86d0ea4987e9f53a565309b3eed797a6bcbd";
enginePlatform = "debian-openssl-3.0.x"; # links libssl/libcrypto .so.3, satisfied by pkgs.openssl
fetchEngine = file: sha256: pkgs.fetchurl {
url = "https://binaries.prisma.sh/all_commits/${engineCommit}/${enginePlatform}/${file}.gz";
inherit sha256;
};
prisma-engines = pkgs.stdenvNoCC.mkDerivation {
pname = "prisma-engines";
version = prismaVersion;
dontUnpack = true;
nativeBuildInputs = [ pkgs.autoPatchelfHook pkgs.gzip ];
buildInputs = [ pkgs.openssl pkgs.stdenv.cc.cc.lib pkgs.zlib ];
dontStrip = true;
queryLib = fetchEngine "libquery_engine.so.node" "0gamcinpfb8gvli48z16a378ziyinsanniddgbmd93v1lisllcz2";
schemaEngine = fetchEngine "schema-engine" "0rjwada7j2gdqx5xwbxqdvhr2c8jk2mjzhyblfbryiazyv3i9ir9";
queryEngine = fetchEngine "query-engine" "0iiknxyygq62g1n64h7nfpbfkmq5pi6d8i8di5hyv09hsmzbaimd";
buildPhase = ''
mkdir -p $out/lib $out/bin
gzip -dc $queryLib > $out/lib/libquery_engine.node
gzip -dc $schemaEngine > $out/bin/schema-engine
gzip -dc $queryEngine > $out/bin/query-engine
chmod +x $out/bin/schema-engine $out/bin/query-engine
'';
};
# =====================================================================
# The environment the toolchain needs in order to behave
#
# ONE definition, consumed by BOTH the devShell and the `nix run` apps.
# Keeping it devShell-only is not a smaller version of this -- it is
# broken, and measurably so. `nix run .#dev` on a clean checkout got as far
# as `pnpm install`, whose postinstall runs `prisma generate`, which then
# tried to download an engine for platform `linux-nixos` and died on a 404.
# The same omission let pnpm re-exec itself as 10.28.1 inside an app whose
# PATH pointed at the flake's 10.34.5.
# =====================================================================
devEnv = {
# Prisma publishes no NixOS engine build, so point it at the patchelf'd
# ones above instead of letting it try to fetch a `linux-nixos` binary
# that has never existed.
PRISMA_QUERY_ENGINE_LIBRARY = "${prisma-engines}/lib/libquery_engine.node";
PRISMA_QUERY_ENGINE_BINARY = "${prisma-engines}/bin/query-engine";
PRISMA_SCHEMA_ENGINE_BINARY = "${prisma-engines}/bin/schema-engine";
# pnpm 10 otherwise downloads and re-execs the exact version named in
# package.json's packageManager field, quietly replacing the pnpm this
# flake pinned. Measured: `pnpm --version` reports 10.28.1 without this
# and 10.34.5 with it, from the same binary on PATH.
npm_config_manage_package_manager_versions = "false";
};
# The same attrset rendered as shell `export` lines, so an app and the
# shell cannot drift apart.
envPreamble = lib.concatStringsSep "\n"
(lib.mapAttrsToList (k: v: "export ${k}=${lib.escapeShellArg v}") devEnv);
# =====================================================================
# Service stack
#
# The SERVERS are docker-compose's job (docker-compose.base.yml), not
# nix's -- four postgres, two redis, minio, meilisearch, clickhouse and
# maildev. What the shell provides is the matching CLIENTS, because you
# talk to those containers by hand constantly:
#
# psql -> the `db` container on :15432
# redis-cli -> the `redis` container on :6379
# clickhouse client -> the `clickhouse` container on :18123
#
# postgresql_17 rather than _16 because the postgres containers are NOT
# all one version: `prisma-pit` and `db` are postgres 17, while
# `notification-db` and `logical-db` are postgres 15. A newer client talks
# to an older server fine; the other direction is the one libpq does not
# promise, so the client tracks the NEWEST server, not the oldest.
# =====================================================================
serviceClients = [
pkgs.postgresql_17
pkgs.redis
pkgs.clickhouse
];
# =====================================================================
# Checks
#
# Deterministic, sandboxed, no network. Two python helpers live in
# scripts/nix/ rather than inline here so they can be read, and run,
# without going through nix (`nix run .#doctor` does exactly that).
# =====================================================================
checkPython = pkgs.python3.withPackages (ps: with ps; [ pyyaml node-semver ]);
nodeCheckArgs = lib.escapeShellArgs [
"--nvmrc" "${./.nvmrc}"
"--package-json" "${./package.json}"
"--flake-node" nodejs.version
"--flake-pnpm" pnpm.version
];
prismaCheckArgs = lib.escapeShellArgs [
"--package-json" "${./package.json}"
"--lockfile" "${./pnpm-lock.yaml}"
"--flake-prisma-version" prismaVersion
"--flake-engine-commit" engineCommit
];
# =====================================================================
# Entrypoints
# =====================================================================
dev = pkgs.writeShellApplication {
name = "dev";
# Docker is deliberately absent: the CLI has to match the daemon the
# developer is already running, so it comes from the host PATH.
runtimeInputs = [ nodejs pnpm pkgs.git pkgs.postgresql_17 pkgs.coreutils ];
text = envPreamble + "\n" + builtins.readFile ./scripts/nix/dev-up.sh;
meta.description = "Bootstrap and run the civitai local dev environment";
};
dev-server = pkgs.writeShellApplication {
name = "dev-server";
runtimeInputs = [ nodejs pnpm pkgs.git ];
# The daemon runs `pnpm install` and `pnpm run db:generate` itself when
# it sees the lockfile or the schema move, so it needs the same env the
# shell has or those background installs hit the identical prisma 404.
text = envPreamble + ''
# The dev-server daemon re-execs itself with process.execPath, so
# whatever node launches the CLI is the node the daemon runs on
# forever. Launching it through this wrapper is what pins it to the
# flake's node instead of whatever happened to be on PATH.
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$ROOT" ] || [ ! -d "$ROOT" ]; then
echo "dev-server: not inside a git checkout." >&2
exit 1
fi
cd "$ROOT"
exec node .claude/skills/dev-server/cli.mjs "$@"
'';
meta.description = "Run the dev-server CLI on the flake's node";
};
doctor = pkgs.writeShellApplication {
name = "doctor";
runtimeInputs = [ checkPython pkgs.git ];
text = ''
# Same assertions `nix flake check` makes, but against your working
# tree rather than the committed source, so you can see the effect of
# an edit before committing it.
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$ROOT" ] || [ ! -d "$ROOT" ]; then
echo "doctor: not inside a git checkout." >&2
exit 1
fi
cd "$ROOT"
rc=0
python3 scripts/nix/check-node-pin.py \
--nvmrc .nvmrc --package-json package.json \
--flake-node ${nodejs.version} --flake-pnpm ${pnpm.version} || rc=1
echo
python3 scripts/nix/check-prisma-pin.py \
--package-json package.json --lockfile pnpm-lock.yaml \
--flake-prisma-version ${prismaVersion} \
--flake-engine-commit ${engineCommit} || rc=1
exit "$rc"
'';
meta.description = "Check the flake's pins against the working tree";
};
in
{
packages.${system} = {
inherit prisma-engines dev dev-server doctor;
default = dev;
};
apps.${system} = {
dev = { type = "app"; program = lib.getExe dev; meta = dev.meta; };
dev-server = { type = "app"; program = lib.getExe dev-server; meta = dev-server.meta; };
doctor = { type = "app"; program = lib.getExe doctor; meta = doctor.meta; };
default = { type = "app"; program = lib.getExe dev; meta = dev.meta; };
};
# `nix flake check` BUILDS checks.* but only EVALUATES packages.* and
# devShells.* -- measured, not assumed. So the three writeShellApplications
# are re-exposed as a check below; without that, their build-time
# shellcheck would never run under `nix flake check`.
checks.${system} = {
# writeShellApplication runs shellcheck + `bash -n` at build time, so
# building these IS the lint.
dev-scripts = pkgs.symlinkJoin {
name = "check-dev-scripts";
paths = [ dev dev-server doctor ];
};
toolchain-pins = pkgs.runCommand "check-toolchain-pins"
{ nativeBuildInputs = [ checkPython ]; }
''
python3 ${./scripts/nix/check-node-pin.py} ${nodeCheckArgs}
touch $out
'';
prisma-pin = pkgs.runCommand "check-prisma-pin"
{ nativeBuildInputs = [ checkPython ]; }
''
python3 ${./scripts/nix/check-prisma-pin.py} ${prismaCheckArgs}
touch $out
'';
# The two guards above only assert that today's pins agree. This one
# asserts the guards can still SEE a disagreement: it breaks each pin on
# purpose and requires the specific guard that owns it to fire, and the
# others to stay silent. Without it, either guard could rot into a
# constant `true` and every check would stay green.
pin-guards-selftest = pkgs.runCommand "check-pin-guards-selftest"
{
nativeBuildInputs = [ checkPython pkgs.bash pkgs.gnused pkgs.gnugrep ];
NODE_SCRIPT = "${./scripts/nix/check-node-pin.py}";
PRISMA_SCRIPT = "${./scripts/nix/check-prisma-pin.py}";
NVMRC = "${./.nvmrc}";
PACKAGE_JSON = "${./package.json}";
LOCKFILE = "${./pnpm-lock.yaml}";
}
''
bash ${./scripts/nix/test-pins.sh}
touch $out
'';
};
devShells.${system}.default = pkgs.mkShell {
buildInputs = [ nodejs pnpm pkgs.openssl ] ++ serviceClients;
# `env` is baked into the cached shell profile, so direnv reloads pay
# nothing for it. Keep anything expensive out of shellHook: nix-direnv
# re-runs the hook on every reload even when the shell itself is cached.
env = devEnv;
shellHook = ''
echo "civitai dev shell: node $(node --version), pnpm $(pnpm --version)"
echo " nix run .#dev bootstrap services + run the app"
echo " nix run .#doctor check toolchain pins"
'';
};
};
}