mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
Zachary Lowden
43d48b42c4
* fix(blocks): the dev-token exemption no longer skips the approved check for every dev token
The App Blocks approved-status predicate short-circuited on `claims.dev === true`,
so a moderator suspension left already-minted dev tokens driving both halves of the
runtime — the 15 tRPC bridge procedures and every `withBlockScope` REST route — for
the remaining life of the token. Dev tokens live 14400s against a 900s default, so
that window was 16x every other token's, on the class with the widest scopes.
The `dev` claim is stamped unconditionally by `signDevScopedPageToken`, which six
mint paths reach. Three of them must run a non-approved app; three must not. Keying
on the bare boolean exempted all six, and the docblock justified it by listing belts
the guard re-checked none of — ownership, an active dev tunnel, the author flags,
the approved-scope clamp.
The predicate now separates them:
- a signed `reviewRunForReal` claim short-circuits ahead of the read, which is the
moderator review sandbox and the one population that must run a non-approved app
without owning it;
- no backing row means nothing to be approved, covering the synthetic-id mints;
- a REAL row that is not approved is exempt only when the subject IS the app's
owner AND that owner has an ACTIVE dev tunnel for the slug — the two preconditions
the owner-dev-tunnel mint enforces, re-derived rather than assumed;
- anything else with a real, non-approved row is refused, which is the `dev:live`
token whose mint required approval and that has simply outlived it.
Ownership comes from one extra selected column on the lookup this path already
performs. The tunnel check is reached only on the dev + real-row + not-approved
path, so no approved app and no non-dev token pays for it.
Revocation is untouched — it was never exempted — and the 4h lifetime is unchanged.
Refs clawgate #571.
* fix(blocks): resolve the dev-token owner in the branch, not as a second query on every request
Findings from this repo's five-lane review, applied. Three of them were wrong
in ways a green suite could not show.
PERF — the owner was read as `app: { select: { userId: true } }` on the row
lookup, and two comments asserted that was "a join on the FK, not a second
query". It is not: the schema enables only `previewFeatures = ["metrics"]`, so
with no `relationJoins` Prisma resolves a nested relation with another round
trip, and because `appId` is a REQUIRED relation that query cannot be skipped on
an empty FK set. It would have fired for every token whose row exists — every
bridge call including the timer-driven `pollWorkflow`, and every block-JWT REST
request — to read a column only the dev branch consults. `claims.appId` IS the
`OauthClient.id`, so the lookup moves into the branch as the same primary-key
read, issued only when needed. The row read returns to `select: { status: true }`
and a test now pins that literal.
TESTS — moving the `dev` check behind the row read silently made the existing
`dev=%p is NOT exempt` table VACUOUS. Its fixture carried no owner, so the
mutant `!claims.dev` fell through to the ownership guard and was refused there:
the table stayed green while the guard it claims to pin never executed. The
repair is in the fixture — clear every other reason to refuse, so the verdict is
attributable to the `dev` comparison alone. Six mutants now each die to their own
test, including this one.
NARROWING — `reviewRunForReal` was exempting without requiring `dev`. Every mint
stamps both, but `sign` accepts the field independently, and a bypass keyed on one
signed boolean without narrowing it is the defect this change exists to fix; alone
it would have been WIDER than the blanket exemption it replaced.
POSTURE — `getActiveDevTunnel` attaches its `.catch()` to the result of
`sysRedis.get(...)`, so a synchronous client throw escapes it, as can the dynamic
import. Unwrapped that becomes a 503 blamed on the replica read. Now wrapped, so
the fail-closed posture is written rather than inherited.
REUSE — `subjectForUserId` already existed in `block-revocation.service` under a
docblock claiming to be "THE ONE PLACE this format is written on the WRITE side".
It was not — the mint open-coded the same template — and this change would have
been a third copy, each pinned by its own literal so no test could see them
diverge. Moved to a zero-import leaf both now use, re-exported so no importer
changes.
Plus the written half: the population table the code referred to now exists as a
table, the `ai:write:budgeted` containment claim is scoped to the population it
is actually true of, and the two sibling resolvers each carry a cross-reference
to the other two.
Refs clawgate #571.
* fix(blocks): log the dev-tunnel re-check failure instead of folding it into the refusal count
Delta round on the previous commit's fixes. The wrapper added there was right
about the verdict and wrong about observability.
A throw out of the dev-tunnel re-check used to reach `resolveRestApprovalVerdict`,
get logged through the throttled limiter with its error message, and answer
`lookup_failed`. Wrapping it turned that into a bare `catch` returning
`not_approved` — correct as a verdict, but it increments the SAME series this
whole change ships to be watched on. A sysRedis fault would have pushed every
population-E owner into `not_approved` fleet-wide with nothing logged anywhere,
and the predicate's own docblock reads that series as "the 4h window closing" —
so the incident would have read as the narrowing working. The one leg the change
added was the one leg it made unobservable.
It now logs, with its own message and its own throttle window. Deliberately a log
rather than a new verdict: a `tunnel_lookup_failed` would have to be mapped by both
callers, and the REST mapping for an unrecognised verdict is 503 — the exact
misattribution (a cache fault blamed on the replica read) the wrapper exists to
avoid. Separate windows because two failure modes sharing one would suppress each
other, and the one you did not see would be the one you needed.
The throttle logic is now written once and closed over per caller, rather than
open-coded twice.
Also from the same round:
- `parseSubjectUserId` and five self-scope gates still spelled 'anon' as a literal
while the new leaf claimed one spelling for the format. They use ANON_SUBJECT now,
so the claim is true rather than nearly true.
- `publisher-ban-revocation` took `subjectForUserId` through
`block-revocation.service`'s re-export — a module wholesale-mocked in a dozen
suites with a factory exporting only `BlockRevocation`, which is the shape the
leaf was extracted to avoid. It imports the leaf directly.
Refs clawgate #571.
* docs(blocks): correct four claims the last round's fix made false
Round 3 of the audit ladder. No behaviour change — every finding was a comment
that the code contradicts, which is the class this file keeps producing because
its docblocks carry the reasoning rather than just describing it.
1. The new tunnel-logger docblock said a dedicated verdict was avoided because
"the REST mapping for an unknown verdict is 503". It is not. The chain in
`withBlockScope` is not_approved -> 403, lookup_failed -> 503, and then an
`else` that asserts `satisfies 'not_found'`, logs "SERVING (observe-only)" and
falls through to the handler. REST's runtime default for an unrecognised
verdict is to SERVE, and it would log the request as a missing row it is not.
The bridge is the opposite: `satisfies never` then an unconditional FORBIDDEN.
So the paragraph told the next author REST fails closed on the exact branch
where it fails open. The real reasons — the two-caller mapping burden, forced
by the compile error at that `satisfies`, and REST's serve-by-default — are
now what it says. It also inverted the trade: a dedicated verdict would be
BETTER attribution than the log, since it would carry its own reason= label
instead of sharing not_approved. Recorded as such, so "add the verdict" stays
available as the deliberate change rather than looking already-rejected.
2. `LOOKUP_FAILURE_LOG_WINDOW_MS`'s docblock now sits above both loggers while
asserting "THE COUNT IS NOT THE ALERTING SIGNAL — the unthrottled
reason=lookup_failed series is". True of the replica-read logger, false of the
tunnel one, whose failures resolve to not_approved and have no dedicated label:
there the throttled log IS the only signal, so a suppressed line is lost
information rather than redundant prose. Same window, opposite relationship to
the metrics.
3. The predicate's "IT DOES NOT CATCH" heading is a blanket claim the previous
commit falsified 130 lines below it, and it names as an anti-pattern exactly
what the tunnel leg now does. Scoped to the ROW reads, with the reason the
exception is right there and wrong for them: an unreachable replica means "we
cannot establish whether this app may run", which is a different question per
caller; an unreachable tunnel cache means "no live tunnel", which is the same
fail-closed answer everywhere.
4. The bridge's "a read that THROWS propagates as the tRPC internal error" is no
longer true of the tunnel read — that caller now gets FORBIDDEN plus a warn
this path never emitted. Scoped, and flagged as the behaviour change it is.
Also: the new log test relied on being the first tunnel failure in the process
for its toHaveBeenCalledTimes(1). It resets the window instead, so it no longer
breaks based on where it sits in the file.
Refs clawgate #571.
* fix(blocks): give the dev-tunnel failure its own verdict — the log it had cannot be read here
Round 4 of the ladder, and it overturns round 2's fix rather than refining it.
Round 2 found the dev-tunnel re-check swallowing a throw into `not_approved`
with no signal, and answered it with a throttled `console.warn`. That answer is
inert on this deployment: `app-block-runtime.metrics.ts` states twice, and
designs around, the fact that application-container logs are NOT collected here
— "the `console.error` shape used elsewhere in the repo would be invisible to a
later investigator". So the fix swapped a silent swallow for an unreadable one,
and the paragraph arguing the log was the signal separating a cache incident
from the stale-token population was wrong about its own environment.
The leg now returns `tunnel_lookup_failed`. It refuses identically to
`not_approved` on both callers — same status, same message, deliberately, since
a bearer learning that the dev-tunnel cache is down would be an infrastructure
oracle — but it carries its own `reason=` label, so an operator can tell a
sysRedis fault from the population this change exists to create. That matters
because `not_approved` is the series the whole narrowing is watched on: folded in,
an incident reads as the fix working.
NOT reused `lookup_failed`, which was the tidier-looking option: that verdict
means the REPLICA read failed, answers 503, and is SERVED on the five routes
declaring `onApprovalLookupFailure: 'serve'`. Both would be wrong — a cache fault
blamed on the database, and a non-approved app served on some routes.
Four edit sites, all compiler-forced: the verdict union, both caller mappings,
and the metric's reason union. The cardinality guard went 3 -> 4 series; its
budget is a deliberate literal, so the new label is argued in place rather than
waved through, and the test now drives the new reason instead of only declaring
it.
Three prose corrections from the same round, each a claim the code denies:
- "the window is shared because the rate argument is identical" — it is not. The
replica logger's case is fleet-wide simultaneity; the tunnel logger is reachable
on one owner's one app. By this repo's own reasoning that shape needs no throttle
at all. Kept, with the real reason.
- "a dedicated verdict would carry its own label" was true only for REST: the
bridge records no verdict metric at all, and the label union is a third edit
site rather than derived.
- "`lookup_failed` -> 503" is route-dependent, the same shape of flat claim round 3
existed to remove, one verdict over.
Refs clawgate #571.
* docs(blocks): teach the verdict docs about the fourth reason, and stop claiming the bridge is covered
Round 5. No behaviour change. Adding `tunnel_lookup_failed` last round left six
places describing a three-verdict world, including the two an operator actually
reads, and repeated this change's own recurring mistake: writing a justification
one surface wider than the fix reaches.
THE ONE THAT MATTERS. The claim that a suppressed log now "loses prose and not
information" is true on REST only. `recordBlockRestApprovalVerdict` has a single
production call site, in `withBlockScope`; `assertAppBlockApproved` resolves the
same verdict and records nothing. So a tunnel-cache fault reached through the
BRIDGE emits no counter at all, and its only trace is the throttled warn — on a
deployment that does not collect container logs. That gap predates this work and
is equally true of `not_approved` (the bridge has never recorded a verdict), so
closing it is a bridge-metrics change rather than a guard one and is not taken
here. What is taken is saying so, in all three places that would otherwise let a
reader infer from the REST series that the bridge is covered — including the
explicit warning that a zero on `reason="tunnel_lookup_failed"` does not mean the
leg is healthy, given the bridge is the higher-rate surface (`pollWorkflow` is
timer-driven).
The operator-facing docs now know the label exists. The metric's `help` string
and its reader table enumerated three reasons and read as exhaustive, which is
the one place a description of `reason` reaches whoever is looking at the series
in Grafana — the whole point of the label was that someone can tell a sysRedis
fault from the stale-token population, and it was undocumented. Likewise the
verdict table in `block-scope.middleware`, which is the gate's own explanation of
the branch chain the last commit edited fourteen lines below it. That table now
carries the row that does NOT bend: `onApprovalLookupFailure` is scoped to
`lookup_failed` alone, so a route wanting lookup-failure tolerance does not get
it here — serving a known non-approved app because a cache was down is not
tolerating an unknown.
Count corrections: 3 -> 4 across the metric docblocks, the middleware table, the
emitter's own docblock and the metrics-test prose.
Two more claims the code denied:
- `resolveRestApprovalVerdict`'s docblock is the canonical mapping (the middleware
points at it rather than restating), and it still listed three verdicts AND
repeated the flat "`lookup_failed` (503) refuse" that the previous commit had
corrected 240 lines above. Fixed one copy, left the authoritative one.
- The `tunnelFailureLog` docblock still opened, present tense, with "converts a
throw into `not_approved`" while its own third paragraph said otherwise.
And two of my own from last round, which is the pattern:
- The new metrics-test comment claimed it proved a production caller emits the
reason. It cannot — the loop iterates the union, so a phantom nobody emits
satisfies it identically. The real pin is in the approved-gate suite, against
the real middleware; this one is the cardinality half and now says so.
- "That second shape does not need throttling at all" understated the tunnel
logger's case: a sysRedis incident hits every pod at once too, so it is the same
simultaneity over a smaller population, not a different shape.
Refs clawgate #571.
* docs(blocks): finish the fourth reason — the count said four where the list said three
Round 6. No behaviour change. Both findings are the previous commit's own half-done
edits, which is the pattern this ladder keeps producing.
The metrics test's reader block was bumped to "the four reasons" over an
enumeration that still listed three. Before that bump it was stale but coherent;
after it, a reader counting rows finds one reason undocumented and has to go
looking for which — and the missing row is the one carrying the new operational
fact, that `tunnel_lookup_failed` is 403 on every route because
`onApprovalLookupFailure` does not reach it. That block is the third reader table
in the same family as the metric `help` string and the middleware table; the other
two got the row and it did not.
Same edit, second instance: a test title went "the three reasons are SEPARATE
series" -> "the four", while its body still drove three. The fourth reason's
separateness was covered elsewhere, so nothing was unproven — but the title read
as the proof and was not, which is exactly the over-claim the same file corrects
two cases below. The case now drives what its title counts.
And the REST-only scoping from last round reached three sites but not the two
that most needed it:
- The `catch` comment inside the SHARED predicate said "the counter is the
signal", flat. That function is the one both callers use and the place a reader
is standing when they ask whether this is observable — and on the bridge there
is no counter.
- The `tunnelFailureLog` docblock closed with "no longer load-bearing", which
directly contradicted the paragraph ninety lines above it stating that a bridge
tunnel failure has this line as its only trace. Both cannot be true of one
logger. The asymmetry is now stated as the argument it is: for giving the bridge
a verdict counter, not for trusting the log.
Refs clawgate #571.
* docs(blocks): the comment welded two slips into one moment, and pointed at the wrong case
Round 7, and both findings are in the five-line comment round 6 added.
It said the reason was "left out when the reason was added — the title said four
while the body drove three". Checked against the branch: at the commit that added
the reason the title still said THREE, and the case was internally consistent —
under-covering the new reason, but claiming nothing it did not prove. The title
was bumped to four in the following docs pass, and that is where it became a
coverage claim wider than the test. Two slips, one commit apart, welded onto a
moment neither of them happened at. The commit message for that pass had the
history right; the in-file comment was a degraded restatement of it.
And "two cases below" is off by one — the case that corrects the same shape is
`emits AT MOST 4 series`, three below. Named rather than counted now, so it
cannot drift again when a case is inserted.
Refs clawgate #571.
Table of Contents
About the Project
Our goal with this project is to create a platform where people can share their stable diffusion models (textual inversions, hypernetworks, aesthetic gradients, VAEs, and any other crazy stuff people do to customize their AI generations), collaborate with others to improve them, and learn from each other's work. The platform allows users to create an account, upload their models, and browse models that have been shared by others. Users can also leave comments and feedback on each other's models to facilitate collaboration and knowledge sharing.
Tech Stack
We've built this project using a combination of modern web technologies, including Next.js for the frontend, TRPC for the API, and Prisma + Postgres for the database. By leveraging these tools, we've been able to create a scalable and maintainable platform that is both user-friendly and powerful.
- DB: Prisma + Postgres
- API: tRPC
- Front-end + Back-end: NextJS
- UI Kit: Mantine
- Storage: Cloudflare
Getting Started
To get a local copy up and running, follow these steps.
Prerequisites
- Docker, with Compose v2 (
docker compose, not the retired hyphenateddocker-compose). The database, Redis, MinIO, Meilisearch, ClickHouse and the mail catcher all run as containers. - Node.js
24.19.0. Not "20 or later" —package.jsondeclaresengines.node: ">=24.0.0 <25". The exact version lives in.nvmrc; CI installs that file's version and the production image is built on the same one, sonvm use(or any tool that reads.nvmrc) is the right way to get it. Note that nothing stops you:pnpm installonly printsWARN Unsupported engineand carries on, so the wrong major surfaces later as odd test failures rather than as a refusal at install time. - pnpm. This repo is pnpm-only, and this one is enforced —
npm installexits 1 via thepreinstallonly-allow pnpmhook.corepack enablewill pick up thepackageManagerfield for you. - Make (optional).
Installation
Standard setup
git clone https://github.com/civitai/civitai.git
cd civitai
nvm use # reads .nvmrc -> 24.19.0
corepack enable
git submodule update --init event-engine-common
cp .env-example .env.development
docker compose -f docker-compose.base.yml up -d
pnpm install
pnpm dev
Optional: Nix flake
Optional, and not the supported default. The standard setup above is what the project expects and what CI builds; nothing in the repo requires Nix, and you can ignore this section entirely. It exists because NixOS cannot use Prisma's published engines (there is no
linux-nixosbuild), so a flake is the practical way to work on this repo there. If you are not on NixOS and not already a flakes user, skip it.
The flake owns the toolchain, so you do not install Node or pnpm yourself:
git clone https://github.com/civitai/civitai.git
cd civitai
nix run .#dev
That single command checks Docker is usable, checks out the
event-engine-common submodule, creates .env.development from .env-example
if you do not already have one, starts the container stack, waits for Postgres,
runs pnpm install, and then starts the dev server on
http://localhost:3000. Every step is idempotent — it is
safe to re-run in a checkout that already works, and it will not overwrite your
.env.development or touch your data.
Useful variants:
nix run .#dev -- --no-start # bootstrap only, leave the services running
nix run .#dev -- --full # also start the signals/buzz containers (see below)
nix run .#doctor # check the flake's pins against the repo
nix flake check # the same checks, plus their own self-test
For an interactive shell with the same toolchain, use nix develop, or copy
.envrc.example to .envrc and run direnv allow to get it
automatically on cd.
With devcontainers
⚠️ Known out of step:
.devcontainer/public/docker-compose.ymlpinsmcr.microsoft.com/devcontainers/typescript-node:1-22, i.e. Node 22, which is outside this repo'sengines.noderange.pnpm installwill warn rather than stop, so the container comes up and then misbehaves in ways that look like your branch. There is no1-24tag (the template major moved on);3-24is the closest equivalent. Not changed here because it could not be exercised.
⚠️ Important Warning for Windows Users: Either clone this repo onto a WSL volume, or use the "clone repository in named container volume" command. Otherwise, you will see performance issues.
- Open the directory up in your IDE of choice
- VS Code should prompt you to "Open in container"
- If not, you may need to manually run
Dev Containers: Open Folder in Container
- If not, you may need to manually run
- For other IDEs, you may need to open the
.devcontainer/devcontainer.jsonfile, and click "Create devcontainer and mount sources" - Note: this may take some time to run initially
- VS Code should prompt you to "Open in container"
- Run
make run
The signals and buzz services
docker-compose.base.yml holds everything a contributor needs (and is also what
nix run .#dev starts). The extra services in docker-compose.yml
(signals, buzz) come from private ghcr.io images, so they only work for
internal members:
- create a GitHub personal access token with
read:packages - set it as
CR_PAT echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin- then
docker compose up -d(or, with the flake,nix run .#dev -- --full)
After the first start
- Edit
.env.development. Most defaults work out of the box; these do not:- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
minioadmin) — note it is port 9001, port 9000 is the S3 API itself — go to "Access Keys", click "Create Access Key", and copy the key and secret intoS3_UPLOAD_KEY/S3_UPLOAD_SECRETandS3_IMAGE_UPLOAD_KEY/S3_IMAGE_UPLOAD_SECRET. WEBHOOK_TOKEN— any random string; it authenticates requests to the webhook endpoint.EMAIL_USER,EMAIL_PASS, andEMAIL_FROM(a valid email format) — any values, but they must be set for user registration to work.
- S3 upload credentials. Open the MinIO console at
http://localhost:9001 (username and password both
- On an empty database, populate it. These are slow and destructive, which is
why no bootstrap runs them for you:
make run-migrations make reseed - Visit http://localhost:3000.
Please report any issues with these commands to us on discord.
* Note that account creation will run emails through maildev, which can be accessed at http://localhost:1080.
Altering your user
- First, create an account for yourself as you normally would through the UI.
- You may wish to set yourself up as a moderator. To do so:
- Use a database editor (like DataGrip) or connect directly to the
DB (
PGPASSWORD=postgres psql -h localhost -p 15432 -U postgres civitai) - Find your user (by email or username), and change
isModeratortotrue
- Use a database editor (like DataGrip) or connect directly to the
DB (
Known limitations
Services that require external input will currently not work locally. These include:
- Orchestration (Generation, Training)
- Signals (Chat, Notifications, other real-time updates)
- Buzz
Contributing
Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the repository to your own GitHub account.
- Create a new branch for your changes.
- Make your changes to the code.
- Commit your changes and push the branch to your forked repository.
- Open a pull request on our repository.
If you would like to be more involved, consider joining the Community Development Team! For more information on the team as well as how to join, see Calling All Developers: Join Civitai's Community Development Team.
Data Migrations
Over the course of development, you may need to change the structure of the database. To do this:
- Make your changes to the
packages/civitai-db-schema/prisma/schema.full.prismafile. Notschema.prisma— that one is gitignored and regenerated fromschema.full.prismabyscripts/generate-slim-schema.json everypnpm run db:generate, so edits to it are silently overwritten. - Run
pnpm run db:migrate:empty "brief description here". This createspackages/civitai-db-schema/prisma/migrations/YYYYMMDDHHmmss_brief_description_here/migration.sqlfor you, in the one directory Prisma reads. To create it by hand instead, use that same path — not theprisma/migrationsdirectory at the repo root, which predates the monorepo layout and is no longer read. - Put your sql changes in the generated
migration.sql- These are usually simple sql commands like
ALTER TABLE ...
- These are usually simple sql commands like
- Run
make run-migrationsandmake gen-prisma - If you are adding/changing a column or table, please try to keep the
gen_seed.tsfile up to date with these changes.
Sponsors
Support this project by becoming a sponsor. Your logo will show up here with a link to your website.
License
Apache License 2.0 - Please have a look at the LICENSE for more details.