Files
civitai__civitai/CLAUDE.md
T
Justin Maier a1bc34d42a feat(cosmetics): perceptual-hash cosmetic artwork (#3534)
* feat(cosmetics): perceptual-hash cosmetic artwork

Cosmetic art never becomes an Image row, so it has never been hashed —
the only duplicate check is a sha256 on the shop item, which a re-encode
defeats. Hash it directly instead: the orchestrator's mediaHash step
stands alone, so a one-step workflow submitted with `wait` returns a
perceptual hash synchronously with no Image row and no scan webhook.

Hashing is fire-and-forget after the write. A cosmetic must land whether
or not the orchestrator answers, and rows left NULL — whether never
attempted or failed — are swept by the backfill endpoint.

Creator submissions still bypass this; they write Cosmetic inline in a
transaction and are handled with submit-time matching in phase 2.

Migration is committed but NOT applied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore(cosmetics): make the pHash migration re-runnable

Migrations here are applied by hand per environment, so a re-run should
no-op rather than error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(cosmetics): re-hash when artwork changes, not when metadata does

Five review findings:

Metadata-only edits re-submitted a workflow. `previous` is only loaded
when `data` is present, so renaming a badge compared against undefined
and always "changed".

Replacing artwork left a stale hash the backfill never revisited.
Several paths swap `data.url` with a raw update, and a row holding the
previous image's hash is worse than holding none — matching would go
looking for artwork that is no longer there. `pHashUrl` records what was
hashed; the backfill re-sweeps wherever it disagrees with `data.url`,
which self-heals regardless of who wrote the url.

Product badges were never hashed on write, and with no cron they would
have stayed NULL until someone ran the endpoint by hand.

One Prisma failure aborted the whole backfill batch, discarding the
tally for every row already done.

A 1000-row default outlived the request that started it; the mod saw a
dropped connection and would re-run against rows still in flight.

Also: abort a submit the orchestrator accepts and abandons, keep the
cosmetic id out of a new Axiom column (civitai-prod is at its field
cap), and skip empty-string urls.

Migration is committed but NOT applied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(cosmetics): correct dryRun parsing, zero hashes, and backfill starvation

`z.coerce.boolean()` is Boolean(value), so `?dryRun=false` asked for a
real run and got a dry one — the operator sees a plausible pending count
and no work done, with nothing to indicate why.

`0n` is a legitimate perceptual hash (solid-colour artwork), and cosmetics
are a corpus where flat frames are ordinary. A falsy check recorded it as
a failure, leaving the row NULL and re-attempted by every later run.

The backfill had no cursor, so rows that can never be hashed — artwork
that 404s on the CDN — held the same head-of-queue slots on every
invocation and stranded everything behind them. Added `afterId`, and
`lastId` in the response to drive it.

Also: warn on the docblock that these hashes are comparable only to other
mediaHash output, not to legacy Image.pHash rows, which came from a
different algorithm and compare as noise with no error to signal it.
Share the one hash conversion rather than keeping two copies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(cosmetics): move the pHash backfill to a one-off script

A mod-only HTTP endpoint is permanent surface area for a temporary job,
and it can't be driven without a browser session — which is why the
backfill hadn't been run yet.

The script requires an explicit --target and prints the database it
resolved before touching anything; there is no default. Arguments are
parsed strictly and an unrecognised one aborts, since a mistyped flag
quietly doing the opposite of what was asked is what this class of bug
looks like.

Its cursor advances past every row it fetches, so artwork that can never
be hashed no longer holds the head of the queue.

If a recurring drift sweep is wanted later, that belongs in a job under
src/server/jobs, not an endpoint.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(cosmetics): correct the backfill invocation and name the lane

The documented command didn't work: npm swallows --target as its own
config option, so the script aborted asking for a target it had been
given. Documents the pnpm exec form, which was verified verbatim.

--target selects the database and nothing else. The orchestrator lane is
separate, and hashes from different lanes aren't comparable, so a dev
lane writing prod rows would look successful and mean nothing. It's now
in the banner rather than left implicit.

Also notes that the `Using PROD database.` line above the banner comes
from importing the db client and ignores --target, and that both targets
are localhost tunnels distinguished only by port.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(cosmetics): load env before the hashing service in the backfill

Every row the backfill submitted failed. ~/env/client snapshots
process.env.NEXT_PUBLIC_* when it is imported; Next inlines those at
build, but a tsx process does not, and ~/env/server calls dotenv in its
module body — after its own import of ~/env/client has already been
evaluated. NEXT_PUBLIC_IMAGE_LOCATION came out undefined, and getEdgeUrl
drops a falsy base from its join rather than failing, so the orchestrator
received a relative path and could not fetch it.

The script now loads dotenv before importing the hashing service.

getPerceptualHash also refuses a media url that isn't absolute, naming
the missing variable. Without that the failure costs one orchestrator
round-trip per row and leaves nothing to read afterwards — the whole
corpus fails and looks like a slow run.

Verified: hashes now returned in ~300-400ms and match the values the
orchestrator returns for the same artwork when called directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(cosmetics): keep the hash helpers out of the search-index graph

Three suites failed to load. cosmetic.service imports ~/server/search-index,
which reaches meilisearch/client — a module that calls pLimit(env...) and
registers prom collectors at import. Importing cosmetic.service from
cosmetic-shop.service and product-badge.service pulled all of that into two
graphs that never had it, past mocks that name modules by hand.

The hash helpers never needed that graph: they want dbWrite, logToAxiom and
getPerceptualHash. Moving them to their own module drops the edge, which
fixes prepaid-membership-jobs with no test change at all.

cosmetic-shop.service still reaches prom/client legitimately, via
orchestrator.service -> workflows -> orchestrator-read-metrics, so those two
mocks now spread the real module with importOriginal instead of listing
exports. A hand-listed mock is coupled to the whole transitive graph of the
thing under test, and nothing warns you when that graph grows.

Verified like-for-like: 3/3 fail before, 11 suites / 53 tests pass after,
matching the same command run against the branch base.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: prefer importOriginal over hand-listed vi.mock exports

Twice in one day a service import dragged a load-time module into a
suite's graph and the hand-listed mock no longer covered it. Typecheck
and lint stay green, so only CI catches it.

Also records the half that's easy to lose: a failing suite may be saying
the code pulled in a dependency it doesn't want, not that the mock is too
narrow. Widening the mock would have hidden that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 20:15:26 -06:00

18 KiB

Civitai Development Guide

How to work with us

We use markdown documents to discuss plans. Documentation goes in the docs/ folder.

Inline Comments

Occasionally, we comment back and forth as we make plans. Comments from us, are marked with @dev: and you can leave comments as well with @ai:. Please make comments inline in the document. If there are actions are requested in my comments, please take them.

New Comment Marking: When you add new comments, use an asterisk after the mention (e.g., @justin:* or @meta:*). Once you reply or acknowledge a comment, remove the asterisk so that I know it's been seen. Note: Sometimes I might forget to add the asterisk to my new comments, so please check all comments regardless of marking.

Example

@dev: This comment has been processed (asterisk removed)
@ai: Of course
@dev:* This is a new comment that needs attention

Tech Stack Overview

Core Technologies

  • Framework: Next.js 14 with TypeScript
  • UI Library: Mantine v7
  • Styling: Tailwind CSS + SCSS Modules
  • Database: PostgreSQL with Prisma ORM
  • API: tRPC
  • State Management: Zustand
  • Authentication: NextAuth
  • Search: Meilisearch
  • Image Processing: Sharp

Additional Libraries

  • React Query (Tanstack Query) for data fetching
  • React Hook Form with Zod validation
  • Tiptap for rich text editing
  • Chart.js for data visualization
  • Stripe/Paddle/PayPal for payments

Build Commands

Development

Always use the /dev-server skill to manage dev servers. Never use pnpm run dev directly.

Build & Deploy

pnpm run build            # Production build

Code Quality

pnpm run typecheck        # Run TypeScript type checking
pnpm run lint             # Run ESLint
pnpm run prettier:check   # Check Prettier formatting
pnpm run prettier:write   # Auto-fix Prettier formatting

Testing

pnpm test                 # Run Playwright tests
pnpm run test:ui          # Run tests with UI

Never put unit tests under src/pages

Next.js 16 treats every .ts/.tsx file under src/pages (incl. nested __tests__/) as a route, and next build runs a route-type validator over it. A Vitest test file there fails the build with Type '...test' does not satisfy the constraint 'ApiRouteConfig'. Property 'default' is missing — and only next build catches it: pnpm typecheck, pnpm test/vitest, and the CI typecheck/unit/component tasks all pass, so it sneaks through to the preview build-image step. Keep handler tests in a __tests__/ dir outside src/pages (e.g. src/server/__tests__/) and import the handler via the ~/pages/... alias. (Bit us on PR #2653.)

Prefer importOriginal over hand-listed vi.mock exports

A vi.mock that lists exports by hand couples the test to the entire transitive import graph of the thing under test, and nothing warns you when that graph grows. Adding one service import can drag in a module that builds pLimit/prom collectors at load (e.g. ~/server/search-indexmeilisearch/client), and the suite then fails to load with an error far from the change — pnpm typecheck and pnpm lint stay green, so only CI catches it. Spread the real module and override only what you need:

vi.mock('~/server/prom/client', async (importOriginal) => ({
  ...(await importOriginal<typeof PromClient>()),
  dbReadFallbackCounter: { inc: vi.fn() },
}));

Use a top-level import type * as PromClient — an inline typeof import('...') trips consistent-type-imports.

Before widening a mock, check whether the import edge is needed at all. A failing suite may be telling you the code pulled in a dependency it doesn't want, not that the mock is too narrow, and widening it would hide that. (Bit us twice in one day, Aug 2026, on two branches; one of those three suites was fixed by extracting the helpers into their own module instead.)

Database

pnpm run db:migrate:empty  # Create an empty migration file

CRITICAL: We do NOT use prisma migrate deploy. Migrations are applied manually.

  • Migration files in packages/civitai-db-schema/prisma/migrations/ exist for review/history but are never auto-run. That is the only directory Prisma reads — the prisma/migrations/ path at the repo root predates the monorepo, no longer exists, and CI blocks re-creating it.
  • Each environment's DB is updated by a human running the SQL directly (psql, retool, etc.)
  • The _prisma_migrations table is not the source of truth — do not rely on it
  • When you add a new migration: write the SQL, commit it, and surface to the user that it needs to be applied manually to wherever they want it (preview / staging / prod)
  • Never suggest prisma migrate deploy, prisma migrate resolve, or any auto-apply path

Release (requires user permission)

pnpm run release          # Patch release (0.0.x) - default
pnpm run release:minor    # Minor release (0.x.0)
pnpm run release:major    # Major release (x.0.0)

IMPORTANT: Never run release commands without explicit user approval. These commands bump the version, push tags, and rebase the release branch.

Server-Side Architecture Map

src/server/ holds the most-edited (and largest) code in the repo. Read the specific file before changing it — several are huge, so grep within them rather than reading end-to-end (services/image.service.ts is ~7.9K lines).

  • tRPC APItrpc.ts (root router + procedure helpers), createContext.ts, middleware.trpc.ts, routers/ (~93 per-domain routers), controllers/, schema/ (zod input contracts), selectors/ (Prisma select fragments).
  • Imagesservices/image.service.ts (~7.9K lines; the hot feed path — getInfiniteImages, getAllImages, NSFW/own-content merge). API surface src/pages/api/v1/images/index.ts; index sync search-index/images.search-index.ts.
  • Modelsservices/model.service.ts, search-index/models.search-index.ts.
  • Search (Meilisearch)meilisearch/client.ts (tags requests with X-Search-Actor), meilisearch/cleanup.ts, search-index/base.search-index.ts (shared sync engine).
  • Redis / cachingredis/client.ts (clients incl. sysRedis), redis/caches.ts (createCachedObject defs + TTLs, e.g. imageMetaCache, tagIdsForImagesCache), utils/cache-helpers.ts.
  • Orchestrator (generation)orchestrator/get-orchestrator-token.ts (getOrchestratorToken), services/orchestrator/orchestrator.service.ts.
  • Authauth/next-auth-options.ts, auth/session-user.ts, auth/token-refresh.ts.
  • Jobs (cron)jobs/job.ts (runner) + individual jobs jobs/*.ts (e.g. entity-moderation.ts, search-index-sync.ts).
  • Metrics / analyticsmetrics/*.metrics.ts (ClickHouse-backed entity metrics), clickhouse/.
  • DBdb/db-helpers.ts (raw pg-pool config: connectionTimeoutMillis, labeled pool gauges), Prisma client; schema prisma/schema.prisma. Migrations are applied manually — see the Database rule above.
  • Telemetrysrc/instrumentation.node.ts (OTEL: Prisma/Redis/HTTP auto-instrumentation + custom withSpan() from utils/otel-helpers.ts), schema/track.schema.ts (ClickHouse action/event tags), prom/client.ts.
  • Healthsrc/pages/api/health.ts runs sub-checks under Promise.all; a single slow check (e.g. searchMetrics) can exceed the kubelet probe budget. HEALTHCHECK_TIMEOUT env gates it.
  • Other server domainsgames/ (new-order/ratings), webhooks/, paddle/ + coinbase/ (payments), notifications/, signals/, rewards/; S3 helpers at src/utils/s3-utils.ts.

Component Standards

File Structure

src/
├── components/          # React components
│   ├── ComponentName/   # Component folder
│   │   ├── ComponentName.tsx
│   │   ├── ComponentName.module.scss  # Optional SCSS module
│   │   └── utils.ts     # Component utilities
├── hooks/              # Custom React hooks
├── server/             # Server-side code
├── utils/              # Shared utilities
└── store/              # Zustand stores

Component Patterns

1. Mantine Components

import { Button, Group, Text } from '@mantine/core';
import { IconBolt } from '@tabler/icons-react';

2. Tailwind Classes with clsx

import clsx from 'clsx';

<div className={clsx('flex items-center gap-2', conditionalClass && 'bg-blue-500')} />

3. SCSS Modules (when needed)

import styles from './Component.module.scss';

<div className={styles.container} />

4. TypeScript Patterns

  • Use type imports when possible: import type { ButtonProps } from '@mantine/core'
  • Define Props interfaces for components
  • Use enums from ~/shared/utils/prisma/enums

Coding Standards

Imports Order

  1. External libraries (React, Mantine, etc.)
  2. Internal components (~/components/...)
  3. Hooks (~/hooks/...)
  4. Server/API code (~/server/...)
  5. Utils and helpers (~/utils/...)
  6. Types and enums
  7. Styles

State Management

  • Use Zustand for global state
  • Use React Query for server state
  • Use React Hook Form for forms

API Calls

import { trpc } from '~/utils/trpc';

const { data, isLoading } = trpc.user.getProfile.useQuery();

Authentication

import { useCurrentUser } from '~/hooks/useCurrentUser';

const currentUser = useCurrentUser();

Comments

Comments are not type-checked, so they rot silently and become misleading. Write the minimum comment needed and bias toward none.

  • Default to no comment. If the code is clear on its own, leave it alone. Prefer a clearer name, smaller method, or better type over a comment that explains confusing code.
  • Only comment the non-obvious why: a rationale, tradeoff, gotcha, invariant, or workaround that the reader cannot recover from the code itself. Link an issue/PR when relevant.
  • Never narrate the what. No comments that restate the next line, label obvious steps (// loop over items), or describe what a well-named symbol already says.
  • Don't describe nearby code's current behavior (e.g. "this gates on X so Y happens"). That is exactly what goes stale when the other code changes. Comment the surprising fact, not the mechanics.
  • No process/banner noise: no change-log narration (// added to fix...), no "I changed X", no section-divider banners, no commented-out code.
  • When you do comment, keep it to a line or two. A long block almost always means the code or naming should be clearer instead.

Explain decisions in your response, not in the file. Rationale for a choice you just made — why you picked this shape, what you deliberately left out, what you considered and rejected — belongs in your reply to us, where we're already reading it. A comment justifying your work to a reviewer is the single most common way this section gets violated. If you catch yourself writing something you'd also say in chat, say it in chat only.

Comment in a separate pass. Write the code first with no comments, then reread it and add back only what's needed. Comments written while authoring never get evaluated — the reasoning is fresh, so it feels non-obvious when it isn't. Judge them against code you're reading, not code you're writing.

The keep test. For every comment that survives, you should be able to name the specific future edit that goes wrong without it. If the answer is "it's helpful context" or "it explains why this is correct," delete it. Being unable to name the failure means the code already says it — or should.

Clean up as you go. When you edit code that already has stale, redundant, or what-narrating comments, delete or fix them — don't preserve them just because they were there. The repo already has many such comments (a lot of them mine); treat touching nearby code as license to remove the noise, but keep edits scoped to what you're already working on rather than going on a separate comment-cleanup sweep.

Environment Setup

Required Environment Variables

  • Database connection strings
  • Authentication providers
  • S3/CloudFlare credentials
  • Payment provider keys
  • Search service endpoints

Local Development

  1. Install dependencies: pnpm install
  2. Generate Prisma client: pnpm run db:generate
  3. Start dev server: Use /dev-server skill

Git Worktrees

When you create a new worktree (git worktree add …), always initialize the event-engine-common submodule in it: git submodule sync --recursive && git submodule update --init event-engine-common. Worktrees don't check out submodules automatically, and without it pnpm typecheck/build fail with a wall of Cannot find module '.../event-engine-common/...' errors (and the missing types cascade into unrelated implicitly has an 'any' type errors) — noise that looks like your change broke something when it didn't.

Important Notes

  • Read the full file before editing. Plan all changes, then make ONE complete edit. If you've edited a file 3+ times, stop and re-read the user's requirements.
  • When the user corrects you, stop and re-read their message. Quote back what they asked for and confirm before proceeding.
  • Every few turns, re-read the original request to make sure you haven't drifted from the goal.
  • Act sooner. Don't read more than 3-5 files before making a change. Get a basic understanding, make the change, then iterate.
  • When stuck, summarize what you've tried and ask the user for guidance instead of retrying the same approach.
  • Re-read the user's last message before responding. Follow through on every instruction completely.
  • After 2 consecutive tool failures, stop and change your approach entirely. Explain what failed and try a different strategy.

Performance

  • Use dynamic imports for heavy components
  • Implement virtual scrolling for large lists
  • Optimize images with Next.js Image component

Security

  • Never commit secrets or API keys
  • Use environment variables
  • Sanitize user input with sanitize-html
  • Follow authentication best practices

Before Committing

  1. Run type checking: pnpm run typecheck
  2. Run linting: pnpm run lint
  3. Format code: pnpm run prettier:write
  4. Test changes locally

Stacked PRs — don't

  • NEVER use stacked PRs — base every PR directly on the integration branch (main, or a feature integration branch like feat/...), never on another open PR's branch. Stacked PRs silently mis-merge: a squash-merged parent doesn't retarget the child, so the child lands on the orphaned parent branch instead of the real base and its changes go missing.
  • If a change depends on an unmerged PR, wait for that PR to merge, then branch off the updated base — or fold both changes into a single PR.
  • (Bit us 2026-06-13: PR #2520's App Blocks W11 F5 was stacked on #2518 (F6) → #2520 squash-merged into the #2518 branch instead of feat/app-blocks-main-v1; corrected via #2525.)

Common Patterns

Infinite Scroll

Use MasonryGrid or virtual scrolling components with React Query infinite queries.

Modals

Use Mantine modals with proper accessibility and keyboard handling.

Dialog Registry System

The project uses a dialog-registry system for managing modals:

  • Register dialogs in src/components/Dialog/dialog-registry.ts or dialog-registry2.ts
  • Use DialogProvider for context-based modal management
  • RoutedDialogProvider for URL-based modal state
  • Access dialogs through the registry for consistent modal handling across the app

Forms

Use React Hook Form with Zod schemas for validation.

File Uploads

Use the S3 upload hooks and providers in the codebase.

Image Handling

Use EdgeImage component for optimized image loading with CDN support.

Debug Endpoints (src/pages/api/testing/*)

src/pages/api/testing/*.ts is the convention for hidden debug endpoints. Each endpoint is guarded by WEBHOOK_TOKEN (via WebhookEndpoint(...), which checks the ?token= query param) and exposes a handful of POST actions for experimenting with a feature without paying real money or hand-editing the DB.

To use one: read the endpoint's source file directly — the top-of-file comment documents the available actions and required params, and the zod schema is the authoritative contract. Agents should never need a wrapper skill; cURL with ?token=$WEBHOOK_TOKEN appended to the URL is enough.

When adding a new debug endpoint:

  1. Drop it at src/pages/api/testing/<feature>.ts
  2. Use WebhookEndpoint(handler) for auth
  3. Lead the file with a block comment listing each action + its params + a one-line description (see src/pages/api/testing/referrals.ts for the pattern)
  4. Scope every destructive action to a single userId/refereeId per call so a misuse can't cascade

Feature Documentation

Feature-specific documentation lives in docs/features/. Before implementing a feature, check if documentation exists:

Core Systems Reference

System Documentation
Image Resources docs/features/image-resources.md
NSFW Filtering docs/features/nsfw-filtering.md
Buzz Accounts docs/features/buzz-accounts.md
Notifications docs/features/notifications.md
Metrics/Analytics docs/features/metrics-analytics.md
Bitwise Flags docs/features/bitwise-flags.md
Civitai LLM Client docs/features/civitai-llm-client.md
Challenge Platform docs/features/challenge-platform.md

Troubleshooting

Memory Issues

Use cross-env NODE_OPTIONS with increased memory:

pnpm run dev-debug  # Includes --max_old_space_size=8192

Build Failures

  1. Clear .next folder
  2. Clear node_modules and reinstall
  3. Check for circular dependencies
  4. Ensure all environment variables are set

Database Issues

  1. Check connection string
  2. Apply pending migrations manually (we do NOT use prisma migrate deploy — see Database section above)
  3. Regenerate client: pnpm run db:generate