mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
chore: bootstrap monorepo package layout (pure file moves + planning docs)
Relocate infrastructure into packages/* as rename-only moves (history preserved); no content changes to moved files. Build is intentionally broken until re-export shims land in follow-up commits.
- packages/civitai-db-schema: Prisma schema, migrations, generated enums/models (contract layer)
- packages/civitai-db: Prisma-client + pg-pool runtime
- packages/civitai-{redis,clickhouse,axiom,telemetry}: infra clients
- docs/: conversion plan, package adaptation plan, directory snapshot, handoff
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Briant Diehl
@@ -0,0 +1,207 @@
|
||||
# Moderator App — Shared Module Boundary
|
||||
|
||||
## Goal
|
||||
|
||||
Make the 22 moderator pages listed below buildable in a **separate Next.js app** that shares Postgres / Redis / ClickHouse connections with the main app. Identify which existing code must be extracted into a git submodule (or submodules) to make this possible without forking the whole repo.
|
||||
|
||||
Related docs: [monorepo-split-overview.md](./monorepo-split-overview.md) (why split), [civitai-schema-common-plan.md](./civitai-schema-common-plan.md) (data-contract submodule, prerequisite for this).
|
||||
|
||||
## Pages in scope (22)
|
||||
|
||||
**Content review**: `images`, `images/to-ingest`, `image-tags`, `image-rating-review`, `downleveled-review`, `ingestion-error-review`, `articles`, `models/index`, `comics-review`, `reports`, `tags`, `blocklists`, `auditor`, `strikes`
|
||||
**Scanner audit**: `scanner-audit/index`, `scanner-audit/[mode]/index`, `scanner-audit/[mode]/[label]` (includes per-label policy editing)
|
||||
**Training**: `training-models`, `review/training-data/index`, `review/training-data/[versionId]`
|
||||
**CSAM**: `csam/index`, `csam/[userId]`
|
||||
**Generation**: `generation`, `generation-config`, `generation-restrictions`
|
||||
|
||||
## Dependency map (from import analysis)
|
||||
|
||||
The 22 pages import code across six tiers. Counts below = how many of the 22 pages import each item.
|
||||
|
||||
### Tier A — schema / data contracts (covered by `civitai-schema-common`)
|
||||
|
||||
| Import | Pages |
|
||||
|---|---|
|
||||
| `~/shared/utils/prisma/enums` | 14 |
|
||||
| `~/server/common/enums` (NsfwLevel, BlockedReason, BlocklistType, etc.) | 6 |
|
||||
| `~/shared/constants/browsingLevel.constants` | 3 |
|
||||
| `~/server/common/constants` | 2 |
|
||||
| `~/server/schema/report.schema`, `strike.schema`, `image.schema`, `scanner-review.schema` | 5 |
|
||||
| `~/shared/constants/basemodel.constants` | 1 (generation-config) |
|
||||
| `~/shared/constants/mime-types`, `~/shared/utils/report-helpers` | 2 |
|
||||
|
||||
**Verdict:** All belong in `civitai-schema-common`. The Prisma `~/server/schema/*.schema.ts` zod schemas are a borderline case — they're tRPC input contracts but also the source-of-truth shape for many of these tables. Worth including the moderator-relevant ones since both apps need to agree on them.
|
||||
|
||||
### Tier B — tRPC client + auth glue (per-consumer, not shared)
|
||||
|
||||
| Import | Pages |
|
||||
|---|---|
|
||||
| `~/utils/trpc` | **22** (all) |
|
||||
| `~/server/utils/server-side-helpers` (createServerSideProps) | 6 |
|
||||
| `~/providers/FeatureFlagsProvider`, `~/hooks/useFeatureFlags` | 5 |
|
||||
| `~/hooks/useCurrentUser`, `~/hooks/useIsMobile`, `~/hooks/useInView`, `~/hooks/useStepper` | 7 (combined) |
|
||||
| `~/types/router` (inferred tRPC types) | 2 |
|
||||
|
||||
**Verdict:** Each app needs its own copy. The satellite app stands up its own tRPC client pointing at its own router (or proxies to the main app's router). Hooks like `useCurrentUser`/`useIsMobile` are small enough to vendor-copy.
|
||||
|
||||
### Tier C — moderator-specific shared components (candidates for a moderator submodule)
|
||||
|
||||
| Import | Pages |
|
||||
|---|---|
|
||||
| `~/components/Meta/Meta` | 9 |
|
||||
| `~/components/AppLayout/Page`, `~/components/AppLayout/NotFound` | 7 |
|
||||
| `~/components/Moderation/*` (ScannerAuditLayout, ScannerPolicySidebar, FlaggedModelsList, RuleDefinitionPopover, GenerationStatusCard, UserGenerationsDrawer) | 5 |
|
||||
| `~/components/Csam/*` (CsamProvider, CsamDetailsForm, CsamImageSelection, useCsamImageSelectStore) | 4 |
|
||||
| `~/components/Dialog/dialogStore`, `~/components/Dialog/Common/TosViolationDialog`, `~/components/Dialog/triggers/*` | 6 |
|
||||
| `~/components/Profile/UserBanModal` | 2 |
|
||||
| `~/store/select.store` | 2 |
|
||||
|
||||
**Verdict:** These are predominantly used by the moderator pages — natural fit for a moderator-specific submodule.
|
||||
|
||||
### Tier D — Civitai UI vocabulary (the awkward middle)
|
||||
|
||||
These are not moderator-specific — they're used everywhere in the main app — but the moderator pages can't render without them.
|
||||
|
||||
| Import | Pages |
|
||||
|---|---|
|
||||
| `~/components/NextLink/NextLink` | 16 |
|
||||
| `~/components/EdgeMedia/*` (EdgeMedia, EdgeVideo, EdgeVideoBase) | 8 |
|
||||
| `~/components/NoContent/NoContent` | 6 |
|
||||
| `~/components/LegacyActionIcon/LegacyActionIcon` | 6 |
|
||||
| `~/components/InView/InViewLoader` | 4 |
|
||||
| `~/components/ImageGuard/ImageGuard2` | 3 |
|
||||
| `~/components/ImageMeta/ImageMeta`, `~/components/VotableTags/VotableTags`, `~/components/ImageHash/ImageHash`, `~/components/MasonryColumns/*`, `~/components/ContentClamp/ContentClamp`, `~/components/RenderHtml/RenderHtml`, `~/components/DescriptionTable/DescriptionTable`, `~/components/PopConfirm/PopConfirm`, `~/components/CivitaiWrapped/ButtonTooltip` | 16 (combined) |
|
||||
|
||||
**Verdict:** Three options:
|
||||
1. **Vendor-copy into satellite** — simple, but the two apps' UI will drift visually over time
|
||||
2. **Promote to a `civitai-ui-common` submodule** — clean long-term, but a *big* extraction (every main-app file importing these gets rewritten)
|
||||
3. **Start vendored, promote later** — pay the duplication cost early, promote when drift becomes painful
|
||||
|
||||
Recommend option 3.
|
||||
|
||||
### Tier E — domain helpers (vendor-copy or duplicate)
|
||||
|
||||
| Import | Pages |
|
||||
|---|---|
|
||||
| `~/utils/notifications` (showError/SuccessNotification) | 15 |
|
||||
| `~/utils/string-helpers` | 9 |
|
||||
| `~/utils/date-helpers` | 6 |
|
||||
| `~/libs/form` (Form, InputTextArea, InputNumber, InputSelect, useForm) | 3 |
|
||||
| `~/utils/moderators/moderator.util`, `~/utils/number-helpers`, `~/utils/file-utils`, `~/utils/lazy`, `~/utils/training`, `~/utils/type-guards`, `~/utils/normalize-text`, `~/utils/metadata/audit`, `~/client-utils/cf-images-utils`, `~/hooks/useCheckProfanity` | scattered |
|
||||
|
||||
**Verdict:** Small, mostly-pure utilities. Easiest path is vendor-copy into satellite; promote anything heavily reused to `civitai-ui-common` along with Tier D when the time comes. `useCheckProfanity` is the exception — it hits a server endpoint, so its hook must travel with the auditor page and the satellite needs the backing endpoint.
|
||||
|
||||
### Tier F — porting blockers (need refactor in place before extraction)
|
||||
|
||||
| Import | Page | Blocker |
|
||||
|---|---|---|
|
||||
| `~/components/ImageGeneration/GenerationForm/generation.utils` (`useUnsupportedResources`) | `generation.tsx` | Pulls in generation form machinery → ecosystem handlers → orchestrator service. Refactor: extract the resource-availability table logic from form-internal helpers. |
|
||||
| `~/server/services/image.service` (type imports) | `image-rating-review`, `downleveled-review`, `ingestion-error-review` | Type-only imports today, but the *underlying functions* must run somewhere. Either satellite hosts these tRPC routes (drags in image.service) or the satellite calls main-app over HTTP. |
|
||||
| `~/server/services/scanner-review.service`, `~/server/services/scanner-content.service` | `scanner-audit/[mode]/[label]` | Scanner audit queue + per-label content fetch. Tied to xguard orchestrator callbacks. Refactor: clarify what's a read-only query vs. what's an orchestrator interaction. |
|
||||
| `~/server/common/moderation-helpers` (`unpublishReasons`) | `articles`, `models` | Moderator-only domain logic. Should be promoted into Tier A (schema-common) — it's effectively a stable enum. |
|
||||
| `~/components/Image/PromptHighlight/PromptHighlight`, `useReportCsamImages` | `images.tsx` | PromptHighlight drags in metadata audit utilities; CSAM hook couples to main app's dialog/notification machinery. Refactor: extract pure highlighter; reimplement CSAM hook against satellite's dialog system. |
|
||||
|
||||
## Per-page portability summary
|
||||
|
||||
| Page | Imports | Portability |
|
||||
|---|---|---|
|
||||
| `images/to-ingest` | 5 | **Easy** |
|
||||
| `image-rating-review` | 15 | **Easy** |
|
||||
| `downleveled-review` | 11 | **Easy** |
|
||||
| `ingestion-error-review` | 9 | **Easy** |
|
||||
| `comics-review` | 18 | **Easy** |
|
||||
| `strikes` | 17 | **Easy** |
|
||||
| `review/training-data/index` | 12 | **Easy** |
|
||||
| `image-tags` | 21 | **Medium** |
|
||||
| `articles` | 20 | **Medium** (unpublishReasons) |
|
||||
| `models/index` | 16 | **Medium** (FlaggedModelsList) |
|
||||
| `reports` | 28 | **Medium** |
|
||||
| `tags` | 20 | **Medium** |
|
||||
| `blocklists` | 11 | **Medium** |
|
||||
| `training-models` | 16 | **Medium** |
|
||||
| `review/training-data/[versionId]` | 19 | **Medium** |
|
||||
| `csam/index` | 8 | **Medium** |
|
||||
| `csam/[userId]` | 15 | **Medium** |
|
||||
| `generation-config` | 9 | **Medium** (basemodel.constants) |
|
||||
| `images` | 34 | **Hard** (PromptHighlight, CSAM hooks) |
|
||||
| `auditor` | 4 | **Hard** (useCheckProfanity) |
|
||||
| `scanner-audit/[mode]/index` | 14 | **Hard** (ScannerAuditLayout, xguard) |
|
||||
| `scanner-audit/[mode]/[label]` | 18 | **Hard** (scanner-content service) |
|
||||
| `generation` | 11 | **Hard** (useUnsupportedResources) |
|
||||
| `generation-restrictions` | 14 | **Hard** (UserGenerationsDrawer) |
|
||||
|
||||
7 Easy / 11 Medium / 6 Hard
|
||||
|
||||
## Recommended module structure
|
||||
|
||||
Three submodules, introduced incrementally:
|
||||
|
||||
### 1. `civitai-schema-common` — already planned
|
||||
|
||||
Per [civitai-schema-common-plan.md](./civitai-schema-common-plan.md). Covers Tier A. Prerequisite for everything below.
|
||||
|
||||
**Addition to that plan from this analysis:**
|
||||
- Include the moderator-relevant `~/server/schema/*.schema.ts` zod files (`report`, `strike`, `image`, `scanner-review`, `buzz-withdrawal-request`, `model-version`)
|
||||
- Promote `~/server/common/moderation-helpers.unpublishReasons` and other stable enum-shaped exports from `~/server/common/enums` and `~/server/common/constants` (NsfwLevel, BlockedReason, BlocklistType, MAX_APPEAL_MESSAGE_LENGTH, etc.)
|
||||
|
||||
### 2. `civitai-moderator-common` — new
|
||||
|
||||
Tier C contents plus the moderator pages themselves once they're portable. Satellite app's `src/pages/moderator/*` are thin re-exports.
|
||||
|
||||
```
|
||||
civitai-moderator-common/
|
||||
├── components/
|
||||
│ ├── Moderation/ # ScannerAuditLayout, ScannerPolicySidebar,
|
||||
│ │ FlaggedModelsList, RuleDefinitionPopover,
|
||||
│ │ GenerationStatusCard, UserGenerationsDrawer
|
||||
│ ├── Csam/ # CsamProvider, CsamDetailsForm,
|
||||
│ │ CsamImageSelection, useCsamImageSelectStore
|
||||
│ ├── Dialog/ # dialogStore + moderator-specific dialogs
|
||||
│ └── Profile/UserBanModal/
|
||||
├── pages/ # the 22 page components (after refactor)
|
||||
├── server/
|
||||
│ ├── routers/ # moderator-* tRPC routers
|
||||
│ └── services/ # moderator-specific service code that satellite
|
||||
│ must run locally (or, alternatively, thin
|
||||
│ clients that call main app's tRPC)
|
||||
└── README.md
|
||||
```
|
||||
|
||||
### 3. `civitai-ui-common` — deferred
|
||||
|
||||
Tier D contents (EdgeMedia, ImageGuard2, MasonryColumns, NextLink, etc.). Promoted from vendor-copy when drift becomes painful. Likely a 6–12 month follow-on, not blocking the initial split.
|
||||
|
||||
## Phased rollout
|
||||
|
||||
**Phase 0** — Ship `civitai-schema-common` per the existing plan (phases 1–5). Satellite app cannot start before this.
|
||||
|
||||
**Phase 1** — Refactor the 6 Tier F blockers **in place** in the main app:
|
||||
- Extract `useUnsupportedResources` from generation form coupling
|
||||
- Extract `PromptHighlight` from metadata-audit coupling
|
||||
- Refactor `useReportCsamImages` to depend on a dialog-system interface, not the concrete main-app dialog store
|
||||
- Decide: scanner-content service satellite-owned, or tRPC-proxied to main app?
|
||||
- Decide: image.service moderator queries satellite-owned, or tRPC-proxied?
|
||||
|
||||
**Phase 2** — Stand up satellite Next.js app with:
|
||||
- `civitai-schema-common` submodule
|
||||
- Vendor-copied Tier D (UI primitives) and Tier E (helpers) — accept duplication
|
||||
- Its own tRPC client + auth setup
|
||||
- Port the 7 Easy pages first as proof-of-concept
|
||||
|
||||
**Phase 3** — Create `civitai-moderator-common` submodule. Move Tier C components into it. Move moderator pages into it as they become portable. Both main app and satellite import from the submodule (during transition, main app keeps the pages working; eventually main app drops them).
|
||||
|
||||
**Phase 4** — Migrate Medium pages (11) once Tier F refactors are done.
|
||||
|
||||
**Phase 5** — Migrate Hard pages (6). Most are hard because of single specific couplings — once those are addressed, they drop to Medium.
|
||||
|
||||
**Phase 6 (deferred)** — Promote Tier D into `civitai-ui-common` when duplication friction is real.
|
||||
|
||||
## Open questions
|
||||
|
||||
@dev:* Decisions needed before Phase 1:
|
||||
|
||||
1. **tRPC topology.** Does the satellite app run its own routers against shared DB (drags in services into the submodule), or call main-app's tRPC over HTTP (simpler, but adds network hop and couples uptime)?
|
||||
2. **Scope of `civitai-moderator-common`.** Just components + server code, or the full pages too?
|
||||
3. **Page authorship during transition.** While moving a page from main app → submodule, do we keep it working in both, or cut over hard?
|
||||
4. **Auth.** Satellite needs to know "is this user a moderator." Does it share the NextAuth session cookie with main app (same domain), use its own login, or call main-app `/api/auth/session` to verify?
|
||||
5. **CSAM page handling.** CSAM is the most sensitive thing in the list. Confirm it should move to the satellite at all (vs. staying in main app for security review/audit-trail reasons).
|
||||
@@ -0,0 +1,122 @@
|
||||
# Session Handoff — Monorepo Bootstrap
|
||||
|
||||
This document captures the state and reasoning from the planning session that produced this worktree, so a fresh Claude session can pick up the work without re-discovering decisions.
|
||||
|
||||
## Where you are
|
||||
|
||||
- **Worktree:** `c:\Work\model-share-monorepo-bootstrap`
|
||||
- **Branch:** `monorepo-bootstrap` (branched from `main`)
|
||||
- **HEAD:** at latest `origin/main` (synced before the moves)
|
||||
- **Nothing is committed on this branch yet** — all changes are staged, awaiting review.
|
||||
|
||||
## What's staged (uncommitted)
|
||||
|
||||
Run `git status --short` to see the live state. As of handoff:
|
||||
|
||||
- **616 renames (R)** — pure file moves, zero content changes. Every file is at 100% similarity (`R100`).
|
||||
- **3 new docs (A)** — the planning docs in `docs/`:
|
||||
- `monorepo-conversion-plan.md` — the full plan with phases, decisions, and operational details
|
||||
- `monorepo-directory-snapshot.md` — shareable view of the post-conversion directory layout
|
||||
- `moderator-app-shared-modules.md` — separate dependency analysis for the future moderator app (referenced from the conversion plan but doesn't gate this work)
|
||||
|
||||
The staged moves implement **Phase 1–5 file relocations** into `packages/civitai-{db,redis,clickhouse,axiom,telemetry}/`. Build is intentionally broken at this point — re-export shims (described in the plan) come in follow-up commits.
|
||||
|
||||
## Key decisions (with reasoning)
|
||||
|
||||
These were settled in conversation; read them before suggesting alternatives.
|
||||
|
||||
1. **Five base packages, not six.** No `civitai-schema-common`. The user's instinct was right: the only files that would have lived there were domain constants (bit-flag interpretations like `browsingLevel.constants.ts`, identifier formats like `air.ts`, and the generic `flags.ts` helper). Those are **not infrastructure** — they're domain conventions. They stay in `src/shared/` in the main app for now. Re-evaluate only when the satellite app actually needs them.
|
||||
|
||||
2. **Base packages are infrastructure only.** `civitai-db`, `civitai-redis`, `civitai-clickhouse`, `civitai-axiom`, `civitai-telemetry`. Each wraps a single piece of runtime infrastructure.
|
||||
|
||||
3. **Base packages don't import from each other.** No `@civitai/*` deps on a sibling base package. External deps only. Higher-level packages (e.g. a future `civitai-moderator-common`) may compose multiple base packages, but base packages stay independent. If two base packages need shared constants, expose them via a subpath (e.g. `@civitai/redis/keys`).
|
||||
- **Amended (2026-06-03):** one exception — a **contract/leaf layer** below the infra packages. `@civitai/db-schema` is a pure schema + generated-types artifact with no runtime; infra packages may depend **downward** on it (like depending on `@prisma/client`). The rule still forbids one infra package importing *another infra package's runtime* — a types/schema package is a lower layer, not a sibling.
|
||||
|
||||
4. **~~All Prisma stuff lives in `civitai-db`.~~ AMENDED (2026-06-03): split into two packages.** The Prisma *schema, migrations, programmability, generated client, `enums.ts`, `models.ts`* move down into **`@civitai/db-schema`** (the source-of-truth contract). **`@civitai/db`** keeps only the Prisma-client *runtime* (`createPrismaClients` factory, pg `Pool` factory, query helpers) and imports the generated client/types from `@civitai/db-schema`.
|
||||
- **Why:** the [`civitai-advertising`](file:///C:/Work/civitai-advertising) app drives **Kysely** off a Prisma-generated schema (two generators on one `schema.prisma`; runtime is `new Kysely<DB>()` over raw `pg`, never the Prisma client). Splitting the contract out lets a future app pick Kysely without dragging in the Prisma-client runtime.
|
||||
- **`prisma-kysely` is baked in now:** the schema package runs a second generator (`prisma-kysely`) emitting Kysely `DB` types via subpath `@civitai/db-schema/kysely`, so a Kysely consumer is unblocked without re-touching the schema package.
|
||||
- Still **6 packages**, distinct from the rejected `civitai-schema-common` (decision 1) — that was *domain constants*; this is the *Prisma DB contract*. A future second DB schema still gets its own package. See `docs/monorepo-package-adaptation-plan.md` §4 for the full layout.
|
||||
|
||||
5. **Main app stays at repo root.** Symmetric `apps/main/` layout (Phase 6 in the plan) is **not planned**. Skipped to avoid a freeze-week mass-move. New apps live in `apps/`; main stays at root indefinitely.
|
||||
|
||||
6. **Re-export shims are kept indefinitely.** The plan to keep old import paths (`~/server/db/client`, etc.) as one-line re-exports forwarding to `@civitai/db` is the drift-tolerant approach. Means main-app call sites never need touching. Two valid import paths for the same code is acceptable.
|
||||
|
||||
7. **OTEL auto-instrumentation stays per-app.** `src/instrumentation.node.ts` keeps the `sdk.start()` call (with the app-specific service name); only the helpers (`withSpan`, etc.) and the prom-client helpers move into `@civitai/telemetry`. Same applies to per-app `prom-client` registration.
|
||||
|
||||
8. **Factory pattern, not module-level singletons.** Today's `db-helpers.ts` exports `dbWrite` as a constant that reads `env` directly. In the monorepo state, each package exports a `createXClients(config)` factory; each app instantiates with its own env and a `serviceLabel` (for prom-client labels). Main-app call sites don't change because a thin wrapper file (the shim) re-exposes the same names.
|
||||
|
||||
## Constraints that affect implementation
|
||||
|
||||
- **Prisma client output** moves into the package via `output = "../generated/client"` in `schema.full.prisma`. Both apps import from `@civitai/db/client`, never from `@prisma/client` directly. This avoids two-copies-with-nominally-different-types.
|
||||
- **Dockerfile** has two existing `COPY` lines pinning Prisma schema paths (lines 11 and 38 of the current Dockerfile) — they need updating to `packages/civitai-db/prisma/...`. Workspace install layer needs to copy `pnpm-workspace.yaml` + all `packages/*/package.json` before `pnpm install`.
|
||||
- **Next.js `output: 'standalone'`** needs `transpilePackages: ['@civitai/*']` added to `next.config.mjs` (the alternative, `outputFileTracingRoot`, only matters if packages have pre-built `tsc` output, which we're not doing).
|
||||
- **Manual migrations.** Civitai applies Prisma migrations manually (`prisma migrate deploy` is forbidden per `CLAUDE.md`). The move doesn't change that — migrations move to `packages/civitai-db/prisma/migrations/` and human still runs them by hand. The `scripts/prisma-migrate-with-views-workaround.mjs` references need their paths updated.
|
||||
- **`db-helpers.ts` has a circular-dep landmine.** Line 7 imports `dbWrite` from `~/server/db/client.ts`. When both move into the same package, this becomes an in-package cycle. Untangle by passing `dbWrite` into helpers that need it, or restructure so the two files don't import each other.
|
||||
- **HMR-safe prom-client registration.** `db-helpers.ts:30-35` has a try/catch fallback for re-registering the Histogram when HMR re-runs the module. Keep that pattern — it also handles multiple apps in the same process during testing.
|
||||
|
||||
## Planning docs in this repo
|
||||
|
||||
Read these (in order) before making decisions:
|
||||
|
||||
1. **`docs/monorepo-conversion-plan.md`** — the source of truth. Has phases, all open-question decisions with `@dev:`/`@ai:` exchanges, operational details (Docker, CI, Next standalone).
|
||||
2. **`docs/monorepo-directory-snapshot.md`** — shareable directory-tree view of the post-conversion state.
|
||||
3. **`docs/moderator-app-shared-modules.md`** — separate analysis for the future moderator app (relevant context, not blocking this work).
|
||||
|
||||
## What's next (the actual work)
|
||||
|
||||
The current staged commit is the **file-relocation step**. After it's committed, the remaining phases are:
|
||||
|
||||
**Phase 0 — Workspace bootstrap** (config only, no file changes):
|
||||
|
||||
- Add `pnpm-workspace.yaml` with `packages: ['.', 'packages/*', 'apps/*']`
|
||||
- Add `transpilePackages: ['@civitai/*']` to `next.config.mjs`
|
||||
- Create `package.json` for each of the 5 new packages (`@civitai/db`, `@civitai/redis`, `@civitai/clickhouse`, `@civitai/axiom`, `@civitai/telemetry`) with name + version + main field
|
||||
- Each gets a `tsconfig.json` (copy the pattern from `event-engine-common/tsconfig.json`)
|
||||
|
||||
**Phase 1 — `@civitai/db`** (this is the heaviest one):
|
||||
|
||||
- Add `@prisma/client`, `prisma`, `pg`, `prom-client` to `packages/civitai-db/package.json`
|
||||
- Update `schema.full.prisma` with `output = "../generated/client"`
|
||||
- Update root scripts (`db:generate`, `db:migrate`, etc.) to point at the new path
|
||||
- Refactor `db-helpers.ts` into a `createDbClients(config)` factory
|
||||
- Untangle the `client.ts` ↔ `db-helpers.ts` circular dep
|
||||
- Write re-export shim at `src/server/db/client.ts` (and others) that calls `createDbClients` with main-app env and re-exports the same names. Existing call sites in main app keep working unchanged.
|
||||
- Verify: `pnpm run typecheck`, `pnpm run build`, `pnpm run dev` all work.
|
||||
|
||||
**Phases 2–5 — same pattern for redis, clickhouse, axiom, telemetry.** Each is mostly mechanical after Phase 1 establishes the factory + shim pattern. See the plan for specifics.
|
||||
|
||||
**Phase 6** — skipped (per decision 5 above).
|
||||
|
||||
## Commit shape (user's preference TBD)
|
||||
|
||||
The user hasn't committed to a commit-split scheme yet. Two reasonable options:
|
||||
|
||||
- **One commit:** `chore: bootstrap monorepo layout (pure file moves + planning docs)`
|
||||
- **Two commits:** `docs: ...` then `refactor: move infra files to packages/* (pure rename)`
|
||||
|
||||
The two-commit split is what the previous worktree used and reads cleaner. Either works for rename-detection. Ask the user before committing.
|
||||
|
||||
## Things to be careful about
|
||||
|
||||
- **Don't combine move + edit in one commit.** Git rename detection has a 50% similarity threshold. If you `git mv` + edit content in the same commit, history may not follow. The current staged state is pure-move; preserve that by committing it before any refactors.
|
||||
- **Don't squash-merge the migration PRs.** Squash collapses the move/refactor/shim commits into one big diff, raising the risk that Git misses renames. Use rebase-merge or merge-commit for the monorepo conversion PR(s).
|
||||
- **Origin's `monorepo-conversion` branch is an artifact.** Previous worktree (now deleted) had an earlier rename-detection test on branch `monorepo-conversion` (commit `22d242dee` with the now-rejected schema-common split). That branch is left untouched on origin as a record of the rename test. Don't try to "fix" it.
|
||||
- **Verify rename detection after any rebase.** If you rebase this branch and the rebase combines commits, re-check `git log --stat HEAD~..HEAD` shows `R100` entries, not paired `A`+`D`.
|
||||
- **The 3 planning docs are currently staged as new files (`A`).** Same commit will include them. If you split commits, decide which commit they belong to.
|
||||
|
||||
## Quick verification commands
|
||||
|
||||
```bash
|
||||
# How many staged renames?
|
||||
git -C c:/Work/model-share-monorepo-bootstrap status --short | awk '{print $1}' | sort | uniq -c
|
||||
|
||||
# Are renames at 100% similarity?
|
||||
git -C c:/Work/model-share-monorepo-bootstrap diff --cached -M --stat | tail -5
|
||||
|
||||
# Where does a file's history live now? (spot-check rename detection)
|
||||
git -C c:/Work/model-share-monorepo-bootstrap log --follow --oneline -5 -- packages/civitai-db/src/db-helpers.ts
|
||||
|
||||
# Confirm worktree is at latest main
|
||||
git -C c:/Work/model-share-monorepo-bootstrap fetch origin main
|
||||
git -C c:/Work/model-share-monorepo-bootstrap log --oneline HEAD..origin/main
|
||||
```
|
||||
@@ -0,0 +1,364 @@
|
||||
# Monorepo Conversion Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Convert the existing repo into a pnpm workspace monorepo, starting by extracting **globally-used infrastructure** (Prisma schema/client, Postgres pools, Redis, ClickHouse, Axiom logging) into shared packages. Future apps (the moderator app from [moderator-app-shared-modules.md](./moderator-app-shared-modules.md)) consume those packages instead of importing from `~/server/...`.
|
||||
|
||||
Supersedes an earlier submodule-based proposal — what was going to be a submodule is now a workspace package, and the package set has narrowed to infrastructure only (db, redis, clickhouse, axiom, telemetry).
|
||||
|
||||
## Why pnpm workspaces (not Turborepo, not Nx)
|
||||
|
||||
- Package manager is already pnpm 10.28
|
||||
- Workspaces are a single-line config change in root `package.json`
|
||||
- No build orchestration needed initially — Next.js compiles packages transparently via `transpilePackages`
|
||||
- Turborepo can be layered on later for CI caching if build times become a problem; not required day-1
|
||||
|
||||
## Base-package rule
|
||||
|
||||
**Base packages do not import from each other.** Each base package (db, redis, clickhouse, axiom, telemetry) is self-contained — external deps only, no `@civitai/*` deps on a sibling base package. Higher-level packages (e.g. future `civitai-moderator-common`) may compose multiple base packages, but base packages stay independent. If two base packages need shared constants, they own their own copy or expose them via a subpath (e.g. `@civitai/redis/keys`). This keeps each base package usable on its own and prevents a brittle dependency graph between low-level layers.
|
||||
|
||||
**Base packages are infrastructure only.** Civitai-specific domain constants (bit-flag interpretations like `browsingLevel.constants.ts`, identifier formats like `air.ts`, bitwise helpers like `flags.ts`) stay in the main app. They're not infrastructure — they're domain conventions. Extracting them is deferred to whenever the satellite app actually needs them.
|
||||
|
||||
## Layout decision: main app stays at root
|
||||
|
||||
```text
|
||||
/
|
||||
├── package.json # root: pnpm workspace config + main app deps
|
||||
├── pnpm-workspace.yaml # new
|
||||
├── next.config.mjs # main app config (unchanged)
|
||||
├── src/ # main app source (unchanged)
|
||||
├── prisma/ # MOVES to packages/civitai-db
|
||||
├── packages/
|
||||
│ ├── civitai-db/
|
||||
│ ├── civitai-redis/
|
||||
│ ├── civitai-clickhouse/
|
||||
│ ├── civitai-axiom/
|
||||
│ └── civitai-telemetry/ # OTEL helpers (withSpan, etc.) — auto-instrumentation stays per-app
|
||||
└── apps/
|
||||
└── moderator/ # added later
|
||||
```
|
||||
|
||||
**Why main at root, not `apps/main/`:** moving 2,500+ files into `apps/main/` triggers the exact catastrophe [monorepo-split-overview.md](./monorepo-split-overview.md) was right to reject. Leaving main at root means `~/` imports never change. New apps live in `apps/`, shared code in `packages/`. The asymmetry is a feature: it lets the conversion be incremental.
|
||||
|
||||
If symmetry becomes important later (e.g., a third app makes the root-as-app pattern feel weird), `git mv src/ apps/main/src/` becomes a one-off cleanup once the workspace exists.
|
||||
|
||||
## Workspace bootstrap (Phase 0)
|
||||
|
||||
1. Add `pnpm-workspace.yaml`:
|
||||
```yaml
|
||||
packages:
|
||||
- '.'
|
||||
- 'packages/*'
|
||||
- 'apps/*'
|
||||
```
|
||||
2. Add to root `package.json`: `"workspaces": ["packages/*", "apps/*"]` (informational; pnpm uses the yaml).
|
||||
3. Add `transpilePackages: ['@civitai/*']` to `next.config.mjs` so Next compiles workspace packages on demand (no separate build step needed).
|
||||
4. Create empty package directories with minimal `package.json` files (`name: "@civitai/db"`, `version: "0.0.0"`, `main: "src/index.ts"`).
|
||||
5. Verify `pnpm install` works and `pnpm run typecheck` still passes.
|
||||
|
||||
**Checkpoint:** workspace is alive, nothing imported from it yet.
|
||||
|
||||
## Package extraction order
|
||||
|
||||
Move packages in dependency order. Each phase ends with `pnpm install`, `pnpm run typecheck`, `pnpm run build` all green.
|
||||
|
||||
### Phase 1: `@civitai/db` (Postgres — schema, client, pools)
|
||||
|
||||
Everything Postgres in one package: the Prisma schema, generated client,
|
||||
migrations, programmability scripts, pg connection pools, and helpers.
|
||||
A future second DB schema (e.g. analytics, separate product) would get
|
||||
its own package (`myapp-db`) — this one is the canonical Civitai schema.
|
||||
|
||||
**Move:**
|
||||
- `prisma/schema.full.prisma` → `packages/civitai-db/prisma/schema.full.prisma`
|
||||
- `prisma/migrations/` → `packages/civitai-db/prisma/migrations/`
|
||||
- `prisma/programmability/` → `packages/civitai-db/prisma/programmability/`
|
||||
- `prisma/seed.ts` → `packages/civitai-db/prisma/seed.ts`
|
||||
- `src/server/db/client.ts` → `packages/civitai-db/src/client.ts` (Prisma client wrapper)
|
||||
- `src/server/db/db-helpers.ts` → `packages/civitai-db/src/db-helpers.ts` (553 lines — pg pools, `cancellableQuery`, prom-client histogram registration)
|
||||
- `src/server/db/pgDb.ts`, `notifDb.ts`, `datapacketDb.ts`, `db-lag-helpers.ts` → `packages/civitai-db/src/`
|
||||
- `src/shared/utils/prisma/enums.ts` → `packages/civitai-db/src/enums.ts` (Prisma-generated)
|
||||
- `src/shared/utils/prisma/models.ts` → `packages/civitai-db/src/models.ts` (Prisma-generated)
|
||||
|
||||
**Prisma client output:** add to schema:
|
||||
```prisma
|
||||
generator client {
|
||||
provider = "prisma-client-js"
|
||||
output = "../generated/client"
|
||||
}
|
||||
```
|
||||
The client now lives *inside* the package, not in root `node_modules/.prisma/client`. Both apps import from `@civitai/db/client`.
|
||||
|
||||
**Package exports** (set up `exports` field in `package.json`):
|
||||
```json
|
||||
{
|
||||
"exports": {
|
||||
".": "./src/index.ts",
|
||||
"./client": "./generated/client/index.js",
|
||||
"./enums": "./src/enums.ts"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Update root scripts:**
|
||||
- `db:generate`: now runs inside the package
|
||||
- `db:migrate`: paths in migration scripts get updated
|
||||
|
||||
**Refactor for monorepo:**
|
||||
|
||||
The current code uses module-level singletons that read `env` directly:
|
||||
```typescript
|
||||
// today
|
||||
import { env } from '~/env/server';
|
||||
const instanceUrlMap = { primary: env.DATABASE_URL, ... };
|
||||
export const dbWrite = createClient('primary');
|
||||
```
|
||||
|
||||
For a package consumed by N apps, expose a **factory**:
|
||||
```typescript
|
||||
// packages/civitai-db/src/index.ts
|
||||
export function createDbClients(config: {
|
||||
databaseUrl: string;
|
||||
databaseReplicaUrl?: string;
|
||||
notificationDbUrl: string;
|
||||
notificationDbReplicaUrl?: string;
|
||||
datapacketReplicaUrl?: string;
|
||||
serviceLabel: string; // for prom-client labels — distinguishes apps
|
||||
}): {
|
||||
dbWrite: PrismaClient;
|
||||
dbRead: PrismaClient;
|
||||
pgDb: AugmentedPool;
|
||||
notifDb: AugmentedPool;
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Main app keeps a thin wrapper at `src/server/db/client.ts`:
|
||||
```typescript
|
||||
import { createDbClients } from '@civitai/db';
|
||||
import { env } from '~/env/server';
|
||||
|
||||
const clients = createDbClients({
|
||||
databaseUrl: env.DATABASE_URL,
|
||||
// ...
|
||||
serviceLabel: 'civitai-app',
|
||||
});
|
||||
|
||||
export const { dbWrite, dbRead, pgDb, notifDb } = clients;
|
||||
```
|
||||
Call sites in main app **don't change** — they still `import { dbWrite } from '~/server/db/client'`. Only the implementation moves.
|
||||
|
||||
**Gotchas:**
|
||||
- `db-helpers.ts:7` imports `dbWrite` from `~/server/db/client` — that's an in-package circular dep once the file moves. Untangle: pass `dbWrite` into helpers that need it, or restructure so `client.ts` and `db-helpers.ts` don't import each other.
|
||||
- The prom-client `Histogram` registration uses a try/catch fallback for HMR (`db-helpers.ts:30-35`). Keep that pattern — it also handles multiple apps in the same process during testing.
|
||||
- `Prisma.Sql` type imports come from `@prisma/client`, which lives in this package. Same runtime, same types — just a different module path.
|
||||
|
||||
### Phase 2: `@civitai/redis`
|
||||
|
||||
**Move:**
|
||||
- `src/server/redis/client.ts` (1,105 lines — clients + helpers)
|
||||
- `src/server/redis/caches.ts` (1,571 lines — the cache key constants, TTLs, and `createCachedObject` infrastructure)
|
||||
- `src/server/redis/queues.ts`, `entity-metric.redis.ts`, `entity-metric-populate.ts`, `resource-data.redis.ts`, `fail-open-log.ts`
|
||||
- `src/utils/cache-helpers.ts` if it's pure helpers (verify)
|
||||
|
||||
**Factory pattern:**
|
||||
```typescript
|
||||
export function createRedisClients(config: {
|
||||
redisUrl: string;
|
||||
sysRedisUrl: string;
|
||||
failOpenLogger?: (event: object) => void;
|
||||
}): {
|
||||
redis: RedisClient;
|
||||
sysRedis: RedisClient;
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
The `fail-open-log.ts` currently uses `logToAxiom` directly. Inject the logger via config so the redis package doesn't hard-depend on `@civitai/axiom` — base packages don't import each other.
|
||||
|
||||
**Cache key constants:** the key strings + TTLs live inside this package (extracted from `caches.ts` into `keys.ts`) and are exported via a subpath: `@civitai/redis/keys`. A consumer that only wants the keys (e.g. a script that invalidates cache without instantiating a redis client) imports `@civitai/redis/keys` directly; tree-shaking keeps the redis client out of their bundle.
|
||||
|
||||
### Phase 3: `@civitai/clickhouse`
|
||||
|
||||
**Move:** `src/server/clickhouse/client.ts` (733 lines).
|
||||
|
||||
Same factory pattern as `@civitai/db`. ClickHouse client is simpler — single URL, no replica routing.
|
||||
|
||||
```typescript
|
||||
export function createClickhouseClient(config: {
|
||||
url: string;
|
||||
database: string;
|
||||
username: string;
|
||||
password: string;
|
||||
}): ClickHouseClient;
|
||||
```
|
||||
|
||||
If ClickHouse query helpers live in services (e.g., event tracking helpers), leave those in the app — only the connection layer moves.
|
||||
|
||||
### Phase 4: `@civitai/axiom`
|
||||
|
||||
**Move:** `src/server/logging/client.ts` (58 lines — `axiom`, `safeError`, `logToAxiom`).
|
||||
|
||||
Smallest, simplest package. Almost a single-file package, but useful as a clean dependency boundary.
|
||||
|
||||
```typescript
|
||||
export function createAxiomLogger(config: {
|
||||
token?: string;
|
||||
orgId?: string;
|
||||
datastream?: string;
|
||||
podName?: string;
|
||||
echoToStderr?: boolean;
|
||||
}): { logToAxiom, safeError };
|
||||
```
|
||||
|
||||
### Phase 5: `@civitai/telemetry`
|
||||
|
||||
OTEL is **two things** that have to be separated:
|
||||
|
||||
1. **Auto-instrumentation registration** — `src/instrumentation.node.ts` calls `sdk.start()` and patches Prisma/Redis/HTTP at process load. **This stays per-app** (every app has its own `instrumentation.node.ts` with its own service name). Cannot move into a package without losing the auto-load behavior.
|
||||
2. **Helpers** — `withSpan`, span attribute helpers, the `src/utils/otel-helpers.ts` utilities. **These move** into `@civitai/telemetry`.
|
||||
|
||||
The package can also export a `bootstrapOtel(config)` function that does what `instrumentation.node.ts` does today, so the per-app file becomes a 3-liner:
|
||||
```typescript
|
||||
// apps/moderator/src/instrumentation.node.ts
|
||||
import { bootstrapOtel } from '@civitai/telemetry/node';
|
||||
bootstrapOtel({ serviceName: 'civitai-moderator' });
|
||||
```
|
||||
|
||||
`prom-client` metrics registration (`src/server/prom/client.ts`) follows the same pattern — helpers in the package, registration call in the app.
|
||||
|
||||
## Cross-cutting concerns
|
||||
|
||||
### env validation
|
||||
|
||||
Currently in `src/env/server.ts` (T3-style zod validation). Two options:
|
||||
- **Per-app env:** each app validates its own env. Packages receive parsed config via factories (preferred — already aligned with the factory pattern above).
|
||||
- **Shared env package:** `@civitai/env` exports the zod schema. Brittle when apps need different subsets.
|
||||
|
||||
Recommend per-app env. Packages never import `env` directly.
|
||||
|
||||
### Migration tooling
|
||||
|
||||
`prisma/migrations/` moves to `packages/civitai-db/prisma/migrations/`. The migration scripts in `scripts/` (e.g., `prisma-migrate-with-views-workaround.mjs`) update their paths. Manual application convention (per CLAUDE.md) doesn't change.
|
||||
|
||||
### Prisma client version pinning
|
||||
|
||||
`@prisma/client` and `prisma` (CLI) move to `packages/civitai-db/package.json`. Main app no longer declares them directly — depends on `@civitai/db` which depends on `@prisma/client`. Single Prisma version across the workspace.
|
||||
|
||||
### CI
|
||||
|
||||
- One `pnpm install` at workspace root installs everything
|
||||
- `pnpm -r run typecheck` typechecks all packages + apps
|
||||
- `pnpm run build` (in main app) still produces a Next standalone bundle
|
||||
- Existing CI workflows mostly survive — main entry points are unchanged
|
||||
- Add `paths:` filters to `pr-check.yml` so changes inside `apps/moderator/` don't retrigger main-app builds (and vice versa); without it every PR rebuilds everything
|
||||
|
||||
### Docker
|
||||
|
||||
Two specific changes to the existing `Dockerfile`:
|
||||
|
||||
1. **Workspace-aware install layer.** Before `pnpm install`, copy `pnpm-workspace.yaml` plus every `package.json` in the workspace (root + `packages/*` + `apps/*`), not just root `package.json`. This preserves the install-layer caching trick — lockfile + package.jsons rarely change, so the install layer stays warm.
|
||||
|
||||
```dockerfile
|
||||
COPY pnpm-lock.yaml pnpm-workspace.yaml package.json ./
|
||||
COPY packages/*/package.json ./packages/
|
||||
COPY apps/*/package.json ./apps/ # only after satellites exist
|
||||
RUN pnpm install --frozen-lockfile
|
||||
```
|
||||
|
||||
2. **Updated Prisma schema paths.** Current Dockerfile has two `COPY` lines that pin `prisma/schema.full.prisma` and `prisma/schema.prisma` for layer caching — update both to `packages/civitai-db/prisma/...`.
|
||||
|
||||
For the satellite app's eventual `apps/moderator/Dockerfile`, use `pnpm deploy --filter <app> --prod /tmp/out` to produce a slim deployment bundle containing only that app's transitive deps. That bundle becomes the COPY source for the runner stage. This is the pnpm-blessed pattern for monorepo Docker images.
|
||||
|
||||
### Next.js `output: 'standalone'` with workspace packages
|
||||
|
||||
The main app's production runtime depends on `.next/standalone` (Dockerfile line 61). Standalone mode uses `nft` (Node File Trace) to determine what to ship. With workspace packages, you need **one of**:
|
||||
|
||||
1. **`transpilePackages: ['@civitai/*']`** in `next.config.mjs` — Next inlines the package source into the build output. Simplest, already in the Phase 0 plan.
|
||||
2. **`outputFileTracingRoot: path.join(__dirname, '../..')`** — tells `nft` to follow symlinks to the workspace root. Required only if packages have their own `tsc --build` step producing pre-built `.js`.
|
||||
|
||||
Going with option 1. Option 2 only matters if you later want each package to have a build output for some reason (publishing, faster cold builds with caching).
|
||||
|
||||
### Tooling explicitly not needed
|
||||
|
||||
- **Turborepo / Nx.** Worth it when many packages × many apps × slow CI. For 6 packages + 2 apps, raw pnpm is simpler. Add later if CI build time becomes a real problem.
|
||||
- **Changesets / lerna.** Only for externally-published packages. `workspace:*` handles internal versioning.
|
||||
- **TypeScript project references** (`references` in tsconfig). Next + `transpilePackages` handles cross-package compilation transparently.
|
||||
- **Separate publish/registry setup.** Packages stay internal.
|
||||
|
||||
### Migration of existing branches
|
||||
|
||||
Best to do this on a **freeze week** with minimal active branches. Each open branch will need a rebase that picks up the package boundary. The shim-export approach (Phase 1's re-export trick) softens the blow: branches that haven't been touched still compile because old import paths keep working.
|
||||
|
||||
### Git history preservation
|
||||
|
||||
Git tracks renames *implicitly* — it infers them at read time by comparing deleted vs. added file content per commit, with a default 50% similarity threshold. To make sure `git log --follow` and `git blame` keep working across the move, follow this pattern per file:
|
||||
|
||||
**Three-commit-per-file pattern:**
|
||||
|
||||
1. **Pure move.** `git mv src/server/db/db-helpers.ts packages/civitai-db/src/db-helpers.ts` — no content changes. Git unambiguously detects the rename.
|
||||
2. **Refactor.** Now edit imports, swap `env` for factory config, untangle circular deps. Small diff, correctly attributed via blame.
|
||||
3. **Add shim** (if applicable). Create the new one-liner re-export at the old path. It's a genuinely new file — no history needed because the original content's history lives at the package path.
|
||||
|
||||
**Tools the team should know:**
|
||||
|
||||
- `git log --follow <path>` — shows full history across renames. Plain `git log <path>` does *not*.
|
||||
- `git blame` — follows simple renames automatically. `git blame -C` also detects content copied from other files (useful for file *splits*).
|
||||
- GitHub web UI — follows renames in "View blame" and file history without flags.
|
||||
|
||||
**Pitfalls to avoid:**
|
||||
|
||||
- **Combining move + significant edit in one commit.** If content similarity drops below 50%, Git misses the rename and history becomes findable but not followable.
|
||||
- **Squash-merging the migration PR.** Collapses the three-commit pattern into one big diff, increasing the chance rename detection fails. Use **rebase-merge** or a **merge commit** for migration PRs.
|
||||
- **File splits.** If `db-helpers.ts` is split into multiple files during the move, Git only auto-detects the rename for the file with the highest content overlap. Prefer move-then-split (move file, then in a later commit split it) over split-then-move.
|
||||
|
||||
## Decisions
|
||||
|
||||
1. **Package naming.** `@civitai/...` scoped, or unscoped (`civitai-...`)?
|
||||
@dev: scoped is good
|
||||
@ai: Confirmed — all packages use the `@civitai/` scope. Reserve the npm org name (even if never publishing publicly, it makes future publishing trivial and avoids name collisions).
|
||||
|
||||
2. **Single `@civitai/data` package vs. four (`db`, `redis`, `clickhouse`, `axiom`)?**
|
||||
@dev: I'm fine with four
|
||||
@ai: Confirmed — four narrow packages. A future text-only utility app can depend on `@civitai/db` alone without pulling in ClickHouse or Axiom.
|
||||
|
||||
3. **Schema-common scope.** Should it include the `~/server/schema/*.schema.ts` zod files used by moderator pages, or defer those to a later phase?
|
||||
@dev: If there is server schema constants that we need, then they should be extracted. I don't think any of these smaller sub-projects should need zod schemas to work
|
||||
@ai: Agreed. The Phase 1–5 infrastructure packages (db/redis/clickhouse/axiom/telemetry) don't import any zod schemas — they only need Prisma types. Any `~/server/schema/*.schema.ts` extraction is deferred until the moderator-app phase, when we know exactly which schemas the satellite needs (`report.schema`, `strike.schema`, `image.schema`, `scanner-review.schema` are the likely candidates from the moderator dependency analysis — see [moderator-app-shared-modules.md](./moderator-app-shared-modules.md)).
|
||||
|
||||
4. **Re-export shims (Phase 1 transition).** Keep them indefinitely (drift-tolerant) or set a deadline to delete them after main-app migration completes?
|
||||
@dev: Not sure what you mean there. We probably want to keep them drift tolerant
|
||||
@ai: Clarified — "re-export shim" means leaving the old file (e.g. `src/server/db/client.ts`) as a one-line file that re-exports from the new package: `export * from '@civitai/db'` (or a thin wrapper that calls `createDbClients` with main-app env). Existing imports `~/server/db/client` continue compiling unchanged. **Drift-tolerant** = keep these shim files indefinitely so we never need to mass-rewrite the ~2,500 call sites in main app. Going with drift-tolerant. Cost is having two valid import paths for the same code (some convention noise in reviews); benefit is zero forced churn.
|
||||
|
||||
5. **Move main app to `apps/main/` eventually?**
|
||||
@dev: I think we would move it to apps/main/ eventually.
|
||||
@dev (update): Skipping Phase 6 for now to avoid needing a freeze week. New apps live in `apps/`; main app stays at root indefinitely. Revisit only if a third app or strong reason emerges.
|
||||
@ai: Acknowledged. Phase 6 is now genuinely optional — the shim approach and factory-pattern packages don't depend on it. Main-at-root is fully supported by pnpm workspaces as a first-class member (`packages: ['.', 'packages/*', 'apps/*']`). The asymmetric layout has no functional cost; the only friction is mild aesthetic inconsistency if a third app eventually joins.
|
||||
|
||||
### Phase 6 (optional, not planned): move main app to `apps/main/`
|
||||
|
||||
Currently **skipped indefinitely** to avoid the freeze-week cost. The asymmetric layout (main at root + satellites in `apps/`) is the planned permanent state.
|
||||
|
||||
If a future trigger justifies revisiting — a third app, strong consistency preference, tooling that breaks on asymmetry — the move would be:
|
||||
|
||||
- `git mv src/ apps/main/src/`
|
||||
- `git mv next.config.mjs apps/main/`
|
||||
- `git mv prisma-related root scripts → apps/main/` (only the app-specific ones, not workspace-level)
|
||||
- Update root `package.json` → workspace-only (no app deps)
|
||||
- Update `pnpm-workspace.yaml` to drop root from the packages list
|
||||
- Update CI workflows that target the root
|
||||
- Requires a freeze week with no active feature branches
|
||||
|
||||
## Phase summary
|
||||
|
||||
| Phase | Package | Effort | Risk |
|
||||
|---|---|---|---|
|
||||
| 0 | Workspace bootstrap | Low | Low — just config |
|
||||
| 1 | `@civitai/db` | High | Medium — Prisma client path change is the trickiest single step; circular dep in `db-helpers.ts` to untangle |
|
||||
| 2 | `@civitai/redis` | Medium | Low — lots of files, mostly mechanical |
|
||||
| 3 | `@civitai/clickhouse` | Low | Low |
|
||||
| 4 | `@civitai/axiom` | Low | Low |
|
||||
| 5 | `@civitai/telemetry` | Medium | Medium — splitting auto-instrumentation from helpers requires care |
|
||||
| 6 | Move main app to `apps/main/` | — | **Not planned** — kept at root indefinitely to avoid freeze-week cost |
|
||||
|
||||
After Phase 5, the foundation is in place for `apps/moderator` to exist as a workspace member that consumes these packages — that's a separate plan (see [moderator-app-shared-modules.md](./moderator-app-shared-modules.md)).
|
||||
@@ -0,0 +1,213 @@
|
||||
# Monorepo — Directory Snapshot
|
||||
|
||||
This shows what the repo will look like after the planned monorepo conversion lands. See [monorepo-conversion-plan.md](./monorepo-conversion-plan.md) for the full plan, phases, and rationale.
|
||||
|
||||
## Scope of this snapshot
|
||||
|
||||
State **after Phase 6** (foundation packages extracted), **before optional Phase 7** (which moves the main app into `apps/main/`).
|
||||
|
||||
## Tooling
|
||||
|
||||
- **pnpm workspaces** (we already use pnpm 10.28). No Turborepo / Nx required.
|
||||
- All shared packages use the `@civitai/` scope.
|
||||
- Four narrow infrastructure packages instead of one bundled `@civitai/data`, so a future text-only utility app can depend on `@civitai/db` alone without pulling in ClickHouse or Axiom.
|
||||
- Re-export shims keep all existing `~/...` imports in the main app working unchanged — no mass-rewrite of call sites.
|
||||
|
||||
## Base-package rule
|
||||
|
||||
**Base packages do not import from each other.** Each package below (`civitai-db`, `civitai-redis`, `civitai-clickhouse`, `civitai-axiom`, `civitai-telemetry`) is self-contained infrastructure: external deps only, no `@civitai/*` deps. Higher-level packages (e.g. a future `civitai-moderator-common`) may depend on multiple base packages — but base packages stay independent. If two base packages need shared constants, they own their own copy or expose them via a subpath (e.g. `@civitai/redis/keys`).
|
||||
|
||||
**Base packages are infrastructure only.** Civitai-specific domain constants (bit-flag interpretations like `browsingLevel.constants.ts`, identifier formats like `air.ts`, bitwise helpers like `flags.ts`) stay in the main app for now. They're not infrastructure — they're domain conventions. We deferred extracting them; the satellite app can re-evaluate when it actually needs them.
|
||||
|
||||
## Directory layout
|
||||
|
||||
```text
|
||||
model-share/
|
||||
├── .browser/ # unchanged
|
||||
├── .claude/ # unchanged
|
||||
├── .devcontainer/
|
||||
├── .github/ # CI workflows — minor edits (pnpm install at root; typecheck adds `-r`)
|
||||
├── .husky/
|
||||
├── .ladle/
|
||||
├── .vscode/
|
||||
├── CLAUDE.md
|
||||
├── Dockerfile # may need adjustment for workspace install
|
||||
├── Makefile
|
||||
├── README.md
|
||||
├── docker-compose.base.yml
|
||||
├── docker-compose.yml
|
||||
│
|
||||
├── package.json # MAIN APP deps still live here (Next, Mantine, etc.)
|
||||
│ # Adds workspace dep references: "@civitai/db": "workspace:*", etc.
|
||||
│ # Drops: @prisma/client, prisma, pg, ioredis, @clickhouse/client,
|
||||
│ # @axiomhq/axiom-node, @opentelemetry/* (moved into packages)
|
||||
├── pnpm-workspace.yaml # NEW — defines packages: '.', 'packages/*', 'apps/*'
|
||||
├── pnpm-lock.yaml # one lockfile for the whole workspace
|
||||
├── tsconfig.json # unchanged (still rooted at ./src for the main app)
|
||||
├── next.config.mjs # adds transpilePackages: ['@civitai/*']
|
||||
├── tailwind.config.ts # unchanged
|
||||
├── postcss.config.cjs # unchanged
|
||||
├── eslint-local-rules.js # unchanged
|
||||
│
|
||||
├── src/ # MAIN APP — entirely unchanged file layout
|
||||
│ ├── app/
|
||||
│ ├── components/
|
||||
│ ├── env/ # unchanged — main app's own env validation
|
||||
│ ├── hooks/
|
||||
│ ├── instrumentation.node.ts # SHRUNK — now calls bootstrapOtel({ serviceName: 'civitai-app' })
|
||||
│ ├── instrumentation.ts # unchanged
|
||||
│ ├── libs/
|
||||
│ ├── middleware.ts
|
||||
│ ├── pages/
|
||||
│ ├── providers/
|
||||
│ ├── server/
|
||||
│ │ ├── db/
|
||||
│ │ │ ├── client.ts # SHIM — re-exports from @civitai/db with main-app env wiring
|
||||
│ │ │ ├── db-helpers.ts # SHIM — re-exports from @civitai/db
|
||||
│ │ │ ├── pgDb.ts # SHIM
|
||||
│ │ │ ├── notifDb.ts # SHIM
|
||||
│ │ │ ├── datapacketDb.ts # SHIM
|
||||
│ │ │ └── db-lag-helpers.ts # SHIM
|
||||
│ │ ├── redis/
|
||||
│ │ │ ├── client.ts # SHIM — re-exports from @civitai/redis
|
||||
│ │ │ ├── caches.ts # SHIM
|
||||
│ │ │ ├── queues.ts # SHIM
|
||||
│ │ │ ├── entity-metric.redis.ts # SHIM
|
||||
│ │ │ └── ... (rest are shims)
|
||||
│ │ ├── clickhouse/
|
||||
│ │ │ └── client.ts # SHIM — re-exports from @civitai/clickhouse
|
||||
│ │ ├── logging/
|
||||
│ │ │ └── client.ts # SHIM — re-exports from @civitai/axiom
|
||||
│ │ ├── services/ # unchanged (still imports via ~/server/db/client, etc.)
|
||||
│ │ ├── routers/ # unchanged
|
||||
│ │ ├── schema/ # unchanged (zod schemas stay here for now)
|
||||
│ │ └── ... (rest unchanged)
|
||||
│ ├── shared/
|
||||
│ │ ├── constants/ # unchanged — domain constants stay in main app
|
||||
│ │ │ ├── browsingLevel.constants.ts # unchanged (DB-encoded NSFW bits)
|
||||
│ │ │ ├── model-version-flags.constants.ts # unchanged
|
||||
│ │ │ ├── user-flags.constants.ts # unchanged
|
||||
│ │ │ └── ...
|
||||
│ │ ├── utils/
|
||||
│ │ │ ├── air.ts # unchanged (AIR identifier parser)
|
||||
│ │ │ ├── flags.ts # unchanged (bitwise helpers)
|
||||
│ │ │ └── prisma/
|
||||
│ │ │ ├── enums.ts # SHIM — re-exports from @civitai/db (Prisma-generated)
|
||||
│ │ │ └── models.ts # SHIM — re-exports from @civitai/db
|
||||
│ │ └── ... (data-graph, tiptap, etc. unchanged)
|
||||
│ ├── store/
|
||||
│ ├── styles/
|
||||
│ ├── types/
|
||||
│ ├── utils/
|
||||
│ │ └── otel-helpers.ts # SHIM — re-exports from @civitai/telemetry
|
||||
│ └── workers/
|
||||
│
|
||||
├── packages/ # NEW — all shared packages live here
|
||||
│ │
|
||||
│ ├── civitai-db/ # @civitai/db — everything Postgres: schema, generated
|
||||
│ │ │ client, migrations, pools, helpers. A future
|
||||
│ │ │ second DB schema gets its own package (e.g. myapp-db).
|
||||
│ │ ├── package.json # deps: pg, @prisma/client, prisma (CLI), prom-client
|
||||
│ │ ├── tsconfig.json
|
||||
│ │ ├── prisma/
|
||||
│ │ │ ├── schema.full.prisma # MOVED from root /prisma/
|
||||
│ │ │ ├── schema.prisma # auto-generated slim
|
||||
│ │ │ ├── migrations/ # MOVED — full migration history
|
||||
│ │ │ ├── programmability/ # MOVED — views, functions, etc.
|
||||
│ │ │ └── seed.ts # MOVED
|
||||
│ │ ├── generated/
|
||||
│ │ │ └── client/ # output of `prisma generate` — apps import from here
|
||||
│ │ └── src/
|
||||
│ │ ├── index.ts # exports createDbClients(config)
|
||||
│ │ ├── client.ts # MOVED from src/server/db/client.ts (Prisma wrapper)
|
||||
│ │ ├── db-helpers.ts # MOVED from src/server/db/db-helpers.ts (553 lines)
|
||||
│ │ ├── pgDb.ts # MOVED
|
||||
│ │ ├── notifDb.ts # MOVED
|
||||
│ │ ├── datapacketDb.ts # MOVED
|
||||
│ │ ├── db-lag-helpers.ts # MOVED
|
||||
│ │ ├── enums.ts # MOVED from src/shared/utils/prisma/enums.ts (Prisma-generated)
|
||||
│ │ └── models.ts # MOVED from src/shared/utils/prisma/models.ts (Prisma-generated)
|
||||
│ │
|
||||
│ ├── civitai-redis/ # @civitai/redis
|
||||
│ │ ├── package.json # deps: redis (or ioredis) — no other base packages
|
||||
│ │ ├── tsconfig.json
|
||||
│ │ └── src/
|
||||
│ │ ├── index.ts # exports createRedisClients(config)
|
||||
│ │ ├── keys.ts # FUTURE — key constants & TTLs (exported via
|
||||
│ │ │ @civitai/redis/keys subpath so consumers can
|
||||
│ │ │ import just the keys without instantiating clients)
|
||||
│ │ ├── client.ts # MOVED from src/server/redis/client.ts (1,105 lines)
|
||||
│ │ ├── caches.ts # MOVED (~1,571 lines — keys stay inside this package)
|
||||
│ │ ├── queues.ts # MOVED
|
||||
│ │ ├── entity-metric.redis.ts # MOVED
|
||||
│ │ ├── entity-metric-populate.ts # MOVED
|
||||
│ │ ├── resource-data.redis.ts # MOVED
|
||||
│ │ └── fail-open-log.ts # MOVED (refactored to accept a logger function via config)
|
||||
│ │
|
||||
│ ├── civitai-clickhouse/ # @civitai/clickhouse
|
||||
│ │ ├── package.json # deps: @clickhouse/client
|
||||
│ │ ├── tsconfig.json
|
||||
│ │ └── src/
|
||||
│ │ ├── index.ts # exports createClickhouseClient(config)
|
||||
│ │ └── client.ts # MOVED from src/server/clickhouse/client.ts (733 lines)
|
||||
│ │
|
||||
│ ├── civitai-axiom/ # @civitai/axiom
|
||||
│ │ ├── package.json # deps: @axiomhq/axiom-node
|
||||
│ │ ├── tsconfig.json
|
||||
│ │ └── src/
|
||||
│ │ ├── index.ts # exports createAxiomLogger(config) — { logToAxiom, safeError }
|
||||
│ │ └── client.ts # MOVED from src/server/logging/client.ts (58 lines)
|
||||
│ │
|
||||
│ └── civitai-telemetry/ # @civitai/telemetry
|
||||
│ ├── package.json # deps: @opentelemetry/*
|
||||
│ ├── tsconfig.json
|
||||
│ └── src/
|
||||
│ ├── index.ts # exports withSpan + helpers (browser-safe subset)
|
||||
│ ├── node.ts # exports bootstrapOtel(config) for instrumentation.node.ts
|
||||
│ ├── otel-helpers.ts # MOVED from src/utils/otel-helpers.ts
|
||||
│ └── prom.ts # MOVED from src/server/prom/client.ts (helpers; per-app registration stays in app)
|
||||
│
|
||||
├── apps/ # NEW — empty for now; populated when moderator app is built
|
||||
│
|
||||
├── event-engine-common/ # EXISTING submodule — untouched (stays a submodule for now)
|
||||
├── designs/ # unchanged
|
||||
├── docs/ # unchanged
|
||||
├── containers/ # unchanged
|
||||
├── analyze/
|
||||
└── (other unchanged files: .env, .env-example, .dockerignore, etc.)
|
||||
```
|
||||
|
||||
## Key things to notice
|
||||
|
||||
- **`src/` looks almost identical to today.** Every file that was moved into a package leaves a one-line shim behind at its old path. The diff to call sites in the main app is zero.
|
||||
- **`prisma/` at root is gone.** It's entirely inside `packages/civitai-schema-common/prisma/`. The `db:generate`, `db:migrate`, etc. scripts now point at the package path.
|
||||
- **No `node_modules/.prisma/client`.** The generated Prisma client lives at `packages/civitai-schema-common/generated/client/`. Both main app and any future apps import from `@civitai/schema-common/client`.
|
||||
- **Per-app `instrumentation.node.ts` shrinks from 97 lines to ~3** — just calls `bootstrapOtel({ serviceName: 'civitai-app' })`. The OTEL SDK setup moves into the telemetry package.
|
||||
- **`apps/` is empty until the moderator app is built.** Phases 0–6 are foundation-only.
|
||||
- **`event-engine-common/` stays a git submodule.** Converting it to a workspace package is a separate decision that doesn't need to block this work.
|
||||
|
||||
## After Phase 7 (the optional final move)
|
||||
|
||||
Everything currently in `src/`, `next.config.mjs`, etc. moves to `apps/main/`. The root becomes a pure workspace shell:
|
||||
|
||||
```text
|
||||
model-share/
|
||||
├── package.json # workspace root only — no app deps
|
||||
├── pnpm-workspace.yaml
|
||||
├── pnpm-lock.yaml
|
||||
├── tsconfig.base.json # shared TS settings
|
||||
├── apps/
|
||||
│ ├── main/ # everything that was at root
|
||||
│ │ ├── package.json
|
||||
│ │ ├── next.config.mjs
|
||||
│ │ ├── tsconfig.json
|
||||
│ │ └── src/
|
||||
│ └── moderator/ # the satellite app
|
||||
├── packages/ # same as Phase 6 snapshot
|
||||
├── docs/
|
||||
├── containers/
|
||||
├── Makefile
|
||||
└── (config files)
|
||||
```
|
||||
|
||||
Phase 7 is deferred until the satellite app exists and proves the workspace setup is stable.
|
||||
@@ -0,0 +1,716 @@
|
||||
# Monorepo — Base Package Adaptation Plan
|
||||
|
||||
**Status:** Planning only. The base-package file *moves* are staged but **not yet committed**;
|
||||
none of the content rewrites below have been applied. This document is the concrete
|
||||
implementation spec for turning the five staged base packages into true,
|
||||
infrastructure-only packages that import **external npm dependencies only**.
|
||||
|
||||
Read first: [`monorepo-bootstrap-handoff.md`](./monorepo-bootstrap-handoff.md) and
|
||||
[`monorepo-conversion-plan.md`](./monorepo-conversion-plan.md). This doc supersedes the
|
||||
"Phase 1–5" sketch in those with worked, code-level detail.
|
||||
|
||||
---
|
||||
|
||||
## 1. The rule we are enforcing
|
||||
|
||||
A base package (`@civitai/db`, `@civitai/redis`, `@civitai/clickhouse`, `@civitai/axiom`,
|
||||
`@civitai/telemetry`) may import **external npm packages only**. It must not import:
|
||||
|
||||
- another base package (no `db → axiom`, no `telemetry → db`),
|
||||
- an app service (`~/server/flipt`, `~/server/auth`, …),
|
||||
- app config (`~/env/server`, `~/env/other`),
|
||||
- app utilities (`~/utils/*`, `~/server/utils/*`),
|
||||
- app domain types/schemas (`~/server/schema/*`, `~/server/common/*`, `~/shared/*`).
|
||||
|
||||
Anything that needs one of those is a **consumer** of infrastructure, not infrastructure,
|
||||
and either (a) gets its dependency **injected**, or (b) **moves back** to the main app.
|
||||
|
||||
**One exception — the contract layer.** `@civitai/db-schema` (Prisma schema + generated
|
||||
types, §4.0) is a **leaf artifact** with no runtime. Infra packages may depend **downward** on
|
||||
it (e.g. `@civitai/db` imports the generated Prisma client from `@civitai/db-schema`), exactly
|
||||
as they depend on any npm package. The no-sibling rule still holds: it only forbids one infra
|
||||
package importing *another infra package's runtime*. A pure types/schema package is a lower
|
||||
layer, not a sibling.
|
||||
|
||||
## 2. The pattern: factory + injected deps + app-side shim
|
||||
|
||||
Every package stops exporting eager module-level singletons and instead exports a
|
||||
**factory**: `createXClients(config)`. The `config` carries two kinds of things:
|
||||
|
||||
1. **Plain values** the package reads from `env` today (URLs, timeouts, booleans).
|
||||
2. **Injected functions** for concerns the package may not own:
|
||||
- a `log` function (replaces `~/utils/logging`'s `createLogger`),
|
||||
- cross-cutting callbacks (`onSlowQuery → axiom`, `isEnhancedFailoverEnabled → flipt`, …).
|
||||
|
||||
Each **app** keeps a thin **shim** at the original import path
|
||||
(`src/server/db/client.ts`, `src/server/redis/client.ts`, …). The shim:
|
||||
|
||||
- calls the factory with that app's real `env`, real logger, real flipt/axiom wiring,
|
||||
- **owns the dev/HMR `global.*` singleton caching** (per decision: globals live where the
|
||||
factory is called, never inside the package),
|
||||
- **re-exports the same names** (`dbRead`, `dbWrite`, `redis`, `sysRedis`, …) so every
|
||||
existing call site keeps working unchanged.
|
||||
|
||||
Because the shims are app code, they may freely compose multiple base packages (e.g. the
|
||||
db shim may import `logToAxiom` from the axiom shim). The **package-level** "no sibling
|
||||
imports" rule is what stays inviolate.
|
||||
|
||||
### 2.1 The injected logger
|
||||
|
||||
Each factory takes an **optional** `log` parameter, defaulting to a no-op. The package never
|
||||
logs on its own; when an app wants visibility it injects a logger (the main app injects one
|
||||
into every factory). To avoid a shared logger package (which would itself be a cross-package
|
||||
import), each package declares its own **structural** logger type; the app's single logger
|
||||
duck-types into all of them:
|
||||
|
||||
```ts
|
||||
// declared independently inside each package — structural, so one app logger satisfies all
|
||||
export type LogFn = (message: string, ...args: unknown[]) => void;
|
||||
const noop: LogFn = () => {};
|
||||
// in the factory: const log = config.log ?? noop;
|
||||
```
|
||||
|
||||
The main app builds one `LogFn` per domain from the existing `createLogger(name, color)` and
|
||||
passes it in. Structured telemetry that currently goes to Axiom (db slow queries, clickhouse
|
||||
insert errors) is a **separate** injected callback, not a log line — see each package below.
|
||||
|
||||
### 2.2 Environment variables — packages never import an env module
|
||||
|
||||
A base package **must not import any `env`** — not the app's `~/env/server`, not a shared one.
|
||||
Factories accept **plain typed values** (`isProd`, URLs, timeouts). The app reads its own env
|
||||
and passes those values in. This keeps packages reuse-agnostic: a package never assumes *how*
|
||||
its config is sourced.
|
||||
|
||||
> The current monolithic `~/env/server` exists only because the project started from a starter
|
||||
> that validated all env vars against one schema. We are **not** carrying that coupling into the
|
||||
> packages. If we later want per-domain env→schema validation back, it lives as a **scoped env
|
||||
> file inside each package** (the package ships its own zod/`@t3-oss/env` schema + a
|
||||
> `configFromEnv(process.env)` helper). That stays **opt-in** — the core factory still takes
|
||||
> plain values, so an app can validate however it likes (or not at all).
|
||||
|
||||
**No `isBuild` in factories.** The old `if (!env.IS_BUILD)` guard (skip opening connections
|
||||
during `next build`) is an *app/runtime* concern — `isBuild` ≠ `isProd` (a production build
|
||||
has `isProd === true` but must still not connect). Since the **shim** owns instantiation, the
|
||||
shim keeps the build guard; the package factory only takes `isProd` where behavior genuinely
|
||||
differs (e.g. slow-query → console vs Axiom). See §4.2.
|
||||
|
||||
---
|
||||
|
||||
## 3. Boundary-import inventory (complete)
|
||||
|
||||
Every `~/…` import found in the five packages, and its resolution:
|
||||
|
||||
| Import | Package(s) | Resolution |
|
||||
|--------|-----------|-----------|
|
||||
| `env` (`~/env/server`) | all | **never imported** — factory takes plain config values; app sources them (§2.2) |
|
||||
| `isProd` (`~/env/other`) | db, redis, clickhouse, axiom | **config boolean** `isProd` (the only env-flag; no `isBuild`) |
|
||||
| `logToAxiom` (`~/server/logging/client`) | db, clickhouse | **inject** `onSlowQuery` / `onError`; axiom stays a sibling, never imported |
|
||||
| `isFlipt` / `FLIPT_FEATURE_FLAGS` (`~/server/flipt/client`) | redis | **inject** `isEnhancedFailoverEnabled` |
|
||||
| `createLogger` (`~/utils/logging`) | db, redis, clickhouse | **inject** optional `log?: LogFn` (default no-op) |
|
||||
| `slugit` (`~/utils/string-helpers`) | redis | **inline** (pure 3-line helper) |
|
||||
| `limitConcurrency` (`~/server/utils/concurrency-helpers`) | db (`db-helpers.ts`) | **vendor or inline** (see §4.6) |
|
||||
| `sleep` (`~/utils/errorHandling`) | clickhouse | moves out with the Tracker (or inline) |
|
||||
| `getServerAuthSession` (`~/server/auth/...`) | clickhouse | **moves back** with the Tracker |
|
||||
| domain type imports (`~/server/common/enums`, `~/server/jobs/...`, `~/server/schema/...`, `~/shared/...`) | clickhouse | **moves back** with the Tracker |
|
||||
| `pgDb`/`notifDb`/`datapacketDb` reads (`~/server/db/...`) | telemetry | pool-gauge block **moves back** to the app |
|
||||
| `dbWrite` (`~/server/db/client`) | db (`db-helpers.ts`) | in-package cycle — **pass `dbWrite` as a param** (§4.5) |
|
||||
|
||||
---
|
||||
|
||||
## 4. The contract layer + `@civitai/db` (heaviest)
|
||||
|
||||
Today's staged `@civitai/db` actually bundles **two separable concerns** — the schema/types
|
||||
*contract* and the Prisma-client *runtime*. The [`civitai-advertising`](file:///C:/Work/civitai-advertising)
|
||||
project drives **Kysely** off a Prisma-generated schema (two generators on one
|
||||
`schema.prisma`: `prisma-client` → `./generated`, `prisma-kysely` → Kysely `DB` types; runtime
|
||||
is `new Kysely<DB>()` over a raw `pg.Pool`, never the Prisma client). To let a future app pick
|
||||
Kysely without dragging in the Prisma-client runtime, the contract is split into its own
|
||||
package.
|
||||
|
||||
### 4.0 `@civitai/db-schema` — source-of-truth contract (LEAF package)
|
||||
|
||||
Pure artifact: schema + migrations + **generated types**. No runtime, no `env`, no `@civitai/*`
|
||||
dependency. Both `@civitai/db` (Prisma runtime) and any future Kysely app/package depend
|
||||
**downward** on it — this is allowed (it's a lower layer, like depending on `@prisma/client`);
|
||||
it does **not** violate the no-sibling-imports rule, which only forbids one infra package
|
||||
importing another infra package.
|
||||
|
||||
**Layout** (`packages/civitai-db-schema/`):
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `prisma/schema.full.prisma` | source of truth — datasource + **both** generators |
|
||||
| `prisma/migrations/`, `prisma/views/`, programmability | migration history (applied **manually**, per CLAUDE.md) |
|
||||
| `generated/client/` | `prisma-client` generator output |
|
||||
| `src/kysely/types.ts`, `src/kysely/enums.ts` | `prisma-kysely` generator output (**baked in now**) |
|
||||
| `src/enums.ts`, `src/models.ts` | Prisma-generated enums/models (moved from `@civitai/db`) |
|
||||
|
||||
**Both generators off one schema:**
|
||||
|
||||
```prisma
|
||||
generator client {
|
||||
provider = "prisma-client"
|
||||
output = "../generated/client"
|
||||
previewFeatures = ["views"]
|
||||
}
|
||||
generator kysely {
|
||||
provider = "prisma-kysely"
|
||||
output = "../src/kysely"
|
||||
fileName = "types.ts"
|
||||
enumFileName = "enums.ts"
|
||||
}
|
||||
datasource db { provider = "postgresql"; url = env("DATABASE_URL") }
|
||||
```
|
||||
|
||||
Owns the `db:generate` / `db:migrate*` scripts — their paths move from `prisma/...` to
|
||||
`packages/civitai-db-schema/prisma/...`. **Migrations stay manual** — this move doesn't change
|
||||
the apply path; `scripts/prisma-migrate-with-views-workaround.mjs` references update to the new
|
||||
path. Exports: generated Prisma client + types, `enums`, `models`, and the Kysely `DB` type via
|
||||
subpath `@civitai/db-schema/kysely`.
|
||||
|
||||
> Adding `prisma-kysely` + creating this package is the **only content edit** authorized here
|
||||
> (schema gets a second generator block + `prisma-kysely` dev-dep); the actual *file relocation*
|
||||
> of `prisma/`, `enums.ts`, `models.ts` into `packages/civitai-db-schema/` is a pure move.
|
||||
|
||||
### 4.1 `@civitai/db` — Prisma-client runtime (depends on `@civitai/db-schema`)
|
||||
|
||||
**Package** (`packages/civitai-db/src/`) — reusable Postgres machinery, no `env`, no globals;
|
||||
imports the generated client + types from `@civitai/db-schema`:
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `client.ts` | `createPrismaClients(config)` — Prisma read/write factory |
|
||||
| `db-helpers.ts` | `getClient(config)` pg-`Pool` factory + pure SQL utils + param-bound stateful helpers |
|
||||
|
||||
(`enums.ts`, `models.ts`, `prisma/` have moved **down** to `@civitai/db-schema`.)
|
||||
|
||||
**App shims** (`src/server/db/`) — instantiate with `env`, own the `global.*` caches,
|
||||
re-export the existing names:
|
||||
|
||||
| Shim | Calls | Re-exports |
|
||||
|------|-------|-----------|
|
||||
| `client.ts` | `createPrismaClients` | `dbRead`, `dbWrite` |
|
||||
| `pgDb.ts` | `getClient` ×3 | `pgDbWrite`, `pgDbRead`, `pgDbReadLong` |
|
||||
| `notifDb.ts` | `getClient` ×2 | `notifDbWrite`, `notifDbRead` |
|
||||
| `datapacketDb.ts` | `getClient` ×1 | `datapacketDbRead` |
|
||||
| `db-helpers.ts` | re-export pkg utils; bind `dbWrite` into stateful helpers | `getCurrentLSN`, `checkNotUpToDate`, `dbKV`, all pure utils |
|
||||
|
||||
> Note: `pgDb.ts`, `notifDb.ts`, `datapacketDb.ts` currently sit **inside** the package
|
||||
> (we moved them there). Under this plan their **singleton** halves move back to
|
||||
> `src/server/db/` as shims; only `getClient` (their factory) stays in the package. This is
|
||||
> a follow-up content commit, done **after** the pure-move commit lands.
|
||||
|
||||
### 4.2 Prisma factory (`client.ts`)
|
||||
|
||||
**Before** — eager singleton, reads `env`, imports axiom
|
||||
([`client.ts:31`](../packages/civitai-db/src/client.ts#L31)):
|
||||
|
||||
```ts
|
||||
import { env } from '~/env/server';
|
||||
import { logToAxiom } from '~/server/logging/client';
|
||||
...
|
||||
export let dbRead: PrismaClient;
|
||||
export let dbWrite: PrismaClient;
|
||||
if (!env.IS_BUILD) { /* isProd ? … : global.globalDbWrite ??= … */ }
|
||||
```
|
||||
|
||||
**After** — package side, factory only:
|
||||
|
||||
```ts
|
||||
// packages/civitai-db/src/client.ts
|
||||
// Prisma client + types come from the contract package, never @prisma/client directly:
|
||||
import type { Prisma } from '@civitai/db-schema';
|
||||
import { PrismaClient } from '@civitai/db-schema';
|
||||
|
||||
export type LogFn = (message: string, ...args: unknown[]) => void;
|
||||
|
||||
export type PrismaClientsConfig = {
|
||||
databaseUrl: string;
|
||||
replicaUrl: string;
|
||||
isProd: boolean; // the only env-flag the factory needs (no isBuild)
|
||||
logging: string[]; // env.LOGGING
|
||||
log?: LogFn; // optional injected logger (defaults to no-op)
|
||||
/** structured slow-query telemetry (the old logToAxiom call). Optional. */
|
||||
onSlowQuery?: (e: { query: string; duration: number; target: 'read' | 'write' }) => void;
|
||||
};
|
||||
|
||||
export type PrismaClients = { dbRead: PrismaClient; dbWrite: PrismaClient };
|
||||
|
||||
export function createPrismaClients(config: PrismaClientsConfig): PrismaClients {
|
||||
const singleClient = config.replicaUrl === config.databaseUrl;
|
||||
|
||||
const logFor = (target: 'read' | 'write') =>
|
||||
(e: { query: string; params: string; duration: number }) => {
|
||||
if (e.duration < 2000) return;
|
||||
const query = substituteParams(e.query, e.params); // existing $1-substitution logic
|
||||
if (!config.isProd) console.log(query);
|
||||
else config.onSlowQuery?.({ query, duration: e.duration, target }); // ← injected, no axiom import
|
||||
};
|
||||
|
||||
const createOne = ({ readonly }: { readonly: boolean }): PrismaClient => {
|
||||
const log = buildPrismaLogDefs(config.logging); // existing log-def logic
|
||||
const url = readonly ? config.replicaUrl : config.databaseUrl;
|
||||
const prisma = new PrismaClient({ log, datasources: { db: { url } } });
|
||||
// prisma-showparams / prisma-slow-* wiring unchanged, gated on config.logging
|
||||
return prisma;
|
||||
};
|
||||
|
||||
// no isBuild here — the shim decides whether to call the factory at all during `next build`
|
||||
const dbWrite = createOne({ readonly: false });
|
||||
const dbRead = singleClient ? dbWrite : createOne({ readonly: true });
|
||||
// slow-query $on wiring uses logFor(...) + config.logging, exactly as today
|
||||
return { dbRead, dbWrite };
|
||||
}
|
||||
```
|
||||
|
||||
**After** — app shim owns env + globals + axiom wiring:
|
||||
|
||||
```ts
|
||||
// src/server/db/client.ts (app shim, original path preserved)
|
||||
import { createPrismaClients, type PrismaClients } from '@civitai/db';
|
||||
import { isProd } from '~/env/other';
|
||||
import { env } from '~/env/server';
|
||||
import { logToAxiom } from '~/server/logging/client';
|
||||
import { createLogger } from '~/utils/logging';
|
||||
|
||||
const log = createLogger('prisma', 'green');
|
||||
|
||||
declare global {
|
||||
// eslint-disable-next-line no-var
|
||||
var __civitaiPrisma: PrismaClients | undefined;
|
||||
}
|
||||
|
||||
// build guard lives in the shim (not the factory): don't open connections during `next build`
|
||||
const clients = env.IS_BUILD
|
||||
? ({ dbRead: undefined as never, dbWrite: undefined as never })
|
||||
: (global.__civitaiPrisma ??= createPrismaClients({
|
||||
databaseUrl: env.DATABASE_URL,
|
||||
replicaUrl: env.DATABASE_REPLICA_URL,
|
||||
isProd,
|
||||
logging: env.LOGGING,
|
||||
log,
|
||||
onSlowQuery: ({ query, duration, target }) => logToAxiom({ query, duration, target }, 'db-logs'),
|
||||
}));
|
||||
|
||||
export const dbRead = clients.dbRead;
|
||||
export const dbWrite = clients.dbWrite;
|
||||
```
|
||||
|
||||
> In production `??=` still assigns once per process; in dev it reuses the HMR global —
|
||||
> identical behavior to today's `global.globalDbWrite` block, just relocated to the shim.
|
||||
|
||||
### 4.3 pg `Pool` factory (`getClient` in `db-helpers.ts`)
|
||||
|
||||
`getClient` ([`db-helpers.ts:85`](../packages/civitai-db/src/db-helpers.ts#L85)) currently reads
|
||||
`env` for URLs, timeouts, pool sizes, `PODNAME`, `IS_DATAPACKET`, SSL. Convert it to take a
|
||||
config object. The `types.setTypeParser(TIMESTAMP, …)` side effect (currently top-of-file in
|
||||
each singleton) moves **into the pool factory** so it runs at pool creation.
|
||||
|
||||
```ts
|
||||
// packages/civitai-db/src/db-helpers.ts (package side)
|
||||
export type PgInstance =
|
||||
| 'primary' | 'primaryRead' | 'primaryReadLong'
|
||||
| 'notification' | 'notificationRead' | 'datapacketRead';
|
||||
|
||||
export type PgClientConfig = {
|
||||
log?: LogFn; // optional, default no-op
|
||||
isDatapacket: boolean;
|
||||
podName?: string;
|
||||
ssl: boolean; // env.DATABASE_SSL !== false
|
||||
urls: Record<PgInstance, string>; // resolved URLs (with fallbacks) from the app
|
||||
timeouts: {
|
||||
connection: number; read?: number; write?: number; poolIdle: number;
|
||||
};
|
||||
poolMax: number; notificationPoolMax?: number;
|
||||
};
|
||||
|
||||
export function getClient(instance: PgInstance, config: PgClientConfig): AugmentedPool {
|
||||
// identical pool construction, but every `env.X` becomes `config.X`
|
||||
// pgPoolAcquireHistogram (raw prom-client) stays — prom-client is an external dep, allowed
|
||||
}
|
||||
```
|
||||
|
||||
App shim instantiates the singletons and owns the globals (one shim file per existing path):
|
||||
|
||||
```ts
|
||||
// src/server/db/pgDb.ts (app shim)
|
||||
import { getClient, type AugmentedPool, type PgClientConfig } from '@civitai/db';
|
||||
import { isProd } from '~/env/other';
|
||||
import { env } from '~/env/server';
|
||||
import { createLogger } from '~/utils/logging';
|
||||
|
||||
const cfg: PgClientConfig = {
|
||||
log: createLogger('pgDb', 'blue'),
|
||||
isDatapacket: env.IS_DATAPACKET,
|
||||
podName: env.PODNAME,
|
||||
ssl: env.DATABASE_SSL !== false,
|
||||
urls: {
|
||||
primary: env.DATABASE_URL,
|
||||
primaryRead: env.DATABASE_REPLICA_URL ?? env.DATABASE_URL,
|
||||
primaryReadLong: env.DATABASE_REPLICA_LONG_URL ?? env.DATABASE_URL,
|
||||
notification: env.NOTIFICATION_DB_URL,
|
||||
notificationRead: env.NOTIFICATION_DB_REPLICA_URL ?? env.NOTIFICATION_DB_URL,
|
||||
datapacketRead: env.DATAPACKET_DATABASE_RO_URL ?? env.DATABASE_URL,
|
||||
},
|
||||
timeouts: {
|
||||
connection: env.DATABASE_CONNECTION_TIMEOUT,
|
||||
read: env.DATABASE_READ_TIMEOUT, write: env.DATABASE_WRITE_TIMEOUT,
|
||||
poolIdle: env.DATABASE_POOL_IDLE_TIMEOUT,
|
||||
},
|
||||
poolMax: env.DATABASE_POOL_MAX, notificationPoolMax: env.NOTIFICATION_POOL_MAX,
|
||||
};
|
||||
|
||||
declare global {
|
||||
// eslint-disable-next-line no-var
|
||||
var globalPgWrite: AugmentedPool | undefined;
|
||||
// eslint-disable-next-line no-var
|
||||
var globalPgRead: AugmentedPool | undefined;
|
||||
// eslint-disable-next-line no-var
|
||||
var globalPgReadLong: AugmentedPool | undefined;
|
||||
}
|
||||
|
||||
const single = (env.DATABASE_REPLICA_URL ?? env.DATABASE_URL) === env.DATABASE_URL;
|
||||
export const pgDbWrite = (global.globalPgWrite ??= getClient('primary', cfg));
|
||||
export const pgDbRead = (global.globalPgRead ??= single ? pgDbWrite : getClient('primaryRead', cfg));
|
||||
export const pgDbReadLong = (global.globalPgReadLong ??= single ? pgDbWrite : getClient('primaryReadLong', cfg));
|
||||
```
|
||||
|
||||
`notifDb.ts` and `datapacketDb.ts` shims follow the same shape (their own globals + instances).
|
||||
|
||||
### 4.4 Pure utilities — re-export untouched
|
||||
|
||||
These functions in `db-helpers.ts` have **no** `env`/client dependency and stay in the
|
||||
package as plain exports: `queryWithTimeout`, `dataProcessor`, `batchProcessor`,
|
||||
`templateHandler`, `parameterizedTemplateHandler`, `combineSqlWithParams`, `getExplainSql`,
|
||||
`jsonbArrayFrom`, `formatSqlType`. The app `db-helpers.ts` shim just `export * from '@civitai/db'`
|
||||
for these.
|
||||
|
||||
### 4.5 Breaking the in-package cycle (`db-helpers → client`)
|
||||
|
||||
`getCurrentLSN`, `checkNotUpToDate`, and `dbKV`
|
||||
([`db-helpers.ts:343-553`](../packages/civitai-db/src/db-helpers.ts#L343-L553)) call `dbWrite`.
|
||||
Today that's `import { dbWrite } from '~/server/db/client'` — an in-package cycle once both
|
||||
files live in `@civitai/db`. **Fix: pass `dbWrite` in.**
|
||||
|
||||
```ts
|
||||
// package side — stateless, takes the client
|
||||
export async function getCurrentLSN(dbWrite: PrismaClient): Promise<string> { /* … */ }
|
||||
export function makeDbKV(dbWrite: PrismaClient) {
|
||||
return { get: async <T>(k: string, d?: T) => { /* … */ }, set: async <T>(k: string, v: T) => { /* … */ } };
|
||||
}
|
||||
```
|
||||
|
||||
```ts
|
||||
// src/server/db/db-helpers.ts (app shim) — bind the app's dbWrite once
|
||||
import { dbWrite } from '~/server/db/client';
|
||||
import * as pkg from '@civitai/db';
|
||||
export * from '@civitai/db'; // pure utils + getClient
|
||||
export const getCurrentLSN = () => pkg.getCurrentLSN(dbWrite);
|
||||
export const checkNotUpToDate = (lsn: string) => pkg.checkNotUpToDate(dbWrite, lsn);
|
||||
export const dbKV = pkg.makeDbKV(dbWrite);
|
||||
```
|
||||
|
||||
Call sites (`import { dbKV } from '~/server/db/db-helpers'`) are unchanged.
|
||||
|
||||
### 4.6 `limitConcurrency`
|
||||
|
||||
`dataProcessor`/`batchProcessor` use `limitConcurrency` from
|
||||
`~/server/utils/concurrency-helpers`. **Resolution:** verify that file is dependency-free
|
||||
(pure Promise scheduling); if so, **vendor a copy** into `packages/civitai-db/src/` (or a
|
||||
small `@civitai/db` internal util). If it has app deps, **inline** the single function. Do
|
||||
**not** import it from the app. _(Sub-task: confirm `concurrency-helpers.ts` purity.)_
|
||||
|
||||
---
|
||||
|
||||
## 5. `@civitai/redis` (keep `client.ts` only)
|
||||
|
||||
Package surface: `createRedisClients(config)` returning `{ redis, sysRedis }` plus the static
|
||||
`REDIS_KEYS` / `REDIS_SYS_KEYS` / `REDIS_SUB_KEYS` key definitions (these are pure constants,
|
||||
exported directly).
|
||||
|
||||
### 5.1 Inject the Flipt-gated failover policy
|
||||
|
||||
**Before** ([`client.ts:333`](../packages/civitai-redis/src/client.ts#L333)):
|
||||
|
||||
```ts
|
||||
import { FLIPT_FEATURE_FLAGS, isFlipt } from '~/server/flipt/client';
|
||||
import { slugit } from '~/utils/string-helpers';
|
||||
const enabled = await isFlipt(FLIPT_FEATURE_FLAGS.REDIS_CLUSTER_ENHANCED_FAILOVER, 'redis-cluster', fliptContext);
|
||||
```
|
||||
|
||||
**After** — package knows nothing about Flipt; the answer is injected:
|
||||
|
||||
```ts
|
||||
// packages/civitai-redis/src/client.ts
|
||||
export type RedisClientsConfig = {
|
||||
url: string; sysUrl: string; timeout: number;
|
||||
cluster: boolean; clusterNodes?: string; clusterRefreshInterval: number;
|
||||
nextAuthUrl?: string; fliptDeploymentId?: string; // failover-context inputs (were env.*)
|
||||
log?: LogFn; // optional; replaces createLogger
|
||||
/** app policy, injected. Defaults to OFF — package never names Flipt. */
|
||||
isEnhancedFailoverEnabled?: (ctx: Record<string, string>) => Promise<boolean>;
|
||||
};
|
||||
|
||||
// inlined — was ~/utils/string-helpers
|
||||
const slugit = (s: string) =>
|
||||
s.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '');
|
||||
|
||||
export function createRedisClients(config: RedisClientsConfig) {
|
||||
// …existing client/cluster construction, env.X → config.X…
|
||||
// in the cluster failover block:
|
||||
const enabled = (await config.isEnhancedFailoverEnabled?.(fliptContext)) ?? false;
|
||||
if (enabled) triggerTopologyRediscovery(baseClient, reason);
|
||||
return { redis, sysRedis };
|
||||
}
|
||||
|
||||
export const REDIS_KEYS = { /* … unchanged constant tree … */ } as const;
|
||||
export const REDIS_SYS_KEYS = { /* … */ } as const;
|
||||
export const REDIS_SUB_KEYS = { /* … */ } as const;
|
||||
```
|
||||
|
||||
**App shim** wires real Flipt + owns globals:
|
||||
|
||||
```ts
|
||||
// src/server/redis/client.ts (app shim)
|
||||
import { createRedisClients, REDIS_KEYS, REDIS_SYS_KEYS, REDIS_SUB_KEYS } from '@civitai/redis';
|
||||
import { env } from '~/env/server';
|
||||
import { createLogger } from '~/utils/logging';
|
||||
import { FLIPT_FEATURE_FLAGS, isFlipt } from '~/server/flipt/client';
|
||||
|
||||
declare global { /* eslint-disable-next-line no-var */ var __civitaiRedis: ReturnType<typeof createRedisClients> | undefined; }
|
||||
|
||||
// build guard in the shim: skip client creation during `next build`
|
||||
const clients = global.__civitaiRedis ??= env.IS_BUILD ? ({} as ReturnType<typeof createRedisClients>) : createRedisClients({
|
||||
url: env.REDIS_URL, sysUrl: env.REDIS_SYS_URL, timeout: env.REDIS_TIMEOUT,
|
||||
cluster: env.REDIS_CLUSTER, clusterNodes: env.REDIS_CLUSTER_NODES,
|
||||
clusterRefreshInterval: env.REDIS_CLUSTER_REFRESH_INTERVAL,
|
||||
nextAuthUrl: env.NEXTAUTH_URL, fliptDeploymentId: env.FLIPT_DEPLOYMENT_ID,
|
||||
log: createLogger('redis', 'green'),
|
||||
isEnhancedFailoverEnabled: (ctx) =>
|
||||
isFlipt(FLIPT_FEATURE_FLAGS.REDIS_CLUSTER_ENHANCED_FAILOVER, 'redis-cluster', ctx),
|
||||
});
|
||||
|
||||
export const { redis, sysRedis } = clients;
|
||||
export { REDIS_KEYS, REDIS_SYS_KEYS, REDIS_SUB_KEYS };
|
||||
// plus re-export the RedisKeyTemplate* types consumers import from here
|
||||
```
|
||||
|
||||
Everything that previously lived in `civitai-redis` (`caches.ts`, `queues.ts`,
|
||||
`resource-data.redis.ts`, `entity-metric.redis.ts`, `entity-metric-populate.ts`,
|
||||
`fail-open-log.ts`) is already moved back to `src/server/redis/` and imports the shim — no
|
||||
change needed there.
|
||||
|
||||
---
|
||||
|
||||
## 6. `@civitai/clickhouse` (split: base client stays, Tracker moves back)
|
||||
|
||||
`clickhouse/client.ts` mixes two concerns:
|
||||
|
||||
- **Infrastructure** — `createClient` + the `$query` / `$exec` template helpers. Needs only
|
||||
`CLICKHOUSE_HOST/USERNAME/PASSWORD` and an injected error hook.
|
||||
- **The Tracker** — request/session-bound event recording. Imports `getServerAuthSession`,
|
||||
`NextApiRequest`/`Response`, `Session`, `request-ip`, and app schemas/enums
|
||||
(`new-order.schema`, `entity-moderation`, `user.schema`, `browsingLevel.constants`). This
|
||||
is **main-app domain** and, per discussion, other apps won't use it the same way — at most
|
||||
they need to pass a `userId` to an insert.
|
||||
|
||||
### 6.1 Package = base client only
|
||||
|
||||
```ts
|
||||
// packages/civitai-clickhouse/src/client.ts
|
||||
import type { ClickHouseClient } from '@clickhouse/client';
|
||||
import { createClient } from '@clickhouse/client';
|
||||
|
||||
export type ClickhouseConfig = {
|
||||
host: string; username: string; password: string;
|
||||
isProd: boolean;
|
||||
log?: LogFn; // optional, default no-op
|
||||
/** insert/query error telemetry (was logToAxiom). Injected. */
|
||||
onError?: (data: Record<string, unknown>, datastream?: string) => void;
|
||||
};
|
||||
|
||||
export type CustomClickHouseClient = ClickHouseClient & {
|
||||
$query: <T extends object>(q: TemplateStringsArray | string, ...v: any[]) => Promise<T[]>;
|
||||
$exec: (q: TemplateStringsArray | string, ...v: any[]) => Promise<void>;
|
||||
};
|
||||
|
||||
export function createClickhouseClient(config: ClickhouseConfig): CustomClickHouseClient {
|
||||
// existing createClient + $query/$exec wiring; env.X → config.X; logToAxiom → config.onError
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 App shim + relocated Tracker
|
||||
|
||||
```ts
|
||||
// src/server/clickhouse/client.ts (app shim)
|
||||
import { createClickhouseClient } from '@civitai/clickhouse';
|
||||
import { isProd } from '~/env/other';
|
||||
import { env } from '~/env/server';
|
||||
import { createLogger } from '~/utils/logging';
|
||||
import { logToAxiom } from '~/server/logging/client';
|
||||
|
||||
declare global { /* … */ var globalClickhouse: ReturnType<typeof createClickhouseClient> | undefined; }
|
||||
|
||||
// build guard in the shim
|
||||
export const clickhouse = global.globalClickhouse ??= env.IS_BUILD
|
||||
? (undefined as never)
|
||||
: createClickhouseClient({
|
||||
host: env.CLICKHOUSE_HOST, username: env.CLICKHOUSE_USERNAME, password: env.CLICKHOUSE_PASSWORD,
|
||||
isProd, log: createLogger('clickhouse', 'blue'),
|
||||
onError: (data, ds) => logToAxiom(data, ds),
|
||||
});
|
||||
```
|
||||
|
||||
The `Tracker` class and all its request/session/schema imports move to a **new app file**
|
||||
`src/server/clickhouse/tracker.ts`, built on top of `clickhouse` from the shim. Its public
|
||||
import path is preserved for existing call sites. _(Sub-task: extract the exact Tracker
|
||||
surface from the current `client.ts`; confirm what other modules import from
|
||||
`~/server/clickhouse/client` so re-exports stay complete.)_
|
||||
|
||||
> Future multi-app note: the base client already supports arbitrary inserts; an app that only
|
||||
> needs "attach a userId" passes it in the row payload — no Tracker required.
|
||||
|
||||
---
|
||||
|
||||
## 7. `@civitai/axiom` (factory-ize the logger)
|
||||
|
||||
`axiom/client.ts` is already free of sibling/app-service imports, but it still reads `env`
|
||||
directly. Make it a factory for consistency and so other apps get their own datastream/pod
|
||||
config. `safeError` is pure — export it directly.
|
||||
|
||||
```ts
|
||||
// packages/civitai-axiom/src/client.ts
|
||||
import { Client } from '@axiomhq/axiom-node';
|
||||
|
||||
export type AxiomConfig = {
|
||||
token?: string; orgId?: string; datastream?: string;
|
||||
podName?: string; isProd: boolean;
|
||||
logErrorsToStdout: boolean; // process.env.LOG_ERRORS_TO_STDOUT === 'true'
|
||||
};
|
||||
|
||||
export function safeError(e: unknown): MixedObject | undefined { /* unchanged, pure */ }
|
||||
|
||||
export function createAxiomLogger(config: AxiomConfig) {
|
||||
const axiom = (config.token && config.orgId)
|
||||
? new Client({ token: config.token, orgId: config.orgId }) : null;
|
||||
async function logToAxiom(data: MixedObject, datastream?: string) {
|
||||
const sendData = { pod: config.podName, ...data };
|
||||
if (!config.isProd) { console.log('logToAxiom', sendData); return; }
|
||||
if (!axiom) return;
|
||||
datastream ??= config.datastream;
|
||||
if (!datastream) return;
|
||||
if (config.logErrorsToStdout) console.error(JSON.stringify({ _axiom: datastream, ...sendData }));
|
||||
await axiom.ingestEvents(datastream, sendData);
|
||||
}
|
||||
return { logToAxiom, safeError };
|
||||
}
|
||||
```
|
||||
|
||||
```ts
|
||||
// src/server/logging/client.ts (app shim — the path db/clickhouse shims import)
|
||||
import { createAxiomLogger, safeError } from '@civitai/axiom';
|
||||
import { isProd } from '~/env/other';
|
||||
import { env } from '~/env/server';
|
||||
|
||||
// build guard in the shim: don't construct the Axiom client during `next build`
|
||||
const noopLog = async (_data: MixedObject, _datastream?: string) => {};
|
||||
export const logToAxiom = env.IS_BUILD
|
||||
? noopLog
|
||||
: createAxiomLogger({
|
||||
token: env.AXIOM_TOKEN, orgId: env.AXIOM_ORG_ID, datastream: env.AXIOM_DATASTREAM,
|
||||
podName: env.PODNAME, isProd,
|
||||
logErrorsToStdout: process.env.LOG_ERRORS_TO_STDOUT === 'true',
|
||||
}).logToAxiom;
|
||||
export { safeError };
|
||||
```
|
||||
|
||||
This shim is the composition seed: the db and clickhouse shims import `logToAxiom` from here
|
||||
to build their `onSlowQuery` / `onError`. No package imports another package.
|
||||
|
||||
---
|
||||
|
||||
## 8. `@civitai/telemetry` (helpers stay, pool gauges go back)
|
||||
|
||||
`telemetry/client.ts` splits cleanly:
|
||||
|
||||
- **Keep in package (no factory needed — pure `prom-client`):** `registerCounter`,
|
||||
`registerCounterWithLabels`, `registerGaugeWithLabels`, `registerHistogram`, and the
|
||||
HMR-safe `getSingleMetric` fallback. These take no `env` and import no app code.
|
||||
- **Move back to the app:** the DB pool-depth gauge block
|
||||
([`telemetry/client.ts:~210-296`](../packages/civitai-telemetry/src/client.ts)) that reads
|
||||
`pgDbRead.totalCount`, `idleCount`, `waitingCount`, … across all six pools. It **composes**
|
||||
`@civitai/db` pools + prom helpers → app-level glue. It originally lived in the app's
|
||||
`prom/client.ts`; it returns there and registers gauges using the package's `register*`
|
||||
helpers plus the pool singletons from the db shims:
|
||||
|
||||
```ts
|
||||
// src/server/prom/client.ts (app)
|
||||
import { registerGaugeWithLabels } from '@civitai/telemetry';
|
||||
import { pgDbRead, pgDbReadLong, pgDbWrite } from '~/server/db/pgDb';
|
||||
import { notifDbRead, notifDbWrite } from '~/server/db/notifDb';
|
||||
import { datapacketDbRead } from '~/server/db/datapacketDb';
|
||||
// …register the node_postgres_pool_* gauges exactly as before…
|
||||
```
|
||||
|
||||
No `@civitai/telemetry → @civitai/db` edge remains.
|
||||
|
||||
---
|
||||
|
||||
## 9. App composition order
|
||||
|
||||
Shims form an acyclic graph (package level has **no** edges; app shims compose upward):
|
||||
|
||||
```
|
||||
@civitai/db-schema (LEAF: schema + generated Prisma client/types + Kysely types)
|
||||
▲
|
||||
│ (generated client/types — downward dep, allowed)
|
||||
@civitai/db ─────────────────────────────────────────────┐
|
||||
@civitai/axiom ─▶ src/server/logging/client.ts (logToAxiom)│
|
||||
│ │
|
||||
┌───────────────┼───────────────┐ │
|
||||
▼ ▼ ▼ ▼
|
||||
db/client.ts clickhouse/client.ts (onSlowQuery / onError)
|
||||
redis/client.ts ─▶ ~/server/flipt/client (isEnhancedFailoverEnabled)
|
||||
db/pgDb,notifDb,datapacketDb ─▶ getClient(config)
|
||||
prom/client.ts ─▶ @civitai/telemetry + db pool shims
|
||||
```
|
||||
|
||||
There is no `db ↔ redis ↔ clickhouse` import among packages; any historical coupling (e.g.
|
||||
the old `db-lag-helpers` needing both) lives in app code, which already moved back. The only
|
||||
cross-package edge is the **downward** `@civitai/db → @civitai/db-schema` (generated types).
|
||||
|
||||
---
|
||||
|
||||
## 10. Verification checklist (per package, after each refactor commit)
|
||||
|
||||
- [ ] `grep -rE "from '~/" packages/<pkg>/src` returns **nothing** (no app imports remain).
|
||||
- [ ] Package imports only external npm deps + its own `./` files.
|
||||
- [ ] `pnpm run typecheck` passes (shims satisfy all existing call-site imports).
|
||||
- [ ] `pnpm run build` (Next standalone) succeeds with `transpilePackages: ['@civitai/*']`.
|
||||
- [ ] Dev server boots; redis/db/clickhouse connect; HMR does not duplicate clients
|
||||
(globals reused).
|
||||
- [ ] Prom metrics still register once (no duplicate-registration crash on HMR).
|
||||
- [ ] Slow-query logs still reach Axiom in a prod-like env (`onSlowQuery` wired).
|
||||
|
||||
## 11. Open items / decisions
|
||||
|
||||
- **@ai:*** `concurrency-helpers.ts` — confirm it's dependency-free so we can vendor it into
|
||||
`@civitai/db` rather than inline `limitConcurrency`. (§4.6)
|
||||
- **@ai:*** Tracker extraction — enumerate exactly what other modules import from
|
||||
`~/server/clickhouse/client` today, so the post-split shim re-exports everything callers
|
||||
expect. (§6.2)
|
||||
- **@ai:*** Confirm the `register*` helpers should stay a bare function module (no factory),
|
||||
given they hold no per-app config. (§8)
|
||||
- **@ai:*** Env handling (§2.2): agreed the factories take plain values and never import a
|
||||
central `env`. Still open — do we add **per-package scoped env files** (each package ships
|
||||
its own zod/`@t3-oss/env` schema + `configFromEnv` helper) now, or keep packages fully
|
||||
env-agnostic and let each app own all env validation? Leaning env-agnostic until a second
|
||||
app exists, then extract the scoped-env helper only where it pays off.
|
||||
- **Six packages now**, not five: `@civitai/db-schema` (contract) + `@civitai/db` (Prisma
|
||||
runtime) + `redis` + `clickhouse` + `axiom` + `telemetry`. This is distinct from the
|
||||
*rejected* `civitai-schema-common` (decision 1 in the handoff) — that was domain constants;
|
||||
this is the Prisma-generated DB contract.
|
||||
- Commit sequencing: pure-move commit first (current staged state **plus** the
|
||||
`prisma/`+`enums.ts`+`models.ts` relocation into `@civitai/db-schema`), then one content
|
||||
commit per package (db-schema generators → db → redis → clickhouse → axiom → telemetry),
|
||||
each independently verifiable.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user