docs(retool): hold the moderator id table privately, genericise internal refs (#4476)

This repository is public. raw/README.md already documented a sanitisation rule for
the Retool exports, but that pass matched on the SHAPE of a value, so content that
looks like ordinary prose went straight through.

Staff identity. moderator-id-mapping.md paired moderation team members' real names
with their Civitai user ids and usernames, and user-lookup-v2.json gated features on
current_user.fullName === a real name, so the authorization model itself was written
in names. The table now lives in the private infra repo; the public doc keeps the
coverage figures and the backfill method and points there. Export literals became
__MODERATOR_A__..__MODERATOR_C__, underscore-delimited because bare MODERATOR_A is a
strict prefix of the live MODERATOR_APP_URL env var.

One staff member had all three legs of a pseudonym-to-identity linkage present in
this repo -- pseudonym, userId in the retained moderator id set, and real name in a
planning doc. That one was reconstructible from repo content alone; the display name
is dropped. Other staff mentions complete no linkage and stand.

End-user identifiers. bulk-ban carried four real banned users' IP addresses and five
real account ids. Now RFC5737 addresses and <accountId>.

Internal service names. Comments, .env.examples and test fixtures named real
in-cluster services and two node hostnames. Every namespace token is now clear.
Deliberately unchanged: five executed fallbacks, verify-runner as an application
identifier, stub-oidc as a local e2e test double, and cnpg-database in immutable
Prisma migrations.

Four adversarial audit rounds. The redactions verified clean every round; the prose
describing them needed four public corrections, including one that misattributed
which moderator was exposed and one that misdiagnosed why a survivor survived -- it
was a term-list defect, not the line wrap I claimed, so the guard I added protected
against the wrong mechanism. Corrections are on the PR.

These files were already committed and pushed, so this narrows further exposure
rather than undoing it. Nothing here is a credential. The history decision -- leave
it, treat the content as disclosed -- is recorded on the tracking ticket.

Verified: 168 tests across four suites (run with --project; without it vitest
silently runs three files and still reports success), 11/11 raw exports parse, byte
deltas match the replacements arithmetically.
This commit is contained in:
Zachary Lowden
2026-08-28 19:06:11 -05:00
committed by GitHub
parent a3f30603ef
commit ed0b0b04e3
29 changed files with 104 additions and 111 deletions
@@ -79,7 +79,8 @@ Add anything new to this checklist and to
- [ ] Apply [the moderator id mapping](../../../docs/moderator-app/retool-exports/moderator-id-mapping.md)
— 72% exact, 21% variants to confirm, 6% unmapped.
- [ ] Decide the fate of `Brittany Widlund`'s 3,888 unmapped notes.
- [ ] Decide the fate of the largest unmapped identifier's 3,888 notes (a former moderator — named in
the private mapping, not here).
- [ ] `TimedMutes.userId` is `text` where the others are `integer` — cast, after checking for
non-numeric values.
- [ ] Rotate the Retool credentials once the migration is done.
@@ -18,7 +18,7 @@ const STUB_KEYS = [
] as const;
// URLs must pass validatedStubUrl (https, or http to an in-cluster .svc host) — use the real shape.
const STUB_BASE = 'http://stub-oidc.civitai-auth-staging.svc.cluster.local:8080';
const STUB_BASE = 'http://stub-oidc.internal.svc.cluster.local:8080';
const FULL = {
AUTH_ENABLE_STUB_PROVIDER: '1',
STUB_AUTHORIZE_URL: `${STUB_BASE}/authorize`,
@@ -127,10 +127,10 @@ describe('stub provider — SSRF-safe URL validation (pinned to the in-cluster S
const https = await load({
...FULL,
STUB_TOKEN_URL: 'https://stub-oidc.civitai-auth-staging.svc.cluster.local/token',
STUB_TOKEN_URL: 'https://stub-oidc.internal.svc.cluster.local/token',
});
expect(https.getProvider('stub')!.tokenUrl).toBe(
'https://stub-oidc.civitai-auth-staging.svc.cluster.local/token'
'https://stub-oidc.internal.svc.cluster.local/token'
);
});
+1 -1
View File
@@ -71,7 +71,7 @@ SIGNALS_ENDPOINT=
DATABASE_URL=postgresql://user:pass@localhost:5432/civitai
# The MODERATOR database - moderation data that has never lived in Civitai's Postgres, plus the
# XGuard label lab tables. Separate instance, no foreign keys into the main database.
# Prod: internal-tools-db-rw.cnpg-database.svc.cluster.local:5432/internal_tools (via bastion)
# Prod: <service>.<namespace>.svc.cluster.local:5432/internal_tools (via bastion)
# Local: docker compose -f xguard-lab/docker-compose.yml up -d
# ONE database for everything the moderator app owns: the XGuard lab tables AND the moderation tables
# (user notes, strikes, image help requests) that `getModeratorDb()` reads. `RETOOL_DATABASE_URL` used to
+1 -1
View File
@@ -20,7 +20,7 @@
-- GRANT USAGE, SELECT ON SEQUENCE abuse_detection_run_id_seq, abuse_detection_finding_id_seq TO internal_tools;
--
-- 🔴 `MODERATOR_DATABASE_URL` and `RETOOL_DATABASE_URL` currently resolve to the SAME instance
-- (`internal-tools-db-rw…/internal_tools`, measured 2026-08-21, post-Retool-cutover). Either works
-- (the same `internal_tools` database, measured 2026-08-21, post-Retool-cutover). Either works
-- today; this names the one whose purpose is new moderator data.
--
-- Idempotent: safe to re-run.
@@ -14,7 +14,7 @@ const abuseDb = () => getModeratorDb().withTables<AbuseDetectionTables>();
* there is only one name: the Retool cutover landed 2026-08-18 (deployment repo `ee835acaf`),
* repointing `RETOOL_DATABASE_URL` at the same `internal_tools` database and retiring it, and
* `moderator-db.ts` now reads `MODERATOR_DATABASE_URL` alone. Verified against the running pod:
* both keys resolve to `internal-tools-db-rw.cnpg-database.svc.cluster.local:5432/internal_tools`,
* both keys resolve to `<service>.<namespace>.svc.cluster.local:5432/internal_tools`,
* whose `public` schema holds the live tables while the pre-cutover snapshot sits in a `cutover`
* schema. One database, therefore one pool.
*/
+1 -1
View File
@@ -28,7 +28,7 @@ REDIS_CLUSTER=false
# System redis — GENERATION.TOKENS cache + generation-version gate + block budget counters. In prod this
# is discovered via REDIS_SYS_SENTINELS (group `sysmaster`); REDIS_SYS_URL still gates the factory. [secret]
REDIS_SYS_URL=redis://localhost:6379
# REDIS_SYS_SENTINELS=civitai-app-sysredis-sentinel-sentinel.civitai-app-sysredis.svc.cluster.local:26379
# REDIS_SYS_SENTINELS=<sentinel-service>.<namespace>.svc.cluster.local:26379
# REDIS_SYS_SENTINEL_NAME=sysmaster
# --- external orchestrator SDK (@civitai/client) ---
+2 -2
View File
@@ -125,8 +125,8 @@ migrations to hand-apply.**
longer forwarded). Optional cleanup of the producer computation + the `ModelSlotContext` field.
**Infra (need cluster write access — see the datapacket-talos handoff doc):**
- HPA `FailedGetResourceMetric`: node pressure on `talos-uvh-ow7` (over the 110-pod cap) and
Tekton completed-pod bloat on `talos-x3r-mnv`; the `pipelinerun-pruner` was recently un-broken —
- HPA `FailedGetResourceMetric`: node pressure on a search-pool node (over the 110-pod cap) and
Tekton completed-pod bloat on a build node; the `pipelinerun-pruner` was recently un-broken —
confirm it's draining the backlog.
- `workflow-completed.ts` still uses the non-atomic `incrBy`+`expire` dedup (the same wedge class
fixed in `build-callback`) — align it to `setNxKeepTtlWithEx`.
+4 -2
View File
@@ -27,8 +27,10 @@ Progress against these lives in
[**retool-db-tables.md**](retool-db-tables.md) covers the other half of the migration: which tables in
Retool's own Postgres these apps actually depend on (7 of 43), and what has to move.
[**moderator-id-mapping.md**](moderator-id-mapping.md) maps Retool's free-text moderator names to real
user ids — needed before 69,100 notes and strikes can keep their attribution.
[**moderator-id-mapping.md**](moderator-id-mapping.md) covers mapping Retool's free-text moderator
names to real user ids — needed before 69,100 notes and strikes can keep their attribution. It holds
the coverage figures and the method; **the name↔id table itself is in the private repo**, because it
pairs staff real names with their Civitai accounts and this repository is public.
## The raw exports ARE here now — sanitized
@@ -73,7 +73,7 @@ resources: REST-WithoutResource, JavascriptQuery, Replicated_Read_Prod, retool_d
SELECT DISTINCT "fromAccountId"
FROM "default"."buzzTransactions"
WHERE amount >= 50
AND "toAccountId" IN(4059236, 5561675, 5603046, 5601367, 5551054)
AND "toAccountId" IN(<accountId>, <accountId>, <accountId>, <accountId>, <accountId>)
AND "type" = 'tip'
AND "fromAccountId" > 5400000
ORDER BY 1 DESC
@@ -111,7 +111,7 @@ resources: REST-WithoutResource, JavascriptQuery, Replicated_Read_Prod, retool_d
### UsersByIp [SqlQuery / Clickhouse]
SELECT DISTINCT targetUserId
FROM default.userActivities
WHERE ip IN('109.236.62.211', '109.236.63.69', '109.236.63.55', '185.245.255.238')
WHERE ip IN('203.0.113.1', '203.0.113.2', '203.0.113.3', '203.0.113.4')
AND type = 'Registration'
ORDER BY 1
@@ -1,38 +1,23 @@
# Retool moderator → Civitai user id
Retool records attribution as free-text display names (`lastUpdateBy`, `createdBy`, `handledBy`,
`ReToolActions.User`). This is the mapping to real user ids, supplied by the moderation team
2026-08-06. Without it, "who wrote this note" is lost for most of 69,100 records.
`ReToolActions.User`). Mapping those to real user ids is what keeps "who wrote this note" from being
lost for most of 69,100 records.
New rows written by the moderator app use the moderator's **Civitai username** instead, so the column
holds two naming schemes until the backfill runs — see
**The mapping table itself is not held in this repository.** It pairs staff members' real names with
their Civitai accounts, which is personal data about identifiable people — and this repository is
public. It lives in the private infrastructure repo instead:
> `claudedocs/moderator-id-mapping.md` in `civitai/talos-infra`
Ask in `#moderation` if you need access. Everything below is the part that carries no personal data:
the coverage figures, the variants that still need a human answer, and how to apply the mapping when
the backfill is written.
New rows written by the moderator app use the moderator's **Civitai username** instead of a free-text
name, so the column holds two naming schemes until the backfill runs — see
[retool-db-tables.md](retool-db-tables.md).
## Mapping
| userId | username | Retool display name |
| --- | --- | --- |
| 5418 | `theally` | Ally Nicoli |
| 149676 | `Valstrix` | Logan Waxler |
| 203133 | `CHESHIRE_OS` | Navi_OS |
| 296765 | `Seb` | Sebastian Widlund |
| 984231 | `Temporarium` | Temporarium |
| 2023372 | `KesWasHere` | Kes Krcha |
| 2342520 | `dolirama126` | Tomas Sitar |
| 2345535 | `DazMakeArt` | Dazzer Wave |
| 11579707 | `Ellie_TheFoxyPaladin` | Ellie King |
| 11841732 | `wade_mod` | *(agent account)* |
### Former moderators
| userId | Retool display name |
| --- | --- |
| 3 | Maxfield Hulker |
| 573 | Cameron Jackson |
| 1019954 | Lars Wilstermann |
| 2709 | Joseph McPeeks |
| 4938487 | Jane Kim |
## Coverage against the live data
Across `UserNotes` + `UserStrikes`**69,100 attributed records, 37 distinct identifiers**:
@@ -45,45 +30,24 @@ Across `UserNotes` + `UserStrikes` — **69,100 attributed records, 37 distinct
## Variants to confirm before the backfill
Retool stores several people under more than one identifier. These are inferred, not given:
Retool stores several people under more than one identifier. These are inferred, not given, and the
identifiers are held with the mapping in the private repo. Two need a human answer rather than a
guess, and both are recorded there:
| identifier in data | records | assumed to be | confidence |
| --- | --- | --- | --- |
| `Valstrix Waxler` | 6,100 | Logan Waxler (149676) | **check this one** — 6k records hinge on it |
| `Logan` | 1,751 | Logan Waxler (149676) | likely |
| `Valstrix` | 135 | Logan Waxler (149676) | likely (matches the username) |
| `Sebastian` | 460 | Sebastian Widlund (296765) | likely |
| `Tomáš` / `Tomáš Sitár` | 1,535 | Tomas Sitar (2342520) | accents differ from the supplied spelling |
| `Ally Nicoll` | 776 | Ally Nicoli (5418) | **spelling differs** — Nicoll vs Nicoli |
| `Temporarium S` | 517 | Temporarium (984231) | likely |
| `Navi_OS null` | 180 | Navi_OS (203133) | likely — a null surname concatenated |
| `Cameron`, `Lars`, `Kes`, `Dazzer`, `Ally`, `Joseph`, `Jane`, `Ellie` | ~3,300 | their full-name entries | likely |
Two need a human answer rather than a guess:
1. **`Valstrix Waxler` (6,100 records)** — the supplied row is `149676 / Valstrix / Logan Waxler`, so
this looks like the same person under a changed Retool display name. It is the single largest
variant; getting it wrong misattributes 6,100 notes.
2. **`Ally Nicoll` vs supplied `Ally Nicoli`** and **`Tomáš Sitár` vs supplied `Tomas Sitar`** — the
data does not match the mapping character-for-character. Match case- and accent-insensitively, or
correct the mapping.
1. **The largest variant accounts for 6,100 records** — it looks like one person under a changed
Retool display name, but it is inferred. Getting it wrong misattributes 6,100 notes.
2. **Two supplied spellings do not match the data character-for-character** — one differs by a single
letter, one by accents. Match case- and accent-insensitively, or correct the mapping.
## Unmapped — 4,317 records
"Not listed means the records can probably be removed", per the moderation team. One entry does not
look disposable:
look disposable: a single former moderator accounts for **3,888 records — 90% of the unmapped total**.
Deleting that discards 3,888 moderator notes, and a former moderator missing from the list is far more
likely than 3,888 junk rows. Confirm before treating it as removable.
| identifier | records | note |
| --- | --- | --- |
| **Brittany Widlund** | **3,888** | **90% of the unmapped total.** Deleting this discards 3,888 moderator notes. Confirm before treating it as removable — a former moderator missing from the list is more likely than 3,888 junk rows. |
| Brittany | 195 | same person |
| Richard / Richard Kelly | 96 | |
| Justin Maier | 64 | |
| Paul Geraghty / Paul | 32 | |
| Bruno Henrique / Bruno Henrique Roda | 29 | |
| civitai-mod | 13 | service account, not a person |
Excluding Brittany, genuinely unmapped volume is **434 records (0.6%)** — that part is safe to drop.
Excluding that one identifier, genuinely unmapped volume is **434 records (0.6%)** — that part is safe
to drop.
## Applying it
@@ -43,6 +43,30 @@ one sentence from being an evasion guide. **Both were already in the committed i
directory existed, and those commits are pushed**, so they are disclosed; the redaction stops the leak
widening rather than undoing it. Neither is a credential, so there is nothing to rotate.
**Personal data is redacted on the same rule, and it is the easiest kind to miss** — it looks like
ordinary content rather than like a secret, so a shape-based sanitiser walks straight past it. Three
classes were found here on 2026-08-28 and are now stripped. **Sanitise a fresh export against all
three** — this list is the checklist, so a class missing from it comes back on the next re-download:
- **Staff real names.** `user-lookup-v2.json` gates features on `current_user.fullName === '<a real
person>'`, so the authorization model itself was written in names. Those are now stable
`__MODERATOR_A__`…`__MODERATOR_C__` placeholders — three names appear in this export, and a
distinct token per person keeps it readable as logic. The tokens are underscore-delimited because
bare `MODERATOR_A` is a prefix of the live `MODERATOR_APP_URL` env var, so an un-redacting
find-and-replace would corrupt real identifiers. The name↔placeholder key lives with the id mapping in the private repo.
- **End-user IP addresses.** `bulk-ban.json` and its rendered `bulk-ban.md` carried four real banned
users' IPs inside a `WHERE ip IN (…)` clause. They are now RFC5737 `203.0.113.x` documentation
addresses.
- **End-user account ids.** The same two files carried five real `toAccountId` values hardcoded into
a tip-farming investigation query. They are now `<accountId>`. This class is easy to miss precisely
because a bare integer has no shape to match on — the only way to catch it is to read what the
query is *for*, and a ban or abuse investigation names real people by id.
All three were already committed and pushed, so — as with the CIDR above — the redaction stops the
leak widening rather than undoing it. None is a credential, so there is nothing to rotate; but a name
tying a **pseudonymous moderator account to a real identity** is the one class here where the harm is
to a person rather than to the system, so treat it as the highest bar when sanitising a fresh export.
The inventories are generated FROM these files, so regenerate rather than hand-editing —
`extract.mjs raw/<app>.json > <app>.md` — or a redacted export will quietly grow an unredacted
inventory beside it.
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -127,7 +127,9 @@ before discarding.
1. **Moderators are identified by name, not id — and the names mostly do not resolve.**
`createdBy`, `lastUpdateBy`, `handledBy` and `ReToolActions.User` are free text holding *Retool
account display names* ("Maxfield Hulker", "Cameron Jackson"), not Civitai usernames.
account display names* — a person's real first and last name — not Civitai usernames. The names
themselves are held with the mapping in the private repo; see
[moderator-id-mapping.md](moderator-id-mapping.md).
Measured 2026-08-06: **37 distinct identifiers across `UserNotes` + `UserStrikes`; only 5 match a
Civitai moderator account** by `username` or `name` (24 moderator accounts exist). None are emails.
+1 -1
View File
@@ -17,7 +17,7 @@ Verify article images are NSFW-scanned. If not, implement scanning so articles o
- [x] Add article image scanning
- [x] Add article content scanning
- [x] Hook up text scanning to use XGuard [@Manuel Emilio](#user_mention#16807894) (Talk with Briant about how to use XGuard)
- [ ] Add PG-XXX ratings to Text Scan category [@Sebastian Widlund](#user_mention#63130867)
- [ ] Add PG-XXX ratings to Text Scan category [@team-member](#user_mention#63130867)
- [x] Scan all articles to apply content and image ratings
- [x] Images
- [x] Content
@@ -162,13 +162,13 @@ describe('createSessionClient — internal identity routing (AUTH_HUB_INTERNAL_U
h.loadAuthEnv.mockReturnValue({
AUTH_JWT_ISSUER: 'https://auth.test',
AUTH_INTERNAL_TOKEN: 'secret-123',
AUTH_HUB_INTERNAL_URL: 'http://civitai-auth.civitai-auth.svc.cluster.local:3000',
AUTH_HUB_INTERNAL_URL: 'http://auth-hub.internal.svc.cluster.local:3000',
});
const fetch = stubFetch(async () => ({ ok: true, status: 200, json: async () => richUser(7) }));
expect(await createSessionClient().getSessionUser('7')).toMatchObject({ id: 7 });
// FETCH target is the internal svc; the bearer is still the user's token.
expect(fetch).toHaveBeenCalledWith(
'http://civitai-auth.civitai-auth.svc.cluster.local:3000/api/auth/identity',
'http://auth-hub.internal.svc.cluster.local:3000/api/auth/identity',
{ headers: { authorization: 'Bearer 7' }, signal: expect.any(AbortSignal) }
);
});
@@ -188,7 +188,7 @@ describe('createSessionClient — internal identity routing (AUTH_HUB_INTERNAL_U
h.loadAuthEnv.mockReturnValue({
AUTH_JWT_ISSUER: 'https://auth.test',
AUTH_INTERNAL_TOKEN: 'secret-123',
AUTH_HUB_INTERNAL_URL: 'http://civitai-auth.civitai-auth.svc.cluster.local:3000',
AUTH_HUB_INTERNAL_URL: 'http://auth-hub.internal.svc.cluster.local:3000',
});
h.verifyToken.mockImplementation(async (t: string) => ({ sub: t, iss: 'https://evil.example' }));
const fetch = stubFetch(async () => ({ ok: true, status: 200, json: async () => richUser(7) }));
@@ -236,12 +236,12 @@ describe('createSessionClient — getSessionUserById internal routing + timeout'
h.loadAuthEnv.mockReturnValue({
AUTH_JWT_ISSUER: 'https://auth.test',
AUTH_INTERNAL_TOKEN: 'secret-123',
AUTH_HUB_INTERNAL_URL: 'http://civitai-auth.civitai-auth.svc.cluster.local:3000',
AUTH_HUB_INTERNAL_URL: 'http://auth-hub.internal.svc.cluster.local:3000',
});
const fetch = stubFetch(async () => ({ ok: true, status: 200, json: async () => richUser(7) }));
await createSessionClient().getSessionUserById(7);
expect(fetch).toHaveBeenCalledWith(
'http://civitai-auth.civitai-auth.svc.cluster.local:3000/api/auth/identity?userId=7',
'http://auth-hub.internal.svc.cluster.local:3000/api/auth/identity?userId=7',
{ headers: { authorization: 'Bearer secret-123' }, signal: expect.any(AbortSignal) }
);
});
+2 -2
View File
@@ -15,12 +15,12 @@ const schema = z.object({
AUTH_JWT_ISSUER: z.string().optional(), // e.g. https://auth.civitai.com
// SERVER-ONLY internal-routing override for the session-identity read (session-client.ts fetchIdentity).
// When set, the identity fetch is sent to THIS base (the in-cluster hub svc, e.g.
// http://civitai-auth.civitai-auth.svc.cluster.local:3000) instead of hairpinning out to the public
// http://<auth-hub-service>.<namespace>.svc.cluster.local:3000) instead of hairpinning out to the public
// AUTH_JWT_ISSUER origin (CF edge → Traefik → back to the same cluster). It ONLY changes the FETCH target —
// the token's `iss` is STILL validated against the public trusted origin(s) first (trustedHubBase), so the
// override can never be used to send a bearer to an untrusted issuer. Deliberately NOT a `NEXT_PUBLIC_*`
// var: it is read on the server only and must never reach the client bundle.
AUTH_HUB_INTERNAL_URL: z.url().optional(), // e.g. http://civitai-auth.civitai-auth.svc.cluster.local:3000
AUTH_HUB_INTERNAL_URL: z.url().optional(), // e.g. http://<auth-hub-service>.<namespace>.svc.cluster.local:3000
// Legacy symmetric secret — kept ONLY for the migration window so spokes can still
// decode pre-cutover next-auth JWE cookies. Drop after the max old-token TTL.
NEXTAUTH_SECRET: z.string().optional(),
@@ -2765,7 +2765,7 @@ model AppOwnershipTransfer {
/// W1 v0 publish request — every version of every app goes through the
/// moderator review queue. The dev uploads a ZIP via /apps/submit (first
/// version) or /apps/<slug>/submit-version (subsequent). civitai-web
/// stores the bundle on ssd-minio-backups MinIO, extracts + validates the
/// stores the bundle on the in-cluster MinIO tenant, extracts + validates the
/// manifest, computes diff summaries, and inserts a row here in
/// status='pending'. A mod approves or rejects via /apps/review.
///
+1 -1
View File
@@ -16,7 +16,7 @@ export const redisEnvSchema = z
REDIS_CLUSTER_REFRESH_INTERVAL: z.coerce.number().default(30000),
// sysRedis HA (Phase 1): when REDIS_SYS_SENTINELS is set the system client is built
// via Sentinel discovery instead of the single REDIS_SYS_URL connection. Comma-separated
// host:port list, e.g. "civitai-app-sysredis-sentinel...:26379".
// host:port list, e.g. "<sentinel-service>.<namespace>.svc.cluster.local:26379".
REDIS_SYS_SENTINELS: z.string().optional(),
// Required whenever REDIS_SYS_SENTINELS is set (cluster uses "sysmaster") — see superRefine.
REDIS_SYS_SENTINEL_NAME: z.string().optional(),
+9 -9
View File
@@ -49,7 +49,7 @@ export const serverSchema = z
// switches the system client to `createSentinel(...)` against this Sentinel
// pool. See claudedocs/sysredis-ha-migration-runbook.md (datapacket-talos)
// for the rollout sequence.
REDIS_SYS_SENTINELS: z.string().optional(), // comma-separated host:port list, e.g. "civitai-app-sysredis-sentinel.civitai-app-sysredis.svc.cluster.local:26379"
REDIS_SYS_SENTINELS: z.string().optional(), // comma-separated host:port list, e.g. "<sentinel-service>.<namespace>.svc.cluster.local:26379"
// Master group name. No default — the cluster uses "sysmaster", and the
// historical Sentinel default ("mymaster") would silently fail every lookup.
// The superRefine below makes this required whenever REDIS_SYS_SENTINELS is set.
@@ -837,7 +837,7 @@ export const serverSchema = z
// APPS_TEKTON_TRIGGER_URL HTTP endpoint that creates PipelineRuns on
// dc-02-a (the app-blocks-trigger receiver,
// reached via the VPN proxy on dp-1). Example:
// http://wireguard-proxy-service.civitai-submodel-proxy.svc.cluster.local:8088/trigger-build
// http://<proxy-service>.<namespace>.svc.cluster.local:8088/trigger-build
// APPS_TEKTON_TRIGGER_SECRET HMAC shared secret between civitai-web and
// the app-blocks-trigger receiver. 32-byte hex.
// APPS_KUBE_NAMESPACE civitai-apps (where apply Jobs are created
@@ -883,7 +883,7 @@ export const serverSchema = z
// (no new secret). OPTIONAL — when unset, triggerReviewBuild derives it from
// APPS_TEKTON_TRIGGER_URL by swapping the trailing `/trigger-build` segment
// for `/trigger-review-build`, so a typical deploy needs no extra env. Example:
// http://wireguard-proxy-service.civitai-submodel-proxy.svc.cluster.local:8088/trigger-review-build
// http://<proxy-service>.<namespace>.svc.cluster.local:8088/trigger-review-build
APPS_TEKTON_REVIEW_TRIGGER_URL: z.string().url().optional(),
APPS_KUBE_NAMESPACE: z.string().default('civitai-apps'),
APPS_DOMAIN: z.string().default('civit.ai'),
@@ -924,19 +924,19 @@ export const serverSchema = z
// the callback falls back to NEXTAUTH_URL (the public origin) so the feature
// keeps working before infra sets the in-cluster value ahead of un-dark.
AGENT_REVIEW_CALLBACK_BASE_URL: z.string().optional(),
// Base URL of the verify-runner screenshot service (warm Playwright Chromium)
// Base URL of the screenshot-runner service (warm Playwright Chromium)
// used to autogenerate a marketplace screenshot for an approved App Block that
// shipped no publisher screenshots. In-cluster service (devpod-devops ns), e.g.
// http://verify-runner.devpod-devops.svc.cluster.local:8080. OPTIONAL — when
// shipped no publisher screenshots. In-cluster service, e.g.
// http://<service>.<namespace>.svc.cluster.local:8080. OPTIONAL — when
// unset, autogeneration is silently skipped (best-effort; never blocks deploy).
BLOCK_SCREENSHOT_RUNNER_URL: z.string().url().optional(),
// App Blocks W1 (publish-request flow). S3-compatible storage for
// dev-uploaded ZIP bundles. Production points at ssd-minio-backups
// MinIO with credentials scoped to the app-block-bundles bucket only.
// dev-uploaded ZIP bundles. Production points at an in-cluster MinIO
// tenant with credentials scoped to the app-block-bundles bucket only.
// All optional so envs without the publish-request feature still boot.
//
// BUNDLE_S3_ENDPOINT e.g. http://minio.minio-ssd-backups.svc.cluster.local
// BUNDLE_S3_ENDPOINT e.g. http://<minio-service>.<namespace>.svc.cluster.local
// BUNDLE_S3_BUCKET e.g. app-block-bundles
// BUNDLE_S3_ACCESS_KEY_ID scoped service-account key
// BUNDLE_S3_SECRET_ACCESS_KEY matching secret
@@ -295,7 +295,7 @@ describe('fetchDocumentsAbortable timeout safety net', () => {
//
// The 5s local timer in #2370 caught the slow-fetch path, but post-deploy
// telemetry showed the dominant failure mode is upstream returning 503
// (civitai-feeds-proxy shed) or 408 (Meilisearch backend timeout). These
// (feeds proxy shed) or 408 (Meilisearch backend timeout). These
// were re-throwing as bare Error from fetchDocumentsAbortable, so the
// post-filter catch in image.service.ts didn't recognise them and the
// 500 bubbled to clients → retry storm → cascade sustained.
+4 -4
View File
@@ -99,7 +99,7 @@ export const FETCH_DOCUMENTS_TIMEOUT_MESSAGE = 'meili-fetch-timeout';
* - `local-timeout` the local 5s deadline fired before the upstream
* responded (PR #2370 the original behaviour;
* preserved for backward-compat dashboards)
* - `upstream-overload` upstream returned HTTP 503 (civitai-feeds-proxy
* - `upstream-overload` upstream returned HTTP 503 (the feeds proxy
* shed because MEILI_MAX_CONCURRENT was hit)
* - `upstream-timeout` upstream returned HTTP 408 (Meilisearch backend
* page-cache thrashing past its own timeout)
@@ -208,7 +208,7 @@ export const TRANSIENT_STATUSTEXT_TO_STATUS: Readonly<Record<string, number>> =
};
/**
* Classify an error caught from a Meilisearch SDK call (or the civitai-feeds
* Classify an error caught from a Meilisearch SDK call (or the feeds
* proxy in front of it) as a genuinely-transient upstream failure that should
* surface to the client as a retryable 503, NOT a hard 500.
*
@@ -425,7 +425,7 @@ export const meiliFetchFailfastTotal = registerCounterWithLabels({
/**
* Typed error thrown by fetchDocumentsAbortable() when the upstream Meili
* backend (or the civitai-feeds-proxy in front of it) returns a non-ok HTTP
* backend (or the feeds proxy in front of it) returns a non-ok HTTP
* status code. Carries the status + response body verbatim so callers can
* pattern-match on `instanceof MeilisearchFetchError` + `.status` instead of
* string-matching the message.
@@ -588,7 +588,7 @@ export class MeiliCallTimeoutError extends Error {
/**
* Backends are limited independently because they fail independently:
* - 'search' SEARCH_HOST (civitai-feeds / searchClient / feed inline client)
* - 'search' SEARCH_HOST (feeds proxy / searchClient / feed inline client)
* - 'metricsSearch' METRICS_SEARCH_HOST (search-meilisearch / metricsSearchClient)
*
* A single shared limiter would let one bad backend exhaust the budget for
@@ -155,7 +155,7 @@ describe('isAllowedOutputHost', () => {
'https://evil.com/?x=civitai.com', // query substring
'https://civitai.com@evil.com/x.jpeg', // userinfo
'http://orchestration.civitai.com/x.jpeg', // not https
'https://orchestration-api.orchestration-poc.svc.cluster.local:8080/x', // the internal API host
'https://orchestration-api.internal.svc.cluster.local:8080/x', // the internal API host
'https://notcivitai.com/x.jpeg',
'not a url',
]) {
@@ -7,7 +7,7 @@
* receiver on dc-02-a, which validates HMAC and
* creates a PipelineRun via its in-pod ServiceAccount.
* Reached via the dp-1 VPN proxy at
* wireguard-proxy-service.civitai-submodel-proxy.svc:8088.
* <proxy-service>.<namespace>.svc:8088.
* Called by the Forgejo push webhook handler.
* 2. triggerApply() POSTs an apply Job to dp-1's civitai-apps
* namespace via the in-pod ServiceAccount token.
@@ -14,7 +14,7 @@
* When ENABLED (historical behaviour): for an APPROVED App Block that ships NO
* publisher screenshots, we render the running block at
* `https://<slug>.<APPS_DOMAIN>` via the verify-runner screenshot service (a
* warm headless Chromium in devpod-devops) and store the PNG through the
* warm headless Chromium in its own namespace) and store the PNG through the
* EXISTING screenshots path (`storeScreenshots` bundle MinIO
* `app_blocks.screenshots`). The marketplace card + detail page then render it
* identically to a dev-uploaded image (served by the gated
@@ -12,7 +12,7 @@
* civitai-team-only, so the admin scope is acceptable; v1 (W5 + W11)
* tightens to per-user OAuth tokens.
*
* Network shape: civitai-web forgejo-http.forgejo.svc.cluster.local:3000
* Network shape: civitai-web <forgejo-service>.<namespace>.svc.cluster.local:3000
* inside the cluster, or https://forgejo.civitai.com from a PR-preview
* env that doesn't have direct cluster DNS. FORGEJO_BASE_URL handles both.
*/
+1 -1
View File
@@ -5950,7 +5950,7 @@ export async function getImagesFromSearchPostFilter(input: ImageSearchInput) {
//
// Status-code rationale:
// - 408 (upstream-timeout) → Meilisearch backend page-cache thrash
// - 503 (upstream-overload) → civitai-feeds-proxy shed (MEILI_MAX_CONCURRENT)
// - 503 (upstream-overload) → the feeds proxy shed (MEILI_MAX_CONCURRENT)
// - other 5xx → upstream brownout / Traefik 504 / etc.
// - 4xx-other (400/401/403) → real client error, MUST bubble up
let results: ImageMetricsSearchIndexRecord[];
+1 -1
View File
@@ -11,7 +11,7 @@ import { env } from '~/env/server';
/**
* S3-compatible client for the App Blocks W1 publish-request bundle store.
*
* Production points at ssd-minio-backups MinIO. Credentials are scoped to
* Production points at an in-cluster MinIO tenant. Credentials are scoped to
* the app-block-bundles bucket only (R/W on objects, ListBucket on the
* bucket itself) the bundle store is intentionally not the same S3
* civitai uses for image uploads.
+2 -2
View File
@@ -12,8 +12,8 @@ import { retryFlaky } from './preview-retry';
* - A broad image.getInfinite (no entity filter) routes server-side through
* getAllImagesIndex the MEILISEARCH index path (image.controller.ts:300-320),
* not the DB. The index `metrics_images_v1` (~114M docs) lives on the in-cluster
* feeds-meilisearch (civitai-feeds-proxy), and previews reach it via
* METRICS_SEARCH_HOST=http://civitai-feeds-proxy.civitai-feeds.svc.cluster.local
* feeds-meilisearch proxy, and previews reach it via
* METRICS_SEARCH_HOST=http://feeds-proxy.internal.svc.cluster.local
* (same host prod uses; verified both feeds backends populated 2026-06-11). So
* the broad feed IS available to previews and this asserts the real /images
* surface end to end (meili query + DB hydration).