2022-10-11 16:56:51 -04:00
|
|
|
# Since .env is gitignored, you can use .env-example to build a new `.env` file when you clone the repo.
|
|
|
|
|
# Keep this file up-to-date when you add new variables to `.env`.
|
|
|
|
|
# This file will be committed to version control, so make sure not to have any secrets in it.
|
|
|
|
|
# If you are cloning this repo, create a copy of this file named `.env` and populate it with your secrets.
|
|
|
|
|
# When adding additional env variables, the schema in /env/schema.mjs should be updated accordingly
|
2023-02-28 04:49:51 +08:00
|
|
|
# The default values for Prisma, Redis, S3, and Email are set to work with the docker-compose setup
|
2022-10-11 16:56:51 -04:00
|
|
|
|
2024-10-30 18:07:05 -04:00
|
|
|
# Database
|
|
|
|
|
DATABASE_SSL=false
|
|
|
|
|
DATABASE_URL=postgresql://postgres:postgres@localhost:15432/civitai
|
|
|
|
|
DATABASE_REPLICA_URL=postgresql://postgres:postgres@localhost:15432/civitai
|
|
|
|
|
|
|
|
|
|
NOTIFICATION_DB_URL=postgresql://postgres:postgres@localhost:15434/postgres
|
|
|
|
|
NOTIFICATION_DB_REPLICA_URL=postgresql://postgres:postgres@localhost:15434/postgres
|
|
|
|
|
|
2026-03-13 12:43:38 -05:00
|
|
|
DATAPACKET_DATABASE_RO_URL=postgresql://postgres:postgres@localhost:15435/postgres
|
2025-08-12 13:20:21 -04:00
|
|
|
|
2024-10-30 18:07:05 -04:00
|
|
|
# Redis
|
2023-02-21 17:44:55 -04:00
|
|
|
REDIS_URL=redis://:redis@localhost:6379
|
2025-01-13 14:05:00 -05:00
|
|
|
REDIS_SYS_URL=redis://:redis@localhost:6378
|
redis: add Sentinel client branch for system Redis (Phase 1, HA migration) (#2331)
* redis: add Sentinel client branch for system Redis (Phase 1 of HA migration)
Add an optional `createSentinel(...)` code path for the `system` Redis
client, gated on a new optional REDIS_SYS_SENTINELS env var. When unset
(default), the existing REDIS_SYS_URL path is used and behavior is
unchanged in any deployed environment — zero risk to ship ahead of the
infra-side cutover.
When REDIS_SYS_SENTINELS is set, the client uses node-redis v5's
Sentinel API:
REDIS_SYS_SENTINELS=civitai-app-sysredis-sentinel.civitai-app-sysredis.svc.cluster.local:26379
REDIS_SYS_SENTINEL_NAME=sysmaster
The credential currently extracted from REDIS_SYS_URL (authConfig.password)
is reused — the infra-side HA cluster is provisioned with the same
password, so no separate REDIS_SYS_PASSWORD env var is needed.
Falls back to the existing single-node createClient on every env where
REDIS_SYS_SENTINELS isn't explicitly set, so this PR is safe to merge
ahead of the per-pool env-var flip during Phase 4 cutover.
Phase 1.5 (atomic HEXPIRE NX helper) and Phase 1.6 (`getOrchestratorToken`
cold-mint cache) are pre-HA-cutover hard prereqs but are tracked as
separate PRs to keep this one small and easy to verify.
Infra side already live + healthy:
datapacket-talos@77fba892b — civitai-app-sysredis namespace with
RedisReplication(1+2) + RedisSentinel(3, quorum 2) on nodes
fug-1v0/wjh-tgy/48r-b3a, all 3 sentinels in quorum agreement.
Refs:
datapacket-talos/claudedocs/sysredis-ha-migration-runbook.md (Phase 1)
datapacket-talos/claudedocs/sysredis-operator-discovery-2026-05-18.md
datapacket-talos/claudedocs/sysredis-ha-handoff-2026-05-27.md
* redis: address PR #2331 audit findings (mymaster default, scanIterator, topology-change logging)
- env/server-schema.ts: drop `REDIS_SYS_SENTINEL_NAME` default of `mymaster`
(live cluster uses `sysmaster`) and add a `superRefine` that rejects boot
when `REDIS_SYS_SENTINELS` is set without `REDIS_SYS_SENTINEL_NAME`. The
old default was a landmine — a missing var silently produced a Sentinel
that never resolved a master. (audit Fix 1)
- server/redis/client.ts: fix `scanIteratorWrapper` TypeError on Sentinel
mode. `RedisSentinel` exposes the SCAN command via WithCommands but not
the JS-only `scanIterator` helper, so `originalScanIterator(options)`
was calling `undefined(...)`. Detect sentinel via `getMasterNode` +
absence of `scanIterator`, then replicate node-redis's scan loop
(`do { reply = await scan(cursor, options); yield reply.keys }
while (cursor !== '0')`). Caller in
`src/pages/api/internal/redis-sys-usage.ts` now works under Sentinel.
(audit Fix 2)
- server/redis/client.ts: add explicit `topology-change` and `client-error`
listeners on the sentinel client (it doesn't emit
`connect`/`reconnecting`/`ready`, so failovers were invisible in Loki).
Logged via the existing `log()` helper to match the file's style.
(audit Fix 3)
- server/redis/client.ts: flip `passthroughClientErrorEvents` to `false`
so one flapping sentinel/replica pod can't flood the top-level `error`
listener — diagnostics now come from the new `client-error` listener.
(audit H4)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* redis: round-2 audit fixes (pingInterval parity, .env-example sysmaster)
- nodeClientOptions now passes the same pingInterval as the standalone
path. Sentinel sub-clients are long-lived against each master/replica
pod; without a heartbeat, idle connections through any intermediate
proxy or a rolling sentinel-pod restart can silently expire and only
surface as latency on the next sysRedis call.
- .env-example REDIS_SYS_SENTINEL_NAME comment now suggests sysmaster
(production) instead of mymaster. After the round-1 superRefine fix
the schema rejects boot if the example value is copy-pasted as-is.
* redis: round-3 audit fixes (masterPoolSize HOL, log payload, prom counters)
- masterPoolSize 1→2: serialized writes through one TCP connection caused
head-of-line blocking — a slow EVAL (PR #2332's atomic helper) could queue
the periodic PING heartbeat behind it and trip the readiness probe.
- topology-change + client-error log payloads now destructure event.node.host
and event.node.port so Loki regex can extract per-pod identifiers during a
multi-pod sentinel flap.
- New Prometheus counters: civitai_sysredis_sentinel_topology_changes_total
and civitai_sysredis_sentinel_client_errors_total, labeled {type,host,deployment}.
Phase 4 cutover SRE-on-call needs these to confirm failovers vs steady state.
Tests cover listener wiring, log string shape, counter increments, and
null-event safety.
* prom/redis: drop TDZ-fragile getters in __civitaiRedisMetrics publish
The 2026-06-15 rebase merge introduced getter-based exposure of the two
sysRedis sentinel counters on globalThis to defer capture past the
source-order TDZ. The Round-4 audit caught that any getRedisMetrics()
lookup fired during prom/client.ts top-level eval — before line ~410
where the consts are declared — would throw ReferenceError, which the
no-op fallback at redis/client.ts:458 (`metrics?.X ?? noopCounter`)
does NOT catch (optional chaining only swallows null/undefined).
No eager reader exists today, but this is one feature-flag-gated
startup probe away from a latent TDZ bomb. Fix: move the publish
block below the four const declarations so direct value capture
works, drop the getters.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-16 13:02:21 -05:00
|
|
|
# Optional: switch the `system` redis client to Sentinel mode. When set, REDIS_SYS_URL
|
|
|
|
|
# is still parsed for credentials but the connection is established via Sentinel.
|
|
|
|
|
# REDIS_SYS_SENTINELS=localhost:26379,localhost:26380,localhost:26381
|
|
|
|
|
# REDIS_SYS_SENTINEL_NAME=sysmaster # production uses "sysmaster"; match the master group your Sentinel CR declares
|
2022-10-11 16:56:51 -04:00
|
|
|
|
2023-01-27 15:34:58 -07:00
|
|
|
# Logging
|
2024-12-16 20:48:31 +00:00
|
|
|
LOGGING=prisma:error,prisma:warn,seed-metrics-search
|
2023-01-27 15:34:58 -07:00
|
|
|
|
2022-10-11 16:56:51 -04:00
|
|
|
# Next Auth
|
2023-02-21 17:44:55 -04:00
|
|
|
NEXTAUTH_SECRET=thisisnotasecret
|
2022-10-11 16:56:51 -04:00
|
|
|
NEXTAUTH_URL=http://localhost:3000
|
|
|
|
|
|
|
|
|
|
# Next Auth Discord Provider
|
|
|
|
|
DISCORD_CLIENT_ID=
|
|
|
|
|
DISCORD_CLIENT_SECRET=
|
2022-10-13 10:26:04 -04:00
|
|
|
|
|
|
|
|
# Next Auth GitHub Provider
|
2022-10-12 16:11:27 -06:00
|
|
|
GITHUB_CLIENT_ID=
|
|
|
|
|
GITHUB_CLIENT_SECRET=
|
2022-10-13 10:26:04 -04:00
|
|
|
|
|
|
|
|
# Next Auth Google Provider
|
2022-10-12 16:11:27 -06:00
|
|
|
GOOGLE_CLIENT_ID=
|
|
|
|
|
GOOGLE_CLIENT_SECRET=
|
2022-10-18 19:58:05 -04:00
|
|
|
|
2022-11-24 09:16:53 -04:00
|
|
|
# Next Auth Reddit Provider
|
|
|
|
|
REDDIT_CLIENT_ID=
|
|
|
|
|
REDDIT_CLIENT_SECRET=
|
|
|
|
|
|
2023-03-03 05:22:43 -07:00
|
|
|
# Integrations
|
|
|
|
|
DISCORD_BOT_TOKEN=
|
2023-05-24 10:28:22 -06:00
|
|
|
DISCORD_GUILD_ID=
|
2026-04-01 16:24:47 -04:00
|
|
|
DISCORD_WEBHOOK_MOD_ALERTS=
|
2023-03-03 05:22:43 -07:00
|
|
|
|
2022-10-18 19:58:05 -04:00
|
|
|
# File uploading
|
2023-02-28 04:49:51 +08:00
|
|
|
S3_UPLOAD_KEY=REFER_TO_README
|
|
|
|
|
S3_UPLOAD_SECRET=REFER_TO_README
|
|
|
|
|
S3_UPLOAD_BUCKET=modelshare
|
|
|
|
|
S3_UPLOAD_REGION=us-east-1
|
2024-11-04 15:53:30 -04:00
|
|
|
S3_UPLOAD_ENDPOINT=http://127.0.0.1:9000
|
2023-05-24 10:28:22 -06:00
|
|
|
|
2024-10-30 18:07:05 -04:00
|
|
|
# Image uploading
|
|
|
|
|
S3_IMAGE_UPLOAD_KEY=
|
|
|
|
|
S3_IMAGE_UPLOAD_SECRET=
|
|
|
|
|
S3_IMAGE_UPLOAD_BUCKET=images
|
2024-11-04 15:53:30 -04:00
|
|
|
S3_IMAGE_UPLOAD_REGION=us-east-1
|
|
|
|
|
S3_IMAGE_UPLOAD_ENDPOINT=http://127.0.0.1:9000
|
2024-10-30 18:07:05 -04:00
|
|
|
S3_IMAGE_CACHE_BUCKET=cache
|
|
|
|
|
S3_IMAGE_UPLOAD_OVERRIDE=
|
2023-02-21 17:44:55 -04:00
|
|
|
|
2024-11-05 12:53:27 -05:00
|
|
|
# Client env vars
|
2024-11-05 11:22:09 -04:00
|
|
|
NEXT_PUBLIC_IMAGE_LOCATION=http://localhost:3000
|
2023-03-02 16:14:23 -05:00
|
|
|
NEXT_PUBLIC_CONTENT_DECTECTION_LOCATION=https://publicstore.civitai.com/content_detection/model.json
|
2024-11-01 11:14:39 -04:00
|
|
|
NEXT_PUBLIC_CIVITAI_LINK=http://localhost:3000
|
2024-10-30 18:07:05 -04:00
|
|
|
NEXT_PUBLIC_UI_CATEGORY_VIEWS=false
|
|
|
|
|
NEXT_PUBLIC_UI_HOMEPAGE_IMAGES=false
|
|
|
|
|
NEXT_PUBLIC_ADS=true
|
2022-11-11 11:42:30 -07:00
|
|
|
|
2024-10-30 18:07:05 -04:00
|
|
|
# Clickhouse
|
|
|
|
|
CLICKHOUSE_HOST=http://localhost:18123
|
|
|
|
|
CLICKHOUSE_USERNAME=default
|
2024-10-31 11:41:05 -04:00
|
|
|
CLICKHOUSE_PASSWORD=
|
2024-10-30 18:07:05 -04:00
|
|
|
CLICKHOUSE_TRACKER_URL=http://localhost:3000
|
2023-04-21 17:13:15 +01:00
|
|
|
|
2023-01-02 15:59:36 -07:00
|
|
|
# Email
|
2023-03-02 16:14:23 -05:00
|
|
|
EMAIL_HOST=localhost
|
2023-03-12 17:40:58 -06:00
|
|
|
EMAIL_PORT=1025
|
2023-01-02 15:59:36 -07:00
|
|
|
EMAIL_USER=
|
|
|
|
|
EMAIL_PASS=
|
|
|
|
|
EMAIL_FROM=
|
|
|
|
|
|
2022-11-11 11:42:30 -07:00
|
|
|
# Endpoint Protection
|
2024-10-30 18:07:05 -04:00
|
|
|
JOB_TOKEN=thisisnotatoken
|
|
|
|
|
WEBHOOK_TOKEN=thisisnotatoken
|
2022-12-20 13:22:13 -07:00
|
|
|
|
|
|
|
|
# Site Configuration
|
|
|
|
|
UNAUTHENTICATED_DOWNLOAD=true
|
2023-02-16 15:18:48 -07:00
|
|
|
UNAUTHENTICATED_LIST_NSFW=false
|
|
|
|
|
SHOW_SFW_IN_NSFW=false
|
|
|
|
|
MAINTENANCE_MODE=false
|
2023-03-12 17:40:58 -06:00
|
|
|
RATE_LIMITING=true
|
2023-05-24 10:28:22 -06:00
|
|
|
TRPC_ORIGINS=
|
2022-11-11 14:22:36 -07:00
|
|
|
|
|
|
|
|
# Security
|
2022-12-20 13:22:13 -07:00
|
|
|
SCANNING_ENDPOINT=http://scan-me.civitai.com/enqueue
|
2023-02-21 17:44:55 -04:00
|
|
|
SCANNING_TOKEN=thisisnotatoken
|
|
|
|
|
|
2023-03-24 20:13:18 +00:00
|
|
|
# Delivery worker
|
|
|
|
|
DELIVERY_WORKER_ENDPOINT=https://delivery-worker.civitai.com/download
|
|
|
|
|
DELIVERY_WORKER_TOKEN=thisisnotatoken
|
|
|
|
|
|
2023-02-21 17:44:55 -04:00
|
|
|
# Payments
|
2024-11-04 15:53:30 -04:00
|
|
|
PADDLE_SECRET_KEY=thisisnotasecret
|
|
|
|
|
PADDLE_WEBHOOK_SECRET=thisisnotasecret
|
|
|
|
|
NEXT_PUBLIC_PADDLE_TOKEN=thisisnotatoken
|
|
|
|
|
NEXT_PUBLIC_DEFAULT_PAYMENT_PROVIDER=Paddle
|
2023-02-21 17:44:55 -04:00
|
|
|
|
|
|
|
|
# Features
|
|
|
|
|
FEATURE_FLAG_EARLY_ACCESS_MODEL=public
|
2023-07-07 12:09:01 -04:00
|
|
|
|
|
|
|
|
# MeiliSearch
|
2024-11-05 11:22:09 -04:00
|
|
|
SEARCH_HOST=http://localhost:7700
|
2024-10-30 18:07:05 -04:00
|
|
|
SEARCH_API_KEY=meilisearch
|
2024-11-05 11:22:09 -04:00
|
|
|
NEXT_PUBLIC_SEARCH_HOST=http://localhost:7700
|
2024-10-30 18:07:05 -04:00
|
|
|
NEXT_PUBLIC_SEARCH_CLIENT_KEY=meilisearch
|
2023-08-24 12:23:31 -04:00
|
|
|
|
2024-11-05 11:22:09 -04:00
|
|
|
METRICS_SEARCH_HOST=http://localhost:7700
|
2024-11-04 15:53:30 -04:00
|
|
|
METRICS_SEARCH_API_KEY=meilisearch
|
2023-09-19 11:24:53 -04:00
|
|
|
|
2026-07-20 10:51:46 -05:00
|
|
|
# Debounce window (ms) for flushing model-metric-affected ids into the model
|
|
|
|
|
# search-index update queue. Widening it collapses more of a hot model's repeated
|
|
|
|
|
# metric changes into a single reindex (cost: metric/popularity staleness lags by
|
|
|
|
|
# up to the window). Fail-soft: a bad value falls back to the 45m default. Requires
|
|
|
|
|
# a restart/rollout to take effect. Default 45m.
|
|
|
|
|
# SEARCH_INDEX_MODEL_METRIC_FLUSH_INTERVAL_MS=2700000
|
|
|
|
|
|
fix(meili): per-call timeout + per-backend concurrency limits + isolated health probe (#2351)
## Trigger
2026-05-29 cascade: civitai-dp-prod-api-primary lost 47 pods to kubelet
SIGKILL (Error exit=137) in 1h. Meilisearch backend returned 503
"service overloaded" at ~150/min; app calls into Meili had no timeout
(SDK + fetch defaults are unlimited), so hung calls accumulated request
contexts until the event loop was blocked enough that liveness TCP
probes timed out at 5s.
## Fix
New `withMeili(backend, fn)` wrapper in `src/server/meilisearch/client.ts`:
- Per-call timeout via Promise.race against MEILI_CALL_TIMEOUT_MS
(default 2500ms); typed `MeiliCallTimeoutError` on expiry.
- Per-pod, per-backend concurrency caps via two p-limit instances
(`search` + `metricsSearch`) at MEILI_CALL_CONCURRENCY (default 50).
Each backend has independent failure modes, so they should not
share a single limiter.
- Proxy-based client wrapping (`wrapMeilisearchClientWithLimiter`) so
only the Meili SDK call runs under the limiter — DB/Redis/ClickHouse
populate work no longer holds Meili semaphore slots or
false-attributes as Meili timeouts.
- Isolated `withMeiliHealthProbe()` with dedicated `pLimit(2)` so
user-traffic saturation cannot starve the kubelet probe (this would
otherwise re-introduce the cascade mechanism).
- Observability: `meili_call_timeouts_total`, `meili_call_active`,
`meili_call_queue_depth`, `meili_call_duration_seconds` — all
labeled by backend.
Wrapped call sites are the actual hot paths bleeding today:
- `getImagesFromSearchPreFilter` / `getImagesFromSearchPostFilter` /
`fetchMeiliUserOwnPass` — reached via tRPC `image.getInfinite` →
`getInfiniteImagesHandler` → `getAllImagesIndex`. Timeouts caught at
`getAllImagesIndex` and rethrown as `TRPCError code:'TIMEOUT'` (HTTP
408 in tRPC v10).
- `getImagesFromFeedSearch` — REST `/api/v1/images` (kept as
defense-in-depth).
- `searchMetrics` health check — wrapped under the isolated probe
limiter.
Background callers (`updateDocs` and friends) are intentionally
unwrapped — they have retries and slowness there does not block the
event loop.
## Concurrency budget
MEILI_CALL_CONCURRENCY=50 applies per backend, so a pod allows up to
50 concurrent calls to `search` + 50 to `metricsSearch` = 100 total
outbound Meili connections per pod. This is intentional.
## Risk
- User-facing: image feed endpoints now return HTTP 408 instead of
hanging under sustained Meili brownouts. Frontend `retry: 0` on
`image.getInfinite` means users see a hard error rather than a
silent retry; surface area is unchanged from the existing failure
mode (Traefik 504 at 30s), but the latency-to-error drops from 30s
to 2.5s.
- Backend: Proxy wrapping covers `search/searchGet/getDocument/
getDocuments` only — read paths in feed code. Write/configure paths
intentionally unwrapped.
## Rollback
Revert this commit. Both env vars (MEILI_CALL_TIMEOUT_MS,
MEILI_CALL_CONCURRENCY) have z.coerce.number().optional().default(...)
in server-schema.ts so removing them from the SOPS-encrypted ConfigMap
`civitai-cfg` (`clusters/production/apps/civitai-dp-prod/secrets/prod-env.enc.yaml`,
mounted via `envFrom: configMapRef`) leaves the app running on
defaults.
The infra-side probe relaxation (livenessProbe failureThreshold 6→12
on civitai-dp-prod-api-primary) is a separate concern and can stay or
be reverted independently.
## Deferred (fast-follow)
- I4: TRPCError TIMEOUT (408) is not retryable by default for REST API
consumers. Document for `/api/v1/images` users separately.
- I5: Background `updateDocs` retry storm (5× exponential backoff)
amplifies upstream saturation. Untouched here.
- N7: Flagger canary SLO may flag a rollout if canary pods produce
different error distributions than primary. Operational, not code.
- X5: `meili_call_duration_seconds` measures queue wait + execute
combined; split into `_wait_seconds` + `_execute_seconds` for cleaner
P99 diagnostics.
- X6: Timed-out callers leave orphan SDK promises holding closures
until backend RSTs. Acceptable for today's fast-503 mode; quantify
acceptable orphan count under TCP-timeout outage.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 13:45:53 -05:00
|
|
|
# Per-call Meilisearch timeout in ms. Calls wrapped via withMeili() fail fast
|
|
|
|
|
# with MeiliCallTimeoutError once exceeded, instead of hanging until Traefik's
|
|
|
|
|
# 30s router timeout fires.
|
|
|
|
|
MEILI_CALL_TIMEOUT_MS=2500
|
|
|
|
|
# Per-pod cap on in-flight Meilisearch calls wrapped via withMeili(). Excess
|
|
|
|
|
# calls fail fast with MeiliCallTimeoutError instead of queueing forever.
|
|
|
|
|
MEILI_CALL_CONCURRENCY=50
|
|
|
|
|
|
fix(meili): add per-backend circuit breaker + wrap fetchDocumentsAbortable (#2362)
* fix(meili): add per-backend circuit breaker + wrap fetchDocumentsAbortable
Hot-fix for the 2026-05-30 chronic Meili brownout. The wrap+timeout from
#2351/#2358/#2360 is still firing throughout 14h cascades — 50 concurrent
callers each waiting the full MEILI_CALL_TIMEOUT_MS (2500ms) before
failing accumulates ~125 worker-seconds of event-loop pressure per pod
per cycle, enough to block past kubelet's 5s TCP probe and SIGKILL.
Two surgical additions:
1. Per-backend circuit breaker (search + metricsSearch, independent).
State machine: CLOSED -> (>=THRESHOLD timeouts in WINDOW) -> OPEN
-> (cooldown elapsed) -> HALF_OPEN -> (trial success) -> CLOSED
or -> (trial fail) -> OPEN. While OPEN, withMeili() throws
MeiliCallTimeoutError at 0ms with no acquire, no setTimeout, no
backend request. healthProbe is intentionally NOT under the circuit
so the kubelet probe stays responsive and remains the canonical
liveness signal. Failures counted = MeiliCallTimeoutError only.
2. Wrap fetchDocumentsAbortable under runWithLimiter('metricsSearch', ...).
The raw-HTTP cancellable path used by getImagesFromSearch{Pre,Post}Filter
was bypassing both the limiter and (now) the circuit. Added a
{ useTimeout: false } option so the caller's AbortSignal stays the
deadline (no double-timeout race), but the semaphore slot + circuit
gate now apply.
New env (defaults in parentheses):
MEILI_CIRCUIT_TRIP_THRESHOLD (10)
MEILI_CIRCUIT_WINDOW_SECONDS (30)
MEILI_CIRCUIT_COOLDOWN_SECONDS (30)
New metrics:
civitai_app_meili_circuit_state{backend=...} (0=closed, 1=half-open, 2=open)
civitai_app_meili_circuit_trips_total{backend=...} (counter)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(meili): audit follow-up — separate circuit-rejection counter + env validation
Two pre-merge fixes from PR #2362 audit:
C1 (counter conflation): split circuit-open rejections out of
meili_call_timeouts_total into a new meili_circuit_rejections_total
counter. Existing alerts keyed on rate(meili_call_timeouts_total[1m])
would have falsely escalated during OPEN state because rejections
accumulate at request-arrival rate (potentially 100× the real timeout
rate). The two counters' semantics are now disjoint; operators can sum
them to get "all fast-fail events."
I4 (env validation): MEILI_CIRCUIT_TRIP_THRESHOLD / WINDOW_SECONDS /
COOLDOWN_SECONDS — add .int().min(1) so a blank env value (coerces to
NaN under z.coerce.number()) or accidental zero is rejected at boot
rather than silently disabling the breaker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 09:09:08 -05:00
|
|
|
# Per-backend circuit breaker. If MEILI_CIRCUIT_TRIP_THRESHOLD wrapped-call
|
|
|
|
|
# timeouts accumulate within MEILI_CIRCUIT_WINDOW_SECONDS on a backend, the
|
|
|
|
|
# circuit OPENs and all calls fail at 0ms for MEILI_CIRCUIT_COOLDOWN_SECONDS,
|
|
|
|
|
# then HALF_OPEN issues a single trial request. healthProbe is excluded.
|
|
|
|
|
MEILI_CIRCUIT_TRIP_THRESHOLD=10
|
|
|
|
|
MEILI_CIRCUIT_WINDOW_SECONDS=30
|
|
|
|
|
MEILI_CIRCUIT_COOLDOWN_SECONDS=30
|
|
|
|
|
|
2024-10-30 18:07:05 -04:00
|
|
|
# BaseURL
|
2023-09-19 11:24:53 -04:00
|
|
|
NEXT_PUBLIC_BASE_URL=http://localhost:3000
|
2024-01-16 10:36:46 -04:00
|
|
|
|
|
|
|
|
# Recaptcha
|
|
|
|
|
RECAPTCHA_PROJECT_ID=aSampleKey
|
2024-02-06 10:56:12 -04:00
|
|
|
NEXT_PUBLIC_RECAPTCHA_KEY=aSampleKey
|
|
|
|
|
|
2024-09-05 16:29:58 -04:00
|
|
|
# CF Turnstile
|
2024-10-30 18:07:05 -04:00
|
|
|
NEXT_PUBLIC_CLOUDFLARE_TURNSTILE_SITEKEY=1x00000000000000000000BB
|
|
|
|
|
CLOUDFLARE_TURNSTILE_SECRET=1x0000000000000000000000000000000AA
|
|
|
|
|
|
2024-09-12 16:24:57 -04:00
|
|
|
NEXT_PUBLIC_CF_INVISIBLE_TURNSTILE_SITEKEY=1x00000000000000000000BB
|
|
|
|
|
CF_INVISIBLE_TURNSTILE_SECRET=1x0000000000000000000000000000000AA
|
|
|
|
|
|
|
|
|
|
NEXT_PUBLIC_CF_MANAGED_TURNSTILE_SITEKEY=1x00000000000000000000AA
|
|
|
|
|
CF_MANAGED_TURNSTILE_SECRET=1x0000000000000000000000000000000AA
|
2024-11-05 12:53:27 -05:00
|
|
|
|
|
|
|
|
ORCHESTRATOR_ENDPOINT=http://localhost
|
|
|
|
|
ORCHESTRATOR_ACCESS_TOKEN=asdf
|
2025-03-12 10:49:38 -04:00
|
|
|
|
|
|
|
|
BUZZ_ENDPOINT=http://localhost
|
|
|
|
|
SIGNALS_ENDPOINT=http://localhost
|
|
|
|
|
NEXT_PUBLIC_SIGNALS_ENDPOINT=http://localhost
|
2025-05-29 10:37:27 -04:00
|
|
|
|
|
|
|
|
NOW_PAYMENTS_API_URL=http://localhost
|
|
|
|
|
NOW_PAYMENTS_API_KEY=key
|
|
|
|
|
NOW_PAYMENTS_IPN_KEY=key
|
|
|
|
|
|
|
|
|
|
COINBASE_API_URL=http://localhost
|
|
|
|
|
COINBASE_API_KEY=key
|
|
|
|
|
COINBASE_WEBHOOK_SECRET=secret
|
2025-07-08 14:50:00 -04:00
|
|
|
|
|
|
|
|
EMERCHANTPAY_WPF_URL=
|
|
|
|
|
EMERCHANTPAY_USERNAME=
|
|
|
|
|
EMERCHANTPAY_PASSWORD=
|
2025-09-16 14:33:24 -04:00
|
|
|
|
feat(merch): Blue Buzz rewards for Shopify merch purchases (#2824)
* feat(merch): Blue Buzz rewards for Shopify merch purchases
Reward Blue Buzz when a customer buys merch on shop.civitai.com (Shopify). An
orders/paid webhook records each order and grants buzz at 250 Blue Buzz/$1
(coupon-boostable via a multiplier map). Grants are idempotent on the Shopify
order id (externalTransactionId).
Identity is resolved with a one-time post-purchase claim: if the order email
matches the user's verified Civitai email it grants instantly; otherwise the
user asserts the order email and confirms ownership via a signed link emailed
to that address. After the first successful claim the Shopify customer is
linked to the Civitai account, so future orders auto-grant with no claim step.
- ShopifyCustomerLink + ShopifyMerchOrder tables (migration; apply manually)
- HMAC-verified webhook at /api/webhooks/shopify (orders/paid)
- merch tRPC router + claim page at /merch/claim
- per-user Redis rate limiting on claim attempts (fail-open)
- SHOPIFY_* env vars; Shopify never grants buzz, it only hands off the order id
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(merch): write civitai.user_id Shopify customer metafield on link
When a Shopify customer is linked to a Civitai account (first claim), stamp a
`civitai.user_id` metafield onto the Shopify customer via the Admin API. The
order-status page gates the claim prompt on `{% unless customer.metafields.civitai.user_id %}`
so linked customers — who auto-redeem from then on — never see it again.
Best-effort and fail-soft: a metafield write failure never blocks the Buzz
grant (the DB link already drives auto-redeem). No-op unless SHOPIFY_SHOP_DOMAIN
+ SHOPIFY_ADMIN_TOKEN are set.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(merch): Shopify client_credentials auth + GraphQL metafield write
The merch store's Shopify app uses the client_credentials grant rather than a
static token. Mint a short-lived (~24h) admin token from SHOPIFY_CLIENT_ID +
SHOPIFY_CLIENT_SECRET, cache it in-process, and re-mint on expiry. A static
SHOPIFY_ADMIN_TOKEN still takes precedence if set.
Switch the customer metafield write to the GraphQL Admin API (2025-01)
metafieldsSet mutation (upsert by owner+namespace+key). Still fail-soft.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(merch): email the buyer a claim link on unlinked orders
shop.civitai.com is on checkout extensibility, so the order-status page can't
host the claim button. Instead, processShopifyOrderPaid emails the buyer a
"Claim your Blue Buzz" link the first time it sees an unlinked order. Once they
claim, the customer is linked and future orders auto-grant with no email.
Retry-safe: the invite only sends on the first insert of an order (guarded by an
existence check before upsert), and is skipped for zero-buzz or email-less
orders. Removes the Shopify-side Liquid/UI-extension dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(merch): gapless claim via signed invite key
The webhook claim email now links to /merch/claim?key=<signed>, where the key
is an HMAC-signed order id (signOrderKey, NEXTAUTH_SECRET, 90d exp). Because the
link was delivered to the order's email, holding a valid key is itself proof of
mailbox ownership — so claimMerchOrderByKey links whatever Civitai account the
clicker is signed into and grants immediately, with no email-match and no
confirmation step. Collapses the entire mismatch sub-flow for the email path.
The unsigned ?order=<id> path (manual entry / optional classic-checkout snippet)
still uses email-match-or-confirm since it carries no proof.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(merch): receipt email on auto-credited (already-linked) orders
Previously only unlinked orders emailed the buyer (the claim invite). When a
Shopify customer is already linked, we now also send a receipt naming the
Civitai account that was credited (merchBuzzCreditedEmail, no claim link), so
they get confirmation and can verify the Buzz landed on the right account.
Same first-insert guard as the invite, so webhook retries don't re-send.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* copy(merch): credited receipt — drop CTA, point to hello@civitai.com
It's sent from a no-reply address, so swap "reply to this email" for "contact
hello@civitai.com", and remove the CTA button (the receipt needs no action).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(merch): drop dead unsigned-claim path; harden webhook
The signed-key email is the only claim entry point now, so the earlier
interactive/unsigned path is dead. Remove it (also closes two security-review
findings by deletion):
- procedures getClaimableOrder/claim/requestEmailConfirmation/confirmClaim
- service getClaimableMerchOrder/claimMerchOrder/requestMerchClaimConfirmation/
confirmMerchClaim, plus signClaimToken/verifyClaimToken (the second token type,
finding: no domain separation) and getVerifiedUserEmail/maskEmail
- getClaimableOrder was also an order-existence oracle (enumeration finding)
- merchClaimConfirmation email template; the ?order=/?token= claim-page routes
(page is now ?key= only)
Harden the webhook path:
- skip Shopify test-mode orders (order.test) so they never grant real Buzz
- cap per-order Buzz (MERCH_BUZZ_MAX_PER_ORDER) against a malformed subtotal
merch router is now a single claimByKey procedure. Typechecks clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(merch): final audit nits — fix stale ?order example URL + doc drift
- merchClaimInvite testData uses ?key= (the live claim param), not the removed ?order=
- doc build-status: claimMerchOrderByKey (not the removed claimMerchOrder), migration
noted as applied, buzz cap noted
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 11:05:14 -06:00
|
|
|
# Shopify merch store — Blue Buzz reward loop
|
|
|
|
|
# SHOPIFY_SHOP_DOMAIN = the *.myshopify.com admin domain (e.g. ff1592-5.myshopify.com)
|
|
|
|
|
SHOPIFY_SHOP_DOMAIN=
|
|
|
|
|
SHOPIFY_WEBHOOK_SECRET=
|
|
|
|
|
# Admin auth: client_credentials grant (preferred). Set ADMIN_TOKEN instead only for a static token.
|
|
|
|
|
SHOPIFY_CLIENT_ID=
|
|
|
|
|
SHOPIFY_CLIENT_SECRET=
|
|
|
|
|
SHOPIFY_ADMIN_TOKEN=
|
|
|
|
|
|
2025-09-04 13:04:08 -05:00
|
|
|
FLIPT_URL=""
|
|
|
|
|
FLIPT_FETCHER_SECRET=placeholder
|
2025-11-25 15:26:53 -04:00
|
|
|
|
|
|
|
|
IMAGE_SCANNER_NEW=false
|
feat(app-blocks): make the cap-limit degrade path observable + rename the absolute-ceiling env knobs (#3528)
Two follow-ups to #3519 (per-app generation spend/velocity caps).
1) OBSERVABILITY of the degrade-to-strictest path.
`resolveAppCapLimits` falls back to STRICTEST_APP_CAP_LIMITS on a DB error
or a missing `app_blocks` row. That behaviour is right — never uncapped, and
a hard deny would turn a transient DB blip into a full generation outage —
but it was SILENT, and an app pinned to the strictest ceiling looks exactly
like an app that is merely busy. The first symptom would be that app's users
hitting abuse rejections they did not earn. (This is the same silent-
degradation shape as #3520, which is why it is worth closing here.)
Adds `civitai_app_block_cap_limits_degraded_total{reason}` —
`db_error` (the read threw: infra; every app degrades at once, page-worthy)
vs `missing_row` (the read succeeded and there is no such app: one app,
points at an id-minting bug) — plus a paired `console.warn` carrying the
specific appBlockId. Follows the existing convention in
src/server/metrics/app-block-runtime.metrics.ts (get-or-create against the
default registry + a fail-soft emit wrapper); no new mechanism.
- NO `app_block_id` prom label, deliberately. `missing_row` fires precisely
for ids absent from the app catalog, i.e. the unbounded population
known-app-blocks.service.ts exists to clamp, and prom-client retains every
distinct label set in the heap forever. The usual clamp needs a DB read —
the very thing broken on the `db_error` path. So: alert on the metric,
attribute from the log.
- NOT a failure path. The metric emit and the log are independently guarded
and `recordAppCapLimitsDegrade` is total on its own side, so neither a
broken registry nor a throwing console can perturb cap resolution.
- NOT on the hot path. Only a cache MISS that DEGRADED emits; a warm hit and
a miss that resolves a real row never reach it. Volume is bounded by the
5s fallback-TTL cache, not by submit rate — a 10k-submit burst against one
degraded app emits once.
2) RENAME the two absolute-ceiling env knobs.
`BLOCK_APP_SPEND_CAP_BUZZ_PER_DAY` / `BLOCK_APP_SPEND_VELOCITY_MAX_GENS`
used to BE the ceilings. Since #3519 they are absolute bounds that clamp the
tier table AND any per-app moderator override, so an operator reaching for
`..._VELOCITY_MAX_GENS` mid-incident would reasonably read it as "set the
limit" rather than "bound it".
BLOCK_APP_SPEND_CAP_BUZZ_PER_DAY -> BLOCK_APP_SPEND_ABSOLUTE_MAX_BUZZ_PER_DAY
BLOCK_APP_SPEND_VELOCITY_MAX_GENS -> BLOCK_APP_SPEND_ABSOLUTE_MAX_GENS_PER_WINDOW
The legacy names are still honoured (dp-prod sets neither, but other
environments are not enumerable from here, and silently ignoring a set spend
guardrail is unacceptable), with a deprecation warning that also states the
changed meaning. A valid new value always wins; a set-but-ignored or
set-but-unusable legacy value warns too. Exported symbols follow the env
names, with the old export names kept as deprecated aliases so pre-rename
importers keep compiling.
Tests: 41 new across two files. Every new guard was mutation-verified —
14 mutations, each killing a specific named test.
2026-08-01 18:37:27 -05:00
|
|
|
|
|
|
|
|
# App Blocks — per-app generation spend/velocity ABSOLUTE CEILINGS (incident knobs).
|
|
|
|
|
# 🔴 These are UPPER BOUNDS, not the limit an app receives. Each app's actual
|
|
|
|
|
# ceilings come from its server-owned `spendTier` (+ any moderator per-app
|
|
|
|
|
# override); these clamp the tier table AND any override from above, so setting
|
|
|
|
|
# one TIGHTENS every app and can never loosen one. Unset = no extra clamp.
|
|
|
|
|
# Formerly BLOCK_APP_SPEND_CAP_BUZZ_PER_DAY / BLOCK_APP_SPEND_VELOCITY_MAX_GENS —
|
|
|
|
|
# those names are DEPRECATED but still honoured (with a startup warning).
|
|
|
|
|
# BLOCK_APP_SPEND_ABSOLUTE_MAX_BUZZ_PER_DAY=
|
|
|
|
|
# BLOCK_APP_SPEND_ABSOLUTE_MAX_GENS_PER_WINDOW=
|
|
|
|
|
# Window (seconds) the gens-per-window ceiling is measured over. Default 60.
|
|
|
|
|
# BLOCK_APP_SPEND_VELOCITY_WINDOW_SECONDS=
|