skills/civitai__civitai
1
0
Fork 0
mirror of https://github.com/civitai/civitai.git synced 2026-09-20 22:08:18 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
03aeb629f24b96fd1c7a1afb67404fbf2defd8ff
civitai__civitai/.env-example
T
Zachary Lowden 03aeb629f2 fix(meili): per-call timeout + per-backend concurrency limits + isolated health probe (#2351) 
## Trigger

2026-05-29 cascade: civitai-dp-prod-api-primary lost 47 pods to kubelet
SIGKILL (Error exit=137) in 1h. Meilisearch backend returned 503
"service overloaded" at ~150/min; app calls into Meili had no timeout
(SDK + fetch defaults are unlimited), so hung calls accumulated request
contexts until the event loop was blocked enough that liveness TCP
probes timed out at 5s.

## Fix

New `withMeili(backend, fn)` wrapper in `src/server/meilisearch/client.ts`:
- Per-call timeout via Promise.race against MEILI_CALL_TIMEOUT_MS
  (default 2500ms); typed `MeiliCallTimeoutError` on expiry.
- Per-pod, per-backend concurrency caps via two p-limit instances
  (`search` + `metricsSearch`) at MEILI_CALL_CONCURRENCY (default 50).
  Each backend has independent failure modes, so they should not
  share a single limiter.
- Proxy-based client wrapping (`wrapMeilisearchClientWithLimiter`) so
  only the Meili SDK call runs under the limiter — DB/Redis/ClickHouse
  populate work no longer holds Meili semaphore slots or
  false-attributes as Meili timeouts.
- Isolated `withMeiliHealthProbe()` with dedicated `pLimit(2)` so
  user-traffic saturation cannot starve the kubelet probe (this would
  otherwise re-introduce the cascade mechanism).
- Observability: `meili_call_timeouts_total`, `meili_call_active`,
  `meili_call_queue_depth`, `meili_call_duration_seconds` — all
  labeled by backend.

Wrapped call sites are the actual hot paths bleeding today:
- `getImagesFromSearchPreFilter` / `getImagesFromSearchPostFilter` /
  `fetchMeiliUserOwnPass` — reached via tRPC `image.getInfinite` →
  `getInfiniteImagesHandler` → `getAllImagesIndex`. Timeouts caught at
  `getAllImagesIndex` and rethrown as `TRPCError code:'TIMEOUT'` (HTTP
  408 in tRPC v10).
- `getImagesFromFeedSearch` — REST `/api/v1/images` (kept as
  defense-in-depth).
- `searchMetrics` health check — wrapped under the isolated probe
  limiter.

Background callers (`updateDocs` and friends) are intentionally
unwrapped — they have retries and slowness there does not block the
event loop.

## Concurrency budget

MEILI_CALL_CONCURRENCY=50 applies per backend, so a pod allows up to
50 concurrent calls to `search` + 50 to `metricsSearch` = 100 total
outbound Meili connections per pod. This is intentional.

## Risk

- User-facing: image feed endpoints now return HTTP 408 instead of
  hanging under sustained Meili brownouts. Frontend `retry: 0` on
  `image.getInfinite` means users see a hard error rather than a
  silent retry; surface area is unchanged from the existing failure
  mode (Traefik 504 at 30s), but the latency-to-error drops from 30s
  to 2.5s.
- Backend: Proxy wrapping covers `search/searchGet/getDocument/
  getDocuments` only — read paths in feed code. Write/configure paths
  intentionally unwrapped.

## Rollback

Revert this commit. Both env vars (MEILI_CALL_TIMEOUT_MS,
MEILI_CALL_CONCURRENCY) have z.coerce.number().optional().default(...)
in server-schema.ts so removing them from the SOPS-encrypted ConfigMap
`civitai-cfg` (`clusters/production/apps/civitai-dp-prod/secrets/prod-env.enc.yaml`,
mounted via `envFrom: configMapRef`) leaves the app running on
defaults.

The infra-side probe relaxation (livenessProbe failureThreshold 6→12
on civitai-dp-prod-api-primary) is a separate concern and can stay or
be reverted independently.

## Deferred (fast-follow)

- I4: TRPCError TIMEOUT (408) is not retryable by default for REST API
  consumers. Document for `/api/v1/images` users separately.
- I5: Background `updateDocs` retry storm (5× exponential backoff)
  amplifies upstream saturation. Untouched here.
- N7: Flagger canary SLO may flag a rollout if canary pods produce
  different error distributions than primary. Operational, not code.
- X5: `meili_call_duration_seconds` measures queue wait + execute
  combined; split into `_wait_seconds` + `_execute_seconds` for cleaner
  P99 diagnostics.
- X6: Timed-out callers leave orphan SDK promises holding closures
  until backend RSTs. Acceptable for today's fast-503 mode; quantify
  acceptable orphan count under TCP-timeout outage.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 13:45:53 -05:00

173 lines
4.8 KiB
Plaintext
Raw Blame History

# Since .env is gitignored, you can use .env-example to build a new `.env` file when you clone the repo.
# Keep this file up-to-date when you add new variables to `.env`.
# This file will be committed to version control, so make sure not to have any secrets in it.
# If you are cloning this repo, create a copy of this file named `.env` and populate it with your secrets.
# When adding additional env variables, the schema in /env/schema.mjs should be updated accordingly
# The default values for Prisma, Redis, S3, and Email are set to work with the docker-compose setup
# Database
DATABASE_SSL=false
DATABASE_URL=postgresql://postgres:postgres@localhost:15432/civitai
DATABASE_REPLICA_URL=postgresql://postgres:postgres@localhost:15432/civitai
NOTIFICATION_DB_URL=postgresql://postgres:postgres@localhost:15434/postgres
NOTIFICATION_DB_REPLICA_URL=postgresql://postgres:postgres@localhost:15434/postgres
DATAPACKET_DATABASE_RO_URL=postgresql://postgres:postgres@localhost:15435/postgres
# Redis
REDIS_URL=redis://:redis@localhost:6379
REDIS_SYS_URL=redis://:redis@localhost:6378
# Logging
LOGGING=prisma:error,prisma:warn,seed-metrics-search
# Next Auth
NEXTAUTH_SECRET=thisisnotasecret
NEXTAUTH_URL=http://localhost:3000
# Next Auth Discord Provider
DISCORD_CLIENT_ID=
DISCORD_CLIENT_SECRET=
# Next Auth GitHub Provider
GITHUB_CLIENT_ID=
GITHUB_CLIENT_SECRET=
# Next Auth Google Provider
GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=
# Next Auth Reddit Provider
REDDIT_CLIENT_ID=
REDDIT_CLIENT_SECRET=
# Integrations
DISCORD_BOT_TOKEN=
DISCORD_GUILD_ID=
DISCORD_WEBHOOK_MOD_ALERTS=
# File uploading
S3_UPLOAD_KEY=REFER_TO_README
S3_UPLOAD_SECRET=REFER_TO_README
S3_UPLOAD_BUCKET=modelshare
S3_UPLOAD_REGION=us-east-1
S3_UPLOAD_ENDPOINT=http://127.0.0.1:9000
# Image uploading
S3_IMAGE_UPLOAD_KEY=
S3_IMAGE_UPLOAD_SECRET=
S3_IMAGE_UPLOAD_BUCKET=images
S3_IMAGE_UPLOAD_REGION=us-east-1
S3_IMAGE_UPLOAD_ENDPOINT=http://127.0.0.1:9000
S3_IMAGE_CACHE_BUCKET=cache
S3_IMAGE_UPLOAD_OVERRIDE=
# Client env vars
NEXT_PUBLIC_IMAGE_LOCATION=http://localhost:3000
NEXT_PUBLIC_CONTENT_DECTECTION_LOCATION=https://publicstore.civitai.com/content_detection/model.json
NEXT_PUBLIC_CIVITAI_LINK=http://localhost:3000
NEXT_PUBLIC_UI_CATEGORY_VIEWS=false
NEXT_PUBLIC_UI_HOMEPAGE_IMAGES=false
NEXT_PUBLIC_ADS=true
# Clickhouse
CLICKHOUSE_HOST=http://localhost:18123
CLICKHOUSE_USERNAME=default
CLICKHOUSE_PASSWORD=
CLICKHOUSE_TRACKER_URL=http://localhost:3000
# Email
EMAIL_HOST=localhost
EMAIL_PORT=1025
EMAIL_USER=
EMAIL_PASS=
EMAIL_FROM=
# Endpoint Protection
JOB_TOKEN=thisisnotatoken
WEBHOOK_TOKEN=thisisnotatoken
# Site Configuration
UNAUTHENTICATED_DOWNLOAD=true
UNAUTHENTICATED_LIST_NSFW=false
SHOW_SFW_IN_NSFW=false
MAINTENANCE_MODE=false
RATE_LIMITING=true
TRPC_ORIGINS=
# Security
SCANNING_ENDPOINT=http://scan-me.civitai.com/enqueue
SCANNING_TOKEN=thisisnotatoken
# Delivery worker
DELIVERY_WORKER_ENDPOINT=https://delivery-worker.civitai.com/download
DELIVERY_WORKER_TOKEN=thisisnotatoken
# Payments
PADDLE_SECRET_KEY=thisisnotasecret
PADDLE_WEBHOOK_SECRET=thisisnotasecret
NEXT_PUBLIC_PADDLE_TOKEN=thisisnotatoken
NEXT_PUBLIC_DEFAULT_PAYMENT_PROVIDER=Paddle
# Features
FEATURE_FLAG_EARLY_ACCESS_MODEL=public
# MeiliSearch
SEARCH_HOST=http://localhost:7700
SEARCH_API_KEY=meilisearch
NEXT_PUBLIC_SEARCH_HOST=http://localhost:7700
NEXT_PUBLIC_SEARCH_CLIENT_KEY=meilisearch
METRICS_SEARCH_HOST=http://localhost:7700
METRICS_SEARCH_API_KEY=meilisearch
# Per-call Meilisearch timeout in ms. Calls wrapped via withMeili() fail fast
# with MeiliCallTimeoutError once exceeded, instead of hanging until Traefik's
# 30s router timeout fires.
MEILI_CALL_TIMEOUT_MS=2500
# Per-pod cap on in-flight Meilisearch calls wrapped via withMeili(). Excess
# calls fail fast with MeiliCallTimeoutError instead of queueing forever.
MEILI_CALL_CONCURRENCY=50
# BaseURL
NEXT_PUBLIC_BASE_URL=http://localhost:3000
# Recaptcha
RECAPTCHA_PROJECT_ID=aSampleKey
NEXT_PUBLIC_RECAPTCHA_KEY=aSampleKey
# CF Turnstile
NEXT_PUBLIC_CLOUDFLARE_TURNSTILE_SITEKEY=1x00000000000000000000BB
CLOUDFLARE_TURNSTILE_SECRET=1x0000000000000000000000000000000AA
NEXT_PUBLIC_CF_INVISIBLE_TURNSTILE_SITEKEY=1x00000000000000000000BB
CF_INVISIBLE_TURNSTILE_SECRET=1x0000000000000000000000000000000AA
NEXT_PUBLIC_CF_MANAGED_TURNSTILE_SITEKEY=1x00000000000000000000AA
CF_MANAGED_TURNSTILE_SECRET=1x0000000000000000000000000000000AA
ORCHESTRATOR_ENDPOINT=http://localhost
ORCHESTRATOR_ACCESS_TOKEN=asdf
BUZZ_ENDPOINT=http://localhost
SIGNALS_ENDPOINT=http://localhost
NEXT_PUBLIC_SIGNALS_ENDPOINT=http://localhost
NOW_PAYMENTS_API_URL=http://localhost
NOW_PAYMENTS_API_KEY=key
NOW_PAYMENTS_IPN_KEY=key
COINBASE_API_URL=http://localhost
COINBASE_API_KEY=key
COINBASE_WEBHOOK_SECRET=secret
EMERCHANTPAY_WPF_URL=
EMERCHANTPAY_USERNAME=
EMERCHANTPAY_PASSWORD=
FLIPT_URL=""
FLIPT_FETCHER_SECRET=placeholder
IMAGE_SCANNER_NEW=false
Reference in New Issue View Git Blame Copy Permalink