feat(app-blocks): make the cap-limit degrade path observable + rename the absolute-ceiling env knobs (#3528)

Two follow-ups to #3519 (per-app generation spend/velocity caps).

1) OBSERVABILITY of the degrade-to-strictest path.

`resolveAppCapLimits` falls back to STRICTEST_APP_CAP_LIMITS on a DB error
or a missing `app_blocks` row. That behaviour is right — never uncapped, and
a hard deny would turn a transient DB blip into a full generation outage —
but it was SILENT, and an app pinned to the strictest ceiling looks exactly
like an app that is merely busy. The first symptom would be that app's users
hitting abuse rejections they did not earn. (This is the same silent-
degradation shape as #3520, which is why it is worth closing here.)

Adds `civitai_app_block_cap_limits_degraded_total{reason}` —
`db_error` (the read threw: infra; every app degrades at once, page-worthy)
vs `missing_row` (the read succeeded and there is no such app: one app,
points at an id-minting bug) — plus a paired `console.warn` carrying the
specific appBlockId. Follows the existing convention in
src/server/metrics/app-block-runtime.metrics.ts (get-or-create against the
default registry + a fail-soft emit wrapper); no new mechanism.

- NO `app_block_id` prom label, deliberately. `missing_row` fires precisely
  for ids absent from the app catalog, i.e. the unbounded population
  known-app-blocks.service.ts exists to clamp, and prom-client retains every
  distinct label set in the heap forever. The usual clamp needs a DB read —
  the very thing broken on the `db_error` path. So: alert on the metric,
  attribute from the log.
- NOT a failure path. The metric emit and the log are independently guarded
  and `recordAppCapLimitsDegrade` is total on its own side, so neither a
  broken registry nor a throwing console can perturb cap resolution.
- NOT on the hot path. Only a cache MISS that DEGRADED emits; a warm hit and
  a miss that resolves a real row never reach it. Volume is bounded by the
  5s fallback-TTL cache, not by submit rate — a 10k-submit burst against one
  degraded app emits once.

2) RENAME the two absolute-ceiling env knobs.

`BLOCK_APP_SPEND_CAP_BUZZ_PER_DAY` / `BLOCK_APP_SPEND_VELOCITY_MAX_GENS`
used to BE the ceilings. Since #3519 they are absolute bounds that clamp the
tier table AND any per-app moderator override, so an operator reaching for
`..._VELOCITY_MAX_GENS` mid-incident would reasonably read it as "set the
limit" rather than "bound it".

  BLOCK_APP_SPEND_CAP_BUZZ_PER_DAY   -> BLOCK_APP_SPEND_ABSOLUTE_MAX_BUZZ_PER_DAY
  BLOCK_APP_SPEND_VELOCITY_MAX_GENS  -> BLOCK_APP_SPEND_ABSOLUTE_MAX_GENS_PER_WINDOW

The legacy names are still honoured (dp-prod sets neither, but other
environments are not enumerable from here, and silently ignoring a set spend
guardrail is unacceptable), with a deprecation warning that also states the
changed meaning. A valid new value always wins; a set-but-ignored or
set-but-unusable legacy value warns too. Exported symbols follow the env
names, with the old export names kept as deprecated aliases so pre-rename
importers keep compiling.

Tests: 41 new across two files. Every new guard was mutation-verified —
14 mutations, each killing a specific named test.
This commit is contained in:
Zachary Lowden
2026-08-01 18:37:27 -05:00
committed by GitHub
parent 917036a1ec
commit 65bfb7ece8
7 changed files with 830 additions and 20 deletions
+12
View File
@@ -198,3 +198,15 @@ FLIPT_URL=""
FLIPT_FETCHER_SECRET=placeholder
IMAGE_SCANNER_NEW=false
# App Blocks — per-app generation spend/velocity ABSOLUTE CEILINGS (incident knobs).
# 🔴 These are UPPER BOUNDS, not the limit an app receives. Each app's actual
# ceilings come from its server-owned `spendTier` (+ any moderator per-app
# override); these clamp the tier table AND any override from above, so setting
# one TIGHTENS every app and can never loosen one. Unset = no extra clamp.
# Formerly BLOCK_APP_SPEND_CAP_BUZZ_PER_DAY / BLOCK_APP_SPEND_VELOCITY_MAX_GENS —
# those names are DEPRECATED but still honoured (with a startup warning).
# BLOCK_APP_SPEND_ABSOLUTE_MAX_BUZZ_PER_DAY=
# BLOCK_APP_SPEND_ABSOLUTE_MAX_GENS_PER_WINDOW=
# Window (seconds) the gens-per-window ceiling is measured over. Default 60.
# BLOCK_APP_SPEND_VELOCITY_WINDOW_SECONDS=