redis: add Sentinel client branch for system Redis (Phase 1, HA migration) (#2331)

* redis: add Sentinel client branch for system Redis (Phase 1 of HA migration)

Add an optional `createSentinel(...)` code path for the `system` Redis
client, gated on a new optional REDIS_SYS_SENTINELS env var. When unset
(default), the existing REDIS_SYS_URL path is used and behavior is
unchanged in any deployed environment — zero risk to ship ahead of the
infra-side cutover.

When REDIS_SYS_SENTINELS is set, the client uses node-redis v5's
Sentinel API:
  REDIS_SYS_SENTINELS=civitai-app-sysredis-sentinel.civitai-app-sysredis.svc.cluster.local:26379
  REDIS_SYS_SENTINEL_NAME=sysmaster

The credential currently extracted from REDIS_SYS_URL (authConfig.password)
is reused — the infra-side HA cluster is provisioned with the same
password, so no separate REDIS_SYS_PASSWORD env var is needed.

Falls back to the existing single-node createClient on every env where
REDIS_SYS_SENTINELS isn't explicitly set, so this PR is safe to merge
ahead of the per-pool env-var flip during Phase 4 cutover.

Phase 1.5 (atomic HEXPIRE NX helper) and Phase 1.6 (`getOrchestratorToken`
cold-mint cache) are pre-HA-cutover hard prereqs but are tracked as
separate PRs to keep this one small and easy to verify.

Infra side already live + healthy:
  datapacket-talos@77fba892b — civitai-app-sysredis namespace with
  RedisReplication(1+2) + RedisSentinel(3, quorum 2) on nodes
  fug-1v0/wjh-tgy/48r-b3a, all 3 sentinels in quorum agreement.

Refs:
  datapacket-talos/claudedocs/sysredis-ha-migration-runbook.md (Phase 1)
  datapacket-talos/claudedocs/sysredis-operator-discovery-2026-05-18.md
  datapacket-talos/claudedocs/sysredis-ha-handoff-2026-05-27.md

* redis: address PR #2331 audit findings (mymaster default, scanIterator, topology-change logging)

- env/server-schema.ts: drop `REDIS_SYS_SENTINEL_NAME` default of `mymaster`
  (live cluster uses `sysmaster`) and add a `superRefine` that rejects boot
  when `REDIS_SYS_SENTINELS` is set without `REDIS_SYS_SENTINEL_NAME`. The
  old default was a landmine — a missing var silently produced a Sentinel
  that never resolved a master. (audit Fix 1)

- server/redis/client.ts: fix `scanIteratorWrapper` TypeError on Sentinel
  mode. `RedisSentinel` exposes the SCAN command via WithCommands but not
  the JS-only `scanIterator` helper, so `originalScanIterator(options)`
  was calling `undefined(...)`. Detect sentinel via `getMasterNode` +
  absence of `scanIterator`, then replicate node-redis's scan loop
  (`do { reply = await scan(cursor, options); yield reply.keys }
  while (cursor !== '0')`). Caller in
  `src/pages/api/internal/redis-sys-usage.ts` now works under Sentinel.
  (audit Fix 2)

- server/redis/client.ts: add explicit `topology-change` and `client-error`
  listeners on the sentinel client (it doesn't emit
  `connect`/`reconnecting`/`ready`, so failovers were invisible in Loki).
  Logged via the existing `log()` helper to match the file's style.
  (audit Fix 3)

- server/redis/client.ts: flip `passthroughClientErrorEvents` to `false`
  so one flapping sentinel/replica pod can't flood the top-level `error`
  listener — diagnostics now come from the new `client-error` listener.
  (audit H4)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* redis: round-2 audit fixes (pingInterval parity, .env-example sysmaster)

- nodeClientOptions now passes the same pingInterval as the standalone
  path. Sentinel sub-clients are long-lived against each master/replica
  pod; without a heartbeat, idle connections through any intermediate
  proxy or a rolling sentinel-pod restart can silently expire and only
  surface as latency on the next sysRedis call.
- .env-example REDIS_SYS_SENTINEL_NAME comment now suggests sysmaster
  (production) instead of mymaster. After the round-1 superRefine fix
  the schema rejects boot if the example value is copy-pasted as-is.

* redis: round-3 audit fixes (masterPoolSize HOL, log payload, prom counters)

- masterPoolSize 1→2: serialized writes through one TCP connection caused
  head-of-line blocking — a slow EVAL (PR #2332's atomic helper) could queue
  the periodic PING heartbeat behind it and trip the readiness probe.
- topology-change + client-error log payloads now destructure event.node.host
  and event.node.port so Loki regex can extract per-pod identifiers during a
  multi-pod sentinel flap.
- New Prometheus counters: civitai_sysredis_sentinel_topology_changes_total
  and civitai_sysredis_sentinel_client_errors_total, labeled {type,host,deployment}.
  Phase 4 cutover SRE-on-call needs these to confirm failovers vs steady state.

Tests cover listener wiring, log string shape, counter increments, and
null-event safety.

* prom/redis: drop TDZ-fragile getters in __civitaiRedisMetrics publish

The 2026-06-15 rebase merge introduced getter-based exposure of the two
sysRedis sentinel counters on globalThis to defer capture past the
source-order TDZ. The Round-4 audit caught that any getRedisMetrics()
lookup fired during prom/client.ts top-level eval — before line ~410
where the consts are declared — would throw ReferenceError, which the
no-op fallback at redis/client.ts:458 (`metrics?.X ?? noopCounter`)
does NOT catch (optional chaining only swallows null/undefined).

No eager reader exists today, but this is one feature-flag-gated
startup probe away from a latent TDZ bomb. Fix: move the publish
block below the four const declarations so direct value capture
works, drop the getters.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.env-example

@@ -18,6 +18,10 @@ DATAPACKET_DATABASE_RO_URL=postgresql://postgres:postgres@localhost:15435/postgr
 # Redis
 REDIS_URL=redis://:redis@localhost:6379
 REDIS_SYS_URL=redis://:redis@localhost:6378
+# Optional: switch the `system` redis client to Sentinel mode. When set, REDIS_SYS_URL
+# is still parsed for credentials but the connection is established via Sentinel.
+# REDIS_SYS_SENTINELS=localhost:26379,localhost:26380,localhost:26381
+# REDIS_SYS_SENTINEL_NAME=sysmaster   # production uses "sysmaster"; match the master group your Sentinel CR declares
 
 # Logging
 LOGGING=prisma:error,prisma:warn,seed-metrics-search