mirror of
https://github.com/civitai/civitai.git
synced 2026-09-20 22:08:18 +08:00
Zachary Lowden
692f50a437
The shared search backend (in-cluster feeds-proxy via METRICS_SEARCH_HOST) is healthy
at rest but intermittently 5xx's under concurrent preview-build load, and /generate SSR
stalls on a cold single-replica pod. Playwright's test-level retries re-run within
seconds — too fast to outlast a load spike — so /moderator/images and whatIf hard-fail
on every PR (suite-wide red). Honest resilience (no skipping):
- preview-auth.setup.ts: add a SEARCH-READINESS gate after the warm-up — poll the
image-search path (and /moderator/images) until 2xx with spacing, so the suite
doesn't start mid-spike. Non-fatal: a sustained outage still surfaces.
- preview-retry.ts: shared retryFlaky(label, fn, {attempts, backoffMs}) — bounded
retry-with-backoff; the wrapped step must still succeed or the last error throws.
- preview-moderation.spec.ts: retry the /moderator/images navigation on a transient 5xx.
- preview-generation.spec.ts: retry the whatIf navigate+wait (extended per-test timeout
to fit ~2 attempts of the 45s wait) for cold-pod /generate load.
Verified root cause against the live cluster: feeds-proxy + search-new both 200 at rest;
failures are load-correlated, recover on their own.
41 lines
1.5 KiB
TypeScript
41 lines
1.5 KiB
TypeScript
/**
|
|
* Bounded retry-with-backoff for riding out TRANSIENT preview-infra flakiness
|
|
* (the shared search backend — feeds-proxy/meili — intermittently 5xx's under
|
|
* concurrent preview-build load; cold-SSR page loads stall) WITHOUT masking a
|
|
* real failure: the wrapped step must still eventually succeed, and if every
|
|
* attempt fails the last error surfaces and the spec fails honestly.
|
|
*
|
|
* Why not rely on Playwright's `retries`: those re-run the WHOLE test within a
|
|
* few seconds of each other — too fast to outlast a load spike (the failing
|
|
* runs exhausted all 3 attempts in <10s). This retries the single flaky step
|
|
* with real spacing so a brief spike is ridden out.
|
|
*
|
|
* NOT a test file (excluded from the preview config testMatch).
|
|
*/
|
|
export async function retryFlaky<T>(
|
|
label: string,
|
|
fn: (attempt: number) => Promise<T>,
|
|
opts: { attempts?: number; backoffMs?: number } = {}
|
|
): Promise<T> {
|
|
const attempts = opts.attempts ?? 3;
|
|
const backoffMs = opts.backoffMs ?? 6000;
|
|
let lastErr: unknown;
|
|
for (let attempt = 1; attempt <= attempts; attempt++) {
|
|
try {
|
|
return await fn(attempt);
|
|
} catch (err) {
|
|
lastErr = err;
|
|
if (attempt < attempts) {
|
|
// eslint-disable-next-line no-console
|
|
console.warn(
|
|
`[retryFlaky] "${label}" failed attempt ${attempt}/${attempts}; retrying in ${
|
|
backoffMs * attempt
|
|
}ms`
|
|
);
|
|
await new Promise((resolve) => setTimeout(resolve, backoffMs * attempt));
|
|
}
|
|
}
|
|
}
|
|
throw lastErr;
|
|
}
|