test(preview): pre-warm all heavy pages + 2 retries (kill the slow-window flake) (#2483)

* test(preview): pre-warm all heavy pages + 2 retries (kill the slow-window flake)

The dominant smoke flake is a cold/slow-window page.goto timeout on a heavy SSR
page — the offender rotates (/, /models, /user/membership, /generate,
/purchase/buzz, /moderator/*) depending on which page the slow window hits.

Two-pronged:
 - Pre-warm EVERY heavy page the suite navigates in preview-auth.setup.ts (was
   only /, /models, /images). Each route JIT-compiles on first hit, so warming
   them up front (gold cookie for member pages, mod cookie for /moderator/*)
   removes the cold-start cost that caused most flakes. Sequential + best-effort
   (a slow warm GET still triggers the server-side compile).
 - retries 1 -> 2: a 2nd retry covers the residual mid-run slow window so it
   flakes-and-recovers instead of surfacing as a failure.

Pairs with the 60s->90s timeout raise (#2480). Together these target a
consistently-clean run — a prerequisite for eventually flipping smoke to gating.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(preview): raise setup timeout for cold-pod sequential warm-up (pr-review)

Addresses the pr-reviewer finding on this PR: the 8+ sequential warm-up GETs
(each 60s-capped) can, on a genuinely cold pod, cumulatively exceed the suite's
90s per-test timeout → the setup test times out → ALL dependent smoke tests are
SKIPPED (worse than the flake this PR fixes). The passing run only saw 1.7s
because the pod was already warm from verify-preview.

Fix: setup.setTimeout(480_000) — a ceiling for worst-case cold sequential
warm-up. Kept sequential (parallel heavy SSR renders OOM the single-replica pod,
per the config comment — the OOM is about concurrent renders, not specifically
playwright workers, so concurrent HTTP GETs are NOT a safe alternative here). The
ceiling doesn't slow the normal path: the setup still finishes as fast as the
warm-ups actually take.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zachary Lowden
2026-06-11 17:03:41 -05:00
committed by GitHub
parent f123f1a5c4
commit 8942b54f8c
2 changed files with 42 additions and 8 deletions
+5 -1
View File
@@ -25,7 +25,11 @@ export default defineConfig({
testMatch: /(^|\/)preview-.*\.(setup|spec)\.ts$/, testMatch: /(^|\/)preview-.*\.(setup|spec)\.ts$/,
fullyParallel: false, fullyParallel: false,
forbidOnly: !!process.env.CI, forbidOnly: !!process.env.CI,
retries: process.env.CI ? 1 : 0, // 2 retries (was 1): the preview pod can hit a transient CPU-throttle window on a
// contended node; with pre-warm (preview-auth.setup.ts) covering cold-start, the
// residual flake is a mid-run slow window. A 2nd retry gives it another recovery
// chance so a slow window flakes-and-recovers instead of surfacing as a failure.
retries: process.env.CI ? 2 : 0,
// Run SERIALLY. The preview is a single-replica, cold, resource-modest pod. // Run SERIALLY. The preview is a single-replica, cold, resource-modest pod.
// Concurrent loads of the heavy pages (/models, /images, /purchase/buzz, // Concurrent loads of the heavy pages (/models, /images, /purchase/buzz,
// /pricing, model detail) across multiple workers segfaulted it (exit 139), // /pricing, model detail) across multiple workers segfaulted it (exit 139),
+37 -7
View File
@@ -84,6 +84,14 @@ async function mintStorageState(role: PreviewRole): Promise<string> {
} }
setup('mint preview sessions', async ({ request }) => { setup('mint preview sessions', async ({ request }) => {
// This setup runs the cold-pod warm-up below: ~9 SEQUENTIAL heavy-SSR GETs (each
// capped at 60s) so one route compiles at a time (parallel heavy renders OOM the
// single-replica pod). On a genuinely cold pod the cumulative warm-up can exceed
// the suite's 90s per-test timeout — and a setup timeout SKIPS every dependent
// smoke test (worse than the flake we're fixing). So give just this setup a large
// ceiling. It's a CEILING, not the runtime: the setup still finishes as fast as
// the warm-ups actually take (~2s when the pod is already warm from verify-preview).
setup.setTimeout(480_000);
if (!SECRET) throw new Error('NEXTAUTH_SECRET is required to mint preview sessions'); if (!SECRET) throw new Error('NEXTAUTH_SECRET is required to mint preview sessions');
if (!PREVIEW_URL) throw new Error('PREVIEW_URL is required for preview smoke tests'); if (!PREVIEW_URL) throw new Error('PREVIEW_URL is required for preview smoke tests');
const jwts: Partial<Record<PreviewRole, string>> = {}; const jwts: Partial<Record<PreviewRole, string>> = {};
@@ -92,16 +100,38 @@ setup('mint preview sessions', async ({ request }) => {
} }
// Warm the freshly-deployed preview before the suite so the first real test // Warm the freshly-deployed preview before the suite so the first real test
// doesn't pay the full cold-SSR cost (Next warm-up + JIT + DB pools). Warm the // doesn't pay the full cold-SSR cost (Next warm-up + JIT + DB pools). Each route
// HEAVY listing pages too, not just `/` — those are the slow ones. They must be // JIT-compiles on its first hit, so we warm EVERY heavy SSR page the suite then
// warmed AUTHENTICATED: on a preview the gate 307s an UNauthenticated /models to // navigates — cold-page timeouts were the dominant smoke flake (a slow-window
// /login, so an anon GET wouldn't touch the real /models render path. Use the // page.goto exceeding the nav budget, then passing on retry once warm). They must
// gold (gate-passing) cookie. Sequential + non-fatal; the suite + retries cover // be warmed AUTHENTICATED: on a preview the gate 307s an UNauthenticated request
// any miss. // to /login, so an anon GET wouldn't touch the real render path. Sequential (one
// concurrent heavy SSR at a time — the single-replica pod OOM'd under parallel
// heavy loads) + non-fatal (.catch): a slow warm-up GET still triggers the
// server-side compile even if the client times out, and the suite + retries
// cover any miss.
const gold = jwts.gold; const gold = jwts.gold;
if (gold) { if (gold) {
const headers = { cookie: `${COOKIE_NAME}=${gold}` }; const headers = { cookie: `${COOKIE_NAME}=${gold}` };
for (const path of ['/', '/models', '/images']) { // gold (gate-passing paid member) reaches all non-mod heavy pages the suite hits.
for (const path of [
'/',
'/models',
'/images',
'/user/membership',
'/generate',
'/purchase/buzz',
'/pricing',
]) {
await request.get(path, { timeout: 60_000, headers }).catch(() => {});
}
}
const mod = jwts.mod;
if (mod) {
// /moderator/* render only for a moderator (gold would be bounced), so warm the
// moderation-spec pages with the mod cookie.
const headers = { cookie: `${COOKIE_NAME}=${mod}` };
for (const path of ['/moderator/reports', '/moderator/images']) {
await request.get(path, { timeout: 60_000, headers }).catch(() => {}); await request.get(path, { timeout: 60_000, headers }).catch(() => {});
} }
} }