* test(preview): fix whatIfFromGraph smoke test against tRPC request batching
The `whatIfFromGraph fires on /generate` preview smoke test has failed 100% of
the time on every PR. It is NOT a cold pod, a DB artifact, or a product defect:
driving a live preview shows the request fires ~1.3s after navigation and
returns HTTP 200 with `cost.total = 8`.
Two test-side defects, both introduced by `trpcBatching` (#2946) ramping on:
1. URL predicate. `httpBatchStreamLink` coalesces concurrent queries into one
request whose path is the COMMA-JOINED procedure list, and whatIf is last:
/api/trpc/content.get,challenge.getInfinite,...,orchestrator.whatIfFromGraph
That does not contain `/api/trpc/orchestrator.whatIfFromGraph`, so
`waitForResponse` never matched and timed out on a request that succeeded.
Now matched by procedure name within the path list (batched or not).
2. Body shape + capture. The client sends `trpc-accept: application/jsonl`, so
the response is newline-delimited chunks — `response.json()` throws, and the
payload sits at a different depth than the old walk expected. Parse raw text,
handling all three wire shapes, and locate the payload structurally (numeric
`cost.total` + boolean `ready`, the whatIf contract) so a sibling procedure in
the batch cannot produce a false pass.
The app also reads the full stream and then abandons the idle reader, which
Chromium reports as `net::ERR_ABORTED` and evicts — making a later
`response.text()` fail with "No data found for resource" on 4 of 8 loads.
The body is now buffered by a route handler, removing that race.
Verified against the live preview for PR #3535: red at origin/main (45s
`waitForResponse` timeout, the exact CI signature), green after, 12/12 runs
(11/12 without the internal retry), ~2s to the assertion. Mutation-checked:
disabling the jsonl branch fails with this guard's own error
("cost.total parsed from whatIfFromGraph response" -> null).
The 45s budget is unchanged and the test still guards the pre-spend cost quote.
* fix(test): narrow capturedBody honestly instead of casting through null
CI Typecheck failed:
tests/preview-generation.spec.ts(177,24): error TS2352: Conversion of type
'null' to type 'string' may be a mistake
`capturedBody` is declared `string | null`, but it is assigned from inside the
`page.route` closure, which TS control-flow analysis cannot see. After the
per-attempt reset to null, TS narrows it to `null` at the return, making
`as string` an illegal null->string conversion.
Replaced the `expect(...).not.toBeNull()` + cast with an explicit null check.
That narrows the type honestly, removes the cast entirely, and fails with a
message distinguishing the two halves: the response was observed, but the
route handler did not capture its body.
Verified: tsc error count 22 at origin/main and 22 here in the same
environment (delta 0), with zero errors mentioning this spec. The 22 are
pre-existing local artifacts (no prisma generate on NixOS, missing submodule).
Why this was not caught before pushing: the fix was verified by RUNNING the
Playwright spec 12 times, and Playwright transpiles without full type
checking. `tests/` is in the tsconfig include set, so CI checks it — a green
Playwright run is not a typecheck.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: audit <alexhurwitz.dev@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shared search backend (in-cluster feeds-proxy via METRICS_SEARCH_HOST) is healthy
at rest but intermittently 5xx's under concurrent preview-build load, and /generate SSR
stalls on a cold single-replica pod. Playwright's test-level retries re-run within
seconds — too fast to outlast a load spike — so /moderator/images and whatIf hard-fail
on every PR (suite-wide red). Honest resilience (no skipping):
- preview-auth.setup.ts: add a SEARCH-READINESS gate after the warm-up — poll the
image-search path (and /moderator/images) until 2xx with spacing, so the suite
doesn't start mid-spike. Non-fatal: a sustained outage still surfaces.
- preview-retry.ts: shared retryFlaky(label, fn, {attempts, backoffMs}) — bounded
retry-with-backoff; the wrapped step must still succeed or the last error throws.
- preview-moderation.spec.ts: retry the /moderator/images navigation on a transient 5xx.
- preview-generation.spec.ts: retry the whatIf navigate+wait (extended per-test timeout
to fit ~2 attempts of the 45s wait) for cold-pod /generate load.
Verified root cause against the live cluster: feeds-proxy + search-new both 200 at rest;
failures are load-correlated, recover on their own.
On a contended preview node the single-replica pod gets CPU-throttled and the
heaviest flows intermittently crossed their timeout ceilings and HARD-failed
(both attempts), instead of flaking-and-recovering on retry:
- /user/membership, /models SSR goto: observed ~66s, just past the 60s test +
nav timeout.
- generation whatIf waitForResponse: pushed past 25s (page+hydrate+form+resource
+orchestrator chain) under load.
Raise per-test + navigationTimeout 60s -> 90s, and the whatIf waitForResponse
25s -> 45s. These are margins for slow windows, not masks for real breaks: the
orchestrator is ~57ms and /generate SSR ~0.4s when healthy, so a healthy run
never approaches these. Removes the hard-fail mode that made the report-only
smoke comment show ❌ on clean PRs — a prerequisite for ever flipping smoke to
gating.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(preview): tranche-2 e2e — generation whatIf, report loop, moderation
Extends the preview smoke harness to the audit's highest-risk untested flows,
self-seeding per-run uniquely-tagged fixtures (no shared-dev-DB collisions; the
mutation isolation answer to "seed per-PR"). All report-only.
- preview-trpc.ts: tiny superjson+batched tRPC client (CSRF Origin/Referer
stamped) for self-seeding via page.request, + uniqueToken().
- preview-generation.spec.ts: assert the REAL orchestrator.whatIfFromGraph cost
quote fires + returns a numeric cost (gold). De-mocks the pricing path; no Buzz.
- preview-report.spec.ts: tester self-seeds a Post (post.create) and reports it
(report.create {type:'post',reason:'TOSViolation',details:{violation}}).
- preview-moderation.spec.ts: mod queues (/moderator/reports + /images) render;
+ a self-seeded report is actioned end-to-end (report.create -> setStatus).
tRPC input shapes verified against the schema files. NOT YET live-validated —
the CSRF origin gate on direct tRPC + the whatIf/mod selectors get confirmed by
the next report-only preview run (deferred: generation-submit, blocked on the
external Buzz service which can't be seeded via Postgres).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(preview): typecheck — reportId is number | undefined
the pipeline tsc (not playwright --list) caught: reportId inferred number
from report?.id, then assigned mine?.id (number | undefined) -> TS2322.
Type it explicitly; the runtime expect() still guards undefined.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(preview): drop the two fragile secondary DOM checks
The live run validated every core tranche-2 flow (real whatIf cost, tRPC
self-seed report round-trip through the CSRF gate, mod render + report->action
loop). The only two reds were optional best-effort DOM checks that duplicate
passing core tests:
- generation "cost near submit button": the submit button is in a gen panel
collapsed by default on the preview viewport (resolves but hidden). The
network whatIf assertion already covers the cost.
- report "image page report affordance": the report control is behind an entity
action menu, not a top-level button. report.create is already covered.
Removed both rather than leave known flakes in the gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zachary Lowden