vercel__eve/e2e

e2e

End-to-end coverage is fixture-owned eve eval runs. The suite only runs fixture eval files from the fixture directory.

CI splits the coverage into two suites so live-model flake never gates world-infrastructure coverage (and vice versa):

• Model suite (e2e-local.yml): every fixture runs against the local workflow world with real matrix models. Fixtures whose behavior varies per provider declare "e2e": { "modelMatrix": "full" } in package.json and run on every matrix model; all other fixtures run once on the default model.
• World suites (e2e-vercel.yml, e2e-postgres.yml, and one workflow per additional workflow world): every fixture builds, deploys, and runs once with deterministic mock models (EVE_E2E_MODEL=mock), proving the world's infrastructure — build, deploy, boot, streaming, durability — without live models. Evals whose assertions need a real model carry the real-model tag and are excluded there via --exclude-tag real-model.

Harness config (@eve-e2e/config)

Fixture agents author their harness-owned configuration through the private @eve-e2e/config workspace package (e2e/fixtures/e2e-config):

• e2eAgentConfig({ mock? }) — spread into the root defineAgent: resolves the matrix model from EVE_E2E_MODEL, and applies the workflow-world override from EVE_E2E_WORKFLOW_WORLD when set.
• e2eSubagentConfig({ mock? }) — the same model resolution for subagents, without root-only settings.
• e2eModel({ mock? }) — a bare model handle for nested slots such as compaction models and dynamic selections.
• e2eJudgeModel() — the judge model for evals.config.ts.

When EVE_E2E_MODEL=mock, all of these return a deterministic mockModel() instead of a gateway model id. The default responder echoes the last user message; fixtures whose evals need tool calls pass a scripted responder via the mock option (see agent-tools-sandbox/agent/agent.ts).

The real-model tag

An eval tagged real-model asserts behavior only a live model produces (judge scoring, provider cache metrics, free-form tool planning). The world suites exclude it; it still runs in the model suite. Untagging an eval is the migration unit for world-suite coverage: script the fixture's mock responder until the eval passes under EVE_E2E_MODEL=mock, then remove the tag. Prefer untagging over new real-model tags — deterministic evals make every world suite stronger.

Local

Run evals from the fixture directory:

cd e2e/fixtures/agent-basic-runtime
EVE_E2E_MODEL="openai/gpt-5.6-sol" pnpm exec eve eval --strict

Mock-model runs work anywhere with no provider credentials, which makes them the fastest way to validate world-suite behavior locally:

EVE_E2E_MODEL=mock pnpm exec eve eval --strict --exclude-tag real-model

Every retained e2e eval is deterministic and self-contained. Coverage that needs external services or injected env is intentionally not part of this suite. Most fixtures use the shared model-provider credentials; dedicated runtime stress fixtures may use an authored deterministic model instead.

Each retained fixture package also exposes the same command as:

pnpm --filter agent-basic-runtime test:e2e

The root convenience command runs every fixture package with a test:e2e script:

pnpm test:e2e

Vercel

Vercel e2e uses the same fixture evals against immutable preview deployment URLs. All fixture deployments link to the same Vercel project id; isolation comes from the deployment URL returned by vc deploy --prebuilt.

One-time project setup:

• Configure the shared Vercel project for Node.js 24.
• Provide the model-provider credentials needed by EVE_E2E_MODEL in the project's Preview environment.
• Provide VERCEL_TOKEN, VERCEL_ORG_ID, and VERCEL_PROJECT_ID in CI.

Run a fixture against Vercel from its directory:

vc link --yes --project "$VERCEL_PROJECT_ID"
vc env pull --yes --environment=preview
VERCEL=1 VERCEL_ENV=preview VERCEL_TARGET_ENV=preview \
  VERCEL_PROJECT_ID="$VERCEL_PROJECT_ID" \
  pnpm exec eve build
DEPLOYMENT_URL="$(vc deploy --prebuilt --yes --target=preview \
  --env "EVE_E2E_MODEL=$EVE_E2E_MODEL" | tail -n 1)"
npx eve eval --strict --url "$DEPLOYMENT_URL"

Do not set VERCEL_TEAM_ID at build: sandbox template keys must derive identically at build and runtime, and Vercel has no team variable at runtime.

Redeploy suite

agent-tools-sandbox/evals/sandbox/redeploy.eval.ts proves sandbox semantics across deployment updates as they behave on preview targets: a parked session keeps working (with its /workspace state intact) when messages route through a new deployment, its turns stay pinned to the deployment that created it (branch-less CLI preview deploys cannot resolve a "latest" deployment; see shouldRouteToLatestDeployment in execution/workflow-runtime.ts), and new sessions adopt the new deployment — a skill added by the redeploy loads there. The pinned-turn assertion is a deliberate tripwire: it must be flipped when turn dispatch gains preview latest-routing (https://github.com/vercel/eve/issues/582).

agent-channels/evals/custom-channels/cross-version-session-inbox.eval.ts deploys the fixture with the published eve@0.30.8, holds a turn active in that old consumer, then redeploys the current checkout and sends a replacement message through the same durable session. It verifies both sides of the codec: the current producer must choose the old consumer's wire version, and the real old consumer must decode and buffer it. The eval then cancels the deliberately blocked turn and verifies that the old session runs the buffered follow-up.

The eval redeploys from inside its test body: it mutates the agent source, runs eve build + vc deploy, and repoints a run-scoped Vercel alias at each new deployment, polling /eve/v1/info until the alias serves it. Because immutable deployment URLs never change what they serve, the eval must run against the alias — the e2e-vercel workflow sets EVE_E2E_REDEPLOY_ALIAS, aliases the deployment, and runs --tag redeploy evals as a second eve eval invocation after the main suite. Without the alias env (local matrix, plain eve eval --strict) the eval skips.

Most fixture agents and their configured judges resolve EVE_E2E_MODEL through @eve-e2e/config, defaulting to openai/gpt-5.6-sol for local runs. agent-prompt-cache is the one fixture that authors a direct @ai-sdk/anthropic model instance instead of a gateway model id: its eval asserts the harness's Anthropic cache-breakpoint placement, which only runs on that path. It uses the matrix model when it is an Anthropic model and otherwise falls back to anthropic/claude-opus-5. The instance points at the AI Gateway's Anthropic-compatible Messages endpoint so it uses the same AI_GATEWAY_API_KEY credential as every other fixture. agent-workflow-stress uses eve's mockModel fixture helper so its 100-turn runs stay fast and deterministic. Its concurrent and sequential evals cover high-volume session execution and repeated session resumption respectively.

Fixtures

E2E fixtures live under e2e/fixtures/*. Fixture discovery also accepts apps/fixtures/* apps with an evals/ directory, but shared development apps should stay out of the e2e matrix unless they intentionally own evals.

When adding e2e coverage:

• Put the eval in the fixture app's evals/ directory.
• Keep it runnable with only eve eval --strict.
• Keep it deterministic: no external service startup or injected env requirements (beyond model-provider credentials).
• If the behavior cannot fit that shape yet, leave it out and rebuild it later as a first-class eval story.

CI

The matrix models and worlds are registered in e2e/matrix.json — edit that file to add either. .github/scripts/discover-e2e-fixtures.mjs discovers every fixture with an evals/ directory and emits the suite matrices from the registry:

• model_matrix — fixture × model legs for e2e-local.yml. The first registry model is the default that every fixture runs on; the rest run only on fixtures with "e2e": { "modelMatrix": "full" } in package.json. A fixture can add a narrowly scoped leg with e2e.additionalModels entries shaped as { "name": "short-check-name", "id": "provider/model" }.
• world_matrix_<world> — one leg per fixture for that world's suite workflow. A registered world's package reaches the job as EVE_E2E_WORKFLOW_WORLD (worlds without one, like vercel, use the deploy target's default).

The short model name is the stable Actions check identifier; the full id selects the provider model. Updating a model version does not rename required checks. Each workflow also publishes one stable aggregate check — e2e-local, e2e-vercel, or e2e-postgres — which succeeds only when every leg succeeds. Require those aggregate checks in the repository ruleset so newly added fixtures and models become required automatically. Add a new aggregate to the ruleset only after its workflow lands on main: a required check nothing reports blocks every PR as permanently "expected".

.github/workflows/e2e-local.yml (the model suite) builds the eve package once per leg, then runs one fixture directory with the leg's real model:

pnpm --filter eve run build
cd "$FIXTURE_DIR"
EVE_E2E_MODEL="$MODEL" pnpm exec eve eval --strict --junit "$JUNIT_PATH"

Always build with the full build script (not build:js); only the full build stamps the package version into dist.

.github/workflows/e2e-vercel.yml (the Vercel world suite) links each fixture directory to the shared Vercel project id, builds Vercel output locally with EVE_E2E_MODEL=mock, deploys that output, and runs the mock-compatible evals:

pnpm exec eve build
DEPLOYMENT_URL="$(vc deploy --prebuilt --yes --target=preview \
  --env "EVE_E2E_MODEL=mock" | tail -n 1)"
npx eve eval --strict --exclude-tag real-model \
  --url "$DEPLOYMENT_URL" --junit "$JUNIT_PATH"

.github/workflows/e2e-postgres.yml (the Postgres world suite) starts a PostgreSQL service container, bootstraps the @workflow/world-postgres schema, builds each fixture with EVE_E2E_WORKFLOW_WORLD=@workflow/world-postgres, runs the mock-compatible evals against a local production server (eve start), and asserts the traffic produced Postgres-backed workflow runs. Every fixture carries @workflow/world-postgres as a dependency so the world module resolves at build time.

A world suite for another workflow world follows the same shape: register the world in e2e/matrix.json, add an e2e-<world>.yml that consumes its world_matrix_<world> output (the registered package arrives as matrix.world_package for EVE_E2E_WORKFLOW_WORLD), set EVE_E2E_MODEL=mock (plus any backing services), and run with --exclude-tag real-model.

TUI smoke scripts are not e2e. They live under packages/eve/test/tui-client and run through pnpm test:tui.