feat(eve): evaluate - expose standalone evaluation with shared auth (#3472)

Signed-off-by: Andrew Barba <barba@hey.com>

.changeset/quiet-evaluations-share-auth.md

@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Add standalone `evaluate` to `eve/experimental/evaluate` for typed evaluations in tools and application code. It shares model authentication with `autoModel`, including the Gateway connection selected during `eve dev`.

docs/guides/evaluate.md

@@ -1,12 +1,13 @@
 ---
 title: Automatic Model Selection
-description: "Choose an agent model from the current request with an AI SDK evaluation model."
+description: "Choose agent models automatically or evaluate typed questions in your tools and application code."
 ---
 
 Use `autoModel` to choose an agent model from an allowlist before inference begins.
 It uses the [AI SDK evaluation API](https://ai-sdk.dev/docs/ai-sdk-core/evaluation),
 so the evaluator can be a Vercel AI Gateway model ID or an evaluation model from
-an installed provider.
+an installed provider. Use `evaluate` from the same entrypoint to ask typed
+questions in your own tools or application code.
 
 `eve/experimental/evaluate` is experimental. Its API can change between eve
 releases, and the AI SDK evaluation model specification can change in patch
@@ -103,6 +104,54 @@ Supported reasoning values are `"provider-default"`, `"none"`, `"minimal"`,
 `"low"`, `"medium"`, `"high"`, and `"xhigh"`. An omitted value inherits the
 agent's reasoning setting.
 
+## Evaluate inside a tool
+
+Use `evaluate` to ask choice, score, or boolean questions about the state you pass to it.
+It defaults to `typesafe-ai/jev` and uses the same authentication as `autoModel`,
+including the Gateway connection selected through `/login` during `eve dev`.
+Pass `model` to use another evaluation model ID or a provider instance. A configured
+AI SDK default provider takes precedence over the local Gateway connection.
+
+```ts title="agent/tools/classify-request.ts"
+import { evaluate } from "eve/experimental/evaluate";
+import { defineTool } from "eve/tools";
+import { z } from "zod";
+
+export default defineTool({
+  description: "Choose the team that can help with a customer request.",
+  inputSchema: z.object({ request: z.string().min(1).max(8000) }),
+  async execute({ request }, ctx) {
+    const result = await evaluate({
+      state: { request },
+      questions: {
+        team: {
+          type: "choice",
+          instructions: "Select the team best suited to handle the request.",
+          criteria: {
+            billing: "Invoices, payments, and refunds",
+            support: "Product questions and troubleshooting",
+          },
+        },
+      },
+      abortSignal: ctx.abortSignal,
+    });
+    return { team: result.answers.team.choice };
+  },
+});
+```
+
+The choice above is typed as `"billing" | "support"`. Each question appears under
+its authored key in `result.answers`. Results also include token usage, warnings,
+provider metadata, and response metadata.
+
+`evaluate` accepts AI SDK evaluation options, including `maxRetries`, `headers`,
+and `providerOptions`. Pass an `abortSignal` to cancel the request. Input and
+answer validation, retries, and provider errors follow AI SDK semantics.
+
+You can also call `evaluate` outside a tool; it does not require an active eve
+session. Each call performs its own evaluation. `autoModel` uses this function
+and adds the per-turn routing behavior described below.
+
 ## Runtime behavior
 
 `autoModel` evaluates at the first `step.started` event, after the incoming prompt

docs/reference/typescript-api.md

@@ -137,7 +137,7 @@ import template from "../../prompts/template.txt?raw";
 | `eve/local-dev`                                                             | `getLocalDevCapability`, `LocalDevCapability`                                                           |
 | `eve/models/openai`                                                         | `openai`, `chatgpt`, deprecated `experimental_chatgpt`                                                  |
 | `eve/models/anthropic`                                                      | `anthropic`                                                                                             |
-| [`eve/experimental/evaluate`](../guides/evaluate)                           | Experimental `autoModel`                                                                                |
+| [`eve/experimental/evaluate`](../guides/evaluate)                           | Experimental `autoModel` and standalone `evaluate`                                                      |
 | `eve/evals`                                                                 | `defineEval`, `defineEvalConfig`, `mockModel`, eval types                                               |
 | `eve/evals/expect`                                                          | `includes`, `equals`, `matches`, `similarity`                                                           |
 | `eve/evals/reporters`                                                       | `Braintrust`, `JUnit`, `EvalReporter`                                                                   |

e2e/fixtures/agent-evaluate/agent/agent.ts

@@ -8,6 +8,21 @@ const { experimental } = e2eAgentConfig();
 export default defineAgent({
   experimental,
   model: fixtureModel(async (request) => {
+    if (request.userMessages.some((text) => text.includes("evaluate-request"))) {
+      const result = request.toolResults.find((result) => result.name === "evaluate-request");
+      if (result) return JSON.stringify({ isError: result.isError, output: result.output });
+      return {
+        toolCalls: [
+          {
+            id: "evaluate-request-1",
+            name: "evaluate-request",
+            input: {
+              missingAnswer: request.userMessages.some((text) => text.includes("missing answer")),
+            },
+          },
+        ],
+      };
+    }
     if (request.userMessages.some((text) => text.includes("parallel investigations"))) {
       const completed = request.messages.filter((message) =>
         message.text.includes("child-result:"),

e2e/fixtures/agent-evaluate/agent/testing.ts

@@ -10,7 +10,7 @@ export const routing = defineState("evaluate-fixture.routing", () => ({
   reasoning: "unselected",
 }));
 
-const evaluationModel = {
+export const evaluationModel: Exclude<Experimental_EvaluationModel, string> = {
   specificationVersion: "v4",
   provider: "fixture",
   modelId: "fixture-evaluator",
@@ -37,7 +37,7 @@ const evaluationModel = {
       response: { modelId: "fixture-evaluator" },
     };
   },
-} satisfies Exclude<Experimental_EvaluationModel, string>;
+};
 
 /** Run the real router with deterministic evaluation and language models. */
 export function fixtureModel(respond: MockModelResponder) {

e2e/fixtures/agent-evaluate/agent/tools/evaluate-request.ts

@@ -0,0 +1,34 @@
+import { evaluate } from "eve/experimental/evaluate";
+import { defineTool } from "eve/tools";
+
+import { evaluationModel } from "../testing";
+
+export default defineTool({
+  description: "Evaluate Alice's request with the fixture evaluation provider.",
+  inputSchema: {
+    type: "object",
+    properties: { missingAnswer: { type: "boolean" } },
+    required: ["missingAnswer"],
+    additionalProperties: false,
+  },
+  async execute({ missingAnswer }, ctx) {
+    const result = await evaluate({
+      model: missingAnswer
+        ? { ...evaluationModel, doEvaluate: async () => ({ answers: {}, warnings: [] }) }
+        : evaluationModel,
+      state: { request: "Alice needs a routine summary." },
+      questions: {
+        route: {
+          type: "choice",
+          instructions: "Choose a model for the request.",
+          criteria: {
+            "openai/large": "Difficult investigations",
+            "openai/small": "Routine requests",
+          },
+        },
+      },
+      abortSignal: ctx.abortSignal,
+    });
+    return { choice: result.answers.route.choice, usage: result.usage };
+  },
+});

e2e/fixtures/agent-evaluate/evals/tool-evaluation-failure.eval.ts

@@ -0,0 +1,14 @@
+import { defineEval } from "eve/evals";
+
+export default defineEval({
+  description: "An invalid evaluation response is reported as a tool error.",
+  async test(t) {
+    const turn = await t.send(
+      "Bob uses evaluate-request to review a missing answer from the evaluation service.",
+    );
+    turn.expectOk();
+    turn.calledTool("evaluate-request", { status: "failed", count: 1 });
+    turn.messageIncludes('"isError":true');
+    t.succeeded();
+  },
+});

e2e/fixtures/agent-evaluate/evals/tool-evaluation.eval.ts

@@ -0,0 +1,16 @@
+import { defineEval } from "eve/evals";
+
+export default defineEval({
+  description: "A tool evaluates structured state and returns typed answers and usage.",
+  async test(t) {
+    const turn = await t.send(
+      "Alice uses evaluate-request to choose a model for her routine summary.",
+    );
+    turn.expectOk();
+    turn.calledTool("evaluate-request");
+    turn.messageIncludes('"isError":false');
+    turn.messageIncludes('"choice":"openai/small"');
+    turn.messageIncludes('"totalTokens":45');
+    t.succeeded();
+  },
+});

packages/eve/extension-contracts/compatibility/tool/v46.ts

@@ -0,0 +1,9 @@
+import { defineTool } from "#public/tools/index.js";
+
+export default defineTool({
+  description: "Report whether the current tool execution was cancelled.",
+  inputSchema: { type: "object", properties: {}, additionalProperties: false },
+  execute(_input, ctx) {
+    return { cancelled: ctx.abortSignal?.aborted ?? false };
+  },
+});

packages/eve/extension-contracts/entrypoints/tool.ts

@@ -18,3 +18,4 @@ export {
   type WorkflowToolInput,
   type WorkflowToolOptions,
 } from "../../src/public/tools/workflow.ts";
+export { evaluate } from "../../src/public/experimental/evaluate/index.ts";

packages/eve/extension-contracts/reports/tool/v47.json

@@ -0,0 +1,23 @@
+{
+  "kind": "eve-extension-capability-contract",
+  "capability": "tool",
+  "epoch": 47,
+  "sha256": "c8f3dea743ed9d8c175769c1249d099bc5265beda83104a8edb9ff841c7f0766",
+  "exports": [
+    "WorkflowTool",
+    "WorkflowToolInput",
+    "WorkflowToolOptions",
+    "defaultWebSearch",
+    "defineTool",
+    "defineWorkflowTool",
+    "disableTool",
+    "evaluate",
+    "isDisabledToolSentinel",
+    "isWebSearchToolDefinition",
+    "toolOutput",
+    "toolOutputPart",
+    "toolResultFrom",
+    "webSearch",
+    "workflow"
+  ]
+}

packages/eve/src/compiler/extension-compatibility.ts

@@ -22,8 +22,10 @@ interface ExtensionCapabilityContract {
 const EXTENSION_CAPABILITY_CONTRACTS = {
   extension: { current: 1, supported: [1], dropped: {} },
   tool: {
-    current: 46,
-    supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 34, 35, 44, 45, 46],
+    current: 47,
+    supported: [
+      1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 34, 35, 44, 45, 46, 47,
+    ],
     dropped: {
       14: "TaskExec.delegated was removed; migrate to workflow-backed background tools",
       15: "TaskExec replaces stageEffect with send",

packages/eve/src/experimental/evaluate/auto-model.ts

@@ -1,9 +1,6 @@
 import { createHash } from "node:crypto";
 
-import {
-  experimental_evaluate as evaluate,
-  type Experimental_EvaluationModel as EvaluationModel,
-} from "ai";
+import { type Experimental_EvaluationModel as EvaluationModel } from "ai";
 
 import { loadContext } from "#context/container.js";
 import { ContextKey } from "#context/key.js";
@@ -13,13 +10,14 @@ import {
   type DynamicSentinel,
 } from "#dynamic/definition.js";
 import { isAgentReasoningDefinition, isRuntimeLanguageModel } from "#internal/runtime-model.js";
-import { localGatewayEvaluationModel } from "#internal/model-auth/transport.js";
 import type {
   AgentReasoningDefinition,
   PublicAgentDynamicModelResult,
   PublicAgentStaticModelDefinition,
 } from "#shared/agent-definition.js";
 
+import { DEFAULT_EVALUATION_MODEL, evaluate } from "./evaluate.js";
+
 type AutoModelOption =
   | string
   | {
@@ -36,8 +34,6 @@ interface AutoModelConfig<
   readonly options: T;
 }
 
-const DEFAULT_EVALUATION_MODEL = "typesafe-ai/jev";
-
 function isRecord(value: unknown): value is Record<string, unknown> {
   return typeof value === "object" && value !== null && !Array.isArray(value);
 }
@@ -169,11 +165,7 @@ export function autoModel<const T extends Readonly<Record<string, AutoModelOptio
         if (previous?.turnId === currentTurnId) return models.get(previous.model)!;
 
         const result = await evaluate({
-          model:
-            typeof evaluationModel === "string" &&
-            Reflect.get(globalThis, "AI_SDK_DEFAULT_PROVIDER") == null
-              ? (localGatewayEvaluationModel(evaluationModel) ?? evaluationModel)
-              : evaluationModel,
+          model: evaluationModel,
           state: routingState(ctx),
           questions: {
             route: {

packages/eve/src/experimental/evaluate/evaluate.integration.test.ts

@@ -0,0 +1,136 @@
+import { Experimental_EvaluationMockModelV4 } from "ai/test";
+import { afterEach, beforeEach, describe, expect, expectTypeOf, it, vi } from "vitest";
+
+import { evaluate } from "#public/experimental/evaluate/index.js";
+
+const localEvaluationModel = vi.hoisted(() => vi.fn());
+vi.mock("#internal/model-auth/transport.js", async (importOriginal) => ({
+  ...(await importOriginal<typeof import("#internal/model-auth/transport.js")>()),
+  localGatewayEvaluationModel: localEvaluationModel,
+}));
+
+const questions = {
+  category: {
+    type: "choice",
+    instructions: "Select the team that can help with the request.",
+    criteria: { billing: "Invoices and payments", support: "Product questions" },
+  },
+  urgent: { type: "boolean", instructions: "Does the request require immediate attention?" },
+  priority: { type: "score", instructions: "Rate urgency.", criteria: ["Low", "High"] },
+} as const;
+const state = { request: "Alice needs a copy of her invoice." };
+
+function evaluationModel() {
+  const doEvaluate = vi.fn(async () => ({
+    answers: {
+      category: { type: "choice" as const, choice: "billing" },
+      urgent: { type: "boolean" as const, probability: 0.1 },
+      priority: { type: "score" as const, score: 0.1 },
+    },
+    usage: { inputTokens: 42, outputTokens: 3 },
+    warnings: [],
+    providerMetadata: { fixture: { requestId: "request-1" } },
+  }));
+  return { doEvaluate, model: new Experimental_EvaluationMockModelV4({ doEvaluate }) };
+}
+
+beforeEach(() => {
+  localEvaluationModel.mockReset();
+  vi.stubGlobal("AI_SDK_DEFAULT_PROVIDER", undefined);
+});
+afterEach(() => vi.unstubAllGlobals());
+
+describe("evaluate", () => {
+  it("defaults to Jev using the local Gateway connection without a session context", async () => {
+    const evaluator = evaluationModel();
+    localEvaluationModel.mockReturnValue(evaluator.model);
+
+    const result = await evaluate({ state, questions });
+
+    expect(localEvaluationModel).toHaveBeenCalledWith("typesafe-ai/jev");
+    expect(result.answers.category.choice).toBe("billing");
+    expectTypeOf(result.answers.category.choice).toEqualTypeOf<"billing" | "support">();
+    expectTypeOf(result.answers.urgent.probability).toEqualTypeOf<number>();
+    expectTypeOf(result.answers.priority.score).toEqualTypeOf<number>();
+    expect(result.usage).toEqual({ inputTokens: 42, outputTokens: 3, totalTokens: 45 });
+    expect(result.providerMetadata).toEqual({ fixture: { requestId: "request-1" } });
+  });
+
+  it("resolves an explicit model ID through the local connection", async () => {
+    localEvaluationModel.mockReturnValue(evaluationModel().model);
+    await evaluate({ model: "typesafe-ai/custom", state, questions });
+    expect(localEvaluationModel).toHaveBeenCalledWith("typesafe-ai/custom");
+  });
+
+  it("preserves the configured default provider for defaults and aliases", async () => {
+    const evaluator = evaluationModel();
+    const factory = vi.fn(() => evaluator.model);
+    vi.stubGlobal("AI_SDK_DEFAULT_PROVIDER", { evaluationModel: factory });
+    localEvaluationModel.mockReturnValue(evaluationModel().model);
+
+    await evaluate({ state, questions });
+    await evaluate({ model: "internal-evaluator", state, questions });
+
+    expect(factory.mock.calls).toEqual([["typesafe-ai/jev"], ["internal-evaluator"]]);
+    expect(localEvaluationModel).not.toHaveBeenCalled();
+  });
+
+  it("passes explicit model instances and request options through unchanged", async () => {
+    const evaluator = evaluationModel();
+    const abortSignal = new AbortController().signal;
+    await evaluate({
+      model: evaluator.model,
+      state,
+      questions,
+      abortSignal,
+      headers: { "x-request-id": "request-1" },
+      providerOptions: { fixture: { mode: "fast" } },
+      maxRetries: 0,
+    });
+
+    expect(localEvaluationModel).not.toHaveBeenCalled();
+    expect(evaluator.doEvaluate).toHaveBeenCalledWith(
+      expect.objectContaining({
+        state,
+        questions,
+        abortSignal,
+        headers: expect.objectContaining({ "x-request-id": "request-1" }),
+        providerOptions: { fixture: { mode: "fast" } },
+      }),
+    );
+  });
+
+  it("preserves input and answer validation", async () => {
+    const evaluator = evaluationModel();
+    await expect(evaluate({ model: evaluator.model, state, questions: {} })).rejects.toThrow();
+    expect(evaluator.doEvaluate).not.toHaveBeenCalled();
+
+    const invalid = new Experimental_EvaluationMockModelV4({
+      doEvaluate: async () => ({ answers: {}, warnings: [] }),
+    });
+    await expect(evaluate({ model: invalid, state, questions })).rejects.toThrow(
+      "exactly one answer",
+    );
+  });
+
+  it("propagates provider errors and aborts before provider I/O", async () => {
+    const error = new Error("Evaluation unavailable.");
+    const failed = new Experimental_EvaluationMockModelV4({
+      doEvaluate: async () => {
+        throw error;
+      },
+    });
+    await expect(evaluate({ model: failed, state, questions, maxRetries: 0 })).rejects.toBe(error);
+
+    const evaluator = evaluationModel();
+    await expect(
+      evaluate({
+        model: evaluator.model,
+        state,
+        questions,
+        abortSignal: AbortSignal.abort(error),
+      }),
+    ).rejects.toBe(error);
+    expect(evaluator.doEvaluate).not.toHaveBeenCalled();
+  });
+});

packages/eve/src/experimental/evaluate/evaluate.ts

@@ -0,0 +1,26 @@
+import {
+  experimental_evaluate as evaluateWithAiSdk,
+  type Experimental_EvaluationModel as EvaluationModel,
+  type Experimental_EvaluationQuestion as EvaluationQuestion,
+} from "ai";
+
+import { localGatewayEvaluationModel } from "#internal/model-auth/transport.js";
+
+export const DEFAULT_EVALUATION_MODEL = "typesafe-ai/jev";
+
+/** Evaluate typed questions about the state you pass to it using eve's model authentication. */
+export function evaluate<const Questions extends Record<string, EvaluationQuestion>>({
+  model = DEFAULT_EVALUATION_MODEL,
+  ...options
+}: Omit<Parameters<typeof evaluateWithAiSdk<Questions>>[0], "model"> & {
+  /** Evaluation model instance or ID. Defaults to TypeSafe Jev. */
+  model?: EvaluationModel;
+}) {
+  return evaluateWithAiSdk({
+    ...options,
+    model:
+      typeof model === "string" && Reflect.get(globalThis, "AI_SDK_DEFAULT_PROVIDER") == null
+        ? (localGatewayEvaluationModel(model) ?? model)
+        : model,
+  });
+}

packages/eve/src/experimental/evaluate/package.scenario.test.ts

@@ -26,6 +26,20 @@ export const evaluationModel = {
 import { autoModel } from "eve/experimental/evaluate";
 import { anthropic } from "eve/models/anthropic";
 export default defineAgent({ model: autoModel({ options: { "openai/gpt-5.6-sol": "Investigations", my_secret_model: { model: anthropic("sonnet-5"), reasoning: "low", description: "Routine work" } } }) });`,
+      "agent/tools/classify.ts": `import { defineTool } from "eve/tools";
+import { evaluate } from "eve/experimental/evaluate";
+export default defineTool({
+  description: "Classify an incident",
+  inputSchema: { type: "object", properties: {}, additionalProperties: false },
+  async execute(_input, ctx) {
+    const result = await evaluate({
+      state: { incident: "Alice needs an export summary." },
+      questions: { category: { type: "choice", instructions: "Choose a category.", criteria: { routine: "Routine work", investigation: "Investigation" } } },
+      abortSignal: ctx.abortSignal,
+    });
+    return result.answers.category.choice;
+  },
+});`,
       "agent/subagents/worker/instructions.md": "Review the assigned evidence.",
       "agent/subagents/worker/agent.ts": `import { defineAgent } from "eve";
 import { autoModel } from "eve/experimental/evaluate";

packages/eve/src/internal/authored-module-loader.ts

@@ -210,7 +210,10 @@ export async function bundleExtensionDistributionGraph(input: {
   readonly entries: readonly ExtensionDistributionGraphEntry[];
   readonly packageRoot: string;
   readonly runtimeDependencies: readonly string[];
-}): Promise<ReadonlyMap<string, string>> {
+}): Promise<{
+  readonly files: ReadonlyMap<string, string>;
+  readonly imports: readonly string[];
+}> {
   const plugins = [
     createAuthoredDirectiveGuardPlugin(),
     createAuthoredRelativeExtensionResolverPlugin({ extensions: RESOLVE_EXTENSIONS }),
@@ -248,12 +251,14 @@ export async function bundleExtensionDistributionGraph(input: {
     });
 
     const files = new Map<string, string>();
+    const imports = new Set<string>();
     for (const item of result.output) {
       if (item.type === "chunk") {
         files.set(item.fileName, removeRolldownModuleRegionComments(item.code));
+        for (const specifier of [...item.imports, ...item.dynamicImports]) imports.add(specifier);
       }
     }
-    return files;
+    return { files, imports: [...imports] };
   } catch (error) {
     throw createAuthoredModuleBundleError(input.packageRoot, error);
   }

packages/eve/src/internal/bundler/nitro-rolldown.ts

@@ -5,6 +5,8 @@ type RolldownOutputChunk = {
   readonly type: "chunk";
   readonly code: string;
   readonly fileName: string;
+  readonly imports: readonly string[];
+  readonly dynamicImports: readonly string[];
 };
 
 type RolldownOutputAsset = {

packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts

@@ -1,4 +1,4 @@
-import { mkdir, mkdtemp, readFile, symlink, writeFile } from "node:fs/promises";
+import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises";
 import { createRequire } from "node:module";
 import { tmpdir } from "node:os";
 import { dirname, join } from "node:path";
@@ -7,7 +7,9 @@ import { pathToFileURL } from "node:url";
 import { describe, expect, it } from "vitest";
 
 import {
+  EXTENSION_CAPABILITY_SUPPORT,
   EXTENSION_CAPABILITY_VERSIONS,
+  findUnsupportedExtensionCapabilities,
   parseExtensionCompatibilityManifest,
 } from "#compiler/extension-compatibility.js";
 import {
@@ -81,6 +83,101 @@ describe("extension build output", () => {
     );
   });
 
+  it.each([
+    ["static re-export", 'export { evaluate as assess } from "eve/experimental/evaluate";'],
+    [
+      "namespace import",
+      'import * as evaluation from "eve/experimental/evaluate"; export const assess = evaluation.evaluate;',
+    ],
+    [
+      "dynamic import",
+      'export async function assess(options: Parameters<typeof import("eve/experimental/evaluate").evaluate>[0]): Promise<void> { await (await import("eve/experimental/evaluate")).evaluate(options); }',
+    ],
+  ])(
+    "stamps evaluation capabilities for a hook-only extension using a %s",
+    async (_name, helper) => {
+      const root = await createExtensionPackage();
+      await rm(join(root, "extension", "tools"), { recursive: true });
+      await mkdir(join(root, "extension", "hooks"));
+      await mkdir(join(root, "extension", "lib"));
+      await writeFile(join(root, "extension", "lib", "evaluation.ts"), helper);
+      await writeFile(
+        join(root, "extension", "hooks", "evaluate.ts"),
+        `import { defineHook } from "eve/hooks";
+import { assess } from "../lib/evaluation";
+export default defineHook({ events: { "turn.started": async () => {
+  await assess({ state: { request: "Alice needs a summary." }, questions: {
+    routine: { type: "boolean", instructions: "Is this routine work?" }
+  } });
+} } });`,
+      );
+      const config = await tryReadExtensionBuildConfig(root);
+      const outDir = await buildExtensionPackage(root, config!);
+      const manifestPath = join(outDir, "extension", "_manifest.json");
+      const manifest = parseExtensionCompatibilityManifest(
+        await readFile(manifestPath, "utf8"),
+        manifestPath,
+      );
+
+      expect(manifest.requires).toEqual({
+        extension: EXTENSION_CAPABILITY_VERSIONS.extension,
+        hook: EXTENSION_CAPABILITY_VERSIONS.hook,
+        tool: EXTENSION_CAPABILITY_VERSIONS.tool,
+        dynamicTool: EXTENSION_CAPABILITY_VERSIONS.dynamicTool,
+      });
+      expect(findUnsupportedExtensionCapabilities(manifest)).toEqual([]);
+      expect(
+        findUnsupportedExtensionCapabilities(manifest, {
+          ...EXTENSION_CAPABILITY_SUPPORT,
+          tool: EXTENSION_CAPABILITY_SUPPORT.tool.filter((version) => version < 47),
+        }),
+      ).toEqual([
+        expect.objectContaining({
+          capability: "tool",
+          requiredVersion: EXTENSION_CAPABILITY_VERSIONS.tool,
+        }),
+      ]);
+    },
+  );
+
+  it.each([
+    [
+      "autoModel",
+      'import { autoModel } from "eve/experimental/evaluate"; export const route: ReturnType<typeof autoModel> = autoModel({ options: { "openai/small": "Routine work" } });',
+      true,
+    ],
+    [
+      "type-only import",
+      'import type { evaluate } from "eve/experimental/evaluate"; export type Evaluate = typeof evaluate;',
+      false,
+    ],
+  ])(
+    "tracks runtime evaluation imports in a tool-free helper: %s",
+    async (_name, helper, runtime) => {
+      const root = await createExtensionPackage();
+      await rm(join(root, "extension", "tools"), { recursive: true });
+      await mkdir(join(root, "extension", "lib"));
+      await writeFile(join(root, "extension", "lib", "evaluation.ts"), helper);
+      const config = await tryReadExtensionBuildConfig(root);
+      const outDir = await buildExtensionPackage(root, config!);
+      const manifestPath = join(outDir, "extension", "_manifest.json");
+      const manifest = parseExtensionCompatibilityManifest(
+        await readFile(manifestPath, "utf8"),
+        manifestPath,
+      );
+
+      expect(manifest.requires).toEqual({
+        extension: EXTENSION_CAPABILITY_VERSIONS.extension,
+        ...(runtime
+          ? {
+              tool: EXTENSION_CAPABILITY_VERSIONS.tool,
+              dynamicTool: EXTENSION_CAPABILITY_VERSIONS.dynamicTool,
+            }
+          : {}),
+      });
+    },
+  );
+
   it("stamps extension-owned external dependencies into compatibility metadata", async () => {
     const root = await createExtensionPackage({
       dependencies: { "layout-sensitive-runtime": "1.0.0" },

packages/eve/src/internal/nitro/host/build-extension.ts

@@ -68,7 +68,7 @@ export async function buildExtensionPackage(
   let preserveTransactionRoot = false;
   try {
     await mkdir(stagedDistRoot, { recursive: true });
-    await emitExtensionDistribution({
+    const runtimeImports = await emitExtensionDistribution({
       appRoot,
       declarationModule,
       declarationsRoot: join(transactionRoot, "declarations"),
@@ -88,6 +88,7 @@ export async function buildExtensionPackage(
         ? {}
         : { build: { externalDependencies: config.externalDependencies } }),
       requires: await deriveExtensionCapabilityRequirements({
+        runtimeImports,
         declarationModule,
         manifest,
         packageName: config.packageName,

packages/eve/src/internal/nitro/host/extension-capability-requirements.ts

@@ -21,6 +21,7 @@ export async function deriveExtensionCapabilityRequirements(input: {
   readonly manifest: AgentSourceManifest;
   readonly packageName: string;
   readonly runtimeDependencies: readonly string[];
+  readonly runtimeImports: readonly string[];
   readonly shortName: string;
   readonly sourceRoot: string;
 }): Promise<ExtensionCapabilityRequirements> {
@@ -89,6 +90,11 @@ export async function deriveExtensionCapabilityRequirements(input: {
     required.add("config");
   }
   if (usesState) required.add("state");
+  // The evaluation entrypoint can be used from any contribution or shared helper.
+  if (input.runtimeImports.includes("eve/experimental/evaluate")) {
+    required.add("tool");
+    required.add("dynamicTool");
+  }
 
   return Object.fromEntries(
     (Object.keys(EXTENSION_CAPABILITY_VERSIONS) as ExtensionCapability[])

packages/eve/src/internal/nitro/host/extension-distribution.ts

@@ -23,7 +23,7 @@ export async function emitExtensionDistribution(input: {
   readonly stagedDistRoot: string;
   readonly stagedOutDir: string;
   readonly transactionRoot: string;
-}): Promise<void> {
+}): Promise<readonly string[]> {
   const sourceFiles = await collectExtensionSourceFiles(input.sourceRoot);
   const skillPackageRoots = input.manifest.skills
     .filter((skill) => skill.sourceKind === "skill-package")
@@ -44,7 +44,7 @@ export async function emitExtensionDistribution(input: {
     packageRoot: input.appRoot,
     runtimeDependencies: input.runtimeDependencies,
   });
-  for (const [fileName, code] of emitted) {
+  for (const [fileName, code] of emitted.files) {
     const outputPath = join(input.stagedOutDir, fileName);
     await mkdir(dirname(outputPath), { recursive: true });
     await writeFile(outputPath, code, "utf8");
@@ -69,6 +69,7 @@ export async function emitExtensionDistribution(input: {
     );
   }
   await emitDeclarationBarrels(input);
+  return emitted.imports;
 }
 
 /**

packages/eve/src/public/experimental/evaluate/index.ts

@@ -1 +1,2 @@
 export { autoModel } from "#experimental/evaluate/auto-model.js";
+export { evaluate } from "#experimental/evaluate/evaluate.js";

research/jev-decision-models.md

@@ -1,24 +1,26 @@
 ---
 issue: "TBD (maintainer-requested research; no matching issue found)"
 status: implemented
-last_updated: "2026-09-16"
+last_updated: "2026-09-17"
 ---
 
 # Evaluation model routing in eve
 
 ## Recommendation
 
-Add only `autoModel` at `eve/experimental/evaluate`. Build it on AI SDK's
-`experimental_evaluate` API instead of owning a TypeSafe client, AI Gateway
-transport, credential resolver, or public decision API. Default to the
-`typesafe-ai/jev` model string. AI SDK model strings use its default
-provider, normally Vercel AI Gateway; explicit model instances use the provider
-package that created them.
+Expose `autoModel` and standalone `evaluate` at `eve/experimental/evaluate`.
+Build both on AI SDK's `experimental_evaluate` API. `evaluate` accepts the SDK's
+state, typed questions, and request options, with an optional model defaulting to
+`typesafe-ai/jev`. It returns the SDK's typed answers and response metadata.
 
-Keep the first integration narrow: select an allowlisted language model from the
-incoming task. Defer a generic agent-facing decision tool. Asking the main model
-to generate tool arguments adds output tokens and latency before the cheaper
-evaluation can happen.
+`autoModel` calls the shared `evaluate` wrapper. Model strings use the configured
+AI SDK default provider; without an override, the wrapper resolves eve's local
+Gateway connection when available and otherwise leaves Gateway resolution to the
+SDK. Explicit provider instances retain their own authentication.
+
+Tool authors can call `evaluate` with structured state and pass `ctx.abortSignal`.
+Application code can use it without an active eve session. Standalone calls are
+not cached; durable per-turn selection remains specific to `autoModel`.
 
 The API is experimental because AI SDK's evaluation model specification is also
 experimental and can change in patch releases. The implemented authoring API is
@@ -160,12 +162,10 @@ direct provider behavior.
 
 ## Scope boundaries
 
-The first release intentionally does not expose `experimental_evaluate`, a
-TypeSafe-specific `decide` function, decision schemas, direct HTTP clients,
-credentials, custom retry logic, provider fallbacks, confidence thresholds, or
-an agent-callable decision tool. Applications that need arbitrary evaluations
-can call AI SDK directly. eve adds only the lifecycle and durable routing policy
-needed to use an evaluation result as an agent model.
+eve owns the `evaluate` wrapper and its default model and authentication behavior.
+AI SDK owns evaluation schemas, validation, retries, errors, and result metadata.
+The integration adds no TypeSafe-specific `decide` function, direct HTTP client,
+provider fallback, confidence threshold, or built-in agent-callable decision tool.
 
 No live Jev inference or independent quality, latency, or cost benchmark was run
 for this research. Deterministic tests use AI SDK evaluation model mocks and