diff --git a/.changeset/unify-background-workflows.md b/.changeset/unify-background-workflows.md new file mode 100644 index 000000000..d36720586 --- /dev/null +++ b/.changeset/unify-background-workflows.md @@ -0,0 +1,11 @@ +--- +"eve": minor +--- + +Require durable background tools to use `defineWorkflowTool`. Remove background execution from `defineTool` and dynamic tools, including the `TaskExec` and `postMessage` authoring APIs, and deliver each background cohort's completed, failed, and cancelled outcomes in one automatic report. + +Background invocations share workflow execution and cancellation cleanup. Agent settlement records usage once before its enclosing workflow returns a tool result. Parent sessions retain task outcomes, and late results cannot overwrite a recorded cancellation; channel task views no longer include executor bindings. Background workflow yields are consumed without publishing progress or retaining a task-progress stream. + +Align `subagent.completed` for blocking and background agents: emit the actual output only after the parent records success. Background receipts remain `action.result` tool outputs; completion events no longer announce admission or wait for cohort reporting. + +Use task lifecycle values in subagent eval assertions: replace `status: "pending"` with `"working"` and `"rejected"` with `"failed"`. Explicit cancelled child outcomes now retain `"cancelled"` instead of appearing as failures. diff --git a/docs/concepts/execution-model-and-durability.mdx b/docs/concepts/execution-model-and-durability.mdx index c0c12ba63..12cfa966d 100644 --- a/docs/concepts/execution-model-and-durability.mdx +++ b/docs/concepts/execution-model-and-durability.mdx @@ -118,12 +118,11 @@ Some work has to wait, including a human approving a [tool](../tools) or an inte Background execution is a separate choice about result delivery. A background workflow tool returns a task receipt so the parent turn can continue; the tool's workflow may then run or suspend independently. A default workflow tool can suspend too, with its original tool call -still pending. Ordinary background tools run their executor inside the initiating step and do -not gain durable waits by setting `execution: "background"`. +still pending. Declare background work with `defineWorkflowTool`; `defineTool` always executes +inside its initiating step. -A generator's `yield` reports progress; an awaited workflow operation provides the durable wait. -For background tools, ordinary yields are stream-only, while `yield task.postMessage(...)` -requests a parent-agent turn. See [background execution](../tools#background-execution) and +An awaited workflow operation provides the durable wait. A generator's `yield` reports progress +in default execution; background workflow tools consume yields without publishing progress. See [background execution](../tools#background-execution) and [workflow suspension](../tools/workflows#how-suspension-works) for the execution and result rules. ## Message delivery and steering diff --git a/docs/concepts/sessions-runs-and-streaming.md b/docs/concepts/sessions-runs-and-streaming.md index 222726544..e99b6d06a 100644 --- a/docs/concepts/sessions-runs-and-streaming.md +++ b/docs/concepts/sessions-runs-and-streaming.md @@ -80,7 +80,7 @@ The stream is newline-delimited JSON (NDJSON), one event per line: | `input.requested` | The run paused for human input ([HITL](/docs/human-in-the-loop) approval or `ask_question`); carries `requests`. | | `input.resolved` | The server accepted terminal human-input outcomes; carries `resolutions` with responses when provided. | | `subagent.called` | A subagent was delegated; carries `childSessionId` to attach to. | -| `subagent.completed` | A background subagent was admitted and returned its task receipt. | +| `subagent.completed` | The parent recorded a successful subagent invocation result; carries the actual output. | | `reasoning.appended` | A reasoning text delta. | | `reasoning.completed` | The finalized reasoning block. | | `message.appended` | An assistant text delta. | @@ -119,7 +119,7 @@ Note: consider the privacy, confidentiality, and user-experience implications fo When a task explicitly requires conditional delivery and there is nothing new to report, the agent can finish with exactly ``. eve emits `message.completed` with `message: null` for that intentional silence. The marker must be the entire response, apart from surrounding whitespace; the HTML-escaped form `<eve-empty-delivery/>` is also accepted. A response that quotes the marker in prose or code is delivered normally. -A delegated subagent publishes progress on its own child-session stream. The parent emits `subagent.called` with a `childSessionId`, which a client uses to attach. `subagent.completed` carries a working task receipt after admission; terminal outcomes arrive as task-triggered `message.received` notifications with `data.kind: "execution.background_task"`. The default frontend reducer retains these events in `events` but does not project their runtime-authored text as participant messages. +A delegated subagent publishes progress on its own child-session stream. The parent emits `subagent.called` with a `childSessionId`, which a client uses to attach. Both blocking and background calls emit `subagent.completed` with the actual output after the parent records a successful outcome. A background working receipt is an `action.result` tool output; it does not emit completion. Older streams can contain receipt-bearing completion events marked by `data.backgroundTask`; those are admission only. Completion does not mean that the reusable child session has ended. Background task terminal outcomes also arrive as task-triggered `message.received` notifications with `data.kind: "execution.background_task"`. The default frontend reducer retains these events in `events` but does not project their runtime-authored text as participant messages. `step.failed` and `turn.failed` carry `{ code, message, details? }` for the failed fragment or turn, and `session.failed` is the terminal session-level variant. `turn.cancelled` is not a failure: the cancelled turn ends without any failure event, `session.waiting` follows, and the session accepts the next message normally. Whatever the turn streamed before cancellation stays on the stream. Durable history keeps the accepted user input and previously settled work, but discards incomplete assistant output and unfinished tool state. When a turn requested an output schema, the finalized payload lands on `result.completed` as `data.result` before the turn boundary. `authorization.required` carries the sign-in challenge (`data.authorization` may include `url`, `userCode`, `expiresAt`, `instructions`), and `authorization.completed` carries `data.outcome` (`"authorized" | "declined" | "failed" | "timed-out"`). diff --git a/docs/evals/assertions.mdx b/docs/evals/assertions.mdx index f85eea8bb..215a11b06 100644 --- a/docs/evals/assertions.mdx +++ b/docs/evals/assertions.mdx @@ -83,7 +83,11 @@ Pick the cheapest builder that captures what "correct" means. When exact match i ## The matcher mini-language -`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default; use `"pending"`, `"failed"`, or `"rejected"` explicitly for lifecycle checks. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls. +`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default. Tool lifecycle checks also accept `"pending"`, `"failed"`, and `"rejected"`. Subagent checks use the shared task status values: `"working"`, `"input_required"`, `"completed"`, `"failed"`, and `"cancelled"`. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls. + +A background subagent that has only returned a working task receipt remains `working`; use `t.calledSubagent(name, { status: "working" })` to assert that delegation. Both blocking and background calls become `completed` when their actual successful result is observed through `subagent.completed`, even when the child session stays available for another call. A rejected dispatch is `failed`; an explicit cancelled child outcome is `cancelled`. + +The runner derives these values from captured events. It does not read stored task state, and the current stream does not expose every task transition. A proxied input request alone does not update the subagent call to `input_required`; assert that request with `requireInputRequest`. Check task notifications for background failure and cancellation when no child outcome event is present. Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean: diff --git a/docs/guides/frontend/overview.mdx b/docs/guides/frontend/overview.mdx index 373ad7fde..4463c9553 100644 --- a/docs/guides/frontend/overview.mdx +++ b/docs/guides/frontend/overview.mdx @@ -172,7 +172,7 @@ supplied `session` remains externally owned and is never replaced by prewarming `data.messages` are eve-owned `EveMessage[]`. Common text, reasoning, file, and dynamic-tool parts follow the [AI SDK `UIMessage`](https://ai-sdk.dev/docs/reference/ai-sdk-core/ui-message) rendering convention, but the types are not interchangeable. eve also exposes authorization and HITL metadata, and a file part's URL can be absent. Adapt those parts before passing messages to an API typed as `UIMessage[]`. -When the root agent delegates, its stream emits `subagent.called` with the child's `childSessionId`, then `subagent.completed` after admission with a working task receipt. Later task notifications wake the parent with completion, failure, or cancellation. Detailed child progress lives on the child session's stream instead of being flattened into the root `data.messages`. Use the lower-level [TypeScript client](../client/overview#sessions) to attach to that ID when your UI needs live subagent activity. See [What the parent sees](../../subagents#what-the-parent-sees) for the complete contract. +When the root agent delegates, its stream emits `subagent.called` with the child's `childSessionId`, then `subagent.completed` when the parent records a successful result, for both blocking and background invocations. A background working receipt appears in `action.result`, without a completion event. The child session can remain available for reuse. Task notifications wake the parent with completion, failure, or cancellation. Detailed child progress lives on the child session's stream instead of being flattened into the root `data.messages`. Use the lower-level [TypeScript client](../client/overview#sessions) to attach to that ID when your UI needs live subagent activity. See [What the parent sees](../../subagents#what-the-parent-sees) for the complete contract. ## Sending and streaming diff --git a/docs/subagents/index.mdx b/docs/subagents/index.mdx index 3dc153e9d..a38a3b293 100644 --- a/docs/subagents/index.mdx +++ b/docs/subagents/index.mdx @@ -9,15 +9,16 @@ eve supports two ways to delegate work: the root-only built-in `agent` tool, whi Overlapping background work owned by a session forms a cohort. Tasks launched in later user turns join the open cohort while earlier work is still pending or its -successful results await delivery. eve holds successful completion notifications -until every task in that cohort has completed, failed, or been cancelled, then -delivers the successful results together in one parent turn. Partial completions -do not invoke the parent model. Work started after the cohort settles forms a new -cohort. No configuration or debounce timer is required. +terminal results await delivery. eve holds completion, failure, and cancellation +notifications until every task in that cohort is terminal, then delivers all outcomes +together in one parent turn. This includes cohorts where every task failed or was +cancelled. Partial settlements do not invoke the parent model. Work started after the +cohort settles forms a new cohort. No configuration or debounce timer is required. -User messages, input requests, authorization events, failures, and cancellation -are handled without waiting for the cohort. Child lifecycle events are processed -before completion delivery so usage accounting and handle cleanup stay ordered. +User messages, input requests, and authorization events are handled without waiting +for the cohort. Failure and cancellation update task lifecycle state immediately; +their automatic conversational report waits for the cohort. Child lifecycle events +are processed before terminal delivery so usage accounting and handle cleanup stay ordered. To follow work in progress, subscribe to the child session streams. ## The built-in `agent` tool @@ -234,6 +235,8 @@ Do not rely on subagent delegation by itself as an approval boundary. Put sensit Each delegated subagent spins up its own child session and stream. The parent stream carries the control-plane events `subagent.called` and `subagent.completed`, plus interactive `input.requested`, `authorization.required`, and `authorization.completed` events proxied from descendants so the root channel can prompt the user. To follow the child's other progress, read `subagent.called.data.childSessionId` and subscribe at `GET /eve/v1/session/:childSessionId/stream`. +Both blocking and background invocations emit `subagent.called` after dispatch and `subagent.completed` after the parent records successful completion. Completion carries the actual output and identifies the same invocation by `callId`; the child session can remain available for reuse. A background working receipt is returned as an `action.result` tool output and does not emit completion. Background completion events do not wait for the cohort report. Failure and cancellation are not successful completion; task notifications retain those outcomes. + Channels with activity reporting attribute the backing agent's tool activity to its background task. Local and remote subagents share the task's progress item rather than adding a separate agent item. Continuing or steering a child with `agentId` attaches its new activity to the new task; the previous task keeps its completed or cancelled status. This applies to the built-in `agent` and declared subagent tools, not to child invocations inside custom workflow tools. Activity reporting is best-effort and does not change task execution or result delivery. A background task that was already admitted survives cancellation of the turn that started it; background work that has not yet been admitted is rejected with the cancelled step. Use `task_cancel` to stop an admitted task. Cancellation delivers the task's final notification to an active parent even if the task must be stopped forcibly. Parent-session finalization cancels remaining live tasks. diff --git a/docs/tools/overview.mdx b/docs/tools/overview.mdx index 808428143..42a83d822 100644 --- a/docs/tools/overview.mdx +++ b/docs/tools/overview.mdx @@ -121,33 +121,25 @@ receive only the projected text. ### Background execution -Background execution controls how a tool delivers its result to the parent agent. Durable -suspension controls whether the executor can pause at a workflow wait and release compute. -These are independent choices: `execution: "background"` selects the task lifecycle; -`defineWorkflowTool` with a leading `"use workflow"` directive enables durable suspension. +Background execution controls how a workflow tool delivers its result to the parent agent. +Set `execution: "background"` on `defineWorkflowTool`; ordinary `defineTool` calls always finish +inside the initiating step. -| Definition | Durable waits inside `execute` | Result delivered to the model | -| ------------------------------------------------ | --------------------------------------------------------------- | ---------------------------------------------------------------------- | -| `defineTool` | No; the executor runs inside the initiating step. | Executor output after it finishes. | -| `defineTool` + `execution: "background"` | No; the initiating step still waits for the executor to finish. | Task receipt, followed by task notifications. | -| `defineWorkflowTool` | Yes; the tool call waits while the workflow can suspend. | Workflow output after it finishes. | -| `defineWorkflowTool` + `execution: "background"` | Yes; the workflow body can outlive the initiating step. | Task receipt before the body finishes, followed by task notifications. | +| Definition | Durable waits inside `execute` | Result delivered to the model | +| ------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| `defineTool` | No; the executor runs inside the initiating step. | Executor output after it finishes. | +| `defineWorkflowTool` | Yes; the tool call waits while the workflow can suspend. | Workflow output after it finishes. | +| `defineWorkflowTool` + `execution: "background"` | Yes; the workflow body can outlive the initiating step. | Task receipt before the body finishes; one combined report after all tasks in its cohort are terminal. | -A background tool receives a third `task` argument. Its tool result is the receipt -`{ status: "working", taskId }`; its eventual output belongs to the task. Use a background -workflow tool when the conversation should continue while the body waits for a person, -webhook, or timer. Setting `execution: "background"` on an ordinary tool does not move its -executor into a separate workflow or make an ordinary Promise a durable wait. +A background workflow tool's result is the receipt `{ status: "working", taskId }`; its eventual +output belongs to the task. Use this mode when the conversation should continue while the body +waits for a person, webhook, or timer. For background tools, `outputSchema` and `toModelOutput` describe the fixed receipt. The body's return value is available through the completed task's output. -`task.delegated()` has been removed. Move external work into a `defineWorkflowTool` executor and -return its result when it finishes. Rebuild extensions using the removed API after migrating. - -`yield task.postMessage(message)` is the only yield that requests a parent-agent turn. Calling -`task.postMessage` only constructs the message descriptor; you must yield it to send it. -In a workflow body, keep values needed after a wait in local variables; workflow replay +The `TaskExec` argument and `task.postMessage` have been removed. Return the terminal result; +background yields are consumed without publishing progress or retaining a task-progress stream. In a workflow body, keep values needed after a wait in local variables; workflow replay reconstructs them across durable waits. Use [`ctx.ask`](/docs/tools/workflows#ask-a-human-ctxask) when the body needs an answer from the human. Background tools are a normal execution mode and need no root-agent flag. Built-in, declared local, @@ -158,10 +150,11 @@ cohort and instructs the model to acknowledge that the work started without wait [Schedule](/docs/schedules)-initiated turns are the exception: no user prompted them, so their launches keep conditional delivery and send no acknowledgement. Later task-triggered parent turns receive fresh state for the same cohort. Once every related -task is terminal, the state includes their outputs so the model can combine the useful results. +task is terminal, one automatic parent turn receives the cohort's completed, failed, and cancelled +outcomes so the model can combine the useful results. eve does not instruct the model to stay silent while sibling tasks are pending; a new user question can receive an answer while background work continues. [Completion batching](/docs/subagents#completion-batching) -combines successful results already queued for the parent. +combines terminal results already queued for the parent. ### Yield and return @@ -169,19 +162,18 @@ combines successful results already queued for the parent. It does not by itself create a durable wait for a person, timer, or external event. The value's meaning depends on the tool's execution mode: -| Executor | Ordinary `yield value` | Final output | -| ------------------------------------------------ | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -| `defineTool`, default execution | Earlier values are `action.partial` snapshots; the final yield becomes the tool result. | Last yielded value; a generator `return` value is ignored. | -| `defineWorkflowTool`, default execution | Every yield is an `action.partial` snapshot. | Explicit return value; falls back to the last yield if the return is `null` or `undefined`, then to `null`. | -| Either definition with `execution: "background"` | Stream-only task progress; does not request a parent-agent turn. | Explicit return value becomes the task output; no return completes with `null`. The last yield is not a fallback. | +| Executor | Ordinary `yield value` | Final output | +| --------------------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| `defineTool`, default execution | Earlier values are `action.partial` snapshots; the final yield becomes the tool result. | Last yielded value; a generator `return` value is ignored. | +| `defineWorkflowTool`, default execution | Every yield is an `action.partial` snapshot. | Explicit return value; falls back to the last yield if the return is `null` or `undefined`, then to `null`. | +| `defineWorkflowTool` with `execution: "background"` | Consumed without publishing progress or requesting a parent-agent turn. | Explicit return value becomes the task output; no return completes with `null`. The last yield is not a fallback. | -For either kind of background tool, `yield task.postMessage(message)` sends a message to the -parent agent while the task remains open. Return completes the task; throw fails it. These -notifications are separate from the original tool call's receipt. +For a background workflow tool, return completes the task and throw fails it. The automatic +cohort report is separate from the original tool call's receipt. Progress snapshots do not enter model history as intermediate tool results. For a default executor, the final output selected by the rules above becomes the tool result. For a background -executor, task messages and completion notifications provide input to later parent-agent turns. +executor, the terminal cohort report provides input to a later parent-agent turn. ### Workflows as tools diff --git a/docs/tools/workflows.mdx b/docs/tools/workflows.mdx index b105ec900..ae701bb0f 100644 --- a/docs/tools/workflows.mdx +++ b/docs/tools/workflows.mdx @@ -178,8 +178,7 @@ agent should receive a tool result: Use default execution when the model needs the answer to continue. Use background execution when the conversation should continue while the task is pending. Background tools need no root-agent -flag. Ordinary `defineTool` also accepts `execution: "background"`, but its executor still runs -inside the initiating step and cannot suspend at workflow waits. See the +flag. `defineTool` does not accept background execution. See the [tool execution comparison](/docs/tools#background-execution). ### How suspension works @@ -429,23 +428,23 @@ This pattern controls specialist selection, not whether the parent model delegat ## Report progress: `yield` -A workflow body may be an async generator. Ordinary yields report progress in both execution -modes. After processing a yield, eve advances the generator; use an awaited workflow operation +A workflow body may be an async generator. In default execution, yields report progress. +Background execution consumes yielded values without publishing them or retaining a task-progress +stream. After processing a yield, eve advances the generator; use an awaited workflow operation when the body needs to suspend. -| Operation | Default execution | `execution: "background"` | -| --------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------- | -| `yield value` | Emits an `action.partial` snapshot for the pending tool call. | Emits stream-only task progress; does not request a parent-agent turn. | -| `yield task.postMessage(message)` | Unavailable; there is no `task` argument. | Sends a message requesting a parent-agent turn; the task stays open. | -| `return value` | Settles the tool call with its output. | Completes the task and delivers its output in a later notification. | -| No return value | Uses the last yield as output, or `null` if there were no yields. | Completes with `null`; yielded progress is not the task output. | +| Operation | Default execution | `execution: "background"` | +| --------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------- | +| `yield value` | Emits an `action.partial` snapshot for the pending tool call. | Consumes the value without publishing progress or requesting a parent-agent turn. | +| `return value` | Settles the tool call with its output. | Completes the task and contributes to its cohort's terminal report. | +| No return value | Uses the last yield as output, or `null` if there were no yields. | Completes with `null`; yielded progress is not the task output. | For default execution, an explicit `return null` also falls back to the last yield. Prefer an explicit object return when progress snapshots and the final result have different shapes. Progress snapshots are last-write-wins by tool call id and do not enter model history as intermediate tool results. A snapshot used as the final output does enter history as the tool result. -This background workflow reports progress, sends the parent one message, and then suspends: +This background workflow consumes an intermediate yield, then suspends until its timer expires: ```ts title="agent/tools/remind_with_progress.ts" import { defineWorkflowTool } from "eve/tools"; @@ -453,25 +452,22 @@ import { sleep } from "workflow"; import { z } from "zod"; export default defineWorkflowTool({ - description: "Schedule a reminder and report progress before waiting.", + description: "Schedule a reminder.", inputSchema: z.object({ note: z.string(), delay: z.string() }), execution: "background", - async *execute({ note, delay }, ctx, task) { + async *execute({ note, delay }) { "use workflow"; - yield { status: "preparing reminder" }; // Stream-only progress. - yield task.postMessage(`Reminder scheduled for ${delay}.`); // Parent-agent message. + yield { status: "preparing reminder" }; // Consumed without publishing progress in background mode. await sleep(delay); // Durable suspension while the timer is pending. return { reminder: note }; // Task completion notification. }, }); ``` -Calling `task.postMessage` constructs a descriptor; yielding it sends the message. It does not -wait for a reply. Use [`ctx.ask`](#ask-a-human-ctxask) when the workflow needs a human answer. -Removing `execution: "background"` requires removing the `task` argument and message yield; the -ordinary progress yield and `await sleep(delay)` still work, but the model waits for the final -reminder as the tool result. See [yield and return](/docs/tools#yield-and-return) for the different -final-output rules of ordinary `defineTool` generators. +Use [`ctx.ask`](#ask-a-human-ctxask) when the workflow needs a human answer. Removing +`execution: "background"` makes the yield visible as progress and keeps the durable sleep; the model +waits for the final reminder as the tool result. See [yield and return](/docs/tools#yield-and-return) for the +final-output rules. ## Cancel and clean up: `ctx.abortSignal` @@ -497,6 +493,11 @@ as cancelled whether or not it did. A body parked on a hook or a `sleep` does no it is abandoned when the grace period ends. Steps that received the signal are how you clean up first. +The caller waits up to 35 seconds for a cooperatively cancelled run to settle before forcing it +to stop. This includes time for the run to publish its outcome after body cleanup. A background +task records its cancelled status before cleanup finishes, so that status alone does not mean +its work has stopped. + ## Workflow tool examples ### Approve with a deadline and an escalation @@ -607,8 +608,10 @@ it. ## Semantics One call, one result. A waiting tool's call resolves once, with the return value, the error, or a -cancellation. A background tool's call resolves once, with the receipt; everything after arrives as -separate session input. +cancellation. A background tool's call resolves once, with the receipt. Its terminal outcome arrives +in one automatic report after every task in its cohort is terminal. A cohort includes overlapping +background work in the same session, including tasks launched in later user turns while earlier +work remains open. While a waiting tool runs, the turn is parked. A `queue` message waits for it. A `steer` message cancels the turn, which cancels the run, which withdraws its requests. Input responses never steer. @@ -616,8 +619,12 @@ cancels the turn, which cancels the run, which withdraws its requests. Input res Background runs belong to the session. They survive turn completion and cancellation, appear in the session's task index, can be cancelled with `task_cancel`, and are cancelled when the session ends. +The parent session records each background task's outcome when it processes the child's notification. +The first terminal outcome it records is final: a late child result cannot replace a recorded +cancellation. Task completion does not require the wrapper workflow to have exited. + Errors follow the SDK. A thrown error in a step retries per the step's policy; `FatalError` does -not. An error that escapes the body fails the run. +not. An error that escapes the body fails the tool invocation. Starting a waiting workflow tool can also be retried. If dispatch is interrupted after starting a run, its retry starts another run, and both may execute. The parent tracks the run returned by the diff --git a/e2e/fixtures/agent-agent-tool-controls/evals/root-tool-false.eval.ts b/e2e/fixtures/agent-agent-tool-controls/evals/root-tool-false.eval.ts index 7686f304b..3eda30d34 100644 --- a/e2e/fixtures/agent-agent-tool-controls/evals/root-tool-false.eval.ts +++ b/e2e/fixtures/agent-agent-tool-controls/evals/root-tool-false.eval.ts @@ -9,7 +9,7 @@ export default defineEval({ turn.expectOk(); turn.messageIncludes("INTERNAL-ROOT-COPY-OK"); turn.calledTool("invoke-self", { count: 1 }); - turn.calledSubagent("agent", { count: 1, status: "pending" }); + turn.calledSubagent("agent", { count: 1, status: "completed" }); t.succeeded(); t.noFailedActions(); }, diff --git a/e2e/fixtures/agent-background-tools/agent/agent.ts b/e2e/fixtures/agent-background-tools/agent/agent.ts index 949d6a11d..84840de7e 100644 --- a/e2e/fixtures/agent-background-tools/agent/agent.ts +++ b/e2e/fixtures/agent-background-tools/agent/agent.ts @@ -2,7 +2,6 @@ import { e2eAgentConfig } from "@eve-e2e/config"; import { defineAgent } from "eve"; import { mockModel, type MockModelRequest, type MockModelResponse } from "eve/evals"; -const PROGRESS = "EXPORT-PROGRESS"; const RESULT = "EXPORT-COMPLETE"; const SCHEDULED = "BACKGROUND-EXPORT-SCHEDULED"; const EMPTY_DELIVERY_SENTINEL = ""; @@ -47,10 +46,6 @@ function respond(request: MockModelRequest): MockModelResponse | string { return "BACKGROUND-EXPORT-STARTED"; } - if (message.includes(PROGRESS)) { - return "BACKGROUND-EXPORT-UPDATE-RECEIVED"; - } - if ( message.includes("is completed") && (message.includes(RESULT) || message.includes("ship-it")) diff --git a/e2e/fixtures/agent-background-tools/agent/tools/export.ts b/e2e/fixtures/agent-background-tools/agent/tools/export.ts index ebbe9ee4b..ce3b36280 100644 --- a/e2e/fixtures/agent-background-tools/agent/tools/export.ts +++ b/e2e/fixtures/agent-background-tools/agent/tools/export.ts @@ -6,11 +6,11 @@ export default defineWorkflowTool({ description: "Start a durable background export.", execution: "background", inputSchema: z.strictObject({ query: z.string() }), - async *execute({ query }, _ctx, task) { + async *execute({ query }) { "use workflow"; yield { progress: 0.5 }; - yield task.postMessage(`Export ${task.taskId}: EXPORT-PROGRESS`); + yield { message: "EXPORT-PROGRESS", progress: 0.75 }; await sleep("250ms"); return { query, result: "EXPORT-COMPLETE" }; }, diff --git a/e2e/fixtures/agent-background-tools/evals/background-export.update-and-complete.eval.ts b/e2e/fixtures/agent-background-tools/evals/background-export.update-and-complete.eval.ts index 46d07674a..303b9495b 100644 --- a/e2e/fixtures/agent-background-tools/evals/background-export.update-and-complete.eval.ts +++ b/e2e/fixtures/agent-background-tools/evals/background-export.update-and-complete.eval.ts @@ -2,9 +2,11 @@ import { defineEval } from "eve/evals"; import { satisfies } from "eve/evals/expect"; import { defaultMessageReducer } from "eve/client"; +const RESULT = "EXPORT-COMPLETE"; + export default defineEval({ description: - "An authored background defineTool yields state and progress, explicitly posts a message, then completes; the parent sees both.", + "A background workflow streams progress and delivers one terminal report to the parent.", async test(t) { const started = await t.send("BACKGROUND-EXPORT-START"); const conversation = started.session; @@ -18,19 +20,8 @@ export default defineEval({ const sessionId = conversation.sessionId; if (sessionId === undefined) throw new Error("Eval has no parent session id."); - const updateLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(started.session, "update wait"), - }); - const updateTurn = await updateLive.result(); - updateTurn.expectOk(); - updateTurn.messageIncludes("BACKGROUND-EXPORT-UPDATE-RECEIVED"); - updateTurn.event("message.received", { - data: (data) => data.kind === "execution.background_task", - count: 1, - }); - const doneLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(updateLive.session, "completion wait"), + startIndex: requireStreamIndex(started.session, "completion wait"), }); const doneTurn = await doneLive.result(); doneTurn.expectOk(); @@ -40,8 +31,24 @@ export default defineEval({ count: 1, }); + await t.require( + doneTurn.events, + satisfies( + (events: typeof doneTurn.events) => + events.some( + (event) => + event.type === "message.received" && + messageText(event.data.message).includes( + `Background task ${taskId} (export) is completed.`, + ) && + messageText(event.data.message).includes(RESULT), + ), + "parent receives the executor completion with task identity", + ), + ); + const reducer = defaultMessageReducer(); - const projection = [...started.events, ...updateTurn.events, ...doneTurn.events].reduce( + const projection = [...started.events, ...doneTurn.events].reduce( (data, event) => reducer.reduce(data, event), reducer.initial(), ); @@ -60,6 +67,10 @@ export default defineEval({ "frontend projection keeps the background result without rendering runtime task input", ), ); + doneTurn.event("turn.started", { count: 1 }); + doneTurn.notEvent("message.received", { + data: (data) => messageText(data.message).includes("PROGRESS"), + }); t.noFailedActions(); }, }); @@ -78,3 +89,18 @@ function requireStreamIndex( if (streamIndex === undefined) throw new Error(`${operation} has no session stream index.`); return streamIndex; } + +function messageText(message: unknown): string { + if (typeof message === "string") return message; + if (!Array.isArray(message)) return ""; + return message + .flatMap((part) => + part !== null && + typeof part === "object" && + Reflect.get(part, "type") === "text" && + typeof Reflect.get(part, "text") === "string" + ? [Reflect.get(part, "text") as string] + : [], + ) + .join("\n"); +} diff --git a/e2e/fixtures/agent-background-tools/evals/scheduled-export.quiet-launch.eval.ts b/e2e/fixtures/agent-background-tools/evals/scheduled-export.quiet-launch.eval.ts index d07db4bef..9de4b9065 100644 --- a/e2e/fixtures/agent-background-tools/evals/scheduled-export.quiet-launch.eval.ts +++ b/e2e/fixtures/agent-background-tools/evals/scheduled-export.quiet-launch.eval.ts @@ -7,8 +7,7 @@ const FINAL = "SCHEDULED-EXPORT-DONE"; * A schedule-launched turn that dispatches a background task sends no launch * acknowledgement even when it runs with a user principal: durable schedule * provenance keeps conditional delivery, so the - * launching turn and the pending message wake both complete with a null - * message, and only the settled wake delivers a report. + * launching turn completes with a null message, and only the settled wake delivers a report. */ export default defineEval({ description: @@ -45,29 +44,8 @@ export default defineEval({ data: (data) => data.finishReason !== "tool-calls" && data.message !== null, }); - // The executor's explicit message wakes the parent while the cohort is - // still pending; that wake stays silent too. - const updateLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(session, "update wait"), - }); - const updateTurn = await updateLive.result(); - updateTurn.expectOk(); - updateTurn.event("message.completed", { - data: (data) => data.finishReason !== "tool-calls" && data.message === null, - count: 1, - }); - updateTurn.notEvent("message.completed", { - data: (data) => data.finishReason !== "tool-calls" && data.message !== null, - }); - await t.require( - updateTurn.message, - satisfies((message) => message === undefined, "the pending message wake is silent"), - ); - - // Agent receives background task result after turn ended. - // Only the settled wake produces the single final non-null delivery. const doneLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(updateLive.session, "completion wait"), + startIndex: requireStreamIndex(session, "completion wait"), }); const doneTurn = await doneLive.result(); doneTurn.expectOk(); @@ -88,21 +66,13 @@ export default defineEval({ ), ); - // Replay all three durable turns. Only the late report survives as assistant output. + // Replay both durable turns. Only the late report survives as assistant output. const replayedLaunch = await t.target.attachSession(sessionId); - const replayedUpdateLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(replayedLaunch, "replayed update"), - }); - await replayedUpdateLive.result(); const replayedDoneLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(replayedUpdateLive.session, "replayed completion"), + startIndex: requireStreamIndex(replayedLaunch, "replayed completion"), }); await replayedDoneLive.result(); - const replayedEvents = [ - ...replayedLaunch.events, - ...replayedUpdateLive.events, - ...replayedDoneLive.events, - ]; + const replayedEvents = [...replayedLaunch.events, ...replayedDoneLive.events]; await t.require( replayedEvents, satisfies( diff --git a/e2e/fixtures/agent-cancellation/evals/cancellation/background-steering.eval.ts b/e2e/fixtures/agent-cancellation/evals/cancellation/background-steering.eval.ts index 7a405a872..ef7ecf415 100644 --- a/e2e/fixtures/agent-cancellation/evals/cancellation/background-steering.eval.ts +++ b/e2e/fixtures/agent-cancellation/evals/cancellation/background-steering.eval.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import { defineEval, type EveEvalTurn } from "eve/evals"; import { equals, satisfies } from "eve/evals/expect"; @@ -165,11 +166,7 @@ export default cases.map(({ parentActive, steering, description }) => const firstChildTurn = await child.result(); const receipts = parentTurns.flatMap((turn) => - turn.events.flatMap((event) => - event.type === "subagent.completed" && event.data.backgroundTask !== undefined - ? [event.data.backgroundTask.taskId] - : [], - ), + taskReceipts(turn.events).map(({ taskId }) => taskId), ); await t.require(new Set(receipts).size, equals(1)); if (steering) { diff --git a/e2e/fixtures/agent-cancellation/evals/cancellation/generated-program-child-hitl.eval.ts b/e2e/fixtures/agent-cancellation/evals/cancellation/generated-program-child-hitl.eval.ts index a31fa4c57..26cf52825 100644 --- a/e2e/fixtures/agent-cancellation/evals/cancellation/generated-program-child-hitl.eval.ts +++ b/e2e/fixtures/agent-cancellation/evals/cancellation/generated-program-child-hitl.eval.ts @@ -23,7 +23,7 @@ export default defineEval({ count: 1, output: /CHILD_HITL_RESULT=.*GENERATED-HITL-MARKER/su, }); - t.calledSubagent("sleeper", { count: 1, status: "pending" }); + t.calledSubagent("sleeper", { count: 1, status: "completed" }); t.messageIncludes("CHILD_HITL_RESULT="); t.messageIncludes("GENERATED-HITL-MARKER"); t.noFailedActions(); diff --git a/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.ts b/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.ts index 5e2370875..0e5cb8753 100644 --- a/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.ts +++ b/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import assert from "node:assert/strict"; import { defineEval, type EveEvalContext, type EveEvalTurn } from "eve/evals"; import { satisfies } from "eve/evals/expect"; @@ -41,7 +42,7 @@ export default ["first", "later"].map((launchTurn) => function expectFiveReviewers(started: EveEvalTurn) { expectHealthyTurn(started); - started.calledSubagent("reviewer", { count: 5 }); + started.calledSubagent("reviewer", { status: "working", count: 5 }); const launchSteps = started.events .filter((event) => event.type === "actions.requested") .flatMap(({ data }) => @@ -52,9 +53,7 @@ function expectFiveReviewers(started: EveEvalTurn) { assert.equal(launchSteps.length, 5, "five reviewer requests"); assert.equal(new Set(launchSteps).size, 1, "all five reviewers launch in one model step"); - const taskIds = started.events - .filter((event) => event.type === "subagent.completed") - .flatMap(({ data }) => (data.backgroundTask ? [data.backgroundTask.taskId] : [])); + const taskIds = taskReceipts(started.events).map(({ taskId }) => taskId); assert.equal(taskIds.length, 5, "five background task receipts"); assert.equal(new Set(taskIds).size, 5, "five distinct background tasks"); return taskIds; diff --git a/e2e/fixtures/agent-router/evals/agent-router.eval.ts b/e2e/fixtures/agent-router/evals/agent-router.eval.ts index 6483427aa..efdaa53ed 100644 --- a/e2e/fixtures/agent-router/evals/agent-router.eval.ts +++ b/e2e/fixtures/agent-router/evals/agent-router.eval.ts @@ -8,7 +8,7 @@ export default defineEval({ turn.expectOk(); turn.messageIncludes("AGENT-ROUTER-ROOT-COPY-OK"); turn.calledTool("agent", { count: 1 }); - turn.calledSubagent("agent", { count: 1, status: "pending" }); + turn.calledSubagent("agent", { count: 1, status: "completed" }); t.succeeded(); t.noFailedActions(); }, diff --git a/e2e/fixtures/agent-subagents-hitl/agent/agent.ts b/e2e/fixtures/agent-subagents-hitl/agent/agent.ts index 637717abb..36bd0de92 100644 --- a/e2e/fixtures/agent-subagents-hitl/agent/agent.ts +++ b/e2e/fixtures/agent-subagents-hitl/agent/agent.ts @@ -26,6 +26,17 @@ function respond(request: MockModelRequest): MockModelResponse | string { if (prompt.includes("Background task reporting") && prompt.includes(STOCK_PRICE)) { return `The stock price is ${STOCK_PRICE}.`; } + if ( + request.messages.some( + (entry) => + entry.role === "user" && + entry.text.startsWith("Background task ") && + entry.text.includes("(collision-child) is completed.\n\nResult:\n") && + entry.text.includes(COLLISION_MARKER), + ) + ) { + return COLLISION_MARKER; + } if (request.lastUserMessage?.includes(COLLISION_MARKER) !== true) { return `Mock reply: ${message}`; } @@ -51,7 +62,8 @@ function respond(request: MockModelRequest): MockModelResponse | string { } if (gateResults.length === 1 && subagentResults.length === 1) { - return COLLISION_MARKER; + // The subagent tool result is a working receipt; its result arrives in a later turn. + return "The gate was approved; the child is still working."; } throw new Error("Mixed runtime-action step resumed before both tool results were available."); diff --git a/e2e/fixtures/agent-subagents-hitl/evals/hitl.eval.ts b/e2e/fixtures/agent-subagents-hitl/evals/hitl.eval.ts index 08f301a07..2f880455c 100644 --- a/e2e/fixtures/agent-subagents-hitl/evals/hitl.eval.ts +++ b/e2e/fixtures/agent-subagents-hitl/evals/hitl.eval.ts @@ -24,11 +24,10 @@ export default defineEval({ const started = await t.send( `Call the stock-price subagent exactly once with message 'Call the get_stock_price tool exactly once with ticker "GOOG". After it returns, do not call any tool again; return the result.'. After that single subagent call finishes, do not call any subagent or tool again; include the exact stock price in your final reply.`, ); - started.event("subagent.completed", { + started.event("action.result", { count: 1, data: { - backgroundTask: { status: "working" }, - subagentName: "stock-price", + result: { kind: "tool-result", output: { status: "working" }, toolName: "stock-price" }, }, }); @@ -44,9 +43,7 @@ export default defineEval({ completed.messageIncludes(GOOG_PRICE); t.succeeded(); - t.calledSubagent("stock-price", { - count: 1, - }); + t.calledSubagent("stock-price", { status: "completed", count: 1 }); t.noFailedActions(); }, }); diff --git a/e2e/fixtures/agent-subagents-hitl/evals/mixed-runtime-action-approval.eval.ts b/e2e/fixtures/agent-subagents-hitl/evals/mixed-runtime-action-approval.eval.ts index eaa633a49..4cb0f5992 100644 --- a/e2e/fixtures/agent-subagents-hitl/evals/mixed-runtime-action-approval.eval.ts +++ b/e2e/fixtures/agent-subagents-hitl/evals/mixed-runtime-action-approval.eval.ts @@ -23,10 +23,19 @@ export default defineEval({ const session = parked.session; parked.calledTool("collision-gate", { count: 1, status: "pending" }); - parked.calledSubagent("collision-child", { count: 1, status: "completed" }); + parked.calledSubagent("collision-child", { count: 1, status: "working" }); parked.eventOrder([ { type: "actions.requested" }, - { type: "subagent.completed" }, + { + type: "action.result", + data: { + result: { + kind: "tool-result", + toolName: "collision-child", + output: { status: "working" }, + }, + }, + }, { type: "input.requested" }, { type: "session.waiting" }, ]); @@ -36,13 +45,17 @@ export default defineEval({ resumed.expectOk(); const completed = resumed.message?.includes(COLLISION_MARKER) ? resumed - : await waitForMessage(t, parked.session, COLLISION_MARKER); + : await waitForMessage(t, resumed.session, COLLISION_MARKER); completed.messageIncludes(COLLISION_MARKER); t.succeeded(); t.noFailedActions(); t.calledTool("collision-gate", { count: 1, status: "completed" }); t.calledSubagent("collision-child", { count: 1, status: "completed" }); + t.event("subagent.completed", { + count: 1, + data: { callId: "collision-child-call", subagentName: "collision-child" }, + }); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/descendant-session-limit.eval.ts b/e2e/fixtures/agent-subagents/evals/descendant-session-limit.eval.ts index 69e87bf6e..d1951c0ce 100644 --- a/e2e/fixtures/agent-subagents/evals/descendant-session-limit.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/descendant-session-limit.eval.ts @@ -80,7 +80,7 @@ export default defineEval({ `Do not call any tool or subagent. Reply with exactly ${ROOT_RECOVERY_TOKEN} and nothing else.`, ); recovered.expectOk(); - stopSession.calledSubagent("limited-worker", { count: 1 }); + stopSession.calledSubagent("limited-worker", { status: "working", count: 1 }); recovered.messageIncludes(ROOT_RECOVERY_TOKEN); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/disabled-subagent-tool.eval.ts b/e2e/fixtures/agent-subagents/evals/disabled-subagent-tool.eval.ts index a2ff63d8d..9defa271a 100644 --- a/e2e/fixtures/agent-subagents/evals/disabled-subagent-tool.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/disabled-subagent-tool.eval.ts @@ -9,8 +9,8 @@ export default defineEval({ turn.expectOk(); turn.messageIncludes("DISABLED-SUBAGENT-OK"); turn.calledTool("invoke-hidden", { count: 1 }); - turn.calledSubagent("disabled-hidden", { count: 1, status: "pending" }); - turn.calledSubagent("tool-hidden", { count: 0, status: "pending" }); + turn.calledSubagent("disabled-hidden", { count: 1, status: "completed" }); + turn.calledSubagent("tool-hidden", { count: 0 }); t.succeeded(); t.noFailedActions(); }, diff --git a/e2e/fixtures/agent-subagents/evals/dynamic-subagent.eval.ts b/e2e/fixtures/agent-subagents/evals/dynamic-subagent.eval.ts index 9bb516702..47d9036ee 100644 --- a/e2e/fixtures/agent-subagents/evals/dynamic-subagent.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/dynamic-subagent.eval.ts @@ -6,9 +6,15 @@ export default defineEval({ async test(t) { const selected = await t.send("Call conditional-marker exactly once."); selected.expectOk(); - selected.event("subagent.completed", { + selected.event("action.result", { count: 1, - data: { backgroundTask: { status: "working" }, subagentName: "conditional-marker" }, + data: { + result: { + kind: "tool-result", + output: { status: "working" }, + toolName: "conditional-marker", + }, + }, }); const completed = await waitForMessage(t, selected.session, "DYNAMIC_SUBAGENT_ENABLED"); diff --git a/e2e/fixtures/agent-subagents/evals/dynamic-workflow.eval.ts b/e2e/fixtures/agent-subagents/evals/dynamic-workflow.eval.ts index 769f6b5f8..180e12b5f 100644 --- a/e2e/fixtures/agent-subagents/evals/dynamic-workflow.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/dynamic-workflow.eval.ts @@ -48,7 +48,7 @@ export default defineEval({ t.succeeded(); t.calledTool("workflow", { input: isFanOutProgram, count: 1 }); - turn.calledSubagent("echo-marker", { count: 2, status: "pending" }); + turn.calledSubagent("echo-marker", { count: 2, status: "completed" }); firstChildTurn.eventsSatisfy( "first child does not complete before both children start", (events) => diff --git a/e2e/fixtures/agent-subagents/evals/hidden-subagents.eval.ts b/e2e/fixtures/agent-subagents/evals/hidden-subagents.eval.ts index ccaf70605..7bf21b00c 100644 --- a/e2e/fixtures/agent-subagents/evals/hidden-subagents.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/hidden-subagents.eval.ts @@ -10,8 +10,8 @@ export default defineEval({ turn.messageIncludes("TOOL-FALSE-SUBAGENT-OK"); turn.messageIncludes("Internal specialist hidden by its agent definition."); turn.calledTool("invoke-hidden", { count: 1 }); - turn.calledSubagent("tool-hidden", { count: 1, status: "pending" }); - turn.calledSubagent("disabled-hidden", { count: 0, status: "pending" }); + turn.calledSubagent("tool-hidden", { count: 1, status: "completed" }); + turn.calledSubagent("disabled-hidden", { count: 0 }); t.succeeded(); t.noFailedActions(); }, diff --git a/e2e/fixtures/agent-subagents/evals/local-delegation.eval.ts b/e2e/fixtures/agent-subagents/evals/local-delegation.eval.ts index b3fa2aa2f..ba1fe5a4f 100644 --- a/e2e/fixtures/agent-subagents/evals/local-delegation.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/local-delegation.eval.ts @@ -22,7 +22,7 @@ export default defineEval({ completed.messageIncludes(SUBAGENT_TOKEN); t.succeeded(); - t.calledSubagent("echo-marker"); + t.calledSubagent("echo-marker", { status: "completed" }); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/recursive-root-only.eval.ts b/e2e/fixtures/agent-subagents/evals/recursive-root-only.eval.ts index a97e07f4d..ae156bdfb 100644 --- a/e2e/fixtures/agent-subagents/evals/recursive-root-only.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/recursive-root-only.eval.ts @@ -25,7 +25,7 @@ export default defineEval({ completed.messageIncludes(CHILD_TOKEN); t.succeeded(); - t.calledSubagent("agent", { count: 1 }); + t.calledSubagent("agent", { status: "completed", count: 1 }); t.noFailedActions(); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/remote-principal-forwarding.eval.ts b/e2e/fixtures/agent-subagents/evals/remote-principal-forwarding.eval.ts index 27958c1f8..8f800c477 100644 --- a/e2e/fixtures/agent-subagents/evals/remote-principal-forwarding.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/remote-principal-forwarding.eval.ts @@ -87,7 +87,9 @@ export default defineEval({ }, }); - t.calledSubagent("remote-loopback", { count: 3 }).soft().label("no repeated delegation"); + t.event("subagent.called", { data: { name: "remote-loopback" }, count: 3 }) + .soft() + .label("no repeated delegation"); t.succeeded(); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/scheduled-remote-completion.eval.ts b/e2e/fixtures/agent-subagents/evals/scheduled-remote-completion.eval.ts index 3ebe182c1..66de3d915 100644 --- a/e2e/fixtures/agent-subagents/evals/scheduled-remote-completion.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/scheduled-remote-completion.eval.ts @@ -30,11 +30,8 @@ export default defineEval({ // both the event stream and the channel. const launch = await t.target.attachSession(sessionId); launch.succeeded(); - launch.calledTool("remote-loopback"); - launch.event("subagent.completed", { - data: (data) => data.subagentName === "remote-loopback" && data.backgroundTask !== undefined, - count: 1, - }); + launch.calledTool("remote-loopback", { output: { status: "working" }, count: 1 }); + launch.notEvent("subagent.completed"); launch.event("message.completed", { data: (data) => data.finishReason !== "tool-calls" && data.message === null, count: 1, @@ -58,6 +55,10 @@ export default defineEval({ const completed = await completedLive.result(); completed.expectOk(); completed.messageIncludes(FINAL); + completed.event("subagent.completed", { + data: { subagentName: "remote-loopback" }, + count: 1, + }); await t.require( completed.events, satisfies( diff --git a/e2e/fixtures/agent-subagents/evals/self-modification/add-capability.eval.ts b/e2e/fixtures/agent-subagents/evals/self-modification/add-capability.eval.ts index f6d333f6f..fa1e5434d 100644 --- a/e2e/fixtures/agent-subagents/evals/self-modification/add-capability.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/self-modification/add-capability.eval.ts @@ -8,6 +8,6 @@ export default defineEval({ started.expectOk(); t.succeeded(); - t.calledSubagent("self-modification"); + t.calledSubagent("self-modification", { status: "working" }); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/self-modification/add-reusable-action.eval.ts b/e2e/fixtures/agent-subagents/evals/self-modification/add-reusable-action.eval.ts index 67b81296b..bd18c0c87 100644 --- a/e2e/fixtures/agent-subagents/evals/self-modification/add-reusable-action.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/self-modification/add-reusable-action.eval.ts @@ -10,6 +10,6 @@ export default defineEval({ started.expectOk(); t.succeeded(); - t.calledSubagent("self-modification"); + t.calledSubagent("self-modification", { status: "working" }); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/self-modification/connect-capability.eval.ts b/e2e/fixtures/agent-subagents/evals/self-modification/connect-capability.eval.ts index 3062825b0..68fbe478c 100644 --- a/e2e/fixtures/agent-subagents/evals/self-modification/connect-capability.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/self-modification/connect-capability.eval.ts @@ -8,6 +8,6 @@ export default defineEval({ started.expectOk(); t.succeeded(); - t.calledSubagent("self-modification"); + t.calledSubagent("self-modification", { status: "working" }); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/self-modification/enable-capability.eval.ts b/e2e/fixtures/agent-subagents/evals/self-modification/enable-capability.eval.ts index 5cc9ccfd1..65e24808e 100644 --- a/e2e/fixtures/agent-subagents/evals/self-modification/enable-capability.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/self-modification/enable-capability.eval.ts @@ -8,6 +8,6 @@ export default defineEval({ started.expectOk(); t.succeeded(); - t.calledSubagent("self-modification"); + t.calledSubagent("self-modification", { status: "working" }); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/self-modification/install-capability.eval.ts b/e2e/fixtures/agent-subagents/evals/self-modification/install-capability.eval.ts index 09dcf6f8a..693f11309 100644 --- a/e2e/fixtures/agent-subagents/evals/self-modification/install-capability.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/self-modification/install-capability.eval.ts @@ -9,6 +9,6 @@ export default defineEval({ started.expectOk(); t.succeeded(); - t.calledSubagent("self-modification"); + t.calledSubagent("self-modification", { status: "working" }); }, }); diff --git a/e2e/fixtures/agent-subagents/evals/software-factory-workflow.eval.ts b/e2e/fixtures/agent-subagents/evals/software-factory-workflow.eval.ts index 12a1a6103..1ab173f7f 100644 --- a/e2e/fixtures/agent-subagents/evals/software-factory-workflow.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/software-factory-workflow.eval.ts @@ -33,9 +33,9 @@ export default defineEval({ count: 1, output: (observed) => isDeepStrictEqual(observed, expected), }); - t.calledSubagent(TRIAGE, { count: 1, status: "pending" }); - t.calledSubagent(REVIEW, { count: 1, status: "pending" }); - t.calledSubagent(REPRODUCE, { count: 1, status: "pending" }); + t.calledSubagent(TRIAGE, { count: 1, status: "completed" }); + t.calledSubagent(REVIEW, { count: 1, status: "completed" }); + t.calledSubagent(REPRODUCE, { count: 1, status: "completed" }); turn.eventsSatisfy("analysis fans out before reproduction consumes both results", (events) => { const called = new Map(); for (const [index, event] of events.entries()) { diff --git a/e2e/fixtures/agent-subagents/evals/workflow-blocking-continuation.eval.ts b/e2e/fixtures/agent-subagents/evals/workflow-blocking-continuation.eval.ts index 76f3dbb76..637f8ed97 100644 --- a/e2e/fixtures/agent-subagents/evals/workflow-blocking-continuation.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/workflow-blocking-continuation.eval.ts @@ -34,7 +34,7 @@ export default defineEval({ t.succeeded(); t.calledTool("workflow", { count: 2 }); - t.calledSubagent("echo-marker", { count: 4, status: "pending" }); + t.calledSubagent("echo-marker", { count: 4, status: "completed" }); t.eventsSatisfy("all workflow-program calls continue one child session", (events) => { const childSessionIds = events.flatMap((event) => event.type === "subagent.called" && event.data.name === "echo-marker" diff --git a/e2e/fixtures/agent-subagents/evals/workflow-subagent-limit.eval.ts b/e2e/fixtures/agent-subagents/evals/workflow-subagent-limit.eval.ts index 5054519a3..b4b3d63c7 100644 --- a/e2e/fixtures/agent-subagents/evals/workflow-subagent-limit.eval.ts +++ b/e2e/fixtures/agent-subagents/evals/workflow-subagent-limit.eval.ts @@ -51,7 +51,7 @@ export default defineEval({ input: isFourCallProgram, output: isFourElementLimitResult, }); - t.calledSubagent("echo-marker", { count: 3, status: "pending" }); + t.calledSubagent("echo-marker", { count: 3, status: "completed" }); t.messageIncludes("WORKFLOW_PROGRAM_SUBAGENT_LIMIT_REACHED"); }, }); diff --git a/e2e/fixtures/agent-task-reporting/evals/reporting.ts b/e2e/fixtures/agent-task-reporting/evals/reporting.ts index 407853c9c..8f29b2976 100644 --- a/e2e/fixtures/agent-task-reporting/evals/reporting.ts +++ b/e2e/fixtures/agent-task-reporting/evals/reporting.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import { e2eModel } from "@eve-e2e/config"; import type { EveEvalContext, EveEvalSession, EveEvalTurn, InputRequest } from "eve/evals"; import { equals, satisfies } from "eve/evals/expect"; @@ -51,7 +52,7 @@ export async function startWarehouseLookups(t: EveEvalContext): Promise - event.type === "subagent.completed" && event.data.backgroundTask !== undefined - ? [{ callId: event.data.callId, taskId: event.data.backgroundTask.taskId }] - : [], - ); + const receipts = taskReceipts(started.events); const actions = started.events.flatMap((event) => event.type === "actions.requested" ? event.data.actions : [], ); @@ -359,7 +356,7 @@ async function releaseCheck(t: EveEvalContext, run: ReportingRun, check: Check): ), equals([]), ); - // ctx.agent completes through its owning workflow tool, not a subagent.completed event. + // Check the owning workflow tool result for the completed lookup. await t.require( lookup, equals([ @@ -428,12 +425,13 @@ async function post(t: EveEvalContext, run: ReportingRun, suffix: "" | "/compact } function hasPostReceiptAcknowledgement(turn: EveEvalTurn): boolean { + const receiptCallIds = new Set( + taskReceipts(turn.events) + .filter(({ toolName }) => toolName === "agent") + .map(({ callId }) => callId), + ); const receiptIndexes = turn.events.flatMap((event, index) => - event.type === "subagent.completed" && - event.data.subagentName === "agent" && - event.data.backgroundTask !== undefined - ? [index] - : [], + event.type === "action.result" && receiptCallIds.has(event.data.result.callId) ? [index] : [], ); return ( receiptIndexes.length === TASK_COUNT && diff --git a/e2e/fixtures/agent-tools-sandbox/evals/sandbox/subagent-sharing.eval.ts b/e2e/fixtures/agent-tools-sandbox/evals/sandbox/subagent-sharing.eval.ts index 7eb47fb65..2aa3966e5 100644 --- a/e2e/fixtures/agent-tools-sandbox/evals/sandbox/subagent-sharing.eval.ts +++ b/e2e/fixtures/agent-tools-sandbox/evals/sandbox/subagent-sharing.eval.ts @@ -36,7 +36,7 @@ export default defineEval({ ); t.succeeded(); - t.calledSubagent("shared-sandbox", { count: 1 }); + t.calledSubagent("shared-sandbox", { status: "completed", count: 1 }); t.check(parentRead.message, includes(CHILD_TOKEN)); }, }); diff --git a/e2e/fixtures/agent-workflow-tools/agent/agent.ts b/e2e/fixtures/agent-workflow-tools/agent/agent.ts index 225b9f2d1..5b4c6c537 100644 --- a/e2e/fixtures/agent-workflow-tools/agent/agent.ts +++ b/e2e/fixtures/agent-workflow-tools/agent/agent.ts @@ -83,9 +83,6 @@ function respond(request: MockModelRequest): MockModelResponse | string { }`; } - if (message.includes("WORKFLOW-REPORT-PROGRESS")) { - return "WORKFLOW-REPORT-UPDATE-RECEIVED"; - } if (message.includes("is completed") && message.includes("WORKFLOW-REPORT-COMPLETE")) { return "WORKFLOW-REPORT-DONE"; } diff --git a/e2e/fixtures/agent-workflow-tools/agent/tools/report_deploy.ts b/e2e/fixtures/agent-workflow-tools/agent/tools/report_deploy.ts index 465d57fa0..1cdd387f3 100644 --- a/e2e/fixtures/agent-workflow-tools/agent/tools/report_deploy.ts +++ b/e2e/fixtures/agent-workflow-tools/agent/tools/report_deploy.ts @@ -4,19 +4,19 @@ import { z } from "zod"; import { describePlan, hashPlan } from "../lib/plan.ts"; /** - * Background workflow tool: the model gets a receipt, `postMessage` delivers a - * progress note, and the agent is woken with the return value. + * Background workflow tool: the model gets a receipt, intermediate yields are consumed, + * and the return value reaches the parent in the terminal cohort report. */ export default defineWorkflowTool({ description: "Plan a deploy in the background and report when it is ready.", execution: "background", inputSchema: z.strictObject({ service: z.string() }), - async *execute({ service }, _ctx, task) { + async *execute({ service }) { "use workflow"; const plan = describePlan(service); yield { plan }; - yield task.postMessage(`Deploy ${task.taskId}: WORKFLOW-REPORT-PROGRESS ${plan}`); + yield { message: "WORKFLOW-REPORT-PROGRESS", plan }; const digest = await hashPlan(plan); return { digest, plan, result: "WORKFLOW-REPORT-COMPLETE" }; }, diff --git a/e2e/fixtures/agent-workflow-tools/evals/report-deploy.background.eval.ts b/e2e/fixtures/agent-workflow-tools/evals/report-deploy.background.eval.ts index e036b3aae..3b5ec057a 100644 --- a/e2e/fixtures/agent-workflow-tools/evals/report-deploy.background.eval.ts +++ b/e2e/fixtures/agent-workflow-tools/evals/report-deploy.background.eval.ts @@ -3,12 +3,13 @@ import { satisfies } from "eve/evals/expect"; export default defineEval({ description: - "A background workflow tool returns a receipt, reports progress, and wakes the agent with its result.", + "A background workflow tool returns a receipt, consumes intermediate yields without progress notifications, and reports its return value.", async test(t) { const started = await t.send("WORKFLOW-REPORT-START"); const conversation = started.session; started.expectOk(); started.calledTool("report_deploy"); + started.notEvent("action.partial"); const receipt = started.requireToolCall("report_deploy"); const taskId = readTaskId(receipt.output); @@ -17,29 +18,8 @@ export default defineEval({ const sessionId = conversation.sessionId; if (sessionId === undefined) throw new Error("Eval has no parent session id."); - const updateLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(started.session, "update wait"), - }); - const updateTurn = await updateLive.result(); - updateTurn.expectOk(); - updateTurn.messageIncludes("WORKFLOW-REPORT-UPDATE-RECEIVED"); - await t.require( - updateTurn.events, - satisfies( - (events: typeof updateTurn.events) => - events.some( - (event) => - event.type === "message.received" && - messageText(event.data.message).includes( - `Deploy ${taskId}: WORKFLOW-REPORT-PROGRESS deploy api`, - ), - ), - "parent receives the run's progress note with task identity", - ), - ); - const doneLive = t.target.watchTurn(sessionId, { - startIndex: requireStreamIndex(updateLive.session, "completion wait"), + startIndex: requireStreamIndex(started.session, "completion wait"), }); const doneTurn = await doneLive.result(); doneTurn.expectOk(); @@ -59,6 +39,11 @@ export default defineEval({ "parent receives the run's return value with task identity", ), ); + doneTurn.event("turn.started", { count: 1 }); + doneTurn.notEvent("action.partial"); + doneTurn.notEvent("message.received", { + data: (data) => messageText(data.message).includes("PROGRESS"), + }); t.noFailedActions(); }, }); diff --git a/e2e/fixtures/e2e-config/package.json b/e2e/fixtures/e2e-config/package.json index d3f33222d..ff6c4367e 100644 --- a/e2e/fixtures/e2e-config/package.json +++ b/e2e/fixtures/e2e-config/package.json @@ -6,7 +6,8 @@ "exports": { ".": "./src/index.ts", "./instrumentation": "./src/instrumentation.ts", - "./instrumentation-otel": "./src/instrumentation-otel.ts" + "./instrumentation-otel": "./src/instrumentation-otel.ts", + "./task-receipts": "./src/task-receipts.ts" }, "scripts": { "test:integration": "node --test test/*.integration.test.mjs", diff --git a/e2e/fixtures/e2e-config/src/task-receipts.ts b/e2e/fixtures/e2e-config/src/task-receipts.ts new file mode 100644 index 000000000..a43db7b02 --- /dev/null +++ b/e2e/fixtures/e2e-config/src/task-receipts.ts @@ -0,0 +1,21 @@ +import type { EveEvalTurn } from "eve/evals"; + +/** Admission receipts are tool results; subagent completion carries the child's final output. */ +export function taskReceipts(events: EveEvalTurn["events"]) { + return events.flatMap((event) => { + if (event.type !== "action.result" || event.data.status !== "completed") return []; + const result = event.data.result; + if (result.kind !== "tool-result") return []; + const output = result.output; + if ( + typeof output !== "object" || + output === null || + Array.isArray(output) || + Reflect.get(output, "status") !== "working" + ) + return []; + const taskId: unknown = Reflect.get(output, "taskId"); + if (typeof taskId !== "string") return []; + return [{ callId: result.callId, taskId, toolName: result.toolName }]; + }); +} diff --git a/e2e/fixtures/fixture-tasks/agent/channels/lifecycle.ts b/e2e/fixtures/fixture-tasks/agent/channels/lifecycle.ts index 70d283508..91bc0c48c 100644 --- a/e2e/fixtures/fixture-tasks/agent/channels/lifecycle.ts +++ b/e2e/fixtures/fixture-tasks/agent/channels/lifecycle.ts @@ -85,7 +85,7 @@ export default defineChannel({ return Response.json({ marker: event.marker, status: await run.status, - deliveries: await ownerDeliveries(event.runId), + notificationCount: await ownerNotificationCount(event.runId), }); } } @@ -94,7 +94,7 @@ export default defineChannel({ ], }); -async function ownerDeliveries(runId: string) { +async function ownerNotificationCount(runId: string) { const world = await getWorld(); const [steps, events] = await Promise.all([ world.steps.list({ runId, pagination: { limit: 100 }, resolveData: "none" }), @@ -105,16 +105,14 @@ async function ownerDeliveries(runId: string) { }), ]); if (steps.hasMore || events.hasMore) throw new Error("Lifecycle owner audit exceeded its bound."); - const names = new Map(steps.data.map((step) => [step.stepId, step.stepName.split("//").at(-1)])); - return events.data.flatMap((event) => { - if (event.eventType !== "step_completed") return []; - const name = names.get(event.correlationId); - return name === "wakeTaskAgentRequestParentStep" - ? ["agent-request"] - : name === "wakeTaskParentStep" - ? ["completed"] - : []; - }); + const notificationSteps = new Set( + steps.data + .filter((step) => step.stepName.endsWith("//notifyTaskParent")) + .map((step) => step.stepId), + ); + return events.data.filter( + (event) => event.eventType === "step_completed" && notificationSteps.has(event.correlationId), + ).length; } async function boundedBody(request: Request) { diff --git a/e2e/fixtures/fixture-tasks/evals/batching.ts b/e2e/fixtures/fixture-tasks/evals/batching.ts index 8abab6466..a8049c6fc 100644 --- a/e2e/fixtures/fixture-tasks/evals/batching.ts +++ b/e2e/fixtures/fixture-tasks/evals/batching.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import { type EveEvalContext, type EveEvalTurn, type InputRequest } from "eve/evals"; import { equals } from "eve/evals/expect"; @@ -14,12 +15,8 @@ export async function startBlockedFanout(t: EveEvalContext, count: number) { started.expectOk(); started.noFailedActions(); started.messageIncludes("TASK-FANOUT-STARTED"); - started.calledSubagent("fanout-worker", { count }); - const receipts = started.events.flatMap((event) => - event.type === "subagent.completed" && event.data.backgroundTask !== undefined - ? [{ callId: event.data.callId, taskId: event.data.backgroundTask.taskId }] - : [], - ); + started.calledSubagent("fanout-worker", { status: "working", count }); + const receipts = taskReceipts(started.events); const taskIds = receipts.map(({ taskId }) => taskId); await t.require( { receipts: taskIds.length, distinct: new Set(taskIds).size }, diff --git a/e2e/fixtures/fixture-tasks/evals/lifecycle.ts b/e2e/fixtures/fixture-tasks/evals/lifecycle.ts index a50efc54c..25a4446ab 100644 --- a/e2e/fixtures/fixture-tasks/evals/lifecycle.ts +++ b/e2e/fixtures/fixture-tasks/evals/lifecycle.ts @@ -80,7 +80,7 @@ export function lifecycleDriver(t: EveEvalContext, key: string) { equals({ marker, status: "completed", - deliveries: agent ? ["agent-request", "agent-request", "completed"] : ["completed"], + notificationCount: agent ? 3 : 1, }), ); }, diff --git a/e2e/fixtures/fixture-tasks/evals/shared.ts b/e2e/fixtures/fixture-tasks/evals/shared.ts index 580025496..d6572d916 100644 --- a/e2e/fixtures/fixture-tasks/evals/shared.ts +++ b/e2e/fixtures/fixture-tasks/evals/shared.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import type { EveEvalContext, EveEvalSession, EveEvalTurn, InputRequest } from "eve/evals"; import { equals, satisfies } from "eve/evals/expect"; @@ -96,14 +97,11 @@ export async function waitForTaskInput( throw new Error(`Task did not surface input for tool "${toolName}" after five turns.`); } -/** Reads the task receipt attached to a background `subagent.completed` event. */ +/** Reads the working task receipt returned by a background tool call. */ export function requireBackgroundTaskId(turn: EveEvalTurn): string { - for (const event of turn.events) { - if (event.type === "subagent.completed" && event.data.backgroundTask !== undefined) { - return event.data.backgroundTask.taskId; - } - } - throw new Error("Turn completed without a background task receipt."); + const receipt = taskReceipts(turn.events)[0]; + if (receipt === undefined) throw new Error("Turn completed without a background task receipt."); + return receipt.taskId; } export function parseToolErrorOutput(output: unknown): unknown { diff --git a/e2e/fixtures/fixture-tasks/evals/task.agent.steer.accepted-busy.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.agent.steer.accepted-busy.eval.ts index 1905f4052..e42709e7b 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.agent.steer.accepted-busy.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.agent.steer.accepted-busy.eval.ts @@ -54,7 +54,7 @@ export default defineTaskEval({ }); raced.calledSubagent("busy-worker", { count: 1, - status: "completed", + status: "working", }); raced.calledTool("busy-worker", { count: 1, @@ -69,7 +69,7 @@ export default defineTaskEval({ held.session, ); later.turn.expectOk(); - later.turn.calledSubagent("busy-worker", { count: 1, status: "completed" }); + later.turn.calledSubagent("busy-worker", { count: 1, status: "working" }); const steeredTaskId = requireBackgroundTaskId(later.turn); await t.require(steeredTaskId, equals(admittedTaskId)); // Steering retains the existing request; it does not publish a replacement approval. diff --git a/e2e/fixtures/fixture-tasks/evals/task.authorization.callback.accepted-current-attempt.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.authorization.callback.accepted-current-attempt.eval.ts index 7e72b54d1..4a99f4469 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.authorization.callback.accepted-current-attempt.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.authorization.callback.accepted-current-attempt.eval.ts @@ -30,9 +30,11 @@ export default defineTaskEval({ const started = await t.send("TASK-C7-AUTHORIZATION"); started.expectOk(); started.messageIncludes("TASK-C7-STARTED"); - started.event("subagent.completed", { + started.event("action.result", { count: 1, - data: { backgroundTask: { status: "working" }, subagentName: "approval-worker" }, + data: { + result: { kind: "tool-result", output: { status: "working" }, toolName: "approval-worker" }, + }, }); const taskId = requireBackgroundTaskId(started); diff --git a/e2e/fixtures/fixture-tasks/evals/task.dispatch-batch.start.accepted-partial-failure.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.dispatch-batch.start.accepted-partial-failure.eval.ts index 7939d152f..e580a82d9 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.dispatch-batch.start.accepted-partial-failure.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.dispatch-batch.start.accepted-partial-failure.eval.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import { type EveEvalTurn } from "eve/evals"; import { satisfies } from "eve/evals/expect"; @@ -52,11 +53,10 @@ export default defineTaskEval({ const firstTaskId = requireReceiptTaskId(receipts, FIRST_CALL_ID); const failedTaskId = requireReceiptTaskId(receipts, FAILED_CALL_ID); const thirdTaskId = requireReceiptTaskId(receipts, THIRD_CALL_ID); - started.event("subagent.completed", { + started.event("action.result", { count: 2, data: { - backgroundTask: { status: "working" }, - subagentName: "busy-worker", + result: { kind: "tool-result", output: { status: "working" }, toolName: "busy-worker" }, }, }); @@ -91,11 +91,7 @@ interface BackgroundReceipt { } function backgroundReceipts(turn: EveEvalTurn): readonly BackgroundReceipt[] { - return turn.events.flatMap((event) => - event.type === "subagent.completed" && event.data.backgroundTask !== undefined - ? [{ callId: event.data.callId, taskId: event.data.backgroundTask.taskId }] - : [], - ); + return taskReceipts(turn.events); } function requireReceiptTaskId(receipts: readonly BackgroundReceipt[], callId: string): string { diff --git a/e2e/fixtures/fixture-tasks/evals/task.dispatch.start.rejected-unreachable.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.dispatch.start.rejected-unreachable.eval.ts index e84581b98..23d192ed9 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.dispatch.start.rejected-unreachable.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.dispatch.start.rejected-unreachable.eval.ts @@ -15,12 +15,15 @@ export default defineTaskEval({ const started = await t.send("TASK-A3-DISPATCH-START-FAILURE"); started.expectOk(); started.messageIncludes("TASK-A3-PARENT-SURVIVED"); - started.event("subagent.completed", { + started.event("action.result", { count: 1, data: { - backgroundTask: { status: "working" }, - callId: CALL_ID, - subagentName: "unstartable-worker", + result: { + kind: "tool-result", + output: { status: "working" }, + callId: CALL_ID, + toolName: "unstartable-worker", + }, }, }); const taskId = requireBackgroundTaskId(started); diff --git a/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.local.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.local.eval.ts index 902b1dd78..657ea1030 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.local.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.local.eval.ts @@ -15,9 +15,11 @@ export default defineTaskEval({ async test(t) { const started = await t.send("TASK-HITL-ROUTING"); started.expectOk(); - started.event("subagent.completed", { + started.event("action.result", { count: 1, - data: { backgroundTask: { status: "working" }, subagentName: "approval-worker" }, + data: { + result: { kind: "tool-result", output: { status: "working" }, toolName: "approval-worker" }, + }, }); const taskId = requireBackgroundTaskId(started); diff --git a/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.remote.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.remote.eval.ts index 997f7d39d..1326745de 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.remote.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.input.answer.accepted-complete.remote.eval.ts @@ -29,12 +29,15 @@ export default defineTaskEval({ const started = await t.send("TASK-C8-REMOTE-HITL"); started.expectOk(); started.messageIncludes("TASK-C8-STARTED"); - started.event("subagent.completed", { + started.event("action.result", { count: 1, data: { - backgroundTask: { status: "working" }, - callId: "task-c8-remote-worker", - subagentName: "remote-loopback", + result: { + kind: "tool-result", + output: { status: "working" }, + callId: "task-c8-remote-worker", + toolName: "remote-loopback", + }, }, }); const taskId = requireBackgroundTaskId(started); diff --git a/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-all-terminal.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-all-terminal.eval.ts index 1258e8aca..e88c19f05 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-all-terminal.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-all-terminal.eval.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import { type EveEvalContext, type EveEvalTurn, type InputRequest } from "eve/evals"; import { satisfies } from "eve/evals/expect"; @@ -31,7 +32,7 @@ export default defineTaskEval({ const started = await t.send("TASK-FAN-IN"); started.expectOk(); started.messageIncludes("TASK-FAN-IN-STARTED"); - started.calledSubagent("fanout-worker", { count: FAN_IN_SIZE }); + started.calledSubagent("fanout-worker", { status: "working", count: FAN_IN_SIZE }); const tasksByMarker = backgroundTasksByMarker(started); const taskIds = [...tasksByMarker.values()]; @@ -146,11 +147,9 @@ async function waitForTurnMessage( function backgroundTasksByMarker(turn: EveEvalTurn): ReadonlyMap { const tasksByMarker = new Map(); - for (const event of turn.events) { - if (event.type !== "subagent.completed" || event.data.backgroundTask === undefined) continue; - const fanInCall = FAN_IN_CALLS.find(({ callId }) => callId === event.data.callId); - if (fanInCall !== undefined) - tasksByMarker.set(fanInCall.marker, event.data.backgroundTask.taskId); + for (const receipt of taskReceipts(turn.events)) { + const fanInCall = FAN_IN_CALLS.find(({ callId }) => callId === receipt.callId); + if (fanInCall !== undefined) tasksByMarker.set(fanInCall.marker, receipt.taskId); } return tasksByMarker; } diff --git a/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-partial.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-partial.eval.ts index f322ce728..12e03d6f0 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-partial.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.join.evaluate.observed-partial.eval.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import type { EveEvalTurn, InputRequest } from "eve/evals"; import { equals } from "eve/evals/expect"; @@ -29,7 +30,7 @@ export default defineTaskEval({ async test(t) { const started = (await t.send("TASK-FAN-IN")).expectOk(); started.messageIncludes("TASK-FAN-IN-STARTED"); - started.calledSubagent("fanout-worker", { count: MARKERS.length }); + started.calledSubagent("fanout-worker", { status: "working", count: MARKERS.length }); let session: TaskEvalSessionDriver = started.session; const requests = new Map(); const setupEvents: EveEvalTurn["events"][number][] = []; @@ -47,20 +48,14 @@ export default defineTaskEval({ const called = setupEvents.find( (event) => event.type === "subagent.called" && event.data.callId === callId, ); - const receipt = started.events.find( - (event) => event.type === "subagent.completed" && event.data.callId === callId, - ); - if ( - called?.type !== "subagent.called" || - receipt?.type !== "subagent.completed" || - receipt.data.backgroundTask === undefined - ) { + const receipt = taskReceipts(started.events).find((receipt) => receipt.callId === callId); + if (called?.type !== "subagent.called" || receipt === undefined) { throw new Error(`No child session and task receipt for ${marker}.`); } return { marker, sessionId: called.data.childSessionId, - taskId: receipt.data.backgroundTask.taskId, + taskId: receipt.taskId, turnId: called.data.turnId, }; }); @@ -124,7 +119,7 @@ export default defineTaskEval({ turn.notEvent("step.started"); } await t.require(reported, equals(true)); - t.calledSubagent("fanout-worker", { count: 2 }); + t.calledSubagent("fanout-worker", { status: "completed", count: 2 }); t.notCalledTool("task_peek"); t.noFailedActions(); diff --git a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.accepted-nonterminal.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.accepted-nonterminal.eval.ts index 9fbd4bd51..e03ff8c89 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.accepted-nonterminal.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.accepted-nonterminal.eval.ts @@ -5,7 +5,6 @@ import { requireTaskView, sendAndFollowQueuedTurn, waitForTaskInput, - waitForTaskStatus, } from "./shared.js"; import { defineTaskEval } from "./task-transition.js"; @@ -21,9 +20,11 @@ export default defineTaskEval({ const started = await t.send("TASK-CANCEL-SETUP"); started.expectOk(); started.messageIncludes("TASK-CANCEL-READY"); - started.event("subagent.completed", { + started.event("action.result", { count: 1, - data: { backgroundTask: { status: "working" }, subagentName: "fanout-worker" }, + data: { + result: { kind: "tool-result", output: { status: "working" }, toolName: "fanout-worker" }, + }, }); const taskId = requireBackgroundTaskId(started); @@ -42,12 +43,12 @@ export default defineTaskEval({ ), ); - const verified = await waitForTaskStatus( + // Cancellation is retained by the parent before the control call returns. + // The next turn must observe it without retrying a child-state read. + const { turn: verified } = await sendAndFollowQueuedTurn( t, + `TASK-CANCEL-VERIFY ${taskId}`, cancelled.session, - "TASK-CANCEL-VERIFY", - taskId, - "cancelled", ); verified.expectOk(); verified.messageIncludes("TASK-CANCEL-STATUS"); diff --git a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.noop-already-cancelled.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.noop-already-cancelled.eval.ts index b16ac58c1..74a7e27db 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.noop-already-cancelled.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.cancel.noop-already-cancelled.eval.ts @@ -25,9 +25,11 @@ export default defineTaskEval({ const started = await t.send("TASK-CANCEL-SETUP"); started.expectOk(); started.messageIncludes("TASK-CANCEL-READY"); - started.event("subagent.completed", { + started.event("action.result", { count: 1, - data: { backgroundTask: { status: "working" }, subagentName: "fanout-worker" }, + data: { + result: { kind: "tool-result", output: { status: "working" }, toolName: "fanout-worker" }, + }, }); const taskId = requireBackgroundTaskId(started); diff --git a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.complete.accepted-nonterminal.child-tool-surface.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.complete.accepted-nonterminal.child-tool-surface.eval.ts index 11f6c745d..435464e99 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.complete.accepted-nonterminal.child-tool-surface.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.complete.accepted-nonterminal.child-tool-surface.eval.ts @@ -16,13 +16,16 @@ export default defineTaskEval({ "Alice asks Bob to summarize the available tools for a background task.", ); started.expectOk(); - started.calledSubagent("tool-surface-worker", { count: 1 }); - started.event("subagent.completed", { + started.calledSubagent("tool-surface-worker", { status: "working", count: 1 }); + started.event("action.result", { count: 1, data: { - backgroundTask: { status: "working" }, - callId: "task-child-tool-surface", - subagentName: "tool-surface-worker", + result: { + kind: "tool-result", + output: { status: "working" }, + callId: "task-child-tool-surface", + toolName: "tool-surface-worker", + }, }, }); const taskId = requireBackgroundTaskId(started); @@ -31,6 +34,11 @@ export default defineTaskEval({ started, ]); completed.turn.expectOk(); + t.event("subagent.completed", { + count: 1, + data: { callId: "task-child-tool-surface", subagentName: "tool-surface-worker" }, + }); + t.calledSubagent("tool-surface-worker", { status: "completed", count: 1 }); const report = completed.turn.message; if (report === undefined) throw new Error("Parent did not return the child's tool report."); await t.require( diff --git a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.fail.accepted-nonterminal.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.fail.accepted-nonterminal.eval.ts index 9ad9907ba..1deb7f708 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.lifecycle.fail.accepted-nonterminal.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.lifecycle.fail.accepted-nonterminal.eval.ts @@ -18,12 +18,15 @@ export default defineTaskEval({ const started = await t.send("TASK-A2-CHILD-FAILURE"); started.expectOk(); started.messageIncludes("TASK-A2-CHILD-FAILURE-STARTED"); - started.event("subagent.completed", { + started.event("action.result", { count: 1, data: { - backgroundTask: { status: "working" }, - callId: CALL_ID, - subagentName: "busy-worker", + result: { + kind: "tool-result", + output: { status: "working" }, + callId: CALL_ID, + toolName: "busy-worker", + }, }, }); const taskId = requireBackgroundTaskId(started); diff --git a/e2e/fixtures/fixture-tasks/evals/task.parent-interaction.send.accepted-live-children.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.parent-interaction.send.accepted-live-children.eval.ts index 250da41e8..92808f803 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.parent-interaction.send.accepted-live-children.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.parent-interaction.send.accepted-live-children.eval.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import { type EveEvalTurn, type InputRequest } from "eve/evals"; import { satisfies } from "eve/evals/expect"; @@ -22,7 +23,7 @@ export default defineTaskEval({ const started = await t.send("TASK-FANOUT-PARENT-UPDATES"); started.expectOk(); started.messageIncludes("TASK-FANOUT-STARTED"); - started.calledSubagent("fanout-worker", { count: FANOUT_SIZE }); + started.calledSubagent("fanout-worker", { status: "working", count: FANOUT_SIZE }); const taskIds = backgroundTaskIds(started); await t.require( @@ -79,9 +80,5 @@ function collectReleaseRequests(turn: EveEvalTurn, requests: Map - event.type === "subagent.completed" && event.data.backgroundTask !== undefined - ? [event.data.backgroundTask.taskId] - : [], - ); + return taskReceipts(turn.events).map(({ taskId }) => taskId); } diff --git a/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.eval.ts index 24cd57ab8..258986bd0 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.eval.ts @@ -1,3 +1,4 @@ +import { taskReceipts } from "@eve-e2e/config/task-receipts"; import { type EveEvalContext, type EveEvalTurn, type InputRequest } from "eve/evals"; import { satisfies } from "eve/evals/expect"; @@ -25,7 +26,7 @@ export default defineTaskEval({ const started = await t.send("TASK-PARENT-WAKE-UPDATES"); started.expectOk(); started.messageIncludes("TASK-FANOUT-STARTED"); - started.calledSubagent("fanout-worker", { count: FANOUT_SIZE }); + started.calledSubagent("fanout-worker", { status: "working", count: FANOUT_SIZE }); const taskIds = backgroundTaskIds(started); await t.require( @@ -129,11 +130,7 @@ function collectReleaseRequests(turn: EveEvalTurn, requests: Map - event.type === "subagent.completed" && event.data.backgroundTask !== undefined - ? [event.data.backgroundTask.taskId] - : [], - ); + return taskReceipts(turn.events).map(({ taskId }) => taskId); } function completedNotificationTaskIds(turn: EveEvalTurn): readonly string[] { diff --git a/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.lifecycle-order.eval.ts b/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.lifecycle-order.eval.ts index 6a4433348..3b400c00d 100644 --- a/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.lifecycle-order.eval.ts +++ b/e2e/fixtures/fixture-tasks/evals/task.parent.wake.emitted-ready.lifecycle-order.eval.ts @@ -32,12 +32,12 @@ export default defineTaskEval({ await driver.active(parent); await driver.release(a); - // Owner completion includes the awaited wakeTaskParentStep, unlike child stream completion. + // Owner completion includes the awaited terminal notification, unlike child stream completion. await driver.settled("A"); await driver.active(parent); await driver.release(b); - // This owner had exactly one agent invocation. Its second agent-request - // forwarding step is settlement, and must precede its successful task wake. + // This owner sends three notifications: agent invocation, agent settlement, + // and task completion. All must be delivered before its run settles. await driver.settled("B", true); const child = await t.target.watchTurn(b.sessionId, { startIndex: 0 }).result(); child.event("session.failed"); diff --git a/packages/eve/extension-contracts/reports/channel/v27.json b/packages/eve/extension-contracts/reports/channel/v27.json new file mode 100644 index 000000000..d29be735f --- /dev/null +++ b/packages/eve/extension-contracts/reports/channel/v27.json @@ -0,0 +1,21 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "channel", + "epoch": 27, + "sha256": "62f1b00ed788ab6ebc5aa37955471f3f51c9c28bab02a31e9550bdd5b2dcbacb", + "exports": [ + "DELETE", + "GET", + "HEAD", + "OPTIONS", + "PATCH", + "POST", + "PUT", + "WS", + "createWebSocketUpgradeServer", + "defineChannel", + "disableRoute", + "isChannel", + "isDisabledRouteSentinel" + ] +} diff --git a/packages/eve/extension-contracts/reports/dynamicTool/v51.json b/packages/eve/extension-contracts/reports/dynamicTool/v51.json new file mode 100644 index 000000000..ae2789d70 --- /dev/null +++ b/packages/eve/extension-contracts/reports/dynamicTool/v51.json @@ -0,0 +1,16 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "dynamicTool", + "epoch": 51, + "sha256": "20a37ed9521bad46ce3e857dd19c92082ce7e7555c53cf54d547a7afd9124a75", + "exports": [ + "DynamicToolEntry", + "DynamicToolEvents", + "DynamicToolResult", + "DynamicToolSet", + "auto", + "defineDurableCallback", + "defineDurableSchema", + "defineDynamic" + ] +} diff --git a/packages/eve/extension-contracts/reports/tool/v53.json b/packages/eve/extension-contracts/reports/tool/v53.json new file mode 100644 index 000000000..dd27fd433 --- /dev/null +++ b/packages/eve/extension-contracts/reports/tool/v53.json @@ -0,0 +1,27 @@ +{ + "kind": "eve-extension-capability-contract", + "capability": "tool", + "epoch": 53, + "sha256": "33f27c953c1408ddcea510b925adbfb7e6eb5147c84a060f9a084d7d5d2c6eee", + "exports": [ + "AgentRouterInput", + "AgentRouterTool", + "WorkflowStepToolContext", + "WorkflowTool", + "WorkflowToolInput", + "WorkflowToolOptions", + "agentRouter", + "defaultWebSearch", + "defineTool", + "defineWorkflowTool", + "disableTool", + "evaluate", + "isDisabledToolSentinel", + "isWebSearchToolDefinition", + "toolOutput", + "toolOutputPart", + "toolResultFrom", + "webSearch", + "workflow" + ] +} diff --git a/packages/eve/src/cli/dev/tui/runner.test.ts b/packages/eve/src/cli/dev/tui/runner.test.ts index 5ff20b0c5..b249d0f34 100644 --- a/packages/eve/src/cli/dev/tui/runner.test.ts +++ b/packages/eve/src/cli/dev/tui/runner.test.ts @@ -3622,6 +3622,67 @@ describe("EveTUIRunner renderer teardown", () => { expect(completeSubagent).toHaveBeenCalledWith({ authoritative: true, callId: "call-child" }); }); + it("keeps a subagent section open when action.result returns a working receipt", async () => { + const backgroundSubagent = vi.fn(); + const completeSubagent = vi.fn(); + const runner = new EveTUIRunner({ + name: "Weather Agent", + renderer: fakeRenderer({ + readPrompt: vi.fn().mockResolvedValueOnce("delegate").mockResolvedValueOnce(undefined), + renderStream: vi.fn(async (result) => { + for await (const event of result.events as AsyncIterable) void event; + }), + subagents: { + begin: vi.fn(), + background: backgroundSubagent, + upsertStep: vi.fn(), + upsertTool: vi.fn(), + removeTool: vi.fn(), + markChildToolCallId: vi.fn(), + complete: completeSubagent, + }, + }), + session: sessionYielding([ + { + type: "subagent.called", + data: { + callId: "call-child", + childSessionId: "child-session", + childStreamPath: "/eve/v1/session/child-session/stream", + name: "researcher", + sequence: 0, + sessionId: "parent-session", + toolName: "researcher", + turnId: "turn-parent", + workflowId: "workflow-parent", + }, + }, + { + type: "action.result", + data: { + status: "completed", + sequence: 1, + stepIndex: 0, + turnId: "turn-parent", + result: { + kind: "tool-result", + callId: "call-child", + output: { agentId: "agent-1", status: "working", taskId: "task_123" }, + toolName: "researcher", + }, + }, + }, + { type: "turn.completed", data: { sequence: 0, turnId: "turn-parent" } }, + { type: "session.waiting", data: { wait: "next-user-message" } }, + ]), + }); + + await runner.run(); + + expect(backgroundSubagent).toHaveBeenCalledWith({ callId: "call-child" }); + expect(completeSubagent).not.toHaveBeenCalled(); + }); + it("does not settle a subagent section when completed carries a background receipt", async () => { const backgroundSubagent = vi.fn(); const completeSubagent = vi.fn(); diff --git a/packages/eve/src/cli/dev/tui/runner.ts b/packages/eve/src/cli/dev/tui/runner.ts index 2854cabcf..42c934f92 100644 --- a/packages/eve/src/cli/dev/tui/runner.ts +++ b/packages/eve/src/cli/dev/tui/runner.ts @@ -1,3 +1,4 @@ +import { isJsonObjectValue } from "#shared/json.js"; import type { ModelAccessChange } from "#shared/model-connection.js"; import { SteeringStream } from "#cli/dev/tui/steering-stream.js"; import { @@ -2428,6 +2429,17 @@ async function* eveEventsToTUIStream( case "action.result": { const resultEvent = event as ActionResultStreamEvent; + const result = resultEvent.data.result; + const output = "output" in result ? result.output : undefined; + if ( + resultEvent.data.status === "completed" && + isJsonObjectValue(output) && + output.status === "working" && + typeof output.taskId === "string" && + typeof output.agentId === "string" + ) { + onSubagentBackgrounded?.(result.callId); + } if (resultEvent.data.result.kind !== "tool-result") { break; } diff --git a/packages/eve/src/cli/dev/tui/subagent-pump.test.ts b/packages/eve/src/cli/dev/tui/subagent-pump.test.ts index 1591ed0a9..c2823d9da 100644 --- a/packages/eve/src/cli/dev/tui/subagent-pump.test.ts +++ b/packages/eve/src/cli/dev/tui/subagent-pump.test.ts @@ -212,6 +212,16 @@ describe("SubagentPump.settleCancelledTurn", () => { }); describe("SubagentPump background receipts", () => { + it("retains a receipt that arrives before child dispatch", () => { + const view = fakeView(); + const pump = new SubagentPump({ view, formatActionResultError: () => "failed" }); + pump.background("call-1"); + pump.begin(subagentCalled("call-1")); + pump.settleCancelledTurn("turn-1"); + expect(view.background).toHaveBeenCalledWith({ callId: "call-1" }); + expect(view.complete).not.toHaveBeenCalled(); + }); + it("keeps the section open until the child stream reaches its own boundary", async () => { const child = pushableChildStream(); const client = new Client({ host: "http://localhost:3000" }); diff --git a/packages/eve/src/cli/dev/tui/subagent-pump.ts b/packages/eve/src/cli/dev/tui/subagent-pump.ts index 690ab5e41..e1ddc6e84 100644 --- a/packages/eve/src/cli/dev/tui/subagent-pump.ts +++ b/packages/eve/src/cli/dev/tui/subagent-pump.ts @@ -137,6 +137,8 @@ export class SubagentPump { | ((subagentName: string, toolName: string, output: unknown) => Promise) | undefined; readonly #runs = new Map(); + // Task admission can return its receipt before the child dispatch event arrives. + readonly #pendingBackgroundCalls = new Set(); readonly #pumps = new Map(); /** Durable child cursor shared by repeated calls into one conversation subagent. */ readonly #childStreamIndices = new Map(); @@ -184,6 +186,7 @@ export class SubagentPump { this.#view?.markChildToolCallId(callId); if (existing !== undefined && existing.status !== "open") return; this.#view?.begin({ callId, name: called.data.name }); + if (this.#pendingBackgroundCalls.delete(callId)) this.background(callId); if (existing !== undefined) return; this.#activateOrQueue(callId); } @@ -204,7 +207,10 @@ export class SubagentPump { */ background(callId: string): void { const run = this.#runs.get(callId); - if (run === undefined) return; + if (run === undefined) { + this.#pendingBackgroundCalls.add(callId); + return; + } run.background = true; if (run.status === "authoritative") return; this.#view?.background({ callId }); @@ -216,6 +222,7 @@ export class SubagentPump { } this.#pumps.clear(); this.#runs.clear(); + this.#pendingBackgroundCalls.clear(); this.#childStreamIndices.clear(); this.#activeChildCalls.clear(); this.#queuedChildCalls.clear(); diff --git a/packages/eve/src/compiler/extension-compatibility.ts b/packages/eve/src/compiler/extension-compatibility.ts index f4207675e..313eeeca4 100644 --- a/packages/eve/src/compiler/extension-compatibility.ts +++ b/packages/eve/src/compiler/extension-compatibility.ts @@ -22,11 +22,8 @@ interface ExtensionCapabilityContract { const EXTENSION_CAPABILITY_CONTRACTS = { extension: { current: 1, supported: [1], dropped: {} }, tool: { - current: 52, - supported: [ - 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 34, 35, 44, 45, 46, 49, 50, 51, - 52, - ], + current: 53, + supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 29, 30, 31, 32, 34, 35, 53], dropped: { 14: "TaskExec.delegated was removed; migrate to workflow-backed background tools", 15: "TaskExec replaces stageEffect with send", @@ -42,6 +39,7 @@ const EXTENSION_CAPABILITY_CONTRACTS = { 25: "TaskExec.delegated was removed; migrate to workflow-backed background tools", 26: "Background tools now use task yield descriptors", 27: "TaskExec.delegated was removed; migrate to workflow-backed background tools", + 28: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", 33: "ctx.agent now accepts the subagent name as its first argument, derives invocation identity internally, and infers structured output types", 36: "experimental_workflow and eve/tools/workflow were removed; migrate to the workflow factory from eve/tools/workflow", 37: "experimental_workflow and eve/tools/workflow were removed; migrate to the workflow factory from eve/tools/workflow", @@ -51,15 +49,21 @@ const EXTENSION_CAPABILITY_CONTRACTS = { 41: "workflow no longer accepts agents and its options argument is optional; use workflow() or workflow({ maxSubagents })", 42: "Legacy session history migration was removed; user-role messages require current provenance kinds.", 43: "Legacy session history migration was removed; user-role messages require current provenance kinds.", + 44: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", + 45: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", + 46: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", 47: "eve/experimental/evaluate was removed; import evaluate from eve/ai", 48: "eve/experimental/evaluate was removed; import evaluate from eve/ai", + 49: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", + 50: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", + 51: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", + 52: "Background defineTool and TaskExec were removed; use defineWorkflowTool for durable background work.", }, }, dynamicTool: { - current: 50, + current: 51, supported: [ - 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 28, 29, 30, 31, 32, - 33, 41, 48, 49, 50, + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 31, 32, 33, 51, ], dropped: { 21: "Message and reasoning append events now expose deltas instead of cumulative snapshots.", @@ -68,6 +72,9 @@ const EXTENSION_CAPABILITY_CONTRACTS = { 25: "TaskExec.delegated was removed; migrate to workflow-backed background tools", 26: "Background tools now use task yield descriptors", 27: "TaskExec.delegated was removed; migrate to workflow-backed background tools", + 28: "Background dynamic tools were removed; use a static defineWorkflowTool for durable background work.", + 29: "Background dynamic tools were removed; use a static defineWorkflowTool for durable background work.", + 30: "Background dynamic tools were removed; use a static defineWorkflowTool for durable background work.", 34: "Legacy session history migration was removed; user-role messages require current provenance kinds.", 35: "workflowMaxSubagents was removed with experimental_workflow; configure generated-program limits with the workflow factory", 36: "workflowMaxSubagents was removed with experimental_workflow; configure generated-program limits with the workflow factory", @@ -75,23 +82,31 @@ const EXTENSION_CAPABILITY_CONTRACTS = { 38: "workflowMaxSubagents was removed with experimental_workflow; configure generated-program limits with the workflow factory", 39: "Legacy session history migration was removed; user-role messages require current provenance kinds.", 40: "Legacy session history migration was removed; user-role messages require current provenance kinds.", + 41: "Background dynamic tools were removed; use a static defineWorkflowTool for durable background work.", 42: "autoModel and eve/experimental/evaluate were removed; import auto from eve/models", 43: "autoModel and eve/experimental/evaluate were removed; import auto from eve/models", 44: "autoModel and eve/experimental/evaluate were removed; import auto from eve/models", 45: "autoModel and eve/experimental/evaluate were removed; import auto from eve/models", 46: "autoModel and eve/experimental/evaluate were removed; import auto from eve/models", 47: "autoModel and eve/experimental/evaluate were removed; import auto from eve/models", + 48: "Background dynamic tools were removed; use a static defineWorkflowTool for durable background work.", + 49: "Background dynamic tools were removed; use a static defineWorkflowTool for durable background work.", + 50: "Background dynamic tools were removed; use a static defineWorkflowTool for durable background work.", }, }, channel: { - current: 26, - supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 23, 24, 25, 26], + current: 27, + supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 27], dropped: { 12: "Message and reasoning append events now expose deltas instead of cumulative snapshots.", 19: "Continuation rekey was removed; channel extensions must use additive continuation.alias instead.", 20: "Continuation rekey was removed; channel extensions must use additive continuation.alias instead.", 21: "Continuation rekey was removed; channel extensions must use additive continuation.alias instead.", 22: "Continuation rekey was removed; channel extensions must use additive continuation.alias instead.", + 23: "Task views no longer expose executor bindings; background work is owned by workflow runs.", + 24: "Task views no longer expose executor bindings; background work is owned by workflow runs.", + 25: "Task views no longer expose executor bindings; background work is owned by workflow runs.", + 26: "Task views no longer expose executor bindings; background work is owned by workflow runs.", }, }, schedule: { diff --git a/packages/eve/src/compiler/normalize-manifest.test.ts b/packages/eve/src/compiler/normalize-manifest.test.ts index 2fc899a30..39785769e 100644 --- a/packages/eve/src/compiler/normalize-manifest.test.ts +++ b/packages/eve/src/compiler/normalize-manifest.test.ts @@ -391,7 +391,6 @@ describe("compileAgentManifest source graph", () => { rootOnly: true, task: { nodeId: "__root__", - resultKind: "subagent", workflowId: expect.stringContaining("subagentToolExecuteWorkflow"), }, }); diff --git a/packages/eve/src/compiler/normalize-tool.ts b/packages/eve/src/compiler/normalize-tool.ts index f16cff632..628c71a54 100644 --- a/packages/eve/src/compiler/normalize-tool.ts +++ b/packages/eve/src/compiler/normalize-tool.ts @@ -102,6 +102,18 @@ export async function compileToolEntry( } const workflowId = readWorkflowFunctionId(entry.definition.execute); + if ( + entry.definition.execution === "background" && + workflowId === undefined && + !( + entry.definition.behavior?.handling?.kind === "dispatch" && + entry.definition.behavior.handling.action === "self-agent" + ) + ) { + throw new Error( + `Background tool "${source.logicalPath}" must use defineWorkflowTool(). defineTool() tools run in the foreground.`, + ); + } const shape = { lifetime: entry.definition.execution === "background" ? ("task" as const) : ("step" as const), suspend: workflowId === undefined ? ("none" as const) : ("workflow" as const), diff --git a/packages/eve/src/context/build-dynamic-tools.ts b/packages/eve/src/context/build-dynamic-tools.ts index 1120d69b8..a67842a79 100644 --- a/packages/eve/src/context/build-dynamic-tools.ts +++ b/packages/eve/src/context/build-dynamic-tools.ts @@ -103,6 +103,12 @@ export function replayDynamicTools( if (metadata.length > 0 && scope.sessionId.length === 0) { throw new Error("Dynamic tool replay requires a session id."); } + const background = metadata.find((entry) => entry.execution === "background"); + if (background !== undefined) { + throw new Error( + `Dynamic tool "${background.name}" used removed background execution. Move durable background work to a static defineWorkflowTool().`, + ); + } return metadata.map((entry) => { const owner = { ...entry, ...scope }; const approvalKeyReference = entry.callbacks.approvalKey; @@ -142,38 +148,15 @@ export function replayDynamicTools( } = { availableInSubagents: entry.availableInSubagents, description: entry.description, - execute: - entry.execution === "background" - ? createToolExecuteWithAuth({ - execution: "background", - scope: entry.name, - execute: (input, context, task) => { - if (execute === undefined) { - throw missingCallbackError(entry, "execute"); - } - return callDurableDynamicCallback( - execute, - executeReference.closure, - input, - context, - task, - ); - }, - }) - : createToolExecuteWithAuth({ - scope: entry.name, - execute: (input, context) => { - if (execute === undefined) { - throw missingCallbackError(entry, "execute"); - } - return callDurableDynamicCallback( - execute, - executeReference.closure, - input, - context, - ); - }, - }), + execute: createToolExecuteWithAuth({ + scope: entry.name, + execute: (input, context) => { + if (execute === undefined) { + throw missingCallbackError(entry, "execute"); + } + return callDurableDynamicCallback(execute, executeReference.closure, input, context); + }, + }), inputSchema: replayDynamicToolSchema(entry, owner, "inputSchema")!, name: entry.name, execution: entry.execution, diff --git a/packages/eve/src/context/dynamic-subagent-lifecycle.test.ts b/packages/eve/src/context/dynamic-subagent-lifecycle.test.ts index d378eb643..052b6eb62 100644 --- a/packages/eve/src/context/dynamic-subagent-lifecycle.test.ts +++ b/packages/eve/src/context/dynamic-subagent-lifecycle.test.ts @@ -74,7 +74,6 @@ describe("dynamic subagent lifecycle", () => { { description: expect.stringContaining("Research the request."), name: "researcher", - resultKind: "subagent", }, ]); expect(getDynamicSubagentSelection(ctx, resolver.nodeId)).toBeDefined(); @@ -185,7 +184,7 @@ describe("dynamic subagent lifecycle", () => { resolvers: [resolver], }); expect(buildDynamicSubagentTools(ctx)[0]?.execution).toBe("background"); - expect(buildDynamicSubagentTools(ctx)[0]?.resultKind).toBe("subagent"); + expect(buildDynamicSubagentTools(ctx)[0]?.nodeId).toEqual(expect.any(String)); await dispatchDynamicSubagentEvent({ ctx, @@ -194,7 +193,7 @@ describe("dynamic subagent lifecycle", () => { resolvers: [resolver], }); expect(buildDynamicSubagentTools(ctx)[0]?.execution).toBe("background"); - expect(buildDynamicSubagentTools(ctx)[0]?.resultKind).toBe("subagent"); + expect(buildDynamicSubagentTools(ctx)[0]?.nodeId).toEqual(expect.any(String)); }); it("exposes a dynamic selection without root configuration", async () => { @@ -328,7 +327,6 @@ describe("dynamic subagent lifecycle", () => { { description: expect.stringContaining("Research on the remote deployment."), name: "researcher", - resultKind: "subagent", }, ]); expect(getDynamicSubagentSelection(ctx, resolver.nodeId)).toMatchObject({ diff --git a/packages/eve/src/context/dynamic-tool-lifecycle.test.ts b/packages/eve/src/context/dynamic-tool-lifecycle.test.ts index 39f097d7b..1fdd359cc 100644 --- a/packages/eve/src/context/dynamic-tool-lifecycle.test.ts +++ b/packages/eve/src/context/dynamic-tool-lifecycle.test.ts @@ -1,8 +1,8 @@ +import { z } from "#compiled/zod/index.js"; import { defineWorkflowTool } from "#tools/workflow-definition.js"; import { asSchema } from "ai"; import { describe, expect, it, vi } from "vitest"; -import { z } from "#compiled/zod/index.js"; import type { DynamicToolEntry } from "#tools/dynamic.js"; import { isCurrentDynamicToolMetadata, @@ -16,7 +16,7 @@ import { type ApprovalResponseContext, } from "#approval/definition.js"; import { defineDurableCallback } from "#public/tools/index.js"; -import { defineTool, type TaskExec, type ToolContext } from "#tools/definition.js"; +import { defineTool, type ToolContext } from "#tools/definition.js"; import type { JsonObject } from "#shared/json.js"; import { serializeOutputSchema, type ToolSchema } from "#tools/schema.js"; @@ -1149,61 +1149,6 @@ describe("dispatchDynamicToolEvent", () => { expect(buildDynamicTools(restored)[0]?.availableInSubagents).toBe(false); }); - it("persists background execution and forwards TaskExec when replaying", async () => { - const ctx = createCtx(); - const stepFn = vi.fn(async function* ( - _closure: unknown, - _input: unknown, - _toolCtx: unknown, - task: TaskExec, - ) { - yield task.postMessage("replayed"); - return { done: true }; - }); - const resolver = createResolver("background", ["session.started"], () => { - const entry = defineTool({ - description: "delegate background work", - execution: "background", - inputSchema: z.strictObject({}), - async *execute(_input, _toolCtx, task) { - yield task.postMessage("replayed"); - return { done: true }; - }, - }); - stampDurableDynamicToolCallbacks(entry, { - inputSchema: { callback: () => entry.inputSchema, closure: {} }, - execute: { callback: stepFn as never, closure: {} }, - }); - return { background_task: entry }; - }); - - await dispatchDynamicToolEvent({ - ctx, - resolvers: [resolver], - messages: [], - event: makeEvent("session.started"), - }); - const restored = await deserializeContext(serializeContext(ctx)); - const [metadata] = restored.get(SessionDynamicToolMetadataKey) ?? []; - expect(metadata?.execution).toBe("background"); - - const [tool] = buildDynamicTools(restored); - expect(tool?.execution).toBe("background"); - const task: TaskExec = { - binding: { taskId: "task-1", token: "token-1" }, - postMessage: (message) => ({ kind: "eve:task-message", message }), - send: vi.fn(), - session: {} as TaskExec["session"], - task: {} as TaskExec["task"], - taskId: "task-1", - }; - const output = tool!.execute!({}, executeOptions, task) as AsyncIterable; - const updates = []; - for await (const update of output) updates.push(update); - expect(updates).toEqual([{ kind: "eve:task-message", message: "replayed" }]); - expect(stepFn).toHaveBeenCalledWith({}, {}, expect.anything(), task); - }); - it("replays session tools from durable metadata on a fresh step", async () => { const ctx = createCtx(); diff --git a/packages/eve/src/context/dynamic-tool-lifecycle.ts b/packages/eve/src/context/dynamic-tool-lifecycle.ts index 2604f7ba5..0491868be 100644 --- a/packages/eve/src/context/dynamic-tool-lifecycle.ts +++ b/packages/eve/src/context/dynamic-tool-lifecycle.ts @@ -334,7 +334,6 @@ function createMetadata(input: { name: input.name, }), description: input.entry.description, - execution: input.entry.execution === "background" ? "background" : undefined, entryKey: input.entryKey, inputSchema: serializeInputSchema(input.entry.inputSchema), name: input.name, diff --git a/packages/eve/src/evals/assertions/run.test.ts b/packages/eve/src/evals/assertions/run.test.ts index 599c47fc1..27c37351b 100644 --- a/packages/eve/src/evals/assertions/run.test.ts +++ b/packages/eve/src/evals/assertions/run.test.ts @@ -220,17 +220,24 @@ describe("run assertions", () => { const result = makeResult({ derived: { subagentCalls: [ - subagentCall("child", "pending"), + subagentCall("child", "working"), subagentCall("child", "completed"), subagentCall("child", "failed"), - subagentCall("child", "rejected"), + subagentCall("child", "cancelled"), + subagentCall("child", "input_required"), ], - subagentCallCount: 4, + subagentCallCount: 5, }, }); expect((await Run.calledSubagent("child").evaluate(result)).score).toBe(1); - for (const status of ["pending", "completed", "failed", "rejected"] as const) { + for (const status of [ + "working", + "input_required", + "completed", + "failed", + "cancelled", + ] as const) { expect((await Run.calledSubagent("child", { status, count: 1 }).evaluate(result)).score).toBe( 1, ); @@ -242,7 +249,7 @@ describe("run assertions", () => { ( await Run.calledSubagent("child", { count: (count) => count >= 4, - status: "pending", + status: "working", }).evaluate(result) ).score, ).toBe(0); diff --git a/packages/eve/src/evals/runner/derive-run-facts.test.ts b/packages/eve/src/evals/runner/derive-run-facts.test.ts index 265451829..0a2109678 100644 --- a/packages/eve/src/evals/runner/derive-run-facts.test.ts +++ b/packages/eve/src/evals/runner/derive-run-facts.test.ts @@ -347,6 +347,80 @@ describe("deriveRunFacts", () => { expect(facts.reasoningBlockCount).toBe(2); }); + it("keeps a working receipt working until an actual result arrives", () => { + const admission: UnstampedMessageStreamEvent = { + type: "subagent.completed", + data: { + callId: "c1", + subagentName: "researcher", + output: "working", + backgroundTask: { status: "working", taskId: "task-1" }, + }, + }; + const receipt: UnstampedMessageStreamEvent = { + type: "action.result", + data: { + sequence: 1, + stepIndex: 0, + turnId: "t1", + status: "completed", + result: { + kind: "subagent-result", + origin: "child", + callId: "c1", + subagentName: "researcher", + backgroundTask: { status: "working", taskId: "task-1" }, + output: { status: "working", taskId: "task-1" }, + outcome: { + kind: "parked", + result: { kind: "succeeded", output: "working" }, + usageDelta: { + inputTokens: 0, + outputTokens: 0, + cacheReadTokens: 0, + cacheWriteTokens: 0, + }, + }, + }, + }, + }; + const toolReceipt = actionResult({ + callId: "c1", + toolName: "researcher", + output: { agentId: "agent-1", status: "working", taskId: "task-1" }, + }); + for (const events of [ + [admission], + [receipt], + [toolReceipt], + [admission, receipt], + [receipt, admission], + ]) { + expect(derive(events).subagentCalls).toEqual([ + expect.objectContaining({ callId: "c1", status: "working" }), + ]); + expect(derive(events).subagentCalls[0]?.output).toBeUndefined(); + } + const completed: UnstampedMessageStreamEvent = { + type: "subagent.completed", + data: { callId: "c1", subagentName: "researcher", output: "actual result" }, + }; + expect(derive([toolReceipt, completed, admission, receipt, toolReceipt]).subagentCalls).toEqual( + [expect.objectContaining({ callId: "c1", status: "completed", output: "actual result" })], + ); + for (const status of ["failed", "rejected"] as const) { + const failure = subagentResult({ + callId: "c1", + subagentName: "researcher", + output: "child failed", + status, + }); + expect(derive([admission, failure, admission, receipt]).subagentCalls).toEqual([ + expect.objectContaining({ callId: "c1", status: "failed", output: "child failed" }), + ]); + } + }); + it("joins subagent.called with subagent.completed by call id", () => { const events: UnstampedMessageStreamEvent[] = [ turnStarted("t1", 0), @@ -419,6 +493,47 @@ describe("deriveRunFacts", () => { ]; const facts = derive(events); expect(facts.subagentCalls.map((call) => call.name)).toEqual(["inline-agent"]); + expect(facts.subagentCalls[0]?.status).toBe("working"); + }); + + it("preserves explicit cancellation even when the action reports failure or a late completion", () => { + const cancelled: UnstampedMessageStreamEvent = { + type: "action.result", + data: { + sequence: 1, + stepIndex: 0, + turnId: "t1", + status: "failed", + result: { + callId: "c1", + kind: "subagent-result", + origin: "child", + subagentName: "researcher", + isError: true, + output: "The agent invocation was cancelled.", + outcome: { + kind: "parked", + result: { kind: "cancelled" }, + usageDelta: { + inputTokens: 0, + outputTokens: 0, + cacheReadTokens: 0, + cacheWriteTokens: 0, + }, + }, + }, + }, + }; + const lateCompletion: UnstampedMessageStreamEvent = { + type: "subagent.completed", + data: { callId: "c1", subagentName: "researcher", output: "late result" }, + }; + for (const events of [[cancelled], [cancelled, lateCompletion]]) { + expect(derive(events).subagentCalls[0]).toMatchObject({ + status: "cancelled", + output: "The agent invocation was cancelled.", + }); + } }); it("records every subagent invocation separately", () => { diff --git a/packages/eve/src/evals/runner/derive-run-facts.ts b/packages/eve/src/evals/runner/derive-run-facts.ts index 741e64385..03c455739 100644 --- a/packages/eve/src/evals/runner/derive-run-facts.ts +++ b/packages/eve/src/evals/runner/derive-run-facts.ts @@ -1,3 +1,4 @@ +import { isJsonObjectValue } from "#shared/json.js"; import type { MessageStreamEvent } from "#protocol/message.js"; import { LOAD_SKILL_TOOL_NAME } from "#runtime/skills/fragment-context.js"; import type { InputRequest } from "#shared/input.js"; @@ -90,7 +91,7 @@ export function deriveRunFacts( const call: MutableSubagentCall = { callId, name, - status: "pending", + status: "working", turnIndex: Math.max(turnIndex, 0), sessionId, }; @@ -123,10 +124,26 @@ export function deriveRunFacts( const call = ensureToolCall(result.callId, result.toolName, {}); call.output = result.output; call.status = status; + const output = result.output; + if ( + status === "completed" && + isJsonObjectValue(output) && + output.status === "working" && + typeof output.taskId === "string" && + typeof output.agentId === "string" + ) { + ensureSubagentCall(result.callId, result.toolName); + } } else if (result.kind === "subagent-result") { const call = ensureSubagentCall(result.callId, result.subagentName); + // A working receipt settles dispatch, not the delegated work. + if (result.origin === "child" && result.backgroundTask !== undefined) break; call.output = call.output ?? result.output; - call.status = status; + if (result.origin === "child" && result.outcome.result.kind === "cancelled") { + call.status = "cancelled"; + } else { + call.status = status === "rejected" ? "failed" : status; + } } break; } @@ -147,8 +164,9 @@ export function deriveRunFacts( case "subagent.completed": { const call = ensureSubagentCall(event.data.callId, event.data.subagentName); + if (event.data.backgroundTask !== undefined || call.status !== "working") break; call.output = event.data.output; - if (call.status === "pending") call.status = "completed"; + call.status = "completed"; break; } diff --git a/packages/eve/src/evals/types.ts b/packages/eve/src/evals/types.ts index 91ac7fd22..9413a172e 100644 --- a/packages/eve/src/evals/types.ts +++ b/packages/eve/src/evals/types.ts @@ -15,6 +15,7 @@ import type { } from "#client/types.js"; import type { InputRequest, InputResponse } from "#shared/input.js"; import type { JsonObject, JsonValue } from "#shared/json.js"; +import type { TaskStatus } from "#tasks/types.js"; import type { AgentModelOptionsDefinition } from "#shared/agent-definition.js"; import type { EvalReporter } from "#evals/runner/reporters/types.js"; import type { @@ -25,7 +26,7 @@ import type { EveEvalToolCallMatchOptions, } from "#evals/match.js"; -/** Lifecycle outcome of an eval-observed tool or subagent action. */ +/** Lifecycle outcome of an eval-observed tool action. */ export type EveEvalActionStatus = "pending" | "completed" | "failed" | "rejected"; /** @@ -62,8 +63,8 @@ export interface EveEvalSubagentCall { readonly remoteUrl?: string; /** Output from the matching `subagent.completed` event; `undefined` when the call never completed. */ readonly output?: JsonValue; - /** Whether the delegation is unresolved, completed, failed, or rejected. */ - readonly status: EveEvalActionStatus; + /** Task lifecycle status inferred from the captured delegation events. */ + readonly status: TaskStatus; /** Zero-based index of the turn the delegation happened in. */ readonly turnIndex: number; /** Owning session id, when the runner knows it. */ diff --git a/packages/eve/src/execution/cancel-descendant-turns-step.test.ts b/packages/eve/src/execution/cancel-descendant-turns-step.test.ts index b1ae34070..d4297f5f3 100644 --- a/packages/eve/src/execution/cancel-descendant-turns-step.test.ts +++ b/packages/eve/src/execution/cancel-descendant-turns-step.test.ts @@ -1,3 +1,4 @@ +import { cancelWorkflowToolRun } from "#execution/tools/workflow/cancel.js"; import { afterEach, describe, expect, it, vi } from "vitest"; import { deserializeContext } from "#context/serialize.js"; @@ -12,6 +13,8 @@ import { requestWorkflowTurnCancellation } from "#execution/workflow-runtime.js" import { AGENT_HANDLES_STATE_KEY, type AgentHandle } from "#subagents/handles/store.js"; import type { HarnessSession } from "#harness/types.js"; +vi.mock("#execution/tools/workflow/cancel.js", () => ({ cancelWorkflowToolRun: vi.fn() })); + vi.mock("#context/serialize.js", () => ({ deserializeContext: vi.fn(), })); @@ -114,6 +117,56 @@ describe("cancelDescendantTurnsStep", () => { }); }); + it("signals owned children while their workflow is still unwinding", async () => { + const settlement = Promise.withResolvers(); + vi.mocked(cancelWorkflowToolRun).mockReturnValueOnce(settlement.promise); + vi.mocked(requestWorkflowTurnCancellation).mockResolvedValue({ + status: "accepted", + sessionId: "local-child", + }); + const cancellation = cancelDescendantTurnsStep({ + serializedContext: {}, + sessionState: createDurableSessionState({ + session: createSession({ + "eve.harness.emission": { + turnId: "turn_0", + stepIndex: 0, + sequence: 0, + sessionStarted: true, + }, + "eve.workflowTool": { + version: 3, + runs: [ + { + lifetime: "turn", + callId: "call", + toolName: "research", + origin: { turnId: "turn_0", stepIndex: 0 }, + address: { runId: "workflow", hookToken: "hook" }, + }, + ], + }, + [AGENT_HANDLES_STATE_KEY]: { + handles: [ + { + phase: "claimed", + ownerId: "workflow", + operationId: "op", + identity: LOCAL_RUNNING_HANDLE.identity, + address: LOCAL_RUNNING_HANDLE.address, + }, + ], + }, + }), + }), + }); + await Promise.resolve(); + expect(cancelWorkflowToolRun).toHaveBeenCalled(); + expect(requestWorkflowTurnCancellation).toHaveBeenCalledWith({ sessionId: "local-child" }); + settlement.resolve(); + await cancellation; + }); + it("does not deserialize remote context for local-only descendants", async () => { vi.mocked(requestWorkflowTurnCancellation).mockResolvedValue({ sessionId: "local-child", diff --git a/packages/eve/src/execution/cancel-descendant-turns-step.ts b/packages/eve/src/execution/cancel-descendant-turns-step.ts index 351907143..e94006fc6 100644 --- a/packages/eve/src/execution/cancel-descendant-turns-step.ts +++ b/packages/eve/src/execution/cancel-descendant-turns-step.ts @@ -1,3 +1,4 @@ +import { getPendingCoordinationBatch } from "#harness/coordination.js"; import type { CancelTurnResult } from "#channel/types.js"; import { deserializeContext } from "#context/serialize.js"; import { BundleKey } from "#runtime/sessions/runtime-context-keys.js"; @@ -9,7 +10,11 @@ import { } from "#subagents/remote-dispatch.js"; import { cancelWorkflowToolRun } from "#execution/tools/workflow/cancel.js"; import { requestWorkflowTurnCancellation } from "#execution/workflow-runtime.js"; -import { getWorkflowToolRuns, type WorkflowToolRunRecord } from "#harness/workflow-tool-runs.js"; +import { + getBlockingWorkflowToolRuns, + type BlockingWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; + import { getAgentHandleStore, type AgentHandle } from "#subagents/handles/store.js"; import { createLogger, logError } from "#internal/logging.js"; import type { RuntimeSubagentRegistry } from "#runtime/subagents/registry.js"; @@ -37,11 +42,15 @@ export async function cancelDescendantTurnsStep(input: { "use step"; let running: readonly RunningAgentHandle[]; - let workflowToolRuns: readonly WorkflowToolRunRecord[]; + let workflowToolRuns: readonly BlockingWorkflowToolRun[]; try { const session = readDurableSession(input.sessionState); - workflowToolRuns = getWorkflowToolRuns(session.state); - const workflowOwnerIds = new Set(workflowToolRuns.map((run) => run.runId)); + workflowToolRuns = getBlockingWorkflowToolRuns( + session.state, + getPendingCoordinationBatch(session.state)?.event.turnId ?? + input.sessionState.emissionState.turnId, + ); + const workflowOwnerIds = new Set(workflowToolRuns.map((run) => run.address.runId)); running = (getAgentHandleStore(session.state)?.handles ?? []).filter( (handle): handle is RunningAgentHandle => handle.phase === "running" || @@ -54,13 +63,6 @@ export async function cancelDescendantTurnsStep(input: { return; } - await Promise.all( - workflowToolRuns.map((record) => - cancelWorkflowToolRun(record, "The turn that called the tool was cancelled."), - ), - ); - if (running.length === 0) return; - let remoteContext: | Promise<{ readonly ctx: ContextContainer; @@ -73,13 +75,16 @@ export async function cancelDescendantTurnsStep(input: { registry: ctx.require(BundleKey).subagentRegistry.subagentsByNodeId, }))); - await Promise.all( - running.map((handle) => + await Promise.all([ + ...workflowToolRuns.map((record) => + cancelWorkflowToolRun(record.address, "The turn that called the tool was cancelled."), + ), + ...running.map((handle) => handle.address.kind === "agent/remote" ? cancelRemoteDescendant({ handle, remoteContext: getRemoteContext() }) : cancelLocalDescendant({ handle }), ), - ); + ]); } async function cancelLocalDescendant(input: { diff --git a/packages/eve/src/execution/cancel-indexed-session-tasks-step.test.ts b/packages/eve/src/execution/cancel-indexed-session-tasks-step.test.ts index e7f4fb418..0e7cf13cc 100644 --- a/packages/eve/src/execution/cancel-indexed-session-tasks-step.test.ts +++ b/packages/eve/src/execution/cancel-indexed-session-tasks-step.test.ts @@ -2,7 +2,11 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import type { DurableSessionState } from "#execution/durable-session-store.js"; import { cancelAllIndexedSessionTasksStep } from "#execution/cancel-indexed-session-tasks-step.js"; -import { SESSION_TASKS_STATE_KEY, type SessionTaskIndexEntry } from "#tasks/session-index.js"; +import { + getBackgroundWorkflowToolRuns, + readWorkflowTaskView, + type BackgroundWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; const { cancelOwnedTaskMock, deserializeContextMock, hydrateDurableSessionMock } = vi.hoisted( () => ({ @@ -22,7 +26,9 @@ vi.mock("#execution/tasks/parent/dispatch.js", () => ({ cancelOwnedTask: cancelO describe("cancelAllIndexedSessionTasksStep", () => { beforeEach(() => { vi.clearAllMocks(); - cancelOwnedTaskMock.mockResolvedValue(undefined); + cancelOwnedTaskMock.mockImplementation( + async ({ entry }: { entry: BackgroundWorkflowToolRun }) => cancelledView(entry), + ); deserializeContextMock.mockResolvedValue({ require: vi.fn(() => "bundle") }); hydrateDurableSessionMock.mockReturnValue("runtime-session"); }); @@ -31,11 +37,17 @@ describe("cancelAllIndexedSessionTasksStep", () => { const task1 = indexedTask("task-1"); const task2 = indexedTask("task-2"); - await cancelAllIndexedSessionTasksStep({ + const result = await cancelAllIndexedSessionTasksStep({ serializedContext: { context: "latest" }, sessionState: makeSessionState([task1, task2]), }); + expect(result.sessionState).toBeDefined(); + expect( + getBackgroundWorkflowToolRuns(result.sessionState?.snapshot.session.state).map((entry) => + readWorkflowTaskView(entry.task), + ), + ).toEqual([cancelledView(task1), cancelledView(task2)]); expect(cancelOwnedTaskMock).toHaveBeenCalledTimes(2); expect(cancelOwnedTaskMock).toHaveBeenNthCalledWith(1, { cancelOwnedWork: expect.any(Function), @@ -54,26 +66,29 @@ describe("cancelAllIndexedSessionTasksStep", () => { it("does not require runtime context when no tasks are indexed", async () => { await expect( cancelAllIndexedSessionTasksStep({ sessionState: makeSessionState([]) }), - ).resolves.toBeUndefined(); + ).resolves.toEqual({ sessionState: makeSessionState([]) }); expect(deserializeContextMock).not.toHaveBeenCalled(); expect(cancelOwnedTaskMock).not.toHaveBeenCalled(); }); }); -function indexedTask(taskId: string): SessionTaskIndexEntry { +function indexedTask(taskId: string): BackgroundWorkflowToolRun { return { - createdByTurnId: "turn_0", - dispatchContext: { auth: { current: null, initiator: null } }, - executor: { data: {}, kind: "workflow" }, - metadata: { kind: "tool", name: "research" }, - taskId, - taskInboxToken: `${taskId}-inbox`, - taskRunId: `${taskId}-run`, + callId: taskId, + toolName: "research", + lifetime: "session" as const, + origin: { turnId: "turn_0", stepIndex: 0 }, + address: { runId: `${taskId}-run`, hookToken: `${taskId}-inbox` }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "research" }, + taskId, + }, }; } -function makeSessionState(tasks: readonly SessionTaskIndexEntry[]): DurableSessionState { +function makeSessionState(tasks: readonly BackgroundWorkflowToolRun[]): DurableSessionState { return { continuationToken: "http:test", emissionState: { sequence: 0, sessionStarted: false, stepIndex: 0, turnId: "" }, @@ -85,9 +100,13 @@ function makeSessionState(tasks: readonly SessionTaskIndexEntry[]): DurableSessi continuationToken: "http:test", history: [], sessionId: "parent-session", - state: { [SESSION_TASKS_STATE_KEY]: { tasks, version: 2 } }, + state: { "eve.workflowTool": { version: 3, runs: tasks } }, }, }, version: 1, }; } + +function cancelledView(entry: BackgroundWorkflowToolRun) { + return { taskId: entry.task.taskId, metadata: entry.task.metadata, status: "cancelled" as const }; +} diff --git a/packages/eve/src/execution/cancel-indexed-session-tasks-step.ts b/packages/eve/src/execution/cancel-indexed-session-tasks-step.ts index fa5b492e7..ea4a7a7bc 100644 --- a/packages/eve/src/execution/cancel-indexed-session-tasks-step.ts +++ b/packages/eve/src/execution/cancel-indexed-session-tasks-step.ts @@ -1,3 +1,6 @@ +import { recordTerminalTaskViewsStep } from "#execution/tasks/parent/hitl-proxy-steps.js"; +import type { SessionStateTransition } from "#execution/session/state-cursor.js"; +import type { TaskView } from "#tasks/types.js"; import { deserializeContext } from "#context/serialize.js"; import { readDurableSession, type DurableSessionState } from "#execution/durable-session-store.js"; import { resolveEffectiveAgentRuntime } from "#execution/effective-agent-config.js"; @@ -6,7 +9,7 @@ import { cancelOwnedTask } from "#execution/tasks/parent/dispatch.js"; import { cancelBackgroundAgentTask } from "#execution/tools/subagent/task-cancel.js"; import { createLogger, logError } from "#internal/logging.js"; import { BundleKey } from "#runtime/sessions/runtime-context-keys.js"; -import { getSessionTaskIndex } from "#tasks/session-index.js"; +import { getBackgroundWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; const log = createLogger("execution.cancel-indexed-session-tasks"); @@ -14,7 +17,7 @@ const log = createLogger("execution.cancel-indexed-session-tasks"); export async function cancelAllIndexedSessionTasksStep(input: { readonly serializedContext?: Record; readonly sessionState: DurableSessionState; -}): Promise { +}): Promise { "use step"; let durable; @@ -24,19 +27,19 @@ export async function cancelAllIndexedSessionTasksStep(input: { logError(log, "failed to read the session for indexed task cancellation", error, { parentSessionId: input.sessionState.sessionId, }); - return; + return { sessionState: input.sessionState }; } let entries; try { - entries = getSessionTaskIndex(durable.state); + entries = getBackgroundWorkflowToolRuns(durable.state); } catch (error) { logError(log, "failed to read the task index", error, { parentSessionId: durable.sessionId, }); - return; + return { sessionState: input.sessionState }; } - if (entries.length === 0) return; + if (entries.length === 0) return { sessionState: input.sessionState }; if (input.serializedContext === undefined) { throw new Error("Indexed task cancellation requires serialized runtime context."); } @@ -49,19 +52,28 @@ export async function cancelAllIndexedSessionTasksStep(input: { turnAgent: effectiveAgent.turnAgent, }); + const views: TaskView[] = []; for (const entry of entries) { try { - await cancelOwnedTask({ + const view = await cancelOwnedTask({ cancelOwnedWork: cancelBackgroundAgentTask, entry, serializedContext: input.serializedContext, session, }); + views.push(view); } catch (error) { logError(log, "failed to cancel indexed task", error, { parentSessionId: durable.sessionId, - taskId: entry.taskId, + taskId: entry.task.taskId, }); } } + // Session finalization closes the inbox before cancellation, so it cannot + // rely on child notifications to record outcomes or settle activity. + return await recordTerminalTaskViewsStep({ + serializedContext: input.serializedContext, + sessionState: input.sessionState, + views, + }); } diff --git a/packages/eve/src/execution/coordination-dispatch-shared.ts b/packages/eve/src/execution/coordination-dispatch-shared.ts index cdc2141ae..e28c8b4a5 100644 --- a/packages/eve/src/execution/coordination-dispatch-shared.ts +++ b/packages/eve/src/execution/coordination-dispatch-shared.ts @@ -65,15 +65,10 @@ export interface CoordinationDispatchInput { readonly sessionState: DurableSessionState; } -/** Owner-side results plus any task-control work that still needs acknowledgement. */ +/** Owner-side results and the updated session. */ export interface CoordinationDispatchResult { readonly results: readonly RuntimeActionResult[]; readonly sessionState: DurableSessionState; - readonly pendingTasks: readonly { - readonly taskInboxToken: string; - readonly taskId: string; - readonly taskRunId: string; - }[]; } /** Everything preflight produces before either step's dispatch loop runs. */ diff --git a/packages/eve/src/execution/coordination-dispatch-step.ts b/packages/eve/src/execution/coordination-dispatch-step.ts index 87abc89df..18af7349a 100644 --- a/packages/eve/src/execution/coordination-dispatch-step.ts +++ b/packages/eve/src/execution/coordination-dispatch-step.ts @@ -7,7 +7,6 @@ import { } from "#execution/coordination-dispatch-shared.js"; import { createDurableSessionState } from "#execution/durable-session-store.js"; import { executeTaskControlAction } from "#execution/tasks/parent/dispatch.js"; -import type { BackgroundTask } from "#execution/tasks/parent/delegate.js"; import { cancelBackgroundAgentTask } from "#execution/tools/subagent/task-cancel.js"; import { startWorkflowTask } from "#execution/tools/workflow/start.js"; import type { RuntimeActionResult } from "#shared/action-types.js"; @@ -29,14 +28,12 @@ export async function dispatchCoordinationStep( return { results: [], sessionState: input.sessionState, - pendingTasks: [], }; } const { batch, session } = prepared; let nextSession = session; const results: RuntimeActionResult[] = []; - const pendingTasks: BackgroundTask[] = []; for (const entry of prepared.plan) { if (entry.kind === "workflow-task") { @@ -62,7 +59,6 @@ export async function dispatchCoordinationStep( session: nextSession, }); nextSession = control.session; - if (control.pendingTask !== undefined) pendingTasks.push(control.pendingTask); results.push(control.result); } } @@ -73,6 +69,5 @@ export async function dispatchCoordinationStep( nextSession === session ? prepared.sessionState : createDurableSessionState({ session: nextSession }), - pendingTasks, }; } diff --git a/packages/eve/src/execution/legacy-session/interrupt-step.test.ts b/packages/eve/src/execution/legacy-session/interrupt-step.test.ts index 389fdcb8d..af0cccbe6 100644 --- a/packages/eve/src/execution/legacy-session/interrupt-step.test.ts +++ b/packages/eve/src/execution/legacy-session/interrupt-step.test.ts @@ -23,10 +23,10 @@ function fixture(turnId = ""): PreparedLegacySession { history: [], agent: { system: "old" }, state: { - "eve.runtime.workflowToolRuns": [ - { runId: "tool-run", callId: "call", toolName: "tool", hookToken: "tool-hook" }, - ], "eve.harness.emission": { sessionStarted: true, turnId, sequence: 4, stepIndex: 2 }, + "eve.runtime.workflowToolRuns": [ + { callId: "call", toolName: "tool", runId: "tool-run", hookToken: "tool-hook" }, + ], }, }; return { @@ -68,6 +68,19 @@ describe("legacy pending work", () => { ).toBe(false); expect(mocks.settle).not.toHaveBeenCalled(); }); + it("discovers both pre-registry formats during conversation import", async () => { + const prepared = fixture(); + const originalSession = { + ...prepared.originalSession, + state: { + "eve.runtime.workflowToolRuns": [{ runId: "waiting-old" }], + "eve.tasks": { version: 2, tasks: [{ taskRunId: "task-old" }] }, + }, + }; + await interruptLegacySessionStep({ ...prepared, originalSession }); + expect(mocks.cancel.mock.calls.map((call) => call[1])).toEqual(["waiting-old", "task-old"]); + }); + it("settles an open turn once after stopping its work", async () => { const prepared = fixture("turn_4"); mocks.settle.mockResolvedValue({ sessionState: prepared.sessionState, serializedContext: {} }); diff --git a/packages/eve/src/execution/legacy-session/interrupt-step.ts b/packages/eve/src/execution/legacy-session/interrupt-step.ts index 6d136bf9f..89cfa091f 100644 --- a/packages/eve/src/execution/legacy-session/interrupt-step.ts +++ b/packages/eve/src/execution/legacy-session/interrupt-step.ts @@ -7,7 +7,6 @@ import { isObject } from "#shared/guards.js"; import { createLogger, logError } from "#internal/logging.js"; import { walkCauseChain } from "#shared/errors.js"; import { cancelRun, getWorld } from "#internal/workflow/runtime.js"; -import { getWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; import { terminateChildSessionsStep } from "#execution/terminate-child-sessions-step.js"; import { settleCancelledTurnStep } from "#execution/settle-cancelled-turn-step.js"; import type { PreparedLegacySession } from "./prepare-step.js"; @@ -38,7 +37,14 @@ export async function interruptLegacySessionStep(prepared: PreparedLegacySession } const world = await getWorld(); const state = prepared.originalSession.state; - const runIds = new Set(getWorkflowToolRuns(state).map((run) => run.runId)); + const runIds = new Set(); + // Import cancels discoverable work even when the old registry cannot pass current validation. + const waitingRuns = state?.["eve.runtime.workflowToolRuns"]; + if (Array.isArray(waitingRuns)) { + for (const entry of waitingRuns) { + if (isObject(entry) && typeof entry.runId === "string") runIds.add(entry.runId); + } + } const handles = state?.["eve.agent.handles"]; if (isObject(handles) && Array.isArray(handles.handles)) { for (const handle of handles.handles) { diff --git a/packages/eve/src/execution/node-step.test.ts b/packages/eve/src/execution/node-step.test.ts index e82a06e2a..955a7cdb3 100644 --- a/packages/eve/src/execution/node-step.test.ts +++ b/packages/eve/src/execution/node-step.test.ts @@ -286,7 +286,7 @@ describe("createNodeHarnessTools", () => { expect(agentTool?.availableInSubagents).toBeUndefined(); expect(agentTool?.execution).toBeUndefined(); - expect(agentTool?.resultKind).toBeUndefined(); + expect(agentTool).not.toHaveProperty("resultKind"); expect(agentTool?.rootOnly).toBeUndefined(); expect(agentTool?.workflowId).toBeUndefined(); }); @@ -333,7 +333,7 @@ describe("createNodeHarnessTools", () => { expect(tools.get(name)?.execution).toBe("background"); expect(tools.get(name)?.execute).toBeDefined(); expect(tools.get(name)?.runtimeAction).toBeUndefined(); - expect(tools.get(name)?.resultKind).toBe("subagent"); + expect(tools.get(name)?.nodeId).toEqual(expect.any(String)); expect(tools.get(name)?.workflowId).toBe("workflow//eve//subagentToolExecuteWorkflow"); } }); diff --git a/packages/eve/src/execution/node-step.ts b/packages/eve/src/execution/node-step.ts index 6b9550ed7..db6064b83 100644 --- a/packages/eve/src/execution/node-step.ts +++ b/packages/eve/src/execution/node-step.ts @@ -227,7 +227,6 @@ function resolveHarnessToolDefinition(input: { rootOnly: input.tool.rootOnly, }), nodeId: input.tool.task.nodeId, - resultKind: input.tool.task.resultKind, workflowId: input.tool.task.workflowId, }); } @@ -324,10 +323,6 @@ function resolveAuthoredExecute(input: { if (rawExecute === undefined) { return undefined; } - const authored = rawExecute as ( - toolInput: unknown, - ctx: unknown, - task?: Parameters>[2], - ) => unknown; + const authored = rawExecute as (toolInput: unknown, ctx: unknown) => unknown; return createToolExecuteWithAuth({ execute: authored, scope }); } diff --git a/packages/eve/src/execution/proxied-deliver-step.ts b/packages/eve/src/execution/proxied-deliver-step.ts index 190cc8c42..882b093f5 100644 --- a/packages/eve/src/execution/proxied-deliver-step.ts +++ b/packages/eve/src/execution/proxied-deliver-step.ts @@ -12,7 +12,7 @@ import { resumeSessionInbox } from "#execution/session-inbox/resume.js"; import { resumeWorkflowToolRunAnswers } from "#execution/tools/workflow/answer.js"; import type { AnswerHookRoute } from "#harness/proxy-input-requests.js"; import type { InputResponse } from "#shared/input.js"; -import { findSessionTaskEntry } from "#tasks/session-index.js"; +import { findBackgroundWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import { createTaskInputRequestId, retireProxyInputRequests, @@ -61,7 +61,7 @@ export async function routeProxiedDeliverStep(input: { const routed = routeDeliverPayload({ allowRoute: (_requestId, route) => route.taskId === undefined || - findSessionTaskEntry(durableSession.state, route.taskId) !== undefined, + findBackgroundWorkflowToolRun(durableSession.state, route.taskId) !== undefined, payload, state: durableSession.state, }); @@ -108,13 +108,13 @@ export async function routeProxiedDeliverStep(input: { // durable decision, so its view cannot claim the child resumed first. const taskId = child.taskId; if (taskId !== undefined) { - const entry = findSessionTaskEntry(durableSession.state, taskId); + const entry = findBackgroundWorkflowToolRun(durableSession.state, taskId); if (entry === undefined) { mergeStrandedResponses(parentPayloads, child, taskId); continue; } const delivery = await sendTaskInboundPayload({ - taskInboxToken: entry.taskInboxToken, + taskInboxToken: entry.address.hookToken, payload: { auth: sourceDelivery.auth, childContinuationToken: child.childContinuationToken, diff --git a/packages/eve/src/execution/route-child-delivery.test.ts b/packages/eve/src/execution/route-child-delivery.test.ts index d2721188a..d0759b85b 100644 --- a/packages/eve/src/execution/route-child-delivery.test.ts +++ b/packages/eve/src/execution/route-child-delivery.test.ts @@ -104,6 +104,89 @@ describe("task HITL delivery routing", () => { ); }); + it("ignores stale requests coalesced with their task's terminal outcome", async () => { + vi.mocked(recordTerminalTaskViewsStep).mockResolvedValue({ + subagentCompletions: [], + views: [], + serializedContext: {}, + sessionState: state(false), + }); + await routeDeliverToChildren({ + delivery: { + kind: "deliver", + payloads: [ + { + task: { + inputRequests: [taskRequest], + agentRequests: [ + { + replyTo: "agent-reply", + taskId: "task-1", + request: { + kind: "agent-invoke", + invocationId: "late-spawn", + input: { target: "research", message: "Find it" }, + }, + }, + ], + views: [ + { + taskId: "task-1", + metadata: { kind: "tool", name: "export" }, + status: "cancelled", + }, + ], + }, + }, + ], + }, + sessionWritable: new WritableStream(), + serializedContext: {}, + sessionState: state(false), + }); + expect(recordTaskInputRequestStep).not.toHaveBeenCalled(); + expect(emitRecordedTaskInputRequestStep).not.toHaveBeenCalled(); + expect(dispatchTaskAgentInvocationStep).not.toHaveBeenCalled(); + expect(recordTerminalTaskViewsStep).toHaveBeenCalledOnce(); + }); + + it("uses the parent's cancelled outcome when a late child reports success", async () => { + const cancelled = { + taskId: "task-1", + metadata: { kind: "tool", name: "export" }, + status: "cancelled" as const, + }; + vi.mocked(recordTerminalTaskViewsStep).mockResolvedValue({ + subagentCompletions: [], + views: [cancelled], + serializedContext: {}, + sessionState: state(false), + }); + const result = await routeDeliverToChildren({ + delivery: { + kind: "deliver", + taskDeliveryId: "task-1:ready:completed", + payloads: [ + { + message: "Success!", + task: { + views: [ + { ...cancelled, status: "completed", lastOutput: { type: "result", data: "late" } }, + ], + }, + }, + ], + }, + sessionWritable: new WritableStream(), + serializedContext: {}, + sessionState: state(false), + }); + expect(result).toMatchObject({ + kind: "continue", + remainder: { payloads: [{ message: "Background task task-1 (export) is cancelled." }] }, + }); + }); + it("adopts instrumentation context returned with terminal task views", async () => { const recordedState = state(false); const view = { @@ -113,6 +196,8 @@ describe("task HITL delivery routing", () => { taskId: "task-1", }; vi.mocked(recordTerminalTaskViewsStep).mockResolvedValue({ + subagentCompletions: [], + views: [view], serializedContext: { trace: "settled" }, sessionState: recordedState, }); diff --git a/packages/eve/src/execution/route-child-delivery.ts b/packages/eve/src/execution/route-child-delivery.ts index 297f1b753..b88de482c 100644 --- a/packages/eve/src/execution/route-child-delivery.ts +++ b/packages/eve/src/execution/route-child-delivery.ts @@ -1,3 +1,6 @@ +import { emitSubagentEventStep } from "#execution/tools/subagent/emit-event-step.js"; +import { formatTaskNotification } from "#tasks/notification.js"; +import type { TaskView } from "#tasks/types.js"; import type { DeliverHookPayload, DeliverPayload } from "#channel/types.js"; import { coalesceDeliverPayloads } from "#execution/deliver-payloads.js"; import type { DurableSessionState } from "#execution/durable-session-store.js"; @@ -35,8 +38,13 @@ export async function routeDeliverToChildren(input: { const payload = coalesceDeliverPayloads(input.delivery.payloads); let serializedContext = input.serializedContext; let sessionState = input.sessionState; + const recordedTaskViews = new Map(); + // Coalescing can put an old request beside its outcome. Do not start new work + // or display new questions for a task settling in this same delivery. + const settlingTaskIds = new Set((payload.task?.views ?? []).map((view) => view.taskId)); for (const request of payload.task?.inputRequests ?? []) { + if (settlingTaskIds.has(request.taskId)) continue; const recorded = await recordTaskInputRequestStep({ request, sessionState, @@ -54,6 +62,7 @@ export async function routeDeliverToChildren(input: { } for (const request of payload.task?.agentRequests ?? []) { + if (request.request.kind === "agent-invoke" && settlingTaskIds.has(request.taskId)) continue; const applied = await applyTaskAgentRequest( { ...request, ownerId: request.taskId }, { @@ -69,6 +78,11 @@ export async function routeDeliverToChildren(input: { // Authorization is display-only: the callback completes against the child, // so the parent re-emits the event without recording a proxy input request. for (const delivery of payload.task?.authorizationEvents ?? []) { + if ( + delivery.hookPayload.event.type === "authorization.required" && + settlingTaskIds.has(delivery.taskId) + ) + continue; const accepted = await acceptTaskAuthorizationEventStep({ delivery, sessionState }); if (!accepted) continue; const emitted = await runProxySubagentEventStep({ @@ -92,6 +106,17 @@ export async function routeDeliverToChildren(input: { }); serializedContext = recorded.serializedContext; sessionState = recorded.sessionState; + for (const view of recorded.views) recordedTaskViews.set(view.taskId, view); + // Publish after the durable write; replay retains these events, while duplicate deliveries return none. + for (const event of recorded.subagentCompletions) { + const emitted = await emitSubagentEventStep({ + event, + sessionWritable: input.sessionWritable, + serializedContext, + sessionState, + }); + serializedContext = emitted.serializedContext; + } } const ordinaryPayloads: DeliverPayload[] = []; @@ -99,6 +124,14 @@ export async function routeDeliverToChildren(input: { for (const [sourcePayloadIndex, sourcePayload] of input.delivery.payloads.entries()) { const ordinaryPayload = { ...sourcePayload }; delete ordinaryPayload.task; + if (ordinaryPayload.message !== undefined && sourcePayload.task?.views !== undefined) { + const notifications = sourcePayload.task.views.flatMap(({ taskId }) => { + const view = recordedTaskViews.get(taskId); + return view === undefined ? [] : [formatTaskNotification(view)]; + }); + if (notifications.length === 0) delete ordinaryPayload.message; + else ordinaryPayload.message = notifications.join("\n\n"); + } if (Object.keys(ordinaryPayload).length === 0) continue; const payloadIndex = ordinaryPayloads.length; ordinaryPayloads.push(ordinaryPayload); diff --git a/packages/eve/src/execution/session-workflow-tool-run.test.ts b/packages/eve/src/execution/session-workflow-tool-run.test.ts new file mode 100644 index 000000000..3b776ba78 --- /dev/null +++ b/packages/eve/src/execution/session-workflow-tool-run.test.ts @@ -0,0 +1,92 @@ +import { beforeEach, expect, it, vi } from "vitest"; + +import { handleWorkflowToolRunMessage } from "#execution/session-workflow-tool-run.js"; +import { applyTaskAgentRequest } from "#execution/tools/subagent/task-agent-requests.js"; +import { cancelAgentInvocationOwnerStep } from "#execution/tools/subagent/task-cancel.js"; +import { releaseAgentInvocationOwnerStep } from "#execution/tools/subagent/invoke-step.js"; +import { registerWorkflowToolRun } from "#harness/workflow-tool-runs.js"; +import { createTestSessionState } from "#internal/testing/session-state.js"; +import { SessionStateCursor } from "#execution/session/state-cursor.js"; + +vi.mock("#execution/tools/subagent/task-agent-requests.js", () => ({ + applyTaskAgentRequest: vi.fn(), +})); +vi.mock("#execution/tools/subagent/task-cancel.js", () => ({ + cancelAgentInvocationOwnerStep: vi.fn(), +})); +vi.mock("#execution/tools/subagent/invoke-step.js", () => ({ + releaseAgentInvocationOwnerStep: vi.fn(), +})); + +beforeEach(() => vi.resetAllMocks()); + +it("settles the agent request once and treats workflow completion as an ordinary tool result", async () => { + const sessionState = createTestSessionState(); + const session = registerWorkflowToolRun(sessionState.snapshot.session, { + callId: "call", + toolName: "agent", + lifetime: "turn", + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "run", hookToken: "control" }, + }); + const state = { ...sessionState, snapshot: { session } }; + const cursor = new SessionStateCursor({ + sessionState: state, + serializedContext: {}, + sessionWritable: new WritableStream(), + inbox: { claimSessionHooks: vi.fn() }, + }); + const from = { + callId: "call", + execution: "blocking" as const, + input: {}, + runId: "run", + sequence: 0, + stepIndex: 0, + toolName: "agent", + turnId: "turn", + }; + const result = { + callId: "call", + kind: "subagent-result" as const, + origin: "child" as const, + subagentName: "agent", + output: "done", + outcome: { + kind: "parked" as const, + result: { kind: "succeeded" as const, output: "done" }, + usageDelta: { inputTokens: 2, outputTokens: 3, cacheReadTokens: 0, cacheWriteTokens: 0 }, + }, + }; + vi.mocked(applyTaskAgentRequest).mockResolvedValue({ + serializedContext: {}, + sessionState: state, + }); + vi.mocked(releaseAgentInvocationOwnerStep).mockResolvedValue({ sessionState: state }); + + await handleWorkflowToolRunMessage({ + callbackMetadataUrl: "https://parent.example", + cursor, + message: { + kind: "request", + from, + replyTo: "reply", + request: { kind: "agent-settled", result }, + }, + }); + const outcome = await handleWorkflowToolRunMessage({ + callbackMetadataUrl: "https://parent.example", + cursor, + message: { kind: "outcome", from, result: { status: "completed", output: "done" } }, + }); + + expect(applyTaskAgentRequest).toHaveBeenCalledTimes(1); + expect(outcome).toEqual({ + kind: "tool-result", + callId: "call", + toolName: "agent", + output: "done", + }); + expect(cancelAgentInvocationOwnerStep).toHaveBeenCalledOnce(); + expect(releaseAgentInvocationOwnerStep).toHaveBeenCalledOnce(); +}); diff --git a/packages/eve/src/execution/session-workflow-tool-run.ts b/packages/eve/src/execution/session-workflow-tool-run.ts index 1f68dd5b1..5a6368896 100644 --- a/packages/eve/src/execution/session-workflow-tool-run.ts +++ b/packages/eve/src/execution/session-workflow-tool-run.ts @@ -1,3 +1,4 @@ +import { deliverWorkflowAuthorization } from "#execution/tools/workflow/owner.js"; import { emitWorkflowToolRunReportStep } from "#execution/tools/workflow/emit-workflow-tool-run-report-step.js"; import type { WorkflowToolRunMessage, @@ -11,21 +12,15 @@ import { cancelAgentInvocationOwnerStep } from "#execution/tools/subagent/task-c import { releaseAgentInvocationOwnerStep } from "#execution/tools/subagent/invoke-step.js"; import { resumeHookStep } from "#execution/tools/workflow/resume-hook-step.js"; import { - workflowToolRunOutcomeToSubagentResult, workflowToolRunOutcomeToToolResult, workflowToolRunRequestToInputRequestPayload, } from "#execution/tools/workflow/owner-inbox.js"; import { - findWorkflowToolRun, - isInboxSubagentResultFromRecordedWorkflowToolRun, + findBlockingWorkflowToolRun, isInboxToolResultFromRecordedWorkflowToolRun, } from "#harness/workflow-tool-runs.js"; import { runProxySubagentEventStep } from "#subagents/event-proxy-step.js"; -import type { - RuntimeActionResult, - RuntimeSubagentResult, - RuntimeToolResultActionResult, -} from "#shared/action-types.js"; +import type { RuntimeActionResult } from "#shared/action-types.js"; interface HandlerInput { readonly callbackMetadataUrl: string; @@ -62,19 +57,16 @@ async function handleWorkflowToolRunOutcome( input: HandlerInput, ): Promise { const { cursor, message } = input; - const recorded = findWorkflowToolRun( + const recorded = findBlockingWorkflowToolRun( cursor.sessionState.snapshot.session.state, message.from.callId, + message.from.turnId, ); - if (recorded?.runId !== message.from.runId) return undefined; + if (recorded?.address.runId !== message.from.runId) return undefined; - const result: RuntimeSubagentResult | RuntimeToolResultActionResult = - recorded.resultKind === "subagent" - ? await settleSubagentOutcome(input) - : workflowToolRunOutcomeToToolResult(message); + const result = workflowToolRunOutcomeToToolResult(message); - // Any workflow tool run may have invoked agents through its request channel, - // so leases are released regardless of the run's result kind. + // A failed or cancelled workflow may leave an agent invocation unfinished. await cancelAgentInvocationOwnerStep({ ownerId: message.from.runId, serializedContext: cursor.serializedContext, @@ -90,34 +82,12 @@ async function handleWorkflowToolRunOutcome( sessionState: released.sessionState, }); - const sessionSnapshotState = cursor.sessionState.snapshot.session.state; - const accepted = - result.kind === "subagent-result" - ? result.callId === message.from.callId && - isInboxSubagentResultFromRecordedWorkflowToolRun(sessionSnapshotState, result) - : isInboxToolResultFromRecordedWorkflowToolRun(sessionSnapshotState, result); - return accepted ? result : undefined; -} - -async function settleSubagentOutcome( - input: HandlerInput, -): Promise { - const { cursor, message } = input; - const result = workflowToolRunOutcomeToSubagentResult(message); - if (result.origin === "child") { - await cursor.apply( - await applyTaskAgentRequest( - { - accumulateUsage: false, - ownerId: message.from.runId, - replyTo: message.from.runId, - request: { kind: "agent-settled", result }, - }, - requestContext(input), - ), - ); - } - return result; + return isInboxToolResultFromRecordedWorkflowToolRun( + cursor.sessionState.snapshot.session.state, + result, + ) + ? result + : undefined; } async function handleWorkflowToolRunRequest( @@ -125,11 +95,12 @@ async function handleWorkflowToolRunRequest( ): Promise { const { cursor, message } = input; if (message.request.kind === "agent-invoke" || message.request.kind === "agent-settled") { - const recorded = findWorkflowToolRun( + const recorded = findBlockingWorkflowToolRun( cursor.sessionState.snapshot.session.state, message.from.callId, + message.from.turnId, ); - if (recorded?.runId !== message.from.runId) { + if (recorded?.address.runId !== message.from.runId) { if (message.request.kind === "agent-invoke") { await resumeHookStep(message.replyTo, { kind: "runtime-action-result", @@ -153,7 +124,6 @@ async function handleWorkflowToolRunRequest( await cursor.apply( await applyTaskAgentRequest( { - accumulateUsage: message.from.resultKind !== "subagent", ownerId: message.from.runId, replyTo: message.replyTo, request: message.request, @@ -164,16 +134,17 @@ async function handleWorkflowToolRunRequest( return; } if (message.request.kind === "authorization-request") { - await cursor.apply( - await runProxySubagentEventStep({ - hookPayload: message.request.event, - sessionWritable: cursor.sessionWritable, - serializedContext: cursor.serializedContext, - sessionState: cursor.sessionState, - }), - ); - if (message.request.event.childSessionId === message.from.runId) - await resumeHookStep(message.replyTo, null, { ifPresent: true }); + const request = message.request; + await deliverWorkflowAuthorization({ ...message, request }, async () => { + await cursor.apply( + await runProxySubagentEventStep({ + hookPayload: request.event, + sessionWritable: cursor.sessionWritable, + serializedContext: cursor.serializedContext, + sessionState: cursor.sessionState, + }), + ); + }); return; } await cursor.apply( diff --git a/packages/eve/src/execution/session/admission.ts b/packages/eve/src/execution/session/admission.ts index b63e7387b..d9f048dd7 100644 --- a/packages/eve/src/execution/session/admission.ts +++ b/packages/eve/src/execution/session/admission.ts @@ -104,10 +104,11 @@ export async function applySessionCancellation( }, ): Promise { if (command.tasks === true) { - await cancelAllIndexedSessionTasksStep({ + const cancelled = await cancelAllIndexedSessionTasksStep({ serializedContext: input.cursor.serializedContext, sessionState: input.cursor.sessionState, }); + await input.cursor.apply(cancelled); } if (command.taskId !== undefined) input.queue.cancelTask(command.taskId); } diff --git a/packages/eve/src/execution/session/entry.integration.test.ts b/packages/eve/src/execution/session/entry.integration.test.ts index df51bda28..ee3839f4c 100644 --- a/packages/eve/src/execution/session/entry.integration.test.ts +++ b/packages/eve/src/execution/session/entry.integration.test.ts @@ -1,6 +1,7 @@ +import type { HandoffWorkflowEntryInput } from "./entry-input.js"; import type { RunCreatedEventRequest } from "@workflow/world"; import { DEFAULT_SESSION_TIMEOUT_MS } from "#execution/session/timeout.js"; -import { afterEach, describe, expect, it, vi } from "vitest"; +import { assert, afterEach, describe, expect, it, vi } from "vitest"; import { getWorld, resumeHook, start } from "#internal/workflow/runtime.js"; import { dehydrateWorkflowArguments, @@ -1473,150 +1474,171 @@ describe("workflowEntry integration", () => { }, ); - it("recovers the original owner when target nested-state validation fails", async () => { - const runtime = await createTestRuntime({ agent: { name: "handoff-validation" } }); - await runtime.run(async () => { - const anchor = await start(workflowEntry, [ - { - kind: "initial", - ownerDeploymentId: "dpl_a", - sessionTimeoutMs: false, - input: { message: "Alice opens a research session." }, - serializedContext: buildSerializedContext({ - acceptedDeploymentId: "dpl_a", - channelKind: "http", - mode: "conversation", - }), - }, - ]); - const stream = captureTurnEvents(anchor); - const world = await getWorld(); - const workflowRuntime = createWorkflowRuntime({ - compiledArtifactsSource: createBundledRuntimeCompiledArtifactsSource(), - }); - const rewritten = new Map>(); - let candidateId: string | undefined; - // Model a value readable by the source but incompatible with the target. - // Only the candidate snapshot changes; the source retains its healthy state. - const incompatibleInput = (runId: string, encoded: unknown): Promise => { - let pending = rewritten.get(runId); - if (pending === undefined) { - pending = (async () => { - const args = (await hydrateWorkflowArguments(encoded, runId, undefined)) as [ - import("./entry-input.js").HandoffWorkflowEntryInput, - ]; - expect(args[0].kind).toBe("handoff"); - candidateId = runId; - const session = args[0].checkpoint.sessionState.snapshot.session; - Object.assign(session, { - state: { - ...session.state, - "eve.tasks": { - version: 2, - tasks: [ - { - taskId: "task", - taskRunId: "run", - taskInboxToken: 42, - createdByTurnId: "turn", - metadata: { kind: "tool", name: "research" }, - terminalView: { - taskId: "task", - metadata: { kind: "tool", name: "research" }, - status: "cancelled", - }, + it.each(["nested state", "checkpoint version"] as const)( + "recovers the original owner when target rejects %s", + async (incompatibility) => { + const runtime = await createTestRuntime({ agent: { name: "handoff-validation" } }); + await runtime.run(async () => { + const anchor = await start(workflowEntry, [ + { + kind: "initial", + ownerDeploymentId: "dpl_a", + sessionTimeoutMs: false, + input: { message: "Alice opens a research session." }, + serializedContext: buildSerializedContext({ + acceptedDeploymentId: "dpl_a", + channelKind: "http", + mode: "conversation", + }), + }, + ]); + const stream = captureTurnEvents(anchor); + const world = await getWorld(); + const workflowRuntime = createWorkflowRuntime({ + compiledArtifactsSource: createBundledRuntimeCompiledArtifactsSource(), + }); + const rewritten = new Map>(); + let candidateId: string | undefined; + // Model a value readable by the source but incompatible with the target. + // Only the candidate snapshot changes; the source retains its healthy state. + const incompatibleInput = (runId: string, encoded: unknown): Promise => { + let pending = rewritten.get(runId); + if (pending === undefined) { + pending = (async () => { + const args = (await hydrateWorkflowArguments(encoded, runId, undefined)) as [ + HandoffWorkflowEntryInput, + ]; + expect(args[0].kind).toBe("handoff"); + candidateId = runId; + const session = args[0].checkpoint.sessionState.snapshot.session; + if (incompatibility === "checkpoint version") { + Object.assign(args[0].checkpoint, { version: 4 }); + } else { + Object.assign(session, { + state: { + ...session.state, + "eve.workflowTool": { + version: 3, + runs: [ + { + callId: "task", + toolName: "research", + lifetime: "session" as const, + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "run", hookToken: 42 }, + task: { + taskId: "task", + metadata: { kind: "tool", name: "research" }, + outcome: { + status: "cancelled", + }, + dispatchContext: { auth: { current: null, initiator: null } }, + }, + }, + ], }, - ], - }, - }, - }); - const operations: Promise[] = []; - const result = await dehydrateWorkflowArguments(args, runId, undefined, operations); - await Promise.all(operations); - return result; - })(); - rewritten.set(runId, pending); - } - return pending; - }; - const createEvent = world.events.create.bind(world.events); - const created = vi.spyOn(world.events, "create").mockImplementation(async (...args) => { - const [runId] = args; - const event = args[1] as (typeof args)[1] | RunCreatedEventRequest; - if (event.eventType === "run_created" && event.eventData.deploymentId === "dpl_b") { - event.eventData.input = await incompatibleInput(runId, event.eventData.input); - } - return createEvent(...args); - }); - const queue = world.queue.bind(world); - const queued = vi.spyOn(world, "queue").mockImplementation(async (...args) => { - const message = args[1] as { - runId?: string; - runInput?: { deploymentId?: string; input: unknown }; + }, + }); + } + const operations: Promise[] = []; + const result = await dehydrateWorkflowArguments(args, runId, undefined, operations); + await Promise.all(operations); + return result; + })(); + rewritten.set(runId, pending); + } + return pending; }; - if (message.runId !== undefined && message.runInput?.deploymentId === "dpl_b") { - message.runInput.input = await incompatibleInput(message.runId, message.runInput.input); + const createEvent = world.events.create.bind(world.events); + const created = vi.spyOn(world.events, "create").mockImplementation(async (...args) => { + const [runId] = args; + const event = args[1] as (typeof args)[1] | RunCreatedEventRequest; + if (event.eventType === "run_created" && event.eventData.deploymentId === "dpl_b") { + event.eventData.input = await incompatibleInput(runId, event.eventData.input); + } + return createEvent(...args); + }); + const queue = world.queue.bind(world); + const queued = vi.spyOn(world, "queue").mockImplementation(async (...args) => { + const message = args[1] as { + runId?: string; + runInput?: { deploymentId?: string; input: unknown }; + }; + if (message.runId !== undefined && message.runInput?.deploymentId === "dpl_b") { + message.runInput.input = await incompatibleInput( + message.runId, + message.runInput.input, + ); + } + return queue(...args); + }); + try { + await stream.nextTurn(); + await workflowRuntime.dispatchSession({ + command: followUp( + "dpl_b", + "Bob requests the next research step.", + "validation-trigger", + ), + sessionId: anchor.runId, + }); + expect((await stream.nextTurn()).at(-1)?.type).toBe("session.waiting"); + assert(candidateId !== undefined); + expect( + ( + await waitForCommandHookOwner( + sessionInboxHookToken(sessionCommandHookToken(anchor.runId)), + ) + ).runId, + ).toBe(anchor.runId); + expect( + created.mock.calls.some( + ([runId, event]) => runId === candidateId && event.eventType === "hook_created", + ), + ).toBe(false); + const candidateHooks = await world.hooks.list({ runId: candidateId }); + expect(candidateHooks.data).toEqual([]); + const turns = await vi.waitFor( + async () => { + const steps = await world.steps.list({ runId: anchor.runId, resolveData: "all" }); + const turns = steps.data.filter((step) => step.stepName.endsWith("//turnStep")); + expect(turns).toHaveLength(2); + // The waiting event is streamed before the step's return value is persisted. + expect(turns.every((step) => step.output !== undefined)).toBe(true); + return turns; + }, + { timeout: 5000 }, + ); + const histories = await Promise.all( + turns.map(async (step) => { + const output = await hydrateStepReturnValue(step.output, anchor.runId, undefined); + return output.sessionState.snapshot.session.history as Array<{ + role: string; + content: unknown; + }>; + }), + ); + const deliveries = histories.map((history) => + history.filter( + (message) => + message.role === "user" && + JSON.stringify(message.content).includes("Bob requests the next research step."), + ), + ); + expect(deliveries.map((messages) => messages.length).sort()).toEqual([0, 1]); + } finally { + created.mockRestore(); + queued.mockRestore(); + await workflowRuntime.dispatchSession({ + command: { kind: "reset", reason: "validation test" }, + sessionId: anchor.runId, + }); + await anchor.returnValue; + stream.dispose(); } - return queue(...args); }); - try { - await stream.nextTurn(); - await workflowRuntime.dispatchSession({ - command: followUp( - "dpl_b", - "Bob requests the next research step.", - "validation-trigger", - ), - sessionId: anchor.runId, - }); - expect((await stream.nextTurn()).at(-1)?.type).toBe("session.waiting"); - expect(candidateId).toBeDefined(); - expect( - ( - await waitForCommandHookOwner( - sessionInboxHookToken(sessionCommandHookToken(anchor.runId)), - ) - ).runId, - ).toBe(anchor.runId); - expect( - created.mock.calls.some( - ([runId, event]) => runId === candidateId && event.eventType === "hook_created", - ), - ).toBe(false); - const candidateHooks = await world.hooks.list({ runId: candidateId! }); - expect(candidateHooks.data).toEqual([]); - const steps = await world.steps.list({ runId: anchor.runId, resolveData: "all" }); - const turns = steps.data.filter((step) => step.stepName.endsWith("//turnStep")); - expect(turns).toHaveLength(2); - const histories = await Promise.all( - turns.map(async (step) => { - const output = await hydrateStepReturnValue(step.output, anchor.runId, undefined); - return output.sessionState.snapshot.session.history as Array<{ - role: string; - content: unknown; - }>; - }), - ); - const deliveries = histories.map((history) => - history.filter( - (message) => - message.role === "user" && - JSON.stringify(message.content).includes("Bob requests the next research step."), - ), - ); - expect(deliveries.map((messages) => messages.length).sort()).toEqual([0, 1]); - } finally { - created.mockRestore(); - queued.mockRestore(); - await workflowRuntime.dispatchSession({ - command: { kind: "reset", reason: "validation test" }, - sessionId: anchor.runId, - }); - await anchor.returnValue; - stream.dispose(); - } - }); - }); + }, + ); it("retains a message accepted just before durable hook disposal", async () => { const runtime = await createTestRuntime({ agent: { name: "workflow-entry-handoff" } }); diff --git a/packages/eve/src/execution/session/handoff-steps.test.ts b/packages/eve/src/execution/session/handoff-steps.test.ts index b2f9fbaae..73da2cef3 100644 --- a/packages/eve/src/execution/session/handoff-steps.test.ts +++ b/packages/eve/src/execution/session/handoff-steps.test.ts @@ -37,19 +37,22 @@ describe("validateSessionCheckpointStep", () => { deserializeContextMock.mockResolvedValue({ require: vi.fn() }); readDurableSessionMock.mockReturnValue({ state: { - "eve.tasks": { - version: 2, - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - taskId: "task", - taskRunId: "run", - taskInboxToken: 42, - createdByTurnId: "turn", - metadata: { kind: "tool", name: "research" }, - terminalView: { + callId: "task", + toolName: "research", + lifetime: "session" as const, + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "run", hookToken: 42 }, + task: { taskId: "task", metadata: { kind: "tool", name: "research" }, - status: "cancelled", + outcome: { + status: "cancelled", + }, + dispatchContext: { auth: { current: null, initiator: null } }, }, }, ], @@ -57,18 +60,24 @@ describe("validateSessionCheckpointStep", () => { }, }); await expect(validateSessionCheckpointStep({ checkpoint: createCheckpoint() })).rejects.toThrow( - "Corrupt task index", + "Corrupt workflow tool run registry", ); }); - it("rejects a checkpoint written by a different contract version", async () => { - const checkpoint: SessionCheckpoint = { ...createCheckpoint(), version: 2 as never }; + it.each([4, 5, 7])( + "rejects checkpoint version %s before reading nested state", + async (version) => { + const checkpoint = createCheckpoint(); + // Simulate an incompatible checkpoint received over the wire. + Object.assign(checkpoint, { version }); - await expect(validateSessionCheckpointStep({ checkpoint })).rejects.toThrow( - /Unsupported session checkpoint version 2.*Start a new session/, - ); - expect(deserializeContextMock).not.toHaveBeenCalled(); - }); + await expect(validateSessionCheckpointStep({ checkpoint })).rejects.toThrow( + `Unsupported session checkpoint version ${version}`, + ); + expect(deserializeContextMock).not.toHaveBeenCalled(); + expect(readDurableSessionMock).not.toHaveBeenCalled(); + }, + ); it.each([undefined, -1, NaN, Infinity, "30000", true])( "rejects an invalid renewal duration (%s)", @@ -84,7 +93,7 @@ describe("validateSessionCheckpointStep", () => { function createCheckpoint(): SessionCheckpoint { return { - version: 4, + version: 6, sessionTimeoutMs: false, mode: "conversation", serializedContext: {}, diff --git a/packages/eve/src/execution/session/handoff-steps.ts b/packages/eve/src/execution/session/handoff-steps.ts index 7f1264f8e..e9531f53d 100644 --- a/packages/eve/src/execution/session/handoff-steps.ts +++ b/packages/eve/src/execution/session/handoff-steps.ts @@ -1,3 +1,4 @@ +import { getWorkflowToolRuns, readWorkflowTaskView } from "#harness/workflow-tool-runs.js"; import { deserializeContext } from "#context/serialize.js"; import { readDurableSession, type DurableSessionState } from "#execution/durable-session-store.js"; import { @@ -9,13 +10,15 @@ import { resumeHook } from "#internal/workflow/runtime.js"; import { BundleKey } from "#runtime/sessions/runtime-context-keys.js"; import { isObject } from "#shared/guards.js"; import { getAgentHandleStore } from "#subagents/handles/store.js"; -import { getSessionTaskIndex } from "#tasks/session-index.js"; /** Parses retained work with this deployment's code before deciding whether it can move. */ export function isSessionStateIdleForHandoff(sessionState: DurableSessionState): boolean { const { state } = readDurableSession(sessionState); // Parse all entries, including terminal tasks, before any busy-work shortcut. - const tasks = getSessionTaskIndex(state); + const invocations = getWorkflowToolRuns(state); + for (const entry of invocations) { + if (entry.lifetime === "session") readWorkflowTaskView(entry.task); + } const handles = getAgentHandleStore(state); // These registries are deleted when work settles. Their ordinary readers @@ -28,10 +31,8 @@ export function isSessionStateIdleForHandoff(sessionState: DurableSessionState): "eve.harness.pendingWorkflowInterrupt", ]; if (pendingKeys.some((key) => state?.[key] !== undefined)) return false; - for (const key of ["eve.runtime.pendingInputBatches", "eve.runtime.workflowToolRuns"]) { - const value = state?.[key]; - if (value !== undefined && (!Array.isArray(value) || value.length > 0)) return false; - } + const batches = state?.["eve.runtime.pendingInputBatches"]; + if (batches !== undefined && (!Array.isArray(batches) || batches.length > 0)) return false; const proxyRequests = state?.["eve.runtime.proxyInputRequests"]; if ( proxyRequests !== undefined && @@ -43,7 +44,7 @@ export function isSessionStateIdleForHandoff(sessionState: DurableSessionState): handles.handles.every( (handle) => handle.phase === "parked" || handle.phase === "available", )) && - tasks.every((task) => task.terminalView !== undefined) + invocations.every((entry) => entry.lifetime === "session" && entry.task.outcome !== undefined) ); } diff --git a/packages/eve/src/execution/session/handoff.test.ts b/packages/eve/src/execution/session/handoff.test.ts index 14b5d0912..741f6a349 100644 --- a/packages/eve/src/execution/session/handoff.test.ts +++ b/packages/eve/src/execution/session/handoff.test.ts @@ -44,7 +44,7 @@ describe("SessionHandoff", () => { checkpoint: expect.objectContaining({ mode: "conversation", sessionTimeoutMs: 60_000, - version: 4, + version: 6, }), delivery: trigger.delivery, targetDeploymentId: "deployment-b", diff --git a/packages/eve/src/execution/session/handoff.ts b/packages/eve/src/execution/session/handoff.ts index 55779ef0f..1e63ebf7b 100644 --- a/packages/eve/src/execution/session/handoff.ts +++ b/packages/eve/src/execution/session/handoff.ts @@ -18,8 +18,9 @@ import type { RunMode } from "#shared/run-mode.js"; * Cross-deployment checkpoint contract. The successor may run a different eve * build than the owner that produced it; bump when any field changes shape so * an incompatible successor rejects the handoff instead of misreading state. + * The shared workflow tool run registry replaces the separate task and waiting-run records. */ -export const SESSION_CHECKPOINT_VERSION = 4; +export const SESSION_CHECKPOINT_VERSION = 6; /** Everything a successor needs to continue an idle session. Hooks are derived from the state. */ export interface SessionCheckpoint { diff --git a/packages/eve/src/execution/session/input-queue.ts b/packages/eve/src/execution/session/input-queue.ts index 7d07fe517..96cd6bb67 100644 --- a/packages/eve/src/execution/session/input-queue.ts +++ b/packages/eve/src/execution/session/input-queue.ts @@ -70,10 +70,16 @@ export class SessionInputQueue { enqueueDelivery(delivery: DeliverHookPayload): DeliveryAdmission | undefined { const deliveryId = taskDeliveryId(delivery); if (deliveryId !== undefined) { - if (this.seenTaskDeliveryIds.has(deliveryId) || this.isCancelledTaskDelivery(deliveryId)) { + const terminalId = terminalTaskId(delivery); + // Competing outcomes for one task must not produce separate cohort reports. + const deduplicationId = terminalId === undefined ? deliveryId : `${terminalId}:ready`; + if ( + this.seenTaskDeliveryIds.has(deduplicationId) || + this.isCancelledTaskDelivery(deliveryId) + ) { return undefined; } - this.seenTaskDeliveryIds.add(deliveryId); + this.seenTaskDeliveryIds.add(deduplicationId); } const admission = { delivery, sequence: this.nextSequence++ }; this.entries.push({ ...admission, kind: "delivery" }); @@ -121,6 +127,15 @@ export class SessionInputQueue { return entry?.delivery; } + /** Task lifecycle effects must be applied even while their cohort report is held. */ + taskDeliveries(): readonly DeliveryAdmission[] { + return this.entries.filter( + (entry): entry is QueuedDelivery => + entry.kind === "delivery" && + entry.delivery.payloads.some((payload) => payload.task !== undefined), + ); + } + replaceDelivery(sequence: number, delivery: DeliverHookPayload | undefined): void { const index = this.entries.findIndex( (entry) => entry.kind === "delivery" && entry.sequence === sequence, @@ -200,25 +215,18 @@ export class SessionInputQueue { } private nextActionableIndex(cohorts: TaskCohorts, deferDeliveries: boolean): number { - const deliveries = this.entries.filter( - (entry): entry is QueuedDelivery => entry.kind === "delivery", - ); - const completed = new Set( - deliveries.flatMap(({ delivery }) => { - const taskId = completionTaskId(delivery); - return taskId === undefined ? [] : [taskId]; - }), - ); const pendingCohorts = new Set(); - for (const [taskId, cohort] of cohorts) { - if (!cohort.settled && !completed.has(taskId) && !this.cancelledTaskIds.has(taskId)) { - pendingCohorts.add(cohort.cohortId); + for (const [taskId, cohortId] of cohorts) { + // A control step can record cancellation before its notification is admitted. + // Wait for that notification too, so it cannot trigger a second cohort report. + if (!this.seenTaskDeliveryIds.has(`${taskId}:ready`) && !this.cancelledTaskIds.has(taskId)) { + pendingCohorts.add(cohortId); } } return this.entries.findIndex((entry) => { if (entry.kind === "control") return true; if (entry.kind === "authorization" || deferDeliveries) return false; - const cohort = completionCohort(entry.delivery, cohorts); + const cohort = terminalCohort(entry.delivery, cohorts); return cohort === undefined || !pendingCohorts.has(cohort); }); } @@ -234,17 +242,17 @@ export class SessionInputQueue { if (selected.kind === "control") return { control: selected.control, kind: "control" }; return { kind: "authorization-resume", payloads: [selected.payload] }; } - const readyCohort = completionCohort(selected.delivery, cohorts); + const readyCohort = terminalCohort(selected.delivery, cohorts); if (readyCohort !== undefined) { const lastSibling = this.entries.findLastIndex( (entry) => - entry.kind === "delivery" && completionCohort(entry.delivery, cohorts) === readyCohort, + entry.kind === "delivery" && terminalCohort(entry.delivery, cohorts) === readyCohort, ); const boundary = this.entries.findIndex( (entry, position) => position > index && position < lastSibling && - (entry.kind !== "delivery" || completionCohort(entry.delivery, cohorts) === undefined), + (entry.kind !== "delivery" || terminalCohort(entry.delivery, cohorts) === undefined), ); if (boundary >= 0) return this.takeSelectionAt(boundary, cohorts, freshSequence); } @@ -252,16 +260,15 @@ export class SessionInputQueue { const first = this.entries.splice(index, 1)[0]!; if (first.kind !== "delivery") throw new Error("Selected a non-delivery entry as a turn."); const turnEntries = [first]; - const cohort = completionCohort(first.delivery, cohorts); + const cohort = terminalCohort(first.delivery, cohorts); if (cohort !== undefined) { const siblings = this.entries.filter( (entry): entry is QueuedDelivery => - entry.kind === "delivery" && completionCohort(entry.delivery, cohorts) === cohort, + entry.kind === "delivery" && terminalCohort(entry.delivery, cohorts) === cohort, ); turnEntries.push(...siblings); this.retain( - (entry) => - entry.kind !== "delivery" || completionCohort(entry.delivery, cohorts) !== cohort, + (entry) => entry.kind !== "delivery" || terminalCohort(entry.delivery, cohorts) !== cohort, ); } else { const authenticated = @@ -346,13 +353,18 @@ function isTaskDelivery( return deliveryId !== undefined && predicate(deliveryId); } -function completionCohort(delivery: DeliverHookPayload, cohorts: TaskCohorts): string | undefined { - const taskId = completionTaskId(delivery); - return taskId === undefined ? undefined : cohorts.get(taskId)?.cohortId; +function terminalCohort(delivery: DeliverHookPayload, cohorts: TaskCohorts): string | undefined { + const taskId = terminalTaskId(delivery); + return taskId === undefined ? undefined : cohorts.get(taskId); } -function completionTaskId(delivery: DeliverHookPayload): string | undefined { - const suffix = ":ready:completed"; - if (delivery.caller !== undefined || !delivery.taskDeliveryId?.endsWith(suffix)) return undefined; - return delivery.taskDeliveryId.slice(0, -suffix.length); +function terminalTaskId(delivery: DeliverHookPayload): string | undefined { + if (delivery.caller !== undefined) return undefined; + for (const status of ["completed", "failed", "cancelled"]) { + const suffix = `:ready:${status}`; + if (delivery.taskDeliveryId?.endsWith(suffix)) { + return delivery.taskDeliveryId.slice(0, -suffix.length); + } + } + return undefined; } diff --git a/packages/eve/src/execution/session/next-input.test.ts b/packages/eve/src/execution/session/next-input.test.ts index 0bea4abf3..0b3fea17c 100644 --- a/packages/eve/src/execution/session/next-input.test.ts +++ b/packages/eve/src/execution/session/next-input.test.ts @@ -1,5 +1,5 @@ import { createTestSessionState } from "#internal/testing/session-state.js"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { assert, afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { DeliverHookPayload, SessionAuthContext } from "#channel/types.js"; import { nextTurnDelivery } from "#execution/session/next-input.js"; @@ -7,7 +7,6 @@ import { SessionInputQueue } from "#execution/session/input-queue.js"; import { routeDeliverToChildren } from "#execution/route-child-delivery.js"; import type { SessionInbox, SessionInboxPayload } from "#execution/session-inbox/inbox.js"; import { SessionStateCursor } from "#execution/session/state-cursor.js"; -import { cacheTerminalTaskView } from "#tasks/session-index.js"; import type { TaskView } from "#tasks/types.js"; vi.mock("#compiled/@workflow/core/index.js", () => ({ @@ -25,6 +24,9 @@ import { cancelAllIndexedSessionTasksStep } from "#execution/cancel-indexed-sess beforeEach(() => { vi.mocked(routeDeliverToChildren).mockReset(); vi.mocked(cancelAllIndexedSessionTasksStep).mockReset(); + vi.mocked(cancelAllIndexedSessionTasksStep).mockImplementation(async ({ sessionState }) => ({ + sessionState, + })); }); interface ScriptedRead { @@ -465,6 +467,28 @@ function completion(taskId: string): DeliverHookPayload { }; } +function terminalDelivery(taskId: string, status: "failed" | "cancelled"): DeliverHookPayload { + const view: TaskView = { + taskId, + metadata: { kind: "subagent", name: "worker" }, + ...(status === "failed" + ? { status, lastOutput: { type: "error", data: "failed" } } + : { status }), + }; + return { + kind: "deliver", + taskDeliveryId: `${taskId}:ready:${status}`, + payloads: [{ message: status, task: { views: [view] } }], + }; +} + +function report(delivery: DeliverHookPayload): DeliverHookPayload { + return { + ...delivery, + payloads: delivery.payloads.map(({ task: _task, ...payload }) => payload), + }; +} + function batchingInput(count = 100, crossTurn = false) { const input = waitInput(createMockInbox([])); const taskSessionState = { @@ -476,26 +500,34 @@ function batchingInput(count = 100, crossTurn = false) { history: [], agent: { system: "" }, state: { - "eve.tasks": { - version: 2, - tasks: Array.from({ length: count }, (_, index) => ({ - taskId: `task_${index}`, - cohortId: "task_0", - taskRunId: `run-${index}`, - taskInboxToken: `inbox-${index}`, - createdByTurnId: crossTurn ? `turn-${index + 1}` : "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "subagent", name: "worker" }, - })).concat([ - { - taskId: "other-cohort", - cohortId: "other-cohort", - taskRunId: "other-run", - taskInboxToken: "other-inbox", - createdByTurnId: "turn-2", + "eve.workflowTool": { + version: 3, + runs: Array.from({ length: count }, (_, index) => ({ + callId: `task_${index}`, + toolName: "worker", + lifetime: "session" as const, + origin: { turnId: crossTurn ? `turn-${index + 1}` : "turn-1", stepIndex: 0 }, + address: { runId: `run-${index}`, hookToken: `inbox-${index}` }, + task: { + taskId: `task_${index}`, + cohortId: "task_0", dispatchContext: { auth: { current: null, initiator: null } }, metadata: { kind: "subagent", name: "worker" }, }, + })).concat([ + { + callId: "other-cohort", + toolName: "worker", + lifetime: "session" as const, + origin: { turnId: "turn-2", stepIndex: 0 }, + address: { runId: "other-run", hookToken: "other-inbox" }, + task: { + taskId: "other-cohort", + cohortId: "other-cohort", + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "subagent", name: "worker" }, + }, + }, ]), }, }, @@ -506,7 +538,7 @@ function batchingInput(count = 100, crossTurn = false) { vi.mocked(routeDeliverToChildren).mockImplementation( async ({ delivery, sessionState, serializedContext }) => ({ kind: "continue", - remainder: delivery, + remainder: report(delivery), sessionState, serializedContext, }), @@ -518,6 +550,35 @@ describe("buffered task completion batching", () => { beforeEach(() => vi.mocked(routeDeliverToChildren).mockReset()); afterEach(() => vi.mocked(routeDeliverToChildren).mockReset()); + it.each(["completed", "failed", "cancelled"] as const)( + "keeps routed %s remainders behind the cohort barrier", + async (status) => { + const input = batchingInput(2); + const original = + status === "completed" ? completion("task_0") : terminalDelivery("task_0", status); + const admission = input.queue.enqueueDelivery(original); + assert(admission !== undefined); + const routed = { + ...original, + payloads: original.payloads.map(({ task: _task, ...payload }) => payload), + }; + input.queue.replaceDelivery(admission.sequence, routed); + const last = completion("task_1"); + input.inbox = createMockInbox([ + messageRead("still working"), + { result: { done: false, value: last } }, + ]); + await expect(nextTurnDelivery(input)).resolves.toMatchObject({ + delivery: { payloads: [{ message: "still working" }] }, + }); + expect(input.queue.pendingCount).toBe(1); + await expect(nextTurnDelivery(input)).resolves.toMatchObject({ + delivery: { payloads: [...routed.payloads, ...report(last).payloads] }, + }); + expect(input.queue.pendingCount).toBe(0); + }, + ); + it("delivers 100 buffered sibling results and their metadata in one parent turn", async () => { const input = batchingInput(); const deliveries = Array.from({ length: 100 }, (_, index) => completion(`task_${index}`)); @@ -528,7 +589,7 @@ describe("buffered task completion batching", () => { kind: "turn", delivery: { taskDeliveryId: "task_0:ready:completed", - payloads: deliveries.flatMap((delivery) => delivery.payloads), + payloads: deliveries.flatMap((delivery) => report(delivery).payloads), deliveryMetadata: deliveries.map((delivery, payloadIndex) => ({ ...delivery.deliveryMetadata![0], payloadIndex, @@ -536,12 +597,10 @@ describe("buffered task completion batching", () => { }, }); expect(queue.pendingCount).toBe(0); - expect(routeDeliverToChildren).toHaveBeenCalledTimes(1); + expect(routeDeliverToChildren).toHaveBeenCalledTimes(101); }); it.each([ - ["failed", { ...completion("task_2"), taskDeliveryId: "task_2:ready:failed" }], - ["cancelled", { ...completion("task_2"), taskDeliveryId: "task_2:ready:cancelled" }], ["input request", { ...completion("task_2"), taskDeliveryId: "task_2:input:request-1" }], ["update", { ...completion("task_2"), taskDeliveryId: "task_2:update:1" }], ["other cohort", completion("other-cohort")], @@ -568,7 +627,7 @@ describe("buffered task completion batching", () => { const later = completion("task_3"); const queue = queueOf(first, second, boundary, later); const next = await nextTurnDelivery({ ...input, queue }); - expect(next).toMatchObject({ kind: "turn", delivery: boundary }); + expect(next).toMatchObject({ kind: "turn", delivery: report(boundary) }); expect(queue.pendingCount).toBe(3); }, ); @@ -592,14 +651,20 @@ describe("buffered task completion batching", () => { delivery: { payloads: [{ message: "user question" }] }, }); expect(input.queue.pendingCount).toBe(2); - expect(routeDeliverToChildren).toHaveBeenCalledTimes(1); + expect(routeDeliverToChildren).toHaveBeenCalledTimes(3); await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", - delivery: { payloads: [...first.payloads, ...second.payloads, ...last.payloads] }, + delivery: { + payloads: [ + ...report(first).payloads, + ...report(second).payloads, + ...report(last).payloads, + ], + }, }); expect(input.queue.pendingCount).toBe(0); - expect(routeDeliverToChildren).toHaveBeenCalledTimes(2); + expect(routeDeliverToChildren).toHaveBeenCalledTimes(5); }, ); @@ -611,9 +676,9 @@ describe("buffered task completion batching", () => { ); await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", - delivery: { payloads: deliveries.flatMap((delivery) => delivery.payloads) }, + delivery: { payloads: deliveries.flatMap((delivery) => report(delivery).payloads) }, }); - expect(routeDeliverToChildren).toHaveBeenCalledTimes(1); + expect(routeDeliverToChildren).toHaveBeenCalledTimes(4); }); it("routes intervening child settlement before releasing the completion cohort", async () => { @@ -661,7 +726,7 @@ describe("buffered task completion batching", () => { kind: "continue", remainder: delivery.payloads.some((payload) => payload.task?.agentRequests !== undefined) ? undefined - : delivery, + : report(delivery), sessionState, serializedContext, }), @@ -669,11 +734,13 @@ describe("buffered task completion batching", () => { await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", - delivery: { payloads: [...first.payloads, ...last.payloads] }, + delivery: { payloads: [...report(first).payloads, ...report(last).payloads] }, }); expect(vi.mocked(routeDeliverToChildren).mock.calls.map(([call]) => call.delivery)).toEqual([ + first, settlement, - expect.objectContaining({ payloads: [...first.payloads, ...last.payloads] }), + last, + expect.objectContaining({ payloads: [...report(first).payloads, ...report(last).payloads] }), ]); expect(input.queue.pendingCount).toBe(0); }); @@ -686,10 +753,13 @@ describe("buffered task completion batching", () => { for (const delivery of [first, other, last]) input.queue.enqueueDelivery(delivery); await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", - delivery: { payloads: [...first.payloads, ...last.payloads] }, + delivery: { payloads: [...report(first).payloads, ...report(last).payloads] }, }); expect(input.queue.pendingCount).toBe(1); - await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", delivery: other }); + await expect(nextTurnDelivery(input)).resolves.toMatchObject({ + kind: "turn", + delivery: report(other), + }); }); it.each(["clear", "compact", "reset", "session-timeout"] as const)( @@ -705,7 +775,7 @@ describe("buffered task completion batching", () => { kind: kind === "session-timeout" ? "expired" : kind, }); expect(input.queue.pendingCount).toBe(1); - expect(routeDeliverToChildren).not.toHaveBeenCalled(); + expect(routeDeliverToChildren).toHaveBeenCalledOnce(); }, ); @@ -714,9 +784,46 @@ describe("buffered task completion batching", () => { const first = completion("task_0"); input.queue.enqueueDelivery(first); input.inbox = createMockInbox([cancelRead({ taskId: "task_1" })]); - await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", delivery: first }); + await expect(nextTurnDelivery(input)).resolves.toMatchObject({ + kind: "turn", + delivery: report(first), + }); }); + it("waits for a recorded cancellation's notification before reporting its cohort", () => { + const queue = new SessionInputQueue(); + const cohorts = new Map([ + ["task_0", "cohort"], + ["task_1", "cohort"], + ]); + queue.enqueueDelivery(completion("task_0")); + expect(queue.takeNext(cohorts)).toBeUndefined(); + queue.enqueueDelivery(terminalDelivery("task_1", "cancelled")); + expect(queue.takeNext(cohorts)).toMatchObject({ kind: "turn" }); + expect(queue.pendingCount).toBe(0); + expect(queue.enqueueDelivery(completion("task_1"))).toBeUndefined(); + }); + + it.each(["failed", "cancelled"] as const)( + "does not admit a second terminal notification with a different %s outcome", + (status) => { + for (const deliveries of [ + [completion("task_0"), terminalDelivery("task_0", status)], + [terminalDelivery("task_0", status), completion("task_0")], + ]) { + const queue = new SessionInputQueue(); + const [first, late] = deliveries; + if (first === undefined || late === undefined) throw new Error("Expected two deliveries."); + expect(queue.enqueueDelivery(first)).toBeDefined(); + expect(queue.enqueueDelivery(late)).toBeUndefined(); + expect(queue.pendingCount).toBe(1); + expect(queue.takeNext(new Map())).toBeDefined(); + expect(queue.enqueueDelivery(late)).toBeUndefined(); + expect(queue.pendingCount).toBe(0); + } + }, + ); + it("ignores duplicate notifications while waiting for the last sibling", async () => { const input = batchingInput(2); const first = completion("task_0"); @@ -729,57 +836,37 @@ describe("buffered task completion batching", () => { ); await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", - delivery: { payloads: [...first.payloads, ...last.payloads] }, + delivery: { payloads: [...report(first).payloads, ...report(last).payloads] }, }); }); it.each(["failed", "cancelled"] as const)( - "delivers %s immediately, then releases successful siblings from the settled cohort", + "batches %s with successful siblings from the same cohort", async (status) => { const input = batchingInput(2); const first = completion("task_0"); - const view: TaskView = { - taskId: "task_1", - metadata: { kind: "subagent", name: "worker" }, - ...(status === "failed" - ? { status, lastOutput: { type: "error", data: "failed" } } - : { status }), - }; - const terminal: DeliverHookPayload = { - kind: "deliver", - taskDeliveryId: `task_1:ready:${status}`, - payloads: [{ message: status, task: { views: [view] } }], - }; + const terminal = terminalDelivery("task_1", status); for (const delivery of [first, terminal]) input.queue.enqueueDelivery(delivery); - vi.mocked(routeDeliverToChildren).mockImplementation( - async ({ delivery, sessionState, serializedContext }) => { - const snapshot = sessionState.snapshot!; - return { - kind: "continue", - remainder: delivery, - serializedContext, - sessionState: { - ...sessionState, - snapshot: { - ...snapshot, - session: { - ...snapshot.session, - state: cacheTerminalTaskView(snapshot.session.state, view), - }, - }, - }, - }; - }, - ); await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", - delivery: terminal, + delivery: { payloads: [...report(first).payloads, ...report(terminal).payloads] }, }); - expect(input.queue.pendingCount).toBe(1); + expect(input.queue.pendingCount).toBe(0); + }, + ); + + it.each(["failed", "cancelled"] as const)( + "batches an all-%s cohort into one report", + async (status) => { + const input = batchingInput(2); + const first = terminalDelivery("task_0", status); + const last = terminalDelivery("task_1", status); + for (const delivery of [first, last]) input.queue.enqueueDelivery(delivery); await expect(nextTurnDelivery(input)).resolves.toMatchObject({ kind: "turn", - delivery: first, + delivery: { payloads: [...report(first).payloads, ...report(last).payloads] }, }); + expect(input.queue.pendingCount).toBe(0); }, ); @@ -790,6 +877,6 @@ describe("buffered task completion batching", () => { const next = await nextTurnDelivery({ ...input, queue, inbox: createMockInbox([]) }); expect(next.kind).toBe("authorization-resume"); expect(queue.pendingCount).toBe(2); - expect(routeDeliverToChildren).not.toHaveBeenCalled(); + expect(routeDeliverToChildren).toHaveBeenCalledTimes(2); }); }); diff --git a/packages/eve/src/execution/session/next-input.ts b/packages/eve/src/execution/session/next-input.ts index a804990ed..b4d2179f6 100644 --- a/packages/eve/src/execution/session/next-input.ts +++ b/packages/eve/src/execution/session/next-input.ts @@ -1,4 +1,5 @@ import type { DeliverPayload } from "#channel/types.js"; +import { routeDeliverToChildren } from "#execution/route-child-delivery.js"; import { routeSelectedDelivery } from "#execution/session/route-selected-delivery.js"; import type { SessionControl, @@ -41,6 +42,17 @@ export async function nextTurnDelivery(input: { // pumped) is the only kind that may move the session to another deployment. let freshSequence: number | undefined; while (true) { + for (const { delivery, sequence } of queue.taskDeliveries()) { + const routed = await routeDeliverToChildren({ + delivery, + sessionWritable: cursor.sessionWritable, + serializedContext: cursor.serializedContext, + sessionState: cursor.sessionState, + }); + await cursor.apply(routed); + queue.replaceDelivery(sequence, routed.kind === "cancel-turn" ? undefined : routed.remainder); + if (routed.kind === "cancel-turn") return routed; + } const selected = queue.takeNext( getSessionTaskCohorts(cursor.sessionState.snapshot.session.state), { diff --git a/packages/eve/src/execution/session/state-compatibility.test.ts b/packages/eve/src/execution/session/state-compatibility.test.ts index 29bf97d38..b7a77901b 100644 --- a/packages/eve/src/execution/session/state-compatibility.test.ts +++ b/packages/eve/src/execution/session/state-compatibility.test.ts @@ -1,11 +1,12 @@ -import { describe, expect, it } from "vitest"; +import { assert, describe, expect, it } from "vitest"; import { createTestSessionState } from "#internal/testing/session-state.js"; import { isSessionStateIdleForHandoff } from "#execution/session/handoff-steps.js"; import { - cacheTerminalTaskView, - getSessionTaskIndex, - recordSessionTask, -} from "#tasks/session-index.js"; + readWorkflowTaskView, + recordWorkflowTaskView, + getBackgroundWorkflowToolRuns, + registerWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import { parseActivityWorkIdentityV1 } from "#protocol/activity.js"; import type { HarnessSession } from "#harness/types.js"; import type { AgentHandle, AgentHandlePhase } from "#subagents/handles/store.js"; @@ -19,31 +20,30 @@ const activity = { futureActivity: { label: "Alice's research" }, }; const task = { - taskId: "task", - taskRunId: "run", - taskInboxToken: "inbox", - createdByTurnId: "turn", - metadata, - activityWorkIdentity: activity, - futureTask: { revision: 2 }, - executor: { kind: "workflow", data: {}, futureExecutor: true }, - terminalView: { + callId: "task", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn", stepIndex: 0, futureOrigin: true }, + address: { runId: "run", hookToken: "inbox", futureAddress: true }, + futureInvocation: true, + task: { taskId: "task", metadata, - status: "completed" as const, - lastOutput: { type: "result" as const, data: "done", futureOutput: true }, - usage: { - inputTokens: 1, - outputTokens: 2, - cacheReadTokens: 0, - cacheWriteTokens: 0, - futureUsage: true, + activityWorkIdentity: activity, + futureTask: { revision: 2 }, + outcome: { + status: "completed" as const, + lastOutput: { type: "result" as const, data: "done", futureOutput: true }, + usage: { + inputTokens: 1, + outputTokens: 2, + cacheReadTokens: 0, + cacheWriteTokens: 0, + futureUsage: true, + }, + futureView: true, }, - executor: { - futureExecutor: true, - binding: { kind: "workflow", data: {}, futureBinding: true }, - }, - futureView: true, + dispatchContext: { auth: { current: null, initiator: null } }, }, }; function checkpoint(state: Record) { @@ -98,64 +98,76 @@ function handle(phase: AgentHandlePhase): AgentHandle { } describe("additive durable state", () => { - it("preserves task extensions through parsing, replayed creation and terminal updates", () => { + it("preserves task extensions through parsing, replayed creation and duplicate terminal deliveries", () => { const state = { - "eve.tasks": { version: 2, tasks: [task], futureIndex: true }, authored: { opaque: true }, + "eve.workflowTool": { version: 3, runs: [task], futureIndex: true }, }; - expect(getSessionTaskIndex(restored(state))).toEqual([ - { ...task, dispatchContext: { legacy: true } }, - ]); + expect(getBackgroundWorkflowToolRuns(restored(state))).toEqual([task]); expect(parseActivityWorkIdentityV1(activity)).toEqual(activity); - const updated = recordSessionTask(session(restored(state)), { - taskId: "task", - taskRunId: "new-run", - taskInboxToken: "inbox", - createdByTurnId: "turn", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata, - activityWorkIdentity: { id: "work", kind: "task", rootSessionId: "root", rootTurnId: "turn" }, - executor: { kind: "workflow", data: {} }, + const updated = registerWorkflowToolRun(session(restored(state)), { + callId: "task", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "new-run", hookToken: "inbox" }, + task: { + taskId: "task", + dispatchContext: { auth: { current: null, initiator: null } }, + metadata, + activityWorkIdentity: { + id: "work", + kind: "task", + rootSessionId: "root", + rootTurnId: "turn", + }, + }, }); - const saved = cacheTerminalTaskView(updated.state, { + const saved = recordWorkflowTaskView(updated.state, { taskId: "task", metadata, status: "completed", lastOutput: { type: "result", data: "updated" }, usage: { inputTokens: 3, outputTokens: 4, cacheReadTokens: 0, cacheWriteTokens: 0 }, - executor: { binding: { kind: "workflow", data: {} } }, }); expect(restored(saved)).toMatchObject({ authored: { opaque: true }, - "eve.tasks": { + "eve.workflowTool": { + version: 3, futureIndex: true, - tasks: [ + runs: [ { - futureTask: { revision: 2 }, - taskRunId: "new-run", - activityWorkIdentity: activity, - executor: { futureExecutor: true }, - terminalView: { - futureView: true, - lastOutput: { data: "updated", futureOutput: true }, - usage: { inputTokens: 3, futureUsage: true }, - executor: { futureExecutor: true, binding: { futureBinding: true } }, + address: { runId: "new-run", futureAddress: true }, + origin: { futureOrigin: true }, + futureInvocation: true, + task: { + futureTask: { revision: 2 }, + activityWorkIdentity: activity, + outcome: { + futureView: true, + lastOutput: { data: "done", futureOutput: true }, + usage: { inputTokens: 1, futureUsage: true }, + }, }, }, ], }, }); - expect(isSessionStateIdleForHandoff(checkpoint(restored(saved!)))).toBe(true); - const cancelled = cacheTerminalTaskView(saved, { + assert(saved !== undefined); + expect(isSessionStateIdleForHandoff(checkpoint(restored(saved)))).toBe(true); + const cancelled = recordWorkflowTaskView(saved, { taskId: "task", metadata, status: "cancelled", }); - expect(getSessionTaskIndex(restored(cancelled))[0]?.terminalView).toMatchObject({ + expect(getBackgroundWorkflowToolRuns(restored(cancelled))[0]?.task.outcome).toMatchObject({ futureView: true, - status: "cancelled", + status: "completed", }); - expect(getSessionTaskIndex(restored(cancelled))[0]?.terminalView?.lastOutput).toBeUndefined(); + expect(cancelled).toBe(saved); + const [retained] = getBackgroundWorkflowToolRuns(restored(cancelled)); + assert(retained !== undefined); + expect(readWorkflowTaskView(retained.task)?.lastOutput?.data).toBe("done"); }); }); @@ -164,9 +176,9 @@ describe("handoff state inspection", () => { expect( isSessionStateIdleForHandoff( checkpoint({ - "eve.tasks": { version: 2, tasks: [task], futureIndex: true }, "eve.agent.handles": { handles: [], futureStore: true }, authored: { version: "anything", values: [null, false] }, + "eve.workflowTool": { version: 3, runs: [task], futureIndex: true }, }), ), ).toBe(true); @@ -203,29 +215,46 @@ describe("handoff state inspection", () => { ).toThrow("Corrupt agent handle store"); }); it("parses settled entries before checking their terminal status", () => { - const incompatible = { ...task, taskInboxToken: 42 }; + const incompatible = { ...task, address: { ...task.address, hookToken: 42 } }; expect(() => isSessionStateIdleForHandoff( - checkpoint({ "eve.tasks": { version: 2, tasks: [incompatible] } }), + checkpoint({ + "eve.workflowTool": { version: 3, runs: [incompatible] }, + }), ), - ).toThrow("Corrupt task index"); + ).toThrow("Corrupt workflow tool run registry"); }); it("does not skip task parsing when another registry is busy", () => { expect(() => isSessionStateIdleForHandoff( checkpoint({ "eve.runtime.pendingAuthorization": {}, - "eve.tasks": { version: 2, tasks: [{ ...task, taskRunId: null }] }, + "eve.workflowTool": { + version: 3, + runs: [{ ...task, address: { ...task.address, runId: null } }], + }, }), ), - ).toThrow("Corrupt task index"); + ).toThrow("Corrupt workflow tool run registry"); + }); + it("validates retained results even when other work prevents handoff", () => { + expect(() => + isSessionStateIdleForHandoff( + checkpoint({ + "eve.runtime.pendingAuthorization": {}, + "eve.workflowTool": { + version: 3, + runs: [{ ...task, task: { ...task.task, outcome: { status: "completed" } } }], + }, + }), + ), + ).toThrow("Corrupt workflow task result"); }); it.each([ ["eve.runtime.pendingAuthorization", false], ["eve.runtime.pendingInputBatch", {}], ["eve.runtime.pendingInputBatches", [null]], ["eve.runtime.pendingCoordinationBatch", {}], - ["eve.runtime.workflowToolRuns", {}], ["eve.runtime.deferredStepInput", {}], ["eve.harness.pendingWorkflowInterrupt", {}], ["eve.runtime.proxyInputRequests", { malformed: null }], diff --git a/packages/eve/src/execution/session/task-settlement.integration.test.ts b/packages/eve/src/execution/session/task-settlement.integration.test.ts new file mode 100644 index 000000000..57dc942ca --- /dev/null +++ b/packages/eve/src/execution/session/task-settlement.integration.test.ts @@ -0,0 +1,201 @@ +import { routeDeliverToChildren } from "#execution/route-child-delivery.js"; +import { emitSubagentEventStep } from "#execution/tools/subagent/emit-event-step.js"; + +vi.mock("#execution/tools/subagent/emit-event-step.js", () => ({ emitSubagentEventStep: vi.fn() })); + +import { expect, it, vi } from "vitest"; + +import type { DeliverHookPayload } from "#channel/types.js"; +import { replaceDurableSessionSnapshot } from "#execution/durable-session-store.js"; +import { SessionInputQueue } from "#execution/session/input-queue.js"; +import { nextTurnDelivery } from "#execution/session/next-input.js"; +import { SessionStateCursor } from "#execution/session/state-cursor.js"; +import { + getProxyInputRequests, + upsertProxyInputRequestState, +} from "#harness/proxy-input-requests.js"; +import { + findBackgroundWorkflowToolRun, + registerWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; +import { createTestSessionState } from "#internal/testing/session-state.js"; +import type { TaskView } from "#tasks/types.js"; + +it.each(["completed", "failed", "cancelled"] as const)( + "records a parked parent's %s task and removes its question before reporting the cohort", + async (status) => { + vi.mocked(emitSubagentEventStep).mockReset().mockResolvedValue({ serializedContext: {} }); + let state = createTestSessionState(); + let session = state.snapshot.session; + for (const taskId of ["A", "B"]) { + session = registerWorkflowToolRun(session, { + callId: taskId, + toolName: "worker", + lifetime: "session", + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: taskId, hookToken: taskId }, + task: { + taskId, + metadata: { kind: "subagent", name: "worker" }, + dispatchContext: { auth: { current: null, initiator: null } }, + }, + }); + } + session = { + ...session, + state: upsertProxyInputRequestState({ + state: session.state, + forChildContinuationToken: "answer-A", + entries: [ + [ + "question-A", + { + childContinuationToken: "answer-A", + childRequestId: "question", + kind: "question", + taskId: "A", + }, + ], + ], + }), + }; + state = replaceDurableSessionSnapshot({ session, state }); + const cursor = new SessionStateCursor({ + inbox: { claimSessionHooks: async () => {} }, + sessionWritable: new WritableStream(), + serializedContext: {}, + sessionState: state, + }); + const queue = new SessionInputQueue(); + const a: TaskView = { + taskId: "A", + metadata: { kind: "subagent", name: "worker" }, + ...(status === "completed" + ? { status, lastOutput: { type: "result", data: "done" } } + : status === "failed" + ? { status, lastOutput: { type: "error", data: "timeout" } } + : { status }), + }; + const notification = (view: TaskView): DeliverHookPayload => ({ + kind: "deliver", + taskDeliveryId: `${view.taskId}:ready:${view.status}`, + payloads: [{ message: "task outcome", task: { views: [view] } }], + }); + queue.enqueueDelivery(notification(a)); + let reads = 0; + const next = await nextTurnDelivery({ + cursor, + queue, + inbox: { + hasPending: () => false, + drain: () => [], + onInterrupt: () => () => {}, + onDelivery: () => () => {}, + restore: () => {}, + async next() { + // The owner is about to wait for B; A must already be settled without a model turn. + reads++; + const current = cursor.sessionState.snapshot.session; + expect(findBackgroundWorkflowToolRun(current.state, "A")?.task.outcome).toEqual({ + status: a.status, + lastOutput: a.lastOutput, + usage: a.usage, + }); + expect(getProxyInputRequests(current.state).size).toBe(0); + expect(cursor.sessionState.hasProxyInputRequests).toBe(false); + expect(queue.pendingCount).toBe(1); + expect(emitSubagentEventStep).toHaveBeenCalledTimes(status === "completed" ? 1 : 0); + return notification({ + taskId: "B", + metadata: { kind: "subagent", name: "worker" }, + status: "completed", + lastOutput: { type: "result", data: "done B" }, + }); + }, + }, + }); + expect(reads).toBe(1); + expect(next).toMatchObject({ + kind: "turn", + delivery: { + payloads: [ + { message: expect.stringContaining("Background task A") }, + { message: expect.stringContaining("Background task B") }, + ], + }, + }); + expect(queue.pendingCount).toBe(0); + }, +); + +it.each(["completed", "failed", "cancelled"] as const)( + "publishes a background subagent result only for a newly recorded success (%s)", + async (status) => { + vi.mocked(emitSubagentEventStep).mockReset(); + const initial = createTestSessionState(); + const metadata = { kind: "subagent", name: "researcher" }; + let sessionState = replaceDurableSessionSnapshot({ + state: initial, + session: registerWorkflowToolRun(initial.snapshot.session, { + callId: "original-call", + toolName: "researcher", + lifetime: "session", + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "run", hookToken: "hook" }, + task: { + taskId: "task", + metadata, + dispatchContext: { auth: { current: null, initiator: null } }, + }, + }), + }); + const view: TaskView = { + taskId: "task", + metadata, + ...(status === "completed" + ? { status, lastOutput: { type: "result", data: { answer: 42 } } } + : status === "failed" + ? { status, lastOutput: { type: "error", data: "failed" } } + : { status }), + }; + vi.mocked(emitSubagentEventStep).mockImplementation(async (input) => { + expect( + findBackgroundWorkflowToolRun(input.sessionState.snapshot.session.state, "task")?.task + .outcome, + ).toEqual({ status: view.status, lastOutput: view.lastOutput, usage: view.usage }); + return { serializedContext: input.serializedContext }; + }); + const deliver = async (outcome: TaskView) => { + const result = await routeDeliverToChildren({ + delivery: { kind: "deliver", payloads: [{ task: { views: [outcome] } }] }, + serializedContext: {}, + sessionState, + sessionWritable: new WritableStream(), + }); + sessionState = result.sessionState; + }; + await deliver(view); + await deliver(view); + // A late success must not replace a failed or cancelled task, nor republish a success. + await deliver({ + taskId: "task", + metadata, + status: "completed", + lastOutput: { type: "result", data: "late" }, + }); + expect(emitSubagentEventStep).toHaveBeenCalledTimes(status === "completed" ? 1 : 0); + if (status === "completed") { + expect(emitSubagentEventStep).toHaveBeenCalledWith( + expect.objectContaining({ + event: { + type: "subagent.completed", + data: { callId: "original-call", subagentName: "researcher", output: '{"answer":42}' }, + }, + }), + ); + } + expect( + findBackgroundWorkflowToolRun(sessionState.snapshot.session.state, "task")?.task.outcome, + ).toEqual({ status: view.status, lastOutput: view.lastOutput, usage: view.usage }); + }, +); diff --git a/packages/eve/src/execution/session/turn-step.test.ts b/packages/eve/src/execution/session/turn-step.test.ts index f80170f4b..e37bae705 100644 --- a/packages/eve/src/execution/session/turn-step.test.ts +++ b/packages/eve/src/execution/session/turn-step.test.ts @@ -61,7 +61,7 @@ import { defineMemory } from "#public/memory/index.js"; import { stampDurableDynamicCallback } from "#tools/durable-callbacks.js"; import { dispatchCoordinationStep } from "#execution/coordination-dispatch-step.js"; import { runProxySubagentEventStep } from "#subagents/event-proxy-step.js"; -import { readLatestTaskView, sendTaskInboundPayload } from "#execution/tasks/parent/run-parent.js"; +import { sendTaskInboundPayload } from "#execution/tasks/parent/run-parent.js"; import { recordTaskInputRequestStep } from "#execution/tasks/parent/hitl-proxy-steps.js"; import { emitTerminalSessionFailureStep } from "#execution/terminal-session-failure-step.js"; import { resolveEffectiveOutputSchema } from "#execution/effective-output-schema.js"; @@ -126,7 +126,6 @@ vi.mock("../durable-session-store.js", async (importOriginal) => { }; }); vi.mock("../tasks/parent/run-parent.js", () => ({ - readLatestTaskView: vi.fn(), sendTaskInboundPayload: vi.fn(), })); @@ -325,7 +324,6 @@ afterEach(() => { vi.unstubAllGlobals(); vi.unstubAllEnvs(); vi.restoreAllMocks(); - vi.mocked(readLatestTaskView).mockReset(); vi.mocked(sendTaskInboundPayload).mockReset(); vi.mocked(sendTaskInboundPayload).mockResolvedValue("delivered"); mockIdentityHistoryViewProjector.mockReset(); @@ -546,18 +544,22 @@ describe("routeProxiedDeliverStep", () => { options?.owned === false ? undefined : { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - taskInboxToken: "task-token", - createdByTurnId: "turn-parent", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "tool", name: "research" }, - taskId: "task-1", - taskRunId: "run-1", + callId: "task-1", + toolName: "research", + lifetime: "session" as const, + origin: { turnId: "turn-parent", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "research" }, + taskId: "task-1", + }, }, ], - version: 2, }, }, }), @@ -700,52 +702,26 @@ describe("recordTaskInputRequestStep", () => { it("records an exact route only for a current task owned by this parent", async () => { const session = createStubSession({ state: { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - taskInboxToken: "task-token", - createdByTurnId: "turn-parent", - dispatchContext: { auth: { current: null, initiator: null } }, - executor: { - data: { - address: { - continuationToken: "child-token", - kind: "agent/local", - sessionId: "child-session", - }, - identity: { id: "agent-1", name: "research", nodeId: "node-1" }, - }, - kind: "subagent", + callId: "task-1", + toolName: "research", + lifetime: "session" as const, + origin: { turnId: "turn-parent", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "research" }, + taskId: "task-1", }, - metadata: { kind: "tool", name: "research" }, - taskId: "task-1", - taskRunId: "run-1", }, ], - version: 2, }, }, }); installSessionStoreMocks([session]); - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: { kind: "tool", name: "research" }, - executor: { - binding: { - data: { - address: { - continuationToken: "child-token", - kind: "agent/local", - sessionId: "child-session", - }, - identity: { id: "agent-1", name: "research", nodeId: "node-1" }, - }, - kind: "subagent", - }, - }, - inputRequests: [taskRequest.request], - status: "input_required", - taskId: "task-1", - }); const result = await recordTaskInputRequestStep({ request: taskRequest, @@ -770,7 +746,6 @@ describe("recordTaskInputRequestStep", () => { it("rejects cross-session and stale batches without recording a route", async () => { const session = createStubSession(); installSessionStoreMocks([session, session]); - vi.mocked(readLatestTaskView).mockResolvedValue(undefined); const result = await recordTaskInputRequestStep({ request: { ...taskRequest, taskId: "foreign-task" }, @@ -1136,13 +1111,16 @@ describe("turnStep", () => { vi.mocked(getCompiledRuntimeAgentBundle).mockResolvedValue(bundle); const task = { - createdByTurnId: "turn_0", - dispatchContext: { auth: { current: null, initiator: null } }, - executor: { data: {}, kind: "workflow-tool" }, - metadata: { kind: "report", name: "daily_report" }, - taskId: "task_report", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task_report", + toolName: "daily_report", + lifetime: "session" as const, + origin: { turnId: "turn_0", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "report", name: "daily_report" }, + taskId: "task_report", + }, }; const emissionState = { sequence: 0, @@ -1153,21 +1131,19 @@ describe("turnStep", () => { const pending = createStubSession({ state: { "eve.harness.emission": emissionState, - "eve.tasks": { tasks: [task], version: 2 }, + "eve.workflowTool": { version: 3, runs: [task] }, }, }); - const terminalView = { + const outcome = { lastOutput: { data: "report complete", type: "result" as const }, - metadata: task.metadata, status: "completed" as const, - taskId: task.taskId, }; const settled = createStubSession({ state: { "eve.harness.emission": { ...emissionState, sequence: 1, turnId: "turn_1" }, - "eve.tasks": { - tasks: [{ ...task, terminalView }], - version: 2, + "eve.workflowTool": { + version: 3, + runs: [{ ...task, task: { ...task.task, outcome } }], }, }, }); @@ -2671,24 +2647,26 @@ describe("turnStep", () => { const metadata = { kind: "report-probe", name: "report_probe" } as const; const session = createStubSession({ state: { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - taskInboxToken: "task-token", - createdByTurnId: "turn-parent", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata, - taskId: "task_1", - taskRunId: "run_1", - terminalView: { - lastOutput: { data: { result: "done" }, type: "result" }, + callId: "task_1", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-parent", stepIndex: 0 }, + address: { runId: "run_1", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, metadata, - status: "completed", taskId: "task_1", + outcome: { + lastOutput: { data: { result: "done" }, type: "result" }, + status: "completed", + }, }, }, ], - version: 2, }, }, }); @@ -2780,20 +2758,22 @@ describe("turnStep", () => { stepIndex: 1, turnId: "turn_0", }, - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - createdByStepIndex: 0, - createdByTurnId: "turn_0", - dispatchContext: { auth: { current: null, initiator: null } }, - executor: { data: {}, kind: "workflow-tool" }, - metadata: { kind: "report-probe", name: "report_probe" }, - taskId: "task_1", - taskInboxToken: "task-token", - taskRunId: "run_1", + callId: "task_1", + toolName: "report_probe", + lifetime: "session" as const, + origin: { turnId: "turn_0", stepIndex: 0 }, + address: { runId: "run_1", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "report-probe", name: "report_probe" }, + taskId: "task_1", + }, }, ], - version: 2, }, }, }); diff --git a/packages/eve/src/execution/session/turn.test.ts b/packages/eve/src/execution/session/turn.test.ts index 98d41895b..253443a08 100644 --- a/packages/eve/src/execution/session/turn.test.ts +++ b/packages/eve/src/execution/session/turn.test.ts @@ -314,7 +314,6 @@ describe("SessionExecution background task checkpoints", () => { }); vi.mocked(dispatchCoordinationStep).mockResolvedValue({ results: [], - pendingTasks: [], sessionState, }); @@ -465,7 +464,6 @@ describe("SessionExecution background task checkpoints", () => { vi.mocked(dispatchCoordinationStep) .mockReset() .mockResolvedValue({ - pendingTasks: [], results: [actionResult], sessionState, }); @@ -619,7 +617,6 @@ describe("SessionExecution background task checkpoints", () => { sessionState, }); vi.mocked(dispatchCoordinationStep).mockResolvedValue({ - pendingTasks: [], results: [], sessionState, }); diff --git a/packages/eve/src/execution/session/turn.ts b/packages/eve/src/execution/session/turn.ts index c8ebaa394..8df490651 100644 --- a/packages/eve/src/execution/session/turn.ts +++ b/packages/eve/src/execution/session/turn.ts @@ -31,10 +31,7 @@ import { activeTurnId } from "#harness/active-turn-id.js"; import { coalesceDeliveries } from "#harness/messages.js"; import { TurnCancelledError } from "#harness/turn-cancellation.js"; import { decodeSessionInboxPayload } from "#execution/session-inbox/protocol.js"; -import { - isInboxSubagentResultFromRecordedWorkflowToolRun, - isInboxToolResultFromRecordedWorkflowToolRun, -} from "#harness/workflow-tool-runs.js"; +import { isInboxToolResultFromRecordedWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import { isInboxSubagentResultFromRunningHandle } from "#subagents/handles/query.js"; import { resolveRuntimeActionResultsForCallIds } from "#runtime/actions/results.js"; import type { RunMode } from "#shared/run-mode.js"; @@ -145,7 +142,6 @@ export class SessionExecution { }); const initialAcceptedAtMs = dispatchResult.results.length === 0 ? undefined : Date.now(); await cursor.apply(dispatchResult); - await acknowledgeDelegatedTasksStep({ tasks: dispatchResult.pendingTasks }); const runtimeResults = await this.waitForRuntimeActionResults({ initialAcceptedAtMs, @@ -234,9 +230,7 @@ export class SessionExecution { } if (result.kind !== "subagent-result") return false; return ( - (result.origin === "child" && - isInboxSubagentResultFromRunningHandle(snapshot, result)) || - isInboxSubagentResultFromRecordedWorkflowToolRun(snapshot, result) + result.origin === "child" && isInboxSubagentResultFromRunningHandle(snapshot, result) ); }); if (accepted.length > 0) { diff --git a/packages/eve/src/execution/settle-cancelled-turn-step.integration.test.ts b/packages/eve/src/execution/settle-cancelled-turn-step.integration.test.ts index 92423a03c..309b232c2 100644 --- a/packages/eve/src/execution/settle-cancelled-turn-step.integration.test.ts +++ b/packages/eve/src/execution/settle-cancelled-turn-step.integration.test.ts @@ -5,7 +5,12 @@ import { createBundledRuntimeCompiledArtifactsSource } from "#runtime/compiled-a import { createDurableSessionState } from "#execution/durable-session-store.js"; import { settleCancelledTurnStep } from "#execution/settle-cancelled-turn-step.js"; import { setHarnessEmissionState } from "#harness/emission.js"; -import { recordWorkflowToolRun } from "#harness/workflow-tool-runs.js"; +import { setPendingCoordinationBatch } from "#harness/coordination.js"; +import { + getWorkflowToolRuns, + registerWorkflowToolRun, + type BackgroundWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import { deriveAgentOperationId } from "#subagents/handles/operation-id.js"; import { AGENT_HANDLES_STATE_KEY, @@ -164,12 +169,12 @@ describe("settleCancelledTurnStep handle store", () => { const runtime = await createTestRuntime({ agent: { name: "settle-cancel-claim" } }); await runtime.run(async () => { - const session = recordWorkflowToolRun(createCancelledTurnSession([CLAIMED_HANDLE]), { + const session = registerWorkflowToolRun(createCancelledTurnSession([CLAIMED_HANDLE]), { callId: "workflow-call", - hookToken: "workflow-hook", - resultKind: "tool", - runId: "workflow-run", toolName: "Workflow", + lifetime: "turn" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "workflow-run", hookToken: "workflow-hook" }, }); const result = await settleCancelledTurnStep({ sessionWritable: new WritableStream({ write() {} }), @@ -189,4 +194,64 @@ describe("settleCancelledTurnStep handle store", () => { }); }); }); + it.each([false, true])( + "retains task payloads on turn cancellation (paused=%s)", + async (paused) => { + const runtime = await createTestRuntime({ agent: { name: "settle-mixed-invocations" } }); + await runtime.run(async () => { + const background: BackgroundWorkflowToolRun = { + callId: "background-call", + lifetime: "session", + toolName: "research", + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "background-run", hookToken: "background-hook" }, + task: { + taskId: "background-task", + metadata: { kind: "tool", name: "research" }, + dispatchContext: { auth: { current: null, initiator: null } }, + }, + }; + let session = registerWorkflowToolRun(createCancelledTurnSession([]), background); + session = registerWorkflowToolRun(session, { + ...background, + callId: "completed-call", + task: { + ...background.task, + taskId: "completed-task", + outcome: { + status: "completed", + lastOutput: { type: "result", data: "retained output" }, + }, + }, + }); + const tasks = getWorkflowToolRuns(session.state); + session = registerWorkflowToolRun(session, { + callId: "waiting-call", + toolName: "research", + lifetime: "turn", + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "waiting-run", hookToken: "waiting-hook" }, + }); + if (paused) + session = setPendingCoordinationBatch({ + event: { sequence: 3, stepIndex: 1, turnId: "turn-1" }, + responseMessages: [], + runtimeActions: [], + tasks: [], + session: setHarnessEmissionState(session, { + sequence: 4, + stepIndex: 0, + sessionStarted: true, + turnId: "", + }), + }); + const result = await settleCancelledTurnStep({ + sessionWritable: new WritableStream({ write() {} }), + serializedContext: buildSerializedContext(), + sessionState: createDurableSessionState({ session }), + }); + expect(getWorkflowToolRuns(result.sessionState.snapshot.session.state)).toEqual(tasks); + }); + }, + ); }); diff --git a/packages/eve/src/execution/settle-cancelled-turn-step.ts b/packages/eve/src/execution/settle-cancelled-turn-step.ts index e32494105..ef64598c8 100644 --- a/packages/eve/src/execution/settle-cancelled-turn-step.ts +++ b/packages/eve/src/execution/settle-cancelled-turn-step.ts @@ -1,3 +1,4 @@ +import { getPendingCoordinationBatch } from "#harness/coordination.js"; import { buildAdapterContext } from "#channel/adapter-context.js"; import { callAdapterEventHandler } from "#channel/adapter.js"; import { dispatchStreamEventHooks } from "#context/hook-lifecycle.js"; @@ -31,7 +32,10 @@ import { abandonRunningAgentTurns, } from "#subagents/handles/transitions.js"; import { clearPendingCoordinationBatch } from "#harness/coordination.js"; -import { clearWorkflowToolRuns, getWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; +import { + removeBlockingWorkflowToolRuns, + getBlockingWorkflowToolRuns, +} from "#harness/workflow-tool-runs.js"; import { bindSessionInstrumentation } from "#instrumentation/runtime.js"; import { getTurnUsageState, toUsage } from "#harness/turn-tag-state.js"; import { @@ -142,10 +146,13 @@ export async function settleCancelledTurnStep(input: { // gone, so a child settlement can never reach this store again. This is the // last write that can park turn-owned `running` and workflow-owned `claimed` // handles. - const workflowToolRuns = getWorkflowToolRuns(session.state); + const owningTurnId = + getPendingCoordinationBatch(session.state)?.event.turnId ?? + input.sessionState.emissionState.turnId; + const workflowToolRuns = getBlockingWorkflowToolRuns(session.state, owningTurnId); session = abandonAgentInvocationOwners( session, - new Set(workflowToolRuns.map((run) => run.runId)), + new Set(workflowToolRuns.map((run) => run.address.runId)), ); const cancelledSession = reconcileSessionContinuationToken( ctx, @@ -153,8 +160,9 @@ export async function settleCancelledTurnStep(input: { clearPendingSessionLimitPrompt( clearAllProxyInputRequests( clearPendingCoordinationBatch( - clearWorkflowToolRuns( + removeBlockingWorkflowToolRuns( abandonRunningAgentTurns({ ...session, outputSchema: undefined }), + owningTurnId, ), ), ), diff --git a/packages/eve/src/execution/stable-workflow-names.ts b/packages/eve/src/execution/stable-workflow-names.ts index 689834f59..9c89f4717 100644 --- a/packages/eve/src/execution/stable-workflow-names.ts +++ b/packages/eve/src/execution/stable-workflow-names.ts @@ -6,7 +6,6 @@ export const TURN_WORKFLOW_NAME = "turnWorkflow"; export const WORKFLOW_ENTRY_NAME = "workflowEntry"; export const SESSION_TIMEOUT_WORKFLOW_NAME = "sessionTimeoutWorkflow"; -export const TASK_RUN_WORKFLOW_NAME = "taskRunWorkflow"; export const WORKFLOW_TOOL_RUN_WORKFLOW_NAME = "workflowToolRunWorkflow"; export const ACTIVITY_COLLECTOR_WORKFLOW_NAME = "activityCollectorWorkflow"; @@ -14,7 +13,6 @@ export const STABLE_WORKFLOW_NAMES: ReadonlySet = new Set([ WORKFLOW_ENTRY_NAME, TURN_WORKFLOW_NAME, SESSION_TIMEOUT_WORKFLOW_NAME, - TASK_RUN_WORKFLOW_NAME, WORKFLOW_TOOL_RUN_WORKFLOW_NAME, ACTIVITY_COLLECTOR_WORKFLOW_NAME, ]); diff --git a/packages/eve/src/execution/tasks/child/steps.test.ts b/packages/eve/src/execution/tasks/child/notify.test.ts similarity index 82% rename from packages/eve/src/execution/tasks/child/steps.test.ts rename to packages/eve/src/execution/tasks/child/notify.test.ts index 4c3708825..1864c1aaa 100644 --- a/packages/eve/src/execution/tasks/child/steps.test.ts +++ b/packages/eve/src/execution/tasks/child/notify.test.ts @@ -1,22 +1,18 @@ +import { HookNotFoundError, RunExpiredError } from "#compiled/@workflow/errors/index.js"; +import { formatTaskNotification } from "#tasks/notification.js"; import { afterEach, describe, expect, it, vi } from "vitest"; import { - appendTaskViewStep, + emitTaskActivityStep, deliverTaskInputResponsesStep, - formatTaskNotification, projectTaskActivity, - wakeTaskAgentRequestParentStep, -} from "#execution/tasks/child/steps.js"; + notifyTaskParent, +} from "#execution/tasks/child/notify.js"; import { resumeWorkflowToolRunAnswers } from "#execution/tools/workflow/answer.js"; import type { TaskView } from "#tasks/types.js"; import { resumeSessionInbox } from "#execution/session-inbox/resume.js"; -import { getWritable } from "#compiled/@workflow/core/index.js"; import { submitActivity } from "#execution/submit-activity.js"; -vi.mock("#compiled/@workflow/core/index.js", async (importOriginal) => ({ - ...(await importOriginal()), - getWritable: vi.fn(), -})); vi.mock("#execution/submit-activity.js", () => ({ submitActivity: vi.fn() })); vi.mock("#execution/session-inbox/resume.js", () => ({ resumeSessionInbox: vi.fn() })); @@ -64,25 +60,20 @@ const notificationCases: readonly { readonly expected: string; readonly view: Ta }, ]; -describe("appendTaskViewStep", () => { +describe("emitTaskActivityStep", () => { it("waits for best-effort activity submission before the step finishes", async () => { const submission = Promise.withResolvers(); const submitted = Promise.withResolvers(); - const releaseLock = vi.fn(); - vi.mocked(getWritable).mockReturnValue({ - getWriter: () => ({ write: vi.fn().mockResolvedValue(undefined), releaseLock }), - } as never); vi.mocked(submitActivity).mockImplementation(() => { submitted.resolve(); return submission.promise; }); let finished = false; - const appended = appendTaskViewStep({ view: notificationCases[0]!.view }).then(() => { + const appended = emitTaskActivityStep({ view: notificationCases[0]!.view }).then(() => { finished = true; }); await submitted.promise; await Promise.resolve(); - expect(releaseLock).toHaveBeenCalledOnce(); expect(finished).toBe(false); submission.resolve(); await appended; @@ -116,7 +107,7 @@ describe("projectTaskActivity", () => { ]); }); - it("projects task work when its initial view is written", () => { + it("projects task work when execution starts", () => { const workIdentity = { id: "work:task", kind: "task" as const, @@ -269,7 +260,7 @@ describe("deliverTaskInputResponsesStep", () => { }); }); -describe("wakeTaskAgentRequestParentStep", () => { +describe("notifyTaskParent", () => { it("forwards an agent invocation through the typed task envelope", async () => { const request = { from: { @@ -290,7 +281,7 @@ describe("wakeTaskAgentRequestParentStep", () => { }, }; - await wakeTaskAgentRequestParentStep({ request, taskId: "task-1", token: "parent-token" }); + await notifyTaskParent({ request, taskId: "task-1", token: "parent-token" }); expect(resumeSessionInbox).toHaveBeenCalledWith("parent-token", { kind: "send", @@ -313,3 +304,41 @@ describe("wakeTaskAgentRequestParentStep", () => { }); }); }); + +describe("notifyTaskParent", () => { + const notification = { + token: "parent-token", + view: { + taskId: "task-1", + metadata, + status: "completed" as const, + lastOutput: { type: "result" as const, data: "done" }, + }, + }; + + it("preserves the payload and deduplication identity", async () => { + await notifyTaskParent(notification); + expect(resumeSessionInbox).toHaveBeenCalledExactlyOnceWith(notification.token, { + kind: "send", + payload: { + message: "Background task task-1 (reviewer) is completed.\n\nResult:\ndone", + task: { views: [notification.view] }, + }, + taskDeliveryId: "task-1:ready:completed", + }); + }); + + it.each([ + new HookNotFoundError("parent-token"), + new Error("delivery failed", { cause: new RunExpiredError("parent ended") }), + ])("tolerates an ended parent", async (error) => { + vi.mocked(resumeSessionInbox).mockRejectedValueOnce(error); + await expect(notifyTaskParent(notification)).resolves.toBeUndefined(); + }); + + it("propagates transient delivery failures so the durable step can retry", async () => { + const error = new Error("storage unavailable"); + vi.mocked(resumeSessionInbox).mockRejectedValueOnce(error); + await expect(notifyTaskParent(notification)).rejects.toBe(error); + }); +}); diff --git a/packages/eve/src/execution/tasks/child/notify.ts b/packages/eve/src/execution/tasks/child/notify.ts new file mode 100644 index 000000000..ef0b57419 --- /dev/null +++ b/packages/eve/src/execution/tasks/child/notify.ts @@ -0,0 +1,235 @@ +import type { ActivityObserverConfig, SessionAuthContext, SessionCommand } from "#channel/types.js"; +import { submitActivity } from "#execution/submit-activity.js"; +import { isTaskWorkflowTargetGone } from "#execution/tasks/workflow-target.js"; +import { resumeSessionInbox } from "#execution/session-inbox/resume.js"; +import { resumeWorkflowToolRunAnswers } from "#execution/tools/workflow/answer.js"; +import type { AnswerHookRoute } from "#harness/proxy-input-requests.js"; +import { createLogger } from "#internal/logging.js"; +import type { ActivityEventV1 } from "#protocol/activity.js"; + +import type { + WorkflowToolAuthorizationRequest, + WorkflowToolRunRequestMessage, + WorkflowToolRunReport, +} from "#execution/tools/workflow/messages.js"; +import { workflowToolRunInputRequests } from "#execution/tools/workflow/owner-inbox.js"; +import { formatTaskNotification, formatTaskOutput } from "#tasks/notification.js"; +import { + isTerminalTaskStatus, + taskAuthorizationRequestId, + type TaskAgentRequestDelivery, + type TaskAuthorizationEventDelivery, + type TaskInputRequestDelivery, + type TaskView, + type TaskInboundAnswerInput, +} from "#tasks/types.js"; + +const log = createLogger("execution.tasks.run"); + +type TaskParentNotification = + | { + readonly view: TaskView; + readonly update?: { readonly report: WorkflowToolRunReport; readonly index: number }; + } + | { + readonly taskId: string; + readonly request: WorkflowToolRunRequestMessage; + }; + +/** Delivers task outcomes, updates, and requests through the parent's session inbox. */ +export async function notifyTaskParent( + input: TaskParentNotification & { readonly token: string }, +): Promise { + "use step"; + + const command = taskNotificationCommand(input); + try { + await resumeSessionInbox(input.token, command); + } catch (error) { + if (!isTaskWorkflowTargetGone(error)) throw error; + log.warn("task notification target is gone; the parent session already ended", { + taskDeliveryId: command.taskDeliveryId, + }); + } +} + +function taskNotificationCommand( + input: TaskParentNotification, +): Extract { + if ("view" in input) { + const { view, update } = input; + if (update !== undefined) { + return { + kind: "send", + payload: { + message: `Background task ${view.taskId} (${view.metadata.name}) update: ${formatTaskOutput(update.report.update)}`, + }, + taskDeliveryId: `${view.taskId}:update:${view.taskId}:${update.index}:${update.report.from.callId}`, + }; + } + const payload: { message: string; task?: { views: readonly TaskView[] } } = { + message: formatTaskNotification(view), + }; + if (isTerminalTaskStatus(view.status)) payload.task = { views: [view] }; + return { + kind: "send", + payload, + taskDeliveryId: `${view.taskId}:ready:${view.status}`, + }; + } + + const { taskId, request: message } = input; + const { request } = message; + if (request.kind === "authorization-request") return taskAuthorizationCommand(request, taskId); + if (request.kind === "agent-invoke" || request.kind === "agent-settled") { + const delivery: TaskAgentRequestDelivery = { replyTo: message.replyTo, request, taskId }; + const invocationId = + request.kind === "agent-invoke" ? request.invocationId : `${request.result.callId}:settled`; + return { + kind: "send", + payload: { task: { agentRequests: [delivery] } }, + taskDeliveryId: `${taskId}:agent:${message.from.runId}:${invocationId}`, + }; + } + + const coordinates = message.requestCoordinates ?? message.from; + const delivery: TaskInputRequestDelivery = { + replyTo: message.replyTo, + requests: workflowToolRunInputRequests(message), + sequence: coordinates.sequence, + stepIndex: coordinates.stepIndex, + taskId, + turnId: coordinates.turnId, + }; + return { + kind: "send", + payload: { task: { inputRequests: [delivery] } }, + taskDeliveryId: `${taskId}:input:${coordinates.turnId}:${coordinates.stepIndex}:${coordinates.sequence}`, + }; +} + +function taskAuthorizationCommand( + request: WorkflowToolAuthorizationRequest, + taskId: string, +): Extract { + const { event: hookPayload } = request; + const data = hookPayload.event.data; + const payload: { + message?: string; + task: { authorizationEvents: TaskAuthorizationEventDelivery[] }; + } = { task: { authorizationEvents: [{ hookPayload, taskId }] } }; + if (hookPayload.event.type === "authorization.required") { + payload.message = `Background task ${taskId} needs authorization.`; + } + return { + kind: "send", + payload, + taskDeliveryId: `${taskId}:authorization:${hookPayload.event.type}:${data.turnId}:${data.stepIndex}:${data.sequence}:${taskAuthorizationRequestId(hookPayload.event)}`, + }; +} + +/** Emits task activity without storing a second task-state record. */ +export async function emitTaskActivityStep(input: { + readonly activityObserver?: ActivityObserverConfig; + readonly view: TaskView; +}): Promise { + "use step"; + + const events = projectTaskActivity({ + activityObserver: input.activityObserver, + settledAt: new Date().toISOString(), + view: input.view, + }); + await submitActivity({ events, sink: input.activityObserver?.sink }); +} + +export function projectTaskActivity(input: { + readonly activityObserver: ActivityObserverConfig | undefined; + readonly settledAt: string; + readonly view: TaskView; +}): readonly ActivityEventV1[] { + const work = input.activityObserver?.workIdentity; + if (work === undefined) return []; + const status = input.view.status; + if (status === "working") { + return [ + { + eventId: `${work.id}:started`, + kind: "work.started", + startedAt: input.settledAt, + work, + }, + ]; + } + if (status !== "completed" && status !== "failed" && status !== "cancelled") return []; + return [ + { + eventId: `${work.id}:settled:${status}`, + kind: "work.settled", + outcome: status, + settledAt: input.settledAt, + workId: work.id, + }, + ]; +} + +/** + * Forwards answered input to the blocked child. + * + * The task run performs this itself so the child unblocks and the + * view leaves `input_required` under one durable decision. Returns + * `unreachable` when the child hook is already gone, which leaves the + * outstanding batch untouched rather than reporting a task as working + * when nothing received the answer. + */ +export async function deliverTaskInputResponsesStep(input: { + readonly answer: TaskInboundAnswerInput; + readonly answerHook?: AnswerHookRoute; + readonly requestIds: readonly string[]; +}): Promise<"delivered" | "unreachable"> { + "use step"; + + const answered = new Set(input.requestIds); + const command: SessionCommand = { + auth: input.answer.auth as SessionAuthContext | null | undefined, + kind: "send", + payload: { + inputResponses: input.answer.inputResponses.filter((response) => + answered.has(response.requestId), + ), + }, + taskDeliveryId: `${input.answer.taskId}:${[...input.requestIds].sort().join(",")}`, + }; + try { + if (input.answer.childResponseUrl !== undefined) { + const response = await fetch(input.answer.childResponseUrl, { + body: JSON.stringify({ inputResponses: command.payload.inputResponses }), + headers: { "content-type": "application/json" }, + method: "POST", + redirect: "error", + }); + if (response.status === 404) return "unreachable"; + if (!response.ok) + throw new Error(`Remote task input delivery failed with HTTP ${response.status}.`); + } else if (input.answerHook !== undefined) { + await resumeWorkflowToolRunAnswers( + input.answer.childContinuationToken, + command.payload.inputResponses, + ); + } else { + await resumeSessionInbox( + input.answer.childSessionInbox ?? input.answer.childContinuationToken, + command, + ); + } + return "delivered"; + } catch (error) { + if (isTaskWorkflowTargetGone(error)) { + log.warn("task input answer target is gone; the child turn already ended", { + taskId: input.answer.taskId, + }); + return "unreachable"; + } + throw error; + } +} diff --git a/packages/eve/src/execution/tasks/child/steps.ts b/packages/eve/src/execution/tasks/child/steps.ts deleted file mode 100644 index 44cdba717..000000000 --- a/packages/eve/src/execution/tasks/child/steps.ts +++ /dev/null @@ -1,365 +0,0 @@ -import { getWritable } from "#compiled/@workflow/core/index.js"; -import type { ActivityObserverConfig, SessionAuthContext, SessionCommand } from "#channel/types.js"; -import type { - WorkflowToolAuthorizationRequest, - WorkflowToolRunRequestMessage, -} from "#execution/tools/workflow/messages.js"; -import type { WorkflowToolRunTaskInputRequest } from "./workflow.js"; -import { submitActivity } from "#execution/submit-activity.js"; -import { isTaskWorkflowTargetGone } from "#execution/tasks/workflow-target.js"; -import { resumeSessionInbox } from "#execution/session-inbox/resume.js"; -import { resumeWorkflowToolRunAnswers } from "#execution/tools/workflow/answer.js"; -import type { AnswerHookRoute } from "#harness/proxy-input-requests.js"; -import { createLogger } from "#internal/logging.js"; -import type { ActivityEventV1 } from "#protocol/activity.js"; -import type { JsonValue } from "#shared/json.js"; -import { - isTerminalTaskStatus, - TASK_PROGRESS_STREAM_NAMESPACE, - TASK_VIEW_STREAM_NAMESPACE, - taskAuthorizationRequestId, - type TaskAgentRequestDelivery, - type TaskAuthorizationEventDelivery, - type TaskInboundAnswerInput, - type TaskInboundMessage, - type TaskInboundUpdate, - type TaskInputRequestDelivery, - type TaskProgress, - type TaskView, -} from "#tasks/types.js"; - -const log = createLogger("execution.tasks.run"); - -/** - * Appends one full task view to the owning task run's `eve.task` - * stream. Only the task run workflow calls this, which is what makes - * the run the single writer readers can trust without re-validating. - */ -export async function appendTaskViewStep(input: { - readonly activityObserver?: ActivityObserverConfig; - readonly view: TaskView; -}): Promise { - "use step"; - - const writable = getWritable({ namespace: TASK_VIEW_STREAM_NAMESPACE }); - const writer = writable.getWriter(); - try { - await writer.write(input.view); - } finally { - writer.releaseLock(); - } - - const events = projectTaskActivity({ - activityObserver: input.activityObserver, - settledAt: new Date().toISOString(), - view: input.view, - }); - await submitActivity({ events, sink: input.activityObserver?.sink }); -} - -export function projectTaskActivity(input: { - readonly activityObserver: ActivityObserverConfig | undefined; - readonly settledAt: string; - readonly view: TaskView; -}): readonly ActivityEventV1[] { - const work = input.activityObserver?.workIdentity; - if (work === undefined) return []; - const status = input.view.status; - if (status === "working") { - return [ - { - eventId: `${work.id}:started`, - kind: "work.started", - startedAt: input.settledAt, - work, - }, - ]; - } - if (status !== "completed" && status !== "failed" && status !== "cancelled") return []; - return [ - { - eventId: `${work.id}:settled:${status}`, - kind: "work.settled", - outcome: status, - settledAt: input.settledAt, - workId: work.id, - }, - ]; -} - -/** Appends task progress without starting a parent turn. */ -export async function appendTaskProgressStep(input: { - readonly progress: TaskProgress; -}): Promise { - "use step"; - - const writable = getWritable({ namespace: TASK_PROGRESS_STREAM_NAMESPACE }); - const writer = writable.getWriter(); - try { - await writer.write(input.progress); - } finally { - writer.releaseLock(); - } -} - -/** - * Wakes the parent session with a framework task notification. - * - * Rides the ordinary session delivery path: a parked parent starts a - * turn carrying this message, while an active turn observes it at the - * next safe boundary through the owner's normal delivery routing. A - * parent whose session already ended is a tolerated no-op. - */ -export async function wakeTaskParentStep(input: { - readonly token: string; - readonly view: TaskView; -}): Promise { - "use step"; - - const payload: { message: string; task?: { views: readonly TaskView[] } } = { - message: formatTaskNotification(input.view), - }; - if (isTerminalTaskStatus(input.view.status)) payload.task = { views: [input.view] }; - const command: SessionCommand = { - kind: "send", - payload, - taskDeliveryId: `${input.view.taskId}:ready:${input.view.status}`, - }; - try { - await resumeSessionInbox(input.token, command); - } catch (error) { - if (isTaskWorkflowTargetGone(error)) { - log.warn("task wake target is gone; the parent session already ended", { - status: input.view.status, - taskId: input.view.taskId, - }); - return; - } - throw error; - } -} - -/** Forwards a running child's intermediate update to its parent session. */ -export async function wakeTaskUpdateParentStep(input: { - readonly token: string; - readonly update: TaskInboundUpdate; - readonly view: TaskView; -}): Promise { - "use step"; - - const command: SessionCommand = { - kind: "send", - payload: { - message: `Background task ${input.view.taskId} (${input.view.metadata.name}) update: ${input.update.message}`, - }, - taskDeliveryId: `${input.view.taskId}:update:${input.update.updateEpoch}:${input.update.updateIndex}:${input.update.callId}`, - }; - try { - await resumeSessionInbox(input.token, command); - } catch (error) { - if (isTaskWorkflowTargetGone(error)) return; - throw error; - } -} - -/** Delivers one task-authored message to the parent as a new turn. */ -export async function wakeTaskMessageParentStep(input: { - readonly message: TaskInboundMessage; - readonly taskId: string; - readonly token: string; -}): Promise { - "use step"; - - const command: SessionCommand = { - kind: "send", - payload: { message: input.message.message }, - taskDeliveryId: `${input.taskId}:message:${input.message.messageEpoch}:${input.message.messageIndex}:${input.message.callId}`, - }; - try { - await resumeSessionInbox(input.token, command); - } catch (error) { - if (isTaskWorkflowTargetGone(error)) return; - throw error; - } -} - -/** Forwards one agent spawn or settlement request to the parent session. */ -export async function wakeTaskAgentRequestParentStep(input: { - readonly request: WorkflowToolRunRequestMessage; - readonly taskId: string; - readonly token: string; -}): Promise { - "use step"; - - const request = input.request.request; - if (request.kind !== "agent-invoke" && request.kind !== "agent-settled") { - throw new Error("Cannot forward task input as an agent request."); - } - const delivery: TaskAgentRequestDelivery = { - replyTo: input.request.replyTo, - request, - taskId: input.taskId, - }; - const invocationId = - request.kind === "agent-invoke" ? request.invocationId : `${request.result.callId}:settled`; - const command: SessionCommand = { - kind: "send", - payload: { task: { agentRequests: [delivery] } }, - taskDeliveryId: `${input.taskId}:agent:${input.request.from.runId}:${invocationId}`, - }; - try { - await resumeSessionInbox(input.token, command); - } catch (error) { - if (!isTaskWorkflowTargetGone(error)) throw error; - } -} - -/** Re-emits a task child's authorization event through the parent channel. */ -export async function wakeTaskAuthorizationParentStep(input: { - readonly request: WorkflowToolAuthorizationRequest; - readonly taskId: string; - readonly token: string; -}): Promise { - "use step"; - - const { event: hookPayload } = input.request; - const data = hookPayload.event.data; - const payload: { - message?: string; - task: { authorizationEvents: TaskAuthorizationEventDelivery[] }; - } = { task: { authorizationEvents: [{ hookPayload, taskId: input.taskId }] } }; - if (hookPayload.event.type === "authorization.required") { - payload.message = `Background task ${input.taskId} needs authorization.`; - } - const command: SessionCommand = { - kind: "send", - payload, - taskDeliveryId: `${input.taskId}:authorization:${hookPayload.event.type}:${data.turnId}:${data.stepIndex}:${data.sequence}:${taskAuthorizationRequestId(hookPayload.event)}`, - }; - try { - await resumeSessionInbox(input.token, command); - } catch (error) { - if (!isTaskWorkflowTargetGone(error)) throw error; - } -} - -/** Sends a workflow-body question to the owning parent's pre-model router. */ -export async function wakeWorkflowTaskInputRequestParentStep(input: { - readonly request: WorkflowToolRunTaskInputRequest; - readonly taskId: string; - readonly token: string; -}): Promise { - "use step"; - - const delivery: TaskInputRequestDelivery = - input.request.requests === undefined - ? { - replyTo: input.request.replyTo, - request: input.request.request, - sequence: input.request.sequence, - stepIndex: input.request.stepIndex, - taskId: input.taskId, - turnId: input.request.turnId, - } - : { - replyTo: input.request.replyTo, - requests: input.request.requests, - sequence: input.request.sequence, - stepIndex: input.request.stepIndex, - taskId: input.taskId, - turnId: input.request.turnId, - }; - const command: SessionCommand = { - kind: "send", - payload: { - task: { - inputRequests: [delivery], - }, - }, - taskDeliveryId: `${input.taskId}:input:${input.request.turnId}:${input.request.stepIndex}:${input.request.sequence}`, - }; - try { - await resumeSessionInbox(input.token, command); - } catch (error) { - if (!isTaskWorkflowTargetGone(error)) throw error; - } -} - -/** - * Forwards answered input to the blocked child. - * - * The task run performs this itself so the child unblocks and the - * view leaves `input_required` under one durable decision. Returns - * `unreachable` when the child hook is already gone, which leaves the - * outstanding batch untouched rather than reporting a task as working - * when nothing received the answer. - */ -export async function deliverTaskInputResponsesStep(input: { - readonly answer: TaskInboundAnswerInput; - readonly answerHook?: AnswerHookRoute; - readonly requestIds: readonly string[]; -}): Promise<"delivered" | "unreachable"> { - "use step"; - - const answered = new Set(input.requestIds); - const command: SessionCommand = { - auth: input.answer.auth as SessionAuthContext | null | undefined, - kind: "send", - payload: { - inputResponses: input.answer.inputResponses.filter((response) => - answered.has(response.requestId), - ), - }, - taskDeliveryId: `${input.answer.taskId}:${[...input.requestIds].sort().join(",")}`, - }; - try { - if (input.answer.childResponseUrl !== undefined) { - const response = await fetch(input.answer.childResponseUrl, { - body: JSON.stringify({ inputResponses: command.payload.inputResponses }), - headers: { "content-type": "application/json" }, - method: "POST", - redirect: "error", - }); - if (response.status === 404) return "unreachable"; - if (!response.ok) - throw new Error(`Remote task input delivery failed with HTTP ${response.status}.`); - } else if (input.answerHook !== undefined) { - await resumeWorkflowToolRunAnswers( - input.answer.childContinuationToken, - command.payload.inputResponses, - ); - } else { - await resumeSessionInbox( - input.answer.childSessionInbox ?? input.answer.childContinuationToken, - command, - ); - } - return "delivered"; - } catch (error) { - if (isTaskWorkflowTargetGone(error)) { - log.warn("task input answer target is gone; the child turn already ended", { - taskId: input.answer.taskId, - }); - return "unreachable"; - } - throw error; - } -} - -export function formatTaskNotification(view: TaskView): string { - const subject = `Background task ${view.taskId} (${view.metadata.name})`; - if (view.status === "input_required") { - return `${subject} needs input.`; - } - if (view.status === "completed") { - return `${subject} is completed.\n\nResult:\n${formatTaskOutput(view.lastOutput.data)}`; - } - if (view.status === "failed") { - return `${subject} failed.\n\nError:\n${formatTaskOutput(view.lastOutput.data)}`; - } - return `${subject} is cancelled.`; -} - -function formatTaskOutput(output: JsonValue): string { - return typeof output === "string" ? output : (JSON.stringify(output) ?? "null"); -} diff --git a/packages/eve/src/execution/tasks/child/workflow.test.ts b/packages/eve/src/execution/tasks/child/workflow.test.ts deleted file mode 100644 index a864b583c..000000000 --- a/packages/eve/src/execution/tasks/child/workflow.test.ts +++ /dev/null @@ -1,587 +0,0 @@ -import { beforeEach, describe, expect, it, vi } from "vitest"; - -import type { WorkflowToolRunMessage } from "#execution/tools/workflow/messages.js"; -import { taskRunWorkflow } from "#execution/tasks/child/workflow.js"; -import type { TaskView } from "#tasks/types.js"; -import { - createAuthorizationRequiredEvent, - createAuthorizationCompletedEvent, -} from "#protocol/message.js"; - -const mocks = vi.hoisted(() => ({ - appendTaskProgressStep: vi.fn(), - appendTaskViewStep: vi.fn(), - cancelWorkflowToolRunStep: vi.fn(), - claimHookOwnership: vi.fn(), - createChannelReader: vi.fn((channel: string) => ({ channel, iterator: [][Symbol.iterator]() })), - createHook: vi.fn(() => ({ token: "task-token" })), - deliverTaskInputResponsesStep: vi.fn(), - openWorkflowToolRunOwnerInbox: vi.fn(() => ({ - owner: { inbox: "generated-owner-token" }, - reader: { channel: "workflow" }, - })), - raceChannelReads: vi.fn(), - resumeHookStep: vi.fn(), - wakeTaskAgentRequestParentStep: vi.fn(), - wakeTaskAuthorizationParentStep: vi.fn(), - wakeTaskMessageParentStep: vi.fn(), - wakeTaskParentStep: vi.fn(), - wakeTaskUpdateParentStep: vi.fn(), - wakeWorkflowTaskInputRequestParentStep: vi.fn(), - executeWorkflowBody: vi.fn(), - createWorkflowBodyRef: vi.fn((input) => ({ - callId: input.callId, - execution: input.execution, - input: input.input, - runId: "task-run", - sequence: input.session.turn.sequence, - stepIndex: input.stepIndex, - toolName: input.toolName, - turnId: input.session.turn.id, - })), -})); - -vi.mock("#compiled/@workflow/core/index.js", async (importOriginal) => ({ - ...(await importOriginal()), - createHook: mocks.createHook, -})); -vi.mock("#execution/hook-ownership.js", () => ({ - claimHookOwnership: mocks.claimHookOwnership, - isHookConflictError: () => false, -})); -vi.mock("#execution/tasks/child/steps.js", () => ({ - appendTaskProgressStep: mocks.appendTaskProgressStep, - appendTaskViewStep: mocks.appendTaskViewStep, - deliverTaskInputResponsesStep: mocks.deliverTaskInputResponsesStep, - wakeTaskAgentRequestParentStep: mocks.wakeTaskAgentRequestParentStep, - wakeTaskMessageParentStep: mocks.wakeTaskMessageParentStep, - wakeTaskAuthorizationParentStep: mocks.wakeTaskAuthorizationParentStep, - wakeTaskParentStep: mocks.wakeTaskParentStep, - wakeTaskUpdateParentStep: mocks.wakeTaskUpdateParentStep, - wakeWorkflowTaskInputRequestParentStep: mocks.wakeWorkflowTaskInputRequestParentStep, -})); -vi.mock("#execution/tools/workflow/cancel.js", () => ({ - cancelWorkflowToolRunStep: mocks.cancelWorkflowToolRunStep, -})); -vi.mock("#execution/tools/workflow/owner-channels.js", () => ({ - createChannelReader: mocks.createChannelReader, - raceChannelReads: mocks.raceChannelReads, -})); -vi.mock("#execution/tools/workflow/owner.js", () => ({ - openWorkflowToolRunOwnerInbox: mocks.openWorkflowToolRunOwnerInbox, -})); -vi.mock("#execution/tools/workflow/resume-hook-step.js", () => ({ - resumeHookStep: mocks.resumeHookStep, -})); -vi.mock("#execution/tools/workflow/body.js", () => ({ - createWorkflowBodyRef: mocks.createWorkflowBodyRef, - executeWorkflowBody: mocks.executeWorkflowBody, -})); - -const initialView = { - metadata: { kind: "tool", name: "approval-worker" }, - status: "working", - taskId: "task-1", -} satisfies TaskView; - -const bufferedAgentRequest = { - kind: "request", - from: { - callId: "tool-call-1", - execution: "background", - input: { message: "authorize" }, - runId: "run-1", - sequence: 0, - stepIndex: 0, - toolName: "approval-worker", - turnId: "turn-parent", - }, - replyTo: "agent-reply", - request: { - input: { message: "authorize", target: "approver" }, - invocationId: "tool-call-1:approver", - kind: "agent-invoke", - }, -} satisfies WorkflowToolRunMessage; - -const workflowAgentRequest = { - ...bufferedAgentRequest, - request: { - input: { message: "authorize", target: "approver" }, - invocationId: "tool-call-1:approver:2", - kind: "agent-invoke", - }, -} satisfies WorkflowToolRunMessage; - -function authorizationRequest(attemptId: string, completed = false) { - const data = { attemptId, name: "github", sequence: 0, stepIndex: 0, turnId: "turn-parent" }; - return { - ...bufferedAgentRequest, - replyTo: `ack-${attemptId}`, - request: { - kind: "authorization-request", - event: { - kind: "subagent-authorization-event", - callId: "tool-call-1", - childSessionId: "run-1", - subagentName: "approval-worker", - event: completed - ? createAuthorizationCompletedEvent({ ...data, outcome: "authorized" }) - : createAuthorizationRequiredEvent({ ...data, description: "Sign in" }), - }, - }, - } satisfies WorkflowToolRunMessage; -} - -function queueOwnerRequest(value: WorkflowToolRunMessage) { - mocks.raceChannelReads.mockResolvedValueOnce({ - channel: "workflow", - next: { done: false, value }, - }); -} - -function queueCommand(command: import("#tasks/types.js").TaskCommand) { - mocks.raceChannelReads.mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { kind: "task-command", command } }, - }); -} - -const workflowInput = { - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", -}; - -describe("taskRunWorkflow", () => { - beforeEach(() => { - vi.resetAllMocks(); - mocks.createHook.mockReturnValue({ token: "task-token" }); - mocks.openWorkflowToolRunOwnerInbox.mockReturnValue({ - owner: { inbox: "generated-owner-token" }, - reader: { channel: "workflow" }, - }); - mocks.executeWorkflowBody.mockResolvedValue({ - outcome: { output: "done", status: "completed" }, - reportCount: 0, - }); - }); - - it("persists auth requests and answers before forwarding and acknowledging each event", async () => { - queueCommand({ kind: "ready" }); - queueOwnerRequest(authorizationRequest("a")); - queueOwnerRequest(authorizationRequest("b")); - queueOwnerRequest(authorizationRequest("a", true)); - queueOwnerRequest(authorizationRequest("b", true)); - mocks.raceChannelReads.mockResolvedValueOnce({ channel: "commands", next: { done: true } }); - - await taskRunWorkflow(workflowInput); - - const views = mocks.appendTaskViewStep.mock.calls.slice(-4).map(([input]) => input.view); - expect(views.map((view) => view.status)).toEqual([ - "input_required", - "input_required", - "input_required", - "working", - ]); - expect( - views - .slice(0, 3) - .map((view) => - view.inputRequests.map((request: { requestId: string }) => request.requestId), - ), - ).toEqual([["a"], ["a", "b"], ["b"]]); - expect(mocks.wakeTaskAuthorizationParentStep).toHaveBeenCalledTimes(4); - expect(mocks.resumeHookStep).toHaveBeenCalledTimes(4); - expect(mocks.wakeTaskParentStep).not.toHaveBeenCalled(); - for (let i = 0; i < 4; i++) { - expect(mocks.appendTaskViewStep.mock.invocationCallOrder[i + 2]).toBeLessThan( - mocks.wakeTaskAuthorizationParentStep.mock.invocationCallOrder[i]!, - ); - expect(mocks.wakeTaskAuthorizationParentStep.mock.invocationCallOrder[i]).toBeLessThan( - mocks.resumeHookStep.mock.invocationCallOrder[i]!, - ); - } - }); - - it("forwards child-agent auth without treating it as the workflow's own request", async () => { - const message = authorizationRequest("child"); - message.request.event.childSessionId = "child-session"; - queueCommand({ kind: "ready" }); - queueOwnerRequest(message); - mocks.raceChannelReads.mockResolvedValueOnce({ channel: "commands", next: { done: true } }); - - await taskRunWorkflow(workflowInput); - - expect(mocks.wakeTaskAuthorizationParentStep).toHaveBeenCalledExactlyOnceWith({ - request: message.request, - taskId: initialView.taskId, - token: workflowInput.parentContinuationToken, - }); - expect(mocks.resumeHookStep).not.toHaveBeenCalled(); - expect( - mocks.appendTaskViewStep.mock.calls.some(([input]) => input.view.status === "input_required"), - ).toBe(false); - }); - - it("acknowledges buffered auth when dispatch is rejected without forwarding it", async () => { - queueOwnerRequest(authorizationRequest("a")); - queueCommand({ kind: "reject-dispatch", data: "rejected" }); - - await taskRunWorkflow(workflowInput); - - expect(mocks.wakeTaskAuthorizationParentStep).not.toHaveBeenCalled(); - expect(mocks.resumeHookStep).toHaveBeenCalledExactlyOnceWith("ack-a", null, { - ifPresent: true, - }); - expect( - mocks.appendTaskViewStep.mock.calls.some(([input]) => input.view.status === "input_required"), - ).toBe(false); - }); - - it.each([false, true])( - "does not reopen a cancelled task for buffered auth (completed=%s)", - async (completed) => { - queueOwnerRequest(authorizationRequest("a", completed)); - queueCommand({ kind: "cancel" }); - queueCommand({ kind: "ready" }); - - await taskRunWorkflow(workflowInput); - - expect(mocks.wakeTaskAuthorizationParentStep).toHaveBeenCalledTimes(completed ? 1 : 0); - expect(mocks.resumeHookStep).toHaveBeenCalledExactlyOnceWith("ack-a", null, { - ifPresent: true, - }); - expect(mocks.appendTaskViewStep.mock.calls.map(([input]) => input.view.status)).toEqual([ - "working", - "cancelled", - ]); - }, - ); - - it.each(["persistence", "forwarding"])( - "does not acknowledge auth after failed %s", - async (failure) => { - queueCommand({ kind: "ready" }); - queueOwnerRequest(authorizationRequest("a")); - if (failure === "persistence") { - mocks.appendTaskViewStep.mockImplementation(async ({ view }) => { - if (view.status === "input_required") throw new Error("failed persistence"); - }); - } else { - mocks.wakeTaskAuthorizationParentStep.mockRejectedValue(new Error("failed forwarding")); - } - - await expect(taskRunWorkflow(workflowInput)).rejects.toThrow(`failed ${failure}`); - expect(mocks.resumeHookStep).not.toHaveBeenCalled(); - }, - ); - - it("delivers an authored message queued before completion and dispatch acknowledgement", async () => { - const message = { - callId: "call-1", - kind: "task-message" as const, - message: "Review the export", - messageEpoch: "task-1", - messageIndex: 0, - }; - for (const value of [ - message, - { kind: "task-command", command: { kind: "complete", data: "done" } }, - { kind: "task-command", command: { kind: "ready" } }, - ]) { - mocks.raceChannelReads.mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value }, - }); - } - await taskRunWorkflow({ - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - }); - expect(mocks.wakeTaskMessageParentStep).toHaveBeenCalledWith({ - message, - taskId: "task-1", - token: "parent-token", - }); - expect(mocks.wakeTaskMessageParentStep.mock.invocationCallOrder[0]).toBeLessThan( - mocks.wakeTaskParentStep.mock.invocationCallOrder[0]!, - ); - }); - - it("buffers agent requests until task dispatch is acknowledged", async () => { - mocks.raceChannelReads - .mockResolvedValueOnce({ - channel: "workflow", - next: { done: false, value: bufferedAgentRequest }, - }) - .mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, - }) - .mockResolvedValueOnce({ channel: "commands", next: { done: true, value: undefined } }); - - await taskRunWorkflow({ - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - }); - - expect(mocks.wakeTaskAgentRequestParentStep).toHaveBeenCalledWith({ - request: bufferedAgentRequest, - taskId: "task-1", - token: "parent-token", - }); - expect(mocks.raceChannelReads.mock.invocationCallOrder[1]).toBeLessThan( - mocks.wakeTaskAgentRequestParentStep.mock.invocationCallOrder[0]!, - ); - }); - - it("forwards admitted agent requests through the task's owner channel", async () => { - mocks.raceChannelReads - .mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, - }) - .mockResolvedValueOnce({ - channel: "workflow", - next: { done: false, value: workflowAgentRequest }, - }) - .mockResolvedValueOnce({ channel: "commands", next: { done: true, value: undefined } }); - - await taskRunWorkflow({ - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - }); - - expect(mocks.wakeTaskAgentRequestParentStep).toHaveBeenCalledWith({ - request: workflowAgentRequest, - taskId: "task-1", - token: "parent-token", - }); - }); - - it("does not execute a workflow body before task admission", async () => { - mocks.raceChannelReads.mockResolvedValueOnce({ - channel: "commands", - next: { done: true, value: undefined }, - }); - - await taskRunWorkflow({ - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - workflow: { - callId: "call-1", - input: {}, - session: { - auth: { current: null, initiator: null }, - id: "session-1", - turn: { id: "turn-1", sequence: 0 }, - }, - stepIndex: 0, - toolName: "worker", - workflowId: "workflow//eve//worker", - }, - }); - - expect(mocks.executeWorkflowBody).not.toHaveBeenCalled(); - }); - - it("starts the workflow body only after ready", async () => { - mocks.raceChannelReads - .mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, - }) - .mockResolvedValueOnce({ channel: "commands", next: { done: true, value: undefined } }); - - await taskRunWorkflow({ - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - workflow: { - callId: "call-1", - input: {}, - session: { - auth: { current: null, initiator: null }, - id: "session-1", - turn: { id: "turn-1", sequence: 0 }, - }, - stepIndex: 0, - toolName: "worker", - workflowId: "workflow//eve//worker", - }, - }); - - expect(mocks.executeWorkflowBody).toHaveBeenCalledOnce(); - expect(mocks.executeWorkflowBody).toHaveBeenCalledWith( - expect.objectContaining({ owner: { inbox: "generated-owner-token" } }), - expect.any(AbortSignal), - ); - }); - - it.each(["tool", "subagent"])( - "routes %s progress without changing subagent delivery", - async (kind) => { - const update = { - callId: "call-1", - kind: "task-update" as const, - message: "progress", - updateEpoch: "task-1", - updateIndex: 0, - }; - mocks.raceChannelReads - .mockResolvedValueOnce({ channel: "commands", next: { done: false, value: update } }) - .mockResolvedValueOnce({ - channel: "commands", - next: { - done: false, - value: { - command: { data: "done", kind: "complete" }, - kind: "task-command", - }, - }, - }) - .mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, - }); - - await taskRunWorkflow({ - initialView: { ...initialView, metadata: { ...initialView.metadata, kind } }, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - }); - - if (kind === "subagent") { - expect(mocks.appendTaskProgressStep).not.toHaveBeenCalled(); - expect(mocks.wakeTaskUpdateParentStep).toHaveBeenCalledWith({ - token: "parent-token", - update, - view: expect.objectContaining({ status: "completed" }), - }); - expect(mocks.wakeTaskUpdateParentStep.mock.invocationCallOrder[0]).toBeLessThan( - mocks.wakeTaskParentStep.mock.invocationCallOrder[0]!, - ); - return; - } - expect(mocks.appendTaskProgressStep).toHaveBeenCalledWith({ - progress: { - callId: "call-1", - kind: "task-progress", - taskId: "task-1", - update: "progress", - updateIndex: 0, - }, - }); - expect(mocks.wakeTaskUpdateParentStep).not.toHaveBeenCalled(); - }, - ); - - it("publishes cancellation after the workflow body observes its abort", async () => { - mocks.raceChannelReads - .mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, - }) - .mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { command: { kind: "cancel" }, kind: "task-command" } }, - }) - .mockResolvedValueOnce({ - channel: "body", - next: { - done: false, - value: { outcome: { reason: "cancelled", status: "cancelled" }, reportCount: 0 }, - }, - }); - mocks.executeWorkflowBody.mockImplementation( - async (_input, signal: AbortSignal) => - await new Promise((resolve) => { - signal.addEventListener( - "abort", - () => - resolve({ outcome: { reason: "cancelled", status: "cancelled" }, reportCount: 0 }), - { once: true }, - ); - }), - ); - - await taskRunWorkflow({ - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - workflow: { - callId: "call-1", - input: {}, - session: { - auth: { current: null, initiator: null }, - id: "session-1", - turn: { id: "turn-1", sequence: 0 }, - }, - stepIndex: 0, - toolName: "worker", - workflowId: "workflow//eve//worker", - }, - }); - - expect(mocks.wakeTaskParentStep).toHaveBeenCalledWith({ - token: "parent-token", - view: expect.objectContaining({ status: "cancelled" }), - }); - }); - - it("consumes every persisted report before accepting a body's completion", async () => { - mocks.raceChannelReads - .mockResolvedValueOnce({ - channel: "commands", - next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, - }) - .mockResolvedValueOnce({ - channel: "body", - next: { - done: false, - value: { outcome: { output: "done", status: "completed" }, reportCount: 1 }, - }, - }) - .mockResolvedValueOnce({ - channel: "workflow", - next: { - done: false, - value: { - kind: "report", - from: bufferedAgentRequest.from, - update: { kind: "eve:task-message", message: "Review the export" }, - }, - }, - }); - await taskRunWorkflow({ - initialView, - parentContinuationToken: "parent-token", - taskInboxToken: "task-token", - workflow: { - callId: "call-1", - input: {}, - session: { - auth: { current: null, initiator: null }, - id: "session-1", - turn: { id: "turn-1", sequence: 0 }, - }, - stepIndex: 0, - toolName: "worker", - workflowId: "workflow//eve//worker", - }, - }); - expect(mocks.wakeTaskMessageParentStep).toHaveBeenCalledWith({ - message: expect.objectContaining({ message: "Review the export" }), - taskId: "task-1", - token: "parent-token", - }); - expect(mocks.wakeTaskMessageParentStep.mock.invocationCallOrder[0]).toBeLessThan( - mocks.wakeTaskParentStep.mock.invocationCallOrder[0]!, - ); - }); -}); diff --git a/packages/eve/src/execution/tasks/child/workflow.ts b/packages/eve/src/execution/tasks/child/workflow.ts deleted file mode 100644 index 29536591f..000000000 --- a/packages/eve/src/execution/tasks/child/workflow.ts +++ /dev/null @@ -1,435 +0,0 @@ -import { createHook } from "#compiled/@workflow/core/index.js"; - -import type { ActivityObserverConfig } from "#channel/types.js"; -import { claimHookOwnership, isHookConflictError } from "#execution/hook-ownership.js"; -import { - appendTaskProgressStep, - appendTaskViewStep, - deliverTaskInputResponsesStep, - wakeTaskAgentRequestParentStep, - wakeTaskAuthorizationParentStep, - wakeTaskMessageParentStep, - wakeTaskParentStep, - wakeTaskUpdateParentStep, - wakeWorkflowTaskInputRequestParentStep, -} from "#execution/tasks/child/steps.js"; -import { - createWorkflowBodyRef, - executeWorkflowBody, - type WorkflowBodyDefinition, - type WorkflowBodyResult, -} from "#execution/tools/workflow/body.js"; -import { resumeHookStep } from "#execution/tools/workflow/resume-hook-step.js"; -import type { - WorkflowToolAuthorizationRequest, - WorkflowToolRunRequestMessage, -} from "#execution/tools/workflow/messages.js"; -import { createChannelReader, raceChannelReads } from "#execution/tools/workflow/owner-channels.js"; -import { openWorkflowToolRunOwnerInbox } from "#execution/tools/workflow/owner.js"; -import { - workflowToolRunOutcomeToTaskCommand, - workflowToolRunReportToTaskPayload, - workflowToolRunRequestToTaskInputRequest, -} from "#execution/tools/workflow/owner-inbox.js"; -import type { AnswerHookRoute } from "#harness/proxy-input-requests.js"; -import { applyTaskTransition } from "#tasks/transitions.js"; -import { - isReadyTaskStatus, - isTerminalTaskStatus, - readTaskInputRequestId, - type TaskCommand, - type TaskInboundAnswerInput, - type TaskInboundMessage, - type TaskInputRequest, - type TaskInboundUpdate, - type TaskRunInboundPayload, - type TaskView, -} from "#tasks/types.js"; - -export interface TaskRunWorkflowInput { - readonly activityObserver?: ActivityObserverConfig; - readonly initialView: TaskView; - readonly parentContinuationToken: string; - readonly taskInboxToken: string; - readonly workflow?: WorkflowBodyDefinition; -} - -/** A workflow-body question routed through the task that owns the workflow tool run. */ -interface WorkflowToolRunTaskInputRequestBase { - readonly kind: "task-input-request"; - readonly replyTo: string; - readonly sequence: number; - readonly stepIndex: number; - readonly turnId: string; -} - -export type WorkflowToolRunTaskInputRequest = WorkflowToolRunTaskInputRequestBase & - ( - | { readonly request: TaskInputRequest; readonly requests?: never } - | { readonly request?: never; readonly requests: readonly TaskInputRequest[] } - ); - -interface PendingWorkflowToolTraffic { - readonly messages: TaskInboundMessage[]; - readonly ownerRequests: WorkflowToolRunRequestMessage[]; -} - -/** Owns lifecycle for one background task and consumes its executor traffic. */ -export async function taskRunWorkflow(input: TaskRunWorkflowInput): Promise { - "use workflow"; - - const commands = createHook({ token: input.taskInboxToken }); - const workflowToolRunInbox = openWorkflowToolRunOwnerInbox(); - const readers = [workflowToolRunInbox.reader, createChannelReader("commands", commands)] as const; - let view = input.initialView; - let dispatchAcknowledged = false; - let dispatchRejected = false; - let pendingInputRequest: WorkflowToolRunTaskInputRequest | undefined; - let pendingUpdates: TaskInboundUpdate[] = []; - let updateIndex = 0; - const pendingTraffic: PendingWorkflowToolTraffic = { messages: [], ownerRequests: [] }; - const answerHooks = new Map(); - const bodyController = new AbortController(); - let bodyReader: - | import("#execution/tools/workflow/owner-channels.js").ChannelReader< - "body", - WorkflowBodyResult - > - | undefined; - let executorSettled = input.workflow === undefined; - let pendingBodyResult: WorkflowBodyResult | undefined; - - try { - await claimHookOwnership(commands); - } catch (error) { - if (isHookConflictError(error)) return; - throw error; - } - - await appendTaskViewStep({ activityObserver: input.activityObserver, view }); - while (true) { - // Hook persistence does not mean the owner has consumed every report yet. - if (pendingBodyResult !== undefined && updateIndex >= pendingBodyResult.reportCount) { - executorSettled = true; - await applyPayload({ - command: workflowToolRunOutcomeToTaskCommand({ - from: createWorkflowBodyRef({ ...input.workflow!, execution: "background" }), - result: pendingBodyResult.outcome, - }), - kind: "task-command", - }); - pendingBodyResult = undefined; - if (view.status === "cancelled" && dispatchAcknowledged && !dispatchRejected) { - await wakeTaskParentStep({ token: input.parentContinuationToken, view }); - } - } - if (isFinished()) break; - const read = await raceChannelReads( - bodyReader === undefined ? readers : [...readers, bodyReader], - ); - if (read.channel === "body") { - bodyReader = undefined; - if (read.next.done) continue; - pendingBodyResult = read.next.value; - continue; - } - if (read.next.done) return; - - if (read.channel === "workflow") { - const message = read.next.value; - if (message.kind === "report") { - await applyPayload(workflowToolRunReportToTaskPayload(message, view.taskId, updateIndex++)); - continue; - } - if (message.kind === "outcome") { - await applyPayload({ - command: workflowToolRunOutcomeToTaskCommand(message), - kind: "task-command", - }); - continue; - } - const request = message; - const kind = request.request.kind; - if (kind === "agent-invoke" || kind === "agent-settled" || kind === "authorization-request") { - await handleOwnerRequest(request); - continue; - } - if (request.requestCoordinates === undefined) { - answerHooks.set(request.replyTo, { runId: request.from.runId }); - } - await applyPayload(workflowToolRunRequestToTaskInputRequest(request)); - continue; - } - - await applyPayload(read.next.value); - } - - function isFinished(): boolean { - return isTerminalTaskStatus(view.status) && dispatchAcknowledged && executorSettled; - } - - async function handleUpdate(update: TaskInboundUpdate): Promise { - if (view.metadata.kind === "subagent") { - if (dispatchRejected) return; - if (dispatchAcknowledged && !isTerminalTaskStatus(view.status)) { - await wakeTaskUpdateParentStep({ token: input.parentContinuationToken, update, view }); - } else { - pendingUpdates.push(update); - } - return; - } - await appendTaskProgressStep({ - progress: { - callId: update.callId, - kind: "task-progress", - taskId: view.taskId, - update: update.message, - updateIndex: update.updateIndex, - }, - }); - } - - async function handleMessage(message: TaskInboundMessage): Promise { - if (dispatchRejected || isTerminalTaskStatus(view.status)) return; - if (!dispatchAcknowledged) { - pendingTraffic.messages.push(message); - return; - } - await wakeTaskMessageParentStep({ - message, - taskId: view.taskId, - token: input.parentContinuationToken, - }); - } - - async function applyPayload( - payload: TaskRunInboundPayload | WorkflowToolRunTaskInputRequest, - ): Promise { - const isReady = payload.kind === "task-command" && payload.command.kind === "ready"; - const isRejected = - payload.kind === "task-command" && payload.command.kind === "reject-dispatch"; - if (isReady || isRejected) dispatchAcknowledged = true; - if (isRejected) dispatchRejected = true; - if (isReady || isRejected) { - if (isRejected || isTerminalTaskStatus(view.status)) { - executorSettled = true; - } else if ( - input.workflow !== undefined && - bodyReader === undefined && - pendingBodyResult === undefined && - !executorSettled - ) { - bodyReader = createChannelReader( - "body", - awaitBodyResult( - executeWorkflowBody( - { - ...input.workflow, - execution: "background", - owner: workflowToolRunInbox.owner, - }, - bodyController.signal, - ), - ), - ); - } - await flushPendingTraffic(); - } - - if (payload.kind === "task-input-request") pendingInputRequest = payload; - if (payload.kind === "task-message") { - await handleMessage(payload); - return; - } - if (payload.kind === "task-update") { - await handleUpdate(payload); - return; - } - - let command: TaskCommand | undefined; - if (payload.kind === "input-response") { - command = - view.status === "input_required" - ? await resolveAnsweredCommand( - view, - payload, - answerHooks.get(payload.childContinuationToken), - ) - : undefined; - } else if (payload.kind === "task-input-request") { - command = { - inputRequests: payload.requests ?? [payload.request], - kind: "require-input", - }; - } else if (payload.kind === "task-command") { - command = payload.command; - } else { - return; - } - if (command === undefined) return; - if (isReady && isTerminalTaskStatus(view.status)) { - await flushUpdates(true); - await wakeTaskParentStep({ token: input.parentContinuationToken, view }); - return; - } - - const previous = view; - const accepted = await transitionTask(command); - if (!accepted) return; - if (command.kind === "cancel") { - bodyController.abort(new Error(`Task ${view.taskId} was cancelled.`)); - if (bodyReader === undefined) executorSettled = true; - } - if (!isTerminalTaskStatus(view.status)) await flushUpdates(); - if ( - pendingInputRequest !== undefined && - dispatchAcknowledged && - view.status === "input_required" - ) { - await wakeWorkflowTaskInputRequestParentStep({ - request: pendingInputRequest, - taskId: view.taskId, - token: input.parentContinuationToken, - }); - pendingInputRequest = undefined; - } else if ( - !dispatchRejected && - dispatchAcknowledged && - (command.kind !== "cancel" || executorSettled) && - ((!isTerminalTaskStatus(previous.status) && isTerminalTaskStatus(view.status)) || - (!isReadyTaskStatus(previous.status) && - isReadyTaskStatus(view.status) && - pendingInputRequest === undefined)) - ) { - await wakeTaskParentStep({ token: input.parentContinuationToken, view }); - } - if (view.status !== "input_required") pendingInputRequest = undefined; - } - - async function flushUpdates(includeTerminal = false): Promise { - if (!dispatchAcknowledged || (isTerminalTaskStatus(view.status) && !includeTerminal)) return; - for (const update of pendingUpdates) { - await wakeTaskUpdateParentStep({ token: input.parentContinuationToken, update, view }); - } - pendingUpdates = []; - } - - async function transitionTask(command: TaskCommand): Promise { - const result = applyTaskTransition(view, command); - if (result.action !== "accepted") return false; - view = result.view; - await appendTaskViewStep({ activityObserver: input.activityObserver, view }); - return true; - } - - // Owner traffic must wait until the parent has acknowledged task dispatch. - async function handleOwnerRequest(message: WorkflowToolRunRequestMessage): Promise { - if (!dispatchAcknowledged) { - pendingTraffic.ownerRequests.push(message); - return; - } - const { request, replyTo } = message; - if ( - request.kind === "authorization-request" && - request.event.childSessionId === message.from.runId - ) { - await handleStepAuthorization(request, replyTo); - return; - } - if (dispatchRejected || isTerminalTaskStatus(view.status)) { - return; - } - if (request.kind === "authorization-request") { - await wakeTaskAuthorizationParentStep({ - request, - taskId: view.taskId, - token: input.parentContinuationToken, - }); - return; - } - await wakeTaskAgentRequestParentStep({ - request: message, - taskId: view.taskId, - token: input.parentContinuationToken, - }); - } - - async function handleStepAuthorization( - request: WorkflowToolAuthorizationRequest, - replyTo: string, - ): Promise { - const event = request.event.event; - const closesDisplayedPrompt = event.type === "authorization.completed"; - const canForward = - !dispatchRejected && (!isTerminalTaskStatus(view.status) || closesDisplayedPrompt); - - if (canForward) { - const requestId = "attemptId" in event.data ? event.data.attemptId : undefined; - if (requestId !== undefined && event.type === "authorization.required") { - const existingRequests = view.status === "input_required" ? view.inputRequests : []; - await transitionTask({ - kind: "require-input", - inputRequests: [ - ...existingRequests, - { kind: "authorization", requestId, name: event.data.name }, - ], - }); - } else if (requestId !== undefined) { - await transitionTask({ kind: "answered", requestIds: [requestId] }); - } - - await wakeTaskAuthorizationParentStep({ - request, - taskId: view.taskId, - token: input.parentContinuationToken, - }); - } - - // Discarded events are acknowledged too; persistence and delivery failures are not. - await resumeHookStep(replyTo, null, { ifPresent: true }); - } - - async function flushPendingTraffic(): Promise { - if (!dispatchRejected) { - for (const message of pendingTraffic.messages) { - await wakeTaskMessageParentStep({ - message, - taskId: view.taskId, - token: input.parentContinuationToken, - }); - } - } - for (const request of pendingTraffic.ownerRequests) { - await handleOwnerRequest(request); - } - pendingTraffic.messages.length = 0; - pendingTraffic.ownerRequests.length = 0; - } -} - -async function* awaitBodyResult( - result: Promise, -): AsyncGenerator { - yield await result; -} - -async function resolveAnsweredCommand( - view: Extract, - answer: TaskInboundAnswerInput, - answerHook: AnswerHookRoute | undefined, -): Promise { - if (answer.taskId !== view.taskId) return undefined; - const outstanding = new Set( - view.inputRequests.flatMap((request) => { - const requestId = readTaskInputRequestId(request); - return requestId === undefined ? [] : [requestId]; - }), - ); - const requestIds = answer.inputResponses - .map((response) => response.requestId) - .filter((id) => outstanding.has(id)); - if (requestIds.length === 0) return undefined; - return (await deliverTaskInputResponsesStep({ answer, answerHook, requestIds })) === "delivered" - ? { kind: "answered", requestIds } - : undefined; -} diff --git a/packages/eve/src/execution/tasks/parent/cancel-notification.integration.test.ts b/packages/eve/src/execution/tasks/parent/cancel-notification.integration.test.ts index bc1f554cd..29fc7cab6 100644 --- a/packages/eve/src/execution/tasks/parent/cancel-notification.integration.test.ts +++ b/packages/eve/src/execution/tasks/parent/cancel-notification.integration.test.ts @@ -1,3 +1,4 @@ +import { WORKFLOW_CANCELLATION_SETTLE_MS } from "#execution/tools/workflow/cancellation-policy.js"; import { describe, expect, it } from "vitest"; import { createTestRuntime } from "#internal/testing/app-harness.js"; @@ -5,26 +6,30 @@ import { taskCancelNotificationWorkflow } from "#internal/testing/task-cancel-no import { start } from "#internal/workflow/runtime.js"; describe("task cancellation parent notification", () => { - it("delivers the committed view from the parent step after forcing a slow lifecycle to stop", async () => { - const runtime = await createTestRuntime({ agent: { name: "task-cancel-notification" } }); - await runtime.run(async () => { - const run = await start(taskCancelNotificationWorkflow, []); - try { - const result = await run.returnValue; - expect(result.taskRunStatus).toBe("cancelled"); - expect(result.view.status).toBe("cancelled"); - expect(result.notification).toMatchObject({ - kind: "send", - payload: { - message: `Background task ${result.view.taskId} (slow-cancel) is cancelled.`, - task: { views: [result.view] }, - }, - taskDeliveryId: `${result.view.taskId}:ready:cancelled`, - }); - } finally { - const status = await run.status; - if (status === "pending" || status === "running") await run.cancel(); - } - }); - }, 30_000); + it( + "delivers cancellation from the parent without a child task-view stream", + async () => { + const runtime = await createTestRuntime({ agent: { name: "task-cancel-notification" } }); + await runtime.run(async () => { + const run = await start(taskCancelNotificationWorkflow, []); + try { + const result = await run.returnValue; + expect(result.taskRunStatus).toBe("cancelled"); + expect(result.view.status).toBe("cancelled"); + expect(result.notification).toMatchObject({ + kind: "send", + payload: { + message: `Background task ${result.view.taskId} (slow-cancel) is cancelled.`, + task: { views: [result.view] }, + }, + taskDeliveryId: `${result.view.taskId}:ready:cancelled`, + }); + } finally { + const status = await run.status; + if (status === "pending" || status === "running") await run.cancel(); + } + }); + }, + WORKFLOW_CANCELLATION_SETTLE_MS + 15_000, + ); }); diff --git a/packages/eve/src/execution/tasks/parent/control-shared.ts b/packages/eve/src/execution/tasks/parent/control-shared.ts index f668d4019..23bee24da 100644 --- a/packages/eve/src/execution/tasks/parent/control-shared.ts +++ b/packages/eve/src/execution/tasks/parent/control-shared.ts @@ -1,9 +1,10 @@ import type { HarnessSession as RuntimeSession } from "#harness/types.js"; -import { readLatestTaskView } from "#execution/tasks/parent/run-parent.js"; -import { isTaskWorkflowTargetGone } from "#execution/tasks/workflow-target.js"; import type { RuntimeActionResult, RuntimeToolCallActionRequest } from "#shared/action-types.js"; import { taskViewsToJson } from "#tasks/json.js"; -import { findSessionTaskEntry, type SessionTaskIndexEntry } from "#tasks/session-index.js"; +import { + findBackgroundWorkflowToolRun, + type BackgroundWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import type { TaskView } from "#tasks/types.js"; /** @@ -17,12 +18,12 @@ export function lookupTaskEntries( session: RuntimeSession, taskIds: readonly string[], ): - | { readonly entries: SessionTaskIndexEntry[]; readonly kind: "found" } + | { readonly entries: BackgroundWorkflowToolRun[]; readonly kind: "found" } | { readonly kind: "unknown"; readonly unknown: string[] } { - const entries: SessionTaskIndexEntry[] = []; + const entries: BackgroundWorkflowToolRun[] = []; const unknown: string[] = []; for (const taskId of taskIds) { - const entry = findSessionTaskEntry(session.state, taskId); + const entry = findBackgroundWorkflowToolRun(session.state, taskId); if (entry === undefined) { unknown.push(taskId); } else { @@ -32,41 +33,6 @@ export function lookupTaskEntries( return unknown.length > 0 ? { kind: "unknown", unknown } : { entries, kind: "found" }; } -/** Reads the latest view of every entry, defaulting to `working`. */ -export async function readTaskViews( - entries: readonly SessionTaskIndexEntry[], -): Promise { - return Promise.all(entries.map(readTaskView)); -} - -export async function readTaskView(entry: SessionTaskIndexEntry): Promise { - try { - return ( - (await readLatestTaskView({ taskRunId: entry.taskRunId })) ?? createPendingTaskView(entry) - ); - } catch (error) { - if (isTaskWorkflowTargetGone(error) && entry.terminalView !== undefined) { - return entry.terminalView; - } - throw error; - } -} - -/** The placeholder view for a run that has not published anything yet. */ -function createPendingTaskView(entry: SessionTaskIndexEntry): TaskView { - const view: TaskView = { - metadata: entry.metadata, - status: "working", - taskId: entry.taskId, - }; - - if (entry.executor === undefined) { - return view; - } - - return { ...view, executor: { binding: entry.executor } }; -} - /** One successful task-control result carrying full task views. */ export function createTaskViewsResult( action: RuntimeToolCallActionRequest, diff --git a/packages/eve/src/execution/tasks/parent/delegate.test.ts b/packages/eve/src/execution/tasks/parent/delegate.test.ts index 21bf61548..84ecfe587 100644 --- a/packages/eve/src/execution/tasks/parent/delegate.test.ts +++ b/packages/eve/src/execution/tasks/parent/delegate.test.ts @@ -4,10 +4,12 @@ import { acknowledgeDelegatedTasksStep } from "#execution/tasks/parent/delegate. import { sendTaskCommandToOwner } from "#execution/tasks/parent/run-parent.js"; vi.mock("#execution/tasks/parent/run-parent.js", () => ({ - readLatestTaskView: vi.fn(), sendTaskCommandToOwner: vi.fn(), })); +const mocks = vi.hoisted(() => ({ getRun: vi.fn() })); +vi.mock("#internal/workflow/runtime.js", () => mocks); + describe("task readiness", () => { beforeEach(() => { vi.resetAllMocks(); @@ -22,4 +24,17 @@ describe("task readiness", () => { expect.objectContaining({ command: { kind: "ready" }, taskInboxToken: "task-token" }), ); }); + it.each(["completed", "cancelled", "failed", "running"] as const)( + "handles a missing readiness hook when the run is %s", + async (status) => { + vi.mocked(sendTaskCommandToOwner).mockResolvedValue(undefined); + mocks.getRun.mockReturnValue({ status: Promise.resolve(status) }); + const ready = acknowledgeDelegatedTasksStep({ + tasks: [{ taskId: "task-1", taskInboxToken: "task-token", taskRunId: "run-1" }], + }); + if (status === "completed" || status === "cancelled") + await expect(ready).resolves.toBeUndefined(); + else await expect(ready).rejects.toThrow("did not accept its readiness command"); + }, + ); }); diff --git a/packages/eve/src/execution/tasks/parent/delegate.ts b/packages/eve/src/execution/tasks/parent/delegate.ts index b04ba5acd..b07b117e8 100644 --- a/packages/eve/src/execution/tasks/parent/delegate.ts +++ b/packages/eve/src/execution/tasks/parent/delegate.ts @@ -1,24 +1,17 @@ /** - * Generic task creation, readiness acknowledgement, and dispatch rejection. - * Task-run transport (start/command/view) lives in `run-parent.ts`, which - * Callers compose these primitives around their own executor policy. + * Session-owned invocation identity, admission acknowledgement, and dispatch rejection. + * The parent commits its session index before releasing the workflow body. */ -import type { ActivityObserverConfig } from "#channel/types.js"; +import { getRun } from "#internal/workflow/runtime.js"; import type { HarnessSession } from "#harness/types.js"; -import type { ActivityWorkIdentityV1 } from "#protocol/activity.js"; -import { - readLatestTaskView, - sendTaskCommand, - sendTaskCommandToOwner, - startTaskRun, - waitForTaskCommandOwner, -} from "#execution/tasks/parent/run-parent.js"; -import { sessionCommandHookToken } from "#execution/session-inbox/address.js"; +import type { + BackgroundWorkflowToolRun, + TaskAgentDispatchContext, +} from "#harness/workflow-tool-runs.js"; +import { sendTaskCommand, sendTaskCommandToOwner } from "#execution/tasks/parent/run-parent.js"; import type { JsonValue } from "#shared/json.js"; -import type { TaskExecutorBinding } from "#tools/task.js"; import { deriveTaskInboxToken, deriveTaskId } from "#tasks/task-id.js"; -import { isTerminalTaskStatus, type TaskMetadata } from "#tasks/types.js"; -import type { TaskAgentDispatchContext } from "#tasks/session-index.js"; +import type { TaskMetadata } from "#tasks/types.js"; import type { ContextReader } from "#context/key.js"; import { SessionDynamicSubagentSelectionsKey, @@ -26,19 +19,6 @@ import { type SessionAuth, } from "#context/keys.js"; -/** A prepared background task: identity plus its started durable run. */ -export interface BackgroundTask { - readonly activityWorkIdentity?: ActivityWorkIdentityV1; - readonly dispatchContext: TaskAgentDispatchContext; - readonly taskInboxToken: string; - readonly createdByStepIndex?: number; - readonly createdByTurnId: string; - readonly executor?: TaskExecutorBinding; - readonly metadata: TaskMetadata; - readonly taskId: string; - readonly taskRunId: string; -} - export function createTaskAgentDispatchContext( ctx: ContextReader, auth: SessionAuth, @@ -50,7 +30,9 @@ export function createTaskAgentDispatchContext( }; } -type BackgroundTaskDraft = Omit; +export type BackgroundTaskDraft = Omit & { + readonly address: { readonly hookToken: string }; +}; /** Derives the replay-stable task identity before its owning run is started. */ export function prepareBackgroundTask(input: { @@ -68,40 +50,20 @@ export function prepareBackgroundTask(input: { parentTurnId: input.parentTurnId, }); return { - taskInboxToken: deriveTaskInboxToken({ - parentContinuationToken: input.session.continuationToken, - taskId, - }), - createdByStepIndex: input.parentStepIndex ?? 0, - createdByTurnId: input.parentTurnId, - dispatchContext: input.dispatchContext, - metadata: input.metadata, - taskId, + callId: input.callId, + toolName: input.metadata.name, + lifetime: "session", + origin: { turnId: input.parentTurnId, stepIndex: input.parentStepIndex ?? 0 }, + address: { + hookToken: deriveTaskInboxToken({ + parentContinuationToken: input.session.continuationToken, + taskId, + }), + }, + task: { dispatchContext: input.dispatchContext, metadata: input.metadata, taskId }, }; } -/** Starts a lifecycle-only task run for a non-workflow external executor. */ -export async function beginBackgroundTask(input: { - readonly activityObserver?: ActivityObserverConfig; - readonly callId: string; - readonly dispatchContext: TaskAgentDispatchContext; - readonly metadata: TaskMetadata; - readonly parentSessionId: string; - readonly parentStepIndex?: number; - readonly parentTurnId: string; - readonly session: HarnessSession; -}): Promise { - const task = prepareBackgroundTask(input); - await startTaskRun({ - activityObserver: input.activityObserver, - taskInboxToken: task.taskInboxToken, - initialView: { metadata: task.metadata, status: "working", taskId: task.taskId }, - parentContinuationToken: sessionCommandHookToken(input.session.sessionId), - }); - const owner = await waitForTaskCommandOwner({ taskInboxToken: task.taskInboxToken }); - return { ...task, taskRunId: owner.runId }; -} - /** Releases task events only after the parent session index committed. */ export async function acknowledgeDelegatedTasksStep(input: { readonly tasks: readonly { @@ -119,8 +81,8 @@ export async function acknowledgeDelegatedTasksStep(input: { retryUnreachable: { attempts: 20, delayMs: 250 }, }); if (owner !== undefined) continue; - const view = await readLatestTaskView({ taskRunId: task.taskRunId }); - if (view !== undefined && isTerminalTaskStatus(view.status)) continue; + const status = await getRun(task.taskRunId).status; + if (status === "completed" || status === "cancelled") continue; throw new Error(`Task run "${task.taskId}" did not accept its readiness command.`); } } @@ -128,11 +90,11 @@ export async function acknowledgeDelegatedTasksStep(input: { /** Silently terminates a task whose child dispatch failed before parent indexing. */ export async function rejectDelegatedDispatch(input: { readonly error: JsonValue; - readonly task: BackgroundTask; + readonly task: BackgroundWorkflowToolRun; }): Promise { await sendTaskCommand({ command: { data: input.error, kind: "reject-dispatch" }, - taskInboxToken: input.task.taskInboxToken, + taskInboxToken: input.task.address.hookToken, retryUnreachable: { attempts: 20, delayMs: 250 }, }); } diff --git a/packages/eve/src/execution/tasks/parent/dispatch.test.ts b/packages/eve/src/execution/tasks/parent/dispatch.test.ts index b21f7fba4..38865401f 100644 --- a/packages/eve/src/execution/tasks/parent/dispatch.test.ts +++ b/packages/eve/src/execution/tasks/parent/dispatch.test.ts @@ -1,8 +1,17 @@ +import { + findBackgroundWorkflowToolRun, + readWorkflowTaskView, +} from "#harness/workflow-tool-runs.js"; +import type { HarnessSession } from "#harness/types.js"; +import { WORKFLOW_CANCELLATION_SETTLE_MS } from "#execution/tools/workflow/cancellation-policy.js"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { cancelOwnedTask, isTaskControlAction } from "#execution/tasks/parent/dispatch.js"; -import { readLatestTaskView, sendTaskCommand } from "#execution/tasks/parent/run-parent.js"; -import { cancelWorkflowToolRun } from "#execution/tools/workflow/cancel.js"; +import { + cancelOwnedTask, + executeTaskControlAction, + isTaskControlAction, +} from "#execution/tasks/parent/dispatch.js"; +import { sendTaskCommand } from "#execution/tasks/parent/run-parent.js"; import { resumeSessionInbox } from "#execution/session-inbox/resume.js"; const { cancelRun, getRun } = vi.hoisted(() => ({ @@ -11,10 +20,8 @@ const { cancelRun, getRun } = vi.hoisted(() => ({ })); vi.mock("#execution/tasks/parent/run-parent.js", () => ({ - readLatestTaskView: vi.fn(), sendTaskCommand: vi.fn(), })); -vi.mock("#execution/tools/workflow/cancel.js", () => ({ cancelWorkflowToolRun: vi.fn() })); vi.mock("#execution/session-inbox/resume.js", () => ({ resumeSessionInbox: vi.fn() })); vi.mock("#internal/workflow/runtime.js", () => ({ cancelRun, @@ -23,13 +30,16 @@ vi.mock("#internal/workflow/runtime.js", () => ({ })); const entry = { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - executor: { data: { hookToken: "run-hook", runId: "run-1" }, kind: "workflow-tool" }, - metadata: { kind: "tool", name: "export" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task-1", + toolName: "export", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "export" }, + taskId: "task-1", + }, } as const; describe("task cancellation", () => { @@ -44,13 +54,48 @@ describe("task cancellation", () => { vi.useRealTimers(); }); - it("cancels task-owned work after cancellation commits", async () => { - vi.mocked(readLatestTaskView).mockResolvedValue({ - executor: { binding: entry.executor }, - metadata: entry.metadata, - status: "cancelled", - taskId: entry.taskId, + it("cancels a live task despite an unrelated malformed retained result", async () => { + const session: HarnessSession = { + agent: { modelReference: { id: "test" }, system: "", tools: [] }, + compaction: { recentWindowSize: 4, threshold: 100_000 }, + history: [], + continuationToken: "parent", + sessionId: "parent-session", + state: { + "eve.workflowTool": { + version: 3, + runs: [ + { + ...entry, + callId: "old", + task: { ...entry.task, taskId: "old", outcome: { status: "completed" } }, + }, + entry, + ], + }, + }, + }; + const cancelled = await executeTaskControlAction({ + action: { + kind: "tool-call", + callId: "cancel", + toolName: "task_cancel", + input: { taskIds: [entry.task.taskId] }, + }, + session, }); + const recorded = findBackgroundWorkflowToolRun(cancelled.session.state, entry.task.taskId); + expect(recorded).toBeDefined(); + expect(recorded === undefined ? undefined : readWorkflowTaskView(recorded.task)).toMatchObject({ + status: "cancelled", + }); + expect(sendTaskCommand).toHaveBeenCalledExactlyOnceWith({ + command: { kind: "cancel" }, + taskInboxToken: entry.address.hookToken, + }); + }); + + it("signals cancellation before stopping task-owned work", async () => { const cancelled = cancelOwnedTask({ entry }); await vi.runAllTimersAsync(); await cancelled; @@ -58,34 +103,11 @@ describe("task cancellation", () => { command: { kind: "cancel" }, taskInboxToken: "task-token", }); - expect(cancelWorkflowToolRun).toHaveBeenCalledWith( - { hookToken: "run-hook", runId: "run-1" }, - "Task task-1 was cancelled.", - ); expect(cancelRun).not.toHaveBeenCalled(); expect(resumeSessionInbox).not.toHaveBeenCalled(); }); - it("does not reinterpret an unknown executor binding", async () => { - const external = { data: { id: "external" }, kind: "external" }; - vi.mocked(readLatestTaskView).mockResolvedValue({ - executor: { binding: external }, - metadata: entry.metadata, - status: "cancelled", - taskId: entry.taskId, - }); - const cancelled = cancelOwnedTask({ entry: { ...entry, executor: external } }); - await vi.runAllTimersAsync(); - await cancelled; - expect(cancelWorkflowToolRun).not.toHaveBeenCalled(); - }); - it("retries child cancellation after the cancelled task's inbox has closed", async () => { - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: entry.metadata, - status: "cancelled", - taskId: entry.taskId, - }); const cancelOwnedWork = vi .fn() .mockRejectedValueOnce(new Error("Child cancellation failed")) @@ -110,24 +132,36 @@ describe("task cancellation", () => { }); it("leaves child work untouched when completion won the cancellation race", async () => { - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: entry.metadata, - lastOutput: { type: "result", data: "Finished" }, - status: "completed", - taskId: entry.taskId, - }); const cancelOwnedWork = vi.fn(); - await cancelOwnedTask({ cancelOwnedWork, entry }); + await cancelOwnedTask({ + cancelOwnedWork, + entry: { + ...entry, + task: { + ...entry.task, + outcome: { + lastOutput: { type: "result", data: "Finished" }, + status: "completed", + }, + }, + }, + }); expect(cancelOwnedWork).not.toHaveBeenCalled(); - expect(cancelWorkflowToolRun).not.toHaveBeenCalled(); + }); + + it("allows cleanup lasting longer than one second without force-stopping or duplicating delivery", async () => { + getRun.mockReturnValue({ status: Promise.resolve("running") }); + const cancelled = cancelOwnedTask({ entry }); + await vi.advanceTimersByTimeAsync(2_000); + expect(cancelRun).not.toHaveBeenCalled(); + getRun.mockReturnValue({ status: Promise.resolve("completed") }); + await vi.advanceTimersByTimeAsync(250); + await cancelled; + expect(cancelRun).not.toHaveBeenCalled(); + expect(resumeSessionInbox).not.toHaveBeenCalled(); }); it("hard-cancels a task run that does not unwind cooperatively", async () => { - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: entry.metadata, - status: "cancelled", - taskId: entry.taskId, - }); getRun.mockReturnValue({ status: Promise.resolve("running") }); const cancelled = cancelOwnedTask({ entry }); @@ -140,15 +174,18 @@ describe("task cancellation", () => { }); it("preserves the committed parent notification when cancellation stops a slow task run", async () => { - const view = { metadata: entry.metadata, status: "cancelled", taskId: entry.taskId } as const; - vi.mocked(readLatestTaskView).mockResolvedValue(view); + const view = { + metadata: entry.task.metadata, + status: "cancelled", + taskId: entry.task.taskId, + } as const; getRun.mockReturnValue({ status: Promise.resolve("running") }); const session = { sessionId: "parent-session" } as Parameters< typeof cancelOwnedTask >[0]["session"]; const cancelled = cancelOwnedTask({ entry, session }); - await vi.advanceTimersByTimeAsync(999); + await vi.advanceTimersByTimeAsync(WORKFLOW_CANCELLATION_SETTLE_MS - 1); expect(cancelRun).not.toHaveBeenCalled(); expect(resumeSessionInbox).not.toHaveBeenCalled(); await vi.advanceTimersByTimeAsync(1); @@ -169,8 +206,11 @@ describe("task cancellation", () => { }); it("retries the parent notification after the cancelled task inbox is gone", async () => { - const view = { metadata: entry.metadata, status: "cancelled", taskId: entry.taskId } as const; - vi.mocked(readLatestTaskView).mockResolvedValue(view); + const view = { + metadata: entry.task.metadata, + status: "cancelled", + taskId: entry.task.taskId, + } as const; getRun.mockReturnValue({ status: Promise.resolve("running") }); vi.mocked(resumeSessionInbox).mockRejectedValueOnce(new Error("temporary delivery failure")); const session = { sessionId: "parent-session" } as Parameters< @@ -185,7 +225,6 @@ describe("task cancellation", () => { getRun.mockReturnValue({ status: Promise.resolve("cancelled") }); await expect(cancelOwnedTask({ entry, session })).resolves.toEqual(view); expect(cancelRun).toHaveBeenCalledTimes(1); - expect(cancelWorkflowToolRun).toHaveBeenCalledTimes(2); expect(resumeSessionInbox).toHaveBeenCalledTimes(2); expect(vi.mocked(resumeSessionInbox).mock.calls[1]).toEqual( vi.mocked(resumeSessionInbox).mock.calls[0], diff --git a/packages/eve/src/execution/tasks/parent/dispatch.ts b/packages/eve/src/execution/tasks/parent/dispatch.ts index eceb86760..2e50b7143 100644 --- a/packages/eve/src/execution/tasks/parent/dispatch.ts +++ b/packages/eve/src/execution/tasks/parent/dispatch.ts @@ -4,24 +4,23 @@ import { createTaskViewsResult, createUnknownTasksError, lookupTaskEntries, - readTaskView, } from "#execution/tasks/parent/control-shared.js"; -import type { BackgroundTask } from "#execution/tasks/parent/delegate.js"; +import { + recordWorkflowTaskView, + readWorkflowTaskView, + type BackgroundWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import { sendTaskCommand } from "#execution/tasks/parent/run-parent.js"; -import { wakeTaskParentStep } from "#execution/tasks/child/steps.js"; +import { notifyTaskParent } from "#execution/tasks/child/notify.js"; import { sessionCommandHookToken } from "#execution/session-inbox/address.js"; import { cancelTaskOwnedWork, type TaskExecutorCancel, } from "#execution/tasks/parent/task-cancel.js"; import type { RuntimeActionResult, RuntimeToolCallActionRequest } from "#shared/action-types.js"; -import type { SessionTaskIndexEntry } from "#tasks/session-index.js"; -import { isTerminalTaskStatus, type TaskView } from "#tasks/types.js"; +import { type TaskView } from "#tasks/types.js"; import { TASK_CANCEL_TOOL_NAME, TASK_TOOL_NAMES } from "#tools/framework/task-contract.js"; -const CANCEL_COMMIT_POLL_ATTEMPTS = 10; -const CANCEL_COMMIT_POLL_DELAY_MS = 250; - export function isTaskControlAction(action: RuntimeToolCallActionRequest): boolean { return action.kind === "tool-call" && TASK_TOOL_NAMES.has(action.toolName); } @@ -34,9 +33,9 @@ export async function executeTaskControlAction(input: { }): Promise<{ readonly result: RuntimeActionResult; readonly session: RuntimeSession; - readonly pendingTask?: BackgroundTask; }> { - const { action, session } = input; + const { action } = input; + let session = input.session; const taskIds = readTaskIds(action.input); if (taskIds === undefined || taskIds.length === 0) { return { @@ -57,56 +56,48 @@ export async function executeTaskControlAction(input: { const views: TaskView[] = []; for (const entry of lookup.entries) { - views.push( - await cancelOwnedTask({ - cancelOwnedWork: input.cancelOwnedWork, - entry, - serializedContext: input.serializedContext, - session, - }), - ); + const view = await cancelOwnedTask({ + cancelOwnedWork: input.cancelOwnedWork, + entry, + serializedContext: input.serializedContext, + session, + }); + session = { ...session, state: recordWorkflowTaskView(session.state, view) }; + views.push(view); } return { result: createTaskViewsResult(action, views), session }; } -/** Commits cancellation, then stops task-owned child work and its lifecycle run. */ +/** Cancels task-owned work; the caller records the outcome in the parent session. */ export async function cancelOwnedTask(input: { readonly cancelOwnedWork?: TaskExecutorCancel; - readonly entry: SessionTaskIndexEntry; + readonly entry: BackgroundWorkflowToolRun; readonly serializedContext?: Record; readonly session?: RuntimeSession; }): Promise { - const delivery = await sendTaskCommand({ + const previous = readWorkflowTaskView(input.entry.task); + if (previous !== undefined && previous.status !== "cancelled") return previous; + const view: TaskView = previous ?? { + metadata: input.entry.task.metadata, + status: "cancelled", + taskId: input.entry.task.taskId, + }; + await sendTaskCommand({ command: { kind: "cancel" }, - taskInboxToken: input.entry.taskInboxToken, + taskInboxToken: input.entry.address.hookToken, }); - let view = await readTaskView(input.entry); - for ( - let attempt = 0; - attempt < CANCEL_COMMIT_POLL_ATTEMPTS && !isTerminalTaskStatus(view.status); - attempt += 1 - ) { - await new Promise((resolve) => setTimeout(resolve, CANCEL_COMMIT_POLL_DELAY_MS)); - view = await readTaskView(input.entry); - } - if (!isTerminalTaskStatus(view.status)) { - throw new Error(`Task "${input.entry.taskId}" did not commit cancellation before timeout.`); - } - if (view.status !== "cancelled") return view; - // The task inbox may be closed after an earlier cancellation committed but // failed to stop its child. Retrying must still finish that cancellation. - const forcedShutdown = await cancelTaskOwnedWork({ + await cancelTaskOwnedWork({ cancelOwnedWork: input.cancelOwnedWork, entry: input.entry, serializedContext: input.serializedContext, session: input.session, }); - if ((delivery === "unreachable" || forcedShutdown) && input.session !== undefined) { - // Forced shutdown can interrupt the lifecycle between its committed view - // and parent wake. Retried cancellation must finish delivery even when the - // inbox is gone; the shared delivery id deduplicates a wake already sent. - await wakeTaskParentStep({ + if (input.session !== undefined) { + // Queue settlement even when the child cannot report. The parent records + // this control result before routing queued notifications; late outcomes lose. + await notifyTaskParent({ token: sessionCommandHookToken(input.session.sessionId), view, }); diff --git a/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.test.ts b/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.test.ts index f0f74dbb4..b70c063e5 100644 --- a/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.test.ts +++ b/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.test.ts @@ -1,6 +1,9 @@ +import { + recordWorkflowTaskView, + getBackgroundWorkflowToolRuns, +} from "#harness/workflow-tool-runs.js"; import { createTestSessionState } from "#internal/testing/session-state.js"; import { beforeEach, describe, expect, it, vi } from "vitest"; - import { ContextContainer } from "#context/container.js"; import { serializeContext } from "#context/serialize.js"; import { readDurableSession } from "#execution/durable-session-store.js"; @@ -8,13 +11,14 @@ import { recordTerminalTaskViewsStep, recordTaskInputRequestStep, } from "#execution/tasks/parent/hitl-proxy-steps.js"; -import { readLatestTaskView } from "#execution/tasks/parent/run-parent.js"; import { bindSessionInstrumentation } from "#instrumentation/runtime.js"; import { BundleKey } from "#runtime/sessions/runtime-context-keys.js"; import { getCompiledRuntimeAgentBundle } from "#runtime/sessions/compiled-agent-cache.js"; import { getAgentHandleStore, setAgentHandleStore } from "#subagents/handles/store.js"; -import { getProxyInputRequests } from "#harness/proxy-input-requests.js"; -import { getSessionTaskIndex } from "#tasks/session-index.js"; +import { + getProxyInputRequests, + upsertProxyInputRequestState, +} from "#harness/proxy-input-requests.js"; const flushInstrumentation = vi.hoisted(() => vi.fn()); const publishBackgroundTaskSettlements = vi.hoisted(() => vi.fn()); @@ -23,7 +27,6 @@ vi.mock("#execution/durable-session-store.js", async (importOriginal) => ({ ...(await importOriginal()), readDurableSession: vi.fn(), })); -vi.mock("#execution/tasks/parent/run-parent.js", () => ({ readLatestTaskView: vi.fn() })); vi.mock("#instrumentation/runtime.js", () => ({ bindSessionInstrumentation: vi.fn(), })); @@ -74,31 +77,28 @@ describe("recordTaskInputRequestStep", () => { history: [], sessionId: "parent-session", state: { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "tool", name: "export" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task-1", + toolName: "export", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "export" }, + taskId: "task-1", + }, }, ], - version: 2, }, }, }); }); - it("records a generic workflow answer route after matching the task view", async () => { - vi.mocked(readLatestTaskView).mockResolvedValue({ - inputRequests: [request.request], - metadata: { kind: "tool", name: "export" }, - status: "input_required", - taskId: "task-1", - }); - + it("records a generic workflow answer route for a parent-owned task", async () => { const result = await recordTaskInputRequestStep({ request, sessionState }); expect(result).toMatchObject({ @@ -116,12 +116,16 @@ describe("recordTaskInputRequestStep", () => { }); }); - it("rejects a request that does not match the task's outstanding batch", async () => { - vi.mocked(readLatestTaskView).mockResolvedValue({ - inputRequests: [{ ...request.request, requestId: "other" }], - metadata: { kind: "tool", name: "export" }, - status: "input_required", - taskId: "task-1", + it("rejects a late input request after parent settlement", async () => { + const session = readDurableSession(sessionState); + vi.mocked(readDurableSession).mockReturnValue({ + ...session, + state: recordWorkflowTaskView(session.state, { + lastOutput: { data: "done", type: "result" }, + metadata: { kind: "tool", name: "export" }, + status: "completed", + taskId: "task-1", + }), }); await expect(recordTaskInputRequestStep({ request, sessionState })).resolves.toEqual({ @@ -138,18 +142,22 @@ describe("recordTaskInputRequestStep", () => { sessionId: "parent-session", state: setAgentHandleStore( { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "tool", name: "export" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task-1", + toolName: "export", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "export" }, + taskId: "task-1", + }, }, ], - version: 2, }, }, { @@ -175,12 +183,6 @@ describe("recordTaskInputRequestStep", () => { replyTo: remoteReplyTo, request: { ...request.request, requestId: "remote-req" }, }; - vi.mocked(readLatestTaskView).mockResolvedValue({ - inputRequests: [remoteRequest.request], - metadata: { kind: "tool", name: "export" }, - status: "input_required", - taskId: "task-1", - }); const result = await recordTaskInputRequestStep({ request: remoteRequest, sessionState }); @@ -202,7 +204,7 @@ describe("recordTerminalTaskViewsStep", () => { } as never); }); - it("caches an owned terminal view and releases the task's agent lease", async () => { + it("records an owned outcome and releases its agent lease and input routes", async () => { vi.mocked(readDurableSession).mockReturnValue({ agent: { system: "" }, continuationToken: "parent-token", @@ -210,18 +212,28 @@ describe("recordTerminalTaskViewsStep", () => { sessionId: "parent-session", state: setAgentHandleStore( { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { agentId: "agent-1", kind: "subagent", mode: "local", name: "research" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task-1", + toolName: { agentId: "agent-1", kind: "subagent", mode: "local", name: "research" } + .name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { + agentId: "agent-1", + kind: "subagent", + mode: "local", + name: "research", + }, + taskId: "task-1", + }, }, ], - version: 2, }, }, { @@ -248,6 +260,20 @@ describe("recordTerminalTaskViewsStep", () => { taskId: "task-1", }; + const current = readDurableSession(sessionState); + vi.mocked(readDurableSession).mockReturnValue({ + ...current, + state: upsertProxyInputRequestState({ + state: current.state, + forChildContinuationToken: "question-hook", + entries: [ + [ + "task-1:question", + { childContinuationToken: "question-hook", kind: "question", taskId: "task-1" }, + ], + ], + }), + }); const result = await recordTerminalTaskViewsStep({ serializedContext: {}, sessionState, @@ -255,7 +281,12 @@ describe("recordTerminalTaskViewsStep", () => { }); const state = result.sessionState.snapshot.session.state; - expect(getSessionTaskIndex(state)[0]?.terminalView).toEqual(view); + expect(getBackgroundWorkflowToolRuns(state)[0]?.task.outcome).toEqual({ + status: view.status, + lastOutput: view.lastOutput, + }); + expect(getProxyInputRequests(state).size).toBe(0); + expect(result.sessionState.hasProxyInputRequests).toBe(false); expect(getAgentHandleStore(state)?.handles).toEqual([ expect.objectContaining({ phase: "available" }), ]); @@ -278,18 +309,22 @@ describe("recordTerminalTaskViewsStep", () => { history: [], sessionId: "parent-session", state: { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "tool", name: "export" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task-1", + toolName: "export", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "export" }, + taskId: "task-1", + }, }, ], - version: 2, }, }, }); @@ -306,6 +341,7 @@ describe("recordTerminalTaskViewsStep", () => { views: [view], }); + expect(result.subagentCompletions).toEqual([]); expect(bindSessionInstrumentation).toHaveBeenCalledWith({ agentName: "parent-agent", ctx: expect.any(ContextContainer), diff --git a/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.ts b/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.ts index bacab73bc..d0a186efe 100644 --- a/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.ts +++ b/packages/eve/src/execution/tasks/parent/hitl-proxy-steps.ts @@ -1,3 +1,7 @@ +import type { SubagentCompletedStreamEvent } from "#protocol/message.js"; +import { ActivityObserverKey } from "#context/keys.js"; +import { projectTaskActivity } from "#execution/tasks/child/notify.js"; +import { submitActivity } from "#execution/submit-activity.js"; import { contextStorage } from "#context/container.js"; import { deserializeContext, serializeContext } from "#context/serialize.js"; import { @@ -5,10 +9,10 @@ import { readDurableSession, replaceDurableSessionSnapshot, } from "#execution/durable-session-store.js"; -import { readLatestTaskView } from "#execution/tasks/parent/run-parent.js"; import { createTaskInputCapabilityToken } from "#execution/task-input-capability.js"; import { createRemoteTaskInputCallbackUrl } from "#execution/workflow-callback-url.js"; import { + clearProxyInputRequestsForTask, createTaskInputRequestId, upsertProxyInputRequestState, type ProxyInputRequest, @@ -20,7 +24,11 @@ import { isInputRequest } from "#shared/input.js"; import { getAgentHandleStore } from "#subagents/handles/store.js"; import { applyTaskAgentHandleCommand } from "#subagents/handles/transitions.js"; import { createEveTaskInputRoutePath } from "#protocol/routes.js"; -import { cacheTerminalTaskView, findSessionTaskEntry } from "#tasks/session-index.js"; +import { + recordWorkflowTaskView, + readWorkflowTaskView, + findBackgroundWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import type { TaskInputRequestDelivery, TaskView } from "#tasks/types.js"; const log = createLogger("execution.tasks.parent"); @@ -40,24 +48,12 @@ export async function recordTaskInputRequestStep(input: { "use step"; const durableSession = readDurableSession(input.sessionState); - const entry = findSessionTaskEntry(durableSession.state, input.request.taskId); + const entry = findBackgroundWorkflowToolRun(durableSession.state, input.request.taskId); const requests = input.request.requests ?? [input.request.request]; if (entry === undefined || requests.length === 0 || !requests.every(isInputRequest)) { return { accepted: false, sessionState: input.sessionState }; } - const view = await readLatestTaskView({ taskRunId: entry.taskRunId }); - const requestIds = requests.map((request) => request.requestId); - if ( - view?.status !== "input_required" || - view.inputRequests.length !== requestIds.length || - !view.inputRequests.every( - (request, index) => - request !== null && - typeof request === "object" && - !Array.isArray(request) && - Reflect.get(request, "requestId") === requestIds[index], - ) - ) { + if (readWorkflowTaskView(entry.task) !== undefined) { return { accepted: false, sessionState: input.sessionState }; } @@ -105,7 +101,7 @@ export async function recordTaskInputRequestStep(input: { }; } -/** Caches terminal task views before their workflow runs expire. */ +/** Records child outcomes in the parent; its first terminal decision wins. */ export async function recordTerminalTaskViewsStep(input: { readonly serializedContext: Record; readonly sessionState: DurableSessionState; @@ -113,16 +109,36 @@ export async function recordTerminalTaskViewsStep(input: { }): Promise<{ readonly serializedContext: Record; readonly sessionState: DurableSessionState; + readonly views: readonly TaskView[]; + readonly subagentCompletions: readonly SubagentCompletedStreamEvent[]; }> { "use step"; const durableSession = readDurableSession(input.sessionState); let session = durableSession; const acceptedViews: TaskView[] = []; + const subagentCompletions: SubagentCompletedStreamEvent[] = []; for (const view of input.views) { - if (findSessionTaskEntry(session.state, view.taskId) === undefined) continue; - const state = cacheTerminalTaskView(session.state, view); - if (state !== session.state) session = { ...session, state }; - acceptedViews.push(view); + const entry = findBackgroundWorkflowToolRun(session.state, view.taskId); + if (entry === undefined) continue; + const state = recordWorkflowTaskView(session.state, view); + if (state !== session.state) { + session = { ...session, state }; + if (entry.task.metadata.kind === "subagent" && view.status === "completed") { + subagentCompletions.push({ + type: "subagent.completed", + data: { + callId: entry.callId, + subagentName: entry.toolName, + output: + typeof view.lastOutput.data === "string" + ? view.lastOutput.data + : JSON.stringify(view.lastOutput.data), + }, + }); + } + } + acceptedViews.push(readWorkflowTaskView(entry.task) ?? view); + session = clearProxyInputRequestsForTask(session, view.taskId); session = applyTaskAgentHandleCommand(session, { kind: "release-owner", ownerId: view.taskId, @@ -137,7 +153,7 @@ export async function recordTerminalTaskViewsStep(input: { session === durableSession ? input.sessionState : replaceDurableSessionSnapshot({ session, state: input.sessionState }); - return { serializedContext, sessionState }; + return { serializedContext, sessionState, views: acceptedViews, subagentCompletions }; } async function settleBackgroundTaskActions(input: { @@ -148,6 +164,23 @@ async function settleBackgroundTaskActions(input: { if (input.views.length === 0) return input.serializedContext; try { const ctx = await deserializeContext(input.serializedContext); + const observer = ctx.get(ActivityObserverKey); + const settledAt = new Date().toISOString(); + const events = input.views.flatMap((view) => { + const entry = findBackgroundWorkflowToolRun(input.session.state, view.taskId); + return projectTaskActivity({ + activityObserver: + observer === undefined + ? undefined + : { + sink: observer.sink, + workIdentity: entry?.task.activityWorkIdentity, + }, + settledAt, + view, + }); + }); + await submitActivity({ events, sink: observer?.sink }); const bundle = ctx.get(BundleKey); if (bundle === undefined) return input.serializedContext; const instrumentation = bindSessionInstrumentation({ diff --git a/packages/eve/src/execution/tasks/parent/run-parent.ts b/packages/eve/src/execution/tasks/parent/run-parent.ts index 6b7bd7300..3e530830f 100644 --- a/packages/eve/src/execution/tasks/parent/run-parent.ts +++ b/packages/eve/src/execution/tasks/parent/run-parent.ts @@ -1,35 +1,28 @@ -import type { TaskRunWorkflowInput } from "#execution/tasks/child/workflow.js"; +import type { BackgroundWorkflowToolRunInput } from "#execution/tools/workflow/types.js"; import { isTaskWorkflowTargetGone } from "#execution/tasks/workflow-target.js"; import { startWorkflowOnCurrentDeployment, - taskRunWorkflowReference, + workflowToolRunWorkflowReference, waitForCommandHookOwner, } from "#execution/workflow-runtime.js"; -import { getRun, resumeHook } from "#internal/workflow/runtime.js"; +import { resumeHook } from "#internal/workflow/runtime.js"; import { - TASK_VIEW_STREAM_NAMESPACE, type TaskCommand, type TaskCommandHookPayload, type TaskRunInboundPayload, - type TaskView, } from "#tasks/types.js"; -const TASK_VIEW_READ_TIMEOUT_MS = 10_000; - /** - * Node-side controls for durable task runs — the generic transport layer. - * This module only speaks `TaskCommand`/`TaskView`; it knows nothing about - * executor implementations, receipts, or the session index. Caller-specific - * policy composes these primitives. + * Node-side admission and cancellation for session-owned workflow invocations. * * Every export must be called from inside a `"use step"` body; none of * these are steps themselves so dispatch and tool steps can compose them * inside one durable boundary. */ -/** Starts the durable run owning one task's lifecycle. */ -export async function startTaskRun(input: TaskRunWorkflowInput): Promise { - await startWorkflowOnCurrentDeployment(taskRunWorkflowReference, [input]); +/** Starts a session-owned invocation through the common workflow entry. */ +export async function startTaskRun(input: BackgroundWorkflowToolRunInput): Promise { + await startWorkflowOnCurrentDeployment(workflowToolRunWorkflowReference, [input]); } /** Resolves the task run that won ownership of one replay-stable command token. */ @@ -111,58 +104,3 @@ export async function sendTaskInboundPayload(input: { return "unreachable"; } } - -/** - * Reads the latest view a task run has published, or `undefined` - * when the run has not committed its first view yet (the caller - * already holds the creation receipt, which is `working`). - * - * Views are trusted without re-validation: the task run is the - * single writer and every write passed the transition function. - */ -export async function readLatestTaskView(input: { - readonly taskRunId: string; -}): Promise { - const stream = getRun(input.taskRunId).getReadable({ - namespace: TASK_VIEW_STREAM_NAMESPACE, - startIndex: -1, - }); - const tailIndex = await stream.getTailIndex(); - const reader = stream.getReader(); - try { - if (tailIndex < 0) { - return undefined; - } - const result = await readWithTimeout(reader, "latest task view"); - return result; - } finally { - await reader.cancel("eve task view read complete").catch(() => {}); - reader.releaseLock(); - } -} - -async function readWithTimeout( - reader: ReadableStreamDefaultReader, - what: string, -): Promise { - let timeout: ReturnType | undefined; - try { - const result = await Promise.race([ - reader.read().then((read) => ({ kind: "read" as const, read })), - new Promise<{ readonly kind: "timeout" }>((resolve) => { - timeout = setTimeout(() => resolve({ kind: "timeout" }), TASK_VIEW_READ_TIMEOUT_MS); - }), - ]); - if (result.kind === "timeout") { - throw new Error(`Timed out reading ${what} after ${TASK_VIEW_READ_TIMEOUT_MS}ms.`); - } - if (result.read.done) { - return undefined; - } - return result.read.value; - } finally { - if (timeout !== undefined) { - clearTimeout(timeout); - } - } -} diff --git a/packages/eve/src/execution/tasks/parent/task-cancel.ts b/packages/eve/src/execution/tasks/parent/task-cancel.ts index 6707a6951..4d4205f42 100644 --- a/packages/eve/src/execution/tasks/parent/task-cancel.ts +++ b/packages/eve/src/execution/tasks/parent/task-cancel.ts @@ -1,44 +1,22 @@ -import { cancelRun, getRun, getWorld } from "#internal/workflow/runtime.js"; -import type { SessionTaskIndexEntry } from "#tasks/session-index.js"; -import { cancelWorkflowToolRun } from "#execution/tools/workflow/cancel.js"; -import { readWorkflowToolExecutorAddress } from "#execution/tools/workflow/types.js"; +import type { HarnessSession } from "#harness/types.js"; +import { settleWorkflowToolRunCancellation } from "#execution/tools/workflow/cancel.js"; +import type { BackgroundWorkflowToolRun } from "#harness/workflow-tool-runs.js"; export interface TaskExecutorCancelContext { - readonly entry: SessionTaskIndexEntry; + readonly entry: BackgroundWorkflowToolRun; readonly serializedContext?: Record; - readonly session?: unknown; + readonly session?: Pick; } export type TaskExecutorCancel = (input: TaskExecutorCancelContext) => Promise; -const TASK_RUN_CANCEL_GRACE_MS = 1_000; -const TASK_RUN_CANCEL_POLL_MS = 50; - -/** Cancels task-owned work and reports whether the lifecycle run was forcibly stopped. */ +/** Cancels task-owned work and waits for the invocation cleanup boundary. */ export async function cancelTaskOwnedWork( input: TaskExecutorCancelContext & { readonly cancelOwnedWork?: TaskExecutorCancel }, -): Promise { - const workflowToolRun = readWorkflowToolExecutorAddress(input.entry.executor); - if (workflowToolRun !== undefined) { - await cancelWorkflowToolRun(workflowToolRun, `Task ${input.entry.taskId} was cancelled.`); - } +): Promise { await input.cancelOwnedWork?.(input); - const deadline = Date.now() + TASK_RUN_CANCEL_GRACE_MS; - while (Date.now() < deadline) { - try { - const status = await getRun(input.entry.taskRunId).status; - if (status !== "pending" && status !== "running") return false; - } catch { - return false; - } - await new Promise((resolve) => setTimeout(resolve, TASK_RUN_CANCEL_POLL_MS)); - } - try { - await cancelRun(await getWorld(), input.entry.taskRunId, { - cancelReason: `Task ${input.entry.taskId} was cancelled.`, - }); - } catch { - // The merged task run may have completed during its cooperative unwind. - } - return true; + await settleWorkflowToolRunCancellation( + input.entry.address.runId, + `Task ${input.entry.task.taskId} was cancelled.`, + ); } diff --git a/packages/eve/src/execution/tasks/parent/tool-execution.integration.test.ts b/packages/eve/src/execution/tasks/parent/tool-execution.integration.test.ts index 5059ab605..cac73fcd1 100644 --- a/packages/eve/src/execution/tasks/parent/tool-execution.integration.test.ts +++ b/packages/eve/src/execution/tasks/parent/tool-execution.integration.test.ts @@ -21,7 +21,10 @@ import { setHarnessEmissionState } from "#harness/emission-state.js"; import { TurnCancelledError } from "#harness/turn-cancellation.js"; import type { HarnessSession } from "#harness/types.js"; import { getAgentHandleStore, setAgentHandleStore } from "#subagents/handles/store.js"; -import { getSessionTaskIndex, recordSessionTask } from "#tasks/session-index.js"; +import { + getBackgroundWorkflowToolRuns, + registerWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; vi.mock("#execution/tools/subagent/steer.js", () => ({ steerBackgroundAgent: vi.fn() })); vi.mock("#execution/tasks/parent/run-parent.js", () => ({ sendTaskCommand: vi.fn(async () => "delivered"), @@ -44,12 +47,16 @@ const handle = { phase: "claimed" as const, }; const entry = { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { agentId: identity.id, kind: "subagent", name: identity.name }, - taskId: handle.ownerId, - taskInboxToken: "original-task-inbox", - taskRunId: "original-task-run", + callId: handle.ownerId, + toolName: { agentId: identity.id, kind: "subagent", name: identity.name }.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "original-task-run", hookToken: "original-task-inbox" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { agentId: identity.id, kind: "subagent", name: identity.name }, + taskId: handle.ownerId, + }, }; function createSession(owned = true): HarnessSession { @@ -64,7 +71,7 @@ function createSession(owned = true): HarnessSession { }, { sessionStarted: true, sequence: 2, stepIndex: 0, turnId: "turn-2" }, ); - return owned ? recordSessionTask(session, entry) : session; + return owned ? registerWorkflowToolRun(session, entry) : session; } async function createScope( @@ -91,15 +98,15 @@ async function createScope( callId = "steering-call", agentId: string | undefined = identity.id, name = identity.name, - resultKind: "subagent" | "tool" = "subagent", + kind: "subagent" | "tool" = "subagent", + label?: (input: unknown) => string, ) { const definition = { execute: vi.fn(), label: label === undefined ? undefined : { start: label }, name, - nodeId: identity.nodeId, - resultKind, + nodeId: kind === "subagent" ? identity.nodeId : undefined, workflowId: "research-workflow", }; const toolInput = { agentId, message: "Use the updated instruction" }; @@ -138,10 +145,10 @@ describe("background subagent steering", () => { input: { agentId: identity.id, message: "Use the updated instruction" }, }), ); - expect(receipt).toEqual({ agentId: identity.id, status: "working", taskId: entry.taskId }); + expect(receipt).toEqual({ agentId: identity.id, status: "working", taskId: entry.task.taskId }); const session = await scope.commit(); expect(getAgentHandleStore(session.state)?.handles).toEqual([handle]); - expect(getSessionTaskIndex(session.state)).toEqual([entry]); + expect(getBackgroundWorkflowToolRuns(session.state)).toEqual([entry]); expect(startTaskRun).not.toHaveBeenCalled(); expect(waitForTaskCommandOwner).not.toHaveBeenCalled(); expect(sendTaskCommand).not.toHaveBeenCalled(); @@ -171,26 +178,30 @@ describe("background subagent steering", () => { await scope.execute("new-call", ""); const committed = await scope.commit(); - const task = getSessionTaskIndex(committed.state).find( - (candidate) => candidate.taskId !== entry.taskId, + const task = getBackgroundWorkflowToolRuns(committed.state).find( + (candidate) => candidate.task.taskId !== entry.task.taskId, ); - expect(task?.dispatchContext).toEqual({ + expect(task?.task.dispatchContext).toEqual({ auth: { current: creatorCurrent, initiator: creatorInitiator }, }); if (task === undefined) throw new Error("Expected created task"); - const replayed = recordSessionTask(committed, { + const replayed = registerWorkflowToolRun(committed, { ...task, - dispatchContext: { - auth: { - current: { ...creatorCurrent, principalId: "later-current" }, - initiator: { ...creatorInitiator, principalId: "later-initiator" }, + task: { + ...task.task, + dispatchContext: { + auth: { + current: { ...creatorCurrent, principalId: "later-current" }, + initiator: { ...creatorInitiator, principalId: "later-initiator" }, + }, }, }, }); expect( - getSessionTaskIndex(replayed.state).find((candidate) => candidate.taskId === task.taskId) - ?.dispatchContext, + getBackgroundWorkflowToolRuns(replayed.state).find( + (candidate) => candidate.task.taskId === task.task.taskId, + )?.task.dispatchContext, ).toEqual({ auth: { current: creatorCurrent, initiator: creatorInitiator } }); }); @@ -209,18 +220,18 @@ describe("background subagent steering", () => { await scope.execute("new-call", ""); const committed = await scope.commit(); - const task = getSessionTaskIndex(committed.state).find( - (candidate) => candidate.taskId !== entry.taskId, + const task = getBackgroundWorkflowToolRuns(committed.state).find( + (candidate) => candidate.task.taskId !== entry.task.taskId, ); - expect(task?.dispatchContext).toEqual({ + expect(task?.task.dispatchContext).toEqual({ auth: { current: null, initiator: sessionInitiator }, }); }); it.each(["subagent", "tool"] as const)( - "persists agent-backed activity identity only (%s)", - async (resultKind) => { + "retains activity identity for parent-owned settlement (%s)", + async (kind) => { const activityObserver = { sink: { url: "https://parent.example/activity", version: 1 as const }, workIdentity: { @@ -232,18 +243,14 @@ describe("background subagent steering", () => { }; const scope = await createScope(createSession(), activityObserver); - await scope.execute("new-call", "", identity.name, resultKind); + await scope.execute("new-call", "", identity.name, kind); const committed = await scope.commit(); - const task = getSessionTaskIndex(committed.state).find( - (candidate) => candidate.taskId !== entry.taskId, + const task = getBackgroundWorkflowToolRuns(committed.state).find( + (candidate) => candidate.task.taskId !== entry.task.taskId, ); expect(task).toBeDefined(); - if (resultKind === "tool") { - expect(task?.activityWorkIdentity).toBeUndefined(); - return; - } - expect(task?.activityWorkIdentity).toMatchObject({ + expect(task?.task.activityWorkIdentity).toMatchObject({ callId: "new-call", kind: "task", name: "research", @@ -278,13 +285,15 @@ describe("background subagent steering", () => { }), ); const committed = await scope.commit(); - const task = getSessionTaskIndex(committed.state).find( - (candidate) => candidate.taskId !== entry.taskId, + const task = getBackgroundWorkflowToolRuns(committed.state).find( + (candidate) => candidate.task.taskId !== entry.task.taskId, ); expect(task).toMatchObject({ - activityWorkIdentity: { label: "Investigator", name: "research" }, - metadata: { name: "research" }, + task: { + activityWorkIdentity: { label: "Investigator", name: "research" }, + metadata: { name: "research" }, + }, }); }); @@ -296,9 +305,9 @@ describe("background subagent steering", () => { it("does not steer a task associated with another child", async () => { const scope = await createScope( - recordSessionTask(createSession(), { + registerWorkflowToolRun(createSession(), { ...entry, - metadata: { ...entry.metadata, agentId: "another-child" }, + task: { ...entry.task, metadata: { ...entry.task.metadata, agentId: "another-child" } }, }), ); await expect(scope.execute()).rejects.toThrow("AGENT_BUSY"); @@ -379,8 +388,8 @@ describe("background subagent steering", () => { const scope = await createScope(); const receipts = await Promise.all([scope.execute("first-call"), scope.execute("second-call")]); expect(receipts).toEqual([ - { agentId: identity.id, status: "working", taskId: entry.taskId }, - { agentId: identity.id, status: "working", taskId: entry.taskId }, + { agentId: identity.id, status: "working", taskId: entry.task.taskId }, + { agentId: identity.id, status: "working", taskId: entry.task.taskId }, ]); expect(steerBackgroundAgent).toHaveBeenCalledTimes(2); expect(startTaskRun).not.toHaveBeenCalled(); diff --git a/packages/eve/src/execution/tasks/parent/tool-execution.test.ts b/packages/eve/src/execution/tasks/parent/tool-execution.test.ts deleted file mode 100644 index 5cfe7d636..000000000 --- a/packages/eve/src/execution/tasks/parent/tool-execution.test.ts +++ /dev/null @@ -1,92 +0,0 @@ -import { beforeEach, describe, expect, it, vi } from "vitest"; - -import { ContextContainer, contextStorage } from "#context/container.js"; -import { SessionKey } from "#context/keys.js"; -import { backgroundToolExecutionProvider } from "#execution/tasks/parent/tool-execution.js"; -import { beginBackgroundTask } from "#execution/tasks/parent/delegate.js"; -import { sendTaskCommand, sendTaskInboundPayload } from "#execution/tasks/parent/run-parent.js"; -import { setHarnessEmissionState } from "#harness/emission.js"; -import { - createBackgroundToolCallBatch, - type BackgroundExecutableTool, -} from "#harness/background-tools.js"; -import type { HarnessSession } from "#harness/types.js"; - -vi.mock("#execution/tasks/parent/delegate.js", () => ({ - beginBackgroundTask: vi.fn(), - createTaskAgentDispatchContext: vi.fn(() => ({ auth: { current: null, initiator: null } })), -})); -vi.mock("#execution/tasks/parent/run-parent.js", () => ({ - sendTaskCommand: vi.fn(), - sendTaskInboundPayload: vi.fn(), -})); - -describe("ordinary background tool execution", () => { - beforeEach(() => { - vi.resetAllMocks(); - vi.mocked(sendTaskCommand).mockResolvedValue("delivered"); - vi.mocked(sendTaskInboundPayload).mockResolvedValue("delivered"); - vi.mocked(beginBackgroundTask).mockResolvedValue({ - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "tool", name: "export" }, - taskId: "task-1", - taskInboxToken: "inbox-1", - taskRunId: "run-1", - }); - }); - - it("returns the fixed receipt while routing yields and the final return separately", async () => { - const ctx = new ContextContainer(); - ctx.setVirtualContext(SessionKey, { - auth: { current: null, initiator: null }, - sessionId: "session-1", - turn: { id: "turn-1", sequence: 0 }, - }); - const session: HarnessSession = setHarnessEmissionState( - { - agent: { modelReference: { id: "openai/gpt-5.4" }, system: "", tools: [] }, - compaction: { recentWindowSize: 10, threshold: 100_000 }, - continuationToken: "parent-token", - history: [], - sessionId: "session-1", - }, - { sequence: 0, sessionStarted: true, stepIndex: 0, turnId: "turn-1" }, - ); - const provider = await backgroundToolExecutionProvider.create(ctx, session); - if (provider === undefined) throw new Error("Background executor was not created."); - const executor = provider.value; - const definition: BackgroundExecutableTool = { - name: "export", - async *execute(_input, _options, task) { - yield "progress"; - yield task.postMessage("Review the export"); - return { result: "done" }; - }, - }; - const batch = createBackgroundToolCallBatch(); - batch.setTool("export", definition); - batch.register({ callId: "call-1", input: {}, toolName: "export" }); - const result = await contextStorage.run(ctx, () => - executor.execute({ - batch, - definition, - options: { toolCallId: "call-1", messages: [] }, - toolInput: {}, - }), - ); - expect(result).toEqual({ status: "working", taskId: "task-1" }); - expect(sendTaskCommand).toHaveBeenCalledWith({ - command: { kind: "complete", data: { result: "done" } }, - taskInboxToken: "inbox-1", - }); - expect(sendTaskInboundPayload).toHaveBeenCalledWith({ - payload: expect.objectContaining({ kind: "task-update", message: "progress" }), - taskInboxToken: "inbox-1", - }); - expect(sendTaskInboundPayload).toHaveBeenCalledWith({ - payload: expect.objectContaining({ kind: "task-message", message: "Review the export" }), - taskInboxToken: "inbox-1", - }); - }); -}); diff --git a/packages/eve/src/execution/tasks/parent/tool-execution.ts b/packages/eve/src/execution/tasks/parent/tool-execution.ts index e0880a3b6..3d8f45e26 100644 --- a/packages/eve/src/execution/tasks/parent/tool-execution.ts +++ b/packages/eve/src/execution/tasks/parent/tool-execution.ts @@ -1,10 +1,14 @@ +import { + type BackgroundWorkflowToolRun, + findBackgroundWorkflowToolRun, + registerWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import type { ContextContainer } from "#context/container.js"; import { loadContext } from "#context/container.js"; import { ActivityObserverKey } from "#context/keys.js"; import type { FrameworkContextProvider } from "#context/provider.js"; import { runStep } from "#context/run-step.js"; import { buildCallbackContext } from "#context/build-callback-context.js"; -import { isAuthorizationSignal } from "#harness/authorization.js"; import { activeTurnId } from "#harness/active-turn-id.js"; import { resolveWorkflowAgentMetadata } from "#execution/tools/subagent/metadata.js"; import { getHarnessEmissionState } from "#harness/emission.js"; @@ -17,27 +21,16 @@ import { type BackgroundToolExecutor, } from "#harness/background-tools.js"; import { deriveBackgroundTaskActivityObserver } from "#execution/activity-work.js"; -import { isAsyncIterable } from "#shared/async-iterable.js"; -import { parseJsonValue } from "#shared/json.js"; import { projectToolStartLabel } from "#harness/action-presentation.js"; import type { ToolExecuteOptions } from "#tools/definition.js"; -import { createTaskMessage, isTaskMessage, type TaskExec } from "#tools/task.js"; -import { findSessionTaskEntry, recordSessionTask } from "#tasks/session-index.js"; import type { AgentView } from "#subagents/handles/prompt.js"; import { - beginBackgroundTask, createTaskAgentDispatchContext, prepareBackgroundTask, rejectDelegatedDispatch, - type BackgroundTask, } from "#execution/tasks/parent/delegate.js"; import { parseWorkflowToolInput } from "#execution/tools/workflow/background.js"; -import { - sendTaskCommand, - sendTaskInboundPayload, - startTaskRun, - waitForTaskCommandOwner, -} from "#execution/tasks/parent/run-parent.js"; +import { startTaskRun, waitForTaskCommandOwner } from "#execution/tasks/parent/run-parent.js"; import { sessionCommandHookToken } from "#execution/session-inbox/address.js"; import { projectSubagentTask } from "#execution/tasks/parent/subagent-task-projection.js"; import { deriveAgentOperationId } from "#subagents/handles/operation-id.js"; @@ -52,8 +45,6 @@ import { import { applyTaskAgentHandleCommand } from "#subagents/handles/transitions.js"; import { steerBackgroundAgent } from "#execution/tools/subagent/steer.js"; -const IN_PROCESS_WORKFLOW_EXECUTOR = { data: {}, kind: "workflow-task" } as const; - interface BackgroundToolExecutionRecord { readonly callId: string; claim?: { @@ -65,7 +56,7 @@ interface BackgroundToolExecutionRecord { readonly operationId: string; }; settled: boolean; - task?: BackgroundTask; + task?: BackgroundWorkflowToolRun; } interface BackgroundToolStepResult { @@ -242,7 +233,7 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { let next = session; for (const record of this.records) { if (!record.settled || record.task === undefined) continue; - next = recordSessionTask(next, record.task); + next = registerWorkflowToolRun(next, record.task); } if (this.agentHandlesChanged) { next = writeHandles(next, getAgentHandleStore(this.agentHandleSession.state)?.handles ?? []); @@ -256,9 +247,9 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { ? [ { callId: record.callId, - taskInboxToken: record.task.taskInboxToken, - taskId: record.task.taskId, - taskRunId: record.task.taskRunId, + taskInboxToken: record.task.address.hookToken, + taskId: record.task.task.taskId, + taskRunId: record.task.address.runId, }, ] : [], @@ -291,37 +282,8 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { const task = started.task; record.task = task; - const taskExec: TaskExec = { - binding: { taskId: task.taskId, token: task.taskInboxToken }, - postMessage: createTaskMessage, - send() { - throw new Error("task.send() was replaced by yielded task descriptors."); - }, - session: this.initialSession, - task, - taskId: task.taskId, - }; - if (input.definition.workflowId !== undefined) { - record.settled = true; - return { - ...started.receipt, - status: "working", - taskId: task.taskId, - }; - } - const output = input.definition.execute(input.toolInput, input.options, taskExec); - const settled = isAsyncIterable(output) - ? await executeBackgroundIterable({ - callId: input.options.toolCallId, - output, - task, - }) - : await output; - if (isAuthorizationSignal(settled)) return settled; - - await deliverTaskCommand(task, { data: parseJsonValue(settled), kind: "complete" }); record.settled = true; - return { status: "working", taskId: task.taskId }; + return { ...started.receipt, status: "working", taskId: task.task.taskId }; } private async startTask(input: { @@ -338,7 +300,7 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { | { readonly kind: "started"; readonly receipt?: { readonly agentId: string }; - readonly task: BackgroundTask; + readonly task: BackgroundWorkflowToolRun; } | { readonly kind: "steered"; @@ -349,20 +311,11 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { }; } > { - const workflow = - input.input.definition.workflowId === undefined - ? undefined - : { - ...input.input.definition, - workflowId: input.input.definition.workflowId, - }; - let workflowInput = - workflow === undefined - ? undefined - : parseWorkflowToolInput(input.input.toolInput, input.input.definition.name); + const workflow = input.input.definition; + let workflowInput = parseWorkflowToolInput(input.input.toolInput, input.input.definition.name); const parentTurnId = activeTurnId(input.emission); let subagentProjection = - workflow?.resultKind === "subagent" && workflowInput !== undefined + workflow.nodeId !== undefined ? projectSubagentTask({ ctx: input.ctx, input: workflowInput, @@ -380,13 +333,13 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { subagentProjection.identity === undefined && !hasAgentHandle(this.agentHandleSession, subagentProjection.metadata.agentId) ) { - const { agentId: _unknownAgentId, ...freshWorkflowInput } = workflowInput!; + const { agentId: _unknownAgentId, ...freshWorkflowInput } = workflowInput; workflowInput = freshWorkflowInput; subagentProjection = projectSubagentTask({ ctx: input.ctx, input: freshWorkflowInput, name: input.input.definition.name, - nodeId: workflow!.nodeId ?? input.input.definition.name, + nodeId: workflow.nodeId ?? input.input.definition.name, taskInput: { callId: input.input.options.toolCallId, parentSessionId: this.initialSession.sessionId, @@ -418,32 +371,16 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { parentTurnId, session: this.initialSession, }; - if (workflow === undefined) { - return { - kind: "started", - task: await beginBackgroundTask({ - activityObserver: taskInput.activityObserver, - callId: taskInput.callId, - dispatchContext: taskInput.dispatchContext, - metadata: taskInput.metadata, - parentSessionId: taskInput.parentSessionId, - parentStepIndex: taskInput.parentStepIndex, - parentTurnId: taskInput.parentTurnId, - session: taskInput.session, - }), - }; - } - if (workflowInput === undefined) { - throw new Error(`Background workflow tool "${input.input.definition.name}" has no input.`); - } - - const task: Omit = { - ...prepareBackgroundTask(taskInput), - activityWorkIdentity: - workflow.resultKind === "subagent" ? taskInput.activityObserver?.workIdentity : undefined, + const prepared = prepareBackgroundTask(taskInput); + const task = { + ...prepared, + task: { + ...prepared.task, + activityWorkIdentity: taskInput.activityObserver?.workIdentity, + }, }; if ( - workflow.resultKind === "subagent" && + workflow.nodeId !== undefined && subagentProjection !== undefined && subagentProjection.identity !== undefined ) { @@ -453,7 +390,7 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { callId: taskInput.callId, kind: "reserve", operationId: identity.operation.id, - ownerId: task.taskId, + ownerId: task.task.taskId, }); if (reservation.kind !== "ready") { throw new Error(`Agent handle store rejected start operation "${identity.operation.id}".`); @@ -464,7 +401,7 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { }; } if ( - workflow.resultKind === "subagent" && + workflow.nodeId !== undefined && subagentProjection !== undefined && subagentProjection.identity === undefined ) { @@ -480,16 +417,16 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { invokedName: subagentProjection.metadata.name, kind: "claim", operationId, - ownerId: task.taskId, + ownerId: task.task.taskId, }); if (claim.kind === "busy" && claim.handle.phase === "claimed") { const handle = claim.handle; - const entry = findSessionTaskEntry(this.agentHandleSession.state, handle.ownerId); + const entry = findBackgroundWorkflowToolRun(this.agentHandleSession.state, handle.ownerId); if ( - entry?.metadata.kind === "subagent" && - entry.metadata.agentId === handle.identity.id && - entry.metadata.name === handle.identity.name && - entry.terminalView === undefined + entry?.task.metadata.kind === "subagent" && + entry.task.metadata.agentId === handle.identity.id && + entry.task.metadata.name === handle.identity.name && + entry.task.outcome === undefined ) { await steerBackgroundAgent({ ctx: input.ctx, @@ -500,7 +437,7 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { }); return { kind: "steered", - receipt: { agentId: handle.identity.id, taskId: entry.taskId, status: "working" }, + receipt: { agentId: handle.identity.id, taskId: entry.task.taskId, status: "working" }, }; } } @@ -509,36 +446,33 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { } else { input.record.claim = { operationId, - taskId: task.taskId, + taskId: task.task.taskId, }; } } await startTaskRun({ activityObserver: taskInput.activityObserver, - initialView: { metadata: task.metadata, status: "working", taskId: task.taskId }, + initialView: { metadata: task.task.metadata, status: "working", taskId: task.task.taskId }, parentContinuationToken: sessionCommandHookToken(this.initialSession.sessionId), - taskInboxToken: task.taskInboxToken, + taskInboxToken: task.address.hookToken, workflow: { agents: resolveWorkflowAgentMetadata(input.ctx), callId: taskInput.callId, executeInput: workflow.executeInput?.(workflowInput), input: workflowInput, - resultKind: workflow.resultKind, session: callbackSession, stepIndex: input.emission.stepIndex, toolName: input.input.definition.name, - taskId: task.taskId, workflowId: workflow.workflowId, }, }); - const owner = await waitForTaskCommandOwner({ taskInboxToken: task.taskInboxToken }); + const owner = await waitForTaskCommandOwner({ taskInboxToken: task.address.hookToken }); const backgroundTask = { ...task, - executor: IN_PROCESS_WORKFLOW_EXECUTOR, - taskRunId: owner.runId, + address: { ...task.address, runId: owner.runId }, }; input.record.task = backgroundTask; - if (workflow.resultKind !== "subagent") { + if (workflow.nodeId === undefined) { return { kind: "started", task: backgroundTask }; } @@ -583,7 +517,7 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { this.applyAgentHandleCommand({ agentId: record.reservation.agentId, kind: "remove", - ownerId: record.task.taskId, + ownerId: record.task.task.taskId, }); } } @@ -597,42 +531,6 @@ class BackgroundToolExecutionScope implements BackgroundToolExecutor { } } -async function executeBackgroundIterable(input: { - readonly callId: string; - readonly output: AsyncIterable; - readonly task: BackgroundTask; -}): Promise { - const iterator = input.output[Symbol.asyncIterator](); - let updateIndex = 0; - let next = await iterator.next(); - while (!next.done) { - const payload = isTaskMessage(next.value) - ? { - callId: input.callId, - kind: "task-message" as const, - message: next.value.message, - messageEpoch: input.task.taskId, - messageIndex: updateIndex++, - } - : { - callId: input.callId, - kind: "task-update" as const, - message: typeof next.value === "string" ? next.value : JSON.stringify(next.value), - updateEpoch: input.task.taskId, - updateIndex: updateIndex++, - }; - const outcome = await sendTaskInboundPayload({ - payload, - taskInboxToken: input.task.taskInboxToken, - }); - if (outcome !== "delivered") { - throw new Error(`Task run "${input.task.taskId}" did not accept "${payload.kind}".`); - } - next = await iterator.next(); - } - return next.value ?? null; -} - function hasAgentHandle(session: HarnessSession, agentId: string): boolean { return ( getAgentHandleStore(session.state)?.handles.some((handle) => handle.identity.id === agentId) === @@ -647,16 +545,6 @@ function requireExecutionScope(executor: BackgroundToolExecutor): BackgroundTool return executor; } -async function deliverTaskCommand( - task: BackgroundTask, - command: Parameters[0]["command"], -): Promise { - const outcome = await sendTaskCommand({ command, taskInboxToken: task.taskInboxToken }); - if (outcome !== "delivered") { - throw new Error(`Task run "${task.taskId}" did not accept "${command.kind}".`); - } -} - function readClaimedHandle(result: AgentHandleStoreCommandResult): boolean { return result.kind === "ready" && result.handle?.phase === "claimed"; } diff --git a/packages/eve/src/execution/terminate-child-sessions-step.test.ts b/packages/eve/src/execution/terminate-child-sessions-step.test.ts index 389675892..5e1719925 100644 --- a/packages/eve/src/execution/terminate-child-sessions-step.test.ts +++ b/packages/eve/src/execution/terminate-child-sessions-step.test.ts @@ -3,7 +3,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import { AGENT_HANDLES_STATE_KEY, type AgentHandle } from "#subagents/handles/store.js"; import type { DurableSessionState } from "#execution/durable-session-store.js"; import { terminateChildSessionsStep } from "#execution/terminate-child-sessions-step.js"; -import { SESSION_TASKS_STATE_KEY, type SessionTaskIndexEntry } from "#tasks/session-index.js"; +import type { BackgroundWorkflowToolRun } from "#harness/workflow-tool-runs.js"; const COMPILED_BUNDLE = { subagentRegistry: { subagentsByNodeId: new Map() }, @@ -61,7 +61,9 @@ vi.mock("#internal/workflow/runtime.js", () => ({ describe("terminateChildSessionsStep", () => { beforeEach(() => { cancelOwnedTaskMock.mockReset(); - cancelOwnedTaskMock.mockResolvedValue(undefined); + cancelOwnedTaskMock.mockImplementation( + async ({ entry }: { entry: BackgroundWorkflowToolRun }) => cancelledView(entry), + ); cancelRunMock.mockReset(); cancelRunMock.mockResolvedValue(undefined); deserializeContextMock.mockReset(); @@ -248,13 +250,15 @@ describe("terminateChildSessionsStep", () => { const secondCancellation = createDeferred(); const order: string[] = []; cancelOwnedTaskMock - .mockImplementationOnce(async () => { + .mockImplementationOnce(async ({ entry }: { entry: BackgroundWorkflowToolRun }) => { await firstCancellation.promise; order.push("task-1-settled"); + return cancelledView(entry); }) - .mockImplementationOnce(async () => { + .mockImplementationOnce(async ({ entry }: { entry: BackgroundWorkflowToolRun }) => { await secondCancellation.promise; order.push("task-2-settled"); + return cancelledView(entry); }); cancelRunMock.mockImplementation(async () => { order.push("child-cancelled"); @@ -289,7 +293,9 @@ describe("terminateChildSessionsStep", () => { const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); cancelOwnedTaskMock .mockRejectedValueOnce(new Error("task cancellation unavailable")) - .mockResolvedValueOnce(undefined); + .mockImplementationOnce(async ({ entry }: { entry: BackgroundWorkflowToolRun }) => + cancelledView(entry), + ); try { await expect( @@ -427,23 +433,30 @@ function startingHandle(input: { }; } -function indexedTask(taskId: string): SessionTaskIndexEntry { +function indexedTask(taskId: string): BackgroundWorkflowToolRun { return { - taskInboxToken: `${taskId}:inbox`, - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { + callId: taskId, + toolName: { kind: "tool", name: "research", + }.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: `run-${taskId}`, hookToken: `${taskId}:inbox` }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { + kind: "tool", + name: "research", + }, + taskId, }, - taskId, - taskRunId: `run-${taskId}`, }; } function makeSessionState( handles: readonly AgentHandle[], - tasks: readonly SessionTaskIndexEntry[] = [], + tasks: readonly BackgroundWorkflowToolRun[] = [], ): DurableSessionState { return { continuationToken: "parent-token", @@ -466,7 +479,7 @@ function makeSessionState( ? { [AGENT_HANDLES_STATE_KEY]: { handles } } : { [AGENT_HANDLES_STATE_KEY]: { handles }, - [SESSION_TASKS_STATE_KEY]: { tasks, version: 2 }, + "eve.workflowTool": { version: 3, runs: tasks }, }, }, }, @@ -481,3 +494,7 @@ function createDeferred(): { readonly promise: Promise; resolve(): void } }); return { promise, resolve }; } + +function cancelledView(entry: BackgroundWorkflowToolRun) { + return { taskId: entry.task.taskId, metadata: entry.task.metadata, status: "cancelled" as const }; +} diff --git a/packages/eve/src/execution/terminate-child-sessions-step.ts b/packages/eve/src/execution/terminate-child-sessions-step.ts index 22d79284b..a424c9e76 100644 --- a/packages/eve/src/execution/terminate-child-sessions-step.ts +++ b/packages/eve/src/execution/terminate-child-sessions-step.ts @@ -51,8 +51,7 @@ export async function terminateChildSessionsStep(input: { readonly ctx: ContextContainer; } | undefined; - // Cooperatively cancel live tasks first: their runs are the single writers - // for task state, so child termination cannot race completion into an ended parent. + // Record cancellation in the parent before terminating its child sessions. await cancelAllIndexedSessionTasksStep({ serializedContext: input.serializedContext, sessionState: input.sessionState, diff --git a/packages/eve/src/execution/tool-auth.ts b/packages/eve/src/execution/tool-auth.ts index 5f2172bc3..18bedbc35 100644 --- a/packages/eve/src/execution/tool-auth.ts +++ b/packages/eve/src/execution/tool-auth.ts @@ -2,26 +2,17 @@ import { buildBaseToolContext } from "#context/build-base-tool-context.js"; import type { SessionAuthContext } from "#channel/types.js"; import type { ApprovalResponseAuth } from "#approval/definition.js"; import type { ToolAuthOptions, ToolContext, ToolExecuteOptions } from "#tools/definition.js"; -import type { TaskExec } from "#tools/task.js"; import { createAuthorizationContext } from "#runtime/authorization-context.js"; import { handleAuthorizationError } from "#runtime/connections/scoped-authorization.js"; type ToolExecuteWithAuthInput = { readonly scope: string; -} & ( - | { - readonly execution: "background"; - readonly execute: (toolInput: TInput, ctx: ToolContext, task: TaskExec) => unknown; - } - | { - readonly execution?: never; - readonly execute: (toolInput: TInput, ctx: ToolContext, task?: TaskExec) => unknown; - } -); + readonly execute: (toolInput: TInput, ctx: ToolContext) => unknown; +}; /** Supplies the shared auth capability to one authored tool execution. */ export function createToolExecuteWithAuth(input: ToolExecuteWithAuthInput) { - return (toolInput: TInput, options: ToolExecuteOptions, task?: TaskExec) => { + return (toolInput: TInput, options: ToolExecuteOptions) => { const auth = createAuthorizationContext({ scope: input.scope }); const ctx: ToolContext = { ...buildBaseToolContext({ options, toolName: input.scope }), @@ -29,13 +20,7 @@ export function createToolExecuteWithAuth(input: ToolExecuteWithAuthInpu requireAuth: auth.requireAuth, }; return auth.run(() => { - if (input.execution === "background") { - if (task === undefined) { - throw new Error("Background tool execution requires a task runtime."); - } - return input.execute(toolInput, ctx, task); - } - return input.execute(toolInput, ctx, task); + return input.execute(toolInput, ctx); }); }; } diff --git a/packages/eve/src/execution/tools/subagent/accept-event-step.test.ts b/packages/eve/src/execution/tools/subagent/accept-event-step.test.ts index 474987a7b..6e72febc3 100644 --- a/packages/eve/src/execution/tools/subagent/accept-event-step.test.ts +++ b/packages/eve/src/execution/tools/subagent/accept-event-step.test.ts @@ -1,8 +1,8 @@ +import { recordWorkflowTaskView } from "#harness/workflow-tool-runs.js"; import { createTestSessionState } from "#internal/testing/session-state.js"; import { beforeEach, describe, expect, it, vi } from "vitest"; import { readDurableSession } from "#execution/durable-session-store.js"; -import { readLatestTaskView } from "#execution/tasks/parent/run-parent.js"; import { acceptTaskAuthorizationEventStep } from "#execution/tools/subagent/accept-event-step.js"; import { setAgentHandleStore } from "#subagents/handles/store.js"; @@ -10,7 +10,6 @@ vi.mock("#execution/durable-session-store.js", async (importOriginal) => ({ ...(await importOriginal()), readDurableSession: vi.fn(), })); -vi.mock("#execution/tasks/parent/run-parent.js", () => ({ readLatestTaskView: vi.fn() })); const sessionState = createTestSessionState({ continuationToken: "parent-token", @@ -36,18 +35,22 @@ const hookPayload = { subagentName: "research", }; const taskIndex = { - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "tool", name: "export" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task-1", + toolName: "export", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "export" }, + taskId: "task-1", + }, }, ], - version: 2, }, }; @@ -78,11 +81,6 @@ describe("acceptTaskAuthorizationEventStep", () => { ownerId: "task-1", }, ]); - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: { kind: "tool", name: "export" }, - status: "working", - taskId: "task-1", - }); }); it("accepts an authorization event from the task's claimed child agent", async () => { @@ -92,7 +90,6 @@ describe("acceptTaskAuthorizationEventStep", () => { sessionState, }), ).resolves.toBe(true); - expect(readLatestTaskView).toHaveBeenCalledWith({ taskRunId: "task-run" }); }); it("accepts the owning workflow tool's event without an agent handle", async () => { @@ -148,11 +145,15 @@ describe("acceptTaskAuthorizationEventStep", () => { }); it("rejects an authorization event once the task is terminal", async () => { - vi.mocked(readLatestTaskView).mockResolvedValue({ - lastOutput: { data: "done", type: "result" }, - metadata: { kind: "tool", name: "export" }, - status: "completed", - taskId: "task-1", + const session = readDurableSession(sessionState); + vi.mocked(readDurableSession).mockReturnValue({ + ...session, + state: recordWorkflowTaskView(session.state, { + lastOutput: { data: "done", type: "result" }, + metadata: { kind: "tool", name: "export" }, + status: "completed", + taskId: "task-1", + }), }); await expect( diff --git a/packages/eve/src/execution/tools/subagent/accept-event-step.ts b/packages/eve/src/execution/tools/subagent/accept-event-step.ts index d0194e6cf..af813a3e6 100644 --- a/packages/eve/src/execution/tools/subagent/accept-event-step.ts +++ b/packages/eve/src/execution/tools/subagent/accept-event-step.ts @@ -1,8 +1,10 @@ import { type DurableSessionState, readDurableSession } from "#execution/durable-session-store.js"; -import { readLatestTaskView } from "#execution/tasks/parent/run-parent.js"; import { getAgentHandleStore } from "#subagents/handles/store.js"; -import { findSessionTaskEntry } from "#tasks/session-index.js"; -import { isTerminalTaskStatus, type TaskAuthorizationEventDelivery } from "#tasks/types.js"; +import { + readWorkflowTaskView, + findBackgroundWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; +import { type TaskAuthorizationEventDelivery } from "#tasks/types.js"; /** Accepts authorization events from the workflow task itself or an agent it owns. */ export async function acceptTaskAuthorizationEventStep(input: { @@ -13,24 +15,21 @@ export async function acceptTaskAuthorizationEventStep(input: { const { hookPayload, taskId } = input.delivery; const durableSession = readDurableSession(input.sessionState); - const entry = findSessionTaskEntry(durableSession.state, taskId); + const entry = findBackgroundWorkflowToolRun(durableSession.state, taskId); if (entry === undefined) return false; // A workflow tool can request authorization without invoking a child agent. // It has no agent handle, so bind its sender to the recorded task run, tool, // and launching turn before accepting the event. if ( - entry.metadata.kind === "tool" && - entry.taskRunId === hookPayload.childSessionId && - entry.metadata.name === hookPayload.subagentName && - entry.createdByTurnId === hookPayload.event.data.turnId + entry.task.metadata.kind === "tool" && + entry.address.runId === hookPayload.childSessionId && + entry.task.metadata.name === hookPayload.subagentName && + entry.origin.turnId === hookPayload.event.data.turnId ) { - const view = await readLatestTaskView({ taskRunId: entry.taskRunId }); + const view = readWorkflowTaskView(entry.task); // Completion can arrive after the body has returned; its sign-in UI must still close. - return ( - view !== undefined && - (hookPayload.event.type === "authorization.completed" || !isTerminalTaskStatus(view.status)) - ); + return hookPayload.event.type === "authorization.completed" || view === undefined; } const handles = getAgentHandleStore(durableSession.state)?.handles ?? []; @@ -52,6 +51,6 @@ export async function acceptTaskAuthorizationEventStep(input: { ); if (claimed === undefined && reserved.length !== 1) return false; - const view = await readLatestTaskView({ taskRunId: entry.taskRunId }); - return view !== undefined && !isTerminalTaskStatus(view.status); + const view = readWorkflowTaskView(entry.task); + return view === undefined; } diff --git a/packages/eve/src/execution/tools/subagent/emit-called-step.ts b/packages/eve/src/execution/tools/subagent/emit-called-step.ts deleted file mode 100644 index 342dab379..000000000 --- a/packages/eve/src/execution/tools/subagent/emit-called-step.ts +++ /dev/null @@ -1,33 +0,0 @@ -import { buildAdapterContext } from "#channel/adapter-context.js"; -import { callAdapterEventHandler } from "#channel/adapter.js"; -import { deserializeContext, serializeContext } from "#context/serialize.js"; -import { - encodeMessageStreamEvent, - stampMessageStreamEvent, - type UnstampedMessageStreamEvent, -} from "#protocol/message.js"; -import { ChannelKey } from "#runtime/sessions/runtime-context-keys.js"; - -/** Emits an already-projected task-owned `subagent.called` event on the parent stream. */ -export async function emitTaskSubagentCalledStep(input: { - readonly event: UnstampedMessageStreamEvent; - readonly sessionWritable: WritableStream; - readonly serializedContext: Record; -}): Promise<{ readonly serializedContext: Record }> { - "use step"; - - const ctx = await deserializeContext(input.serializedContext); - const adapter = ctx.require(ChannelKey); - const emitted = await callAdapterEventHandler( - adapter, - input.event, - buildAdapterContext(adapter, ctx), - ); - const writer = input.sessionWritable.getWriter(); - try { - await writer.write(encodeMessageStreamEvent(stampMessageStreamEvent(emitted))); - } finally { - writer.releaseLock(); - } - return { serializedContext: serializeContext(ctx) }; -} diff --git a/packages/eve/src/execution/tools/subagent/emit-event-step.integration.test.ts b/packages/eve/src/execution/tools/subagent/emit-event-step.integration.test.ts new file mode 100644 index 000000000..13c78ea6b --- /dev/null +++ b/packages/eve/src/execution/tools/subagent/emit-event-step.integration.test.ts @@ -0,0 +1,105 @@ +import { beforeEach, expect, it, vi } from "vitest"; + +import { ContextContainer, loadContext } from "#context/container.js"; +import { SessionIdKey, SessionKey, SessionTitleKey } from "#context/keys.js"; +import { deserializeContext, serializeContext } from "#context/serialize.js"; +import { emitSubagentEventStep } from "#execution/tools/subagent/emit-event-step.js"; +import { createTestSessionState } from "#internal/testing/session-state.js"; +import { createRuntimeHookRegistry } from "#runtime/hooks/registry.js"; +import { + BundleKey, + ChannelKey, + type CompiledBundle, +} from "#runtime/sessions/runtime-context-keys.js"; + +vi.mock("#context/serialize.js", () => ({ + deserializeContext: vi.fn(), + serializeContext: vi.fn(), +})); + +beforeEach(() => vi.resetAllMocks()); + +it.each([false, true])( + "delivers completion to typed and wildcard hooks and releases the writer (hook fails: %s)", + async (fails) => { + const calls: string[] = []; + const ctx = new ContextContainer(); + ctx.set(SessionIdKey, "parent"); + ctx.set(SessionKey, { + sessionId: "parent", + auth: { current: null, initiator: null }, + turn: { id: "turn", sequence: 0 }, + }); + ctx.set(ChannelKey, { + kind: "test", + state: {}, + "subagent.completed"(_data, adapterCtx) { + calls.push("adapter"); + adapterCtx.state = { completed: true }; + }, + }); + const hookRegistry = createRuntimeHookRegistry([ + { + slug: "completion", + logicalPath: "hooks/completion.ts", + sourceId: "hooks/completion.ts", + sourceKind: "module", + exportName: undefined, + events: { + "subagent.completed": async (event, hookCtx) => { + expect(hookCtx.session.id).toBe("parent"); + if (event.type !== "subagent.completed") throw new Error("Unexpected event type"); + expect(event.data.output).toBe("done"); + expect(loadContext()).toBe(ctx); + calls.push("typed"); + if (fails) throw new Error("completion subscriber failed"); + ctx.set(SessionTitleKey, "Completed research"); + }, + "*": async (event) => { + calls.push(`wildcard:${event.type}`); + }, + }, + }, + ]); + ctx.set(BundleKey, { + graph: { root: {} }, + resolvedAgent: { config: {} }, + turnAgent: { id: "parent" }, + subagentRegistry: {}, + hookRegistry, + } as CompiledBundle); + vi.mocked(deserializeContext).mockResolvedValue(ctx); + vi.mocked(serializeContext).mockImplementation((context) => ({ + title: context.get(SessionTitleKey), + channelState: context.get(ChannelKey)?.state, + })); + const chunks: Uint8Array[] = []; + const stream = new WritableStream({ + write(chunk) { + calls.push("stream"); + chunks.push(chunk); + }, + }); + const emitted = emitSubagentEventStep({ + event: { + type: "subagent.completed", + data: { callId: "call", subagentName: "research", output: "done" }, + }, + sessionWritable: stream, + serializedContext: {}, + sessionState: createTestSessionState({ sessionId: "parent" }), + }); + if (fails) { + await expect(emitted).rejects.toThrow("completion subscriber failed"); + expect(calls).toEqual(["adapter", "stream", "typed"]); + } else { + await expect(emitted).resolves.toEqual({ + serializedContext: { title: "Completed research", channelState: { completed: true } }, + }); + expect(calls).toEqual(["adapter", "stream", "typed", "wildcard:subagent.completed"]); + } + expect(chunks).toHaveLength(1); + expect(new TextDecoder().decode(chunks[0])).toContain('"type":"subagent.completed"'); + expect(stream.locked).toBe(false); + }, +); diff --git a/packages/eve/src/execution/tools/subagent/emit-event-step.ts b/packages/eve/src/execution/tools/subagent/emit-event-step.ts new file mode 100644 index 000000000..5ac31aede --- /dev/null +++ b/packages/eve/src/execution/tools/subagent/emit-event-step.ts @@ -0,0 +1,38 @@ +import { contextStorage } from "#context/container.js"; +import { deserializeContext, serializeContext } from "#context/serialize.js"; +import { readDurableSession, type DurableSessionState } from "#execution/durable-session-store.js"; +import { resolveEffectiveAgentRuntime } from "#execution/effective-agent-config.js"; +import { createSessionEventSink } from "#execution/session/event-sink.js"; +import type { UnstampedMessageStreamEvent } from "#protocol/message.js"; +import { BundleKey, ChannelKey } from "#runtime/sessions/runtime-context-keys.js"; + +/** Emits an agent invocation event on the parent stream. */ +export async function emitSubagentEventStep(input: { + readonly event: UnstampedMessageStreamEvent; + readonly sessionWritable: WritableStream; + readonly serializedContext: Record; + readonly sessionState: DurableSessionState; +}): Promise<{ readonly serializedContext: Record }> { + "use step"; + + const ctx = await deserializeContext(input.serializedContext); + const bundle = ctx.require(BundleKey); + const session = readDurableSession(input.sessionState); + const sink = createSessionEventSink({ + abortSignal: undefined, + adapter: ctx.require(ChannelKey), + bundle, + ctx, + effectiveAgent: resolveEffectiveAgentRuntime(bundle, ctx), + instrumentation: undefined, + isFirstTurn: input.sessionState.emissionState.sequence === 0, + sessionWritable: input.sessionWritable, + sessionId: session.sessionId, + }); + try { + await contextStorage.run(ctx, () => sink.handleEvent(input.event, session.history)); + } finally { + sink.release(); + } + return { serializedContext: serializeContext(ctx) }; +} diff --git a/packages/eve/src/execution/tools/subagent/invoke-agent.test.ts b/packages/eve/src/execution/tools/subagent/invoke-agent.test.ts index 803d37271..b28b6beb2 100644 --- a/packages/eve/src/execution/tools/subagent/invoke-agent.test.ts +++ b/packages/eve/src/execution/tools/subagent/invoke-agent.test.ts @@ -115,7 +115,10 @@ describe("background agent invocation routing", () => { output: { findings: ["available"] }, subagentName: "research", }; - const replies: AgentInvocationReply[] = [{ kind: "runtime-action-result", results: [result] }]; + const replies: AgentInvocationReply[] = [ + { kind: "runtime-action-result", results: [result] }, + { kind: "agent-settled", callId: result.callId }, + ]; mocks.createHook.mockReturnValue({ [Symbol.asyncIterator]: () => ({ next: async () => @@ -180,6 +183,7 @@ describe("background agent invocation routing", () => { } as never, ], }, + { kind: "agent-settled", callId: `call-1:${token}` }, ]; mocks.createHook.mockReturnValueOnce({ [Symbol.asyncIterator]: () => ({ @@ -249,6 +253,10 @@ describe("background agent invocation routing", () => { done: false, value: { kind: "runtime-action-result", results: [result] }, }) + .mockResolvedValueOnce({ + done: false, + value: { kind: "agent-settled", callId: "call-1:agent-reply" }, + }) .mockResolvedValue({ done: true }), }), token: "agent-reply", @@ -284,6 +292,66 @@ describe("background agent invocation routing", () => { }); }); + it.each([false, true])( + "waits for settlement acknowledgement before returning a child result (error=%s)", + async (isError) => { + const acknowledgement = Promise.withResolvers>(); + const childResult = { + callId: "call-1:agent-reply", + kind: "subagent-result", + origin: "child", + isError, + output: isError ? "failed" : "done", + subagentName: "research", + }; + const next = vi + .fn() + .mockResolvedValueOnce({ + done: false, + value: { kind: "runtime-action-result", results: [childResult] }, + }) + .mockImplementationOnce(() => acknowledgement.promise); + mocks.createHook.mockReturnValue({ + token: "agent-reply", + [Symbol.asyncIterator]: () => ({ next }), + }); + const ctx = { callId: "call-1" } as ToolContext; + attachWorkflowToolRunContext(ctx, { + from: { + callId: "call-1", + execution: "blocking", + input: {}, + runId: "run-1", + sequence: 0, + stepIndex: 0, + toolName: "research", + turnId: "turn-1", + }, + owner: { inbox: "owner" }, + }); + const returned = vi.fn(); + const output = agent(ctx, "research", { message: "Find it" }).then( + (value) => { + returned(); + return value; + }, + (error) => { + returned(); + return error; + }, + ); + await vi.waitFor(() => expect(next).toHaveBeenCalledTimes(2)); + expect(returned).not.toHaveBeenCalled(); + expect(mocks.disposeHook).not.toHaveBeenCalled(); + acknowledgement.resolve({ + done: false, + value: { kind: "agent-settled", callId: childResult.callId }, + }); + expect(await output).toBe(childResult.output); + expect(mocks.disposeHook).toHaveBeenCalledOnce(); + }, + ); + it("rejects dispatch failures without reporting a child settlement", async () => { const failure = { callId: "call-1:agent-reply", @@ -304,6 +372,10 @@ describe("background agent invocation routing", () => { done: false, value: { kind: "runtime-action-result", results: [failure] }, }) + .mockResolvedValueOnce({ + done: false, + value: { kind: "agent-settled", callId: "call-1:agent-reply" }, + }) .mockResolvedValue({ done: true }), }), token: "agent-reply", @@ -380,6 +452,7 @@ describe("background agent invocation routing", () => { } as never, ], }, + { kind: "agent-settled", callId: "call-1:agent-reply" }, ]; mocks.createHook.mockReturnValue({ [Symbol.asyncIterator]: () => ({ @@ -464,6 +537,7 @@ describe("background agent invocation routing", () => { } as never, ], }, + { kind: "agent-settled", callId: "call-1:agent-reply" }, ]; mocks.createHook.mockReturnValue({ [Symbol.asyncIterator]: () => ({ diff --git a/packages/eve/src/execution/tools/subagent/invoke-agent.ts b/packages/eve/src/execution/tools/subagent/invoke-agent.ts index 32973a3f9..94da39f05 100644 --- a/packages/eve/src/execution/tools/subagent/invoke-agent.ts +++ b/packages/eve/src/execution/tools/subagent/invoke-agent.ts @@ -49,7 +49,10 @@ export type AgentInvocationEvent = | SubagentAuthorizationEventHookPayload | SubagentInputRequestHookPayload; -export type AgentInvocationReply = AgentInvocationEvent | RuntimeActionResultHookPayload; +export type AgentInvocationReply = + | AgentInvocationEvent + | RuntimeActionResultHookPayload + | { readonly kind: "agent-settled"; readonly callId: string }; /** Invokes an agent from a workflow tool. */ export async function agent( @@ -71,18 +74,8 @@ export async function agent( export async function invokeAgent( ctx: ToolContext, input: InternalAgentInput, - options: { readonly invocationId: string; readonly returnResult: true }, -): Promise; -export async function invokeAgent( - ctx: ToolContext, - input: InternalAgentInput, - options?: { readonly invocationId?: string; readonly returnResult?: false }, -): Promise; -export async function invokeAgent( - ctx: ToolContext, - input: InternalAgentInput, - options: { readonly invocationId?: string; readonly returnResult?: boolean } = {}, -): Promise { + options: { readonly invocationId?: string } = {}, +): Promise { validateAgentInput(input); const run = readWorkflowToolRunRef(ctx); const owner = readWorkflowToolRunOwner(ctx); @@ -114,13 +107,24 @@ export async function invokeAgent( replyTo: replies.token, request: { kind: "agent-settled", result }, }); + // The enclosing workflow cannot finish before its owner applies settlement. + for (;;) { + const acknowledgement = await nextAgentReply(iterator, ctx.abortSignal); + if (acknowledgement.done) + throw new Error(`Agent "${input.target}" closed before settlement.`); + if ( + acknowledgement.value.kind === "agent-settled" && + acknowledgement.value.callId === invocationId + ) + break; + } } - if (options.returnResult === true && run.execution === "blocking") return result; if (result.isError === true) throw result.output; return result.output; } continue; } + if (reply.kind === "agent-settled") continue; if (reply.kind === "subagent-input-request") { await resumeHookStep(owner.inbox, { kind: "request", diff --git a/packages/eve/src/execution/tools/subagent/invoke-origin.test.ts b/packages/eve/src/execution/tools/subagent/invoke-origin.test.ts index 8bc84df63..8c62f1e63 100644 --- a/packages/eve/src/execution/tools/subagent/invoke-origin.test.ts +++ b/packages/eve/src/execution/tools/subagent/invoke-origin.test.ts @@ -5,7 +5,7 @@ import { prepareActionDispatch } from "#execution/coordination-dispatch-shared.j import { createDurableSessionState } from "#execution/durable-session-store.js"; import { setHarnessEmissionState } from "#harness/emission-state.js"; import { BundleKey } from "#runtime/sessions/runtime-context-keys.js"; -import { recordSessionTask } from "#tasks/session-index.js"; +import { registerWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import { prepareOwnerAgentInvocation } from "./invoke-preparation.js"; vi.mock("#context/serialize.js", () => ({ deserializeContext: vi.fn() })); @@ -38,7 +38,7 @@ describe("background invocation origin", () => { it.each([true, false])( "uses the task's creating turn after the parent advances: task=%s", async (background) => { - const session = recordSessionTask( + const session = registerWorkflowToolRun( setHarnessEmissionState( { agent: { dynamicModel: true, system: "", tools: [] }, @@ -50,13 +50,16 @@ describe("background invocation origin", () => { { sessionStarted: true, sequence: 3, stepIndex: 2, turnId: "turn-3" }, ), { - taskId: "task", - taskRunId: "task-run", - taskInboxToken: "task-inbox", - createdByTurnId: "turn-1", - createdByStepIndex: 0, - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "subagent", name: "research", agentId: "agent" }, + callId: "task", + toolName: { kind: "subagent", name: "research", agentId: "agent" }.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-inbox" }, + task: { + taskId: "task", + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "subagent", name: "research", agentId: "agent" }, + }, }, ); await prepareOwnerAgentInvocation({ diff --git a/packages/eve/src/execution/tools/subagent/invoke-preparation.ts b/packages/eve/src/execution/tools/subagent/invoke-preparation.ts index 30c643158..135dbbe0a 100644 --- a/packages/eve/src/execution/tools/subagent/invoke-preparation.ts +++ b/packages/eve/src/execution/tools/subagent/invoke-preparation.ts @@ -38,7 +38,7 @@ import { import type { SubagentStartTarget } from "#execution/tools/subagent/start.js"; import type { SubagentInputSource } from "#subagents/tool.js"; import { createLogger } from "#internal/logging.js"; -import { findSessionTaskEntry } from "#tasks/session-index.js"; +import { findBackgroundWorkflowToolRun } from "#harness/workflow-tool-runs.js"; const log = createLogger("execution.agent-invocation"); @@ -67,7 +67,7 @@ export async function prepareOwnerAgentInvocation(input: { const task = input.taskId === undefined ? undefined - : findSessionTaskEntry(durableSession.state, input.taskId); + : findBackgroundWorkflowToolRun(durableSession.state, input.taskId); const action = resolveAgentInvocationAction({ ctx, input: input.invocation, @@ -78,8 +78,8 @@ export async function prepareOwnerAgentInvocation(input: { requests: [action], event: { ...event, - stepIndex: task?.createdByStepIndex ?? event.stepIndex, - turnId: task?.createdByTurnId ?? activeTurnId(event), + stepIndex: task?.origin.stepIndex ?? event.stepIndex, + turnId: task?.origin.turnId ?? activeTurnId(event), }, }, ctx, diff --git a/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.ts b/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.ts index 4fff302f7..1f81dcabc 100644 --- a/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.ts +++ b/packages/eve/src/execution/tools/subagent/invoke-step.integration.test.ts @@ -1,3 +1,4 @@ +import { getTurnUsageState } from "#harness/turn-tag-state.js"; import { beforeEach, describe, expect, it, vi } from "vitest"; import { createDurableSessionState, readDurableSession } from "#execution/durable-session-store.js"; @@ -159,7 +160,7 @@ describe("blocking workflow agent continuation", () => { expect(startSubagent).toHaveBeenCalledWith(expect.objectContaining({ activityObserver })); }); - it("reuses one handle for two calls from the same workflow run", async () => { + it("settles each invocation once and ignores an old result after the handle is reused", async () => { const session = { agent: { dynamicModel: true as const, system: "", tools: [] }, compaction: { recentWindowSize: 5, threshold: 10_000 }, @@ -171,6 +172,7 @@ describe("blocking workflow agent continuation", () => { }), }; let sessionState = createDurableSessionState({ session }); + let previousSettlement: Parameters[0] | undefined; for (const [index, message] of ["first", "second"].entries()) { const callId = `workflow-call:${String(index)}`; @@ -193,7 +195,16 @@ describe("blocking workflow agent continuation", () => { expect.objectContaining({ identity, ownerId: "workflow-run-1", phase: "claimed" }), ]); - const settled = await settleTaskAgentInvocationStep({ + if (previousSettlement !== undefined) { + const stale = await settleTaskAgentInvocationStep({ + ...previousSettlement, + sessionState: dispatched.sessionState, + }); + expect(stale.settled).toBe(false); + expect(stale.completion).toBeUndefined(); + expect(stale.sessionState).toBe(dispatched.sessionState); + } + const settlement: Parameters[0] = { serializedContext: dispatched.serializedContext ?? {}, ownerId: "workflow-run-1", result: { @@ -206,15 +217,34 @@ describe("blocking workflow agent continuation", () => { usageDelta: { cacheReadTokens: 0, cacheWriteTokens: 0, - inputTokens: 0, - outputTokens: 0, + inputTokens: 2, + outputTokens: 3, }, }, output: message, subagentName: identity.name, }, sessionState: dispatched.sessionState, + }; + const settled = await settleTaskAgentInvocationStep(settlement); + expect(settled.settled).toBe(true); + expect(settled.completion).toEqual({ + type: "subagent.completed", + data: { callId, subagentName: identity.name, output: message }, }); + const restored = JSON.parse(JSON.stringify(settled.sessionState)); + const duplicate = await settleTaskAgentInvocationStep({ + ...settlement, + sessionState: restored, + }); + expect(duplicate.settled).toBe(false); + expect(duplicate.completion).toBeUndefined(); + expect(duplicate.sessionState).toBe(restored); + expect(getTurnUsageState(restored.snapshot.session.state)?.session).toMatchObject({ + inputTokens: 2 * (index + 1), + outputTokens: 3 * (index + 1), + }); + previousSettlement = settlement; sessionState = settled.sessionState; expect(getAgentHandleStore(sessionState.snapshot.session.state)?.handles).toEqual([ { address, identity, phase: "available" }, diff --git a/packages/eve/src/execution/tools/subagent/invoke-step.test.ts b/packages/eve/src/execution/tools/subagent/invoke-step.test.ts index 5f4419466..cb86468e3 100644 --- a/packages/eve/src/execution/tools/subagent/invoke-step.test.ts +++ b/packages/eve/src/execution/tools/subagent/invoke-step.test.ts @@ -11,8 +11,7 @@ import { startSubagent } from "#execution/tools/subagent/start.js"; import { prepareOwnerAgentInvocation } from "#execution/tools/subagent/invoke-preparation.js"; import { readDurableSession } from "#execution/durable-session-store.js"; import { getAgentHandleStore, setAgentHandleStore } from "#subagents/handles/store.js"; -import { readLatestTaskView } from "#execution/tasks/parent/run-parent.js"; -import { recordSessionTask } from "#tasks/session-index.js"; +import { registerWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import { AuthKey, InitiatorAuthKey, @@ -41,7 +40,6 @@ vi.mock("#execution/durable-session-store.js", async (importOriginal) => ({ ...(await importOriginal()), readDurableSession: vi.fn(), })); -vi.mock("#execution/tasks/parent/run-parent.js", () => ({ readLatestTaskView: vi.fn() })); const action = { callId: "call-1", description: "Research", @@ -252,21 +250,20 @@ describe("owner agent invocation dispatch", () => { rootSessionId: "root-session", rootTurnId: "root-turn", }; - const indexedSession = recordSessionTask(session as never, { - activityWorkIdentity: taskWork, - createdByTurnId: "turn-1", - dispatchContext: taskDispatchContext, - metadata: { kind: taskKind, name: "research" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + const indexedSession = registerWorkflowToolRun(session, { + callId: "task-1", + toolName: { kind: taskKind, name: "research" }.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + activityWorkIdentity: taskWork, + dispatchContext: taskDispatchContext, + metadata: { kind: taskKind, name: "research" }, + taskId: "task-1", + }, }); vi.mocked(readDurableSession).mockReturnValue(indexedSession as never); - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: { kind: "subagent", name: "research" }, - status: "working", - taskId: "task-1", - }); vi.mocked(prepareOwnerAgentInvocation).mockResolvedValue({ ...prepared, auth: creatorAuth, @@ -341,22 +338,21 @@ describe("owner agent invocation dispatch", () => { [AuthKey.name]: null, [InitiatorAuthKey.name]: sessionInitiatorAuth, }; - const indexedSession = recordSessionTask(session as never, { - createdByTurnId: "turn-1", - dispatchContext: { - auth: { current: null, initiator: sessionInitiatorAuth }, + const indexedSession = registerWorkflowToolRun(session, { + callId: "task-1", + toolName: "research", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { + auth: { current: null, initiator: sessionInitiatorAuth }, + }, + metadata: { kind: "subagent", name: "research" }, + taskId: "task-1", }, - metadata: { kind: "subagent", name: "research" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", }); vi.mocked(readDurableSession).mockReturnValue(indexedSession as never); - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: { kind: "subagent", name: "research" }, - status: "working", - taskId: "task-1", - }); vi.mocked(prepareOwnerAgentInvocation).mockResolvedValue({ ...prepared, auth: null, @@ -389,30 +385,29 @@ describe("owner agent invocation dispatch", () => { expect(result).toMatchObject({ serializedContext }); }); - it("rejects only nested dispatch for a legacy task without creator context", async () => { + it("rejects missing creator context before dispatching with receiver authentication", async () => { vi.mocked(readDurableSession).mockReturnValue({ ...session, state: { ...session.state, - "eve.tasks": { - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { - createdByTurnId: "turn-1", - metadata: { kind: "subagent", name: "research" }, - taskId: "task-1", - taskInboxToken: "task-token", - taskRunId: "task-run", + callId: "task-1", + toolName: "research", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + metadata: { kind: "subagent", name: "research" }, + taskId: "task-1", + }, }, ], - version: 2, }, }, } as never); - vi.mocked(readLatestTaskView).mockResolvedValue({ - metadata: { kind: "subagent", name: "research" }, - status: "working", - taskId: "task-1", - }); await expect( dispatchTaskAgentInvocationStep({ @@ -427,13 +422,7 @@ describe("owner agent invocation dispatch", () => { sessionState: { sessionId: "parent" } as never, taskId: "task-1", }), - ).resolves.toMatchObject({ - kind: "failed", - result: { - isError: true, - output: { code: "AGENT_INVOCATION_AUTH_UNAVAILABLE" }, - }, - }); + ).rejects.toThrow("Corrupt workflow tool run registry"); expect(prepareOwnerAgentInvocation).not.toHaveBeenCalled(); }); @@ -484,16 +473,36 @@ describe("owner agent invocation dispatch", () => { }); describe("task-owned agent settlement", () => { - it.each(["parked", "terminal"] as const)("applies a %s child outcome", async (kind) => { + it.each( + (["parked", "terminal"] as const).flatMap((kind) => + ([undefined, "tool", "subagent"] as const).map((taskKind) => ({ kind, taskKind })), + ), + )("applies a $kind child outcome under $taskKind ownership", async ({ kind, taskKind }) => { const claimed = { ...availableRecord, + callId: "call-1", operationId: "operation-1", phase: "claimed" as const, ownerId: "task-1", }; + const owner = + taskKind === undefined + ? session + : registerWorkflowToolRun(session, { + callId: "call-1", + toolName: "research", + lifetime: "session", + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "run", hookToken: "hook" }, + task: { + taskId: "task-1", + metadata: { kind: taskKind, name: "research" }, + dispatchContext: { auth: { current: null, initiator: null } }, + }, + }); vi.mocked(readDurableSession).mockReturnValue({ - ...session, - state: setAgentHandleStore(undefined, { handles: [claimed] }), + ...owner, + state: setAgentHandleStore(owner.state, { handles: [claimed] }), } as never); const settled = await settleTaskAgentInvocationStep({ @@ -517,9 +526,17 @@ describe("task-owned agent settlement", () => { }, ownerId: "task-1", sessionState: {} as never, - taskId: "task-1", + taskId: taskKind === undefined ? undefined : "task-1", }); + expect(settled.completion).toEqual( + taskKind === "subagent" + ? undefined + : { + type: "subagent.completed", + data: { callId: "call-1", subagentName: "research", output: "done" }, + }, + ); const handles = getAgentHandleStore(settled.sessionState.snapshot.session.state)?.handles ?? []; expect(handles).toEqual( kind === "parked" ? [expect.objectContaining({ phase: "available" })] : [], @@ -529,6 +546,7 @@ describe("task-owned agent settlement", () => { it("releases every remaining claim for a completed workflow run", async () => { const claimed = { ...availableRecord, + callId: "call-1", operationId: "operation-1", ownerId: "workflow-run-1", phase: "claimed" as const, @@ -551,6 +569,7 @@ describe("task-owned agent settlement", () => { it("parks every remaining claim for a cancelled workflow run", async () => { const claimed = { ...availableRecord, + callId: "call-1", operationId: "operation-1", ownerId: "workflow-run-1", phase: "claimed" as const, @@ -579,6 +598,7 @@ describe("task-owned agent settlement", () => { it("keeps a cancelled parked child resumable after settlement", async () => { const claimed = { ...availableRecord, + callId: "call-1", operationId: "operation-1", phase: "claimed" as const, ownerId: "workflow-run-1", @@ -611,6 +631,7 @@ describe("task-owned agent settlement", () => { sessionState: {} as never, }); + expect(settled.completion).toBeUndefined(); expect(getAgentHandleStore(settled.sessionState.snapshot.session.state)?.handles).toEqual([ { address: availableRecord.address, diff --git a/packages/eve/src/execution/tools/subagent/invoke-step.ts b/packages/eve/src/execution/tools/subagent/invoke-step.ts index 5daa651ad..97de3979b 100644 --- a/packages/eve/src/execution/tools/subagent/invoke-step.ts +++ b/packages/eve/src/execution/tools/subagent/invoke-step.ts @@ -10,7 +10,11 @@ import type { AgentInvocationRequest } from "#execution/tools/subagent/invoke-ag import type { RuntimeSubagentResult } from "#shared/action-types.js"; import type { HandleEventFn } from "#harness/types.js"; import type { ActivityWorkIdentityV1 } from "#protocol/activity.js"; -import { createSubagentCalledEvent, type SubagentCalledStreamEvent } from "#protocol/message.js"; +import { + createSubagentCalledEvent, + type SubagentCalledStreamEvent, + type SubagentCompletedStreamEvent, +} from "#protocol/message.js"; import { workflowEntryReference } from "#execution/workflow-runtime.js"; import { getWorkflowMetadata } from "#compiled/@workflow/core/index.js"; import { resolveWorkflowCallbackBaseUrl } from "#execution/workflow-callback-url.js"; @@ -22,7 +26,6 @@ import { type DurableSessionState, } from "#execution/durable-session-store.js"; import { projectToDurableSession } from "#execution/session.js"; -import { readLatestTaskView } from "#execution/tasks/parent/run-parent.js"; import { getAgentHandleStore, writeHandles, @@ -39,8 +42,11 @@ import { AGENT_UNREACHABLE, formatAgentBusyMessage, } from "#subagents/agent-handle-errors.js"; -import { findSessionTaskEntry } from "#tasks/session-index.js"; -import { isTerminalTaskStatus } from "#tasks/types.js"; +import { + readWorkflowTaskView, + findBackgroundWorkflowToolRun, + type TaskAgentDispatchContext, +} from "#harness/workflow-tool-runs.js"; import type { RuntimeSubagentChildResult } from "#shared/action-types.js"; import { clearProxyInputRequestsForChild, @@ -62,7 +68,6 @@ import { SessionDynamicSubagentSelectionsKey, TurnDynamicSubagentSelectionsKey, } from "#context/keys.js"; -import type { TaskAgentDispatchContext } from "#tasks/session-index.js"; export type AgentInvocationDispatchResult = | { @@ -349,18 +354,15 @@ export async function dispatchTaskAgentInvocationStep( let taskDispatchContext: TaskAgentDispatchContext | undefined; if (input.taskId !== undefined) { const session = readDurableSession(input.sessionState); - const entry = findSessionTaskEntry(session.state, input.taskId); + const entry = findBackgroundWorkflowToolRun(session.state, input.taskId); if (entry === undefined) return { kind: "not-admitted", sessionState: input.sessionState }; - const view = await readLatestTaskView({ taskRunId: entry.taskRunId }); - if (view === undefined || isTerminalTaskStatus(view.status)) { + const view = readWorkflowTaskView(entry.task); + if (view !== undefined) { return { kind: "not-admitted", sessionState: input.sessionState }; } - if ("legacy" in entry.dispatchContext) { - return missingTaskDispatchContext(input); - } - taskDispatchContext = entry.dispatchContext; - if (entry.metadata.kind === "subagent") { - activityWorkIdentity = entry.activityWorkIdentity; + taskDispatchContext = entry.task.dispatchContext; + if (entry.task.metadata.kind === "subagent") { + activityWorkIdentity = entry.task.activityWorkIdentity; } } const dispatched = await dispatchAgentInvocation({ @@ -375,26 +377,6 @@ export async function dispatchTaskAgentInvocationStep( return dispatched; } -function missingTaskDispatchContext( - input: Parameters[0], -): TaskAgentInvocationDispatchResult { - return { - kind: "failed", - result: { - callId: input.request.invocationId, - isError: true, - kind: "subagent-result", - origin: "dispatch", - output: { - code: "AGENT_INVOCATION_AUTH_UNAVAILABLE", - message: "The background task predates captured authentication context.", - }, - subagentName: input.request.input.target, - }, - sessionState: input.sessionState, - }; -} - function applyTaskDispatchContext( parent: Record, task: TaskAgentDispatchContext, @@ -416,19 +398,34 @@ function applyTaskDispatchContext( /** Applies an owner-scoped child settlement to the parent session's canonical state. */ export async function settleTaskAgentInvocationStep(input: { - readonly accumulateUsage?: boolean; readonly ownerId: string; readonly result: RuntimeSubagentChildResult; readonly serializedContext: Record; readonly sessionState: DurableSessionState; readonly taskId?: string | undefined; }): Promise<{ + readonly settled: boolean; + readonly completion?: SubagentCompletedStreamEvent; readonly serializedContext: Record; readonly sessionState: DurableSessionState; }> { "use step"; const durable = readDurableSession(input.sessionState); + const handles = getAgentHandleStore(durable.state)?.handles ?? []; + const handle = handles.find( + (candidate) => + candidate.phase === "claimed" && + candidate.ownerId === input.ownerId && + candidate.callId === input.result.callId, + ); + if (handle?.phase !== "claimed") { + return { + settled: false, + serializedContext: input.serializedContext, + sessionState: input.sessionState, + }; + } const serializedContext = await flushAgentInvocationTraces( settleAgentInvocationTrace({ acceptedAtMs: Date.now(), @@ -437,17 +434,6 @@ export async function settleTaskAgentInvocationStep(input: { sessionId: durable.sessionId, }), ); - const handles = getAgentHandleStore(durable.state)?.handles ?? []; - const candidates = handles.filter( - (candidate) => candidate.phase === "claimed" && candidate.ownerId === input.ownerId, - ); - const handle = - candidates.find( - (candidate) => candidate.phase === "claimed" && candidate.callId === input.result.callId, - ) ?? (candidates.length === 1 ? candidates[0] : undefined); - if (handle?.phase !== "claimed") { - return { serializedContext, sessionState: input.sessionState }; - } const nextHandles = input.result.outcome.kind === "terminal" @@ -480,16 +466,38 @@ export async function settleTaskAgentInvocationStep(input: { : session : clearProxyInputRequestsForTask(session, input.taskId); } - if (input.accumulateUsage !== false) { - session = setTurnUsageState( - session, - accumulateSessionUsage({ - previous: getTurnUsageState(session.state), - usage: input.result.outcome.usageDelta, - }), - ); - } + session = setTurnUsageState( + session, + accumulateSessionUsage({ + previous: getTurnUsageState(session.state), + usage: input.result.outcome.usageDelta, + }), + ); + const task = + input.taskId === undefined + ? undefined + : findBackgroundWorkflowToolRun(session.state, input.taskId); + // A generated subagent task completes when the parent records its task outcome. + // Nested agents inside authored background workflows settle independently of that task. + const isTaskAgent = + task?.task.metadata.kind === "subagent" && task.callId === input.result.callId; + const completion: SubagentCompletedStreamEvent | undefined = + input.result.outcome.result.kind !== "succeeded" || isTaskAgent + ? undefined + : { + type: "subagent.completed", + data: { + callId: input.result.callId, + subagentName: input.result.subagentName, + output: + typeof input.result.output === "string" + ? input.result.output + : JSON.stringify(input.result.output), + }, + }; return { + settled: true, + completion, serializedContext, sessionState: replaceDurableSessionSnapshot({ session, state: input.sessionState }), }; diff --git a/packages/eve/src/execution/tools/subagent/task-activity.integration.test.ts b/packages/eve/src/execution/tools/subagent/task-activity.integration.test.ts index 64c4a3a52..2f349a0b5 100644 --- a/packages/eve/src/execution/tools/subagent/task-activity.integration.test.ts +++ b/packages/eve/src/execution/tools/subagent/task-activity.integration.test.ts @@ -12,7 +12,7 @@ import { } from "#execution/activity-work.js"; import { createActivitySnapshot, reduceActivityBatch } from "#execution/session-activity.js"; import { projectSessionActivity } from "#execution/session-activity-projection.js"; -import { projectTaskActivity } from "#execution/tasks/child/steps.js"; +import { projectTaskActivity } from "#execution/tasks/child/notify.js"; import { startSubagent, type SubagentStartTarget } from "#execution/tools/subagent/start.js"; import type { ActivityEventV1, ActivitySnapshotV1 } from "#protocol/activity.js"; import type { MessageStreamEvent } from "#protocol/message.js"; diff --git a/packages/eve/src/execution/tools/subagent/task-agent-requests.test.ts b/packages/eve/src/execution/tools/subagent/task-agent-requests.test.ts index f8201998c..4331a745e 100644 --- a/packages/eve/src/execution/tools/subagent/task-agent-requests.test.ts +++ b/packages/eve/src/execution/tools/subagent/task-agent-requests.test.ts @@ -1,3 +1,4 @@ +import { resumeHookStep } from "#execution/tools/workflow/resume-hook-step.js"; import { beforeEach, describe, expect, it, vi } from "vitest"; import { applyTaskAgentRequest } from "#execution/tools/subagent/task-agent-requests.js"; @@ -5,15 +6,15 @@ import { dispatchTaskAgentInvocationStep, settleTaskAgentInvocationStep, } from "#execution/tools/subagent/invoke-step.js"; -import { emitTaskSubagentCalledStep } from "#execution/tools/subagent/emit-called-step.js"; +import { emitSubagentEventStep } from "#execution/tools/subagent/emit-event-step.js"; import type { RuntimeSubagentChildResult } from "#shared/action-types.js"; vi.mock("#execution/tools/subagent/invoke-step.js", () => ({ dispatchTaskAgentInvocationStep: vi.fn(), settleTaskAgentInvocationStep: vi.fn(), })); -vi.mock("#execution/tools/subagent/emit-called-step.js", () => ({ - emitTaskSubagentCalledStep: vi.fn(), +vi.mock("#execution/tools/subagent/emit-event-step.js", () => ({ + emitSubagentEventStep: vi.fn(), })); vi.mock("#execution/tools/workflow/resume-hook-step.js", () => ({ resumeHookStep: vi.fn(), @@ -40,6 +41,7 @@ describe("workflow-owned agent requests", () => { const serializedContext = { "eve.test": "before" }; const flushedContext = { "eve.test": "after" }; vi.mocked(settleTaskAgentInvocationStep).mockResolvedValue({ + settled: false, serializedContext: flushedContext, sessionState, }); @@ -52,7 +54,6 @@ describe("workflow-owned agent requests", () => { const settled = await applyTaskAgentRequest(delivery, context); expect(settleTaskAgentInvocationStep).toHaveBeenCalledWith({ - accumulateUsage: undefined, ownerId: "workflow-run", result, serializedContext, @@ -64,6 +65,45 @@ describe("workflow-owned agent requests", () => { expect(replay).toEqual(settled); }); + it("emits completion and acknowledges only after settlement", async () => { + const updatedState = { sessionId: "updated" } as never; + vi.mocked(settleTaskAgentInvocationStep).mockResolvedValue({ + settled: true, + completion: { + type: "subagent.completed", + data: { callId: "nested", subagentName: "research", output: "done" }, + }, + serializedContext: {}, + sessionState: updatedState, + }); + const publication = Promise.withResolvers<{ serializedContext: Record }>(); + vi.mocked(emitSubagentEventStep).mockReturnValue(publication.promise); + const applying = applyTaskAgentRequest( + { ownerId: "workflow-run", replyTo: "reply", request: { kind: "agent-settled", result } }, + { sessionWritable: {} as never, serializedContext: {}, sessionState }, + ); + await vi.waitFor(() => expect(emitSubagentEventStep).toHaveBeenCalledOnce()); + expect(resumeHookStep).not.toHaveBeenCalled(); + publication.resolve({ serializedContext: {} }); + expect((await applying).sessionState).toBe(updatedState); + expect(resumeHookStep).toHaveBeenCalledExactlyOnceWith( + "reply", + { kind: "agent-settled", callId: "nested" }, + { ifPresent: true }, + ); + }); + + it("does not acknowledge a failed settlement", async () => { + vi.mocked(settleTaskAgentInvocationStep).mockRejectedValue(new Error("write failed")); + await expect( + applyTaskAgentRequest( + { ownerId: "workflow-run", replyTo: "reply", request: { kind: "agent-settled", result } }, + { sessionWritable: {} as never, serializedContext: {}, sessionState }, + ), + ).rejects.toThrow("write failed"); + expect(resumeHookStep).not.toHaveBeenCalled(); + }); + it("retains existing context when replaying a dispatch result without tracing state", async () => { const receiver = { attributes: {}, @@ -83,7 +123,7 @@ describe("workflow-owned agent requests", () => { kind: "dispatched", sessionState, }); - vi.mocked(emitTaskSubagentCalledStep).mockResolvedValue({ serializedContext }); + vi.mocked(emitSubagentEventStep).mockResolvedValue({ serializedContext }); const applied = await applyTaskAgentRequest( { @@ -98,10 +138,11 @@ describe("workflow-owned agent requests", () => { { sessionWritable: {} as never, serializedContext, sessionState }, ); - expect(emitTaskSubagentCalledStep).toHaveBeenCalledWith({ + expect(emitSubagentEventStep).toHaveBeenCalledWith({ event, sessionWritable: {}, serializedContext, + sessionState, }); expect(applied.serializedContext).toBe(serializedContext); }); diff --git a/packages/eve/src/execution/tools/subagent/task-agent-requests.ts b/packages/eve/src/execution/tools/subagent/task-agent-requests.ts index edb828778..80ad967fe 100644 --- a/packages/eve/src/execution/tools/subagent/task-agent-requests.ts +++ b/packages/eve/src/execution/tools/subagent/task-agent-requests.ts @@ -1,5 +1,5 @@ import type { DurableSessionState } from "#execution/durable-session-store.js"; -import { emitTaskSubagentCalledStep } from "#execution/tools/subagent/emit-called-step.js"; +import { emitSubagentEventStep } from "#execution/tools/subagent/emit-event-step.js"; import { dispatchTaskAgentInvocationStep, settleTaskAgentInvocationStep, @@ -8,7 +8,6 @@ import { resumeHookStep } from "#execution/tools/workflow/resume-hook-step.js"; import type { TaskAgentRequestDelivery } from "#tasks/types.js"; export interface AgentRequestDelivery { - readonly accumulateUsage?: boolean; readonly ownerId: string; readonly replyTo: TaskAgentRequestDelivery["replyTo"]; readonly request: TaskAgentRequestDelivery["request"]; @@ -39,15 +38,29 @@ export async function applyTaskAgentRequest( switch (request.kind) { case "agent-settled": { const settled = await settleTaskAgentInvocationStep({ - accumulateUsage: delivery.accumulateUsage, ownerId: delivery.ownerId, result: request.result, serializedContext: ctx.serializedContext, sessionState: ctx.sessionState, taskId: delivery.taskId, }); + let serializedContext = settled.serializedContext; + if (settled.completion !== undefined) { + const emitted = await emitSubagentEventStep({ + event: settled.completion, + sessionWritable: ctx.sessionWritable, + serializedContext, + sessionState: settled.sessionState, + }); + serializedContext = emitted.serializedContext; + } + await resumeHookStep( + delivery.replyTo, + { kind: "agent-settled", callId: request.result.callId }, + { ifPresent: true }, + ); return { - serializedContext: settled.serializedContext, + serializedContext, sessionState: settled.sessionState, }; } @@ -62,10 +75,11 @@ export async function applyTaskAgentRequest( }); switch (dispatched.kind) { case "dispatched": { - const emitted = await emitTaskSubagentCalledStep({ + const emitted = await emitSubagentEventStep({ event: dispatched.event, sessionWritable: ctx.sessionWritable, serializedContext: dispatched.serializedContext ?? ctx.serializedContext, + sessionState: dispatched.sessionState, }); return { serializedContext: emitted.serializedContext, diff --git a/packages/eve/src/execution/tools/subagent/task-cancel.test.ts b/packages/eve/src/execution/tools/subagent/task-cancel.test.ts new file mode 100644 index 000000000..1376ef339 --- /dev/null +++ b/packages/eve/src/execution/tools/subagent/task-cancel.test.ts @@ -0,0 +1,143 @@ +import { ContextContainer } from "#context/container.js"; +import { beforeEach, expect, it, vi } from "vitest"; +import { cancelBackgroundAgentTask, cancelAgentInvocationOwnerStep } from "./task-cancel.js"; +import { requestWorkflowTurnCancellation } from "#execution/workflow-runtime.js"; +import { createTestSessionState } from "#internal/testing/session-state.js"; +import { deserializeContext } from "#context/serialize.js"; +import { cancelRemoteAgentTurn, resolveRemoteAgentForAction } from "#subagents/remote-dispatch.js"; +import { setAgentHandleStore, type AgentHandle } from "#subagents/handles/store.js"; +import type { BackgroundWorkflowToolRun } from "#harness/workflow-tool-runs.js"; + +vi.mock("#execution/workflow-runtime.js", () => ({ requestWorkflowTurnCancellation: vi.fn() })); +vi.mock("#context/serialize.js", () => ({ deserializeContext: vi.fn() })); +vi.mock("#context/dynamic-subagent-lifecycle.js", () => ({ getDynamicSubagentSelection: vi.fn() })); +vi.mock("#subagents/remote-dispatch.js", () => ({ + cancelRemoteAgentTurn: vi.fn(), + resolveRemoteAgentForAction: vi.fn(), +})); +vi.mock("#internal/logging.js", () => ({ createLogger: vi.fn(() => ({})), logError: vi.fn() })); + +const entry: BackgroundWorkflowToolRun = { + callId: "call", + toolName: "research", + lifetime: "session", + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "run", hookToken: "hook" }, + task: { + taskId: "owner", + metadata: { kind: "tool", name: "research" }, + dispatchContext: { auth: { current: null, initiator: null } }, + }, +}; +const claimed = (id: string, ownerId = "owner"): AgentHandle => ({ + phase: "claimed", + ownerId, + operationId: `op-${id}`, + identity: { id, name: "research", nodeId: "subagents/research" }, + address: { kind: "agent/local", sessionId: id, continuationToken: `token-${id}` }, +}); +const session = { + state: setAgentHandleStore(undefined, { + handles: [ + claimed("child-a"), + claimed("child-b"), + claimed("other-child", "other-owner"), + { + phase: "available", + identity: { id: "idle", name: "research", nodeId: "subagents/research" }, + address: { kind: "agent/local", sessionId: "idle", continuationToken: "idle" }, + }, + ], + }), +}; + +beforeEach(() => vi.resetAllMocks()); + +it.each(["background", "waiting"])( + "cancels every claimed child of a %s owner, leaving unrelated children alone", + async (mode) => { + if (mode === "background") { + await cancelBackgroundAgentTask({ entry, session, serializedContext: {} }); + } else { + const sessionState = createTestSessionState(); + await cancelAgentInvocationOwnerStep({ + ownerId: "owner", + serializedContext: {}, + sessionState: { + ...sessionState, + snapshot: { session: { ...sessionState.snapshot.session, ...session } }, + }, + }); + } + expect(requestWorkflowTurnCancellation).toHaveBeenCalledTimes(2); + expect(requestWorkflowTurnCancellation).toHaveBeenCalledWith({ sessionId: "child-a" }); + expect(requestWorkflowTurnCancellation).toHaveBeenCalledWith({ sessionId: "child-b" }); + expect(deserializeContext).not.toHaveBeenCalled(); + }, +); + +it("propagates child cancellation failure so background cancellation can retry", async () => { + vi.mocked(requestWorkflowTurnCancellation).mockRejectedValueOnce(new Error("unavailable")); + await expect( + cancelBackgroundAgentTask({ entry, session, serializedContext: {} }), + ).rejects.toThrow("Failed to cancel owned agent turns"); + expect(requestWorkflowTurnCancellation).toHaveBeenCalledTimes(2); +}); + +it("waits for sibling cancellation requests before returning a failure", async () => { + const release = Promise.withResolvers(); + vi.mocked(requestWorkflowTurnCancellation) + .mockRejectedValueOnce(new Error("first child failed")) + .mockImplementationOnce(async () => { + await release.promise; + return { status: "accepted", sessionId: "child-b" }; + }); + const cancellation = cancelBackgroundAgentTask({ entry, session, serializedContext: {} }); + const rejected = vi.fn(); + cancellation.catch(rejected); + await Promise.resolve(); + await Promise.resolve(); + expect(rejected).not.toHaveBeenCalled(); + release.resolve(); + await expect(cancellation).rejects.toThrow("Failed to cancel owned agent turns"); +}); + +it("cancels remote children at their recorded address", async () => { + const remoteSession = { + state: setAgentHandleStore(undefined, { + handles: [ + { + ...claimed("remote"), + phase: "claimed", + ownerId: "owner", + operationId: "op-remote", + address: { + kind: "agent/remote", + sessionId: "remote", + url: "https://original.example", + callbackBaseUrl: "https://parent.example", + }, + }, + ], + }), + }; + const ctx = new ContextContainer(); + vi.spyOn(ctx, "require").mockReturnValue({ subagentRegistry: { subagentsByNodeId: new Map() } }); + vi.mocked(deserializeContext).mockResolvedValue(ctx); + vi.mocked(resolveRemoteAgentForAction).mockReturnValue({ + kind: "remote", + name: "research", + description: "Research", + nodeId: "subagents/research", + path: "/research", + logicalPath: "subagents/research", + sourceId: "research", + sourceKind: "module", + url: "https://new.example", + }); + await cancelBackgroundAgentTask({ entry, session: remoteSession, serializedContext: {} }); + expect(cancelRemoteAgentTurn).toHaveBeenCalledWith({ + remote: expect.objectContaining({ url: "https://original.example" }), + sessionId: "remote", + }); +}); diff --git a/packages/eve/src/execution/tools/subagent/task-cancel.ts b/packages/eve/src/execution/tools/subagent/task-cancel.ts index ebcb258a1..b1e3bdcc0 100644 --- a/packages/eve/src/execution/tools/subagent/task-cancel.ts +++ b/packages/eve/src/execution/tools/subagent/task-cancel.ts @@ -11,30 +11,13 @@ import { createLogger, logError } from "#internal/logging.js"; const log = createLogger("execution.agent-invocation-cancel"); -/** Cancels the active child turn owned by a background subagent task. */ +/** Cancels all child turns still owned by a background task. */ export const cancelBackgroundAgentTask: TaskExecutorCancel = async (input) => { if (input.session === undefined || input.serializedContext === undefined) return; - const session = input.session as RuntimeSession; - const handle = getAgentHandleStore(session.state)?.handles.find( - (candidate) => candidate.phase === "claimed" && candidate.ownerId === input.entry.taskId, - ); - if (handle === undefined || handle.phase !== "claimed") return; - if (handle.address.kind !== "agent/remote") { - await requestWorkflowTurnCancellation({ sessionId: handle.address.sessionId }); - return; - } - const ctx = await deserializeContext(input.serializedContext); - const bundle = ctx.require(BundleKey); - const selection = getDynamicSubagentSelection(ctx, handle.identity.nodeId); - const remote = resolveRemoteAgentForAction({ - dynamicRemoteAgent: selection?.kind === "remote" ? selection.remoteAgent : undefined, - nodeId: handle.identity.nodeId, - registry: bundle.subagentRegistry.subagentsByNodeId, - remoteAgentName: handle.identity.name, - }); - await cancelRemoteAgentTurn({ - remote: { ...remote, url: handle.address.url }, - sessionId: handle.address.sessionId, + await cancelAgentInvocationOwner({ + ownerId: input.entry.task.taskId, + serializedContext: input.serializedContext, + session: input.session, }); }; @@ -46,41 +29,54 @@ export async function cancelAgentInvocationOwnerStep(input: { }): Promise { "use step"; - const session = readDurableSession(input.sessionState); - const handles = (getAgentHandleStore(session.state)?.handles ?? []).filter( + try { + await cancelAgentInvocationOwner({ + ownerId: input.ownerId, + serializedContext: input.serializedContext, + session: readDurableSession(input.sessionState), + }); + } catch (error) { + logError(log, "failed to cancel workflow-owned agent turn", error, { ownerId: input.ownerId }); + } +} + +async function cancelAgentInvocationOwner(input: { + readonly ownerId: string; + readonly serializedContext: Record; + readonly session: Pick; +}): Promise { + const handles = (getAgentHandleStore(input.session.state)?.handles ?? []).filter( (candidate): candidate is Extract => candidate.phase === "claimed" && candidate.ownerId === input.ownerId, ); if (handles.length === 0) return; - const remoteContext = handles.some((handle) => handle.address.kind === "agent/remote") - ? await deserializeContext(input.serializedContext) - : undefined; - await Promise.all( + let remoteContext: ReturnType | undefined; + const results = await Promise.allSettled( handles.map(async (handle) => { - try { - if (handle.address.kind !== "agent/remote") { - await requestWorkflowTurnCancellation({ sessionId: handle.address.sessionId }); - return; - } - const bundle = remoteContext!.require(BundleKey); - const selection = getDynamicSubagentSelection(remoteContext!, handle.identity.nodeId); - const remote = resolveRemoteAgentForAction({ - dynamicRemoteAgent: selection?.kind === "remote" ? selection.remoteAgent : undefined, - nodeId: handle.identity.nodeId, - registry: bundle.subagentRegistry.subagentsByNodeId, - remoteAgentName: handle.identity.name, - }); - await cancelRemoteAgentTurn({ - remote: { ...remote, url: handle.address.url }, - sessionId: handle.address.sessionId, - }); - } catch (error) { - logError(log, "failed to cancel workflow-owned agent turn", error, { - agentId: handle.identity.id, - childSessionId: handle.address.sessionId, - ownerId: input.ownerId, - }); + if (handle.address.kind !== "agent/remote") { + await requestWorkflowTurnCancellation({ sessionId: handle.address.sessionId }); + return; } + remoteContext ??= deserializeContext(input.serializedContext); + const ctx = await remoteContext; + const bundle = ctx.require(BundleKey); + const selection = getDynamicSubagentSelection(ctx, handle.identity.nodeId); + const remote = resolveRemoteAgentForAction({ + dynamicRemoteAgent: selection?.kind === "remote" ? selection.remoteAgent : undefined, + nodeId: handle.identity.nodeId, + registry: bundle.subagentRegistry.subagentsByNodeId, + remoteAgentName: handle.identity.name, + }); + await cancelRemoteAgentTurn({ + remote: { ...remote, url: handle.address.url }, + sessionId: handle.address.sessionId, + }); }), ); + const failures = results.filter((result) => result.status === "rejected"); + if (failures.length > 0) + throw new AggregateError( + failures.map((result) => result.reason), + "Failed to cancel owned agent turns.", + ); } diff --git a/packages/eve/src/execution/tools/workflow/background.test.ts b/packages/eve/src/execution/tools/workflow/background.test.ts index 110537d6d..4a1533ee8 100644 --- a/packages/eve/src/execution/tools/workflow/background.test.ts +++ b/packages/eve/src/execution/tools/workflow/background.test.ts @@ -4,7 +4,7 @@ import { jsonSchema } from "ai"; import { createWorkflowToolHarnessDefinition, parseWorkflowToolInput } from "./background.js"; describe("createWorkflowToolHarnessDefinition", () => { - it("preserves subagent resultKind on workflow-backed tools", () => { + it("preserves agent identity on workflow-backed tools", () => { expect( createWorkflowToolHarnessDefinition({ definition: { @@ -14,10 +14,10 @@ describe("createWorkflowToolHarnessDefinition", () => { inputSchema: jsonSchema({ type: "object" }), name: "research", }, - resultKind: "subagent", + nodeId: "subagents/research", workflowId: "workflow//eve//subagentToolExecuteWorkflow", }), - ).toMatchObject({ resultKind: "subagent" }); + ).toMatchObject({ nodeId: "subagents/research" }); }); }); diff --git a/packages/eve/src/execution/tools/workflow/background.ts b/packages/eve/src/execution/tools/workflow/background.ts index d046fb8b7..e1d9af1d1 100644 --- a/packages/eve/src/execution/tools/workflow/background.ts +++ b/packages/eve/src/execution/tools/workflow/background.ts @@ -2,14 +2,14 @@ import type { HarnessToolDefinition } from "#harness/execute-tool.js"; import type { PreparedRuntimeTool } from "#runtime/sessions/turn.js"; import { parseJsonObject, type JsonValue } from "#shared/json.js"; import type { ToolExecuteOptions } from "#tools/definition.js"; -import type { TaskExec } from "#tools/task.js"; import { UNSPECIFIED_INPUT_SCHEMA, toInputSchema, toOutputSchema } from "#tools/schema.js"; export interface WorkflowToolHarnessDefinitionInput { readonly definition: HarnessToolDefinition; readonly executeInput?: (input: unknown) => JsonValue; + /** Selected agent definition's runtime graph ID; absent for authored workflow tools. */ readonly nodeId?: string; - readonly resultKind?: "subagent" | "tool"; + readonly workflowId: string; } @@ -20,7 +20,6 @@ export function createWorkflowToolHarnessDefinition( const workflow = { executeInput: input.executeInput, nodeId: input.nodeId, - resultKind: input.resultKind, workflowId: input.workflowId, }; if (definition.execution !== "background") { @@ -57,7 +56,6 @@ export function createPreparedWorkflowToolHarnessDefinition( rootOnly: tool.rootOnly, }, nodeId: tool.task.nodeId, - resultKind: tool.task.resultKind, workflowId: tool.task.workflowId, }; return createWorkflowToolHarnessDefinition(input); @@ -68,7 +66,7 @@ export function createWorkflowToolBackgroundExecute(input: { readonly toolName: string; readonly workflowId: string; }): NonNullable { - return (_toolInput: unknown, _options: ToolExecuteOptions, _task?: TaskExec): never => { + return (_toolInput: unknown, _options: ToolExecuteOptions): never => { throw new Error( `Background workflow tool "${input.toolName}" must be started by the task runtime (${input.workflowId}).`, ); diff --git a/packages/eve/src/execution/tools/workflow/body.ts b/packages/eve/src/execution/tools/workflow/body.ts index b51a22b45..4ae20ffa8 100644 --- a/packages/eve/src/execution/tools/workflow/body.ts +++ b/packages/eve/src/execution/tools/workflow/body.ts @@ -15,7 +15,6 @@ import { normalizeSerializableError } from "#execution/workflow-errors.js"; import { readRegisteredWorkflow } from "#execution/workflow-registry.js"; import type { JsonObject, JsonValue } from "#shared/json.js"; import type { ToolContext } from "#tools/definition.js"; -import { createTaskMessage, type TaskExec } from "#tools/task.js"; export interface WorkflowBodyDefinition { /** Snapshot added for new runs; absent only when resuming an older durable payload. */ @@ -23,10 +22,9 @@ export interface WorkflowBodyDefinition { readonly callId: string; readonly executeInput?: JsonValue; readonly input: JsonObject; - readonly resultKind?: "subagent" | "tool"; + readonly session: SessionContext["session"]; readonly stepIndex: number; - readonly taskId?: string; readonly toolName: string; readonly workflowId: string; } @@ -43,7 +41,6 @@ export interface WorkflowBodyResult { type WorkflowToolExecute = ( input: unknown, ctx: WorkflowToolContext, - task?: TaskExec, ) => Promise | AsyncIterable; /** Executes one registered workflow body and reports progress to its owner. */ @@ -64,8 +61,7 @@ export async function executeWorkflowBody( try { const execute = resolveWorkflowToolExecute(input); - const task = input.execution === "background" ? createWorkflowTaskExec(input) : undefined; - const result = execute(input.executeInput ?? input.input, ctx, task); + const result = execute(input.executeInput ?? input.input, ctx); let output: JsonValue; if (!isAsyncIterable(result)) { output = await result; @@ -111,7 +107,6 @@ export function createWorkflowBodyRef( callId: input.callId, execution: input.execution, input: input.input, - resultKind: input.resultKind, runId: input.runId ?? getWorkflowMetadata().workflowRunId, sequence: input.session.turn.sequence, stepIndex: input.stepIndex, @@ -168,22 +163,6 @@ function createWorkflowBodyContext( return ctx; } -function createWorkflowTaskExec(input: WorkflowBodyInput): TaskExec { - if (input.taskId === undefined) { - throw new Error(`Background workflow tool "${input.toolName}" has no task id.`); - } - return { - binding: { taskId: input.taskId, token: input.taskId }, - postMessage: createTaskMessage, - send() { - throw new Error("task.send() was replaced by yielded task descriptors."); - }, - session: undefined as never, - task: undefined as never, - taskId: input.taskId, - }; -} - function isAsyncIterable(value: unknown): value is AsyncIterable { return ( typeof value === "object" && diff --git a/packages/eve/src/execution/tools/workflow/cancel.test.ts b/packages/eve/src/execution/tools/workflow/cancel.test.ts index 95657b873..5cae3390f 100644 --- a/packages/eve/src/execution/tools/workflow/cancel.test.ts +++ b/packages/eve/src/execution/tools/workflow/cancel.test.ts @@ -1,12 +1,13 @@ -import { beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { EntityConflictError, HookNotFoundError } from "#compiled/@workflow/errors/index.js"; -import { cancelRun, getWorld, resumeHook } from "#internal/workflow/runtime.js"; +import { cancelRun, getRun, getWorld, resumeHook } from "#internal/workflow/runtime.js"; import { logError } from "#internal/logging.js"; -import { cancelWorkflowToolRun } from "./cancel.js"; +import { cancelWorkflowToolRun, settleWorkflowToolRunCancellation } from "./cancel.js"; vi.mock("#compiled/@workflow/core/runtime.js", () => ({ cancelRun: vi.fn(), + getRun: vi.fn(), getWorld: vi.fn(), resumeHook: vi.fn(), })); @@ -23,9 +24,15 @@ const reason = "The calling turn was cancelled."; describe("cancelWorkflowToolRun", () => { beforeEach(() => { vi.resetAllMocks(); + vi.useFakeTimers(); + vi.mocked(getRun).mockReturnValue({ status: Promise.resolve("completed") } as ReturnType< + typeof getRun + >); vi.mocked(getWorld).mockResolvedValue(world); }); + afterEach(() => vi.useRealTimers()); + it("lets a registered workflow cancel cooperatively", async () => { await cancelWorkflowToolRun(address, reason); @@ -33,6 +40,30 @@ describe("cancelWorkflowToolRun", () => { expect(cancelRun).not.toHaveBeenCalled(); }); + it("allows slow cooperative cleanup before escalating a stuck waiting run", async () => { + vi.mocked(getRun).mockReturnValue({ status: Promise.resolve("running") } as ReturnType< + typeof getRun + >); + const cancelled = cancelWorkflowToolRun(address, reason); + await vi.advanceTimersByTimeAsync(2_000); + expect(cancelRun).not.toHaveBeenCalled(); + await vi.runAllTimersAsync(); + await cancelled; + expect(cancelRun).toHaveBeenCalledExactlyOnceWith(world, address.runId, { + cancelReason: reason, + }); + }); + + it("does not mistake a status lookup failure for successful cancellation", async () => { + vi.mocked(getRun).mockImplementation(() => { + throw new Error("status unavailable"); + }); + await expect(settleWorkflowToolRunCancellation(address.runId, reason)).rejects.toThrow( + "status unavailable", + ); + expect(cancelRun).not.toHaveBeenCalled(); + }); + it("cancels by run ID before the control hook is registered", async () => { vi.mocked(resumeHook).mockRejectedValue(new HookNotFoundError(address.hookToken)); diff --git a/packages/eve/src/execution/tools/workflow/cancel.ts b/packages/eve/src/execution/tools/workflow/cancel.ts index 82b50424f..759815182 100644 --- a/packages/eve/src/execution/tools/workflow/cancel.ts +++ b/packages/eve/src/execution/tools/workflow/cancel.ts @@ -1,24 +1,25 @@ +import { WORKFLOW_CANCELLATION_SETTLE_MS } from "#execution/tools/workflow/cancellation-policy.js"; import type { WorkflowToolRunControlMessage } from "#execution/tools/workflow/messages.js"; import type { WorkflowToolRunAddress } from "#execution/tools/workflow/types.js"; import { isTaskWorkflowTargetGone } from "#execution/tasks/workflow-target.js"; -import { cancelRun, getWorld, resumeHook } from "#internal/workflow/runtime.js"; +import { cancelRun, getRun, getWorld, resumeHook } from "#internal/workflow/runtime.js"; import { createLogger, logError } from "#internal/logging.js"; const log = createLogger("execution.workflow-tool-run"); /** - * Asks the run to cancel itself; cancels it outright only if the message cannot - * be delivered. Failures are logged, not thrown: the caller has already - * committed the cancellation the run was serving. + * Asks the run to cancel itself, then bounds cooperative cleanup. Failures are + * logged: the caller has already committed cancellation of the calling turn. */ export async function cancelWorkflowToolRun( run: WorkflowToolRunAddress, reason: string, ): Promise { const cancel: WorkflowToolRunControlMessage = { kind: "cancel", reason }; + let signalled = false; try { await resumeHook(run.hookToken, cancel); - return; + signalled = true; } catch (error) { // A fresh run may not have registered its control hook yet. if (!isTaskWorkflowTargetGone(error)) { @@ -34,7 +35,7 @@ export async function cancelWorkflowToolRun( } try { - await cancelRun(await getWorld(), run.runId, { cancelReason: reason }); + await settleWorkflowToolRunCancellation(run.runId, reason, signalled); } catch (error) { if (isTaskWorkflowTargetGone(error)) return; logError(log, "failed to cancel workflow tool run; it may run to completion", error, { @@ -43,6 +44,33 @@ export async function cancelWorkflowToolRun( } } +/** Allows cooperative cleanup, then forcibly stops a still-live run. */ +export async function settleWorkflowToolRunCancellation( + runId: string, + reason: string, + cooperative = true, +): Promise { + if (cooperative) { + const deadline = Date.now() + WORKFLOW_CANCELLATION_SETTLE_MS; + while (Date.now() < deadline) { + try { + const status = await getRun(runId).status; + if (status !== "pending" && status !== "running") return; + } catch (error) { + if (isTaskWorkflowTargetGone(error)) return; + throw error; + } + await new Promise((resolve) => setTimeout(resolve, 250)); + } + } + try { + await cancelRun(await getWorld(), runId, { cancelReason: reason }); + } catch (error) { + if (isTaskWorkflowTargetGone(error)) return; + throw error; + } +} + export async function cancelWorkflowToolRunStep(input: { readonly reason: string; readonly run: WorkflowToolRunAddress; diff --git a/packages/eve/src/execution/tools/workflow/cancellation-policy.ts b/packages/eve/src/execution/tools/workflow/cancellation-policy.ts new file mode 100644 index 000000000..22ddadda9 --- /dev/null +++ b/packages/eve/src/execution/tools/workflow/cancellation-policy.ts @@ -0,0 +1,3 @@ +export const WORKFLOW_CANCELLATION_CLEANUP_MS = 30_000; +// Give the owner time to persist its outcome and wake its parent after body cleanup. +export const WORKFLOW_CANCELLATION_SETTLE_MS = WORKFLOW_CANCELLATION_CLEANUP_MS + 5_000; diff --git a/packages/eve/src/execution/tools/workflow/messages.ts b/packages/eve/src/execution/tools/workflow/messages.ts index 95830d247..07c8e22d7 100644 --- a/packages/eve/src/execution/tools/workflow/messages.ts +++ b/packages/eve/src/execution/tools/workflow/messages.ts @@ -54,7 +54,7 @@ export interface WorkflowToolRunRef { readonly callId: string; readonly execution: "background" | "blocking"; readonly input: JsonObject; - readonly resultKind?: "subagent" | "tool"; + readonly runId: string; readonly sequence: number; readonly stepIndex: number; diff --git a/packages/eve/src/execution/tools/workflow/owner-channels.ts b/packages/eve/src/execution/tools/workflow/owner-channels.ts index 4318ea5f5..3c8bf694b 100644 --- a/packages/eve/src/execution/tools/workflow/owner-channels.ts +++ b/packages/eve/src/execution/tools/workflow/owner-channels.ts @@ -20,11 +20,14 @@ export function createChannelReader( return { channel, iterator: iterable[Symbol.asyncIterator](), landed: [] }; } -export type ChannelRead[]> = { - [I in keyof R]: R[I] extends ChannelReader +type ChannelReadResult = + R extends ChannelReader ? { readonly channel: C; readonly next: IteratorResult } : never; -}[number]; + +export type ChannelRead[]> = ChannelReadResult< + R[number] +>; /** * Channel reads wake the loop with `undefined` after buffering their result; diff --git a/packages/eve/src/execution/tools/workflow/owner-inbox.test.ts b/packages/eve/src/execution/tools/workflow/owner-inbox.test.ts index 162f74f2e..f16baa431 100644 --- a/packages/eve/src/execution/tools/workflow/owner-inbox.test.ts +++ b/packages/eve/src/execution/tools/workflow/owner-inbox.test.ts @@ -1,9 +1,8 @@ import { describe, expect, it } from "vitest"; import { - workflowToolRunOutcomeToTaskCommand, - workflowToolRunReportToTaskPayload, - workflowToolRunRequestToTaskInputRequest, + workflowToolRunFailureOutput, + workflowToolRunRequestToInputRequestPayload, } from "#execution/tools/workflow/owner-inbox.js"; const from = { @@ -32,18 +31,19 @@ describe("workflow-tool task input", () => { }; expect( - workflowToolRunRequestToTaskInputRequest({ + workflowToolRunRequestToInputRequestPayload({ from, replyTo: "subagent:parent:call-1", request, }), - ).toEqual({ - kind: "task-input-request", - replyTo: "subagent:parent:call-1", - request, - sequence: 0, - stepIndex: 0, - turnId: "turn-1", + ).toMatchObject({ + childContinuationToken: "subagent:parent:call-1", + event: { + requests: [request], + sequence: 0, + stepIndex: 0, + turnId: "turn-1", + }, }); }); @@ -61,25 +61,26 @@ describe("workflow-tool task input", () => { }; expect( - workflowToolRunRequestToTaskInputRequest({ + workflowToolRunRequestToInputRequestPayload({ from, replyTo: "subagent:parent:call-1", request, requestCoordinates: { sequence: 4, stepIndex: 2, turnId: "turn-child" }, }), - ).toEqual({ - kind: "task-input-request", - replyTo: "subagent:parent:call-1", - request, - sequence: 4, - stepIndex: 2, - turnId: "turn-child", + ).toMatchObject({ + childContinuationToken: "subagent:parent:call-1", + event: { + requests: [request], + sequence: 4, + stepIndex: 2, + turnId: "turn-child", + }, }); }); it("does not normalize workflow agent requests as human input", () => { expect(() => - workflowToolRunRequestToTaskInputRequest({ + workflowToolRunRequestToInputRequestPayload({ from, replyTo: "subagent:parent:call-1", request: { @@ -92,41 +93,11 @@ describe("workflow-tool task input", () => { }); }); -describe("workflow-tool task reports", () => { - it("maps postMessage to a distinct parent delivery", () => { - expect( - workflowToolRunReportToTaskPayload( - { from, update: { kind: "eve:task-message", message: "Review this output." } }, - "task-1", - 2, - ), - ).toEqual({ - callId: "call-1", - kind: "task-message", - message: "Review this output.", - messageEpoch: "task-1", - messageIndex: 2, - }); - }); - - it("keeps untagged yields as progress", () => { - expect( - workflowToolRunReportToTaskPayload({ from, update: { progress: 0.5 } }, "task-1", 1), - ).toEqual({ - callId: "call-1", - kind: "task-update", - message: '{"progress":0.5}', - updateEpoch: "task-1", - updateIndex: 1, - }); - }); -}); - describe("workflow-tool task outcomes", () => { - it("keeps a subagent failure object as task failure data", () => { + it("keeps a structured workflow failure as task failure data", () => { expect( - workflowToolRunOutcomeToTaskCommand({ - from: { ...from, resultKind: "subagent" }, + workflowToolRunFailureOutput({ + from, result: { error: { code: "SUBAGENT_EXECUTION_FAILED", @@ -136,17 +107,14 @@ describe("workflow-tool task outcomes", () => { }, }), ).toEqual({ - data: { - code: "SUBAGENT_EXECUTION_FAILED", - message: "child crashed", - }, - kind: "fail", + code: "SUBAGENT_EXECUTION_FAILED", + message: "child crashed", }); }); it("keeps ordinary workflow-tool task failures as message strings", () => { expect( - workflowToolRunOutcomeToTaskCommand({ + workflowToolRunFailureOutput({ from, result: { error: { @@ -155,9 +123,6 @@ describe("workflow-tool task outcomes", () => { status: "failed", }, }), - ).toEqual({ - data: "export failed", - kind: "fail", - }); + ).toEqual("export failed"); }); }); diff --git a/packages/eve/src/execution/tools/workflow/owner-inbox.ts b/packages/eve/src/execution/tools/workflow/owner-inbox.ts index 57644b6d6..8430e8dd6 100644 --- a/packages/eve/src/execution/tools/workflow/owner-inbox.ts +++ b/packages/eve/src/execution/tools/workflow/owner-inbox.ts @@ -1,20 +1,14 @@ import type { SubagentInputRequestHookPayload } from "#channel/types.js"; import type { WorkflowToolRunOutcomeMessage, - WorkflowToolRunReport, WorkflowToolRunRef, WorkflowToolInputRequestBatch, WorkflowToolRequest, WorkflowToolRunRequestMessage, } from "#execution/tools/workflow/messages.js"; import type { RuntimeToolResultActionResult } from "#shared/action-types.js"; -import type { RuntimeSubagentResult } from "#shared/action-types.js"; import type { InputRequest } from "#shared/input.js"; import type { ToolInputRequest } from "#tools/definition.js"; -import type { WorkflowToolRunTaskInputRequest } from "#execution/tasks/child/workflow.js"; -import type { TaskCommand, TaskInboundMessage, TaskInboundUpdate } from "#tasks/types.js"; -import { isTaskMessage } from "#tools/task.js"; -import { SUBAGENT_EXECUTION_FAILED } from "#subagents/agent-handle-errors.js"; import { parseJsonValue, type JsonValue } from "#shared/json.js"; export function workflowToolRunOutcomeToToolResult( @@ -35,100 +29,23 @@ export function workflowToolRunOutcomeToToolResult( kind: "tool-result", output: result.status === "failed" - ? errorMessage(result.error) + ? workflowToolRunFailureOutput(message) : (result.reason ?? "The workflow tool run was cancelled."), toolName: from.toolName, }; } -/** Reads a blocking agent result returned through an ordinary run outcome. */ -export function workflowToolRunOutcomeToSubagentResult( - message: WorkflowToolRunOutcomeMessage, -): RuntimeSubagentResult { - if (message.result.status === "completed" && isRuntimeSubagentResult(message.result.output)) { - return message.result.output; - } - const output = - message.result.status === "failed" - ? errorMessage(message.result.error) - : message.result.status === "cancelled" - ? (message.result.reason ?? "The agent invocation was cancelled.") - : "The agent invocation returned an invalid result."; - return { - callId: message.from.callId, - isError: true, - kind: "subagent-result", - origin: "dispatch", - output, - subagentName: message.from.toolName, - }; -} - -export function workflowToolRunOutcomeToTaskCommand( - message: WorkflowToolRunOutcomeMessage, -): TaskCommand { - if (message.result.status === "completed") { - return { data: message.result.output, kind: "complete" }; - } - if (message.result.status === "failed") { - return { - data: - message.from.resultKind === "subagent" - ? subagentFailureOutput(message.result.error) - : errorMessage(message.result.error), - kind: "fail", - }; - } - return { kind: "cancel" }; -} - -function subagentFailureOutput(error: unknown): JsonValue { - const parsed = parseJsonValueOrUndefined(error); - if ( - parsed !== undefined && +export function workflowToolRunFailureOutput(message: WorkflowToolRunOutcomeMessage): JsonValue { + if (message.result.status !== "failed") + throw new TypeError("Expected a failed workflow outcome."); + const parsed = parseJsonValueOrUndefined(message.result.error); + return parsed !== undefined && typeof parsed === "object" && parsed !== null && !Array.isArray(parsed) && typeof Reflect.get(parsed, "code") === "string" - ) { - return parsed; - } - return { - code: SUBAGENT_EXECUTION_FAILED, - message: errorMessage(error), - }; -} - -export function workflowToolRunReportToTaskPayload( - report: WorkflowToolRunReport, - taskId: string, - updateIndex: number, -): TaskInboundMessage | TaskInboundUpdate { - if (isTaskMessage(report.update)) { - return { - callId: report.from.callId, - kind: "task-message", - message: report.update.message, - messageEpoch: taskId, - messageIndex: updateIndex, - }; - } - return { - callId: report.from.callId, - kind: "task-update", - message: typeof report.update === "string" ? report.update : JSON.stringify(report.update), - updateEpoch: taskId, - updateIndex, - }; -} - -function isRuntimeSubagentResult(value: unknown): value is RuntimeSubagentResult { - if (typeof value !== "object" || value === null) return false; - const origin = Reflect.get(value, "origin"); - return ( - Reflect.get(value, "kind") === "subagent-result" && - (origin === "child" || origin === "dispatch") - ); + ? parsed + : errorMessage(message.result.error); } function errorMessage(error: unknown): string { @@ -150,16 +67,13 @@ function parseJsonValueOrUndefined(value: unknown): JsonValue | undefined { export function workflowToolRunRequestToInputRequestPayload( message: WorkflowToolRunRequestMessage, ): SubagentInputRequestHookPayload { - const { from, replyTo, request, requestCoordinates } = message; + const { from, replyTo, requestCoordinates } = message; return { callId: from.callId, childContinuationToken: replyTo, childSessionId: from.runId, event: { - requests: - request.kind === "input-batch" - ? request.requests - : [normalizeInputRequest(request, from, replyTo)], + requests: workflowToolRunInputRequests(message), sequence: requestCoordinates?.sequence ?? from.sequence, stepIndex: requestCoordinates?.stepIndex ?? from.stepIndex, turnId: requestCoordinates?.turnId ?? from.turnId, @@ -169,20 +83,12 @@ export function workflowToolRunRequestToInputRequestPayload( }; } -export function workflowToolRunRequestToTaskInputRequest( +export function workflowToolRunInputRequests( message: WorkflowToolRunRequestMessage, -): WorkflowToolRunTaskInputRequest { - const { from, replyTo, request, requestCoordinates } = message; - const base = { - kind: "task-input-request" as const, - replyTo, - sequence: requestCoordinates?.sequence ?? from.sequence, - stepIndex: requestCoordinates?.stepIndex ?? from.stepIndex, - turnId: requestCoordinates?.turnId ?? from.turnId, - }; - return request.kind === "input-batch" - ? { ...base, requests: request.requests } - : { ...base, request: normalizeInputRequest(request, from, replyTo) }; +): readonly InputRequest[] { + return message.request.kind === "input-batch" + ? message.request.requests + : [normalizeInputRequest(message.request, message.from, message.replyTo)]; } function normalizeInputRequest( diff --git a/packages/eve/src/execution/tools/workflow/owner.ts b/packages/eve/src/execution/tools/workflow/owner.ts index a3a44787b..78fe000b2 100644 --- a/packages/eve/src/execution/tools/workflow/owner.ts +++ b/packages/eve/src/execution/tools/workflow/owner.ts @@ -1,12 +1,15 @@ import { createHook } from "#compiled/@workflow/core/index.js"; import type { WorkflowToolRunMessage, + WorkflowToolRunRequestMessage, + WorkflowToolAuthorizationRequest, WorkflowToolRunOwner, } from "#execution/tools/workflow/messages.js"; import { createChannelReader, type ChannelReader, } from "#execution/tools/workflow/owner-channels.js"; +import { resumeHookStep } from "#execution/tools/workflow/resume-hook-step.js"; import { disposeHook } from "#execution/hook-ownership.js"; export interface WorkflowToolRunOwnerInbox { @@ -15,7 +18,7 @@ export interface WorkflowToolRunOwnerInbox { readonly reader: ChannelReader<"workflow", WorkflowToolRunMessage>; } -/** Background task workflows have their own lifecycle, outside a session inbox. */ +/** Receives body messages before routing them to the turn or session owner. */ export function openWorkflowToolRunOwnerInbox(): WorkflowToolRunOwnerInbox { const hook = createHook(); return { @@ -24,3 +27,14 @@ export function openWorkflowToolRunOwnerInbox(): WorkflowToolRunOwnerInbox { reader: createChannelReader("workflow", hook), }; } + +/** Acknowledge only step-owned events, after delivery or deliberate discard. */ +export async function deliverWorkflowAuthorization( + message: WorkflowToolRunRequestMessage & { readonly request: WorkflowToolAuthorizationRequest }, + deliver: () => Promise, +): Promise { + await deliver(); + // Agent events reuse their invocation reply channel; it is not an event acknowledgement. + if (message.request.event.childSessionId === message.from.runId) + await resumeHookStep(message.replyTo, null, { ifPresent: true }); +} diff --git a/packages/eve/src/execution/tools/workflow/run-control.ts b/packages/eve/src/execution/tools/workflow/run-control.ts deleted file mode 100644 index 6d846ed10..000000000 --- a/packages/eve/src/execution/tools/workflow/run-control.ts +++ /dev/null @@ -1,58 +0,0 @@ -import { createHook } from "#compiled/@workflow/core/index.js"; - -import { - isWorkflowToolRunControlMessage, - type WorkflowToolRunControlMessage, -} from "#execution/tools/workflow/messages.js"; - -/** - * The run's control inbox. An unawaited hook read is not scheduled under - * replay, so `cancelled` must be raced for a cancel to be observed at all. - */ -export interface WorkflowToolRunControlInbox { - readonly signal: AbortSignal; - /** Rejects with {@link WorkflowToolRunCancelledError} when a cancel message arrives; never resolves. */ - readonly cancelled: Promise; - reason(): string | undefined; -} - -export function openWorkflowToolRunControlInbox(hookToken: string): WorkflowToolRunControlInbox { - const hook = createHook({ token: hookToken }); - const iterator = hook[Symbol.asyncIterator](); - const controller = new AbortController(); - let cancelReason: string | undefined; - - const cancelled = consumeCancel(iterator, (reason) => { - cancelReason = reason; - controller.abort(new WorkflowToolRunCancelledError(reason)); - }); - // Racing drives the read; a lone reference must not surface as unhandled. - cancelled.catch(() => {}); - - return { - cancelled, - reason: () => cancelReason, - signal: controller.signal, - }; -} - -// End-of-stream parks forever so the race is simply never won. -async function consumeCancel( - iterator: AsyncIterator, - onCancel: (reason: string) => void, -): Promise { - while (true) { - const next = await iterator.next(); - if (next.done === true) return await new Promise(() => {}); - if (!isWorkflowToolRunControlMessage(next.value)) continue; - onCancel(next.value.reason); - throw new WorkflowToolRunCancelledError(next.value.reason); - } -} - -export class WorkflowToolRunCancelledError extends Error { - constructor(reason: string) { - super(reason); - this.name = "WorkflowToolRunCancelledError"; - } -} diff --git a/packages/eve/src/execution/tools/workflow/start.ts b/packages/eve/src/execution/tools/workflow/start.ts index 436c76c7a..12ecdf835 100644 --- a/packages/eve/src/execution/tools/workflow/start.ts +++ b/packages/eve/src/execution/tools/workflow/start.ts @@ -1,6 +1,6 @@ import type { SessionAuth, SessionParent } from "#context/session-context.js"; import { createRuntimeToolResultFromValue } from "#harness/action-result-helpers.js"; -import { recordWorkflowToolRun } from "#harness/workflow-tool-runs.js"; +import { registerWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import { createLogger, logError } from "#internal/logging.js"; import type { RuntimeSession } from "#subagents/handle-dispatch.js"; import type { @@ -54,7 +54,6 @@ export async function startWorkflowTask(input: { executeInput: task.executeInput, input: task.input, owner: input.owner, - resultKind: task.resultKind, session: { auth: { current: input.auth, initiator: input.initiatorAuth }, id: session.sessionId, @@ -66,11 +65,11 @@ export async function startWorkflowTask(input: { workflowId: task.workflowId, }); return { - session: recordWorkflowToolRun(session, { + session: registerWorkflowToolRun(session, { callId: task.callId, - hookToken: started.hookToken, - resultKind: task.resultKind ?? "tool", - runId: started.runId, + lifetime: "turn", + origin: { turnId: batchEvent.turnId, stepIndex: batchEvent.stepIndex }, + address: started, toolName: task.toolName, }), }; diff --git a/packages/eve/src/execution/tools/workflow/types.ts b/packages/eve/src/execution/tools/workflow/types.ts index cc67e39db..ce99ffa12 100644 --- a/packages/eve/src/execution/tools/workflow/types.ts +++ b/packages/eve/src/execution/tools/workflow/types.ts @@ -1,50 +1,23 @@ -import type { SessionContext } from "#context/session-context.js"; -import type { JsonObject, JsonValue } from "#shared/json.js"; -import type { TaskExecutorBinding } from "#tools/task.js"; +import type { ActivityObserverConfig } from "#channel/types.js"; +import type { TaskView } from "#tasks/types.js"; +import type { WorkflowBodyDefinition } from "#execution/tools/workflow/body.js"; import type { WorkflowToolRunOwner } from "#execution/tools/workflow/messages.js"; -import type { WorkflowAgentMetadata } from "#tools/workflow-definition.js"; -export type WorkflowToolRunSessionContext = SessionContext["session"]; - -export const WORKFLOW_TOOL_EXECUTOR_KIND = "workflow-tool"; - -/** Private task executor binding for the workflow tool run doing the task's work. */ -export function createWorkflowToolExecutorBinding( - input: WorkflowToolRunAddress, -): TaskExecutorBinding { - return { - data: { hookToken: input.hookToken, runId: input.runId }, - kind: WORKFLOW_TOOL_EXECUTOR_KIND, - }; -} - -export function readWorkflowToolExecutorAddress( - executor: TaskExecutorBinding | undefined, -): WorkflowToolRunAddress | undefined { - if (executor?.kind !== WORKFLOW_TOOL_EXECUTOR_KIND) return undefined; - const hookToken = executor.data.hookToken; - const runId = executor.data.runId; - return typeof hookToken === "string" && typeof runId === "string" - ? { hookToken, runId } - : undefined; -} - -export interface WorkflowToolRunInput { - readonly agents: Readonly>; - readonly callId: string; +export interface WorkflowToolRunInput extends WorkflowBodyDefinition { readonly execution?: "background" | "blocking"; - readonly executeInput?: JsonValue; readonly hookToken: string; - readonly input: JsonObject; readonly owner: WorkflowToolRunOwner; - readonly resultKind?: "subagent" | "tool"; - readonly session: WorkflowToolRunSessionContext; - readonly stepIndex: number; - readonly toolName: string; - readonly workflowId: string; } export interface WorkflowToolRunAddress { readonly hookToken: string; readonly runId: string; } + +export interface BackgroundWorkflowToolRunInput { + readonly activityObserver?: ActivityObserverConfig; + readonly initialView: TaskView; + readonly parentContinuationToken: string; + readonly taskInboxToken: string; + readonly workflow: WorkflowBodyDefinition; +} diff --git a/packages/eve/src/execution/tools/workflow/workflow-owner-background.test.ts b/packages/eve/src/execution/tools/workflow/workflow-owner-background.test.ts new file mode 100644 index 000000000..8fd3a02a0 --- /dev/null +++ b/packages/eve/src/execution/tools/workflow/workflow-owner-background.test.ts @@ -0,0 +1,561 @@ +import { assert, beforeEach, describe, expect, it, vi } from "vitest"; + +import type { WorkflowToolRunMessage } from "#execution/tools/workflow/messages.js"; +import { workflowToolRunWorkflow } from "#execution/tools/workflow/workflow.js"; +import type { TaskCommand, TaskView } from "#tasks/types.js"; +import { + createAuthorizationRequiredEvent, + createAuthorizationCompletedEvent, +} from "#protocol/message.js"; + +const mocks = vi.hoisted(() => ({ + emitTaskActivityStep: vi.fn(), + claimHookOwnership: vi.fn(), + createChannelReader: vi.fn((channel: string) => ({ + channel, + landed: [], + iterator: [][Symbol.iterator](), + })), + createHook: vi.fn(() => ({ token: "task-token" })), + deliverTaskInputResponsesStep: vi.fn(), + raceChannelReads: vi.fn(), + resumeHookStep: vi.fn(), + notifyTaskParent: vi.fn(), + executeWorkflowBody: vi.fn(), + sleep: vi.fn(), +})); + +vi.mock("#compiled/@workflow/core/index.js", async (importOriginal) => ({ + ...(await importOriginal()), + createHook: mocks.createHook, + sleep: mocks.sleep, +})); +vi.mock("#execution/hook-ownership.js", () => ({ + claimHookOwnership: mocks.claimHookOwnership, + isHookConflictError: () => false, +})); +vi.mock("#execution/tasks/child/notify.js", () => ({ + emitTaskActivityStep: mocks.emitTaskActivityStep, + deliverTaskInputResponsesStep: mocks.deliverTaskInputResponsesStep, + notifyTaskParent: mocks.notifyTaskParent, +})); +vi.mock("#execution/tools/workflow/owner-channels.js", () => ({ + createChannelReader: mocks.createChannelReader, + raceChannelReads: mocks.raceChannelReads, +})); +vi.mock("#execution/tools/workflow/resume-hook-step.js", () => ({ + resumeHookStep: mocks.resumeHookStep, +})); +vi.mock("#execution/tools/workflow/body.js", () => ({ + executeWorkflowBody: mocks.executeWorkflowBody, + createWorkflowBodyRef: () => bufferedAgentRequest.from, +})); +vi.mock("#execution/tools/workflow/owner.js", async (importOriginal) => ({ + ...(await importOriginal()), + openWorkflowToolRunOwnerInbox: () => ({ + owner: { inbox: "owner" }, + reader: { channel: "workflow" }, + }), +})); + +const initialView = { + metadata: { kind: "tool", name: "approval-worker" }, + status: "working", + taskId: "task-1", +} satisfies TaskView; + +const bufferedAgentRequest = { + kind: "request", + from: { + callId: "tool-call-1", + execution: "background", + input: { message: "authorize" }, + runId: "run-1", + sequence: 0, + stepIndex: 0, + toolName: "approval-worker", + turnId: "turn-parent", + }, + replyTo: "agent-reply", + request: { + input: { message: "authorize", target: "approver" }, + invocationId: "tool-call-1:approver", + kind: "agent-invoke", + }, +} satisfies WorkflowToolRunMessage; + +const workflowAgentRequest = { + ...bufferedAgentRequest, + request: { + input: { message: "authorize", target: "approver" }, + invocationId: "tool-call-1:approver:2", + kind: "agent-invoke", + }, +} satisfies WorkflowToolRunMessage; + +function authorizationRequest(attemptId: string, completed = false) { + const data = { attemptId, name: "github", sequence: 0, stepIndex: 0, turnId: "turn-parent" }; + return { + ...bufferedAgentRequest, + replyTo: `ack-${attemptId}`, + request: { + kind: "authorization-request", + event: { + kind: "subagent-authorization-event", + callId: "tool-call-1", + childSessionId: "run-1", + subagentName: "approval-worker", + event: completed + ? createAuthorizationCompletedEvent({ ...data, outcome: "authorized" }) + : createAuthorizationRequiredEvent({ ...data, description: "Sign in" }), + }, + }, + } satisfies WorkflowToolRunMessage; +} + +function queueOwnerRequest(value: WorkflowToolRunMessage) { + mocks.raceChannelReads.mockResolvedValueOnce( + value.kind === "outcome" + ? { + channel: "body", + next: { done: false, value: { outcome: value.result, reportCount: 0 } }, + } + : { + channel: "workflow", + next: { done: false, value }, + }, + ); +} + +function queueCommand(command: TaskCommand) { + mocks.raceChannelReads.mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { kind: "task-command", command } }, + }); +} + +const workflowInput = { + initialView, + parentContinuationToken: "parent-token", + taskInboxToken: "task-token", + workflow: { + callId: "call-1", + input: {}, + session: { + auth: { current: null, initiator: null }, + id: "session-1", + turn: { id: "turn-1", sequence: 0 }, + }, + stepIndex: 0, + toolName: "worker", + workflowId: "workflow//eve//worker", + }, +}; + +describe("workflowToolRunWorkflow", () => { + beforeEach(() => { + vi.resetAllMocks(); + mocks.createHook.mockReturnValue({ token: "task-token" }); + mocks.executeWorkflowBody.mockReturnValue(new Promise(() => {})); + mocks.sleep.mockReturnValue(new Promise(() => {})); + }); + + it("forwards auth events before acknowledging them", async () => { + queueCommand({ kind: "ready" }); + queueOwnerRequest(authorizationRequest("a")); + queueOwnerRequest(authorizationRequest("b")); + queueOwnerRequest(authorizationRequest("a", true)); + queueOwnerRequest(authorizationRequest("b", true)); + mocks.raceChannelReads.mockResolvedValueOnce({ channel: "commands", next: { done: true } }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.notifyTaskParent).toHaveBeenCalledTimes(4); + expect(mocks.resumeHookStep).toHaveBeenCalledTimes(4); + for (let i = 0; i < 4; i++) { + const notified = mocks.notifyTaskParent.mock.invocationCallOrder[i]; + const acknowledged = mocks.resumeHookStep.mock.invocationCallOrder[i]; + assert(notified !== undefined && acknowledged !== undefined); + expect(notified).toBeLessThan(acknowledged); + } + }); + + it("keeps ordinary input answerable when authorization completes with the same request id", async () => { + const requestId = "request-1"; + const answer = { + kind: "input-response" as const, + childContinuationToken: "answer-hook", + taskId: initialView.taskId, + inputResponses: [{ requestId, optionId: "approve" }], + }; + queueCommand({ kind: "ready" }); + queueOwnerRequest({ + ...bufferedAgentRequest, + replyTo: "answer-hook", + request: { + kind: "tool-approval", + requestId, + prompt: "Approve deployment?", + action: { kind: "tool-call", callId: "deploy", toolName: "deploy", input: {} }, + }, + }); + queueOwnerRequest(authorizationRequest(requestId)); + queueOwnerRequest(authorizationRequest(requestId, true)); + mocks.deliverTaskInputResponsesStep.mockResolvedValue("delivered"); + mocks.raceChannelReads.mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: answer }, + }); + mocks.raceChannelReads.mockResolvedValueOnce({ channel: "commands", next: { done: true } }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.deliverTaskInputResponsesStep).toHaveBeenCalledExactlyOnceWith({ + answer, + answerHook: { runId: "run-1" }, + requestIds: [requestId], + }); + }); + + it("acknowledges discarded authorization prompts after cancellation", async () => { + queueCommand({ kind: "ready" }); + queueCommand({ kind: "cancel" }); + queueOwnerRequest(authorizationRequest("late")); + mocks.raceChannelReads.mockResolvedValueOnce({ channel: "commands", next: { done: true } }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.notifyTaskParent).toHaveBeenCalledExactlyOnceWith({ + token: "parent-token", + view: { ...initialView, status: "cancelled" }, + }); + expect(mocks.resumeHookStep).toHaveBeenCalledExactlyOnceWith("ack-late", null, { + ifPresent: true, + }); + }); + + it("forwards child-agent auth without treating it as the workflow's own request", async () => { + const message = authorizationRequest("child"); + message.request.event.childSessionId = "child-session"; + queueCommand({ kind: "ready" }); + queueOwnerRequest(message); + mocks.raceChannelReads.mockResolvedValueOnce({ channel: "commands", next: { done: true } }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.notifyTaskParent).toHaveBeenCalledExactlyOnceWith({ + request: message, + taskId: initialView.taskId, + token: workflowInput.parentContinuationToken, + }); + expect(mocks.resumeHookStep).not.toHaveBeenCalled(); + expect( + mocks.emitTaskActivityStep.mock.calls.some( + ([input]) => input.view.status === "input_required", + ), + ).toBe(false); + }); + + it("does not acknowledge auth after failed forwarding", async () => { + queueCommand({ kind: "ready" }); + queueOwnerRequest(authorizationRequest("a")); + mocks.notifyTaskParent.mockRejectedValue(new Error("failed forwarding")); + await expect(workflowToolRunWorkflow(workflowInput)).rejects.toThrow("failed forwarding"); + expect(mocks.resumeHookStep).not.toHaveBeenCalled(); + }); + + it("forwards admitted agent requests through the task's owner channel", async () => { + mocks.raceChannelReads + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ + channel: "workflow", + next: { done: false, value: workflowAgentRequest }, + }) + .mockResolvedValueOnce({ channel: "commands", next: { done: true, value: undefined } }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.notifyTaskParent).toHaveBeenCalledWith({ + request: workflowAgentRequest, + taskId: "task-1", + token: "parent-token", + }); + }); + + it("waits for agent settlement delivery before publishing task completion", async () => { + const delivery = Promise.withResolvers(); + const delivering = Promise.withResolvers(); + mocks.notifyTaskParent.mockImplementationOnce(() => { + delivering.resolve(); + return delivery.promise; + }); + const settlement = { + ...bufferedAgentRequest, + request: { + kind: "agent-settled", + result: { + callId: "nested", + kind: "subagent-result", + origin: "child", + outcome: { + kind: "parked", + result: { kind: "succeeded", output: "done" }, + usageDelta: { + cacheReadTokens: 0, + cacheWriteTokens: 0, + inputTokens: 2, + outputTokens: 3, + }, + }, + output: "done", + subagentName: "research", + }, + }, + } satisfies WorkflowToolRunMessage; + queueCommand({ kind: "ready" }); + queueOwnerRequest(settlement); + queueOwnerRequest({ + kind: "outcome", + from: bufferedAgentRequest.from, + result: { status: "completed", output: "done" }, + }); + const execution = workflowToolRunWorkflow(workflowInput); + await delivering.promise; + expect(mocks.notifyTaskParent).toHaveBeenCalledTimes(1); + delivery.resolve(); + await execution; + expect(mocks.notifyTaskParent).toHaveBeenNthCalledWith(1, { + request: settlement, + taskId: "task-1", + token: "parent-token", + }); + expect(mocks.notifyTaskParent).toHaveBeenNthCalledWith(2, { + token: "parent-token", + view: expect.objectContaining({ status: "completed" }), + }); + }); + + it("does not execute a workflow body before task admission", async () => { + mocks.raceChannelReads.mockResolvedValueOnce({ + channel: "commands", + next: { done: true, value: undefined }, + }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.executeWorkflowBody).not.toHaveBeenCalled(); + }); + + it.each(["ready", "reject-dispatch"] as const)( + "does not start pre-admission cancelled work after %s", + async (kind) => { + queueCommand({ kind: "cancel" }); + queueCommand(kind === "ready" ? { kind } : { kind, data: "step failed" }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.executeWorkflowBody).not.toHaveBeenCalled(); + expect(mocks.notifyTaskParent).toHaveBeenCalledTimes(kind === "ready" ? 1 : 0); + if (kind === "ready") + expect(mocks.notifyTaskParent).toHaveBeenLastCalledWith({ + token: "parent-token", + view: { ...initialView, status: "cancelled" }, + }); + }, + ); + + it("starts the workflow body only after ready", async () => { + mocks.raceChannelReads + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ channel: "commands", next: { done: true, value: undefined } }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.executeWorkflowBody).toHaveBeenCalledOnce(); + expect(mocks.executeWorkflowBody).toHaveBeenCalledWith( + expect.objectContaining({ execution: "background" }), + expect.any(AbortSignal), + ); + }); + + it.each(["tool", "subagent"])( + "consumes %s progress and only forwards subagent updates", + async (kind) => { + const report = { + from: { ...bufferedAgentRequest.from, callId: "call-1" }, + update: "progress", + }; + mocks.raceChannelReads + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ + channel: "workflow", + next: { done: false, value: { ...report, kind: "report" } }, + }) + .mockResolvedValueOnce({ + channel: "body", + next: { + done: false, + value: { + reportCount: 0, + outcome: { status: "completed", output: "done" }, + }, + }, + }); + + await workflowToolRunWorkflow({ + ...workflowInput, + initialView: { ...initialView, metadata: { ...initialView.metadata, kind } }, + }); + + if (kind === "subagent") { + expect(mocks.notifyTaskParent).toHaveBeenNthCalledWith(1, { + token: "parent-token", + update: { report: expect.objectContaining(report), index: 0 }, + view: expect.objectContaining({ status: "working" }), + }); + expect(mocks.notifyTaskParent).toHaveBeenNthCalledWith(2, { + token: "parent-token", + view: expect.objectContaining({ status: "completed" }), + }); + return; + } + expect(mocks.notifyTaskParent).toHaveBeenCalledTimes(1); + }, + ); + + it("publishes cancellation after the workflow body observes its abort", async () => { + mocks.raceChannelReads + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "cancel" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ + channel: "body", + next: { + done: false, + value: { + reportCount: 0, + outcome: { reason: "cancelled", status: "cancelled" }, + }, + }, + }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.notifyTaskParent).toHaveBeenCalledWith({ + token: "parent-token", + view: expect.objectContaining({ status: "cancelled" }), + }); + }); + + it("ignores duplicate admission commands while cancelled work finishes cleanup", async () => { + queueCommand({ kind: "ready" }); + queueCommand({ kind: "cancel" }); + queueCommand({ kind: "ready" }); + queueCommand({ kind: "reject-dispatch", data: "late rejection" }); + queueOwnerRequest(authorizationRequest("cleanup", true)); + queueOwnerRequest({ + from: bufferedAgentRequest.from, + kind: "outcome", + result: { status: "cancelled", reason: "stop" }, + }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.raceChannelReads).toHaveBeenCalledTimes(6); + expect(mocks.executeWorkflowBody).toHaveBeenCalledOnce(); + expect(mocks.resumeHookStep).toHaveBeenCalledExactlyOnceWith("ack-cleanup", null, { + ifPresent: true, + }); + expect(mocks.notifyTaskParent).toHaveBeenCalledTimes(2); + const acknowledged = mocks.resumeHookStep.mock.invocationCallOrder[0]; + const completed = mocks.notifyTaskParent.mock.invocationCallOrder[1]; + assert(acknowledged !== undefined && completed !== undefined); + expect(acknowledged).toBeLessThan(completed); + }); + + it("keeps explicit cancellation final when the invocation completes late", async () => { + mocks.raceChannelReads + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "cancel" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ + channel: "body", + next: { + done: false, + value: { + reportCount: 0, + outcome: { output: "late success", status: "completed" }, + }, + }, + }); + + await workflowToolRunWorkflow(workflowInput); + + expect(mocks.notifyTaskParent).toHaveBeenCalledWith({ + token: "parent-token", + view: expect.objectContaining({ status: "cancelled" }), + }); + expect(mocks.notifyTaskParent).toHaveBeenLastCalledWith({ + token: "parent-token", + view: expect.objectContaining({ status: "cancelled" }), + }); + }); + + it("drains background yields before delivering the return value without progress notifications", async () => { + mocks.raceChannelReads + .mockResolvedValueOnce({ + channel: "commands", + next: { done: false, value: { command: { kind: "ready" }, kind: "task-command" } }, + }) + .mockResolvedValueOnce({ + channel: "body", + next: { + done: false, + value: { + reportCount: 1, + outcome: { output: "done", status: "completed" }, + }, + }, + }) + .mockResolvedValueOnce({ + channel: "workflow", + next: { + done: false, + value: { + kind: "report", + from: bufferedAgentRequest.from, + update: "Review the export", + }, + }, + }); + await workflowToolRunWorkflow(workflowInput); + expect(mocks.raceChannelReads).toHaveBeenCalledTimes(3); + expect(mocks.notifyTaskParent).toHaveBeenCalledTimes(1); + expect(mocks.notifyTaskParent).toHaveBeenCalledExactlyOnceWith({ + token: "parent-token", + view: expect.objectContaining({ + status: "completed", + lastOutput: { type: "result", data: "done" }, + }), + }); + }); +}); diff --git a/packages/eve/src/execution/tools/workflow/workflow-owner-background.ts b/packages/eve/src/execution/tools/workflow/workflow-owner-background.ts new file mode 100644 index 000000000..5fde345a3 --- /dev/null +++ b/packages/eve/src/execution/tools/workflow/workflow-owner-background.ts @@ -0,0 +1,204 @@ +import { createHook } from "#compiled/@workflow/core/index.js"; + +import { claimHookOwnership, isHookConflictError } from "#execution/hook-ownership.js"; +import { + emitTaskActivityStep, + deliverTaskInputResponsesStep, + notifyTaskParent, +} from "#execution/tasks/child/notify.js"; +import type { BackgroundWorkflowToolRunInput } from "#execution/tools/workflow/types.js"; +import { deliverWorkflowAuthorization } from "#execution/tools/workflow/owner.js"; +import type { + WorkflowToolRunRequestMessage, + WorkflowToolRunReport, + WorkflowToolRunMessage, +} from "#execution/tools/workflow/messages.js"; +import { + createChannelReader, + type ChannelReader, +} from "#execution/tools/workflow/owner-channels.js"; +import { workflowToolRunInputRequests } from "#execution/tools/workflow/owner-inbox.js"; +import type { AnswerHookRoute } from "#harness/proxy-input-requests.js"; +import { applyTaskTransition } from "#tasks/transitions.js"; +import { + isTerminalTaskStatus, + readTaskInputRequestId, + type TaskCommand, + type TaskInboundAnswerInput, + type TaskRunInboundPayload, + type TaskView, +} from "#tasks/types.js"; + +export interface BackgroundWorkflowOwner { + readonly kind: "session"; + readonly commands: ChannelReader<"commands", TaskRunInboundPayload>; + readonly signal: AbortSignal; + handleCommand(payload: TaskRunInboundPayload): Promise<"start" | "stop" | undefined>; + handleMessage(message: WorkflowToolRunMessage): Promise; +} + +/** Routes child messages; the parent session owns persisted task state. */ +export async function createBackgroundWorkflowOwner( + input: BackgroundWorkflowToolRunInput, +): Promise { + const commands = createHook({ token: input.taskInboxToken }); + let view = input.initialView; + let updateIndex = 0; + const answerHooks = new Map(); + const bodyController = new AbortController(); + try { + await claimHookOwnership(commands); + } catch (error) { + if (isHookConflictError(error)) return; + throw error; + } + await emitTaskActivityStep({ activityObserver: input.activityObserver, view }); + return { + kind: "session", + commands: createChannelReader("commands", commands), + signal: bodyController.signal, + handleCommand, + handleMessage, + }; + + async function handleCommand( + payload: TaskRunInboundPayload, + ): Promise<"start" | "stop" | undefined> { + if (payload.kind === "task-command") { + if (payload.command.kind === "ready") { + if (!isTerminalTaskStatus(view.status)) return "start"; + await notifyTaskParent({ token: input.parentContinuationToken, view }); + return "stop"; + } + if (payload.command.kind === "reject-dispatch") { + applyTransition(payload.command); + return "stop"; + } + } + await applyPayload(payload); + } + + async function handleMessage(message: WorkflowToolRunMessage): Promise { + if (message.kind === "report") { + await handleReport(message); + return; + } + if (message.kind === "outcome") { + const transitioned = applyTransition(message); + if (transitioned || view.status === "cancelled") { + await notifyTaskParent({ token: input.parentContinuationToken, view }); + } + return; + } + const request = message; + const kind = request.request.kind; + if (kind === "agent-invoke" || kind === "agent-settled" || kind === "authorization-request") { + await handleOwnerRequest(request); + return; + } + if (request.requestCoordinates === undefined) { + answerHooks.set(request.replyTo, { runId: request.from.runId }); + } + const accepted = applyTransition({ + kind: "require-input", + inputRequests: workflowToolRunInputRequests(request), + }); + if (!accepted) return; + await notifyTaskParent({ + request, + taskId: view.taskId, + token: input.parentContinuationToken, + }); + } + + async function handleReport(report: WorkflowToolRunReport): Promise { + const index = updateIndex++; + if (view.metadata.kind !== "subagent" || isTerminalTaskStatus(view.status)) return; + await notifyTaskParent({ + token: input.parentContinuationToken, + update: { report, index }, + view, + }); + } + + async function applyPayload(payload: TaskRunInboundPayload): Promise { + let command: TaskCommand | undefined; + if (payload.kind === "input-response") { + command = + view.status === "input_required" + ? await resolveAnsweredCommand( + view, + payload, + answerHooks.get(payload.childContinuationToken), + ) + : undefined; + } else if (payload.kind === "task-command") { + command = payload.command; + } else { + return; + } + if (command === undefined) return; + + const accepted = applyTransition(command); + if (!accepted) return; + if (command.kind === "cancel") { + bodyController.abort(new Error(`Task ${view.taskId} was cancelled.`)); + } + } + + function applyTransition( + command: TaskCommand | Extract, + ): boolean { + const result = applyTaskTransition(view, command); + if (result.action !== "accepted") return false; + view = result.view; + return true; + } + + // Owner traffic must wait until the parent has acknowledged task dispatch. + async function handleOwnerRequest(message: WorkflowToolRunRequestMessage): Promise { + const { request } = message; + if (request.kind === "authorization-request") { + await deliverWorkflowAuthorization({ ...message, request }, async () => { + // A cancelled workflow may still need to close its own displayed prompt. + const closesPrompt = + request.event.childSessionId === message.from.runId && + request.event.event.type === "authorization.completed"; + if (isTerminalTaskStatus(view.status) && !closesPrompt) return; + await notifyTaskParent({ + request: message, + taskId: view.taskId, + token: input.parentContinuationToken, + }); + }); + return; + } + if (isTerminalTaskStatus(view.status)) return; + await notifyTaskParent({ + request: message, + taskId: view.taskId, + token: input.parentContinuationToken, + }); + } +} + +async function resolveAnsweredCommand( + view: Extract, + answer: TaskInboundAnswerInput, + answerHook: AnswerHookRoute | undefined, +): Promise { + if (answer.taskId !== view.taskId) return undefined; + const outstanding = new Set( + view.inputRequests.flatMap((request) => { + const requestId = readTaskInputRequestId(request); + return requestId === undefined ? [] : [requestId]; + }), + ); + const requestIds = answer.inputResponses + .map((response) => response.requestId) + .filter((id) => outstanding.has(id)); + if (requestIds.length === 0) return undefined; + return (await deliverTaskInputResponsesStep({ answer, answerHook, requestIds })) === "delivered" + ? { kind: "answered", requestIds } + : undefined; +} diff --git a/packages/eve/src/execution/tools/workflow/workflow-owner-blocking.ts b/packages/eve/src/execution/tools/workflow/workflow-owner-blocking.ts new file mode 100644 index 000000000..bbe7b3f7c --- /dev/null +++ b/packages/eve/src/execution/tools/workflow/workflow-owner-blocking.ts @@ -0,0 +1,47 @@ +import { createHook } from "#compiled/@workflow/core/index.js"; +import { + createChannelReader, + type ChannelReader, +} from "#execution/tools/workflow/owner-channels.js"; +import { + isWorkflowToolRunControlMessage, + type WorkflowToolRunMessage, + type WorkflowToolRunControlMessage, +} from "#execution/tools/workflow/messages.js"; +import { resumeHookStep } from "#execution/tools/workflow/resume-hook-step.js"; +import type { WorkflowToolRunInput } from "#execution/tools/workflow/types.js"; + +export interface BlockingWorkflowOwner { + readonly kind: "turn"; + readonly commands: ChannelReader<"control", WorkflowToolRunControlMessage>; + readonly signal: AbortSignal; + handleCommand(message: WorkflowToolRunControlMessage): void; + handleMessage(message: WorkflowToolRunMessage): Promise; +} + +/** Routes invocation messages to the waiting turn and accepts cancellation. */ +export function createBlockingWorkflow(input: WorkflowToolRunInput): BlockingWorkflowOwner { + const controller = new AbortController(); + const hook = createHook({ token: input.hookToken }); + return { + kind: "turn", + commands: createChannelReader("control", hook), + signal: controller.signal, + handleCommand(message: WorkflowToolRunControlMessage) { + if (isWorkflowToolRunControlMessage(message)) + controller.abort(new WorkflowToolRunCancelledError(message.reason)); + }, + handleMessage(message: WorkflowToolRunMessage) { + return resumeHookStep(input.owner.inbox, message, { + ifPresent: message.kind === "outcome" && message.result.status === "cancelled", + }); + }, + }; +} + +class WorkflowToolRunCancelledError extends Error { + constructor(reason: string) { + super(reason); + this.name = "WorkflowToolRunCancelledError"; + } +} diff --git a/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts b/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts index ec11e4b60..096ea9741 100644 --- a/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts +++ b/packages/eve/src/execution/tools/workflow/workflow-tool-run.integration.test.ts @@ -601,7 +601,7 @@ describe("workflow tools", () => { event.data.result.kind === "tool-result" && event.data.result.toolName === "confirm_deploy", ); - expect(progress).toBeGreaterThanOrEqual(0); + expect(progress, JSON.stringify(answered)).toBeGreaterThanOrEqual(0); expect(resultIndex).toBeGreaterThan(progress); const results = filterEventsByType(answered, "action.result"); expect(results.map((event) => JSON.stringify(event.data.result.output))).toContainEqual( @@ -745,7 +745,7 @@ describe("workflow tools", () => { }); }, 60_000); - it("runs a background workflow tool as its task's executor", async () => { + it("runs a session-owned background workflow invocation", async () => { vi.stubEnv("VERCEL_DEPLOYMENT_ID", "dpl_inline"); const runtime = await createWorkflowToolRuntime({ agentName: "workflow-tool-background", @@ -789,7 +789,7 @@ describe("workflow tools", () => { notifications.push(eventsText(filterEventsByType(woken, "message.received"))); } const text = notifications.join("\n"); - expect(text).toContain("Review plan:api"); + expect(text).not.toContain("Review plan:api"); expect(text).not.toContain("update: planned api"); expect(text).toContain("is completed"); expect(text).toContain("plan:api"); diff --git a/packages/eve/src/execution/tools/workflow/workflow.test.ts b/packages/eve/src/execution/tools/workflow/workflow.test.ts new file mode 100644 index 000000000..c9f698c4d --- /dev/null +++ b/packages/eve/src/execution/tools/workflow/workflow.test.ts @@ -0,0 +1,375 @@ +import { beforeEach, expect, it, vi } from "vitest"; + +import type { TaskRunInboundPayload } from "#tasks/types.js"; +import { createBackgroundWorkflowOwner } from "#execution/tools/workflow/workflow-owner-background.js"; +import type { WorkflowToolRunMessage } from "#execution/tools/workflow/messages.js"; +import { workflowToolRunWorkflow } from "#execution/tools/workflow/workflow.js"; + +const mocks = vi.hoisted(() => ({ + sleep: vi.fn(), + control: vi.fn(), + deliver: vi.fn(), + executeWorkflowBody: vi.fn(), + openWorkflowToolRunOwnerInbox: vi.fn(), +})); + +vi.mock("#execution/tools/workflow/workflow-owner-background.js", () => ({ + createBackgroundWorkflowOwner: vi.fn(), +})); + +vi.mock("#compiled/@workflow/core/index.js", () => ({ sleep: mocks.sleep })); + +vi.mock("#execution/tools/workflow/workflow-owner-blocking.js", () => ({ + createBlockingWorkflow: mocks.control, +})); +vi.mock("#execution/tools/workflow/resume-hook-step.js", () => ({ resumeHookStep: mocks.deliver })); + +vi.mock("#execution/tools/workflow/body.js", () => ({ + createWorkflowBodyRef: (input: { + callId: string; + execution: "background" | "blocking"; + input: object; + session: { turn: { id: string; sequence: number } }; + stepIndex: number; + toolName: string; + }) => ({ + callId: input.callId, + execution: input.execution, + input: input.input, + runId: "run-1", + sequence: input.session.turn.sequence, + stepIndex: input.stepIndex, + toolName: input.toolName, + turnId: input.session.turn.id, + }), + executeWorkflowBody: mocks.executeWorkflowBody, +})); +vi.mock("#execution/tools/workflow/owner.js", () => ({ + openWorkflowToolRunOwnerInbox: mocks.openWorkflowToolRunOwnerInbox, +})); + +import { createChannelReader } from "#execution/tools/workflow/owner-channels.js"; + +const input = { + hookToken: "control", + owner: { inbox: "parent" }, + callId: "call-1", + execution: "background" as const, + input: {}, + session: { + auth: { current: null, initiator: null }, + id: "session-1", + turn: { id: "turn-1", sequence: 0 }, + }, + stepIndex: 0, + toolName: "worker", + workflowId: "workflow//eve//worker", +}; + +beforeEach(() => { + vi.resetAllMocks(); + setControl(new AbortController()); + mocks.executeWorkflowBody.mockResolvedValue({ + outcome: { output: "done", status: "completed" }, + reportCount: 1, + }); +}); + +it("emits every persisted report before the terminal outcome", async () => { + const report = { + from: { + callId: "call-1", + execution: "background" as const, + input: {}, + runId: "run-1", + sequence: 0, + stepIndex: 0, + toolName: "worker", + turnId: "turn-1", + }, + kind: "report" as const, + update: "halfway", + }; + mocks.openWorkflowToolRunOwnerInbox.mockReturnValue({ + owner: { inbox: "invocation-owner" }, + reader: createChannelReader( + "workflow", + (async function* () { + // The body settles before the persisted report reaches its owner. + await new Promise((resolve) => setTimeout(resolve, 0)); + yield report; + })(), + ), + }); + + await workflowToolRunWorkflow(input); + expect(mocks.deliver).toHaveBeenNthCalledWith(1, "parent", report, { ifPresent: false }); + expect(mocks.deliver).toHaveBeenNthCalledWith( + 2, + "parent", + { + from: expect.objectContaining({ callId: "call-1", execution: "background" }), + kind: "outcome", + result: { output: "done", status: "completed" }, + }, + { ifPresent: false }, + ); + expect(mocks.executeWorkflowBody).toHaveBeenCalledWith( + expect.objectContaining({ owner: { inbox: "invocation-owner" } }), + expect.any(AbortSignal), + ); +}); + +for (const execution of ["blocking", "background"] as const) { + it.each(["completed", "failed", "cancelled", "throw", "blocked"] as const)( + `${execution} keeps cancellation final when cleanup is %s`, + async (status) => { + const controller = new AbortController(); + const release = Promise.withResolvers(); + const started = Promise.withResolvers(); + mocks.sleep.mockImplementation(async () => { + if (status === "blocked") return; + await new Promise(() => {}); + }); + mocks.openWorkflowToolRunOwnerInbox.mockReturnValue({ + owner: { inbox: "owner" }, + reader: createChannelReader("workflow", { + [Symbol.asyncIterator]: () => ({ + next: () => new Promise>(() => {}), + }), + }), + }); + mocks.executeWorkflowBody.mockImplementation(async () => { + started.resolve(); + await release.promise; + if (status === "throw") throw new Error("cleanup failed"); + return { + reportCount: 0, + outcome: + status === "failed" + ? { status, error: "failed" } + : status === "cancelled" + ? { status, reason: "body cancelled" } + : { status: "completed", output: "late success" }, + }; + }); + setControl(controller); + const completion = workflowToolRunWorkflow({ ...input, execution }); + await started.promise; + controller.abort(new Error("stop")); + if (status !== "blocked") release.resolve(); + await completion; + expect(mocks.deliver).toHaveBeenCalledExactlyOnceWith( + "parent", + expect.objectContaining({ + kind: "outcome", + result: { status: "cancelled", reason: "stop" }, + }), + { ifPresent: true }, + ); + }, + ); +} + +it("does not start a body cancelled before its first read", async () => { + const controller = new AbortController(); + controller.abort(new Error("never admitted")); + setControl(controller); + await workflowToolRunWorkflow(input); + expect(mocks.deliver).toHaveBeenCalledWith( + "parent", + expect.objectContaining({ + result: expect.objectContaining({ status: "cancelled" }), + }), + { ifPresent: true }, + ); + expect(mocks.executeWorkflowBody).not.toHaveBeenCalled(); + expect(mocks.openWorkflowToolRunOwnerInbox).not.toHaveBeenCalled(); + expect(mocks.sleep).not.toHaveBeenCalled(); +}); + +it("preserves the pending inbox read across cancellation and drains the report before settling", async () => { + const controller = new AbortController(); + const pending = Promise.withResolvers>(); + const next = vi.fn(() => pending.promise); + const report: WorkflowToolRunMessage = { + from: { + callId: input.callId, + execution: input.execution, + input: input.input, + runId: "run-1", + sequence: 0, + stepIndex: 0, + toolName: input.toolName, + turnId: "turn-1", + }, + kind: "report", + update: "cleanup progress", + }; + mocks.sleep.mockReturnValue(new Promise(() => {})); + mocks.openWorkflowToolRunOwnerInbox.mockReturnValue({ + owner: { inbox: "owner" }, + reader: createChannelReader("workflow", { [Symbol.asyncIterator]: () => ({ next }) }), + }); + setControl(controller); + const completion = workflowToolRunWorkflow(input); + await vi.waitFor(() => expect(next).toHaveBeenCalledOnce()); + controller.abort(new Error("stop")); + await vi.waitFor(() => expect(mocks.sleep).toHaveBeenCalledOnce()); + pending.resolve({ done: false, value: report }); + await completion; + expect(mocks.deliver).toHaveBeenNthCalledWith(1, "parent", report, { ifPresent: false }); + expect(next).toHaveBeenCalledOnce(); + expect(mocks.deliver).toHaveBeenNthCalledWith( + 2, + "parent", + expect.objectContaining({ + kind: "outcome", + result: { status: "cancelled", reason: "stop" }, + }), + { ifPresent: true }, + ); +}); + +function setControl(controller: AbortController) { + const { signal } = controller; + mocks.control.mockReturnValue({ + kind: "turn", + signal, + commands: createChannelReader( + "control", + (async function* () { + if (!signal.aborted) + await new Promise((resolve) => + signal.addEventListener("abort", () => resolve(), { once: true }), + ); + yield { kind: "cancel", reason: "stop" }; + await new Promise(() => {}); + })(), + ), + handleCommand: vi.fn(), + handleMessage: (message: WorkflowToolRunMessage) => + mocks.deliver("parent", message, { + ifPresent: message.kind === "outcome" && message.result.status === "cancelled", + }), + }); +} + +it("applies cancellation buffered during the last report delivery before publishing completion", async () => { + const controller = new AbortController(); + const cancel = Promise.withResolvers(); + const commands = createChannelReader( + "commands", + (async function* (): AsyncGenerator { + yield { kind: "task-command", command: { kind: "ready" } }; + await cancel.promise; + yield { kind: "task-command", command: { kind: "cancel" } }; + await new Promise(() => {}); + })(), + ); + const report: WorkflowToolRunMessage = { + from: { + callId: input.callId, + execution: input.execution, + input: {}, + runId: "run-1", + sequence: 0, + stepIndex: 0, + toolName: input.toolName, + turnId: "turn-1", + }, + kind: "report", + update: "finished", + }; + const deliver = vi.fn(async (message: WorkflowToolRunMessage) => { + if (message.kind === "report") { + cancel.resolve(); + await vi.waitFor(() => expect(commands.landed).toHaveLength(1)); + } + }); + vi.mocked(createBackgroundWorkflowOwner).mockResolvedValue({ + kind: "session", + commands, + signal: controller.signal, + handleMessage: deliver, + async handleCommand(payload) { + if (payload.kind !== "task-command") return; + if (payload.command.kind === "ready") return "start"; + if (payload.command.kind === "cancel") controller.abort(new Error("stop")); + }, + }); + mocks.sleep.mockReturnValue(new Promise(() => {})); + mocks.openWorkflowToolRunOwnerInbox.mockReturnValue({ + owner: { inbox: "owner" }, + reader: createChannelReader( + "workflow", + (async function* () { + await new Promise((resolve) => setTimeout(resolve, 0)); + yield report; + await new Promise(() => {}); + })(), + ), + }); + await workflowToolRunWorkflow({ + workflow: input, + initialView: { status: "working", taskId: "task", metadata: { kind: "tool", name: "worker" } }, + taskInboxToken: "commands", + parentContinuationToken: "parent", + }); + expect(deliver).toHaveBeenNthCalledWith(1, report); + expect(deliver).toHaveBeenNthCalledWith( + 2, + expect.objectContaining({ + kind: "outcome", + result: { status: "cancelled", reason: "stop" }, + }), + ); + expect(deliver).toHaveBeenCalledTimes(2); +}); + +it("keeps waiting for the body after the control hook closes", async () => { + mocks.control.mockReturnValue({ + kind: "turn", + signal: new AbortController().signal, + commands: createChannelReader("control", (async function* () {})()), + handleCommand: vi.fn(), + handleMessage: mocks.deliver, + }); + mocks.executeWorkflowBody.mockResolvedValue({ + reportCount: 0, + outcome: { status: "completed", output: "done" }, + }); + mocks.openWorkflowToolRunOwnerInbox.mockReturnValue({ + owner: { inbox: "owner" }, + reader: createChannelReader("workflow", { + [Symbol.asyncIterator]: () => ({ + next: () => new Promise>(() => {}), + }), + }), + }); + await workflowToolRunWorkflow(input); + expect(mocks.deliver).toHaveBeenCalledExactlyOnceWith( + expect.objectContaining({ + kind: "outcome", + result: { status: "completed", output: "done" }, + }), + ); +}); + +it("propagates terminal delivery failure instead of replacing the invocation outcome", async () => { + mocks.executeWorkflowBody.mockResolvedValue({ + reportCount: 0, + outcome: { status: "completed", output: "done" }, + }); + mocks.openWorkflowToolRunOwnerInbox.mockReturnValue({ + owner: { inbox: "owner" }, + reader: createChannelReader("workflow", { + [Symbol.asyncIterator]: () => ({ + next: () => new Promise>(() => {}), + }), + }), + }); + mocks.deliver.mockRejectedValue(new Error("delivery failed")); + await expect(workflowToolRunWorkflow(input)).rejects.toThrow("delivery failed"); + expect(mocks.deliver).toHaveBeenCalledOnce(); +}); diff --git a/packages/eve/src/execution/tools/workflow/workflow.ts b/packages/eve/src/execution/tools/workflow/workflow.ts index e0bfa6cac..6c40703b7 100644 --- a/packages/eve/src/execution/tools/workflow/workflow.ts +++ b/packages/eve/src/execution/tools/workflow/workflow.ts @@ -1,48 +1,163 @@ -import { sleep as workflowSleep } from "#compiled/@workflow/core/index.js"; - -import type { - WorkflowToolRunOutcome, - WorkflowToolRunOutcomeMessage, -} from "#execution/tools/workflow/messages.js"; -import { createWorkflowBodyRef, executeWorkflowBody } from "#execution/tools/workflow/body.js"; -import { openWorkflowToolRunControlInbox } from "#execution/tools/workflow/run-control.js"; -import type { WorkflowToolRunInput } from "#execution/tools/workflow/types.js"; -import { resumeHookStep } from "#execution/tools/workflow/resume-hook-step.js"; +import { WORKFLOW_CANCELLATION_CLEANUP_MS } from "#execution/tools/workflow/cancellation-policy.js"; +import { sleep } from "#compiled/@workflow/core/index.js"; import { normalizeSerializableError } from "#execution/workflow-errors.js"; +import { + createWorkflowBodyRef, + executeWorkflowBody, + type WorkflowBodyResult, +} from "#execution/tools/workflow/body.js"; +import type { WorkflowToolRunOutcome } from "#execution/tools/workflow/messages.js"; +import { + createChannelReader, + raceChannelReads, + type ChannelReader, +} from "#execution/tools/workflow/owner-channels.js"; +import { + openWorkflowToolRunOwnerInbox, + type WorkflowToolRunOwnerInbox, +} from "#execution/tools/workflow/owner.js"; +import { createBackgroundWorkflowOwner } from "#execution/tools/workflow/workflow-owner-background.js"; +import { createBlockingWorkflow } from "#execution/tools/workflow/workflow-owner-blocking.js"; +import type { + BackgroundWorkflowToolRunInput, + WorkflowToolRunInput, +} from "#execution/tools/workflow/types.js"; -const CANCEL_GRACE = "30s"; - -/** Runs one authored workflow tool call and reports to its declared owner hook. */ -export async function workflowToolRunWorkflow(input: WorkflowToolRunInput): Promise { +/** Owns admission, command intake, body execution, and settlement for either lifetime. */ +export async function workflowToolRunWorkflow( + input: WorkflowToolRunInput | BackgroundWorkflowToolRunInput, +): Promise { "use workflow"; - const control = openWorkflowToolRunControlInbox(input.hookToken); - const bodyInput = { ...input, execution: input.execution ?? "blocking" } as const; - const from = createWorkflowBodyRef(bodyInput); - const body = executeWorkflowBody(bodyInput, control.signal).then(({ outcome }) => { - if (outcome.status === "completed") return outcome.output; - if (outcome.status === "failed") throw outcome.error; - throw control.signal.reason ?? new Error(outcome.reason ?? "Workflow tool run cancelled."); - }); - const settled = body.catch(() => {}); - let outcome: WorkflowToolRunOutcome; - try { - outcome = { output: await Promise.race([body, control.cancelled]), status: "completed" }; - } catch (error) { - if (!control.signal.aborted) { - outcome = { error: normalizeSerializableError(error), status: "failed" }; - } else { - await Promise.race([settled, workflowSleep(CANCEL_GRACE)]); - outcome = { reason: control.reason(), status: "cancelled" }; - } - } + const owner = + "workflow" in input + ? await createBackgroundWorkflowOwner(input) + : createBlockingWorkflow(input); + if (owner === undefined) return; + const definition = + "workflow" in input + ? { ...input.workflow, execution: "background" as const } + : { ...input, execution: input.execution ?? "blocking" }; + const { signal } = owner; + let admitted = owner.kind === "turn"; + let commandsOpen = true; + let body: + | { + readonly inbox: WorkflowToolRunOwnerInbox; + readonly reader: ChannelReader<"body", WorkflowBodyResult>; + } + | undefined; + let consumedReports = 0; + let bodyResult: WorkflowBodyResult | undefined; + let cleanupDeadline: Promise<"cancel"> | undefined; + let outcome: WorkflowToolRunOutcome | undefined; - const message: WorkflowToolRunOutcomeMessage = { from, result: outcome }; - await resumeHookStep( - input.owner.inbox, - { kind: "outcome", ...message }, - { - ifPresent: outcome.status === "cancelled", - }, - ); + while (true) { + if (admitted && signal.aborted) { + if (body === undefined) break; + cleanupDeadline ??= sleep(WORKFLOW_CANCELLATION_CLEANUP_MS).then(() => "cancel"); + } + if ( + // Wait for the body to produce its final outcome. + bodyResult !== undefined && + // Persisted reports must also finish delivery before settlement. + consumedReports >= bodyResult.reportCount && + // Handle buffered commands, especially cancellation, before publishing the outcome. + owner.commands.landed.length === 0 && + // Propagate a command-read failure instead of hiding it behind completion. + owner.commands.failure === undefined + ) { + outcome = bodyResult.outcome; + break; + } + let read; + try { + if (admitted && body === undefined) { + const inbox = openWorkflowToolRunOwnerInbox(); + body = { + inbox, + reader: createChannelReader( + "body", + awaitBodyResult(executeWorkflowBody({ ...definition, owner: inbox.owner }, signal)), + ), + }; + } + const readers: Array< + | typeof owner.commands + | WorkflowToolRunOwnerInbox["reader"] + | ChannelReader<"body", WorkflowBodyResult> + > = []; + if (commandsOpen) readers.push(owner.commands); + if (body !== undefined) { + readers.push(body.inbox.reader); + if (bodyResult === undefined) readers.push(body.reader); + } + read = await raceChannelReads(readers, cleanupDeadline); + } catch (error) { + if (owner.commands.failure !== undefined) throw error; + outcome = { status: "failed", error: normalizeSerializableError(error) }; + break; + } + if (read === "cancel") break; + if (read.next.done) { + if (read.channel === "control") { + commandsOpen = false; + continue; + } + if (read.channel === "body") { + outcome = { status: "failed", error: "Workflow body ended without an outcome." }; + break; + } + if (admitted && signal.aborted) break; + return; + } + if (read.channel === "control") { + if (owner.kind !== "turn") + throw new Error("Session-owned workflow run received a turn command."); + owner.handleCommand(read.next.value); + continue; + } + if (read.channel === "commands") { + if (owner.kind !== "session") + throw new Error("Turn-owned workflow run received a task command."); + const payload = read.next.value; + if ( + admitted && + payload.kind === "task-command" && + (payload.command.kind === "ready" || payload.command.kind === "reject-dispatch") + ) + continue; + const action = await owner.handleCommand(payload); + if (action === "stop") return; + if (action === "start") admitted = true; + continue; + } + if (read.channel === "body") { + bodyResult = read.next.value; + continue; + } + if (read.next.value.kind === "outcome") continue; + if (read.next.value.kind === "report") consumedReports += 1; + await owner.handleMessage(read.next.value); + } + // Cleanup cannot undo cancellation, even when the body returns success. + if (signal.aborted) { + outcome = { + status: "cancelled", + reason: signal.reason instanceof Error ? signal.reason.message : String(signal.reason ?? ""), + }; + } + if (outcome !== undefined) { + await owner.handleMessage({ + from: createWorkflowBodyRef(definition), + kind: "outcome", + result: outcome, + }); + } +} + +async function* awaitBodyResult( + result: Promise, +): AsyncGenerator { + yield await result; } diff --git a/packages/eve/src/execution/workflow-runtime.test.ts b/packages/eve/src/execution/workflow-runtime.test.ts index 8282ac1bf..7484a250e 100644 --- a/packages/eve/src/execution/workflow-runtime.test.ts +++ b/packages/eve/src/execution/workflow-runtime.test.ts @@ -160,7 +160,7 @@ describe("session owner starts", () => { getRunMock.mockReturnValue({ getWritable: () => sessionWritable }); startMock.mockResolvedValue({ runId: "owner-2" }); const checkpoint = { - version: 4, + version: 6, mode: "conversation", serializedContext: {}, sessionState: { continuationToken: "continuation-1", sessionId: "session-1" }, diff --git a/packages/eve/src/execution/workflow-runtime.ts b/packages/eve/src/execution/workflow-runtime.ts index 2ad04ee15..1dbc99599 100644 --- a/packages/eve/src/execution/workflow-runtime.ts +++ b/packages/eve/src/execution/workflow-runtime.ts @@ -84,7 +84,6 @@ import { initializeSessionInstrumentation } from "#instrumentation/runtime.js"; import { ACTIVITY_COLLECTOR_WORKFLOW_NAME, SESSION_TIMEOUT_WORKFLOW_NAME, - TASK_RUN_WORKFLOW_NAME, WORKFLOW_TOOL_RUN_WORKFLOW_NAME, WORKFLOW_ENTRY_NAME, } from "#execution/stable-workflow-names.js"; @@ -119,11 +118,6 @@ export const sessionTimeoutWorkflowReference = { workflowId: `workflow//${STABLE_ID_BASE}//${SESSION_TIMEOUT_WORKFLOW_NAME}`, }; -/** Stable workflow reference for durable task runs (`experimental.tasks`). */ -export const taskRunWorkflowReference = { - workflowId: `workflow//${STABLE_ID_BASE}//${TASK_RUN_WORKFLOW_NAME}`, -}; - /** Stable workflow reference for root-session activity collectors. */ export const activityCollectorWorkflowReference = { workflowId: `workflow//${STABLE_ID_BASE}//${ACTIVITY_COLLECTOR_WORKFLOW_NAME}`, diff --git a/packages/eve/src/harness/advertised-tools.test.ts b/packages/eve/src/harness/advertised-tools.test.ts index ab136248f..f192d8dbc 100644 --- a/packages/eve/src/harness/advertised-tools.test.ts +++ b/packages/eve/src/harness/advertised-tools.test.ts @@ -134,7 +134,6 @@ function createTool(name: string): HarnessToolDefinition { function createSubagentTool(name: string): HarnessToolDefinition { return { ...createTool(name), - resultKind: "subagent", workflowId: "workflow//./agent/subagents/researcher//execute", }; } diff --git a/packages/eve/src/harness/background-tools.ts b/packages/eve/src/harness/background-tools.ts index 9516fa7c9..1be2bac92 100644 --- a/packages/eve/src/harness/background-tools.ts +++ b/packages/eve/src/harness/background-tools.ts @@ -1,19 +1,18 @@ import { loadContext } from "#context/container.js"; import { ContextKey } from "#context/key.js"; import type { InternalToolLabelDefinition, ToolExecuteOptions } from "#tools/definition.js"; -import type { TaskExec } from "#tools/task.js"; import type { AgentView } from "#subagents/handles/prompt.js"; import type { JsonValue } from "#shared/json.js"; export interface BackgroundExecutableTool { readonly label?: InternalToolLabelDefinition; - readonly execute: (input: unknown, options: ToolExecuteOptions, task: TaskExec) => unknown; readonly executeInput?: (input: unknown) => JsonValue; readonly name: string; + /** Selected agent definition's runtime graph ID; absent for authored workflow tools. */ readonly nodeId?: string; - readonly resultKind?: "subagent" | "tool"; - /** Present when the execute body runs in the task-owned durable workflow. */ - readonly workflowId?: string; + + /** Registered durable workflow body run by the session-owned task. */ + readonly workflowId: string; } export interface BackgroundToolCall { diff --git a/packages/eve/src/harness/coordination.test.ts b/packages/eve/src/harness/coordination.test.ts index 788b31267..4a517b684 100644 --- a/packages/eve/src/harness/coordination.test.ts +++ b/packages/eve/src/harness/coordination.test.ts @@ -3,7 +3,6 @@ import { isResultBoundToRunningHandle, } from "#subagents/handles/query.js"; import { describe, expect, it } from "vitest"; - import { createPresentedRuntimeActionRequestFromToolCall } from "#harness/action-presentation.js"; import { createCoordinationRequestFromToolCall, @@ -17,7 +16,11 @@ import { deriveAgentOperationId } from "#subagents/handles/operation-id.js"; import { deriveAgentId, getAgentHandleStore } from "#subagents/handles/store.js"; import { confirmAgentStarted, prepareAgentStart } from "#subagents/handles/transitions.js"; import { getProxyInputRequests, upsertProxyInputRequests } from "#harness/proxy-input-requests.js"; -import { getWorkflowToolRuns, recordWorkflowToolRun } from "#harness/workflow-tool-runs.js"; +import { + getBlockingWorkflowToolRuns, + registerWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; + import { toolOutput } from "#tools/model-output.js"; import { getSessionTokenUsage, setTurnUsageState } from "#harness/turn-tag-state.js"; import type { HarnessSession } from "#harness/types.js"; @@ -239,7 +242,6 @@ describe("createCoordinationRequestFromToolCall", () => { description: "Delegate research.", inputSchema: jsonSchema({ type: "object" }), name: "researcher", - resultKind: "subagent" as const, workflowId: "workflow://subagent-tool", }, ], @@ -252,7 +254,6 @@ describe("createCoordinationRequestFromToolCall", () => { executeInput: undefined, input: { message: "research this" }, kind: "workflow-task", - resultKind: "subagent", toolName: "researcher", workflowId: "workflow://subagent-tool", }, @@ -318,7 +319,6 @@ function createParkedSession(): HarnessSession { executeInput: { message: "go", target: "researcher" }, input: { description: "Research the topic", message: "go" }, kind: "workflow-task", - resultKind: "subagent", toolName: "researcher", workflowId: "workflow://subagent-tool", }, @@ -336,7 +336,6 @@ describe("coordination batch identity", () => { executeInput: { message: "go", target: "researcher" }, input: { message: "go" }, kind: "workflow-task" as const, - resultKind: "subagent" as const, toolName: "researcher", workflowId: "workflow://subagent-tool", }; @@ -380,7 +379,7 @@ function createSessionWithRunningChild(): HarnessSession { } describe("resolvePendingCoordination", () => { - it("marks a working task receipt as backgrounded on subagent.completed", async () => { + it("does not emit subagent completion for a working task receipt", async () => { const events: UnstampedMessageStreamEvent[] = []; const taskId = "task_0123456789abcdef"; @@ -412,9 +411,8 @@ describe("resolvePendingCoordination", () => { }, }); - expect(events.find((event) => event.type === "subagent.completed")).toMatchObject({ - data: { backgroundTask: { status: "working", taskId } }, - }); + expect(events.some((event) => event.type === "subagent.completed")).toBe(false); + expect(events).toContainEqual(expect.objectContaining({ type: "action.result" })); expect(getAgentHandleStore(resolved.session.state)).toBeUndefined(); }); @@ -477,6 +475,29 @@ describe("resolvePendingCoordination", () => { expect(getAgentHandleStore(resolved.session.state)).toEqual({ handles: [] }); }); + it("does not report a cancelled child outcome as successful completion", async () => { + const events: UnstampedMessageStreamEvent[] = []; + await resolvePendingCoordination({ + emit: async (event) => { + events.push(event); + }, + session: createSessionWithRunningChild(), + stepInput: { + runtimeActionResults: [ + { + callId: "call-1", + kind: "subagent-result", + origin: "child", + subagentName: "researcher", + output: "cancelled", + outcome: { kind: "parked", result: { kind: "cancelled" }, usageDelta: ZERO_USAGE }, + }, + ], + }, + }); + expect(events.some((event) => event.type === "subagent.completed")).toBe(false); + }); + it("clears the child's proxy-input entries before settling its handle", async () => { const session = upsertProxyInputRequests({ entries: [ @@ -526,11 +547,12 @@ describe("resolvePendingCoordination", () => { }, ], }); - const withRun = recordWorkflowToolRun(parked, { + const withRun = registerWorkflowToolRun(parked, { callId: "call-1", - hookToken: "eve:workflow-tool-run:op-1", - runId: "run-1", toolName: "deploy", + lifetime: "turn" as const, + origin: { turnId: "turn_0", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "eve:workflow-tool-run:op-1" }, }); const answerToken = "eve:workflow-tool-run-answer:run-1:0"; const session = upsertProxyInputRequests({ @@ -564,7 +586,7 @@ describe("resolvePendingCoordination", () => { }); expect(resolved.outcome).toBe("resolved"); - expect(getWorkflowToolRuns(resolved.session.state)).toEqual([]); + expect(getBlockingWorkflowToolRuns(resolved.session.state)).toEqual([]); expect([...getProxyInputRequests(resolved.session.state).keys()]).toEqual(["other-request"]); }); @@ -758,7 +780,6 @@ describe("resolvePendingCoordination", () => { executeInput: { agentId, message: "continue", target: "researcher" }, input: { agentId, message: "continue" }, kind: "workflow-task", - resultKind: "subagent", toolName: "researcher", workflowId: "workflow://subagent-tool", }, @@ -878,7 +899,6 @@ describe("resolvePendingCoordination", () => { executeInput: { agentId, message: "continue", target: "researcher" }, input: { agentId, message: "continue" }, kind: "workflow-task", - resultKind: "subagent", toolName: "researcher", workflowId: "workflow://subagent-tool", }, diff --git a/packages/eve/src/harness/coordination.ts b/packages/eve/src/harness/coordination.ts index df37a5339..5cde5662e 100644 --- a/packages/eve/src/harness/coordination.ts +++ b/packages/eve/src/harness/coordination.ts @@ -18,9 +18,8 @@ import { clearProxyInputRequestsWhere, } from "#harness/proxy-input-requests.js"; import { - findWorkflowToolRun, - isInboxSubagentResultFromRecordedWorkflowToolRun, - removeWorkflowToolRun, + findBlockingWorkflowToolRun, + removeBlockingWorkflowToolRuns, } from "#harness/workflow-tool-runs.js"; import { normalizeToolModelOutput } from "#harness/tool-model-output.js"; import type { HarnessToolDefinition } from "#harness/execute-tool.js"; @@ -199,12 +198,7 @@ function resolveResultsForCoordinationBatch(input: { pendingCallIds: [...input.batch.runtimeActions, ...input.batch.tasks].map( (request) => request.callId, ), - results: input.results.filter( - (result) => - isResultBoundToRunningHandle(input.state, result) || - (result.kind === "subagent-result" && - isInboxSubagentResultFromRecordedWorkflowToolRun(input.state, result)), - ), + results: input.results.filter((result) => isResultBoundToRunningHandle(input.state, result)), }); } @@ -246,32 +240,6 @@ export async function resolvePendingCoordination(input: { }; } - if (input.emit !== undefined) { - for (const result of readyResults) { - if (result.kind === "subagent-result" && result.isError !== true) { - const backgroundTask = readBackgroundTaskReceipt(result); - const data = { - callId: result.callId, - output: typeof result.output === "string" ? result.output : JSON.stringify(result.output), - subagentName: result.subagentName, - }; - await input.emit({ - data: backgroundTask === undefined ? data : { ...data, backgroundTask }, - type: "subagent.completed", - } satisfies Extract); - } - - await input.emit( - createActionResultEvent({ - result, - sequence: batch.event.sequence, - stepIndex: batch.event.stepIndex, - turnId: batch.event.turnId, - }), - ); - } - } - // Settle each bound child result against its running handle from the // outcome the child engine reported: `parked` keeps the handle (the child // is idle and resumable), `terminal` deletes it. Before a terminal @@ -313,22 +281,18 @@ export async function resolvePendingCoordination(input: { // Drop a finished run's unanswered requests so a late click cannot reach it. for (const result of readyResults) { if (result.kind !== "tool-result") continue; - const record = findWorkflowToolRun(nextSession.state, result.callId); + const record = findBlockingWorkflowToolRun( + nextSession.state, + result.callId, + batch.event.turnId, + ); if (record === undefined) continue; - nextSession = removeWorkflowToolRun( + nextSession = removeBlockingWorkflowToolRuns( clearProxyInputRequestsWhere( nextSession, - (route) => route.answerHook?.runId === record.runId, + (route) => route.answerHook?.runId === record.address.runId, ), - record.callId, - ); - } - for (const result of readyResults) { - if (result.kind !== "subagent-result") continue; - const record = findWorkflowToolRun(nextSession.state, result.callId); - if (record?.resultKind !== "subagent") continue; - nextSession = removeWorkflowToolRun( - clearProxyInputRequestsForChild(nextSession, record.hookToken), + batch.event.turnId, record.callId, ); } @@ -364,6 +328,36 @@ export async function resolvePendingCoordination(input: { ); } + if (input.emit !== undefined) { + for (const result of readyResults) { + if ( + result.kind === "subagent-result" && + result.origin === "child" && + result.outcome.result.kind === "succeeded" && + readBackgroundTaskReceipt(result) === undefined + ) { + const data = { + callId: result.callId, + output: typeof result.output === "string" ? result.output : JSON.stringify(result.output), + subagentName: result.subagentName, + }; + await input.emit({ + data, + type: "subagent.completed", + } satisfies Extract); + } + + await input.emit( + createActionResultEvent({ + result, + sequence: batch.event.sequence, + stepIndex: batch.event.stepIndex, + turnId: batch.event.turnId, + }), + ); + } + } + const toolResults: ToolResultPart[] = []; for (const result of readyResults) { switch (result.kind) { @@ -472,7 +466,6 @@ export function createCoordinationRequestFromToolCall(input: { executeInput: definition.executeInput?.(inputObject), input: inputObject, kind: "workflow-task", - resultKind: definition.resultKind, toolName: input.toolCall.toolName, workflowId: definition.workflowId, }, diff --git a/packages/eve/src/harness/emission.test.ts b/packages/eve/src/harness/emission.test.ts index 9f64cf9e6..97a85c1f7 100644 --- a/packages/eve/src/harness/emission.test.ts +++ b/packages/eve/src/harness/emission.test.ts @@ -579,7 +579,7 @@ describe("emitStreamContent action requests", () => { description: "Delegate work to a subagent.", inputSchema: jsonSchema({ type: "object" }), name: "delegate", - resultKind: "subagent", + nodeId: "subagents/researcher", workflowId: "workflow//./agent/subagents/researcher//execute", }, ], @@ -718,7 +718,7 @@ describe("emitStreamContent action requests", () => { ]); }); - it("marks a background subagent receipt on subagent.completed", async () => { + it("returns a background receipt without announcing subagent completion", async () => { const emit = createEmitStub(); const tools = new Map([ [ @@ -728,7 +728,7 @@ describe("emitStreamContent action requests", () => { execution: "background", inputSchema: jsonSchema({ type: "object" }), name: "delegate", - resultKind: "subagent", + nodeId: "subagents/researcher", workflowId: "workflow//./agent/subagents/researcher//execute", }, ], @@ -756,71 +756,12 @@ describe("emitStreamContent action requests", () => { ); const events = vi.mocked(emit).mock.calls.map(([event]) => event); - expect(events.map((event) => event.type)).toEqual([ - "actions.requested", - "subagent.completed", - "action.result", - ]); + expect(events.map((event) => event.type)).toEqual(["actions.requested", "action.result"]); expect(events[1]).toMatchObject({ data: { - backgroundTask: { status: "working", taskId: "task-1" }, - callId: "call-delegate", - subagentName: "delegate", + result: { callId: "call-delegate", output: { status: "working", taskId: "task-1" } }, }, - type: "subagent.completed", - }); - }); - - it("marks a background subagent receipt on subagent.completed", async () => { - const emit = createEmitStub(); - const tools = new Map([ - [ - "delegate", - { - description: "Delegate work to a subagent.", - execution: "background", - inputSchema: jsonSchema({ type: "object" }), - name: "delegate", - resultKind: "subagent", - workflowId: "workflow//./agent/subagents/researcher//execute", - }, - ], - ]); - - await emitStreamContent( - emit, - EMISSION_STATE, - streamOf([ - { - input: { message: "research the release" }, - toolCallId: "call-delegate", - toolName: "delegate", - type: "tool-call", - }, - { - output: { status: "working", taskId: "task-1" }, - toolCallId: "call-delegate", - toolName: "delegate", - type: "tool-result", - }, - { finishReason: "tool-calls", type: "finish-step" }, - ] as TextStreamPart[]), - { excludedActionToolNames: new Set(), tools }, - ); - - const events = vi.mocked(emit).mock.calls.map(([event]) => event); - expect(events.map((event) => event.type)).toEqual([ - "actions.requested", - "subagent.completed", - "action.result", - ]); - expect(events[1]).toMatchObject({ - data: { - backgroundTask: { status: "working", taskId: "task-1" }, - callId: "call-delegate", - subagentName: "delegate", - }, - type: "subagent.completed", + type: "action.result", }); }); diff --git a/packages/eve/src/harness/emission.ts b/packages/eve/src/harness/emission.ts index de2f411ab..c562ec84c 100644 --- a/packages/eve/src/harness/emission.ts +++ b/packages/eve/src/harness/emission.ts @@ -273,21 +273,6 @@ interface StreamActionEmissionOptions { readonly tools: HarnessToolMap; } -function readSubagentBackgroundTaskReceipt( - result: RuntimeToolResultActionResult, - tools: HarnessToolMap | undefined, -): { readonly status: "working"; readonly taskId: string } | undefined { - if (result.isError === true || tools?.get(result.toolName)?.resultKind !== "subagent") { - return undefined; - } - if (typeof result.output !== "object" || result.output === null || Array.isArray(result.output)) { - return undefined; - } - const status = Reflect.get(result.output, "status"); - const taskId = Reflect.get(result.output, "taskId"); - return status === "working" && typeof taskId === "string" ? { status, taskId } : undefined; -} - /** * Consumes the AI SDK `fullStream` and emits real-time text and reasoning * events. @@ -439,18 +424,6 @@ async function consumeStreamContent( return; } emittedActionResultCallIds.add(result.callId); - const backgroundTask = readSubagentBackgroundTaskReceipt(result, options?.tools); - if (backgroundTask !== undefined) { - await emitFn({ - data: { - backgroundTask, - callId: result.callId, - output: typeof result.output === "string" ? result.output : JSON.stringify(result.output), - subagentName: result.toolName, - }, - type: "subagent.completed", - }); - } const resultPresentation = result.isError === true ? undefined diff --git a/packages/eve/src/harness/execute-tool.ts b/packages/eve/src/harness/execute-tool.ts index b5412dcd4..593281b30 100644 --- a/packages/eve/src/harness/execute-tool.ts +++ b/packages/eve/src/harness/execute-tool.ts @@ -2,7 +2,6 @@ import type { FlexibleSchema } from "ai"; import type { Approval } from "#approval/definition.js"; import type { InternalToolLabelDefinition, ToolExecuteOptions } from "#tools/definition.js"; -import type { TaskExec } from "#tools/task.js"; import type { JsonValue } from "#shared/json.js"; import type { PreparedToolBehavior } from "#tools/behavior.js"; @@ -23,43 +22,17 @@ export interface HarnessToolDefinition { readonly approvalKey?: (toolInput: Readonly>) => string; readonly behavior?: PreparedToolBehavior; readonly description: string; - readonly execute?: (input: any, options: ToolExecuteOptions, task?: TaskExec) => any; + readonly execute?: (input: any, options: ToolExecuteOptions) => any; /** Optional JSON input substituted when this tool starts its workflow body. */ readonly executeInput?: (input: unknown) => JsonValue; readonly execution?: "background"; readonly frameworkAction?: "load-skill"; readonly inputSchema: FlexibleSchema; readonly name: string; - /** Runtime graph node for a framework subagent workflow body. */ + /** Selected agent definition's runtime graph ID; absent for authored workflow tools. */ readonly nodeId?: string; readonly approval?: Approval; readonly outputSchema?: FlexibleSchema; - /** - * How the result of this workflow-backed tool is settled: as a - * `subagent-result` (delegation tools — `behavior.handling.target.kind` is - * `subagent-call`, `remote-agent-call`, or `self-agent-call`) or as an - * ordinary `tool-result` (authored workflow tools). Absent means `"tool"`. - * - * On the definition itself this duplicates the dispatch target kind; it - * exists because the value must survive past the tool map. `buildToolSet`, - * `createCoordinationRequestFromToolCall`, and the workflow sandbox host - * tool copy it into the `RuntimeWorkflowTaskRequest`, which `startWorkflowTask` - * persists on the `WorkflowToolRunRecord` in session state and the run echoes - * back on every `WorkflowToolRunRef` inbox message. The owner turn then routes - * outcomes, counts the workflow subagent budget, and decides whether child - * usage accrues without access to a `HarnessToolMap`. Harness-side readers - * (`advertised-tools`, `emission`, the background tool executor) use it to - * expose only delegation tools inside workflow sandboxes, emit task receipts, - * and reserve/claim agent handles for subagent starts. - * - * The persisted copy is dropped by `removeWorkflowToolRun` when the run - * settles, or `clearWorkflowToolRuns` at turn end. - * - * TODO: once subagent starts no longer need harness-specific handling, - * derive this from `behavior.handling.target.kind` at the projection points - * above and remove the field. - */ - readonly resultKind?: "subagent" | "tool"; /** * Advertise this tool only to the root session, hiding it from subagent * sessions. Set on the injected `agent` self-delegation tool so children diff --git a/packages/eve/src/harness/tool-loop-generate-approval-resume.integration.test.ts b/packages/eve/src/harness/tool-loop-generate-approval-resume.integration.test.ts index c1f016092..e917595c6 100644 --- a/packages/eve/src/harness/tool-loop-generate-approval-resume.integration.test.ts +++ b/packages/eve/src/harness/tool-loop-generate-approval-resume.integration.test.ts @@ -2,7 +2,6 @@ import { setTurnClientContextState } from "#harness/turn-client-context.js"; import { jsonSchema, type LanguageModel, type ModelMessage, simulateReadableStream } from "ai"; import { MockLanguageModelV4 } from "ai/test"; import { describe, expect, it, vi } from "vitest"; - import { ContextContainer, contextStorage } from "#context/container.js"; import { dispatchDynamicToolEvent, @@ -29,7 +28,7 @@ import { getPendingInputBatches } from "#harness/pending-input-batches.js"; import { createToolLoopHarness } from "#harness/tool-loop.js"; import { setTurnUsageState } from "#harness/turn-tag-state.js"; import type { HarnessSession, ToolLoopHarnessConfig } from "#harness/types.js"; -import { recordSessionTask } from "#tasks/session-index.js"; +import { registerWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import { once } from "#tools/approval/policies.js"; import { defineTool } from "#tools/definition.js"; import { @@ -876,14 +875,17 @@ describe("tool loop generate approval resume (real AI SDK)", () => { : "Available skills\n- policy: Tenant policy"; if (historyKey === "taskState") { ctx.set(TurnTaskDeliveryKey, "initiating"); - session = recordSessionTask(session, { - createdByTurnId: "turn-1", - dispatchContext: { auth: { current: null, initiator: null } }, - executor: { data: {}, kind: "workflow-tool" }, - metadata: { kind: "report-probe", name: "analysis" }, - taskId: "analysis", - taskInboxToken: "task-token", - taskRunId: "task-run", + session = registerWorkflowToolRun(session, { + callId: "analysis", + toolName: "analysis", + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "report-probe", name: "analysis" }, + taskId: "analysis", + }, }); } else { ctx.set(PendingSkillAnnouncementKey, runtimeContextAnnouncement); diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts index 203268a0b..2522b5ebb 100644 --- a/packages/eve/src/harness/tool-loop.test.ts +++ b/packages/eve/src/harness/tool-loop.test.ts @@ -11,7 +11,6 @@ import { } from "ai"; import { MockLanguageModelV3 } from "ai/test"; import { afterEach, describe, expect, it, vi } from "vitest"; - import { ContextContainer, contextStorage } from "#context/container.js"; import { DynamicModelSelectionError } from "#context/dynamic-model-lifecycle.js"; import { dispatchDynamicInstructionEvent } from "#context/dynamic-instruction-lifecycle.js"; @@ -80,7 +79,7 @@ import { appendPendingInputBatch, } from "#harness/input-requests.js"; import { activeTurnId } from "#harness/active-turn-id.js"; -import { recordSessionTask } from "#tasks/session-index.js"; +import { registerWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import { getPendingCoordinationBatch } from "#harness/coordination.js"; import { AGENT_HANDLES_STATE_KEY } from "#subagents/handles/store.js"; import { BackgroundToolExecutorKey } from "#harness/background-tools.js"; @@ -302,14 +301,17 @@ const analysisTaskAnnouncement = '[Task state]\n{"tasks":[{"name":"analysis","status":"pending","taskId":"analysis"}]}'; function recordBackgroundTask(session: HarnessSession, taskId = "analysis"): HarnessSession { - return recordSessionTask(session, { - createdByTurnId: activeTurnId(getHarnessEmissionState(session.state)), - dispatchContext: { auth: { current: null, initiator: null } }, - executor: { data: {}, kind: "workflow-tool" }, - metadata: { kind: "report-probe", name: taskId }, - taskId, - taskInboxToken: `token-${taskId}`, - taskRunId: `run-${taskId}`, + return registerWorkflowToolRun(session, { + callId: taskId, + toolName: { kind: "report-probe", name: taskId }.name, + lifetime: "session" as const, + origin: { turnId: activeTurnId(getHarnessEmissionState(session.state)), stepIndex: 0 }, + address: { runId: `run-${taskId}`, hookToken: `token-${taskId}` }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "report-probe", name: taskId }, + taskId, + }, }); } @@ -355,7 +357,6 @@ function createDelegationToolMap(): ToolLoopHarnessConfig["tools"] { description: "Delegate to a subagent.", inputSchema: jsonSchema({ type: "object" }), name: "delegate", - resultKind: "subagent", workflowId: "workflow//./agent/subagents/researcher//execute", }, ], @@ -1243,6 +1244,7 @@ describe("createToolLoopHarness", () => { execution: "background" as const, inputSchema: jsonSchema({ type: "object" }), name: "background_work", + workflowId: "workflow//test//background_work", }, ], ]), @@ -1843,7 +1845,6 @@ describe("createToolLoopHarness", () => { callId: "call-1", input: { message: "delegate from child" }, kind: "workflow-task", - resultKind: "subagent", toolName: "delegate", }), ]); @@ -4173,7 +4174,6 @@ describe("createToolLoopHarness", () => { description: "Delegate to a subagent.", inputSchema: jsonSchema({ type: "object" }), name: "delegate", - resultKind: "subagent", workflowId: "workflow//./agent/subagents/researcher//execute", }, ], @@ -4230,7 +4230,6 @@ describe("createToolLoopHarness", () => { description: "Delegate to a subagent.", inputSchema: jsonSchema({ type: "object" }), name: "delegate", - resultKind: "subagent", workflowId: "workflow//./agent/subagents/researcher//execute", }, ], diff --git a/packages/eve/src/harness/tools.test.ts b/packages/eve/src/harness/tools.test.ts index 97f8c1389..98025e864 100644 --- a/packages/eve/src/harness/tools.test.ts +++ b/packages/eve/src/harness/tools.test.ts @@ -152,6 +152,7 @@ describe("buildToolSet", () => { execution: "background", inputSchema: jsonSchema({ type: "object" }), name: "background_work", + workflowId: "workflow//test//background_work", }, ], ]); diff --git a/packages/eve/src/harness/tools.ts b/packages/eve/src/harness/tools.ts index e601a5248..e849a91d3 100644 --- a/packages/eve/src/harness/tools.ts +++ b/packages/eve/src/harness/tools.ts @@ -77,12 +77,10 @@ export function buildToolSet(input: { definition.name, definition.execution === "background" && definition.execute !== undefined ? { - execute: definition.execute, executeInput: definition.executeInput, name: definition.name, nodeId: definition.nodeId, - resultKind: definition.resultKind, - workflowId: definition.workflowId, + workflowId: requireBackgroundWorkflowId(definition), } : undefined, ); @@ -171,6 +169,15 @@ export function buildToolSet(input: { return tools as ToolSet; } +function requireBackgroundWorkflowId(definition: HarnessToolDefinition): string { + if (definition.workflowId === undefined) { + throw new Error( + `Background tool "${definition.name}" must be defined with defineWorkflowTool().`, + ); + } + return definition.workflowId; +} + /** * Builds a ToolSet from an ordered list of harness definitions. * diff --git a/packages/eve/src/harness/workflow-tool-run-registry.test.ts b/packages/eve/src/harness/workflow-tool-run-registry.test.ts new file mode 100644 index 000000000..a8fd286cc --- /dev/null +++ b/packages/eve/src/harness/workflow-tool-run-registry.test.ts @@ -0,0 +1,291 @@ +import { runInNewContext } from "node:vm"; +import { describe, expect, it } from "vitest"; +import type { SessionStateMap } from "#harness/types.js"; +import { + readWorkflowTaskView, + recordWorkflowTaskView, + findBlockingWorkflowToolRun, + getWorkflowToolRuns, + registerWorkflowToolRun, + removeBlockingWorkflowToolRuns, + type BackgroundWorkflowToolRun, + type BlockingWorkflowToolRun, +} from "./workflow-tool-runs.js"; +import { getSessionTaskCohorts } from "#tasks/session-task-cohorts.js"; +import { resolveTaskDeliveryContext } from "#tasks/delivery-context.js"; + +const waiting = (turnId: string): BlockingWorkflowToolRun => ({ + lifetime: "turn", + callId: "same-call", + toolName: "research", + origin: { turnId, stepIndex: 0 }, + address: { runId: `run-${turnId}`, hookToken: `hook-${turnId}` }, +}); +const task = (taskId: string): BackgroundWorkflowToolRun => ({ + ...waiting("turn-a"), + callId: taskId, + lifetime: "session", + address: { runId: `run-${taskId}`, hookToken: `hook-${taskId}` }, + task: { + taskId, + metadata: { kind: "tool", name: "research" }, + dispatchContext: { auth: { current: null, initiator: null } }, + }, +}); + +describe("shared workflow invocation ownership", () => { + it("stores blocking and background runs together under eve.workflowTool", () => { + const blocking = waiting("turn-a"); + const background = task("task-a"); + const initial: { state?: SessionStateMap } = {}; + const session = registerWorkflowToolRun(registerWorkflowToolRun(initial, blocking), background); + expect(session.state).toEqual({ + "eve.workflowTool": { version: 3, runs: [blocking, background] }, + }); + }); + + it("clears only one turn while retaining another turn, live tasks and completed payloads", () => { + let session: { state?: SessionStateMap } = registerWorkflowToolRun({}, waiting("turn-a")); + session = registerWorkflowToolRun(session, waiting("turn-b")); + session = registerWorkflowToolRun(session, task("task-a")); + session = registerWorkflowToolRun(session, task("task-b")); + session = { + ...session, + state: recordWorkflowTaskView(session.state, { + taskId: "task-a", + metadata: task("task-a").task.metadata, + status: "completed", + lastOutput: { type: "result", data: "first" }, + }), + }; + session = removeBlockingWorkflowToolRuns(session, "turn-a"); + const entries = getWorkflowToolRuns(session.state); + expect(entries.map((entry) => [entry.lifetime, entry.callId])).toEqual([ + ["turn", "same-call"], + ["session", "task-a"], + ["session", "task-b"], + ]); + expect(findBlockingWorkflowToolRun(session.state, "same-call", "turn-a")).toBeUndefined(); + expect(findBlockingWorkflowToolRun(session.state, "same-call", "turn-b")).toEqual( + waiting("turn-b"), + ); + expect( + resolveTaskDeliveryContext({ state: session.state, taskDeliveryId: "task-a:ready:completed" }) + ?.phase, + ).toBe("pending"); + session = { + ...session, + state: recordWorkflowTaskView(session.state, { + taskId: "task-b", + metadata: task("task-b").task.metadata, + status: "failed", + lastOutput: { type: "error", data: "second" }, + }), + }; + const restored = JSON.parse( + JSON.stringify(removeBlockingWorkflowToolRuns(session, "turn-b").state), + ); + const beforeReport = JSON.stringify(restored); + const report = resolveTaskDeliveryContext({ + state: restored, + taskDeliveryId: "task-b:ready:failed", + }); + expect(report?.phase).toBe("settled"); + expect(report?.context).toContain("first"); + expect(report?.context).toContain("second"); + expect([...getSessionTaskCohorts(restored).values()]).toEqual(["task-a", "task-a"]); + expect( + JSON.stringify(removeBlockingWorkflowToolRuns({ state: restored }, "turn-a").state), + ).toBe(beforeReport); + }); + + it("preserves malformed historical results during unrelated ownership mutations", () => { + const old = task("old"); + const retained = { ...old, task: { ...old.task, outcome: { status: "completed" } } }; + let session: { state?: SessionStateMap } = { + state: { "eve.workflowTool": { version: 3, runs: [retained] } }, + }; + session = registerWorkflowToolRun(session, waiting("turn-b")); + session = registerWorkflowToolRun(session, task("live")); + session = { + ...session, + state: recordWorkflowTaskView(session.state, { + taskId: "live", + metadata: old.task.metadata, + status: "cancelled", + }), + }; + session = removeBlockingWorkflowToolRuns(session, "turn-b"); + const entries = getWorkflowToolRuns(session.state); + expect(entries[0]).toEqual(retained); + expect(() => readWorkflowTaskView(retained.task)).toThrow("Corrupt workflow task result"); + expect(entries).toHaveLength(2); + }); + + it("retains creator auth and opaque selections without sharing parsed mutable containers", () => { + const entry = task("task-a"); + const principal = { + attributes: { roles: ["researcher"], team: "eve" }, + authenticator: "test", + principalId: "alice", + principalType: "user", + }; + const selections = { researcher: { futureSelection: true } }; + const retained = { + ...entry, + task: { + ...entry.task, + dispatchContext: { + auth: { current: principal, initiator: principal }, + sessionDynamicSubagentSelections: selections, + }, + }, + }; + const [parsed] = getWorkflowToolRuns({ + "eve.workflowTool": { version: 3, runs: [retained] }, + }); + expect(parsed).toEqual(retained); + expect(parsed?.lifetime).toBe("session"); + if (parsed?.lifetime !== "session") throw new Error("Expected a background run."); + const context = parsed.task.dispatchContext; + expect(context.auth.current).not.toBe(principal); + expect(context.auth.current?.attributes.roles).not.toBe(principal.attributes.roles); + expect(Object.isFrozen(context.auth.current?.attributes.roles)).toBe(true); + expect(context.sessionDynamicSubagentSelections).not.toBe(selections); + expect(context.sessionDynamicSubagentSelections?.researcher).toBe(selections.researcher); + }); + + it("reads creator auth and dynamic selections restored in the workflow VM", () => { + const entry = task("task-a"); + const state = { + "eve.workflowTool": { + version: 3, + runs: [ + { + ...entry, + task: { + ...entry.task, + dispatchContext: { + auth: { + current: { + attributes: { roles: ["researcher"] }, + authenticator: "test", + principalId: "alice", + principalType: "user", + }, + initiator: null, + }, + sessionDynamicSubagentSelections: { researcher: { futureSelection: true } }, + turnDynamicSubagentSelections: {}, + }, + }, + }, + ], + }, + }; + const restored = runInNewContext("JSON.parse(input)", { input: JSON.stringify(state) }); + expect(Object.getPrototypeOf(restored)).not.toBe(Object.prototype); + expect(getWorkflowToolRuns(restored)).toEqual(state["eve.workflowTool"].runs); + expect(() => registerWorkflowToolRun({ state: restored }, waiting("turn-b"))).not.toThrow(); + }); + + it.each([ + { auth: { current: null } }, + { auth: { current: null, initiator: null, injected: true } }, + { + auth: { + current: { + attributes: { roles: Array(1) }, + authenticator: "test", + principalId: "alice", + principalType: "user", + }, + initiator: null, + }, + }, + { + auth: { + current: { + attributes: { roles: ["researcher", 1] }, + authenticator: "test", + principalId: "alice", + principalType: "user", + }, + initiator: null, + }, + }, + { + auth: { + current: { + attributes: {}, + authenticator: "test", + principalId: "alice", + principalType: "user", + injected: true, + }, + initiator: null, + }, + }, + { auth: { current: null, initiator: null }, sessionDynamicSubagentSelections: [] }, + ])("rejects malformed retained creator authority: %j", (dispatchContext) => { + const entry = task("task-a"); + expect(() => + getWorkflowToolRuns({ + "eve.workflowTool": { + version: 3, + runs: [{ ...entry, task: { ...entry.task, dispatchContext } }], + }, + }), + ).toThrow("Corrupt workflow tool run registry"); + }); + + it("rejects duplicate originating call identities even without task payloads", () => { + expect(() => + getWorkflowToolRuns({ + "eve.workflowTool": { + version: 3, + runs: [waiting("turn-a"), waiting("turn-a")], + }, + }), + ).toThrow("Run identities must be unique"); + }); + + it.each([Infinity, NaN, -1, "0"])("rejects corrupt terminal usage %s", (costUsd) => { + const entry = task("task-a"); + expect(() => + readWorkflowTaskView({ + ...entry.task, + outcome: { + status: "completed", + lastOutput: { type: "result", data: "done" }, + usage: { + cacheReadTokens: 0, + cacheWriteTokens: 0, + inputTokens: 1, + outputTokens: 1, + costUsd, + }, + }, + }), + ).toThrow("Corrupt workflow task result"); + }); + + it("does not change lifetime on replay", () => { + const session = registerWorkflowToolRun({}, waiting("turn-a")); + expect(() => + registerWorkflowToolRun(session, { ...task("task-a"), callId: "same-call" }), + ).toThrow("Replayed invocation changed its ownership"); + }); + + it("rejects the task-only index from main", () => { + expect(() => getWorkflowToolRuns({ "eve.tasks": { version: 2, tasks: [] } })).toThrow( + "Unsupported workflow tool run state", + ); + }); + + it("rejects the separate blocking-run store from main", () => { + expect(() => getWorkflowToolRuns({ "eve.runtime.workflowToolRuns": [] })).toThrow( + "Unsupported workflow tool run state", + ); + }); +}); diff --git a/packages/eve/src/harness/workflow-tool-run-tasks.test.ts b/packages/eve/src/harness/workflow-tool-run-tasks.test.ts new file mode 100644 index 000000000..1f346b45c --- /dev/null +++ b/packages/eve/src/harness/workflow-tool-run-tasks.test.ts @@ -0,0 +1,465 @@ +import { assert, describe, expect, it } from "vitest"; +import type { HarnessSession } from "#harness/types.js"; +import { + readWorkflowTaskView, + recordWorkflowTaskView, + findBackgroundWorkflowToolRun, + getBackgroundWorkflowToolRuns, + registerWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; +import { getTaskCohortId, getSessionTaskCohorts } from "#tasks/session-task-cohorts.js"; +import { deriveTaskId } from "#tasks/task-id.js"; +import type { TaskView } from "#tasks/types.js"; + +function createSession(state?: HarnessSession["state"]): HarnessSession { + return { + agent: { + modelReference: { id: "model_test" }, + system: "", + tools: [], + }, + compaction: { recentWindowSize: 4, threshold: 1_000_000 }, + continuationToken: "continuation_test", + history: [], + sessionId: "session_parent", + state, + }; +} + +describe("session task index", () => { + const metadata = { + kind: "tool" as const, + name: "research", + }; + const dispatchContext = { auth: { current: null, initiator: null } } as const; + it("returns an empty index when the key is absent", () => { + expect(getBackgroundWorkflowToolRuns({})).toEqual([]); + expect(getBackgroundWorkflowToolRuns(undefined)).toEqual([]); + }); + + it("records a task and finds it by id", () => { + const session = registerWorkflowToolRun(createSession(), { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { dispatchContext, metadata, taskId: "task_a" }, + }); + + expect(findBackgroundWorkflowToolRun(session.state, "task_a")).toEqual({ + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { dispatchContext, metadata, taskId: "task_a" }, + }); + expect(findBackgroundWorkflowToolRun(session.state, "task_other")).toBeUndefined(); + }); + + it("keeps activity identity in the persisted task index", () => { + const activityWorkIdentity = { + callId: "call-1", + id: "work:task", + kind: "task" as const, + name: "research", + parentId: "work:root", + rootSessionId: "root-session", + rootTurnId: "root-turn", + }; + const session = registerWorkflowToolRun(createSession(), { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { activityWorkIdentity, dispatchContext, metadata, taskId: "task_a" }, + }); + + const restoredState = JSON.parse(JSON.stringify(session.state)); + expect( + findBackgroundWorkflowToolRun(restoredState, "task_a")?.task.activityWorkIdentity, + ).toEqual(activityWorkIdentity); + }); + + it("keeps subagent metadata in the persisted task index", () => { + const subagentMetadata = { + agentId: "ag_worker", + kind: "subagent", + mode: "remote", + name: "research", + } as const; + + const session = registerWorkflowToolRun(createSession(), { + callId: "task_a", + toolName: subagentMetadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { dispatchContext, metadata: subagentMetadata, taskId: "task_a" }, + }); + + expect(findBackgroundWorkflowToolRun(session.state, "task_a")?.task.metadata).toEqual( + subagentMetadata, + ); + }); + + it("keeps a terminal view when replayed activity presentation changes", () => { + let session = registerWorkflowToolRun(createSession(), { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { + activityWorkIdentity: { + callId: "call-1", + id: "work:task", + kind: "task", + label: "First label", + name: "research", + parentId: "work:root", + rootSessionId: "root-session", + rootTurnId: "root-turn", + }, + dispatchContext, + metadata, + taskId: "task_a", + }, + }); + session = { + ...session, + state: recordWorkflowTaskView(session.state, terminal("task_a", "completed")), + }; + session = registerWorkflowToolRun(session, { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-2", hookToken: "task:token-2" }, + task: { + activityWorkIdentity: { + callId: "call-1", + id: "work:task", + kind: "task", + label: "Second label", + name: "research", + parentId: "work:root", + rootSessionId: "root-session", + rootTurnId: "root-turn", + }, + dispatchContext, + metadata, + taskId: "task_a", + }, + }); + + expect(findBackgroundWorkflowToolRun(session.state, "task_a")).toMatchObject({ + task: { + activityWorkIdentity: { label: "Second label" }, + outcome: { status: "completed", lastOutput: { type: "result", data: "done" } }, + }, + }); + }); + + it("replaces the entry on replayed creation instead of duplicating it", () => { + let session = registerWorkflowToolRun(createSession(), { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { dispatchContext, metadata, taskId: "task_a" }, + }); + session = registerWorkflowToolRun(session, { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-2", hookToken: "task:token-2" }, + task: { dispatchContext, metadata, taskId: "task_a" }, + }); + + const entries = getBackgroundWorkflowToolRuns(session.state); + expect(entries).toHaveLength(1); + expect(entries[0]?.address.runId).toBe("run-2"); + }); + + function task(taskId: string, createdByTurnId: string) { + return { + callId: taskId, + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: createdByTurnId, stepIndex: 0 }, + address: { runId: `run-${taskId}`, hookToken: `inbox-${taskId}` }, + task: { dispatchContext, metadata, taskId }, + }; + } + + function terminal(taskId: string, status: "completed" | "failed" | "cancelled"): TaskView { + if (status === "cancelled") return { metadata, status, taskId }; + return status === "completed" + ? { metadata, status, taskId, lastOutput: { type: "result", data: "done" } } + : { metadata, status, taskId, lastOutput: { type: "error", data: "failed" } }; + } + + it("durably joins overlapping work across turns", () => { + const first = task("task_a", "turn-1"); + const initial = registerWorkflowToolRun(createSession(), first); + const second = task("task_b", "turn-2"); + const session = registerWorkflowToolRun(initial, second); + const entries = getBackgroundWorkflowToolRuns(session.state); + expect(entries.map((entry) => getTaskCohortId(entry.task))).toEqual(["task_a", "task_a"]); + expect(entries.map((entry) => entry.origin.turnId)).toEqual(["turn-1", "turn-2"]); + expect(entries[0]?.task.cohortId).toBeUndefined(); + expect(entries[1]?.task.cohortId).toBe("task_a"); + const restored = createSession(JSON.parse(JSON.stringify(initial.state))); + expect(registerWorkflowToolRun(restored, second).state).toEqual(session.state); + expect(getBackgroundWorkflowToolRuns(initial.state)).toHaveLength(1); + }); + + it.each(["completed", "failed", "cancelled"] as const)( + "keeps a %s sibling in a pending cohort, then starts a new cohort after settlement", + (status) => { + let session = registerWorkflowToolRun(createSession(), task("task_a", "turn-1")); + session = registerWorkflowToolRun(session, task("task_b", "turn-1")); + session = { + ...session, + state: recordWorkflowTaskView(session.state, terminal("task_a", status)), + }; + session = registerWorkflowToolRun(session, task("task_c", "turn-2")); + expect( + getBackgroundWorkflowToolRuns(session.state).map((entry) => getTaskCohortId(entry.task)), + ).toEqual(["task_a", "task_a", "task_a"]); + expect([...getSessionTaskCohorts(session.state).values()]).toEqual([ + "task_a", + "task_a", + "task_a", + ]); + for (const taskId of ["task_b", "task_c"]) { + session = { + ...session, + state: recordWorkflowTaskView(session.state, terminal(taskId, status)), + }; + } + // Even another creation in the same turn must not reopen a settled cohort. + session = registerWorkflowToolRun(session, task("task_d", "turn-2")); + expect( + getBackgroundWorkflowToolRuns(session.state).map((entry) => getTaskCohortId(entry.task)), + ).toEqual(["task_a", "task_a", "task_a", "task_d"]); + }, + ); + + it("preserves replayed membership, creation provenance, order, and settlement", () => { + let session = registerWorkflowToolRun(createSession(), task("task_a", "turn-1")); + session = registerWorkflowToolRun(session, task("task_b", "turn-2")); + for (const taskId of ["task_a", "task_b"]) { + session = { + ...session, + state: recordWorkflowTaskView(session.state, terminal(taskId, "completed")), + }; + } + session = registerWorkflowToolRun(session, task("task_c", "turn-3")); + session = registerWorkflowToolRun(session, { + ...task("task_a", "turn-1"), + origin: { ...task("task_a", "turn-1").origin, stepIndex: 9 }, + address: { ...task("task_a", "turn-1").address, runId: "run-replayed" }, + }); + session = registerWorkflowToolRun(session, { + ...task("task_b", "turn-2"), + origin: { ...task("task_b", "turn-2").origin, stepIndex: 9 }, + }); + expect( + getBackgroundWorkflowToolRuns(session.state).map((entry) => ({ + taskId: entry.task.taskId, + cohortId: getTaskCohortId(entry.task), + turnId: entry.origin.turnId, + stepIndex: entry.origin.stepIndex, + settled: entry.task.outcome !== undefined, + })), + ).toEqual([ + { taskId: "task_a", cohortId: "task_a", turnId: "turn-1", stepIndex: 0, settled: true }, + { taskId: "task_b", cohortId: "task_a", turnId: "turn-2", stepIndex: 0, settled: true }, + { taskId: "task_c", cohortId: "task_c", turnId: "turn-3", stepIndex: 0, settled: false }, + ]); + expect(findBackgroundWorkflowToolRun(session.state, "task_a")?.address.runId).toBe( + "run-replayed", + ); + session = registerWorkflowToolRun(session, task("task_d", "turn-4")); + expect(findBackgroundWorkflowToolRun(session.state, "task_d")?.task.cohortId).toBe("task_c"); + }); + + it.each(["", null, 42])("rejects an invalid additive cohort identity: %j", (cohortId) => { + expect(() => + getBackgroundWorkflowToolRuns({ + "eve.workflowTool": { + version: 3, + runs: [ + { + ...task("task_a", "turn-1"), + task: { ...task("task_a", "turn-1").task, cohortId: cohortId }, + }, + ], + }, + }), + ).toThrow(/Corrupt workflow tool run registry/u); + }); + + it("validates retained outcomes when they are consumed", () => { + const base = { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { dispatchContext, metadata, taskId: "task_a" }, + }; + const outcome = { + lastOutput: { data: "done", type: "result" as const }, + status: "completed" as const, + }; + + const session = registerWorkflowToolRun(createSession(), { + ...base, + task: { ...base.task, outcome }, + }); + expect(findBackgroundWorkflowToolRun(session.state, "task_a")?.task.outcome).toEqual(outcome); + for (const invalidOutcome of [ + { status: "working" }, + { status: "completed" }, + { + lastOutput: { data: "wrong", type: "result" }, + status: "failed", + }, + { + lastOutput: { data: "wrong", type: "result" }, + status: "cancelled", + }, + { + inputRequests: [{ requestId: "stale" }], + lastOutput: { data: "done", type: "result" }, + status: "completed", + }, + ]) { + const [entry] = getBackgroundWorkflowToolRuns({ + "eve.workflowTool": { + version: 3, + runs: [{ ...base, task: { ...base.task, outcome: invalidOutcome } }], + }, + }); + expect(entry?.address).toEqual(base.address); + assert(entry !== undefined); + expect(() => readWorkflowTaskView(entry.task)).toThrow("Corrupt workflow task result"); + expect(() => + registerWorkflowToolRun(createSession(), { + ...base, + task: { ...base.task, outcome: invalidOutcome }, + }), + ).toThrow("Corrupt workflow task result"); + } + }); + + it.each(["completed", "failed", "cancelled"] as const)( + "keeps the parent's first %s outcome across duplicates and competing deliveries", + (status) => { + const session = registerWorkflowToolRun(createSession(), task("task_a", "turn-1")); + const usage = { inputTokens: 3, outputTokens: 5, cacheReadTokens: 0, cacheWriteTokens: 0 }; + const first = { ...terminal("task_a", status), usage }; + const state = recordWorkflowTaskView(session.state, first); + for (const late of ["completed", "failed", "cancelled"] as const) { + expect(recordWorkflowTaskView(state, terminal("task_a", late))).toBe(state); + } + const entry = findBackgroundWorkflowToolRun(state, "task_a"); + assert(entry !== undefined); + expect(entry.task.outcome).toEqual( + status === "cancelled" + ? { status, usage } + : { status, usage, lastOutput: first.lastOutput }, + ); + expect(readWorkflowTaskView(entry.task)).toEqual(first); + }, + ); + + it("rejects an incoming result with metadata belonging to a different task", () => { + const session = registerWorkflowToolRun(createSession(), task("task_a", "turn-1")); + expect(() => + recordWorkflowTaskView(session.state, { + ...terminal("task_a", "cancelled"), + metadata: { kind: "tool", name: "other" }, + }), + ).toThrow("Task view metadata does not match"); + }); + + it("throws on a corrupt index instead of treating it as absent", () => { + expect(() => + getBackgroundWorkflowToolRuns({ + "eve.workflowTool": { version: 3, runs: [{ taskId: 42 }] }, + }), + ).toThrow("Corrupt workflow tool run registry"); + }); + + it("rejects missing creator context", () => { + const entry = task("task_a", "turn-1"); + expect(() => + getBackgroundWorkflowToolRuns({ + "eve.workflowTool": { + version: 3, + runs: [{ ...entry, task: { ...entry.task, dispatchContext: undefined } }], + }, + }), + ).toThrow("Corrupt workflow tool run registry"); + }); + + it("rejects reassigning a task id to another originating turn", () => { + const session = registerWorkflowToolRun(createSession(), task("task_a", "turn-1")); + expect(() => registerWorkflowToolRun(session, task("task_a", "turn-2"))).toThrow( + "Task ids must be unique", + ); + }); + + it("rejects unrecognized task dispatch context fields", () => { + expect(() => + getBackgroundWorkflowToolRuns({ + "eve.workflowTool": { + version: 3, + runs: [ + { + callId: "task_a", + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "run-1", hookToken: "task:token-1" }, + task: { + dispatchContext: { + auth: { current: null, initiator: null }, + unexpected: "receiver-context", + }, + metadata, + taskId: "task_a", + }, + }, + ], + }, + }), + ).toThrow("Corrupt workflow tool run registry"); + }); + + it("rejects an unsupported registry version", () => { + expect(() => + getBackgroundWorkflowToolRuns({ + "eve.workflowTool": { version: 99, runs: [] }, + }), + ).toThrow("Corrupt workflow tool run registry"); + }); +}); + +describe("deriveTaskId", () => { + it("is deterministic for the same originating call and distinct otherwise", () => { + const input = { callId: "call-1", parentSessionId: "session-1", parentTurnId: "turn-1" }; + + expect(deriveTaskId(input)).toBe(deriveTaskId(input)); + expect(deriveTaskId(input)).toMatch(/^task_[0-9a-f]{24}$/); + expect(deriveTaskId({ ...input, callId: "call-2" })).not.toBe(deriveTaskId(input)); + }); +}); diff --git a/packages/eve/src/harness/workflow-tool-runs.test.ts b/packages/eve/src/harness/workflow-tool-runs.test.ts index 575ba9767..4c34f6d88 100644 --- a/packages/eve/src/harness/workflow-tool-runs.test.ts +++ b/packages/eve/src/harness/workflow-tool-runs.test.ts @@ -1,20 +1,20 @@ +import { + registerWorkflowToolRun, + removeBlockingWorkflowToolRuns, + findBlockingWorkflowToolRun, + getBlockingWorkflowToolRuns, + isInboxToolResultFromRecordedWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import { describe, expect, it } from "vitest"; -import { - clearWorkflowToolRuns, - findWorkflowToolRun, - getWorkflowToolRuns, - isInboxToolResultFromRecordedWorkflowToolRun, - recordWorkflowToolRun, - removeWorkflowToolRun, -} from "#harness/workflow-tool-runs.js"; import type { HarnessSession } from "#harness/types.js"; const RECORD = { callId: "call_1", - hookToken: "eve:workflow-tool-run:abc", - runId: "wrun_1", toolName: "deploy", + lifetime: "turn" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "wrun_1", hookToken: "eve:workflow-tool-run:abc" }, }; function session(state?: HarnessSession["state"]): HarnessSession { @@ -22,28 +22,51 @@ function session(state?: HarnessSession["state"]): HarnessSession { } describe("workflow tool run records", () => { + it("rejects the previous registry format before using its run records", () => { + expect(() => + getBlockingWorkflowToolRuns({ + "eve.workflowTool": { version: 1, runs: [RECORD] }, + }), + ).toThrow("Corrupt workflow tool run registry"); + }); + it("records, finds, and removes runs by call id", () => { - const recorded = recordWorkflowToolRun(session({ other: true }), RECORD); - expect(getWorkflowToolRuns(recorded.state)).toEqual([RECORD]); - expect(findWorkflowToolRun(recorded.state, "call_1")).toEqual(RECORD); + const recorded = registerWorkflowToolRun(session({ other: true }), RECORD); + expect(getBlockingWorkflowToolRuns(recorded.state)).toEqual([RECORD]); + expect(findBlockingWorkflowToolRun(recorded.state, "call_1", "turn-1")).toEqual(RECORD); - const replaced = recordWorkflowToolRun(recorded, { ...RECORD, runId: "wrun_2" }); - expect(getWorkflowToolRuns(replaced.state)).toEqual([{ ...RECORD, runId: "wrun_2" }]); + const replaced = registerWorkflowToolRun(recorded, { + ...RECORD, + address: { ...RECORD.address, runId: "wrun_2" }, + }); + expect(getBlockingWorkflowToolRuns(replaced.state)).toEqual([ + { ...RECORD, address: { ...RECORD.address, runId: "wrun_2" } }, + ]); - const removed = removeWorkflowToolRun(replaced, "call_1"); - expect(getWorkflowToolRuns(removed.state)).toEqual([]); + const removed = removeBlockingWorkflowToolRuns(replaced, "turn-1", "call_1"); + expect(getBlockingWorkflowToolRuns(removed.state)).toEqual([]); expect(removed.state).toEqual({ other: true }); - expect(removeWorkflowToolRun(removed, "call_1")).toBe(removed); + expect(removeBlockingWorkflowToolRuns(removed, "turn-1", "call_1")).toBe(removed); }); it("drops the state map entirely when nothing else is recorded", () => { - const recorded = recordWorkflowToolRun(session(), RECORD); - expect(clearWorkflowToolRuns(recorded).state).toBeUndefined(); - expect(clearWorkflowToolRuns(session())).toEqual(session()); + const recorded = registerWorkflowToolRun(session(), RECORD); + expect(removeBlockingWorkflowToolRuns(recorded, "turn-1").state).toBeUndefined(); + expect(removeBlockingWorkflowToolRuns(session(), "turn-1")).toEqual(session()); }); it("binds inbox tool results to the recorded run by call id and tool name", () => { - const state = recordWorkflowToolRun(session(), RECORD).state; + const state = registerWorkflowToolRun( + session({ + "eve.harness.emission": { + turnId: "turn-1", + sequence: 0, + stepIndex: 0, + sessionStarted: true, + }, + }), + RECORD, + ).state; const result = { callId: "call_1", kind: "tool-result" as const, @@ -61,9 +84,22 @@ describe("workflow tool run records", () => { expect(isInboxToolResultFromRecordedWorkflowToolRun(undefined, result)).toBe(false); }); - it("ignores malformed state", () => { - expect(getWorkflowToolRuns({ "eve.runtime.workflowToolRuns": { not: "an array" } })).toEqual( - [], - ); + it("finds a paused call after authorization has ended the visible turn", () => { + const recorded = registerWorkflowToolRun(session(), RECORD); + expect(findBlockingWorkflowToolRun(recorded.state, RECORD.callId)).toEqual(RECORD); + const overlapping = registerWorkflowToolRun(recorded, { + ...RECORD, + origin: { turnId: "another-turn", stepIndex: 0 }, + }); + expect(findBlockingWorkflowToolRun(overlapping.state, RECORD.callId)).toBeUndefined(); + expect(findBlockingWorkflowToolRun(overlapping.state, RECORD.callId, "turn-1")).toEqual(RECORD); + }); + + it("rejects malformed state", () => { + expect(() => + getBlockingWorkflowToolRuns({ + "eve.workflowTool": { version: 3, runs: { not: "an array" } }, + }), + ).toThrow("Corrupt workflow tool run registry"); }); }); diff --git a/packages/eve/src/harness/workflow-tool-runs.ts b/packages/eve/src/harness/workflow-tool-runs.ts index d6ec3a4f7..8e883aec8 100644 --- a/packages/eve/src/harness/workflow-tool-runs.ts +++ b/packages/eve/src/harness/workflow-tool-runs.ts @@ -1,86 +1,450 @@ -import type { HarnessSession, SessionStateMap } from "#harness/types.js"; import type { RuntimeToolResultActionResult } from "#shared/action-types.js"; +import { isNonEmptyString, isObject } from "#shared/guards.js"; +import type { SessionStateMap } from "#harness/types.js"; +import { parseActivityWorkIdentityV1, type ActivityWorkIdentityV1 } from "#protocol/activity.js"; +import type { SessionAuthContext } from "#channel/types.js"; +import { + sameTaskMetadata, + type TaskMetadata, + type TaskOutput, + type TaskUsage, + type TaskView, +} from "#tasks/types.js"; +import type { DurableDynamicSubagentSelection, SessionAuth } from "#context/keys.js"; -const WORKFLOW_TOOL_RUNS_STATE_KEY = "eve.runtime.workflowToolRuns"; +// Version 3 replaces the task-only index with the shared workflow tool run registry. +export const WORKFLOW_TOOL_RUNS_STATE_KEY = "eve.workflowTool"; +const WORKFLOW_TOOL_RUNS_VERSION = 3; -/** A workflow tool run the active turn waits on. Background runs live in the task index. */ -export interface WorkflowToolRunRecord { - readonly callId: string; - readonly hookToken: string; - readonly runId: string; - readonly toolName: string; - readonly resultKind?: "subagent" | "tool"; +export interface WorkflowTaskPayload { + readonly taskId: string; + readonly metadata: TaskMetadata; + readonly dispatchContext: TaskAgentDispatchContext; + readonly activityWorkIdentity?: ActivityWorkIdentityV1; + readonly cohortId?: string; + /** Parent-owned outcome. Read through readWorkflowTaskView before consuming it. */ + readonly outcome?: unknown; +} + +/** Settled task data; identity and metadata belong to the owning task. */ +type TaskOutcome = { + readonly usage?: TaskUsage; +} & ( + | { readonly status: "completed"; readonly lastOutput: Extract } + | { readonly status: "failed"; readonly lastOutput: Extract } + | { readonly status: "cancelled"; readonly lastOutput?: never } +); + +interface WorkflowToolRunBase { + readonly callId: string; + readonly toolName: string; + + readonly origin: { readonly turnId: string; readonly stepIndex: number }; + readonly address: { readonly runId: string; readonly hookToken: string }; +} +export type BlockingWorkflowToolRun = WorkflowToolRunBase & { readonly lifetime: "turn" }; +export type BackgroundWorkflowToolRun = WorkflowToolRunBase & { + readonly lifetime: "session"; + readonly task: WorkflowTaskPayload; +}; +export type WorkflowToolRun = BlockingWorkflowToolRun | BackgroundWorkflowToolRun; + +export interface TaskAgentDispatchContext { + readonly auth: SessionAuth; + readonly sessionDynamicSubagentSelections?: Readonly< + Record + >; + readonly turnDynamicSubagentSelections?: Readonly< + Record + >; +} + +interface WorkflowToolRunRegistry { + readonly version: typeof WORKFLOW_TOOL_RUNS_VERSION; + readonly runs: readonly WorkflowToolRun[]; + readonly [key: string]: unknown; +} + +// These readers run inside the workflow driver: importing a schema runtime here also +// embeds it and its source map in every deployed workflow function. +function isTaskMetadata(value: unknown): value is TaskMetadata { + return isObject(value) && isNonEmptyString(value.kind) && isNonEmptyString(value.name); +} + +/** Workflow checkpoints can carry records created in another VM realm. */ +function isRecord(value: unknown): value is Record { + if (!isObject(value)) return false; + const constructor = value.constructor; + return ( + typeof constructor !== "function" || + (isObject(constructor.prototype) && Object.hasOwn(constructor.prototype, "isPrototypeOf")) + ); +} + +function isSessionAuthContext(value: unknown): value is SessionAuthContext | null { + return ( + value === null || + (isObject(value) && + Object.keys(value).every((key) => + [ + "attributes", + "authenticator", + "issuer", + "principalId", + "principalType", + "subject", + ].includes(key), + ) && + isRecord(value.attributes) && + Object.values(value.attributes).every( + (attribute) => + typeof attribute === "string" || + (Array.isArray(attribute) && + Array.from(attribute).every((item) => typeof item === "string")), + ) && + typeof value.authenticator === "string" && + typeof value.principalId === "string" && + typeof value.principalType === "string" && + (value.issuer === undefined || typeof value.issuer === "string") && + (value.subject === undefined || typeof value.subject === "string")) + ); +} + +function isTaskAgentDispatchContext(value: unknown): value is TaskAgentDispatchContext { + return ( + isObject(value) && + Object.keys(value).every((key) => + ["auth", "sessionDynamicSubagentSelections", "turnDynamicSubagentSelections"].includes(key), + ) && + isObject(value.auth) && + Object.keys(value.auth).every((key) => key === "current" || key === "initiator") && + isSessionAuthContext(value.auth.current) && + isSessionAuthContext(value.auth.initiator) && + (value.sessionDynamicSubagentSelections === undefined || + isRecord(value.sessionDynamicSubagentSelections)) && + (value.turnDynamicSubagentSelections === undefined || + isRecord(value.turnDynamicSubagentSelections)) + ); +} + +function isWorkflowToolRun(value: unknown): value is WorkflowToolRun { + if ( + !isObject(value) || + !isNonEmptyString(value.callId) || + !isNonEmptyString(value.toolName) || + !isObject(value.origin) || + !isNonEmptyString(value.origin.turnId) || + typeof value.origin.stepIndex !== "number" || + !Number.isSafeInteger(value.origin.stepIndex) || + value.origin.stepIndex < 0 || + !isObject(value.address) || + !isNonEmptyString(value.address.runId) || + !isNonEmptyString(value.address.hookToken) + ) + return false; + if (value.lifetime === "turn") return value.task === undefined; + if (value.lifetime !== "session" || !isObject(value.task)) return false; + const task = value.task; + return ( + isNonEmptyString(task.taskId) && + isTaskMetadata(task.metadata) && + isTaskAgentDispatchContext(task.dispatchContext) && + (task.cohortId === undefined || isNonEmptyString(task.cohortId)) && + (task.activityWorkIdentity === undefined || + parseActivityWorkIdentityV1(task.activityWorkIdentity) !== undefined) + ); +} + +function parseRegistry(value: unknown): WorkflowToolRunRegistry { + if ( + !isObject(value) || + value.version !== WORKFLOW_TOOL_RUNS_VERSION || + !Array.isArray(value.runs) || + !Array.from(value.runs).every(isWorkflowToolRun) + ) { + throw new Error("Corrupt workflow tool run registry: invalid version or run."); + } + const identities = new Set(); + const tasks = new Set(); + for (const entry of value.runs) { + const identity = JSON.stringify([entry.origin.turnId, entry.callId]); + if (identities.has(identity)) + throw new Error("Corrupt workflow tool run registry: Run identities must be unique."); + identities.add(identity); + if (entry.lifetime !== "session") continue; + if (tasks.has(entry.task.taskId)) + throw new Error("Corrupt workflow tool run registry: Task ids must be unique."); + tasks.add(entry.task.taskId); + } + return { + ...value, + version: WORKFLOW_TOOL_RUNS_VERSION, + runs: value.runs.map(copyWorkflowToolRun), + }; +} + +function copySessionAuthContext(value: SessionAuthContext | null): SessionAuthContext | null { + if (value === null) return null; + return { + ...value, + attributes: Object.fromEntries( + Object.entries(value.attributes).map(([key, attribute]) => [ + key, + typeof attribute === "string" ? attribute : Object.freeze([...attribute]), + ]), + ), + }; +} + +function copyWorkflowToolRun(entry: WorkflowToolRun): WorkflowToolRun { + const base = { ...entry, origin: { ...entry.origin }, address: { ...entry.address } }; + if (entry.lifetime === "turn") return base; + const dispatch = entry.task.dispatchContext; + const dispatchContext = { + ...dispatch, + auth: { + current: copySessionAuthContext(dispatch.auth.current), + initiator: copySessionAuthContext(dispatch.auth.initiator), + }, + }; + if (dispatch.sessionDynamicSubagentSelections !== undefined) + dispatchContext.sessionDynamicSubagentSelections = { + ...dispatch.sessionDynamicSubagentSelections, + }; + if (dispatch.turnDynamicSubagentSelections !== undefined) + dispatchContext.turnDynamicSubagentSelections = { ...dispatch.turnDynamicSubagentSelections }; + return { + ...base, + lifetime: "session", + task: { ...entry.task, metadata: { ...entry.task.metadata }, dispatchContext }, + }; +} + +function parseTaskOutcome(value: unknown): TaskOutcome | undefined { + if (!isObject(value) || value.inputRequests !== undefined) return undefined; + const usage = value.usage; + if ( + usage !== undefined && + (!isObject(usage) || + ![ + usage.cacheReadTokens, + usage.cacheWriteTokens, + usage.inputTokens, + usage.outputTokens, + ...(usage.costUsd === undefined ? [] : [usage.costUsd]), + ].every((count) => typeof count === "number" && Number.isFinite(count) && count >= 0)) + ) + return undefined; + if (value.status === "cancelled") { + if (value.lastOutput !== undefined) return undefined; + } else { + if (value.status !== "completed" && value.status !== "failed") return undefined; + const outputType = value.status === "completed" ? "result" : "error"; + if (!isObject(value.lastOutput) || value.lastOutput.type !== outputType) return undefined; + } + // Output data and additive fields are opaque; only the known lifecycle fields are decoded. + const outcome = { ...value }; + if (usage !== undefined) outcome.usage = { ...usage }; + if (isObject(value.lastOutput)) outcome.lastOutput = { ...value.lastOutput }; + return outcome as TaskOutcome; +} + +/** Decode retained output only when it is consumed, independently of ownership reads. */ +export function readWorkflowTaskView(task: WorkflowTaskPayload): TaskView | undefined { + if (task.outcome === undefined) return undefined; + const outcome = parseTaskOutcome(task.outcome); + if (outcome === undefined) + throw new Error(`Corrupt workflow task result "${task.taskId}": invalid outcome.`); + return { ...outcome, taskId: task.taskId, metadata: { ...task.metadata } }; +} + +function readRegistry(state: SessionStateMap | undefined): WorkflowToolRunRegistry { + if (state?.["eve.tasks"] !== undefined || state?.["eve.runtime.workflowToolRuns"] !== undefined) { + throw new Error( + "Unsupported workflow tool run state: start a new session or import its conversation.", + ); + } + const raw = state?.[WORKFLOW_TOOL_RUNS_STATE_KEY]; + if (raw === undefined) return { version: WORKFLOW_TOOL_RUNS_VERSION, runs: [] }; + return parseRegistry(raw); } -// Schema-free: this is bundled into the workflow driver. export function getWorkflowToolRuns( state: SessionStateMap | undefined, -): readonly WorkflowToolRunRecord[] { - const raw = state?.[WORKFLOW_TOOL_RUNS_STATE_KEY]; - return Array.isArray(raw) ? (raw as readonly WorkflowToolRunRecord[]) : []; +): readonly WorkflowToolRun[] { + return readRegistry(state).runs; } -export function findWorkflowToolRun( +export function getBackgroundWorkflowToolRuns( + state: SessionStateMap | undefined, +): readonly BackgroundWorkflowToolRun[] { + return getWorkflowToolRuns(state).filter( + (entry): entry is BackgroundWorkflowToolRun => entry.lifetime === "session", + ); +} + +export function getBlockingWorkflowToolRuns( + state: SessionStateMap | undefined, + turnId?: string, +): readonly BlockingWorkflowToolRun[] { + return getWorkflowToolRuns(state).filter( + (entry): entry is BlockingWorkflowToolRun => + entry.lifetime === "turn" && (turnId === undefined || entry.origin.turnId === turnId), + ); +} + +export function findBackgroundWorkflowToolRun( + state: SessionStateMap | undefined, + taskId: string, +): BackgroundWorkflowToolRun | undefined { + return getBackgroundWorkflowToolRuns(state).find((entry) => entry.task.taskId === taskId); +} + +function writeRegistry( + state: SessionStateMap | undefined, + registry: WorkflowToolRunRegistry, +): SessionStateMap | undefined { + if (registry.runs.length === 0) { + const next = { ...state }; + delete next[WORKFLOW_TOOL_RUNS_STATE_KEY]; + return Object.keys(next).length === 0 ? undefined : next; + } + return { + ...state, + [WORKFLOW_TOOL_RUNS_STATE_KEY]: parseRegistry(registry), + }; +} + +/** Registration is idempotent by originating turn and call; retained task facts survive replay. */ +export function registerWorkflowToolRun( + session: T, + entry: WorkflowToolRun, +): T { + const registry = readRegistry(session.state); + const runs = [...registry.runs]; + const index = runs.findIndex( + (candidate) => + candidate.origin.turnId === entry.origin.turnId && candidate.callId === entry.callId, + ); + const previous = runs[index]; + if ( + previous !== undefined && + (previous.lifetime !== entry.lifetime || previous.toolName !== entry.toolName) + ) { + throw new Error("Replayed invocation changed its ownership or tool identity."); + } + if (entry.lifetime === "session") { + if (previous?.lifetime === "session") { + if (entry.task.taskId !== previous.task.taskId) + throw new Error("Replayed invocation changed its task identity."); + entry = { + ...entry, + task: { + ...previous.task, + ...entry.task, + metadata: { ...previous.task.metadata, ...entry.task.metadata }, + activityWorkIdentity: + entry.task.activityWorkIdentity === undefined + ? previous.task.activityWorkIdentity + : { ...previous.task.activityWorkIdentity, ...entry.task.activityWorkIdentity }, + cohortId: previous.task.cohortId, + dispatchContext: previous.task.dispatchContext, + outcome: previous.task.outcome ?? entry.task.outcome, + }, + }; + } else { + const pending = runs.find( + (candidate): candidate is BackgroundWorkflowToolRun => + candidate.lifetime === "session" && candidate.task.outcome === undefined, + ); + entry = { + ...entry, + task: { + ...entry.task, + cohortId: + pending === undefined ? undefined : (pending.task.cohortId ?? pending.task.taskId), + }, + }; + } + } + if (entry.lifetime === "session") readWorkflowTaskView(entry.task); + if (previous === undefined) runs.push(entry); + else + runs[index] = { + ...previous, + ...entry, + origin: previous.origin, + address: { ...previous.address, ...entry.address }, + }; + return { ...session, state: writeRegistry(session.state, { ...registry, runs }) }; +} + +/** Task payloads remain available for the session lifetime, including after report delivery. */ +export function recordWorkflowTaskView( + state: SessionStateMap | undefined, + view: TaskView, +): SessionStateMap | undefined { + const { taskId, metadata, ...result } = view; + const outcome = parseTaskOutcome(result); + if (!isNonEmptyString(taskId) || !isTaskMetadata(metadata) || outcome === undefined) + throw new Error("Invalid terminal workflow task view."); + const registry = readRegistry(state); + const runs = [...registry.runs]; + const index = runs.findIndex( + (entry) => entry.lifetime === "session" && entry.task.taskId === taskId, + ); + const entry = runs[index]; + if (entry === undefined || entry.lifetime !== "session") return state; + if (!sameTaskMetadata(entry.task.metadata, metadata)) + throw new Error(`Task view metadata does not match invocation "${view.taskId}".`); + const previous = readWorkflowTaskView(entry.task); + // Parent delivery order decides settlement. Replays and late outcomes cannot replace it. + if (previous !== undefined) return state; + runs[index] = { + ...entry, + task: { + ...entry.task, + outcome, + }, + }; + return writeRegistry(state, { ...registry, runs }); +} + +/** Removes only this turn's waiting calls. Session-owned task payloads are never pruned here. */ +export function removeBlockingWorkflowToolRuns( + session: T, + turnId: string, + callId?: string, +): T { + const registry = readRegistry(session.state); + const entries = registry.runs; + const remaining = entries.filter( + (entry) => + entry.lifetime !== "turn" || + entry.origin.turnId !== turnId || + (callId !== undefined && entry.callId !== callId), + ); + return remaining.length === entries.length + ? session + : { ...session, state: writeRegistry(session.state, { ...registry, runs: remaining }) }; +} + +/** Results without an originating turn may bind only when exactly one recorded turn owns the call. */ +export function findBlockingWorkflowToolRun( state: SessionStateMap | undefined, callId: string, -): WorkflowToolRunRecord | undefined { - return getWorkflowToolRuns(state).find((record) => record.callId === callId); -} - -export function recordWorkflowToolRun( - session: T, - record: WorkflowToolRunRecord, -): T { - const others = getWorkflowToolRuns(session.state).filter( - (entry) => entry.callId !== record.callId, + turnId?: string, +): BlockingWorkflowToolRun | undefined { + const candidates = getBlockingWorkflowToolRuns(state, turnId).filter( + (entry) => entry.callId === callId, ); - return writeWorkflowToolRuns(session, [...others, record]); + return candidates.length === 1 ? candidates[0] : undefined; } - -export function removeWorkflowToolRun( - session: T, - callId: string, -): T { - const records = getWorkflowToolRuns(session.state); - const remaining = records.filter((entry) => entry.callId !== callId); - return remaining.length === records.length ? session : writeWorkflowToolRuns(session, remaining); -} - -export function clearWorkflowToolRuns(session: HarnessSession): HarnessSession { - return getWorkflowToolRuns(session.state).length === 0 - ? session - : writeWorkflowToolRuns(session, []); -} - /** The turn inbox is shared; a result settles a call only if the turn recorded that run. */ export function isInboxToolResultFromRecordedWorkflowToolRun( state: SessionStateMap | undefined, result: RuntimeToolResultActionResult, ): boolean { - const record = findWorkflowToolRun(state, result.callId); - return ( - record !== undefined && record.resultKind !== "subagent" && record.toolName === result.toolName - ); -} - -/** A child result reported through a shared subagent execute run. */ -export function isInboxSubagentResultFromRecordedWorkflowToolRun( - state: SessionStateMap | undefined, - result: { readonly callId: string; readonly subagentName: string }, -): boolean { - const record = findWorkflowToolRun(state, result.callId); - return record?.resultKind === "subagent" && record.toolName === result.subagentName; -} - -function writeWorkflowToolRuns( - session: T, - records: readonly WorkflowToolRunRecord[], -): T { - const state = { ...session.state }; - if (records.length === 0) { - delete state[WORKFLOW_TOOL_RUNS_STATE_KEY]; - } else { - state[WORKFLOW_TOOL_RUNS_STATE_KEY] = records; - } - return { ...session, state: Object.keys(state).length > 0 ? state : undefined }; + const record = findBlockingWorkflowToolRun(state, result.callId); + return record !== undefined && record.toolName === result.toolName; } diff --git a/packages/eve/src/internal/authored-definition/schema-backed.test.ts b/packages/eve/src/internal/authored-definition/schema-backed.test.ts index ef25a1a3b..fd01ec3da 100644 --- a/packages/eve/src/internal/authored-definition/schema-backed.test.ts +++ b/packages/eve/src/internal/authored-definition/schema-backed.test.ts @@ -45,24 +45,6 @@ describe("normalizeToolDefinition", () => { expect(entry.definition.availableInSubagents).toBe(false); }); - it("preserves the background execution discriminator", () => { - const tool = defineTool({ - description: "Starts an export.", - execution: "background", - inputSchema: z.object({ exportId: z.string() }), - async *execute(input) { - yield { exportId: input.exportId }; - return { exportId: input.exportId }; - }, - }); - - const entry = normalizeToolDefinition(tool, FAILURE_MESSAGE); - - expect(entry.kind).toBe("tool"); - if (entry.kind !== "tool") throw new Error("expected tool kind"); - expect(entry.definition.execution).toBe("background"); - }); - it("normalizes a tool with a Zod 3 input schema", () => { const tool = defineTool({ description: "Gets weather for a city.", diff --git a/packages/eve/src/internal/testing/task-cancel-notification-workflow.ts b/packages/eve/src/internal/testing/task-cancel-notification-workflow.ts index 4dd1bdde3..0a9e2277b 100644 --- a/packages/eve/src/internal/testing/task-cancel-notification-workflow.ts +++ b/packages/eve/src/internal/testing/task-cancel-notification-workflow.ts @@ -2,15 +2,14 @@ import { createHook, getWorkflowMetadata, sleep } from "#compiled/@workflow/core import { createSessionInbox } from "#execution/session-inbox/inbox.js"; import { sessionCommandHookToken } from "#execution/session-inbox/address.js"; -import { appendTaskViewStep } from "#execution/tasks/child/steps.js"; import { cancelOwnedTask } from "#execution/tasks/parent/dispatch.js"; import { waitForCommandHookOwner } from "#execution/workflow-runtime.js"; import { getRun, start } from "#internal/workflow/runtime.js"; import type { HarnessSession } from "#harness/types.js"; -import type { SessionTaskIndexEntry } from "#tasks/session-index.js"; +import type { BackgroundWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import type { TaskCommandHookPayload } from "#tasks/types.js"; -/** Models a task whose view commits before its executor finishes unwinding. */ +/** Models a task whose executor cannot finish cooperative cleanup. */ export async function slowCancelledTaskWorkflow(input: { readonly taskId: string; readonly taskInboxToken: string; @@ -18,19 +17,16 @@ export async function slowCancelledTaskWorkflow(input: { "use workflow"; using commands = createHook({ token: input.taskInboxToken }); - const metadata = { kind: "tool", name: "slow-cancel" } as const; - await appendTaskViewStep({ view: { metadata, status: "working", taskId: input.taskId } }); const delivery = await commands; if (delivery.kind !== "task-command" || delivery.command.kind !== "cancel") { throw new Error("Expected the task cancellation command."); } - await appendTaskViewStep({ view: { metadata, status: "cancelled", taskId: input.taskId } }); await sleep("1h"); } export async function startSlowCancelledTaskStep(input: { readonly sessionId: string; -}): Promise { +}): Promise { "use step"; const taskId = `${input.sessionId}-task`; @@ -38,17 +34,21 @@ export async function startSlowCancelledTaskStep(input: { const run = await start(slowCancelledTaskWorkflow, [{ taskId, taskInboxToken }]); await waitForCommandHookOwner(taskInboxToken); return { - createdByTurnId: "turn_0", - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "tool", name: "slow-cancel" }, - taskId, - taskInboxToken, - taskRunId: run.runId, + callId: taskId, + toolName: { kind: "tool", name: "slow-cancel" }.name, + lifetime: "session" as const, + origin: { turnId: "turn_0", stepIndex: 0 }, + address: { runId: run.runId, hookToken: taskInboxToken }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "tool", name: "slow-cancel" }, + taskId, + }, }; } export async function cancelSlowTaskFromParentStep(input: { - readonly entry: SessionTaskIndexEntry; + readonly entry: BackgroundWorkflowToolRun; readonly sessionId: string; }) { "use step"; @@ -57,7 +57,7 @@ export async function cancelSlowTaskFromParentStep(input: { entry: input.entry, session: { sessionId: input.sessionId } as HarnessSession, }); - return { view, taskRunStatus: await getRun(input.entry.taskRunId).status }; + return { view, taskRunStatus: await getRun(input.entry.address.runId).status }; } export async function taskCancelNotificationWorkflow() { diff --git a/packages/eve/src/internal/testing/workflow-tool-fixtures.ts b/packages/eve/src/internal/testing/workflow-tool-fixtures.ts index 7af04f022..86a37d348 100644 --- a/packages/eve/src/internal/testing/workflow-tool-fixtures.ts +++ b/packages/eve/src/internal/testing/workflow-tool-fixtures.ts @@ -13,7 +13,6 @@ import { } from "#compiled/@workflow/core/index.js"; import type { WorkflowToolContext } from "#tools/workflow-definition.js"; -import type { TaskExec, TaskMessage } from "#tools/task.js"; import { ConnectionAuthorizationFailedError, ConnectionAuthorizationRequiredError, @@ -183,13 +182,12 @@ export async function* reportingDeployWorkflow( export async function* backgroundDeployWorkflow( input: DeployInput, _ctx: WorkflowToolContext, - task: TaskExec, -): AsyncGenerator { +): AsyncGenerator { "use workflow"; const plan = await planDeployStep(input.service); yield `planned ${input.service}`; - yield task.postMessage(`Review ${plan}`); + yield `review ${plan}`; return { plan }; } diff --git a/packages/eve/src/internal/workflow-bundle/builder.scenario.test.ts b/packages/eve/src/internal/workflow-bundle/builder.scenario.test.ts index 3ed261b6f..2e6b93418 100644 --- a/packages/eve/src/internal/workflow-bundle/builder.scenario.test.ts +++ b/packages/eve/src/internal/workflow-bundle/builder.scenario.test.ts @@ -662,6 +662,11 @@ describe("WorkflowBundleBuilder", () => { it.each([ ["sleep tool", "src/execution/tools/sleep-workflow.ts", "executeSleepTool"], ["session owner", "src/execution/session/entry.ts", "nextTurnDelivery"], + [ + "workflow tool owner", + "src/execution/tools/workflow/workflow.ts", + "createBackgroundWorkflowOwner", + ], ])("keeps the %s schemas out of the workflow driver", async (_name, sourcePath, marker) => { const tempRoot = await mkdtemp(join(tmpdir(), "eve-workflow-bundle-no-schemas-")); const outDir = join(tempRoot, "workflow-build"); diff --git a/packages/eve/src/protocol/message.ts b/packages/eve/src/protocol/message.ts index 2d8b297de..776cfa2b3 100644 --- a/packages/eve/src/protocol/message.ts +++ b/packages/eve/src/protocol/message.ts @@ -412,14 +412,14 @@ export interface SubagentChildEventStreamEvent { } /** - * Stream event emitted when an inline subagent completes. + * Stream event emitted after the parent accepts a successful subagent invocation result. */ export interface SubagentCompletedStreamEvent { data: { /** - * Present when the originating call completed with a background-task - * receipt while the child itself kept running. Consumers must not treat - * this as the child's terminal boundary; the child stream owns that. + * Historical admission marker retained for reading existing streams. + * A marked event is a working receipt, not a completed invocation. + * New receipts are published only as action.result tool outputs. */ backgroundTask?: { taskId: string; diff --git a/packages/eve/src/public/definitions/exact.test.ts b/packages/eve/src/public/definitions/exact.test.ts index 08d515b0d..02cdcd497 100644 --- a/packages/eve/src/public/definitions/exact.test.ts +++ b/packages/eve/src/public/definitions/exact.test.ts @@ -25,12 +25,8 @@ import { import { defineSandbox } from "#public/definitions/sandbox.js"; import { defineSchedule } from "#public/definitions/schedule.js"; import { defineSkill } from "#public/definitions/skill.js"; -import { - defineTool, - type TaskExec, - type TaskReceipt, - type ToolDefinition, -} from "#public/tools/index.js"; +import { defineTool, type TaskReceipt, type ToolDefinition } from "#public/tools/index.js"; +import { defineWorkflowTool } from "#public/tools/index.js"; describe("definition helper exact inputs", () => { it("preserves literal inference for valid definitions", () => { @@ -111,15 +107,12 @@ describe("definition helper exact inputs", () => { >(); }); - it("types background tools in terms of the durable task capability", () => { - const backgroundTool = defineTool({ + it("types background workflow tools in terms of their receipt", () => { + const backgroundTool = defineWorkflowTool({ description: "Start a durable export.", execution: "background", inputSchema: z.object({ jobId: z.string() }), - async *execute(input, _ctx, task) { - expectTypeOf(task).toEqualTypeOf(); - expectTypeOf(task.taskId).toEqualTypeOf(); - expectTypeOf(task).not.toHaveProperty("delegated"); + async *execute(input) { yield { jobId: input.jobId }; return { jobId: input.jobId }; }, @@ -132,6 +125,16 @@ describe("definition helper exact inputs", () => { expect(backgroundTool.execution).toBe("background"); }); + it("rejects background execution on ordinary tools", () => { + const definition = { + description: "Start a durable export.", + execution: "background", + inputSchema: z.object({ jobId: z.string() }), + execute: async () => null, + }; + expect(() => defineTool(definition)).toThrow("Use defineWorkflowTool for background work"); + }); + it("infers tool input from Zod 3 schemas", () => { const tool = defineTool({ description: "Fetch current weather for a city.", diff --git a/packages/eve/src/public/tools/index.ts b/packages/eve/src/public/tools/index.ts index 1da7215e6..22686ab2d 100644 --- a/packages/eve/src/public/tools/index.ts +++ b/packages/eve/src/public/tools/index.ts @@ -3,13 +3,10 @@ */ export { - type BackgroundToolDefinition, type DisabledToolSentinel, defineTool, disableTool, isDisabledToolSentinel, - type TaskExec, - type TaskReceipt, type ToolLabelDefinition, type ToolAuthOptions, type ToolAuthProvider, @@ -40,6 +37,7 @@ export { export { defineWorkflowTool, type WorkflowStepToolContext, + type TaskReceipt, type WorkflowToolContext, type WorkflowToolDefinition, type AgentInput, diff --git a/packages/eve/src/runtime/sessions/turn.ts b/packages/eve/src/runtime/sessions/turn.ts index d57dd688c..d3668c523 100644 --- a/packages/eve/src/runtime/sessions/turn.ts +++ b/packages/eve/src/runtime/sessions/turn.ts @@ -5,18 +5,17 @@ import type { AgentSourceOwner } from "#compiler/source-graph.js"; import type { PreparedToolBehavior } from "#tools/behavior.js"; /** Grouped durable workflow metadata for one prepared harness tool. */ -export type PreparedRuntimeWorkflowTask = - | { - readonly nodeId?: never; - readonly resultKind?: "tool"; - readonly workflowId: string; - } - | { - readonly nodeId: string; - readonly resultKind: "subagent"; - readonly workflowId: string; - }; - +export interface PreparedRuntimeWorkflowTask { + /** + * Runtime graph ID of the agent definition this tool delegates to, including + * the root agent for the framework `agent` tool. Used for receipts and handle + * reservations; `agentId` identifies the resulting agent instance. + * Absent for authored workflow tools, even if their body calls `ctx.agent()`. + */ + readonly nodeId?: string; + /** Registered workflow definition to execute. */ + readonly workflowId: string; +} /** * Serializable authored tool descriptor prepared by the runtime for one * harness turn. diff --git a/packages/eve/src/runtime/subagents/registry.ts b/packages/eve/src/runtime/subagents/registry.ts index 8fffb917f..9a104019a 100644 --- a/packages/eve/src/runtime/subagents/registry.ts +++ b/packages/eve/src/runtime/subagents/registry.ts @@ -165,7 +165,6 @@ export function createPreparedRuntimeSubagentTool( sourceId: definition.sourceId, task: { nodeId: definition.nodeId, - resultKind: "subagent", workflowId: subagentToolExecuteWorkflowReference.workflowId, }, }; diff --git a/packages/eve/src/runtime/subagents/workflow.ts b/packages/eve/src/runtime/subagents/workflow.ts index 4afdeb158..f6bfc7dcc 100644 --- a/packages/eve/src/runtime/subagents/workflow.ts +++ b/packages/eve/src/runtime/subagents/workflow.ts @@ -28,5 +28,5 @@ export async function subagentToolExecuteWorkflow( outputSchema: input.outputSchema as JsonObject | undefined, target: ctx.toolName, }; - return await invokeAgent(ctx, invocation, { invocationId: ctx.callId, returnResult: true }); + return await invokeAgent(ctx, invocation, { invocationId: ctx.callId }); } diff --git a/packages/eve/src/runtime/tools/registry.ts b/packages/eve/src/runtime/tools/registry.ts index 98eeba8b5..78a29046f 100644 --- a/packages/eve/src/runtime/tools/registry.ts +++ b/packages/eve/src/runtime/tools/registry.ts @@ -114,7 +114,7 @@ async function createPreparedRuntimeTool( : isSelfAgent ? { nodeId: ROOT_RUNTIME_AGENT_NODE_ID, - resultKind: "subagent", + workflowId, } : { workflowId }, diff --git a/packages/eve/src/shared/action-types.ts b/packages/eve/src/shared/action-types.ts index a28d2df12..f941cfdff 100644 --- a/packages/eve/src/shared/action-types.ts +++ b/packages/eve/src/shared/action-types.ts @@ -135,7 +135,6 @@ export const runtimeWorkflowTaskRequestSchema = z input: jsonObjectSchema, kind: z.literal("workflow-task"), nodeId: z.string().optional(), - resultKind: z.enum(["subagent", "tool"]).optional(), toolName: z.string(), workflowId: z.string(), }) diff --git a/packages/eve/src/tasks/delivery-context.test.ts b/packages/eve/src/tasks/delivery-context.test.ts index 3c121d727..d89737b6b 100644 --- a/packages/eve/src/tasks/delivery-context.test.ts +++ b/packages/eve/src/tasks/delivery-context.test.ts @@ -11,7 +11,7 @@ import { TASK_DELIVERY_INITIATING_INSTRUCTION, TASK_DELIVERY_SETTLED_INSTRUCTION, } from "#tasks/delivery-context.js"; -import { SESSION_TASKS_STATE_KEY, type SessionTaskIndexEntry } from "#tasks/session-index.js"; +import type { BackgroundWorkflowToolRun } from "#harness/workflow-tool-runs.js"; import type { TaskView } from "#tasks/types.js"; const metadata = { kind: "report-probe", name: "report_probe" } as const; @@ -75,10 +75,10 @@ describe("resolveInitiatingTaskContext", () => { expect( resolveInitiatingTaskContext({ state: taskState([ - taskEntry("task_1", "turn_1", undefined, { data: {}, kind: "workflow-tool" }), + taskEntry("task_1", "turn_1"), { - ...taskEntry("task_2", "turn_2", undefined, { data: {}, kind: "workflow-tool" }), - cohortId: "task_1", + ...taskEntry("task_2", "turn_2"), + task: { ...taskEntry("task_2", "turn_2").task, cohortId: "task_1" }, }, ]), turnId: "turn_1", @@ -90,13 +90,13 @@ describe("resolveInitiatingTaskContext", () => { }); }); - it("ignores task records that were not accepted by an executor", () => { + it("recognizes an indexed invocation without an executor binding", () => { expect( resolveInitiatingTaskContext({ state: taskState([taskEntry("task_1", "turn_1")]), turnId: "turn_1", }), - ).toBeUndefined(); + ).toMatchObject({ phase: "initiating" }); }); }); @@ -110,7 +110,10 @@ describe("resolveTaskDeliveryContext", () => { } satisfies TaskView; const state = taskState([ taskEntry("task_1", "turn_1", completed), - { ...taskEntry("task_2", "turn_2"), cohortId: "task_1" }, + { + ...taskEntry("task_2", "turn_2"), + task: { ...taskEntry("task_2", "turn_2").task, cohortId: "task_1" }, + }, taskEntry("task_3", "turn_1"), ]); @@ -142,7 +145,10 @@ describe("resolveTaskDeliveryContext", () => { resolveTaskDeliveryContext({ state: taskState([ taskEntry("task_1", "turn_1", first), - { ...taskEntry("task_2", "turn_2", second), cohortId: "task_1" }, + { + ...taskEntry("task_2", "turn_2", second), + task: { ...taskEntry("task_2", "turn_2", second).task, cohortId: "task_1" }, + }, ]), taskDeliveryId: "task_2:ready:completed", }), @@ -171,7 +177,10 @@ describe("resolveTaskDeliveryContext", () => { const state = taskState([ taskEntry("task_previous", "turn_1", previous), taskEntry("task_failed", "turn_1", failed), - { ...taskEntry("task_cancelled", "turn_2", cancelled), cohortId: "task_failed" }, + { + ...taskEntry("task_cancelled", "turn_2", cancelled), + task: { ...taskEntry("task_cancelled", "turn_2", cancelled).task, cohortId: "task_failed" }, + }, ]); const result = resolveTaskDeliveryContext({ state, @@ -203,21 +212,32 @@ describe("resolveTaskDeliveryContext", () => { function taskEntry( taskId: string, createdByTurnId: string, - terminalView?: TaskView, - executor?: { readonly data: Record; readonly kind: string }, -): SessionTaskIndexEntry { + outcome?: TaskView, +): BackgroundWorkflowToolRun { return { - createdByTurnId, - dispatchContext: { auth: { current: null, initiator: null } }, - executor, - metadata, - taskId, - taskInboxToken: `inbox-${taskId}`, - taskRunId: `run-${taskId}`, - terminalView, + callId: taskId, + toolName: metadata.name, + lifetime: "session" as const, + origin: { turnId: createdByTurnId, stepIndex: 0 }, + address: { runId: `run-${taskId}`, hookToken: `inbox-${taskId}` }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + metadata, + taskId, + outcome: + outcome === undefined + ? undefined + : { + status: outcome.status, + lastOutput: outcome.lastOutput, + usage: outcome.usage, + }, + }, }; } function taskState(tasks: readonly ReturnType[]): SessionStateMap { - return { [SESSION_TASKS_STATE_KEY]: { tasks, version: 2 } } as SessionStateMap; + return { + "eve.workflowTool": { version: 3, runs: tasks }, + } as SessionStateMap; } diff --git a/packages/eve/src/tasks/delivery-context.ts b/packages/eve/src/tasks/delivery-context.ts index e0f32c401..741dc4f15 100644 --- a/packages/eve/src/tasks/delivery-context.ts +++ b/packages/eve/src/tasks/delivery-context.ts @@ -2,7 +2,11 @@ import type { DeliverHookPayload } from "#channel/types.js"; import { markFrameworkStepInput } from "#harness/messages.js"; import type { SessionStateMap, StepInput } from "#harness/types.js"; import { EMPTY_DELIVERY_SENTINEL } from "#shared/empty-delivery.js"; -import { getSessionTaskIndex, type SessionTaskIndexEntry } from "#tasks/session-index.js"; +import { + readWorkflowTaskView, + getBackgroundWorkflowToolRuns, + type BackgroundWorkflowToolRun, +} from "#harness/workflow-tool-runs.js"; import { getTaskCohortId } from "#tasks/session-task-cohorts.js"; export const TASK_DELIVERY_CONTEXT_LABEL = "[Task state]"; @@ -45,12 +49,16 @@ export function resolveTaskDeliveryContext(input: { readonly rootTurnId: string; } | undefined { - const entries = getSessionTaskIndex(input.state); - const delivered = entries.find((entry) => input.taskDeliveryId.startsWith(`${entry.taskId}:`)); + const entries = getBackgroundWorkflowToolRuns(input.state); + const delivered = entries.find((entry) => + input.taskDeliveryId.startsWith(`${entry.task.taskId}:`), + ); if (delivered === undefined) return undefined; - const cohort = entries.filter((entry) => getTaskCohortId(entry) === getTaskCohortId(delivered)); - return { ...projectTaskCohort(cohort), rootTurnId: delivered.createdByTurnId }; + const cohort = entries.filter( + (entry) => getTaskCohortId(entry.task) === getTaskCohortId(delivered.task), + ); + return { ...projectTaskCohort(cohort), rootTurnId: delivered.origin.turnId }; } /** Returns model context for durable tasks launched by the active parent turn. */ @@ -58,26 +66,29 @@ export function resolveInitiatingTaskContext(input: { readonly state: SessionStateMap | undefined; readonly turnId: string; }): { readonly context: string; readonly phase: "initiating" } | undefined { - const cohort = getSessionTaskIndex(input.state).filter( - (entry) => entry.createdByTurnId === input.turnId, + const cohort = getBackgroundWorkflowToolRuns(input.state).filter( + (entry) => entry.origin.turnId === input.turnId, ); - if (!cohort.some((entry) => entry.executor !== undefined && entry.terminalView === undefined)) { + if (!cohort.some((entry) => entry.task.outcome === undefined)) { return undefined; } return { ...projectTaskCohort(cohort), phase: "initiating" }; } -function projectTaskCohort(cohort: readonly SessionTaskIndexEntry[]): { +function projectTaskCohort(cohort: readonly BackgroundWorkflowToolRun[]): { readonly context: string; readonly phase: "pending" | "settled"; } { - const settled = cohort.every((entry) => entry.terminalView !== undefined); - const tasks = cohort.map((entry) => ({ - name: entry.metadata.name, - output: settled ? entry.terminalView?.lastOutput : undefined, - status: entry.terminalView?.status ?? "pending", - taskId: entry.taskId, - })); + const settled = cohort.every((entry) => entry.task.outcome !== undefined); + const tasks = cohort.map((entry) => { + const view = readWorkflowTaskView(entry.task); + return { + name: entry.task.metadata.name, + output: settled ? view?.lastOutput : undefined, + status: view?.status ?? "pending", + taskId: entry.task.taskId, + }; + }); return { context: `${TASK_DELIVERY_CONTEXT_LABEL}\n${JSON.stringify({ tasks })}`, diff --git a/packages/eve/src/tasks/json.test.ts b/packages/eve/src/tasks/json.test.ts index d62cc4182..c8570283f 100644 --- a/packages/eve/src/tasks/json.test.ts +++ b/packages/eve/src/tasks/json.test.ts @@ -8,9 +8,6 @@ import type { TaskView } from "#tasks/types.js"; describe("taskViewToJson", () => { it("projects only the model-visible task fields", () => { const view: TaskView = { - executor: { - binding: { data: { runId: "run-1" }, kind: "workflow-tool" }, - }, lastOutput: { data: { answer: 42 }, type: "result" }, metadata: { agentId: "agent-1", diff --git a/packages/eve/src/tasks/notification.ts b/packages/eve/src/tasks/notification.ts new file mode 100644 index 000000000..e3f0976a1 --- /dev/null +++ b/packages/eve/src/tasks/notification.ts @@ -0,0 +1,20 @@ +import type { JsonValue } from "#shared/json.js"; +import type { TaskView } from "#tasks/types.js"; + +export function formatTaskNotification(view: TaskView): string { + const subject = `Background task ${view.taskId} (${view.metadata.name})`; + if (view.status === "input_required") { + return `${subject} needs input.`; + } + if (view.status === "completed") { + return `${subject} is completed.\n\nResult:\n${formatTaskOutput(view.lastOutput.data)}`; + } + if (view.status === "failed") { + return `${subject} failed.\n\nError:\n${formatTaskOutput(view.lastOutput.data)}`; + } + return `${subject} is cancelled.`; +} + +export function formatTaskOutput(output: JsonValue): string { + return typeof output === "string" ? output : (JSON.stringify(output) ?? "null"); +} diff --git a/packages/eve/src/tasks/session-index.test.ts b/packages/eve/src/tasks/session-index.test.ts deleted file mode 100644 index 0db2dacdc..000000000 --- a/packages/eve/src/tasks/session-index.test.ts +++ /dev/null @@ -1,422 +0,0 @@ -import { describe, expect, it } from "vitest"; - -import type { HarnessSession } from "#harness/types.js"; -import { - LEGACY_TASK_AGENT_DISPATCH_CONTEXT, - SESSION_TASKS_STATE_KEY, - cacheTerminalTaskView, - findSessionTaskEntry, - getSessionTaskIndex, - recordSessionTask, -} from "#tasks/session-index.js"; -import { getTaskCohortId, getSessionTaskCohorts } from "#tasks/session-task-cohorts.js"; -import { deriveTaskId } from "#tasks/task-id.js"; -import type { TaskView } from "#tasks/types.js"; - -function createSession(state?: HarnessSession["state"]): HarnessSession { - return { - agent: { - modelReference: { id: "model_test" }, - system: "", - tools: [], - }, - compaction: { recentWindowSize: 4, threshold: 1_000_000 }, - continuationToken: "continuation_test", - history: [], - sessionId: "session_parent", - state, - }; -} - -describe("session task index", () => { - const metadata = { - kind: "tool" as const, - name: "research", - }; - const dispatchContext = { auth: { current: null, initiator: null } } as const; - it("returns an empty index when the key is absent", () => { - expect(getSessionTaskIndex({})).toEqual([]); - expect(getSessionTaskIndex(undefined)).toEqual([]); - }); - - it("records a task and finds it by id", () => { - const session = recordSessionTask(createSession(), { - dispatchContext, - taskInboxToken: "task:token-1", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-1", - }); - - expect(findSessionTaskEntry(session.state, "task_a")).toEqual({ - dispatchContext, - taskInboxToken: "task:token-1", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-1", - }); - expect(findSessionTaskEntry(session.state, "task_other")).toBeUndefined(); - }); - - it("keeps activity identity in the persisted task index", () => { - const activityWorkIdentity = { - callId: "call-1", - id: "work:task", - kind: "task" as const, - name: "research", - parentId: "work:root", - rootSessionId: "root-session", - rootTurnId: "root-turn", - }; - const session = recordSessionTask(createSession(), { - activityWorkIdentity, - dispatchContext, - taskInboxToken: "task:token-1", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-1", - }); - - const restoredState = JSON.parse(JSON.stringify(session.state)); - expect(findSessionTaskEntry(restoredState, "task_a")?.activityWorkIdentity).toEqual( - activityWorkIdentity, - ); - }); - - it("keeps subagent metadata in the persisted task index", () => { - const subagentMetadata = { - agentId: "ag_worker", - kind: "subagent", - mode: "remote", - name: "research", - } as const; - - const session = recordSessionTask(createSession(), { - dispatchContext, - taskInboxToken: "task:token-1", - createdByTurnId: "turn-1", - metadata: subagentMetadata, - taskId: "task_a", - taskRunId: "run-1", - }); - - expect(findSessionTaskEntry(session.state, "task_a")?.metadata).toEqual(subagentMetadata); - }); - - it("keeps a terminal view when replayed activity presentation changes", () => { - let session = recordSessionTask(createSession(), { - activityWorkIdentity: { - callId: "call-1", - id: "work:task", - kind: "task", - label: "First label", - name: "research", - parentId: "work:root", - rootSessionId: "root-session", - rootTurnId: "root-turn", - }, - dispatchContext, - taskInboxToken: "task:token-1", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-1", - }); - session = { - ...session, - state: cacheTerminalTaskView(session.state, terminal("task_a", "completed")), - }; - session = recordSessionTask(session, { - activityWorkIdentity: { - callId: "call-1", - id: "work:task", - kind: "task", - label: "Second label", - name: "research", - parentId: "work:root", - rootSessionId: "root-session", - rootTurnId: "root-turn", - }, - dispatchContext, - taskInboxToken: "task:token-2", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-2", - }); - - expect(findSessionTaskEntry(session.state, "task_a")).toMatchObject({ - activityWorkIdentity: { label: "Second label" }, - terminalView: terminal("task_a", "completed"), - }); - }); - - it("replaces the entry on replayed creation instead of duplicating it", () => { - let session = recordSessionTask(createSession(), { - dispatchContext, - taskInboxToken: "task:token-1", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-1", - }); - session = recordSessionTask(session, { - dispatchContext, - taskInboxToken: "task:token-2", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-2", - }); - - const entries = getSessionTaskIndex(session.state); - expect(entries).toHaveLength(1); - expect(entries[0]?.taskRunId).toBe("run-2"); - }); - - function task(taskId: string, createdByTurnId: string) { - return { - createdByStepIndex: 0, - createdByTurnId, - dispatchContext, - metadata, - taskId, - taskInboxToken: `inbox-${taskId}`, - taskRunId: `run-${taskId}`, - }; - } - - function terminal(taskId: string, status: "completed" | "failed" | "cancelled"): TaskView { - if (status === "cancelled") return { metadata, status, taskId }; - return status === "completed" - ? { metadata, status, taskId, lastOutput: { type: "result", data: "done" } } - : { metadata, status, taskId, lastOutput: { type: "error", data: "failed" } }; - } - - it("durably joins overlapping work across turns and executor kinds", () => { - const first = task("task_a", "turn-1"); - const initial = recordSessionTask(createSession(), first); - const second = { - ...task("task_b", "turn-2"), - executor: { kind: "workflow-tool", data: {} }, - }; - const session = recordSessionTask(initial, second); - const entries = getSessionTaskIndex(session.state); - expect(entries.map(getTaskCohortId)).toEqual(["task_a", "task_a"]); - expect(entries.map((entry) => entry.createdByTurnId)).toEqual(["turn-1", "turn-2"]); - expect(entries[0]?.cohortId).toBeUndefined(); - expect(entries[1]?.cohortId).toBe("task_a"); - const restored = createSession(JSON.parse(JSON.stringify(initial.state))); - expect(recordSessionTask(restored, second).state).toEqual(session.state); - expect(getSessionTaskIndex(initial.state)).toHaveLength(1); - }); - - it.each(["completed", "failed", "cancelled"] as const)( - "keeps a %s sibling in a pending cohort, then starts a new cohort after settlement", - (status) => { - let session = recordSessionTask(createSession(), task("task_a", "turn-1")); - session = recordSessionTask(session, task("task_b", "turn-1")); - session = { - ...session, - state: cacheTerminalTaskView(session.state, terminal("task_a", status)), - }; - session = recordSessionTask(session, task("task_c", "turn-2")); - expect(getSessionTaskIndex(session.state).map(getTaskCohortId)).toEqual([ - "task_a", - "task_a", - "task_a", - ]); - expect([...getSessionTaskCohorts(session.state).values()]).toEqual([ - { cohortId: "task_a", settled: true }, - { cohortId: "task_a", settled: false }, - { cohortId: "task_a", settled: false }, - ]); - for (const taskId of ["task_b", "task_c"]) { - session = { - ...session, - state: cacheTerminalTaskView(session.state, terminal(taskId, status)), - }; - } - // Even another creation in the same turn must not reopen a settled cohort. - session = recordSessionTask(session, task("task_d", "turn-2")); - expect(getSessionTaskIndex(session.state).map(getTaskCohortId)).toEqual([ - "task_a", - "task_a", - "task_a", - "task_d", - ]); - }, - ); - - it("preserves replayed membership, creation provenance, order, and settlement", () => { - let session = recordSessionTask(createSession(), task("task_a", "turn-1")); - session = recordSessionTask(session, task("task_b", "turn-2")); - for (const taskId of ["task_a", "task_b"]) { - session = { - ...session, - state: cacheTerminalTaskView(session.state, terminal(taskId, "completed")), - }; - } - session = recordSessionTask(session, task("task_c", "turn-3")); - session = recordSessionTask(session, { - ...task("task_a", "turn-replay"), - createdByStepIndex: 9, - taskRunId: "run-replayed", - }); - session = recordSessionTask(session, { - ...task("task_b", "turn-replay"), - createdByStepIndex: 9, - }); - expect( - getSessionTaskIndex(session.state).map((entry) => ({ - taskId: entry.taskId, - cohortId: getTaskCohortId(entry), - turnId: entry.createdByTurnId, - stepIndex: entry.createdByStepIndex, - settled: entry.terminalView !== undefined, - })), - ).toEqual([ - { taskId: "task_a", cohortId: "task_a", turnId: "turn-1", stepIndex: 0, settled: true }, - { taskId: "task_b", cohortId: "task_a", turnId: "turn-2", stepIndex: 0, settled: true }, - { taskId: "task_c", cohortId: "task_c", turnId: "turn-3", stepIndex: 0, settled: false }, - ]); - expect(findSessionTaskEntry(session.state, "task_a")?.taskRunId).toBe("run-replayed"); - session = recordSessionTask(session, task("task_d", "turn-4")); - expect(findSessionTaskEntry(session.state, "task_d")?.cohortId).toBe("task_c"); - }); - - it.each(["", null, 42])("rejects an invalid additive cohort identity: %j", (cohortId) => { - expect(() => - getSessionTaskIndex({ - [SESSION_TASKS_STATE_KEY]: { - tasks: [{ ...task("task_a", "turn-1"), cohortId }], - version: 2, - }, - }), - ).toThrow(/Corrupt task index/u); - }); - - it("retains only terminal views as expired-run fallbacks", () => { - const base = { - dispatchContext, - taskInboxToken: "task:token-1", - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskRunId: "run-1", - }; - const terminalView = { - lastOutput: { data: "done", type: "result" as const }, - metadata, - status: "completed" as const, - taskId: "task_a", - }; - - const session = recordSessionTask(createSession(), { ...base, terminalView }); - expect(findSessionTaskEntry(session.state, "task_a")?.terminalView).toEqual(terminalView); - for (const invalidView of [ - { metadata, status: "working", taskId: "task_a" }, - { metadata, status: "completed", taskId: "task_a" }, - { - lastOutput: { data: "wrong", type: "result" }, - metadata, - status: "failed", - taskId: "task_a", - }, - { - lastOutput: { data: "wrong", type: "result" }, - metadata, - status: "cancelled", - taskId: "task_a", - }, - { - inputRequests: [{ requestId: "stale" }], - lastOutput: { data: "done", type: "result" }, - metadata, - status: "completed", - taskId: "task_a", - }, - { ...terminalView, taskId: "task_other" }, - ]) { - expect(() => - getSessionTaskIndex({ - [SESSION_TASKS_STATE_KEY]: { - tasks: [{ ...base, terminalView: invalidView }], - version: 2, - }, - }), - ).toThrow(`Corrupt task index under session state key "${SESSION_TASKS_STATE_KEY}"`); - } - }); - - it("throws on a corrupt index instead of treating it as absent", () => { - expect(() => - getSessionTaskIndex({ - [SESSION_TASKS_STATE_KEY]: { tasks: [{ taskId: 42 }], version: 2 }, - }), - ).toThrow(`Corrupt task index under session state key "${SESSION_TASKS_STATE_KEY}"`); - }); - - it("normalizes a task without creator context to an explicit legacy state", () => { - expect( - getSessionTaskIndex({ - [SESSION_TASKS_STATE_KEY]: { - tasks: [ - { - createdByTurnId: "turn-1", - metadata, - taskId: "task_a", - taskInboxToken: "task:token-1", - taskRunId: "run-1", - }, - ], - version: 2, - }, - })[0]?.dispatchContext, - ).toEqual(LEGACY_TASK_AGENT_DISPATCH_CONTEXT); - }); - - it("rejects unrecognized task dispatch context fields", () => { - expect(() => - getSessionTaskIndex({ - [SESSION_TASKS_STATE_KEY]: { - tasks: [ - { - createdByTurnId: "turn-1", - dispatchContext: { - auth: { current: null, initiator: null }, - unexpected: "receiver-context", - }, - metadata, - taskId: "task_a", - taskInboxToken: "task:token-1", - taskRunId: "run-1", - }, - ], - version: 2, - }, - }), - ).toThrow(`Corrupt task index under session state key "${SESSION_TASKS_STATE_KEY}"`); - }); - - it("rejects the old task index version explicitly", () => { - expect(() => - getSessionTaskIndex({ [SESSION_TASKS_STATE_KEY]: { tasks: [], version: 1 } }), - ).toThrow( - `Unsupported task index version 1 under session state key "${SESSION_TASKS_STATE_KEY}"`, - ); - }); -}); - -describe("deriveTaskId", () => { - it("is deterministic for the same originating call and distinct otherwise", () => { - const input = { callId: "call-1", parentSessionId: "session-1", parentTurnId: "turn-1" }; - - expect(deriveTaskId(input)).toBe(deriveTaskId(input)); - expect(deriveTaskId(input)).toMatch(/^task_[0-9a-f]{24}$/); - expect(deriveTaskId({ ...input, callId: "call-2" })).not.toBe(deriveTaskId(input)); - }); -}); diff --git a/packages/eve/src/tasks/session-index.ts b/packages/eve/src/tasks/session-index.ts deleted file mode 100644 index a7845e186..000000000 --- a/packages/eve/src/tasks/session-index.ts +++ /dev/null @@ -1,367 +0,0 @@ -import { z } from "#compiled/zod/index.js"; - -import type { HarnessSession, SessionStateMap } from "#harness/types.js"; -import { parseActivityWorkIdentityV1, type ActivityWorkIdentityV1 } from "#protocol/activity.js"; -import type { JsonValue } from "#shared/json.js"; -import type { TaskExecutorBinding } from "#tools/task.js"; -import { sameTaskMetadata, type TaskMetadata, type TaskView } from "#tasks/types.js"; -import { type DurableDynamicSubagentSelection, type SessionAuth } from "#context/keys.js"; -import { - getTaskCohortId, - SESSION_TASKS_STATE_KEY, - SESSION_TASKS_STATE_VERSION, -} from "#tasks/session-task-cohorts.js"; - -/** - * Session-state key for the parent's live-task index. - * - * The parent session stores only this index; the mutable task record - * lives in the dedicated durable task run. The PR #1190 spike found the - * session-state boundary unworkable for task state itself: session state - * threads through step results, while callback routes and child - * executors must update tasks without holding the current snapshot. - */ -export { SESSION_TASKS_STATE_KEY } from "#tasks/session-task-cohorts.js"; - -/** - * One task owned by this session. Immutable model-safe metadata keeps the - * task-to-agent join available before the task run publishes its first view. - * - * `taskInboxToken` is the private routing credential for the task run's - * inbound hook. It must never render into model context, history, task - * views, or compaction summaries — the model addresses tasks by - * `taskId` only, and lookup verifies ownership through this index. - */ -export interface SessionTaskIndexEntry { - readonly activityWorkIdentity?: ActivityWorkIdentityV1; - readonly dispatchContext: SessionTaskDispatchContext; - readonly taskId: string; - readonly taskRunId: string; - /** Immutable fallback once the owning workflow run expires. */ - readonly terminalView?: TaskView; - readonly taskInboxToken: string; - readonly createdByStepIndex?: number; - readonly createdByTurnId: string; - /** Immutable join target; absent on the task that starts a cohort. */ - readonly cohortId?: string; - readonly executor?: TaskExecutorBinding; - readonly metadata: TaskMetadata; -} - -const taskMetadataSchema = z.looseObject({ - kind: z.string().min(1), - name: z.string().min(1), -}) as z.ZodType; - -export interface TaskAgentDispatchContext { - readonly auth: SessionAuth; - readonly sessionDynamicSubagentSelections?: Readonly< - Record - >; - readonly turnDynamicSubagentSelections?: Readonly< - Record - >; -} - -export const LEGACY_TASK_AGENT_DISPATCH_CONTEXT = { legacy: true } as const; -export type SessionTaskDispatchContext = - | TaskAgentDispatchContext - | typeof LEGACY_TASK_AGENT_DISPATCH_CONTEXT; - -const sessionAuthContextSchema = z.strictObject({ - attributes: z.record(z.string(), z.union([z.string(), z.array(z.string()).readonly()])), - authenticator: z.string(), - issuer: z.string().optional(), - principalId: z.string(), - principalType: z.string(), - subject: z.string().optional(), -}); -const dynamicSubagentSelectionsSchema = z.record( - z.string(), - z.custom(), -); - -const taskAgentDispatchContextSchema: z.ZodType = z.strictObject({ - auth: z.strictObject({ - current: sessionAuthContextSchema.nullable(), - initiator: sessionAuthContextSchema.nullable(), - }), - sessionDynamicSubagentSelections: dynamicSubagentSelectionsSchema.optional(), - turnDynamicSubagentSelections: dynamicSubagentSelectionsSchema.optional(), -}); -const sessionTaskDispatchContextSchema = z.union([ - taskAgentDispatchContextSchema, - z.strictObject({ legacy: z.literal(true) }), -]); - -const taskViewBaseShape = { - // Terminal views never carry pending requests; the loose object must say so explicitly. - inputRequests: z.never().optional(), - executor: z - .looseObject({ - binding: z - .looseObject({ - data: z.record(z.string(), z.custom()), - kind: z.string().min(1), - }) - .optional(), - }) - .optional(), - metadata: taskMetadataSchema, - taskId: z.string().min(1), - usage: z - .looseObject({ - cacheReadTokens: z.number().nonnegative(), - cacheWriteTokens: z.number().nonnegative(), - costUsd: z.number().finite().nonnegative().optional(), - inputTokens: z.number().nonnegative(), - outputTokens: z.number().nonnegative(), - }) - .optional(), -}; - -/** - * Terminal views only, on purpose: the index caches a view solely as - * the expired-run fallback, and the discriminated arms encode the terminal - * status/output invariants structurally (explicit fields reject - * `inputRequests` and mismatched outputs while preserving additive metadata). - */ -const taskViewSchema: z.ZodType = z.discriminatedUnion("status", [ - z.looseObject({ - ...taskViewBaseShape, - lastOutput: z.looseObject({ data: z.custom(), type: z.literal("result") }), - status: z.literal("completed"), - }), - z.looseObject({ - ...taskViewBaseShape, - lastOutput: z.looseObject({ data: z.custom(), type: z.literal("error") }), - status: z.literal("failed"), - }), - z.looseObject({ - ...taskViewBaseShape, - lastOutput: z.never().optional(), - status: z.literal("cancelled"), - }), -]); - -type StoredSessionTaskIndexEntry = Omit & { - readonly dispatchContext?: SessionTaskDispatchContext; -}; - -const storedSessionTaskIndexEntrySchema: z.ZodType = z.looseObject({ - activityWorkIdentity: z - .custom((value) => parseActivityWorkIdentityV1(value) !== undefined) - .optional(), - taskInboxToken: z.string().min(1), - createdByStepIndex: z.number().int().nonnegative().optional(), - createdByTurnId: z.string().min(1), - cohortId: z.string().min(1).optional(), - dispatchContext: sessionTaskDispatchContextSchema.optional(), - executor: z - .looseObject({ - data: z.record(z.string(), z.custom()), - kind: z.string().min(1), - }) - .optional(), - metadata: taskMetadataSchema, - taskId: z.string().min(1), - taskRunId: z.string().min(1), - terminalView: taskViewSchema.optional(), -}); - -const sessionTaskIndexSchema = z - .looseObject({ - tasks: z.array(storedSessionTaskIndexEntrySchema), - version: z.literal(SESSION_TASKS_STATE_VERSION), - }) - .refine( - (index) => new Set(index.tasks.map((entry) => entry.taskId)).size === index.tasks.length, - { - message: "Task ids must be unique.", - }, - ) - .refine( - (index) => - index.tasks.every( - (entry) => - entry.terminalView === undefined || - (entry.terminalView.taskId === entry.taskId && - sameTaskMetadata(entry.terminalView.metadata, entry.metadata)), - ), - { message: "Cached terminal views must match their task index entry." }, - ); - -interface SessionTaskIndex { - readonly [key: string]: unknown; - readonly tasks: readonly SessionTaskIndexEntry[]; - readonly version: typeof SESSION_TASKS_STATE_VERSION; -} - -/** - * Reads and validates the task index from session state. - * - * A present but invalid index throws: treating corruption as absence - * would silently orphan every live task's routing credential. - */ -export function getSessionTaskIndex( - state: SessionStateMap | undefined, -): readonly SessionTaskIndexEntry[] { - return readSessionTaskIndex(state).tasks; -} - -function readSessionTaskIndex(state: SessionStateMap | undefined): SessionTaskIndex { - const raw = state?.[SESSION_TASKS_STATE_KEY]; - if (raw === undefined) { - return { tasks: [], version: SESSION_TASKS_STATE_VERSION }; - } - const version = typeof raw === "object" && raw !== null ? Reflect.get(raw, "version") : undefined; - if (version !== SESSION_TASKS_STATE_VERSION) { - throw new Error( - `Unsupported task index version ${JSON.stringify(version)} under session state key "${SESSION_TASKS_STATE_KEY}"; expected version ${SESSION_TASKS_STATE_VERSION}.`, - ); - } - const parsed = sessionTaskIndexSchema.safeParse(raw); - if (!parsed.success) { - throw new Error( - `Corrupt task index under session state key "${SESSION_TASKS_STATE_KEY}": ${parsed.error.message}`, - ); - } - return { - ...parsed.data, - tasks: parsed.data.tasks.map((entry): SessionTaskIndexEntry => ({ - ...entry, - dispatchContext: entry.dispatchContext ?? LEGACY_TASK_AGENT_DISPATCH_CONTEXT, - })), - }; -} - -/** Caches one terminal view beside its task-run address. */ -export function cacheTerminalTaskView( - state: SessionStateMap | undefined, - view: TaskView, -): SessionStateMap | undefined { - if (!isValidTerminalView(view)) { - throw new Error(`Cannot cache invalid terminal task "${view.taskId}".`); - } - const stored = readSessionTaskIndex(state); - const entries = stored.tasks; - const index = entries.findIndex((entry) => entry.taskId === view.taskId); - if (index < 0) return state; - if (!sameTaskMetadata(entries[index]!.metadata, view.metadata)) { - throw new Error(`Task view metadata does not match index entry "${view.taskId}".`); - } - const tasks = [...entries]; - const previous = tasks[index]!.terminalView; - tasks[index] = { - ...tasks[index]!, - terminalView: taskViewSchema.parse({ - ...previous, - ...view, - metadata: { ...previous?.metadata, ...view.metadata }, - lastOutput: - view.lastOutput === undefined - ? undefined - : { - ...previous?.lastOutput, - ...view.lastOutput, - }, - usage: view.usage === undefined ? undefined : { ...previous?.usage, ...view.usage }, - executor: - view.executor === undefined - ? undefined - : { - ...previous?.executor, - ...view.executor, - binding: - view.executor.binding === undefined - ? undefined - : { - ...previous?.executor?.binding, - ...view.executor.binding, - }, - }, - }), - }; - return { - ...state, - [SESSION_TASKS_STATE_KEY]: { ...stored, tasks }, - }; -} - -function isValidTerminalView(view: TaskView): boolean { - if (view.inputRequests !== undefined) return false; - switch (view.status) { - case "completed": - return view.lastOutput?.type === "result"; - case "failed": - return view.lastOutput?.type === "error"; - case "cancelled": - return view.lastOutput === undefined; - // "input_required" is already excluded: its arm requires `inputRequests`. - case "working": - return false; - } -} - -/** Finds one owned task; `undefined` enforces parent-session ownership. */ -export function findSessionTaskEntry( - state: SessionStateMap | undefined, - taskId: string, -): SessionTaskIndexEntry | undefined { - return getSessionTaskIndex(state).find((entry) => entry.taskId === taskId); -} - -/** - * Joins the indexed cohort that still has unreported/nonterminal work. Cached - * terminal siblings remain members until the whole cohort settles. Membership - * and creation provenance survive replay, even after that cohort has settled. - */ -export function recordSessionTask( - session: HarnessSession, - entry: Omit, -): HarnessSession { - const stored = readSessionTaskIndex(session.state); - const tasks = [...stored.tasks]; - const index = tasks.findIndex((candidate) => candidate.taskId === entry.taskId); - const previous = tasks[index]; - if (previous !== undefined) { - tasks[index] = { - ...previous, - ...entry, - metadata: { ...previous.metadata, ...entry.metadata }, - activityWorkIdentity: - entry.activityWorkIdentity === undefined - ? previous.activityWorkIdentity - : { - ...previous.activityWorkIdentity, - ...entry.activityWorkIdentity, - }, - executor: - entry.executor === undefined - ? previous.executor - : { ...previous.executor, ...entry.executor }, - cohortId: previous.cohortId, - createdByStepIndex: previous.createdByStepIndex, - createdByTurnId: previous.createdByTurnId, - dispatchContext: previous.dispatchContext, - terminalView: previous.terminalView ?? entry.terminalView, - }; - } else { - const pending = tasks.find((candidate) => candidate.terminalView === undefined); - tasks.push({ - ...entry, - cohortId: pending === undefined ? undefined : getTaskCohortId(pending), - }); - } - return { - ...session, - state: { - ...session.state, - [SESSION_TASKS_STATE_KEY]: { - ...stored, - tasks, - version: SESSION_TASKS_STATE_VERSION, - } satisfies SessionTaskIndex, - }, - }; -} diff --git a/packages/eve/src/tasks/session-task-cohorts.test.ts b/packages/eve/src/tasks/session-task-cohorts.test.ts index 9b3d375d0..59e5ab43b 100644 --- a/packages/eve/src/tasks/session-task-cohorts.test.ts +++ b/packages/eve/src/tasks/session-task-cohorts.test.ts @@ -1,32 +1,32 @@ import { describe, expect, it } from "vitest"; -import { getSessionTaskIndex } from "#tasks/session-index.js"; -import { - getTaskCohortId, - getSessionTaskCohorts, - SESSION_TASKS_STATE_KEY, -} from "#tasks/session-task-cohorts.js"; +import { getBackgroundWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; +import { getTaskCohortId, getSessionTaskCohorts } from "#tasks/session-task-cohorts.js"; describe("workflow task cohort lookup", () => { it("projects the same identities as the full task index", () => { const state = { - [SESSION_TASKS_STATE_KEY]: { - version: 2, - tasks: ["turn-1", "turn-2", "turn-2"].map((createdByTurnId, index) => ({ - cohortId: index === 1 ? "task_0" : undefined, - taskId: `task_${index}`, - taskRunId: `run-${index}`, - taskInboxToken: `inbox-${index}`, - createdByTurnId, - dispatchContext: { auth: { current: null, initiator: null } }, - metadata: { kind: "subagent", name: "worker" }, + "eve.workflowTool": { + version: 3, + runs: ["turn-1", "turn-2", "turn-2"].map((createdByTurnId, index) => ({ + callId: `task_${index}`, + toolName: "worker", + lifetime: "session" as const, + origin: { turnId: createdByTurnId, stepIndex: 0 }, + address: { runId: `run-${index}`, hookToken: `inbox-${index}` }, + task: { + cohortId: index === 1 ? "task_0" : undefined, + taskId: `task_${index}`, + dispatchContext: { auth: { current: null, initiator: null } }, + metadata: { kind: "subagent", name: "worker" }, + }, })), }, }; expect([...getSessionTaskCohorts(state)]).toEqual( - getSessionTaskIndex(state).map((task) => [ - task.taskId, - { cohortId: getTaskCohortId(task), settled: task.terminalView !== undefined }, + getBackgroundWorkflowToolRuns(state).map((task) => [ + task.task.taskId, + getTaskCohortId(task.task), ]), ); }); @@ -38,25 +38,12 @@ describe("workflow task cohort lookup", () => { it.each([ null, - { version: 1, tasks: [] }, - { version: 3, tasks: [] }, - { version: 2, tasks: null }, - { version: 2, tasks: [null] }, - { version: 2, tasks: [{ taskId: "", createdByTurnId: "turn-1" }] }, - { version: 2, tasks: [{ taskId: "task_1", createdByTurnId: "" }] }, - { version: 2, tasks: [{ taskId: "task_1", createdByTurnId: 1 }] }, - ...["", null, 42].map((cohortId) => ({ - version: 2, - tasks: [{ taskId: "task_1", createdByTurnId: "turn-1", cohortId }], - })), - { - version: 2, - tasks: [ - { taskId: "task_1", createdByTurnId: "turn-1" }, - { taskId: "task_1", createdByTurnId: "turn-2" }, - ], - }, - ])("rejects ambiguous or invalid cohort identities: %j", (raw) => { - expect(() => getSessionTaskCohorts({ [SESSION_TASKS_STATE_KEY]: raw })).toThrow(/task index/u); + { version: 99, runs: [] }, + { version: 1, runs: null }, + { version: 1, runs: [null] }, + ])("rejects invalid registry state: %j", (raw) => { + expect(() => getSessionTaskCohorts({ "eve.workflowTool": raw })).toThrow( + "Corrupt workflow tool run registry", + ); }); }); diff --git a/packages/eve/src/tasks/session-task-cohorts.ts b/packages/eve/src/tasks/session-task-cohorts.ts index ce3ac0e38..5fce49af3 100644 --- a/packages/eve/src/tasks/session-task-cohorts.ts +++ b/packages/eve/src/tasks/session-task-cohorts.ts @@ -1,9 +1,5 @@ import type { SessionStateMap } from "#harness/types.js"; -import { isNonEmptyString, isObject } from "#shared/guards.js"; - -/** Session-state key for the parent's task index; mutable task records live in task runs. */ -export const SESSION_TASKS_STATE_KEY = "eve.tasks"; -export const SESSION_TASKS_STATE_VERSION = 2; +import { getBackgroundWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; /** An entry without a join target starts its own cohort. */ export function getTaskCohortId(task: { @@ -13,44 +9,10 @@ export function getTaskCohortId(task: { return task.cohortId ?? task.taskId; } -/** - * Reads cohort identities and settlement for workflow-side completion batching. - * Full task validation stays in getSessionTaskIndex on the step side, so - * the workflow bundle does not retain the task schemas and their dependencies. - */ export function getSessionTaskCohorts( state: SessionStateMap | undefined, -): ReadonlyMap { - const cohorts = new Map(); - const raw = state?.[SESSION_TASKS_STATE_KEY]; - if (raw === undefined) return cohorts; - - if (!isObject(raw) || raw.version !== SESSION_TASKS_STATE_VERSION) { - throw new Error( - `Unsupported task index version under session state key "${SESSION_TASKS_STATE_KEY}".`, - ); - } - if (!Array.isArray(raw.tasks)) { - throw new Error( - `Corrupt task index under session state key "${SESSION_TASKS_STATE_KEY}": expected tasks array.`, - ); - } - for (const task of raw.tasks) { - if ( - !isObject(task) || - !isNonEmptyString(task.taskId) || - !isNonEmptyString(task.createdByTurnId) || - (task.cohortId !== undefined && !isNonEmptyString(task.cohortId)) || - cohorts.has(task.taskId) - ) { - throw new Error( - `Corrupt task index under session state key "${SESSION_TASKS_STATE_KEY}": invalid task cohort identity.`, - ); - } - cohorts.set(task.taskId, { - cohortId: getTaskCohortId({ taskId: task.taskId, cohortId: task.cohortId }), - settled: task.terminalView !== undefined, - }); - } - return cohorts; +): ReadonlyMap { + return new Map( + getBackgroundWorkflowToolRuns(state).map(({ task }) => [task.taskId, getTaskCohortId(task)]), + ); } diff --git a/packages/eve/src/tasks/transitions.test.ts b/packages/eve/src/tasks/transitions.test.ts index a0b16d943..52b9905a1 100644 --- a/packages/eve/src/tasks/transitions.test.ts +++ b/packages/eve/src/tasks/transitions.test.ts @@ -1,3 +1,4 @@ +import type { JsonValue } from "#shared/json.js"; import { describe, expect, it } from "vitest"; import { applyTaskTransition } from "#tasks/transitions.js"; @@ -13,34 +14,6 @@ function view(status: TaskStatus, overrides: Partial = {}): TaskView { } describe("applyTaskTransition", () => { - it("binds one opaque executor idempotently", () => { - const command = { - executor: { data: { runId: "run-1" }, kind: "workflow-tool" }, - kind: "bind", - } as const; - const bound = applyTaskTransition(view("working"), command); - expect(bound).toMatchObject({ - action: "accepted", - view: { executor: { binding: command.executor } }, - }); - expect(applyTaskTransition(bound.view, command).action).toBe("noop"); - expect( - applyTaskTransition(bound.view, { - executor: { data: { runId: "run-2" }, kind: "workflow-tool" }, - kind: "bind", - }).action, - ).toBe("rejected"); - }); - - it("retains a late binding after fast completion", () => { - const completed = applyTaskTransition(view("working"), { data: "done", kind: "complete" }); - const bound = applyTaskTransition(completed.view, { - executor: { data: { runId: "run-1" }, kind: "workflow-tool" }, - kind: "bind", - }); - expect(bound).toMatchObject({ action: "accepted", view: { status: "completed" } }); - }); - it("moves through input, answer, and completion", () => { const blocked = applyTaskTransition(view("working"), { inputRequests: [{ prompt: "Continue?", requestId: "req-1" }], @@ -49,7 +22,7 @@ describe("applyTaskTransition", () => { expect(blocked).toMatchObject({ action: "accepted", view: { status: "input_required" } }); const resumed = applyTaskTransition(blocked.view, { kind: "answered", requestIds: ["req-1"] }); expect(resumed).toMatchObject({ action: "accepted", view: { status: "working" } }); - const completed = applyTaskTransition(resumed.view, { data: { answer: 42 }, kind: "complete" }); + const completed = applyTaskTransition(resumed.view, outcome({ answer: 42 })); expect(completed).toMatchObject({ action: "accepted", view: { lastOutput: { data: { answer: 42 }, type: "result" }, status: "completed" }, @@ -72,8 +45,23 @@ describe("applyTaskTransition", () => { const cancelled = applyTaskTransition(view("working"), { kind: "cancel" }); expect(cancelled).toMatchObject({ action: "accepted", view: { status: "cancelled" } }); expect(applyTaskTransition(cancelled.view, { kind: "cancel" }).action).toBe("noop"); - expect(applyTaskTransition(cancelled.view, { data: "late", kind: "complete" }).action).toBe( - "rejected", - ); + expect(applyTaskTransition(cancelled.view, outcome("late")).action).toBe("rejected"); }); }); + +function outcome(output: JsonValue) { + return { + kind: "outcome" as const, + result: { status: "completed" as const, output }, + from: { + callId: "call", + execution: "background" as const, + input: {}, + runId: "run", + sequence: 0, + stepIndex: 0, + toolName: "export", + turnId: "turn", + }, + }; +} diff --git a/packages/eve/src/tasks/transitions.ts b/packages/eve/src/tasks/transitions.ts index bdfebdf48..5017efeb4 100644 --- a/packages/eve/src/tasks/transitions.ts +++ b/packages/eve/src/tasks/transitions.ts @@ -1,11 +1,10 @@ -import { jsonValuesEqual } from "#shared/json.js"; +import type { WorkflowToolRunOutcomeMessage } from "#execution/tools/workflow/messages.js"; +import { workflowToolRunFailureOutput } from "#execution/tools/workflow/owner-inbox.js"; import { isTerminalTaskStatus, readTaskInputRequestId, type TaskCommand, type TaskInputRequest, - type TaskOutput, - type TaskUsage, type TaskView, } from "#tasks/types.js"; @@ -14,55 +13,11 @@ export type TaskTransitionResult = | { readonly action: "noop"; readonly view: TaskView } | { readonly action: "rejected"; readonly view: TaskView; readonly reason: string }; -function terminalView( +/** Pure transition function for the background invocation view. */ +export function applyTaskTransition( view: TaskView, - command: Extract, - settled: - | { readonly lastOutput: Extract; readonly status: "completed" } - | { readonly lastOutput: Extract; readonly status: "failed" } - | { readonly status: "cancelled" }, -): TaskView { - const usage = "usage" in command ? command.usage : undefined; - const base: Pick & { usage?: TaskUsage } = { - executor: view.executor, - metadata: view.metadata, - taskId: view.taskId, - }; - if (usage !== undefined) base.usage = usage; - switch (settled.status) { - case "completed": - return { ...base, lastOutput: settled.lastOutput, status: "completed" }; - case "failed": - return { ...base, lastOutput: settled.lastOutput, status: "failed" }; - case "cancelled": - return { ...base, status: "cancelled" }; - } -} - -/** Pure, executor-neutral transition function for one durable task. */ -export function applyTaskTransition(view: TaskView, command: TaskCommand): TaskTransitionResult { - if (command.kind === "bind") { - const binding = view.executor?.binding; - if ( - binding !== undefined && - binding.kind === command.executor.kind && - jsonValuesEqual(binding.data, command.executor.data) - ) { - return { action: "noop", view }; - } - if (binding !== undefined) { - return { - action: "rejected", - reason: `Task "${view.taskId}" already has an executor binding.`, - view, - }; - } - return { - action: "accepted", - view: { ...view, executor: { ...view.executor, binding: command.executor } }, - }; - } - + command: TaskCommand | ({ readonly kind: "outcome" } & WorkflowToolRunOutcomeMessage), +): TaskTransitionResult { if (isTerminalTaskStatus(view.status)) { if (command.kind === "cancel" && view.status === "cancelled") { return { action: "noop", view }; @@ -74,26 +29,38 @@ export function applyTaskTransition(view: TaskView, command: TaskCommand): TaskT }; } + const base = { metadata: view.metadata, taskId: view.taskId }; + if (command.kind === "outcome") { + const result = command.result; + return { + action: "accepted", + view: + result.status === "completed" + ? { ...base, status: "completed", lastOutput: { type: "result", data: result.output } } + : result.status === "failed" + ? { + ...base, + status: "failed", + lastOutput: { type: "error", data: workflowToolRunFailureOutput(command) }, + } + : { ...base, status: "cancelled" }, + }; + } + switch (command.kind) { - case "complete": - return { - action: "accepted", - view: terminalView(view, command, { - lastOutput: { data: command.data, type: "result" }, - status: "completed", - }), - }; - case "fail": case "reject-dispatch": return { action: "accepted", - view: terminalView(view, command, { - lastOutput: { data: command.data, type: "error" }, - status: "failed", - }), + view: { ...base, lastOutput: { data: command.data, type: "error" }, status: "failed" }, }; case "cancel": - return { action: "accepted", view: terminalView(view, command, { status: "cancelled" }) }; + return { + action: "accepted", + view: + command.usage === undefined + ? { ...base, status: "cancelled" } + : { ...base, status: "cancelled", usage: command.usage }, + }; case "require-input": if (!isValidInputRequestBatch(command.inputRequests)) { return { @@ -106,7 +73,6 @@ export function applyTaskTransition(view: TaskView, command: TaskCommand): TaskT action: "accepted", view: { inputRequests: command.inputRequests, - executor: view.executor, metadata: view.metadata, status: "input_required", taskId: view.taskId, @@ -131,7 +97,6 @@ export function applyTaskTransition(view: TaskView, command: TaskCommand): TaskT return { action: "accepted", view: { - executor: view.executor, metadata: view.metadata, status: "working", taskId: view.taskId, diff --git a/packages/eve/src/tasks/types.ts b/packages/eve/src/tasks/types.ts index 9995fa0ea..102b38e47 100644 --- a/packages/eve/src/tasks/types.ts +++ b/packages/eve/src/tasks/types.ts @@ -5,7 +5,6 @@ import type { import type { WorkflowToolAgentRequest } from "#execution/tools/workflow/messages.js"; import type { SessionInboxAddress } from "#execution/session-inbox/address.js"; import { jsonValuesEqual, type JsonValue } from "#shared/json.js"; -import type { TaskExecutorBinding } from "#tools/task.js"; /** Durable lifecycle status for one unit of background work. */ export type TaskStatus = "working" | "input_required" | "completed" | "failed" | "cancelled"; @@ -78,8 +77,6 @@ export function readTaskInputRequestId(request: TaskInputRequest): string | unde interface TaskViewBase { readonly taskId: string; readonly metadata: TaskMetadata; - /** Private executor state, excluded from model-visible JSON. */ - readonly executor?: { readonly binding?: TaskExecutorBinding }; /** Retained for accounting, excluded from model-visible JSON. */ readonly usage?: TaskUsage; } @@ -113,19 +110,8 @@ export type TaskView = TaskViewBase & } ); -/** Executor-neutral commands accepted by the durable task run. */ +/** Admission, cancellation, and input state changes for a background invocation. */ export type TaskCommand = - | { readonly executor: TaskExecutorBinding; readonly kind: "bind" } - | { - readonly kind: "complete"; - readonly data: JsonValue; - readonly usage?: TaskUsage; - } - | { - readonly kind: "fail"; - readonly data: JsonValue; - readonly usage?: TaskUsage; - } | { readonly kind: "reject-dispatch"; readonly data: JsonValue } | { readonly kind: "cancel"; @@ -140,34 +126,6 @@ export interface TaskCommandHookPayload { readonly command: TaskCommand; } -/** One authored message delivered to the parent as a new turn. */ -export interface TaskInboundMessage { - readonly callId: string; - readonly kind: "task-message"; - readonly message: string; - readonly messageIndex: number; - readonly messageEpoch: string; -} - -/** Intermediate progress reported by an executor. */ -export interface TaskInboundUpdate { - readonly callId: string; - readonly updateIndex: number; - readonly updateEpoch: string; - readonly kind: "task-update"; - readonly message: string; -} - -/** One workflow-executor request bound to its private answer hook. */ -export interface TaskInboundInputRequest { - readonly kind: "task-input-request"; - readonly replyTo: string; - readonly requests: readonly TaskInputRequest[]; - readonly sequence: number; - readonly stepIndex: number; - readonly turnId: string; -} - /** One human answer routed through the task that owns the blocked executor. */ export interface TaskInboundAnswerInput { readonly auth?: unknown; @@ -183,11 +141,7 @@ export interface TaskInboundAnswerInput { readonly taskId: string; } -export type TaskRunInboundPayload = - | TaskCommandHookPayload - | TaskInboundAnswerInput - | TaskInboundMessage - | TaskInboundUpdate; +export type TaskRunInboundPayload = TaskCommandHookPayload | TaskInboundAnswerInput; /** Generic task-owned request sent through the parent session payload. */ interface TaskInputRequestDeliveryBase { @@ -233,17 +187,6 @@ export interface TaskAgentRequestDelivery { readonly taskId: string; } -export interface TaskProgress { - readonly callId: string; - readonly kind: "task-progress"; - readonly taskId: string; - readonly update: JsonValue; - readonly updateIndex: number; -} - -export const TASK_PROGRESS_STREAM_NAMESPACE = "eve.task.progress"; -export const TASK_VIEW_STREAM_NAMESPACE = "eve.task"; - export function isTerminalTaskStatus(status: TaskStatus): boolean { return status === "completed" || status === "failed" || status === "cancelled"; } diff --git a/packages/eve/src/tools/definition-auth.test.ts b/packages/eve/src/tools/definition-auth.test.ts index a72080aa0..adebfeb8f 100644 --- a/packages/eve/src/tools/definition-auth.test.ts +++ b/packages/eve/src/tools/definition-auth.test.ts @@ -17,7 +17,7 @@ describe("defineTool auth field", () => { execute: () => null, }; - expect(() => defineTool(definition as never)).toThrow(/"auth" field is no longer supported/); + expect(() => defineTool(definition)).toThrow(/"auth" field is no longer supported/); }); }); @@ -35,17 +35,13 @@ describe("defineTool approvalKey", () => { expect(definition.approvalKey?.({ scope: "repo" })).toBe("write:repo"); }); - it("infers readonly input for background tools with input schemas", () => { - const definition = defineTool({ + it("rejects background execution", () => { + const definition = { description: "Scoped background write", execution: "background", inputSchema: z.object({ scope: z.string() }), - approvalKey(input) { - expectTypeOf(input).toEqualTypeOf>(); - return `write:${input.scope}`; - }, - execute: async (input) => input.scope, - }); - expect(definition.approvalKey?.({ scope: "repo" })).toBe("write:repo"); + execute: async () => null, + }; + expect(() => defineTool(definition)).toThrow("Use defineWorkflowTool for background work"); }); }); diff --git a/packages/eve/src/tools/definition.ts b/packages/eve/src/tools/definition.ts index 9e61ba368..1109b64df 100644 --- a/packages/eve/src/tools/definition.ts +++ b/packages/eve/src/tools/definition.ts @@ -17,20 +17,17 @@ import { } from "#tools/durable-callbacks.js"; import { TOOL_BRAND } from "#tools/dynamic.js"; import type { ToolModelOutput } from "#tools/model-output.js"; -import type { TaskExec, TaskReceipt } from "#tools/task.js"; type ApprovalContextInput = unknown extends TInput ? Record : TInput; export type { ToolAuthDefinition, ToolAuthOptions, ToolAuthProvider } from "#tools/auth.js"; export type { ToolModelOutput, ToolModelOutputPart } from "#tools/model-output.js"; -export type { TaskExec, TaskExecutorBinding, TaskReceipt } from "#tools/task.js"; export type ToolExecuteOptions = Omit, "context">; export type ToolExecuteFn = ( input: TInput, options: ToolExecuteOptions, - task?: TaskExec, ) => Promise | TOutput | AsyncIterable; export type ToolExecution = "background"; @@ -218,21 +215,6 @@ export interface ToolDefinition extends Pub toModelOutput?: (output: TOutput) => ToolModelOutput | Promise; } -/** A tool whose executor can outlive the model tool-call phase as a durable task. */ -export interface BackgroundToolDefinition< - TInput = unknown, - TOutput = unknown, -> extends PublicToolDefinition { - readonly execution: "background"; - execute( - input: TInput, - ctx: ToolContext, - task: TaskExec, - ): Promise | TOutput | AsyncIterable; - approval?: Approval>; - toModelOutput?: (output: TaskReceipt) => ToolModelOutput | Promise; -} - type ToolOutputFromExecuteReturn = TReturn extends Promise ? TOutput @@ -240,24 +222,10 @@ type ToolOutputFromExecuteReturn = ? TOutput : TReturn; -type BackgroundToolOutputFromExecuteReturn = - TReturn extends AsyncGenerator - ? TOutput - : TReturn extends AsyncIterable - ? null - : Awaited; - type ToolDefinitionWithExecuteReturn = ToolDefinition & { execute(input: TInput, ctx: ToolContext): TReturn; }; -type BackgroundToolDefinitionWithExecuteReturn = BackgroundToolDefinition< - TInput, - TOutput -> & { - execute(input: TInput, ctx: ToolContext, task: TaskExec): TReturn; -}; - /** * Defines a tool configuration, used both for static tools (default export * from `agent/tools/*.ts`) and as the entry wrapper inside `defineDynamic` @@ -266,33 +234,6 @@ type BackgroundToolDefinitionWithExecuteReturn = Backg * For static tools, the runtime tool name is the filename slug. `defineTool` * stamps a brand that lifecycle code validates; it rejects raw object literals. */ -export function defineTool< - TSchema extends StandardSchemaV1 | StandardJSONSchemaV1, - TReturn, ->(definition: { - description: BackgroundToolDefinition["description"]; - execution: "background"; - inputSchema: TSchema; - outputSchema?: PublicToolDefinition["outputSchema"]; - execute(input: StandardSchemaV1.InferOutput, ctx: ToolContext, task: TaskExec): TReturn; - label?: BackgroundToolDefinition< - StandardSchemaV1.InferOutput, - BackgroundToolOutputFromExecuteReturn - >["label"]; - approval?: BackgroundToolDefinition, unknown>["approval"]; - approvalKey?: BackgroundToolDefinition< - StandardSchemaV1.InferOutput, - unknown - >["approvalKey"]; - toModelOutput?: BackgroundToolDefinition< - unknown, - BackgroundToolOutputFromExecuteReturn - >["toModelOutput"]; -}): BackgroundToolDefinitionWithExecuteReturn< - StandardSchemaV1.InferOutput, - BackgroundToolOutputFromExecuteReturn, - TReturn ->; export function defineTool< TInputSchema extends StandardSchemaV1 | StandardJSONSchemaV1, TOutputSchema extends StandardJSONSchemaV1, @@ -384,8 +325,13 @@ export function defineTool( definition: ToolDefinition, ): ToolDefinition; export function defineTool( - definition: ToolDefinition | BackgroundToolDefinition, -): ToolDefinition | BackgroundToolDefinition { + definition: ToolDefinition, +): ToolDefinition { + if ("execution" in definition && definition.execution !== undefined) { + throw new Error( + 'defineTool: "execution" is not supported. Use defineWorkflowTool for background work.', + ); + } return stampToolDefinition(definition, "defineTool"); } diff --git a/packages/eve/src/tools/dynamic.ts b/packages/eve/src/tools/dynamic.ts index 7eb72a148..ce63a51e4 100644 --- a/packages/eve/src/tools/dynamic.ts +++ b/packages/eve/src/tools/dynamic.ts @@ -6,7 +6,6 @@ import type { ToolLabelDefinition, ToolContext, } from "#tools/definition.js"; -import type { TaskExec } from "#tools/task.js"; import type { ToolModelOutput } from "#tools/model-output.js"; /** @@ -28,8 +27,7 @@ export interface DynamicToolEntry, TOutput = an readonly description: string; readonly inputSchema: PublicToolInputSchema; readonly outputSchema?: PublicToolOutputSchema; - readonly execution?: "background"; - execute(input: TInput, ctx: ToolContext, task?: TaskExec): TOutput | Promise; + execute(input: TInput, ctx: ToolContext): TOutput | Promise; readonly toModelOutput?: (output: TOutput) => ToolModelOutput | Promise; /** * Optional per-call approval gate, mirroring the authored-tool diff --git a/packages/eve/src/tools/framework/agent.ts b/packages/eve/src/tools/framework/agent.ts index 581c5ffaa..23fd58f30 100644 --- a/packages/eve/src/tools/framework/agent.ts +++ b/packages/eve/src/tools/framework/agent.ts @@ -1,5 +1,5 @@ import type { z } from "#compiled/zod/index.js"; -import { defineTool } from "#tools/definition.js"; +import { stampToolDefinition } from "#tools/definition.js"; import { AGENT_TOOL_DESCRIPTION, SUBAGENT_TOOL_INPUT_SCHEMA, @@ -8,20 +8,20 @@ import { SUBAGENT_TASK_RECEIPT_OUTPUT_SCHEMA } from "#tools/framework/task-contr import { attachToolBehavior } from "#tools/behavior.js"; export const agent = attachToolBehavior( - defineTool({ - description: `${AGENT_TOOL_DESCRIPTION} This call starts a background task and returns a task receipt immediately.`, - execution: "background", - inputSchema: SUBAGENT_TOOL_INPUT_SCHEMA, - outputSchema: SUBAGENT_TASK_RECEIPT_OUTPUT_SCHEMA, - // `defineTool` requires a body, but the runtime tool registry rebinds this - // tool to the shared subagent workflow before it can ever run. - execute(): z.infer { - "use workflow"; - throw new Error( - 'The framework "agent" tool was executed directly. It must be resolved through the runtime tool registry, which dispatches it to the shared subagent workflow.', - ); + stampToolDefinition( + { + description: `${AGENT_TOOL_DESCRIPTION} This call starts a background task and returns a task receipt immediately.`, + execution: "background", + inputSchema: SUBAGENT_TOOL_INPUT_SCHEMA, + outputSchema: SUBAGENT_TASK_RECEIPT_OUTPUT_SCHEMA, + execute(): z.infer { + throw new Error( + 'The framework "agent" tool was executed directly. It must be resolved through the runtime tool registry, which dispatches it to the shared subagent workflow.', + ); + }, }, - }), + "defineTool", + ), { availability: ["root-session"], handling: { action: "self-agent", kind: "dispatch" } }, ); diff --git a/packages/eve/src/tools/task.test.ts b/packages/eve/src/tools/task.test.ts deleted file mode 100644 index 91e957cf1..000000000 --- a/packages/eve/src/tools/task.test.ts +++ /dev/null @@ -1,13 +0,0 @@ -import { describe, expect, it } from "vitest"; - -import { createTaskMessage } from "#tools/task.js"; - -describe("TaskExec descriptors", () => { - it("builds non-empty parent messages", () => { - expect(createTaskMessage("Review the export.")).toEqual({ - kind: "eve:task-message", - message: "Review the export.", - }); - expect(() => createTaskMessage(" ")).toThrow("Task messages must not be empty."); - }); -}); diff --git a/packages/eve/src/tools/task.ts b/packages/eve/src/tools/task.ts deleted file mode 100644 index 0fad91591..000000000 --- a/packages/eve/src/tools/task.ts +++ /dev/null @@ -1,63 +0,0 @@ -import type { BackgroundTask } from "#execution/tasks/parent/delegate.js"; -import type { HarnessSession } from "#harness/types.js"; -import type { JsonObject, JsonValue } from "#shared/json.js"; - -const TASK_MESSAGE_KIND = "eve:task-message" as const; - -export type TaskMessage = JsonObject & { - readonly kind: typeof TASK_MESSAGE_KIND; - readonly message: string; -}; - -export function createTaskMessage(message: string): TaskMessage { - if (message.trim() === "") throw new TypeError("Task messages must not be empty."); - return { kind: TASK_MESSAGE_KIND, message }; -} - -export function isTaskMessage(value: unknown): value is TaskMessage { - return ( - typeof value === "object" && value !== null && Reflect.get(value, "kind") === TASK_MESSAGE_KIND - ); -} - -/** Opaque, framework-private address used to control a task executor. */ -export interface TaskExecutorBinding { - readonly kind: string; - readonly data: JsonObject; -} - -/** @deprecated Use workflow-backed background tools with yield descriptors. */ -export interface TaskBinding { - readonly taskId: string; - readonly token: string; - readonly url?: string; -} - -/** @deprecated Use workflow-backed background tools with yield descriptors. */ -export type TaskSendCommand = - | { readonly kind: "update"; readonly message: string } - | { readonly kind: "complete"; readonly data: JsonValue } - | { readonly kind: "fail"; readonly data: JsonValue } - | { readonly kind: "cancel" }; - -/** Fixed acknowledgement returned when a background task is admitted. */ -export interface TaskReceipt { - readonly status: "working"; - readonly taskId: string; -} - -/** Capability passed only to tools declared with `execution: "background"`. */ -export interface TaskExec { - /** Model-facing durable task identity. */ - readonly taskId: string; - /** Returns a descriptor which sends one message to the parent when yielded. */ - postMessage(message: string): TaskMessage; - /** @deprecated Use yields from a workflow-backed background tool. */ - readonly binding: TaskBinding; - /** @deprecated Use yields from a workflow-backed background tool. */ - readonly send: (command: TaskSendCommand) => Promise; - /** @deprecated Use ctx.session. */ - readonly session: HarnessSession; - /** @deprecated Framework-owned task internals are not part of the authoring API. */ - readonly task: BackgroundTask; -} diff --git a/packages/eve/src/tools/workflow-definition.test.ts b/packages/eve/src/tools/workflow-definition.test.ts index 0f6f0bfca..b4daa10b6 100644 --- a/packages/eve/src/tools/workflow-definition.test.ts +++ b/packages/eve/src/tools/workflow-definition.test.ts @@ -7,6 +7,7 @@ import { isWorkflowToolDefinition, type WorkflowAgentMetadata, type WorkflowStepToolContext, + type TaskReceipt, type WorkflowToolContext, } from "#tools/workflow-definition.js"; import { normalizeToolDefinition } from "#internal/authored-definition/schema-backed.js"; @@ -69,21 +70,19 @@ describe("defineWorkflowTool", () => { expectTypeOf(useStepContext).parameter(0).toEqualTypeOf(); }); - it("provides task messages and receipt projections for background workflows", () => { + it("provides progress yields and receipt projections for background workflows", () => { const definition = defineWorkflowTool({ description: "Report a deployment", execution: "background", inputSchema: z.object({ service: z.string() }), - async *execute(input, ctx, task) { + async *execute(input, ctx) { expectTypeOf(input).toEqualTypeOf<{ service: string }>(); expectTypeOf(ctx).toEqualTypeOf(); - expectTypeOf(task.taskId).toEqualTypeOf(); yield { status: "planning" }; - yield task.postMessage(input.service); return { deployed: input.service }; }, toModelOutput(receipt) { - expectTypeOf(receipt).toEqualTypeOf(); + expectTypeOf(receipt).toEqualTypeOf(); return { type: "text", value: receipt.taskId }; }, }); diff --git a/packages/eve/src/tools/workflow-definition.ts b/packages/eve/src/tools/workflow-definition.ts index 3fd9f6a2d..f98c96915 100644 --- a/packages/eve/src/tools/workflow-definition.ts +++ b/packages/eve/src/tools/workflow-definition.ts @@ -7,14 +7,18 @@ import type { JsonObject, JsonValue } from "#shared/json.js"; import { stampToolDefinition, type PublicToolDefinition, - type BackgroundToolDefinition, type ToolContext, type ToolInputRequest, type ToolInputResponse, } from "#tools/definition.js"; -import type { TaskExec } from "#tools/task.js"; import type { ToolModelOutput } from "#tools/model-output.js"; +/** Fixed acknowledgement returned when a background task is admitted. */ +export interface TaskReceipt { + readonly status: "working"; + readonly taskId: string; +} + export interface AgentInput { readonly agentId?: string; readonly message: string; @@ -112,16 +116,15 @@ export interface BlockingWorkflowToolDefinition< toModelOutput?: (output: TOutput) => ToolModelOutput | Promise; } -type BackgroundWorkflowToolDefinition = Omit< - BackgroundToolDefinition, - "execute" +export type BackgroundWorkflowToolDefinition = PublicToolDefinition< + TInput, + TaskReceipt > & { readonly [WORKFLOW_TOOL_BRAND]: true; - execute( - input: TInput, - ctx: WorkflowToolContext, - task: TaskExec, - ): Promise | AsyncIterable; + readonly execution: "background"; + execute(input: TInput, ctx: WorkflowToolContext): Promise | AsyncIterable; + approval?: Approval : TInput>; + toModelOutput?: (output: TaskReceipt) => ToolModelOutput | Promise; }; /** A static tool whose executor runs as a durable workflow. */ @@ -140,7 +143,7 @@ type BackgroundDefinition = Omit< BackgroundWorkflowToolDefinition>, typeof WORKFLOW_TOOL_BRAND | "execute" > & { - execute(input: TInput, ctx: WorkflowToolContext, task: TaskExec): TReturn; + execute(input: TInput, ctx: WorkflowToolContext): TReturn; }; type WorkflowReturn = T extends AsyncIterable ? Output : Awaited; @@ -152,6 +155,24 @@ type Definition = Omit< execute(input: TInput, ctx: WorkflowToolContext): TReturn; }; +export function defineWorkflowTool< + TInputSchema extends Schema, + TOutputSchema extends StandardJSONSchemaV1, + TReturn extends + | Promise> + | AsyncIterable>, +>( + definition: Omit< + Definition, TReturn>, + "inputSchema" | "outputSchema" + > & { + inputSchema: TInputSchema; + outputSchema: TOutputSchema; + }, +): BlockingWorkflowToolDefinition< + StandardSchemaV1.InferOutput, + StandardJSONSchemaV1.InferOutput +>; export function defineWorkflowTool< TSchema extends Schema, TReturn extends Promise | AsyncIterable, diff --git a/packages/eve/src/tracing/agent-invocation-coordinator.test.ts b/packages/eve/src/tracing/agent-invocation-coordinator.test.ts index 9aabf564e..822079ca8 100644 --- a/packages/eve/src/tracing/agent-invocation-coordinator.test.ts +++ b/packages/eve/src/tracing/agent-invocation-coordinator.test.ts @@ -21,14 +21,18 @@ const conversation: ConversationContext = { }; const sessionState = { - "eve.runtime.workflowToolRuns": [ - { - callId: "workflow", - hookToken: "workflow-hook", - runId: "workflow-run", - toolName: "coordinate", - }, - ], + "eve.workflowTool": { + version: 3, + runs: [ + { + callId: "workflow", + toolName: "coordinate", + lifetime: "turn" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "workflow-run", hookToken: "workflow-hook" }, + }, + ], + }, }; describe("agent invocation trace coordinator", () => { diff --git a/packages/eve/src/tracing/agent-invocation-coordinator.ts b/packages/eve/src/tracing/agent-invocation-coordinator.ts index 6f0b87fb3..7150754ca 100644 --- a/packages/eve/src/tracing/agent-invocation-coordinator.ts +++ b/packages/eve/src/tracing/agent-invocation-coordinator.ts @@ -3,7 +3,7 @@ import { ConversationIdKey } from "#context/keys.js"; import { readConversationId } from "#tracing/conversation-context.js"; import type { RuntimeSubagentResult } from "#shared/action-types.js"; import type { SessionStateMap } from "#harness/types.js"; -import { getWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; +import { getBlockingWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; import { type ChannelAudience } from "#shared/channel-audience.js"; import type { ConversationContext } from "#shared/conversation-context.js"; import { @@ -55,7 +55,9 @@ export function prepareAgentInvocationTrace(input: { : readTaskActionTrace(input.serializedContext, input.sessionId, input.taskId); const parentActionCallId = input.taskId === undefined - ? getWorkflowToolRuns(input.sessionState).find((run) => run.runId === input.ownerId)?.callId + ? getBlockingWorkflowToolRuns(input.sessionState).find( + (run) => run.address.runId === input.ownerId, + )?.callId : taskAction?.callId; const turnId = taskAction?.turnId ?? input.turnId; const parentTurnContext = readTurnTraceContext(input.serializedContext, input.sessionId, turnId); diff --git a/packages/eve/src/tracing/agent-otel-provider.test.ts b/packages/eve/src/tracing/agent-otel-provider.test.ts index 676e475db..63d8c26de 100644 --- a/packages/eve/src/tracing/agent-otel-provider.test.ts +++ b/packages/eve/src/tracing/agent-otel-provider.test.ts @@ -1559,14 +1559,18 @@ describe("createAgentOtelInstrumentation", () => { serializedContext: serializeContext(context), sessionId: scope.sessionId, sessionState: { - "eve.runtime.workflowToolRuns": [ - { - callId: "workflow", - hookToken: "workflow-hook", - runId: "workflow-run", - toolName: "coordinate", - }, - ], + "eve.workflowTool": { + version: 3, + runs: [ + { + callId: "workflow", + toolName: "coordinate", + lifetime: "turn" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "workflow-run", hookToken: "workflow-hook" }, + }, + ], + }, }, startTimeMs: 2, turnId: scope.turnId, diff --git a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts index d840c2361..eaf5b1e89 100644 --- a/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts +++ b/packages/eve/src/tracing/agent-telemetry-contract.integration.test.ts @@ -359,14 +359,18 @@ describe("exported agent telemetry contract", () => { sessionId: "parent", turnId: "turn_0", sessionState: { - "eve.runtime.workflowToolRuns": [ - { - callId: "workflow", - hookToken: "hook", - runId: "workflow-run", - toolName: "coordinate", - }, - ], + "eve.workflowTool": { + version: 3, + runs: [ + { + callId: "workflow", + toolName: "coordinate", + lifetime: "turn" as const, + origin: { turnId: "turn-1", stepIndex: 0 }, + address: { runId: "workflow-run", hookToken: "hook" }, + }, + ], + }, }, }); }); diff --git a/packages/eve/src/tracing/agent-trace-context-store.ts b/packages/eve/src/tracing/agent-trace-context-store.ts index 9c2e3b328..c37924bea 100644 --- a/packages/eve/src/tracing/agent-trace-context-store.ts +++ b/packages/eve/src/tracing/agent-trace-context-store.ts @@ -15,8 +15,11 @@ import type { import { actionIdempotencyKey } from "#instrumentation/lifecycle.js"; import { deriveTaskId } from "#tasks/task-id.js"; import type { SessionStateMap } from "#harness/types.js"; -import { getWorkflowToolRuns } from "#harness/workflow-tool-runs.js"; -import { getSessionTaskIndex } from "#tasks/session-index.js"; +import { + getBlockingWorkflowToolRuns, + getBackgroundWorkflowToolRuns, +} from "#harness/workflow-tool-runs.js"; + import { createLogger } from "#internal/logging.js"; import type { InstrumentationDecision } from "#shared/instrumentation-decision.js"; import { @@ -61,11 +64,11 @@ function pruneTraceOwnership( ): void { const state = context.get(AgentTraceContextKey); if (state === undefined) return; - const calls = new Set(getWorkflowToolRuns(sessionState).map((run) => run.callId)); + const calls = new Set(getBlockingWorkflowToolRuns(sessionState).map((run) => run.callId)); const tasks = new Set( - getSessionTaskIndex(sessionState) - .filter((task) => task.terminalView === undefined) - .map((task) => task.taskId), + getBackgroundWorkflowToolRuns(sessionState) + .filter((task) => task.task.outcome === undefined) + .map((task) => task.task.taskId), ); const actionAnchors = Object.fromEntries( Object.entries(state.actionAnchors).filter( diff --git a/packages/eve/src/tracing/agent-trace-retention.test.ts b/packages/eve/src/tracing/agent-trace-retention.test.ts index 0ac13e487..56b5883ad 100644 --- a/packages/eve/src/tracing/agent-trace-retention.test.ts +++ b/packages/eve/src/tracing/agent-trace-retention.test.ts @@ -32,7 +32,9 @@ describe("trace retention by live work", () => { ); const before = serializeContext(context); expect(() => - pruneAgentTraceState(context, "session", { "eve.tasks": { version: 1, tasks: [] } }), + pruneAgentTraceState(context, "session", { + "eve.workflowTool": { version: 99, runs: [] }, + }), ).not.toThrow(); expect(serializeContext(context)).toEqual(before); expect(warn).toHaveBeenCalledWith( @@ -64,28 +66,35 @@ describe("trace retention by live work", () => { new ContextAgentTraceStateStore().setActionAnchor("key", anchor), ); const task = { - dispatchContext: { auth: { current: null, initiator: null } }, - taskId: deriveTaskId({ callId: "call", parentSessionId: "session", parentTurnId: "turn" }), - taskRunId: "task-run", - taskInboxToken: "task-token", - createdByTurnId: "turn", - metadata: { kind: "tool", name: "workflow" }, + callId: deriveTaskId({ callId: "call", parentSessionId: "session", parentTurnId: "turn" }), + toolName: "workflow", + lifetime: "session" as const, + origin: { turnId: "turn", stepIndex: 0 }, + address: { runId: "task-run", hookToken: "task-token" }, + task: { + dispatchContext: { auth: { current: null, initiator: null } }, + taskId: deriveTaskId({ callId: "call", parentSessionId: "session", parentTurnId: "turn" }), + metadata: { kind: "tool", name: "workflow" }, + }, }; - pruneAgentTraceState(context, "session", { "eve.tasks": { version: 2, tasks: [task] } }); + pruneAgentTraceState(context, "session", { + "eve.workflowTool": { version: 3, runs: [task] }, + }); expect(serializeContext(context)[AGENT_TRACE_CONTEXT_KEY]).toMatchObject({ actionAnchors: { key: anchor }, }); pruneAgentTraceState(context, "session", { - "eve.tasks": { - version: 2, - tasks: [ + "eve.workflowTool": { + version: 3, + runs: [ { ...task, - terminalView: { - taskId: task.taskId, - metadata: task.metadata, - status: "completed", - lastOutput: { type: "result", data: "done" }, + task: { + ...task.task, + outcome: { + status: "completed", + lastOutput: { type: "result", data: "done" }, + }, }, }, ], diff --git a/packages/eve/test/runtime-subagent-registry.test.ts b/packages/eve/test/runtime-subagent-registry.test.ts index 09e84ea6e..0dac92142 100644 --- a/packages/eve/test/runtime-subagent-registry.test.ts +++ b/packages/eve/test/runtime-subagent-registry.test.ts @@ -113,7 +113,6 @@ describe("createRuntimeSubagentRegistry", () => { expect(prepared.execution).toBe("background"); expect(prepared.task).toEqual({ nodeId: definition.nodeId, - resultKind: "subagent", workflowId: expect.stringContaining("subagentToolExecuteWorkflow"), }); }); diff --git a/packages/eve/test/runtime-tool-registry.test.ts b/packages/eve/test/runtime-tool-registry.test.ts index 6279b5ab3..69176b9c4 100644 --- a/packages/eve/test/runtime-tool-registry.test.ts +++ b/packages/eve/test/runtime-tool-registry.test.ts @@ -139,7 +139,6 @@ describe("createRuntimeToolRegistry", () => { const prepared = registry.preparedTools[0]; expect(prepared?.task).toEqual({ nodeId: "__root__", - resultKind: "subagent", workflowId: subagentToolExecuteWorkflowReference.workflowId, }); expect(prepared?.behavior?.handling).toEqual({ diff --git a/packages/eve/test/scenarios/runtime-sandbox-keys.scenario.test.ts b/packages/eve/test/scenarios/runtime-sandbox-keys.scenario.test.ts index 6da942199..12e4be169 100644 --- a/packages/eve/test/scenarios/runtime-sandbox-keys.scenario.test.ts +++ b/packages/eve/test/scenarios/runtime-sandbox-keys.scenario.test.ts @@ -44,7 +44,7 @@ async function createTemporaryAppRoot(options?: { sourceGraphHash?: string }): P generator: { name: "eve", version: "0.0.0-test" }, kind: "eve-compile-metadata", status: "ready", - version: 5, + version: 6, })}\n`, ); return appRoot; diff --git a/patches/workflow-world-postgres-5.0.0-beta.44.patch b/patches/workflow-world-postgres-5.0.0-beta.44.patch new file mode 100644 index 000000000..9c4658be6 --- /dev/null +++ b/patches/workflow-world-postgres-5.0.0-beta.44.patch @@ -0,0 +1,48 @@ +diff --git a/dist/queue.js b/dist/queue.js +--- a/dist/queue.js ++++ b/dist/queue.js +@@ -150,7 +150,6 @@ + }; + const completedMessages = new Set(); + const inflightMessages = new Map(); +- const inflightWorkflowRuns = new Map(); + let workerUtils = null; + let runner = null; + let runnerStart = null; +@@ -438,10 +437,6 @@ + const queueName = `${queue}${messageData.id}`; + const body = await deserializeMessageBody(messageData.data); + QueuePayloadSchema.parse(body); +- const workflowInvoke = WorkflowInvokePayloadSchema.safeParse(body); +- const workflowRunSerializationKey = workflowInvoke.success && !workflowInvoke.data.stepId +- ? `workflow:${workflowInvoke.data.runId}` +- : undefined; + const executeTask = async () => { + const result = await executeMessageOverHttp({ + queueName, +@@ -475,23 +470,8 @@ + }; + const idempotencyKey = messageData.idempotencyKey; + if (!idempotencyKey) { +- if (workflowRunSerializationKey) { +- // Preserve step fan-out while preventing two workflow replays from +- // mutating the same run's event log at the same time. +- const previous = inflightWorkflowRuns.get(workflowRunSerializationKey); +- const execution = (previous ?? Promise.resolve()) +- .catch(() => { }) +- .then(() => executeTask()) +- .finally(() => { +- if (inflightWorkflowRuns.get(workflowRunSerializationKey) === +- execution) { +- inflightWorkflowRuns.delete(workflowRunSerializationKey); +- } +- }); +- inflightWorkflowRuns.set(workflowRunSerializationKey, execution); +- await execution; +- return; +- } ++ // A delivery can hold an inline step until another wake aborts it. ++ // Run-level exclusion here would also exclude that required wake. + await executeTask(); + return; + } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 6c94de3a5..3a0d35377 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -249,6 +249,9 @@ catalogs: specifier: 4.5.4 version: 4.5.4 +patchedDependencies: + '@workflow/world-postgres@5.0.0-beta.44': 446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723 + importers: .: @@ -573,7 +576,7 @@ importers: version: link:../../../e2e/fixtures/e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) dd-trace: specifier: 6.13.0 version: 6.13.0 @@ -755,7 +758,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -774,7 +777,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -796,7 +799,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -818,7 +821,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -840,7 +843,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -862,7 +865,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -887,7 +890,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -906,7 +909,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -925,7 +928,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -947,7 +950,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -969,7 +972,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) ai: specifier: 'catalog:' version: 7.0.105(zod@4.5.4) @@ -991,7 +994,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1013,7 +1016,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1032,7 +1035,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1051,7 +1054,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1073,7 +1076,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1095,7 +1098,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1114,7 +1117,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1139,7 +1142,7 @@ importers: version: 2.2.0(@ai-sdk/mcp@2.0.52(zod@4.5.4))(@auth/core@0.41.2)(@chat-adapter/slack@4.34.0(ai@7.0.105(zod@4.5.4))(bufferutil@4.1.0)(supports-color@10.2.2)(zod@4.5.4))(ai@7.0.105(zod@4.5.4))(eve@packages+eve) '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1164,7 +1167,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1186,7 +1189,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1205,7 +1208,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1227,7 +1230,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1249,7 +1252,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1271,7 +1274,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1299,7 +1302,7 @@ importers: version: file:e2e/fixtures/agent-tools/fixtures/dynamic-import-dependency '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1321,7 +1324,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1343,7 +1346,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1365,7 +1368,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1387,7 +1390,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1406,7 +1409,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1428,7 +1431,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1475,7 +1478,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -1506,7 +1509,7 @@ importers: version: link:../e2e-config '@workflow/world-postgres': specifier: 'catalog:' - version: 5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) + version: 5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2) eve: specifier: workspace:* version: link:../../../packages/eve @@ -23342,7 +23345,7 @@ snapshots: optionalDependencies: '@opentelemetry/api': 1.9.1 - '@workflow/world-postgres@5.0.0-beta.44(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2)': + '@workflow/world-postgres@5.0.0-beta.44(patch_hash=446cac9dc1c5f1ac72dc037fbf61d4fd93dce20623d805305dd495ab11395723)(@opentelemetry/api@1.9.1)(@types/pg@8.23.1)(@upstash/redis@1.38.4)(sql.js@1.14.2)(supports-color@10.2.2)(typescript@7.0.2)': dependencies: '@vercel/queue': 0.5.1(@opentelemetry/api@1.9.1) '@workflow/errors': 5.0.0-beta.21 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index f190d2d10..74dde3677 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -1,5 +1,10 @@ enableGlobalVirtualStore: false +# Backport https://github.com/vercel/workflow/pull/4219 until its next Postgres release. +# The per-run queue lock holds steering wakes behind the inline step they must interrupt. +patchedDependencies: + "@workflow/world-postgres@5.0.0-beta.44": patches/workflow-world-postgres-5.0.0-beta.44.patch + packages: - apps/* - apps/fixtures/* diff --git a/research/background-tasks-redesign.md b/research/background-tasks-redesign.md new file mode 100644 index 000000000..0f8411e37 --- /dev/null +++ b/research/background-tasks-redesign.md @@ -0,0 +1,426 @@ +--- +issue: https://github.com/vercel/eve/issues/1084 +status: draft +last_updated: "2026-09-18" +--- + +# Background tasks: one workflow runtime + +**Prototype result: background work is a session-owned workflow tool run.** Waiting and +background tools now enter through one durable workflow kind, use one session invocation registry, +and share body execution, report drainage, and cancellation cleanup. The parent session retains +task outcomes, pending input routes, child ownership, and accounting. The background owner handles +admission and routes child messages through the existing session inbox. + +This prototype is based on `32aca9b1485ce8fce0eb9c636d741456c1779f25`. The linked, closed issue +is provenance only. The branch is an investigation, not a production-ready migration. + +## Approved decisions + +The prototype implements three approved scope decisions: + +1. Authored background work uses `defineWorkflowTool({ execution: "background" })`. + `defineTool` and dynamic tools no longer accept background execution. +2. The authored `TaskExec` third argument, `TaskMessage`, `task.postMessage`, and deprecated task + fields are removed. Background workflow yields are consumed without a task-progress stream; `ctx.ask`, authorization, and + framework-owned workflow requests remain. +3. Completed, failed, and cancelled outcomes wait for their existing session cohort and become one + automatic report, including all-failed and all-cancelled cohorts. Lifecycle state still updates + when each task settles. User input, human-input requests, and authorization do not wait for the + cohort. + +## Resulting execution model + +`workflowToolRunWorkflow` in `workflow.ts` is the durable entry and execution loop for both +modes. It starts `executeWorkflowBody`, owns the internal workflow inbox, reads commands and +requests, and drains every persisted report before emitting the outcome. Background work starts +only after `ready`; cancellation cleanup is bounded. [Shared workflow][prototype-invocation] + +`workflow-owner-blocking.ts` routes messages to the waiting turn. `workflow-owner-background.ts` +handles task admission and routes messages through the parent session inbox. The parent records +task outcomes and pending input routes. Admitted background work survives the initiating turn. +[Blocking owner][prototype-blocking], [background owner][prototype-background] + +Both paths use `deliverWorkflowAuthorization` to deliver or deliberately discard an authorization +event before acknowledging it. Only step-owned events receive that acknowledgement; forwarded +agent events retain their invocation reply channel. Blocking delivery completes after parent event +processing; background delivery completes when the parent inbox accepts the notification. +The background owner filters events after cancellation, allowing workflow authorization completion +to close a displayed prompt. Pending authorization attempts remain in the shared workflow step; +the background owner tracks only ordinary input requests for answer routing. + +| Concern | Before | Prototype | +| ----------------------------------- | ---------------------------------------------------- | --------------------------------------------------- | +| Workflow-body execution owners | 2 direct callers of `executeWorkflowBody` | 1 shared invocation loop | +| Background executor implementations | Workflow body or inline `defineTool` body | Workflow body only | +| Durable workflow kinds | Foreground workflow-tool run and background task run | One workflow-tool run entry | +| Persistent records | Separate workflow-tool-run and task registries | One invocation registry, with task payloads | +| Authored background protocols | Return/yield plus `TaskExec`/`TaskMessage` | Return/yield plus workflow context | +| Terminal report classifier | Successful `:ready:completed` delivery ID suffix | Stable terminal delivery ID, retained after routing | + +A task is the public handle for an admitted session-owned invocation. Both lifetimes now live in +`eve.workflowTool` as `{ version: 3, runs: WorkflowToolRun[] }`; task lookup and blocking-run lookup +are filtered views of that registry. Version 3 replaces the version-2 task-only index on main. +Cleanup selects the originating turn and `lifetime: "turn"`, so it cannot discard +session-owned work. [Registry][prototype-registry] + +## Shared state and retention + +Store invocation identity and ownership once; keep task-specific behavior in a typed `task` payload. +The identity is `(origin.turnId, callId)` within the owning session. `taskId` remains the public task +handle. No additional invocation ID or generic extension system is introduced. + +```ts +type WorkflowToolRun = { + callId: string; + toolName: string; + origin: { turnId: string; stepIndex: number }; + address: { runId: string; hookToken: string }; +} & ( + | { lifetime: "turn" } + | { + lifetime: "session"; + task: { + taskId: string; + metadata: TaskMetadata; + dispatchContext: TaskAgentDispatchContext; + activityWorkIdentity?: ActivityWorkIdentityV1; + cohortId?: string; + outcome?: TaskOutcome; + }; + } +); +``` + +`WorkflowToolRun` has `BlockingWorkflowToolRun` and `BackgroundWorkflowToolRun` variants; +`WorkflowTaskPayload` holds the background task fields. `TaskAgentDispatchContext` +captures the creator's authentication and dynamic subagent selections; it must not be replaced +with the authentication of a later input delivery. `TaskMetadata` describes the public task, +`ActivityWorkIdentityV1` links its activity stream, and `TaskView` is its public status/output view. +`cohortId` groups overlapping work for one combined report. + +| State | Owner and lifetime | +| ------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| Common identity, origin, run address | Session registry; waiting entries are removed when their call settles or their turn is cancelled | +| Task metadata, creator context, cohort membership | Typed task payload; retained for the session lifetime | +| Pending questions | Parent session proxy-input state | +| Final task status and output | Parent-owned `task.outcome`; retained for the session lifetime, including after the combined report | +| Pending deliveries and report deduplication | Existing session input queue | + +Registration precedes the `ready` admission command. A waiting outcome is matched against its +originating turn, call ID, and run ID. Authorization can end the visible turn while its workflow +call remains pending; cancellation uses the pending coordination batch's originating turn in that +case. Results without an origin can bind only to an unambiguous recorded call. Replaying +registration preserves original creator context, +cohort membership, and any recorded terminal outcome. Reusing a task ID for another turn is rejected. + +**Retention decision:** completed, failed, and cancelled task payloads stay until the session ends. +Report delivery does not prune them. This avoids a second cleanup protocol for now; retained state +therefore grows with task count and final output size. Live progress is not copied into this registry. + +## What was eliminated + +The prototype deletes these responsibilities rather than renaming them: + +- Parent-step execution of ordinary background tool bodies, async-iterable drainage, authorization + return handling, and final task-command delivery. +- Lifecycle-only task runs with no workflow body and the optional no-body branch in + the background workflow input. +- Two `TaskExec` constructors, `TaskMessage` detection, message buffering, and the dedicated + task-message parent wake step. +- Background dynamic-tool persistence and replay. +- Success-only cohort classification; all three terminal delivery suffixes now share the barrier. +- `TaskExecutorBinding`, the `bind` command, fixed executor tags, and the separate executor-run + cancellation lookup. +- The `taskRunWorkflow` durable entry and its stable workflow registration. +- Workflow-to-task outcome, progress, and question wrappers. The background owner consumes + workflow messages directly; only session delivery and public view projection adapt their shape. +- Pre-admission progress buffering: the workflow body cannot emit before `ready` starts it. +- The separate `eve.runtime.workflowToolRuns` store and independent task-index write path; + `eve.workflowTool` now holds both lifetimes without duplicate creation provenance or run addresses. + +These responsibilities were retained or relocated: + +- Body start, report drainage, cancellation cleanup, and final outcome construction live in the shared invocation loop. +- Admission, compensation, child reservation/claiming, steering, task cancellation, and session + indexing remain in the task owner. +- Waiting-run and task lookup remain filtered projections of the shared invocation registry. + The lifetime discriminator controls cleanup. +- The session input queue still buffers active-turn deliveries and performs cohort release. + +## Size comparison + +The reduction comes from deleting ordinary background executors, the authored task-message +protocol, executor bindings, duplicate message conversions, and their associated tests. Shared +ordering and cancellation tests exercise the real invocation and channel readers for both modes. + +Measure the current patch with `git diff --numstat 32aca9b1485ce8fce0eb9c636d741456c1779f25`. +For production source, include `packages/eve/src/**` and exclude `*.test.ts` and +`src/internal/testing/**`. Report documentation, generated extension reports, test support, +and E2E fixtures separately. The PR description records the current diff totals. + +## Observable semantics + +Background authoring has one supported shape: + +```ts +import { defineWorkflowTool } from "eve/tools"; +import { z } from "zod"; + +export default defineWorkflowTool({ + description: "Review a change in the background.", + execution: "background", + inputSchema: z.object({ request: z.string() }), + async *execute({ request }, ctx) { + "use workflow"; + yield { phase: "reviewing" }; + return await ctx.agent("reviewer", { message: request }); + }, +}); +``` + +The original call receives `{ status: "working", taskId }` after admission. Progress yields update +the stream without starting a parent turn. `return` completes the task, an escaping error fails it, +and task cancellation remains final against a late successful invocation outcome. + +A **cohort** is overlapping background work owned by one session. Existing membership is +preserved across overlapping turns. The automatic report becomes eligible only when every member +is terminal, and it contains success, failure, and cancellation payloads. A user message remains a +normal turn, and workflow input/authorization requests remain serviceable while a cohort is open. +Explicit cancellation checks the parent's recorded outcome before cancelling owned work. It records +cancellation when the control step returns and queues a settlement notification with the existing +delivery ID. Late child outcomes cannot replace that parent decision. Steering cancellation continues to mark +superseded task deliveries for suppression. + +## Compatibility and migration + +This is a breaking authoring change, so the prototype includes a minor changeset and updated public +docs. Extension capability generation records `tool` epoch 45 and `dynamicTool` epoch 42. Channel epoch 24 removes executor bindings from task views and drops epoch 23. The authoring change also +drops retained epochs that explicitly exercised the removed surfaces: tool epoch 28, dynamic-tool +epochs 28–30, and the previous current epochs 44 and 41. + +Migrate an ordinary background definition by replacing `defineTool` with `defineWorkflowTool`, +adding `"use workflow"`, moving nondeterministic side effects into `"use step"` helpers, removing +the third executor argument, and replacing parent messages with progress or the final return value. +A dependency on an intermediate result belongs inside the owning workflow through `ctx.agent`, +`ctx.ask`, or another durable operation. + +Existing sessions containing either old registry key are rejected by the new runtime. Conversation +import can stop discoverable old runs and retain conversation history, but does not migrate pending +work. Completed task payloads also require the new format before deployment handoff. + +The cross-deployment checkpoint version is now **6**. Version 4 and 5 readers also enforce exact +version equality, so both old-to-new and new-to-old handoffs reject the incompatible checkpoint +before hydrating nested state or claiming session hooks. The original owner recovers its hooks +and processes the triggering message. This version boundary is necessary even for idle sessions: +an older reader could otherwise overlook the new registry key and lose retained task outputs. + +Within version 6, unknown fields on the registry, invocation, origin, address, task payload, and +terminal output survive parsing, replayed registration, and terminal-cache updates. Authentication +and dispatch-context schemas remain strict; incompatible changes there require another checkpoint +version bump. Session turns execute on their owning deployment. Legacy import returns its prepared +conversation snapshot to the parked old driver, rather than exporting the new owner's registry. + +These checks protect the handoff boundary; they do not migrate in-flight workflows. A rollout must +keep the original deployments available for retained sessions and old task runs, or drain them first. + +## Demonstrated behavior + +| Requirement | Evidence | +| ------------------------------------------------------------- | ----------------------------------------------------------------- | +| Waiting workflow completion, failure, and progress | Workflow integration suite | +| Background receipt before later completion | Workflow integration suite exercises both root and child owners | +| Commit before body start | Task-owner unit test starts the workflow body only after `ready` | +| Report before outcome | Shared invocation unit test and task-owner report/outcome test | +| Explicit cancellation wins over late completion | Background-owner tests and invocation-loop tests for both modes | +| Workflow human input and authorization routing | Workflow integration suite and task-owner authorization tests | +| Mixed success/failure and success/cancellation report | Session next-input unit tests | +| All-failed and all-cancelled report | Session next-input unit tests | +| Active parent does not steer on terminal failure/cancellation | Session input queue and active-turn unit tests | +| Duplicate terminal delivery | Existing session next-input deduplication unit test | +| Cross-turn cohort membership | Existing session next-input cross-turn unit test | +| Forced-stop cancellation notification | Cancellation integration test | +| Mixed invocation lifetimes and retained terminal payloads | Shared registry unit test and turn-cancellation integration tests | +| Completion after authorization ends the visible turn | Workflow authorization integration tests | +| Extension migration boundary | Generated capability reports and invariant guard | + +Checks run before the checkpoint-version follow-up: + +- Full unit tier: **799 files passed; 8,676 tests passed; 1 skipped**. +- Workflow/session integration slices: **7 files, 107 tests passed**, including authorization, handoff, + telemetry, forced-stop cancellation, and retention during active or paused turn cancellation. +- TypeScript `--noEmit`, fresh production TypeScript/Rolldown build, focused lint, formatting, + `git diff --check`, extension-contract generation, and `guard:invariants`: passed. +- Documentation frontmatter/navigation, import snippets, and MDX compilation: passed for all 89 + published pages. + +The checkpoint-version follow-up passed 101 focused unit tests and all 35 session-entry and +legacy-import integration tests, plus typechecking and a fresh production build. The integration +suite initially failed a compatible legacy handoff while a build ran concurrently; the complete +rerun and an isolated repeat of that handoff passed. Build interference is unconfirmed. The tests verify version rejection before nested +state reads, recovery without duplicate input, and preservation of additive invocation fields. + +The normal package-manager wrapper remains unavailable in this checkout because the private +registry requires `SOCKET_PASSWORD_B64`. Dependencies were installed from the local store with a +frozen lockfile, and the underlying checks were invoked directly. No credential value was read or +printed. Local E2E was not run because the repository marks it CI-only. + +## Correctness and entropy review + +Admission starts an invocation; only its outcome settles running work. A duplicate `ready` after +cancellation previously set the settled flag and exited before cleanup messages were consumed. +That defect was carried through from the old task owner. Admission is now accepted once; rejection +before admission returns without starting the body. The regression test first failed after consuming +only three of six messages, then passed with cleanup acknowledged before terminal delivery. + +The review also removed seven registry/type aliases and the task-index facade. Callers now use the +workflow tool run registry directly. Mutations reuse their parsed registry rather than reading it +again during the write, and replay registration merges common fields once. Persisted formats and +task-payload retention are unchanged by these corrections. + +This pass ran the full unit suite (799 files, 8,678 passed, one skipped), then 51 focused tests +after the final registry and admission edits. All 64 integration tests across workflow execution, +background dispatch, cancellation, and approval passed on rerun. The progress-after-answer test +failed once before passing in isolation and in the full rerun; the cause remains unconfirmed, and +its assertion now includes captured events for diagnosis. Typechecking, production build, focused +lint, formatting, and invariant guards passed. + +## Ownership and cancellation consolidation + +Ownership reads no longer decode every retained result, and both invocation lifetimes share +cancellation escalation. The persisted registry is version 2 and checkpoints are version 6. + +- Removed the unused coordination `pendingTasks` acknowledgement list. Real background admission + still persists ownership before sending `ready`. +- `settleWorkflowToolRunCancellation` owns polling and forced stop for both lifetimes. Body cleanup + retains its 30-second limit; callers allow 35 seconds for cleanup and outcome publication. This + replaces the task-specific one-second cutoff. The parent records cancellation after the control step finishes. +- Both child-owner paths use the same cancellation function and attempt every claimed child. + Turn cancellation starts child cancellation alongside workflow cancellation. Background child + failures remain retryable; all child requests settle before the helper returns or throws. +- Registry ownership and routing reads validate addresses, identity, and ownership metadata. + `readWorkflowTaskView` validates the selected retained result and its task identity when consumed. + A malformed old result cannot block cancelling an unrelated live invocation or removing a waiting + invocation. Handoff still validates every retained result, even when other work is pending. + +Cohort membership, delivery eligibility, creator context, and session-lifetime retention are unchanged. +No corruption recovery or quarantine mechanism was added. + +Validation: all 800 unit files passed (8,692 tests, one skipped), and 73 integration tests passed +across workflow execution, task dispatch, turn cancellation, forced-stop notification, and handoff. +After final helper changes, the 29 workflow/cancellation integration tests and five child-cancellation +unit tests passed again. Typechecking, production build, lint, formatting, invariant guards, and all +89 published-doc checks passed. The first unit run exposed a missing active-turn marker in the new +cancellation fixture; correcting the fixture made it exercise the intended path. + +## Remaining proof gaps and migration risks + +The prototype does not yet demonstrate these full boundaries: + +- A live upgrade and rollback between actual released deployments. Version rejection and owner + recovery are tested locally; the old reader's version check was inspected in the base revision. + +- An explicit initiating-turn cancellation racing after task admission while the background body + remains blocked. Ownership is unchanged and existing retention code covers this path, but the + exact combined prototype path lacks a dedicated test. +- One end-to-end model-call assertion that active-parent buffered terminal outcomes later become a + single report. The tests separately prove active buffering and parked cohort release. +- Parent-session finalization cancelling a currently blocked invocation through the new reader. +- Reusable-child steering with a late superseded success through the complete combined path. +- Replay at the exact parent commit/`ready` boundary and a crash during final cohort release. + +Removing the old `taskRunWorkflow` registration changes replay compatibility for in-flight runs +created by older deployments, including workflow-backed tasks. A production rollout needs deployment +pinning or a drain/migration plan; the prototype does not provide a legacy workflow fallback. + +Delaying automatic failure output behind a slow sibling is an intentional behavior change. Task +state and activity still show failure promptly, but the model does not receive the failure as +conversation input until the cohort closes. Long-lived overlapping cohorts can therefore delay the +automatic report. No new timeout or grouping option was added. + +The framework `agent` tool remains an internal exception to authored compilation: its definition is +runtime-rebound to the shared subagent workflow. The compiler exemption is limited to the existing +`self-agent` handling marker; authored background tools without a workflow ID fail with a migration +error. + +## Parent-owned task state + +The parent session owns durable task outcomes and pending input routes. Children send outcomes, +input requests, and authorization events through the existing session inbox; they no longer write +an `eve.task` snapshot stream. The child keeps only execution-local state for admission, abort, +and answer routing. Task notifications share one durable delivery step, preserving their existing +payloads and deduplication IDs. The unused `eve.task.progress` stream is removed: authored background +yields are consumed without publishing progress, while subagent update notifications still reach +the parent. + +The first terminal outcome recorded by the parent wins. Duplicate or competing child deliveries +cannot overwrite it. Terminal notifications deduplicate by task, and their model-facing text uses +the parent's recorded outcome. A cohort waits for each terminal notification, including cancellation +already recorded by a control step, before releasing its report. Settlement clears pending task input routes. Coalesced input and spawn +requests are ignored when the same delivery also settles their task; agent settlement still runs +before its ownership lease is released. + +`task_cancel` reads the parent's outcome instead of polling a child view. An already recorded +success or failure is returned unchanged. Otherwise cancellation signals the child, attempts +owned-agent cleanup, retains the existing bounded workflow-status wait, and records the cancelled +outcome in the parent. A queued notification ensures cohort reporting also works after forced stop. +The task-view polling loop and stream read timeout are deleted. Startup and reset waits are unchanged. + +Validation for the parent-owned state change: all 8,911 unit tests passed (one skipped), and all 82 tests across seven +workflow, cancellation, session, and reset integration suites passed. Typechecking, the production +build, lint, formatting, invariant guards, and published-doc checks passed. The cancellation eval +now checks the retained outcome on the next turn without retrying; it remains CI-only. + +After notification consolidation and progress-stream removal, all 8,915 unit tests passed (one +skipped), followed by 33 focused tests including the added agent-settlement delivery ordering test. +All 46 selected workflow and cancellation integration tests passed. The background-workflow eval +now asserts that yields emit no progress events; fixture lifecycle audits count the shared delivery +step and retain their handle-cleanup and accounting checks. E2E remains CI-only. + +The wire registry is version 3. Each task stores its identity and metadata once; `outcome` +contains only terminal status, output, and optional usage. Readers combine them into a `TaskView`. +Proxy-input routes reuse their existing store. No legacy task-stream fallback is added. + +## Agent settlement + +Workflow runs carry no `resultKind`: every run returns a tool outcome. `agent-settled` owns +agent handle settlement and usage accounting. The agent helper waits for an acknowledgement on +its existing reply hook before returning output or throwing the child failure. The parent applies +settlement before acknowledging; workflow completion does not settle that agent again. + +Settlement requires the exact owner and invocation call ID. Duplicate results are no-ops, including +late deliveries after a reusable agent has been claimed by another call. Agent preparation keeps its +existing node identity for receipts and reservations; this identity is not copied onto workflow runs. +The registry version changes to 2 and checkpoint version to 6 because older readers require the +removed discriminator and expect the former result protocol. + +## Subagent result observation + +Both execution modes use `subagent.called` for dispatch and `subagent.completed` for a +successful parent-recorded invocation result. The completion carries actual output and the +original `callId`; it does not imply termination of the reusable child session. + +A background receipt is an `action.result` tool output with `status: "working"`. It does not +emit `subagent.completed`. For a generated subagent task, completion follows the parent's +first recorded successful task outcome. Replayed notifications and late success after failure +or cancellation cannot publish another completion. An agent invoked inside an authored +background workflow settles independently of its enclosing task, as in a blocking workflow. + +Completion events do not wait for sibling tasks; the existing combined report still does. +Failure and cancellation retain their distinct task outcomes and do not emit successful +completion. Evals use `pending` until they observe a successful result, then `completed`. +The task itself uses `working` or `input_required` while active. No new event or stream +version is added; existing receipt-marked events remain readable as admission. + +## Recommendation + +Retain the shared invocation runtime and parent-owned task state. A task settles when the parent +records its outcome; this does not claim that the wrapper workflow has already exited. Workflow +status remains useful for cancellation cleanup, but it does not replace the parent's task result. +An infrastructure failure before outcome delivery still needs separate reconciliation; this change +does not introduce a runtime completion subscription or repair the local runtime cancellation race. + +[prototype-invocation]: ../packages/eve/src/execution/tools/workflow/workflow.ts +[prototype-blocking]: ../packages/eve/src/execution/tools/workflow/workflow-owner-blocking.ts +[prototype-background]: ../packages/eve/src/execution/tools/workflow/workflow-owner-background.ts +[prototype-registry]: ../packages/eve/src/harness/workflow-tool-runs.ts