[core] Settle a hook's awaiter in-process instead of re-invoking, on creation and on conflict (#3938)

* [core] Settle a hook's awaiter in-process instead of re-invoking, on creation and on conflict

* [core] Address review: deterministic hook signal tests, split changesets, document the boundary

- hook.test.ts: drive the idle poll with explicit macrotask turns instead of a
  fixed 20ms sleep (Copilot)
- Split the changeset so each package's entry says only what changed in it
- runtime-tuning docs: hook-only suspensions no longer always park; the hook
  write continuation is the one exception

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Pranay Prakash
2026-09-03 13:35:26 -07:00
committed by GitHub
parent 1280163551
commit 7cc5c88a8b
14 changed files with 1422 additions and 117 deletions
+7
View File
@@ -0,0 +1,7 @@
---
'@workflow/world': patch
'@workflow/world-local': patch
'@workflow/world-sim': patch
---
Answer the `sinceCursor` event-log delta on a `hook_conflict` write the same way as on `hook_created`.
+5
View File
@@ -0,0 +1,5 @@
---
'@workflow/core': patch
---
Settle a hook's awaiter in the invocation that wrote its `hook_created` or `hook_conflict`, instead of re-invoking through the queue and replaying. Falls back to an incremental read when the World returns no delta, and to the re-invocation under `WORKFLOW_RETAINED_VM=0`.
@@ -186,7 +186,7 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
- Default: enabled
- Node.js VM engine only. Keeps the suspended workflow VM alive across inline steps within one invocation, so each iteration of the inline loop appends only the newly written events instead of replaying the whole event log in a fresh VM. QuickJS manages its own retained inline loop independently of this setting.
- A step- or attribute-driven suspension can keep the VM retained even when hooks or waits are open or created at the same boundary. Hook- or wait-only suspensions park the invocation because nothing in the current delivery can advance them. Any replay divergence falls back to a full replay.
- A step- or attribute-driven suspension can keep the VM retained even when hooks or waits are open or created at the same boundary. Hook- or wait-only suspensions park the invocation because nothing in the current delivery can advance them, with one exception: when the hook's own create is what the workflow is waiting on (a `hook.getConflict()` awaiter, or a create whose token is already claimed), the invocation resumes the retained VM over the committed `hook_created` or `hook_conflict` instead of re-invoking through the queue. Any replay divergence falls back to a full replay.
- Step inputs made of plain data (objects, arrays, primitives) and standard built-ins (`Map`, `Set`, `Date`, `RegExp`, typed arrays, `ArrayBuffer`, `URL`, `Headers`) keep the VM retained. Patching or polyfilling built-in prototypes doesn't change that because serialization never calls them. A boundary falls back to a full replay only when serializing its arguments runs code the workflow controls, such as a getter, a proxy, or a custom class serializer, or computes an `Error`'s stack trace.
- Set `0` or `false` to replay the Node.js workflow from scratch in a fresh VM on every iteration.
+393 -61
View File
@@ -362,6 +362,62 @@ registerStepFunction('r_s1', async () => 10);
registerStepFunction('r_s2', async () => 20);
registerStepFunction('r_echo', async (value) => value);
// A `hook.getConflict()` awaiter, whose whole continuation is the
// `hook_created` the suspension commits. The step after it proves the
// resumed VM keeps running past the awaiter rather than just settling it.
const hookConflictWorkflow = `const s1 = globalThis[Symbol.for("WORKFLOW_USE_STEP")]("r_s1");
const createHook = globalThis[Symbol.for("WORKFLOW_CREATE_HOOK")];
async function workflow() {
const hook = createHook({ token: "retained-conflict-token" });
const conflict = await hook.getConflict();
const a = await s1();
return conflict === null ? a : -1;
}
globalThis.__private_workflows = new Map([["workflow", workflow]]);`;
/** The token `drive({ conflictToken })` answers with a `hook_conflict`. */
const CONFLICTING_TOKEN = 'retained-taken-token';
// The same awaiter, against a token another run already holds. The create
// commits `hook_conflict` rather than `hook_created`, which settles the
// awaiter just as durably — `getConflict()` resolves with the conflicting run
// (or rejects, when no `Run` can be constructed for it). The step after it
// proves the VM kept running past the branch rather than the run going
// dormant on the conflict.
const conflictingGetConflictWorkflow = `const s1 = globalThis[Symbol.for("WORKFLOW_USE_STEP")]("r_s1");
const createHook = globalThis[Symbol.for("WORKFLOW_CREATE_HOOK")];
async function workflow() {
const hook = createHook({ token: "${CONFLICTING_TOKEN}" });
let observed;
try {
observed = (await hook.getConflict()) === null ? "clean" : "conflict";
} catch {
observed = "conflict";
}
const a = await s1();
return observed + ":" + a;
}
globalThis.__private_workflows = new Map([["workflow", workflow]]);`;
// A plain payload await against a taken token — the conflict shape with NO
// `getConflict()` awaiter, so the suspension reports only `hasHookConflict`.
// The `hook_conflict` rejects the await, and the run continues into the step.
const conflictingAwaitWorkflow = `const s1 = globalThis[Symbol.for("WORKFLOW_USE_STEP")]("r_s1");
const createHook = globalThis[Symbol.for("WORKFLOW_CREATE_HOOK")];
async function workflow() {
const hook = createHook({ token: "${CONFLICTING_TOKEN}" });
let observed;
try {
await hook;
observed = "payload";
} catch {
observed = "rejected";
}
const a = await s1();
return observed + ":" + a;
}
globalThis.__private_workflows = new Map([["workflow", workflow]]);`;
// Drive the full workflow handler over a stateful (dynamic) event log so the
// inline loop makes real progress across its own writes, exactly like a World.
// Non-turbo (no runInput, attempt 2) to keep the path simple and deterministic.
@@ -438,10 +494,41 @@ function startedStepResult(
};
}
/**
* How the harness's World answers the writes a drive makes, beyond the plain
* in-memory log: the shapes a real World can take that the runtime has to
* cope with, each opt-in so the default drive is the simplest one.
*/
type DriveWorldOptions = {
/**
* Answer `sinceCursor` with no delta, the way a World that does not
* implement it does. The runtime must then read from its cursor instead of
* assuming the log was carried forward.
*/
withholdDelta?: boolean;
/**
* Commit `hook_conflict` instead of `hook_created` for a create carrying
* this token, the way a World does when another run already holds it. The
* conflict lands on the same slot the create asked for and rides the same
* inline delta, so what the caller gets back differs only in which event
* the delta carries.
*/
conflictToken?: string;
/**
* Serve the read that follows the initial load short of the newest event,
* the way an eventually-consistent replica can. Combined with
* `withholdDelta` this is the one shape a continuation cannot get anywhere
* from: the pass runs over a log that still does not hold the event it is
* continuing over.
*/
staleFirstReload?: boolean;
};
async function drive(
runId: string,
workflowCode = twoStepWorkflow,
initialMode: DriveMode = { type: 'normal' }
initialMode: DriveMode = { type: 'normal' },
options: DriveWorldOptions = {}
) {
let mode = initialMode;
const run: WorkflowRun = {
@@ -456,67 +543,132 @@ async function drive(
};
const events: Event[] = [];
const createdEvents: any[] = [];
const createParams: any[] = [];
let seq = 0;
const eventsCreate = vi.fn(async (_runId: string, data: any) => {
if (mode.type === 'fail-event' && data.eventType === mode.eventType) {
mode = { type: 'normal' };
throw new PreconditionFailedError('stale snapshot (test-injected)');
}
createdEvents.push(data);
if (data.eventType === 'run_started') {
return { run, events };
}
const event = {
eventId: slotToEventId(++seq),
runId,
createdAt: new Date(),
...data,
} as Event;
events.push(event);
if (data.eventType === 'step_started' && mode.type === 'inject-hook') {
mode = { type: 'normal' };
const hookCreated = events.find(
(candidate) => candidate.eventType === 'hook_created'
);
assert(hookCreated, 'expected hook_created before step');
events.push({
// Cursors, positioned like a World's: each one is issued for a log of a
// known length, so the delta since it is everything appended after.
let cursorSeq = 0;
const cursorPosition = new Map<string, number>();
const nextCursor = (at = events.length): string => {
const cursor = `cursor_${++cursorSeq}`;
cursorPosition.set(cursor, at);
return cursor;
};
const eventsCreate = vi.fn(
async (_runId: string, data: any, params?: any) => {
if (mode.type === 'fail-event' && data.eventType === mode.eventType) {
mode = { type: 'normal' };
throw new PreconditionFailedError('stale snapshot (test-injected)');
}
createdEvents.push(data);
createParams.push({ eventType: data.eventType, ...params });
if (data.eventType === 'run_started') {
return { run, events };
}
// A create whose token is already claimed commits `hook_conflict` on the
// slot the `hook_created` asked for — not an error for the caller, an
// event its workflow has to observe.
const conflicting =
data.eventType === 'hook_created' &&
options.conflictToken !== undefined &&
data.eventData?.token === options.conflictToken;
const committed = conflicting
? {
eventType: 'hook_conflict',
specVersion: data.specVersion,
correlationId: data.correlationId,
eventData: {
token: data.eventData.token,
conflictingRunId: 'wrun_retained_token_owner',
},
}
: data;
const event = {
eventId: slotToEventId(++seq),
runId,
eventType: 'hook_received',
specVersion: SPEC_VERSION_CURRENT,
correlationId: hookCreated.correlationId,
eventData: {
token: hookCreated.eventData.token,
payload: await dehydrateStepReturnValue(
{ source: 'external-hook' },
runId,
undefined
),
},
createdAt: new Date(),
});
...committed,
} as Event;
events.push(event);
if (data.eventType === 'step_started' && mode.type === 'inject-hook') {
mode = { type: 'normal' };
const hookCreated = events.find(
(candidate) => candidate.eventType === 'hook_created'
);
assert(hookCreated, 'expected hook_created before step');
events.push({
eventId: slotToEventId(++seq),
runId,
eventType: 'hook_received',
specVersion: SPEC_VERSION_CURRENT,
correlationId: hookCreated.correlationId,
eventData: {
token: hookCreated.eventData.token,
payload: await dehydrateStepReturnValue(
{ source: 'external-hook' },
runId,
undefined
),
},
createdAt: new Date(),
});
}
if (data.eventType === 'step_started' && mode.type === 'inject-wait') {
mode = { type: 'normal' };
const waitCreated = events.find(
(candidate) => candidate.eventType === 'wait_created'
);
assert(waitCreated, 'expected wait_created before step');
events.push({
eventId: slotToEventId(++seq),
runId,
eventType: 'wait_completed',
specVersion: SPEC_VERSION_CURRENT,
correlationId: waitCreated.correlationId,
eventData: { resumeAt: waitCreated.eventData.resumeAt },
createdAt: new Date(),
});
}
// Inline delta: everything appended since the caller's cursor, this write
// included — the same page an `events.list` from that cursor would return
// right now. Any event type may be asked (see world-local), and the page
// is the same slice whichever event the write committed onto it.
const delta =
typeof params?.sinceCursor === 'string' && !options.withholdDelta
? {
events: events.slice(cursorPosition.get(params.sinceCursor) ?? 0),
cursor: nextCursor(),
hasMore: false,
}
: undefined;
// step_started returns a running step entity so executeStep proceeds to
// run the body and write step_completed.
return {
...(startedStepResult(runId, data, event) ?? { event }),
...delta,
};
}
if (data.eventType === 'step_started' && mode.type === 'inject-wait') {
mode = { type: 'normal' };
const waitCreated = events.find(
(candidate) => candidate.eventType === 'wait_created'
);
assert(waitCreated, 'expected wait_created before step');
events.push({
eventId: slotToEventId(++seq),
runId,
eventType: 'wait_completed',
specVersion: SPEC_VERSION_CURRENT,
correlationId: waitCreated.correlationId,
eventData: { resumeAt: waitCreated.eventData.resumeAt },
createdAt: new Date(),
});
}
// step_started returns a running step entity so executeStep proceeds to
// run the body and write step_completed.
return startedStepResult(runId, data, event) ?? { event };
);
let listCallCount = 0;
const eventsList = vi.fn(async () => {
listCallCount++;
// The cursor is positioned at what the read actually showed, so a stale
// read hands back a short log AND a cursor that still covers the events it
// withheld — a prefix, the way a replica behind on replication serves one.
const visible =
options.staleFirstReload && listCallCount === 2
? events.slice(0, -1)
: [...events];
return {
data: visible,
hasMore: false,
cursor: nextCursor(visible.length),
};
});
const queueSend = vi.fn(async () => ({ messageId: null }));
setWorld({
specVersion: SPEC_VERSION_CURRENT,
@@ -537,14 +689,10 @@ async function drive(
),
events: {
create: eventsCreate,
list: vi.fn(async () => ({
data: [...events],
hasMore: false,
cursor: 'cursor_retained',
})),
list: eventsList,
},
runs: { get: vi.fn(async () => run) },
queue: vi.fn(async () => ({ messageId: null })),
queue: queueSend,
getEncryptionKeyForRun: vi.fn(async () => undefined),
} as any);
@@ -555,6 +703,12 @@ async function drive(
return {
vmBuilds: createContextSpy.mock.calls.length,
durableLog: normalizeDurableLog(events),
listCalls: eventsList.mock.calls.length,
queueSends: queueSend.mock.calls.length,
createParams,
createdHook: createdEvents.some((e) => e.eventType === 'hook_created'),
/** What the log actually holds, which a conflicting create diverges from. */
committedTypes: events.map((e) => e.eventType),
output,
result:
output === undefined
@@ -1124,4 +1278,182 @@ describe('retained VM through the inline replay loop', () => {
expect(result).toBe(30);
expect(vmBuilds).toBe(1);
});
/**
* A hook's awaiter is settled by the event its own create commits, so the
* parked VM is one `await` away from continuing. These pin that it continues
* HERE — in the delivery that made the write — rather than through a queue
* message whose only job would be to read that event back and replay to the
* same point.
*
* The harness invokes the handler exactly once, so "the run completed" is
* also "no re-invocation was needed": the pre-change path returns a
* visibility timeout to the queue and leaves the run unfinished.
*/
describe('hook write continuation', () => {
it('resolves the awaiter in-process, off the hook write response', async () => {
const { result, listCalls, queueSends, createParams } = await drive(
'wrun_retained_hook_conflict',
hookConflictWorkflow
);
// The awaiter saw a clean registration and the step after it ran, all
// within this one delivery.
expect(result).toBe(10);
// The hook create asked for the delta that made that possible.
expect(
createParams.find((p) => p.eventType === 'hook_created')?.sinceCursor
).toEqual(expect.any(String));
// One list: the invocation's initial load. The hook write carried the
// log forward from there, so the continuation read nothing.
expect(listCalls).toBe(1);
// Nothing was enqueued: no re-invocation, and the run's only step ran
// inline after the awaiter had already settled.
expect(queueSends).toBe(0);
});
it('reads from its cursor and still continues when the World returns no delta', async () => {
// `sinceCursor` is optional by contract. Without a delta the log is
// short of the hook_created, so the continuation must load from the
// cursor before resuming — and must not resume over the stale log.
const { result, listCalls, queueSends } = await drive(
'wrun_retained_hook_conflict_no_delta',
hookConflictWorkflow,
{ type: 'normal' },
{ withholdDelta: true }
);
expect(result).toBe(10);
// The initial load plus the continuation's incremental read — still one
// list against the delivery round-trip and full replay it replaces.
expect(listCalls).toBeGreaterThan(1);
expect(queueSends).toBe(0);
});
it('hands the awaiter back to the queue under the kill switch', async () => {
// With retention off there is no parked VM to resume, so the awaiter
// falls back to the re-invocation this path has always used: the
// handler returns a visibility timeout and the run finishes on the
// next delivery, which this single-invocation harness never makes.
process.env.WORKFLOW_RETAINED_VM = '0';
const { result, createdHook } = await drive(
'wrun_retained_hook_conflict_off',
hookConflictWorkflow
);
expect(createdHook).toBe(true);
expect(result).toBeUndefined();
});
/**
* The other outcome of the same write. A create whose token is already
* claimed commits `hook_conflict`, which settles the hook's awaiters just
* as durably as a `hook_created` would — so it rides the same delta and
* takes the same in-process continuation, instead of the re-invocation
* this path used to answer every conflict with.
*/
describe('when the create commits hook_conflict', () => {
it('settles a getConflict() awaiter in-process, off the write response', async () => {
const { result, committedTypes, listCalls, queueSends, createParams } =
await drive(
'wrun_retained_taken_get_conflict',
conflictingGetConflictWorkflow,
{ type: 'normal' },
{ conflictToken: CONFLICTING_TOKEN }
);
// The log holds the conflict, not a creation — and the workflow both
// branched on it and ran the step after it, in this one delivery.
expect(committedTypes).toContain('hook_conflict');
expect(committedTypes).not.toContain('hook_created');
expect(result).toBe('conflict:10');
// The create asked for the delta on the way in; the conflict came
// back on it.
expect(
createParams.find((p) => p.eventType === 'hook_created')?.sinceCursor
).toEqual(expect.any(String));
// One list: the invocation's initial load. The conflicting write
// carried the log forward from there, so the continuation read
// nothing and nothing was enqueued.
expect(listCalls).toBe(1);
expect(queueSends).toBe(0);
});
it('settles a payload await in-process, with no getConflict() awaiter', async () => {
// The conflict shape the suspension reports as `hasHookConflict`
// alone: no `hook.getConflict()` is parked, so the awaited-creation
// branch never runs and the continuation is the conflict branch's
// own. The rejection is what advances the workflow.
const { result, listCalls, queueSends } = await drive(
'wrun_retained_taken_await',
conflictingAwaitWorkflow,
{ type: 'normal' },
{ conflictToken: CONFLICTING_TOKEN }
);
expect(result).toBe('rejected:10');
expect(listCalls).toBe(1);
expect(queueSends).toBe(0);
});
it('reads from its cursor and still continues when the World returns no delta', async () => {
// `sinceCursor` is optional by contract. Without a delta the log is
// short of the `hook_conflict`, so the continuation must load from the
// cursor before resuming — and must not resume over the stale log.
const { result, listCalls, queueSends } = await drive(
'wrun_retained_taken_no_delta',
conflictingAwaitWorkflow,
{ type: 'normal' },
{ conflictToken: CONFLICTING_TOKEN, withholdDelta: true }
);
expect(result).toBe('rejected:10');
expect(listCalls).toBeGreaterThan(1);
expect(queueSends).toBe(0);
});
it('hands the conflict back to the queue under the kill switch', async () => {
// With retention off there is no parked VM to resume, so a conflict
// falls back to the re-invocation it has always used.
process.env.WORKFLOW_RETAINED_VM = '0';
const { result, committedTypes } = await drive(
'wrun_retained_taken_off',
conflictingAwaitWorkflow,
{ type: 'normal' },
{ conflictToken: CONFLICTING_TOKEN }
);
expect(committedTypes).toContain('hook_conflict');
expect(result).toBeUndefined();
});
it('re-invokes instead of spinning when a repeat pass still cannot see the conflict', async () => {
// The repeat guard, which both continuation branches share. With no
// delta AND a read that has not caught up, the pass continues over a
// log that still does not hold the `hook_conflict`, so the same hook
// comes back wanting the same continuation. Continuing again could not
// change that, so the run goes to the queue — which is what keeps this
// bounded rather than a loop.
const { result, createParams } = await drive(
'wrun_retained_taken_stale_read',
conflictingAwaitWorkflow,
{ type: 'normal' },
{
conflictToken: CONFLICTING_TOKEN,
withholdDelta: true,
staleFirstReload: true,
}
);
// Handed back: the run finishes on a delivery this single-invocation
// harness never makes.
expect(result).toBeUndefined();
// Two create attempts for the hook — the first pass, and the one
// repeat the guard then refuses to continue.
expect(
createParams.filter((p) => p.eventType === 'hook_created')
).toHaveLength(2);
});
});
});
});
+182 -20
View File
@@ -70,7 +70,7 @@ import {
} from './runtime/deployment-guard.js';
import {
absorbSkippedSlotReport,
appendUniqueEvents,
appendEventLog,
getQueueOverhead,
getWorkflowQueueName,
handleHealthCheckMessage,
@@ -559,6 +559,23 @@ type RetentionDecision =
* this policy's precondition and could bind one ordinal to two logical branches
* before the step-ownership claim has a chance to arbitrate them.
*
* A hook-write continuation is the one boundary retained without a step or
* attribute driver. The suspension committed the event a hook's own awaiter is
* parked on the `hook_created` a `hook.getConflict()` waits for, or the
* `hook_conflict` a create whose token was already claimed committed instead
* and the caller advances the workflow over that event by resuming the session
* in this process rather than re-invoking (see `continueOverHookWrite` in the
* replay loop), so the runtime itself drives the next iteration. One arm for
* both outcomes because it is one boundary: same write, same event slot, same
* continuation. Steps in the same suspension ride along queued: an awaiter
* empties `lazyInlineSteps` and the conflict branch returns before any inline
* execution, so nothing this invocation does can order the continuation behind
* a step body. The hook this suspension just created is an open hook by
* definition, and is no more a hazard than any other open hook here: a
* `hook_received` landing out of band is absent from the log the resume reads
* exactly as it is absent from a fetch that returned a moment before it a
* prefix, never a hole, corrected on the next write.
*
* Quiescence assumes workflow code stays inside the sandbox's determinism
* contract. Escaping to the host realm (for example, recovering a host
* `Function` constructor to schedule real timers) already makes ordinary cold
@@ -568,9 +585,17 @@ type RetentionDecision =
function getRetentionDecision({
suspension,
serializationBlockerCount,
hookContinuation = false,
}: {
suspension: WorkflowSuspension;
serializationBlockerCount: number;
/**
* Whether this suspension committed the event a hook's own awaiter is
* waiting on and the caller will continue over it in-process, making the
* runtime the replay driver for a boundary that has no step or attribute
* write of its own. See the policy above.
*/
hookContinuation?: boolean;
}): RetentionDecision {
if (!isVmRetentionEnabled()) {
return { retain: false, reason: 'disabled' };
@@ -581,6 +606,9 @@ function getRetentionDecision({
reason: 'serialization_executed_workflow_code',
};
}
if (hookContinuation) {
return { retain: true };
}
if (suspension.stepCount === 0 && suspension.attributeCount === 0) {
return { retain: false, reason: 'no_replay_driver' };
}
@@ -627,11 +655,6 @@ function nextEventLogLoad(log: LoadedEventLog): ReplayEventLog {
};
}
function appendEventLog(log: LoadedEventLog, appended: LoadedEventLog): void {
appendUniqueEvents(log.events, appended.events);
log.cursor = appended.cursor ?? log.cursor;
}
/**
* Maximum inline-execution duration for a single handler invocation.
*
@@ -2730,6 +2753,24 @@ export function workflowEntrypoint(
// replays. Invocation-scoped: dies with this delivery.
let retainedSession: WorkflowSession | null = null;
// Hooks whose create this invocation has already answered by
// continuing in this process instead of re-invoking —
// whether the create committed the `hook_created` a
// `hook.getConflict()` was parked on or the `hook_conflict`
// a claimed token produced. One set, because a hook takes
// one of those outcomes and never both.
//
// Tracked by hook, not by count, because a repeat for the
// SAME hook is the only shape that cannot make progress: the
// continuation is resolved by an event the suspension
// already committed, so a pass that comes back asking for
// the same one ran over a log that still did not hold that
// event, and continuing again would spin. A hook that has
// not been seen here before is a pass that got somewhere, so
// a workflow creating one such hook after another keeps
// continuing in-process for each.
const continuedHookIds = new Set<string>();
// Main replay loop
while (true) {
loopIteration++;
@@ -3454,9 +3495,16 @@ export function workflowEntrypoint(
});
return;
}
// Open hooks/waits in the log as loaded for this
// replay. Computed lazily, at most once, for the
// delta/turbo gates below — the attr-detour and
// Open hooks/waits in the log this replay ran over,
// plus whatever the suspension's own writes folded back
// into it — so a `hook_created` this suspension
// committed and got a delta for IS in the scan, while
// one it wrote without a delta is not. Neither reading
// changes an outcome below: every gate that consults
// `openHook` also treats "this suspension created a
// hook" as equivalent. Computed lazily, at most once,
// and shared between the hook-write continuation and
// the delta/turbo gates below — the attr-detour and
// hook-conflict paths return/continue before the gates
// and usually avoid the scan entirely.
const openHookWait = once(() => {
@@ -3469,6 +3517,12 @@ export function workflowEntrypoint(
suspension: err,
serializationBlockerCount:
suspensionResult.serializationBlockerCount,
// This suspension committed the event a hook's
// own awaiter is parked on; the continuation
// below drives the next iteration in-process.
hookContinuation:
suspensionResult.hasAwaitedHookCreation ||
suspensionResult.hasHookConflict,
})
: undefined;
if (retentionDecision?.retain === false) {
@@ -3512,8 +3566,104 @@ export function workflowEntrypoint(
: {}),
});
// Hook conflict: break loop, re-invoke via queue
/**
* Advance the workflow HERE over the event a hook
* create just committed, instead of handing the run
* back to the queue for a delivery whose only job
* would be to read that event and replay to the same
* point.
*
* The parked VM is one `await` away from consuming it,
* so resuming it over the carried-forward log costs
* neither the queue hop nor the cold replay. Steps
* stay queued either way: this invocation runs none of
* them, so the continuation is not serialized behind a
* step body the property an awaiter emptying
* `lazyInlineSteps` exists to protect, and which the
* conflict path gets by returning before any inline
* execution at all.
*
* The suspension's writes carried the log forward only
* if the hook create's delta accounted for all of
* them; otherwise read from the cursor first, which is
* still one list against the delivery round-trip and
* full replay it replaces. An open wait also forces
* the read, for the reason the inline-delta gate below
* gives: a `wait_completed` is a resolution the replay
* is waiting on rather than an event it can observe an
* iteration late.
*
* False when this invocation cannot get anywhere that
* way no session to resume (retention off, or a
* boundary the predicate refused), or every hook here
* already had its continuation and still wants one, so
* the pass ran over a log that did not hold the event
* and repeating it would spin. The caller re-invokes.
*/
const continueOverHookWrite = (
hookIds: readonly string[],
settles: 'hook_conflict' | 'hook_created'
): boolean => {
if (!retainedSession) return false;
const fresh = hookIds.filter(
(id) => !continuedHookIds.has(id)
);
if (fresh.length === 0) return false;
for (const id of fresh) {
continuedHookIds.add(id);
}
const resumeWithoutRead =
suspensionResult.eventLogCarriedForward &&
!openHookWait.value.openWait;
if (!resumeWithoutRead) {
// Narrowing does not survive into this closure;
// the replay that raised this suspension ran over
// a ready log, same as `openHookWait` asserts.
assert(eventLog.type === 'ready');
eventLog = nextEventLogLoad(eventLog);
}
runtimeLogger.debug(
'Continuing over a hook write in-process',
{
workflowRunId: runId,
loopIteration,
settles,
hookIds: fresh,
carriedForward:
suspensionResult.eventLogCarriedForward,
readBeforeResume: !resumeWithoutRead,
}
);
span?.setAttributes({
'workflow.hook_write_continuations':
continuedHookIds.size,
});
return true;
};
// Hook conflict: the token was already claimed, so
// this run's hook was never created and the
// `hook_conflict` this suspension committed is what
// settles its awaiters — rejecting a payload await,
// resolving a `hook.getConflict()` with the
// conflicting run. The workflow must observe that
// before anything else this suspension scheduled runs,
// which is why this branch comes ahead of the attr
// detour and all step dispatch: a `Promise.race`
// between the hook and a step must let the durable
// conflict win without executing the losing step.
// Continue in this process when the boundary allows
// it; otherwise hand the run back for a fresh replay
// over the conflict.
if (suspensionResult.hasHookConflict) {
if (
continueOverHookWrite(
suspensionResult.hookConflictCorrelationIds,
'hook_conflict'
)
) {
continue;
}
return await reinvoke(0);
}
@@ -3606,14 +3756,15 @@ export function workflowEntrypoint(
// is present. That awaiter case must execute nothing
// inline: an inline `await executeStep(...)` blocks this
// handler for the full step duration, so the awaiter's
// continuation (which only advances on the next replay)
// would be serialized behind the step, defeating work
// continuation (which only advances on the next pass)
// would be serialized behind the step defeating work
// the workflow expressed as parallel (e.g.
// `hook.getConflict().then(() => stepB())` racing `await
// stepA()`). In that case `lazyInlineSteps` is empty and
// every step is queued for re-invocation, which replays
// over the just-committed hook_created and resolves the
// awaiter while queued steps run in parallel invocations.
// every step is queued, so the continuation below —
// which resumes the parked VM over the just-committed
// hook_created — races those queued steps rather than
// waiting on any of them.
const lazyInlineSteps =
suspensionResult.lazyInlineSteps;
const inlineCorrelationIds = new Set(
@@ -3984,13 +4135,24 @@ export function workflowEntrypoint(
dispatchesSettled,
suspensionResult.deferredBatchWork,
]);
// A `hook.getConflict()` awaiter needs an immediate
// re-invocation: the replay consumes the
// just-committed hook_created and resolves the
// awaiter. Without it (no inline step, all work
// queued or none pending) the run would sit idle
// A `hook.getConflict()` awaiter needs the workflow
// to continue: the `hook_created` this suspension
// just committed is what resolves it, and nothing
// else will — no step ran here, and every step this
// suspension scheduled went to the queue — so
// without a continuation the run would sit idle
// until some unrelated message woke it.
if (suspensionResult.hasAwaitedHookCreation) {
if (
continueOverHookWrite(
suspensionResult.awaitedHookCorrelationIds,
'hook_created'
)
) {
continue;
}
// Hand the run back for a fresh replay over the
// committed hook_created.
return await reinvoke(0);
}
return;
+17
View File
@@ -710,6 +710,23 @@ export interface LoadedEventLog {
cursor: string | null;
}
/**
* Extend a loaded log with a page that continues it a listed page, or the
* inline delta a write handed back and move its read position with it.
*
* The cursor is only advanced when the page carries one, so a source with no
* position of its own (a skipped-slot report) cannot walk the read position
* past events it did not carry. Appending does not re-sort; see
* {@link appendUniqueEvents} for why receipt order is the order to keep.
*/
export function appendEventLog(
log: LoadedEventLog,
appended: { events: readonly Event[]; cursor?: string | null }
): void {
appendUniqueEvents(log.events, appended.events);
log.cursor = appended.cursor ?? log.cursor;
}
/**
* Whether a replay refuses to run over a log with a hole in it (see
* {@link findEventSlotGap}). **On by default**; set
@@ -631,6 +631,402 @@ describe('handleSuspension', () => {
expect(maxEventSlot(eventLog.events)).toBe(1);
});
});
// The hook create asks the World for the event-log delta since the caller's
// cursor, so a caller holding the hook's awaiter can settle it off the
// response instead of re-invoking to read the event back — on whichever
// event the create commits.
describe('hook creation inline delta', () => {
function slotEvent(slot: number, eventType: Event['eventType']): Event {
return {
eventId: slotToEventId(slot),
eventType,
runId: run.runId,
createdAt: new Date(),
} as Event;
}
function awaitedHook(correlationId = 'hook_awaited') {
return [
correlationId,
{
type: 'hook' as const,
correlationId,
token: `tok-${correlationId}`,
hasConflictAwaiter: true,
},
] as const;
}
/**
* A World that answers `sinceCursor` with the write it just committed.
*
* `conflict` commits `hook_conflict` in place of a `hook_created`, the way
* a World does when another run already holds the token: same slot, same
* delta, a different event on it.
*/
function deltaWorld(startSlot = 2, { conflict = false } = {}) {
let slot = startSlot;
return vi.fn(async (_runId, event, params) => {
const substituted =
conflict && event.eventType === 'hook_created'
? {
...event,
eventType: 'hook_conflict',
eventData: {
token: event.eventData?.token,
conflictingRunId: 'wrun_token_owner',
},
}
: event;
const committed = {
...substituted,
eventId: slotToEventId(slot++),
} as Event;
if (typeof params?.sinceCursor !== 'string') {
return { event: committed };
}
return {
event: committed,
events: [committed],
cursor: `eid:${committed.eventId}`,
hasMore: false,
};
});
}
it('folds the created hook event into the caller log and says so', async () => {
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const eventsCreate = deltaWorld();
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([awaitedHook()]),
globalThis
),
world: createWorld(eventsCreate),
run,
eventLog,
});
expect(eventsCreate).toHaveBeenCalledWith(
run.runId,
expect.objectContaining({ eventType: 'hook_created' }),
expect.objectContaining({ sinceCursor: 'eid:cursor_1' })
);
// The log now holds the event that resolves the awaiter, and its read
// position moved with it — so a caller can replay/resume off it with no
// read of its own.
expect(eventLog.events.map((e) => e.eventType)).toEqual([
'run_started',
'hook_created',
]);
expect(eventLog.cursor).toBe(`eid:${slotToEventId(2)}`);
expect(result.eventLogCarriedForward).toBe(true);
expect(result.awaitedHookCorrelationIds).toEqual(['hook_awaited']);
expect(result.hookConflictCorrelationIds).toEqual([]);
// A delta is not a skipped-slot report: it extends the tail, so the
// caller's cached scan positions stay valid.
expect(result.reportedEventCount).toBe(0);
});
it('folds a committed hook_conflict into the caller log and says so', async () => {
// The conflict outcome of the same write. It is the event the hook's
// awaiters settle on, so it carries the log forward exactly as a
// `hook_created` does — and is reported by hook id, so a caller
// continuing in-process can tell a fresh pass from a repeating one.
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const eventsCreate = deltaWorld(2, { conflict: true });
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([awaitedHook('hook_taken')]),
globalThis
),
world: createWorld(eventsCreate),
run,
eventLog,
});
// Asked for on the way in, before the outcome was known — the request is
// still a `hook_created`.
expect(eventsCreate).toHaveBeenCalledWith(
run.runId,
expect.objectContaining({ eventType: 'hook_created' }),
expect.objectContaining({ sinceCursor: 'eid:cursor_1' })
);
expect(eventLog.events.map((e) => e.eventType)).toEqual([
'run_started',
'hook_conflict',
]);
expect(eventLog.cursor).toBe(`eid:${slotToEventId(2)}`);
expect(result.eventLogCarriedForward).toBe(true);
expect(result.hasHookConflict).toBe(true);
expect(result.hookConflictCorrelationIds).toEqual(['hook_taken']);
// A conflict means the hook was never created, so its `getConflict()`
// awaiter is settled by the conflict rather than by a creation — the
// caller takes the conflict branch, not the awaited-creation one.
expect(result.hasAwaitedHookCreation).toBe(false);
expect(result.awaitedHookCorrelationIds).toEqual([]);
});
it('does not carry the log forward on a conflict when a wait also wrote', async () => {
// Same accounting as the creation case: the `wait_created` lands above
// the delta the hook write returned, so the caller has to read before
// continuing over the conflict. Also pins that a conflict suppresses the
// wait timeout — the caller advances the workflow over the conflict
// before scheduling anything, and the pass after it reports the wait.
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([
awaitedHook('hook_taken'),
[
'w1',
{
type: 'wait' as const,
correlationId: 'w1',
resumeAt: new Date(Date.now() + 30_000),
},
],
]),
globalThis
),
world: createWorld(deltaWorld(2, { conflict: true })),
run,
eventLog,
});
expect(result.hookConflictCorrelationIds).toEqual(['hook_taken']);
expect(result.eventLogCarriedForward).toBe(false);
expect(result.waitTimeout).toBeUndefined();
});
it('defers a step on a conflict and still carries the log forward', async () => {
// A conflict with no `getConflict()` awaiter leaves `lazyInlineSteps`
// populated, so the step's `step_created` is deferred and the hook write
// is the suspension's only one — the log really is carried forward. The
// step is not stranded: the caller returns before dispatching it, and
// the pass that continues over the conflict schedules it.
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const plainHook = [
'hook_taken',
{
type: 'hook' as const,
correlationId: 'hook_taken',
token: 'tok-hook_taken',
},
] as const;
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([
plainHook,
[
's1',
{
type: 'step' as const,
correlationId: 's1',
stepName: 's1',
args: [],
},
],
]),
globalThis
),
world: createWorld(deltaWorld(2, { conflict: true })),
run,
eventLog,
});
expect(result.hookConflictCorrelationIds).toEqual(['hook_taken']);
expect(result.lazyInlineSteps.map((s) => s.correlationId)).toEqual([
's1',
]);
expect(result.createdStepCorrelationIds).not.toContain('s1');
expect(result.eventLogCarriedForward).toBe(true);
});
it('leaves the log alone on a conflict when the World returns no delta', async () => {
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const eventsCreate = vi.fn(async (_runId, event) => ({
event: {
...event,
eventType: 'hook_conflict',
eventId: slotToEventId(2),
},
}));
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([awaitedHook('hook_taken')]),
globalThis
),
world: createWorld(eventsCreate),
run,
eventLog,
});
expect(eventLog.events.map((e) => e.eventType)).toEqual(['run_started']);
expect(eventLog.cursor).toBe('eid:cursor_1');
expect(result.eventLogCarriedForward).toBe(false);
expect(result.hookConflictCorrelationIds).toEqual(['hook_taken']);
});
it('does not carry the log forward when a step also wrote', async () => {
// The step_created lands above the delta the hook write returned, so the
// log is short of it and the caller has to read before continuing.
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([
awaitedHook(),
[
's1',
{
type: 'step' as const,
correlationId: 's1',
stepName: 's1',
args: [],
},
],
]),
globalThis
),
world: createWorld(deltaWorld()),
run,
eventLog,
});
// An awaiter still means nothing runs inline, so the step keeps its
// eager step_created and is queued by the caller.
expect(result.lazyInlineSteps).toEqual([]);
expect(result.createdStepCorrelationIds).toContain('s1');
expect(result.hasAwaitedHookCreation).toBe(true);
expect(result.eventLogCarriedForward).toBe(false);
});
it('asks for no delta when the suspension creates two hooks', async () => {
// Both creates would diff against the same cursor and only one delta
// could be folded in, so the log would end up short of the other's event
// with nothing to say so.
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const eventsCreate = deltaWorld();
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([awaitedHook('hook_a'), awaitedHook('hook_b')]),
globalThis
),
world: createWorld(eventsCreate),
run,
eventLog,
});
for (const call of eventsCreate.mock.calls) {
expect(call[2]?.sinceCursor).toBeUndefined();
}
expect(result.eventLogCarriedForward).toBe(false);
expect([...result.awaitedHookCorrelationIds].sort()).toEqual([
'hook_a',
'hook_b',
]);
});
it('declines a truncated delta rather than moving the cursor past it', async () => {
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const eventsCreate = vi.fn(async (_runId, event) => ({
event: { ...event, eventId: slotToEventId(2) },
events: [slotEvent(2, 'hook_created')],
cursor: 'eid:cursor_2',
hasMore: true,
}));
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([awaitedHook()]),
globalThis
),
world: createWorld(eventsCreate),
run,
eventLog,
});
expect(eventLog.events.map((e) => e.eventType)).toEqual(['run_started']);
expect(eventLog.cursor).toBe('eid:cursor_1');
expect(result.eventLogCarriedForward).toBe(false);
});
it('leaves the log alone when the World returns no delta', async () => {
// Any World may ignore `sinceCursor`; the caller then reads instead.
const eventLog = {
events: [slotEvent(1, 'run_started')],
cursor: 'eid:cursor_1',
};
const eventsCreate = vi.fn(async (_runId, event) => ({
event: { ...event, eventId: slotToEventId(2) },
}));
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([awaitedHook()]),
globalThis
),
world: createWorld(eventsCreate),
run,
eventLog,
});
expect(eventLog.events.map((e) => e.eventType)).toEqual(['run_started']);
expect(eventLog.cursor).toBe('eid:cursor_1');
expect(result.eventLogCarriedForward).toBe(false);
expect(result.hasAwaitedHookCreation).toBe(true);
});
it('asks for no delta on a log with no cursor (turbo)', async () => {
const eventLog = { events: [], cursor: null };
const eventsCreate = deltaWorld(1);
const result = await handleSuspension({
suspension: new WorkflowSuspension(
new Map([awaitedHook()]),
globalThis
),
world: createWorld(eventsCreate),
run,
eventLog,
});
expect(eventsCreate.mock.calls[0][2]?.sinceCursor).toBeUndefined();
expect(result.eventLogCarriedForward).toBe(false);
});
});
});
describe('resilient step dispatch', () => {
+147 -18
View File
@@ -55,6 +55,7 @@ import {
} from './constants.js';
import {
absorbSkippedSlotReport,
appendEventLog,
type EventCreator,
type LoadedEventLog,
maxEventSlot,
@@ -81,6 +82,11 @@ export interface SuspensionHandlerParams {
* seeded sequence, so re-committing it against a corrected log would persist
* an event no correct replay produces. The caller restarts the replay
* instead.
*
* Extended in place by what those writes report back the events on slots
* they skipped over, and (on the hook create) the inline delta since its
* cursor. Whether the log is complete afterwards is answered by
* {@link SuspensionHandlerResult.eventLogCarriedForward}.
*/
eventLog?: LoadedEventLog;
/**
@@ -267,10 +273,58 @@ export interface SuspensionHandlerResult {
* pending wait collapse into a single delayed continuation.
*/
waitTimeout?: { seconds: number; correlationId: string };
/** Whether a hook conflict was detected (should re-invoke immediately) */
/**
* Whether a hook create committed a `hook_conflict` the token was already
* claimed, so this run's hook was never created and the workflow must
* observe the conflict before anything else this suspension scheduled runs.
* The caller answers it by advancing the workflow over the committed event.
*/
hasHookConflict: boolean;
/** Whether a `hook.getConflict()` awaiter needs the workflow to continue immediately */
hasAwaitedHookCreation: boolean;
/**
* Correlation ids of the hooks this suspension committed a `hook_created`
* for while a `hook.getConflict()` awaiter was waiting on it the events
* that resolve those awaiters on the next pass. Empty exactly when
* {@link hasAwaitedHookCreation} is false.
*
* Reported by id, not just counted, so a caller that continues in this
* process can tell a continuation that got somewhere from one that is
* repeating: a workflow may create one awaited hook after another, and each
* new id is a pass that made progress, while the same id coming back means
* the pass ran over a log that still did not hold its event and continuing
* again cannot change that.
*/
awaitedHookCorrelationIds: string[];
/**
* Correlation ids of the hooks whose create committed a `hook_conflict`.
* Empty exactly when {@link hasHookConflict} is false.
*
* By id for the same reason as {@link awaitedHookCorrelationIds}, and
* against the same hazard: a conflict is resolved by the `hook_conflict`
* this suspension committed, and until the workflow observes it the hook
* stays in the invocations queue and the next pass writes the create again.
* A fresh id is progress; the same id coming back is a pass that ran over a
* log which still did not hold the event, so continuing again cannot change
* that.
*/
hookConflictCorrelationIds: string[];
/**
* Whether the caller's `eventLog` now holds every event this suspension
* committed, so a caller that continues in this process can replay or
* resume a retained VM straight off it with no read.
*
* True only when the hook create's inline delta came back complete and was
* folded in (see `hookDeltaCursor` below), and nothing else in this
* suspension wrote an event. False whenever a read is needed first: no
* delta was asked for or returned, it was truncated, or a step / wait /
* attribute / abort write landed above it and is therefore not in it.
*
* Indifferent to which event the create committed: the delta is the slice
* of the log after the caller's cursor either way, so it carries a
* `hook_conflict` exactly as it carries a `hook_created`.
*/
eventLogCarriedForward: boolean;
/** Whether native workflow attribute events were written for replay. */
hasAttributeEvents: boolean;
/**
@@ -304,12 +358,19 @@ async function createHookEvent({
hookEvent,
queueItem,
requestId,
sinceCursor,
createEvent,
}: {
runId: string;
hookEvent: CreateEventRequest;
queueItem: HookInvocationQueueItem;
requestId?: string;
/**
* Cursor to ask the World for the event-log delta against, or undefined to
* not ask. See `hookDeltaCursor` in {@link handleSuspension} for when it is
* set and why it is at most one write per suspension.
*/
sinceCursor?: string;
createEvent: (
data: CreateEventRequest,
params?: CreateEventParams
@@ -321,11 +382,15 @@ async function createHookEvent({
try {
const result = await createEvent(hookEvent, {
requestId,
...(sinceCursor === undefined ? {} : { sinceCursor }),
});
// Check if the world returned a hook_conflict event instead of hook_created.
// The hook_conflict event is stored in the event log and will be replayed
// on the next workflow invocation, causing the hook's promise to reject.
// The hook_conflict event is stored in the event log and is what the next
// pass consumes to settle the hook's awaiters — rejecting a payload await,
// resolving a `hook.getConflict()` with the conflicting run. An inline
// delta asked for above carries it just as it would have carried the
// hook_created, so the caller can advance over it without a re-invocation.
if (result.event?.eventType === 'hook_conflict') {
return {
hasHookConflict: true,
@@ -460,7 +525,15 @@ export async function handleSuspension({
// sequence, so re-committing it against a corrected log would persist an
// event no correct replay produces.
let reportedEvents = 0;
// Writes this suspension issued, and whether one of them handed back a
// complete inline delta that was folded into the caller's log. Together they
// answer `eventLogCarriedForward`: the delta covers the log up to the write
// that returned it, so it accounts for every event this suspension committed
// only if that write was the only one.
let guardedWrites = 0;
let deltaAbsorbed = false;
const createGuarded: EventCreator = async (data, params) => {
guardedWrites++;
if (!eventLog) {
return createEvent(data, params);
}
@@ -469,6 +542,29 @@ export async function handleSuspension({
...params,
...slotSnapshotParams(log.events),
});
// An inline delta this call asked for (`sinceCursor`) is everything the
// log gained since that cursor, this write included, so it is folded onto
// the tail and carries the cursor with it — unlike a skipped-slot report,
// which is a window strictly below the write and has to be sorted back
// into place. A World returns one or the other, never both (the delta is a
// strict superset), so the two are handled apart rather than merged.
//
// Declining is always safe — an unabsorbed delta is one the next read
// returns — so the guards match the replay loop's `absorbCreateDelta`: a
// truncated page (`hasMore`) is dropped whole rather than advancing the
// cursor past events it did not carry, and the log must still be where the
// request was computed from, since appending does not re-sort.
if (typeof params?.sinceCursor === 'string') {
if (
log.cursor === params.sinceCursor &&
result.events !== undefined &&
result.hasMore !== true
) {
appendEventLog(log, { events: result.events, cursor: result.cursor });
deltaAbsorbed = true;
}
return result;
}
// Bump-and-report: the write landed above the slot it asked for, so the
// report holds the events it was decided without. Absorbing here rather
// than at each call site means the rest of this phase's writes (which read
@@ -618,12 +714,32 @@ export async function handleSuspension({
}
// Process hooks first to prevent race conditions with webhook receivers.
// Track any hook conflicts that occur: these are returned to the caller
// so the V2 handler can re-invoke immediately.
let hasHookConflict = false;
let hasAwaitedHookCreation = false;
// Track any hook conflicts that occur these are returned to the caller so
// it can advance the workflow over the committed `hook_conflict` before
// anything else this suspension scheduled runs.
const hookConflictCorrelationIds: string[] = [];
const awaitedHookCorrelationIds: string[] = [];
let hookCreationMs = 0;
// Ask the hook create for the event-log delta since the cursor the caller's
// log was read at. The hook's awaiters are settled by the event this write
// commits and by nothing else — a `hook_created` for a clean registration,
// a `hook_conflict` when the token was already claimed — so the caller can
// continue the workflow in its own process on either outcome, but only over
// a log that holds that event, and this write is the one request that can
// hand it back together with anything another writer landed in the meantime.
// Optional by contract: a World that ignores `sinceCursor` returns no delta
// and the caller reads instead.
//
// Asked for on the single-hook suspension only. Two creates issued from one
// snapshot each diff against the same cursor, and only the first delta back
// can be folded in (the cursor moves with it), so the log would end up short
// of the other's event with nothing to say so.
const hookDeltaCursor =
hooksNeedingCreation.length === 1 && typeof eventLog?.cursor === 'string'
? eventLog.cursor
: undefined;
if (hookItemsByToken.size > 0) {
const hookPhaseStart = Date.now();
await ensureRunReady();
@@ -657,10 +773,15 @@ export async function handleSuspension({
hookEvent,
queueItem,
requestId,
sinceCursor: hookDeltaCursor,
createEvent: createGuarded,
});
hasHookConflict ||= result.hasHookConflict;
hasAwaitedHookCreation ||= result.hasAwaitedHookCreation;
if (result.hasHookConflict) {
hookConflictCorrelationIds.push(queueItem.correlationId);
}
if (result.hasAwaitedHookCreation) {
awaitedHookCorrelationIds.push(queueItem.correlationId);
}
creationConflicted = result.hasHookConflict;
}
@@ -902,12 +1023,12 @@ export async function handleSuspension({
// step is created on the fly by the lazy `step_started` executeStep sends
// (saving a round-trip per step). We never defer when a `hook.getConflict()`
// awaiter is present, because in that case the caller executes nothing inline
// (it re-invokes immediately to resolve the awaiter), so deferring would
// leave the steps uncreated and unqueued. We pick the first N uncreated
// steps (matching the caller's inline-candidate selection) and dehydrate
// (it continues the workflow to resolve the awaiter instead), so deferring
// would leave the steps uncreated and unqueued. We pick the first N uncreated
// steps matching the caller's inline-candidate selection and dehydrate
// their input here so executeStep can ship it as the step_started payload.
const lazyInlineCorrelationIds = new Set<string>(
hasAwaitedHookCreation === false
awaitedHookCorrelationIds.length === 0
? stepItems
.filter((item) => stepsNeedingCreation.has(item.correlationId))
.slice(0, getMaxInlineSteps())
@@ -1838,11 +1959,19 @@ export async function handleSuspension({
inlineClaims,
batchCommittedSlotCeiling,
deferredBatchWork,
// On hook conflict the caller re-invokes immediately and never reads
// the wait timeout, so don't report one.
waitTimeout: hasHookConflict ? undefined : soonestWait,
hasHookConflict,
hasAwaitedHookCreation,
// On hook conflict the caller advances the workflow over the conflict
// before scheduling anything and never reads the wait timeout, so don't
// report one. The next pass, which sees the conflict settled, reports it.
waitTimeout:
hookConflictCorrelationIds.length > 0 ? undefined : soonestWait,
hasHookConflict: hookConflictCorrelationIds.length > 0,
hookConflictCorrelationIds,
hasAwaitedHookCreation: awaitedHookCorrelationIds.length > 0,
awaitedHookCorrelationIds,
// The delta accounts for the whole log only if the write that returned it
// was this suspension's only one — anything written after it landed above
// the delta and is missing from the caller's log.
eventLogCarriedForward: deltaAbsorbed && guardedWrites === 1,
hasAttributeEvents: attributeItems.length > 0,
hasHookEvents: hooksNeedingCreation.length > 0,
hookCreationMs,
+65
View File
@@ -55,9 +55,22 @@ function setupWorkflowContext(events: Event[]): WorkflowOrchestratorContext {
onWorkflowError: vi.fn(),
promiseQueue: Promise.resolve(),
pendingDeliveries: 0,
suspensionGeneration: 0,
};
}
/**
* Let the idle poll behind `scheduleWhenIdle` make progress. Each poll is one
* `setTimeout(0)` turn (plus a `promiseQueue` hop while a delivery is held),
* so a handful of explicit macrotask turns covers arming, re-polling against
* a held delivery, and firing once it is released, without a wall-clock wait.
*/
async function settleTimers(turns = 4): Promise<void> {
for (let i = 0; i < turns; i++) {
await new Promise<void>((resolve) => setTimeout(resolve, 0));
}
}
describe('createCreateHook', () => {
it('should resolve with payload when hook_received event is received', async () => {
const ops: Promise<any>[] = [];
@@ -348,6 +361,58 @@ describe('createCreateHook', () => {
}
});
// The hook consumer's suspension signal carries the generation guard (see
// `scheduleWorkflowSuspension`), which is what lets the runtime resume a
// retained VM over the `hook_created` (or `hook_conflict`) it just committed
// instead of re-invoking. Without it, a signal armed at the boundary the
// resume moved past would raise a suspension the workflow never reached —
// carrying none of the work the resume kicked off, leaving the run dormant.
it('drops a suspension signal armed for a boundary the run has moved past', async () => {
const ctx = setupWorkflowContext([]);
const errors: Error[] = [];
ctx.onWorkflowError = (error) => {
errors.push(error);
};
// Hold the idle gate so the signal is armed but cannot fire yet — the
// window a resume lands in.
ctx.pendingDeliveries = 1;
const hook = createCreateHook(ctx)({ token: 'stale-signal' });
void hook.getConflict();
await settleTimers();
expect(errors).toHaveLength(0);
// What the runtime does when it resumes the parked VM.
ctx.suspensionGeneration++;
ctx.pendingDeliveries = 0;
await settleTimers();
expect(errors).toHaveLength(0);
});
it('still signals when the run has not moved past the boundary', async () => {
// The control for the test above: the same held-then-released signal
// reaches the runtime when the generation has not moved, so the guard
// cannot be swallowing signals that are still wanted.
const ctx = setupWorkflowContext([]);
const errors: Error[] = [];
ctx.onWorkflowError = (error) => {
errors.push(error);
};
ctx.pendingDeliveries = 1;
const hook = createCreateHook(ctx)({ token: 'live-signal' });
void hook.getConflict();
await settleTimers();
expect(errors).toHaveLength(0);
ctx.pendingDeliveries = 0;
await settleTimers();
expect(errors).toHaveLength(1);
expect(errors[0]).toBeInstanceOf(WorkflowSuspension);
});
it('should resolve getConflict with the conflicting run when hook_conflict event is received', async () => {
const ctx = setupWorkflowContext([
{
+56
View File
@@ -1658,6 +1658,62 @@ describe('Storage', () => {
).toBe(true);
expect(result.hasMore).toBe(false);
});
it('returns a delta for a create that committed hook_conflict', async () => {
await updateRun(storage, testRunId, 'run_started');
await storage.events.create(testRunId, {
eventType: 'hook_created' as const,
correlationId: 'corr_hook_owner',
eventData: { token: 'delta-conflict-token' },
});
const sinceCursor = await currentCursor();
// A create whose token is taken commits `hook_conflict` and returns
// early, ahead of the shared delta block — but the conflict is the
// event the create's awaiters settle on, so the caller has to get it
// here or pay a re-invocation to read back an event this response
// already held.
const result = await storage.events.create(
testRunId,
{
eventType: 'hook_created' as const,
correlationId: 'corr_hook_loser',
eventData: { token: 'delta-conflict-token' },
},
{ sinceCursor }
);
expect(result.event?.eventType).toBe('hook_conflict');
const expected = await storage.events.list({
runId: testRunId,
pagination: { sortOrder: 'asc', cursor: sinceCursor },
});
expect(result.events?.map((e) => e.eventId)).toEqual(
expected.data.map((e) => e.eventId)
);
expect(result.events?.at(-1)?.eventType).toBe('hook_conflict');
expect(result.cursor).toBe(expected.cursor);
expect(result.hasMore).toBe(expected.hasMore);
});
it('does not return a delta on a hook_conflict when sinceCursor is omitted', async () => {
await updateRun(storage, testRunId, 'run_started');
await storage.events.create(testRunId, {
eventType: 'hook_created' as const,
correlationId: 'corr_hook_owner2',
eventData: { token: 'no-delta-conflict-token' },
});
const result = await storage.events.create(testRunId, {
eventType: 'hook_created' as const,
correlationId: 'corr_hook_loser2',
eventData: { token: 'no-delta-conflict-token' },
});
expect(result.event?.eventType).toBe('hook_conflict');
expect(result.events).toBeUndefined();
expect(result.cursor).toBeUndefined();
});
});
describe('list', () => {
@@ -2397,12 +2397,39 @@ export function createEventsStorage(
const storedConflict = await storeEvent(conflictEvent);
const resolveData =
params?.resolveData ?? DEFAULT_RESOLVE_DATA_OPTION;
return {
// Inline delta, when the writer asked for one. This return is
// ahead of the shared `sinceCursor` block below, and the conflict
// it carries is the event the create's awaiters settle on — so a
// caller that asked has to get the delta here too, or it pays a
// re-invocation to read back an event this response could have
// handed it. Same single-page-or-fallback contract as below: a
// truncated page comes back with `hasMore` and the SDK falls back
// to `events.list`.
const conflictDelta =
typeof params?.sinceCursor === 'string'
? await queryRunEvents(effectiveRunId, {
sortOrder: 'asc',
cursor: params.sinceCursor,
})
: undefined;
const conflictResult = {
event: stripEventDataRefs(storedConflict, resolveData),
run,
step,
hook: undefined,
};
if (!conflictDelta) return conflictResult;
return {
...conflictResult,
events:
resolveData === 'none'
? conflictDelta.data.map((delta) =>
stripEventDataRefs(delta, resolveData)
)
: conflictDelta.data,
cursor: conflictDelta.cursor,
hasMore: conflictDelta.hasMore,
};
}
// Defer the Hook entity write until the event publish succeeds. A
+47
View File
@@ -266,6 +266,53 @@ describe('sim store', () => {
expect(store.hookByToken('approval:1')?.hookId).toBe('hook_1');
});
it('answers the sinceCursor delta on a conflicting create', async () => {
// The conflict is the event the create's awaiters settle on, so a
// caller that asked for the delta gets it here for the same reason it
// gets one on a clean `hook_created`: it continues over the event in its
// own process instead of re-invoking to read it back. Withholding it
// would cost a delivery on exactly the path that asked to avoid one.
await store.events.create(RUN, hookCreated('hook_1', 'approval:1'));
const before = await store.events.list({
runId: RUN,
pagination: { sortOrder: 'asc' },
});
const conflict = await store.events.create(
RUN,
hookCreated('hook_2', 'approval:1'),
{ sinceCursor: before.cursor ?? undefined }
);
expect(conflict.event?.eventType).toBe('hook_conflict');
expect(conflict.events?.map((e) => e.eventType)).toEqual([
'hook_conflict',
]);
expect(conflict.hasMore).toBe(false);
// Byte-identical to the page a list from the same cursor returns now.
const after = await store.events.list({
runId: RUN,
pagination: { sortOrder: 'asc', cursor: before.cursor ?? undefined },
});
expect(conflict.events?.map((e) => e.eventId)).toEqual(
after.data.map((e) => e.eventId)
);
expect(conflict.cursor).toBe(after.cursor);
});
it('omits the delta on a conflicting create that did not ask', async () => {
await store.events.create(RUN, hookCreated('hook_1', 'approval:1'));
const conflict = await store.events.create(
RUN,
hookCreated('hook_2', 'approval:1')
);
expect(conflict.event?.eventType).toBe('hook_conflict');
expect(conflict.events).toBeUndefined();
expect(conflict.cursor).toBeUndefined();
expect(conflict.hasMore).toBeUndefined();
});
it('releases the token on dispose and refuses later resumes', async () => {
await store.events.create(RUN, hookCreated('hook_1', 'approval:1'));
await store.events.create(RUN, {
+52 -15
View File
@@ -911,6 +911,44 @@ export function createSimStore(options: SimStoreOptions): SimStore {
delete (event as Record<string, unknown>).eventData;
}
/**
* The optional `sinceCursor` inline delta: everything appended strictly
* after the caller's cursor, this write included.
*
* Answered only for the writes the Vercel World computes one for a
* step-terminal event (the inline sequential loop) and a hook create (the
* hook's own awaited continuation) rather than for every type, so a
* scenario sees the same delta-or-fall-back split the backend actually
* produces. Keyed on the REQUESTED type, which is what makes the delta
* ride along on a create that commits `hook_conflict` instead of
* `hook_created`: the same awaiter is settled either way, so the caller
* continues off either event.
*
* `undefined` when the caller did not ask, or asked on a write that does
* not answer.
*/
function sinceCursorDelta():
| { events: Event[]; cursor: string | null; hasMore: boolean }
| undefined {
if (typeof params?.sinceCursor !== 'string') return undefined;
if (
!isTerminalStepEventType(data.eventType) &&
data.eventType !== 'hook_created'
) {
return undefined;
}
const page = paginate(applyWithhold(eventsForRun(runId)), {
pagination: { cursor: params.sinceCursor, sortOrder: 'asc' },
getCreatedAt: (e) => e.createdAt,
getId: (e) => e.eventId,
});
return {
events: page.data.map((e) => stripEventDataRefs(e, resolveData)),
cursor: page.cursor,
hasMore: page.hasMore,
};
}
// ---- Per-event-type validation ----------------------------------------
// Everything the write path *refuses*. What it does to the entity rows is
// `applyEvent` below: the same fold the seed path runs.
@@ -961,10 +999,18 @@ export function createSimStore(options: SimStoreOptions): SimStore {
conflictingRunId: hooks.get(owner)?.runId,
},
} as Event);
return {
// The conflict answers the inline delta the same way the
// `hook_created` below it would: it is the event the create's
// awaiters settle on, so a caller that asked can continue over it
// in its own process instead of re-invoking to read it back. This
// return is ahead of the shared delta block at the end of the
// write, so it computes its own.
const delta = sinceCursorDelta();
const conflictResult = {
event: stripEventDataRefs(clone(conflict), resolveData),
run: currentRun ? clone(currentRun) : undefined,
};
return delta ? { ...conflictResult, ...delta } : conflictResult;
}
if (hooks.has(data.correlationId)) {
throw new EntityConflictError(
@@ -1099,20 +1145,11 @@ export function createSimStore(options: SimStoreOptions): SimStore {
cursor: page.cursor,
hasMore: page.hasMore,
};
} else if (
isTerminalStepEventType(data.eventType) &&
typeof params?.sinceCursor === 'string'
) {
const page = paginate(applyWithhold(eventsForRun(runId)), {
pagination: { cursor: params.sinceCursor, sortOrder: 'asc' },
getCreatedAt: (e) => e.createdAt,
getId: (e) => e.eventId,
});
deltaPage = {
events: page.data.map((e) => stripEventDataRefs(e, resolveData)),
cursor: page.cursor,
hasMore: page.hasMore,
};
} else {
// See `sinceCursorDelta` above for which writes answer one; a create
// that committed `hook_conflict` returned before reaching here and
// computed its own.
deltaPage = sinceCursorDelta();
}
const result = {
+26 -1
View File
@@ -802,6 +802,25 @@ export interface CreateEventParams {
* point there is to keep the first invocation's writes as cheap as
* possible, and it has no loaded log to extend.
*
* The suspension handler sets it too, on the hook create of a single-hook
* suspension. That write is the whole continuation for the hook's own
* awaiter the event it commits is what settles it so a delta lets the
* runtime advance the workflow in the same process instead of enqueueing a
* message whose only job is to read back the event it just wrote. It is
* asked for on one hook create per suspension because two creates issued
* from the same cursor each diff against it, and only one of the returned
* deltas can be folded into the log.
*
* A World that answers it on `hook_created` MUST answer it on the
* `hook_conflict` a create whose token is already claimed commits instead.
* That event settles the same awaiter a payload await rejects, a
* `hook.getConflict()` resolves with the conflicting run and the runtime
* continues over it in-process just the same, so withholding the delta
* there would silently cost a delivery on exactly the path the caller
* asked to avoid one on. The delta is keyed on the requested event type,
* not the committed one; there is nothing extra to compute, since it is the
* same slice of the log either way.
*
* The cursor MUST share `events.list` semantics: the returned `events`
* are everything sorted strictly after `sinceCursor`, `cursor` is the
* position past the last returned event, and `hasMore` indicates a
@@ -911,7 +930,7 @@ export type EventResult<T extends EventType = EventType> = {
} & (
| {
/**
* Events with data resolved. Four producers populate this:
* Events with data resolved. Five producers populate this:
*
* - On a `run_started` response: all events up to this point, so the
* runtime can skip the initial `events.list` call and reduce TTFB.
@@ -919,6 +938,12 @@ export type EventResult<T extends EventType = EventType> = {
* the caller passed {@link CreateEventParams.sinceCursor}: the delta
* of events written strictly after that cursor, so the inline loop
* can skip the per-step incremental `events.list` round-trip.
* - On a hook-create write when the caller passed
* {@link CreateEventParams.sinceCursor}: the same delta, which
* includes the event the create committed the `hook_created`, or
* the `hook_conflict` of an already-claimed token so the hook's
* awaiter can be settled in the writing process rather than by a
* re-invocation that reads the event back.
* - On a `hook_received` response when the caller passed
* {@link CreateEventParams.preloadEvents}: the run's current replay
* log through the canonical `hook_received`, so the lazy hook queue