feat(ios): drive ASWebAuthenticationSession sign-in sheets in place (#2438) (#2448)

* 0.21.1

* feat(ios): drive ASWebAuthenticationSession sign-in sheets in place (#2438)

iOS apps that sign in via ASWebAuthenticationSession present the identity
provider in com.apple.SafariViewService, out of the app's process. Two facts,
both verified live on the iOS 26.2 Simulator, made these flows unautomatable:
activating or launching the host cancels the auth session, and the host AX
bridge cannot see the sheet because the app stays the AX primaryApp.

Serve and drive the sheet in place. A closed registry names the host (shared by
the TypeScript and Swift sides under a parity test); the runner reads and drives
it without activation and never adopts it as the session target; and the
Simulator route detects a running host with a cheap device-scoped ps probe and
takes the runner path, since the bridge would serve the occluded app tree as if
healthy. open refuses to launch a registered host, and captures carry a
system-surface disclosure.

Presence is foreground state, not tree content: a torn-down host serves a richer
tree than a live one, so content heuristics cannot tell them apart. The
never-activate guard is what keeps the foreground predicate sound, which also
makes the stale-tree failure mode unrepresentable for this flow.

Closes #2438

* chore(gates): register contracts/ios-system-surface in the export snapshot

* fix(ios): close the system-surface correctness gaps from review

Presence probe: absence and probe failure are no longer reported as "no
surface". The probe returns present/absent/unknown and the route takes the
runner for anything but a proven absent, so a sheet opened between two captures,
or a probe that cannot answer, can no longer fall through to a bridge capture
that would answer confidently from the occluded app tree. Only a positive
observation is memoized. The probe now matches with pgrep and reads only a
matched pid's environment, which is ~3x cheaper than the previous full
process-environment dump and stops copying every process's environment.

Open guard: the refusal moved to every resolved-host launch and terminate, so
the URL, deep-link and launch-args branches that returned before the old check
can no longer launch the host. Terminating a host is refused too, since that
cancels the presented session just as launching it does.

Comparison: the surface identity now reaches SnapshotState, and tap-failure
corroboration refuses outright when a baseline and a post-action capture
disagree about it, instead of letting app and sheet captures meet in legacy
same-presentation matching. Selector routes disclose an iOS system surface
through the shared disclosure seam rather than reading only the Android field.

The contracts import in the launch path is deferred so the app-lifecycle
facade's eager closure stays flat, and the runner's comment prose is trimmed
because apple/runner ships to npm as uncompiled source.

* fix(ios): route the system-surface probe through the Apple tool provider

The probe shelled out with runCmd, so every eligible capture spawned a real
process even in provider-backed tests that stub the Apple tool seam — 17 real
spawns in one scenario file, which is both wasted work and added latency on
timing-sensitive settle paths. It now goes through runAppleToolCommand like the
sibling ps probe, so a stubbed provider answers instead of spawning.

* fix(ios): disclose a skipped bridge when the surface probe cannot answer

Routing an unprovable probe to the runner is right, but the early return also
skipped runFallback, so the response lost its warning and kept an identity that
could still be compared against a bridge publication. An unknown probe now falls
back through the same disclosed path as a bridge failure, with its own reason.

* fix(ios): keep surface identity through comparison, find, and probe scope

A ps read that carries no SIMULATOR_UDID at all was reported as absence, so an
unreadable or truncated environment could route a live sheet to the occluded app
tree. Only a scope naming a different device is a real negative now; a missing
one stays unknown.

The shared post-gesture comparison token used comparisonKey or the backend
alone, so an app capture and a sheet capture — both XCTest — compared equal and
a sheet appearing or dismissing read as a stable surface. The token now carries
the surface, which covers stabilization, verify and settle through the one path
they share.

Mutating find rebuilt its capture without iosSystemSurfaceBundleId, so the
shared disclosure helper could not report the sheet on either outcome. It is
preserved now.

Each fix has a regression that fails without it.

* fix(ios): keep surface identity in verify and settle comparisons

`--verify` compared node digests and `--settle` diffed node-only baselines, so an app
baseline and an in-place system-surface capture (a web sign-in sheet) were treated as one
presentation: a meaningless changed verdict, and a whole-surface replacement presented as an
in-surface diff with refs.

The pre-action baseline now travels with the surface its capture described, from the resolution
and the session frame through to the settled capture, and one module owns the comparison for
both routes. Across a surface change no same-surface claim is made: evidence reports the
transition instead of a digest comparison, the settled diff and its refs are withheld, and both
payloads disclose the transition.

* refactor(test): move the cross-surface settle tests onto their source mirror

The #2438 cross-surface cases were appended to `settle.test.ts`, taking it over the
test-file size ratchet (2528 lines, 2359 at the merge-base). They assert the
comparison `post-action-surface.ts` owns, so they move to that module's mirror test
file, and the device double plus the trees both files drive move to a sibling
fixtures module under `__tests__/` rather than being duplicated.

Pure move: every test and every assertion is unchanged, and `settle.test.ts` is back
under its merge-base length.

* test(daemon): cover the cross-surface settle refusal on the generic route

`scroll --settle` and `back --settle` plumb the baseline's surface identity
through `baselineSurfaceBundleId`, but nothing asserted it: the generic route
had zero coverage of the #2438 refusal, so a regression there would have been
silent while the element-targeted route stayed green.

Assert the same contract the targeted route guarantees, in both directions and
for both commands: no diff is attached across an app/sheet boundary — therefore
no tail and no `refsGeneration` — the transition is disclosed, and the settle
observation still reports its own verdict alongside that disclosure.

Each direction falsifies a different half of the plumbing, so both are needed:
dropping the baseline's surface identity fails only the sheet-to-app tests (an
app baseline has no surface id to lose), and dropping the settled capture's
fails only the app-to-sheet tests. No production change: the plumbing was
correct, only untested.

* refactor(ios): inline the single-caller surface disclosure wrapper

iosSystemSurfaceDisclosure() only mapped provenance-or-nothing onto the shared
constant for one caller, so the caller now reads the constant directly and the
wrapper is gone. Its test becomes a test of the transition disclosure, which is
the function that still earns its place (the "sheet is gone" sentence).

readAppleSnapshotResult also called readSystemSurfaceProvenance twice inside one
spread; it is bound to a local and read once.

* docs(adr): state that a presented surface outranks a requested bundle id

prepareActiveCommandContext checks for a presented system surface before it
resolves or activates command.appBundleId, so a command naming a different app is
still served the sheet. That is intended, but the code does not read that way;
the amendment now says it plainly.

* refactor(ios): carry the system surface in the capture's comparison lineage

A capture of an in-place system surface (a web sign-in sheet) describes a
different presentation than a capture of the app, so it must never compare
equal to one. The `present` branch of the iOS snapshot route returned a bare
fallback, so that capture carried no comparison identity at all, and two
comparison sites hand-rolled the distinction from `iosSystemSurfaceBundleId`
instead.

The probe now reports which host it matched, and the `present` branch goes
through `runFallback` like the `unknown` branch beside it, lineaged to
`<device>:<host bundle>`. The comparison key then differs from an app
capture's by construction, so the surface branch in `hasMatchingPresentation`
and the surface concatenation in `snapshotComparisonKey` are gone: both sites
are plain key equality again, and neither knows that system surfaces exist.
Two captures of the same surface still share a lineage, so they stay
comparable with each other.

A presented surface is not a bridge failure, so it gets its own warning
wording: the bridge is inapplicable here, not unavailable.

* refactor(interaction): carry the pre-action baseline as one surface-scoped value

The same pre-action tree travelled as a flattened nodes/surface pair at every
boundary, and each boundary rebuilt it with a conditional spread. Carry
SurfaceScopedNodes itself instead:

- ResolvedInteractionTarget gets preAction?: SurfaceScopedNodes, replacing the
  preActionNodes/preActionSurfaceBundleId pair and the PreActionBaselineFields
  intersection on all three arms of the union.
- SettleObservationCommandOptions gets baseline: SurfaceScopedNodes, replacing
  baselineNodes/baselineSurfaceBundleId.
- RefResolution carries tree: SurfaceScopedNodes instead of nodes plus a loose
  surfaceBundleId.

That retires preActionBaselineFields(), preActionBaseline(), evidenceBaseline(),
the local SettleBaseline type, the split-then-reassemble in
settleObservationCommand, and the 'preActionNodes' in resolved narrowing tests.
SurfaceScopedNodes moves to contracts, where ResolvedInteractionTarget can name
it; only two sites now mint one from a SnapshotState.

Behaviour is unchanged: the cross-surface guarantees keep their existing tests.

* fix(ios): identify a surface capture by what the runner served

The `present` path stamped the capture's comparison lineage from the host-side
presence probe. That probe answers about a host PROCESS and deliberately stays
positive while a dismissed host lingers, so during that window the runner
truthfully returned APP content while the route lineaged it to the HOST: the
sheet capture before the dismissal and the app capture after it compared equal,
and a post-gesture poll could read the transition as a stable surface.

Derive the identity from the returned capture's `systemSurface` instead - the
runner stamps the surface it actually served - and say which of the two the
capture holds in the warning. The probe's host is now evidence only: it names
the matched host in a route diagnostic so a lingering window is legible in the
daemon log. Other reasons keep their lineage and wording byte for byte.

Captures that bypass the route's planning (a pinned backend, a custom-actions
read) also reach the runner, and the runner serves the sheet there too. They
carried no comparison key at all, so a sheet and app content fell through to
legacy presentation matching as one presentation and could corroborate a tap
across the two. The capture owner now gives those a surface-scoped identity as
well, with no fallback-source residue: nothing fell back. An app capture off
the route is untouched.

* fix(ios): derive a served surface identity at the one stamping point

A runner fallback's comparison identity was decided per call site. The
`present` path and the off-route path read the runner's `systemSurface`
stamp, but the plain `runFallback` path did not: it stamped the app
lineage the route had planned, whatever the runner returned.

The probe and the capture are separate observations, so a sheet can
appear in the gap between them. With the bridge circuit already disabled
for the generation, an app capture and a later sheet capture both
received the same app-generation key, so tap corroboration could treat
two different surfaces as comparable.

`stampFallback` now owns the decision for every runner fallback: the
surface the runner served outranks the app lineage the route planned.
The reason the bridge was skipped survives either way, and
app-generation evidence leaves with the app lineage it describes, so two
captures of the same sheet still compare equal. `runSurfaceFallback`
keeps only the reason, which is the one thing that path decides.
This commit is contained in:
Michał Pierzchała
2026-09-11 17:37:35 +02:00
committed by GitHub
parent bda6d42c9a
commit 0feb4e26a0
46 changed files with 2404 additions and 203 deletions
@@ -0,0 +1,72 @@
import Foundation
// iOS out-of-process system surfaces observed and driven IN PLACE, never activated: activating such
// a host cancels what it presents (issue #2438; rationale in docs/adr/0004). Membership is the
// golden fixture contracts/fixtures/ios-system-surface-hosts.json, mirrored by the TS twin
// packages/contracts/src/ios-system-surface.ts; drift fails on either side without a simulator.
enum SystemSurfaceHostKind: String {
case webAuth = "web-auth"
}
struct SystemSurfaceHost: Equatable {
let bundleId: String
let kind: SystemSurfaceHostKind
}
enum SystemSurfaceHostRegistry {
static let hosts: [SystemSurfaceHost] = [
SystemSurfaceHost(bundleId: "com.apple.SafariViewService", kind: .webAuth)
]
static func host(forBundleId bundleId: String?) -> SystemSurfaceHost? {
guard let bundleId else { return nil }
return hosts.first { $0.bundleId == bundleId }
}
static func isSystemSurfaceHost(_ bundleId: String?) -> Bool {
host(forBundleId: bundleId) != nil
}
}
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
import XCTest
private struct SystemSurfaceHostFixture: Decodable {
struct Host: Decodable {
let bundleId: String
let kind: String
}
let hosts: [Host]
}
extension RunnerTests {
func testSystemSurfaceHostRegistryMirrorsGoldenFixture() throws {
let fixtureURL = URL(fileURLWithPath: #filePath)
.deletingLastPathComponent() // AgentDeviceRunnerUITests
.deletingLastPathComponent() // AgentDeviceRunner
.deletingLastPathComponent() // runner
.deletingLastPathComponent() // apple
.deletingLastPathComponent() // repo root
.appendingPathComponent("contracts")
.appendingPathComponent("fixtures")
.appendingPathComponent("ios-system-surface-hosts.json")
let fixture = try JSONDecoder().decode(
SystemSurfaceHostFixture.self,
from: Data(contentsOf: fixtureURL)
)
let registry = SystemSurfaceHostRegistry.hosts.map { [$0.bundleId, $0.kind.rawValue] }
let golden = fixture.hosts.map { [$0.bundleId, $0.kind] }
XCTAssertEqual(registry, golden, "SystemSurfaceHostRegistry drifted from the golden fixture")
}
func testSystemSurfaceHostRegistryRecognizesRegisteredHosts() {
XCTAssertTrue(SystemSurfaceHostRegistry.isSystemSurfaceHost("com.apple.SafariViewService"))
XCTAssertFalse(SystemSurfaceHostRegistry.isSystemSurfaceHost("com.example.app"))
XCTAssertFalse(SystemSurfaceHostRegistry.isSystemSurfaceHost(nil))
XCTAssertEqual(
SystemSurfaceHostRegistry.host(forBundleId: "com.apple.SafariViewService")?.kind,
.webAuth
)
}
}
#endif
@@ -986,6 +986,8 @@ extension RunnerTests {
struct ActiveCommandContext {
let app: XCUIApplication
/// Set when `app` is a system surface served in place over the still-bound session app (#2438).
var systemSurface: SystemSurfaceHost? = nil
}
enum ActiveCommandPreparation {
@@ -1342,7 +1344,11 @@ extension RunnerTests {
case .response(let response):
return response
case .context(let context):
return try executeSnapshotPrepared(command: command, activeApp: context.app)
return try executeSnapshotPrepared(
command: command,
activeApp: context.app,
systemSurface: context.systemSurface
)
}
}
@@ -1364,15 +1370,25 @@ extension RunnerTests {
)
}
private func executeSnapshotPrepared(command: Command, activeApp: XCUIApplication) throws -> Response {
private func executeSnapshotPrepared(
command: Command,
activeApp: XCUIApplication,
systemSurface: SystemSurfaceHost? = nil
) throws -> Response {
let options = Self.presentationOptions(from: command)
do {
let payload: DataPayload
var payload: DataPayload
if options.raw {
payload = try snapshotRaw(app: activeApp, options: options)
} else {
payload = try snapshotFast(app: activeApp, options: options)
}
if let systemSurface {
payload.systemSurface = SystemSurfaceProvenancePayload(
bundleId: systemSurface.bundleId,
kind: systemSurface.kind.rawValue
)
}
setNeedsPostSnapshotInteractionDelay()
return Response(ok: true, data: payload)
} catch let failure as SnapshotCaptureFailure {
@@ -1575,10 +1591,20 @@ extension RunnerTests {
routeToSpringboard: Bool = false
) -> ActiveCommandPreparation {
var activeApp = currentApp ?? app
var systemSurface: SystemSurfaceHost? = nil
if routeToSpringboard {
activeApp = springboard
} else if shouldSkipAppActivationPreflight(command) {
activeApp = resolveAppWithoutActivation(command: command)
} else if let presented = presentedSystemSurfaceHost() {
// Serve and drive the presented surface IN PLACE: never activate it (that cancels what it
// presents) and never adopt it as the cached session target, so once it is gone the next
// command resolves back to the still-bound session app (#2438).
activeApp = presented.app
systemSurface = presented.host
if isInteractionCommand(command.command) {
applyInteractionStabilizationIfNeeded()
}
} else if !isRunnerLifecycleCommand(command.command) {
let normalizedBundleId = command.appBundleId?
.trimmingCharacters(in: .whitespacesAndNewlines)
@@ -1639,7 +1665,25 @@ extension RunnerTests {
applyInteractionStabilizationIfNeeded()
}
}
return .context(ActiveCommandContext(app: activeApp))
return .context(ActiveCommandContext(app: activeApp, systemSurface: systemSurface))
}
/// A registered system surface host that is genuinely on screen, or nil. Presence is foreground
/// state, not tree content: a torn-down host still serves a rich tree, and it can only be
/// foreground-with-a-stale-tree if something activated it, which the open guard refuses. `state`
/// never activates and is cheap when the host is absent. See docs/adr/0004.
private func presentedSystemSurfaceHost() -> (host: SystemSurfaceHost, app: XCUIApplication)? {
#if os(iOS)
for host in SystemSurfaceHostRegistry.hosts {
let candidate = XCUIApplication(bundleIdentifier: host.bundleId)
if candidate.state == .runningForeground {
return (host, candidate)
}
}
return nil
#else
return nil
#endif
}
func executeOnMainPrepared(
@@ -241,6 +241,8 @@ struct DataPayload: Codable {
var truncated: Bool?
var qualityPayload: SnapshotQualityPayload? = nil
var snapshotQuality: SnapshotQuality?
/// Set when the capture describes an in-place system surface, not the app itself (#2438).
var systemSurface: SystemSurfaceProvenancePayload?
var gestureStartUptimeMs: Double?
var gestureEndUptimeMs: Double?
var x: Double?
@@ -275,6 +277,12 @@ struct DataPayload: Codable {
var sequenceResults: [SequenceStepResult]?
}
/// `kind` mirrors the TS `IosSystemSurfaceKind` (e.g. "web-auth").
struct SystemSurfaceProvenancePayload: Codable {
let bundleId: String
let kind: String
}
struct SnapshotQualityPayload: Codable {
let nodes: [PresentedNode]
let truncated: Bool
@@ -0,0 +1,11 @@
{
"description": "iOS out-of-process system surfaces that agent-device observes and drives IN PLACE, never by activation. Activating or relaunching such a host destroys what it presents: com.apple.SafariViewService hosts ASWebAuthenticationSession / SFSafariViewController, and any XCUIApplication.activate() or simctl launch cancels the auth session (issue #2438). Source of truth shared by the TypeScript registry (packages/contracts/src/ios-system-surface.ts) and the Swift runner registry (RunnerSystemSurfaceHostPolicy.swift); a change here must keep both parity tests green. `processExecutable` is the simulator app-binary path fragment the TypeScript host-side presence probe matches with `pgrep -f`, confirming device scope from the matched process's environment; the Swift runner detects the host by bundle id via XCUIApplication.state and ignores it.",
"hosts": [
{
"bundleId": "com.apple.SafariViewService",
"kind": "web-auth",
"processExecutable": "SafariViewService.app/SafariViewService",
"note": "Hosts ASWebAuthenticationSession and SFSafariViewController out of the app's process. Presented over a still-foreground app; read and driven in place via the XCTest runner (the host AX bridge cannot see it: the app remains the AX primaryApp)."
}
]
}
@@ -311,3 +311,47 @@ When adding new iOS snapshot behavior, maintainers should first decide which str
change tries to make regular snapshots fast by dropping visible controls behind a node budget, or
tries to make raw snapshots safe by silently truncating, it is probably crossing strategy
boundaries.
## Amendment: in-place system surfaces (issue #2438)
Some UI is presented out of the app's process by a system bundle — `com.apple.SafariViewService`,
which hosts `ASWebAuthenticationSession` and `SFSafariViewController` for delegated OAuth/OIDC
sign-in. Two facts, both verified live on the iOS 26.2 Simulator, shape how it is captured:
- The surface dies if activated. `XCUIApplication.activate()` or `simctl launch` on the host cancels
the authentication session and blacks the view. So the host must be observed and driven **in
place**, never activated, and `open` refuses to launch a registered host.
- The local host AX bridge cannot see it. While the sheet is up the app remains the AX `primaryApp`,
so the bridge serves the (occluded) app tree as if healthy. Only the XCTest runner, addressing the
host by bundle id, can read and drive the sheet.
Decision. A closed registry names these hosts (`contracts/fixtures/ios-system-surface-hosts.json`,
mirrored by the TypeScript and Swift registries under a parity test). When a registered host is
genuinely presented, the runner serves and drives it in place and never adopts it as the cached
session target; the session binding stays on the app, so once the surface is gone the next command
resolves back to the app. On the Simulator a cheap, device-scoped host-side probe (a registered
host process running for the device) routes the capture to the runner instead of the bridge; when no
host is running the bridge fast path is untouched.
A presented surface also outranks an explicitly requested bundle id: the runner checks for a
presented host before it resolves or activates `command.appBundleId`, so a command that names a
*different* app is still served the sheet. That is deliberate — the sheet occludes the screen, so
the named app has nothing readable under it, and the capture discloses which surface it describes —
and it costs nothing once the sheet is gone, because the session binding never moved.
Presence is `XCUIApplication.state == .runningForeground`, not tree content. The live spike showed a
torn-down host still serving a *richer* tree than a live one, so content heuristics cannot separate
live from dead; foreground state can. Crucially, the only way a host is foreground with a stale tree
is if it was activated or relaunched — which the open guard and the in-place policy both refuse — so
this fix and the never-activate guard are one design: the guard is what makes the foreground
predicate sound. This also makes issue #2438's second bug (a stale tree served confidently after
teardown) unrepresentable for the delegated-auth flow, because the session never binds to the host.
Captures of a system surface carry a response-level `systemSurface` provenance and the shared
`IOS_SYSTEM_SURFACE_DISCLOSURE`, so the agent is told the controls belong to a system sheet rather
than the app. They are also lineaged to the host rather than the app, so their comparison identity
differs from an app capture's by construction: every consumer that asks "are these two captures the
same presentation" refuses a cross-surface pair through ordinary key equality, and no comparison
site carries a surface check of its own. Physical devices always use the runner, so the in-place
serve applies there without a route change; the Simulator route probe is the only
Simulator-specific piece.
@@ -22,6 +22,7 @@ import { coveredAndroidReplacementNodeIndexes } from './snapshot/android-replace
import { scopeSnapshotNodes } from './snapshot-desktop-projection.ts';
import { normalizeSnapshotTree, pruneGroupNodes } from './snapshot-tree-ingestion.ts';
import { iosSnapshotComparisonIdentityKey } from './ios-snapshot-planning.ts';
import type { IosSystemSurfaceProvenance } from '@agent-device/contracts/ios-system-surface';
import type { IosSnapshotComparisonIdentity } from '@agent-device/contracts/ios-snapshot';
/**
@@ -43,6 +44,7 @@ export function buildSnapshotState(
truncated?: boolean;
quality?: unknown;
comparisonIdentity?: IosSnapshotComparisonIdentity;
systemSurface?: IosSystemSurfaceProvenance;
} & SnapshotCaptureProvenance,
flags:
| (Pick<CommandFlags, 'snapshotDepth' | 'snapshotInteractiveOnly' | 'snapshotRaw'> &
@@ -78,6 +80,7 @@ export function buildSnapshotState(
...(data.comparisonIdentity
? { comparisonKey: iosSnapshotComparisonIdentityKey(data.comparisonIdentity) }
: {}),
...(data.systemSurface ? { iosSystemSurfaceBundleId: data.systemSurface.bundleId } : {}),
presentationKey: buildSnapshotPresentationKey(snapshotPresentationOptionsFromFlags(flags)),
// Only broad Android snapshots become freshness baselines. If the user asked for a scoped
// or filtered view, preserve that output contract but avoid pretending it is safe for
+4
View File
@@ -240,6 +240,10 @@
"types": "./src/ios-snapshot.ts",
"default": "./src/ios-snapshot.ts"
},
"./ios-system-surface": {
"types": "./src/ios-system-surface.ts",
"default": "./src/ios-system-surface.ts"
},
"./interactor-operation-catalog": {
"types": "./src/interactor-operation-catalog.ts",
"default": "./src/interactor-operation-catalog.ts"
+58 -4
View File
@@ -121,13 +121,35 @@ export type PreresolvedInteractionTarget = {
node: SnapshotNode;
/** The tree `node` came from — the guards read its siblings for occlusion/viewport. */
nodes: SnapshotNode[];
/**
* The in-place iOS system surface `nodes` describes (#2438), absent for ordinary app content.
* Travels with the tree so the adopting consumer's post-action comparison knows which surface
* its baseline came from.
*/
iosSystemSurfaceBundleId?: string;
};
/**
* One side of a post-action comparison: the nodes, and the SURFACE the capture they came from
* described (#2438: the bundle id of an in-place iOS system surface such as a web sign-in sheet,
* absent for ordinary app content).
*
* One value, never two channels: a capture of the sheet and a capture of the app describe
* different surfaces, so a `--verify` digest comparison or a `--settle` diff across that boundary
* is not about one presentation. Every boundary that carries a baseline carries this type, so
* nodes cannot arrive without the surface they describe.
*/
export type SurfaceScopedNodes = {
nodes: SnapshotNode[];
/** Bundle id of the in-place iOS system surface; absent for ordinary app content. */
surfaceBundleId?: string;
};
export type ResolvedInteractionTarget =
| {
kind: 'point';
point: Point;
preActionNodes?: SnapshotNode[];
preAction?: SurfaceScopedNodes;
}
| {
kind: 'ref';
@@ -138,9 +160,9 @@ export type ResolvedInteractionTarget =
refLabel?: string;
targetHittable?: boolean;
hint?: string;
preActionNodes?: SnapshotNode[];
resolution?: ResolutionDisclosure;
recordingTarget?: RecordingTargetOverride;
preAction?: SurfaceScopedNodes;
}
| {
kind: 'selector';
@@ -151,11 +173,28 @@ export type ResolvedInteractionTarget =
refLabel?: string;
targetHittable?: boolean;
hint?: string;
preActionNodes?: SnapshotNode[];
resolution?: ResolutionDisclosure;
recordingTarget?: RecordingTargetOverride;
preAction?: SurfaceScopedNodes;
};
/**
* A post-action capture that describes a DIFFERENT surface than the pre-action baseline (#2438): an
* in-place iOS system surface (a web sign-in sheet, hosted out of the app's process) was presented
* over the app, or left it. `from`/`to` name the two surfaces a host bundle id, or `app` for
* ordinary app content.
*
* Its presence IS the refusal of a same-surface claim: the two captures are not one presentation,
* so `--verify` reports `changedFromBefore` from this transition instead of from a digest
* comparison across it, and `--settle` attaches no settled diff (and therefore no refs) across it.
*/
export type PostActionSurfaceChange = {
from: string;
to: string;
/** The one agent-facing sentence for this transition (`@agent-device/contracts/ios-system-surface`). */
disclosure: string;
};
/**
* Opt-in (`--verify`) cheap post-condition evidence for mutating interaction
* commands (#1047). `digest`/`nodeCount`/`interactiveNodeCount` describe a single
@@ -164,6 +203,10 @@ export type ResolvedInteractionTarget =
* held, so no extra device round trip is spent beyond the one verify capture.
* `changedFromBefore: false` is evidence, not failure the command still
* succeeded.
*
* When `surfaceChange` is present the two captures describe different surfaces, so the digest
* comparison is not made at all: `changedFromBefore` then reports that transition, which replaced
* the whole observed surface.
*/
export type InteractionEvidence = {
foregroundApp?: string;
@@ -171,6 +214,7 @@ export type InteractionEvidence = {
interactiveNodeCount: number;
digest: string;
changedFromBefore: boolean;
surfaceChange?: PostActionSurfaceChange;
};
export type SettleDiffLine = {
@@ -249,7 +293,17 @@ export type SettleObservation = {
* intentionally omitted.
*/
refs?: Array<{ ref: string }>;
/** Present only for `settled: true` observations that stored the settled tree. */
/**
* Present when the settled capture describes a different surface than the pre-action baseline
* (#2438). The settled tree then replaced the whole surface rather than changing within one, so
* `diff` is omitted: its lines (and their refs) would present a surface replacement as an
* in-surface change. `hint` says what to do instead.
*/
surfaceChange?: PostActionSurfaceChange;
/**
* Present only for `settled: true` observations that stored the settled tree, and never across a
* `surfaceChange` a diff describes change WITHIN one surface.
*/
diff?: {
summary: { additions: number; removals: number; unchanged: number };
lines: SettleDiffLine[];
@@ -1,4 +1,5 @@
import type { BackMode } from './back-mode.ts';
import type { IosSystemSurfaceProvenance } from './ios-system-surface.ts';
import type { DeviceRotation } from './device-rotation.ts';
import type { ScrollDirection } from './scroll-gesture.ts';
import type { ScrollExecutionOptions } from './scroll-command.ts';
@@ -255,6 +256,11 @@ export type KeyboardEnterResult =
export type SnapshotResult = Omit<BackendSnapshotResult, 'backend' | 'nodes'> & {
nodes?: RawSnapshotNode[];
comparisonIdentity?: IosSnapshotComparisonIdentity;
/**
* Set when the capture describes an in-place iOS system surface (a web sign-in sheet) presented
* over the session app rather than the app itself (#2438).
*/
systemSurface?: IosSystemSurfaceProvenance;
} & SnapshotProvenance;
export type SnapshotRuntimeAcquiredResult = Readonly<{
@@ -0,0 +1,68 @@
import fs from 'node:fs';
import path from 'node:path';
import { expect, test } from 'vitest';
import {
IOS_SYSTEM_SURFACE_DISCLOSURE,
IOS_SYSTEM_SURFACE_HOSTS,
iosSystemSurfaceOpenRefusal,
iosSystemSurfaceTransitionDisclosure,
isIosSystemSurfaceHost,
} from './ios-system-surface.ts';
const FIXTURE_PATH = path.resolve(
import.meta.dirname,
'..',
'..',
'..',
'contracts',
'fixtures',
'ios-system-surface-hosts.json',
);
type Fixture = {
hosts: Array<{ bundleId: string; kind: string; processExecutable: string; note: string }>;
};
function readFixture(): Fixture {
return JSON.parse(fs.readFileSync(FIXTURE_PATH, 'utf8')) as Fixture;
}
// Cross-language parity: the TS registry must mirror the canonical fixture that the Swift
// SystemSurfaceHostRegistry also reads. A drift on either side fails here or in the Swift test.
test('the TS registry mirrors the canonical fixture exactly', () => {
const fixture = readFixture();
expect(
IOS_SYSTEM_SURFACE_HOSTS.map((host) => ({
bundleId: host.bundleId,
kind: host.kind,
processExecutable: host.processExecutable,
})),
).toEqual(
fixture.hosts.map((host) => ({
bundleId: host.bundleId,
kind: host.kind,
processExecutable: host.processExecutable,
})),
);
});
test('isIosSystemSurfaceHost recognizes registered hosts and rejects others', () => {
expect(isIosSystemSurfaceHost('com.apple.SafariViewService')).toBe(true);
expect(isIosSystemSurfaceHost('com.example.app')).toBe(false);
expect(isIosSystemSurfaceHost(undefined)).toBe(false);
});
test('the open refusal names the bundle and does not claim to open it', () => {
const refusal = iosSystemSurfaceOpenRefusal('com.apple.SafariViewService');
expect(refusal).toContain('com.apple.SafariViewService');
expect(refusal.toLowerCase()).not.toContain('opened it');
});
test('the transition disclosure says the sheet is gone only when it left', () => {
expect(iosSystemSurfaceTransitionDisclosure('com.apple.SafariViewService')).toBe(
IOS_SYSTEM_SURFACE_DISCLOSURE,
);
const departed = iosSystemSurfaceTransitionDisclosure(undefined);
expect(departed).not.toBe(IOS_SYSTEM_SURFACE_DISCLOSURE);
expect(departed).toContain('gone now');
});
@@ -0,0 +1,88 @@
/**
* iOS out-of-process system surfaces that agent-device observes and drives IN PLACE, never by
* activation.
*
* `com.apple.SafariViewService` hosts `ASWebAuthenticationSession` and `SFSafariViewController`
* out of the app's process. It is presented over a still-foreground app, and any
* `XCUIApplication.activate()` or `simctl launch` on it cancels the authentication session and
* blacks the view (issue #2438). So the runner reads and drives it without activation, and the
* `open` path refuses to launch it. The set is deliberately closed and tiny; add a host only with
* live evidence that it presents out of process and dies on activation.
*
* The canonical membership lives in `contracts/fixtures/ios-system-surface-hosts.json`; this module
* and the Swift `SystemSurfaceHostRegistry` both mirror it, each guarded by a parity test.
*/
/** Why a system surface is served in place; carried at snapshot-response level as provenance. */
export type IosSystemSurfaceKind = 'web-auth';
export type IosSystemSurfaceHost = Readonly<{
bundleId: string;
kind: IosSystemSurfaceKind;
/**
* Simulator app-binary path fragment the host-side presence probe matches with `pgrep -f`. Only
* a matched pid's environment is then read, to confirm the process belongs to the requested
* device. The Swift runner detects the host by bundle id (`XCUIApplication.state`) instead.
*/
processExecutable: string;
}>;
export const IOS_SYSTEM_SURFACE_HOSTS: readonly IosSystemSurfaceHost[] = Object.freeze([
Object.freeze({
bundleId: 'com.apple.SafariViewService',
kind: 'web-auth' as const,
processExecutable: 'SafariViewService.app/SafariViewService',
}),
]);
const HOST_BY_BUNDLE_ID: ReadonlyMap<string, IosSystemSurfaceHost> = new Map(
IOS_SYSTEM_SURFACE_HOSTS.map((host) => [host.bundleId, host] as const),
);
/** The bundle id, if any, is a known observe-in-place system surface host. */
export function isIosSystemSurfaceHost(bundleId: string | undefined): boolean {
return bundleId !== undefined && HOST_BY_BUNDLE_ID.has(bundleId);
}
/**
* Refusal shown when a user tries to `open` a system surface host directly. These surfaces are not
* launched; while genuinely presented they appear in the session app's snapshots on their own, and
* are driven in place. Keyed by callers off `UNSUPPORTED_OPERATION`; the text is the agent-facing
* explanation.
*/
export function iosSystemSurfaceOpenRefusal(bundleId: string): string {
return `${bundleId} is a system-hosted surface (e.g. a web sign-in sheet) that cannot be opened directly — launching or activating it cancels what it presents. While it is on screen it already appears in this session's snapshots; read it and interact with it there without opening it.`;
}
/**
* Whole-snapshot provenance: the capture describes a system surface presented over the session app,
* not the app itself. Carried at response level (it applies to the entire snapshot) and folded into
* iOS snapshot lineage so `--verify`/`--settle` never compare an app baseline against a sheet
* capture. Mirrors the Android system-chrome/system-surface provenance model.
*/
export type IosSystemSurfaceProvenance = Readonly<{
bundleId: string;
kind: IosSystemSurfaceKind;
}>;
/**
* The one agent-facing explanation for an iOS capture that faithfully shows an occluding system
* surface (a web sign-in sheet) instead of app content. Shared by the direct snapshot warning and
* every selector-backed consumer (find/wait/get/is) so the disclosure cannot silently drop on one
* route while surviving on another; generalizes the Android system-surface disclosure.
*/
export const IOS_SYSTEM_SURFACE_DISCLOSURE =
'A system web sign-in sheet is presented over the app, so this snapshot shows that sheet (hosted out of the app process). Its controls are real and interactive; complete or dismiss the sheet to return to app content.';
/**
* The agent-facing sentence for a surface TRANSITION between two captures the post-action
* observation's case, where the pre-action baseline and the capture taken after the action describe
* different surfaces. `to` is the surface the AFTER capture describes: a host bundle id when the
* sheet is now on screen (the standing disclosure applies verbatim), or `undefined` when the sheet
* has left and the capture shows app content again, which the standing sentence cannot say.
*/
export function iosSystemSurfaceTransitionDisclosure(to: string | undefined): string {
return to === undefined
? 'A system web sign-in sheet was presented over the app before this action and is gone now, so this observation describes app content while the pre-action tree described that sheet.'
: IOS_SYSTEM_SURFACE_DISCLOSURE;
}
+6
View File
@@ -401,6 +401,12 @@ export type SnapshotState = {
* occlusion (see `@agent-device/contracts/android-system-surface-disclosure`).
*/
systemSurfaceOnly?: boolean;
/**
* iOS: the bundle id of the in-place system surface this capture describes (a web sign-in sheet
* presented over the app, #2438). Two captures that disagree here describe different surfaces and
* must never be compared as the same presentation; consumers that surface the tree disclose it.
*/
iosSystemSurfaceBundleId?: string;
} & SnapshotStateProvenance;
export type SnapshotUnchanged = {
@@ -0,0 +1,97 @@
import { beforeEach, test, vi } from 'vitest';
import assert from 'node:assert/strict';
import { IOS_TEST_SIMULATOR } from './apple-core-stub-helpers.ts';
vi.mock('@agent-device/host-kit/command', async (importOriginal) => {
const actual = await importOriginal<typeof import('@agent-device/host-kit/command')>();
return { ...actual, runCmd: vi.fn(actual.runCmd) };
});
vi.mock('../simulator.ts', async (importOriginal) => {
const actual = await importOriginal<typeof import('../simulator.ts')>();
return { ...actual, ensureBootedSimulator: vi.fn(actual.ensureBootedSimulator) };
});
import { runCmd } from '@agent-device/host-kit/command';
import { ensureBootedSimulator } from '../simulator.ts';
import { closeIosApp, openIosApp } from '../app-launch.ts';
import { AppError } from '@agent-device/kernel/errors';
const mockRunCmd = vi.mocked(runCmd);
const mockEnsureBootedSimulator = vi.mocked(ensureBootedSimulator);
beforeEach(() => {
vi.resetAllMocks();
mockEnsureBootedSimulator.mockResolvedValue();
mockRunCmd.mockResolvedValue({ stdout: '', stderr: '', exitCode: 0 });
});
// Bug A (issue #2438): opening the web-auth host must be refused, not launched — a simctl launch or
// activation cancels the ASWebAuthenticationSession it presents. The refusal must fire BEFORE any
// process-touching command runs.
test('open refuses a system-surface host and never launches it', async () => {
await assert.rejects(
openIosApp(IOS_TEST_SIMULATOR, 'com.apple.SafariViewService', {
appBundleId: 'com.apple.SafariViewService',
}),
(error: unknown) => {
assert.ok(error instanceof AppError);
assert.equal(error.code, 'UNSUPPORTED_OPERATION');
assert.equal(error.details?.reason, 'system-surface-host-not-openable');
assert.match(error.message, /com\.apple\.SafariViewService/);
return true;
},
);
assert.equal(mockRunCmd.mock.calls.length, 0);
});
// The refusal must sit at every resolved-host launch/terminate, not only the plain `open <bundle>`
// branch: the URL and deep-link branches return before it and would otherwise still launch the host.
type OpenOptions = NonNullable<Parameters<typeof openIosApp>[2]>;
test.for<[string, OpenOptions]>([
['web URL', { url: 'https://example.com' }],
['web URL with relaunch', { url: 'https://example.com', terminateRunningApp: true }],
['deep link with relaunch', { url: 'myapp://path', terminateRunningApp: true }],
['launch args', { launchArgs: ['--flag'] }],
])('open refuses a system-surface host via the %s branch', async ([, options]) => {
await assert.rejects(
openIosApp(IOS_TEST_SIMULATOR, 'com.apple.SafariViewService', {
appBundleId: 'com.apple.SafariViewService',
...options,
}),
(error: unknown) => {
assert.ok(error instanceof AppError);
assert.equal(error.details?.reason, 'system-surface-host-not-openable');
return true;
},
);
const touchedHost = mockRunCmd.mock.calls.some(
([, args]) => Array.isArray(args) && args.includes('com.apple.SafariViewService'),
);
assert.equal(touchedHost, false, 'no simctl command may name the host bundle');
});
// Terminating the host cancels the presented session just as launching it does.
test('close refuses to terminate a system-surface host', async () => {
await assert.rejects(
closeIosApp(IOS_TEST_SIMULATOR, 'com.apple.SafariViewService'),
(error: unknown) => {
assert.ok(error instanceof AppError);
assert.equal(error.details?.reason, 'system-surface-host-not-openable');
return true;
},
);
assert.equal(mockRunCmd.mock.calls.length, 0);
});
test('open still launches an ordinary app', async () => {
await openIosApp(IOS_TEST_SIMULATOR, 'MyApp', { appBundleId: 'com.example.app' });
const launched = mockRunCmd.mock.calls.some(
([cmd, args]) =>
cmd === 'xcrun' &&
Array.isArray(args) &&
args.includes('launch') &&
args.includes('com.example.app'),
);
assert.ok(launched, 'ordinary app launch must still dispatch simctl launch');
});
@@ -179,13 +179,34 @@ export async function closeIosApp(
return;
}
await assertNotSystemSurfaceHost(bundleId);
await resolveIosPhysicalDeviceControl(device).terminateApp(device, bundleId, {
runnerOptions,
runRunnerCommand: runAppleRunnerCommand,
});
}
/**
* A system-hosted surface (e.g. the web sign-in sheet in `com.apple.SafariViewService`) must never
* be launched, activated, or terminated: doing so cancels what it presents (issue #2438). While it
* is on screen it appears in the session app's snapshots on its own and is driven in place.
*
* Every resolved-bundle launch and terminate calls this, so the URL and deep-link branches are
* covered too, not just a plain `open <bundle>`. The registry import is deferred to keep the
* app-lifecycle facade's eager closure flat.
*/
async function assertNotSystemSurfaceHost(bundleId: string): Promise<void> {
const { isIosSystemSurfaceHost, iosSystemSurfaceOpenRefusal } =
await import('@agent-device/contracts/ios-system-surface');
if (!isIosSystemSurfaceHost(bundleId)) return;
throw new AppError('UNSUPPORTED_OPERATION', iosSystemSurfaceOpenRefusal(bundleId), {
reason: 'system-surface-host-not-openable',
appBundleId: bundleId,
});
}
async function terminateIosSimulatorApp(device: DeviceInfo, bundleId: string): Promise<void> {
await assertNotSystemSurfaceHost(bundleId);
await ensureBootedSimulator(device);
const terminateArgs = simctlArgs(device, ['terminate', device.id, bundleId]);
const result = await runXcrun(terminateArgs, {
@@ -207,6 +228,7 @@ async function launchIosSimulatorApp(
bundleId: string,
options?: { launchConsole?: string; launchArgs?: string[]; terminateRunningApp?: boolean },
): Promise<void> {
await assertNotSystemSurfaceHost(bundleId);
await ensureBootedSimulator(device);
let consecutiveFBSFailures = 0;
@@ -337,6 +359,7 @@ async function launchIosDeviceProcess(
runnerOptions?: AppleRunnerCommandOptions;
},
): Promise<void> {
await assertNotSystemSurfaceHost(bundleId);
await resolveIosPhysicalDeviceControl(device).launchApp(device, bundleId, {
...options,
runRunnerCommand: runAppleRunnerCommand,
+16 -2
View File
@@ -34,6 +34,7 @@ import {
readAppleSnapshotResult,
} from './runner/snapshot-presentation.ts';
import type { AppleRunnerSnapshotResult } from './runner/snapshot-presentation.ts';
import { IOS_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/ios-system-surface';
export function createAppleInteractor(
device: DeviceInfo,
@@ -235,17 +236,30 @@ async function captureAppleRunnerSnapshot(
if (nodes.length === 0 && device.kind === 'simulator' && !isValidEmptyScope) {
throw new AppError('COMMAND_FAILED', 'XCTest snapshot returned 0 nodes on iOS simulator.');
}
const warnings = runnerSnapshotWarnings(result);
return {
nodes: presentRunnerSnapshotForDevice(device, options, result),
truncated: result.truncated ?? false,
backend: 'xctest' as const,
producer: 'apple-runner' as const,
...(result.quality ? { quality: result.quality } : {}),
// Legacy runners without a quality verdict still surface their message text.
...(!result.quality && result.message ? { warnings: [result.message] } : {}),
...(result.systemSurface ? { systemSurface: result.systemSurface } : {}),
...(warnings.length > 0 ? { warnings } : {}),
};
}
/**
* Agent-facing warnings for a runner capture: a legacy runner's message text when it carried no
* quality verdict, and the shared disclosure when the capture describes an in-place system surface
* (e.g. the web sign-in sheet) rather than the app itself (#2438).
*/
function runnerSnapshotWarnings(result: AppleRunnerSnapshotResult): string[] {
const warnings: string[] = [];
if (!result.quality && result.message) warnings.push(result.message);
if (result.systemSurface) warnings.push(IOS_SYSTEM_SURFACE_DISCLOSURE);
return warnings;
}
function presentRunnerSnapshotForDevice(
device: DeviceInfo,
options: SnapshotOptions | undefined,
@@ -17,6 +17,11 @@ import {
} from '@agent-device/capture-kit/ios-snapshot-planning';
import { AppError } from '@agent-device/kernel/errors';
import type { RawSnapshotNode, SnapshotQualityVerdict } from '@agent-device/kernel/snapshot';
import {
isIosSystemSurfaceHost,
type IosSystemSurfaceKind,
type IosSystemSurfaceProvenance,
} from '@agent-device/contracts/ios-system-surface';
export type AppleRunnerSnapshotResult = Readonly<{
nodes?: RawSnapshotNode[];
@@ -25,17 +30,20 @@ export type AppleRunnerSnapshotResult = Readonly<{
quality?: SnapshotQualityVerdict;
qualityPayload?: IosRunnerQualityPayloadFacts;
runnerFatal?: boolean;
systemSurface?: IosSystemSurfaceProvenance;
}>;
export function readAppleSnapshotResult(
result: Record<string, unknown>,
): AppleRunnerSnapshotResult {
const systemSurface = readSystemSurfaceProvenance(result.systemSurface);
return {
nodes: Array.isArray(result.nodes) ? (result.nodes as RawSnapshotNode[]) : undefined,
truncated: typeof result.truncated === 'boolean' ? result.truncated : undefined,
quality: readSnapshotQualityVerdict(result.snapshotQuality),
qualityPayload: readQualityPayload(result.qualityPayload),
runnerFatal: result.runnerFatal === true,
...(systemSurface ? { systemSurface } : {}),
message:
typeof result.message === 'string' && result.message.trim().length > 0
? result.message
@@ -43,6 +51,17 @@ export function readAppleSnapshotResult(
};
}
function readSystemSurfaceProvenance(value: unknown): IosSystemSurfaceProvenance | undefined {
if (!isRecord(value)) return undefined;
const bundleId = value.bundleId;
const kind = value.kind;
// Trust only a bundle id the shared registry recognizes; an unknown value is dropped rather than
// surfaced, mirroring the wire-reader discipline elsewhere in this module.
if (typeof bundleId !== 'string' || !isIosSystemSurfaceHost(bundleId)) return undefined;
if (typeof kind !== 'string') return undefined;
return { bundleId, kind: kind as IosSystemSurfaceKind };
}
export function presentAppleRunnerSnapshot(
deviceId: string,
options: SnapshotOptions | undefined,
@@ -1,6 +1,14 @@
import { expect, test, vi } from 'vitest';
import type { DeviceInfo } from '@agent-device/kernel/device';
// Keep the route tests hermetic: the default system-surface presence probe shells out to `ps`, which
// never resolves under the fake timers these tests drive. Tests that exercise the bypass inject
// their own probe through the `systemSurfacePresent` option.
vi.mock('./system-surface-presence.ts', () => ({
createSystemSurfacePresenceProbe: () => async () => 'absent',
}));
import { areIosSnapshotComparisonIdentitiesEqual } from '@agent-device/capture-kit/ios-snapshot-planning';
import { IOS_SYSTEM_SURFACE_HOSTS } from '@agent-device/contracts/ios-system-surface';
import { createLocalAppleToolProvider, withAppleToolProvider } from './core/tool-provider.ts';
import { platformRuntimeHostFixture } from './runtime.fixtures.ts';
import { createAppleSnapshotRoute } from './snapshot-route.ts';
@@ -28,6 +36,9 @@ const target = {
const input = { options: { appBundleId: 'com.example.app' } } as const;
/** A proven-present surface as the probe reports it: the matched host travels with the verdict. */
const presentSurface = { kind: 'present', host: IOS_SYSTEM_SURFACE_HOSTS[0]! } as const;
test('eligible simulator capture publishes bridge acquisition without touching XCTest', async () => {
const acquired = bridgeAcquisition();
const source = sourceReturning(acquired);
@@ -52,6 +63,215 @@ test('eligible simulator capture publishes bridge acquisition without touching X
expect(fallback).not.toHaveBeenCalled();
});
test.for([presentSurface, 'unknown'] as const)(
'an unabsent system surface routes the capture to the runner and never touches the bridge',
async (presence) => {
const source = sourceReturning(bridgeAcquisition());
const fallback = vi.fn(async () => runnerResult());
const route = createAppleSnapshotRoute(platformRuntimeHostFixture(), {
source,
resolveTarget: vi.fn(async () => target),
systemSurfacePresent: async () => presence,
});
await route.capture(ios, input, signal(), fallback);
// The host AX bridge cannot see the sheet, so the runner (fallback) must serve it and the bridge
// source must never be asked. `unknown` takes the same path: an unproven probe must not fall
// through to a bridge capture that would answer from the occluded app tree (#2438).
expect(fallback).toHaveBeenCalledOnce();
expect(source.acquire).not.toHaveBeenCalled();
},
);
// The structural guarantee every comparison site downstream relies on (#2438): a surface capture is
// lineaged to the HOST, not to the app (`target.targetId` here), so its comparison key cannot equal
// an app capture's and no comparison site needs its own surface check.
test('a presented system surface captures under the host lineage, never the app lineage', async () => {
const route = createAppleSnapshotRoute(platformRuntimeHostFixture(), {
source: sourceReturning(bridgeAcquisition()),
resolveTarget: vi.fn(async () => target),
systemSurfacePresent: async () => presentSurface,
});
const first = await route.capture(ios, input, signal(), async () => surfaceRunnerResult());
const second = await route.capture(ios, input, signal(), async () => surfaceRunnerResult());
expect(first.comparisonIdentity).toMatchObject({
producer: 'apple-runner',
lineage: { targetId: `${ios.id}:${presentSurface.host.bundleId}` },
// No `unknown-generation` residue: the surface is not an app generation, and a per-capture
// residue id would make two captures of the same sheet incomparable with each other too.
residue: [{ kind: 'fallback-source', producer: 'apple-runner' }],
});
expect(
areIosSnapshotComparisonIdentitiesEqual(first.comparisonIdentity!, second.comparisonIdentity!),
).toBe(true);
// The bridge is healthy here and simply cannot see the surface, so the app's wording would lie.
expect(first.warnings).toEqual([
'Simulator AX snapshot inapplicable (system-surface-presented); used XCTest to read the system surface presented over the app.',
]);
});
// The host-side probe answers about a host PROCESS, which stays positive while a dismissed host
// lingers — a documented false positive. Only the runner answers about the screen, and it stamps the
// surface it served on the capture. Reading the probe for identity instead would lineage the app
// capture to the host, make it compare EQUAL to the preceding sheet capture, and let a post-gesture
// poll read the dismissal as a stable surface (#2438).
test('a lingering probe cannot make a sheet capture and an app capture compare equal', async () => {
const route = createAppleSnapshotRoute(platformRuntimeHostFixture(), {
source: sourceReturning(bridgeAcquisition()),
resolveTarget: vi.fn(async () => target),
systemSurfacePresent: async () => presentSurface,
});
const sheet = await route.capture(ios, input, signal(), async () => surfaceRunnerResult());
const app = await route.capture(ios, input, signal(), async () => runnerResult());
const stillApp = await route.capture(ios, input, signal(), async () => runnerResult());
expect(sheet.comparisonIdentity?.lineage).toEqual({
targetId: `${ios.id}:${presentSurface.host.bundleId}`,
});
expect(app.comparisonIdentity?.lineage).toEqual({ targetId: target.targetId });
expect(
areIosSnapshotComparisonIdentitiesEqual(sheet.comparisonIdentity!, app.comparisonIdentity!),
).toBe(false);
// Two app captures taken in the same lingering window still compare equal, so a poll can settle on
// app content: the capture decides the lineage, and nothing here carries a per-capture residue.
expect(
areIosSnapshotComparisonIdentitiesEqual(app.comparisonIdentity!, stillApp.comparisonIdentity!),
).toBe(true);
expect(app.warnings).toEqual([
'Simulator AX snapshot inapplicable (system-surface-host-lingering); used XCTest, which read app content: the system surface host process was still running but no longer presenting.',
]);
});
// The probe is a separate observation from the capture, so a sheet can appear in the gap between
// them. With the bridge circuit already disabled for this generation the route takes the plain
// fallback, which used to stamp the planned app lineage onto whatever came back — so the sheet
// capture carried the app generation and could corroborate a tap against an app capture (#2438).
test('a sheet that appears after an absent probe is identified by the surface, not the app', async () => {
const route = createAppleSnapshotRoute(platformRuntimeHostFixture(), {
source: sourceReturning({
stage: 'failed',
failure: { kind: 'transport-failure', code: 'bridge-disconnected' },
}),
resolveTarget: vi.fn(async () => target),
systemSurfacePresent: async () => 'absent',
});
// The first capture disables the circuit for this generation; both later captures take the
// `circuit-disabled` path, where the route plans nothing about a surface.
const app = await route.capture(ios, input, signal(), async () => runnerResult());
const sheet = await route.capture(ios, input, signal(), async () => surfaceRunnerResult());
const sheetAgain = await route.capture(ios, input, signal(), async () => surfaceRunnerResult());
expect(sheet.comparisonIdentity).toMatchObject({
producer: 'apple-runner',
lineage: { targetId: `${ios.id}:${presentSurface.host.bundleId}` },
residue: [{ kind: 'fallback-source', producer: 'apple-runner' }],
});
expect(app.comparisonIdentity?.lineage).toEqual({
targetId: target.targetId,
generation: target.generation,
});
expect(
areIosSnapshotComparisonIdentitiesEqual(app.comparisonIdentity!, sheet.comparisonIdentity!),
).toBe(false);
// Two captures of the same sheet still compare equal, so a poll can settle on the sheet.
expect(
areIosSnapshotComparisonIdentitiesEqual(
sheet.comparisonIdentity!,
sheetAgain.comparisonIdentity!,
),
).toBe(true);
// The reason the bridge was skipped survives — it is independent of what the runner found — but
// the sentence cannot claim an app generation this capture did not read.
expect(sheet.warnings).toEqual([
'Simulator AX snapshot unavailable (circuit-disabled); used XCTest, which read the system surface presented over the app.',
]);
});
// Same gap, reached through the probe-unavailable arm: that arm adds a per-capture
// `unknown-generation` residue for an app capture, and app-generation evidence must not ride along
// on a capture of a surface — it would make two captures of the same sheet incomparable too.
test('a surface capture drops the app-generation residue of the arm that reached it', async () => {
const route = createAppleSnapshotRoute(platformRuntimeHostFixture(), {
source: sourceReturning(bridgeAcquisition()),
resolveTarget: vi.fn(async () => target),
systemSurfacePresent: async () => 'unknown',
});
const sheet = await route.capture(ios, input, signal(), async () => surfaceRunnerResult());
const sheetAgain = await route.capture(ios, input, signal(), async () => surfaceRunnerResult());
expect(sheet.comparisonIdentity).toMatchObject({
lineage: { targetId: `${ios.id}:${presentSurface.host.bundleId}` },
residue: [{ kind: 'fallback-source', producer: 'apple-runner' }],
});
expect(
areIosSnapshotComparisonIdentitiesEqual(
sheet.comparisonIdentity!,
sheetAgain.comparisonIdentity!,
),
).toBe(true);
});
// A pinned backend and a custom-actions read bypass the route's planning, but they still reach the
// runner, and the runner serves the sheet there too. Without an identity that pair falls back to
// legacy presentation matching, where a sheet and app content read as one presentation and could
// corroborate a tap across the two (#2438).
test.each([
['a pinned backend', { preferredBackend: 'private-ax' }],
['a custom-actions read', { customActions: true }],
] as const)(
'a route-bypassing capture of a system surface is incomparable (%s)',
async (_label, bypass) => {
const route = createAppleSnapshotRoute(platformRuntimeHostFixture(), {
source: sourceReturning(bridgeAcquisition()),
resolveTarget: vi.fn(async () => target),
});
const bypassInput = { options: { ...input.options, ...bypass } };
const sheet = await route.capture(ios, bypassInput, signal(), async () =>
surfaceRunnerResult(),
);
const app = await route.capture(ios, bypassInput, signal(), async () => runnerResult());
expect(sheet.comparisonIdentity).toMatchObject({
producer: 'apple-runner',
lineage: { targetId: `${ios.id}:${presentSurface.host.bundleId}` },
// Nothing fell back here: the runner is the requested producer, not a replacement for the bridge.
residue: [],
});
// An app capture off the route is untouched — identity included, as before.
expect(app).toEqual(runnerResult());
},
);
// Losing the bridge fast path must never be silent: an unprovable probe still owes the caller a
// warning and an identity that cannot be compared against a bridge publication.
test('a probe that cannot answer discloses the skipped bridge and stays incomparable', async () => {
const route = createAppleSnapshotRoute(platformRuntimeHostFixture(), {
source: sourceReturning(bridgeAcquisition()),
resolveTarget: vi.fn(async () => target),
systemSurfacePresent: async () => 'unknown',
});
const result = await route.capture(ios, input, signal(), async () => runnerResult());
expect(result.warnings).toEqual([
'Simulator AX snapshot unavailable (system-surface-probe-unavailable); used XCTest for an unverified app generation.',
]);
expect(result.comparisonIdentity).toMatchObject({
producer: 'apple-runner',
residue: [
{ kind: 'unknown-generation', captureId: expect.any(String) },
{ kind: 'fallback-source', producer: 'apple-runner' },
],
});
});
test('typed bridge failure falls back once and disables retries for that app generation', async () => {
const source = sourceReturning({
stage: 'failed',
@@ -364,6 +584,14 @@ function runnerResult() {
return { backend: 'xctest' as const, producer: 'apple-runner' as const, nodes: [] };
}
/** The runner's capture OF the sheet: it stamps the surface it actually served onto the result. */
function surfaceRunnerResult() {
return {
...runnerResult(),
systemSurface: { bundleId: presentSurface.host.bundleId, kind: presentSurface.host.kind },
};
}
function signal(): AbortSignal {
return new AbortController().signal;
}
+180 -14
View File
@@ -32,9 +32,23 @@ import {
type SimulatorSnapshotTarget,
type SimulatorSnapshotTargetResolver,
} from './snapshot-target.ts';
import {
createSystemSurfacePresenceProbe,
type SystemSurfacePresenceProbe,
} from './system-surface-presence.ts';
type SnapshotFallback = (input: CaptureSnapshotInput) => Promise<SnapshotResult>;
/** Why this capture left the bridge: a system surface the bridge cannot see was on screen. */
const SYSTEM_SURFACE_PRESENTED = 'system-surface-presented';
/**
* The same decision, against the host-side probe's one documented false positive: the surface host
* process outlives the dismissal of its sheet (see `system-surface-presence.ts`), so the bridge was
* skipped for a surface the runner then did not serve.
*/
const SYSTEM_SURFACE_HOST_LINGERING = 'system-surface-host-lingering';
export type AppleSnapshotRoute = LaunchObservationPort &
Readonly<{
capture(
@@ -51,10 +65,12 @@ export function createAppleSnapshotRoute(
options: Readonly<{
source?: SimulatorSnapshotSource;
resolveTarget?: SimulatorSnapshotTargetResolver;
systemSurfacePresent?: SystemSurfacePresenceProbe;
}> = {},
): AppleSnapshotRoute {
const source = options.source ?? createSimulatorSnapshotSource();
const resolveTarget = options.resolveTarget ?? createSimulatorSnapshotTargetResolver();
const systemSurfacePresent = options.systemSurfacePresent ?? createSystemSurfacePresenceProbe();
const disabledGenerations = new Set<string>();
const latestGeneration = new Map<string, string>();
/**
@@ -78,7 +94,32 @@ export function createAppleSnapshotRoute(
awaitObservable: observation.awaitObservable,
shutdown: async () => await source.close(),
capture: async (device, input, signal, fallback) => {
if (!isEligible(device, input)) return await fallback(input);
if (!isEligible(device, input)) return await captureOffRoute(device, input, fallback);
// A system surface (e.g. the web sign-in sheet) presented over the app is invisible to the
// host AX bridge — the app is still the AX primaryApp, so the bridge would serve the occluded
// app tree as if healthy (#2438). The XCTest runner can see and drive the surface, so route
// this capture to it. The runner serves the surface only while it is genuinely foreground and
// otherwise serves the app, so this is correct even while a dismissed host lingers. Anything
// but a proven `absent` takes the runner: an unproven probe must not fall through to a bridge
// capture that would answer confidently from the occluded app tree.
const surfacePresence = await systemSurfacePresent(device, signal);
if (surfacePresence === 'unknown') {
// The probe could not answer. Take the runner rather than a bridge capture that would
// answer confidently from the occluded app tree — but say so: silently losing the bridge
// fast path, with no warning and a comparable identity, would be its own defect.
return await runFallback(
device.id,
input,
fallback,
appLineage(device.id, input),
requestFor(input),
'system-surface-probe-unavailable',
[unknownGenerationResidue()],
);
}
if (surfacePresence !== 'absent') {
return await runSurfaceFallback(device, input, fallback, surfacePresence.host.bundleId);
}
let target: SimulatorSnapshotTarget;
try {
target = await resolveTargetForObservation(host, resolveTarget, device, input, signal);
@@ -86,16 +127,24 @@ export function createAppleSnapshotRoute(
signal.throwIfAborted();
emitRouteDiagnostic('target-resolution-failed', device, undefined, error);
return await runFallback(
device.id,
input,
fallback,
{ targetId: `${device.id}:${input.options!.appBundleId!}` },
appLineage(device.id, input),
requestFor(input),
'target-resolution-failed',
[unknownGenerationResidue()],
);
}
if (isBridgeDisabled(target)) {
return await runFallback(input, fallback, target, requestFor(input), 'circuit-disabled');
return await runFallback(
device.id,
input,
fallback,
target,
requestFor(input),
'circuit-disabled',
);
}
const request = requestFor(input);
@@ -218,6 +267,7 @@ async function fallbackAfterFailure(
failure.details,
);
return await runFallback(
failedTarget.udid,
input,
fallback,
identity.lineage,
@@ -228,15 +278,117 @@ async function fallbackAfterFailure(
}
async function runFallback(
deviceId: string,
input: CaptureSnapshotInput,
fallback: SnapshotFallback,
lineage: IosSnapshotLineage,
request: ReturnType<typeof createIosSnapshotRequest>,
reason: string,
residue: readonly IosAcquisitionResidue[] = [],
): Promise<SnapshotResult> {
return stampFallback(deviceId, await fallback(input), lineage, request, reason, residue);
}
/**
* The `present` path's capture. The probe answers about a host PROCESS and stays positive while a
* dismissed host lingers, while the runner answers about the screen so only the reason is decided
* here, from what the runner served. The identity comes from the shared stamping point, which reads
* the same stamp: if the probe decided identity instead, an app capture taken in the lingering
* window would be lineaged to the host and compare EQUAL to the sheet capture before the dismissal,
* which is exactly the transition a post-gesture poll must not miss (#2438).
*
* `detectedHost` is therefore evidence, not identity: it names the host the probe matched so a
* lingering window is legible in the daemon log instead of looking like a missing bridge capture.
*/
async function runSurfaceFallback(
device: DeviceInfo,
input: CaptureSnapshotInput,
fallback: SnapshotFallback,
detectedHost: string,
): Promise<SnapshotResult> {
const result = await fallback(input);
const comparisonIdentity: IosSnapshotComparisonIdentity = Object.freeze({
const reason = result.systemSurface ? SYSTEM_SURFACE_PRESENTED : SYSTEM_SURFACE_HOST_LINGERING;
if (!result.systemSurface) {
emitRouteDiagnostic(reason, device, undefined, undefined, { detectedHost });
}
return stampFallback(device.id, result, appLineage(device.id, input), requestFor(input), reason);
}
/**
* A capture the route cannot plan a pinned backend or a custom-actions read, see
* {@link isEligible} still reaches the XCTest runner, and the runner serves a presented system
* surface on those paths too. Such a capture describes the surface rather than the app, so it is
* identified like any other surface capture: without an identity it would fall back to legacy
* presentation matching, where a sheet and the app read as the same presentation and could
* corroborate a tap across the two (#2438). An app capture off the route carries no identity, as
* before: the route planned nothing about it.
*/
async function captureOffRoute(
device: DeviceInfo,
input: CaptureSnapshotInput,
fallback: SnapshotFallback,
): Promise<SnapshotResult> {
const result = await fallback(input);
const served = result.systemSurface;
if (!served) return result;
return {
...result,
comparisonIdentity: runnerComparisonIdentity(
surfaceLineage(device.id, served.bundleId),
requestFor(input),
[],
),
};
}
/**
* The one place a runner fallback's identity is decided. The runner stamps the surface it actually
* served onto its result, and that stamp is the authority over the app lineage the route planned:
* a capture OF a system surface is identified by that surface whatever reason sent the route here.
* Deriving this per call site is what let `circuit-disabled` stamp app lineage onto a sheet capture,
* so a sheet and the app could compare equal and corroborate a tap across the two (#2438).
*
* `reason` survives either way why the bridge was skipped is independent of what the runner found.
* App-generation evidence leaves with the app lineage it describes: a surface is not an app
* generation, and a per-capture residue id would make two captures of the same sheet incomparable
* with each other too.
*/
function stampFallback(
deviceId: string,
result: SnapshotResult,
lineage: IosSnapshotLineage,
request: ReturnType<typeof createIosSnapshotRequest>,
reason: string,
residue: readonly IosAcquisitionResidue[] = [],
): SnapshotResult {
const served = result.systemSurface;
return {
...result,
comparisonIdentity: runnerComparisonIdentity(
served ? surfaceLineage(deviceId, served.bundleId) : lineage,
request,
[...(served ? [] : residue), { kind: 'fallback-source', producer: 'apple-runner' }],
),
warnings: [...(result.warnings ?? []), fallbackWarning(reason, lineage, served !== undefined)],
};
}
/** The app generation the route planned this capture against. */
function appLineage(deviceId: string, input: CaptureSnapshotInput): IosSnapshotLineage {
return { targetId: `${deviceId}:${input.options!.appBundleId!}` };
}
/** A served system surface, which is identified by the surface host and by no app generation. */
function surfaceLineage(deviceId: string, bundleId: string): IosSnapshotLineage {
return { targetId: `${deviceId}:${bundleId}` };
}
function runnerComparisonIdentity(
lineage: IosSnapshotLineage,
request: ReturnType<typeof createIosSnapshotRequest>,
residue: readonly IosAcquisitionResidue[],
): IosSnapshotComparisonIdentity {
return Object.freeze({
producer: 'apple-runner',
intent: request.acquisitionIntent,
lineage: Object.freeze({
@@ -244,18 +396,32 @@ async function runFallback(
...(lineage.generation ? { generation: lineage.generation } : {}),
}),
presentationKey: buildIosSnapshotPresentationKey(request),
residue: Object.freeze([
...residue,
{ kind: 'fallback-source', producer: 'apple-runner' } as const,
]),
residue: Object.freeze([...residue]),
});
}
/**
* A presented system surface is not a bridge failure: the bridge is healthy and simply cannot see
* the surface, so it is inapplicable here rather than unavailable and the capture belongs to that
* surface, not to an app generation. A lingering host is the same decision over a surface the runner
* did not serve, so that sentence says what the capture holds instead. Every other reason keeps the
* unavailable sentence.
*/
function fallbackWarning(reason: string, lineage: IosSnapshotLineage, served: boolean): string {
if (reason === SYSTEM_SURFACE_PRESENTED) {
return `Simulator AX snapshot inapplicable (${reason}); used XCTest to read the system surface presented over the app.`;
}
if (reason === SYSTEM_SURFACE_HOST_LINGERING) {
return `Simulator AX snapshot inapplicable (${reason}); used XCTest, which read app content: the system surface host process was still running but no longer presenting.`;
}
// The bridge was skipped for its own reason and the runner then found a surface over the app. The
// app-generation sentence would describe a capture this is not, so the reason keeps its wording
// and the content sentence says what arrived.
if (served) {
return `Simulator AX snapshot unavailable (${reason}); used XCTest, which read the system surface presented over the app.`;
}
const generation = lineage.generation ? 'this app generation' : 'an unverified app generation';
const warning = `Simulator AX snapshot unavailable (${reason}); used XCTest for ${generation}.`;
return {
...result,
comparisonIdentity,
warnings: [...(result.warnings ?? []), warning],
};
return `Simulator AX snapshot unavailable (${reason}); used XCTest for ${generation}.`;
}
type FallbackIdentity = Readonly<{
@@ -0,0 +1,140 @@
import { beforeEach, expect, test, vi } from 'vitest';
import type { DeviceInfo } from '@agent-device/kernel/device';
vi.mock('./core/tool-provider.ts', async (importOriginal) => {
const actual = await importOriginal<typeof import('./core/tool-provider.ts')>();
return { ...actual, runAppleToolCommand: vi.fn(actual.runAppleToolCommand) };
});
import { runAppleToolCommand } from './core/tool-provider.ts';
import { IOS_SYSTEM_SURFACE_HOSTS } from '@agent-device/contracts/ios-system-surface';
import { createSystemSurfacePresenceProbe } from './system-surface-presence.ts';
const mockRunCmd = vi.mocked(runAppleToolCommand);
const sim = {
platform: 'apple',
appleOs: 'ios',
id: 'UDID-1',
name: 'iPhone',
kind: 'simulator',
target: 'mobile',
booted: true,
} as const satisfies DeviceInfo;
type ProbeReply = { exitCode: number; stdout: string };
/** Routes the two probe commands independently so each failure mode can be exercised alone. */
function stubProbes(replies: { pgrep?: ProbeReply | Error; ps?: ProbeReply | Error }): void {
mockRunCmd.mockImplementation(async (command: string) => {
const reply = command === 'pgrep' ? replies.pgrep : replies.ps;
if (reply === undefined) throw new Error(`unexpected probe command ${command}`);
if (reply instanceof Error) throw reply;
return { exitCode: reply.exitCode, stdout: reply.stdout, stderr: '' };
});
}
const RUNNING = { exitCode: 0, stdout: '900\n' } as const;
const NOT_RUNNING = { exitCode: 1, stdout: '' } as const;
/** The verdict a matched host produces: the host travels with it, to become the capture's lineage. */
const PRESENT = { kind: 'present', host: IOS_SYSTEM_SURFACE_HOSTS[0]! } as const;
const scopedTo = (udid: string): ProbeReply => ({
exitCode: 0,
stdout: `/…/SafariViewService.app/SafariViewService SIMULATOR_UDID=${udid}`,
});
beforeEach(() => {
vi.resetAllMocks();
});
test('a host process scoped to this device is present, and names the host it matched', async () => {
stubProbes({ pgrep: RUNNING, ps: scopedTo('UDID-1') });
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toEqual(PRESENT);
});
test('the same host running for another device is absent', async () => {
stubProbes({ pgrep: RUNNING, ps: scopedTo('OTHER') });
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toBe('absent');
});
test('no host process at all is absent without reading any environment', async () => {
stubProbes({ pgrep: NOT_RUNNING });
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toBe('absent');
expect(mockRunCmd).toHaveBeenCalledOnce();
});
test('a non-simulator is absent without probing', async () => {
const probe = createSystemSurfacePresenceProbe();
await expect(probe({ ...sim, kind: 'device' } as DeviceInfo)).resolves.toBe('absent');
expect(mockRunCmd).not.toHaveBeenCalled();
});
// A probe that cannot answer must not be reported as absence: absence sends the capture to the AX
// bridge, which would answer confidently from the occluded app tree (#2438).
test('a failing process scan is unknown, never absent', async () => {
stubProbes({ pgrep: new Error('pgrep unavailable') });
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toBe('unknown');
});
test('an unreadable process environment is unknown, never absent', async () => {
stubProbes({ pgrep: RUNNING, ps: new Error('ps failed') });
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toBe('unknown');
});
// A successful read that carries no device scope at all proves nothing: reporting it as absence
// would route a live sheet to the occluded app tree.
test('a process environment with no device scope at all is unknown, never absent', async () => {
stubProbes({
pgrep: RUNNING,
ps: { exitCode: 0, stdout: '/…/SafariViewService.app/SafariViewService' },
});
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toBe('unknown');
});
test('an empty process environment read is unknown, never absent', async () => {
stubProbes({ pgrep: RUNNING, ps: { exitCode: 0, stdout: '' } });
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toBe('unknown');
});
test('a non-zero process scan exit that is not "no match" is unknown', async () => {
stubProbes({ pgrep: { exitCode: 2, stdout: '' } });
await expect(createSystemSurfacePresenceProbe()(sim)).resolves.toBe('unknown');
});
// The regression thymikee named: a sheet opened right after an app capture must be seen by the very
// next capture, so absence is never memoized.
test('absence is not cached: a sheet opening within the TTL is seen immediately', async () => {
let clock = 1_000;
const probe = createSystemSurfacePresenceProbe(() => clock);
stubProbes({ pgrep: NOT_RUNNING });
await expect(probe(sim)).resolves.toBe('absent');
stubProbes({ pgrep: RUNNING, ps: scopedTo('UDID-1') });
clock += 10; // far inside the memo TTL
await expect(probe(sim)).resolves.toEqual(PRESENT);
});
test('unknown is not cached either', async () => {
let clock = 1_000;
const probe = createSystemSurfacePresenceProbe(() => clock);
stubProbes({ pgrep: new Error('transient') });
await expect(probe(sim)).resolves.toBe('unknown');
stubProbes({ pgrep: RUNNING, ps: scopedTo('UDID-1') });
clock += 10;
await expect(probe(sim)).resolves.toEqual(PRESENT);
});
test('a positive observation is memoized within the TTL and re-probed after it', async () => {
let clock = 1_000;
const probe = createSystemSurfacePresenceProbe(() => clock);
stubProbes({ pgrep: RUNNING, ps: scopedTo('UDID-1') });
await probe(sim);
const callsAfterFirst = mockRunCmd.mock.calls.length;
await expect(probe(sim)).resolves.toEqual(PRESENT);
expect(mockRunCmd.mock.calls.length).toBe(callsAfterFirst);
clock += 2_000; // past the TTL
stubProbes({ pgrep: NOT_RUNNING });
await expect(probe(sim)).resolves.toBe('absent');
});
@@ -0,0 +1,137 @@
import type { DeviceInfo } from '@agent-device/kernel/device';
import {
IOS_SYSTEM_SURFACE_HOSTS,
type IosSystemSurfaceHost,
} from '@agent-device/contracts/ios-system-surface';
import { runAppleToolCommand } from './core/tool-provider.ts';
/**
* Whether a registered iOS system surface host is running for a Simulator (issue #2438).
*
* `unknown` is deliberate and is NOT collapsed into `absent`: the local host AX bridge cannot see a
* system surface presented over the app while the sheet is up the app remains the AX `primaryApp`,
* so the bridge serves the (occluded) app tree, which looks perfectly healthy. Answering `absent`
* when we do not actually know would silently route such a capture to the bridge and return that
* occluded tree. Callers route anything that is not `absent` to the XCTest runner, which
* authoritatively serves the surface only while it is genuinely foreground and otherwise serves the
* app, so the cost of a false positive is one runner capture instead of a bridge capture.
*
* A positive answer names the host it matched, because the capture it routes describes that host's
* surface rather than the app: the caller stamps the host into the capture's comparison lineage, so
* a surface capture cannot compare equal to an app capture.
*/
export type SystemSurfacePresence =
| Readonly<{ kind: 'present'; host: IosSystemSurfaceHost }>
| 'absent'
| 'unknown';
export type SystemSurfacePresenceProbe = (
device: DeviceInfo,
signal?: AbortSignal,
) => Promise<SystemSurfacePresence>;
/**
* Only a positive observation is memoized. Absence must never be cached: a sheet opens between two
* captures, and a cached `absent` would send the very next capture to the bridge and answer from the
* occluded app tree. Re-probing on every non-present capture is what keeps that window closed.
*/
const PRESENT_MEMO_TTL_MS = 1_000;
const PROBE_TIMEOUT_MS = 3_000;
export function createSystemSurfacePresenceProbe(
now: () => number = Date.now,
): SystemSurfacePresenceProbe {
const observedPresent = new Map<string, { at: number; host: IosSystemSurfaceHost }>();
return async (device, signal) => {
if (device.kind !== 'simulator') return 'absent';
const seen = observedPresent.get(device.id);
if (seen !== undefined && now() - seen.at < PRESENT_MEMO_TTL_MS) {
return { kind: 'present', host: seen.host };
}
observedPresent.delete(device.id);
const presence = await probeSystemSurfacePresence(device, signal);
if (presence !== 'absent' && presence !== 'unknown') {
observedPresent.set(device.id, { at: now(), host: presence.host });
}
return presence;
};
}
async function probeSystemSurfacePresence(
device: DeviceInfo,
signal: AbortSignal | undefined,
): Promise<SystemSurfacePresence> {
let sawUnknown = false;
for (const host of IOS_SYSTEM_SURFACE_HOSTS) {
const pids = await hostProcessIds(host.processExecutable, signal);
if (pids === 'unknown') {
sawUnknown = true;
continue;
}
for (const pid of pids) {
const scoped = await isProcessScopedToDevice(pid, device.id, signal);
if (scoped === 'unknown') sawUnknown = true;
else if (scoped) return { kind: 'present', host };
}
}
return sawUnknown ? 'unknown' : 'absent';
}
/**
* Pids of a host's simulator app binary, host-wide. `pgrep` exits 1 with no output when nothing
* matches, which is a real negative; any other failure is `unknown`. This stays cheap in the common
* case no match means one small process-table scan and no environment read at all.
*/
async function hostProcessIds(
processExecutable: string,
signal: AbortSignal | undefined,
): Promise<number[] | 'unknown'> {
const result = await runProbe('pgrep', ['-f', processExecutable], signal);
if (result === 'unknown') return 'unknown';
if (result.exitCode === 1) return [];
if (result.exitCode !== 0) return 'unknown';
return result.stdout
.split('\n')
.map((line) => Number.parseInt(line.trim(), 10))
.filter((pid) => Number.isSafeInteger(pid) && pid > 0);
}
/**
* Whether one host process belongs to this Simulator. Simulator processes carry `SIMULATOR_UDID` in
* their environment, so the device scope is exact rather than "some booted simulator". A process
* that vanished between the scan and this read reports `unknown` rather than a negative, because a
* dead pid and an unreadable one are indistinguishable here.
*/
async function isProcessScopedToDevice(
pid: number,
deviceId: string,
signal: AbortSignal | undefined,
): Promise<boolean | 'unknown'> {
const result = await runProbe('ps', ['eww', '-p', String(pid), '-o', 'command='], signal);
if (result === 'unknown' || result.exitCode !== 0) return 'unknown';
if (result.stdout.includes(`SIMULATOR_UDID=${deviceId}`)) return true;
// Only a scope naming a DIFFERENT device is a real negative. A read that carries no device scope
// at all proves nothing — the environment may have been truncated or withheld — and reporting it
// as absence would route a live sheet to the occluded app tree.
return result.stdout.includes('SIMULATOR_UDID=') ? false : 'unknown';
}
async function runProbe(
command: string,
args: string[],
signal: AbortSignal | undefined,
): Promise<{ exitCode: number; stdout: string } | 'unknown'> {
try {
// Through the Apple tool provider, like every other host probe here, so a stubbed provider
// answers instead of spawning a real process.
const result = await runAppleToolCommand(command, args, {
allowFailure: true,
timeoutMs: PROBE_TIMEOUT_MS,
...(signal ? { signal } : {}),
});
return { exitCode: result.exitCode, stdout: result.stdout };
} catch {
// A timeout or spawn failure is not evidence of absence.
return 'unknown';
}
}
@@ -58,6 +58,7 @@
"@agent-device/contracts/interactor-operation-catalog",
"@agent-device/contracts/interactor-types",
"@agent-device/contracts/ios-snapshot",
"@agent-device/contracts/ios-system-surface",
"@agent-device/contracts/is-predicate",
"@agent-device/contracts/keyboard",
"@agent-device/contracts/keyboard-runtime",
@@ -0,0 +1,89 @@
// Doubles the `--settle` routes share: the backend whose captures the settle loop
// reads, and the small trees it serves. `settle.test.ts` drives the loop itself and
// `post-action-surface.test.ts` the cross-surface comparison over it, so both read
// these builders from here instead of keeping a private copy.
//
// Budgets are injected (fake clock) — no real waiting.
import type { AgentDeviceBackend, BackendSnapshotResult } from '../../../../backend.ts';
import type { SnapshotState } from '@agent-device/kernel/snapshot';
import { createLocalArtifactAdapter } from '../../../../io.ts';
import {
createAgentDevice,
createMemorySessionStore,
localCommandPolicy,
} from '../../../../runtime.ts';
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
export function createFakeClock(stepMs = 300): {
now: () => number;
sleep: (ms: number) => Promise<void>;
advance: (ms: number) => void;
} {
let elapsed = 0;
return {
now: () => elapsed,
sleep: async (ms: number) => {
elapsed += ms > 0 ? ms : stepMs;
},
advance: (ms: number) => {
elapsed += ms;
},
};
}
export function buttonSnapshot(): SnapshotState {
return makeSnapshotState([
{
index: 0,
depth: 0,
type: 'Button',
label: 'Continue',
rect: { x: 10, y: 20, width: 100, height: 40 },
hittable: true,
},
]);
}
// Five nodes so a settled capture clears the tiny-tree readiness heuristic.
export function welcomeSnapshot(): SnapshotState {
return makeSnapshotState(
['Welcome!', 'Next', 'Back', 'Home', 'Menu'].map((label, index) => ({
index,
depth: index === 0 ? 0 : 1,
...(index === 0 ? {} : { parentIndex: 0 }),
type: index === 0 ? 'StaticText' : 'Button',
label,
rect: { x: 10, y: 20 + index * 60, width: 100, height: 40 },
hittable: true,
})),
);
}
export function createSettleDevice(params: {
stored: SnapshotState;
captureSnapshot: () => Promise<BackendSnapshotResult> | BackendSnapshotResult;
tap?: () => Promise<Record<string, unknown>>;
clock?: ReturnType<typeof createFakeClock>;
appBundleId?: string;
}): ReturnType<typeof createAgentDevice> {
return createAgentDevice({
backend: {
platform: 'ios',
captureSnapshot: async () => await params.captureSnapshot(),
tap: async () => (params.tap ? await params.tap() : { ok: true }),
fill: async () => ({ ok: true }),
longPress: async () => ({ ok: true }),
} satisfies AgentDeviceBackend,
artifacts: createLocalArtifactAdapter(),
sessions: createMemorySessionStore([
{
name: 'default',
snapshot: params.stored,
...(params.appBundleId ? { appBundleId: params.appBundleId } : {}),
},
]),
policy: localCommandPolicy(),
clock: params.clock ?? createFakeClock(),
});
}
+2 -2
View File
@@ -336,14 +336,14 @@ function recordedDragTarget(target: ResolvedInteractionTarget): DragRecordingTar
!('selectorChain' in target) ||
!target.selectorChain?.length ||
!target.node ||
!target.preActionNodes
!target.preAction
) {
return undefined;
}
return {
selectorChain: target.selectorChain,
node: target.node,
preActionNodes: target.preActionNodes,
preActionNodes: target.preAction.nodes,
};
}
@@ -8,7 +8,9 @@ import {
createMemorySessionStore,
localCommandPolicy,
} from '../../../runtime.ts';
import type { Point } from '@agent-device/kernel/snapshot';
import type { Point, SnapshotState } from '@agent-device/kernel/snapshot';
import { summarizeAxEvidence } from '@agent-device/capture-kit/snapshot-evidence';
import { IOS_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/ios-system-surface';
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
import {
coveredByTabBarSnapshot,
@@ -130,7 +132,7 @@ test('runtime fill uses backend ref primitive without resolving snapshot geometr
// ADR 0012 decision 3: the preflight's guard lookup supplies the
// record-time evidence node on the runtime result.
assert.equal(result.node?.ref, 'e1');
assert.ok(Array.isArray(result.preActionNodes));
assert.ok(Array.isArray(result.preAction?.nodes));
assert.deepEqual(result.target, { kind: 'ref', ref: '@e1' });
assert.equal(result.text, 'hello');
assert.deepEqual(result.backendResult, { ref: 'e1', text: 'hello' });
@@ -491,6 +493,92 @@ test('runtime press with verify reports changedFromBefore true when the post-act
assert.equal(result.evidence?.nodeCount, 2);
});
// #2438: iOS serves a web sign-in sheet (com.apple.SafariViewService) IN PLACE over a
// still-foreground app, so a capture of the sheet and a capture of the app describe DIFFERENT
// surfaces. `--verify` must not answer "did this change?" by comparing their node digests.
const WEB_SIGN_IN_SHEET_BUNDLE_ID = 'com.apple.SafariViewService';
function webSignInSheetSnapshot(labels: string[]): SnapshotState {
return {
...makeSnapshotState(
labels.map((label, index) => ({
index,
depth: 0,
type: 'Button',
label,
rect: { x: 10, y: 20 + index * 50, width: 200, height: 40 },
hittable: true,
})),
),
iosSystemSurfaceBundleId: WEB_SIGN_IN_SHEET_BUNDLE_ID,
};
}
test('runtime press with verify discloses the surface change when a sign-in sheet replaces the app', async () => {
let captureCount = 0;
const device = createInteractionDevice(selectorSnapshot(), {
captureSnapshot: async () => {
captureCount += 1;
// Capture 1 resolves the target against app content; capture 2 is the verify capture, taken
// after the sheet was presented in place over that app.
return {
snapshot:
captureCount === 1
? selectorSnapshot()
: webSignInSheetSnapshot(['Sign in with Example', 'Cancel']),
};
},
tap: async () => ({ ok: true }),
});
const result = await device.interactions.press(selector('label=Continue'), {
session: 'default',
verify: true,
});
assert.equal(result.kind, 'selector');
assert.deepEqual(result.evidence?.surfaceChange, {
from: 'app',
to: WEB_SIGN_IN_SHEET_BUNDLE_ID,
disclosure: IOS_SYSTEM_SURFACE_DISCLOSURE,
});
// The transition is what changed, not a digest comparison between two different surfaces.
assert.equal(result.evidence?.changedFromBefore, true);
});
test('runtime press with verify reports the sheet leaving even when the two digests coincide', async () => {
const sheet: SnapshotState = {
...selectorSnapshot(),
iosSystemSurfaceBundleId: WEB_SIGN_IN_SHEET_BUNDLE_ID,
};
const app = selectorSnapshot();
// The premise this test exists for: the sheet tree and the app tree that replaces it digest
// identically, so a digest comparison would report "nothing changed" across a whole-surface
// replacement. The surfaces, not the digests, answer the question.
assert.equal(summarizeAxEvidence(sheet.nodes).digest, summarizeAxEvidence(app.nodes).digest);
let captureCount = 0;
const device = createInteractionDevice(sheet, {
captureSnapshot: async () => {
captureCount += 1;
return { snapshot: captureCount === 1 ? sheet : app };
},
tap: async () => ({ ok: true }),
});
const result = await device.interactions.press(selector('label=Continue'), {
session: 'default',
verify: true,
});
assert.equal(result.kind, 'selector');
assert.equal(result.evidence?.surfaceChange?.from, WEB_SIGN_IN_SHEET_BUNDLE_ID);
assert.equal(result.evidence?.surfaceChange?.to, 'app');
assert.match(result.evidence?.surfaceChange?.disclosure ?? '', /sign-in sheet/);
// The sheet is gone, so the standing "is presented over the app" sentence cannot be the one used.
assert.notEqual(result.evidence?.surfaceChange?.disclosure, IOS_SYSTEM_SURFACE_DISCLOSURE);
assert.equal(result.evidence?.changedFromBefore, true);
});
test('runtime fill without verify omits evidence entirely', async () => {
const device = createInteractionDevice(fillableSnapshot(), {
fill: async () => ({ ok: true }),
@@ -1,6 +1,4 @@
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
import type { AgentDeviceRuntime, CommandContext } from '../../../runtime-contract.ts';
import { summarizeAxEvidence } from '@agent-device/capture-kit/snapshot-evidence';
import type {
InteractionEvidence,
ResolvedInteractionTarget,
@@ -8,6 +6,7 @@ import type {
SettleParams,
} from '@agent-device/contracts/interaction';
import { captureInteractionSnapshot } from './resolution.ts';
import { summarizePostActionEvidence, surfaceScopedNodes } from './post-action-surface.ts';
import { settleAfterInteraction, settleEvidence } from './settle.ts';
type ObservedResult<T extends object> = T & {
@@ -82,10 +81,7 @@ async function observeAfterInteraction(
resolved,
});
const evidence = params.verify
? settleEvidence(
outcome.settledNodes,
'preActionNodes' in resolved ? resolved.preActionNodes : undefined,
)
? settleEvidence(outcome.settledCapture, resolved.preAction)
: undefined;
return { settle: outcome.observation, ...(evidence ? { evidence } : {}) };
}
@@ -98,23 +94,17 @@ async function observeAfterInteraction(
* Post-action side of `--verify` (#1047): one interactive-only capture through
* the same capture helper the resolution path already uses, digested and then
* discarded. The node tree itself is never attached to the result, only the
* cheap summary.
* cheap summary. A missing baseline still yields `changedFromBefore: false`
* no baseline, no claim.
*/
async function captureVerifyEvidence(
runtime: AgentDeviceRuntime,
options: CommandContext,
resolved: ResolvedInteractionTarget,
): Promise<InteractionEvidence | undefined> {
const preActionNodes: SnapshotNode[] | undefined =
'preActionNodes' in resolved ? resolved.preActionNodes : undefined;
try {
const capture = await captureInteractionSnapshot(runtime, options, true);
const after = summarizeAxEvidence(capture.snapshot.nodes);
// No pre-action baseline means we cannot claim a change happened; default
// to false rather than asserting a change we did not actually observe.
const changedFromBefore =
preActionNodes !== undefined && after.digest !== summarizeAxEvidence(preActionNodes).digest;
return { ...after, changedFromBefore };
return summarizePostActionEvidence(surfaceScopedNodes(capture.snapshot), resolved.preAction);
} catch {
return undefined;
}
@@ -150,6 +140,9 @@ function hasMaterialPostActionChange(view: {
settle?: SettleObservation;
}): boolean {
if (view.evidence?.changedFromBefore === true) return true;
// A surface replacement (#2438) carries no settled diff by design, and it is the most material
// post-action change there is: the screen is now a different surface.
if (view.settle?.surfaceChange !== undefined) return true;
const summary = view.settle?.diff?.summary;
return !!summary && (summary.additions > 0 || summary.removals > 0);
}
@@ -0,0 +1,179 @@
import assert from 'node:assert/strict';
import { test } from 'vitest';
import type { SnapshotState } from '@agent-device/kernel/snapshot';
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
import { summarizeAxEvidence } from '@agent-device/capture-kit/snapshot-evidence';
import { IOS_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/ios-system-surface';
import { selector } from './selector-read-utils.ts';
import {
buttonSnapshot,
createSettleDevice,
welcomeSnapshot,
} from './__tests__/settle-device-fixtures.ts';
// #2438 cross-surface settle: iOS serves a web sign-in sheet
// (com.apple.SafariViewService) IN PLACE over a still-foreground app, so a settled
// capture of the sheet and a pre-action capture of the app describe DIFFERENT
// surfaces. A diff between them is a whole-surface replacement dressed as change
// within one surface — and, since diff presence is what issues refs, it would hand
// the caller refs for that claim.
const WEB_SIGN_IN_SHEET_BUNDLE_ID = 'com.apple.SafariViewService';
function webSignInSheetSnapshot(labels: string[]): SnapshotState {
return {
...makeSnapshotState(
labels.map((label, index) => ({
index,
depth: 0,
type: 'Button',
label,
rect: { x: 10, y: 20 + index * 50, width: 200, height: 40 },
hittable: true,
})),
),
iosSystemSurfaceBundleId: WEB_SIGN_IN_SHEET_BUNDLE_ID,
};
}
// Five labels so a settled sheet clears the tiny-tree readiness heuristic, and a
// `Continue` control so the same selector also resolves on the sheet.
const WEB_SIGN_IN_SHEET_LABELS = [
'Continue',
'Sign in with Example',
'Email',
'Password',
'Cancel',
];
test('press --settle attaches no diff across an app-to-sheet surface change and discloses it', async () => {
const before = buttonSnapshot();
const sheet = webSignInSheetSnapshot(WEB_SIGN_IN_SHEET_LABELS);
let captures = 0;
const device = createSettleDevice({
stored: before,
captureSnapshot: () => {
captures += 1;
// Capture 1 = selector resolution (app baseline). Captures 2+ = the settle
// loop, reading the sheet now presented over that app.
return { snapshot: captures === 1 ? before : sheet };
},
});
const result = await device.interactions.press(selector('label=Continue'), {
session: 'default',
settle: {},
});
const settle = result.settle;
assert.ok(settle);
assert.equal(settle.settled, true);
assert.deepEqual(settle.surfaceChange, {
from: 'app',
to: WEB_SIGN_IN_SHEET_BUNDLE_ID,
disclosure: IOS_SYSTEM_SURFACE_DISCLOSURE,
});
// No same-surface claim: no diff, so no issued refs, and no tail either.
assert.equal(settle.diff, undefined);
assert.equal(settle.tail, undefined);
assert.match(settle.hint ?? '', /different surfaces/);
assert.match(settle.hint ?? '', /take a snapshot/i);
// Disclosed, not hidden: the settled sheet still becomes the stored observation
// a follow-up snapshot reads.
const stored = (await device.sessions.get('default')) as { snapshot?: SnapshotState };
assert.equal(stored.snapshot?.iosSystemSurfaceBundleId, WEB_SIGN_IN_SHEET_BUNDLE_ID);
});
test('press --settle attaches no diff across a sheet-to-app surface change and discloses it', async () => {
const sheet = webSignInSheetSnapshot(WEB_SIGN_IN_SHEET_LABELS);
const app = welcomeSnapshot();
let captures = 0;
const device = createSettleDevice({
stored: sheet,
captureSnapshot: () => {
captures += 1;
// The baseline is the sheet this press acts on; the settled tree is the app
// content that returns once the sheet completes and dismisses itself.
return { snapshot: captures === 1 ? sheet : app };
},
});
const result = await device.interactions.press(selector('label=Continue'), {
session: 'default',
settle: {},
});
const settle = result.settle;
assert.ok(settle);
assert.equal(settle.settled, true);
assert.equal(settle.surfaceChange?.from, WEB_SIGN_IN_SHEET_BUNDLE_ID);
assert.equal(settle.surfaceChange?.to, 'app');
assert.match(settle.surfaceChange?.disclosure ?? '', /sign-in sheet/);
// The sheet is gone, so the standing "is presented over the app" sentence cannot
// be the one used.
assert.notEqual(settle.surfaceChange?.disclosure, IOS_SYSTEM_SURFACE_DISCLOSURE);
assert.equal(settle.diff, undefined);
assert.equal(settle.tail, undefined);
assert.match(settle.hint ?? '', /different surfaces/);
});
test('press --settle --verify reports one app-to-sheet surface change on both payloads', async () => {
const before = buttonSnapshot();
const sheet = webSignInSheetSnapshot(WEB_SIGN_IN_SHEET_LABELS);
let captures = 0;
const device = createSettleDevice({
stored: before,
captureSnapshot: () => {
captures += 1;
return { snapshot: captures === 1 ? before : sheet };
},
});
const result = await device.interactions.press(selector('label=Continue'), {
session: 'default',
settle: {},
verify: true,
});
// The settle loop's final capture doubles as the verify evidence source, so the
// shared evidence path must refuse the cross-surface digest comparison too.
assert.deepEqual(result.evidence?.surfaceChange, {
from: 'app',
to: WEB_SIGN_IN_SHEET_BUNDLE_ID,
disclosure: IOS_SYSTEM_SURFACE_DISCLOSURE,
});
assert.equal(result.evidence?.changedFromBefore, true);
assert.equal(result.settle?.surfaceChange?.to, WEB_SIGN_IN_SHEET_BUNDLE_ID);
assert.equal(result.settle?.diff, undefined);
});
test('press --settle --verify reports a sheet-to-app surface change when the digests coincide', async () => {
const sheet: SnapshotState = {
...welcomeSnapshot(),
iosSystemSurfaceBundleId: WEB_SIGN_IN_SHEET_BUNDLE_ID,
};
const app = welcomeSnapshot();
// The premise this test exists for: the sheet tree and the app tree that replaces
// it digest identically, so a digest comparison would report "nothing changed"
// across a whole-surface replacement.
assert.equal(summarizeAxEvidence(sheet.nodes).digest, summarizeAxEvidence(app.nodes).digest);
let captures = 0;
const device = createSettleDevice({
stored: sheet,
captureSnapshot: () => {
captures += 1;
return { snapshot: captures === 1 ? sheet : app };
},
});
const result = await device.interactions.press(selector('label=Next'), {
session: 'default',
settle: {},
verify: true,
});
assert.equal(result.evidence?.surfaceChange?.from, WEB_SIGN_IN_SHEET_BUNDLE_ID);
assert.equal(result.evidence?.surfaceChange?.to, 'app');
assert.equal(result.evidence?.changedFromBefore, true);
assert.equal(result.settle?.surfaceChange?.from, WEB_SIGN_IN_SHEET_BUNDLE_ID);
assert.equal(result.settle?.diff, undefined);
});
@@ -0,0 +1,75 @@
import type { SnapshotState } from '@agent-device/kernel/snapshot';
import { summarizeAxEvidence } from '@agent-device/capture-kit/snapshot-evidence';
import type {
InteractionEvidence,
PostActionSurfaceChange,
SurfaceScopedNodes,
} from '@agent-device/contracts/interaction';
import { iosSystemSurfaceTransitionDisclosure } from '@agent-device/contracts/ios-system-surface';
/**
* The surface question every post-action observation owes (#2438): iOS serves an in-place system
* surface a web sign-in sheet hosted out of the app's process over a still-foreground app, so a
* capture of the sheet and a capture of the app describe DIFFERENT surfaces. Comparing their node
* digests yields a meaningless "changed" verdict, and diffing them presents a whole-surface
* replacement as an in-surface diff, with refs.
*
* Both `--verify` and `--settle` route their comparison through this module, so the refusal and its
* disclosure cannot hold on one route and drop on the other.
*/
/** How a capture of ordinary app content names its surface in a {@link PostActionSurfaceChange}. */
const APP_SURFACE = 'app';
/** Mints the one carried value from a capture: the nodes together with the surface they describe. */
export function surfaceScopedNodes(snapshot: SnapshotState): SurfaceScopedNodes {
return {
nodes: snapshot.nodes,
...(snapshot.iosSystemSurfaceBundleId
? { surfaceBundleId: snapshot.iosSystemSurfaceBundleId }
: {}),
};
}
/**
* The transition between the two compared captures, or `undefined` when both describe the same
* surface and an ordinary same-surface comparison is therefore valid. An absent baseline has no
* surface to disagree with, so it reports no transition.
*/
export function resolvePostActionSurfaceChange(
baseline: SurfaceScopedNodes | undefined,
after: SurfaceScopedNodes,
): PostActionSurfaceChange | undefined {
if (!baseline || baseline.surfaceBundleId === after.surfaceBundleId) return undefined;
return {
from: baseline.surfaceBundleId ?? APP_SURFACE,
to: after.surfaceBundleId ?? APP_SURFACE,
disclosure: iosSystemSurfaceTransitionDisclosure(after.surfaceBundleId),
};
}
/**
* `--verify` evidence for one post-action capture (#1047). Same surface: the digest comparison
* against the pre-action baseline, as before. Cross-surface: no digest comparison is made the
* surface itself was replaced, which is a change by construction, and `surfaceChange` discloses
* that this is what `changedFromBefore` reports.
*/
export function summarizePostActionEvidence(
after: SurfaceScopedNodes,
baseline: SurfaceScopedNodes | undefined,
): InteractionEvidence {
const summary = summarizeAxEvidence(after.nodes);
const surfaceChange = resolvePostActionSurfaceChange(baseline, after);
if (surfaceChange) return { ...summary, changedFromBefore: true, surfaceChange };
const changedFromBefore =
baseline !== undefined && summary.digest !== summarizeAxEvidence(baseline.nodes).digest;
return { ...summary, changedFromBefore };
}
/**
* What `--settle` says instead of a diff it refuses to build: the transition itself, then the
* observation the caller should take to read the surface that is now on screen.
*/
export function crossSurfaceSettleHint(change: PostActionSurfaceChange): string {
return `${change.disclosure} The settled tree and the pre-action tree describe different surfaces (${change.from}${change.to}), so no settled diff or refs are shown; take a snapshot to read the current surface.`;
}
+32 -18
View File
@@ -34,6 +34,7 @@ import {
type OffscreenScrollDirection,
} from '@agent-device/capture-kit/mobile-snapshot-semantics';
import { truncateUtf8 } from './truncate-utf8.ts';
import { surfaceScopedNodes } from './post-action-surface.ts';
import type {
InteractionTarget,
PointTarget,
@@ -42,6 +43,7 @@ import type {
ResolutionDiagnosticEntry,
ResolutionDisclosure,
ResolvedInteractionTarget,
SurfaceScopedNodes,
} from '@agent-device/contracts/interaction';
import { INTERACTION_ERROR_REASONS } from '@agent-device/selectors/interaction-error';
import type {
@@ -224,11 +226,11 @@ async function resolvePointInteractionTarget(
...(warning ? { warning } : {}),
};
}
const preActionNodes = await tryCaptureEvidenceBaseline(runtime, options);
const baseline = await tryCaptureEvidenceBaseline(runtime, options);
return {
kind: 'point',
point: { x: target.x, y: target.y },
...(preActionNodes ? { preActionNodes } : {}),
...(baseline ? { preAction: baseline } : {}),
...(warning ? { warning } : {}),
};
}
@@ -236,10 +238,10 @@ async function resolvePointInteractionTarget(
async function tryCaptureEvidenceBaseline(
runtime: AgentDeviceRuntime,
options: CommandContext,
): Promise<SnapshotNode[] | undefined> {
): Promise<SurfaceScopedNodes | undefined> {
try {
const capture = await captureInteractionSnapshot(runtime, options, true);
return capture.snapshot.nodes;
return surfaceScopedNodes(capture.snapshot);
} catch {
// Evidence is best-effort: a failed baseline capture must not fail the
// action itself. Post-action evidence (if any) will simply omit
@@ -249,7 +251,10 @@ async function tryCaptureEvidenceBaseline(
}
/** The node a ref target acts on, plus the tree the shared guards read it against. */
type RefResolution = { nodes: SnapshotState['nodes']; resolved: ResolvedRefNode };
type RefResolution = {
tree: SurfaceScopedNodes;
resolved: ResolvedRefNode;
};
/**
* #1654: adopt the node the caller already resolved instead of resolving the
@@ -276,7 +281,12 @@ function adoptPreresolvedRefTarget(
);
}
return {
nodes: preresolved.nodes,
tree: {
nodes: preresolved.nodes,
...(preresolved.iosSystemSurfaceBundleId
? { surfaceBundleId: preresolved.iosSystemSurfaceBundleId }
: {}),
},
resolved: buildRefResolution(ref, preresolved.node, 'exact'),
};
}
@@ -287,7 +297,7 @@ async function readRefResolution(
target: Extract<InteractionTarget, { kind: 'ref' }>,
): Promise<RefResolution> {
const capture = await resolveSnapshotForRef(runtime, options, target);
return { nodes: capture.snapshot.nodes, resolved: capture.resolved };
return { tree: surfaceScopedNodes(capture.snapshot), resolved: capture.resolved };
}
async function resolveRefInteractionTarget(
@@ -296,9 +306,10 @@ async function resolveRefInteractionTarget(
target: Extract<InteractionTarget, { kind: 'ref' }>,
params: ResolveInteractionTargetParams,
): Promise<ResolvedInteractionTarget> {
const { nodes, resolved } = params.preresolvedTarget
const { tree, resolved } = params.preresolvedTarget
? adoptPreresolvedRefTarget(target, params.preresolvedTarget)
: await readRefResolution(runtime, options, target);
const nodes = tree.nodes;
// #1542: point/response read from the returned (possibly rescue-patched) node.
const visibleNode = await runInteractionPipelineStages({
policy: params.pipeline,
@@ -324,7 +335,7 @@ async function resolveRefInteractionTarget(
...describeResolvedInteractionNode(
runtime,
visibleNode,
nodes,
tree,
params.action,
resolved.resolution,
),
@@ -389,7 +400,7 @@ async function resolveSelectorInteractionTarget(
...describeResolvedInteractionNode(
runtime,
visibleNode,
capture.snapshot.nodes,
surfaceScopedNodes(capture.snapshot),
params.action,
buildSelectorResolutionDisclosure(resolved, capture.snapshot.nodes),
),
@@ -532,11 +543,12 @@ function buildResolutionDiagnosticEntry(
// Shared tail of a resolved ref/selector interaction target: the node itself
// plus everything derived from it for the response. Every response field
// describes the DISPATCHED node — the #1280 retarget rides only on the
// `recordingTarget` side channel below.
// `recordingTarget` side channel below. `tree` is the capture the node was
// resolved from, and becomes the pre-action baseline this publishes.
function describeResolvedInteractionNode(
runtime: AgentDeviceRuntime,
node: SnapshotNode,
nodes: SnapshotState['nodes'],
tree: SurfaceScopedNodes,
action: InteractionAction,
resolution: ResolutionDisclosure,
): {
@@ -545,10 +557,11 @@ function describeResolvedInteractionNode(
refLabel: string | undefined;
targetHittable?: boolean;
hint?: string;
preActionNodes: SnapshotState['nodes'];
preAction: SurfaceScopedNodes;
resolution: ResolutionDisclosure;
recordingTarget?: RecordingTargetOverride;
} {
const nodes = tree.nodes;
return {
node,
selectorChain: buildSelectorChainForNode(node, runtime.backend.platform, {
@@ -557,7 +570,7 @@ function describeResolvedInteractionNode(
}),
refLabel: resolveRefLabel(node, nodes),
...describeNonHittableTarget(node, action),
preActionNodes: nodes,
preAction: tree,
resolution,
...pressRecordingTargetOverride(runtime, node, nodes, action),
};
@@ -961,11 +974,12 @@ export async function preflightNativeRefInteraction(
targetHittable?: boolean;
hint?: string;
node?: SnapshotNode;
preActionNodes?: SnapshotNode[];
preAction?: SurfaceScopedNodes;
}> {
const session = await runtime.sessions.get(options.session ?? 'default');
const nodes = session?.snapshot?.nodes;
if (!nodes || normalizeRef(target.ref) === null) return {};
const storedSnapshot = session?.snapshot;
const nodes = storedSnapshot?.nodes;
if (!storedSnapshot || !nodes || normalizeRef(target.ref) === null) return {};
const resolved = tryResolveRefNode(nodes, target.ref, {
fallbackLabel: target.fallbackLabel ?? '',
});
@@ -994,7 +1008,7 @@ export async function preflightNativeRefInteraction(
// ADR 0012 decision 3: the guard lookup above doubles as the record-time
// evidence source for the fast path, at zero extra capture cost.
node: visibleNode,
preActionNodes: nodes,
preAction: surfaceScopedNodes(storedSnapshot),
};
}
@@ -56,7 +56,7 @@ test('settle recovers the session baseline when resolved target evidence is abse
assert.equal(outcome.observation.settled, true);
assert.equal(
outcome.settledNodes?.some((node) => node.label === 'action file'),
outcome.settledCapture?.nodes.some((node) => node.label === 'action file'),
false,
);
assert.ok(
@@ -99,7 +99,7 @@ test('settle uses the authorized ref frame instead of a polluted evidence captur
kind: 'ref',
point: { x: 201, y: 795 },
target: { kind: 'ref', ref: '@e5' },
preActionNodes: elementTransientRoomSnapshot.nodes,
preAction: { nodes: elementTransientRoomSnapshot.nodes },
},
quietMs: 500,
timeoutMs: 5_000,
@@ -108,7 +108,7 @@ test('settle uses the authorized ref frame instead of a polluted evidence captur
assert.equal(outcome.observation.settled, true);
assert.equal(
outcome.settledNodes?.some((node) => node.label === 'action file'),
outcome.settledCapture?.nodes.some((node) => node.label === 'action file'),
false,
);
assert.ok(
@@ -1,6 +1,6 @@
import assert from 'node:assert/strict';
import { test } from 'vitest';
import type { AgentDeviceBackend, BackendSnapshotResult } from '../../../backend.ts';
import type { AgentDeviceBackend } from '../../../backend.ts';
import type { SnapshotState } from '@agent-device/kernel/snapshot';
import { createLocalArtifactAdapter } from '../../../io.ts';
import {
@@ -11,84 +11,17 @@ import {
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
import { ref, selector } from './selector-read-utils.ts';
import { buildSettleTailEntries, NEVER_SETTLED_HINT } from './settle.ts';
import {
buttonSnapshot,
createFakeClock,
createSettleDevice,
welcomeSnapshot,
} from './__tests__/settle-device-fixtures.ts';
import { readSnapshotQualityVerdict } from '@agent-device/capture-kit/snapshot-quality-verdict';
// #1101 --settle: quiet-window settle loop composition on the interaction
// commands. Budgets are injected (fake clock) — no real waiting.
function createFakeClock(stepMs = 300): {
now: () => number;
sleep: (ms: number) => Promise<void>;
advance: (ms: number) => void;
} {
let elapsed = 0;
return {
now: () => elapsed,
sleep: async (ms: number) => {
elapsed += ms > 0 ? ms : stepMs;
},
advance: (ms: number) => {
elapsed += ms;
},
};
}
function buttonSnapshot(): SnapshotState {
return makeSnapshotState([
{
index: 0,
depth: 0,
type: 'Button',
label: 'Continue',
rect: { x: 10, y: 20, width: 100, height: 40 },
hittable: true,
},
]);
}
// Five nodes so a settled capture clears the tiny-tree readiness heuristic.
function welcomeSnapshot(): SnapshotState {
return makeSnapshotState(
['Welcome!', 'Next', 'Back', 'Home', 'Menu'].map((label, index) => ({
index,
depth: index === 0 ? 0 : 1,
...(index === 0 ? {} : { parentIndex: 0 }),
type: index === 0 ? 'StaticText' : 'Button',
label,
rect: { x: 10, y: 20 + index * 60, width: 100, height: 40 },
hittable: true,
})),
);
}
function createSettleDevice(params: {
stored: SnapshotState;
captureSnapshot: () => Promise<BackendSnapshotResult> | BackendSnapshotResult;
tap?: () => Promise<Record<string, unknown>>;
clock?: ReturnType<typeof createFakeClock>;
appBundleId?: string;
}): ReturnType<typeof createAgentDevice> {
return createAgentDevice({
backend: {
platform: 'ios',
captureSnapshot: async () => await params.captureSnapshot(),
tap: async () => (params.tap ? await params.tap() : { ok: true }),
fill: async () => ({ ok: true }),
longPress: async () => ({ ok: true }),
} satisfies AgentDeviceBackend,
artifacts: createLocalArtifactAdapter(),
sessions: createMemorySessionStore([
{
name: 'default',
snapshot: params.stored,
...(params.appBundleId ? { appBundleId: params.appBundleId } : {}),
},
]),
policy: localCommandPolicy(),
clock: params.clock ?? createFakeClock(),
});
}
test('press --settle returns the settled diff and stores the settled tree', async () => {
const before = buttonSnapshot();
const after = welcomeSnapshot();
+67 -39
View File
@@ -11,13 +11,14 @@ import {
collectSettleChromeRefs,
withoutSettleChrome,
} from '@agent-device/capture-kit/snapshot-chrome';
import { summarizeAxEvidence } from '@agent-device/capture-kit/snapshot-evidence';
import type {
InteractionEvidence,
PostActionSurfaceChange,
ResolvedInteractionTarget,
SettleObservation,
SettleParams,
SettleTailEntry,
SurfaceScopedNodes,
} from '@agent-device/contracts/interaction';
import type { RuntimeCommand } from '../../runtime-types.ts';
import type { CapturedSnapshot } from './selector-read-shared.ts';
@@ -28,6 +29,12 @@ import {
TINY_STABLE_TREE_HINT,
TINY_STABLE_TREE_NODE_COUNT,
} from './stable-capture.ts';
import {
crossSurfaceSettleHint,
resolvePostActionSurfaceChange,
summarizePostActionEvidence,
surfaceScopedNodes,
} from './post-action-surface.ts';
/**
* `--settle` (#1101): after a mutating command, wait for the UI to go quiet
@@ -49,8 +56,8 @@ import {
export type SettleOutcome = {
observation: SettleObservation;
/** Nodes of the final capture; doubles as the `--verify` evidence source. */
settledNodes?: SnapshotNode[];
/** The final capture; doubles as the `--verify` evidence source. */
settledCapture?: SurfaceScopedNodes;
};
// Changed-lines bound: the settled diff is the response payload, and unbounded
@@ -76,15 +83,15 @@ export async function settleAfterInteraction(
): Promise<SettleOutcome> {
return await settleAfterAction(runtime, options, {
...params,
baselineNodes: await resolveBaselineNodes(runtime, options, params.resolved),
baseline: await resolveSettleBaseline(runtime, options, params.resolved),
actionPoint: params.resolved.point,
});
}
export type SettleObservationCommandOptions = CommandContext &
SettleParams & {
/** The pre-action tree the settled diff is taken against. */
baselineNodes: SnapshotNode[];
/** The pre-action tree the settled diff is taken against, and the surface it describes. */
baseline: SurfaceScopedNodes;
};
/**
@@ -106,17 +113,18 @@ export const settleObservationCommand: RuntimeCommand<
* interaction entry point only the two things a resolution would have
* supplied come from the caller:
*
* - `baselineNodes` is the diff baseline. On the generic route it is the
* session's STORED pre-action tree, which may be several commands older than
* the action, so the diff honestly reads "settled tree vs the last tree you
* observed" rather than press's freshly resolved pre-action capture.
* - `baseline` is the diff baseline and the surface it describes. On the generic
* route the nodes are the session's STORED pre-action tree, which may be
* several commands older than the action, so the diff honestly reads "settled
* tree vs the last tree you observed" rather than press's freshly resolved
* pre-action capture.
* - `actionPoint` is absent: with no point there is nothing to self-echo
* against, so the tail's self-echo exclusion simply never fires.
*/
async function settleAfterAction(
runtime: AgentDeviceRuntime,
options: CommandContext,
params: SettleParams & { baselineNodes: SnapshotNode[]; actionPoint?: Point },
params: SettleParams & { baseline: SurfaceScopedNodes; actionPoint?: Point },
): Promise<SettleOutcome> {
const quietMs = params.quietMs ?? DEFAULT_STABLE_QUIET_MS;
const timeoutMs = params.timeoutMs ?? DEFAULT_STABLE_TIMEOUT_MS;
@@ -126,7 +134,7 @@ async function settleAfterAction(
quietMs,
timeoutMs,
resetBudgetOnPrivateAxRecovery: true,
broadTransitionBaselineNodes: params.baselineNodes,
broadTransitionBaselineNodes: params.baseline.nodes,
});
return await readSettledOutcome(runtime, options, params, base, outcome);
} catch (error) {
@@ -145,7 +153,7 @@ async function settleAfterAction(
async function readSettledOutcome(
runtime: AgentDeviceRuntime,
options: CommandContext,
params: { baselineNodes: SnapshotNode[]; actionPoint?: Point },
params: { baseline: SurfaceScopedNodes; actionPoint?: Point },
base: SettleObservation,
outcome: Awaited<ReturnType<typeof runStableCaptureLoop>>,
): Promise<SettleOutcome> {
@@ -164,10 +172,17 @@ async function readSettledOutcome(
};
}
const { stored, session } = await storeSettledSnapshot(runtime, options, outcome.lastCapture);
const settledNodes = outcome.lastCapture.snapshot.nodes;
const settledCapture = surfaceScopedNodes(outcome.lastCapture.snapshot);
const settledNodes = settledCapture.nodes;
// A settled capture of an in-place system surface (a web sign-in sheet) and a pre-action capture
// of the app describe different surfaces (#2438). The diff below would then be a whole-surface
// replacement presented as change within one surface, refs included, so it is refused and the
// transition is disclosed instead.
const surfaceChange = resolvePostActionSurfaceChange(params.baseline, settledCapture);
return {
observation: {
...observation,
...(surfaceChange ? { surfaceChange } : {}),
// The diff (with its added-line refs) is only attached when the settled
// tree actually became the stored session snapshot: those refs must be
// valid against the tree the next @ref command resolves on. The daemon
@@ -175,17 +190,17 @@ async function readSettledOutcome(
// captures are intentionally diff-less: they are not a stable
// observation, so surfacing refs would invite agents to act on
// advisory state.
...(outcome.settled && stored
...(outcome.settled && stored && !surfaceChange
? buildSettleDiffAndTail(
params.baselineNodes,
params.baseline.nodes,
settledNodes,
params.actionPoint,
session?.appBundleId,
)
: {}),
...resolveSettleHint(outcome, stored, settledNodes.length),
...resolveSettleHint(outcome, stored, settledNodes.length, surfaceChange),
},
settledNodes,
settledCapture,
};
}
@@ -195,21 +210,18 @@ async function readSettledOutcome(
* final capture there is no evidence best-effort, like verify itself.
*/
export function settleEvidence(
settledNodes: SnapshotNode[] | undefined,
preActionNodes: SnapshotNode[] | undefined,
settledCapture: SurfaceScopedNodes | undefined,
baseline: SurfaceScopedNodes | undefined,
): InteractionEvidence | undefined {
if (!settledNodes) return undefined;
const after = summarizeAxEvidence(settledNodes);
const changedFromBefore =
preActionNodes !== undefined && after.digest !== summarizeAxEvidence(preActionNodes).digest;
return { ...after, changedFromBefore };
if (!settledCapture) return undefined;
return summarizePostActionEvidence(settledCapture, baseline);
}
async function resolveBaselineNodes(
async function resolveSettleBaseline(
runtime: AgentDeviceRuntime,
options: CommandContext,
resolved: ResolvedInteractionTarget,
): Promise<SnapshotNode[]> {
): Promise<SurfaceScopedNodes> {
const session = await runtime.sessions.get(options.session ?? 'default');
// A ref is authorized against the stored ref frame. Keep that visible presentation as the
// transition baseline: a best-effort evidence recapture can recover through private AX and see
@@ -220,7 +232,7 @@ async function resolveBaselineNodes(
// and pre-frame sessions.
return (
authorizedRefBaseline(resolved, session) ??
evidenceBaseline(resolved) ??
nonEmptyBaseline(resolved.preAction) ??
sessionBaseline(session)
);
}
@@ -228,22 +240,21 @@ async function resolveBaselineNodes(
function authorizedRefBaseline(
resolved: ResolvedInteractionTarget,
session: CommandSessionRecord | undefined,
): SnapshotNode[] | undefined {
): SurfaceScopedNodes | undefined {
if (resolved.kind !== 'ref') return undefined;
return nonEmptyNodes(session?.refFrameSnapshot?.nodes);
const frame = session?.refFrameSnapshot;
return frame ? nonEmptyBaseline(surfaceScopedNodes(frame)) : undefined;
}
function evidenceBaseline(resolved: ResolvedInteractionTarget): SnapshotNode[] | undefined {
if (!('preActionNodes' in resolved)) return undefined;
return nonEmptyNodes(resolved.preActionNodes);
function sessionBaseline(session: CommandSessionRecord | undefined): SurfaceScopedNodes {
const tree = session?.refFrameSnapshot ?? session?.snapshot;
return tree ? surfaceScopedNodes(tree) : { nodes: [] };
}
function sessionBaseline(session: CommandSessionRecord | undefined): SnapshotNode[] {
return session?.refFrameSnapshot?.nodes ?? session?.snapshot?.nodes ?? [];
}
function nonEmptyNodes(nodes: SnapshotNode[] | undefined): SnapshotNode[] | undefined {
return nodes?.length ? nodes : undefined;
function nonEmptyBaseline(
baseline: SurfaceScopedNodes | undefined,
): SurfaceScopedNodes | undefined {
return baseline?.nodes.length ? baseline : undefined;
}
function buildSettleDiff(
@@ -405,10 +416,27 @@ function capSettleDiffLines<T extends { kind: string }>(changed: T[]): T[] {
return kept;
}
/**
* The settled observation's hint. A surface change (#2438) is stated alongside whatever the loop
* itself reports rather than in place of it: the transition explains the missing diff, and the
* loop's own verdict (stalled, never settled, sparse, tiny tree) still explains the capture.
*/
function resolveSettleHint(
outcome: { settled: boolean; stalled: boolean },
stored: boolean,
settledNodeCount: number,
surfaceChange: PostActionSurfaceChange | undefined,
): { hint?: string } {
const loopHint = resolveSettleLoopHint(outcome, stored, settledNodeCount).hint;
if (!surfaceChange) return loopHint === undefined ? {} : { hint: loopHint };
const surfaceHint = crossSurfaceSettleHint(surfaceChange);
return { hint: loopHint === undefined ? surfaceHint : `${surfaceHint} ${loopHint}` };
}
function resolveSettleLoopHint(
outcome: { settled: boolean; stalled: boolean },
stored: boolean,
settledNodeCount: number,
): { hint?: string } {
if (outcome.stalled) return { hint: SETTLE_CAPTURE_STALLED_HINT };
if (!outcome.settled) return { hint: NEVER_SETTLED_HINT };
+255 -5
View File
@@ -1,5 +1,11 @@
import { beforeEach, expect, test, vi } from 'vitest';
import type { CommandFlags } from '@agent-device/contracts/command';
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
import {
IOS_SYSTEM_SURFACE_DISCLOSURE,
iosSystemSurfaceTransitionDisclosure,
type IosSystemSurfaceProvenance,
} from '@agent-device/contracts/ios-system-surface';
import { makeIosSession } from '../../__tests__/test-utils/session-factories.ts';
import { makeSessionStore } from '../../__tests__/test-utils/store-factory.ts';
import { activateCompleteRefFrame, refFrameState } from '../ref-frame.ts';
@@ -70,6 +76,7 @@ type SettlePayload = {
lines: Array<{ kind: string; text: string; ref?: string }>;
};
tail?: Array<{ ref: string; role: string; label?: string }>;
surfaceChange?: { from: string; to: string; disclosure: string };
hint?: string;
};
@@ -119,17 +126,49 @@ const platformExecution = vi.fn(
/** Stands in for the device work a bound generic leaf performs, keyed by command name. */
const mockDispatch = vi.fn<(command: string) => Promise<Record<string, unknown>>>(async () => ({}));
function seedSession(sessionName: string, sessionStore: SessionStore): SessionState {
/**
* The session as it stands before the generic command runs. `baseline` is the STORED pre-action
* tree the settled diff is taken against, and the surface that capture described which is how a
* session whose last observation was an in-place system surface (#2438) is set up.
*/
function seedSession(
sessionName: string,
sessionStore: SessionStore,
baseline: GenericSettleTree = { nodes: BEFORE_NODES },
): SessionState {
const session = makeIosSession(sessionName);
setSessionSnapshot(
session,
buildSnapshotState({ nodes: BEFORE_NODES, backend: 'xctest', producer: 'apple-runner' }, {}),
);
setSessionSnapshot(session, buildSnapshotState(snapshotPayload(baseline), {}));
activateCompleteRefFrame(session);
sessionStore.set(sessionName, session);
return session;
}
/** A tree plus the surface a capture of it describes: app content, or an in-place system surface. */
type GenericSettleTree = {
nodes: RawSnapshotNode[];
systemSurface?: IosSystemSurfaceProvenance;
};
function snapshotPayload(tree: GenericSettleTree) {
return {
nodes: tree.nodes,
backend: 'xctest' as const,
producer: 'apple-runner' as const,
...(tree.systemSurface ? { systemSurface: tree.systemSurface } : {}),
};
}
/**
* Every settle capture reads `tree`; any other command answers `result`. Unlike
* {@link mockCommandDispatch} the captured tree carries its surface provenance, so the settled
* capture can describe a surface other than the one the stored baseline described.
*/
function mockSurfaceDispatch(tree: GenericSettleTree, result: Record<string, unknown> = {}) {
mockDispatch.mockImplementation(async (command) =>
command === 'snapshot' ? snapshotPayload(tree) : result,
);
}
async function dispatchGeneric(params: {
sessionName: string;
sessionStore: SessionStore;
@@ -250,6 +289,217 @@ test('back --settle answers with the settled diff alongside the command result',
});
});
// #2438 cross-surface settle on the GENERIC route: iOS serves a web sign-in sheet
// (com.apple.SafariViewService) IN PLACE over a still-foreground app, so a settled capture of the
// sheet and the session's stored pre-action tree of the app describe DIFFERENT surfaces. The diff
// above would then be a whole-surface replacement presented as change within one surface — and
// since the daemon treats `diff` presence as "this response issues refs", it would hand the caller
// refs for that claim. This route plumbs the baseline's surface identity through the
// `SurfaceScopedNodes` it hands the settle engine; the element-targeted route guarantees the same contract in
// `src/commands/interaction/runtime/post-action-surface.test.ts`, and these tests assert it for
// scroll/back so the two routes cannot drift.
//
// Both directions are covered because each falsifies a DIFFERENT half of that plumbing, and
// neither substitutes for the other: drop the baseline's surface identity here and only the
// sheet-to-app tests fail (an app baseline has no surface id to lose); drop the settled capture's
// surface identity and only the app-to-sheet tests fail.
const WEB_SIGN_IN_SHEET: IosSystemSurfaceProvenance = {
bundleId: 'com.apple.SafariViewService',
kind: 'web-auth',
};
/** Five nodes, so a settled sheet clears the tiny-tree readiness hint and the hint under test is
* the cross-surface one. No Application root: the sheet is hosted out of the app's process. */
const WEB_SIGN_IN_SHEET_NODES = [
'Sign in with Example',
'Email',
'Password',
'Continue',
'Cancel',
].map((label, index) => ({
index,
type: 'Button',
label,
rect: { x: 10, y: 120 + index * 50, width: 200, height: 44 },
hittable: true,
}));
/** The app content that returns once the sheet completes and dismisses itself, also past the
* tiny-tree count so the return direction asserts the same hint contract. */
const APP_CONTENT_NODES = [
{ index: 0, type: 'Application', rect: { x: 0, y: 0, width: 390, height: 844 } },
...['Load more', 'Profile', 'Settings', 'Sign out'].map((label, offset) => ({
index: offset + 1,
parentIndex: 0,
type: 'Button',
label,
rect: { x: 10, y: 20 + offset * 50, width: 120, height: 44 },
hittable: true,
})),
];
/**
* The cross-surface contract a settle route owes, asserted as one unit so `scroll` and `back`
* cannot drift from each other or from the targeted route: no diff is attached across the
* boundary therefore no tail and no issued refs the transition is disclosed, and the settle
* observation still reports its own verdict alongside that disclosure rather than instead of it.
*/
function expectCrossSurfaceSettle(
settle: SettlePayload,
change: { from: string; to: string; disclosure: string },
): void {
expect(settle.surfaceChange).toEqual(change);
expect(settle.diff).toBeUndefined();
expect(settle.tail).toBeUndefined();
// `refsGeneration` is folded in only when the settled diff published refs (ADR 0014); a refused
// diff publishes none, so the payload must not name a generation either.
expect(settle.refsGeneration).toBeUndefined();
expect(settle.hint).toContain('different surfaces');
expect(settle.hint).toMatch(/take a snapshot/i);
expect(settle.settled).toBe(true);
expect(settle.captures).toBeGreaterThanOrEqual(2);
expect(settle.quietMs).toBe(25);
expect(settle.timeoutMs).toBe(2_000);
}
/** No refs were published, so the frame the mutating leaf expired stays expired (ADR 0014). */
function expectNoPublishedRefFrame(sessionStore: SessionStore, sessionName: string): SessionState {
const stored = sessionStore.get(sessionName) as SessionState;
expect(refFrameState(stored)).toBe('expired');
return stored;
}
test('scroll --settle attaches no diff across an app-to-sheet surface change and discloses it', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'generic-settle-scroll-to-sheet';
const session = seedSession(sessionName, sessionStore);
// The stored pre-action tree is app content; every settle capture reads the sheet now presented
// over that still-foreground app.
mockSurfaceDispatch({ nodes: WEB_SIGN_IN_SHEET_NODES, systemSurface: WEB_SIGN_IN_SHEET });
const response = await dispatchGeneric({
sessionName,
sessionStore,
session,
command: 'scroll',
positionals: ['down'],
flags: { ...SETTLE_FLAGS },
});
expectCrossSurfaceSettle(expectOkData(response).settle as SettlePayload, {
from: 'app',
to: WEB_SIGN_IN_SHEET.bundleId,
disclosure: IOS_SYSTEM_SURFACE_DISCLOSURE,
});
// Disclosed, not hidden: the settled sheet still becomes the stored observation a follow-up
// snapshot reads — and the surface identity the NEXT command's baseline is built from.
const stored = expectNoPublishedRefFrame(sessionStore, sessionName);
expect(stored.snapshot?.iosSystemSurfaceBundleId).toBe(WEB_SIGN_IN_SHEET.bundleId);
});
test('scroll --settle attaches no diff across a sheet-to-app surface change and discloses it', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'generic-settle-scroll-from-sheet';
// The baseline is the sheet this scroll acts on; the settled tree is the app content that
// returns once the sheet completes and dismisses itself.
const session = seedSession(sessionName, sessionStore, {
nodes: WEB_SIGN_IN_SHEET_NODES,
systemSurface: WEB_SIGN_IN_SHEET,
});
mockSurfaceDispatch({ nodes: APP_CONTENT_NODES });
const response = await dispatchGeneric({
sessionName,
sessionStore,
session,
command: 'scroll',
positionals: ['down'],
flags: { ...SETTLE_FLAGS },
});
const settle = expectOkData(response).settle as SettlePayload;
expectCrossSurfaceSettle(settle, {
from: WEB_SIGN_IN_SHEET.bundleId,
to: 'app',
// The sheet is gone, so the standing "is presented over the app" sentence cannot be the one
// used — the transition disclosure has to say it left.
disclosure: iosSystemSurfaceTransitionDisclosure(undefined),
});
expect(settle.surfaceChange?.disclosure).not.toBe(IOS_SYSTEM_SURFACE_DISCLOSURE);
expect(settle.surfaceChange?.disclosure).toMatch(/sign-in sheet/);
const stored = expectNoPublishedRefFrame(sessionStore, sessionName);
expect(stored.snapshot?.iosSystemSurfaceBundleId).toBeUndefined();
});
test('back --settle attaches no diff across an app-to-sheet surface change and discloses it', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'generic-settle-back-to-sheet';
const session = seedSession(sessionName, sessionStore);
mockSurfaceDispatch(
{ nodes: WEB_SIGN_IN_SHEET_NODES, systemSurface: WEB_SIGN_IN_SHEET },
{
action: 'back',
mode: 'in-app',
message: 'Back',
},
);
const response = await dispatchGeneric({
sessionName,
sessionStore,
session,
command: 'back',
flags: { ...SETTLE_FLAGS },
});
const data = expectOkData(response);
// The refusal rides ALONGSIDE the command's own closed result shape, same as the diff does.
expect(data.action).toBe('back');
expect(data.message).toBe('Back');
expectCrossSurfaceSettle(data.settle as SettlePayload, {
from: 'app',
to: WEB_SIGN_IN_SHEET.bundleId,
disclosure: IOS_SYSTEM_SURFACE_DISCLOSURE,
});
const stored = expectNoPublishedRefFrame(sessionStore, sessionName);
expect(stored.snapshot?.iosSystemSurfaceBundleId).toBe(WEB_SIGN_IN_SHEET.bundleId);
});
test('back --settle attaches no diff across a sheet-to-app surface change and discloses it', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'generic-settle-back-from-sheet';
const session = seedSession(sessionName, sessionStore, {
nodes: WEB_SIGN_IN_SHEET_NODES,
systemSurface: WEB_SIGN_IN_SHEET,
});
mockSurfaceDispatch(
{ nodes: APP_CONTENT_NODES },
{
action: 'back',
mode: 'in-app',
message: 'Back',
},
);
const response = await dispatchGeneric({
sessionName,
sessionStore,
session,
command: 'back',
flags: { ...SETTLE_FLAGS },
});
const data = expectOkData(response);
expect(data.action).toBe('back');
expectCrossSurfaceSettle(data.settle as SettlePayload, {
from: WEB_SIGN_IN_SHEET.bundleId,
to: 'app',
disclosure: iosSystemSurfaceTransitionDisclosure(undefined),
});
const stored = expectNoPublishedRefFrame(sessionStore, sessionName);
expect(stored.snapshot?.iosSystemSurfaceBundleId).toBeUndefined();
});
test('the settle observation runs after the post-gesture stabilization marker', async () => {
const sessionStore = makeSessionStore();
const sessionName = 'generic-settle-order';
@@ -0,0 +1,31 @@
import {
buildIosSnapshotPresentationKey,
createIosSnapshotRequest,
iosSnapshotComparisonIdentityKey,
} from '@agent-device/capture-kit/ios-snapshot-planning';
import { IOS_SYSTEM_SURFACE_HOSTS } from '@agent-device/contracts/ios-system-surface';
/**
* The comparison keys the iOS snapshot route stamps on XCTest captures, built through the production
* key function: an app capture is lineaged to the app, a capture of an in-place system surface (a web
* sign-in sheet) to its HOST, because that is the surface it describes (#2438). Regressions that rely
* on a cross-surface pair being incomparable build both keys here, so they fail if the construction
* ever stops distinguishing them.
*/
export function appCaptureComparisonKey(deviceId: string, appBundleId: string): string {
return runnerComparisonKey(`${deviceId}:${appBundleId}`);
}
export function systemSurfaceCaptureComparisonKey(deviceId: string): string {
return runnerComparisonKey(`${deviceId}:${IOS_SYSTEM_SURFACE_HOSTS[0]!.bundleId}`);
}
function runnerComparisonKey(targetId: string): string {
return iosSnapshotComparisonIdentityKey({
producer: 'apple-runner',
intent: 'full',
lineage: { targetId },
presentationKey: buildIosSnapshotPresentationKey(createIosSnapshotRequest()),
residue: [{ kind: 'fallback-source', producer: 'apple-runner' }],
});
}
@@ -20,6 +20,10 @@ import {
pickupSnapshot,
pickupSnapshotWithExtraText,
} from './post-gesture-stabilization-fixtures.ts';
import {
appCaptureComparisonKey,
systemSurfaceCaptureComparisonKey,
} from './ios-comparison-key-fixture.ts';
// Pure verdict/classifier coverage (decidePostGestureStabilityVerdict) lives
// in the sibling post-gesture-stabilization-verdict.test.ts, and the
@@ -492,6 +496,87 @@ test('capturePostGestureStabilizedResult re-baselines instead of concluding when
assert.equal(staleAccepts + settled, 1);
});
// --- #2438: a system surface appearing or dismissing mid-poll is not comparable evidence ---
test('capturePostGestureStabilizedResult re-baselines when an in-place system surface appears (iOS)', async () => {
// A web sign-in sheet is hosted out of the app's process. Its tree and the app's tree describe
// different surfaces, so a quiet capture of the sheet says nothing about a gesture taken against
// the app. Both captures are XCTest: what keeps them incomparable is that the route lineages a
// surface capture to its host, so their comparison keys differ by construction (#2438).
vi.useFakeTimers();
const session = makeSession('ios');
session.snapshot = makeSnapshotState(pickupSnapshot(500).nodes, {
snapshotQuality: { state: 'healthy', backend: 'tree' },
comparisonKey: appCaptureComparisonKey(session.device.id, 'com.example.app'),
});
markPostGestureStabilization(session, 'scroll');
const capture = vi.fn(async () =>
makeSnapshotState(pickupSnapshot(500).nodes, {
snapshotQuality: { state: 'healthy', backend: 'tree' },
comparisonKey: systemSurfaceCaptureComparisonKey(session.device.id),
iosSystemSurfaceBundleId: 'com.apple.SafariViewService',
}),
);
const resultPromise = withDiagnosticsScope({}, async () => {
await capturePostGestureStabilizedResult({
session,
capture,
readSnapshot: (snapshot) => snapshot,
});
return {
rebased: countDiagnosticEventsByPhase(['post_gesture_snapshot_baseline_rebased']),
staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']),
settled: countDiagnosticEventsByPhase(['post_gesture_snapshot_stabilized']),
};
});
await vi.advanceTimersByTimeAsync(6_000);
const { rebased, staleAccepts, settled } = await resultPromise;
assert.equal(rebased, 1);
assert.equal(staleAccepts + settled, 1);
});
test('capturePostGestureStabilizedResult re-baselines when an in-place system surface dismisses (iOS)', async () => {
// The mirror case: the baseline was the sheet and the app returns underneath it.
vi.useFakeTimers();
const session = makeSession('ios');
session.snapshot = makeSnapshotState(pickupSnapshot(500).nodes, {
snapshotQuality: { state: 'healthy', backend: 'tree' },
comparisonKey: systemSurfaceCaptureComparisonKey(session.device.id),
iosSystemSurfaceBundleId: 'com.apple.SafariViewService',
});
markPostGestureStabilization(session, 'scroll');
const capture = vi.fn(async () =>
makeSnapshotState(pickupSnapshot(500).nodes, {
snapshotQuality: { state: 'healthy', backend: 'tree' },
comparisonKey: appCaptureComparisonKey(session.device.id, 'com.example.app'),
}),
);
const resultPromise = withDiagnosticsScope({}, async () => {
await capturePostGestureStabilizedResult({
session,
capture,
readSnapshot: (snapshot) => snapshot,
});
return {
rebased: countDiagnosticEventsByPhase(['post_gesture_snapshot_baseline_rebased']),
staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']),
settled: countDiagnosticEventsByPhase(['post_gesture_snapshot_stabilized']),
};
});
await vi.advanceTimersByTimeAsync(6_000);
const { rebased, staleAccepts, settled } = await resultPromise;
assert.equal(rebased, 1);
assert.equal(staleAccepts + settled, 1);
});
test('capturePostGestureStabilizedResult still distrusts a same-backend baseline match (iOS)', async () => {
// The guard above must not become a blanket escape hatch: when the backend is
// stable, an unchanged surface is still the stale-read signal #1542 added.
@@ -6,9 +6,13 @@ import { dispatchFindReadOnlyViaRuntime } from '../selector-runtime.ts';
import { dispatchWaitViaRuntime } from '../wait-runtime.ts';
import type { DaemonRequest, DaemonResponse } from '../daemon-request.ts';
import { ANDROID_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/android-system-surface-disclosure';
import { IOS_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/ios-system-surface';
import { snapshotRuntimeFixture } from './snapshot-runtime-fixture.ts';
import { makeSessionStore } from '../../__tests__/test-utils/store-factory.ts';
import { makeAndroidSession } from '../../__tests__/test-utils/session-factories.ts';
import {
makeAndroidSession,
makeIosSession,
} from '../../__tests__/test-utils/session-factories.ts';
import { platformResourceCleanup } from '../../platform-runtime-resource-cleanup.ts';
vi.mock('@agent-device/device-selection/dispatch-resolve', async (importOriginal) => {
@@ -257,3 +261,110 @@ test('sessionless wait timeout still discloses the occluding system surface', as
expect(sessionStore.get('default')).toBeUndefined();
expect(String(response.error.details?.hint)).toContain(ANDROID_SYSTEM_SURFACE_DISCLOSURE);
});
// --- #2438: an in-place iOS system surface (web sign-in sheet) discloses on the same shared seam ---
// The runner stamps `systemSurface` on a capture it served from the sheet; buildSnapshotState turns
// that into `iosSystemSurfaceBundleId`, which selector routes must carry and disclose.
const SHEET_SNAPSHOT_DATA = {
backend: 'xctest',
producer: 'apple-runner',
systemSurface: { bundleId: 'com.apple.SafariViewService', kind: 'web-auth' },
nodes: [
{
index: 0,
depth: 0,
type: 'Application',
label: 'Safari',
rect: { x: 0, y: 0, width: 402, height: 874 },
},
{
index: 1,
depth: 1,
parentIndex: 0,
type: 'Button',
label: 'LOGIN',
hittable: true,
rect: { x: 32, y: 366, width: 338, height: 50 },
},
],
};
function serveSheetCapture(): void {
legacyDispatchCapture.mockReset();
legacyDispatchCapture.mockImplementation(async (_device: unknown, command: string) =>
command === 'snapshot' ? SHEET_SNAPSHOT_DATA : {},
);
}
test('mutating find on an in-place system surface discloses it on the found outcome', async () => {
serveSheetCapture();
const sessionStore = makeSessionStore();
sessionStore.set('default', makeIosSession('default', { appBundleId: 'com.example.app' }));
const response = await handleFindCommands({
req: {
token: 't',
session: 'default',
command: 'find',
positionals: ['LOGIN', 'click'],
flags: {},
},
sessionName: 'default',
logPath: '/tmp/test.log',
sessionStore,
invoke: async () => ({ ok: true, data: {} }) as DaemonResponse,
...getRuntimeBindings(),
});
expect(response?.ok).toBe(true);
if (!response?.ok) return;
expect(String((response.data as Record<string, unknown>).warning)).toContain(
IOS_SYSTEM_SURFACE_DISCLOSURE,
);
});
test('mutating find that misses on an in-place system surface still discloses it', async () => {
serveSheetCapture();
const sessionStore = makeSessionStore();
sessionStore.set('default', makeIosSession('default', { appBundleId: 'com.example.app' }));
const response = await handleFindCommands({
req: {
token: 't',
session: 'default',
command: 'find',
positionals: ['Bakery list', 'click'],
flags: {},
},
sessionName: 'default',
logPath: '/tmp/test.log',
sessionStore,
invoke: async () => ({ ok: true, data: {} }) as DaemonResponse,
...getRuntimeBindings(),
});
expect(response?.ok).toBe(false);
if (response?.ok) return;
expect(String(response?.error.details?.hint)).toContain(IOS_SYSTEM_SURFACE_DISCLOSURE);
});
test('the shared disclosure helper reports an iOS system surface on both outcomes', () => {
const ok = withSystemSurfaceDisclosure(
{ ok: true, data: { found: true } },
{ iosSystemSurfaceBundleId: 'com.apple.SafariViewService' },
);
expect(ok.ok).toBe(true);
if (!ok.ok) return;
expect(String((ok.data as Record<string, unknown>).warning)).toContain(
IOS_SYSTEM_SURFACE_DISCLOSURE,
);
const failed = withSystemSurfaceDisclosure(
{ ok: false, error: { code: 'NOT_FOUND', message: 'no match' } },
{ iosSystemSurfaceBundleId: 'com.apple.SafariViewService' },
);
expect(failed.ok).toBe(false);
if (failed.ok) return;
expect(String(failed.error.details?.hint)).toContain(IOS_SYSTEM_SURFACE_DISCLOSURE);
});
@@ -363,6 +363,12 @@ export async function capturePostGestureStabilizedResult<T>(params: {
return outcome;
}
/**
* What makes two captures comparable at all. The iOS comparison key already carries the whole
* presentation identity, including the surface the capture described an in-place system surface (a
* web sign-in sheet) is captured under its own host lineage (#2438) so a sheet appearing or
* dismissing mid-poll reads as incomparable rather than as a stable surface.
*/
function snapshotComparisonKey(snapshot: SnapshotState | undefined): string | undefined {
return snapshot?.comparisonKey ?? snapshot?.snapshotQuality?.backend;
}
+15 -6
View File
@@ -1,6 +1,9 @@
import type { CommandFlags } from '@agent-device/contracts/command';
import type { SettleObservation, SettleParams } from '@agent-device/contracts/interaction';
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
import type {
SettleObservation,
SettleParams,
SurfaceScopedNodes,
} from '@agent-device/contracts/interaction';
import { commandSupportsSettleObservation } from '@agent-device/command-registry/registry';
import {
captureSnapshotForSession,
@@ -73,8 +76,14 @@ export function planGenericSettleObservation(
if (invalidSettleFlags) return { response: invalidSettleFlags };
const settle = readSettleRequest(params.flags);
if (!settle) return {};
const baselineNodes = params.session.snapshot?.nodes ?? [];
return { observe: async () => await observeSettled(params, settle, baselineNodes) };
const snapshot = params.session.snapshot;
const baseline: SurfaceScopedNodes = {
nodes: snapshot?.nodes ?? [],
...(snapshot?.iosSystemSurfaceBundleId
? { surfaceBundleId: snapshot.iosSystemSurfaceBundleId }
: {}),
};
return { observe: async () => await observeSettled(params, settle, baseline) };
}
/**
@@ -87,7 +96,7 @@ export function planGenericSettleObservation(
async function observeSettled(
context: GenericSettleContext,
settle: SettleParams,
baselineNodes: SnapshotNode[],
baseline: SurfaceScopedNodes,
): Promise<SettleObservation | undefined> {
const runtime = createGenericSettleRuntime(context);
if (!runtime) return undefined;
@@ -96,7 +105,7 @@ async function observeSettled(
// use for press/fill.
const observation = await runtime.interactions.settleObservation({
...settle,
baselineNodes,
baseline,
session: context.sessionName,
requestId: context.req.meta?.requestId,
});
@@ -23,6 +23,10 @@ import {
snapshotPayload,
} from './interaction-ios-tap-outcome-fixtures.ts';
import { snapshotRuntimeFixture } from '../../../__tests__/snapshot-runtime-fixture.ts';
import {
appCaptureComparisonKey,
systemSurfaceCaptureComparisonKey,
} from '../../../__tests__/ios-comparison-key-fixture.ts';
import { IOS_SIMULATOR } from '../../../../__tests__/test-utils/device-fixtures.ts';
import {
getRuntimeBindings,
@@ -316,6 +320,38 @@ test('a producer or generation switch cannot corroborate a failed tap', async ()
).resolves.toBeUndefined();
});
// An app baseline and an in-place system-surface capture (a web sign-in sheet) describe different
// surfaces, so neither may corroborate a tap taken against the other. The route lineages a surface
// capture to its host, so the refusal is the ordinary comparison-key mismatch above (#2438) — this
// pins that it still holds for the surface pair specifically.
test('a capture of a system surface cannot corroborate a tap taken against the app', async () => {
const sessionName = 'ios-system-surface-mismatch';
const sessionStore = makeSessionStore();
const baseline = snapshot(profileNodes);
baseline.comparisonKey = appCaptureComparisonKey(IOS_SIMULATOR.id, 'com.example.app');
const session = makeIosSession(sessionName, {
appBundleId: 'com.example.app',
snapshot: baseline,
});
sessionStore.set(sessionName, session);
const after = snapshot(imageViewerNodes);
after.comparisonKey = systemSurfaceCaptureComparisonKey(IOS_SIMULATOR.id);
after.iosSystemSurfaceBundleId = 'com.apple.SafariViewService';
await expect(
corroborateIosTapFailure({
error: new AppError('XCTEST_RECORDED_FAILURE', 'tap failed'),
command: 'click',
requestId: undefined,
flags: {},
session,
sessionStore,
contextFromFlags,
captureSnapshotForSession: async () => after,
}),
).resolves.toBeUndefined();
});
test('a sparse changed capture keeps the tap failure', async () => {
const sessionName = 'ios-sparse-tap-corroboration';
const sessionStore = makeSessionStore();
@@ -109,7 +109,7 @@ test('press @ref while recording attaches target-v1 evidence to the recorded act
expect(response?.ok).toBe(true);
if (response?.ok) {
expect(response.data).not.toHaveProperty('node');
expect(response.data).not.toHaveProperty('preActionNodes');
expect(response.data).not.toHaveProperty('preAction');
expect(response.data).not.toHaveProperty('targetEvidence');
}
@@ -125,7 +125,7 @@ test('press @ref while recording attaches target-v1 evidence to the recorded act
verification: 'verified',
});
expect(recordedAction?.result).not.toHaveProperty('node');
expect(recordedAction?.result).not.toHaveProperty('preActionNodes');
expect(recordedAction?.result).not.toHaveProperty('preAction');
});
test('press @ref without recording never computes target-v1 evidence', async () => {
@@ -139,7 +139,7 @@ test('press @ref without recording never computes target-v1 evidence', async ()
const recordedAction = sessionStore.get(sessionName)?.actions[0];
expect(recordedAction?.targetEvidence).toBeUndefined();
expect(recordedAction?.result).not.toHaveProperty('node');
expect(recordedAction?.result).not.toHaveProperty('preActionNodes');
expect(recordedAction?.result).not.toHaveProperty('preAction');
});
test('get text @ref while recording attaches target-v1 evidence to the recorded action, never to session history payloads', async () => {
@@ -285,7 +285,7 @@ function expectContainerBasedResponse(data: Record<string, unknown>): void {
expect(data.targetHittable).toBeUndefined();
expect(data).not.toHaveProperty('recordingTarget');
expect(data).not.toHaveProperty('node');
expect(data).not.toHaveProperty('preActionNodes');
expect(data).not.toHaveProperty('preAction');
}
// The recorded action entry — the .ad writer's source — carries the
@@ -12,6 +12,7 @@ export type FindTargetTree = {
nodes: SnapshotState['nodes'];
snapshotQuality?: SnapshotQualityVerdict;
systemSurfaceOnly?: boolean;
iosSystemSurfaceBundleId?: string;
};
/**
@@ -65,6 +66,7 @@ export function createFindTargetCapture(
nodes: snapshot.nodes,
snapshotQuality: snapshot.snapshotQuality,
systemSurfaceOnly: snapshot.systemSurfaceOnly,
iosSystemSurfaceBundleId: snapshot.iosSystemSurfaceBundleId,
};
};
}
+15 -1
View File
@@ -55,6 +55,8 @@ type ResolvedMatch = {
resolvedNode: SnapshotState['nodes'][number];
ref: string;
nodes: SnapshotState['nodes'];
/** The in-place iOS system surface the target capture described (#2438), if any. */
iosSystemSurfaceBundleId?: string;
actionFlags: Record<string, unknown>;
/**
* Set when find's row refuses this match as covered. Only the focus/type
@@ -172,6 +174,9 @@ export async function handleFindCommands(params: FindRouteInput): Promise<Daemon
resolvedNode,
ref,
nodes,
...(snapshotResult.iosSystemSurfaceBundleId
? { iosSystemSurfaceBundleId: snapshotResult.iosSystemSurfaceBundleId }
: {}),
actionFlags,
...(target.kind === 'occluded' ? { occludedNode: target.node } : {}),
};
@@ -217,7 +222,16 @@ async function dispatchFindAction(
* (occlusion, promotion, off-screen) still run on this node.
*/
function preresolvedTarget(match: ResolvedMatch): PreresolvedInteractionTarget {
return { ref: match.ref, node: match.resolvedNode, nodes: match.nodes };
return {
ref: match.ref,
node: match.resolvedNode,
nodes: match.nodes,
// #2438: the leaf's post-action verify/settle compares against this tree, so it must know
// whether the tree describes the app or an in-place system surface served over it.
...(match.iosSystemSurfaceBundleId
? { iosSystemSurfaceBundleId: match.iosSystemSurfaceBundleId }
: {}),
};
}
async function handleFindClick(ctx: FindContext, match: ResolvedMatch): Promise<DaemonResponse> {
@@ -166,6 +166,12 @@ function hasMatchingPresentation(
return hasMatchingLegacyPresentation(baseline, after, command);
}
/**
* Key equality, where the key is the capture's whole presentation identity producer, generation,
* and the surface the capture described, since an in-place system surface (a web sign-in sheet) is
* captured under its own host lineage (#2438). So a cross-surface pair is refused here, without
* this module knowing that system surfaces exist.
*/
function compareSnapshotIdentity(
baseline: SnapshotState,
after: SnapshotState,
@@ -244,7 +244,7 @@ function recordedTargetCapture(
// retargeted descendant, in lockstep with `recordingTargetExtra`'s chain
// override — the recorded entry and its evidence always name ONE node.
const node = readRecordingTarget(result)?.node ?? ('node' in result ? result.node : undefined);
const preActionNodes = 'preActionNodes' in result ? result.preActionNodes : undefined;
const preActionNodes = 'preAction' in result ? result.preAction?.nodes : undefined;
return node && preActionNodes ? { recordedTarget: { node, preActionNodes } } : {};
}
+9 -5
View File
@@ -1,18 +1,22 @@
import type { SnapshotState } from '@agent-device/kernel/snapshot';
import { systemSurfaceDisclosure } from '@agent-device/contracts/android-system-surface-disclosure';
import { IOS_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/ios-system-surface';
import type { DaemonResponse } from './daemon-request.ts';
/**
* Append the occluding-system-surface disclosure to a selector-route response whose consumed
* snapshot was a system surface (notification shade / quick settings). Both found and not-found
* outcomes must explain that app content is occluded: a match found inside the shade is not app
* content, and a miss is expected while the shade covers the app.
* snapshot was a system surface: an Android notification shade / quick settings, or an iOS in-place
* web sign-in sheet (#2438). Both found and not-found outcomes must explain that app content is
* occluded: a match found inside the surface is not app content, and a miss is expected while the
* surface covers the app.
*/
export function withSystemSurfaceDisclosure(
response: DaemonResponse,
snapshot: Pick<SnapshotState, 'systemSurfaceOnly'> | undefined,
snapshot: Pick<SnapshotState, 'systemSurfaceOnly' | 'iosSystemSurfaceBundleId'> | undefined,
): DaemonResponse {
const disclosure = systemSurfaceDisclosure(snapshot);
const disclosure = snapshot?.iosSystemSurfaceBundleId
? IOS_SYSTEM_SURFACE_DISCLOSURE
: systemSurfaceDisclosure(snapshot);
if (!disclosure) return response;
if (response.ok) {
const warning = appended(response.data?.warning, disclosure);
+16 -1
View File
@@ -209,6 +209,19 @@ const resolutionDisclosureSchema: JsonSchema = {
],
};
// PostActionSurfaceChange (packages/contracts/src/interaction.ts) — the post-action capture
// describes a different surface than the pre-action baseline (#2438), so no same-surface
// comparison is presented across it.
const postActionSurfaceChangeSchema: JsonSchema = objectSchema(
{
from: stringSchema('Surface the pre-action baseline described: a host bundle id, or app.'),
to: stringSchema('Surface the post-action capture describes: a host bundle id, or app.'),
disclosure: stringSchema('Agent-facing sentence explaining the surface transition.'),
},
['from', 'to', 'disclosure'],
'Present when an in-place system surface (web sign-in sheet) was presented over the app, or left it.',
);
// InteractionEvidence (packages/contracts/src/interaction.ts) — opt-in `--verify` cheap
// post-condition evidence (#1047).
const interactionEvidenceSchema: JsonSchema = objectSchema(
@@ -218,8 +231,9 @@ const interactionEvidenceSchema: JsonSchema = objectSchema(
interactiveNodeCount: numberSchema('Subset of nodeCount the platform reports as hittable.'),
digest: stringSchema('Order-independent digest of the post-action node multiset.'),
changedFromBefore: booleanSchema(
'Whether the post-action digest differs from the pre-action capture digest. false is evidence, not failure.',
'Whether the post-action digest differs from the pre-action capture digest. false is evidence, not failure. With surfaceChange present, no digest comparison is made: it reports that surface transition.',
),
surfaceChange: postActionSurfaceChangeSchema,
},
['nodeCount', 'interactiveNodeCount', 'digest', 'changedFromBefore'],
);
@@ -247,6 +261,7 @@ const settleObservationSchema: JsonSchema = objectSchema(
['ref'],
),
},
surfaceChange: postActionSurfaceChangeSchema,
diff: objectSchema(
{
summary: objectSchema(
@@ -173,7 +173,7 @@ test(scenario('responseConstruction'), async () => {
// ADR 0012 decision 3: the preflight's guard lookup supplies the
// record-time evidence node on the runtime result.
assert.equal(result.node?.ref, 'e1');
assert.ok(Array.isArray(result.preActionNodes));
assert.ok(Array.isArray(result.preAction?.nodes));
const {
result: visualization,
@@ -188,7 +188,7 @@ test(scenario('responseConstruction'), async () => {
assert.equal(recordedTarget?.node.ref, 'e1');
for (const payload of [visualization, responseData]) {
assert.equal('node' in payload, false);
assert.equal('preActionNodes' in payload, false);
assert.equal('preAction' in payload, false);
assert.equal('targetEvidence' in payload, false);
}
});