mirror of https://github.com/callstack/agent-device.git synced 2026-09-14 20:06:34 +08:00

Files

T

Michał Pierzchała ab0c7a4328 fix(scroll): keep the swipe above the keyboard, refuse when it cannot (#2503 )

* fix(ios): clip a scroll's swipe above the keyboard, refuse when it cannot

The runner owns the live keyboard frame, so it does the clip and reports what it left: a scroll
answers with `keyboardAvoided` and `keyboardMinY` beside its plan, and refuses with
`SCROLL_KEYBOARD_OCCLUDES_SURFACE` when the keys leave too little band to swipe in instead of flinging
into them. It never dismisses the keyboard, which would drop focus and mutate state that
session-action provenance does not record.

Scroll's keyboard policy moves to `requiredWhenAvailable`. The probe costs a live AX fetch, but
gating it on a healthy tree left the first scroll of a session swiping under the keys, which is the
failure this is for. Every scroll logs its decision, including the two ways it avoids reading the
keyboard at all.

Scroll no longer shares `frameAvoidingKeyboard`, whose 25% fail-open was a tap-reference-frame rule;
that path is unchanged for its remaining callers.

* chore(gates): run the scroll viewport policy tests on the iOS lane

The parity table only detects drift if both halves run in CI. Two of these three were reachable by no
lane, so the Swift half of the table was a local assertion.

* fix(ios): keep the keyboard clip out of the scroll's rotation basis

`resolvedScrollViewport` handed the command one frame for both jobs, and the coordinate rotation reads
a frame's HEIGHT to map a `landscapeRight` native x. Clipping an 834pt landscape viewport to 576pt
therefore moved the dispatched gesture 258pt sideways off the lane the plan had just been built for:
the clip fixed the keyboard and broke the gesture.

The resolved viewport now names both frames, and the gesture comes from one dispatch decision, so the
band the plan is planned inside and the frame its coordinates rotate against cannot be swapped. The
landscape case asserts through that decision and fails on the swap.

* fix(ios): report a scroll's clipped band in its response

2026-09-13 13:55:37 +02:00

AgentDeviceRunner

fix(scroll): keep the swipe above the keyboard, refuse when it cannot (#2503 )

2026-09-13 13:55:37 +02:00

README.md

refactor(apple): colocate the XCUITest runner client into packages/platform-apple (#2040 ) (#2050 )

2026-08-26 15:53:01 +02:00

RUNNER_PROTOCOL.md

refactor(apple): colocate the XCUITest runner client into packages/platform-apple (#2040 ) (#2050 )

2026-08-26 15:53:01 +02:00

README.md

agent-device iOS Runner

This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.

Intent

Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
Allow local builds via xcodebuild and caching for faster subsequent runs.
Support simulator prebuilds where compatible.

Status

Current internal runner for iOS, tvOS, and macOS desktop automation.

Protocol and maintenance references:

Protocol overview: RUNNER_PROTOCOL.md
TypeScript client: ../../packages/platform-apple/src/runner/runner-client.ts
Swift wire models: AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift

UITest Runner File Map

AgentDeviceRunnerUITests/RunnerTests is split into focused files to reduce context size for contributors and LLM agents.

RunnerTests.swift: shared state/constants, setUp(), and testCommand() entry flow.
RunnerTests+Models.swift: wire protocol models (Command, Response, snapshot payload models).
RunnerTests+Environment.swift: environment and CLI argument helpers (RunnerEnv).
RunnerTests+Transport.swift: TCP request handling and HTTP parsing/encoding.
RunnerTests+CommandExecution.swift: command dispatch (execute*) and command switch.
RunnerTests+Lifecycle.swift: activation/retry/stabilization and recording lifecycle helpers.
RunnerTests+Interaction.swift: tap/drag/swipe/type/home/rotate/app-switcher helpers.
RunnerTests+Navigation.swift: back/navigation-control helpers.
RunnerTests+Snapshot.swift: fast/raw snapshot builders and include/filter helpers.
RunnerTests+SystemModal.swift: SpringBoard/system modal detection and modal snapshot shaping.
RunnerTests+ScreenRecorder.swift: nested ScreenRecorder implementation.

Snapshot Strategy

iOS snapshots have two explicit public capture modes:

full/raw snapshots use recursive XCTest snapshots for rich hierarchy and diagnostics;
interactive snapshots filter the same visible tree down to agent-facing refs.

Some iOS apps expose accessibility trees that lower-level AX services can inspect but XCTest cannot serialize reliably. In those cases interactive snapshots may return a sparse root quickly, while full snapshots preserve the XCTest error. A penalized simulator can recover through private AX; physical devices use a short XCTest probe because no non-XCTest semantic backend is available there. See ../../docs/adr/0004-ios-snapshot-backend-strategy.md for the backend boundary and future simulator AX-service direction.

Protocol Notes

The daemon posts JSON commands to POST /command on the runner's local HTTP listener.
The runner responds with a JSON envelope shaped as { ok, data?, error? }.
The protocol is internal to agent-device; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.