mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
Michał Pierzchała
3a02e514e1
* refactor(ios): consolidate series batching onto the sequence runner command Closes #767 Routes every Apple multi-press variant (plain, double-tap, hold, jitter) and swipe series through budget-chunked sequence requests, retiring the daemon-side tapSeries and dragSeries senders: - Add a doubleTap step kind to the sequence allowlist on both ends, mirroring the retired tapSeries doubleTapAt branch. - The single doubleTap interactor sends a one-step sequence and parses the result, surfacing step failures as errors. - Swipe series unroll ping-pong daemon-side into per-step endpoints; the runner's coordinate-drag path ignores durationMs exactly as the daemon-sent (non-synthesized) dragSeries did. - Extract runIosSequenceChunks so press and swipe share the chunking, aggregation, and global step-index rebasing. - Keep tapSeries/dragSeries runner handlers for wire compatibility with older daemons, annotated like interactionFrame; remove both from the preflight-skip allowlist (daemon never sends them) and update ADR 0005 / protocol-optimizations docs. This also closes the latent watchdog exposure where press --count N --interval-ms M routed to tapSeries and executed all pauses inside one 30s-watchdog main-thread block with no chunking. Behavior note: plain tap series now use the synthesized HID tap path on iOS non-tv (with runner-side tapAt fallback), matching the individual tap command instead of the retired tapSeries' XCUICoordinate taps. https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo * refactor(ios): drop dead series wire surface from the daemon - Remove chunkRunnerSequenceSteps: superseded by the budget-aware chunker; no production callers remained. - Remove tapSeries/dragSeries from the RunnerCommand union along with their orphaned fields (count, intervalMs, doubleTap, pauseMs, pattern) and protocol fixtures: this type is the send surface of the current daemon, which no longer sends either command. The Swift runner keeps serving both for wire compatibility with older daemons. - Retarget the ready-mutation preflight test from tapSeries to sequence. https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo * refactor(ios): remove retired series and frame wire commands entirely Drops the runner-side wire compatibility for tapSeries, dragSeries, and interactionFrame now that no daemon path sends them (series fuse into sequence since this branch; interactionFrame was fused into scroll in #760): - Swift: delete the three handler cases, performDragSeries, runSeries (no remaining callers), the CommandType enum cases, journal-retention and traits entries, and the Command fields (count, intervalMs, doubleTap, pauseMs, pattern) that existed only for them. The never-sent synthesized dragSeries branch goes with it. - TS: drop interactionFrame from the RunnerCommand union and isReadOnlyRunnerCommand, and its protocol fixture. - Update stale perf scenario labels referencing the retired commands. Verified dead before removal: no dynamic command construction anywhere (runner-command-recovery only echoes in-flight command ids), no raw-string references in Swift, no docs references. Helpers shared with live paths (synthesizedDragAt, doubleTapAt, keyboardAvoidingDragPoints, sleepFor) all retain callers. Compat: an old daemon paired with a runner built from these sources gets a CommandType decode rejection; the source-fingerprint check rebuilds a matching runner on the next session. https://claude.ai/code/session_01VokBZWESTDgcnbYwS4DkJo --------- Co-authored-by: Claude <noreply@anthropic.com>
agent-device iOS Runner
This folder contains the lightweight XCUITest runner used to provide element-level automation for Apple-family targets.
Intent
- Provide a minimal XCTest target that exposes UI automation over a small HTTP server.
- Allow local builds via
xcodebuildand caching for faster subsequent runs. - Support simulator prebuilds where compatible.
Status
Current internal runner for iOS, tvOS, and macOS desktop automation.
Protocol and maintenance references:
- Protocol overview:
RUNNER_PROTOCOL.md - TypeScript client:
../src/platforms/ios/runner-client.ts - Swift wire models:
AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift
UITest Runner File Map
AgentDeviceRunnerUITests/RunnerTests is split into focused files to reduce context size for contributors and LLM agents.
RunnerTests.swift: shared state/constants,setUp(), andtestCommand()entry flow.RunnerTests+Models.swift: wire protocol models (Command,Response, snapshot payload models).RunnerTests+Environment.swift: environment and CLI argument helpers (RunnerEnv).RunnerTests+Transport.swift: TCP request handling and HTTP parsing/encoding.RunnerTests+CommandExecution.swift: command dispatch (execute*) and command switch.RunnerTests+Lifecycle.swift: activation/retry/stabilization and recording lifecycle helpers.RunnerTests+Interaction.swift: tap/drag/swipe/type/back/home/rotate/app-switcher helpers.RunnerTests+Snapshot.swift: fast/raw snapshot builders and include/filter helpers.RunnerTests+SystemModal.swift: SpringBoard/system modal detection and modal snapshot shaping.RunnerTests+ScreenRecorder.swift: nestedScreenRecorderimplementation.
Snapshot Strategy
iOS snapshots have two explicit capture modes:
- full/raw snapshots use recursive XCTest snapshots for rich hierarchy and diagnostics;
- compact interactive snapshots use a bounded flat XCTest query path for fast agent-facing refs.
Some simulator apps expose accessibility trees that lower-level AX services can inspect but XCTest
cannot serialize reliably. In those cases compact interactive snapshots may return a sparse root
quickly, while full snapshots preserve the XCTest error. See
../docs/adr/0004-ios-snapshot-backend-strategy.md
for the backend boundary and the rationale for a future simulator AX-service backend.
Protocol Notes
- The daemon posts JSON commands to
POST /commandon the runner's local HTTP listener. - The runner responds with a JSON envelope shaped as
{ ok, data?, error? }. - The protocol is internal to
agent-device; when adding or renaming commands, update both wire models and the protocol tests/docs in the same change.