Michał Pierzchała
2bdbef3f2b
* fix(daemon): distrust post-gesture stability that matches the pre-gesture baseline #1542 defect 2: post-gesture-stabilization.ts treated two consecutive matching AX-signature polls as proof the screen settled. On iOS's AX-free synthesized gesture lane, XCTest's tree can serve a stale-but-internally- consistent read for a window after a scroll/swipe, so that "match" can be false: the daemon then evaluates pre-gesture node positions on the very next interaction. Fix: capture the interaction-surface signature before the gesture dispatches (reusing session.snapshot, no extra capture), and when a quiet poll-to-poll match still equals that baseline, don't trust it — keep polling past the normal 1.5s deadline up to a bounded 3.5s cap. On cap expiry with the signature still identical, accept the result (a genuine no-op gesture is the honest answer) but flag it via a new post_gesture_snapshot_stale_accept diagnostic so a stale-accept is observable in ndjson. Baseline comparison is subset-tolerant (interactionSurfaceMatchesBaseline) rather than whole-array equality: the pre-gesture baseline and the post-gesture capture are routinely fetched with different snapshot scopes, so naive equality reported "changed" from scope drift alone and never caught the real staleness on first implementation — live-verified and fixed before shipping. Platform-scoped to Apple only (requiresPostGestureBaselineDistrust): Android's persistent helper clears its accessibility-node cache before every capture (AccessibilityTreeCapture.capture, #1254/#1259), so an Android post-gesture read is fresh by construction and never computes a baseline signature — latency and semantics unchanged, confirmed live (checkout-form-android.ad + gesture-lab-android.ad 2/2 on Pixel_7_CI). Does not close #1542: live validation on checkout-form.ad still fails at step 11, but now for a distinct reason this fix correctly surfaces rather than causes — a corrupted ScrollView-ancestor viewport frame ((18,381,366,109) vs the true (18,62,366,729)) that the off-screen guard's findNearestScrollableAncestorRect trusts, independent of whether the signature matches the baseline. gesture-lab.ad (iOS) remains 2/2 clean, confirming no regression on the passing scenario. _Generated by Claude Code_ * refactor(daemon): decompose the stabilization loop's diagnostics and capture pair The distrust integration pushed capturePostGestureStabilizedResult over the complexity gate (cyclomatic 15, cognitive 24); extracting the settle-diagnostic branching and the capture+signature pair restores a clean fallow pass with no behavior change. * fix(daemon): require discriminating overlap for a post-gesture baseline match PR review on #1563 (P1): interactionSurfaceMatchesBaseline returned true whenever ANY shared entry was frozen, including the application/window viewport root, whose rect is invariant under any gesture. In the exact scope-drift case this PR supports, a broad pre-gesture baseline and a narrow post-gesture selector capture can share only that root after a real, successful scroll — the boolean predicate called that a baseline match and extended the interaction to the 3.5s stale-read cap on zero real evidence. Fix: replace the boolean with classifyBaselineSurfaceEvidence, a subset-tolerant classifier reusing this module's existing InteractionSurfaceChange vocabulary ('changed' | 'unchanged' | 'ambiguous') instead of a bespoke boolean or an Application-only special case. An entry only counts as evidence when it is `discriminating` — excludes the viewport root (minimal local equivalent of snapshot-occlusion.ts's isViewportRoot) and keyboard chrome (minimal local equivalent of snapshot-chrome.ts's keyboard-container check), both computed once at signature-build time since the flat signature-entry representation has no ref/parentIndex to reuse those modules' full ancestor-walk classifiers directly. Zero discriminating overlap is now 'ambiguous' (insufficient evidence) rather than a match, and decidePostGestureStabilityVerdict falls through 'ambiguous' to 'trust' — the safe default, same as 'changed'. Tests: the reviewer's exact shape (signatures sharing only the Application root, with the real content swapped) at three layers — classifyBaselineSurfaceEvidence directly, decidePostGestureStabilityVerdict, and the full capturePostGestureStabilizedResult async loop (proving no cap-tax: settles in 2 capture attempts, not 3.5s). Also: root+one real element both frozen still matches (guards against over-excluding), and keyboard chrome excluded from discriminating overlap. All prior tests kept green unchanged. Counterfactual: reverted to the old boolean predicate and reran — 5 tests went red, including the async regression test, which didn't just fail an assertion but timed out after 5s because the boolean predicate extended the interaction to the 3.5s distrust cap the test's 1s timer advance never covered — exactly the "extends to cap" failure mode the review predicted. Restored, 37/37 green. _Generated by Claude Code_ * fix(daemon): exclude keyboard descendants (not just the container) from baseline evidence PR review on #1563 (two findings, blocking merge): 1. isKeyboardChromeKind excluded only the [Keyboard] container node itself. collectKeyboardChrome (src/core/snapshot-chrome.ts, the established source of truth) classifies the WHOLE keyboard window/subtree — keys, AND the "Next keyboard"/"Dictate" assistant buttons, which are documented siblings of the container, not descendants, so a container-descendant walk alone provably misses them. In the scope-drift case this PR supports, a successful scroll can leave only those keyboard descendants shared between a baseline and a later capture, and the narrower check called that a baseline match — extending a fresh result to the 3.5s stale-read cap. Fixed by exporting a narrow predicate, collectKeyboardChromeRefs(nodes), from snapshot-chrome.ts (returns collectKeyboardChrome(nodes).refs, no Android union — this caller has no appBundleId in scope and only needs the iOS half). buildInteractionSurfaceSignature computes it once per signature build and threads it into buildInteractionSurfaceEntry, so discriminating is now `!isViewportRootKind(node) && !keyboardChromeRefs .has(node.ref)` — reusing the real ancestor-walk classification instead of a per-node type check, no ancestry needed in the signature entries themselves. 2. post-gesture-stabilization.test.ts had grown to 550 LOC, past the repository's 500-line extraction tripwire (AGENTS.md: "past 500, extract before adding behavior... Tests are not exempt"). Split along subject lines: the pure decidePostGestureStabilityVerdict coverage moved to a new sibling post-gesture-stabilization-verdict.test.ts, and shared fixtures (pickupSnapshot, deliverySnapshot, applicationRootNode, keyboardWindowNodes, makeSession) moved to a new non-test post-gesture-stabilization-fixtures.ts. The async capturePostGesture- StabilizedResult loop tests stay in the original file. Assertions unchanged, only relocation, plus the new regression tests below. Resulting LOC: post-gesture-stabilization.test.ts 381, -verdict.test.ts 208, -fixtures.ts 129 (interaction-outcome-policy.test.ts grew to 413, still under the tripwire). Tests: the reviewer's exact regression — a shared overlap consisting only of keyboard descendants (a key + the "Next keyboard" sibling button, NOT the container) plus real content that changed (Pickup -> Delivery) — at three layers: classifyBaselineSurfaceEvidence directly (ambiguous), the verdict function (trust, elapsedMs: 0), and the full async capture loop (settles in 2 attempts, no cap tax). Counterfactual: reverted isNonDiscriminatingSurfaceNode to a container-only check (normalizeType(node.type) === 'keyboard') and reran — 3 of the new tests went red across all three layers, including the async test, which timed out after 5s (not just a failed assertion) because the container-only exclusion genuinely extended the interaction to the 3.5s distrust cap the test's 1s timer advance never covers — the same "extends to cap" failure shape as the review's finding 1. Restored, 40/40 green. _Generated by [Claude Code](https://claude.ai/code)_
agent-device
Let your coding agent verify its changes in the running app.
agent-device lets coding agents inspect, control, and verify apps on iOS, Android, tvOS, Android TV, Amazon Vega OS TV through the Vega Virtual Device (VVD), web, macOS, and Linux. Agents can read token-efficient accessibility snapshots where supported, find elements by ref or selector, run device actions, and save evidence for review. Initial Vega OS support is VVD-only and covers discovery, app lifecycle, and complete TV-remote control; physical Fire TV, capture, and selector backends remain unsupported.
Your coding agent or QA tool reads each result and chooses the next command. agent-device runs the command and saves evidence when asked.
agent-device uses the inspect-act-verify process from Vercel's agent-browser for mobile, TV, and desktop apps. Basic --platform web support runs agent-browser in the same session and replay system.
Quick start
Install the CLI and check setup. It requires Node.js 22.12 or newer; web automation requires Node.js 24 or newer. See Installation for target requirements.
npm install -g agent-device@latest
agent-device doctor
agent-device help workflow
Run agent-device doctor yourself before handing the CLI to an agent. The installed CLI help defines current behavior. agent-device help workflow links to guides for debugging, replay, React Native profiling, and other tasks.
Add a contact in the built-in iOS Contacts app:
# Start a session.
agent-device open Contacts --platform ios
# Inspect the screen. The example below shows the output; refs vary.
agent-device snapshot -i
# @e2 [button] "Add"
# Use the ref and wait for the UI to settle.
agent-device press @e2 --settle
# The diff includes:
# + @e7 [text-field] "First name"
agent-device fill @e7 "Ada" --settle
# The next diff shows changed values and current refs:
# - @e7 [text-field] "First name"
# + @e14 [text-field] "Ada"
# = @e15 [text-field] "Last name"
# Capture evidence and close the session.
agent-device screenshot ./contact-form.png
agent-device close
Use refs only from the latest output. Do not assume an earlier @eN still identifies the same element. After a command with --settle, use the refs in its diff. Take another snapshot only if the diff omits what you need.
Snapshots use the app's accessibility tree. Clear labels, roles, and test IDs make agent runs more reliable. Use screenshots and videos as evidence or when accessibility data is poor. Use refs and selectors for actions and assertions when you can.
What agents can do
- Inspect app state through accessibility snapshots, refs, selectors, and React Native component trees.
- Act on visible UI by tapping or pressing elements, filling fields, scrolling, making gestures, waiting, asserting state, and handling alerts.
- Diagnose failures with screenshots, video, logs, traces, network data, performance samples, crash details, and React profiles.
- Repeat workflows by saving working steps as
.adscripts for local use or CI. Export strict Maestro YAML when needed.
See Commands for the commands and evidence each target supports.
Next steps
- Set up your agent: run the CLI from Cursor, Codex, Claude Code, Windsurf, or another agent terminal. See AI Agent Setup for skills, rules, MCP tools, and setup for each client.
- Try the sample app: clone the repo and run the bundled Expo test app. Quick Start covers a guided run with screenshots, replay, and performance data.
- Build repeatable tests: use Replay & E2E to repeat tests. Use Debugging & Profiling to find bugs.
Articles and videos
Articles
- Build an AI QA agent for Expo apps with EAS Workflows
- Agent Device: iOS & Android automation for AI agents
- Building mobile QA agents with Vercel Eve
- How we optimized Agent Device for mobile app automation
Videos
- Verifying mobile apps with agent-device
- Using agent-device in an AI coding workflow
- Cloud agents that test mobile apps on real devices
Where to run agent-device
| Path | Best for | Start with |
|---|---|---|
| Local | Trying commands and debugging apps on simulators, emulators, physical devices, macOS, and Linux. | Follow the Quick Start. |
| CI/CD | Automated pull request and merge validation with replay scripts and captured artifacts. | Try the EAS workflow template. GitHub Actions template coming soon. |
| Cloud / remote | Linux runners, managed devices, and remote jobs. | Use Agent Device Cloud, set a remote profile with Commands, or contact Callstack for team QA. |
How it works
agent-device keeps device state in sessions. It sends commands to XCTest on iOS and tvOS, ADB and the snapshot helper on Android, Vega CLI/VDA on the Vega Virtual Device, a local helper on macOS, and AT-SPI on Linux.
Node.js apps can use the typed client or public subpaths. agent-device/android-adb provides the Android ADB provider interface, helpers for logcat, the clipboard, the keyboard, and apps, and port reverse management.
FAQ
What is agent-device?
agent-device is a command-line tool that lets coding agents inspect, control, and verify apps and save evidence for review. It supports iOS, Android, TV, web, macOS, and Linux.
Does it work with React Native, Expo, Flutter, and native apps?
Yes. agent-device supports native iOS and Android apps, plus React Native, Expo, and Flutter apps on supported targets. The commands and evidence vary by target.
How is it different from Appium, Detox, or Maestro?
With agent-device, an agent reads app state and chooses each command at run time. Teams use Appium, Detox, and Maestro to write and maintain test suites. agent-device can complement them by saving its runs as .ad scripts or exporting them as strict Maestro YAML.
Can agent-device run in CI?
Yes. Record a run as an .ad script, replay it locally or in CI, and save screenshots, logs, and other artifacts for review. See Replay & E2E or start with the EAS workflow template.
Who uses agent-device?
Teams and developers at Callstack, JPMorgan Chase, Expensify, Shopify, Kindred, Total Wine & More, LegendList, HerLyfe, App & Flow, and others use agent-device.
Documentation
Contributing
See CONTRIBUTING.md.
Made at Callstack
agent-device is open source under the MIT license. Visit agent-device.dev or contact Callstack.