mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
docs: restructure agent-device skill docs (#260)
* docs: restructure agent-device skill docs * docs: keep agent-device references grouped * docs: clarify batch use in verification * docs: restore skill operational detail * docs: add qa loop for acceptance criteria * docs: clarify qa modes in skill router * docs: improve skill mental model wording * docs: simplify skill router labels * docs: add accessibility audit guidance * docs: update iOS simulator example device * docs: clarify follow-up routing after session open * docs: tighten session follow-up rule
This commit is contained in:
Michał Pierzchała
committed by
GitHub
GitHub
parent
7100fc5920
commit
bb8d9ddeb8
@@ -116,6 +116,11 @@ Minimal operating guide for AI coding agents in this repo.
|
||||
- For behavior/CLI surface changes, evaluate docs/skills updates.
|
||||
- Update `README.md` and relevant `website/docs/**` pages for command behavior/flags/aliases/workflows.
|
||||
- Update relevant `skills/**/SKILL.md` when usage examples/workflow recommendations change.
|
||||
- Keep skill docs task-first:
|
||||
- top-level `SKILL.md` should stay a thin router, not a full manual.
|
||||
- keep detailed workflows/troubleshooting in a `references/` folder instead of growing the router.
|
||||
- isolate true platform/infra exceptions (for example macOS-only or remote-tenancy-only guidance) in dedicated files.
|
||||
- do not delete high-value operational guidance during refactors; move or condense it unless the behavior is obsolete.
|
||||
- In final summaries, state whether docs/skills were updated; if not, explain why.
|
||||
|
||||
## When Blocked
|
||||
|
||||
+24
-107
@@ -3,120 +3,37 @@ name: agent-device
|
||||
description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets.
|
||||
---
|
||||
|
||||
# Apple and Android Automation with agent-device
|
||||
# agent-device
|
||||
|
||||
Use this skill as a router.
|
||||
|
||||
Core rule:
|
||||
## QA modes
|
||||
|
||||
- explore with `snapshot -i` and `@ref`
|
||||
- stabilize with selectors
|
||||
- use plain `snapshot` when you need to verify whether text is visible
|
||||
- re-snapshot after every meaningful UI change
|
||||
- Open-ended bug hunt with reporting: use [../dogfood/SKILL.md](../dogfood/SKILL.md).
|
||||
- Pass/fail QA from acceptance criteria: stay in this skill, start with [references/bootstrap-install.md](references/bootstrap-install.md), then use the QA loop in [references/exploration.md](references/exploration.md).
|
||||
|
||||
For exploratory QA bug hunts and reporting, use [../dogfood/SKILL.md](../dogfood/SKILL.md).
|
||||
## Mental model
|
||||
|
||||
## Quick route
|
||||
- First choose the correct target and open the app or session you want to work on.
|
||||
- Then inspect the current UI with `snapshot -i` and pick targets from the actual UI state.
|
||||
- Act with `press`, `fill`, `get`, `is`, `wait`, or `find`.
|
||||
- Re-snapshot after meaningful UI changes instead of reusing stale refs.
|
||||
- End by capturing proof if needed, then `close`.
|
||||
|
||||
- Normal UI task: `open` -> `snapshot -i` -> `click/fill/press` -> `close`
|
||||
- Debug task: `open` -> `logs clear --restart` -> reproduce -> `network dump` -> `logs path`
|
||||
- Replay drift: `replay -u <path>`
|
||||
- No target context yet: `devices` -> pick target -> `open`
|
||||
## Decision rules
|
||||
|
||||
## Target rules
|
||||
- Use plain `snapshot` when you need to verify whether text is visible.
|
||||
- Use `snapshot -i` mainly for interactive exploration and choosing refs.
|
||||
- Use `fill` to replace text.
|
||||
- Use `type` to append text.
|
||||
- Prefer `@ref` or selector targeting over raw coordinates.
|
||||
- Keep the default loop short: `open` -> explore/act -> optional debug or verify -> `close`.
|
||||
|
||||
- iOS local QA: prefer simulators
|
||||
- Android binary flow: `install` or `reinstall` first, then `open <package> --relaunch`
|
||||
- In mixed-device labs, always pin the target with `--device`, `--udid`, `--serial`, or an isolation scope
|
||||
- For session-bound automation, prefer `AGENT_DEVICE_SESSION` + `AGENT_DEVICE_PLATFORM`
|
||||
## Choose a reference
|
||||
|
||||
## macOS rules
|
||||
|
||||
- Use `open <app> --platform macos` for normal Mac app automation
|
||||
- Use `open --platform macos --surface frontmost-app|desktop|menubar` when you need desktop-global inspection first
|
||||
- Use `app` sessions for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record`
|
||||
- Use `frontmost-app`, `desktop`, and `menubar` mainly for `snapshot`, `get`, `is`, and `wait`
|
||||
- If you inspect with `desktop` or `menubar` and then need to act inside one app, open that app in a normal `app` session
|
||||
- Prefer `@ref` or selectors over raw `x y` on macOS
|
||||
- Use `click --button secondary` for context menus, then run `snapshot -i` again
|
||||
|
||||
## Canonical flows
|
||||
|
||||
### Normal flow
|
||||
|
||||
```bash
|
||||
agent-device open Settings --platform ios
|
||||
agent-device snapshot -i
|
||||
agent-device press @e3
|
||||
agent-device fill @e5 "test"
|
||||
agent-device close
|
||||
```
|
||||
|
||||
### macOS app flow
|
||||
|
||||
```bash
|
||||
agent-device open TextEdit --platform macos
|
||||
agent-device snapshot -i
|
||||
agent-device fill @e3 "desktop smoke test"
|
||||
agent-device screenshot /tmp/macos-textedit.png
|
||||
agent-device close
|
||||
```
|
||||
|
||||
### macOS desktop-global inspect flow
|
||||
|
||||
```bash
|
||||
agent-device open --platform macos --surface desktop
|
||||
agent-device snapshot -i
|
||||
agent-device get attrs @e4
|
||||
agent-device is visible 'role="window" label="Notes"'
|
||||
agent-device wait text "Notes"
|
||||
agent-device close
|
||||
```
|
||||
|
||||
### Android relaunch flow
|
||||
|
||||
```bash
|
||||
agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554
|
||||
agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch
|
||||
agent-device snapshot -i
|
||||
agent-device close
|
||||
```
|
||||
|
||||
### Debug flow
|
||||
|
||||
```bash
|
||||
agent-device open MyApp --platform ios
|
||||
agent-device logs clear --restart
|
||||
agent-device network dump 25
|
||||
agent-device logs path
|
||||
```
|
||||
|
||||
### Replay maintenance
|
||||
|
||||
```bash
|
||||
agent-device replay -u ./session.ad
|
||||
```
|
||||
|
||||
## High-value guardrails
|
||||
|
||||
- Prefer `snapshot -i`; use `--raw` only for structure debugging
|
||||
- Use plain `snapshot` to verify text visibility; use `snapshot -i` mainly for interactive exploration and choosing refs
|
||||
- Use refs for discovery, selectors for replay/assertions
|
||||
- `fill` clears then types; `type` only types into the focused field
|
||||
- `network dump` is best-effort and reads from the session app log
|
||||
- `logs clear --restart` requires an active app session
|
||||
- On macOS, helper-backed flows cover permissions, alerts, and desktop-global snapshot surfaces
|
||||
- On macOS, do not assume `desktop` or `menubar` are the best surface for real interactions yet
|
||||
|
||||
## References
|
||||
|
||||
- [references/macos-desktop.md](references/macos-desktop.md)
|
||||
- [references/snapshot-refs.md](references/snapshot-refs.md)
|
||||
- [references/logs-and-debug.md](references/logs-and-debug.md)
|
||||
- [references/session-management.md](references/session-management.md)
|
||||
- [references/permissions.md](references/permissions.md)
|
||||
- [references/video-recording.md](references/video-recording.md)
|
||||
- [references/coordinate-system.md](references/coordinate-system.md)
|
||||
- [references/batching.md](references/batching.md)
|
||||
- [references/perf-metrics.md](references/perf-metrics.md)
|
||||
- [references/remote-tenancy.md](references/remote-tenancy.md)
|
||||
- Pick target device, install, open, or manage sessions: [references/bootstrap-install.md](references/bootstrap-install.md)
|
||||
- Need to discover UI, pick refs, wait, query, or interact: [references/exploration.md](references/exploration.md)
|
||||
- Need logs, network, alerts, permissions, or failure triage: [references/debugging.md](references/debugging.md)
|
||||
- Need screenshots, diff, recording, replay maintenance, or perf data: [references/verification.md](references/verification.md)
|
||||
- Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: [references/macos-desktop.md](references/macos-desktop.md)
|
||||
- Need to connect to a remote `agent-device` daemon over HTTP or use tenant leases: [references/remote-tenancy.md](references/remote-tenancy.md)
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
# Batching
|
||||
|
||||
## When to use batch
|
||||
|
||||
- The agent already knows a short sequence of commands.
|
||||
- Steps belong to one logical screen flow.
|
||||
- You want one result object with per-step timing and failure context.
|
||||
|
||||
## When not to use batch
|
||||
|
||||
- Flows are unrelated and should be retried independently.
|
||||
- The workflow is highly dynamic and requires replanning after each step.
|
||||
- You need human approvals between steps.
|
||||
|
||||
## CLI patterns
|
||||
|
||||
From file:
|
||||
|
||||
```bash
|
||||
agent-device batch --session sim --platform ios --steps-file /tmp/batch-steps.json --json
|
||||
```
|
||||
|
||||
Inline (small payloads only):
|
||||
|
||||
```bash
|
||||
agent-device batch --steps '[{"command":"open","positionals":["settings"]}]'
|
||||
```
|
||||
|
||||
## Step payload contract
|
||||
|
||||
```json
|
||||
[
|
||||
{ "command": "open", "positionals": ["settings"], "flags": {} },
|
||||
{ "command": "wait", "positionals": ["label=\"Privacy & Security\"", "3000"], "flags": {} },
|
||||
{ "command": "click", "positionals": ["label=\"Privacy & Security\""], "flags": {} },
|
||||
{ "command": "get", "positionals": ["text", "label=\"Tracking\""], "flags": {} }
|
||||
]
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `positionals` optional, defaults to `[]`.
|
||||
- `flags` optional, defaults to `{}`.
|
||||
- nested `batch` and `replay` are rejected.
|
||||
- stop-on-first-error is the supported mode (`--on-error stop`).
|
||||
|
||||
## Response handling
|
||||
|
||||
Success includes:
|
||||
|
||||
- `total`, `executed`, `totalDurationMs`
|
||||
- `results[]` entries with `step`, `command`, `durationMs`, and optional `data`
|
||||
|
||||
Failure includes:
|
||||
|
||||
- `details.step`
|
||||
- `details.command`
|
||||
- `details.executed`
|
||||
- `details.partialResults`
|
||||
|
||||
Use these fields to replan from the first failing step.
|
||||
|
||||
## Common error categories and agent actions
|
||||
|
||||
- `INVALID_ARGS`: payload/step shape issue; fix payload and retry.
|
||||
- `SESSION_NOT_FOUND`: open or select the correct session, then retry.
|
||||
- `UNSUPPORTED_OPERATION`: switch command/target to supported operation.
|
||||
- `AMBIGUOUS_MATCH`: refine selector/locator, then retry failed step.
|
||||
- `COMMAND_FAILED`: add sync guard (`wait`, `is exists`) and retry from failed step.
|
||||
|
||||
## Reliability guardrails
|
||||
|
||||
- Add sync guards after mutating steps.
|
||||
- Assume snapshot/ref drift after navigation.
|
||||
- Keep batch size moderate (about 5-20 steps).
|
||||
- Split long workflows into phases:
|
||||
1. navigate
|
||||
2. verify/extract
|
||||
3. cleanup
|
||||
@@ -0,0 +1,167 @@
|
||||
# Bootstrap and Install
|
||||
|
||||
## When to open this file
|
||||
|
||||
Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting.
|
||||
|
||||
## Main commands to reach for first
|
||||
|
||||
- `devices`
|
||||
- `ensure-simulator`
|
||||
- `open`
|
||||
- `install` or `reinstall`
|
||||
- `close`
|
||||
- `session list`
|
||||
|
||||
## Most common mistake to avoid
|
||||
|
||||
Do not start acting before you have pinned the correct target and opened an `app` session. In mixed-device environments, always pass `--device`, `--udid`, or `--serial`.
|
||||
|
||||
## Canonical loop
|
||||
|
||||
```bash
|
||||
agent-device ensure-simulator --platform ios --device "iPhone 17 Pro" --boot
|
||||
agent-device open MyApp --platform ios --device "iPhone 17 Pro" --relaunch
|
||||
agent-device snapshot -i
|
||||
agent-device close
|
||||
```
|
||||
|
||||
## Choose the right starting point
|
||||
|
||||
- iOS local QA: prefer simulators unless the task explicitly requires physical hardware.
|
||||
- iOS in mixed simulator and device environments: run `ensure-simulator` first, then keep using `--device` or `--udid`.
|
||||
- TV targets: use `--target tv` together with `--platform` when the task is for tvOS or Android TV rather than phone or tablet surfaces.
|
||||
- Android binary flow: use `install` or `reinstall` for `.apk` or `.aab`, then open by installed package name.
|
||||
- Android React Native plus Metro flow: `reinstall <app> <apk>` first, then `open <package> --remote-config <path> --relaunch`.
|
||||
- macOS desktop app flow: use `open <app> --platform macos`. Only load [macos-desktop.md](macos-desktop.md) if a desktop surface or macOS-specific behavior matters.
|
||||
|
||||
TV example:
|
||||
|
||||
```bash
|
||||
agent-device open MyTvApp --platform ios --target tv
|
||||
agent-device open com.example.androidtv --platform android --target tv
|
||||
```
|
||||
|
||||
## Session rules
|
||||
|
||||
- Use `--session <name>` when you need a named session:
|
||||
|
||||
```bash
|
||||
agent-device --session auth open Settings --platform ios
|
||||
agent-device --session auth snapshot -i
|
||||
```
|
||||
|
||||
- Use `open <app>` before interactions.
|
||||
- Use `close` when done. Add `--shutdown` when you want simulators or emulators torn down with the session.
|
||||
- Use semantic session names when you need multiple concurrent runs.
|
||||
- Use `--save-script=<path>` on `close` when you want to keep a replay script.
|
||||
- For dev loops where state can linger, prefer `open <app> --relaunch`.
|
||||
- In iOS sessions, use `open <app>` for the app itself. Use `open <url>` for deep links, and `open <app> <url>` when you need to launch the app and deep link in one step.
|
||||
- On iOS, `appstate` is session-scoped and requires the matching active session on the target device.
|
||||
|
||||
## After a session is established
|
||||
|
||||
Once you have opened the correct session on the correct target, default to the conservative rule: keep the session binding on follow-up commands, and stop repeating device-routing flags unless you are intentionally retargeting.
|
||||
|
||||
- Prefer `--session <name>` on follow-up commands, or use sandboxed `AGENT_DEVICE_SESSION`.
|
||||
- Do not keep repeating `--platform`, `--target`, `--device`, `--udid`, `--serial`, or similar target-selection flags on normal follow-up commands.
|
||||
- Only omit follow-up session flags when the environment explicitly guarantees isolation.
|
||||
|
||||
Good shared-host pattern:
|
||||
|
||||
```bash
|
||||
agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro"
|
||||
agent-device --session auth snapshot -i
|
||||
agent-device --session auth press @e3
|
||||
agent-device --session auth close
|
||||
```
|
||||
|
||||
Bad shared-host pattern:
|
||||
|
||||
```bash
|
||||
agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro"
|
||||
agent-device --session auth snapshot -i --platform ios --device "iPhone 17 Pro"
|
||||
```
|
||||
|
||||
Use target-selection flags again only when you are choosing the target before opening a session, or when you explicitly mean to retarget.
|
||||
|
||||
## Session-bound automation
|
||||
|
||||
Use this when an orchestrator must keep plain CLI calls on one session and device.
|
||||
|
||||
```bash
|
||||
export AGENT_DEVICE_SESSION=qa-ios
|
||||
export AGENT_DEVICE_PLATFORM=ios
|
||||
export AGENT_DEVICE_SESSION_LOCK=strip
|
||||
|
||||
agent-device open MyApp --relaunch
|
||||
agent-device snapshot -i
|
||||
agent-device close
|
||||
```
|
||||
|
||||
- `AGENT_DEVICE_SESSION` plus `AGENT_DEVICE_PLATFORM` provides the default binding.
|
||||
- `--session-lock reject|strip` controls whether conflicting per-call routing flags fail or are ignored.
|
||||
- Conflicts include explicit retargeting flags such as `--platform`, `--target`, `--device`, `--udid`, `--serial`, `--ios-simulator-device-set`, and `--android-device-allowlist`.
|
||||
- Lock policy applies to nested `batch` steps too.
|
||||
- Compatibility aliases remain supported: `--session-locked`, `--session-lock-conflicts`, `AGENT_DEVICE_SESSION_LOCKED`, and `AGENT_DEVICE_SESSION_LOCK_CONFLICTS`.
|
||||
|
||||
Android emulator variant:
|
||||
|
||||
```bash
|
||||
export AGENT_DEVICE_SESSION=qa-android
|
||||
export AGENT_DEVICE_PLATFORM=android
|
||||
|
||||
agent-device reinstall MyApp /path/to/app-debug.apk --serial emulator-5554
|
||||
agent-device --session-lock reject open com.example.myapp --relaunch
|
||||
agent-device snapshot -i
|
||||
agent-device close --shutdown
|
||||
```
|
||||
|
||||
## Scoped discovery
|
||||
|
||||
Use scoped discovery when one run must not see host-global device lists.
|
||||
|
||||
```bash
|
||||
agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators
|
||||
agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234
|
||||
```
|
||||
|
||||
- Scope is applied before `--device`, `--udid`, and `--serial`.
|
||||
- Out-of-scope selectors fail with `DEVICE_NOT_FOUND`.
|
||||
- With iOS simulator-set scope enabled, iOS physical devices are not enumerated.
|
||||
- If the scoped iOS simulator set is empty, the error should point at the set path and suggest creating a simulator in that set.
|
||||
- Environment equivalents:
|
||||
- `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET`
|
||||
- `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST`
|
||||
|
||||
## Session inspection and replay
|
||||
|
||||
```bash
|
||||
agent-device session list
|
||||
agent-device replay ./session.ad --session auth
|
||||
agent-device replay -u ./session.ad --session auth
|
||||
```
|
||||
|
||||
- iOS session entries include `device_udid` and `ios_simulator_device_set`. Use them to confirm routing in concurrent runs.
|
||||
- Prefer selector-based actions and assertions in saved replay scripts.
|
||||
- Tenant isolation namespaces sessions as `<tenant>:<session>` during tenant-scoped runs.
|
||||
|
||||
## When to leave this file
|
||||
|
||||
- Once the correct target and session are pinned, move to [exploration.md](exploration.md).
|
||||
- If opening, startup, permissions, or logs become the blocker, switch to [debugging.md](debugging.md).
|
||||
|
||||
## Install and open examples
|
||||
|
||||
```bash
|
||||
agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554
|
||||
agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch
|
||||
```
|
||||
|
||||
Do not use `open <apk|aab> --relaunch` on Android.
|
||||
|
||||
## Security and trust notes
|
||||
|
||||
- Treat signing, provisioning, and daemon auth values as host secrets. Do not paste them into shared logs or commit them to source control.
|
||||
- Prefer Xcode Automatic Signing over manual overrides when a physical iOS device is involved.
|
||||
- Keep persistent host-specific defaults in environment variables rather than checked-in project config.
|
||||
@@ -1,8 +1,28 @@
|
||||
# Coordinate System
|
||||
|
||||
All coordinate-based actions use device screen coordinates:
|
||||
## When to open this file
|
||||
|
||||
- Origin: top-left of the device screen
|
||||
- Units: device points for iOS, pixels for Android
|
||||
Open this file only when you must use raw coordinates instead of selectors or `@ref` targeting.
|
||||
|
||||
Use screenshots to reason about coordinates.
|
||||
## Main commands to reach for first
|
||||
|
||||
- `screenshot`
|
||||
- coordinate-based `click` or `swipe`
|
||||
|
||||
## Most common mistake to avoid
|
||||
|
||||
Do not assume coordinates mean the same thing across platforms or runs. Prefer selectors and refs first.
|
||||
|
||||
## Canonical loop
|
||||
|
||||
```bash
|
||||
agent-device screenshot /tmp/current-screen.png
|
||||
agent-device click 120 240
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- Origin is the top-left of the device screen.
|
||||
- iOS uses device points.
|
||||
- Android uses pixels.
|
||||
- Use screenshots to reason about coordinates before acting.
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
# Debugging
|
||||
|
||||
## When to open this file
|
||||
|
||||
Open this file when the task turns into failure triage, logs, network inspection, permission prompts, setup trouble, or unstable session behavior.
|
||||
|
||||
## Main commands to reach for first
|
||||
|
||||
- `logs clear --restart`
|
||||
- `network dump`
|
||||
- `logs path`
|
||||
- `logs doctor`
|
||||
- `alert wait`
|
||||
- `alert accept` or `alert dismiss`
|
||||
|
||||
## Most common mistake to avoid
|
||||
|
||||
Do not leave logging on for normal flows or dump full log files into context. Keep debug windows short and inspect logs with `grep` or `tail`.
|
||||
|
||||
## Canonical loop
|
||||
|
||||
```bash
|
||||
agent-device open MyApp --platform ios
|
||||
agent-device logs clear --restart
|
||||
agent-device network dump 25
|
||||
agent-device logs path
|
||||
agent-device close
|
||||
```
|
||||
|
||||
## Log and network flow
|
||||
|
||||
Logging is off by default. Enable it only when you need a debugging window.
|
||||
|
||||
- Default app logs live under `~/.agent-device/sessions/<session>/app.log`.
|
||||
- `logs clear --restart` is the fastest clean repro loop.
|
||||
- `network dump [limit] [summary|headers|body|all]` parses recent HTTP(s) entries from the same session app log.
|
||||
- `logs doctor` checks backend and runtime readiness for the current session and device.
|
||||
- `logs mark "before tap"` inserts a timestamped marker into the app log.
|
||||
- Session app logs can contain runtime data, headers, or payload fragments. Review them before sharing.
|
||||
- `logs start` requires an active app session and appends to `app.log`.
|
||||
- `logs stop` stops streaming. `close` also stops logging.
|
||||
- `logs clear` truncates `app.log` and removes rotated `app.log.N` files, and requires logging to be stopped first.
|
||||
- `logs path` returns the log path plus metadata about the active backend and file state.
|
||||
- `network log` is an alias for `network dump`.
|
||||
|
||||
Operational limits:
|
||||
|
||||
- `app.log` rotates to `app.log.1` after 5 MB by default.
|
||||
- `network dump` scans the last 4000 app-log lines, returns up to 200 entries, and truncates header or payload fields at 2048 characters.
|
||||
- Retention knobs:
|
||||
- `AGENT_DEVICE_APP_LOG_MAX_BYTES`
|
||||
- `AGENT_DEVICE_APP_LOG_MAX_FILES`
|
||||
- Redaction hook:
|
||||
- `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS`
|
||||
|
||||
Useful shell follow-up after `logs path`:
|
||||
|
||||
```bash
|
||||
grep -n -E "Error|Exception|Fatal|crash" <path>
|
||||
tail -50 <path>
|
||||
```
|
||||
|
||||
## Alerts and permissions
|
||||
|
||||
Use `alert` for iOS simulator permission dialogs instead of tapping coordinates.
|
||||
|
||||
```bash
|
||||
agent-device alert wait 5000
|
||||
agent-device alert accept
|
||||
```
|
||||
|
||||
- `alert` is only supported on iOS simulators.
|
||||
- `alert accept` and `alert dismiss` retry internally for a short window, so you usually do not need manual sleeps.
|
||||
- iOS 16+ "Allow Paste" prompts are suppressed under XCUITest. Use `xcrun simctl pbcopy booted` when you need to seed simulator clipboard content directly.
|
||||
|
||||
## Setup problems worth recognizing early
|
||||
|
||||
- iOS snapshots do not require macOS Accessibility permissions.
|
||||
- iOS physical-device XCTest setup does require valid signing and provisioning.
|
||||
- If physical-device runner setup fails, prefer Xcode Automatic Signing first.
|
||||
- Optional overrides are:
|
||||
- `AGENT_DEVICE_IOS_TEAM_ID`
|
||||
- `AGENT_DEVICE_IOS_SIGNING_IDENTITY`
|
||||
- `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`
|
||||
- `AGENT_DEVICE_IOS_BUNDLE_ID`
|
||||
- If daemon startup is timing out during setup, increase `AGENT_DEVICE_DAEMON_TIMEOUT_MS`.
|
||||
- If daemon startup fails with stale metadata hints, clean `~/.agent-device/daemon.json` and `~/.agent-device/daemon.lock`, then retry.
|
||||
- Free Apple Developer personal-team accounts may reject generic bundle IDs. Use a unique reverse-DNS value for `AGENT_DEVICE_IOS_BUNDLE_ID` when that happens.
|
||||
|
||||
## Common failure patterns
|
||||
|
||||
- `snapshot` returns 0 nodes: the app may no longer be foregrounded or the UI is not stable yet. Re-open the app or retry when state settles.
|
||||
- Logs are empty: confirm you opened an app session before `logs clear --restart`.
|
||||
- Android logs look stale after relaunch: retry the repro window after the process rebinds.
|
||||
- Permission prompts block the flow: wait for the alert and handle it explicitly.
|
||||
- If snapshots keep returning 0 nodes on an iOS simulator, restart Simulator and re-open the app.
|
||||
- If a macOS snapshot looks incomplete, compare with `snapshot --raw --platform macos` to separate collector filtering from missing AX content.
|
||||
|
||||
## Crash triage fast path
|
||||
|
||||
Always start from the session app log, then branch by platform.
|
||||
|
||||
```bash
|
||||
agent-device logs path
|
||||
grep -n -E "SIGABRT|SIGSEGV|EXC_|fatal|exception|terminated|killed|jetsam|memorystatus|FATAL EXCEPTION|Abort message" <path>
|
||||
```
|
||||
|
||||
- iOS: if the log suggests `ReportCrash`, `SIGABRT`, or `EXC_*`, inspect `~/Library/Logs/DiagnosticReports`.
|
||||
- Android: if the app log is not enough, use `adb logcat` for `FATAL EXCEPTION`, `Abort message`, or `signal` lines around process death.
|
||||
- If no crash signature appears in app logs, stop collecting broad logs and switch to the platform-native crash source.
|
||||
|
||||
## When to leave this file
|
||||
|
||||
- Return to [exploration.md](exploration.md) once the app is stable again.
|
||||
- Load [verification.md](verification.md) if you need evidence artifacts after reproducing the issue.
|
||||
@@ -0,0 +1,192 @@
|
||||
# Exploration
|
||||
|
||||
## When to open this file
|
||||
|
||||
Open this file when the app or screen is already running and you need to discover the UI, choose targets, read state, wait for conditions, or perform normal interactions.
|
||||
|
||||
## Main commands to reach for first
|
||||
|
||||
- `snapshot`
|
||||
- `snapshot -i`
|
||||
- `press`
|
||||
- `fill`
|
||||
- `get`
|
||||
- `is`
|
||||
- `wait`
|
||||
- `find`
|
||||
|
||||
## Most common mistake to avoid
|
||||
|
||||
Do not treat `@ref` values as durable after navigation or dynamic updates. Re-snapshot after the UI changes, and switch to selectors when the flow must stay stable.
|
||||
|
||||
## Canonical loop
|
||||
|
||||
```bash
|
||||
agent-device open Settings --platform ios
|
||||
agent-device snapshot -i
|
||||
agent-device press @e3
|
||||
agent-device wait visible 'label="Privacy & Security"' 3000
|
||||
agent-device get text 'label="Privacy & Security"'
|
||||
agent-device close
|
||||
```
|
||||
|
||||
## Snapshot choices
|
||||
|
||||
- Use plain `snapshot` when you only need to verify whether visible text or structure is on screen.
|
||||
- Use `snapshot -i` when you need refs such as `@e3` for interactive exploration.
|
||||
- Use `snapshot -i -s "Camera"` or `snapshot -i -s @e3` when you want a smaller, scoped result.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
agent-device snapshot -i
|
||||
```
|
||||
|
||||
Sample output:
|
||||
|
||||
```text
|
||||
Page: com.apple.Preferences
|
||||
App: com.apple.Preferences
|
||||
|
||||
@e1 [ioscontentgroup]
|
||||
@e2 [button] "Camera"
|
||||
@e3 [button] "Privacy & Security"
|
||||
```
|
||||
|
||||
## Refs vs selectors
|
||||
|
||||
- Use refs for discovery, debugging, and short local loops.
|
||||
- Use selectors for deterministic scripts, assertions, and replay-friendly actions.
|
||||
- Prefer selector or `@ref` targeting over raw coordinates.
|
||||
- For tap interactions, `press` is canonical and `click` is an equivalent alias.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
agent-device press @e2
|
||||
agent-device fill @e5 "test"
|
||||
agent-device press 'id="camera_row" || label="Camera" role=button'
|
||||
agent-device is visible 'id="camera_settings_anchor"'
|
||||
```
|
||||
|
||||
## Text entry rules
|
||||
|
||||
- Use `fill` to replace text in an editable field.
|
||||
- Use `type` to append text to the current insertion point.
|
||||
|
||||
## Query and sync rules
|
||||
|
||||
- Use `get` to read text, attrs, or state from a known target.
|
||||
- Use `is` for assertions.
|
||||
- Use `wait` when the UI needs time to settle after a mutation.
|
||||
- Use `find "<query>" click --json` when you need search-driven targeting plus matched-target metadata.
|
||||
- If you are forced onto raw coordinates, open [coordinate-system.md](coordinate-system.md) first.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
agent-device find "Increment" click --json
|
||||
```
|
||||
|
||||
Returned metadata comes from the matched snapshot node and can be used for observability or replay maintenance.
|
||||
|
||||
## QA from acceptance criteria
|
||||
|
||||
Use this loop when the task starts from acceptance criteria and you need to turn them into concrete checks.
|
||||
|
||||
Preferred mapping:
|
||||
|
||||
- visibility or presence claim: `is visible` or plain `snapshot`
|
||||
- exact text, label, or value claim: `get text`
|
||||
- post-action state change: act, then `wait`, then `is` or `get`
|
||||
- nearby structural UI change: `diff snapshot`
|
||||
- proof artifact for the final result: `screenshot` or `record`
|
||||
|
||||
Anti-hallucination rules:
|
||||
|
||||
- Do not invent app names, device ids, session names, refs, selectors, or package names.
|
||||
- Discover them first with `devices`, `open`, `snapshot -i`, `find`, or `session list`.
|
||||
- If refs drift after navigation, re-snapshot or switch to selectors instead of guessing.
|
||||
|
||||
Canonical QA loop:
|
||||
|
||||
```bash
|
||||
agent-device open MyApp --platform ios
|
||||
agent-device snapshot -i
|
||||
agent-device press @e3
|
||||
agent-device wait visible 'label="Success"' 3000
|
||||
agent-device is visible 'label="Success"'
|
||||
agent-device screenshot /tmp/qa-proof.png
|
||||
agent-device close
|
||||
```
|
||||
|
||||
## Accessibility audit
|
||||
|
||||
Use this pattern when you need to find UI that is visible to a user but missing from the accessibility tree.
|
||||
|
||||
Audit loop:
|
||||
|
||||
1. Capture a `screenshot` to see what is visually rendered.
|
||||
2. Capture a `snapshot` or `snapshot -i` to see what the accessibility tree exposes.
|
||||
3. Compare the two:
|
||||
- visible in screenshot and present in snapshot: exposed to accessibility
|
||||
- visible in screenshot and missing from snapshot: likely accessibility gap
|
||||
4. If you suspect the node exists in AX but is filtered from interactive output, retry with `snapshot --raw`.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
agent-device screenshot /tmp/accessibility-screen.png
|
||||
agent-device snapshot -i
|
||||
```
|
||||
|
||||
Use `screenshot` as the visual source of truth and `snapshot` as the accessibility source of truth for this audit.
|
||||
|
||||
## Batch only when the sequence is already known
|
||||
|
||||
Use `batch` when a short command sequence is already planned and belongs to one logical screen flow.
|
||||
|
||||
```bash
|
||||
agent-device batch --session sim --platform ios --steps-file /tmp/batch-steps.json --json
|
||||
```
|
||||
|
||||
- Keep batch size moderate, roughly 5 to 20 steps.
|
||||
- Add `wait` or `is exists` guards after mutating steps.
|
||||
- Do not use `batch` for highly dynamic flows that need replanning after each step.
|
||||
|
||||
Step payload contract:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "command": "open", "positionals": ["Settings"], "flags": { "platform": "ios" } },
|
||||
{ "command": "wait", "positionals": ["label=\"Privacy & Security\"", "3000"], "flags": {} },
|
||||
{ "command": "click", "positionals": ["label=\"Privacy & Security\""], "flags": {} },
|
||||
{ "command": "get", "positionals": ["text", "label=\"Tracking\""], "flags": {} }
|
||||
]
|
||||
```
|
||||
|
||||
- `positionals` is optional and defaults to `[]`.
|
||||
- `flags` is optional and defaults to `{}`.
|
||||
- Nested `batch` and `replay` are rejected.
|
||||
- Supported error mode is stop-on-first-error.
|
||||
|
||||
Response handling:
|
||||
|
||||
- Success returns fields such as `total`, `executed`, `totalDurationMs`, and `results[]`.
|
||||
- Failed runs include `details.step`, `details.command`, `details.executed`, and `details.partialResults`.
|
||||
- Replan from the first failing step instead of rerunning the whole flow blindly.
|
||||
|
||||
Common batch error categories:
|
||||
|
||||
- `INVALID_ARGS`: fix the payload shape and retry.
|
||||
- `SESSION_NOT_FOUND`: open or select the correct session, then retry.
|
||||
- `UNSUPPORTED_OPERATION`: switch to a supported command or surface.
|
||||
- `AMBIGUOUS_MATCH`: refine the selector or locator, then retry the failed step.
|
||||
- `COMMAND_FAILED`: add sync guards and retry from the failing step.
|
||||
|
||||
## Stop conditions
|
||||
|
||||
- If refs drift after transitions, switch to selectors.
|
||||
- If a desktop surface or context menu is involved on macOS, load [macos-desktop.md](macos-desktop.md).
|
||||
- If logs, network, alerts, or setup failures become the blocker, switch to [debugging.md](debugging.md).
|
||||
- If the flow is stable and you need proof or replay maintenance, switch to [verification.md](verification.md).
|
||||
@@ -1,114 +0,0 @@
|
||||
# Logs (Token-Efficient Debugging)
|
||||
|
||||
Logging is off by default in normal flows. Enable it on demand for debugging windows. App output is written to a session-scoped file so agents can grep it instead of loading full logs into context.
|
||||
`network dump` parses recent HTTP(s) entries from this same session app log file.
|
||||
|
||||
## Data Handling
|
||||
|
||||
- Default app logs are stored under `~/.agent-device/sessions/<session>/app.log`.
|
||||
- Replay scripts saved with `--save-script` are written to the explicit path you provide.
|
||||
- Log files may contain sensitive runtime data; review before sharing and clean up when finished.
|
||||
- Use `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS` to redact sensitive patterns at write time when needed.
|
||||
|
||||
## Retention and Cleanup
|
||||
|
||||
- Keep logging scoped to active debug windows (`logs clear --restart` before repro, `logs stop` after repro).
|
||||
- Prefer bounded inspection (`grep -n`, `tail -50`) instead of reading full logs into context.
|
||||
- Clear session logs when finished:
|
||||
`agent-device logs clear`
|
||||
- Close session to stop background logging state:
|
||||
`agent-device close`
|
||||
|
||||
## Quick Flow
|
||||
|
||||
```bash
|
||||
agent-device open MyApp --platform ios # or --platform android
|
||||
agent-device logs clear --restart # Preferred: stop stream, clear logs, and start streaming again
|
||||
agent-device network dump 25 # Parse latest HTTP(s) requests (method/url/status) from app.log
|
||||
agent-device logs path # Print path, e.g. ~/.agent-device/sessions/default/app.log
|
||||
agent-device logs doctor # Check tool/runtime readiness for current session/device
|
||||
agent-device logs mark "before tap" # Insert a timeline marker into app.log
|
||||
# ... run flows; on failure, grep the path (see below)
|
||||
agent-device logs stop # Stop streaming (optional; close also stops)
|
||||
```
|
||||
|
||||
Precondition: `logs clear --restart` requires an active app session (`open <app>` first).
|
||||
|
||||
## Command Notes
|
||||
|
||||
- `logs path`: returns log file path and metadata (`active`, `state`, `backend`, size, timestamps).
|
||||
- `logs start`: starts streaming; requires an active app session (`open` first). Supported on iOS simulator, iOS device, Android, and macOS app sessions.
|
||||
- `logs stop`: stops streaming. Session `close` also stops logging.
|
||||
- `logs clear`: truncates `app.log` and removes rotated `app.log.N` files. Requires logging to be stopped first.
|
||||
- `logs clear --restart`: convenience reset for repro loops (stop stream, clear files, restart stream).
|
||||
- `logs doctor`: reports backend/tool checks and readiness notes for troubleshooting.
|
||||
- `logs mark`: writes a timestamped marker line to the session log.
|
||||
- `network dump [limit] [summary|headers|body|all]`: parses recent HTTP(s) lines from the session app log and returns request summaries.
|
||||
- `network log ...`: alias for `network dump`.
|
||||
- On macOS, logs/network are still app-scoped. They rely on Unified Logging output associated with the active session app rather than packet capture.
|
||||
|
||||
## Behavior and Limits
|
||||
|
||||
- `logs start` appends to `app.log` and rotates to `app.log.1` when `app.log` exceeds 5 MB.
|
||||
- `network dump` scans the last 4000 app-log lines, returns up to 200 entries, and truncates payload/header fields at 2048 characters.
|
||||
- Android log streaming automatically rebinds to the app PID after process restarts.
|
||||
- iOS log capture relies on Unified Logging signals (for example `os_log`); plain stdout/stderr output may be limited depending on app/runtime.
|
||||
- Retention knobs:
|
||||
- `AGENT_DEVICE_APP_LOG_MAX_BYTES`
|
||||
- `AGENT_DEVICE_APP_LOG_MAX_FILES`
|
||||
- Optional write-time redaction patterns:
|
||||
- `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS` (comma-separated regex)
|
||||
|
||||
## Grep Patterns
|
||||
|
||||
After getting the path from `logs path`, run `grep` (or `grep -E`) so only matching lines enter context.
|
||||
|
||||
```bash
|
||||
# Get path first, then grep it; -n adds line numbers
|
||||
grep -n "Error\|Exception\|Fatal" <path>
|
||||
grep -n -E "Error|Exception|Fatal|crash" <path>
|
||||
|
||||
# Bounded context: last N lines only
|
||||
tail -50 <path>
|
||||
```
|
||||
|
||||
- Use `-n` for line numbers.
|
||||
- Use `-E` for extended regex so `|` in the pattern does not need escaping.
|
||||
- Prefer targeted patterns (e.g. `Error`, `Exception`, or app-specific tags) over reading the full file.
|
||||
|
||||
## Crash Triage Fast Path
|
||||
|
||||
Always start from the session app log, then branch by platform.
|
||||
|
||||
```bash
|
||||
agent-device logs path
|
||||
grep -n -E "SIGABRT|SIGSEGV|EXC_|fatal|exception|terminated|killed|jetsam|memorystatus|FATAL EXCEPTION|Abort message" <path>
|
||||
nl -ba <path> | sed -n '<start>,<end>p'
|
||||
```
|
||||
|
||||
### iOS
|
||||
|
||||
```bash
|
||||
# If log shows ReportCrash / SIGABRT / EXC_*, inspect simulator DiagnosticReports:
|
||||
ls -lt ~/Library/Logs/DiagnosticReports | grep -E "<AppName>|<BundleId>" | head
|
||||
```
|
||||
|
||||
- `SIGABRT`: app/runtime abort; inspect `.ips` triggered thread and top frames.
|
||||
- `SIGKILL` + jetsam/memorystatus markers: memory-pressure kill.
|
||||
- `EXC_BAD_ACCESS`/`SIGSEGV`: native memory access issue.
|
||||
|
||||
### Android
|
||||
|
||||
```bash
|
||||
# Capture fatal crash lines around app process death:
|
||||
adb -s <serial> logcat -d | grep -n -E "FATAL EXCEPTION|Process: <package>|Abort message|signal [0-9]+ \\(SIG"
|
||||
```
|
||||
|
||||
- `FATAL EXCEPTION` with Java stack: uncaught Java/Kotlin exception.
|
||||
- `signal 6 (SIGABRT)` or `signal 11 (SIGSEGV)` with tombstone refs: native crash path (NDK/JNI/runtime).
|
||||
- `Low memory killer` / `Killing <pid>` entries: OS memory-pressure/process reclaim.
|
||||
|
||||
## Stop Conditions
|
||||
|
||||
- If no crash signature appears in app log, switch to platform-native crash sources (`.ips` on iOS, logcat/tombstone flow on Android).
|
||||
- If signatures are present and root cause class is identified (abort, native fault, memory pressure), stop collecting broad logs and focus on reproducing the specific path.
|
||||
@@ -1,129 +1,87 @@
|
||||
# macOS Desktop Automation
|
||||
# macOS Desktop
|
||||
|
||||
Use this reference for host Mac apps such as Finder, TextEdit, System Settings, Preview, or browser apps running as normal desktop windows.
|
||||
## When to open this file
|
||||
|
||||
## Start here
|
||||
Open this file only when `--platform macos` is involved or the task needs `frontmost-app`, `desktop`, or `menubar` surfaces.
|
||||
|
||||
- Use `open <app> --platform macos` when you need to act inside one app.
|
||||
- Use `open --platform macos --surface frontmost-app|desktop|menubar` when you need to inspect desktop-global UI first.
|
||||
- Use `app` sessions for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record`.
|
||||
- Use `frontmost-app`, `desktop`, and `menubar` mainly for `snapshot`, `get`, `is`, and `wait`.
|
||||
- Prefer `@ref` or selectors. Avoid raw coordinates unless there is no better target.
|
||||
## Main commands to reach for first
|
||||
|
||||
## Mental model
|
||||
- `open <app> --platform macos`
|
||||
- `open --platform macos --surface frontmost-app|desktop|menubar`
|
||||
- `snapshot -i`
|
||||
- `get`
|
||||
- `is`
|
||||
- `click --button secondary`
|
||||
|
||||
- `snapshot -i` should describe UI visible to a human.
|
||||
- Context menus are not ambient UI. Open them explicitly with `click --button secondary`, then re-snapshot.
|
||||
- Prefer refs for exploration and selectors for deterministic replay/assertions.
|
||||
- If you inspect with `desktop` or `menubar` and then need to act on one app, switch to a normal `app` session.
|
||||
## Most common mistake to avoid
|
||||
|
||||
## Canonical app flow
|
||||
Do not treat every macOS surface the same. Use the normal `app` surface when you want to act inside one app. Use `frontmost-app`, `desktop`, or `menubar` mainly to inspect what is visible before switching back to `app` for most interactions.
|
||||
|
||||
## Canonical loop
|
||||
|
||||
```bash
|
||||
agent-device open Finder --platform macos
|
||||
agent-device snapshot -i
|
||||
agent-device click @e66 --button secondary --platform macos
|
||||
agent-device open TextEdit --platform macos
|
||||
agent-device snapshot -i
|
||||
agent-device fill @e3 "desktop smoke test"
|
||||
agent-device close
|
||||
```
|
||||
|
||||
## Canonical desktop-global flow
|
||||
## Surface rules
|
||||
|
||||
```bash
|
||||
agent-device open --platform macos --surface desktop
|
||||
agent-device snapshot -i
|
||||
agent-device get attrs @e4
|
||||
agent-device is visible 'role="window" label="Finder"'
|
||||
agent-device close
|
||||
```
|
||||
|
||||
Surface variants:
|
||||
|
||||
```bash
|
||||
agent-device open --platform macos --surface frontmost-app
|
||||
agent-device open --platform macos --surface desktop
|
||||
agent-device open --platform macos --surface menubar
|
||||
```
|
||||
|
||||
- `app`: default session surface; use this for most real interaction work.
|
||||
- `app`: default surface and the normal choice for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record`.
|
||||
- `frontmost-app`: inspect the currently focused app without naming it first.
|
||||
- `desktop`: inspect visible desktop windows across apps.
|
||||
- `menubar`: inspect the active app menu bar and system menu extras.
|
||||
|
||||
Use `frontmost-app`, `desktop`, and `menubar` for read/inspect flows first. If the next step is a click/fill/press/scroll in one app, switch back to `app`.
|
||||
Use inspect-first surfaces to understand desktop-global UI, then switch back to `app` when you need to act in one app.
|
||||
|
||||
## What to expect from snapshots
|
||||
## Snapshot expectations
|
||||
|
||||
- `app` snapshots should focus on the chosen app window.
|
||||
- `desktop` snapshots can contain multiple windows from multiple apps.
|
||||
- `menubar` snapshots can contain both app-menu items and system menu extras.
|
||||
- File rows, sidebar items, toolbar controls, search fields, and visible context menus should appear.
|
||||
- Finder and other native apps may expose duplicate-looking structures such as row wrapper nodes, `cell` nodes, and child `text` or `text-field` nodes.
|
||||
- Treat those as distinct AX nodes unless you have a stronger selector anchor.
|
||||
- `snapshot -i` should describe UI visible to a human.
|
||||
- `desktop` snapshots can include multiple windows from multiple apps.
|
||||
- `menubar` snapshots can include both app-menu items and system menu extras.
|
||||
- Finder-style rows, sidebar items, toolbar controls, search fields, and opened context menus should appear when visible.
|
||||
- Finder and other native apps may expose duplicate-looking row, cell, and child text nodes. Treat them as distinct AX nodes unless you have a stronger selector anchor.
|
||||
|
||||
## Context menus
|
||||
|
||||
Use secondary click when the app exposes actions only through the contextual menu.
|
||||
Context menus are not ambient UI. Open them explicitly, then re-snapshot.
|
||||
|
||||
```bash
|
||||
agent-device click @e66 --button secondary --platform macos
|
||||
agent-device snapshot -i
|
||||
```
|
||||
|
||||
Expected pattern:
|
||||
Expected loop:
|
||||
|
||||
1. Snapshot visible content.
|
||||
2. Secondary-click the target row/item.
|
||||
2. Secondary-click the target item.
|
||||
3. Snapshot again.
|
||||
4. Interact with newly visible `menu-item` nodes.
|
||||
4. Interact with the new `menu-item` nodes.
|
||||
|
||||
Do not expect context-menu items to appear before the menu is opened.
|
||||
## Targeting rules
|
||||
|
||||
Do not use `longpress` as a substitute for right-click on macOS.
|
||||
- Prefer selectors or `@ref` values over raw coordinates.
|
||||
- On macOS, window position can vary across runs, so coordinate-only flows are fragile.
|
||||
- If the task only needs shared exploration rules, return to [exploration.md](exploration.md).
|
||||
|
||||
## Finder-specific guidance
|
||||
Selector guidance:
|
||||
|
||||
- `snapshot -i` should still expose visible folder rows even when nothing is selected.
|
||||
- Unselected folder contents should still be visible in `snapshot -i` through list/table rows.
|
||||
- A file row may expose multiple nodes with the same label, including a row container, name cell, and child text/text-field.
|
||||
- For opening a context menu, prefer the outer visible row/cell ref over a nested text child if both exist.
|
||||
- After secondary click, expect actions such as `Rename`, `Quick Look`, `Copy`, `Compress`, and tag-related items in the next snapshot.
|
||||
- Good selectors usually anchor on stable labels or app-owned identifiers such as `label="Downloads"` or `role=menu-item label="Rename"`.
|
||||
- Avoid relying on framework-generated `_NS:*` identifiers as stable selectors.
|
||||
|
||||
## Raw snapshots
|
||||
Use `snapshot --raw --platform macos` only when debugging AX structure or collector filtering. Do not make raw snapshots the default agent loop.
|
||||
|
||||
Use `snapshot --raw` only when debugging AX structure or collector issues.
|
||||
Things not to rely on:
|
||||
|
||||
```bash
|
||||
agent-device snapshot --raw --platform macos
|
||||
```
|
||||
- Mobile-only helpers such as `install`, `reinstall`, or `push`.
|
||||
- Desktop-global click or fill parity from `desktop` or `menubar` sessions.
|
||||
- Raw coordinate assumptions across runs.
|
||||
|
||||
- Raw output is larger and less token-efficient.
|
||||
- It is useful for verifying whether missing UI is absent from the AX tree or only filtered from interactive output.
|
||||
- Do not use raw output as the default agent loop when `snapshot -i` already shows the visible window content you need.
|
||||
Troubleshooting:
|
||||
|
||||
## Selector guidance
|
||||
|
||||
Good macOS selectors usually anchor on one of:
|
||||
|
||||
- `label="Downloads"`
|
||||
- `label="failed-step.json"`
|
||||
- `role=button label="Search"`
|
||||
- `role=menu-item label="Rename"`
|
||||
- `role=window label="Notes"`
|
||||
|
||||
Prefer exact labels when the desktop UI is stable. Use `id=...` when the AX identifier is clearly app-owned and not a framework-generated `_NS:*` value.
|
||||
|
||||
## Things not to rely on
|
||||
|
||||
- Mobile-only helpers like `install`, `reinstall`, and `push`
|
||||
- Desktop-global click/fill parity from `desktop` or `menubar` sessions
|
||||
- Raw coordinate assumptions across runs; macOS windows can move
|
||||
- Framework-generated `_NS:*` identifiers as stable selectors
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- If visible window content is missing from `snapshot -i`, re-snapshot once after the UI settles.
|
||||
- If `desktop` is too broad, retry with `frontmost-app` to narrow the inspect surface.
|
||||
- If `menubar` is missing the menu you expect, make the target app frontmost first, then re-open the `menubar` surface and snapshot again.
|
||||
- If the wrong menu opened or no menu appeared, retry secondary-clicking the row/cell wrapper instead of the nested text node.
|
||||
- If the app has multiple windows, ensure the correct one is frontmost before relying on refs.
|
||||
- If visible content is missing from `snapshot -i`, re-snapshot after the UI settles.
|
||||
- If `desktop` is too broad, retry with `frontmost-app`.
|
||||
- If `menubar` is missing the expected menu, make the app frontmost first and retry.
|
||||
- If the wrong menu opened, retry secondary-clicking the row or cell wrapper rather than the nested text node.
|
||||
- If the app has multiple windows, make the correct window frontmost before relying on refs.
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
# Performance Metrics (`perf` / `metrics`)
|
||||
|
||||
Use this reference when you need to measure launch performance in agent workflows.
|
||||
|
||||
## Quick flow
|
||||
|
||||
```bash
|
||||
agent-device open Settings --platform ios
|
||||
agent-device perf --json
|
||||
```
|
||||
|
||||
Alias:
|
||||
|
||||
```bash
|
||||
agent-device metrics --json
|
||||
```
|
||||
|
||||
## What is measured today
|
||||
|
||||
- Session-scoped `startup` timing only.
|
||||
- Sampling method: `open-command-roundtrip`.
|
||||
- Unit: milliseconds (`ms`).
|
||||
- Source: elapsed wall-clock time around each session `open` command dispatch for the active app target.
|
||||
|
||||
## Output fields to use
|
||||
|
||||
- `metrics.startup.lastDurationMs`: most recent startup sample.
|
||||
- `metrics.startup.lastMeasuredAt`: ISO timestamp of most recent sample.
|
||||
- `metrics.startup.sampleCount`: number of retained samples.
|
||||
- `metrics.startup.samples[]`: recent startup history for the current session.
|
||||
- `sampling.startup.method`: current sampling method identifier.
|
||||
|
||||
## Platform support (current)
|
||||
|
||||
- iOS simulator: supported for startup sampling.
|
||||
- iOS physical device: supported for startup sampling.
|
||||
- Android emulator/device: supported for startup sampling.
|
||||
- `fps`, `memory`, and `cpu`: currently placeholders (`available: false`).
|
||||
|
||||
## Interpretation guidance
|
||||
|
||||
- Treat startup values as command round-trip timing, not true app first-frame or first-interactive telemetry.
|
||||
- Compare like-for-like runs:
|
||||
- same device target
|
||||
- same app build
|
||||
- same workflow/session steps
|
||||
- Use multiple runs and compare trend/median, not one-off samples.
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
- Running `perf` before any `open` in the session yields no startup sample yet.
|
||||
- Comparing values across different devices/runtimes introduces large noise.
|
||||
- Interpreting current `startup` as CPU/FPS/memory would be incorrect.
|
||||
@@ -1,70 +0,0 @@
|
||||
# Permissions and Setup
|
||||
|
||||
## iOS snapshots
|
||||
|
||||
iOS snapshots use XCTest and do not require macOS Accessibility permissions.
|
||||
|
||||
## iOS physical device runner
|
||||
|
||||
For iOS physical devices, XCTest runner setup requires valid signing/provisioning.
|
||||
Use Automatic Signing in Xcode, or provide optional overrides:
|
||||
|
||||
- `AGENT_DEVICE_IOS_TEAM_ID`
|
||||
- `AGENT_DEVICE_IOS_SIGNING_IDENTITY`
|
||||
- `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`
|
||||
- `AGENT_DEVICE_IOS_BUNDLE_ID` (optional runner bundle-id base override)
|
||||
|
||||
Free Apple Developer (Personal Team) accounts may reject generic bundle IDs as unavailable.
|
||||
Set `AGENT_DEVICE_IOS_BUNDLE_ID` to a unique reverse-DNS identifier when that happens.
|
||||
|
||||
Security guidance for these overrides:
|
||||
|
||||
- These variables are optional and only needed for physical-device XCTest setup.
|
||||
- Treat values as sensitive host configuration; do not share in chat logs or commit to source control.
|
||||
- Do not provide private keys or unrelated secrets; use the minimum values required for signing.
|
||||
- Prefer Xcode Automatic Signing when possible to reduce manual secret/config handling.
|
||||
- For autonomous/CI runs, keep these unset by default and require explicit opt-in for physical-device workflows.
|
||||
|
||||
If setup/build takes long, increase:
|
||||
|
||||
- `AGENT_DEVICE_DAEMON_TIMEOUT_MS` (default `90000`, for example `120000`)
|
||||
|
||||
If daemon startup fails with stale metadata hints, clean stale files and retry:
|
||||
|
||||
- `~/.agent-device/daemon.json`
|
||||
- `~/.agent-device/daemon.lock`
|
||||
|
||||
## iOS permission alerts (simulator only)
|
||||
|
||||
iOS apps trigger system permission dialogs (camera, location, notifications, etc.) on first use.
|
||||
Use `alert` to handle them without tapping coordinates:
|
||||
|
||||
```bash
|
||||
agent-device alert wait # block until an alert appears (default 10 s timeout)
|
||||
agent-device alert accept # accept the frontmost alert
|
||||
agent-device alert dismiss # dismiss the frontmost alert
|
||||
agent-device alert get # read alert title/message without acting
|
||||
```
|
||||
|
||||
**Timing note:** `alert accept` and `alert dismiss` include a built-in 2 s retry window.
|
||||
If the alert is present in the UI hierarchy but not yet interactive, the command retries every 300 ms
|
||||
rather than failing immediately. You do not need to add manual sleeps between triggering the alert
|
||||
and accepting it.
|
||||
|
||||
**Preferred pattern for clean simulator sessions:**
|
||||
|
||||
```bash
|
||||
agent-device open MyApp --platform ios
|
||||
agent-device alert wait 5000 # wait up to 5 s for the permission prompt
|
||||
agent-device alert accept # accept; retries internally if not yet actionable
|
||||
```
|
||||
|
||||
`alert` is only supported on iOS simulators; iOS physical devices are not supported.
|
||||
|
||||
## iOS: "Allow Paste" dialog
|
||||
|
||||
iOS 16+ shows an "Allow Paste" prompt when an app reads the system pasteboard. Under XCUITest (which `agent-device` uses), this prompt is suppressed by the testing runtime. Use `xcrun simctl pbcopy booted` to set clipboard content directly on the simulator instead.
|
||||
|
||||
## Simulator troubleshooting
|
||||
|
||||
- If snapshots return 0 nodes, restart Simulator and re-open the app.
|
||||
@@ -1,56 +1,56 @@
|
||||
# Remote Tenancy and Lease Admission
|
||||
# Remote Tenancy
|
||||
|
||||
Use this reference for remote daemon HTTP flows that require explicit
|
||||
tenant/run admission control.
|
||||
## When to open this file
|
||||
|
||||
## Transport prerequisites
|
||||
Open this file only for remote daemon HTTP flows that require explicit daemon URL setup, authentication, lease allocation, or tenant-scoped command admission.
|
||||
|
||||
- Start daemon in HTTP mode (`AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual`).
|
||||
- Point remote clients at the host with `AGENT_DEVICE_DAEMON_BASE_URL=http(s)://host:port[/base-path]`
|
||||
or `--daemon-base-url <url>` so the CLI skips local daemon discovery/startup.
|
||||
- Use `AGENT_DEVICE_DAEMON_AUTH_TOKEN` / `--daemon-auth-token` when the client should send the
|
||||
shared daemon token automatically.
|
||||
- Direct JSON-RPC callers can use a token in params, `Authorization: Bearer <token>`, or
|
||||
`x-agent-device-token`.
|
||||
- Prefer an auth hook (`AGENT_DEVICE_HTTP_AUTH_HOOK`) for caller validation and
|
||||
tenant injection.
|
||||
## Main commands to reach for first
|
||||
|
||||
## Lease lifecycle (JSON-RPC)
|
||||
- `AGENT_DEVICE_DAEMON_BASE_URL=...`
|
||||
- `AGENT_DEVICE_DAEMON_AUTH_TOKEN=...`
|
||||
- `curl ... agent_device.lease.allocate`
|
||||
- `curl ... agent_device.lease.heartbeat`
|
||||
- `curl ... agent_device.lease.release`
|
||||
- `agent-device --tenant ... --session-isolation tenant --run-id ... --lease-id ...`
|
||||
|
||||
Use `POST /rpc` with JSON-RPC 2.0 methods:
|
||||
## Most common mistake to avoid
|
||||
|
||||
- `agent_device.lease.allocate`
|
||||
- `agent_device.lease.heartbeat`
|
||||
- `agent_device.lease.release`
|
||||
Do not run a tenant-isolated command without matching `tenant`, `run`, and `lease` scope. Admission checks require all three to line up.
|
||||
|
||||
Example allocate:
|
||||
## Canonical loop
|
||||
|
||||
```bash
|
||||
export AGENT_DEVICE_DAEMON_BASE_URL=http://mac-host.example:4310
|
||||
export AGENT_DEVICE_DAEMON_AUTH_TOKEN=<token>
|
||||
|
||||
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
||||
-H "content-type: application/json" \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
-d '{"jsonrpc":"2.0","id":"alloc-1","method":"agent_device.lease.allocate","params":{"tenantId":"acme","runId":"run-123","ttlMs":60000}}'
|
||||
|
||||
agent-device \
|
||||
--tenant acme \
|
||||
--session-isolation tenant \
|
||||
--run-id run-123 \
|
||||
--lease-id <lease-id> \
|
||||
session list --json
|
||||
```
|
||||
|
||||
Example heartbeat:
|
||||
Heartbeat and release example:
|
||||
|
||||
```bash
|
||||
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
||||
-H "content-type: application/json" \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
-d '{"jsonrpc":"2.0","id":"hb-1","method":"agent_device.lease.heartbeat","params":{"leaseId":"<lease-id>","ttlMs":60000}}'
|
||||
```
|
||||
|
||||
Example release:
|
||||
|
||||
```bash
|
||||
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
||||
-H "content-type: application/json" \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
-d '{"jsonrpc":"2.0","id":"rel-1","method":"agent_device.lease.release","params":{"leaseId":"<lease-id>"}}'
|
||||
```
|
||||
|
||||
Example session-locked command request:
|
||||
Session-locked RPC command example:
|
||||
|
||||
```bash
|
||||
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
||||
@@ -59,11 +59,34 @@ curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
|
||||
-d '{"jsonrpc":"2.0","id":"cmd-1","method":"agent_device.command","params":{"session":"qa-ios","command":"snapshot","positionals":[],"meta":{"lockPolicy":"reject","lockPlatform":"ios","tenantId":"acme","runId":"run-123","leaseId":"<lease-id>"}}}'
|
||||
```
|
||||
|
||||
Direct RPC callers can send the same session lock concept as the CLI and typed client through `meta.lockPolicy` and optional `meta.lockPlatform`.
|
||||
## Transport prerequisites
|
||||
|
||||
- Start the daemon in HTTP mode with `AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual`.
|
||||
- Point the client at the remote host with `AGENT_DEVICE_DAEMON_BASE_URL=http(s)://host:port[/base-path]`.
|
||||
- Use `AGENT_DEVICE_DAEMON_AUTH_TOKEN` or `--daemon-auth-token` when the client should send the shared daemon token automatically.
|
||||
- Direct JSON-RPC callers can authenticate with request params, `Authorization: Bearer <token>`, or `x-agent-device-token`.
|
||||
- Prefer an auth hook such as `AGENT_DEVICE_HTTP_AUTH_HOOK` when the host needs caller validation or tenant injection.
|
||||
|
||||
## Lease lifecycle
|
||||
|
||||
Use JSON-RPC methods on `POST /rpc`:
|
||||
|
||||
- `agent_device.lease.allocate`
|
||||
- `agent_device.lease.heartbeat`
|
||||
- `agent_device.lease.release`
|
||||
|
||||
Keep the lease alive for the duration of the run and release it when the tenant-scoped work is complete.
|
||||
|
||||
Host-level lease knobs:
|
||||
|
||||
- `AGENT_DEVICE_MAX_SIMULATOR_LEASES`
|
||||
- `AGENT_DEVICE_LEASE_TTL_MS`
|
||||
- `AGENT_DEVICE_LEASE_MIN_TTL_MS`
|
||||
- `AGENT_DEVICE_LEASE_MAX_TTL_MS`
|
||||
|
||||
## Command admission contract
|
||||
|
||||
For tenant-isolated command execution, pass all four flags:
|
||||
For tenant-isolated command execution, pass all four CLI flags together:
|
||||
|
||||
```bash
|
||||
agent-device \
|
||||
@@ -74,25 +97,11 @@ agent-device \
|
||||
session list --json
|
||||
```
|
||||
|
||||
Admission checks require tenant/run/lease scope alignment.
|
||||
The CLI sends `AGENT_DEVICE_DAEMON_AUTH_TOKEN` in both the JSON-RPC request token field and HTTP auth headers so existing daemon auth paths continue to work.
|
||||
|
||||
The CLI sends `AGENT_DEVICE_DAEMON_AUTH_TOKEN` in both the JSON-RPC request token field and HTTP
|
||||
auth headers so existing daemon auth paths continue to work.
|
||||
## Failure semantics and trust notes
|
||||
|
||||
## Failure semantics
|
||||
|
||||
- Missing tenant/run/lease fields in tenant isolation mode: `INVALID_ARGS`
|
||||
- Lease not active or wrong scope: `UNAUTHORIZED`
|
||||
- Method mismatch: JSON-RPC `-32601` (HTTP 404)
|
||||
|
||||
## Operational guidance
|
||||
|
||||
- Keep TTL short and heartbeat only while a run is active.
|
||||
- Release lease immediately on run completion/error paths.
|
||||
- For remote debug sessions, inspect logs on the remote host; client-side `--debug` no longer tails
|
||||
a local daemon log when `AGENT_DEVICE_DAEMON_BASE_URL` is set.
|
||||
- For bounded hosts, configure:
|
||||
- `AGENT_DEVICE_MAX_SIMULATOR_LEASES`
|
||||
- `AGENT_DEVICE_LEASE_TTL_MS`
|
||||
- `AGENT_DEVICE_LEASE_MIN_TTL_MS`
|
||||
- `AGENT_DEVICE_LEASE_MAX_TTL_MS`
|
||||
- Missing tenant, run, or lease fields in tenant-isolation mode should fail as `INVALID_ARGS`.
|
||||
- Inactive or scope-mismatched leases should fail as `UNAUTHORIZED`.
|
||||
- Inspect logs on the remote host during remote debugging. Client-side `--debug` does not tail a local daemon log once `AGENT_DEVICE_DAEMON_BASE_URL` is set.
|
||||
- Treat daemon auth tokens and lease identifiers as sensitive operational data.
|
||||
|
||||
@@ -1,101 +0,0 @@
|
||||
# Session Management
|
||||
|
||||
## Named sessions
|
||||
|
||||
```bash
|
||||
agent-device --session auth open Settings --platform ios
|
||||
agent-device --session auth snapshot -i
|
||||
```
|
||||
|
||||
Sessions isolate device context. A device can only be held by one session at a time.
|
||||
|
||||
## Best practices
|
||||
|
||||
- Name sessions semantically.
|
||||
- Close sessions when done.
|
||||
- Use separate sessions for parallel work.
|
||||
- For orchestrated QA runs, prefer a pre-bound session/platform over repeating per-command selectors.
|
||||
- For remote tenant-scoped automation, run commands with:
|
||||
`--tenant <id> --session-isolation tenant --run-id <id> --lease-id <id>`
|
||||
- In iOS sessions, use `open <app>`. `open <url>` opens deep links; on devices `http(s)://` opens Safari when no app is active, and custom schemes require an active app in the session.
|
||||
- In iOS sessions, `open <app> <url>` opens a deep link.
|
||||
- On iOS, `appstate` is session-scoped and requires a matching active session on the target device.
|
||||
- For dev loops where runtime state can persist (for example React Native Fast Refresh), use `open <app> --relaunch` to restart the app process in the same session.
|
||||
- Use `--save-script [path]` to record replay scripts on `close`; path is a file path and parent directories are created automatically.
|
||||
- Use `close --shutdown` on iOS simulators or Android emulators to shut down the target as part of session teardown, preventing resource leakage in multi-tenant or CI workloads.
|
||||
- For ambiguous bare `--save-script` values, prefer `--save-script=workflow.ad` or `./workflow.ad`.
|
||||
- For deterministic replay scripts, prefer selector-based actions and assertions.
|
||||
- Use `replay -u` to update selector drift during maintenance.
|
||||
|
||||
## Session-bound automation
|
||||
|
||||
Use this when an external orchestrator must keep every CLI call on the same session/device without a wrapper script.
|
||||
|
||||
```bash
|
||||
export AGENT_DEVICE_SESSION=qa-ios
|
||||
export AGENT_DEVICE_PLATFORM=ios
|
||||
export AGENT_DEVICE_SESSION_LOCK=strip
|
||||
|
||||
agent-device open MyApp --relaunch
|
||||
agent-device snapshot -i
|
||||
agent-device press @e3
|
||||
agent-device close
|
||||
```
|
||||
|
||||
- `AGENT_DEVICE_SESSION` and `AGENT_DEVICE_PLATFORM` provide the default binding when `--session` and `--platform` are omitted.
|
||||
- A configured `AGENT_DEVICE_SESSION` enables lock policy enforcement by convention. The default mode is `reject`.
|
||||
- `--session-lock reject|strip` or `AGENT_DEVICE_SESSION_LOCK=reject|strip` controls whether conflicting selectors fail or are ignored.
|
||||
- The daemon enforces the same lock policy for CLI requests, typed client calls, and direct RPC commands.
|
||||
- Conflicts include explicit retargeting selectors such as `--platform`, `--target`, `--device`, `--udid`, `--serial`, `--ios-simulator-device-set`, and `--android-device-allowlist`.
|
||||
- `--session-locked`, `--session-lock-conflicts`, `AGENT_DEVICE_SESSION_LOCKED`, and `AGENT_DEVICE_SESSION_LOCK_CONFLICTS` remain supported as compatibility aliases.
|
||||
- Lock policy applies to nested `batch` steps too. If a step omits `platform`, it still inherits the parent batch `--platform` instead of being silently replaced by an environment default.
|
||||
|
||||
Android emulator variant:
|
||||
|
||||
```bash
|
||||
export AGENT_DEVICE_SESSION=qa-android
|
||||
export AGENT_DEVICE_PLATFORM=android
|
||||
|
||||
agent-device reinstall MyApp /path/to/app-debug.apk --serial emulator-5554
|
||||
agent-device --session-lock reject open com.example.myapp --relaunch
|
||||
agent-device snapshot -i
|
||||
agent-device close --shutdown
|
||||
```
|
||||
|
||||
## Scoped device isolation
|
||||
|
||||
Use scoped discovery when sessions must not see host-global device lists.
|
||||
|
||||
```bash
|
||||
agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators
|
||||
agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234
|
||||
```
|
||||
|
||||
- Scope is applied before selectors (`--device`, `--udid`, `--serial`).
|
||||
- If selector target is outside scope, resolution fails with `DEVICE_NOT_FOUND`.
|
||||
- If the scoped iOS simulator set is empty (first-run), the error includes the set path and a suggested `xcrun simctl --set <path> create ...` command.
|
||||
- With iOS simulator-set scope enabled, iOS physical devices are not enumerated.
|
||||
- Environment equivalents:
|
||||
- `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` (compat: `IOS_SIMULATOR_DEVICE_SET`)
|
||||
- `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST` (compat: `ANDROID_DEVICE_ALLOWLIST`)
|
||||
|
||||
## Listing sessions
|
||||
|
||||
```bash
|
||||
agent-device session list
|
||||
```
|
||||
|
||||
iOS session entries include `device_udid` and `ios_simulator_device_set` (null when using the default set). Use these fields to confirm device routing in concurrent multi-session runs without additional `simctl` calls.
|
||||
|
||||
## Replay within sessions
|
||||
|
||||
```bash
|
||||
agent-device replay ./session.ad --session auth
|
||||
agent-device replay -u ./session.ad --session auth
|
||||
```
|
||||
|
||||
## Tenant isolation note
|
||||
|
||||
When session isolation is set to tenant mode, session namespace is scoped as
|
||||
`<tenant>:<session>`. For remote runs, allocate and maintain an active lease
|
||||
for the same tenant/run scope before executing tenant-isolated commands.
|
||||
@@ -1,105 +0,0 @@
|
||||
# Snapshot Refs and Selectors
|
||||
|
||||
## Purpose
|
||||
|
||||
Refs are useful for discovery/debugging. For deterministic scripts, use selectors.
|
||||
For tap interactions, `press` is canonical; `click` is an equivalent alias.
|
||||
For host Mac desktop apps, pair this reference with [macos-desktop.md](macos-desktop.md) because context menus and native list/table structures need desktop-specific handling.
|
||||
|
||||
## Snapshot
|
||||
|
||||
```bash
|
||||
agent-device snapshot -i
|
||||
```
|
||||
|
||||
Use plain `snapshot` when you need to verify whether text is visible on screen.
|
||||
Use `snapshot -i` mainly for interactive exploration and choosing refs.
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
Page: com.apple.Preferences
|
||||
App: com.apple.Preferences
|
||||
|
||||
@e1 [ioscontentgroup]
|
||||
@e2 [button] "Camera"
|
||||
@e3 [button] "Privacy & Security"
|
||||
```
|
||||
|
||||
## Using refs (discovery/debug)
|
||||
|
||||
```bash
|
||||
agent-device press @e2
|
||||
agent-device fill @e5 "test"
|
||||
```
|
||||
|
||||
On macOS, if actions live in a context menu, use:
|
||||
|
||||
```bash
|
||||
agent-device click @e5 --button secondary --platform macos
|
||||
agent-device snapshot -i
|
||||
```
|
||||
|
||||
## Using selectors (deterministic)
|
||||
|
||||
```bash
|
||||
agent-device press 'id="camera_row" || label="Camera" role=button'
|
||||
agent-device fill 'id="search_input" editable=true' "test"
|
||||
agent-device is visible 'id="camera_settings_anchor"'
|
||||
```
|
||||
|
||||
## Ref lifecycle
|
||||
|
||||
Refs can become invalid when UI changes (navigation, modal, dynamic list updates).
|
||||
Re-snapshot after transitions if you keep using refs.
|
||||
|
||||
## Scope snapshots
|
||||
|
||||
Use `-s` to scope to labels/identifiers. This reduces size and speeds up results:
|
||||
|
||||
```bash
|
||||
agent-device snapshot -i -s "Camera"
|
||||
agent-device snapshot -i -s @e3
|
||||
```
|
||||
|
||||
## Diff snapshots (structural)
|
||||
|
||||
Use `diff snapshot` when you need compact state-change visibility between nearby UI states:
|
||||
|
||||
```bash
|
||||
agent-device snapshot -i # First snapshot initializes baseline
|
||||
agent-device press @e5
|
||||
agent-device diff snapshot -i # Shows +/− structural lines vs prior snapshot
|
||||
```
|
||||
|
||||
Efficient pattern:
|
||||
|
||||
- Initialize once at a stable point.
|
||||
- Mutate UI (`press`, `fill`, `swipe`).
|
||||
- Run `diff snapshot` after interactions to confirm expected change shape with bounded output.
|
||||
- Re-run full/scoped `snapshot` only when you need fresh refs for next step selection.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Ref not found: re-snapshot.
|
||||
- If `snapshot` returns 0 nodes, foreground app state or accessibility state may have changed. Re-open the app or retry after state is stable.
|
||||
- On macOS, use `snapshot --raw --platform macos` to distinguish collector filtering from truly missing AX content.
|
||||
|
||||
## Stop Conditions
|
||||
|
||||
- If refs are unstable after UI transitions, switch to selector-based targeting and stop investing in ref-only flows.
|
||||
|
||||
## find click response
|
||||
|
||||
`find "<query>" click --json` returns deterministic matched-target metadata:
|
||||
|
||||
```json
|
||||
{ "ref": "@e3", "locator": "any", "query": "Increment", "x": 195, "y": 422 }
|
||||
```
|
||||
|
||||
Fields come from the matched snapshot node, not the platform runner. Use these for observability and replay quality — they are stable across runs for the same UI state.
|
||||
|
||||
## Replay note
|
||||
|
||||
- Prefer selector-based actions in recorded `.ad` replays.
|
||||
- Use `agent-device replay -u <path>` to update selector drift and rewrite replay scripts in place.
|
||||
@@ -0,0 +1,103 @@
|
||||
# Verification
|
||||
|
||||
## When to open this file
|
||||
|
||||
Open this file when the task needs evidence, regression checks, replay maintenance, or startup performance measurements after the main interaction flow is already working.
|
||||
|
||||
## Main commands to reach for first
|
||||
|
||||
- `screenshot`
|
||||
- `diff snapshot`
|
||||
- `record`
|
||||
- `replay -u`
|
||||
- `perf`
|
||||
|
||||
## Most common mistake to avoid
|
||||
|
||||
Do not use verification tools as the first exploration step. First get the app into the correct state with the normal interaction flow, then capture proof or maintain replay assets.
|
||||
|
||||
## Canonical loop
|
||||
|
||||
```bash
|
||||
agent-device open Settings --platform ios
|
||||
agent-device snapshot -i
|
||||
agent-device press @e5
|
||||
agent-device diff snapshot -i
|
||||
agent-device screenshot /tmp/settings-proof.png
|
||||
agent-device close
|
||||
```
|
||||
|
||||
## Structural verification with diff snapshot
|
||||
|
||||
Use `diff snapshot` when you need a compact view of how the UI changed between nearby states.
|
||||
|
||||
```bash
|
||||
agent-device snapshot -i
|
||||
agent-device press @e5
|
||||
agent-device diff snapshot -i
|
||||
```
|
||||
|
||||
- Initialize the baseline at a stable point.
|
||||
- Perform the mutation.
|
||||
- Run `diff snapshot` to confirm the expected structural change.
|
||||
- Re-run full `snapshot` only when you need fresh refs.
|
||||
|
||||
## Visual artifacts
|
||||
|
||||
Use `screenshot` when the proof needs a rendered image instead of a structural tree.
|
||||
|
||||
## Session recording
|
||||
|
||||
Use `record` for debugging, documentation, or shareable verification artifacts.
|
||||
|
||||
```bash
|
||||
agent-device record start ./recordings/ios.mov
|
||||
agent-device open App
|
||||
agent-device snapshot -i
|
||||
agent-device press @e3
|
||||
agent-device close
|
||||
agent-device record stop
|
||||
```
|
||||
|
||||
- `record` supports iOS simulators, iOS devices, and Android.
|
||||
- On iOS, recording is a wrapper around `simctl` for simulators and the corresponding device capture path for physical devices.
|
||||
- On Android, recording is a wrapper around `adb`.
|
||||
- Recording writes a video artifact and a gesture-telemetry sidecar JSON.
|
||||
- On macOS hosts, touch overlay burn-in is available for supported recordings.
|
||||
- On non-macOS hosts, recording still succeeds but the video stays raw and `record stop` can return an `overlayWarning`.
|
||||
- If the agent already knows the interaction sequence and wants a more lifelike, uninterrupted recording, drive the flow with `batch` while recording instead of replanning between each step.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
agent-device record start ./recordings/smoke.mov
|
||||
agent-device batch --session sim --platform ios --steps-file /tmp/smoke-steps.json --json
|
||||
agent-device record stop
|
||||
```
|
||||
|
||||
- Use this only after exploration has stabilized the flow.
|
||||
- Keep the batch short and add `wait` or `is exists` guards after mutating steps so the recorded flow still tracks realistic UI timing.
|
||||
|
||||
## Replay maintenance
|
||||
|
||||
Use replay updates when selectors drift but the recorded scenario is still correct.
|
||||
|
||||
```bash
|
||||
agent-device replay -u ./session.ad
|
||||
```
|
||||
|
||||
- Prefer selector-based actions in recorded `.ad` replays.
|
||||
- Use update mode for maintenance, not as a substitute for fixing a broken interaction strategy.
|
||||
|
||||
## Performance checks
|
||||
|
||||
Use `perf --json` or `metrics --json` when you need startup timing for the active session.
|
||||
|
||||
```bash
|
||||
agent-device open Settings --platform ios
|
||||
agent-device perf --json
|
||||
```
|
||||
|
||||
- Current startup data is command round-trip timing around `open`.
|
||||
- It is not true first-frame or first-interactive telemetry.
|
||||
- `fps`, `memory`, and `cpu` are currently placeholders.
|
||||
@@ -1,51 +0,0 @@
|
||||
# Video Recording
|
||||
|
||||
Capture device automation sessions as video for debugging, documentation, or verification
|
||||
|
||||
## iOS Simulator
|
||||
|
||||
Use `agent-device record` commands (wrapper around simctl):
|
||||
|
||||
```bash
|
||||
# Start recording
|
||||
agent-device record start ./recordings/ios.mov
|
||||
|
||||
# Perform actions
|
||||
agent-device open App
|
||||
agent-device snapshot -i
|
||||
agent-device click @e3
|
||||
agent-device close
|
||||
|
||||
# Stop recording
|
||||
agent-device record stop
|
||||
```
|
||||
|
||||
`record` supports iOS simulators, physical iOS devices, and Android.
|
||||
|
||||
Recording outputs:
|
||||
|
||||
- a video artifact
|
||||
- a gesture-telemetry sidecar JSON next to the video
|
||||
|
||||
Touch overlay support:
|
||||
|
||||
- macOS host: telemetry can be burned into the video as visible touch overlays
|
||||
- non-macOS host: recording still succeeds, but the video stays raw and `record stop` returns an `overlayWarning`
|
||||
|
||||
## Android Emulator/Device
|
||||
|
||||
Use `agent-device record` commands (wrapper around adb):
|
||||
|
||||
```bash
|
||||
# Start recording
|
||||
agent-device record start ./recordings/android.mp4
|
||||
|
||||
# Perform actions
|
||||
agent-device open App
|
||||
agent-device snapshot -i
|
||||
agent-device click @e3
|
||||
agent-device close
|
||||
|
||||
# Stop recording
|
||||
agent-device record stop
|
||||
```
|
||||
Reference in New Issue
Block a user