diff --git a/AGENTS.md b/AGENTS.md index 2bcec4b73..cb8d9f691 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -116,6 +116,11 @@ Minimal operating guide for AI coding agents in this repo. - For behavior/CLI surface changes, evaluate docs/skills updates. - Update `README.md` and relevant `website/docs/**` pages for command behavior/flags/aliases/workflows. - Update relevant `skills/**/SKILL.md` when usage examples/workflow recommendations change. +- Keep skill docs task-first: + - top-level `SKILL.md` should stay a thin router, not a full manual. + - keep detailed workflows/troubleshooting in a `references/` folder instead of growing the router. + - isolate true platform/infra exceptions (for example macOS-only or remote-tenancy-only guidance) in dedicated files. + - do not delete high-value operational guidance during refactors; move or condense it unless the behavior is obsolete. - In final summaries, state whether docs/skills were updated; if not, explain why. ## When Blocked diff --git a/skills/agent-device/SKILL.md b/skills/agent-device/SKILL.md index c45d05bb9..2008beb4a 100644 --- a/skills/agent-device/SKILL.md +++ b/skills/agent-device/SKILL.md @@ -3,120 +3,37 @@ name: agent-device description: Automates interactions for Apple-platform apps (iOS, tvOS, macOS) and Android devices. Use when navigating apps, taking snapshots/screenshots, tapping, typing, scrolling, or extracting UI info across mobile, TV, and desktop targets. --- -# Apple and Android Automation with agent-device +# agent-device Use this skill as a router. -Core rule: +## QA modes -- explore with `snapshot -i` and `@ref` -- stabilize with selectors -- use plain `snapshot` when you need to verify whether text is visible -- re-snapshot after every meaningful UI change +- Open-ended bug hunt with reporting: use [../dogfood/SKILL.md](../dogfood/SKILL.md). +- Pass/fail QA from acceptance criteria: stay in this skill, start with [references/bootstrap-install.md](references/bootstrap-install.md), then use the QA loop in [references/exploration.md](references/exploration.md). -For exploratory QA bug hunts and reporting, use [../dogfood/SKILL.md](../dogfood/SKILL.md). +## Mental model -## Quick route +- First choose the correct target and open the app or session you want to work on. +- Then inspect the current UI with `snapshot -i` and pick targets from the actual UI state. +- Act with `press`, `fill`, `get`, `is`, `wait`, or `find`. +- Re-snapshot after meaningful UI changes instead of reusing stale refs. +- End by capturing proof if needed, then `close`. -- Normal UI task: `open` -> `snapshot -i` -> `click/fill/press` -> `close` -- Debug task: `open` -> `logs clear --restart` -> reproduce -> `network dump` -> `logs path` -- Replay drift: `replay -u ` -- No target context yet: `devices` -> pick target -> `open` +## Decision rules -## Target rules +- Use plain `snapshot` when you need to verify whether text is visible. +- Use `snapshot -i` mainly for interactive exploration and choosing refs. +- Use `fill` to replace text. +- Use `type` to append text. +- Prefer `@ref` or selector targeting over raw coordinates. +- Keep the default loop short: `open` -> explore/act -> optional debug or verify -> `close`. -- iOS local QA: prefer simulators -- Android binary flow: `install` or `reinstall` first, then `open --relaunch` -- In mixed-device labs, always pin the target with `--device`, `--udid`, `--serial`, or an isolation scope -- For session-bound automation, prefer `AGENT_DEVICE_SESSION` + `AGENT_DEVICE_PLATFORM` +## Choose a reference -## macOS rules - -- Use `open --platform macos` for normal Mac app automation -- Use `open --platform macos --surface frontmost-app|desktop|menubar` when you need desktop-global inspection first -- Use `app` sessions for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record` -- Use `frontmost-app`, `desktop`, and `menubar` mainly for `snapshot`, `get`, `is`, and `wait` -- If you inspect with `desktop` or `menubar` and then need to act inside one app, open that app in a normal `app` session -- Prefer `@ref` or selectors over raw `x y` on macOS -- Use `click --button secondary` for context menus, then run `snapshot -i` again - -## Canonical flows - -### Normal flow - -```bash -agent-device open Settings --platform ios -agent-device snapshot -i -agent-device press @e3 -agent-device fill @e5 "test" -agent-device close -``` - -### macOS app flow - -```bash -agent-device open TextEdit --platform macos -agent-device snapshot -i -agent-device fill @e3 "desktop smoke test" -agent-device screenshot /tmp/macos-textedit.png -agent-device close -``` - -### macOS desktop-global inspect flow - -```bash -agent-device open --platform macos --surface desktop -agent-device snapshot -i -agent-device get attrs @e4 -agent-device is visible 'role="window" label="Notes"' -agent-device wait text "Notes" -agent-device close -``` - -### Android relaunch flow - -```bash -agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554 -agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch -agent-device snapshot -i -agent-device close -``` - -### Debug flow - -```bash -agent-device open MyApp --platform ios -agent-device logs clear --restart -agent-device network dump 25 -agent-device logs path -``` - -### Replay maintenance - -```bash -agent-device replay -u ./session.ad -``` - -## High-value guardrails - -- Prefer `snapshot -i`; use `--raw` only for structure debugging -- Use plain `snapshot` to verify text visibility; use `snapshot -i` mainly for interactive exploration and choosing refs -- Use refs for discovery, selectors for replay/assertions -- `fill` clears then types; `type` only types into the focused field -- `network dump` is best-effort and reads from the session app log -- `logs clear --restart` requires an active app session -- On macOS, helper-backed flows cover permissions, alerts, and desktop-global snapshot surfaces -- On macOS, do not assume `desktop` or `menubar` are the best surface for real interactions yet - -## References - -- [references/macos-desktop.md](references/macos-desktop.md) -- [references/snapshot-refs.md](references/snapshot-refs.md) -- [references/logs-and-debug.md](references/logs-and-debug.md) -- [references/session-management.md](references/session-management.md) -- [references/permissions.md](references/permissions.md) -- [references/video-recording.md](references/video-recording.md) -- [references/coordinate-system.md](references/coordinate-system.md) -- [references/batching.md](references/batching.md) -- [references/perf-metrics.md](references/perf-metrics.md) -- [references/remote-tenancy.md](references/remote-tenancy.md) +- Pick target device, install, open, or manage sessions: [references/bootstrap-install.md](references/bootstrap-install.md) +- Need to discover UI, pick refs, wait, query, or interact: [references/exploration.md](references/exploration.md) +- Need logs, network, alerts, permissions, or failure triage: [references/debugging.md](references/debugging.md) +- Need screenshots, diff, recording, replay maintenance, or perf data: [references/verification.md](references/verification.md) +- Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: [references/macos-desktop.md](references/macos-desktop.md) +- Need to connect to a remote `agent-device` daemon over HTTP or use tenant leases: [references/remote-tenancy.md](references/remote-tenancy.md) diff --git a/skills/agent-device/references/batching.md b/skills/agent-device/references/batching.md deleted file mode 100644 index 2432db641..000000000 --- a/skills/agent-device/references/batching.md +++ /dev/null @@ -1,79 +0,0 @@ -# Batching - -## When to use batch - -- The agent already knows a short sequence of commands. -- Steps belong to one logical screen flow. -- You want one result object with per-step timing and failure context. - -## When not to use batch - -- Flows are unrelated and should be retried independently. -- The workflow is highly dynamic and requires replanning after each step. -- You need human approvals between steps. - -## CLI patterns - -From file: - -```bash -agent-device batch --session sim --platform ios --steps-file /tmp/batch-steps.json --json -``` - -Inline (small payloads only): - -```bash -agent-device batch --steps '[{"command":"open","positionals":["settings"]}]' -``` - -## Step payload contract - -```json -[ - { "command": "open", "positionals": ["settings"], "flags": {} }, - { "command": "wait", "positionals": ["label=\"Privacy & Security\"", "3000"], "flags": {} }, - { "command": "click", "positionals": ["label=\"Privacy & Security\""], "flags": {} }, - { "command": "get", "positionals": ["text", "label=\"Tracking\""], "flags": {} } -] -``` - -Rules: - -- `positionals` optional, defaults to `[]`. -- `flags` optional, defaults to `{}`. -- nested `batch` and `replay` are rejected. -- stop-on-first-error is the supported mode (`--on-error stop`). - -## Response handling - -Success includes: - -- `total`, `executed`, `totalDurationMs` -- `results[]` entries with `step`, `command`, `durationMs`, and optional `data` - -Failure includes: - -- `details.step` -- `details.command` -- `details.executed` -- `details.partialResults` - -Use these fields to replan from the first failing step. - -## Common error categories and agent actions - -- `INVALID_ARGS`: payload/step shape issue; fix payload and retry. -- `SESSION_NOT_FOUND`: open or select the correct session, then retry. -- `UNSUPPORTED_OPERATION`: switch command/target to supported operation. -- `AMBIGUOUS_MATCH`: refine selector/locator, then retry failed step. -- `COMMAND_FAILED`: add sync guard (`wait`, `is exists`) and retry from failed step. - -## Reliability guardrails - -- Add sync guards after mutating steps. -- Assume snapshot/ref drift after navigation. -- Keep batch size moderate (about 5-20 steps). -- Split long workflows into phases: - 1. navigate - 2. verify/extract - 3. cleanup diff --git a/skills/agent-device/references/bootstrap-install.md b/skills/agent-device/references/bootstrap-install.md new file mode 100644 index 000000000..181272799 --- /dev/null +++ b/skills/agent-device/references/bootstrap-install.md @@ -0,0 +1,167 @@ +# Bootstrap and Install + +## When to open this file + +Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting. + +## Main commands to reach for first + +- `devices` +- `ensure-simulator` +- `open` +- `install` or `reinstall` +- `close` +- `session list` + +## Most common mistake to avoid + +Do not start acting before you have pinned the correct target and opened an `app` session. In mixed-device environments, always pass `--device`, `--udid`, or `--serial`. + +## Canonical loop + +```bash +agent-device ensure-simulator --platform ios --device "iPhone 17 Pro" --boot +agent-device open MyApp --platform ios --device "iPhone 17 Pro" --relaunch +agent-device snapshot -i +agent-device close +``` + +## Choose the right starting point + +- iOS local QA: prefer simulators unless the task explicitly requires physical hardware. +- iOS in mixed simulator and device environments: run `ensure-simulator` first, then keep using `--device` or `--udid`. +- TV targets: use `--target tv` together with `--platform` when the task is for tvOS or Android TV rather than phone or tablet surfaces. +- Android binary flow: use `install` or `reinstall` for `.apk` or `.aab`, then open by installed package name. +- Android React Native plus Metro flow: `reinstall ` first, then `open --remote-config --relaunch`. +- macOS desktop app flow: use `open --platform macos`. Only load [macos-desktop.md](macos-desktop.md) if a desktop surface or macOS-specific behavior matters. + +TV example: + +```bash +agent-device open MyTvApp --platform ios --target tv +agent-device open com.example.androidtv --platform android --target tv +``` + +## Session rules + +- Use `--session ` when you need a named session: + +```bash +agent-device --session auth open Settings --platform ios +agent-device --session auth snapshot -i +``` + +- Use `open ` before interactions. +- Use `close` when done. Add `--shutdown` when you want simulators or emulators torn down with the session. +- Use semantic session names when you need multiple concurrent runs. +- Use `--save-script=` on `close` when you want to keep a replay script. +- For dev loops where state can linger, prefer `open --relaunch`. +- In iOS sessions, use `open ` for the app itself. Use `open ` for deep links, and `open ` when you need to launch the app and deep link in one step. +- On iOS, `appstate` is session-scoped and requires the matching active session on the target device. + +## After a session is established + +Once you have opened the correct session on the correct target, default to the conservative rule: keep the session binding on follow-up commands, and stop repeating device-routing flags unless you are intentionally retargeting. + +- Prefer `--session ` on follow-up commands, or use sandboxed `AGENT_DEVICE_SESSION`. +- Do not keep repeating `--platform`, `--target`, `--device`, `--udid`, `--serial`, or similar target-selection flags on normal follow-up commands. +- Only omit follow-up session flags when the environment explicitly guarantees isolation. + +Good shared-host pattern: + +```bash +agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro" +agent-device --session auth snapshot -i +agent-device --session auth press @e3 +agent-device --session auth close +``` + +Bad shared-host pattern: + +```bash +agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro" +agent-device --session auth snapshot -i --platform ios --device "iPhone 17 Pro" +``` + +Use target-selection flags again only when you are choosing the target before opening a session, or when you explicitly mean to retarget. + +## Session-bound automation + +Use this when an orchestrator must keep plain CLI calls on one session and device. + +```bash +export AGENT_DEVICE_SESSION=qa-ios +export AGENT_DEVICE_PLATFORM=ios +export AGENT_DEVICE_SESSION_LOCK=strip + +agent-device open MyApp --relaunch +agent-device snapshot -i +agent-device close +``` + +- `AGENT_DEVICE_SESSION` plus `AGENT_DEVICE_PLATFORM` provides the default binding. +- `--session-lock reject|strip` controls whether conflicting per-call routing flags fail or are ignored. +- Conflicts include explicit retargeting flags such as `--platform`, `--target`, `--device`, `--udid`, `--serial`, `--ios-simulator-device-set`, and `--android-device-allowlist`. +- Lock policy applies to nested `batch` steps too. +- Compatibility aliases remain supported: `--session-locked`, `--session-lock-conflicts`, `AGENT_DEVICE_SESSION_LOCKED`, and `AGENT_DEVICE_SESSION_LOCK_CONFLICTS`. + +Android emulator variant: + +```bash +export AGENT_DEVICE_SESSION=qa-android +export AGENT_DEVICE_PLATFORM=android + +agent-device reinstall MyApp /path/to/app-debug.apk --serial emulator-5554 +agent-device --session-lock reject open com.example.myapp --relaunch +agent-device snapshot -i +agent-device close --shutdown +``` + +## Scoped discovery + +Use scoped discovery when one run must not see host-global device lists. + +```bash +agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators +agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234 +``` + +- Scope is applied before `--device`, `--udid`, and `--serial`. +- Out-of-scope selectors fail with `DEVICE_NOT_FOUND`. +- With iOS simulator-set scope enabled, iOS physical devices are not enumerated. +- If the scoped iOS simulator set is empty, the error should point at the set path and suggest creating a simulator in that set. +- Environment equivalents: + - `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` + - `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST` + +## Session inspection and replay + +```bash +agent-device session list +agent-device replay ./session.ad --session auth +agent-device replay -u ./session.ad --session auth +``` + +- iOS session entries include `device_udid` and `ios_simulator_device_set`. Use them to confirm routing in concurrent runs. +- Prefer selector-based actions and assertions in saved replay scripts. +- Tenant isolation namespaces sessions as `:` during tenant-scoped runs. + +## When to leave this file + +- Once the correct target and session are pinned, move to [exploration.md](exploration.md). +- If opening, startup, permissions, or logs become the blocker, switch to [debugging.md](debugging.md). + +## Install and open examples + +```bash +agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554 +agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch +``` + +Do not use `open --relaunch` on Android. + +## Security and trust notes + +- Treat signing, provisioning, and daemon auth values as host secrets. Do not paste them into shared logs or commit them to source control. +- Prefer Xcode Automatic Signing over manual overrides when a physical iOS device is involved. +- Keep persistent host-specific defaults in environment variables rather than checked-in project config. diff --git a/skills/agent-device/references/coordinate-system.md b/skills/agent-device/references/coordinate-system.md index 8547b2c85..03b8f2ef0 100644 --- a/skills/agent-device/references/coordinate-system.md +++ b/skills/agent-device/references/coordinate-system.md @@ -1,8 +1,28 @@ # Coordinate System -All coordinate-based actions use device screen coordinates: +## When to open this file -- Origin: top-left of the device screen -- Units: device points for iOS, pixels for Android +Open this file only when you must use raw coordinates instead of selectors or `@ref` targeting. -Use screenshots to reason about coordinates. +## Main commands to reach for first + +- `screenshot` +- coordinate-based `click` or `swipe` + +## Most common mistake to avoid + +Do not assume coordinates mean the same thing across platforms or runs. Prefer selectors and refs first. + +## Canonical loop + +```bash +agent-device screenshot /tmp/current-screen.png +agent-device click 120 240 +``` + +## Rules + +- Origin is the top-left of the device screen. +- iOS uses device points. +- Android uses pixels. +- Use screenshots to reason about coordinates before acting. diff --git a/skills/agent-device/references/debugging.md b/skills/agent-device/references/debugging.md new file mode 100644 index 000000000..c2ead081e --- /dev/null +++ b/skills/agent-device/references/debugging.md @@ -0,0 +1,115 @@ +# Debugging + +## When to open this file + +Open this file when the task turns into failure triage, logs, network inspection, permission prompts, setup trouble, or unstable session behavior. + +## Main commands to reach for first + +- `logs clear --restart` +- `network dump` +- `logs path` +- `logs doctor` +- `alert wait` +- `alert accept` or `alert dismiss` + +## Most common mistake to avoid + +Do not leave logging on for normal flows or dump full log files into context. Keep debug windows short and inspect logs with `grep` or `tail`. + +## Canonical loop + +```bash +agent-device open MyApp --platform ios +agent-device logs clear --restart +agent-device network dump 25 +agent-device logs path +agent-device close +``` + +## Log and network flow + +Logging is off by default. Enable it only when you need a debugging window. + +- Default app logs live under `~/.agent-device/sessions//app.log`. +- `logs clear --restart` is the fastest clean repro loop. +- `network dump [limit] [summary|headers|body|all]` parses recent HTTP(s) entries from the same session app log. +- `logs doctor` checks backend and runtime readiness for the current session and device. +- `logs mark "before tap"` inserts a timestamped marker into the app log. +- Session app logs can contain runtime data, headers, or payload fragments. Review them before sharing. +- `logs start` requires an active app session and appends to `app.log`. +- `logs stop` stops streaming. `close` also stops logging. +- `logs clear` truncates `app.log` and removes rotated `app.log.N` files, and requires logging to be stopped first. +- `logs path` returns the log path plus metadata about the active backend and file state. +- `network log` is an alias for `network dump`. + +Operational limits: + +- `app.log` rotates to `app.log.1` after 5 MB by default. +- `network dump` scans the last 4000 app-log lines, returns up to 200 entries, and truncates header or payload fields at 2048 characters. +- Retention knobs: + - `AGENT_DEVICE_APP_LOG_MAX_BYTES` + - `AGENT_DEVICE_APP_LOG_MAX_FILES` +- Redaction hook: + - `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS` + +Useful shell follow-up after `logs path`: + +```bash +grep -n -E "Error|Exception|Fatal|crash" +tail -50 +``` + +## Alerts and permissions + +Use `alert` for iOS simulator permission dialogs instead of tapping coordinates. + +```bash +agent-device alert wait 5000 +agent-device alert accept +``` + +- `alert` is only supported on iOS simulators. +- `alert accept` and `alert dismiss` retry internally for a short window, so you usually do not need manual sleeps. +- iOS 16+ "Allow Paste" prompts are suppressed under XCUITest. Use `xcrun simctl pbcopy booted` when you need to seed simulator clipboard content directly. + +## Setup problems worth recognizing early + +- iOS snapshots do not require macOS Accessibility permissions. +- iOS physical-device XCTest setup does require valid signing and provisioning. +- If physical-device runner setup fails, prefer Xcode Automatic Signing first. +- Optional overrides are: + - `AGENT_DEVICE_IOS_TEAM_ID` + - `AGENT_DEVICE_IOS_SIGNING_IDENTITY` + - `AGENT_DEVICE_IOS_PROVISIONING_PROFILE` + - `AGENT_DEVICE_IOS_BUNDLE_ID` +- If daemon startup is timing out during setup, increase `AGENT_DEVICE_DAEMON_TIMEOUT_MS`. +- If daemon startup fails with stale metadata hints, clean `~/.agent-device/daemon.json` and `~/.agent-device/daemon.lock`, then retry. +- Free Apple Developer personal-team accounts may reject generic bundle IDs. Use a unique reverse-DNS value for `AGENT_DEVICE_IOS_BUNDLE_ID` when that happens. + +## Common failure patterns + +- `snapshot` returns 0 nodes: the app may no longer be foregrounded or the UI is not stable yet. Re-open the app or retry when state settles. +- Logs are empty: confirm you opened an app session before `logs clear --restart`. +- Android logs look stale after relaunch: retry the repro window after the process rebinds. +- Permission prompts block the flow: wait for the alert and handle it explicitly. +- If snapshots keep returning 0 nodes on an iOS simulator, restart Simulator and re-open the app. +- If a macOS snapshot looks incomplete, compare with `snapshot --raw --platform macos` to separate collector filtering from missing AX content. + +## Crash triage fast path + +Always start from the session app log, then branch by platform. + +```bash +agent-device logs path +grep -n -E "SIGABRT|SIGSEGV|EXC_|fatal|exception|terminated|killed|jetsam|memorystatus|FATAL EXCEPTION|Abort message" +``` + +- iOS: if the log suggests `ReportCrash`, `SIGABRT`, or `EXC_*`, inspect `~/Library/Logs/DiagnosticReports`. +- Android: if the app log is not enough, use `adb logcat` for `FATAL EXCEPTION`, `Abort message`, or `signal` lines around process death. +- If no crash signature appears in app logs, stop collecting broad logs and switch to the platform-native crash source. + +## When to leave this file + +- Return to [exploration.md](exploration.md) once the app is stable again. +- Load [verification.md](verification.md) if you need evidence artifacts after reproducing the issue. diff --git a/skills/agent-device/references/exploration.md b/skills/agent-device/references/exploration.md new file mode 100644 index 000000000..fe6b3e503 --- /dev/null +++ b/skills/agent-device/references/exploration.md @@ -0,0 +1,192 @@ +# Exploration + +## When to open this file + +Open this file when the app or screen is already running and you need to discover the UI, choose targets, read state, wait for conditions, or perform normal interactions. + +## Main commands to reach for first + +- `snapshot` +- `snapshot -i` +- `press` +- `fill` +- `get` +- `is` +- `wait` +- `find` + +## Most common mistake to avoid + +Do not treat `@ref` values as durable after navigation or dynamic updates. Re-snapshot after the UI changes, and switch to selectors when the flow must stay stable. + +## Canonical loop + +```bash +agent-device open Settings --platform ios +agent-device snapshot -i +agent-device press @e3 +agent-device wait visible 'label="Privacy & Security"' 3000 +agent-device get text 'label="Privacy & Security"' +agent-device close +``` + +## Snapshot choices + +- Use plain `snapshot` when you only need to verify whether visible text or structure is on screen. +- Use `snapshot -i` when you need refs such as `@e3` for interactive exploration. +- Use `snapshot -i -s "Camera"` or `snapshot -i -s @e3` when you want a smaller, scoped result. + +Example: + +```bash +agent-device snapshot -i +``` + +Sample output: + +```text +Page: com.apple.Preferences +App: com.apple.Preferences + +@e1 [ioscontentgroup] + @e2 [button] "Camera" + @e3 [button] "Privacy & Security" +``` + +## Refs vs selectors + +- Use refs for discovery, debugging, and short local loops. +- Use selectors for deterministic scripts, assertions, and replay-friendly actions. +- Prefer selector or `@ref` targeting over raw coordinates. +- For tap interactions, `press` is canonical and `click` is an equivalent alias. + +Examples: + +```bash +agent-device press @e2 +agent-device fill @e5 "test" +agent-device press 'id="camera_row" || label="Camera" role=button' +agent-device is visible 'id="camera_settings_anchor"' +``` + +## Text entry rules + +- Use `fill` to replace text in an editable field. +- Use `type` to append text to the current insertion point. + +## Query and sync rules + +- Use `get` to read text, attrs, or state from a known target. +- Use `is` for assertions. +- Use `wait` when the UI needs time to settle after a mutation. +- Use `find "" click --json` when you need search-driven targeting plus matched-target metadata. +- If you are forced onto raw coordinates, open [coordinate-system.md](coordinate-system.md) first. + +Example: + +```bash +agent-device find "Increment" click --json +``` + +Returned metadata comes from the matched snapshot node and can be used for observability or replay maintenance. + +## QA from acceptance criteria + +Use this loop when the task starts from acceptance criteria and you need to turn them into concrete checks. + +Preferred mapping: + +- visibility or presence claim: `is visible` or plain `snapshot` +- exact text, label, or value claim: `get text` +- post-action state change: act, then `wait`, then `is` or `get` +- nearby structural UI change: `diff snapshot` +- proof artifact for the final result: `screenshot` or `record` + +Anti-hallucination rules: + +- Do not invent app names, device ids, session names, refs, selectors, or package names. +- Discover them first with `devices`, `open`, `snapshot -i`, `find`, or `session list`. +- If refs drift after navigation, re-snapshot or switch to selectors instead of guessing. + +Canonical QA loop: + +```bash +agent-device open MyApp --platform ios +agent-device snapshot -i +agent-device press @e3 +agent-device wait visible 'label="Success"' 3000 +agent-device is visible 'label="Success"' +agent-device screenshot /tmp/qa-proof.png +agent-device close +``` + +## Accessibility audit + +Use this pattern when you need to find UI that is visible to a user but missing from the accessibility tree. + +Audit loop: + +1. Capture a `screenshot` to see what is visually rendered. +2. Capture a `snapshot` or `snapshot -i` to see what the accessibility tree exposes. +3. Compare the two: + - visible in screenshot and present in snapshot: exposed to accessibility + - visible in screenshot and missing from snapshot: likely accessibility gap +4. If you suspect the node exists in AX but is filtered from interactive output, retry with `snapshot --raw`. + +Example: + +```bash +agent-device screenshot /tmp/accessibility-screen.png +agent-device snapshot -i +``` + +Use `screenshot` as the visual source of truth and `snapshot` as the accessibility source of truth for this audit. + +## Batch only when the sequence is already known + +Use `batch` when a short command sequence is already planned and belongs to one logical screen flow. + +```bash +agent-device batch --session sim --platform ios --steps-file /tmp/batch-steps.json --json +``` + +- Keep batch size moderate, roughly 5 to 20 steps. +- Add `wait` or `is exists` guards after mutating steps. +- Do not use `batch` for highly dynamic flows that need replanning after each step. + +Step payload contract: + +```json +[ + { "command": "open", "positionals": ["Settings"], "flags": { "platform": "ios" } }, + { "command": "wait", "positionals": ["label=\"Privacy & Security\"", "3000"], "flags": {} }, + { "command": "click", "positionals": ["label=\"Privacy & Security\""], "flags": {} }, + { "command": "get", "positionals": ["text", "label=\"Tracking\""], "flags": {} } +] +``` + +- `positionals` is optional and defaults to `[]`. +- `flags` is optional and defaults to `{}`. +- Nested `batch` and `replay` are rejected. +- Supported error mode is stop-on-first-error. + +Response handling: + +- Success returns fields such as `total`, `executed`, `totalDurationMs`, and `results[]`. +- Failed runs include `details.step`, `details.command`, `details.executed`, and `details.partialResults`. +- Replan from the first failing step instead of rerunning the whole flow blindly. + +Common batch error categories: + +- `INVALID_ARGS`: fix the payload shape and retry. +- `SESSION_NOT_FOUND`: open or select the correct session, then retry. +- `UNSUPPORTED_OPERATION`: switch to a supported command or surface. +- `AMBIGUOUS_MATCH`: refine the selector or locator, then retry the failed step. +- `COMMAND_FAILED`: add sync guards and retry from the failing step. + +## Stop conditions + +- If refs drift after transitions, switch to selectors. +- If a desktop surface or context menu is involved on macOS, load [macos-desktop.md](macos-desktop.md). +- If logs, network, alerts, or setup failures become the blocker, switch to [debugging.md](debugging.md). +- If the flow is stable and you need proof or replay maintenance, switch to [verification.md](verification.md). diff --git a/skills/agent-device/references/logs-and-debug.md b/skills/agent-device/references/logs-and-debug.md deleted file mode 100644 index df048ad0b..000000000 --- a/skills/agent-device/references/logs-and-debug.md +++ /dev/null @@ -1,114 +0,0 @@ -# Logs (Token-Efficient Debugging) - -Logging is off by default in normal flows. Enable it on demand for debugging windows. App output is written to a session-scoped file so agents can grep it instead of loading full logs into context. -`network dump` parses recent HTTP(s) entries from this same session app log file. - -## Data Handling - -- Default app logs are stored under `~/.agent-device/sessions//app.log`. -- Replay scripts saved with `--save-script` are written to the explicit path you provide. -- Log files may contain sensitive runtime data; review before sharing and clean up when finished. -- Use `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS` to redact sensitive patterns at write time when needed. - -## Retention and Cleanup - -- Keep logging scoped to active debug windows (`logs clear --restart` before repro, `logs stop` after repro). -- Prefer bounded inspection (`grep -n`, `tail -50`) instead of reading full logs into context. -- Clear session logs when finished: - `agent-device logs clear` -- Close session to stop background logging state: - `agent-device close` - -## Quick Flow - -```bash -agent-device open MyApp --platform ios # or --platform android -agent-device logs clear --restart # Preferred: stop stream, clear logs, and start streaming again -agent-device network dump 25 # Parse latest HTTP(s) requests (method/url/status) from app.log -agent-device logs path # Print path, e.g. ~/.agent-device/sessions/default/app.log -agent-device logs doctor # Check tool/runtime readiness for current session/device -agent-device logs mark "before tap" # Insert a timeline marker into app.log -# ... run flows; on failure, grep the path (see below) -agent-device logs stop # Stop streaming (optional; close also stops) -``` - -Precondition: `logs clear --restart` requires an active app session (`open ` first). - -## Command Notes - -- `logs path`: returns log file path and metadata (`active`, `state`, `backend`, size, timestamps). -- `logs start`: starts streaming; requires an active app session (`open` first). Supported on iOS simulator, iOS device, Android, and macOS app sessions. -- `logs stop`: stops streaming. Session `close` also stops logging. -- `logs clear`: truncates `app.log` and removes rotated `app.log.N` files. Requires logging to be stopped first. -- `logs clear --restart`: convenience reset for repro loops (stop stream, clear files, restart stream). -- `logs doctor`: reports backend/tool checks and readiness notes for troubleshooting. -- `logs mark`: writes a timestamped marker line to the session log. -- `network dump [limit] [summary|headers|body|all]`: parses recent HTTP(s) lines from the session app log and returns request summaries. -- `network log ...`: alias for `network dump`. -- On macOS, logs/network are still app-scoped. They rely on Unified Logging output associated with the active session app rather than packet capture. - -## Behavior and Limits - -- `logs start` appends to `app.log` and rotates to `app.log.1` when `app.log` exceeds 5 MB. -- `network dump` scans the last 4000 app-log lines, returns up to 200 entries, and truncates payload/header fields at 2048 characters. -- Android log streaming automatically rebinds to the app PID after process restarts. -- iOS log capture relies on Unified Logging signals (for example `os_log`); plain stdout/stderr output may be limited depending on app/runtime. -- Retention knobs: - - `AGENT_DEVICE_APP_LOG_MAX_BYTES` - - `AGENT_DEVICE_APP_LOG_MAX_FILES` -- Optional write-time redaction patterns: - - `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS` (comma-separated regex) - -## Grep Patterns - -After getting the path from `logs path`, run `grep` (or `grep -E`) so only matching lines enter context. - -```bash -# Get path first, then grep it; -n adds line numbers -grep -n "Error\|Exception\|Fatal" -grep -n -E "Error|Exception|Fatal|crash" - -# Bounded context: last N lines only -tail -50 -``` - -- Use `-n` for line numbers. -- Use `-E` for extended regex so `|` in the pattern does not need escaping. -- Prefer targeted patterns (e.g. `Error`, `Exception`, or app-specific tags) over reading the full file. - -## Crash Triage Fast Path - -Always start from the session app log, then branch by platform. - -```bash -agent-device logs path -grep -n -E "SIGABRT|SIGSEGV|EXC_|fatal|exception|terminated|killed|jetsam|memorystatus|FATAL EXCEPTION|Abort message" -nl -ba | sed -n ',p' -``` - -### iOS - -```bash -# If log shows ReportCrash / SIGABRT / EXC_*, inspect simulator DiagnosticReports: -ls -lt ~/Library/Logs/DiagnosticReports | grep -E "|" | head -``` - -- `SIGABRT`: app/runtime abort; inspect `.ips` triggered thread and top frames. -- `SIGKILL` + jetsam/memorystatus markers: memory-pressure kill. -- `EXC_BAD_ACCESS`/`SIGSEGV`: native memory access issue. - -### Android - -```bash -# Capture fatal crash lines around app process death: -adb -s logcat -d | grep -n -E "FATAL EXCEPTION|Process: |Abort message|signal [0-9]+ \\(SIG" -``` - -- `FATAL EXCEPTION` with Java stack: uncaught Java/Kotlin exception. -- `signal 6 (SIGABRT)` or `signal 11 (SIGSEGV)` with tombstone refs: native crash path (NDK/JNI/runtime). -- `Low memory killer` / `Killing ` entries: OS memory-pressure/process reclaim. - -## Stop Conditions - -- If no crash signature appears in app log, switch to platform-native crash sources (`.ips` on iOS, logcat/tombstone flow on Android). -- If signatures are present and root cause class is identified (abort, native fault, memory pressure), stop collecting broad logs and focus on reproducing the specific path. diff --git a/skills/agent-device/references/macos-desktop.md b/skills/agent-device/references/macos-desktop.md index 67f7ab605..d86051590 100644 --- a/skills/agent-device/references/macos-desktop.md +++ b/skills/agent-device/references/macos-desktop.md @@ -1,129 +1,87 @@ -# macOS Desktop Automation +# macOS Desktop -Use this reference for host Mac apps such as Finder, TextEdit, System Settings, Preview, or browser apps running as normal desktop windows. +## When to open this file -## Start here +Open this file only when `--platform macos` is involved or the task needs `frontmost-app`, `desktop`, or `menubar` surfaces. -- Use `open --platform macos` when you need to act inside one app. -- Use `open --platform macos --surface frontmost-app|desktop|menubar` when you need to inspect desktop-global UI first. -- Use `app` sessions for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record`. -- Use `frontmost-app`, `desktop`, and `menubar` mainly for `snapshot`, `get`, `is`, and `wait`. -- Prefer `@ref` or selectors. Avoid raw coordinates unless there is no better target. +## Main commands to reach for first -## Mental model +- `open --platform macos` +- `open --platform macos --surface frontmost-app|desktop|menubar` +- `snapshot -i` +- `get` +- `is` +- `click --button secondary` -- `snapshot -i` should describe UI visible to a human. -- Context menus are not ambient UI. Open them explicitly with `click --button secondary`, then re-snapshot. -- Prefer refs for exploration and selectors for deterministic replay/assertions. -- If you inspect with `desktop` or `menubar` and then need to act on one app, switch to a normal `app` session. +## Most common mistake to avoid -## Canonical app flow +Do not treat every macOS surface the same. Use the normal `app` surface when you want to act inside one app. Use `frontmost-app`, `desktop`, or `menubar` mainly to inspect what is visible before switching back to `app` for most interactions. + +## Canonical loop ```bash -agent-device open Finder --platform macos -agent-device snapshot -i -agent-device click @e66 --button secondary --platform macos +agent-device open TextEdit --platform macos agent-device snapshot -i +agent-device fill @e3 "desktop smoke test" agent-device close ``` -## Canonical desktop-global flow +## Surface rules -```bash -agent-device open --platform macos --surface desktop -agent-device snapshot -i -agent-device get attrs @e4 -agent-device is visible 'role="window" label="Finder"' -agent-device close -``` - -Surface variants: - -```bash -agent-device open --platform macos --surface frontmost-app -agent-device open --platform macos --surface desktop -agent-device open --platform macos --surface menubar -``` - -- `app`: default session surface; use this for most real interaction work. +- `app`: default surface and the normal choice for `click`, `fill`, `press`, `scroll`, `screenshot`, and `record`. - `frontmost-app`: inspect the currently focused app without naming it first. - `desktop`: inspect visible desktop windows across apps. - `menubar`: inspect the active app menu bar and system menu extras. -Use `frontmost-app`, `desktop`, and `menubar` for read/inspect flows first. If the next step is a click/fill/press/scroll in one app, switch back to `app`. +Use inspect-first surfaces to understand desktop-global UI, then switch back to `app` when you need to act in one app. -## What to expect from snapshots +## Snapshot expectations -- `app` snapshots should focus on the chosen app window. -- `desktop` snapshots can contain multiple windows from multiple apps. -- `menubar` snapshots can contain both app-menu items and system menu extras. -- File rows, sidebar items, toolbar controls, search fields, and visible context menus should appear. -- Finder and other native apps may expose duplicate-looking structures such as row wrapper nodes, `cell` nodes, and child `text` or `text-field` nodes. -- Treat those as distinct AX nodes unless you have a stronger selector anchor. +- `snapshot -i` should describe UI visible to a human. +- `desktop` snapshots can include multiple windows from multiple apps. +- `menubar` snapshots can include both app-menu items and system menu extras. +- Finder-style rows, sidebar items, toolbar controls, search fields, and opened context menus should appear when visible. +- Finder and other native apps may expose duplicate-looking row, cell, and child text nodes. Treat them as distinct AX nodes unless you have a stronger selector anchor. ## Context menus -Use secondary click when the app exposes actions only through the contextual menu. +Context menus are not ambient UI. Open them explicitly, then re-snapshot. ```bash agent-device click @e66 --button secondary --platform macos agent-device snapshot -i ``` -Expected pattern: +Expected loop: 1. Snapshot visible content. -2. Secondary-click the target row/item. +2. Secondary-click the target item. 3. Snapshot again. -4. Interact with newly visible `menu-item` nodes. +4. Interact with the new `menu-item` nodes. -Do not expect context-menu items to appear before the menu is opened. +## Targeting rules -Do not use `longpress` as a substitute for right-click on macOS. +- Prefer selectors or `@ref` values over raw coordinates. +- On macOS, window position can vary across runs, so coordinate-only flows are fragile. +- If the task only needs shared exploration rules, return to [exploration.md](exploration.md). -## Finder-specific guidance +Selector guidance: -- `snapshot -i` should still expose visible folder rows even when nothing is selected. -- Unselected folder contents should still be visible in `snapshot -i` through list/table rows. -- A file row may expose multiple nodes with the same label, including a row container, name cell, and child text/text-field. -- For opening a context menu, prefer the outer visible row/cell ref over a nested text child if both exist. -- After secondary click, expect actions such as `Rename`, `Quick Look`, `Copy`, `Compress`, and tag-related items in the next snapshot. +- Good selectors usually anchor on stable labels or app-owned identifiers such as `label="Downloads"` or `role=menu-item label="Rename"`. +- Avoid relying on framework-generated `_NS:*` identifiers as stable selectors. -## Raw snapshots +Use `snapshot --raw --platform macos` only when debugging AX structure or collector filtering. Do not make raw snapshots the default agent loop. -Use `snapshot --raw` only when debugging AX structure or collector issues. +Things not to rely on: -```bash -agent-device snapshot --raw --platform macos -``` +- Mobile-only helpers such as `install`, `reinstall`, or `push`. +- Desktop-global click or fill parity from `desktop` or `menubar` sessions. +- Raw coordinate assumptions across runs. -- Raw output is larger and less token-efficient. -- It is useful for verifying whether missing UI is absent from the AX tree or only filtered from interactive output. -- Do not use raw output as the default agent loop when `snapshot -i` already shows the visible window content you need. +Troubleshooting: -## Selector guidance - -Good macOS selectors usually anchor on one of: - -- `label="Downloads"` -- `label="failed-step.json"` -- `role=button label="Search"` -- `role=menu-item label="Rename"` -- `role=window label="Notes"` - -Prefer exact labels when the desktop UI is stable. Use `id=...` when the AX identifier is clearly app-owned and not a framework-generated `_NS:*` value. - -## Things not to rely on - -- Mobile-only helpers like `install`, `reinstall`, and `push` -- Desktop-global click/fill parity from `desktop` or `menubar` sessions -- Raw coordinate assumptions across runs; macOS windows can move -- Framework-generated `_NS:*` identifiers as stable selectors - -## Troubleshooting - -- If visible window content is missing from `snapshot -i`, re-snapshot once after the UI settles. -- If `desktop` is too broad, retry with `frontmost-app` to narrow the inspect surface. -- If `menubar` is missing the menu you expect, make the target app frontmost first, then re-open the `menubar` surface and snapshot again. -- If the wrong menu opened or no menu appeared, retry secondary-clicking the row/cell wrapper instead of the nested text node. -- If the app has multiple windows, ensure the correct one is frontmost before relying on refs. +- If visible content is missing from `snapshot -i`, re-snapshot after the UI settles. +- If `desktop` is too broad, retry with `frontmost-app`. +- If `menubar` is missing the expected menu, make the app frontmost first and retry. +- If the wrong menu opened, retry secondary-clicking the row or cell wrapper rather than the nested text node. +- If the app has multiple windows, make the correct window frontmost before relying on refs. diff --git a/skills/agent-device/references/perf-metrics.md b/skills/agent-device/references/perf-metrics.md deleted file mode 100644 index ad1ebc9a3..000000000 --- a/skills/agent-device/references/perf-metrics.md +++ /dev/null @@ -1,53 +0,0 @@ -# Performance Metrics (`perf` / `metrics`) - -Use this reference when you need to measure launch performance in agent workflows. - -## Quick flow - -```bash -agent-device open Settings --platform ios -agent-device perf --json -``` - -Alias: - -```bash -agent-device metrics --json -``` - -## What is measured today - -- Session-scoped `startup` timing only. -- Sampling method: `open-command-roundtrip`. -- Unit: milliseconds (`ms`). -- Source: elapsed wall-clock time around each session `open` command dispatch for the active app target. - -## Output fields to use - -- `metrics.startup.lastDurationMs`: most recent startup sample. -- `metrics.startup.lastMeasuredAt`: ISO timestamp of most recent sample. -- `metrics.startup.sampleCount`: number of retained samples. -- `metrics.startup.samples[]`: recent startup history for the current session. -- `sampling.startup.method`: current sampling method identifier. - -## Platform support (current) - -- iOS simulator: supported for startup sampling. -- iOS physical device: supported for startup sampling. -- Android emulator/device: supported for startup sampling. -- `fps`, `memory`, and `cpu`: currently placeholders (`available: false`). - -## Interpretation guidance - -- Treat startup values as command round-trip timing, not true app first-frame or first-interactive telemetry. -- Compare like-for-like runs: - - same device target - - same app build - - same workflow/session steps -- Use multiple runs and compare trend/median, not one-off samples. - -## Common pitfalls - -- Running `perf` before any `open` in the session yields no startup sample yet. -- Comparing values across different devices/runtimes introduces large noise. -- Interpreting current `startup` as CPU/FPS/memory would be incorrect. diff --git a/skills/agent-device/references/permissions.md b/skills/agent-device/references/permissions.md deleted file mode 100644 index 89e906772..000000000 --- a/skills/agent-device/references/permissions.md +++ /dev/null @@ -1,70 +0,0 @@ -# Permissions and Setup - -## iOS snapshots - -iOS snapshots use XCTest and do not require macOS Accessibility permissions. - -## iOS physical device runner - -For iOS physical devices, XCTest runner setup requires valid signing/provisioning. -Use Automatic Signing in Xcode, or provide optional overrides: - -- `AGENT_DEVICE_IOS_TEAM_ID` -- `AGENT_DEVICE_IOS_SIGNING_IDENTITY` -- `AGENT_DEVICE_IOS_PROVISIONING_PROFILE` -- `AGENT_DEVICE_IOS_BUNDLE_ID` (optional runner bundle-id base override) - -Free Apple Developer (Personal Team) accounts may reject generic bundle IDs as unavailable. -Set `AGENT_DEVICE_IOS_BUNDLE_ID` to a unique reverse-DNS identifier when that happens. - -Security guidance for these overrides: - -- These variables are optional and only needed for physical-device XCTest setup. -- Treat values as sensitive host configuration; do not share in chat logs or commit to source control. -- Do not provide private keys or unrelated secrets; use the minimum values required for signing. -- Prefer Xcode Automatic Signing when possible to reduce manual secret/config handling. -- For autonomous/CI runs, keep these unset by default and require explicit opt-in for physical-device workflows. - -If setup/build takes long, increase: - -- `AGENT_DEVICE_DAEMON_TIMEOUT_MS` (default `90000`, for example `120000`) - -If daemon startup fails with stale metadata hints, clean stale files and retry: - -- `~/.agent-device/daemon.json` -- `~/.agent-device/daemon.lock` - -## iOS permission alerts (simulator only) - -iOS apps trigger system permission dialogs (camera, location, notifications, etc.) on first use. -Use `alert` to handle them without tapping coordinates: - -```bash -agent-device alert wait # block until an alert appears (default 10 s timeout) -agent-device alert accept # accept the frontmost alert -agent-device alert dismiss # dismiss the frontmost alert -agent-device alert get # read alert title/message without acting -``` - -**Timing note:** `alert accept` and `alert dismiss` include a built-in 2 s retry window. -If the alert is present in the UI hierarchy but not yet interactive, the command retries every 300 ms -rather than failing immediately. You do not need to add manual sleeps between triggering the alert -and accepting it. - -**Preferred pattern for clean simulator sessions:** - -```bash -agent-device open MyApp --platform ios -agent-device alert wait 5000 # wait up to 5 s for the permission prompt -agent-device alert accept # accept; retries internally if not yet actionable -``` - -`alert` is only supported on iOS simulators; iOS physical devices are not supported. - -## iOS: "Allow Paste" dialog - -iOS 16+ shows an "Allow Paste" prompt when an app reads the system pasteboard. Under XCUITest (which `agent-device` uses), this prompt is suppressed by the testing runtime. Use `xcrun simctl pbcopy booted` to set clipboard content directly on the simulator instead. - -## Simulator troubleshooting - -- If snapshots return 0 nodes, restart Simulator and re-open the app. diff --git a/skills/agent-device/references/remote-tenancy.md b/skills/agent-device/references/remote-tenancy.md index 0080371a0..36e355a12 100644 --- a/skills/agent-device/references/remote-tenancy.md +++ b/skills/agent-device/references/remote-tenancy.md @@ -1,56 +1,56 @@ -# Remote Tenancy and Lease Admission +# Remote Tenancy -Use this reference for remote daemon HTTP flows that require explicit -tenant/run admission control. +## When to open this file -## Transport prerequisites +Open this file only for remote daemon HTTP flows that require explicit daemon URL setup, authentication, lease allocation, or tenant-scoped command admission. -- Start daemon in HTTP mode (`AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual`). -- Point remote clients at the host with `AGENT_DEVICE_DAEMON_BASE_URL=http(s)://host:port[/base-path]` - or `--daemon-base-url ` so the CLI skips local daemon discovery/startup. -- Use `AGENT_DEVICE_DAEMON_AUTH_TOKEN` / `--daemon-auth-token` when the client should send the - shared daemon token automatically. -- Direct JSON-RPC callers can use a token in params, `Authorization: Bearer `, or - `x-agent-device-token`. -- Prefer an auth hook (`AGENT_DEVICE_HTTP_AUTH_HOOK`) for caller validation and - tenant injection. +## Main commands to reach for first -## Lease lifecycle (JSON-RPC) +- `AGENT_DEVICE_DAEMON_BASE_URL=...` +- `AGENT_DEVICE_DAEMON_AUTH_TOKEN=...` +- `curl ... agent_device.lease.allocate` +- `curl ... agent_device.lease.heartbeat` +- `curl ... agent_device.lease.release` +- `agent-device --tenant ... --session-isolation tenant --run-id ... --lease-id ...` -Use `POST /rpc` with JSON-RPC 2.0 methods: +## Most common mistake to avoid -- `agent_device.lease.allocate` -- `agent_device.lease.heartbeat` -- `agent_device.lease.release` +Do not run a tenant-isolated command without matching `tenant`, `run`, and `lease` scope. Admission checks require all three to line up. -Example allocate: +## Canonical loop ```bash +export AGENT_DEVICE_DAEMON_BASE_URL=http://mac-host.example:4310 +export AGENT_DEVICE_DAEMON_AUTH_TOKEN= + curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \ -H "content-type: application/json" \ -H "Authorization: Bearer " \ -d '{"jsonrpc":"2.0","id":"alloc-1","method":"agent_device.lease.allocate","params":{"tenantId":"acme","runId":"run-123","ttlMs":60000}}' + +agent-device \ + --tenant acme \ + --session-isolation tenant \ + --run-id run-123 \ + --lease-id \ + session list --json ``` -Example heartbeat: +Heartbeat and release example: ```bash curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \ -H "content-type: application/json" \ -H "Authorization: Bearer " \ -d '{"jsonrpc":"2.0","id":"hb-1","method":"agent_device.lease.heartbeat","params":{"leaseId":"","ttlMs":60000}}' -``` -Example release: - -```bash curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \ -H "content-type: application/json" \ -H "Authorization: Bearer " \ -d '{"jsonrpc":"2.0","id":"rel-1","method":"agent_device.lease.release","params":{"leaseId":""}}' ``` -Example session-locked command request: +Session-locked RPC command example: ```bash curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \ @@ -59,11 +59,34 @@ curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \ -d '{"jsonrpc":"2.0","id":"cmd-1","method":"agent_device.command","params":{"session":"qa-ios","command":"snapshot","positionals":[],"meta":{"lockPolicy":"reject","lockPlatform":"ios","tenantId":"acme","runId":"run-123","leaseId":""}}}' ``` -Direct RPC callers can send the same session lock concept as the CLI and typed client through `meta.lockPolicy` and optional `meta.lockPlatform`. +## Transport prerequisites + +- Start the daemon in HTTP mode with `AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual`. +- Point the client at the remote host with `AGENT_DEVICE_DAEMON_BASE_URL=http(s)://host:port[/base-path]`. +- Use `AGENT_DEVICE_DAEMON_AUTH_TOKEN` or `--daemon-auth-token` when the client should send the shared daemon token automatically. +- Direct JSON-RPC callers can authenticate with request params, `Authorization: Bearer `, or `x-agent-device-token`. +- Prefer an auth hook such as `AGENT_DEVICE_HTTP_AUTH_HOOK` when the host needs caller validation or tenant injection. + +## Lease lifecycle + +Use JSON-RPC methods on `POST /rpc`: + +- `agent_device.lease.allocate` +- `agent_device.lease.heartbeat` +- `agent_device.lease.release` + +Keep the lease alive for the duration of the run and release it when the tenant-scoped work is complete. + +Host-level lease knobs: + +- `AGENT_DEVICE_MAX_SIMULATOR_LEASES` +- `AGENT_DEVICE_LEASE_TTL_MS` +- `AGENT_DEVICE_LEASE_MIN_TTL_MS` +- `AGENT_DEVICE_LEASE_MAX_TTL_MS` ## Command admission contract -For tenant-isolated command execution, pass all four flags: +For tenant-isolated command execution, pass all four CLI flags together: ```bash agent-device \ @@ -74,25 +97,11 @@ agent-device \ session list --json ``` -Admission checks require tenant/run/lease scope alignment. +The CLI sends `AGENT_DEVICE_DAEMON_AUTH_TOKEN` in both the JSON-RPC request token field and HTTP auth headers so existing daemon auth paths continue to work. -The CLI sends `AGENT_DEVICE_DAEMON_AUTH_TOKEN` in both the JSON-RPC request token field and HTTP -auth headers so existing daemon auth paths continue to work. +## Failure semantics and trust notes -## Failure semantics - -- Missing tenant/run/lease fields in tenant isolation mode: `INVALID_ARGS` -- Lease not active or wrong scope: `UNAUTHORIZED` -- Method mismatch: JSON-RPC `-32601` (HTTP 404) - -## Operational guidance - -- Keep TTL short and heartbeat only while a run is active. -- Release lease immediately on run completion/error paths. -- For remote debug sessions, inspect logs on the remote host; client-side `--debug` no longer tails - a local daemon log when `AGENT_DEVICE_DAEMON_BASE_URL` is set. -- For bounded hosts, configure: - - `AGENT_DEVICE_MAX_SIMULATOR_LEASES` - - `AGENT_DEVICE_LEASE_TTL_MS` - - `AGENT_DEVICE_LEASE_MIN_TTL_MS` - - `AGENT_DEVICE_LEASE_MAX_TTL_MS` +- Missing tenant, run, or lease fields in tenant-isolation mode should fail as `INVALID_ARGS`. +- Inactive or scope-mismatched leases should fail as `UNAUTHORIZED`. +- Inspect logs on the remote host during remote debugging. Client-side `--debug` does not tail a local daemon log once `AGENT_DEVICE_DAEMON_BASE_URL` is set. +- Treat daemon auth tokens and lease identifiers as sensitive operational data. diff --git a/skills/agent-device/references/session-management.md b/skills/agent-device/references/session-management.md deleted file mode 100644 index 08ea93cde..000000000 --- a/skills/agent-device/references/session-management.md +++ /dev/null @@ -1,101 +0,0 @@ -# Session Management - -## Named sessions - -```bash -agent-device --session auth open Settings --platform ios -agent-device --session auth snapshot -i -``` - -Sessions isolate device context. A device can only be held by one session at a time. - -## Best practices - -- Name sessions semantically. -- Close sessions when done. -- Use separate sessions for parallel work. -- For orchestrated QA runs, prefer a pre-bound session/platform over repeating per-command selectors. -- For remote tenant-scoped automation, run commands with: - `--tenant --session-isolation tenant --run-id --lease-id ` -- In iOS sessions, use `open `. `open ` opens deep links; on devices `http(s)://` opens Safari when no app is active, and custom schemes require an active app in the session. -- In iOS sessions, `open ` opens a deep link. -- On iOS, `appstate` is session-scoped and requires a matching active session on the target device. -- For dev loops where runtime state can persist (for example React Native Fast Refresh), use `open --relaunch` to restart the app process in the same session. -- Use `--save-script [path]` to record replay scripts on `close`; path is a file path and parent directories are created automatically. -- Use `close --shutdown` on iOS simulators or Android emulators to shut down the target as part of session teardown, preventing resource leakage in multi-tenant or CI workloads. -- For ambiguous bare `--save-script` values, prefer `--save-script=workflow.ad` or `./workflow.ad`. -- For deterministic replay scripts, prefer selector-based actions and assertions. -- Use `replay -u` to update selector drift during maintenance. - -## Session-bound automation - -Use this when an external orchestrator must keep every CLI call on the same session/device without a wrapper script. - -```bash -export AGENT_DEVICE_SESSION=qa-ios -export AGENT_DEVICE_PLATFORM=ios -export AGENT_DEVICE_SESSION_LOCK=strip - -agent-device open MyApp --relaunch -agent-device snapshot -i -agent-device press @e3 -agent-device close -``` - -- `AGENT_DEVICE_SESSION` and `AGENT_DEVICE_PLATFORM` provide the default binding when `--session` and `--platform` are omitted. -- A configured `AGENT_DEVICE_SESSION` enables lock policy enforcement by convention. The default mode is `reject`. -- `--session-lock reject|strip` or `AGENT_DEVICE_SESSION_LOCK=reject|strip` controls whether conflicting selectors fail or are ignored. -- The daemon enforces the same lock policy for CLI requests, typed client calls, and direct RPC commands. -- Conflicts include explicit retargeting selectors such as `--platform`, `--target`, `--device`, `--udid`, `--serial`, `--ios-simulator-device-set`, and `--android-device-allowlist`. -- `--session-locked`, `--session-lock-conflicts`, `AGENT_DEVICE_SESSION_LOCKED`, and `AGENT_DEVICE_SESSION_LOCK_CONFLICTS` remain supported as compatibility aliases. -- Lock policy applies to nested `batch` steps too. If a step omits `platform`, it still inherits the parent batch `--platform` instead of being silently replaced by an environment default. - -Android emulator variant: - -```bash -export AGENT_DEVICE_SESSION=qa-android -export AGENT_DEVICE_PLATFORM=android - -agent-device reinstall MyApp /path/to/app-debug.apk --serial emulator-5554 -agent-device --session-lock reject open com.example.myapp --relaunch -agent-device snapshot -i -agent-device close --shutdown -``` - -## Scoped device isolation - -Use scoped discovery when sessions must not see host-global device lists. - -```bash -agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators -agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234 -``` - -- Scope is applied before selectors (`--device`, `--udid`, `--serial`). -- If selector target is outside scope, resolution fails with `DEVICE_NOT_FOUND`. -- If the scoped iOS simulator set is empty (first-run), the error includes the set path and a suggested `xcrun simctl --set create ...` command. -- With iOS simulator-set scope enabled, iOS physical devices are not enumerated. -- Environment equivalents: - - `AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET` (compat: `IOS_SIMULATOR_DEVICE_SET`) - - `AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST` (compat: `ANDROID_DEVICE_ALLOWLIST`) - -## Listing sessions - -```bash -agent-device session list -``` - -iOS session entries include `device_udid` and `ios_simulator_device_set` (null when using the default set). Use these fields to confirm device routing in concurrent multi-session runs without additional `simctl` calls. - -## Replay within sessions - -```bash -agent-device replay ./session.ad --session auth -agent-device replay -u ./session.ad --session auth -``` - -## Tenant isolation note - -When session isolation is set to tenant mode, session namespace is scoped as -`:`. For remote runs, allocate and maintain an active lease -for the same tenant/run scope before executing tenant-isolated commands. diff --git a/skills/agent-device/references/snapshot-refs.md b/skills/agent-device/references/snapshot-refs.md deleted file mode 100644 index f1adf1c4a..000000000 --- a/skills/agent-device/references/snapshot-refs.md +++ /dev/null @@ -1,105 +0,0 @@ -# Snapshot Refs and Selectors - -## Purpose - -Refs are useful for discovery/debugging. For deterministic scripts, use selectors. -For tap interactions, `press` is canonical; `click` is an equivalent alias. -For host Mac desktop apps, pair this reference with [macos-desktop.md](macos-desktop.md) because context menus and native list/table structures need desktop-specific handling. - -## Snapshot - -```bash -agent-device snapshot -i -``` - -Use plain `snapshot` when you need to verify whether text is visible on screen. -Use `snapshot -i` mainly for interactive exploration and choosing refs. - -Output: - -``` -Page: com.apple.Preferences -App: com.apple.Preferences - -@e1 [ioscontentgroup] - @e2 [button] "Camera" - @e3 [button] "Privacy & Security" -``` - -## Using refs (discovery/debug) - -```bash -agent-device press @e2 -agent-device fill @e5 "test" -``` - -On macOS, if actions live in a context menu, use: - -```bash -agent-device click @e5 --button secondary --platform macos -agent-device snapshot -i -``` - -## Using selectors (deterministic) - -```bash -agent-device press 'id="camera_row" || label="Camera" role=button' -agent-device fill 'id="search_input" editable=true' "test" -agent-device is visible 'id="camera_settings_anchor"' -``` - -## Ref lifecycle - -Refs can become invalid when UI changes (navigation, modal, dynamic list updates). -Re-snapshot after transitions if you keep using refs. - -## Scope snapshots - -Use `-s` to scope to labels/identifiers. This reduces size and speeds up results: - -```bash -agent-device snapshot -i -s "Camera" -agent-device snapshot -i -s @e3 -``` - -## Diff snapshots (structural) - -Use `diff snapshot` when you need compact state-change visibility between nearby UI states: - -```bash -agent-device snapshot -i # First snapshot initializes baseline -agent-device press @e5 -agent-device diff snapshot -i # Shows +/− structural lines vs prior snapshot -``` - -Efficient pattern: - -- Initialize once at a stable point. -- Mutate UI (`press`, `fill`, `swipe`). -- Run `diff snapshot` after interactions to confirm expected change shape with bounded output. -- Re-run full/scoped `snapshot` only when you need fresh refs for next step selection. - -## Troubleshooting - -- Ref not found: re-snapshot. -- If `snapshot` returns 0 nodes, foreground app state or accessibility state may have changed. Re-open the app or retry after state is stable. -- On macOS, use `snapshot --raw --platform macos` to distinguish collector filtering from truly missing AX content. - -## Stop Conditions - -- If refs are unstable after UI transitions, switch to selector-based targeting and stop investing in ref-only flows. - -## find click response - -`find "" click --json` returns deterministic matched-target metadata: - -```json -{ "ref": "@e3", "locator": "any", "query": "Increment", "x": 195, "y": 422 } -``` - -Fields come from the matched snapshot node, not the platform runner. Use these for observability and replay quality — they are stable across runs for the same UI state. - -## Replay note - -- Prefer selector-based actions in recorded `.ad` replays. -- Use `agent-device replay -u ` to update selector drift and rewrite replay scripts in place. diff --git a/skills/agent-device/references/verification.md b/skills/agent-device/references/verification.md new file mode 100644 index 000000000..2bb5d5fec --- /dev/null +++ b/skills/agent-device/references/verification.md @@ -0,0 +1,103 @@ +# Verification + +## When to open this file + +Open this file when the task needs evidence, regression checks, replay maintenance, or startup performance measurements after the main interaction flow is already working. + +## Main commands to reach for first + +- `screenshot` +- `diff snapshot` +- `record` +- `replay -u` +- `perf` + +## Most common mistake to avoid + +Do not use verification tools as the first exploration step. First get the app into the correct state with the normal interaction flow, then capture proof or maintain replay assets. + +## Canonical loop + +```bash +agent-device open Settings --platform ios +agent-device snapshot -i +agent-device press @e5 +agent-device diff snapshot -i +agent-device screenshot /tmp/settings-proof.png +agent-device close +``` + +## Structural verification with diff snapshot + +Use `diff snapshot` when you need a compact view of how the UI changed between nearby states. + +```bash +agent-device snapshot -i +agent-device press @e5 +agent-device diff snapshot -i +``` + +- Initialize the baseline at a stable point. +- Perform the mutation. +- Run `diff snapshot` to confirm the expected structural change. +- Re-run full `snapshot` only when you need fresh refs. + +## Visual artifacts + +Use `screenshot` when the proof needs a rendered image instead of a structural tree. + +## Session recording + +Use `record` for debugging, documentation, or shareable verification artifacts. + +```bash +agent-device record start ./recordings/ios.mov +agent-device open App +agent-device snapshot -i +agent-device press @e3 +agent-device close +agent-device record stop +``` + +- `record` supports iOS simulators, iOS devices, and Android. +- On iOS, recording is a wrapper around `simctl` for simulators and the corresponding device capture path for physical devices. +- On Android, recording is a wrapper around `adb`. +- Recording writes a video artifact and a gesture-telemetry sidecar JSON. +- On macOS hosts, touch overlay burn-in is available for supported recordings. +- On non-macOS hosts, recording still succeeds but the video stays raw and `record stop` can return an `overlayWarning`. +- If the agent already knows the interaction sequence and wants a more lifelike, uninterrupted recording, drive the flow with `batch` while recording instead of replanning between each step. + +Example: + +```bash +agent-device record start ./recordings/smoke.mov +agent-device batch --session sim --platform ios --steps-file /tmp/smoke-steps.json --json +agent-device record stop +``` + +- Use this only after exploration has stabilized the flow. +- Keep the batch short and add `wait` or `is exists` guards after mutating steps so the recorded flow still tracks realistic UI timing. + +## Replay maintenance + +Use replay updates when selectors drift but the recorded scenario is still correct. + +```bash +agent-device replay -u ./session.ad +``` + +- Prefer selector-based actions in recorded `.ad` replays. +- Use update mode for maintenance, not as a substitute for fixing a broken interaction strategy. + +## Performance checks + +Use `perf --json` or `metrics --json` when you need startup timing for the active session. + +```bash +agent-device open Settings --platform ios +agent-device perf --json +``` + +- Current startup data is command round-trip timing around `open`. +- It is not true first-frame or first-interactive telemetry. +- `fps`, `memory`, and `cpu` are currently placeholders. diff --git a/skills/agent-device/references/video-recording.md b/skills/agent-device/references/video-recording.md deleted file mode 100644 index b94741ebf..000000000 --- a/skills/agent-device/references/video-recording.md +++ /dev/null @@ -1,51 +0,0 @@ -# Video Recording - -Capture device automation sessions as video for debugging, documentation, or verification - -## iOS Simulator - -Use `agent-device record` commands (wrapper around simctl): - -```bash -# Start recording -agent-device record start ./recordings/ios.mov - -# Perform actions -agent-device open App -agent-device snapshot -i -agent-device click @e3 -agent-device close - -# Stop recording -agent-device record stop -``` - -`record` supports iOS simulators, physical iOS devices, and Android. - -Recording outputs: - -- a video artifact -- a gesture-telemetry sidecar JSON next to the video - -Touch overlay support: - -- macOS host: telemetry can be burned into the video as visible touch overlays -- non-macOS host: recording still succeeds, but the video stays raw and `record stop` returns an `overlayWarning` - -## Android Emulator/Device - -Use `agent-device record` commands (wrapper around adb): - -```bash -# Start recording -agent-device record start ./recordings/android.mp4 - -# Perform actions -agent-device open App -agent-device snapshot -i -agent-device click @e3 -agent-device close - -# Stop recording -agent-device record stop -```