diff --git a/android-cli/SKILL.md b/android-cli/SKILL.md new file mode 100644 index 0000000..4a5b44b --- /dev/null +++ b/android-cli/SKILL.md @@ -0,0 +1,197 @@ +--- +name: android-cli +description: Orchestrates Android development tasks including project creation, deployment, SDK management, and environment diagnostics using the `android` command-line tool. +--- +# Android CLI Specialist + +This skill provides instructions for using the `android` CLI tool. The tool includes various commands for creating projects, running applications, interacting with devices, and managing the CLI environment. + +## SDK management +To manage the installation of Android SDKs and tools, use the `sdk` command. For example: + +- `android sdk install [@]...`: Install specific packages. Multiple packages can be specified, separated by spaces. `` defaults to latest. For example: `android sdk install platforms/android-30@2 platforms/android-34` +- `android sdk update []`: Update a specific package or all packages to the latest version. +- `android sdk remove `: Remove a package from the local SDK. +- `android sdk list --all`: List installed and available SDK packages. + +## Project creation +Create projects from templates using the `create` command. + +For example: `android create empty-activity --name="My App" --output=./my-app` + +## Interacting with devices +For more information on interacting with running devices, see [here](references/interact.md) + +## Running journey tests +For more information on running journeys, see [here](references/journeys.md) + +## Doc searching +The `docs` command searches authoritative, high-quality Android developer documentation in the Android Knowledge Base. +By providing a few keywords, this tool will return high quality articles that contain examples or guidance on how to use Android APIs or libraries. +Use this tool to obtain additional information on how to achieve Android-specific tasks or to know more about Android APIs, surfaces, libraries, or devices. + +Always use this tool to get the most up-to-date information about Android concepts. Typical good use cases are: + - Finding migration guides for APIs. + - Finding examples for APIs. + - Finding up-to-date information about Android APIs. + - Finding best practices for Android concepts. + +## Running APKs +Use the `run` command to run Android apps. + +## Managing emulators + +Manage Android Virtual Devices (AVDs) using the `android emulator` command + +## Capturing screenshots + +Capture an image of the current screen of a connected Android device and output it to a file using the `android screenshot` command. + +## Managing skills + +Manage antigravity agent skills for Android using the `android skills` command. + +## Inspecting UI Layouts + +Use the `android layout` command to inspect the UI layout of an Android application. It returns the layout tree of an Android application in JSON format. When debugging UI errors, this is often a much faster approach than taking a screenshot. + +## Updating the CLI + +Update the Android CLI using the `android update` command. + +# `android help` output + +Usage: android [-hV] [--sdk=PARAM] [COMMAND] + -h, --help Show this help message and exit. + --sdk=PARAM Path to the Android SDK + -V, --version Print version information and exit. +Commands: + create Create a new Android project + describe Analyzes an Android project to generate descriptive metadata. + docs Android documentation commands + emulator Emulator commands + help Shows the help of all commands + info Print environment information (SDK Location, etc.) + init Initializes the environment (eg. skills) for Android CLI. + layout Returns the layout tree of an application + run Deploy an Android Application + screen Commands to view the device + sdk Download and list SDK packages + skills Manage skills + update Update the Android CLI + +create + Usage: android create [-h] [--verbose] [--list] [--minSdk=api] + --name=applicationName [-o=dest-path] [template-name] + Create a new Android project + [template-name] The template name + -h, --help Show this help message and exit. + --minSdk=api The 'minSdk' supported by the application (default + is defined in the template) + --name=applicationName + The name of the application (e.g. 'My Application') + -o, --output=dest-path The destination project directory path (default is + '.') + --verbose Enables verbose output + --list List all available templates + +describe + Usage: android describe [-hV] [--project_dir=PARAM] + Analyzes an Android project to generate descriptive metadata. + This command identifies and outputs the paths to JSON files that detail the + project's structure, including build targets and their corresponding output + artifact locations (e.g., APKs). This information enables other tools and + commands to locate build artifacts efficiently. + -h, --help Show this help message and exit. + --project_dir=PARAM The project directory to describe + -V, --version Print version information and exit. + +docs + Usage: android docs [-h] [COMMAND] + Android documentation commands + -h, --help Show this help message and exit. + Commands: + search Search Android documentation + fetch Fetch Android documentation + +emulator + Usage: android emulator [-h] [COMMAND] + Emulator commands + -h, --help Show this help message and exit. + Commands: + create Creates a virtual device + start Launches the specified virtual device. This command will return when + the emulator is fully started and ready to use. + stop Stops the specified virtual device + list Lists available virtual devices + remove Delete a virtual device + +help + Usage: android help [COMMAND] + Shows the help of all commands + [COMMAND] The command to show help for + +info + Usage: android info + Print environment information (SDK Location, etc.) + The specific field to print the value of. If omitted print all. + +init + Usage: android init + Initializes the environment (eg. skills) for Android CLI. + +layout + Usage: android layout [-dhp] [--device=PARAM] [-o=PARAM] + Returns the layout tree of an application + -d, --diff Returns a flat list of the layout elements that have + changed since the last invocation of ui-dump + --device=PARAM The device serial number + -h, --help Show this help message and exit. + -o, --output=PARAM Writes the layout tree to the specified file or + directory. If omitted, prints the tree to standard + output + -p, --pretty Pretty-prints the returned JSON + +run + Usage: android run [-h] [--debug] [--activity=PARAM] [--device=PARAM] + [--type=PARAM] [--apks=PARAM[,PARAM...]]... + Deploy an Android Application + --activity=PARAM The activity name + --apks=PARAM[,PARAM...] + The paths to the APKs + --debug Run in debug mode + --device=PARAM The device serial number + -h, --help Show this help message and exit. + --type=PARAM The component type (ACTIVITY, SERVICE, etc.) + +screen + Usage: android screen [-h] [COMMAND] + Commands to view the device + -h, --help Show this help message and exit. + Commands: + capture Outputs the device screen to a PNG + resolve Target UI elements visually + +sdk + Usage: android sdk [COMMAND] + Download and list SDK packages + Commands: + install Install SDK packages + update Update one or all packages to the latest version + remove Remove a package from the SDK + list List installed and available SDK packages + +skills + Usage: android skills [COMMAND] + Manage skills + Commands: + add Install a skill + remove Remove a skill + list List available skills + find Find skills by keyword + +update + Usage: android update [--url=PARAM] + Update the Android CLI + --url=PARAM The URL to download the update from + diff --git a/android-cli/base/SKILL.md b/android-cli/base/SKILL.md deleted file mode 100644 index 07ff90e..0000000 --- a/android-cli/base/SKILL.md +++ /dev/null @@ -1,207 +0,0 @@ ---- -name: base -description: Orchestrates Android development tasks -license: Complete terms in LICENSE.txt -metadata: - author: Google LLC - keywords: - - AndroidCLI ---- - -including project creation, deployment, SDK management, and environment -diagnostics using the `android` command-line tool. - -This skill provides instructions for using the `android` CLI tool. The tool includes various commands for creating projects, running applications, interacting with devices, and managing the CLI environment. - -## SDK management - -To manage the installation of Android SDKs and tools, use the `sdk` command. For example: - -- `android sdk install [@]...`: Install specific packages. Multiple packages can be specified, separated by spaces. `` defaults to latest. For example: `android sdk install platforms/android-30@2 platforms/android-34` -- `android sdk update []`: Update a specific package or all packages to the latest version. -- `android sdk remove `: Remove a package from the local SDK. -- `android sdk list --all`: List installed and available SDK packages. - -## Project creation - -Create projects from templates using the `create` command. - -For example: `android create empty-activity --name="My App" --output=./my-app` - -## Interacting with devices - -For more information on interacting with running devices, see [here](https://developer.android.com/agents/skills/android-cli/base/references/interact) - -## Running journey tests - -For more information on running journeys, see [here](https://developer.android.com/agents/skills/android-cli/base/references/journeys) - -## Doc searching - -The `docs` command searches authoritative, high-quality Android developer documentation in the Android Knowledge Base. -By providing a few keywords, this tool will return high quality articles that contain examples or guidance on how to use Android APIs or libraries. -Use this tool to obtain additional information on how to achieve Android-specific tasks or to know more about Android APIs, surfaces, libraries, or devices. - -Always use this tool to get the most up-to-date information about Android concepts. Typical good use cases are: -- Finding migration guides for APIs. -- Finding examples for APIs. -- Finding up-to-date information about Android APIs. -- Finding best practices for Android concepts. - -## Running APKs - -Use the `run` command to run Android apps. - -## Managing emulators - -Manage Android Virtual Devices (AVDs) using the `android emulator` command - -## Capturing screenshots - -Capture an image of the current screen of a connected Android device and output it to a file using the `android screenshot` command. - -## Managing skills - -Manage antigravity agent skills for Android using the `android skills` command. - -## Inspecting UI Layouts - -Use the `android layout` command to inspect the UI layout of an Android application. It returns the layout tree of an Android application in JSON format. When debugging UI errors, this is often a much faster approach than taking a screenshot. - -## Updating the CLI - -Update the Android CLI using the `android update` command. - -# `android help` output - -Usage: android \[-hV\] \[--sdk=PARAM\] \[COMMAND\] --h, --help Show this help message and exit. ---sdk=PARAM Path to the Android SDK --V, --version Print version information and exit. -Commands: -create Create a new Android project -describe Analyzes an Android project to generate descriptive metadata. -docs Android documentation commands -emulator Emulator commands -help Shows the help of all commands -info Print environment information (SDK Location, etc.) -init Initializes the environment (eg. skills) for Android CLI. -layout Returns the layout tree of an application -run Deploy an Android Application -screen Commands to view the device -sdk Download and list SDK packages -skills Manage skills -update Update the Android CLI - -create -Usage: android create \[-h\] \[--verbose\] \[--list\] \[--minSdk=api\] ---name=applicationName \[-o=dest-path\] \[template-name\] -Create a new Android project -\[template-name\] The template name --h, --help Show this help message and exit. ---minSdk=api The 'minSdk' supported by the application (default -is defined in the template) ---name=applicationName -The name of the application (e.g. 'My Application') --o, --output=dest-path The destination project directory path (default is -'.') ---verbose Enables verbose output ---list List all available templates - -describe -Usage: android describe \[-hV\] \[--project_dir=PARAM\] -Analyzes an Android project to generate descriptive metadata. -This command identifies and outputs the paths to JSON files that detail the -project's structure, including build targets and their corresponding output -artifact locations (e.g., APKs). This information enables other tools and -commands to locate build artifacts efficiently. --h, --help Show this help message and exit. ---project_dir=PARAM The project directory to describe --V, --version Print version information and exit. - -docs -Usage: android docs \[-h\] \[COMMAND\] -Android documentation commands --h, --help Show this help message and exit. -Commands: -search Search Android documentation -fetch Fetch Android documentation - -emulator -Usage: android emulator \[-h\] \[COMMAND\] -Emulator commands --h, --help Show this help message and exit. -Commands: -create Creates a virtual device -start Launches the specified virtual device. This command will return when -the emulator is fully started and ready to use. -stop Stops the specified virtual device -list Lists available virtual devices -remove Delete a virtual device - -help -Usage: android help \[COMMAND\] -Shows the help of all commands -\[COMMAND\] The command to show help for - -info -Usage: android info Print environment information (SDK Location, etc.) The specific field to print the value of. If omitted print all. - -init -Usage: android init -Initializes the environment (eg. skills) for Android CLI. - -layout -Usage: android layout \[-dhp\] \[--device=PARAM\] \[-o=PARAM\] -Returns the layout tree of an application --d, --diff Returns a flat list of the layout elements that have -changed since the last invocation of ui-dump ---device=PARAM The device serial number --h, --help Show this help message and exit. --o, --output=PARAM Writes the layout tree to the specified file or -directory. If omitted, prints the tree to standard -output --p, --pretty Pretty-prints the returned JSON - -run -Usage: android run \[-h\] \[--debug\] \[--activity=PARAM\] \[--device=PARAM\] -\[--type=PARAM\] \[--apks=PARAM\[,PARAM...\]\]... -Deploy an Android Application ---activity=PARAM The activity name ---apks=PARAM\[,PARAM...\] -The paths to the APKs ---debug Run in debug mode ---device=PARAM The device serial number --h, --help Show this help message and exit. ---type=PARAM The component type (ACTIVITY, SERVICE, etc.) - -screen -Usage: android screen \[-h\] \[COMMAND\] -Commands to view the device --h, --help Show this help message and exit. -Commands: -capture Outputs the device screen to a PNG -resolve Target UI elements visually - -sdk -Usage: android sdk \[COMMAND\] -Download and list SDK packages -Commands: -install Install SDK packages -update Update one or all packages to the latest version -remove Remove a package from the SDK -list List installed and available SDK packages - -skills -Usage: android skills \[COMMAND\] -Manage skills -Commands: -add Install a skill -remove Remove a skill -list List available skills -find Find skills by keyword - -update -Usage: android update \[--url=PARAM\] -Update the Android CLI ---url=PARAM The URL to download the update from \ No newline at end of file diff --git a/android-cli/references/interact.md b/android-cli/references/interact.md new file mode 100644 index 0000000..099e66b --- /dev/null +++ b/android-cli/references/interact.md @@ -0,0 +1,83 @@ +# Tools +Run `android layout --help` and `android screen --help`. + +## UI Dump +`android layout` returns a flat JSON list of the UI elements on screen. +`android layout --diff` returns a flat JSON list of the UI elements that have changed since the last call to `layout` or `layout --diff` + +Each JSON object represents a UI element in the Android app. The following properties may be present: +- `text` - any literal text the element contains +- `resourceId` - the Android resource id used to refer to the element +- `contentDesc` - a description of a UI element for use by accessibility tools +- `interactions` - the set of user interactions the element supports. May contain one or more of: `checkable`, `clickable`, `focusable`, `scrollable`, `long-clickable`, `password` +- `state` - the set of states the element is in. May contain one or more of `checked`, `focused`, `selected` +- `bounds` - the screen coordinates of the bounding rectangle of the element, in the format `[min X,min Y][max X, max Y]` +- `center` - the screen coordinates of the center of the element, in the format `[x,y]` +- `off-screen` - if true, the element is in the UI hierarchy but not visible; it may require scrolling to view. + +Use `layout` as a primary means of examining an Android app. Use `layout --diff` to focus on changes and to keep your context small. +Example: When entering digits into a calculator, use `layout --diff` to output only the digit readout element. + +`layout` may fail due to the app displaying a WebView or animation; in these cases, use `android screen --annotate` to inspect the app. +This failure will likely resolve after navigating away from the current screen. + +## Screenshot +`android screen capture -o ` saves a PNG of the current device screen to `` + +Use `screen capture` as a secondary means of examining an Android app +Examples: +- Understanding the content of an on-screen image +- Looking at a `WebView` (web content does not always appear in the ui dump) +- Trying to find a UI element by its visual appearance + +**IMPORTANT**: Always *VISUALLY* examine the PNG image returned from `android screen` BEFORE doing anything else. + +## Annotated Screenshot +`android screen capture --annotate -o ` +`android screen resolve --screen --string ` + +The `--annotate` command adds numerical labels and bounding boxes around UI elements. Use this command to locate UI elements that cannot +be located in the `layout` output. + +**IMPORTANT**: When using `android screen --annotate`, always *VISUALLY* examine the resulting PNG file. + +To refer to these labels in input commands, use `screen resolve` to convert labels into coordinates: + +`android screen resolve --screen --string "#3"` returns ` ` + +To save turns, you can combine shell commands: + +`adb shell input $(android screen resolve --screen screen.png --string "tap #34")` + +This command taps on region #34 from `screen.png` + +## Input +Use `adb shell input` for interacting with Android devices. +Refer to the `"interactions"` property of an element for what interactions can be performed on a particular element. + +Interact with UI elements with their `center` coordinate or their `bounds` coordinates: +```json +{ + "key": -248568265, + "class": "android.widget.Button", + "bounds": "[138,9][167,38]", + "center": "[152,23]" +} +``` +To tap on this button, you would execute `adb shell input tap 152 23`. This taps the center. + +```json +{ + "key": 12487234, + "class": "com.example.ui.ScrollableList", + "bounds": "[100,200][400,600]", + "center": "[250,400]" +} +``` +To scroll down on this list, you would execute `adb shell input swipe 250 400 600 500`. This swipes from the center to the bottom over 500ms. + +# Android Interaction Rules +1. Always ensure text input fields have `"focused"` in their `"state"` list before entering text +2. If an element has `"scrollable"` in its `"interactions"` list, try scrolling it when looking for missing UI elements +2. Always scroll slowly when executing scroll inputs. The 5th argument to `adb shell input swipe` controls scroll duration. +3. Content may take time to load; if a `layout` is missing information after you take an action, wait a few seconds, then perform `layout --diff` to see if anything changes. \ No newline at end of file diff --git a/android-cli/references/journeys.md b/android-cli/references/journeys.md new file mode 100644 index 0000000..7ed45ad --- /dev/null +++ b/android-cli/references/journeys.md @@ -0,0 +1,97 @@ +A journey is an XML-specified test of an Android app's behavior. It consists of a list of `` elements. For example: +```xml + + + A sample journey to illustrate the format + + + + Tap the "Home" icon + + + Verify that the app is on its Home screen + + + +``` + +Evaluate a journey by proceeding through the `` list in sequential order. Evaluate each `` block individually. +A journey succeeds if all elements in the `` list succeed. + +A journey is a test case for an app. The journey XML is the source of truth; if the app disagrees with the journey, the app has failed. +Additionally, if the app exits, crashes, or freezes, journey evaluation stops and the journey fails. + +**IMPORTANT** - Execute each step EXACTLY as written, and independently of other steps! If an action says to `"tap the first search result"`, +you MUST find the search results and tap the first one. Do this even if you believe you know the intent behind the action. + +## Taking Actions +Some `` elements specify UI interactions to perform on the running Android app. Perform the interaction and verify that the app does +not crash or behave in an unexpected manner. This is the *only* verification you should perform for an ``. + +If the interaction cannot be performed as specified, the journey fails. +Example: +```Click the red button``` +If you determine a red button is not present in the UI, the journey fails. + +If the text of an `` specifies a list of actions, break it into sub-actions and evaluate them individually: +Example: +```Search for soda and add the first result to the cart``` +This should be evaluated as: +``` +Search for soda +Add the first result to the cart +``` + +If an `` contains something that is not a specification for a UI interaction, alert the user that the journey is malformed and exit +early, specifying the error in question. + +## Verifying Expectations +`` elements that begin with "check" or "verify" specify expectations for the current state of the Android app. Determine the current +state of the app and check if the expectations are met. + +Determine the current state of the app by inspecting the current screen of the device without interacting with it. +Example: +```Check if "Switch 2" is visible on the screen``` +This requires only inspecting the current screen, not scrolling or interacting. If "Switch 2" is not currently visible, the action fails. + +If the expectations are not met, mark the `` as a failure and the journey evaluation ends. A single `` may contain +multiple expectations. +Example: +```Verify that the app is on the Home screen, the Home icon is blue, and the temperature is displayed``` +This `` fails if ANY of the following are false: +- The app is on the Home screen +- There is a Home icon, and it is blue +- A temperature is displayed + +## Handling failure +When running a journey, evaluate it as a test. Failure is acceptable, and often expected. Proper reporting of failures is the priority. + +Keep debugging and troubleshooting to a minimum; assume that tools are showing you the correct output every time. The goal is to determine +if the *current* Android app can correctly handle the *current* steps outlined in the journey. Suggestions for bug fixes, clarification, or +other improvements should be kept to journey evaluation summary at the end. + +## Summarizing +For each `` you evaluated, output JSON describing the results. + +``` +{ + "journey:", The name of the journey + "results:" [ + { + // A string containing the full text of the + "action": "Click the blue button, + // "PASSED" if the instruction was evaluated, "FAILED" if the instruction could not be evaluated, or "SKIPPED" if journey evaluation ended early because an instruction failed + "status": "PASSED", + // A list of the ADB commands executed while evaluating the instruction, + "commands": [ "adb input swipe 490 200 500 500 500", "adb input tap 45 920" ], + // Failure reasons, feedback, or other useful information + "comment": "The journey step doesn't specify that the button requires scrolling to see", + }, + { + "action": "The home screen is shown", + "status": "FAILED", + "comment": "The settings page was shown", + }, + ] +} +```