feat(telemetry): report device_kind next to platform on tool events (#1049)

## Why

Telemetry cannot tell a physical iPhone from a simulator: `platform` is
`ios` for both, and the tool-server's `kind` never reaches the events.
Physical-iOS support shipped in 0.24.0 with no way to count its adoption
except through `ios_device_*` failure stages on `tool:fail`. Physical
Android phones have the same gap.

## What

`tool:invoke` / `tool:complete` / `tool:fail` gain `device_kind:
simulator | emulator | vvd | device | app`, derived from the same device
id as `platform` (single classifier in `telemetry-platform.ts`, which
`debugger:tool_outcome` and `lens:*` also go through for their
platform). `platform: ios` + `device_kind: device` is a physical iPhone.

The kind is emitted only for positively recognised id shapes, so the
metric is not polluted by ids that are not devices:

| id shape | platform | device_kind |
|---|---|---|
| simulator UUID / `remote:` | ios / ios-remote (tvos when the cache is
warm) | simulator |
| physical iPhone UDID | ios | device |
| `emulator-NNNN` | android (android-tv when warm) | emulator |
| USB serial, 6–20 alnum with a digit and a letter | android | device |
| non-loopback `ip:port` | android | device (best effort) |
| loopback `127.0.0.1:port` / `localhost:port` / `::1:port` | android |
emulator |
| anything else non-iOS-shaped (Metro 40-hex handle, `booted`,
`Pixel_7`, IPv6, mDNS) | android | omitted |
| `chromium-cdp-<port>` / `amazon-…` | chromium / vega | app / vvd |
| `avdName`-only boot | android | omitted |

A child sub-tool that names its own device replaces the parent's
platform, kind and provider label wholesale, so an `avdName`-only child
never reports the parent's `simulator` next to its own `android`. A
logical-keyed Metro session strips the kind.

`unknown` (in the registry `DeviceKind`) is rejected by the sanitizer
like `platform: "unknown"`; no registry change.

## Querying

- Physical iPhone adoption: `tool:invoke` where `platform = 'ios' and
device_kind = 'device'`, distinct users over it. `isIosPhysicalUdid` is
a positive match, so invoke is clean.
- Physical Android: count on `tool:complete` (the device answered).
`tool:invoke` / `tool:fail` still carry residual mistakes that happen to
look like a serial (`Pixel7`), and a non-loopback `ip:port` may be a
networked virtual device.

## Docs

- `Telemetry.md` → 1.04, effective date 7 September 2026 (adjust to the
release date on merge). Discloses the new category and, while touching
that section, the external-provider label that #735 added without a
notice line — happy to drop that second bullet if it should ship
separately.
- No docs-site change: `docs/reference/telemetry.mdx` links to
`Telemetry.md` and has no per-property list.

## Notes

- Pre-existing, unchanged: HTTP-edge failures carry no
`tool_invocation_id`; `stop-all-simulator-servers` declares no
capability, so its invoke row has neither field while its fail row has
both. `device_kind` follows `platform` in each site.
- Legacy 40-hex iPhone UDIDs already classify as `android` today; they
now get no kind rather than a wrong one.
- Design went through three adversarial review rounds before
implementation and one on the diff; the fallback-bucket and
child-attribution rules above came out of those.

## Verification

- `packages/telemetry`: 319 tests, `packages/tool-server`: full suite
5670 passed / 1 skipped; `typecheck:tests` both packages; `eslint
--max-warnings 0`; `knip`; prettier.
- New: `test/telemetry-device-kind.test.ts` (the shape table above),
plus cases in `http-tools-meta`, `http-platform-alias`, `sanitize`,
`registry-listener`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BSGpeAwvoCv2JAhbLsuto2


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Telemetry now records a device category—such as simulator, emulator,
physical device, virtual device, or desktop app—alongside the platform.
* Device categories are inferred from device information and may include
a short label for external device providers.
* Device category information is included in tool invocation,
completion, and failure telemetry when available.
* Device identifiers themselves continue to be excluded from telemetry.
* **Documentation**
* Updated the Argent Privacy Notice (Telemetry) to version 1.04,
effective 7 September 2026.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
filip131311
2026-09-07 11:55:57 +02:00
committed by GitHub
parent 1a7e97fb14
commit c573e9c524
13 changed files with 476 additions and 49 deletions
+3 -2
View File
@@ -1,6 +1,6 @@
# Argent Privacy Notice (Telemetry)
Effective date: 31 July 2026 · Version: 1.03
Effective date: 7 September 2026 · Version: 1.04
This notice is a product-specific supplement to the [Software Mansion Privacy Policy](https://swmansion.com/) (the "Policy") and applies to telemetry collected by Argent, a Software Mansion Software Product. Capitalised terms used but not defined here (including Personal Data, Usage Data, Legitimate Interest, EEA and Software Mansion Software Product) have the meaning given to them in the Policy. Where this notice and the Policy differ in respect of Argent telemetry, this notice prevails.
@@ -63,7 +63,8 @@ When you use the Argent Lens design-review flow (previewing and choosing between
- Argent version, Node.js version, operating system, processor architecture;
- whether the process runs in an interactive terminal and whether it runs in a CI environment;
- whether Argent is used in connection with Android, iOS, tvOS (Apple TV), Android TV, VegaOS, or a Chromium-based target.
- whether Argent is used in connection with Android, iOS, tvOS (Apple TV), Android TV, VegaOS, or a Chromium-based target, and whether that target is a simulator or emulator, a virtual device, a physical device, or a desktop application. This category is inferred from the form of the device identifier; the identifier itself is never recorded;
- when a device is supplied by an external device provider, a short label for that provider (the leading segment of the name the provider registers under), never the identifier of the device it supplies.
### Diagnostics
+23 -1
View File
@@ -1,6 +1,6 @@
// sanitize.ts enforces this same event surface at runtime.
import type { FailureSignal } from "@argent/registry";
import type { DeviceKind, FailureSignal } from "@argent/registry";
import type { AiTelemetryProps } from "./ai-identity.js";
// Single source of truth for the telemetry device-platform enum: sanitize.ts's
@@ -23,6 +23,19 @@ export const PLATFORMS = [
] as const;
export type Platform = (typeof PLATFORMS)[number];
// Telemetry-only subset of the registry's `DeviceKind`: `unknown` is never
// produced by `resolveDevice` and is rejected by the sanitizer, as
// `platform: "unknown"` already is. Reported next to `platform` from the same
// device id — `platform: "ios"` + `device_kind: "device"` is a physical iPhone.
export const DEVICE_KINDS = [
"simulator",
"emulator",
"vvd",
"device",
"app",
] as const satisfies readonly DeviceKind[];
export type TelemetryDeviceKind = (typeof DEVICE_KINDS)[number];
type FailureTelemetryProps = Partial<FailureSignal>;
export interface InstallationCliInitStartProps {
@@ -139,6 +152,13 @@ export interface ToolInvokeProps extends AiTelemetryProps {
tool: string;
tool_invocation_id: string;
platform?: Platform;
/**
* Simulator / emulator / physical device / desktop app, derived with
* `platform` from the same device id. Omitted when `platform` is omitted, when
* the call names no concrete device (an `avdName`-only boot), or when the id
* has no positively recognised shape (see tool-server telemetry-platform.ts).
*/
device_kind?: TelemetryDeviceKind;
}
export interface ToolCompleteProps extends AiTelemetryProps {
@@ -146,6 +166,7 @@ export interface ToolCompleteProps extends AiTelemetryProps {
tool: string;
tool_invocation_id: string;
platform?: Platform;
device_kind?: TelemetryDeviceKind;
duration_ms: number;
}
@@ -154,6 +175,7 @@ export interface ToolFailProps extends FailureTelemetryProps, AiTelemetryProps {
tool: string;
tool_invocation_id?: string;
platform?: Platform;
device_kind?: TelemetryDeviceKind;
duration_ms: number;
/**
* Parameter names that failed zod validation on an HTTP tool call, plus the
+1
View File
@@ -30,6 +30,7 @@ export type {
EventName,
EventPropertyMap,
Platform,
TelemetryDeviceKind,
} from "./events.js";
export { DEBUGGER_NOT_CONNECTED_REASONS, DEBUGGER_TOOL_OUTCOMES, PLATFORMS } from "./events.js";
export type { Runtime } from "./base-props.js";
+5 -1
View File
@@ -1,6 +1,6 @@
import { FAILURE_CODES, getFailureSignalOrFallback, type Registry } from "@argent/registry";
import { track } from "./index.js";
import type { Platform } from "./events.js";
import type { Platform, TelemetryDeviceKind } from "./events.js";
import { aiTelemetryFromMeta, type AiTelemetryProps } from "./ai-identity.js";
// Filled by the HTTP layer so registry lifecycle events carry platform and
@@ -9,6 +9,7 @@ interface InvocationMeta extends AiTelemetryProps {
/** Vendor label of the external provider supplying the target device. */
device_provider?: string;
platform?: Platform;
device_kind?: TelemetryDeviceKind;
}
interface AttachHandle {
@@ -37,6 +38,7 @@ export function attachRegistryTelemetry(registry: Registry): AttachHandle {
tool_invocation_id: toolInvocationId,
...(meta.device_provider ? { device_provider: meta.device_provider } : {}),
...(meta.platform ? { platform: meta.platform } : {}),
...(meta.device_kind ? { device_kind: meta.device_kind } : {}),
...aiTelemetryFromMeta(meta),
});
};
@@ -48,6 +50,7 @@ export function attachRegistryTelemetry(registry: Registry): AttachHandle {
tool_invocation_id: toolInvocationId,
...(meta.device_provider ? { device_provider: meta.device_provider } : {}),
...(meta.platform ? { platform: meta.platform } : {}),
...(meta.device_kind ? { device_kind: meta.device_kind } : {}),
duration_ms: durationMs,
...aiTelemetryFromMeta(meta),
});
@@ -71,6 +74,7 @@ export function attachRegistryTelemetry(registry: Registry): AttachHandle {
tool_invocation_id: toolInvocationId,
...(meta.device_provider ? { device_provider: meta.device_provider } : {}),
...(meta.platform ? { platform: meta.platform } : {}),
...(meta.device_kind ? { device_kind: meta.device_kind } : {}),
duration_ms: durationMs,
...signal,
...aiTelemetryFromMeta(meta),
+5
View File
@@ -9,6 +9,7 @@ import {
} from "@argent/registry";
import {
DEBUGGER_TOOL_OUTCOMES,
DEVICE_KINDS,
PLATFORMS,
type EventName,
type EventPropertyMap,
@@ -53,6 +54,7 @@ const arrayOf =
const TOOL_NAME = matches(/^[a-z][a-z0-9_-]{0,63}$/, 64);
const PLATFORM = oneOf(PLATFORMS);
const DEVICE_KIND = oneOf(DEVICE_KINDS);
const UUID = matches(
/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/,
36
@@ -224,6 +226,7 @@ export const ALLOWED: ValidatorMap = {
tool: TOOL_NAME,
tool_invocation_id: UUID,
platform: PLATFORM,
device_kind: DEVICE_KIND,
...AI_TELEMETRY,
},
"tool:complete": {
@@ -231,6 +234,7 @@ export const ALLOWED: ValidatorMap = {
tool: TOOL_NAME,
tool_invocation_id: UUID,
platform: PLATFORM,
device_kind: DEVICE_KIND,
duration_ms: DURATION_MS,
...AI_TELEMETRY,
},
@@ -239,6 +243,7 @@ export const ALLOWED: ValidatorMap = {
tool: TOOL_NAME,
tool_invocation_id: UUID,
platform: PLATFORM,
device_kind: DEVICE_KIND,
duration_ms: DURATION_MS,
// Emit side sends only names declared in the tool's zod shape, capped at 16
// because arrayOf voids the whole array once it is longer.
@@ -50,6 +50,57 @@ describe("attachRegistryTelemetry", () => {
handle.detach();
});
it("threads the device kind through invoke / complete / fail events", () => {
const trackSpy = vi.spyOn(telemetry, "track");
const registry = new Registry();
const handle = attachRegistryTelemetry(registry);
handle.recordInvocation(INVOCATION_ID_1, { platform: "ios", device_kind: "device" });
registry.events.emit("toolInvoked", "gesture-tap", INVOCATION_ID_1, "Starting tool.");
registry.events.emit("toolCompleted", "gesture-tap", INVOCATION_ID_1, 10, "Completed tool.");
handle.recordInvocation(INVOCATION_ID_2, { platform: "ios", device_kind: "device" });
registry.events.emit("toolInvoked", "gesture-tap", INVOCATION_ID_2, "Starting tool.");
registry.events.emit(
"toolFailed",
"gesture-tap",
INVOCATION_ID_2,
withFailureSignal(new Error("boom"), {
error_code: FAILURE_CODES.REGISTRY_TOOL_FAILURE_UNCLASSIFIED,
failure_stage: "test",
failure_area: "registry",
error_kind: "unknown",
}),
5,
"Failed tool."
);
expect(trackSpy).toHaveBeenCalledTimes(4);
for (const call of trackSpy.mock.calls) {
expect(call[1]).toMatchObject({ platform: "ios", device_kind: "device" });
}
handle.detach();
});
it("omits device_kind when the invocation meta carries none", () => {
const trackSpy = vi.spyOn(telemetry, "track");
const registry = new Registry();
const handle = attachRegistryTelemetry(registry);
handle.recordInvocation(INVOCATION_ID_1, { platform: "android" });
registry.events.emit("toolInvoked", "boot-device", INVOCATION_ID_1, "Starting tool.");
expect(trackSpy.mock.calls[0]![1]).toEqual({
tool: "boot-device",
tool_invocation_id: INVOCATION_ID_1,
platform: "android",
});
expect(trackSpy.mock.calls[0]![1]).not.toHaveProperty("device_kind");
handle.detach();
});
it("threads the AI client through invoke / complete / fail events", () => {
const trackSpy = vi.spyOn(telemetry, "track");
const registry = new Registry();
+37 -1
View File
@@ -1,7 +1,7 @@
import { describe, expect, it } from "vitest";
import { FAILURE_CODES } from "@argent/registry";
import { sanitize, ALLOWED } from "../src/sanitize.js";
import { DEBUGGER_TOOL_OUTCOMES, EVENT_NAMES, PLATFORMS } from "../src/events.js";
import { DEBUGGER_TOOL_OUTCOMES, DEVICE_KINDS, EVENT_NAMES, PLATFORMS } from "../src/events.js";
describe("sanitize", () => {
describe("event allowlist", () => {
@@ -60,6 +60,42 @@ describe("sanitize", () => {
expect(PLATFORMS).toContain("android-tv");
});
it("accepts every device kind on the tool lifecycle events", () => {
// `device_kind` rides next to `platform`; `platform: "ios"` + `device_kind:
// "device"` is how a physical iPhone is told apart from a simulator.
for (const event of ["tool:invoke", "tool:complete", "tool:fail"] as const) {
for (const device_kind of DEVICE_KINDS) {
expect(sanitize(event, { tool: "describe", platform: "ios", device_kind })).toEqual({
tool: "describe",
platform: "ios",
device_kind,
});
}
}
expect(DEVICE_KINDS).toContain("device");
});
it("drops a device kind outside the enum, including the registry's `unknown`", () => {
// Mirror of the `platform: "unknown"` rule above: `resolveDevice` never
// produces `unknown`, so a bucket for it could only be filled by a bug.
for (const device_kind of ["unknown", "iphone", "Simulator", 1, null]) {
expect(sanitize("tool:invoke", { tool: "describe", platform: "ios", device_kind })).toEqual(
{ tool: "describe", platform: "ios" }
);
}
});
it("drops device_kind on events that do not declare it", () => {
expect(
sanitize("debugger:tool_outcome", {
tool: "debugger-status",
outcome: "connected",
platform: "ios",
device_kind: "device",
})
).toEqual({ tool: "debugger-status", outcome: "connected", platform: "ios" });
});
it("drops the legacy `from_tar` decision (developer-only path is off the books)", () => {
expect(sanitize("installation:global_install_decision", { decision: "from_tar" })).toEqual(
{}
+44 -22
View File
@@ -19,6 +19,7 @@ import {
AI_CLIENTS,
type AiTelemetryProps,
type Platform as TelemetryPlatform,
type TelemetryDeviceKind,
} from "@argent/telemetry";
import { ToolNotFoundError } from "@argent/registry";
import { createIdleTimer, IDLE_CHECK_INTERVAL_MS } from "./utils/idle-timer";
@@ -47,8 +48,8 @@ import {
externalSupportHint,
isExternalId,
} from "./utils/external-devices";
import { canonicalDeviceId } from "./utils/debugger/device-alias";
import { refineTvPlatform } from "./utils/telemetry-platform";
import { canonicalDeviceId, isLogicalKeyedDevice } from "./utils/debugger/device-alias";
import { attributeDeviceForTelemetry, type DeviceAttribution } from "./utils/telemetry-platform";
import { deriveInvalidParams } from "./utils/invalid-params";
import type { Server as HttpServer } from "node:http";
import {
@@ -182,6 +183,7 @@ type InvocationMeta = {
/** Coarse vendor label only — see {@link externalProviderLabel}. */
device_provider?: string;
platform?: TelemetryPlatform;
device_kind?: TelemetryDeviceKind;
} & AiTelemetryProps;
// Coarse context only: the raw device id (UDID / serial) infers a platform and is
// never stored or forwarded; invalid_params carries schema-declared parameter
@@ -189,10 +191,11 @@ type InvocationMeta = {
type HttpFailureMeta = {
device_provider?: string;
platform?: TelemetryPlatform;
device_kind?: TelemetryDeviceKind;
invalid_params?: string[];
} & AiTelemetryProps;
function inferPlatform(deviceId: string | null): TelemetryPlatform | null {
function inferDeviceAttribution(deviceId: string | null): DeviceAttribution | null {
if (!deviceId) return null;
try {
// Telemetry-only: rewrite a forwarded Metro logicalDeviceId back to the id
@@ -200,7 +203,14 @@ function inferPlatform(deviceId: string | null): TelemetryPlatform | null {
// the same platform for one invocation — the opaque hex handle would
// otherwise shape-classify as android. Unaliased ids pass through unchanged.
const canonical = canonicalDeviceId(deviceId) ?? deviceId;
return refineTvPlatform(resolveDevice(canonical).platform, canonical);
const attribution = attributeDeviceForTelemetry(canonical);
// A session addressable only by its logicalDeviceId has no device id to
// derive a kind from. The handle's shape already fails the hardware-serial
// gate today, but Metro ids are opaque and their shape is not contractual.
if (attribution.device_kind && isLogicalKeyedDevice(canonical)) {
return { platform: attribution.platform };
}
return attribution;
} catch {
return null;
}
@@ -230,8 +240,8 @@ function extractInvocationMeta(
): InvocationMeta | null {
const meta: InvocationMeta = { ...aiMeta };
if (hasCapability && data && typeof data === "object") {
const platform = platformFromArgs(data);
if (platform) meta.platform = platform;
const attribution = deviceAttributionFromArgs(data);
if (attribution) Object.assign(meta, attribution);
}
const deviceArg = extractDeviceArg(data);
const provider = deviceArg ? externalProviderLabel(deviceArg) : undefined;
@@ -240,42 +250,54 @@ function extractInvocationMeta(
}
/**
* Telemetry platform from a tool call's device arg, or null when it carries none.
* A device id refines to `tvos` / `android-tv` once the runtime-kind cache is warm
* (coarse `ios` / `android` until then); the `avdName`-only fallback is always
* coarse.
* Telemetry platform and device kind from a tool call's device arg, or null when
* it carries none. A device id refines to `tvos` / `android-tv` once the
* runtime-kind cache is warm (coarse `ios` / `android` until then); the
* `avdName`-only fallback is always coarse and carries no kind.
* Exported for tests (http-platform-alias.test.ts).
*/
export function platformFromArgs(data: unknown): TelemetryPlatform | null {
export function deviceAttributionFromArgs(data: unknown): DeviceAttribution | null {
if (!data || typeof data !== "object") return null;
const deviceArg = extractDeviceArg(data);
if (deviceArg) return inferPlatform(deviceArg) ?? null;
if (deviceArg) return inferDeviceAttribution(deviceArg);
// An `avdName`-only call (boot-device before the emulator exists) has no serial
// to resolve a runtime kind from, so it stays coarse `android`; later `udid` /
// `device_id` calls refine an Android TV AVD once the cache is warm.
if (typeof (data as Record<string, unknown>).avdName === "string") return "android";
if (typeof (data as Record<string, unknown>).avdName === "string") {
return { platform: "android" };
}
return null;
}
/** Platform half of {@link deviceAttributionFromArgs}. */
export function platformFromArgs(data: unknown): TelemetryPlatform | null {
return deviceAttributionFromArgs(data)?.platform ?? null;
}
/**
* Attribution for a sub-tool an orchestrator dispatches: the AI client is
* inherited, but the platform is re-derived from the child's OWN device arg.
* Orchestrators like flow-execute carry no platform (and a flow can span several
* devices), so the parent's platform is only the fallback.
* inherited, but platform, device kind and provider label are re-derived from
* the child's OWN device arg. Orchestrators like flow-execute carry no platform
* (and a flow can span several devices), so the parent's trio is the fallback
* only when the child names no device.
*/
function deriveChildInvocationMeta(parentMeta: InvocationMeta, childArgs: unknown): InvocationMeta {
const childPlatform = platformFromArgs(childArgs);
const childAttribution = deviceAttributionFromArgs(childArgs);
if (!childAttribution) return parentMeta;
const childDeviceArg = extractDeviceArg(childArgs);
/**
* Re-derived like the platform. A flow can dispatch across several devices,
* so inheriting the parent's label would misattribute them.
*/
const childProvider = childDeviceArg ? externalProviderLabel(childDeviceArg) : undefined;
if (!childPlatform && !childProvider) return parentMeta;
// The child named its own device, so its attribution replaces the parent's
// wholesale: an `avdName`-only child must not report the parent's kind next
// to its own platform, nor an `ext:` parent's label next to a native device.
const { platform: _p, device_kind: _k, device_provider: _d, ...inherited } = parentMeta;
return {
...parentMeta,
...inherited,
...(childProvider ? { device_provider: childProvider } : {}),
...(childPlatform ? { platform: childPlatform } : {}),
...childAttribution,
};
}
@@ -671,12 +693,12 @@ export function createHttpApp(registry: Registry, options?: HttpAppOptions): Htt
if (!options?.recordFailure) return;
const failedDeviceArg = extractDeviceArg(parsedDataForMeta);
const provider = failedDeviceArg ? externalProviderLabel(failedDeviceArg) : undefined;
const platform = inferPlatform(failedDeviceArg);
const attribution = inferDeviceAttribution(failedDeviceArg);
options.recordFailure(
name,
{
...(provider ? { device_provider: provider } : {}),
...(platform ? { platform } : {}),
...attribution,
...(extraMeta?.invalid_params?.length
? { invalid_params: extraMeta.invalid_params }
: {}),
+1
View File
@@ -382,6 +382,7 @@ export function start(): void {
tool: toolId,
...(meta.device_provider ? { device_provider: meta.device_provider } : {}),
...(meta.platform ? { platform: meta.platform } : {}),
...(meta.device_kind ? { device_kind: meta.device_kind } : {}),
...(meta.invalid_params?.length ? { invalid_params: meta.invalid_params } : {}),
duration_ms: durationMs,
...signal,
@@ -1,8 +1,9 @@
import type { Platform as DevicePlatform } from "@argent/registry";
import type { Platform as TelemetryPlatform } from "@argent/telemetry";
import { classifyDevice } from "./device-info";
import type { DeviceInfo, Platform as DevicePlatform } from "@argent/registry";
import type { Platform as TelemetryPlatform, TelemetryDeviceKind } from "@argent/telemetry";
import { resolveDevice } from "./device-info";
import { externalNativeId } from "./external-devices";
import { getCachedSimulatorRuntimeKind } from "./ios-devices";
import { getCachedAndroidRuntimeKind } from "./adb";
import { consolePortFromAdbSerial, getCachedAndroidRuntimeKind } from "./adb";
export type { TelemetryPlatform };
@@ -15,10 +16,7 @@ export type { TelemetryPlatform };
* before a describe/interaction path warms the runtime-kind cache report the base
* platform.
*/
export function refineTvPlatform(
basePlatform: DevicePlatform,
deviceId: string
): TelemetryPlatform {
function refineTvPlatform(basePlatform: DevicePlatform, deviceId: string): TelemetryPlatform {
if (basePlatform === "ios" && getCachedSimulatorRuntimeKind(deviceId) === "tv") {
return "tvos";
}
@@ -28,11 +26,64 @@ export function refineTvPlatform(
return basePlatform;
}
/** What `tool:*` events report about the device a call targets. */
export interface DeviceAttribution {
platform: TelemetryPlatform;
/** Absent when the id has no positively recognised shape — see `telemetryDeviceKind`. */
device_kind?: TelemetryDeviceKind;
}
/**
* Android `kind: "device"` is `resolveDevice`'s bucket for ANY id that is neither
* iOS-shaped nor `emulator-`: a typo'd udid, a simulator name, a 40-hex Metro
* logicalDeviceId, a legacy 40-hex iPhone UDID. Reporting those as hardware
* would poison the one metric `device_kind` exists for, so `device` is emitted
* only when the native id looks like adb hardware:
* - a USB serial: 620 alphanumerics with at least one digit AND one letter
* (`R5CR30ABCDE`, `HT82A0203045`; pure-letter or pure-digit strings are far more
* often mistakes such as `booted` than serials), or
* - a non-loopback IPv4 `host:port` (shape-only: octets and port are not bounded).
* A loopback `host:port` is an emulator console slot per `consolePortFromAdbSerial`
* and reports `emulator` a phone reached through an `adb connect 127.0.0.1:<port>`
* tunnel is the one false positive. Known false negatives (no kind): IPv6 or
* hostname serials, serials with `_`/`-` or longer than 20 chars, adb-over-Wi-Fi
* mDNS names. A non-loopback `ip:port` may also be a virtual device on another
* host (Genymotion, Waydroid, a Docker-hosted emulator) best effort.
*/
const ANDROID_USB_SERIAL = /^(?=.*\d)(?=.*[A-Za-z])[A-Za-z0-9]{6,20}$/;
const ANDROID_TCP_SERIAL = /^(?:\d{1,3}\.){3}\d{1,3}:\d{1,5}$/;
function telemetryDeviceKind(
device: DeviceInfo,
nativeId: string
): TelemetryDeviceKind | undefined {
if (device.kind === "unknown") return undefined;
// Every other kind is already a positive shape match in `resolveDevice`:
// simulator (UUID / `remote:`), device on iOS (`isIosPhysicalUdid`), emulator
// (`emulator-`), vvd (`amazon-`), app (`chromium-cdp-`).
if (device.platform !== "android" || device.kind !== "device") return device.kind;
if (consolePortFromAdbSerial(nativeId) !== null) return "emulator";
if (ANDROID_USB_SERIAL.test(nativeId) || ANDROID_TCP_SERIAL.test(nativeId)) return "device";
return undefined;
}
/**
* Telemetry platform and device kind for a raw device id. The single classifier
* behind every event that attributes a device, so `tool:*`, `debugger:tool_outcome`
* and `lens:*` cannot drift apart. An `ext:` id is attributed by its native id.
*/
export function attributeDeviceForTelemetry(deviceId: string): DeviceAttribution {
const device = resolveDevice(deviceId);
const platform = refineTvPlatform(device.platform, deviceId);
const device_kind = telemetryDeviceKind(device, externalNativeId(deviceId));
return device_kind ? { platform, device_kind } : { platform };
}
/**
* Telemetry platform for a raw device id, for events that classify the device
* themselves (Lens funnel, debugger outcomes) so a TV target is attributed the
* same way the `tool:*` path in http.ts attributes it.
*/
export function classifyDeviceForTelemetry(deviceId: string): TelemetryPlatform {
return refineTvPlatform(classifyDevice(deviceId), deviceId);
return attributeDeviceForTelemetry(deviceId).platform;
}
@@ -1,6 +1,11 @@
import { describe, it, expect, afterEach } from "vitest";
import { platformFromArgs } from "../src/http";
import { rememberDeviceAlias, resetDeviceAliases } from "../src/utils/debugger/device-alias";
import { deviceAttributionFromArgs, platformFromArgs } from "../src/http";
import {
forgetLogicalKeyedDevice,
rememberDeviceAlias,
rememberLogicalKeyedDevice,
resetDeviceAliases,
} from "../src/utils/debugger/device-alias";
/**
* The platform on tool:invoke / tool:complete / tool:fail must agree with the
@@ -14,6 +19,7 @@ import { rememberDeviceAlias, resetDeviceAliases } from "../src/utils/debugger/d
const LOGICAL_ID = "8b9223b1392be193fa9058e0cef5cefb2bddeb68";
const IOS_UDID = "BE1DCAD9-43CE-40C4-B8B2-9CB30BC03227";
const PHYSICAL_UDID = "00008030-000A1B2C3D4E5F60";
afterEach(() => {
resetDeviceAliases();
@@ -33,4 +39,39 @@ describe("http platform inference and the device alias", () => {
expect(platformFromArgs({ udid: IOS_UDID })).toBe("ios");
expect(platformFromArgs({ device_id: "chromium-cdp-9222" })).toBe("chromium");
});
it("derives the device kind from the aliased id, not the opaque handle", () => {
// A learned alias to a physical iPhone makes the debugger call count as
// hardware; the 40-hex handle alone never would.
rememberDeviceAlias(LOGICAL_ID, PHYSICAL_UDID);
expect(deviceAttributionFromArgs({ device_id: LOGICAL_ID })).toEqual({
platform: "ios",
device_kind: "device",
});
});
it("an un-aliased handle keeps the shape fallback platform and carries no kind", () => {
// The fallback `android` is pre-existing; what must not happen is the
// handle being counted as a physical Android phone.
expect(deviceAttributionFromArgs({ device_id: LOGICAL_ID })).toEqual({ platform: "android" });
});
it("a logical-keyed session carries no kind even if its id looked like a serial", () => {
// Two devices on one Metro: the caller connects with the logicalDeviceId
// itself, so there is no alias to canonicalise through. Marked sessions
// strip the kind regardless of what the handle's shape suggests.
const serialShaped = "R5CT12345678";
rememberLogicalKeyedDevice(serialShaped, serialShaped);
try {
expect(deviceAttributionFromArgs({ device_id: serialShaped })).toEqual({
platform: "android",
});
} finally {
forgetLogicalKeyedDevice(serialShaped);
}
expect(deviceAttributionFromArgs({ device_id: serialShaped })).toEqual({
platform: "android",
device_kind: "device",
});
});
});
+135 -11
View File
@@ -151,6 +151,7 @@ describe("GET /tools progressive-loading metadata", () => {
expect(recordInvocation).toHaveBeenCalledWith(expect.any(String), {
platform: "ios",
device_kind: "simulator",
});
expect(seenMeta).not.toHaveProperty("bundleId");
expect(seenMeta).not.toHaveProperty("deviceId");
@@ -218,7 +219,7 @@ describe("GET /tools progressive-loading metadata", () => {
// The first id is enough for the coarse platform; a mixed-platform scope
// is not something this dimension tries to represent.
expect(seenMeta).toEqual({ platform: "android" });
expect(seenMeta).toEqual({ platform: "android", device_kind: "emulator" });
});
it("ignores a devices list that holds no usable id", async () => {
@@ -269,7 +270,11 @@ describe("GET /tools progressive-loading metadata", () => {
.expect(400);
expect(recordFailure).toHaveBeenCalled();
expect(recordFailure.mock.calls[0][1]).toMatchObject({ platform: "android" });
expect(recordFailure.mock.calls[0][1]).toEqual({
platform: "android",
device_kind: "emulator",
invalid_params: ["unrecognized_keys"],
});
});
it("refines an iOS device to `tvos` when its cached runtime kind is tv", async () => {
@@ -288,7 +293,8 @@ describe("GET /tools progressive-loading metadata", () => {
.expect(200);
// Same UDID shape as an iPhone sim, but the warm cache splits it out as tvOS.
expect(seenMeta).toEqual({ platform: "tvos" });
// The kind never depends on the cache: a tvOS target is still a simulator.
expect(seenMeta).toEqual({ platform: "tvos", device_kind: "simulator" });
});
it("refines an Android device to `android-tv` when its cached runtime kind is tv", async () => {
@@ -306,7 +312,7 @@ describe("GET /tools progressive-loading metadata", () => {
.send({ udid: "emulator-5554" })
.expect(200);
expect(seenMeta).toEqual({ platform: "android-tv" });
expect(seenMeta).toEqual({ platform: "android-tv", device_kind: "emulator" });
});
it("keeps the coarse `ios` platform when the cached kind is mobile (not tv)", async () => {
@@ -324,7 +330,7 @@ describe("GET /tools progressive-loading metadata", () => {
.send({ udid: "11111111-1111-1111-1111-111111111111" })
.expect(200);
expect(seenMeta).toEqual({ platform: "ios" });
expect(seenMeta).toEqual({ platform: "ios", device_kind: "simulator" });
});
it("keeps the coarse platform when the cache is cold (first call before warm-up)", async () => {
@@ -344,7 +350,7 @@ describe("GET /tools progressive-loading metadata", () => {
.send({ udid: "11111111-1111-1111-1111-111111111111" })
.expect(200);
expect(seenMeta).toEqual({ platform: "ios" });
expect(seenMeta).toEqual({ platform: "ios", device_kind: "simulator" });
});
it("re-derives a child sub-tool's TV platform from its own device arg", async () => {
@@ -366,7 +372,10 @@ describe("GET /tools progressive-loading metadata", () => {
// A sub-tool targeting the same TV device is attributed to android-tv too.
recordChildInvocation("tv-child", { udid: "emulator-5554" });
expect(recordInvocation).toHaveBeenCalledWith("tv-child", { platform: "android-tv" });
expect(recordInvocation).toHaveBeenCalledWith("tv-child", {
platform: "android-tv",
device_kind: "emulator",
});
});
it("records the AI client from request headers alongside platform", async () => {
@@ -384,7 +393,7 @@ describe("GET /tools progressive-loading metadata", () => {
.send({ udid: "11111111-1111-1111-1111-111111111111" })
.expect(200);
expect(seenMeta).toEqual({ platform: "ios", ai_client: "codex" });
expect(seenMeta).toEqual({ platform: "ios", device_kind: "simulator", ai_client: "codex" });
});
it("forwards a child-invocation recorder bound to the request's attribution", async () => {
@@ -402,7 +411,11 @@ describe("GET /tools progressive-loading metadata", () => {
// The parent invocation is recorded with the resolved attribution.
expect(recordInvocation).toHaveBeenCalledTimes(1);
expect(recordInvocation.mock.calls[0]![1]).toEqual({ platform: "ios", ai_client: "codex" });
expect(recordInvocation.mock.calls[0]![1]).toEqual({
platform: "ios",
device_kind: "simulator",
ai_client: "codex",
});
// A recorder is threaded into the tool context so orchestrator tools can
// attribute the sub-tools they dispatch. The AI client is inherited; a child
@@ -415,6 +428,7 @@ describe("GET /tools progressive-loading metadata", () => {
const childRelease = opts.recordChildInvocation!("child-id");
expect(recordInvocation).toHaveBeenCalledWith("child-id", {
platform: "ios",
device_kind: "simulator",
ai_client: "codex",
});
expect(childRelease).toBe(release);
@@ -444,14 +458,124 @@ describe("GET /tools progressive-loading metadata", () => {
expect(recordInvocation).toHaveBeenCalledWith("android-child", {
ai_client: "codex",
platform: "android",
device_kind: "emulator",
});
// A child with no device arg falls back to the parent's platform.
// A child with no device arg falls back to the parent's platform and kind.
recordChildInvocation("no-device-child", { message: "hi" });
expect(recordInvocation).toHaveBeenCalledWith("no-device-child", {
ai_client: "codex",
platform: "ios",
device_kind: "simulator",
});
// A child that names a physical phone reports hardware, never the parent's
// simulator kind. `R5CT12345678` is the serial the adb tests use.
recordChildInvocation("phone-child", { udid: "R5CT12345678" });
expect(recordInvocation).toHaveBeenCalledWith("phone-child", {
ai_client: "codex",
platform: "android",
device_kind: "device",
});
// An `avdName`-only child has a platform but no serial to derive a kind
// from: it must not report the parent's `simulator` next to `android`.
recordChildInvocation("avd-child", { avdName: "Pixel_9" });
expect(recordInvocation).toHaveBeenCalledWith("avd-child", {
ai_client: "codex",
platform: "android",
});
});
it("records `device` for a physical iPhone UDID, `simulator` for a simulator one", async () => {
// The whole point of `device_kind`: the two share `platform: "ios"`, and only
// the UDID shape (`isIosPhysicalUdid`) tells hardware from a simulator.
let seenMeta: Record<string, unknown> | undefined;
const recordInvocation = vi.fn((_id: string, meta: Record<string, unknown>) => {
seenMeta = meta;
return vi.fn();
});
const registry = stubRegistry();
// `device-tool` above is simulator-only, so the HTTP capability gate would
// reject the phone before attribution; this one supports both.
(registry.getTool as unknown as ReturnType<typeof vi.fn>).mockReturnValue({
id: "phone-tool",
description: "Phone tool",
inputSchema: { type: "object", properties: {} },
capability: { apple: { simulator: true, device: true } },
services: () => ({}),
execute: async () => ({}),
});
handle.dispose();
handle = createHttpApp(registry, { recordInvocation });
await request(handle.app)
.post("/tools/phone-tool")
.send({ udid: "00008030-000A1B2C3D4E5F60" })
.expect(200);
expect(seenMeta).toEqual({ platform: "ios", device_kind: "device" });
await request(handle.app)
.post("/tools/phone-tool")
.send({ udid: "11111111-1111-1111-1111-111111111111" })
.expect(200);
expect(seenMeta).toEqual({ platform: "ios", device_kind: "simulator" });
});
it("drops the parent's external-provider label when a child names a native device", async () => {
const recordInvocation = vi.fn((_id: string, _meta: Record<string, unknown>) => vi.fn());
const registry = stubRegistry();
handle.dispose();
handle = createHttpApp(registry, { recordInvocation });
// Parent targets a device an external provider attached.
await request(handle.app)
.post("/tools/device-tool")
.send({ udid: "ext:acme-3f2a9c:11111111-1111-1111-1111-111111111111" })
.expect(200);
expect(recordInvocation.mock.calls[0]![1]).toEqual({
device_provider: "acme",
platform: "ios",
device_kind: "simulator",
});
const { recordChildInvocation } = vi.mocked(registry.invokeTool).mock.calls[0]![2] as {
recordChildInvocation: (id: string, childArgs?: unknown) => () => void;
};
// The child named its own, non-external device: the provider label is
// re-derived from it (to nothing), not inherited from the parent.
recordChildInvocation("native-child", { udid: "emulator-5554" });
expect(recordInvocation).toHaveBeenCalledWith("native-child", {
platform: "android",
device_kind: "emulator",
});
});
it("classifies a FAILED call's platform but not its kind when the udid is garbage", async () => {
// `emitHttpFailure` reads the raw body before validation, so a typo'd udid
// reaches the classifier. It still shape-classifies as `android` (the
// pre-existing fallback), but it must not count as physical hardware.
const recordFailure = vi.fn();
const registry = stubRegistry();
(registry.getTool as unknown as ReturnType<typeof vi.fn>).mockReturnValue({
id: "strict-tool",
description: "Strict tool",
inputSchema: { type: "object", properties: { udid: {} } },
zodSchema: z.object({ udid: z.string() }).strict(),
services: () => ({}),
execute: async () => ({}),
});
handle.dispose();
handle = createHttpApp(registry, { recordFailure });
await request(handle.app)
.post("/tools/strict-tool")
.send({ udid: "booted", extra: 1 })
.expect(400);
expect(recordFailure.mock.calls[0]![1]).toMatchObject({ platform: "android" });
expect(recordFailure.mock.calls[0]![1]).not.toHaveProperty("device_kind");
});
it("does not forward a child recorder when there is no attribution to propagate", async () => {
@@ -527,7 +651,7 @@ describe("GET /tools progressive-loading metadata", () => {
.send({ udid: "11111111-1111-1111-1111-111111111111" })
.expect(200);
expect(seenMeta).toEqual({ platform: "ios", ai_client: "codex" });
expect(seenMeta).toEqual({ platform: "ios", device_kind: "simulator", ai_client: "codex" });
});
it("drops a client name sent with no ai_client header", async () => {
@@ -0,0 +1,68 @@
import { describe, expect, it, vi } from "vitest";
import {
attributeDeviceForTelemetry,
classifyDeviceForTelemetry,
} from "../src/utils/telemetry-platform";
// Cache-only readers: cold here, so every platform stays coarse and the kind can
// be pinned independently of the TV refinement.
vi.mock("../src/utils/ios-devices", async (importOriginal) => {
const actual = await importOriginal<typeof import("../src/utils/ios-devices")>();
return { ...actual, getCachedSimulatorRuntimeKind: () => undefined };
});
vi.mock("../src/utils/adb", async (importOriginal) => {
const actual = await importOriginal<typeof import("../src/utils/adb")>();
return { ...actual, getCachedAndroidRuntimeKind: () => undefined };
});
const SIM_UDID = "11111111-1111-1111-1111-111111111111";
const PHYSICAL_UDID = "00008030-000A1B2C3D4E5F60";
const METRO_HANDLE = "8b9223b1392be193fa9058e0cef5cefb2bddeb68";
describe("attributeDeviceForTelemetry", () => {
it.each([
// Positive shapes `resolveDevice` already recognises.
[SIM_UDID, { platform: "ios", device_kind: "simulator" }],
[PHYSICAL_UDID, { platform: "ios", device_kind: "device" }],
[`remote:${SIM_UDID}`, { platform: "ios-remote", device_kind: "simulator" }],
["emulator-5554", { platform: "android", device_kind: "emulator" }],
["chromium-cdp-9222", { platform: "chromium", device_kind: "app" }],
["amazon-1a2b3c", { platform: "vega", device_kind: "vvd" }],
// Android hardware: USB serials with a digit and a letter, non-loopback ip:port.
["R5CT12345678", { platform: "android", device_kind: "device" }],
["HT82A0203045", { platform: "android", device_kind: "device" }],
["192.168.1.5:5555", { platform: "android", device_kind: "device" }],
// Loopback ip:port is an emulator console slot (adb.ts consolePortFromAdbSerial).
["127.0.0.1:5555", { platform: "android", device_kind: "emulator" }],
["localhost:5555", { platform: "android", device_kind: "emulator" }],
["::1:5555", { platform: "android", device_kind: "emulator" }],
// `ext:` ids attribute by their native id.
[`ext:acme-3f2a9c:${PHYSICAL_UDID}`, { platform: "ios", device_kind: "device" }],
["ext:acme-3f2a9c:emulator-5554", { platform: "android", device_kind: "emulator" }],
["ext:acme-3f2a9c:R5CT12345678", { platform: "android", device_kind: "device" }],
])("%s → %o", (id, expected) => {
expect(attributeDeviceForTelemetry(id)).toEqual(expected);
});
it.each([
// `resolveDevice` sends every one of these to android/device; none is hardware.
METRO_HANDLE, // 40-hex Metro logicalDeviceId, also a legacy iPhone UDID shape
"oops",
"booted",
"Pixel_7",
"abcdefgh",
"1234567890",
"[fe80::1]:5555",
"pixel.local:5555",
"adb-R5CT12345678-AbCdEf._adb-tls-connect._tcp.",
"0123456789ABCDEF0123456789ABCDEF", // 32-hex, over the 20-char cap
])("%s carries the fallback platform but no kind", (id) => {
expect(attributeDeviceForTelemetry(id)).toEqual({ platform: "android" });
});
it("classifyDeviceForTelemetry is the platform half of the same classifier", () => {
for (const id of [SIM_UDID, PHYSICAL_UDID, "emulator-5554", "R5CT12345678", METRO_HANDLE]) {
expect(classifyDeviceForTelemetry(id)).toBe(attributeDeviceForTelemetry(id).platform);
}
});
});