mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
* fix(cloud): unblock iOS snapshot and gate cloud fill on text-entry focus (#1658)
Two bugs isolated to the cloud-webdriver iOS path.
`snapshot`/`diff` refused every capture on a live BrowserStack session with
SESSION_NOT_FOUND, instantly and without a driver round trip. The app-session
guard they ran belongs to the local XCUITest runner, which must attach to a
target app; a cloud capture reads the provider's own driver session and needs
no app identity, so it now applies to local Apple targets only. The session
was empty in the first place because the provider open path skips local app
resolution wholesale — no simctl/devicectl reaches a hosted device — and
dropped an explicitly spelled bundle id along with it. A dotted, non-deep-link
target is the bundle id under the same convention resolveIosApp applies
locally, so a cloud `open com.example.app` now records it.
`fill` tapped and sent its keys in back-to-back requests. A WebView input —
an OAuth page in a Safari view controller — takes first responder
asynchronously, so the keys landed with nothing focused while the command
still answered "Filled N chars"; tapping and filling as two separate commands
worked only because the round trip between them gave the field time to focus.
The cloud interactor now waits on the same signal the Apple runner uses, the
software keyboard going from hidden to shown after its tap, and discloses what
it observed as `textEntryReadiness` so a fill with no witness cannot pass for
a filled field. Where keyboard visibility cannot witness the focus move —
back-to-back fills into one form, the shape that failed most often — it spends
the runner's full readiness budget rather than racing the app with a short
settle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Zjbzf7HdziX9SzFpNP7WW
* fix(cloud): let a new bundle-id open replace the tracked cloud iOS app
Adopting an explicitly spelled bundle id on a provider-backed open (the
fix that makes snapshot/diff work at all) also made a previously dead
precedence rule live: the provider branch returned currentAppBundleId
first, so once a first open had populated it, `open com.a` followed by
`open com.b` left the session still reporting com.a to every
appBundleId-gated command.
The local path does the opposite, and is the convention this branch is
meant to mirror: resolveIosApp returns a dotted target unchanged and
never consults the session's current app. Only its deep-link branches
prefer the tracked id. Flip the provider branch to match — an explicit
bundle-id target wins, and everything the branch cannot name (deep
links, display names, bare open) still falls back to the tracked id.
* fix(cloud): fail a witness-less cloud fill instead of reporting it filled
Review follow-ups on #1658.
`not-observed` was still a success: it sent the keys and answered "Filled N
chars", and nothing renders `textEntryReadiness` in default CLI output — so the
exact silent success this branch exists to remove survived whenever focus never
happened. A tap that raises no keyboard now fails with
`text_entry_focus_not_observed` and sends no keys, leaving the field untouched
rather than half-written, and the readiness vocabulary keeps only outcomes that
describe a fill that did type.
The readiness budget was advertised but not enforced at the request boundary:
each keyboard probe inherited the client's 30s default, so one hung probe could
hold a 2s wait for far longer. Probes now carry their own bound, threaded
through the client as a per-request timeout override.
The probe also swallowed every error as "this driver cannot answer", which
degraded a dead session, an auth rejection, or a grid outage into a blind text
entry. Only a positively classified unimplemented route counts as unsupported
now — classified on the W3C error code rather than the status, since `unknown
command` and `invalid session id` share HTTP 404 — and everything else
propagates.
The provider scenario proved request ordering against a stub that always
accepted keys. Its fake now models the device: focus lands a beat after the
tap, and keys arriving while the keyboard is down are accepted and dropped,
exactly as an unfocused field does. The tests assert the field's own value, and
both go red against the pre-fix `fill`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Zjbzf7HdziX9SzFpNP7WW
* fix(cloud): witness the tapped field's focus before a cloud fill types
Review of cc23f2b found two ways a fill could still report success
without evidence that OUR tap focused the field it was aimed at.
P1. `settled-keyboard-up` and `settled-unknown` both typed and returned
normal success. Keyboard visibility can only witness that *a* field took
focus, never *which*: filling a second field in an already-open form
reads the same before and after, so a missed tap left the first field
focused and `POST /keys` — which the driver routes to whatever holds
first responder — appended to it while every request returned 200.
Failing those closed outright would have broken ordinary multi-field
form fills, which do work: a live AWS Device Farm run types both fields
of a WebView login correctly. So witness focus properly instead. W3C
`GET /element/active` answers the question keyboard visibility cannot —
is the thing focused now the thing I tapped — and answers it whether or
not the keyboard was already up. That becomes the primary signal
(`focused-element`); the keyboard transition stays as the fallback for
drivers without the route, and a keyboard already up on such a driver
now refuses rather than typing.
The test is identity, not geometry. Containment of the tap point looks
like the obvious rule and is wrong: focusing a field can re-lay it out.
On a live iPhone 16, tapping Safari's collapsed address bar expands it
into a taller field that no longer covers the tapped point, and a
containment-only rule refused a fill that plainly worked. So a tap that
MOVES focus counts, with containment as the second half of the test —
re-filling the already-focused field moves nothing, and only geometry
tells that from a tap that missed. Both readings are taken before the
tap, since each is evidence only as a change.
P2. The 2s budget bounded the loop but not the calls inside it: every
probe got a fixed 1500ms, so one begun near the deadline finished well
past it. Both the probe timeout and the sleep are now capped by the
remaining budget.
Also fixes a related escape the review did not name: the poll loop had
no catch, so one transient grid error aborted a fill the next poll would
have satisfied. Probe failures are now tolerated within the budget, but
a budget that expires without a single answered probe rethrows, so a
dead session surfaces as itself rather than as "the tap missed".
The provider scenario gains the two-field case the review asked for: it
begins keyboard-up with the email field focused, misses the password
tap, and asserts no keys reach the email field.
Verified on AWS Device Farm iPhone 16 / iOS 18.0 at this exact tree:
address bar (the re-layout case) and both WebView login fields all
report `focused-element`, the second with the keyboard already up, and
the device reads back `tomsmith` and a 20-character password.
* fix(cloud): refuse a cloud fill no focus probe can witness, and bound the composite probe
Two blockers from the review of 3a9aceb9.
P1. `settled-unknown` was the last path that typed without evidence: when
both the active-element and keyboard routes are positively unsupported,
`fill` settled 350ms, typed, and returned ordinary success. Nothing
renders `textEntryReadiness`, so that reached a caller looking exactly
like a fill that worked — the same silent false success #1658 is about,
just narrowed to one branch. It now refuses with a distinct reason,
`text_entry_focus_unobservable`: nothing is wrong with the target, the
driver simply cannot answer, so the caller's next move differs from a
missed tap and the hint names it — `press` then `type` stays the
deliberate way to enter text unwitnessed.
`CLOUD_TEXT_ENTRY_READINESS` is now `focused-element` and
`keyboard-shown` only. Every value describes a fill that witnessed focus
before sending a key; there is deliberately no value for typing blind.
P2. `activeElement(timeoutMs)` bounded each of its two sequential
requests by the full timeout rather than bounding the operation, so a
probe handed the 1.5s left of a 2s readiness deadline could spend ~3s
across `/element/active` and `/element/{id}/rect` and overrun the
deadline it was derived from. It now derives one deadline at entry and
gives the second request only what the first left, floored at zero so an
already-spent budget aborts immediately instead of falling back to the
client default.
The regression pins elapsed transport time across both calls, which is
what the defect is made of: the rect request answers only its own abort,
so the time it was allowed to run IS the budget it was handed. It
measures ~202ms of a shared 200ms budget before the fix and ~120ms
after.
Also updates the generic Cloud WebDriver facade scenario, whose stub
answered `{value: null}` to everything and so read as a driver with
neither route. It now answers the two focus probes, since that scenario
exercises facade wiring rather than text-entry semantics — those live in
cloud-webdriver-ios-text-entry.test.ts, which models focus properly.
---------
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
committed by
GitHub
parent
9b4239b2f3
commit
e14c9d8d7b
+2
-1
@@ -3,7 +3,8 @@
|
||||
## Unreleased
|
||||
|
||||
- `agent-device help workflow` is now a compact ~8KB card instead of a ~41KB dump; the same depth still exists, split into `help scripting` (save-script, secret-safe fills, batch JSON, replay divergence/repair, recording) and `help gestures` (multi-touch shapes and platform quirks), plus a few paragraphs folded into the topics that already owned the subject (`help debugging`, `help physical-device`, `help validate`). Every `help <topic>` first line is now `agent-device <version> — <topic>` so an agent can read the installed version from its mandatory first help read instead of a separate `agent-device --version` call.
|
||||
|
||||
- Cloud iOS (BrowserStack, AWS Device Farm): `snapshot` and `diff` no longer fail with `SESSION_NOT_FOUND` on a live provider session (#1658). The app-session guard they ran belongs to the local XCUITest runner, which must attach to a target app; a cloud capture reads the provider's own driver session and needs no app identity, so the guard now applies to local Apple targets only. Relatedly, a cloud iOS `open com.example.app` now records that bundle id on the session — the provider path skips local app resolution (no simctl/devicectl reaches a hosted device), and used to drop an explicitly spelled bundle id along with it, leaving the session with no app identity at all. Opening a second bundle id replaces the first, matching the local path, where an explicitly spelled target always wins over the session's current app; deep links, display names, and bare `open` still keep the app already tracked.
|
||||
- Cloud `fill` (BrowserStack, AWS Device Farm) now witnesses that the field it tapped actually holds text-entry focus before sending its keys, instead of dispatching tap and keys in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The witness is the focused element's own geometry: `fill` polls the active element and proceeds only once it contains the point it tapped, which is the one signal that identifies *which* field took focus. Keyboard visibility cannot — it reads the same before and after a second fill into an already-open form, so it could not tell a focused password field from the email field the previous fill left focused. The response discloses `textEntryReadiness`: `focused-element`, or `keyboard-shown` when the driver has no active-element route but the keyboard rose from hidden after the tap. Both describe a fill that witnessed focus before typing; there is deliberately no value for typing without evidence, because nothing renders this field and such a value would reach a caller as an ordinary success. Breaking: when focus cannot be witnessed, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. That covers a tap that focused nothing, a keyboard already up on a driver that cannot name the focused field, and a driver that reports neither (`text_entry_focus_unobservable`, which points at `press` + `type` as the deliberate way to enter text unwitnessed). Only a positively classified unimplemented route counts as unsupported, so a dead session, an auth rejection, or a grid outage surfaces instead of degrading into a blind text entry.
|
||||
- `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands.
|
||||
- Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags.
|
||||
- `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied.
|
||||
|
||||
@@ -106,10 +106,16 @@ export type {
|
||||
SettleParams,
|
||||
SettleTailEntry,
|
||||
} from '../interaction.ts';
|
||||
export { MAESTRO_NON_HITTABLE_FALLBACK_MESSAGE, TEXT_ENTRY_ROUTES } from '../interactor-types.ts';
|
||||
export {
|
||||
CLOUD_TEXT_ENTRY_READINESS,
|
||||
MAESTRO_NON_HITTABLE_FALLBACK_MESSAGE,
|
||||
TEXT_ENTRY_ROUTES,
|
||||
} from '../interactor-types.ts';
|
||||
export type {
|
||||
CloudTextEntryReadiness,
|
||||
ElementSelectorKey,
|
||||
ElementSelectorTapOptions,
|
||||
FillBackendResult,
|
||||
Interactor,
|
||||
RunnerCallOptions,
|
||||
RunnerContext,
|
||||
|
||||
@@ -98,6 +98,49 @@ export type TypeTextBackendResult = {
|
||||
textEntryRoute?: TextEntryRoute;
|
||||
};
|
||||
|
||||
/**
|
||||
* What a cloud-WebDriver `fill` could establish about its target taking text
|
||||
* entry focus before it sent the keys (#1658). The local Apple runner owns this
|
||||
* discipline in Swift and reports it as a `textEntryRoute`; the cloud path has
|
||||
* no runner, only what the driver will tell it, so it names what it observed
|
||||
* instead of which channel it typed through. Listed strongest evidence first:
|
||||
*
|
||||
* - `focused-element`: the element holding text-entry focus after the tap
|
||||
* contains the tapped point — positive evidence that THIS field, not merely
|
||||
* some field, took focus.
|
||||
* - `keyboard-shown`: no active-element route, but the software keyboard went
|
||||
* from hidden to shown after our tap. Witnesses that focus arrived somewhere,
|
||||
* which only means our field when the keyboard was down beforehand.
|
||||
* Every value describes a fill that sent its keys AND witnessed focus first.
|
||||
* There is deliberately no value for "typed without evidence": nothing renders
|
||||
* this field, so such a value would reach a caller as an ordinary success and
|
||||
* recreate the #1658 silent false success it exists to remove. Every
|
||||
* unwitnessed case throws without typing instead —
|
||||
*
|
||||
* - `text_entry_focus_not_observed`: the tap focused nothing, or a keyboard
|
||||
* already up on a driver that cannot say which field owns it (keys would have
|
||||
* landed in the PREVIOUS field).
|
||||
* - `text_entry_focus_unobservable`: the driver implements neither route, so no
|
||||
* wait could establish anything. `press` + `type` remains the deliberate way
|
||||
* to enter text unwitnessed.
|
||||
*
|
||||
* Closed set: the cloud interactor is its only producer and the boundary that
|
||||
* narrows it (readFillBackendResult, src/core/dispatch-interactions.ts) drops a
|
||||
* value it cannot name.
|
||||
*/
|
||||
export const CLOUD_TEXT_ENTRY_READINESS = ['focused-element', 'keyboard-shown'] as const;
|
||||
|
||||
export type CloudTextEntryReadiness = (typeof CLOUD_TEXT_ENTRY_READINESS)[number];
|
||||
|
||||
/**
|
||||
* What `Interactor.fill` reports back about the entry it performed. Only the
|
||||
* cloud-WebDriver interactor populates `textEntryReadiness`; the Apple runner
|
||||
* carries its own equivalent in Swift and every other platform fills blind.
|
||||
*/
|
||||
export type FillBackendResult = {
|
||||
textEntryReadiness?: CloudTextEntryReadiness;
|
||||
};
|
||||
|
||||
export type SnapshotOptions = BaseSnapshotOptions & {
|
||||
appBundleId?: string;
|
||||
signal?: AbortSignal;
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { afterEach, test } from 'vitest';
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import { WebDriverClient } from './webdriver-client.ts';
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = realFetch;
|
||||
});
|
||||
|
||||
// #1658: `fill` blind-settles and types when the keyboard route is unsupported.
|
||||
// Anything else reaching that branch would degrade a real provider failure into
|
||||
// a blind text entry, so the split is pinned against the actual wire shapes.
|
||||
test('is_keyboard_shown reports an unimplemented route as unsupported', async () => {
|
||||
// Appium's own answer for a route the driver does not have.
|
||||
const client = await connectedClient(
|
||||
webDriverResponse(404, { value: { error: 'unknown command', message: 'Unknown command' } }),
|
||||
);
|
||||
|
||||
assert.equal(await client.isKeyboardShown(), 'unsupported');
|
||||
});
|
||||
|
||||
test('is_keyboard_shown reports a non-boolean answer as unsupported', async () => {
|
||||
const client = await connectedClient(webDriverResponse(200, { value: null }));
|
||||
|
||||
assert.equal(await client.isKeyboardShown(), 'unsupported');
|
||||
});
|
||||
|
||||
// The status is identical to an unimplemented route's, so a status-only test
|
||||
// would read a dead session as a missing feature and keep typing into it.
|
||||
test('is_keyboard_shown propagates a dead session, which shares the 404', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverResponse(404, {
|
||||
value: {
|
||||
error: 'invalid session id',
|
||||
message: 'A session is either terminated or not started',
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
await assert.rejects(client.isKeyboardShown(), (error: AppError) => {
|
||||
assert.match(error.message, /terminated or not started/);
|
||||
return true;
|
||||
});
|
||||
});
|
||||
|
||||
test('is_keyboard_shown propagates a grid failure', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverResponse(503, { value: { message: 'provider backend unavailable' } }),
|
||||
);
|
||||
|
||||
await assert.rejects(client.isKeyboardShown(), (error: AppError) => {
|
||||
assert.match(error.message, /provider backend unavailable/);
|
||||
return true;
|
||||
});
|
||||
});
|
||||
|
||||
// The readiness poll advertises a 2s budget; without the override each probe
|
||||
// would inherit the client's 30s default and one hung request could hold it
|
||||
// there. A stub that answers only the abort proves the caller's bound is the
|
||||
// one in force — were it ignored, this test would hang to its own timeout.
|
||||
test('is_keyboard_shown honors a caller timeout shorter than the client default', async () => {
|
||||
const client = await connectedClient(
|
||||
async (_input, init) =>
|
||||
await new Promise<Response>((_resolve, reject) => {
|
||||
init?.signal?.addEventListener('abort', () => reject(init.signal?.reason as Error));
|
||||
}),
|
||||
);
|
||||
|
||||
await assert.rejects(client.isKeyboardShown(50), (error: Error) => {
|
||||
assert.match(`${error.name} ${error.message}`, /timeout|abort/i);
|
||||
return true;
|
||||
});
|
||||
});
|
||||
|
||||
// The focused element is the only signal that can say WHICH field took focus,
|
||||
// so `fill` fails closed when it is unavailable. Each wire shape it
|
||||
// distinguishes is pinned here.
|
||||
test('activeElement returns the focused element id and rect', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } },
|
||||
'/rect': { body: { value: { x: 20, y: 300, width: 350, height: 44 } } },
|
||||
}),
|
||||
);
|
||||
|
||||
assert.deepEqual(await client.activeElement(), {
|
||||
id: 'el-1',
|
||||
rect: { x: 20, y: 300, width: 350, height: 44 },
|
||||
});
|
||||
});
|
||||
|
||||
// A form with nothing focused yet is an expected state, not a broken driver —
|
||||
// the caller must keep polling rather than give up.
|
||||
test('activeElement reports nothing focused as none', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': {
|
||||
status: 404,
|
||||
body: { value: { error: 'no such element', message: 'nothing focused' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'none');
|
||||
});
|
||||
|
||||
// Focus moved between the two round trips. A race the caller can simply retry
|
||||
// out of, not a failure worth ending the fill on.
|
||||
test('activeElement reports an element that vanished before its rect as none', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } },
|
||||
'/rect': {
|
||||
status: 404,
|
||||
body: { value: { error: 'no such element', message: 'stale element' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'none');
|
||||
});
|
||||
|
||||
// A grid that answers 200/null is not implementing the route: the spec reports
|
||||
// "nothing focused" as a `no such element` error. Reading null as "nothing
|
||||
// focused" would fail every fill on such a grid instead of falling back.
|
||||
test('activeElement reports a non-element answer as unsupported', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({ '/element/active': { body: { value: null } } }),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'unsupported');
|
||||
});
|
||||
|
||||
test('activeElement reports an unimplemented route as unsupported', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': {
|
||||
status: 404,
|
||||
body: { value: { error: 'unknown command', message: 'Unknown command' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'unsupported');
|
||||
});
|
||||
|
||||
// Same discipline as the keyboard route: a dead session shares the 404 and must
|
||||
// never be mistaken for a driver that simply lacks the feature.
|
||||
test('activeElement propagates a dead session, which shares the 404', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': {
|
||||
status: 404,
|
||||
body: {
|
||||
value: { error: 'invalid session id', message: 'A session is either terminated' },
|
||||
},
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
await assert.rejects(client.activeElement(), (error: AppError) => {
|
||||
assert.match(error.message, /terminated/);
|
||||
return true;
|
||||
});
|
||||
});
|
||||
|
||||
// `timeoutMs` bounds the OPERATION, not each request in it. Given the full
|
||||
// budget twice, these two sequential round trips would together take double it
|
||||
// — a probe handed the 1.5s left of a 2s readiness deadline would run 3s and
|
||||
// overrun the deadline it came from. Real time is unavoidable here: the defect
|
||||
// IS elapsed transport time across two calls.
|
||||
test('activeElement bounds its two sequential requests by one shared budget', async () => {
|
||||
const firstCallMs = 80;
|
||||
const budgetMs = 200;
|
||||
let rectRequestBudgetMs: number | undefined;
|
||||
const client = await connectedClient(async (input, init) => {
|
||||
const url = String(input instanceof Request ? input.url : input);
|
||||
if (url.includes('/rect')) {
|
||||
// Never answers; only its own abort ends it, so how long it was allowed
|
||||
// to run is exactly the budget it was handed.
|
||||
const startedAt = Date.now();
|
||||
return await new Promise<Response>((_resolve, reject) => {
|
||||
init?.signal?.addEventListener('abort', () => {
|
||||
rectRequestBudgetMs = Date.now() - startedAt;
|
||||
reject(init.signal?.reason as Error);
|
||||
});
|
||||
});
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, firstCallMs));
|
||||
return new Response(JSON.stringify({ value: { [W3C_ELEMENT_KEY]: 'el-1' } }), {
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
});
|
||||
|
||||
await assert.rejects(client.activeElement(budgetMs));
|
||||
|
||||
assert.ok(rectRequestBudgetMs !== undefined, 'the rect request should have been made');
|
||||
// It must get what the first call left (~120ms), never a fresh 200ms.
|
||||
assert.ok(
|
||||
rectRequestBudgetMs < budgetMs - firstCallMs / 2,
|
||||
`rect request was allowed ${rectRequestBudgetMs}ms of a shared ${budgetMs}ms budget already ${firstCallMs}ms spent`,
|
||||
);
|
||||
});
|
||||
|
||||
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
|
||||
|
||||
/** Dispatches by URL suffix, since `activeElement` makes two round trips. */
|
||||
function webDriverRoutes(
|
||||
routes: Record<string, { status?: number; body: unknown }>,
|
||||
): typeof globalThis.fetch {
|
||||
const keys = Object.keys(routes).sort((a, b) => b.length - a.length);
|
||||
return async (input) => {
|
||||
const url = String(input instanceof Request ? input.url : input);
|
||||
const key = keys.find((candidate) => url.includes(candidate));
|
||||
const route = key
|
||||
? routes[key]!
|
||||
: { status: 500, body: { value: { message: `unrouted ${url}` } } };
|
||||
return new Response(JSON.stringify(route.body), {
|
||||
status: route.status ?? 200,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
/** A client with a live session id, so the probes reach the session routes. */
|
||||
async function connectedClient(respond: typeof globalThis.fetch): Promise<WebDriverClient> {
|
||||
globalThis.fetch = webDriverResponse(200, {
|
||||
value: { sessionId: 'wd-1', capabilities: {} },
|
||||
});
|
||||
const client = new WebDriverClient({
|
||||
clientVersion: '0.0.0-test',
|
||||
endpoint: 'http://cloud-webdriver.test/wd/hub/',
|
||||
requestPolicy: { timeoutMs: 30_000, retryAttempts: 0 },
|
||||
});
|
||||
await client.createSession({ platformName: 'iOS' });
|
||||
globalThis.fetch = respond;
|
||||
return client;
|
||||
}
|
||||
|
||||
function webDriverResponse(status: number, body: unknown): typeof globalThis.fetch {
|
||||
return async () =>
|
||||
new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
}
|
||||
@@ -35,6 +35,12 @@ export type WebDriverWindowRect = {
|
||||
height: number;
|
||||
};
|
||||
|
||||
export type WebDriverActiveElement = {
|
||||
/** Opaque W3C element reference; only ever compared for equality. */
|
||||
id: string;
|
||||
rect: WebDriverWindowRect;
|
||||
};
|
||||
|
||||
export type W3CPointerAction =
|
||||
| {
|
||||
type: 'pointerMove';
|
||||
@@ -65,6 +71,12 @@ type WebDriverResponse = {
|
||||
|
||||
type WebDriverRequestOverrides = {
|
||||
retryAttempts?: number;
|
||||
/**
|
||||
* Per-request transport bound, for callers whose own budget is far shorter
|
||||
* than the client's default. Without it a caller waiting 2s on a poll can be
|
||||
* held for the full default timeout by one hung request.
|
||||
*/
|
||||
timeoutMs?: number;
|
||||
};
|
||||
|
||||
export class WebDriverClient {
|
||||
@@ -142,6 +154,89 @@ export class WebDriverClient {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the software keyboard is up, or `unsupported` when this driver has
|
||||
* no such route. Polled between a text field tap and the keys it should
|
||||
* receive, so it stays one cheap round trip — no retries, and `timeoutMs`
|
||||
* lets the caller bound it by its own budget rather than the client default.
|
||||
*
|
||||
* The unsupported/failed split matters: a caller that treats every error as
|
||||
* "this driver cannot answer" would silently degrade a dead session, an auth
|
||||
* rejection, or a grid outage into a blind text entry. Only a route the
|
||||
* driver does not implement — or one that answers with a non-boolean — is
|
||||
* reported as unsupported; everything else throws.
|
||||
*/
|
||||
async isKeyboardShown(timeoutMs?: number): Promise<boolean | 'unsupported'> {
|
||||
let value: unknown;
|
||||
try {
|
||||
value = await this.sessionRequest('GET', '/appium/device/is_keyboard_shown', undefined, {
|
||||
retryAttempts: 0,
|
||||
...(timeoutMs === undefined ? {} : { timeoutMs }),
|
||||
});
|
||||
} catch (error) {
|
||||
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
|
||||
throw error;
|
||||
}
|
||||
return typeof value === 'boolean' ? value : 'unsupported';
|
||||
}
|
||||
|
||||
/**
|
||||
* Which element holds text-entry focus and where it sits, `none` when nothing
|
||||
* does, or `unsupported` when this driver has no active-element route.
|
||||
*
|
||||
* Keyboard visibility can only witness that *a* field took focus; it says
|
||||
* nothing about *which*, which is why a second fill into an already-open form
|
||||
* could send its keys to the field the first one focused (#1658). The focused
|
||||
* element's identity answers that. Its rect comes along because identity
|
||||
* alone cannot distinguish a deliberate re-fill of the already-focused field
|
||||
* from a tap that moved nothing.
|
||||
*
|
||||
* `no such element` is a real, expected answer here (a form with nothing
|
||||
* focused yet), so it maps to `none` and leaves the caller polling. A route
|
||||
* that answers 200 with something that is not an element reference is not
|
||||
* implementing this at all — the spec reports "nothing focused" as that
|
||||
* error, never as an empty value — so it degrades to `unsupported` alongside
|
||||
* a genuinely unimplemented route, the same way a non-boolean keyboard answer
|
||||
* does. Reading it as `none` would fail every fill on such a grid.
|
||||
*/
|
||||
async activeElement(
|
||||
timeoutMs?: number,
|
||||
): Promise<WebDriverActiveElement | 'none' | 'unsupported'> {
|
||||
// `timeoutMs` bounds this OPERATION, not each request in it. Two sequential
|
||||
// round trips each handed the full budget would together take twice it — a
|
||||
// probe given the 1.5s left of a 2s readiness deadline could run 3s and
|
||||
// overrun the deadline it was derived from. One deadline, and the second
|
||||
// request gets only what the first left.
|
||||
const deadline = timeoutMs === undefined ? undefined : Date.now() + timeoutMs;
|
||||
let elementId: string | undefined;
|
||||
try {
|
||||
elementId = readW3CElementId(
|
||||
await this.sessionRequest('GET', '/element/active', undefined, requestBudget(deadline)),
|
||||
);
|
||||
} catch (error) {
|
||||
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
|
||||
if (isNoSuchElementError(error)) return 'none';
|
||||
throw error;
|
||||
}
|
||||
if (elementId === undefined) return 'unsupported';
|
||||
try {
|
||||
const value = await this.sessionRequest(
|
||||
'GET',
|
||||
`/element/${encodeURIComponent(elementId)}/rect`,
|
||||
undefined,
|
||||
requestBudget(deadline),
|
||||
);
|
||||
return { id: elementId, rect: readWindowRect(value) };
|
||||
} catch (error) {
|
||||
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
|
||||
// The focused element went away between the two calls — a stale answer,
|
||||
// not a broken driver. Report it as "nothing focused" so the caller polls
|
||||
// again rather than failing on a race it can simply retry out of.
|
||||
if (isNoSuchElementError(error)) return 'none';
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async back(): Promise<void> {
|
||||
await this.sessionRequest('POST', '/back');
|
||||
}
|
||||
@@ -209,9 +304,10 @@ export class WebDriverClient {
|
||||
): Promise<unknown> {
|
||||
let lastError: unknown;
|
||||
const retryAttempts = overrides?.retryAttempts ?? this.requestPolicy.retryAttempts;
|
||||
const timeoutMs = overrides?.timeoutMs ?? this.requestPolicy.timeoutMs;
|
||||
for (let attempt = 0; attempt <= retryAttempts; attempt += 1) {
|
||||
try {
|
||||
return await this.requestValueOnce(method, path, body);
|
||||
return await this.requestValueOnce(method, path, body, timeoutMs);
|
||||
} catch (error) {
|
||||
lastError = error;
|
||||
if (!isRetriableWebDriverError(error) || attempt >= retryAttempts) {
|
||||
@@ -223,7 +319,12 @@ export class WebDriverClient {
|
||||
throw lastError;
|
||||
}
|
||||
|
||||
private async requestValueOnce(method: string, path: string, body?: unknown): Promise<unknown> {
|
||||
private async requestValueOnce(
|
||||
method: string,
|
||||
path: string,
|
||||
body: unknown,
|
||||
timeoutMs: number,
|
||||
): Promise<unknown> {
|
||||
const response = await fetch(new URL(trimLeadingSlash(path), this.endpoint), {
|
||||
method,
|
||||
headers: {
|
||||
@@ -232,7 +333,7 @@ export class WebDriverClient {
|
||||
...this.headers,
|
||||
},
|
||||
body: body === undefined ? undefined : JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(this.requestPolicy.timeoutMs),
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
});
|
||||
const text = await response.text();
|
||||
const payload = text ? parseJsonResponse(text) : {};
|
||||
@@ -329,6 +430,65 @@ function webdriverError(status: number, payload: unknown): AppError {
|
||||
return new AppError('COMMAND_FAILED', message, { status, response: payload });
|
||||
}
|
||||
|
||||
/**
|
||||
* A route this driver does not implement, as opposed to one that failed.
|
||||
*
|
||||
* Classified from the W3C error code, NOT from the HTTP status: `unknown
|
||||
* command` and `invalid session id` are both 404, so a status test would read a
|
||||
* dead session as a missing feature — exactly the confusion that turns a broken
|
||||
* session into a blind text entry. Only 405/501 are unambiguous enough to stand
|
||||
* on their own. A 5xx, an auth rejection, or a timeout is a real failure.
|
||||
*/
|
||||
const UNIMPLEMENTED_WEBDRIVER_STATUSES = new Set([405, 501]);
|
||||
const UNIMPLEMENTED_WEBDRIVER_ERRORS = new Set(['unknown command', 'unknown method']);
|
||||
|
||||
/**
|
||||
* The W3C element identifier, whose key is the spec's fixed UUID rather than a
|
||||
* readable name. Appium also echoes the legacy `ELEMENT` key; accept either so
|
||||
* the caller works across grid versions.
|
||||
*/
|
||||
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
|
||||
|
||||
function readW3CElementId(value: unknown): string | undefined {
|
||||
if (!value || typeof value !== 'object') return undefined;
|
||||
const record = value as Record<string, unknown>;
|
||||
const id = record[W3C_ELEMENT_KEY] ?? record.ELEMENT;
|
||||
return typeof id === 'string' && id.length > 0 ? id : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* One request's share of a multi-request operation's budget: whatever is left
|
||||
* of it. Floored at zero rather than omitted, so an already-spent budget aborts
|
||||
* the request immediately instead of silently falling back to the client
|
||||
* default and outliving the deadline entirely.
|
||||
*/
|
||||
function requestBudget(deadline: number | undefined): WebDriverRequestOverrides {
|
||||
if (deadline === undefined) return { retryAttempts: 0 };
|
||||
return { retryAttempts: 0, timeoutMs: Math.max(0, deadline - Date.now()) };
|
||||
}
|
||||
|
||||
/** Nothing is focused right now — an expected state, not a driver defect. */
|
||||
function isNoSuchElementError(error: unknown): boolean {
|
||||
if (!(error instanceof AppError)) return false;
|
||||
return readWebDriverErrorCode(error).toLowerCase() === 'no such element';
|
||||
}
|
||||
|
||||
function isUnimplementedWebDriverRoute(error: unknown): boolean {
|
||||
if (!(error instanceof AppError)) return false;
|
||||
const status = error.details?.status;
|
||||
if (typeof status === 'number' && UNIMPLEMENTED_WEBDRIVER_STATUSES.has(status)) return true;
|
||||
return UNIMPLEMENTED_WEBDRIVER_ERRORS.has(readWebDriverErrorCode(error).toLowerCase());
|
||||
}
|
||||
|
||||
function readWebDriverErrorCode(error: AppError): string {
|
||||
const response = error.details?.response;
|
||||
const value =
|
||||
response && typeof response === 'object' ? (response as { value?: unknown }).value : undefined;
|
||||
const code =
|
||||
value && typeof value === 'object' ? (value as { error?: unknown }).error : undefined;
|
||||
return typeof code === 'string' ? code : '';
|
||||
}
|
||||
|
||||
function isRetriableWebDriverError(error: unknown): boolean {
|
||||
if (error instanceof AppError) {
|
||||
const status = error.details?.status;
|
||||
|
||||
@@ -1,10 +1,343 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { test } from 'vitest';
|
||||
import { test, vi } from 'vitest';
|
||||
import { buildGesturePlan } from '@agent-device/contracts/interaction';
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import { createCloudWebDriverCapabilities } from './capabilities.ts';
|
||||
import type { WebDriverClient, W3CActionSequence } from './webdriver-client.ts';
|
||||
import { createWebDriverInteractor } from './webdriver-interactor.ts';
|
||||
|
||||
// #1658: `fill` used to send its keys in the request right after the tap. A
|
||||
// WebView input takes first responder asynchronously, so on a web login form
|
||||
// the keys landed with nothing focused and `fill` still reported success.
|
||||
test('fill withholds keys until the tapped point holds text-entry focus', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
// Nothing focused at first, then the tapped field takes it two polls later.
|
||||
world.activeRects = ['none', 'none', CONTAINS_TAP];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
assert.deepEqual(world.transcript, [
|
||||
'keyboard',
|
||||
'active',
|
||||
'tap',
|
||||
'active',
|
||||
'active',
|
||||
'active',
|
||||
'keys',
|
||||
]);
|
||||
});
|
||||
|
||||
// The case keyboard visibility can never decide: a second fill into a form
|
||||
// whose keyboard is already up. The keyboard says "a field is focused" both
|
||||
// before and after, so only the focused element's own geometry can tell whether
|
||||
// OUR tap moved focus.
|
||||
test('fill accepts a keyboard-up second field once focus lands on the tapped point', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.keyboardShown = [true];
|
||||
world.focusedBeforeTap = PREVIOUS_FIELD;
|
||||
world.activeRects = [CONTAINS_TAP];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
assert.deepEqual(world.transcript, ['keyboard', 'active', 'tap', 'active', 'keys']);
|
||||
});
|
||||
|
||||
// P1, the two-field misdelivery: email filled, keyboard stays up, the password
|
||||
// tap misses. Focus never leaves the email field, so a global `/keys` would
|
||||
// append the password to the EMAIL field while reporting success.
|
||||
test('fill refuses when a keyboard-up tap misses and focus stays on the previous field', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.keyboardShown = [true];
|
||||
world.focusedBeforeTap = PREVIOUS_FIELD;
|
||||
world.activeRects = [PREVIOUS_FIELD];
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.equal(error.code, 'COMMAND_FAILED');
|
||||
assert.match(error.message, /nothing there took text-entry focus/);
|
||||
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
|
||||
return true;
|
||||
});
|
||||
|
||||
// The decisive assertion: no keys reached the field the previous fill focused.
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
// Found on a live AWS Device Farm iPhone 16: tapping Safari's collapsed address
|
||||
// bar expands it into a taller field whose rect no longer covers the tapped
|
||||
// point. A containment-only rule refused a fill that plainly worked, so a tap
|
||||
// that MOVES focus counts even when the focused element has since been re-laid
|
||||
// out.
|
||||
test('fill accepts focus that moved to an element re-laid out away from the tap', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRects = ['none', MOVED_AFTER_FOCUS];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
});
|
||||
|
||||
// The other half of the rule: re-filling the field that is ALREADY focused
|
||||
// moves nothing, so only containment can tell it from a tap that missed.
|
||||
test('fill accepts a re-fill of the already-focused field it tapped', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.focusedBeforeTap = CONTAINS_TAP;
|
||||
world.activeRects = [CONTAINS_TAP];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
});
|
||||
|
||||
test('fill falls back to the keyboard transition when there is no active-element route', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRoute = 'unimplemented';
|
||||
world.keyboardShown = [false, false, true];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'keyboard-shown');
|
||||
assert.deepEqual(world.transcript, [
|
||||
'keyboard',
|
||||
'active',
|
||||
'tap',
|
||||
'active',
|
||||
'keyboard',
|
||||
'keyboard',
|
||||
'keys',
|
||||
]);
|
||||
});
|
||||
|
||||
// Without an active-element route, a keyboard that was ALREADY up witnesses
|
||||
// nothing about the new field — the exact silent misdelivery P1 names.
|
||||
test('fill refuses a keyboard-up tap when the driver cannot report the focused field', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRoute = 'unimplemented';
|
||||
world.keyboardShown = [true];
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.match(error.message, /cannot report which field holds focus/);
|
||||
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
|
||||
return true;
|
||||
});
|
||||
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
// The last path that typed without evidence. It reported ordinary success, and
|
||||
// nothing renders `textEntryReadiness`, so it read to a caller exactly like a
|
||||
// fill that worked — the #1658 false success this change exists to remove.
|
||||
test('fill refuses when the driver implements neither focus route', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRoute = 'unimplemented';
|
||||
world.keyboardRoute = 'unimplemented';
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.equal(error.code, 'COMMAND_FAILED');
|
||||
assert.match(error.message, /reports neither the focused element nor keyboard state/);
|
||||
// A capability gap, not a missed tap: the caller's next move differs.
|
||||
assert.equal(error.details?.reason, 'text_entry_focus_unobservable');
|
||||
assert.match(String(error.details?.hint), /press <target> followed by type <text>/);
|
||||
return true;
|
||||
});
|
||||
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
// The #1658 report is "fill answered Filled N chars while the field kept its
|
||||
// placeholder". Reporting a readiness value nobody renders would preserve that,
|
||||
// so a tap that focuses nothing refuses instead — and sends no keys, leaving
|
||||
// the field untouched rather than half-written.
|
||||
test('fill refuses without typing when nothing ever takes focus', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRects = ['none'];
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.equal(error.code, 'COMMAND_FAILED');
|
||||
assert.match(error.message, /nothing there took text-entry focus/);
|
||||
return true;
|
||||
});
|
||||
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
// A probe that FAILED is not a driver without the feature. Swallowing a dead
|
||||
// session or a grid outage as "unsupported" would degrade it into a blind type.
|
||||
test('fill propagates a failing keyboard probe instead of typing blind', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.keyboardRoute = 'failing';
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.match(error.message, /invalid session id/);
|
||||
return true;
|
||||
});
|
||||
|
||||
// It threw on a pre-tap probe, so the device was never touched at all.
|
||||
assert.equal(world.transcript.includes('tap'), false);
|
||||
});
|
||||
|
||||
// Post-tap the device HAS been touched, so one transient grid error must not
|
||||
// fail a fill the next poll would have satisfied.
|
||||
test('fill tolerates a transient probe failure and still witnesses focus', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeFailuresAfterTap = 2;
|
||||
world.activeRects = [CONTAINS_TAP];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
});
|
||||
|
||||
// ...but a budget that expires without a single answered probe is a broken
|
||||
// session, and must surface as itself rather than as "the tap missed".
|
||||
test('fill rethrows the probe failure when no probe ever answers', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRoute = 'failing';
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.match(error.message, /active element exploded/);
|
||||
return true;
|
||||
});
|
||||
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
// P2: a probe handed the full fixed timeout near the deadline would finish well
|
||||
// after the budget it was meant to respect, so each one is capped by what is
|
||||
// actually left.
|
||||
test('fill caps each probe timeout by the readiness budget remaining', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRects = ['none'];
|
||||
|
||||
await assert.rejects(runFill(world), () => true);
|
||||
|
||||
const timeouts = world.probeTimeouts.filter((t): t is number => t !== undefined);
|
||||
assert.equal(
|
||||
timeouts.every((t) => t <= 1_500),
|
||||
true,
|
||||
);
|
||||
// The budget is 2s and probes start at t=0, so once under 1.5s remain the cap
|
||||
// is the remainder, and the final probe gets exactly one poll interval.
|
||||
assert.equal(timeouts.at(-1), 100);
|
||||
});
|
||||
|
||||
/**
|
||||
* Drives `fill` on a fake clock: the readiness wait is real production time on
|
||||
* a device and must never be real time here (docs/agents/testing.md). Advancing
|
||||
* past the whole readiness budget is safe for every case — once readiness
|
||||
* resolves there are no pending timers left to fire.
|
||||
*/
|
||||
async function runFill(world: ReturnType<typeof createTextEntryWorld>) {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const interactor = createWebDriverInteractor({
|
||||
client: world.client,
|
||||
backend: 'xctest',
|
||||
capabilities: createCloudWebDriverCapabilities({ provider: 'test', platform: 'ios' }),
|
||||
});
|
||||
// Settled-shaped from the start: a rejection that lands while the clock is
|
||||
// being advanced would otherwise be unhandled until the await below.
|
||||
const pending = interactor.fill(12, 24, 'user@example.com').then(
|
||||
(value) => ({ rejected: false, value }) as const,
|
||||
(error: unknown) => ({ rejected: true, error }) as const,
|
||||
);
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
const settled = await pending;
|
||||
if (settled.rejected) throw settled.error;
|
||||
return settled.value;
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Records the request order `fill` produces. The ordering IS the fix: a `keys`
|
||||
* entry that follows `tap` with no keyboard evidence between them is the bug.
|
||||
*/
|
||||
type FocusedElement = {
|
||||
id: string;
|
||||
rect: { x: number; y: number; width: number; height: number };
|
||||
};
|
||||
|
||||
/** `fill` is driven at (12, 24) throughout, so these read as hit and miss. */
|
||||
const CONTAINS_TAP = { id: 'tapped', rect: { x: 0, y: 0, width: 100, height: 100 } } as const;
|
||||
const PREVIOUS_FIELD = {
|
||||
id: 'previous',
|
||||
rect: { x: 200, y: 200, width: 100, height: 100 },
|
||||
} as const;
|
||||
/** Focused by our tap, but re-laid out by focusing so it no longer covers the tap point. */
|
||||
const MOVED_AFTER_FOCUS = {
|
||||
id: 'expanded',
|
||||
rect: { x: 200, y: 600, width: 180, height: 40 },
|
||||
} as const;
|
||||
|
||||
function createTextEntryWorld() {
|
||||
const world = {
|
||||
transcript: [] as string[],
|
||||
probeTimeouts: [] as Array<number | undefined>,
|
||||
keyboardShown: [true] as boolean[],
|
||||
/** How the driver answers each route: normally, not at all, or with a real failure. */
|
||||
keyboardRoute: 'ok' as 'ok' | 'unimplemented' | 'failing',
|
||||
activeRoute: 'ok' as 'ok' | 'unimplemented' | 'failing',
|
||||
/** Successive answers from the active-element route; the last one repeats. */
|
||||
activeRects: ['none'] as Array<'none' | FocusedElement>,
|
||||
/** What held focus when the fill began — a previous fill's field, or nothing. */
|
||||
focusedBeforeTap: 'none' as 'none' | FocusedElement,
|
||||
/** Transient post-tap failures to emit before the route starts answering. */
|
||||
activeFailuresAfterTap: 0,
|
||||
tapped: false,
|
||||
client: undefined as unknown as WebDriverClient,
|
||||
};
|
||||
world.client = {
|
||||
performActions: async () => {
|
||||
world.transcript.push('tap');
|
||||
world.tapped = true;
|
||||
},
|
||||
releaseActions: async () => {},
|
||||
sendKeys: async () => {
|
||||
world.transcript.push('keys');
|
||||
},
|
||||
activeElement: async (timeoutMs?: number) => {
|
||||
world.transcript.push('active');
|
||||
world.probeTimeouts.push(timeoutMs);
|
||||
if (world.activeRoute === 'unimplemented') return 'unsupported' as const;
|
||||
if (world.activeRoute === 'failing') {
|
||||
throw new AppError('COMMAND_FAILED', 'active element exploded', { status: 500 });
|
||||
}
|
||||
// Before the tap the form is in whatever state the previous fill left it.
|
||||
if (!world.tapped) {
|
||||
return world.focusedBeforeTap === 'none'
|
||||
? ('none' as const)
|
||||
: { id: world.focusedBeforeTap.id, rect: world.focusedBeforeTap.rect };
|
||||
}
|
||||
if (world.activeFailuresAfterTap > 0) {
|
||||
world.activeFailuresAfterTap -= 1;
|
||||
throw new AppError('COMMAND_FAILED', 'transient grid blip', { status: 502 });
|
||||
}
|
||||
const next =
|
||||
world.activeRects.length > 1 ? world.activeRects.shift()! : world.activeRects[0]!;
|
||||
return next === 'none' ? ('none' as const) : { id: next.id, rect: next.rect };
|
||||
},
|
||||
isKeyboardShown: async (timeoutMs?: number) => {
|
||||
world.transcript.push('keyboard');
|
||||
world.probeTimeouts.push(timeoutMs);
|
||||
// The real client classifies the wire error; this stands in for its verdict.
|
||||
if (world.keyboardRoute === 'unimplemented') return 'unsupported' as const;
|
||||
if (world.keyboardRoute === 'failing') {
|
||||
// A dead session: 404, same status as an unimplemented route, which is
|
||||
// why the client classifies on the W3C error code instead.
|
||||
throw new AppError('COMMAND_FAILED', 'invalid session id', { status: 404 });
|
||||
}
|
||||
// The last programmed reading is what the keyboard stays at.
|
||||
return world.keyboardShown.length > 1
|
||||
? world.keyboardShown.shift()!
|
||||
: world.keyboardShown[0]!;
|
||||
},
|
||||
} as unknown as WebDriverClient;
|
||||
return world;
|
||||
}
|
||||
|
||||
test('endpoint plans become one timed W3C pointer move', async () => {
|
||||
const performed: W3CActionSequence[][] = [];
|
||||
let released = false;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { DeviceRotation } from '@agent-device/contracts/device';
|
||||
import type {
|
||||
BackMode,
|
||||
CloudTextEntryReadiness,
|
||||
GesturePlan,
|
||||
Interactor,
|
||||
ScreenshotOptions,
|
||||
@@ -24,6 +25,74 @@ import { scrollFrameFromWebDriverSource } from './webdriver-scroll-frame.ts';
|
||||
import { parseWebDriverSource } from './webdriver-source.ts';
|
||||
import { setWebDriverOrientation } from './webdriver-orientation.ts';
|
||||
|
||||
/**
|
||||
* How long a tapped field gets to take text entry focus before `fill` gives up.
|
||||
* Mirrors the Apple runner's `TextEntryTiming.readinessTimeout`
|
||||
* (RunnerTests+TextEntry.swift), which owns this discipline locally.
|
||||
*/
|
||||
const TEXT_ENTRY_READINESS_TIMEOUT_MS = 2_000;
|
||||
const TEXT_ENTRY_READINESS_POLL_MS = 100;
|
||||
/**
|
||||
* One keyboard probe's own transport bound. Well under the readiness budget it
|
||||
* serves, so a single hung probe cannot stretch that budget out toward the
|
||||
* WebDriver client's much longer default request timeout.
|
||||
*/
|
||||
const TEXT_ENTRY_PROBE_TIMEOUT_MS = 1_500;
|
||||
|
||||
/** Global-timer based, so text-entry readiness can be exercised on a fake clock. */
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
/**
|
||||
* Nothing took text-entry focus within the readiness budget, so `sendKeys`
|
||||
* would go to whatever else holds first responder — or nowhere. #1658 is
|
||||
* precisely the report of that being answered with "Filled N chars", so this
|
||||
* fails instead of typing: the caller gets a refusal it must handle, and the
|
||||
* field is left untouched rather than half-written.
|
||||
*/
|
||||
function textEntryFocusNotObservedError(x: number, y: number, detail: string): AppError {
|
||||
return new AppError('COMMAND_FAILED', `fill tapped (${x}, ${y}) but ${detail}`, {
|
||||
reason: 'text_entry_focus_not_observed',
|
||||
x,
|
||||
y,
|
||||
hint: 'The tap most likely missed the field. Re-check the target with snapshot -i and fill the element it reports, rather than retrying the same coordinates.',
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The driver reports neither which element holds focus nor whether the keyboard
|
||||
* is up, so no wait could establish that the tap landed on a text input. A
|
||||
* distinct reason from a tap that missed: nothing about the target is wrong,
|
||||
* the grid simply cannot answer, and the caller's next move is different.
|
||||
*/
|
||||
function textEntryFocusUnobservableError(x: number, y: number): AppError {
|
||||
return new AppError(
|
||||
'COMMAND_FAILED',
|
||||
`fill tapped (${x}, ${y}) but this driver reports neither the focused element nor keyboard state, so the text was not sent`,
|
||||
{
|
||||
reason: 'text_entry_focus_unobservable',
|
||||
x,
|
||||
y,
|
||||
hint: 'This provider cannot confirm a field took focus. Use press <target> followed by type <text> to enter it anyway, accepting that nothing witnesses which field receives the keys.',
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
function rectContains(rect: WebDriverWindowRect, x: number, y: number): boolean {
|
||||
return x >= rect.x && x <= rect.x + rect.width && y >= rect.y && y <= rect.y + rect.height;
|
||||
}
|
||||
|
||||
/**
|
||||
* What text entry looked like before the tap. Both readings are only evidence
|
||||
* as a change, so both must be captured before the device is touched.
|
||||
*/
|
||||
type TextEntryStateBeforeTap = {
|
||||
keyboard: boolean | 'unsupported';
|
||||
/** `undefined` when nothing was focused, or when the driver cannot say. */
|
||||
focusedElementId: string | undefined;
|
||||
};
|
||||
|
||||
export type WebDriverInteractorOptions = {
|
||||
client: WebDriverClient;
|
||||
backend: Extract<SnapshotResult['backend'], 'android' | 'xctest'>;
|
||||
@@ -147,9 +216,17 @@ class WebDriverInteractor implements Interactor {
|
||||
_delayMs?: number,
|
||||
): Promise<Record<string, unknown>> {
|
||||
this.requireSupport('fill');
|
||||
// #1658: both readings are taken BEFORE the tap, because each is only
|
||||
// evidence as a change. A keyboard that was already up says nothing about
|
||||
// which field owns first responder now, and a focused element is only
|
||||
// proof our tap did something if it differs from the one already focused.
|
||||
// Probe failures throw here, before the tap, so a broken session fails a
|
||||
// fill that has not touched the device rather than typing into the dark.
|
||||
const beforeTap = await this.readTextEntryState();
|
||||
await this.tap(x, y);
|
||||
const textEntryReadiness = await this.awaitTextEntryReadiness(beforeTap, x, y);
|
||||
await this.type(text);
|
||||
return { backend: 'webdriver', x, y, text };
|
||||
return { backend: 'webdriver', x, y, text, textEntryReadiness };
|
||||
}
|
||||
|
||||
async scroll(
|
||||
@@ -256,6 +333,129 @@ class WebDriverInteractor implements Interactor {
|
||||
this.unsupported('settings');
|
||||
}
|
||||
|
||||
/**
|
||||
* The cloud twin of the local Apple runner's focus -> readiness -> type
|
||||
* pipeline (RunnerTests+TextEntry.swift). A WebView input does not take first
|
||||
* responder synchronously with the tap: the page has to process the touch and
|
||||
* raise the keyboard. `fill` used to send its keys in the very next request,
|
||||
* so on a slow web login form they landed with nothing focused — reported as
|
||||
* "Filled N chars" while the field kept its placeholder. That is also why
|
||||
* tapping and filling as two separate commands was the reliable workaround:
|
||||
* only the round trip between them gave the field time to focus.
|
||||
*/
|
||||
private async awaitTextEntryReadiness(
|
||||
beforeTap: TextEntryStateBeforeTap,
|
||||
x: number,
|
||||
y: number,
|
||||
): Promise<CloudTextEntryReadiness> {
|
||||
const keyboardBeforeTap = beforeTap.keyboard;
|
||||
const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS;
|
||||
// Strongest evidence first: the focused element itself. Keyboard visibility
|
||||
// can only witness that *a* field took focus, never *which* — which is why
|
||||
// filling a second field in an already-open form could send its keys to the
|
||||
// first one.
|
||||
//
|
||||
// The question is whether OUR tap moved focus, so compare against what held
|
||||
// it before. Identity rather than geometry, because focusing a field can
|
||||
// re-lay it out: tapping Safari's collapsed address bar expands it into a
|
||||
// taller search field that no longer contains the point that was tapped, so
|
||||
// a containment-only rule refuses a fill that plainly worked. Containment
|
||||
// still earns its place as the second half of the test — re-filling the
|
||||
// field that is ALREADY focused moves nothing, and only geometry can tell
|
||||
// that from a tap that missed everything.
|
||||
const focusedBeforeTap = beforeTap.focusedElementId;
|
||||
const focus = await this.pollWithinBudget(deadline, async (timeoutMs) => {
|
||||
const active = await this.client.activeElement(timeoutMs);
|
||||
if (active === 'unsupported') return 'abandon';
|
||||
if (active === 'none') return 'retry';
|
||||
const moved = active.id !== focusedBeforeTap;
|
||||
return moved || rectContains(active.rect, x, y) ? 'satisfied' : 'retry';
|
||||
});
|
||||
if (focus === 'satisfied') return 'focused-element';
|
||||
if (focus === 'expired') {
|
||||
throw textEntryFocusNotObservedError(
|
||||
x,
|
||||
y,
|
||||
'nothing there took text-entry focus, so the text was not sent',
|
||||
);
|
||||
}
|
||||
|
||||
// No active-element route. Fall back to keyboard evidence, which can still
|
||||
// witness focus arriving when the keyboard was down before our tap.
|
||||
if (keyboardBeforeTap === 'unsupported') {
|
||||
// Neither route: nothing this fill can wait for would mean anything. It
|
||||
// used to settle briefly and type, reporting ordinary success — but
|
||||
// nothing renders `textEntryReadiness`, so that read to a caller exactly
|
||||
// like a fill that worked, which is the #1658 false success this whole
|
||||
// change exists to remove. Refusing is the only honest answer, and it
|
||||
// leaves `press` + `type` as the deliberate way to enter text unwitnessed.
|
||||
throw textEntryFocusUnobservableError(x, y);
|
||||
}
|
||||
if (keyboardBeforeTap) {
|
||||
// Keyboard already up and no way to ask which field owns it: every signal
|
||||
// available here describes the PREVIOUS field. Typing anyway is exactly
|
||||
// #1658's silent misdelivery, so refuse instead.
|
||||
throw textEntryFocusNotObservedError(
|
||||
x,
|
||||
y,
|
||||
'the keyboard was already up and this driver cannot report which field holds focus, so the text was not sent',
|
||||
);
|
||||
}
|
||||
const keyboard = await this.pollWithinBudget(deadline, async (timeoutMs) =>
|
||||
(await this.client.isKeyboardShown(timeoutMs)) === true ? 'satisfied' : 'retry',
|
||||
);
|
||||
if (keyboard === 'satisfied') return 'keyboard-shown';
|
||||
throw textEntryFocusNotObservedError(x, y, 'no keyboard appeared, so the text was not sent');
|
||||
}
|
||||
|
||||
/** Both pre-tap readings, taken together so neither is charged an extra tap-to-probe gap. */
|
||||
private async readTextEntryState(): Promise<TextEntryStateBeforeTap> {
|
||||
const [keyboard, active] = await Promise.all([
|
||||
this.client.isKeyboardShown(TEXT_ENTRY_PROBE_TIMEOUT_MS),
|
||||
this.client.activeElement(TEXT_ENTRY_PROBE_TIMEOUT_MS),
|
||||
]);
|
||||
return {
|
||||
keyboard,
|
||||
focusedElementId: active === 'none' || active === 'unsupported' ? undefined : active.id,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Polls until `probe` is satisfied or the readiness budget runs out, bounding
|
||||
* BOTH the probe's own transport timeout and the sleep between attempts by
|
||||
* what remains — a probe handed the full fixed timeout near the deadline
|
||||
* would otherwise finish well after the budget it was meant to respect.
|
||||
*
|
||||
* A probe that throws is tolerated: the loop has further attempts, and one
|
||||
* transient grid error should not fail a fill that would have succeeded. But
|
||||
* a budget that expires without a single answered probe rethrows, so a dead
|
||||
* session or an auth rejection surfaces as itself instead of being reported
|
||||
* as a tap that missed its field.
|
||||
*/
|
||||
private async pollWithinBudget(
|
||||
deadline: number,
|
||||
probe: (timeoutMs: number) => Promise<'satisfied' | 'abandon' | 'retry'>,
|
||||
): Promise<'satisfied' | 'abandon' | 'expired'> {
|
||||
let answered = false;
|
||||
let lastError: unknown;
|
||||
for (;;) {
|
||||
const remaining = deadline - Date.now();
|
||||
if (remaining <= 0) break;
|
||||
try {
|
||||
const verdict = await probe(Math.min(TEXT_ENTRY_PROBE_TIMEOUT_MS, remaining));
|
||||
answered = true;
|
||||
if (verdict !== 'retry') return verdict;
|
||||
} catch (error) {
|
||||
lastError = error;
|
||||
}
|
||||
const left = deadline - Date.now();
|
||||
if (left <= 0) break;
|
||||
await sleep(Math.min(TEXT_ENTRY_READINESS_POLL_MS, left));
|
||||
}
|
||||
if (!answered && lastError !== undefined) throw lastError;
|
||||
return 'expired';
|
||||
}
|
||||
|
||||
private async pointerGesture(name: string, actions: W3CPointerAction[]): Promise<void> {
|
||||
await this.client.performActions([touchPointer(name, actions)]);
|
||||
// Some Appium grids accept W3C actions but reject DELETE /actions. A failed
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import {
|
||||
assertExclusiveScrollDistanceInputs,
|
||||
CLOUD_TEXT_ENTRY_READINESS,
|
||||
getClickButtonValidationError,
|
||||
honoredScrollDurationMs,
|
||||
MAESTRO_NON_HITTABLE_FALLBACK_MESSAGE,
|
||||
@@ -7,6 +8,8 @@ import {
|
||||
parseScrollDirection,
|
||||
resolveClickButton,
|
||||
type ClickButton,
|
||||
type CloudTextEntryReadiness,
|
||||
type FillBackendResult,
|
||||
type Interactor,
|
||||
type RunnerCallOptions,
|
||||
type ScrollCommandOptions,
|
||||
@@ -125,10 +128,28 @@ export async function handleFillCommand(
|
||||
...(result?.maestroNonHittableCoordinateFallbackUsed === true
|
||||
? { maestroNonHittableCoordinateFallbackUsed: true }
|
||||
: {}),
|
||||
...readFillBackendResult(result),
|
||||
...successText(formatTextLengthMessage('Filled', text)),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The interactor result is untrusted; this is the one place a `fill` response
|
||||
* becomes the typed {@link FillBackendResult}. Only the cloud-WebDriver
|
||||
* interactor reports a readiness today, and a value this boundary cannot name
|
||||
* is dropped rather than forwarded.
|
||||
*/
|
||||
function readFillBackendResult(result: Record<string, unknown> | void): FillBackendResult {
|
||||
const readiness = result ? result.textEntryReadiness : undefined;
|
||||
return isCloudTextEntryReadiness(readiness) ? { textEntryReadiness: readiness } : {};
|
||||
}
|
||||
|
||||
function isCloudTextEntryReadiness(value: unknown): value is CloudTextEntryReadiness {
|
||||
return (
|
||||
typeof value === 'string' && (CLOUD_TEXT_ENTRY_READINESS as readonly string[]).includes(value)
|
||||
);
|
||||
}
|
||||
|
||||
async function handleDirectElementSelectorFill(
|
||||
interactor: Interactor,
|
||||
selector: NonNullable<DispatchContext['directElementSelector']>,
|
||||
|
||||
@@ -41,7 +41,13 @@ test('inferAndroidPackageAfterOpen reads foreground package for Android URL open
|
||||
).resolves.toBe('host.exp.exponent');
|
||||
});
|
||||
|
||||
test('provider iOS keeps the known bundle id without local app resolution', async () => {
|
||||
// Opening a bundle id switches the session's app, so the target wins over what
|
||||
// the session tracked before — the same precedence the local path has, where
|
||||
// resolveIosApp returns a dotted target unchanged and never reads the current
|
||||
// id. This case only became reachable once a first open started populating the
|
||||
// id at all (#1658); preferring the stored value left `open com.a` then
|
||||
// `open com.b` reporting com.a for every appBundleId-gated command after it.
|
||||
test('provider iOS adopts a new bundle-id target over the tracked app', async () => {
|
||||
mockIsActiveProviderDevice.mockReturnValue(true);
|
||||
const resolveAndroidPackageForOpen = vi.fn(async () => 'com.example.android');
|
||||
|
||||
@@ -52,6 +58,60 @@ test('provider iOS keeps the known bundle id without local app resolution', asyn
|
||||
resolveAndroidPackageForOpen,
|
||||
);
|
||||
|
||||
expect(bundleId).toBe('com.example.installed');
|
||||
expect(bundleId).toBe('com.example.demo');
|
||||
expect(resolveAndroidPackageForOpen).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test.each([
|
||||
['a deep link', 'myapp://login'],
|
||||
['a web URL', 'https://example.com'],
|
||||
['an app display name', 'Demo App'],
|
||||
['no target', undefined],
|
||||
])('provider iOS keeps the tracked app across %s', async (_label, openTarget) => {
|
||||
mockIsActiveProviderDevice.mockReturnValue(true);
|
||||
|
||||
await expect(
|
||||
resolveSessionAppBundleIdForTarget(
|
||||
IOS_SIMULATOR,
|
||||
openTarget,
|
||||
'com.example.installed',
|
||||
vi.fn(async () => 'com.example.android'),
|
||||
),
|
||||
).resolves.toBe('com.example.installed');
|
||||
});
|
||||
|
||||
// #1658: a first open on a cloud device has no known bundle id, and no local
|
||||
// tooling can look one up. Dropping the target left the session with no app
|
||||
// identity at all, so every appBundleId-gated command refused a live session.
|
||||
test('provider iOS adopts a bundle-id open target when none is known', async () => {
|
||||
mockIsActiveProviderDevice.mockReturnValue(true);
|
||||
const resolveAndroidPackageForOpen = vi.fn(async () => 'com.example.android');
|
||||
|
||||
const bundleId = await resolveSessionAppBundleIdForTarget(
|
||||
IOS_SIMULATOR,
|
||||
'com.example.demo',
|
||||
undefined,
|
||||
resolveAndroidPackageForOpen,
|
||||
);
|
||||
|
||||
expect(bundleId).toBe('com.example.demo');
|
||||
expect(resolveAndroidPackageForOpen).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test.each([
|
||||
['an app display name', 'Demo App'],
|
||||
['a deep link', 'myapp://login'],
|
||||
['a web URL', 'https://example.com'],
|
||||
['no target', undefined],
|
||||
])('provider iOS adopts nothing from %s', async (_label, openTarget) => {
|
||||
mockIsActiveProviderDevice.mockReturnValue(true);
|
||||
|
||||
await expect(
|
||||
resolveSessionAppBundleIdForTarget(
|
||||
IOS_SIMULATOR,
|
||||
openTarget,
|
||||
undefined,
|
||||
vi.fn(async () => undefined),
|
||||
),
|
||||
).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { test, expect, vi, beforeEach } from 'vitest';
|
||||
import { test, expect, vi, afterEach, beforeEach } from 'vitest';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { PNG } from '../../../utils/png.ts';
|
||||
@@ -6,6 +6,8 @@ import { handleSnapshotCommands } from '../snapshot.ts';
|
||||
import { withSessionlessRunnerCleanup } from '../snapshot-session.ts';
|
||||
import { captureSnapshot } from '../snapshot-capture.ts';
|
||||
import { SessionStore } from '../../session-store.ts';
|
||||
import { setActiveProviderDeviceRuntimes } from '../../../provider-device-runtime.ts';
|
||||
import type { ProviderDeviceRuntime } from '@agent-device/contracts/device';
|
||||
import type { DaemonResponse, SessionState } from '../../types.ts';
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import { buildSnapshotSignatures } from '../../android-snapshot-freshness.ts';
|
||||
@@ -104,6 +106,30 @@ const androidDevice: SessionState['device'] = {
|
||||
booted: true,
|
||||
};
|
||||
|
||||
const providerIosDevice: SessionState['device'] = {
|
||||
platform: 'apple',
|
||||
id: 'browserstack:ios:lease-a',
|
||||
name: 'iPhone 16',
|
||||
kind: 'device',
|
||||
target: 'mobile',
|
||||
booted: true,
|
||||
};
|
||||
|
||||
function makeProviderRuntimeOwning(device: SessionState['device']): ProviderDeviceRuntime {
|
||||
return {
|
||||
provider: 'browserstack',
|
||||
leaseLifecycle: {},
|
||||
deviceInventoryProvider: async () => [device],
|
||||
ownsDevice: (candidate) => candidate.id === device.id,
|
||||
getInteractor: () => undefined,
|
||||
shutdown: async () => undefined,
|
||||
};
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
setActiveProviderDeviceRuntimes([]);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
mockDispatch.mockReset();
|
||||
mockDispatch.mockResolvedValue({});
|
||||
@@ -413,6 +439,68 @@ test('snapshot on iOS without a tracked app carries the detected open command as
|
||||
expect(mockBuildIosOpenCommandHint).toHaveBeenCalledWith(iosSimulatorDevice);
|
||||
});
|
||||
|
||||
// #1658: the app-session requirement is the local XCUITest runner's, not the
|
||||
// capture's. A cloud device reads its page source through the provider's own
|
||||
// driver session, and its open path cannot resolve a bundle id locally, so the
|
||||
// guard refused every capture on a live, healthy session.
|
||||
test('snapshot on provider-backed iOS runs without a tracked app', async () => {
|
||||
const sessionStore = makeSessionStore();
|
||||
const sessionName = 'ios-cloud-no-app';
|
||||
sessionStore.set(sessionName, makeSession(sessionName, providerIosDevice));
|
||||
setActiveProviderDeviceRuntimes([makeProviderRuntimeOwning(providerIosDevice)]);
|
||||
mockDispatch.mockResolvedValue({
|
||||
nodes: [{ index: 0, depth: 0, type: 'XCUIElementTypeButton', label: 'Sign in' }],
|
||||
truncated: false,
|
||||
backend: 'xctest',
|
||||
});
|
||||
|
||||
const response = await handleSnapshotCommands({
|
||||
req: {
|
||||
token: 't',
|
||||
session: sessionName,
|
||||
command: 'snapshot',
|
||||
positionals: [],
|
||||
flags: {},
|
||||
},
|
||||
sessionName,
|
||||
logPath: '/tmp/daemon.log',
|
||||
sessionStore,
|
||||
});
|
||||
|
||||
expect(response?.ok).toBe(true);
|
||||
expect(mockDispatch).toHaveBeenCalled();
|
||||
// The bypass has to happen before the hint probe (#1662), which shells out to
|
||||
// simctl and can only ever see local simulators — for a hosted device it is a
|
||||
// guaranteed-useless spawn on what is now a success path.
|
||||
expect(mockBuildIosOpenCommandHint).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('diff on local iOS still requires a tracked app', async () => {
|
||||
const sessionStore = makeSessionStore();
|
||||
const sessionName = 'ios-sim-no-app-diff';
|
||||
sessionStore.set(sessionName, makeSession(sessionName, iosSimulatorDevice));
|
||||
|
||||
const response = await handleSnapshotCommands({
|
||||
req: {
|
||||
token: 't',
|
||||
session: sessionName,
|
||||
command: 'diff',
|
||||
positionals: ['snapshot'],
|
||||
flags: {},
|
||||
},
|
||||
sessionName,
|
||||
logPath: '/tmp/daemon.log',
|
||||
sessionStore,
|
||||
});
|
||||
|
||||
expect(response?.ok).toBe(false);
|
||||
if (response?.ok === false) {
|
||||
expect(response.error.code).toBe('SESSION_NOT_FOUND');
|
||||
expect(response.error.message).toMatch(/iOS diff requires an active app session/i);
|
||||
}
|
||||
expect(mockDispatch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('snapshot on iOS runs when the session tracks an app', async () => {
|
||||
const sessionStore = makeSessionStore();
|
||||
const sessionName = 'ios-sim-app';
|
||||
|
||||
@@ -97,6 +97,23 @@ function shouldPreserveAndroidPackageContext(
|
||||
return device.platform === 'android' && Boolean(openTarget && isDeepLinkTarget(openTarget));
|
||||
}
|
||||
|
||||
/**
|
||||
* #1658: a provider-backed iOS device has no local simctl/devicectl to resolve
|
||||
* an app NAME against, which is why the open path skips resolution there
|
||||
* entirely. Skipping it wholesale also dropped the case where the caller
|
||||
* already spelled the bundle id — `open com.example.app` against a cloud device
|
||||
* left the session with no app identity at all, so every appBundleId-gated
|
||||
* command (snapshot/diff first among them) refused a device that was perfectly
|
||||
* healthy. A dotted, non-deep-link target IS the bundle id under the same
|
||||
* convention the local resolver applies (resolveIosApp returns a dotted target
|
||||
* unchanged), so adopt it without any device round trip.
|
||||
*/
|
||||
function providerIosBundleIdFromOpenTarget(openTarget: string | undefined): string | undefined {
|
||||
const trimmed = openTarget?.trim();
|
||||
if (!trimmed || isDeepLinkTarget(trimmed) || !trimmed.includes('.')) return undefined;
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
export async function resolveSessionAppBundleIdForTarget(
|
||||
device: DeviceInfo,
|
||||
openTarget: string | undefined,
|
||||
@@ -107,7 +124,13 @@ export async function resolveSessionAppBundleIdForTarget(
|
||||
) => Promise<string | undefined>,
|
||||
): Promise<string | undefined> {
|
||||
if (isIosFamily(device) && isActiveProviderDevice(device)) {
|
||||
return currentAppBundleId;
|
||||
// An explicit bundle-id target wins over the session's current app, exactly
|
||||
// as the local path does: resolveIosApp returns a dotted target unchanged
|
||||
// and never consults currentAppBundleId. Preferring the stored id instead
|
||||
// would leave `open com.a` then `open com.b` reporting com.a. Everything
|
||||
// this cannot name — deep links, display names, no target — still falls
|
||||
// back to the known id, which is what the local deep-link branches do too.
|
||||
return providerIosBundleIdFromOpenTarget(openTarget) ?? currentAppBundleId;
|
||||
}
|
||||
return (
|
||||
(await resolveIosBundleIdForOpen(device, openTarget, currentAppBundleId)) ??
|
||||
|
||||
@@ -9,6 +9,7 @@ import { AppError } from '@agent-device/kernel/errors';
|
||||
import type { AgentDeviceBackend, BackendSnapshotResult } from '../backend.ts';
|
||||
import type { CommandSessionRecord } from '../runtime.ts';
|
||||
import { createAgentDevice } from '../runtime.ts';
|
||||
import { isActiveProviderDevice } from '../provider-device-runtime.ts';
|
||||
import { maybeBuildAndroidSnapshotTimeoutFailure } from './android-snapshot-timeout-evidence.ts';
|
||||
import { errorResponse, requireCommandSupported } from './handlers/response.ts';
|
||||
import { buildIosOpenCommandHint } from './ios-app-session-hint.ts';
|
||||
@@ -197,12 +198,25 @@ async function dispatchSnapshotRuntimeCommand(
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The guard exists for the LOCAL Apple path: an iOS capture rides the XCUITest
|
||||
* runner, which must attach to a target app, so a session with no app is a
|
||||
* capture that cannot succeed. A provider-backed (cloud) device captures
|
||||
* through the provider's own driver session instead — the page source needs no
|
||||
* app identity, which is why every other command on that path works without
|
||||
* one. Refusing there turned a healthy live session into an instant
|
||||
* SESSION_NOT_FOUND with no device round trip at all (#1658).
|
||||
*
|
||||
* The provider check belongs in this early return rather than in the hint
|
||||
* below: buildIosOpenCommandHint probes simctl, which can only ever see local
|
||||
* simulators, so running it for a hosted device is a guaranteed-useless spawn.
|
||||
*/
|
||||
async function requireIosAppSessionForSnapshot(
|
||||
command: 'snapshot' | 'diff',
|
||||
session: SessionState | undefined,
|
||||
device: SessionState['device'],
|
||||
): Promise<DaemonResponse | null> {
|
||||
if (!isIosFamily(device) || session?.appBundleId) {
|
||||
if (!isIosFamily(device) || session?.appBundleId || isActiveProviderDevice(device)) {
|
||||
return null;
|
||||
}
|
||||
// An unambiguous environment (one booted simulator, one foreground app)
|
||||
|
||||
@@ -0,0 +1,421 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { test } from 'vitest';
|
||||
import {
|
||||
CLOUD_WEBDRIVER_PROVIDERS,
|
||||
createProviderWebDriver,
|
||||
} from '@agent-device/provider-webdriver';
|
||||
import type { DeviceLease } from '@agent-device/contracts/device';
|
||||
import { createProviderDeviceRuntimeRequestProviders } from '../../../src/provider-device-runtime.ts';
|
||||
import type { DaemonRequest } from '../../../src/daemon/types.ts';
|
||||
import { assertRpcOk } from './assertions.ts';
|
||||
import {
|
||||
createProviderScenarioHarness,
|
||||
withProviderScenarioResource,
|
||||
type ProviderScenarioHarness,
|
||||
} from './harness.ts';
|
||||
import {
|
||||
CloudWebDriverTestServer,
|
||||
type CloudWebDriverHttpCall,
|
||||
type CloudWebDriverTestResponse,
|
||||
cloudWebDriverTestJson,
|
||||
startCloudWebDriverTestServer,
|
||||
type StartedCloudWebDriverTestServer,
|
||||
} from './cloud-webdriver-test-server.ts';
|
||||
|
||||
const WEBDRIVER_PROVIDER = CLOUD_WEBDRIVER_PROVIDERS.browserStack;
|
||||
const CLIENT_VERSION = '0.20.6-test';
|
||||
|
||||
/**
|
||||
* #1658: `open <bundle id>` against a cloud iOS device reported success but
|
||||
* left the session with no app identity, because the provider branch of the
|
||||
* open path skips local app resolution wholesale. Every capture then hit the
|
||||
* local runner's app-session guard and failed instantly with SESSION_NOT_FOUND
|
||||
* — no driver round trip, on a session where press/find/fill/screenshot all
|
||||
* worked.
|
||||
*/
|
||||
test('cloud iOS snapshot captures through the provider session after open', async () => {
|
||||
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
|
||||
const lease = await openCloudIosSession(daemon);
|
||||
|
||||
const response = await daemon.callCommand(
|
||||
'snapshot',
|
||||
[],
|
||||
{ ...leaseFlags(lease.leaseId), snapshotInteractiveOnly: true },
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
|
||||
const data = assertRpcOk<{ nodes?: Array<{ label?: string; identifier?: string }> }>(response);
|
||||
assert.equal(
|
||||
data.nodes?.some((node) => node.identifier === 'email'),
|
||||
true,
|
||||
);
|
||||
// The capture went to the provider's driver, not to a local runner.
|
||||
assert.equal(
|
||||
server.calls.some(
|
||||
(call) => call.method === 'GET' && call.path === '/wd/hub/session/wd-1/source',
|
||||
),
|
||||
true,
|
||||
);
|
||||
// The explicit bundle id is the one identity a cloud open can adopt without
|
||||
// local tooling, so the session keeps it for every appBundleId-gated command.
|
||||
assert.equal(daemon.session()?.appBundleId, 'com.example.demo');
|
||||
});
|
||||
}, 15_000);
|
||||
|
||||
/**
|
||||
* #1658: `fill` tapped and sent its keys back-to-back. A WebView input takes
|
||||
* first responder asynchronously, so the keys arrived with nothing focused and
|
||||
* the command still answered "Filled N chars" — which is why tapping and
|
||||
* filling as two separate commands was the only reliable route.
|
||||
*
|
||||
* The device model, not just the request order, is what makes this a
|
||||
* regression test: the fake drops keys that arrive while the keyboard is down,
|
||||
* exactly as an unfocused field does, so the assertion below is the field's own
|
||||
* value.
|
||||
*/
|
||||
test('cloud iOS fill lands text in the field it tapped', async () => {
|
||||
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
|
||||
const lease = await openCloudIosSession(daemon);
|
||||
// Focus lands a poll late, as a WebView input's does. The keys must wait.
|
||||
server.pollsUntilFocus = 2;
|
||||
|
||||
const response = await daemon.callCommand(
|
||||
'fill',
|
||||
['label=Email', 'user@example.com'],
|
||||
leaseFlags(lease.leaseId),
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
|
||||
const data = assertRpcOk<{ text?: string; textEntryReadiness?: string }>(response);
|
||||
assert.equal(data.text, 'user@example.com');
|
||||
assert.equal(data.textEntryReadiness, 'focused-element');
|
||||
// The keys reached the focused field, not the void.
|
||||
assert.equal(server.fieldValues.email, 'user@example.com');
|
||||
// Both pre-tap readings, the tap, then polling until focus lands. The keys
|
||||
// come last — that ordering IS the fix.
|
||||
assert.deepEqual(textEntryTranscript(server), [
|
||||
'keyboard',
|
||||
'active',
|
||||
'tap',
|
||||
'active',
|
||||
'active',
|
||||
'keys',
|
||||
]);
|
||||
|
||||
// And the device agrees: the field reports the typed value back.
|
||||
const snapshot = await daemon.callCommand(
|
||||
'snapshot',
|
||||
[],
|
||||
{ ...leaseFlags(lease.leaseId), snapshotInteractiveOnly: true },
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
const nodes = assertRpcOk<{ nodes?: Array<{ identifier?: string; value?: string }> }>(
|
||||
snapshot,
|
||||
).nodes;
|
||||
assert.equal(nodes?.find((node) => node.identifier === 'email')?.value, 'user@example.com');
|
||||
});
|
||||
}, 15_000);
|
||||
|
||||
/**
|
||||
* #1658 again, from the other side: when the tap raises no keyboard at all,
|
||||
* nothing focused a field and `sendKeys` would go nowhere. Answering that with
|
||||
* "Filled N chars" is the original bug, so the fill refuses — and sends no keys,
|
||||
* leaving the field untouched rather than half-written.
|
||||
*/
|
||||
test('cloud iOS fill refuses, without typing, when nothing takes focus', async () => {
|
||||
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
|
||||
const lease = await openCloudIosSession(daemon);
|
||||
server.pollsUntilFocus = Number.POSITIVE_INFINITY;
|
||||
|
||||
const response = await daemon.callCommand(
|
||||
'fill',
|
||||
['label=Email', 'user@example.com'],
|
||||
leaseFlags(lease.leaseId),
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
|
||||
const error = response.json.error;
|
||||
assert.ok(error, 'expected fill to refuse');
|
||||
assert.match(error.message ?? '', /nothing there took text-entry focus/);
|
||||
assert.equal(error.data?.details?.reason, 'text_entry_focus_not_observed');
|
||||
assert.equal(server.fieldValues.email, '');
|
||||
assert.equal(textEntryTranscript(server).includes('keys'), false);
|
||||
});
|
||||
}, 15_000);
|
||||
|
||||
/**
|
||||
* The misdelivery keyboard visibility can never catch (#1658 P1). The form is
|
||||
* already filled and focused — keyboard up — and the second fill's tap misses
|
||||
* the password field. Focus therefore never leaves `email`, so a `POST /keys`
|
||||
* here would append the password to the EMAIL field and every request would
|
||||
* still return 200. The fill must refuse before sending anything.
|
||||
*/
|
||||
test('cloud iOS fill sends no keys when a keyboard-up tap misses the second field', async () => {
|
||||
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
|
||||
const lease = await openCloudIosSession(daemon);
|
||||
server.focusField('email', 'user@example.com');
|
||||
// Focus never moves, reproducing a tap that landed outside both fields.
|
||||
server.pollsUntilFocus = Number.POSITIVE_INFINITY;
|
||||
|
||||
const response = await daemon.callCommand(
|
||||
'fill',
|
||||
['label=Password', 'hunter2'],
|
||||
leaseFlags(lease.leaseId),
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
|
||||
assert.ok(response.json.error, 'expected fill to refuse');
|
||||
assert.equal(response.json.error.data?.details?.reason, 'text_entry_focus_not_observed');
|
||||
// The decisive assertion: the previous field is untouched and the password
|
||||
// never reached it.
|
||||
assert.equal(server.fieldValues.email, 'user@example.com');
|
||||
assert.equal(server.fieldValues.password, '');
|
||||
assert.equal(textEntryTranscript(server).includes('keys'), false);
|
||||
});
|
||||
}, 15_000);
|
||||
|
||||
/** The tap/keys/keyboard requests in the order the driver received them. */
|
||||
function textEntryTranscript(server: FakeIosWebDriverServer): string[] {
|
||||
const labels: Record<string, string> = {
|
||||
'POST /wd/hub/session/wd-1/actions': 'tap',
|
||||
'POST /wd/hub/session/wd-1/keys': 'keys',
|
||||
'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': 'keyboard',
|
||||
'GET /wd/hub/session/wd-1/element/active': 'active',
|
||||
};
|
||||
return server.calls
|
||||
.map((call) => labels[`${call.method} ${call.path}`])
|
||||
.filter((label): label is string => label !== undefined);
|
||||
}
|
||||
|
||||
async function createCloudIosWorld() {
|
||||
const server = await FakeIosWebDriverServer.start();
|
||||
const providerWebDriver = createProviderWebDriver({
|
||||
clientVersion: CLIENT_VERSION,
|
||||
runHostCommand: async () => {
|
||||
throw new Error('BrowserStack scenario must not run host commands');
|
||||
},
|
||||
});
|
||||
const runtimes = providerWebDriver.createDefaultRuntimes({
|
||||
BROWSERSTACK_USERNAME: 'browser-user',
|
||||
BROWSERSTACK_ACCESS_KEY: 'browser-key',
|
||||
BROWSERSTACK_WEBDRIVER_ENDPOINT: `${server.url}/wd/hub/`,
|
||||
BROWSERSTACK_SESSION_DETAILS_ENDPOINT: `${server.url}/app-automate/sessions`,
|
||||
});
|
||||
const providers = createProviderDeviceRuntimeRequestProviders(runtimes);
|
||||
const daemon = await createProviderScenarioHarness({
|
||||
...providers,
|
||||
deviceInventoryProvider: providers.deviceInventoryProvider!,
|
||||
});
|
||||
return {
|
||||
daemon,
|
||||
server,
|
||||
close: async () => {
|
||||
await Promise.allSettled(runtimes.map(async (runtime) => await runtime.shutdown()));
|
||||
await daemon.close();
|
||||
await server.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Allocates the provider lease and opens the app by bundle id, as `connect` + `open` do. */
|
||||
async function openCloudIosSession(daemon: ProviderScenarioHarness): Promise<DeviceLease> {
|
||||
const allocate = await daemon.callCommand('lease_allocate', [], leaseFlags(), {
|
||||
meta: leaseMeta(),
|
||||
});
|
||||
const { lease } = assertRpcOk<{ lease: DeviceLease }>(allocate);
|
||||
const open = await daemon.callCommand('open', ['com.example.demo'], leaseFlags(lease.leaseId), {
|
||||
meta: leaseMeta(lease.leaseId),
|
||||
});
|
||||
assertRpcOk(open);
|
||||
return lease;
|
||||
}
|
||||
|
||||
/**
|
||||
* A device model, not just a request recorder: the field takes focus a beat
|
||||
* after the tap (as a WebView input does), the keyboard follows focus, and
|
||||
* `POST /keys` — which the real driver routes to whatever holds first responder
|
||||
* — writes the field only while it is focused, and silently succeeds otherwise.
|
||||
* That last part is the #1658 bug reproduced faithfully: keys sent too early
|
||||
* are accepted by the grid and dropped by the device.
|
||||
*/
|
||||
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
|
||||
/** Two stacked inputs, as a login form has. Geometry is what decides focus. */
|
||||
const FIELDS = {
|
||||
email: { x: 20, y: 300, width: 350, height: 44 },
|
||||
password: { x: 20, y: 380, width: 350, height: 44 },
|
||||
} as const;
|
||||
type FieldName = keyof typeof FIELDS;
|
||||
|
||||
class FakeIosWebDriverServer extends CloudWebDriverTestServer {
|
||||
/** How many probes after a tap before focus settles. Infinity = focus never moves. */
|
||||
pollsUntilFocus = 1;
|
||||
fieldValues: Record<FieldName, string> = { email: '', password: '' };
|
||||
focused: FieldName | undefined;
|
||||
|
||||
private pollsRemaining = Number.POSITIVE_INFINITY;
|
||||
private pendingFocus: FieldName | undefined;
|
||||
|
||||
static async start(): Promise<StartedCloudWebDriverTestServer<FakeIosWebDriverServer>> {
|
||||
return await startCloudWebDriverTestServer(new FakeIosWebDriverServer());
|
||||
}
|
||||
|
||||
/** Puts the form in the state a completed first fill leaves it in. */
|
||||
focusField(field: FieldName, value: string): void {
|
||||
this.focused = field;
|
||||
this.fieldValues[field] = value;
|
||||
this.pollsRemaining = 0;
|
||||
this.pendingFocus = field;
|
||||
}
|
||||
|
||||
protected respond(call: CloudWebDriverHttpCall) {
|
||||
const rectMatch = /^\/wd\/hub\/session\/wd-1\/element\/([^/]+)\/rect$/.exec(call.path);
|
||||
if (call.method === 'GET' && rectMatch) return this.elementRect(rectMatch[1]!);
|
||||
const route = this.routes[`${call.method} ${call.path}`];
|
||||
return route ? route() : cloudWebDriverTestJson({ value: null });
|
||||
}
|
||||
|
||||
private get routes(): Record<string, () => CloudWebDriverTestResponse> {
|
||||
return {
|
||||
'POST /wd/hub/session': () =>
|
||||
cloudWebDriverTestJson({
|
||||
value: { sessionId: 'wd-1', capabilities: { platformName: 'iOS' } },
|
||||
}),
|
||||
'GET /wd/hub/session/wd-1/source': () => cloudWebDriverTestJson({ value: this.source() }),
|
||||
'GET /wd/hub/session/wd-1/window/rect': () =>
|
||||
cloudWebDriverTestJson({ value: { x: 0, y: 0, width: 390, height: 844 } }),
|
||||
'POST /wd/hub/session/wd-1/actions': () => this.tap(),
|
||||
'POST /wd/hub/session/wd-1/keys': () => this.sendKeys(),
|
||||
'GET /wd/hub/session/wd-1/element/active': () => this.activeElement(),
|
||||
'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': () =>
|
||||
cloudWebDriverTestJson({ value: this.keyboardShown() }),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A tap moves focus only if it actually lands inside a field. A miss leaves
|
||||
* the previous field focused — which is exactly how keys can reach the wrong
|
||||
* one while every request still succeeds.
|
||||
*/
|
||||
private tap(): CloudWebDriverTestResponse {
|
||||
const point = this.lastTapPoint();
|
||||
const hit =
|
||||
point && (Object.keys(FIELDS) as FieldName[]).find((n) => contains(FIELDS[n], point));
|
||||
this.pendingFocus = hit ?? this.focused;
|
||||
this.pollsRemaining = hit ? this.pollsUntilFocus : Number.POSITIVE_INFINITY;
|
||||
if (hit) this.focused = undefined;
|
||||
return cloudWebDriverTestJson({ value: null });
|
||||
}
|
||||
|
||||
/** Focus is not instant: it settles only after the configured number of probes. */
|
||||
private settle(): void {
|
||||
if (this.pollsRemaining <= 0) return;
|
||||
this.pollsRemaining -= 1;
|
||||
if (this.pollsRemaining <= 0) this.focused = this.pendingFocus;
|
||||
}
|
||||
|
||||
private activeElement(): CloudWebDriverTestResponse {
|
||||
this.settle();
|
||||
if (!this.focused) {
|
||||
return cloudWebDriverTestJson(
|
||||
{ value: { error: 'no such element', message: 'nothing is focused' } },
|
||||
404,
|
||||
);
|
||||
}
|
||||
return cloudWebDriverTestJson({ value: { [W3C_ELEMENT_KEY]: this.focused } });
|
||||
}
|
||||
|
||||
private elementRect(elementId: string): CloudWebDriverTestResponse {
|
||||
const rect = FIELDS[elementId as FieldName];
|
||||
if (!rect) {
|
||||
return cloudWebDriverTestJson(
|
||||
{ value: { error: 'no such element', message: `unknown element ${elementId}` } },
|
||||
404,
|
||||
);
|
||||
}
|
||||
return cloudWebDriverTestJson({ value: rect });
|
||||
}
|
||||
|
||||
private keyboardShown(): boolean {
|
||||
this.settle();
|
||||
return this.focused !== undefined;
|
||||
}
|
||||
|
||||
private sendKeys(): CloudWebDriverTestResponse {
|
||||
// Unfocused keys are accepted by the grid and land nowhere; keys sent while
|
||||
// the WRONG field holds focus land in that field. Both are the bug's shape.
|
||||
const text = this.lastSentKeys();
|
||||
if (this.focused && text !== undefined) this.fieldValues[this.focused] = text;
|
||||
return cloudWebDriverTestJson({ value: null });
|
||||
}
|
||||
|
||||
private lastTapPoint(): { x: number; y: number } | undefined {
|
||||
const body = this.calls.at(-1)?.body as { actions?: unknown } | undefined;
|
||||
const sequences = Array.isArray(body?.actions) ? body.actions : [];
|
||||
for (const sequence of sequences) {
|
||||
const actions = (sequence as { actions?: unknown }).actions;
|
||||
if (!Array.isArray(actions)) continue;
|
||||
const move = actions.find((a) => (a as { type?: string }).type === 'pointerMove') as
|
||||
| { x?: unknown; y?: unknown }
|
||||
| undefined;
|
||||
if (typeof move?.x === 'number' && typeof move.y === 'number') {
|
||||
return { x: move.x, y: move.y };
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
private lastSentKeys(): string | undefined {
|
||||
const body = this.calls.at(-1)?.body as { value?: unknown } | undefined;
|
||||
return Array.isArray(body?.value) ? body.value.join('') : undefined;
|
||||
}
|
||||
|
||||
private source(): string {
|
||||
const field = (name: FieldName, label: string) =>
|
||||
`<XCUIElementTypeTextField name="${name}" label="${label}" value="${this.fieldValues[name]}" ` +
|
||||
`x="${FIELDS[name].x}" y="${FIELDS[name].y}" width="${FIELDS[name].width}" ` +
|
||||
`height="${FIELDS[name].height}" visible="true" enabled="true" />`;
|
||||
return (
|
||||
'<XCUIElementTypeApplication name="Demo" x="0" y="0" width="390" height="844" visible="true">' +
|
||||
field('email', 'Email') +
|
||||
field('password', 'Password') +
|
||||
'</XCUIElementTypeApplication>'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function contains(
|
||||
rect: { x: number; y: number; width: number; height: number },
|
||||
p: { x: number; y: number },
|
||||
): boolean {
|
||||
return (
|
||||
p.x >= rect.x && p.x <= rect.x + rect.width && p.y >= rect.y && p.y <= rect.y + rect.height
|
||||
);
|
||||
}
|
||||
|
||||
function leaseFlags(leaseId?: string): DaemonRequest['flags'] {
|
||||
return {
|
||||
platform: 'ios',
|
||||
tenant: 'team-a',
|
||||
runId: 'run-a',
|
||||
leaseId,
|
||||
leaseProvider: WEBDRIVER_PROVIDER,
|
||||
device: 'iPhone 16',
|
||||
providerApp: 'bs://app-id',
|
||||
providerOsVersion: '18',
|
||||
providerProject: 'agent-device',
|
||||
providerBuild: 'run-a',
|
||||
providerSessionName: leaseId,
|
||||
};
|
||||
}
|
||||
|
||||
function leaseMeta(leaseId?: string): DaemonRequest['meta'] {
|
||||
return {
|
||||
tenantId: 'team-a',
|
||||
runId: 'run-a',
|
||||
leaseId,
|
||||
leaseBackend: 'ios-instance',
|
||||
leaseProvider: WEBDRIVER_PROVIDER,
|
||||
deviceKey: 'webdriver-ios-a',
|
||||
clientId: 'client-a',
|
||||
};
|
||||
}
|
||||
@@ -411,6 +411,10 @@ function assertWebDriverCalls(calls: readonly CloudWebDriverHttpCall[], leaseId:
|
||||
}
|
||||
}
|
||||
|
||||
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
|
||||
/** Spans the viewport, so whatever coordinate a scenario fills lands inside it. */
|
||||
const FOCUSED_FIELD_RECT = { x: 0, y: 0, width: 1080, height: 1920 };
|
||||
|
||||
class FakeWebDriverServer extends CloudWebDriverTestServer {
|
||||
artifactFailuresRemaining = 0;
|
||||
sessionDeleteFailuresRemaining = 0;
|
||||
@@ -428,6 +432,9 @@ class FakeWebDriverServer extends CloudWebDriverTestServer {
|
||||
protected respond(call: CloudWebDriverHttpCall) {
|
||||
// Routes are a table rather than a switch: every added endpoint would otherwise widen one
|
||||
// function's branch count, and this fake grows an endpoint per provider feature.
|
||||
if (call.method === 'GET' && /\/element\/[^/]+\/rect$/.test(call.path)) {
|
||||
return cloudWebDriverTestJson({ value: FOCUSED_FIELD_RECT });
|
||||
}
|
||||
const route = this.routes[`${call.method} ${call.path}`];
|
||||
return route ? route() : cloudWebDriverTestJson({ value: null });
|
||||
}
|
||||
@@ -448,6 +455,15 @@ class FakeWebDriverServer extends CloudWebDriverTestServer {
|
||||
{ value: { message: 'The requested resource could not be found.' } },
|
||||
500,
|
||||
),
|
||||
// This fake stands in for a conformant driver, so it answers the focus
|
||||
// probes `fill` witnesses with. Text-entry semantics themselves live in
|
||||
// cloud-webdriver-ios-text-entry.test.ts, which models focus properly;
|
||||
// here the field is simply always focused so the facade wiring is what
|
||||
// the scenario exercises.
|
||||
'GET /wd/hub/session/wd-1/element/active': () =>
|
||||
cloudWebDriverTestJson({ value: { [W3C_ELEMENT_KEY]: 'focused-field' } }),
|
||||
'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': () =>
|
||||
cloudWebDriverTestJson({ value: true }),
|
||||
'POST /wd/hub/session/wd-1/rotation': () => this.rotationResponse(),
|
||||
'POST /wd/hub/session/wd-1/orientation': () => cloudWebDriverTestJson({ value: null }),
|
||||
'DELETE /wd/hub/session/wd-1': () => this.deleteSessionResponse(),
|
||||
|
||||
@@ -355,3 +355,7 @@ BrowserStack can return session video, Appium logs, device logs, dashboard URL,
|
||||
- If AWS connect fails, use its reported `aws devicefarm get-*` error to check the credential chain, ARN, region, resource platform, or upload readiness. Allocation has not happened yet.
|
||||
- If Limrun connect fails, check `LIMRUN_API_KEY` and the optional `LIMRUN_REGION`. A successful connect verifies the selected instance service without creating an instance.
|
||||
- If artifact lookup is pending immediately after `close`, retry `agent-device artifacts --json`. Some providers finalize video/log URLs asynchronously after the hosted session stops.
|
||||
- On hosted WebDriver sessions (BrowserStack, AWS Device Farm), `fill` waits until the field it tapped actually holds text entry focus before sending its keys, and reports what it observed as `textEntryReadiness`: `focused-element` (the tap moved focus, or landed inside the element already focused — the strongest signal, and the only one that identifies *which* field took focus) or `keyboard-shown` (the driver has no active-element route, but the keyboard rose from hidden after the tap). Both mean focus was witnessed before any key was sent; `fill` never types without evidence.
|
||||
- If focus cannot be witnessed, `fill` **fails** and sends no keys, rather than reporting a filled field that never received the text.
|
||||
- `text_entry_focus_not_observed` — the tap focused nothing, or the keyboard was already up on a driver that cannot name the focused field (there, unwitnessed keys would land in the field the previous fill focused). This almost always means the tap missed: re-check the target with `snapshot -i` and fill the element it reports instead of retrying the same coordinates.
|
||||
- `text_entry_focus_unobservable` — the driver reports neither the focused element nor keyboard state, so nothing could establish focus. Use `press <target>` followed by `type <text>` to enter the text anyway, accepting that nothing witnesses which field receives the keys.
|
||||
|
||||
Reference in New Issue
Block a user