From 3a9aceb935b0ed81888c02a266e092a51261c364 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 7 Aug 2026 14:06:56 +0200 Subject: [PATCH] fix(cloud): witness the tapped field's focus before a cloud fill types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of cc23f2b found two ways a fill could still report success without evidence that OUR tap focused the field it was aimed at. P1. `settled-keyboard-up` and `settled-unknown` both typed and returned normal success. Keyboard visibility can only witness that *a* field took focus, never *which*: filling a second field in an already-open form reads the same before and after, so a missed tap left the first field focused and `POST /keys` — which the driver routes to whatever holds first responder — appended to it while every request returned 200. Failing those closed outright would have broken ordinary multi-field form fills, which do work: a live AWS Device Farm run types both fields of a WebView login correctly. So witness focus properly instead. W3C `GET /element/active` answers the question keyboard visibility cannot — is the thing focused now the thing I tapped — and answers it whether or not the keyboard was already up. That becomes the primary signal (`focused-element`); the keyboard transition stays as the fallback for drivers without the route, and a keyboard already up on such a driver now refuses rather than typing. The test is identity, not geometry. Containment of the tap point looks like the obvious rule and is wrong: focusing a field can re-lay it out. On a live iPhone 16, tapping Safari's collapsed address bar expands it into a taller field that no longer covers the tapped point, and a containment-only rule refused a fill that plainly worked. So a tap that MOVES focus counts, with containment as the second half of the test — re-filling the already-focused field moves nothing, and only geometry tells that from a tap that missed. Both readings are taken before the tap, since each is evidence only as a change. P2. The 2s budget bounded the loop but not the calls inside it: every probe got a fixed 1500ms, so one begun near the deadline finished well past it. Both the probe timeout and the sleep are now capped by the remaining budget. Also fixes a related escape the review did not name: the poll loop had no catch, so one transient grid error aborted a fill the next poll would have satisfied. Probe failures are now tolerated within the budget, but a budget that expires without a single answered probe rethrows, so a dead session surfaces as itself rather than as "the tap missed". The provider scenario gains the two-field case the review asked for: it begins keyboard-up with the email field focused, misses the password tap, and asserts no keys reach the email field. Verified on AWS Device Farm iPhone 16 / iOS 18.0 at this exact tree: address bar (the re-layout case) and both WebView login fields all report `focused-element`, the second with the keyboard already up, and the device reads back `tomsmith` and a 20-character password. --- CHANGELOG.md | 2 +- packages/contracts/src/interactor-types.ts | 34 +-- .../src/webdriver-client.test.ts | 114 ++++++++- .../src/webdriver-client.ts | 78 ++++++ .../src/webdriver-interactor.test.ts | 224 +++++++++++++++--- .../src/webdriver-interactor.ts | 178 ++++++++++---- .../cloud-webdriver-ios-text-entry.test.ts | 177 ++++++++++++-- website/docs/docs/device-clouds.md | 4 +- 8 files changed, 695 insertions(+), 116 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a8d7f3b7..287470354 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ - `agent-device help workflow` is now a compact ~8KB card instead of a ~41KB dump; the same depth still exists, split into `help scripting` (save-script, secret-safe fills, batch JSON, replay divergence/repair, recording) and `help gestures` (multi-touch shapes and platform quirks), plus a few paragraphs folded into the topics that already owned the subject (`help debugging`, `help physical-device`, `help validate`). Every `help ` first line is now `agent-device — ` so an agent can read the installed version from its mandatory first help read instead of a separate `agent-device --version` call. - Cloud iOS (BrowserStack, AWS Device Farm): `snapshot` and `diff` no longer fail with `SESSION_NOT_FOUND` on a live provider session (#1658). The app-session guard they ran belongs to the local XCUITest runner, which must attach to a target app; a cloud capture reads the provider's own driver session and needs no app identity, so the guard now applies to local Apple targets only. Relatedly, a cloud iOS `open com.example.app` now records that bundle id on the session — the provider path skips local app resolution (no simctl/devicectl reaches a hosted device), and used to drop an explicitly spelled bundle id along with it, leaving the session with no app identity at all. Opening a second bundle id replaces the first, matching the local path, where an explicitly spelled target always wins over the session's current app; deep links, display names, and bare `open` still keep the app already tracked. -- Cloud `fill` (BrowserStack, AWS Device Farm) waits for the tapped field to take text-entry focus before sending its keys, instead of dispatching both in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The response now discloses `textEntryReadiness`: `keyboard-shown`, `settled-keyboard-up` (back-to-back fills into one form, where keyboard visibility cannot witness the new field taking focus, so the fill waits out the same readiness budget the Apple runner spends there), or `settled-unknown` (the driver implements no keyboard route at all). Breaking: when the tap raises no keyboard within that budget, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. A driver that cannot report keyboard state at all is unaffected (it blind-settles as before); only a positively classified unimplemented route counts as such, so a dead session, an auth rejection, or a grid outage now surfaces instead of degrading into a blind text entry. +- Cloud `fill` (BrowserStack, AWS Device Farm) now witnesses that the field it tapped actually holds text-entry focus before sending its keys, instead of dispatching tap and keys in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The witness is the focused element's own geometry: `fill` polls the active element and proceeds only once it contains the point it tapped, which is the one signal that identifies *which* field took focus. Keyboard visibility cannot — it reads the same before and after a second fill into an already-open form, so it could not tell a focused password field from the email field the previous fill left focused. The response discloses `textEntryReadiness`: `focused-element`, `keyboard-shown` (no active-element route, but the keyboard rose from hidden after the tap), or `settled-unknown` (the driver implements neither route, the only path that types without evidence). Breaking: when focus cannot be witnessed, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. That covers a tap that focused nothing and a keyboard already up on a driver that cannot name the focused field. Only a positively classified unimplemented route counts as unsupported, so a dead session, an auth rejection, or a grid outage surfaces instead of degrading into a blind text entry. - `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands. - Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags. - `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied. diff --git a/packages/contracts/src/interactor-types.ts b/packages/contracts/src/interactor-types.ts index a80d9e5d8..13bd15c48 100644 --- a/packages/contracts/src/interactor-types.ts +++ b/packages/contracts/src/interactor-types.ts @@ -102,29 +102,35 @@ export type TypeTextBackendResult = { * What a cloud-WebDriver `fill` could establish about its target taking text * entry focus before it sent the keys (#1658). The local Apple runner owns this * discipline in Swift and reports it as a `textEntryRoute`; the cloud path has - * no runner, only the driver's keyboard-visibility signal, so it names what it - * observed instead of which channel it typed through: + * no runner, only what the driver will tell it, so it names what it observed + * instead of which channel it typed through. Listed strongest evidence first: * - * - `keyboard-shown`: the software keyboard went from hidden to shown after our - * tap — positive evidence the tap moved focus to a text input. - * - `settled-keyboard-up`: the keyboard was ALREADY up (back-to-back fills into - * one form), so its visibility says nothing about the NEW field. Entry waited - * the full readiness budget, the same answer the Apple runner gives here. - * - `settled-unknown`: the driver implements no keyboard route at all, so - * nothing could be waited for. Entry followed a short blind settle. + * - `focused-element`: the element holding text-entry focus after the tap + * contains the tapped point — positive evidence that THIS field, not merely + * some field, took focus. + * - `keyboard-shown`: no active-element route, but the software keyboard went + * from hidden to shown after our tap. Witnesses that focus arrived somewhere, + * which only means our field when the keyboard was down beforehand. + * - `settled-unknown`: the driver implements neither an active-element nor a + * keyboard route, so nothing could be waited for. Entry followed a short + * blind settle. This is the only value that describes a fill with no evidence + * behind it, and it takes two positively classified unsupported answers to + * reach. * - * Every value describes a fill that DID send its keys. The fourth outcome — a - * tap that raised no keyboard at all — is deliberately absent: reporting it - * here would recreate #1658's silent success, so the interactor throws - * (`text_entry_focus_not_observed`) without typing instead. + * Every value describes a fill that DID send its keys. Outcomes where focus + * could not be witnessed are deliberately absent: reporting one here would + * recreate #1658's silent success, so the interactor throws + * (`text_entry_focus_not_observed`) without typing instead. That covers a tap + * that focused nothing, and a keyboard already up on a driver that cannot say + * which field owns it — the case where keys would land in the PREVIOUS field. * * Closed set: the cloud interactor is its only producer and the boundary that * narrows it (readFillBackendResult, src/core/dispatch-interactions.ts) drops a * value it cannot name. */ export const CLOUD_TEXT_ENTRY_READINESS = [ + 'focused-element', 'keyboard-shown', - 'settled-keyboard-up', 'settled-unknown', ] as const; diff --git a/packages/provider-webdriver/src/webdriver-client.test.ts b/packages/provider-webdriver/src/webdriver-client.test.ts index 144444c00..3258987bb 100644 --- a/packages/provider-webdriver/src/webdriver-client.test.ts +++ b/packages/provider-webdriver/src/webdriver-client.test.ts @@ -74,7 +74,119 @@ test('is_keyboard_shown honors a caller timeout shorter than the client default' }); }); -/** A client with a live session id, so `isKeyboardShown` reaches the session route. */ +// The focused element is the only signal that can say WHICH field took focus, +// so `fill` fails closed when it is unavailable. Each wire shape it +// distinguishes is pinned here. +test('activeElement returns the focused element id and rect', async () => { + const client = await connectedClient( + webDriverRoutes({ + '/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } }, + '/rect': { body: { value: { x: 20, y: 300, width: 350, height: 44 } } }, + }), + ); + + assert.deepEqual(await client.activeElement(), { + id: 'el-1', + rect: { x: 20, y: 300, width: 350, height: 44 }, + }); +}); + +// A form with nothing focused yet is an expected state, not a broken driver — +// the caller must keep polling rather than give up. +test('activeElement reports nothing focused as none', async () => { + const client = await connectedClient( + webDriverRoutes({ + '/element/active': { + status: 404, + body: { value: { error: 'no such element', message: 'nothing focused' } }, + }, + }), + ); + + assert.equal(await client.activeElement(), 'none'); +}); + +// Focus moved between the two round trips. A race the caller can simply retry +// out of, not a failure worth ending the fill on. +test('activeElement reports an element that vanished before its rect as none', async () => { + const client = await connectedClient( + webDriverRoutes({ + '/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } }, + '/rect': { + status: 404, + body: { value: { error: 'no such element', message: 'stale element' } }, + }, + }), + ); + + assert.equal(await client.activeElement(), 'none'); +}); + +// A grid that answers 200/null is not implementing the route: the spec reports +// "nothing focused" as a `no such element` error. Reading null as "nothing +// focused" would fail every fill on such a grid instead of falling back. +test('activeElement reports a non-element answer as unsupported', async () => { + const client = await connectedClient( + webDriverRoutes({ '/element/active': { body: { value: null } } }), + ); + + assert.equal(await client.activeElement(), 'unsupported'); +}); + +test('activeElement reports an unimplemented route as unsupported', async () => { + const client = await connectedClient( + webDriverRoutes({ + '/element/active': { + status: 404, + body: { value: { error: 'unknown command', message: 'Unknown command' } }, + }, + }), + ); + + assert.equal(await client.activeElement(), 'unsupported'); +}); + +// Same discipline as the keyboard route: a dead session shares the 404 and must +// never be mistaken for a driver that simply lacks the feature. +test('activeElement propagates a dead session, which shares the 404', async () => { + const client = await connectedClient( + webDriverRoutes({ + '/element/active': { + status: 404, + body: { + value: { error: 'invalid session id', message: 'A session is either terminated' }, + }, + }, + }), + ); + + await assert.rejects(client.activeElement(), (error: AppError) => { + assert.match(error.message, /terminated/); + return true; + }); +}); + +const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf'; + +/** Dispatches by URL suffix, since `activeElement` makes two round trips. */ +function webDriverRoutes( + routes: Record, +): typeof globalThis.fetch { + const keys = Object.keys(routes).sort((a, b) => b.length - a.length); + return async (input) => { + const url = String(input instanceof Request ? input.url : input); + const key = keys.find((candidate) => url.includes(candidate)); + const route = key + ? routes[key]! + : { status: 500, body: { value: { message: `unrouted ${url}` } } }; + return new Response(JSON.stringify(route.body), { + status: route.status ?? 200, + headers: { 'Content-Type': 'application/json' }, + }); + }; +} + +/** A client with a live session id, so the probes reach the session routes. */ async function connectedClient(respond: typeof globalThis.fetch): Promise { globalThis.fetch = webDriverResponse(200, { value: { sessionId: 'wd-1', capabilities: {} }, diff --git a/packages/provider-webdriver/src/webdriver-client.ts b/packages/provider-webdriver/src/webdriver-client.ts index 78251961d..80a453184 100644 --- a/packages/provider-webdriver/src/webdriver-client.ts +++ b/packages/provider-webdriver/src/webdriver-client.ts @@ -35,6 +35,12 @@ export type WebDriverWindowRect = { height: number; }; +export type WebDriverActiveElement = { + /** Opaque W3C element reference; only ever compared for equality. */ + id: string; + rect: WebDriverWindowRect; +}; + export type W3CPointerAction = | { type: 'pointerMove'; @@ -174,6 +180,58 @@ export class WebDriverClient { return typeof value === 'boolean' ? value : 'unsupported'; } + /** + * Which element holds text-entry focus and where it sits, `none` when nothing + * does, or `unsupported` when this driver has no active-element route. + * + * Keyboard visibility can only witness that *a* field took focus; it says + * nothing about *which*, which is why a second fill into an already-open form + * could send its keys to the field the first one focused (#1658). The focused + * element's identity answers that. Its rect comes along because identity + * alone cannot distinguish a deliberate re-fill of the already-focused field + * from a tap that moved nothing. + * + * `no such element` is a real, expected answer here (a form with nothing + * focused yet), so it maps to `none` and leaves the caller polling. A route + * that answers 200 with something that is not an element reference is not + * implementing this at all — the spec reports "nothing focused" as that + * error, never as an empty value — so it degrades to `unsupported` alongside + * a genuinely unimplemented route, the same way a non-boolean keyboard answer + * does. Reading it as `none` would fail every fill on such a grid. + */ + async activeElement( + timeoutMs?: number, + ): Promise { + const overrides = { retryAttempts: 0, ...(timeoutMs === undefined ? {} : { timeoutMs }) }; + let elementId: string | undefined; + try { + elementId = readW3CElementId( + await this.sessionRequest('GET', '/element/active', undefined, overrides), + ); + } catch (error) { + if (isUnimplementedWebDriverRoute(error)) return 'unsupported'; + if (isNoSuchElementError(error)) return 'none'; + throw error; + } + if (elementId === undefined) return 'unsupported'; + try { + const value = await this.sessionRequest( + 'GET', + `/element/${encodeURIComponent(elementId)}/rect`, + undefined, + overrides, + ); + return { id: elementId, rect: readWindowRect(value) }; + } catch (error) { + if (isUnimplementedWebDriverRoute(error)) return 'unsupported'; + // The focused element went away between the two calls — a stale answer, + // not a broken driver. Report it as "nothing focused" so the caller polls + // again rather than failing on a race it can simply retry out of. + if (isNoSuchElementError(error)) return 'none'; + throw error; + } + } + async back(): Promise { await this.sessionRequest('POST', '/back'); } @@ -379,6 +437,26 @@ function webdriverError(status: number, payload: unknown): AppError { const UNIMPLEMENTED_WEBDRIVER_STATUSES = new Set([405, 501]); const UNIMPLEMENTED_WEBDRIVER_ERRORS = new Set(['unknown command', 'unknown method']); +/** + * The W3C element identifier, whose key is the spec's fixed UUID rather than a + * readable name. Appium also echoes the legacy `ELEMENT` key; accept either so + * the caller works across grid versions. + */ +const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf'; + +function readW3CElementId(value: unknown): string | undefined { + if (!value || typeof value !== 'object') return undefined; + const record = value as Record; + const id = record[W3C_ELEMENT_KEY] ?? record.ELEMENT; + return typeof id === 'string' && id.length > 0 ? id : undefined; +} + +/** Nothing is focused right now — an expected state, not a driver defect. */ +function isNoSuchElementError(error: unknown): boolean { + if (!(error instanceof AppError)) return false; + return readWebDriverErrorCode(error).toLowerCase() === 'no such element'; +} + function isUnimplementedWebDriverRoute(error: unknown): boolean { if (!(error instanceof AppError)) return false; const status = error.details?.status; diff --git a/packages/provider-webdriver/src/webdriver-interactor.test.ts b/packages/provider-webdriver/src/webdriver-interactor.test.ts index 4afe7c39b..54af2e403 100644 --- a/packages/provider-webdriver/src/webdriver-interactor.test.ts +++ b/packages/provider-webdriver/src/webdriver-interactor.test.ts @@ -9,60 +9,146 @@ import { createWebDriverInteractor } from './webdriver-interactor.ts'; // #1658: `fill` used to send its keys in the request right after the tap. A // WebView input takes first responder asynchronously, so on a web login form // the keys landed with nothing focused and `fill` still reported success. -test('fill withholds keys until the tapped field raises the keyboard', async () => { +test('fill withholds keys until the tapped point holds text-entry focus', async () => { const world = createTextEntryWorld(); - // Hidden at the tap, then up two polls later: the transition our tap caused. - world.keyboardShown = [false, false, false, true]; + // Nothing focused at first, then the tapped field takes it two polls later. + world.activeRects = ['none', 'none', CONTAINS_TAP]; + + const result = await runFill(world); + + assert.equal(result?.textEntryReadiness, 'focused-element'); + assert.deepEqual(world.transcript, [ + 'keyboard', + 'active', + 'tap', + 'active', + 'active', + 'active', + 'keys', + ]); +}); + +// The case keyboard visibility can never decide: a second fill into a form +// whose keyboard is already up. The keyboard says "a field is focused" both +// before and after, so only the focused element's own geometry can tell whether +// OUR tap moved focus. +test('fill accepts a keyboard-up second field once focus lands on the tapped point', async () => { + const world = createTextEntryWorld(); + world.keyboardShown = [true]; + world.focusedBeforeTap = PREVIOUS_FIELD; + world.activeRects = [CONTAINS_TAP]; + + const result = await runFill(world); + + assert.equal(result?.textEntryReadiness, 'focused-element'); + assert.deepEqual(world.transcript, ['keyboard', 'active', 'tap', 'active', 'keys']); +}); + +// P1, the two-field misdelivery: email filled, keyboard stays up, the password +// tap misses. Focus never leaves the email field, so a global `/keys` would +// append the password to the EMAIL field while reporting success. +test('fill refuses when a keyboard-up tap misses and focus stays on the previous field', async () => { + const world = createTextEntryWorld(); + world.keyboardShown = [true]; + world.focusedBeforeTap = PREVIOUS_FIELD; + world.activeRects = [PREVIOUS_FIELD]; + + await assert.rejects(runFill(world), (error: AppError) => { + assert.equal(error.code, 'COMMAND_FAILED'); + assert.match(error.message, /nothing there took text-entry focus/); + assert.equal(error.details?.reason, 'text_entry_focus_not_observed'); + return true; + }); + + // The decisive assertion: no keys reached the field the previous fill focused. + assert.equal(world.transcript.includes('keys'), false); +}); + +// Found on a live AWS Device Farm iPhone 16: tapping Safari's collapsed address +// bar expands it into a taller field whose rect no longer covers the tapped +// point. A containment-only rule refused a fill that plainly worked, so a tap +// that MOVES focus counts even when the focused element has since been re-laid +// out. +test('fill accepts focus that moved to an element re-laid out away from the tap', async () => { + const world = createTextEntryWorld(); + world.activeRects = ['none', MOVED_AFTER_FOCUS]; + + const result = await runFill(world); + + assert.equal(result?.textEntryReadiness, 'focused-element'); +}); + +// The other half of the rule: re-filling the field that is ALREADY focused +// moves nothing, so only containment can tell it from a tap that missed. +test('fill accepts a re-fill of the already-focused field it tapped', async () => { + const world = createTextEntryWorld(); + world.focusedBeforeTap = CONTAINS_TAP; + world.activeRects = [CONTAINS_TAP]; + + const result = await runFill(world); + + assert.equal(result?.textEntryReadiness, 'focused-element'); +}); + +test('fill falls back to the keyboard transition when there is no active-element route', async () => { + const world = createTextEntryWorld(); + world.activeRoute = 'unimplemented'; + world.keyboardShown = [false, false, true]; const result = await runFill(world); assert.equal(result?.textEntryReadiness, 'keyboard-shown'); assert.deepEqual(world.transcript, [ 'keyboard', + 'active', 'tap', - 'keyboard', + 'active', 'keyboard', 'keyboard', 'keys', ]); }); -test('fill settles when the keyboard was already up before the tap', async () => { +// Without an active-element route, a keyboard that was ALREADY up witnesses +// nothing about the new field — the exact silent misdelivery P1 names. +test('fill refuses a keyboard-up tap when the driver cannot report the focused field', async () => { const world = createTextEntryWorld(); - // Back-to-back fills (email then password): a keyboard that never went down - // cannot witness focus moving to the NEW field, so no poll can prove anything - // — polling it would only re-observe evidence about the previous field. + world.activeRoute = 'unimplemented'; world.keyboardShown = [true]; - const result = await runFill(world); + await assert.rejects(runFill(world), (error: AppError) => { + assert.match(error.message, /cannot report which field holds focus/); + assert.equal(error.details?.reason, 'text_entry_focus_not_observed'); + return true; + }); - assert.equal(result?.textEntryReadiness, 'settled-keyboard-up'); - assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']); + assert.equal(world.transcript.includes('keys'), false); }); -test('fill settles when the driver implements no keyboard route', async () => { +test('fill settles blind only when the driver implements neither route', async () => { const world = createTextEntryWorld(); + world.activeRoute = 'unimplemented'; world.keyboardRoute = 'unimplemented'; const result = await runFill(world); - // A route the driver does not have is not a failure — it just leaves nothing to wait for. + // Routes the driver does not have are not failures — they just leave nothing + // to wait for. This is the only path that types without evidence. assert.equal(result?.textEntryReadiness, 'settled-unknown'); - assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']); + assert.deepEqual(world.transcript, ['keyboard', 'active', 'tap', 'active', 'keys']); }); // The #1658 report is "fill answered Filled N chars while the field kept its // placeholder". Reporting a readiness value nobody renders would preserve that, -// so a tap that raises no keyboard refuses instead — and sends no keys, leaving +// so a tap that focuses nothing refuses instead — and sends no keys, leaving // the field untouched rather than half-written. -test('fill refuses without typing when no keyboard ever appears', async () => { +test('fill refuses without typing when nothing ever takes focus', async () => { const world = createTextEntryWorld(); - world.keyboardShown = [false]; + world.activeRects = ['none']; await assert.rejects(runFill(world), (error: AppError) => { assert.equal(error.code, 'COMMAND_FAILED'); - assert.match(error.message, /no keyboard appeared, so the text was not sent/); - assert.equal(error.details?.reason, 'text_entry_focus_not_observed'); + assert.match(error.message, /nothing there took text-entry focus/); return true; }); @@ -80,18 +166,53 @@ test('fill propagates a failing keyboard probe instead of typing blind', async ( return true; }); - // It threw on the pre-tap probe, so the device was never touched at all. - assert.deepEqual(world.transcript, ['keyboard']); + // It threw on a pre-tap probe, so the device was never touched at all. + assert.equal(world.transcript.includes('tap'), false); }); -test('fill bounds each keyboard probe well under its readiness budget', async () => { +// Post-tap the device HAS been touched, so one transient grid error must not +// fail a fill the next poll would have satisfied. +test('fill tolerates a transient probe failure and still witnesses focus', async () => { const world = createTextEntryWorld(); - world.keyboardShown = [false, true]; + world.activeFailuresAfterTap = 2; + world.activeRects = [CONTAINS_TAP]; - await runFill(world); + const result = await runFill(world); - // Left at the client default, one hung probe could hold a 2s wait for 30s. - assert.deepEqual(world.probeTimeouts, [1_500, 1_500]); + assert.equal(result?.textEntryReadiness, 'focused-element'); +}); + +// ...but a budget that expires without a single answered probe is a broken +// session, and must surface as itself rather than as "the tap missed". +test('fill rethrows the probe failure when no probe ever answers', async () => { + const world = createTextEntryWorld(); + world.activeRoute = 'failing'; + + await assert.rejects(runFill(world), (error: AppError) => { + assert.match(error.message, /active element exploded/); + return true; + }); + + assert.equal(world.transcript.includes('keys'), false); +}); + +// P2: a probe handed the full fixed timeout near the deadline would finish well +// after the budget it was meant to respect, so each one is capped by what is +// actually left. +test('fill caps each probe timeout by the readiness budget remaining', async () => { + const world = createTextEntryWorld(); + world.activeRects = ['none']; + + await assert.rejects(runFill(world), () => true); + + const timeouts = world.probeTimeouts.filter((t): t is number => t !== undefined); + assert.equal( + timeouts.every((t) => t <= 1_500), + true, + ); + // The budget is 2s and probes start at t=0, so once under 1.5s remain the cap + // is the remainder, and the final probe gets exactly one poll interval. + assert.equal(timeouts.at(-1), 100); }); /** @@ -127,23 +248,70 @@ async function runFill(world: ReturnType) { * Records the request order `fill` produces. The ordering IS the fix: a `keys` * entry that follows `tap` with no keyboard evidence between them is the bug. */ +type FocusedElement = { + id: string; + rect: { x: number; y: number; width: number; height: number }; +}; + +/** `fill` is driven at (12, 24) throughout, so these read as hit and miss. */ +const CONTAINS_TAP = { id: 'tapped', rect: { x: 0, y: 0, width: 100, height: 100 } } as const; +const PREVIOUS_FIELD = { + id: 'previous', + rect: { x: 200, y: 200, width: 100, height: 100 }, +} as const; +/** Focused by our tap, but re-laid out by focusing so it no longer covers the tap point. */ +const MOVED_AFTER_FOCUS = { + id: 'expanded', + rect: { x: 200, y: 600, width: 180, height: 40 }, +} as const; + function createTextEntryWorld() { const world = { transcript: [] as string[], probeTimeouts: [] as Array, - keyboardShown: [] as boolean[], - /** How the driver answers the keyboard route: normally, not at all, or with a real failure. */ + keyboardShown: [true] as boolean[], + /** How the driver answers each route: normally, not at all, or with a real failure. */ keyboardRoute: 'ok' as 'ok' | 'unimplemented' | 'failing', + activeRoute: 'ok' as 'ok' | 'unimplemented' | 'failing', + /** Successive answers from the active-element route; the last one repeats. */ + activeRects: ['none'] as Array<'none' | FocusedElement>, + /** What held focus when the fill began — a previous fill's field, or nothing. */ + focusedBeforeTap: 'none' as 'none' | FocusedElement, + /** Transient post-tap failures to emit before the route starts answering. */ + activeFailuresAfterTap: 0, + tapped: false, client: undefined as unknown as WebDriverClient, }; world.client = { performActions: async () => { world.transcript.push('tap'); + world.tapped = true; }, releaseActions: async () => {}, sendKeys: async () => { world.transcript.push('keys'); }, + activeElement: async (timeoutMs?: number) => { + world.transcript.push('active'); + world.probeTimeouts.push(timeoutMs); + if (world.activeRoute === 'unimplemented') return 'unsupported' as const; + if (world.activeRoute === 'failing') { + throw new AppError('COMMAND_FAILED', 'active element exploded', { status: 500 }); + } + // Before the tap the form is in whatever state the previous fill left it. + if (!world.tapped) { + return world.focusedBeforeTap === 'none' + ? ('none' as const) + : { id: world.focusedBeforeTap.id, rect: world.focusedBeforeTap.rect }; + } + if (world.activeFailuresAfterTap > 0) { + world.activeFailuresAfterTap -= 1; + throw new AppError('COMMAND_FAILED', 'transient grid blip', { status: 502 }); + } + const next = + world.activeRects.length > 1 ? world.activeRects.shift()! : world.activeRects[0]!; + return next === 'none' ? ('none' as const) : { id: next.id, rect: next.rect }; + }, isKeyboardShown: async (timeoutMs?: number) => { world.transcript.push('keyboard'); world.probeTimeouts.push(timeoutMs); diff --git a/packages/provider-webdriver/src/webdriver-interactor.ts b/packages/provider-webdriver/src/webdriver-interactor.ts index cd9e60079..97eb42e46 100644 --- a/packages/provider-webdriver/src/webdriver-interactor.ts +++ b/packages/provider-webdriver/src/webdriver-interactor.ts @@ -47,25 +47,35 @@ function sleep(ms: number): Promise { } /** - * The tap raised no keyboard within the readiness budget, so nothing focused a - * text input and `sendKeys` would go to whatever else holds first responder — - * or nowhere. #1658 is precisely the report of that being answered with - * "Filled N chars", so this fails instead of typing: the caller gets a refusal - * it must handle, and the field is left untouched rather than half-written. + * Nothing took text-entry focus within the readiness budget, so `sendKeys` + * would go to whatever else holds first responder — or nowhere. #1658 is + * precisely the report of that being answered with "Filled N chars", so this + * fails instead of typing: the caller gets a refusal it must handle, and the + * field is left untouched rather than half-written. */ -function textEntryFocusNotObservedError(x: number, y: number): AppError { - return new AppError( - 'COMMAND_FAILED', - `fill tapped (${x}, ${y}) but no keyboard appeared, so the text was not sent`, - { - reason: 'text_entry_focus_not_observed', - x, - y, - hint: 'The tap most likely missed the field. Re-check the target with snapshot -i and fill the element it reports, rather than retrying the same coordinates.', - }, - ); +function textEntryFocusNotObservedError(x: number, y: number, detail: string): AppError { + return new AppError('COMMAND_FAILED', `fill tapped (${x}, ${y}) but ${detail}`, { + reason: 'text_entry_focus_not_observed', + x, + y, + hint: 'The tap most likely missed the field. Re-check the target with snapshot -i and fill the element it reports, rather than retrying the same coordinates.', + }); } +function rectContains(rect: WebDriverWindowRect, x: number, y: number): boolean { + return x >= rect.x && x <= rect.x + rect.width && y >= rect.y && y <= rect.y + rect.height; +} + +/** + * What text entry looked like before the tap. Both readings are only evidence + * as a change, so both must be captured before the device is touched. + */ +type TextEntryStateBeforeTap = { + keyboard: boolean | 'unsupported'; + /** `undefined` when nothing was focused, or when the driver cannot say. */ + focusedElementId: string | undefined; +}; + export type WebDriverInteractorOptions = { client: WebDriverClient; backend: Extract; @@ -189,14 +199,15 @@ class WebDriverInteractor implements Interactor { _delayMs?: number, ): Promise> { this.requireSupport('fill'); - // #1658: read the keyboard BEFORE the tap. Only a hidden -> shown - // transition proves that OUR tap moved focus; a keyboard that was already - // up says nothing about which field owns first responder now. A probe - // failure throws here, before the tap, so a broken session fails a fill - // that has not touched the device rather than typing into the dark. - const keyboardBeforeTap = await this.readKeyboardShown(); + // #1658: both readings are taken BEFORE the tap, because each is only + // evidence as a change. A keyboard that was already up says nothing about + // which field owns first responder now, and a focused element is only + // proof our tap did something if it differs from the one already focused. + // Probe failures throw here, before the tap, so a broken session fails a + // fill that has not touched the device rather than typing into the dark. + const beforeTap = await this.readTextEntryState(); await this.tap(x, y); - const textEntryReadiness = await this.awaitTextEntryReadiness(keyboardBeforeTap, x, y); + const textEntryReadiness = await this.awaitTextEntryReadiness(beforeTap, x, y); await this.type(text); return { backend: 'webdriver', x, y, text, textEntryReadiness }; } @@ -316,40 +327,115 @@ class WebDriverInteractor implements Interactor { * only the round trip between them gave the field time to focus. */ private async awaitTextEntryReadiness( - keyboardBeforeTap: boolean | 'unsupported', + beforeTap: TextEntryStateBeforeTap, x: number, y: number, ): Promise { - // The driver implements no keyboard route, so no wait can learn anything. - // Keep a blind path cheap: settle briefly, then type. This is the ONE case - // that types without evidence, and it takes a positively classified - // unsupported answer to reach — never a probe that merely failed. + const keyboardBeforeTap = beforeTap.keyboard; + const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS; + // Strongest evidence first: the focused element itself. Keyboard visibility + // can only witness that *a* field took focus, never *which* — which is why + // filling a second field in an already-open form could send its keys to the + // first one. + // + // The question is whether OUR tap moved focus, so compare against what held + // it before. Identity rather than geometry, because focusing a field can + // re-lay it out: tapping Safari's collapsed address bar expands it into a + // taller search field that no longer contains the point that was tapped, so + // a containment-only rule refuses a fill that plainly worked. Containment + // still earns its place as the second half of the test — re-filling the + // field that is ALREADY focused moves nothing, and only geometry can tell + // that from a tap that missed everything. + const focusedBeforeTap = beforeTap.focusedElementId; + const focus = await this.pollWithinBudget(deadline, async (timeoutMs) => { + const active = await this.client.activeElement(timeoutMs); + if (active === 'unsupported') return 'abandon'; + if (active === 'none') return 'retry'; + const moved = active.id !== focusedBeforeTap; + return moved || rectContains(active.rect, x, y) ? 'satisfied' : 'retry'; + }); + if (focus === 'satisfied') return 'focused-element'; + if (focus === 'expired') { + throw textEntryFocusNotObservedError( + x, + y, + 'nothing there took text-entry focus, so the text was not sent', + ); + } + + // No active-element route. Fall back to keyboard evidence, which can still + // witness focus arriving when the keyboard was down before our tap. if (keyboardBeforeTap === 'unsupported') { + // The driver implements neither route, so no wait can learn anything. + // Keep a blind path cheap: settle briefly, then type. This is the ONE + // case that types without evidence, and it takes two positively + // classified unsupported answers to reach — never a probe that failed. await sleep(TEXT_ENTRY_BLIND_SETTLE_MS); return 'settled-unknown'; } - // The keyboard was ALREADY up — back-to-back fills into one form, the shape - // that failed most often in #1658. Its visibility cannot witness focus - // moving to the NEW field, so polling it would resolve instantly on - // evidence about the PREVIOUS one. The Apple runner spends its whole - // readiness budget on exactly this case rather than race the app with a - // short settle; the cloud path has even more reason to, because its keys go - // to whatever holds first responder rather than to a scoped element. if (keyboardBeforeTap) { - await sleep(TEXT_ENTRY_READINESS_TIMEOUT_MS); - return 'settled-keyboard-up'; + // Keyboard already up and no way to ask which field owns it: every signal + // available here describes the PREVIOUS field. Typing anyway is exactly + // #1658's silent misdelivery, so refuse instead. + throw textEntryFocusNotObservedError( + x, + y, + 'the keyboard was already up and this driver cannot report which field holds focus, so the text was not sent', + ); } - const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS; - while (Date.now() < deadline) { - await sleep(TEXT_ENTRY_READINESS_POLL_MS); - if ((await this.readKeyboardShown()) === true) return 'keyboard-shown'; - } - throw textEntryFocusNotObservedError(x, y); + const keyboard = await this.pollWithinBudget(deadline, async (timeoutMs) => + (await this.client.isKeyboardShown(timeoutMs)) === true ? 'satisfied' : 'retry', + ); + if (keyboard === 'satisfied') return 'keyboard-shown'; + throw textEntryFocusNotObservedError(x, y, 'no keyboard appeared, so the text was not sent'); } - /** Bounded by the readiness budget it serves, not the client's default request timeout. */ - private async readKeyboardShown(): Promise { - return await this.client.isKeyboardShown(TEXT_ENTRY_PROBE_TIMEOUT_MS); + /** Both pre-tap readings, taken together so neither is charged an extra tap-to-probe gap. */ + private async readTextEntryState(): Promise { + const [keyboard, active] = await Promise.all([ + this.client.isKeyboardShown(TEXT_ENTRY_PROBE_TIMEOUT_MS), + this.client.activeElement(TEXT_ENTRY_PROBE_TIMEOUT_MS), + ]); + return { + keyboard, + focusedElementId: active === 'none' || active === 'unsupported' ? undefined : active.id, + }; + } + + /** + * Polls until `probe` is satisfied or the readiness budget runs out, bounding + * BOTH the probe's own transport timeout and the sleep between attempts by + * what remains — a probe handed the full fixed timeout near the deadline + * would otherwise finish well after the budget it was meant to respect. + * + * A probe that throws is tolerated: the loop has further attempts, and one + * transient grid error should not fail a fill that would have succeeded. But + * a budget that expires without a single answered probe rethrows, so a dead + * session or an auth rejection surfaces as itself instead of being reported + * as a tap that missed its field. + */ + private async pollWithinBudget( + deadline: number, + probe: (timeoutMs: number) => Promise<'satisfied' | 'abandon' | 'retry'>, + ): Promise<'satisfied' | 'abandon' | 'expired'> { + let answered = false; + let lastError: unknown; + for (;;) { + const remaining = deadline - Date.now(); + if (remaining <= 0) break; + try { + const verdict = await probe(Math.min(TEXT_ENTRY_PROBE_TIMEOUT_MS, remaining)); + answered = true; + if (verdict !== 'retry') return verdict; + } catch (error) { + lastError = error; + } + const left = deadline - Date.now(); + if (left <= 0) break; + await sleep(Math.min(TEXT_ENTRY_READINESS_POLL_MS, left)); + } + if (!answered && lastError !== undefined) throw lastError; + return 'expired'; } private async pointerGesture(name: string, actions: W3CPointerAction[]): Promise { diff --git a/test/integration/provider-scenarios/cloud-webdriver-ios-text-entry.test.ts b/test/integration/provider-scenarios/cloud-webdriver-ios-text-entry.test.ts index 648a33771..82153f49d 100644 --- a/test/integration/provider-scenarios/cloud-webdriver-ios-text-entry.test.ts +++ b/test/integration/provider-scenarios/cloud-webdriver-ios-text-entry.test.ts @@ -76,20 +76,31 @@ test('cloud iOS snapshot captures through the provider session after open', asyn test('cloud iOS fill lands text in the field it tapped', async () => { await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => { const lease = await openCloudIosSession(daemon); + // Focus lands a poll late, as a WebView input's does. The keys must wait. + server.pollsUntilFocus = 2; const response = await daemon.callCommand( 'fill', - ['editable=true', 'user@example.com'], + ['label=Email', 'user@example.com'], leaseFlags(lease.leaseId), { meta: leaseMeta(lease.leaseId) }, ); const data = assertRpcOk<{ text?: string; textEntryReadiness?: string }>(response); assert.equal(data.text, 'user@example.com'); - assert.equal(data.textEntryReadiness, 'keyboard-shown'); + assert.equal(data.textEntryReadiness, 'focused-element'); // The keys reached the focused field, not the void. - assert.equal(server.fieldValue, 'user@example.com'); - assert.deepEqual(textEntryTranscript(server), ['keyboard', 'tap', 'keyboard', 'keys']); + assert.equal(server.fieldValues.email, 'user@example.com'); + // Both pre-tap readings, the tap, then polling until focus lands. The keys + // come last — that ordering IS the fix. + assert.deepEqual(textEntryTranscript(server), [ + 'keyboard', + 'active', + 'tap', + 'active', + 'active', + 'keys', + ]); // And the device agrees: the field reports the typed value back. const snapshot = await daemon.callCommand( @@ -111,23 +122,54 @@ test('cloud iOS fill lands text in the field it tapped', async () => { * "Filled N chars" is the original bug, so the fill refuses — and sends no keys, * leaving the field untouched rather than half-written. */ -test('cloud iOS fill refuses, without typing, when the tap raises no keyboard', async () => { +test('cloud iOS fill refuses, without typing, when nothing takes focus', async () => { await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => { const lease = await openCloudIosSession(daemon); - server.keyboardPollsUntilShown = Number.POSITIVE_INFINITY; + server.pollsUntilFocus = Number.POSITIVE_INFINITY; const response = await daemon.callCommand( 'fill', - ['editable=true', 'user@example.com'], + ['label=Email', 'user@example.com'], leaseFlags(lease.leaseId), { meta: leaseMeta(lease.leaseId) }, ); const error = response.json.error; assert.ok(error, 'expected fill to refuse'); - assert.match(error.message ?? '', /no keyboard appeared, so the text was not sent/); + assert.match(error.message ?? '', /nothing there took text-entry focus/); assert.equal(error.data?.details?.reason, 'text_entry_focus_not_observed'); - assert.equal(server.fieldValue, ''); + assert.equal(server.fieldValues.email, ''); + assert.equal(textEntryTranscript(server).includes('keys'), false); + }); +}, 15_000); + +/** + * The misdelivery keyboard visibility can never catch (#1658 P1). The form is + * already filled and focused — keyboard up — and the second fill's tap misses + * the password field. Focus therefore never leaves `email`, so a `POST /keys` + * here would append the password to the EMAIL field and every request would + * still return 200. The fill must refuse before sending anything. + */ +test('cloud iOS fill sends no keys when a keyboard-up tap misses the second field', async () => { + await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => { + const lease = await openCloudIosSession(daemon); + server.focusField('email', 'user@example.com'); + // Focus never moves, reproducing a tap that landed outside both fields. + server.pollsUntilFocus = Number.POSITIVE_INFINITY; + + const response = await daemon.callCommand( + 'fill', + ['label=Password', 'hunter2'], + leaseFlags(lease.leaseId), + { meta: leaseMeta(lease.leaseId) }, + ); + + assert.ok(response.json.error, 'expected fill to refuse'); + assert.equal(response.json.error.data?.details?.reason, 'text_entry_focus_not_observed'); + // The decisive assertion: the previous field is untouched and the password + // never reached it. + assert.equal(server.fieldValues.email, 'user@example.com'); + assert.equal(server.fieldValues.password, ''); assert.equal(textEntryTranscript(server).includes('keys'), false); }); }, 15_000); @@ -138,6 +180,7 @@ function textEntryTranscript(server: FakeIosWebDriverServer): string[] { 'POST /wd/hub/session/wd-1/actions': 'tap', 'POST /wd/hub/session/wd-1/keys': 'keys', 'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': 'keyboard', + 'GET /wd/hub/session/wd-1/element/active': 'active', }; return server.calls .map((call) => labels[`${call.method} ${call.path}`]) @@ -195,18 +238,38 @@ async function openCloudIosSession(daemon: ProviderScenarioHarness): Promise = { email: '', password: '' }; + focused: FieldName | undefined; private pollsRemaining = Number.POSITIVE_INFINITY; + private pendingFocus: FieldName | undefined; static async start(): Promise> { return await startCloudWebDriverTestServer(new FakeIosWebDriverServer()); } + /** Puts the form in the state a completed first fill leaves it in. */ + focusField(field: FieldName, value: string): void { + this.focused = field; + this.fieldValues[field] = value; + this.pollsRemaining = 0; + this.pendingFocus = field; + } + protected respond(call: CloudWebDriverHttpCall) { + const rectMatch = /^\/wd\/hub\/session\/wd-1\/element\/([^/]+)\/rect$/.exec(call.path); + if (call.method === 'GET' && rectMatch) return this.elementRect(rectMatch[1]!); const route = this.routes[`${call.method} ${call.path}`]; return route ? route() : cloudWebDriverTestJson({ value: null }); } @@ -222,47 +285,113 @@ class FakeIosWebDriverServer extends CloudWebDriverTestServer { cloudWebDriverTestJson({ value: { x: 0, y: 0, width: 390, height: 844 } }), 'POST /wd/hub/session/wd-1/actions': () => this.tap(), 'POST /wd/hub/session/wd-1/keys': () => this.sendKeys(), + 'GET /wd/hub/session/wd-1/element/active': () => this.activeElement(), 'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': () => cloudWebDriverTestJson({ value: this.keyboardShown() }), }; } + /** + * A tap moves focus only if it actually lands inside a field. A miss leaves + * the previous field focused — which is exactly how keys can reach the wrong + * one while every request still succeeds. + */ private tap(): CloudWebDriverTestResponse { - this.pollsRemaining = this.keyboardPollsUntilShown; + const point = this.lastTapPoint(); + const hit = + point && (Object.keys(FIELDS) as FieldName[]).find((n) => contains(FIELDS[n], point)); + this.pendingFocus = hit ?? this.focused; + this.pollsRemaining = hit ? this.pollsUntilFocus : Number.POSITIVE_INFINITY; + if (hit) this.focused = undefined; return cloudWebDriverTestJson({ value: null }); } - /** Focus is not instant: it lands only after the configured number of probes. */ - private keyboardShown(): boolean { - if (this.pollsRemaining <= 0) return true; + /** Focus is not instant: it settles only after the configured number of probes. */ + private settle(): void { + if (this.pollsRemaining <= 0) return; this.pollsRemaining -= 1; - return this.pollsRemaining <= 0; + if (this.pollsRemaining <= 0) this.focused = this.pendingFocus; + } + + private activeElement(): CloudWebDriverTestResponse { + this.settle(); + if (!this.focused) { + return cloudWebDriverTestJson( + { value: { error: 'no such element', message: 'nothing is focused' } }, + 404, + ); + } + return cloudWebDriverTestJson({ value: { [W3C_ELEMENT_KEY]: this.focused } }); + } + + private elementRect(elementId: string): CloudWebDriverTestResponse { + const rect = FIELDS[elementId as FieldName]; + if (!rect) { + return cloudWebDriverTestJson( + { value: { error: 'no such element', message: `unknown element ${elementId}` } }, + 404, + ); + } + return cloudWebDriverTestJson({ value: rect }); + } + + private keyboardShown(): boolean { + this.settle(); + return this.focused !== undefined; } private sendKeys(): CloudWebDriverTestResponse { - // Unfocused keys are accepted by the grid and land nowhere — the bug's shape. - if (this.pollsRemaining <= 0) { - const text = this.lastSentKeys(); - if (text !== undefined) this.fieldValue = text; - } + // Unfocused keys are accepted by the grid and land nowhere; keys sent while + // the WRONG field holds focus land in that field. Both are the bug's shape. + const text = this.lastSentKeys(); + if (this.focused && text !== undefined) this.fieldValues[this.focused] = text; return cloudWebDriverTestJson({ value: null }); } + private lastTapPoint(): { x: number; y: number } | undefined { + const body = this.calls.at(-1)?.body as { actions?: unknown } | undefined; + const sequences = Array.isArray(body?.actions) ? body.actions : []; + for (const sequence of sequences) { + const actions = (sequence as { actions?: unknown }).actions; + if (!Array.isArray(actions)) continue; + const move = actions.find((a) => (a as { type?: string }).type === 'pointerMove') as + | { x?: unknown; y?: unknown } + | undefined; + if (typeof move?.x === 'number' && typeof move.y === 'number') { + return { x: move.x, y: move.y }; + } + } + return undefined; + } + private lastSentKeys(): string | undefined { const body = this.calls.at(-1)?.body as { value?: unknown } | undefined; return Array.isArray(body?.value) ? body.value.join('') : undefined; } private source(): string { + const field = (name: FieldName, label: string) => + ``; return ( '' + - `' + + field('email', 'Email') + + field('password', 'Password') + '' ); } } +function contains( + rect: { x: number; y: number; width: number; height: number }, + p: { x: number; y: number }, +): boolean { + return ( + p.x >= rect.x && p.x <= rect.x + rect.width && p.y >= rect.y && p.y <= rect.y + rect.height + ); +} + function leaseFlags(leaseId?: string): DaemonRequest['flags'] { return { platform: 'ios', diff --git a/website/docs/docs/device-clouds.md b/website/docs/docs/device-clouds.md index f75eeed9c..90eaa87be 100644 --- a/website/docs/docs/device-clouds.md +++ b/website/docs/docs/device-clouds.md @@ -355,5 +355,5 @@ BrowserStack can return session video, Appium logs, device logs, dashboard URL, - If AWS connect fails, use its reported `aws devicefarm get-*` error to check the credential chain, ARN, region, resource platform, or upload readiness. Allocation has not happened yet. - If Limrun connect fails, check `LIMRUN_API_KEY` and the optional `LIMRUN_REGION`. A successful connect verifies the selected instance service without creating an instance. - If artifact lookup is pending immediately after `close`, retry `agent-device artifacts --json`. Some providers finalize video/log URLs asynchronously after the hosted session stops. -- On hosted WebDriver sessions (BrowserStack, AWS Device Farm), `fill` waits for the tapped field to take text entry focus before sending its keys, and reports what it observed as `textEntryReadiness`: `keyboard-shown` (the keyboard came up in response to the tap), `settled-keyboard-up` (the keyboard was already up — back-to-back fills into one form — so it could not witness the new field taking focus and the fill waited out its full readiness budget), or `settled-unknown` (the driver implements no keyboard route, so there was nothing to wait for). -- If the tap raises no keyboard at all, `fill` **fails** with `text_entry_focus_not_observed` and sends no keys, rather than reporting a filled field that never received the text. That almost always means the tap missed: re-check the target with `snapshot -i` and fill the element it reports instead of retrying the same coordinates. +- On hosted WebDriver sessions (BrowserStack, AWS Device Farm), `fill` waits until the field it tapped actually holds text entry focus before sending its keys, and reports what it observed as `textEntryReadiness`: `focused-element` (the focused element contains the tapped point — the strongest signal, and the only one that identifies *which* field took focus), `keyboard-shown` (the driver has no active-element route, but the keyboard rose from hidden after the tap), or `settled-unknown` (the driver implements neither route, so there was nothing to wait for). +- If focus cannot be witnessed, `fill` **fails** with `text_entry_focus_not_observed` and sends no keys, rather than reporting a filled field that never received the text. That almost always means the tap missed: re-check the target with `snapshot -i` and fill the element it reports instead of retrying the same coordinates. The same refusal covers a second fill into a form whose keyboard is already up on a driver that cannot name the focused field — there, unwitnessed keys would land in the field the previous fill focused.