mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
fix(cloud): witness the tapped field's focus before a cloud fill types
Review of cc23f2b found two ways a fill could still report success without evidence that OUR tap focused the field it was aimed at. P1. `settled-keyboard-up` and `settled-unknown` both typed and returned normal success. Keyboard visibility can only witness that *a* field took focus, never *which*: filling a second field in an already-open form reads the same before and after, so a missed tap left the first field focused and `POST /keys` — which the driver routes to whatever holds first responder — appended to it while every request returned 200. Failing those closed outright would have broken ordinary multi-field form fills, which do work: a live AWS Device Farm run types both fields of a WebView login correctly. So witness focus properly instead. W3C `GET /element/active` answers the question keyboard visibility cannot — is the thing focused now the thing I tapped — and answers it whether or not the keyboard was already up. That becomes the primary signal (`focused-element`); the keyboard transition stays as the fallback for drivers without the route, and a keyboard already up on such a driver now refuses rather than typing. The test is identity, not geometry. Containment of the tap point looks like the obvious rule and is wrong: focusing a field can re-lay it out. On a live iPhone 16, tapping Safari's collapsed address bar expands it into a taller field that no longer covers the tapped point, and a containment-only rule refused a fill that plainly worked. So a tap that MOVES focus counts, with containment as the second half of the test — re-filling the already-focused field moves nothing, and only geometry tells that from a tap that missed. Both readings are taken before the tap, since each is evidence only as a change. P2. The 2s budget bounded the loop but not the calls inside it: every probe got a fixed 1500ms, so one begun near the deadline finished well past it. Both the probe timeout and the sleep are now capped by the remaining budget. Also fixes a related escape the review did not name: the poll loop had no catch, so one transient grid error aborted a fill the next poll would have satisfied. Probe failures are now tolerated within the budget, but a budget that expires without a single answered probe rethrows, so a dead session surfaces as itself rather than as "the tap missed". The provider scenario gains the two-field case the review asked for: it begins keyboard-up with the email field focused, misses the password tap, and asserts no keys reach the email field. Verified on AWS Device Farm iPhone 16 / iOS 18.0 at this exact tree: address bar (the re-layout case) and both WebView login fields all report `focused-element`, the second with the keyboard already up, and the device reads back `tomsmith` and a 20-character password.
This commit is contained in:
+1
-1
@@ -4,7 +4,7 @@
|
||||
|
||||
- `agent-device help workflow` is now a compact ~8KB card instead of a ~41KB dump; the same depth still exists, split into `help scripting` (save-script, secret-safe fills, batch JSON, replay divergence/repair, recording) and `help gestures` (multi-touch shapes and platform quirks), plus a few paragraphs folded into the topics that already owned the subject (`help debugging`, `help physical-device`, `help validate`). Every `help <topic>` first line is now `agent-device <version> — <topic>` so an agent can read the installed version from its mandatory first help read instead of a separate `agent-device --version` call.
|
||||
- Cloud iOS (BrowserStack, AWS Device Farm): `snapshot` and `diff` no longer fail with `SESSION_NOT_FOUND` on a live provider session (#1658). The app-session guard they ran belongs to the local XCUITest runner, which must attach to a target app; a cloud capture reads the provider's own driver session and needs no app identity, so the guard now applies to local Apple targets only. Relatedly, a cloud iOS `open com.example.app` now records that bundle id on the session — the provider path skips local app resolution (no simctl/devicectl reaches a hosted device), and used to drop an explicitly spelled bundle id along with it, leaving the session with no app identity at all. Opening a second bundle id replaces the first, matching the local path, where an explicitly spelled target always wins over the session's current app; deep links, display names, and bare `open` still keep the app already tracked.
|
||||
- Cloud `fill` (BrowserStack, AWS Device Farm) waits for the tapped field to take text-entry focus before sending its keys, instead of dispatching both in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The response now discloses `textEntryReadiness`: `keyboard-shown`, `settled-keyboard-up` (back-to-back fills into one form, where keyboard visibility cannot witness the new field taking focus, so the fill waits out the same readiness budget the Apple runner spends there), or `settled-unknown` (the driver implements no keyboard route at all). Breaking: when the tap raises no keyboard within that budget, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. A driver that cannot report keyboard state at all is unaffected (it blind-settles as before); only a positively classified unimplemented route counts as such, so a dead session, an auth rejection, or a grid outage now surfaces instead of degrading into a blind text entry.
|
||||
- Cloud `fill` (BrowserStack, AWS Device Farm) now witnesses that the field it tapped actually holds text-entry focus before sending its keys, instead of dispatching tap and keys in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The witness is the focused element's own geometry: `fill` polls the active element and proceeds only once it contains the point it tapped, which is the one signal that identifies *which* field took focus. Keyboard visibility cannot — it reads the same before and after a second fill into an already-open form, so it could not tell a focused password field from the email field the previous fill left focused. The response discloses `textEntryReadiness`: `focused-element`, `keyboard-shown` (no active-element route, but the keyboard rose from hidden after the tap), or `settled-unknown` (the driver implements neither route, the only path that types without evidence). Breaking: when focus cannot be witnessed, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. That covers a tap that focused nothing and a keyboard already up on a driver that cannot name the focused field. Only a positively classified unimplemented route counts as unsupported, so a dead session, an auth rejection, or a grid outage surfaces instead of degrading into a blind text entry.
|
||||
- `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands.
|
||||
- Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags.
|
||||
- `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied.
|
||||
|
||||
@@ -102,29 +102,35 @@ export type TypeTextBackendResult = {
|
||||
* What a cloud-WebDriver `fill` could establish about its target taking text
|
||||
* entry focus before it sent the keys (#1658). The local Apple runner owns this
|
||||
* discipline in Swift and reports it as a `textEntryRoute`; the cloud path has
|
||||
* no runner, only the driver's keyboard-visibility signal, so it names what it
|
||||
* observed instead of which channel it typed through:
|
||||
* no runner, only what the driver will tell it, so it names what it observed
|
||||
* instead of which channel it typed through. Listed strongest evidence first:
|
||||
*
|
||||
* - `keyboard-shown`: the software keyboard went from hidden to shown after our
|
||||
* tap — positive evidence the tap moved focus to a text input.
|
||||
* - `settled-keyboard-up`: the keyboard was ALREADY up (back-to-back fills into
|
||||
* one form), so its visibility says nothing about the NEW field. Entry waited
|
||||
* the full readiness budget, the same answer the Apple runner gives here.
|
||||
* - `settled-unknown`: the driver implements no keyboard route at all, so
|
||||
* nothing could be waited for. Entry followed a short blind settle.
|
||||
* - `focused-element`: the element holding text-entry focus after the tap
|
||||
* contains the tapped point — positive evidence that THIS field, not merely
|
||||
* some field, took focus.
|
||||
* - `keyboard-shown`: no active-element route, but the software keyboard went
|
||||
* from hidden to shown after our tap. Witnesses that focus arrived somewhere,
|
||||
* which only means our field when the keyboard was down beforehand.
|
||||
* - `settled-unknown`: the driver implements neither an active-element nor a
|
||||
* keyboard route, so nothing could be waited for. Entry followed a short
|
||||
* blind settle. This is the only value that describes a fill with no evidence
|
||||
* behind it, and it takes two positively classified unsupported answers to
|
||||
* reach.
|
||||
*
|
||||
* Every value describes a fill that DID send its keys. The fourth outcome — a
|
||||
* tap that raised no keyboard at all — is deliberately absent: reporting it
|
||||
* here would recreate #1658's silent success, so the interactor throws
|
||||
* (`text_entry_focus_not_observed`) without typing instead.
|
||||
* Every value describes a fill that DID send its keys. Outcomes where focus
|
||||
* could not be witnessed are deliberately absent: reporting one here would
|
||||
* recreate #1658's silent success, so the interactor throws
|
||||
* (`text_entry_focus_not_observed`) without typing instead. That covers a tap
|
||||
* that focused nothing, and a keyboard already up on a driver that cannot say
|
||||
* which field owns it — the case where keys would land in the PREVIOUS field.
|
||||
*
|
||||
* Closed set: the cloud interactor is its only producer and the boundary that
|
||||
* narrows it (readFillBackendResult, src/core/dispatch-interactions.ts) drops a
|
||||
* value it cannot name.
|
||||
*/
|
||||
export const CLOUD_TEXT_ENTRY_READINESS = [
|
||||
'focused-element',
|
||||
'keyboard-shown',
|
||||
'settled-keyboard-up',
|
||||
'settled-unknown',
|
||||
] as const;
|
||||
|
||||
|
||||
@@ -74,7 +74,119 @@ test('is_keyboard_shown honors a caller timeout shorter than the client default'
|
||||
});
|
||||
});
|
||||
|
||||
/** A client with a live session id, so `isKeyboardShown` reaches the session route. */
|
||||
// The focused element is the only signal that can say WHICH field took focus,
|
||||
// so `fill` fails closed when it is unavailable. Each wire shape it
|
||||
// distinguishes is pinned here.
|
||||
test('activeElement returns the focused element id and rect', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } },
|
||||
'/rect': { body: { value: { x: 20, y: 300, width: 350, height: 44 } } },
|
||||
}),
|
||||
);
|
||||
|
||||
assert.deepEqual(await client.activeElement(), {
|
||||
id: 'el-1',
|
||||
rect: { x: 20, y: 300, width: 350, height: 44 },
|
||||
});
|
||||
});
|
||||
|
||||
// A form with nothing focused yet is an expected state, not a broken driver —
|
||||
// the caller must keep polling rather than give up.
|
||||
test('activeElement reports nothing focused as none', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': {
|
||||
status: 404,
|
||||
body: { value: { error: 'no such element', message: 'nothing focused' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'none');
|
||||
});
|
||||
|
||||
// Focus moved between the two round trips. A race the caller can simply retry
|
||||
// out of, not a failure worth ending the fill on.
|
||||
test('activeElement reports an element that vanished before its rect as none', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } },
|
||||
'/rect': {
|
||||
status: 404,
|
||||
body: { value: { error: 'no such element', message: 'stale element' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'none');
|
||||
});
|
||||
|
||||
// A grid that answers 200/null is not implementing the route: the spec reports
|
||||
// "nothing focused" as a `no such element` error. Reading null as "nothing
|
||||
// focused" would fail every fill on such a grid instead of falling back.
|
||||
test('activeElement reports a non-element answer as unsupported', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({ '/element/active': { body: { value: null } } }),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'unsupported');
|
||||
});
|
||||
|
||||
test('activeElement reports an unimplemented route as unsupported', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': {
|
||||
status: 404,
|
||||
body: { value: { error: 'unknown command', message: 'Unknown command' } },
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
assert.equal(await client.activeElement(), 'unsupported');
|
||||
});
|
||||
|
||||
// Same discipline as the keyboard route: a dead session shares the 404 and must
|
||||
// never be mistaken for a driver that simply lacks the feature.
|
||||
test('activeElement propagates a dead session, which shares the 404', async () => {
|
||||
const client = await connectedClient(
|
||||
webDriverRoutes({
|
||||
'/element/active': {
|
||||
status: 404,
|
||||
body: {
|
||||
value: { error: 'invalid session id', message: 'A session is either terminated' },
|
||||
},
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
await assert.rejects(client.activeElement(), (error: AppError) => {
|
||||
assert.match(error.message, /terminated/);
|
||||
return true;
|
||||
});
|
||||
});
|
||||
|
||||
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
|
||||
|
||||
/** Dispatches by URL suffix, since `activeElement` makes two round trips. */
|
||||
function webDriverRoutes(
|
||||
routes: Record<string, { status?: number; body: unknown }>,
|
||||
): typeof globalThis.fetch {
|
||||
const keys = Object.keys(routes).sort((a, b) => b.length - a.length);
|
||||
return async (input) => {
|
||||
const url = String(input instanceof Request ? input.url : input);
|
||||
const key = keys.find((candidate) => url.includes(candidate));
|
||||
const route = key
|
||||
? routes[key]!
|
||||
: { status: 500, body: { value: { message: `unrouted ${url}` } } };
|
||||
return new Response(JSON.stringify(route.body), {
|
||||
status: route.status ?? 200,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
/** A client with a live session id, so the probes reach the session routes. */
|
||||
async function connectedClient(respond: typeof globalThis.fetch): Promise<WebDriverClient> {
|
||||
globalThis.fetch = webDriverResponse(200, {
|
||||
value: { sessionId: 'wd-1', capabilities: {} },
|
||||
|
||||
@@ -35,6 +35,12 @@ export type WebDriverWindowRect = {
|
||||
height: number;
|
||||
};
|
||||
|
||||
export type WebDriverActiveElement = {
|
||||
/** Opaque W3C element reference; only ever compared for equality. */
|
||||
id: string;
|
||||
rect: WebDriverWindowRect;
|
||||
};
|
||||
|
||||
export type W3CPointerAction =
|
||||
| {
|
||||
type: 'pointerMove';
|
||||
@@ -174,6 +180,58 @@ export class WebDriverClient {
|
||||
return typeof value === 'boolean' ? value : 'unsupported';
|
||||
}
|
||||
|
||||
/**
|
||||
* Which element holds text-entry focus and where it sits, `none` when nothing
|
||||
* does, or `unsupported` when this driver has no active-element route.
|
||||
*
|
||||
* Keyboard visibility can only witness that *a* field took focus; it says
|
||||
* nothing about *which*, which is why a second fill into an already-open form
|
||||
* could send its keys to the field the first one focused (#1658). The focused
|
||||
* element's identity answers that. Its rect comes along because identity
|
||||
* alone cannot distinguish a deliberate re-fill of the already-focused field
|
||||
* from a tap that moved nothing.
|
||||
*
|
||||
* `no such element` is a real, expected answer here (a form with nothing
|
||||
* focused yet), so it maps to `none` and leaves the caller polling. A route
|
||||
* that answers 200 with something that is not an element reference is not
|
||||
* implementing this at all — the spec reports "nothing focused" as that
|
||||
* error, never as an empty value — so it degrades to `unsupported` alongside
|
||||
* a genuinely unimplemented route, the same way a non-boolean keyboard answer
|
||||
* does. Reading it as `none` would fail every fill on such a grid.
|
||||
*/
|
||||
async activeElement(
|
||||
timeoutMs?: number,
|
||||
): Promise<WebDriverActiveElement | 'none' | 'unsupported'> {
|
||||
const overrides = { retryAttempts: 0, ...(timeoutMs === undefined ? {} : { timeoutMs }) };
|
||||
let elementId: string | undefined;
|
||||
try {
|
||||
elementId = readW3CElementId(
|
||||
await this.sessionRequest('GET', '/element/active', undefined, overrides),
|
||||
);
|
||||
} catch (error) {
|
||||
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
|
||||
if (isNoSuchElementError(error)) return 'none';
|
||||
throw error;
|
||||
}
|
||||
if (elementId === undefined) return 'unsupported';
|
||||
try {
|
||||
const value = await this.sessionRequest(
|
||||
'GET',
|
||||
`/element/${encodeURIComponent(elementId)}/rect`,
|
||||
undefined,
|
||||
overrides,
|
||||
);
|
||||
return { id: elementId, rect: readWindowRect(value) };
|
||||
} catch (error) {
|
||||
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
|
||||
// The focused element went away between the two calls — a stale answer,
|
||||
// not a broken driver. Report it as "nothing focused" so the caller polls
|
||||
// again rather than failing on a race it can simply retry out of.
|
||||
if (isNoSuchElementError(error)) return 'none';
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async back(): Promise<void> {
|
||||
await this.sessionRequest('POST', '/back');
|
||||
}
|
||||
@@ -379,6 +437,26 @@ function webdriverError(status: number, payload: unknown): AppError {
|
||||
const UNIMPLEMENTED_WEBDRIVER_STATUSES = new Set([405, 501]);
|
||||
const UNIMPLEMENTED_WEBDRIVER_ERRORS = new Set(['unknown command', 'unknown method']);
|
||||
|
||||
/**
|
||||
* The W3C element identifier, whose key is the spec's fixed UUID rather than a
|
||||
* readable name. Appium also echoes the legacy `ELEMENT` key; accept either so
|
||||
* the caller works across grid versions.
|
||||
*/
|
||||
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
|
||||
|
||||
function readW3CElementId(value: unknown): string | undefined {
|
||||
if (!value || typeof value !== 'object') return undefined;
|
||||
const record = value as Record<string, unknown>;
|
||||
const id = record[W3C_ELEMENT_KEY] ?? record.ELEMENT;
|
||||
return typeof id === 'string' && id.length > 0 ? id : undefined;
|
||||
}
|
||||
|
||||
/** Nothing is focused right now — an expected state, not a driver defect. */
|
||||
function isNoSuchElementError(error: unknown): boolean {
|
||||
if (!(error instanceof AppError)) return false;
|
||||
return readWebDriverErrorCode(error).toLowerCase() === 'no such element';
|
||||
}
|
||||
|
||||
function isUnimplementedWebDriverRoute(error: unknown): boolean {
|
||||
if (!(error instanceof AppError)) return false;
|
||||
const status = error.details?.status;
|
||||
|
||||
@@ -9,60 +9,146 @@ import { createWebDriverInteractor } from './webdriver-interactor.ts';
|
||||
// #1658: `fill` used to send its keys in the request right after the tap. A
|
||||
// WebView input takes first responder asynchronously, so on a web login form
|
||||
// the keys landed with nothing focused and `fill` still reported success.
|
||||
test('fill withholds keys until the tapped field raises the keyboard', async () => {
|
||||
test('fill withholds keys until the tapped point holds text-entry focus', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
// Hidden at the tap, then up two polls later: the transition our tap caused.
|
||||
world.keyboardShown = [false, false, false, true];
|
||||
// Nothing focused at first, then the tapped field takes it two polls later.
|
||||
world.activeRects = ['none', 'none', CONTAINS_TAP];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
assert.deepEqual(world.transcript, [
|
||||
'keyboard',
|
||||
'active',
|
||||
'tap',
|
||||
'active',
|
||||
'active',
|
||||
'active',
|
||||
'keys',
|
||||
]);
|
||||
});
|
||||
|
||||
// The case keyboard visibility can never decide: a second fill into a form
|
||||
// whose keyboard is already up. The keyboard says "a field is focused" both
|
||||
// before and after, so only the focused element's own geometry can tell whether
|
||||
// OUR tap moved focus.
|
||||
test('fill accepts a keyboard-up second field once focus lands on the tapped point', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.keyboardShown = [true];
|
||||
world.focusedBeforeTap = PREVIOUS_FIELD;
|
||||
world.activeRects = [CONTAINS_TAP];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
assert.deepEqual(world.transcript, ['keyboard', 'active', 'tap', 'active', 'keys']);
|
||||
});
|
||||
|
||||
// P1, the two-field misdelivery: email filled, keyboard stays up, the password
|
||||
// tap misses. Focus never leaves the email field, so a global `/keys` would
|
||||
// append the password to the EMAIL field while reporting success.
|
||||
test('fill refuses when a keyboard-up tap misses and focus stays on the previous field', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.keyboardShown = [true];
|
||||
world.focusedBeforeTap = PREVIOUS_FIELD;
|
||||
world.activeRects = [PREVIOUS_FIELD];
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.equal(error.code, 'COMMAND_FAILED');
|
||||
assert.match(error.message, /nothing there took text-entry focus/);
|
||||
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
|
||||
return true;
|
||||
});
|
||||
|
||||
// The decisive assertion: no keys reached the field the previous fill focused.
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
// Found on a live AWS Device Farm iPhone 16: tapping Safari's collapsed address
|
||||
// bar expands it into a taller field whose rect no longer covers the tapped
|
||||
// point. A containment-only rule refused a fill that plainly worked, so a tap
|
||||
// that MOVES focus counts even when the focused element has since been re-laid
|
||||
// out.
|
||||
test('fill accepts focus that moved to an element re-laid out away from the tap', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRects = ['none', MOVED_AFTER_FOCUS];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
});
|
||||
|
||||
// The other half of the rule: re-filling the field that is ALREADY focused
|
||||
// moves nothing, so only containment can tell it from a tap that missed.
|
||||
test('fill accepts a re-fill of the already-focused field it tapped', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.focusedBeforeTap = CONTAINS_TAP;
|
||||
world.activeRects = [CONTAINS_TAP];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
});
|
||||
|
||||
test('fill falls back to the keyboard transition when there is no active-element route', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRoute = 'unimplemented';
|
||||
world.keyboardShown = [false, false, true];
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'keyboard-shown');
|
||||
assert.deepEqual(world.transcript, [
|
||||
'keyboard',
|
||||
'active',
|
||||
'tap',
|
||||
'keyboard',
|
||||
'active',
|
||||
'keyboard',
|
||||
'keyboard',
|
||||
'keys',
|
||||
]);
|
||||
});
|
||||
|
||||
test('fill settles when the keyboard was already up before the tap', async () => {
|
||||
// Without an active-element route, a keyboard that was ALREADY up witnesses
|
||||
// nothing about the new field — the exact silent misdelivery P1 names.
|
||||
test('fill refuses a keyboard-up tap when the driver cannot report the focused field', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
// Back-to-back fills (email then password): a keyboard that never went down
|
||||
// cannot witness focus moving to the NEW field, so no poll can prove anything
|
||||
// — polling it would only re-observe evidence about the previous field.
|
||||
world.activeRoute = 'unimplemented';
|
||||
world.keyboardShown = [true];
|
||||
|
||||
const result = await runFill(world);
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.match(error.message, /cannot report which field holds focus/);
|
||||
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
|
||||
return true;
|
||||
});
|
||||
|
||||
assert.equal(result?.textEntryReadiness, 'settled-keyboard-up');
|
||||
assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']);
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
test('fill settles when the driver implements no keyboard route', async () => {
|
||||
test('fill settles blind only when the driver implements neither route', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRoute = 'unimplemented';
|
||||
world.keyboardRoute = 'unimplemented';
|
||||
|
||||
const result = await runFill(world);
|
||||
|
||||
// A route the driver does not have is not a failure — it just leaves nothing to wait for.
|
||||
// Routes the driver does not have are not failures — they just leave nothing
|
||||
// to wait for. This is the only path that types without evidence.
|
||||
assert.equal(result?.textEntryReadiness, 'settled-unknown');
|
||||
assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']);
|
||||
assert.deepEqual(world.transcript, ['keyboard', 'active', 'tap', 'active', 'keys']);
|
||||
});
|
||||
|
||||
// The #1658 report is "fill answered Filled N chars while the field kept its
|
||||
// placeholder". Reporting a readiness value nobody renders would preserve that,
|
||||
// so a tap that raises no keyboard refuses instead — and sends no keys, leaving
|
||||
// so a tap that focuses nothing refuses instead — and sends no keys, leaving
|
||||
// the field untouched rather than half-written.
|
||||
test('fill refuses without typing when no keyboard ever appears', async () => {
|
||||
test('fill refuses without typing when nothing ever takes focus', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.keyboardShown = [false];
|
||||
world.activeRects = ['none'];
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.equal(error.code, 'COMMAND_FAILED');
|
||||
assert.match(error.message, /no keyboard appeared, so the text was not sent/);
|
||||
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
|
||||
assert.match(error.message, /nothing there took text-entry focus/);
|
||||
return true;
|
||||
});
|
||||
|
||||
@@ -80,18 +166,53 @@ test('fill propagates a failing keyboard probe instead of typing blind', async (
|
||||
return true;
|
||||
});
|
||||
|
||||
// It threw on the pre-tap probe, so the device was never touched at all.
|
||||
assert.deepEqual(world.transcript, ['keyboard']);
|
||||
// It threw on a pre-tap probe, so the device was never touched at all.
|
||||
assert.equal(world.transcript.includes('tap'), false);
|
||||
});
|
||||
|
||||
test('fill bounds each keyboard probe well under its readiness budget', async () => {
|
||||
// Post-tap the device HAS been touched, so one transient grid error must not
|
||||
// fail a fill the next poll would have satisfied.
|
||||
test('fill tolerates a transient probe failure and still witnesses focus', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.keyboardShown = [false, true];
|
||||
world.activeFailuresAfterTap = 2;
|
||||
world.activeRects = [CONTAINS_TAP];
|
||||
|
||||
await runFill(world);
|
||||
const result = await runFill(world);
|
||||
|
||||
// Left at the client default, one hung probe could hold a 2s wait for 30s.
|
||||
assert.deepEqual(world.probeTimeouts, [1_500, 1_500]);
|
||||
assert.equal(result?.textEntryReadiness, 'focused-element');
|
||||
});
|
||||
|
||||
// ...but a budget that expires without a single answered probe is a broken
|
||||
// session, and must surface as itself rather than as "the tap missed".
|
||||
test('fill rethrows the probe failure when no probe ever answers', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRoute = 'failing';
|
||||
|
||||
await assert.rejects(runFill(world), (error: AppError) => {
|
||||
assert.match(error.message, /active element exploded/);
|
||||
return true;
|
||||
});
|
||||
|
||||
assert.equal(world.transcript.includes('keys'), false);
|
||||
});
|
||||
|
||||
// P2: a probe handed the full fixed timeout near the deadline would finish well
|
||||
// after the budget it was meant to respect, so each one is capped by what is
|
||||
// actually left.
|
||||
test('fill caps each probe timeout by the readiness budget remaining', async () => {
|
||||
const world = createTextEntryWorld();
|
||||
world.activeRects = ['none'];
|
||||
|
||||
await assert.rejects(runFill(world), () => true);
|
||||
|
||||
const timeouts = world.probeTimeouts.filter((t): t is number => t !== undefined);
|
||||
assert.equal(
|
||||
timeouts.every((t) => t <= 1_500),
|
||||
true,
|
||||
);
|
||||
// The budget is 2s and probes start at t=0, so once under 1.5s remain the cap
|
||||
// is the remainder, and the final probe gets exactly one poll interval.
|
||||
assert.equal(timeouts.at(-1), 100);
|
||||
});
|
||||
|
||||
/**
|
||||
@@ -127,23 +248,70 @@ async function runFill(world: ReturnType<typeof createTextEntryWorld>) {
|
||||
* Records the request order `fill` produces. The ordering IS the fix: a `keys`
|
||||
* entry that follows `tap` with no keyboard evidence between them is the bug.
|
||||
*/
|
||||
type FocusedElement = {
|
||||
id: string;
|
||||
rect: { x: number; y: number; width: number; height: number };
|
||||
};
|
||||
|
||||
/** `fill` is driven at (12, 24) throughout, so these read as hit and miss. */
|
||||
const CONTAINS_TAP = { id: 'tapped', rect: { x: 0, y: 0, width: 100, height: 100 } } as const;
|
||||
const PREVIOUS_FIELD = {
|
||||
id: 'previous',
|
||||
rect: { x: 200, y: 200, width: 100, height: 100 },
|
||||
} as const;
|
||||
/** Focused by our tap, but re-laid out by focusing so it no longer covers the tap point. */
|
||||
const MOVED_AFTER_FOCUS = {
|
||||
id: 'expanded',
|
||||
rect: { x: 200, y: 600, width: 180, height: 40 },
|
||||
} as const;
|
||||
|
||||
function createTextEntryWorld() {
|
||||
const world = {
|
||||
transcript: [] as string[],
|
||||
probeTimeouts: [] as Array<number | undefined>,
|
||||
keyboardShown: [] as boolean[],
|
||||
/** How the driver answers the keyboard route: normally, not at all, or with a real failure. */
|
||||
keyboardShown: [true] as boolean[],
|
||||
/** How the driver answers each route: normally, not at all, or with a real failure. */
|
||||
keyboardRoute: 'ok' as 'ok' | 'unimplemented' | 'failing',
|
||||
activeRoute: 'ok' as 'ok' | 'unimplemented' | 'failing',
|
||||
/** Successive answers from the active-element route; the last one repeats. */
|
||||
activeRects: ['none'] as Array<'none' | FocusedElement>,
|
||||
/** What held focus when the fill began — a previous fill's field, or nothing. */
|
||||
focusedBeforeTap: 'none' as 'none' | FocusedElement,
|
||||
/** Transient post-tap failures to emit before the route starts answering. */
|
||||
activeFailuresAfterTap: 0,
|
||||
tapped: false,
|
||||
client: undefined as unknown as WebDriverClient,
|
||||
};
|
||||
world.client = {
|
||||
performActions: async () => {
|
||||
world.transcript.push('tap');
|
||||
world.tapped = true;
|
||||
},
|
||||
releaseActions: async () => {},
|
||||
sendKeys: async () => {
|
||||
world.transcript.push('keys');
|
||||
},
|
||||
activeElement: async (timeoutMs?: number) => {
|
||||
world.transcript.push('active');
|
||||
world.probeTimeouts.push(timeoutMs);
|
||||
if (world.activeRoute === 'unimplemented') return 'unsupported' as const;
|
||||
if (world.activeRoute === 'failing') {
|
||||
throw new AppError('COMMAND_FAILED', 'active element exploded', { status: 500 });
|
||||
}
|
||||
// Before the tap the form is in whatever state the previous fill left it.
|
||||
if (!world.tapped) {
|
||||
return world.focusedBeforeTap === 'none'
|
||||
? ('none' as const)
|
||||
: { id: world.focusedBeforeTap.id, rect: world.focusedBeforeTap.rect };
|
||||
}
|
||||
if (world.activeFailuresAfterTap > 0) {
|
||||
world.activeFailuresAfterTap -= 1;
|
||||
throw new AppError('COMMAND_FAILED', 'transient grid blip', { status: 502 });
|
||||
}
|
||||
const next =
|
||||
world.activeRects.length > 1 ? world.activeRects.shift()! : world.activeRects[0]!;
|
||||
return next === 'none' ? ('none' as const) : { id: next.id, rect: next.rect };
|
||||
},
|
||||
isKeyboardShown: async (timeoutMs?: number) => {
|
||||
world.transcript.push('keyboard');
|
||||
world.probeTimeouts.push(timeoutMs);
|
||||
|
||||
@@ -47,25 +47,35 @@ function sleep(ms: number): Promise<void> {
|
||||
}
|
||||
|
||||
/**
|
||||
* The tap raised no keyboard within the readiness budget, so nothing focused a
|
||||
* text input and `sendKeys` would go to whatever else holds first responder —
|
||||
* or nowhere. #1658 is precisely the report of that being answered with
|
||||
* "Filled N chars", so this fails instead of typing: the caller gets a refusal
|
||||
* it must handle, and the field is left untouched rather than half-written.
|
||||
* Nothing took text-entry focus within the readiness budget, so `sendKeys`
|
||||
* would go to whatever else holds first responder — or nowhere. #1658 is
|
||||
* precisely the report of that being answered with "Filled N chars", so this
|
||||
* fails instead of typing: the caller gets a refusal it must handle, and the
|
||||
* field is left untouched rather than half-written.
|
||||
*/
|
||||
function textEntryFocusNotObservedError(x: number, y: number): AppError {
|
||||
return new AppError(
|
||||
'COMMAND_FAILED',
|
||||
`fill tapped (${x}, ${y}) but no keyboard appeared, so the text was not sent`,
|
||||
{
|
||||
reason: 'text_entry_focus_not_observed',
|
||||
x,
|
||||
y,
|
||||
hint: 'The tap most likely missed the field. Re-check the target with snapshot -i and fill the element it reports, rather than retrying the same coordinates.',
|
||||
},
|
||||
);
|
||||
function textEntryFocusNotObservedError(x: number, y: number, detail: string): AppError {
|
||||
return new AppError('COMMAND_FAILED', `fill tapped (${x}, ${y}) but ${detail}`, {
|
||||
reason: 'text_entry_focus_not_observed',
|
||||
x,
|
||||
y,
|
||||
hint: 'The tap most likely missed the field. Re-check the target with snapshot -i and fill the element it reports, rather than retrying the same coordinates.',
|
||||
});
|
||||
}
|
||||
|
||||
function rectContains(rect: WebDriverWindowRect, x: number, y: number): boolean {
|
||||
return x >= rect.x && x <= rect.x + rect.width && y >= rect.y && y <= rect.y + rect.height;
|
||||
}
|
||||
|
||||
/**
|
||||
* What text entry looked like before the tap. Both readings are only evidence
|
||||
* as a change, so both must be captured before the device is touched.
|
||||
*/
|
||||
type TextEntryStateBeforeTap = {
|
||||
keyboard: boolean | 'unsupported';
|
||||
/** `undefined` when nothing was focused, or when the driver cannot say. */
|
||||
focusedElementId: string | undefined;
|
||||
};
|
||||
|
||||
export type WebDriverInteractorOptions = {
|
||||
client: WebDriverClient;
|
||||
backend: Extract<SnapshotResult['backend'], 'android' | 'xctest'>;
|
||||
@@ -189,14 +199,15 @@ class WebDriverInteractor implements Interactor {
|
||||
_delayMs?: number,
|
||||
): Promise<Record<string, unknown>> {
|
||||
this.requireSupport('fill');
|
||||
// #1658: read the keyboard BEFORE the tap. Only a hidden -> shown
|
||||
// transition proves that OUR tap moved focus; a keyboard that was already
|
||||
// up says nothing about which field owns first responder now. A probe
|
||||
// failure throws here, before the tap, so a broken session fails a fill
|
||||
// that has not touched the device rather than typing into the dark.
|
||||
const keyboardBeforeTap = await this.readKeyboardShown();
|
||||
// #1658: both readings are taken BEFORE the tap, because each is only
|
||||
// evidence as a change. A keyboard that was already up says nothing about
|
||||
// which field owns first responder now, and a focused element is only
|
||||
// proof our tap did something if it differs from the one already focused.
|
||||
// Probe failures throw here, before the tap, so a broken session fails a
|
||||
// fill that has not touched the device rather than typing into the dark.
|
||||
const beforeTap = await this.readTextEntryState();
|
||||
await this.tap(x, y);
|
||||
const textEntryReadiness = await this.awaitTextEntryReadiness(keyboardBeforeTap, x, y);
|
||||
const textEntryReadiness = await this.awaitTextEntryReadiness(beforeTap, x, y);
|
||||
await this.type(text);
|
||||
return { backend: 'webdriver', x, y, text, textEntryReadiness };
|
||||
}
|
||||
@@ -316,40 +327,115 @@ class WebDriverInteractor implements Interactor {
|
||||
* only the round trip between them gave the field time to focus.
|
||||
*/
|
||||
private async awaitTextEntryReadiness(
|
||||
keyboardBeforeTap: boolean | 'unsupported',
|
||||
beforeTap: TextEntryStateBeforeTap,
|
||||
x: number,
|
||||
y: number,
|
||||
): Promise<CloudTextEntryReadiness> {
|
||||
// The driver implements no keyboard route, so no wait can learn anything.
|
||||
// Keep a blind path cheap: settle briefly, then type. This is the ONE case
|
||||
// that types without evidence, and it takes a positively classified
|
||||
// unsupported answer to reach — never a probe that merely failed.
|
||||
const keyboardBeforeTap = beforeTap.keyboard;
|
||||
const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS;
|
||||
// Strongest evidence first: the focused element itself. Keyboard visibility
|
||||
// can only witness that *a* field took focus, never *which* — which is why
|
||||
// filling a second field in an already-open form could send its keys to the
|
||||
// first one.
|
||||
//
|
||||
// The question is whether OUR tap moved focus, so compare against what held
|
||||
// it before. Identity rather than geometry, because focusing a field can
|
||||
// re-lay it out: tapping Safari's collapsed address bar expands it into a
|
||||
// taller search field that no longer contains the point that was tapped, so
|
||||
// a containment-only rule refuses a fill that plainly worked. Containment
|
||||
// still earns its place as the second half of the test — re-filling the
|
||||
// field that is ALREADY focused moves nothing, and only geometry can tell
|
||||
// that from a tap that missed everything.
|
||||
const focusedBeforeTap = beforeTap.focusedElementId;
|
||||
const focus = await this.pollWithinBudget(deadline, async (timeoutMs) => {
|
||||
const active = await this.client.activeElement(timeoutMs);
|
||||
if (active === 'unsupported') return 'abandon';
|
||||
if (active === 'none') return 'retry';
|
||||
const moved = active.id !== focusedBeforeTap;
|
||||
return moved || rectContains(active.rect, x, y) ? 'satisfied' : 'retry';
|
||||
});
|
||||
if (focus === 'satisfied') return 'focused-element';
|
||||
if (focus === 'expired') {
|
||||
throw textEntryFocusNotObservedError(
|
||||
x,
|
||||
y,
|
||||
'nothing there took text-entry focus, so the text was not sent',
|
||||
);
|
||||
}
|
||||
|
||||
// No active-element route. Fall back to keyboard evidence, which can still
|
||||
// witness focus arriving when the keyboard was down before our tap.
|
||||
if (keyboardBeforeTap === 'unsupported') {
|
||||
// The driver implements neither route, so no wait can learn anything.
|
||||
// Keep a blind path cheap: settle briefly, then type. This is the ONE
|
||||
// case that types without evidence, and it takes two positively
|
||||
// classified unsupported answers to reach — never a probe that failed.
|
||||
await sleep(TEXT_ENTRY_BLIND_SETTLE_MS);
|
||||
return 'settled-unknown';
|
||||
}
|
||||
// The keyboard was ALREADY up — back-to-back fills into one form, the shape
|
||||
// that failed most often in #1658. Its visibility cannot witness focus
|
||||
// moving to the NEW field, so polling it would resolve instantly on
|
||||
// evidence about the PREVIOUS one. The Apple runner spends its whole
|
||||
// readiness budget on exactly this case rather than race the app with a
|
||||
// short settle; the cloud path has even more reason to, because its keys go
|
||||
// to whatever holds first responder rather than to a scoped element.
|
||||
if (keyboardBeforeTap) {
|
||||
await sleep(TEXT_ENTRY_READINESS_TIMEOUT_MS);
|
||||
return 'settled-keyboard-up';
|
||||
// Keyboard already up and no way to ask which field owns it: every signal
|
||||
// available here describes the PREVIOUS field. Typing anyway is exactly
|
||||
// #1658's silent misdelivery, so refuse instead.
|
||||
throw textEntryFocusNotObservedError(
|
||||
x,
|
||||
y,
|
||||
'the keyboard was already up and this driver cannot report which field holds focus, so the text was not sent',
|
||||
);
|
||||
}
|
||||
const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS;
|
||||
while (Date.now() < deadline) {
|
||||
await sleep(TEXT_ENTRY_READINESS_POLL_MS);
|
||||
if ((await this.readKeyboardShown()) === true) return 'keyboard-shown';
|
||||
}
|
||||
throw textEntryFocusNotObservedError(x, y);
|
||||
const keyboard = await this.pollWithinBudget(deadline, async (timeoutMs) =>
|
||||
(await this.client.isKeyboardShown(timeoutMs)) === true ? 'satisfied' : 'retry',
|
||||
);
|
||||
if (keyboard === 'satisfied') return 'keyboard-shown';
|
||||
throw textEntryFocusNotObservedError(x, y, 'no keyboard appeared, so the text was not sent');
|
||||
}
|
||||
|
||||
/** Bounded by the readiness budget it serves, not the client's default request timeout. */
|
||||
private async readKeyboardShown(): Promise<boolean | 'unsupported'> {
|
||||
return await this.client.isKeyboardShown(TEXT_ENTRY_PROBE_TIMEOUT_MS);
|
||||
/** Both pre-tap readings, taken together so neither is charged an extra tap-to-probe gap. */
|
||||
private async readTextEntryState(): Promise<TextEntryStateBeforeTap> {
|
||||
const [keyboard, active] = await Promise.all([
|
||||
this.client.isKeyboardShown(TEXT_ENTRY_PROBE_TIMEOUT_MS),
|
||||
this.client.activeElement(TEXT_ENTRY_PROBE_TIMEOUT_MS),
|
||||
]);
|
||||
return {
|
||||
keyboard,
|
||||
focusedElementId: active === 'none' || active === 'unsupported' ? undefined : active.id,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Polls until `probe` is satisfied or the readiness budget runs out, bounding
|
||||
* BOTH the probe's own transport timeout and the sleep between attempts by
|
||||
* what remains — a probe handed the full fixed timeout near the deadline
|
||||
* would otherwise finish well after the budget it was meant to respect.
|
||||
*
|
||||
* A probe that throws is tolerated: the loop has further attempts, and one
|
||||
* transient grid error should not fail a fill that would have succeeded. But
|
||||
* a budget that expires without a single answered probe rethrows, so a dead
|
||||
* session or an auth rejection surfaces as itself instead of being reported
|
||||
* as a tap that missed its field.
|
||||
*/
|
||||
private async pollWithinBudget(
|
||||
deadline: number,
|
||||
probe: (timeoutMs: number) => Promise<'satisfied' | 'abandon' | 'retry'>,
|
||||
): Promise<'satisfied' | 'abandon' | 'expired'> {
|
||||
let answered = false;
|
||||
let lastError: unknown;
|
||||
for (;;) {
|
||||
const remaining = deadline - Date.now();
|
||||
if (remaining <= 0) break;
|
||||
try {
|
||||
const verdict = await probe(Math.min(TEXT_ENTRY_PROBE_TIMEOUT_MS, remaining));
|
||||
answered = true;
|
||||
if (verdict !== 'retry') return verdict;
|
||||
} catch (error) {
|
||||
lastError = error;
|
||||
}
|
||||
const left = deadline - Date.now();
|
||||
if (left <= 0) break;
|
||||
await sleep(Math.min(TEXT_ENTRY_READINESS_POLL_MS, left));
|
||||
}
|
||||
if (!answered && lastError !== undefined) throw lastError;
|
||||
return 'expired';
|
||||
}
|
||||
|
||||
private async pointerGesture(name: string, actions: W3CPointerAction[]): Promise<void> {
|
||||
|
||||
@@ -76,20 +76,31 @@ test('cloud iOS snapshot captures through the provider session after open', asyn
|
||||
test('cloud iOS fill lands text in the field it tapped', async () => {
|
||||
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
|
||||
const lease = await openCloudIosSession(daemon);
|
||||
// Focus lands a poll late, as a WebView input's does. The keys must wait.
|
||||
server.pollsUntilFocus = 2;
|
||||
|
||||
const response = await daemon.callCommand(
|
||||
'fill',
|
||||
['editable=true', 'user@example.com'],
|
||||
['label=Email', 'user@example.com'],
|
||||
leaseFlags(lease.leaseId),
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
|
||||
const data = assertRpcOk<{ text?: string; textEntryReadiness?: string }>(response);
|
||||
assert.equal(data.text, 'user@example.com');
|
||||
assert.equal(data.textEntryReadiness, 'keyboard-shown');
|
||||
assert.equal(data.textEntryReadiness, 'focused-element');
|
||||
// The keys reached the focused field, not the void.
|
||||
assert.equal(server.fieldValue, 'user@example.com');
|
||||
assert.deepEqual(textEntryTranscript(server), ['keyboard', 'tap', 'keyboard', 'keys']);
|
||||
assert.equal(server.fieldValues.email, 'user@example.com');
|
||||
// Both pre-tap readings, the tap, then polling until focus lands. The keys
|
||||
// come last — that ordering IS the fix.
|
||||
assert.deepEqual(textEntryTranscript(server), [
|
||||
'keyboard',
|
||||
'active',
|
||||
'tap',
|
||||
'active',
|
||||
'active',
|
||||
'keys',
|
||||
]);
|
||||
|
||||
// And the device agrees: the field reports the typed value back.
|
||||
const snapshot = await daemon.callCommand(
|
||||
@@ -111,23 +122,54 @@ test('cloud iOS fill lands text in the field it tapped', async () => {
|
||||
* "Filled N chars" is the original bug, so the fill refuses — and sends no keys,
|
||||
* leaving the field untouched rather than half-written.
|
||||
*/
|
||||
test('cloud iOS fill refuses, without typing, when the tap raises no keyboard', async () => {
|
||||
test('cloud iOS fill refuses, without typing, when nothing takes focus', async () => {
|
||||
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
|
||||
const lease = await openCloudIosSession(daemon);
|
||||
server.keyboardPollsUntilShown = Number.POSITIVE_INFINITY;
|
||||
server.pollsUntilFocus = Number.POSITIVE_INFINITY;
|
||||
|
||||
const response = await daemon.callCommand(
|
||||
'fill',
|
||||
['editable=true', 'user@example.com'],
|
||||
['label=Email', 'user@example.com'],
|
||||
leaseFlags(lease.leaseId),
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
|
||||
const error = response.json.error;
|
||||
assert.ok(error, 'expected fill to refuse');
|
||||
assert.match(error.message ?? '', /no keyboard appeared, so the text was not sent/);
|
||||
assert.match(error.message ?? '', /nothing there took text-entry focus/);
|
||||
assert.equal(error.data?.details?.reason, 'text_entry_focus_not_observed');
|
||||
assert.equal(server.fieldValue, '');
|
||||
assert.equal(server.fieldValues.email, '');
|
||||
assert.equal(textEntryTranscript(server).includes('keys'), false);
|
||||
});
|
||||
}, 15_000);
|
||||
|
||||
/**
|
||||
* The misdelivery keyboard visibility can never catch (#1658 P1). The form is
|
||||
* already filled and focused — keyboard up — and the second fill's tap misses
|
||||
* the password field. Focus therefore never leaves `email`, so a `POST /keys`
|
||||
* here would append the password to the EMAIL field and every request would
|
||||
* still return 200. The fill must refuse before sending anything.
|
||||
*/
|
||||
test('cloud iOS fill sends no keys when a keyboard-up tap misses the second field', async () => {
|
||||
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
|
||||
const lease = await openCloudIosSession(daemon);
|
||||
server.focusField('email', 'user@example.com');
|
||||
// Focus never moves, reproducing a tap that landed outside both fields.
|
||||
server.pollsUntilFocus = Number.POSITIVE_INFINITY;
|
||||
|
||||
const response = await daemon.callCommand(
|
||||
'fill',
|
||||
['label=Password', 'hunter2'],
|
||||
leaseFlags(lease.leaseId),
|
||||
{ meta: leaseMeta(lease.leaseId) },
|
||||
);
|
||||
|
||||
assert.ok(response.json.error, 'expected fill to refuse');
|
||||
assert.equal(response.json.error.data?.details?.reason, 'text_entry_focus_not_observed');
|
||||
// The decisive assertion: the previous field is untouched and the password
|
||||
// never reached it.
|
||||
assert.equal(server.fieldValues.email, 'user@example.com');
|
||||
assert.equal(server.fieldValues.password, '');
|
||||
assert.equal(textEntryTranscript(server).includes('keys'), false);
|
||||
});
|
||||
}, 15_000);
|
||||
@@ -138,6 +180,7 @@ function textEntryTranscript(server: FakeIosWebDriverServer): string[] {
|
||||
'POST /wd/hub/session/wd-1/actions': 'tap',
|
||||
'POST /wd/hub/session/wd-1/keys': 'keys',
|
||||
'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': 'keyboard',
|
||||
'GET /wd/hub/session/wd-1/element/active': 'active',
|
||||
};
|
||||
return server.calls
|
||||
.map((call) => labels[`${call.method} ${call.path}`])
|
||||
@@ -195,18 +238,38 @@ async function openCloudIosSession(daemon: ProviderScenarioHarness): Promise<Dev
|
||||
* That last part is the #1658 bug reproduced faithfully: keys sent too early
|
||||
* are accepted by the grid and dropped by the device.
|
||||
*/
|
||||
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
|
||||
/** Two stacked inputs, as a login form has. Geometry is what decides focus. */
|
||||
const FIELDS = {
|
||||
email: { x: 20, y: 300, width: 350, height: 44 },
|
||||
password: { x: 20, y: 380, width: 350, height: 44 },
|
||||
} as const;
|
||||
type FieldName = keyof typeof FIELDS;
|
||||
|
||||
class FakeIosWebDriverServer extends CloudWebDriverTestServer {
|
||||
/** How many keyboard probes after the tap before focus lands. Infinity = the tap missed. */
|
||||
keyboardPollsUntilShown = 1;
|
||||
fieldValue = '';
|
||||
/** How many probes after a tap before focus settles. Infinity = focus never moves. */
|
||||
pollsUntilFocus = 1;
|
||||
fieldValues: Record<FieldName, string> = { email: '', password: '' };
|
||||
focused: FieldName | undefined;
|
||||
|
||||
private pollsRemaining = Number.POSITIVE_INFINITY;
|
||||
private pendingFocus: FieldName | undefined;
|
||||
|
||||
static async start(): Promise<StartedCloudWebDriverTestServer<FakeIosWebDriverServer>> {
|
||||
return await startCloudWebDriverTestServer(new FakeIosWebDriverServer());
|
||||
}
|
||||
|
||||
/** Puts the form in the state a completed first fill leaves it in. */
|
||||
focusField(field: FieldName, value: string): void {
|
||||
this.focused = field;
|
||||
this.fieldValues[field] = value;
|
||||
this.pollsRemaining = 0;
|
||||
this.pendingFocus = field;
|
||||
}
|
||||
|
||||
protected respond(call: CloudWebDriverHttpCall) {
|
||||
const rectMatch = /^\/wd\/hub\/session\/wd-1\/element\/([^/]+)\/rect$/.exec(call.path);
|
||||
if (call.method === 'GET' && rectMatch) return this.elementRect(rectMatch[1]!);
|
||||
const route = this.routes[`${call.method} ${call.path}`];
|
||||
return route ? route() : cloudWebDriverTestJson({ value: null });
|
||||
}
|
||||
@@ -222,47 +285,113 @@ class FakeIosWebDriverServer extends CloudWebDriverTestServer {
|
||||
cloudWebDriverTestJson({ value: { x: 0, y: 0, width: 390, height: 844 } }),
|
||||
'POST /wd/hub/session/wd-1/actions': () => this.tap(),
|
||||
'POST /wd/hub/session/wd-1/keys': () => this.sendKeys(),
|
||||
'GET /wd/hub/session/wd-1/element/active': () => this.activeElement(),
|
||||
'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': () =>
|
||||
cloudWebDriverTestJson({ value: this.keyboardShown() }),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A tap moves focus only if it actually lands inside a field. A miss leaves
|
||||
* the previous field focused — which is exactly how keys can reach the wrong
|
||||
* one while every request still succeeds.
|
||||
*/
|
||||
private tap(): CloudWebDriverTestResponse {
|
||||
this.pollsRemaining = this.keyboardPollsUntilShown;
|
||||
const point = this.lastTapPoint();
|
||||
const hit =
|
||||
point && (Object.keys(FIELDS) as FieldName[]).find((n) => contains(FIELDS[n], point));
|
||||
this.pendingFocus = hit ?? this.focused;
|
||||
this.pollsRemaining = hit ? this.pollsUntilFocus : Number.POSITIVE_INFINITY;
|
||||
if (hit) this.focused = undefined;
|
||||
return cloudWebDriverTestJson({ value: null });
|
||||
}
|
||||
|
||||
/** Focus is not instant: it lands only after the configured number of probes. */
|
||||
private keyboardShown(): boolean {
|
||||
if (this.pollsRemaining <= 0) return true;
|
||||
/** Focus is not instant: it settles only after the configured number of probes. */
|
||||
private settle(): void {
|
||||
if (this.pollsRemaining <= 0) return;
|
||||
this.pollsRemaining -= 1;
|
||||
return this.pollsRemaining <= 0;
|
||||
if (this.pollsRemaining <= 0) this.focused = this.pendingFocus;
|
||||
}
|
||||
|
||||
private activeElement(): CloudWebDriverTestResponse {
|
||||
this.settle();
|
||||
if (!this.focused) {
|
||||
return cloudWebDriverTestJson(
|
||||
{ value: { error: 'no such element', message: 'nothing is focused' } },
|
||||
404,
|
||||
);
|
||||
}
|
||||
return cloudWebDriverTestJson({ value: { [W3C_ELEMENT_KEY]: this.focused } });
|
||||
}
|
||||
|
||||
private elementRect(elementId: string): CloudWebDriverTestResponse {
|
||||
const rect = FIELDS[elementId as FieldName];
|
||||
if (!rect) {
|
||||
return cloudWebDriverTestJson(
|
||||
{ value: { error: 'no such element', message: `unknown element ${elementId}` } },
|
||||
404,
|
||||
);
|
||||
}
|
||||
return cloudWebDriverTestJson({ value: rect });
|
||||
}
|
||||
|
||||
private keyboardShown(): boolean {
|
||||
this.settle();
|
||||
return this.focused !== undefined;
|
||||
}
|
||||
|
||||
private sendKeys(): CloudWebDriverTestResponse {
|
||||
// Unfocused keys are accepted by the grid and land nowhere — the bug's shape.
|
||||
if (this.pollsRemaining <= 0) {
|
||||
const text = this.lastSentKeys();
|
||||
if (text !== undefined) this.fieldValue = text;
|
||||
}
|
||||
// Unfocused keys are accepted by the grid and land nowhere; keys sent while
|
||||
// the WRONG field holds focus land in that field. Both are the bug's shape.
|
||||
const text = this.lastSentKeys();
|
||||
if (this.focused && text !== undefined) this.fieldValues[this.focused] = text;
|
||||
return cloudWebDriverTestJson({ value: null });
|
||||
}
|
||||
|
||||
private lastTapPoint(): { x: number; y: number } | undefined {
|
||||
const body = this.calls.at(-1)?.body as { actions?: unknown } | undefined;
|
||||
const sequences = Array.isArray(body?.actions) ? body.actions : [];
|
||||
for (const sequence of sequences) {
|
||||
const actions = (sequence as { actions?: unknown }).actions;
|
||||
if (!Array.isArray(actions)) continue;
|
||||
const move = actions.find((a) => (a as { type?: string }).type === 'pointerMove') as
|
||||
| { x?: unknown; y?: unknown }
|
||||
| undefined;
|
||||
if (typeof move?.x === 'number' && typeof move.y === 'number') {
|
||||
return { x: move.x, y: move.y };
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
private lastSentKeys(): string | undefined {
|
||||
const body = this.calls.at(-1)?.body as { value?: unknown } | undefined;
|
||||
return Array.isArray(body?.value) ? body.value.join('') : undefined;
|
||||
}
|
||||
|
||||
private source(): string {
|
||||
const field = (name: FieldName, label: string) =>
|
||||
`<XCUIElementTypeTextField name="${name}" label="${label}" value="${this.fieldValues[name]}" ` +
|
||||
`x="${FIELDS[name].x}" y="${FIELDS[name].y}" width="${FIELDS[name].width}" ` +
|
||||
`height="${FIELDS[name].height}" visible="true" enabled="true" />`;
|
||||
return (
|
||||
'<XCUIElementTypeApplication name="Demo" x="0" y="0" width="390" height="844" visible="true">' +
|
||||
`<XCUIElementTypeTextField name="email" label="Email" value="${this.fieldValue}" ` +
|
||||
'x="20" y="300" width="350" height="44" visible="true" enabled="true" />' +
|
||||
field('email', 'Email') +
|
||||
field('password', 'Password') +
|
||||
'</XCUIElementTypeApplication>'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function contains(
|
||||
rect: { x: number; y: number; width: number; height: number },
|
||||
p: { x: number; y: number },
|
||||
): boolean {
|
||||
return (
|
||||
p.x >= rect.x && p.x <= rect.x + rect.width && p.y >= rect.y && p.y <= rect.y + rect.height
|
||||
);
|
||||
}
|
||||
|
||||
function leaseFlags(leaseId?: string): DaemonRequest['flags'] {
|
||||
return {
|
||||
platform: 'ios',
|
||||
|
||||
@@ -355,5 +355,5 @@ BrowserStack can return session video, Appium logs, device logs, dashboard URL,
|
||||
- If AWS connect fails, use its reported `aws devicefarm get-*` error to check the credential chain, ARN, region, resource platform, or upload readiness. Allocation has not happened yet.
|
||||
- If Limrun connect fails, check `LIMRUN_API_KEY` and the optional `LIMRUN_REGION`. A successful connect verifies the selected instance service without creating an instance.
|
||||
- If artifact lookup is pending immediately after `close`, retry `agent-device artifacts --json`. Some providers finalize video/log URLs asynchronously after the hosted session stops.
|
||||
- On hosted WebDriver sessions (BrowserStack, AWS Device Farm), `fill` waits for the tapped field to take text entry focus before sending its keys, and reports what it observed as `textEntryReadiness`: `keyboard-shown` (the keyboard came up in response to the tap), `settled-keyboard-up` (the keyboard was already up — back-to-back fills into one form — so it could not witness the new field taking focus and the fill waited out its full readiness budget), or `settled-unknown` (the driver implements no keyboard route, so there was nothing to wait for).
|
||||
- If the tap raises no keyboard at all, `fill` **fails** with `text_entry_focus_not_observed` and sends no keys, rather than reporting a filled field that never received the text. That almost always means the tap missed: re-check the target with `snapshot -i` and fill the element it reports instead of retrying the same coordinates.
|
||||
- On hosted WebDriver sessions (BrowserStack, AWS Device Farm), `fill` waits until the field it tapped actually holds text entry focus before sending its keys, and reports what it observed as `textEntryReadiness`: `focused-element` (the focused element contains the tapped point — the strongest signal, and the only one that identifies *which* field took focus), `keyboard-shown` (the driver has no active-element route, but the keyboard rose from hidden after the tap), or `settled-unknown` (the driver implements neither route, so there was nothing to wait for).
|
||||
- If focus cannot be witnessed, `fill` **fails** with `text_entry_focus_not_observed` and sends no keys, rather than reporting a filled field that never received the text. That almost always means the tap missed: re-check the target with `snapshot -i` and fill the element it reports instead of retrying the same coordinates. The same refusal covers a second fill into a form whose keyboard is already up on a driver that cannot name the focused field — there, unwitnessed keys would land in the field the previous fill focused.
|
||||
|
||||
Reference in New Issue
Block a user