fix(cloud): witness the tapped field's focus before a cloud fill types

Review of cc23f2b found two ways a fill could still report success
without evidence that OUR tap focused the field it was aimed at.

P1. `settled-keyboard-up` and `settled-unknown` both typed and returned
normal success. Keyboard visibility can only witness that *a* field took
focus, never *which*: filling a second field in an already-open form
reads the same before and after, so a missed tap left the first field
focused and `POST /keys` — which the driver routes to whatever holds
first responder — appended to it while every request returned 200.

Failing those closed outright would have broken ordinary multi-field
form fills, which do work: a live AWS Device Farm run types both fields
of a WebView login correctly. So witness focus properly instead. W3C
`GET /element/active` answers the question keyboard visibility cannot —
is the thing focused now the thing I tapped — and answers it whether or
not the keyboard was already up. That becomes the primary signal
(`focused-element`); the keyboard transition stays as the fallback for
drivers without the route, and a keyboard already up on such a driver
now refuses rather than typing.

The test is identity, not geometry. Containment of the tap point looks
like the obvious rule and is wrong: focusing a field can re-lay it out.
On a live iPhone 16, tapping Safari's collapsed address bar expands it
into a taller field that no longer covers the tapped point, and a
containment-only rule refused a fill that plainly worked. So a tap that
MOVES focus counts, with containment as the second half of the test —
re-filling the already-focused field moves nothing, and only geometry
tells that from a tap that missed. Both readings are taken before the
tap, since each is evidence only as a change.

P2. The 2s budget bounded the loop but not the calls inside it: every
probe got a fixed 1500ms, so one begun near the deadline finished well
past it. Both the probe timeout and the sleep are now capped by the
remaining budget.

Also fixes a related escape the review did not name: the poll loop had
no catch, so one transient grid error aborted a fill the next poll would
have satisfied. Probe failures are now tolerated within the budget, but
a budget that expires without a single answered probe rethrows, so a
dead session surfaces as itself rather than as "the tap missed".

The provider scenario gains the two-field case the review asked for: it
begins keyboard-up with the email field focused, misses the password
tap, and asserts no keys reach the email field.

Verified on AWS Device Farm iPhone 16 / iOS 18.0 at this exact tree:
address bar (the re-layout case) and both WebView login fields all
report `focused-element`, the second with the keyboard already up, and
the device reads back `tomsmith` and a 20-character password.
This commit is contained in:
Michał Pierzchała
2026-08-07 14:06:56 +02:00
parent 8af981e84b
commit 3a9aceb935
8 changed files with 695 additions and 116 deletions
+1 -1
View File
@@ -4,7 +4,7 @@
- `agent-device help workflow` is now a compact ~8KB card instead of a ~41KB dump; the same depth still exists, split into `help scripting` (save-script, secret-safe fills, batch JSON, replay divergence/repair, recording) and `help gestures` (multi-touch shapes and platform quirks), plus a few paragraphs folded into the topics that already owned the subject (`help debugging`, `help physical-device`, `help validate`). Every `help <topic>` first line is now `agent-device <version> — <topic>` so an agent can read the installed version from its mandatory first help read instead of a separate `agent-device --version` call.
- Cloud iOS (BrowserStack, AWS Device Farm): `snapshot` and `diff` no longer fail with `SESSION_NOT_FOUND` on a live provider session (#1658). The app-session guard they ran belongs to the local XCUITest runner, which must attach to a target app; a cloud capture reads the provider's own driver session and needs no app identity, so the guard now applies to local Apple targets only. Relatedly, a cloud iOS `open com.example.app` now records that bundle id on the session — the provider path skips local app resolution (no simctl/devicectl reaches a hosted device), and used to drop an explicitly spelled bundle id along with it, leaving the session with no app identity at all. Opening a second bundle id replaces the first, matching the local path, where an explicitly spelled target always wins over the session's current app; deep links, display names, and bare `open` still keep the app already tracked.
- Cloud `fill` (BrowserStack, AWS Device Farm) waits for the tapped field to take text-entry focus before sending its keys, instead of dispatching both in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The response now discloses `textEntryReadiness`: `keyboard-shown`, `settled-keyboard-up` (back-to-back fills into one form, where keyboard visibility cannot witness the new field taking focus, so the fill waits out the same readiness budget the Apple runner spends there), or `settled-unknown` (the driver implements no keyboard route at all). Breaking: when the tap raises no keyboard within that budget, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. A driver that cannot report keyboard state at all is unaffected (it blind-settles as before); only a positively classified unimplemented route counts as such, so a dead session, an auth rejection, or a grid outage now surfaces instead of degrading into a blind text entry.
- Cloud `fill` (BrowserStack, AWS Device Farm) now witnesses that the field it tapped actually holds text-entry focus before sending its keys, instead of dispatching tap and keys in back-to-back requests (#1658). A WebView input — an OAuth/SSO page in a Safari view controller, for example — takes first responder asynchronously, so the keys used to land with nothing focused while `fill` still answered "Filled N chars"; tapping and filling as two separate commands worked only because the round trip between them gave the field time to focus. The witness is the focused element's own geometry: `fill` polls the active element and proceeds only once it contains the point it tapped, which is the one signal that identifies *which* field took focus. Keyboard visibility cannot — it reads the same before and after a second fill into an already-open form, so it could not tell a focused password field from the email field the previous fill left focused. The response discloses `textEntryReadiness`: `focused-element`, `keyboard-shown` (no active-element route, but the keyboard rose from hidden after the tap), or `settled-unknown` (the driver implements neither route, the only path that types without evidence). Breaking: when focus cannot be witnessed, cloud `fill` now FAILS with `COMMAND_FAILED` / `text_entry_focus_not_observed` and sends no keys, instead of typing into whatever holds first responder and answering "Filled N chars" — a fill with no witness must not read as a filled field. That covers a tap that focused nothing and a keyboard already up on a driver that cannot name the focused field. Only a positively classified unimplemented route counts as unsupported, so a dead session, an auth rejection, or a grid outage surfaces instead of degrading into a blind text entry.
- `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands.
- Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags.
- `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied.
+20 -14
View File
@@ -102,29 +102,35 @@ export type TypeTextBackendResult = {
* What a cloud-WebDriver `fill` could establish about its target taking text
* entry focus before it sent the keys (#1658). The local Apple runner owns this
* discipline in Swift and reports it as a `textEntryRoute`; the cloud path has
* no runner, only the driver's keyboard-visibility signal, so it names what it
* observed instead of which channel it typed through:
* no runner, only what the driver will tell it, so it names what it observed
* instead of which channel it typed through. Listed strongest evidence first:
*
* - `keyboard-shown`: the software keyboard went from hidden to shown after our
* tap — positive evidence the tap moved focus to a text input.
* - `settled-keyboard-up`: the keyboard was ALREADY up (back-to-back fills into
* one form), so its visibility says nothing about the NEW field. Entry waited
* the full readiness budget, the same answer the Apple runner gives here.
* - `settled-unknown`: the driver implements no keyboard route at all, so
* nothing could be waited for. Entry followed a short blind settle.
* - `focused-element`: the element holding text-entry focus after the tap
* contains the tapped point — positive evidence that THIS field, not merely
* some field, took focus.
* - `keyboard-shown`: no active-element route, but the software keyboard went
* from hidden to shown after our tap. Witnesses that focus arrived somewhere,
* which only means our field when the keyboard was down beforehand.
* - `settled-unknown`: the driver implements neither an active-element nor a
* keyboard route, so nothing could be waited for. Entry followed a short
* blind settle. This is the only value that describes a fill with no evidence
* behind it, and it takes two positively classified unsupported answers to
* reach.
*
* Every value describes a fill that DID send its keys. The fourth outcome — a
* tap that raised no keyboard at all — is deliberately absent: reporting it
* here would recreate #1658's silent success, so the interactor throws
* (`text_entry_focus_not_observed`) without typing instead.
* Every value describes a fill that DID send its keys. Outcomes where focus
* could not be witnessed are deliberately absent: reporting one here would
* recreate #1658's silent success, so the interactor throws
* (`text_entry_focus_not_observed`) without typing instead. That covers a tap
* that focused nothing, and a keyboard already up on a driver that cannot say
* which field owns it — the case where keys would land in the PREVIOUS field.
*
* Closed set: the cloud interactor is its only producer and the boundary that
* narrows it (readFillBackendResult, src/core/dispatch-interactions.ts) drops a
* value it cannot name.
*/
export const CLOUD_TEXT_ENTRY_READINESS = [
'focused-element',
'keyboard-shown',
'settled-keyboard-up',
'settled-unknown',
] as const;
@@ -74,7 +74,119 @@ test('is_keyboard_shown honors a caller timeout shorter than the client default'
});
});
/** A client with a live session id, so `isKeyboardShown` reaches the session route. */
// The focused element is the only signal that can say WHICH field took focus,
// so `fill` fails closed when it is unavailable. Each wire shape it
// distinguishes is pinned here.
test('activeElement returns the focused element id and rect', async () => {
const client = await connectedClient(
webDriverRoutes({
'/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } },
'/rect': { body: { value: { x: 20, y: 300, width: 350, height: 44 } } },
}),
);
assert.deepEqual(await client.activeElement(), {
id: 'el-1',
rect: { x: 20, y: 300, width: 350, height: 44 },
});
});
// A form with nothing focused yet is an expected state, not a broken driver —
// the caller must keep polling rather than give up.
test('activeElement reports nothing focused as none', async () => {
const client = await connectedClient(
webDriverRoutes({
'/element/active': {
status: 404,
body: { value: { error: 'no such element', message: 'nothing focused' } },
},
}),
);
assert.equal(await client.activeElement(), 'none');
});
// Focus moved between the two round trips. A race the caller can simply retry
// out of, not a failure worth ending the fill on.
test('activeElement reports an element that vanished before its rect as none', async () => {
const client = await connectedClient(
webDriverRoutes({
'/element/active': { body: { value: { [W3C_ELEMENT_KEY]: 'el-1' } } },
'/rect': {
status: 404,
body: { value: { error: 'no such element', message: 'stale element' } },
},
}),
);
assert.equal(await client.activeElement(), 'none');
});
// A grid that answers 200/null is not implementing the route: the spec reports
// "nothing focused" as a `no such element` error. Reading null as "nothing
// focused" would fail every fill on such a grid instead of falling back.
test('activeElement reports a non-element answer as unsupported', async () => {
const client = await connectedClient(
webDriverRoutes({ '/element/active': { body: { value: null } } }),
);
assert.equal(await client.activeElement(), 'unsupported');
});
test('activeElement reports an unimplemented route as unsupported', async () => {
const client = await connectedClient(
webDriverRoutes({
'/element/active': {
status: 404,
body: { value: { error: 'unknown command', message: 'Unknown command' } },
},
}),
);
assert.equal(await client.activeElement(), 'unsupported');
});
// Same discipline as the keyboard route: a dead session shares the 404 and must
// never be mistaken for a driver that simply lacks the feature.
test('activeElement propagates a dead session, which shares the 404', async () => {
const client = await connectedClient(
webDriverRoutes({
'/element/active': {
status: 404,
body: {
value: { error: 'invalid session id', message: 'A session is either terminated' },
},
},
}),
);
await assert.rejects(client.activeElement(), (error: AppError) => {
assert.match(error.message, /terminated/);
return true;
});
});
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
/** Dispatches by URL suffix, since `activeElement` makes two round trips. */
function webDriverRoutes(
routes: Record<string, { status?: number; body: unknown }>,
): typeof globalThis.fetch {
const keys = Object.keys(routes).sort((a, b) => b.length - a.length);
return async (input) => {
const url = String(input instanceof Request ? input.url : input);
const key = keys.find((candidate) => url.includes(candidate));
const route = key
? routes[key]!
: { status: 500, body: { value: { message: `unrouted ${url}` } } };
return new Response(JSON.stringify(route.body), {
status: route.status ?? 200,
headers: { 'Content-Type': 'application/json' },
});
};
}
/** A client with a live session id, so the probes reach the session routes. */
async function connectedClient(respond: typeof globalThis.fetch): Promise<WebDriverClient> {
globalThis.fetch = webDriverResponse(200, {
value: { sessionId: 'wd-1', capabilities: {} },
@@ -35,6 +35,12 @@ export type WebDriverWindowRect = {
height: number;
};
export type WebDriverActiveElement = {
/** Opaque W3C element reference; only ever compared for equality. */
id: string;
rect: WebDriverWindowRect;
};
export type W3CPointerAction =
| {
type: 'pointerMove';
@@ -174,6 +180,58 @@ export class WebDriverClient {
return typeof value === 'boolean' ? value : 'unsupported';
}
/**
* Which element holds text-entry focus and where it sits, `none` when nothing
* does, or `unsupported` when this driver has no active-element route.
*
* Keyboard visibility can only witness that *a* field took focus; it says
* nothing about *which*, which is why a second fill into an already-open form
* could send its keys to the field the first one focused (#1658). The focused
* element's identity answers that. Its rect comes along because identity
* alone cannot distinguish a deliberate re-fill of the already-focused field
* from a tap that moved nothing.
*
* `no such element` is a real, expected answer here (a form with nothing
* focused yet), so it maps to `none` and leaves the caller polling. A route
* that answers 200 with something that is not an element reference is not
* implementing this at all — the spec reports "nothing focused" as that
* error, never as an empty value — so it degrades to `unsupported` alongside
* a genuinely unimplemented route, the same way a non-boolean keyboard answer
* does. Reading it as `none` would fail every fill on such a grid.
*/
async activeElement(
timeoutMs?: number,
): Promise<WebDriverActiveElement | 'none' | 'unsupported'> {
const overrides = { retryAttempts: 0, ...(timeoutMs === undefined ? {} : { timeoutMs }) };
let elementId: string | undefined;
try {
elementId = readW3CElementId(
await this.sessionRequest('GET', '/element/active', undefined, overrides),
);
} catch (error) {
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
if (isNoSuchElementError(error)) return 'none';
throw error;
}
if (elementId === undefined) return 'unsupported';
try {
const value = await this.sessionRequest(
'GET',
`/element/${encodeURIComponent(elementId)}/rect`,
undefined,
overrides,
);
return { id: elementId, rect: readWindowRect(value) };
} catch (error) {
if (isUnimplementedWebDriverRoute(error)) return 'unsupported';
// The focused element went away between the two calls — a stale answer,
// not a broken driver. Report it as "nothing focused" so the caller polls
// again rather than failing on a race it can simply retry out of.
if (isNoSuchElementError(error)) return 'none';
throw error;
}
}
async back(): Promise<void> {
await this.sessionRequest('POST', '/back');
}
@@ -379,6 +437,26 @@ function webdriverError(status: number, payload: unknown): AppError {
const UNIMPLEMENTED_WEBDRIVER_STATUSES = new Set([405, 501]);
const UNIMPLEMENTED_WEBDRIVER_ERRORS = new Set(['unknown command', 'unknown method']);
/**
* The W3C element identifier, whose key is the spec's fixed UUID rather than a
* readable name. Appium also echoes the legacy `ELEMENT` key; accept either so
* the caller works across grid versions.
*/
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
function readW3CElementId(value: unknown): string | undefined {
if (!value || typeof value !== 'object') return undefined;
const record = value as Record<string, unknown>;
const id = record[W3C_ELEMENT_KEY] ?? record.ELEMENT;
return typeof id === 'string' && id.length > 0 ? id : undefined;
}
/** Nothing is focused right now — an expected state, not a driver defect. */
function isNoSuchElementError(error: unknown): boolean {
if (!(error instanceof AppError)) return false;
return readWebDriverErrorCode(error).toLowerCase() === 'no such element';
}
function isUnimplementedWebDriverRoute(error: unknown): boolean {
if (!(error instanceof AppError)) return false;
const status = error.details?.status;
@@ -9,60 +9,146 @@ import { createWebDriverInteractor } from './webdriver-interactor.ts';
// #1658: `fill` used to send its keys in the request right after the tap. A
// WebView input takes first responder asynchronously, so on a web login form
// the keys landed with nothing focused and `fill` still reported success.
test('fill withholds keys until the tapped field raises the keyboard', async () => {
test('fill withholds keys until the tapped point holds text-entry focus', async () => {
const world = createTextEntryWorld();
// Hidden at the tap, then up two polls later: the transition our tap caused.
world.keyboardShown = [false, false, false, true];
// Nothing focused at first, then the tapped field takes it two polls later.
world.activeRects = ['none', 'none', CONTAINS_TAP];
const result = await runFill(world);
assert.equal(result?.textEntryReadiness, 'focused-element');
assert.deepEqual(world.transcript, [
'keyboard',
'active',
'tap',
'active',
'active',
'active',
'keys',
]);
});
// The case keyboard visibility can never decide: a second fill into a form
// whose keyboard is already up. The keyboard says "a field is focused" both
// before and after, so only the focused element's own geometry can tell whether
// OUR tap moved focus.
test('fill accepts a keyboard-up second field once focus lands on the tapped point', async () => {
const world = createTextEntryWorld();
world.keyboardShown = [true];
world.focusedBeforeTap = PREVIOUS_FIELD;
world.activeRects = [CONTAINS_TAP];
const result = await runFill(world);
assert.equal(result?.textEntryReadiness, 'focused-element');
assert.deepEqual(world.transcript, ['keyboard', 'active', 'tap', 'active', 'keys']);
});
// P1, the two-field misdelivery: email filled, keyboard stays up, the password
// tap misses. Focus never leaves the email field, so a global `/keys` would
// append the password to the EMAIL field while reporting success.
test('fill refuses when a keyboard-up tap misses and focus stays on the previous field', async () => {
const world = createTextEntryWorld();
world.keyboardShown = [true];
world.focusedBeforeTap = PREVIOUS_FIELD;
world.activeRects = [PREVIOUS_FIELD];
await assert.rejects(runFill(world), (error: AppError) => {
assert.equal(error.code, 'COMMAND_FAILED');
assert.match(error.message, /nothing there took text-entry focus/);
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
return true;
});
// The decisive assertion: no keys reached the field the previous fill focused.
assert.equal(world.transcript.includes('keys'), false);
});
// Found on a live AWS Device Farm iPhone 16: tapping Safari's collapsed address
// bar expands it into a taller field whose rect no longer covers the tapped
// point. A containment-only rule refused a fill that plainly worked, so a tap
// that MOVES focus counts even when the focused element has since been re-laid
// out.
test('fill accepts focus that moved to an element re-laid out away from the tap', async () => {
const world = createTextEntryWorld();
world.activeRects = ['none', MOVED_AFTER_FOCUS];
const result = await runFill(world);
assert.equal(result?.textEntryReadiness, 'focused-element');
});
// The other half of the rule: re-filling the field that is ALREADY focused
// moves nothing, so only containment can tell it from a tap that missed.
test('fill accepts a re-fill of the already-focused field it tapped', async () => {
const world = createTextEntryWorld();
world.focusedBeforeTap = CONTAINS_TAP;
world.activeRects = [CONTAINS_TAP];
const result = await runFill(world);
assert.equal(result?.textEntryReadiness, 'focused-element');
});
test('fill falls back to the keyboard transition when there is no active-element route', async () => {
const world = createTextEntryWorld();
world.activeRoute = 'unimplemented';
world.keyboardShown = [false, false, true];
const result = await runFill(world);
assert.equal(result?.textEntryReadiness, 'keyboard-shown');
assert.deepEqual(world.transcript, [
'keyboard',
'active',
'tap',
'keyboard',
'active',
'keyboard',
'keyboard',
'keys',
]);
});
test('fill settles when the keyboard was already up before the tap', async () => {
// Without an active-element route, a keyboard that was ALREADY up witnesses
// nothing about the new field — the exact silent misdelivery P1 names.
test('fill refuses a keyboard-up tap when the driver cannot report the focused field', async () => {
const world = createTextEntryWorld();
// Back-to-back fills (email then password): a keyboard that never went down
// cannot witness focus moving to the NEW field, so no poll can prove anything
// — polling it would only re-observe evidence about the previous field.
world.activeRoute = 'unimplemented';
world.keyboardShown = [true];
const result = await runFill(world);
await assert.rejects(runFill(world), (error: AppError) => {
assert.match(error.message, /cannot report which field holds focus/);
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
return true;
});
assert.equal(result?.textEntryReadiness, 'settled-keyboard-up');
assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']);
assert.equal(world.transcript.includes('keys'), false);
});
test('fill settles when the driver implements no keyboard route', async () => {
test('fill settles blind only when the driver implements neither route', async () => {
const world = createTextEntryWorld();
world.activeRoute = 'unimplemented';
world.keyboardRoute = 'unimplemented';
const result = await runFill(world);
// A route the driver does not have is not a failure — it just leaves nothing to wait for.
// Routes the driver does not have are not failures — they just leave nothing
// to wait for. This is the only path that types without evidence.
assert.equal(result?.textEntryReadiness, 'settled-unknown');
assert.deepEqual(world.transcript, ['keyboard', 'tap', 'keys']);
assert.deepEqual(world.transcript, ['keyboard', 'active', 'tap', 'active', 'keys']);
});
// The #1658 report is "fill answered Filled N chars while the field kept its
// placeholder". Reporting a readiness value nobody renders would preserve that,
// so a tap that raises no keyboard refuses instead — and sends no keys, leaving
// so a tap that focuses nothing refuses instead — and sends no keys, leaving
// the field untouched rather than half-written.
test('fill refuses without typing when no keyboard ever appears', async () => {
test('fill refuses without typing when nothing ever takes focus', async () => {
const world = createTextEntryWorld();
world.keyboardShown = [false];
world.activeRects = ['none'];
await assert.rejects(runFill(world), (error: AppError) => {
assert.equal(error.code, 'COMMAND_FAILED');
assert.match(error.message, /no keyboard appeared, so the text was not sent/);
assert.equal(error.details?.reason, 'text_entry_focus_not_observed');
assert.match(error.message, /nothing there took text-entry focus/);
return true;
});
@@ -80,18 +166,53 @@ test('fill propagates a failing keyboard probe instead of typing blind', async (
return true;
});
// It threw on the pre-tap probe, so the device was never touched at all.
assert.deepEqual(world.transcript, ['keyboard']);
// It threw on a pre-tap probe, so the device was never touched at all.
assert.equal(world.transcript.includes('tap'), false);
});
test('fill bounds each keyboard probe well under its readiness budget', async () => {
// Post-tap the device HAS been touched, so one transient grid error must not
// fail a fill the next poll would have satisfied.
test('fill tolerates a transient probe failure and still witnesses focus', async () => {
const world = createTextEntryWorld();
world.keyboardShown = [false, true];
world.activeFailuresAfterTap = 2;
world.activeRects = [CONTAINS_TAP];
await runFill(world);
const result = await runFill(world);
// Left at the client default, one hung probe could hold a 2s wait for 30s.
assert.deepEqual(world.probeTimeouts, [1_500, 1_500]);
assert.equal(result?.textEntryReadiness, 'focused-element');
});
// ...but a budget that expires without a single answered probe is a broken
// session, and must surface as itself rather than as "the tap missed".
test('fill rethrows the probe failure when no probe ever answers', async () => {
const world = createTextEntryWorld();
world.activeRoute = 'failing';
await assert.rejects(runFill(world), (error: AppError) => {
assert.match(error.message, /active element exploded/);
return true;
});
assert.equal(world.transcript.includes('keys'), false);
});
// P2: a probe handed the full fixed timeout near the deadline would finish well
// after the budget it was meant to respect, so each one is capped by what is
// actually left.
test('fill caps each probe timeout by the readiness budget remaining', async () => {
const world = createTextEntryWorld();
world.activeRects = ['none'];
await assert.rejects(runFill(world), () => true);
const timeouts = world.probeTimeouts.filter((t): t is number => t !== undefined);
assert.equal(
timeouts.every((t) => t <= 1_500),
true,
);
// The budget is 2s and probes start at t=0, so once under 1.5s remain the cap
// is the remainder, and the final probe gets exactly one poll interval.
assert.equal(timeouts.at(-1), 100);
});
/**
@@ -127,23 +248,70 @@ async function runFill(world: ReturnType<typeof createTextEntryWorld>) {
* Records the request order `fill` produces. The ordering IS the fix: a `keys`
* entry that follows `tap` with no keyboard evidence between them is the bug.
*/
type FocusedElement = {
id: string;
rect: { x: number; y: number; width: number; height: number };
};
/** `fill` is driven at (12, 24) throughout, so these read as hit and miss. */
const CONTAINS_TAP = { id: 'tapped', rect: { x: 0, y: 0, width: 100, height: 100 } } as const;
const PREVIOUS_FIELD = {
id: 'previous',
rect: { x: 200, y: 200, width: 100, height: 100 },
} as const;
/** Focused by our tap, but re-laid out by focusing so it no longer covers the tap point. */
const MOVED_AFTER_FOCUS = {
id: 'expanded',
rect: { x: 200, y: 600, width: 180, height: 40 },
} as const;
function createTextEntryWorld() {
const world = {
transcript: [] as string[],
probeTimeouts: [] as Array<number | undefined>,
keyboardShown: [] as boolean[],
/** How the driver answers the keyboard route: normally, not at all, or with a real failure. */
keyboardShown: [true] as boolean[],
/** How the driver answers each route: normally, not at all, or with a real failure. */
keyboardRoute: 'ok' as 'ok' | 'unimplemented' | 'failing',
activeRoute: 'ok' as 'ok' | 'unimplemented' | 'failing',
/** Successive answers from the active-element route; the last one repeats. */
activeRects: ['none'] as Array<'none' | FocusedElement>,
/** What held focus when the fill began — a previous fill's field, or nothing. */
focusedBeforeTap: 'none' as 'none' | FocusedElement,
/** Transient post-tap failures to emit before the route starts answering. */
activeFailuresAfterTap: 0,
tapped: false,
client: undefined as unknown as WebDriverClient,
};
world.client = {
performActions: async () => {
world.transcript.push('tap');
world.tapped = true;
},
releaseActions: async () => {},
sendKeys: async () => {
world.transcript.push('keys');
},
activeElement: async (timeoutMs?: number) => {
world.transcript.push('active');
world.probeTimeouts.push(timeoutMs);
if (world.activeRoute === 'unimplemented') return 'unsupported' as const;
if (world.activeRoute === 'failing') {
throw new AppError('COMMAND_FAILED', 'active element exploded', { status: 500 });
}
// Before the tap the form is in whatever state the previous fill left it.
if (!world.tapped) {
return world.focusedBeforeTap === 'none'
? ('none' as const)
: { id: world.focusedBeforeTap.id, rect: world.focusedBeforeTap.rect };
}
if (world.activeFailuresAfterTap > 0) {
world.activeFailuresAfterTap -= 1;
throw new AppError('COMMAND_FAILED', 'transient grid blip', { status: 502 });
}
const next =
world.activeRects.length > 1 ? world.activeRects.shift()! : world.activeRects[0]!;
return next === 'none' ? ('none' as const) : { id: next.id, rect: next.rect };
},
isKeyboardShown: async (timeoutMs?: number) => {
world.transcript.push('keyboard');
world.probeTimeouts.push(timeoutMs);
@@ -47,25 +47,35 @@ function sleep(ms: number): Promise<void> {
}
/**
* The tap raised no keyboard within the readiness budget, so nothing focused a
* text input and `sendKeys` would go to whatever else holds first responder —
* or nowhere. #1658 is precisely the report of that being answered with
* "Filled N chars", so this fails instead of typing: the caller gets a refusal
* it must handle, and the field is left untouched rather than half-written.
* Nothing took text-entry focus within the readiness budget, so `sendKeys`
* would go to whatever else holds first responder — or nowhere. #1658 is
* precisely the report of that being answered with "Filled N chars", so this
* fails instead of typing: the caller gets a refusal it must handle, and the
* field is left untouched rather than half-written.
*/
function textEntryFocusNotObservedError(x: number, y: number): AppError {
return new AppError(
'COMMAND_FAILED',
`fill tapped (${x}, ${y}) but no keyboard appeared, so the text was not sent`,
{
reason: 'text_entry_focus_not_observed',
x,
y,
hint: 'The tap most likely missed the field. Re-check the target with snapshot -i and fill the element it reports, rather than retrying the same coordinates.',
},
);
function textEntryFocusNotObservedError(x: number, y: number, detail: string): AppError {
return new AppError('COMMAND_FAILED', `fill tapped (${x}, ${y}) but ${detail}`, {
reason: 'text_entry_focus_not_observed',
x,
y,
hint: 'The tap most likely missed the field. Re-check the target with snapshot -i and fill the element it reports, rather than retrying the same coordinates.',
});
}
function rectContains(rect: WebDriverWindowRect, x: number, y: number): boolean {
return x >= rect.x && x <= rect.x + rect.width && y >= rect.y && y <= rect.y + rect.height;
}
/**
* What text entry looked like before the tap. Both readings are only evidence
* as a change, so both must be captured before the device is touched.
*/
type TextEntryStateBeforeTap = {
keyboard: boolean | 'unsupported';
/** `undefined` when nothing was focused, or when the driver cannot say. */
focusedElementId: string | undefined;
};
export type WebDriverInteractorOptions = {
client: WebDriverClient;
backend: Extract<SnapshotResult['backend'], 'android' | 'xctest'>;
@@ -189,14 +199,15 @@ class WebDriverInteractor implements Interactor {
_delayMs?: number,
): Promise<Record<string, unknown>> {
this.requireSupport('fill');
// #1658: read the keyboard BEFORE the tap. Only a hidden -> shown
// transition proves that OUR tap moved focus; a keyboard that was already
// up says nothing about which field owns first responder now. A probe
// failure throws here, before the tap, so a broken session fails a fill
// that has not touched the device rather than typing into the dark.
const keyboardBeforeTap = await this.readKeyboardShown();
// #1658: both readings are taken BEFORE the tap, because each is only
// evidence as a change. A keyboard that was already up says nothing about
// which field owns first responder now, and a focused element is only
// proof our tap did something if it differs from the one already focused.
// Probe failures throw here, before the tap, so a broken session fails a
// fill that has not touched the device rather than typing into the dark.
const beforeTap = await this.readTextEntryState();
await this.tap(x, y);
const textEntryReadiness = await this.awaitTextEntryReadiness(keyboardBeforeTap, x, y);
const textEntryReadiness = await this.awaitTextEntryReadiness(beforeTap, x, y);
await this.type(text);
return { backend: 'webdriver', x, y, text, textEntryReadiness };
}
@@ -316,40 +327,115 @@ class WebDriverInteractor implements Interactor {
* only the round trip between them gave the field time to focus.
*/
private async awaitTextEntryReadiness(
keyboardBeforeTap: boolean | 'unsupported',
beforeTap: TextEntryStateBeforeTap,
x: number,
y: number,
): Promise<CloudTextEntryReadiness> {
// The driver implements no keyboard route, so no wait can learn anything.
// Keep a blind path cheap: settle briefly, then type. This is the ONE case
// that types without evidence, and it takes a positively classified
// unsupported answer to reach — never a probe that merely failed.
const keyboardBeforeTap = beforeTap.keyboard;
const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS;
// Strongest evidence first: the focused element itself. Keyboard visibility
// can only witness that *a* field took focus, never *which* — which is why
// filling a second field in an already-open form could send its keys to the
// first one.
//
// The question is whether OUR tap moved focus, so compare against what held
// it before. Identity rather than geometry, because focusing a field can
// re-lay it out: tapping Safari's collapsed address bar expands it into a
// taller search field that no longer contains the point that was tapped, so
// a containment-only rule refuses a fill that plainly worked. Containment
// still earns its place as the second half of the test — re-filling the
// field that is ALREADY focused moves nothing, and only geometry can tell
// that from a tap that missed everything.
const focusedBeforeTap = beforeTap.focusedElementId;
const focus = await this.pollWithinBudget(deadline, async (timeoutMs) => {
const active = await this.client.activeElement(timeoutMs);
if (active === 'unsupported') return 'abandon';
if (active === 'none') return 'retry';
const moved = active.id !== focusedBeforeTap;
return moved || rectContains(active.rect, x, y) ? 'satisfied' : 'retry';
});
if (focus === 'satisfied') return 'focused-element';
if (focus === 'expired') {
throw textEntryFocusNotObservedError(
x,
y,
'nothing there took text-entry focus, so the text was not sent',
);
}
// No active-element route. Fall back to keyboard evidence, which can still
// witness focus arriving when the keyboard was down before our tap.
if (keyboardBeforeTap === 'unsupported') {
// The driver implements neither route, so no wait can learn anything.
// Keep a blind path cheap: settle briefly, then type. This is the ONE
// case that types without evidence, and it takes two positively
// classified unsupported answers to reach — never a probe that failed.
await sleep(TEXT_ENTRY_BLIND_SETTLE_MS);
return 'settled-unknown';
}
// The keyboard was ALREADY up — back-to-back fills into one form, the shape
// that failed most often in #1658. Its visibility cannot witness focus
// moving to the NEW field, so polling it would resolve instantly on
// evidence about the PREVIOUS one. The Apple runner spends its whole
// readiness budget on exactly this case rather than race the app with a
// short settle; the cloud path has even more reason to, because its keys go
// to whatever holds first responder rather than to a scoped element.
if (keyboardBeforeTap) {
await sleep(TEXT_ENTRY_READINESS_TIMEOUT_MS);
return 'settled-keyboard-up';
// Keyboard already up and no way to ask which field owns it: every signal
// available here describes the PREVIOUS field. Typing anyway is exactly
// #1658's silent misdelivery, so refuse instead.
throw textEntryFocusNotObservedError(
x,
y,
'the keyboard was already up and this driver cannot report which field holds focus, so the text was not sent',
);
}
const deadline = Date.now() + TEXT_ENTRY_READINESS_TIMEOUT_MS;
while (Date.now() < deadline) {
await sleep(TEXT_ENTRY_READINESS_POLL_MS);
if ((await this.readKeyboardShown()) === true) return 'keyboard-shown';
}
throw textEntryFocusNotObservedError(x, y);
const keyboard = await this.pollWithinBudget(deadline, async (timeoutMs) =>
(await this.client.isKeyboardShown(timeoutMs)) === true ? 'satisfied' : 'retry',
);
if (keyboard === 'satisfied') return 'keyboard-shown';
throw textEntryFocusNotObservedError(x, y, 'no keyboard appeared, so the text was not sent');
}
/** Bounded by the readiness budget it serves, not the client's default request timeout. */
private async readKeyboardShown(): Promise<boolean | 'unsupported'> {
return await this.client.isKeyboardShown(TEXT_ENTRY_PROBE_TIMEOUT_MS);
/** Both pre-tap readings, taken together so neither is charged an extra tap-to-probe gap. */
private async readTextEntryState(): Promise<TextEntryStateBeforeTap> {
const [keyboard, active] = await Promise.all([
this.client.isKeyboardShown(TEXT_ENTRY_PROBE_TIMEOUT_MS),
this.client.activeElement(TEXT_ENTRY_PROBE_TIMEOUT_MS),
]);
return {
keyboard,
focusedElementId: active === 'none' || active === 'unsupported' ? undefined : active.id,
};
}
/**
* Polls until `probe` is satisfied or the readiness budget runs out, bounding
* BOTH the probe's own transport timeout and the sleep between attempts by
* what remains — a probe handed the full fixed timeout near the deadline
* would otherwise finish well after the budget it was meant to respect.
*
* A probe that throws is tolerated: the loop has further attempts, and one
* transient grid error should not fail a fill that would have succeeded. But
* a budget that expires without a single answered probe rethrows, so a dead
* session or an auth rejection surfaces as itself instead of being reported
* as a tap that missed its field.
*/
private async pollWithinBudget(
deadline: number,
probe: (timeoutMs: number) => Promise<'satisfied' | 'abandon' | 'retry'>,
): Promise<'satisfied' | 'abandon' | 'expired'> {
let answered = false;
let lastError: unknown;
for (;;) {
const remaining = deadline - Date.now();
if (remaining <= 0) break;
try {
const verdict = await probe(Math.min(TEXT_ENTRY_PROBE_TIMEOUT_MS, remaining));
answered = true;
if (verdict !== 'retry') return verdict;
} catch (error) {
lastError = error;
}
const left = deadline - Date.now();
if (left <= 0) break;
await sleep(Math.min(TEXT_ENTRY_READINESS_POLL_MS, left));
}
if (!answered && lastError !== undefined) throw lastError;
return 'expired';
}
private async pointerGesture(name: string, actions: W3CPointerAction[]): Promise<void> {
@@ -76,20 +76,31 @@ test('cloud iOS snapshot captures through the provider session after open', asyn
test('cloud iOS fill lands text in the field it tapped', async () => {
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
const lease = await openCloudIosSession(daemon);
// Focus lands a poll late, as a WebView input's does. The keys must wait.
server.pollsUntilFocus = 2;
const response = await daemon.callCommand(
'fill',
['editable=true', 'user@example.com'],
['label=Email', 'user@example.com'],
leaseFlags(lease.leaseId),
{ meta: leaseMeta(lease.leaseId) },
);
const data = assertRpcOk<{ text?: string; textEntryReadiness?: string }>(response);
assert.equal(data.text, 'user@example.com');
assert.equal(data.textEntryReadiness, 'keyboard-shown');
assert.equal(data.textEntryReadiness, 'focused-element');
// The keys reached the focused field, not the void.
assert.equal(server.fieldValue, 'user@example.com');
assert.deepEqual(textEntryTranscript(server), ['keyboard', 'tap', 'keyboard', 'keys']);
assert.equal(server.fieldValues.email, 'user@example.com');
// Both pre-tap readings, the tap, then polling until focus lands. The keys
// come last — that ordering IS the fix.
assert.deepEqual(textEntryTranscript(server), [
'keyboard',
'active',
'tap',
'active',
'active',
'keys',
]);
// And the device agrees: the field reports the typed value back.
const snapshot = await daemon.callCommand(
@@ -111,23 +122,54 @@ test('cloud iOS fill lands text in the field it tapped', async () => {
* "Filled N chars" is the original bug, so the fill refuses — and sends no keys,
* leaving the field untouched rather than half-written.
*/
test('cloud iOS fill refuses, without typing, when the tap raises no keyboard', async () => {
test('cloud iOS fill refuses, without typing, when nothing takes focus', async () => {
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
const lease = await openCloudIosSession(daemon);
server.keyboardPollsUntilShown = Number.POSITIVE_INFINITY;
server.pollsUntilFocus = Number.POSITIVE_INFINITY;
const response = await daemon.callCommand(
'fill',
['editable=true', 'user@example.com'],
['label=Email', 'user@example.com'],
leaseFlags(lease.leaseId),
{ meta: leaseMeta(lease.leaseId) },
);
const error = response.json.error;
assert.ok(error, 'expected fill to refuse');
assert.match(error.message ?? '', /no keyboard appeared, so the text was not sent/);
assert.match(error.message ?? '', /nothing there took text-entry focus/);
assert.equal(error.data?.details?.reason, 'text_entry_focus_not_observed');
assert.equal(server.fieldValue, '');
assert.equal(server.fieldValues.email, '');
assert.equal(textEntryTranscript(server).includes('keys'), false);
});
}, 15_000);
/**
* The misdelivery keyboard visibility can never catch (#1658 P1). The form is
* already filled and focused — keyboard up — and the second fill's tap misses
* the password field. Focus therefore never leaves `email`, so a `POST /keys`
* here would append the password to the EMAIL field and every request would
* still return 200. The fill must refuse before sending anything.
*/
test('cloud iOS fill sends no keys when a keyboard-up tap misses the second field', async () => {
await withProviderScenarioResource(createCloudIosWorld, async ({ daemon, server }) => {
const lease = await openCloudIosSession(daemon);
server.focusField('email', 'user@example.com');
// Focus never moves, reproducing a tap that landed outside both fields.
server.pollsUntilFocus = Number.POSITIVE_INFINITY;
const response = await daemon.callCommand(
'fill',
['label=Password', 'hunter2'],
leaseFlags(lease.leaseId),
{ meta: leaseMeta(lease.leaseId) },
);
assert.ok(response.json.error, 'expected fill to refuse');
assert.equal(response.json.error.data?.details?.reason, 'text_entry_focus_not_observed');
// The decisive assertion: the previous field is untouched and the password
// never reached it.
assert.equal(server.fieldValues.email, 'user@example.com');
assert.equal(server.fieldValues.password, '');
assert.equal(textEntryTranscript(server).includes('keys'), false);
});
}, 15_000);
@@ -138,6 +180,7 @@ function textEntryTranscript(server: FakeIosWebDriverServer): string[] {
'POST /wd/hub/session/wd-1/actions': 'tap',
'POST /wd/hub/session/wd-1/keys': 'keys',
'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': 'keyboard',
'GET /wd/hub/session/wd-1/element/active': 'active',
};
return server.calls
.map((call) => labels[`${call.method} ${call.path}`])
@@ -195,18 +238,38 @@ async function openCloudIosSession(daemon: ProviderScenarioHarness): Promise<Dev
* That last part is the #1658 bug reproduced faithfully: keys sent too early
* are accepted by the grid and dropped by the device.
*/
const W3C_ELEMENT_KEY = 'element-6066-11e4-a52e-4f735466cecf';
/** Two stacked inputs, as a login form has. Geometry is what decides focus. */
const FIELDS = {
email: { x: 20, y: 300, width: 350, height: 44 },
password: { x: 20, y: 380, width: 350, height: 44 },
} as const;
type FieldName = keyof typeof FIELDS;
class FakeIosWebDriverServer extends CloudWebDriverTestServer {
/** How many keyboard probes after the tap before focus lands. Infinity = the tap missed. */
keyboardPollsUntilShown = 1;
fieldValue = '';
/** How many probes after a tap before focus settles. Infinity = focus never moves. */
pollsUntilFocus = 1;
fieldValues: Record<FieldName, string> = { email: '', password: '' };
focused: FieldName | undefined;
private pollsRemaining = Number.POSITIVE_INFINITY;
private pendingFocus: FieldName | undefined;
static async start(): Promise<StartedCloudWebDriverTestServer<FakeIosWebDriverServer>> {
return await startCloudWebDriverTestServer(new FakeIosWebDriverServer());
}
/** Puts the form in the state a completed first fill leaves it in. */
focusField(field: FieldName, value: string): void {
this.focused = field;
this.fieldValues[field] = value;
this.pollsRemaining = 0;
this.pendingFocus = field;
}
protected respond(call: CloudWebDriverHttpCall) {
const rectMatch = /^\/wd\/hub\/session\/wd-1\/element\/([^/]+)\/rect$/.exec(call.path);
if (call.method === 'GET' && rectMatch) return this.elementRect(rectMatch[1]!);
const route = this.routes[`${call.method} ${call.path}`];
return route ? route() : cloudWebDriverTestJson({ value: null });
}
@@ -222,47 +285,113 @@ class FakeIosWebDriverServer extends CloudWebDriverTestServer {
cloudWebDriverTestJson({ value: { x: 0, y: 0, width: 390, height: 844 } }),
'POST /wd/hub/session/wd-1/actions': () => this.tap(),
'POST /wd/hub/session/wd-1/keys': () => this.sendKeys(),
'GET /wd/hub/session/wd-1/element/active': () => this.activeElement(),
'GET /wd/hub/session/wd-1/appium/device/is_keyboard_shown': () =>
cloudWebDriverTestJson({ value: this.keyboardShown() }),
};
}
/**
* A tap moves focus only if it actually lands inside a field. A miss leaves
* the previous field focused — which is exactly how keys can reach the wrong
* one while every request still succeeds.
*/
private tap(): CloudWebDriverTestResponse {
this.pollsRemaining = this.keyboardPollsUntilShown;
const point = this.lastTapPoint();
const hit =
point && (Object.keys(FIELDS) as FieldName[]).find((n) => contains(FIELDS[n], point));
this.pendingFocus = hit ?? this.focused;
this.pollsRemaining = hit ? this.pollsUntilFocus : Number.POSITIVE_INFINITY;
if (hit) this.focused = undefined;
return cloudWebDriverTestJson({ value: null });
}
/** Focus is not instant: it lands only after the configured number of probes. */
private keyboardShown(): boolean {
if (this.pollsRemaining <= 0) return true;
/** Focus is not instant: it settles only after the configured number of probes. */
private settle(): void {
if (this.pollsRemaining <= 0) return;
this.pollsRemaining -= 1;
return this.pollsRemaining <= 0;
if (this.pollsRemaining <= 0) this.focused = this.pendingFocus;
}
private activeElement(): CloudWebDriverTestResponse {
this.settle();
if (!this.focused) {
return cloudWebDriverTestJson(
{ value: { error: 'no such element', message: 'nothing is focused' } },
404,
);
}
return cloudWebDriverTestJson({ value: { [W3C_ELEMENT_KEY]: this.focused } });
}
private elementRect(elementId: string): CloudWebDriverTestResponse {
const rect = FIELDS[elementId as FieldName];
if (!rect) {
return cloudWebDriverTestJson(
{ value: { error: 'no such element', message: `unknown element ${elementId}` } },
404,
);
}
return cloudWebDriverTestJson({ value: rect });
}
private keyboardShown(): boolean {
this.settle();
return this.focused !== undefined;
}
private sendKeys(): CloudWebDriverTestResponse {
// Unfocused keys are accepted by the grid and land nowhere — the bug's shape.
if (this.pollsRemaining <= 0) {
const text = this.lastSentKeys();
if (text !== undefined) this.fieldValue = text;
}
// Unfocused keys are accepted by the grid and land nowhere; keys sent while
// the WRONG field holds focus land in that field. Both are the bug's shape.
const text = this.lastSentKeys();
if (this.focused && text !== undefined) this.fieldValues[this.focused] = text;
return cloudWebDriverTestJson({ value: null });
}
private lastTapPoint(): { x: number; y: number } | undefined {
const body = this.calls.at(-1)?.body as { actions?: unknown } | undefined;
const sequences = Array.isArray(body?.actions) ? body.actions : [];
for (const sequence of sequences) {
const actions = (sequence as { actions?: unknown }).actions;
if (!Array.isArray(actions)) continue;
const move = actions.find((a) => (a as { type?: string }).type === 'pointerMove') as
| { x?: unknown; y?: unknown }
| undefined;
if (typeof move?.x === 'number' && typeof move.y === 'number') {
return { x: move.x, y: move.y };
}
}
return undefined;
}
private lastSentKeys(): string | undefined {
const body = this.calls.at(-1)?.body as { value?: unknown } | undefined;
return Array.isArray(body?.value) ? body.value.join('') : undefined;
}
private source(): string {
const field = (name: FieldName, label: string) =>
`<XCUIElementTypeTextField name="${name}" label="${label}" value="${this.fieldValues[name]}" ` +
`x="${FIELDS[name].x}" y="${FIELDS[name].y}" width="${FIELDS[name].width}" ` +
`height="${FIELDS[name].height}" visible="true" enabled="true" />`;
return (
'<XCUIElementTypeApplication name="Demo" x="0" y="0" width="390" height="844" visible="true">' +
`<XCUIElementTypeTextField name="email" label="Email" value="${this.fieldValue}" ` +
'x="20" y="300" width="350" height="44" visible="true" enabled="true" />' +
field('email', 'Email') +
field('password', 'Password') +
'</XCUIElementTypeApplication>'
);
}
}
function contains(
rect: { x: number; y: number; width: number; height: number },
p: { x: number; y: number },
): boolean {
return (
p.x >= rect.x && p.x <= rect.x + rect.width && p.y >= rect.y && p.y <= rect.y + rect.height
);
}
function leaseFlags(leaseId?: string): DaemonRequest['flags'] {
return {
platform: 'ios',
+2 -2
View File
@@ -355,5 +355,5 @@ BrowserStack can return session video, Appium logs, device logs, dashboard URL,
- If AWS connect fails, use its reported `aws devicefarm get-*` error to check the credential chain, ARN, region, resource platform, or upload readiness. Allocation has not happened yet.
- If Limrun connect fails, check `LIMRUN_API_KEY` and the optional `LIMRUN_REGION`. A successful connect verifies the selected instance service without creating an instance.
- If artifact lookup is pending immediately after `close`, retry `agent-device artifacts --json`. Some providers finalize video/log URLs asynchronously after the hosted session stops.
- On hosted WebDriver sessions (BrowserStack, AWS Device Farm), `fill` waits for the tapped field to take text entry focus before sending its keys, and reports what it observed as `textEntryReadiness`: `keyboard-shown` (the keyboard came up in response to the tap), `settled-keyboard-up` (the keyboard was already up — back-to-back fills into one form — so it could not witness the new field taking focus and the fill waited out its full readiness budget), or `settled-unknown` (the driver implements no keyboard route, so there was nothing to wait for).
- If the tap raises no keyboard at all, `fill` **fails** with `text_entry_focus_not_observed` and sends no keys, rather than reporting a filled field that never received the text. That almost always means the tap missed: re-check the target with `snapshot -i` and fill the element it reports instead of retrying the same coordinates.
- On hosted WebDriver sessions (BrowserStack, AWS Device Farm), `fill` waits until the field it tapped actually holds text entry focus before sending its keys, and reports what it observed as `textEntryReadiness`: `focused-element` (the focused element contains the tapped point — the strongest signal, and the only one that identifies *which* field took focus), `keyboard-shown` (the driver has no active-element route, but the keyboard rose from hidden after the tap), or `settled-unknown` (the driver implements neither route, so there was nothing to wait for).
- If focus cannot be witnessed, `fill` **fails** with `text_entry_focus_not_observed` and sends no keys, rather than reporting a filled field that never received the text. That almost always means the tap missed: re-check the target with `snapshot -i` and fill the element it reports instead of retrying the same coordinates. The same refusal covers a second fill into a form whose keyboard is already up on a driver that cannot name the focused field — there, unwitnessed keys would land in the field the previous fill focused.