mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
* diag: instrument Linux CI to gather evidence for #1935 input/a11y defects Temporary — adds a diagnostic step that dumps raw AT-SPI interfaces/actions for gnome-calculator's digit buttons, tests a raw xdotool click at a button's own rect (bypassing our promotion logic), and isolates the typed '=' character in several configurations. Will be removed once the real fixes land. * diag: harden diagnostic step against bash -e and AT-SPI registration races The prior version crashed 7s in: GH Actions runs steps under bash -e, and an unguarded python3 heredoc threw (iterating a dict instead of a list when the app wasn't found yet), aborting the rest of the script silently under continue-on-error. Guards every fallible command, and replaces the fixed 2s sleep with inspect.py's own poll-until-found loop. * diag: test WINDOW coordtype and static Text.get_text call (round 3) Round 2 proved Component.get_extents(SCREEN) returns (0,0) for every non-toplevel widget (real click miss confirmed on-screen), and Text.get_text() throws — a documented PyGObject binding collision with the deprecated 1-arg Accessible.get_text(). This narrows to the two candidate fixes before writing them: does CoordType.WINDOW give usable relative offsets, and does Atspi.Text.get_text(accessible, ...) (static call) return the real typed text. * fix(linux): resolve click-miss, dropped '=', and GTK4 text exposure defects Three defects surfaced by CI on #1935 (Linux Smoke lane), all confirmed live via instrumented CI runs before being fixed here: 1. Click misses its target: Component.get_extents(Atspi.CoordType.SCREEN) returns (0, 0) as the origin for every non-toplevel widget under this GTK4 build — confirmed by a raw click at the computed rect center landing on the window's own header-bar button instead of the intended digit button. CoordType.WINDOW gives correct, distinct per-widget offsets, so get_rect() now computes screen-absolute rects as that offset plus the enclosing top-level frame's own (correct) screen origin, threaded through traverse_node() alongside the existing window-title tracking. Complementary hardening: role "label" is now excluded from `hittable`, since GTK4 wraps every button's caption in a same-rect "label" child, and the shared cross-platform promotion logic in interaction-targeting.ts would otherwise retarget a click from the button onto that non-interactive label. 2. Typed '=' never arrives: a single isolated synthetic keystroke sent right after a focus change is unreliably delivered — confirmed live, both `xdotool type -- "="` and `xdotool key equal` sent alone produced no character at all, while multi-character bursts always landed in full. typeLinux and sendKey now wait a short settle margin before dispatching to xdotool/ydotool, absorbing the race regardless of which action last changed focus. 3. GTK4 apps expose no editable text: accessible.get_text_iface().get_text() throws "Atspi.Accessible.get_text() takes exactly 1 argument (3 given)" — a documented PyGObject binding collision between Text.get_text and the deprecated 1-argument Accessible.get_text, silently swallowed as "no text" by the broad exception handler. get_text_value() now calls the unbound Atspi.Text.get_text(accessible, ...) form, which correctly returns the real content. The Linux smoke replay is restored to exercise all three fixes together (click a resolved digit button, type a full calculation including the '=' keystroke, wait on the computed result through the tree) instead of staying at the weakened, contract-tier assertions the defects had forced. The coverage manifest promotes click and type from command-contract to live accordingly. * fix(linux): drop unproven keyboard-settle and hittable changes per review Addresses thymikee's review on #1949 (both points correct): P1: the keyboard settle (typeLinux/sendKey) was unjustified. The cited diagnostic evidence for a dropped '=' actually shows the opposite — "100+55=" and "5=5" both computed correctly with zero settle, proving '=' was delivered in every multi-character burst tested. Sending '=' alone to an empty entry showing a blank display is normal calculator semantics (nothing to evaluate), not a lost keystroke. The likelier explanation for the original "100+55" screenshot (run 32487868346) is that its attempt-3 hit the already-fixed mousemove --sync hang, not an independent keyboard-dispatch defect. Reverted; no keyboard-dispatch change was needed. P2: the `role_name != "label"` hittable narrowing was extra surface beyond what the click-miss fix required. The corrected AT-SPI coordinates alone fix the observed miss — the button and its same-rect label child resolve to nearly identical centers, so descendant promotion still lands inside the button either way, and the replay can't distinguish which node it actually targeted. Reverted; only the coordinate fix remains.
This commit is contained in:
committed by
GitHub
parent
1f80c92a27
commit
3bf3ff130a
+54
-9
@@ -22,19 +22,48 @@ MAX_DESKTOP_APPS = 24
|
||||
VALID_SURFACES = ("desktop", "frontmost-app")
|
||||
|
||||
|
||||
def get_rect(accessible):
|
||||
def get_screen_origin(accessible):
|
||||
"""The screen-absolute (x, y) of a top-level frame/window's own Component extents.
|
||||
|
||||
Unlike descendant widgets (see get_rect), a top-level's own SCREEN-coordinate extents are
|
||||
correct — GTK4's AT-SPI bridge only loses the translation when walking from a widget up
|
||||
through its ancestor chain to the root, not for the root itself.
|
||||
"""
|
||||
try:
|
||||
component = accessible.get_component_iface()
|
||||
if not component:
|
||||
return None
|
||||
extents = component.get_extents(Atspi.CoordType.SCREEN)
|
||||
if not extents:
|
||||
return None
|
||||
return (extents.x, extents.y)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def get_rect(accessible, frame_origin):
|
||||
"""A node's screen-absolute rect.
|
||||
|
||||
GTK4's AT-SPI bridge returns (0, 0) as the origin of Component.get_extents(SCREEN) for every
|
||||
non-toplevel widget — confirmed live (CI run 32503838660): a raw xdotool click at the
|
||||
computed center of a digit button's "screen" rect landed on the window's own header-bar
|
||||
button instead. CoordType.WINDOW gives correct, distinct per-widget offsets, so the
|
||||
screen-absolute rect is that offset plus the enclosing top-level frame's own (correct)
|
||||
screen origin.
|
||||
"""
|
||||
try:
|
||||
component = accessible.get_component_iface()
|
||||
if not component:
|
||||
return None
|
||||
extents = component.get_extents(Atspi.CoordType.WINDOW)
|
||||
if not extents:
|
||||
return None
|
||||
if extents.width <= 0 or extents.height <= 0:
|
||||
return None
|
||||
origin_x, origin_y = frame_origin if frame_origin else (0, 0)
|
||||
return {
|
||||
"x": extents.x,
|
||||
"y": extents.y,
|
||||
"x": extents.x + origin_x,
|
||||
"y": extents.y + origin_y,
|
||||
"width": extents.width,
|
||||
"height": extents.height,
|
||||
}
|
||||
@@ -43,14 +72,24 @@ def get_rect(accessible):
|
||||
|
||||
|
||||
def get_text_value(accessible):
|
||||
"""The Text interface's content, if any.
|
||||
|
||||
Must call Atspi.Text.get_text(accessible, ...) as an unbound/static call, NOT
|
||||
accessible.get_text_iface().get_text(...) — the bound form resolves to the deprecated
|
||||
1-argument Atspi.Accessible.get_text() instead (a documented PyGObject binding collision:
|
||||
https://discourse.gnome.org/t/how-can-i-explicitly-call-atspi-text-get-text/36684), raising
|
||||
"takes exactly 1 argument (3 given)" for every node, silently swallowed as "no text" by the
|
||||
except-Exception below. Confirmed live (CI run 32505154107): the static form correctly
|
||||
returns typed text ("155") where the bound form threw on the same node.
|
||||
"""
|
||||
try:
|
||||
text_iface = accessible.get_text_iface()
|
||||
if not text_iface:
|
||||
return None
|
||||
count = text_iface.get_character_count()
|
||||
count = Atspi.Text.get_character_count(accessible)
|
||||
if count <= 0:
|
||||
return None
|
||||
value = text_iface.get_text(0, count)
|
||||
value = Atspi.Text.get_text(accessible, 0, count)
|
||||
return value if value else None
|
||||
except Exception:
|
||||
return None
|
||||
@@ -76,7 +115,7 @@ def has_state(state_set, state_type):
|
||||
return False
|
||||
|
||||
|
||||
def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=None):
|
||||
def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=None, frame_origin=None):
|
||||
if len(ctx["nodes"]) >= ctx["max_nodes"] or depth > ctx["max_depth"] or not accessible:
|
||||
return
|
||||
|
||||
@@ -96,7 +135,13 @@ def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=N
|
||||
description = ""
|
||||
|
||||
label = name or description or None
|
||||
rect = get_rect(accessible)
|
||||
|
||||
# Entering a new top-level resets the frame origin used to translate its descendants'
|
||||
# WINDOW-relative extents to screen-absolute (see get_rect) — each frame/dialog is a
|
||||
# separate X11 top-level with its own screen position.
|
||||
is_frame = role_name in ("frame", "window", "dialog")
|
||||
effective_frame_origin = (get_screen_origin(accessible) or frame_origin) if is_frame else frame_origin
|
||||
rect = get_rect(accessible, effective_frame_origin)
|
||||
|
||||
try:
|
||||
state_set = accessible.get_state_set()
|
||||
@@ -110,7 +155,7 @@ def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=N
|
||||
hittable = (enabled is not False) and visible and showing and (rect is not None)
|
||||
|
||||
current_window_title = window_title
|
||||
if current_window_title is None and role_name in ("frame", "window", "dialog"):
|
||||
if current_window_title is None and is_frame:
|
||||
current_window_title = label
|
||||
|
||||
nodes = ctx["nodes"]
|
||||
@@ -147,7 +192,7 @@ def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=N
|
||||
if child:
|
||||
traverse_node(
|
||||
child, depth + 1, node_index, ctx, app_info,
|
||||
current_window_title
|
||||
current_window_title, effective_frame_origin
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -160,11 +160,11 @@ export const LINUX_PLATFORM_COVERAGE = {
|
||||
[C.record]: gap('No Linux-specific recording command evidence exists yet'),
|
||||
[C.trace]: gap('No Linux-specific trace command evidence exists yet'),
|
||||
[C.find]: gap('No Linux-specific find command evidence exists yet'),
|
||||
[C.click]: contract(
|
||||
LINUX_PROVIDER_EVIDENCE.path,
|
||||
LINUX_PROVIDER_EVIDENCE.test,
|
||||
'Linux provider scenario executes primary, secondary, middle, and double clicks',
|
||||
),
|
||||
// Promoted from command-contract to live: the desktop replay now clicks a resolved digit
|
||||
// button on real Linux hardware and the downstream wait only passes if the click landed
|
||||
// (formerly missed — AT-SPI extents were computed screen-absolute-wrong under GTK4; see
|
||||
// linux/atspi-dump.py).
|
||||
[C.click]: live('the Linux desktop replay clicks a resolved calculator digit button'),
|
||||
[C.fill]: contract(
|
||||
LINUX_PROVIDER_EVIDENCE.path,
|
||||
LINUX_PROVIDER_EVIDENCE.test,
|
||||
@@ -181,15 +181,11 @@ export const LINUX_PLATFORM_COVERAGE = {
|
||||
LINUX_PROVIDER_EVIDENCE.test,
|
||||
'Linux provider scenario presses a snapshot ref and coordinate target',
|
||||
),
|
||||
// The desktop replay runs the migrated typeText path on real hardware and uploads pixel
|
||||
// evidence of the typed entry each run, but GTK4 gnome-calculator exposes no Text-interface
|
||||
// content to selectors, so no tree-level assertion can hold and the claim stays at the
|
||||
// contract tier until that platform defect is fixed.
|
||||
[C.type]: contract(
|
||||
'src/platforms/linux/__tests__/input-actions.test.ts',
|
||||
'typeLinux uses ydotool type',
|
||||
'Linux type dispatch uses the Wayland ydotool type primitive',
|
||||
),
|
||||
// Promoted from command-contract to live: GTK4 gnome-calculator's entry previously exposed no
|
||||
// Text-interface content to selectors (a PyGObject binding call-pattern bug — see
|
||||
// linux/atspi-dump.py), so no tree-level assertion could hold. Fixed, so the desktop replay's
|
||||
// typed calculation now has a real wait assertion on the computed result.
|
||||
[C.type]: live('the Linux desktop replay types a calculation and its result is selectable'),
|
||||
[C.get]: contract(
|
||||
LINUX_PROVIDER_EVIDENCE.path,
|
||||
LINUX_PROVIDER_EVIDENCE.test,
|
||||
|
||||
@@ -16,13 +16,27 @@ snapshot -i
|
||||
focus 100 100
|
||||
# The session survives the focus: a crashed desktop would fail here, not silently pass above.
|
||||
is exists "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5"
|
||||
# A resolved button press: proves pointer input lands on a resolved target. role= disambiguates
|
||||
# from the calculator's [text] "1" node — a bare label=1 is an AMBIGUOUS_MATCH rejection by
|
||||
# design. Formerly missed: AT-SPI's Component.get_extents(SCREEN) returns (0,0) for every
|
||||
# non-toplevel widget under this GTK4 build, so the click landed on the window's own header-bar
|
||||
# button instead — fixed by computing screen-absolute rects from CoordType.WINDOW plus the
|
||||
# toplevel frame's own screen origin (linux/atspi-dump.py).
|
||||
click "role=button label=1"
|
||||
# R41 (#1739): `type` executes through the bound `typeText` operation rather than the retired
|
||||
# interactor leaf. The typed-state screenshot below is the live evidence: run 32490373693's
|
||||
# artifact shows the typed digits in the calculator entry. A selector assertion on the value is
|
||||
# impossible today — GTK4 gnome-calculator exposes no Text-interface content through the AT-SPI
|
||||
# dumper (display showed "155" while a wait for value=155 timed out) — so the tree-level claim
|
||||
# for `type` stays at the command-contract tier until the exposure defect is fixed.
|
||||
type "155"
|
||||
# interactor leaf. The keystrokes spell a full calculation: the clicked 1 plus the typed 00+55=
|
||||
# can only produce a 155 result if every keystroke — digits, the shift-composed '+', and the
|
||||
# trailing '=' — actually landed; no calculator button is labelled 155. (Run 32487868346's
|
||||
# "100+55" without '=' motivated a keystroke-loss investigation, but the diagnostic evidence for
|
||||
# that (CI run 32503838660) turned out to show '=' delivered correctly in every multi-character
|
||||
# burst tested; the run's own attempt-3 hit the already-fixed mousemove --sync hang, which is the
|
||||
# more likely explanation for that screenshot. No keyboard-dispatch change was needed here.)
|
||||
type "00+55="
|
||||
screenshot "./test/screenshots/replays/linux-calculator-typed.png"
|
||||
# The session survives the typing; a wedged desktop fails here instead of silently passing.
|
||||
is exists "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5"
|
||||
# GTK4 gnome-calculator's entry widget did not expose its content to selectors: AT-SPI's
|
||||
# Text.get_text() called as accessible.get_text_iface().get_text(...) throws — a PyGObject
|
||||
# binding collision with the deprecated 1-arg Accessible.get_text() — silently swallowed as "no
|
||||
# text". Fixed by calling the unbound Atspi.Text.get_text(accessible, ...) form instead. 155
|
||||
# appears only as the computed result: no button carries that label, and deleting any keystroke
|
||||
# above turns this wait red.
|
||||
wait "value=155 || label=155 || text=155" 10000
|
||||
|
||||
@@ -40,11 +40,12 @@ test('Linux coverage exhaustively classifies the public catalog', () => {
|
||||
test('Linux coverage report has the expected classification counts', () => {
|
||||
assert.deepEqual(LINUX_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY, {
|
||||
capabilityDenial: 11,
|
||||
// focus (#1925) is live via the replay; type runs there too but GTK4 blocks a tree-level
|
||||
// assertion, so its claim stays contract-tier (see the manifest entry).
|
||||
contract: 19,
|
||||
// focus (#1925), click, and type are live via the replay: click resolves and lands on a
|
||||
// digit button, and the typed calculation's result is now selectable (see the manifest
|
||||
// entries for the platform defects this fixed).
|
||||
contract: 17,
|
||||
gap: 18,
|
||||
live: 6,
|
||||
live: 8,
|
||||
total: 54,
|
||||
});
|
||||
|
||||
@@ -61,7 +62,7 @@ test('Linux live claims reference commands in the existing smoke replay', () =>
|
||||
parseReplayScriptDetailed(replaySource).actions.map((action) => action.command),
|
||||
);
|
||||
const liveCommands = liveCommandsForLinuxReplay();
|
||||
assert.deepEqual(liveCommands.length, 6);
|
||||
assert.deepEqual(liveCommands.length, 8);
|
||||
for (const command of liveCommands) {
|
||||
assert.equal(
|
||||
replayCommands.has(command),
|
||||
|
||||
Reference in New Issue
Block a user