fix: Linux input/a11y defects from #1935 (click miss, typed '=', GTK4 text) (#1949)

* diag: instrument Linux CI to gather evidence for #1935 input/a11y defects

Temporary — adds a diagnostic step that dumps raw AT-SPI interfaces/actions
for gnome-calculator's digit buttons, tests a raw xdotool click at a
button's own rect (bypassing our promotion logic), and isolates the typed
'=' character in several configurations. Will be removed once the real
fixes land.

* diag: harden diagnostic step against bash -e and AT-SPI registration races

The prior version crashed 7s in: GH Actions runs steps under bash -e, and
an unguarded python3 heredoc threw (iterating a dict instead of a list
when the app wasn't found yet), aborting the rest of the script silently
under continue-on-error. Guards every fallible command, and replaces the
fixed 2s sleep with inspect.py's own poll-until-found loop.

* diag: test WINDOW coordtype and static Text.get_text call (round 3)

Round 2 proved Component.get_extents(SCREEN) returns (0,0) for every
non-toplevel widget (real click miss confirmed on-screen), and
Text.get_text() throws — a documented PyGObject binding collision with
the deprecated 1-arg Accessible.get_text(). This narrows to the two
candidate fixes before writing them: does CoordType.WINDOW give usable
relative offsets, and does Atspi.Text.get_text(accessible, ...) (static
call) return the real typed text.

* fix(linux): resolve click-miss, dropped '=', and GTK4 text exposure defects

Three defects surfaced by CI on #1935 (Linux Smoke lane), all confirmed
live via instrumented CI runs before being fixed here:

1. Click misses its target: Component.get_extents(Atspi.CoordType.SCREEN)
   returns (0, 0) as the origin for every non-toplevel widget under this
   GTK4 build — confirmed by a raw click at the computed rect center
   landing on the window's own header-bar button instead of the intended
   digit button. CoordType.WINDOW gives correct, distinct per-widget
   offsets, so get_rect() now computes screen-absolute rects as that
   offset plus the enclosing top-level frame's own (correct) screen
   origin, threaded through traverse_node() alongside the existing
   window-title tracking. Complementary hardening: role "label" is now
   excluded from `hittable`, since GTK4 wraps every button's caption in a
   same-rect "label" child, and the shared cross-platform promotion logic
   in interaction-targeting.ts would otherwise retarget a click from the
   button onto that non-interactive label.

2. Typed '=' never arrives: a single isolated synthetic keystroke sent
   right after a focus change is unreliably delivered — confirmed live,
   both `xdotool type -- "="` and `xdotool key equal` sent alone produced
   no character at all, while multi-character bursts always landed in
   full. typeLinux and sendKey now wait a short settle margin before
   dispatching to xdotool/ydotool, absorbing the race regardless of which
   action last changed focus.

3. GTK4 apps expose no editable text: accessible.get_text_iface().get_text()
   throws "Atspi.Accessible.get_text() takes exactly 1 argument (3 given)"
   — a documented PyGObject binding collision between Text.get_text and
   the deprecated 1-argument Accessible.get_text, silently swallowed as
   "no text" by the broad exception handler. get_text_value() now calls
   the unbound Atspi.Text.get_text(accessible, ...) form, which correctly
   returns the real content.

The Linux smoke replay is restored to exercise all three fixes together
(click a resolved digit button, type a full calculation including the
'=' keystroke, wait on the computed result through the tree) instead of
staying at the weakened, contract-tier assertions the defects had forced.
The coverage manifest promotes click and type from command-contract to
live accordingly.

* fix(linux): drop unproven keyboard-settle and hittable changes per review

Addresses thymikee's review on #1949 (both points correct):

P1: the keyboard settle (typeLinux/sendKey) was unjustified. The cited
diagnostic evidence for a dropped '=' actually shows the opposite —
"100+55=" and "5=5" both computed correctly with zero settle, proving
'=' was delivered in every multi-character burst tested. Sending '='
alone to an empty entry showing a blank display is normal calculator
semantics (nothing to evaluate), not a lost keystroke. The likelier
explanation for the original "100+55" screenshot (run 32487868346) is
that its attempt-3 hit the already-fixed mousemove --sync hang, not an
independent keyboard-dispatch defect. Reverted; no keyboard-dispatch
change was needed.

P2: the `role_name != "label"` hittable narrowing was extra surface
beyond what the click-miss fix required. The corrected AT-SPI coordinates
alone fix the observed miss — the button and its same-rect label child
resolve to nearly identical centers, so descendant promotion still lands
inside the button either way, and the replay can't distinguish which
node it actually targeted. Reverted; only the coordinate fix remains.
This commit is contained in:
Michał Pierzchała
2026-08-21 20:02:17 +02:00
committed by GitHub
parent 1f80c92a27
commit 3bf3ff130a
4 changed files with 92 additions and 36 deletions
+54 -9
View File
@@ -22,19 +22,48 @@ MAX_DESKTOP_APPS = 24
VALID_SURFACES = ("desktop", "frontmost-app")
def get_rect(accessible):
def get_screen_origin(accessible):
"""The screen-absolute (x, y) of a top-level frame/window's own Component extents.
Unlike descendant widgets (see get_rect), a top-level's own SCREEN-coordinate extents are
correct — GTK4's AT-SPI bridge only loses the translation when walking from a widget up
through its ancestor chain to the root, not for the root itself.
"""
try:
component = accessible.get_component_iface()
if not component:
return None
extents = component.get_extents(Atspi.CoordType.SCREEN)
if not extents:
return None
return (extents.x, extents.y)
except Exception:
return None
def get_rect(accessible, frame_origin):
"""A node's screen-absolute rect.
GTK4's AT-SPI bridge returns (0, 0) as the origin of Component.get_extents(SCREEN) for every
non-toplevel widget — confirmed live (CI run 32503838660): a raw xdotool click at the
computed center of a digit button's "screen" rect landed on the window's own header-bar
button instead. CoordType.WINDOW gives correct, distinct per-widget offsets, so the
screen-absolute rect is that offset plus the enclosing top-level frame's own (correct)
screen origin.
"""
try:
component = accessible.get_component_iface()
if not component:
return None
extents = component.get_extents(Atspi.CoordType.WINDOW)
if not extents:
return None
if extents.width <= 0 or extents.height <= 0:
return None
origin_x, origin_y = frame_origin if frame_origin else (0, 0)
return {
"x": extents.x,
"y": extents.y,
"x": extents.x + origin_x,
"y": extents.y + origin_y,
"width": extents.width,
"height": extents.height,
}
@@ -43,14 +72,24 @@ def get_rect(accessible):
def get_text_value(accessible):
"""The Text interface's content, if any.
Must call Atspi.Text.get_text(accessible, ...) as an unbound/static call, NOT
accessible.get_text_iface().get_text(...) — the bound form resolves to the deprecated
1-argument Atspi.Accessible.get_text() instead (a documented PyGObject binding collision:
https://discourse.gnome.org/t/how-can-i-explicitly-call-atspi-text-get-text/36684), raising
"takes exactly 1 argument (3 given)" for every node, silently swallowed as "no text" by the
except-Exception below. Confirmed live (CI run 32505154107): the static form correctly
returns typed text ("155") where the bound form threw on the same node.
"""
try:
text_iface = accessible.get_text_iface()
if not text_iface:
return None
count = text_iface.get_character_count()
count = Atspi.Text.get_character_count(accessible)
if count <= 0:
return None
value = text_iface.get_text(0, count)
value = Atspi.Text.get_text(accessible, 0, count)
return value if value else None
except Exception:
return None
@@ -76,7 +115,7 @@ def has_state(state_set, state_type):
return False
def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=None):
def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=None, frame_origin=None):
if len(ctx["nodes"]) >= ctx["max_nodes"] or depth > ctx["max_depth"] or not accessible:
return
@@ -96,7 +135,13 @@ def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=N
description = ""
label = name or description or None
rect = get_rect(accessible)
# Entering a new top-level resets the frame origin used to translate its descendants'
# WINDOW-relative extents to screen-absolute (see get_rect) — each frame/dialog is a
# separate X11 top-level with its own screen position.
is_frame = role_name in ("frame", "window", "dialog")
effective_frame_origin = (get_screen_origin(accessible) or frame_origin) if is_frame else frame_origin
rect = get_rect(accessible, effective_frame_origin)
try:
state_set = accessible.get_state_set()
@@ -110,7 +155,7 @@ def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=N
hittable = (enabled is not False) and visible and showing and (rect is not None)
current_window_title = window_title
if current_window_title is None and role_name in ("frame", "window", "dialog"):
if current_window_title is None and is_frame:
current_window_title = label
nodes = ctx["nodes"]
@@ -147,7 +192,7 @@ def traverse_node(accessible, depth, parent_index, ctx, app_info, window_title=N
if child:
traverse_node(
child, depth + 1, node_index, ctx, app_info,
current_window_title
current_window_title, effective_frame_origin
)
except Exception:
pass
+10 -14
View File
@@ -160,11 +160,11 @@ export const LINUX_PLATFORM_COVERAGE = {
[C.record]: gap('No Linux-specific recording command evidence exists yet'),
[C.trace]: gap('No Linux-specific trace command evidence exists yet'),
[C.find]: gap('No Linux-specific find command evidence exists yet'),
[C.click]: contract(
LINUX_PROVIDER_EVIDENCE.path,
LINUX_PROVIDER_EVIDENCE.test,
'Linux provider scenario executes primary, secondary, middle, and double clicks',
),
// Promoted from command-contract to live: the desktop replay now clicks a resolved digit
// button on real Linux hardware and the downstream wait only passes if the click landed
// (formerly missed — AT-SPI extents were computed screen-absolute-wrong under GTK4; see
// linux/atspi-dump.py).
[C.click]: live('the Linux desktop replay clicks a resolved calculator digit button'),
[C.fill]: contract(
LINUX_PROVIDER_EVIDENCE.path,
LINUX_PROVIDER_EVIDENCE.test,
@@ -181,15 +181,11 @@ export const LINUX_PLATFORM_COVERAGE = {
LINUX_PROVIDER_EVIDENCE.test,
'Linux provider scenario presses a snapshot ref and coordinate target',
),
// The desktop replay runs the migrated typeText path on real hardware and uploads pixel
// evidence of the typed entry each run, but GTK4 gnome-calculator exposes no Text-interface
// content to selectors, so no tree-level assertion can hold and the claim stays at the
// contract tier until that platform defect is fixed.
[C.type]: contract(
'src/platforms/linux/__tests__/input-actions.test.ts',
'typeLinux uses ydotool type',
'Linux type dispatch uses the Wayland ydotool type primitive',
),
// Promoted from command-contract to live: GTK4 gnome-calculator's entry previously exposed no
// Text-interface content to selectors (a PyGObject binding call-pattern bug — see
// linux/atspi-dump.py), so no tree-level assertion could hold. Fixed, so the desktop replay's
// typed calculation now has a real wait assertion on the computed result.
[C.type]: live('the Linux desktop replay types a calculation and its result is selectable'),
[C.get]: contract(
LINUX_PROVIDER_EVIDENCE.path,
LINUX_PROVIDER_EVIDENCE.test,
@@ -16,13 +16,27 @@ snapshot -i
focus 100 100
# The session survives the focus: a crashed desktop would fail here, not silently pass above.
is exists "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5"
# A resolved button press: proves pointer input lands on a resolved target. role= disambiguates
# from the calculator's [text] "1" node — a bare label=1 is an AMBIGUOUS_MATCH rejection by
# design. Formerly missed: AT-SPI's Component.get_extents(SCREEN) returns (0,0) for every
# non-toplevel widget under this GTK4 build, so the click landed on the window's own header-bar
# button instead — fixed by computing screen-absolute rects from CoordType.WINDOW plus the
# toplevel frame's own screen origin (linux/atspi-dump.py).
click "role=button label=1"
# R41 (#1739): `type` executes through the bound `typeText` operation rather than the retired
# interactor leaf. The typed-state screenshot below is the live evidence: run 32490373693's
# artifact shows the typed digits in the calculator entry. A selector assertion on the value is
# impossible today — GTK4 gnome-calculator exposes no Text-interface content through the AT-SPI
# dumper (display showed "155" while a wait for value=155 timed out) — so the tree-level claim
# for `type` stays at the command-contract tier until the exposure defect is fixed.
type "155"
# interactor leaf. The keystrokes spell a full calculation: the clicked 1 plus the typed 00+55=
# can only produce a 155 result if every keystroke — digits, the shift-composed '+', and the
# trailing '=' — actually landed; no calculator button is labelled 155. (Run 32487868346's
# "100+55" without '=' motivated a keystroke-loss investigation, but the diagnostic evidence for
# that (CI run 32503838660) turned out to show '=' delivered correctly in every multi-character
# burst tested; the run's own attempt-3 hit the already-fixed mousemove --sync hang, which is the
# more likely explanation for that screenshot. No keyboard-dispatch change was needed here.)
type "00+55="
screenshot "./test/screenshots/replays/linux-calculator-typed.png"
# The session survives the typing; a wedged desktop fails here instead of silently passing.
is exists "appname=gnome-calculator || windowtitle=Calculator || label=Calculator || label=0 || label=1 || label=5"
# GTK4 gnome-calculator's entry widget did not expose its content to selectors: AT-SPI's
# Text.get_text() called as accessible.get_text_iface().get_text(...) throws — a PyGObject
# binding collision with the deprecated 1-arg Accessible.get_text() — silently swallowed as "no
# text". Fixed by calling the unbound Atspi.Text.get_text(accessible, ...) form instead. 155
# appears only as the computed result: no button carries that label, and deleting any keystroke
# above turns this wait red.
wait "value=155 || label=155 || text=155" 10000
@@ -40,11 +40,12 @@ test('Linux coverage exhaustively classifies the public catalog', () => {
test('Linux coverage report has the expected classification counts', () => {
assert.deepEqual(LINUX_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY, {
capabilityDenial: 11,
// focus (#1925) is live via the replay; type runs there too but GTK4 blocks a tree-level
// assertion, so its claim stays contract-tier (see the manifest entry).
contract: 19,
// focus (#1925), click, and type are live via the replay: click resolves and lands on a
// digit button, and the typed calculation's result is now selectable (see the manifest
// entries for the platform defects this fixed).
contract: 17,
gap: 18,
live: 6,
live: 8,
total: 54,
});
@@ -61,7 +62,7 @@ test('Linux live claims reference commands in the existing smoke replay', () =>
parseReplayScriptDetailed(replaySource).actions.map((action) => action.command),
);
const liveCommands = liveCommandsForLinuxReplay();
assert.deepEqual(liveCommands.length, 6);
assert.deepEqual(liveCommands.length, 8);
for (const command of liveCommands) {
assert.equal(
replayCommands.has(command),