feat(screenshot): add --crop-on to crop captures to a selector frame (#2276)

* feat(screenshot): add crop-on geometry core and cropTarget selector rows

* feat(screenshot): declare crop-on flag, script round-trip, and snapshot runtime plan

* feat(screenshot): run the crop leaf after the platform write and before scale

* feat(screenshot): expose --crop-on in the CLI and surface crop warnings

* chore(gates): declare crop-on capture-kit subpaths and scope the crop scenario exemption

* refactor(screenshot): split crop target/policy module and trim redundant coverage

Address review comments at 570da2c417:
- Split the 328-line screenshot-crop.ts leaf: the target acceptance matrix,
  classifier, and pre-device argument policy move to screenshot-crop-target.ts,
  so both implementation modules meet the 300-line target.
- Reuse kernel isPositiveFiniteRect/rectArea in the rect-projection module
  instead of redefining them locally.
- Drop the crop-on CLI forwarding case (redundant with screenshot-options
  flag-mapping coverage + the generic dispatcher) and the transport-based
  warnings case, replacing the latter with a focused screenshot-result unit
  test. This also returns the two legacy aggregate test files to their
  merge-base length for the test-file size ratchet.

* refactor(screenshot): extract macOS crop-target decision to keep classifier under the complexity budget

classifyAppleCropTarget inlined the macOS surface decision, pushing its
cyclomatic complexity to the fallow threshold. Move it back out to a
small helper so the target classifier stays within budget.

* refactor(screenshot): dedupe the meaningful-signal predicate and polish png-crop

- Hoist isMeaningfulSignal into @agent-device/contracts/snapshot (next to
  normalizeType/isMeaningfulLabel) so the ref overlay and the crop
  rect-projection share one copy instead of each carrying an identical
  private predicate. Behavior is unchanged.
- png-crop: isCropBox was a no-op 'box is Rect' predicate (input already
  Rect) — make it a plain boolean, and tighten the doc to the contract.

* refactor(screenshot): drop the dead crop outcome flag and cover the projection seams

- ScreenshotCropOutcome.cropped was a constant true that no caller read;
  the crop either returns (success) or throws, so the outcome reduces to
  the partialIntersection observation.
- resolveScreenshotRectSpace and resolveSnapshotBounds were the only
  projection exports without coverage: pin the accepted-backend map, the
  unaccepted-backend typed refusal, and the viewport-root / union / empty
  bounds branches.
This commit is contained in:
Michał Pierzchała
2026-09-04 11:18:57 +02:00
committed by GitHub
parent 658f822c40
commit 172ee149cf
41 changed files with 1899 additions and 155 deletions
@@ -0,0 +1,92 @@
[
{
"name": "iOS simulator 1x capture: viewport-space rect projects 1:1 against a zero-origin bounds",
"space": "viewport-points",
"bounds": { "x": 0, "y": 0, "width": 402, "height": 874 },
"rect": { "x": 16, "y": 293.33, "width": 370, "height": 52 },
"image": { "width": 402, "height": 874 },
"expectedProjection": { "x": 16, "y": 293, "width": 370, "height": 52 },
"expectedIntersection": { "x": 16, "y": 293, "width": 370, "height": 52 }
},
{
"name": "iOS simulator 3x capture: the same points rect scales by imgW/bounds.width and rounds",
"space": "viewport-points",
"bounds": { "x": 0, "y": 0, "width": 402, "height": 874 },
"rect": { "x": 16, "y": 293.33, "width": 370, "height": 52 },
"image": { "width": 1206, "height": 2622 },
"expectedProjection": { "x": 48, "y": 880, "width": 1110, "height": 156 },
"expectedIntersection": { "x": 48, "y": 880, "width": 1110, "height": 156 }
},
{
"name": "Android: device-pixel rects project 1:1 with no bounds lookup",
"space": "device-pixels",
"bounds": null,
"rect": { "x": 210, "y": 678, "width": 436, "height": 71 },
"image": { "width": 1080, "height": 2400 },
"expectedProjection": { "x": 210, "y": 678, "width": 436, "height": 71 },
"expectedIntersection": { "x": 210, "y": 678, "width": 436, "height": 71 }
},
{
"name": "macOS app window: non-zero-origin bounds shift the rect before the 2x scale",
"space": "viewport-points",
"bounds": { "x": 898, "y": 74, "width": 586, "height": 488 },
"rect": { "x": 906, "y": 114, "width": 20, "height": 20 },
"image": { "width": 1172, "height": 976 },
"expectedProjection": { "x": 16, "y": 80, "width": 40, "height": 40 },
"expectedIntersection": { "x": 16, "y": 80, "width": 40, "height": 40 }
},
{
"name": "viewport-points without bounds degrades to a rounded 1:1 rect instead of dividing by zero",
"space": "viewport-points",
"bounds": null,
"rect": { "x": 16.6, "y": 293.33, "width": 370, "height": 52 },
"image": { "width": 402, "height": 874 },
"expectedProjection": { "x": 17, "y": 293, "width": 370, "height": 52 },
"expectedIntersection": { "x": 17, "y": 293, "width": 370, "height": 52 }
},
{
"name": "scrolled-out row above the image floor intersects nothing",
"space": "viewport-points",
"bounds": { "x": 0, "y": 0, "width": 402, "height": 874 },
"rect": { "x": 16, "y": 900, "width": 370, "height": 52 },
"image": { "width": 402, "height": 874 },
"expectedProjection": { "x": 16, "y": 900, "width": 370, "height": 52 },
"expectedIntersection": null
},
{
"name": "row grazing the bottom edge clips to the image and stays partial",
"space": "device-pixels",
"bounds": null,
"rect": { "x": 210, "y": 2360, "width": 436, "height": 111 },
"image": { "width": 1080, "height": 2400 },
"expectedProjection": { "x": 210, "y": 2360, "width": 436, "height": 111 },
"expectedIntersection": { "x": 210, "y": 2360, "width": 436, "height": 40 }
},
{
"name": "rect left of the image edge clips to x=0",
"space": "device-pixels",
"bounds": null,
"rect": { "x": -40, "y": 100, "width": 200, "height": 50 },
"image": { "width": 1080, "height": 2400 },
"expectedProjection": { "x": -40, "y": 100, "width": 200, "height": 50 },
"expectedIntersection": { "x": 0, "y": 100, "width": 160, "height": 50 }
},
{
"name": "rect fully past the right edge intersects nothing",
"space": "device-pixels",
"bounds": null,
"rect": { "x": 1080, "y": 100, "width": 100, "height": 50 },
"image": { "width": 1080, "height": 2400 },
"expectedProjection": { "x": 1080, "y": 100, "width": 100, "height": 50 },
"expectedIntersection": null
},
{
"name": "zero-area intersection (edge touch only) is empty, not a 1px sliver",
"space": "device-pixels",
"bounds": null,
"rect": { "x": 1079, "y": 2400, "width": 10, "height": 10 },
"image": { "width": 1080, "height": 2400 },
"expectedProjection": { "x": 1079, "y": 2400, "width": 10, "height": 10 },
"expectedIntersection": null
}
]
@@ -135,6 +135,34 @@ test('screenshot replay script round-trips screenshot flags', () => {
assert.equal(parsed[0]?.flags.screenshotNoStabilize, true);
});
test('screenshot replay script round-trips a quoted --crop-on selector', () => {
const actions: SessionAction[] = [
{
ts: Date.now(),
command: 'screenshot',
positionals: ['./page.png'],
flags: {
screenshotCropOn: 'role=cell label=General || role=button label=General',
screenshotScale: 0.3,
},
},
];
const script = formatReplayScriptForTest(actions);
assert.match(
script,
/screenshot "\.\/page\.png" --crop-on "role=cell label=General \|\| role=button label=General" --scale 0\.3/,
);
const parsed = parseReplayScriptDetailed(script).actions;
assert.deepEqual(parsed[0]?.positionals, ['./page.png']);
assert.equal(
parsed[0]?.flags.screenshotCropOn,
'role=cell label=General || role=button label=General',
);
assert.equal(parsed[0]?.flags.screenshotScale, 0.3);
});
test('snapshot replay script parses full refresh flags', () => {
const ignoredLegacyFlag = '-' + 'c';
const parsed = parseReplayScriptDetailed(
@@ -227,6 +227,10 @@ export function appendScreenshotActionScriptArgs(parts: string[], action: Sessio
for (const positional of action.positionals ?? []) {
parts.push(formatScriptArg(positional));
}
const cropOn = action.flags?.screenshotCropOn;
if (typeof cropOn === 'string' && cropOn.length > 0) {
parts.push('--crop-on', formatScriptArgQuoteIfNeeded(cropOn));
}
appendScreenshotScriptFlags(parts, action.flags);
}
+8
View File
@@ -34,6 +34,10 @@
"types": "./src/png.ts",
"default": "./src/png.ts"
},
"./png-crop": {
"types": "./src/png-crop.ts",
"default": "./src/png-crop.ts"
},
"./png-resize": {
"types": "./src/png-resize.ts",
"default": "./src/png-resize.ts"
@@ -73,6 +77,10 @@
"./snapshot-quality-verdict": {
"types": "./src/snapshot-quality-verdict.ts",
"default": "./src/snapshot-quality-verdict.ts"
},
"./snapshot-rect-projection": {
"types": "./src/snapshot-rect-projection.ts",
"default": "./src/snapshot-rect-projection.ts"
}
},
"devDependencies": {
+97
View File
@@ -0,0 +1,97 @@
import { afterAll, test } from 'vitest';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { PNG } from './png.ts';
import { cropPngFile } from './png-crop.ts';
import { terminatePngWorker } from './png-worker-client.ts';
import { mkdtempForTestSync } from './tmp-dir.fixtures.ts';
afterAll(async () => {
await terminatePngWorker();
});
test('cropPngFile keeps only the box rows and columns, in place', async () => {
const filePath = writeCheckedPng();
await cropPngFile(filePath, { x: 2, y: 1, width: 3, height: 2 });
const cropped = PNG.sync.read(fs.readFileSync(filePath));
assert.equal(cropped.width, 3);
assert.equal(cropped.height, 2);
assert.deepEqual(readPngPixel(cropped, 0, 0), pixel(2, 1));
assert.deepEqual(readPngPixel(cropped, 2, 1), pixel(4, 2));
});
test('a box grazing the image edges crops to the edge without clamping the origin', async () => {
const filePath = writeCheckedPng();
await cropPngFile(filePath, { x: 4, y: 2, width: 2, height: 2 });
const cropped = PNG.sync.read(fs.readFileSync(filePath));
assert.equal(cropped.width, 2);
assert.equal(cropped.height, 2);
assert.deepEqual(readPngPixel(cropped, 1, 1), pixel(5, 3));
});
test('a full-image box is a no-op that leaves the file decodable at the same size', async () => {
const filePath = writeCheckedPng();
const before = fs.readFileSync(filePath);
await cropPngFile(filePath, { x: 0, y: 0, width: 6, height: 4 });
assert.deepEqual(fs.readFileSync(filePath), before);
});
test('boxes that exceed the image refuse instead of clamping', async () => {
const filePath = writeCheckedPng();
await assert.rejects(
() => cropPngFile(filePath, { x: 5, y: 3, width: 3, height: 3 }),
(error: unknown) => error instanceof Error && error.message.includes('exceeds'),
);
});
test('non-integer or non-positive boxes refuse', async () => {
const filePath = writeCheckedPng();
const boxes = [
{ x: 1.5, y: 0, width: 2, height: 2 },
{ x: 0, y: -1, width: 2, height: 2 },
{ x: 0, y: 0, width: 0, height: 2 },
{ x: 0, y: 0, width: 2, height: 0 },
] as const;
for (const box of boxes) {
await assert.rejects(
() => cropPngFile(filePath, box),
(error: unknown) => error instanceof Error && error.message.includes('positive integer'),
JSON.stringify(box),
);
}
});
// A 6x4 grid whose pixel (x, y) carries (x*10, y*10) so a wrong source offset
// is caught by the value, not just the size.
function writeCheckedPng(): string {
const filePath = path.join(mkdtempForTestSync('agent-device-png-crop-'), 'image.png');
const png = new PNG({ width: 6, height: 4 });
for (let y = 0; y < png.height; y += 1) {
for (let x = 0; x < png.width; x += 1) {
const offset = (y * png.width + x) * 4;
png.data[offset] = x * 10;
png.data[offset + 1] = y * 10;
png.data[offset + 2] = 0;
png.data[offset + 3] = 255;
}
}
fs.writeFileSync(filePath, PNG.sync.write(png));
return filePath;
}
function pixel(x: number, y: number): number[] {
return [x * 10, y * 10, 0, 255];
}
function readPngPixel(png: PNG, x: number, y: number): number[] {
const offset = (y * png.width + x) * 4;
return [
png.data[offset] ?? 0,
png.data[offset + 1] ?? 0,
png.data[offset + 2] ?? 0,
png.data[offset + 3] ?? 0,
];
}
+54
View File
@@ -0,0 +1,54 @@
import { promises as fs } from 'node:fs';
import { AppError } from '@agent-device/kernel/errors';
import type { Rect } from '@agent-device/kernel/snapshot';
import { PNG } from './png.ts';
import { decodePngAsync, encodePngAsync } from './png-worker-client.ts';
/**
* Crops `filePath` in place to `box` (positive integer pixels). `box` is the caller's
* already-intersected region, so one outside the image is a caller bug — refused, not clamped.
* Decode and encode run on the PNG worker thread; a full-image box is a no-op.
*/
export async function cropPngFile(filePath: string, box: Rect): Promise<void> {
if (!isCropBox(box)) {
throw new AppError(
'INVALID_ARGS',
'Screenshot crop box must be positive integer pixel offsets',
);
}
const source = await decodePngAsync(await fs.readFile(filePath), 'screenshot');
if (box.x + box.width > source.width || box.y + box.height > source.height) {
throw new AppError(
'INVALID_ARGS',
`Screenshot crop box ${box.width}x${box.height} at (${box.x}, ${box.y}) exceeds the ${source.width}x${source.height} image`,
);
}
if (box.x === 0 && box.y === 0 && box.width === source.width && box.height === source.height) {
return;
}
await fs.writeFile(filePath, await encodePngAsync(cropPngBox(source, box)));
}
function isCropBox(box: Rect): boolean {
return (
Number.isInteger(box.x) &&
box.x >= 0 &&
Number.isInteger(box.y) &&
box.y >= 0 &&
Number.isInteger(box.width) &&
box.width > 0 &&
Number.isInteger(box.height) &&
box.height > 0
);
}
function cropPngBox(source: PNG, box: Rect): PNG {
const output = new PNG({ width: box.width, height: box.height });
for (let row = 0; row < box.height; row += 1) {
const sourceStart = ((row + box.y) * source.width + box.x) * 4;
source.data.copy(output.data, row * output.width * 4, sourceStart, sourceStart + box.width * 4);
}
return output;
}
@@ -0,0 +1,114 @@
import { test } from 'vitest';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
import { AppError } from '@agent-device/kernel/errors';
import type { Rect } from '@agent-device/kernel/snapshot';
import {
intersectScreenshotRect,
projectSnapshotRectToScreenshot,
resolveScreenshotRectSpace,
resolveSnapshotBounds,
type ScreenshotRectSpace,
} from './snapshot-rect-projection.ts';
// Golden geometry table for `screenshot --crop-on`: the numbers are the live
// evidence baseline (iOS simulator 402/1206, Android 1080, macOS 586/1172),
// so a projection-law change that moves any accepted cell's crop box turns this
// red before the pixel-identity cross-check could ever notice.
type FixtureCase = {
name: string;
space: ScreenshotRectSpace;
bounds: Rect | null;
rect: Rect;
image: { width: number; height: number };
expectedProjection: Rect;
expectedIntersection: Rect | null;
};
const TABLE_PATH = path.resolve(
import.meta.dirname,
'..',
'..',
'..',
'contracts',
'fixtures',
'screenshot-crop-geometry.json',
);
test('the crop projection law agrees with every golden geometry case', () => {
const cases = JSON.parse(fs.readFileSync(TABLE_PATH, 'utf8')) as FixtureCase[];
assert.ok(cases.length > 0, 'geometry table must not be empty');
const names = new Set(cases.map((fixture) => fixture.name));
assert.equal(names.size, cases.length, 'geometry table case names must be unique');
for (const fixture of cases) {
const projected = projectSnapshotRectToScreenshot(
fixture.space,
fixture.bounds,
fixture.rect,
fixture.image.width,
fixture.image.height,
);
assert.deepEqual(projected, fixture.expectedProjection, fixture.name);
const intersection = intersectScreenshotRect(
projected,
fixture.image.width,
fixture.image.height,
);
assert.deepEqual(intersection, fixture.expectedIntersection, fixture.name);
}
});
test('resolveScreenshotRectSpace maps each accepted backend to its projection space and refuses the rest', () => {
assert.equal(resolveScreenshotRectSpace('android'), 'device-pixels');
assert.equal(resolveScreenshotRectSpace('xctest'), 'viewport-points');
assert.equal(resolveScreenshotRectSpace('macos-helper'), 'viewport-points');
for (const backend of [undefined, 'linux', 'web', 'harmonyos', 'vega']) {
assert.throws(
() => resolveScreenshotRectSpace(backend),
(error: unknown) =>
error instanceof AppError &&
error.code === 'UNSUPPORTED_OPERATION' &&
error.details?.reason === SCREENSHOT_CROP_REASONS.targetNotAccepted,
);
}
});
test('resolveSnapshotBounds prefers the largest viewport root, else unions every positive rect excluding unlabeled images', () => {
assert.deepEqual(
resolveSnapshotBounds([
{ type: 'XCUIElementTypeImage', label: '', rect: { x: 0, y: 0, width: 50, height: 50 } },
{
type: 'XCUIElementTypeApplication',
label: 'Settings',
rect: { x: 0, y: 0, width: 402, height: 874 },
},
{
type: 'XCUIElementTypeButton',
label: 'Continue',
rect: { x: 100, y: 100, width: 200, height: 50 },
},
]),
{ x: 0, y: 0, width: 402, height: 874 },
);
assert.deepEqual(
resolveSnapshotBounds([
{ type: 'XCUIElementTypeButton', label: 'A', rect: { x: 10, y: 20, width: 100, height: 40 } },
{
type: 'XCUIElementTypeButton',
label: 'B',
rect: { x: 300, y: 400, width: 60, height: 30 },
},
{ type: 'XCUIElementTypeImage', label: '', rect: { x: 0, y: 0, width: 999, height: 999 } },
]),
{ x: 10, y: 20, width: 350, height: 410 },
);
assert.equal(
resolveSnapshotBounds([
{ type: 'XCUIElementTypeButton', label: 'A', rect: { x: 0, y: 0, width: 0, height: 50 } },
]),
null,
);
});
@@ -0,0 +1,138 @@
import { AppError } from '@agent-device/kernel/errors';
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
import {
isMeaningfulSignal,
isViewportRootNode,
normalizeType,
} from '@agent-device/contracts/snapshot';
import { isPositiveFiniteRect, rectArea } from '@agent-device/kernel/rect';
import type { Rect, SnapshotNode } from '@agent-device/kernel/snapshot';
/**
* The space a snapshot's rects are expressed in, as far as the captured image is concerned:
* `device-pixels` rects land 1:1 on the PNG (Android), `viewport-points` rects must be shifted
* by the viewport bounds and scaled to the image (Apple, where 1x/2x/3x captures all share the
* points-space tree).
*/
export type ScreenshotRectSpace = 'device-pixels' | 'viewport-points';
/**
* The single decision site for `screenshot --crop-on`: which projection law a snapshot backend's
* rects obey. Backends the crop feature has not accepted project nothing, so an unaccepted
* backend is a typed refusal, never a guess.
*/
export function resolveScreenshotRectSpace(backend: string | undefined): ScreenshotRectSpace {
switch (backend) {
case 'android':
return 'device-pixels';
case 'xctest':
case 'macos-helper':
return 'viewport-points';
default:
throw new AppError(
'UNSUPPORTED_OPERATION',
`screenshot --crop-on does not accept snapshot backend "${backend ?? 'unknown'}"`,
{ reason: SCREENSHOT_CROP_REASONS.targetNotAccepted },
);
}
}
/**
* Where a snapshot rect falls in the captured image: the space-specific law only, with NO
* clamping. Callers decide what out-of-image means — the overlay clamps to a 1px box, the crop
* intersects and refuses the empty case.
*/
export function projectSnapshotRectToScreenshot(
space: ScreenshotRectSpace,
bounds: Rect | null,
rect: Rect,
imageWidth: number,
imageHeight: number,
): Rect {
if (space === 'device-pixels' || bounds === null) {
return roundRect(rect);
}
const scaleX = imageWidth / bounds.width;
const scaleY = imageHeight / bounds.height;
return {
x: Math.round((rect.x - bounds.x) * scaleX),
y: Math.round((rect.y - bounds.y) * scaleY),
width: Math.round(rect.width * scaleX),
height: Math.round(rect.height * scaleY),
};
}
/**
* The crop box a projected rect actually yields: its intersection with the image frame. `null`
* means the rect occupies no image pixel — a crop cannot be taken, not a 1px sliver.
*/
export function intersectScreenshotRect(
rect: Rect,
imageWidth: number,
imageHeight: number,
): Rect | null {
const x = Math.max(rect.x, 0);
const y = Math.max(rect.y, 0);
const right = Math.min(rect.x + rect.width, imageWidth);
const bottom = Math.min(rect.y + rect.height, imageHeight);
if (right <= x || bottom <= y) return null;
return { x, y, width: right - x, height: bottom - y };
}
/**
* The viewport the tree's points-space rects are measured against: the largest viewport root
* with a positive rect, else the union of every rect-carrying node (unlabeled images excluded as
* bounds outliers). One copy, shared by the ref overlay and the crop projection.
*/
export function resolveSnapshotBounds(
nodes: ReadonlyArray<Pick<SnapshotNode, 'type' | 'label' | 'rect'>>,
): Rect | null {
let viewport: Rect | null = null;
for (const node of nodes) {
if (!isViewportRootNode(node) || !isPositiveFiniteRect(node.rect)) continue;
if (!viewport || rectArea(node.rect) > rectArea(viewport)) {
viewport = node.rect;
}
}
if (viewport) return viewport;
return measureSnapshotBounds(
nodes.filter((node) => isPositiveFiniteRect(node.rect) && !isSnapshotBoundsOutlier(node)),
);
}
function measureSnapshotBounds(nodes: ReadonlyArray<Pick<SnapshotNode, 'rect'>>): Rect | null {
let minX = Number.POSITIVE_INFINITY;
let minY = Number.POSITIVE_INFINITY;
let maxRight = Number.NEGATIVE_INFINITY;
let maxBottom = Number.NEGATIVE_INFINITY;
for (const node of nodes) {
if (!isPositiveFiniteRect(node.rect)) continue;
minX = Math.min(minX, node.rect.x);
minY = Math.min(minY, node.rect.y);
maxRight = Math.max(maxRight, node.rect.x + node.rect.width);
maxBottom = Math.max(maxBottom, node.rect.y + node.rect.height);
}
if (!Number.isFinite(minX) || !Number.isFinite(minY) || maxRight <= minX || maxBottom <= minY) {
return null;
}
return {
x: minX,
y: minY,
width: maxRight - minX,
height: maxBottom - minY,
};
}
function isSnapshotBoundsOutlier(node: Pick<SnapshotNode, 'type' | 'label'>): boolean {
return normalizeType(node.type ?? '') === 'image' && !isMeaningfulSignal(node.label);
}
function roundRect(rect: Rect): Rect {
return {
x: Math.round(rect.x),
y: Math.round(rect.y),
width: Math.round(rect.width),
height: Math.round(rect.height),
};
}
+2
View File
@@ -67,6 +67,8 @@ export type CaptureSnapshotResult = {
export type CaptureScreenshotOptions = AgentDeviceRequestOverrides & {
path?: string;
overlayRefs?: boolean;
/** Crop the capture to the frame of the selector resolved on the same screen. */
cropOn?: string;
pixelDensity?: number;
fullscreen?: boolean;
scale?: number;
@@ -3,6 +3,7 @@ export {
RETIRED_SCREENSHOT_MAX_SIZE,
SCREENSHOT_ACTION_FLAG_KEYS,
SCREENSHOT_COMMAND_FLAG_KEYS,
SCREENSHOT_CROP_REASONS,
SCREENSHOT_SCALE_LIMITS,
SCREENSHOT_SPECIFIC_FLAG_DEFINITIONS,
appendScreenshotScriptFlags,
@@ -15,6 +16,7 @@ export {
validateScreenshotScale,
} from '../screenshot.ts';
export type {
ScreenshotCropReason,
ScreenshotDispatchFlags,
ScreenshotPublicOptions,
ScreenshotRequestFlags,
@@ -26,5 +26,6 @@ export {
extractNodeText,
isFillableType,
isMeaningfulLabel,
isMeaningfulSignal,
normalizeType,
} from '../snapshot-text.ts';
@@ -602,35 +602,46 @@ export function resolveSnapshotRuntimePlan(input: {
const captureScreenshotUse = defineUse({ required: ['captureScreenshot'] });
/**
* `--overlay-refs` annotates the capture with the refs of a snapshot taken in the same request, so
* the snapshot is part of what the command requires — not something to discover after the PNG is
* Screenshot post-processing that resolves a snapshot taken in the same request — `--overlay-refs`
* annotates the capture with snapshot refs, `--crop-on` crops it to a snapshot node's frame. The
* snapshot is part of what the command requires, not something to discover after the PNG is
* already on disk. Declaring it in the use is what lets admission refuse the whole request up
* front on a target that can capture pixels but not a tree.
*/
const captureScreenshotWithOverlayRefsUse = defineUse({
const captureScreenshotWithSnapshotUse = defineUse({
required: ['captureScreenshot', 'captureSnapshot'],
});
export const screenshotRuntimePlanUses = Object.freeze([
captureScreenshotUse,
captureScreenshotWithOverlayRefsUse,
captureScreenshotWithSnapshotUse,
] as const);
export type ScreenshotRuntimePlan =
| Readonly<{ kind: 'capture'; use: typeof captureScreenshotUse }>
| Readonly<{
kind: 'capture-with-overlay-refs';
use: typeof captureScreenshotWithOverlayRefsUse;
use: typeof captureScreenshotWithSnapshotUse;
}>
| Readonly<{
kind: 'capture-with-crop-on';
use: typeof captureScreenshotWithSnapshotUse;
}>;
/** Selects one owner-fact-backed capture plan from normalized command intent. */
export function resolveScreenshotRuntimePlan(
input: Readonly<{ overlayRefs: boolean }>,
input: Readonly<{ overlayRefs: boolean; cropOn: boolean }>,
): ScreenshotRuntimePlan {
if (input.cropOn) {
return Object.freeze({
kind: 'capture-with-crop-on',
use: captureScreenshotWithSnapshotUse,
});
}
return input.overlayRefs
? Object.freeze({
kind: 'capture-with-overlay-refs',
use: captureScreenshotWithOverlayRefsUse,
use: captureScreenshotWithSnapshotUse,
})
: Object.freeze({ kind: 'capture', use: captureScreenshotUse });
}
+66 -2
View File
@@ -44,9 +44,33 @@ export function validateNoRetiredScreenshotMaxSize(
}
}
/**
* Machine-readable `screenshot --crop-on` outcome taxonomy. Failures carry these
* values in `error.details.reason`; `partialIntersection` doubles as the
* success-path warning token. Callers branch on the values, never on message
* text. `pendingPixelIdentityEvidence` is the acceptance-matrix rejection
* reason, not a runtime failure.
*/
export const SCREENSHOT_CROP_REASONS = {
selectorInvalid: 'CROP_SELECTOR_INVALID',
targetNotFound: 'CROP_TARGET_NOT_FOUND',
targetAmbiguous: 'CROP_TARGET_AMBIGUOUS',
captureUnreadable: 'CROP_CAPTURE_UNREADABLE',
captureIncomplete: 'CROP_CAPTURE_INCOMPLETE',
emptyIntersection: 'CROP_EMPTY_INTERSECTION',
partialIntersection: 'CROP_PARTIAL_INTERSECTION',
targetNotAccepted: 'CROP_TARGET_NOT_ACCEPTED',
frameMismatch: 'CROP_FRAME_MISMATCH',
pendingPixelIdentityEvidence: 'PENDING_PIXEL_IDENTITY_EVIDENCE',
} as const;
export type ScreenshotCropReason =
(typeof SCREENSHOT_CROP_REASONS)[keyof typeof SCREENSHOT_CROP_REASONS];
export const SCREENSHOT_COMMAND_FLAG_KEYS = [
'out',
'overlayRefs',
'screenshotCropOn',
'screenshotPixelDensity',
'screenshotFullscreen',
'screenshotScale',
@@ -55,6 +79,7 @@ export const SCREENSHOT_COMMAND_FLAG_KEYS = [
] as const;
export const SCREENSHOT_ACTION_FLAG_KEYS = [
'screenshotCropOn',
'screenshotPixelDensity',
'screenshotFullscreen',
'screenshotScale',
@@ -67,7 +92,7 @@ type ScreenshotSpecificFlagKey = (typeof SCREENSHOT_ACTION_FLAG_KEYS)[number];
type ScreenshotSpecificFlagDefinition = {
key: ScreenshotSpecificFlagKey;
names: readonly string[];
type: 'boolean' | 'int' | 'number';
type: 'boolean' | 'int' | 'number' | 'string';
min?: number;
max?: number;
usageLabel: string;
@@ -75,6 +100,14 @@ type ScreenshotSpecificFlagDefinition = {
};
export const SCREENSHOT_SPECIFIC_FLAG_DEFINITIONS: readonly ScreenshotSpecificFlagDefinition[] = [
{
key: 'screenshotCropOn',
names: ['--crop-on'],
type: 'string',
usageLabel: '--crop-on <selector-expression>',
usageDescription:
'Screenshot: crop the capture to the frame of the selector resolved on the same screen',
},
{
key: 'screenshotPixelDensity',
names: ['--pixel-density'],
@@ -144,9 +177,18 @@ const SCREENSHOT_SCRIPT_NUMBER_FLAGS = [
},
] as const;
const SCREENSHOT_SCRIPT_STRING_FLAGS = [
{
token: '--crop-on',
key: 'screenshotCropOn',
label: 'screenshot --crop-on',
},
] as const;
export type ScreenshotRequestFlags = {
out?: string;
overlayRefs?: boolean;
screenshotCropOn?: string;
screenshotPixelDensity?: number;
screenshotFullscreen?: boolean;
screenshotScale?: number;
@@ -164,6 +206,7 @@ export type ScreenshotDispatchFlags = Pick<
export type ScreenshotRuntimeFlags = Pick<
ScreenshotRequestFlags,
| 'screenshotCropOn'
| 'screenshotPixelDensity'
| 'screenshotFullscreen'
| 'screenshotScale'
@@ -173,6 +216,7 @@ export type ScreenshotRuntimeFlags = Pick<
export type ScreenshotPublicOptions = {
overlayRefs?: boolean;
cropOn?: string;
pixelDensity?: number;
fullscreen?: boolean;
scale?: number;
@@ -182,6 +226,7 @@ export type ScreenshotPublicOptions = {
export type ScreenshotRuntimeOptions = {
overlayRefs?: boolean;
cropOn?: string;
pixelDensity?: number;
fullscreen?: boolean;
scale?: number;
@@ -194,6 +239,7 @@ export function screenshotOptionsFromFlags(
): ScreenshotRuntimeOptions {
return stripUndefined({
overlayRefs: flags?.overlayRefs,
cropOn: flags?.screenshotCropOn,
pixelDensity: flags?.screenshotPixelDensity,
fullscreen: flags?.screenshotFullscreen,
scale: flags?.screenshotScale,
@@ -207,6 +253,7 @@ export function screenshotFlagsFromOptions(
): Partial<ScreenshotRequestFlags> {
return stripUndefined({
overlayRefs: options.overlayRefs,
screenshotCropOn: options.screenshotCropOn ?? options.cropOn,
screenshotPixelDensity: options.screenshotPixelDensity ?? options.pixelDensity,
screenshotFullscreen: options.screenshotFullscreen ?? options.fullscreen,
screenshotScale: options.screenshotScale,
@@ -269,7 +316,8 @@ export function readScreenshotScriptFlag(params: {
return (
readScreenshotBooleanScriptFlag(token, flags, index) ??
readScreenshotIntScriptFlag({ args, index, flags, token }) ??
readScreenshotNumberScriptFlag({ args, index, flags, token }) ?? { handled: false }
readScreenshotNumberScriptFlag({ args, index, flags, token }) ??
readScreenshotStringScriptFlag({ args, index, flags, token }) ?? { handled: false }
);
}
@@ -324,3 +372,19 @@ function readScreenshotNumberScriptFlag(params: {
params.flags[definition.key] = parsed;
return { handled: true, nextIndex: params.index + 1 };
}
function readScreenshotStringScriptFlag(params: {
args: readonly string[];
index: number;
flags: Partial<ScreenshotRequestFlags>;
token: string | undefined;
}): { handled: true; nextIndex: number } | undefined {
const definition = SCREENSHOT_SCRIPT_STRING_FLAGS.find((entry) => entry.token === params.token);
if (!definition) return undefined;
const value = params.args[params.index + 1];
if (typeof value !== 'string' || value.length === 0) {
throw new AppError('INVALID_ARGS', `${definition.label} requires a selector expression`);
}
params.flags[definition.key] = value;
return { handled: true, nextIndex: params.index + 1 };
}
+8
View File
@@ -38,6 +38,14 @@ export function isMeaningfulLabel(value: string): boolean {
return true;
}
/** A non-empty, non-boolean `label`/`value` is a usable overlay or crop signal. */
export function isMeaningfulSignal(value: string | undefined): boolean {
if (typeof value !== 'string') return false;
const trimmed = value.trim();
if (!trimmed) return false;
return !/^(true|false)$/i.test(trimmed);
}
export function extractNodeText(
node: Pick<RawSnapshotNode, 'label' | 'value' | 'identifier'>,
): string {
+1
View File
@@ -15,6 +15,7 @@ export type ScreenshotResultData = {
logicalHeight?: number;
pixelDensity?: number;
overlayRefs?: ScreenshotOverlayRef[];
warnings?: string[];
};
export type BackendSnapshotResult = {
nodes?: SnapshotNode[];
@@ -89,6 +89,11 @@ export const SELECTOR_RESOLUTION_POLICIES = {
ambiguity: 'reject-candidates',
requireRect: false,
},
/** `screenshot --crop-on` — crops the capture to the resolved node's frame. */
cropTarget: {
ambiguity: 'fail-closed',
requireRect: true,
},
} as const satisfies Record<string, SelectorResolutionPolicy>;
/**
+9
View File
@@ -361,6 +361,15 @@ function summarizeProviderScenarioFlagExclusions() {
owner: 'runner XCTest unit, snapshot-lines, and snapshot-quality tests',
keys: ['snapshotCustomActions'],
},
{
// The crop is daemon-level post-processing: the platform write happens first, then the
// daemon crops the PNG against a fresh snapshot whose pixel/tree identity the fake
// provider scenario fixtures cannot fabricate. Covered instead by the daemon crop-leaf
// unit tests and the live device verification in the feature's PR evidence.
name: 'daemon screenshot selector crop',
owner: 'daemon screenshot-crop unit and live device verification',
keys: ['screenshotCropOn'],
},
];
}
@@ -468,6 +468,7 @@ test('the real tree parses, declares, and passes R11', () => {
'@agent-device/capture-kit/ios-snapshot-planning',
'@agent-device/capture-kit/mobile-snapshot-semantics',
'@agent-device/capture-kit/png',
'@agent-device/capture-kit/png-crop',
'@agent-device/capture-kit/png-resize',
'@agent-device/capture-kit/png-rgb-difference',
'@agent-device/capture-kit/png-size',
@@ -478,6 +479,7 @@ test('the real tree parses, declares, and passes R11', () => {
'@agent-device/capture-kit/snapshot-occlusion',
'@agent-device/capture-kit/snapshot-quality-backend-capabilities',
'@agent-device/capture-kit/snapshot-quality-verdict',
'@agent-device/capture-kit/snapshot-rect-projection',
]);
const provisionKitPackage = packages.find((pkg) => pkg.name === '@agent-device/provision-kit');
@@ -53,6 +53,14 @@ export const WEB_DESKTOP_DEVICE: DeviceInfo = {
booted: true,
};
export const ANDROID_DEVICE: DeviceInfo = {
platform: 'android',
id: 'and-1',
name: 'Pixel 8',
kind: 'device',
booted: true,
};
export const ANDROID_TV_DEVICE: DeviceInfo = {
platform: 'android',
id: 'and-tv-1',
+2 -12
View File
@@ -79,7 +79,7 @@ import {
} from './metro/metro-session-hints.ts';
import { isRecord } from '@agent-device/kernel/record';
import { createLeaseClient } from './client/lease-client.ts';
import { readScreenshotResultData } from './client/screenshot-result.ts';
import { normalizeScreenshotCaptureResult } from './client/screenshot-result.ts';
type ProjectedSystemCommandClient = ProjectedNavigationCommandClient<InternalRequestOptions> &
Pick<AgentDeviceCommandClient, 'appState' | 'keyboard' | 'clipboard'>;
@@ -372,17 +372,7 @@ export function createAgentDeviceClient(
// (The caller opted into a non-default level, so the static type is the
// default shape; the runtime value is the leveled payload.)
if (isLeveledResponse(options)) return data as unknown as CaptureScreenshotResult;
const screenshot = readScreenshotResultData(data);
return {
path: readRequiredString(data, 'path'),
width: screenshot?.width,
height: screenshot?.height,
logicalWidth: screenshot?.logicalWidth,
logicalHeight: screenshot?.logicalHeight,
pixelDensity: screenshot?.pixelDensity,
overlayRefs: screenshot?.overlayRefs,
identifiers: { session },
};
return normalizeScreenshotCaptureResult(data, session);
},
diff: async (options) => await executeCommand<CommandResult<'diff'>>('diff', options),
},
@@ -28,6 +28,8 @@ test('usageForCommand documents screenshot web aliases and stabilization flags',
assert.match(help, /low-latency Android capture loops/);
assert.match(help, /--normalize-status-bar/);
assert.match(help, /deterministic iOS simulator chrome/);
assert.match(help, /--crop-on <selector-expression>/);
assert.match(help, /crop the capture to the frame of the selector/);
});
test('usageForCommand documents screenshot diff normalization', async () => {
@@ -53,6 +53,31 @@ test('screenshot --level digest --json preserves the digest payload through the
assert.deepEqual(parsed.data, digest);
});
const CROP_WARNING =
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame';
test('screenshot surfaces response-level warnings under the summary', async () => {
const full = { path: '/tmp/shot.png', width: 40, height: 20, warnings: [CROP_WARNING] };
const client = clientReturning(full);
const flags = { json: false } as CliFlags;
const out = await captureStdout(() => screenshotCommand({ positionals: [], flags, client }));
assert.equal(out, `/tmp/shot.png (40x20)\n${CROP_WARNING}\n`);
});
test('screenshot --json carries response-level warnings in the data', async () => {
const full = { path: '/tmp/shot.png', width: 40, height: 20, warnings: [CROP_WARNING] };
const client = clientReturning(full);
const flags = { json: true } as CliFlags;
const out = await captureStdout(() => screenshotCommand({ positionals: [], flags, client }));
const parsed = JSON.parse(out) as { data: { path: string; warnings?: string[] } };
assert.equal(parsed.data.path, '/tmp/shot.png');
assert.deepEqual(parsed.data.warnings, [CROP_WARNING]);
});
test('screenshot --json at the default level still emits normalized screenshot metadata', async () => {
const full = {
path: '/tmp/shot.png',
+6 -4
View File
@@ -26,11 +26,13 @@ export const screenshotCommand: ClientCommandHandler = async ({ positionals, fla
return true;
}
const data = pickScreenshotResultData(result);
await writeCommandOutput(flags, data, () =>
result.overlayRefs
await writeCommandOutput(flags, data, () => {
const summary = result.overlayRefs
? `Annotated ${result.overlayRefs.length} refs onto ${result.path}`
: formatScreenshotSummary(result),
);
: formatScreenshotSummary(result);
const warnings = result.warnings ?? [];
return warnings.length > 0 ? [summary, ...warnings].join('\n') : summary;
});
return true;
};
@@ -0,0 +1,43 @@
import assert from 'node:assert/strict';
import { test } from 'vitest';
import {
normalizeScreenshotCaptureResult,
pickScreenshotResultData,
} from '../screenshot-result.ts';
const CROP_WARNING =
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame';
test('normalizeScreenshotCaptureResult surfaces response-level warnings', () => {
const result = normalizeScreenshotCaptureResult(
{
path: '/tmp/screenshot.png',
width: 40,
height: 20,
warnings: [CROP_WARNING],
},
'qa',
);
assert.equal(result.path, '/tmp/screenshot.png');
assert.equal(result.width, 40);
assert.equal(result.height, 20);
assert.deepEqual(result.warnings, [CROP_WARNING]);
assert.deepEqual(result.identifiers, { session: 'qa' });
});
test('normalizeScreenshotCaptureResult omits warnings when the response carries none', () => {
const result = normalizeScreenshotCaptureResult({ path: '/tmp/screenshot.png' }, 'qa');
assert.equal(result.path, '/tmp/screenshot.png');
assert.ok(!('warnings' in result));
assert.deepEqual(result.identifiers, { session: 'qa' });
});
test('pickScreenshotResultData keeps warnings only when present and non-empty', () => {
assert.deepEqual(
pickScreenshotResultData({ path: '/tmp/a.png', width: 40, warnings: [CROP_WARNING] }),
{ path: '/tmp/a.png', width: 40, warnings: [CROP_WARNING] },
);
assert.deepEqual(pickScreenshotResultData({ path: '/tmp/a.png', warnings: [] }), {
path: '/tmp/a.png',
});
});
+55 -21
View File
@@ -1,6 +1,7 @@
import type { ScreenshotResultData } from '@agent-device/contracts/capture';
import type { CaptureScreenshotResult } from '@agent-device/contracts/client';
import { isRecord, parsePoint, parseRect, readRequiredString } from '@agent-device/kernel/record';
import type { ScreenshotOverlayRef } from '@agent-device/kernel/snapshot';
import { isRecord, parsePoint, parseRect } from '@agent-device/kernel/record';
export function pickScreenshotResultData(value: ScreenshotResultData): ScreenshotResultData {
return {
@@ -11,6 +12,26 @@ export function pickScreenshotResultData(value: ScreenshotResultData): Screensho
...(typeof value.logicalHeight === 'number' ? { logicalHeight: value.logicalHeight } : {}),
...(typeof value.pixelDensity === 'number' ? { pixelDensity: value.pixelDensity } : {}),
...(value.overlayRefs ? { overlayRefs: value.overlayRefs } : {}),
...(value.warnings && value.warnings.length > 0 ? { warnings: value.warnings } : {}),
};
}
/** The default-level daemon payload, normalized into the typed client result. */
export function normalizeScreenshotCaptureResult(
data: Record<string, unknown>,
session: string,
): CaptureScreenshotResult {
const screenshot = readScreenshotResultData(data);
return {
path: readRequiredString(data, 'path'),
width: screenshot?.width,
height: screenshot?.height,
logicalWidth: screenshot?.logicalWidth,
logicalHeight: screenshot?.logicalHeight,
pixelDensity: screenshot?.pixelDensity,
overlayRefs: screenshot?.overlayRefs,
...(screenshot?.warnings ? { warnings: screenshot.warnings } : {}),
identifiers: { session },
};
}
@@ -22,31 +43,44 @@ type ScreenshotOverlayRefData = {
center?: unknown;
};
export function readScreenshotResultData(value: unknown): ScreenshotResultData | undefined {
function readScreenshotResultData(value: unknown): ScreenshotResultData | undefined {
if (!isRecord(value)) return undefined;
const path = typeof value.path === 'string' ? value.path : undefined;
const width = typeof value.width === 'number' ? value.width : undefined;
const height = typeof value.height === 'number' ? value.height : undefined;
const logicalWidth = typeof value.logicalWidth === 'number' ? value.logicalWidth : undefined;
const logicalHeight = typeof value.logicalHeight === 'number' ? value.logicalHeight : undefined;
const pixelDensity = typeof value.pixelDensity === 'number' ? value.pixelDensity : undefined;
const overlayRefs = Array.isArray(value.overlayRefs)
? value.overlayRefs.filter(isScreenshotOverlayRefData).flatMap((entry) => {
const overlayRef = readScreenshotOverlayRef(entry);
return overlayRef ? [overlayRef] : [];
})
: undefined;
const warnings = readScreenshotWarnings(value.warnings);
return pickScreenshotResultData({
path,
width,
height,
logicalWidth,
logicalHeight,
pixelDensity,
overlayRefs,
path: readStringField(value, 'path'),
width: readNumberField(value, 'width'),
height: readNumberField(value, 'height'),
logicalWidth: readNumberField(value, 'logicalWidth'),
logicalHeight: readNumberField(value, 'logicalHeight'),
pixelDensity: readNumberField(value, 'pixelDensity'),
overlayRefs: readScreenshotOverlayRefs(value.overlayRefs),
...(warnings !== undefined ? { warnings } : {}),
});
}
function readNumberField(value: Record<string, unknown>, key: string): number | undefined {
const field = value[key];
return typeof field === 'number' ? field : undefined;
}
function readStringField(value: Record<string, unknown>, key: string): string | undefined {
const field = value[key];
return typeof field === 'string' ? field : undefined;
}
function readScreenshotOverlayRefs(value: unknown): ScreenshotOverlayRef[] | undefined {
if (!Array.isArray(value)) return undefined;
return value.filter(isScreenshotOverlayRefData).flatMap((entry) => {
const overlayRef = readScreenshotOverlayRef(entry);
return overlayRef ? [overlayRef] : [];
});
}
function readScreenshotWarnings(value: unknown): string[] | undefined {
if (!Array.isArray(value)) return undefined;
return value.filter((entry): entry is string => typeof entry === 'string' && entry.length > 0);
}
function readScreenshotOverlayRef(
record: ScreenshotOverlayRefData,
): ScreenshotOverlayRef | undefined {
@@ -11,12 +11,14 @@ import {
screenshotOptionsFromFlags,
validateNoRetiredScreenshotMaxSize,
validateScreenshotScale,
type ScreenshotRequestFlags,
} from '@agent-device/contracts/capture';
test('screenshot flag projection maps CLI flags to runtime options', () => {
assert.deepEqual(
screenshotOptionsFromFlags({
overlayRefs: true,
screenshotCropOn: 'role=button label=Save',
screenshotPixelDensity: 2,
screenshotFullscreen: true,
screenshotScale: 0.3,
@@ -25,6 +27,7 @@ test('screenshot flag projection maps CLI flags to runtime options', () => {
}),
{
overlayRefs: true,
cropOn: 'role=button label=Save',
pixelDensity: 2,
fullscreen: true,
scale: 0.3,
@@ -38,6 +41,7 @@ test('screenshot flag projection maps public options to request flags', () => {
assert.deepEqual(
screenshotFlagsFromOptions({
overlayRefs: true,
cropOn: 'label="Network & internet"',
pixelDensity: 3,
fullscreen: true,
stabilize: false,
@@ -45,6 +49,7 @@ test('screenshot flag projection maps public options to request flags', () => {
}),
{
overlayRefs: true,
screenshotCropOn: 'label="Network & internet"',
screenshotPixelDensity: 3,
screenshotFullscreen: true,
screenshotNoStabilize: true,
@@ -84,7 +89,7 @@ test('retired max-size inputs are refused with migration guidance', () => {
test('screenshot script flags use the shared recorded flag contract', () => {
const parts: string[] = [];
const flags = {};
const flags: Partial<ScreenshotRequestFlags> = {};
let result = readScreenshotScriptFlag({ args: ['--full'], index: 0, flags });
assert.deepEqual(result, { handled: true, nextIndex: 0 });
@@ -100,6 +105,17 @@ test('screenshot script flags use the shared recorded flag contract', () => {
assert.deepEqual(result, { handled: true, nextIndex: 0 });
result = readScreenshotScriptFlag({ args: ['--pixel-density', '3'], index: 0, flags });
assert.deepEqual(result, { handled: true, nextIndex: 1 });
result = readScreenshotScriptFlag({
args: ['--crop-on', 'role=cell label=General || role=button label=General'],
index: 0,
flags,
});
assert.deepEqual(result, { handled: true, nextIndex: 1 });
assert.equal(flags.screenshotCropOn, 'role=cell label=General || role=button label=General');
assert.throws(() => readScreenshotScriptFlag({ args: ['--crop-on'], index: 0, flags: {} }), {
code: 'INVALID_ARGS',
message: /requires a selector expression/,
});
appendScreenshotScriptFlags(parts, flags);
@@ -113,6 +129,7 @@ test('screenshot script flags use the shared recorded flag contract', () => {
'--normalize-status-bar',
]);
assert.deepEqual(SCREENSHOT_ACTION_FLAG_KEYS, [
'screenshotCropOn',
'screenshotPixelDensity',
'screenshotFullscreen',
'screenshotScale',
@@ -126,6 +143,7 @@ test('screenshot script flags use the shared recorded flag contract', () => {
assert.deepEqual(SCREENSHOT_COMMAND_FLAG_KEYS, [
'out',
'overlayRefs',
'screenshotCropOn',
'screenshotPixelDensity',
'screenshotFullscreen',
'screenshotScale',
+4 -1
View File
@@ -34,6 +34,9 @@ const screenshotCommandMetadata = defineFieldCommandMetadata(
screenshotCommandDescription,
{
path: stringField('Output path.'),
cropOn: stringField(
'Selector expression; the capture is cropped to the frame the selector resolves on the same screen.',
),
overlayRefs: booleanField(),
pixelDensity: integerField('Output screenshot pixel density in pixels per logical point.', {
min: 1,
@@ -77,7 +80,7 @@ export const screenshotCommandFacet = defineCommandFacet({
text: {
summary: 'Capture a screenshot',
cliDetail:
'Web defaults to the viewport; use --fullscreen, --full, or -f for the entire page. iOS simulators default to 1x logical-point output; use --pixel-density to request a different screenshot density. macOS app sessions default to the app window; use --fullscreen for full desktop, --scale to downscale, --overlay-refs to annotate current refs, --normalize-status-bar for deterministic iOS simulator chrome, or --no-stabilize for low-latency Android capture loops.',
'Web defaults to the viewport; use --fullscreen, --full, or -f for the entire page. iOS simulators default to 1x logical-point output; use --pixel-density to request a different screenshot density. macOS app sessions default to the app window; use --fullscreen for full desktop, --scale to downscale, --crop-on <selector> to crop the capture to the frame the selector resolves on the same screen (currently iOS simulators and Android emulators), --overlay-refs to annotate current refs, --normalize-status-bar for deterministic iOS simulator chrome, or --no-stabilize for low-latency Android capture loops.',
},
metadata: screenshotCommandMetadata,
definition: screenshotCommandDefinition,
@@ -128,9 +128,11 @@ test('disambiguation declines when candidates are genuinely indistinguishable',
});
test('fail-closed rows refuse an ambiguous tree instead of guessing', () => {
const outcome = outcomeFor('readUnique', AMBIGUOUS_TREE);
assert.equal(outcome.kind, 'ambiguous');
if (outcome.kind === 'ambiguous') assert.equal(outcome.matchedNodes.length, 2);
for (const name of ['readUnique', 'cropTarget'] as const) {
const outcome = outcomeFor(name, AMBIGUOUS_TREE);
assert.equal(outcome.kind, 'ambiguous', name);
if (outcome.kind === 'ambiguous') assert.equal(outcome.matchedNodes.length, 2, name);
}
});
test('first-match rows take the head of an ambiguous tree', () => {
@@ -155,7 +157,7 @@ test('reject-candidates surfaces every candidate for the caller to narrow or ref
});
test('rect-requiring rows skip rectless nodes; read and wait rows accept them', () => {
for (const name of ['act', 'findAct', 'actCoveredDiagnosis'] as const) {
for (const name of ['act', 'findAct', 'actCoveredDiagnosis', 'cropTarget'] as const) {
assert.equal(outcomeFor(name, RECTLESS_TREE).kind, 'none', name);
}
for (const name of ['readUnique', 'readAny', 'readText', 'wait'] as const) {
@@ -189,4 +191,5 @@ test('the documented per-caller contracts are the ones declared', () => {
assert.equal(SELECTOR_RESOLUTION_POLICIES.readAny.ambiguity, 'first-match');
assert.equal(SELECTOR_RESOLUTION_POLICIES.wait.ambiguity, 'first-match');
assert.equal(SELECTOR_RESOLUTION_POLICIES.findAct.ambiguity, 'reject-candidates');
assert.equal(SELECTOR_RESOLUTION_POLICIES.cropTarget.ambiguity, 'fail-closed');
});
+8
View File
@@ -190,6 +190,14 @@ export const SELECTOR_PIPELINE_POLICIES = {
promotion: 'hittable-ancestor-below-root',
poll: 'none',
},
/** `screenshot --crop-on`: crops the capture to the resolved node's frame. */
cropTarget: {
resolution: SELECTOR_RESOLUTION_POLICIES.cropTarget,
occlusion: 'ignore',
offscreen: 'ignore',
promotion: 'none',
poll: 'none',
},
} as const satisfies Record<string, SelectorPipelinePolicy | SelectorListPolicy>;
export type SelectorPipelinePolicyName = keyof typeof SELECTOR_PIPELINE_POLICIES;
+26 -3
View File
@@ -25,6 +25,7 @@ const NODE_STAGE_ROWS = [
'findWait',
'wait',
'findAct',
'cropTarget',
] as const;
/**
@@ -149,7 +150,14 @@ test('the occlusion stage decides candidacy: acting rows drop covered nodes, the
);
assert.equal(listed.list?.matchedNodes.length, 1, row);
}
for (const row of ['readText', 'readUnique', 'readAny', 'wait', 'findWait'] as const) {
for (const row of [
'readText',
'readUnique',
'readAny',
'wait',
'findWait',
'cropTarget',
] as const) {
const outcome = await resolveSelectorPipeline(
SELECTOR_PIPELINE_POLICIES[row],
nodes,
@@ -168,7 +176,14 @@ test('the occlusion stage decides refusal: every row that refuses a covered targ
}
// `readList` is absent by construction: a listing row declares no node
// stages, so it has no occlusion verdict to make on a target.
for (const row of ['readText', 'readUnique', 'readAny', 'wait', 'findWait'] as const) {
for (const row of [
'readText',
'readUnique',
'readAny',
'wait',
'findWait',
'cropTarget',
] as const) {
const target = await stagedIndex(row, COVERED_TREE, 1);
assert.equal(target.kind, 'target', row);
assert.equal(target.node.index, 1, row);
@@ -179,7 +194,14 @@ test('the promotion stage retargets only for the rows that declare it', async ()
// Same tree, same node: the row is the whole difference.
assert.equal((await stagedIndex('promotedTarget', PROMOTABLE_TREE, 1)).node.index, 0);
assert.equal((await stagedIndex('findAct', PROMOTABLE_TREE, 1)).node.index, 0);
for (const row of ['resolvedTarget', 'readText', 'readUnique', 'readAny', 'wait'] as const) {
for (const row of [
'resolvedTarget',
'readText',
'readUnique',
'readAny',
'wait',
'cropTarget',
] as const) {
assert.equal((await stagedIndex(row, PROMOTABLE_TREE, 1)).node.index, 1, row);
}
});
@@ -305,6 +327,7 @@ test('the documented per-caller pipelines are the ones declared', () => {
findWait: ['ignore', 'ignore', 'none', 'poll'],
wait: ['ignore', 'ignore', 'none', 'poll'],
findAct: ['refuse', 'ignore', 'hittable-ancestor-below-root', 'no-poll'],
cropTarget: ['ignore', 'ignore', 'none', 'no-poll'],
},
);
});
@@ -158,6 +158,28 @@ test('screenshot digest tolerates a path-only result with no overlay refs', () =
expect(digest).toEqual({ path: '/tmp/s.png', overlayCount: 0, overlayRefs: [] });
});
test('screenshot digest keeps response-level warnings emitted once', () => {
const digest = screenshotView!(
{
path: '/tmp/s.png',
width: 40,
height: 20,
warnings: [
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame',
],
},
'digest',
);
expect(digest).toMatchObject({
path: '/tmp/s.png',
width: 40,
height: 20,
warnings: [
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame',
],
});
});
// A verbose matched node as it appears on the `find`/`get` wire: the semantic
// attributes (kept) plus the geometry/index/process plumbing (the token sink).
const MATCHED_NODE = {
@@ -0,0 +1,152 @@
import {
ANDROID_DEVICE,
ANDROID_EMULATOR,
IOS_DEVICE,
IOS_SIMULATOR,
LINUX_DEVICE,
MACOS_DEVICE,
TVOS_SIMULATOR,
WEB_DESKTOP_DEVICE,
} from '../../__tests__/test-utils/device-fixtures.ts';
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
import type { SessionSurface } from '@agent-device/contracts/session';
import type { DeviceInfo } from '@agent-device/kernel/device';
import { expect, test } from 'vitest';
import {
SCREENSHOT_CROP_TARGET_CELLS,
assertScreenshotCropPolicy,
classifyScreenshotCropTarget,
} from '../screenshot-crop-target.ts';
type CropTargetDevice = Readonly<{
target: (typeof SCREENSHOT_CROP_TARGET_CELLS)[number]['target'];
device: DeviceInfo;
surface: SessionSurface | undefined;
}>;
/** One device per matrix cell: the classifier's whole reachable output, named by its cell. */
const CROP_TARGET_DEVICES: readonly CropTargetDevice[] = [
{ target: 'ios-simulator', device: IOS_SIMULATOR, surface: undefined },
{ target: 'android-emulator', device: ANDROID_EMULATOR, surface: undefined },
{ target: 'android-device', device: ANDROID_DEVICE, surface: undefined },
{ target: 'macos-app-window', device: MACOS_DEVICE, surface: 'app' },
{ target: 'ios-physical', device: IOS_DEVICE, surface: undefined },
{ target: 'macos-helper', device: MACOS_DEVICE, surface: undefined },
{ target: 'web', device: WEB_DESKTOP_DEVICE, surface: undefined },
{ target: 'linux', device: LINUX_DEVICE, surface: undefined },
{ target: 'tvos', device: TVOS_SIMULATOR, surface: undefined },
{
target: 'harmonyos',
device: { platform: 'harmonyos', id: 'hmy-1', name: 'HarmonyOS', kind: 'device' },
surface: undefined,
},
{
target: 'vega',
device: { platform: 'vega', id: 'vega-1', name: 'Vega TV', kind: 'device', target: 'tv' },
surface: undefined,
},
];
test('the classifier and the acceptance matrix agree one-to-one, and the accepted cells are exactly the evidenced ones', () => {
const matrixTargets = SCREENSHOT_CROP_TARGET_CELLS.map((cell) => cell.target);
expect(new Set(matrixTargets).size).toBe(matrixTargets.length);
for (const row of CROP_TARGET_DEVICES) {
expect(classifyScreenshotCropTarget(row.device, row.surface)).toBe(row.target);
}
expect(CROP_TARGET_DEVICES.map((row) => row.target)).toEqual(matrixTargets);
for (const cell of SCREENSHOT_CROP_TARGET_CELLS) {
if (cell.target === 'ios-simulator' || cell.target === 'android-emulator') {
expect(cell.status).toBe('accepted');
} else {
expect(cell).toEqual({
target: cell.target,
status: 'rejected',
rejectionReason: SCREENSHOT_CROP_REASONS.pendingPixelIdentityEvidence,
});
}
}
});
test('an apple device with an unpopulated reserved OS is a typed refusal, not a guess', () => {
const device: DeviceInfo = {
platform: 'apple',
id: 'watch-1',
name: 'Watch',
kind: 'simulator',
appleOs: 'watchos',
};
let refusal: unknown;
try {
classifyScreenshotCropTarget(device, undefined);
} catch (error) {
refusal = error;
}
expect(refusal).toMatchObject({
code: 'UNSUPPORTED_OPERATION',
details: { reason: SCREENSHOT_CROP_REASONS.targetNotAccepted },
});
});
function expectPolicyRefusal(
params: Readonly<{
device: DeviceInfo;
surface: SessionSurface | undefined;
cropOn: string;
overlayRefs: boolean;
fullscreen: boolean;
}>,
expected: Record<string, unknown>,
): void {
try {
assertScreenshotCropPolicy(params);
} catch (error) {
expect(error).toMatchObject(expected);
return;
}
throw new Error('the crop policy must refuse before device work');
}
const POLICY_BASE = { cropOn: 'label="Save"', overlayRefs: false, fullscreen: false } as const;
test('combination refusals are answered before selector validation and the matrix', () => {
expectPolicyRefusal(
{ device: MACOS_DEVICE, surface: 'app', ...POLICY_BASE, overlayRefs: true },
{ code: 'INVALID_ARGS', details: { reason: SCREENSHOT_CROP_REASONS.frameMismatch } },
);
expectPolicyRefusal(
{ device: ANDROID_EMULATOR, surface: undefined, ...POLICY_BASE, fullscreen: true },
{ code: 'INVALID_ARGS', details: { reason: SCREENSHOT_CROP_REASONS.frameMismatch } },
);
});
test('an invalid selector expression is refused with the selector reason', () => {
expectPolicyRefusal(
{
device: ANDROID_EMULATOR,
surface: undefined,
cropOn: 'label="unterminated',
overlayRefs: false,
fullscreen: false,
},
{ code: 'INVALID_ARGS', details: { reason: SCREENSHOT_CROP_REASONS.selectorInvalid } },
);
});
test('a matrix-rejected target is refused with the pending-evidence reason', () => {
expectPolicyRefusal(
{ device: ANDROID_DEVICE, surface: undefined, ...POLICY_BASE },
{
code: 'UNSUPPORTED_OPERATION',
details: {
reason: SCREENSHOT_CROP_REASONS.targetNotAccepted,
rejectionReason: SCREENSHOT_CROP_REASONS.pendingPixelIdentityEvidence,
},
},
);
});
test('an accepted target with a valid selector passes the policy', () => {
expect(() =>
assertScreenshotCropPolicy({ device: ANDROID_EMULATOR, surface: undefined, ...POLICY_BASE }),
).not.toThrow();
});
@@ -0,0 +1,294 @@
import { ANDROID_EMULATOR, IOS_SIMULATOR } from '../../__tests__/test-utils/device-fixtures.ts';
import { makeSession } from '../../__tests__/test-utils/session-factories.ts';
import { readPngSize } from '@agent-device/capture-kit/png-size';
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
import type { SessionSurface } from '@agent-device/contracts/session';
import type { SnapshotResult } from '@agent-device/contracts/snapshot-runtime';
import type { DeviceInfo } from '@agent-device/kernel/device';
import type {
RawSnapshotNode,
Rect,
SnapshotProvenance,
SnapshotQualityVerdict,
} from '@agent-device/kernel/snapshot';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { expect, test, vi } from 'vitest';
import type { SessionState } from '../types.ts';
import { buildScreenshotCropWarnings, cropScreenshotToSelector } from '../screenshot-crop.ts';
import { writeSolidPng } from './screenshot-runtime-fixture.ts';
test('the warning composition is the single owner: partial intersection and only', () => {
expect(buildScreenshotCropWarnings(undefined)).toEqual([]);
expect(buildScreenshotCropWarnings({ partialIntersection: false })).toEqual([]);
const warnings = buildScreenshotCropWarnings({ partialIntersection: true });
expect(warnings).toHaveLength(1);
expect(warnings[0]).toMatch(new RegExp(`^${SCREENSHOT_CROP_REASONS.partialIntersection}: `));
});
const ANDROID_ROOT: RawSnapshotNode = {
index: 0,
depth: 0,
type: 'android.widget.FrameLayout',
rect: { x: 0, y: 0, width: 100, height: 50 },
};
function androidTree(saveRect: Rect): RawSnapshotNode[] {
return [
ANDROID_ROOT,
{
index: 1,
depth: 1,
parentIndex: 0,
type: 'android.widget.Button',
label: 'Save',
rect: saveRect,
},
];
}
const IOS_TREE: RawSnapshotNode[] = [
{
index: 0,
depth: 0,
type: 'XCUIElementTypeApplication',
rect: { x: 0, y: 0, width: 390, height: 844 },
hittable: true,
},
{
index: 1,
depth: 1,
parentIndex: 0,
type: 'XCUIElementTypeButton',
label: 'Save',
rect: { x: 10, y: 10, width: 100, height: 40 },
hittable: true,
},
];
type CropSeamParams = Readonly<{
device: DeviceInfo;
surface?: SessionSurface;
cropOn?: string;
nodes: RawSnapshotNode[];
png: Readonly<{ width: number; height: number }>;
truncated?: boolean;
quality?: SnapshotQualityVerdict;
provenance: SnapshotProvenance;
}>;
type CropSeam = Readonly<{
session: SessionState;
screenshotPath: string;
captureSnapshot: ReturnType<typeof vi.fn>;
run: () => Promise<unknown>;
dispose: () => void;
}>;
function cropSeam(params: CropSeamParams): CropSeam {
const session = makeSession('default', { device: params.device, surface: params.surface });
const screenshotPath = path.join(
os.tmpdir(),
`agent-device-crop-on-${Date.now()}-${Math.random().toString(36).slice(2)}.png`,
);
writeSolidPng(screenshotPath, params.png.width, params.png.height);
const captureSnapshot = vi.fn(async (): Promise<SnapshotResult> => ({
nodes: params.nodes,
...params.provenance,
...(params.truncated === undefined ? {} : { truncated: params.truncated }),
...(params.quality === undefined ? {} : { quality: params.quality }),
}));
return {
session,
screenshotPath,
captureSnapshot,
run: () =>
cropScreenshotToSelector({
device: params.device,
session,
surface: params.surface,
cropOn: params.cropOn ?? 'label="Save"',
screenshotPath,
logPath: path.join(os.tmpdir(), 'agent-device-crop-on-daemon.log'),
dispatchContext: {},
captureSnapshot,
}),
dispose: () => {
fs.rmSync(screenshotPath, { force: true });
},
};
}
test('an android crop runs the fresh full-tree capture once and leaves the session snapshot untouched', async () => {
const seam = cropSeam({
device: ANDROID_EMULATOR,
nodes: androidTree({ x: 10, y: 10, width: 40, height: 20 }),
provenance: { backend: 'android', producer: 'android-uiautomator' },
png: { width: 100, height: 50 },
});
try {
const outcome = await seam.run();
expect(outcome).toEqual({ partialIntersection: false });
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 40, height: 20 });
expect(seam.captureSnapshot).toHaveBeenCalledTimes(1);
const options = seam.captureSnapshot.mock.calls[0]?.[0].options;
expect(options.interactiveOnly).toBe(false);
expect(options.includeRects).toBe(true);
expect(options.surface).toBeUndefined();
expect(options.appBundleId).toBeUndefined();
expect(seam.session.snapshot).toBeUndefined();
expect(seam.session.snapshotGeneration).toBeUndefined();
} finally {
seam.dispose();
}
});
test('an iOS simulator crop projects the points-space frame into the 3x capture', async () => {
const seam = cropSeam({
device: IOS_SIMULATOR,
nodes: IOS_TREE,
provenance: { backend: 'xctest', producer: 'apple-runner' },
png: { width: 1170, height: 2532 },
});
try {
const outcome = await seam.run();
expect(outcome).toEqual({ partialIntersection: false });
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 300, height: 120 });
} finally {
seam.dispose();
}
});
test('a frame that runs past the image is clipped and reported partial', async () => {
const seam = cropSeam({
device: ANDROID_EMULATOR,
nodes: androidTree({ x: 80, y: 10, width: 50, height: 20 }),
provenance: { backend: 'android', producer: 'android-uiautomator' },
png: { width: 100, height: 50 },
});
try {
const outcome = await seam.run();
expect(outcome).toEqual({ partialIntersection: true });
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 20, height: 20 });
} finally {
seam.dispose();
}
});
test('a frame that occupies no image pixel refuses without writing', async () => {
const seam = cropSeam({
device: ANDROID_EMULATOR,
nodes: androidTree({ x: 200, y: 100, width: 10, height: 10 }),
provenance: { backend: 'android', producer: 'android-uiautomator' },
png: { width: 100, height: 50 },
});
try {
await expect(seam.run()).rejects.toMatchObject({
code: 'COMMAND_FAILED',
details: { reason: SCREENSHOT_CROP_REASONS.emptyIntersection },
});
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 100, height: 50 });
} finally {
seam.dispose();
}
});
test('a sparse crop capture is an unreadable refusal, not a missing target', async () => {
const seam = cropSeam({
device: ANDROID_EMULATOR,
nodes: androidTree({ x: 10, y: 10, width: 40, height: 20 }),
provenance: { backend: 'android', producer: 'android-uiautomator' },
png: { width: 100, height: 50 },
quality: { state: 'sparse', backend: 'tree' },
});
try {
await expect(seam.run()).rejects.toMatchObject({
code: 'COMMAND_FAILED',
details: { reason: SCREENSHOT_CROP_REASONS.captureUnreadable },
});
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 100, height: 50 });
} finally {
seam.dispose();
}
});
test('a truncated no-match cannot prove the target absent', async () => {
const seam = cropSeam({
device: ANDROID_EMULATOR,
nodes: [ANDROID_ROOT],
cropOn: 'label="Missing"',
provenance: { backend: 'android', producer: 'android-uiautomator' },
png: { width: 100, height: 50 },
truncated: true,
});
try {
await expect(seam.run()).rejects.toMatchObject({
code: 'COMMAND_FAILED',
details: { reason: SCREENSHOT_CROP_REASONS.captureIncomplete },
});
} finally {
seam.dispose();
}
});
test('a complete no-match is a missing target with the find pointer', async () => {
const seam = cropSeam({
device: ANDROID_EMULATOR,
nodes: [ANDROID_ROOT],
cropOn: 'label="Missing"',
provenance: { backend: 'android', producer: 'android-uiautomator' },
png: { width: 100, height: 50 },
});
try {
await expect(seam.run()).rejects.toMatchObject({
code: 'COMMAND_FAILED',
details: {
reason: SCREENSHOT_CROP_REASONS.targetNotFound,
hint: 'find label="Missing" list',
},
});
} finally {
seam.dispose();
}
});
test('an ambiguous match refuses with the candidates, not a first-wins crop', async () => {
const seam = cropSeam({
device: ANDROID_EMULATOR,
nodes: [
ANDROID_ROOT,
{
index: 1,
depth: 1,
parentIndex: 0,
type: 'android.widget.Button',
label: 'Save',
rect: { x: 5, y: 5, width: 20, height: 10 },
},
{
index: 2,
depth: 1,
parentIndex: 0,
type: 'android.widget.Button',
label: 'Save',
rect: { x: 60, y: 5, width: 20, height: 10 },
},
],
provenance: { backend: 'android', producer: 'android-uiautomator' },
png: { width: 100, height: 50 },
});
try {
await expect(seam.run()).rejects.toMatchObject({
code: 'COMMAND_FAILED',
details: {
reason: SCREENSHOT_CROP_REASONS.targetAmbiguous,
candidates: ['Save', 'Save'],
matches: 2,
},
});
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 100, height: 50 });
} finally {
seam.dispose();
}
});
+141 -1
View File
@@ -2,6 +2,8 @@ import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { expect, test } from 'vitest';
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
import { ANDROID_EMULATOR, IOS_SIMULATOR } from '../../__tests__/test-utils/device-fixtures.ts';
import { makeSession } from '../../__tests__/test-utils/session-factories.ts';
import type { GenericPlatformExecutionParams } from '../request-generic-dispatch.ts';
@@ -28,6 +30,7 @@ function screenshotRequest(overrides: Partial<DaemonRequest> = {}): DaemonReques
function executionParams(
session: SessionState,
req: DaemonRequest,
dispatchContext: GenericPlatformExecutionParams['dispatchContext'] = {},
): GenericPlatformExecutionParams {
return {
session,
@@ -37,10 +40,22 @@ function executionParams(
request: req,
positionals: req.positionals ?? [],
out: req.flags?.out,
dispatchContext: {},
dispatchContext,
};
}
async function executeResult(
resolved: Awaited<ReturnType<typeof resolveScreenshotGenericExecution>>,
session: SessionState,
req: DaemonRequest,
dispatchContext?: GenericPlatformExecutionParams['dispatchContext'],
): Promise<Record<string, unknown>> {
if (!resolved.ok) throw new Error('the screenshot plan must be admitted');
const result = await resolved.execute(executionParams(session, req, dispatchContext));
if (result === undefined) throw new Error('the screenshot leaf returns a result record');
return result;
}
test('admits one capture plan, binds once, and hands the runtime the resolved destination', async () => {
const fixture = screenshotRuntimeFixture();
const session = makeSession('default', { device: ANDROID_EMULATOR });
@@ -196,3 +211,128 @@ test('rejects --pixel-density outside iOS-family simulators before inspecting ow
).rejects.toMatchObject({ code: 'UNSUPPORTED_OPERATION' });
expect(fixture.binds).toHaveLength(0);
});
const CROP_TREE_ROOT = {
index: 0,
depth: 0,
type: 'android.widget.FrameLayout',
rect: { x: 0, y: 0, width: 100, height: 50 },
} satisfies RawSnapshotNode;
const CROP_TREE = [
CROP_TREE_ROOT,
{
index: 1,
depth: 1,
parentIndex: 0,
type: 'android.widget.Button',
label: 'Save',
rect: { x: 10, y: 10, width: 40, height: 20 },
},
];
test('the crop runs after the platform write and before the shared scale', async () => {
const fixture = screenshotRuntimeFixture({
snapshotResult: () => ({
nodes: CROP_TREE,
backend: 'android',
producer: 'android-uiautomator',
}),
});
const session = makeSession('default', { device: ANDROID_EMULATOR });
const outPath = path.join(os.tmpdir(), `agent-device-crop-order-${Date.now()}.png`);
const req = screenshotRequest({
positionals: [outPath],
flags: { screenshotCropOn: 'label="Save"', screenshotScale: 0.5 },
});
const resolved = await resolveScreenshotGenericExecution({
req,
session,
inspectFacts: fixture.inspectFacts,
bindDevice: fixture.bindDevice,
});
const result = await executeResult(resolved, session, req, { screenshotScale: 0.5 });
expect(fixture.captureScreenshot).toHaveBeenCalledTimes(1);
expect(fixture.captureSnapshot).toHaveBeenCalledTimes(1);
// The platform wrote 100x50; the pre-scale box is 40x20, so only a crop-then-scale pipeline
// lands on 20x10 — a scale-then-crop pipeline would halve the image first and the same box
// would overflow it.
expect(result).toMatchObject({ width: 20, height: 10 });
expect(fixture.captureSnapshot.mock.calls[0]?.[0].options).toMatchObject({
interactiveOnly: false,
includeRects: true,
});
});
test('a partial crop surfaces its warning once in the result record and annotates nothing', async () => {
const fixture = screenshotRuntimeFixture({
snapshotResult: () => ({
nodes: [
CROP_TREE_ROOT,
{
index: 1,
depth: 1,
parentIndex: 0,
type: 'android.widget.Button',
label: 'Save',
rect: { x: 80, y: 10, width: 50, height: 20 },
},
],
backend: 'android',
producer: 'android-uiautomator',
}),
});
const session = makeSession('default', { device: ANDROID_EMULATOR });
const outPath = path.join(os.tmpdir(), `agent-device-crop-warning-${Date.now()}.png`);
const req = screenshotRequest({
positionals: [outPath],
flags: { screenshotCropOn: 'label="Save"' },
});
const resolved = await resolveScreenshotGenericExecution({
req,
session,
inspectFacts: fixture.inspectFacts,
bindDevice: fixture.bindDevice,
});
const result = await executeResult(resolved, session, req);
expect(result.warnings).toEqual([
expect.stringMatching(new RegExp(`^${SCREENSHOT_CROP_REASONS.partialIntersection}:`)),
]);
// The crop plan binds the snapshot for the crop alone: no overlay refs, no republished tree.
expect(result).not.toHaveProperty('overlayRefs');
expect(session.snapshot).toBeUndefined();
});
test('the crop plan is refused up front when the target cannot capture a tree', async () => {
const fixture = screenshotRuntimeFixture({
snapshot: { available: false, reason: 'unsupported-platform-leaf' },
});
const session = makeSession('default', { device: ANDROID_EMULATOR });
const resolved = await resolveScreenshotGenericExecution({
req: screenshotRequest({
positionals: ['/tmp/crop.png'],
flags: { screenshotCropOn: 'label="Save"' },
}),
session,
inspectFacts: fixture.inspectFacts,
bindDevice: fixture.bindDevice,
});
expect(resolved.ok).toBe(false);
if (resolved.ok) return;
expect(resolved.response).toMatchObject({
ok: false,
error: {
code: 'UNSUPPORTED_OPERATION',
details: { reason: 'unsupported-platform-leaf' },
hint: 'Re-run screenshot without --crop-on.',
},
});
expect(fixture.binds).toHaveLength(0);
expect(fixture.captureScreenshot).not.toHaveBeenCalled();
});
+1
View File
@@ -76,6 +76,7 @@ function screenshotView(data: DaemonResponseData, level: ResponseLevel): DaemonR
...pickScreenshotDigestMetadata(data),
overlayCount: overlays.length,
overlayRefs,
...(data.warnings !== undefined ? { warnings: data.warnings } : {}),
...(data.artifacts !== undefined ? { artifacts: data.artifacts } : {}),
};
}
+148
View File
@@ -0,0 +1,148 @@
import {
SCREENSHOT_CROP_REASONS,
type ScreenshotCropReason,
} from '@agent-device/contracts/capture';
import type { SessionSurface } from '@agent-device/contracts/session';
import { resolveDeviceAppleOs, type DeviceInfo } from '@agent-device/kernel/device';
import { AppError } from '@agent-device/kernel/errors';
import { validateSelectorExpression } from '@agent-device/selectors';
/**
* The `--crop-on` target policy: which device capture frame the crop is accepted on, and the
* argument policy answered before any device work. The crop leaf (screenshot-crop.ts) runs the
* crop; this module decides whether it may run at all.
*/
type CropTarget =
| 'ios-simulator'
| 'android-emulator'
| 'android-device'
| 'macos-app-window'
| 'ios-physical'
| 'macos-helper'
| 'web'
| 'linux'
| 'tvos'
| 'harmonyos'
| 'vega';
type CropTargetCell =
| Readonly<{ target: CropTarget; status: 'accepted' }>
| Readonly<{
target: CropTarget;
status: 'rejected';
rejectionReason: ScreenshotCropReason;
}>;
/**
* The machine-readable support policy. A cell is accepted only once the live pixel-identity
* cross-check against an independent capture has been collected; until then the target is
* rejected with the pending-evidence reason, and the evidence itself lives in the completeness
* gate, the geometry fixtures, and the PR history never in this table.
*/
const PENDING = SCREENSHOT_CROP_REASONS.pendingPixelIdentityEvidence;
export const SCREENSHOT_CROP_TARGET_CELLS: readonly CropTargetCell[] = [
{ target: 'ios-simulator', status: 'accepted' },
{ target: 'android-emulator', status: 'accepted' },
{ target: 'android-device', status: 'rejected', rejectionReason: PENDING },
{ target: 'macos-app-window', status: 'rejected', rejectionReason: PENDING },
{ target: 'ios-physical', status: 'rejected', rejectionReason: PENDING },
{ target: 'macos-helper', status: 'rejected', rejectionReason: PENDING },
{ target: 'web', status: 'rejected', rejectionReason: PENDING },
{ target: 'linux', status: 'rejected', rejectionReason: PENDING },
{ target: 'tvos', status: 'rejected', rejectionReason: PENDING },
{ target: 'harmonyos', status: 'rejected', rejectionReason: PENDING },
{ target: 'vega', status: 'rejected', rejectionReason: PENDING },
];
/** The one target cell a device's capture frame falls into. */
export function classifyScreenshotCropTarget(
device: DeviceInfo,
surface: SessionSurface | undefined,
): CropTarget {
switch (device.platform) {
case 'apple':
return classifyAppleCropTarget(device, surface);
case 'android':
return device.kind === 'device' ? 'android-device' : 'android-emulator';
case 'harmonyos':
return 'harmonyos';
case 'vega':
return 'vega';
case 'linux':
return 'linux';
case 'web':
return 'web';
}
}
function classifyAppleCropTarget(
device: DeviceInfo,
surface: SessionSurface | undefined,
): CropTarget {
const appleOs = resolveDeviceAppleOs(device);
switch (appleOs) {
case 'ios':
case 'ipados':
return device.kind === 'device' ? 'ios-physical' : 'ios-simulator';
case 'tvos':
return 'tvos';
case 'macos':
return classifyMacOsCropTarget(surface);
case 'watchos':
case 'visionos':
// Stored records carry the reserved OSes although discovery never populates them; a
// session device on one has no acceptance cell to fall into, so it is a typed refusal,
// not a guess.
throw cropRefusal(appleOs);
}
}
function classifyMacOsCropTarget(surface: SessionSurface | undefined): CropTarget {
return surface === 'app' || surface === 'frontmost-app' ? 'macos-app-window' : 'macos-helper';
}
function cropRefusal(target: string, rejectionReason?: ScreenshotCropReason): AppError {
return new AppError(
'UNSUPPORTED_OPERATION',
`screenshot --crop-on is not accepted on ${target} targets`,
{
reason: SCREENSHOT_CROP_REASONS.targetNotAccepted,
...(rejectionReason ? { rejectionReason } : {}),
},
);
}
/**
* The crop argument policy, answered before any device work: combination refusals, the
* selector expression, and the acceptance matrix.
*/
export function assertScreenshotCropPolicy(params: {
device: DeviceInfo;
surface: SessionSurface | undefined;
cropOn: string;
overlayRefs: boolean;
fullscreen: boolean;
}): void {
if (params.overlayRefs || params.fullscreen) {
throw new AppError(
'INVALID_ARGS',
'--crop-on cannot be combined with --overlay-refs or --fullscreen: both move the captured frame away from the snapshot viewport the crop is measured against',
{ reason: SCREENSHOT_CROP_REASONS.frameMismatch },
);
}
try {
validateSelectorExpression(params.cropOn);
} catch {
throw new AppError(
'INVALID_ARGS',
'screenshot --crop-on requires a valid selector expression',
{ reason: SCREENSHOT_CROP_REASONS.selectorInvalid },
);
}
const target = classifyScreenshotCropTarget(params.device, params.surface);
const cell = SCREENSHOT_CROP_TARGET_CELLS.find((candidate) => candidate.target === target);
if (!cell || cell.status === 'rejected') {
throw cropRefusal(target, cell?.status === 'rejected' ? cell.rejectionReason : undefined);
}
}
+141
View File
@@ -0,0 +1,141 @@
import type { CommandFlags } from '@agent-device/contracts/command';
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
import type { SessionSurface } from '@agent-device/contracts/session';
import { publicPlatformString, type DeviceInfo } from '@agent-device/kernel/device';
import { AppError } from '@agent-device/kernel/errors';
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
import { readPngSize } from '@agent-device/capture-kit/png-size';
import { cropPngFile } from '@agent-device/capture-kit/png-crop';
import {
intersectScreenshotRect,
projectSnapshotRectToScreenshot,
resolveScreenshotRectSpace,
resolveSnapshotBounds,
} from '@agent-device/capture-kit/snapshot-rect-projection';
import { buildSnapshotState } from '../core/snapshot-state.ts';
import { SELECTOR_PIPELINE_POLICIES } from '../core/selector-pipeline-policy.ts';
import { resolveSelectorPipeline } from '../core/selector-pipeline.ts';
import type { DaemonCommandContext } from './context.ts';
import { captureSnapshotData } from './snapshot-capture.ts';
import { runtimeExecutionFromContext } from './snapshot-runtime-capture-input.ts';
import type { BoundScreenshotRuntime } from './screenshot-runtime-binding.ts';
import type { SessionState } from './types.ts';
export type ScreenshotCropOutcome = Readonly<{ partialIntersection: boolean }>;
const CROP_PARTIAL_INTERSECTION_WARNING = `${SCREENSHOT_CROP_REASONS.partialIntersection}: the selector frame extends past the captured image; the crop was clipped to the image frame`;
/** The single warning-composition owner: append-only, typed reason, stable text. */
export function buildScreenshotCropWarnings(outcome: ScreenshotCropOutcome | undefined): string[] {
return outcome?.partialIntersection ? [CROP_PARTIAL_INTERSECTION_WARNING] : [];
}
/**
* Crops `screenshotPath` to the frame of the selector resolved against a fresh full-tree
* snapshot taken on the same screen, through the same admitted binding as the capture. The crop
* snapshot is request-scoped it never becomes the session snapshot.
*/
export async function cropScreenshotToSelector(params: {
device: DeviceInfo;
session: SessionState;
surface: SessionSurface | undefined;
cropOn: string;
screenshotPath: string;
logPath: string;
dispatchContext: DaemonCommandContext;
captureSnapshot: NonNullable<BoundScreenshotRuntime['captureSnapshot']>;
}): Promise<ScreenshotCropOutcome> {
const { device, session, surface, cropOn, screenshotPath, logPath, dispatchContext } = params;
const flags = { snapshotInteractiveOnly: false } satisfies CommandFlags;
const snapshotData = await captureSnapshotData({
device,
session,
flags,
logPath,
snapshotScope: undefined,
captureData: async () =>
await params.captureSnapshot({
options: {
appBundleId: dispatchContext.appBundleId,
interactiveOnly: false,
includeRects: true,
surface,
},
execution: runtimeExecutionFromContext(dispatchContext),
}),
});
const snapshot = buildSnapshotState(snapshotData, flags);
if (snapshot.snapshotQuality?.state === 'sparse') {
throw new AppError(
'COMMAND_FAILED',
'the snapshot taken for --crop-on was sparse and cannot be read as the screen',
{ reason: SCREENSHOT_CROP_REASONS.captureUnreadable },
);
}
const node = await resolveCropTargetNode(snapshot.nodes, cropOn, snapshot.truncated, device);
const image = await readPngSize(screenshotPath);
const projected = projectSnapshotRectToScreenshot(
resolveScreenshotRectSpace(snapshot.backend),
resolveSnapshotBounds(snapshot.nodes),
node.rect!,
image.width,
image.height,
);
const box = intersectScreenshotRect(projected, image.width, image.height);
if (box === null) {
throw new AppError(
'COMMAND_FAILED',
`the frame resolved by --crop-on occupies no pixel of the capture`,
{ reason: SCREENSHOT_CROP_REASONS.emptyIntersection },
);
}
const partialIntersection =
box.x !== projected.x ||
box.y !== projected.y ||
box.width < projected.width ||
box.height < projected.height;
await cropPngFile(screenshotPath, box);
return { partialIntersection };
}
/** Resolve the selector to exactly one framed node, or fail with a typed crop reason. */
async function resolveCropTargetNode(
nodes: SnapshotNode[],
cropOn: string,
truncated: boolean | undefined,
device: DeviceInfo,
) {
const outcome = await resolveSelectorPipeline(
SELECTOR_PIPELINE_POLICIES.cropTarget,
nodes,
cropOn,
{ platform: publicPlatformString(device) },
);
if (outcome.kind === 'ambiguous') {
throw new AppError(
'COMMAND_FAILED',
`--crop-on matched ${outcome.matchedNodes.length} nodes and must resolve to one`,
{
reason: SCREENSHOT_CROP_REASONS.targetAmbiguous,
candidates: outcome.matchedNodes.map((node) => node.label ?? node.ref),
matches: outcome.matchedNodes.length,
},
);
}
if (outcome.kind === 'none') {
if (truncated) {
throw new AppError(
'COMMAND_FAILED',
'the snapshot taken for --crop-on was truncated, so a missing match is not proven',
{ reason: SCREENSHOT_CROP_REASONS.captureIncomplete },
);
}
throw new AppError('COMMAND_FAILED', `--crop-on matched no node: ${cropOn}`, {
reason: SCREENSHOT_CROP_REASONS.targetNotFound,
hint: `find ${cropOn} list`,
});
}
return outcome.node;
}
+7 -74
View File
@@ -7,9 +7,14 @@ import {
type SnapshotState,
} from '@agent-device/kernel/snapshot';
import { decodePngAsync, encodePngAsync } from '@agent-device/capture-kit/png-worker-client';
import {
projectSnapshotRectToScreenshot,
resolveSnapshotBounds,
} from '@agent-device/capture-kit/snapshot-rect-projection';
import { analyzeReactNativeOverlay } from '../core/react-native-overlay.ts';
import {
findNearestAncestor,
isMeaningfulSignal,
isViewportRootNode,
normalizeType,
} from '@agent-device/contracts/snapshot';
@@ -259,69 +264,14 @@ function projectRectToScreenshot(
screenshotWidth: number,
screenshotHeight: number,
): Rect {
if (snapshot.backend === 'android') {
return clampRect(roundRect(rect), screenshotWidth, screenshotHeight);
}
if (!bounds) {
return clampRect(roundRect(rect), screenshotWidth, screenshotHeight);
}
const scaleX = screenshotWidth / bounds.width;
const scaleY = screenshotHeight / bounds.height;
const space = snapshot.backend === 'android' ? 'device-pixels' : 'viewport-points';
return clampRect(
{
x: Math.round((rect.x - bounds.x) * scaleX),
y: Math.round((rect.y - bounds.y) * scaleY),
width: Math.max(1, Math.round(rect.width * scaleX)),
height: Math.max(1, Math.round(rect.height * scaleY)),
},
projectSnapshotRectToScreenshot(space, bounds, rect, screenshotWidth, screenshotHeight),
screenshotWidth,
screenshotHeight,
);
}
function resolveSnapshotBounds(nodes: SnapshotState['nodes']): Rect | null {
let viewport: Rect | null = null;
for (const node of nodes) {
if (!isViewportRootNode(node) || !hasPositiveRect(node.rect)) continue;
if (!viewport || rectArea(node.rect) > rectArea(viewport)) {
viewport = node.rect;
}
}
if (viewport) return viewport;
return measureSnapshotBounds(
nodes.filter((node) => hasPositiveRect(node.rect) && !isSnapshotBoundsOutlier(node)),
);
}
function measureSnapshotBounds(nodes: Array<Pick<SnapshotNode, 'rect'>>): Rect | null {
let minX = Number.POSITIVE_INFINITY;
let minY = Number.POSITIVE_INFINITY;
let maxRight = Number.NEGATIVE_INFINITY;
let maxBottom = Number.NEGATIVE_INFINITY;
for (const node of nodes) {
if (!node.rect || !hasPositiveRect(node.rect)) continue;
minX = Math.min(minX, node.rect.x);
minY = Math.min(minY, node.rect.y);
maxRight = Math.max(maxRight, node.rect.x + node.rect.width);
maxBottom = Math.max(maxBottom, node.rect.y + node.rect.height);
}
if (!Number.isFinite(minX) || !Number.isFinite(minY) || maxRight <= minX || maxBottom <= minY) {
return null;
}
return {
x: minX,
y: minY,
width: maxRight - minX,
height: maxBottom - minY,
};
}
function isSnapshotBoundsOutlier(node: SnapshotNode): boolean {
const normalizedType = normalizeType(node.type ?? '');
return normalizedType === 'image' && !isMeaningfulSignal(node.label);
}
function hasActionableRole(node: SnapshotNode): boolean {
const roleText = [node.type, node.role, node.subrole]
.map((value) => normalizeType(value ?? ''))
@@ -347,14 +297,6 @@ function isUsableOverlayTarget(node: SnapshotNode | null): node is SnapshotNode
return Boolean(node?.rect && hasPositiveRect(node.rect) && !isViewportRootNode(node));
}
function isMeaningfulSignal(value: string | undefined): boolean {
if (typeof value !== 'string') return false;
const trimmed = value.trim();
if (!trimmed) return false;
if (/^(true|false)$/i.test(trimmed)) return false;
return true;
}
function isOverlaySignal(value: string | undefined): boolean {
if (!isMeaningfulSignal(value)) return false;
return !isGenericOverlayLabel(value);
@@ -433,15 +375,6 @@ function compareNumericRefs(left: string, right: string): number {
return leftValue - rightValue;
}
function roundRect(rect: Rect): Rect {
return {
x: Math.round(rect.x),
y: Math.round(rect.y),
width: Math.round(rect.width),
height: Math.round(rect.height),
};
}
function clampRect(rect: Rect, width: number, height: number): Rect {
const x = clamp(rect.x, 0, Math.max(0, width - 1));
const y = clamp(rect.y, 0, Math.max(0, height - 1));
+26 -10
View File
@@ -30,9 +30,9 @@ export type ScreenshotRuntimeBindings = Readonly<{
}>;
/**
* One request's admitted capture authority. `captureSnapshot` is present exactly when the
* overlay-refs plan was admitted, so a caller cannot annotate a capture it never declared and
* both operations come from the same single binding.
* One request's admitted capture authority. `captureSnapshot` is present exactly when a
* tree-requiring plan (overlay-refs or crop-on) was admitted, so a caller cannot post-process a
* capture it never declared a snapshot for and both operations come from the same single binding.
*/
export type BoundScreenshotRuntime = Readonly<{
captureScreenshot(input: CaptureScreenshotInput): Promise<void>;
@@ -45,9 +45,13 @@ export type ResolvedScreenshotRuntime =
/** Resolves one plan, inspects its owner facts once, then binds once on the admitted device. */
export async function resolveBoundScreenshotRuntime(
params: Readonly<{ device: DeviceInfo; overlayRefs: boolean }> & ScreenshotRuntimeBindings,
params: Readonly<{ device: DeviceInfo; overlayRefs: boolean; cropOn?: string }> &
ScreenshotRuntimeBindings,
): Promise<ResolvedScreenshotRuntime> {
const plan = resolveScreenshotRuntimePlan({ overlayRefs: params.overlayRefs });
const plan = resolveScreenshotRuntimePlan({
overlayRefs: params.overlayRefs,
cropOn: params.cropOn !== undefined,
});
const admission = await admitRuntimePlan({
device: params.device,
plan,
@@ -56,7 +60,7 @@ export async function resolveBoundScreenshotRuntime(
if (!admission.admitted) {
return {
ok: false,
response: screenshotPlanUnavailableResponse(admission.operation, admission.fact),
response: screenshotPlanUnavailableResponse(plan.kind, admission.operation, admission.fact),
};
}
return { ok: true, runtime: await bindScreenshotRuntime(params.bindDevice, admission) };
@@ -78,11 +82,12 @@ async function bindScreenshotRuntime(
const runtime = await bind(device, plan.use);
return Object.freeze({ captureScreenshot: selectScreenshotCapture(runtime) });
}
case 'capture-with-overlay-refs': {
case 'capture-with-overlay-refs':
case 'capture-with-crop-on': {
const runtime = await bind(device, plan.use);
return Object.freeze({
captureScreenshot: selectScreenshotCapture(runtime),
captureSnapshot: selectOverlaySnapshot(runtime),
captureSnapshot: selectSnapshotCapture(runtime),
});
}
}
@@ -97,20 +102,31 @@ function selectScreenshotCapture(runtime: BoundScreenshotOperation<'captureScree
return async (input: CaptureScreenshotInput) => await runtime.operations.captureScreenshot(input);
}
/** The overlay plan's tree read, from the same binding as its capture. */
function selectOverlaySnapshot(
/** The tree-requiring plans' snapshot read, from the same binding as the capture. */
function selectSnapshotCapture(
runtime: Readonly<{ operations: Readonly<Pick<SnapshotRuntimeOperations, 'captureSnapshot'>> }>,
) {
return async (input: CaptureSnapshotInput) => await runtime.operations.captureSnapshot(input);
}
function screenshotPlanUnavailableResponse(
kind: ScreenshotRuntimePlan['kind'],
operation: ScreenshotRuntimePlan['use']['required'][number],
fact: RuntimeOperationFact,
): DaemonResponse {
if (operation === 'captureScreenshot') {
return unavailableRuntimeOperationResponse('screenshot', fact)!;
}
if (kind === 'capture-with-crop-on') {
return errorResponse(
'UNSUPPORTED_OPERATION',
'--crop-on crops the capture to a selector frame using a snapshot taken on the same screen, which this target cannot capture.',
{ reason: fact.available ? undefined : fact.reason },
{
hint: (fact.available ? undefined : fact.hint) ?? 'Re-run screenshot without --crop-on.',
},
);
}
return errorResponse(
'UNSUPPORTED_OPERATION',
'--overlay-refs annotates a capture with the refs of a snapshot taken on the same screen, which this target cannot capture.',
+111 -15
View File
@@ -27,6 +27,8 @@ import type {
ResolvedGenericExecution,
} from './request-generic-dispatch.ts';
import { createDaemonRuntimeSessionStore } from './runtime-session.ts';
import { assertScreenshotCropPolicy } from './screenshot-crop-target.ts';
import { buildScreenshotCropWarnings, cropScreenshotToSelector } from './screenshot-crop.ts';
import { annotateScreenshotWithRefs } from './screenshot-overlay.ts';
import {
resolveBoundScreenshotRuntime,
@@ -55,15 +57,30 @@ export async function resolveScreenshotGenericExecution(
assertSupportedScreenshotPixelDensity(session.device, req.flags?.screenshotPixelDensity);
const request = readScreenshotRequest(req);
const cropOn =
typeof req.flags?.screenshotCropOn === 'string' && req.flags.screenshotCropOn.length > 0
? req.flags.screenshotCropOn
: undefined;
if (cropOn !== undefined) {
assertScreenshotCropPolicy({
device: session.device,
surface: session.surface,
cropOn,
overlayRefs: req.flags?.overlayRefs === true,
fullscreen: req.flags?.screenshotFullscreen === true,
});
}
const resolved = await resolveBoundScreenshotRuntime({
device: session.device,
overlayRefs: req.flags?.overlayRefs === true,
cropOn,
inspectFacts: params.inspectFacts,
bindDevice: params.bindDevice,
});
if (!resolved.ok) return resolved;
const runtime = resolved.runtime;
const cropRun: ScreenshotCropRun = { warnings: [] };
return {
ok: true,
recorded: request.recorded,
@@ -76,6 +93,8 @@ export async function resolveScreenshotGenericExecution(
flags: execution.request.flags,
outPath: request.outPath,
runtime,
cropOn,
cropRun,
}),
};
}
@@ -92,6 +111,7 @@ export async function captureScreenshotArtifact(
outPath?: string;
dispatchContext: DaemonCommandContext;
captureScreenshot: BoundScreenshotRuntime['captureScreenshot'];
crop?: ScreenshotCropBinding;
}>,
): Promise<CapturedScreenshot> {
const { session, sessionName, outPath, dispatchContext } = params;
@@ -122,7 +142,18 @@ export async function captureScreenshotArtifact(
* `commands/`: the daemon sits below the command surface (R2), and this adapter's artifact
* publisher emits no descriptors, so the destination and its message are the whole result.
*/
type CapturedScreenshot = Readonly<{ path: string; message?: string }>;
type CapturedScreenshot = Readonly<{ path: string; message?: string; warnings?: string[] }>;
/** One request's crop state: the backend closure appends, the result record reads. */
type ScreenshotCropRun = { warnings: string[] };
/** The crop orchestration the backend closure runs after the platform write, before scale. */
type ScreenshotCropBinding = Readonly<{
cropOn: string;
captureSnapshot: NonNullable<BoundScreenshotRuntime['captureSnapshot']>;
run: ScreenshotCropRun;
logPath: string;
}>;
/**
* Runner metadata the capture needs; cancellation comes from the request binding, not from here.
@@ -146,30 +177,37 @@ async function executeScreenshot(
flags: CommandFlags | undefined;
outPath: string | undefined;
runtime: BoundScreenshotRuntime;
cropOn?: string;
cropRun?: ScreenshotCropRun;
}>,
): Promise<Record<string, unknown>> {
const { session, runtime, flags } = params;
const crop = buildScreenshotCropBinding({
cropOn: params.cropOn,
captureSnapshot: runtime.captureSnapshot,
cropRun: params.cropRun,
logPath: params.logPath,
});
const captured = await captureScreenshotArtifact({
session,
sessionName: params.sessionName,
outPath: params.outPath,
dispatchContext: params.dispatchContext,
captureScreenshot: runtime.captureScreenshot,
...(crop ? { crop } : {}),
});
const captureSnapshot = runtime.captureSnapshot;
const warnings = [...(captured.warnings ?? []), ...(crop?.run?.warnings ?? [])];
return {
...captured,
...(captureSnapshot
? {
overlayRefs: await annotateScreenshotWithSessionRefs({
session,
logPath: params.logPath,
screenshotPath: captured.path,
dispatchContext: params.dispatchContext,
captureSnapshot,
}),
}
: {}),
...(warnings.length > 0 ? { warnings } : {}),
...(await screenshotOverlayRefsField({
session,
cropOn: params.cropOn,
captureSnapshot: runtime.captureSnapshot,
screenshotPath: captured.path,
logPath: params.logPath,
dispatchContext: params.dispatchContext,
})),
...(await readScreenshotResultMetadata({
device: session.device,
path: captured.path,
@@ -179,10 +217,54 @@ async function executeScreenshot(
};
}
/** Present exactly when the admitted plan carries a crop and the snapshot binding to serve it. */
function buildScreenshotCropBinding(
params: Readonly<{
cropOn: string | undefined;
captureSnapshot: BoundScreenshotRuntime['captureSnapshot'];
cropRun: ScreenshotCropRun | undefined;
logPath: string;
}>,
): ScreenshotCropBinding | undefined {
const { cropOn, captureSnapshot, cropRun } = params;
if (cropOn === undefined || captureSnapshot === undefined || cropRun === undefined) {
return undefined;
}
return { cropOn, captureSnapshot, run: cropRun, logPath: params.logPath };
}
/**
* `--overlay-refs` republishes the annotated tree as the session's snapshot, so the refs it drew
* are the refs a following interaction resolves. The tree comes from the same admitted binding as
* the capture the plan required both operations before either ran.
* the capture the plan required both operations before either ran. The crop plan binds the
* snapshot for the crop alone, so a crop request never annotates or republishes.
*/
async function screenshotOverlayRefsField(
params: Readonly<{
session: SessionState;
cropOn: string | undefined;
captureSnapshot: BoundScreenshotRuntime['captureSnapshot'];
screenshotPath: string;
logPath: string;
dispatchContext: DaemonCommandContext;
}>,
): Promise<Readonly<Record<string, unknown>>> {
const { session, cropOn, captureSnapshot } = params;
if (cropOn !== undefined || captureSnapshot === undefined) return {};
return {
overlayRefs: await annotateScreenshotWithSessionRefs({
session,
logPath: params.logPath,
screenshotPath: params.screenshotPath,
dispatchContext: params.dispatchContext,
captureSnapshot,
}),
};
}
/**
* The overlay-refs tree capture: interactive-only, through the same admitted binding as the
* screenshot, stored as the session snapshot so the burned-in refs stay the authorized frame.
*/
async function annotateScreenshotWithSessionRefs(
params: Readonly<{
@@ -247,9 +329,10 @@ function createBoundScreenshotBackend(
session: SessionState;
dispatchContext: DaemonCommandContext;
captureScreenshot: BoundScreenshotRuntime['captureScreenshot'];
crop?: ScreenshotCropBinding;
}>,
): AgentDeviceBackend {
const { session, dispatchContext, captureScreenshot } = params;
const { session, dispatchContext, captureScreenshot, crop } = params;
return {
platform: publicPlatformString(session.device),
captureScreenshot: async (_context, outPath, options) => {
@@ -275,6 +358,19 @@ function createBoundScreenshotBackend(
},
execution: screenshotExecutionFromContext(dispatchContext),
});
if (crop) {
const outcome = await cropScreenshotToSelector({
device: session.device,
session,
surface: session.surface,
cropOn: crop.cropOn,
screenshotPath: outPath,
logPath: crop.logPath,
dispatchContext,
captureSnapshot: crop.captureSnapshot,
});
crop.run.warnings.push(...buildScreenshotCropWarnings(outcome));
}
},
};
}
+2
View File
@@ -916,6 +916,7 @@ agent-device screenshot # Auto filename
agent-device screenshot page.png # Explicit screenshot path
agent-device screenshot page.png --scale 0.3 # Resize both dimensions to 30% for agent context
agent-device screenshot page.png --overlay-refs # Draw current @eN refs and target rectangles onto the PNG
agent-device screenshot page.png --crop-on 'label="Save"' # Crop the capture to the frame the selector resolves on the same screen
agent-device screenshot baseline.png --normalize-status-bar # Normalize iOS simulator chrome for reusable diff baselines
agent-device screenshot page.png --platform web --fullscreen # On web, --fullscreen/--full/-f captures the entire document
agent-device viewport 1280 900 --platform web # Resize the active web viewport for fixed-layout or 100vh apps
@@ -938,6 +939,7 @@ agent-device record stop # Stop active recording
- Set `AGENT_DEVICE_SCREENSHOT_SCALE=0.3` (or `screenshotScale` in config) as a token-conscious screenshot default for agent workflows. An explicit `--scale` overrides it.
- Keep the scale default unset, or use `--scale 1`, when full-resolution screenshots are required for reusable pixel-diff baselines.
- `screenshot --overlay-refs` captures a fresh full snapshot and burns visible `@eN` refs plus their target rectangles into the saved PNG.
- `screenshot --crop-on <selector>` captures a fresh full snapshot of the same screen and crops the saved PNG to the frame the selector resolves to. The selector must resolve to exactly one framed node; the result carries a `warnings` entry when the frame is clipped to the image. Currently accepted on iOS simulators and Android emulators — every other target is refused before any device work, and the flag cannot be combined with `--overlay-refs` or `--fullscreen` because both move the captured frame away from the snapshot viewport the crop is measured against.
- `screenshot --normalize-status-bar` temporarily normalizes iOS simulator status-bar chrome for deterministic screenshot baselines; ordinary screenshots leave the simulator's current chrome visible.
- `screenshot --scale <factor> --overlay-refs` writes a smaller image and draws refs for that final image size; avoid very small scales when text, icons, or labels need to remain readable.
- `diff screenshot` compares the current live screenshot to `--baseline`, or compares `--baseline` to an optional saved `current.png` path without requiring an active session. Its text output reports ranked changed regions with screen-space rectangles, changed-pixel counts, and each region's share of the diff; JSON also includes normalized rectangles. The earlier best-effort `ocr` and `nonTextDeltas` analyzers are retired; their optional result fields remain for source compatibility but are no longer emitted, so use the baseline/current images and diff artifact with vision for qualitative interpretation. It writes a diff PNG with a light grayscale current-screen context, red-tinted changed pixels, and outlined changed regions when `--out` is provided. Live iOS simulator diffs normalize status-bar chrome by default; use `screenshot --normalize-status-bar` when capturing reusable baselines.