mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
feat(screenshot): add --crop-on to crop captures to a selector frame (#2276)
* feat(screenshot): add crop-on geometry core and cropTarget selector rows
* feat(screenshot): declare crop-on flag, script round-trip, and snapshot runtime plan
* feat(screenshot): run the crop leaf after the platform write and before scale
* feat(screenshot): expose --crop-on in the CLI and surface crop warnings
* chore(gates): declare crop-on capture-kit subpaths and scope the crop scenario exemption
* refactor(screenshot): split crop target/policy module and trim redundant coverage
Address review comments at 570da2c417:
- Split the 328-line screenshot-crop.ts leaf: the target acceptance matrix,
classifier, and pre-device argument policy move to screenshot-crop-target.ts,
so both implementation modules meet the 300-line target.
- Reuse kernel isPositiveFiniteRect/rectArea in the rect-projection module
instead of redefining them locally.
- Drop the crop-on CLI forwarding case (redundant with screenshot-options
flag-mapping coverage + the generic dispatcher) and the transport-based
warnings case, replacing the latter with a focused screenshot-result unit
test. This also returns the two legacy aggregate test files to their
merge-base length for the test-file size ratchet.
* refactor(screenshot): extract macOS crop-target decision to keep classifier under the complexity budget
classifyAppleCropTarget inlined the macOS surface decision, pushing its
cyclomatic complexity to the fallow threshold. Move it back out to a
small helper so the target classifier stays within budget.
* refactor(screenshot): dedupe the meaningful-signal predicate and polish png-crop
- Hoist isMeaningfulSignal into @agent-device/contracts/snapshot (next to
normalizeType/isMeaningfulLabel) so the ref overlay and the crop
rect-projection share one copy instead of each carrying an identical
private predicate. Behavior is unchanged.
- png-crop: isCropBox was a no-op 'box is Rect' predicate (input already
Rect) — make it a plain boolean, and tighten the doc to the contract.
* refactor(screenshot): drop the dead crop outcome flag and cover the projection seams
- ScreenshotCropOutcome.cropped was a constant true that no caller read;
the crop either returns (success) or throws, so the outcome reduces to
the partialIntersection observation.
- resolveScreenshotRectSpace and resolveSnapshotBounds were the only
projection exports without coverage: pin the accepted-backend map, the
unaccepted-backend typed refusal, and the viewport-root / union / empty
bounds branches.
This commit is contained in:
committed by
GitHub
parent
658f822c40
commit
172ee149cf
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"name": "iOS simulator 1x capture: viewport-space rect projects 1:1 against a zero-origin bounds",
|
||||
"space": "viewport-points",
|
||||
"bounds": { "x": 0, "y": 0, "width": 402, "height": 874 },
|
||||
"rect": { "x": 16, "y": 293.33, "width": 370, "height": 52 },
|
||||
"image": { "width": 402, "height": 874 },
|
||||
"expectedProjection": { "x": 16, "y": 293, "width": 370, "height": 52 },
|
||||
"expectedIntersection": { "x": 16, "y": 293, "width": 370, "height": 52 }
|
||||
},
|
||||
{
|
||||
"name": "iOS simulator 3x capture: the same points rect scales by imgW/bounds.width and rounds",
|
||||
"space": "viewport-points",
|
||||
"bounds": { "x": 0, "y": 0, "width": 402, "height": 874 },
|
||||
"rect": { "x": 16, "y": 293.33, "width": 370, "height": 52 },
|
||||
"image": { "width": 1206, "height": 2622 },
|
||||
"expectedProjection": { "x": 48, "y": 880, "width": 1110, "height": 156 },
|
||||
"expectedIntersection": { "x": 48, "y": 880, "width": 1110, "height": 156 }
|
||||
},
|
||||
{
|
||||
"name": "Android: device-pixel rects project 1:1 with no bounds lookup",
|
||||
"space": "device-pixels",
|
||||
"bounds": null,
|
||||
"rect": { "x": 210, "y": 678, "width": 436, "height": 71 },
|
||||
"image": { "width": 1080, "height": 2400 },
|
||||
"expectedProjection": { "x": 210, "y": 678, "width": 436, "height": 71 },
|
||||
"expectedIntersection": { "x": 210, "y": 678, "width": 436, "height": 71 }
|
||||
},
|
||||
{
|
||||
"name": "macOS app window: non-zero-origin bounds shift the rect before the 2x scale",
|
||||
"space": "viewport-points",
|
||||
"bounds": { "x": 898, "y": 74, "width": 586, "height": 488 },
|
||||
"rect": { "x": 906, "y": 114, "width": 20, "height": 20 },
|
||||
"image": { "width": 1172, "height": 976 },
|
||||
"expectedProjection": { "x": 16, "y": 80, "width": 40, "height": 40 },
|
||||
"expectedIntersection": { "x": 16, "y": 80, "width": 40, "height": 40 }
|
||||
},
|
||||
{
|
||||
"name": "viewport-points without bounds degrades to a rounded 1:1 rect instead of dividing by zero",
|
||||
"space": "viewport-points",
|
||||
"bounds": null,
|
||||
"rect": { "x": 16.6, "y": 293.33, "width": 370, "height": 52 },
|
||||
"image": { "width": 402, "height": 874 },
|
||||
"expectedProjection": { "x": 17, "y": 293, "width": 370, "height": 52 },
|
||||
"expectedIntersection": { "x": 17, "y": 293, "width": 370, "height": 52 }
|
||||
},
|
||||
{
|
||||
"name": "scrolled-out row above the image floor intersects nothing",
|
||||
"space": "viewport-points",
|
||||
"bounds": { "x": 0, "y": 0, "width": 402, "height": 874 },
|
||||
"rect": { "x": 16, "y": 900, "width": 370, "height": 52 },
|
||||
"image": { "width": 402, "height": 874 },
|
||||
"expectedProjection": { "x": 16, "y": 900, "width": 370, "height": 52 },
|
||||
"expectedIntersection": null
|
||||
},
|
||||
{
|
||||
"name": "row grazing the bottom edge clips to the image and stays partial",
|
||||
"space": "device-pixels",
|
||||
"bounds": null,
|
||||
"rect": { "x": 210, "y": 2360, "width": 436, "height": 111 },
|
||||
"image": { "width": 1080, "height": 2400 },
|
||||
"expectedProjection": { "x": 210, "y": 2360, "width": 436, "height": 111 },
|
||||
"expectedIntersection": { "x": 210, "y": 2360, "width": 436, "height": 40 }
|
||||
},
|
||||
{
|
||||
"name": "rect left of the image edge clips to x=0",
|
||||
"space": "device-pixels",
|
||||
"bounds": null,
|
||||
"rect": { "x": -40, "y": 100, "width": 200, "height": 50 },
|
||||
"image": { "width": 1080, "height": 2400 },
|
||||
"expectedProjection": { "x": -40, "y": 100, "width": 200, "height": 50 },
|
||||
"expectedIntersection": { "x": 0, "y": 100, "width": 160, "height": 50 }
|
||||
},
|
||||
{
|
||||
"name": "rect fully past the right edge intersects nothing",
|
||||
"space": "device-pixels",
|
||||
"bounds": null,
|
||||
"rect": { "x": 1080, "y": 100, "width": 100, "height": 50 },
|
||||
"image": { "width": 1080, "height": 2400 },
|
||||
"expectedProjection": { "x": 1080, "y": 100, "width": 100, "height": 50 },
|
||||
"expectedIntersection": null
|
||||
},
|
||||
{
|
||||
"name": "zero-area intersection (edge touch only) is empty, not a 1px sliver",
|
||||
"space": "device-pixels",
|
||||
"bounds": null,
|
||||
"rect": { "x": 1079, "y": 2400, "width": 10, "height": 10 },
|
||||
"image": { "width": 1080, "height": 2400 },
|
||||
"expectedProjection": { "x": 1079, "y": 2400, "width": 10, "height": 10 },
|
||||
"expectedIntersection": null
|
||||
}
|
||||
]
|
||||
@@ -135,6 +135,34 @@ test('screenshot replay script round-trips screenshot flags', () => {
|
||||
assert.equal(parsed[0]?.flags.screenshotNoStabilize, true);
|
||||
});
|
||||
|
||||
test('screenshot replay script round-trips a quoted --crop-on selector', () => {
|
||||
const actions: SessionAction[] = [
|
||||
{
|
||||
ts: Date.now(),
|
||||
command: 'screenshot',
|
||||
positionals: ['./page.png'],
|
||||
flags: {
|
||||
screenshotCropOn: 'role=cell label=General || role=button label=General',
|
||||
screenshotScale: 0.3,
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
const script = formatReplayScriptForTest(actions);
|
||||
assert.match(
|
||||
script,
|
||||
/screenshot "\.\/page\.png" --crop-on "role=cell label=General \|\| role=button label=General" --scale 0\.3/,
|
||||
);
|
||||
|
||||
const parsed = parseReplayScriptDetailed(script).actions;
|
||||
assert.deepEqual(parsed[0]?.positionals, ['./page.png']);
|
||||
assert.equal(
|
||||
parsed[0]?.flags.screenshotCropOn,
|
||||
'role=cell label=General || role=button label=General',
|
||||
);
|
||||
assert.equal(parsed[0]?.flags.screenshotScale, 0.3);
|
||||
});
|
||||
|
||||
test('snapshot replay script parses full refresh flags', () => {
|
||||
const ignoredLegacyFlag = '-' + 'c';
|
||||
const parsed = parseReplayScriptDetailed(
|
||||
|
||||
@@ -227,6 +227,10 @@ export function appendScreenshotActionScriptArgs(parts: string[], action: Sessio
|
||||
for (const positional of action.positionals ?? []) {
|
||||
parts.push(formatScriptArg(positional));
|
||||
}
|
||||
const cropOn = action.flags?.screenshotCropOn;
|
||||
if (typeof cropOn === 'string' && cropOn.length > 0) {
|
||||
parts.push('--crop-on', formatScriptArgQuoteIfNeeded(cropOn));
|
||||
}
|
||||
appendScreenshotScriptFlags(parts, action.flags);
|
||||
}
|
||||
|
||||
|
||||
@@ -34,6 +34,10 @@
|
||||
"types": "./src/png.ts",
|
||||
"default": "./src/png.ts"
|
||||
},
|
||||
"./png-crop": {
|
||||
"types": "./src/png-crop.ts",
|
||||
"default": "./src/png-crop.ts"
|
||||
},
|
||||
"./png-resize": {
|
||||
"types": "./src/png-resize.ts",
|
||||
"default": "./src/png-resize.ts"
|
||||
@@ -73,6 +77,10 @@
|
||||
"./snapshot-quality-verdict": {
|
||||
"types": "./src/snapshot-quality-verdict.ts",
|
||||
"default": "./src/snapshot-quality-verdict.ts"
|
||||
},
|
||||
"./snapshot-rect-projection": {
|
||||
"types": "./src/snapshot-rect-projection.ts",
|
||||
"default": "./src/snapshot-rect-projection.ts"
|
||||
}
|
||||
},
|
||||
"devDependencies": {
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import { afterAll, test } from 'vitest';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { PNG } from './png.ts';
|
||||
import { cropPngFile } from './png-crop.ts';
|
||||
import { terminatePngWorker } from './png-worker-client.ts';
|
||||
import { mkdtempForTestSync } from './tmp-dir.fixtures.ts';
|
||||
|
||||
afterAll(async () => {
|
||||
await terminatePngWorker();
|
||||
});
|
||||
|
||||
test('cropPngFile keeps only the box rows and columns, in place', async () => {
|
||||
const filePath = writeCheckedPng();
|
||||
await cropPngFile(filePath, { x: 2, y: 1, width: 3, height: 2 });
|
||||
|
||||
const cropped = PNG.sync.read(fs.readFileSync(filePath));
|
||||
assert.equal(cropped.width, 3);
|
||||
assert.equal(cropped.height, 2);
|
||||
assert.deepEqual(readPngPixel(cropped, 0, 0), pixel(2, 1));
|
||||
assert.deepEqual(readPngPixel(cropped, 2, 1), pixel(4, 2));
|
||||
});
|
||||
|
||||
test('a box grazing the image edges crops to the edge without clamping the origin', async () => {
|
||||
const filePath = writeCheckedPng();
|
||||
await cropPngFile(filePath, { x: 4, y: 2, width: 2, height: 2 });
|
||||
|
||||
const cropped = PNG.sync.read(fs.readFileSync(filePath));
|
||||
assert.equal(cropped.width, 2);
|
||||
assert.equal(cropped.height, 2);
|
||||
assert.deepEqual(readPngPixel(cropped, 1, 1), pixel(5, 3));
|
||||
});
|
||||
|
||||
test('a full-image box is a no-op that leaves the file decodable at the same size', async () => {
|
||||
const filePath = writeCheckedPng();
|
||||
const before = fs.readFileSync(filePath);
|
||||
await cropPngFile(filePath, { x: 0, y: 0, width: 6, height: 4 });
|
||||
assert.deepEqual(fs.readFileSync(filePath), before);
|
||||
});
|
||||
|
||||
test('boxes that exceed the image refuse instead of clamping', async () => {
|
||||
const filePath = writeCheckedPng();
|
||||
await assert.rejects(
|
||||
() => cropPngFile(filePath, { x: 5, y: 3, width: 3, height: 3 }),
|
||||
(error: unknown) => error instanceof Error && error.message.includes('exceeds'),
|
||||
);
|
||||
});
|
||||
|
||||
test('non-integer or non-positive boxes refuse', async () => {
|
||||
const filePath = writeCheckedPng();
|
||||
const boxes = [
|
||||
{ x: 1.5, y: 0, width: 2, height: 2 },
|
||||
{ x: 0, y: -1, width: 2, height: 2 },
|
||||
{ x: 0, y: 0, width: 0, height: 2 },
|
||||
{ x: 0, y: 0, width: 2, height: 0 },
|
||||
] as const;
|
||||
for (const box of boxes) {
|
||||
await assert.rejects(
|
||||
() => cropPngFile(filePath, box),
|
||||
(error: unknown) => error instanceof Error && error.message.includes('positive integer'),
|
||||
JSON.stringify(box),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
// A 6x4 grid whose pixel (x, y) carries (x*10, y*10) so a wrong source offset
|
||||
// is caught by the value, not just the size.
|
||||
function writeCheckedPng(): string {
|
||||
const filePath = path.join(mkdtempForTestSync('agent-device-png-crop-'), 'image.png');
|
||||
const png = new PNG({ width: 6, height: 4 });
|
||||
for (let y = 0; y < png.height; y += 1) {
|
||||
for (let x = 0; x < png.width; x += 1) {
|
||||
const offset = (y * png.width + x) * 4;
|
||||
png.data[offset] = x * 10;
|
||||
png.data[offset + 1] = y * 10;
|
||||
png.data[offset + 2] = 0;
|
||||
png.data[offset + 3] = 255;
|
||||
}
|
||||
}
|
||||
fs.writeFileSync(filePath, PNG.sync.write(png));
|
||||
return filePath;
|
||||
}
|
||||
|
||||
function pixel(x: number, y: number): number[] {
|
||||
return [x * 10, y * 10, 0, 255];
|
||||
}
|
||||
|
||||
function readPngPixel(png: PNG, x: number, y: number): number[] {
|
||||
const offset = (y * png.width + x) * 4;
|
||||
return [
|
||||
png.data[offset] ?? 0,
|
||||
png.data[offset + 1] ?? 0,
|
||||
png.data[offset + 2] ?? 0,
|
||||
png.data[offset + 3] ?? 0,
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import { promises as fs } from 'node:fs';
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import type { Rect } from '@agent-device/kernel/snapshot';
|
||||
import { PNG } from './png.ts';
|
||||
import { decodePngAsync, encodePngAsync } from './png-worker-client.ts';
|
||||
|
||||
/**
|
||||
* Crops `filePath` in place to `box` (positive integer pixels). `box` is the caller's
|
||||
* already-intersected region, so one outside the image is a caller bug — refused, not clamped.
|
||||
* Decode and encode run on the PNG worker thread; a full-image box is a no-op.
|
||||
*/
|
||||
export async function cropPngFile(filePath: string, box: Rect): Promise<void> {
|
||||
if (!isCropBox(box)) {
|
||||
throw new AppError(
|
||||
'INVALID_ARGS',
|
||||
'Screenshot crop box must be positive integer pixel offsets',
|
||||
);
|
||||
}
|
||||
|
||||
const source = await decodePngAsync(await fs.readFile(filePath), 'screenshot');
|
||||
if (box.x + box.width > source.width || box.y + box.height > source.height) {
|
||||
throw new AppError(
|
||||
'INVALID_ARGS',
|
||||
`Screenshot crop box ${box.width}x${box.height} at (${box.x}, ${box.y}) exceeds the ${source.width}x${source.height} image`,
|
||||
);
|
||||
}
|
||||
if (box.x === 0 && box.y === 0 && box.width === source.width && box.height === source.height) {
|
||||
return;
|
||||
}
|
||||
|
||||
await fs.writeFile(filePath, await encodePngAsync(cropPngBox(source, box)));
|
||||
}
|
||||
|
||||
function isCropBox(box: Rect): boolean {
|
||||
return (
|
||||
Number.isInteger(box.x) &&
|
||||
box.x >= 0 &&
|
||||
Number.isInteger(box.y) &&
|
||||
box.y >= 0 &&
|
||||
Number.isInteger(box.width) &&
|
||||
box.width > 0 &&
|
||||
Number.isInteger(box.height) &&
|
||||
box.height > 0
|
||||
);
|
||||
}
|
||||
|
||||
function cropPngBox(source: PNG, box: Rect): PNG {
|
||||
const output = new PNG({ width: box.width, height: box.height });
|
||||
for (let row = 0; row < box.height; row += 1) {
|
||||
const sourceStart = ((row + box.y) * source.width + box.x) * 4;
|
||||
source.data.copy(output.data, row * output.width * 4, sourceStart, sourceStart + box.width * 4);
|
||||
}
|
||||
return output;
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
import { test } from 'vitest';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import type { Rect } from '@agent-device/kernel/snapshot';
|
||||
import {
|
||||
intersectScreenshotRect,
|
||||
projectSnapshotRectToScreenshot,
|
||||
resolveScreenshotRectSpace,
|
||||
resolveSnapshotBounds,
|
||||
type ScreenshotRectSpace,
|
||||
} from './snapshot-rect-projection.ts';
|
||||
|
||||
// Golden geometry table for `screenshot --crop-on`: the numbers are the live
|
||||
// evidence baseline (iOS simulator 402/1206, Android 1080, macOS 586/1172),
|
||||
// so a projection-law change that moves any accepted cell's crop box turns this
|
||||
// red before the pixel-identity cross-check could ever notice.
|
||||
|
||||
type FixtureCase = {
|
||||
name: string;
|
||||
space: ScreenshotRectSpace;
|
||||
bounds: Rect | null;
|
||||
rect: Rect;
|
||||
image: { width: number; height: number };
|
||||
expectedProjection: Rect;
|
||||
expectedIntersection: Rect | null;
|
||||
};
|
||||
|
||||
const TABLE_PATH = path.resolve(
|
||||
import.meta.dirname,
|
||||
'..',
|
||||
'..',
|
||||
'..',
|
||||
'contracts',
|
||||
'fixtures',
|
||||
'screenshot-crop-geometry.json',
|
||||
);
|
||||
|
||||
test('the crop projection law agrees with every golden geometry case', () => {
|
||||
const cases = JSON.parse(fs.readFileSync(TABLE_PATH, 'utf8')) as FixtureCase[];
|
||||
assert.ok(cases.length > 0, 'geometry table must not be empty');
|
||||
const names = new Set(cases.map((fixture) => fixture.name));
|
||||
assert.equal(names.size, cases.length, 'geometry table case names must be unique');
|
||||
for (const fixture of cases) {
|
||||
const projected = projectSnapshotRectToScreenshot(
|
||||
fixture.space,
|
||||
fixture.bounds,
|
||||
fixture.rect,
|
||||
fixture.image.width,
|
||||
fixture.image.height,
|
||||
);
|
||||
assert.deepEqual(projected, fixture.expectedProjection, fixture.name);
|
||||
const intersection = intersectScreenshotRect(
|
||||
projected,
|
||||
fixture.image.width,
|
||||
fixture.image.height,
|
||||
);
|
||||
assert.deepEqual(intersection, fixture.expectedIntersection, fixture.name);
|
||||
}
|
||||
});
|
||||
|
||||
test('resolveScreenshotRectSpace maps each accepted backend to its projection space and refuses the rest', () => {
|
||||
assert.equal(resolveScreenshotRectSpace('android'), 'device-pixels');
|
||||
assert.equal(resolveScreenshotRectSpace('xctest'), 'viewport-points');
|
||||
assert.equal(resolveScreenshotRectSpace('macos-helper'), 'viewport-points');
|
||||
for (const backend of [undefined, 'linux', 'web', 'harmonyos', 'vega']) {
|
||||
assert.throws(
|
||||
() => resolveScreenshotRectSpace(backend),
|
||||
(error: unknown) =>
|
||||
error instanceof AppError &&
|
||||
error.code === 'UNSUPPORTED_OPERATION' &&
|
||||
error.details?.reason === SCREENSHOT_CROP_REASONS.targetNotAccepted,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('resolveSnapshotBounds prefers the largest viewport root, else unions every positive rect excluding unlabeled images', () => {
|
||||
assert.deepEqual(
|
||||
resolveSnapshotBounds([
|
||||
{ type: 'XCUIElementTypeImage', label: '', rect: { x: 0, y: 0, width: 50, height: 50 } },
|
||||
{
|
||||
type: 'XCUIElementTypeApplication',
|
||||
label: 'Settings',
|
||||
rect: { x: 0, y: 0, width: 402, height: 874 },
|
||||
},
|
||||
{
|
||||
type: 'XCUIElementTypeButton',
|
||||
label: 'Continue',
|
||||
rect: { x: 100, y: 100, width: 200, height: 50 },
|
||||
},
|
||||
]),
|
||||
{ x: 0, y: 0, width: 402, height: 874 },
|
||||
);
|
||||
assert.deepEqual(
|
||||
resolveSnapshotBounds([
|
||||
{ type: 'XCUIElementTypeButton', label: 'A', rect: { x: 10, y: 20, width: 100, height: 40 } },
|
||||
{
|
||||
type: 'XCUIElementTypeButton',
|
||||
label: 'B',
|
||||
rect: { x: 300, y: 400, width: 60, height: 30 },
|
||||
},
|
||||
{ type: 'XCUIElementTypeImage', label: '', rect: { x: 0, y: 0, width: 999, height: 999 } },
|
||||
]),
|
||||
{ x: 10, y: 20, width: 350, height: 410 },
|
||||
);
|
||||
assert.equal(
|
||||
resolveSnapshotBounds([
|
||||
{ type: 'XCUIElementTypeButton', label: 'A', rect: { x: 0, y: 0, width: 0, height: 50 } },
|
||||
]),
|
||||
null,
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,138 @@
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
|
||||
import {
|
||||
isMeaningfulSignal,
|
||||
isViewportRootNode,
|
||||
normalizeType,
|
||||
} from '@agent-device/contracts/snapshot';
|
||||
import { isPositiveFiniteRect, rectArea } from '@agent-device/kernel/rect';
|
||||
import type { Rect, SnapshotNode } from '@agent-device/kernel/snapshot';
|
||||
|
||||
/**
|
||||
* The space a snapshot's rects are expressed in, as far as the captured image is concerned:
|
||||
* `device-pixels` rects land 1:1 on the PNG (Android), `viewport-points` rects must be shifted
|
||||
* by the viewport bounds and scaled to the image (Apple, where 1x/2x/3x captures all share the
|
||||
* points-space tree).
|
||||
*/
|
||||
export type ScreenshotRectSpace = 'device-pixels' | 'viewport-points';
|
||||
|
||||
/**
|
||||
* The single decision site for `screenshot --crop-on`: which projection law a snapshot backend's
|
||||
* rects obey. Backends the crop feature has not accepted project nothing, so an unaccepted
|
||||
* backend is a typed refusal, never a guess.
|
||||
*/
|
||||
export function resolveScreenshotRectSpace(backend: string | undefined): ScreenshotRectSpace {
|
||||
switch (backend) {
|
||||
case 'android':
|
||||
return 'device-pixels';
|
||||
case 'xctest':
|
||||
case 'macos-helper':
|
||||
return 'viewport-points';
|
||||
default:
|
||||
throw new AppError(
|
||||
'UNSUPPORTED_OPERATION',
|
||||
`screenshot --crop-on does not accept snapshot backend "${backend ?? 'unknown'}"`,
|
||||
{ reason: SCREENSHOT_CROP_REASONS.targetNotAccepted },
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Where a snapshot rect falls in the captured image: the space-specific law only, with NO
|
||||
* clamping. Callers decide what out-of-image means — the overlay clamps to a 1px box, the crop
|
||||
* intersects and refuses the empty case.
|
||||
*/
|
||||
export function projectSnapshotRectToScreenshot(
|
||||
space: ScreenshotRectSpace,
|
||||
bounds: Rect | null,
|
||||
rect: Rect,
|
||||
imageWidth: number,
|
||||
imageHeight: number,
|
||||
): Rect {
|
||||
if (space === 'device-pixels' || bounds === null) {
|
||||
return roundRect(rect);
|
||||
}
|
||||
const scaleX = imageWidth / bounds.width;
|
||||
const scaleY = imageHeight / bounds.height;
|
||||
return {
|
||||
x: Math.round((rect.x - bounds.x) * scaleX),
|
||||
y: Math.round((rect.y - bounds.y) * scaleY),
|
||||
width: Math.round(rect.width * scaleX),
|
||||
height: Math.round(rect.height * scaleY),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The crop box a projected rect actually yields: its intersection with the image frame. `null`
|
||||
* means the rect occupies no image pixel — a crop cannot be taken, not a 1px sliver.
|
||||
*/
|
||||
export function intersectScreenshotRect(
|
||||
rect: Rect,
|
||||
imageWidth: number,
|
||||
imageHeight: number,
|
||||
): Rect | null {
|
||||
const x = Math.max(rect.x, 0);
|
||||
const y = Math.max(rect.y, 0);
|
||||
const right = Math.min(rect.x + rect.width, imageWidth);
|
||||
const bottom = Math.min(rect.y + rect.height, imageHeight);
|
||||
if (right <= x || bottom <= y) return null;
|
||||
return { x, y, width: right - x, height: bottom - y };
|
||||
}
|
||||
|
||||
/**
|
||||
* The viewport the tree's points-space rects are measured against: the largest viewport root
|
||||
* with a positive rect, else the union of every rect-carrying node (unlabeled images excluded as
|
||||
* bounds outliers). One copy, shared by the ref overlay and the crop projection.
|
||||
*/
|
||||
export function resolveSnapshotBounds(
|
||||
nodes: ReadonlyArray<Pick<SnapshotNode, 'type' | 'label' | 'rect'>>,
|
||||
): Rect | null {
|
||||
let viewport: Rect | null = null;
|
||||
for (const node of nodes) {
|
||||
if (!isViewportRootNode(node) || !isPositiveFiniteRect(node.rect)) continue;
|
||||
if (!viewport || rectArea(node.rect) > rectArea(viewport)) {
|
||||
viewport = node.rect;
|
||||
}
|
||||
}
|
||||
if (viewport) return viewport;
|
||||
|
||||
return measureSnapshotBounds(
|
||||
nodes.filter((node) => isPositiveFiniteRect(node.rect) && !isSnapshotBoundsOutlier(node)),
|
||||
);
|
||||
}
|
||||
|
||||
function measureSnapshotBounds(nodes: ReadonlyArray<Pick<SnapshotNode, 'rect'>>): Rect | null {
|
||||
let minX = Number.POSITIVE_INFINITY;
|
||||
let minY = Number.POSITIVE_INFINITY;
|
||||
let maxRight = Number.NEGATIVE_INFINITY;
|
||||
let maxBottom = Number.NEGATIVE_INFINITY;
|
||||
for (const node of nodes) {
|
||||
if (!isPositiveFiniteRect(node.rect)) continue;
|
||||
minX = Math.min(minX, node.rect.x);
|
||||
minY = Math.min(minY, node.rect.y);
|
||||
maxRight = Math.max(maxRight, node.rect.x + node.rect.width);
|
||||
maxBottom = Math.max(maxBottom, node.rect.y + node.rect.height);
|
||||
}
|
||||
if (!Number.isFinite(minX) || !Number.isFinite(minY) || maxRight <= minX || maxBottom <= minY) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
x: minX,
|
||||
y: minY,
|
||||
width: maxRight - minX,
|
||||
height: maxBottom - minY,
|
||||
};
|
||||
}
|
||||
|
||||
function isSnapshotBoundsOutlier(node: Pick<SnapshotNode, 'type' | 'label'>): boolean {
|
||||
return normalizeType(node.type ?? '') === 'image' && !isMeaningfulSignal(node.label);
|
||||
}
|
||||
|
||||
function roundRect(rect: Rect): Rect {
|
||||
return {
|
||||
x: Math.round(rect.x),
|
||||
y: Math.round(rect.y),
|
||||
width: Math.round(rect.width),
|
||||
height: Math.round(rect.height),
|
||||
};
|
||||
}
|
||||
@@ -67,6 +67,8 @@ export type CaptureSnapshotResult = {
|
||||
export type CaptureScreenshotOptions = AgentDeviceRequestOverrides & {
|
||||
path?: string;
|
||||
overlayRefs?: boolean;
|
||||
/** Crop the capture to the frame of the selector resolved on the same screen. */
|
||||
cropOn?: string;
|
||||
pixelDensity?: number;
|
||||
fullscreen?: boolean;
|
||||
scale?: number;
|
||||
|
||||
@@ -3,6 +3,7 @@ export {
|
||||
RETIRED_SCREENSHOT_MAX_SIZE,
|
||||
SCREENSHOT_ACTION_FLAG_KEYS,
|
||||
SCREENSHOT_COMMAND_FLAG_KEYS,
|
||||
SCREENSHOT_CROP_REASONS,
|
||||
SCREENSHOT_SCALE_LIMITS,
|
||||
SCREENSHOT_SPECIFIC_FLAG_DEFINITIONS,
|
||||
appendScreenshotScriptFlags,
|
||||
@@ -15,6 +16,7 @@ export {
|
||||
validateScreenshotScale,
|
||||
} from '../screenshot.ts';
|
||||
export type {
|
||||
ScreenshotCropReason,
|
||||
ScreenshotDispatchFlags,
|
||||
ScreenshotPublicOptions,
|
||||
ScreenshotRequestFlags,
|
||||
|
||||
@@ -26,5 +26,6 @@ export {
|
||||
extractNodeText,
|
||||
isFillableType,
|
||||
isMeaningfulLabel,
|
||||
isMeaningfulSignal,
|
||||
normalizeType,
|
||||
} from '../snapshot-text.ts';
|
||||
|
||||
@@ -602,35 +602,46 @@ export function resolveSnapshotRuntimePlan(input: {
|
||||
|
||||
const captureScreenshotUse = defineUse({ required: ['captureScreenshot'] });
|
||||
/**
|
||||
* `--overlay-refs` annotates the capture with the refs of a snapshot taken in the same request, so
|
||||
* the snapshot is part of what the command requires — not something to discover after the PNG is
|
||||
* Screenshot post-processing that resolves a snapshot taken in the same request — `--overlay-refs`
|
||||
* annotates the capture with snapshot refs, `--crop-on` crops it to a snapshot node's frame. The
|
||||
* snapshot is part of what the command requires, not something to discover after the PNG is
|
||||
* already on disk. Declaring it in the use is what lets admission refuse the whole request up
|
||||
* front on a target that can capture pixels but not a tree.
|
||||
*/
|
||||
const captureScreenshotWithOverlayRefsUse = defineUse({
|
||||
const captureScreenshotWithSnapshotUse = defineUse({
|
||||
required: ['captureScreenshot', 'captureSnapshot'],
|
||||
});
|
||||
|
||||
export const screenshotRuntimePlanUses = Object.freeze([
|
||||
captureScreenshotUse,
|
||||
captureScreenshotWithOverlayRefsUse,
|
||||
captureScreenshotWithSnapshotUse,
|
||||
] as const);
|
||||
|
||||
export type ScreenshotRuntimePlan =
|
||||
| Readonly<{ kind: 'capture'; use: typeof captureScreenshotUse }>
|
||||
| Readonly<{
|
||||
kind: 'capture-with-overlay-refs';
|
||||
use: typeof captureScreenshotWithOverlayRefsUse;
|
||||
use: typeof captureScreenshotWithSnapshotUse;
|
||||
}>
|
||||
| Readonly<{
|
||||
kind: 'capture-with-crop-on';
|
||||
use: typeof captureScreenshotWithSnapshotUse;
|
||||
}>;
|
||||
|
||||
/** Selects one owner-fact-backed capture plan from normalized command intent. */
|
||||
export function resolveScreenshotRuntimePlan(
|
||||
input: Readonly<{ overlayRefs: boolean }>,
|
||||
input: Readonly<{ overlayRefs: boolean; cropOn: boolean }>,
|
||||
): ScreenshotRuntimePlan {
|
||||
if (input.cropOn) {
|
||||
return Object.freeze({
|
||||
kind: 'capture-with-crop-on',
|
||||
use: captureScreenshotWithSnapshotUse,
|
||||
});
|
||||
}
|
||||
return input.overlayRefs
|
||||
? Object.freeze({
|
||||
kind: 'capture-with-overlay-refs',
|
||||
use: captureScreenshotWithOverlayRefsUse,
|
||||
use: captureScreenshotWithSnapshotUse,
|
||||
})
|
||||
: Object.freeze({ kind: 'capture', use: captureScreenshotUse });
|
||||
}
|
||||
|
||||
@@ -44,9 +44,33 @@ export function validateNoRetiredScreenshotMaxSize(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Machine-readable `screenshot --crop-on` outcome taxonomy. Failures carry these
|
||||
* values in `error.details.reason`; `partialIntersection` doubles as the
|
||||
* success-path warning token. Callers branch on the values, never on message
|
||||
* text. `pendingPixelIdentityEvidence` is the acceptance-matrix rejection
|
||||
* reason, not a runtime failure.
|
||||
*/
|
||||
export const SCREENSHOT_CROP_REASONS = {
|
||||
selectorInvalid: 'CROP_SELECTOR_INVALID',
|
||||
targetNotFound: 'CROP_TARGET_NOT_FOUND',
|
||||
targetAmbiguous: 'CROP_TARGET_AMBIGUOUS',
|
||||
captureUnreadable: 'CROP_CAPTURE_UNREADABLE',
|
||||
captureIncomplete: 'CROP_CAPTURE_INCOMPLETE',
|
||||
emptyIntersection: 'CROP_EMPTY_INTERSECTION',
|
||||
partialIntersection: 'CROP_PARTIAL_INTERSECTION',
|
||||
targetNotAccepted: 'CROP_TARGET_NOT_ACCEPTED',
|
||||
frameMismatch: 'CROP_FRAME_MISMATCH',
|
||||
pendingPixelIdentityEvidence: 'PENDING_PIXEL_IDENTITY_EVIDENCE',
|
||||
} as const;
|
||||
|
||||
export type ScreenshotCropReason =
|
||||
(typeof SCREENSHOT_CROP_REASONS)[keyof typeof SCREENSHOT_CROP_REASONS];
|
||||
|
||||
export const SCREENSHOT_COMMAND_FLAG_KEYS = [
|
||||
'out',
|
||||
'overlayRefs',
|
||||
'screenshotCropOn',
|
||||
'screenshotPixelDensity',
|
||||
'screenshotFullscreen',
|
||||
'screenshotScale',
|
||||
@@ -55,6 +79,7 @@ export const SCREENSHOT_COMMAND_FLAG_KEYS = [
|
||||
] as const;
|
||||
|
||||
export const SCREENSHOT_ACTION_FLAG_KEYS = [
|
||||
'screenshotCropOn',
|
||||
'screenshotPixelDensity',
|
||||
'screenshotFullscreen',
|
||||
'screenshotScale',
|
||||
@@ -67,7 +92,7 @@ type ScreenshotSpecificFlagKey = (typeof SCREENSHOT_ACTION_FLAG_KEYS)[number];
|
||||
type ScreenshotSpecificFlagDefinition = {
|
||||
key: ScreenshotSpecificFlagKey;
|
||||
names: readonly string[];
|
||||
type: 'boolean' | 'int' | 'number';
|
||||
type: 'boolean' | 'int' | 'number' | 'string';
|
||||
min?: number;
|
||||
max?: number;
|
||||
usageLabel: string;
|
||||
@@ -75,6 +100,14 @@ type ScreenshotSpecificFlagDefinition = {
|
||||
};
|
||||
|
||||
export const SCREENSHOT_SPECIFIC_FLAG_DEFINITIONS: readonly ScreenshotSpecificFlagDefinition[] = [
|
||||
{
|
||||
key: 'screenshotCropOn',
|
||||
names: ['--crop-on'],
|
||||
type: 'string',
|
||||
usageLabel: '--crop-on <selector-expression>',
|
||||
usageDescription:
|
||||
'Screenshot: crop the capture to the frame of the selector resolved on the same screen',
|
||||
},
|
||||
{
|
||||
key: 'screenshotPixelDensity',
|
||||
names: ['--pixel-density'],
|
||||
@@ -144,9 +177,18 @@ const SCREENSHOT_SCRIPT_NUMBER_FLAGS = [
|
||||
},
|
||||
] as const;
|
||||
|
||||
const SCREENSHOT_SCRIPT_STRING_FLAGS = [
|
||||
{
|
||||
token: '--crop-on',
|
||||
key: 'screenshotCropOn',
|
||||
label: 'screenshot --crop-on',
|
||||
},
|
||||
] as const;
|
||||
|
||||
export type ScreenshotRequestFlags = {
|
||||
out?: string;
|
||||
overlayRefs?: boolean;
|
||||
screenshotCropOn?: string;
|
||||
screenshotPixelDensity?: number;
|
||||
screenshotFullscreen?: boolean;
|
||||
screenshotScale?: number;
|
||||
@@ -164,6 +206,7 @@ export type ScreenshotDispatchFlags = Pick<
|
||||
|
||||
export type ScreenshotRuntimeFlags = Pick<
|
||||
ScreenshotRequestFlags,
|
||||
| 'screenshotCropOn'
|
||||
| 'screenshotPixelDensity'
|
||||
| 'screenshotFullscreen'
|
||||
| 'screenshotScale'
|
||||
@@ -173,6 +216,7 @@ export type ScreenshotRuntimeFlags = Pick<
|
||||
|
||||
export type ScreenshotPublicOptions = {
|
||||
overlayRefs?: boolean;
|
||||
cropOn?: string;
|
||||
pixelDensity?: number;
|
||||
fullscreen?: boolean;
|
||||
scale?: number;
|
||||
@@ -182,6 +226,7 @@ export type ScreenshotPublicOptions = {
|
||||
|
||||
export type ScreenshotRuntimeOptions = {
|
||||
overlayRefs?: boolean;
|
||||
cropOn?: string;
|
||||
pixelDensity?: number;
|
||||
fullscreen?: boolean;
|
||||
scale?: number;
|
||||
@@ -194,6 +239,7 @@ export function screenshotOptionsFromFlags(
|
||||
): ScreenshotRuntimeOptions {
|
||||
return stripUndefined({
|
||||
overlayRefs: flags?.overlayRefs,
|
||||
cropOn: flags?.screenshotCropOn,
|
||||
pixelDensity: flags?.screenshotPixelDensity,
|
||||
fullscreen: flags?.screenshotFullscreen,
|
||||
scale: flags?.screenshotScale,
|
||||
@@ -207,6 +253,7 @@ export function screenshotFlagsFromOptions(
|
||||
): Partial<ScreenshotRequestFlags> {
|
||||
return stripUndefined({
|
||||
overlayRefs: options.overlayRefs,
|
||||
screenshotCropOn: options.screenshotCropOn ?? options.cropOn,
|
||||
screenshotPixelDensity: options.screenshotPixelDensity ?? options.pixelDensity,
|
||||
screenshotFullscreen: options.screenshotFullscreen ?? options.fullscreen,
|
||||
screenshotScale: options.screenshotScale,
|
||||
@@ -269,7 +316,8 @@ export function readScreenshotScriptFlag(params: {
|
||||
return (
|
||||
readScreenshotBooleanScriptFlag(token, flags, index) ??
|
||||
readScreenshotIntScriptFlag({ args, index, flags, token }) ??
|
||||
readScreenshotNumberScriptFlag({ args, index, flags, token }) ?? { handled: false }
|
||||
readScreenshotNumberScriptFlag({ args, index, flags, token }) ??
|
||||
readScreenshotStringScriptFlag({ args, index, flags, token }) ?? { handled: false }
|
||||
);
|
||||
}
|
||||
|
||||
@@ -324,3 +372,19 @@ function readScreenshotNumberScriptFlag(params: {
|
||||
params.flags[definition.key] = parsed;
|
||||
return { handled: true, nextIndex: params.index + 1 };
|
||||
}
|
||||
|
||||
function readScreenshotStringScriptFlag(params: {
|
||||
args: readonly string[];
|
||||
index: number;
|
||||
flags: Partial<ScreenshotRequestFlags>;
|
||||
token: string | undefined;
|
||||
}): { handled: true; nextIndex: number } | undefined {
|
||||
const definition = SCREENSHOT_SCRIPT_STRING_FLAGS.find((entry) => entry.token === params.token);
|
||||
if (!definition) return undefined;
|
||||
const value = params.args[params.index + 1];
|
||||
if (typeof value !== 'string' || value.length === 0) {
|
||||
throw new AppError('INVALID_ARGS', `${definition.label} requires a selector expression`);
|
||||
}
|
||||
params.flags[definition.key] = value;
|
||||
return { handled: true, nextIndex: params.index + 1 };
|
||||
}
|
||||
|
||||
@@ -38,6 +38,14 @@ export function isMeaningfulLabel(value: string): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
/** A non-empty, non-boolean `label`/`value` is a usable overlay or crop signal. */
|
||||
export function isMeaningfulSignal(value: string | undefined): boolean {
|
||||
if (typeof value !== 'string') return false;
|
||||
const trimmed = value.trim();
|
||||
if (!trimmed) return false;
|
||||
return !/^(true|false)$/i.test(trimmed);
|
||||
}
|
||||
|
||||
export function extractNodeText(
|
||||
node: Pick<RawSnapshotNode, 'label' | 'value' | 'identifier'>,
|
||||
): string {
|
||||
|
||||
@@ -15,6 +15,7 @@ export type ScreenshotResultData = {
|
||||
logicalHeight?: number;
|
||||
pixelDensity?: number;
|
||||
overlayRefs?: ScreenshotOverlayRef[];
|
||||
warnings?: string[];
|
||||
};
|
||||
export type BackendSnapshotResult = {
|
||||
nodes?: SnapshotNode[];
|
||||
|
||||
@@ -89,6 +89,11 @@ export const SELECTOR_RESOLUTION_POLICIES = {
|
||||
ambiguity: 'reject-candidates',
|
||||
requireRect: false,
|
||||
},
|
||||
/** `screenshot --crop-on` — crops the capture to the resolved node's frame. */
|
||||
cropTarget: {
|
||||
ambiguity: 'fail-closed',
|
||||
requireRect: true,
|
||||
},
|
||||
} as const satisfies Record<string, SelectorResolutionPolicy>;
|
||||
|
||||
/**
|
||||
|
||||
@@ -361,6 +361,15 @@ function summarizeProviderScenarioFlagExclusions() {
|
||||
owner: 'runner XCTest unit, snapshot-lines, and snapshot-quality tests',
|
||||
keys: ['snapshotCustomActions'],
|
||||
},
|
||||
{
|
||||
// The crop is daemon-level post-processing: the platform write happens first, then the
|
||||
// daemon crops the PNG against a fresh snapshot whose pixel/tree identity the fake
|
||||
// provider scenario fixtures cannot fabricate. Covered instead by the daemon crop-leaf
|
||||
// unit tests and the live device verification in the feature's PR evidence.
|
||||
name: 'daemon screenshot selector crop',
|
||||
owner: 'daemon screenshot-crop unit and live device verification',
|
||||
keys: ['screenshotCropOn'],
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
|
||||
@@ -468,6 +468,7 @@ test('the real tree parses, declares, and passes R11', () => {
|
||||
'@agent-device/capture-kit/ios-snapshot-planning',
|
||||
'@agent-device/capture-kit/mobile-snapshot-semantics',
|
||||
'@agent-device/capture-kit/png',
|
||||
'@agent-device/capture-kit/png-crop',
|
||||
'@agent-device/capture-kit/png-resize',
|
||||
'@agent-device/capture-kit/png-rgb-difference',
|
||||
'@agent-device/capture-kit/png-size',
|
||||
@@ -478,6 +479,7 @@ test('the real tree parses, declares, and passes R11', () => {
|
||||
'@agent-device/capture-kit/snapshot-occlusion',
|
||||
'@agent-device/capture-kit/snapshot-quality-backend-capabilities',
|
||||
'@agent-device/capture-kit/snapshot-quality-verdict',
|
||||
'@agent-device/capture-kit/snapshot-rect-projection',
|
||||
]);
|
||||
|
||||
const provisionKitPackage = packages.find((pkg) => pkg.name === '@agent-device/provision-kit');
|
||||
|
||||
@@ -53,6 +53,14 @@ export const WEB_DESKTOP_DEVICE: DeviceInfo = {
|
||||
booted: true,
|
||||
};
|
||||
|
||||
export const ANDROID_DEVICE: DeviceInfo = {
|
||||
platform: 'android',
|
||||
id: 'and-1',
|
||||
name: 'Pixel 8',
|
||||
kind: 'device',
|
||||
booted: true,
|
||||
};
|
||||
|
||||
export const ANDROID_TV_DEVICE: DeviceInfo = {
|
||||
platform: 'android',
|
||||
id: 'and-tv-1',
|
||||
|
||||
@@ -79,7 +79,7 @@ import {
|
||||
} from './metro/metro-session-hints.ts';
|
||||
import { isRecord } from '@agent-device/kernel/record';
|
||||
import { createLeaseClient } from './client/lease-client.ts';
|
||||
import { readScreenshotResultData } from './client/screenshot-result.ts';
|
||||
import { normalizeScreenshotCaptureResult } from './client/screenshot-result.ts';
|
||||
|
||||
type ProjectedSystemCommandClient = ProjectedNavigationCommandClient<InternalRequestOptions> &
|
||||
Pick<AgentDeviceCommandClient, 'appState' | 'keyboard' | 'clipboard'>;
|
||||
@@ -372,17 +372,7 @@ export function createAgentDeviceClient(
|
||||
// (The caller opted into a non-default level, so the static type is the
|
||||
// default shape; the runtime value is the leveled payload.)
|
||||
if (isLeveledResponse(options)) return data as unknown as CaptureScreenshotResult;
|
||||
const screenshot = readScreenshotResultData(data);
|
||||
return {
|
||||
path: readRequiredString(data, 'path'),
|
||||
width: screenshot?.width,
|
||||
height: screenshot?.height,
|
||||
logicalWidth: screenshot?.logicalWidth,
|
||||
logicalHeight: screenshot?.logicalHeight,
|
||||
pixelDensity: screenshot?.pixelDensity,
|
||||
overlayRefs: screenshot?.overlayRefs,
|
||||
identifiers: { session },
|
||||
};
|
||||
return normalizeScreenshotCaptureResult(data, session);
|
||||
},
|
||||
diff: async (options) => await executeCommand<CommandResult<'diff'>>('diff', options),
|
||||
},
|
||||
|
||||
@@ -28,6 +28,8 @@ test('usageForCommand documents screenshot web aliases and stabilization flags',
|
||||
assert.match(help, /low-latency Android capture loops/);
|
||||
assert.match(help, /--normalize-status-bar/);
|
||||
assert.match(help, /deterministic iOS simulator chrome/);
|
||||
assert.match(help, /--crop-on <selector-expression>/);
|
||||
assert.match(help, /crop the capture to the frame of the selector/);
|
||||
});
|
||||
|
||||
test('usageForCommand documents screenshot diff normalization', async () => {
|
||||
|
||||
@@ -53,6 +53,31 @@ test('screenshot --level digest --json preserves the digest payload through the
|
||||
assert.deepEqual(parsed.data, digest);
|
||||
});
|
||||
|
||||
const CROP_WARNING =
|
||||
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame';
|
||||
|
||||
test('screenshot surfaces response-level warnings under the summary', async () => {
|
||||
const full = { path: '/tmp/shot.png', width: 40, height: 20, warnings: [CROP_WARNING] };
|
||||
const client = clientReturning(full);
|
||||
const flags = { json: false } as CliFlags;
|
||||
|
||||
const out = await captureStdout(() => screenshotCommand({ positionals: [], flags, client }));
|
||||
|
||||
assert.equal(out, `/tmp/shot.png (40x20)\n${CROP_WARNING}\n`);
|
||||
});
|
||||
|
||||
test('screenshot --json carries response-level warnings in the data', async () => {
|
||||
const full = { path: '/tmp/shot.png', width: 40, height: 20, warnings: [CROP_WARNING] };
|
||||
const client = clientReturning(full);
|
||||
const flags = { json: true } as CliFlags;
|
||||
|
||||
const out = await captureStdout(() => screenshotCommand({ positionals: [], flags, client }));
|
||||
const parsed = JSON.parse(out) as { data: { path: string; warnings?: string[] } };
|
||||
|
||||
assert.equal(parsed.data.path, '/tmp/shot.png');
|
||||
assert.deepEqual(parsed.data.warnings, [CROP_WARNING]);
|
||||
});
|
||||
|
||||
test('screenshot --json at the default level still emits normalized screenshot metadata', async () => {
|
||||
const full = {
|
||||
path: '/tmp/shot.png',
|
||||
|
||||
@@ -26,11 +26,13 @@ export const screenshotCommand: ClientCommandHandler = async ({ positionals, fla
|
||||
return true;
|
||||
}
|
||||
const data = pickScreenshotResultData(result);
|
||||
await writeCommandOutput(flags, data, () =>
|
||||
result.overlayRefs
|
||||
await writeCommandOutput(flags, data, () => {
|
||||
const summary = result.overlayRefs
|
||||
? `Annotated ${result.overlayRefs.length} refs onto ${result.path}`
|
||||
: formatScreenshotSummary(result),
|
||||
);
|
||||
: formatScreenshotSummary(result);
|
||||
const warnings = result.warnings ?? [];
|
||||
return warnings.length > 0 ? [summary, ...warnings].join('\n') : summary;
|
||||
});
|
||||
return true;
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { test } from 'vitest';
|
||||
import {
|
||||
normalizeScreenshotCaptureResult,
|
||||
pickScreenshotResultData,
|
||||
} from '../screenshot-result.ts';
|
||||
|
||||
const CROP_WARNING =
|
||||
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame';
|
||||
|
||||
test('normalizeScreenshotCaptureResult surfaces response-level warnings', () => {
|
||||
const result = normalizeScreenshotCaptureResult(
|
||||
{
|
||||
path: '/tmp/screenshot.png',
|
||||
width: 40,
|
||||
height: 20,
|
||||
warnings: [CROP_WARNING],
|
||||
},
|
||||
'qa',
|
||||
);
|
||||
assert.equal(result.path, '/tmp/screenshot.png');
|
||||
assert.equal(result.width, 40);
|
||||
assert.equal(result.height, 20);
|
||||
assert.deepEqual(result.warnings, [CROP_WARNING]);
|
||||
assert.deepEqual(result.identifiers, { session: 'qa' });
|
||||
});
|
||||
|
||||
test('normalizeScreenshotCaptureResult omits warnings when the response carries none', () => {
|
||||
const result = normalizeScreenshotCaptureResult({ path: '/tmp/screenshot.png' }, 'qa');
|
||||
assert.equal(result.path, '/tmp/screenshot.png');
|
||||
assert.ok(!('warnings' in result));
|
||||
assert.deepEqual(result.identifiers, { session: 'qa' });
|
||||
});
|
||||
|
||||
test('pickScreenshotResultData keeps warnings only when present and non-empty', () => {
|
||||
assert.deepEqual(
|
||||
pickScreenshotResultData({ path: '/tmp/a.png', width: 40, warnings: [CROP_WARNING] }),
|
||||
{ path: '/tmp/a.png', width: 40, warnings: [CROP_WARNING] },
|
||||
);
|
||||
assert.deepEqual(pickScreenshotResultData({ path: '/tmp/a.png', warnings: [] }), {
|
||||
path: '/tmp/a.png',
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { ScreenshotResultData } from '@agent-device/contracts/capture';
|
||||
import type { CaptureScreenshotResult } from '@agent-device/contracts/client';
|
||||
import { isRecord, parsePoint, parseRect, readRequiredString } from '@agent-device/kernel/record';
|
||||
import type { ScreenshotOverlayRef } from '@agent-device/kernel/snapshot';
|
||||
import { isRecord, parsePoint, parseRect } from '@agent-device/kernel/record';
|
||||
|
||||
export function pickScreenshotResultData(value: ScreenshotResultData): ScreenshotResultData {
|
||||
return {
|
||||
@@ -11,6 +12,26 @@ export function pickScreenshotResultData(value: ScreenshotResultData): Screensho
|
||||
...(typeof value.logicalHeight === 'number' ? { logicalHeight: value.logicalHeight } : {}),
|
||||
...(typeof value.pixelDensity === 'number' ? { pixelDensity: value.pixelDensity } : {}),
|
||||
...(value.overlayRefs ? { overlayRefs: value.overlayRefs } : {}),
|
||||
...(value.warnings && value.warnings.length > 0 ? { warnings: value.warnings } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
/** The default-level daemon payload, normalized into the typed client result. */
|
||||
export function normalizeScreenshotCaptureResult(
|
||||
data: Record<string, unknown>,
|
||||
session: string,
|
||||
): CaptureScreenshotResult {
|
||||
const screenshot = readScreenshotResultData(data);
|
||||
return {
|
||||
path: readRequiredString(data, 'path'),
|
||||
width: screenshot?.width,
|
||||
height: screenshot?.height,
|
||||
logicalWidth: screenshot?.logicalWidth,
|
||||
logicalHeight: screenshot?.logicalHeight,
|
||||
pixelDensity: screenshot?.pixelDensity,
|
||||
overlayRefs: screenshot?.overlayRefs,
|
||||
...(screenshot?.warnings ? { warnings: screenshot.warnings } : {}),
|
||||
identifiers: { session },
|
||||
};
|
||||
}
|
||||
|
||||
@@ -22,31 +43,44 @@ type ScreenshotOverlayRefData = {
|
||||
center?: unknown;
|
||||
};
|
||||
|
||||
export function readScreenshotResultData(value: unknown): ScreenshotResultData | undefined {
|
||||
function readScreenshotResultData(value: unknown): ScreenshotResultData | undefined {
|
||||
if (!isRecord(value)) return undefined;
|
||||
const path = typeof value.path === 'string' ? value.path : undefined;
|
||||
const width = typeof value.width === 'number' ? value.width : undefined;
|
||||
const height = typeof value.height === 'number' ? value.height : undefined;
|
||||
const logicalWidth = typeof value.logicalWidth === 'number' ? value.logicalWidth : undefined;
|
||||
const logicalHeight = typeof value.logicalHeight === 'number' ? value.logicalHeight : undefined;
|
||||
const pixelDensity = typeof value.pixelDensity === 'number' ? value.pixelDensity : undefined;
|
||||
const overlayRefs = Array.isArray(value.overlayRefs)
|
||||
? value.overlayRefs.filter(isScreenshotOverlayRefData).flatMap((entry) => {
|
||||
const overlayRef = readScreenshotOverlayRef(entry);
|
||||
return overlayRef ? [overlayRef] : [];
|
||||
})
|
||||
: undefined;
|
||||
const warnings = readScreenshotWarnings(value.warnings);
|
||||
return pickScreenshotResultData({
|
||||
path,
|
||||
width,
|
||||
height,
|
||||
logicalWidth,
|
||||
logicalHeight,
|
||||
pixelDensity,
|
||||
overlayRefs,
|
||||
path: readStringField(value, 'path'),
|
||||
width: readNumberField(value, 'width'),
|
||||
height: readNumberField(value, 'height'),
|
||||
logicalWidth: readNumberField(value, 'logicalWidth'),
|
||||
logicalHeight: readNumberField(value, 'logicalHeight'),
|
||||
pixelDensity: readNumberField(value, 'pixelDensity'),
|
||||
overlayRefs: readScreenshotOverlayRefs(value.overlayRefs),
|
||||
...(warnings !== undefined ? { warnings } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
function readNumberField(value: Record<string, unknown>, key: string): number | undefined {
|
||||
const field = value[key];
|
||||
return typeof field === 'number' ? field : undefined;
|
||||
}
|
||||
|
||||
function readStringField(value: Record<string, unknown>, key: string): string | undefined {
|
||||
const field = value[key];
|
||||
return typeof field === 'string' ? field : undefined;
|
||||
}
|
||||
|
||||
function readScreenshotOverlayRefs(value: unknown): ScreenshotOverlayRef[] | undefined {
|
||||
if (!Array.isArray(value)) return undefined;
|
||||
return value.filter(isScreenshotOverlayRefData).flatMap((entry) => {
|
||||
const overlayRef = readScreenshotOverlayRef(entry);
|
||||
return overlayRef ? [overlayRef] : [];
|
||||
});
|
||||
}
|
||||
|
||||
function readScreenshotWarnings(value: unknown): string[] | undefined {
|
||||
if (!Array.isArray(value)) return undefined;
|
||||
return value.filter((entry): entry is string => typeof entry === 'string' && entry.length > 0);
|
||||
}
|
||||
|
||||
function readScreenshotOverlayRef(
|
||||
record: ScreenshotOverlayRefData,
|
||||
): ScreenshotOverlayRef | undefined {
|
||||
|
||||
@@ -11,12 +11,14 @@ import {
|
||||
screenshotOptionsFromFlags,
|
||||
validateNoRetiredScreenshotMaxSize,
|
||||
validateScreenshotScale,
|
||||
type ScreenshotRequestFlags,
|
||||
} from '@agent-device/contracts/capture';
|
||||
|
||||
test('screenshot flag projection maps CLI flags to runtime options', () => {
|
||||
assert.deepEqual(
|
||||
screenshotOptionsFromFlags({
|
||||
overlayRefs: true,
|
||||
screenshotCropOn: 'role=button label=Save',
|
||||
screenshotPixelDensity: 2,
|
||||
screenshotFullscreen: true,
|
||||
screenshotScale: 0.3,
|
||||
@@ -25,6 +27,7 @@ test('screenshot flag projection maps CLI flags to runtime options', () => {
|
||||
}),
|
||||
{
|
||||
overlayRefs: true,
|
||||
cropOn: 'role=button label=Save',
|
||||
pixelDensity: 2,
|
||||
fullscreen: true,
|
||||
scale: 0.3,
|
||||
@@ -38,6 +41,7 @@ test('screenshot flag projection maps public options to request flags', () => {
|
||||
assert.deepEqual(
|
||||
screenshotFlagsFromOptions({
|
||||
overlayRefs: true,
|
||||
cropOn: 'label="Network & internet"',
|
||||
pixelDensity: 3,
|
||||
fullscreen: true,
|
||||
stabilize: false,
|
||||
@@ -45,6 +49,7 @@ test('screenshot flag projection maps public options to request flags', () => {
|
||||
}),
|
||||
{
|
||||
overlayRefs: true,
|
||||
screenshotCropOn: 'label="Network & internet"',
|
||||
screenshotPixelDensity: 3,
|
||||
screenshotFullscreen: true,
|
||||
screenshotNoStabilize: true,
|
||||
@@ -84,7 +89,7 @@ test('retired max-size inputs are refused with migration guidance', () => {
|
||||
|
||||
test('screenshot script flags use the shared recorded flag contract', () => {
|
||||
const parts: string[] = [];
|
||||
const flags = {};
|
||||
const flags: Partial<ScreenshotRequestFlags> = {};
|
||||
|
||||
let result = readScreenshotScriptFlag({ args: ['--full'], index: 0, flags });
|
||||
assert.deepEqual(result, { handled: true, nextIndex: 0 });
|
||||
@@ -100,6 +105,17 @@ test('screenshot script flags use the shared recorded flag contract', () => {
|
||||
assert.deepEqual(result, { handled: true, nextIndex: 0 });
|
||||
result = readScreenshotScriptFlag({ args: ['--pixel-density', '3'], index: 0, flags });
|
||||
assert.deepEqual(result, { handled: true, nextIndex: 1 });
|
||||
result = readScreenshotScriptFlag({
|
||||
args: ['--crop-on', 'role=cell label=General || role=button label=General'],
|
||||
index: 0,
|
||||
flags,
|
||||
});
|
||||
assert.deepEqual(result, { handled: true, nextIndex: 1 });
|
||||
assert.equal(flags.screenshotCropOn, 'role=cell label=General || role=button label=General');
|
||||
assert.throws(() => readScreenshotScriptFlag({ args: ['--crop-on'], index: 0, flags: {} }), {
|
||||
code: 'INVALID_ARGS',
|
||||
message: /requires a selector expression/,
|
||||
});
|
||||
|
||||
appendScreenshotScriptFlags(parts, flags);
|
||||
|
||||
@@ -113,6 +129,7 @@ test('screenshot script flags use the shared recorded flag contract', () => {
|
||||
'--normalize-status-bar',
|
||||
]);
|
||||
assert.deepEqual(SCREENSHOT_ACTION_FLAG_KEYS, [
|
||||
'screenshotCropOn',
|
||||
'screenshotPixelDensity',
|
||||
'screenshotFullscreen',
|
||||
'screenshotScale',
|
||||
@@ -126,6 +143,7 @@ test('screenshot script flags use the shared recorded flag contract', () => {
|
||||
assert.deepEqual(SCREENSHOT_COMMAND_FLAG_KEYS, [
|
||||
'out',
|
||||
'overlayRefs',
|
||||
'screenshotCropOn',
|
||||
'screenshotPixelDensity',
|
||||
'screenshotFullscreen',
|
||||
'screenshotScale',
|
||||
|
||||
@@ -34,6 +34,9 @@ const screenshotCommandMetadata = defineFieldCommandMetadata(
|
||||
screenshotCommandDescription,
|
||||
{
|
||||
path: stringField('Output path.'),
|
||||
cropOn: stringField(
|
||||
'Selector expression; the capture is cropped to the frame the selector resolves on the same screen.',
|
||||
),
|
||||
overlayRefs: booleanField(),
|
||||
pixelDensity: integerField('Output screenshot pixel density in pixels per logical point.', {
|
||||
min: 1,
|
||||
@@ -77,7 +80,7 @@ export const screenshotCommandFacet = defineCommandFacet({
|
||||
text: {
|
||||
summary: 'Capture a screenshot',
|
||||
cliDetail:
|
||||
'Web defaults to the viewport; use --fullscreen, --full, or -f for the entire page. iOS simulators default to 1x logical-point output; use --pixel-density to request a different screenshot density. macOS app sessions default to the app window; use --fullscreen for full desktop, --scale to downscale, --overlay-refs to annotate current refs, --normalize-status-bar for deterministic iOS simulator chrome, or --no-stabilize for low-latency Android capture loops.',
|
||||
'Web defaults to the viewport; use --fullscreen, --full, or -f for the entire page. iOS simulators default to 1x logical-point output; use --pixel-density to request a different screenshot density. macOS app sessions default to the app window; use --fullscreen for full desktop, --scale to downscale, --crop-on <selector> to crop the capture to the frame the selector resolves on the same screen (currently iOS simulators and Android emulators), --overlay-refs to annotate current refs, --normalize-status-bar for deterministic iOS simulator chrome, or --no-stabilize for low-latency Android capture loops.',
|
||||
},
|
||||
metadata: screenshotCommandMetadata,
|
||||
definition: screenshotCommandDefinition,
|
||||
|
||||
@@ -128,9 +128,11 @@ test('disambiguation declines when candidates are genuinely indistinguishable',
|
||||
});
|
||||
|
||||
test('fail-closed rows refuse an ambiguous tree instead of guessing', () => {
|
||||
const outcome = outcomeFor('readUnique', AMBIGUOUS_TREE);
|
||||
assert.equal(outcome.kind, 'ambiguous');
|
||||
if (outcome.kind === 'ambiguous') assert.equal(outcome.matchedNodes.length, 2);
|
||||
for (const name of ['readUnique', 'cropTarget'] as const) {
|
||||
const outcome = outcomeFor(name, AMBIGUOUS_TREE);
|
||||
assert.equal(outcome.kind, 'ambiguous', name);
|
||||
if (outcome.kind === 'ambiguous') assert.equal(outcome.matchedNodes.length, 2, name);
|
||||
}
|
||||
});
|
||||
|
||||
test('first-match rows take the head of an ambiguous tree', () => {
|
||||
@@ -155,7 +157,7 @@ test('reject-candidates surfaces every candidate for the caller to narrow or ref
|
||||
});
|
||||
|
||||
test('rect-requiring rows skip rectless nodes; read and wait rows accept them', () => {
|
||||
for (const name of ['act', 'findAct', 'actCoveredDiagnosis'] as const) {
|
||||
for (const name of ['act', 'findAct', 'actCoveredDiagnosis', 'cropTarget'] as const) {
|
||||
assert.equal(outcomeFor(name, RECTLESS_TREE).kind, 'none', name);
|
||||
}
|
||||
for (const name of ['readUnique', 'readAny', 'readText', 'wait'] as const) {
|
||||
@@ -189,4 +191,5 @@ test('the documented per-caller contracts are the ones declared', () => {
|
||||
assert.equal(SELECTOR_RESOLUTION_POLICIES.readAny.ambiguity, 'first-match');
|
||||
assert.equal(SELECTOR_RESOLUTION_POLICIES.wait.ambiguity, 'first-match');
|
||||
assert.equal(SELECTOR_RESOLUTION_POLICIES.findAct.ambiguity, 'reject-candidates');
|
||||
assert.equal(SELECTOR_RESOLUTION_POLICIES.cropTarget.ambiguity, 'fail-closed');
|
||||
});
|
||||
|
||||
@@ -190,6 +190,14 @@ export const SELECTOR_PIPELINE_POLICIES = {
|
||||
promotion: 'hittable-ancestor-below-root',
|
||||
poll: 'none',
|
||||
},
|
||||
/** `screenshot --crop-on`: crops the capture to the resolved node's frame. */
|
||||
cropTarget: {
|
||||
resolution: SELECTOR_RESOLUTION_POLICIES.cropTarget,
|
||||
occlusion: 'ignore',
|
||||
offscreen: 'ignore',
|
||||
promotion: 'none',
|
||||
poll: 'none',
|
||||
},
|
||||
} as const satisfies Record<string, SelectorPipelinePolicy | SelectorListPolicy>;
|
||||
|
||||
export type SelectorPipelinePolicyName = keyof typeof SELECTOR_PIPELINE_POLICIES;
|
||||
|
||||
@@ -25,6 +25,7 @@ const NODE_STAGE_ROWS = [
|
||||
'findWait',
|
||||
'wait',
|
||||
'findAct',
|
||||
'cropTarget',
|
||||
] as const;
|
||||
|
||||
/**
|
||||
@@ -149,7 +150,14 @@ test('the occlusion stage decides candidacy: acting rows drop covered nodes, the
|
||||
);
|
||||
assert.equal(listed.list?.matchedNodes.length, 1, row);
|
||||
}
|
||||
for (const row of ['readText', 'readUnique', 'readAny', 'wait', 'findWait'] as const) {
|
||||
for (const row of [
|
||||
'readText',
|
||||
'readUnique',
|
||||
'readAny',
|
||||
'wait',
|
||||
'findWait',
|
||||
'cropTarget',
|
||||
] as const) {
|
||||
const outcome = await resolveSelectorPipeline(
|
||||
SELECTOR_PIPELINE_POLICIES[row],
|
||||
nodes,
|
||||
@@ -168,7 +176,14 @@ test('the occlusion stage decides refusal: every row that refuses a covered targ
|
||||
}
|
||||
// `readList` is absent by construction: a listing row declares no node
|
||||
// stages, so it has no occlusion verdict to make on a target.
|
||||
for (const row of ['readText', 'readUnique', 'readAny', 'wait', 'findWait'] as const) {
|
||||
for (const row of [
|
||||
'readText',
|
||||
'readUnique',
|
||||
'readAny',
|
||||
'wait',
|
||||
'findWait',
|
||||
'cropTarget',
|
||||
] as const) {
|
||||
const target = await stagedIndex(row, COVERED_TREE, 1);
|
||||
assert.equal(target.kind, 'target', row);
|
||||
assert.equal(target.node.index, 1, row);
|
||||
@@ -179,7 +194,14 @@ test('the promotion stage retargets only for the rows that declare it', async ()
|
||||
// Same tree, same node: the row is the whole difference.
|
||||
assert.equal((await stagedIndex('promotedTarget', PROMOTABLE_TREE, 1)).node.index, 0);
|
||||
assert.equal((await stagedIndex('findAct', PROMOTABLE_TREE, 1)).node.index, 0);
|
||||
for (const row of ['resolvedTarget', 'readText', 'readUnique', 'readAny', 'wait'] as const) {
|
||||
for (const row of [
|
||||
'resolvedTarget',
|
||||
'readText',
|
||||
'readUnique',
|
||||
'readAny',
|
||||
'wait',
|
||||
'cropTarget',
|
||||
] as const) {
|
||||
assert.equal((await stagedIndex(row, PROMOTABLE_TREE, 1)).node.index, 1, row);
|
||||
}
|
||||
});
|
||||
@@ -305,6 +327,7 @@ test('the documented per-caller pipelines are the ones declared', () => {
|
||||
findWait: ['ignore', 'ignore', 'none', 'poll'],
|
||||
wait: ['ignore', 'ignore', 'none', 'poll'],
|
||||
findAct: ['refuse', 'ignore', 'hittable-ancestor-below-root', 'no-poll'],
|
||||
cropTarget: ['ignore', 'ignore', 'none', 'no-poll'],
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -158,6 +158,28 @@ test('screenshot digest tolerates a path-only result with no overlay refs', () =
|
||||
expect(digest).toEqual({ path: '/tmp/s.png', overlayCount: 0, overlayRefs: [] });
|
||||
});
|
||||
|
||||
test('screenshot digest keeps response-level warnings emitted once', () => {
|
||||
const digest = screenshotView!(
|
||||
{
|
||||
path: '/tmp/s.png',
|
||||
width: 40,
|
||||
height: 20,
|
||||
warnings: [
|
||||
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame',
|
||||
],
|
||||
},
|
||||
'digest',
|
||||
);
|
||||
expect(digest).toMatchObject({
|
||||
path: '/tmp/s.png',
|
||||
width: 40,
|
||||
height: 20,
|
||||
warnings: [
|
||||
'CROP_PARTIAL_INTERSECTION: the selector frame extends past the captured image; the crop was clipped to the image frame',
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
// A verbose matched node as it appears on the `find`/`get` wire: the semantic
|
||||
// attributes (kept) plus the geometry/index/process plumbing (the token sink).
|
||||
const MATCHED_NODE = {
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
import {
|
||||
ANDROID_DEVICE,
|
||||
ANDROID_EMULATOR,
|
||||
IOS_DEVICE,
|
||||
IOS_SIMULATOR,
|
||||
LINUX_DEVICE,
|
||||
MACOS_DEVICE,
|
||||
TVOS_SIMULATOR,
|
||||
WEB_DESKTOP_DEVICE,
|
||||
} from '../../__tests__/test-utils/device-fixtures.ts';
|
||||
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
|
||||
import type { SessionSurface } from '@agent-device/contracts/session';
|
||||
import type { DeviceInfo } from '@agent-device/kernel/device';
|
||||
import { expect, test } from 'vitest';
|
||||
import {
|
||||
SCREENSHOT_CROP_TARGET_CELLS,
|
||||
assertScreenshotCropPolicy,
|
||||
classifyScreenshotCropTarget,
|
||||
} from '../screenshot-crop-target.ts';
|
||||
|
||||
type CropTargetDevice = Readonly<{
|
||||
target: (typeof SCREENSHOT_CROP_TARGET_CELLS)[number]['target'];
|
||||
device: DeviceInfo;
|
||||
surface: SessionSurface | undefined;
|
||||
}>;
|
||||
|
||||
/** One device per matrix cell: the classifier's whole reachable output, named by its cell. */
|
||||
const CROP_TARGET_DEVICES: readonly CropTargetDevice[] = [
|
||||
{ target: 'ios-simulator', device: IOS_SIMULATOR, surface: undefined },
|
||||
{ target: 'android-emulator', device: ANDROID_EMULATOR, surface: undefined },
|
||||
{ target: 'android-device', device: ANDROID_DEVICE, surface: undefined },
|
||||
{ target: 'macos-app-window', device: MACOS_DEVICE, surface: 'app' },
|
||||
{ target: 'ios-physical', device: IOS_DEVICE, surface: undefined },
|
||||
{ target: 'macos-helper', device: MACOS_DEVICE, surface: undefined },
|
||||
{ target: 'web', device: WEB_DESKTOP_DEVICE, surface: undefined },
|
||||
{ target: 'linux', device: LINUX_DEVICE, surface: undefined },
|
||||
{ target: 'tvos', device: TVOS_SIMULATOR, surface: undefined },
|
||||
{
|
||||
target: 'harmonyos',
|
||||
device: { platform: 'harmonyos', id: 'hmy-1', name: 'HarmonyOS', kind: 'device' },
|
||||
surface: undefined,
|
||||
},
|
||||
{
|
||||
target: 'vega',
|
||||
device: { platform: 'vega', id: 'vega-1', name: 'Vega TV', kind: 'device', target: 'tv' },
|
||||
surface: undefined,
|
||||
},
|
||||
];
|
||||
|
||||
test('the classifier and the acceptance matrix agree one-to-one, and the accepted cells are exactly the evidenced ones', () => {
|
||||
const matrixTargets = SCREENSHOT_CROP_TARGET_CELLS.map((cell) => cell.target);
|
||||
expect(new Set(matrixTargets).size).toBe(matrixTargets.length);
|
||||
for (const row of CROP_TARGET_DEVICES) {
|
||||
expect(classifyScreenshotCropTarget(row.device, row.surface)).toBe(row.target);
|
||||
}
|
||||
expect(CROP_TARGET_DEVICES.map((row) => row.target)).toEqual(matrixTargets);
|
||||
for (const cell of SCREENSHOT_CROP_TARGET_CELLS) {
|
||||
if (cell.target === 'ios-simulator' || cell.target === 'android-emulator') {
|
||||
expect(cell.status).toBe('accepted');
|
||||
} else {
|
||||
expect(cell).toEqual({
|
||||
target: cell.target,
|
||||
status: 'rejected',
|
||||
rejectionReason: SCREENSHOT_CROP_REASONS.pendingPixelIdentityEvidence,
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('an apple device with an unpopulated reserved OS is a typed refusal, not a guess', () => {
|
||||
const device: DeviceInfo = {
|
||||
platform: 'apple',
|
||||
id: 'watch-1',
|
||||
name: 'Watch',
|
||||
kind: 'simulator',
|
||||
appleOs: 'watchos',
|
||||
};
|
||||
let refusal: unknown;
|
||||
try {
|
||||
classifyScreenshotCropTarget(device, undefined);
|
||||
} catch (error) {
|
||||
refusal = error;
|
||||
}
|
||||
expect(refusal).toMatchObject({
|
||||
code: 'UNSUPPORTED_OPERATION',
|
||||
details: { reason: SCREENSHOT_CROP_REASONS.targetNotAccepted },
|
||||
});
|
||||
});
|
||||
|
||||
function expectPolicyRefusal(
|
||||
params: Readonly<{
|
||||
device: DeviceInfo;
|
||||
surface: SessionSurface | undefined;
|
||||
cropOn: string;
|
||||
overlayRefs: boolean;
|
||||
fullscreen: boolean;
|
||||
}>,
|
||||
expected: Record<string, unknown>,
|
||||
): void {
|
||||
try {
|
||||
assertScreenshotCropPolicy(params);
|
||||
} catch (error) {
|
||||
expect(error).toMatchObject(expected);
|
||||
return;
|
||||
}
|
||||
throw new Error('the crop policy must refuse before device work');
|
||||
}
|
||||
|
||||
const POLICY_BASE = { cropOn: 'label="Save"', overlayRefs: false, fullscreen: false } as const;
|
||||
|
||||
test('combination refusals are answered before selector validation and the matrix', () => {
|
||||
expectPolicyRefusal(
|
||||
{ device: MACOS_DEVICE, surface: 'app', ...POLICY_BASE, overlayRefs: true },
|
||||
{ code: 'INVALID_ARGS', details: { reason: SCREENSHOT_CROP_REASONS.frameMismatch } },
|
||||
);
|
||||
expectPolicyRefusal(
|
||||
{ device: ANDROID_EMULATOR, surface: undefined, ...POLICY_BASE, fullscreen: true },
|
||||
{ code: 'INVALID_ARGS', details: { reason: SCREENSHOT_CROP_REASONS.frameMismatch } },
|
||||
);
|
||||
});
|
||||
|
||||
test('an invalid selector expression is refused with the selector reason', () => {
|
||||
expectPolicyRefusal(
|
||||
{
|
||||
device: ANDROID_EMULATOR,
|
||||
surface: undefined,
|
||||
cropOn: 'label="unterminated',
|
||||
overlayRefs: false,
|
||||
fullscreen: false,
|
||||
},
|
||||
{ code: 'INVALID_ARGS', details: { reason: SCREENSHOT_CROP_REASONS.selectorInvalid } },
|
||||
);
|
||||
});
|
||||
|
||||
test('a matrix-rejected target is refused with the pending-evidence reason', () => {
|
||||
expectPolicyRefusal(
|
||||
{ device: ANDROID_DEVICE, surface: undefined, ...POLICY_BASE },
|
||||
{
|
||||
code: 'UNSUPPORTED_OPERATION',
|
||||
details: {
|
||||
reason: SCREENSHOT_CROP_REASONS.targetNotAccepted,
|
||||
rejectionReason: SCREENSHOT_CROP_REASONS.pendingPixelIdentityEvidence,
|
||||
},
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test('an accepted target with a valid selector passes the policy', () => {
|
||||
expect(() =>
|
||||
assertScreenshotCropPolicy({ device: ANDROID_EMULATOR, surface: undefined, ...POLICY_BASE }),
|
||||
).not.toThrow();
|
||||
});
|
||||
@@ -0,0 +1,294 @@
|
||||
import { ANDROID_EMULATOR, IOS_SIMULATOR } from '../../__tests__/test-utils/device-fixtures.ts';
|
||||
import { makeSession } from '../../__tests__/test-utils/session-factories.ts';
|
||||
import { readPngSize } from '@agent-device/capture-kit/png-size';
|
||||
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
|
||||
import type { SessionSurface } from '@agent-device/contracts/session';
|
||||
import type { SnapshotResult } from '@agent-device/contracts/snapshot-runtime';
|
||||
import type { DeviceInfo } from '@agent-device/kernel/device';
|
||||
import type {
|
||||
RawSnapshotNode,
|
||||
Rect,
|
||||
SnapshotProvenance,
|
||||
SnapshotQualityVerdict,
|
||||
} from '@agent-device/kernel/snapshot';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { expect, test, vi } from 'vitest';
|
||||
import type { SessionState } from '../types.ts';
|
||||
import { buildScreenshotCropWarnings, cropScreenshotToSelector } from '../screenshot-crop.ts';
|
||||
import { writeSolidPng } from './screenshot-runtime-fixture.ts';
|
||||
|
||||
test('the warning composition is the single owner: partial intersection and only', () => {
|
||||
expect(buildScreenshotCropWarnings(undefined)).toEqual([]);
|
||||
expect(buildScreenshotCropWarnings({ partialIntersection: false })).toEqual([]);
|
||||
const warnings = buildScreenshotCropWarnings({ partialIntersection: true });
|
||||
expect(warnings).toHaveLength(1);
|
||||
expect(warnings[0]).toMatch(new RegExp(`^${SCREENSHOT_CROP_REASONS.partialIntersection}: `));
|
||||
});
|
||||
|
||||
const ANDROID_ROOT: RawSnapshotNode = {
|
||||
index: 0,
|
||||
depth: 0,
|
||||
type: 'android.widget.FrameLayout',
|
||||
rect: { x: 0, y: 0, width: 100, height: 50 },
|
||||
};
|
||||
|
||||
function androidTree(saveRect: Rect): RawSnapshotNode[] {
|
||||
return [
|
||||
ANDROID_ROOT,
|
||||
{
|
||||
index: 1,
|
||||
depth: 1,
|
||||
parentIndex: 0,
|
||||
type: 'android.widget.Button',
|
||||
label: 'Save',
|
||||
rect: saveRect,
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
const IOS_TREE: RawSnapshotNode[] = [
|
||||
{
|
||||
index: 0,
|
||||
depth: 0,
|
||||
type: 'XCUIElementTypeApplication',
|
||||
rect: { x: 0, y: 0, width: 390, height: 844 },
|
||||
hittable: true,
|
||||
},
|
||||
{
|
||||
index: 1,
|
||||
depth: 1,
|
||||
parentIndex: 0,
|
||||
type: 'XCUIElementTypeButton',
|
||||
label: 'Save',
|
||||
rect: { x: 10, y: 10, width: 100, height: 40 },
|
||||
hittable: true,
|
||||
},
|
||||
];
|
||||
|
||||
type CropSeamParams = Readonly<{
|
||||
device: DeviceInfo;
|
||||
surface?: SessionSurface;
|
||||
cropOn?: string;
|
||||
nodes: RawSnapshotNode[];
|
||||
png: Readonly<{ width: number; height: number }>;
|
||||
truncated?: boolean;
|
||||
quality?: SnapshotQualityVerdict;
|
||||
provenance: SnapshotProvenance;
|
||||
}>;
|
||||
|
||||
type CropSeam = Readonly<{
|
||||
session: SessionState;
|
||||
screenshotPath: string;
|
||||
captureSnapshot: ReturnType<typeof vi.fn>;
|
||||
run: () => Promise<unknown>;
|
||||
dispose: () => void;
|
||||
}>;
|
||||
|
||||
function cropSeam(params: CropSeamParams): CropSeam {
|
||||
const session = makeSession('default', { device: params.device, surface: params.surface });
|
||||
const screenshotPath = path.join(
|
||||
os.tmpdir(),
|
||||
`agent-device-crop-on-${Date.now()}-${Math.random().toString(36).slice(2)}.png`,
|
||||
);
|
||||
writeSolidPng(screenshotPath, params.png.width, params.png.height);
|
||||
const captureSnapshot = vi.fn(async (): Promise<SnapshotResult> => ({
|
||||
nodes: params.nodes,
|
||||
...params.provenance,
|
||||
...(params.truncated === undefined ? {} : { truncated: params.truncated }),
|
||||
...(params.quality === undefined ? {} : { quality: params.quality }),
|
||||
}));
|
||||
return {
|
||||
session,
|
||||
screenshotPath,
|
||||
captureSnapshot,
|
||||
run: () =>
|
||||
cropScreenshotToSelector({
|
||||
device: params.device,
|
||||
session,
|
||||
surface: params.surface,
|
||||
cropOn: params.cropOn ?? 'label="Save"',
|
||||
screenshotPath,
|
||||
logPath: path.join(os.tmpdir(), 'agent-device-crop-on-daemon.log'),
|
||||
dispatchContext: {},
|
||||
captureSnapshot,
|
||||
}),
|
||||
dispose: () => {
|
||||
fs.rmSync(screenshotPath, { force: true });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
test('an android crop runs the fresh full-tree capture once and leaves the session snapshot untouched', async () => {
|
||||
const seam = cropSeam({
|
||||
device: ANDROID_EMULATOR,
|
||||
nodes: androidTree({ x: 10, y: 10, width: 40, height: 20 }),
|
||||
provenance: { backend: 'android', producer: 'android-uiautomator' },
|
||||
png: { width: 100, height: 50 },
|
||||
});
|
||||
try {
|
||||
const outcome = await seam.run();
|
||||
expect(outcome).toEqual({ partialIntersection: false });
|
||||
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 40, height: 20 });
|
||||
expect(seam.captureSnapshot).toHaveBeenCalledTimes(1);
|
||||
const options = seam.captureSnapshot.mock.calls[0]?.[0].options;
|
||||
expect(options.interactiveOnly).toBe(false);
|
||||
expect(options.includeRects).toBe(true);
|
||||
expect(options.surface).toBeUndefined();
|
||||
expect(options.appBundleId).toBeUndefined();
|
||||
expect(seam.session.snapshot).toBeUndefined();
|
||||
expect(seam.session.snapshotGeneration).toBeUndefined();
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('an iOS simulator crop projects the points-space frame into the 3x capture', async () => {
|
||||
const seam = cropSeam({
|
||||
device: IOS_SIMULATOR,
|
||||
nodes: IOS_TREE,
|
||||
provenance: { backend: 'xctest', producer: 'apple-runner' },
|
||||
png: { width: 1170, height: 2532 },
|
||||
});
|
||||
try {
|
||||
const outcome = await seam.run();
|
||||
expect(outcome).toEqual({ partialIntersection: false });
|
||||
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 300, height: 120 });
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('a frame that runs past the image is clipped and reported partial', async () => {
|
||||
const seam = cropSeam({
|
||||
device: ANDROID_EMULATOR,
|
||||
nodes: androidTree({ x: 80, y: 10, width: 50, height: 20 }),
|
||||
provenance: { backend: 'android', producer: 'android-uiautomator' },
|
||||
png: { width: 100, height: 50 },
|
||||
});
|
||||
try {
|
||||
const outcome = await seam.run();
|
||||
expect(outcome).toEqual({ partialIntersection: true });
|
||||
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 20, height: 20 });
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('a frame that occupies no image pixel refuses without writing', async () => {
|
||||
const seam = cropSeam({
|
||||
device: ANDROID_EMULATOR,
|
||||
nodes: androidTree({ x: 200, y: 100, width: 10, height: 10 }),
|
||||
provenance: { backend: 'android', producer: 'android-uiautomator' },
|
||||
png: { width: 100, height: 50 },
|
||||
});
|
||||
try {
|
||||
await expect(seam.run()).rejects.toMatchObject({
|
||||
code: 'COMMAND_FAILED',
|
||||
details: { reason: SCREENSHOT_CROP_REASONS.emptyIntersection },
|
||||
});
|
||||
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 100, height: 50 });
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('a sparse crop capture is an unreadable refusal, not a missing target', async () => {
|
||||
const seam = cropSeam({
|
||||
device: ANDROID_EMULATOR,
|
||||
nodes: androidTree({ x: 10, y: 10, width: 40, height: 20 }),
|
||||
provenance: { backend: 'android', producer: 'android-uiautomator' },
|
||||
png: { width: 100, height: 50 },
|
||||
quality: { state: 'sparse', backend: 'tree' },
|
||||
});
|
||||
try {
|
||||
await expect(seam.run()).rejects.toMatchObject({
|
||||
code: 'COMMAND_FAILED',
|
||||
details: { reason: SCREENSHOT_CROP_REASONS.captureUnreadable },
|
||||
});
|
||||
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 100, height: 50 });
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('a truncated no-match cannot prove the target absent', async () => {
|
||||
const seam = cropSeam({
|
||||
device: ANDROID_EMULATOR,
|
||||
nodes: [ANDROID_ROOT],
|
||||
cropOn: 'label="Missing"',
|
||||
provenance: { backend: 'android', producer: 'android-uiautomator' },
|
||||
png: { width: 100, height: 50 },
|
||||
truncated: true,
|
||||
});
|
||||
try {
|
||||
await expect(seam.run()).rejects.toMatchObject({
|
||||
code: 'COMMAND_FAILED',
|
||||
details: { reason: SCREENSHOT_CROP_REASONS.captureIncomplete },
|
||||
});
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('a complete no-match is a missing target with the find pointer', async () => {
|
||||
const seam = cropSeam({
|
||||
device: ANDROID_EMULATOR,
|
||||
nodes: [ANDROID_ROOT],
|
||||
cropOn: 'label="Missing"',
|
||||
provenance: { backend: 'android', producer: 'android-uiautomator' },
|
||||
png: { width: 100, height: 50 },
|
||||
});
|
||||
try {
|
||||
await expect(seam.run()).rejects.toMatchObject({
|
||||
code: 'COMMAND_FAILED',
|
||||
details: {
|
||||
reason: SCREENSHOT_CROP_REASONS.targetNotFound,
|
||||
hint: 'find label="Missing" list',
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('an ambiguous match refuses with the candidates, not a first-wins crop', async () => {
|
||||
const seam = cropSeam({
|
||||
device: ANDROID_EMULATOR,
|
||||
nodes: [
|
||||
ANDROID_ROOT,
|
||||
{
|
||||
index: 1,
|
||||
depth: 1,
|
||||
parentIndex: 0,
|
||||
type: 'android.widget.Button',
|
||||
label: 'Save',
|
||||
rect: { x: 5, y: 5, width: 20, height: 10 },
|
||||
},
|
||||
{
|
||||
index: 2,
|
||||
depth: 1,
|
||||
parentIndex: 0,
|
||||
type: 'android.widget.Button',
|
||||
label: 'Save',
|
||||
rect: { x: 60, y: 5, width: 20, height: 10 },
|
||||
},
|
||||
],
|
||||
provenance: { backend: 'android', producer: 'android-uiautomator' },
|
||||
png: { width: 100, height: 50 },
|
||||
});
|
||||
try {
|
||||
await expect(seam.run()).rejects.toMatchObject({
|
||||
code: 'COMMAND_FAILED',
|
||||
details: {
|
||||
reason: SCREENSHOT_CROP_REASONS.targetAmbiguous,
|
||||
candidates: ['Save', 'Save'],
|
||||
matches: 2,
|
||||
},
|
||||
});
|
||||
expect(await readPngSize(seam.screenshotPath)).toEqual({ width: 100, height: 50 });
|
||||
} finally {
|
||||
seam.dispose();
|
||||
}
|
||||
});
|
||||
@@ -2,6 +2,8 @@ import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { expect, test } from 'vitest';
|
||||
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
|
||||
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
|
||||
import { ANDROID_EMULATOR, IOS_SIMULATOR } from '../../__tests__/test-utils/device-fixtures.ts';
|
||||
import { makeSession } from '../../__tests__/test-utils/session-factories.ts';
|
||||
import type { GenericPlatformExecutionParams } from '../request-generic-dispatch.ts';
|
||||
@@ -28,6 +30,7 @@ function screenshotRequest(overrides: Partial<DaemonRequest> = {}): DaemonReques
|
||||
function executionParams(
|
||||
session: SessionState,
|
||||
req: DaemonRequest,
|
||||
dispatchContext: GenericPlatformExecutionParams['dispatchContext'] = {},
|
||||
): GenericPlatformExecutionParams {
|
||||
return {
|
||||
session,
|
||||
@@ -37,10 +40,22 @@ function executionParams(
|
||||
request: req,
|
||||
positionals: req.positionals ?? [],
|
||||
out: req.flags?.out,
|
||||
dispatchContext: {},
|
||||
dispatchContext,
|
||||
};
|
||||
}
|
||||
|
||||
async function executeResult(
|
||||
resolved: Awaited<ReturnType<typeof resolveScreenshotGenericExecution>>,
|
||||
session: SessionState,
|
||||
req: DaemonRequest,
|
||||
dispatchContext?: GenericPlatformExecutionParams['dispatchContext'],
|
||||
): Promise<Record<string, unknown>> {
|
||||
if (!resolved.ok) throw new Error('the screenshot plan must be admitted');
|
||||
const result = await resolved.execute(executionParams(session, req, dispatchContext));
|
||||
if (result === undefined) throw new Error('the screenshot leaf returns a result record');
|
||||
return result;
|
||||
}
|
||||
|
||||
test('admits one capture plan, binds once, and hands the runtime the resolved destination', async () => {
|
||||
const fixture = screenshotRuntimeFixture();
|
||||
const session = makeSession('default', { device: ANDROID_EMULATOR });
|
||||
@@ -196,3 +211,128 @@ test('rejects --pixel-density outside iOS-family simulators before inspecting ow
|
||||
).rejects.toMatchObject({ code: 'UNSUPPORTED_OPERATION' });
|
||||
expect(fixture.binds).toHaveLength(0);
|
||||
});
|
||||
|
||||
const CROP_TREE_ROOT = {
|
||||
index: 0,
|
||||
depth: 0,
|
||||
type: 'android.widget.FrameLayout',
|
||||
rect: { x: 0, y: 0, width: 100, height: 50 },
|
||||
} satisfies RawSnapshotNode;
|
||||
|
||||
const CROP_TREE = [
|
||||
CROP_TREE_ROOT,
|
||||
{
|
||||
index: 1,
|
||||
depth: 1,
|
||||
parentIndex: 0,
|
||||
type: 'android.widget.Button',
|
||||
label: 'Save',
|
||||
rect: { x: 10, y: 10, width: 40, height: 20 },
|
||||
},
|
||||
];
|
||||
|
||||
test('the crop runs after the platform write and before the shared scale', async () => {
|
||||
const fixture = screenshotRuntimeFixture({
|
||||
snapshotResult: () => ({
|
||||
nodes: CROP_TREE,
|
||||
backend: 'android',
|
||||
producer: 'android-uiautomator',
|
||||
}),
|
||||
});
|
||||
const session = makeSession('default', { device: ANDROID_EMULATOR });
|
||||
const outPath = path.join(os.tmpdir(), `agent-device-crop-order-${Date.now()}.png`);
|
||||
const req = screenshotRequest({
|
||||
positionals: [outPath],
|
||||
flags: { screenshotCropOn: 'label="Save"', screenshotScale: 0.5 },
|
||||
});
|
||||
|
||||
const resolved = await resolveScreenshotGenericExecution({
|
||||
req,
|
||||
session,
|
||||
inspectFacts: fixture.inspectFacts,
|
||||
bindDevice: fixture.bindDevice,
|
||||
});
|
||||
const result = await executeResult(resolved, session, req, { screenshotScale: 0.5 });
|
||||
|
||||
expect(fixture.captureScreenshot).toHaveBeenCalledTimes(1);
|
||||
expect(fixture.captureSnapshot).toHaveBeenCalledTimes(1);
|
||||
// The platform wrote 100x50; the pre-scale box is 40x20, so only a crop-then-scale pipeline
|
||||
// lands on 20x10 — a scale-then-crop pipeline would halve the image first and the same box
|
||||
// would overflow it.
|
||||
expect(result).toMatchObject({ width: 20, height: 10 });
|
||||
expect(fixture.captureSnapshot.mock.calls[0]?.[0].options).toMatchObject({
|
||||
interactiveOnly: false,
|
||||
includeRects: true,
|
||||
});
|
||||
});
|
||||
|
||||
test('a partial crop surfaces its warning once in the result record and annotates nothing', async () => {
|
||||
const fixture = screenshotRuntimeFixture({
|
||||
snapshotResult: () => ({
|
||||
nodes: [
|
||||
CROP_TREE_ROOT,
|
||||
{
|
||||
index: 1,
|
||||
depth: 1,
|
||||
parentIndex: 0,
|
||||
type: 'android.widget.Button',
|
||||
label: 'Save',
|
||||
rect: { x: 80, y: 10, width: 50, height: 20 },
|
||||
},
|
||||
],
|
||||
backend: 'android',
|
||||
producer: 'android-uiautomator',
|
||||
}),
|
||||
});
|
||||
const session = makeSession('default', { device: ANDROID_EMULATOR });
|
||||
const outPath = path.join(os.tmpdir(), `agent-device-crop-warning-${Date.now()}.png`);
|
||||
const req = screenshotRequest({
|
||||
positionals: [outPath],
|
||||
flags: { screenshotCropOn: 'label="Save"' },
|
||||
});
|
||||
|
||||
const resolved = await resolveScreenshotGenericExecution({
|
||||
req,
|
||||
session,
|
||||
inspectFacts: fixture.inspectFacts,
|
||||
bindDevice: fixture.bindDevice,
|
||||
});
|
||||
const result = await executeResult(resolved, session, req);
|
||||
|
||||
expect(result.warnings).toEqual([
|
||||
expect.stringMatching(new RegExp(`^${SCREENSHOT_CROP_REASONS.partialIntersection}:`)),
|
||||
]);
|
||||
// The crop plan binds the snapshot for the crop alone: no overlay refs, no republished tree.
|
||||
expect(result).not.toHaveProperty('overlayRefs');
|
||||
expect(session.snapshot).toBeUndefined();
|
||||
});
|
||||
|
||||
test('the crop plan is refused up front when the target cannot capture a tree', async () => {
|
||||
const fixture = screenshotRuntimeFixture({
|
||||
snapshot: { available: false, reason: 'unsupported-platform-leaf' },
|
||||
});
|
||||
const session = makeSession('default', { device: ANDROID_EMULATOR });
|
||||
|
||||
const resolved = await resolveScreenshotGenericExecution({
|
||||
req: screenshotRequest({
|
||||
positionals: ['/tmp/crop.png'],
|
||||
flags: { screenshotCropOn: 'label="Save"' },
|
||||
}),
|
||||
session,
|
||||
inspectFacts: fixture.inspectFacts,
|
||||
bindDevice: fixture.bindDevice,
|
||||
});
|
||||
|
||||
expect(resolved.ok).toBe(false);
|
||||
if (resolved.ok) return;
|
||||
expect(resolved.response).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
code: 'UNSUPPORTED_OPERATION',
|
||||
details: { reason: 'unsupported-platform-leaf' },
|
||||
hint: 'Re-run screenshot without --crop-on.',
|
||||
},
|
||||
});
|
||||
expect(fixture.binds).toHaveLength(0);
|
||||
expect(fixture.captureScreenshot).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
@@ -76,6 +76,7 @@ function screenshotView(data: DaemonResponseData, level: ResponseLevel): DaemonR
|
||||
...pickScreenshotDigestMetadata(data),
|
||||
overlayCount: overlays.length,
|
||||
overlayRefs,
|
||||
...(data.warnings !== undefined ? { warnings: data.warnings } : {}),
|
||||
...(data.artifacts !== undefined ? { artifacts: data.artifacts } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
import {
|
||||
SCREENSHOT_CROP_REASONS,
|
||||
type ScreenshotCropReason,
|
||||
} from '@agent-device/contracts/capture';
|
||||
import type { SessionSurface } from '@agent-device/contracts/session';
|
||||
import { resolveDeviceAppleOs, type DeviceInfo } from '@agent-device/kernel/device';
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import { validateSelectorExpression } from '@agent-device/selectors';
|
||||
|
||||
/**
|
||||
* The `--crop-on` target policy: which device capture frame the crop is accepted on, and the
|
||||
* argument policy answered before any device work. The crop leaf (screenshot-crop.ts) runs the
|
||||
* crop; this module decides whether it may run at all.
|
||||
*/
|
||||
|
||||
type CropTarget =
|
||||
| 'ios-simulator'
|
||||
| 'android-emulator'
|
||||
| 'android-device'
|
||||
| 'macos-app-window'
|
||||
| 'ios-physical'
|
||||
| 'macos-helper'
|
||||
| 'web'
|
||||
| 'linux'
|
||||
| 'tvos'
|
||||
| 'harmonyos'
|
||||
| 'vega';
|
||||
|
||||
type CropTargetCell =
|
||||
| Readonly<{ target: CropTarget; status: 'accepted' }>
|
||||
| Readonly<{
|
||||
target: CropTarget;
|
||||
status: 'rejected';
|
||||
rejectionReason: ScreenshotCropReason;
|
||||
}>;
|
||||
|
||||
/**
|
||||
* The machine-readable support policy. A cell is accepted only once the live pixel-identity
|
||||
* cross-check against an independent capture has been collected; until then the target is
|
||||
* rejected with the pending-evidence reason, and the evidence itself lives in the completeness
|
||||
* gate, the geometry fixtures, and the PR history — never in this table.
|
||||
*/
|
||||
const PENDING = SCREENSHOT_CROP_REASONS.pendingPixelIdentityEvidence;
|
||||
export const SCREENSHOT_CROP_TARGET_CELLS: readonly CropTargetCell[] = [
|
||||
{ target: 'ios-simulator', status: 'accepted' },
|
||||
{ target: 'android-emulator', status: 'accepted' },
|
||||
{ target: 'android-device', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'macos-app-window', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'ios-physical', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'macos-helper', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'web', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'linux', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'tvos', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'harmonyos', status: 'rejected', rejectionReason: PENDING },
|
||||
{ target: 'vega', status: 'rejected', rejectionReason: PENDING },
|
||||
];
|
||||
|
||||
/** The one target cell a device's capture frame falls into. */
|
||||
export function classifyScreenshotCropTarget(
|
||||
device: DeviceInfo,
|
||||
surface: SessionSurface | undefined,
|
||||
): CropTarget {
|
||||
switch (device.platform) {
|
||||
case 'apple':
|
||||
return classifyAppleCropTarget(device, surface);
|
||||
case 'android':
|
||||
return device.kind === 'device' ? 'android-device' : 'android-emulator';
|
||||
case 'harmonyos':
|
||||
return 'harmonyos';
|
||||
case 'vega':
|
||||
return 'vega';
|
||||
case 'linux':
|
||||
return 'linux';
|
||||
case 'web':
|
||||
return 'web';
|
||||
}
|
||||
}
|
||||
|
||||
function classifyAppleCropTarget(
|
||||
device: DeviceInfo,
|
||||
surface: SessionSurface | undefined,
|
||||
): CropTarget {
|
||||
const appleOs = resolveDeviceAppleOs(device);
|
||||
switch (appleOs) {
|
||||
case 'ios':
|
||||
case 'ipados':
|
||||
return device.kind === 'device' ? 'ios-physical' : 'ios-simulator';
|
||||
case 'tvos':
|
||||
return 'tvos';
|
||||
case 'macos':
|
||||
return classifyMacOsCropTarget(surface);
|
||||
case 'watchos':
|
||||
case 'visionos':
|
||||
// Stored records carry the reserved OSes although discovery never populates them; a
|
||||
// session device on one has no acceptance cell to fall into, so it is a typed refusal,
|
||||
// not a guess.
|
||||
throw cropRefusal(appleOs);
|
||||
}
|
||||
}
|
||||
|
||||
function classifyMacOsCropTarget(surface: SessionSurface | undefined): CropTarget {
|
||||
return surface === 'app' || surface === 'frontmost-app' ? 'macos-app-window' : 'macos-helper';
|
||||
}
|
||||
|
||||
function cropRefusal(target: string, rejectionReason?: ScreenshotCropReason): AppError {
|
||||
return new AppError(
|
||||
'UNSUPPORTED_OPERATION',
|
||||
`screenshot --crop-on is not accepted on ${target} targets`,
|
||||
{
|
||||
reason: SCREENSHOT_CROP_REASONS.targetNotAccepted,
|
||||
...(rejectionReason ? { rejectionReason } : {}),
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* The crop argument policy, answered before any device work: combination refusals, the
|
||||
* selector expression, and the acceptance matrix.
|
||||
*/
|
||||
export function assertScreenshotCropPolicy(params: {
|
||||
device: DeviceInfo;
|
||||
surface: SessionSurface | undefined;
|
||||
cropOn: string;
|
||||
overlayRefs: boolean;
|
||||
fullscreen: boolean;
|
||||
}): void {
|
||||
if (params.overlayRefs || params.fullscreen) {
|
||||
throw new AppError(
|
||||
'INVALID_ARGS',
|
||||
'--crop-on cannot be combined with --overlay-refs or --fullscreen: both move the captured frame away from the snapshot viewport the crop is measured against',
|
||||
{ reason: SCREENSHOT_CROP_REASONS.frameMismatch },
|
||||
);
|
||||
}
|
||||
try {
|
||||
validateSelectorExpression(params.cropOn);
|
||||
} catch {
|
||||
throw new AppError(
|
||||
'INVALID_ARGS',
|
||||
'screenshot --crop-on requires a valid selector expression',
|
||||
{ reason: SCREENSHOT_CROP_REASONS.selectorInvalid },
|
||||
);
|
||||
}
|
||||
const target = classifyScreenshotCropTarget(params.device, params.surface);
|
||||
const cell = SCREENSHOT_CROP_TARGET_CELLS.find((candidate) => candidate.target === target);
|
||||
if (!cell || cell.status === 'rejected') {
|
||||
throw cropRefusal(target, cell?.status === 'rejected' ? cell.rejectionReason : undefined);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import type { CommandFlags } from '@agent-device/contracts/command';
|
||||
import { SCREENSHOT_CROP_REASONS } from '@agent-device/contracts/capture';
|
||||
import type { SessionSurface } from '@agent-device/contracts/session';
|
||||
import { publicPlatformString, type DeviceInfo } from '@agent-device/kernel/device';
|
||||
import { AppError } from '@agent-device/kernel/errors';
|
||||
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
|
||||
import { readPngSize } from '@agent-device/capture-kit/png-size';
|
||||
import { cropPngFile } from '@agent-device/capture-kit/png-crop';
|
||||
import {
|
||||
intersectScreenshotRect,
|
||||
projectSnapshotRectToScreenshot,
|
||||
resolveScreenshotRectSpace,
|
||||
resolveSnapshotBounds,
|
||||
} from '@agent-device/capture-kit/snapshot-rect-projection';
|
||||
import { buildSnapshotState } from '../core/snapshot-state.ts';
|
||||
import { SELECTOR_PIPELINE_POLICIES } from '../core/selector-pipeline-policy.ts';
|
||||
import { resolveSelectorPipeline } from '../core/selector-pipeline.ts';
|
||||
import type { DaemonCommandContext } from './context.ts';
|
||||
import { captureSnapshotData } from './snapshot-capture.ts';
|
||||
import { runtimeExecutionFromContext } from './snapshot-runtime-capture-input.ts';
|
||||
import type { BoundScreenshotRuntime } from './screenshot-runtime-binding.ts';
|
||||
import type { SessionState } from './types.ts';
|
||||
|
||||
export type ScreenshotCropOutcome = Readonly<{ partialIntersection: boolean }>;
|
||||
|
||||
const CROP_PARTIAL_INTERSECTION_WARNING = `${SCREENSHOT_CROP_REASONS.partialIntersection}: the selector frame extends past the captured image; the crop was clipped to the image frame`;
|
||||
|
||||
/** The single warning-composition owner: append-only, typed reason, stable text. */
|
||||
export function buildScreenshotCropWarnings(outcome: ScreenshotCropOutcome | undefined): string[] {
|
||||
return outcome?.partialIntersection ? [CROP_PARTIAL_INTERSECTION_WARNING] : [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Crops `screenshotPath` to the frame of the selector resolved against a fresh full-tree
|
||||
* snapshot taken on the same screen, through the same admitted binding as the capture. The crop
|
||||
* snapshot is request-scoped — it never becomes the session snapshot.
|
||||
*/
|
||||
export async function cropScreenshotToSelector(params: {
|
||||
device: DeviceInfo;
|
||||
session: SessionState;
|
||||
surface: SessionSurface | undefined;
|
||||
cropOn: string;
|
||||
screenshotPath: string;
|
||||
logPath: string;
|
||||
dispatchContext: DaemonCommandContext;
|
||||
captureSnapshot: NonNullable<BoundScreenshotRuntime['captureSnapshot']>;
|
||||
}): Promise<ScreenshotCropOutcome> {
|
||||
const { device, session, surface, cropOn, screenshotPath, logPath, dispatchContext } = params;
|
||||
const flags = { snapshotInteractiveOnly: false } satisfies CommandFlags;
|
||||
const snapshotData = await captureSnapshotData({
|
||||
device,
|
||||
session,
|
||||
flags,
|
||||
logPath,
|
||||
snapshotScope: undefined,
|
||||
captureData: async () =>
|
||||
await params.captureSnapshot({
|
||||
options: {
|
||||
appBundleId: dispatchContext.appBundleId,
|
||||
interactiveOnly: false,
|
||||
includeRects: true,
|
||||
surface,
|
||||
},
|
||||
execution: runtimeExecutionFromContext(dispatchContext),
|
||||
}),
|
||||
});
|
||||
const snapshot = buildSnapshotState(snapshotData, flags);
|
||||
|
||||
if (snapshot.snapshotQuality?.state === 'sparse') {
|
||||
throw new AppError(
|
||||
'COMMAND_FAILED',
|
||||
'the snapshot taken for --crop-on was sparse and cannot be read as the screen',
|
||||
{ reason: SCREENSHOT_CROP_REASONS.captureUnreadable },
|
||||
);
|
||||
}
|
||||
|
||||
const node = await resolveCropTargetNode(snapshot.nodes, cropOn, snapshot.truncated, device);
|
||||
const image = await readPngSize(screenshotPath);
|
||||
const projected = projectSnapshotRectToScreenshot(
|
||||
resolveScreenshotRectSpace(snapshot.backend),
|
||||
resolveSnapshotBounds(snapshot.nodes),
|
||||
node.rect!,
|
||||
image.width,
|
||||
image.height,
|
||||
);
|
||||
const box = intersectScreenshotRect(projected, image.width, image.height);
|
||||
if (box === null) {
|
||||
throw new AppError(
|
||||
'COMMAND_FAILED',
|
||||
`the frame resolved by --crop-on occupies no pixel of the capture`,
|
||||
{ reason: SCREENSHOT_CROP_REASONS.emptyIntersection },
|
||||
);
|
||||
}
|
||||
const partialIntersection =
|
||||
box.x !== projected.x ||
|
||||
box.y !== projected.y ||
|
||||
box.width < projected.width ||
|
||||
box.height < projected.height;
|
||||
await cropPngFile(screenshotPath, box);
|
||||
return { partialIntersection };
|
||||
}
|
||||
|
||||
/** Resolve the selector to exactly one framed node, or fail with a typed crop reason. */
|
||||
async function resolveCropTargetNode(
|
||||
nodes: SnapshotNode[],
|
||||
cropOn: string,
|
||||
truncated: boolean | undefined,
|
||||
device: DeviceInfo,
|
||||
) {
|
||||
const outcome = await resolveSelectorPipeline(
|
||||
SELECTOR_PIPELINE_POLICIES.cropTarget,
|
||||
nodes,
|
||||
cropOn,
|
||||
{ platform: publicPlatformString(device) },
|
||||
);
|
||||
if (outcome.kind === 'ambiguous') {
|
||||
throw new AppError(
|
||||
'COMMAND_FAILED',
|
||||
`--crop-on matched ${outcome.matchedNodes.length} nodes and must resolve to one`,
|
||||
{
|
||||
reason: SCREENSHOT_CROP_REASONS.targetAmbiguous,
|
||||
candidates: outcome.matchedNodes.map((node) => node.label ?? node.ref),
|
||||
matches: outcome.matchedNodes.length,
|
||||
},
|
||||
);
|
||||
}
|
||||
if (outcome.kind === 'none') {
|
||||
if (truncated) {
|
||||
throw new AppError(
|
||||
'COMMAND_FAILED',
|
||||
'the snapshot taken for --crop-on was truncated, so a missing match is not proven',
|
||||
{ reason: SCREENSHOT_CROP_REASONS.captureIncomplete },
|
||||
);
|
||||
}
|
||||
throw new AppError('COMMAND_FAILED', `--crop-on matched no node: ${cropOn}`, {
|
||||
reason: SCREENSHOT_CROP_REASONS.targetNotFound,
|
||||
hint: `find ${cropOn} list`,
|
||||
});
|
||||
}
|
||||
return outcome.node;
|
||||
}
|
||||
@@ -7,9 +7,14 @@ import {
|
||||
type SnapshotState,
|
||||
} from '@agent-device/kernel/snapshot';
|
||||
import { decodePngAsync, encodePngAsync } from '@agent-device/capture-kit/png-worker-client';
|
||||
import {
|
||||
projectSnapshotRectToScreenshot,
|
||||
resolveSnapshotBounds,
|
||||
} from '@agent-device/capture-kit/snapshot-rect-projection';
|
||||
import { analyzeReactNativeOverlay } from '../core/react-native-overlay.ts';
|
||||
import {
|
||||
findNearestAncestor,
|
||||
isMeaningfulSignal,
|
||||
isViewportRootNode,
|
||||
normalizeType,
|
||||
} from '@agent-device/contracts/snapshot';
|
||||
@@ -259,69 +264,14 @@ function projectRectToScreenshot(
|
||||
screenshotWidth: number,
|
||||
screenshotHeight: number,
|
||||
): Rect {
|
||||
if (snapshot.backend === 'android') {
|
||||
return clampRect(roundRect(rect), screenshotWidth, screenshotHeight);
|
||||
}
|
||||
if (!bounds) {
|
||||
return clampRect(roundRect(rect), screenshotWidth, screenshotHeight);
|
||||
}
|
||||
const scaleX = screenshotWidth / bounds.width;
|
||||
const scaleY = screenshotHeight / bounds.height;
|
||||
const space = snapshot.backend === 'android' ? 'device-pixels' : 'viewport-points';
|
||||
return clampRect(
|
||||
{
|
||||
x: Math.round((rect.x - bounds.x) * scaleX),
|
||||
y: Math.round((rect.y - bounds.y) * scaleY),
|
||||
width: Math.max(1, Math.round(rect.width * scaleX)),
|
||||
height: Math.max(1, Math.round(rect.height * scaleY)),
|
||||
},
|
||||
projectSnapshotRectToScreenshot(space, bounds, rect, screenshotWidth, screenshotHeight),
|
||||
screenshotWidth,
|
||||
screenshotHeight,
|
||||
);
|
||||
}
|
||||
|
||||
function resolveSnapshotBounds(nodes: SnapshotState['nodes']): Rect | null {
|
||||
let viewport: Rect | null = null;
|
||||
for (const node of nodes) {
|
||||
if (!isViewportRootNode(node) || !hasPositiveRect(node.rect)) continue;
|
||||
if (!viewport || rectArea(node.rect) > rectArea(viewport)) {
|
||||
viewport = node.rect;
|
||||
}
|
||||
}
|
||||
if (viewport) return viewport;
|
||||
|
||||
return measureSnapshotBounds(
|
||||
nodes.filter((node) => hasPositiveRect(node.rect) && !isSnapshotBoundsOutlier(node)),
|
||||
);
|
||||
}
|
||||
|
||||
function measureSnapshotBounds(nodes: Array<Pick<SnapshotNode, 'rect'>>): Rect | null {
|
||||
let minX = Number.POSITIVE_INFINITY;
|
||||
let minY = Number.POSITIVE_INFINITY;
|
||||
let maxRight = Number.NEGATIVE_INFINITY;
|
||||
let maxBottom = Number.NEGATIVE_INFINITY;
|
||||
for (const node of nodes) {
|
||||
if (!node.rect || !hasPositiveRect(node.rect)) continue;
|
||||
minX = Math.min(minX, node.rect.x);
|
||||
minY = Math.min(minY, node.rect.y);
|
||||
maxRight = Math.max(maxRight, node.rect.x + node.rect.width);
|
||||
maxBottom = Math.max(maxBottom, node.rect.y + node.rect.height);
|
||||
}
|
||||
if (!Number.isFinite(minX) || !Number.isFinite(minY) || maxRight <= minX || maxBottom <= minY) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
x: minX,
|
||||
y: minY,
|
||||
width: maxRight - minX,
|
||||
height: maxBottom - minY,
|
||||
};
|
||||
}
|
||||
|
||||
function isSnapshotBoundsOutlier(node: SnapshotNode): boolean {
|
||||
const normalizedType = normalizeType(node.type ?? '');
|
||||
return normalizedType === 'image' && !isMeaningfulSignal(node.label);
|
||||
}
|
||||
|
||||
function hasActionableRole(node: SnapshotNode): boolean {
|
||||
const roleText = [node.type, node.role, node.subrole]
|
||||
.map((value) => normalizeType(value ?? ''))
|
||||
@@ -347,14 +297,6 @@ function isUsableOverlayTarget(node: SnapshotNode | null): node is SnapshotNode
|
||||
return Boolean(node?.rect && hasPositiveRect(node.rect) && !isViewportRootNode(node));
|
||||
}
|
||||
|
||||
function isMeaningfulSignal(value: string | undefined): boolean {
|
||||
if (typeof value !== 'string') return false;
|
||||
const trimmed = value.trim();
|
||||
if (!trimmed) return false;
|
||||
if (/^(true|false)$/i.test(trimmed)) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
function isOverlaySignal(value: string | undefined): boolean {
|
||||
if (!isMeaningfulSignal(value)) return false;
|
||||
return !isGenericOverlayLabel(value);
|
||||
@@ -433,15 +375,6 @@ function compareNumericRefs(left: string, right: string): number {
|
||||
return leftValue - rightValue;
|
||||
}
|
||||
|
||||
function roundRect(rect: Rect): Rect {
|
||||
return {
|
||||
x: Math.round(rect.x),
|
||||
y: Math.round(rect.y),
|
||||
width: Math.round(rect.width),
|
||||
height: Math.round(rect.height),
|
||||
};
|
||||
}
|
||||
|
||||
function clampRect(rect: Rect, width: number, height: number): Rect {
|
||||
const x = clamp(rect.x, 0, Math.max(0, width - 1));
|
||||
const y = clamp(rect.y, 0, Math.max(0, height - 1));
|
||||
|
||||
@@ -30,9 +30,9 @@ export type ScreenshotRuntimeBindings = Readonly<{
|
||||
}>;
|
||||
|
||||
/**
|
||||
* One request's admitted capture authority. `captureSnapshot` is present exactly when the
|
||||
* overlay-refs plan was admitted, so a caller cannot annotate a capture it never declared — and
|
||||
* both operations come from the same single binding.
|
||||
* One request's admitted capture authority. `captureSnapshot` is present exactly when a
|
||||
* tree-requiring plan (overlay-refs or crop-on) was admitted, so a caller cannot post-process a
|
||||
* capture it never declared a snapshot for — and both operations come from the same single binding.
|
||||
*/
|
||||
export type BoundScreenshotRuntime = Readonly<{
|
||||
captureScreenshot(input: CaptureScreenshotInput): Promise<void>;
|
||||
@@ -45,9 +45,13 @@ export type ResolvedScreenshotRuntime =
|
||||
|
||||
/** Resolves one plan, inspects its owner facts once, then binds once on the admitted device. */
|
||||
export async function resolveBoundScreenshotRuntime(
|
||||
params: Readonly<{ device: DeviceInfo; overlayRefs: boolean }> & ScreenshotRuntimeBindings,
|
||||
params: Readonly<{ device: DeviceInfo; overlayRefs: boolean; cropOn?: string }> &
|
||||
ScreenshotRuntimeBindings,
|
||||
): Promise<ResolvedScreenshotRuntime> {
|
||||
const plan = resolveScreenshotRuntimePlan({ overlayRefs: params.overlayRefs });
|
||||
const plan = resolveScreenshotRuntimePlan({
|
||||
overlayRefs: params.overlayRefs,
|
||||
cropOn: params.cropOn !== undefined,
|
||||
});
|
||||
const admission = await admitRuntimePlan({
|
||||
device: params.device,
|
||||
plan,
|
||||
@@ -56,7 +60,7 @@ export async function resolveBoundScreenshotRuntime(
|
||||
if (!admission.admitted) {
|
||||
return {
|
||||
ok: false,
|
||||
response: screenshotPlanUnavailableResponse(admission.operation, admission.fact),
|
||||
response: screenshotPlanUnavailableResponse(plan.kind, admission.operation, admission.fact),
|
||||
};
|
||||
}
|
||||
return { ok: true, runtime: await bindScreenshotRuntime(params.bindDevice, admission) };
|
||||
@@ -78,11 +82,12 @@ async function bindScreenshotRuntime(
|
||||
const runtime = await bind(device, plan.use);
|
||||
return Object.freeze({ captureScreenshot: selectScreenshotCapture(runtime) });
|
||||
}
|
||||
case 'capture-with-overlay-refs': {
|
||||
case 'capture-with-overlay-refs':
|
||||
case 'capture-with-crop-on': {
|
||||
const runtime = await bind(device, plan.use);
|
||||
return Object.freeze({
|
||||
captureScreenshot: selectScreenshotCapture(runtime),
|
||||
captureSnapshot: selectOverlaySnapshot(runtime),
|
||||
captureSnapshot: selectSnapshotCapture(runtime),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -97,20 +102,31 @@ function selectScreenshotCapture(runtime: BoundScreenshotOperation<'captureScree
|
||||
return async (input: CaptureScreenshotInput) => await runtime.operations.captureScreenshot(input);
|
||||
}
|
||||
|
||||
/** The overlay plan's tree read, from the same binding as its capture. */
|
||||
function selectOverlaySnapshot(
|
||||
/** The tree-requiring plans' snapshot read, from the same binding as the capture. */
|
||||
function selectSnapshotCapture(
|
||||
runtime: Readonly<{ operations: Readonly<Pick<SnapshotRuntimeOperations, 'captureSnapshot'>> }>,
|
||||
) {
|
||||
return async (input: CaptureSnapshotInput) => await runtime.operations.captureSnapshot(input);
|
||||
}
|
||||
|
||||
function screenshotPlanUnavailableResponse(
|
||||
kind: ScreenshotRuntimePlan['kind'],
|
||||
operation: ScreenshotRuntimePlan['use']['required'][number],
|
||||
fact: RuntimeOperationFact,
|
||||
): DaemonResponse {
|
||||
if (operation === 'captureScreenshot') {
|
||||
return unavailableRuntimeOperationResponse('screenshot', fact)!;
|
||||
}
|
||||
if (kind === 'capture-with-crop-on') {
|
||||
return errorResponse(
|
||||
'UNSUPPORTED_OPERATION',
|
||||
'--crop-on crops the capture to a selector frame using a snapshot taken on the same screen, which this target cannot capture.',
|
||||
{ reason: fact.available ? undefined : fact.reason },
|
||||
{
|
||||
hint: (fact.available ? undefined : fact.hint) ?? 'Re-run screenshot without --crop-on.',
|
||||
},
|
||||
);
|
||||
}
|
||||
return errorResponse(
|
||||
'UNSUPPORTED_OPERATION',
|
||||
'--overlay-refs annotates a capture with the refs of a snapshot taken on the same screen, which this target cannot capture.',
|
||||
|
||||
@@ -27,6 +27,8 @@ import type {
|
||||
ResolvedGenericExecution,
|
||||
} from './request-generic-dispatch.ts';
|
||||
import { createDaemonRuntimeSessionStore } from './runtime-session.ts';
|
||||
import { assertScreenshotCropPolicy } from './screenshot-crop-target.ts';
|
||||
import { buildScreenshotCropWarnings, cropScreenshotToSelector } from './screenshot-crop.ts';
|
||||
import { annotateScreenshotWithRefs } from './screenshot-overlay.ts';
|
||||
import {
|
||||
resolveBoundScreenshotRuntime,
|
||||
@@ -55,15 +57,30 @@ export async function resolveScreenshotGenericExecution(
|
||||
assertSupportedScreenshotPixelDensity(session.device, req.flags?.screenshotPixelDensity);
|
||||
|
||||
const request = readScreenshotRequest(req);
|
||||
const cropOn =
|
||||
typeof req.flags?.screenshotCropOn === 'string' && req.flags.screenshotCropOn.length > 0
|
||||
? req.flags.screenshotCropOn
|
||||
: undefined;
|
||||
if (cropOn !== undefined) {
|
||||
assertScreenshotCropPolicy({
|
||||
device: session.device,
|
||||
surface: session.surface,
|
||||
cropOn,
|
||||
overlayRefs: req.flags?.overlayRefs === true,
|
||||
fullscreen: req.flags?.screenshotFullscreen === true,
|
||||
});
|
||||
}
|
||||
const resolved = await resolveBoundScreenshotRuntime({
|
||||
device: session.device,
|
||||
overlayRefs: req.flags?.overlayRefs === true,
|
||||
cropOn,
|
||||
inspectFacts: params.inspectFacts,
|
||||
bindDevice: params.bindDevice,
|
||||
});
|
||||
if (!resolved.ok) return resolved;
|
||||
|
||||
const runtime = resolved.runtime;
|
||||
const cropRun: ScreenshotCropRun = { warnings: [] };
|
||||
return {
|
||||
ok: true,
|
||||
recorded: request.recorded,
|
||||
@@ -76,6 +93,8 @@ export async function resolveScreenshotGenericExecution(
|
||||
flags: execution.request.flags,
|
||||
outPath: request.outPath,
|
||||
runtime,
|
||||
cropOn,
|
||||
cropRun,
|
||||
}),
|
||||
};
|
||||
}
|
||||
@@ -92,6 +111,7 @@ export async function captureScreenshotArtifact(
|
||||
outPath?: string;
|
||||
dispatchContext: DaemonCommandContext;
|
||||
captureScreenshot: BoundScreenshotRuntime['captureScreenshot'];
|
||||
crop?: ScreenshotCropBinding;
|
||||
}>,
|
||||
): Promise<CapturedScreenshot> {
|
||||
const { session, sessionName, outPath, dispatchContext } = params;
|
||||
@@ -122,7 +142,18 @@ export async function captureScreenshotArtifact(
|
||||
* `commands/`: the daemon sits below the command surface (R2), and this adapter's artifact
|
||||
* publisher emits no descriptors, so the destination and its message are the whole result.
|
||||
*/
|
||||
type CapturedScreenshot = Readonly<{ path: string; message?: string }>;
|
||||
type CapturedScreenshot = Readonly<{ path: string; message?: string; warnings?: string[] }>;
|
||||
|
||||
/** One request's crop state: the backend closure appends, the result record reads. */
|
||||
type ScreenshotCropRun = { warnings: string[] };
|
||||
|
||||
/** The crop orchestration the backend closure runs after the platform write, before scale. */
|
||||
type ScreenshotCropBinding = Readonly<{
|
||||
cropOn: string;
|
||||
captureSnapshot: NonNullable<BoundScreenshotRuntime['captureSnapshot']>;
|
||||
run: ScreenshotCropRun;
|
||||
logPath: string;
|
||||
}>;
|
||||
|
||||
/**
|
||||
* Runner metadata the capture needs; cancellation comes from the request binding, not from here.
|
||||
@@ -146,30 +177,37 @@ async function executeScreenshot(
|
||||
flags: CommandFlags | undefined;
|
||||
outPath: string | undefined;
|
||||
runtime: BoundScreenshotRuntime;
|
||||
cropOn?: string;
|
||||
cropRun?: ScreenshotCropRun;
|
||||
}>,
|
||||
): Promise<Record<string, unknown>> {
|
||||
const { session, runtime, flags } = params;
|
||||
const crop = buildScreenshotCropBinding({
|
||||
cropOn: params.cropOn,
|
||||
captureSnapshot: runtime.captureSnapshot,
|
||||
cropRun: params.cropRun,
|
||||
logPath: params.logPath,
|
||||
});
|
||||
const captured = await captureScreenshotArtifact({
|
||||
session,
|
||||
sessionName: params.sessionName,
|
||||
outPath: params.outPath,
|
||||
dispatchContext: params.dispatchContext,
|
||||
captureScreenshot: runtime.captureScreenshot,
|
||||
...(crop ? { crop } : {}),
|
||||
});
|
||||
const captureSnapshot = runtime.captureSnapshot;
|
||||
const warnings = [...(captured.warnings ?? []), ...(crop?.run?.warnings ?? [])];
|
||||
return {
|
||||
...captured,
|
||||
...(captureSnapshot
|
||||
? {
|
||||
overlayRefs: await annotateScreenshotWithSessionRefs({
|
||||
session,
|
||||
logPath: params.logPath,
|
||||
screenshotPath: captured.path,
|
||||
dispatchContext: params.dispatchContext,
|
||||
captureSnapshot,
|
||||
}),
|
||||
}
|
||||
: {}),
|
||||
...(warnings.length > 0 ? { warnings } : {}),
|
||||
...(await screenshotOverlayRefsField({
|
||||
session,
|
||||
cropOn: params.cropOn,
|
||||
captureSnapshot: runtime.captureSnapshot,
|
||||
screenshotPath: captured.path,
|
||||
logPath: params.logPath,
|
||||
dispatchContext: params.dispatchContext,
|
||||
})),
|
||||
...(await readScreenshotResultMetadata({
|
||||
device: session.device,
|
||||
path: captured.path,
|
||||
@@ -179,10 +217,54 @@ async function executeScreenshot(
|
||||
};
|
||||
}
|
||||
|
||||
/** Present exactly when the admitted plan carries a crop and the snapshot binding to serve it. */
|
||||
function buildScreenshotCropBinding(
|
||||
params: Readonly<{
|
||||
cropOn: string | undefined;
|
||||
captureSnapshot: BoundScreenshotRuntime['captureSnapshot'];
|
||||
cropRun: ScreenshotCropRun | undefined;
|
||||
logPath: string;
|
||||
}>,
|
||||
): ScreenshotCropBinding | undefined {
|
||||
const { cropOn, captureSnapshot, cropRun } = params;
|
||||
if (cropOn === undefined || captureSnapshot === undefined || cropRun === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
return { cropOn, captureSnapshot, run: cropRun, logPath: params.logPath };
|
||||
}
|
||||
|
||||
/**
|
||||
* `--overlay-refs` republishes the annotated tree as the session's snapshot, so the refs it drew
|
||||
* are the refs a following interaction resolves. The tree comes from the same admitted binding as
|
||||
* the capture — the plan required both operations before either ran.
|
||||
* the capture — the plan required both operations before either ran. The crop plan binds the
|
||||
* snapshot for the crop alone, so a crop request never annotates or republishes.
|
||||
*/
|
||||
async function screenshotOverlayRefsField(
|
||||
params: Readonly<{
|
||||
session: SessionState;
|
||||
cropOn: string | undefined;
|
||||
captureSnapshot: BoundScreenshotRuntime['captureSnapshot'];
|
||||
screenshotPath: string;
|
||||
logPath: string;
|
||||
dispatchContext: DaemonCommandContext;
|
||||
}>,
|
||||
): Promise<Readonly<Record<string, unknown>>> {
|
||||
const { session, cropOn, captureSnapshot } = params;
|
||||
if (cropOn !== undefined || captureSnapshot === undefined) return {};
|
||||
return {
|
||||
overlayRefs: await annotateScreenshotWithSessionRefs({
|
||||
session,
|
||||
logPath: params.logPath,
|
||||
screenshotPath: params.screenshotPath,
|
||||
dispatchContext: params.dispatchContext,
|
||||
captureSnapshot,
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The overlay-refs tree capture: interactive-only, through the same admitted binding as the
|
||||
* screenshot, stored as the session snapshot so the burned-in refs stay the authorized frame.
|
||||
*/
|
||||
async function annotateScreenshotWithSessionRefs(
|
||||
params: Readonly<{
|
||||
@@ -247,9 +329,10 @@ function createBoundScreenshotBackend(
|
||||
session: SessionState;
|
||||
dispatchContext: DaemonCommandContext;
|
||||
captureScreenshot: BoundScreenshotRuntime['captureScreenshot'];
|
||||
crop?: ScreenshotCropBinding;
|
||||
}>,
|
||||
): AgentDeviceBackend {
|
||||
const { session, dispatchContext, captureScreenshot } = params;
|
||||
const { session, dispatchContext, captureScreenshot, crop } = params;
|
||||
return {
|
||||
platform: publicPlatformString(session.device),
|
||||
captureScreenshot: async (_context, outPath, options) => {
|
||||
@@ -275,6 +358,19 @@ function createBoundScreenshotBackend(
|
||||
},
|
||||
execution: screenshotExecutionFromContext(dispatchContext),
|
||||
});
|
||||
if (crop) {
|
||||
const outcome = await cropScreenshotToSelector({
|
||||
device: session.device,
|
||||
session,
|
||||
surface: session.surface,
|
||||
cropOn: crop.cropOn,
|
||||
screenshotPath: outPath,
|
||||
logPath: crop.logPath,
|
||||
dispatchContext,
|
||||
captureSnapshot: crop.captureSnapshot,
|
||||
});
|
||||
crop.run.warnings.push(...buildScreenshotCropWarnings(outcome));
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -916,6 +916,7 @@ agent-device screenshot # Auto filename
|
||||
agent-device screenshot page.png # Explicit screenshot path
|
||||
agent-device screenshot page.png --scale 0.3 # Resize both dimensions to 30% for agent context
|
||||
agent-device screenshot page.png --overlay-refs # Draw current @eN refs and target rectangles onto the PNG
|
||||
agent-device screenshot page.png --crop-on 'label="Save"' # Crop the capture to the frame the selector resolves on the same screen
|
||||
agent-device screenshot baseline.png --normalize-status-bar # Normalize iOS simulator chrome for reusable diff baselines
|
||||
agent-device screenshot page.png --platform web --fullscreen # On web, --fullscreen/--full/-f captures the entire document
|
||||
agent-device viewport 1280 900 --platform web # Resize the active web viewport for fixed-layout or 100vh apps
|
||||
@@ -938,6 +939,7 @@ agent-device record stop # Stop active recording
|
||||
- Set `AGENT_DEVICE_SCREENSHOT_SCALE=0.3` (or `screenshotScale` in config) as a token-conscious screenshot default for agent workflows. An explicit `--scale` overrides it.
|
||||
- Keep the scale default unset, or use `--scale 1`, when full-resolution screenshots are required for reusable pixel-diff baselines.
|
||||
- `screenshot --overlay-refs` captures a fresh full snapshot and burns visible `@eN` refs plus their target rectangles into the saved PNG.
|
||||
- `screenshot --crop-on <selector>` captures a fresh full snapshot of the same screen and crops the saved PNG to the frame the selector resolves to. The selector must resolve to exactly one framed node; the result carries a `warnings` entry when the frame is clipped to the image. Currently accepted on iOS simulators and Android emulators — every other target is refused before any device work, and the flag cannot be combined with `--overlay-refs` or `--fullscreen` because both move the captured frame away from the snapshot viewport the crop is measured against.
|
||||
- `screenshot --normalize-status-bar` temporarily normalizes iOS simulator status-bar chrome for deterministic screenshot baselines; ordinary screenshots leave the simulator's current chrome visible.
|
||||
- `screenshot --scale <factor> --overlay-refs` writes a smaller image and draws refs for that final image size; avoid very small scales when text, icons, or labels need to remain readable.
|
||||
- `diff screenshot` compares the current live screenshot to `--baseline`, or compares `--baseline` to an optional saved `current.png` path without requiring an active session. Its text output reports ranked changed regions with screen-space rectangles, changed-pixel counts, and each region's share of the diff; JSON also includes normalized rectangles. The earlier best-effort `ocr` and `nonTextDeltas` analyzers are retired; their optional result fields remain for source compatibility but are no longer emitted, so use the baseline/current images and diff artifact with vision for qualitative interpretation. It writes a diff PNG with a light grayscale current-screen context, red-tinted changed pixels, and outlined changed regions when `--out` is provided. Live iOS simulator diffs normalize status-bar chrome by default; use `screenshot --normalize-status-bar` when capturing reusable baselines.
|
||||
|
||||
Reference in New Issue
Block a user