mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
feat(bench): renderer-pinned samples, topic-coverage gate, error-recovery quizzes; trim skillgym to agentic checks (#1411)
* feat(bench): renderer-pinned samples, topic-coverage gate, error quizzes; trim skillgym to agentic checks The help conformance bench's quoted CLI output is now sourced from scripts/help-conformance-sample-outputs.mjs, and every sample is rebuilt through the real production renderers (settle output formatters, printHumanError, formatSnapshotText, refMutationAdmissionResponse) by scripts/__tests__/help-conformance-sample-outputs.test.ts — a rendering or message change fails deterministically instead of leaving the bench grading against output the CLI no longer prints. This retires the fabricated recoverable-failure envelope (production never throws a textual settle timeout; that case is replaced by a real DEVICE_IN_USE recovery quiz). Bench cases move to scripts/help-conformance-cases.mjs and are enumerated against the help-topic registry: helpTopicIds() is exported from cli-help, and scripts/__tests__/help-conformance-topic-coverage.test.ts fails when a help topic has neither a bench case nor an explicit waiver. New case families: error-envelope recovery quizzes (device-in-use, stale pinned ref, ambiguous find match, app-not-installed) pinned to real error text, topic coverage for tv/web/react-native/debugging/workflow, and a metamorphic twin of the settled-diff quiz. The skillgym smoke suite shrinks from 119 cases to the 5 that measure what only an agentic runner can show: skill routing plus output interpretation with a proven local CLI help probe (local-cli-help-policy). Its embedded samples now import the same pinned constants, replacing hand-transcribed output that had already drifted from the renderer. Knowledge checks belong to the bench; live fixture behavior belongs to the iOS simulator e2e suite. * review: drive error samples through the real producers; enforce local-help on the routing smoke The DEVICE_IN_USE, AMBIGUOUS_MATCH, and APP_NOT_INSTALLED parity tests no longer hand-author the producer message before rendering: each drives the actual producer — buildDeviceInUseBySessionError (extracted in session-open.ts and called by the handler), buildAmbiguousMatchError (now exported from find.ts), and buildAppNotInstalledError (extracted in app-resolution.ts and thrown by the resolver). Because each factory is exported from its producer file and called by the production path, dropping the production call would make it test-only and fail check:production-exports — the wiring is gate-enforced, not conventional. open-and-snapshot now sets requireLocalCliHelp and allowOnlyLocalCliHelpCommands, so the 'skill plus local help' claim is observed rather than assumed; without them the case can pass on model prior alone.
This commit is contained in:
Michał Pierzchała
committed by
GitHub
GitHub
parent
f19864e486
commit
2d1d70613f
+10
-1
@@ -30,7 +30,8 @@
|
||||
"scripts/layering/**",
|
||||
"scripts/maestro-conformance/**",
|
||||
"apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests.xctestplan",
|
||||
"scripts/write-xcuitest-cache-metadata.mjs"
|
||||
"scripts/write-xcuitest-cache-metadata.mjs",
|
||||
"scripts/help-conformance-sample-outputs.d.mts"
|
||||
],
|
||||
"ignoreDependencies": [
|
||||
"@theme"
|
||||
@@ -79,6 +80,14 @@
|
||||
"PlatformDescriptorsAreTotal",
|
||||
"GatedKeysAreResolverKeys"
|
||||
]
|
||||
},
|
||||
{
|
||||
"comment": "Help-benchmark conformance seams: helpTopicIds feeds the topic-coverage gate and PRIVATE_AX_RECOVERY_SAMPLE feeds the skillgym suite; both consumers are test-tree files outside --production analysis.",
|
||||
"file": "{src/cli/parser/cli-help.ts,scripts/help-conformance-sample-outputs.mjs}",
|
||||
"exports": [
|
||||
"helpTopicIds",
|
||||
"PRIVATE_AX_RECOVERY_SAMPLE"
|
||||
]
|
||||
}
|
||||
],
|
||||
"usedClassMembers": [
|
||||
|
||||
@@ -16,8 +16,9 @@ The mapping it encodes, for when you need to run a gate directly or reason about
|
||||
| Platform/device response — anything emitting `platform`/`appleOs` on the wire, or shaping a daemon response | `pnpm test:integration:provider` **and** `pnpm test:coverage` |
|
||||
| Cross-platform behavior | `pnpm test:integration` |
|
||||
| iOS runner / Swift | `pnpm build:xcuitest` |
|
||||
| CLI help/guidance (`src/cli/parser/cli-help.ts`, `src/cli-schema/`) | `pnpm exec vitest run src/cli/parser/__tests__ src/cli-schema/command-schema-guards.test.ts` |
|
||||
| SkillGym prompts/assertions | `pnpm test:skillgym:case <case-id>` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) |
|
||||
| CLI help/guidance (`src/cli/parser/cli-help.ts`, `src/cli-schema/`) | `pnpm exec vitest run src/cli/parser/__tests__ src/cli-schema/command-schema-guards.test.ts scripts/__tests__` — the `scripts/__tests__` gates enforce help-topic benchmark coverage and pin the bench's quoted CLI samples to the real renderers |
|
||||
| Help benchmark cases (`scripts/help-conformance-*.mjs`) | `pnpm exec vitest run scripts/__tests__` (deterministic gates); model-backed: `pnpm bench:help-conformance` (paid LLM calls, local only) |
|
||||
| SkillGym prompts/assertions | `pnpm test:skillgym:case <case-id>` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) — agentic routing + local-help-consumption proof only; command-planning knowledge checks belong in the help bench |
|
||||
| Anything in `src/`, `test/`, `skills/` | `pnpm format` |
|
||||
|
||||
Two traps worth naming:
|
||||
|
||||
+2
-2
@@ -109,8 +109,8 @@
|
||||
"perf:ios": "node --experimental-strip-types scripts/perf/run.ts --platform ios",
|
||||
"perf:android": "node --experimental-strip-types scripts/perf/run.ts --platform android",
|
||||
"lint": "oxlint . --deny-warnings",
|
||||
"format": "node ./node_modules/oxfmt/bin/oxfmt --write src test skills scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/__tests__/help-conformance-bench.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
|
||||
"format:check": "node ./node_modules/oxfmt/bin/oxfmt --check src test skills scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/__tests__/help-conformance-bench.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
|
||||
"format": "node ./node_modules/oxfmt/bin/oxfmt --write src test skills scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
|
||||
"format:check": "node ./node_modules/oxfmt/bin/oxfmt --check src test skills scripts/help-conformance-bench.mjs scripts/help-conformance-case-checks.mjs scripts/help-conformance-command-validator.ts scripts/help-conformance-expectations.mjs scripts/help-conformance-plan-validator.mjs scripts/help-conformance-runner-output.mjs scripts/help-conformance-summary.mjs scripts/help-conformance-cases.mjs scripts/help-conformance-sample-outputs.mjs scripts/__tests__/help-conformance-bench.test.ts scripts/__tests__/help-conformance-sample-outputs.test.ts scripts/__tests__/help-conformance-topic-coverage.test.ts package.json tsconfig.json tsconfig.lib.json tsdown.config.ts vitest.config.ts .github/actions/setup-node-pnpm/action.yml .oxlintrc.json .oxfmtrc.json '!test/skillgym/.skillgym-results/**'",
|
||||
"fallow": "fallow audit --base origin/main",
|
||||
"fallow:all": "fallow --summary",
|
||||
"fallow:baseline": "(fallow dead-code --save-baseline fallow-baselines/dead-code.json --summary || true) && (fallow health --save-baseline fallow-baselines/health.json --summary || true)",
|
||||
|
||||
@@ -214,8 +214,12 @@ test('runtime-output cases do not coach the command that their output should imp
|
||||
[
|
||||
'settle-diff-is-observation',
|
||||
'sample-output-settled-diff-next-target',
|
||||
'metamorphic-settled-diff-next-target-notes',
|
||||
'sample-output-not-settled-needs-observe',
|
||||
'sample-output-recoverable-failure-retries-in-session',
|
||||
'sample-output-device-in-use-reuses-session',
|
||||
'sample-output-stale-ref-resnapshots',
|
||||
'sample-output-ambiguous-match-reobserves',
|
||||
'sample-output-app-not-installed-discovers-first',
|
||||
].join(','),
|
||||
'--runner',
|
||||
'claude:test-model',
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { test } from 'vitest';
|
||||
import {
|
||||
AMBIGUOUS_MATCH_SAMPLE,
|
||||
APP_NOT_INSTALLED_SAMPLE,
|
||||
DEVICE_IN_USE_SAMPLE,
|
||||
NOT_SETTLED_SAMPLE,
|
||||
PRIVATE_AX_RECOVERY_SAMPLE,
|
||||
SETTLE_DIFF_SAMPLE,
|
||||
SETTLE_DIFF_SAMPLE_NOTES,
|
||||
SETTLE_TAIL_SAMPLE,
|
||||
STALE_REF_SAMPLE,
|
||||
} from '../help-conformance-sample-outputs.mjs';
|
||||
import { interactionCliOutputFormatters } from '../../src/commands/interaction/output.ts';
|
||||
import { NEVER_SETTLED_HINT } from '../../src/commands/interaction/runtime/settle.ts';
|
||||
import { buildAmbiguousMatchError } from '../../src/daemon/handlers/find.ts';
|
||||
import { refMutationAdmissionResponse } from '../../src/daemon/handlers/interaction-ref-policy.ts';
|
||||
import { buildDeviceInUseBySessionError } from '../../src/daemon/handlers/session-open.ts';
|
||||
import { resolveRefStalenessWarning } from '../../src/daemon/session-snapshot.ts';
|
||||
import type { SessionState } from '../../src/daemon/types.ts';
|
||||
import { buildAppNotInstalledError } from '../../src/platforms/apple/core/app-resolution.ts';
|
||||
import { AppError, normalizeError } from '../../src/kernel/errors.ts';
|
||||
import type { SnapshotQualityVerdict } from '../../src/snapshot/snapshot-quality.ts';
|
||||
import { renderSnapshotQualityWarnings } from '../../src/snapshot/snapshot-quality.ts';
|
||||
import { formatSnapshotText, printHumanError } from '../../src/utils/output.ts';
|
||||
|
||||
// Every quiz case in scripts/help-conformance-cases.mjs quotes "captured"
|
||||
// agent-device output. These tests rebuild each quoted sample through the real
|
||||
// production renderer, so a rendering or message change fails HERE instead of
|
||||
// leaving the benchmark grading models against output the CLI no longer
|
||||
// prints — the drift class that made hand-transcribed samples untrustworthy.
|
||||
|
||||
const formatPress = (result: Record<string, unknown>) =>
|
||||
interactionCliOutputFormatters.press({ input: {}, result });
|
||||
|
||||
const formatFill = (result: Record<string, unknown>) =>
|
||||
interactionCliOutputFormatters.fill({ input: {}, result });
|
||||
|
||||
function renderHumanError(error: AppError): string {
|
||||
const lines: string[] = [];
|
||||
const originalWrite = process.stderr.write;
|
||||
process.stderr.write = ((chunk: string | Uint8Array) => {
|
||||
lines.push(String(chunk));
|
||||
return true;
|
||||
}) as typeof process.stderr.write;
|
||||
try {
|
||||
printHumanError(normalizeError(error));
|
||||
} finally {
|
||||
process.stderr.write = originalWrite;
|
||||
}
|
||||
return lines.join('').trimEnd();
|
||||
}
|
||||
|
||||
test('settle tail sample matches the press renderer, including ADR 0014 pinning', () => {
|
||||
const output = formatPress({
|
||||
ref: 'e37',
|
||||
x: 203,
|
||||
y: 88,
|
||||
settle: {
|
||||
settled: true,
|
||||
waitedMs: 540,
|
||||
refsGeneration: 5,
|
||||
diff: {
|
||||
summary: { additions: 0, removals: 1, unchanged: 15 },
|
||||
lines: [{ kind: 'removed', text: '@e50 [text] "Suggested for you"' }],
|
||||
},
|
||||
tail: [
|
||||
{ ref: 'e64', role: 'text-field', label: 'Search' },
|
||||
{ ref: 'e65', role: 'text', label: 'Recent searches' },
|
||||
{ ref: 'e12', role: 'tab', label: 'Home' },
|
||||
{ ref: 'e40', role: 'tab', label: 'Profile' },
|
||||
],
|
||||
},
|
||||
});
|
||||
assert.equal(output.text, SETTLE_TAIL_SAMPLE.output);
|
||||
});
|
||||
|
||||
test('settled diff sample matches the fill renderer', () => {
|
||||
const output = formatFill({
|
||||
text: 'callstack',
|
||||
message: 'Filled 9 chars',
|
||||
settle: {
|
||||
settled: true,
|
||||
waitedMs: 610,
|
||||
refsGeneration: 6,
|
||||
diff: {
|
||||
summary: { additions: 2, removals: 0, unchanged: 18 },
|
||||
lines: [
|
||||
{ kind: 'added', text: '@e64 [button] "@callstack.com"' },
|
||||
{ kind: 'added', text: '@e65 [text] "Callstack"' },
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
assert.equal(output.text, SETTLE_DIFF_SAMPLE.output);
|
||||
});
|
||||
|
||||
test('metamorphic settled diff sample matches the fill renderer', () => {
|
||||
const output = formatFill({
|
||||
text: 'groceries',
|
||||
message: 'Filled 9 chars',
|
||||
settle: {
|
||||
settled: true,
|
||||
waitedMs: 480,
|
||||
refsGeneration: 3,
|
||||
diff: {
|
||||
summary: { additions: 2, removals: 0, unchanged: 11 },
|
||||
lines: [
|
||||
{ kind: 'added', text: '@e21 [button] "Groceries list"' },
|
||||
{ kind: 'added', text: '@e22 [text] "3 items"' },
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
assert.equal(output.text, SETTLE_DIFF_SAMPLE_NOTES.output);
|
||||
});
|
||||
|
||||
test('not-settled sample matches the press renderer and the production hint', () => {
|
||||
const output = formatPress({
|
||||
ref: 'e12',
|
||||
x: 166,
|
||||
y: 240,
|
||||
settle: { settled: false, waitedMs: 10_000, hint: NEVER_SETTLED_HINT },
|
||||
});
|
||||
assert.equal(output.text, NOT_SETTLED_SAMPLE.output);
|
||||
});
|
||||
|
||||
test('private-ax recovery sample matches the snapshot renderer and quality warning', () => {
|
||||
const nodes = [
|
||||
{
|
||||
index: 1,
|
||||
ref: 'e5',
|
||||
type: 'Button',
|
||||
label: 'Search',
|
||||
interactive: true,
|
||||
rect: { x: 20, y: 120, width: 200, height: 44 },
|
||||
},
|
||||
{
|
||||
index: 2,
|
||||
ref: 'e8',
|
||||
type: 'Tab',
|
||||
label: 'Home',
|
||||
selected: true,
|
||||
interactive: true,
|
||||
rect: { x: 0, y: 780, width: 100, height: 60 },
|
||||
},
|
||||
];
|
||||
const verdict = { state: 'recovered', backend: 'private-ax' } as SnapshotQualityVerdict;
|
||||
const text = formatSnapshotText({
|
||||
nodes,
|
||||
backend: 'private-ax',
|
||||
snapshotQuality: verdict,
|
||||
warnings: renderSnapshotQualityWarnings(verdict, nodes),
|
||||
interactiveOnly: true,
|
||||
});
|
||||
assert.equal(text.trimEnd(), PRIVATE_AX_RECOVERY_SAMPLE.output);
|
||||
});
|
||||
|
||||
test('device-in-use sample matches the real session-open producer', () => {
|
||||
const owningSession = { name: 'checkout' } as SessionState;
|
||||
const device = { id: 'SIM-001', name: 'iPhone 17 Pro' } as Parameters<
|
||||
typeof buildDeviceInUseBySessionError
|
||||
>[1];
|
||||
const response = buildDeviceInUseBySessionError(owningSession, device);
|
||||
assert.ok(!response.ok, 'the by-session conflict must be an error response');
|
||||
const rendered = renderHumanError(
|
||||
new AppError(
|
||||
response.error.code as ConstructorParameters<typeof AppError>[0],
|
||||
response.error.message,
|
||||
response.error.details,
|
||||
),
|
||||
);
|
||||
assert.equal(rendered, DEVICE_IN_USE_SAMPLE.output);
|
||||
});
|
||||
|
||||
test('stale-ref sample matches the real admission rejection and staleness hint', () => {
|
||||
const session = { refFrameGeneration: 7 } as SessionState;
|
||||
const response = refMutationAdmissionResponse({
|
||||
session,
|
||||
ref: '@e12',
|
||||
mintedGeneration: 5,
|
||||
staleRefsWarning: resolveRefStalenessWarning({ session, ref: '@e12', mintedGeneration: 5 }),
|
||||
});
|
||||
assert.ok(response && !response.ok, 'a superseded pin must be rejected');
|
||||
const rendered = renderHumanError(
|
||||
new AppError(
|
||||
response.error.code as ConstructorParameters<typeof AppError>[0],
|
||||
response.error.message,
|
||||
response.error.details,
|
||||
),
|
||||
);
|
||||
assert.equal(rendered, STALE_REF_SAMPLE.output);
|
||||
});
|
||||
|
||||
test('ambiguous-match sample matches the real find producer and default hint', () => {
|
||||
const matches = [
|
||||
{ ref: 'e2', label: 'Follow' },
|
||||
{ ref: 'e5', label: 'Follow' },
|
||||
{ ref: 'e9', label: 'Follow' },
|
||||
] as Parameters<typeof buildAmbiguousMatchError>[0];
|
||||
const response = buildAmbiguousMatchError(matches, 'text', 'Follow');
|
||||
assert.ok(!response.ok, 'an ambiguous find must be an error response');
|
||||
const rendered = renderHumanError(
|
||||
new AppError(
|
||||
response.error.code as ConstructorParameters<typeof AppError>[0],
|
||||
response.error.message,
|
||||
response.error.details,
|
||||
),
|
||||
);
|
||||
assert.equal(rendered, AMBIGUOUS_MATCH_SAMPLE.output);
|
||||
});
|
||||
|
||||
test('app-not-installed sample matches the real app-resolution producer and default hint', () => {
|
||||
const rendered = renderHumanError(buildAppNotInstalledError('Shoply'));
|
||||
assert.equal(rendered, APP_NOT_INSTALLED_SAMPLE.output);
|
||||
});
|
||||
@@ -0,0 +1,67 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { test } from 'vitest';
|
||||
import { CASES } from '../help-conformance-cases.mjs';
|
||||
import { helpTopicIds } from '../../src/cli/parser/cli-help.ts';
|
||||
|
||||
// "What enumerates N": benchmark cases are keyed to help topics, and this gate
|
||||
// keys the case list to the topic registry itself. A new help topic must gain
|
||||
// a benchmark case (docs: [..., '<topic>']) or an explicit waiver here — the
|
||||
// waiver names why the topic's guidance is not yet benchmarked, so uncovered
|
||||
// topics are a visible decision instead of silent drift.
|
||||
const WAIVED_TOPICS: Record<string, string> = {
|
||||
cdp: 'JS-heap forensics niche; add cases when heap-guidance regressions show up in practice.',
|
||||
macos: 'macOS surface guidance is thin and stable; no observed planning regressions yet.',
|
||||
maestro: 'Compatibility reference, not a planning loop; conformance is oracle-tested instead.',
|
||||
'physical-device': 'Needs device-specific setup guidance; no portable planning task defined yet.',
|
||||
'react-devtools':
|
||||
'Profiling-window guidance; add cases when render-diagnosis planning regresses.',
|
||||
remote: 'Remote/cloud lease setup; niche until remote workflows are benchmarked end to end.',
|
||||
};
|
||||
|
||||
const FIRST_SCREEN_DOC = '--help:first30';
|
||||
|
||||
test('every case doc id is the first screen or a real help topic', () => {
|
||||
const topics = new Set(helpTopicIds());
|
||||
for (const testCase of CASES) {
|
||||
for (const doc of testCase.docs) {
|
||||
assert.ok(
|
||||
doc === FIRST_SCREEN_DOC || topics.has(doc),
|
||||
`case "${testCase.id}" references unknown help doc "${doc}"`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('every help topic has a benchmark case or an explicit waiver', () => {
|
||||
const covered = new Set(CASES.flatMap((testCase) => testCase.docs));
|
||||
const uncovered = helpTopicIds().filter(
|
||||
(topic) => !covered.has(topic) && !(topic in WAIVED_TOPICS),
|
||||
);
|
||||
assert.deepEqual(
|
||||
uncovered,
|
||||
[],
|
||||
'new help topics need a benchmark case in scripts/help-conformance-cases.mjs or a waiver above',
|
||||
);
|
||||
});
|
||||
|
||||
test('waivers only name real, uncovered topics', () => {
|
||||
const topics = new Set(helpTopicIds());
|
||||
const covered = new Set(CASES.flatMap((testCase) => testCase.docs));
|
||||
for (const [topic, reason] of Object.entries(WAIVED_TOPICS)) {
|
||||
assert.ok(topics.has(topic), `waived topic "${topic}" no longer exists — remove the waiver`);
|
||||
assert.ok(
|
||||
!covered.has(topic),
|
||||
`waived topic "${topic}" is now covered by a case — remove the waiver`,
|
||||
);
|
||||
assert.ok(reason.trim().length > 0, `waiver for "${topic}" needs a reason`);
|
||||
}
|
||||
});
|
||||
|
||||
test('the first help screen is exercised by every case family', () => {
|
||||
for (const testCase of CASES) {
|
||||
assert.ok(
|
||||
testCase.docs.includes(FIRST_SCREEN_DOC),
|
||||
`case "${testCase.id}" must include the first-screen doc — it is the only text every runner sees before choosing a topic`,
|
||||
);
|
||||
}
|
||||
});
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
countChecks,
|
||||
scoreExpectations,
|
||||
} from './help-conformance-case-checks.mjs';
|
||||
import { CASES } from './help-conformance-cases.mjs';
|
||||
import { validatePlanCommands } from './help-conformance-plan-validator.mjs';
|
||||
import { detectRunnerError, extractCommands } from './help-conformance-runner-output.mjs';
|
||||
import { summarizeResults } from './help-conformance-summary.mjs';
|
||||
@@ -83,241 +84,6 @@ const OPTION_APPLIERS = {
|
||||
},
|
||||
};
|
||||
|
||||
// Raw-coordinate fallback the ported skillgym quiz cases forbid: a
|
||||
// click/fill/press targeting bare numbers instead of a ref or selector.
|
||||
const RAW_COORDINATE_TARGET =
|
||||
/(?:^|\n)(?:agent-device\s+)?(?:click|fill|press)\s+-?\d+(?:\.\d+)?\s+-?\d+(?:\.\d+)?/i;
|
||||
|
||||
const CASES = [
|
||||
{
|
||||
id: 'raw-first-screen-bluesky',
|
||||
docs: ['--help:first30'],
|
||||
task: 'Plan commands to open an already installed Bluesky app, search "callstack", open the @callstack.com account, press Follow or Following, and close.',
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'noWaitStable',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'metamorphic-community-search',
|
||||
docs: ['--help:first30'],
|
||||
task: 'Plan commands to open the already installed app com.example.community, open the visible Discover destination, fill the People search field with "react native", open the @react.dev account, press Connect or Connected, and close.',
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'noWaitStable',
|
||||
'opensAndCloses',
|
||||
],
|
||||
matchers: [
|
||||
{
|
||||
id: 'opensKnownCommunityApp',
|
||||
pattern: /\bagent-device\s+open\s+com\.example\.community\b/i,
|
||||
},
|
||||
{
|
||||
id: 'fillsExpectedSearch',
|
||||
pattern: /\bagent-device\s+fill\b[^\n]*(?:"react native"|'react native')[^\n]*--settle\b/i,
|
||||
},
|
||||
{
|
||||
id: 'usesLiteralHandleSelector',
|
||||
pattern:
|
||||
/\bagent-device\s+(?:press|click|tap)\b[^\n]*(?:label|text)=@react\.dev\b[^\n]*--settle\b/i,
|
||||
},
|
||||
],
|
||||
forbidden: [
|
||||
{
|
||||
id: 'noBlueskyLeakage',
|
||||
pattern: /(?:bluesky|callstack|@e64|@callstack\.com)/i,
|
||||
},
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'manual-qa-bluesky-script',
|
||||
docs: ['--help:first30', 'manual-qa'],
|
||||
task: 'You are following a manual QA script: on Bluesky, open Search, search "callstack", open @callstack.com, press Follow or Following, verify the button state changed, then close. Plan commands only.',
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'verifiesNamedExpectation',
|
||||
'noWaitStable',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'dogfood-mode',
|
||||
docs: ['--help:first30', 'dogfood'],
|
||||
task: 'Plan a short dogfood pass for the logged-in iOS shop app com.example.shop. Exercise the visible Home, Search, and Cart destinations and capture reproducible evidence for any issue found.',
|
||||
allowedExternalCommands: ['mkdir'],
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'usesDogfoodEvidence',
|
||||
'opensAndCloses',
|
||||
],
|
||||
matchers: [
|
||||
{ id: 'opensKnownDogfoodApp', pattern: /\bagent-device\s+open\s+com\.example\.shop\b/i },
|
||||
{
|
||||
id: 'capturesStrongIssueEvidence',
|
||||
pattern: /\b(?:screenshot\b[^\n]*--overlay-refs|record\s+start\b|logs\s+mark\b)/i,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'engineering-validate-mode',
|
||||
docs: ['--help:first30', 'validate'],
|
||||
task: 'Plan commands to validate a TypeScript-only CLI/runtime change to settled press output against the already installed iOS Settings app. Use the known General control, prove current built output is running, and clean up. Swift runner code did not change.',
|
||||
allowedExternalCommands: ['pnpm'],
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'usesValidationPrep',
|
||||
'opensAndCloses',
|
||||
],
|
||||
matchers: [
|
||||
{
|
||||
id: 'opensSettings',
|
||||
pattern: /\bagent-device\s+open\s+(?:settings|com\.apple\.Preferences)\b/i,
|
||||
},
|
||||
],
|
||||
forbidden: [
|
||||
{
|
||||
id: 'avoidsUnrelatedPlatformBuild',
|
||||
pattern: /\bpnpm\s+(?:run\s+)?build:(?:android|xcuitest)\b/i,
|
||||
},
|
||||
],
|
||||
},
|
||||
// The first three cases below are ported from
|
||||
// test/skillgym/suites/agent-device-smoke-suite.ts (settle-diff-is-observation,
|
||||
// sample-output-settled-diff-next-target, sample-output-not-settled-needs-observe).
|
||||
// The fourth comes from test/output-economy/routine-workflow.ts. They are
|
||||
// self-contained "next-command quiz" cases: a captured agent-device
|
||||
// output plus a task, scored by regex instead of the named expectation
|
||||
// scorers above. Output text mirrors the CURRENT settle rendering in
|
||||
// src/commands/interaction/output.ts, including the "unchanged interactive
|
||||
// (N):" tail added by #1167/#1172 for diffs with no meaningful added ref.
|
||||
{
|
||||
id: 'settle-diff-is-observation',
|
||||
docs: ['--help:first30'],
|
||||
task: `You already ran this command and observed its settled output:
|
||||
|
||||
agent-device press @e37 --settle
|
||||
Tapped @e37 (203, 88)
|
||||
settled after 540ms: +0 -1 (~15 unchanged)
|
||||
- @e50 [text] "Suggested for you"
|
||||
unchanged interactive (4):
|
||||
= @e64 [text-field] "Search"
|
||||
= @e65 [text] "Recent searches"
|
||||
= @e12 [tab] "Home"
|
||||
= @e40 [tab] "Profile"
|
||||
|
||||
Use the output already shown to determine whether the feed-search UI is present, then close the session. What command should run next?`,
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [{ id: 'plansClose', pattern: /(?:^|\n)(?:agent-device\s+)?close\b/i }],
|
||||
forbidden: [
|
||||
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
|
||||
{ id: 'noWait', pattern: /\bwait\b/i },
|
||||
{ id: 'noFind', pattern: /\bfind\b/i },
|
||||
{ id: 'noGet', pattern: /\bget\b/i },
|
||||
{ id: 'noIs', pattern: /\bis\b/i },
|
||||
{ id: 'noPressOrClick', pattern: /\b(?:press|click)\b/i },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-settled-diff-next-target',
|
||||
docs: ['--help:first30'],
|
||||
task: `Read this previous agent-device output, then plan the next command:
|
||||
|
||||
agent-device fill 'id="account-search"' "callstack" --settle
|
||||
Filled 9 chars
|
||||
settled after 610ms: +2 -0 (~18 unchanged)
|
||||
+ @e64 [button] "@callstack.com"
|
||||
+ @e65 [text] "Callstack"
|
||||
|
||||
The task is to open the matching account result. What command should run next?`,
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{ id: 'pressOrClickOrTap', pattern: /\b(?:press|click|tap)\b/i },
|
||||
{ id: 'usesE64RefOrLabel', pattern: /@e64\b|label=(?:["']?@callstack\.com["']?)/i },
|
||||
{ id: 'usesSettleFlag', pattern: /--settle\b/i },
|
||||
],
|
||||
forbidden: [
|
||||
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
|
||||
{ id: 'noWaitStable', pattern: /wait\s+stable/i },
|
||||
{ id: 'noFill', pattern: /\bfill\b/i },
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-not-settled-needs-observe',
|
||||
docs: ['--help:first30'],
|
||||
task: `Read this previous agent-device output, then plan the next command:
|
||||
|
||||
agent-device press @e12 --settle
|
||||
Tapped @e12 (166, 240)
|
||||
not settled after 10000ms
|
||||
hint: The UI kept changing for the whole settle budget (animation, carousel, or ticker?), so no settled diff is shown. Raise --timeout, wait for specific content, or take a fresh snapshot.
|
||||
|
||||
The next target is not known yet. What command should run next?`,
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'observesBeforeActing',
|
||||
pattern: /(?:^|\n)(?:agent-device\s+)?(?:wait\b|snapshot\b[^\n]*-i\b)/i,
|
||||
},
|
||||
],
|
||||
forbidden: [
|
||||
{
|
||||
id: 'noBareRefMutation',
|
||||
pattern: /(?:^|\n)(?:agent-device\s+)?(?:press|click|fill|longpress)\s+@e\d+/i,
|
||||
},
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
// The next-step decision measured by the routine-workflow oracle
|
||||
// (test/output-economy/routine-workflow.ts): an actionable failure that
|
||||
// preserved the session, the failing ref, its snapshot generation, a retry
|
||||
// signal, and retry guidance should be recovered IN SESSION with a targeted
|
||||
// retry — not by reopening the session or re-observing evidence the error
|
||||
// already kept valid.
|
||||
{
|
||||
id: 'sample-output-recoverable-failure-retries-in-session',
|
||||
docs: ['--help:first30'],
|
||||
task: `Read this previous agent-device output, then plan the next command:
|
||||
|
||||
agent-device press @e6 --settle
|
||||
error COMMAND_FAILED: Tap on @e6 did not settle within 10000ms
|
||||
retriable: true
|
||||
hint: The tap did not settle in time. Retry press @e6 --settle with a higher --timeout; refs from this session are still valid.
|
||||
details: { reason: "timeout", timeoutMs: 10000, ref: "@e6", session: "checkout", refsGeneration: 22 }
|
||||
|
||||
What command should run next?`,
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{ id: 'retriesSameRef', pattern: /(?:^|\n)(?:agent-device\s+)?press\s+@e6\b/i },
|
||||
{ id: 'keepsSettle', pattern: /--settle\b/i },
|
||||
{ id: 'raisesTimeout', pattern: /--timeout\s+\d+/i },
|
||||
],
|
||||
forbidden: [
|
||||
{ id: 'noReopen', pattern: /(?:^|\n)(?:agent-device\s+)?open\b/i },
|
||||
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
|
||||
{ id: 'noFind', pattern: /\bfind\b/i },
|
||||
{ id: 'noGet', pattern: /\bget\b/i },
|
||||
{ id: 'noWait', pattern: /\bwait\b/i },
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = { runners: undefined, cases: undefined, dryRun: false };
|
||||
readArgs(args, argv, 0);
|
||||
|
||||
@@ -0,0 +1,426 @@
|
||||
import {
|
||||
AMBIGUOUS_MATCH_SAMPLE,
|
||||
APP_NOT_INSTALLED_SAMPLE,
|
||||
DEVICE_IN_USE_SAMPLE,
|
||||
NOT_SETTLED_SAMPLE,
|
||||
SETTLE_DIFF_SAMPLE,
|
||||
SETTLE_DIFF_SAMPLE_NOTES,
|
||||
SETTLE_TAIL_SAMPLE,
|
||||
STALE_REF_SAMPLE,
|
||||
sampleText,
|
||||
} from './help-conformance-sample-outputs.mjs';
|
||||
|
||||
// Raw-coordinate fallback the quiz cases forbid: a click/fill/press targeting
|
||||
// bare numbers instead of a ref or selector.
|
||||
const RAW_COORDINATE_TARGET =
|
||||
/(?:^|\n)(?:agent-device\s+)?(?:click|fill|press)\s+-?\d+(?:\.\d+)?\s+-?\d+(?:\.\d+)?/i;
|
||||
|
||||
function quiz(sample, question) {
|
||||
return `Read this previous agent-device output, then plan the next command:
|
||||
|
||||
${sampleText(sample)}
|
||||
|
||||
${question}`;
|
||||
}
|
||||
|
||||
// Case docs reference help topic ids from src/cli/parser/cli-help.ts plus the
|
||||
// synthetic '--help:first30' first-screen slice. Topic coverage is enforced by
|
||||
// scripts/__tests__/help-conformance-topic-coverage.test.ts: a new help topic
|
||||
// needs a case here or an explicit waiver there.
|
||||
export const CASES = [
|
||||
{
|
||||
id: 'raw-first-screen-bluesky',
|
||||
docs: ['--help:first30'],
|
||||
task: 'Plan commands to open an already installed Bluesky app, search "callstack", open the @callstack.com account, press Follow or Following, and close.',
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'noWaitStable',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'metamorphic-community-search',
|
||||
docs: ['--help:first30'],
|
||||
task: 'Plan commands to open the already installed app com.example.community, open the visible Discover destination, fill the People search field with "react native", open the @react.dev account, press Connect or Connected, and close.',
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'noWaitStable',
|
||||
'opensAndCloses',
|
||||
],
|
||||
matchers: [
|
||||
{
|
||||
id: 'opensKnownCommunityApp',
|
||||
pattern: /\bagent-device\s+open\s+com\.example\.community\b/i,
|
||||
},
|
||||
{
|
||||
id: 'fillsExpectedSearch',
|
||||
pattern: /\bagent-device\s+fill\b[^\n]*(?:"react native"|'react native')[^\n]*--settle\b/i,
|
||||
},
|
||||
{
|
||||
id: 'usesLiteralHandleSelector',
|
||||
pattern:
|
||||
/\bagent-device\s+(?:press|click|tap)\b[^\n]*(?:label|text)=@react\.dev\b[^\n]*--settle\b/i,
|
||||
},
|
||||
],
|
||||
forbidden: [
|
||||
{
|
||||
id: 'noBlueskyLeakage',
|
||||
pattern: /(?:bluesky|callstack|@e64|@callstack\.com)/i,
|
||||
},
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'manual-qa-bluesky-script',
|
||||
docs: ['--help:first30', 'manual-qa'],
|
||||
task: 'You are following a manual QA script: on Bluesky, open Search, search "callstack", open @callstack.com, press Follow or Following, verify the button state changed, then close. Plan commands only.',
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'verifiesNamedExpectation',
|
||||
'noWaitStable',
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'dogfood-mode',
|
||||
docs: ['--help:first30', 'dogfood'],
|
||||
task: 'Plan a short dogfood pass for the logged-in iOS shop app com.example.shop. Exercise the visible Home, Search, and Cart destinations and capture reproducible evidence for any issue found.',
|
||||
allowedExternalCommands: ['mkdir'],
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'usesDogfoodEvidence',
|
||||
'opensAndCloses',
|
||||
],
|
||||
matchers: [
|
||||
{ id: 'opensKnownDogfoodApp', pattern: /\bagent-device\s+open\s+com\.example\.shop\b/i },
|
||||
{
|
||||
id: 'capturesStrongIssueEvidence',
|
||||
pattern: /\b(?:screenshot\b[^\n]*--overlay-refs|record\s+start\b|logs\s+mark\b)/i,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'engineering-validate-mode',
|
||||
docs: ['--help:first30', 'validate'],
|
||||
task: 'Plan commands to validate a TypeScript-only CLI/runtime change to settled press output against the already installed iOS Settings app. Use the known General control, prove current built output is running, and clean up. Swift runner code did not change.',
|
||||
allowedExternalCommands: ['pnpm'],
|
||||
expectations: [
|
||||
'validPlanCommands',
|
||||
'fullPrefix',
|
||||
'usesSnapshotI',
|
||||
'usesSettleOnMutations',
|
||||
'usesValidationPrep',
|
||||
'opensAndCloses',
|
||||
],
|
||||
matchers: [
|
||||
{
|
||||
id: 'opensSettings',
|
||||
pattern: /\bagent-device\s+open\s+(?:settings|com\.apple\.Preferences)\b/i,
|
||||
},
|
||||
],
|
||||
forbidden: [
|
||||
{
|
||||
id: 'avoidsUnrelatedPlatformBuild',
|
||||
pattern: /\bpnpm\s+(?:run\s+)?build:(?:android|xcuitest)\b/i,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'tv-focus-first-remote',
|
||||
docs: ['--help:first30', 'tv'],
|
||||
task: 'On an Android TV emulator, open the installed app com.example.tvhub, move focus to the "Continue watching" tile two positions to the right of the initially focused tile, activate it, verify the player screen appeared, and close. Plan commands only.',
|
||||
expectations: ['validPlanCommands', 'fullPrefix', 'usesSnapshotI', 'opensAndCloses'],
|
||||
matchers: [
|
||||
{ id: 'movesFocusWithRemote', pattern: /\btv-remote\s+press\s+right\b/i },
|
||||
{ id: 'activatesWithSelect', pattern: /\btv-remote\s+press\s+select\b/i },
|
||||
{ id: 'verifiesOutcome', pattern: /\b(?:is\s+focused|wait\b|find\b)/i },
|
||||
],
|
||||
forbidden: [
|
||||
// Focus-first surface: activation goes through tv-remote select, not a
|
||||
// coordinate/element tap (help tv "Do not assume press/click @ref works").
|
||||
{
|
||||
id: 'noDirectTapActivation',
|
||||
pattern: /(?:^|\n)agent-device\s+(?:press|click|tap)\s/i,
|
||||
},
|
||||
{ id: 'noRawAdbKeyevent', pattern: /\badb\s+shell\s+input\b/i },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'web-managed-backend-loop',
|
||||
docs: ['--help:first30', 'web'],
|
||||
task: 'On a fresh machine that has never run web automation, plan commands to set up and verify the managed web backend, open https://shop.example/login, fill the Email field with "qa@example.com", press the "Sign in" button, verify the "Welcome back" text appears, capture a screenshot to ./artifacts/web-login.png, and close. Plan commands only.',
|
||||
expectations: ['validPlanCommands', 'fullPrefix', 'usesSnapshotI', 'opensAndCloses'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'setsUpBackendBeforeOpen',
|
||||
pattern: /\bagent-device\s+web\s+setup\b[\s\S]*\n[^\n]*\bopen\s+https:\/\//i,
|
||||
},
|
||||
{ id: 'verifiesBackendWithDoctor', pattern: /\bagent-device\s+web\s+doctor\b/i },
|
||||
{ id: 'usesWebPlatform', pattern: /--platform\s+web\b/i },
|
||||
{ id: 'verifiesWelcomeText', pattern: /\b(?:wait|is|find)\b[^\n]*welcome/i },
|
||||
],
|
||||
forbidden: [
|
||||
// help web: native mobile/desktop setup commands are out of scope for
|
||||
// --platform web sessions.
|
||||
{
|
||||
id: 'noNativeSetupCommands',
|
||||
pattern: /(?:^|\n)agent-device\s+(?:boot|apps|install|alert|keyboard|perf|logs)\b/i,
|
||||
},
|
||||
{ id: 'noStandaloneAgentBrowser', pattern: /(?:^|\n)agent-browser\b/i },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'react-native-overlay-before-tap',
|
||||
docs: ['--help:first30', 'react-native'],
|
||||
task: 'An Expo dev-client app on the iOS simulator shows a React Native warning overlay in the latest snapshot. Plan the commands that safely get past it and then press the control with id "submit-order". Plan commands only.',
|
||||
expectations: ['validPlanCommands', 'fullPrefix', 'usesSettleOnMutations'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'usesDismissOverlayCommand',
|
||||
pattern: /(?:^|\n)agent-device\s+react-native\s+dismiss-overlay\b/i,
|
||||
},
|
||||
{
|
||||
id: 'refreshesRefsAfterDismiss',
|
||||
pattern: /dismiss-overlay\b[\s\S]*\n[^\n]*\bsnapshot\s+-i\b/i,
|
||||
},
|
||||
{ id: 'pressesSubmitTarget', pattern: /(?:^|\n)agent-device\s+press\s+[^\n]*submit-order/i },
|
||||
],
|
||||
forbidden: [
|
||||
// help react-native: never press warning/error overlay text manually;
|
||||
// the dismiss-overlay command owns LogBox/RedBox targeting.
|
||||
{
|
||||
id: 'noManualOverlayPress',
|
||||
pattern: /(?:^|\n)agent-device\s+(?:press|click)\s+[^\n]*(?:warning|error|logbox|redbox)/i,
|
||||
},
|
||||
{ id: 'noPlainReloadCommand', pattern: /(?:^|\n)agent-device\s+reload\b/i },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'debugging-small-log-window',
|
||||
docs: ['--help:first30', 'debugging'],
|
||||
task: 'The "Load diagnostics" control (id "load-diagnostics") in the already-open iOS app intermittently fails. Plan commands to capture a small fresh log window plus request/response metadata around one reproduction. Plan commands only.',
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'clearsAndRestartsLogs',
|
||||
pattern: /(?:^|\n)agent-device\s+logs\s+clear\s+--restart\b/i,
|
||||
},
|
||||
{ id: 'marksBeforeRepro', pattern: /\blogs\s+mark\b/i },
|
||||
{
|
||||
id: 'reproducesTargetPress',
|
||||
pattern: /(?:^|\n)agent-device\s+press\s+[^\n]*load-diagnostics/i,
|
||||
},
|
||||
{ id: 'readsLogPath', pattern: /\blogs\s+path\b/i },
|
||||
{ id: 'dumpsNetworkMetadata', pattern: /\bnetwork\s+dump\b/i },
|
||||
],
|
||||
forbidden: [
|
||||
{ id: 'noSessionReopen', pattern: /(?:^|\n)agent-device\s+open\b/i },
|
||||
{ id: 'noSplitLogRestart', pattern: /\blogs\s+stop\b/i },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'workflow-install-artifact-before-open',
|
||||
docs: ['--help:first30', 'workflow'],
|
||||
task: 'A local Android build artifact ./dist/app-release.apk contains the app com.example.orders, which is not yet on the emulator. Plan commands to get it running with fresh state and confirm its first screen shows "Orders". Plan commands only.',
|
||||
expectations: ['validPlanCommands', 'fullPrefix', 'usesSnapshotI'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'installsIdThenArtifact',
|
||||
pattern: /(?:^|\n)agent-device\s+install\s+com\.example\.orders\s+\S*app-release\.apk/i,
|
||||
},
|
||||
{
|
||||
id: 'opensFreshAfterInstall',
|
||||
pattern: /\binstall\b[\s\S]*\n[^\n]*\bopen\s+com\.example\.orders\b[^\n]*--relaunch\b/i,
|
||||
},
|
||||
{ id: 'verifiesFirstScreen', pattern: /\b(?:wait|find|is|get)\b[^\n]*orders/i },
|
||||
],
|
||||
forbidden: [
|
||||
// help workflow: install for a first install; reinstall only when
|
||||
// explicitly requested; never open an artifact path.
|
||||
{ id: 'noReinstall', pattern: /(?:^|\n)agent-device\s+reinstall\b/i },
|
||||
{ id: 'noOpenArtifactPath', pattern: /(?:^|\n)agent-device\s+open\s+[^\n]*\.apk\b/i },
|
||||
],
|
||||
},
|
||||
// Next-command quiz cases: captured output (pinned to the real renderer by
|
||||
// scripts/__tests__/help-conformance-sample-outputs.test.ts) plus a task,
|
||||
// scored by regex instead of the named expectation scorers above.
|
||||
{
|
||||
id: 'settle-diff-is-observation',
|
||||
docs: ['--help:first30'],
|
||||
task: `You already ran this command and observed its settled output:
|
||||
|
||||
${sampleText(SETTLE_TAIL_SAMPLE)}
|
||||
|
||||
Use the output already shown to determine whether the feed-search UI is present, then close the session. What command should run next?`,
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [{ id: 'plansClose', pattern: /(?:^|\n)(?:agent-device\s+)?close\b/i }],
|
||||
forbidden: [
|
||||
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
|
||||
{ id: 'noWait', pattern: /\bwait\b/i },
|
||||
{ id: 'noFind', pattern: /\bfind\b/i },
|
||||
{ id: 'noGet', pattern: /\bget\b/i },
|
||||
{ id: 'noIs', pattern: /\bis\b/i },
|
||||
{ id: 'noPressOrClick', pattern: /\b(?:press|click)\b/i },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-settled-diff-next-target',
|
||||
docs: ['--help:first30'],
|
||||
task: quiz(
|
||||
SETTLE_DIFF_SAMPLE,
|
||||
'The task is to open the matching account result. What command should run next?',
|
||||
),
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{ id: 'pressOrClickOrTap', pattern: /\b(?:press|click|tap)\b/i },
|
||||
{ id: 'usesE64RefOrLabel', pattern: /@e64\b|label=(?:["']?@callstack\.com["']?)/i },
|
||||
{ id: 'usesSettleFlag', pattern: /--settle\b/i },
|
||||
],
|
||||
forbidden: [
|
||||
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
|
||||
{ id: 'noWaitStable', pattern: /wait\s+stable/i },
|
||||
{ id: 'noFill', pattern: /\bfill\b/i },
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'metamorphic-settled-diff-next-target-notes',
|
||||
docs: ['--help:first30'],
|
||||
task: quiz(
|
||||
SETTLE_DIFF_SAMPLE_NOTES,
|
||||
'The task is to open the matching list result. What command should run next?',
|
||||
),
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{ id: 'pressOrClickOrTap', pattern: /\b(?:press|click|tap)\b/i },
|
||||
{ id: 'usesE21RefOrLabel', pattern: /@e21\b|label=(?:["']?groceries list["']?)/i },
|
||||
{ id: 'usesSettleFlag', pattern: /--settle\b/i },
|
||||
],
|
||||
forbidden: [
|
||||
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
|
||||
{ id: 'noWaitStable', pattern: /wait\s+stable/i },
|
||||
{ id: 'noFill', pattern: /\bfill\b/i },
|
||||
{ id: 'noCallstackLeakage', pattern: /(?:callstack|@e64)/i },
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-not-settled-needs-observe',
|
||||
docs: ['--help:first30'],
|
||||
task: quiz(
|
||||
NOT_SETTLED_SAMPLE,
|
||||
'The next target is not known yet. What command should run next?',
|
||||
),
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'observesBeforeActing',
|
||||
pattern: /(?:^|\n)(?:agent-device\s+)?(?:wait\b|snapshot\b[^\n]*-i\b)/i,
|
||||
},
|
||||
],
|
||||
forbidden: [
|
||||
{
|
||||
id: 'noBareRefMutation',
|
||||
pattern: /(?:^|\n)(?:agent-device\s+)?(?:press|click|fill|longpress)\s+@e\d+/i,
|
||||
},
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-device-in-use-reuses-session',
|
||||
docs: ['--help:first30'],
|
||||
task: quiz(
|
||||
DEVICE_IN_USE_SAMPLE,
|
||||
'You are continuing the checkout flow that the "checkout" session was already running on this device. What command should run next?',
|
||||
),
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'retriesWithOwningSession',
|
||||
pattern: /(?:^|\n)agent-device\s+press\b[^\n]*--session\s+checkout\b/i,
|
||||
},
|
||||
{ id: 'keepsSettle', pattern: /--settle\b/i },
|
||||
],
|
||||
forbidden: [
|
||||
{ id: 'noClose', pattern: /(?:^|\n)agent-device\s+close\b/i },
|
||||
{ id: 'noReopen', pattern: /(?:^|\n)agent-device\s+open\b/i },
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-stale-ref-resnapshots',
|
||||
docs: ['--help:first30'],
|
||||
task: quiz(
|
||||
STALE_REF_SAMPLE,
|
||||
'The Continue control this ref pointed at may have moved. What command should run next?',
|
||||
),
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{ id: 'refreshesInteractiveRefs', pattern: /(?:^|\n)agent-device\s+snapshot\s+-i\b/i },
|
||||
],
|
||||
forbidden: [
|
||||
{
|
||||
id: 'noBareRefRetry',
|
||||
pattern: /(?:^|\n)agent-device\s+(?:press|click|fill|longpress)\s+@e\d/i,
|
||||
},
|
||||
{ id: 'noReopen', pattern: /(?:^|\n)agent-device\s+open\b/i },
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-ambiguous-match-reobserves',
|
||||
docs: ['--help:first30'],
|
||||
task: quiz(
|
||||
AMBIGUOUS_MATCH_SAMPLE,
|
||||
'The intent is to follow the @callstack.com account row. The candidate refs were not shown. What command should run next?',
|
||||
),
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [
|
||||
{
|
||||
id: 'reobservesOrNarrows',
|
||||
pattern:
|
||||
/(?:^|\n)agent-device\s+(?:snapshot\s+-i\b|(?:find|press|click)\s+[^\n]*(?:role=|id=|label="?@callstack\.com))/i,
|
||||
},
|
||||
],
|
||||
forbidden: [
|
||||
// The candidates live in error details the human output never printed,
|
||||
// so a ref-targeting command here would be a guess.
|
||||
{ id: 'noGuessedRef', pattern: /(?:^|\n)agent-device\s+(?:press|click)\s+@e\d/i },
|
||||
{
|
||||
id: 'noVerbatimRetry',
|
||||
pattern: /(?:^|\n)agent-device\s+find\s+text\s+"?follow"?\s+press\b/i,
|
||||
},
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'sample-output-app-not-installed-discovers-first',
|
||||
docs: ['--help:first30'],
|
||||
task: quiz(
|
||||
APP_NOT_INSTALLED_SAMPLE,
|
||||
'The goal is still to open the shop app on this simulator; no build artifact was provided. What command should run next?',
|
||||
),
|
||||
expectations: ['validPlanCommands', 'fullPrefix'],
|
||||
matchers: [{ id: 'discoversInstalledApps', pattern: /(?:^|\n)agent-device\s+apps\b/i }],
|
||||
forbidden: [
|
||||
{ id: 'noBlindReopen', pattern: /(?:^|\n)agent-device\s+open\s+"?shoply\b/i },
|
||||
// No artifact exists to install; inventing one is the failure mode help
|
||||
// workflow forbids ("Do not open artifact paths or invent package ids").
|
||||
{
|
||||
id: 'noInventedInstall',
|
||||
pattern: /(?:^|\n)agent-device\s+(?:install|install-from-source)\b/i,
|
||||
},
|
||||
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
|
||||
],
|
||||
},
|
||||
];
|
||||
@@ -0,0 +1,13 @@
|
||||
export type CapturedSample = { command: string; output: string };
|
||||
|
||||
export declare function sampleText(sample: CapturedSample): string;
|
||||
|
||||
export declare const SETTLE_TAIL_SAMPLE: CapturedSample;
|
||||
export declare const SETTLE_DIFF_SAMPLE: CapturedSample;
|
||||
export declare const SETTLE_DIFF_SAMPLE_NOTES: CapturedSample;
|
||||
export declare const NOT_SETTLED_SAMPLE: CapturedSample;
|
||||
export declare const PRIVATE_AX_RECOVERY_SAMPLE: CapturedSample;
|
||||
export declare const DEVICE_IN_USE_SAMPLE: CapturedSample;
|
||||
export declare const STALE_REF_SAMPLE: CapturedSample;
|
||||
export declare const AMBIGUOUS_MATCH_SAMPLE: CapturedSample;
|
||||
export declare const APP_NOT_INSTALLED_SAMPLE: CapturedSample;
|
||||
@@ -0,0 +1,105 @@
|
||||
// Captured agent-device output embedded in benchmark quiz cases. Each sample
|
||||
// is { command, output }: the command line the prompt claims was run, and the
|
||||
// exact text the CLI would print for it. Every `output` is pinned to the real
|
||||
// renderer by scripts/__tests__/help-conformance-sample-outputs.test.ts, which
|
||||
// rebuilds it through src/commands/interaction/output.ts or
|
||||
// src/utils/output.ts printHumanError — a rendering change fails that test
|
||||
// instead of silently leaving the benchmark grading against stale output.
|
||||
|
||||
export function sampleText(sample) {
|
||||
return `${sample.command}\n${sample.output}`;
|
||||
}
|
||||
|
||||
// Settled press whose diff carried no added ref, so the response appended the
|
||||
// unchanged-interactive tail (#1167/#1172). The response carried a diff plus
|
||||
// refsGeneration, so the tail refs render pinned (ADR 0014).
|
||||
export const SETTLE_TAIL_SAMPLE = {
|
||||
command: 'agent-device press @e37 --settle',
|
||||
output: `Tapped @e37 (203, 88)
|
||||
settled after 540ms: +0 -1 (~15 unchanged)
|
||||
- @e50 [text] "Suggested for you"
|
||||
unchanged interactive (4):
|
||||
= @e64~s5 [text-field] "Search"
|
||||
= @e65~s5 [text] "Recent searches"
|
||||
= @e12~s5 [tab] "Home"
|
||||
= @e40~s5 [tab] "Profile"`,
|
||||
};
|
||||
|
||||
// Settled fill whose diff exposes the next target directly.
|
||||
export const SETTLE_DIFF_SAMPLE = {
|
||||
command: `agent-device fill 'id="account-search"' "callstack" --settle`,
|
||||
output: `Filled 9 chars
|
||||
settled after 610ms: +2 -0 (~18 unchanged)
|
||||
+ @e64 [button] "@callstack.com"
|
||||
+ @e65 [text] "Callstack"`,
|
||||
};
|
||||
|
||||
// Metamorphic twin of SETTLE_DIFF_SAMPLE on a different app and ref range, so
|
||||
// a pass cannot come from memorizing the callstack sample.
|
||||
export const SETTLE_DIFF_SAMPLE_NOTES = {
|
||||
command: `agent-device fill 'id="notes-search"' "groceries" --settle`,
|
||||
output: `Filled 9 chars
|
||||
settled after 480ms: +2 -0 (~11 unchanged)
|
||||
+ @e21 [button] "Groceries list"
|
||||
+ @e22 [text] "3 items"`,
|
||||
};
|
||||
|
||||
// Never-settled press: success response, no diff, NEVER_SETTLED_HINT attached.
|
||||
export const NOT_SETTLED_SAMPLE = {
|
||||
command: 'agent-device press @e12 --settle',
|
||||
output: `Tapped @e12 (166, 240)
|
||||
not settled after 10000ms
|
||||
hint: The UI kept changing for the whole settle budget (animation, carousel, or ticker?), so no settled diff is shown. Raise --timeout, wait for specific content, or take a fresh snapshot.`,
|
||||
};
|
||||
|
||||
// Recovered snapshot: the private-ax fallback fired but still exposed
|
||||
// actionable refs. Warning wording is renderSnapshotQualityWarnings
|
||||
// (src/snapshot/snapshot-quality.ts); lines are the structured snapshot
|
||||
// renderer (src/utils/output.ts formatSnapshotText).
|
||||
export const PRIVATE_AX_RECOVERY_SAMPLE = {
|
||||
command: 'agent-device snapshot -i',
|
||||
output: `Snapshot: 2 nodes
|
||||
Detected an overly complex or slow accessibility tree. Fell back to the private-ax snapshot backend. It is OK to continue; use --json to inspect snapshotQuality.reason if you need recovery details.
|
||||
@e5 [button] "Search"
|
||||
@e8 [tab] "Home" [selected]`,
|
||||
};
|
||||
|
||||
// DEVICE_IN_USE from buildDeviceInUseBySessionError
|
||||
// (src/daemon/handlers/session-open.ts) — the parity test drives that exact
|
||||
// producer.
|
||||
export const DEVICE_IN_USE_SAMPLE = {
|
||||
command: `agent-device press 'label="Place order"' --settle`,
|
||||
output: `Error (DEVICE_IN_USE): Device is already in use by session "checkout".
|
||||
Hint: Run agent-device session list to inspect active sessions. To reuse this device, rerun the command with --session checkout. To open a new session on this device, first run agent-device close --session checkout.`,
|
||||
};
|
||||
|
||||
// ADR 0014 mutation rejection from
|
||||
// src/daemon/handlers/interaction-ref-policy.ts: a pinned ref minted from a
|
||||
// superseded generation is rejected before dispatch. The daemon strips the
|
||||
// `~s5` pin at the boundary (interaction-touch-targets.ts), so the message
|
||||
// names the plain ref; the hint is the precise resolveRefStalenessWarning.
|
||||
export const STALE_REF_SAMPLE = {
|
||||
command: 'agent-device press @e12~s5 --settle',
|
||||
output: `Error (COMMAND_FAILED): Ref @e12 was minted from a superseded snapshot generation
|
||||
Hint: Ref @e12 was minted from snapshot s5 but the session's ref frame is now s7 — re-run snapshot -i.`,
|
||||
};
|
||||
|
||||
// AMBIGUOUS_MATCH from buildAmbiguousMatchError (src/daemon/handlers/find.ts)
|
||||
// — the parity test drives that exact producer. The by-design rejection
|
||||
// instead of silent disambiguation: candidate refs live in details, which the
|
||||
// human rendering does not print, so the agent must re-observe or narrow, not
|
||||
// guess a ref it never saw.
|
||||
export const AMBIGUOUS_MATCH_SAMPLE = {
|
||||
command: 'agent-device find text "Follow" press',
|
||||
output: `Error (AMBIGUOUS_MATCH): find matched 3 elements for text "Follow". Use a more specific locator or selector.
|
||||
Hint: Multiple candidates matched. Narrow the query or pass an exact identifier.`,
|
||||
};
|
||||
|
||||
// APP_NOT_INSTALLED from buildAppNotInstalledError
|
||||
// (src/platforms/apple/core/app-resolution.ts) — the parity test drives that
|
||||
// exact producer; the hint is defaultHintForCode('APP_NOT_INSTALLED').
|
||||
export const APP_NOT_INSTALLED_SAMPLE = {
|
||||
command: 'agent-device open Shoply --platform ios',
|
||||
output: `Error (APP_NOT_INSTALLED): No app found matching "Shoply"
|
||||
Hint: Run apps to discover the exact installed package or bundle id, or install the app before open.`,
|
||||
};
|
||||
@@ -1285,6 +1285,15 @@ Usage:
|
||||
`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Topic-id registry view for conformance tooling: the help benchmark's topic
|
||||
* coverage gate enumerates this instead of a hand-maintained list, so adding a
|
||||
* topic without benchmark coverage (or an explicit waiver) fails a test.
|
||||
*/
|
||||
export function helpTopicIds(): string[] {
|
||||
return Object.keys(HELP_TOPICS);
|
||||
}
|
||||
|
||||
function buildHelpTopicUsageText(topicName: string): string | null {
|
||||
const topic = HELP_TOPICS[topicName as keyof typeof HELP_TOPICS];
|
||||
if (!topic) return null;
|
||||
|
||||
@@ -542,7 +542,10 @@ function publicFindFlags(flags: DaemonRequest['flags']): Record<string, unknown>
|
||||
return { ...(stripInternalInteractionFlags(flags) ?? {}) };
|
||||
}
|
||||
|
||||
function buildAmbiguousMatchError(
|
||||
// Exported as the single AMBIGUOUS_MATCH producer so the help-benchmark
|
||||
// sample parity test renders the exact error this handler returns; a message
|
||||
// change here fails that gate instead of drifting past it.
|
||||
export function buildAmbiguousMatchError(
|
||||
matches: SnapshotState['nodes'],
|
||||
locator: FindLocator,
|
||||
query: string,
|
||||
|
||||
@@ -559,6 +559,17 @@ function findNewSessionDeviceConflict(params: {
|
||||
},
|
||||
);
|
||||
}
|
||||
return buildDeviceInUseBySessionError(inUse, device);
|
||||
}
|
||||
|
||||
// Exported as the single by-session DEVICE_IN_USE producer so the
|
||||
// help-benchmark sample parity test renders the exact error this handler
|
||||
// returns; a message or hint change here fails that gate instead of drifting
|
||||
// past it.
|
||||
export function buildDeviceInUseBySessionError(
|
||||
inUse: SessionState,
|
||||
device: DeviceInfo,
|
||||
): DaemonResponse {
|
||||
return errorResponse('DEVICE_IN_USE', `Device is already in use by session "${inUse.name}".`, {
|
||||
session: inUse.name,
|
||||
deviceId: device.id,
|
||||
|
||||
@@ -58,7 +58,14 @@ export async function resolveIosApp(device: DeviceInfo, app: string): Promise<st
|
||||
throw new AppError('INVALID_ARGS', `Multiple apps matched "${app}"`, { matches });
|
||||
}
|
||||
|
||||
throw new AppError('APP_NOT_INSTALLED', `No app found matching "${app}"`);
|
||||
throw buildAppNotInstalledError(app);
|
||||
}
|
||||
|
||||
// Exported as the single name-lookup APP_NOT_INSTALLED producer so the
|
||||
// help-benchmark sample parity test renders the exact error this resolver
|
||||
// throws; a message change here fails that gate instead of drifting past it.
|
||||
export function buildAppNotInstalledError(app: string): AppError {
|
||||
return new AppError('APP_NOT_INSTALLED', `No app found matching "${app}"`);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+19
-54
@@ -1,58 +1,29 @@
|
||||
# Skillgym For agent-device
|
||||
|
||||
This folder is a starter `skillgym` setup for benchmarking the `agent-device` skill with a controlled Expo target app.
|
||||
This folder benchmarks the one thing only an agentic runner can prove: given the `agent-device` skill, does a real agent route to it, consult local CLI help before answering, and read captured `agent-device` output correctly in that agentic setting?
|
||||
|
||||
## Why `skillgym` fits here
|
||||
Everything else this suite used to cover moved to cheaper, stronger owners:
|
||||
|
||||
`skillgym` is useful for `agent-device` in three layers:
|
||||
- Command-planning knowledge checks (can a model choose the right next command from a help slice, a captured output, or an error?) live in the help conformance bench (`scripts/help-conformance-bench.mjs`). Its plans are validated by the production CLI parser, its case list is enumerated against the help-topic registry (`scripts/__tests__/help-conformance-topic-coverage.test.ts`), and its quoted samples are pinned to the real renderers (`scripts/__tests__/help-conformance-sample-outputs.test.ts`).
|
||||
- Live fixture-app behavior is owned by the deterministic iOS simulator e2e suite (`test/integration/ios-simulator-e2e/`) and its coverage manifest.
|
||||
|
||||
1. Skill-routing checks: verify that the runner loads `skills/agent-device/SKILL.md` and its required references before it answers.
|
||||
2. Workflow-planning checks: verify that the agent describes the right `agent-device` loop for a known fixture app.
|
||||
3. Optional live-device smoke runs: locally, you can extend prompts so the agent actually drives `agent-device` against a simulator or device.
|
||||
## What stays here
|
||||
|
||||
The included suite focuses on the first two layers so it stays stable and CI-safe.
|
||||
The suite uses SkillGym v0.8 case tags:
|
||||
The suite uses two case tags:
|
||||
|
||||
- `fixture-smoke`: fixture-specific app surface coverage
|
||||
- `skill-guidance`: command-planning guidance regressions
|
||||
- `fixture-smoke`: one routing smoke (`open-and-snapshot`) — a sane fixture-app plan from the skill plus local help, without reading project source.
|
||||
- `skill-guidance`: four output-interpretation cases that require a REAL observed local help probe before scoring (`requireLocalCliHelp` + `suites/local-cli-help-policy.ts` over observed command events): `settle-diff-is-observation`, `sample-output-settled-diff-next-target`, `sample-output-not-settled-needs-observe`, `sample-output-private-ax-recovery-continues`. Allowing help without verifying that the runner actually read it can misclassify a model-prior failure as a help-guidance failure.
|
||||
|
||||
The captured output quoted in the skill-guidance cases is imported from `scripts/help-conformance-sample-outputs.mjs` — the same constants the bench embeds — so a renderer change fails the pinning test instead of silently grading agents against output the CLI no longer prints.
|
||||
|
||||
## Included files
|
||||
|
||||
- `../../examples/test-app/`: minimal Expo SDK 56 development-build fixture app for broad UI coverage
|
||||
- `../../examples/test-app/`: minimal Expo SDK 56 development-build fixture app
|
||||
- `skillgym.config.ts`: starter config that runs Codex and Claude Haiku against this repo
|
||||
- `suites/agent-device-smoke-suite.ts`: planning suite for skill routing, fixture-aware flows, and skill-guidance regressions
|
||||
- `suites/agent-device-smoke-suite.ts`: the routing + output-interpretation suite
|
||||
- `suites/local-cli-help-policy.ts`: the observed-command policy that proves a real local help command ran
|
||||
|
||||
## Current coverage
|
||||
|
||||
The suite keeps the app small while separating coverage into two non-overlapping groups.
|
||||
|
||||
Fixture smoke cases cover concrete app surfaces:
|
||||
|
||||
- open/snapshot/close defaults with the installed Expo development build
|
||||
- banners, alerts, toggles, and quick actions on Home
|
||||
- search debounce, filters, long-list scroll, favorites, and cart updates in Catalog
|
||||
- detail navigation, quantity edits, note append, and save-to-cart on Product
|
||||
- form validation, success submit, iOS keyboard-dismiss fallback, and reset on Checkout form
|
||||
- diagnostics load/error/retry plus reset alert handling in Settings
|
||||
- accessibility audit via screenshot + snapshot
|
||||
|
||||
Skill-guidance regression cases cover distinct command-planning habits:
|
||||
|
||||
- read-only inspection versus mutation
|
||||
- fresh `@ref` targeting, durable selectors, raw-rect fallbacks, and off-screen scroll recovery
|
||||
- interpreting representative `agent-device` output, including settled diffs, not-settled hints, and private-AX recovery warnings
|
||||
- text replacement, append semantics, supported field clearing, keyboard status, and keyboard fallback
|
||||
- install/open setup, Expo Go/dev-client launch paths, app discovery, session scoping, and app-owned navigation fallbacks
|
||||
- Metro reload, logs, network dump, alert fallback, and screenshot evidence
|
||||
- performance metrics, React DevTools profiling, gestures, settings, and trace capture
|
||||
- Android TV and Vega OS remote planning, including focus moves, exact holds, and Vega VVD lifecycle
|
||||
- remote config, macOS menu bar surfaces, replay update, same-session mutation ordering, and batch schema/recording
|
||||
|
||||
Use SkillGym for stable behavior regressions: can a runner choose the right next command from help, app-contract facts, or representative CLI output? For rapid help-layout A/B testing, prefer the lighter help conformance bench (`scripts/help-conformance-bench.mjs`) because it can feed only the top-level first screen or one help topic without letting the runner read the full help page, it runs runner x case pairs concurrently (`HELP_BENCH_CONCURRENCY`, default 4), and it can grade a draft help rewrite with zero rebuild via `--override-doc <topicId>=<path>` (repeatable; last occurrence per topic wins), which loads that file's contents in place of shelling out to `node bin/agent-device.mjs help <topicId>` for that one topic while still applying the same post-processing as the live source (the `--help:first30` doc id stays capped to its first 30 lines), so the A/B grade compares like with like; a topic id no selected case uses fails fast instead of silently grading the real doc. Returned plans are checked against the production CLI parser and interaction grammar, plus a narrow executable allowlist and shell-projection guard, so invented flags, malformed selectors, placeholder refs, redirection, and unsupported helper commands fail even when keyword checks pass. Use `--repeat <n>` to measure variance: every runner x case pair gets numbered independent trials plus an aggregate pass-rate and failure-taxonomy report. The `metamorphic-community-search` case renames the app, controls, query, and account from the original search flow and explicitly forbids original-answer leakage, providing a cheap generalization check. The bench also includes four deliberately uncoached "next-command quiz" cases (`settle-diff-is-observation`, `sample-output-settled-diff-next-target`, `sample-output-not-settled-needs-observe`, and `sample-output-recoverable-failure-retries-in-session`) ported from this suite's skill-guidance regressions and the routine-workflow oracle (`test/output-economy/routine-workflow.ts`), for scoring whether a runner can infer the next action from representative captured `agent-device` output — including the `--settle` "unchanged interactive (N):" tail and an actionable settle timeout that must be recovered in-session rather than by reopening or re-observing — instead of being told the answer by the task. Filter with `--cases`/`--case` and `--runners`/`--runner` (both repeatable/CSV) the same way as this suite.
|
||||
|
||||
Parser-backed validation proves command syntax, positional bounds, and interaction grammar; it does not prove runtime semantics for intentionally free-form values such as `open <appOrUrl>`. A selector-looking app string can therefore be syntactically valid even when it would not resolve to an installed app.
|
||||
|
||||
The SkillGym output-interpretation cases require an observed local `--help` lookup before scoring. Allowing help without verifying that the runner actually read it can misclassify a model-prior failure as a help-guidance failure.
|
||||
For help-layout A/B testing and command-planning regressions, use the bench instead of adding cases here: it feeds only the top-level first screen or one help topic, runs runner x case pairs concurrently (`HELP_BENCH_CONCURRENCY`, default 4), grades draft help rewrites with zero rebuild via `--override-doc <topicId>=<path>`, measures variance with `--repeat <n>` plus an aggregate pass-rate and failure-taxonomy report, and filters with `--cases`/`--case` and `--runners`/`--runner` (both repeatable/CSV) the same way as this suite.
|
||||
|
||||
`assertAgentDeviceEvidence` is intentionally soft when a runner does not expose skill-detection telemetry. When telemetry exists, the suite asserts that `agent-device` was loaded; when it is absent, the cases still judge command-planning output instead of failing on missing runner metadata.
|
||||
|
||||
@@ -62,15 +33,9 @@ SkillGym v0.8 command assertions are for observed command events. This suite pri
|
||||
The source-read guardrails use `assert.soft.*` plus deferred explain questions so one failing run can report multiple routing mistakes and can later be inspected with `skillgym explain`.
|
||||
Suite types use the v0.8 root export name `Case`; older `TestCase` imports no longer typecheck.
|
||||
|
||||
## Suggested workflow
|
||||
|
||||
1. Start with the included smoke suite to benchmark routing and default guidance.
|
||||
2. Extend the suite with app-specific prompts that cover a new command-planning category rather than duplicating an existing one.
|
||||
3. Add local-only cases that expect real `agent-device` shell commands once you are ready to involve a running simulator.
|
||||
|
||||
## Running the suite
|
||||
|
||||
`skillgym` is installed as a repo dev dependency, so run the starter suite from the project root:
|
||||
`skillgym` is installed as a repo dev dependency, so run the suite from the project root:
|
||||
|
||||
```bash
|
||||
cd /absolute/path/to/agent-device
|
||||
@@ -89,7 +54,7 @@ pnpm test:skillgym:case open-and-snapshot
|
||||
Useful v0.8 filters, reporters, and recovery options:
|
||||
|
||||
```bash
|
||||
pnpm test:skillgym -- --tag fixture-smoke
|
||||
pnpm test:skillgym -- --tag skill-guidance
|
||||
pnpm test:skillgym -- --reporter json
|
||||
pnpm test:skillgym -- --repeat 3 --repeat-failure 1
|
||||
```
|
||||
@@ -133,6 +98,6 @@ The configured runners call external Codex and Claude model backends. In Codex s
|
||||
|
||||
## Where to extend next
|
||||
|
||||
- Add suite cases that ask for selector-based plans against `Agent Device Tester`.
|
||||
- Add local-only prompts that expect `agent-device open`, `snapshot`, `snapshot -i`, `get`, and `wait`.
|
||||
- Add regression snapshots once the prompt set stabilizes.
|
||||
- Add a case here ONLY when it measures agentic behavior: skill routing, local-help consultation, or multi-turn discovery a single non-agentic call cannot show.
|
||||
- Add command-planning knowledge checks to `scripts/help-conformance-cases.mjs` instead; the topic-coverage gate tells you which help topics are unbenchmarked.
|
||||
- Add local-only cases that drive a live simulator only when the deterministic e2e suite cannot own the behavior.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -38,6 +38,8 @@ export default defineConfig({
|
||||
include: [
|
||||
'src/**/*.test.ts',
|
||||
'scripts/__tests__/help-conformance-bench.test.ts',
|
||||
'scripts/__tests__/help-conformance-sample-outputs.test.ts',
|
||||
'scripts/__tests__/help-conformance-topic-coverage.test.ts',
|
||||
'test/skillgym/suites/local-cli-help-policy.test.ts',
|
||||
// The Maestro conformance oracle runs via `node --test` in its own CI
|
||||
// job (scripts/maestro-conformance), like the layering guard.
|
||||
|
||||
Reference in New Issue
Block a user