mirror of
https://github.com/callstack/agent-device.git
synced 2026-09-14 20:06:34 +08:00
fix(eval): narrow the conclusions to the invocations actually tested
Review follow-up on #2400. Three claims went further than the evidence. - The report recommended waiting for a terse output mode. --legend=compact is already documented at the tested revision. Measured: it costs no recall and saves 55% on --callers, 10% on --affected, 1% on --for. The arms ran on default output, and their calls were mostly --for-shaped, so compact narrows the +10% token gap rather than closing it — stated as the open question it is rather than a reason to defer. retrieval-bench gains a for-compact variant so the comparison is reproducible. - The CLI-flag head-to-head tested --for and --pack-task, which reach 1-3 of the five documented declaration sites, and concluded a call graph cannot recover a convention. --recall, the verb built for document questions, finds all five. That conclusion was an artifact of the verb chosen and is withdrawn. - --affected scored a precision figure against the historical commit's file list. It answers transitive reach, so a test it names that the commit left alone is not a false positive and that list is not a precision oracle. The field is removed; recall stands, and selection breadth is reported as read cost rather than a defect. Recall and every A/B number are unchanged; the corrections are to framing, one withdrawn conclusion, and one removed metric. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F7vMn8ehPq3NTPYMfxs7ro
This commit is contained in:
Claude
@@ -26,11 +26,15 @@ What did show up, and is worth keeping in view:
|
||||
1. **It is consistent where grep is lucky.** On the one task with a non-obvious touch point — a
|
||||
scripted provider fake that throws on unscripted calls — both ripwire runs found it and only one
|
||||
of two baseline runs did.
|
||||
2. **The token cost is a fixable implementation detail, not a design limit.** ripwire's
|
||||
self-documenting preamble is a *fixed* 1.6–3.1 KB per invocation, up to 62% of a small verb's
|
||||
whole response, re-sent on every call. A terse mode would likely flip the token column.
|
||||
3. **The cheap deterministic verbs stand on their own.** `--recall` answers from 799 KB of markdown
|
||||
in 15 KB; `--affected` names the right test files at 2–5 KB when the seed set is narrow.
|
||||
2. **Part of the token cost is already avoidable today.** ripwire's self-documenting preamble is a
|
||||
*fixed* 1.6–3.1 KB per invocation, up to 62% of a small verb's whole response. The documented
|
||||
`--legend=compact` removes most of it — but by verb: **−55% on `--callers`, −10% on `--affected`,
|
||||
−1% on `--for`**. The A/B arm's calls were mostly `--for`-shaped, so compact would not have
|
||||
closed the +10% gap; it would have helped an agent that leaned on the narrow verbs.
|
||||
3. **The document verbs are the strongest thing here.** `--recall` answers from 799 KB of markdown
|
||||
in 15 KB, and it finds all five CLI-flag declaration sites this repo documents by hand — where
|
||||
the graph verbs find one to three. `--affected` recalls 9 of 12 touched test files at 2–5 KB
|
||||
when the seed set is narrow.
|
||||
|
||||
## What was measured
|
||||
|
||||
@@ -133,10 +137,16 @@ Both misses are real. T1's are two `packages/maestro` harnesses the walk did not
|
||||
`snapshot-route.test.ts`, which covers the direct caller of the changed module and should have
|
||||
been a short hop.
|
||||
|
||||
Selection breadth is the sharper problem. T2 named **185** test files for a 5-file change: the
|
||||
seeds reach into `packages/kernel`, whose symbols are called from everywhere, and the walk has no
|
||||
notion of "this hub is not evidence". At that width the answer costs more to read than it saves.
|
||||
`--affected` is useful here at 2–24 selected files and not useful at 185.
|
||||
**Recall only, deliberately.** `--affected` answers "which tests can transitively reach this
|
||||
change" — a strictly larger set than "which tests this commit happened to edit". A test it names
|
||||
that the commit left alone is not a false positive, so the commit's file list cannot score
|
||||
precision, and an earlier draft of this report was wrong to treat the extra names as excess.
|
||||
`selected` below is breadth — what it would cost to read or run the set — not a quality score.
|
||||
|
||||
Breadth still varies enough to matter in practice. T2 named **185** test files for a 5-file change,
|
||||
because its seeds reach into `packages/kernel`, whose symbols are called from everywhere. That is a
|
||||
correct transitive answer and an expensive one; at 2–24 files the same verb is cheap to act on.
|
||||
Whether the wide answer is *useful* is a judgement about read cost, not a measured defect.
|
||||
|
||||
This does not overlap `pnpm check:affected`, which selects CI *lanes* from a diff. `--affected`
|
||||
selects test *files* from source files. They answer different questions.
|
||||
@@ -231,36 +241,38 @@ repository, ripwire's self-documenting XML comment preamble is **1.6 KB on `--fo
|
||||
hardest on exactly the cheap, narrow verbs that should be the tool's best value. Whole-file reads
|
||||
went down; total context did not.
|
||||
|
||||
This is the single most actionable finding here, and it is a fixable one: the preamble is
|
||||
documentation aimed at a first-time reader, re-sent to an agent that has already read it. A
|
||||
`--terse` mode that emits the header once per session — or not at all — would likely flip the token
|
||||
column without touching the ranking.
|
||||
**How much of this is avoidable today.** ripwire already ships `--legend=compact` (documented in
|
||||
its `docs/COMMANDS.md`); the arms were run on the default legend, so the +10% above is a
|
||||
default-output number. Measured on this repository, compact costs nothing in recall and saves
|
||||
**55% on `--callers`, 10% on `--affected`, and 1% on `--for`** — it strips the whole preamble from
|
||||
the narrow verbs and only part of it from `--for`, whose bulk is ranked rows, not header. Since the
|
||||
ripwire arm's calls were mostly `--for`/`--pack-task`-shaped, compact would have trimmed the gap
|
||||
rather than closed it. Re-running the A/B under `--legend=compact` is the open question this
|
||||
evaluation does not answer.
|
||||
|
||||
## 4. The one question this repo has already answered in prose
|
||||
|
||||
`docs/agents/cli-flags.md` names, by hand, the declaration sites a new CLI flag must be threaded
|
||||
through. That makes it the cleanest possible head-to-head between a call graph and a maintained
|
||||
routing doc. Asked the same question, one ripwire call names:
|
||||
through — the cleanest available head-to-head between a maintained routing doc and the tool.
|
||||
|
||||
| Declaration site (from `docs/agents/cli-flags.md`) | `--for="<task in prose>"` | `--for="<the type and helper names>"` |
|
||||
| --- | --- | --- |
|
||||
| `packages/contracts/src/cli-flags.ts` | — | yes |
|
||||
| `src/commands/cli-grammar/*` | yes | yes |
|
||||
| `src/commands/command-projection.ts` | — | — |
|
||||
| `src/cli-schema/command-overrides.ts` | — | — |
|
||||
| `src/cli-schema/cli-config.ts` | — | yes |
|
||||
|
||||
`--pack-task --partition=3`, the verb aimed at fanning work out to parallel agents, produces three
|
||||
slices with `overlap_max=0.000` in 1.3 s and 25 KB total — a clean split, naming 44 files, 2 of
|
||||
these 5 sites among them.
|
||||
|
||||
**A ranked call graph does not recover a convention.** These sites are related by a rule the team
|
||||
wrote down, not by call edges: `PROJECT_CONFIG_FLAG_KEYS` is a positive allowlist, and
|
||||
`SCHEMA_ONLY_CLI_COMMAND_SCHEMAS` is a merge path. Nothing in the graph says "and also this". The
|
||||
routing doc stays the better answer to this particular question, and that is the shape of the
|
||||
boundary — ripwire finds what the code *does*, `AGENTS.md` records what the team *decided*.
|
||||
| Declaration site (from `docs/agents/cli-flags.md`) | `--for` (prose) | `--for` (identifiers) | `--recall` |
|
||||
| --- | --- | --- | --- |
|
||||
| `packages/contracts/src/cli-flags.ts` | — | yes | **yes** |
|
||||
| `src/commands/cli-grammar/*` | yes | yes | **yes** |
|
||||
| `src/commands/command-projection.ts` | — | — | **yes** |
|
||||
| `src/cli-schema/command-overrides.ts` | — | — | **yes** |
|
||||
| `src/cli-schema/cli-config.ts` | — | yes | **yes** |
|
||||
|
||||
**`--recall` finds all five, in 15.7 KB.** That is the verb built for this question: it searches the
|
||||
written corpus, and the answer to "where does a CLI flag get threaded" lives in a document, not in
|
||||
call edges. An earlier draft of this report tested only `--for` and `--pack-task`, which reach 1–3
|
||||
of the five, and concluded from that "a ranked call graph does not recover a convention". That
|
||||
conclusion was an artifact of the verb chosen, and is withdrawn.
|
||||
|
||||
What survives is narrower and less interesting: **the graph verbs are the wrong tool for a
|
||||
convention, and ripwire knows it** — the routing table in its own skills sends this question to
|
||||
`--recall`. `--pack-task --partition=3`, the fan-out verb, produces three slices with
|
||||
`overlap_max=0.000` in 1.3 s and 25 KB total, naming 44 files, 2 of these 5 sites among them.
|
||||
|
||||
## Adoption cost, if we wanted it
|
||||
|
||||
@@ -292,13 +304,14 @@ line, sends nothing anywhere and needs no key, so the cost of one engineer tryin
|
||||
The verbs worth trying first here are `--recall` (52× cheaper than the doc corpus it searches) and
|
||||
`--affected` on a narrow seed set.
|
||||
|
||||
**Re-run this harness if ripwire ships a terse output mode.** `scripts/ripwire-eval/` is written to
|
||||
be re-run against a new binary with two commands; the token result is the one number most likely to
|
||||
move, and it is the one currently deciding the verdict.
|
||||
**Re-run the A/B under `--legend=compact` before treating the token result as settled.** The arms
|
||||
ran on default output. Compact is measured above and is worth 55% on `--callers` but only 1% on
|
||||
`--for`, so it should narrow rather than close the gap — but that is an inference from the
|
||||
deterministic benches, not a measurement of the agent arm, and the harness exists to settle it.
|
||||
|
||||
**Two findings are worth sending upstream**, since both are measured rather than impressionistic:
|
||||
the fixed preamble cost per invocation, and `--affected` selecting 185 test files for a 5-file
|
||||
change once its seeds reach a hub module in `packages/kernel`.
|
||||
**One finding is worth sending upstream:** the fixed preamble is 15% of a `--for` response and 62%
|
||||
of a `--callers` one, and `--legend=compact` clears it from the latter but not the former — the
|
||||
default is the expensive one, on the verb agents reach for most.
|
||||
|
||||
## Caveats
|
||||
|
||||
@@ -314,3 +327,6 @@ change once its seeds reach a hub module in `packages/kernel`.
|
||||
than folded into per-run wall clock.
|
||||
- Change-set localization is one job among many. This says nothing about ripwire's refactoring,
|
||||
security or quality lenses beyond the single `--quality-panel` run noted above.
|
||||
- **Results are bounded by the invocations tested, and the choice of verb changed conclusions.**
|
||||
The A/B ran on default-legend output; §4's original finding reversed once `--recall` was tried
|
||||
instead of `--for`. Read every number here as "this verb, this flag set", not "the tool".
|
||||
|
||||
@@ -6,6 +6,12 @@
|
||||
// truth files of the real commit and asks whether the commit's own TEST files come back, and at
|
||||
// what cost.
|
||||
//
|
||||
// RECALL ONLY, deliberately. `--affected` answers "which tests can transitively reach this change",
|
||||
// which is a strictly larger set than "which tests the commit happened to edit". A test it names
|
||||
// that the commit left alone is not a false positive — the commit's file list is not a precision
|
||||
// oracle for a reach query. `selected` is therefore reported as breadth (what it would cost to run
|
||||
// or read the set), never scored against the ground truth.
|
||||
//
|
||||
// Usage: node scripts/ripwire-eval/affected-bench.mjs --ripwire=<bin> --worktrees=<dir> [--out=<file>]
|
||||
|
||||
import { writeFileSync } from 'node:fs';
|
||||
@@ -68,8 +74,6 @@ for (const task of loadTasks()) {
|
||||
helpers_not_scored: helpers,
|
||||
hit: hit.length,
|
||||
recall: Number((hit.length / expected.length).toFixed(3)),
|
||||
// Of the tests it named, how many were actually touched — the cost of running the whole set.
|
||||
precision: selected.length === 0 ? 0 : Number((hit.length / selected.length).toFixed(3)),
|
||||
missed: expected.filter((path) => !selected.includes(path)),
|
||||
});
|
||||
process.stderr.write(
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
{
|
||||
"generated": "2026-09-08T16:04:40.961Z",
|
||||
"generated": "2026-09-09T06:24:35.118Z",
|
||||
"results": [
|
||||
{
|
||||
"task": "T1",
|
||||
"failed": null,
|
||||
"ms": 3972,
|
||||
"ms": 543,
|
||||
"bytes": 8998,
|
||||
"seeds": 10,
|
||||
"selected": 65,
|
||||
@@ -13,7 +13,6 @@
|
||||
"helpers_not_scored": ["packages/maestro/src/internal/__tests__/runtime-port-fixtures.ts"],
|
||||
"hit": 2,
|
||||
"recall": 0.5,
|
||||
"precision": 0.031,
|
||||
"missed": [
|
||||
"packages/maestro/src/internal/__tests__/program-ir-parser.test.ts",
|
||||
"packages/maestro/src/internal/__tests__/runtime-port.test.ts"
|
||||
@@ -22,7 +21,7 @@
|
||||
{
|
||||
"task": "T2",
|
||||
"failed": null,
|
||||
"ms": 2757,
|
||||
"ms": 486,
|
||||
"bytes": 19711,
|
||||
"seeds": 6,
|
||||
"selected": 185,
|
||||
@@ -31,13 +30,12 @@
|
||||
"helpers_not_scored": [],
|
||||
"hit": 1,
|
||||
"recall": 1,
|
||||
"precision": 0.005,
|
||||
"missed": []
|
||||
},
|
||||
{
|
||||
"task": "T3",
|
||||
"failed": null,
|
||||
"ms": 2484,
|
||||
"ms": 925,
|
||||
"bytes": 2447,
|
||||
"seeds": 1,
|
||||
"selected": 5,
|
||||
@@ -50,13 +48,12 @@
|
||||
],
|
||||
"hit": 1,
|
||||
"recall": 1,
|
||||
"precision": 0.2,
|
||||
"missed": []
|
||||
},
|
||||
{
|
||||
"task": "T4",
|
||||
"failed": null,
|
||||
"ms": 2524,
|
||||
"ms": 493,
|
||||
"bytes": 4380,
|
||||
"seeds": 4,
|
||||
"selected": 24,
|
||||
@@ -65,13 +62,12 @@
|
||||
"helpers_not_scored": ["test/wire-compat/surface.ts"],
|
||||
"hit": 2,
|
||||
"recall": 1,
|
||||
"precision": 0.083,
|
||||
"missed": []
|
||||
},
|
||||
{
|
||||
"task": "T5",
|
||||
"failed": null,
|
||||
"ms": 2840,
|
||||
"ms": 475,
|
||||
"bytes": 3652,
|
||||
"seeds": 2,
|
||||
"selected": 18,
|
||||
@@ -80,13 +76,12 @@
|
||||
"helpers_not_scored": [],
|
||||
"hit": 2,
|
||||
"recall": 1,
|
||||
"precision": 0.111,
|
||||
"missed": []
|
||||
},
|
||||
{
|
||||
"task": "T6",
|
||||
"failed": null,
|
||||
"ms": 2179,
|
||||
"ms": 477,
|
||||
"bytes": 2123,
|
||||
"seeds": 1,
|
||||
"selected": 2,
|
||||
@@ -95,7 +90,6 @@
|
||||
"helpers_not_scored": [],
|
||||
"hit": 1,
|
||||
"recall": 0.5,
|
||||
"precision": 0.5,
|
||||
"missed": ["packages/platform-apple/src/snapshot-route.test.ts"]
|
||||
}
|
||||
]
|
||||
|
||||
@@ -32,6 +32,10 @@ const VERBS = [
|
||||
},
|
||||
// Same verb, but fed only the identifiers the task text itself puts in backticks — a mechanical
|
||||
// distillation, not a hand-tuned query. Isolates how much of --for's result is phrasing.
|
||||
// Same query, with the documented compact legend. Isolates how much of the default output is
|
||||
// the self-documenting preamble — the answer differs sharply by verb, so it is measured, not
|
||||
// assumed (docs/COMMANDS.md `--legend=full|compact`).
|
||||
{ id: 'for-compact', args: (task) => ['.', `--for=${task.prompt}`, '--legend=compact'] },
|
||||
{
|
||||
id: 'for-idents',
|
||||
args: (task) => ['.', `--for=${backtickedTerms(task.prompt)}`],
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
{
|
||||
"generated": "2026-09-08T13:06:03.406Z",
|
||||
"generated": "2026-09-09T06:24:31.660Z",
|
||||
"results": [
|
||||
{
|
||||
"task": "T1",
|
||||
"verb": "for",
|
||||
"failed": null,
|
||||
"ms": 5252,
|
||||
"ms": 1464,
|
||||
"bytes": 10354,
|
||||
"est_tokens": 2589,
|
||||
"paths_mentioned": 43,
|
||||
@@ -85,7 +85,7 @@
|
||||
"task": "T1",
|
||||
"verb": "pack-task",
|
||||
"failed": null,
|
||||
"ms": 4661,
|
||||
"ms": 1273,
|
||||
"bytes": 12413,
|
||||
"est_tokens": 3103,
|
||||
"paths_mentioned": 20,
|
||||
@@ -165,7 +165,7 @@
|
||||
"task": "T1",
|
||||
"verb": "pack-task-4k",
|
||||
"failed": null,
|
||||
"ms": 869,
|
||||
"ms": 747,
|
||||
"bytes": 8821,
|
||||
"est_tokens": 2205,
|
||||
"paths_mentioned": 13,
|
||||
@@ -241,11 +241,91 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T1",
|
||||
"verb": "for-compact",
|
||||
"failed": null,
|
||||
"ms": 1341,
|
||||
"bytes": 10173,
|
||||
"est_tokens": 2543,
|
||||
"paths_mentioned": 47,
|
||||
"ground_truth": 16,
|
||||
"ground_truth_basis": "existing-files-only",
|
||||
"hits": 5,
|
||||
"recall": 0.313,
|
||||
"best_rank": 2,
|
||||
"per_file": [
|
||||
{
|
||||
"path": "packages/maestro/src/internal/__tests__/program-ir-parser.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/__tests__/runtime-port-fixtures.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/__tests__/runtime-port.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/conformance-normalize.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/program-ir-command-parser.ts",
|
||||
"rank": 18
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/program-ir.ts",
|
||||
"rank": 10
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/runtime-port-commands.ts",
|
||||
"rank": 3
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/runtime-port-types.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/maestro/src/internal/support-matrix.ts",
|
||||
"rank": 8
|
||||
},
|
||||
{
|
||||
"path": "scripts/fuzz/validation-arbitraries-maestro.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "scripts/maestro-conformance/build-manifest.mjs",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "scripts/maestro-conformance/corpus/manifest.json",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/adapters/maestro/__tests__/daemon-runtime-port.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/adapters/maestro/__tests__/daemon-runtime-public-operation.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/adapters/maestro/daemon-runtime-port.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/adapters/maestro/daemon-runtime-public-operation.ts",
|
||||
"rank": 2
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T1",
|
||||
"verb": "for-idents",
|
||||
"failed": null,
|
||||
"ms": 947,
|
||||
"ms": 1007,
|
||||
"bytes": 10613,
|
||||
"est_tokens": 2653,
|
||||
"paths_mentioned": 45,
|
||||
@@ -325,7 +405,7 @@
|
||||
"task": "T2",
|
||||
"verb": "for",
|
||||
"failed": null,
|
||||
"ms": 5186,
|
||||
"ms": 695,
|
||||
"bytes": 10296,
|
||||
"est_tokens": 2574,
|
||||
"paths_mentioned": 43,
|
||||
@@ -369,7 +449,7 @@
|
||||
"task": "T2",
|
||||
"verb": "pack-task",
|
||||
"failed": null,
|
||||
"ms": 4583,
|
||||
"ms": 1076,
|
||||
"bytes": 11590,
|
||||
"est_tokens": 2898,
|
||||
"paths_mentioned": 10,
|
||||
@@ -413,7 +493,7 @@
|
||||
"task": "T2",
|
||||
"verb": "pack-task-4k",
|
||||
"failed": null,
|
||||
"ms": 951,
|
||||
"ms": 658,
|
||||
"bytes": 8137,
|
||||
"est_tokens": 2034,
|
||||
"paths_mentioned": 5,
|
||||
@@ -453,11 +533,55 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T2",
|
||||
"verb": "for-compact",
|
||||
"failed": null,
|
||||
"ms": 696,
|
||||
"bytes": 10131,
|
||||
"est_tokens": 2533,
|
||||
"paths_mentioned": 46,
|
||||
"ground_truth": 7,
|
||||
"ground_truth_basis": "existing-files-only",
|
||||
"hits": 2,
|
||||
"recall": 0.286,
|
||||
"best_rank": 1,
|
||||
"per_file": [
|
||||
{
|
||||
"path": "android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeXml.java",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/kernel/src/snapshot.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/platform-android/src/ui-hierarchy-builder.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/platform-android/src/ui-hierarchy-node.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/platform-android/src/ui-hierarchy.ts",
|
||||
"rank": 42
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/__tests__/response-views.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/response-views.ts",
|
||||
"rank": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T2",
|
||||
"verb": "for-idents",
|
||||
"failed": null,
|
||||
"ms": 875,
|
||||
"ms": 646,
|
||||
"bytes": 10653,
|
||||
"est_tokens": 2663,
|
||||
"paths_mentioned": 37,
|
||||
@@ -501,7 +625,7 @@
|
||||
"task": "T3",
|
||||
"verb": "for",
|
||||
"failed": null,
|
||||
"ms": 4943,
|
||||
"ms": 806,
|
||||
"bytes": 9748,
|
||||
"est_tokens": 2437,
|
||||
"paths_mentioned": 34,
|
||||
@@ -537,7 +661,7 @@
|
||||
"task": "T3",
|
||||
"verb": "pack-task",
|
||||
"failed": null,
|
||||
"ms": 4737,
|
||||
"ms": 1457,
|
||||
"bytes": 9695,
|
||||
"est_tokens": 2424,
|
||||
"paths_mentioned": 8,
|
||||
@@ -573,7 +697,7 @@
|
||||
"task": "T3",
|
||||
"verb": "pack-task-4k",
|
||||
"failed": null,
|
||||
"ms": 981,
|
||||
"ms": 776,
|
||||
"bytes": 8476,
|
||||
"est_tokens": 2119,
|
||||
"paths_mentioned": 7,
|
||||
@@ -605,11 +729,47 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T3",
|
||||
"verb": "for-compact",
|
||||
"failed": null,
|
||||
"ms": 690,
|
||||
"bytes": 9607,
|
||||
"est_tokens": 2402,
|
||||
"paths_mentioned": 37,
|
||||
"ground_truth": 5,
|
||||
"ground_truth_basis": "existing-files-only",
|
||||
"hits": 1,
|
||||
"recall": 0.2,
|
||||
"best_rank": 8,
|
||||
"per_file": [
|
||||
{
|
||||
"path": "packages/platform-android/src/__tests__/input-actions.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/platform-android/src/__tests__/test-utils/fake-adb.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/platform-android/src/input-actions.ts",
|
||||
"rank": 8
|
||||
},
|
||||
{
|
||||
"path": "test/integration/provider-scenarios/android-ime-lifecycle-world.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "test/integration/provider-scenarios/android-world.ts",
|
||||
"rank": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T3",
|
||||
"verb": "for-idents",
|
||||
"failed": null,
|
||||
"ms": 1016,
|
||||
"ms": 1044,
|
||||
"bytes": 10718,
|
||||
"est_tokens": 2680,
|
||||
"paths_mentioned": 41,
|
||||
@@ -645,7 +805,7 @@
|
||||
"task": "T4",
|
||||
"verb": "for",
|
||||
"failed": null,
|
||||
"ms": 5223,
|
||||
"ms": 740,
|
||||
"bytes": 10118,
|
||||
"est_tokens": 2530,
|
||||
"paths_mentioned": 41,
|
||||
@@ -689,7 +849,7 @@
|
||||
"task": "T4",
|
||||
"verb": "pack-task",
|
||||
"failed": null,
|
||||
"ms": 4522,
|
||||
"ms": 1428,
|
||||
"bytes": 12724,
|
||||
"est_tokens": 3181,
|
||||
"paths_mentioned": 19,
|
||||
@@ -733,7 +893,7 @@
|
||||
"task": "T4",
|
||||
"verb": "pack-task-4k",
|
||||
"failed": null,
|
||||
"ms": 960,
|
||||
"ms": 730,
|
||||
"bytes": 9044,
|
||||
"est_tokens": 2261,
|
||||
"paths_mentioned": 15,
|
||||
@@ -773,11 +933,55 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T4",
|
||||
"verb": "for-compact",
|
||||
"failed": null,
|
||||
"ms": 731,
|
||||
"bytes": 10059,
|
||||
"est_tokens": 2515,
|
||||
"paths_mentioned": 45,
|
||||
"ground_truth": 7,
|
||||
"ground_truth_basis": "existing-files-only",
|
||||
"hits": 3,
|
||||
"recall": 0.429,
|
||||
"best_rank": 1,
|
||||
"per_file": [
|
||||
{
|
||||
"path": "src/daemon/__tests__/http-server-tenant-trust.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/__tests__/request-diagnostics-http.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/request-diagnostics-http.ts",
|
||||
"rank": 1
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/server/http-server.ts",
|
||||
"rank": 5
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/server/tenant-trust.ts",
|
||||
"rank": 4
|
||||
},
|
||||
{
|
||||
"path": "src/daemon/session-tenant-scope.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "test/wire-compat/surface.ts",
|
||||
"rank": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T4",
|
||||
"verb": "for-idents",
|
||||
"failed": null,
|
||||
"ms": 914,
|
||||
"ms": 692,
|
||||
"bytes": 9641,
|
||||
"est_tokens": 2410,
|
||||
"paths_mentioned": 44,
|
||||
@@ -821,7 +1025,7 @@
|
||||
"task": "T5",
|
||||
"verb": "for",
|
||||
"failed": null,
|
||||
"ms": 4826,
|
||||
"ms": 746,
|
||||
"bytes": 10059,
|
||||
"est_tokens": 2515,
|
||||
"paths_mentioned": 35,
|
||||
@@ -853,7 +1057,7 @@
|
||||
"task": "T5",
|
||||
"verb": "pack-task",
|
||||
"failed": null,
|
||||
"ms": 4562,
|
||||
"ms": 1010,
|
||||
"bytes": 13150,
|
||||
"est_tokens": 3288,
|
||||
"paths_mentioned": 28,
|
||||
@@ -885,7 +1089,7 @@
|
||||
"task": "T5",
|
||||
"verb": "pack-task-4k",
|
||||
"failed": null,
|
||||
"ms": 948,
|
||||
"ms": 657,
|
||||
"bytes": 9018,
|
||||
"est_tokens": 2255,
|
||||
"paths_mentioned": 14,
|
||||
@@ -913,11 +1117,43 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T5",
|
||||
"verb": "for-compact",
|
||||
"failed": null,
|
||||
"ms": 713,
|
||||
"bytes": 9868,
|
||||
"est_tokens": 2467,
|
||||
"paths_mentioned": 39,
|
||||
"ground_truth": 4,
|
||||
"ground_truth_basis": "existing-files-only",
|
||||
"hits": 2,
|
||||
"recall": 0.5,
|
||||
"best_rank": 1,
|
||||
"per_file": [
|
||||
{
|
||||
"path": "src/commands/interaction/runtime/wait-polling.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/commands/interaction/runtime/wait-polling.ts",
|
||||
"rank": 1
|
||||
},
|
||||
{
|
||||
"path": "src/commands/interaction/runtime/wait-selector.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "src/commands/interaction/runtime/wait-selector.ts",
|
||||
"rank": 5
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T5",
|
||||
"verb": "for-idents",
|
||||
"failed": null,
|
||||
"ms": 959,
|
||||
"ms": 684,
|
||||
"bytes": 10785,
|
||||
"est_tokens": 2696,
|
||||
"paths_mentioned": 39,
|
||||
@@ -949,7 +1185,7 @@
|
||||
"task": "T6",
|
||||
"verb": "for",
|
||||
"failed": null,
|
||||
"ms": 5362,
|
||||
"ms": 716,
|
||||
"bytes": 10390,
|
||||
"est_tokens": 2598,
|
||||
"paths_mentioned": 33,
|
||||
@@ -977,7 +1213,7 @@
|
||||
"task": "T6",
|
||||
"verb": "pack-task",
|
||||
"failed": null,
|
||||
"ms": 4830,
|
||||
"ms": 1369,
|
||||
"bytes": 11137,
|
||||
"est_tokens": 2784,
|
||||
"paths_mentioned": 7,
|
||||
@@ -1005,7 +1241,7 @@
|
||||
"task": "T6",
|
||||
"verb": "pack-task-4k",
|
||||
"failed": null,
|
||||
"ms": 996,
|
||||
"ms": 1087,
|
||||
"bytes": 8256,
|
||||
"est_tokens": 2064,
|
||||
"paths_mentioned": 3,
|
||||
@@ -1029,11 +1265,39 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T6",
|
||||
"verb": "for-compact",
|
||||
"failed": null,
|
||||
"ms": 999,
|
||||
"bytes": 10487,
|
||||
"est_tokens": 2622,
|
||||
"paths_mentioned": 38,
|
||||
"ground_truth": 3,
|
||||
"ground_truth_basis": "existing-files-only",
|
||||
"hits": 0,
|
||||
"recall": 0,
|
||||
"best_rank": null,
|
||||
"per_file": [
|
||||
{
|
||||
"path": "packages/platform-apple/src/snapshot-route.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/platform-apple/src/snapshot-target.test.ts",
|
||||
"rank": null
|
||||
},
|
||||
{
|
||||
"path": "packages/platform-apple/src/snapshot-target.ts",
|
||||
"rank": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"task": "T6",
|
||||
"verb": "for-idents",
|
||||
"failed": null,
|
||||
"ms": 1026,
|
||||
"ms": 693,
|
||||
"bytes": 10545,
|
||||
"est_tokens": 2636,
|
||||
"paths_mentioned": 43,
|
||||
|
||||
Reference in New Issue
Block a user