skills/boshu2__agentops
1
0
Fork 0
mirror of https://github.com/boshu2/agentops.git synced 2026-09-14 15:08:13 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor(cli): delete the unconsumed eval/redact surfaces — the estate audit's mechanical cut (#1082)

Browse Source 
> **Review findings closed.** The re-check's residue (app-seam family
count) is applied in `9a2790ae7` along with the full-tier CI
settlements: regenerated documentation index (generated file, hand-edit
drifted it), regenerated CLI-surface count fixtures (top=18 sub=44
all=62), `Test-Removal-Reason` trailer for the deliberate test
deletions, and the release-tag bats output list updated to the real
changes-job set. 67/67 full-tier gates green locally. Merging on Bo's
instruction.

## What

Deletes the provably-dead 28% of the `ao` CLI and every reference to it,
per the 2026-08-23 estate audit. −19.5K lines in the lane commit plus
integration fixups.

**Removed (each with zero live consumers, verified by consumer-grep +
`go list -deps`):**
- `ao eval` — 13 subcommands, ~10.9K LOC. Its would-be consumers were
already tombstones (`scripts/eval-agentops.sh` printed `RETIRED`),
`release.yml` hardcoded `--eval pass`, release evidence recorded
`suite_count: 0`, and three of its module tests exercised subcommands
that could never register (nil composition seats).
- `ao redact` — its only declared caller
(`skills/compile/scripts/compile.sh`) never existed.
- `cli/internal/types/memrl_policy.go` + the orphan cascade it and eval
left behind (`internal/scenario`, `internal/wiki`,
`internal/runtimecmd`, `internal/redact`) — all with zero importers,
verified before and after.
- `scripts/check-memrl-health.sh` +
`examples/schedules/feedback-drain-hourly.yaml` — a health check for the
feedback loop amputated on 2026-07-14; it exits 1 on main today and the
example instructs a verb (`ao feedback-loop`) that no longer exists.
- `corpus.secret-scan` gate — vacuous: its file filter excluded the
single tracked path its globs could match, so it scanned zero files;
secrets are covered by the pinned gitleaks steps in nightly and release
(validate's quick toolchain mode skips gitleaks).
- Docs for the deleted surface:
`docs/architecture/eval-architecture.md`,
`docs/code-map/eval-lid-primitives.md`; `contracts/eval-baseline-ab.md`
already carried a RETIRED banner and stays as history (delisted from the
live index).

**Kept, deliberately:**
- `ao robot-docs` — the audit's "duplicate of `doctor robot-docs`"
premise was false: they render different handbooks (whole-CLI vs
doctor-scoped). Verified before acting.
- `completion`, `demo`, `quick-start` — interactive human furniture, not
dead code.
- `corpus.witness-dolt-jsonl-crosscheck` gate — retargeted, not retired:
its backing script is a hermetic self-test over real tracked fixtures;
globs now point at the paths it actually exercises.
- `cli/internal/evalsubstrate` — Go-dead but it is the declared mirror
of `schemas/outcomes-rubric.v1.schema.json`; retiring it needs a paired
schemas/docs/scripts decision (package doc comment records this).
- `scripts/ci-local-release.sh` eval-evidence stanza — self-contained
honest bookkeeping (`status: not_applicable`), invokes nothing removed.

**Tombstones + migration:** `eval` and `redact` added to
`removed_command_hint.go` and `docs/MIGRATION.md`; the now-false "(`ao
eval` returned in 3.3 …)" parenthetical deleted; `go-cli.md` spine and
the "Eval — the Learn seat" section updated; the dated research snapshot
got a HISTORICAL banner via the docs-scope self-declaration mechanism
(history not rewritten).

## Why

v3.6.0 binary downloads: 4 darwin-arm64, 3 linux-amd64. Only 7 of 53
shipped skills invoke `ao` at all, and none of them touch this surface.
The eval family was the single largest command surface in the CLI with
zero live consumers — 28% of non-test Go maintained for nobody.

## Evidence

- `cd cli && go build ./... && go vet ./... && go test ./...` — exit 0
(previously-failing `TestGoCLIDocSpineMatchesApprovedSpine` and
`TestRemovedVerbsHaveMigrationRows` now pass)
- `scripts/check-docs-cli-snippets.sh` PASS ·
`check-cmdao-surface-parity.sh` PASS (54 leaf commands) ·
`check-corpus-path-guard.sh` PASS · `check-new-scripts-use-preamble.sh`
PASS · `ao gate check --dry-run` PASS
- Implemented by a worktree-isolated lane, independently validated by a
fresh context that re-ran the suite itself; the two failures it found
were doc files outside the lane's write scope, fixed in the integration
commit. Cross-family (Codex) review verdict included in the final
session report.

## Cross-family review (Codex, fresh context)

First pass: **FAIL** with two majors — (1) `quality.DeprecatedCommands`
still mapped five rewrite entries onto the removed eval family, so `ao
doctor --fix` would have introduced dead commands; (2) retained docs
(formal-verification research links, applied-ood README run block,
evalsubstrate hint strings) still prescribed removed commands. Both
repaired in `4da85a0d4` (one bounded round), plus its two minors
(types/AGENTS.md row, .gitignore unignore, family counts,
gitleaks-coverage comment). Re-verified: full suite green, snippets gate
PASS. Focused re-check: first-round findings confirmed closed; one new
residue (the family count above) stopped the loop under the spiral rule.

## Follow-ups (not in this PR)

- `cli/internal/quality/stale_refs.go` `DeprecatedCommands`: the five
eval-target entries are pruned here; the older pre-existing dead targets
(forge, inject, flywheel, ratchet, …) still need a map-wide
reconciliation against the live registry.
- `cli/internal/evalsubstrate` retirement decision (paired
schemas/docs/scripts change).
- `evals/scenarios/applied-ood/`, `evals/tier2-premortem/`,
`evals/_stats/` retain historical `ao eval` mentions in prereg/holdout
records — dated artifacts, left as history.
This commit is contained in:
Bo
2026-08-24 23:34:00 -04:00
committed by GitHub
parent 68ac1481f5
commit ffb9f122af
135 changed files with 126 additions and 20249 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cli/docs/COMMANDS.md
-402
View File
@@ -225,16 +225,6 @@ ao gate check [flags]
---
### `ao redact`
Read text on stdin, apply the canonical secret redactor (the same
```
ao redact [flags]
```
---
### `ao robot-docs`
Print a paste-ready, agent-targeted handbook for the whole ao CLI.
@@ -265,398 +255,6 @@ ao version [flags]
---
### `ao eval`
Run deterministic AgentOps evaluation suites and compare run records.
```
ao eval [command]
```
**Subcommands:**
#### `ao eval baseline`
Promote an eval run record as a baseline
```
ao eval baseline <run.json> [flags]
```
**Flags:**
```
-h, --help help for baseline
--out string write promoted baseline run record to path
--promoted-by string identity promoting the baseline
--rationale string rationale for promoting the baseline
```
#### `ao eval baseline-audit`
Audit eval suite baseline policy against promoted baselines
```
ao eval baseline-audit [suite.json ...] [flags]
```
**Flags:**
```
--baseline-dir string promoted baseline directory (default ".agents/evals/baselines")
-h, --help help for baseline-audit
--root string suite root to scan when no suite paths are provided (default "evals/agentops-core")
```
#### `ao eval cleanup`
Per SCHEMA.md §4 cleanup state-transition rule (rc2):
```
ao eval cleanup [flags]
```
**Flags:**
```
--delete Remove Run directories whose status is failed or aborted (never retracted)
--dry-run Preview without mutations
-h, --help help for cleanup
--tmp-age int Minimum tmp-file age in seconds before sweep (0 = sweep all) (default 60)
--tmp-files Sweep orphan *.tmp files older than --tmp-age
```
#### `ao eval compare`
Compare an eval run against a baseline
```
ao eval compare <candidate-run.json> <baseline-run.json> [flags]
```
**Flags:**
```
-h, --help help for compare
--max-aggregate-regression float allowed aggregate regression before verdict becomes regression
--max-dimension-regression float allowed per-dimension regression before verdict becomes regression
--out string write compared eval run record to path
```
#### `ao eval coverage`
Summarize eval suite coverage
```
ao eval coverage [suite.json ...] [flags]
```
**Flags:**
```
-h, --help help for coverage
--require-dimension stringArray required score dimension for missing-dimension reporting (default [correctness,process_adherence,artifact_quality,runtime_compatibility,efficiency,safety,learning_closure])
--require-domain stringArray required product domain for missing-domain reporting (default [cli,hook,skill,rpi,runtime,retrieval,scenario,mixed,security])
--require-evidence-kind stringArray required evidence kind for missing-evidence-kind reporting
--require-runtime stringArray required deterministic runtime for missing-runtime reporting (default [static,shell,mock])
--root string suite root to scan when no suite paths are provided (default "evals/agentops-core")
```
#### `ao eval outcomes`
Outcomes is a derived projection of the locked eval substrate (SCHEMA.md), never an alternate authority.
```
ao eval outcomes [command]
```
##### `ao eval outcomes compile`
Compile a holdout-safe Outcomes rubric payload from a locked Task + criteria
```
ao eval outcomes compile <input.json> [flags]
```
##### `ao eval outcomes ingest`
Ingest an Outcomes score payload into the one council verdict record
```
ao eval outcomes ingest <score.json> [flags]
```
**Flags:**
```
--burn-ledger string path to a JSON HoldoutBurnLedger; when set, a holdout-split score registers a burn and is REFUSED if the (suite,gt) quota is exhausted (gate #3 runtime enforcement), persisted across invocations
--expect-judge-hash string refuse the ingest if the score's judge_content_hash does not match this value (gate #2 rubric-drift parity)
-h, --help help for ingest
--manifest-out string also write an eval-run.v1 manifest to <dir>/<run-id>/manifest.json for a declared downstream evidence consumer
--run-id string run id for the --manifest-out manifest; defaults to the score's run_id, then source_task_id (sanitized to the eval-run.v1 pattern)
```
#### `ao eval run`
Run a deterministic eval suite.
```
ao eval run <suite.json> [flags]
```
**Flags:**
```
--baseline string compare the run against a baseline run record
--baseline-mode string skill-on | skill-off | both — runs the suite once with skills loaded, once with hooks suppressed, or both for a delta scorecard (default "skill-on")
--context-mode string none | ab — run context-off/context-on legs over isolated AO_AGENTS_DIR roots (default "none")
--context-off-agents-dir string AO_AGENTS_DIR root for the context-off leg (defaults to suite fixtures)
--context-on-agents-dir string AO_AGENTS_DIR root for the context-on leg (defaults to suite fixtures)
--delta-out string write delta scorecard JSON to path (with --baseline-mode=both or --context-mode=ab)
-h, --help help for run
--out string write eval run record to path
--run-id string stable run id to use in the run record
--runtime string runtime override (static, mock, shell, claude, codex)
```
#### `ao eval scenario`
Create, list, validate, and evaluate holdout scenarios stored in .agents/holdout/.
```
ao eval scenario [command]
```
##### `ao eval scenario add`
Author a holdout scenario from a goal description
```
ao eval scenario add <goal> [flags]
```
**Flags:**
```
--expected-outcome string Expected observable outcome (default: inferred from goal)
-h, --help help for add
--narrative string Narrative description (default: inferred from goal)
--source string Scenario source (human, agent, prod-telemetry) (default "human")
--status string Scenario status (active, draft, retired) (default "draft")
--threshold float Satisfaction threshold in [0,1] (default 0.8)
```
##### `ao eval scenario evaluate`
Evaluate directive-linked scenarios and record satisfaction results
```
ao eval scenario evaluate [flags]
```
**Flags:**
```
--all Evaluate every directive's linked scenarios
--directive string Evaluate only the directive with this stable Directive ID
-h, --help help for evaluate
--json Emit the machine-readable evaluation report
--run-id string run_id recorded in the results artifact (default "ao-scenario-evaluate")
--timeout duration Per-check execution timeout (default 2m0s)
```
##### `ao eval scenario init`
Initialize .agents/holdout/ directory for scenario storage
```
ao eval scenario init [flags]
```
##### `ao eval scenario list`
List holdout scenarios
```
ao eval scenario list [flags]
```
**Flags:**
```
-h, --help help for list
--status string Filter by status (active, draft, retired)
```
##### `ao eval scenario validate`
Validate holdout scenarios against schema
```
ao eval scenario validate [flags]
```
#### `ao eval scenario-ab`
Run a knowledge-reuse holdout scenario with vs. without the gold pull (the discriminating A/B)
```
ao eval scenario-ab [flags]
```
**Flags:**
```
--control-only Run only the without-gold control arm and fail on ceiling/no-headroom
-h, --help help for scenario-ab
--output string Write the ScenarioDeltaScorecard JSON to this path
--scenario string Path to the scenario.v1 JSON file (required)
--timeout duration Per-arm timeout (0 = default 5m)
--token-budget int Fail the gate if summed arm token cost exceeds this (0 = default 200000)
```
#### `ao eval scenario-moat`
Aggregate moat-eligible scenario A/B scorecards into a publication verdict
```
ao eval scenario-moat [flags]
```
**Flags:**
```
-h, --help help for scenario-moat
--output string Write the MoatClaimResult JSON to this path
--scorecard stringArray Path to a ScenarioDeltaScorecard JSON (repeatable)
```
#### `ao eval scorecard`
Build an eval scorecard from run records
```
ao eval scorecard <candidate-run.json> [baseline-run.json] [flags]
```
**Flags:**
```
-h, --help help for scorecard
--kind string scorecard kind (rpi, skill-change) (default "rpi")
--max-category-regression float allowed per-category regression before verdict becomes regression
--out string write scorecard JSON to path
```
#### `ao eval suite`
Suite-level operations against the §6.5 statistical contract.
```
ao eval suite [command]
```
##### `ao eval suite n-required`
Compute power-derived n_required (gate #6 input on Day 3+)
```
ao eval suite n-required [flags]
```
**Flags:**
```
--alpha float Type-I error rate (default 0.05)
--baseline-rate float Baseline rate (binomial worst-case fallback) (default 0.5)
-h, --help help for n-required
--mde float Minimum detectable effect (default 0.05)
--paired Paired comparison (default true)
--power float Statistical power (1-beta) (default 0.8)
```
##### `ao eval suite verdict`
Compute the §6.5 paired cluster-bootstrap verdict
```
ao eval suite verdict <suite-id> --arms a,b --inputs <bootstrap-inputs.json> [flags]
```
**Flags:**
```
--B int Bootstrap resamples (default 10000)
--arms string Comma-separated arm ids (default: from suite varied_axis)
-h, --help help for verdict
--inputs string Path to canonical bootstrap-inputs JSON (REQUIRED)
--mde float Minimum detectable effect (used for inconclusive_high_variance)
--n-required int Override n_required (default: derived from suite power block)
```
#### `ao eval task`
Operate on the §3 Task primitive of the eval substrate.
```
ao eval task [command]
```
##### `ao eval task add`
Register a Task by copying its yaml + samples into the substrate
```
ao eval task add <task.yaml> [flags]
```
##### `ao eval task list`
List registered Task ids
```
ao eval task list [flags]
```
##### `ao eval task run`
Open a new Run manifest for <task-id>; refuses on gate failure
```
ao eval task run <task-id> [flags]
```
**Flags:**
```
--allow-weak-labels Allow runs against confidence=weak ground-truth rows (gate #7)
--cross-spec Allow ModelSpec drift (gate #4)
--dry-run Run gates and exit without writing a Run manifest
--ground-truth string Ground-truth row id (head of supersession chain)
--harness string Harness id (recorded into manifest)
--harness-dir string Path to harness source dir for snapshot + gate #8
-h, --help help for run
--inspect-command string Inspect command recorded into the Run manifest (not executed yet)
--inspect-version string Inspect AI version stamped into manifest (default "0.3.216")
--model-spec string ModelSpec id, resolved from <evals-root>/models/<id>/spec.yaml
--n-samples int Override Suite.n_samples
--quick Mark Run as quick_session=true (excluded from --vs auto-baseline pool)
--rig-id string Rig identifier stamped into the Run manifest
--sample-split string Sample split (dev|holdout); default from suite
--seeds string Comma-separated seeds (>=3, per §4)
--suite string Suite id or path to suite.yaml (required)
```
##### `ao eval task show`
Print a registered Task summary
```
ao eval task show <task-id> [flags]
```
---
### `ao gc`
Prepare and qualify the stock Gas City maintainer pack without owning a pack.