skills/boshu2__agentops
1
0
Fork 0
mirror of https://github.com/boshu2/agentops.git synced 2026-09-14 15:08:13 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs(goals): F2.4 docs regen for scenario-satisfaction gate + epic CLI artifacts

Browse Source 
Regenerate cli/docs/COMMANDS.md and registry.json from a clean worktree at
HEAD so the soc-58nt command surface is documented without leaking unrelated
peer WIP into the generated docs.

F2 (soc-58nt.2.7): document scenario_satisfaction JSON shape, --scenarios-only,
result-artifact resolution, and exit codes in skills/goals/SKILL.md +
references/executable-spec-chain.md; cross-ref ao goals measure/trace from
skills/scenario/SKILL.md.

Epic-wide artifacts also landed here because they regenerate atomically across
F2-F5: COMMANDS.md/registry.json (all new ao goals/rpi surface), cli-skills-map
heading count, skills-codex hashes, ADR-0004/0005/0006 documentation-index
links, and the F2-F5 e2e CI lanes in validate.yml.

ao capabilities and ao robot-docs need no manual edits — both build their
command/flag surface from the live cobra tree, so new commands register
automatically (Global Rule G5 satisfied by construction).

bead: soc-58nt.2.7
This commit is contained in:
Boden Fuller
2026-05-17 09:34:37 -04:00
parent 5c66e9fd63
commit 37798abe57
10 changed files with 316 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/validate.yml
+20
View File
@@ -378,6 +378,26 @@ jobs:
chmod +x tests/e2e/goals-scenarios-link.sh
bash tests/e2e/goals-scenarios-link.sh
- name: F2 scenario-satisfaction gate e2e (epic soc-58nt)
run: |
chmod +x tests/e2e/goals-measure-scenarios.sh
bash tests/e2e/goals-measure-scenarios.sh
- name: F3 domain-scoped RPI e2e (epic soc-58nt)
run: |
chmod +x tests/e2e/rpi-phased-domain.sh
bash tests/e2e/rpi-phased-domain.sh
- name: F4 trace-chain e2e (epic soc-58nt)
run: |
chmod +x tests/e2e/goals-trace-chain.sh
bash tests/e2e/goals-trace-chain.sh
- name: F5 auto re-steer e2e (epic soc-58nt)
run: |
chmod +x tests/e2e/goals-steer-auto.sh
bash tests/e2e/goals-steer-auto.sh
validate-wiring-closure:
needs: [changes]
runs-on: ubuntu-latest
cli/docs/COMMANDS.md
+68
View File
@@ -1959,6 +1959,7 @@ ao goals measure [flags]
--exclude-tag string Skip goals whose Tags include this value (e.g. long-cycle)
--goal string Measure a single goal by ID
-h, --help help for measure
--scenarios-only Evaluate only executable-spec scenario satisfaction; skip shell gate-command execution
--total-timeout int Overall measurement timeout in seconds (0 disables)
```
@@ -2026,6 +2027,21 @@ ao goals history [flags]
--since string Show entries since date (YYYY-MM-DD)
```
#### `ao goals render`
Render the executable-spec layer as BDD/Gherkin text.
```
ao goals render [flags]
```
**Flags:**
```
-h, --help help for render
--out string Write Gherkin to this file instead of stdout
```
#### `ao goals scenarios`
List or create the executable-spec scenarios linked to GOALS.md directives.
@@ -2048,6 +2064,23 @@ ao goals scenarios [flags]
--threshold float Satisfaction threshold for a created scenario (default 0.8)
```
#### `ao goals trace`
Walk the executable-spec trace chain defined in docs/adr/ADR-0005.
```
ao goals trace [flags]
```
**Flags:**
```
--from string Render the trace lineage rooted at this directive, scenario, or bead ID
-h, --help help for trace
--orphans Audit the whole chain for broken references (errors) and missing yields (warnings)
--strict Escalate warning-class defects to a non-zero exit (ADR-0005 §4.2)
```
#### `ao goals add`
Add a new goal
@@ -2154,6 +2187,23 @@ ao goals steer add <title> [flags]
--steer string Steer direction (increase, decrease, hold, explore) (default "increase")
```
##### `ao goals steer apply`
Apply the top re-steer recommendation to GOALS.md via the non-lossy directive-block patcher. Requires policy auto_apply:true AND explicit human confirmation (interactive prompt, or --auto --yes for scripts). A run without confirmation never changes GOALS.md.
```
ao goals steer apply [flags]
```
**Flags:**
```
--auto Equivalent to --yes: explicit non-interactive consent to apply
-h, --help help for apply
--policy string Re-steer policy path (default: docs/re-steer-policy.json)
--yes Pre-confirm the apply for non-interactive/scripted use (explicit consent)
```
##### `ao goals steer prioritize`
Move a directive to a new position
@@ -2162,6 +2212,21 @@ Move a directive to a new position
ao goals steer prioritize <number> <new-position> [flags]
```
##### `ao goals steer recommend`
Run the re-steer policy engine over the verdict ledger and print recommended directive mutations and skip reasons. GOALS.md is never modified. Use `ao goals steer apply` to apply a recommendation.
```
ao goals steer recommend [flags]
```
**Flags:**
```
-h, --help help for recommend
--policy string Re-steer policy path (default: docs/re-steer-policy.json)
```
##### `ao goals steer remove`
Remove a directive by number
@@ -2753,7 +2818,9 @@ ao rpi phased <goal> [flags]
--daemon-token string agentopsd mutation token for --daemon-submit
--daemon-url string agentopsd base URL for --daemon-submit (default: activation file)
--discovery-artifact string Path to a pre-validated discovery artifact (markdown) used to skip Phase 1 when combined with --from=implementation
--domain string Scope the run to a domain slice (loads docs/domains/<name>/manifest.yaml; phase prompts carry its boundaries)
--fast-path Force fast path (--quick for gates)
--force With --scaffold-domain: overwrite an existing manifest
--from string Start from phase (discovery, implementation, validation; aliases: research, plan, pre-mortem, crank, vibe, post-mortem) (default "discovery")
-h, --help help for phased
--interactive Enable human gates at research and plan phases
@@ -2767,6 +2834,7 @@ ao rpi phased <goal> [flags]
--phase-timeout duration Maximum wall-clock runtime per phase (0 disables timeout) (default 1h30m0s)
--runtime string Phase runtime mode: auto|direct|stream|tmux|gc (default "auto")
--runtime-cmd string Runtime command used for phase prompts (Claude uses '-p'; Codex uses 'exec') (default "claude")
--scaffold-domain string Write a domain-slice manifest template at docs/domains/<name>/manifest.yaml and exit (does NOT run RPI)
--stall-timeout duration Maximum time without progress before declaring stall (0 disables) (default 10m0s)
--stream-startup-timeout duration Maximum time to wait for first stream event before falling back to direct execution (0 disables) (default 45s)
--swarm-first Default each phase to swarm/agent-team execution; fall back to direct execution if swarm runtime is unavailable (default true)
docs/documentation-index.md
+3
View File
@@ -53,6 +53,9 @@ Bridge / framing docs:
- [ADR-0001: Adopt DDD + Hexagonal Architecture](adr/ADR-0001-ddd-hexagonal-adoption.md) — Decision record for encoding DDD + Hexagonal with `ExecutionPacket` as the tracer-bullet aggregate
- [ADR-0002: AgentOps 3.0 Hookless-First CDLC Rearchitecture](adr/ADR-0002-agentops-3-hookless-cdlc-rearchitecture.md) — Proposed 3.0 direction: demote hooks to optional runtime adapters and center CDLC bounded contexts
- [ADR-0003: Executable-Spec Artifact Durability](adr/ADR-0003-executable-spec-artifact-durability.md) — Where executable-spec scenarios and domain manifests live: promoted spec scenarios in tracked `spec/scenarios/`, ad hoc holdout scenarios stay in `.agents/holdout/`
- [ADR-0004: Domain-Slice Manifest Contract](adr/ADR-0004-domain-slice-manifest-contract.md) — Schema and resolution rules for `docs/domains/<name>/manifest.yaml`, the domain-scoped boundary contract consumed by `ao rpi phased --domain`
- [ADR-0005: Trace-Link Convention](adr/ADR-0005-trace-link-convention.md) — The directive→scenario→bead→verdict→learning link grammar that `ao goals trace` renders and audits, including warning/error defect classes and `--strict` escalation
- [ADR-0006: Re-Steer Policy and Mutation Safety](adr/ADR-0006-re-steer-policy-and-mutation-safety.md) — The `docs/re-steer-policy.json` engine, human-gate, and non-lossy GOALS.md patcher that govern `ao goals steer recommend`/`apply`
- [PDC Framework](architecture/pdc-framework.md) — Prevent, Detect, Correct quality control approach
- [FAAFO Alignment](architecture/faafo-alignment.md) — FAAFO promise framework for vibe coding value
- [Failure Patterns](architecture/failure-patterns.md) — The 12 failure patterns reference guide
registry.json
+21 -5
View File
@@ -1,13 +1,13 @@
{
"schema_version": 1,
"generated_at": "2026-05-17T11:16:29Z",
"generated_at": "2026-05-17T13:33:34Z",
"summary": {
"skills": 77,
"hooks": 43,
"knowledge_stores": 4,
"job_types": 14,
"eval_files": 62,
"cli_commands": 166
"cli_commands": 170
},
"surfaces": {
"skills": [
@@ -25,7 +25,7 @@
"path": "skills/forge/",
"has_skill_md": true,
"has_references": true,
"reference_count": 2
"reference_count": 3
},
{
"name": "inject",
@@ -329,7 +329,7 @@
"path": "skills/post-mortem/",
"has_skill_md": true,
"has_references": true,
"reference_count": 20
"reference_count": 21
},
{
"name": "pre-mortem",
@@ -537,7 +537,7 @@
"path": "skills/goals/",
"has_skill_md": true,
"has_references": true,
"reference_count": 2
"reference_count": 3
},
{
"name": "product",
@@ -1436,6 +1436,10 @@
"name": "goals_prune",
"path": "cli/cmd/ao/goals_prune.go"
},
{
"name": "goals_render",
"path": "cli/cmd/ao/goals_render.go"
},
{
"name": "goals_scenarios",
"path": "cli/cmd/ao/goals_scenarios.go"
@@ -1444,6 +1448,18 @@
"name": "goals_steer",
"path": "cli/cmd/ao/goals_steer.go"
},
{
"name": "goals_steer_auto",
"path": "cli/cmd/ao/goals_steer_auto.go"
},
{
"name": "goals_trace",
"path": "cli/cmd/ao/goals_trace.go"
},
{
"name": "goals_trace_orphans",
"path": "cli/cmd/ao/goals_trace_orphans.go"
},
{
"name": "goals_validate",
"path": "cli/cmd/ao/goals_validate.go"
skills-codex/.agentops-manifest.json
+6 -6
View File
@@ -757,7 +757,7 @@
{
"name": "domain",
"source_skill": "skills/domain",
"source_hash": "ea4a560f7677854bc8e667ef2742dc75c3d056e00fe61382ef7737d57e45e84b",
"source_hash": "c30d182d4035fd7fabda09f312e3ad31494f6eff0ab826a15af540232ef5336a",
"generated_hash": "5de534737803c76fab3ad06e1a47dba85eb8f631a159cc6157ce4f932c2e99e4"
},
{
@@ -787,7 +787,7 @@
{
"name": "goals",
"source_skill": "skills/goals",
"source_hash": "45070be3934ed6393398517f0048a1e86849470522598352b3872c028581425f",
"source_hash": "8a386eff82a8f0eca27d1940277d00187e83582c2146652f67487521b848a4d6",
"generated_hash": "dc9b388da75af17c85440fa0481c9c69b16b9641b1832269e302856dd48ded8d"
},
{
@@ -871,8 +871,8 @@
{
"name": "post-mortem",
"source_skill": "skills/post-mortem",
"source_hash": "72fb9bbcf256c96ff75e880fe41e8e0f058b645af8440718603166b720d4efd6",
"generated_hash": "d627fe306330cace90675dee4251363dc899988d882faf40c2a51ef4a95775ac"
"source_hash": "d7f2fb1ed9e3ed843b1d2ee65693d8c53dee41e789269a1a15071922417b8134",
"generated_hash": "1a7ca0bfa802a2414e10e50fa0e69408d29b6397ac287c12e16d352a6c0beba3"
},
{
"name": "pr-implement",
@@ -1009,13 +1009,13 @@
{
"name": "scaffold",
"source_skill": "skills/scaffold",
"source_hash": "210cb4fa18c5418f599246c50a42d82f4f7081f299dd6c1285c73d9badbe6eb2",
"source_hash": "047a6d3af2988cf5deee6ac1b0e32bfb360e9bd2e4776c0c1f8923ee5f16397c",
"generated_hash": "800870469aab63a0445703a1019cbc0d382c1bb0865d1ceb03ae3a9c1a86ac75"
},
{
"name": "scenario",
"source_skill": "skills/scenario",
"source_hash": "ed3aeec01e23a14176746c9ec9dc4b3958d601e24c0bea85833f23ffddb17db1",
"source_hash": "7c14ca7bf2632a1130611d391899ea45862d7aa9bbe6c582fd0f3087a9ee8193",
"generated_hash": "1736927301013e0f0aa5492ac60c2b909caee1a7ec232752998d85f1c9767de5"
},
{
skills-codex/goals/.agentops-generated.json
+1 -1
View File
@@ -2,6 +2,6 @@
"generator": "manual-maintained",
"source_skill": "skills/goals",
"layout": "modular",
"source_hash": "45070be3934ed6393398517f0048a1e86849470522598352b3872c028581425f",
"source_hash": "8a386eff82a8f0eca27d1940277d00187e83582c2146652f67487521b848a4d6",
"generated_hash": "dc9b388da75af17c85440fa0481c9c69b16b9641b1832269e302856dd48ded8d"
}
skills-codex/scenario/.agentops-generated.json
+1 -1
View File
@@ -2,6 +2,6 @@
"generator": "manual",
"source_skill": "skills/scenario",
"layout": "modular",
"source_hash": "ed3aeec01e23a14176746c9ec9dc4b3958d601e24c0bea85833f23ffddb17db1",
"source_hash": "7c14ca7bf2632a1130611d391899ea45862d7aa9bbe6c582fd0f3087a9ee8193",
"generated_hash": "1736927301013e0f0aa5492ac60c2b909caee1a7ec232752998d85f1c9767de5"
}
skills/goals/SKILL.md
+88
View File
@@ -46,6 +46,8 @@ output_contract: GOALS.md
/goals validate # Validate structure
/goals prune # Remove stale gates
/goals migrate # Migrate YAML to Markdown
/goals trace # Render/audit the executable-spec chain
/goals render # Export directive scenarios as Gherkin
```
## Format Support
@@ -75,6 +77,8 @@ Parse the user's input:
| `/goals prune`, "prune goals", "clean goals" | **prune** | `ao goals prune` |
| `/goals migrate`, "migrate goals" | **migrate** | `ao goals migrate` |
| `/goals scenarios`, "directive scenarios", "link a scenario" | **scenarios** | `ao goals scenarios` |
| `/goals trace`, "trace lineage", "orphan audit" | **trace** | `ao goals trace` |
| `/goals render`, "export gherkin", "feature file" | **render** | `ao goals render` |
`ao goals scenarios` links each directive to behavioral scenarios (the
`ao scenario` family) so GOALS.md is an executable BDD spec: bare lists every
@@ -82,6 +86,10 @@ directive's linked scenarios with link health; `--create "<goal>" --directive N`
scaffolds and bidirectionally links a scenario; `--lint` checks the link graph.
See `docs/adr/ADR-0003`.
`ao goals trace` and `ao goals render` render and audit the executable-spec
chain; `ao goals steer recommend`/`apply` drive the auto re-steer loop. Their
contracts are documented in `references/executable-spec-chain.md`.
## Measure Mode (default) — Observe
### Step 1: Run Measurement
@@ -105,6 +113,35 @@ For each directive, assess whether recent work has addressed it:
- Check beads/issues related to the directive topic
- Rate each directive: addressed / partially-addressed / gap
### Step 2b: Scenario Satisfaction (executable-spec fitness)
`ao goals measure` also gates on **scenario satisfaction** — the fraction of a
directive's linked behavioral scenarios that currently pass. Each directive in
the `--json` / `--directives` output carries a `scenario_satisfaction` block:
```jsonc
"scenario_satisfaction": {
"linked": 4, // scenarios linked to this directive
"satisfied": 3, // scenarios whose latest result is PASS
"ratio": 0.75, // satisfied / linked
"threshold": 0.8, // directive's required ratio
"status": "RED" // GREEN | YELLOW | RED — RED when ratio < threshold
}
```
A directive below its threshold is `RED` and drags overall fitness down.
Use `--scenarios-only` to evaluate just the executable-spec layer and skip the
shell gate-command execution — fast feedback while iterating on scenarios:
```bash
ao goals measure --scenarios-only -o json
```
Scenario results are read from the scenario result artifacts (see
`ao scenario` family); the exact aggregation path and exit-code semantics are
in `references/executable-spec-chain.md`.
### Step 3: Report
Present fitness dashboard:
@@ -213,6 +250,26 @@ ao goals steer remove 3
ao goals steer prioritize 2 1
```
### Step 4: Auto Re-Steer (F5) — chronic failure mutates the directive
When a directive's scenarios fail chronically, the re-steer engine recommends a
directive mutation from the verdict ledger:
```bash
ao goals steer recommend # show recommendations; GOALS.md untouched
ao goals steer apply # apply the top recommendation (human-gated)
ao goals steer apply --auto --yes # non-interactive consent for scripts
ao goals steer apply --policy docs/re-steer-policy.json
```
`recommend` is read-only — it runs the policy engine over the verdict ledger and
prints recommended mutations plus skip reasons. `apply` mutates GOALS.md only
when the policy's `auto_apply` is true **and** the operator confirms (interactive
prompt, or `--auto`/`--yes` for scripts). Every mutation routes through the
non-lossy directive-block patcher — never a full re-render. Policy, ledger, and
human-gate semantics: `references/executable-spec-chain.md` and
`docs/adr/ADR-0006`.
## Add Mode
Add a single goal to the goals file. Format-aware — writes to GOALS.yaml or GOALS.md depending on which format is detected.
@@ -303,6 +360,36 @@ ao goals migrate # Migrate GOALS.yaml to latest YAML version
The `--to-md` flag creates a GOALS.md with mission, north/anti stars sections, and converts existing goals into the Gates table format. The original YAML file is backed up.
## Trace Mode (F4) — render and audit the executable-spec chain
`ao goals trace` renders the directive → scenario → bead → verdict → learning
lineage and audits it for defects:
```bash
ao goals trace --from d-fitness-gate-bdd # lineage tree from a directive
ao goals trace --from s-2026-05-17-001 -o json # line-delimited JSON graph
ao goals trace --orphans # whole-chain gap audit
ao goals trace --orphans --strict # warnings also fail (exit 1)
```
- `--from <id>` roots the trace at any directive, scenario, or bead ID.
- `--orphans` audits the whole chain: broken references are **errors** (always
non-zero exit), missing yields are **warnings**.
- `--strict` escalates warning-class defects to a non-zero exit (ADR-0005 §4.2).
Stable directive IDs (`d-...`) are the link anchors — never display numbers.
The link grammar is `docs/adr/ADR-0005`.
## Render Mode (F4) — export Gherkin feature files
`ao goals render` exports the directive-linked scenarios as a Gherkin feature
file so the executable spec can be consumed by external BDD tooling:
```bash
ao goals render # print Gherkin to stdout
ao goals render --out spec.feature # write Gherkin to a file
```
## Examples
### Checking fitness and directive gaps
@@ -358,3 +445,4 @@ The `--to-md` flag creates a GOALS.md with mission, north/anti stars sections, a
- [references/generation-heuristics.md](references/generation-heuristics.md)
- [references/goals-schema.md](references/goals-schema.md)
- [references/executable-spec-chain.md](references/executable-spec-chain.md)
skills/goals/references/executable-spec-chain.md
+102
View File
@@ -0,0 +1,102 @@
# Executable-Spec Chain — Reference
Detailed contracts for the executable-spec layer of `ao goals`: scenario
satisfaction (F2), trace/render (F4), and auto re-steer (F5). The `goals`
SKILL.md links here; this file holds the precise schemas and exit-code rules.
## Scenario satisfaction (F2)
`ao goals measure` aggregates the latest result of every behavioral scenario
linked to a directive and computes a satisfaction ratio. The producer reads
scenario result artifacts written by the `ao scenario` family.
### `scenario_satisfaction` JSON shape
Every directive object in `ao goals measure --json` and
`ao goals measure --directives` carries:
```jsonc
"scenario_satisfaction": {
"linked": 4, // count of scenarios linked to the directive
"satisfied": 3, // count whose latest result artifact is PASS
"ratio": 0.75, // satisfied / linked (0.0 when linked == 0)
"threshold": 0.8, // directive's required ratio (default in policy)
"status": "RED" // GREEN (ratio >= threshold)
// YELLOW (linked == 0 — nothing to satisfy yet)
// RED (ratio < threshold)
}
```
### `--scenarios-only`
`ao goals measure --scenarios-only` evaluates ONLY the executable-spec layer and
skips shell gate-command execution. Use it for fast iteration on scenarios
without paying for the full gate suite. Combine with `-o json` for CI.
### Result-artifact resolution order
Scenario results are resolved from result artifacts (ADR-0003 durability
contract):
1. Promoted spec scenarios — tracked `spec/scenarios/`.
2. Ad hoc holdout scenarios — `.agents/holdout/<id>.json`.
### Exit-code semantics
| Exit | Meaning |
|------|---------|
| 0 | All gates and all directive scenario thresholds satisfied. |
| 1 | One or more gates failed, or a directive is `RED` (ratio below threshold). |
| 2 | Partial result — a scenario artifact was missing or unreadable. |
## Trace and render (F4)
### `ao goals trace`
Renders and audits the directive → scenario → bead → verdict → learning chain.
- `--from <id>` — render the lineage tree rooted at a directive (`d-...`),
scenario (`s-...`), or bead ID. Add `-o json` for a line-delimited JSON graph.
- `--orphans` — audit the whole chain. Broken references are **errors**;
missing downstream yields (e.g. a scenario with no verdict) are **warnings**.
- `--strict` — escalate warning-class defects to a non-zero exit (ADR-0005
§4.2). Errors always exit non-zero regardless of `--strict`.
Link anchors are stable directive IDs (`^d-[a-z0-9][a-z0-9-]*$`) — never the
display numbers, which are not stable across edits. The full link grammar and
defect taxonomy are in `docs/adr/ADR-0005`.
### `ao goals render`
Exports directive-linked scenarios as a Gherkin `.feature` file:
- bare — print Gherkin to stdout.
- `--out <path>` — write the Gherkin to a file instead.
## Auto re-steer (F5)
When a directive's scenarios fail chronically, the re-steer engine recommends a
directive mutation. This is the last and most safety-gated part of the chain.
### `ao goals steer recommend`
Read-only. Runs the re-steer policy engine over the verdict ledger and prints
recommended directive mutations plus skip reasons. GOALS.md is never modified.
### `ao goals steer apply`
Applies the top recommendation to GOALS.md. Two conditions must BOTH hold:
1. The policy's `auto_apply` is `true`.
2. The operator confirms — interactive prompt, or `--auto` / `--yes` for
non-interactive scripted consent.
A run without confirmation never changes GOALS.md. Every mutation routes through
the non-lossy directive-block patcher (`cli/internal/goals/patcher.go`) — never
`RenderGoalsMD` / `WriteMDGoals`, which are lossy full re-renders.
- `--policy <path>` — re-steer policy file (default `docs/re-steer-policy.json`).
- `--auto` / `--yes` — pre-confirm for non-interactive use.
Policy schema, verdict-ledger format, mutation-safety invariants, and the
human-gate contract are in `docs/adr/ADR-0006`.
skills/scenario/SKILL.md
+6
View File
@@ -113,6 +113,12 @@ lists each directive's linked scenarios; `ao goals scenarios --lint` checks
the link graph. Ad hoc holdout scenarios authored with `ao scenario add`
stay unlinked until promoted. See the `/goals` skill and `docs/adr/ADR-0003`.
Once linked, a scenario's pass/fail feeds the directive's fitness:
`ao goals measure` rolls linked scenario results into a per-directive
`scenario_satisfaction` ratio (RED below threshold), and `ao goals trace`
renders the directive → scenario → bead → verdict → learning lineage and
audits it for orphans. See the `/goals` skill for both surfaces.
### Step 5: Integration with Validation
Scenarios are consumed by **STEP 1.8** in the `/validation` skill. During