mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
Bo
5908024782
## Summary The two structural beads deferred from the skill-overhaul reboot waves. **goals → fitness** (`age-skill-overhaul-reboot-sjv7v.10`, 07-24 plan D9) - The semantic skill is now `fitness`; the `ao goals` CLI command family deliberately keeps its name. - `skills/goals` remains a thin, non-advertised compatibility alias: user-invocable, zero capabilities/effects/dependencies, an `alias-of` context_rel edge to fitness, physical deletion only under the observed-zero policy. - `alias-of` added to the frontmatter schema enums (the catalog schema's own description already listed it as valid relationship vocabulary; the enum lagged). - bootstrap's `goals` consumer edge migrated; the stale `goals.feature` row in the scenario-linkage contract corrected (the feature file was already absent). **shared retirement** (`age-skill-overhaul-reboot-sjv7v.11`, 07-24 plan D9) - The runtime-neutrality contract moved to its declared owner: `docs/contracts/runtime-neutrality.md`, scope unchanged (shared references a consuming skill loads). - The last consumer edge (`bootstrap consumes: shared`) removed; `skills/shared` reduced to a non-routable tombstone pending the observed-zero deletion window; the ports-and-adapters contract updated to executed state. ## Validation - Strict frontmatter 50/50 (fitness + alias); regen byte-current; scenario-linkage PASS; liveness/anti-spiral/desc-budget bats green; `cli/internal/skills` Go tests pass - Cross-family review, two rounds: round 1 — one finding fixed (unintended contract-scope broadening reverted), one rejected with evidence (the reviewer inferred a pre-existing feature file from a stale doc row; the pre-change tree had none); round 2 delta **VERDICT: PASS** Tracker: `age-skill-overhaul-reboot-sjv7v.10`, `.11`
105 lines
4.6 KiB
Markdown
105 lines
4.6 KiB
Markdown
# Scenario → Test Linkage Contract
|
|
|
|
Status: active · Bead: soc-63xfx · Gate: `validate-scenario-test-linkage` (T1, blocking)
|
|
|
|
## Why
|
|
|
|
71 `.feature` files live under `skills/*/references/`. Historically only a
|
|
handful had an executing test; the rest were documentation that drifted free of
|
|
any check. A scenario nobody runs is a claim nobody verifies.
|
|
|
|
This contract makes the scenario→test relationship **mechanical**: every Gherkin
|
|
`Scenario:` in the corpus must either name the test that covers it, or its
|
|
feature file must be explicitly declared documentation-only. The gate fails on
|
|
any scenario that is neither.
|
|
|
|
It is the sibling of `executable-spec-link-integrity` (soc-58nt / soc-m8tdn),
|
|
which links **GOALS directives → scenarios**. This one links **scenarios →
|
|
tests**. Together they close the spec-to-evidence chain:
|
|
|
|
```
|
|
GOALS directive ──► scenario (.feature) ──► test (executing)
|
|
(exec-spec-link-integrity) (scenario-test-linkage)
|
|
```
|
|
|
|
## The convention
|
|
|
|
Tag a scenario with the test that covers it using a Gherkin tag:
|
|
|
|
```gherkin
|
|
@covered-by:tests/e2e/rpi-phased-domain.sh
|
|
Scenario: Phases run in order and never compress
|
|
When /rpi executes
|
|
Then it runs Research, then Plan, then Implement in order
|
|
```
|
|
|
|
Rules:
|
|
|
|
- The tag goes on its **own line directly above** the `Scenario:` it covers —
|
|
the standard Gherkin tag position. A non-tag line (a step, `Background`, a
|
|
blank then content) between the tag and the scenario breaks the association.
|
|
- **File-level tags** (placed above the `Feature:` line) apply to **every**
|
|
scenario in the file.
|
|
- Multiple tags may stack; a scenario passes only if **all** of its
|
|
`@covered-by:` targets resolve.
|
|
- Two target forms:
|
|
- `@covered-by:<test-path>` — the test file must exist (repo-relative path).
|
|
- `@covered-by:<test-path>::<Name>` — additionally, `<Name>` (a Go func, a
|
|
bats `@test` label, a bash function, etc.) must appear in that file. The
|
|
name match is a substring check, which keeps the gate language-agnostic
|
|
while still catching a renamed or deleted test.
|
|
|
|
`<test-path>` is any executing test surface: `tests/e2e/*.sh`,
|
|
`tests/scripts/*.bats`, `cli/**/*_test.go`, and so on. The gate only asserts the
|
|
target **exists** — it does not run it. Running is the test suite's job; linking
|
|
is this gate's job.
|
|
|
|
## The allowlist
|
|
|
|
`scripts/.scenario-linkage-allow` lists feature files (repo-relative, one per
|
|
line, `#` comments allowed) that are intentionally documentation-only for now.
|
|
The gate skips scenarios in those files.
|
|
|
|
The allowlist is a **draining backlog**, not a parking lot:
|
|
|
|
- When a skill gains an executing test, tag its scenarios with `@covered-by:`
|
|
and **remove the file from the allowlist**.
|
|
- A file may **not** be both allowlisted **and** carry `@covered-by:` tags —
|
|
that ambiguous intent is a gate error. Pick one.
|
|
- A stale allowlist entry (a listed file that no longer exists) is a gate error.
|
|
- Do not add new feature files to the allowlist without a tracking bead; new
|
|
specs should ship with their covering test.
|
|
|
|
## Gate behavior
|
|
|
|
`scripts/check-scenario-test-linkage.sh`:
|
|
|
|
| Condition | Result |
|
|
|---|---|
|
|
| Scenario has a resolving `@covered-by:` tag | pass |
|
|
| Scenario's feature file is allowlisted | pass (counted as doc-only) |
|
|
| Scenario has no tag and file is not allowlisted | **FAIL** |
|
|
| `@covered-by:` path does not exist | **FAIL** (dangling) |
|
|
| `@covered-by:...::Name` not found in the file | **FAIL** (dangling) |
|
|
| Allowlisted file also carries a `@covered-by:` tag | **FAIL** (ambiguous) |
|
|
| Allowlist entry points at a deleted file | **FAIL** (stale) |
|
|
|
|
Flags: `--warn-only` (advisory, exit 0), `--json` (machine-readable summary),
|
|
`-h`/`--help`. Exit codes: `0` pass, `1` fail, `2` misuse.
|
|
|
|
Wired into `.github/workflows/validate.yml` as `validate-scenario-test-linkage`,
|
|
gated on changes to `skills/**`, `**/*.sh`, or `.github/**`.
|
|
|
|
## Starter set (at landing, soc-63xfx)
|
|
|
|
3 of 71 feature files are fully linked to executing tests:
|
|
|
|
| Feature | Covering test |
|
|
|---|---|
|
|
| `skills/rpi/references/rpi.feature` | `tests/e2e/rpi-phased-domain.sh` |
|
|
| ~~`skills/goals/references/goals.feature`~~ (stale row: the feature file was already absent from the tree before 2026-07-29; the skill is now `fitness`) | `tests/e2e/goals-measure-scenarios.sh`, `goals-trace-chain.sh` still execute against the `ao goals` CLI |
|
|
| ~~`skills/scenario/references/scenario.feature`~~ (removed: `scenario` folded into `eval-outcomes`, 2026-06-12) | `tests/e2e/goals-scenarios-link.sh` |
|
|
|
|
The remaining 68 files are allowlisted in `scripts/.scenario-linkage-allow` with
|
|
the standing goal to drain the list as skills gain executing tests.
|