2026-07-14 22:01:50 -04:00
# Repository CI and delivery
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
2026-07-15 19:17:08 -04:00
Repository authority and the core work boundary are defined in [AGENTS.md ](../AGENTS.md ).
AgentOps reads or refines the existing bead/caller intent, runs one bounded
implementation experiment, derives exact content identity, obtains one
2026-07-29 19:50:56 -04:00
author-distinct Validate result, and optionally stores that result when a caller
or declared downstream consumer requests it. It does not own Git delivery,
merge policy, retries, queues, work ownership, or release transitions.
2026-07-14 17:39:11 -04:00
2026-07-15 00:21:02 -04:00
Repositories own delivery policy for local and cloud agents.
2026-07-14 22:01:50 -04:00
## Separation of responsibilities
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
``` text
Loop restore: converge and crank as control flow under the verdict contract (ADR-0017) (#1099)
## Loop restore: converge and crank as control flow under the verdict
contract (ADR-0017)
Intent source: `docs/plans/2026-09-03-loop-restore.md` (in this PR).
Decision record:
`docs/adr/ADR-0017-loop-as-control-flow-not-knowledge.md`.
**Why.** The 2026-07-14 single-pass cut (`482307762`) removed the
iterate loop (discovery, crank, converge, evolve, the learn write-half)
together with the unproven compounding claim, although ADR-0011 demoted
only the latter. The control flow was never demoted, and its absence
showed on 2026-09-02, when a three-lane fix needed eight validators and
two stops because the contract had no repair phase. This restores the
loop as control flow and nothing else: no knowledge store, no `ao
converge`/`ao crank`, no evolve, no canary. ADR-0004 and ADR-0011 stay
in force.
**What changes.**
- **RPI gains a bounded repair phase.** On `FAIL` or `NOT_PROVEN` with
findings, repair and re-validate freshly under the convergence law:
caller-declared `repair_rounds` (default 2); open finding set keyed by
stable `findings[].id`, union across validator families, non-growing; no
closed id reopens; the subject digest changed or, for `NOT_PROVEN`, new
digest-bound evidence resolved a named gap. Converged = fresh PASS plus
cross-family PASS on risky surfaces. Plan and Implement keep their
single dispatch. `skills/rpi/scripts/run_once.py` models the law as pure
data (33 tests): rounds are validated for shape (digest required, no
duplicate ids, no PASS with findings, no FAIL without findings),
condition 4's evidence branch needs a NOT_PROVEN previous round, a
non-FAIL current round, new evidence, and a resolved finding, and a PASS
over unchanged bytes after a FAIL is a flip that reports NOT_PROVEN.
`workflows/rpi.js` runs validation as legs (spawned or external primary,
plus a caller-supplied `crossFamily.command` on risky surfaces) merged
worst-of with a union of stable ids; a risky surface without a
cross-family leg is `diversity_unsatisfied` and never converges or
enters repair; a failed repair or re-validation returns NOT_PROVEN with
no stale verdict. Validators return `subjectDigest`, stable finding ids,
and `evidenceRefs`.
- **crank returns as a thin wave executor** (113 lines): the caller
selects the wave and the repair bound, crank invokes RPI per lane
(parallel only on disjoint write and regen scopes), runs the wave
acceptance once, returns evidence, and stops. No retry, budget, queue,
claim, lease, Git, closure, or next-work ownership. Routing golden
`rq-07-wave-execution` ranks it first.
- **validate is cross-family by default on risky surfaces**
(`cli/internal/gates/**`, `scripts/check-*.sh`, `tests/**`,
`skills/*/scripts/**`, hook policies, `lib/**`, security-scanned paths)
with the LAW-0 dispatch table: Claude orchestrating uses read-only
`codex exec`; Codex orchestrating uses an interactive Claude session in
an NTM pane, never `claude -p`. No live adapter means
`diversity_unsatisfied`, which on a risky surface is `NOT_PROVEN`. The
full literal CI command set runs once on the final integrated subject;
routine rounds keep the receipt-driven freshness contract.
- **Conformance assertions flipped under ADR-0017 only:**
`scripts/check-cathedral-cut-conformance.py` (crank live; "Stop
regardless" replaced by positive canaries for the law's four conditions;
a bounded `for` loop that compares against `repair_rounds` is required
in `run_repair_phase`, and the gate executes the law's canaries against
the reference behavior), `workflows/rpi.js`,
`skills/rpi/scripts/validate.sh`,
`evals/agentops-core/rpi-behavior.json`,
`skills/rpi/references/rpi.feature`. Every single-pass public surface
(README, AGENTS.md, PRODUCT.md, CI-CD, agent-workflow-reference,
rpi-traversal, cli/README, quickstart and demo commands, the
operating-contract and product-boundary bats, the Codex-description
oracle) now states repair to convergence.
**Known approximation, disclosed.** The Claude conveyor has no
deterministic shell primitive, so changed paths are derived by the fresh
validator (git status and diff against the clean pre-run tree) and
unioned with the implementer's report; risk is classified over that
union and unreported paths are coverage findings. A validator is still a
model; runtime derivation outside every agent is a follow-up. Family
distinctness of the cross-family leg is asserted by the caller's choice
of command and not verified by the script.
**Not in scope.** Premortem stays a single advisory judge and Plan still
only names the first check (phase boundaries unchanged). No `verdict.v2`
or `rpi-report.v1` change. The loop's own effect on outcomes is
unmeasured and owed a seeded-defect probe, like the rest of the corpus.
**Evidence on the tip.** Regen check clean; full gate green with a
HEAD-built binary; CI's bats command green; Go build/vet/test green;
golangci-lint clean; security gate quick PASS; one fresh validator over
the whole diff; one cross-family read of the design before
implementation (13 findings folded) and two of the integrated diff (9
findings in round one, 11 by round two, 15 by round three, each round
repaired and re-reviewed; the fresh validator passed the tip after round
two and the final tip 1e8adb72d passed a fresh validator (14-scenario
independent harness of the law, full gate 71/71 with a HEAD-built
binary, CI bats 1164/0) and a cross-family read by Gemini 3.8 via AGY,
which closed all six remaining residues with no new findings; Codex was
unreachable at push time).
**Follow-ups filed from the final reviews, not blockers:** the JS
violation check tests growth before reopen while Python tests reopen
first (same stop, different label when both occur in one round);
`cli/testdata/compatibility-baseline/families/{demo,quickstart}/case.json`
assert help-text substrings Cobra never prints (pre-existing, no
consumer); runtime derivation of changed paths outside every agent in
the Claude conveyor.
2026-09-03 11:16:08 -04:00
AgentOps: Plan -> Implement once -> fresh Validate -> bounded repair -> report
2026-07-14 22:01:50 -04:00
Repository: deterministic checks -> repository-selected Git/CI/release policy
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
```
2026-07-14 22:01:50 -04:00
`ao gate check` is an ordinary deterministic repository test runner. Success
means only that its selected checks passed. It cannot create, strengthen, or
replace a semantic verdict.
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
2026-07-15 00:21:02 -04:00
AgentOps installs no push hook and does not choose how a repository invokes
these checks. A repository may call `ao gate check` from a local command, CI,
pull request, merge queue, or another delivery process.
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
2026-07-14 22:01:50 -04:00
## GitHub workflows
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
2026-07-14 22:01:50 -04:00
| Workflow | Purpose |
|---|---|
| `.github/workflows/validate.yml` | Optional hosted execution of repository deterministic checks |
| `.github/workflows/release.yml` | Repository-owned tagged release publication |
| `.github/workflows/nightly.yml` | Scheduled deterministic regression and security coverage |
2026-05-16 09:18:33 -04:00
2026-07-14 22:01:50 -04:00
No workflow is a semantic validator. No workflow writes a Validate verdict.
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
2026-07-14 22:01:50 -04:00
## Current CI jobs
2026-07-14 17:39:11 -04:00
2026-07-14 22:01:50 -04:00
<!-- BEGIN GENERATED CI JOBS -->
<!-- Generated from docs/contracts/ci-jobs.yaml by scripts/generate-ci-jobs-table.sh. -->
2026-07-14 17:39:11 -04:00
| Job | What it validates | Common failure |
|-----|-------------------|----------------|
2026-07-14 22:01:50 -04:00
| **go-gate-shadow ** | Runs the ordinary Go gate registry with workflow-coverage reporting | A deterministic check failure or workflow/registry coverage mismatch |
| **correctness ** | Builds `ao` ; runs Go, schema, generated-surface, shell, and smoke tests | Compilation, test, schema, generated-drift, or portability failure |
| **security ** | Runs secret, dependency, static-analysis, and dangerous-pattern checks | A blocking security finding |
<!-- END GENERATED CI JOBS -->
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
2026-07-14 22:01:50 -04:00
## Local verification
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
2026-07-14 22:01:50 -04:00
Use the smallest focused checks while editing, then run the ordinary full suite
once for the complete candidate:
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
``` bash
2026-07-14 22:01:50 -04:00
cd cli && go test ./...
2026-07-15 00:21:02 -04:00
python3 scripts/check-cathedral-cut-conformance.py
python3 scripts/generate-skill-mesh.py --check
RPI flow: premortem on risky plans, finding class in the law, adjudicated splits, orphaned evidence, doc claims gated (#1104)
## RPI flow: premortem on risky plans, finding class in the law,
adjudicated splits, orphaned evidence, doc claims gated
Provenance: running the loop for real across three trains on 2026-09-03
(#1099, #1100, #1101). The spine held; the flow around it did not. A
risky-surface design shipped with no premortem and needed six repair
passes. The open finding set grew three rounds running inside one class
while the id-counting law watched. A same-family PASS and a cross-family
FAIL had no tie-break until the caller named one afterwards. Every
harness edit orphaned bound evidence with no warning until verify
failed. A doc called an egress log published while the repository's
`*.log` rule kept it out of the tree, and the same overclaim class came
back four rounds.
**What changes.**
- **Premortem at plan exit.** When the write scope hits a risky surface,
one fresh judge reads the frozen plan before Implement; a blocking
finding returns NOT_PLANNED with the findings. The caller may declare
`premortem: skip`; the report says so.
- **A split is recorded, never elected and never closed inside the
loop.** The law stands: a risky surface converges only when both legs
PASS, and no finding leaves the open set because someone chose a judge.
Plan declares `binding_judge` as the caller's disposition for a split
that survives repair; it is bound in the plan identity, carried in the
report, and never mutates the verdict. On a risky split that survives
the repair budget (never after a law stop), one council leg receives a
bounded packet marked untrusted and returns per-finding rulings, exactly
one per id, which the traversal validates and records under
`council.rulings` for the caller's next intent; the verdict and the open
finding set stay exactly as repair left them. A council closure
mechanism was built, drew a new finding class in three consecutive judge
rounds (path containment, digest exclusion, presence-as-evidence,
partial closure downgrading a FAIL), and was cut under the train's own
class rule rather than hardened again. A non-risky split stays worst-of.
- **Finding class in the convergence law.** Findings carry an optional
stable `class`. A class carried by a resolved id that reappears on a new
id with no surviving prior id carrying it is `class_reopened`, whether
or not a classless round sits between; a blank class is an invalid
round; a round can carry both a reopened id and a reopened class and
stops on either. Repair stops, the traversal returns NOT_PROVEN, and the
caller goes back to Plan, because the design is wrong, not the patch.
Same rule in `run_once.py` and `workflows/rpi.js`, driven by one shared
fixture file, with executed canaries in the cathedral gate.
- **Plan identity, and the premortem's risky test.** On a risky scope
the Plan carries `binding_judge` and the evidence it expects to orphan;
`plan_digest`, when the plan declares one, must equal the script's
digest over the acceptance, the write scope, the binding judge, and the
intent-source digest, or the traversal stops before Implement. The risky
test over a declared scope is glob intersection: witness paths, the
risky regexes over the glob with wildcards instantiated, and any literal
prefix that reaches a risky root; a bare `**` or `*` is risky.
- **Intent is verified before anything is built.** An `intent-snapshot`
receipt re-derives the digest and compares the snapshot bytes to the
caller intent before premortem or Implement; absent, mis-hashed, or
wrong stops the traversal with `plan_identity_mismatch`. The Python
reference refuses on the same rule. Declared scopes are canonicalized
(repeated `./` collapsed; absolute paths and `..` refused at plan exit),
and the four spellings of `tests/**` share one plan digest while
`tests/` does not.
- **Orphaned evidence is a receipt, not a surprise.** After Implement
and after every repair round the workflow runs
`scripts/evidence-orphans.sh` over the runtime-derived path union, keyed
on the round so a same-set repair still reruns it and appends every
scorecard and fixture set whose bound evaluator or skill digests cover
them to the check receipts the validator reads, each with a cause
(`changed_path`, `digest_drift`, `both`, `skill_changed`). Plan budgets
that recapture up front. On today's tree the receipt already lists the
08-16 and 08-26 sets whose bound harness digests moved long ago.
- **Doc claims are gated.** The stronger scan already caught one stale
claim in the tree (a 2026-06-22 harvest note calling evidence committed
after Train 2 had removed that tree); the sentence now says where the
evidence went. `docs.claims-tracked` (blocking) requires backticked repo
paths in `evals/` and `docs/evals/` markdown to be tracked, and
sentences that say published, tracked, or committed to name files that
exist.
- **Risky scope is a glob intersection, not a heuristic.** One
segment-wise `globsIntersect` over one list of risky segment globs
answers both the declared-scope question at Plan exit and the
changed-path question for the cross-family rule; two heuristic designs
missed in opposite directions and were replaced rather than patched.
- **Contracts.** rpi gains a judgment dispatch table: risky scope routes
to premortem, a risky split to council, an irreversible landing to
one-way-door as a caller choice. plan, implement, validate, premortem,
council, and learn carry their halves; learn proposes a deterministic
check for any finding class that repeats.
`docs/architecture/rpi-traversal.md` is the source of record and was
updated with the flow.
**Evidence on the tip.** Regen check clean; Go bar; full gate green with
a HEAD-built binary (the new gate fires and passes); CI's literal bats
command green; rpi unit tests; cathedral gate with executed canaries for
the class law; routing goldens; token budgets; gate-tightening ratchet;
coverage 1 of 12 unchanged.
**Judgment, stated plainly.** Six judging rounds, each on a frozen tip.
The caller elected Fable 5.1 as the binding validator; its round-6
verdict is PASS with no verdict-driving findings (lows: a council
payload outside the schema throws a raw error instead of recording
`invalid-rulings`; two stale comments; a dead parameter; a dead fixture
element; the contracts say exactly one ruling per finding while the code
enforces at most one). The cross-family Codex read of the same tip
failed on three input-validation items, which the final commit closes:
control characters are now refused on the original scope value with the
full Unicode control class and the caller scope is canonicalized in
place so one spelling reaches every prompt; the intent receipt no longer
forgives a trailing-newline difference; and the Python reference accepts
a digest string only from a verifier callback and matches digests with
`fullmatch`. One half of one Codex item stays open and is disclosed: the
workflow does not itself hash the caller intent's bytes to compare with
the plan's declared digest (it relies on the receipt's exact-bytes match
plus digest equality); binding that locally means every probe fixture
carries a real digest of its intent text, which is the next pass. Also
open, pre-existing and outside this train: the risky-surface list names
`lib/**` while the preamble every gate sources lives under
`scripts/lib/`. Across the rounds the loop behaved as designed once: the
council closure mechanism drew a fresh finding class three rounds
running and was cut under the train's own class rule instead of hardened
a fourth time.
2026-09-05 19:03:51 -04:00
bash scripts/check-doc-claims-tracked.sh
2026-07-15 00:21:02 -04:00
bash scripts/ci-local-release.sh
feat(docs,tests): wave 1 — testing & documentation improvements (na-xjw)
- Fix 11 stale `tests/claude-code/` paths in testing-skills.md
- Remove phantom e2e-install-test.sh reference and --skip-e2e-install flag
- Add validate.sh to athena, grafana-platform-dashboard, push skills
- Create docs/TESTING.md umbrella testing guide
- Create docs/CI-CD.md CI architecture document
- Add README.md to 6 test subdirectories
- Add 17 BATS unit tests for lib/hook-helpers.sh functions
- Wire orphaned tests into runners (run-all.sh + validate.yml)
Issues: na-xjw.1, na-xjw.2, na-xjw.3, na-xjw.4, na-xjw.5, na-xjw.6, na-xjw.8, na-xjw.10
2026-03-05 10:37:45 -05:00
```
2026-07-14 22:01:50 -04:00
The local release script validates this repository's release artifacts. That is
repository policy, not an AgentOps lifecycle transition.
