feat(goals): gate acceptance vectors for mappable scenario directives + nightly dream-cycle evaluate step (age-26y1)

This commit is contained in:
boshu
2026-07-01 21:57:11 -04:00
parent 39c4067649
commit 1ab0b09219
9 changed files with 54 additions and 0 deletions
+30
View File
@@ -251,6 +251,35 @@ jobs:
fi
scripts/nightly-knowledge-cycle.sh record-stage dream-cycle ok "summary at dream-cycle/summary.md"
- name: Dream cycle — scenario satisfaction (report-only)
if: steps.precondition.outputs.decision == 'RUN'
env:
NIGHTLY_KNOWLEDGE_CYCLE_DIR: ${{ runner.temp }}/knowledge-cycle
run: |
# Producer side of the satisfaction panel (ag-26y1): evaluate every
# directive-linked scenario against its acceptance vectors and append
# the verdicts to .agents/rpi/scenario-results.json so satisfaction is
# measured nightly alongside the dream cycle. Runs at the repo root —
# it needs GOALS.md, spec/scenarios/, and the real gate scripts — NOT
# inside nightly-dream-cycle.sh, which snapshots .agents/ into an
# ephemeral, GOALS-less workspace and is deliberately non-mutating.
# REPORT-ONLY: a gate-shaped scenario that fails its own threshold must
# not fail the cycle — mirror the compile-mine warn-on-nonzero guard.
SCEN_DIR="${RUNNER_TEMP}/knowledge-cycle/scenario-satisfaction"
mkdir -p "$SCEN_DIR"
set +e
./cli/bin/ao eval scenario evaluate --all --json > "$SCEN_DIR/report.json"
rc=$?
if [[ $rc -ne 0 ]]; then
scripts/nightly-knowledge-cycle.sh record-stage scenario-satisfaction warn "ao eval scenario evaluate exited $rc (report-only; does not block the cycle)"
else
written=$(jq -r '.written // 0' "$SCEN_DIR/report.json" 2>/dev/null || echo 0)
gate_pass=$(jq -r '[.evaluations[] | select(.shape == "gate" and .verdict == "pass")] | length' "$SCEN_DIR/report.json" 2>/dev/null || echo 0)
gate_total=$(jq -r '[.evaluations[] | select(.shape == "gate")] | length' "$SCEN_DIR/report.json" 2>/dev/null || echo 0)
scripts/nightly-knowledge-cycle.sh record-stage scenario-satisfaction ok "wrote ${written} result(s); ${gate_pass}/${gate_total} gate-shaped scenarios satisfied"
fi
exit 0
- name: Athena follow-up (placeholder — shares compile substrate)
if: steps.precondition.outputs.decision == 'RUN'
env:
@@ -269,6 +298,7 @@ jobs:
run: |
scripts/nightly-knowledge-cycle.sh record-stage compile skip "${{ steps.precondition.outputs.reason }}"
scripts/nightly-knowledge-cycle.sh record-stage dream-cycle skip "${{ steps.precondition.outputs.reason }}"
scripts/nightly-knowledge-cycle.sh record-stage scenario-satisfaction skip "${{ steps.precondition.outputs.reason }}"
scripts/nightly-knowledge-cycle.sh record-stage athena skip "${{ steps.precondition.outputs.reason }}"
- name: Append knowledge-cycle summary to step summary
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given the three install scripts (install.sh, install-codex.sh, install-opencode.sh), when the install-smoke CI gate runs tests/install/test-install-smoke.sh, then all three scripts pass syntax+structure validation and, when cli/bin/ao exists, ao --version / ao help / and the flywheel, goals, inject subcommands are confirmed registered..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given the three install scripts (install.sh, install-codex.sh, install-opencode.sh), when the install-smoke CI gate runs tests/install/test-install-smoke.sh, then all three scripts pass syntax+structure validation and, when cli/bin/ao exists, ao --version / ao help / and the flywheel, goals, inject subcommands are confirmed registered..",
"satisfaction_threshold": 0.9,
"acceptance_vectors": [
{ "dimension": "correctness", "threshold": 1.0, "check": "gate:install-smoke" }
],
"source": "human",
"status": "active"
}
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given the tests/_quarantine/ directory, when the validate-quarantine-empty CI gate runs, then it finds zero active quarantined test suites (the directory stays empty or every entry is promoted, deleted, or tracked as explicit follow-up)..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given the tests/_quarantine/ directory, when the validate-quarantine-empty CI gate runs, then it finds zero active quarantined test suites (the directory stays empty or every entry is promoted, deleted, or tracked as explicit follow-up)..",
"satisfaction_threshold": 1,
"acceptance_vectors": [
{ "dimension": "correctness", "threshold": 1.0, "check": "gate:quarantine-empty" }
],
"source": "human",
"status": "active"
}
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given a single learning in the knowledge corpus, when scripts/check-flywheel-lifecycle.sh runs, then all five stages (capture, retrieval, inject, round-trip, citation) are traced and the flywheel-lifecycle CI gate passes..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given a single learning in the knowledge corpus, when scripts/check-flywheel-lifecycle.sh runs, then all five stages (capture, retrieval, inject, round-trip, citation) are traced and the flywheel-lifecycle CI gate passes..",
"satisfaction_threshold": 0.9,
"acceptance_vectors": [
{ "dimension": "correctness", "threshold": 1.0, "check": "gate:flywheel-lifecycle" }
],
"source": "human",
"status": "active"
}
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given the cli/ and cli/internal/ Go packages, when the go-complexity CI gate runs golangci-lint, then zero functions exceed cyclomatic complexity 20 (cli/) and 18 (cli/internal/)..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given the cli/ and cli/internal/ Go packages, when the go-complexity CI gate runs golangci-lint, then zero functions exceed cyclomatic complexity 20 (cli/) and 18 (cli/internal/)..",
"satisfaction_threshold": 1,
"acceptance_vectors": [
{ "dimension": "maintainability", "threshold": 1.0, "check": "gate:go-complexity-ceiling" }
],
"source": "human",
"status": "active"
}
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given a change that introduces codex parity drift, when the codex-parity CI gate (scripts/audit-codex-parity.sh) runs at push time, then it reports zero parity findings and blocks the push if any drift exists..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given a change that introduces codex parity drift, when the codex-parity CI gate (scripts/audit-codex-parity.sh) runs at push time, then it reports zero parity findings and blocks the push if any drift exists..",
"satisfaction_threshold": 1,
"acceptance_vectors": [
{ "dimension": "consistency", "threshold": 1.0, "check": "gate:codex-parity-drift" }
],
"source": "human",
"status": "active"
}
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given the workbench-behavioral-v1 eval suite (12 tasks), when make -C evals/workbench verify runs, then golden solutions score 12/12 pass and broken variants are detected 12/12, and the eval-skill-delta CI gate validates the structure..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given the workbench-behavioral-v1 eval suite (12 tasks), when make -C evals/workbench verify runs, then golden solutions score 12/12 pass and broken variants are detected 12/12, and the eval-skill-delta CI gate validates the structure..",
"satisfaction_threshold": 0.9,
"acceptance_vectors": [
{ "dimension": "correctness", "threshold": 1.0, "check": "gate:eval-workbench-verify" }
],
"source": "human",
"status": "active"
}
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given a fresh checkout after a runtime-state wipe, when the .agents/ snapshot/restore mechanism (soc-rv5p) rehydrates from durable storage, then scripts/corpus-stats.sh reports non-degraded learning/pattern/decision counts and the corpus-freshness gate confirms coverage..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given a fresh checkout after a runtime-state wipe, when the .agents/ snapshot/restore mechanism (soc-rv5p) rehydrates from durable storage, then scripts/corpus-stats.sh reports non-degraded learning/pattern/decision counts and the corpus-freshness gate confirms coverage..",
"satisfaction_threshold": 0.85,
"acceptance_vectors": [
{ "dimension": "durability", "threshold": 1.0, "check": "gate:corpus-freshness" }
],
"source": "human",
"status": "active"
}
+3
View File
@@ -7,6 +7,9 @@
"narrative": "A user or evaluator exercises the system behavior for this goal: Given a surface the teardown (epic ag-097) explicitly removed and recorded in scripts/removed-apparatus.txt, when an /evolve cycle or any change reintroduces that surface, then the no-apparatus-regrowth gate fails — teardown-removed apparatus stays removed..",
"expected_outcome": "The implementation satisfies the goal in observable behavior: Given a surface the teardown (epic ag-097) explicitly removed and recorded in scripts/removed-apparatus.txt, when an /evolve cycle or any change reintroduces that surface, then the no-apparatus-regrowth gate fails — teardown-removed apparatus stays removed..",
"satisfaction_threshold": 0.85,
"acceptance_vectors": [
{ "dimension": "regression", "threshold": 1.0, "check": "gate:no-apparatus-regrowth" }
],
"source": "human",
"status": "active"
}