proffesor-for-testing__agentic-qe/docs/policies/release-verification.md

Release Verification Policy

The release-gate corpus is the primary verification. Manual checks supplement the gate, they do not replace it. This file is the canonical release-process policy; for the maintainer-facing how-to (gate architecture, failure interpretation, fixture-add procedure) see docs/VERIFICATION.md.

Critical requirements

This is a critical policy to ensure release quality:

• ✅ ALWAYS let the pre-publish-gate job in .github/workflows/npm-publish.yml decide whether a release publishes. The gate is load-bearing.
• ❌ NEVER bypass the gate via gh workflow run --ref <tag> after the gate has rejected a tag. Fix the bug, push the fix, re-tag.
• ✅ ALWAYS verify version numbers are updated in ALL documentation BEFORE creating the release tag (see "Version Update Policy" below — this is the one part of pre-release verification the gate cannot do for you).
• ❌ NEVER delete + re-create a release tag without the underlying fix landing on main first.
• ✅ ALWAYS read the post-publish-canary result before declaring a release done. The canary catches packaging mismatches the pre-publish gate can't.

Pre-#401 manual checklist (now obsolete — superseded by the gate)

Before #401, this file documented a long manual aqe init checklist (create test project, install RC, run init, eyeball outputs). That checklist is now the pre-publish-gate job's responsibility, with 22 assertions per fixture across 4 pinned real public repositories. The gate catches the failure modes the manual checklist tried to catch — and several it never could, like phase 06 delta-scan hangs that only reproduce on the second init pass.

If you find yourself wanting to run the manual checklist instead of trusting the gate, that is a signal that the gate has a coverage gap. Open an issue and add a fixture to the corpus rather than reinstating the manual workflow.

Manual sanity checks (post-gate, supplementary)

After the gate has passed and npm publish has run, the following manual checks are still useful — they cover things the gate intentionally doesn't:

# 1. Verify the published version matches what main says
npm view agentic-qe version
node -p "require('./package.json').version"

# 2. Spot-check that documented agent / skill counts still match reality
find .claude/agents -name "*.md" | wc -l
find .claude/skills -name "*.md" | wc -l

# 3. After the post-publish-canary run completes, confirm it stayed green
gh run list --workflow=post-publish-canary.yml --limit=5

# 4. Look at the verification matrix in the new release notes file
#    (generated by scripts/embed-verification-matrix.sh — see VERIFICATION.md)
cat docs/releases/v$(node -p "require('./package.json').version").md

These are smoke tests, not gates. If any of them surfaces a real problem, the corpus is the place to add coverage so the gate catches it next time.

Version Update Policy (CRITICAL)

The gate verifies that init works on the new version. It does NOT verify that documentation references the new version. That is your responsibility before tagging.

• ❌ NEVER create a release PR without updating version numbers in ALL documentation.
• ✅ ALWAYS update version numbers BEFORE creating the release tag.
• ✅ ALWAYS check and update these files:
  • README.md (version line and Recent Changes section)
  • package.json (the source of truth)
  • Any other docs referencing the current version
• ✅ ALWAYS search for old version references: grep -r "v3\.9\.[0-9]" README.md docs/ --exclude-dir=docs/releases

Version Update Workflow

# 1. Search for old version references (excluding historical docs)
grep -r "v3.9.7\|Version 3.9.7" README.md --exclude-dir=docs/releases

# 2. Update all found references to the new version
#    - README.md header: Version X.Y.Z
#    - README.md Recent Changes: add new version section
#    - docs/releases/vX.Y.Z.md: new file (use the previous one as a template)

# 3. Embed the verification matrix into the new release notes file
#    (after npm-publish.yml has run for the tag)
./scripts/embed-verification-matrix.sh <run-id> >> docs/releases/vX.Y.Z.md

# 4. Commit the docs update + open the release PR

Examples of correct behavior

• User: "prepare release X.Y.Z" → Update README + package.json versions FIRST, write docs/releases/vX.Y.Z.md, then propose the tag.
• User: "the gate rejected the tag" → Read init-corpus-logs artifact, reproduce locally, fix, push to main, delete the rejected tag, recreate it.
• User: "tag and ship vX.Y.Z" → Confirm: README updated? release notes file written? gate green on main? canary green after publish? matrix embedded?
• User: "let's bypass the gate just this once" → STOP. The gate exists because v3.9.1–v3.9.4 shipped without one. Fix the bug or fix the gate, do not bypass it.

Purpose

This policy ensures that:

1. The release-gate corpus is treated as authoritative — not as advice.
2. Version-update mistakes (the one class of pre-release error the gate can't catch) are blocked by maintainer process.
3. Post-publish verification (the canary, the matrix in release notes) is visible to humans, not just to CI.

The motivating incident is #401 — the v3.9.1–v3.9.4 init regression series, which shipped because pre-release verification was honor-system. This policy + the gate it points to are the structural fix.

---

Related:

• docs/VERIFICATION.md — gate architecture, failure interpretation, fixture-add how-to
• tests/fixtures/init-corpus/README.md — the corpus
• tests/fixtures/init-chaos/README.md — the weekly chaos workflow
• Issue #401 — the post-mortem
• Git Operations Policy
• Test Execution Policy