mirror of
https://github.com/conorbronsdon/avoid-ai-writing.git
synced 2026-09-19 01:32:11 +08:00
Catch promo-surface drift from the release that causes it (#80)
* Catch promo-surface drift from the release that causes it The pattern count and word-table count are already policed inside this repo: check-pattern-count.sh derives both from SKILL.md and fails CI if the README disagrees. Nothing policed the second link — four surfaces in other repos quote those numbers, and a release moves the number here while they sit still. That drift cannot be caught from the other side. Their files do not change on release day, so no pre-commit hook over there ever has a commit to fire on. It went unnoticed for three weeks: three of the four surfaces were still on 53 pattern categories and 109 table entries. So the check runs here, on release, with a weekly backstop: - .ssot.yaml declares the README bullets as canonical and the four surfaces as copies (dogfoods conorbronsdon/ssot-check) - promo-drift.yml clones the surfaces as siblings and runs the check - promo-drift-report.py turns the JSON into a readable issue body A surface that cannot be read is reported as NOT CHECKED and does not fail the job — conorbronsdon-site is private, and failing red every week over a missing token trains people to ignore the check. It is always printed. Set SITE_REPO_TOKEN to bring it into coverage; no code change needed. Verified against real clones of all four surfaces: in-sync exits 0, a stale number exits 1 with file:line, an unreadable surface exits 0 and says so, and a reworded canonical bullet is a hard failure rather than a silent pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YA7143ow5qvWZghbeQKiUd * Drop the site from the manifest; the check now needs no credential conorbronsdon.com/builds floors both counts instead of quoting them, so there is nothing left to keep in sync there. That removes the only private surface, which removes the only reason this workflow wanted a PAT. Every tracked surface is now a public repo and the job runs on its own GITHUB_TOKEN. The not-checked path stays — a surface repo can still be renamed or fail to clone, and that must read as a coverage gap, not a pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YA7143ow5qvWZghbeQKiUd --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Conor Bronsdon
@@ -0,0 +1,100 @@
|
||||
name: promo-drift
|
||||
|
||||
# Every release moves the pattern count and the word-table count. The promo
|
||||
# surfaces that quote those numbers live in other repos and do not change on
|
||||
# that day, so nothing over there has a commit to hook onto — the drift is
|
||||
# created here and can only be caught from here.
|
||||
#
|
||||
# Runs on release, weekly as a backstop, and on demand. Opens (or updates) a
|
||||
# single issue when a surface has gone stale.
|
||||
#
|
||||
# Every tracked surface is a public repo, so this needs no secret beyond the
|
||||
# job's own GITHUB_TOKEN. conorbronsdon.com/builds floors both counts instead
|
||||
# of quoting them, which is why it is not tracked and no PAT is required here.
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [published]
|
||||
schedule:
|
||||
# Mondays 15:00 UTC. Weekly is enough: releases fire the check directly,
|
||||
# and this only has to catch a release that was cut some other way.
|
||||
- cron: "0 15 * * 1"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
concurrency:
|
||||
group: promo-drift
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
check:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
# ssot-check resolves `../<repo>/...` copies against real directories on
|
||||
# disk, so the surfaces have to be cloned as siblings of this checkout.
|
||||
# Depth 1: only the current content of each file matters.
|
||||
- name: Clone the public promo surfaces
|
||||
env:
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
run: |
|
||||
cd "$GITHUB_WORKSPACE/.."
|
||||
for repo in agent-skills ai-tools-for-creators conorbronsdon; do
|
||||
git clone --depth 1 --quiet \
|
||||
"https://github.com/$OWNER/$repo.git" "$repo"
|
||||
done
|
||||
|
||||
- name: Get ssot-check
|
||||
run: |
|
||||
cd "$GITHUB_WORKSPACE/.."
|
||||
git clone --depth 1 --quiet https://github.com/conorbronsdon/ssot-check.git ssot-check
|
||||
|
||||
- name: Check the promo surfaces
|
||||
id: check
|
||||
run: |
|
||||
set -o pipefail
|
||||
# `check` exits 1 on any problem; the report script decides what is
|
||||
# actually actionable, so do not let the raw exit code kill the step.
|
||||
python3 ../ssot-check/ssot_check.py check --json > result.json || true
|
||||
if ! python3 scripts/promo-drift-report.py < result.json > report.md; then
|
||||
echo "drift=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "drift=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
cat report.md >> "$GITHUB_STEP_SUMMARY"
|
||||
cat report.md
|
||||
|
||||
- name: Open or update the drift issue
|
||||
if: steps.check.outputs.drift == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
title="Promo surfaces have drifted from the canonical counts"
|
||||
{
|
||||
echo "The counts in this repo's README moved and one or more"
|
||||
echo "surfaces elsewhere still quote the old ones."
|
||||
echo
|
||||
cat report.md
|
||||
echo
|
||||
echo "Canonical is this repo. Fix the surfaces, not the README —"
|
||||
echo "the README is already asserted against SKILL.md by"
|
||||
echo "\`scripts/check-pattern-count.sh\`."
|
||||
echo
|
||||
echo "_Opened by [promo-drift](.github/workflows/promo-drift.yml)._"
|
||||
} > issue.md
|
||||
|
||||
# Loose search, then exact title match, so a differently-worded issue
|
||||
# never gets commented on by mistake.
|
||||
existing="$(gh issue list --state open --search "in:title drifted from the canonical counts" \
|
||||
--json number,title \
|
||||
--jq --arg t "$title" 'map(select(.title == $t)) | .[0].number // empty')"
|
||||
|
||||
if [ -n "$existing" ]; then
|
||||
gh issue comment "$existing" --body-file issue.md
|
||||
else
|
||||
gh issue create --title "$title" --body-file issue.md
|
||||
fi
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
# Which promo surfaces copy this repo's counts, and where the real number lives.
|
||||
#
|
||||
# There are two links in this chain and they are policed separately:
|
||||
#
|
||||
# SKILL.md -> README.md scripts/check-pattern-count.sh (every push)
|
||||
# README.md -> promo surfaces this manifest (.github/workflows/promo-drift.yml)
|
||||
#
|
||||
# The first link derives the count from the catalog, so the README cannot lie
|
||||
# about SKILL.md. The second link is the one that actually rots: a release moves
|
||||
# the number here, the four surfaces below sit still, and nothing in *their*
|
||||
# repos changes — so no pre-commit hook over there can ever fire on it. That is
|
||||
# why this check is a release/cron job and not a hook.
|
||||
#
|
||||
# Checked with https://github.com/conorbronsdon/ssot-check:
|
||||
# python3 ssot_check.py check # from this repo root, siblings cloned at ../
|
||||
#
|
||||
# Adding a surface is two lines under `copies`. Do not add a surface that floors
|
||||
# the number ("60+ pattern categories") — a floored number is a different,
|
||||
# looser fact, and tracking it as a copy fires false drift on every release.
|
||||
#
|
||||
# conorbronsdon.com/builds is deliberately absent for exactly that reason: it
|
||||
# floors both counts, so there is nothing to keep in sync. It is also the one
|
||||
# private surface, and excluding it keeps this check free of any credential.
|
||||
|
||||
facts:
|
||||
- name: avoid-ai-pattern-count
|
||||
type: integer
|
||||
note: >
|
||||
The human-facing prose-catalog category count. Canonical is the README
|
||||
bullet, which check-pattern-count.sh derives from SKILL.md's `###` entries
|
||||
under "What to remove or fix". This is NOT the detector engine's `type`
|
||||
category count — that number is deliberately different and the mapping
|
||||
lives in detector/CATEGORIES.md. Never reconcile the two.
|
||||
canonical:
|
||||
file: README.md
|
||||
pattern: '\*\*(\d+) pattern categories\*\*'
|
||||
copies:
|
||||
- file: ../agent-skills/README.md
|
||||
pattern: '(\d+) pattern categories'
|
||||
- file: ../ai-tools-for-creators/README.md
|
||||
pattern: '(\d+) pattern categories'
|
||||
- file: ../conorbronsdon/README.md
|
||||
pattern: '(\d+) pattern categories'
|
||||
|
||||
- name: avoid-ai-replacement-table
|
||||
type: integer
|
||||
note: >
|
||||
Word replacement table entry count (Tier 1 + 2 + 3 data rows; the Tier 3
|
||||
*phrases* table is counted separately in the README bullet and excluded
|
||||
here). Canonical is the README bullet, derived from SKILL.md by
|
||||
check-pattern-count.sh. agent-skills is absent below on purpose: it cites
|
||||
the pattern count but not this one.
|
||||
canonical:
|
||||
file: README.md
|
||||
pattern: '\*\*(\d+)-entry word replacement table'
|
||||
copies:
|
||||
- file: ../ai-tools-for-creators/README.md
|
||||
pattern: '(\d+)-entry replacement table'
|
||||
- file: ../conorbronsdon/README.md
|
||||
pattern: '(\d+)-entry replacement table'
|
||||
Executable
+98
@@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Turn `ssot_check.py check --json` output into a readable drift report.
|
||||
|
||||
Reads the JSON on stdin, writes markdown to stdout. Exit code decides whether
|
||||
the workflow opens an issue:
|
||||
|
||||
0 every surface that could be read is in sync
|
||||
1 at least one surface carries a stale number
|
||||
|
||||
A surface that could not be read at all (repo renamed, moved, or a clone that
|
||||
failed) is reported as NOT CHECKED and does *not* fail the job. That case is a
|
||||
coverage gap, not drift, and conflating the two makes a red check meaningless.
|
||||
It is always printed, never silently dropped.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
|
||||
# Statuses ssot-check emits for a copy it managed to read and compare.
|
||||
REAL_DRIFT = {"drifted", "stale_entry"}
|
||||
UNREADABLE = {"unverified"}
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
result = json.load(sys.stdin)
|
||||
except json.JSONDecodeError as exc:
|
||||
print(f"could not parse ssot-check output: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
drifted, unreadable, checked = [], [], 0
|
||||
|
||||
for fact in result.get("facts", []):
|
||||
name = fact.get("name", "?")
|
||||
canon = fact.get("canonical", {})
|
||||
cval = canon.get("value")
|
||||
|
||||
# A broken canonical means the README bullet moved or was reworded.
|
||||
# That is this repo's own problem and always a hard failure.
|
||||
if fact.get("status") == "canonical_moved":
|
||||
drifted.append(
|
||||
f"- **{name}** — canonical unreadable in "
|
||||
f"`{canon.get('file')}`: {canon.get('error')}. "
|
||||
f"The pattern in `.ssot.yaml` no longer matches the README."
|
||||
)
|
||||
continue
|
||||
|
||||
for copy in fact.get("copies", []):
|
||||
status = copy.get("status")
|
||||
if status in REAL_DRIFT:
|
||||
drifted.append(
|
||||
f"- **{name}** — `{copy.get('file')}`"
|
||||
f"{':' + str(copy['line']) if copy.get('line') else ''} "
|
||||
f"says **{copy.get('value', '?')}**, canonical is "
|
||||
f"**{cval}**"
|
||||
+ (f" ({copy['note']})" if copy.get("note") else "")
|
||||
)
|
||||
elif status in UNREADABLE:
|
||||
# ssot-check reports the *source* it tried, so a missing file
|
||||
# comes back as the bare word "local". Say what that means.
|
||||
note = copy.get("note") or ""
|
||||
if note in ("", "local"):
|
||||
note = ("file not present on the runner — the surface repo "
|
||||
"was renamed, moved, or failed to clone")
|
||||
unreadable.append(f"- **{name}** — `{copy.get('file')}` ({note})")
|
||||
else:
|
||||
checked += 1
|
||||
|
||||
lines = []
|
||||
if drifted:
|
||||
lines.append(
|
||||
"The number moved here and these surfaces still carry the old one.\n"
|
||||
)
|
||||
lines.append("### Drifted\n")
|
||||
lines.extend(drifted)
|
||||
lines.append("")
|
||||
|
||||
if unreadable:
|
||||
lines.append("### Not checked\n")
|
||||
lines.append(
|
||||
"These surfaces could not be read on the runner, so they are "
|
||||
"unverified either way — not a pass.\n"
|
||||
)
|
||||
lines.extend(unreadable)
|
||||
lines.append("")
|
||||
|
||||
lines.append(
|
||||
f"_{checked} copies verified in sync · "
|
||||
f"{len(drifted)} drifted · {len(unreadable)} not checked · "
|
||||
f"ssot-check {result.get('version', '?')} on "
|
||||
f"{result.get('generated', '?')}_"
|
||||
)
|
||||
|
||||
print("\n".join(lines))
|
||||
return 1 if drifted else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user