scripts/test_validate_compliance_fixtures.py

"""Unit tests for validate_compliance_fixtures.py."""
import unittest
from pathlib import Path
from tempfile import TemporaryDirectory

from tests.test_helpers import run_script

SCRIPT = Path(__file__).resolve().parent / "validate_compliance_fixtures.py"
REAL_FIXTURES = Path(__file__).resolve().parent.parent / "examples" / "compliance"


class TestValidateComplianceFixtures(unittest.TestCase):
    def test_real_fixtures_all_pass(self) -> None:
        result = run_script(SCRIPT, str(REAL_FIXTURES))
        self.assertEqual(result.returncode, 0, msg=result.stdout + result.stderr)
        self.assertIn("OK", result.stdout)
        self.assertNotIn("FAIL", result.stdout)
        self.assertNotIn("no fixture", result.stderr.lower())

    def test_empty_dir_fails(self) -> None:
        with TemporaryDirectory() as tmp:
            result = run_script(SCRIPT, tmp)
            self.assertEqual(result.returncode, 1)
            self.assertIn("no fixture", result.stderr.lower())

    def test_invalid_fixture_fails(self) -> None:
        with TemporaryDirectory() as tmp:
            bad = Path(tmp) / "fixture_invalid.yaml"
            bad.write_text("mode: not_a_valid_mode\n", encoding="utf-8")
            result = run_script(SCRIPT, tmp)
            self.assertEqual(result.returncode, 1)
            self.assertIn("FAIL", result.stdout)


if __name__ == "__main__":
    unittest.main()
ci: wire compliance validator + freshness check into workflows Plan Task 13 asked for all three steps (fixture loop, validator unit tests, freshness check) to land in spec-consistency.yml as a `for` loop with embedded python heredoc. Two deviations: 1. Replace the YAML→JSON inline shell heredoc with a helper script scripts/validate_compliance_fixtures.py (+ 3 unit tests). The heredoc approach is fragile under shell escaping and breaks if a fixture filename contains spaces; the helper converts each YAML fixture to a temp JSON file and delegates to check_compliance_report.py. 2. Move the freshness check to a separate workflow (.github/workflows/freshness-check.yml) with a weekly cron (Monday 09:00 UTC) + path-filtered push trigger + workflow_dispatch. Per efficiency review: a 180-day staleness threshold does not benefit from running on every push; weekly cron is the right cadence and push triggers only when the protocol file itself changes. spec-consistency.yml now runs the compliance validator on every push/PR (fixture loop + unit tests covering 21 + 3 = 24 tests). Freshness runs weekly or when prisma_trAIce_protocol.md changes. All 29 unit tests green (compliance 21 + freshness 5 + fixture validator 3). 2026-04-20 17:39:44 +08:00			`"""Unit tests for validate_compliance_fixtures.py."""`
			`import unittest`
			`from pathlib import Path`
			`from tempfile import TemporaryDirectory`

refactor(tests): move shared helpers into tests package (#311) 2026-06-04 09:33:43 +08:00			`from tests.test_helpers import run_script`
ci: wire compliance validator + freshness check into workflows Plan Task 13 asked for all three steps (fixture loop, validator unit tests, freshness check) to land in spec-consistency.yml as a `for` loop with embedded python heredoc. Two deviations: 1. Replace the YAML→JSON inline shell heredoc with a helper script scripts/validate_compliance_fixtures.py (+ 3 unit tests). The heredoc approach is fragile under shell escaping and breaks if a fixture filename contains spaces; the helper converts each YAML fixture to a temp JSON file and delegates to check_compliance_report.py. 2. Move the freshness check to a separate workflow (.github/workflows/freshness-check.yml) with a weekly cron (Monday 09:00 UTC) + path-filtered push trigger + workflow_dispatch. Per efficiency review: a 180-day staleness threshold does not benefit from running on every push; weekly cron is the right cadence and push triggers only when the protocol file itself changes. spec-consistency.yml now runs the compliance validator on every push/PR (fixture loop + unit tests covering 21 + 3 = 24 tests). Freshness runs weekly or when prisma_trAIce_protocol.md changes. All 29 unit tests green (compliance 21 + freshness 5 + fixture validator 3). 2026-04-20 17:39:44 +08:00
			`SCRIPT = Path(__file__).resolve().parent / "validate_compliance_fixtures.py"`
			`REAL_FIXTURES = Path(__file__).resolve().parent.parent / "examples" / "compliance"`


			`class TestValidateComplianceFixtures(unittest.TestCase):`
			`def test_real_fixtures_all_pass(self) -> None:`
			`result = run_script(SCRIPT, str(REAL_FIXTURES))`
			`self.assertEqual(result.returncode, 0, msg=result.stdout + result.stderr)`
			`self.assertIn("OK", result.stdout)`
			`self.assertNotIn("FAIL", result.stdout)`
fix(task13-17): simplify review — in-process fixture validation + CI dedup + concurrency Reviewers flagged 4 actionable items across the last 3 commits. Must-fix (merged here): - validate_compliance_fixtures.py: replace subprocess+tempfile roundtrip with direct `from check_compliance_report import validate`. Drops ~20 lines + ~100ms per fixture (process startup). The old design dumped YAML→JSON to a temp file just to re-parse via the CLI; the validator exposes a clean `validate(report: dict) -> list[str]` API since Task 3. - spec-consistency.yml: drop the standalone "Validate compliance_report fixtures against Schema 12" step. The unit test suite (test_real_fixtures_all_pass) already validates all real fixtures, so running both was double-work. Rename the remaining step to make the coverage explicit. - Both workflows: add `concurrency: cancel-in-progress: true` so rapid pushes to a branch don't queue duplicate runs. - test_validate_compliance_fixtures.py: tighten test_real_fixtures_all_pass with assertNotIn("no fixture", stderr) so an accidentally-empty fixture dir surfaces as a clear diagnostic instead of a false "all OK". Deferred (reviewer-flagged but not urgent): - check_spec_consistency.py v3.3.x guards stacking linearly per release (pre-smell before v3.5). - .claude/CLAUDE.md Skills Overview table has no CI guard against drift from SKILL.md frontmatter (real DRY gap, out of scope here). 29/29 unit tests green; spec_consistency passes. 2026-04-20 17:51:55 +08:00			`self.assertNotIn("no fixture", result.stderr.lower())`
ci: wire compliance validator + freshness check into workflows Plan Task 13 asked for all three steps (fixture loop, validator unit tests, freshness check) to land in spec-consistency.yml as a `for` loop with embedded python heredoc. Two deviations: 1. Replace the YAML→JSON inline shell heredoc with a helper script scripts/validate_compliance_fixtures.py (+ 3 unit tests). The heredoc approach is fragile under shell escaping and breaks if a fixture filename contains spaces; the helper converts each YAML fixture to a temp JSON file and delegates to check_compliance_report.py. 2. Move the freshness check to a separate workflow (.github/workflows/freshness-check.yml) with a weekly cron (Monday 09:00 UTC) + path-filtered push trigger + workflow_dispatch. Per efficiency review: a 180-day staleness threshold does not benefit from running on every push; weekly cron is the right cadence and push triggers only when the protocol file itself changes. spec-consistency.yml now runs the compliance validator on every push/PR (fixture loop + unit tests covering 21 + 3 = 24 tests). Freshness runs weekly or when prisma_trAIce_protocol.md changes. All 29 unit tests green (compliance 21 + freshness 5 + fixture validator 3). 2026-04-20 17:39:44 +08:00
			`def test_empty_dir_fails(self) -> None:`
			`with TemporaryDirectory() as tmp:`
			`result = run_script(SCRIPT, tmp)`
			`self.assertEqual(result.returncode, 1)`
			`self.assertIn("no fixture", result.stderr.lower())`

			`def test_invalid_fixture_fails(self) -> None:`
			`with TemporaryDirectory() as tmp:`
			`bad = Path(tmp) / "fixture_invalid.yaml"`
			`bad.write_text("mode: not_a_valid_mode\n", encoding="utf-8")`
			`result = run_script(SCRIPT, tmp)`
			`self.assertEqual(result.returncode, 1)`
			`self.assertIn("FAIL", result.stdout)`


			`if __name__ == "__main__":`
			`unittest.main()`