mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
df183d1ea4
Addresses the CRITICAL/HIGH/MED findings from the brutal-honesty review of the learning-integrity cluster. The theme: the machinery was real but several claims outran the evidence. This makes the claims honest and the evidence falsifiable. CRIT-1 / HIGH-3 (flywheel gated vapor / toy anchor can't move): - verification/anchors/qe-anchor-v2.json — 5 oracle-verified DISCRIMINATING items (13 mutants), each with a trap edge a happy-path test misses, so the anchor CAN actually move. v1's items were too easy to discriminate a retrieval effect. - scripts/coupling-experiment.ts — falsifiable NONE/IRRELEVANT/RELEVANT controlled test (N reps, temp>0, $0 mutation-oracle grading). Live run remains blocked on model availability; the harness refuses to fabricate a verdict. CRIT-2 (DoE "stats" were a range proxy): - scripts/doe-run.ts — N real replicates of a FIXED item at temp=0.6 (was 5 different items x1 at temp=0), per-replicate seed, plus a deterministic $0 --stub. - scripts/doe-aggregate.mjs — real main-effects factorial ANOVA: within-cell error term, F-test via dependency-free regularized incomplete beta; beads verdict now gates on p<0.05, not an eyeballed mean delta. - tests/unit/scripts/doe-aggregate.test.ts — F-CDF + incomplete-beta vs closed forms/R, planted-effect recovery (p<0.001), null factor (p>0.5), unbalanced cells. HIGH-4 / MED-6 (over-claimed gate guarantees): - ADR-120 gains a "Threat model and limits (honest)" section: gateReExecutes catches a lying log, not a buggy gate (same-process seal+rule). - ADR-118 status reframed to "infrastructure awaiting a validated signal." - tests/unit/validation/rule-consistency.test.ts — property test scoping what the gate actually guarantees (determinism, stable fingerprints, flipped-verdict). - DOE-HARNESS-SCREEN-DESIGN.md — unequal-n note. HIGH-5 / MED-7 (self-fulfilling test / brittle scripts): - tests/unit/learning/mmr-scaling.test.ts — FALSIFIABLE regression for the real O(n^4) scoreHeldOut hang the MMR_POOL_SIZE cap fixed (96s->420ms at n=300). - coupled-anchor.test.ts de-faked: relabeled as a MECHANISM/wiring test, documents that the scripted generator hardcodes the outcome and the real model was an honest null — no longer readable as evidence that coupling helps. - tests/unit/validation/sealed-hash-roundtrip.test.ts — regression for the canonicalJson undefined-key seal bug. Verification: tsc 0 errors; 33/33 new remediation tests + 290/290 full validation+flywheel cluster green. No live model run (Ollama offline + Anthropic account at usage limit) — coupling live-validation surfaced as the one honest open item, not faked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>