mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
be28f15f9c
Live runner executed for real: OpenAI gpt-4o-mini under test, Gemini 2.5-flash as the non-same-family judge, hidden-test ground truth, seeded X/Y counterbalance, pre-registered RUBRIC-v1 hash embedded per row. The real run surfaced and fixed two harness bugs: 1. Ground-truth fix-rate was coupled to judge-row validity — a fixed bug that zeroed a real B-arm fix when its judge pass returned no winner. Ground truth is now tracked per (scenario, arm), independent of the judge; numbers are internally consistent. 2. Judge row model: emit ONE comparison row per (scenario, pass) — the unit the sign test clusters — instead of duplicated per-arm rows. Plus judge-verdict parse hardened (last-verdict wins, broader regex, more tokens): unparseable rate 50% -> 25%, the rest excluded in code. Result at n=2: ground truth A=0.50 B=0.00, judge clustered 1-1 p=1.0 — no arm signal, honestly underpowered. Corpus scaling is the path to a claim (stated in results README + ADR). Verified: 30 deterministic tests (stats + safety) green; real run $0.003. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>