A self-contained harness that measures, per quality-engineering task, the
cheapest tier that clears the quality bar against api.cognitum.one — so tier
routing (e.g. the qe-court model table) is set by evidence, not guessed.
Structure (scripts/cognitum-qe-bench/):
- 7 QE tasks spanning the oracle-provenance ladder: test-generation &
mutation-adequacy (execution), security-triage / pr-severity / flaky-diagnosis
/ adversarial-review (labeled), coverage-gap (structural).
- test-generation is graded by MUTATION SCORE over a diverse mutant set with
per-test validity, truncation-repair, and emission-vs-quality separation —
operators grounded in .claude/skills/mutation-testing/references.
- Three policies per task: P1 always-cheap, P2 always-frontier, P3 cheap+escalate
(high tier only as an advisor on low cheap-confidence).
- Cost is ground truth from each x_cognitum.price_usd receipt; each row carries
resolvedModel + requestId (reconcilable 1:1 against usage_ledger).
- Keys come from local config (env / .env), never code; two-key split (worker
low/mid, judge high) is enforced by scope. mint-bench-keys.mjs is a separate
one-time owner tool (safe two-part write: metered subscription + scope-split
key). README documents setup for team members.
- Safety: dry-run default, --max-cost pre-flight abort, mid-run cost stop,
AQE_MAX_BUDGET_USD backstop. Emits ruflo-compatible cognitum-qe.v1 records.
First measured finding: on these QE tasks the cheap tier (glm-5.2) matches or
beats the frontier tier at 3-4x less cost; on test-generation the frontier tier
failed to emit runnable code (emit 0%) while cheap scored a full mutation score.
Co-Authored-By: Ruflo & AQE
Dragan Spiridonov
e841d1b094