skills/imbad0202__academic-research-skills
1
0
Fork 0
mirror of https://github.com/Imbad0202/academic-research-skills.git synced 2026-09-14 13:51:17 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#787)

Browse Source 
Probe set: 30 refs (10 easy DOI-keyed journal articles; 10 hard: 3 arXiv,
2 DOI-less NeurIPS, 5 non-English; 10 fabrications), every real row
resolver-confirmed same-day, every fabrication negative-checked. 180
same-day paired calls (30 x 3 repeats x 2 models), majority verdicts.

Result: all five measures PASS with superiority — recall 1.00 vs 0.80,
grounded completion 0.933 vs 0.900, p95 latency 26.5s vs 58.7s, zero
guard misfires, false disagreement 0.00 = 0.00. Transport-qualified:
gpt-5.6-sol stays provisional on the first-party API route (jq guards
unexercised; allowlist unchanged). Report + probe-set sha256 under
audits/; per-call index committed beside the probe set.

Campaign side-product (transport): page-open webSearch items
(action.type != "search") are skipped for binding instead of failing the
stream (opened-page URLs still can never become bound sources), and
DEVELOPER_INSTRUCTIONS requires an empty sources array for
NOT_FOUND/NOT_SEARCHED. 52 transport tests green. Defective-tool run 1
archived unscored; three probe-row transcription errors were flagged
MISMATCH by both models, independently re-verified, corrected, re-run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
This commit is contained in:
Imbad0202
2026-08-19 21:15:33 +08:00
parent 5714f3a3eb
commit 3fc6ddb910
9 changed files with 579 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+4
View File
@@ -8,6 +8,10 @@ All notable changes to this project will be documented in this file.
- **Codex subscription transport repaired against three live codex-cli 0.147.0 drifts (#785).** The contained citation adapter had become unusable end-to-end: (1) the ChatGPT-subscription attestation is emitted on stderr in non-TTY invocation, but detection read stdout only — every detect returned `AUTH_NOT_CHATGPT_SUBSCRIPTION`; detection now accepts the exact line on either stream (the #684 harness idiom), with a stderr-emitting fake-codex regression test. (2) The provider's structured-output validator now rejects `uniqueItems` (`invalid_json_schema`, HTTP 400) — the keyword is dropped from the provider-sent `MODEL_OUTPUT_SCHEMA`; duplicate-source refusal was already enforced fail-closed in the local validator. (3) `--disable code_mode_host` silently removed the standalone web-search tool (the tool executes through the code-mode host on this build; isolated by live bisection of the 22-entry disable list) so every call failed closed as `MODEL_RETURNED_NOT_SEARCHED`; the host is no longer disabled while `code_mode` itself stays disabled and the forbidden-event scan still fails the receipt on any item type outside the four-type allowlist. Live `cross_model_smoke_test_codex.sh`: PASS for both `gpt-5.5` and `gpt-5.6-sol` (2026-08-19).
### Added
- **First Promotion Bakeoff run: `gpt-5.6-sol` validated for the ChatGPT-subscription citation transport (#787).** The § Promotion Bakeoff procedure ran end-to-end over the #630 contained codex transport (entry gate: live codex smoke PASS for baseline and candidate): a 30-reference versioned probe set (`evals/bakeoff/2026-08-19-gpt-5-6-sol-codex/`, every real row resolver-confirmed same-day, every fabrication negative-checked), 30 × 3 repeats × 2 models = 180 same-day paired calls, majority-verdict scoring. All five non-inferiority measures passed with a stated superiority case: fabrication recall 1.00 vs 0.80, grounded completion 0.933 vs 0.900, p95 latency 26.5 s vs 58.7 s, zero receipt-guard misfires, false-disagreement 0.00 = 0.00. Run report with probe-set sha256: `audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md`. The result is transport-qualified: `gpt-5.6-sol` remains **provisional on the first-party API route** (its jq grounding guards were not exercised), and the API-route id-status allowlist is unchanged; the recommendation-policy note, Supported Models row, bakeoff section, and the codex-subscription setup examples record the split explicitly. Campaign side-product: two further receipt-grammar drifts fixed in the transport (page-open `webSearch` items — `action.type != "search"` — are skipped for binding instead of failing the stream, an opened page's URL still can never become a bound source; `DEVELOPER_INSTRUCTIONS` now requires an empty `sources` array for NOT_FOUND/NOT_SEARCHED), with a page-open regression test (52 transport tests). A defective-tool first fleet was archived unscored; the probe-set ground truth itself was corrected mid-campaign after both models correctly flagged three operator transcription errors — recorded in the run report.
### Changed
- **Cross-model recommendation surfaces follow generation currency (#783).** The recommended-verifier surfaces (`shared/cross_model_verification.md` Supported Models table + recommendation sentence + Option A example, `docs/SETUP.md` / `docs/SETUP.zh-TW.md` quick-setup blocks, `.claude/CLAUDE.md` live description) stop naming superseded GPT-5.5 as the default: `gpt-5.6-sol` (current OpenAI flagship) becomes the lead OpenAI example while staying **provisional** — the flip is a lifecycle decision recorded with an explicit no-measurement-claim note, the § Promotion Bakeoff remains the only route to `validated`, and `gpt-5.5` / `gpt-5.5-pro` stay in the table as validated previous-generation options (`gpt-5.5` remains the designated bakeoff baseline; no bakeoff run has been recorded for any id). Gemini 3.1 Pro stays recommended: first-party check (2026-08-19) confirms `gemini-3.1-pro-preview` is still Google's most capable Pro model. Docs-only; no env-var, id-status, or transport change.
audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md
+58
View File
@@ -0,0 +1,58 @@
# Promotion Bakeoff run — `gpt-5.6-sol` over the ChatGPT-subscription codex transport
**Date:** 2026-08-19 (all calls same-day, paired) · **Issue:** #787 · **Operator:** maintainer session
**Procedure:** `shared/cross_model_verification.md` § Promotion Bakeoff, executed over the #630 contained citation transport (codex-cli 0.147.0, `ARS_CROSS_MODEL_TRANSPORT=codex`).
## Transport qualification (what this run licenses)
The canonical bakeoff was written for the first-party API route. This run substitutes the codex subscription transport end-to-end, so its result is **transport-qualified**:
- It licenses `validated` for `gpt-5.6-sol` **on the ChatGPT-subscription citation transport only**.
- The first-party API route's jq grounding guards were not exercised; `gpt-5.6-sol` **remains provisional on the API route** and the API-route allowlist in the canonical doc is deliberately unchanged.
- Measure analogues: grounding evidence = receipt `searched == true` (webSearch-event binding); measure 4 = zero fail-closed receipt-guard misfires (`EVENT_STREAM_INVALID` / `FINAL_OUTPUT_INVALID` / `TURN_NOT_COMPLETED` families, and transport errors).
## Entry gate
`scripts/cross_model_smoke_test_codex.sh` PASS for both `gpt-5.5` and `gpt-5.6-sol` (2026-08-19), after the #785/#786 transport repair (attestation stream, provider `uniqueItems` rejection, `code_mode_host` disable killing standalone search).
## Probe set
`evals/bakeoff/2026-08-19-gpt-5-6-sol-codex/probe_set.json`
**sha256 `6db7c1ffeb20d4b6819010f7c7ca79f422acfef560c14cfbaf6896c78db305c2`**
30 references: 10 easy (DOI-keyed journal articles), 10 hard (3 arXiv preprints, 2 DOI-less NeurIPS proceedings papers, 5 non-English — 3 zh-TW higher-education-QA articles, 1 de, 1 fr), 10 synthetic fabrications. Every real row was resolver-confirmed the same day (Crossref direct DOI lookup / arXiv API id lookup / live papers.nips.cc fetch); every fabricated row was negative-checked against Crossref bibliographic search.
**Ground-truth correction during the campaign:** three real rows initially carried metadata defects introduced while transcribing truncated discovery output (missing co-authors 許羿梃 / 陳瑩; a wrong author initial and a fabricated completion of a truncated German title). Both models flagged all three as MISMATCH — correctly. The rows were re-verified against full Crossref records, corrected in the fixture, and their 18 cells re-run; no other cell was touched. The episode is itself evidence for the citation-integrity design: the check caught the operator's own transcription/hallucination errors.
## Tool-defect run (run 1, archived, not scored)
The first 180-call fleet surfaced two receipt-grammar drifts against real codex-cli 0.147.0 streams, concentrated on fabricated references (absence checks legitimately open result pages):
1. Page-open `webSearch` items (`action.type == "other"`, no query string) made the whole stream `EVENT_STREAM_INVALID`.
2. `NOT_FOUND` verdicts carrying absence-evidence URLs died as `FINAL_OUTPUT_INVALID`.
Both were fixed in the transport before the scored run (page-open items skipped for binding, never stream-fatal — an opened page's URL still can never become a bound source; `DEVELOPER_INSTRUCTIONS` now instructs an empty `sources` array for NOT_FOUND/NOT_SEARCHED, contract unchanged). Run-1 artifacts are retained off-repo as defect evidence; run 1 produced no scored measures (recall was tool-suppressed to 0.2 for both models identically).
## Scored run (run 2): 30 refs × 3 repeats × 2 models = 180 calls
Per-reference verdict = majority of 3 repeats; no 1–1–1 split occurred in the scored run. Concurrency 3, one call per reference per repeat. Per-call index: `evals/bakeoff/2026-08-19-gpt-5-6-sol-codex/run2_call_index.jsonl` (verdict, searched, reason_code, latency, receipt digests per call).
| Measure | `gpt-5.5` (baseline) | `gpt-5.6-sol` (candidate) | Threshold | Result |
|---|---|---|---|---|
| 1. Grounded-search completion (per call) | 0.900 | **0.933** | ≥ base − 5 pp | PASS |
| 2. Fabrication recall (10 fabs, majority) | 0.80 | **1.00** | ≥ base − 5 pp AND ≥ 0.80 | PASS |
| 3. False disagreement (20 real, majority) | 0.00 | 0.00 | ≤ base + 5 pp | PASS |
| 4. Receipt-guard misfires (shape families) | 0 | **0** | zero (candidate) | PASS |
| 5. p95 latency | 58.7 s | **26.5 s** | ≤ 2× base | PASS |
Baseline misses on measure 2: `fab-01`, `fab-05` returned majority `NOT_SEARCHED` for `gpt-5.5` (scored as misses, conservatively). Median latency: 16.9 s (base) / 17.6 s (candidate).
## Outcome (per the two-promotion rule)
1. **All five measures pass → `gpt-5.6-sol` is `validated` for the ChatGPT-subscription citation transport.**
2. **Superiority case (stated, measured):** +20 pp fabrication recall (1.00 vs 0.80), +3.3 pp grounded completion, and p95 latency under half of baseline, with no inferiority on any measure. This licenses measured-superiority claims for this transport per § Promotion Bakeoff.
3. `gpt-5.6-sol` **stays provisional on the first-party API route** — an API-route bakeoff (same probe set reusable by hash) is the remaining step to full validation; tracked as follow-up in #787.
## Reproduction
Runner and scorer are deterministic consumers of the probe set + receipts; the live calls consume ChatGPT-subscription capacity and web-search results vary day to day — the 3-repeat majority and same-day pairing are the fairness mechanism, per the canonical procedure.
docs/SETUP.md
+4 -4
View File
@@ -225,10 +225,10 @@ Devil's Advocate, Reviewer 2, calibration, re-review, or checkpoint judgments.
```bash
# Citation-integrity calls only. General DA/reviewer/judgment calls remain on API transport.
export ARS_CROSS_MODEL_TRANSPORT="codex"
# Deliberate transport-specific exception to the generation-currency recommendation:
# this closed citation adapter's example stays on validated gpt-5.5. Set another id
# only after scripts/cross_model_smoke_test_codex.sh passes against your subscription.
export ARS_CROSS_MODEL="gpt-5.5"
# gpt-5.6-sol is validated for THIS transport (2026-08-19 codex-transport bakeoff,
# superiority on recall + latency — audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md).
# gpt-5.5 remains the validated bakeoff baseline alternative.
export ARS_CROSS_MODEL="gpt-5.6-sol"
python3 scripts/cross_model_codex_transport.py detect
# The producer sends one closed codex_citation_request/1.0 object on stdin:
docs/SETUP.zh-TW.md
+4 -4
View File
@@ -221,10 +221,10 @@ OpenAI API key 而改走該訂閱。這不涵蓋魔鬼代言人、Reviewer 2、
```bash
# Citation-integrity calls only. General DA/reviewer/judgment calls remain on API transport.
export ARS_CROSS_MODEL_TRANSPORT="codex"
# Deliberate transport-specific exception to the generation-currency recommendation:
# this closed citation adapter's example stays on validated gpt-5.5. Set another id
# only after scripts/cross_model_smoke_test_codex.sh passes against your subscription.
export ARS_CROSS_MODEL="gpt-5.5"
# gpt-5.6-sol is validated for THIS transport (2026-08-19 codex-transport bakeoff,
# superiority on recall + latency — audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md).
# gpt-5.5 remains the validated bakeoff baseline alternative.
export ARS_CROSS_MODEL="gpt-5.6-sol"
python3 scripts/cross_model_codex_transport.py detect
# The producer sends one closed codex_citation_request/1.0 object on stdin:
evals/bakeoff/2026-08-19-gpt-5-6-sol-codex/probe_set.json
+249
View File
@@ -0,0 +1,249 @@
{
"schema_version": "ars-bakeoff-probe-set/1.0",
"created": "2026-08-19",
"issue": 787,
"purpose": "Promotion Bakeoff probe set (shared/cross_model_verification.md § Promotion Bakeoff): 30 references — 20 real (10 easy DOI-keyed journal articles; 10 hard: arXiv preprints, DOI-less conference papers, non-English articles) + 10 synthetic plausible fabrications. Every real row was resolver-confirmed on 2026-08-19 (Crossref direct DOI lookup, arXiv API id lookup, or live papers.nips.cc fetch); every fabricated row was negative-checked against Crossref bibliographic search on the same day.",
"labels": {"real": "the work exists; ground_truth records the confirmed identifier", "fabricated": "synthetic; no such work exists (negative-checked)"},
"references": [
{
"id": "real-easy-01",
"label": "real",
"difficulty": "easy",
"reference_text": "Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32.",
"citation_context": "Cited as the origin of the random forest ensemble method.",
"ground_truth": {"doi": "10.1023/a:1010933404324", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-02",
"label": "real",
"difficulty": "easy",
"reference_text": "Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350.",
"citation_context": "Cited for common misinterpretations of significance testing.",
"ground_truth": {"doi": "10.1007/s10654-016-0149-3", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-03",
"label": "real",
"difficulty": "easy",
"reference_text": "Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71.",
"citation_context": "Cited as the reporting guideline followed by the systematic review.",
"ground_truth": {"doi": "10.1136/bmj.n71", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-04",
"label": "real",
"difficulty": "easy",
"reference_text": "Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.",
"citation_context": "Cited as the large-scale replication project in psychology.",
"ground_truth": {"doi": "10.1126/science.aac4716", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-05",
"label": "real",
"difficulty": "easy",
"reference_text": "Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.",
"citation_context": "Cited for the argument that many published findings do not replicate.",
"ground_truth": {"doi": "10.1371/journal.pmed.0020124", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-06",
"label": "real",
"difficulty": "easy",
"reference_text": "Christensen, S. H., & Erno-Kjolhede, E. (2011). Academic drift in Danish professional engineering education. Myth or reality? Opportunity or threat? European Journal of Engineering Education, 36(3), 285-299.",
"citation_context": "Cited as evidence on academic drift in professional higher education.",
"ground_truth": {"doi": "10.1080/03043797.2011.585225", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-07",
"label": "real",
"difficulty": "easy",
"reference_text": "Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.",
"citation_context": "Cited as the source of the LSTM architecture.",
"ground_truth": {"doi": "10.1162/neco.1997.9.8.1735", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-08",
"label": "real",
"difficulty": "easy",
"reference_text": "Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018). The Moral Machine experiment. Nature, 563(7729), 59-64.",
"citation_context": "Cited for cross-cultural preferences in machine ethics dilemmas.",
"ground_truth": {"doi": "10.1038/s41586-018-0637-6", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-09",
"label": "real",
"difficulty": "easy",
"reference_text": "Harvey, L., & Green, D. (1993). Defining quality. Assessment & Evaluation in Higher Education, 18(1), 9-34.",
"citation_context": "Cited as the classic taxonomy of quality conceptions in higher education.",
"ground_truth": {"doi": "10.1080/0260293930180102", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-easy-10",
"label": "real",
"difficulty": "easy",
"reference_text": "Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289-310.",
"citation_context": "Cited for the distinction between explanatory and predictive modeling.",
"ground_truth": {"doi": "10.1214/10-sts330", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-hard-01",
"label": "real",
"difficulty": "hard",
"reference_text": "Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations (ICLR 2015). arXiv:1412.6980.",
"citation_context": "Cited as the optimizer used for model training.",
"ground_truth": {"arxiv": "1412.6980", "verified_via": "arxiv-api-2026-08-19"}
},
{
"id": "real-hard-02",
"label": "real",
"difficulty": "hard",
"reference_text": "Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.14165.",
"citation_context": "Cited as the GPT-3 paper establishing in-context learning.",
"ground_truth": {"arxiv": "2005.14165", "verified_via": "arxiv-api-2026-08-19"}
},
{
"id": "real-hard-03",
"label": "real",
"difficulty": "hard",
"reference_text": "Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019. arXiv:1810.04805.",
"citation_context": "Cited as the source of the BERT pre-training objective.",
"ground_truth": {"arxiv": "1810.04805", "verified_via": "arxiv-api-2026-08-19"}
},
{
"id": "real-hard-04",
"label": "real",
"difficulty": "hard",
"reference_text": "Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 (NIPS 2012).",
"citation_context": "Cited as the AlexNet result on ImageNet; the NIPS 2012 proceedings version carries no DOI.",
"ground_truth": {"url": "https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html", "verified_via": "papers.nips.cc-fetch-2026-08-19"}
},
{
"id": "real-hard-05",
"label": "real",
"difficulty": "hard",
"reference_text": "Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26 (NIPS 2013).",
"citation_context": "Cited as the word2vec negative-sampling paper; the NIPS 2013 proceedings version carries no DOI.",
"ground_truth": {"url": "https://papers.nips.cc/paper_files/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html", "verified_via": "papers.nips.cc-fetch-2026-08-19"}
},
{
"id": "real-hard-06",
"label": "real",
"difficulty": "hard",
"reference_text": "周華琪、許羿梃（2023）。建構跨境高等教育之品質保證合作模式：臺灣經驗。教育研究月刊，353，4-19。",
"citation_context": "引用於跨境高等教育品保合作模式之臺灣個案討論。",
"ground_truth": {"doi": "10.53106/168063602023090353001", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-hard-07",
"label": "real",
"difficulty": "hard",
"reference_text": "葉玉賢、陳怡如（2023）。印尼伊斯蘭高等教育品質保證與評鑑制度之探討：衝擊與困境。比較教育，95。",
"citation_context": "引用於亞洲高等教育評鑑制度比較之討論。",
"ground_truth": {"doi": "10.53106/160957582023110095003", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-hard-08",
"label": "real",
"difficulty": "hard",
"reference_text": "侯永琪、林芳伃、蘇泓誠、陳瑩（2023）。高等教育微型證書的品質保證及資歷認可初探：全球趨勢與地方實踐。教育研究月刊，353，120-137。",
"citation_context": "引用於微型證書品保與資歷認可趨勢之討論。",
"ground_truth": {"doi": "10.53106/168063602023090353008", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-hard-09",
"label": "real",
"difficulty": "hard",
"reference_text": "Ulrich, M., & Hachfeld, A. (2024). Familien-Schul-Kooperation zu Beginn der Sekundarstufe I: zum Aufgabenverständnis von Müttern und zur Bedeutung ihrer Passungswahrnehmung. Zeitschrift für Erziehungswissenschaft, 27, 803-828.",
"citation_context": "Zitiert für die Aufgabenteilung zwischen Eltern und Lehrkräften in der Sekundarstufe.",
"ground_truth": {"doi": "10.1007/s11618-024-01230-x", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "real-hard-10",
"label": "real",
"difficulty": "hard",
"reference_text": "Bireaud, A. (1979). Attitudes d'enseignants à l'égard d'une innovation portant sur l'autonomie. Revue française de pédagogie.",
"citation_context": "Cité pour les attitudes enseignantes face à une innovation pédagogique.",
"ground_truth": {"doi": "10.3406/rfp.1979.1695", "verified_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-01",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Tanaka, K., & Bergström, L. (2021). Hierarchical consensus networks for multi-institutional peer review. Journal of Machine Learning Research, 22(148), 1-37.",
"citation_context": "Cited as evidence that consensus architectures improve review reliability.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-02",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Okonkwo, C., & Falkenberg, M. (2019). Quality assurance regimes and the myth of convergence: A six-country panel study. Higher Education Policy, 32(4), 511-534.",
"citation_context": "Cited for cross-national divergence of QA regimes.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-03",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Verhoeven, P., Salim, A., & Grunwald, T. (2023). Attention decay in long-context transformers: An empirical atlas. Advances in Neural Information Processing Systems 36 (NeurIPS 2023).",
"citation_context": "Cited for degradation of attention over long contexts.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-04",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Lim, S. H., & Watanabe, R. (2020). The Shanghai Protocol: Standardizing cross-border credit recognition in East Asian higher education. Comparative Education Review, 64(3), 401-425.",
"citation_context": "Cited as the origin of a regional credit-recognition standard.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-05",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Almeida, R. (2022). Reverse distillation of citation graphs for retraction prediction. Scientometrics, 127(9), 5231-5256.",
"citation_context": "Cited for graph-based prediction of retractions.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-06",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Kaufmann, D., & Öztürk, E. (2018). Cognitive cartography of supervisory feedback loops in doctoral committees. Studies in Higher Education, 43(11), 1987-2003.",
"citation_context": "Cited for feedback dynamics in doctoral supervision.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-07",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Petrov, I., & Nakagawa, H. (2021). GraphMend: Self-healing knowledge graphs for scholarly databases. Proceedings of the VLDB Endowment, 14(12), 3020-3033.",
"citation_context": "Cited for automated repair of bibliographic knowledge graphs.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-08",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "林建甫、張維倫（2021）。外部品質保證與大學自主性的再平衡：東亞四地比較。教育政策論壇，24(2)，1-32。",
"citation_context": "引用於外部品保與大學自主之張力討論。",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-09",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Delgado-Ruiz, M., & Chen, Y. (2020). Stochastic curriculum alignment for assurance of learning in business schools. Academy of Management Learning & Education, 19(4), 488-509.",
"citation_context": "Cited for probabilistic curriculum-alignment methods in AoL.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
},
{
"id": "fab-10",
"label": "fabricated",
"difficulty": "n/a",
"reference_text": "Brandmüller, S. (2017). Die unsichtbare Hochschule: Qualitätskultur als organisationale Fiktion. Zeitschrift für Hochschulentwicklung, 12(3), 45-63.",
"citation_context": "Zitiert für Qualitätskultur als organisationale Fiktion.",
"ground_truth": {"negative_checked_via": "crossref-api-2026-08-19"}
}
]
}
evals/bakeoff/2026-08-19-gpt-5-6-sol-codex/run2_call_index.jsonl
+180
View File
@@ -0,0 +1,180 @@
{"model": "gpt-5.5", "ref_id": "fab-01", "repeat": 1, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 25.89, "receipt_request_digest": "ec6b1c52b12706d5fdf4688f15ec330faf8dfd6399aea6c38d3e7a44d750fb59", "event_stream_digest": "96b7c3b67456ca0b0fa359a0708692ced0c495e02b433cc188cb580cc49a5b9a"}
{"model": "gpt-5.5", "ref_id": "fab-01", "repeat": 2, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 22.64, "receipt_request_digest": "1fa604230fba8682786dd4421ccd79169191127cbc0592931ac6ed432084142c", "event_stream_digest": "6a6baa7bcecd2a32c4a1e4f3cf960aab01eb3a8592e9cc7fffe576311f0c9aaf"}
{"model": "gpt-5.5", "ref_id": "fab-01", "repeat": 3, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 19.34, "receipt_request_digest": "4afc8179ef0621e94f6a552931fcacdedab926af082ea24891b08cd22d38fbc4", "event_stream_digest": "b1a3dac0d5003588ad85ab75ea80cf3b51ca465b51dc63657f2da76593a1ee6f"}
{"model": "gpt-5.5", "ref_id": "fab-02", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 71.95, "receipt_request_digest": "4b45ebab5fab8a2465469e3ee00b3b174f4237556ef2112a4f80e7e35a8eff0b", "event_stream_digest": "7e8fe8ff96312667aa83537f0c8a26b403fe7939687b689884d634d7c94c9b6a"}
{"model": "gpt-5.5", "ref_id": "fab-02", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 49.31, "receipt_request_digest": "1b4168d8ed7c9c88e27c4475cd75db73667b346bdaae9ed155f5ad03cec1dd31", "event_stream_digest": "6c61605017727042937b2d6b45255c38922ac99e1e1042ec2c64a4b94097c9ae"}
{"model": "gpt-5.5", "ref_id": "fab-02", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 57.66, "receipt_request_digest": "0dc06ef057da809b9091cd22b13686d3502ccb311c48c2b84007967f6e7caa40", "event_stream_digest": "ec6ba44c0d9e5a2559fc92b2ae5794b906e3fa800b2b389b74bb581462aed42c"}
{"model": "gpt-5.5", "ref_id": "fab-03", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 34.04, "receipt_request_digest": "cb916116e6bedd2fdb2b15fd421508317d860dd86997c049e2cfd124ea74d8c0", "event_stream_digest": "7c10a8fb597a5ac87cef94d7374d693a4b5fc20e8bad79df20024e9a34347460"}
{"model": "gpt-5.5", "ref_id": "fab-03", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 31.94, "receipt_request_digest": "4389a54217e2609f2e7ceec931d96545e985f598e5a67185f486555aa5916531", "event_stream_digest": "118f4f7b8922086d176f33e0529fb36bb2ee2b0d6af938dec83042bfae78032f"}
{"model": "gpt-5.5", "ref_id": "fab-03", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 26.92, "receipt_request_digest": "2bcebe53ed4af738b0010c43841eeae2539bccfd4641407f9543c5bbc9ba8027", "event_stream_digest": "eaec5822ad74162821171059b6286200f9fded8b9a44d033b123b22f54e25b7d"}
{"model": "gpt-5.5", "ref_id": "fab-04", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 28.44, "receipt_request_digest": "c854a2622e29009165500d6aa5f0c313111d0d2003c5bb86d296c5e77e7694eb", "event_stream_digest": "a09855b9e5627b9ca2072493a72fdaeecb7b980a66e9814223b7b1c7dbad9e83"}
{"model": "gpt-5.5", "ref_id": "fab-04", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 30.44, "receipt_request_digest": "60c19ccc0a83f38d4db160aef9bd338ece51eb10779745649e3007bedbea86ee", "event_stream_digest": "40d0ed7e4b9eb290ac314bbd371867983baa806a57a65144eda236921a3edd1f"}
{"model": "gpt-5.5", "ref_id": "fab-04", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 21.14, "receipt_request_digest": "59c5f73739d588ec03f5dc7588b9bb244ab6ebd3dc5e6684d4b6dd84c17e32ca", "event_stream_digest": "f285fad00562ed53b2b1e53861996b1d7b78d06d331ed654a00497789666430d"}
{"model": "gpt-5.5", "ref_id": "fab-05", "repeat": 1, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 42.07, "receipt_request_digest": "1d0e22a1419029687e483ac32e9c973b22b50a6dba146aba686dd40adfa62a85", "event_stream_digest": "05c3a50dc401adff073dcb62ce84fc6c39f1c855e3107d41ef92e62bb5371804"}
{"model": "gpt-5.5", "ref_id": "fab-05", "repeat": 2, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 52.63, "receipt_request_digest": "a65e32b2e6fe1e0b45bc580ec6f8ea031e5fdb0e25a818a458ae52e4c3a3725c", "event_stream_digest": "77f26ca817dbba89f5e8c270e16b13dc1d337dd9eb2d678375707036f59d29a0"}
{"model": "gpt-5.5", "ref_id": "fab-05", "repeat": 3, "verdict": "MISMATCH", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 30.14, "receipt_request_digest": "32af64575b29f42f9f1d07e5e66f8d4b5d1389e72fcde49a5df10ebae58f149a", "event_stream_digest": "3fe6b0d2bf18cf3dde4f1e01baf1158961acf673133f1733aaaa503f2cc38dfc"}
{"model": "gpt-5.5", "ref_id": "fab-06", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 77.54, "receipt_request_digest": "86874138ea8071303e069eabfd8fe1f7b5459f6716e913307998fb80efbf35fe", "event_stream_digest": "fb2c9fb1239975e967bccab7c5603774bd9b544284e4b812486746ec37c254d7"}
{"model": "gpt-5.5", "ref_id": "fab-06", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 45.33, "receipt_request_digest": "5ebcfd2a629084730ddea3c308abd7da08aa5f811aea740113ff133089249ac3", "event_stream_digest": "15d3ab2bf89e78625c04ee85d7a3314763073323a673521fdf4dfb36495b6b23"}
{"model": "gpt-5.5", "ref_id": "fab-06", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 43.13, "receipt_request_digest": "c5eaab4fad120d0d3c7cb62b00a07b5d9b2b8c80a042fb171175d1c22ccc9b1f", "event_stream_digest": "f45cae4bcedeec35a9906a1150a056bbae16978556c890ce2b59f0c04b6c8527"}
{"model": "gpt-5.5", "ref_id": "fab-07", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 52.96, "receipt_request_digest": "a3889ff6cb3e2b12d36b524c2652ac65a34edc6e4f119482400efee1ab336f7d", "event_stream_digest": "bb975e074c958b6595264ecad6b11b8602774965190ab46b7352ae876acf17f5"}
{"model": "gpt-5.5", "ref_id": "fab-07", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 36.9, "receipt_request_digest": "08fd3d224933cfc199a2d62ee6b4f0caaf3509894c355378a912dfaa6dba61e8", "event_stream_digest": "aff30df880c88a96036ceccd55d7eed49dee0a20006b0ecdc51b432a049e8f40"}
{"model": "gpt-5.5", "ref_id": "fab-07", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 64.22, "receipt_request_digest": "88a5502d720e6c6e50c47e6b0e3361c2ee99b4050f10fd7cfb64b8ff63954bca", "event_stream_digest": "532aea56220ec4266c6832f2709866dcbf8669abbd1ea2d32dd477a9f72a93fc"}
{"model": "gpt-5.5", "ref_id": "fab-08", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 58.69, "receipt_request_digest": "789b8107b3f5ac239fac112dbdab81114820e241c22e25bcf2fa3adaa49872b8", "event_stream_digest": "d6a0131d1ba24c37e6050d196ce2decffaf22614924c3998f55a6e80c9df63c9"}
{"model": "gpt-5.5", "ref_id": "fab-08", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 41.57, "receipt_request_digest": "505a7740b5d59f5277e50215b7365058e37c3af12b7aa18f878ad3d75131d7a4", "event_stream_digest": "99144ba81f80cc4630984be0b534393f947832ae9646bad8d097f7c5c5e7c3ea"}
{"model": "gpt-5.5", "ref_id": "fab-08", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 73.93, "receipt_request_digest": "1db87d8114f45c0e52d8f447376163e0a6d66d2240f23bbbf2e5eccb98c886b9", "event_stream_digest": "d1ae561aa558e147703b221bf38cc65911586d0942777ee28daf6beb6895210b"}
{"model": "gpt-5.5", "ref_id": "fab-09", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 40.27, "receipt_request_digest": "464ed3eaca76878d4c3e023e2de196df4da3b0ea200f15cc1a6b33050d8865a4", "event_stream_digest": "30da6b7690aa6a8f4bc421a2c15e8b3128f669863ec946370f40eb8e96b14c98"}
{"model": "gpt-5.5", "ref_id": "fab-09", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 35.27, "receipt_request_digest": "74c8cdb67397c24c220edaddc4771fe2431b98e4a9258f1294da503da6f7b8b2", "event_stream_digest": "4c6fa21377d7fcb2d212c141613c97f9841a2dd872644b19ba3b2ce0d278b9b7"}
{"model": "gpt-5.5", "ref_id": "fab-09", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 43.88, "receipt_request_digest": "efbfda2ce3aa0af0e02515920afcc09d6b864e3d3123f4176304974ec3d81ba4", "event_stream_digest": "addb516bc58d85a4b4ac9932573601b302f8b21511ad3045e036786e8d6dc647"}
{"model": "gpt-5.5", "ref_id": "fab-10", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 79.05, "receipt_request_digest": "7ff605fd2070d17f86c6da69710eadab3b432ed3347d6eeb9bbefd08a7cb7def", "event_stream_digest": "0595d3fd7a2c80b5f86953dc1b41e6f52449708be2640c57f32f5a6deaa33871"}
{"model": "gpt-5.5", "ref_id": "fab-10", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 54.97, "receipt_request_digest": "388b1e65e10f00c39602bac28f7f74d818790715bda56e8bd96567a3e887b7c2", "event_stream_digest": "f1f4341fefdda2f48effe953cc869e95bf1e2d35e7833bf270eb50aa80b5fba6"}
{"model": "gpt-5.5", "ref_id": "fab-10", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 56.14, "receipt_request_digest": "bdccb4a5b4ca1d4a03f8c67cbfde61010e4c3020eea955a6eeec880acbee5296", "event_stream_digest": "60f73ea0142fc3c139ab734ca23c7af80ebe9dce2b39222344ba86f5045d9c15"}
{"model": "gpt-5.5", "ref_id": "real-easy-01", "repeat": 1, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 11.76, "receipt_request_digest": "104637603e9ff6ea41f79def798c0a169363812ff82b4269abca40ce1b2afa68", "event_stream_digest": "526b6ed919f60e535a07daa733fc61ed2bef0744f9d829cb196cc34335ea9977"}
{"model": "gpt-5.5", "ref_id": "real-easy-01", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.23, "receipt_request_digest": "17ee241620825f3af29fb42c88c61080b653348c68d07d817afe375c128d7453", "event_stream_digest": "e9f36be9531627a709e8b6151ca8196bd05a4420569a10c5d27dd788e6963300"}
{"model": "gpt-5.5", "ref_id": "real-easy-01", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 11.04, "receipt_request_digest": "41141bd0e2d8495c7459255daed32efab608c80bba13151204ca98091506529b", "event_stream_digest": "8f75b42b40871e3b55ce64a7a4fa61474438471e83e6c09524555dded173d46b"}
{"model": "gpt-5.5", "ref_id": "real-easy-02", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 3, "wall_seconds": 13.9, "receipt_request_digest": "42bd71fc0222efe7599a10ef46c2df2336099e676c21ac97f52189ea21accf0b", "event_stream_digest": "ea0bcfe8f741c6ffe102d5fb8bb10773d02228b5058c45d1d99253885f333e1f"}
{"model": "gpt-5.5", "ref_id": "real-easy-02", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.95, "receipt_request_digest": "e8631cfbbd59266dcd39808199e5b8d7053ed050068b983b59855c0ebfa31728", "event_stream_digest": "9a81206289deed2151e08f2399ddfa80a304c7346fa1c35b8b1ab3ecac0c9d35"}
{"model": "gpt-5.5", "ref_id": "real-easy-02", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.9, "receipt_request_digest": "a1145859d97bdb281d5553709752cba47506a45342b65a61e212ac06beede291", "event_stream_digest": "9590ebdfd5e7aa6c58949c124a9410b1290fdf1b1bb4cff3f2eeb89e2683ca17"}
{"model": "gpt-5.5", "ref_id": "real-easy-03", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.4, "receipt_request_digest": "5e64c32b88111a1cca9f595d03bba45e052ae8da992a74cb2183a0b5dee1a2be", "event_stream_digest": "cfecda67d1205e017cc2f73bfa3cfc5308957c02807c8c78034d15718c1f1f14"}
{"model": "gpt-5.5", "ref_id": "real-easy-03", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 11.46, "receipt_request_digest": "6e398772fb4c7b3f8eebfc5fcba532203062b1ad0aa7d8fe91ce3e328ef72b42", "event_stream_digest": "ab1ac60015baa0f03e9b3cfa3f0ebdc830f0aa9b535f0a98865f058fe35b6e36"}
{"model": "gpt-5.5", "ref_id": "real-easy-03", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.87, "receipt_request_digest": "acff1454acb5fffb9e963f8d0cc5c7671b6400238f4cf8b66f0d25a34e7e8e4d", "event_stream_digest": "e725d8eacc4c398e76f202f39656ef1914801c18220cf89361b1fdc375ad823e"}
{"model": "gpt-5.5", "ref_id": "real-easy-04", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.83, "receipt_request_digest": "3464521350e98ee20fdcf05736521f3d36d85f0f2d6f986180ff389ba1f29056", "event_stream_digest": "8700fc4e75b3ea244be6a6f546377f199275b9ad2efcf5bca0fb3f77c0a663a4"}
{"model": "gpt-5.5", "ref_id": "real-easy-04", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.92, "receipt_request_digest": "0968f24ff16091e24b73aac935b73e9e7da70a43904c34cc945edb6bc4fb2a67", "event_stream_digest": "22b10e436a680810a8a674e6ec4cbfc34ddbe138762dca2fc8cc292d46851ae6"}
{"model": "gpt-5.5", "ref_id": "real-easy-04", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.59, "receipt_request_digest": "30eade65985c10cdfa9fe57b881d92a66a1f21fc3bbeeac6b071ca70d288f31a", "event_stream_digest": "6d9b91cc1d1eee5e3c7355771855bf219a12a77965f6a6bc42025b6aaa9ed4fb"}
{"model": "gpt-5.5", "ref_id": "real-easy-05", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.89, "receipt_request_digest": "91bc65163def4fd9ccd4a2a6501e961a935699293dedcafb3f465e2a25dba99a", "event_stream_digest": "27901a75b33c720bceabb2a0da1d46fda469f78ccb2043740ea62523b3865b7b"}
{"model": "gpt-5.5", "ref_id": "real-easy-05", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.81, "receipt_request_digest": "3c4188ab914877a9449411ae8ea9d053bac1b6f3aacb4bdf38f4b23e9d2665c1", "event_stream_digest": "84909ae5e47f3575ed35f7306146be0f708493b77b1991090b11adf066791ca8"}
{"model": "gpt-5.5", "ref_id": "real-easy-05", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.07, "receipt_request_digest": "4236b7ac96a1fc53ffe4e5211abfff5ef85c8595fbdb7d0a6eaf3798d8a6958e", "event_stream_digest": "baaa5d3bb6ae78af40420a7585add27051255a81c550914cb1fa1cbcf5bd7034"}
{"model": "gpt-5.5", "ref_id": "real-easy-06", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.43, "receipt_request_digest": "9286e6411c4c9a052bdb84ef677949640a234ac06dc493755aa92b0ae6ae5365", "event_stream_digest": "7683e191a77455a4d88cff267a5def7cbc3be97c71ec1db8231e6089629c196a"}
{"model": "gpt-5.5", "ref_id": "real-easy-06", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.79, "receipt_request_digest": "2aaac12193a2c6eb94260154b3b28106070c794d5319fcab84245ce564f6010c", "event_stream_digest": "7850a8e67b20bbbe0c66aab6712fc8cb747d029dd88339dd92c83967e2e6f445"}
{"model": "gpt-5.5", "ref_id": "real-easy-06", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.31, "receipt_request_digest": "c07d9edd9f9908aea89bd6f5afe155b0d527f0bdcc780880c8ea43ff2862ac47", "event_stream_digest": "eb59aafc2cb47c7b5692cc11dc6fd617d1963df74e07cff9bd4db3b8049b311b"}
{"model": "gpt-5.5", "ref_id": "real-easy-07", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.63, "receipt_request_digest": "848a9efe1612db263dd7df5576236b0fee65f317e61980b9567a11c5e67134d3", "event_stream_digest": "fecb6cfb0bf31c75c334118deec36e82bbcbc467ed12ffbbb6550fb65c72bd39"}
{"model": "gpt-5.5", "ref_id": "real-easy-07", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 11.83, "receipt_request_digest": "708a9181b1f9d26287d61787d70093e1ac41b86dcc5de37f5a68b81b2d732cf2", "event_stream_digest": "304ca1fd72729f01129c879418c207b550da1818bfe03eb5d24bbd389847cdee"}
{"model": "gpt-5.5", "ref_id": "real-easy-07", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.02, "receipt_request_digest": "5100f8328ad104c9b6dc8b49d3f8d921503b354fe75ee12e548bf20ff846366e", "event_stream_digest": "d4a445aacb079e37e366619195d87e87b765b2f9e197fa2e8cb7f29b75ca43ef"}
{"model": "gpt-5.5", "ref_id": "real-easy-08", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 11.53, "receipt_request_digest": "80bdf68ee9f3e7f8d89919808a0173a07b6d4bbeff6cf15529b7da3a069b72a2", "event_stream_digest": "b1d291b23781afc50c379497bb0d6257161f3f034e549f0dfb422cc696c773fa"}
{"model": "gpt-5.5", "ref_id": "real-easy-08", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.96, "receipt_request_digest": "74bace325f4f33de857bead92198acf4045410c9fb74dd511f7728174fe0938c", "event_stream_digest": "c87e73d2bbbbe6d305e43105e7092780de225679d8e57e55bb06a331967f9816"}
{"model": "gpt-5.5", "ref_id": "real-easy-08", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 12.01, "receipt_request_digest": "d8f8d48a0cfafaf865fb89e3ae6aa815f0a0c2323393a37be8e8857366688831", "event_stream_digest": "90a02576d3bda2080f148b1f169e3457b2c58af5298d8a466c5e170980222081"}
{"model": "gpt-5.5", "ref_id": "real-easy-09", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 11.92, "receipt_request_digest": "85749b9d236d73befeda1e7eb270cd0e8b846d389786ad10a43e82e65d7a8542", "event_stream_digest": "1ed4d01e3f456d8e5a4380617567a29ea2bb10d9bf067e02dbc1e10fdaffed8b"}
{"model": "gpt-5.5", "ref_id": "real-easy-09", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 13.9, "receipt_request_digest": "1b69910525f4973d86aee0f6662015fad62968ae1edb64767c66e2c52b0eb999", "event_stream_digest": "a3d62ae1b0bf29b9b685eb42fd5dee604667d88949e71f60e1643c200510965a"}
{"model": "gpt-5.5", "ref_id": "real-easy-09", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 16.22, "receipt_request_digest": "4989c8b27a6446383ecdf3505bba99155ab8467086949a1146f87185aa76cce5", "event_stream_digest": "e7de6a2e8e2bb01da002e15d7819ec4341aa976b7cbeae25a9543b12ae095cf9"}
{"model": "gpt-5.5", "ref_id": "real-easy-10", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 11.98, "receipt_request_digest": "16dce18f219b08300c20d8854a713af7e946c91acf48d971e222925f057c1ae6", "event_stream_digest": "b1174edb9511928dad7023ea036179c4a315094c4ba51999a504977645d08540"}
{"model": "gpt-5.5", "ref_id": "real-easy-10", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 10.86, "receipt_request_digest": "1f0975f001c638d3286bc1adc94f8b604b80902be3cd68f2aa3c0d1a3b2fc1ff", "event_stream_digest": "3779143af78a00090c915173b7b11271af2193811d9ee781a336dedca29596c8"}
{"model": "gpt-5.5", "ref_id": "real-easy-10", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 18.02, "receipt_request_digest": "8cdbd2affff0479eaf06fe3f345bcacb9c57c181a47bedfca999bc0e5c5bb8ce", "event_stream_digest": "a3d3c1fb71abbd77915a62e3e39e3f1635414b5322137dac05ac216b564946a8"}
{"model": "gpt-5.5", "ref_id": "real-hard-01", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.89, "receipt_request_digest": "f33780bb1534fe35b6188f7c5837b7b9abf471325e9c1273a650ba5b56526dcf", "event_stream_digest": "f9a07a4eaa7296f5e912abfd67d2c8f39ab305506b2cd9e5f946b151f62cc778"}
{"model": "gpt-5.5", "ref_id": "real-hard-01", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.24, "receipt_request_digest": "75fe7cd91c6d671f564d82432f2994ca01d57343245b9a0930a2058137d0076a", "event_stream_digest": "40b03abeb99cd93b33412fbe2a612f020aec9b521c3212e29cc0b7ca8c0b2718"}
{"model": "gpt-5.5", "ref_id": "real-hard-01", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.22, "receipt_request_digest": "b741646de458ccacb4ae457b8e5cde22aec75de3fdf8532a5a27123c3cc3ffd8", "event_stream_digest": "95b0b5453f7c14bd380837d4787db22771057c034989773b0ee8995ac4450656"}
{"model": "gpt-5.5", "ref_id": "real-hard-02", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 21.86, "receipt_request_digest": "9e490363a77b95f4e668b4247b698df6adee59c31fb3a8171df91770ed3704d2", "event_stream_digest": "2a045bd71ed504029438becbb73a4ecdce61442aee716d24a8195b0b97ef529f"}
{"model": "gpt-5.5", "ref_id": "real-hard-02", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.92, "receipt_request_digest": "bd425d209c7a2d93813090bc889b204cf11332649423c2b3c7d4fd7da7284891", "event_stream_digest": "20d3f82b49e6c6594f43d03759a65548c4e1c65f6c1e3d359690f52764638cd8"}
{"model": "gpt-5.5", "ref_id": "real-hard-02", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 20.34, "receipt_request_digest": "ee295951837649a570e9219c88da9135761e537af52f331d441b357f56e9c9bf", "event_stream_digest": "defcab044d6363bd11c53f5ce43e4e0308dc7f4ae8cd85f566ae69ff4b89ef15"}
{"model": "gpt-5.5", "ref_id": "real-hard-03", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.01, "receipt_request_digest": "0f7d543bfbcd0da0b5369f76bda00df1f0e4bb041cd57b4fb88d8aa0960f536c", "event_stream_digest": "f636453c9a5a83e3d64843fbde6eef5d3ba215e71fefb57f4fbc9780524cea72"}
{"model": "gpt-5.5", "ref_id": "real-hard-03", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.37, "receipt_request_digest": "ff63eeac63b186f3481bf429aec1843dca88e4bb29c244517fd805c955e91791", "event_stream_digest": "0732009b0e5e90a94d583f2e7cb592902781e30cb459ead1e5984405bf008b84"}
{"model": "gpt-5.5", "ref_id": "real-hard-03", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.01, "receipt_request_digest": "0d3df20540618f16a8a00fe75bdd8f4776817e3320ae0d413ceac768f341234d", "event_stream_digest": "1ff3e4d8afaedf27fff04e5a5b57fb52efc55feb36a642cda769bfe5f25064ea"}
{"model": "gpt-5.5", "ref_id": "real-hard-04", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 21.18, "receipt_request_digest": "c5f6ce7d1f7f76c3e4e8ec4aa305a4cf11e4701763a7edb73ce2936593109bda", "event_stream_digest": "b197dd5e5f17d131f3eba36a1e771c045a3ceacc8436d4a74637f6d1a3d2ac3b"}
{"model": "gpt-5.5", "ref_id": "real-hard-04", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 21.92, "receipt_request_digest": "cfe24fa6f8d1ebfcae2aab8e6859495144ac51e95346289ae1f1279100068711", "event_stream_digest": "5c1d3b0453fce6d350da295e26009d1c3834c003bf8f4ef623a7567553fc42a1"}
{"model": "gpt-5.5", "ref_id": "real-hard-04", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 45.75, "receipt_request_digest": "adeaf310fdd55d0ad8017a1983464e30498431e5b326da3651496032ee2876df", "event_stream_digest": "764a6d67312f60e53774790698370b1a75756b6c50f5896f00241599dca60474"}
{"model": "gpt-5.5", "ref_id": "real-hard-05", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.7, "receipt_request_digest": "9a3947eef53b1334fa77aaf96ab8bf7c231c49c1eb582a2a0bf017ad8902e51c", "event_stream_digest": "1677b67c88335e69dd84c015d71cbbc2b829144828ad7016eee01e6a53bf7760"}
{"model": "gpt-5.5", "ref_id": "real-hard-05", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.92, "receipt_request_digest": "42452c6876d0e37be567f676bf7ba78f6b0a5a8537bcc67bebd4beac8977ec8d", "event_stream_digest": "3268ee4ebfb500a6ed454f2803ea63b3f5d426e836ff37a527d617f764114dc0"}
{"model": "gpt-5.5", "ref_id": "real-hard-05", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 3, "wall_seconds": 21.82, "receipt_request_digest": "9e424fa225608c66f30b45e81286f97521f053d5444c861bde2602a5e10cca59", "event_stream_digest": "56ed2ad7c045903a5c3e501b842b83d49eb9e6420dadf863e3af8773d61055cd"}
{"model": "gpt-5.5", "ref_id": "real-hard-06", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 16.43, "receipt_request_digest": "581c40e9af758d8fe7104b14e131b1bcec0253a2aa8c45a177dae4ab0afbcfb3", "event_stream_digest": "51b1033ec435a4e8b44083879affeb08a0f5f4ea88f38b5208b0cc5787713bc0"}
{"model": "gpt-5.5", "ref_id": "real-hard-06", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 13.64, "receipt_request_digest": "faa51812ca6900c82d7519dea3aa53b0a9e6f86d1b8d9363eae55bb14d802ddf", "event_stream_digest": "a13ef2a1a851c08a6f6b58db7f3909d505a6eaff6ef2777106c2ad3b0cbfb5c1"}
{"model": "gpt-5.5", "ref_id": "real-hard-06", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 12.92, "receipt_request_digest": "66791c40cf8f5a9f23bdb7a93be4144308b7cb50b21465dc892eb4fe6875774e", "event_stream_digest": "57632e08d111dcbdd5df7f03a331ae49b8df9333e5ab7f11db4d52fecbf5853c"}
{"model": "gpt-5.5", "ref_id": "real-hard-07", "repeat": 1, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 24.58, "receipt_request_digest": "3072a3857c5abb6cd07494dc5db94e426776fd4550ebaef5820a66e35cf58467", "event_stream_digest": "1953f69232f19fc873954336e921866ae5edf421069e869c6ca147c1551bfff1"}
{"model": "gpt-5.5", "ref_id": "real-hard-07", "repeat": 2, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 22.37, "receipt_request_digest": "22e8e8c42ca85f291ace377f690d328b337b2b9f4c18567b6fc04ca1787549d6", "event_stream_digest": "402d4ace318cff189f3aee94a7ec5128a1c4a65666a71cbe12bdcbb085034e83"}
{"model": "gpt-5.5", "ref_id": "real-hard-07", "repeat": 3, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 22.72, "receipt_request_digest": "cdd1bea51fa5ba5997a60efb2d36ff2ca79b014367c4a5a8e0334d299f0f80c6", "event_stream_digest": "00bffcf0046f212a1ed1d8a84cba257efb9896b96a8fd0e39a4b8a30b693db51"}
{"model": "gpt-5.5", "ref_id": "real-hard-08", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 18.09, "receipt_request_digest": "d509674fd0a6ec150640635a64fb26309c8895e4697a346d4187c53076d0cd53", "event_stream_digest": "686c6e0ee1d2242760c8bfae5ab84cbdd464a4373744c51d7219b30a8b909b0f"}
{"model": "gpt-5.5", "ref_id": "real-hard-08", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 17.9, "receipt_request_digest": "44498e9b24c3597271a0de7cf35291437b334b07768b74078e04345749e1b895", "event_stream_digest": "19ef7bccab198a08a1b78efcd2fb95ce88b27e65d89c908d74926e35fbaa3657"}
{"model": "gpt-5.5", "ref_id": "real-hard-08", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.11, "receipt_request_digest": "976e74f879fd4ce7a6fd026be16445b48a844fa2a283ac3aaeefa0aad7309fbd", "event_stream_digest": "3c969b14a1303be18dd93bdffff3e51a9837c1d20beb88bd6c19b02dea2a00fb"}
{"model": "gpt-5.5", "ref_id": "real-hard-09", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 14.77, "receipt_request_digest": "232aefa1ca31b4b25876100d48cae4d55da43ce3eb4081badc4bb235e2794cfb", "event_stream_digest": "9693ac1ef8b04325b16a9147096f1b664ca5170c894af04b6026d20aea8a92b3"}
{"model": "gpt-5.5", "ref_id": "real-hard-09", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 12.68, "receipt_request_digest": "81abb523945740af658044dbd20e9634a9afb750d83928ee1d22d022fd90d8b6", "event_stream_digest": "be0e7a6fd97d66ff43dc91a46090424026098e64db2962297f80010e3ccdb1f9"}
{"model": "gpt-5.5", "ref_id": "real-hard-09", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 17.29, "receipt_request_digest": "9e9b0c073cf817ad343a235db1a4a610769caad5a6d01ebf8f0e46a8285b7244", "event_stream_digest": "4dde0d7a291fd697556380a8a86112680730c54f64e7ad72cd04d5b02caaf167"}
{"model": "gpt-5.5", "ref_id": "real-hard-10", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 22.53, "receipt_request_digest": "a92d2bdea23d4c5fbb5aad7b3e215d6100bf37ad4d8d969f8ad8a0c189e10560", "event_stream_digest": "48f1a38e069d574beb8883e541785d01eb734fa9886ec64818e608fae071e800"}
{"model": "gpt-5.5", "ref_id": "real-hard-10", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 13.39, "receipt_request_digest": "d6684f6bc91f1ee2c712c988664c981e45d80ba45476e65e81274367da46a628", "event_stream_digest": "cd1b8cac929ef1a23f16d8296dd3fe5d35c2aeda417fb1b62afd179d640f7018"}
{"model": "gpt-5.5", "ref_id": "real-hard-10", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 17.77, "receipt_request_digest": "a61bbb254e245b2d085b8114d6726caf01399561e9ca0b44f0cd3128037dd85e", "event_stream_digest": "58120cce7467c41462bd89680e7d7e2549ff6f67f3c90ad4b07b930ac9127c67"}
{"model": "gpt-5.6-sol", "ref_id": "fab-01", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 23.07, "receipt_request_digest": "5200cd2aad40946ab53abe8d9081b61322af06a40ae28c9a622ca437a5537510", "event_stream_digest": "4310c2b009a6acd6628955010f95b63fbb3e6114abfda9e1ec1c3435ac4f0bc5"}
{"model": "gpt-5.6-sol", "ref_id": "fab-01", "repeat": 2, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 22.71, "receipt_request_digest": "2bb3cebcbefe30618cd3235fa2867c594c9c9e1be199e447d1dea5de628195e5", "event_stream_digest": "92593df2ab7c83d1f1ddeaf9973f41bfdb4df0d241c12e6803c64e5fd6ef021d"}
{"model": "gpt-5.6-sol", "ref_id": "fab-01", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 16.1, "receipt_request_digest": "e522e3734b0961eb29bc3d52c79568245650f67093b9234620b80862d35acb18", "event_stream_digest": "20b7562db91aad5c1097caf7b1a77714f12b35c501d69b9f5b8d026494b14c93"}
{"model": "gpt-5.6-sol", "ref_id": "fab-02", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 26.48, "receipt_request_digest": "7eb7a2acd32e4536b7136331b9fb2a6ae9d44c09a05c70df094d75c936ebb5a5", "event_stream_digest": "d31a2b4256fa200964c8fb4b7708d79f14dc15adc2cd789340304ea95ae430c1"}
{"model": "gpt-5.6-sol", "ref_id": "fab-02", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 26.56, "receipt_request_digest": "495f23cdc6785b221b4b99908b1c0d07c9a6c3e8fc1589b634937016cfc2810e", "event_stream_digest": "b587dbc1167548924412e70199a91ae8acd6c5c9c96ec2abe2244429027b8ea6"}
{"model": "gpt-5.6-sol", "ref_id": "fab-02", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 25.45, "receipt_request_digest": "a1b81550dc7e8fa003afca7dbf717ab2eba75297493ee9393edcf3048792fdd5", "event_stream_digest": "a8d5cc364aa2c904afbae808baa659af96246aa97da908f05a926922abc4b6ef"}
{"model": "gpt-5.6-sol", "ref_id": "fab-03", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 35.55, "receipt_request_digest": "4f26b9d605bc8376c0156534ce528c5a2c25801ab0f27a22df4d446eaf7e7e2f", "event_stream_digest": "67ca833d6da919d1dcab6f7d1adf1ea52014bf7f803636692350898ff09fd41e"}
{"model": "gpt-5.6-sol", "ref_id": "fab-03", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 23.6, "receipt_request_digest": "4592161530c46f841885d49a6402ae95d965509b75ca2eeb8fd5985e79222b6b", "event_stream_digest": "aa910f13540c7f9feacf5b55ed597364b3a2f3bbdab8e28f0f6fc799aff30dc0"}
{"model": "gpt-5.6-sol", "ref_id": "fab-03", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 22.14, "receipt_request_digest": "626474d57769fd5653e09e46c068c2363f3cf4cba97339ba6dc1ea5bd8f2dc21", "event_stream_digest": "cbf717fa62a438d1e19b160081265632731d9c589df2fa305405f3ac00c32e74"}
{"model": "gpt-5.6-sol", "ref_id": "fab-04", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 24.18, "receipt_request_digest": "eb4cc76078a40d2ee8d83cf5111a2c63d08fd29818b5b014997ed474e7406e4a", "event_stream_digest": "0e08bc2dc3d4a6873ddd80fcede9252dd599b0085dd711a9cc2d0b01b98ec975"}
{"model": "gpt-5.6-sol", "ref_id": "fab-04", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 22.37, "receipt_request_digest": "da3845fe5720115ad34138b212c854c8e9e81e6928e1ba9aba931c152f8b0c3b", "event_stream_digest": "9d98aff2354720e4553494e8b4c7e0fbbe5dd4094480a721a46f7d7c33bce6f0"}
{"model": "gpt-5.6-sol", "ref_id": "fab-04", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 15.98, "receipt_request_digest": "9eac1ec49c6bfdcda9826cd9a2ddb84220a4cd684dbca24688c059ee541252e9", "event_stream_digest": "dc35a955b9cc32362b1f6f41c62a5c0f7d80730059fb1d83160af80a2473113b"}
{"model": "gpt-5.6-sol", "ref_id": "fab-05", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 23.04, "receipt_request_digest": "b593aeda54dda52e6008710af08da875f2950d81cd02c89cdae5864f16db2bba", "event_stream_digest": "994b5067183e31ded9e16429275c24cc129a314ac7ff665db612310d35f67ce5"}
{"model": "gpt-5.6-sol", "ref_id": "fab-05", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 34.37, "receipt_request_digest": "de64dedf6156c4546b05e6ccb67eeedfb9aadd9ac5949668297a62785b35fe8c", "event_stream_digest": "04c3bb8103643ea03a9daa2045717c179e09bf3da1ec938e68219e8b5b86fae3"}
{"model": "gpt-5.6-sol", "ref_id": "fab-05", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 23.89, "receipt_request_digest": "7af7354fa2ad62cc16be1103289fde141451347d951d1fc574d76caf02a16a0c", "event_stream_digest": "7b67176fa14573c911ec67671a7794c5cf6fef8610c2046dd12c447de5901535"}
{"model": "gpt-5.6-sol", "ref_id": "fab-06", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 25.84, "receipt_request_digest": "b3b7f99a251e50144feda543040cf0fb1288d35cfc9f5e24b41b08957fd0ad13", "event_stream_digest": "f964c88afbaa8bdf0bce52368d00c0110d084ae6d30d2fcde60bd8b62b2ae1e7"}
{"model": "gpt-5.6-sol", "ref_id": "fab-06", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 20.38, "receipt_request_digest": "6952deb7f8149fddaa5d31e40ff15a0231401333046ec443d0486a4b769d2d92", "event_stream_digest": "77f22d3b140a9e056f2a92ebae86b6c7c7acbd86a23e919f8f0340b9169a1b95"}
{"model": "gpt-5.6-sol", "ref_id": "fab-06", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 18.73, "receipt_request_digest": "9d2860c222a1eab7abcf6c74270c0d1f64397502fe16ea453d05ccfccda8d3d2", "event_stream_digest": "100d45aa2b6593904ebf1aae515923078774f0d46318a58fd738395d9b9cadee"}
{"model": "gpt-5.6-sol", "ref_id": "fab-07", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 21.82, "receipt_request_digest": "c1d8c623b4319bf6b21f5293f1f497706cac16dec0fc9c394dd461a28ce6c28a", "event_stream_digest": "e71fd26ba8ddb657ca06d99258fb964c4ca1685e302653b80e26ca4d1884dbe1"}
{"model": "gpt-5.6-sol", "ref_id": "fab-07", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 20.04, "receipt_request_digest": "b3ac0ca0d746b268efa31cbe5734ee1354fd25b0487535b0e30fdaa1052f071c", "event_stream_digest": "e6a953b6885f975944d16a6691ad502e81542f570319552ab4abdcfb4dbd8ffc"}
{"model": "gpt-5.6-sol", "ref_id": "fab-07", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 12.3, "receipt_request_digest": "60666da5f4791f815fdb0742e82b5c120aeebe0cb011d6aabd3c08d0ad4b0999", "event_stream_digest": "5132b758527f55c9492c973719fbead708cd859948e65e9bef96deb2d89d5221"}
{"model": "gpt-5.6-sol", "ref_id": "fab-08", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 22.74, "receipt_request_digest": "5eda7c750ab47fcee0c23bfedcab4be92dff14f9f56ee1997dae0c3e9d209275", "event_stream_digest": "f59e977500927fc8134264ff9978da650e8d0871d34f3aaca476898c701efa11"}
{"model": "gpt-5.6-sol", "ref_id": "fab-08", "repeat": 2, "verdict": "MISMATCH", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 22.43, "receipt_request_digest": "7f859e0134e2c5e61180093a791b8bfc8e2ac7cec253a11e31929ca0caf838bb", "event_stream_digest": "9f59d10bd76a1ebebf6c9ad6de88151d0c7d5b9249a4ad430b6ea9a144dd1968"}
{"model": "gpt-5.6-sol", "ref_id": "fab-08", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 22.24, "receipt_request_digest": "cb2ea6d06a6cbf756e8c436e4992838763bb75aebc9b5321decbef8c7b4973b9", "event_stream_digest": "4604103d5b499abe70659eba5a7e17370800890481472eab22d5382076f87112"}
{"model": "gpt-5.6-sol", "ref_id": "fab-09", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 20.89, "receipt_request_digest": "342553a20efc266b1e98b9a84c5cdcbd8cb0632ed0cc0b2f01ad272b04db3c49", "event_stream_digest": "af6232a86a9f9aaca53afb432fa7e6cb6305425b014bf669dfbdc039c7bf4cd3"}
{"model": "gpt-5.6-sol", "ref_id": "fab-09", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 28.87, "receipt_request_digest": "acb35e1276902cdef63b8997f86336b3044a6ae97512573182ca04e2162b047c", "event_stream_digest": "efb95f9e1928e39763b9235ab62f589a5c3f76a2209941da11d93d1a52c44056"}
{"model": "gpt-5.6-sol", "ref_id": "fab-09", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 20.32, "receipt_request_digest": "c45ceb4d9867c516a9973544d6d1d5a9cbdfbf10ac2bbbcdc3d1b1e0f5ffbba7", "event_stream_digest": "ab84431e3c95c8184f0397826ac8af43f28daafc6c4caf1300333598e52dc452"}
{"model": "gpt-5.6-sol", "ref_id": "fab-10", "repeat": 1, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 24.87, "receipt_request_digest": "de1e7469ac017146e8393bdb778aaf53864126ed606642324faff7b4c8987ecd", "event_stream_digest": "31a4e6adad9cfd25be314853a35d9648117c0b16b018c58a5c0dd05b1bfb95e7"}
{"model": "gpt-5.6-sol", "ref_id": "fab-10", "repeat": 2, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 26.32, "receipt_request_digest": "42939bdbf9ee3833b98c4d2286256712745362f0fed4db94fe5636b0ff0d39cb", "event_stream_digest": "83d70b80a9a74aef3ad558122ec73709a102b9177d9aff7a4cdff9e150857dea"}
{"model": "gpt-5.6-sol", "ref_id": "fab-10", "repeat": 3, "verdict": "NOT_FOUND", "searched": true, "reason_code": null, "n_sources": 0, "wall_seconds": 24.23, "receipt_request_digest": "a2e53f25e0aecdcd5002f7421bc8e92875d616ab4fd509a73307ba4956ff54ba", "event_stream_digest": "81baf6ea5e531986634101ed41867c922d63e8c96e940e0b65d6ab3b99d7fe94"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-01", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 13.27, "receipt_request_digest": "37309b778057e7edf6db83202992b15c703b68209599ebf80fd39ebc03a7532a", "event_stream_digest": "cdb4d84233c0286e4beaa617daca6fb3ad6d895d79477714b76ebf607d2d4fea"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-01", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.19, "receipt_request_digest": "b05cd4e961cf1ac8eca0a47e8b773c97d450194e59a5ff4c99c0b37f631c1b70", "event_stream_digest": "654b76aceed8d41e8d7bbb3ec524a49167cf0a748b6983a7e3a56f1eb171f050"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-01", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.0, "receipt_request_digest": "381d6d035a0b031d1788ef80e199fb50c4f36123d532e4272f7c54a46ed09230", "event_stream_digest": "139661fd894964964bfd500b0934a517c5d9ca7cf29dc415d3edba0690504de9"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-02", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.66, "receipt_request_digest": "16811b87c363755019c4806e1cdc0a618287e6637493d5a68d3bbc2d174f49bf", "event_stream_digest": "535e1412d4b668aa09866b800f3cb0eeda6df373567ee557109aef6095fe651d"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-02", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.53, "receipt_request_digest": "fe198ff26cbc9f618ff6644f31ba305a785f2d0fab602fe32b6016ef61729bf0", "event_stream_digest": "a049fa3de4a35e00bbdca62730b152ab342c380aa18f82a6cab79af859c3a23e"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-02", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 17.16, "receipt_request_digest": "52f4d1876fad5f55adb3519b13498861198ff59aaa4ca861ce363641ceffafd7", "event_stream_digest": "3e869d0e842547a009f826693cde8f3978da69e22796af774818f31a2621aa7e"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-03", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.92, "receipt_request_digest": "9675ba1cdad2a6bdb0388647ffc0bf361a8595f3fb28e486b51432b6d5b92992", "event_stream_digest": "3488202cd86d43c41fc0e8759228d88e90ee515d338a03319a6d982851e2595a"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-03", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 16.33, "receipt_request_digest": "86ee97aab697e8cf4d2f45fe81362da7b3d9224200afd66823492ccb4b4e4387", "event_stream_digest": "bafc4de97b25730595de4bf555e6691264357049b1901352685e7e32dc25afe0"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-03", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 17.0, "receipt_request_digest": "c117abed6f9dff6d6f24be2fbb9ac48f8613bb5af888089a08f7cf0e89c74932", "event_stream_digest": "8b6c750539244e4124025331bf50d811585a44ce88d613e33f7270ab054726db"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-04", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 17.77, "receipt_request_digest": "c37f87e987f09892dd0d70a097078e79c3406dea37f190355427b8a247f6c0a7", "event_stream_digest": "567e558f2ff0842137ab38c5030fd204d9b2f4d5a2ca9441d0bdfcbdbb11c70f"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-04", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 12.47, "receipt_request_digest": "00b48afdd4c6fa9ad7e4acbf6146345cba0f4cba678311124624086a89076f40", "event_stream_digest": "cf1083c8eee0b3652ef0c814ddca344d1440524100a6d8eb2595c6a8aefe2462"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-04", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 14.21, "receipt_request_digest": "60f909b0eef9f0833acdda6292b902a1051f4a85cbcadf7c78711e42b0ee3d88", "event_stream_digest": "885a15940dfa11ea64c148a815716c2f46da90545c96bb72b3887f3c1c1a7829"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-05", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 15.82, "receipt_request_digest": "35db4b02ceb6d64fd02d2a999d9d3d2ccbba093fc46c6bcd4526f5e4d400acad", "event_stream_digest": "363587bce619b6ff3032bbb4f5e59fab926687610843bb2012d3905ac02bf1f2"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-05", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 13.03, "receipt_request_digest": "ac1436af1bf62b1849511b7e2ca0eb4fe8c1c13c4b66af18692e05fc1c7c06b5", "event_stream_digest": "e8445c0a51bf1ef3204785c20ac53d953ae3b2789c5435123ad9a67e42c3511a"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-05", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 19.84, "receipt_request_digest": "656477bcdce0abe1797e50ac689d00e5b05a6433f36938b380abe841d0283e7f", "event_stream_digest": "9c23c050ec384a9f3004edaa09bd1ebc5678eefb7e2bcee2c6acdcc4ed931db3"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-06", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 19.5, "receipt_request_digest": "bc1154d8dbd55314838e5e567c30d9af7ee7b3b6860b36494e9152bf74d634c0", "event_stream_digest": "7d8a622fc97c8d2d0e1656e6d2bdf7186515450aa5a9b1069e7c9f39cb8c8cde"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-06", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 17.5, "receipt_request_digest": "1d206f722e935276b9e447a5f497e2b5201baded3ed6576edec264748624ccf6", "event_stream_digest": "2c0120e1a6fd25b6fab61ee4f4bb43b1ff8a470104da04f9a90109340c337992"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-06", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.32, "receipt_request_digest": "382346c9dc652ed31eae407c36f2ce4dcf9db8341a4b3111625902f1163c7e92", "event_stream_digest": "1313958bcc17b039ad770380ec4764916cc7a97c97a63aaec6ef8fdd7ef09164"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-07", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 13.25, "receipt_request_digest": "5bff5beaba3c748491c940d5900c2279680298a41729be5c60fc1d6f7a101b1e", "event_stream_digest": "d016206bb82c546a72af453ca045f28b98afbbd3c54f54b2322a30e79c7a37f8"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-07", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.62, "receipt_request_digest": "488044ebc522aa9c6d92a5bee47d5318042c629db9eeb68ab01b8515b7d32a0f", "event_stream_digest": "1cad2ce65cbd323b3fd1bdc4d32b336c691156bd5c2acfd765f061d04e951c57"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-07", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.05, "receipt_request_digest": "724b7c4747cedc1b3929f494b95ea78fad7162911d6aeee44f80e3b4779b5cfe", "event_stream_digest": "7681aa45c1ad8e70ad5c6baec6c075373b7469d57ac5ea6b14af7b667009a73c"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-08", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 15.28, "receipt_request_digest": "caeab6c250682c7f2e0b767bfbb20f206fea3399f0d0a051d5b307cf4e116187", "event_stream_digest": "202c4e7dacb9f512b217649732e056ea7f6f061eb0b73a7224d956271b7551d4"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-08", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.0, "receipt_request_digest": "476326f4541996531657d0536b5c412f994a533f2e5cbcdfb6a4c7691d6c30fb", "event_stream_digest": "39efb677e1ac4739bf990b35d834a9fc2e03b0200125dac4b34a5dacf3439a46"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-08", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.99, "receipt_request_digest": "76b6435896b944e98141d718eb17d4449f112c0367d409d1032316e492235674", "event_stream_digest": "ece5cd25235642ebf5fad2f5f53b7b98ecb6799916351fc889399da28b18d303"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-09", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.04, "receipt_request_digest": "e3af01d3f6b7b86df62bc7c44c0711ea2b0cd12986cbf58eb46052321ec9e9d9", "event_stream_digest": "72a19ff858ae6a09ae2205586dbb545b66eea2431f66fdaa68a16e09bb3ae477"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-09", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 28.53, "receipt_request_digest": "04b504bd9432be1415fb68e3b4e5db3187f1cb9c156c8d068882584730affc04", "event_stream_digest": "10d104f7da5011bad018a92e916d5263d7ef2307cee0a2debf34fd973416b39b"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-09", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 22.16, "receipt_request_digest": "4fe0bfe810d412dd531538d5471899d6c4e692747cc77ca8be3139c516230642", "event_stream_digest": "8e3ef4d1b3d062c6048d437e4537f2e57e28ecc41464b4c392906f9dc5d083ea"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-10", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 16.55, "receipt_request_digest": "f540d65fc6a2b1ae652c44ec31c82a673b614149abbac8aea544b7a098e3fb51", "event_stream_digest": "302f3c6e6ab4039ff4268818bc5092c6c0c4c1b974fa2bd204b7d013fdbd713f"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-10", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 17.96, "receipt_request_digest": "dcedbc7f6f0cac207b1d14d67ac278a5550fd8ffea8487f4750d4631e8753653", "event_stream_digest": "c2ab5d90b6c0471216d5243b6c0925846a825b87b1df0a2bd4a40d91d10b15ea"}
{"model": "gpt-5.6-sol", "ref_id": "real-easy-10", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 14.55, "receipt_request_digest": "531b81b1a70cd5daee4cbf6d40f5fb6668b54f5cf039e0a96d696bf6209f2055", "event_stream_digest": "2a1860a301e41967f86dfaff95f0e608b39cbeacd9c9699c1de0377ff86ee536"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-01", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 18.64, "receipt_request_digest": "45716b73948e5572858491c7f4b06d2ed3a1ce9893065d4777b0281a5201e814", "event_stream_digest": "ad85601b568b19e632e1b162da438420fb4e6465dd008ccaeee3f76ad4612d2f"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-01", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.8, "receipt_request_digest": "0efd46b05dd5366c0f938a86ece7c8044341eef905ccdf1402eeba3c8080a427", "event_stream_digest": "8494ce3cbb69a4abdc7120e15287ef49e90b7f9f458f0ae9cdf4ddd581032102"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-01", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.28, "receipt_request_digest": "df1a123b7a2ffb13e8cfe4c3ce235365ceaf7357d7c159c25767c2c9fe18b3d1", "event_stream_digest": "01344095b5afe9e9c3ad436cce12884d601f6fc279a016be24025968c92e6de3"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-02", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.58, "receipt_request_digest": "76830c74bddd7b981f6bc9b19fb9f7bf8b5848da15a967a721b04b20cf23d4ef", "event_stream_digest": "f67fe779a1538a5f8d3fd179726b0985e79b9e2ec71ca15ddc3ad757c4d6d3d9"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-02", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 16.07, "receipt_request_digest": "3eb9beab157706001a7314f359bae64b586aa1242776f7859476d4dd4e1a61c1", "event_stream_digest": "d6ce37803e647e02facb96f2b3379d1565a74047a502a6dcc0d0c59d608e54dd"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-02", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.15, "receipt_request_digest": "1686f967a132ed6a713af136daf5fa69f00347e2b66b00624c37f57585a95d55", "event_stream_digest": "c39c569a3e5b5c0fca0786be7f992f12acc7c865a29c0237209f78fc82e19527"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-03", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 3, "wall_seconds": 14.45, "receipt_request_digest": "c1c3c8075aa30a155359ab6e91f6de6c0b0214e6bc4e46798bda21c0686f9d20", "event_stream_digest": "6bb7f8934286eeb96079bf13b570b2be827dc003839e9d8ca801bf7df2dd3a96"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-03", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.38, "receipt_request_digest": "72a1bf182880bdedc3f3ba77ca64f7ae9abaa56b9d53ad0817784cc53e945ccd", "event_stream_digest": "f96c82615a5e77255ad5128a88c4dd52b85300661578a47c5362eb7cc7f536c4"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-03", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.93, "receipt_request_digest": "27fac6eb5e93d979293fe542960b9bb7f6854afa64790fd81e574991b71afe9b", "event_stream_digest": "727294a6728810fe809af18c0ec858eb2b7cfac53ea11f5ae438ebb52f6a4adc"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-04", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.99, "receipt_request_digest": "3f223cadb717dedea1fc8de35a59d077d6c6a9f8f8200692315bdd1d4576da2b", "event_stream_digest": "81644b770aa8286ed7ca7cfd4b3120b772debd3f98f8e089a29c6bcb54b7b585"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-04", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.72, "receipt_request_digest": "64b9db2055916ac79e2f5fc681dd11c40b3c55fe251997de09574ef05c20b325", "event_stream_digest": "20477cd5c013608cf2b15e5a893895d0281fbea16df54d91c0bde66c305d8341"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-04", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 13.56, "receipt_request_digest": "cd7f28a69a8fb76af920d3e5a733ed6bd15258a78e31517b41154cc140ccb84b", "event_stream_digest": "972d98a0373d777545f734eabad3903f076099aa563b1aadcf9a618706e59452"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-05", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 16.87, "receipt_request_digest": "a8114a08e3a65d24b5b2f619ee6d30c47917a445f3ebcea10e3e7f3d2b4fd3d7", "event_stream_digest": "0820fc1442d86da9464bcf0dafcd612e64cf8714c096c10e6c02a91fd588657a"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-05", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.2, "receipt_request_digest": "8257c395ead71dcd8668224144552e262618f2eeaef1d259b89019263787e836", "event_stream_digest": "db99c08270b6b4601e051e15a3ec4e930c237230b0007207f4b1f4dd3ed3b697"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-05", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 15.83, "receipt_request_digest": "a7f2f008ca68d966670b480134b462c92c1e2feb83dfcd5a67532ab2033f7450", "event_stream_digest": "bacaf38e8d8be44b52f7835deb561bdb12f930e21948542111a0d4243fa0e2fe"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-06", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 14.07, "receipt_request_digest": "cd77deacfa1a619818a27fe05d66bb3be4999d6849691076e539e5e42932e8b1", "event_stream_digest": "bfe49a94c65dd9d39f5ec7e9f39dde63938431fcd0a3cb49007a7f037e888edc"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-06", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 2, "wall_seconds": 20.26, "receipt_request_digest": "bf7bac82a2c95fca48f05af1c7c0098139c7618d7b8142354fa22f13eccb6b78", "event_stream_digest": "faae9b31b317e3af312d122de25dc732fb7f742f8d81cbc85596b6a890004e41"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-06", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 15.47, "receipt_request_digest": "9f548f31b31f4a1c64cf94c9d949ee9f34387f69a7782c1bb589b7efed660137", "event_stream_digest": "cd59c4c38807bb724b347bbe58e29dce965d4b39bfc1c84263b1a93837e5a1fd"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-07", "repeat": 1, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 25.81, "receipt_request_digest": "05d37780b360611c2d065a67f865da4a64b87ace5d6ba2eab0d41278d06c33a2", "event_stream_digest": "09c1c1f67040af69f8e99f01ecb7e89a839141f53c6edac469426dd7bf3ad355"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-07", "repeat": 2, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 23.08, "receipt_request_digest": "9d2ac46226b31d995be559a33095bc4b8a20d87d5e240addff81a0f71ed9da42", "event_stream_digest": "d84fa31d3e9b89283a906d8ce22ec5c018ec6ac6b1fae3ab89fdf2a866718703"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-07", "repeat": 3, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 24.12, "receipt_request_digest": "19d49f57261059bae9a087af98da787123780494cee53b4c02a34863157ebe11", "event_stream_digest": "52b1cf77b731d26abe9f016588635ad27b07a2f654910c2df697097698e51413"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-08", "repeat": 1, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 21.87, "receipt_request_digest": "c1a9908f1847d1374a54040116c6fa56e2cb9d2682bb2edd91023ea918c3506e", "event_stream_digest": "5612d541cad912410ecc771445ee7238c053018c691ef97b8d86fc3918b06001"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-08", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 16.92, "receipt_request_digest": "8b694692f9b62ae6f0f235f82eeaa86c2e9d95192215320f4d3669d368e78bc6", "event_stream_digest": "1c68e0bdfaffa8ba0366efe089942e439d07bd1cf35fd57fe96f831e361823f7"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-08", "repeat": 3, "verdict": "NOT_SEARCHED", "searched": false, "reason_code": "SOURCE_NOT_IN_SEARCH_RESULTS", "n_sources": 0, "wall_seconds": 19.9, "receipt_request_digest": "fb09076143c33203e975af5525a433e13d3bf4e2e3b360971b7225fcaab52c6a", "event_stream_digest": "f478331eded15aa25fe749520d95541cd972355fa29d38768e3328539f5a8e34"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-09", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 13.16, "receipt_request_digest": "23720d287d4f67a9921418127d43f3f5bcaa979263c0deebca4ff9c1ded45530", "event_stream_digest": "3ac88aa70e0c41f02c27d498161ab8df85c49b4facd6baf92a001bcab1a2cadc"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-09", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 18.35, "receipt_request_digest": "12ba95cf68d3c418bb4b932c0c132ab2b6b2670bdd00d65bb16df11c9b6c92e9", "event_stream_digest": "16a5e3336f1b6f05e47c9436036437c4ed0f4c826591c57b06956501a7b16ce9"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-09", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 23.1, "receipt_request_digest": "56a7ac55662b6953745685e787d26cc9e6e569f854338f15f8ac5129d39f686e", "event_stream_digest": "d95f5febb4e2487de6d55a58f7b34c39dc1b057cf8aec2a4b17920655275d84c"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-10", "repeat": 1, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 26.19, "receipt_request_digest": "eec8cb07a82e10e436a24da454da7cb1f6aa0937b8892ab9106a169d501ab930", "event_stream_digest": "8e1e72881b92a403c846422ae3bb40f9e5436d97b42c06a8b59415380e623d2a"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-10", "repeat": 2, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 23.63, "receipt_request_digest": "0c1bd1632dfd4faf9bf83045be87695628a6a520f1fb47c1b591e8d6ab193f23", "event_stream_digest": "dd42f8f19659666a1a450b16d5711383cdae930026ef32349e157a454e24b04c"}
{"model": "gpt-5.6-sol", "ref_id": "real-hard-10", "repeat": 3, "verdict": "VERIFIED", "searched": true, "reason_code": null, "n_sources": 1, "wall_seconds": 22.56, "receipt_request_digest": "10ca435920771043f808942b04e7745c9fdc3362594f1a89f91db627f7943e57", "event_stream_digest": "23697c902ea672a20ed129043458e199d896dda252afa2ac10e5fab4c0dbc806"}
scripts/cross_model_codex_transport.py
+18 -2
View File
@@ -171,7 +171,9 @@ NOT_FOUND means a reference-bound search completed but no matching work was foun
NOT_SEARCHED means you could not complete a reference-bound live search.
For VERIFIED or MISMATCH, sources must contain at least one exact HTTPS URL from the
structured search results you actually received. Do not copy a URL merely because it
appears in REFERENCE_DATA. Keep detail factual and under 2,048 characters."""
appears in REFERENCE_DATA. For NOT_FOUND or NOT_SEARCHED, sources must be an empty
array — describe any absence evidence in detail instead of listing URLs. Keep detail
factual and under 2,048 characters."""
class TransportError(RuntimeError):
@@ -1051,10 +1053,24 @@ def parse_app_server_messages(
receipt["detail"] = model_output["detail"]
return receipt
# codex-cli 0.147.0 also emits webSearch items for follow-up page opens
# (action.type == "other", no query string). Those are not standalone
# searches: they are excluded before the item cap and skipped for binding
# purposes — a URL seen only in an opened page can never become a bound
# source — but they are not stream-fatal (#787; previously every
# fabricated-reference run died as EVENT_STREAM_INVALID because absence
# checks legitimately open result pages). Search-typed items keep the
# exact strict validation below.
def _is_page_open(item: dict[str, Any]) -> bool:
action = item.get("action")
return isinstance(action, dict) and action.get("type") != "search"
all_searches = [
(index, item)
for index, item in completed_items
if index < final_index and item["type"] == "webSearch"
if index < final_index
and item["type"] == "webSearch"
and not _is_page_open(item)
]
if len(all_searches) > MAX_SEARCH_ITEMS:
return _empty_receipt(request, model, event_digest, "EVENT_STREAM_INVALID")
scripts/test_cross_model_codex_transport.py
+51
View File
@@ -352,6 +352,57 @@ def test_grounded_mismatch_and_model_not_searched_are_closed() -> None:
assert receipt["reason_code"] == "MODEL_RETURNED_NOT_SEARCHED"
def _page_open_event(url: str = "https://example.org/toc") -> dict[str, Any]:
# codex-cli 0.147.0 emits follow-up page opens as webSearch items with a
# non-"search" action type and no query string (#787).
return {
"method": "item/completed",
"params": {
"item": {
"id": "exec-view-1",
"type": "webSearch",
"action": {"type": "other"},
"results": [{"type": "text_result", "url": url, "title": "TOC"}],
}
},
}
def test_page_open_web_search_items_are_skipped_not_stream_fatal() -> None:
messages, raw = _fixture("grounded_verified.jsonl")
messages = [_page_open_event()] + messages
receipt = runtime.parse_app_server_messages(
messages, raw_stream=raw, request=_request(), model="gpt-5.6"
)
assert receipt["verdict"] == "VERIFIED"
assert receipt["searched"] is True
# A page-open's URL never becomes a bindable source.
model_output = {
"verdict": "VERIFIED",
"detail": "Claimed from an opened page, not a search result.",
"sources": ["https://example.org/toc"],
}
messages2, raw2 = _fixture("grounded_verified.jsonl")
messages2 = [_page_open_event()] + messages2
messages2[2]["params"]["item"]["text"] = json.dumps(model_output)
receipt = runtime.parse_app_server_messages(
messages2, raw_stream=raw2, request=_request(), model="gpt-5.6"
)
assert receipt["reason_code"] == "SOURCE_NOT_IN_SEARCH_RESULTS"
# A stream whose only webSearch items are page opens has no bound search.
messages3, raw3 = _fixture("grounded_verified.jsonl")
messages3 = [_page_open_event()] + [
m for m in messages3 if m.get("params", {}).get("item", {}).get("type") != "webSearch"
]
receipt = runtime.parse_app_server_messages(
messages3, raw_stream=raw3, request=_request(), model="gpt-5.6"
)
assert receipt["verdict"] == "NOT_SEARCHED"
assert receipt["reason_code"] == "NO_BOUND_SEARCH_RESULTS"
def test_positive_without_source_fails_closed() -> None:
messages, raw = _fixture("grounded_verified.jsonl")
messages[1]["params"]["item"]["text"] = json.dumps(
shared/cross_model_verification.md
+11 -6
View File
@@ -42,7 +42,7 @@ A stress test of 68 AI-generated citations found 31% had problems — and all pa
| Model | API ID | Provider | Best For |
|-------|--------|----------|----------|
| Claude (session model) | _(inherited Claude Code session model — e.g., Fable 5)_ | Anthropic | Primary model (default for all ARS skills) |
| GPT-5.6 Sol | `gpt-5.6-sol` | OpenAI | Cross-verification — current OpenAI flagship, recommended OpenAI verifier; **provisional pending ARS validation** (same standard rates as GPT-5.5) |
| GPT-5.6 Sol | `gpt-5.6-sol` | OpenAI | Cross-verification — current OpenAI flagship, recommended OpenAI verifier; **validated for the ChatGPT-subscription citation transport** (2026-08-19 bakeoff, superiority on recall + latency — `audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md`); **provisional pending ARS validation** on the first-party API route (same standard rates as GPT-5.5) |
| Gemini 3.1 Pro | `gemini-3.1-pro-preview` | Google | Cross-verification — current Google flagship (validated); strong at factual verification |
| GPT-5.5 | `gpt-5.5` | OpenAI | Cross-verification — previous generation, superseded by GPT-5.6 (2026-07-09); validated, remains fully supported (supports `xhigh` reasoning) |
| GPT-5.5 Pro | `gpt-5.5-pro` | OpenAI | Cross-verification — previous generation; validated; strongest GPT-5.5-line reasoning (premium pricing: ~6× GPT-5.5) |
@@ -61,7 +61,7 @@ A stress test of 68 AI-generated citations found 31% had problems — and all pa
> The primary row deliberately names no version: the primary is always the session model, so the row cannot go stale on the next Anthropic release. Verifier IDs stay concrete because they are literal API strings the user must export. (`gpt-5.4` / `gpt-5.4-pro` remain accepted for existing setups.)
> **GPT-5.6 Sol is provisional (listed 2026-07-11, three days after release).** Its endpoint support (Responses API), hosted `web_search` tool, and reasoning-effort values are confirmed against OpenAI's model documentation, but its ARS-specific behavior — grounded-search completion rate, citation-mismatch recall, false-disagreement rate, response-shape stability against the jq grounding guards, p95 latency — is unvalidated. **Recommendation policy (2026-08-19):** GPT-5.5 was superseded by the GPT-5.6 family on 2026-07-09, so the recommendation names the current generation rather than a superseded id — a lifecycle decision, not a measurement claim. `gpt-5.6-sol` stays **provisional** until it passes the § Promotion Bakeoff below; `validated` is earned only there, and no parity or superiority over GPT-5.5 is claimed in the meantime. Run `scripts/cross_model_smoke_test.sh` against your key before adopting it; users who prefer a validated id can stay on `gpt-5.5` or `gemini-3.1-pro-preview` (validated = the id-status allowlist below; no bakeoff run has been recorded for any id). Two facts that differ from the GPT-5.5 lineup: GPT-5.6 ships **no `-pro` model ID** — premium operation is standard `gpt-5.6-sol` plus `reasoning: {mode: "pro"}` in the request, billed at standard token rates with more model work per request (the old fixed ~6× unit-price split does not carry over); and its reasoning effort accepts `none|low|medium|high|xhigh|max` (GPT-5.5 tops out at `xhigh`), defaulting to `medium` in both standard and pro modes.
> **GPT-5.6 Sol is provisional (listed 2026-07-11, three days after release).** Its endpoint support (Responses API), hosted `web_search` tool, and reasoning-effort values are confirmed against OpenAI's model documentation, but its ARS-specific behavior — grounded-search completion rate, citation-mismatch recall, false-disagreement rate, response-shape stability against the jq grounding guards, p95 latency — is unvalidated. **Recommendation policy (2026-08-19):** GPT-5.5 was superseded by the GPT-5.6 family on 2026-07-09, so the recommendation names the current generation rather than a superseded id — a lifecycle decision, not a measurement claim. `validated` is earned only there — and on 2026-08-19 a codex-transport bakeoff run earned it for the **ChatGPT-subscription citation transport**, with a measured superiority case (fabrication recall 1.00 vs 0.80, p95 latency 26.5 s vs 58.7 s; `audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md`). On the **first-party API route** `gpt-5.6-sol` stays **provisional** — that run did not exercise the API route's jq grounding guards, and no parity or superiority is claimed there. For the API route, run `scripts/cross_model_smoke_test.sh` against your key before adopting it; users who prefer an API-route-validated id can stay on `gpt-5.5` or `gemini-3.1-pro-preview` (validated = the id-status allowlist below; the API route has no recorded bakeoff run). Two facts that differ from the GPT-5.5 lineup: GPT-5.6 ships **no `-pro` model ID** — premium operation is standard `gpt-5.6-sol` plus `reasoning: {mode: "pro"}` in the request, billed at standard token rates with more model work per request (the old fixed ~6× unit-price split does not carry over); and its reasoning effort accepts `none|low|medium|high|xhigh|max` (GPT-5.5 tops out at `xhigh`), defaulting to `medium` in both standard and pro modes.
Using two non-Anthropic models as primary+verifier is possible but not tested with ARS prompts.
@@ -133,10 +133,10 @@ by detection and execution.
```bash
# Citation-integrity calls only. General DA/reviewer/judgment calls remain on API transport.
export ARS_CROSS_MODEL_TRANSPORT="codex"
# Deliberate transport-specific exception to the generation-currency recommendation:
# this closed citation adapter's example stays on validated gpt-5.5. Set another id
# only after scripts/cross_model_smoke_test_codex.sh passes against your subscription.
export ARS_CROSS_MODEL="gpt-5.5"
# gpt-5.6-sol is validated for THIS transport (2026-08-19 codex-transport bakeoff,
# superiority on recall + latency — audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md).
# gpt-5.5 remains the validated bakeoff baseline alternative.
export ARS_CROSS_MODEL="gpt-5.6-sol"
python3 scripts/cross_model_codex_transport.py detect
# The producer sends one closed codex_citation_request/1.0 object on stdin:
@@ -593,6 +593,9 @@ if [ -n "$ARS_CROSS_MODEL" ]; then
case " gpt-5.5 gpt-5.5-pro gpt-5.4 gpt-5.4-pro gemini-3.1-pro-preview " in
*" $1 "*) echo "validated"; return ;;
esac
# gpt-5.6-sol: validated for the codex subscription citation transport
# (2026-08-19 bakeoff); provisional HERE because this allowlist gates the
# first-party API route, which has no recorded bakeoff run.
case " gpt-5.6-sol " in
*" $1 "*) echo "provisional"; return ;;
esac
@@ -656,6 +659,8 @@ an API route.
The run that flips a provisional id (today: `gpt-5.6-sol`) to validated is defined here so a future promotion argues against numbers, not vibes (#518). Validation and recommendation are separate axes. (2026-08-19, #783: the recommendation moved to the current generation on lifecycle grounds — GPT-5.5 was superseded — ahead of validation; that flip carries no measurement claim. This bakeoff remains the only route to `validated`, and any claim of measured parity or superiority still requires the run below.)
> **Recorded run (2026-08-19, #787 — codex-transport variant).** The procedure below was executed over the #630 ChatGPT-subscription citation transport (entry gate: `scripts/cross_model_smoke_test_codex.sh` PASS for baseline and candidate; measure analogues: grounding evidence = receipt `searched`, measure 4 = zero fail-closed receipt-guard misfires). All five measures passed with superiority on measures 1, 2, and 5 — see `audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md` (probe set `evals/bakeoff/2026-08-19-gpt-5-6-sol-codex/`, sha256 in the report). The result is **transport-qualified**: `gpt-5.6-sol` is validated for the subscription citation transport; it remains provisional on the first-party API route, whose jq grounding guards that run did not exercise. An API-route run may reuse the same probe set by hash.
- **Entry gate:** `scripts/cross_model_smoke_test.sh` passes against the candidate id.
- **Probe-set precondition (reproducibility):** before any run counts, the probe set must be committed as a versioned fixture (under `evals/` or `audits/`) listing each reference's full text, its ground-truth label (`real` / `fabricated`, with source DOI/URL for the real ones), and the file's sha256 recorded in the run report. A bakeoff against an ad-hoc, unversioned probe set is not a gate result. Composition: 30 references — 20 real (10 easy: DOI-keyed journal articles; 10 hard: preprints, DOI-less, non-English) + 10 synthetic plausible fabrications.
- **Procedure:** run the baseline (`gpt-5.5`) and the candidate the same day, one call per reference, 3 repeats. Per-reference verdict = the verdict returned by ≥ 2 of 3 repeats; if no verdict reaches 2 (a 1–1–1 split), the reference is **indeterminate** and scored conservatively against the model that produced it — a miss for recall (measure 2), a false disagreement for measure 3. Grounded-search completion (measure 1) is computed per call, so ties don't apply.