feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
# Cross-Model Verification Protocol (v3.0)
## Overview
2026-08-15 15:04:28 +08:00
This protocol enables optional blind cross-model checks for high-stakes AI judgments. When enabled, another model family can inspect bounded inputs without seeing the primary result. That adds typed substrate diversity and may expose shared-frame blind spots; it does not by itself establish independent error processes or higher accuracy.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
2026-06-06 10:22:45 +08:00
**This is entirely optional. ** All ARS skills work with the primary Claude model alone. Cross-model verification is an additional layer for users who want higher confidence in integrity checks, devil's advocate challenges, and review judgments.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
2026-06-04 20:44:03 +08:00
**Consent boundary: ** Before unpublished manuscripts, private notes, corpus text,
reviewer comments, decision letters, response letters, or other review material
is sent to an external provider, the agent must identify the provider, model,
and content class that would be sent, then obtain explicit user consent. An
environment variable alone is not consent to upload user content. If consent is
not granted, continue with single-model verification.
2026-08-11 12:56:01 +08:00
**Citation-only ChatGPT-subscription transport (#630): ** A fourth, deliberately
narrow transport is available only for the one-reference citation-integrity calls
in Stage 2.5 / 4.5. Setting `ARS_CROSS_MODEL_TRANSPORT=codex` selects the contained
Codex app-server adapter described below. It does not authorize or implement DA,
Reviewer 2, calibration, re-review judgment, design-freeze, editorial-decision, or
generic handoff calls; those paths continue to require their documented provider
API credentials. The selector is closed: unset or `api` keeps the existing API
route, `codex` selects this citation-only route, and every other value fails visibly
without falling back.
2026-08-11 15:29:59 +08:00
This runtime boundary does not forbid a separately preregistered, offline
held-out suite from choosing Codex CLI as its subject transport. In particular,
the #684 constructive-value plan is a synthetic evaluation with its own frozen
call plan, USD 0 API ceiling, and human expert labels; it is not a generic
reviewer/DA handoff and must not call this citation adapter with reviewer data.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
## Why Cross-Model Verification
A stress test of 68 AI-generated citations found 31% had problems — and all passed three rounds of same-model integrity checks. The root cause: the verifying AI and the generating AI share the same training data distribution, so they share the same blind spots. A different model (trained on overlapping but not identical data, with different RLHF tuning) can catch errors that the primary model systematically misses.
Align distribution-surface claims with evidence ceilings (#753) (#766)
* fix(claims): align distribution-surface claims with evidence ceilings (#753)
- plugin.json / marketplace.json: drop 'Production-grade' / '39-agent
ensemble' for matrix-licensed wording ('contract-audited', '39 prompt
roles (3 plugin-exposed agents; the rest run inline by default)')
- academic-pipeline/SKILL.md: no-bypass prose rewritten to the actual
mechanism (mandatory checkpoints; overrides require recorded user
reasoning); #528 content-lock hash updated in the same commit
- shared/cross_model_verification.md: 31%->5-10% relabeled as an
unvalidated working hypothesis
- shared/ground_truth_isolation_pattern.md: gold-labels rule rewritten to
the intended boundary (no unconditional loading into operational agent
context)
- version-consistency invariant 8: binds 'N prompt roles' spelling too,
checks every count token (finditer)
- new scripts/check_distribution_surface_claims.py (D1-D5, 20 mutation
tests, CI-wired): fail-closed manifest load, shared claim vocabulary
imported from check_stage_capability_matrix, percentage refusal,
mandatory bindable count token, plugin-exposed count bound to MIRRORS
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki
* fix(claims): codex R1 — integrity-family must-PASS sweep + lint case/boundary fixes (#753)
- integrity 'must PASS with zero issues' absolutes now name the recorded
3-round FAIL-loop exit (integrity_review_protocol, reinforcement_content,
team_collaboration_protocol, integrity_verification_agent, SKILL.md flow
row); 'recorded with reasoning' weakened to 'recorded user decision'
(rationale escalates per compliance override ladder)
- D3 percent check lowercases input (matrix caller parity)
- D5 plugin-exposed regex case-insensitive
- AGENT_CLAIM_RE gains trailing boundaries (39-agentic / singular 'prompt
role' no longer count as bound); 4 new mutation tests (20 -> 24)
- SKILL.md #528 content-lock hash rebumped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki
* fix(claims): codex R2 — Stage 2.5 routing parity, passport-state honesty, gold-set scope, strict JSON (#753)
- Stage 2.5 flow row + both state-machine checkpoint triggers name the
recorded FAIL-loop exit (SKILL.md + pipeline_state_machine.md, both
content-lock hashes rebumped)
- team protocol handoff checklist: FAIL-loop continuation keeps passport
verification_status UNVERIFIED; VERIFIED only on zero-issue PASS
- ground-truth gold exception scoped to synthetic/public-safe content;
live-reviewer calibration sets stay runtime-supplied
- D1 rejects non-standard JSON constants (NaN/Infinity) via parse_constant;
2 new tests (24 -> 26)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki
* fix(claims): codex R3 — prerequisite checker + handoff materials accept the recorded FAIL-loop route (#753)
- state_tracker_agent prerequisite table: Stage 3 / Stage 5 entry rows
accept a recorded Integrity Check FAIL Loop resolution (previously the
documented continuation route was unreachable at the checker)
- SKILL.md handoff lines 2.5->3 and 4.5->5 no longer mislabel a FAIL-loop
continuation draft as verified; team protocol Materials/Approval rows
aligned the same way
- SKILL.md + state_tracker_agent content-lock hashes rebumped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki
* fix(claims): codex R4 — orchestrator transfer rows + advisory dispatch accept the recorded FAIL-loop route (#753)
- orchestrator 2.5->3 and 4.5->5 transfer rows no longer require a
'Verified'-labeled draft on a recorded FAIL-loop continuation
- #660/#672 advisory dispatch anchors to the Stage 4.5 terminal resolution
(PASS, or recorded FAIL-loop continuation) instead of exact PASS only
- orchestrator content-lock hash rebumped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011n3WG8Z3Us8fX8UhXS51Ki
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 20:02:59 +08:00
**What it improves: ** Different models catch different types of hallucination patterns. The post-verification error rate has never been measured — the residual-rate hypothesis (that cross-model checks cut the 31% above to single digits) is unvalidated.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
**What it doesn't solve: ** Frame-lock (all LLMs share most training data), sycophancy (all RLHF models have this tendency). These are degree improvements, not kind improvements.
## Supported Models
| Model | API ID | Provider | Best For |
|-------|--------|----------|----------|
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
| Claude (session model) | _ (inherited Claude Code session model — e.g., Fable 5.1) _ | Anthropic | Primary model (default for all ARS skills) |
| GPT-6 Astra | `gpt-6-astra` | OpenAI | Cross-verification — current OpenAI flagship (released 2026-09-03), recommended OpenAI verifier under the recommendation policy below; **provisional pending ARS validation ** on both the first-party API route and the ChatGPT-subscription citation transport (no recorded bakeoff run; entry-gate smoke PASS on the citation transport 2026-09-05, codex-cli 0.153.4 — see the GPT-6 Astra note below) |
| GPT-5.6 Sol | `gpt-5.6-sol` | OpenAI | Cross-verification — previous generation, superseded by GPT-6 Astra (2026-09-03); **validated for the ChatGPT-subscription citation transport ** (2026-08-19/20 bakeoff, superiority on recall + latency — `audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md` ), the only id with a measured ARS run on any transport; **provisional pending ARS validation ** on the first-party API route (same standard rates as GPT-5.5) |
2026-08-19 18:00:30 +08:00
| Gemini 3.1 Pro | `gemini-3.1-pro-preview` | Google | Cross-verification — current Google flagship (validated); strong at factual verification |
| GPT-5.5 | `gpt-5.5` | OpenAI | Cross-verification — previous generation, superseded by GPT-5.6 (2026-07-09); validated, remains fully supported (supports `xhigh` reasoning) |
| GPT-5.5 Pro | `gpt-5.5-pro` | OpenAI | Cross-verification — previous generation; validated; strongest GPT-5.5-line reasoning (premium pricing: ~6× GPT-5.5) |
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
### OpenAI-compatible providers (Chat Completions API — UNGROUNDED, opt-in)
| Provider | Example API ID(s) | Endpoint (`ARS_OPENAI_COMPAT_BASE_URL` ) | Notes |
|----------|-------------------|------------------------------------------|-------|
| Xiaomi MiMo | `mimo-v2.5-pro` | `https://token-plan-cn.xiaomimimo.com/v1` | Set `ARS_OPENAI_COMPAT_API_KEY` + `ARS_CROSS_MODEL` . Ungrounded: positive verdicts never count as citation agreement. |
| DeepSeek | `deepseek-v4-pro` | `https://api.deepseek.com/v1` | Set `ARS_OPENAI_COMPAT_API_KEY` + `ARS_CROSS_MODEL` . Ungrounded. |
| Any OpenAI-compatible | any non-`gpt-*` /`gemini-*` id | any `/v1/chat/completions` endpoint | Routing is governed solely by `ARS_OPENAI_COMPAT_BASE_URL` ; the model id must NOT match a first-party prefix or it takes the grounded first-party route instead. |
> **Compatible providers are ungrounded.** They expose no hosted web-search tool, so there is no grounding evidence behind a verdict. A positive `VERIFIED` is downgraded to `NOT_SEARCHED` and never counts as agreement in citation verification; a `NOT_FOUND`/`MISMATCH` survives as a disagreement. They ARE first-class for Devil's Advocate critique (which needs no grounding) — but a DA finding from any provider is an adversarial hypothesis, not standalone evidence, unless independently sourced.
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
**Recommended cross-verification pair: ** the inherited Claude session model (primary) + a current-generation second-family verifier — Gemini 3.1 Pro (validated) or GPT-6 Astra (provisional; see the note below). Users who want a measured OpenAI id can stay on GPT-5.6 Sol for the ChatGPT-subscription citation transport (validated there) or on GPT-5.5 for the API route.
2026-06-10 03:57:09 +08:00
> The primary row deliberately names no version: the primary is always the session model, so the row cannot go stale on the next Anthropic release. Verifier IDs stay concrete because they are literal API strings the user must export. (`gpt-5.4` / `gpt-5.4-pro` remain accepted for existing setups.)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
fix: Astra request compat, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823–#826) (#827)
* fix: Astra request compatibility, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823 #824 #825 #826)
#823 — OpenAI request builders (smoke entrypoint + documented example) drop
`temperature`, which GPT-6 Astra rejects; the per-model effort vocabulary
lives in scripts/cross_model_verification/openai_effort_guard.sh, sourced by
both, and an unsupported explicit Astra value fails before curl. Hermetic
fake-curl test runs both surfaces.
#824 — the contained Codex citation transport rejects effort=ultra with
REASONING_EFFORT_REQUIRES_DELEGATION before detection/auth/tempdir/launch on
both entry paths (codex-cli 0.153.4 defines ultra as the multiAgentMode
replacement). Model-independent by design.
#825 — hedging can no longer rescue an unsupported claim (writer recovery
tree, CER fallback row, temporal rule 5 in writer + both compiler mirrors,
writer contract D2); universal prose quotas in the writer, compilers,
writing_quality_check.md, academic-paper/SKILL.md, and contract D6 become
diagnostics subordinate to author/venue requirements. Audit inventory
corrected in place; held-out seed evals/heldout/unsupported_claim_recovery
(NOT_RUN) registered.
#826 — run_codex_audit.sh pins gpt-6-astra/xhigh and records both in a new
sidecar `model` block; claim_audit_pipeline binds an unknown judge identity
to a run-local cache key (no cross-run reuse) instead of defaulting to
gpt-5.5-xhigh.
Review: /simplify (4 angles), codex gpt-5.6-sol xhigh 2 rounds (r1: 1 P1 +
1 P2 + 2 P3 fixed; r2: 0 P1/P2), /security-review 0 findings; all 102
spec-consistency steps + pytest manifest replayed locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNKiXpdHx1T5F5RbXT2Ueu
* docs(claude): record the #824 ultra reversal in the v3.21.2 key-additions line
The v3.21.2 bullet still said the contained Codex citation transport accepts
ultra; #824 on this branch rejects it as a delegation request. Add the
reversal so the live instruction surface matches the transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7emV5r2aqZDJzAyYVuuDo
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 08:08:30 +09:00
> **GPT-6 Astra is provisional (listed 2026-09-05, two days after its 2026-09-03 release).** Its ARS-specific behavior on the first-party API route — the five Promotion Bakeoff measures below — is unvalidated, while its API effort vocabulary is documented (see § Reasoning effort below; API support is separate from ARS bakeoff validation). On the ChatGPT-subscription citation transport it passed the entry-gate smoke (`scripts/cross_model_smoke_test_codex.sh`, 2026-09-05, codex-cli 0.153.4: `VERIFIED` with a bound source on the Vaswani et al. fixture) — the precondition for a Promotion Bakeoff, not a bakeoff. Under the recommendation policy recorded in the GPT-5.6 Sol note below (#783) the recommendation moves to the current generation on lifecycle grounds; `validated` still requires the sealed bakeoff, on each transport separately. Two vendor-reported facts shape how ARS treats this verifier (GPT-6 Astra system card, 2026-09-03): provider-side misalignment and misuse monitoring can pause, end, or block a call (§ Provider-side monitoring and safety interventions below — never a verdict), and its verbalized evaluation awareness is high (§8.6, §8.8.1 — see the Promotion Bakeoff caveat).
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
> **GPT-5.6 Sol status (listed 2026-07-11, three days after release; superseded by GPT-6 Astra on 2026-09-03).** Its endpoint support (Responses API), hosted `web_search` tool, and reasoning-effort values are confirmed against OpenAI's model documentation, but its ARS-specific behavior — grounded-search completion rate, citation-mismatch recall, false-disagreement rate, response-shape stability against the jq grounding guards, p95 latency — is unvalidated. **Recommendation policy (2026-08-19):** GPT-5.5 was superseded by the GPT-5.6 family on 2026-07-09, so the recommendation names the current generation rather than a superseded id — a lifecycle decision, not a measurement claim. `validated` is earned only there — and on 2026-08-19 a codex-transport bakeoff run earned it for the **ChatGPT-subscription citation transport**, with a measured superiority case from the counterbalanced gate fleet (fabrication recall 0.90 vs 0.80, p95 latency 25.0 s vs 49.6 s nearest-rank, grounded completion tied, no inferiority on any measure; recall and latency led in all five paired fleets — `audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md`). On the **first-party API route** `gpt-5.6-sol` stays **provisional** — that run did not exercise the API route's jq grounding guards, and no parity or superiority is claimed there. For the API route, run `scripts/cross_model_smoke_test.sh` against your key before adopting it; users who prefer an API-route-validated id can stay on `gpt-5.5` or `gemini-3.1-pro-preview` (validated = the id-status allowlist below; the API route has no recorded bakeoff run). Two facts that differ from the GPT-5.5 lineup: GPT-5.6 ships **no `-pro` model ID** — premium operation is standard `gpt-5.6-sol` plus `reasoning: {mode: "pro"}` in the request, billed at standard token rates with more model work per request (the old fixed ~6× unit-price split does not carry over); and its reasoning effort accepts `none|low|medium|high|xhigh|max` (GPT-5.5 tops out at `xhigh`), defaulting to `medium` in both standard and pro modes.
2026-07-12 02:06:59 +08:00
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
Using two non-Anthropic models as primary+verifier is possible but not tested with ARS prompts.
## Setup Guide
### Prerequisites
You need API keys from at least one additional provider. ARS itself runs inside Claude Code, so Claude is always available as the primary model.
### Step 1: Get API Keys
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
**OpenAI (GPT-6 Astra / GPT-5.6 Sol / GPT-5.5): **
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
1. Go to [platform.openai.com/api-keys ](https://platform.openai.com/api-keys )
2. Create a new API key
3. Copy the key (starts with `sk-` )
**Google (Gemini 3.1 Pro): **
1. Go to [aistudio.google.com/apikey ](https://aistudio.google.com/apikey )
2. Create a new API key
3. Copy the key (starts with `AIza` )
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
**OpenAI-compatible providers (MiMo / DeepSeek / self-hosted): **
1. Get an API key from your provider (e.g. [platform.deepseek.com ](https://platform.deepseek.com ) or the Xiaomi MiMo platform)
2. Note the provider's API root including `/v1` (e.g. `https://api.deepseek.com/v1` )
3. The key goes in `ARS_OPENAI_COMPAT_API_KEY` and the endpoint in `ARS_OPENAI_COMPAT_BASE_URL` — NOT in `OPENAI_API_KEY` /`OPENAI_BASE_URL` (your real OpenAI key is never sent to a third-party endpoint)
4. The compatible model id (`ARS_CROSS_MODEL` ) must NOT begin with a `gpt-` or `gemini-` prefix. Any such id is claimed by the first-party grounded route, so a self-hosted compatible model named that way would be routed to the (unavailable) first-party path instead of your compatible endpoint.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
### Step 2: Set Environment Variables
Add to your shell profile (`~/.zshrc` or `~/.bashrc` ):
``` bash
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
# Cross-model verification for ARS — pick exactly ONE provider tuple.
# --- Option A: OpenAI (first-party, grounded) ---
2026-06-04 20:44:03 +08:00
export OPENAI_API_KEY = "<your-openai-api-key>"
2026-08-19 18:00:30 +08:00
# Current OpenAI flagship — provisional pending ARS validation (see Supported Models;
# run scripts/cross_model_smoke_test.sh against your key before relying on it):
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
export ARS_CROSS_MODEL = "gpt-6-astra"
# Previous generation — validated on the ChatGPT-subscription citation transport,
# provisional on this API route:
# export ARS_CROSS_MODEL="gpt-5.6-sol"
# Previous generation, validated (designated API-route bakeoff baseline):
2026-08-19 18:00:30 +08:00
# export ARS_CROSS_MODEL="gpt-5.5"
2026-07-12 02:06:59 +08:00
# Optional: reasoning effort for OpenAI verifier calls (unset = the provider's own
# default for the chosen model). GPT-5.6 accepts none|low|medium|high|xhigh|max;
fix: Astra request compat, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823–#826) (#827)
* fix: Astra request compatibility, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823 #824 #825 #826)
#823 — OpenAI request builders (smoke entrypoint + documented example) drop
`temperature`, which GPT-6 Astra rejects; the per-model effort vocabulary
lives in scripts/cross_model_verification/openai_effort_guard.sh, sourced by
both, and an unsupported explicit Astra value fails before curl. Hermetic
fake-curl test runs both surfaces.
#824 — the contained Codex citation transport rejects effort=ultra with
REASONING_EFFORT_REQUIRES_DELEGATION before detection/auth/tempdir/launch on
both entry paths (codex-cli 0.153.4 defines ultra as the multiAgentMode
replacement). Model-independent by design.
#825 — hedging can no longer rescue an unsupported claim (writer recovery
tree, CER fallback row, temporal rule 5 in writer + both compiler mirrors,
writer contract D2); universal prose quotas in the writer, compilers,
writing_quality_check.md, academic-paper/SKILL.md, and contract D6 become
diagnostics subordinate to author/venue requirements. Audit inventory
corrected in place; held-out seed evals/heldout/unsupported_claim_recovery
(NOT_RUN) registered.
#826 — run_codex_audit.sh pins gpt-6-astra/xhigh and records both in a new
sidecar `model` block; claim_audit_pipeline binds an unknown judge identity
to a run-local cache key (no cross-run reuse) instead of defaulting to
gpt-5.5-xhigh.
Review: /simplify (4 angles), codex gpt-5.6-sol xhigh 2 rounds (r1: 1 P1 +
1 P2 + 2 P3 fixed; r2: 0 P1/P2), /security-review 0 findings; all 102
spec-consistency steps + pytest manifest replayed locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNKiXpdHx1T5F5RbXT2Ueu
* docs(claude): record the #824 ultra reversal in the v3.21.2 key-additions line
The v3.21.2 bullet still said the contained Codex citation transport accepts
ultra; #824 on this branch rejects it as a delegation request. Add the
reversal so the live instruction surface matches the transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7emV5r2aqZDJzAyYVuuDo
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 08:08:30 +09:00
# GPT-5.5 tops out at xhigh; GPT-6 Astra accepts low|medium|high|xhigh|max
# (the contained Codex citation transport rejects ultra: it requests delegation).
2026-07-12 02:06:59 +08:00
# export ARS_CROSS_MODEL_REASONING_EFFORT="medium"
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
# --- Option B: Google Gemini (first-party, grounded) ---
2026-06-04 20:44:03 +08:00
export GOOGLE_AI_API_KEY = "<your-google-ai-api-key>"
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
export ARS_CROSS_MODEL = "gemini-3.1-pro-preview"
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
# --- Option C: OpenAI-compatible provider (MiMo / DeepSeek / self-hosted) — UNGROUNDED ---
# Uses a DEDICATED key; your real OPENAI_API_KEY is never sent to a third-party endpoint.
export ARS_OPENAI_COMPAT_BASE_URL = "https://api.deepseek.com/v1" # API root incl. /v1
export ARS_OPENAI_COMPAT_API_KEY = "<your-provider-api-key>"
export ARS_CROSS_MODEL = "deepseek-v4-pro" # provider id, NOT gpt-*/gemini-*
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
```
Then reload: `source ~/.zshrc`
2026-08-11 12:56:01 +08:00
### ChatGPT subscription option — citation integrity only
Users already authenticated to Codex with a ChatGPT subscription may select the
contained citation adapter without supplying `OPENAI_API_KEY` . This is not a
general API-key replacement and does not activate any non-citation cross-model
surface. Codex CLI 0.147.0 or newer is required; `codex login status` must return
exactly `Logged in using ChatGPT` . A custom `CODEX_HOME` is honored consistently
by detection and execution.
``` bash
# Citation-integrity calls only. General DA/reviewer/judgment calls remain on API transport.
export ARS_CROSS_MODEL_TRANSPORT = "codex"
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
# gpt-6-astra: current OpenAI flagship — provisional on this transport (entry-gate
# smoke PASS 2026-09-05 on codex-cli 0.153.4; no bakeoff run yet).
export ARS_CROSS_MODEL = "gpt-6-astra"
feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#788)
* feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#787)
Probe set: 30 refs (10 easy DOI-keyed journal articles; 10 hard: 3 arXiv,
2 DOI-less NeurIPS, 5 non-English; 10 fabrications), every real row
resolver-confirmed same-day, every fabrication negative-checked. 180
same-day paired calls (30 x 3 repeats x 2 models), majority verdicts.
Result: all five measures PASS with superiority — recall 1.00 vs 0.80,
grounded completion 0.933 vs 0.900, p95 latency 26.5s vs 58.7s, zero
guard misfires, false disagreement 0.00 = 0.00. Transport-qualified:
gpt-5.6-sol stays provisional on the first-party API route (jq guards
unexercised; allowlist unchanged). Report + probe-set sha256 under
audits/; per-call index committed beside the probe set.
Campaign side-product (transport): page-open webSearch items
(action.type != "search") are skipped for binding instead of failing the
stream (opened-page URLs still can never become bound sources), and
DEVELOPER_INSTRUCTIONS requires an empty sources array for
NOT_FOUND/NOT_SEARCHED. 52 transport tests green. Defective-tool run 1
archived unscored; three probe-row transcription errors were flagged
MISMATCH by both models, independently re-verified, corrected, re-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: narrow the page-open exemption to the observed action.type == "other" shape (#788 codex P2)
An empty action object, unknown action type, or non-dict action on a
webSearch item is stream-fatal again; only the observed page-open shape
is skipped. Mutation test sweeps four bad shapes (52 -> 53 tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: anchor the page-open exemption to the first-party closed WebSearchAction set
Run-3 surfaced a third real shape ({"type": "openPage", "url": ...}) that
the single-observation exemption rejected, tool-suppressing the baseline's
measures (13 EVENT_STREAM_INVALID cells). The exempt set is now exactly
the non-search members of the app-server protocol's closed WebSearchAction
oneOf — {other, openPage, findInPage} plus the Responses-API spellings —
verified against `codex app-server generate-json-schema` on 0.147.0.
Unknown shapes stay stream-fatal (mutation sweep unchanged); 54 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: score preregistered run 4 as the gate result; runs 1-3 recorded as exploratory
Run 4 (frozen fixture @ 3fc6ddb, frozen parser @ c9c865d, both pushed
pre-run): all five measures PASS, zero misfires on BOTH models, recall
1.00 vs 0.80, grounded completion 0.933 vs 0.867, p95 27.5s vs 51.1s.
Report rewritten with the preregistration statement and the full
exploratory-round accounting; call index replaced with run-4 data;
claim surfaces and CHANGELOG updated to run-4 numbers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs(code): pin the bare-discriminator decision against the first-party schema (#788 codex round-2 P2 rejected with evidence)
The round-2 finding claimed openPage/findInPage require url/pattern; the
protocol schema (generate-json-schema, 0.147.0) marks every non-search
variant required:["type"] with url/pattern nullable optionals. Demanding
optional fields is the exact false-fatality class that invalidated
bakeoff runs 1 and 3. Decision recorded in the comment and pinned by
bare-discriminator test rows (54 tests, +2 param rows).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-3 findings — ordering, replayability, exposure analysis
P2 (ordering): webSearch action-shape validation now runs BEFORE the
MODEL_RETURNED_NOT_SEARCHED early return, so a model NOT_SEARCHED verdict
can never mask response-shape drift; mutation test added (55 tests).
P2 (replayability): the 180 full receipt rows, the offline scorer
(verified to reproduce the gate byte-for-byte from committed artifacts
alone), and the parameterized fleet runner are committed beside the
probe set; the report states the replayability boundary plainly (raw
event streams are digest-only by transport design).
P1 (answer-key exposure): empirical scan across all 540 retained
receipts finds zero repo-referencing bound queries/sources; report gains
an exposure-analysis section with scope caveats and corroboration; the
structural fix (sealed hash-commit preregistration, fresh fabrication
pool per run) is filed as #789 for future bakeoffs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-4 P2s — fleet gate, timeout margin, fresh-probe rule, nearest-rank p95
Scorer refuses truncated/duplicated/partial fleets (exactly one row per
(ref_id, repeat) across 30x3) before computing any measure; p95 moves to
the nearest-rank order statistic (51.13/27.46 -> 51.20/28.09, matching
the review's own recomputation; gate unchanged) and the method is named
on every surface. Runner outer timeout raised above the transport's
inner 300s deadline so its finally-block cleanup always fires first.
The canonical recorded-run note and report outcome now require a FRESH
probe set for the API-route run per #789 (this set's labels are public),
resolving the self-contradiction with the exposure analysis.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-5 P2s — scorer consumes runner output + binds receipt identity
score_run.py now scores either the committed run-4 JSONLs (default) or a
fleet-runner output directory (argv[1]), so a reproduced fleet can never
silently re-report the old result; every scored row must pass identity
binding (outer model/ref/repeat, receipt.model, receipt.request_id, and
a request_digest recomputed from the probe set via the transport's
canonical form), refusing mis-associated or edited fleets. Verified:
committed data reproduces the gate unchanged, results-dir mode scores
the live run-4 cells, and a cross-model receipt swap is refused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-6 P2s — dual-fleet hard-zero + unambiguous shape code
Measure 4 now requires zero guard misfires in BOTH fleets (a baseline
suppressed by tool misfires cannot anchor a fair comparison — the run-3
lesson, now enforced by the scorer). The transport emits
EVENT_STREAM_INVALID for a non-null non-list search `results` value
instead of silently skipping into NO_BOUND_SEARCH_RESULTS
(wrong_search_shape fixture expectation updated in lockstep), and the
scorer's shape family is trimmed to exactly the emitted shape codes with
the behavior-family classification documented. Provably no effect on the
scored run: run-4 contains zero rows in any affected code family (only
SOURCE_NOT_IN_SEARCH_RESULTS 12/6, a behavior code) and both fleets
already sit at zero misfires; all gate numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-7 P2s — results-shape check before verdict return + same-day fleet enforcement
The results-shape validation joins the pre-verdict scan loop so a model
NOT_SEARCHED verdict can never mask dict-shaped search results (mutation
test added: wrong_search_shape + NOT_SEARCHED -> EVENT_STREAM_INVALID;
56 tests). The runner refuses to resume over cells from an earlier date,
and the scorer refuses mixed-date fleets across both models (run-4 is
single-date; gate numbers unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-8 P2s — complete pre-verdict search validation + runner failure semantics
The pre-verdict scan now performs the COMPLETE search-item strict
validation (query type/length/control chars + results shape), covering
legacy action-less items — the round-7 placement validated only
action-typed items, which also made the round-7 mutation test fail (a
red test my verification pipeline masked via tail; committed here only
with PYTEST_EXIT=0 verified directly). Runner: a fleet with any failed
call now exits nonzero instead of printing ALL DONE, and an outer-
timeout kill sweeps the adapter's orphaned temp dirs (the detached
app-server exits on stdin EOF; the ephemeral auth copy is what the
verifier's skipped finally-block would have removed). 56 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-9 P2s — NOT_SEARCHED source contract, retry-not-skip, quiescent sweep, dated rows
Transport: NOT_SEARCHED with a populated sources array fails closed as
FINAL_OUTPUT_INVALID before the early return (mutation test; 57 tests).
Runner: a same-day cell that recorded a failure is discarded and retried
on resume instead of silently counting as complete; the orphan-tempdir
sweep runs only after the executor drains so it can never delete a live
worker's ephemeral CODEX_HOME. Scorer: every row must carry a real ISO
date — an undated fleet cannot satisfy the same-day gate on empty
strings. Committed run-4 data re-verified green under all new gates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-10 findings — single-path validation, scorer gate exit, probe-hash pin
P1 root treatment: the entire search-processing pipeline (cap, strict
per-item validation, reference-bound filter, URL binding incl. the
result-entry object-shape check) now runs BEFORE any verdict branch, so
every shape-fatal path fires identically regardless of the model's
answer — the verdict-masking bug class (rounds 3/7/8/9/10) is closed by
construction, not by another patch. Emptiness outcomes stay verdict-
conditional (an honest NOT_SEARCHED with no bound search remains model
behavior). Mutation test: bound search with a non-object result entry +
NOT_SEARCHED verdict -> EVENT_STREAM_INVALID (58 tests).
Scorer: refuses a probe set whose whole-file sha256 differs from the
frozen hash (labels now inside the scoring identity), and exits nonzero
when any gate fails. The P1's rerun demand is accepted: a run-5 fleet
under this frozen parser follows as the scored gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: run 5 under the frozen final instrument is the gate result (#788 codex round-10 P1 accepted)
Fleet rerun 2026-08-20 under parser+scorer db6ed67 (pushed pre-run):
all five measures PASS with zero misfires on both fleets — recall 0.90
vs 0.80, grounded completion 0.911 vs 0.889, p95 26.1s vs 47.6s. Run 4
reclassified as a prior-instrument exploratory round; committed
receipts/index/scorer default swapped to run-5 data (committed scorer
replays the gate from repo artifacts alone, exit 0); all claim surfaces
carry run-5 numbers and the cross-fleet consistency note (candidate led
measures 1/2/5 in every full paired fleet).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-11 P1s — no failed-trial erasure + full-fleet entry validation (instrument fixpoint)
Runner: a recorded failed trial is never deleted on resume; it is
carried into the failure count and forces a nonzero exit, so the only
path past a failure is rerunning the ENTIRE fleet fresh — selective
retry-until-green is structurally impossible. (Provably no scored fleet
was affected: runs 4 and 5 each completed in a single invocation with
zero failures and no retry/carried lines in their logs.)
Transport: the strict pre-verdict loop now validates every consumed
field of EVERY search item — id, query, results-list shape, and each
entry's object shape, bound or unbound — reaching the instrument
fixpoint: no field the pipeline reads is unvalidated, so no future
verdict-masking variant of this class exists. Mutation tests for
unbound-malformed-entries and id shapes (60 tests). A run-6 fleet under
this frozen instrument follows as the gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: run 6 under the fixpoint instrument is the gate result
Fleet rerun 2026-08-20 under adf18f9 (parser fixpoint + no-erasure
runner + full-gate scorer, all pushed pre-run): 5/5 PASS, zero misfires
both fleets — recall 1.00 vs 0.70, grounded completion 0.911 vs 0.867,
p95 28.8s vs 43.3s. Run 5 reclassified prior-instrument; artifacts and
scorer default swapped to run-6; leak scan clean across all 900 retained
receipts (runs 2-6); candidate led measures 1/2/5 in all four full
paired fleets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-12 P2s — pinned effort, fleet-private temp root, split disclosure
Runner pins ARS_CROSS_MODEL_REASONING_EFFORT to the provider default
(explicitly unset per call, recorded per row) and routes all transport
temp dirs into a fleet-private mkdtemp root so the timeout sweep can
never touch another invocation's dirs. The audit now discloses the one
run-6 1-1-1 split (baseline fab-05: MISMATCH/NOT_SEARCHED/NOT_FOUND ->
INDETERMINATE, conservative miss) and names the actual baseline misses
(fab-01, fab-08 majority NOT_SEARCHED; fab-05 split) — verified against
the committed receipts, correcting a stale carried-over sentence. The
effort variable was verified unset for every fleet (shell env + profile
carry no export); gate numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-13 P2s — receipt-contract validation, effort-marker gates, always-sweep
Scorer validates every non-null receipt against the closed contract
(required keys, verdict/searched types, positive-verdict grounding with
fully-bound sources, empty sources on NOT_FOUND/NOT_SEARCHED, queries
present when searched) before any metric trusts it, and results-dir
scoring requires the pinned-effort marker on every row (committed
gate-run rows predate the marker; the audit attests their configuration).
Runner refuses carried cells without the marker and sweeps the
fleet-private temp root in a finally-block on every outcome — a
signal/OOM-killed verifier no longer leaves its ephemeral auth copy.
Committed gate scoring still exits 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-14 P2s — per-verdict receipt invariants + runner rejects malformed parsed receipts
Scorer: grounded verdicts (VERIFIED/MISMATCH/NOT_FOUND) require
searched=true and a null reason_code; NOT_SEARCHED requires
searched=false and a reason from the transport's closed emitted set —
a fabricated NOT_FOUND-without-search or NOT_SEARCHED-with-search row
can no longer contribute to recall or completion. Runner: a verifier
exiting 0 with parsed-but-malformed output records RECEIPT_INVALID,
counts as a failure, and forces nonzero exit. Committed run-6 gate
scoring re-verified: exit 0, numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-15 P2s — uniform item-field validation + shared receipt contract (axis terminal)
Transport: every webSearch item — page-opens included — now has its
action payload validated against the closed WebSearchAction variant
types (url/pattern/query string-or-null, queries string-array), plus
uniform id and results/entry shape checks; a recognized discriminator
with a wrong-typed payload is stream-fatal (mutation sweep; 60 tests).
Tooling: the full closed receipt contract (exact key set, transport/
auth_mode/containment, digest formats, per-verdict cross-field
invariants) moves into a shared receipt_contract.py imported by BOTH
run_fleet.py and score_run.py — one implementation, applied to fresh
cells, resumed cells, and every scored row, so the two consumers cannot
diverge. This terminates the validation axis: every field of every
webSearch item and every key of every receipt is now checked; committed
run-6 gate scoring re-verified exit 0 with numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-16 — strict source bindings; instrument-freeze boundary pinned (P1 declined with recorded rationale)
receipt_contract.py enforces the full canonical source-binding shape
(closed 4-key object, non-trivial https URL, non-empty item id,
result_index 0-127 with bool exclusion) — committed run-6 data passes
unchanged. The round-16 rerun demand is DECLINED under a pinned
maintainer boundary, recorded in the report and the canonical
recorded-run note: runs 4/5 were discarded because consumed-data gaps
could alter scored outcomes; post-run-6 hardening validates only
surfaces outside every consumed path and cannot change any verdict,
binding, latency, or measure of a past fleet — such hardening applies
from the next fleet. The disagreement is recorded, not hidden.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-17 P2s — complete schema mirror + latency-sample validation
receipt_contract.py is now a COMPLETE stdlib mirror of the canonical
receipt schema: identifier/event-id/https-url patterns and length
bounds, array caps (queries<=32, sources<=16), closed entry objects,
auth_mode const, NOT_SEARCHED => empty queries+sources with a mandatory
reason, unknown reason codes refused globally. Scorer refuses boolean,
negative, non-numeric, or absurd wall_seconds before the percentile
gate. Committed run-6 gate scoring re-verified: exit 0, numbers
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-18 P2s — hashability, boolean identity, SIGTERM cleanup
Validator: array/object verdict or reason_code becomes a contract
failure instead of an uncaught TypeError (which would have escaped the
runner's SystemExit handling and re-opened the no-reroll gap);
containment flags are checked by identity (`is True`) so integer 1
cannot satisfy the schema's boolean constants. Runner: SIGTERM/SIGINT
raise SystemExit so the finally-block sweep of the fleet-private auth
copies also runs on cancellation. Mutation-verified (3/3 caught);
committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-19 P2s — discriminator hashability, integer repeats, real ISO timestamps
Transport: the WebSearchAction discriminator is type-checked before set
membership in both _is_page_open and the uniform loop — an array/object
type fails closed as EVENT_STREAM_INVALID instead of crashing the
verifier past shape accounting. Scorer: repeat must be an exact int in
1..3 (1.0 satisfied the completeness Counter while minting ...-r1.0),
and ts must parse as a full ISO timestamp with offset instead of a
digit-shaped prefix. Committed run-6 scoring exit 0 unchanged; 60
transport tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-20 — source-query binding, midnight guard; null-action P2 declined with schema evidence
receipt_contract.py rejects sources whose search_item_id has no retained
entry in search_queries (unretained evidence never counts as grounding).
run_fleet.py fails visibly BEFORE reporting success when a fresh fleet's
cells span two calendar dates. The explicit-null-action P2 is declined
on first-party evidence: ThreadItem types action as
anyOf[WebSearchAction, null] (generate-json-schema, 0.147.0), so null is
protocol-legal and follows the legacy path where the item still faces
the complete strict validation — fatal-izing it is the run-1/run-3
false-fatality class; decision pinned in the code comment. Run-6
scoring exit 0 unchanged; 60 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-21 — cancellation stops queued quota burn, worker failures persist; open-variant P2 declined with schema evidence
P1: SIGTERM/SIGINT set a STOP event making every queued job a no-op
(marked [CANCELLED], counted as failure), so shutdown waits only for
the at-most-3 in-flight calls instead of burning the rest of a paid
180-call fleet. P2: a worker exception after the paid call persists a
failed cell with the job identity, so a resume can never treat the
consumed trial as missing and re-roll it. The closed-variant P2 is
declined on first-party evidence: no WebSearchAction variant sets
additionalProperties, so extra fields are schema-legal and rejecting
them would make any future informational field fleet-fatal; decision
pinned in the code comment, known fields stay type-checked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-22 P1 — stray STOP=None placeholder no longer nullifies the cancellation event
The round-21 placeholder assignment landed AFTER the Event creation in
module order, resetting STOP to None and disabling queued-call
cancellation exactly as the review read it. The placeholder is removed;
the Event created before worker start is the one the signal handler
sets. Static check pins that no STOP=None assignment remains.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-23 P2 — refuse contradictory receipt+error rows
A row carrying both a valid receipt and a truthy error is structurally
impossible from the runner and is refused as corrupted/external instead
of being scored as grounded evidence; error rows with a null receipt
stay counted as misfires. Committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-24 P2s — output-contract precedence, portable probe digest, runner preflight
Transport: NOT_FOUND carrying sources is FINAL_OUTPUT_INVALID even when
the stream also lacks a bound search — output-contract violations now
outrank emptiness outcomes so the shape event cannot be misfiled as a
behavior code. Probe digest verification moves into the shared module
with CRLF->LF normalization (a Windows autocrlf checkout is not probe
drift) and the runner runs the same preflight BEFORE any paid call, so
180 subscription calls can never be spent on a fixture the scorer will
refuse. Run-6 scoring exit 0 unchanged; 60 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-25 — scorer-equivalent resume preflight + exclusive fleet lock
validate_row (shared receipt_contract) now carries the COMPLETE row
validation — contradiction, outer identity, exact-integer repeat, full
ISO timestamp, sane latency, receipt identity binding to the probe row,
closed receipt contract — and is the single implementation used by both
the scorer and the runner's resume preflight, so a misnamed or copied
cell fails before any further quota is spent. The runner takes an
exclusive flock on the output dir, refusing a second concurrent
invocation that would duplicate paid calls and race cell writes.
Committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-26 — URL-binding drift hits measure 4; cross-platform fleet lock
Transport: a bound search whose non-empty result entries yield no
extractable URL is EVENT_STREAM_INVALID (provider moved/renamed the URL
key = response-shape drift), no longer the behavioral
NO_BOUND_SEARCH_RESULTS; the pre-existing pin of the old classification
is updated in lockstep and a canonical_url regression test added (62
tests). Runner: the fleet lock falls back to msvcrt.locking on Windows,
keeping the documented reproduction path viable.
Note: run-6's receipts contain zero NO_BOUND_SEARCH_RESULTS /
SOURCE_NOT-with-empty-binding rows of the reclassified kind (reason
distribution: only SOURCE_NOT_IN_SEARCH_RESULTS with non-empty
bindings and clean rows), so the gate numbers are provably unaffected;
the change also falls under the pinned instrument-freeze boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-27 P2s — key-drift/value split, full resume validation, Windows invoke, UTF-8 I/O
Transport: EVENT_STREAM_INVALID for empty bindings now requires that NO
recognized URL key exists across the bound entries (true key drift); a
recognized key with an unusable value stays behavioral — both sides
test-pinned (62 tests). Runner: every resumed cell, failed ones
included, faces validate_row + the effort check before further quota is
spent; the transport is invoked via sys.executable (Windows honors no
shebang); all subprocess/artifact text I/O pinned to strict UTF-8 in
runner and scorer. Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-28 P2 — URL-key drift determined pre-verdict
The key-drift determination (bound entries present, no recognized URL
key anywhere) moves before the NOT_SEARCHED early return, so a model
NOT_SEARCHED answer can no longer mask renamed-URL-key response drift;
the post-verdict emptiness branch keeps only behavioral outcomes.
Masking regression test added (63 tests); run-6 scoring exit 0
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-29 P1 — fleet runner gated to POSIX
The #630 transport's process-group containment (start_new_session +
os.killpg in _stop_process) is POSIX-only, so a native-Windows fleet
would consume paid calls while every cell fails during cleanup — the
rounds-26/27 surface accommodations implied support the deeper stack
never had. The runner now refuses non-POSIX up front with a WSL
pointer; the dead msvcrt lock branch is removed (the scorer, which is
genuinely offline and portable, keeps its CRLF-tolerant digest and
UTF-8 reads). Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-30 — receipt-level invariance proof replaces live-validation claim; session isolation; stray-file preflight
P1 resolved by PROOF rather than a third rerun: run 6's 180 receipts
carry exactly two reason states (null; SOURCE_NOT_IN_SEARCH_RESULTS
12/8) with zero receipt-less, error, NO_BOUND, NO_REFERENCE,
FINAL_OUTPUT_INVALID, MODEL_RETURNED_NOT_SEARCHED, or
EVENT_STREAM_INVALID rows — each post-run-6 transport change either
touches unconsumed surfaces or only relabels cells in code families
that provably never occurred, so no run-6 cell can differ under the
shipped parser. The stale "run 6 live-validates the shipped parser" and
"instrument FIXPOINT / no masking path remains" sentences are replaced
with the precise provable statements; the freeze policy now REQUIRES
this proof standard (no proof on a consumed path = rerun, as runs 4/5
were). P2s: verifier subprocesses start in their own session so an
interactive Ctrl-C cannot turn in-flight calls into resume-poisoning
EXIT failures; the runner refuses unexpected result files before
spending quota.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-31 P2s — orphaned .tmp cells refused, fresh cells identity-bound
The preflight refuses orphaned atomic-write temp files (a crash between
write and rename must not silently re-roll a completed paid trial), and
fresh cells face the same validate_row identity binding as resumed
cells and the scorer before being persisted as success — a receipt with
the wrong model/request_id/digest becomes a recorded RECEIPT_INVALID
failure. Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* feat: counterbalanced interleaved scheduling for the bakeoff fleet (#788 round-32 P1 accepted)
The two models' calls for each (reference, repeat) cell are adjacent in
the queue with deterministic parity-alternating pair order, so model
identity is decorrelated from execution time — provider load or
web-search drift during the fleet can no longer masquerade as a model
effect. A counterbalanced run-7 follows as the scored gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: counterbalanced run 7 is the gate result; run 6 superseded for order confound
Run 7 under frozen instrument 69cd04a (interleaved parity-alternating
pair scheduling): all five measures PASS — recall 0.90 vs 0.80, p95
25.0s vs 49.6s (median 14.8 vs 17.5), grounded completion tied at
0.900, zero misfires both fleets, two 1-1-1 splits disclosed and scored
as conservative misses. Honesty note carried on every claim surface:
the sequential fleets' grounded-completion edge did NOT survive
counterbalancing and is not claimed; recall and latency led in all five
paired fleets. Artifacts and scorer default swapped to run-7 (committed
scorer replays the gate, exit 0); leak scan clean across 1,080 retained
receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-33 P1 — resume refuses half-complete counterbalanced pairs
Every (reference, repeat) pair must be wholly present or wholly missing
on resume: a one-sided pair would run the counterpart far from its
partner and silently reintroduce the model-vs-time confound. Run 7 is
unaffected (single uninterrupted invocation); scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: close codex round-34 P2 — recorded-run note limited to the measured superiority (2 and 5, tie on 1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 07:13:23 +08:00
# gpt-5.6-sol is validated for THIS transport (2026-08-19 codex-transport bakeoff,
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
# superiority on recall + latency — audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md):
# export ARS_CROSS_MODEL="gpt-5.6-sol"
2026-08-11 12:56:01 +08:00
python3 scripts/cross_model_codex_transport.py detect
# The producer sends one closed codex_citation_request/1.0 object on stdin:
printf '%s' " $CITATION_REQUEST_JSON " | scripts/cross_model_codex_verify.sh
```
The adapter launches an ephemeral app-server thread from an empty working root,
copies only `auth.json` into a private temporary `CODEX_HOME` , strips credential
environment variables, sets read-only sandboxing with approvals disabled, exposes
no dynamic tools or workspace roots, and disables local execution/file/app/plugin/
skill/browser/agent capabilities. It accepts only the closed request schema and
emits only the closed receipt schema under `shared/contracts/cross_model/` ; it
never accepts a path or caller-authored prompt. Authentication detection reports
the mode only—credentials never enter the receipt or diagnostic output.
Grounding authority is the app-server `webSearch.results[]` event, not the model's
text and not a URL echoed from the request. Every positive source receipt binds an
exact canonical HTTPS URL to the search item id, result index, and canonical result
digest. A reference-bound search with no matching work may yield `NOT_FOUND` ; a
missing/malformed search, unrelated query, unbound source, multiple final answer,
forbidden tool event, or wrong event shape fails closed to `NOT_SEARCHED` . The
ordinary `codex exec --json` projection is intentionally not used because it does
not preserve the structured standalone-search results needed for that binding.
`scripts/cross_model_smoke_test_codex.sh` is a manual live smoke using public
Vaswani et al. citation metadata. It consumes subscription/model/network capacity,
so CI never invokes it; run it only after consenting to that live call. CI uses a
fake Codex app-server and checked-in event fixtures exclusively.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
### Step 3: Verify Setup
In Claude Code, you can test by asking:
```
Check if cross-model verification is available for ARS
```
The system will check for the environment variables and report which models are available.
### Step 4: Enable Per-Session (Optional)
If you don't want cross-model verification running all the time, you can enable it per session:
``` bash
# Enable for this session only
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
export ARS_CROSS_MODEL = "gpt-6-astra"
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
# Disable for this session
unset ARS_CROSS_MODEL
```
## How It Works in Each Skill
### Integrity Verification (academic-pipeline, Stage 2.5 / 4.5)
**When `ARS_CROSS_MODEL` is set: **
- Primary model (Claude) runs full Phase A-E verification as normal
2026-08-15 15:04:28 +08:00
- After Phase A completes, a **risk-stratified ** selection of references is sent for a blind cross-model verification pass (see step 2 below; replaces the pre-#518 uniform random 30%)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
- Cross-model receives only the reference text and paper context — not Claude's verification result (to prevent anchoring)
- Disagreements are flagged as `[CROSS-MODEL-DISAGREEMENT]` and prioritized for human review
**When `ARS_CROSS_MODEL` is not set: **
- Standard single-model verification (unchanged from v2.7+)
**Implementation for agents: **
When the integrity_verification_agent detects `ARS_CROSS_MODEL` in the environment, it should:
1. Complete Phase A verification normally
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
2. Select references by **risk stratification ** (#518 ; replaces uniform random 30%). Classify each reference at selection time and record the tier in the results table. Four tiers; a reference qualifying for more than one is classified once at the highest tier that applies (precedence: `HIGH-IMPACT` > `NEW-CHANGED` > `CONTROL` /`RANDOM` ) and verified once:
- **HIGH-IMPACT — verify 100%, no cap (both gates).** A reference is high-impact if it supports any of: (a) a headline conclusion (abstract- or conclusions-level claim); (b) a numerical claim (statistic, effect size, percentage, threshold); (c) a causal claim; (d) a methods-critical claim (the validity of the chosen method rests on it); (e) a disputed claim (already carrying a contradiction disclosure or reviewer split).
- **RANDOM (Stage 2.5 only) — the non-high-impact remainder**, sampled at 10%, rounded up (minimum 3, maximum 10; if the remainder has fewer than 3 references, sample all of it).
- **NEW-CHANGED (Stage 4.5 only) — verify 100%, no cap:** every reference supporting a claim that is **new or changed ** since Stage 2.5, whatever its impact class.
- **CONTROL (Stage 4.5 only) — the unchanged, non-high-impact remainder**, sampled at 10%, rounded up (minimum 3, maximum 10; fewer than 3 → all of it) to catch silent drift. At Stage 4.5, CONTROL replaces RANDOM — there is no separate RANDOM tier at the final gate.
- Cost scales with the count of high-impact (and, at Stage 4.5, new/changed) citations instead of total reference count — a results-dense paper approaches 100% coverage, which is the point: verification budget concentrates where the paper's weight rests. The old flat cap (max 15) is retired; only the sampled tiers (RANDOM/CONTROL) carry a cap (max 10 each).
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
3. Issue **one API call per reference ** — not a batch. (Batching hides which reference the model actually grounded: a single grounding-metadata trace on a 5-reference response proves * something * was searched, not that * each * reference was. One reference per call makes the grounding evidence 1:1 with the verdict.) For each reference, construct a verification prompt:
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
```
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
Verify this academic reference. Check: Does it exist? Are the author
names, year, title, journal, and DOI correct? Search the web to
confirm — do not answer from memory.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
Respond with exactly one verdict:
- VERIFIED — found online; include at least one source URL or DOI you found
- MISMATCH — found, but a field is wrong (state which); include the source
- NOT_FOUND — searched, no matching record exists
- NOT_SEARCHED — you could not actually search the web for this reference
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
Reference: [full reference text] — Context: [sentence where cited]
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
` ``
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
A ` VERIFIED` verdict with no accompanying source URL/DOI is treated as ` NOT_SEARCHED` (the model claimed a result it cannot evidence).
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
4. Send to the cross-model via the appropriate API (see API Call Patterns below). **For first-party providers the call patterns enable the hosted web-search/grounding tool and reject the response as ` NOT_SEARCHED` when the API returns no grounding evidence** — a model that ignores the "search the web" instruction cannot fake an absent grounding trace, so this is the real safety boundary, not the prompt wording. **An OpenAI-compatible provider has no grounding tool, so its positive verdicts are downgraded to ` NOT_SEARCHED` by the handler (rejections pass through); a compatible provider therefore never contributes a grounded agreement.**
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
5. Compare results: if Claude said VERIFIED but cross-model said NOT_FOUND or MISMATCH, flag as ` [CROSS-MODEL-DISAGREEMENT]`. Treat ` NOT_SEARCHED` / ungrounded exactly as **not verified** — it never counts as agreement with a Claude ` VERIFIED`, and a sample that returns ` NOT_SEARCHED` is surfaced for re-run or human review, never silently passed.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
6. Include disagreements in the integrity report under a new section:
` ``markdown
### Cross-Model Verification Results
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
- References selected: X/Y (Z%) — HIGH-IMPACT: H (100% of tier), RANDOM: R (Stage 2.5), NEW-CHANGED: N + CONTROL: C (Stage 4.5)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
- Agreements: N
- Disagreements: M (listed below, prioritized for human review)
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
- Ungrounded (NOT_SEARCHED): U (the cross-model could not actually search — these are NOT confirmations; re-run or human-review)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
| # | Reference | Tier | Claude | Cross-Model | Source (URL/DOI) | Status |
|---|-----------|------|--------|-------------|------------------|--------|
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
` ``
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
The ` Tier` column is ` HIGH-IMPACT` / ` RANDOM` / ` NEW-CHANGED` / ` CONTROL` per step 2 (one tier per reference, highest-precedence tier wins). The ` Source` column carries the URL/DOI the cross-model returned for a ` VERIFIED` row; a blank source on a ` VERIFIED` verdict downgrades it to ` NOT_SEARCHED`.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
### Devil's Advocate (deep-research + academic-paper-reviewer)
**When ` ARS_CROSS_MODEL` is set:**
2026-08-15 15:04:28 +08:00
- After the DA completes its standard review/checkpoint, the cross-model receives the same material without the DA findings and generates a blind critique
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
- The DA then compares: any CRITICAL or MAJOR issues found by the cross-model but not by the DA are added as ` [CROSS-MODEL-FINDING]`
- This directly addresses frame-lock — a different model may attack from a different angle
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
> A compatible (ungrounded) provider is first-class for DA critique — surfacing weaknesses and attack angles needs no web grounding. But "first-class" is scoped to critique, not factual adjudication: a DA finding from any provider is an adversarial hypothesis, never standalone evidence, unless it carries an independently-checkable source. Do not treat a compatible-provider DA "finding" as a verified defect.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
**When ` ARS_CROSS_MODEL` is not set:**
- Standard single-model DA (unchanged)
**Implementation:**
The DA agent, after completing its checkpoint report, should:
1. Send the reviewed material + a simplified DA prompt to the cross-model:
` ``
You are a devil's advocate reviewing this [research/paper].
feat(reviewer): #574 behavior batch — quota removal + coverage receipts, typed evidence anchors, severity transport, decision symmetry (#581)
A1: remove 3-5/2-3 finding quotas across EIC/R1/R2/R3 + template; empty
finding lists require a Coverage Receipt. A2: six-type evidence-anchor
vocabulary (text/table/figure/equation/dataset/absence), Critical/Major
findings must carry an adequate applicable anchor; DA tables gain typed
Evidence Anchor (+Confidence) columns. A3: Schema 6 severity enum becomes
the declared single source with per-finding Severity/Confidence/competence
basis on every scoring seat; synthesizer transports severity/confidence
with [SEVERITY-SOURCE]/[CONFIDENCE-SOURCE] fallback tags; Serious/Moderate
parallel vocabulary folded into the enum. B1: acceptance-rate/base-rate
anchors removed (EIC Q1 ~10-15%, "Rare — most papers", "< 5%"); Decision
Symmetry and Register Independence section added. Also closes the #578
P0-3 residue in SKILL.md ("no duplicate criticisms").
New fail-closed lints: check_reviewer_data_fences.py (PR #578
paper_content/phase1_output fences across 5 agents + protocol delivery +
cross-model transport) and check_reviewer_finding_contract.py (all
behavior-batch surfaces), 30 mutation tests, CI + unified pytest manifest
wired. test_check_215_field_norm.py DA-table literal re-baselined to the
new header (companion-fixture-same-commit convention).
Gated on the #574 E4 baseline (307ef24: MS01 0.90 / MS02 1.00 recall,
clean-control 0, severity agreement 0.625); post-change E4 re-measurement
per evals/heldout/reviewer_seeded_defects/README.md is the acceptance step.
Claude-Session: https://claude.ai/code/session_01EUq9orZoNHtHrBZgYSXFaH
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 01:09:11 +08:00
Find the most serious weaknesses — every one the evidence supports,
ranked most severe first; no fixed count, and do not pad to reach one
(#574 A1). For each, state:
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
- What the weakness is
- Why it matters
- What the strongest counter-argument would be
Material: [the reviewed content]
` ``
2. Compare cross-model findings with own findings
3. Any cross-model finding not already covered → add to report as ` [CROSS-MODEL-FINDING]`
4. Log: ` [CROSS-MODEL: X findings received, Y novel (not in primary DA report)]`
2026-07-18 14:38:26 +08:00
### Cross-Model Reviewer Track (#540 — academic-paper-reviewer full mode)
**Activation — consent, not configuration:** the track activates only inside the same consent boundary as every cross-model feature in this document (the manuscript is uploaded to the external provider): ` ARS_CROSS_MODEL` being set is configuration, and the user's explicit cross-model consent for the session is the authorization. Configured-but-unconsented runs behave exactly like the not-set case below.
**When active:**
- ONE existing peer-reviewer slot (Reviewer 2 by default) runs on the cross-model family instead of the session model. The panel stays FIVE seats — this is a substrate swap inside a fixed slot, NOT the retired "6th reviewer" (see the retirement note above: its five counterproductive conditions — score averaging, role duplication, findings-as-confirmed-defects, majority-vote false confidence, synthesizer context burn — all attach to an ADDED generic seat; none applies to swapping the substrate of an existing persona with an unchanged role and an unchanged vote).
fix(reviewer): #574 PR 1 — contradiction batch + manuscript data-fence + no-invention rule + Top Blocking Issues + overlap-prohibition removal (#578)
* fix(reviewer): #574 PR 1 — contradiction batch (C1/C2/C4/C6) + manuscript data-fence (A6) + no-invention rule (A5) + Top Blocking Issues (E7) + overlap-prohibition removal (P0-3)
Seven prompt-level defects, none changing review behavioral direction
(behavior batch waits on the E4 baseline):
- C1: eic frontmatter no longer claims decision ownership
- C2: "Recommendation to Peer Reviewers" retired (no delivery channel
under Iron Rule #2); SKILL.md "Sets the review tone" line replaced
- C4: [CONSENSUS-2] -> corroborated finding (agree=2, conflict=0)
- C6: first-impression score (1-10) retired (fed no output)
- A6: <paper_content> data-fence in all 5 reviewer agents + protocol
delivery side + re-review untrusted-input annotations
- A5: panel-suggested references need attestation or [UNVERIFIED];
perspective 3-5 quota removed (zero allowed)
- E7: letter gains "Top Blocking Issues (0-3, ranked)" with anchors,
zero valid for genuine Accept (template + synthesizer lockstep)
- P0-3: DA "no repeating other reviewers" reframed — report findings,
dedup is synthesis work
Full suite 3518 green; 215/216/phase-boundary/spec-consistency lints green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017bHgbm9bstvmorumtPa65J
* refactor: /simplify pass — delete C2-residual quality gate, align E7 comment pair, C6 rationale to tombstone
- eic_agent quality gate "includes focus suggestions for other reviewers"
required the very field C2 retired — deleted (the actual second
overlap-class instance; field_analyst's config-time non-overlap gate
reviewed and deliberately kept)
- the two Top Blocking Issues HTML comments start life identical instead
of divergent (template <-> synthesizer parallel pair)
- C6 retirement rationale moved from live instructional prose into a
tombstone comment, matching the C2 placement
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017bHgbm9bstvmorumtPa65J
* fix: close codex ultra round-1 findings — acknowledgment_only scope, cross-model fence lockstep, mode-neutral C6, Rn syntax, registry ownership, Step 2 pointer
- P1: re-review response-letter note scoped to manuscript-change claims
(acknowledgment_only commitments keep the letter as their evidence)
- P1: cross-model Reviewer 2 call-2 transport wraps the paper in
<paper_content> in lockstep with sprint_contract_protocol §2 step 4
- P2: C6 replacement wording mode-neutral (calibration emits rubric
scores without a contract)
- P2: blocker-table placeholder uses the Roadmap's own Rn ID syntax
(both letter mirrors)
- P2: SKILL.md registry row no longer calls the template "EIC final
decision letter" (synthesizer Phase 2 ownership)
- P2: corroborated-finding pointer corrected to Step 2 Consensus
Identification
Full suite green; cross-model handoff + parity lints green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017bHgbm9bstvmorumtPa65J
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 12:47:46 +08:00
- Transport follows #523 ownership: the dispatching layer (the main session running the reviewer skill — not a Bucket A agent) executes the API calls, mirroring the in-session phase inputs exactly: call 1 = the Phase 1 system persona + the contract JSON + the paper METADATA that in-session Phase 1 receives (paper content withheld, per the sprint protocol's Phase 1 input spec); call 2 = the re-injected contract + the Phase 2 system prompt + call 1's output wrapped in the ` <phase1_output>` data delimiter + the paper wrapped in the ` <paper_content>` data delimiter (#574 A6, in lockstep with ` sprint_contract_protocol.md` §2 step 4 — the cross-model seat receives the manuscript inside the same fence as in-session seats). The delimiters are the conversation linkage — no server-side session state is assumed.
2026-08-15 15:04:28 +08:00
- The dispatching layer hands the synthesizer the slot's report PLUS the actual seat-level provenance observation (role ID, invocation-context ID, peer-output visibility, actor type, model family, provider, and any accountable human-reviewer ID). It builds the ` reviewer_full`-bound ` review-panel-provenance/1.0` artifact over the exact EIC/R1/R2/R3/DA roster, then raw-byte and replay-validates its closed Schema 6 carrier with ` scripts/review_panel_provenance.py`; the synthesizer fills the Review Panel Provenance block from that artifact, never from a persona, intended route, or configured provider.
2026-07-18 14:38:26 +08:00
- The slot's report enters the panel matrix exactly as that slot's report always does — heterogeneity itself is the §5.2 safeguard. The synthesizer computes NO cross-family aggregate and NO "same-model majority" (any such aggregation is on its forbidden-operations list): cross-family splits are visible by inspection in the panel matrix the user already receives, and the provenance block names which seat ran on which family.
- An ungrounded compatible provider is first-class here (same class as DA critique: persona judgment needs no web grounding); its factual claims about literature remain subject to the normal citation gates.
2026-08-15 15:04:28 +08:00
- Degradation: a failed/unavailable cross-model dispatch falls back to the normal primary-family routing for that seat (the session model, as adjusted by any active ` ARS_MODEL_TIERING` policy — tiering is orthogonal and never overridden by this track), and the actual fallback execution is recorded in that seat's typed provenance — never a silent swap-back. If the actual family or provider cannot be established, the observation is omitted/null and the corresponding axis becomes ` unknown`; the intended route MUST NOT fill the gap.
2026-07-18 14:38:26 +08:00
**When not active** (env unset, or consent not given):
2026-08-15 15:04:28 +08:00
- All five personas run on the normal primary-family routing (session model + any active ` ARS_MODEL_TIERING` policy), and the Editorial Decision Letter carries the correlated-error disclosure derived from the typed provenance artifact (see the template's Review Panel Provenance block) instead of silently implying independence.
**Typed provenance is not a binary independence score (#740).** The closed
contract and field semantics are defined in
` academic-paper-reviewer/references/review_panel_provenance_protocol.md`.
` role_separated`, ` fresh_context`, ` blind_to_peer_outputs`,
` model_family_distinct`, ` provider_distinct`, and ` human_distinct` remain
separate ` true` / ` false` / ` unknown` axes. A fixed seat or persona label does
not fill even the role observation; the dispatcher records the actual role and
all other axes from execution. No label establishes a binary ` independent`
value. Same-family execution requires the fixed correlated-error
disclosure, while missing family evidence stays ` unknown` and carries the
unknown-family disclosure. ` fresh_context` is fixed to
` fresh_context_scope: within_panel_attempt_only`: it compares the five contexts
within one artifact and does not prove that a retry or later round used contexts
new to attempt history.
2026-07-18 14:38:26 +08:00
External motivation: Ren et al. (2026, arXiv:2607.13104 §5.2) — consistency-derived feedback is fragile when errors correlate across samples of one model, and repeated sampling may amplify a confidently-wrong conclusion; heterogeneous critique models are among the safeguards it names.
2026-08-01 19:18:11 +08:00
#### Calibration transport exception (#611 — non-sprint, attempt-atomic)
This branch applies only to the opt-in ` reviewer_calibration` mode. It does not opt calibration into the sprint contract or change the ordinary ` reviewer_full` transport above. For each calibration panel, the Reviewer 2 substrate swap is exactly one stateless provider call that byte-for-byte mirrors the same replicate's primary-family calibration Reviewer 2 invocation: the same ` domain_reviewer_agent` system persona, that paper's already-frozen Reviewer Configuration Card #3, and the complete manuscript inside the same ` <paper_content>...</paper_content>` data fence. The call MUST NOT send a sprint contract, a paper-blind Phase 1 request, ` <phase1_output>`, any gold label, human score, per-dimension gold, or gold rationale. Its return is the complete standard-mode Reviewer 2 report plus a substrate-provenance stamp for the existing calibration synthesizer. This is a transport-only substitution; it does not change any reviewer prompt, rubric, panel cardinality, or synthesis semantics.
**Calibration data-fence collision preflight (closed).** The single-call payload carries Reviewer Configuration Card #3 byte-for-byte inside ` <reviewer_configuration>...</reviewer_configuration>` and the manuscript byte-for-byte inside ` <paper_content>...</paper_content>`. Before a payload is sent, test each raw source independently against its own wrapper with the case-insensitive predicates ` </\s*reviewer_configuration\b[^>]*>` and ` </\s*paper_content\b[^>]*>`, respectively. If either matches, refuse the entire calibration attempt before transport and send no provider call containing either payload. MUST NOT escape, strip, rewrite, truncate, switch delimiters, or fall back to primary routing: those paths break byte parity or send the same colliding content. Exact, whitespace, case, self-closing, and attributed/tolerant-parser closing forms match; a different longer tag such as ` </paper_contents>` does not match. This preflight is closed at exactly these two tag names; fragments without ` >`, entity encodings, and Unicode confusables are outside its delimiter grammar. Expanding this boundary requires the normative paragraph, lint witnesses, and mutation tests to change together.
Calibration is repeated-panel measurement, so its fallback is
**attempt-atomic** rather than per-seat:
1. Before any scored panel completes, lock one ` attempt_id` and one ` substrate_plan` (` cross_model_r2` or ` primary_only`) without consulting any gold material. Configuration, consent, and a non-content transport preflight happen before this lock. If any is unavailable, warn, lock ` primary_only`, disclose the reason, and begin the complete schedule on that plan.
2. Under ` cross_model_r2`, every paper and replicate uses the single-call branch above. If a later Reviewer 2 dispatch fails after the attempt begins, mark the entire attempt invalid; every completed panel in that attempt becomes diagnostic-only and MUST NOT enter any aggregate. Never continue the failed paper or a later replicate on primary routing.
3. The only result-producing recovery is a new ` attempt_id`, an empty aggregate, and a restart at paper 1 / replicate 1 on one homogeneous plan. Restarting all-primary may spend the whole schedule again, so stop for explicit user authorization unless that retry cost was already authorized. Before a homogeneous attempt finishes, MUST NOT emit full-tier metrics, a directional readout, or either session disclosure.
This attempt-atomic override is calibration-only; ordinary ` reviewer_full` keeps the per-seat disclosed fallback above.
2026-08-15 15:04:28 +08:00
### Re-Review Judge Provenance (#539/#740 — Stage 3' verification round)
feat(reviewer): #539 re-review judge independence + judge record (#559)
* feat(reviewer): #539 re-review judge independence + judge record
Stage 3' Priority-1 addressed-ness verdicts get an independent cross-model
pass when ARS_CROSS_MODEL is set (dispatching-layer transport per #523;
[CROSS-FAMILY-DIVERGENCE] rows are a review trigger for the EIC, never a
vote); single-provider runs carry a never-omitted §8.1.2 disclosure line.
Judge Record block (verification vs revision-driving judge, rubric version,
evidence seen, judging budget separate from generation) + Schema 6 optional
judge_record field.
External motivation: Ren et al. (2026, arXiv:2607.13104 §8.1.2).
Closes #539
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 1 — dispatchable pass, fillable Judge Record, defined output surfaces, §8.1.2 precision
P1: the per-item pass is integrity-sampling-style direct dispatch by the
dispatching layer (no #527 envelope — no fenced owner hands off); full verdict
set incl. MADE_WORSE; results land in a new R&R matrix Cross-model column
(agree / diverges / unavailable) with diverges as a Phase 2 review trigger.
P1: Judge Record made fillable — Round-1 panel provenance copied seat-level
from the #540 provenance block (singular revision-driving judge undefined for
a mixed-family panel); cross-model status is a three-way enum; prompt/rubric
surfaces recorded. P2: disclosure line verbatim in the output template; Schema
11 optional per-row fields; §8.1.2 attribution tightened + calibration claim
qualified.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 2 — judgment-specific transport, letter carried into Stage 4->3', usable matrix, not_configured status
P1: the per-item pass uses the provider transport with a judgment-specific
request (no grounding gate, closed verdict set, non-conforming -> unavailable,
never coerced) instead of the citation handlers. P1: the Editorial Decision
Letter joins the Stage 4->3' handoff on both locked surfaces (orchestrator +
pipeline SKILL; locks re-hashed) so the Judge Record's Round-1 provenance is
actually fillable post-#540. P1: the R&R matrix template fixed to eight
delimiter/example cells with real Cross-model values. P2: not_configured added
to the cell/status vocabulary (single-family rows have a defined value).
P2: "exact configuration" softened to the analogous family-level correlation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 3 — letter named on all handoff surfaces, status scoped to Priority 1 rows, CHANGELOG synced
P1: the Editorial Decision Letter joins the Stage 4->3' handoff on the
remaining three operative surfaces (state-machine transition + material rows,
lock re-hashed; reviewer SKILL re-review input line; protocol input list).
P1: cross_model_status always-carried requirement scoped to Priority 1 rows
(the pass does not evaluate P2/P3; their tables omit the column). P2:
CHANGELOG entry synced to the softened/consented/diverges-vocabulary text.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 4 — letter on team/state-tracker/example surfaces, neutral provenance fallback
P1: the Editorial Decision Letter joins the Stage 4->3' transfer in the team
collaboration protocol, the state tracker's prerequisite row (lock re-hashed),
and both pipeline examples. P2: the absent-provenance fallback is neutral —
"unknown (provenance block absent)" with an optional recorded reason (guided
Round 1 emits no letter; pre-#540 letters lack the block) instead of falsely
labeling #540-era guided runs as pre-#540.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 5 — letter on the two-stage protocol + recovery example; partial status for mixed runs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 6 — partial + N/M counts in the Schema 6 enum; shared-doc disclosure scoped to not_configured / all-unavailable
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 14:56:41 +08:00
2026-08-15 15:04:28 +08:00
**When active** (configured + consented): after the re-review commits its Priority 1 verdicts, the dispatching layer runs a direct blind per-item pass over the § API Call Patterns TRANSPORT (endpoint + auth) with a judgment-specific request — not the citation handlers: no grounding requirement (persona-judgment class), closed verdict set {FULLY_ADDRESSED, PARTIALLY_ADDRESSED, NOT_ADDRESSED, MADE_WORSE}, non-conforming responses → ` unavailable`, never coerced; item + author claim + revised passage sent minimized and as data. Results land in the R&R Traceability Matrix's ` Cross-model` column (` agree` / ` diverges: <verdict>` / ` unavailable` / ` not_configured`) — a ` diverges` cell is a review trigger for the Phase 2 synthesis decision, never a vote; ` unavailable` is ROW-level (that row carries the same-family caveat). **Run-level disclosure** (the verbatim same-family line in the Re-Review Output, never omitted) applies only when the pass is ` not_configured` or EVERY item came back unavailable; mixed runs record ` partial — N/M items judged`. Both cases record the Judge Record (actual verification-judge identity; exact replay-validated Round-1 provenance artifact reference/digest and axes, or explicit unknown; prompt/rubric surfaces; evidence seen; judging budget separate from generation) — Schema 6 optional ` judge_record`. Cross-family routing and input blinding are typed provenance facts, never a binary independence claim. Authority: ` academic-paper-reviewer/references/re_review_mode_protocol.md` § Judge Provenance and Correlated-Error Boundary. External motivation: Ren et al. §8.1.2 — a distinct judge configuration for final reporting plus transparency about the judge's identity, prompt, rubric, and budget; the reviewer's calibration mode approximates the same section's calibration-against-a-verifiable-subset safeguard to the extent the user's gold labels reflect real outcomes.
feat(reviewer): #539 re-review judge independence + judge record (#559)
* feat(reviewer): #539 re-review judge independence + judge record
Stage 3' Priority-1 addressed-ness verdicts get an independent cross-model
pass when ARS_CROSS_MODEL is set (dispatching-layer transport per #523;
[CROSS-FAMILY-DIVERGENCE] rows are a review trigger for the EIC, never a
vote); single-provider runs carry a never-omitted §8.1.2 disclosure line.
Judge Record block (verification vs revision-driving judge, rubric version,
evidence seen, judging budget separate from generation) + Schema 6 optional
judge_record field.
External motivation: Ren et al. (2026, arXiv:2607.13104 §8.1.2).
Closes #539
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 1 — dispatchable pass, fillable Judge Record, defined output surfaces, §8.1.2 precision
P1: the per-item pass is integrity-sampling-style direct dispatch by the
dispatching layer (no #527 envelope — no fenced owner hands off); full verdict
set incl. MADE_WORSE; results land in a new R&R matrix Cross-model column
(agree / diverges / unavailable) with diverges as a Phase 2 review trigger.
P1: Judge Record made fillable — Round-1 panel provenance copied seat-level
from the #540 provenance block (singular revision-driving judge undefined for
a mixed-family panel); cross-model status is a three-way enum; prompt/rubric
surfaces recorded. P2: disclosure line verbatim in the output template; Schema
11 optional per-row fields; §8.1.2 attribution tightened + calibration claim
qualified.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 2 — judgment-specific transport, letter carried into Stage 4->3', usable matrix, not_configured status
P1: the per-item pass uses the provider transport with a judgment-specific
request (no grounding gate, closed verdict set, non-conforming -> unavailable,
never coerced) instead of the citation handlers. P1: the Editorial Decision
Letter joins the Stage 4->3' handoff on both locked surfaces (orchestrator +
pipeline SKILL; locks re-hashed) so the Judge Record's Round-1 provenance is
actually fillable post-#540. P1: the R&R matrix template fixed to eight
delimiter/example cells with real Cross-model values. P2: not_configured added
to the cell/status vocabulary (single-family rows have a defined value).
P2: "exact configuration" softened to the analogous family-level correlation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 3 — letter named on all handoff surfaces, status scoped to Priority 1 rows, CHANGELOG synced
P1: the Editorial Decision Letter joins the Stage 4->3' handoff on the
remaining three operative surfaces (state-machine transition + material rows,
lock re-hashed; reviewer SKILL re-review input line; protocol input list).
P1: cross_model_status always-carried requirement scoped to Priority 1 rows
(the pass does not evaluate P2/P3; their tables omit the column). P2:
CHANGELOG entry synced to the softened/consented/diverges-vocabulary text.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 4 — letter on team/state-tracker/example surfaces, neutral provenance fallback
P1: the Editorial Decision Letter joins the Stage 4->3' transfer in the team
collaboration protocol, the state tracker's prerequisite row (lock re-hashed),
and both pipeline examples. P2: the absent-provenance fallback is neutral —
"unknown (provenance block absent)" with an optional recorded reason (guided
Round 1 emits no letter; pre-#540 letters lack the block) instead of falsely
labeling #540-era guided runs as pre-#540.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 5 — letter on the two-stage protocol + recovery example; partial status for mixed runs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
* fix(reviewer): #539 codex round 6 — partial + N/M counts in the Schema 6 enum; shared-doc disclosure scoped to not_configured / all-unavailable
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hi625UBf6GX7aeWJpJpSnE
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 14:56:41 +08:00
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
### Blind Disagreement Checkpoints (research-design freeze + final editorial decision)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
Two irreversible checkpoints gain an optional cross-model check when ` ARS_CROSS_MODEL` is set and the consent gate has been passed:
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
| Checkpoint | Primary owner | Cross-model input (never the primary's decision) | Structured decision enum |
|---|---|---|---|
| Research-design freeze | ` research_architect_agent` (deep-research) | RQ Brief + draft Methodology Blueprint | ` sound` / ` revise_before_freeze` / ` fundamental_concern` |
| Final editorial decision | ` editorial_synthesizer_agent` (academic-paper-reviewer) | The panel's usable reviewer cards (all ` panel_size` N of them — 5 in the default full-mode panel, 2 under ` methodology_focus`) + paper metadata | ` accept` / ` minor_revision` / ` major_revision` / ` reject` |
**Mechanics:**
feat(cross-model): #527 canonical handoff envelope + dispatcher consumer contract (#536)
* feat(cross-model): #527 canonical handoff envelope + dispatcher consumer contract
One canonical [CROSS-MODEL-HANDOFF v1] envelope for the #523
owner->dispatcher->owner transport, with scripts/cross_model_handoff.py
as the normative grammar (parse + outcome routing as pure functions)
and 20 deterministic fixtures on a fake transport. The three checkpoint
owners emit the envelope with closed kind/result-shape pairs; the
Mode-A orchestrator pins the consumer contract (recognition, fail-safe
malformed->unavailable mapping, agreement-no-reinvoke,
divergence-minimum-context, DA full-return, flag-unset unchanged).
New check_cross_model_handoff_contract.py pins the five surfaces with
14 mutation tests; both suites manifest- and CI-wired. Orchestrator
CONTENT_LOCKS hash updated alongside its edit; research_architect
plugin mirror re-synced. Closes #527.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-1 closures — fail-closed versioning, owner binding, full-contract validation
(1) Unknown envelope versions (a v2 fence) are malformed_handoff, never
an ordinary deliverable. (2) Closed kind->owner binding (EXPECTED_OWNERS)
so a divergence can only return to the original owner. (3) The #518
output contract is fully required on both sides — decision + drivers
(list of <=3 strings) + confidence; a bare decision never routes to a
judgment. (4) Owner-side validation errors classify as malformed_handoff,
not malformed_result. (5) The lint pins full backticked header
declarations + the owners' blindness clauses instead of bare values.
Six new witnesses; 40 tests across the two #527 suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-2 closures — header strictness, fence-collision rejection, three-field owners
(1) Both enum owners' committed-decision instructions now include the
required confidence field (the parser rejects a bare decision, so the
old instructions would have disabled the checkpoint); shared Mechanics
step 1 aligned. (2) The lint pins the per-owner payload exclusions and
the shared owner_agent header / ALL-THREE-fields / fence-shaped-line
declarations. (3) full_return envelopes reject an owner_decision header
(REQUIRED iff enum_comparison) and unknown headers fail closed. (4) A
fence-shaped line in the payload (or a second envelope) rejects the
whole output as ambiguous instead of silently truncating. (5) Boundary
advisory recorded in the payload definition: sanitized also means
data-minimized (names/affiliations/private URLs stripped unless covered
by consent). Eight new witnesses; 47 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-3 closures — literal pins against self-referential tests
Outcome strings, EXPECTED_OWNERS, and fence literals are asserted as
hardcoded values with a distinctness check, and wrong-owner rejection
is exercised per checkpoint kind with hardcoded owner names — a mutated
module constant can no longer stay green by feeding the assertions its
own value. 51 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-4 closures — minimization pin, limit pins, full-return context
(1) The data-minimization payload rule is pinned in the lint with an
adverse witness. (2) MAX_DRIVERS / CONFIDENCE_VALUES are pinned as
literals with rejection tests for a fourth driver and an
out-of-contract confidence. (3) The DA full-return fixture asserts the
complete required context (correlation_id + owner + verbatim response)
against hardcoded values. 55 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-5 closures — strict column-0 acceptance, consent pins, authoritative transport_ok
(1) Fence detection stays generous (any version, stripped) but
acceptance is strict: only the exact column-0 v1 fences are valid — an
indented or re-versioned fence is malformed_handoff, never transported
and never a deliverable; the Mode-A trigger widened to match (any
handoff-shaped fence) and its narrowing is a lint witness. (2) The
three owners' consent predicates are pinned with an inversion witness.
(3) The divergence return context is asserted exactly (a truncated
cross_model_decision no longer stays green). (4) transport_ok is
authoritative — a failed transport with a valid-looking residual body
routes to unavailable, pinned by witness. (5) Non-string driver
elements have an adverse witness. Orchestrator CONTENT_LOCKS hash
updated with its edit. 60 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-6 closures — parse-level fence rule, detection witnesses, count sync
(1) parse_handoff enforces the no-fence-inside-envelope rule itself, so
a caller that skips extract_handoff_block gets the same rejection; a
direct nested-opener regression added. (2) The generous any-version
detection is independently witnessed by an unclosed v2 opener that must
raise (kills a detection-narrowing edit that all prior tests missed).
(3) Zero-width-prefixed fences are detected via Cf-fold (the #524
fold-before-compare lesson) and rejected by raw column-0 acceptance.
(4) CHANGELOG test counts synced to the actual suites (41 + 22).
63 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-7 closures — prompt-shape separation, RecursionError fail-closed, property witnesses
(1) The dispatcher reuses the § API Call Patterns transport mechanics
(endpoint/auth/model/error handling) but with the checkpoint's
structured-decision prompt (or DA-critique prompt) — NEVER the
citation-verification prompt or its grounding-status normalization,
which would corrupt a judgment into a citation verdict; pinned on both
the shared contract and the Mode-A block. (2) RecursionError from
pathologically nested JSON fails closed on both parse paths, with
concrete 100k-deep witnesses. (3) Detection properties are pinned at
the property level: any unknown version token (v3/v99/empty) raises
even opener-only, every sampled Cf format char folds in detection, and
parse_handoff itself catches a Cf-masked nested fence. Orchestrator
CONTENT_LOCKS hash updated. 68 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-8 closures — per-field witnesses + count-free changelog
Missing-drivers and missing-confidence are witnessed independently on
both the result and owner paths (a single-field default can no longer
reach agreement green); the CHANGELOG entry drops exact test counts so
it cannot go stale against the suites again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 round-9 probe closures — extraction/parse guard witnesses
Six witnesses for the guards codex's round-9 mutation probes showed
unpinned: transport_ok=True with a missing body stays unavailable;
closing-fence-only and close-before-open outputs raise; parse_handoff
called directly rejects a wrong first line or missing last-line closer.
75 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-10 closures — duplicate-key rejection, blank-body guard, triple pins
(1) Structured-decision JSON is parsed with a duplicate-rejecting
decoder on both paths (two decision values in one object is ambiguity,
never a judgment). (2) A blank/whitespace full_return body is a failed
transport, not a critique handed back to the owner. (3) Invariant 4 now
derives the complete kind (owner) result triples from CHECKPOINT_KINDS
+ EXPECTED_OWNERS, with a rewire witness. 77 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-11 closures — Cf-blank guards, per-triple pins, dispatcher clause pins, header witnesses
(1) Blankness folds Cf characters on both the payload and full-return
paths (an invisible-only payload/response is never substance), with
witnesses. (2) The shared prose states each kind/owner/result triple
independently and the lint derives ALL of them from the module (an enum
owner flipped to full_return now fires). (3) The dispatcher's
never-forwarded blindness clause and the flag-unset not-transported
clause are pinned with inversion witnesses. (4) Every required header
has an independent omission witness (an exempted header can no longer
surface as KeyError). 84 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-12 closure — full per-owner header pins
Owner instructions now spell every operative envelope header explicitly
(checkpoint_kind, owner_agent — derived from EXPECTED_OWNERS —,
correlation_id, owner_decision where applicable, expected_result), and
the lint pins each spelling with typo/wrong-owner witnesses; the
research_architect plugin mirror re-synced.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-13 closures — strict JSON constants, Cf-blank headers, boundary witnesses
(1) _loads_strict rejects NaN/Infinity via parse_constant, witnessed on
both paths. (2) A required header whose value is blank after Cf-folding
(zero-width-only correlation_id) is missing. (3) The inclusive
three-driver maximum is witnessed as VALID (off-by-one tightening now
fails). (4) A marker-less envelope raises HandoffError, never
TypeError, with a direct witness. 88 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* test(review): #527 round-14 P2 survivor closures — verbatim body, close-fence parse witness, blank-header guard
Three witnesses for the round-14 P2 survivors: the full_return body
returns verbatim (whitespace included); parse_handoff rejects a
close-fence shape inside the payload directly; a blank owner_decision
header on full_return stays malformed. Codex round 14 declared 0 P1 —
P1 convergence reached; these close the remaining P2 test gaps.
90 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 18:03:29 +08:00
1. The primary reaches its decision as normal and records it in the SAME structured form as step 3 (the enum + up to 3 drivers + confidence — all three fields) **before** the cross-model is called — both sides commit blind, so the comparison in step 4 is enum-against-enum, not enum-against-prose. Under a sprint contract, the editorial checkpoint runs **after** the mechanical three-step protocol has emitted ` editorial_decision` (a post-Step-3 comparison; the contract arithmetic itself is never extended or re-run).
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
2. The cross-model receives the same input material and a structured-decision prompt. It **never** sees the primary's decision, scores, or reasoning first — the same anchoring-prevention rule as the integrity samples.
3. Output contract: ` {decision: <enum>, drivers: [up to 3 one-sentence reasons], confidence: low|medium|high}`.
4. Mechanical comparison: **material divergence = differing enum values.** Adjacent categories (e.g. minor vs major revision) are still material; the report notes adjacency.
5. On divergence: a **targeted rebuttal** — the primary must address each cross-model driver specifically against the evidence already on file (reviewer cards / blueprint content), no generic reassurance. Both decisions and the rebuttal surface to the user. The primary's decision stands unless the **user** changes it: disagreement is a review trigger, never a vote, and the two decisions are never averaged.
6. On agreement: one log line ` [CROSS-MODEL-CHECKPOINT: agreement — <checkpoint>]`; both structured decisions are still recorded.
7. Graceful degradation: transport failure → ` [CROSS-MODEL-ERROR]`, proceed single-model, note in the report (see § Graceful Degradation).
2026-08-15 15:04:28 +08:00
**Transport ownership (#523).** Both checkpoint owners are fenced single-phase (Bucket A) agents: the runtime write-scope guard (` scripts/ars_write_scope_guard.py`) denies them ALL Bash, and ` research_architect_agent` additionally carries the #514 frontmatter ` tools:` allowlist (` Read, Write, Edit, Grep, Glob` — no shell) at dispatch time. A checkpoint owner therefore never executes the § API Call Patterns transport itself when it runs as a dispatched subagent. The contract: the owner commits its structured decision (step 1) and emits the sanitized cross-model input as a **handoff artifact**; the **dispatching layer** — the context that invoked the agent and holds shell capability (the main session running the skill, or ` pipeline_orchestrator_agent` in pipeline Mode A; neither is Bucket A) — executes the transport, parses the structured output, and applies the mechanical enum comparison (step 4). Agreement or transport failure → the dispatching layer records the outcome (the audit-surface fill is a mechanical template population from the two committed decisions); divergence → it re-invokes the owner with the cross-model's ` {decision, drivers, confidence}` to produce the targeted rebuttal (step 5) — the comparison is mechanical, the rebuttal is the owner's judgment against the evidence on file and is never written by the dispatcher. When the owning role executes inline in a context that itself holds shell capability, owner and dispatching layer are the same context and the handoff is a no-op. **This rule generalizes:** any cross-model call whose primary owner is a Bucket A agent routes its transport through the dispatching layer the same way (e.g. ` devils_advocate_reviewer_agent`'s blind, separately executed cross-model DA critique) — with one outcome-routing difference: a call with no mechanical enum comparison (the DA critique) has nothing the dispatcher can resolve itself, so every successful response is returned to the owner for the follow-on judgment, not only divergences. Non-fenced owners with shell capability (` integrity_verification_agent` at the Stage 2.5/4.5 gates, ` devils_advocate_agent` in deep-research, the main session) execute § API Call Patterns directly, unchanged.
fix(cross-model): #523 blind-checkpoint transport moves to the dispatching layer (#525)
* fix(cross-model): #523 blind-checkpoint transport moves to the dispatching layer
The #518 blind disagreement checkpoints instructed their Bucket A primary
owners (research_architect_agent, editorial_synthesizer_agent) to execute
the curl transport themselves — unexecutable under the runtime Bash deny
(ars_write_scope_guard.py) and, for the architect, the #514 dispatch-time
tools allowlist. New Transport ownership contract in
shared/cross_model_verification.md: the owner commits its structured
decision and emits a sanitized handoff artifact; the dispatching layer
(main session / pipeline_orchestrator_agent — neither Bucket A) executes
the API call, applies the mechanical enum comparison, and re-invokes the
owner only for the divergence rebuttal. Inline execution in a
shell-capable context is its own dispatching layer (no-op handoff).
Rule stated generally: it covers any Bucket A cross-model owner
(devils_advocate_reviewer_agent's DA critique gets the same clause);
non-fenced owners (integrity_verification_agent, deep-research
devils_advocate_agent, main session) are unchanged. No Bash/WebFetch
re-added anywhere (resolution (a); (c) rejected). Mirror re-copied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s
* fix(cross-model): address codex xhigh P1s — dispatched-exception ordering + DA return path
P1-1: the editorial checkpoint's dispatched shape is now an explicit,
justified exception to the before-the-roadmap ordering (safe because the
sprint-contract boundary keeps cross-model drivers out of the roadmap),
instead of a parenthetical contradicting the same sentence. P1-2: the DA
critique has no mechanical enum comparison, so the dispatcher now returns
every successful response to the DA owner for the findings comparison —
stated in both the agent clause and the shared generalization rule.
Codex P2 (#514-vs-#521 provenance) rejected: #514 is the spec issue and
the repo's canonical reference convention; #521 is its implementing PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 13:26:23 +08:00
feat(cross-model): #527 canonical handoff envelope + dispatcher consumer contract (#536)
* feat(cross-model): #527 canonical handoff envelope + dispatcher consumer contract
One canonical [CROSS-MODEL-HANDOFF v1] envelope for the #523
owner->dispatcher->owner transport, with scripts/cross_model_handoff.py
as the normative grammar (parse + outcome routing as pure functions)
and 20 deterministic fixtures on a fake transport. The three checkpoint
owners emit the envelope with closed kind/result-shape pairs; the
Mode-A orchestrator pins the consumer contract (recognition, fail-safe
malformed->unavailable mapping, agreement-no-reinvoke,
divergence-minimum-context, DA full-return, flag-unset unchanged).
New check_cross_model_handoff_contract.py pins the five surfaces with
14 mutation tests; both suites manifest- and CI-wired. Orchestrator
CONTENT_LOCKS hash updated alongside its edit; research_architect
plugin mirror re-synced. Closes #527.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-1 closures — fail-closed versioning, owner binding, full-contract validation
(1) Unknown envelope versions (a v2 fence) are malformed_handoff, never
an ordinary deliverable. (2) Closed kind->owner binding (EXPECTED_OWNERS)
so a divergence can only return to the original owner. (3) The #518
output contract is fully required on both sides — decision + drivers
(list of <=3 strings) + confidence; a bare decision never routes to a
judgment. (4) Owner-side validation errors classify as malformed_handoff,
not malformed_result. (5) The lint pins full backticked header
declarations + the owners' blindness clauses instead of bare values.
Six new witnesses; 40 tests across the two #527 suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-2 closures — header strictness, fence-collision rejection, three-field owners
(1) Both enum owners' committed-decision instructions now include the
required confidence field (the parser rejects a bare decision, so the
old instructions would have disabled the checkpoint); shared Mechanics
step 1 aligned. (2) The lint pins the per-owner payload exclusions and
the shared owner_agent header / ALL-THREE-fields / fence-shaped-line
declarations. (3) full_return envelopes reject an owner_decision header
(REQUIRED iff enum_comparison) and unknown headers fail closed. (4) A
fence-shaped line in the payload (or a second envelope) rejects the
whole output as ambiguous instead of silently truncating. (5) Boundary
advisory recorded in the payload definition: sanitized also means
data-minimized (names/affiliations/private URLs stripped unless covered
by consent). Eight new witnesses; 47 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-3 closures — literal pins against self-referential tests
Outcome strings, EXPECTED_OWNERS, and fence literals are asserted as
hardcoded values with a distinctness check, and wrong-owner rejection
is exercised per checkpoint kind with hardcoded owner names — a mutated
module constant can no longer stay green by feeding the assertions its
own value. 51 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-4 closures — minimization pin, limit pins, full-return context
(1) The data-minimization payload rule is pinned in the lint with an
adverse witness. (2) MAX_DRIVERS / CONFIDENCE_VALUES are pinned as
literals with rejection tests for a fourth driver and an
out-of-contract confidence. (3) The DA full-return fixture asserts the
complete required context (correlation_id + owner + verbatim response)
against hardcoded values. 55 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-5 closures — strict column-0 acceptance, consent pins, authoritative transport_ok
(1) Fence detection stays generous (any version, stripped) but
acceptance is strict: only the exact column-0 v1 fences are valid — an
indented or re-versioned fence is malformed_handoff, never transported
and never a deliverable; the Mode-A trigger widened to match (any
handoff-shaped fence) and its narrowing is a lint witness. (2) The
three owners' consent predicates are pinned with an inversion witness.
(3) The divergence return context is asserted exactly (a truncated
cross_model_decision no longer stays green). (4) transport_ok is
authoritative — a failed transport with a valid-looking residual body
routes to unavailable, pinned by witness. (5) Non-string driver
elements have an adverse witness. Orchestrator CONTENT_LOCKS hash
updated with its edit. 60 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-6 closures — parse-level fence rule, detection witnesses, count sync
(1) parse_handoff enforces the no-fence-inside-envelope rule itself, so
a caller that skips extract_handoff_block gets the same rejection; a
direct nested-opener regression added. (2) The generous any-version
detection is independently witnessed by an unclosed v2 opener that must
raise (kills a detection-narrowing edit that all prior tests missed).
(3) Zero-width-prefixed fences are detected via Cf-fold (the #524
fold-before-compare lesson) and rejected by raw column-0 acceptance.
(4) CHANGELOG test counts synced to the actual suites (41 + 22).
63 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-7 closures — prompt-shape separation, RecursionError fail-closed, property witnesses
(1) The dispatcher reuses the § API Call Patterns transport mechanics
(endpoint/auth/model/error handling) but with the checkpoint's
structured-decision prompt (or DA-critique prompt) — NEVER the
citation-verification prompt or its grounding-status normalization,
which would corrupt a judgment into a citation verdict; pinned on both
the shared contract and the Mode-A block. (2) RecursionError from
pathologically nested JSON fails closed on both parse paths, with
concrete 100k-deep witnesses. (3) Detection properties are pinned at
the property level: any unknown version token (v3/v99/empty) raises
even opener-only, every sampled Cf format char folds in detection, and
parse_handoff itself catches a Cf-masked nested fence. Orchestrator
CONTENT_LOCKS hash updated. 68 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-8 closures — per-field witnesses + count-free changelog
Missing-drivers and missing-confidence are witnessed independently on
both the result and owner paths (a single-field default can no longer
reach agreement green); the CHANGELOG entry drops exact test counts so
it cannot go stale against the suites again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 round-9 probe closures — extraction/parse guard witnesses
Six witnesses for the guards codex's round-9 mutation probes showed
unpinned: transport_ok=True with a missing body stays unavailable;
closing-fence-only and close-before-open outputs raise; parse_handoff
called directly rejects a wrong first line or missing last-line closer.
75 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-10 closures — duplicate-key rejection, blank-body guard, triple pins
(1) Structured-decision JSON is parsed with a duplicate-rejecting
decoder on both paths (two decision values in one object is ambiguity,
never a judgment). (2) A blank/whitespace full_return body is a failed
transport, not a critique handed back to the owner. (3) Invariant 4 now
derives the complete kind (owner) result triples from CHECKPOINT_KINDS
+ EXPECTED_OWNERS, with a rewire witness. 77 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-11 closures — Cf-blank guards, per-triple pins, dispatcher clause pins, header witnesses
(1) Blankness folds Cf characters on both the payload and full-return
paths (an invisible-only payload/response is never substance), with
witnesses. (2) The shared prose states each kind/owner/result triple
independently and the lint derives ALL of them from the module (an enum
owner flipped to full_return now fires). (3) The dispatcher's
never-forwarded blindness clause and the flag-unset not-transported
clause are pinned with inversion witnesses. (4) Every required header
has an independent omission witness (an exempted header can no longer
surface as KeyError). 84 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-12 closure — full per-owner header pins
Owner instructions now spell every operative envelope header explicitly
(checkpoint_kind, owner_agent — derived from EXPECTED_OWNERS —,
correlation_id, owner_decision where applicable, expected_result), and
the lint pins each spelling with typo/wrong-owner witnesses; the
research_architect plugin mirror re-synced.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-13 closures — strict JSON constants, Cf-blank headers, boundary witnesses
(1) _loads_strict rejects NaN/Infinity via parse_constant, witnessed on
both paths. (2) A required header whose value is blank after Cf-folding
(zero-width-only correlation_id) is missing. (3) The inclusive
three-driver maximum is witnessed as VALID (off-by-one tightening now
fails). (4) A marker-less envelope raises HandoffError, never
TypeError, with a direct witness. 88 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* test(review): #527 round-14 P2 survivor closures — verbatim body, close-fence parse witness, blank-header guard
Three witnesses for the round-14 P2 survivors: the full_return body
returns verbatim (whitespace included); parse_handoff rejects a
close-fence shape inside the payload directly; a blank owner_decision
header on full_return stays malformed. Codex round 14 declared 0 P1 —
P1 convergence reached; these close the remaining P2 test gaps.
90 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 18:03:29 +08:00
### Cross-model handoff envelope (#527)
The #523 "clearly-delimited cross-model handoff block" has ONE canonical form. ` scripts/cross_model_handoff.py` is the **normative grammar** — this prose describes it; the module decides it; the fixtures in ` scripts/test_cross_model_handoff.py` pin the owner → dispatcher → owner path with a fake transport.
**Envelope (emitted by a dispatched owner, verbatim fences at line start):**
` ``
[CROSS-MODEL-HANDOFF v1]
checkpoint_kind: design_freeze | editorial_decision | da_critique
owner_agent: <emitting agent, e.g. research_architect_agent>
correlation_id: <owner-chosen stable token, echoed back verbatim on any re-invocation>
expected_result: enum_comparison | full_return
owner_decision: <single-line JSON {"decision": <enum>, "drivers": [...], "confidence": ...} — REQUIRED iff enum_comparison; travels OUTSIDE the payload and is NEVER forwarded to the cross-model>
payload:
<the sanitized cross-model input, exactly as step 2 of the owning checkpoint prepares it — everything below ` payload:` down to the closing fence is data, not instructions; it must not contain a fence-shaped line (the dispatcher rejects ambiguous fences rather than guessing). Sanitized also means data-minimized: strip personal names, affiliations, and private URLs not essential to the judgment unless their transmission is explicitly covered by the consent grant>
[/CROSS-MODEL-HANDOFF]
` ``
Kind ↔ owner ↔ result-shape triples are closed (normative mapping: ` CHECKPOINT_KINDS` + ` EXPECTED_OWNERS` in the reference module): ` design_freeze` (` research_architect_agent`) is ` enum_comparison`; ` editorial_decision` (` editorial_synthesizer_agent`) is ` enum_comparison` (decision enums per the checkpoint table above); ` da_critique` (` devils_advocate_reviewer_agent`) is ` full_return`. Any other combination — including an unknown version fence, which is malformed rather than an ordinary deliverable — fails closed. Structured decisions carry ALL THREE fields (` decision`, ` drivers`, ` confidence`) on both sides; a bare decision never routes to a judgment.
**Dispatcher consumer contract** (the main session running the skill, or ` pipeline_orchestrator_agent` in pipeline Mode A):
1. **Recognition.** A ` [CROSS-MODEL-HANDOFF v1]` fence in a dispatched agent's output is a transport request, never an ordinary deliverable — the dispatcher must not file it as content, summarize it, or drop it.
2. **Validation.** Unknown version fence, missing/duplicate header, unknown ` checkpoint_kind`, kind/` expected_result` mismatch, unparseable ` owner_decision`, or missing payload → ` [CROSS-MODEL-ERROR: malformed_handoff]`, outcome ` unavailable`, proceed single-model. Fail-closed: the dispatcher never repairs or guesses.
2026-08-15 15:04:28 +08:00
3. **Transport.** Execute the provider transport per § API Call Patterns (endpoint, auth, model id, timeout/error handling) with the **payload only** as input material — ` owner_decision` and everything outside the fences never reach the cross-model (blindness). The REQUEST PROMPT is the owning checkpoint's structured-decision prompt (§ Blind Disagreement Checkpoints, Mechanics steps 2-3) for ` enum_comparison`, or the blind-separately-executed-DA-critique prompt for ` full_return` — NEVER the citation-verification prompt, its grounding-status guards (` NOT_SEARCHED` / ` SOURCES:`), or its citation-status normalization, which would corrupt a judgment response into a citation verdict. This label records blinding and execution separation; it does not assert independent error processes.
feat(cross-model): #527 canonical handoff envelope + dispatcher consumer contract (#536)
* feat(cross-model): #527 canonical handoff envelope + dispatcher consumer contract
One canonical [CROSS-MODEL-HANDOFF v1] envelope for the #523
owner->dispatcher->owner transport, with scripts/cross_model_handoff.py
as the normative grammar (parse + outcome routing as pure functions)
and 20 deterministic fixtures on a fake transport. The three checkpoint
owners emit the envelope with closed kind/result-shape pairs; the
Mode-A orchestrator pins the consumer contract (recognition, fail-safe
malformed->unavailable mapping, agreement-no-reinvoke,
divergence-minimum-context, DA full-return, flag-unset unchanged).
New check_cross_model_handoff_contract.py pins the five surfaces with
14 mutation tests; both suites manifest- and CI-wired. Orchestrator
CONTENT_LOCKS hash updated alongside its edit; research_architect
plugin mirror re-synced. Closes #527.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-1 closures — fail-closed versioning, owner binding, full-contract validation
(1) Unknown envelope versions (a v2 fence) are malformed_handoff, never
an ordinary deliverable. (2) Closed kind->owner binding (EXPECTED_OWNERS)
so a divergence can only return to the original owner. (3) The #518
output contract is fully required on both sides — decision + drivers
(list of <=3 strings) + confidence; a bare decision never routes to a
judgment. (4) Owner-side validation errors classify as malformed_handoff,
not malformed_result. (5) The lint pins full backticked header
declarations + the owners' blindness clauses instead of bare values.
Six new witnesses; 40 tests across the two #527 suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-2 closures — header strictness, fence-collision rejection, three-field owners
(1) Both enum owners' committed-decision instructions now include the
required confidence field (the parser rejects a bare decision, so the
old instructions would have disabled the checkpoint); shared Mechanics
step 1 aligned. (2) The lint pins the per-owner payload exclusions and
the shared owner_agent header / ALL-THREE-fields / fence-shaped-line
declarations. (3) full_return envelopes reject an owner_decision header
(REQUIRED iff enum_comparison) and unknown headers fail closed. (4) A
fence-shaped line in the payload (or a second envelope) rejects the
whole output as ambiguous instead of silently truncating. (5) Boundary
advisory recorded in the payload definition: sanitized also means
data-minimized (names/affiliations/private URLs stripped unless covered
by consent). Eight new witnesses; 47 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-3 closures — literal pins against self-referential tests
Outcome strings, EXPECTED_OWNERS, and fence literals are asserted as
hardcoded values with a distinctness check, and wrong-owner rejection
is exercised per checkpoint kind with hardcoded owner names — a mutated
module constant can no longer stay green by feeding the assertions its
own value. 51 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-4 closures — minimization pin, limit pins, full-return context
(1) The data-minimization payload rule is pinned in the lint with an
adverse witness. (2) MAX_DRIVERS / CONFIDENCE_VALUES are pinned as
literals with rejection tests for a fourth driver and an
out-of-contract confidence. (3) The DA full-return fixture asserts the
complete required context (correlation_id + owner + verbatim response)
against hardcoded values. 55 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-5 closures — strict column-0 acceptance, consent pins, authoritative transport_ok
(1) Fence detection stays generous (any version, stripped) but
acceptance is strict: only the exact column-0 v1 fences are valid — an
indented or re-versioned fence is malformed_handoff, never transported
and never a deliverable; the Mode-A trigger widened to match (any
handoff-shaped fence) and its narrowing is a lint witness. (2) The
three owners' consent predicates are pinned with an inversion witness.
(3) The divergence return context is asserted exactly (a truncated
cross_model_decision no longer stays green). (4) transport_ok is
authoritative — a failed transport with a valid-looking residual body
routes to unavailable, pinned by witness. (5) Non-string driver
elements have an adverse witness. Orchestrator CONTENT_LOCKS hash
updated with its edit. 60 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-6 closures — parse-level fence rule, detection witnesses, count sync
(1) parse_handoff enforces the no-fence-inside-envelope rule itself, so
a caller that skips extract_handoff_block gets the same rejection; a
direct nested-opener regression added. (2) The generous any-version
detection is independently witnessed by an unclosed v2 opener that must
raise (kills a detection-narrowing edit that all prior tests missed).
(3) Zero-width-prefixed fences are detected via Cf-fold (the #524
fold-before-compare lesson) and rejected by raw column-0 acceptance.
(4) CHANGELOG test counts synced to the actual suites (41 + 22).
63 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-7 closures — prompt-shape separation, RecursionError fail-closed, property witnesses
(1) The dispatcher reuses the § API Call Patterns transport mechanics
(endpoint/auth/model/error handling) but with the checkpoint's
structured-decision prompt (or DA-critique prompt) — NEVER the
citation-verification prompt or its grounding-status normalization,
which would corrupt a judgment into a citation verdict; pinned on both
the shared contract and the Mode-A block. (2) RecursionError from
pathologically nested JSON fails closed on both parse paths, with
concrete 100k-deep witnesses. (3) Detection properties are pinned at
the property level: any unknown version token (v3/v99/empty) raises
even opener-only, every sampled Cf format char folds in detection, and
parse_handoff itself catches a Cf-masked nested fence. Orchestrator
CONTENT_LOCKS hash updated. 68 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-8 closures — per-field witnesses + count-free changelog
Missing-drivers and missing-confidence are witnessed independently on
both the result and owner paths (a single-field default can no longer
reach agreement green); the CHANGELOG entry drops exact test counts so
it cannot go stale against the suites again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 round-9 probe closures — extraction/parse guard witnesses
Six witnesses for the guards codex's round-9 mutation probes showed
unpinned: transport_ok=True with a missing body stays unavailable;
closing-fence-only and close-before-open outputs raise; parse_handoff
called directly rejects a wrong first line or missing last-line closer.
75 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-10 closures — duplicate-key rejection, blank-body guard, triple pins
(1) Structured-decision JSON is parsed with a duplicate-rejecting
decoder on both paths (two decision values in one object is ambiguity,
never a judgment). (2) A blank/whitespace full_return body is a failed
transport, not a critique handed back to the owner. (3) Invariant 4 now
derives the complete kind (owner) result triples from CHECKPOINT_KINDS
+ EXPECTED_OWNERS, with a rewire witness. 77 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-11 closures — Cf-blank guards, per-triple pins, dispatcher clause pins, header witnesses
(1) Blankness folds Cf characters on both the payload and full-return
paths (an invisible-only payload/response is never substance), with
witnesses. (2) The shared prose states each kind/owner/result triple
independently and the lint derives ALL of them from the module (an enum
owner flipped to full_return now fires). (3) The dispatcher's
never-forwarded blindness clause and the flag-unset not-transported
clause are pinned with inversion witnesses. (4) Every required header
has an independent omission witness (an exempted header can no longer
surface as KeyError). 84 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-12 closure — full per-owner header pins
Owner instructions now spell every operative envelope header explicitly
(checkpoint_kind, owner_agent — derived from EXPECTED_OWNERS —,
correlation_id, owner_decision where applicable, expected_result), and
the lint pins each spelling with typo/wrong-owner witnesses; the
research_architect plugin mirror re-synced.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* fix(review): #527 codex round-13 closures — strict JSON constants, Cf-blank headers, boundary witnesses
(1) _loads_strict rejects NaN/Infinity via parse_constant, witnessed on
both paths. (2) A required header whose value is blank after Cf-folding
(zero-width-only correlation_id) is missing. (3) The inclusive
three-driver maximum is witnessed as VALID (off-by-one tightening now
fails). (4) A marker-less envelope raises HandoffError, never
TypeError, with a direct witness. 88 tests across the suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
* test(review): #527 round-14 P2 survivor closures — verbatim body, close-fence parse witness, blank-header guard
Three witnesses for the round-14 P2 survivors: the full_return body
returns verbatim (whitespace included); parse_handoff rejects a
close-fence shape inside the payload directly; a blank owner_decision
header on full_return stays malformed. Codex round 14 declared 0 P1 —
P1 convergence reached; these close the remaining P2 test gaps.
90 tests across the two suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013y1bcrbtdSmARbBRmA8LHq
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 18:03:29 +08:00
4. **Result validation.** For ` enum_comparison` the response must parse as ` {decision ∈ the kind's enum, drivers ≤ 3, confidence ∈ low|medium|high}`; malformed JSON or an unknown enum value → ` [CROSS-MODEL-ERROR: malformed_result]`, outcome ` unavailable` — the dispatcher never fabricates or coerces a judgment.
5. **Agreement** (` enum_comparison`, equal enums): the dispatcher performs the mechanical fill (log line + audit-surface population from the two committed decisions) and does **not** re-invoke the owner.
6. **Divergence** (` enum_comparison`, differing enums): the dispatcher re-invokes the ORIGINAL owner with the minimum return context — ` correlation_id`, the owner's committed ` owner_decision`, the cross-model's full structured result, and the original payload (or a pointer to the same artifact on file) — and the owner writes the targeted rebuttal. The dispatcher never authors it.
7. **Full return** (` full_return`): no comparison exists for the dispatcher to resolve, so EVERY successful response is returned to the owner (` correlation_id` + the response verbatim); the findings comparison is the owner's.
8. **Flag unset.** With ` ARS_CROSS_MODEL` unset, owners emit no envelope and behavior is byte-equivalent pre-#527; a stray envelope encountered with the flag unset is logged ` [CROSS-MODEL-SKIPPED]` and not transported.
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
Checkpoint decisions are judgment, not lookup — an ungrounded/compatible provider is first-class here, with the same scoping as DA critique: a divergence from any provider is an adversarial hypothesis and a review trigger, never a confirmed defect.
> **Why there is no generic "6th reviewer."** An earlier version of this document planned a cross-model 6th reviewer for peer review. That design is retired, not deferred (#518, 2026-07): the conditions under which an extra generic reviewer becomes counterproductive — score averaging, role duplication, findings treated as confirmed defects, majority-vote false confidence, synthesizer context burn — match ARS's documented anti-patterns one-for-one. The blind disagreement checkpoints above are the replacement: cross-model judgment concentrated at the two decisions that are hardest to reverse, compared blind, with divergence escalated to the human instead of blended into a consensus.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
## API Call Patterns
2026-08-11 12:56:01 +08:00
Four patterns are documented below. The first is the citation-only Codex subscription adapter; the next two (OpenAI and Gemini) are first-party API routes and share the same contract: enable the provider's hosted web-search tool, and **gate the model's text on proof that a search actually happened** — no grounding evidence (an OpenAI ` web_search_call` item / a Gemini ` groundingMetadata` block) emits ` NOT_SEARCHED` and the text is discarded, so this guard, not the prompt wording, is what prevents a from-memory guess being laundered into ` VERIFIED`. Both first-party web-search tools are hosted/server-side: one request, no client-side tool-call round-trip. The fourth (OpenAI-compatible) is ungrounded by construction: it has no web-search tool, so the handler downgrades positive verdicts to ` NOT_SEARCHED` and lets rejections through, and a compatible verdict never counts as a grounded agreement. ` PROMPT` holds the single-reference verification prompt from step 3 for API routes; the Codex adapter instead accepts its closed request object.
### Codex app-server (ChatGPT subscription; citation integrity only)
When and only when ` ARS_CROSS_MODEL_TRANSPORT=codex`, the Stage 2.5 / 4.5
integrity producer constructs one ` ars-codex-citation-request/1.0` JSON object per
selected reference (` request_id`, exact ` reference_text`, exact
` citation_context`) and invokes ` scripts/cross_model_codex_verify.sh` on stdin.
It validates the one-line response against
` codex_citation_receipt.schema.json` before reading ` verdict`, ` searched`,
` reason_code`, or bound ` sources[]`. Nonzero exit is a transport error; a valid
` NOT_SEARCHED` receipt is an ungrounded result and follows the existing separate
handling. No other agent or checkpoint may consume this adapter.
The runtime drives app-server v2 rather than ` codex exec --json`: only the former
retains the structured standalone-search result objects needed for exact source
binding. The minimum-privilege, auth-attestation, event-grammar, and containment
contract is normative in
` docs/design/2026-08-11-630-codex-subscription-citation-transport-spec.md` and
machine-checked by the #630 test suite. The Bash entrypoints use syntax compatible
with macOS Bash 3.2.
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
### OpenAI (GPT-6 Astra / GPT-5.6 Sol / GPT-5.5 / GPT-5.5 Pro)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
Use the **Responses API** (` /v1/responses`) — the hosted ` web_search` tool lives there. (Chat Completions does not take ` tools: [{type: "web_search"}]`; web search on that endpoint requires the separate ` gpt-5-search-api` model, so this example targets Responses to stay model-agnostic across ` gpt-6-astra` / ` gpt-5.6-sol` / ` gpt-5.5` / ` gpt-5.5-pro` / the legacy ` gpt-5.4*` ids.)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
` ``bash
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
# PROMPT holds the single-reference verification prompt (step 3). One reference per call.
fix: Astra request compat, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823–#826) (#827)
* fix: Astra request compatibility, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823 #824 #825 #826)
#823 — OpenAI request builders (smoke entrypoint + documented example) drop
`temperature`, which GPT-6 Astra rejects; the per-model effort vocabulary
lives in scripts/cross_model_verification/openai_effort_guard.sh, sourced by
both, and an unsupported explicit Astra value fails before curl. Hermetic
fake-curl test runs both surfaces.
#824 — the contained Codex citation transport rejects effort=ultra with
REASONING_EFFORT_REQUIRES_DELEGATION before detection/auth/tempdir/launch on
both entry paths (codex-cli 0.153.4 defines ultra as the multiAgentMode
replacement). Model-independent by design.
#825 — hedging can no longer rescue an unsupported claim (writer recovery
tree, CER fallback row, temporal rule 5 in writer + both compiler mirrors,
writer contract D2); universal prose quotas in the writer, compilers,
writing_quality_check.md, academic-paper/SKILL.md, and contract D6 become
diagnostics subordinate to author/venue requirements. Audit inventory
corrected in place; held-out seed evals/heldout/unsupported_claim_recovery
(NOT_RUN) registered.
#826 — run_codex_audit.sh pins gpt-6-astra/xhigh and records both in a new
sidecar `model` block; claim_audit_pipeline binds an unknown judge identity
to a run-local cache key (no cross-run reuse) instead of defaulting to
gpt-5.5-xhigh.
Review: /simplify (4 angles), codex gpt-5.6-sol xhigh 2 rounds (r1: 1 P1 +
1 P2 + 2 P3 fixed; r2: 0 P1/P2), /security-review 0 findings; all 102
spec-consistency steps + pytest manifest replayed locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNKiXpdHx1T5F5RbXT2Ueu
* docs(claude): record the #824 ultra reversal in the v3.21.2 key-additions line
The v3.21.2 bullet still said the contained Codex citation transport accepts
ultra; #824 on this branch rejects it as a delegation request. Add the
reversal so the live instruction surface matches the transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7emV5r2aqZDJzAyYVuuDo
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 08:08:30 +09:00
GUARD=scripts/cross_model_verification
# Per-model effort vocabulary (#823): reject an unsupported explicit value before
# any request leaves; unset stays the provider default.
. "$GUARD/openai_effort_guard.sh"
ars_openai_effort_check "$ARS_CROSS_MODEL" "${ARS_CROSS_MODEL_REASONING_EFFORT:-}" || exit 1
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
resp="$(curl -sS -w '\n%{http_code}' https://api.openai.com/v1/responses \
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
2026-07-12 02:06:59 +08:00
-d "$(jq -n --arg model "$ARS_CROSS_MODEL" --arg prompt "$PROMPT" \
--arg effort "${ARS_CROSS_MODEL_REASONING_EFFORT:-}" '{
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
model: $model,
instructions: "You are a citation-verification assistant. Search the web before every verdict; never answer from memory. If you could not search, respond NOT_SEARCHED.",
input: $prompt,
fix: Astra request compat, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823–#826) (#827)
* fix: Astra request compatibility, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823 #824 #825 #826)
#823 — OpenAI request builders (smoke entrypoint + documented example) drop
`temperature`, which GPT-6 Astra rejects; the per-model effort vocabulary
lives in scripts/cross_model_verification/openai_effort_guard.sh, sourced by
both, and an unsupported explicit Astra value fails before curl. Hermetic
fake-curl test runs both surfaces.
#824 — the contained Codex citation transport rejects effort=ultra with
REASONING_EFFORT_REQUIRES_DELEGATION before detection/auth/tempdir/launch on
both entry paths (codex-cli 0.153.4 defines ultra as the multiAgentMode
replacement). Model-independent by design.
#825 — hedging can no longer rescue an unsupported claim (writer recovery
tree, CER fallback row, temporal rule 5 in writer + both compiler mirrors,
writer contract D2); universal prose quotas in the writer, compilers,
writing_quality_check.md, academic-paper/SKILL.md, and contract D6 become
diagnostics subordinate to author/venue requirements. Audit inventory
corrected in place; held-out seed evals/heldout/unsupported_claim_recovery
(NOT_RUN) registered.
#826 — run_codex_audit.sh pins gpt-6-astra/xhigh and records both in a new
sidecar `model` block; claim_audit_pipeline binds an unknown judge identity
to a run-local cache key (no cross-run reuse) instead of defaulting to
gpt-5.5-xhigh.
Review: /simplify (4 angles), codex gpt-5.6-sol xhigh 2 rounds (r1: 1 P1 +
1 P2 + 2 P3 fixed; r2: 0 P1/P2), /security-review 0 findings; all 102
spec-consistency steps + pytest manifest replayed locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNKiXpdHx1T5F5RbXT2Ueu
* docs(claude): record the #824 ultra reversal in the v3.21.2 key-additions line
The v3.21.2 bullet still said the contained Codex citation transport accepts
ultra; #824 on this branch rejects it as a delegation request. Add the
reversal so the live instruction surface matches the transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7emV5r2aqZDJzAyYVuuDo
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 08:08:30 +09:00
tools: [{type: "web_search"}]
2026-07-12 02:06:59 +08:00
} + (if $effort == "" then {} else {reasoning: {effort: $effort}} end)')")"
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
http="${resp##*$'\n'}"; body="${resp%$'\n'*}"
test(cross-model): behavior-test grounding guards + close malformed-response fail-opens (#349) (#350)
* test(cross-model): behavior-test grounding guards + fix Gemini source fail-open (#349)
The #346 grounding guards shipped as bash/jq inside cross_model_verification.md with no
automated test — a future jq edit or provider response-shape change could silently stop the
guard failing closed (the silent-false-VERIFIED class it exists to prevent). This extracts the
contract-bearing jq into canonical files, behavior-tests them, pins the doc wiring, and closes
a fail-open the extraction surfaced.
- Extract 5 canonical filters → scripts/cross_model_verification/*.jq (OpenAI search-guard /
text / sources; Gemini grounded-guard / sources). The doc loads them via `jq -f`, not inline.
- FAIL-OPEN FIX (Gemini sources): the extractor indexed groundingChunks by the model-supplied
groundingChunkIndices without validating them — a negative index silently picked a chunk from
the END (fabricating a real-but-wrong source URL that defeats the blank-source downgrade →
false VERIFIED resurrected), a string index crashed jq. Now admits only in-range non-negative
numeric indices, so a malformed support set yields blank sources → NOT_SEARCHED. First-party
verified the bug (negative index returned the last chunk) and the fix.
- test_cross_model_verification_guards.py: run each .jq against synthetic fixtures (grounded /
from-memory / negative-string-out-of-range index / non-2xx), + 2 mutation tests proving the
fixtures discriminate a working guard from accept-all / the pre-fix naive sources filter.
- check_cross_model_verification_sync.py: doc-sync lint — REQUIRED_FILTERS cross-checked against
the on-disk .jq set (a new filter can't escape the lint), every filter must be wired via
`jq -f`, NOT_SEARCHED / CROSS-MODEL-ERROR branches must remain, and no provider block may
re-inline a jq program referencing a grounding token. + its own mutation tests.
- CI: both wired into the pytest manifest + spec-consistency.yml; the workflow now ensures jq
is present on the runner (tests require jq, fail clear if absent — never skip a safety test).
Design approach (canonical .jq + behavior test + doc-sync lint) chosen after a cross-model
design consult; the fail-open was surfaced by that same consult and confirmed first-party.
/simplify applied (REQUIRED_FILTERS↔disk cross-check, regex -f detection, reword-proof re-inline
guard, parametrized mutation tests, cached jq lookup). spec + version consistency + 36-entry
pytest manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): harden jq guards against malformed response types (codex #349 review)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security). Security: 0 P1/P2
(verified the fail-open fix first-party, no shell injection, no doc-extracted shell in CI,
no PII leak). Codex: 0 P1, 2 P2, 3 P3 — all addressed.
The #349 index fix closed bad *indices* but the filters still trusted container/value TYPES.
First-party confirmed each hole before fixing:
- [P2] Gemini grounded-guard used `length > 0`, which is truthy for non-empty strings/objects —
so groundingSupports/webSearchQueries arriving as a string/object passed the guard. Now
requires non-empty ARRAYS.
- [P2] Gemini source extractor: a string `groundingChunks` crashed `$chunks[.]`; a non-string
`uri` (number) was emitted as a source. Now array-normalizes every container and filters URIs
to non-empty strings.
- [P2] OpenAI source extractor: a `url_citation.url` of true/123 was emitted as a fabricated
source; an object url crashed `join`. Now filters url to non-empty strings.
→ all yield blank SOURCES → NOT_SEARCHED, never a fabricated or crashing result.
- [P3] Parse test now asserts `rc == 0` (catches undefined-function compile errors a
"syntax error" stderr-substring check would miss).
- [P3] Doc-sync lint now scans executable bash lines only (comment-stripped) and both quote
styles, so a commented-out `jq -f` (filename surviving in a comment) or a double-quoted inline
grounding jq can no longer bypass it. + 2 regression tests.
- [P3] Dropped the stale "12 pytest invocations" workflow comment (manifest has 36).
+6 malformed-type behavior tests (Gemini non-array containers / non-string uri; OpenAI
non-string/object url). guards 14→20, lint 6→8. spec + version consistency + 36-entry manifest
green; pyflakes clean. CHANGELOG #349 entry broadened to the full malformed-response hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize top-level candidates/output + strip trailing comments (R2 #349)
R2 convergence pass (codex gpt-5.5/xhigh) on the R1 fix round found 3 P3; first-party
verified each, fixed 2, rejected 1 as a non-reproducible false positive.
- [P3, confirmed] Top-level `candidates` (Gemini) / `output` (OpenAI) were not array-normalized,
so a malformed response with them as an OBJECT passed the guard (.candidates[]? / .output[]?
iterate an object's values) and then crashed (.candidates[0] on an object) or leaked a nested
url in the source extractor. This contradicted the "every container is array-normalized" claim.
Now `arr(.candidates)` / `arr(.output)` in all five filters: a non-array top-level container
yields no items → guard false → NOT_SEARCHED, sources blank, no crash/leak. First-party
confirmed the object-output leak (https://leak surfaced) and the fix.
- [P3, confirmed] The doc-sync lint's bash-line extraction dropped whole-line comments but kept
trailing comments, so `jq -r ".x" # jq -f "$GUARD/sources.jq"` satisfied the wiring check via
the filename in the trailing comment. _bash_code_lines now strips trailing bash comments
(quote-aware: a `#` only counts at word boundary outside quotes) before the check.
- [P3, rejected] codex claimed the double-quoted inline-jq detector is not escape-aware and a
PoC bypasses it; first-party the PoC was still caught (rc=1) by the current DOTALL regex. Not
adding escape-handling complexity for a non-reproducible case (no unrequested flexibility).
+4 top-level-container behavior tests (guards 20→24) + 1 trailing-comment lint test (8→9).
spec + version consistency + 36-entry manifest green; pyflakes clean. The CHANGELOG's "every
container is array-normalized" claim is now literally accurate (top-level included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize nested OpenAI content/annotations — full path closed (R3 #349)
R3 convergence pass (codex gpt-5.5/xhigh) confirmed R2's top-level fixes work, and found the last
residual: `arr(.output)` normalized the top level but `.content[]?` / `.annotations[]?` still
iterated an object's values — a valid array `output` with a completed web_search_call but an
object-shaped `content` or `annotations` still leaked a nested url into SOURCES (and text from an
object content). First-party confirmed the leak (https://leak1 / https://leak2 surfaced) and the fix.
This is the bottom of the OpenAI container nesting: the path is output[] → content[] →
annotations[] → url(scalar). All three containers are now array-normalized and the url leaf is
already string-filtered (even a url arriving as an array-of-strings is rejected). No deeper level
exists, so this closes the malformed-container class for the OpenAI filters — not an open-ended loop.
- openai_sources.jq / openai_text.jq: `arr(.content)[]` + `arr(.annotations)[]`.
- +3 nested-container behavior tests (guards 24→27).
- CHANGELOG: the "every container is array-normalized" clause is made explicit about the full
per-path container set (OpenAI output→content→annotations; Gemini candidates→chunks/supports/
indices) so the claim is precise, not overbroad (the self-consistency point codex raised).
Threat model note: these malformed shapes come from the provider's own authenticated API
response, not an attacker — but the fix reuses the same one-line `arr` pattern, makes the
CHANGELOG claim literally true, and converges the malformed-container hardening. spec + version
consistency + 36-entry manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:20:11 +08:00
# The grounding guard and source extraction are kept as canonical jq filters under
# scripts/cross_model_verification/ so they are behavior-tested in CI (a from-memory verdict, a
# malformed grounding index, etc.) and cannot silently stop failing closed. Reference them via
fix: Astra request compat, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823–#826) (#827)
* fix: Astra request compatibility, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823 #824 #825 #826)
#823 — OpenAI request builders (smoke entrypoint + documented example) drop
`temperature`, which GPT-6 Astra rejects; the per-model effort vocabulary
lives in scripts/cross_model_verification/openai_effort_guard.sh, sourced by
both, and an unsupported explicit Astra value fails before curl. Hermetic
fake-curl test runs both surfaces.
#824 — the contained Codex citation transport rejects effort=ultra with
REASONING_EFFORT_REQUIRES_DELEGATION before detection/auth/tempdir/launch on
both entry paths (codex-cli 0.153.4 defines ultra as the multiAgentMode
replacement). Model-independent by design.
#825 — hedging can no longer rescue an unsupported claim (writer recovery
tree, CER fallback row, temporal rule 5 in writer + both compiler mirrors,
writer contract D2); universal prose quotas in the writer, compilers,
writing_quality_check.md, academic-paper/SKILL.md, and contract D6 become
diagnostics subordinate to author/venue requirements. Audit inventory
corrected in place; held-out seed evals/heldout/unsupported_claim_recovery
(NOT_RUN) registered.
#826 — run_codex_audit.sh pins gpt-6-astra/xhigh and records both in a new
sidecar `model` block; claim_audit_pipeline binds an unknown judge identity
to a run-local cache key (no cross-run reuse) instead of defaulting to
gpt-5.5-xhigh.
Review: /simplify (4 angles), codex gpt-5.6-sol xhigh 2 rounds (r1: 1 P1 +
1 P2 + 2 P3 fixed; r2: 0 P1/P2), /security-review 0 findings; all 102
spec-consistency steps + pytest manifest replayed locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNKiXpdHx1T5F5RbXT2Ueu
* docs(claude): record the #824 ultra reversal in the v3.21.2 key-additions line
The v3.21.2 bullet still said the contained Codex citation transport accepts
ultra; #824 on this branch rejects it as a delegation request. Add the
reversal so the live instruction surface matches the transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7emV5r2aqZDJzAyYVuuDo
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 08:08:30 +09:00
# ` jq -f` rather than inlining, so the doc and the test share one definition ($GUARD above).
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
if [ "$http" -lt 200 ] || [ "$http" -ge 300 ]; then
# Transport/API failure (401/429/5xx, or curl's 000 on a network error) — NOT the same as
# "searched but found nothing". Surface as a transport error so the consumer falls back to
# single-model (see § Graceful Degradation); never relabel it NOT_SEARCHED, which would
# imply a completed-but-ungrounded lookup.
echo "CROSS-MODEL-ERROR: openai_http_$http"
test(cross-model): behavior-test grounding guards + close malformed-response fail-opens (#349) (#350)
* test(cross-model): behavior-test grounding guards + fix Gemini source fail-open (#349)
The #346 grounding guards shipped as bash/jq inside cross_model_verification.md with no
automated test — a future jq edit or provider response-shape change could silently stop the
guard failing closed (the silent-false-VERIFIED class it exists to prevent). This extracts the
contract-bearing jq into canonical files, behavior-tests them, pins the doc wiring, and closes
a fail-open the extraction surfaced.
- Extract 5 canonical filters → scripts/cross_model_verification/*.jq (OpenAI search-guard /
text / sources; Gemini grounded-guard / sources). The doc loads them via `jq -f`, not inline.
- FAIL-OPEN FIX (Gemini sources): the extractor indexed groundingChunks by the model-supplied
groundingChunkIndices without validating them — a negative index silently picked a chunk from
the END (fabricating a real-but-wrong source URL that defeats the blank-source downgrade →
false VERIFIED resurrected), a string index crashed jq. Now admits only in-range non-negative
numeric indices, so a malformed support set yields blank sources → NOT_SEARCHED. First-party
verified the bug (negative index returned the last chunk) and the fix.
- test_cross_model_verification_guards.py: run each .jq against synthetic fixtures (grounded /
from-memory / negative-string-out-of-range index / non-2xx), + 2 mutation tests proving the
fixtures discriminate a working guard from accept-all / the pre-fix naive sources filter.
- check_cross_model_verification_sync.py: doc-sync lint — REQUIRED_FILTERS cross-checked against
the on-disk .jq set (a new filter can't escape the lint), every filter must be wired via
`jq -f`, NOT_SEARCHED / CROSS-MODEL-ERROR branches must remain, and no provider block may
re-inline a jq program referencing a grounding token. + its own mutation tests.
- CI: both wired into the pytest manifest + spec-consistency.yml; the workflow now ensures jq
is present on the runner (tests require jq, fail clear if absent — never skip a safety test).
Design approach (canonical .jq + behavior test + doc-sync lint) chosen after a cross-model
design consult; the fail-open was surfaced by that same consult and confirmed first-party.
/simplify applied (REQUIRED_FILTERS↔disk cross-check, regex -f detection, reword-proof re-inline
guard, parametrized mutation tests, cached jq lookup). spec + version consistency + 36-entry
pytest manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): harden jq guards against malformed response types (codex #349 review)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security). Security: 0 P1/P2
(verified the fail-open fix first-party, no shell injection, no doc-extracted shell in CI,
no PII leak). Codex: 0 P1, 2 P2, 3 P3 — all addressed.
The #349 index fix closed bad *indices* but the filters still trusted container/value TYPES.
First-party confirmed each hole before fixing:
- [P2] Gemini grounded-guard used `length > 0`, which is truthy for non-empty strings/objects —
so groundingSupports/webSearchQueries arriving as a string/object passed the guard. Now
requires non-empty ARRAYS.
- [P2] Gemini source extractor: a string `groundingChunks` crashed `$chunks[.]`; a non-string
`uri` (number) was emitted as a source. Now array-normalizes every container and filters URIs
to non-empty strings.
- [P2] OpenAI source extractor: a `url_citation.url` of true/123 was emitted as a fabricated
source; an object url crashed `join`. Now filters url to non-empty strings.
→ all yield blank SOURCES → NOT_SEARCHED, never a fabricated or crashing result.
- [P3] Parse test now asserts `rc == 0` (catches undefined-function compile errors a
"syntax error" stderr-substring check would miss).
- [P3] Doc-sync lint now scans executable bash lines only (comment-stripped) and both quote
styles, so a commented-out `jq -f` (filename surviving in a comment) or a double-quoted inline
grounding jq can no longer bypass it. + 2 regression tests.
- [P3] Dropped the stale "12 pytest invocations" workflow comment (manifest has 36).
+6 malformed-type behavior tests (Gemini non-array containers / non-string uri; OpenAI
non-string/object url). guards 14→20, lint 6→8. spec + version consistency + 36-entry manifest
green; pyflakes clean. CHANGELOG #349 entry broadened to the full malformed-response hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize top-level candidates/output + strip trailing comments (R2 #349)
R2 convergence pass (codex gpt-5.5/xhigh) on the R1 fix round found 3 P3; first-party
verified each, fixed 2, rejected 1 as a non-reproducible false positive.
- [P3, confirmed] Top-level `candidates` (Gemini) / `output` (OpenAI) were not array-normalized,
so a malformed response with them as an OBJECT passed the guard (.candidates[]? / .output[]?
iterate an object's values) and then crashed (.candidates[0] on an object) or leaked a nested
url in the source extractor. This contradicted the "every container is array-normalized" claim.
Now `arr(.candidates)` / `arr(.output)` in all five filters: a non-array top-level container
yields no items → guard false → NOT_SEARCHED, sources blank, no crash/leak. First-party
confirmed the object-output leak (https://leak surfaced) and the fix.
- [P3, confirmed] The doc-sync lint's bash-line extraction dropped whole-line comments but kept
trailing comments, so `jq -r ".x" # jq -f "$GUARD/sources.jq"` satisfied the wiring check via
the filename in the trailing comment. _bash_code_lines now strips trailing bash comments
(quote-aware: a `#` only counts at word boundary outside quotes) before the check.
- [P3, rejected] codex claimed the double-quoted inline-jq detector is not escape-aware and a
PoC bypasses it; first-party the PoC was still caught (rc=1) by the current DOTALL regex. Not
adding escape-handling complexity for a non-reproducible case (no unrequested flexibility).
+4 top-level-container behavior tests (guards 20→24) + 1 trailing-comment lint test (8→9).
spec + version consistency + 36-entry manifest green; pyflakes clean. The CHANGELOG's "every
container is array-normalized" claim is now literally accurate (top-level included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize nested OpenAI content/annotations — full path closed (R3 #349)
R3 convergence pass (codex gpt-5.5/xhigh) confirmed R2's top-level fixes work, and found the last
residual: `arr(.output)` normalized the top level but `.content[]?` / `.annotations[]?` still
iterated an object's values — a valid array `output` with a completed web_search_call but an
object-shaped `content` or `annotations` still leaked a nested url into SOURCES (and text from an
object content). First-party confirmed the leak (https://leak1 / https://leak2 surfaced) and the fix.
This is the bottom of the OpenAI container nesting: the path is output[] → content[] →
annotations[] → url(scalar). All three containers are now array-normalized and the url leaf is
already string-filtered (even a url arriving as an array-of-strings is rejected). No deeper level
exists, so this closes the malformed-container class for the OpenAI filters — not an open-ended loop.
- openai_sources.jq / openai_text.jq: `arr(.content)[]` + `arr(.annotations)[]`.
- +3 nested-container behavior tests (guards 24→27).
- CHANGELOG: the "every container is array-normalized" clause is made explicit about the full
per-path container set (OpenAI output→content→annotations; Gemini candidates→chunks/supports/
indices) so the claim is precise, not overbroad (the self-consistency point codex raised).
Threat model note: these malformed shapes come from the provider's own authenticated API
response, not an attacker — but the fix reuses the same one-line `arr` pattern, makes the
CHANGELOG claim literally true, and converges the malformed-container hardening. spec + version
consistency + 36-entry manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:20:11 +08:00
elif ! jq -e -f "$GUARD/openai_has_completed_web_search.jq" <<<"$body" >/dev/null; then
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
echo "NOT_SEARCHED: no_web_search_call" # no search happened at all — discard the text
else
# A completed web_search_call proves *a* search ran, not that THIS reference's verdict
# is supported by it. Emit the verdict text together with the url_citation annotations the
# model attached; step 5 downgrades a VERIFIED with no citation to NOT_SEARCHED.
test(cross-model): behavior-test grounding guards + close malformed-response fail-opens (#349) (#350)
* test(cross-model): behavior-test grounding guards + fix Gemini source fail-open (#349)
The #346 grounding guards shipped as bash/jq inside cross_model_verification.md with no
automated test — a future jq edit or provider response-shape change could silently stop the
guard failing closed (the silent-false-VERIFIED class it exists to prevent). This extracts the
contract-bearing jq into canonical files, behavior-tests them, pins the doc wiring, and closes
a fail-open the extraction surfaced.
- Extract 5 canonical filters → scripts/cross_model_verification/*.jq (OpenAI search-guard /
text / sources; Gemini grounded-guard / sources). The doc loads them via `jq -f`, not inline.
- FAIL-OPEN FIX (Gemini sources): the extractor indexed groundingChunks by the model-supplied
groundingChunkIndices without validating them — a negative index silently picked a chunk from
the END (fabricating a real-but-wrong source URL that defeats the blank-source downgrade →
false VERIFIED resurrected), a string index crashed jq. Now admits only in-range non-negative
numeric indices, so a malformed support set yields blank sources → NOT_SEARCHED. First-party
verified the bug (negative index returned the last chunk) and the fix.
- test_cross_model_verification_guards.py: run each .jq against synthetic fixtures (grounded /
from-memory / negative-string-out-of-range index / non-2xx), + 2 mutation tests proving the
fixtures discriminate a working guard from accept-all / the pre-fix naive sources filter.
- check_cross_model_verification_sync.py: doc-sync lint — REQUIRED_FILTERS cross-checked against
the on-disk .jq set (a new filter can't escape the lint), every filter must be wired via
`jq -f`, NOT_SEARCHED / CROSS-MODEL-ERROR branches must remain, and no provider block may
re-inline a jq program referencing a grounding token. + its own mutation tests.
- CI: both wired into the pytest manifest + spec-consistency.yml; the workflow now ensures jq
is present on the runner (tests require jq, fail clear if absent — never skip a safety test).
Design approach (canonical .jq + behavior test + doc-sync lint) chosen after a cross-model
design consult; the fail-open was surfaced by that same consult and confirmed first-party.
/simplify applied (REQUIRED_FILTERS↔disk cross-check, regex -f detection, reword-proof re-inline
guard, parametrized mutation tests, cached jq lookup). spec + version consistency + 36-entry
pytest manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): harden jq guards against malformed response types (codex #349 review)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security). Security: 0 P1/P2
(verified the fail-open fix first-party, no shell injection, no doc-extracted shell in CI,
no PII leak). Codex: 0 P1, 2 P2, 3 P3 — all addressed.
The #349 index fix closed bad *indices* but the filters still trusted container/value TYPES.
First-party confirmed each hole before fixing:
- [P2] Gemini grounded-guard used `length > 0`, which is truthy for non-empty strings/objects —
so groundingSupports/webSearchQueries arriving as a string/object passed the guard. Now
requires non-empty ARRAYS.
- [P2] Gemini source extractor: a string `groundingChunks` crashed `$chunks[.]`; a non-string
`uri` (number) was emitted as a source. Now array-normalizes every container and filters URIs
to non-empty strings.
- [P2] OpenAI source extractor: a `url_citation.url` of true/123 was emitted as a fabricated
source; an object url crashed `join`. Now filters url to non-empty strings.
→ all yield blank SOURCES → NOT_SEARCHED, never a fabricated or crashing result.
- [P3] Parse test now asserts `rc == 0` (catches undefined-function compile errors a
"syntax error" stderr-substring check would miss).
- [P3] Doc-sync lint now scans executable bash lines only (comment-stripped) and both quote
styles, so a commented-out `jq -f` (filename surviving in a comment) or a double-quoted inline
grounding jq can no longer bypass it. + 2 regression tests.
- [P3] Dropped the stale "12 pytest invocations" workflow comment (manifest has 36).
+6 malformed-type behavior tests (Gemini non-array containers / non-string uri; OpenAI
non-string/object url). guards 14→20, lint 6→8. spec + version consistency + 36-entry manifest
green; pyflakes clean. CHANGELOG #349 entry broadened to the full malformed-response hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize top-level candidates/output + strip trailing comments (R2 #349)
R2 convergence pass (codex gpt-5.5/xhigh) on the R1 fix round found 3 P3; first-party
verified each, fixed 2, rejected 1 as a non-reproducible false positive.
- [P3, confirmed] Top-level `candidates` (Gemini) / `output` (OpenAI) were not array-normalized,
so a malformed response with them as an OBJECT passed the guard (.candidates[]? / .output[]?
iterate an object's values) and then crashed (.candidates[0] on an object) or leaked a nested
url in the source extractor. This contradicted the "every container is array-normalized" claim.
Now `arr(.candidates)` / `arr(.output)` in all five filters: a non-array top-level container
yields no items → guard false → NOT_SEARCHED, sources blank, no crash/leak. First-party
confirmed the object-output leak (https://leak surfaced) and the fix.
- [P3, confirmed] The doc-sync lint's bash-line extraction dropped whole-line comments but kept
trailing comments, so `jq -r ".x" # jq -f "$GUARD/sources.jq"` satisfied the wiring check via
the filename in the trailing comment. _bash_code_lines now strips trailing bash comments
(quote-aware: a `#` only counts at word boundary outside quotes) before the check.
- [P3, rejected] codex claimed the double-quoted inline-jq detector is not escape-aware and a
PoC bypasses it; first-party the PoC was still caught (rc=1) by the current DOTALL regex. Not
adding escape-handling complexity for a non-reproducible case (no unrequested flexibility).
+4 top-level-container behavior tests (guards 20→24) + 1 trailing-comment lint test (8→9).
spec + version consistency + 36-entry manifest green; pyflakes clean. The CHANGELOG's "every
container is array-normalized" claim is now literally accurate (top-level included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize nested OpenAI content/annotations — full path closed (R3 #349)
R3 convergence pass (codex gpt-5.5/xhigh) confirmed R2's top-level fixes work, and found the last
residual: `arr(.output)` normalized the top level but `.content[]?` / `.annotations[]?` still
iterated an object's values — a valid array `output` with a completed web_search_call but an
object-shaped `content` or `annotations` still leaked a nested url into SOURCES (and text from an
object content). First-party confirmed the leak (https://leak1 / https://leak2 surfaced) and the fix.
This is the bottom of the OpenAI container nesting: the path is output[] → content[] →
annotations[] → url(scalar). All three containers are now array-normalized and the url leaf is
already string-filtered (even a url arriving as an array-of-strings is rejected). No deeper level
exists, so this closes the malformed-container class for the OpenAI filters — not an open-ended loop.
- openai_sources.jq / openai_text.jq: `arr(.content)[]` + `arr(.annotations)[]`.
- +3 nested-container behavior tests (guards 24→27).
- CHANGELOG: the "every container is array-normalized" clause is made explicit about the full
per-path container set (OpenAI output→content→annotations; Gemini candidates→chunks/supports/
indices) so the claim is precise, not overbroad (the self-consistency point codex raised).
Threat model note: these malformed shapes come from the provider's own authenticated API
response, not an attacker — but the fix reuses the same one-line `arr` pattern, makes the
CHANGELOG claim literally true, and converges the malformed-container hardening. spec + version
consistency + 36-entry manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:20:11 +08:00
text="$(jq -r -f "$GUARD/openai_text.jq" <<<"$body")"
cites="$(jq -r -f "$GUARD/openai_sources.jq" <<<"$body")"
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
printf '%s\nSOURCES: %s\n' "$text" "${cites:-(none)}"
fi
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
` ``
### Google Gemini (Gemini 3.1 Pro)
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
The hosted grounding tool is ` google_search` (REST uses snake_case; the JS SDK's ` googleSearch` is the same tool). A grounded response carries ` candidates[].groundingMetadata`; its absence means the model did not search.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
` ``bash
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
# PROMPT holds the single-reference verification prompt (step 3). One reference per call.
resp="$(curl -sS -w '\n%{http_code}' \
"https://generativelanguage.googleapis.com/v1beta/models/${ARS_CROSS_MODEL}:generateContent?key=$GOOGLE_AI_API_KEY" \
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
-H "Content-Type: application/json" \
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
-d "$(jq -n --arg prompt "$PROMPT" '{
contents: [{parts: [{text: $prompt}]}],
tools: [{google_search: {}}],
generationConfig: {temperature: 0.1}
}')")"
http="${resp##*$'\n'}"; body="${resp%$'\n'*}"
test(cross-model): behavior-test grounding guards + close malformed-response fail-opens (#349) (#350)
* test(cross-model): behavior-test grounding guards + fix Gemini source fail-open (#349)
The #346 grounding guards shipped as bash/jq inside cross_model_verification.md with no
automated test — a future jq edit or provider response-shape change could silently stop the
guard failing closed (the silent-false-VERIFIED class it exists to prevent). This extracts the
contract-bearing jq into canonical files, behavior-tests them, pins the doc wiring, and closes
a fail-open the extraction surfaced.
- Extract 5 canonical filters → scripts/cross_model_verification/*.jq (OpenAI search-guard /
text / sources; Gemini grounded-guard / sources). The doc loads them via `jq -f`, not inline.
- FAIL-OPEN FIX (Gemini sources): the extractor indexed groundingChunks by the model-supplied
groundingChunkIndices without validating them — a negative index silently picked a chunk from
the END (fabricating a real-but-wrong source URL that defeats the blank-source downgrade →
false VERIFIED resurrected), a string index crashed jq. Now admits only in-range non-negative
numeric indices, so a malformed support set yields blank sources → NOT_SEARCHED. First-party
verified the bug (negative index returned the last chunk) and the fix.
- test_cross_model_verification_guards.py: run each .jq against synthetic fixtures (grounded /
from-memory / negative-string-out-of-range index / non-2xx), + 2 mutation tests proving the
fixtures discriminate a working guard from accept-all / the pre-fix naive sources filter.
- check_cross_model_verification_sync.py: doc-sync lint — REQUIRED_FILTERS cross-checked against
the on-disk .jq set (a new filter can't escape the lint), every filter must be wired via
`jq -f`, NOT_SEARCHED / CROSS-MODEL-ERROR branches must remain, and no provider block may
re-inline a jq program referencing a grounding token. + its own mutation tests.
- CI: both wired into the pytest manifest + spec-consistency.yml; the workflow now ensures jq
is present on the runner (tests require jq, fail clear if absent — never skip a safety test).
Design approach (canonical .jq + behavior test + doc-sync lint) chosen after a cross-model
design consult; the fail-open was surfaced by that same consult and confirmed first-party.
/simplify applied (REQUIRED_FILTERS↔disk cross-check, regex -f detection, reword-proof re-inline
guard, parametrized mutation tests, cached jq lookup). spec + version consistency + 36-entry
pytest manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): harden jq guards against malformed response types (codex #349 review)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security). Security: 0 P1/P2
(verified the fail-open fix first-party, no shell injection, no doc-extracted shell in CI,
no PII leak). Codex: 0 P1, 2 P2, 3 P3 — all addressed.
The #349 index fix closed bad *indices* but the filters still trusted container/value TYPES.
First-party confirmed each hole before fixing:
- [P2] Gemini grounded-guard used `length > 0`, which is truthy for non-empty strings/objects —
so groundingSupports/webSearchQueries arriving as a string/object passed the guard. Now
requires non-empty ARRAYS.
- [P2] Gemini source extractor: a string `groundingChunks` crashed `$chunks[.]`; a non-string
`uri` (number) was emitted as a source. Now array-normalizes every container and filters URIs
to non-empty strings.
- [P2] OpenAI source extractor: a `url_citation.url` of true/123 was emitted as a fabricated
source; an object url crashed `join`. Now filters url to non-empty strings.
→ all yield blank SOURCES → NOT_SEARCHED, never a fabricated or crashing result.
- [P3] Parse test now asserts `rc == 0` (catches undefined-function compile errors a
"syntax error" stderr-substring check would miss).
- [P3] Doc-sync lint now scans executable bash lines only (comment-stripped) and both quote
styles, so a commented-out `jq -f` (filename surviving in a comment) or a double-quoted inline
grounding jq can no longer bypass it. + 2 regression tests.
- [P3] Dropped the stale "12 pytest invocations" workflow comment (manifest has 36).
+6 malformed-type behavior tests (Gemini non-array containers / non-string uri; OpenAI
non-string/object url). guards 14→20, lint 6→8. spec + version consistency + 36-entry manifest
green; pyflakes clean. CHANGELOG #349 entry broadened to the full malformed-response hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize top-level candidates/output + strip trailing comments (R2 #349)
R2 convergence pass (codex gpt-5.5/xhigh) on the R1 fix round found 3 P3; first-party
verified each, fixed 2, rejected 1 as a non-reproducible false positive.
- [P3, confirmed] Top-level `candidates` (Gemini) / `output` (OpenAI) were not array-normalized,
so a malformed response with them as an OBJECT passed the guard (.candidates[]? / .output[]?
iterate an object's values) and then crashed (.candidates[0] on an object) or leaked a nested
url in the source extractor. This contradicted the "every container is array-normalized" claim.
Now `arr(.candidates)` / `arr(.output)` in all five filters: a non-array top-level container
yields no items → guard false → NOT_SEARCHED, sources blank, no crash/leak. First-party
confirmed the object-output leak (https://leak surfaced) and the fix.
- [P3, confirmed] The doc-sync lint's bash-line extraction dropped whole-line comments but kept
trailing comments, so `jq -r ".x" # jq -f "$GUARD/sources.jq"` satisfied the wiring check via
the filename in the trailing comment. _bash_code_lines now strips trailing bash comments
(quote-aware: a `#` only counts at word boundary outside quotes) before the check.
- [P3, rejected] codex claimed the double-quoted inline-jq detector is not escape-aware and a
PoC bypasses it; first-party the PoC was still caught (rc=1) by the current DOTALL regex. Not
adding escape-handling complexity for a non-reproducible case (no unrequested flexibility).
+4 top-level-container behavior tests (guards 20→24) + 1 trailing-comment lint test (8→9).
spec + version consistency + 36-entry manifest green; pyflakes clean. The CHANGELOG's "every
container is array-normalized" claim is now literally accurate (top-level included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize nested OpenAI content/annotations — full path closed (R3 #349)
R3 convergence pass (codex gpt-5.5/xhigh) confirmed R2's top-level fixes work, and found the last
residual: `arr(.output)` normalized the top level but `.content[]?` / `.annotations[]?` still
iterated an object's values — a valid array `output` with a completed web_search_call but an
object-shaped `content` or `annotations` still leaked a nested url into SOURCES (and text from an
object content). First-party confirmed the leak (https://leak1 / https://leak2 surfaced) and the fix.
This is the bottom of the OpenAI container nesting: the path is output[] → content[] →
annotations[] → url(scalar). All three containers are now array-normalized and the url leaf is
already string-filtered (even a url arriving as an array-of-strings is rejected). No deeper level
exists, so this closes the malformed-container class for the OpenAI filters — not an open-ended loop.
- openai_sources.jq / openai_text.jq: `arr(.content)[]` + `arr(.annotations)[]`.
- +3 nested-container behavior tests (guards 24→27).
- CHANGELOG: the "every container is array-normalized" clause is made explicit about the full
per-path container set (OpenAI output→content→annotations; Gemini candidates→chunks/supports/
indices) so the claim is precise, not overbroad (the self-consistency point codex raised).
Threat model note: these malformed shapes come from the provider's own authenticated API
response, not an attacker — but the fix reuses the same one-line `arr` pattern, makes the
CHANGELOG claim literally true, and converges the malformed-container hardening. spec + version
consistency + 36-entry manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:20:11 +08:00
# Grounding guard + source extraction are canonical jq filters under scripts/cross_model_verification/
fix(cross-model): Gemini guard requires valid supported index + fail-closed hardening (#351) (#353)
Closes #351
## Context
Post-squash integration review of #349 (auto-opened as #351) surfaced residual malformed-response paths the four #349 review rounds missed. Every finding across the review rounds was reproduced first-party before fixing.
## Findings fixed
**[P2] `gemini_is_grounded.jq` false-passed an ungrounded verdict.** The guard only checked that `groundingSupports` was a non-empty array — but a `groundingSupports: [{}]`, or one whose only `groundingChunkIndices` are empty / negative / string / fractional / out-of-range, links the verdict to **no** retrieved chunk. The source filter blanks those indices, but the blank-source downgrade only rescues `VERIFIED`, so an ungrounded `NOT_FOUND` / `MISMATCH` could still be trusted as a grounded result.
**[P3] `openai_text.jq` crashed on a non-string `text`.** An `output_text.text` that is an object made `join("\n")` exit with jq error 5. It now filters `.text` to strings, mirroring the URL string-filter.
## Structural fix (guard derives from extractor)
Rather than keep the guard and the source extractor as two parallel predicates asserted to agree (which kept diverging on malformed input), `gemini_is_grounded.jq` now **embeds the exact extraction `gemini_sources.jq` performs** — same `candidates[0]` selection, same valid-index predicate, same non-empty-string `uri` filter — and passes iff that extraction yields ≥1 url **and** `webSearchQueries` is non-empty. So divergence is impossible by construction.
The safety invariant is **one-directional**: `guard-pass ⟹ ≥1 source extractable` (NOT the converse — a response carrying valid chunks but an empty `webSearchQueries` extracts a source yet correctly fails the guard, because a real search signal is also required).
## Fail-closed hardening
Both Gemini filters now array-normalize every container (`arr/1`) **and** object-normalize every value that is then field-dereferenced (`obj/1`), including the root `.candidates` access. A non-object at any point — root, `groundingMetadata`, a `groundingSupports` element, a cited `groundingChunks` element, or its `web` — yields blank extraction / a clean non-grounded verdict instead of crashing jq (exit 5). Verified by an exhaustive fuzz over every JSON primitive at every nesting level (including non-object roots): zero crashes.
## Tests
Guard behavior tests grow to 61. New coverage: the no-valid-index false-pass (×5: empty / negative / string / fractional / out-of-range / bare `{}`), the guard-derives-from-extractor cases, the one-directional `guard-pass ⟹ sources-nonblank` invariant, the OpenAI text-crash path, the non-object array-element cases, the Gemini non-object-dereference cases (×5), and the non-object-root cases (×7). A mutation test confirms the fixtures are not vacuously green.
spec + version consistency + the cross-model doc-sync lint + pytest manifest green; pyflakes clean.
This is a follow-up hardening of the #346/#349 guards (same canonical `.jq` files); no agent-prompt or schema change.
2026-06-06 14:19:18 +08:00
# (same rationale as the OpenAI block: behavior-tested, referenced via ` jq -f`). The guard is
# rederived from the source extractor: it passes iff the SAME extraction the source filter performs
# yields at least one url AND the model issued a search (a non-empty webSearchQueries). So
# guard-pass ⟹ a source is extractable — a groundingSupports linking to no valid chunk
# (empty/negative/string/out-of-range/fractional index), the wrong candidate, or a non-string uri
# all leave the extraction blank and fail the guard closed. See the .jq file headers for the full
# contract.
test(cross-model): behavior-test grounding guards + close malformed-response fail-opens (#349) (#350)
* test(cross-model): behavior-test grounding guards + fix Gemini source fail-open (#349)
The #346 grounding guards shipped as bash/jq inside cross_model_verification.md with no
automated test — a future jq edit or provider response-shape change could silently stop the
guard failing closed (the silent-false-VERIFIED class it exists to prevent). This extracts the
contract-bearing jq into canonical files, behavior-tests them, pins the doc wiring, and closes
a fail-open the extraction surfaced.
- Extract 5 canonical filters → scripts/cross_model_verification/*.jq (OpenAI search-guard /
text / sources; Gemini grounded-guard / sources). The doc loads them via `jq -f`, not inline.
- FAIL-OPEN FIX (Gemini sources): the extractor indexed groundingChunks by the model-supplied
groundingChunkIndices without validating them — a negative index silently picked a chunk from
the END (fabricating a real-but-wrong source URL that defeats the blank-source downgrade →
false VERIFIED resurrected), a string index crashed jq. Now admits only in-range non-negative
numeric indices, so a malformed support set yields blank sources → NOT_SEARCHED. First-party
verified the bug (negative index returned the last chunk) and the fix.
- test_cross_model_verification_guards.py: run each .jq against synthetic fixtures (grounded /
from-memory / negative-string-out-of-range index / non-2xx), + 2 mutation tests proving the
fixtures discriminate a working guard from accept-all / the pre-fix naive sources filter.
- check_cross_model_verification_sync.py: doc-sync lint — REQUIRED_FILTERS cross-checked against
the on-disk .jq set (a new filter can't escape the lint), every filter must be wired via
`jq -f`, NOT_SEARCHED / CROSS-MODEL-ERROR branches must remain, and no provider block may
re-inline a jq program referencing a grounding token. + its own mutation tests.
- CI: both wired into the pytest manifest + spec-consistency.yml; the workflow now ensures jq
is present on the runner (tests require jq, fail clear if absent — never skip a safety test).
Design approach (canonical .jq + behavior test + doc-sync lint) chosen after a cross-model
design consult; the fail-open was surfaced by that same consult and confirmed first-party.
/simplify applied (REQUIRED_FILTERS↔disk cross-check, regex -f detection, reword-proof re-inline
guard, parametrized mutation tests, cached jq lookup). spec + version consistency + 36-entry
pytest manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): harden jq guards against malformed response types (codex #349 review)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security). Security: 0 P1/P2
(verified the fail-open fix first-party, no shell injection, no doc-extracted shell in CI,
no PII leak). Codex: 0 P1, 2 P2, 3 P3 — all addressed.
The #349 index fix closed bad *indices* but the filters still trusted container/value TYPES.
First-party confirmed each hole before fixing:
- [P2] Gemini grounded-guard used `length > 0`, which is truthy for non-empty strings/objects —
so groundingSupports/webSearchQueries arriving as a string/object passed the guard. Now
requires non-empty ARRAYS.
- [P2] Gemini source extractor: a string `groundingChunks` crashed `$chunks[.]`; a non-string
`uri` (number) was emitted as a source. Now array-normalizes every container and filters URIs
to non-empty strings.
- [P2] OpenAI source extractor: a `url_citation.url` of true/123 was emitted as a fabricated
source; an object url crashed `join`. Now filters url to non-empty strings.
→ all yield blank SOURCES → NOT_SEARCHED, never a fabricated or crashing result.
- [P3] Parse test now asserts `rc == 0` (catches undefined-function compile errors a
"syntax error" stderr-substring check would miss).
- [P3] Doc-sync lint now scans executable bash lines only (comment-stripped) and both quote
styles, so a commented-out `jq -f` (filename surviving in a comment) or a double-quoted inline
grounding jq can no longer bypass it. + 2 regression tests.
- [P3] Dropped the stale "12 pytest invocations" workflow comment (manifest has 36).
+6 malformed-type behavior tests (Gemini non-array containers / non-string uri; OpenAI
non-string/object url). guards 14→20, lint 6→8. spec + version consistency + 36-entry manifest
green; pyflakes clean. CHANGELOG #349 entry broadened to the full malformed-response hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize top-level candidates/output + strip trailing comments (R2 #349)
R2 convergence pass (codex gpt-5.5/xhigh) on the R1 fix round found 3 P3; first-party
verified each, fixed 2, rejected 1 as a non-reproducible false positive.
- [P3, confirmed] Top-level `candidates` (Gemini) / `output` (OpenAI) were not array-normalized,
so a malformed response with them as an OBJECT passed the guard (.candidates[]? / .output[]?
iterate an object's values) and then crashed (.candidates[0] on an object) or leaked a nested
url in the source extractor. This contradicted the "every container is array-normalized" claim.
Now `arr(.candidates)` / `arr(.output)` in all five filters: a non-array top-level container
yields no items → guard false → NOT_SEARCHED, sources blank, no crash/leak. First-party
confirmed the object-output leak (https://leak surfaced) and the fix.
- [P3, confirmed] The doc-sync lint's bash-line extraction dropped whole-line comments but kept
trailing comments, so `jq -r ".x" # jq -f "$GUARD/sources.jq"` satisfied the wiring check via
the filename in the trailing comment. _bash_code_lines now strips trailing bash comments
(quote-aware: a `#` only counts at word boundary outside quotes) before the check.
- [P3, rejected] codex claimed the double-quoted inline-jq detector is not escape-aware and a
PoC bypasses it; first-party the PoC was still caught (rc=1) by the current DOTALL regex. Not
adding escape-handling complexity for a non-reproducible case (no unrequested flexibility).
+4 top-level-container behavior tests (guards 20→24) + 1 trailing-comment lint test (8→9).
spec + version consistency + 36-entry manifest green; pyflakes clean. The CHANGELOG's "every
container is array-normalized" claim is now literally accurate (top-level included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize nested OpenAI content/annotations — full path closed (R3 #349)
R3 convergence pass (codex gpt-5.5/xhigh) confirmed R2's top-level fixes work, and found the last
residual: `arr(.output)` normalized the top level but `.content[]?` / `.annotations[]?` still
iterated an object's values — a valid array `output` with a completed web_search_call but an
object-shaped `content` or `annotations` still leaked a nested url into SOURCES (and text from an
object content). First-party confirmed the leak (https://leak1 / https://leak2 surfaced) and the fix.
This is the bottom of the OpenAI container nesting: the path is output[] → content[] →
annotations[] → url(scalar). All three containers are now array-normalized and the url leaf is
already string-filtered (even a url arriving as an array-of-strings is rejected). No deeper level
exists, so this closes the malformed-container class for the OpenAI filters — not an open-ended loop.
- openai_sources.jq / openai_text.jq: `arr(.content)[]` + `arr(.annotations)[]`.
- +3 nested-container behavior tests (guards 24→27).
- CHANGELOG: the "every container is array-normalized" clause is made explicit about the full
per-path container set (OpenAI output→content→annotations; Gemini candidates→chunks/supports/
indices) so the claim is precise, not overbroad (the self-consistency point codex raised).
Threat model note: these malformed shapes come from the provider's own authenticated API
response, not an attacker — but the fix reuses the same one-line `arr` pattern, makes the
CHANGELOG claim literally true, and converges the malformed-container hardening. spec + version
consistency + 36-entry manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:20:11 +08:00
GUARD=scripts/cross_model_verification
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
if [ "$http" -lt 200 ] || [ "$http" -ge 300 ]; then
# Transport/API failure (401/429/5xx, or curl's 000) — surface as a transport error so the
# consumer falls back to single-model (see § Graceful Degradation), not NOT_SEARCHED.
echo "CROSS-MODEL-ERROR: gemini_http_$http"
test(cross-model): behavior-test grounding guards + close malformed-response fail-opens (#349) (#350)
* test(cross-model): behavior-test grounding guards + fix Gemini source fail-open (#349)
The #346 grounding guards shipped as bash/jq inside cross_model_verification.md with no
automated test — a future jq edit or provider response-shape change could silently stop the
guard failing closed (the silent-false-VERIFIED class it exists to prevent). This extracts the
contract-bearing jq into canonical files, behavior-tests them, pins the doc wiring, and closes
a fail-open the extraction surfaced.
- Extract 5 canonical filters → scripts/cross_model_verification/*.jq (OpenAI search-guard /
text / sources; Gemini grounded-guard / sources). The doc loads them via `jq -f`, not inline.
- FAIL-OPEN FIX (Gemini sources): the extractor indexed groundingChunks by the model-supplied
groundingChunkIndices without validating them — a negative index silently picked a chunk from
the END (fabricating a real-but-wrong source URL that defeats the blank-source downgrade →
false VERIFIED resurrected), a string index crashed jq. Now admits only in-range non-negative
numeric indices, so a malformed support set yields blank sources → NOT_SEARCHED. First-party
verified the bug (negative index returned the last chunk) and the fix.
- test_cross_model_verification_guards.py: run each .jq against synthetic fixtures (grounded /
from-memory / negative-string-out-of-range index / non-2xx), + 2 mutation tests proving the
fixtures discriminate a working guard from accept-all / the pre-fix naive sources filter.
- check_cross_model_verification_sync.py: doc-sync lint — REQUIRED_FILTERS cross-checked against
the on-disk .jq set (a new filter can't escape the lint), every filter must be wired via
`jq -f`, NOT_SEARCHED / CROSS-MODEL-ERROR branches must remain, and no provider block may
re-inline a jq program referencing a grounding token. + its own mutation tests.
- CI: both wired into the pytest manifest + spec-consistency.yml; the workflow now ensures jq
is present on the runner (tests require jq, fail clear if absent — never skip a safety test).
Design approach (canonical .jq + behavior test + doc-sync lint) chosen after a cross-model
design consult; the fail-open was surfaced by that same consult and confirmed first-party.
/simplify applied (REQUIRED_FILTERS↔disk cross-check, regex -f detection, reword-proof re-inline
guard, parametrized mutation tests, cached jq lookup). spec + version consistency + 36-entry
pytest manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): harden jq guards against malformed response types (codex #349 review)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security). Security: 0 P1/P2
(verified the fail-open fix first-party, no shell injection, no doc-extracted shell in CI,
no PII leak). Codex: 0 P1, 2 P2, 3 P3 — all addressed.
The #349 index fix closed bad *indices* but the filters still trusted container/value TYPES.
First-party confirmed each hole before fixing:
- [P2] Gemini grounded-guard used `length > 0`, which is truthy for non-empty strings/objects —
so groundingSupports/webSearchQueries arriving as a string/object passed the guard. Now
requires non-empty ARRAYS.
- [P2] Gemini source extractor: a string `groundingChunks` crashed `$chunks[.]`; a non-string
`uri` (number) was emitted as a source. Now array-normalizes every container and filters URIs
to non-empty strings.
- [P2] OpenAI source extractor: a `url_citation.url` of true/123 was emitted as a fabricated
source; an object url crashed `join`. Now filters url to non-empty strings.
→ all yield blank SOURCES → NOT_SEARCHED, never a fabricated or crashing result.
- [P3] Parse test now asserts `rc == 0` (catches undefined-function compile errors a
"syntax error" stderr-substring check would miss).
- [P3] Doc-sync lint now scans executable bash lines only (comment-stripped) and both quote
styles, so a commented-out `jq -f` (filename surviving in a comment) or a double-quoted inline
grounding jq can no longer bypass it. + 2 regression tests.
- [P3] Dropped the stale "12 pytest invocations" workflow comment (manifest has 36).
+6 malformed-type behavior tests (Gemini non-array containers / non-string uri; OpenAI
non-string/object url). guards 14→20, lint 6→8. spec + version consistency + 36-entry manifest
green; pyflakes clean. CHANGELOG #349 entry broadened to the full malformed-response hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize top-level candidates/output + strip trailing comments (R2 #349)
R2 convergence pass (codex gpt-5.5/xhigh) on the R1 fix round found 3 P3; first-party
verified each, fixed 2, rejected 1 as a non-reproducible false positive.
- [P3, confirmed] Top-level `candidates` (Gemini) / `output` (OpenAI) were not array-normalized,
so a malformed response with them as an OBJECT passed the guard (.candidates[]? / .output[]?
iterate an object's values) and then crashed (.candidates[0] on an object) or leaked a nested
url in the source extractor. This contradicted the "every container is array-normalized" claim.
Now `arr(.candidates)` / `arr(.output)` in all five filters: a non-array top-level container
yields no items → guard false → NOT_SEARCHED, sources blank, no crash/leak. First-party
confirmed the object-output leak (https://leak surfaced) and the fix.
- [P3, confirmed] The doc-sync lint's bash-line extraction dropped whole-line comments but kept
trailing comments, so `jq -r ".x" # jq -f "$GUARD/sources.jq"` satisfied the wiring check via
the filename in the trailing comment. _bash_code_lines now strips trailing bash comments
(quote-aware: a `#` only counts at word boundary outside quotes) before the check.
- [P3, rejected] codex claimed the double-quoted inline-jq detector is not escape-aware and a
PoC bypasses it; first-party the PoC was still caught (rc=1) by the current DOTALL regex. Not
adding escape-handling complexity for a non-reproducible case (no unrequested flexibility).
+4 top-level-container behavior tests (guards 20→24) + 1 trailing-comment lint test (8→9).
spec + version consistency + 36-entry manifest green; pyflakes clean. The CHANGELOG's "every
container is array-normalized" claim is now literally accurate (top-level included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize nested OpenAI content/annotations — full path closed (R3 #349)
R3 convergence pass (codex gpt-5.5/xhigh) confirmed R2's top-level fixes work, and found the last
residual: `arr(.output)` normalized the top level but `.content[]?` / `.annotations[]?` still
iterated an object's values — a valid array `output` with a completed web_search_call but an
object-shaped `content` or `annotations` still leaked a nested url into SOURCES (and text from an
object content). First-party confirmed the leak (https://leak1 / https://leak2 surfaced) and the fix.
This is the bottom of the OpenAI container nesting: the path is output[] → content[] →
annotations[] → url(scalar). All three containers are now array-normalized and the url leaf is
already string-filtered (even a url arriving as an array-of-strings is rejected). No deeper level
exists, so this closes the malformed-container class for the OpenAI filters — not an open-ended loop.
- openai_sources.jq / openai_text.jq: `arr(.content)[]` + `arr(.annotations)[]`.
- +3 nested-container behavior tests (guards 24→27).
- CHANGELOG: the "every container is array-normalized" clause is made explicit about the full
per-path container set (OpenAI output→content→annotations; Gemini candidates→chunks/supports/
indices) so the claim is precise, not overbroad (the self-consistency point codex raised).
Threat model note: these malformed shapes come from the provider's own authenticated API
response, not an attacker — but the fix reuses the same one-line `arr` pattern, makes the
CHANGELOG claim literally true, and converges the malformed-container hardening. spec + version
consistency + 36-entry manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:20:11 +08:00
elif ! jq -e -f "$GUARD/gemini_is_grounded.jq" <<<"$body" >/dev/null; then
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
echo "NOT_SEARCHED: no_grounding_support" # no search, or text not supported by it — discard
else
text="$(jq -r '.candidates[0].content.parts[]?.text // empty' <<<"$body")"
test(cross-model): behavior-test grounding guards + close malformed-response fail-opens (#349) (#350)
* test(cross-model): behavior-test grounding guards + fix Gemini source fail-open (#349)
The #346 grounding guards shipped as bash/jq inside cross_model_verification.md with no
automated test — a future jq edit or provider response-shape change could silently stop the
guard failing closed (the silent-false-VERIFIED class it exists to prevent). This extracts the
contract-bearing jq into canonical files, behavior-tests them, pins the doc wiring, and closes
a fail-open the extraction surfaced.
- Extract 5 canonical filters → scripts/cross_model_verification/*.jq (OpenAI search-guard /
text / sources; Gemini grounded-guard / sources). The doc loads them via `jq -f`, not inline.
- FAIL-OPEN FIX (Gemini sources): the extractor indexed groundingChunks by the model-supplied
groundingChunkIndices without validating them — a negative index silently picked a chunk from
the END (fabricating a real-but-wrong source URL that defeats the blank-source downgrade →
false VERIFIED resurrected), a string index crashed jq. Now admits only in-range non-negative
numeric indices, so a malformed support set yields blank sources → NOT_SEARCHED. First-party
verified the bug (negative index returned the last chunk) and the fix.
- test_cross_model_verification_guards.py: run each .jq against synthetic fixtures (grounded /
from-memory / negative-string-out-of-range index / non-2xx), + 2 mutation tests proving the
fixtures discriminate a working guard from accept-all / the pre-fix naive sources filter.
- check_cross_model_verification_sync.py: doc-sync lint — REQUIRED_FILTERS cross-checked against
the on-disk .jq set (a new filter can't escape the lint), every filter must be wired via
`jq -f`, NOT_SEARCHED / CROSS-MODEL-ERROR branches must remain, and no provider block may
re-inline a jq program referencing a grounding token. + its own mutation tests.
- CI: both wired into the pytest manifest + spec-consistency.yml; the workflow now ensures jq
is present on the runner (tests require jq, fail clear if absent — never skip a safety test).
Design approach (canonical .jq + behavior test + doc-sync lint) chosen after a cross-model
design consult; the fail-open was surfaced by that same consult and confirmed first-party.
/simplify applied (REQUIRED_FILTERS↔disk cross-check, regex -f detection, reword-proof re-inline
guard, parametrized mutation tests, cached jq lookup). spec + version consistency + 36-entry
pytest manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): harden jq guards against malformed response types (codex #349 review)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security). Security: 0 P1/P2
(verified the fail-open fix first-party, no shell injection, no doc-extracted shell in CI,
no PII leak). Codex: 0 P1, 2 P2, 3 P3 — all addressed.
The #349 index fix closed bad *indices* but the filters still trusted container/value TYPES.
First-party confirmed each hole before fixing:
- [P2] Gemini grounded-guard used `length > 0`, which is truthy for non-empty strings/objects —
so groundingSupports/webSearchQueries arriving as a string/object passed the guard. Now
requires non-empty ARRAYS.
- [P2] Gemini source extractor: a string `groundingChunks` crashed `$chunks[.]`; a non-string
`uri` (number) was emitted as a source. Now array-normalizes every container and filters URIs
to non-empty strings.
- [P2] OpenAI source extractor: a `url_citation.url` of true/123 was emitted as a fabricated
source; an object url crashed `join`. Now filters url to non-empty strings.
→ all yield blank SOURCES → NOT_SEARCHED, never a fabricated or crashing result.
- [P3] Parse test now asserts `rc == 0` (catches undefined-function compile errors a
"syntax error" stderr-substring check would miss).
- [P3] Doc-sync lint now scans executable bash lines only (comment-stripped) and both quote
styles, so a commented-out `jq -f` (filename surviving in a comment) or a double-quoted inline
grounding jq can no longer bypass it. + 2 regression tests.
- [P3] Dropped the stale "12 pytest invocations" workflow comment (manifest has 36).
+6 malformed-type behavior tests (Gemini non-array containers / non-string uri; OpenAI
non-string/object url). guards 14→20, lint 6→8. spec + version consistency + 36-entry manifest
green; pyflakes clean. CHANGELOG #349 entry broadened to the full malformed-response hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize top-level candidates/output + strip trailing comments (R2 #349)
R2 convergence pass (codex gpt-5.5/xhigh) on the R1 fix round found 3 P3; first-party
verified each, fixed 2, rejected 1 as a non-reproducible false positive.
- [P3, confirmed] Top-level `candidates` (Gemini) / `output` (OpenAI) were not array-normalized,
so a malformed response with them as an OBJECT passed the guard (.candidates[]? / .output[]?
iterate an object's values) and then crashed (.candidates[0] on an object) or leaked a nested
url in the source extractor. This contradicted the "every container is array-normalized" claim.
Now `arr(.candidates)` / `arr(.output)` in all five filters: a non-array top-level container
yields no items → guard false → NOT_SEARCHED, sources blank, no crash/leak. First-party
confirmed the object-output leak (https://leak surfaced) and the fix.
- [P3, confirmed] The doc-sync lint's bash-line extraction dropped whole-line comments but kept
trailing comments, so `jq -r ".x" # jq -f "$GUARD/sources.jq"` satisfied the wiring check via
the filename in the trailing comment. _bash_code_lines now strips trailing bash comments
(quote-aware: a `#` only counts at word boundary outside quotes) before the check.
- [P3, rejected] codex claimed the double-quoted inline-jq detector is not escape-aware and a
PoC bypasses it; first-party the PoC was still caught (rc=1) by the current DOTALL regex. Not
adding escape-handling complexity for a non-reproducible case (no unrequested flexibility).
+4 top-level-container behavior tests (guards 20→24) + 1 trailing-comment lint test (8→9).
spec + version consistency + 36-entry manifest green; pyflakes clean. The CHANGELOG's "every
container is array-normalized" claim is now literally accurate (top-level included).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): array-normalize nested OpenAI content/annotations — full path closed (R3 #349)
R3 convergence pass (codex gpt-5.5/xhigh) confirmed R2's top-level fixes work, and found the last
residual: `arr(.output)` normalized the top level but `.content[]?` / `.annotations[]?` still
iterated an object's values — a valid array `output` with a completed web_search_call but an
object-shaped `content` or `annotations` still leaked a nested url into SOURCES (and text from an
object content). First-party confirmed the leak (https://leak1 / https://leak2 surfaced) and the fix.
This is the bottom of the OpenAI container nesting: the path is output[] → content[] →
annotations[] → url(scalar). All three containers are now array-normalized and the url leaf is
already string-filtered (even a url arriving as an array-of-strings is rejected). No deeper level
exists, so this closes the malformed-container class for the OpenAI filters — not an open-ended loop.
- openai_sources.jq / openai_text.jq: `arr(.content)[]` + `arr(.annotations)[]`.
- +3 nested-container behavior tests (guards 24→27).
- CHANGELOG: the "every container is array-normalized" clause is made explicit about the full
per-path container set (OpenAI output→content→annotations; Gemini candidates→chunks/supports/
indices) so the claim is precise, not overbroad (the self-consistency point codex raised).
Threat model note: these malformed shapes come from the provider's own authenticated API
response, not an attacker — but the fix reuses the same one-line `arr` pattern, makes the
CHANGELOG claim literally true, and converges the malformed-container hardening. spec + version
consistency + 36-entry manifest green; pyflakes clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:20:11 +08:00
cites="$(jq -r -f "$GUARD/gemini_sources.jq" <<<"$body")"
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
printf '%s\nSOURCES: %s\n' "$text" "${cites:-(none)}"
fi
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
` ``
fix: Astra request compat, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823–#826) (#827)
* fix: Astra request compatibility, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823 #824 #825 #826)
#823 — OpenAI request builders (smoke entrypoint + documented example) drop
`temperature`, which GPT-6 Astra rejects; the per-model effort vocabulary
lives in scripts/cross_model_verification/openai_effort_guard.sh, sourced by
both, and an unsupported explicit Astra value fails before curl. Hermetic
fake-curl test runs both surfaces.
#824 — the contained Codex citation transport rejects effort=ultra with
REASONING_EFFORT_REQUIRES_DELEGATION before detection/auth/tempdir/launch on
both entry paths (codex-cli 0.153.4 defines ultra as the multiAgentMode
replacement). Model-independent by design.
#825 — hedging can no longer rescue an unsupported claim (writer recovery
tree, CER fallback row, temporal rule 5 in writer + both compiler mirrors,
writer contract D2); universal prose quotas in the writer, compilers,
writing_quality_check.md, academic-paper/SKILL.md, and contract D6 become
diagnostics subordinate to author/venue requirements. Audit inventory
corrected in place; held-out seed evals/heldout/unsupported_claim_recovery
(NOT_RUN) registered.
#826 — run_codex_audit.sh pins gpt-6-astra/xhigh and records both in a new
sidecar `model` block; claim_audit_pipeline binds an unknown judge identity
to a run-local cache key (no cross-run reuse) instead of defaulting to
gpt-5.5-xhigh.
Review: /simplify (4 angles), codex gpt-5.6-sol xhigh 2 rounds (r1: 1 P1 +
1 P2 + 2 P3 fixed; r2: 0 P1/P2), /security-review 0 findings; all 102
spec-consistency steps + pytest manifest replayed locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNKiXpdHx1T5F5RbXT2Ueu
* docs(claude): record the #824 ultra reversal in the v3.21.2 key-additions line
The v3.21.2 bullet still said the contained Codex citation transport accepts
ultra; #824 on this branch rejects it as a delegation request. Add the
reversal so the live instruction surface matches the transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7emV5r2aqZDJzAyYVuuDo
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 08:08:30 +09:00
> **Sampling parameters:** the OpenAI Responses request omits ` temperature`, ` top_p`, and ` top_logprobs`; GPT-6 Astra does not support them. Gemini and compatible-provider examples retain their provider-specific parameters. Grounding guards, rather than a sampling setting, enforce an actual lookup.
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
fix: Astra request compat, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823–#826) (#827)
* fix: Astra request compatibility, no-delegation citation transport, hedge/quota prompt repairs, audit provenance (#823 #824 #825 #826)
#823 — OpenAI request builders (smoke entrypoint + documented example) drop
`temperature`, which GPT-6 Astra rejects; the per-model effort vocabulary
lives in scripts/cross_model_verification/openai_effort_guard.sh, sourced by
both, and an unsupported explicit Astra value fails before curl. Hermetic
fake-curl test runs both surfaces.
#824 — the contained Codex citation transport rejects effort=ultra with
REASONING_EFFORT_REQUIRES_DELEGATION before detection/auth/tempdir/launch on
both entry paths (codex-cli 0.153.4 defines ultra as the multiAgentMode
replacement). Model-independent by design.
#825 — hedging can no longer rescue an unsupported claim (writer recovery
tree, CER fallback row, temporal rule 5 in writer + both compiler mirrors,
writer contract D2); universal prose quotas in the writer, compilers,
writing_quality_check.md, academic-paper/SKILL.md, and contract D6 become
diagnostics subordinate to author/venue requirements. Audit inventory
corrected in place; held-out seed evals/heldout/unsupported_claim_recovery
(NOT_RUN) registered.
#826 — run_codex_audit.sh pins gpt-6-astra/xhigh and records both in a new
sidecar `model` block; claim_audit_pipeline binds an unknown judge identity
to a run-local cache key (no cross-run reuse) instead of defaulting to
gpt-5.5-xhigh.
Review: /simplify (4 angles), codex gpt-5.6-sol xhigh 2 rounds (r1: 1 P1 +
1 P2 + 2 P3 fixed; r2: 0 P1/P2), /security-review 0 findings; all 102
spec-consistency steps + pytest manifest replayed locally.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNKiXpdHx1T5F5RbXT2Ueu
* docs(claude): record the #824 ultra reversal in the v3.21.2 key-additions line
The v3.21.2 bullet still said the contained Codex citation transport accepts
ultra; #824 on this branch rejects it as a delegation request. Add the
reversal so the live instruction surface matches the transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7emV5r2aqZDJzAyYVuuDo
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 08:08:30 +09:00
> **Reasoning effort (OpenAI only):** when ` ARS_CROSS_MODEL_REASONING_EFFORT` is set, the payload passes it as ` reasoning.effort`, making the effort a verification run uses visible and reproducible. When it is **unset, the field is omitted entirely and the provider's own default for the chosen model applies** — defaults differ across the lineup (GPT-5.6 documents ` medium`; other ids carry their own), so forcing one value here would silently change behavior for existing setups. Citation lookup is search-bound, not reasoning-bound, so higher efforts mostly buy latency and cost; set the variable deliberately (never silently run at ` xhigh`) if a run shows shallow search behavior. Ids without a row in the per-model table below are passed through unvalidated (the API rejects unknown values): GPT-5.5 accepts up to ` xhigh`, GPT-5.6 adds ` max`. GPT-6 Astra's API values are ` low|medium|high|xhigh|max` per the [official model guide](https://developers.openai.com/api/docs/guides/latest-model?model=gpt-6-astra) (` none`/` minimal`/` ultra` are not API values); the per-model table lives in ` scripts/cross_model_verification/openai_effort_guard.sh`, sourced by both the example above and the smoke entrypoint, so an unsupported explicit value fails before any request leaves. **Contained Codex citation transport (#824):** ` ultra` is rejected with ` REASONING_EFFORT_REQUIRES_DELEGATION` before detection, auth, temporary state, or launch — the codex-cli 0.153.4 schema defines it as the replacement for the deprecated ` multiAgentMode` (proactive delegation), outside this single-reference transport's contract; rationale on ` ACCEPTED_REASONING_EFFORTS` in ` scripts/cross_model_codex_transport.py`. A general Codex research session may still use it.
2026-07-12 02:06:59 +08:00
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
### OpenAI-Compatible API (MiMo, DeepSeek, self-hosted) — ungrounded
When ` CROSS_MODEL_AVAILABLE=openai_compatible`, use the **Chat Completions API** at
` ARS_OPENAI_COMPAT_BASE_URL`, authenticated with the dedicated ` ARS_OPENAI_COMPAT_API_KEY`.
These providers expose no hosted web-search tool, so there is **no grounding guard**. The
handler therefore normalizes the verdict by invoking the canonical
` normalize_compat_verdict.py` unit, which emits a single-line JSON object
(` {"status","provider","context"}`): a positive ` VERIFIED` is downgraded to ` NOT_SEARCHED` (an
ungrounded confirmation can never count as a grounded agreement), while a genuine rejection
(` NOT_FOUND` / ` MISMATCH`) passes through as a useful disagreement. The consumer reads ` .status`
only; the raw model text is JSON-escaped into ` .context` as human-readable context and is
**never** placed in a verdict slot the agreement counter parses — embedded newlines become
literal ` \n` inside the string, so a model response cannot inject a second status line. ` PROMPT`
holds the single-reference verification prompt from step 3.
` ``bash
# ARS_OPENAI_COMPAT_BASE_URL is the API root INCLUDING /v1 (e.g. https://api.deepseek.com/v1).
# Trailing slash is normalized so the endpoint is built exactly once — no double /v1.
endpoint="${ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions"
GUARD=scripts/cross_model_verification
resp="$(curl -sS -w '\n%{http_code}' "$endpoint" \
-H "Authorization: Bearer $ARS_OPENAI_COMPAT_API_KEY" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg model "$ARS_CROSS_MODEL" --arg prompt "$PROMPT" '{
model: $model,
messages: [
{role: "system", content: "You are a citation-verification assistant. If you did not actually perform an external lookup, respond NOT_SEARCHED. Use NOT_FOUND only if you are confident no such record exists; MISMATCH if a field is wrong; VERIFIED only with a source URL/DOI."},
{role: "user", content: $prompt}
],
temperature: 0.1
}')")"
http="${resp##*$'\n'}"; body="${resp%$'\n'*}"
if [ "$http" -lt 200 ] || [ "$http" -ge 300 ]; then
# Transport/API failure (401/429/5xx, or curl's 000) — distinct from NOT_SEARCHED, so the
# consumer falls back to single-model (see § Graceful Degradation), never an ungrounded verdict.
echo "CROSS-MODEL-ERROR: openai_compatible_http_$http"
else
text="$(jq -r '.choices[0].message.content // empty' <<<"$body")"
if [ -z "$text" ]; then
echo "CROSS-MODEL-ERROR: openai_compatible_empty_response"
else
# Canonical normalization lives in scripts/cross_model_verification/normalize_compat_verdict.py
# (behavior-tested in scripts/test_normalize_compat_verdict.py) and is INVOKED here rather than
# re-implemented in bash — the same canonical-and-referenced pattern the first-party blocks use
# with ` jq -f`. It emits ONE line of JSON: {"status","provider","context"}. The consumer reads
# .status only; raw model text is JSON-escaped in .context so it can never inject a second
# status line (the producer/consumer anti-laundering contract holds at the output-format level).
# VERIFIED -> status NOT_SEARCHED (ungrounded positive can never agree)
# NOT_FOUND/MISMATCH -> status passes through (useful disagreement)
# anything else/empty -> status NOT_SEARCHED (fail closed)
printf '%s' "$text" | python3 "$GUARD/normalize_compat_verdict.py"
fi
fi
` ``
> **No grounding guard for compatible providers.** The grounding guard (an API-level
> ` web_search_call` / ` groundingMetadata` trace) exists only for first-party OpenAI and
> Gemini. A compatible provider cannot evidence a lookup, so its positive verdicts are
> downgraded to ` NOT_SEARCHED` and never count as agreement. Its rejections survive as
> disagreements. The block emits a single-line JSON object (` {"status","provider","context"}`)
> from ` normalize_compat_verdict.py`, and the grounded-agreement count is computed solely from
> its ` .status` field — never from the raw text, which lives JSON-escaped in ` .context`.
> For the OpenAI-compatible block, read the verdict from the JSON ` .status` field only
> (e.g. ` jq -r .status`); never grep the emitted line or ` .context` for a verdict token — the
> raw model text is preserved JSON-escaped in ` .context` precisely so it cannot be mistaken for
> a verdict.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
### Detecting Available Models
Agents should check at the start of a verification/review session:
` ``bash
# Check which cross-model APIs are available
# Requires: jq (for JSON parsing). Fallback: python3 -c "import sys,json; ..."
if ! command -v jq &>/dev/null; then
echo "WARNING: jq not installed. Cross-model API calls will use python3 fallback."
fi
2026-08-11 12:56:01 +08:00
case "${ARS_CROSS_MODEL_TRANSPORT:-api}" in
codex)
# Citation-integrity availability only. This does not make any DA/reviewer/
# judgment transport available. The detector shares auth/model/version logic
# with the production verifier and honors a custom CODEX_HOME.
python3 scripts/cross_model_codex_transport.py detect ;;
api)
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
if [ -n "$ARS_CROSS_MODEL" ]; then
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
# PRECEDENCE: a first-party model id ALWAYS takes the grounded route, even if
# ARS_OPENAI_COMPAT_BASE_URL is set. This prevents a grounded->ungrounded downgrade. ANY gpt-*
# id (not just today's gpt-5.5/gpt-5.4) and any gemini-* id route grounded, so a future
# first-party release keeps the grounded path instead of silently falling through to the
# ungrounded compatible branch. The compatible path is reachable only for a model id that
# matches no first-party prefix, and only when its dedicated opt-in env vars are both present.
# OPENAI_BASE_URL is never read.
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
# ID STATUS is a separate axis from routing (#518): routing answers "which provider
# endpoint", the allowlist answers "is this id known-good". An unlisted gpt-*/gemini-* id
# still routes grounded (never falls through to the ungrounded compatible branch) but is
# announced as unlisted so nobody trusts results from a typo'd or made-up id the API has
# never accepted. Applies to first-party routes only — compatible-route ids are
# user-declared and carry no allowlist.
id_status() {
case " gpt-5.5 gpt-5.5-pro gpt-5.4 gpt-5.4-pro gemini-3.1-pro-preview " in
*" $1 "*) echo "validated"; return ;;
esac
feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#788)
* feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#787)
Probe set: 30 refs (10 easy DOI-keyed journal articles; 10 hard: 3 arXiv,
2 DOI-less NeurIPS, 5 non-English; 10 fabrications), every real row
resolver-confirmed same-day, every fabrication negative-checked. 180
same-day paired calls (30 x 3 repeats x 2 models), majority verdicts.
Result: all five measures PASS with superiority — recall 1.00 vs 0.80,
grounded completion 0.933 vs 0.900, p95 latency 26.5s vs 58.7s, zero
guard misfires, false disagreement 0.00 = 0.00. Transport-qualified:
gpt-5.6-sol stays provisional on the first-party API route (jq guards
unexercised; allowlist unchanged). Report + probe-set sha256 under
audits/; per-call index committed beside the probe set.
Campaign side-product (transport): page-open webSearch items
(action.type != "search") are skipped for binding instead of failing the
stream (opened-page URLs still can never become bound sources), and
DEVELOPER_INSTRUCTIONS requires an empty sources array for
NOT_FOUND/NOT_SEARCHED. 52 transport tests green. Defective-tool run 1
archived unscored; three probe-row transcription errors were flagged
MISMATCH by both models, independently re-verified, corrected, re-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: narrow the page-open exemption to the observed action.type == "other" shape (#788 codex P2)
An empty action object, unknown action type, or non-dict action on a
webSearch item is stream-fatal again; only the observed page-open shape
is skipped. Mutation test sweeps four bad shapes (52 -> 53 tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: anchor the page-open exemption to the first-party closed WebSearchAction set
Run-3 surfaced a third real shape ({"type": "openPage", "url": ...}) that
the single-observation exemption rejected, tool-suppressing the baseline's
measures (13 EVENT_STREAM_INVALID cells). The exempt set is now exactly
the non-search members of the app-server protocol's closed WebSearchAction
oneOf — {other, openPage, findInPage} plus the Responses-API spellings —
verified against `codex app-server generate-json-schema` on 0.147.0.
Unknown shapes stay stream-fatal (mutation sweep unchanged); 54 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: score preregistered run 4 as the gate result; runs 1-3 recorded as exploratory
Run 4 (frozen fixture @ 3fc6ddb, frozen parser @ c9c865d, both pushed
pre-run): all five measures PASS, zero misfires on BOTH models, recall
1.00 vs 0.80, grounded completion 0.933 vs 0.867, p95 27.5s vs 51.1s.
Report rewritten with the preregistration statement and the full
exploratory-round accounting; call index replaced with run-4 data;
claim surfaces and CHANGELOG updated to run-4 numbers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs(code): pin the bare-discriminator decision against the first-party schema (#788 codex round-2 P2 rejected with evidence)
The round-2 finding claimed openPage/findInPage require url/pattern; the
protocol schema (generate-json-schema, 0.147.0) marks every non-search
variant required:["type"] with url/pattern nullable optionals. Demanding
optional fields is the exact false-fatality class that invalidated
bakeoff runs 1 and 3. Decision recorded in the comment and pinned by
bare-discriminator test rows (54 tests, +2 param rows).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-3 findings — ordering, replayability, exposure analysis
P2 (ordering): webSearch action-shape validation now runs BEFORE the
MODEL_RETURNED_NOT_SEARCHED early return, so a model NOT_SEARCHED verdict
can never mask response-shape drift; mutation test added (55 tests).
P2 (replayability): the 180 full receipt rows, the offline scorer
(verified to reproduce the gate byte-for-byte from committed artifacts
alone), and the parameterized fleet runner are committed beside the
probe set; the report states the replayability boundary plainly (raw
event streams are digest-only by transport design).
P1 (answer-key exposure): empirical scan across all 540 retained
receipts finds zero repo-referencing bound queries/sources; report gains
an exposure-analysis section with scope caveats and corroboration; the
structural fix (sealed hash-commit preregistration, fresh fabrication
pool per run) is filed as #789 for future bakeoffs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-4 P2s — fleet gate, timeout margin, fresh-probe rule, nearest-rank p95
Scorer refuses truncated/duplicated/partial fleets (exactly one row per
(ref_id, repeat) across 30x3) before computing any measure; p95 moves to
the nearest-rank order statistic (51.13/27.46 -> 51.20/28.09, matching
the review's own recomputation; gate unchanged) and the method is named
on every surface. Runner outer timeout raised above the transport's
inner 300s deadline so its finally-block cleanup always fires first.
The canonical recorded-run note and report outcome now require a FRESH
probe set for the API-route run per #789 (this set's labels are public),
resolving the self-contradiction with the exposure analysis.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-5 P2s — scorer consumes runner output + binds receipt identity
score_run.py now scores either the committed run-4 JSONLs (default) or a
fleet-runner output directory (argv[1]), so a reproduced fleet can never
silently re-report the old result; every scored row must pass identity
binding (outer model/ref/repeat, receipt.model, receipt.request_id, and
a request_digest recomputed from the probe set via the transport's
canonical form), refusing mis-associated or edited fleets. Verified:
committed data reproduces the gate unchanged, results-dir mode scores
the live run-4 cells, and a cross-model receipt swap is refused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-6 P2s — dual-fleet hard-zero + unambiguous shape code
Measure 4 now requires zero guard misfires in BOTH fleets (a baseline
suppressed by tool misfires cannot anchor a fair comparison — the run-3
lesson, now enforced by the scorer). The transport emits
EVENT_STREAM_INVALID for a non-null non-list search `results` value
instead of silently skipping into NO_BOUND_SEARCH_RESULTS
(wrong_search_shape fixture expectation updated in lockstep), and the
scorer's shape family is trimmed to exactly the emitted shape codes with
the behavior-family classification documented. Provably no effect on the
scored run: run-4 contains zero rows in any affected code family (only
SOURCE_NOT_IN_SEARCH_RESULTS 12/6, a behavior code) and both fleets
already sit at zero misfires; all gate numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-7 P2s — results-shape check before verdict return + same-day fleet enforcement
The results-shape validation joins the pre-verdict scan loop so a model
NOT_SEARCHED verdict can never mask dict-shaped search results (mutation
test added: wrong_search_shape + NOT_SEARCHED -> EVENT_STREAM_INVALID;
56 tests). The runner refuses to resume over cells from an earlier date,
and the scorer refuses mixed-date fleets across both models (run-4 is
single-date; gate numbers unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-8 P2s — complete pre-verdict search validation + runner failure semantics
The pre-verdict scan now performs the COMPLETE search-item strict
validation (query type/length/control chars + results shape), covering
legacy action-less items — the round-7 placement validated only
action-typed items, which also made the round-7 mutation test fail (a
red test my verification pipeline masked via tail; committed here only
with PYTEST_EXIT=0 verified directly). Runner: a fleet with any failed
call now exits nonzero instead of printing ALL DONE, and an outer-
timeout kill sweeps the adapter's orphaned temp dirs (the detached
app-server exits on stdin EOF; the ephemeral auth copy is what the
verifier's skipped finally-block would have removed). 56 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-9 P2s — NOT_SEARCHED source contract, retry-not-skip, quiescent sweep, dated rows
Transport: NOT_SEARCHED with a populated sources array fails closed as
FINAL_OUTPUT_INVALID before the early return (mutation test; 57 tests).
Runner: a same-day cell that recorded a failure is discarded and retried
on resume instead of silently counting as complete; the orphan-tempdir
sweep runs only after the executor drains so it can never delete a live
worker's ephemeral CODEX_HOME. Scorer: every row must carry a real ISO
date — an undated fleet cannot satisfy the same-day gate on empty
strings. Committed run-4 data re-verified green under all new gates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-10 findings — single-path validation, scorer gate exit, probe-hash pin
P1 root treatment: the entire search-processing pipeline (cap, strict
per-item validation, reference-bound filter, URL binding incl. the
result-entry object-shape check) now runs BEFORE any verdict branch, so
every shape-fatal path fires identically regardless of the model's
answer — the verdict-masking bug class (rounds 3/7/8/9/10) is closed by
construction, not by another patch. Emptiness outcomes stay verdict-
conditional (an honest NOT_SEARCHED with no bound search remains model
behavior). Mutation test: bound search with a non-object result entry +
NOT_SEARCHED verdict -> EVENT_STREAM_INVALID (58 tests).
Scorer: refuses a probe set whose whole-file sha256 differs from the
frozen hash (labels now inside the scoring identity), and exits nonzero
when any gate fails. The P1's rerun demand is accepted: a run-5 fleet
under this frozen parser follows as the scored gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: run 5 under the frozen final instrument is the gate result (#788 codex round-10 P1 accepted)
Fleet rerun 2026-08-20 under parser+scorer db6ed67 (pushed pre-run):
all five measures PASS with zero misfires on both fleets — recall 0.90
vs 0.80, grounded completion 0.911 vs 0.889, p95 26.1s vs 47.6s. Run 4
reclassified as a prior-instrument exploratory round; committed
receipts/index/scorer default swapped to run-5 data (committed scorer
replays the gate from repo artifacts alone, exit 0); all claim surfaces
carry run-5 numbers and the cross-fleet consistency note (candidate led
measures 1/2/5 in every full paired fleet).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-11 P1s — no failed-trial erasure + full-fleet entry validation (instrument fixpoint)
Runner: a recorded failed trial is never deleted on resume; it is
carried into the failure count and forces a nonzero exit, so the only
path past a failure is rerunning the ENTIRE fleet fresh — selective
retry-until-green is structurally impossible. (Provably no scored fleet
was affected: runs 4 and 5 each completed in a single invocation with
zero failures and no retry/carried lines in their logs.)
Transport: the strict pre-verdict loop now validates every consumed
field of EVERY search item — id, query, results-list shape, and each
entry's object shape, bound or unbound — reaching the instrument
fixpoint: no field the pipeline reads is unvalidated, so no future
verdict-masking variant of this class exists. Mutation tests for
unbound-malformed-entries and id shapes (60 tests). A run-6 fleet under
this frozen instrument follows as the gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: run 6 under the fixpoint instrument is the gate result
Fleet rerun 2026-08-20 under adf18f9 (parser fixpoint + no-erasure
runner + full-gate scorer, all pushed pre-run): 5/5 PASS, zero misfires
both fleets — recall 1.00 vs 0.70, grounded completion 0.911 vs 0.867,
p95 28.8s vs 43.3s. Run 5 reclassified prior-instrument; artifacts and
scorer default swapped to run-6; leak scan clean across all 900 retained
receipts (runs 2-6); candidate led measures 1/2/5 in all four full
paired fleets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-12 P2s — pinned effort, fleet-private temp root, split disclosure
Runner pins ARS_CROSS_MODEL_REASONING_EFFORT to the provider default
(explicitly unset per call, recorded per row) and routes all transport
temp dirs into a fleet-private mkdtemp root so the timeout sweep can
never touch another invocation's dirs. The audit now discloses the one
run-6 1-1-1 split (baseline fab-05: MISMATCH/NOT_SEARCHED/NOT_FOUND ->
INDETERMINATE, conservative miss) and names the actual baseline misses
(fab-01, fab-08 majority NOT_SEARCHED; fab-05 split) — verified against
the committed receipts, correcting a stale carried-over sentence. The
effort variable was verified unset for every fleet (shell env + profile
carry no export); gate numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-13 P2s — receipt-contract validation, effort-marker gates, always-sweep
Scorer validates every non-null receipt against the closed contract
(required keys, verdict/searched types, positive-verdict grounding with
fully-bound sources, empty sources on NOT_FOUND/NOT_SEARCHED, queries
present when searched) before any metric trusts it, and results-dir
scoring requires the pinned-effort marker on every row (committed
gate-run rows predate the marker; the audit attests their configuration).
Runner refuses carried cells without the marker and sweeps the
fleet-private temp root in a finally-block on every outcome — a
signal/OOM-killed verifier no longer leaves its ephemeral auth copy.
Committed gate scoring still exits 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-14 P2s — per-verdict receipt invariants + runner rejects malformed parsed receipts
Scorer: grounded verdicts (VERIFIED/MISMATCH/NOT_FOUND) require
searched=true and a null reason_code; NOT_SEARCHED requires
searched=false and a reason from the transport's closed emitted set —
a fabricated NOT_FOUND-without-search or NOT_SEARCHED-with-search row
can no longer contribute to recall or completion. Runner: a verifier
exiting 0 with parsed-but-malformed output records RECEIPT_INVALID,
counts as a failure, and forces nonzero exit. Committed run-6 gate
scoring re-verified: exit 0, numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-15 P2s — uniform item-field validation + shared receipt contract (axis terminal)
Transport: every webSearch item — page-opens included — now has its
action payload validated against the closed WebSearchAction variant
types (url/pattern/query string-or-null, queries string-array), plus
uniform id and results/entry shape checks; a recognized discriminator
with a wrong-typed payload is stream-fatal (mutation sweep; 60 tests).
Tooling: the full closed receipt contract (exact key set, transport/
auth_mode/containment, digest formats, per-verdict cross-field
invariants) moves into a shared receipt_contract.py imported by BOTH
run_fleet.py and score_run.py — one implementation, applied to fresh
cells, resumed cells, and every scored row, so the two consumers cannot
diverge. This terminates the validation axis: every field of every
webSearch item and every key of every receipt is now checked; committed
run-6 gate scoring re-verified exit 0 with numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-16 — strict source bindings; instrument-freeze boundary pinned (P1 declined with recorded rationale)
receipt_contract.py enforces the full canonical source-binding shape
(closed 4-key object, non-trivial https URL, non-empty item id,
result_index 0-127 with bool exclusion) — committed run-6 data passes
unchanged. The round-16 rerun demand is DECLINED under a pinned
maintainer boundary, recorded in the report and the canonical
recorded-run note: runs 4/5 were discarded because consumed-data gaps
could alter scored outcomes; post-run-6 hardening validates only
surfaces outside every consumed path and cannot change any verdict,
binding, latency, or measure of a past fleet — such hardening applies
from the next fleet. The disagreement is recorded, not hidden.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-17 P2s — complete schema mirror + latency-sample validation
receipt_contract.py is now a COMPLETE stdlib mirror of the canonical
receipt schema: identifier/event-id/https-url patterns and length
bounds, array caps (queries<=32, sources<=16), closed entry objects,
auth_mode const, NOT_SEARCHED => empty queries+sources with a mandatory
reason, unknown reason codes refused globally. Scorer refuses boolean,
negative, non-numeric, or absurd wall_seconds before the percentile
gate. Committed run-6 gate scoring re-verified: exit 0, numbers
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-18 P2s — hashability, boolean identity, SIGTERM cleanup
Validator: array/object verdict or reason_code becomes a contract
failure instead of an uncaught TypeError (which would have escaped the
runner's SystemExit handling and re-opened the no-reroll gap);
containment flags are checked by identity (`is True`) so integer 1
cannot satisfy the schema's boolean constants. Runner: SIGTERM/SIGINT
raise SystemExit so the finally-block sweep of the fleet-private auth
copies also runs on cancellation. Mutation-verified (3/3 caught);
committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-19 P2s — discriminator hashability, integer repeats, real ISO timestamps
Transport: the WebSearchAction discriminator is type-checked before set
membership in both _is_page_open and the uniform loop — an array/object
type fails closed as EVENT_STREAM_INVALID instead of crashing the
verifier past shape accounting. Scorer: repeat must be an exact int in
1..3 (1.0 satisfied the completeness Counter while minting ...-r1.0),
and ts must parse as a full ISO timestamp with offset instead of a
digit-shaped prefix. Committed run-6 scoring exit 0 unchanged; 60
transport tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-20 — source-query binding, midnight guard; null-action P2 declined with schema evidence
receipt_contract.py rejects sources whose search_item_id has no retained
entry in search_queries (unretained evidence never counts as grounding).
run_fleet.py fails visibly BEFORE reporting success when a fresh fleet's
cells span two calendar dates. The explicit-null-action P2 is declined
on first-party evidence: ThreadItem types action as
anyOf[WebSearchAction, null] (generate-json-schema, 0.147.0), so null is
protocol-legal and follows the legacy path where the item still faces
the complete strict validation — fatal-izing it is the run-1/run-3
false-fatality class; decision pinned in the code comment. Run-6
scoring exit 0 unchanged; 60 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-21 — cancellation stops queued quota burn, worker failures persist; open-variant P2 declined with schema evidence
P1: SIGTERM/SIGINT set a STOP event making every queued job a no-op
(marked [CANCELLED], counted as failure), so shutdown waits only for
the at-most-3 in-flight calls instead of burning the rest of a paid
180-call fleet. P2: a worker exception after the paid call persists a
failed cell with the job identity, so a resume can never treat the
consumed trial as missing and re-roll it. The closed-variant P2 is
declined on first-party evidence: no WebSearchAction variant sets
additionalProperties, so extra fields are schema-legal and rejecting
them would make any future informational field fleet-fatal; decision
pinned in the code comment, known fields stay type-checked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-22 P1 — stray STOP=None placeholder no longer nullifies the cancellation event
The round-21 placeholder assignment landed AFTER the Event creation in
module order, resetting STOP to None and disabling queued-call
cancellation exactly as the review read it. The placeholder is removed;
the Event created before worker start is the one the signal handler
sets. Static check pins that no STOP=None assignment remains.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-23 P2 — refuse contradictory receipt+error rows
A row carrying both a valid receipt and a truthy error is structurally
impossible from the runner and is refused as corrupted/external instead
of being scored as grounded evidence; error rows with a null receipt
stay counted as misfires. Committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-24 P2s — output-contract precedence, portable probe digest, runner preflight
Transport: NOT_FOUND carrying sources is FINAL_OUTPUT_INVALID even when
the stream also lacks a bound search — output-contract violations now
outrank emptiness outcomes so the shape event cannot be misfiled as a
behavior code. Probe digest verification moves into the shared module
with CRLF->LF normalization (a Windows autocrlf checkout is not probe
drift) and the runner runs the same preflight BEFORE any paid call, so
180 subscription calls can never be spent on a fixture the scorer will
refuse. Run-6 scoring exit 0 unchanged; 60 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-25 — scorer-equivalent resume preflight + exclusive fleet lock
validate_row (shared receipt_contract) now carries the COMPLETE row
validation — contradiction, outer identity, exact-integer repeat, full
ISO timestamp, sane latency, receipt identity binding to the probe row,
closed receipt contract — and is the single implementation used by both
the scorer and the runner's resume preflight, so a misnamed or copied
cell fails before any further quota is spent. The runner takes an
exclusive flock on the output dir, refusing a second concurrent
invocation that would duplicate paid calls and race cell writes.
Committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-26 — URL-binding drift hits measure 4; cross-platform fleet lock
Transport: a bound search whose non-empty result entries yield no
extractable URL is EVENT_STREAM_INVALID (provider moved/renamed the URL
key = response-shape drift), no longer the behavioral
NO_BOUND_SEARCH_RESULTS; the pre-existing pin of the old classification
is updated in lockstep and a canonical_url regression test added (62
tests). Runner: the fleet lock falls back to msvcrt.locking on Windows,
keeping the documented reproduction path viable.
Note: run-6's receipts contain zero NO_BOUND_SEARCH_RESULTS /
SOURCE_NOT-with-empty-binding rows of the reclassified kind (reason
distribution: only SOURCE_NOT_IN_SEARCH_RESULTS with non-empty
bindings and clean rows), so the gate numbers are provably unaffected;
the change also falls under the pinned instrument-freeze boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-27 P2s — key-drift/value split, full resume validation, Windows invoke, UTF-8 I/O
Transport: EVENT_STREAM_INVALID for empty bindings now requires that NO
recognized URL key exists across the bound entries (true key drift); a
recognized key with an unusable value stays behavioral — both sides
test-pinned (62 tests). Runner: every resumed cell, failed ones
included, faces validate_row + the effort check before further quota is
spent; the transport is invoked via sys.executable (Windows honors no
shebang); all subprocess/artifact text I/O pinned to strict UTF-8 in
runner and scorer. Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-28 P2 — URL-key drift determined pre-verdict
The key-drift determination (bound entries present, no recognized URL
key anywhere) moves before the NOT_SEARCHED early return, so a model
NOT_SEARCHED answer can no longer mask renamed-URL-key response drift;
the post-verdict emptiness branch keeps only behavioral outcomes.
Masking regression test added (63 tests); run-6 scoring exit 0
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-29 P1 — fleet runner gated to POSIX
The #630 transport's process-group containment (start_new_session +
os.killpg in _stop_process) is POSIX-only, so a native-Windows fleet
would consume paid calls while every cell fails during cleanup — the
rounds-26/27 surface accommodations implied support the deeper stack
never had. The runner now refuses non-POSIX up front with a WSL
pointer; the dead msvcrt lock branch is removed (the scorer, which is
genuinely offline and portable, keeps its CRLF-tolerant digest and
UTF-8 reads). Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-30 — receipt-level invariance proof replaces live-validation claim; session isolation; stray-file preflight
P1 resolved by PROOF rather than a third rerun: run 6's 180 receipts
carry exactly two reason states (null; SOURCE_NOT_IN_SEARCH_RESULTS
12/8) with zero receipt-less, error, NO_BOUND, NO_REFERENCE,
FINAL_OUTPUT_INVALID, MODEL_RETURNED_NOT_SEARCHED, or
EVENT_STREAM_INVALID rows — each post-run-6 transport change either
touches unconsumed surfaces or only relabels cells in code families
that provably never occurred, so no run-6 cell can differ under the
shipped parser. The stale "run 6 live-validates the shipped parser" and
"instrument FIXPOINT / no masking path remains" sentences are replaced
with the precise provable statements; the freeze policy now REQUIRES
this proof standard (no proof on a consumed path = rerun, as runs 4/5
were). P2s: verifier subprocesses start in their own session so an
interactive Ctrl-C cannot turn in-flight calls into resume-poisoning
EXIT failures; the runner refuses unexpected result files before
spending quota.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-31 P2s — orphaned .tmp cells refused, fresh cells identity-bound
The preflight refuses orphaned atomic-write temp files (a crash between
write and rename must not silently re-roll a completed paid trial), and
fresh cells face the same validate_row identity binding as resumed
cells and the scorer before being persisted as success — a receipt with
the wrong model/request_id/digest becomes a recorded RECEIPT_INVALID
failure. Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* feat: counterbalanced interleaved scheduling for the bakeoff fleet (#788 round-32 P1 accepted)
The two models' calls for each (reference, repeat) cell are adjacent in
the queue with deterministic parity-alternating pair order, so model
identity is decorrelated from execution time — provider load or
web-search drift during the fleet can no longer masquerade as a model
effect. A counterbalanced run-7 follows as the scored gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: counterbalanced run 7 is the gate result; run 6 superseded for order confound
Run 7 under frozen instrument 69cd04a (interleaved parity-alternating
pair scheduling): all five measures PASS — recall 0.90 vs 0.80, p95
25.0s vs 49.6s (median 14.8 vs 17.5), grounded completion tied at
0.900, zero misfires both fleets, two 1-1-1 splits disclosed and scored
as conservative misses. Honesty note carried on every claim surface:
the sequential fleets' grounded-completion edge did NOT survive
counterbalancing and is not claimed; recall and latency led in all five
paired fleets. Artifacts and scorer default swapped to run-7 (committed
scorer replays the gate, exit 0); leak scan clean across 1,080 retained
receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-33 P1 — resume refuses half-complete counterbalanced pairs
Every (reference, repeat) pair must be wholly present or wholly missing
on resume: a one-sided pair would run the counterpart far from its
partner and silently reintroduce the model-vs-time confound. Run 7 is
unaffected (single uninterrupted invocation); scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: close codex round-34 P2 — recorded-run note limited to the measured superiority (2 and 5, tie on 1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 07:13:23 +08:00
# gpt-5.6-sol: validated for the codex subscription citation transport
# (2026-08-19 bakeoff); provisional HERE because this allowlist gates the
# first-party API route, which has no recorded bakeoff run.
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
# gpt-6-astra: listed 2026-09-05; provisional on every transport (entry-gate
# smoke only, no bakeoff run).
case " gpt-5.6-sol gpt-6-astra " in
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
*" $1 "*) echo "provisional"; return ;;
esac
echo "unlisted"
}
announce_id_status() {
status="$(id_status "$ARS_CROSS_MODEL")"
echo "CROSS_MODEL_ID_STATUS=$status"
case "$status" in
provisional) echo "NOTE: $ARS_CROSS_MODEL is provisional — endpoint support confirmed, ARS-specific behavior unvalidated (see Supported Models). Run scripts/cross_model_smoke_test.sh before relying on it." ;;
unlisted) echo "WARNING: $ARS_CROSS_MODEL matches a first-party prefix and routes grounded, but is NOT a known-good id — the API may reject it. Check the id, or run scripts/cross_model_smoke_test.sh before trusting results." ;;
esac
}
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
case "$ARS_CROSS_MODEL" in
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
gpt-*)
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
if [ -n "$OPENAI_API_KEY" ]; then
echo "CROSS_MODEL_AVAILABLE=openai"; announce_id_status
else
echo "WARNING: ARS_CROSS_MODEL=$ARS_CROSS_MODEL but OPENAI_API_KEY is not set"
fi ;;
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
gemini*)
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
if [ -n "$GOOGLE_AI_API_KEY" ]; then
echo "CROSS_MODEL_AVAILABLE=google"; announce_id_status
else
echo "WARNING: ARS_CROSS_MODEL=$ARS_CROSS_MODEL but GOOGLE_AI_API_KEY is not set"
fi ;;
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
*)
# Unrecognized id: only an explicit, credential-isolated opt-in enables the ungrounded
# OpenAI-compatible path. Both the base URL AND the dedicated key are required; the
# standard OPENAI_API_KEY is NEVER sent to a third-party endpoint (see Credential
# isolation in the API Call Patterns section).
if [ -n "$ARS_OPENAI_COMPAT_BASE_URL" ] && [ -n "$ARS_OPENAI_COMPAT_API_KEY" ]; then
echo "CROSS_MODEL_AVAILABLE=openai_compatible"
elif [ -n "$ARS_OPENAI_COMPAT_BASE_URL" ]; then
echo "WARNING: ARS_OPENAI_COMPAT_BASE_URL is set but ARS_OPENAI_COMPAT_API_KEY is not — refusing to send another provider's key. Set ARS_OPENAI_COMPAT_API_KEY."
echo "CROSS_MODEL_AVAILABLE=none"
else
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
echo "WARNING: ARS_CROSS_MODEL=$ARS_CROSS_MODEL is not a recognized model. First-party grounded route: any gpt-* id (e.g. gpt-6-astra, gpt-5.6-sol, gpt-5.5, gpt-5.5-pro, legacy gpt-5.4*) or gemini-* id (e.g. gemini-3.1-pro-preview). For an OpenAI-compatible provider set ARS_OPENAI_COMPAT_BASE_URL + ARS_OPENAI_COMPAT_API_KEY and use that provider's model id (must not match a gpt-*/gemini-* prefix, or it takes the grounded first-party route instead)."
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe)
Reworks external PR #453 to make the cross-model verifier provider-agnostic
(MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant:
explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL
downgrade), compatible verdicts always NOT_SEARCHED in citation verification,
DA critique equivalent, lint+mutation coverage for the new contracts.
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: revise PR#453 spec after dual-track (codex+gemini) review
Six corrections from the dual-LLM spec review:
- D1: first-party model ids always win the grounded route (resolve the
reviewer divergence toward safety; compat opt-in only for unrecognized ids)
- D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED;
NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer
contract so raw text never leaks into agreement counting
- D3: DA compatible is first-class for critique only, not factual adjudication
- D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync
lint narrowed to executable-bash assignment/expansion patterns
- D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send
OPENAI_API_KEY to a third-party endpoint
- Follow-up: first-party grounded verification through Azure/proxy filed as a
separate pre-existing gap, out of this PR's scope
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(xmv): collapse verdict normalization branch to a single expression
/simplify cleanup — behavior-preserving (26 tests unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-*
Addresses dual-model (codex+gemini) review of the implementation diff:
- P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw
text is JSON-escaped in .context and cannot inject a second STATUS line
- P1b: removes the non-portable grep \b word-boundary (logic now in tested Python)
- P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new
behavioral test exercises the CLI output contract incl. the injection case
- P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep
the grounded route instead of silently downgrading to ungrounded compatible
Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract
Addresses dual-model re-review residuals (runtime vulns already closed):
- check 8 false-pass: re-scope to the compatible block + require the pipe so a
comment mention or unpiped reference can't satisfy the wiring check
- U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output
- consumer contract: doc now states read .status only, never grep the line/.context
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual
codex final-pass residuals on the wiring check:
- false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches
(`(?:-\S+\s+)*` allows flags between python3 and the path)
- documented out-of-scope residual: the wiring check proves invoke-by-pipe but
does not parse bash control flow, so a contrived discard-output-then-rederive
block is a code-review concern, not a static-lint one (behavioral tests carry
the real JSON output contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 08:16:42 +08:00
echo "CROSS_MODEL_AVAILABLE=none"
fi ;;
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
esac
else
echo "CROSS_MODEL_AVAILABLE=none"
fi
2026-08-11 12:56:01 +08:00
;;
*)
echo "CROSS-MODEL-ERROR: invalid ARS_CROSS_MODEL_TRANSPORT selector"
echo "CROSS_MODEL_AVAILABLE=none"
;;
esac
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
` ``
2026-08-11 12:56:01 +08:00
If the API route is selected and ` ARS_CROSS_MODEL` is set but the corresponding
API key is missing or the model name is unsupported, the agent should warn the
user and proceed with single-model verification. If the citation-only ` codex`
route is selected, consume the detector's closed status instead; an invalid
transport selector is a visible configuration error and never falls through to
an API route.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
2026-08-19 18:00:30 +08:00
### Promotion Bakeoff (provisional → validated)
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
The run that flips a provisional id (today: ` gpt-6-astra` on both transports, and ` gpt-5.6-sol` on the first-party API route) to validated is defined here so a future promotion argues against numbers, not vibes (#518). Validation and recommendation are separate axes. (2026-08-19, #783: the recommendation moved to the current generation on lifecycle grounds — GPT-5.5 was superseded — ahead of validation; that flip carries no measurement claim. This bakeoff remains the only route to ` validated`, and any claim of measured parity or superiority still requires the run below.)
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
2026-08-24 12:08:01 +08:00
> **Recorded run (2026-08-19/20, #787 — codex-transport variant).** The procedure below was executed over the #630 ChatGPT-subscription citation transport (entry gate: ` scripts/cross_model_smoke_test_codex.sh` PASS for baseline and candidate; measure analogues: grounding evidence = receipt ` searched`, measure 4 = zero fail-closed receipt-guard misfires). All five measures passed in the counterbalanced gate fleet, with superiority on measures 2 (fabrication recall) and 5 (latency) and a tie on measure 1 — see ` audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md` (probe set ` evals/bakeoff/2026-08-19-gpt-5-6-sol-codex/`, sha256 in the report). The result is **transport-qualified**: ` gpt-5.6-sol` is validated for the subscription citation transport; it remains provisional on the first-party API route, whose jq grounding guards that run did not exercise. A scored fleet is bound to its preregistered frozen instrument; later instrument hardening that validates only surfaces outside every consumed path applies from the next fleet and does not retroactively invalidate a recorded gate result (boundary rationale in the run report's Instrument-freeze decision record). An API-route run requires a FRESH probe set under the #789 sealed-preregistration protocol below — the 2026-08-19 set's labels are public, so reusing it would expose a live-search run to answer-key retrieval.
feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#788)
* feat: first Promotion Bakeoff run — gpt-5.6-sol validated for the codex subscription transport (#787)
Probe set: 30 refs (10 easy DOI-keyed journal articles; 10 hard: 3 arXiv,
2 DOI-less NeurIPS, 5 non-English; 10 fabrications), every real row
resolver-confirmed same-day, every fabrication negative-checked. 180
same-day paired calls (30 x 3 repeats x 2 models), majority verdicts.
Result: all five measures PASS with superiority — recall 1.00 vs 0.80,
grounded completion 0.933 vs 0.900, p95 latency 26.5s vs 58.7s, zero
guard misfires, false disagreement 0.00 = 0.00. Transport-qualified:
gpt-5.6-sol stays provisional on the first-party API route (jq guards
unexercised; allowlist unchanged). Report + probe-set sha256 under
audits/; per-call index committed beside the probe set.
Campaign side-product (transport): page-open webSearch items
(action.type != "search") are skipped for binding instead of failing the
stream (opened-page URLs still can never become bound sources), and
DEVELOPER_INSTRUCTIONS requires an empty sources array for
NOT_FOUND/NOT_SEARCHED. 52 transport tests green. Defective-tool run 1
archived unscored; three probe-row transcription errors were flagged
MISMATCH by both models, independently re-verified, corrected, re-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: narrow the page-open exemption to the observed action.type == "other" shape (#788 codex P2)
An empty action object, unknown action type, or non-dict action on a
webSearch item is stream-fatal again; only the observed page-open shape
is skipped. Mutation test sweeps four bad shapes (52 -> 53 tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: anchor the page-open exemption to the first-party closed WebSearchAction set
Run-3 surfaced a third real shape ({"type": "openPage", "url": ...}) that
the single-observation exemption rejected, tool-suppressing the baseline's
measures (13 EVENT_STREAM_INVALID cells). The exempt set is now exactly
the non-search members of the app-server protocol's closed WebSearchAction
oneOf — {other, openPage, findInPage} plus the Responses-API spellings —
verified against `codex app-server generate-json-schema` on 0.147.0.
Unknown shapes stay stream-fatal (mutation sweep unchanged); 54 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: score preregistered run 4 as the gate result; runs 1-3 recorded as exploratory
Run 4 (frozen fixture @ 3fc6ddb, frozen parser @ c9c865d, both pushed
pre-run): all five measures PASS, zero misfires on BOTH models, recall
1.00 vs 0.80, grounded completion 0.933 vs 0.867, p95 27.5s vs 51.1s.
Report rewritten with the preregistration statement and the full
exploratory-round accounting; call index replaced with run-4 data;
claim surfaces and CHANGELOG updated to run-4 numbers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs(code): pin the bare-discriminator decision against the first-party schema (#788 codex round-2 P2 rejected with evidence)
The round-2 finding claimed openPage/findInPage require url/pattern; the
protocol schema (generate-json-schema, 0.147.0) marks every non-search
variant required:["type"] with url/pattern nullable optionals. Demanding
optional fields is the exact false-fatality class that invalidated
bakeoff runs 1 and 3. Decision recorded in the comment and pinned by
bare-discriminator test rows (54 tests, +2 param rows).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-3 findings — ordering, replayability, exposure analysis
P2 (ordering): webSearch action-shape validation now runs BEFORE the
MODEL_RETURNED_NOT_SEARCHED early return, so a model NOT_SEARCHED verdict
can never mask response-shape drift; mutation test added (55 tests).
P2 (replayability): the 180 full receipt rows, the offline scorer
(verified to reproduce the gate byte-for-byte from committed artifacts
alone), and the parameterized fleet runner are committed beside the
probe set; the report states the replayability boundary plainly (raw
event streams are digest-only by transport design).
P1 (answer-key exposure): empirical scan across all 540 retained
receipts finds zero repo-referencing bound queries/sources; report gains
an exposure-analysis section with scope caveats and corroboration; the
structural fix (sealed hash-commit preregistration, fresh fabrication
pool per run) is filed as #789 for future bakeoffs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-4 P2s — fleet gate, timeout margin, fresh-probe rule, nearest-rank p95
Scorer refuses truncated/duplicated/partial fleets (exactly one row per
(ref_id, repeat) across 30x3) before computing any measure; p95 moves to
the nearest-rank order statistic (51.13/27.46 -> 51.20/28.09, matching
the review's own recomputation; gate unchanged) and the method is named
on every surface. Runner outer timeout raised above the transport's
inner 300s deadline so its finally-block cleanup always fires first.
The canonical recorded-run note and report outcome now require a FRESH
probe set for the API-route run per #789 (this set's labels are public),
resolving the self-contradiction with the exposure analysis.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-5 P2s — scorer consumes runner output + binds receipt identity
score_run.py now scores either the committed run-4 JSONLs (default) or a
fleet-runner output directory (argv[1]), so a reproduced fleet can never
silently re-report the old result; every scored row must pass identity
binding (outer model/ref/repeat, receipt.model, receipt.request_id, and
a request_digest recomputed from the probe set via the transport's
canonical form), refusing mis-associated or edited fleets. Verified:
committed data reproduces the gate unchanged, results-dir mode scores
the live run-4 cells, and a cross-model receipt swap is refused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-6 P2s — dual-fleet hard-zero + unambiguous shape code
Measure 4 now requires zero guard misfires in BOTH fleets (a baseline
suppressed by tool misfires cannot anchor a fair comparison — the run-3
lesson, now enforced by the scorer). The transport emits
EVENT_STREAM_INVALID for a non-null non-list search `results` value
instead of silently skipping into NO_BOUND_SEARCH_RESULTS
(wrong_search_shape fixture expectation updated in lockstep), and the
scorer's shape family is trimmed to exactly the emitted shape codes with
the behavior-family classification documented. Provably no effect on the
scored run: run-4 contains zero rows in any affected code family (only
SOURCE_NOT_IN_SEARCH_RESULTS 12/6, a behavior code) and both fleets
already sit at zero misfires; all gate numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-7 P2s — results-shape check before verdict return + same-day fleet enforcement
The results-shape validation joins the pre-verdict scan loop so a model
NOT_SEARCHED verdict can never mask dict-shaped search results (mutation
test added: wrong_search_shape + NOT_SEARCHED -> EVENT_STREAM_INVALID;
56 tests). The runner refuses to resume over cells from an earlier date,
and the scorer refuses mixed-date fleets across both models (run-4 is
single-date; gate numbers unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-8 P2s — complete pre-verdict search validation + runner failure semantics
The pre-verdict scan now performs the COMPLETE search-item strict
validation (query type/length/control chars + results shape), covering
legacy action-less items — the round-7 placement validated only
action-typed items, which also made the round-7 mutation test fail (a
red test my verification pipeline masked via tail; committed here only
with PYTEST_EXIT=0 verified directly). Runner: a fleet with any failed
call now exits nonzero instead of printing ALL DONE, and an outer-
timeout kill sweeps the adapter's orphaned temp dirs (the detached
app-server exits on stdin EOF; the ephemeral auth copy is what the
verifier's skipped finally-block would have removed). 56 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-9 P2s — NOT_SEARCHED source contract, retry-not-skip, quiescent sweep, dated rows
Transport: NOT_SEARCHED with a populated sources array fails closed as
FINAL_OUTPUT_INVALID before the early return (mutation test; 57 tests).
Runner: a same-day cell that recorded a failure is discarded and retried
on resume instead of silently counting as complete; the orphan-tempdir
sweep runs only after the executor drains so it can never delete a live
worker's ephemeral CODEX_HOME. Scorer: every row must carry a real ISO
date — an undated fleet cannot satisfy the same-day gate on empty
strings. Committed run-4 data re-verified green under all new gates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-10 findings — single-path validation, scorer gate exit, probe-hash pin
P1 root treatment: the entire search-processing pipeline (cap, strict
per-item validation, reference-bound filter, URL binding incl. the
result-entry object-shape check) now runs BEFORE any verdict branch, so
every shape-fatal path fires identically regardless of the model's
answer — the verdict-masking bug class (rounds 3/7/8/9/10) is closed by
construction, not by another patch. Emptiness outcomes stay verdict-
conditional (an honest NOT_SEARCHED with no bound search remains model
behavior). Mutation test: bound search with a non-object result entry +
NOT_SEARCHED verdict -> EVENT_STREAM_INVALID (58 tests).
Scorer: refuses a probe set whose whole-file sha256 differs from the
frozen hash (labels now inside the scoring identity), and exits nonzero
when any gate fails. The P1's rerun demand is accepted: a run-5 fleet
under this frozen parser follows as the scored gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: run 5 under the frozen final instrument is the gate result (#788 codex round-10 P1 accepted)
Fleet rerun 2026-08-20 under parser+scorer db6ed67 (pushed pre-run):
all five measures PASS with zero misfires on both fleets — recall 0.90
vs 0.80, grounded completion 0.911 vs 0.889, p95 26.1s vs 47.6s. Run 4
reclassified as a prior-instrument exploratory round; committed
receipts/index/scorer default swapped to run-5 data (committed scorer
replays the gate from repo artifacts alone, exit 0); all claim surfaces
carry run-5 numbers and the cross-fleet consistency note (candidate led
measures 1/2/5 in every full paired fleet).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-11 P1s — no failed-trial erasure + full-fleet entry validation (instrument fixpoint)
Runner: a recorded failed trial is never deleted on resume; it is
carried into the failure count and forces a nonzero exit, so the only
path past a failure is rerunning the ENTIRE fleet fresh — selective
retry-until-green is structurally impossible. (Provably no scored fleet
was affected: runs 4 and 5 each completed in a single invocation with
zero failures and no retry/carried lines in their logs.)
Transport: the strict pre-verdict loop now validates every consumed
field of EVERY search item — id, query, results-list shape, and each
entry's object shape, bound or unbound — reaching the instrument
fixpoint: no field the pipeline reads is unvalidated, so no future
verdict-masking variant of this class exists. Mutation tests for
unbound-malformed-entries and id shapes (60 tests). A run-6 fleet under
this frozen instrument follows as the gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: run 6 under the fixpoint instrument is the gate result
Fleet rerun 2026-08-20 under adf18f9 (parser fixpoint + no-erasure
runner + full-gate scorer, all pushed pre-run): 5/5 PASS, zero misfires
both fleets — recall 1.00 vs 0.70, grounded completion 0.911 vs 0.867,
p95 28.8s vs 43.3s. Run 5 reclassified prior-instrument; artifacts and
scorer default swapped to run-6; leak scan clean across all 900 retained
receipts (runs 2-6); candidate led measures 1/2/5 in all four full
paired fleets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-12 P2s — pinned effort, fleet-private temp root, split disclosure
Runner pins ARS_CROSS_MODEL_REASONING_EFFORT to the provider default
(explicitly unset per call, recorded per row) and routes all transport
temp dirs into a fleet-private mkdtemp root so the timeout sweep can
never touch another invocation's dirs. The audit now discloses the one
run-6 1-1-1 split (baseline fab-05: MISMATCH/NOT_SEARCHED/NOT_FOUND ->
INDETERMINATE, conservative miss) and names the actual baseline misses
(fab-01, fab-08 majority NOT_SEARCHED; fab-05 split) — verified against
the committed receipts, correcting a stale carried-over sentence. The
effort variable was verified unset for every fleet (shell env + profile
carry no export); gate numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-13 P2s — receipt-contract validation, effort-marker gates, always-sweep
Scorer validates every non-null receipt against the closed contract
(required keys, verdict/searched types, positive-verdict grounding with
fully-bound sources, empty sources on NOT_FOUND/NOT_SEARCHED, queries
present when searched) before any metric trusts it, and results-dir
scoring requires the pinned-effort marker on every row (committed
gate-run rows predate the marker; the audit attests their configuration).
Runner refuses carried cells without the marker and sweeps the
fleet-private temp root in a finally-block on every outcome — a
signal/OOM-killed verifier no longer leaves its ephemeral auth copy.
Committed gate scoring still exits 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-14 P2s — per-verdict receipt invariants + runner rejects malformed parsed receipts
Scorer: grounded verdicts (VERIFIED/MISMATCH/NOT_FOUND) require
searched=true and a null reason_code; NOT_SEARCHED requires
searched=false and a reason from the transport's closed emitted set —
a fabricated NOT_FOUND-without-search or NOT_SEARCHED-with-search row
can no longer contribute to recall or completion. Runner: a verifier
exiting 0 with parsed-but-malformed output records RECEIPT_INVALID,
counts as a failure, and forces nonzero exit. Committed run-6 gate
scoring re-verified: exit 0, numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-15 P2s — uniform item-field validation + shared receipt contract (axis terminal)
Transport: every webSearch item — page-opens included — now has its
action payload validated against the closed WebSearchAction variant
types (url/pattern/query string-or-null, queries string-array), plus
uniform id and results/entry shape checks; a recognized discriminator
with a wrong-typed payload is stream-fatal (mutation sweep; 60 tests).
Tooling: the full closed receipt contract (exact key set, transport/
auth_mode/containment, digest formats, per-verdict cross-field
invariants) moves into a shared receipt_contract.py imported by BOTH
run_fleet.py and score_run.py — one implementation, applied to fresh
cells, resumed cells, and every scored row, so the two consumers cannot
diverge. This terminates the validation axis: every field of every
webSearch item and every key of every receipt is now checked; committed
run-6 gate scoring re-verified exit 0 with numbers unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-16 — strict source bindings; instrument-freeze boundary pinned (P1 declined with recorded rationale)
receipt_contract.py enforces the full canonical source-binding shape
(closed 4-key object, non-trivial https URL, non-empty item id,
result_index 0-127 with bool exclusion) — committed run-6 data passes
unchanged. The round-16 rerun demand is DECLINED under a pinned
maintainer boundary, recorded in the report and the canonical
recorded-run note: runs 4/5 were discarded because consumed-data gaps
could alter scored outcomes; post-run-6 hardening validates only
surfaces outside every consumed path and cannot change any verdict,
binding, latency, or measure of a past fleet — such hardening applies
from the next fleet. The disagreement is recorded, not hidden.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-17 P2s — complete schema mirror + latency-sample validation
receipt_contract.py is now a COMPLETE stdlib mirror of the canonical
receipt schema: identifier/event-id/https-url patterns and length
bounds, array caps (queries<=32, sources<=16), closed entry objects,
auth_mode const, NOT_SEARCHED => empty queries+sources with a mandatory
reason, unknown reason codes refused globally. Scorer refuses boolean,
negative, non-numeric, or absurd wall_seconds before the percentile
gate. Committed run-6 gate scoring re-verified: exit 0, numbers
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-18 P2s — hashability, boolean identity, SIGTERM cleanup
Validator: array/object verdict or reason_code becomes a contract
failure instead of an uncaught TypeError (which would have escaped the
runner's SystemExit handling and re-opened the no-reroll gap);
containment flags are checked by identity (`is True`) so integer 1
cannot satisfy the schema's boolean constants. Runner: SIGTERM/SIGINT
raise SystemExit so the finally-block sweep of the fleet-private auth
copies also runs on cancellation. Mutation-verified (3/3 caught);
committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-19 P2s — discriminator hashability, integer repeats, real ISO timestamps
Transport: the WebSearchAction discriminator is type-checked before set
membership in both _is_page_open and the uniform loop — an array/object
type fails closed as EVENT_STREAM_INVALID instead of crashing the
verifier past shape accounting. Scorer: repeat must be an exact int in
1..3 (1.0 satisfied the completeness Counter while minting ...-r1.0),
and ts must parse as a full ISO timestamp with offset instead of a
digit-shaped prefix. Committed run-6 scoring exit 0 unchanged; 60
transport tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-20 — source-query binding, midnight guard; null-action P2 declined with schema evidence
receipt_contract.py rejects sources whose search_item_id has no retained
entry in search_queries (unretained evidence never counts as grounding).
run_fleet.py fails visibly BEFORE reporting success when a fresh fleet's
cells span two calendar dates. The explicit-null-action P2 is declined
on first-party evidence: ThreadItem types action as
anyOf[WebSearchAction, null] (generate-json-schema, 0.147.0), so null is
protocol-legal and follows the legacy path where the item still faces
the complete strict validation — fatal-izing it is the run-1/run-3
false-fatality class; decision pinned in the code comment. Run-6
scoring exit 0 unchanged; 60 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-21 — cancellation stops queued quota burn, worker failures persist; open-variant P2 declined with schema evidence
P1: SIGTERM/SIGINT set a STOP event making every queued job a no-op
(marked [CANCELLED], counted as failure), so shutdown waits only for
the at-most-3 in-flight calls instead of burning the rest of a paid
180-call fleet. P2: a worker exception after the paid call persists a
failed cell with the job identity, so a resume can never treat the
consumed trial as missing and re-roll it. The closed-variant P2 is
declined on first-party evidence: no WebSearchAction variant sets
additionalProperties, so extra fields are schema-legal and rejecting
them would make any future informational field fleet-fatal; decision
pinned in the code comment, known fields stay type-checked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-22 P1 — stray STOP=None placeholder no longer nullifies the cancellation event
The round-21 placeholder assignment landed AFTER the Event creation in
module order, resetting STOP to None and disabling queued-call
cancellation exactly as the review read it. The placeholder is removed;
the Event created before worker start is the one the signal handler
sets. Static check pins that no STOP=None assignment remains.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-23 P2 — refuse contradictory receipt+error rows
A row carrying both a valid receipt and a truthy error is structurally
impossible from the runner and is refused as corrupted/external instead
of being scored as grounded evidence; error rows with a null receipt
stay counted as misfires. Committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-24 P2s — output-contract precedence, portable probe digest, runner preflight
Transport: NOT_FOUND carrying sources is FINAL_OUTPUT_INVALID even when
the stream also lacks a bound search — output-contract violations now
outrank emptiness outcomes so the shape event cannot be misfiled as a
behavior code. Probe digest verification moves into the shared module
with CRLF->LF normalization (a Windows autocrlf checkout is not probe
drift) and the runner runs the same preflight BEFORE any paid call, so
180 subscription calls can never be spent on a fixture the scorer will
refuse. Run-6 scoring exit 0 unchanged; 60 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-25 — scorer-equivalent resume preflight + exclusive fleet lock
validate_row (shared receipt_contract) now carries the COMPLETE row
validation — contradiction, outer identity, exact-integer repeat, full
ISO timestamp, sane latency, receipt identity binding to the probe row,
closed receipt contract — and is the single implementation used by both
the scorer and the runner's resume preflight, so a misnamed or copied
cell fails before any further quota is spent. The runner takes an
exclusive flock on the output dir, refusing a second concurrent
invocation that would duplicate paid calls and race cell writes.
Committed run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-26 — URL-binding drift hits measure 4; cross-platform fleet lock
Transport: a bound search whose non-empty result entries yield no
extractable URL is EVENT_STREAM_INVALID (provider moved/renamed the URL
key = response-shape drift), no longer the behavioral
NO_BOUND_SEARCH_RESULTS; the pre-existing pin of the old classification
is updated in lockstep and a canonical_url regression test added (62
tests). Runner: the fleet lock falls back to msvcrt.locking on Windows,
keeping the documented reproduction path viable.
Note: run-6's receipts contain zero NO_BOUND_SEARCH_RESULTS /
SOURCE_NOT-with-empty-binding rows of the reclassified kind (reason
distribution: only SOURCE_NOT_IN_SEARCH_RESULTS with non-empty
bindings and clean rows), so the gate numbers are provably unaffected;
the change also falls under the pinned instrument-freeze boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-27 P2s — key-drift/value split, full resume validation, Windows invoke, UTF-8 I/O
Transport: EVENT_STREAM_INVALID for empty bindings now requires that NO
recognized URL key exists across the bound entries (true key drift); a
recognized key with an unusable value stays behavioral — both sides
test-pinned (62 tests). Runner: every resumed cell, failed ones
included, faces validate_row + the effort check before further quota is
spent; the transport is invoked via sys.executable (Windows honors no
shebang); all subprocess/artifact text I/O pinned to strict UTF-8 in
runner and scorer. Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-28 P2 — URL-key drift determined pre-verdict
The key-drift determination (bound entries present, no recognized URL
key anywhere) moves before the NOT_SEARCHED early return, so a model
NOT_SEARCHED answer can no longer mask renamed-URL-key response drift;
the post-verdict emptiness branch keeps only behavioral outcomes.
Masking regression test added (63 tests); run-6 scoring exit 0
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-29 P1 — fleet runner gated to POSIX
The #630 transport's process-group containment (start_new_session +
os.killpg in _stop_process) is POSIX-only, so a native-Windows fleet
would consume paid calls while every cell fails during cleanup — the
rounds-26/27 surface accommodations implied support the deeper stack
never had. The runner now refuses non-POSIX up front with a WSL
pointer; the dead msvcrt lock branch is removed (the scorer, which is
genuinely offline and portable, keeps its CRLF-tolerant digest and
UTF-8 reads). Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix+docs: close codex round-30 — receipt-level invariance proof replaces live-validation claim; session isolation; stray-file preflight
P1 resolved by PROOF rather than a third rerun: run 6's 180 receipts
carry exactly two reason states (null; SOURCE_NOT_IN_SEARCH_RESULTS
12/8) with zero receipt-less, error, NO_BOUND, NO_REFERENCE,
FINAL_OUTPUT_INVALID, MODEL_RETURNED_NOT_SEARCHED, or
EVENT_STREAM_INVALID rows — each post-run-6 transport change either
touches unconsumed surfaces or only relabels cells in code families
that provably never occurred, so no run-6 cell can differ under the
shipped parser. The stale "run 6 live-validates the shipped parser" and
"instrument FIXPOINT / no masking path remains" sentences are replaced
with the precise provable statements; the freeze policy now REQUIRES
this proof standard (no proof on a consumed path = rerun, as runs 4/5
were). P2s: verifier subprocesses start in their own session so an
interactive Ctrl-C cannot turn in-flight calls into resume-poisoning
EXIT failures; the runner refuses unexpected result files before
spending quota.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-31 P2s — orphaned .tmp cells refused, fresh cells identity-bound
The preflight refuses orphaned atomic-write temp files (a crash between
write and rename must not silently re-roll a completed paid trial), and
fresh cells face the same validate_row identity binding as resumed
cells and the scorer before being persisted as success — a receipt with
the wrong model/request_id/digest becomes a recorded RECEIPT_INVALID
failure. Run-6 scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* feat: counterbalanced interleaved scheduling for the bakeoff fleet (#788 round-32 P1 accepted)
The two models' calls for each (reference, repeat) cell are adjacent in
the queue with deterministic parity-alternating pair order, so model
identity is decorrelated from execution time — provider load or
web-search drift during the fleet can no longer masquerade as a model
effect. A counterbalanced run-7 follows as the scored gate run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: counterbalanced run 7 is the gate result; run 6 superseded for order confound
Run 7 under frozen instrument 69cd04a (interleaved parity-alternating
pair scheduling): all five measures PASS — recall 0.90 vs 0.80, p95
25.0s vs 49.6s (median 14.8 vs 17.5), grounded completion tied at
0.900, zero misfires both fleets, two 1-1-1 splits disclosed and scored
as conservative misses. Honesty note carried on every claim surface:
the sequential fleets' grounded-completion edge did NOT survive
counterbalancing and is not claimed; recall and latency led in all five
paired fleets. Artifacts and scorer default swapped to run-7 (committed
scorer replays the gate, exit 0); leak scan clean across 1,080 retained
receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* fix: close codex round-33 P1 — resume refuses half-complete counterbalanced pairs
Every (reference, repeat) pair must be wholly present or wholly missing
on resume: a one-sided pair would run the counterpart far from its
partner and silently reintroduce the model-vs-time confound. Run 7 is
unaffected (single uninterrupted invocation); scoring exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
* docs: close codex round-34 P2 — recorded-run note limited to the measured superiority (2 and 5, tie on 1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mo3QXHj2yzwNQ3VQKVEM2j
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 07:13:23 +08:00
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
- **Entry gate:** ` scripts/cross_model_smoke_test.sh` passes against the candidate id.
2026-08-24 12:08:01 +08:00
- **Probe-set precondition — sealed preregistration (#789; both API and codex transports):** every future gate run uses ` scripts/check_promotion_bakeoff_preregistration.py` and the closed ` shared/contracts/cross_model/promotion_bakeoff_sealed_commitment.schema.json` + ` shared/contracts/cross_model/promotion_bakeoff_sealed_reveal.schema.json` contracts. A bakeoff against an ad-hoc/unsealed set, or one revealed before its fleet completed, is not a gate result.
1. **Prepare privately.** Build one ` ars-bakeoff-probe-set/1.0` fixture with 30 references — 20 real (10 easy DOI-keyed + 10 hard preprint/DOI-less/non-English) and 10 synthetic plausible fabrications. Real rows carry resolver-confirmed DOI/arXiv/URL ground truth; fabricated rows carry a fresh negative-check witness. Keep the labeled file outside Git (an untracked canonical path is permitted, but a private path is safer). ` python3 scripts/check_promotion_bakeoff_preregistration.py prepare --campaign-id <id> --probe-set <private-file>` validates the shape and historical non-reuse, then prints a closed commitment containing only the campaign id, LF-normalized file sha256, fixed row count, and aggregate composition — never a row, label, ground-truth identifier, fixture path, or free-text escape hatch.
2. **Commit and publish the seal before any call.** Save that output as ` evals/bakeoff/<id>/sealed_commitment.json` in a dedicated commit whose only changed path is that file; do not stage the fixture or reveal carrier. Push it, wait until the commit and its passing CI result are publicly reachable, record the immutable commit permalink, then run ` python3 scripts/check_promotion_bakeoff_preregistration.py preflight --commitment evals/bakeoff/<id>/sealed_commitment.json --probe-set <private-file>`. A local commit, timestamp, or later ancestry proof is not a substitute for this public-before-fleet witness.
3. **Run while sealed.** Run the counterbalanced baseline/candidate fleet with the fixture local. No scored call may precede the successful preflight; neither ` probe_set.json` nor ` sealed_reveal.json` may enter Git while any fleet call remains pending.
4. **Reveal after the fleet.** Once all calls have reached terminal retained rows, place the unchanged fixture at ` evals/bakeoff/<id>/probe_set.json`; run the checker's ` make-reveal` command to produce ` evals/bakeoff/<id>/sealed_reveal.json`; add those two files together in one later commit. ` verify-reveal` (one campaign) or ` verify-tree` (CI, all campaigns) fails closed on digest/composition drift, duplicate JSON keys, symlink/path substitution, non-isolated or rewritten commitments, non-ancestor/same-commit seals, probe/reveal introduction drift, or post-reveal mutation. Squash/cherry-pick copies are accepted only when they descend the same seal and carry the identical bound probe/reveal lifecycle; the receipt exposes every qualifying introduction in ` reveal_copy_git_commits` rather than hiding source-ref copies.
5. **Never reuse a published answer key.** Once labels appear in any Git version, those exact probe bytes are retired permanently and every later gate gets a fresh fabrication pool. The verifier scans every historical version of every ` evals/bakeoff/**/probe_set.json`; a fabricated reference remains reused even if its id, context, case, Unicode width, spacing, or punctuation changes. Previously used real references may remain, but no previously labeled reference may enter the new fabricated pool. The 2026-08-19 fixture is the sole explicitly grandfathered unsealed artifact: its canonical path and LF-normalized SHA-256 are pinned, its blob bytes and regular-file mode must remain immutable across reachable history, and it remains part of the published-history scan. Verification requires a complete non-shallow local history and fails closed when a referenced historical object cannot be read.
The reveal verifier proves byte binding, composition, Git immutability/order, and detectable historical non-reuse. It cannot prove when a commit became visible on a remote or when an external call ran. The run report therefore MUST record the public commitment permalink and CI result, successful preflight output, fleet start/end bounds, final verifier receipt, and—for every listed reveal-copy commit in ` reveal_copy_git_commits`—a public permalink plus evidence that it first became publicly reachable only after the fleet ended. A copy published before or during the fleet invalidates the gate even when a later squash commit is clean. Missing remote/timing evidence makes the fleet exploratory, not a gate result.
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
- **Procedure:** run the transport's validated baseline (` gpt-5.5` on the first-party API route; ` gpt-5.6-sol` on the ChatGPT-subscription citation transport) and the candidate the same day, one call per reference, 3 repeats. Per-reference verdict = the verdict returned by ≥ 2 of 3 repeats; if no verdict reaches 2 (a 1– 1– 1 split), the reference is **indeterminate** and scored conservatively against the model that produced it — a miss for recall (measure 2), a false disagreement for measure 3. Grounded-search completion (measure 1) is computed per call, so ties don't apply.
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
- **Non-inferiority thresholds — all five must pass:**
1. **Grounded-search completion rate** (share of calls returning grounding evidence) ≥ baseline − 5 pp.
2. **Citation-mismatch recall** on the 10 fabrications (share flagged ` NOT_FOUND`/` MISMATCH`) ≥ baseline − 5 pp AND ≥ 80% absolute.
3. **False-disagreement rate** on the 20 real references (share incorrectly flagged ` NOT_FOUND`/` MISMATCH`) ≤ baseline + 5 pp.
4. **jq-guard shape stability:** zero guard misfires attributable to response-shape change across all calls (hard requirement — a shape change that trips the fail-closed guards disqualifies regardless of the other measures).
5. **p95 latency** ≤ 2× baseline.
2026-08-19 18:00:30 +08:00
- **Outcome — validation, plus a separate claim rule:**
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
- **All five pass → ` provisional` becomes ` validated`** (the id-status allowlist and the Supported Models note update; a promotion PR records the run under ` audits/` with the probe-set hash). Non-inferiority earns trust, nothing more.
2026-08-19 18:00:30 +08:00
- **A measured-superiority claim additionally requires observed superiority on at least one of the five measures, with no inferiority elsewhere, named explicitly in the promotion PR.** An operational benefit (cost, latency, capability) is a recommendation-policy argument, never a measurement claim. A candidate that merely scraped under every tolerance (− 5 pp grounding, − 5 pp recall, +5 pp false disagreements, 2× latency) is validated, nothing more. (Recommendation itself follows generation currency — see the 2026-08-19 note above — but stays caveated as provisional/unmeasured until this run passes.)
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
- Any fail → the id stays provisional; the results are still recorded.
Web-search results vary day to day; the 3-repeat majority verdict and same-day paired runs are what make the comparison fair. Thresholds are the #518 spec's choice and are tunable in a future spec without redesigning the procedure.
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
A further caveat that no threshold absorbs: a candidate with high verbalized evaluation awareness (GPT-6 Astra's vendor reports it in 9.6% of simulated Codex trajectories, and 41– 51% of samples at ` xhigh`/` max` effort in one external evaluation — system card §8.6, §8.8.1) may behave differently on a 30-reference probe fleet than in a live pipeline. The sealed preregistration prevents answer-key retrieval, not evaluation awareness; a promotion PR for such a model records this as a known limitation of the measurement.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
## Cost Considerations
Cross-model verification adds API costs from the second provider:
| Scenario | Additional Calls | Estimated Additional Cost |
|----------|-----------------|--------------------------|
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
| Integrity verification (risk-stratified: HIGH-IMPACT — and at Stage 4.5 NEW-CHANGED — 100% uncapped + sampled remainder, min 3 / max 10; **one call per reference**) | worked example: 60 refs, 12 high-impact → 12 + 5 = 17 calls. No fixed upper bound — a results-dense paper approaches all references | ~$1.35-2.95 (the example; scales linearly with calls) |
2026-06-10 03:57:09 +08:00
| DA cross-check (1 per checkpoint, 3 checkpoints) | 3 calls | ~$0.30-0.55 |
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
| Blind disagreement checkpoints (design freeze + final editorial decision, 1 structured-decision call each; editorial repeats on re-review) | 2-3 calls | ~$0.20-0.55 |
| **Full pipeline (the worked example)** | **~22-23 calls** | **~$1.85-4.05 — no fixed ceiling; grows with the high-impact / new-changed count** |
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
These are rough estimates based on GPT-5.5 pricing ($5/1M input, $30/1M output) and typical prompt sizes; GPT-5.5 Pro runs ~6× higher ($30/1M input, $180/1M output). GPT-5.6 Sol bills at the same standard rates as GPT-5.5 ($5/1M input, $0.50/1M cached input, $30/1M output); its pro mode keeps those rates but performs more model work per request, so total tokens (and latency) rise instead of the unit price. GPT-6 Astra's list pricing is not recorded in this document; re-derive the table from the provider's price list before budgeting a run on it. One-call-per-reference (rather than batching) is a deliberate cost-for-provenance trade: it is the only way the grounding-evidence check maps 1:1 to each verdict. Web-search-tool calls also cost more than plain completions.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
## Limitations
1. **Does not solve frame-lock fully.** All major LLMs share substantial training data. Cross-model catches different surface errors but may share deep structural biases.
feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff (#519)
* feat(cross-model): #518 gate hardening — risk-stratified sampling, blind disagreement checkpoints, id-status allowlist, promotion bakeoff
Four upgrades to shared/cross_model_verification.md + consumers, per the
2026-07-11 cross-model consult frozen in #518:
1. Integrity-gate cross-model sample: uniform random 30% (min 5 max 15) →
risk stratification (100% of high-impact refs uncapped + 10% random
remainder min 3 max 10; Stage 4.5 adds 100% of new/changed-claim refs
+ control sample). integrity_verification_agent updated in lockstep;
results table gains a Tier column.
2. Blind disagreement checkpoints at the two irreversible decisions
(research_architect design freeze; editorial_synthesizer final
decision): independent structured cross-model decision, anchoring-
blind, divergence = targeted rebuttal + user escalation — review
trigger, never a vote, never averaged. Sprint-contract boundary kept
closed (drivers never enter the scoring matrix).
3. "6th reviewer — Planned" retired, not deferred (counterproductive-
conditions list matches documented anti-patterns); live mirrors
(.claude/CLAUDE.md, raise_framework, SETUP en/zh-TW) drop the claim;
historical release notes untouched.
4. Detection snippet separates endpoint routing from id trust:
CROSS_MODEL_ID_STATUS=validated|provisional|unlisted with explicit
unlisted-id warning; routing byte-identical (grounded precedence
preserved). Plus §Promotion Bakeoff operationalizing the gpt-5.6-sol
provisional→validated non-inferiority run (5 thresholds, smoke-test
entry gate).
Spec: docs/design/2026-07-12-518-cross-model-gate-hardening-spec.md
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 codex fixes — NEW-CHANGED tier, primary pre-commitment, panel_size N, bakeoff reproducibility + validated-vs-default split
Addresses all 6 findings from the round-1 codex review (gpt-5.6-sol, xhigh):
- Stage 4.5 tier gap: 4-tier set (HIGH-IMPACT > NEW-CHANGED > CONTROL/RANDOM),
mutually exclusive by precedence, one verification per reference; CONTROL
replaces RANDOM at the final gate.
- Cap contradiction: stale "max 15 survives" sentence removed; RANDOM/CONTROL
capped at 10 each, 10% round-up rounding pinned.
- Design-freeze primary decision: architect commits the same enum + drivers
BEFORE the blind call, with criteria per enum value; blueprint Output Format
gains a Design-Freeze Checkpoint Audit section.
- Sprint-contract integration: editorial checkpoint is explicitly post-Step-3
(never extends the three-step arithmetic); input is the panel's panel_size N
usable cards (5 full-mode / 2 methodology_focus), not a hardcoded five.
- Bakeoff reproducibility: versioned, labeled, sha256-recorded probe-set
fixture is a precondition; >=2/3 majority verdict with conservative
1-1-1-split handling defined.
- Non-inferiority != default flip: full pass earns provisional->validated;
the recommended default flips only with a stated superiority or
operational-benefit reason.
Spec + CHANGELOG updated to as-built; agents/ mirror re-copied.
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-2 verify fixes — blindness leak, promotion wording, audit transport-error, cost bounds, SETUP tier coverage
Addresses all 6 findings from the round-2 codex verify (1 P1 + 5 P2):
- P1 blindness leak: the architect now commits its structured decision
SEPARATELY from the blueprint and sends the cross-model a sanitized
payload with the audit section stripped; the audit section is populated
only after the comparison.
- Two remaining canonical statements implying non-inferiority => default
flip aligned to the validated-vs-default split.
- Design-Freeze Checkpoint Audit gains unavailable / transport-error
representations.
- Cost table: bounded ranges replaced with a worked example (17 calls)
+ explicit no-fixed-upper-bound statement.
- SETUP en/zh-TW feature rows now mention the Stage 4.5 NEW-CHANGED tier.
- Spec Files-touched inventory adds the agents/ mirror; review-round
annotations normalized (round 1 = first review, round 2 = verify).
All CI lints + 60-entry pytest manifest green locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
* fix(cross-model): #518 round-3 P2 residuals — audit N/A fields on transport failure, Limitations tier phrasing
- Design-Freeze Checkpoint Audit: drivers/confidence gain explicit
none/N/A values when the cross-model is unavailable; step 5 failure
representation aligned to the schema.
- Limitations §2 rewritten per-gate: RANDOM at Stage 2.5, NEW-CHANGED +
CONTROL at Stage 4.5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FczvB1WJjfumdJTg9YyL3X
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:07:04 +08:00
2. **API latency.** Cross-model calls add 2-5 seconds per call, plus web-search round-trip time. With one call per reference (no batching) and a web-search tool, a risk-stratified integrity selection (uncapped HIGH-IMPACT plus the capped RANDOM sample at Stage 2.5; uncapped HIGH-IMPACT + NEW-CHANGED plus the capped CONTROL sample at Stage 4.5) can add several minutes on a results-dense paper; the calls can be issued concurrently to bound wall-clock time.
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
3. **Response format differences.** Different models structure responses differently. The agent must parse varied formats — keep verification prompts simple and structured to minimize parsing issues.
4. **Cost scales with paper size.** Longer papers with more references = more cross-model calls.
## Graceful Degradation
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
If cross-model verification fails **at the transport level** (API error, rate limit, key expired):
feat: v3.0 — Anti-sycophancy, intent detection, cross-model verification, AI self-reflection
Three batches of optimizations addressing structural AI limitations discovered through
a 4-round dialectic experiment:
Batch 1 — Anti-sycophancy + Intent Detection:
- Devil's Advocate Concession Threshold Protocol (deep-research + reviewer): 1-5 rebuttal
scoring, no concession below 4/5, no consecutive concessions, frame-lock detection
- Socratic Mentor Intent Detection Layer: exploratory vs goal-oriented mode classification,
anti-premature-closure rules, re-assessment every 5 turns
- Socratic Mentor Dialogue Health Indicator: silent self-check every 5 turns for persistent
agreement, conflict avoidance, premature convergence
Batch 2 — Cross-Model Verification (optional):
- New shared/cross_model_verification.md: protocol for GPT-5.4 Pro / Gemini 3.1 Pro as
independent second reviewer. Setup guide, API patterns, cost estimates
- Integrity verification: 30% sample cross-checked by second model (batched 5/call)
- DA agents: independent cross-model critique after each checkpoint
- Activated by ARS_CROSS_MODEL env var; zero overhead when not set
Batch 3 — AI Self-Reflection:
- Pipeline Stage 6 AI Self-Reflection Report: tracks DA concession rate, health alerts,
sycophancy risk rating (LOW/MEDIUM/HIGH), frame-lock incidents, convergence patterns
- Irony caveat: report states it is produced by the same AI it is assessing
All changes reviewed via 3x parallel /simplify passes (reuse + quality + efficiency).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:14:18 +08:00
- Log the failure: ` [CROSS-MODEL-ERROR: reason]`
- Continue with single-model verification — never block the pipeline on cross-model failure
- Include a note in the report: "Cross-model verification was configured but unavailable for this run. Results are single-model only."
fix(cross-model): wire grounding tool + fail loud on ungrounded verify (#346) (#348)
* fix(cross-model): wire web-search grounding + fail loud on ungrounded verify (#346)
The cross-model citation-verification prompt told the verifier to 'search
the web to confirm', but the shipped API examples enabled no search tool —
so a copied example produced a verifier that answered from parametric
memory and could return a confident false VERIFIED. Silent false
provenance is the exact failure this protocol exists to catch.
- OpenAI example migrated from chat/completions to the Responses API
(/v1/responses), the endpoint that hosts the web_search tool. Chat
Completions does not take tools:[{type:web_search}] (that needs the
separate gpt-5-search-api model), so Responses keeps it model-agnostic
across gpt-5.4 / gpt-5.4-pro.
- Gemini example adds the hosted google_search grounding tool.
- Both examples now GATE the model text on proof a search happened: an
OpenAI web_search_call item / a Gemini groundingMetadata block. Absent
grounding -> the call emits NOT_SEARCHED and the text is discarded. A
model that ignores 'search the web' cannot fake an absent grounding
trace, so the jq guard, not the prompt wording, is the safety boundary.
- Verify contract: one call per reference (grounding evidence maps 1:1 to
the verdict; batching proves 'something' was searched, not each ref),
new NOT_SEARCHED verdict, VERIFIED must carry a source URL/DOI or it is
downgraded to NOT_SEARCHED.
- Step 5 / results table / graceful-degradation updated: NOT_SEARCHED is
treated as not-verified, counted separately, never silent agreement,
and distinguished from a transport-level failure (which still falls
back to single-model). Cost/latency estimates updated for per-reference
calls.
Closes #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): tighten grounding guard to require citation linkage (#346)
Codex review flagged the first guard as too loose: a completed web_search_call / nonempty groundingChunks proves a search ran, not that THIS reference's verdict is supported. OpenAI now emits url_citation annotations (SOURCES line) so step 5 downgrades a citation-less VERIFIED to NOT_SEARCHED; Gemini requires BOTH webSearchQueries AND groundingChunks. NOT_FOUND preserved. Guard verified with pos/neg fixtures. Refs #346
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): align integrity-verification consumer to grounded one-call-per-ref protocol (#346)
The two prior #346 commits updated shared/cross_model_verification.md (wire
grounding tool + fail loud as NOT_SEARCHED + one call per reference) but left
the consuming agent, academic-pipeline/agents/integrity_verification_agent.md,
asserting the old behavior — a silent spec↔consumer drift that re-opens the
exact failure class #346 closes at the agent layer.
- Behavior summary: drop stale "batches of 5" (Quality Standards table + summary
bullets), state one grounded API call per reference, add the NOT_SEARCHED
verdict and its "never counts as agreement with a Claude VERIFIED" rule.
- Graceful degradation: split transport failure (fall back to single-model) from
NOT_SEARCHED (call succeeded, no grounded evidence — surface, never launder).
- CHANGELOG: add [Unreleased] entry covering the spec change + this alignment.
Agent-prompt + CHANGELOG text only; no schema or script change.
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cross-model): close dual-track review gaps in grounding guard (#346)
Dual-track pre-merge review (codex gpt-5.5/xhigh + opus security agent) on the
branch. Security: 0 P1/P2 (noted the jq -n --arg migration closes the old
$ARS_CROSS_MODEL splice seam). Codex: 0 P1, 4 P2, 2 P3 — all addressed:
- [P2] Gemini guard only checked webSearchQueries + groundingChunks, so a search
that returned chunks but a from-memory verdict (text not tied to any chunk)
passed as grounded, and SOURCES pulled every chunk uri so the blank-source
downgrade never fired. Now require groundingSupports (the chunkIndices that
link answer spans to sources) and derive SOURCES only from supported chunks —
an unsupported VERIFIED leaves SOURCES blank → downgraded to NOT_SEARCHED.
Both jq paths mutation-tested (grounded vs from-memory) with synthetic responses.
- [P2] Both bash patterns emitted NOT_SEARCHED on ANY non-2xx, contradicting the
spec's own transport-failure distinction (lines 296-301) and agent line 474.
Non-2xx (401/429/5xx/000) now emits CROSS-MODEL-ERROR (→ single-model fallback);
NOT_SEARCHED is reserved for 2xx-with-no-grounding.
- [P2] Cost math: 30% of 60 refs is capped at max 15, not 18. Cost table +
latency text + CHANGELOG corrected to ~15 integrity / ~18 full-pipeline calls.
- [P2] Neutralized "Codex cross-check" provenance in the public CHANGELOG to
"a second-model cross-check pass" (kept the public #301 framing).
- [P3] "no agent-prompt logic change" was inaccurate → "Documentation +
agent-prompt/protocol text only; no executable script or schema change".
- Aligned agent line 467 to the 2xx-only NOT_SEARCHED wording so it can't drift
from the new transport-error split.
OpenAI jq path independently re-verified (codex confirmed; mutation-tested here).
check_spec_consistency + check_version_consistency + 34-entry pytest manifest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 12:10:31 +08:00
A ` NOT_SEARCHED` result is **not** a transport failure and is handled differently. It means the call succeeded but the model could not (or did not) ground the lookup, so its verdict carries no evidence. Do not fall back to single-model and do not treat it as agreement: record the reference as ` NOT_SEARCHED` in the results table, count it separately from agreements/disagreements, and surface it for re-run or human review. The distinction matters — a transport failure means "we have no cross-model opinion"; a ` NOT_SEARCHED` means "the cross-model gave an opinion we have decided not to trust as a confirmation."
chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra (#819) (#820)
* chore(models): align docs and guardrails to Claude Fable 5.1 and GPT-6 Astra
Read both vendor system cards in full and applied the model-update pass:
- Claude Fable 5.1 named as the current frontier model (PERFORMANCE en/zh-TW
with a dated list-price re-derivation; cross-model primary-row example).
- gpt-6-astra listed as a provisional cross-model verifier on both transports
and recommended under the #783 lifecycle policy; gpt-5.6-sol keeps its
validated status on the ChatGPT-subscription citation transport. Entry-gate
smoke PASS on that transport (2026-09-05, codex-cli 0.153.4). SETUP en/zh-TW
example sets, id-status allowlist, bakeoff baseline text, and .claude/CLAUDE.md
move together.
- Codex citation transport: `ultra` joins the closed reasoning-effort set as a
named constant, with a test pinning turn/start forwarding and fail-closed
rejection of unknown values.
- New guardrail: checkpoint decision provenance (authority in the pipeline
state machine, operational mirror in the orchestrator), indexed as risk R11;
both content-lock hashes updated in this commit.
- Provider-side monitoring / safety interventions named as a never-a-verdict
case in the cross-model doc and the degradation registry row.
- Model tiering records that the resolved tier is the declared model; risk
register R1/R4/R5/R6 residual gaps updated.
- Harness-retirement audit for the model change:
audits/harness-retirement-2026-09-model-update.md (0 prompt retirements,
4 applied currency fixes, 2 deferred, 8 keep-as-debt annotations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* docs(changelog): align the model-update entries with the final text
The [Unreleased] entries were written before the simplify pass moved the
checkpoint-decision authority into the pipeline state machine, reused the
existing transport-failure markers for provider-side interventions, and
de-numbered the model-tiering note. Wording now matches the files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
* test: scope the checkpoint-authority section out of the v3.6.7 orchestrator line budget
The v3.6.7 Phase 6.6 budget test measures the orchestrator prompt minus every
later independent extension, each with its own bounded cap. The new
`## Checkpoint authority fidelity` section (13 lines) pushed the v3.6.7-attributed
count to 652 against a 639 ceiling. Following the existing convention, the
section gets its own measurement helper, an 18-line cap (5 lines of headroom),
a dedicated test, and is subtracted from the historical budget.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sWwwG3oCbtL4cGhRsr5US
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 00:46:14 +09:00
### Provider-side monitoring and safety interventions (2026-09)
GPT-6 Astra ships with provider-side monitoring that acts on live calls (GPT-6 Astra system card, 2026-09-03): a misalignment monitor over Codex, ChatGPT, and the Responses API that can pause or end a conversation, with stopped API conversations not resumable (§10.2.3.1); misuse monitors and activation classifiers that can block a generation mid-stream (§10.2.3.2); a stricter cyber refusal boundary for higher-risk accounts (§10.2.2.2); and account-level escalation to automated and, in some cases, manual review (§10.2.5). None of this changes the contract above; it adds concrete cases to it:
- **An intervention is never a verdict.** A paused, ended, blocked, or refused call surfaces on the API route either as an HTTP error — the existing transport-failure marker (` CROSS-MODEL-ERROR: openai_http_<status>`); the run continues single-model with the standard disclosure — or as a completed response with no grounding evidence, which the existing ` NOT_SEARCHED` guard already catches (counted separately, surfaced for re-run or human review). On the contained codex adapter it is the adapter's nonzero exit or fail-closed receipt. None of these is ever a citation judgment, a reviewer finding, or a checkpoint decision. Because every ARS call is one stateless request per reference or judgment, an ended conversation loses no ARS state: re-run the affected item.
- **Content can trigger the intervention.** A manuscript on a security topic, or a review payload dense with attack vocabulary, can trip the cyber boundary even when the task is ordinary scholarship. Do not rephrase the *manuscript* to route around a provider's safety boundary; that is the provider's call to make. For ARS's own prompt wording, prefer process vocabulary (consistency, drift, witness, discrepancy) over attack vocabulary where the meaning is the same, and treat a content-triggered refusal as ` unavailable` for that item with the reason recorded.
- **Escalation is part of the consent boundary.** Because flagged conversations can reach human review at the provider, the consent obtained under the Consent boundary above must assume that provider staff may read what is sent. This is the residual gap recorded as R4 in ` docs/RISK_REGISTER.md`.
- **Why ARS never consumes the verifier's reasoning.** The same vendor reports reduced chain-of-thought monitorability for this model and euphemistic framing of workarounds in its reasoning (§9). ARS binds every verifier result to grounding evidence, a bound source receipt, a typed anchor, or a closed enum — never to the model's narrative. That is a design rationale for the guards in this document, not a runtime claim about the verifier.