mirror of
https://github.com/Imbad0202/academic-research-skills.git
synced 2026-09-14 13:51:17 +08:00
feat: provider-agnostic cross-model verification (OpenAI-compatible: MiMo, DeepSeek, self-hosted) (#455)
* docs: spec for provider-agnostic cross-model verifier (PR #453 reframe) Reworks external PR #453 to make the cross-model verifier provider-agnostic (MiMo/DeepSeek/self-hosted) while holding the grounding-evidence invariant: explicit opt-in via ARS_OPENAI_COMPAT_BASE_URL (no passive OPENAI_BASE_URL downgrade), compatible verdicts always NOT_SEARCHED in citation verification, DA critique equivalent, lint+mutation coverage for the new contracts. Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: revise PR#453 spec after dual-track (codex+gemini) review Six corrections from the dual-LLM spec review: - D1: first-party model ids always win the grounded route (resolve the reviewer divergence toward safety; compat opt-in only for unrecognized ids) - D2: selective normalization — only VERIFIED downgrades to NOT_SEARCHED; NOT_FOUND/MISMATCH pass through as valid disagreements + producer/consumer contract so raw text never leaks into agreement counting - D3: DA compatible is first-class for critique only, not factual adjudication - D5: behavioral fixtures first (VERIFIED->NOT_SEARCHED->agreement 0), doc-sync lint narrowed to executable-bash assignment/expansion patterns - D6 (new): credential isolation via ARS_OPENAI_COMPAT_API_KEY — never send OPENAI_API_KEY to a third-party endpoint - Follow-up: first-party grounded verification through Azure/proxy filed as a separate pre-existing gap, out of this PR's scope Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(xmv): compatible verdict normalization — VERIFIED downgrades, rejections survive Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(xmv): pin case-insensitivity, None-safety, and VERIFIED-first fail-closed precedence Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(xmv): detection precedence — first-party grounded wins; compat opt-in + key isolation Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(xmv): compatible call path — isolated key, normalized endpoint, selective verdict downgrade Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(xmv): align table/setup/integrity/DA prose with grounding-aware compatible path Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(xmv): doc-sync lint backstops compat downgrade, key isolation, no double /v1, precedence parity Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(xmv): harden lint against unterminated-fence drop + compat-block identifier loss Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(xmv): collapse verdict normalization branch to a single expression /simplify cleanup — behavior-preserving (26 tests unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: record altitude follow-up from /simplify (canonical-unit vs inline-bash) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(xmv): call canonical normalizer from bash (kills output injection + \b portability + precedence drift); widen first-party to gpt-* Addresses dual-model (codex+gemini) review of the implementation diff: - P1: bash now invokes normalize_compat_verdict.py emitting single-line JSON; raw text is JSON-escaped in .context and cannot inject a second STATUS line - P1b: removes the non-portable grep \b word-boundary (logic now in tested Python) - P2: lint check 8 becomes a wiring check (must invoke the canonical unit); new behavioral test exercises the CLI output contract incl. the injection case - P3: detection widened gpt-5.5*/gpt-5.4* -> gpt-* so future first-party ids keep the grounded route instead of silently downgrading to ungrounded compatible Co-Authored-By: kzccIneko <kzccIneko@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * harden(xmv): scope+pipe-require lint check 8, ascii-escape JSON context, document .status-only consumer contract Addresses dual-model re-review residuals (runtime vulns already closed): - check 8 false-pass: re-scope to the compatible block + require the pipe so a comment mention or unpiped reference can't satisfy the wiring check - U+2028/U+2029: ensure_ascii=True so a unicode line-separator can't re-split output - consumer contract: doc now states read .status only, never grep the line/.context Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * harden(xmv): allow interpreter flags in check-8 normalizer regex; document residual codex final-pass residuals on the wiring check: - false-fail fixed: `python3 -u .../normalize_compat_verdict.py` now matches (`(?:-\S+\s+)*` allows flags between python3 and the path) - documented out-of-scope residual: the wiring check proves invoke-by-pipe but does not parse bash control flow, so a contrived discard-output-then-rederive block is a code-review concern, not a static-lint one (behavioral tests carry the real JSON output contract) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: kzccIneko <kzccIneko@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
committed by
GitHub
GitHub
parent
d54d43a001
commit
88fc003e6a
@@ -0,0 +1,128 @@
|
||||
# Provider-agnostic cross-model verifier (PR #453 reframe)
|
||||
|
||||
**Issue/PR:** reworks external PR #453 (`kzccIneko:feat/openai-compatible-cross-model`)
|
||||
**Date:** 2026-06-16
|
||||
**File touched:** `shared/cross_model_verification.md` + `scripts/check_cross_model_verification_sync.py` + `scripts/test_check_cross_model_verification_sync.py`
|
||||
**Status:** design approved, pre-implementation
|
||||
|
||||
## Problem
|
||||
|
||||
The cross-model verification layer hardcodes the verifier endpoint to `api.openai.com`, and the model-detection `case` statement only recognises `gpt-5.*` / `gemini-*`. Researchers without an OpenAI account but with access to an OpenAI-Chat-Completions-compatible provider (Xiaomi MiMo, DeepSeek, or a self-hosted endpoint) cannot use the cross-model layer at all. The intent is legitimate: make the verifier **provider-agnostic** for any endpoint speaking the OpenAI Chat Completions protocol.
|
||||
|
||||
The contributor's PR delivers this intent but, read as **actual diff** (not PR prose), introduces three real defects confirmed by an independent codex review (REWORK verdict):
|
||||
|
||||
- **P1 — laundering:** the compatible path emits the model's raw `VERIFIED` text with no grounding trace and no downgrade, so an ungrounded from-memory verdict can be counted as an agreement in the integrity results table. This violates the protocol's load-bearing invariant (grounding evidence, not prompt wording, is the safety boundary).
|
||||
- **P1 — passive downgrade:** the detection logic routes any model id to `openai_compatible` whenever the **standard SDK env var** `OPENAI_BASE_URL` is set. Existing GPT users who set `OPENAI_BASE_URL` for an Azure/proxy/local route get silently downgraded from grounded first-party OpenAI to ungrounded Chat Completions. Behaviour changes for users who never opted in.
|
||||
- **P2 cluster:** documented endpoints build to `…/v1/v1/chat/completions` (double `/v1`); the setup guide exports `OPENAI_API_KEY` twice and leaves a misleading `gpt-5.5`; the compatible prompt collapses `NOT_SEARCHED` into `NOT_FOUND`; the Integrity section and the new compatible section contradict each other on the guard.
|
||||
|
||||
## Security invariant (must hold after this change)
|
||||
|
||||
A cross-model `VERIFIED` verdict counts as agreement **only** when backed by API-level grounding evidence (OpenAI `web_search_call` / Gemini `groundingMetadata`). Compatible providers have no hosted web-search tool, so they can never produce that evidence. Therefore: **compatible-provider verdicts are never counted as grounded agreement in citation verification.** The line is drawn on the "needs grounding?" axis, not the "is it compatible?" axis — so a compatible provider is a first-class verifier for tasks that don't need grounding (Devil's Advocate critique) and a non-confirming voice for tasks that do (citation existence).
|
||||
|
||||
**Producer/consumer split (dual-track review, 2026-06-16).** Holding the invariant requires pinning BOTH ends, not just the producer. The producer (handler) emits a normalized machine-readable status; the consumer (agreement counting) must read ONLY that normalized status, never the raw model text. The compatible provider's raw text must never land in a verdict column or any parseable verdict slot — otherwise a raw `VERIFIED` substring leaks back into the agreement count even though the status field says `NOT_SEARCHED`. The behavioral fixture that proves this (compatible returns `VERIFIED` → final agreement count is 0) is the load-bearing test, more than any doc-string lint.
|
||||
|
||||
## Design
|
||||
|
||||
### D1 — Explicit opt-in via `ARS_OPENAI_COMPAT_BASE_URL`
|
||||
|
||||
A dedicated, ARS-namespaced env var both signals opt-in and supplies the endpoint. The standard `OPENAI_BASE_URL` is **never** read by the detection or call logic — removing the PR's passive `OPENAI_BASE_URL`-triggered downgrade is the fix for the P1 passive-downgrade defect.
|
||||
|
||||
**Precedence: first-party recognized model ids always win the grounded route.** Detection `case` after change, in order:
|
||||
|
||||
- `gpt-5.5*|gpt-5.4*` → `openai` (grounded; key check unchanged) — **even if `ARS_OPENAI_COMPAT_BASE_URL` is set.**
|
||||
- `gemini*` → `google` (grounded; key check unchanged) — likewise.
|
||||
- `*)` catch-all (an otherwise-unrecognized model id) → if `ARS_OPENAI_COMPAT_BASE_URL` is set **and** `ARS_OPENAI_COMPAT_API_KEY` is set (see D6) → `openai_compatible`; else warn + `none`. **No `OPENAI_BASE_URL` branch.**
|
||||
- The `mimo*|deepseek*` prefixes from the PR are dropped as load-bearing routing — they're documented as *examples* of compatible model ids, but routing is governed solely by `ARS_OPENAI_COMPAT_BASE_URL` (so "any self-hosted OpenAI-compatible endpoint", the broadest case the contributor wanted, works without a prefix allowlist to maintain).
|
||||
|
||||
**The explicit precedence trade-off (dual-track review divergence, resolved toward safety).** The two reviewers split here: one flagged "first-party-first" as a bug that locks out a user running `gpt-5.5` through a LiteLLM/proxy; the other flagged "compat-first" as the real hazard, because a recognized first-party model id silently routed to the ungrounded compatible path is a grounded→ungrounded **downgrade** — the exact regression class this rework exists to kill. We choose first-party-first: a recognized `gpt-*`/`gemini-*` id never loses its grounding guard to an opt-in env var. The cost is acknowledged and documented: a user who wants to verify *through* a compatible proxy must name the model with an id that does not match a first-party prefix (the proxy's own model id), not `gpt-5.5`. First-party grounded verification through `OPENAI_BASE_URL`-style proxies is a separate, currently-absent feature (see Follow-up), not this PR's job.
|
||||
|
||||
### D2 — Compatible verdicts: only `VERIFIED` is downgraded; rejections pass through
|
||||
|
||||
When `CROSS_MODEL_AVAILABLE=openai_compatible`, the citation-verification call:
|
||||
|
||||
- builds the endpoint as `endpoint="${ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions"` (base URL is the API root including `/v1`; trailing slash normalised — fixes double-`/v1`).
|
||||
- on transport failure (non-2xx / curl 000) → `CROSS-MODEL-ERROR: openai_compatible_http_<code>` (unchanged contract: transport error ≠ NOT_SEARCHED).
|
||||
|
||||
**Selective normalization (not a blanket downgrade).** The first spec draft downgraded *every* compatible verdict to `NOT_SEARCHED`. That was too blunt — the invariant only forbids an ungrounded *positive* confirmation; it says nothing about rejections. A compatible model that correctly flags a hallucinated paper as `NOT_FOUND`/`MISMATCH` produces a **valid disagreement** worth keeping; throwing it away forces needless human review of a hallucination the model already caught. So the handler normalizes by verdict:
|
||||
|
||||
- model says `VERIFIED` → **downgrade to `NOT_SEARCHED`** (no grounding trace to evidence the positive claim; an ungrounded confirmation can never count as agreement — the existing line-127 rule "VERIFIED with no source → NOT_SEARCHED" applies, intensified here because no source channel exists at all).
|
||||
- model says `NOT_FOUND` / `MISMATCH` → **pass through unchanged** as a disagreement (a rejection needs no grounding evidence to be useful; it is surfaced for human review either way).
|
||||
- model says `NOT_SEARCHED` or returns unparseable text → `NOT_SEARCHED`.
|
||||
|
||||
**Producer/consumer contract (the load-bearing part).** The handler emits exactly ONE machine-readable normalized status. The compatible model's raw text is NOT placed in the verdict column or any slot the agreement counter parses; if shown at all it is in a clearly-labelled context/notes field the counter ignores. Agreement counting consumes only the normalized status. This closes the leak where a raw `VERIFIED` substring (e.g. `NOT_SEARCHED\nraw: VERIFIED`) is re-grepped into the agreement total. The behavioral fixture in D5 proves: compatible returns `VERIFIED` → normalized status `NOT_SEARCHED` → grounded-agreement count increments by 0.
|
||||
|
||||
This holds the invariant with the minimal change: it reuses the existing `NOT_SEARCHED` channel (no new status enum), keeps genuinely useful rejections, and a compatible provider still appears in the results table without a positive verdict being laundered into a confirmation.
|
||||
|
||||
### D3 — Task scope: citation downgraded, DA equivalent *for critique only*, peer-review untouched
|
||||
|
||||
- **Citation existence verification** (Stage 2.5/4.5): compatible → handled per D2.
|
||||
- **Devil's Advocate critique** (deep-research + academic-paper-reviewer): compatible is a **first-class verifier for idea generation and critique** — surfacing weaknesses, alternative framings, and attack angles needs no web grounding, so DeepSeek/MiMo findings are used like GPT/Gemini findings *at that layer*. **But "first-class" is scoped to critique, not factual adjudication.** An ungrounded model can invent missing literature, assert a methodological flaw resting on a false fact, or fabricate a policy constraint. So the contract narrows: a DA finding from **any** provider (compatible or first-party) is an adversarial hypothesis, never standalone evidence, unless it carries an independently-checkable source. The spec wording in the DA section must state this explicitly so a compatible-provider DA "finding" is not mistaken for a verified defect. The DA section already routes through the shared "API Call Patterns"; no DA agent file hardcodes an endpoint (verified by grep), so no agent file changes.
|
||||
- **Peer-review sixth reviewer:** remains `Planned, not yet implemented`. Out of scope for this change.
|
||||
|
||||
### D4 — Setup guide: mutually exclusive provider blocks
|
||||
|
||||
Replace the single copy-paste block (which exports `OPENAI_API_KEY` twice and leaves a misleading `gpt-5.5`) with three mutually exclusive examples, each a complete self-consistent tuple:
|
||||
|
||||
- **OpenAI (first-party, grounded):** `OPENAI_API_KEY` + `ARS_CROSS_MODEL=gpt-5.5`.
|
||||
- **Gemini (first-party, grounded):** `GOOGLE_AI_API_KEY` + `ARS_CROSS_MODEL=gemini-3.1-pro-preview`.
|
||||
- **OpenAI-compatible (ungrounded):** `ARS_OPENAI_COMPAT_API_KEY=<provider key>` + `ARS_OPENAI_COMPAT_BASE_URL=https://api.deepseek.com/v1` (or MiMo / self-hosted) + `ARS_CROSS_MODEL=<provider model id, not gpt-*/gemini-*>`, with a one-line note that this path is ungrounded (positive verdicts never count as citation-agreement confirmations) and that the model id must not collide with a first-party prefix (per D1 precedence).
|
||||
|
||||
The Supported Models table keeps the compatible row but states the ungrounded boundary in the table itself (not only in a footnote that the Integrity section can contradict).
|
||||
|
||||
### D5 — Verification: behavioral fixtures first, lint second
|
||||
|
||||
Both reviewers landed on the same correction: the load-bearing proof is **behavioral** (does an ungrounded `VERIFIED` reach the agreement count?), not a doc-string lint (does the block contain the word `NOT_SEARCHED`?). So the verification has two tiers, behavioral first.
|
||||
|
||||
**Tier 1 — behavioral fixtures (the real guard).** Drive the documented handler logic against canned compatible-provider responses and assert the normalized output + agreement effect:
|
||||
|
||||
1. compatible returns `VERIFIED` (with a plausible-looking DOI) → normalized status `NOT_SEARCHED`, grounded-agreement count increments by 0, raw `VERIFIED` text appears in no parseable verdict slot.
|
||||
2. compatible returns `NOT_FOUND` → passes through as a disagreement (not silently dropped).
|
||||
3. compatible returns `MISMATCH` → passes through as a disagreement.
|
||||
4. base URL with a trailing slash → endpoint has exactly one `/chat/completions` and no `/v1/v1`.
|
||||
|
||||
These mirror the existing canonical-jq behavioral tests (`test_cross_model_verification_guards.py`): extract the handler's verdict-normalization into a small testable unit so the fixtures run in CI, rather than asserting against prose.
|
||||
|
||||
**Tier 2 — doc-sync lint (regression backstop, narrowed).** Extend `check_cross_model_verification_sync.py` to pin, against **executable bash only** (the lint already strips comments/prose via `_bash_code_lines` + `_strip_trailing_comment`):
|
||||
|
||||
1. **Compatible downgrade present:** the compatible block's executable bash references `NOT_SEARCHED`.
|
||||
2. **No passive base-url downgrade:** no executable *assignment or expansion* of `OPENAI_BASE_URL` in the detection/call blocks — match `OPENAI_BASE_URL=` and `${OPENAI_BASE_URL` patterns, not a raw substring anywhere (so explanatory prose/comments like "we deliberately do not read `OPENAI_BASE_URL`" don't false-fail, and `${OPENAI_BASE_URL:-…}` variants don't false-pass).
|
||||
3. **Endpoint normalised:** executable bash constructing the compatible endpoint contains no literal `/v1/v1` and no hardcoded `api.openai.com` fallback.
|
||||
|
||||
Mutation discipline (per prior linter-mutation lessons): commit working tree before mutating; align mutation string case with the lint's matching (avoid IGNORECASE false-greens); each new check gets a red-then-green mutation proving it fails when the contract is broken.
|
||||
|
||||
### D6 — Credential isolation: `ARS_OPENAI_COMPAT_API_KEY`
|
||||
|
||||
The compatible path must NOT reuse `OPENAI_API_KEY` as its bearer token. A user who sets `ARS_OPENAI_COMPAT_BASE_URL=https://api.deepseek.com/v1` but still has a real, billing-enabled `OPENAI_API_KEY` in their environment would otherwise send that OpenAI key in the `Authorization` header to a third-party endpoint — a real credential-leak across a trust boundary. The compatible call uses a dedicated `ARS_OPENAI_COMPAT_API_KEY`:
|
||||
|
||||
- detection's `openai_compatible` branch requires `ARS_OPENAI_COMPAT_API_KEY` (not `OPENAI_API_KEY`) to be set; missing → warn + `none`.
|
||||
- the compatible curl uses `Authorization: Bearer $ARS_OPENAI_COMPAT_API_KEY`.
|
||||
- `OPENAI_API_KEY` is sent only to `api.openai.com` (the grounded first-party path), never to a compatible endpoint.
|
||||
|
||||
This is the same trust-boundary class as the prior path-fix POSIX side-effect lesson: fixing one defect (passive downgrade) must not silently open another (key egress to an untrusted host).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Peer-review sixth reviewer (stays planned).
|
||||
- Any new status enum beyond the existing `NOT_SEARCHED` (decision A in brainstorm: reuse, don't invent).
|
||||
- DA agent file edits (no hardcoded endpoint exists; grep-verified).
|
||||
- Grounding for compatible providers (structurally impossible; not a goal).
|
||||
- First-party grounded verification *through* an `OPENAI_BASE_URL`-style Azure/proxy (see Follow-up).
|
||||
|
||||
## Follow-up (separate issue, not this PR)
|
||||
|
||||
The dual-track review surfaced a genuine pre-existing gap: ARS has no way to run **grounded first-party OpenAI** verification through an enterprise proxy / Azure endpoint. This predates PR #453 (the original code hardcodes `api.openai.com`) and is a distinct feature, not a regression introduced here. File it as its own enhancement issue; do not bundle it into this rework. The D1 precedence note documents the current limitation for users.
|
||||
|
||||
**Altitude follow-up (from the /simplify altitude review, 2026-06-16).** The first-party blocks use a *canonical-and-referenced* pattern: the grounding logic lives in `.jq` files the doc bash **executes** via `jq -f`, so there is one definition, no drift, and no parity-lint needed. The compatible block instead **inlines** the normalization in bash while `normalize_compat_verdict.py` is referenced only in comments (never executed) — which is why lint check 8 (token-set parity) and check 5's literal-endpoint block-locator exist at all. The deeper altitude: have the compatible bash *call* `python3 normalize_compat_verdict.py` (the way other blocks call their `.jq`), making the Python canonical, letting the behavioral test cover the executed path, and dissolving check 8 + check 5's fragility. Deferred, not done in this PR, because it adds a `python3` runtime assumption to a call path that currently needs only curl+jq+grep (jq is already in the first-party path; python3 in the verdict-parse step is new surface for agents running the bash in varied environments). Worth a separate issue weighing the python3-in-call-path cost against the canonical-unit win.
|
||||
|
||||
## Adoption
|
||||
|
||||
Local rebuild (not fetch+cherry-pick of the fork) per repo fork-PR discipline: a single commit on a feature branch off `main`, carrying `Co-Authored-By: kzccIneko`. Then a courteous PR reply crediting the contributor and stating the provider-agnostic intent was adopted with a grounding-aware reshaping — without dissecting their diff line by line.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `shared/cross_model_verification.md` | D1 precedence rewrite, D2 selective normalization + producer/consumer contract, D3 DA critique-only scope, D4 setup guide, D6 credential isolation, contradiction cleanup |
|
||||
| `scripts/check_cross_model_verification_sync.py` | D5 Tier-2 three narrowed contract checks (executable-bash only) |
|
||||
| `scripts/test_check_cross_model_verification_sync.py` | D5 Tier-2 mutation tests (red-then-green per check) |
|
||||
| `scripts/cross_model_verification/` + a new behavioral test | D5 Tier-1 verdict-normalization unit + fixtures (compatible `VERIFIED`→`NOT_SEARCHED`, agreement 0; `NOT_FOUND`/`MISMATCH` pass-through; endpoint normalization) |
|
||||
@@ -188,6 +188,10 @@ path = "scripts/test_cross_model_verification_guards.py"
|
||||
id = "346-349-cross-model-sync-lint"
|
||||
path = "scripts/test_check_cross_model_verification_sync.py"
|
||||
|
||||
[[pytest]]
|
||||
id = "453-compat-verdict-normalization"
|
||||
path = "scripts/test_normalize_compat_verdict.py"
|
||||
|
||||
[[pytest]]
|
||||
id = "361-judge-prompt-version-drift"
|
||||
path = "scripts/test_check_judge_prompt_version.py"
|
||||
|
||||
@@ -65,6 +65,39 @@ def _strip_trailing_comment(line: str) -> str:
|
||||
return line
|
||||
|
||||
|
||||
def _bash_blocks(text: str) -> list[list[str]]:
|
||||
"""Return per-fence lists of executable bash lines (comments removed).
|
||||
|
||||
Each element is one ```bash … ``` fence's executable lines, with comment-only lines dropped
|
||||
and trailing comments stripped (see `_strip_trailing_comment`). Keeping fences separate lets a
|
||||
check scope itself to a single block (e.g. the compatible-provider block) instead of the whole
|
||||
doc — necessary when a token (`NOT_SEARCHED`) legitimately appears in several blocks but a
|
||||
given contract only binds one of them.
|
||||
"""
|
||||
blocks: list[list[str]] = []
|
||||
current: list[str] | None = None
|
||||
for raw in text.splitlines():
|
||||
stripped = raw.strip()
|
||||
if current is None and stripped.startswith("```bash"):
|
||||
current = []
|
||||
continue
|
||||
if current is not None and stripped == "```":
|
||||
blocks.append(current)
|
||||
current = None
|
||||
continue
|
||||
if current is not None and not stripped.startswith("#"):
|
||||
code = _strip_trailing_comment(raw)
|
||||
if code.strip():
|
||||
current.append(code)
|
||||
# An unterminated trailing ```bash block (no closing fence at EOF) must still be scanned —
|
||||
# otherwise its lines silently drop, which is fail-OPEN for checks 4/6/7/8 (a re-inlined guard,
|
||||
# a passive OPENAI_BASE_URL, or a missing normalizer invocation in the final block would escape
|
||||
# the lint). Flush it here.
|
||||
if current is not None:
|
||||
blocks.append(current)
|
||||
return blocks
|
||||
|
||||
|
||||
def _bash_code_lines(text: str) -> list[str]:
|
||||
"""Return the executable lines inside ```bash fenced blocks, comments removed.
|
||||
|
||||
@@ -74,19 +107,8 @@ def _bash_code_lines(text: str) -> list[str]:
|
||||
doc loading that filter via `jq -f`.
|
||||
"""
|
||||
lines: list[str] = []
|
||||
in_block = False
|
||||
for raw in text.splitlines():
|
||||
stripped = raw.strip()
|
||||
if not in_block and stripped.startswith("```bash"):
|
||||
in_block = True
|
||||
continue
|
||||
if in_block and stripped == "```":
|
||||
in_block = False
|
||||
continue
|
||||
if in_block and not stripped.startswith("#"):
|
||||
code = _strip_trailing_comment(raw)
|
||||
if code.strip():
|
||||
lines.append(code)
|
||||
for block in _bash_blocks(text):
|
||||
lines.extend(block)
|
||||
return lines
|
||||
|
||||
|
||||
@@ -153,6 +175,75 @@ def main() -> int:
|
||||
f"`jq -f` instead so the guard stays behavior-tested"
|
||||
)
|
||||
|
||||
# 5. (#453) REMOVED. The compatible block no longer re-implements verdict precedence /
|
||||
# fail-closed defaulting inline — it INVOKES the canonical normalize_compat_verdict.py
|
||||
# (check 8 below pins the wiring; the unit's behavioral tests pin the fail-closed contract).
|
||||
# The old check 5 pinned an inline `*) status="NOT_SEARCHED"` case that no longer exists, so
|
||||
# keeping it would either fail-vacuously or false-fail. Its intent now lives in check 8.
|
||||
|
||||
# 6. (#453) No passive OPENAI_BASE_URL assignment/expansion in executable bash (prose is fine).
|
||||
# Target assignment (`OPENAI_BASE_URL=`) and expansion (`${OPENAI_BASE_URL`/`$OPENAI_BASE_URL`),
|
||||
# NOT a raw substring, so an explanatory comment/prose mention doesn't false-fail and a
|
||||
# `${OPENAI_BASE_URL:-…}` variant doesn't false-pass.
|
||||
if re.search(r"(?<![A-Z_])OPENAI_BASE_URL=", bash_code) or re.search(
|
||||
r"\$\{?OPENAI_BASE_URL\b", bash_code
|
||||
):
|
||||
failures.append(
|
||||
"executable bash reads/sets OPENAI_BASE_URL; the compatible path must use "
|
||||
"ARS_OPENAI_COMPAT_BASE_URL (reading the standard SDK var silently downgrades "
|
||||
"existing first-party users — the #453 passive-downgrade regression)"
|
||||
)
|
||||
|
||||
# 7. (#453) Endpoint construction never builds a double /v1 or falls back to api.openai.com.
|
||||
if "/v1/v1" in bash_code:
|
||||
failures.append("executable bash contains a literal '/v1/v1' (double-/v1 endpoint bug)")
|
||||
if re.search(r"api\.openai\.com[^\n]*chat/completions", bash_code):
|
||||
failures.append(
|
||||
"compatible endpoint must not fall back to api.openai.com/chat/completions; "
|
||||
"require ARS_OPENAI_COMPAT_BASE_URL"
|
||||
)
|
||||
|
||||
# 8. (#453) The compatible block must INVOKE the canonical normalizer via a PIPE, not
|
||||
# re-implement verdict logic in bash. Scope to the compatible block (located by its
|
||||
# endpoint identifier) and require `| python3 ... normalize_compat_verdict.py` so a bare
|
||||
# comment mention or an unpiped/dead reference can't satisfy the check. Anti-vacuity:
|
||||
# if the compatible path is present but its block can't be located, fail loud.
|
||||
# (Earlier inline bash re-implemented leftmost-of-four precedence and risked a head->tail
|
||||
# regression + a raw-text injection of a second STATUS line; calling the behavior-tested
|
||||
# Python unit, which emits single-line JSON, removes both. _bash_blocks already strips
|
||||
# comment-only and trailing comments, so a `# ... normalize_compat_verdict.py` comment is
|
||||
# gone before this check sees it — the pipe requirement is belt-and-braces on top.)
|
||||
if "openai_compatible" in bash_code:
|
||||
compat_blocks = [
|
||||
"\n".join(b) for b in _bash_blocks(text)
|
||||
if "ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions" in "\n".join(b)
|
||||
]
|
||||
if not compat_blocks:
|
||||
failures.append(
|
||||
"compatible path is present (openai_compatible) but the lint could not locate "
|
||||
"the compatible call block by its endpoint identifier — keep the "
|
||||
"`ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions` endpoint line so the wiring "
|
||||
"check stays live"
|
||||
)
|
||||
else:
|
||||
compat_code = "\n".join(compat_blocks)
|
||||
# Require a PIPE into the canonical normalizer. `(?:-\S+\s+)*` allows interpreter
|
||||
# flags (e.g. `python3 -u .../normalize_compat_verdict.py`). This wiring check proves
|
||||
# the canonical unit is invoked-by-pipe; it does NOT parse bash control flow, so it
|
||||
# cannot catch a contrived block that pipes to the normalizer, discards its output,
|
||||
# and re-derives status in bash. That residual is out of scope by design — the real
|
||||
# output contract is carried by the behavioral tests on the JSON-emitting unit; a
|
||||
# determined wrong rewrite is a code-review concern, not a static-lint one.
|
||||
if not re.search(
|
||||
r"\|\s*python3?\s+(?:-\S+\s+)*\S*normalize_compat_verdict\.py", compat_code
|
||||
):
|
||||
failures.append(
|
||||
"the compatible block must pipe the model text into "
|
||||
"normalize_compat_verdict.py (`... | python3 .../normalize_compat_verdict.py`); "
|
||||
"a comment mention or unpiped reference does not count — verdict normalization "
|
||||
"must call the canonical, behavior-tested unit, not re-implement it in bash"
|
||||
)
|
||||
|
||||
if failures:
|
||||
print(f"[cross-model-sync] FAIL: {len(failures)} issue(s):")
|
||||
for f in failures:
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Normalize an OpenAI-compatible provider's citation-verification response to ONE status.
|
||||
|
||||
Compatible providers (MiMo / DeepSeek / self-hosted) expose no hosted web-search tool, so
|
||||
there is no grounding trace to evidence a positive verdict. The security invariant:
|
||||
|
||||
- a positive `VERIFIED` is downgraded to NOT_SEARCHED (an ungrounded confirmation can
|
||||
never count as a grounded agreement);
|
||||
- a rejection (NOT_FOUND / MISMATCH) passes through — it is a useful disagreement and
|
||||
needs no grounding to be acted on;
|
||||
- anything else (self-reported NOT_SEARCHED, unparseable text, empty) fails closed to
|
||||
NOT_SEARCHED.
|
||||
|
||||
The consumer (agreement counter) must read ONLY the returned `status`. Raw model text is
|
||||
kept in `context` for humans and is never parsed for a verdict.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# First explicit verdict token wins; matched case-insensitively at a word boundary.
|
||||
_VERDICT_RE = re.compile(r"\b(VERIFIED|NOT_FOUND|MISMATCH|NOT_SEARCHED)\b", re.IGNORECASE)
|
||||
|
||||
# Rejections survive as-is; a positive is downgraded; everything else fails closed.
|
||||
_PASS_THROUGH = {"NOT_FOUND", "MISMATCH"}
|
||||
|
||||
|
||||
def normalize_compat_verdict(raw: str) -> dict:
|
||||
"""Return {"status": <normalized>, "context": <raw>} for a compatible-provider response."""
|
||||
raw = raw or ""
|
||||
m = _VERDICT_RE.search(raw)
|
||||
token = m.group(1).upper() if m else None
|
||||
# Only rejections survive; VERIFIED (downgrade), self-reported NOT_SEARCHED, or no
|
||||
# recognizable token all fail closed to NOT_SEARCHED.
|
||||
status = token if token in _PASS_THROUGH else "NOT_SEARCHED"
|
||||
return {"status": status, "context": raw}
|
||||
|
||||
|
||||
def _main() -> int:
|
||||
import sys, json
|
||||
raw = sys.stdin.read()
|
||||
result = normalize_compat_verdict(raw)
|
||||
# Single-line JSON: the consumer reads .status; raw text lives JSON-escaped in .context
|
||||
# where embedded newlines become literal \n and cannot inject a second status line.
|
||||
# ensure_ascii=True (the default) additionally \u-escapes ALL non-ASCII, including the
|
||||
# Unicode line separators U+2028/U+2029 that some Unicode-aware consumers treat as line
|
||||
# breaks — so a model response cannot smuggle a second output line via those either. The
|
||||
# context is diagnostic text; ASCII-escaping it is fine.
|
||||
print(json.dumps({
|
||||
"status": result["status"],
|
||||
"provider": "openai_compatible",
|
||||
"context": result["context"],
|
||||
}, ensure_ascii=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
raise SystemExit(_main())
|
||||
@@ -106,3 +106,98 @@ def test_lint_fails_on_double_quoted_inline_grounding_jq(tmp_path, monkeypatch):
|
||||
REINLINE_OLD,
|
||||
'cites="$(jq -r "[.candidates[0].groundingMetadata.groundingChunks[].web.uri]" <<<"$body")"',
|
||||
)
|
||||
|
||||
|
||||
# --- #453 narrowed regression checks (Task 5) ------------------------------------------------
|
||||
|
||||
def test_lint_fails_if_compat_drops_normalizer_invocation(tmp_path, monkeypatch):
|
||||
"""(#453) The compatible block must INVOKE normalize_compat_verdict.py (check 8 wiring).
|
||||
Mutate the canonical-unit invocation away — re-implementing verdict logic inline instead of
|
||||
calling the behavior-tested unit must fail the lint. This single test replaces the three
|
||||
removed check-5 tests (drops-NOT_SEARCHED / precedence-rejection-only / block-identifier-lost),
|
||||
which all pinned the now-deleted inline `case`/`grep` precedence logic."""
|
||||
_assert_lint_fails_on_mutation(
|
||||
tmp_path, monkeypatch,
|
||||
'printf \'%s\' "$text" | python3 "$GUARD/normalize_compat_verdict.py"',
|
||||
'first="$(printf \'%s\' "$text" | grep -oiE \'(NOT_FOUND|MISMATCH)\' | head -1)"',
|
||||
)
|
||||
|
||||
|
||||
def test_lint_fails_if_normalizer_only_in_comment(tmp_path, monkeypatch):
|
||||
"""A commented-out normalizer invocation (with inline logic restored) must NOT satisfy
|
||||
check 8 — the pipe + comment-stripping require a real piped invocation."""
|
||||
_assert_lint_fails_on_mutation(
|
||||
tmp_path, monkeypatch,
|
||||
'printf \'%s\' "$text" | python3 "$GUARD/normalize_compat_verdict.py"',
|
||||
'# normalize via python3 "$GUARD/normalize_compat_verdict.py"\n echo "STATUS: $text"',
|
||||
)
|
||||
|
||||
|
||||
def test_lint_fails_if_compat_block_identifier_lost_v2(tmp_path, monkeypatch):
|
||||
"""(#453) Anti-vacuity: if the endpoint identifier that locates the compatible block is
|
||||
lost, check 8 cannot scope itself and must fail loud (not silently pass). `openai_compatible`
|
||||
still appears in the detection block (`echo "CROSS_MODEL_AVAILABLE=openai_compatible"`), so the
|
||||
guard path runs but the block can no longer be located."""
|
||||
_assert_lint_fails_on_mutation(
|
||||
tmp_path, monkeypatch,
|
||||
'endpoint="${ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions"',
|
||||
'endpoint="$(build_endpoint)"',
|
||||
)
|
||||
|
||||
|
||||
def test_lint_fails_if_openai_base_url_expansion_reintroduced(tmp_path, monkeypatch):
|
||||
"""A passive OPENAI_BASE_URL expansion in executable bash must fail (the passive-downgrade
|
||||
regression). Reintroduce the PR's endpoint line."""
|
||||
_assert_lint_fails_on_mutation(
|
||||
tmp_path, monkeypatch,
|
||||
'endpoint="${ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions"',
|
||||
'endpoint="${OPENAI_BASE_URL:-https://api.openai.com}/v1/chat/completions"',
|
||||
)
|
||||
|
||||
|
||||
def test_lint_fails_if_double_v1_reintroduced(tmp_path, monkeypatch):
|
||||
"""A literal /v1/v1 in executable bash must fail (the double-v1 endpoint bug)."""
|
||||
_assert_lint_fails_on_mutation(
|
||||
tmp_path, monkeypatch,
|
||||
'endpoint="${ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions"',
|
||||
'endpoint="${ARS_OPENAI_COMPAT_BASE_URL%/}/v1/v1/chat/completions"',
|
||||
)
|
||||
|
||||
|
||||
def test_lint_allows_openai_base_url_in_prose(tmp_path, monkeypatch):
|
||||
"""A PROSE/comment mention of OPENAI_BASE_URL (explaining we don't read it) must NOT
|
||||
false-fail — the check targets executable assignment/expansion only."""
|
||||
mod = _load_lint()
|
||||
text = DOC.read_text(encoding="utf-8")
|
||||
injected = text.replace(
|
||||
"## API Call Patterns",
|
||||
"We deliberately never read `OPENAI_BASE_URL` here.\n\n## API Call Patterns",
|
||||
1,
|
||||
)
|
||||
assert injected != text
|
||||
fake = tmp_path / "cross_model_verification.md"
|
||||
fake.write_text(injected, encoding="utf-8")
|
||||
monkeypatch.setattr(mod, "DOC", fake)
|
||||
assert mod.main() == 0
|
||||
|
||||
|
||||
# --- Defensive hardenings against future silent-vacuity (Task 5 follow-up) -------------------
|
||||
|
||||
def test_bash_blocks_includes_unterminated_final_block():
|
||||
"""An unterminated trailing ```bash block must still be scanned (fail-closed parsing).
|
||||
|
||||
Directly exercises `_bash_blocks`: without the EOF flush, a final block with no closing fence
|
||||
silently drops — fail-OPEN for the bash-scanning checks. We assert the OPENAI_BASE_URL
|
||||
expansion inside the unterminated block is recovered, so a downstream check could see it.
|
||||
(A whole-doc main()==1 assertion would be vacuously green here — a minimal synthetic doc fails
|
||||
for unrelated reasons like missing filters — so we pin the parse contract, not the exit code.)"""
|
||||
mod = _load_lint()
|
||||
text = 'intro\n\n```bash\nendpoint="${OPENAI_BASE_URL:-x}/chat/completions"\n' # no closing fence
|
||||
blocks = mod._bash_blocks(text)
|
||||
recovered = [ln for b in blocks for ln in b]
|
||||
assert any("OPENAI_BASE_URL" in ln for ln in recovered), (
|
||||
"unterminated final ```bash block was dropped — its OPENAI_BASE_URL expansion would "
|
||||
f"escape the bash-scanning checks (fail-OPEN). Recovered lines: {recovered!r}"
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Tier-1 behavioral guard for the OpenAI-compatible verdict normalization (#453).
|
||||
|
||||
The security invariant: an ungrounded compatible provider can never launder a positive
|
||||
VERIFIED into a grounded agreement, but a genuine rejection (NOT_FOUND/MISMATCH) is a
|
||||
useful disagreement and must survive. The consumer (agreement counter) reads ONLY the
|
||||
returned `status`; raw model text lives in `context` and is never parsed for a verdict.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
MOD_PATH = REPO / "scripts" / "cross_model_verification" / "normalize_compat_verdict.py"
|
||||
|
||||
|
||||
def _load():
|
||||
spec = importlib.util.spec_from_file_location("normalize_compat_verdict", MOD_PATH)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def _counts_as_grounded_agreement(result) -> bool:
|
||||
"""Mirror the consumer: a row counts toward grounded agreement iff its status is a
|
||||
grounded positive. Compatible never produces one, so this must be False for VERIFIED."""
|
||||
return result["status"] == "VERIFIED"
|
||||
|
||||
|
||||
def test_verified_is_downgraded_and_never_agrees():
|
||||
mod = _load()
|
||||
r = mod.normalize_compat_verdict("VERIFIED — found at https://doi.org/10.1/fake")
|
||||
assert r["status"] == "NOT_SEARCHED"
|
||||
assert _counts_as_grounded_agreement(r) is False
|
||||
|
||||
|
||||
def test_verified_raw_text_not_in_a_parseable_verdict_slot():
|
||||
mod = _load()
|
||||
r = mod.normalize_compat_verdict("VERIFIED https://doi.org/10.1/fake")
|
||||
# raw text may be retained for humans, but only in `context`, never in `status`.
|
||||
assert r["status"] == "NOT_SEARCHED"
|
||||
assert "VERIFIED" not in r["status"]
|
||||
assert r.get("context", "").startswith("VERIFIED") # preserved, but consumer ignores it
|
||||
|
||||
|
||||
def test_not_found_passes_through_as_disagreement():
|
||||
mod = _load()
|
||||
r = mod.normalize_compat_verdict("NOT_FOUND — no matching record exists")
|
||||
assert r["status"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_mismatch_passes_through_as_disagreement():
|
||||
mod = _load()
|
||||
r = mod.normalize_compat_verdict("MISMATCH — year is 2021 not 2019")
|
||||
assert r["status"] == "MISMATCH"
|
||||
|
||||
|
||||
def test_self_reported_not_searched_stays_not_searched():
|
||||
mod = _load()
|
||||
assert mod.normalize_compat_verdict("NOT_SEARCHED — could not search")["status"] == "NOT_SEARCHED"
|
||||
|
||||
|
||||
def test_unparseable_text_defaults_closed_to_not_searched():
|
||||
mod = _load()
|
||||
assert mod.normalize_compat_verdict("the paper looks plausible to me")["status"] == "NOT_SEARCHED"
|
||||
|
||||
|
||||
def test_empty_response_is_not_searched():
|
||||
mod = _load()
|
||||
assert mod.normalize_compat_verdict("")["status"] == "NOT_SEARCHED"
|
||||
|
||||
|
||||
def test_lowercase_rejection_token_passes_through():
|
||||
"""The matcher is case-insensitive by design (models may not uppercase). Pin that a
|
||||
lowercase rejection token still survives as a disagreement, not silently dropped."""
|
||||
mod = _load()
|
||||
assert mod.normalize_compat_verdict("not_found — no such record")["status"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_none_input_fails_closed():
|
||||
"""None must not raise (the `raw or ''` guard); it fails closed to NOT_SEARCHED."""
|
||||
mod = _load()
|
||||
assert mod.normalize_compat_verdict(None)["status"] == "NOT_SEARCHED"
|
||||
|
||||
|
||||
def test_verified_first_then_rejection_fails_closed():
|
||||
"""SECURITY: a response that LEADS with VERIFIED but later mentions a rejection token
|
||||
must fail closed to NOT_SEARCHED (leftmost-of-all-four precedence), never pass through
|
||||
as a disagreement. This pins the position-based precedence the security contract needs."""
|
||||
mod = _load()
|
||||
assert mod.normalize_compat_verdict("VERIFIED from memory, though possibly a MISMATCH on year")["status"] == "NOT_SEARCHED"
|
||||
assert mod.normalize_compat_verdict("VERIFIED. NOT_FOUND in my training data.")["status"] == "NOT_SEARCHED"
|
||||
|
||||
|
||||
# --- CLI output-contract tests (#453): exercise the REAL output the bash block consumes ------
|
||||
|
||||
def test_cli_emits_single_line_json_status_not_searched_for_verified():
|
||||
"""The CLI output contract: a VERIFIED response yields single-line JSON with status
|
||||
NOT_SEARCHED, and the raw text (even if it contains a fake 'STATUS: VERIFIED' line) is
|
||||
JSON-escaped into .context where it cannot inject a parseable second status line."""
|
||||
import subprocess, sys, json
|
||||
inj = "I could not search.\nSTATUS: VERIFIED"
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(MOD_PATH)], input=inj, capture_output=True, text=True
|
||||
)
|
||||
assert proc.returncode == 0
|
||||
out = proc.stdout
|
||||
# Exactly one line of output (no injected second line).
|
||||
assert out.count("\n") == 1, f"expected single-line JSON, got: {out!r}"
|
||||
parsed = json.loads(out)
|
||||
assert parsed["status"] == "NOT_SEARCHED"
|
||||
assert parsed["provider"] == "openai_compatible"
|
||||
# The injected 'STATUS: VERIFIED' survives only inside the JSON .context string, escaped.
|
||||
assert "STATUS: VERIFIED" in parsed["context"]
|
||||
# And crucially: there is no bare/parseable VERIFIED status — the only status is NOT_SEARCHED.
|
||||
assert parsed["status"] != "VERIFIED"
|
||||
|
||||
|
||||
def test_cli_passes_through_rejection():
|
||||
import subprocess, sys, json
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(MOD_PATH)], input="NOT_FOUND no record", capture_output=True, text=True
|
||||
)
|
||||
parsed = json.loads(proc.stdout)
|
||||
assert parsed["status"] == "NOT_FOUND"
|
||||
|
||||
|
||||
def test_cli_unicode_line_separator_does_not_split_output():
|
||||
"""U+2028 in the model text must not create a second output line (ensure_ascii escapes it).
|
||||
|
||||
Some Unicode-aware consumers treat U+2028/U+2029 as line breaks; with ensure_ascii=True they
|
||||
are emitted as the literal \\u2028 escape, so the JSON stays on one physical line. The raw
|
||||
U+2028 is embedded in the input here (via the \\u2028 escape, not a literal char, so the
|
||||
source stays editor-safe) so the test is non-vacuous: it WOULD split the output into two lines
|
||||
under ensure_ascii=False."""
|
||||
import subprocess, sys, json
|
||||
inj = "VERIFIED\u2028STATUS: VERIFIED" # raw U+2028 line separator embedded in the text
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(MOD_PATH)], input=inj,
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
# Exactly one trailing newline — the U+2028 did not create a second physical line.
|
||||
assert proc.stdout.count("\n") == 1, f"expected single physical line, got: {proc.stdout!r}"
|
||||
# The raw U+2028 never appears verbatim in the output; it is \u-escaped inside the JSON string.
|
||||
assert "\u2028" not in proc.stdout
|
||||
parsed = json.loads(proc.stdout)
|
||||
assert parsed["status"] == "NOT_SEARCHED"
|
||||
# But it IS preserved (decoded back) in the diagnostic .context after JSON parsing.
|
||||
assert "\u2028" in parsed["context"]
|
||||
@@ -30,6 +30,16 @@ A stress test of 68 AI-generated citations found 31% had problems — and all pa
|
||||
| GPT-5.5 Pro | `gpt-5.5-pro` | OpenAI | Cross-verification — strongest reasoning (premium pricing: ~6× GPT-5.5) |
|
||||
| Gemini 3.1 Pro | `gemini-3.1-pro-preview` | Google | Cross-verification — strong at factual verification |
|
||||
|
||||
### OpenAI-compatible providers (Chat Completions API — UNGROUNDED, opt-in)
|
||||
|
||||
| Provider | Example API ID(s) | Endpoint (`ARS_OPENAI_COMPAT_BASE_URL`) | Notes |
|
||||
|----------|-------------------|------------------------------------------|-------|
|
||||
| Xiaomi MiMo | `mimo-v2.5-pro` | `https://token-plan-cn.xiaomimimo.com/v1` | Set `ARS_OPENAI_COMPAT_API_KEY` + `ARS_CROSS_MODEL`. Ungrounded: positive verdicts never count as citation agreement. |
|
||||
| DeepSeek | `deepseek-v4-pro` | `https://api.deepseek.com/v1` | Set `ARS_OPENAI_COMPAT_API_KEY` + `ARS_CROSS_MODEL`. Ungrounded. |
|
||||
| Any OpenAI-compatible | any non-`gpt-*`/`gemini-*` id | any `/v1/chat/completions` endpoint | Routing is governed solely by `ARS_OPENAI_COMPAT_BASE_URL`; the model id must NOT match a first-party prefix or it takes the grounded first-party route instead. |
|
||||
|
||||
> **Compatible providers are ungrounded.** They expose no hosted web-search tool, so there is no grounding evidence behind a verdict. A positive `VERIFIED` is downgraded to `NOT_SEARCHED` and never counts as agreement in citation verification; a `NOT_FOUND`/`MISMATCH` survives as a disagreement. They ARE first-class for Devil's Advocate critique (which needs no grounding) — but a DA finding from any provider is an adversarial hypothesis, not standalone evidence, unless independently sourced.
|
||||
|
||||
**Recommended cross-verification pair:** the inherited Claude session model (primary) + GPT-5.5 or Gemini 3.1 Pro (verifier).
|
||||
|
||||
> The primary row deliberately names no version: the primary is always the session model, so the row cannot go stale on the next Anthropic release. Verifier IDs stay concrete because they are literal API strings the user must export. (`gpt-5.4` / `gpt-5.4-pro` remain accepted for existing setups.)
|
||||
@@ -54,18 +64,32 @@ You need API keys from at least one additional provider. ARS itself runs inside
|
||||
2. Create a new API key
|
||||
3. Copy the key (starts with `AIza`)
|
||||
|
||||
**OpenAI-compatible providers (MiMo / DeepSeek / self-hosted):**
|
||||
1. Get an API key from your provider (e.g. [platform.deepseek.com](https://platform.deepseek.com) or the Xiaomi MiMo platform)
|
||||
2. Note the provider's API root including `/v1` (e.g. `https://api.deepseek.com/v1`)
|
||||
3. The key goes in `ARS_OPENAI_COMPAT_API_KEY` and the endpoint in `ARS_OPENAI_COMPAT_BASE_URL` — NOT in `OPENAI_API_KEY`/`OPENAI_BASE_URL` (your real OpenAI key is never sent to a third-party endpoint)
|
||||
4. The compatible model id (`ARS_CROSS_MODEL`) must NOT begin with a `gpt-` or `gemini-` prefix. Any such id is claimed by the first-party grounded route, so a self-hosted compatible model named that way would be routed to the (unavailable) first-party path instead of your compatible endpoint.
|
||||
|
||||
### Step 2: Set Environment Variables
|
||||
|
||||
Add to your shell profile (`~/.zshrc` or `~/.bashrc`):
|
||||
|
||||
```bash
|
||||
# Optional: Cross-model verification for ARS
|
||||
export OPENAI_API_KEY="<your-openai-api-key>"
|
||||
export GOOGLE_AI_API_KEY="<your-google-ai-api-key>"
|
||||
# Cross-model verification for ARS — pick exactly ONE provider tuple.
|
||||
|
||||
# Choose your preferred cross-verification model
|
||||
# Options: gpt-5.5, gpt-5.5-pro, gemini-3.1-pro-preview (gpt-5.4 / gpt-5.4-pro still accepted)
|
||||
# --- Option A: OpenAI (first-party, grounded) ---
|
||||
export OPENAI_API_KEY="<your-openai-api-key>"
|
||||
export ARS_CROSS_MODEL="gpt-5.5"
|
||||
|
||||
# --- Option B: Google Gemini (first-party, grounded) ---
|
||||
export GOOGLE_AI_API_KEY="<your-google-ai-api-key>"
|
||||
export ARS_CROSS_MODEL="gemini-3.1-pro-preview"
|
||||
|
||||
# --- Option C: OpenAI-compatible provider (MiMo / DeepSeek / self-hosted) — UNGROUNDED ---
|
||||
# Uses a DEDICATED key; your real OPENAI_API_KEY is never sent to a third-party endpoint.
|
||||
export ARS_OPENAI_COMPAT_BASE_URL="https://api.deepseek.com/v1" # API root incl. /v1
|
||||
export ARS_OPENAI_COMPAT_API_KEY="<your-provider-api-key>"
|
||||
export ARS_CROSS_MODEL="deepseek-v4-pro" # provider id, NOT gpt-*/gemini-*
|
||||
```
|
||||
|
||||
Then reload: `source ~/.zshrc`
|
||||
@@ -125,7 +149,7 @@ When the integrity_verification_agent detects `ARS_CROSS_MODEL` in the environme
|
||||
Reference: [full reference text] — Context: [sentence where cited]
|
||||
```
|
||||
A `VERIFIED` verdict with no accompanying source URL/DOI is treated as `NOT_SEARCHED` (the model claimed a result it cannot evidence).
|
||||
4. Send to the cross-model via the appropriate API (see API Call Patterns below). **The call patterns enable the provider's web-search/grounding tool and reject the response as `NOT_SEARCHED` when the API returns no grounding evidence** — a model that ignores the "search the web" instruction cannot fake an absent grounding trace, so this is the real safety boundary, not the prompt wording.
|
||||
4. Send to the cross-model via the appropriate API (see API Call Patterns below). **For first-party providers the call patterns enable the hosted web-search/grounding tool and reject the response as `NOT_SEARCHED` when the API returns no grounding evidence** — a model that ignores the "search the web" instruction cannot fake an absent grounding trace, so this is the real safety boundary, not the prompt wording. **An OpenAI-compatible provider has no grounding tool, so its positive verdicts are downgraded to `NOT_SEARCHED` by the handler (rejections pass through); a compatible provider therefore never contributes a grounded agreement.**
|
||||
5. Compare results: if Claude said VERIFIED but cross-model said NOT_FOUND or MISMATCH, flag as `[CROSS-MODEL-DISAGREEMENT]`. Treat `NOT_SEARCHED` / ungrounded exactly as **not verified** — it never counts as agreement with a Claude `VERIFIED`, and a sample that returns `NOT_SEARCHED` is surfaced for re-run or human review, never silently passed.
|
||||
6. Include disagreements in the integrity report under a new section:
|
||||
```markdown
|
||||
@@ -147,6 +171,8 @@ When the integrity_verification_agent detects `ARS_CROSS_MODEL` in the environme
|
||||
- The DA then compares: any CRITICAL or MAJOR issues found by the cross-model but not by the DA are added as `[CROSS-MODEL-FINDING]`
|
||||
- This directly addresses frame-lock — a different model may attack from a different angle
|
||||
|
||||
> A compatible (ungrounded) provider is first-class for DA critique — surfacing weaknesses and attack angles needs no web grounding. But "first-class" is scoped to critique, not factual adjudication: a DA finding from any provider is an adversarial hypothesis, never standalone evidence, unless it carries an independently-checkable source. Do not treat a compatible-provider DA "finding" as a verified defect.
|
||||
|
||||
**When `ARS_CROSS_MODEL` is not set:**
|
||||
- Standard single-model DA (unchanged)
|
||||
|
||||
@@ -179,7 +205,7 @@ The DA agent, after completing its checkpoint report, should:
|
||||
|
||||
## API Call Patterns
|
||||
|
||||
Both patterns below share the same contract: enable the provider's hosted web-search tool, and **gate the model's text on proof that a search actually happened**. If the API returns no grounding evidence (an OpenAI `web_search_call` item / a Gemini `groundingMetadata` block), the call emits `NOT_SEARCHED` and the text is discarded — a model that ignored "search the web" cannot fake an absent grounding trace, so this guard, not the prompt wording, is what prevents a from-memory guess being laundered into `VERIFIED`. Both web-search tools are hosted/server-side: one request, no client-side tool-call round-trip. `PROMPT` holds the single-reference verification prompt from step 3.
|
||||
Three patterns are documented below. The first two (OpenAI and Gemini) are first-party and share the same contract: enable the provider's hosted web-search tool, and **gate the model's text on proof that a search actually happened** — no grounding evidence (an OpenAI `web_search_call` item / a Gemini `groundingMetadata` block) emits `NOT_SEARCHED` and the text is discarded, so this guard, not the prompt wording, is what prevents a from-memory guess being laundered into `VERIFIED`. Both first-party web-search tools are hosted/server-side: one request, no client-side tool-call round-trip. The third (OpenAI-compatible) is ungrounded by construction: it has no web-search tool, so the handler downgrades positive verdicts to `NOT_SEARCHED` and lets rejections through, and a compatible verdict never counts as a grounded agreement. `PROMPT` holds the single-reference verification prompt from step 3.
|
||||
|
||||
### OpenAI (GPT-5.5 / GPT-5.5 Pro)
|
||||
|
||||
@@ -262,6 +288,75 @@ fi
|
||||
|
||||
> **Why `temperature: 0.1`:** reference existence/metadata checking is a deterministic factual task, so low temperature reduces run-to-run variance in the verdict. It is not a grounding control — the grounding guard above is what enforces an actual lookup.
|
||||
|
||||
### OpenAI-Compatible API (MiMo, DeepSeek, self-hosted) — ungrounded
|
||||
|
||||
When `CROSS_MODEL_AVAILABLE=openai_compatible`, use the **Chat Completions API** at
|
||||
`ARS_OPENAI_COMPAT_BASE_URL`, authenticated with the dedicated `ARS_OPENAI_COMPAT_API_KEY`.
|
||||
These providers expose no hosted web-search tool, so there is **no grounding guard**. The
|
||||
handler therefore normalizes the verdict by invoking the canonical
|
||||
`normalize_compat_verdict.py` unit, which emits a single-line JSON object
|
||||
(`{"status","provider","context"}`): a positive `VERIFIED` is downgraded to `NOT_SEARCHED` (an
|
||||
ungrounded confirmation can never count as a grounded agreement), while a genuine rejection
|
||||
(`NOT_FOUND` / `MISMATCH`) passes through as a useful disagreement. The consumer reads `.status`
|
||||
only; the raw model text is JSON-escaped into `.context` as human-readable context and is
|
||||
**never** placed in a verdict slot the agreement counter parses — embedded newlines become
|
||||
literal `\n` inside the string, so a model response cannot inject a second status line. `PROMPT`
|
||||
holds the single-reference verification prompt from step 3.
|
||||
|
||||
```bash
|
||||
# ARS_OPENAI_COMPAT_BASE_URL is the API root INCLUDING /v1 (e.g. https://api.deepseek.com/v1).
|
||||
# Trailing slash is normalized so the endpoint is built exactly once — no double /v1.
|
||||
endpoint="${ARS_OPENAI_COMPAT_BASE_URL%/}/chat/completions"
|
||||
GUARD=scripts/cross_model_verification
|
||||
|
||||
resp="$(curl -sS -w '\n%{http_code}' "$endpoint" \
|
||||
-H "Authorization: Bearer $ARS_OPENAI_COMPAT_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$(jq -n --arg model "$ARS_CROSS_MODEL" --arg prompt "$PROMPT" '{
|
||||
model: $model,
|
||||
messages: [
|
||||
{role: "system", content: "You are a citation-verification assistant. If you did not actually perform an external lookup, respond NOT_SEARCHED. Use NOT_FOUND only if you are confident no such record exists; MISMATCH if a field is wrong; VERIFIED only with a source URL/DOI."},
|
||||
{role: "user", content: $prompt}
|
||||
],
|
||||
temperature: 0.1
|
||||
}')")"
|
||||
|
||||
http="${resp##*$'\n'}"; body="${resp%$'\n'*}"
|
||||
if [ "$http" -lt 200 ] || [ "$http" -ge 300 ]; then
|
||||
# Transport/API failure (401/429/5xx, or curl's 000) — distinct from NOT_SEARCHED, so the
|
||||
# consumer falls back to single-model (see § Graceful Degradation), never an ungrounded verdict.
|
||||
echo "CROSS-MODEL-ERROR: openai_compatible_http_$http"
|
||||
else
|
||||
text="$(jq -r '.choices[0].message.content // empty' <<<"$body")"
|
||||
if [ -z "$text" ]; then
|
||||
echo "CROSS-MODEL-ERROR: openai_compatible_empty_response"
|
||||
else
|
||||
# Canonical normalization lives in scripts/cross_model_verification/normalize_compat_verdict.py
|
||||
# (behavior-tested in scripts/test_normalize_compat_verdict.py) and is INVOKED here rather than
|
||||
# re-implemented in bash — the same canonical-and-referenced pattern the first-party blocks use
|
||||
# with `jq -f`. It emits ONE line of JSON: {"status","provider","context"}. The consumer reads
|
||||
# .status only; raw model text is JSON-escaped in .context so it can never inject a second
|
||||
# status line (the producer/consumer anti-laundering contract holds at the output-format level).
|
||||
# VERIFIED -> status NOT_SEARCHED (ungrounded positive can never agree)
|
||||
# NOT_FOUND/MISMATCH -> status passes through (useful disagreement)
|
||||
# anything else/empty -> status NOT_SEARCHED (fail closed)
|
||||
printf '%s' "$text" | python3 "$GUARD/normalize_compat_verdict.py"
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
> **No grounding guard for compatible providers.** The grounding guard (an API-level
|
||||
> `web_search_call` / `groundingMetadata` trace) exists only for first-party OpenAI and
|
||||
> Gemini. A compatible provider cannot evidence a lookup, so its positive verdicts are
|
||||
> downgraded to `NOT_SEARCHED` and never count as agreement. Its rejections survive as
|
||||
> disagreements. The block emits a single-line JSON object (`{"status","provider","context"}`)
|
||||
> from `normalize_compat_verdict.py`, and the grounded-agreement count is computed solely from
|
||||
> its `.status` field — never from the raw text, which lives JSON-escaped in `.context`.
|
||||
> For the OpenAI-compatible block, read the verdict from the JSON `.status` field only
|
||||
> (e.g. `jq -r .status`); never grep the emitted line or `.context` for a verdict token — the
|
||||
> raw model text is preserved JSON-escaped in `.context` precisely so it cannot be mistaken for
|
||||
> a verdict.
|
||||
|
||||
### Detecting Available Models
|
||||
|
||||
Agents should check at the start of a verification/review session:
|
||||
@@ -274,15 +369,34 @@ if ! command -v jq &>/dev/null; then
|
||||
fi
|
||||
|
||||
if [ -n "$ARS_CROSS_MODEL" ]; then
|
||||
# PRECEDENCE: a first-party model id ALWAYS takes the grounded route, even if
|
||||
# ARS_OPENAI_COMPAT_BASE_URL is set. This prevents a grounded->ungrounded downgrade. ANY gpt-*
|
||||
# id (not just today's gpt-5.5/gpt-5.4) and any gemini-* id route grounded, so a future
|
||||
# first-party release keeps the grounded path instead of silently falling through to the
|
||||
# ungrounded compatible branch. The compatible path is reachable only for a model id that
|
||||
# matches no first-party prefix, and only when its dedicated opt-in env vars are both present.
|
||||
# OPENAI_BASE_URL is never read.
|
||||
case "$ARS_CROSS_MODEL" in
|
||||
gpt-5.5*|gpt-5.4*)
|
||||
gpt-*)
|
||||
[ -n "$OPENAI_API_KEY" ] && echo "CROSS_MODEL_AVAILABLE=openai" \
|
||||
|| echo "WARNING: ARS_CROSS_MODEL=$ARS_CROSS_MODEL but OPENAI_API_KEY is not set" ;;
|
||||
gemini*)
|
||||
gemini*)
|
||||
[ -n "$GOOGLE_AI_API_KEY" ] && echo "CROSS_MODEL_AVAILABLE=google" \
|
||||
|| echo "WARNING: ARS_CROSS_MODEL=$ARS_CROSS_MODEL but GOOGLE_AI_API_KEY is not set" ;;
|
||||
*) echo "WARNING: ARS_CROSS_MODEL=$ARS_CROSS_MODEL is not a supported model. Supported: gpt-5.5, gpt-5.5-pro, gemini-3.1-pro-preview (legacy gpt-5.4* accepted)"
|
||||
echo "CROSS_MODEL_AVAILABLE=none" ;;
|
||||
*)
|
||||
# Unrecognized id: only an explicit, credential-isolated opt-in enables the ungrounded
|
||||
# OpenAI-compatible path. Both the base URL AND the dedicated key are required; the
|
||||
# standard OPENAI_API_KEY is NEVER sent to a third-party endpoint (see Credential
|
||||
# isolation in the API Call Patterns section).
|
||||
if [ -n "$ARS_OPENAI_COMPAT_BASE_URL" ] && [ -n "$ARS_OPENAI_COMPAT_API_KEY" ]; then
|
||||
echo "CROSS_MODEL_AVAILABLE=openai_compatible"
|
||||
elif [ -n "$ARS_OPENAI_COMPAT_BASE_URL" ]; then
|
||||
echo "WARNING: ARS_OPENAI_COMPAT_BASE_URL is set but ARS_OPENAI_COMPAT_API_KEY is not — refusing to send another provider's key. Set ARS_OPENAI_COMPAT_API_KEY."
|
||||
echo "CROSS_MODEL_AVAILABLE=none"
|
||||
else
|
||||
echo "WARNING: ARS_CROSS_MODEL=$ARS_CROSS_MODEL is not a recognized model. First-party grounded route: any gpt-* id (e.g. gpt-5.5, gpt-5.5-pro, legacy gpt-5.4*) or gemini-* id (e.g. gemini-3.1-pro-preview). For an OpenAI-compatible provider set ARS_OPENAI_COMPAT_BASE_URL + ARS_OPENAI_COMPAT_API_KEY and use that provider's model id (must not match a gpt-*/gemini-* prefix, or it takes the grounded first-party route instead)."
|
||||
echo "CROSS_MODEL_AVAILABLE=none"
|
||||
fi ;;
|
||||
esac
|
||||
else
|
||||
echo "CROSS_MODEL_AVAILABLE=none"
|
||||
|
||||
Reference in New Issue
Block a user