skills/imbad0202__academic-research-skills
1
0
Fork 0
mirror of https://github.com/Imbad0202/academic-research-skills.git synced 2026-09-14 13:51:17 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(v3.6.7 Step 6 Phase 6.1): codex audit wrapper + JSONL parser + byte-exact snapshot helper (#55)

Browse Source 
Phase 6.1 deliverables per spec §10 line 2287: a Bash 4+ wrapper that invokes
codex CLI to produce 4 audit contract files (jsonl + sidecar + verdict +
proposal entry), a Python JSONL parser that converts codex 0.125+ event-stream
output into a verdict YAML, and a Python snapshot helper that captures bundle
file bytes + computes per-file SHA-256 + bundle manifest SHA in a single read
(eliminating the TOCTOU window that 5 rounds of Bash-only iteration could not
close).

## Deliverables

- scripts/run_codex_audit.sh (1191 LoC, NEW) — Bash 4+ wrapper per spec §4
  contract end-to-end: input parsing (§4.2), output construction (§4.3),
  atomicity (§4.4 / §4.8), multi-file bundle support (§4.5), failure modes
  (§4.6), wrapper-not-LLM-callable enforcement comment (§4.7), passport
  lifecycle (§4.9). Delegates byte-level snapshot operations to
  audit_snapshot.py. Bash-side responsibility narrows to: input validation,
  process orchestration (codex invocation + tee | PIPESTATUS), atomic write
  (tmp+fsync+rename), SIGTERM trap with cleanup, AUDIT_FAILED short-circuit,
  EXIT trap for partial-file cleanup.

- scripts/audit_snapshot.py (545 LoC, NEW) — Byte-exact snapshot + manifest
  helper per §3.6 + §4.4. Two CLI modes: `snapshot` reads each bundle file
  ONCE via `open(p, "rb").read()`, rejects NUL-containing files (binary not
  supported), computes per-file SHA-256 + bundle manifest SHA from in-memory
  bytes, renders the audit prompt with substituted Section 1/2/4/5 (round
  metadata / commit SHA / agent-specific Section 4(f)) plus verbatim Sections
  3/6/7 from the canonical template, writes manifest.txt + prompt.txt, emits
  JSON summary on stdout. `verify` mode recomputes file SHAs against manifest
  for Step 3a post-codex mutation detection. The single-read architecture
  closes the F-004 TOCTOU window root: in-memory content and SHA derive from
  the same bytes — no second file read, no subshell SHA loss, no Bash
  command-substitution NUL-strip.

- scripts/parse_audit_verdict.py (802 LoC, NEW) — Python 3 stdlib-only JSONL
  parser per §3.5. Two CLI modes: `--probe` (validate stream shape only,
  no YAML output, used by wrapper to gate AUDIT_FAILED branch) and `--jsonl`
  (parse + emit verdict YAML to stdout for atomic write capture). 10 stream-
  shape checks: exactly one thread.started, second event is turn.started,
  no error event, at least one agent_message, first turn.completed appears
  after last agent_message, no events after first turn.completed, all four
  usage.* fields present + non-negative int, input_tokens > 0, canonical
  UUID thread_id, item.id + item.type non-empty strings on item.* events.
  Section 6 verdict text parsing: anchored regex matches the audit-template
  output format, cross-validates summary `(N total)` against bucket sum,
  requires summary be the LAST non-empty line, classifies status with the
  full four-tier rule (PASS / MINOR p3<=3 / MATERIAL p3>3 / AUDIT_FAILED).

- docs/PERFORMANCE.md — added v3.6.7 Step 6 onboarding subsection per spec
  §4 R1 mitigation 2: codex CLI install (brew / vendor installer), required
  dependencies (Bash 4+ via brew on macOS, jq, sha256sum optional), required
  environment (OPENAI_API_KEY or codex SSO), threat-model boundary (§1.2 /
  §4.7 wrapper-not-LLM-callable rule + §3.7 family E lifecycle ownership),
  exit-code contract (§4.6: 0 → PASS/MINOR/MATERIAL, 64 → input validation,
  70 → AUDIT_FAILED, 73 → EX_CANTCREAT, 75 → EX_TEMPFAIL).

- docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md +
  shared/contracts/README.md + shared/contracts/audit/audit_jsonl.schema.json
  — codex 0.125 → 0.125+ forward-compat sweep (17 references). Empirical
  verification: codex 0.128 --json stream shape is byte-identical to 0.125
  (thread.started + thread_id UUID + turn.started + item.completed
  agent_message + turn.completed usage{4 ints}). Boundary history preserved:
  "pre-0.125 drafts" / "Codex 0.125 retired pre-0.125 fields" / "codex
  0.125.0 dropped the pre-0.121 flag set" remain unchanged. Example
  values (codex_cli_version) updated 0.125.0 → 0.128.0.

## Codex review iteration history (10 rounds, 30 unique findings)

R1 (11 findings: 4P1 + 7P2): initial review surfaced jq crash-tolerance
under pipefail, parser stream-shape gaps, full-parse pipe-to-atomic-write
race, TOCTOU snapshot/SHA window, missing operand exit code mismatch,
unvalidated repo-relative paths, lost probe stderr, schema-violating
failure_reason length, duration_seconds clamp, fs failure exit normalization,
Section 6 summary scan with no break.

R2 (8 findings, 7 R1 closures): F-002/004/010/011 partial; new F-012 (P1)
caught a contract drift — original brief omitted MINOR's `p3<=3` upper bound,
caused MATERIAL-class audits with p3>3 to be silently downgraded to MINOR.

R3 (4 findings, 5 R2 closures): F-002/004/010/011 deeper; new F-016 (P3)
on dry-run path validation order.

R4 (4 findings, 4 R3 closures): F-004 STILL partial after 3 rounds (Bash
$(cat)/sha256(file) two-read window); new F-017 (P1) on --previous-findings
not in manifest; new F-018/019 (P2).

R5 (5 findings, 2 R4 closures): F-004 5th-iteration partial. F-018 fix
broke the wrapper (`grep -q $'\\0'` is empty pattern, every non-empty file
rejected as binary). F-020 NEW (P1) on trailing-newline drift. Triggered
ARS memory feedback_architectural_inflection_after_repeated_p1: "3+
consecutive P1 findings on same rule = architectural inflection, stop
whack-a-mole, redesign."

R5→R6 ARCHITECTURAL REDESIGN: Bash-side byte operations (file reads,
SHA computation, NUL detection, manifest building, prompt rendering) moved
to scripts/audit_snapshot.py (NEW). Wrapper simplified from 1342 → 1178
LoC (net -164). End-to-end smoke test 6/6 green (snapshot / verify /
mutation detection / mock codex JSONL / parser probe / parser full / NUL
rejection).

R6 (3 findings, 22/22 closures verified post-redesign): F-004/018/020/021
/022 all verified closed by architecture. New: F-023 (P1, audit template
placeholder substitution missed in render_prompt), F-024 (P2, verify
internal error continued instead of forcing AUDIT_FAILED), F-025 (P3,
parser type guards incomplete on thread_id / usage container).

R7 (2 findings, 25 closures): F-026 (P2, _extract_template_sections ran
Section 7 to EOF embedding the appendix worked example which contained
synthesis-agent 4(f) in non-synthesis prompts), F-027 (P3, JSONL row /
item / item.text type guards incomplete).

R8 (3 findings, 27 closures): all P3 — F-028 (fenced code-block boundary,
not reachable in canonical template), F-029 (ValueError uncaught + manifest
left behind), F-030 (item:null silently skipped).

R9 (1 finding, 28 closures): F-030 partial — empty item dict still skipped.

R10 (0 findings, 29 closures + F-028 documented known limitation):
**CONVERGENCE REACHED.** Codex round 10 verdict:
"Architecturally, the redesign is sound: byte-exact snapshotting in Python,
proposal-last lifecycle, JSONL evidence plus sidecar metadata, and
parser-before-policy separation are coherent."

## Verification gates

- bash -n scripts/run_codex_audit.sh: PASS
- python3 -m py_compile scripts/audit_snapshot.py scripts/parse_audit_verdict.py: PASS
- 6 end-to-end smoke tests (snapshot / verify no-mutation / verify with-mutation /
  mock codex JSONL / parser probe / parser full / NUL rejection): all PASS
- Boundary matrix tests: PASS (PASS/MINOR p3=1, p3=3 / MATERIAL p3=4 / MATERIAL
  p1>0 / MATERIAL p2>0)
- F-022 string usage.input_tokens, F-027 item:null + item.text non-string,
  F-030 R9 empty item + missing item.id: all produce clean ParseError without
  traceback
- F-026 prompt for research_architect_agent verified: no Worked example
  appendix, no fake commit a1b2c3d, no fake chapter4_synthesis.md, no
  synthesis 4(f) leak

Wrapper end-to-end dispatch (real codex invocation) deferred: macOS
stock /bin/bash 3.2 blocks the §4.1 Bash 4+ guard inside this session.
Wrapper preflight correctly exits 64 on Bash 3.2 (verified). Full
end-to-end gate per §10 Phase 6.1 verification list (3 invocation modes:
--dry-run / synthetic / TOCTOU SHA mutation injection) defers to Phase 6.8
or to a deployment running Bash 4+ (brew install bash on macOS).

Known limitation: F-028 (fenced code-block boundary in
_extract_template_sections) not fixed per ARS memory
feedback_codex_review_vs_resume_audit_scope.md "补 counter 不改 rule → 停"
principle. ARS canonical audit template at
shared/templates/codex_audit_multifile_template.md does not contain `## `
inside fenced code blocks; the wrapper pins this template path as a const
(rejecting --audit-template flag), so the bug is not reachable through
the Phase 6.1 production code path. Full Markdown fence-state tracking
deferred to Phase 6.8 if needed.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
2026-05-02 13:30:39 +08:00
committed by GitHub
parent f594d3e2fd
commit 092a0b9655
7 changed files with 2612 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PERFORMANCE.md
+59
View File
@@ -134,3 +134,62 @@ When the Material Passport carries a non-empty `literature_corpus[]`, Phase 1 re
| ~500 entries (large library) | +~25-40K input + ~8-12K output | Consider trimming the corpus before passport emit |
Step 2 search-fills-gap reduces external-DB cost when `uncovered_topics` is small (case A), which can offset Step 1 cost. Empirical net delta pending real systematic-review run instrumentation; until then, no aggregate numeric claim is made. Parse failures cost roughly one short turn (parse + emit `[CORPUS PARSE FAILURE]` + fall back).
## v3.6.7 Step 6 cross-model audit wrapper (onboarding)
v3.6.7 Step 6 ships `scripts/run_codex_audit.sh` and `scripts/parse_audit_verdict.py`, which dispatch a separate codex CLI process to audit `synthesis_agent`, `research_architect_agent` (survey-designer mode), and `report_compiler_agent` (abstract-only mode) deliverables before stage transitions. The wrapper is the boundary object between deployment-side audit execution and ARS-side artifact verification — see [spec §4](design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md) for the full contract.
### codex CLI install + credentials
The wrapper invokes `codex exec --json -m gpt-5.5 -c 'model_reasoning_effort="xhigh"'`. Required setup before first audit run:
| Step | macOS | Linux / WSL |
|---|---|---|
| Install codex CLI | `brew install codex` (or vendor installer) | vendor installer |
| Verify install | `codex --version` should print a `codex-cli X.Y.Z` line; the wrapper requires bare-semver match `^[0-9]+\.[0-9]+\.[0-9]+$` | same |
| Authenticate | `codex login` (browser SSO) OR set `OPENAI_API_KEY=...` in shell rc | same |
| Bash 4+ | `brew install bash` (stock macOS ships 3.2 — not supported) | distro default usually 5.x |
| `jq` | `brew install jq` | distro package |
| `sha256sum` (optional — wrapper falls back to `shasum -a 256`) | `brew install coreutils` | preinstalled |
The wrapper preflights every dependency at startup (§4.1 dependency table) and exits 64 (`EX_USAGE`) with `missing dependency: <name>` before touching any artifact file. No partial state is written when a dependency is missing.
### Required environment
| Variable | Required? | Purpose |
|---|---|---|
| `OPENAI_API_KEY` (or codex SSO session) | yes | codex CLI authentication |
| Standard PATH containing `codex`, `git`, `jq`, `python3` | yes | wrapper preflight checks each |
| Working directory inside the ARS repo | yes | `git rev-parse --short HEAD` for `runner.git_sha`, repo-relative paths in sidecar |
The wrapper does NOT read any ARS-specific environment variables (no `ARS_*` flags affect Step 6). Audit dispatch is purely flag-driven.
### Threat-model boundary
Per [spec §1.2](design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md), Step 6 defends against Pattern C3 (sub-agent fabricated audit metadata, where a downstream agent claims to have run audit and surfaces simulated findings). The artifact-as-contract design closes the inadvertent-overclaim attack surface: the wrapper produces four cryptographically-pinned files (jsonl + sidecar + verdict + proposal entry), and the orchestrator runs eleven layered verification checks before merging any audit outcome into the Material Passport.
Step 6 is **known-insufficient against motivated forgery** (a determined attacker with shell access could invoke codex with a different model and the JSONL stream alone would not betray them — see §9.1 R2). Two structural mitigations apply:
- **§4.7 wrapper-not-LLM-callable rule.** The wrapper MUST NOT be invoked by an in-LLM Bash tool call from the same session producing the audited deliverable. Allowed invokers: human interactive shell, CI workflow step, SubagentStop hook, second-terminal Bash from outside the LLM session. Same-session in-LLM invocation creates the Pattern C3 motive-and-means combination the artifact-as-contract design was chosen to disrupt.
- **Lifecycle ownership (§3.7 family E).** The wrapper writes only to `--output-dir`, never to the passport file. The orchestrator is the sole writer of `audit_artifact[]` entries. A wrapper-emitted proposal carrying `verdict.verified_at` or `verdict.verified_by` is rejected as malformed at lifecycle step 4 (Pattern C3 attack surface).
Deployment-side discipline is the load-bearing defense for the residual motivated-forgery surface; ARS spec-level enforcement is the partial inversion of the three downstream agent prompts (§6) so the agent has no awareness that a downstream audit exists, removing the trigger for fabricated tool-call hallucination.
### Wrapper exit-code contract
The wrapper's process exit code always agrees with the verdict it just wrote (§4.6):
| Exit code | Meaning | Verdict status | Orchestrator response |
|---|---|---|---|
| `0` | Audit completed cleanly | `PASS` / `MINOR` / `MATERIAL` | Read verdict, run §5.2 eleven gating checks, ship or block per §5.3 |
| `64` (`EX_USAGE`) | Input validation failed | none (no files written) | Block, surface `<missing flag>` error to user |
| `70` (`EX_SOFTWARE`) | codex itself exited 70 OR `parse_audit_verdict.py --probe` rejected JSONL OR bundle TOCTOU mutation detected | `AUDIT_FAILED` (with `failure_reason`) | §5.6 Path B5 short-circuits to BLOCK without gating |
| `73` (`EX_CANTCREAT`) | Tee write failed (disk full / EIO) | none / partial (cleaned up) | Block, surface filesystem error |
| `75` (`EX_TEMPFAIL`) | codex rate-limited OR SIGTERM/SIGINT received | `AUDIT_FAILED` | Same as 70: BLOCK without gating; deployment may apply backoff before retry |
| Other non-zero (1, 2, 137, …) | codex exited with a code not enumerated above; wrapper preserves the code rather than normalizing | `AUDIT_FAILED` | Same as 70: BLOCK without gating |
Even on AUDIT_FAILED, the wrapper writes all four contract files (jsonl placeholder + sidecar with `process.exit_code` carrying codex's actual exit + verdict.yaml carrying `status: AUDIT_FAILED` + proposal entry) so orchestrator can distinguish "audit ran but failed" (proposal exists with `AUDIT_FAILED`) from "audit never ran" (no proposal at all). Both states block transition; only `PASS / MINOR / MATERIAL` proposals reach the eleven gating checks.
### Cost posture
A typical Phase 2 chapter audit (synthesis + verification + bibliography bundle) runs codex `gpt-5.5` at `xhigh` reasoning effort for 30-90 seconds wall-clock per round. ARS-side cost is constant: the wrapper adds ~1-2 KB of metadata (sidecar + proposal entry) per audit run regardless of bundle size; the orchestrator's eleven-gate verification is sub-second per audit. The dominant cost is codex API usage on the deployment side, governed by audit template Section 1's three-round convergence target (§10 ship-quality target update).
docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md
+13 -13
View File
@@ -60,7 +60,7 @@ Two rounds of decisions converged before this spec started: a four-question brai
|---|----------|----------|-------------|
| Q1 | Trigger scope — which agents auto-trigger audit? Just the three v3.6.7 downstream agents (A), Phase 2 + bibliography (B), or all stages (C)? | **A — v3.6.7-only.** Trigger only on `synthesis_agent`, `research_architect_agent` (survey-designer mode), `report_compiler_agent` (abstract-only mode). `bibliography_agent` patterns and other agents tracked separately. | §5 stage-transition gate fires only at the three v3.6.7 agents' exit transitions. The five `bibliography_agent` hallucination patterns documented in `feedback_ars_bibliography_agent_hallucination_patterns.md` are out of scope for v3.6.7; they are candidates for v3.6.8+ on their own subline. Future scope expansion follows the same artifact-as-contract structure but is not a Step 6 deliverable. |
| Q2 | Execution model — where does audit run? Inline orchestrator dispatch (A), external hook script (B), sub-agent dispatch (C), or artifact-as-contract (D)? | **D — artifact-as-contract.** Orchestrator does not execute codex; it verifies that an audit artifact exists in the passport, that it passes Layer 2 + Layer 3 anti-fake-audit checks, and reads its verdict. Audit execution is a deployment concern. | §4 specifies the wrapper script contract; §5 specifies the orchestrator block protocol. The orchestrator agent prompt grows by one new "Audit Artifact Gate" subsection (§5.6) and does not internalize codex CLI semantics. Token cost of audit becomes a deployment-side operational cost, not a per-pipeline-run LLM cost. |
| Q3 | Anti-fake-audit guard — how does the orchestrator detect a forged audit artifact? Existence check only (Layer 1), JSONL schema (Layer 2), sidecar metadata (Layer 3), user attestation (Layer 4), or cryptographic signing (Layer 5)? | **Layer 2 + Layer 3 + partial inversion.** Layer 2 validates codex 0.125 `--json` event-stream shape (opens with `thread.started` + `turn.started`; closes with `turn.completed` carrying non-zero `usage` integers; the body interleaves `item.completed` `agent_message` events — codex thinking aloud and the final verdict — with `item.started` / `item.completed` pairs for any tool calls, in arbitrary order; canonical UUID `thread_id`; parseable verdict text in the **last** `agent_message` `item.completed` per audit template Section 6 — see §3.3 for the full schema, the tool-call event pair, and §3.7 family A row A7 for the stream-shape pairing invariant). Layer 3 validates a sidecar `<run_id>.meta.json` written by the wrapper script (CLI version, hostname, cwd, git SHA, timestamps, exit code, `jsonl_thread_id` matching the JSONL stream's `thread.started` event). Partial inversion: agent prompts retain the "DO NOT simulate any audit step" prohibition but remove sentences that reveal the audit pipeline exists downstream (e.g., "The orchestrator runs codex audit afterward" — those sentences trigger Pattern C3 sub-agent fake-audit hallucination per `feedback_subagent_tool_hallucination.md`). | §5.2 specifies the eleven gating verification checks (4 Layer 2 gates from L2-2/L2-3/L2-4/L2-5 + 7 Layer 3 gates from L3-2 through L3-8); two precondition slots L2-1/L3-1 are file/schema existence checks reported as "audit not conducted" rather than gating failures, so they are not counted in the eleven. §6 specifies the partial inversion sweep across the three downstream agent prompts. Layer 4 (user attestation) was rejected as violating the "automatic audit" goal; Layer 5 (signing) was rejected as over-engineering relative to the threat model. Two independent failure modes (Layer 2 + Layer 3) raise forgery cost to the point where running real codex is the economical path under the §1.2 threat model. |
| Q3 | Anti-fake-audit guard — how does the orchestrator detect a forged audit artifact? Existence check only (Layer 1), JSONL schema (Layer 2), sidecar metadata (Layer 3), user attestation (Layer 4), or cryptographic signing (Layer 5)? | **Layer 2 + Layer 3 + partial inversion.** Layer 2 validates codex 0.125+ `--json` event-stream shape (opens with `thread.started` + `turn.started`; closes with `turn.completed` carrying non-zero `usage` integers; the body interleaves `item.completed` `agent_message` events — codex thinking aloud and the final verdict — with `item.started` / `item.completed` pairs for any tool calls, in arbitrary order; canonical UUID `thread_id`; parseable verdict text in the **last** `agent_message` `item.completed` per audit template Section 6 — see §3.3 for the full schema, the tool-call event pair, and §3.7 family A row A7 for the stream-shape pairing invariant). Layer 3 validates a sidecar `<run_id>.meta.json` written by the wrapper script (CLI version, hostname, cwd, git SHA, timestamps, exit code, `jsonl_thread_id` matching the JSONL stream's `thread.started` event). Partial inversion: agent prompts retain the "DO NOT simulate any audit step" prohibition but remove sentences that reveal the audit pipeline exists downstream (e.g., "The orchestrator runs codex audit afterward" — those sentences trigger Pattern C3 sub-agent fake-audit hallucination per `feedback_subagent_tool_hallucination.md`). | §5.2 specifies the eleven gating verification checks (4 Layer 2 gates from L2-2/L2-3/L2-4/L2-5 + 7 Layer 3 gates from L3-2 through L3-8); two precondition slots L2-1/L3-1 are file/schema existence checks reported as "audit not conducted" rather than gating failures, so they are not counted in the eleven. §6 specifies the partial inversion sweep across the three downstream agent prompts. Layer 4 (user attestation) was rejected as violating the "automatic audit" goal; Layer 5 (signing) was rejected as over-engineering relative to the threat model. Two independent failure modes (Layer 2 + Layer 3) raise forgery cost to the point where running real codex is the economical path under the §1.2 threat model. |
| Q4 | Relation to v3.6.6 + Step 8 fixture shape. Is v3.6.6 generator/evaluator contract the same axis as Step 6 audit, and does Step 8 use single corpus (A), per-pattern micro-fixtures (B), or hybrid (C)? | **Orthogonal + Hybrid (C).** v3.6.6 (`evaluator_full` Schema 13.1 contract) is in-pipeline LLM self-discipline at the writer/evaluator pair; Step 6 audit is cross-model external codex verification at the deliverable boundary. They run in parallel: a deliverable can carry both an `evaluator_full` contract and an `audit_artifact` entry. Step 8 fixture: 18 per-pattern micro-fixtures + 1 chapter-level integration fixture. | §8 records the orthogonality with a one-paragraph distinction and an explicit "do not merge" non-goal. §7 specifies both fixture tiers; §7.6 wires CI to run unit-level + integration-level test scripts separately. The 18 micro-fixtures align with the existing 29-mutation lint suite's unit philosophy already shipping in v3.6.7 Step 1+2. |
### 2.2 Architecture choice (Approach 1–3)
@@ -241,7 +241,7 @@ These rules live in `scripts/check_audit_artifact_consistency.py`.
Schema for the codex CLI JSONL output that the wrapper script produces. This is the **Layer 2** anti-fake-audit check from Q3: orchestrator validates the JSONL against this schema before reading any verdict.
**Codex 0.125 `--json` event-stream shape (load-bearing).** Codex 0.125 `--json` emits a typed event stream over stdout: each line is a JSON object with a `type` field naming one of `thread.started`, `turn.started`, `item.started`, `item.completed`, `turn.completed`, or `error`. There is no per-row `model` field, no per-row `reasoning_effort` field, no `session_id` field, no `final_message` field, and no per-row `usage` field — those names belonged to a pre-0.125 draft and are retired here. The stable run identifier is `thread_id` carried on the opening `thread.started` event; `usage` lands on the closing `turn.completed`; the assistant verdict text lands inside an `item.completed` event whose `item.type == "agent_message"` and `item.text` carries the structured verdict (severity-bucket count summary per audit template Section 6).
**Codex 0.125+ `--json` event-stream shape (load-bearing).** Codex 0.125+ `--json` emits a typed event stream over stdout (verified compatible across 0.125 through 0.128): each line is a JSON object with a `type` field naming one of `thread.started`, `turn.started`, `item.started`, `item.completed`, `turn.completed`, or `error`. There is no per-row `model` field, no per-row `reasoning_effort` field, no `session_id` field, no `final_message` field, and no per-row `usage` field — those names belonged to a pre-0.125 draft and are retired here. The stable run identifier is `thread_id` carried on the opening `thread.started` event; `usage` lands on the closing `turn.completed`; the assistant verdict text lands inside an `item.completed` event whose `item.type == "agent_message"` and `item.text` carries the structured verdict (severity-bucket count summary per audit template Section 6).
**Tool-call events.** Whenever codex invokes a tool during the run (a `git diff`, a file read, etc.), the stream emits an `item.started` event when the tool call begins and a matching `item.completed` event when it returns. Both events carry an `item` block whose `item.type` names the tool kind (`command_execution`, `file_change`, etc.) — neither is an `agent_message`, so neither participates in verdict extraction. The audit prompt at `shared/templates/codex_audit_multifile_template.md` typically reads bundle files via tools, so most real audits emit at least one `item.started` / `item.completed` pair before the final `agent_message`; the no-tool case is reachable but rare (e.g., codex deciding the bundle context already in the prompt is sufficient and answering directly). L2-5 enforces only the pairing invariant — every observed `item.started` is matched 1:1 by a later `item.completed` sharing the same `item.id` (parallel tool starts complete in any order, see the schema-rule paragraph below) — not a minimum tool count, because requiring ≥1 tool pair would over-scope the anti-fake-audit guard (Q3 minimal-Layer-2 surface) and reject legitimate no-tool audits. Pre-0.125 drafts that enumerated only four events were wrong about the wire format; PR #52 round-4 surfaced the omission via empirical capture of `codex exec --json` against a live tool-using run.
@@ -283,9 +283,9 @@ Schema-level rules:
- A clean run ends with `turn.completed`. A failed run may end with `error` (or be truncated by SIGKILL — see §4.4 Step 2a JSONL placeholder).
- The **last** `agent_message` `item.completed` event in the stream carries the verdict text. Tool-using runs may emit intermediate `agent_message` events between tool-call `item.started` / `item.completed` pairs (codex thinking aloud); the verdict-bearing event is always the last `agent_message` `item.completed`, but it is NOT the final stream event — `turn.completed` (carrying `usage`) is always emitted after it as the closing event. The wrapper extracts this via `parse_audit_verdict.py` (§4.4 Step 4); the verdict text format is contracted by `shared/templates/codex_audit_multifile_template.md` Section 6 (severity-bucket count summary), NOT by this schema. (Pre-PR-#52 drafts said "exactly one" `agent_message`; that was correct for the no-tool minimum but rejected the actual tool-using audit shape.)
- `item.started` events appear only in tool-using runs and pair 1:1 with later `item.completed` events sharing the same `item.id`. Codex may start multiple tools concurrently (e.g., `item_1`, `item_2`, `item_3` start in that order) and complete them in any order (e.g., `item_1`, `item_3`, `item_2`); pairing is by `item.id` only, not by completion-order FIFO. They are evidence that codex actually invoked tools (anti-fake-audit signal); verdict logic ignores them. **Pairing is a stream-level invariant, not a per-row schema rule** — Layer 2 (`audit_jsonl.schema.json`) validates each row's shape and §5.2 L2-3/L2-4 cover the canonical opening / closing / final-`agent_message` slots; the 1:1 `item.started` ↔ `item.completed` pairing check is enumerated in §3.7 family A row A7 and enforced by `scripts/check_audit_artifact_consistency.py` (Phase 6.3). Tool-event pairing was added to the canonical wire-format description here so implementers know the shape, but the gating burden stays per-row at Layer 2 by design (Q3 chose minimal Layer 2 surface; stream-level is Phase 6.3 territory). The §3.7 row makes the rule discoverable to the lint-script author without re-reading §3.3 prose.
- Model + reasoning effort enforcement is **invocation-side, not stream-side**: the wrapper invokes codex with `-m gpt-5.5 -c 'model_reasoning_effort="xhigh"'` per §4.4 Step 2b. There is no event-stream field that records these — they are properties of the CLI invocation captured in the sidecar's `prompt` block, not Layer 2 evidence per se. Layer 3 therefore cannot cross-check the stream against `model = "gpt-5.5"`; this is a known limit of the 0.125 `--json` shape and is captured in §9.1 R2 motivated-forgery boundary (a forger with shell access could invoke codex with a different model and the JSONL alone would not betray them; deployment-side discipline per §3.7 E9 is the defense).
- Model + reasoning effort enforcement is **invocation-side, not stream-side**: the wrapper invokes codex with `-m gpt-5.5 -c 'model_reasoning_effort="xhigh"'` per §4.4 Step 2b. There is no event-stream field that records these — they are properties of the CLI invocation captured in the sidecar's `prompt` block, not Layer 2 evidence per se. Layer 3 therefore cannot cross-check the stream against `model = "gpt-5.5"`; this is a known limit of the 0.125+ `--json` shape and is captured in §9.1 R2 motivated-forgery boundary (a forger with shell access could invoke codex with a different model and the JSONL alone would not betray them; deployment-side discipline per §3.7 E9 is the defense).
**Why these specific fields:** they are the events codex 0.125 actually emits, validated for shape. A forger needs to fabricate a canonical UUID `thread_id` (8-4-4-4-12 layout), the canonical event ordering (with tool-call pairs if claiming a tool-using audit), `usage` integers on `turn.completed`, AND a syntactically-valid final `item.completed.item.text` `agent_message` that passes `parse_audit_verdict.py` (which itself enforces audit-template Section 6 schema). Layer 2 alone does not guarantee genuineness, but it raises the floor — `echo '{}' >> fake.jsonl` no longer passes; the canonical event sequence is the minimum forgery surface.
**Why these specific fields:** they are the events codex 0.125+ actually emits, validated for shape. A forger needs to fabricate a canonical UUID `thread_id` (8-4-4-4-12 layout), the canonical event ordering (with tool-call pairs if claiming a tool-using audit), `usage` integers on `turn.completed`, AND a syntactically-valid final `item.completed.item.text` `agent_message` that passes `parse_audit_verdict.py` (which itself enforces audit-template Section 6 schema). Layer 2 alone does not guarantee genuineness, but it raises the floor — `echo '{}' >> fake.jsonl` no longer passes; the canonical event sequence is the minimum forgery surface.
**`parse_audit_verdict.py` contract** (Phase 6.1 deliverable per §10): reads the JSONL stream, extracts the **last** `item.completed` event whose `item.type == "agent_message"` (intermediate `agent_message` events from codex thinking aloud between tool calls are skipped), and converts the audit-template Section 6 structured text into the `<run_id>.verdict.yaml` shape (§3.5). On parse failure (no agent_message event at all, malformed Section 6 text in the final agent_message, etc.), emits AUDIT_FAILED verdict with `failure_reason: "JSONL parse error: <reason>"` per §4.6 case (b). The wrapper invokes parser with `--probe` first to validate the JSONL has parseable shape; absence of parseable shape triggers the AUDIT_FAILED branch in §4.4 Step 4.
@@ -296,7 +296,7 @@ Schema for the sidecar metadata file. This is the **Layer 3** anti-fake-audit ch
```yaml
# <run_id>.meta.json structure (validated by Layer 3 schema)
run_id: 2026-04-30T15-22-04Z-d8f3
codex_cli_version: 0.125.0 # from `codex --version`
codex_cli_version: 0.128.0 # from `codex --version`
runner:
hostname: imbad-mbp.local # `uname -n`
cwd: /Users/imbad/Projects/academic-research-skills
@@ -668,9 +668,9 @@ _sha256() {
# AUDIT_FAILED; this helper is the canonical path that emits the empty value
# cleanly without crashing.
#
# codex 0.125 emits the thread_id only on the opening `thread.started` event
# codex 0.125+ emits the thread_id only on the opening `thread.started` event
# (per §3.3 event-stream contract). Earlier draft parsed `.session_id` per
# every row — that field does not exist in 0.125 output and the parse
# every row — that field does not exist in 0.125+ output and the parse
# returned null on every clean run. The current jq filter selects only events
# whose `type == "thread.started"` and pulls their `thread_id`.
_extract_jsonl_thread_id() {
@@ -683,8 +683,8 @@ _extract_jsonl_thread_id() {
| head -1
}
# Extract semver from `codex --version` output. codex 0.125 prints
# "codex-cli 0.125.0"; §3.4 sidecar schema's `codex_cli_version` field is a
# Extract semver from `codex --version` output. codex 0.125+ prints
# "codex-cli X.Y.Z" (e.g., "codex-cli 0.128.0"); §3.4 sidecar schema's `codex_cli_version` field is a
# bare semver string with regex constraint `^[0-9]+\.[0-9]+\.[0-9]+$`. Match
# the first dotted-triple in stdout and reject (EX_USAGE) if absent — defends
# against future CLI versions that print multi-line stdout or unexpected
@@ -814,7 +814,7 @@ prompt=$(render_template "$audit_template_path" \
# pre-0.121 flag set; the current (and only) way to control reasoning effort
# is the `-c model_reasoning_effort=...` config override (TOML-quoted), and
# `--json` replaces the older `--output-format jsonl`. There are no
# `--output`, `--stdout-log`, or `--stderr-log` flags in 0.125 — codex prints
# `--output`, `--stdout-log`, or `--stderr-log` flags in 0.125+ — codex prints
# the event-stream to stdout, which we split via a real pipeline + tee into
# the JSONL contract file plus the <run_id>.stdout diagnostic file. Stderr
# goes to <run_id>.stderr at shell level.
@@ -1007,7 +1007,7 @@ emit_proposal_entry \
**Atomicity guarantee:**
Three of the four contract files (sidecar, verdict, proposal entry) are written via the tmp-file-rename pattern: wrapper writes to `<file>.tmp`, fsyncs, then `mv <file>.tmp <file>`. The fourth file, JSONL, is intentionally exempt: codex 0.125 `--json` writes its event stream to stdout (the CLI has no `--output` flag in 0.125; see §3.3 + §4.4 Step 2b), and the wrapper splits stdout via a `tee | >` pipeline into `<run_id>.jsonl` (the contract artifact) and `<run_id>.stdout` (diagnostic copy). The pipeline runs over the audit's runtime (potentially minutes), and tmp-rename of a streamed file would defeat live debugging via `tail -f`. Step 2a additionally pre-touches an empty JSONL placeholder so the file exists at a known path even when codex is killed before emitting any events (F-059 closure). JSONL therefore may be observed mid-write or empty during an audit run, but **no orchestrator code path ever consumes JSONL until a proposal entry file references it** (see proposal-last guarantee below) — so JSONL's lack of tmp-rename atomicity is not observable to orchestrator. Step 2b's `${PIPESTATUS[@]}` capture additionally surfaces tee-side write failures (disk full / EIO during the streaming write) as wrapper exit 73 with cleanup of all partial artifacts, so a corrupted JSONL never reaches the proposal-write step. For AUDIT_FAILED proposals, §5.6 Path B5 short-circuits BLOCK before Layer 2 validation runs (gating against incomplete evidence is meaningless — see §5.6 B5). For PASS / MINOR / MATERIAL proposals, §5.2 L2-1 (file exists) and L2-2 (every event validates against `audit_jsonl.schema.json`) reject missing or partial JSONL at the gate, so a non-AUDIT_FAILED proposal whose JSONL is partially-written never ships.
Three of the four contract files (sidecar, verdict, proposal entry) are written via the tmp-file-rename pattern: wrapper writes to `<file>.tmp`, fsyncs, then `mv <file>.tmp <file>`. The fourth file, JSONL, is intentionally exempt: codex 0.125+ `--json` writes its event stream to stdout (the CLI has no `--output` flag in 0.125+; see §3.3 + §4.4 Step 2b), and the wrapper splits stdout via a `tee | >` pipeline into `<run_id>.jsonl` (the contract artifact) and `<run_id>.stdout` (diagnostic copy). The pipeline runs over the audit's runtime (potentially minutes), and tmp-rename of a streamed file would defeat live debugging via `tail -f`. Step 2a additionally pre-touches an empty JSONL placeholder so the file exists at a known path even when codex is killed before emitting any events (F-059 closure). JSONL therefore may be observed mid-write or empty during an audit run, but **no orchestrator code path ever consumes JSONL until a proposal entry file references it** (see proposal-last guarantee below) — so JSONL's lack of tmp-rename atomicity is not observable to orchestrator. Step 2b's `${PIPESTATUS[@]}` capture additionally surfaces tee-side write failures (disk full / EIO during the streaming write) as wrapper exit 73 with cleanup of all partial artifacts, so a corrupted JSONL never reaches the proposal-write step. For AUDIT_FAILED proposals, §5.6 Path B5 short-circuits BLOCK before Layer 2 validation runs (gating against incomplete evidence is meaningless — see §5.6 B5). For PASS / MINOR / MATERIAL proposals, §5.2 L2-1 (file exists) and L2-2 (every event validates against `audit_jsonl.schema.json`) reject missing or partial JSONL at the gate, so a non-AUDIT_FAILED proposal whose JSONL is partially-written never ships.
If the wrapper crashes between any two writes, only `.tmp` artifacts (for the three tmp-renamed files), the empty/partial JSONL placeholder, and possibly `.stdout`/`.stderr` remain. Orchestrator finds no proposal entry → treats run as not-conducted (per §5 rule). No false PASS reachable through partial writes.
@@ -2206,7 +2206,7 @@ R1 does NOT motivate a "skip audit" command (§5.7 hard rule). The friction is t
The risk: a future maintainer reads the eleven gating checks (§5.2) and assumes they defend against any audit-artifact tampering, then adds new features (e.g., "auto-resume on audit pass") that depend on stronger trust than Layer 2 + Layer 3 actually provide. Such a feature would be safe under the inadvertent-Pattern-C3 threat model but unsafe under the motivated-forgery model that real-world deployment may face.
**Concrete gap in 0.125 stream evidence (load-bearing for §3.3).** Codex 0.125 `--json` carries no `model` field and no `reasoning_effort` field anywhere in the event stream — both are invocation-side properties of the CLI flags (`-m gpt-5.5`, `-c 'model_reasoning_effort="xhigh"'`) rather than stream-side properties codex echoes back. Layer 2 therefore cannot verify which model actually ran; Layer 3 only records the wrapper-declared invocation in `sidecar.prompt`. A motivated actor with shell access could invoke `codex exec -m gpt-5.6 -c 'model_reasoning_effort="low"' ...` and write a sidecar declaring `model: gpt-5.5` / `reasoning_effort: xhigh`; the eleven gates would not detect that mismatch — every JSONL field they cross-check (`thread_id`, `usage` integers, the `agent_message` event's text) is unaffected by which model produced them. This gap stays inside the rejected motivated-forgery scope: it is exactly the case §3.7 family E row E9 (wrapper not LLM-callable from same session) and the §4.7 invocation-site discipline rule are designed to bound, and exactly the case Layer 5 (cryptographic signing on the codex CLI side) would close. The §1.2 closing paragraph already declares "Layer 5 was considered and rejected as over-engineering for the inadvertent-Pattern-C3 threat surface"; this paragraph names the specific gap that decision tolerates.
**Concrete gap in 0.125+ stream evidence (load-bearing for §3.3).** Codex 0.125+ `--json` carries no `model` field and no `reasoning_effort` field anywhere in the event stream — both are invocation-side properties of the CLI flags (`-m gpt-5.5`, `-c 'model_reasoning_effort="xhigh"'`) rather than stream-side properties codex echoes back. Layer 2 therefore cannot verify which model actually ran; Layer 3 only records the wrapper-declared invocation in `sidecar.prompt`. A motivated actor with shell access could invoke `codex exec -m gpt-5.6 -c 'model_reasoning_effort="low"' ...` and write a sidecar declaring `model: gpt-5.5` / `reasoning_effort: xhigh`; the eleven gates would not detect that mismatch — every JSONL field they cross-check (`thread_id`, `usage` integers, the `agent_message` event's text) is unaffected by which model produced them. This gap stays inside the rejected motivated-forgery scope: it is exactly the case §3.7 family E row E9 (wrapper not LLM-callable from same session) and the §4.7 invocation-site discipline rule are designed to bound, and exactly the case Layer 5 (cryptographic signing on the codex CLI side) would close. The §1.2 closing paragraph already declares "Layer 5 was considered and rejected as over-engineering for the inadvertent-Pattern-C3 threat surface"; this paragraph names the specific gap that decision tolerates.
Mitigation:
@@ -2286,7 +2286,7 @@ Each phase has a defined deliverable, a defined verification gate, and an explic
### Phase 6.1 — Wrapper script `scripts/run_codex_audit.sh`
**Deliverables:** `scripts/run_codex_audit.sh` AND `scripts/parse_audit_verdict.py` (the JSONL→verdict parser the wrapper invokes per §4.4 Step 4 to extract the **last** `item.completed.item.text` `agent_message` event's text from the codex 0.125 event stream — intermediate `agent_message` events between tool-call `item.started` / `item.completed` pairs are skipped — and convert it into the `<run_id>.verdict.yaml` file per §3.5). Both implement §4 contract end-to-end.
**Deliverables:** `scripts/run_codex_audit.sh` AND `scripts/parse_audit_verdict.py` (the JSONL→verdict parser the wrapper invokes per §4.4 Step 4 to extract the **last** `item.completed.item.text` `agent_message` event's text from the codex 0.125+ event stream — intermediate `agent_message` events between tool-call `item.started` / `item.completed` pairs are skipped — and convert it into the `<run_id>.verdict.yaml` file per §3.5). Both implement §4 contract end-to-end.
**Includes:**
- Bash 4+ shebang and dependency check (`set -euo pipefail`).
scripts/audit_snapshot.py
Executable
+545
View File
@@ -0,0 +1,545 @@
#!/usr/bin/env python3
"""audit_snapshot.py — ARS v3.6.7 Step 6 Phase 6.1 byte-exact snapshot helper.
Reads bundle files (primary deliverable + supporting context + audit template +
optional previous-findings), validates them as text-only (rejects NUL bytes),
computes SHA-256 from in-memory bytes (single read per file — no TOCTOU window),
emits the canonical bundle manifest, and writes a JSON summary on stdout for
the wrapper to consume.
This script REPLACES the Bash-side `_snapshot_file` helper. The Bash version
had three structural issues (codex review rounds 1-5 surfaced these):
F-004 (P1, 5 rounds partial): Bash command substitution runs the helper in a
subshell. Variable assignments (e.g., `_LAST_SNAPSHOT_SHA`) made inside the
subshell don't propagate to the parent. The wrapper saw empty SHAs.
F-018 (P1): `grep -q $'\\0'` in Bash treats `$'\\0'` as an empty pattern,
matching every non-empty file. Every text file was rejected as binary.
F-020 (P1): `$(cat file)` strips trailing newlines and drops NUL bytes.
Manifest SHA (computed from in-memory content) drifted from `sha256sum file`
(computed from disk bytes). Step 3a always reported false mutation.
The Python implementation reads bytes verbatim, hashes from those exact bytes,
and emits both manifest and SHAs to the wrapper via JSON — no subshell, no
trailing-newline drift, binary-safe NUL detection. The wrapper consumes the
JSON via `python3 audit_snapshot.py ... | jq` or by reading from a tmp file.
CLI modes:
--snapshot
Compute snapshot for a bundle. Writes:
- <output-dir>/<run_id>.manifest.txt (per spec §3.6)
- <output-dir>/<run_id>.prompt.txt (rendered audit prompt)
- JSON summary to stdout: {primary_shas, supporting_shas, template_sha,
manifest_sha, bundle_files, prompt_path}
--verify
Recompute SHAs of bundle files and compare against manifest. Used by Step 3a
of the wrapper to detect post-snapshot disk mutation. Exits 0 if no
mutation, exits 1 with mutated paths on stdout otherwise.
Exit codes:
0 success
64 EX_USAGE — bad arguments or NUL-containing input
1 verify mode: mutation detected (paths printed to stdout, one per line)
2 internal error (file not found, JSON serialization failure, etc.)
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import sys
from pathlib import Path
from typing import Optional
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def read_bytes_or_die(path: str) -> bytes:
"""Read a file's exact bytes. Exits 2 on missing/unreadable file."""
try:
with open(path, "rb") as f:
return f.read()
except FileNotFoundError:
print(f"audit_snapshot: file not found: {path}", file=sys.stderr)
sys.exit(2)
except PermissionError:
print(f"audit_snapshot: permission denied: {path}", file=sys.stderr)
sys.exit(2)
def reject_if_binary(path: str, content: bytes) -> None:
"""Reject NUL-containing input (F-018 closure).
Phase 6.1 audits are UTF-8 text deliverables (markdown, JSON, YAML).
Bash command-substitution would silently strip NUL bytes; Python's
binary-safe `b"\\0" in content` test catches them cleanly.
"""
if b"\0" in content:
print(
f"audit_snapshot: file contains NUL bytes (binary not supported): {path}",
file=sys.stderr,
)
sys.exit(64)
def sha256_hex(content: bytes) -> str:
"""SHA-256 hex digest from in-memory bytes."""
return hashlib.sha256(content).hexdigest()
def _extract_template_sections(template_str: str, section_numbers: list[int]) -> str:
"""Extract specified Section blocks from the audit template.
The template uses `## Section N — Title` headers with `---` separators.
F-023 (P1, R6): we substitute Sections 1/2/4/5 with real round metadata
above; the template's placeholder versions of those sections must not be
embedded too, or codex sees `{N}` / `{git_sha}` literals in the prompt.
F-026 (P2, R7): the LAST extracted section must terminate at the next
top-level `## ` heading (e.g. `## Worked example`, `## Cross-references`),
not at end-of-file. Without this guard, Section 7's extraction would pull
the appendix worked example — which embeds a synthesis_agent 4(f) clause —
into prompts dispatched for non-synthesis agents, contradicting the real
Section 4(f) we substituted above.
Returns concatenated section text (with the leading `## Section N` header
intact and the appendix excluded).
"""
import re as _re
# Match either Section heading or any other top-level `## ` heading
# (e.g. "## Worked example", "## Cross-references"); the latter terminate
# the last extracted section.
boundary_re = _re.compile(r"^## (?:Section (\d+)( —|$)|.+)", _re.MULTILINE)
matches = list(boundary_re.finditer(template_str))
if not matches:
return template_str # no headings found — fall back to verbatim
# Identify which matches are real Section N starts (group(1) populated)
section_starts: list[tuple[int, int]] = [] # (section_num, match_index)
for i, m in enumerate(matches):
if m.group(1) is not None:
section_starts.append((int(m.group(1)), i))
requested = set(section_numbers)
if not requested.issubset({n for n, _ in section_starts}):
missing = requested - {n for n, _ in section_starts}
raise ValueError(
f"audit_snapshot: requested template sections {sorted(missing)} not found "
f"(template has Sections {sorted({n for n, _ in section_starts})})"
)
parts: list[str] = []
for sec_num, match_idx in section_starts:
if sec_num not in requested:
continue
start = matches[match_idx].start()
# End at the next `## ` heading of any kind (Section or appendix),
# not just the next Section. This catches the post-Section-7 appendix.
end = (
matches[match_idx + 1].start()
if match_idx + 1 < len(matches)
else len(template_str)
)
parts.append(template_str[start:end].rstrip() + "\n\n")
return "".join(parts)
_SECTION_4F_BY_AGENT = {
"synthesis_agent": (
"(f) Cross-section consistency check: for every source cited in 2+ "
"sections of the primary deliverable, verify the source's "
"characterization is compatible across sections; flag any pair of "
"sections that pull the source's effect in incompatible directions "
"(Pattern A1)."
),
"research_architect_agent": (
"(f) Construct-equivalence test: for every survey item labelled "
"'reverse-coded', verify it meets the construct-equivalence "
"definition in `shared/references/psychometric_terminology_glossary.md`."
),
"report_compiler_agent": (
"(f) Mandatory three-part check: (i) word count = "
"`len(body.split())` <= publisher cap minus 3-5% buffer per "
"`shared/references/word_count_conventions.md`; (ii) every entry of "
"the upstream `protected_hedges` block per "
"`shared/references/protected_hedging_phrases.md` appears verbatim "
"in the abstract; (iii) no claim in the abstract is less hedged than "
"its anchor in the body. Failure of any sub-check is a P1 finding."
),
}
def render_prompt(
audit_template: bytes,
primary_paths: list[str],
primary_contents: list[bytes],
supporting_paths: list[str],
supporting_contents: list[bytes],
round_n: int,
target_rounds: int,
git_sha: str,
stage: int,
agent: str,
prior_findings: Optional[bytes],
) -> bytes:
"""Render the audit prompt sent to codex stdin.
The prompt embeds the AT-SNAPSHOT-TIME bytes of every bundle file. Codex
sees exactly the bytes whose SHA-256 went into the manifest — there is
no second file read, no TOCTOU window.
F-023 (P1, R6): the audit template at
shared/templates/codex_audit_multifile_template.md contains placeholders
in {curly_braces} that the orchestrator fills before sending to codex
(Section 1 round metadata, Section 2 git_sha, Section 4(f) bundle-specific
check). Round 5 redesign initially embedded the template verbatim,
leaving placeholders unsubstituted — codex would see literal `{N}` in
the prompt and either omit Section 4(f) entirely or produce an
unparsable Section 6 verdict. This implementation now does explicit
bundle-specific substitution and rebuilds Section 1, 2, and 4 around
real values, while preserving Section 3 / 5 / 6 / 7 verbatim from the
template (they describe codex's expected output, not the bundle).
Returns bytes (not str) to preserve binary fidelity.
"""
section_4f = _SECTION_4F_BY_AGENT.get(
agent,
f"(f) Bundle-specific check (no agent-specific clause registered for {agent}; "
"treat 4(f) as N/A and run only (a)-(e)).",
)
if round_n <= 1:
prior_summary = "none (first round, baseline audit)"
elif prior_findings is not None:
prior_summary = (
"(prior findings attached as supporting context — see file listing "
"below; verify each carries-forward / closes per (a) and (e))"
)
else:
prior_summary = "(no previous-findings file provided)"
primary_listing = "\n".join(f"- {p}" for p in primary_paths) or "- (none)"
supporting_listing = (
"\n".join(f"- {p}" for p in supporting_paths) if supporting_paths else "- (none)"
)
# Extract audit template Sections 3, 6, 7 (verbatim — describe codex's
# expected output / dimensions / anti-fake guard). Sections 1, 2, 4, 5 are
# rendered above with real values, so we skip the template's placeholder
# versions to avoid showing codex two copies of the same section (one with
# placeholders, one substituted) — F-023 closure also requires the
# placeholder text not appear.
template_str = audit_template.decode("utf-8", errors="replace")
section_3_to_7 = _extract_template_sections(template_str, [3, 6, 7])
rendered_intro = (
f"# ARS v3.6.7 cross-model audit — round {round_n} of {target_rounds}\n"
f"# Stage: {stage} | Agent: {agent}\n"
f"# Git SHA at audit start: {git_sha}\n\n"
f"## Section 1 — Round metadata\n\n"
f"Audit round: {round_n} of {target_rounds}\n"
f"Previous rounds: {prior_summary}\n"
f"Bundle scope: Stage {stage} deliverable for {agent}\n\n"
f"## Section 2 — Bundle inventory\n\n"
f"Authoritative context (commit {git_sha}):\n\n"
f"Primary deliverables (audit target):\n{primary_listing}\n\n"
f"Supporting context (do not audit; reference only):\n{supporting_listing}\n\n"
f"## Section 4 — Round {round_n} job\n\n"
f"(a) Verify each round-{round_n - 1} finding closed correctly. List by ID.\n"
f"(b) Audit for new issues introduced by round-{round_n - 1} corrections (cascade audit).\n"
f"(c) Run the 7 audit dimensions (§3.1-§3.7) plus the bundle-specific Section 4(f) check on the primary deliverables. Report each finding with the dimension or `4(f)` that surfaced it.\n"
f"(d) Anchoring-bias residual check on closed findings.\n"
f"(e) PARTIAL-vs-CLOSED check.\n"
f"{section_4f}\n\n"
f"## Section 5 — Convergence target\n\n"
f"Convergence target: ZERO findings of ANY severity in one round.\n\n"
)
parts: list[bytes] = []
parts.append(rendered_intro.encode("utf-8"))
parts.append(section_3_to_7.encode("utf-8"))
parts.append(b"\n\n## Primary deliverables (audit target)\n\n")
for path, content in zip(primary_paths, primary_contents):
parts.append(f"--- PRIMARY: {path} ---\n".encode("utf-8"))
parts.append(content)
parts.append(b"\n")
if supporting_paths:
parts.append(b"\n## Supporting context (reference only)\n\n")
for path, content in zip(supporting_paths, supporting_contents):
parts.append(f"--- SUPPORTING: {path} ---\n".encode("utf-8"))
parts.append(content)
parts.append(b"\n")
return b"".join(parts)
def write_manifest(
out_path: str,
primary_shas: list[tuple[str, str]],
supporting_shas: list[tuple[str, str]],
audit_template_path: str,
audit_template_sha: str,
) -> str:
"""Write the canonical bundle manifest per §3.6.
Format: <role>:<repo-relative-path>:<sha256-hex>, one line per file,
sorted by (role, path) via the equivalent of `LC_ALL=C sort`.
Returns the manifest's own SHA-256 (== `bundle_manifest_sha`).
"""
lines: list[str] = []
for path, sha in primary_shas:
lines.append(f"primary:{path}:{sha}")
for path, sha in supporting_shas:
lines.append(f"supporting:{path}:{sha}")
lines.append(f"template:{audit_template_path}:{audit_template_sha}")
lines.sort()
manifest_text = "\n".join(lines) + "\n"
manifest_bytes = manifest_text.encode("utf-8")
try:
with open(out_path, "wb") as f:
f.write(manifest_bytes)
except OSError as e:
print(f"audit_snapshot: manifest write failed: {e}", file=sys.stderr)
sys.exit(2)
return sha256_hex(manifest_bytes)
def write_prompt(out_path: str, prompt_bytes: bytes) -> None:
"""Write the rendered prompt to a file the wrapper feeds to codex stdin."""
try:
with open(out_path, "wb") as f:
f.write(prompt_bytes)
except OSError as e:
print(f"audit_snapshot: prompt write failed: {e}", file=sys.stderr)
sys.exit(2)
def dedupe_preserving_order(items: list[str]) -> list[str]:
"""Remove duplicates while preserving first occurrence order (F-021)."""
seen: set[str] = set()
out: list[str] = []
for x in items:
if x not in seen:
seen.add(x)
out.append(x)
return out
# ---------------------------------------------------------------------------
# CLI modes
# ---------------------------------------------------------------------------
def cmd_snapshot(args: argparse.Namespace) -> int:
"""Snapshot bundle files, write manifest + prompt, emit JSON summary."""
primary = list(args.primary or [])
supporting = list(args.supporting or [])
if args.previous_findings:
# Inject --previous-findings into supporting (preserves §3.6 3-role enum).
# Dedup happens AFTER normalization (which the wrapper does upstream).
supporting.append(args.previous_findings)
supporting = dedupe_preserving_order(supporting)
if not primary:
print("audit_snapshot: --primary is required", file=sys.stderr)
return 64
if not args.audit_template:
print("audit_snapshot: --audit-template is required", file=sys.stderr)
return 64
if not args.output_dir or not args.run_id:
print("audit_snapshot: --output-dir and --run-id are required", file=sys.stderr)
return 64
audit_template_bytes = read_bytes_or_die(args.audit_template)
reject_if_binary(args.audit_template, audit_template_bytes)
audit_template_sha = sha256_hex(audit_template_bytes)
primary_contents: list[bytes] = []
primary_shas: list[tuple[str, str]] = []
for p in primary:
content = read_bytes_or_die(p)
reject_if_binary(p, content)
primary_contents.append(content)
primary_shas.append((p, sha256_hex(content)))
supporting_contents: list[bytes] = []
supporting_shas: list[tuple[str, str]] = []
for s in supporting:
content = read_bytes_or_die(s)
reject_if_binary(s, content)
supporting_contents.append(content)
supporting_shas.append((s, sha256_hex(content)))
# F-029 (P3, R8): render the prompt BEFORE writing manifest/prompt to disk.
# If _extract_template_sections raises ValueError (requested sections not
# found in template), no partial artifact is left in --output-dir.
# F-023 (P1, R6): pass stage/agent/prior_findings so render_prompt can
# substitute audit-template placeholders and select the agent-specific
# Section 4(f) clause.
prior_findings_bytes: Optional[bytes] = None
if args.previous_findings:
# PREV_FINDINGS was just snapshotted as part of supporting; locate its content.
for path, content in zip([s for s, _ in supporting_shas], supporting_contents):
if path == args.previous_findings:
prior_findings_bytes = content
break
try:
prompt_bytes = render_prompt(
audit_template_bytes,
[p for p, _ in primary_shas],
primary_contents,
[s for s, _ in supporting_shas],
supporting_contents,
args.round,
args.target_rounds,
args.git_sha or "unknown",
args.stage,
args.agent,
prior_findings_bytes,
)
except ValueError as e:
# F-029: clean exit 64 with no partial artifact left behind.
print(f"audit_snapshot: prompt render failed: {e}", file=sys.stderr)
return 64
# Now safe to write — render succeeded, no traceback path.
manifest_path = os.path.join(args.output_dir, f"{args.run_id}.manifest.txt")
bundle_manifest_sha = write_manifest(
manifest_path,
primary_shas,
supporting_shas,
args.audit_template,
audit_template_sha,
)
prompt_path = os.path.join(args.output_dir, f"{args.run_id}.prompt.txt")
write_prompt(prompt_path, prompt_bytes)
summary = {
"manifest_path": manifest_path,
"manifest_sha": bundle_manifest_sha,
"prompt_path": prompt_path,
"audit_template_path": args.audit_template,
"audit_template_sha": audit_template_sha,
"primary_files": [{"path": p, "sha": sha} for p, sha in primary_shas],
"supporting_files": [{"path": s, "sha": sha} for s, sha in supporting_shas],
}
json.dump(summary, sys.stdout, indent=2)
sys.stdout.write("\n")
return 0
def cmd_verify(args: argparse.Namespace) -> int:
"""Verify bundle files against the snapshot manifest.
Reads the manifest file written by --snapshot mode, recomputes each
referenced file's SHA-256 from current disk content, and reports any
mismatches. Used by the wrapper's Step 3a (post-codex mutation detection).
"""
if not args.manifest:
print("audit_snapshot: --manifest is required for --verify", file=sys.stderr)
return 64
try:
with open(args.manifest, encoding="utf-8") as f:
manifest_text = f.read()
except OSError as e:
print(f"audit_snapshot: manifest read failed: {e}", file=sys.stderr)
return 2
mutated_paths: list[str] = []
for line in manifest_text.splitlines():
line = line.strip()
if not line:
continue
# Format: <role>:<path>:<sha>
# The path may contain colons (POSIX paths can include them, though
# rare); split into 3 parts maximum from the left (role + path + sha)
# but the SHA is always the LAST 64 hex chars after a colon.
if ":" not in line:
continue
# Find the SHA suffix: last colon, then 64 hex chars.
if len(line) < 65 or line[-65] != ":":
continue
path_with_role = line[:-65]
expected_sha = line[-64:]
# Split role:path
role_sep = path_with_role.find(":")
if role_sep == -1:
continue
# role = path_with_role[:role_sep] # not needed for verify
path = path_with_role[role_sep + 1 :]
# Recompute SHA from current disk content
try:
with open(path, "rb") as f:
actual_sha = sha256_hex(f.read())
except FileNotFoundError:
mutated_paths.append(path)
continue
if actual_sha != expected_sha:
mutated_paths.append(path)
if mutated_paths:
for p in mutated_paths:
print(p)
return 1
return 0
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(
description="ARS v3.6.7 audit bundle snapshot helper. "
"Modes: --snapshot (initial) or --verify (post-codex)."
)
sub = p.add_subparsers(dest="mode", required=True)
snap = sub.add_parser("snapshot", help="Snapshot bundle, write manifest + prompt")
snap.add_argument("--primary", action="append", required=True, help="Primary deliverable path (repeatable)")
snap.add_argument("--supporting", action="append", default=[], help="Supporting context path (repeatable)")
snap.add_argument("--previous-findings", help="Optional --previous-findings path (auto-injected into supporting with dedup)")
snap.add_argument("--audit-template", required=True, help="Audit template path (single)")
snap.add_argument("--output-dir", required=True, help="Where to write manifest.txt and prompt.txt")
snap.add_argument("--run-id", required=True, help="Audit run identifier")
snap.add_argument("--round", type=int, required=True, help="Current audit round")
snap.add_argument("--target-rounds", type=int, required=True, help="Total target rounds")
snap.add_argument("--git-sha", help="Repo HEAD short SHA (informational, embedded in prompt)")
snap.add_argument("--stage", type=int, required=True, help="Stage transition (1-6) the audit gates")
snap.add_argument(
"--agent",
required=True,
choices=["synthesis_agent", "research_architect_agent", "report_compiler_agent"],
help="Which v3.6.7 downstream agent produced the deliverable (selects Section 4(f) clause)",
)
ver = sub.add_parser("verify", help="Verify bundle files against snapshot manifest")
ver.add_argument("--manifest", required=True, help="Path to <run_id>.manifest.txt")
return p
def main(argv: Optional[list[str]] = None) -> int:
parser = build_parser()
args = parser.parse_args(argv)
if args.mode == "snapshot":
return cmd_snapshot(args)
elif args.mode == "verify":
return cmd_verify(args)
return 2
if __name__ == "__main__":
sys.exit(main())
scripts/parse_audit_verdict.py
Executable
+802
View File
@@ -0,0 +1,802 @@
#!/usr/bin/env python3
"""parse_audit_verdict.py — ARS v3.6.7 Step 6 Phase 6.1
Reads a codex CLI 0.125+ --json JSONL event stream and converts the structured
Section 6 verdict text (from the LAST agent_message item.completed event) into
a <run_id>.verdict.yaml file.
Two CLI modes:
--probe <jsonl> Validate only; exit 0 on success, non-zero on failure.
--jsonl <jsonl> --round N --target-rounds M Parse + emit YAML to stdout.
Exit codes:
0 success (probe passed OR full parse emitted YAML)
non-zero parse failure (reason on stderr, one line)
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
from datetime import datetime, timezone
from typing import Optional
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
GENERATED_BY = "scripts/run_codex_audit.sh"
GENERATOR_VERSION = "1.0.0"
# run_id filename pattern: YYYY-MM-DDTHH-MM-SSZ-XXXX (hyphens, not colons)
RUN_ID_RE = re.compile(
r"^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}-[0-9]{2}-[0-9]{2}Z-[0-9a-f]{4}$"
)
# F-019 (§3.3): canonical UUID regex for thread.started.thread_id validation.
# codex 0.125+ always emits a lowercase-hex UUID in this exact format.
# A non-matching value indicates a malformed or forged stream.
_THREAD_ID_RE = re.compile(
r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"
)
# Valid dimension values (string enum per audit_verdict.schema.json)
VALID_DIMENSIONS = {"3.1", "3.2", "3.3", "3.4", "3.5", "3.6", "3.7", "4(f)"}
# ---------------------------------------------------------------------------
# JSONL parsing
# ---------------------------------------------------------------------------
def load_events(jsonl_path: str) -> list[dict]:
"""Read all JSONL rows from the given file path.
Returns a list of parsed dicts. Raises ParseError on file-level failures.
"""
if not os.path.exists(jsonl_path):
raise ParseError(f"jsonl file not found: {jsonl_path}")
with open(jsonl_path, encoding="utf-8") as fh:
raw = fh.read().strip()
if not raw:
raise ParseError(f"jsonl file is empty: {jsonl_path}")
events: list[dict] = []
for lineno, line in enumerate(raw.splitlines(), start=1):
line = line.strip()
if not line:
continue
try:
row = json.loads(line)
except json.JSONDecodeError as exc:
raise ParseError(f"invalid JSON at row {lineno}: {exc}") from exc
# F-027 (P3, R7): each row must be a JSON object so .get/[] work later.
if not isinstance(row, dict):
raise ParseError(
f"jsonl row {lineno} is not an object: {type(row).__name__}"
)
events.append(row)
return events
def validate_stream_shape(events: list[dict]) -> None:
"""Validate that the JSONL event stream has the expected clean-run shape.
F-002 (§3.6) — Round 3 fix: anchors checks 6, 7, 8 on the FIRST
turn.completed event (not the last). Rounds 1-2 used last_turn_completed_idx
throughout, which allowed the multi-turn.completed forgery:
thread.started → turn.started → real_agent_message → turn.completed_1
→ forged_agent_message → turn.completed_2
Under the Round 2 logic: last_turn_completed_idx pointed to turn.completed_2,
so trailing = [] (nothing after TC_2, check 7 passed), and TC_2 < TC_2 was
False (check 6 passed), but extract_verdict_text returned the forged message.
Fix: locate the FIRST turn.completed. Per §3.3 a clean run has exactly ONE
turn.completed, at the end. Check 7 (no trailing events after first TC) rejects
any second TC as an illegal trailing event, and check 6 verifies the first TC
appears after the last agent_message. Both checks together enforce exactly-one-TC.
Per §3.3 canonical no-tool run sequence:
thread.started → turn.started → item.completed(agent_message) → turn.completed
Checks:
1. Exactly one thread.started event (multiple → forgery surface).
2. First event is thread.started.
3. Second event is turn.started (per §3.3 canonical sequence).
4. No error event anywhere in the stream.
5. Stream contains at least one agent_message.
6. FIRST turn.completed appears after the last agent_message.
7. No events after the FIRST turn.completed (clean run ends with first TC;
any trailing event — including a second TC — is a forgery indicator).
8. first turn.completed.usage.input_tokens > 0 (real codex always consumes input).
9. thread.started.thread_id matches canonical UUID regex (F-019 §3.3).
10. All four turn.completed.usage fields present, integers, >= 0 (F-019 §3.3).
Raises ParseError if any check fails. Probe rejection cascades into the
wrapper's AUDIT_FAILED branch per §4.6 case (b).
"""
if not events:
raise ParseError("stream is empty")
# Check 1: exactly one thread.started event (§3.3: thread_id emitted ONCE)
thread_started_count = sum(1 for ev in events if ev.get("type") == "thread.started")
if thread_started_count == 0:
raise ParseError(
f"stream missing thread.started event (got first: {events[0].get('type')!r})"
)
if thread_started_count > 1:
raise ParseError(
f"stream has {thread_started_count} thread.started events; expected exactly 1"
)
# Check 2: first event is thread.started
if events[0].get("type") != "thread.started":
raise ParseError(
f"stream first event is not thread.started (got: {events[0].get('type')!r})"
)
# Check 3: second event is turn.started (per §3.3 canonical sequence)
if len(events) < 2 or events[1].get("type") != "turn.started":
got = events[1].get("type") if len(events) >= 2 else "(stream too short)"
raise ParseError(
f"stream second event is not turn.started (got: {got!r})"
)
# Check 4: no error event anywhere in the stream
if any(ev.get("type") == "error" for ev in events):
raise ParseError("stream contains error event (codex was killed or errored mid-run)")
# Locate the FIRST turn.completed (F-002 Round 3: must use FIRST, not last).
# Per §3.3 a clean run has exactly one turn.completed at the stream end.
# Using last_turn_completed_idx (Rounds 1-2) allowed a forged second TC to
# make trailing[] appear empty while hiding a forged agent_message between TC_1
# and TC_2. Anchoring on FIRST TC collapses that window.
first_turn_completed_idx = -1
for idx, ev in enumerate(events):
if ev.get("type") == "turn.completed":
first_turn_completed_idx = idx
break # stop at FIRST occurrence
# Locate the last agent_message (verdict-bearing event per §3.3 contract).
# F-027 (P3, R7) + F-030 (P3, R8/R9): malformed `item` on item.started/
# item.completed events is rejected outright. Per spec §3.3 schema, `item`
# is an object with required `id` (non-empty string) + `type` (non-empty
# string); object-shaped rows missing those keys are still malformed.
last_agent_msg_idx = -1
for idx, ev in enumerate(events):
ev_type = ev.get("type")
if ev_type not in ("item.completed", "item.started"):
continue
item = ev.get("item")
if not isinstance(item, dict):
raise ParseError(
f"event {idx} ({ev_type}) has malformed item field: "
f"{type(item).__name__}"
)
# F-030 R9 closure: enforce required item.id + item.type as non-empty strings.
item_id = item.get("id")
item_type = item.get("type")
if not isinstance(item_id, str) or not item_id:
raise ParseError(
f"event {idx} ({ev_type}) item.id is not a non-empty string: {item_id!r}"
)
if not isinstance(item_type, str) or not item_type:
raise ParseError(
f"event {idx} ({ev_type}) item.type is not a non-empty string: {item_type!r}"
)
if ev_type == "item.completed" and item_type == "agent_message":
last_agent_msg_idx = idx
# Check 5: at least one agent_message present
if last_agent_msg_idx == -1:
raise ParseError("no agent_message in stream")
# Check 6: FIRST turn.completed must exist and appear after the last agent_message.
# (F-002 R3: anchored on first_turn_completed_idx, not last_turn_completed_idx)
if first_turn_completed_idx == -1:
raise ParseError(
"stream missing turn.completed event (codex was killed mid-run)"
)
if first_turn_completed_idx < last_agent_msg_idx:
raise ParseError(
"turn.completed appears before last agent_message "
"(stream truncated or forged; first turn.completed must follow all agent messages)"
)
# Check 7: NO events after the FIRST turn.completed.
# (F-002 R3: anchored on first_turn_completed_idx, not last_turn_completed_idx)
# This implicitly enforces exactly-one-TC: any second turn.completed would appear
# here as a trailing event and be rejected. Per §3.3 §line 284, a clean codex run
# always ends at the single turn.completed event.
trailing = events[first_turn_completed_idx + 1:]
if trailing:
trailing_types = [ev.get("type") for ev in trailing]
raise ParseError(
f"unexpected event(s) after turn.completed: {trailing_types!r} "
f"(stream must end at first turn.completed per §3.3)"
)
# Check 9 (F-019 §3.3): thread_id must match canonical UUID format.
# codex 0.125+ always emits a lowercase-hex UUID on the thread.started event.
# A non-canonical value indicates a malformed or forged stream header.
# F-025 (P3, R6): thread_id must be a string before regex match.
# `events[0].get("thread_id", "")` returns None / list / int verbatim if
# the key holds those types — feeding non-string to _THREAD_ID_RE.match
# raises TypeError. Explicit type guard turns it into a clean ParseError.
thread_id = events[0].get("thread_id")
if not isinstance(thread_id, str):
raise ParseError(
f"thread.started.thread_id is not a string: {type(thread_id).__name__}"
)
if not _THREAD_ID_RE.match(thread_id):
raise ParseError(
f"thread.started.thread_id is not canonical UUID: {thread_id!r}"
)
# Check 10 (F-019 §3.3): all four usage fields must be present, integers,
# and non-negative. Type validation runs FIRST (F-022 §3.3) so non-int
# values (e.g., "1" as string) raise a one-line ParseError instead of an
# unhandled traceback in the input_tokens > 0 comparison below.
turn_completed = events[first_turn_completed_idx]
# F-025 (P3, R6): usage must be a dict before `in` / index operations.
# JSONL with `usage: null` or `usage: []` would otherwise raise TypeError.
usage = turn_completed.get("usage")
if not isinstance(usage, dict):
raise ParseError(
f"turn.completed.usage is not an object: {type(usage).__name__}"
)
_USAGE_FIELDS = (
"input_tokens",
"cached_input_tokens",
"output_tokens",
"reasoning_output_tokens",
)
for fld in _USAGE_FIELDS:
if fld not in usage:
raise ParseError(f"turn.completed.usage missing field: {fld!r}")
if not isinstance(usage[fld], int) or isinstance(usage[fld], bool):
raise ParseError(
f"turn.completed.usage.{fld} is not int: {usage[fld]!r}"
)
if usage[fld] < 0:
raise ParseError(
f"turn.completed.usage.{fld} is negative: {usage[fld]}"
)
# Check 8: first turn.completed.usage.input_tokens > 0 (§3.3 line 277, Q3 line 63).
# Real codex always consumes input; zero → forgery indicator.
# Runs AFTER Check 10 so input_tokens is guaranteed-int by this point.
if usage["input_tokens"] <= 0:
raise ParseError(
f"turn.completed.usage.input_tokens is {usage['input_tokens']} "
"(real codex runs always consume input tokens; zero indicates forgery)"
)
def extract_verdict_text(events: list[dict]) -> str:
"""Return the text of the LAST item.completed agent_message event.
Per §3.3 contract: tool-using runs emit intermediate agent_message events
between tool calls; the verdict-bearing event is always the LAST one.
F-027 (P3, R7): each event's `item` field must be a dict before .get;
`item.text` for agent_message must be a non-empty string. Malformed types
raise clean ParseError instead of TypeError tracebacks.
"""
agent_messages: list[dict] = []
for idx, row in enumerate(events):
if row.get("type") != "item.completed":
continue
item = row.get("item")
if not isinstance(item, dict):
raise ParseError(
f"event {idx} (item.completed) has malformed item field: "
f"{type(item).__name__}"
)
if item.get("type") != "agent_message":
continue
agent_messages.append(row)
if not agent_messages:
raise ParseError("no agent_message in stream")
last_item = agent_messages[-1].get("item", {})
verdict_text = last_item.get("text")
if not isinstance(verdict_text, str) or not verdict_text:
raise ParseError(
f"agent_message.item.text is not a non-empty string: "
f"{type(verdict_text).__name__}"
)
return verdict_text
# ---------------------------------------------------------------------------
# Section 6 text parsing
# ---------------------------------------------------------------------------
# Summary line patterns (two canonical forms per audit template Section 6):
#
# Form A (findings present):
# "Round N: P1×n1 / P2×n2 / P3×n3 (M total)"
# Form B (zero findings):
# "Round N: 0 findings of any severity. Convergence reached."
#
# Both forms may be preceded by "Previous rounds: Round 1: ... ; all P1 ..." on
# the same or adjacent line — we anchor on the ROUND value passed via CLI to
# pick the authoritative summary line rather than a "previous rounds" recap.
# Matches: Round N: P1×n1 / P2×n2 / P3×n3 (M total)
#
# F-011 (§3.6) Round 2 fixes:
# 1. (M total) parenthetical is now MANDATORY (was optional "(?:...)?").
# A summary line without the parenthetical is malformed per audit template
# Section 6 line 160 and should cascade to "no parseable Section 6 summary"
# → AUDIT_FAILED rather than silently accepting an unchecked total.
# 2. Anchored to start-of-line (^) and end-of-line ($) with re.MULTILINE to
# prevent partial matches on lines containing "Round N:" as a substring
# (e.g., finding entries that restate the round number in their description).
_SUMMARY_A = re.compile(
r"^\s*Round\s+(\d+)\s*:\s*P1[×x×](\d+)\s*/\s*P2[×x×](\d+)\s*/\s*P3[×x×](\d+)"
r"\s*\((\d+)\s*total\)\s*$",
re.IGNORECASE | re.MULTILINE,
)
# Matches: Round N: 0 findings of any severity. Convergence reached.
# F-011: anchored similarly for consistency — avoids substring hits in
# "Previous rounds: Round 1: 0 findings ..." recap lines.
_SUMMARY_B = re.compile(
r"^\s*Round\s+(\d+)\s*:\s*0\s+findings\s+of\s+any\s+severity[.,]?\s*Convergence\s+reached.*$",
re.IGNORECASE | re.MULTILINE,
)
# Finding entry pattern (tolerant of formatting variation):
#
# Codex emits lines like:
# 1. **F-007** P3 §3.7 chapter_4/synthesis.md:482 — description. Fix: fix text.
# 2. **F-001** P1 3.2 path/file.md:10 - description here. Fix: suggested fix.
#
# Groups: (list_num, finding_id, severity, dimension, file, line, description, fix_text)
#
# Strategy:
# - Finding ID: **F-NNN** (bold markdown) or bare F-NNN
# - Severity: P1 / P2 / P3 (standalone word)
# - Dimension: §3.1–§3.7 (with or without § prefix) or 4(f)
# - file:line anchor (greedy non-space path, colon, integer)
# - Description and Fix separated by "Fix:" label (case-insensitive)
#
# We use a two-pass approach: first match the structural header (id/sev/dim/file:line),
# then extract description + fix from the tail.
_FINDING_HEADER = re.compile(
r"^\s*\d+\.\s+" # numbered list item: "1. "
r"\*{0,2}(F-[0-9]{3,})\*{0,2}" # finding ID: **F-007** or F-007
r"\s+"
r"(P[123])" # severity: P1 P2 P3
r"\s+"
r"(?:§)?(3\.[1-7]|4\(f\))" # dimension: §3.x or 4(f) (with or without §)
r"\s+"
r"([^\s:]+)" # file path (no spaces, stops before colon)
r":([0-9]+)" # :line
r"\s*[-—–]+\s*" # separator: — or - or –
r"(.+)", # rest of line (description + fix)
re.IGNORECASE,
)
# Fix label separator within the rest-of-line tail
# e.g.: "deictic phrase. Fix: replace with ..."
# "description here. Fix: suggested fix here."
_FIX_SEPARATOR = re.compile(r"\.\s*Fix\s*:\s*", re.IGNORECASE)
def _parse_dimension(raw: str) -> str:
"""Normalize raw dimension string to the schema enum value."""
d = raw.lstrip("§").strip()
if d in VALID_DIMENSIONS:
return d
raise ParseError(f"malformed finding entry: unknown dimension '{raw}'")
def _parse_finding_line(line: str) -> Optional[dict]:
"""Attempt to parse one numbered finding line. Returns None if no match."""
m = _FINDING_HEADER.match(line)
if not m:
return None
finding_id, severity, dimension_raw, file_path, line_num, tail = m.groups()
finding_id = finding_id.upper()
severity = severity.upper()
dimension = _parse_dimension(dimension_raw)
line_int = int(line_num)
# Split tail into description and suggested_fix at ". Fix: "
fix_parts = _FIX_SEPARATOR.split(tail, maxsplit=1)
if len(fix_parts) == 2:
description = fix_parts[0].strip().rstrip(".")
suggested_fix = fix_parts[1].strip().rstrip(".")
else:
# No "Fix:" separator found — description consumes entire tail,
# suggested_fix is missing → malformed entry.
raise ParseError(
f"malformed finding entry: no 'Fix:' separator in line: {line!r}"
)
if not description:
raise ParseError(f"malformed finding entry: empty description for {finding_id}")
if not suggested_fix:
raise ParseError(
f"malformed finding entry: empty suggested_fix for {finding_id}"
)
return {
"id": finding_id,
"severity": severity,
"dimension": dimension,
"file": file_path,
"line": line_int,
"description": description,
"suggested_fix": suggested_fix,
}
def parse_section6(
text: str,
current_round: Optional[int] = None,
) -> tuple[dict, list[dict]]:
"""Parse Section 6 verdict text.
Returns (finding_counts, findings_list) where:
finding_counts = {"p1": int, "p2": int, "p3": int}
findings_list = list of finding dicts
Raises ParseError if no authoritative summary line is found or if
finding_counts disagrees with the parsed findings list.
current_round: when provided, use to locate the authoritative summary line
(avoiding "Previous rounds: Round 1: …" recaps). When None (probe mode),
accept the last parseable summary line.
"""
lines = text.splitlines()
# ---- Step 1: locate the authoritative summary line -----
# We scan ALL lines and collect all summary-line matches.
# If current_round is known, we require the summary to match that round.
# If not (probe), we accept any match and take the last one.
summary_counts: Optional[tuple[int, int, int]] = None # (p1, p2, p3)
# F-011: track the authoritative summary line's text for last-line check
_authoritative_summary_line: Optional[str] = None
for line in lines:
# Try Form B first (zero-findings convergence line)
mb = _SUMMARY_B.search(line)
if mb:
r = int(mb.group(1))
if current_round is None or r == current_round:
summary_counts = (0, 0, 0)
_authoritative_summary_line = line.strip()
if current_round is not None:
break
continue
# Try Form A (P1×n / P2×n / P3×n)
ma = _SUMMARY_A.search(line)
if ma:
r, n1, n2, n3 = (
int(ma.group(1)),
int(ma.group(2)),
int(ma.group(3)),
int(ma.group(4)),
)
if current_round is None or r == current_round:
# F-011: cross-validate (N total) parenthetical when present.
# "Round 2: P1×0 / P2×3 / P3×1 (5 total)" should reject because
# 0+3+1=4 ≠ 5. group(5) is None when parenthetical is absent.
total_group = ma.group(5)
if total_group is not None:
total_claimed = int(total_group)
total_computed = n1 + n2 + n3
if total_claimed != total_computed:
raise ParseError(
f"summary line total {total_claimed} disagrees with "
f"bucket sum {total_computed} (P1×{n1}+P2×{n2}+P3×{n3})"
)
summary_counts = (n1, n2, n3)
_authoritative_summary_line = line.strip()
if current_round is not None:
break
if summary_counts is None:
raise ParseError("no parseable Section 6 summary")
# F-011: require the authoritative summary line to be the LAST non-empty
# line of the verdict text. Audit template Section 6 format puts the summary
# at the end; an interior summary line indicates a malformed or shadowed
# verdict (e.g. "Previous rounds" recap in the wrong position).
non_empty_lines = [ln.strip() for ln in lines if ln.strip()]
if non_empty_lines and _authoritative_summary_line is not None:
last_nonempty = non_empty_lines[-1]
# The authoritative summary line must match the last non-empty line.
# We do a substring check because the line may be embedded in a longer
# line (e.g. with markdown emphasis markers around it).
if _authoritative_summary_line not in last_nonempty and last_nonempty not in _authoritative_summary_line:
raise ParseError(
"summary line is not the last non-empty line of the verdict text "
"(found interior summary; expected it at the end per audit template Section 6)"
)
# ---- Step 2: parse individual finding entries ----
findings: list[dict] = []
for line in lines:
# Skip summary lines to avoid spurious matches
if _SUMMARY_A.search(line) or _SUMMARY_B.search(line):
continue
result = _parse_finding_line(line)
if result is not None:
findings.append(result)
# ---- Step 3: cross-validate counts vs parsed findings ----
p1_exp, p2_exp, p3_exp = summary_counts
p1_got = sum(1 for f in findings if f["severity"] == "P1")
p2_got = sum(1 for f in findings if f["severity"] == "P2")
p3_got = sum(1 for f in findings if f["severity"] == "P3")
if (p1_got, p2_got, p3_got) != (p1_exp, p2_exp, p3_exp):
raise ParseError(
f"finding_counts disagrees with findings[] "
f"(summary: P1×{p1_exp}/P2×{p2_exp}/P3×{p3_exp}, "
f"parsed: P1×{p1_got}/P2×{p2_got}/P3×{p3_got})"
)
finding_counts = {"p1": p1_exp, "p2": p2_exp, "p3": p3_exp}
return finding_counts, findings
# ---------------------------------------------------------------------------
# Status classification (§3.2 cross-field rule)
# ---------------------------------------------------------------------------
def classify_status(finding_counts: dict) -> str:
"""Derive verdict_status from finding_counts per spec §3.2 cross-field rules.
F-012 (§3.2): original implementation omitted the p3 upper bound, classifying
p3=4 as MINOR instead of MATERIAL. Per spec lines 231-233:
PASS: p1 == 0 AND p2 == 0 AND p3 == 0
MINOR: p1 == 0 AND p2 == 0 AND 1 <= p3 <= 3
MATERIAL: p1 > 0 OR p2 > 0 OR p3 > 3
Contract-critical: a 4-P3-finding audit mis-classified as MINOR lets the
orchestrator escalate to user instead of BLOCKing — silent severity downgrade.
"""
p1, p2, p3 = finding_counts["p1"], finding_counts["p2"], finding_counts["p3"]
if p1 == 0 and p2 == 0 and p3 == 0:
return "PASS"
if p1 == 0 and p2 == 0 and 1 <= p3 <= 3:
return "MINOR"
return "MATERIAL"
# ---------------------------------------------------------------------------
# YAML serialisation (hand-rolled — standard library only)
# ---------------------------------------------------------------------------
def _yaml_str(value: str) -> str:
"""Double-quote a string value to safely embed colons, brackets, and backslashes."""
escaped = value.replace("\\", "\\\\").replace('"', '\\"')
return f'"{escaped}"'
def render_verdict_yaml(
run_id: str,
verdict_status: str,
round_num: int,
target_rounds: int,
finding_counts: dict,
findings: list[dict],
generated_at: str,
) -> str:
"""Serialize verdict fields to YAML string.
Hand-rolled to avoid PyYAML dependency and to produce a deterministic,
human-readable output matching the §3.5 example shape.
"""
lines: list[str] = []
lines.append(f"run_id: {run_id}")
lines.append(f"verdict_status: {verdict_status}")
lines.append(f"round: {round_num}")
lines.append(f"target_rounds: {target_rounds}")
lines.append("finding_counts:")
lines.append(f" p1: {finding_counts['p1']}")
lines.append(f" p2: {finding_counts['p2']}")
lines.append(f" p3: {finding_counts['p3']}")
if findings:
lines.append("findings:")
for f in findings:
lines.append(f" - id: {f['id']}")
lines.append(f" severity: {f['severity']}")
lines.append(f" dimension: {_yaml_str(f['dimension'])}")
lines.append(f" file: {_yaml_str(f['file'])}")
lines.append(f" line: {f['line']}")
lines.append(f" description: {_yaml_str(f['description'])}")
lines.append(f" suggested_fix: {_yaml_str(f['suggested_fix'])}")
else:
lines.append("findings: []")
# F-013 (§3.5): ISO 8601 timestamps are parsed as Python datetime objects by
# PyYAML's default loader (Loader=FullLoader / SafeLoader), violating the
# schema's type:string constraint. Wrap in _yaml_str() to force YAML string.
lines.append(f"generated_at: {_yaml_str(generated_at)}")
lines.append(f"generated_by: {_yaml_str(GENERATED_BY)}")
lines.append(f"generator_version: {_yaml_str(GENERATOR_VERSION)}")
return "\n".join(lines) + "\n"
# ---------------------------------------------------------------------------
# Error type
# ---------------------------------------------------------------------------
class ParseError(Exception):
"""Raised when the JSONL stream or verdict text cannot be parsed."""
# ---------------------------------------------------------------------------
# CLI entry point
# ---------------------------------------------------------------------------
def _now_rfc3339_ms() -> str:
"""Return current UTC timestamp with millisecond precision: YYYY-MM-DDTHH:MM:SS.mmmZ"""
now = datetime.now(timezone.utc)
return now.strftime("%Y-%m-%dT%H:%M:%S.") + f"{now.microsecond // 1000:03d}Z"
def _extract_run_id(jsonl_path: str) -> str:
"""Extract run_id from JSONL filename basename (strip .jsonl extension)."""
basename = os.path.basename(jsonl_path)
# Strip .jsonl extension (required)
if basename.endswith(".jsonl"):
run_id = basename[: -len(".jsonl")]
else:
run_id = basename
if not RUN_ID_RE.match(run_id):
raise ParseError(
f"jsonl filename '{basename}' does not match run_id pattern "
f"YYYY-MM-DDTHH-MM-SSZ-XXXX (got '{run_id}')"
)
return run_id
def cmd_probe(jsonl_path: str) -> int:
"""Run probe mode: validate JSONL has parseable agent_message + Section 6 summary.
Exit 0 on success, non-zero on failure. No YAML output.
Per spec §4.4 Step 4 and §4.6 case (b): probe does not require
--round / --target-rounds. Cross-field count validation is deferred to
full parse (requires --round to anchor the authoritative summary line).
F-002 (§3.6): validates stream shape (thread.started first, no error event,
turn.completed after last agent_message) before accepting PASS/MINOR/MATERIAL.
"""
try:
events = load_events(jsonl_path)
# F-002: stream-shape sanity check before content parsing
validate_stream_shape(events)
verdict_text = extract_verdict_text(events)
# Probe: accept any parseable summary line (current_round=None)
parse_section6(verdict_text, current_round=None)
return 0
except ParseError as exc:
print(str(exc), file=sys.stderr)
return 1
def cmd_jsonl(
jsonl_path: str,
round_num: int,
target_rounds: int,
) -> int:
"""Full parse mode: parse JSONL and emit verdict YAML to stdout.
Exit 0 on success; non-zero on failure (reason on stderr).
F-002 (§3.6): validates stream shape before classification to ensure a
SIGKILL'd or errored codex run cannot yield PASS/MINOR/MATERIAL status.
"""
try:
run_id = _extract_run_id(jsonl_path)
events = load_events(jsonl_path)
# F-002: stream-shape sanity check before classification
validate_stream_shape(events)
verdict_text = extract_verdict_text(events)
finding_counts, findings = parse_section6(verdict_text, current_round=round_num)
verdict_status = classify_status(finding_counts)
generated_at = _now_rfc3339_ms()
yaml_out = render_verdict_yaml(
run_id=run_id,
verdict_status=verdict_status,
round_num=round_num,
target_rounds=target_rounds,
finding_counts=finding_counts,
findings=findings,
generated_at=generated_at,
)
sys.stdout.write(yaml_out)
return 0
except ParseError as exc:
print(str(exc), file=sys.stderr)
return 1
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(
description=(
"Parse codex CLI 0.125+ --json JSONL stream into verdict YAML.\n"
"Modes: --probe (validate only) or --jsonl (emit YAML to stdout)."
)
)
mode = p.add_mutually_exclusive_group(required=True)
mode.add_argument(
"--probe",
metavar="JSONL",
dest="probe_path",
help="Validate JSONL has parseable agent_message + Section 6 summary; "
"exit 0 on success, non-zero on failure. No YAML output.",
)
mode.add_argument(
"--jsonl",
metavar="JSONL",
dest="jsonl_path",
help="Parse JSONL and emit verdict YAML to stdout. "
"Requires --round and --target-rounds.",
)
p.add_argument(
"--round",
type=int,
metavar="ROUND",
help="Audit round number (required with --jsonl).",
)
p.add_argument(
"--target-rounds",
type=int,
metavar="TARGET_ROUNDS",
help="Total target rounds (required with --jsonl).",
)
return p
def main() -> int:
parser = build_parser()
args = parser.parse_args()
if args.probe_path is not None:
return cmd_probe(args.probe_path)
# --jsonl mode: validate required companion flags
if args.round is None or args.target_rounds is None:
parser.error("--jsonl requires both --round and --target-rounds")
if args.round < 1:
parser.error("--round must be >= 1")
if args.target_rounds < 1:
parser.error("--target-rounds must be >= 1")
if args.round > args.target_rounds:
parser.error(f"--round ({args.round}) must be <= --target-rounds ({args.target_rounds})")
return cmd_jsonl(
jsonl_path=args.jsonl_path,
round_num=args.round,
target_rounds=args.target_rounds,
)
if __name__ == "__main__":
sys.exit(main())
scripts/run_codex_audit.sh
Executable
+1191
View File
File diff suppressed because it is too large Load Diff
shared/contracts/README.md
+1 -1
View File
@@ -55,7 +55,7 @@ the four-schema contract that `scripts/run_codex_audit.sh` (Phase 6.1) writes an
orchestrator agent reads at every per-agent audit gate.
- `audit/audit_jsonl.schema.json` — Layer 2 evidence: per-row schema for the codex CLI
0.125 `--json` event stream (`thread.started` / `turn.started` / `item.completed` /
0.125+ `--json` event stream (`thread.started` / `turn.started` / `item.completed` /
`turn.completed` / `error`). One JSONL line per event row.
- `audit/audit_sidecar.schema.json` — Layer 3 evidence: runner / timing / process /
stream / prompt metadata. Cross-file rules linking sidecar fields to JSONL events,
shared/contracts/audit/audit_jsonl.schema.json
+1 -1
View File
@@ -2,7 +2,7 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://github.com/Imbad0202/academic-research-skills/shared/contracts/audit/audit_jsonl.schema.json",
"title": "Codex Audit JSONL Event",
"description": "Schema for one row of the codex CLI 0.125 --json event stream produced by scripts/run_codex_audit.sh. Each JSONL line in <run_id>.jsonl validates against this schema. This is the Layer 2 anti-fake-audit check (§3.3): orchestrator validates the JSONL row-by-row before reading any verdict. Stream-level rules (first event must be thread.started; clean run ends with turn.completed; exactly one item.completed.agent_message carries the verdict text) live in scripts/check_audit_artifact_consistency.py (Phase 6.3), NOT in this schema. Codex 0.125 retired pre-0.125 fields (model, reasoning_effort, session_id, final_message, per-row usage); see §3.3 'Codex 0.125 --json event-stream shape' for the load-bearing shape contract. NOTE: item.started events appear in tool-using runs (command_execution start, etc.) and are accepted here even though spec §3.3's enumerated four-event canonical run did not list them — empirical capture from codex 0.125 confirms they precede every item.completed of types other than agent_message.",
"description": "Schema for one row of the codex CLI 0.125+ --json event stream produced by scripts/run_codex_audit.sh (verified compatible across 0.125 through 0.128). Each JSONL line in <run_id>.jsonl validates against this schema. This is the Layer 2 anti-fake-audit check (§3.3): orchestrator validates the JSONL row-by-row before reading any verdict. Stream-level rules (first event must be thread.started; clean run ends with turn.completed; exactly one item.completed.agent_message carries the verdict text) live in scripts/check_audit_artifact_consistency.py (Phase 6.3), NOT in this schema. Codex 0.125 retired pre-0.125 fields (model, reasoning_effort, session_id, final_message, per-row usage); see §3.3 'Codex 0.125+ --json event-stream shape' for the load-bearing shape contract. NOTE: item.started events appear in tool-using runs (command_execution start, etc.) and are accepted here even though spec §3.3's enumerated four-event canonical run did not list them — empirical capture from codex 0.125+ confirms they precede every item.completed of types other than agent_message.",
"type": "object",
"required": ["type"],