skills/imbad0202__academic-research-skills
1
0
Fork 0
mirror of https://github.com/Imbad0202/academic-research-skills.git synced 2026-09-14 13:51:17 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(lint): #524 defrift lock pins the #514 tools-allowlist content (#526)

Browse Source 
* feat(lint): #524 defrift lock pins the #514 tools-allowlist content

New scripts/check_tools_allowlist.py + 18 mutation tests, wired into
spec-consistency.yml and the unified pytest manifest. Invariant 1 pins
the exact `tools: Read, Write, Edit, Grep, Glob` frontmatter line on all
six #514 surfaces — the symmetric source+mirror Bash re-add that
previously passed every CI gate green (mirror-sync pins the pair, never
the value; the runtime guard keys on name, never frontmatter) now fails.
Invariant 2 reconciles frontmatter against the runtime channel: a Bucket
A agent must not advertise Bash in a tools key, fail-closed on a
missing/unparseable manifest. Placement of invariant 2 in this lint
follows the #524 proposal (considered and declined: a write-scope-lint
I6, to avoid touching the frozen I1-I5 suite for an optional guard).
Plus the #524 checklist docs: one-line allowlist mention in
docs/PERFORMANCE.md en/zh-TW § plugin agents and the announce script's
plugin-agents line, and the CHANGELOG [Unreleased] backfill covering
#514/#521 (retrospective, by @madtriceps), #523, and this change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-2 — YAML-semantic invariant 2, duplicate-key/CRLF pins, curated manifest diagnostics

Addresses the codex gpt-5.6-sol xhigh round-1 findings (P1 convergent
with the first-party security pass): invariant 2 now parses frontmatter
as YAML, so quoted values, quoted names, flow/block lists, inline
comments, and Bash(...) specifiers no longer evade the Bucket A
reconciliation (BashOutput stays unflagged — exact base-name match);
Bucket A files with unparseable YAML or unrecognized tools shapes fail
closed. Invariant 1 gains a broadened tools-key count (a quoted
"tools": duplicate cannot hide behind the pinned line) and a semantic
YAML belt (last-wins divergence fails even if a line trick slips the
count), and reads raw bytes so a symmetric LF->CRLF conversion is
drift. Valid-JSON non-object manifests get a curated fail-closed
diagnostic instead of a traceback. Suite grows 18 -> 31 with a failing
witness per new branch; CHANGELOG wording corrected accordingly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-3 — node-tree merge/alias authority, col-0 fence, byte-witness tightening

Closes the codex gpt-5.6-sol xhigh round-3 P1s, both rooted in the same
lesson (text parsing is not a sound YAML authority):

- P1-1: an indented `---` inside a `description: |` block scalar was read
  as the closing frontmatter fence, truncating the block and hiding a
  Bucket A `name` + `tools: Read, Bash` below it. The fence is now a
  column-0 `---` only (optional retained CR).
- P1-2: merge/alias detection was a text scan that a `#` inside a quoted
  flow key and a `<<` inside a flow mapping both evaded. Replaced with a
  composed-node-tree walk: a merge key carries the yaml merge tag on its
  scalar node, an alias surfaces as the same node object reached twice
  (shared identity). No text parsing.

Also: P2-1 byte-witness now requires the verbatim pinned line as the sole
raw tools line (an escaped/tagged/folded re-spelling that empties
raw_lines still fires). P2-2 (`!!binary`/`!!str` tag tricks) fixed by the
simplified `_scalar_py` — construct from the original composed node so its
resolved tag is honored (a `!!binary` list member becomes bytes, rejected
as a non-string shape; `!!str null` stays the string 'null'). Dropped the
redundant resolve()+reconstruct and the now-unused re import. Suite 39 ->
42 with witnesses for the indented fence, flow-merge-quoted-hash, escaped
duplicate key, and the block-scalar-triple-dash false-positive. CHANGELOG
corrected (duplicate-preserving, 42 tests, three review rounds).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 self-probe — strip leading BOM so a BOM-prefixed agent can't skip invariant 2

A UTF-8 BOM before the first `---` made the column-0 fence match fail, so
the file read as frontmatter-less and invariant 2 skipped it — while a real
YAML reader (and Claude Code) strips the BOM and sees the frontmatter. A
BOM-prefixed Bucket A agent could smuggle `tools: Read, Bash` through that
skip. _read_raw now strips a leading BOM so the lint and the real consumer
agree. Suite 42 -> 44 (BOM-Bash fail-closed + BOM-clean pass witnesses).
Found by first-party self-probe before the codex round-4 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-4 — fail closed on parser-dependent duplicate name/tools in invariant 2

codex gpt-5.6-sol xhigh round-4 P2: invariant 2 resolved name/tools with
last-wins (values[-1]), but duplicate YAML keys are parser-dependent. A
Bucket A file with `tools: Read, Bash` then `tools: Read, Grep` passed
(last-wins picks Grep) while a first-wins consumer would grant Bash;
symmetrically a duplicate `name` where only one resolution is Bucket A
could skip the file. Invariant 2 now rejects a duplicate `tools` on any
in-scope file and a duplicate `name` where any resolution is a Bucket A
key — fail closed, matching invariant 1's existing duplicate-tools
rejection. (round-4's other finding, the leading BOM, was already fixed in
645cfa2 — round-4 reviewed the pre-BOM-fix commit and noted the worktree
already carried the fix.) Suite 44 -> 48 with last-wins/first-wins/dup-name
witnesses. CHANGELOG updated (four rounds, 48 tests, full fail-open list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-5 — merge tag on any node type + RecursionError fails closed

codex gpt-5.6-sol xhigh round-5 found two residual issues:

- P1: _uses_merge_or_alias checked the merge tag only inside the ScalarNode
  branch, but PyYAML applies merge semantics for a merge-tagged key of ANY
  type — `? !!merge [x]` is a merge-tagged SequenceNode key that injects
  `tools: [Read, Bash]` while the scalar-only check returned false. The tag
  is now checked on every node before dispatching on type.
- P2: the recursive node walk (and yaml.compose) could raise RecursionError
  on pathologically deep valid YAML, crashing the lint with a traceback.
  Both now map to fail-closed (compose RecursionError -> None -> error;
  walk RecursionError -> True).

Suite 51 -> 53 with merge-tagged-complex-key (seq + map) and
deep-nesting-fails-closed witnesses. CHANGELOG updated (five rounds, 53
tests, full fail-open list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* test(lint): #524 round-6 self-probe witnesses — no residual fail-open or false-positive

Adds regression anchors for the round-6 adversarial probes, all confirmed
correct without code change: lowercase `bash` / `BashOutput` are distinct
non-shell tool names (not flagged); `yaml.compose` does not intern
identical scalars, so the alias-by-shared-identity detector has no
false-positive on a clean file repeating a value; a non-string manifest
`agents` key does not break reconciliation of the real Bucket A agents.
Suite 53 -> 56.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* test(lint): #524 make recursion tests Python-version-independent (CI 3.14 fix)

The deep-nesting test assumed 400 flow-sequence levels trip RecursionError,
but Python 3.14 tolerates far deeper nesting than 3.11, so compose
succeeded and the (clean, merge/alias-free) file legitimately passed —
red-lighting the test on CI's 3.14 while local 3.11 was green. A depth
that does not exceed the interpreter limit is a harmless file and SHOULD
pass. Split into: (1) does-not-crash (contract = check() returns, any
depth), (2) compose-RecursionError->None via monkeypatch, (3)
walk-RecursionError->fail-closed via an iterable that raises on iteration.
All three are deterministic and version-independent. Suite 56 -> 58.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-6 — anchor byte-witness to composed key line (block-scalar false-positive)

codex gpt-5.6-sol xhigh round-6 found one non-blocking false-positive (no
fail-open): _raw_tools_lines text-scanned every frontmatter line for
`tools:`, so a clean allowlisted file whose `description: |` block scalar
documents a `tools: ...` line produced a spurious second match and failed
CI. Replaced with _raw_tools_line anchored to the composed `tools` key
node's start_mark.line (+ _key_nodes helper) — a `tools:`-looking line
inside a block scalar is no longer mistaken for the key. Semantics for
real drift (CRLF, whitespace, escaped/tagged spelling) unchanged; the
node-tree semantic check remains the security floor. Suite 58 -> 59.
CHANGELOG updated (six rounds, 59 tests, convergence: r6 = one
false-positive fixed, zero fail-open).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-7 — rglob nested agents + byte-witness anchored to start_mark.index

codex gpt-5.6-sol xhigh round-7 found two real issues:

- P1 (fail-open): invariant 2's glob("*.md") only scanned immediate
  children of each AGENT_DIRS entry, so a nested agents/subdir/x.md with a
  Bucket A name + Bash would be missed while the runtime guard (keying on
  name regardless of path) would still fence it inconsistently. Now
  rglob("*.md"). No nested agent files exist today; this is a forward
  guard.
- P2 (byte-witness weakening / false-positive): the witness read the raw
  line via start_mark.line indexed into split("\n"), but YAML counts the
  Unicode line breaks NEL (U+0085) / LS (U+2028) / PS (U+2029) that
  split("\n") does not — the two diverge, which could false-reject an exact
  line or let a non-verbatim key line pass. Now anchored to
  start_mark.index (byte offset), sliced to the surrounding physical-\n
  line, so the two agree by construction. The node-tree semantic check
  remains the security floor regardless.

Suite 59 -> 61 with nested-Bucket-A-Bash and Unicode-line-break witnesses.
CHANGELOG updated (seven rounds, 61 tests, full fail-open + false-positive
list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-8 — fail closed on directory symlinks + bare-CR frontmatter

codex gpt-5.6-sol xhigh round-8 found two more fail-opens:

- P1: rglob does not descend into directory symlinks, so a tracked
  agents/nested -> ../payload could hide a Bucket A .md declaring Bash.
  Invariant 2 now fails closed on any directory symlink under an agent dir
  (these hand-authored trees have no reason for one). Real repo has none.
- P1: _frontmatter split only on \n, so bare-\r (old-Mac) frontmatter —
  which YAML parses fine — was read as absent and skipped, hiding a Bucket
  A tools: Bash. Fences are now found with splitlines() (recognizes \n,
  \r\n, bare \r, and Unicode breaks); the block is still the byte-faithful
  substring so compose sees true bytes and the byte-witness (start_mark.index)
  stays exact — a bare-CR/CRLF file still fires the witness as drift.

Suite 61 -> 63 with dir-symlink and bare-CR witnesses. CHANGELOG updated
(eight rounds, 63 tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-9 — fold invisible/homoglyph tool tokens so padded Bash can't slip

codex gpt-5.6-sol xhigh round-9 (via its own probe) surfaced a real
fail-open in invariant 2: the Bash check is an exact-string membership
test (`"Bash" in declared`), and str.strip() does NOT remove Unicode
format characters (category Cf). So a fenced Bucket A agent declaring

    tools: Read, Bash      (BOM U+FEFF padding)
    tools: Read, ​Bash​      (zero-width space U+200B)
    tools: Read, Ｂａｓｈ        (fullwidth homoglyph)

kept the token distinct from `Bash`, passed the lint GREEN, yet reads as
Bash to the eye and to any NFKC-normalizing consumer — exactly the
symmetric-re-add bypass the lock exists to stop.

Fix: new _fold_token strips Cf format chars then NFKC-normalizes each base
tool token before comparison. All six real tool names + the Bash(git:*)
permission form are pure ASCII and pass through byte-identical, so no
legitimate value is altered. Folding lives in _normalized_tools, so it
protects BOTH invariants; invariant 1's additive byte-witness is confirmed
to STILL fire on an invisible-char canonical value (folding the semantic
check did not open a hole there).

Suite 63 -> 68: BOM / ZWSP / ZWNJ / fullwidth padded-Bash witnesses + a
byte-witness-still-fires regression for the invariant-1 companion. CHANGELOG
updated (nine rounds, 68 tests, invisible-char fail-open listed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-10 — fold before the permission-specifier split so fullwidth-paren Bash can't slip

codex gpt-5.6-sol xhigh round-10 found an ordering corollary of the r9
fold: _fold ran AFTER the ASCII "(" split, so a fullwidth-paren specifier

    tools: Read, Bash（git:*）      (U+FF08 / U+FF09)

was never split (ASCII "(" misses the fullwidth "（"), leaving the whole
"Bash（git:*）" as the base token; NFKC then folded it to "Bash(git:*)"
(one token) which != "Bash", so the membership test missed it and the
lint passed green — while an NFKC-normalizing runtime consumer would read
a valid Bash(git:*) grant.

Fix: fold each item FIRST (renamed _fold_token -> _fold, now content-only,
no internal strip), THEN split on ASCII "(" and strip. Fullwidth parens
fold to ASCII before the split, so Bash（git:*） and Ｂａｓｈ（git:*） both
reduce to "Bash". ASCII Bash(git:*), the canonical five, and BashOutput
are all unchanged.

Suite 68 -> 70: fullwidth-paren specifier + fullwidth-name-and-paren
witnesses. CHANGELOG updated (ten rounds, 70 tests, ordering corollary
listed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* fix(lint): #524 round-11 — fold the whole tools value before ANY split so fullwidth-comma Bash can't slip

codex gpt-5.6-sol xhigh round-11 found the comma-side twin of the r10
paren corollary: _fold still ran per-item AFTER value.split(","), so a
fullwidth comma

    tools: Read，Bash          (U+FF0C)

kept "Read，Bash" as one item through the ASCII comma split; folding the
item then yielded "Read,Bash" (one token) != "Bash", so the membership
test missed it — while an NFKC-normalizing consumer would split on the
folded comma and grant Bash.

Root cause is the r10 class generalized: splitting on ASCII separators
before folding. Fix: fold ONCE up front — the whole string in the comma
form (so U+FF0C and U+FE50 become "," before split), and each list member
whole in the list form (a fullwidth comma inside a YAML list element is
not a separator, so it correctly stays one token). Then split on "," and
"(". ASCII values, the canonical five, and BashOutput are all unchanged.

Suite 70 -> 73: fullwidth-comma + small-comma + combined
comma/name/paren witnesses. CHANGELOG updated (eleven rounds, 73 tests,
both separator corollaries listed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

* test(lint): #524 round-12 — pin the separator-fold convergence boundary as a non-bug

Round-12 review: codex's automated pass tripped its own cybersecurity
content filter mid-run (it went spelunking in the Claude Code binary) and
returned no verdict, so I resolved convergence by first-party analysis
instead.

Exhaustively scanned the residual surface the r9-r11 fold left open:
(A) every NFKC-stable alternate separator (ideographic/Arabic commas
U+3001/U+060C, division/fraction slashes, semicolons, middle dots) — none
can isolate a bare "Bash" token, because they do NOT fold to the ASCII
","/"(" the split (or any NFKC-normalizing consumer) honors, so
"Read、Bash" stays ONE non-Bash token for everyone (proven: it is a single
YAML scalar, and this lint splits only on ASCII ","); (B) zero codepoints
NFKC-decompose INTO "ash" (no false-positive/hiding risk); (C) the
per-letter compat variants are already folded whole; (D) zero non-Cf
codepoints fold to empty (no token-merging attack). The fullwidth
comma/paren cases fail precisely because NFKC folds THEM into ASCII
separators; the stable ones cannot, so not flagging them is correct.

Added test_nfkc_stable_alt_separator_is_not_a_bash_grant documenting this
boundary as a NON-bug (suite 73 -> 74) so a future maintainer does not
"fix" it by over-broadening the separator set into false positives.
CHANGELOG updated (twelve rounds, 74 tests, convergence boundary recorded).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPGvAbphbuZ4mhx2Ketj7s

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Edward Cheng-I Wu
2026-07-13 16:27:44 +08:00
committed by GitHub
parent 3977b9fb80
commit d22b1634d6
8 changed files with 1401 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/spec-consistency.yml
+7
View File
@@ -127,6 +127,13 @@ jobs:
# unified manifest.
run: python3 scripts/check_agents_mirror_sync.py
- name: Check tools-allowlist content lock (#524)
# Pins the #514 allowlist VALUE (mirror-sync pins only the pair) +
# the Bucket-A-must-not-advertise-Bash reconciliation. The pytest
# companion `test_check_tools_allowlist.py` runs via the unified
# manifest.
run: python3 scripts/check_tools_allowlist.py
- name: Check revision-patch discipline (#390 Slice B)
# The pytest companion `test_check_390_revision_patch_discipline.py`
# runs via the unified manifest.
CHANGELOG.md
+12
View File
File diff suppressed because one or more lines are too long
docs/PERFORMANCE.md
+2
View File
@@ -42,6 +42,8 @@ When ARS is installed as a Claude Code plugin (`/plugin install academic-researc
- A Sonnet session gets Sonnet agents, matching the cost/latency profile of the parent run.
- The agents never silently fall back to Haiku — `inherit` resolves through the parent session's model, which is itself gated by the project policy of "no Haiku for ARS runs."
Since #514 (shipped in #521), each of the three also carries a pinned tools allowlist in the same frontmatter — `tools: Read, Write, Edit, Grep, Glob`, no shell and no network fetch — so dispatch-time capability is least-privilege; the exact value is CI-locked by `scripts/check_tools_allowlist.py` (#524).
This means **plugin-agent token costs track the per-mode estimates above unchanged** (with `ARS_MODEL_TIERING` unset); there is no separate plugin agent surcharge or discount, because dispatched agents inherit the same model the parent run already pays for. Under `ARS_MODEL_TIERING=economy`, plugin-exposed execution-type agents (e.g. `report_compiler_agent`) follow the tiering rule instead — one tier below the session model, floor Opus-class (see `shared/model_tiering.md`). If you change the main session model mid-pipeline (e.g., downshift to Sonnet for a long revision pass), the next agent dispatch picks up the new floor automatically.
Other ARS agents (`bibliography_agent`, `literature_strategist_agent`, etc.) are not plugin-exposed in v3.7.0; they remain in-skill prompt templates that the main session executes inline, with no separate model routing layer **by default**. The opt-in `ARS_MODEL_TIERING` switch (#517) adds a dispatch-time routing rule on top: when a tiering direction applies to a role, the session dispatches it as a subagent pinned to the target tier (inline roles included — dispatch-as-subagent is the mechanism); with the flag unset, this paragraph describes behavior unchanged. See `shared/model_tiering.md`. Wider plugin-agent coverage is deferred to a future release.
docs/PERFORMANCE.zh-TW.md
+2
View File
@@ -42,6 +42,8 @@
- Sonnet session 取得 Sonnet agent，跟主 session cost / latency 對齊。
- Agent 永遠不會默默掉到 Haiku — `inherit` 走的是主 session 模型，主 session 本身又被「ARS 全程不用 Haiku」政策守住。
自 #514（於 #521 出貨）起，這三個 agent 的 frontmatter 同時帶固定的 tools 白名單——`tools: Read, Write, Edit, Grep, Glob`，無 shell、無網路抓取——派工當下即是最小權限；白名單內容由 `scripts/check_tools_allowlist.py`（#524）在 CI 鎖定。
意涵：**plugin agent 的 token 成本完全跟著上表各模式估算走，沒有額外加減**（`ARS_MODEL_TIERING` 未設定時）。dispatched agent 跟主 session 同一個模型，主 session 已經付的成本沒有再多一層 plugin agent 收費。設定 `ARS_MODEL_TIERING=economy` 時，plugin 暴露的 execution 型 agent（如 `report_compiler_agent`）改走分層規則——比 session model 低一階、樓地板 Opus 級（見 `shared/model_tiering.md`）。如果 pipeline 中途換模型（例如 revision pass 改用 Sonnet 省成本），下一輪 agent 派工自動跟上。
其他 ARS agent（`bibliography_agent`、`literature_strategist_agent` 等）在 v3.7.0 不暴露為 plugin agent；它們仍是 in-skill prompt template，由主 session 內聯執行，**預設**沒有獨立的模型路由層。Opt-in 的 `ARS_MODEL_TIERING`（#517）在其上加了一層 dispatch 時的路由規則：當分層方向適用於某角色時，session 會把該角色以子代理形式派發、鎖定目標層級（內聯角色也一樣——「派發為子代理」正是其機制）；flag 未設定時，本段描述的行為完全不變。見 `shared/model_tiering.md`。更廣的 plugin agent 覆蓋留到後續版本。
scripts/_ci_pytest_manifest.toml
+4
View File
@@ -271,3 +271,7 @@ path = "scripts/test_check_setup_cross_model_parity.py"
[[pytest]]
id = "517-model-tiering-classification"
path = "scripts/test_check_model_tiering.py"
[[pytest]]
id = "524-tools-allowlist-lock"
path = "scripts/test_check_tools_allowlist.py"
scripts/announce-ars-loaded.sh
+1 -1
View File
@@ -77,7 +77,7 @@ Slash commands (16) — light modes pin sonnet in frontmatter; the three heavy m
/ars-unmark-read sonnet Rescind a prior human-read mark for one or more citation keys
/ars-cache-invalidate sonnet Drop cached verification rows for one or more citation keys
Plugin agents (3, v3.6.7-hardened, model: inherit) — dispatched by ARS pipeline:
Plugin agents (3, v3.6.7-hardened, model: inherit, tools allowlist: Read/Write/Edit/Grep/Glob per #514) — dispatched by ARS pipeline:
synthesis_agent Cross-source integration, contradiction resolution, gap analysis
research_architect_agent Methodology blueprint (paradigm, method, data strategy)
report_compiler_agent APA 7.0 report drafting (Phase 4 + Phase 6)
scripts/check_tools_allowlist.py
Executable
+538
View File
@@ -0,0 +1,538 @@
#!/usr/bin/env python3
"""Lint: pin the #514 tools-allowlist CONTENT on the three plugin agents (#524).
#521 shipped `tools: Read, Write, Edit, Grep, Glob` on the three top-level
plugin agents (deep-research sources + agents/ mirrors), but no lint read a
frontmatter `tools` key: check_agents_mirror_sync.py pins mirror==source
byte-equality (the PAIR, never the VALUE), and the runtime write-scope guard
(scripts/ars_write_scope_guard.py) keys on agent NAME, never frontmatter. A
future PR editing a source+mirror pair symmetrically back to `..., Bash` (or
dropping `Grep`, or typoing a tool name) would pass every CI gate green —
exactly the drift class the repo's defrift locks exist to catch (cf. the
v3.15 locks). This lint pins the VALUE.
Design: YAML is the authority, not a line scan. Claude Code defines these
files as YAML frontmatter, and a raw-line/regex scan can never see every
YAML-legal spelling of a key or value (quoted, tagged, aliased, escaped,
`\\u0073`-escaped, tab-indented, flow/block list, `Bash(...)` specifier). So
every semantic decision is read from a DUPLICATE-PRESERVING YAML node tree
(`yaml.compose`, which — unlike `safe_load` — keeps a shadowed duplicate key
visible AND resolves an alias into shared node identity). Any frontmatter
that will not compose to a mapping, OR uses a merge key (`<<`) / alias
(constructs that inject or share keys the literal-key scan cannot see), is a
fail-closed ERROR, never a skip. The frontmatter fence is a COLUMN-0 `---`
only, so an indented `---` inside a block scalar cannot truncate the block
and hide keys below it. The exact PINNED_TOOLS_LINE raw-line check is kept
ON TOP as an additive, stricter witness (it also pins byte-level form:
CR-sensitive, so a symmetric LF→CRLF conversion is drift; and it fires when
the verbatim pinned line is absent), but it can only ADD findings, never
subtract them — the semantic node-tree check stands alone as the security
floor.
Invariants:
1. Every file in ALLOWLISTED_FILES exists, its frontmatter composes to a
YAML mapping with EXACTLY ONE `tools` key (counted from the
duplicate-preserving node tree, so a shadowed `"tools":` / `"tool\\u0073":`
duplicate is caught), and that key's value normalizes to exactly the
canonical five tools. On top, the single raw `tools:` frontmatter line
must be byte-equal to PINNED_TOOLS_LINE. Changing the allowlist is a
deliberate security-surface change: edit the agent files AND this
lint's PINNED_TOOLS_LINE in the same commit (standard lock semantics).
2. Frontmatter/guard reconciliation: any agent file under AGENT_DIRS whose
frontmatter `name` is a Bucket A key in
scripts/ars_phase_scope_manifest.json must NOT declare Bash in a
`tools:` key — in ANY YAML-legal form. `Bash` is matched as an exact
base tool name (`BashOutput` is a different tool and is not flagged;
`Bash(git:*)` normalizes to `Bash` and IS). The runtime guard denies
Bucket A agents ALL Bash (zero fail-open); a frontmatter advertising
Bash would silently widen capability in hook-less installs while
contradicting the guard in hook-active ones. Agents with no `tools:`
key inherit and are untouched (the runtime guard still fences them).
Fail-closed: an agent file whose frontmatter will not compose to a
mapping is treated as a POSSIBLE Bucket A member and errors (we cannot
read its `name` to clear it), and a Bucket A agent whose `tools` value
has an unrecognized shape (non-string list member, mapping) errors.
The manifest is load-bearing for invariant 2, so a missing, unparseable, or
non-mapping manifest FAILS the lint (fail-closed) rather than skipping.
"""
from __future__ import annotations
import json
import sys
import unicodedata
from pathlib import Path
import yaml
REPO_ROOT = Path(__file__).resolve().parent.parent
# The exact frontmatter line shipped by #521 (frozen #514 spec). Single
# source of truth for the VALUE — a symmetric source+mirror edit cannot
# change it without touching this lint in the same commit.
PINNED_TOOLS_LINE = "tools: Read, Write, Edit, Grep, Glob"
CANONICAL_TOOLS = ("Read", "Write", "Edit", "Grep", "Glob")
# The six #514 surfaces: three canonical sources + three agents/ mirrors
# (mirror==source byte-equality is check_agents_mirror_sync.py's job; the
# mirrors are still listed here so THIS lint stays correct even if that one
# is skipped or edited — deliberately re-derived, not imported, per the
# repo's independent-second-witness lint convention).
ALLOWLISTED_FILES = (
"deep-research/agents/report_compiler_agent.md",
"deep-research/agents/research_architect_agent.md",
"deep-research/agents/synthesis_agent.md",
"agents/report_compiler_agent.md",
"agents/research_architect_agent.md",
"agents/synthesis_agent.md",
)
# Every directory that holds agent prompt files (invariant 2 scan surface).
AGENT_DIRS = (
"deep-research/agents",
"academic-paper/agents",
"academic-paper-reviewer/agents",
"academic-pipeline/agents",
"shared/agents",
"agents",
)
MANIFEST = "scripts/ars_phase_scope_manifest.json"
def _read_raw(path: Path) -> str:
"""Read WITHOUT universal-newline translation, so a CRLF file keeps its
`\\r` bytes and cannot satisfy an exact LF line pin. A leading UTF-8 BOM
is stripped: a YAML reader (and Claude Code) skips it, so leaving it on
would make `\\ufeff---` fail the column-0 fence match and the whole file
read as frontmatter-less — a fail-open that lets a BOM-prefixed Bucket A
agent smuggle Bash past invariant 2's skip-when-no-frontmatter branch."""
return path.read_bytes().decode("utf-8").lstrip("")
def _frontmatter(text: str) -> str | None:
"""The raw YAML frontmatter block (the byte-faithful substring between the
two `---` fences), or None when the file has no frontmatter.
Line breaks are found with `str.splitlines`, which recognizes EVERY YAML
line break — `\\n`, `\\r\\n`, and a bare `\\r` (old-Mac) plus the Unicode
breaks NEL/LS/PS — so a bare-CR-delimited file is not silently read as
frontmatter-less and skipped (a real fail-open, #524 r8). A fence is
exactly `---` at column 0: an INDENTED `---` is block-scalar content, not
a fence (matching it would truncate the block and hide keys). The block
returned is the raw substring (original break bytes intact) so `compose`
sees the true bytes and the byte-witness — anchored to `start_mark.index`
— stays byte-faithful; a bare `\\r`/CRLF file therefore still fires the
byte-witness as drift."""
lines = text.splitlines(keepends=True)
if not lines or lines[0].splitlines()[0] != "---":
return None
# Byte offset just past the opening fence line (start of the block body).
body_start = len(lines[0])
offset = body_start
for raw in lines[1:]:
content = raw.splitlines()[0] if raw.splitlines() else raw
if content == "---": # column-0 closing fence
return text[body_start:offset]
offset += len(raw)
return None
def _mapping_node(block: str) -> yaml.MappingNode | None:
"""The frontmatter composed to a DUPLICATE-PRESERVING mapping node, or
None when it is not a mapping / will not compose / is too deeply nested
to compose safely. Unlike `safe_load`, `compose` keeps a shadowed
duplicate key visible in the node tree. A RecursionError on pathological
nesting maps to None so the caller fails closed rather than crashing."""
try:
node = yaml.compose(block, Loader=yaml.SafeLoader)
except (yaml.YAMLError, RecursionError):
return None
return node if isinstance(node, yaml.MappingNode) else None
_UNRESOLVED = object()
def _scalar_py(scalar: yaml.ScalarNode) -> object:
"""The Python value a scalar node resolves to, so `"tool\\u0073"`,
`'tools'`, and `tools` all compare equal. `compose` already stamps each
node with its resolved tag, so the SafeLoader constructs it exactly as
`safe_load` would (quotes, `\\u` escapes, explicit `!!` tags, block/flow
styles all honored). Any tag the safe constructor cannot build (a `<<`
merge scalar, an unknown `!Tag`) yields _UNRESOLVED so callers fail
closed rather than crash."""
try:
return yaml.SafeLoader("").construct_object(scalar)
except yaml.YAMLError:
return _UNRESOLVED
_MERGE_TAG = "tag:yaml.org,2002:merge"
def _uses_merge_or_alias(node: yaml.Node) -> bool:
"""True if the composed frontmatter uses a YAML merge key (`<<`) or an
alias (`*name`). safe_load resolves both — a `<<: *base` injects a
`tools`/`name` the literal-key scan never sees, and an alias shares a
value node — so their presence makes the duplicate-preserving key scan
unsound. Detected from the COMPOSED NODE TREE, not text (text scanning is
not a sound YAML authority — the reviews' recurring lesson): the `merge`
tag can land on a node of ANY type (`<<:` is a scalar key, but `? !!merge
[x]` is a merge-tagged sequence/mapping key), so we check the tag on
every node before dispatching on its type; an alias surfaces as the SAME
node object reachable by two paths (compose resolves `*a` to shared
identity). These hand-authored files never need either, so we fail closed
rather than reimplement merge resolution. A RecursionError on pathological
nesting also fails closed (True) — we could not prove the tree clean."""
seen: set[int] = set()
def walk(nd: yaml.Node) -> bool:
if nd is None:
return False
if nd.tag == _MERGE_TAG: # merge tag on ANY node type (key or value)
return True
if id(nd) in seen:
return True # reached twice = an alias shares this node
seen.add(id(nd))
if isinstance(nd, yaml.SequenceNode):
return any(walk(i) for i in nd.value)
if isinstance(nd, yaml.MappingNode):
return any(walk(k) or walk(v) for k, v in nd.value)
return False
try:
return walk(node)
except RecursionError:
return True
def _key_values(node: yaml.MappingNode, key: str) -> list[yaml.Node]:
"""Every value node whose key resolves to `key`, duplicates included
(the node tree preserves a shadowed key that safe_load would collapse)."""
return [v for k, v in node.value
if isinstance(k, yaml.ScalarNode) and _scalar_py(k) == key]
def _key_nodes(node: yaml.MappingNode, key: str) -> list[yaml.ScalarNode]:
"""Every KEY node that resolves to `key` (for line marks), duplicates
included."""
return [k for k, v in node.value
if isinstance(k, yaml.ScalarNode) and _scalar_py(k) == key]
def _node_to_py(value_node: yaml.Node) -> object:
"""A value node converted to its Python value, or the sentinel
_UNRESOLVED when it is not a plain scalar/sequence of scalars (a nested
list, a mapping member, or a mapping value — all unrecognized shapes)."""
if isinstance(value_node, yaml.ScalarNode):
return _scalar_py(value_node)
if isinstance(value_node, yaml.SequenceNode):
if not all(isinstance(i, yaml.ScalarNode) for i in value_node.value):
return _UNRESOLVED # nested list / mapping member
return [_scalar_py(i) for i in value_node.value]
return _UNRESOLVED
def _fold(text: str) -> str:
"""Text folded so an invisible/compatibility re-spelling collapses onto its
plain ASCII form. Removes Unicode format characters (category `Cf` —
BOM/`\\ufeff`, zero-width space/`\\u200b`, zero-width non-joiner/`\\u200c`,
etc.), which `str.strip()` does NOT remove, so `\\ufeffBash\\ufeff` would
otherwise survive as a token distinct from `Bash` and slip the `"Bash" in
declared` membership test (#524 r9) — a fenced Bucket A agent could declare
a zero-width-padded Bash and pass. Then NFKC-normalizes to fold
compatibility homoglyphs (fullwidth, etc.) onto ASCII. All six real tool
names + the `Bash(git:*)` permission form are pure ASCII and fold to
themselves, so no legitimate value is altered."""
stripped = "".join(c for c in text if unicodedata.category(c) != "Cf")
return unicodedata.normalize("NFKC", stripped)
def _normalized_tools(value: object) -> list[str] | None:
"""A `tools` value normalized to base tool names, or None when the shape
is unrecognized. Accepts the comma-string form and a list of strings; a
`Bash(git:*)`-style permission specifier normalizes to `Bash`. Folding
(`_fold`) happens BEFORE any split, on the whole string (or on each list
member whole), so every compatibility separator becomes its ASCII form
first: a fullwidth comma `，` (U+FF0C) in the string form — which
`split(",")` would miss, leaving `Read，Bash` as one token that an
NFKC-normalizing consumer would re-split to grant Bash (#524 r11) — and a
fullwidth-paren specifier `Bash（git:*）` (U+FF08/U+FF09) that the ASCII `(`
split would miss (#524 r10) both reduce correctly. Splitting before folding
reintroduces either hole. `value` comes from `_node_to_py`, which already
collapses a non-scalar list member to `_UNRESOLVED`, so a list reaching
here is all-strings; any non-str/list value (including `_UNRESOLVED`) is
unrecognized."""
if isinstance(value, str):
items: list[str] = _fold(value).split(",")
elif isinstance(value, list) and all(isinstance(i, str) for i in value):
# A YAML list is already tokenized; fold each member whole (a
# fullwidth comma inside a member is not a YAML separator, so it stays
# one token — correctly, since no list consumer re-splits an element).
items = [_fold(i) for i in value]
else:
return None
out = []
for item in items:
base = item.split("(", 1)[0].strip()
if base:
out.append(base)
return out
def _raw_tools_line(block: str, key_node: yaml.ScalarNode) -> str | None:
"""The raw frontmatter line the composed `tools` key SITS ON, for the
byte-exact PINNED_TOOLS_LINE witness. Anchored to the node's
`start_mark.index` — the character offset into `block` — sliced to the
surrounding physical-`\\n` line, NOT `start_mark.line` (which counts
YAML's Unicode line breaks NEL/LS/PS that `split("\\n")` does not, so the
two indexings can diverge and select the wrong line — #524 r7). Using the
byte offset makes the two agree by construction. Anchoring (not a text
scan) also means a `tools:`-looking line inside a block scalar — e.g. a
`description: |` documenting the tools — is never mistaken for the key
line (round-6 false-positive). An ADDITIVE (still CI-gating) layer on top
of the node-tree check: it only adds findings a capability-equivalent
re-spelling would slip past (CRLF, trailing/interior whitespace, exact
spelling), never clears the semantic check. Returns None when the offset
is out of range."""
idx = key_node.start_mark.index
if not (0 <= idx <= len(block)):
return None
start = block.rfind("\n", 0, idx) + 1 # char after the prev newline
end = block.find("\n", idx) # next newline, or end
return block[start:] if end == -1 else block[start:end]
def _tools_value(node: yaml.MappingNode) -> object:
"""The effective (last-wins) `tools` Python value from the node tree, or
_UNRESOLVED when the key is absent or its value has an unreadable shape."""
values = _key_values(node, "tools")
if not values:
return _UNRESOLVED
return _node_to_py(values[-1]) # YAML last-wins
def _name_value(node: yaml.MappingNode) -> str:
"""The effective `name` from the node tree ('' when absent/non-scalar)."""
values = _key_values(node, "name")
if not values:
return ""
py = _node_to_py(values[-1])
return str(py).strip() if isinstance(py, str) else ""
def _bucket_a_names(root: Path) -> tuple[set[str] | None, str | None]:
"""Bucket A agent names from the manifest, or (None, error)."""
mp = root / MANIFEST
try:
data = json.loads(mp.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
return None, (
f"{MANIFEST}: unreadable or unparseable ({exc}) — invariant 2 "
"(frontmatter/guard reconciliation) cannot run; failing closed."
)
agents = data.get("agents") if isinstance(data, dict) else None
if not isinstance(agents, dict):
return None, (
f"{MANIFEST}: no `agents` mapping — invariant 2 cannot run; "
"failing closed."
)
return set(agents), None
def check(root: Path) -> list[str]:
errors: list[str] = []
# --- invariant 1: exactly-one canonical `tools` on the six files ---------
for rel in ALLOWLISTED_FILES:
path = root / rel
if not path.is_file():
errors.append(
f"{rel}: allowlisted agent file is missing — the #514 "
"surface changed; update ALLOWLISTED_FILES in "
"check_tools_allowlist.py deliberately or restore the file."
)
continue
block = _frontmatter(_read_raw(path))
if block is None:
errors.append(f"{rel}: no YAML frontmatter block found.")
continue
node = _mapping_node(block)
if node is None:
errors.append(
f"{rel}: frontmatter does not compose to a YAML mapping — "
"the tools allowlist cannot be verified; failing closed."
)
continue
if _uses_merge_or_alias(node):
errors.append(
f"{rel}: frontmatter uses a YAML merge key / alias — these "
"resolve to keys the pin cannot see (a `<<` can inject a "
"`tools` value); not supported in agent frontmatter, "
"failing closed."
)
continue
tools_nodes = _key_values(node, "tools")
if not tools_nodes:
errors.append(
f"{rel}: frontmatter has no `tools:` key — the #514 "
"allowlist was dropped (a silent capability widening: the "
"agent would inherit ALL tools). Restore "
f"`{PINNED_TOOLS_LINE}`."
)
elif len(tools_nodes) > 1:
errors.append(
f"{rel}: {len(tools_nodes)} `tools` keys in frontmatter "
"(duplicate-preserving parse — quoted / escaped variants "
"included) — exactly one expected; a duplicate key overrides "
"the pinned value under YAML last-wins resolution."
)
else:
normalized = _normalized_tools(_node_to_py(tools_nodes[0]))
if normalized != list(CANONICAL_TOOLS):
errors.append(
f"{rel}: the `tools` value diverges from the canonical "
f"{', '.join(CANONICAL_TOOLS)} — the effective allowlist "
"is not what #514 froze."
)
# Byte-exact witness ON TOP (also pins CRLF / spelling). Additive
# layer: it can only add findings, never clear the semantic check.
# Anchored to the composed `tools` key's own line (not a text scan),
# so a `tools:`-looking line inside a block scalar is not mistaken for
# it. Only meaningful for the exactly-one-key case; the missing /
# duplicate cases already fired via the semantic branch above. A
# non-verbatim line (escaped/tagged/folded spelling, or CRLF /
# whitespace drift) fires — the semantic check independently catches
# the value-changing subset.
key_nodes = _key_nodes(node, "tools")
if len(key_nodes) == 1:
raw_line = _raw_tools_line(block, key_nodes[0])
if raw_line != PINNED_TOOLS_LINE:
found = raw_line if raw_line is not None else \
"(tools key not on its own line)"
errors.append(
f"{rel}: the `tools` line is not byte-equal to the frozen "
f"#514 form.\n expected: {PINNED_TOOLS_LINE}\n "
f"found: {found!r}\n Changing the allowlist is a "
"deliberate security-surface change: update "
"PINNED_TOOLS_LINE in check_tools_allowlist.py in the "
"same commit."
)
# --- invariant 2: no Bucket A agent declares Bash -------------------------
bucket_a, manifest_err = _bucket_a_names(root)
if manifest_err:
errors.append(manifest_err)
return errors
for rel_dir in AGENT_DIRS:
d = root / rel_dir
if not d.is_dir():
continue
# rglob does NOT descend into directory symlinks, so a tracked
# `agents/nested -> ../payload` could hide a Bucket A `.md` declaring
# Bash (#524 r8). Fail closed on any directory symlink under an agent
# dir — these hand-authored trees have no reason for one, and
# reconciliation cannot see through it.
for sub in sorted(d.rglob("*")):
if sub.is_symlink() and sub.is_dir():
errors.append(
f"{sub.relative_to(root).as_posix()}: directory symlink "
"under an agent dir — rglob does not descend into it, so a "
"Bucket A agent declaring Bash could hide behind it; "
"failing closed. Replace with real files."
)
# rglob, not glob: a nested `agents/subdir/x.md` could carry a Bucket
# A `name` + Bash and the runtime guard keys on name regardless of
# path, so the reconciliation must reach nested files too (#524 r7).
for path in sorted(d.rglob("*.md")):
rel = path.relative_to(root).as_posix()
block = _frontmatter(_read_raw(path))
if block is None:
continue
node = _mapping_node(block)
if node is None:
# Cannot read `name` to clear it — treat as a possible
# Bucket A member and fail closed.
errors.append(
f"{rel}: agent frontmatter does not compose to a YAML "
"mapping — cannot confirm it is not a Bucket A agent "
"advertising Bash; failing closed."
)
continue
if _uses_merge_or_alias(node):
# A `<<`/alias could inject `tools` or rewrite `name`
# invisibly to the literal-key scan — fail closed rather than
# clear the file on a name it may not really carry.
errors.append(
f"{rel}: agent frontmatter uses a YAML merge key / alias "
"— cannot soundly confirm it is not a Bucket A agent "
"advertising Bash; failing closed."
)
continue
# Duplicate `name`/`tools` handling is parser-dependent (last-wins
# here, but another consumer may take first-wins). If ANY resolved
# `name` is a Bucket A key, the file is in scope — and if it also
# carries a duplicate `tools`, one resolution could hide Bash
# behind the other. Fail closed rather than pick a winner, exactly
# as invariant 1 rejects a duplicate `tools`.
name_nodes = _key_values(node, "name")
resolved_names = {_node_to_py(n) for n in name_nodes}
if not resolved_names & bucket_a:
continue
if len(name_nodes) > 1:
errors.append(
f"{rel}: {len(name_nodes)} `name` keys in frontmatter, one "
"resolving to a Bucket A agent — duplicate-key resolution "
"is parser-dependent; failing closed."
)
continue
if len(_key_values(node, "tools")) > 1:
errors.append(
f"{rel}: Bucket A agent has {len(_key_values(node, 'tools'))} "
"`tools` keys — duplicate-key resolution is "
"parser-dependent and one could hide Bash behind another; "
"failing closed."
)
continue
tools_value = _tools_value(node)
if tools_value is _UNRESOLVED:
if _key_values(node, "tools"):
errors.append(
f"{rel}: Bucket A agent has a `tools` value of "
"unrecognized shape — cannot verify it excludes "
"Bash; failing closed."
)
continue
declared = _normalized_tools(tools_value)
if declared is None:
errors.append(
f"{rel}: Bucket A agent has a `tools` value of "
"unrecognized shape — cannot verify it excludes Bash; "
"failing closed."
)
elif "Bash" in declared:
errors.append(
f"{rel}: frontmatter declares Bash but this is a Bucket "
f"A agent in {MANIFEST} — the runtime guard denies "
"Bucket A agents ALL Bash (zero fail-open), so this "
"grant is either dead (hook-active) or a silent widening "
"(hook-less). Remove Bash from the tools list."
)
return errors
def main() -> int:
errors = check(REPO_ROOT)
if errors:
print("tools allowlist check failed (#524):")
for err in errors:
print(f"- {err}")
return 1
print("tools allowlist check passed.")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/test_check_tools_allowlist.py
+835
View File
@@ -0,0 +1,835 @@
#!/usr/bin/env python3
"""Tests for check_tools_allowlist.py (#524).
Mutation discipline: every invariant branch has a passing case (green fixture
tree + the real repo tree) and a failing case proving the check fires when
the guarded property is broken. Two load-bearing families:
* The SYMMETRIC source+mirror edit re-adding Bash — it passes
check_agents_mirror_sync.py (byte-equal pair) and the name-keyed runtime
guard lint, so before #524 it sailed through CI green (the drift the
issue documents).
* YAML-form bypasses of the semantic checks — quoted/flow/block/escaped
spellings that a raw line scan cannot see. These were found by the codex
xhigh review; the node-tree parse closes them and each has a witness
here.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
from check_tools_allowlist import (
ALLOWLISTED_FILES,
CANONICAL_TOOLS,
MANIFEST,
PINNED_TOOLS_LINE,
REPO_ROOT,
check,
)
def make_tree(tmp_path: Path) -> Path:
"""A green fixture tree: six allowlisted files carrying the pinned line,
a minimal Bucket A manifest, a fenced no-tools agent, and an unfenced
Bash-holding agent."""
for rel in ALLOWLISTED_FILES:
p = tmp_path / rel
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(
f"---\nname: {p.stem}\ndescription: \"x\"\nmodel: inherit\n"
f"{PINNED_TOOLS_LINE}\n---\n\nbody\n",
encoding="utf-8",
)
manifest = tmp_path / MANIFEST
manifest.parent.mkdir(parents=True, exist_ok=True)
manifest.write_text(json.dumps({"agents": {
"report_compiler_agent": {},
"research_architect_agent": {},
"synthesis_agent": {},
"eic_agent": {},
}}), encoding="utf-8")
# A Bucket A agent with NO tools key (inherit) — must pass untouched.
eic = tmp_path / "academic-paper-reviewer/agents/eic_agent.md"
eic.parent.mkdir(parents=True, exist_ok=True)
eic.write_text("---\nname: eic_agent\n---\n\nbody\n", encoding="utf-8")
# A NON-Bucket-A agent advertising Bash — allowed (invariant 2 is
# scoped to fenced agents; the orchestrator legitimately holds shell).
orch = tmp_path / "academic-pipeline/agents/pipeline_orchestrator_agent.md"
orch.parent.mkdir(parents=True, exist_ok=True)
orch.write_text(
"---\nname: pipeline_orchestrator_agent\ntools: Read, Bash\n---\n\nbody\n",
encoding="utf-8",
)
return tmp_path
def first_pair() -> tuple[str, str]:
"""One (source, mirror) pair for symmetric-edit mutations."""
return ("deep-research/agents/research_architect_agent.md",
"agents/research_architect_agent.md")
def rewrite_tools_line(path: Path, new_line: str) -> None:
text = path.read_text(encoding="utf-8")
path.write_text(text.replace(PINNED_TOOLS_LINE, new_line),
encoding="utf-8")
def errs_for(tmp_path: Path, rel: str) -> list[str]:
return [e for e in check(tmp_path) if rel in e]
# --- invariant 0: the real tree is green --------------------------------------
def test_real_repo_passes():
assert check(REPO_ROOT) == []
# --- green fixture -------------------------------------------------------------
def test_green_tree_passes(tmp_path):
make_tree(tmp_path)
assert check(tmp_path) == []
# --- invariant 1: exact value + semantic parse ---------------------------------
def test_symmetric_bash_readd_fails_on_both_files(tmp_path):
# THE #524 drift scenario: source+mirror edited together to re-add Bash.
# Mirror-sync stays green (byte-equal pair); this lint must fire on BOTH,
# via the semantic value check AND the byte-exact witness.
make_tree(tmp_path)
src, mirror = first_pair()
for rel in (src, mirror):
rewrite_tools_line(tmp_path / rel, PINNED_TOOLS_LINE + ", Bash")
assert any("diverges from the canonical" in e for e in errs_for(tmp_path, src))
assert any("diverges from the canonical" in e for e in errs_for(tmp_path, mirror))
def test_dropped_tool_fails(tmp_path):
make_tree(tmp_path)
src, _ = first_pair()
rewrite_tools_line(tmp_path / src, "tools: Read, Write, Edit, Glob")
assert any("diverges from the canonical" in e for e in errs_for(tmp_path, src))
def test_typoed_tool_name_fails(tmp_path):
make_tree(tmp_path)
src, _ = first_pair()
rewrite_tools_line(tmp_path / src, "tools: Read, Write, Edit, Gerp, Glob")
assert any("diverges from the canonical" in e for e in errs_for(tmp_path, src))
def test_trailing_whitespace_is_byte_drift(tmp_path):
# A trailing space keeps the semantic value intact but breaks the
# byte-exact witness — the exact-form pin must still fire.
make_tree(tmp_path)
src, _ = first_pair()
rewrite_tools_line(tmp_path / src, PINNED_TOOLS_LINE + " ")
assert any("not byte-equal" in e for e in errs_for(tmp_path, src))
def test_crlf_conversion_is_byte_drift(tmp_path):
# A symmetric LF->CRLF conversion leaves YAML semantics intact, so only
# the CR-sensitive byte witness catches it (codex round-1 P2).
make_tree(tmp_path)
src, mirror = first_pair()
for rel in (src, mirror):
p = tmp_path / rel
p.write_bytes(p.read_bytes().replace(b"\n", b"\r\n"))
assert any("not byte-equal" in e for e in errs_for(tmp_path, src))
assert any("not byte-equal" in e for e in errs_for(tmp_path, mirror))
def test_missing_tools_key_fails_as_widening(tmp_path):
# Dropping the key silently widens capability (agent inherits ALL tools).
make_tree(tmp_path)
src, _ = first_pair()
p = tmp_path / src
p.write_text(p.read_text(encoding="utf-8").replace(
PINNED_TOOLS_LINE + "\n", ""), encoding="utf-8")
assert any("no `tools:` key" in e for e in errs_for(tmp_path, src))
def test_escaped_duplicate_key_fails(tmp_path):
# The sharpest round-2 bypass: keep the pinned bare line AND add a
# `s`-escaped `"tools":` duplicate. safe_load collapses the two
# to the last-wins canonical value; the duplicate-preserving node tree
# sees TWO `tools` keys and fires.
make_tree(tmp_path)
src, _ = first_pair()
rewrite_tools_line(
tmp_path / src,
'"tool\\u0073": Read, Write, Edit, Grep, Glob, Bash\n'
+ PINNED_TOOLS_LINE,
)
assert any("2 `tools` keys" in e for e in errs_for(tmp_path, src))
def test_quoted_duplicate_key_fails(tmp_path):
make_tree(tmp_path)
src, _ = first_pair()
rewrite_tools_line(tmp_path / src,
f'{PINNED_TOOLS_LINE}\n"tools": Read, Bash')
assert any("2 `tools` keys" in e for e in errs_for(tmp_path, src))
def test_missing_allowlisted_file_fails(tmp_path):
make_tree(tmp_path)
src, _ = first_pair()
(tmp_path / src).unlink()
assert any("missing" in e for e in errs_for(tmp_path, src))
def test_no_frontmatter_fails(tmp_path):
make_tree(tmp_path)
src, _ = first_pair()
(tmp_path / src).write_text("body only\n", encoding="utf-8")
assert any("no YAML frontmatter" in e for e in errs_for(tmp_path, src))
def test_uncomposable_frontmatter_fails_closed(tmp_path):
make_tree(tmp_path)
src, _ = first_pair()
p = tmp_path / src
p.write_text(p.read_text(encoding="utf-8").replace(
PINNED_TOOLS_LINE, "tools: [unclosed"), encoding="utf-8")
assert any("does not compose" in e for e in errs_for(tmp_path, src))
def test_body_tools_line_does_not_satisfy_pin(tmp_path):
# The pinned line must live in FRONTMATTER; a body mention is not a grant.
make_tree(tmp_path)
src, _ = first_pair()
p = tmp_path / src
p.write_text(p.read_text(encoding="utf-8").replace(
PINNED_TOOLS_LINE + "\n", "") + f"\n{PINNED_TOOLS_LINE}\n",
encoding="utf-8")
assert any("no `tools:` key" in e for e in errs_for(tmp_path, src))
# --- invariant 2: Bucket A frontmatter must not declare Bash --------------------
def bash_fixture(tmp_path, frontmatter_body: str) -> list[str]:
"""Write a Bucket A agent with the given frontmatter body and return the
errors mentioning it."""
eic = tmp_path / "academic-paper-reviewer/agents/eic_agent.md"
eic.write_text(f"---\n{frontmatter_body}\n---\n\nbody\n",
encoding="utf-8")
return [e for e in check(tmp_path) if "eic_agent" in e]
def test_bucket_a_agent_advertising_bash_fails(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, Bash")
assert any("declares Bash" in e for e in errs)
def test_quoted_string_value_bash_fails(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path, 'name: eic_agent\ntools: "Read, Bash"')
assert any("declares Bash" in e for e in errs)
def test_quoted_name_with_bash_fails(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path, 'name: "eic_agent"\ntools: Read, Bash')
assert any("declares Bash" in e for e in errs)
def test_flow_list_bash_fails(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: [Read, Bash]")
assert any("declares Bash" in e for e in errs)
def test_block_list_bash_fails(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
"name: eic_agent\ntools:\n - Read\n - Bash")
assert any("declares Bash" in e for e in errs)
def test_inline_comment_bash_fails(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
"name: eic_agent\ntools: Read, Bash # reviewed")
assert any("declares Bash" in e for e in errs)
def test_permission_specifier_bash_fails(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, Bash(git:*)")
assert any("declares Bash" in e for e in errs)
def test_bashoutput_is_a_different_tool_and_passes(tmp_path):
# Exact base-name match: BashOutput grants no shell; a prefix match
# would false-fire on it.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, BashOutput")
assert errs == []
def test_lowercase_bash_is_a_different_tool_and_passes(tmp_path):
# Tool names are exact: `bash` (lowercase) is not the shell-granting
# `Bash` tool. Case-folding would false-fire on a legitimate value.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, bash")
assert errs == []
def test_bom_padded_bash_fails(tmp_path):
# #524 r9: a BOM (U+FEFF) is NOT stripped by str.strip(), so `Bash`
# would survive as a token != `Bash` and slip the membership test.
# _fold drops Cf format chars so it collapses back to Bash.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, Bash")
assert any("declares Bash" in e for e in errs)
def test_zero_width_space_padded_bash_fails(tmp_path):
# #524 r9: zero-width space (U+200B) — a Cf format char str.strip() leaves.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, ​Bash​")
assert any("declares Bash" in e for e in errs)
def test_zero_width_non_joiner_padded_bash_fails(tmp_path):
# #524 r9: zero-width non-joiner (U+200C) — another Cf format char.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, ‌Bash‌")
assert any("declares Bash" in e for e in errs)
def test_fullwidth_bash_fails(tmp_path):
# #524 r9: fullwidth "Ｂａｓｈ" (U+FF22 etc.) is a compatibility homoglyph;
# NFKC in _fold folds it onto ASCII "Bash".
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, Ｂａｓｈ")
assert any("declares Bash" in e for e in errs)
def test_fullwidth_paren_permission_specifier_bash_fails(tmp_path):
# #524 r10: fullwidth parens U+FF08/U+FF09 in a permission specifier. The
# ASCII "(" split misses them, so folding must happen BEFORE the split —
# otherwise NFKC leaves the whole "Bash（git:*）" as one token != "Bash".
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, Bash（git:*）")
assert any("declares Bash" in e for e in errs)
def test_fullwidth_bash_and_paren_specifier_fails(tmp_path):
# #524 r10: both the tool name AND its parens fullwidth — the whole thing
# must fold to ASCII "Bash" and be caught.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, Ｂａｓｈ（git:*）")
assert any("declares Bash" in e for e in errs)
def test_fullwidth_comma_separated_bash_fails(tmp_path):
# #524 r11: fullwidth comma U+FF0C. split(",") misses it, so "Read，Bash"
# stays one token — folding must happen on the WHOLE value BEFORE the comma
# split (the r11 corollary of r10), or "Read，Bash" folds to "Read,Bash"
# (one token != "Bash") and slips.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read，Bash")
assert any("declares Bash" in e for e in errs)
def test_small_comma_separated_bash_fails(tmp_path):
# #524 r11: small comma U+FE50 — another compatibility comma NFKC folds to
# ASCII "," only if the fold precedes the split.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read﹐Bash")
assert any("declares Bash" in e for e in errs)
def test_fullwidth_comma_and_name_and_paren_bash_fails(tmp_path):
# #524 r11: fullwidth comma + fullwidth name + fullwidth parens all at once
# — the whole-value fold must reduce it to Read + Bash.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read，Ｂａｓｈ（git:*）")
assert any("declares Bash" in e for e in errs)
def test_nfkc_stable_alt_separator_is_not_a_bash_grant(tmp_path):
# #524 r12 (convergence boundary, documented as a NON-bug): an alternate
# separator that NFKC does NOT fold to ASCII "," — e.g. an ideographic comma
# U+3001, an Arabic comma U+060C, or a semicolon — keeps "Read、Bash" as ONE
# token for EVERYONE: the string is a single YAML scalar, this lint splits
# only on ASCII "," (and any NFKC-normalizing consumer would too), so no
# consumer extracts a bare "Bash" from it and no shell is granted. The
# fullwidth comma/paren cases fail (above) precisely because NFKC DOES fold
# them into the ASCII separators the split honors; these do not. Not
# flagging this is correct — flagging it would be a false positive a future
# maintainer might "fix" by over-broadening the separator set.
make_tree(tmp_path)
assert bash_fixture(tmp_path, "name: eic_agent\ntools: Read、Bash") == []
assert bash_fixture(tmp_path, "name: eic_agent\ntools: Read؍Bash") == []
assert bash_fixture(tmp_path, "name: eic_agent\ntools: Read;Bash") == []
def test_bom_canonical_allowlist_value_still_fires_byte_witness(tmp_path):
# #524 r9 companion: folding the SEMANTIC check must not weaken invariant 1.
# A plugin agent whose tools value is BOM-padded canonical now folds to the
# five canonical tools semantically — but the additive byte-witness must
# STILL fire (the raw line is not byte-equal to PINNED_TOOLS_LINE).
make_tree(tmp_path)
target = tmp_path / sorted(ALLOWLISTED_FILES)[0]
target.write_text(
"---\nname: report_compiler_agent\n"
"tools: Read, Write, Edit, Grep, Glob\n---\n\nbody\n",
encoding="utf-8")
errs = [e for e in check(tmp_path) if "byte-equal" in e]
assert errs, "byte-witness must still fire on an invisible-char canonical value"
def test_repeated_identical_scalars_are_not_aliases(tmp_path):
# False-positive guard for the alias-by-shared-identity detector:
# `yaml.compose` does NOT intern identical scalar values (each gets a
# distinct node), so a clean file repeating a value must still pass.
make_tree(tmp_path)
errs = bash_fixture(
tmp_path,
"name: eic_agent\ntools: Read, Read, Grep\ndescription: Read Read")
assert errs == []
def test_non_string_manifest_agent_key_still_reconciles(tmp_path):
# A manifest whose `agents` mapping carries a non-string key alongside
# the real ones must not break reconciliation of the real Bucket A agent.
make_tree(tmp_path)
(tmp_path / MANIFEST).write_text(
'{"agents": {"123": {}, "report_compiler_agent": {}, '
'"research_architect_agent": {}, "synthesis_agent": {}, '
'"eic_agent": {}}}', encoding="utf-8")
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Read, Bash")
assert any("declares Bash" in e for e in errs)
def test_nested_list_member_fails_closed(tmp_path):
# A non-scalar list member is an unrecognized shape — must not be
# silently stringified into a passing value (codex round-2 P2).
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: [Read, [Bash]]")
assert any("unrecognized shape" in e for e in errs)
def test_mapping_list_member_fails_closed(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: [Read, {Bash: 1}]")
assert any("unrecognized shape" in e for e in errs)
def test_mapping_tools_value_fails_closed(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: {Read: yes}")
assert any("unrecognized shape" in e for e in errs)
def test_typed_scalar_tools_on_bucket_a_fails_closed(tmp_path):
# A Bucket A `tools` that is an int/bool/null/timestamp scalar is an
# unrecognized shape — the reconciliation cannot confirm it excludes
# Bash, so it fails closed (scoped to Bucket A; out-of-scope agents with
# a nonsense tools value are not this lint's concern).
make_tree(tmp_path)
for val in ("5", "true", "null", "2020-01-01"):
errs = bash_fixture(tmp_path, f"name: eic_agent\ntools: {val}")
assert any("unrecognized shape" in e for e in errs), val
def test_bare_bash_scalar_fails_closed(tmp_path):
# Bash as the whole scalar value (not a list member) still resolves to
# the `Bash` base name.
make_tree(tmp_path)
errs = bash_fixture(tmp_path, "name: eic_agent\ntools: Bash")
assert any("declares Bash" in e for e in errs)
def test_merge_nested_in_sequence_fails_closed(tmp_path):
# The merge/alias walk must recurse into sequence values, not only
# mapping values — a `<<`/alias buried in a list still fails closed.
make_tree(tmp_path)
errs = bash_fixture(
tmp_path,
"name: eic_agent\n_a: &a {tools: [Read, Bash]}\nx: [<<, *a]")
assert any("merge key / alias" in e for e in errs)
def test_merge_tagged_complex_key_fails_closed(tmp_path):
# codex round-5 P1: the merge tag can land on a non-scalar KEY
# (`? !!merge [x]` is a merge-tagged sequence key). safe_load applies
# merge semantics and injects `tools: [Read, Bash]`; the tag check must
# fire on any node type, not just scalars.
make_tree(tmp_path)
for key in ("!!merge [x]", "!!merge {a: 1}"):
errs = bash_fixture(
tmp_path,
f"name: eic_agent\n? {key}\n: {{tools: [Read, Bash]}}")
assert any("merge key / alias" in e for e in errs), key
def test_deeply_nested_frontmatter_does_not_crash(tmp_path):
# codex round-5 P2: a pathologically deep flow sequence must never crash
# the lint with an unhandled traceback — check() must RETURN. Whether a
# given depth trips RecursionError is Python-version-dependent (3.14
# tolerates far deeper nesting than 3.11), so the version-independent
# contract is "returns, does not raise", not "fails at depth N". A
# depth that does NOT recurse past the limit is a harmless (if odd) file
# and legitimately passes; the fail-closed path is exercised
# deterministically by test_recursion_error_fails_closed below.
make_tree(tmp_path)
deep = "name: eic_agent\ntools: Read, Grep\nx: " + "[" * 400 + "]" * 400
errs = bash_fixture(tmp_path, deep) # must not raise
assert isinstance(errs, list)
def test_compose_recursion_error_maps_to_none():
# _mapping_node must turn a RecursionError from yaml.compose into None
# (→ the caller emits a fail-closed "does not compose" error), not let it
# escape. Driven by monkeypatching compose to raise, so it is
# deterministic and Python-version-independent (unlike relying on a
# specific nesting depth tripping the interpreter's own limit).
import check_tools_allowlist as m
def boom(*a, **k):
raise RecursionError("maximum recursion depth exceeded")
orig = m.yaml.compose
m.yaml.compose = boom
try:
assert m._mapping_node("name: x\n") is None
finally:
m.yaml.compose = orig
def test_walk_recursion_error_fails_closed():
# _uses_merge_or_alias must treat a RecursionError in the walk as True
# (fail closed — the tree could not be proven clean). Forced
# deterministically: a MappingNode whose `.value` is an iterable that
# raises RecursionError on iteration (standing in for a walk that
# recurses past the interpreter limit), so the test does not depend on
# any Python-version recursion depth.
import check_tools_allowlist as m
import yaml as y
class ExplodingValue:
def __iter__(self):
raise RecursionError("maximum recursion depth exceeded")
node = y.MappingNode("tag:yaml.org,2002:map", [])
node.value = ExplodingValue()
assert m._uses_merge_or_alias(node) is True
def test_uncomposable_bucket_a_frontmatter_fails_closed(tmp_path):
# Frontmatter that won't compose can't be cleared by name — fail closed
# (codex round-2 P2: a quoted/indented name under malformed YAML must
# not be silently skipped).
make_tree(tmp_path)
errs = bash_fixture(tmp_path, 'name: "eic_agent"\ndescription: [unclosed')
assert any("does not compose" in e for e in errs)
def test_merge_key_injecting_bash_fails_closed(tmp_path):
# codex round-3: `<<: *base` merges a `tools: [..., Bash]` the
# duplicate-preserving node scan never sees (the top level only shows a
# literal `<<` key). The composed node tree carries the `merge` tag —
# detected there, not by text, and failed closed.
make_tree(tmp_path)
errs = bash_fixture(
tmp_path,
"_base: &b {tools: [Read, Bash]}\n<<: *b\nname: eic_agent")
assert any("merge key / alias" in e for e in errs)
def test_flow_merge_with_quoted_hash_key_fails_closed(tmp_path):
# codex round-3 P1-2: a `#` inside a quoted flow key fooled the old
# text-scan comment strip; the node-tree merge-tag detector is immune.
make_tree(tmp_path)
eic = tmp_path / "academic-paper-reviewer/agents/eic_agent.md"
eic.write_text(
'---\n{ "x#": y, name: eic_agent, <<: &b {tools: "Read, Bash"} }\n'
"---\n\nbody\n", encoding="utf-8")
assert any("merge key / alias" in e
for e in check(tmp_path) if "eic_agent" in e)
def test_alias_value_bash_fails_closed(tmp_path):
# An alias makes `tools` share another node's value — the node tree sees
# the SAME node object twice (shared identity). Fail closed.
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
"name: eic_agent\n_x: &a [Read, Bash]\ntools: *a")
assert any("merge key / alias" in e for e in errs)
def test_ampersand_in_quoted_value_is_not_an_anchor(tmp_path):
# False-positive guard: a literal `&`/`*` inside a quoted scalar is not a
# YAML anchor/alias (the node carries a plain str value) and must NOT
# fail the clean file — the node-tree detector, unlike a text scan, sees
# this correctly.
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
'name: eic_agent\ndescription: "A & B, 3 * 4"\n'
"tools: Read, Grep")
assert errs == []
def test_allowlisted_file_with_alias_fails_closed(tmp_path):
# Invariant 1 also fails closed on aliases (a `<<`/alias could inject the
# pinned value from elsewhere, defeating the value lock).
make_tree(tmp_path)
src, _ = first_pair()
p = tmp_path / src
p.write_text(
"---\nname: research_architect_agent\n_t: &t Read, Write, Edit, "
"Grep, Glob\ntools: *t\n---\n\nbody\n", encoding="utf-8")
assert any("merge key / alias" in e for e in errs_for(tmp_path, src))
def test_indented_fence_in_block_scalar_does_not_truncate(tmp_path):
# codex round-3 P1-1: an indented `---` inside a `description: |` block
# scalar must NOT be read as the closing fence — doing so truncated the
# block and hid a Bucket A `name` + `tools: Read, Bash` below it. Only a
# column-0 `---` closes frontmatter.
make_tree(tmp_path)
eic = tmp_path / "academic-paper-reviewer/agents/eic_agent.md"
eic.write_text(
"---\ndescription: |\n ---\nname: eic_agent\ntools: Read, Bash\n"
"---\n\nbody\n", encoding="utf-8")
assert any("declares Bash" in e
for e in check(tmp_path) if "eic_agent" in e)
def test_bom_prefixed_bucket_a_bash_fails_closed(tmp_path):
# A leading UTF-8 BOM makes `---` fail the column-0 fence match, so
# the file would read as frontmatter-less and skip invariant 2 — while a
# real YAML reader strips the BOM and sees `tools: Read, Bash`. _read_raw
# strips the BOM so the two agree; the smuggled Bash fails closed.
make_tree(tmp_path)
eic = tmp_path / "academic-paper-reviewer/agents/eic_agent.md"
eic.write_bytes(
"---\nname: eic_agent\ntools: Read, Bash\n---\n\nbody\n"
.encode("utf-8"))
assert any("declares Bash" in e
for e in check(tmp_path) if "eic_agent" in e)
def test_bom_prefixed_clean_file_passes(tmp_path):
make_tree(tmp_path)
eic = tmp_path / "academic-paper-reviewer/agents/eic_agent.md"
eic.write_bytes(
"---\nname: eic_agent\ntools: Read, Grep\n---\n\nbody\n"
.encode("utf-8"))
assert not [e for e in check(tmp_path) if "eic_agent" in e]
def test_block_scalar_containing_tools_text_no_false_positive(tmp_path):
# codex round-6 P2: a `description: |` block scalar whose body contains a
# `tools: ...` line must NOT trip the byte-witness — that line is not the
# `tools` KEY. The witness is anchored to the composed key's own line, so
# a clean allowlisted file with such documentation passes.
make_tree(tmp_path)
src, _ = first_pair()
p = tmp_path / src
p.write_text(
f"---\nname: research_architect_agent\n{PINNED_TOOLS_LINE}\n"
"description: |\n tools: this is documentation, not the key\n"
"---\n\nbody\n", encoding="utf-8")
assert errs_for(tmp_path, src) == []
def test_block_scalar_containing_triple_dash_is_not_a_fence(tmp_path):
# The companion false-positive: a block scalar that legitimately contains
# an indented `---` line must still parse the real keys below it and PASS
# a clean file.
make_tree(tmp_path)
errs = bash_fixture(
tmp_path,
"name: eic_agent\ndescription: |\n intro\n --- not a fence\n"
"tools: Read, Grep")
assert errs == []
def test_escaped_tools_key_fires_byte_witness_on_allowlisted(tmp_path):
# codex round-3 P2-1: replacing the pinned line with an escaped-key
# spelling makes raw_lines empty; the byte witness must still fire
# (require the verbatim pinned line), and the semantic check fires too.
make_tree(tmp_path)
src, _ = first_pair()
p = tmp_path / src
p.write_text(
'---\nname: research_architect_agent\n"tool\\u0073": Read, Write, '
"Edit, Grep, Glob\n---\n\nbody\n", encoding="utf-8")
assert any("not byte-equal" in e for e in errs_for(tmp_path, src))
def test_duplicate_tools_on_bucket_a_fails_closed(tmp_path):
# codex round-4 P2: last-wins would pick `Read, Grep` and pass, but a
# first-wins parser would grant Bash. Duplicate-key resolution is
# parser-dependent, so invariant 2 fails closed (as invariant 1 does).
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
"name: eic_agent\ntools: Read, Bash\ntools: Read, Grep")
assert any("`tools` keys" in e and "parser-dependent" in e for e in errs)
def test_duplicate_tools_first_wins_bash_fails_closed(tmp_path):
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
"name: eic_agent\ntools: Read, Grep\ntools: Read, Bash")
assert any("`tools` keys" in e and "parser-dependent" in e for e in errs)
def test_duplicate_name_one_bucket_a_fails_closed(tmp_path):
# A duplicate `name` where one resolution is Bucket A: a non-Bucket-A
# last-wins name would skip the file, hiding a Bucket A first-wins name +
# Bash. Fail closed.
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
"name: safe_agent\nname: eic_agent\ntools: Read, Bash")
assert any("`name` keys" in e and "parser-dependent" in e for e in errs)
def test_duplicate_name_neither_bucket_a_passes(tmp_path):
# If NO resolution is a Bucket A name, the file is out of scope whichever
# way a parser resolves it — no need to fail closed.
make_tree(tmp_path)
errs = bash_fixture(tmp_path,
"name: safe_one\nname: safe_two\ntools: Read, Bash")
assert errs == []
def test_nested_bucket_a_agent_declaring_bash_fails_closed(tmp_path):
# codex round-7 P1: invariant 2 must reach nested agent files (rglob, not
# glob) — the runtime guard keys on `name` regardless of path, so a
# `agents/subdir/x.md` with a Bucket A name + Bash is a real exposure.
make_tree(tmp_path)
nested = (tmp_path
/ "academic-paper-reviewer/agents/subdir/eic_agent.md")
nested.parent.mkdir(parents=True, exist_ok=True)
nested.write_text("---\nname: eic_agent\ntools: Read, Bash\n---\nbody\n",
encoding="utf-8")
assert any("declares Bash" in e for e in check(tmp_path)
if "eic_agent" in e)
def test_directory_symlink_under_agent_dir_fails_closed(tmp_path):
# codex round-8 P1: rglob does not descend into directory symlinks, so a
# tracked `agents/nested -> ../payload` could hide a Bucket A agent
# declaring Bash. Fail closed on the symlink itself.
make_tree(tmp_path)
agents = tmp_path / "academic-paper-reviewer/agents"
agents.mkdir(parents=True, exist_ok=True)
payload = tmp_path / "payload"
payload.mkdir()
(payload / "eic_agent.md").write_text(
"---\nname: eic_agent\ntools: Read, Bash\n---\nbody\n",
encoding="utf-8")
try:
(agents / "nested").symlink_to(payload, target_is_directory=True)
except (OSError, NotImplementedError):
import pytest
pytest.skip("symlinks unavailable on this platform")
assert any("directory symlink" in e for e in check(tmp_path))
def test_bare_cr_frontmatter_bucket_a_bash_fails_closed(tmp_path):
# codex round-8 P1: bare `\r` (old-Mac) is a YAML line break, but a
# split-on-`\n` fence scan reads the file as frontmatter-less and skips
# it, hiding a Bucket A `tools: Bash`. splitlines() recognizes bare CR so
# the declaration is caught.
make_tree(tmp_path)
eic = tmp_path / "academic-paper-reviewer/agents/eic_agent.md"
eic.write_bytes(
"---\rname: eic_agent\rtools: Read, Bash\r---\r".encode("utf-8"))
assert any("declares Bash" in e for e in check(tmp_path)
if "eic_agent" in e)
def test_unicode_line_break_before_tools_no_false_positive(tmp_path):
# codex round-7 P2: YAML counts NEL (U+0085) / LS (U+2028) / PS (U+2029)
# as line breaks but str.split("\n") does not. Placed in a quoted value
# BEFORE the tools key, they shift YAML's start_mark.line off the
# split("\n") index, so a line-based anchor would read the WRONG physical
# line. The byte witness anchors via start_mark.index (byte offset), so
# the clean allowlisted file is not falsely rejected and no non-verbatim
# key line slips past.
make_tree(tmp_path)
src, _ = first_pair()
p = tmp_path / src
p.write_text(
f"---\nname: research_architect_agent\n"
'description: "a…b
c
d"\n'
f"{PINNED_TOOLS_LINE}\n---\n\nbody\n",
encoding="utf-8")
assert errs_for(tmp_path, src) == []
def test_non_bucket_a_agent_with_bash_passes(tmp_path):
# Baked into the green fixture (pipeline_orchestrator_agent declares
# Bash); assert it raises nothing on its own.
make_tree(tmp_path)
assert check(tmp_path) == []
def test_bucket_a_agent_without_tools_key_passes(tmp_path):
# eic_agent in the green fixture has no tools key — inherit is fine;
# the runtime guard still fences it.
make_tree(tmp_path)
assert not [e for e in check(tmp_path) if "eic_agent" in e]
def test_missing_manifest_fails_closed(tmp_path):
make_tree(tmp_path)
(tmp_path / MANIFEST).unlink()
errs = check(tmp_path)
assert any(MANIFEST in e and "failing closed" in e for e in errs)
def test_unparseable_manifest_fails_closed(tmp_path):
make_tree(tmp_path)
(tmp_path / MANIFEST).write_text("{not json", encoding="utf-8")
errs = check(tmp_path)
assert any(MANIFEST in e and "failing closed" in e for e in errs)
def test_valid_json_non_object_manifest_fails_closed(tmp_path):
# A JSON array parses fine but has no `agents` mapping — must be a
# curated diagnostic, not a traceback (codex round-1 P2).
make_tree(tmp_path)
(tmp_path / MANIFEST).write_text("[]", encoding="utf-8")
errs = check(tmp_path)
assert any(MANIFEST in e and "no `agents` mapping" in e for e in errs)
def test_non_mapping_agents_value_fails_closed(tmp_path):
make_tree(tmp_path)
(tmp_path / MANIFEST).write_text('{"agents": []}', encoding="utf-8")
errs = check(tmp_path)
assert any(MANIFEST in e and "no `agents` mapping" in e for e in errs)
# --- lock shape ------------------------------------------------------------------
def test_pinned_line_is_the_frozen_514_value():
# Editing the allowlist is a deliberate security-surface change: it must
# touch this lint in the same commit. This test is the second witness.
assert PINNED_TOOLS_LINE == "tools: Read, Write, Edit, Grep, Glob"
assert CANONICAL_TOOLS == ("Read", "Write", "Edit", "Grep", "Glob")
def test_allowlisted_files_are_the_three_pairs():
assert set(ALLOWLISTED_FILES) == {
"deep-research/agents/report_compiler_agent.md",
"deep-research/agents/research_architect_agent.md",
"deep-research/agents/synthesis_agent.md",
"agents/report_compiler_agent.md",
"agents/research_architect_agent.md",
"agents/synthesis_agent.md",
}