skills/boshu2__agentops
1
0
Fork 0
mirror of https://github.com/boshu2/agentops.git synced 2026-09-14 15:08:13 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
e1fae0dae63d73be508bb8348a12106df043cab0
boshu2__agentops/docs/contracts/codex-skill-api.md
T
Bo 8cdcb5a903 Train 1: measurement substrate, context diet, retrieval-eval contract (instrument-panel roadmap) (#1087) 
> **Residues closed on the caller's merge instruction** (`499d916a6`):
the round-2 findings were the same failure shape — round-1 repairs
patched cited lines instead of sweeping the class — so this commit
sweeps each file whole: every remaining SATURATED-row-append site in
skill-eval now routes to RUNBOOK retirement, the human-only-skills
*description* is runtime-conditional, premortem's "(MEASURED)" label is
gone, SKILL-API's context table carries all 25 rows and the enforcement
table gains `disable-model-invocation`, and the fixture-identity claim
is stated precisely (probe id, honesty note, and control arm are the
only differing fields — as the acceptance permits). Post-sweep:
validators, full Go suite, 68/68 gates, goldens + headroom bats green,
projections current, gemini in sync. Merging per Bo's instruction.

## What

Train 1 of the accepted [instrument-panel
roadmap](docs/plans/2026-08-26-instrument-panel-roadmap.md) (intent
landed at `986a4feaf`): the measurement substrate, the skill-context
diet, and the retrieval-eval contract. Three worktree-isolated lanes,
each independently validated by a fresh context, plus one integration
commit. 103 files, +5,510/−76.

**L1 — measurement substrate** (`instrument/measurement-substrate`)
- Gate `skill.probe-headroom` (advisory, Fast|Full): answers the
question `skill.probe-coverage` cannot — not "does a probe result exist"
but "could one have existed at all". The rule, ported to Go
(`cli/internal/probeheadroom` + `cli/cmd/probe-headroom` behind a thin
check script — the witness-crosscheck pattern, **no new `ao` root
command**): control arm ≥ 0.75 with ≥ 2 usable reps at ≥ 2 effort levels
⇒ SATURATED (void row, not an honest null); UNMEASURED outranks it;
treatment-silent ⇒ FLOOR; else SEPARATED. RED first: both committed
fixture pairs read `INERT` to everything else in the repo; the failing
separation test predates the implementation, and a bats negative-control
swaps fixture bytes and asserts the gate flips.
- **First reading on real data: 7 of 11 historical probe groups are
SATURATED** — including both `validate-not-proven` runs. Those INERT
rows were never honest nulls; they were void. The 0/12 ledger number now
argues itself.
- Declared denominator for probe-coverage:
`scripts/.skill-probe-denominator-exclusions`, fail-closed parser (entry
without an argument, stale slug, or duplicate ⇒ exit 2). One entry
(`goals`, a pure alias-of `fitness`). Net effect deliberately zero (0/12
→ 0/12: alias left, `one-way-door` entered) — the gain is a declared
number, not a better-looking one.
- Re-landed from the recovered clean-room commit (`9872483bd`),
re-validated against *current* main: `skill-eval` (defers saturation to
the gate id; its shell scripts dropped, not shipped — ratchet intent),
`route`, `one-way-door`, premortem reversibility check, council
`caller_challenge` (schema + validator, per the agent-core boundary that
the panel may challenge, never overrule).

**L4 — context diet** (`instrument/context-diet`)
- `disable-model-invocation: true` on 4 human-only skills (key verified
verbatim against Anthropic's docs). The plan guessed 35 candidates; the
graph said otherwise — 23 carry `user-invocable: true`, and 19 of those
are excluded on cited evidence (rpi consumes
anti-ceremony/implement/plan/validate; workflow scripts reach others;
`goals` is a live migration tombstone). The exclusion evidence is
retained in the lane report.
- One router skill (`human-only-skills`) — the single always-loaded
description that replaces four; it hints, never fires.
- `.out-of-scope/` formalized with this week's three refusals
(checked-in knowledge corpus; ee self-improvement loops; whole-skill A/B
as the measurement unit), each citing its evidence.
- Deterministic proof, no model eval: before/after bytes of
always-loaded description load reported in the lane summary.

**L5 — retrieval-eval contract** (`instrument/retrieval-contract`, lane
verdict PASS 10/10)
- `AGENTS.md` federated row now names **ee (eidetic-engine)** as a
concrete caller-selected memory system — consume, never build; symlink
intact.
- `schemas/pack-quality-expectations.v1.schema.json` + 4 routing goldens
+ `scripts/check-routing-probe-goldens.sh` graded against `ao skills
find`, wired as an **advisory** nightly job. Zero goldens is a failing
state — no new zero-denominator green.
- **The instrument caught a real miss on day one — and its own
prescription fixed it.** Golden `rq-04` expects `validate` for "judge
whether this finished change is actually proven before I merge it"; at
authoring, `ao skills find` ranked the *forbidden* `premortem` first and
`validate` nowhere in six natural phrasings. The pointer-wording-first
repair (validate's description gained the caller's own words: finished,
proven, verdict, merge) now ranks it #1 at 0.333; grader 6/6, and the
golden pins the repair — a description regression reopens it.

## Integration

`regen-all.sh` once over the merged lanes (catalog 52 → 56, four new
codex twins, mesh, router, manifests); `skills/route/SKILL.md`
catalog/router links became prose repo-root references (the projected
twin cannot resolve `../catalog.json` — this was both the
portable-conformance failure and the sole broken doc link);
`codex-portable-conformance.bats` pin 52 → 56.

## Evidence

- `cd cli && go build ./... && go vet ./... && go test ./...` exit 0 ·
`ao gate check --full` **68/68** · four skill validators PASS ·
probe-headroom / routing-goldens / probe-coverage bats PASS ·
`regen-all.sh --check` all current.
- Per-lane fresh validators re-ran every suite on detached content; L5
PASS; L1/L4 NOT_PROVEN solely on the projection-regen clause reserved
for integration (their remaining acceptance observed green), settled
above. Cross-family (Codex) review of the integrated diff recorded in
the session report.
- Two disclosed scope stretches accepted at integration: a one-line
`.gitignore` entry mirroring the witness-crosscheck precedent, and the
probe LEDGER.md fact-correction L1's own change made necessary (noted
for Train 2's L2, which owns that file next).

## Cross-family review (Codex, fresh context)

Round 1: **FAIL** — two blockers (the RED fixtures didn't isolate the
control arm; the goldens grader was red where the plan's acceptance says
green) and eight majors (contract contradictions in the re-landed
skills, a converter-substitution false claim in the codex router twin,
two overreaching `.out-of-scope` entries, stale SKILL-API counts). All
repaired in one bounded round (`db68935a3`): fixtures now byte-identical
outside the control arm, the routing miss actually fixed rather than
tolerated, every cited contradiction reconciled at the source and
re-projected. Post-repair: full Go suite exit 0, `gate check --full`
68/68, all validators and probe/goldens bats green, projections current,
gemini byte-identity restored. Focused re-check verdict recorded in the
session report.

## Follow-ups (Train 2, already planned)

Seeded-defect probes for the judgment spine (every ledger row citing a
passing headroom pre-screen) and the gate-hardening pair
(`Gate-Loosen-Reason` tightening ratchet; mechanical
grounding-validation over evidence docs). Plus, surfaced by this train:
a latent `valid_keys`/schema divergence in `validate-skill-schema.sh`
(two keys the schema defines are absent from the script's allowlist —
pre-existing).
2026-08-26 23:16:51 +00:00

310 lines
13 KiB
Markdown
Raw Blame History

# Codex Skill API Contract
> Source of truth for what the Codex runtime actually supports. All converter output and validation must conform to this contract.
> Orientation contract: [`AGENTS.md`](../../AGENTS.md).
**Official docs:**
- [Codex Skills](https://developers.openai.com/codex/skills/)
- [Codex Multi-Agent](https://developers.openai.com/codex/multi-agent/)
- [Codex CLI Features](https://developers.openai.com/codex/cli/features)
---
## SKILL.md Frontmatter
Every generated AgentOps Codex projection emits only the two fields Codex uses
for discovery:
```yaml
---
name: skill-name
description: 'Explain when this skill triggers and when it does not.'
---
```
The portable Agent Skills specification requires `name` and `description`
and also permits `license`, `compatibility`, string-to-string `metadata`,
and the experimental space-separated `allowed-tools` field. The AgentOps
generator does not currently emit those optional fields. AgentOps-only fields
such as `skill_api_version`, `context`, `model`, `user-invocable`, and
`output_contract` must be stripped from Codex output.
---
## Optional: agents/openai.yaml
Codex skills may include `agents/openai.yaml` for display metadata and policy:
```yaml
interface:
display_name: "User-facing name"
short_description: "Brief description"
icon_small: "./assets/small-logo.svg"
icon_large: "./assets/large-logo.png"
brand_color: "#3B82F6"
default_prompt: "Optional surrounding prompt"
policy:
allow_implicit_invocation: false
dependencies:
tools:
- type: "mcp"
value: "toolName"
description: "Tool description"
transport: "streamable_http"
url: "https://example.com"
```
| Field | Purpose |
|-------|---------|
| `interface.display_name` | User-visible name in Codex UI |
| `interface.short_description` | Brief description for skill browser |
| `policy.allow_implicit_invocation` | `false` prevents auto-activation (explicit `$skill` only) |
| `dependencies.tools` | MCP server dependencies |
---
## Skill Discovery And Installation
The portable Codex skill root is `.agents/skills/` at repository or parent
scope and `$HOME/.agents/skills/` at user scope. Current Codex desktop installs
may also index `$HOME/.codex/skills/`; on Bo's development fleet both user roots
are symlinks to this repository's canonical `skills/` tree.
| Scope | Path | Use Case |
|-------|------|----------|
| Repo (nearest) | `.agents/skills/` from CWD | Folder-specific workflows |
| Repo (parent) | `../.agents/skills/` | Nested repo organization |
| Repo (root) | `$REPO_ROOT/.agents/skills/` | Organization-wide skills |
| User | `$HOME/.agents/skills/` | Portable personal skill collection |
| Host compatibility | `$HOME/.codex/skills/` | Codex-host symlink root when configured |
| Admin | `/etc/codex/skills/` | System-wide defaults |
| System | Bundled with Codex | Built-in skills |
Development symlinks may point directly at `skills/<name>/`; checked-in
`skills-codex/` packages are the portable Agent Skills release projection.
Canonical `skills/` frontmatter is intentionally host-extended and is evaluated
against the AgentOps profile, not mislabeled as portable. The release gate
`scripts/validate-codex-api-conformance.sh` validates every projected package
against the portable field, type, identity, body, and relative-resource-link
contract before applying Codex-specific checks. Plugin caches are neither
source nor an installation target and may be deleted without affecting the
canonical repository skills.
---
## Skill Invocation
| Method | Syntax | Description |
|--------|--------|-------------|
| Explicit | `$skill-name` or `/skills` menu | User directly requests the skill |
| Implicit | Automatic | Codex matches task to skill description |
Skills are loaded via **progressive disclosure**: metadata first (name,
description), full SKILL.md only when activated. Because the description is
the implicit-routing surface, a projection must preserve the source's complete
`Triggers:` clause while keeping its introductory prose within the catalog
budget; a generic summary or dropped trigger is a behavioral defect, even when
the body remains byte-complete.
---
## Multi-Agent (Sub-Agents)
Current Codex releases enable subagent workflows by default. Codex only spawns
subagents when the operator or parent agent explicitly asks it to do so.
Subagents inherit the parent sandbox policy and live runtime overrides.
### Agent Roles
Codex ships with these built-in roles:
| Role | Purpose |
|------|---------|
| `default` | General-purpose fallback |
| `worker` | Execution-focused implementation |
| `explorer` | Read-heavy codebase exploration |
Custom agents live as standalone TOML files under `$HOME/.codex/agents/` for
personal agents or `.codex/agents/` for project-scoped agents. Each custom
agent file must define `name`, `description`, and `developer_instructions`.
Optional fields such as `nickname_candidates`, `model`,
`model_reasoning_effort`, `sandbox_mode`, `mcp_servers`, and `skills.config`
inherit from the parent session when omitted.
Global subagent limits stay in the `[agents]` section of `config.toml`:
```toml
[agents]
max_threads = 6
max_depth = 1
job_max_runtime_seconds = 1800
[agents.reviewer]
description = "Code review specialist"
config_file = "codex-reviewer.toml"
```
`agents.max_threads` defaults to `6`; `agents.max_depth` defaults to `1`,
which allows direct child agents but prevents deeper recursive fan-out.
### Batch Processing
`spawn_agents_on_csv` processes batches of similar tasks:
| Parameter | Description |
|-----------|-------------|
| `csv_path` | Source CSV file |
| `instruction` | Worker prompt template with `{column_name}` placeholders |
| `id_column` | Stable identifiers |
| `output_schema` | Fixed JSON structure for worker results |
| `output_csv_path` | Destination CSV containing row metadata and results |
| `max_concurrency` | Parallel worker limit |
| `max_runtime_seconds` | Worker timeout |
Workers call `report_agent_job_result` exactly once.
`sqlite_home` controls where Codex stores the SQLite-backed state used for
agent jobs and exported results.
### Codex Built-in Tools
Tools available inside a Codex agent session:
| Tool | Purpose |
|------|---------|
| `read_file` | Read file contents |
| `list_dir` | List directory contents |
| `glob_file_search` | Find files by pattern |
| `apply_patch` | Apply file edits (diff-based) |
| `rg` | Ripgrep search |
| `git` | Git operations |
| Shell/terminal tool | Shell command execution |
| `spawn_agent` | Create a focused sub-agent |
| `send_input` | Send follow-up input to a sub-agent |
| `wait_agent` | Wait for one or more sub-agents |
| `close_agent` | Stop a stuck or no-longer-needed sub-agent |
### Claude → Codex Primitive Mapping
| Claude Code | Codex Equivalent | Converter Action |
|-------------|-----------------|------------------|
| `Read` tool | `read_file` | Map |
| `Edit` tool | `apply_patch` | Map |
| `Grep` tool | `rg` | Map |
| `Glob` tool | `glob_file_search` | Map |
| `Agent(subagent_type="Explore")` | Explorer agent role | Map |
| `Skill(skill="name")` | `$name` invocation | Map |
| `TaskCreate` / `TaskList` / `TaskUpdate` | No equivalent (`todo_write`/`update_plan` not available — empirically verified) | Strip |
| `TeamCreate` / `TeamDelete` | No equivalent | Strip |
| `SendMessage` | `send_input` for brief follow-up only | Rewrite or strip |
| `EnterPlanMode` / `ExitPlanMode` | No equivalent | Strip |
| `EnterWorktree` | No equivalent | Strip |
| `context.window` | No equivalent | Strip from frontmatter |
| `context.sections.exclude` | No equivalent | Strip from frontmatter |
| `context.intel_scope` | Deprecated — ignored, no reader | Does not exist |
| `disable-model-invocation` | No equivalent — Codex has no model-invocation switch and always advertises a `$name` skill | Strip from frontmatter |
Skills referencing these primitives produce **broken instructions** in Codex.
`disable-model-invocation` is a Claude Code context-budget control, not a
behavioral instruction, so stripping it loses nothing a Codex reader could act
on. The consequence is intended and asymmetric: a human-only skill is invisible
to Claude's implicit routing but stays `$name`-invocable under Codex. Requirement
1 below already strips it — the row exists so the divergence is declared rather
than discovered.
---
## Converter Requirements
When generating Codex skills from source skills:
1. **Strip all non-Codex frontmatter** — emit only `name` + `description`,
compacting introductory prose while preserving the complete source
`Triggers:` clause as the activation catalog signal
2. **Map Claude tools to Codex tools** — Read→read_file, Edit→apply_patch, Grep→rg, Glob→glob_file_search
3. **Rewrite `Skill(skill="X")` to `$X`** — Codex uses dollar-prefix invocation
4. **Strip ALL task/team primitives** — TaskCreate, TaskList, TeamCreate, SendMessage (none have working Codex equivalents as direct tool calls — `todo_write`/`update_plan` empirically unavailable, and `send_input` is follow-up-only)
5. **Fix paths** — avoid host-specific installed-skill paths; use repository-relative links in shipped instructions
6. **Rewrite reference files** — `.md` files in references/ pass through `codex_rewrite_text()` during copy
7. **Preserve skill body** — the SKILL.md body (instructions) is the skill's value; keep it functional
---
## Validation Criteria
A Codex-conformant skill must:
1. Have frontmatter with only `name` and `description`
2. Contain no Claude-only primitive names (TaskCreate, TeamCreate, SendMessage, etc.)
3. Contain no Claude-specific paths and no dependency on one host's installed-skill root
4. Have valid `agents/openai.yaml` if present
5. Not reference non-existent Codex features (context controls, plan mode, etc.)
---
## Generated inventories
`skills/<name>/SKILL.md` metadata owns the skill inventory and dependency graph.
Run `scripts/regen-all.sh` after changing it. The CLI command surface is a
separate executable projection generated from Cobra; no handwritten CLI-to-skill
map is maintained.
---
## Codex Skill Maintenance
Codex is a first-class runtime in this repo.
- `skills/<name>/SKILL.md` is the canonical behavior contract.
- `skills-codex-overrides/<name>/` is the Codex-specific tailoring layer.
- `skills-codex-overrides/catalog.json` is the machine-readable treatment map for the full catalog.
- `skills-codex/<name>/` is the generated checked-in Codex runtime artifact.
**Editing an EXISTING parity skill regenerates its Codex twin — not just hashes.**
`make regen-all` / `scripts/codex-sync.sh` refresh parity-only twins from
`skills/<name>/` whenever their generated body, prompt, or mirrored references
drift. `scripts/regen-codex-hashes.sh` remains the bookkeeping step after
content is current. Manual edits under `skills-codex/<name>/` are reserved for
bespoke skills or deliberate Codex-only divergence recorded in
`skills-codex-overrides/catalog.json`; otherwise fix the source skill or the
codex-sync transform/template and regenerate.
The current catalog has no bespoke or pointer twins. Every live Codex package is
a generated parity projection of `skills/<name>/SKILL.md` plus its linked local
files. `scripts/codex-sync.sh` owns those packages; manual edits under
`skills-codex/<name>/` are drift and will be overwritten.
`skills-codex-overrides/catalog.json` remains the explicit treatment registry.
If a future runtime-specific implementation is genuinely necessary, declare it
there before editing a twin and add focused parity tests in the same change.
Do not create an undocumented exception or a second skill inventory.
When a skill change affects Codex behavior, phrasing, orchestration, or UX:
1. Update the source skill under `skills/` when the shared contract changes.
2. For parity-only skills, update source or the codex-sync transform/template and regenerate. Update `skills-codex/<name>/SKILL.md` directly only when the Codex runtime copy is bespoke, or update `skills-codex-overrides/<name>/` when the Codex experience should differ from Claude.
- Prompt/operator-layer changes belong in `skills-codex-overrides/<name>/prompt.md`.
- Durable Codex-only body rewrites belong in `skills-codex-overrides/<name>/SKILL.md`.
3. Run the semantic audit if the checked-in Codex body looks suspicious:
```bash
bash scripts/audit-codex-parity.sh
# or target one skill
bash scripts/audit-codex-parity.sh --skill <name>
```
4. Validate the checked-in Codex artifacts:
```bash
bash scripts/audit-codex-parity.sh
bash scripts/validate-codex-override-coverage.sh
bash scripts/validate-codex-generated-artifacts.sh --scope worktree
bash scripts/validate-headless-runtime-skills.sh
python3 scripts/check-cathedral-cut-conformance.py
```
Think of `skills/` as the shared contract, `skills-codex-overrides/` as the durable Codex-only tailoring layer, and `skills-codex/` as the checked-in Codex artifact shipped to users.
Reference in New Issue View Git Blame Copy Permalink