skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity

Define and validate native Codex plugin support (#1164)

Browse Source 
* fix: validate native Codex plugin support

Separate Codex-compatible plugin components from GitHub Copilot custom agents, validate Codex manifest fields and MCP shapes, and document the native agent installation boundary.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix: strengthen Codex plugin verification

Validate Codex MCP per-tool settings and add a pinned native smoke lane that installs the marketplace, discovers skills and MCP, and calls binlog_overview.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix: avoid restarting Codex MCP smoke client

Use the real tool call as the app-server initialization so the Linux smoke test does not start and tear down a redundant status client first.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix: harden Codex validation and smoke cleanup

Constrain smoke artifacts to a dedicated child directory, validate Codex manifest field shapes and required skill paths, and document the separate Agents SDK handoff model.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix: align Codex smoke and hooks parsing

Use the clean Codex installation's default provider for the direct MCP call and reject heterogeneous hooks arrays like the runtime parser.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix: verify every Codex plugin skill

Compare native skill discovery with the full repository inventory, match Codex lexical path rejection, and allow null optional MCP tool settings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix: address Codex compatibility review

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Amaury Levé
2026-09-15 10:47:14 +02:00
committed by GitHub
parent 24f7cfbd42
commit 460a01882f
23 changed files with 1224 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/plugins/marketplace.json
+2 -2
View File
@@ -10,7 +10,7 @@
{
"name": "dotnet",
"source": "./plugins/dotnet",
"description": "C# language server (LSP) integration for coding agents and high-level .NET development skills."
"description": "Core C# and .NET development skills."
},
{
"name": "dotnet-advanced",
@@ -65,7 +65,7 @@
{
"name": "dotnet-test-migration",
"source": "./plugins/dotnet-test-migration",
"description": "Skills and an orchestrator agent for migrating .NET test frameworks and platforms: MSTest and xUnit version upgrades, xUnit-to-MSTest conversion, and VSTest to Microsoft.Testing.Platform."
"description": "Skills for migrating .NET test frameworks and platforms: MSTest and xUnit version upgrades, xUnit-to-MSTest conversion, and VSTest to Microsoft.Testing.Platform."
},
{
"name": "dotnet-aspnetcore",
.github/workflows/codex-plugin-smoke.yml
+70
View File
@@ -0,0 +1,70 @@
name: codex-plugin-smoke
on:
pull_request:
paths:
- ".agents/plugins/marketplace.json"
- "plugins/*/plugin.json"
- "plugins/*/.codex-plugin/plugin.json"
- "plugins/*/.mcp.json"
- "plugins/*/mcp.json"
- "plugins/*/skills/**"
- "eng/codex/**"
- ".github/workflows/codex-plugin-smoke.yml"
push:
branches: [main]
paths:
- ".agents/plugins/marketplace.json"
- "plugins/*/plugin.json"
- "plugins/*/.codex-plugin/plugin.json"
- "plugins/*/.mcp.json"
- "plugins/*/mcp.json"
- "plugins/*/skills/**"
- "eng/codex/**"
- ".github/workflows/codex-plugin-smoke.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
smoke:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
with:
persist-credentials: false
- name: Setup .NET
uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v5
with:
global-json-file: global.json
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24
- name: Install Codex CLI
env:
CODEX_SHA256: d7e18b2597ae8f242f5f31ee9e90deef48dbc9edd634d9868fb6435d08c07f02
CODEX_VERSION: 0.154.0
shell: bash
run: |
mkdir -p artifacts/codex-cli
curl -fsSLo artifacts/codex-cli/codex.tar.gz \
"https://github.com/openai/codex/releases/download/rust-v${CODEX_VERSION}/codex-x86_64-unknown-linux-musl.tar.gz"
echo "${CODEX_SHA256} artifacts/codex-cli/codex.tar.gz" | sha256sum -c -
tar -xzf artifacts/codex-cli/codex.tar.gz -C artifacts/codex-cli
mv artifacts/codex-cli/codex-x86_64-unknown-linux-musl artifacts/codex-cli/codex
chmod +x artifacts/codex-cli/codex
# This calls MCP transport directly without exercising authentication or model inference.
- name: Run credential-free Codex plugin discovery and MCP transport smoke test
run: node eng/codex/codex-plugin-smoke.mjs --codex "${{ github.workspace }}/artifacts/codex-cli/codex"
CONTRIBUTING.md
+47 -13
View File
@@ -5,7 +5,7 @@ Thanks for your interest in contributing. We expect to accept external contribut
This repository contains shared building blocks for coding agents:
- Skills: reusable, task focused instruction packs
- Agents: role based configurations that bundle tool expectations and skill selection
- Agents: GitHub Copilot custom-agent configurations that bundle tool expectations and skill selection
Because these artifacts can affect many users and workflows, we prioritize correctness, clarity, and long term maintainability over speed.
@@ -52,10 +52,12 @@ If your skill does not fit any existing plugin, consider creating a new one.
To create a new plugin:
1. Add `plugins/<plugin-name>/plugin.json`, identical copies at
`plugins/<plugin-name>/.claude-plugin/plugin.json` and
1. Add `plugins/<plugin-name>/plugin.json`, an identical copy at
`plugins/<plugin-name>/.claude-plugin/plugin.json`, a Codex-specific compatibility manifest at
`plugins/<plugin-name>/.codex-plugin/plugin.json`, and a `skills/` directory beneath them.
2. Add a matching entry in `.github/plugin/marketplace.json`, `.claude-plugin/marketplace.json`, `.cursor-plugin/marketplace.json`, and `.agents/plugins/marketplace.json`. Keep plugin entries consistent across all marketplace manifests (including `plugins[].source` format) to reduce drift and make future updates safer.
The Codex manifest must contain only components its runtime loads; do not copy `agents` or
`lspServers` into it.
2. Add a matching entry in `.github/plugin/marketplace.json`, `.claude-plugin/marketplace.json`, `.cursor-plugin/marketplace.json`, and `.agents/plugins/marketplace.json`. Keep names and `plugins[].source` values consistent across all marketplace manifests. Descriptions may be host-specific when a capability is unavailable there; for example, the Codex marketplace must not advertise `.agent.md` agents or LSP integration.
Also add a `plugins/<plugin-name>/version.json` (copy an existing one) so the plugin participates in automated versioning. Start its `plugin.json` version at `0.1.0`.
3. Add a CODEOWNERS entry for the new plugin and its tests (see [Code ownership](#code-ownership)).
4. Add the plugin to the **What's Included** table in the root `README.md`.
@@ -63,6 +65,13 @@ To create a new plugin:
See existing plugins for the expected format.
This legacy multi-host layout and its Codex parity validation are a compatibility bridge, not the
intended permanent Agent Plugins package format. A future Agent Plugins 1.0 migration would use a
schema-qualified root `plugin.json`, fixed `skills/` and `mcp.json` locations, and namespaced client
extensions instead of top-level host fields. That migration should remove the duplicated
host-specific manifests and most custom parity and shape validation while retaining a smaller Codex
integration smoke test for installation and discovery.
### The `dotnet-experimental` plugin
Use `dotnet-experimental` when you want to try out a skill idea but are not yet confident it belongs in a stable plugin — for example, when the skill is outside your usual area of responsibility, the approach is unproven, or you want community feedback before committing to a long-term home.
@@ -80,8 +89,9 @@ Place experimental skills under `plugins/dotnet-experimental/skills/` with match
Each plugin is versioned independently. Every plugin carries the manifests its consumers read:
`plugins/<plugin>/plugin.json`, `plugins/<plugin>/.codex-plugin/plugin.json`, and
`plugins/<plugin>/.claude-plugin/plugin.json`. The Claude manifest is an exact generated copy of
the root manifest. Consumers (Copilot CLI, Claude, Codex, Cursor) read the version directly from
this repository.
the root manifest. The Codex manifest shares the stamped version but is host-specific and may omit
unsupported root-manifest fields. Consumers (Copilot CLI, Claude, Codex, Cursor) read the version
directly from this repository.
Each `plugins/<plugin>/version.json` declares the plugin's major/minor release base and the files
that count as effective plugin content. A calculated manifest version transition is a release
@@ -117,10 +127,12 @@ do not trigger skill evaluations.
## Bundled MCP servers
A plugin that bundles an MCP server must declare it in **every** manifest it ships. Each host reads a
different one — Copilot reads `plugin.json`, Codex reads `.codex-plugin/plugin.json`, and Claude reads
`.claude-plugin/plugin.json` — so a server declared in only one of them is silently unavailable in the
others.
Under the repository's current legacy layout, a plugin that bundles an MCP server must declare it
in **every** manifest it ships. Each host reads a different one — Copilot reads `plugin.json`, Codex
reads `.codex-plugin/plugin.json`, and Claude reads `.claude-plugin/plugin.json` — so a server
declared in only one of them is silently unavailable in the others. Keep the same server names and
transport settings, but preserve host-specific schema differences instead of copying optional
fields blindly.
Declare the servers inline as an object. The alternative form, a relative path to a companion
`.mcp.json`, is resolved by hosts against the **plugin root** and not against the directory holding the
@@ -128,7 +140,29 @@ manifest, so `"mcpServers": "./.mcp.json"` inside `.codex-plugin/plugin.json` po
`plugins/<plugin>/.mcp.json`, never `plugins/<plugin>/.codex-plugin/.mcp.json`.
`skill-validator check` enforces both rules: it fails when a referenced `.mcp.json` does not resolve
from the plugin root, and when the manifests do not declare the same set of servers.
from the plugin root, when the manifests do not declare the same set of servers, or when the Codex
manifest uses a server shape that Codex rejects. In particular, Codex interprets `tools` as a map of
per-tool settings, not an allow-list array. Omit `tools` to enable all tools.
The `codex-plugin-smoke` workflow pins a supported Codex CLI release and, without API credentials,
verifies marketplace installation, skill discovery, MCP discovery, and direct MCP transport through
a real `binlog_overview` call against a generated binary log. It does not cover authentication,
model inference, prompt-based skill selection, or model-driven tool selection.
## Host-specific agents
[Agent Plugins 1.0](https://agent-plugins.org/specification) standardizes skills and MCP
servers, not custom agents or LSP servers. Files under `plugins/<plugin>/agents/*.agent.md` use
[GitHub Copilot custom-agent](https://docs.github.com/copilot/reference/custom-agents-configuration)
frontmatter such as `tools`, `agents`, and `handoffs`.
Do not list `.agent.md` files in `.codex-plugin/plugin.json`: Codex ignores that field and does not
install those files as native agents. Codex discovers native agents from `.codex/agents/*.toml` in a
trusted project or `~/.codex/agents/*.toml` in a user profile. Plugin installation does not populate
those locations, so do not generate or claim native Codex agents until the repository defines and
tests a documented installation mechanism. Copilot handoffs also have no static Codex equivalent;
Codex delegation is model-driven. OpenAI Agents SDK handoffs are an application-level orchestration
API between SDK-defined agents, not a plugin packaging or discovery mechanism.
## Before you start
@@ -227,7 +261,7 @@ Also:
- Avoid duplicating text across multiple skills. Prefer referencing shared patterns.
- Do not include content copied from other repositories. If you are inspired by existing work, rewrite in your own words and adapt it to our conventions.
## Proposing a new agent
## Proposing a new GitHub Copilot agent
An agent definition should be opinionated but bounded:
@@ -235,7 +269,7 @@ An agent definition should be opinionated but bounded:
- Define boundaries (what the agent should not do).
- List the skills it expects to use and how it chooses among them.
Add an agent file under a plugin's `agents/` directory:
Add a GitHub Copilot custom-agent file under a plugin's `agents/` directory:
```text
plugins/<plugin>/agents/<agent-name>.agent.md
README.md
+23 -3
View File
@@ -2,18 +2,29 @@
[![Dashboard](https://github.com/dotnet/skills/actions/workflows/pages/pages-build-deployment/badge.svg)](https://dotnet.github.io/skills/)
This repository contains the .NET team's curated set of core skills and custom agents for coding agents. For information about the Agent Skills standard, see [agentskills.io](https://agentskills.io).
This repository contains the .NET team's curated set of portable skills and host-specific custom
agents for coding agents. For information about the Agent Skills standard, see
[agentskills.io](https://agentskills.io).
> [!TIP]
> **Compare skill value:** [Open the Skill Value dashboard](https://dotnet.github.io/skills/)
> See token use, elapsed time, activation, and not-passed rates by plugin, skill,
> executor model, and judge model.
Plugin support is component-specific:
- Skills and MCP servers are the portable component types defined by
[Agent Plugins 1.0](https://agent-plugins.org/specification).
- Files under `agents/*.agent.md` use GitHub Copilot custom-agent conventions. They are not
installed as native OpenAI Codex agents.
- Native Codex agents use `.codex/agents/*.toml`. This repository does not currently ship them
because Codex plugin installation does not place agent files in those discovery locations.
## What's Included
| Plugin | Description |
|--------|-------------|
| [dotnet](plugins/dotnet/) | C# language server (LSP) integration for coding agents and high-level .NET development skills. |
| [dotnet](plugins/dotnet/) | High-level .NET development skills plus C# language server (LSP) integration for hosts that support it. |
| [dotnet-advanced](plugins/dotnet-advanced/) | Collection of .NET skills for handling specific .NET tasks for special scenarios. |
| [dotnet-data](plugins/dotnet-data/) | Skills for .NET data access and Entity Framework related tasks. |
| [dotnet-diag](plugins/dotnet-diag/) | Skills for .NET performance investigations, debugging, and incident analysis. |
@@ -24,7 +35,7 @@ This repository contains the .NET team's curated set of core skills and custom a
| [dotnet-ai](plugins/dotnet-ai/) | AI and ML skills for .NET: technology selection, LLM integration, agentic workflows, RAG pipelines, MCP, and classic ML with ML.NET. |
| [dotnet-template-engine](plugins/dotnet-template-engine/) | .NET Template Engine skills: template discovery, project scaffolding, and template authoring. |
| [dotnet-test](plugins/dotnet-test/) | Skills for running, generating, analyzing, and improving .NET tests: test execution, filtering, platform detection, coverage, testability, and MSTest workflows. |
| [dotnet-test-migration](plugins/dotnet-test-migration/) | Skills and an orchestrator agent for migrating .NET test frameworks and platforms: MSTest and xUnit version upgrades, xUnit-to-MSTest conversion, and VSTest to Microsoft.Testing.Platform. |
| [dotnet-test-migration](plugins/dotnet-test-migration/) | Skills and a GitHub Copilot orchestrator agent for migrating .NET test frameworks and platforms: MSTest and xUnit version upgrades, xUnit-to-MSTest conversion, and VSTest to Microsoft.Testing.Platform. |
| [dotnet-aspnetcore](plugins/dotnet-aspnetcore/) | ASP.NET Core web development skills including middleware, endpoints, real-time communication, and API patterns. |
| [dotnet-blazor](plugins/dotnet-blazor/) | Skills for Blazor development: component authoring, interactivity, and web application patterns. |
| [dotnet11](plugins/dotnet11/) | Skills for new .NET 11 APIs and language features. |
@@ -89,6 +100,15 @@ For local development or unpublished changes, import plugins from a local checko
Skills in this repository follow the [agentskills.io](https://agentskills.io) open standard
and are compatible with [OpenAI Codex](https://developers.openai.com/codex/skills).
Codex plugin installs expose the skills and any Codex-compatible MCP servers declared by the
plugin. They do not expose the repository's GitHub Copilot `.agent.md` files, their static
handoffs, or host-specific LSP declarations. Use Codex's built-in dynamic subagent delegation
instead. Native Codex custom agents are a separate configuration mechanism documented under
[multi-agent workflows](https://developers.openai.com/codex/multi-agent/). OpenAI
[Agents SDK handoffs](https://openai.github.io/openai-agents-python/handoffs/) are an
application-level transfer between SDK-defined agents; they are separate from both Copilot's
static UI handoff metadata and Codex plugin or custom-agent packaging.
#### Plugin marketplace (recommended)
Codex CLI v0.121.0 and later supports a [plugin marketplace](https://developers.openai.com/codex/plugins).
eng/codex/codex-plugin-smoke.mjs
+382
View File
@@ -0,0 +1,382 @@
import assert from "node:assert/strict";
import { spawn, spawnSync } from "node:child_process";
import { createInterface } from "node:readline";
import {
existsSync,
mkdirSync,
readFileSync,
readdirSync,
rmSync,
writeFileSync,
} from "node:fs";
import { isAbsolute, join, resolve } from "node:path";
const supportedCodexVersion = "0.154.0";
const marketplaceName = "dotnet-agent-skills";
const expectedMcpServer = "binlog";
const expectedMcpTool = "binlog_overview";
const options = parseArguments(process.argv.slice(2));
const repositoryRoot = resolve(options.repository ?? process.cwd());
const codex = options.codex ?? process.env.CODEX_BIN ?? "codex";
const stateParent = resolve(
process.env.CODEX_SMOKE_HOME ??
join(repositoryRoot, "artifacts"),
);
const stateRoot = join(stateParent, "codex-plugin-smoke");
const codexHome = join(stateRoot, "codex-home");
const dotnetHome = join(stateRoot, "dotnet-home");
const nugetPackages = join(stateRoot, "nuget-packages");
const environment = {
...process.env,
CODEX_HOME: codexHome,
DOTNET_CLI_HOME: dotnetHome,
DOTNET_ADD_GLOBAL_TOOLS_TO_PATH: "false",
DOTNET_CLI_TELEMETRY_OPTOUT: "1",
DOTNET_GENERATE_ASPNET_CERTIFICATE: "false",
DOTNET_NOLOGO: "1",
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "1",
NUGET_PACKAGES: nugetPackages,
};
delete environment.CODEX_API_KEY;
delete environment.OPENAI_API_KEY;
try {
rmSync(stateRoot, { recursive: true, force: true });
mkdirSync(codexHome, { recursive: true });
mkdirSync(dotnetHome, { recursive: true });
mkdirSync(nugetPackages, { recursive: true });
const version = runCodex(["--version"]).trim();
assert.match(
version,
new RegExp(`\\b${supportedCodexVersion.replaceAll(".", "\\.")}$`),
`Expected Codex ${supportedCodexVersion}, got: ${version}`,
);
const marketplace = runCodexJson([
"plugin",
"marketplace",
"add",
repositoryRoot,
"--json",
]);
assert.equal(marketplace.marketplaceName, marketplaceName);
const marketplaceManifest = JSON.parse(
readFileSync(
join(repositoryRoot, ".agents", "plugins", "marketplace.json"),
"utf8",
),
);
const expectedPlugins = marketplaceManifest.plugins.map((plugin) => plugin.name);
const expectedSkills = expectedSkillInventory(
repositoryRoot,
marketplaceManifest.plugins,
);
for (const name of expectedPlugins) {
const installed = runCodexJson([
"plugin",
"add",
`${name}@${marketplaceName}`,
"--json",
]);
assert.equal(installed.name, name);
assert.equal(installed.marketplaceName, marketplaceName);
}
const plugins = runCodexJson([
"plugin",
"list",
"--marketplace",
marketplaceName,
"--json",
]);
const installedPlugins = new Set(
plugins.installed
.filter((plugin) => plugin.enabled)
.map((plugin) => plugin.name),
);
assert.deepEqual(installedPlugins, new Set(expectedPlugins));
const mcpServers = runCodexJson(["mcp", "list", "--json"]);
const binlog = mcpServers.find(
(server) => server.name === expectedMcpServer && server.enabled,
);
assert.ok(binlog, "Codex did not discover the enabled binlog MCP server");
assert.equal(binlog.transport.type, "stdio");
runCommand("dotnet", [
"tool",
"install",
"Microsoft.AITools.BinlogMcp",
"--tool-path",
join(stateRoot, "binlog-tool"),
"--prerelease",
]);
const sampleRoot = join(stateRoot, "sample");
const sampleBinlog = join(stateRoot, "sample.binlog");
mkdirSync(sampleRoot, { recursive: true });
writeFileSync(
join(sampleRoot, "sample.csproj"),
'<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><OutputType>Exe</OutputType><TargetFramework>net11.0</TargetFramework></PropertyGroup></Project>',
);
writeFileSync(
join(sampleRoot, "Program.cs"),
'System.Console.WriteLine("smoke");',
);
runCommand("dotnet", [
"build",
join(sampleRoot, "sample.csproj"),
`-bl:${sampleBinlog}`,
"--nologo",
]);
await testAppServer(sampleBinlog, expectedSkills);
console.log(
`Codex ${supportedCodexVersion} installed ${expectedPlugins.length} plugins, discovered all ${expectedSkills.size} skills and ${expectedMcpServer}, and called ${expectedMcpTool}.`,
);
} finally {
if (process.env.CODEX_SMOKE_KEEP_HOME !== "1") {
await removeStateRoot();
}
}
async function testAppServer(sampleBinlog, expectedSkills) {
const appServer = spawn(codex, ["app-server", "--stdio"], {
cwd: repositoryRoot,
env: environment,
stdio: ["pipe", "pipe", "pipe"],
});
const pending = new Map();
const stderr = [];
let nextId = 1;
createInterface({ input: appServer.stdout }).on("line", (line) => {
let message;
try {
message = JSON.parse(line);
} catch {
return;
}
if (message.id === undefined) {
return;
}
const request = pending.get(message.id);
if (!request) {
return;
}
pending.delete(message.id);
clearTimeout(request.timeout);
if (message.error) {
request.reject(
new Error(
`${request.method} failed: ${JSON.stringify(message.error)}`,
),
);
} else {
request.resolve(message.result);
}
});
createInterface({ input: appServer.stderr }).on("line", (line) => {
stderr.push(line);
console.error(`[codex app-server] ${line}`);
});
const exit = new Promise((resolveExit) => {
appServer.once("exit", (code, signal) => {
const error = new Error(
`Codex app-server exited before the smoke test completed (code ${code}, signal ${signal}).\n${stderr.join("\n")}`,
);
for (const request of pending.values()) {
clearTimeout(request.timeout);
request.reject(error);
}
pending.clear();
resolveExit();
});
});
function notify(method, params) {
const message = params === undefined ? { method } : { method, params };
appServer.stdin.write(`${JSON.stringify(message)}\n`);
}
function request(method, params, timeoutMs = 180_000) {
const id = nextId++;
return new Promise((resolveRequest, rejectRequest) => {
const timeout = setTimeout(() => {
pending.delete(id);
rejectRequest(
new Error(
`${method} timed out after ${timeoutMs} ms.\n${stderr.join("\n")}`,
),
);
}, timeoutMs);
pending.set(id, {
method,
resolve: resolveRequest,
reject: rejectRequest,
timeout,
});
appServer.stdin.write(`${JSON.stringify({ id, method, params })}\n`);
});
}
try {
await request("initialize", {
clientInfo: {
name: "dotnet-skills-codex-smoke",
version: "1.0.0",
},
capabilities: {
experimentalApi: true,
},
});
notify("initialized");
const skills = await request("skills/list", {
cwds: [repositoryRoot],
forceReload: true,
});
const discoveryErrors = skills.data.flatMap((entry) => entry.errors);
assert.deepEqual(
discoveryErrors,
[],
`Codex reported skill discovery errors: ${JSON.stringify(discoveryErrors)}`,
);
const discoveredSkills = skills.data.flatMap((entry) => entry.skills);
const discoveredPluginSkills = new Set(
discoveredSkills
.filter((skill) => skill.pluginId?.endsWith(`@${marketplaceName}`))
.map((skill) => skill.name),
);
assert.deepEqual(
discoveredPluginSkills,
expectedSkills,
"Codex plugin skill inventory does not match the repository",
);
const thread = await request("thread/start", {
cwd: repositoryRoot,
ephemeral: true,
});
const threadId = thread.thread.id;
const result = await request("mcpServer/tool/call", {
threadId,
server: expectedMcpServer,
tool: expectedMcpTool,
arguments: {
binlog_file: sampleBinlog,
},
});
assert.notEqual(result.isError, true, `${expectedMcpTool} returned an error`);
assert.ok(
(result.content?.length ?? 0) > 0 || result.structuredContent,
`${expectedMcpTool} returned no build overview`,
);
} finally {
appServer.stdin.end();
const exitedGracefully = await Promise.race([
exit.then(() => true),
new Promise((resolveTimeout) => {
setTimeout(() => resolveTimeout(false), 15_000);
}),
]);
if (!exitedGracefully) {
appServer.kill();
await exit;
}
}
}
function runCodex(args) {
return runCommand(codex, args);
}
function runCommand(command, args) {
const result = spawnSync(command, args, {
cwd: repositoryRoot,
env: environment,
encoding: "utf8",
timeout: 180_000,
});
assert.equal(
result.status,
0,
`${command} ${args.join(" ")} failed.\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`,
);
return result.stdout;
}
function runCodexJson(args) {
return JSON.parse(runCodex(args));
}
function expectedSkillInventory(root, plugins) {
const skills = [];
for (const plugin of plugins) {
const skillsRoot = join(root, plugin.source, "skills");
if (!existsSync(skillsRoot)) {
continue;
}
for (const entry of readdirSync(skillsRoot, { withFileTypes: true })) {
if (
entry.isDirectory() &&
existsSync(join(skillsRoot, entry.name, "SKILL.md"))
) {
skills.push(`${plugin.name}:${entry.name}`);
}
}
}
return new Set(skills);
}
function parseArguments(args) {
const parsed = {};
for (let index = 0; index < args.length; index += 1) {
const argument = args[index];
if (argument === "--codex" || argument === "--repository") {
const value = args[index + 1];
if (!value || value.startsWith("--")) {
throw new Error(`${argument} requires a path`);
}
index += 1;
if (argument === "--codex") {
parsed.codex = resolve(value);
} else {
parsed.repository = resolve(value);
}
} else {
throw new Error(`Unknown argument: ${argument}`);
}
}
if (parsed.codex && !isAbsolute(parsed.codex)) {
throw new Error("--codex must be an absolute path");
}
return parsed;
}
async function removeStateRoot() {
for (let attempt = 1; attempt <= 20; attempt += 1) {
try {
rmSync(stateRoot, { recursive: true, force: true });
return;
} catch (error) {
if (attempt === 20) {
throw error;
}
await new Promise((resolveTimeout) => setTimeout(resolveTimeout, 1_000));
}
}
}
eng/skill-validator/src/Check/ExternalDependencyChecker.cs
+5 -4
View File
@@ -19,8 +19,9 @@ public static partial class ExternalDependencyChecker
"ask_user", "bash", "powershell", "grep", "glob", "view", "sql",
"report_intent", "store_memory", "fetch_copilot_cli_documentation",
// Cross-host spellings that are not case-insensitive matches of the names
// above. Each supported host (Copilot CLI / VS Code, Claude Code, Gemini
// CLI) spells some built-in tools differently:
// above. Each supported .agent.md host (Copilot CLI / VS Code, Claude Code,
// Gemini CLI) spells some built-in tools differently. Native Codex agents use
// .codex/agents/*.toml and do not expose a matching generic tools array:
// "agent" Copilot CLI / VS Code subagent fan-out (Claude: "task")
// "write" Claude Code file creation (Copilot: "create", Gemini: "write_file")
// "execute" Copilot CLI / VS Code run-command (Claude: "bash", Gemini: "run_shell_command")
@@ -188,7 +189,7 @@ public static partial class ExternalDependencyChecker
/// <summary>
/// Check that an agent's <c>tools:</c> list is portable across every
/// supported host. When a capability is granted for one host (e.g. the
/// supported .agent.md host. When a capability is granted for one host (e.g. the
/// Copilot CLI alias <c>edit</c>) but the equivalent for another host
/// (Claude Code's <c>Edit</c> or Gemini CLI's <c>replace</c>) is absent, the
/// agent works on one host and is silently tool-less on the others. Returns
@@ -234,7 +235,7 @@ public static partial class ExternalDependencyChecker
findings.Add(
$"Agent tool '{capability.Name}' is declared for {presentLabel} but not {missingLabel} — " +
$"add {additions} to the tools list so the agent works across all supported hosts. (allow: {key})");
$"add {additions} to the tools list so the agent works across all supported .agent.md hosts. (allow: {key})");
}
return findings;
eng/skill-validator/src/Check/PluginProfiler.cs
+375 -1
View File
@@ -1,14 +1,42 @@
using System.Text.Json;
using SkillValidator.Shared;
namespace SkillValidator.Check;
/// <summary>
/// Validates plugin.json files against the agent plugin conventions.
/// Validates plugin.json files against the repository's host-specific plugin conventions.
/// See: https://code.visualstudio.com/docs/copilot/customization/agent-plugins
/// See: https://code.claude.com/docs/en/plugins-reference (Plugin manifest schema)
/// See: https://developers.openai.com/plugins/build/plugins
/// </summary>
public static class PluginProfiler
{
// Codex's legacy compatibility manifest has no published JSON schema. Keep this allowlist
// aligned with openai/codex/codex-rs/core-plugins/src/manifest.rs.
private static readonly HashSet<string> CodexManifestFields = new(StringComparer.Ordinal)
{
"name",
"version",
"description",
"keywords",
"skills",
"mcpServers",
"apps",
"hooks",
"interface",
"commands",
};
// Keep per-tool validation aligned with McpServerToolConfig and AppToolApproval in
// openai/codex/codex-rs/config/src/mcp_types.rs.
private static readonly HashSet<string> CodexMcpToolApprovalModes = new(StringComparer.Ordinal)
{
"auto",
"prompt",
"writes",
"approve",
};
public static PluginCheckResult ValidatePlugin(PluginInfo plugin)
{
var errors = new List<string>();
@@ -81,6 +109,8 @@ public static class PluginProfiler
}
}
ValidateCodexManifest(plugin, errors);
// --- MCP server parity across manifests ---
ValidateMcpServerParity(plugin, errors);
@@ -98,6 +128,350 @@ public static class PluginProfiler
return result;
}
/// <summary>
/// Validates the legacy Codex compatibility manifest against the fields and MCP shape
/// consumed by the Codex runtime. Agent Plugins 1.0 has a separate portable layout and
/// Codex native agents are discovered from .codex/agents/*.toml, not this manifest.
/// </summary>
private static void ValidateCodexManifest(PluginInfo plugin, List<string> errors)
{
const string relativePath = ".codex-plugin/plugin.json";
var manifestPath = Path.Combine(plugin.DirectoryPath, relativePath.Replace('/', Path.DirectorySeparatorChar));
if (!File.Exists(manifestPath))
return;
if (!PluginDiscovery.TryReadJsonObject(manifestPath, out var manifest, out _))
return; // ValidateMcpServerParity reports malformed companion manifests.
ValidateCodexManifestFields(plugin.DirectoryPath, relativePath, manifest, errors);
if (!PluginDiscovery.TryGetManifestMcpServers(
plugin.DirectoryPath,
manifestPath,
out var servers,
out _)
|| servers is not { } serverObject)
{
return;
}
foreach (var server in serverObject.EnumerateObject())
{
if (server.Value.ValueKind != JsonValueKind.Object)
{
errors.Add($"{relativePath} MCP server '{server.Name}' must be an object.");
continue;
}
if (server.Value.TryGetProperty("tools", out var tools))
ValidateCodexMcpTools(relativePath, server.Name, tools, errors);
}
}
private static void ValidateCodexManifestFields(
string pluginDirectory,
string relativePath,
JsonElement manifest,
List<string> errors)
{
foreach (var property in manifest.EnumerateObject())
{
if (!CodexManifestFields.Contains(property.Name))
{
errors.Add(
$"{relativePath} declares unsupported Codex field '{property.Name}'. " +
"Codex ignores unknown compatibility-manifest fields; keep host-specific components out of this manifest.");
continue;
}
switch (property.Name)
{
case "name":
case "version":
case "description":
ValidateJsonKind(relativePath, $"field '{property.Name}'", property.Value, JsonValueKind.String, errors);
break;
case "apps":
ValidateJsonKind(relativePath, $"field '{property.Name}'", property.Value, JsonValueKind.String, errors);
if (property.Value.ValueKind == JsonValueKind.String)
{
ValidateCodexManifestPaths(
pluginDirectory,
relativePath,
property.Name,
property.Value,
requireAtLeastOne: false,
errors);
}
break;
case "keywords":
ValidateStringArray(relativePath, $"field '{property.Name}'", property.Value, errors);
break;
case "skills":
ValidateCodexManifestPaths(
pluginDirectory,
relativePath,
property.Name,
property.Value,
requireAtLeastOne: true,
errors);
break;
case "commands":
ValidateCodexManifestPaths(
pluginDirectory,
relativePath,
property.Name,
property.Value,
requireAtLeastOne: false,
errors);
break;
case "mcpServers":
if (property.Value.ValueKind is not (JsonValueKind.String or JsonValueKind.Object))
errors.Add($"{relativePath} field '{property.Name}' must be a string or object.");
break;
case "hooks":
ValidateCodexHooks(pluginDirectory, relativePath, property.Value, errors);
break;
case "interface":
ValidateCodexInterface(relativePath, property.Value, errors);
break;
}
}
if (!manifest.TryGetProperty("name", out _))
errors.Add($"{relativePath} has no 'name' field — required by repository policy.");
if (!manifest.TryGetProperty("skills", out var skills))
{
errors.Add($"{relativePath} has no 'skills' field — required by repository policy.");
}
}
private static void ValidateCodexManifestPaths(
string pluginDirectory,
string relativePath,
string field,
JsonElement value,
bool requireAtLeastOne,
List<string> errors)
{
IReadOnlyList<string> paths;
if (value.ValueKind == JsonValueKind.String)
{
paths = [value.GetString()!];
}
else if (value.ValueKind == JsonValueKind.Array &&
value.EnumerateArray().All(item => item.ValueKind == JsonValueKind.String))
{
paths = [.. value.EnumerateArray().Select(item => item.GetString()!)];
}
else
{
errors.Add($"{relativePath} field '{field}' must be a string or an array of strings.");
return;
}
if (requireAtLeastOne && paths.Count == 0)
{
errors.Add($"{relativePath} field '{field}' must contain at least one path.");
return;
}
foreach (var path in paths)
{
if (string.IsNullOrWhiteSpace(path) || !path.StartsWith("./", StringComparison.Ordinal))
{
errors.Add($"{relativePath} field '{field}' path '{path}' must start with './'.");
continue;
}
string pathSuffix = path[2..];
if (pathSuffix.Length == 0)
{
errors.Add($"{relativePath} field '{field}' path must not be './'.");
continue;
}
if (pathSuffix.Split(['/', '\\']).Contains("..", StringComparer.Ordinal))
{
errors.Add($"{relativePath} field '{field}' path '{path}' must not contain '..'.");
continue;
}
if (!PluginDiscovery.TryGetSafeSubdirectory(pluginDirectory, path, out var resolved, out var pathError))
{
errors.Add($"{relativePath} field '{field}' path is invalid: {pathError}");
}
else if (field == "skills" && !Directory.Exists(resolved!) && !File.Exists(resolved!))
{
errors.Add($"{relativePath} field 'skills' path '{path}' does not exist at '{resolved}'.");
}
}
}
private static void ValidateCodexHooks(
string pluginDirectory,
string relativePath,
JsonElement hooks,
List<string> errors)
{
if (hooks.ValueKind == JsonValueKind.String)
{
ValidateCodexManifestPaths(
pluginDirectory,
relativePath,
"hooks",
hooks,
requireAtLeastOne: false,
errors);
return;
}
if (hooks.ValueKind == JsonValueKind.Object)
return;
if (hooks.ValueKind == JsonValueKind.Array)
{
bool allStrings = hooks.EnumerateArray().All(item => item.ValueKind == JsonValueKind.String);
bool allObjects = hooks.EnumerateArray().All(item => item.ValueKind == JsonValueKind.Object);
if (allStrings)
{
ValidateCodexManifestPaths(
pluginDirectory,
relativePath,
"hooks",
hooks,
requireAtLeastOne: false,
errors);
return;
}
if (allObjects)
return;
}
errors.Add(
$"{relativePath} field 'hooks' must be a string, object, or a homogeneous array of strings or objects.");
}
private static void ValidateCodexInterface(string relativePath, JsonElement value, List<string> errors)
{
if (value.ValueKind != JsonValueKind.Object)
errors.Add($"{relativePath} field 'interface' must be an object.");
}
private static void ValidateCodexMcpTools(
string relativePath,
string serverName,
JsonElement tools,
List<string> errors)
{
if (tools.ValueKind != JsonValueKind.Object)
{
errors.Add(
$"{relativePath} MCP server '{serverName}' has an invalid 'tools' value. " +
"Codex expects a map of per-tool settings; omit it to enable all server tools.");
return;
}
foreach (var tool in tools.EnumerateObject())
{
if (tool.Value.ValueKind != JsonValueKind.Object)
{
errors.Add(
$"{relativePath} MCP server '{serverName}' tool '{tool.Name}' settings must be an object.");
continue;
}
foreach (var setting in tool.Value.EnumerateObject())
{
switch (setting.Name)
{
case "approval_mode":
if (setting.Value.ValueKind != JsonValueKind.Null &&
(setting.Value.ValueKind != JsonValueKind.String ||
!CodexMcpToolApprovalModes.Contains(setting.Value.GetString()!)))
{
errors.Add(
$"{relativePath} MCP server '{serverName}' tool '{tool.Name}' has an invalid 'approval_mode'. " +
"Expected one of: auto, prompt, writes, approve.");
}
break;
case "output_token_limit":
if (setting.Value.ValueKind != JsonValueKind.Null &&
(setting.Value.ValueKind != JsonValueKind.Number ||
!setting.Value.TryGetUInt64(out var limit) ||
limit == 0))
{
errors.Add(
$"{relativePath} MCP server '{serverName}' tool '{tool.Name}' has an invalid 'output_token_limit'. " +
"Expected a positive integer.");
}
break;
default:
errors.Add(
$"{relativePath} MCP server '{serverName}' tool '{tool.Name}' declares unsupported setting '{setting.Name}'. " +
"Codex supports only 'approval_mode' and 'output_token_limit'.");
break;
}
}
}
}
private static void ValidateStringArray(
string relativePath,
string field,
JsonElement value,
List<string> errors)
{
if (value.ValueKind != JsonValueKind.Array ||
value.EnumerateArray().Any(item => item.ValueKind != JsonValueKind.String))
{
errors.Add($"{relativePath} {field} must be an array of strings.");
}
}
private static void ValidateStringOrStringArray(
string relativePath,
string field,
JsonElement value,
List<string> errors)
{
if (value.ValueKind == JsonValueKind.String)
return;
ValidateStringArray(relativePath, field, value, errors);
}
private static void ValidateJsonKind(
string relativePath,
string field,
JsonElement value,
JsonValueKind expected,
List<string> errors)
{
ValidateJsonKind(relativePath, field, value, expected, expected, errors);
}
private static void ValidateJsonKind(
string relativePath,
string field,
JsonElement value,
JsonValueKind expectedOne,
JsonValueKind expectedTwo,
List<string> errors)
{
if (value.ValueKind != expectedOne && value.ValueKind != expectedTwo)
{
string expected = expectedOne == expectedTwo
? expectedOne.ToString().ToLowerInvariant()
: $"{expectedOne.ToString().ToLowerInvariant()} or {expectedTwo.ToString().ToLowerInvariant()}";
errors.Add($"{relativePath} {field} must be {expected}.");
}
}
/// <summary>
/// Verifies that the root plugin.json and every companion manifest declare the same MCP
/// servers, and that a manifest referencing an external .mcp.json resolves it from the
eng/skill-validator/src/README.md
+7
View File
@@ -115,6 +115,13 @@ skill-validator check --verbose --plugin ./plugins/my-plugin
skill-validator check --json --plugin ./plugins/my-plugin
```
Plugin checks also validate the Codex compatibility manifest when
`.codex-plugin/plugin.json` is present. Unsupported Codex component fields are rejected, and MCP
server declarations are checked for the known incompatible `tools` array and malformed per-tool
settings. The pinned `codex-plugin-smoke` workflow uses Codex itself to validate the complete
server shape during marketplace installation and a real MCP tool call, avoiding a brittle duplicate
of Codex's evolving server parser in `skill-validator`.
## `check` flags
| Flag | Default | Description |
eng/skill-validator/src/Shared/PluginDiscovery.cs
+23 -7
View File
@@ -114,20 +114,36 @@ public static class PluginDiscovery
out string? error)
{
serverNames = [];
if (!TryGetManifestMcpServers(pluginRoot, manifestPath, out var servers, out error))
return false;
if (servers is { } serverObject)
serverNames = ReadServerNames(serverObject);
return true;
}
internal static bool TryGetManifestMcpServers(
string pluginRoot,
string manifestPath,
out JsonElement? mcpServers,
out string? error)
{
mcpServers = null;
error = null;
if (!TryReadJsonObject(manifestPath, out var doc, out var readError))
if (!TryReadJsonObject(manifestPath, out var manifest, out var readError))
{
error = $"could not be parsed as a JSON object: {readError}";
return false;
}
if (!doc.TryGetProperty("mcpServers", out var servers))
if (!manifest.TryGetProperty("mcpServers", out var servers))
return true;
if (servers.ValueKind == JsonValueKind.Object)
{
serverNames = ReadServerNames(servers);
mcpServers = servers;
return true;
}
@@ -150,19 +166,19 @@ public static class PluginDiscovery
return false;
}
if (!TryReadJsonObject(resolved!, out var mcpDoc, out var mcpReadError))
if (!TryReadJsonObject(resolved!, out var mcpDocument, out var mcpReadError))
{
error = $"'mcpServers' references '{referencePath}', which could not be parsed as a JSON object: {mcpReadError}";
return false;
}
if (!mcpDoc.TryGetProperty("mcpServers", out var referenced) || referenced.ValueKind != JsonValueKind.Object)
if (!mcpDocument.TryGetProperty("mcpServers", out var referenced) || referenced.ValueKind != JsonValueKind.Object)
{
error = $"'mcpServers' references '{referencePath}', which has no 'mcpServers' object.";
return false;
}
serverNames = ReadServerNames(referenced);
mcpServers = referenced;
return true;
}
@@ -171,7 +187,7 @@ public static class PluginDiscovery
/// other root kind, so the kind is checked here and reported as a structured error rather
/// than escaping as an unhandled exception.
/// </summary>
private static bool TryReadJsonObject(string path, out JsonElement doc, out string? error)
internal static bool TryReadJsonObject(string path, out JsonElement doc, out string? error)
{
try
{
eng/skill-validator/tests/Check/AgentPluginTests.cs
+108 -1
View File
@@ -307,6 +307,114 @@ public class PluginProfilerTests
}
}
[Theory]
[InlineData("agents")]
[InlineData("lspServers")]
public void CodexManifestWithUnsupportedComponentErrors(string fieldName)
{
var pluginDir = Path.Combine(Path.GetTempPath(), "plugin-test-" + Guid.NewGuid().ToString("N"));
try
{
Directory.CreateDirectory(Path.Combine(pluginDir, "skills"));
Directory.CreateDirectory(Path.Combine(pluginDir, ".codex-plugin"));
var dirName = Path.GetFileName(pluginDir);
File.WriteAllText(
Path.Combine(pluginDir, ".codex-plugin", "plugin.json"),
$$"""{"name":"{{dirName}}","version":"1.0.0","description":"A test plugin.","skills":["./skills/"],"{{fieldName}}":[]}""");
var plugin = new PluginInfo(dirName, "1.0.0", "A test plugin.", ["./skills/"], [], pluginDir, dirName);
var result = PluginProfiler.ValidatePlugin(plugin);
Assert.Contains(
result.Errors,
e => e.Contains(".codex-plugin/plugin.json") &&
e.Contains($"unsupported Codex field '{fieldName}'"));
}
finally
{
Directory.Delete(pluginDir, true);
}
}
[Theory]
[InlineData("name", "[]", "field 'name' must be string")]
[InlineData("version", "{}", "field 'version' must be string")]
[InlineData("description", "[]", "field 'description' must be string")]
[InlineData("keywords", """["valid",1]""", "field 'keywords' must be an array of strings")]
[InlineData("skills", "{}", "field 'skills' must be a string or an array of strings")]
[InlineData("skills", "[]", "field 'skills' must contain at least one path")]
[InlineData("skills", """["skills"]""", "field 'skills' path 'skills' must start with './'")]
[InlineData("skills", """["./"]""", "field 'skills' path must not be './'")]
[InlineData("skills", """["./packs/../packs/"]""", "field 'skills' path './packs/../packs/' must not contain '..'")]
[InlineData("commands", "{}", "field 'commands' must be a string or an array of strings")]
[InlineData("apps", "[]", "field 'apps' must be string")]
[InlineData("hooks", "[true]", "field 'hooks' must be a string, object")]
[InlineData("hooks", """["./hooks.json",{"hooks":{}}]""", "homogeneous array of strings or objects")]
[InlineData("interface", "[]", "field 'interface' must be an object")]
public void CodexManifestWithInvalidFieldShapeErrors(string fieldName, string invalidJson, string expectedError)
{
var pluginDir = Path.Combine(Path.GetTempPath(), "plugin-test-" + Guid.NewGuid().ToString("N"));
try
{
Directory.CreateDirectory(Path.Combine(pluginDir, "skills"));
Directory.CreateDirectory(Path.Combine(pluginDir, ".codex-plugin"));
var dirName = Path.GetFileName(pluginDir);
var properties = new Dictionary<string, string>
{
["name"] = JsonSerializer.Serialize(dirName),
["version"] = "\"1.0.0\"",
["description"] = "\"A test plugin.\"",
["skills"] = """["./skills/"]""",
};
properties[fieldName] = invalidJson;
var json = "{" + string.Join(",", properties.Select(p => $"\"{p.Key}\":{p.Value}")) + "}";
File.WriteAllText(Path.Combine(pluginDir, ".codex-plugin", "plugin.json"), json);
var plugin = new PluginInfo(dirName, "1.0.0", "A test plugin.", ["./skills/"], [], pluginDir, dirName);
var result = PluginProfiler.ValidatePlugin(plugin);
Assert.Contains(result.Errors, error => error.Contains(expectedError));
}
finally
{
Directory.Delete(pluginDir, true);
}
}
[Theory]
[InlineData("name", "has no 'name' field")]
[InlineData("skills", "has no 'skills' field")]
public void CodexManifestMissingRequiredRepositoryFieldErrors(string omittedField, string expectedError)
{
var pluginDir = Path.Combine(Path.GetTempPath(), "plugin-test-" + Guid.NewGuid().ToString("N"));
try
{
Directory.CreateDirectory(Path.Combine(pluginDir, "skills"));
Directory.CreateDirectory(Path.Combine(pluginDir, ".codex-plugin"));
var dirName = Path.GetFileName(pluginDir);
var properties = new Dictionary<string, string>
{
["name"] = JsonSerializer.Serialize(dirName),
["version"] = "\"1.0.0\"",
["description"] = "\"A test plugin.\"",
["skills"] = """["./skills/"]""",
};
properties.Remove(omittedField);
var json = "{" + string.Join(",", properties.Select(p => $"\"{p.Key}\":{p.Value}")) + "}";
File.WriteAllText(Path.Combine(pluginDir, ".codex-plugin", "plugin.json"), json);
var plugin = new PluginInfo(dirName, "1.0.0", "A test plugin.", ["./skills/"], [], pluginDir, dirName);
var result = PluginProfiler.ValidatePlugin(plugin);
Assert.Contains(result.Errors, error => error.Contains(expectedError));
}
finally
{
Directory.Delete(pluginDir, true);
}
}
[Fact]
public void ValidSkillPathsArrayProducesNoErrors()
{
@@ -516,4 +624,3 @@ public class PluginProfilerTests
Assert.Equal("my-plugin", result.Name);
}
}
eng/skill-validator/tests/Check/PluginMcpManifestTests.cs
+142 -2
View File
@@ -16,8 +16,7 @@ public class PluginMcpManifestTests
"binlog": {
"type": "stdio",
"command": "dotnet",
"args": ["dnx", "Microsoft.AITools.BinlogMcp", "--yes", "--prerelease"],
"tools": ["*"]
"args": ["dnx", "Microsoft.AITools.BinlogMcp", "--yes", "--prerelease"]
}
}
""";
@@ -51,6 +50,18 @@ public class PluginMcpManifestTests
return PluginProfiler.ValidatePlugin(plugin);
}
private static string BinlogServersWithTools(string toolsJson) =>
$$"""
{
"binlog": {
"type": "stdio",
"command": "dotnet",
"args": ["dnx", "Microsoft.AITools.BinlogMcp", "--yes", "--prerelease"],
"tools": {{toolsJson}}
}
}
""";
[Fact]
public void CodexManifestPointingAtNestedMcpJsonErrors()
{
@@ -109,6 +120,113 @@ public class PluginMcpManifestTests
}
}
[Fact]
public void CodexManifestWithToolsArrayErrors()
{
var pluginDir = CreatePluginDir();
try
{
var servers = BinlogServersWithTools("""["*"]""");
WriteManifest(pluginDir, "plugin.json", servers);
WriteManifest(pluginDir, ".codex-plugin/plugin.json", servers);
var result = Validate(pluginDir);
Assert.Contains(
result.Errors,
e => e.Contains(".codex-plugin/plugin.json") &&
e.Contains("binlog") &&
e.Contains("map of per-tool settings"));
}
finally
{
Directory.Delete(pluginDir, true);
}
}
[Theory]
[InlineData("[]", "settings must be an object")]
[InlineData("true", "settings must be an object")]
[InlineData("null", "settings must be an object")]
[InlineData("""{"approval_mode":true}""", "invalid 'approval_mode'")]
[InlineData("""{"approval_mode":"always"}""", "invalid 'approval_mode'")]
[InlineData("""{"output_token_limit":-1}""", "invalid 'output_token_limit'")]
[InlineData("""{"output_token_limit":0}""", "invalid 'output_token_limit'")]
[InlineData("""{"output_token_limit":1.5}""", "invalid 'output_token_limit'")]
[InlineData("""{"output_token_limit":"1"}""", "invalid 'output_token_limit'")]
[InlineData("""{"output_token_limit":18446744073709551616}""", "invalid 'output_token_limit'")]
[InlineData("""{"enabled":true}""", "unsupported setting 'enabled'")]
public void CodexManifestWithInvalidPerToolSettingsErrors(string toolSettingsJson, string expectedError)
{
var pluginDir = CreatePluginDir();
try
{
var servers = BinlogServersWithTools($$"""{"*":{{toolSettingsJson}}}""");
WriteManifest(pluginDir, "plugin.json", servers);
WriteManifest(pluginDir, ".codex-plugin/plugin.json", servers);
var result = Validate(pluginDir);
Assert.Contains(
result.Errors,
e => e.Contains(".codex-plugin/plugin.json") &&
e.Contains("binlog") &&
e.Contains("'*'") &&
e.Contains(expectedError));
}
finally
{
Directory.Delete(pluginDir, true);
}
}
[Fact]
public void CodexManifestWithValidPerToolSettingsSucceeds()
{
var pluginDir = CreatePluginDir();
try
{
var servers = BinlogServersWithTools(
"""{"search":{"approval_mode":"prompt","output_token_limit":30000},"max":{"output_token_limit":18446744073709551615},"defaults":{"approval_mode":null,"output_token_limit":null},"list":{}}""");
WriteManifest(pluginDir, "plugin.json", servers);
WriteManifest(pluginDir, ".codex-plugin/plugin.json", servers);
Assert.Empty(Validate(pluginDir).Errors);
}
finally
{
Directory.Delete(pluginDir, true);
}
}
[Theory]
[InlineData("\"apps\":\"apps.json\"", "field 'apps' path 'apps.json' must start with './'")]
[InlineData("\"hooks\":[\"../outside-hooks.json\"]", "field 'hooks' path '../outside-hooks.json' must start with './'")]
public void CodexManifestWithInvalidComponentPathErrors(string componentJson, string expectedError)
{
var pluginDir = CreatePluginDir();
try
{
WriteManifest(pluginDir, "plugin.json", BinlogServers);
var manifestPath = Path.Combine(pluginDir, ".codex-plugin", "plugin.json");
Directory.CreateDirectory(Path.GetDirectoryName(manifestPath)!);
File.WriteAllText(manifestPath, $$"""
{
"name": "{{Path.GetFileName(pluginDir)}}",
"version": "0.1.0",
"description": "A test plugin.",
"skills": ["./skills/"],
"mcpServers": {{BinlogServers}},
{{componentJson}}
}
""");
Assert.Contains(Validate(pluginDir).Errors, error => error.Contains(expectedError));
}
finally
{
Directory.Delete(pluginDir, true);
}
}
[Fact]
public void CompanionManifestMissingServerErrors()
{
@@ -266,6 +384,28 @@ public class PluginMcpManifestTests
}
}
[Fact]
public void RepositoryCodexManifestsUseSupportedFieldsAndMcpShapes()
{
var pluginsRoot = Path.Combine(FindRepositoryRoot(), "plugins");
foreach (var pluginDirectory in Directory.GetDirectories(pluginsRoot))
{
var rootManifest = Path.Combine(pluginDirectory, "plugin.json");
var codexManifest = Path.Combine(pluginDirectory, ".codex-plugin", "plugin.json");
if (!File.Exists(rootManifest) || !File.Exists(codexManifest))
continue;
var plugin = PluginDiscovery.ParsePluginJson(rootManifest);
Assert.NotNull(plugin);
var result = PluginProfiler.ValidatePlugin(plugin);
Assert.DoesNotContain(
result.Errors,
error => error.Contains(".codex-plugin/plugin.json", StringComparison.Ordinal));
}
}
/// <summary>
/// Walks the ancestors of the test output directory looking for the repository root, which is
/// identified by markers that exist nowhere else. The walk is unbounded and stops at the
plugins/dotnet-diag/.codex-plugin/plugin.json
+1 -2
View File
@@ -2,6 +2,5 @@
"name": "dotnet-diag",
"version": "0.1.2",
"description": "Skills for .NET performance investigations, debugging, and incident analysis.",
"skills": ["./skills/"],
"agents": ["./agents/optimizing-dotnet-performance.agent.md"]
"skills": ["./skills/"]
}
plugins/dotnet-diag/README.md
+7
View File
@@ -0,0 +1,7 @@
# dotnet-diag
Skills for .NET performance investigations, debugging, and incident analysis.
The `optimizing-dotnet-performance` file under `agents/` is a GitHub Copilot custom-agent
definition. Codex plugin installs expose this plugin's skills, but do not install that `.agent.md`
file as a native Codex agent.
plugins/dotnet-msbuild/.codex-plugin/plugin.json
+1 -7
View File
@@ -3,11 +3,6 @@
"version": "0.1.10",
"description": "Comprehensive MSBuild and .NET build skills: failure diagnosis, performance optimization, code quality, and modernization.",
"skills": ["./skills/"],
"agents": [
"./agents/build-perf.agent.md",
"./agents/msbuild-code-review.agent.md",
"./agents/msbuild.agent.md"
],
"mcpServers": {
"binlog": {
"type": "stdio",
@@ -17,8 +12,7 @@
"Microsoft.AITools.BinlogMcp",
"--yes",
"--prerelease"
],
"tools": ["*"]
]
}
}
}
plugins/dotnet-msbuild/README.md
+8
View File
@@ -0,0 +1,8 @@
# dotnet-msbuild
Skills for MSBuild failure diagnosis, performance optimization, code quality, and modernization.
The plugin also exposes the `binlog` MCP server to supported hosts.
Files under `agents/` are GitHub Copilot custom-agent definitions. Codex plugin installs expose the
skills and the Codex-compatible `binlog` MCP declaration, but do not install those `.agent.md` files
as native Codex agents.
plugins/dotnet-template-engine/.codex-plugin/plugin.json
+1 -2
View File
@@ -2,6 +2,5 @@
"name": "dotnet-template-engine",
"version": "0.1.4",
"description": ".NET Template Engine skills for dotnet new: create projects (console app, class library, web API, Blazor, MAUI), discover and search templates, inspect template parameters and frameworks (net8.0, net9.0, net10.0), scaffold solutions, author and validate custom templates, install template packages from NuGet.",
"skills": ["./skills/"],
"agents": ["./agents/template-engine.agent.md"]
"skills": ["./skills/"]
}
plugins/dotnet-template-engine/README.md
+7
View File
@@ -0,0 +1,7 @@
# dotnet-template-engine
Skills for discovering, instantiating, authoring, and validating .NET templates.
The `template-engine` file under `agents/` is a GitHub Copilot custom-agent definition. Codex
plugin installs expose this plugin's skills, but do not install that `.agent.md` file as a native
Codex agent.
plugins/dotnet-test-migration/.codex-plugin/plugin.json
+2 -5
View File
@@ -1,9 +1,6 @@
{
"name": "dotnet-test-migration",
"version": "0.1.9",
"description": "Skills and an orchestrator agent for migrating .NET test frameworks and platforms: MSTest and xUnit version upgrades, xUnit/NUnit-to-MSTest conversion, and VSTest to Microsoft.Testing.Platform.",
"skills": ["./skills/"],
"agents": [
"./agents/test-migration.agent.md"
]
"description": "Skills for migrating .NET test frameworks and platforms: MSTest and xUnit version upgrades, xUnit/NUnit-to-MSTest conversion, and VSTest to Microsoft.Testing.Platform.",
"skills": ["./skills/"]
}
plugins/dotnet-test-migration/README.md
+4 -1
View File
@@ -1,6 +1,6 @@
# dotnet-test-migration
Skills and an orchestrator agent for migrating .NET test frameworks and platforms. Covers framework version upgrades (MSTest, xUnit), cross-framework conversion (xUnit/NUnit → MSTest), and test platform migration (VSTest → Microsoft.Testing.Platform).
Skills and a GitHub Copilot orchestrator agent for migrating .NET test frameworks and platforms. Covers framework version upgrades (MSTest, xUnit), cross-framework conversion (xUnit/NUnit → MSTest), and test platform migration (VSTest → Microsoft.Testing.Platform).
## When to use this plugin
@@ -24,6 +24,9 @@ Skills and an orchestrator agent for migrating .NET test frameworks and platform
## Agents
The agent below is a GitHub Copilot `.agent.md` definition. Codex plugin installs expose the
migration skills, but not this agent or its static handoff.
| Agent | Purpose |
|---|---|
| **test-migration** | Auto-detects framework/version/platform and routes to the correct migration skill; coordinates multi-step migrations (e.g., MSTest v1 → v3 → v4) |
plugins/dotnet-test/.codex-plugin/plugin.json
+1 -13
View File
@@ -2,17 +2,5 @@
"name": "dotnet-test",
"version": "0.2.22",
"description": "Skills for running, generating, analyzing, and improving .NET tests: test execution, filtering, platform detection, coverage, testability, and MSTest workflows.",
"skills": ["./skills/"],
"agents": [
"./agents/code-testing-generator.agent.md",
"./agents/code-testing-researcher.agent.md",
"./agents/code-testing-planner.agent.md",
"./agents/code-testing-implementer.agent.md",
"./agents/code-testing-builder.agent.md",
"./agents/code-testing-tester.agent.md",
"./agents/code-testing-fixer.agent.md",
"./agents/code-testing-linter.agent.md",
"./agents/testability-migration.agent.md",
"./agents/test-quality-auditor.agent.md"
]
"skills": ["./skills/"]
}
plugins/dotnet-test/README.md
+4 -1
View File
@@ -1,6 +1,6 @@
# dotnet-test
Skills and agents for running, generating, analyzing, and improving tests. Originally built for .NET (MSTest, xUnit, NUnit, TUnit) and platforms (VSTest, Microsoft.Testing.Platform); the test-generation pipeline and the six test-analysis skills (anti-patterns, smells, assertion quality, gap analysis, tagging, grade tests) plus the `test-quality-auditor` agent are **polyglot** and also work with Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit 4/5/TestNG), Go (testing/testify), Ruby (RSpec/Minitest), Rust (built-in/proptest), Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest/Boost.Test).
Skills and GitHub Copilot custom agents for running, generating, analyzing, and improving tests. Originally built for .NET (MSTest, xUnit, NUnit, TUnit) and platforms (VSTest, Microsoft.Testing.Platform); the test-generation pipeline and the six test-analysis skills (anti-patterns, smells, assertion quality, gap analysis, tagging, grade tests) plus the `test-quality-auditor` agent are **polyglot** and also work with Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit 4/5/TestNG), Go (testing/testify), Ruby (RSpec/Minitest), Rust (built-in/proptest), Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest/Boost.Test).
> **Test framework/platform migration** (MSTest/xUnit upgrades, xUnit → MSTest, VSTest → Microsoft.Testing.Platform) lives in the separate [`dotnet-test-migration`](../dotnet-test-migration/) plugin.
@@ -95,6 +95,9 @@ filtered-command scenarios in the `run-tests` eval.
## Agents
The agents below are GitHub Copilot `.agent.md` definitions. Codex plugin installs expose this
plugin's skills, but not these agents or their static handoffs.
### User-facing agents
These are the entry-point agents you invoke directly:
plugins/dotnet/.codex-plugin/plugin.json
+2 -3
View File
@@ -1,7 +1,6 @@
{
"name": "dotnet",
"version": "0.2.4",
"description": "C# language server (LSP) integration for coding agents and high-level .NET development skills.",
"skills": ["./skills/"],
"lspServers": "./lsp.json"
"description": "Core C# and .NET development skills for coding agents.",
"skills": ["./skills/"]
}
plugins/dotnet/README.md
+2
View File
@@ -10,6 +10,8 @@ Core .NET and C# skills for coding agents.
## LSP
This plugin declares a C# LSP server that is launched through the .NET CLI.
The LSP declaration is available to hosts that support the plugin `lspServers` extension. Codex
plugin installs expose this plugin's skills but do not load that host-specific LSP declaration.
Prerequisites:
- .NET 10 SDK installed