mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
main
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
460a01882f |
Define and validate native Codex plugin support (#1164)
* fix: validate native Codex plugin support Separate Codex-compatible plugin components from GitHub Copilot custom agents, validate Codex manifest fields and MCP shapes, and document the native agent installation boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: strengthen Codex plugin verification Validate Codex MCP per-tool settings and add a pinned native smoke lane that installs the marketplace, discovers skills and MCP, and calls binlog_overview. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: avoid restarting Codex MCP smoke client Use the real tool call as the app-server initialization so the Linux smoke test does not start and tear down a redundant status client first. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: harden Codex validation and smoke cleanup Constrain smoke artifacts to a dedicated child directory, validate Codex manifest field shapes and required skill paths, and document the separate Agents SDK handoff model. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: align Codex smoke and hooks parsing Use the clean Codex installation's default provider for the direct MCP call and reject heterogeneous hooks arrays like the runtime parser. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: verify every Codex plugin skill Compare native skill discovery with the full repository inventory, match Codex lexical path rejection, and allow null optional MCP tool settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: address Codex compatibility review Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
ef5f7cada6 |
Merge pull request #1095 from dotnet/bot/weekly-version-sync
Weekly plugin version sync |
||
|
Amaury Levé
|
a7a6170880 |
Improve template and JSON skill guidance (#1097)
* Improve template and JSON skill guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align template validation guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refine skills from evaluation evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify SDK and parse guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make local SDK setup self-contained Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Improve template eval discrimination Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix template comparison wording Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden multiline command graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align multiline positive graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Collapse dry-run command graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep parse errors out of findings tables Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Stage template fixtures explicitly Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce safe template command boundaries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify template pack target framework Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Cover inline negative command checks Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align template framework selection guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Scope workspace framework grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Require options in template commands Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
github-actions[bot]
|
6cc9e1e1fc | Weekly plugin version sync | ||
|
Amaury Levé
|
47dd914532 |
Add Claude plugin manifests (#1071)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
14d805d08f |
Improve template skill eval coverage (#1068)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
github-actions[bot]
|
62ff166f2b | Weekly plugin version sync | ||
|
YuliiaKovalova
|
b39a041ed0 |
Strengthen dotnet-template-engine skills per cross-family eval (#898) (#904)
* Strengthen dotnet-template-engine skills per cross-family eval (#898) Cross-family skill eval (issue #898) flagged 5 of 6 skills as STRENGTHEN ("mostly ties - too generic/non-prescriptive") and template-instantiation as KEEP-POLISH. Invocation is already ~100% and token overhead is a penalty, so these edits add decisive, opinionated OUTPUT requirements (not volume). All skills remain in the recommended "detailed" tier. - template-validation: require a one-line verdict header + single findings table, and a concrete fix for every ERROR/WARNING. - template-smart-defaults: require a "Defaults applied" log + the exact single dotnet new command line (only flags actually passed). - template-comparison: require a decisive Recommendation verdict and add opinionated decision shortcuts for canonical template pairs. - template-discovery: lead the answer with a ready-to-run command line. - template-authoring: require a complete template.json + a conventions-preserved confirmation table. - template-instantiation: add a lean "match the workspace, then stop" directive to cut the excess turns/tokens flagged for the Opus gap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: remove template-specific AOT claims, narrow authoring validation routing - template-smart-defaults: reword the AOT note, rules row, and pitfall so they no longer claim --aot is unavailable on webapi/webapp (dotnet new webapi --aot exists since .NET 8 and template-instantiation references it). Now say --aot is present only where dotnet new <template> --help lists it, without naming specific templates as unsupported. - template-authoring: narrow the frontmatter DO NOT USE to standalone validation (use template-validation) and align the When-to-Use bullet, so it no longer contradicts the body, which validates as a step during authoring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: keep Defaults-applied Source strictly user|rule The sample row used 'rule (AOT)', which made the Source column look free-form. Set it to 'rule' and moved the AOT context into the Why column, matching the guidance that Source is strictly user or rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: tighten discovery description, make comparison AOT row conditional - template-discovery: simplify the redundant 'two or more templates against each other' phrasing to 'comparing templates side by side' and rewrap so each '(use <skill>)' stays with its clause. - template-comparison: reword the example AOT row so it is conditional on what 'dotnet new <template> --help' exposes instead of asserting webapi has no --aot (which contradicted template-instantiation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: align find/compare routing between frontmatter and body - template-discovery: narrow the frontmatter exclusion to a *detailed* side-by-side comparison (use template-comparison), matching the body which supports light comparison but routes detailed comparisons away. - template-instantiation: split the frontmatter 'finding or comparing' exclusion into finding (template-discovery) vs detailed comparison (template-comparison), and align the When-Not-to-Use bullet accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: mention validation in template-authoring description The description said only 'Guides creation' while the skill validates the template it authors (intro + Step 2). Restored 'creation and validation' and added a USE FOR entry for validating the template.json you are authoring, while keeping standalone validation routed to template-validation in DO NOT USE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: mention high-level compare in discovery; clarify defaults log scope - template-discovery: description now says 'find, inspect, and compare (at a high level)' so it isn't under-scoped vs the body (detailed side-by-side still routes to template-comparison via DO NOT USE). - template-smart-defaults: clarify the 'Defaults applied' log has one row per parameter covering both preserved user values (Source=user) and rule-applied defaults (Source=rule). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: symmetric AOT cell + clarify clean verdict - template-comparison: make the webapp AOT cell reference 'dotnet new webapp --help' so it is symmetric with the webapi cell and unambiguous. - template-validation: the 'Ready to publish' verdict now says '0 errors, 0 warnings' and notes optional suggestions may still apply, instead of 'clean'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: allow line:col location for JSON parse errors Widened the findings-table Location column to 'JSON path or line:col' and added an example parse-error row, so the required table can represent the JSON parse errors that Step 2 asks to report with a line number. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Improve template-discovery resilience: answer before any dotnet new call Eval investigation (PR #904 results, run 29503063336) showed template-discovery scoring 1.0-2.0 on Opus because sessions died with '(no output)' after a 'persistence'/'mutex' error from concurrent dotnet new calls. When the session survived, the same content scored 4.0-4.3. The failing runs invoked the skill then ran 'dotnet new --help' as the first/last action and the mutex error left the turn empty. Make answer-first an imperative rule: write the recommendation and ready-to-run command from the intent mapping BEFORE running any dotnet new command, and never end the turn on a dotnet new call. This makes the answer survive the mutex error (and gives users a faster answer). Skill stays in the 'detailed' token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Iterate on eval quality: shadow-verb shortName warning + discovery resilience From PR #904 eval run 29509140219: - template-validation & template-authoring: add a WARNING tier for shortNames that *shadow* a well-known top-level dotnet verb (build/run/test/publish/pack/clean/ restore). These do NOT technically conflict with dotnet new (correctness kept), but 'dotnet new build' is easily confused with 'dotnet build', so it warrants a warning + a distinctive-name suggestion. This is a genuine authoring footgun and also resolves the 'Validate a template.json file' scenario where the agent was penalized for correctly saying shortName 'build' is fine. - template-discovery: require the pre-CLI answer to be complete and standalone (never end on a 'let me confirm from the CLI…' teaser, which the mutex error was truncating), and add a dry-run-failure fallback (describe expected files from template knowledge) so preview requests never return empty. Skill stays 'detailed'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Rubber-duck follow-up: revert eval-gaming, fix real defects Independent review + empirical checks (dotnet new build/restore/list --help; EvaluateCommand.cs Task.WhenAll) confirmed: - The mutex '(no output)' deaths come from the validator running Baseline+Isolated +Plugin concurrently, not cross-skill parallelism — eval.yaml can't fix it. - build/restore/test do NOT conflict with dotnet new (only real subcommands like 'list' do), so the shortName-shadow WARNING added last commit was appeasing a factually-wrong rubric (and the 'restore' entry contradicted the validation eval). Changes: - Revert the shortName-shadow WARNING in template-validation and template-authoring (the skills were already correct; do not game an incorrect rubric). - template-discovery: fix a real mapping bug — 'windows auth' mapped to SingleOrg; it is --auth Windows. Split azure ad/entra (SingleOrg) from windows auth (Windows). - template-discovery: tighten the dry-run-failure fallback to give only a representative structure and forbid inventing specific parameter values/choices/ file paths (avoids confident hallucination), keeping the 'answer-first, never end on a teaser' resilience. Skill stays in the detailed token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: fix validation findings-table example (real subcommand + line:col) - Use a real dotnet new subcommand ('list') in the ERROR example instead of 'restore' (which does not conflict), matching the corrected shortName rules. - Make the parse-error location example '12:5' match the 'line:col' column header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: consistency fixes across three skills - template-authoring: add ExcludeAssets to the conventions-preserved table (the workflow above lists it, so include it to avoid implying partial carry-over). - template-discovery: add SingleOrg to the example --auth choices so options aren't under-reported. - template-validation: broaden the concrete-fix requirement to allow a specific edit instruction (e.g. 'remove the trailing comma'), matching the parse-error example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Fix incorrect eval rubrics: shortName 'build'/'restore' are not dotnet new conflicts The 'Validate a template.json' (authoring) and 'Validate template with multiple errors' (validation) scenarios used shortName 'build'/'restore' and asserted they conflict with dotnet new. Verified empirically they do NOT (dotnet new build/ restore --help treat them as template-name lookups; only real subcommands like 'list'/'search' intercept). The skills correctly report this, so the wrong rubrics were penalizing correct behavior and causing regressions. Change both fixtures to shortName 'list' (a real dotnet new subcommand that genuinely conflicts) and update the rubric/assertion wording accordingly, so the eval tests correct behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * template-discovery: faithful dry-run output + built-in package guidance Fresh eval (run 29581297999) confirmed the earlier rubric fix worked (authoring Validate 4.3->5.0). Two repeatable clean-output defects remained in template-discovery (seen in surviving Plugin runs, not just mutex deaths): - Preview: the agent summarized/regrouped the dry-run output and invented files (e.g. ReconnectModal), and didn't explain key files. Now: present the actual dry-run file list faithfully (no summarizing/inventing) and give a one-line purpose for key entry points. - Search: for a built-in template the agent never clearly answered 'which package to install'. Now: name the exact package, or say 'no install needed - ships with the SDK' for built-in templates. Compressed the resilience note to keep the skill in the detailed token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Reduce eval overfitting: outcome-focused rubrics + de-cued prompts The overfitting judge flagged moderate overfit on smart-defaults (0.45), comparison (0.34), and instantiation (0.28). Per its per-element analysis (assertions were all 'broad'/fine), make the rubrics/prompts outcome-focused: - Drop every 'The agent used the template-* skill' rubric item (tests activation, not outcome — expect_tools already enforces skill usage as a hard gate). - Reword skill-vocabulary items (e.g. 'native AOT needs a recent AOT-capable TFM', 'smart defaults only fill gaps') and skill-method items ('verify against dotnet new --help', 'inspected via --help', 'logged each applied default', 'verified the project was created') into result-focused criteria that any valid approach satisfies. - De-cue the three smart-defaults prompts that mirrored the skill's 'Required Output B' verbatim ('output the exact single dotnet new command line ... only the flags you will actually pass'); they still elicit a concrete command so the negative assertions remain meaningful. - Drop the blazor prompt's 'Inspect both with dotnet new --help' methodology cue. Assertions and expect_tools are unchanged, so activation and correctness are still verified — the eval now measures outcomes instead of parroting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * template-smart-defaults: defer --help to creation time (trim advice-only cost) Eval run 29586558625 showed the advice-only AOT scenario tie at 4.0 but the skilled run cost 45.7k tokens / 3 turns vs baseline 12.7k / 1 turn — driven entirely by running 'dotnet new worker --help' (feeding ~30k tokens of help output into context) just to advise, which flipped the weighted score negative. Make --help confirmation a creation-time step: for advice-only requests, answer from the rules and note names are confirmed at creation, rather than spending a --help call to advise on well-known parameters. The quality wins come from the decisive output (defaults log + exact command), not from --help, so quality holds while cost drops. Addresses the #898 TRIM-COST concern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: workspace-framework precedence + sync eval.vally.yaml - Precedence conflict (instantiation 'mirror workspace framework' vs smart-defaults 'latest AOT-capable when --framework unset'): treat the workspace framework discovered in Step 2 as an explicit --framework value that smart-defaults must not override; deviate only when incompatible with a requested feature (then flag it). Added a matching note to the smart-defaults AOT rule (a framework pinned by the workspace/global.json counts as set). - Synced eval.vally.yaml (cross-family harness) with the eval.yaml changes that had drifted: outcome-focused rubrics + de-cued prompts for smart-defaults/comparison/ instantiation, and the shortName build/restore -> list fixture+rubric fix for authoring/validation (including the referenced fixture template.json files). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Assert required skill outputs with broad checks (address review) Reviewer flagged that the outcome-focused rubric rewrite left several scenarios without any assertion of the skill's required outputs. Add broad (non-overfitting) assertions in eval.yaml and eval.vally.yaml: - smart-defaults (all 4 scenarios): assert a 'dotnet new' command line is present (output B) and a markdown table is present (the 'Defaults applied' log, output A). - comparison (webapi/webapp, blazor): assert a recommendation is present. - validation (multiple errors): assert a verdict summary (Not ready / N errors) and a findings table are present. - authoring (validate): assert the agent flagged that shortName 'list' conflicts. These are broad presence checks (any valid phrasing passes), so they verify the required output without re-introducing the overfitting the rubric rewrite removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
ca1b47371f |
Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop (#813)
* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop WHY Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a plugin's version directly from its checked-in manifest. With no versioning discipline, a plugin's behavior can change while its advertised version stays flat, so clients never learn to re-pull, and there is no human-readable signal of what changed. We want correct, current versions in the repo with minimal manual work and without bloating the marketplace clone. WHAT - Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin owns a version.json whose pathFilters exclude the generated manifests and the version.json itself, so version height tracks real content changes only. - The computed version is materialized into the checked-in manifests (plugin.json and .codex-plugin/plugin.json) so every consumer reads a current value with no build step on their side. - eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the set of changed plugins from a git diff, computes each version with nbgv (predicting the squash-merge height for PRs), and either reports or stamps. AUTOMATIONS (two, low-touch by design) - /version-bump: an admin/maintainer comments the command on a PR and the affected plugins are stamped on the PR branch. Gated on collaborator permission (admin/write/maintain); forks are rejected before any privileged step. No other PRs are auto-modified. - weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any drift on main, opens/updates a single bot PR, and explains the per-plugin reason. This self-heals anything that merged without a bump. We deliberately did NOT auto-edit contributor PRs or add a noisy advisory comment bot; maintainers stay in control and the signal stays clean. SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro) - Supply chain (High, both models): dotnet tool restore would have honored a nuget.config authored in the PR tree, letting an attacker remap the nbgv package source to a malicious feed and run code in the privileged contents:write context. Mitigated with a trusted eng/version/nuget.config (clear + nuget.org-only + packageSourceMapping), overlaid from main and used via --configfile so PR-supplied configs are ignored. No nuget.config is tracked in the repo today, so this path was genuinely exploitable. - TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather than the mutable branch name; a racing push fails non-fast-forward, which is the safe outcome. - Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement string) so a "$"-bearing version cannot re-expand, plus a strict major.minor.patch guard that throws on a malformed base, leaving manifests untouched. - A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped. VERIFIED End-to-end against a real NBGV git harness: content-scoped predict, base-only bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard, and --configfile restore (exit 0). actionlint passes on both workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review feedback - Add missing plugins/dotnet-test-migration/version.json so it participates in versioning (it was the only plugin without one; manifests are at 0.1.0). - CONTRIBUTING: the two manifests are not byte-identical; say the version is duplicated across two manifest files instead. - weekly-version-sync: include version.json in commit attribution so a base-only bump is explained rather than showing 'no attributable commits'. - Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI failures are diagnosable, while keeping stdout clean for JSON parsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
74c58505e5 |
Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills (#745)
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills Fix inaccurate reserved-shortName guidance, consolidate validation rules into a single skill, expand discovery mappings, add explicit CPM/version steps, and introduce template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .codex-plugin manifest indentation; add evals for new skills Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space indent on the agents line). Add eval.yaml + eval.vally.yaml capability evals for the new template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh - Clarify the reserved shortName set is the current dotnet new subcommands (authoritative source: dotnet new --help); create is verified as a real subcommand (alias behind dotnet new <template>). - template-discovery: note that some mapped short names (maui, winui3, aspire, func, orleans) need workloads/template packages, with fallback to dotnet new list/search. - template-instantiation: keep template versions by default; if refreshing, use dotnet list package --outdated + user confirmation and constrain to same major/minor rather than always latest stable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list - smart-defaults evals: enforce --no-https absence (auth scenario), absence of minimal-API flag (controllers scenario), and no newer --framework TFM when net8.0 is explicitly required, using output_not_contains/output_not_matches. - comparison eval: split the combined (auth|aot|docker|controllers) check into four separate output_matches assertions so partial comparisons fail. - template-validation/authoring: reframe the reserved shortName list as non-exhaustive examples and source the authoritative set from dotnet new --help; drop the specific create-alias assertion in favor of parsing-ambiguity wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make template-comparison evals robust to CI SDK currency Switch the Blazor comparison scenario from blazorserver (absent in the CI SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in .NET 8+, and instruct the agent to inspect each via --help. Update the SKILL.md example reference for currency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CLI-failure resilience guidance to discovery/comparison skills The isolated eval runs failed because the agent ran 'dotnet new <t> --help', hit the template engine's global-mutex/persistence error (common when the command runs concurrently in a sandbox), and then returned no answer at all. Instruct both skills to run 'dotnet new' calls sequentially, retry once on a transient mutex/persistence error, and fall back to the intent/parameter mapping so a concrete answer is always produced instead of empty output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: accurate flags + robust negative-assertion prompts - template-smart-defaults SKILL.md: drop the non-existent --publish-aot flag. Clarify --aot is a dotnet new flag only on templates that expose it (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild PublishAot=true property, not a dotnet new flag. - template-discovery SKILL.md: replace the hardcoded --enable-docker mapping (not a real flag on common templates) with generic 'confirm with --help'. - smart-defaults evals: tighten the negative-assertion prompts to output only the command line and not mention unused flags, so a negated explanation can't trip output_not_contains/output_not_matches. Switch the AOT scenario from webapi to worker (which actually supports --aot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: scope negative checks, de-emphasize stale lists - smart-defaults evals: anchor the negative assertions to the 'dotnet new' command line (same-line regex) instead of whole-output substring/regex, so a flag mentioned only in prose can't fail the test. - template-validation / template-authoring: mark the dotnet new subcommand examples as illustrative/version-dependent and tell readers not to hardcode them; the live 'dotnet new --help' output is canonical. - template-comparison: fix the example table's AOT row — webapi/webapp do not expose a --aot template flag; native AOT is publish-time via PublishAot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
2ab3501158 |
Add .codex-plugin/plugin.json manifests for Codex CLI plugin install (#726)
The Codex CLI requires .codex-plugin/plugin.json as the plugin manifest entry point. Without it, 'codex plugin add' fails with 'missing plugin.json' even though the marketplace listing works. This adds .codex-plugin/plugin.json to all 14 plugin directories, with paths relative to the plugin root per the Codex docs. Also updates the agents marketplace to use dotnet-aspnetcore (per #711 rename) and adds missing dotnet-blazor and dotnet11 entries. Fixes #578 Fixes #724 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
05aeb657e6 | Add license to agent files (#568) | ||
|
YuliiaKovalova
|
59d4485e14 |
Improve dotnet-template-engine skills and eval suite, add template-validation skill (#480)
* Improve dotnet-template-engine eval quality and plugin activation - Replace useless 'Create project with specific framework' test (vanilla=10/10 in all 30 runs, zero skill delta) with harder scenarios that vanilla cannot trivially solve - Expand eval suite from 5 to 10 scenarios to reduce overfitting signal and improve coverage of untested capabilities (dry-run, search, install, template creation from existing project, ambiguous intent resolution) - Add file_not_exists assertion to dry-run scenario to verify projects are not actually created during preview - Add output_matches assertions with specific patterns (e.g. 'individual', 'blazor') instead of relying solely on output_contains - Increase all timeouts from 120s to 180s to reduce MCP cold-start timeouts - Enrich plugin.json description with activation keywords (console app, class library, web API, Blazor, MAUI, framework versions) to fix Plugin mode activation failures (19/21 not-activated events were from one scenario) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add template-validation skill with eval scenarios New skill encoding template.json validation rules: required fields, identity format, shortName conflicts with CLI commands, symbol type/datatype validation, choice parameter default checking, computed/generated symbol completeness, parameter prefix collisions, source condition syntax, post-action requirements, constraint configuration, and tag recommendations. Two eval scenarios: - Validate a template with 8+ deliberate errors (missing identity, CLI- conflicting shortName, invalid defaults, incomplete symbols, missing actionId) - Validate a correct template and confirm it's ready to publish Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - Add ./test-output to dry-run prompt so file_not_exists assertion checks the correct location - Add missing expect_tools: skill to template-authoring validate scenario - Add 'using Xunit;' to seeded test file so the setup project is compilable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten assertions and replace low-delta scenario - Replace low-delta 'worker service search' (vanilla solves trivially) with 'create multi-project solution' (two projects + project reference — vanilla struggles with multi-step orchestration) - Fix gRPC search assertion: replace trivial output_contains 'template' with specific pattern matching gRPC service/package names - Fix validation restore assertion: require 'shortName' context near 'restore' to prevent false pass from post-action description - Fix correct-template assertion: replace fragile output_not_contains with positive assertion that agent mentioned specific fields it checked - Fix create-from-existing assertion: replace pass-anything pattern with assertion requiring template.json / sourceName / .template.config mention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 2) - Change datatype 'integer' to 'int' in eval fixture and SKILL.md to align with actual template engine schema - Tighten correct-template assertion: require unambiguous success signal (e.g. 'no errors', 'template is valid') instead of permissive pattern that matches 'not ready' or 'pass' in other contexts - Replace output_not_contains 'missing required' (fails on 'no missing required fields') with positive assertion checking field names - Remove int/integer dual listing from valid datatypes in SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval failures from CI run 4 scenarios failed; fixes: - Web API + AOT: increase timeout 180s -> 300s (auth scaffolding + NuGet restore needs more time) - Dry-run preview: rephrase prompt to explicitly mention 'template' for better skill activation matching - Multi-project solution: replace with simpler 'create xUnit test project with net8.0' — the multi-project scenario gave vanilla a HIGHER score (4.7) than skilled (4.3) because the skill added overhead without helping - Create-from-existing: increase timeout 180s -> 300s (template generation burned 256K tokens and timed out) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dishonest eval adjustments - Revert dry-run prompt to natural language — don't coach activation by injecting 'template' keyword. If skill doesn't activate, that's a real signal about activation quality. - Replace recycled 'xUnit + net8.0' test (same ceiling problem as the classlib + net9.0 test we removed) with 'Blazor + interactive rendering' which tests parameter knowledge vanilla is less likely to have (render mode selection, blazor vs blazorserver vs blazorwasm template choice). - Keep timeout increases (300s for web API + auth, 300s for create-from- existing) as these address real infrastructure constraints. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move dry-run scenario from template-instantiation to template-discovery The dry-run scenario failed with 'NOT ACTIVATED' because the platform couldn't match a preview/dry-run prompt to the template-instantiation skill. This is correct behavior — previewing what files a template produces is a discovery operation, not instantiation. The template-discovery SKILL.md explicitly covers dry-run in Step 4: 'Use dotnet new <template> --dry-run to show what files a template would create without writing anything to disk.' This is an objective reclassification, not prompt tuning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill content causing eval failures template-instantiation: Add parameter reference table with common parameter combinations (auth values, interactivity modes, AOT flag) for webapi, blazor, grpc, worker templates. The web API + AOT scenario scored 2.0 for both vanilla and skilled because the skill mentioned --auth and --aot but never explained what values they accept. Now the agent gets actionable knowledge. template-authoring: Trim verbose content to reduce token overhead. The create-from-existing scenario burned 256K tokens (vs 185K vanilla) and timed out because the agent read packaging/distribution steps it didn't need. Condensed Step 1 example, removed Step 5 (packaging), compressed Step 4 (testing). Reduces skill size by ~30%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 3) - Web API+AOT: replace output_matches for AOT with file_contains checking for PublishAot in the generated .csproj — proves the parameter was actually passed, not just mentioned in text - Validation multi-error: add assertions for bool default (enableLogging/ yes), int default (maxRetries/abc), computed missing value, generated missing generator — now enforces 7 of 8 rubric items via assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 4) - Fix YAML double-quote escaping for regex word boundaries - Fix worker template example to include --aot flag - Fix webapi example to show --auth Individual --aot together - Fix grammar: Available Symbol Generators Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove web API+AOT and Blazor eval scenarios Web API+AOT consistently degraded quality (run 1: 2.0->2.0, run 2: 3.0->2.7). The skill adds reading overhead on execution tasks without improving the result. Blazor hit ceiling (5.0->5.0) — vanilla already creates Blazor apps perfectly, so the scenario contributes zero signal. template-instantiation now has 1 honest scenario (console app, +1.0 delta). The skill's value is modest for execution tasks; the PR's strength is in discovery (+2.3 to +3.7), validation (+1.7 to +2.0), and authoring (+2.0) where knowledge matters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 5) - Tighten bool/int validation assertions to require explicit error wording (invalid, not valid, must be) alongside field/value — prevents passing when agent just echoes the JSON without diagnosing the issue - Add actionId assertion to enforce detection of missing postAction actionId (was in rubric but not in assertions) - Use single-quoted YAML for patterns with backslash sequences Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
a895c226b3 |
Fix agents field in plugin.json to use explicit file paths (#419)
* Fix agents field in plugin.json to use explicit file paths Claude Code's manifest validator rejects directory paths in the agents field. Replace directory references with explicit .agent.md file paths in all affected plugins (dotnet-diag, dotnet-msbuild, dotnet-template-engine). Fixes https://github.com/dotnet/skills/issues/416 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Validate that agents paths are explicit files, not directories Claude Code's manifest validator rejects directory paths in the agents field. Update PluginValidator to error when an agent path resolves to a directory, guiding authors to use explicit .agent.md file paths instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
6face98ff6 |
Change agents and skills fields in plugin.json to array format for Claude Code CLI compatibility (#407)
* Initial plan * Remove agents string field from plugin.json and handle array format in parser Claude Code CLI validates `agents` as array-of-strings per its plugin manifest schema. Our plugin.json files had `"agents": "./agents/"` (a string), which caused "agents: Invalid input" validation errors on install. Remove the field from the 3 affected plugin.json files — agents are still discovered by convention from the `agents/` directory. Also make ParsePluginJson resilient to both string and array formats for forward-compatibility. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Remove `agents` string field from plugin.json for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change agents field to array format in plugin.json, update validator to read array-first with string fallback Plugin.json files now declare agents as an array of file paths: "agents": ["./agents/foo.agent.md", "./agents/bar.agent.md"] ParsePluginJson reads the array first (preferred, Claude Code schema), falls back to a string path (legacy). PluginInfo gains an AgentPaths field. Validator and discovery use the array when present, else fall back to directory-based convention. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Simplify agents to directory-path array form ["./agents/"], handle directories in array entries Per Copilot CLI spec, agents field values are "Path(s) to agent directories" — no need to list individual files. Use ["./agents/"] to mirror how skills uses "./skills/". Updated validator and discovery to handle both directory and file entries in the array. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Convert skills field to array format in plugin.json, update validator to handle both forms Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Align AgentPaths null checks to use { Count: > 0 } pattern for consistency Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change `agents` and `skills` fields in plugin.json to array format for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Consolidate SkillsPath/SkillPaths and AgentsPath/AgentPaths into single array fields Remove dual string/array fields from PluginInfo. The parser now normalizes legacy string values into single-element arrays, so all downstream code uses only SkillPaths and AgentPaths (IReadOnlyList<string>). This eliminates all array-vs-string fallback branching. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
dcba4f3d8e |
Add dotnet-template-engine plugin (#341)
* Add dotnet-template-engine plugin Add template-engine skills for .NET project scaffolding powered by the DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent: Skills: - template-discovery: find, inspect, and compare templates with intent resolution (70+ keyword mappings) - template-instantiation: create projects with CPM adaptation, latest NuGet versions, parameter validation, and multi-template composition - template-authoring: create templates from existing projects and validate template.json for authoring issues Agent: - template-engine: expert agent that routes to the appropriate skill based on user intent Also adds CODEOWNERS entries, marketplace.json registration, eval tests, and .mcp.json for DotnetTemplateMCP server configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: mcpServers in plugin.json, expect_tools, path fix - Move MCP server config into plugin.json under mcpServers so SkillValidator discovers the template_* tools during evaluation runs - Add expect_tools constraints to all eval scenarios to verify actual MCP tool invocation instead of relying on rubric text alone - Fix frontmatter path: template.config → .template.config (leading dot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update DotnetTemplateMCP to v1.2.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include 'dotnet new' in agent description for discoverability Address review feedback from danmoseley: the frontmatter description is how the caller decides whether to load the agent tokens, so it needs to mention 'dotnet new' as a trigger keyword. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove MCP server configuration and references - Delete .mcp.json MCP server config file - Remove mcpServers section from plugin.json - Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions - Skills and eval files remain unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove ghost tool references and add CLI fallback guidance - Remove references to 4 non-existent tools: template_compose, template_suggest_parameters, template_validate, solution_analyze - Replace with real alternatives (template_inspect, manual review, sequential template_instantiate, direct file inspection) - Add CLI fallback guidance so LLMs degrade gracefully when template tools are unavailable - Fix authoring eval to not expect template_validate tool Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: CODEOWNERS naming, claude marketplace, README - Rename CODEOWNERS header to dotnet-template-engine for consistency - Mirror plugin entry in .claude-plugin/marketplace.json - Add plugin to README What's Included table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json - Updated template-discovery eval.yaml rubric to match expect_tools (removed 'or template_search' since expect_tools enforces template_from_intent) - Made .claude-plugin/marketplace.json byte-for-byte identical to .github/plugin/marketplace.json (removed extra trailing newline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: strengthen eval assertions and fix rubrics - Add exit_success assertions to all eval scenarios - Add file_exists and file_contains assertions for template-instantiation - Add output_contains for shortName and datatype in template-authoring - Fix rubrics to reference actual tools (skill + dotnet new CLI) instead of non-existent MCP tools (template_from_intent, template_inspect, template_instantiate) which caused the judge to penalize quality scores - Update expect_tools from non-existent MCP tools to 'skill' - Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins - Remove extra blank line in README.md after plugin table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools The SKILL.md files and agent.md referenced MCP tools (template_from_intent, template_inspect, template_instantiate, template_dry_run, etc.) that are not available at runtime since no MCP server is declared in plugin.json. This caused the agent to waste tokens attempting to find these tools before falling back to dotnet new CLI commands. Changes: - template-discovery/SKILL.md: Replace MCP tool calls with dotnet new search, dotnet new list, dotnet new --help, and dotnet new --dry-run. Add intent-to-template keyword mapping table. - template-instantiation/SKILL.md: Replace template_instantiate with dotnet new CLI commands. Add explicit CPM adaptation steps and multi-project composition with dotnet sln/dotnet add reference. - template-authoring/SKILL.md: Replace template_create_from_existing with manual template.json creation guidance including example JSON. Replace template_dry_run/template_instantiate with CLI equivalents. - template-engine.agent.md: Replace Available Tools table with CLI Commands Reference. Update all workflow steps to use CLI commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix file assertion paths: remove ./ prefix for glob matcher compatibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |