skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
653 Commits 131 Branches 2 Tags
62ff166f2bf806137fe121b5328d68bc7db4a743
Commit Graph

10 Commits

Author SHA1 Message Date
github-actions[bot] 62ff166f2b Weekly plugin version sync 2026-08-03 10:19:29 +00:00
YuliiaKovalova b39a041ed0 Strengthen dotnet-template-engine skills per cross-family eval (#898) (#904) 
* Strengthen dotnet-template-engine skills per cross-family eval (#898)

Cross-family skill eval (issue #898) flagged 5 of 6 skills as STRENGTHEN
("mostly ties - too generic/non-prescriptive") and template-instantiation
as KEEP-POLISH. Invocation is already ~100% and token overhead is a
penalty, so these edits add decisive, opinionated OUTPUT requirements
(not volume). All skills remain in the recommended "detailed" tier.

- template-validation: require a one-line verdict header + single findings
  table, and a concrete fix for every ERROR/WARNING.
- template-smart-defaults: require a "Defaults applied" log + the exact
  single dotnet new command line (only flags actually passed).
- template-comparison: require a decisive Recommendation verdict and add
  opinionated decision shortcuts for canonical template pairs.
- template-discovery: lead the answer with a ready-to-run command line.
- template-authoring: require a complete template.json + a
  conventions-preserved confirmation table.
- template-instantiation: add a lean "match the workspace, then stop"
  directive to cut the excess turns/tokens flagged for the Opus gap.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: remove template-specific AOT claims, narrow authoring validation routing

- template-smart-defaults: reword the AOT note, rules row, and pitfall so they
  no longer claim --aot is unavailable on webapi/webapp (dotnet new webapi --aot
  exists since .NET 8 and template-instantiation references it). Now say --aot is
  present only where dotnet new <template> --help lists it, without naming
  specific templates as unsupported.
- template-authoring: narrow the frontmatter DO NOT USE to standalone validation
  (use template-validation) and align the When-to-Use bullet, so it no longer
  contradicts the body, which validates as a step during authoring.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: keep Defaults-applied Source strictly user|rule

The sample row used 'rule (AOT)', which made the Source column look free-form.
Set it to 'rule' and moved the AOT context into the Why column, matching the
guidance that Source is strictly user or rule.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: tighten discovery description, make comparison AOT row conditional

- template-discovery: simplify the redundant 'two or more templates against each
  other' phrasing to 'comparing templates side by side' and rewrap so each
  '(use <skill>)' stays with its clause.
- template-comparison: reword the example AOT row so it is conditional on what
  'dotnet new <template> --help' exposes instead of asserting webapi has no --aot
  (which contradicted template-instantiation).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: align find/compare routing between frontmatter and body

- template-discovery: narrow the frontmatter exclusion to a *detailed*
  side-by-side comparison (use template-comparison), matching the body which
  supports light comparison but routes detailed comparisons away.
- template-instantiation: split the frontmatter 'finding or comparing' exclusion
  into finding (template-discovery) vs detailed comparison (template-comparison),
  and align the When-Not-to-Use bullet accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: mention validation in template-authoring description

The description said only 'Guides creation' while the skill validates the
template it authors (intro + Step 2). Restored 'creation and validation' and
added a USE FOR entry for validating the template.json you are authoring, while
keeping standalone validation routed to template-validation in DO NOT USE.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: mention high-level compare in discovery; clarify defaults log scope

- template-discovery: description now says 'find, inspect, and compare (at a high
  level)' so it isn't under-scoped vs the body (detailed side-by-side still routes
  to template-comparison via DO NOT USE).
- template-smart-defaults: clarify the 'Defaults applied' log has one row per
  parameter covering both preserved user values (Source=user) and rule-applied
  defaults (Source=rule).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: symmetric AOT cell + clarify clean verdict

- template-comparison: make the webapp AOT cell reference 'dotnet new webapp
  --help' so it is symmetric with the webapi cell and unambiguous.
- template-validation: the 'Ready to publish' verdict now says '0 errors, 0
  warnings' and notes optional suggestions may still apply, instead of 'clean'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: allow line:col location for JSON parse errors

Widened the findings-table Location column to 'JSON path or line:col' and added
an example parse-error row, so the required table can represent the JSON parse
errors that Step 2 asks to report with a line number.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Improve template-discovery resilience: answer before any dotnet new call

Eval investigation (PR #904 results, run 29503063336) showed template-discovery
scoring 1.0-2.0 on Opus because sessions died with '(no output)' after a
'persistence'/'mutex' error from concurrent dotnet new calls. When the session
survived, the same content scored 4.0-4.3. The failing runs invoked the skill
then ran 'dotnet new --help' as the first/last action and the mutex error left
the turn empty.

Make answer-first an imperative rule: write the recommendation and ready-to-run
command from the intent mapping BEFORE running any dotnet new command, and never
end the turn on a dotnet new call. This makes the answer survive the mutex error
(and gives users a faster answer). Skill stays in the 'detailed' token tier.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Iterate on eval quality: shadow-verb shortName warning + discovery resilience

From PR #904 eval run 29509140219:

- template-validation & template-authoring: add a WARNING tier for shortNames that
  *shadow* a well-known top-level dotnet verb (build/run/test/publish/pack/clean/
  restore). These do NOT technically conflict with dotnet new (correctness kept),
  but 'dotnet new build' is easily confused with 'dotnet build', so it warrants a
  warning + a distinctive-name suggestion. This is a genuine authoring footgun and
  also resolves the 'Validate a template.json file' scenario where the agent was
  penalized for correctly saying shortName 'build' is fine.
- template-discovery: require the pre-CLI answer to be complete and standalone
  (never end on a 'let me confirm from the CLI…' teaser, which the mutex error was
  truncating), and add a dry-run-failure fallback (describe expected files from
  template knowledge) so preview requests never return empty. Skill stays 'detailed'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Rubber-duck follow-up: revert eval-gaming, fix real defects

Independent review + empirical checks (dotnet new build/restore/list --help;
EvaluateCommand.cs Task.WhenAll) confirmed:
- The mutex '(no output)' deaths come from the validator running Baseline+Isolated
  +Plugin concurrently, not cross-skill parallelism — eval.yaml can't fix it.
- build/restore/test do NOT conflict with dotnet new (only real subcommands like
  'list' do), so the shortName-shadow WARNING added last commit was appeasing a
  factually-wrong rubric (and the 'restore' entry contradicted the validation eval).

Changes:
- Revert the shortName-shadow WARNING in template-validation and template-authoring
  (the skills were already correct; do not game an incorrect rubric).
- template-discovery: fix a real mapping bug — 'windows auth' mapped to SingleOrg;
  it is --auth Windows. Split azure ad/entra (SingleOrg) from windows auth (Windows).
- template-discovery: tighten the dry-run-failure fallback to give only a
  representative structure and forbid inventing specific parameter values/choices/
  file paths (avoids confident hallucination), keeping the 'answer-first, never end
  on a teaser' resilience. Skill stays in the detailed token tier.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: fix validation findings-table example (real subcommand + line:col)

- Use a real dotnet new subcommand ('list') in the ERROR example instead of
  'restore' (which does not conflict), matching the corrected shortName rules.
- Make the parse-error location example '12:5' match the 'line:col' column header.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: consistency fixes across three skills

- template-authoring: add ExcludeAssets to the conventions-preserved table (the
  workflow above lists it, so include it to avoid implying partial carry-over).
- template-discovery: add SingleOrg to the example --auth choices so options
  aren't under-reported.
- template-validation: broaden the concrete-fix requirement to allow a specific
  edit instruction (e.g. 'remove the trailing comma'), matching the parse-error
  example.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Fix incorrect eval rubrics: shortName 'build'/'restore' are not dotnet new conflicts

The 'Validate a template.json' (authoring) and 'Validate template with multiple
errors' (validation) scenarios used shortName 'build'/'restore' and asserted they
conflict with dotnet new. Verified empirically they do NOT (dotnet new build/
restore --help treat them as template-name lookups; only real subcommands like
'list'/'search' intercept). The skills correctly report this, so the wrong rubrics
were penalizing correct behavior and causing regressions.

Change both fixtures to shortName 'list' (a real dotnet new subcommand that genuinely
conflicts) and update the rubric/assertion wording accordingly, so the eval tests
correct behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* template-discovery: faithful dry-run output + built-in package guidance

Fresh eval (run 29581297999) confirmed the earlier rubric fix worked (authoring
Validate 4.3->5.0). Two repeatable clean-output defects remained in template-discovery
(seen in surviving Plugin runs, not just mutex deaths):
- Preview: the agent summarized/regrouped the dry-run output and invented files
  (e.g. ReconnectModal), and didn't explain key files. Now: present the actual
  dry-run file list faithfully (no summarizing/inventing) and give a one-line
  purpose for key entry points.
- Search: for a built-in template the agent never clearly answered 'which package
  to install'. Now: name the exact package, or say 'no install needed - ships with
  the SDK' for built-in templates.

Compressed the resilience note to keep the skill in the detailed token tier.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Reduce eval overfitting: outcome-focused rubrics + de-cued prompts

The overfitting judge flagged moderate overfit on smart-defaults (0.45),
comparison (0.34), and instantiation (0.28). Per its per-element analysis
(assertions were all 'broad'/fine), make the rubrics/prompts outcome-focused:

- Drop every 'The agent used the template-* skill' rubric item (tests activation,
  not outcome — expect_tools already enforces skill usage as a hard gate).
- Reword skill-vocabulary items (e.g. 'native AOT needs a recent AOT-capable TFM',
  'smart defaults only fill gaps') and skill-method items ('verify against
  dotnet new --help', 'inspected via --help', 'logged each applied default',
  'verified the project was created') into result-focused criteria that any valid
  approach satisfies.
- De-cue the three smart-defaults prompts that mirrored the skill's 'Required
  Output B' verbatim ('output the exact single dotnet new command line ... only
  the flags you will actually pass'); they still elicit a concrete command so the
  negative assertions remain meaningful.
- Drop the blazor prompt's 'Inspect both with dotnet new --help' methodology cue.

Assertions and expect_tools are unchanged, so activation and correctness are still
verified — the eval now measures outcomes instead of parroting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* template-smart-defaults: defer --help to creation time (trim advice-only cost)

Eval run 29586558625 showed the advice-only AOT scenario tie at 4.0 but the
skilled run cost 45.7k tokens / 3 turns vs baseline 12.7k / 1 turn — driven
entirely by running 'dotnet new worker --help' (feeding ~30k tokens of help
output into context) just to advise, which flipped the weighted score negative.

Make --help confirmation a creation-time step: for advice-only requests, answer
from the rules and note names are confirmed at creation, rather than spending a
--help call to advise on well-known parameters. The quality wins come from the
decisive output (defaults log + exact command), not from --help, so quality holds
while cost drops. Addresses the #898 TRIM-COST concern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Address PR review: workspace-framework precedence + sync eval.vally.yaml

- Precedence conflict (instantiation 'mirror workspace framework' vs smart-defaults
  'latest AOT-capable when --framework unset'): treat the workspace framework
  discovered in Step 2 as an explicit --framework value that smart-defaults must
  not override; deviate only when incompatible with a requested feature (then flag
  it). Added a matching note to the smart-defaults AOT rule (a framework pinned by
  the workspace/global.json counts as set).
- Synced eval.vally.yaml (cross-family harness) with the eval.yaml changes that had
  drifted: outcome-focused rubrics + de-cued prompts for smart-defaults/comparison/
  instantiation, and the shortName build/restore -> list fixture+rubric fix for
  authoring/validation (including the referenced fixture template.json files).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

* Assert required skill outputs with broad checks (address review)

Reviewer flagged that the outcome-focused rubric rewrite left several scenarios
without any assertion of the skill's required outputs. Add broad (non-overfitting)
assertions in eval.yaml and eval.vally.yaml:

- smart-defaults (all 4 scenarios): assert a 'dotnet new' command line is present
  (output B) and a markdown table is present (the 'Defaults applied' log, output A).
- comparison (webapi/webapp, blazor): assert a recommendation is present.
- validation (multiple errors): assert a verdict summary (Not ready / N errors) and
  a findings table are present.
- authoring (validate): assert the agent flagged that shortName 'list' conflicts.

These are broad presence checks (any valid phrasing passes), so they verify the
required output without re-introducing the overfitting the rubric rewrite removed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-07-20 16:43:57 +02:00
Abhitej John ca1b47371f Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop (#813) 
* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop

WHY
Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a
plugin's version directly from its checked-in manifest. With no versioning
discipline, a plugin's behavior can change while its advertised version stays
flat, so clients never learn to re-pull, and there is no human-readable signal
of what changed. We want correct, current versions in the repo with minimal
manual work and without bloating the marketplace clone.

WHAT
- Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin
  owns a version.json whose pathFilters exclude the generated manifests and the
  version.json itself, so version height tracks real content changes only.
- The computed version is materialized into the checked-in manifests
  (plugin.json and .codex-plugin/plugin.json) so every consumer reads a current
  value with no build step on their side.
- eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the
  set of changed plugins from a git diff, computes each version with nbgv
  (predicting the squash-merge height for PRs), and either reports or stamps.

AUTOMATIONS (two, low-touch by design)
- /version-bump: an admin/maintainer comments the command on a PR and the
  affected plugins are stamped on the PR branch. Gated on collaborator
  permission (admin/write/maintain); forks are rejected before any privileged
  step. No other PRs are auto-modified.
- weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any
  drift on main, opens/updates a single bot PR, and explains the per-plugin
  reason. This self-heals anything that merged without a bump.

We deliberately did NOT auto-edit contributor PRs or add a noisy advisory
comment bot; maintainers stay in control and the signal stays clean.

SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro)
- Supply chain (High, both models): dotnet tool restore would have honored a
  nuget.config authored in the PR tree, letting an attacker remap the nbgv
  package source to a malicious feed and run code in the privileged
  contents:write context. Mitigated with a trusted eng/version/nuget.config
  (clear + nuget.org-only + packageSourceMapping), overlaid from main and used
  via --configfile so PR-supplied configs are ignored. No nuget.config is
  tracked in the repo today, so this path was genuinely exploitable.
- TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather
  than the mutable branch name; a racing push fails non-fast-forward, which is
  the safe outcome.
- Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement
  string) so a "$"-bearing version cannot re-expand, plus a strict
  major.minor.patch guard that throws on a malformed base, leaving manifests
  untouched.
- A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped.

VERIFIED
End-to-end against a real NBGV git harness: content-scoped predict, base-only
bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard,
and --configfile restore (exit 0). actionlint passes on both workflows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Copilot review feedback

- Add missing plugins/dotnet-test-migration/version.json so it participates
  in versioning (it was the only plugin without one; manifests are at 0.1.0).
- CONTRIBUTING: the two manifests are not byte-identical; say the version is
  duplicated across two manifest files instead.
- weekly-version-sync: include version.json in commit attribution so a
  base-only bump is explained rather than showing 'no attributable commits'.
- Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI
  failures are diagnosable, while keeping stdout clean for JSON parsing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-07-13 10:07:45 -07:00
YuliiaKovalova 74c58505e5 Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills (#745) 
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills

Fix inaccurate reserved-shortName guidance, consolidate validation rules into
a single skill, expand discovery mappings, add explicit CPM/version steps, and
introduce template-comparison and template-smart-defaults skills.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix .codex-plugin manifest indentation; add evals for new skills

Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space
indent on the agents line). Add eval.yaml + eval.vally.yaml capability
evals for the new template-comparison and template-smart-defaults skills.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh

- Clarify the reserved shortName set is the current dotnet new subcommands
  (authoritative source: dotnet new --help); create is verified as a real
  subcommand (alias behind dotnet new <template>).
- template-discovery: note that some mapped short names (maui, winui3, aspire,
  func, orleans) need workloads/template packages, with fallback to
  dotnet new list/search.
- template-instantiation: keep template versions by default; if refreshing,
  use dotnet list package --outdated + user confirmation and constrain to
  same major/minor rather than always latest stable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list

- smart-defaults evals: enforce --no-https absence (auth scenario), absence of
  minimal-API flag (controllers scenario), and no newer --framework TFM when
  net8.0 is explicitly required, using output_not_contains/output_not_matches.
- comparison eval: split the combined (auth|aot|docker|controllers) check into
  four separate output_matches assertions so partial comparisons fail.
- template-validation/authoring: reframe the reserved shortName list as
  non-exhaustive examples and source the authoritative set from dotnet new --help;
  drop the specific create-alias assertion in favor of parsing-ambiguity wording.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Make template-comparison evals robust to CI SDK currency

Switch the Blazor comparison scenario from blazorserver (absent in the CI
SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in
.NET 8+, and instruct the agent to inspect each via --help. Update the
SKILL.md example reference for currency.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add CLI-failure resilience guidance to discovery/comparison skills

The isolated eval runs failed because the agent ran 'dotnet new <t> --help',
hit the template engine's global-mutex/persistence error (common when the
command runs concurrently in a sandbox), and then returned no answer at all.

Instruct both skills to run 'dotnet new' calls sequentially, retry once on a
transient mutex/persistence error, and fall back to the intent/parameter
mapping so a concrete answer is always produced instead of empty output.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review: accurate flags + robust negative-assertion prompts

- template-smart-defaults SKILL.md: drop the non-existent --publish-aot
  flag. Clarify --aot is a dotnet new flag only on templates that expose it
  (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild
  PublishAot=true property, not a dotnet new flag.
- template-discovery SKILL.md: replace the hardcoded --enable-docker mapping
  (not a real flag on common templates) with generic 'confirm with --help'.
- smart-defaults evals: tighten the negative-assertion prompts to output only
  the command line and not mention unused flags, so a negated explanation
  can't trip output_not_contains/output_not_matches. Switch the AOT scenario
  from webapi to worker (which actually supports --aot).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review round 3: scope negative checks, de-emphasize stale lists

- smart-defaults evals: anchor the negative assertions to the 'dotnet new'
  command line (same-line regex) instead of whole-output substring/regex, so a
  flag mentioned only in prose can't fail the test.
- template-validation / template-authoring: mark the dotnet new subcommand
  examples as illustrative/version-dependent and tell readers not to hardcode
  them; the live 'dotnet new --help' output is canonical.
- template-comparison: fix the example table's AOT row — webapi/webapp do not
  expose a --aot template flag; native AOT is publish-time via PublishAot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-12 15:15:56 +02:00
Abhitej John 2ab3501158 Add .codex-plugin/plugin.json manifests for Codex CLI plugin install (#726) 
The Codex CLI requires .codex-plugin/plugin.json as the plugin manifest
entry point. Without it, 'codex plugin add' fails with 'missing plugin.json'
even though the marketplace listing works.

This adds .codex-plugin/plugin.json to all 14 plugin directories, with paths
relative to the plugin root per the Codex docs. Also updates the agents
marketplace to use dotnet-aspnetcore (per #711 rename) and adds missing
dotnet-blazor and dotnet11 entries.

Fixes #578
Fixes #724

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-09 09:29:42 -07:00
Jan Krivanek 05aeb657e6 Add license to agent files (#568) 2026-04-21 12:57:18 +00:00
YuliiaKovalova 59d4485e14 Improve dotnet-template-engine skills and eval suite, add template-validation skill (#480) 
* Improve dotnet-template-engine eval quality and plugin activation

- Replace useless 'Create project with specific framework' test (vanilla=10/10
  in all 30 runs, zero skill delta) with harder scenarios that vanilla cannot
  trivially solve
- Expand eval suite from 5 to 10 scenarios to reduce overfitting signal and
  improve coverage of untested capabilities (dry-run, search, install,
  template creation from existing project, ambiguous intent resolution)
- Add file_not_exists assertion to dry-run scenario to verify projects are
  not actually created during preview
- Add output_matches assertions with specific patterns (e.g. 'individual',
  'blazor') instead of relying solely on output_contains
- Increase all timeouts from 120s to 180s to reduce MCP cold-start timeouts
- Enrich plugin.json description with activation keywords (console app, class
  library, web API, Blazor, MAUI, framework versions) to fix Plugin mode
  activation failures (19/21 not-activated events were from one scenario)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add template-validation skill with eval scenarios

New skill encoding template.json validation rules: required fields, identity
format, shortName conflicts with CLI commands, symbol type/datatype validation,
choice parameter default checking, computed/generated symbol completeness,
parameter prefix collisions, source condition syntax, post-action requirements,
constraint configuration, and tag recommendations.

Two eval scenarios:
- Validate a template with 8+ deliberate errors (missing identity, CLI-
  conflicting shortName, invalid defaults, incomplete symbols, missing actionId)
- Validate a correct template and confirm it's ready to publish

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments

- Add ./test-output to dry-run prompt so file_not_exists assertion checks
  the correct location
- Add missing expect_tools: skill to template-authoring validate scenario
- Add 'using Xunit;' to seeded test file so the setup project is compilable

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tighten assertions and replace low-delta scenario

- Replace low-delta 'worker service search' (vanilla solves trivially) with
  'create multi-project solution' (two projects + project reference — vanilla
  struggles with multi-step orchestration)
- Fix gRPC search assertion: replace trivial output_contains 'template' with
  specific pattern matching gRPC service/package names
- Fix validation restore assertion: require 'shortName' context near 'restore'
  to prevent false pass from post-action description
- Fix correct-template assertion: replace fragile output_not_contains with
  positive assertion that agent mentioned specific fields it checked
- Fix create-from-existing assertion: replace pass-anything pattern with
  assertion requiring template.json / sourceName / .template.config mention

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments (round 2)

- Change datatype 'integer' to 'int' in eval fixture and SKILL.md to align
  with actual template engine schema
- Tighten correct-template assertion: require unambiguous success signal
  (e.g. 'no errors', 'template is valid') instead of permissive pattern
  that matches 'not ready' or 'pass' in other contexts
- Replace output_not_contains 'missing required' (fails on 'no missing
  required fields') with positive assertion checking field names
- Remove int/integer dual listing from valid datatypes in SKILL.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix eval failures from CI run

4 scenarios failed; fixes:

- Web API + AOT: increase timeout 180s -> 300s (auth scaffolding + NuGet
  restore needs more time)
- Dry-run preview: rephrase prompt to explicitly mention 'template' for
  better skill activation matching
- Multi-project solution: replace with simpler 'create xUnit test project
  with net8.0' — the multi-project scenario gave vanilla a HIGHER score
  (4.7) than skilled (4.3) because the skill added overhead without helping
- Create-from-existing: increase timeout 180s -> 300s (template generation
  burned 256K tokens and timed out)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix dishonest eval adjustments

- Revert dry-run prompt to natural language — don't coach activation by
  injecting 'template' keyword. If skill doesn't activate, that's a real
  signal about activation quality.
- Replace recycled 'xUnit + net8.0' test (same ceiling problem as the
  classlib + net9.0 test we removed) with 'Blazor + interactive rendering'
  which tests parameter knowledge vanilla is less likely to have (render
  mode selection, blazor vs blazorserver vs blazorwasm template choice).
- Keep timeout increases (300s for web API + auth, 300s for create-from-
  existing) as these address real infrastructure constraints.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move dry-run scenario from template-instantiation to template-discovery

The dry-run scenario failed with 'NOT ACTIVATED' because the platform
couldn't match a preview/dry-run prompt to the template-instantiation
skill. This is correct behavior — previewing what files a template
produces is a discovery operation, not instantiation.

The template-discovery SKILL.md explicitly covers dry-run in Step 4:
'Use dotnet new <template> --dry-run to show what files a template
would create without writing anything to disk.'

This is an objective reclassification, not prompt tuning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix skill content causing eval failures

template-instantiation: Add parameter reference table with common parameter
combinations (auth values, interactivity modes, AOT flag) for webapi, blazor,
grpc, worker templates. The web API + AOT scenario scored 2.0 for both
vanilla and skilled because the skill mentioned --auth and --aot but never
explained what values they accept. Now the agent gets actionable knowledge.

template-authoring: Trim verbose content to reduce token overhead. The
create-from-existing scenario burned 256K tokens (vs 185K vanilla) and
timed out because the agent read packaging/distribution steps it didn't
need. Condensed Step 1 example, removed Step 5 (packaging), compressed
Step 4 (testing). Reduces skill size by ~30%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments (round 3)

- Web API+AOT: replace output_matches for AOT with file_contains checking
  for PublishAot in the generated .csproj — proves the parameter was
  actually passed, not just mentioned in text
- Validation multi-error: add assertions for bool default (enableLogging/
  yes), int default (maxRetries/abc), computed missing value, generated
  missing generator — now enforces 7 of 8 rubric items via assertions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments (round 4)

- Fix YAML double-quote escaping for regex word boundaries
- Fix worker template example to include --aot flag
- Fix webapi example to show --auth Individual --aot together
- Fix grammar: Available Symbol Generators

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove web API+AOT and Blazor eval scenarios

Web API+AOT consistently degraded quality (run 1: 2.0->2.0, run 2:
3.0->2.7). The skill adds reading overhead on execution tasks without
improving the result.

Blazor hit ceiling (5.0->5.0) — vanilla already creates Blazor apps
perfectly, so the scenario contributes zero signal.

template-instantiation now has 1 honest scenario (console app, +1.0
delta). The skill's value is modest for execution tasks; the PR's
strength is in discovery (+2.3 to +3.7), validation (+1.7 to +2.0),
and authoring (+2.0) where knowledge matters.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments (round 5)

- Tighten bool/int validation assertions to require explicit error
  wording (invalid, not valid, must be) alongside field/value — prevents
  passing when agent just echoes the JSON without diagnosing the issue
- Add actionId assertion to enforce detection of missing postAction
  actionId (was in rubric but not in assertions)
- Use single-quoted YAML for patterns with backslash sequences

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-01 14:16:24 +02:00
Viktor Hofer a895c226b3 Fix agents field in plugin.json to use explicit file paths (#419) 
* Fix agents field in plugin.json to use explicit file paths

Claude Code's manifest validator rejects directory paths in the agents
field. Replace directory references with explicit .agent.md file paths
in all affected plugins (dotnet-diag, dotnet-msbuild, dotnet-template-engine).

Fixes https://github.com/dotnet/skills/issues/416

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Validate that agents paths are explicit files, not directories

Claude Code's manifest validator rejects directory paths in the agents
field. Update PluginValidator to error when an agent path resolves to a
directory, guiding authors to use explicit .agent.md file paths instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-23 08:36:20 +00:00
Copilot 6face98ff6 Change agents and skills fields in plugin.json to array format for Claude Code CLI compatibility (#407) 
* Initial plan

* Remove agents string field from plugin.json and handle array format in parser

Claude Code CLI validates `agents` as array-of-strings per its plugin manifest
schema. Our plugin.json files had `"agents": "./agents/"` (a string), which
caused "agents: Invalid input" validation errors on install.

Remove the field from the 3 affected plugin.json files — agents are still
discovered by convention from the `agents/` directory. Also make ParsePluginJson
resilient to both string and array formats for forward-compatibility.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Remove `agents` string field from plugin.json for Claude Code CLI compatibility

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Change agents field to array format in plugin.json, update validator to read array-first with string fallback

Plugin.json files now declare agents as an array of file paths:
  "agents": ["./agents/foo.agent.md", "./agents/bar.agent.md"]

ParsePluginJson reads the array first (preferred, Claude Code schema),
falls back to a string path (legacy). PluginInfo gains an AgentPaths
field. Validator and discovery use the array when present, else fall
back to directory-based convention.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Simplify agents to directory-path array form ["./agents/"], handle directories in array entries

Per Copilot CLI spec, agents field values are "Path(s) to agent directories"
— no need to list individual files. Use ["./agents/"] to mirror how
skills uses "./skills/". Updated validator and discovery to handle both
directory and file entries in the array.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Delete .nuget/nuget.exe

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Convert skills field to array format in plugin.json, update validator to handle both forms

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Align AgentPaths null checks to use { Count: > 0 } pattern for consistency

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Change `agents` and `skills` fields in plugin.json to array format for Claude Code CLI compatibility

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Delete .nuget/nuget.exe

* Consolidate SkillsPath/SkillPaths and AgentsPath/AgentPaths into single array fields

Remove dual string/array fields from PluginInfo. The parser now
normalizes legacy string values into single-element arrays, so all
downstream code uses only SkillPaths and AgentPaths (IReadOnlyList<string>).
This eliminates all array-vs-string fallback branching.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-03-20 10:27:28 +01:00
YuliiaKovalova dcba4f3d8e Add dotnet-template-engine plugin (#341) 
* Add dotnet-template-engine plugin

Add template-engine skills for .NET project scaffolding powered by the
DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent:

Skills:
- template-discovery: find, inspect, and compare templates with intent
  resolution (70+ keyword mappings)
- template-instantiation: create projects with CPM adaptation, latest NuGet
  versions, parameter validation, and multi-template composition
- template-authoring: create templates from existing projects and validate
  template.json for authoring issues

Agent:
- template-engine: expert agent that routes to the appropriate skill based
  on user intent

Also adds CODEOWNERS entries, marketplace.json registration, eval tests,
and .mcp.json for DotnetTemplateMCP server configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: mcpServers in plugin.json, expect_tools, path fix

- Move MCP server config into plugin.json under mcpServers so SkillValidator
  discovers the template_* tools during evaluation runs
- Add expect_tools constraints to all eval scenarios to verify actual MCP
  tool invocation instead of relying on rubric text alone
- Fix frontmatter path: template.config → .template.config (leading dot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update DotnetTemplateMCP to v1.2.0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Include 'dotnet new' in agent description for discoverability

Address review feedback from danmoseley: the frontmatter description
is how the caller decides whether to load the agent tokens, so it
needs to mention 'dotnet new' as a trigger keyword.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove MCP server configuration and references

- Delete .mcp.json MCP server config file
- Remove mcpServers section from plugin.json
- Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions
- Skills and eval files remain unchanged

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove ghost tool references and add CLI fallback guidance

- Remove references to 4 non-existent tools: template_compose,
  template_suggest_parameters, template_validate, solution_analyze
- Replace with real alternatives (template_inspect, manual review,
  sequential template_instantiate, direct file inspection)
- Add CLI fallback guidance so LLMs degrade gracefully when
  template tools are unavailable
- Fix authoring eval to not expect template_validate tool

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: CODEOWNERS naming, claude marketplace, README

- Rename CODEOWNERS header to dotnet-template-engine for consistency
- Mirror plugin entry in .claude-plugin/marketplace.json
- Add plugin to README What's Included table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json

- Updated template-discovery eval.yaml rubric to match expect_tools
  (removed 'or template_search' since expect_tools enforces template_from_intent)
- Made .claude-plugin/marketplace.json byte-for-byte identical to
  .github/plugin/marketplace.json (removed extra trailing newline)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review comments: strengthen eval assertions and fix rubrics

- Add exit_success assertions to all eval scenarios
- Add file_exists and file_contains assertions for template-instantiation
- Add output_contains for shortName and datatype in template-authoring
- Fix rubrics to reference actual tools (skill + dotnet new CLI) instead
  of non-existent MCP tools (template_from_intent, template_inspect,
  template_instantiate) which caused the judge to penalize quality scores
- Update expect_tools from non-existent MCP tools to 'skill'
- Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins
- Remove extra blank line in README.md after plugin table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools

The SKILL.md files and agent.md referenced MCP tools (template_from_intent,
template_inspect, template_instantiate, template_dry_run, etc.) that are
not available at runtime since no MCP server is declared in plugin.json.
This caused the agent to waste tokens attempting to find these tools before
falling back to dotnet new CLI commands.

Changes:
- template-discovery/SKILL.md: Replace MCP tool calls with dotnet new
  search, dotnet new list, dotnet new --help, and dotnet new --dry-run.
  Add intent-to-template keyword mapping table.
- template-instantiation/SKILL.md: Replace template_instantiate with
  dotnet new CLI commands. Add explicit CPM adaptation steps and
  multi-project composition with dotnet sln/dotnet add reference.
- template-authoring/SKILL.md: Replace template_create_from_existing
  with manual template.json creation guidance including example JSON.
  Replace template_dry_run/template_instantiate with CLI equivalents.
- template-engine.agent.md: Replace Available Tools table with CLI
  Commands Reference. Update all workflow steps to use CLI commands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix file assertion paths: remove ./ prefix for glob matcher compatibility

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
 2026-03-18 22:51:20 +01:00