* Make evaluate a subcommand with tests-dir as required argument
Change EvaluateCommand from RootCommand to a regular Command('evaluate')
so it is a subcommand like check, consolidate, and rejudge. Convert
--tests-dir from an optional Option to a required positional Argument
placed after the skill paths.
Update .csproj RunArguments, evaluation-run workflow, README, and
CONTRIBUTING.md to reflect the new CLI syntax.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --parallel-skills/scenarios/runs to 3
Update the default from 1 to 3 for all three parallelism options and
remove the now-redundant overrides from RunArguments in the .csproj.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace RunArguments with launchSettings.json, publish in skill-check
Add Properties/launchSettings.json with default evaluate args for local
development. Remove RunArguments from .csproj. Update skill-check.yml to
publish the validator and invoke the binary directly instead of using
dotnet run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Keep --tests-dir as a required option, not positional
A variadic paths argument greedily consumes all positional values,
making a trailing positional tests-dir impossible to parse. Keep
--tests-dir as a named required option which matches the previous UX.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move testsDirOpt after resultsDirOpt, drop evaluate from CONTRIBUTING examples
Reorder testsDirOpt declaration and registration to follow resultsDirOpt.
Remove the evaluate subcommand from CONTRIBUTING.md dotnet run examples
since the launch profile already provides it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
15 KiB
Contributing
Thanks for your interest in contributing. We expect to accept external contributions, but the bar for merging is intentionally high.
This repository contains shared building blocks for coding agents:
- Skills: reusable, task focused instruction packs
- Agents: role based configurations that bundle tool expectations and skill selection
Because these artifacts can affect many users and workflows, we prioritize correctness, clarity, and long term maintainability over speed.
Code ownership
Every plugin, skill, agent, and agentic workflow must have designated owners in the .github/CODEOWNERS file. When you add a new skill, agent, or workflow, add a matching CODEOWNERS entry. Ownership must be either:
- Two or more FTE GitHub aliases (e.g.,
@user1 @user2), or - A GitHub team alias (e.g.,
@dotnet/my-team)
This ensures that every contribution area has accountable reviewers and that PRs are automatically routed to the right people.
Repository layout
plugins/
<plugin>/
plugin.json
skills/
<skill-name>/
SKILL.md
scripts/
references/
assets/
agents/
<agent-name>.agent.md
tests/
<plugin>/
<skill-name>/
eval.yaml
<fixture files>
agentic-workflows/
<plugin>/
Every plugin must have a plugin.json file in the plugin root that is linked to from the marketplace.json file.
Plugin organization
Skills are grouped into domain-specific plugins. When proposing a new skill, place it in the plugin that best matches its domain:
| Plugin | Domain |
|---|---|
dotnet |
Common everyday C#/.NET coding tasks useful to all .NET developers |
dotnet-upgrade |
Migrating and upgrading .NET projects across framework versions, language features, and compatibility targets |
dotnet-diag |
Performance investigations, debugging, and incident analysis |
dotnet-data |
Data access and Entity Framework |
dotnet-msbuild |
MSBuild and project system |
dotnet-ai |
AI and ML: technology selection, LLM integration, agentic workflows, RAG pipelines, and classic ML |
If your skill does not fit any existing plugin, consider creating a new one. The following plugin names are reserved for future use and are good candidates for new skills in those areas:
dotnet-aspnet— ASP.NETdotnet-wpf— WPFdotnet-winforms— Windows Forms (WinForms)dotnet-maui— .NET MAUI
To create a new plugin:
- Add
plugins/<plugin-name>/plugin.jsonand askills/directory beneath it. - Add a matching entry in both
.github/plugin/marketplace.jsonand.claude-plugin/marketplace.json. The.claude-plugin/marketplace.jsonfile must remain an exact copy of.github/plugin/marketplace.json, so any change to one file (adding, removing, or editing a plugin entry) must be applied to the other in the same way. - Add a CODEOWNERS entry for the new plugin and its tests (see Code ownership).
- Add the plugin to the What's Included table in the root
README.md. - Create a
tests/<plugin-name>/directory for skill tests.
See existing plugins for the expected format.
Before you start
- Search existing issues and pull requests to avoid duplicates.
- Start with an issue before you submit a pull request for a new skill, a new agent, or any non trivial change. This helps us align on scope and avoids wasted work.
- Small fixes like typos, broken links, or clearly isolated corrections can go straight to a pull request.
- Keep changes small and focused. One skill or one agent per pull request is a good default.
What we look for
We are most likely to accept contributions that are:
- Addresses a LLM gap and is clearly motivated by a real use case
- Likely to be used frequently and is general (not repo-specific)
- Narrow in scope and easy to review
- Tool conscious and explicit about assumptions
- Verifiable with concrete validation steps
- Written to be durable across repo changes
We are less likely to accept contributions that:
- Add broad frameworks, meta tooling, or large reorganizations
- Duplicate guidance that already exists in another skill
- Encode private environment details, credentials, or company specific secrets
- Depend on proprietary tools or access that most contributors will not have
- Skills that make use of third party tools will be evaluated on a case by case basis. Acceptance of such skills will depend on our evaluation of the provenance and maturity of any such tools.
Proposing a new skill
Please review the What we look for section and add justification for the skill in your issue and PR.
A skill should be self-contained and:
- Clearly state what it does and when to use it.
- Frontmatter (name and description) is small and minimal, just enough for LLM to understand when to use it
- Keep the SKILL.md body under 500 lines for optimal performance. Split content into separate files when you approach this limit. Use a progressive disclosure pattern, referring to those files from the SKILL.md file where needed.
- Specify required inputs (repo context, environment, access needs).
- Prefer concrete checklists and verification steps over vague guidance.
Create a new folder under a plugin's skills/ directory:
plugins/<plugin>/skills/<skill-name>/SKILL.md
A skill should answer three questions up front:
- What outcome does the skill produce
- When should an agent use it
- How does the agent validate success
Skill naming
Use short, kebab-case names that mirror how developers naturally phrase the task, prioritizing keyword overlap over grammar — e.g., add-aspnet-auth, configure-jwt-auth, setup-identity-server. Optionally using gerund style (verb-ing) is acceptable as well - e.g., configuring-caching.
Optimize for intent matching: lead with the action verb users actually say (add, configure, setup, deploy) followed the outcome the skill is aiming to assist.
The SKILL.md is required to have front-matter at a minimum:
Create the file with required YAML frontmatter:
---
name: <skill-name>
description: <description of what the skill does, when to use it, and when not to use it>
---
Tip: The
descriptionfield is used by the agent runtime to decide whether to load the full skill. Include when to use and when not to use guidance directly in the description so the agent can select or skip skills without reading the entireSKILL.md. This avoids unnecessary token usage. Seethread-abort-migration/SKILL.mdfor a good example.
Recommended SKILL.md sections
- Purpose: one paragraph describing the outcome.
- When to use / When not to use (put the essentials in the frontmatter
description; expand here only if more detail is needed). - Inputs: what the agent needs (files, commands, permissions).
- Workflow: numbered steps with checkpoints.
- Validation: how to confirm the result (tests, linters, manual checks).
- Common pitfalls: known traps and how to avoid them.
Skill checklist
Include a SKILL.md that covers:
- Purpose and non goals
- When to use and when not to use (summarized in the frontmatter
description; body section for extended detail) - Inputs and prerequisites
- Step by step workflow with checkpoints
- Validation steps that can be run or observed
- Failure modes and recovery guidance
Also:
- Avoid duplicating text across multiple skills. Prefer referencing shared patterns.
- Do not include content copied from other repositories. If you are inspired by existing work, rewrite in your own words and adapt it to our conventions.
Proposing a new agent
An agent definition should be opinionated but bounded:
- Describe the role (e.g., "WinForms Expert", "Security Reviewer", "Docs Maintainer").
- Define boundaries (what the agent should not do).
- List the skills it expects to use and how it chooses among them.
Add an agent file under a plugin's agents/ directory:
plugins/<plugin>/agents/<agent-name>.agent.md
Agent checklist
Include documentation that explains:
- Role and intended tasks
- Boundaries and safety constraints
- Tooling assumptions
- How the agent chooses which skills to apply
- What a good completion looks like, including validation expectations
Testing and validation
Skills and agents are documentation driven, but we still treat them as production assets.
- Every change should include a validation section that a reviewer can follow.
- If your change references commands, keep them cross platform when practical. If not, state the supported environment.
- If your change depends on external services, document how a reviewer can validate without privileged access, or explain why validation is not possible.
Writing skill tests
Each skill should have an eval.yaml file that defines test scenarios. Tests live under the repo root tests/ directory, matching the plugin and skill name:
tests/<plugin>/<skill-name>/eval.yaml
A minimal eval file:
scenarios:
- name: "Describe what the agent should do"
prompt: "The prompt sent to the agent"
assertions:
- type: "output_contains"
value: "expected text in agent output"
rubric:
- "The agent correctly identified the issue"
- "The agent suggested a concrete fix"
timeout: 120
Test fixture files
If a scenario requires files in the agent's working directory (e.g. .csproj, .sln, .cs files), place them alongside eval.yaml and opt into auto-copy:
tests/<plugin>/<skill-name>/
eval.yaml
MyProject.csproj
Program.cs
scenarios:
- name: "Diagnose build failure"
prompt: "Why does this project fail to build?"
setup:
copy_test_files: true # copies MyProject.csproj, Program.cs into work dir
assertions:
- type: "output_matches"
pattern: "CS\\d{4}"
You can also create files inline or reference files from the skill directory:
setup:
files:
- path: "input.txt"
content: "inline file content"
- path: "data.csv"
source: "fixtures/sample-data.csv" # relative to skill directory
See the skill-validator README for the full list of assertion types, constraints, and rubric options.
Running tests locally
Prerequisites: .NET 10 SDK or later and gh auth login.
# Run tests for a single plugin
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate --tests-dir tests/dotnet-msbuild plugins/dotnet-msbuild/skills
# Run tests for a single skill (pass the skill directory directly)
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate --tests-dir tests/dotnet-msbuild plugins/dotnet-msbuild/skills/common-build-errors
# Fewer runs for faster iteration (default is 5)
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate --runs 3 --tests-dir tests/dotnet-msbuild plugins/dotnet-msbuild/skills
# Use a specific model
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate --model claude-opus-4.6 --tests-dir tests/dotnet-msbuild plugins/dotnet-msbuild/skills
# Run with verbose logging
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate --verbose --tests-dir tests/dotnet-msbuild plugins/dotnet-msbuild/skills
Warning
If you share the results in a Pull Request, make sure to have
--runsconfigured to at least 3 but better 5 for reliable results.
CI evaluation
Tests run automatically on pull requests that modify files under plugins/. The evaluation workflow discovers changed plugins and runs the skill-validator for each one. Results are posted as a PR comment and uploaded as build artifacts.
Writing style
- Be concise and specific.
- Prefer numbered steps for workflows.
- Prefer checklists for requirements.
- Define terminology the first time it appears.
- Avoid excessive formatting and avoid clever wording that could be misread by an agent.
Security and safety
- Do not include secrets, tokens, or internal URLs.
- If you discover a security issue, do not open a public issue with sensitive details. Use the repository or organization security reporting process instead.
External references
Skills often reference external tools, documentation, and projects — this is
expected and welcome, including community and third-party resources. To help
reviewers stay aware of external dependencies, the repository includes an
automated reference scanner (eng/reference-scanner/scan.ps1) that runs in CI.
The scanner treats all of the following as CI-blocking errors:
http://URLs wherehttps://should be used<script>tags loading external resources without anintegrity(SRI) attribute- Pipe-to-shell patterns (
curl ... | bash) - URLs pointing to domains not listed in
eng/reference-scanner/known-domains.txt
Community tools and third-party projects are evaluated on a case-by-case basis
(see "What we look for" above). If your skill references a new external domain,
add it to eng/reference-scanner/known-domains.txt in the same PR — the reviewer will
approve it alongside the skill content.
Review process
Maintainers may request changes for:
- Clarity and unambiguous instructions
- Reduced scope
- More explicit validation
- Compatibility with multiple agent runtimes
- Consistency with existing conventions
We may close pull requests that are out of scope or too large to review. If that happens, we are happy to suggest a smaller path forward.
Licensing and provenance
Only submit content that you have the right to contribute.
- Do not include copyrighted text from other projects.
- You may be asked to confirm that your contribution is original or appropriately licensed.
Getting help
If you are unsure where a change belongs or how to structure a skill or agent, open an issue describing:
- The user problem
- The proposed outcome
- A small example of the desired behavior
If you're not sure whether something belongs under skills/ or agents/, a good rule of thumb is:
- Put reusable task playbooks in
skills/. - Put role + operating model in
agents/.
Quality bar
Skills and agents in this repo should be:
- Actionable: the agent can follow them without guesswork.
- Minimal: no extra features or scope creep; focus on the task.
- Verifiable: always include a way to validate success.
- Tool-conscious: don't assume capabilities that might not exist in every runtime.
Skill-Validator & Evaluation workflow
Changes to eng/skill-validator or the .github/workflows/evaluation*.yml workflows must be made from a branch in the dotnet/skills repository (i.e., not from a fork). This is a security measure.
For pull requests from forks, the evaluation workflow (triggered via /evaluate) always uses the workflow YAML from the default branch of dotnet/skills and builds the validator from that default-branch checkout, so any changes to these files in the forked PR will be ignored during evaluation.