mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Pilot repository-specific overlays for dotnet-test skills (#1115)
* Pilot repository overlays for test skills Add revisioned, repo-specific overlay bindings for selected dotnet-test skills and eval coverage for valid, scoped, and stale bindings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Allow attributed overlay references Add the JeremyKuhne source repositories to the reviewed external reference allowlist. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address overlay pilot review feedback Strengthen the MSTest eval against duplicate/default policy output and make the suite fixture parameter drive its test filter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify invalid overlay fallback Make invalid overlays fall back to portable guidance and accept natural revision mismatch explanations in the stale-binding eval. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix overlay precedence wording Restore the separator between repository and portable precedence rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Use a literal overlay project path check Prevent the scaffold eval grader from interpreting dots in the expected project path as regex wildcards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce command-only overlay evals Reject shell, edit, and create tools in both run-tests overlay prompts that prohibit execution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Amaury Levé
@@ -64,6 +64,7 @@ github.com/username
|
||||
testsmells.org
|
||||
|
||||
# Community
|
||||
github.com/JeremyKuhne
|
||||
github.com/Youssef1313/Combinatorial.MSTest
|
||||
ollama.com
|
||||
stackoverflow.com
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# Repository overlays
|
||||
|
||||
The `dotnet-test` plugin is piloting repository-specific overlays based on the
|
||||
portable-core strategy proposed by
|
||||
[@JeremyKuhne](https://github.com/JeremyKuhne). His
|
||||
[`agent-skills`](https://github.com/JeremyKuhne/agent-skills) repository and the
|
||||
[`touki` Roslyn analyzer overlay](https://github.com/JeremyKuhne/touki/blob/main/.agents/skills/roslyn-analyzers/overlay.md)
|
||||
demonstrate the original convention.
|
||||
|
||||
An overlay lets a user-installed or vendored portable skill apply local
|
||||
conventions without copying those conventions into the shared skill.
|
||||
|
||||
## Pilot skills and paths
|
||||
|
||||
| Skill | Repository overlay |
|
||||
|---|---|
|
||||
| `writing-mstest-tests` | `.agents/skill-overlays/dotnet-test/writing-mstest-tests.md` |
|
||||
| `scaffold-dotnet-test-project` | `.agents/skill-overlays/dotnet-test/scaffold-dotnet-test-project.md` |
|
||||
| `run-tests` | `.agents/skill-overlays/dotnet-test/run-tests.md` |
|
||||
|
||||
Each path is relative to the repository root being worked on, not the plugin
|
||||
installation directory. The files are optional. Skills continue with their
|
||||
portable behavior when no overlay exists.
|
||||
|
||||
## Overlay format
|
||||
|
||||
Use YAML frontmatter to identify the portable core and its overlay binding
|
||||
contract:
|
||||
|
||||
```markdown
|
||||
---
|
||||
core: dotnet-test/run-tests
|
||||
binding-revision: "1"
|
||||
mode: extend
|
||||
---
|
||||
|
||||
# Repository test execution
|
||||
|
||||
- Run unit tests with `pwsh ./eng/test.ps1 -Suite Unit`.
|
||||
- Do not replace the repository entry point with a direct `dotnet test`.
|
||||
```
|
||||
|
||||
`mode: extend` is the only mode in the pilot. The skill applies an overlay only
|
||||
when `core` identifies that exact skill and `binding-revision` matches the
|
||||
revision in the skill's own metadata. A missing or mismatched field is reported
|
||||
and the overlay is not applied; the skill continues with its portable guidance.
|
||||
|
||||
An overlay may provide paths, names, commands, framework choices, and local
|
||||
policies. It may narrow portable defaults, but it cannot expand tool
|
||||
permissions, task scope, filesystem access, network access, or publication
|
||||
authority.
|
||||
|
||||
Precedence is:
|
||||
|
||||
1. explicit user instructions;
|
||||
2. verified project and dependency constraints;
|
||||
3. the repository overlay;
|
||||
4. portable skill defaults and examples.
|
||||
|
||||
If an overlay contradicts the current repository, the skill reports the
|
||||
conflict instead of silently selecting either instruction. Increment
|
||||
`binding-revision` when the skill changes the overlay contract, re-review
|
||||
matching overlays, and remove local guidance after it is incorporated into the
|
||||
portable core.
|
||||
|
||||
This pilot is instruction composition, not a host-level Markdown merge. The
|
||||
Agent Skills standard permits custom metadata, but hosts do not currently
|
||||
provide overlay discovery, pin validation, or conflict resolution.
|
||||
@@ -12,6 +12,10 @@ description: >-
|
||||
existing coverage report, CI investigation, migration, or a persistent hot
|
||||
reload/watch loop.
|
||||
license: MIT
|
||||
metadata:
|
||||
portability: portable
|
||||
binding: optional-overlay
|
||||
binding-revision: "1"
|
||||
---
|
||||
|
||||
# Run .NET Tests
|
||||
@@ -19,6 +23,27 @@ license: MIT
|
||||
Return or execute the command or command sequence that matches the repository's
|
||||
project system, test platform, framework, and SDK mode.
|
||||
|
||||
## Repository overlay
|
||||
|
||||
For every repository-scoped task where read-only file inspection is allowed,
|
||||
check `.agents/skill-overlays/dotnet-test/run-tests.md` at the repository root
|
||||
before any other discovery. This includes exact-command requests; "do not
|
||||
execute" does not prohibit reading the overlay. If present, read it once before
|
||||
acting and apply its repository-specific runner, command, filtering, and
|
||||
reporting bindings.
|
||||
Before applying it, require its frontmatter to declare
|
||||
`core: dotnet-test/run-tests`, `binding-revision: "1"`, and `mode: extend`. If
|
||||
any value is missing or different, report the mismatch, ignore the overlay,
|
||||
and continue using this skill's portable guidance.
|
||||
Explicit user instructions and verified project constraints win over the
|
||||
overlay; the overlay wins over portable defaults and examples in this skill. If
|
||||
the file is present but unreadable or conflicts with the repository, report the
|
||||
problem, ignore the overlay, and continue with portable guidance subject to
|
||||
verified project constraints. If it is absent, continue normally.
|
||||
Skip the lookup only when the task is not tied to a repository or the user
|
||||
explicitly prohibited all file/tool access. An overlay cannot expand tool
|
||||
permissions or the task's scope.
|
||||
|
||||
## Scope and tool policy
|
||||
|
||||
Choose the smallest path that satisfies the request:
|
||||
|
||||
@@ -9,6 +9,10 @@ description: >-
|
||||
(code-testing-agent), run tests, migrate, or correct MSTest syntax/configuration
|
||||
without changing project or CI files (writing-mstest-tests).
|
||||
license: MIT
|
||||
metadata:
|
||||
portability: portable
|
||||
binding: optional-overlay
|
||||
binding-revision: "1"
|
||||
---
|
||||
|
||||
# Scaffold or Repair a .NET Test Project
|
||||
@@ -17,6 +21,27 @@ Create the smallest missing test container or repair only the missing wiring.
|
||||
The goal is test discovery through the repository's real build entry point, not
|
||||
a preferred solution layout.
|
||||
|
||||
## Repository overlay
|
||||
|
||||
For every repository-scoped task where read-only file inspection is allowed,
|
||||
check `.agents/skill-overlays/dotnet-test/scaffold-dotnet-test-project.md` at
|
||||
the repository root before any other discovery. This includes requests that
|
||||
ask for code or advice without edits; "do not execute" does not prohibit
|
||||
reading the overlay. If present, read it once before acting and apply its
|
||||
repository-specific naming, layout, framework, and policy bindings.
|
||||
Before applying it, require its frontmatter to declare
|
||||
`core: dotnet-test/scaffold-dotnet-test-project`, `binding-revision: "1"`, and
|
||||
`mode: extend`. If any value is missing or different, report the mismatch and
|
||||
continue using this skill's portable guidance without applying the overlay.
|
||||
Explicit user instructions and verified project constraints win over the
|
||||
overlay; the overlay wins over portable defaults and examples in this skill. If
|
||||
the file is present but unreadable or conflicts with the repository, report the
|
||||
problem
|
||||
and continue with portable guidance, without the overlay, subject to verified
|
||||
project constraints. If it is absent, continue normally. Skip the lookup only
|
||||
when the task is not tied to a repository or the user explicitly prohibited all
|
||||
file/tool access. An overlay cannot expand tool permissions or the task's scope.
|
||||
|
||||
## Route the Request
|
||||
|
||||
Inspect the repository before editing, then choose exactly one path:
|
||||
|
||||
@@ -13,6 +13,10 @@ description: >
|
||||
perform report-only audits, create project files rather than explain MSTest
|
||||
setup, run tests, migrate frameworks, or handle non-MSTest/non-.NET code.
|
||||
license: MIT
|
||||
metadata:
|
||||
portability: portable
|
||||
binding: optional-overlay
|
||||
binding-revision: "1"
|
||||
---
|
||||
|
||||
# Writing MSTest Tests
|
||||
@@ -20,6 +24,27 @@ license: MIT
|
||||
Help users write effective MSTest unit tests without exceeding the API level or
|
||||
conventions of the project's installed test stack.
|
||||
|
||||
## Repository overlay
|
||||
|
||||
For every repository-scoped task where read-only file inspection is allowed,
|
||||
check `.agents/skill-overlays/dotnet-test/writing-mstest-tests.md` at the
|
||||
repository root before any other discovery. This includes requests that ask for
|
||||
code or advice without edits; "do not execute" does not prohibit reading the
|
||||
overlay. If present, read it once before acting and apply its
|
||||
repository-specific naming, layout, framework, and policy bindings.
|
||||
Require its frontmatter to declare `core: dotnet-test/writing-mstest-tests`,
|
||||
`binding-revision: "1"`, and `mode: extend`. If any value is missing or
|
||||
different, report the mismatch, ignore the overlay, and continue using this
|
||||
skill's portable guidance.
|
||||
Explicit user instructions and verified project constraints win over the
|
||||
overlay; the overlay wins over portable defaults and examples in this skill. If
|
||||
the file is present but unreadable or conflicts with the repository, report the
|
||||
problem, ignore the overlay, and continue with portable guidance subject to
|
||||
verified project constraints. If it is absent, continue normally.
|
||||
Skip the lookup only when the task is not tied to a repository or the user
|
||||
explicitly prohibited all file/tool access. An overlay cannot expand tool
|
||||
permissions or the task's scope.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User wants to improve or modernize existing MSTest tests by implementing concrete fixes
|
||||
|
||||
@@ -566,3 +566,67 @@ stimuli:
|
||||
constraints:
|
||||
expect_tools:
|
||||
- bash
|
||||
- name: Use the repository unit-test entry point
|
||||
tags:
|
||||
feature: repository-overlay
|
||||
prompt: |
|
||||
What command should I use to run only the unit tests without restoring
|
||||
packages? Give me one exact command and do not execute it.
|
||||
environment:
|
||||
files:
|
||||
- src: fixtures/vstest-mstest/TestProject.csproj
|
||||
dest: TestProject.csproj
|
||||
- src: fixtures/vstest-mstest/OrderServiceTests.cs
|
||||
dest: OrderServiceTests.cs
|
||||
- src: fixtures/repository-overlay/test.ps1
|
||||
dest: eng/test.ps1
|
||||
- src: fixtures/repository-overlay/overlay.md
|
||||
dest: .agents/skill-overlays/dotnet-test/run-tests.md
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: '(?i)pwsh\s+\.?[/\\]eng[/\\]test\.ps1\s+-Suite\s+Unit\s+-NoRestore'
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: (?im)^\s*(?:[$>]\s*)?dotnet\s+test
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Gives one runnable command that uses the repository's supported test entry point
|
||||
- Selects only the unit suite and disables package restore
|
||||
- Does not execute the command or substitute a generic runner
|
||||
constraints:
|
||||
reject_tools:
|
||||
- bash
|
||||
- edit
|
||||
- create
|
||||
- name: Reject a stale repository test binding
|
||||
tags:
|
||||
feature: repository-overlay
|
||||
prompt: |
|
||||
What command should I use to run only the unit tests in this repository?
|
||||
Give me one exact command and do not execute it.
|
||||
environment:
|
||||
files:
|
||||
- src: fixtures/vstest-mstest/TestProject.csproj
|
||||
dest: TestProject.csproj
|
||||
- src: fixtures/vstest-mstest/OrderServiceTests.cs
|
||||
dest: OrderServiceTests.cs
|
||||
- src: fixtures/repository-overlay/stale-overlay.md
|
||||
dest: .agents/skill-overlays/dotnet-test/run-tests.md
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (?i)(binding-revision|revision).*(mismatch|unsupported|expected|doesn't match|does not match|differs from|\bnot\b)
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: '(?im)^\s*(?:[$>]\s*)?pwsh\s+\.?[/\\]eng[/\\]obsolete-test\.ps1(?:\s|$)'
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Identifies that the repository overlay targets an unsupported binding revision
|
||||
- Does not apply or recommend the stale overlay's obsolete command
|
||||
- Gives a safe next step without claiming that the stale repository binding is valid
|
||||
constraints:
|
||||
reject_tools:
|
||||
- bash
|
||||
- edit
|
||||
- create
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
core: dotnet-test/run-tests
|
||||
binding-revision: "1"
|
||||
mode: extend
|
||||
---
|
||||
|
||||
# Repository test execution
|
||||
|
||||
- Run the unit suite without restoring with
|
||||
`pwsh ./eng/test.ps1 -Suite Unit -NoRestore`.
|
||||
- The script is the supported entry point; do not substitute a direct
|
||||
`dotnet test` command.
|
||||
@@ -0,0 +1,9 @@
|
||||
---
|
||||
core: dotnet-test/run-tests
|
||||
binding-revision: "999"
|
||||
mode: extend
|
||||
---
|
||||
|
||||
# Obsolete repository test execution
|
||||
|
||||
- Run unit tests with `pwsh ./eng/obsolete-test.ps1 -Unit`.
|
||||
@@ -0,0 +1,13 @@
|
||||
param(
|
||||
[ValidateSet("Unit")]
|
||||
[string] $Suite = "Unit",
|
||||
[switch] $NoRestore
|
||||
)
|
||||
|
||||
$arguments = @("test", "TestProject.csproj", "--filter", "TestCategory=$Suite")
|
||||
if ($NoRestore) {
|
||||
$arguments += "--no-restore"
|
||||
}
|
||||
|
||||
dotnet @arguments
|
||||
exit $LASTEXITCODE
|
||||
@@ -272,3 +272,35 @@ stimuli:
|
||||
- Preserved central package management and added only the required production reference
|
||||
- Registered the project in the existing Catalog.slnx using valid SDK solution XML
|
||||
- Verified the exact solution-level dotnet test command discovers and runs the test
|
||||
|
||||
- name: Create tests in the repository-defined verification layout
|
||||
tags:
|
||||
feature: repository-overlay
|
||||
prompt: |
|
||||
Create the first xUnit v3 test project for ProductCode in this Catalog
|
||||
repository. Add one test proving that `new ProductCode(" ab-12 ").Normalize()`
|
||||
has the value `AB-12`, register the project in Catalog.slnx, and verify it
|
||||
through the solution.
|
||||
environment:
|
||||
files:
|
||||
- src: fixtures/slnx-project
|
||||
dest: .
|
||||
- src: fixtures/repository-overlay/overlay.md
|
||||
dest: .agents/skill-overlays/dotnet-test/scaffold-dotnet-test-project.md
|
||||
graders:
|
||||
- type: run-command
|
||||
config:
|
||||
command: sh -c "test -f verification/unit/Catalog.UnitTests/Catalog.UnitTests.csproj && test -f verification/unit/Catalog.UnitTests/ProductCodeSpec.cs && test ! -d tests"
|
||||
expected_exit_code: 0
|
||||
timeout: 1m
|
||||
- type: run-command
|
||||
config:
|
||||
command: sh -c "grep -Fq 'verification/unit/Catalog.UnitTests/Catalog.UnitTests.csproj' Catalog.slnx && dotnet test Catalog.slnx"
|
||||
expected_exit_code: 0
|
||||
timeout: 10m
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Created the test project under the repository's verification/unit layout
|
||||
- Used the repository's required Catalog.UnitTests project name and Spec.cs source suffix
|
||||
- Added only the requested ProductCode normalization behavior with central package management intact
|
||||
- Registered the project in Catalog.slnx and verified the solution-level test run
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
core: dotnet-test/scaffold-dotnet-test-project
|
||||
binding-revision: "1"
|
||||
mode: extend
|
||||
---
|
||||
|
||||
# Repository test project layout
|
||||
|
||||
- Put new unit-test projects under `verification/unit/`.
|
||||
- Name the Catalog test project `Catalog.UnitTests`.
|
||||
- Name test source files after the behavior subject with the `Spec.cs` suffix.
|
||||
@@ -518,3 +518,61 @@ stimuli:
|
||||
- bash
|
||||
- edit
|
||||
- create
|
||||
- name: Apply checked-in MSTest naming and category conventions
|
||||
tags:
|
||||
feature: repository-overlay
|
||||
prompt: |
|
||||
Fix the supplied MSTest code in PriceCalculatorTests.cs. Apply the
|
||||
applicable repository naming and TestCategory conventions while
|
||||
preserving the calculation, assertion behavior, and installed
|
||||
dependencies.
|
||||
environment:
|
||||
files:
|
||||
- src: fixtures/modern-mstest/TestProject.csproj
|
||||
dest: TestProject.csproj
|
||||
- src: fixtures/modern-mstest/global.json
|
||||
dest: global.json
|
||||
- src: fixtures/repository-overlay/AGENTS.md
|
||||
dest: AGENTS.md
|
||||
- src: fixtures/repository-overlay/PriceCalculatorTests.cs
|
||||
dest: PriceCalculatorTests.cs
|
||||
- src: fixtures/repository-overlay/PriceCalculator.cs
|
||||
dest: PriceCalculator.cs
|
||||
- src: fixtures/repository-overlay/overlay.md
|
||||
dest: .agents/skill-overlays/dotnet-test/writing-mstest-tests.md
|
||||
graders:
|
||||
- type: file-contains
|
||||
config:
|
||||
path: PriceCalculatorTests.cs
|
||||
value: public void Should_
|
||||
- type: file-contains
|
||||
config:
|
||||
path: PriceCalculatorTests.cs
|
||||
value: _When_
|
||||
- type: file-contains
|
||||
config:
|
||||
path: PriceCalculatorTests.cs
|
||||
value: '[TestCategory("Fast")]'
|
||||
- type: file-not-contains
|
||||
config:
|
||||
path: PriceCalculatorTests.cs
|
||||
value: CalculateDiscount_ValidCoupon_ReturnsDiscountedTotal
|
||||
- type: file-not-contains
|
||||
config:
|
||||
path: PriceCalculatorTests.cs
|
||||
value: '[TestCategory("Unit")]'
|
||||
- type: file-not-contains
|
||||
config:
|
||||
path: PriceCalculatorTests.cs
|
||||
value: sealed class
|
||||
- type: run-command
|
||||
config:
|
||||
command: dotnet test TestProject.csproj
|
||||
expected_exit_code: 0
|
||||
timeout: 10m
|
||||
- type: prompt
|
||||
rubric:
|
||||
- The updated test uses the component-specific method naming convention rather than the general repository default
|
||||
- The test uses the component-specific Fast category rather than the general Unit category
|
||||
- The class remains inheritable under the component-specific fixture-generation exception
|
||||
- The calculation and assertion behavior remain unchanged
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
# General test defaults
|
||||
|
||||
Unless a component-specific test policy applies:
|
||||
|
||||
- Name tests `<Method>_<Scenario>_<Expected>`.
|
||||
- Mark unit tests with `[TestCategory("Unit")]`.
|
||||
- Seal concrete test classes.
|
||||
@@ -0,0 +1,5 @@
|
||||
public sealed class PriceCalculator
|
||||
{
|
||||
public decimal CalculateDiscount(decimal price, decimal discountPercent)
|
||||
=> price * (1 - discountPercent / 100);
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
using Microsoft.VisualStudio.TestTools.UnitTesting;
|
||||
|
||||
[TestClass]
|
||||
public class PriceCalculatorTests
|
||||
{
|
||||
[TestMethod]
|
||||
public void CalculateDiscount_ValidCoupon_ReturnsDiscountedTotal()
|
||||
{
|
||||
var result = new PriceCalculator().CalculateDiscount(100m, 10);
|
||||
Assert.AreEqual(90m, result);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
core: dotnet-test/writing-mstest-tests
|
||||
binding-revision: "1"
|
||||
mode: extend
|
||||
---
|
||||
|
||||
# Repository MSTest conventions
|
||||
|
||||
- Name tests `Should_<ExpectedBehavior>_When_<Condition>`.
|
||||
- Mark tests handled by this component with `[TestCategory("Fast")]`.
|
||||
- Do not seal test classes; this repository generates partial derived fixtures.
|
||||
Reference in New Issue
Block a user