Amaury Levé
cb9651deae
* Pilot repository overlays for test skills Add revisioned, repo-specific overlay bindings for selected dotnet-test skills and eval coverage for valid, scoped, and stale bindings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Allow attributed overlay references Add the JeremyKuhne source repositories to the reviewed external reference allowlist. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address overlay pilot review feedback Strengthen the MSTest eval against duplicate/default policy output and make the suite fixture parameter drive its test filter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify invalid overlay fallback Make invalid overlays fall back to portable guidance and accept natural revision mismatch explanations in the stale-binding eval. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix overlay precedence wording Restore the separator between repository and portable precedence rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Use a literal overlay project path check Prevent the scaffold eval grader from interpreting dots in the expected project path as regex wildcards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce command-only overlay evals Reject shell, edit, and create tools in both run-tests overlay prompts that prohibit execution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2.6 KiB
Repository overlays
The dotnet-test plugin is piloting repository-specific overlays based on the
portable-core strategy proposed by
@JeremyKuhne. His
agent-skills repository and the
touki Roslyn analyzer overlay
demonstrate the original convention.
An overlay lets a user-installed or vendored portable skill apply local conventions without copying those conventions into the shared skill.
Pilot skills and paths
| Skill | Repository overlay |
|---|---|
writing-mstest-tests |
.agents/skill-overlays/dotnet-test/writing-mstest-tests.md |
scaffold-dotnet-test-project |
.agents/skill-overlays/dotnet-test/scaffold-dotnet-test-project.md |
run-tests |
.agents/skill-overlays/dotnet-test/run-tests.md |
Each path is relative to the repository root being worked on, not the plugin installation directory. The files are optional. Skills continue with their portable behavior when no overlay exists.
Overlay format
Use YAML frontmatter to identify the portable core and its overlay binding contract:
---
core: dotnet-test/run-tests
binding-revision: "1"
mode: extend
---
# Repository test execution
- Run unit tests with `pwsh ./eng/test.ps1 -Suite Unit`.
- Do not replace the repository entry point with a direct `dotnet test`.
mode: extend is the only mode in the pilot. The skill applies an overlay only
when core identifies that exact skill and binding-revision matches the
revision in the skill's own metadata. A missing or mismatched field is reported
and the overlay is not applied; the skill continues with its portable guidance.
An overlay may provide paths, names, commands, framework choices, and local policies. It may narrow portable defaults, but it cannot expand tool permissions, task scope, filesystem access, network access, or publication authority.
Precedence is:
- explicit user instructions;
- verified project and dependency constraints;
- the repository overlay;
- portable skill defaults and examples.
If an overlay contradicts the current repository, the skill reports the
conflict instead of silently selecting either instruction. Increment
binding-revision when the skill changes the overlay contract, re-review
matching overlays, and remove local guidance after it is incorporated into the
portable core.
This pilot is instruction composition, not a host-level Markdown merge. The Agent Skills standard permits custom metadata, but hosts do not currently provide overlay discovery, pin validation, or conflict resolution.