Abhitej John
780c999546
Fail closed on missing or duplicate comparison slots, reject duplicate stimulus names before execution, and clarify how the practical floor changes power at its boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1c6716a2-172b-461a-adb5-127b1ba7e96b
1.9 KiB
Repository Instructions
This repository contains skill plugins under plugins/. Each subdirectory in plugins/ is an independent plugin (e.g., plugins/dotnet-msbuild, plugins/dotnet).
Working on skills and evals
Use the repository's own authoring skills under .agents/skills/ instead of improvising:
create-skill— scaffolding a new skill, and writing adescriptionthe runtime will route to.create-skill-test— writing or resizing aneval.yaml. Evals use the Vally schema (stimuli:/graders:/defaults:);scenarios:/assertions:no longer load.improve-skill-quality— an eval regressed, produced no verdict, or the skill did not activate. Classify the failure before editing skill content; broken fixtures, underpowered trial counts and harness errors routinely masquerade as skill regressions.
Before pushing eval changes, run python eng/eval-quality/check_eval_quality.py. It blocks eleven
structural defect classes documented in eng/eval-quality/README.md that can corrupt a real
evaluation result.
The distilled quality rules — what makes a skill beat its own baseline — live in the "Quality bar"
section of CONTRIBUTING.md.
Skill-Validator
The skill-validator is a shipping tool — its NuGet package and .tar.gz archives are built from eng/skill-validator/src/. Content referenced at runtime or bundled with the tool (docs, README, etc.) must live under src/ so it is included in the published output. Do not add references from src/ to files outside of it, except for explicitly linked packaging assets (such as the repo-root LICENSE file) referenced by the project file.
When modifying the evaluation pipeline (evaluation.yml), results JSON schema (Models.cs), or the skill-validator evaluation logic, review and update eng/skill-validator/src/docs/InvestigatingResults.md to keep the failure investigation guidance, schema documentation, and example scripts in sync.