skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
4801b2ccce09ffd801fa804c79655932171fe238
dotnet__skills/.gitignore
T
Amaury Levé 38bf797c27 Capture the repo's hard-won skill-authoring lessons as guidance (#979) 
* Capture the repo's hard-won skill-authoring lessons as guidance

Mines ~35 merged PRs (#830-#976), issue #899 and eng/eval-quality/README.md
into reusable guidance so contributors do not have to rediscover it.

- Add .agents/skills/improve-skill-quality: a triage playbook for evals that
  regress, return no verdict, or fail to activate. Classify before rewriting -
  broken fixtures, underpowered trial counts, forced tools, stale spec keys and
  harness errors have all masqueraded as skill regressions. Two reference docs
  carry the content patterns and the symptom/cause/fix catalogue with citations.
- Rewrite .agents/skills/create-skill-test. It documented a pre-Vally schema
  (scenarios:/assertions:/setup.copy_test_files) that no longer loads: all 97
  eval specs use stimuli:/graders:/environment. Anyone following it authored a
  broken eval. Also folds in the trial floor, dormancy-guard and fixture rules.
- Extend .agents/skills/create-skill with the description-as-router rules and a
  "write for delta over the baseline model" section.
- CONTRIBUTING.md: add the quality playbook, document eval sizing, and correct
  the eval snippet and the stale "raise runs in dotnet-skills.experiment.yaml"
  advice - an experiment-level runs overwrites every eval's own value.
- .gitignore: .agents/skills/ was silently swallowing the repo's own authoring
  skills. Replaced with .agents/skills/* plus negations for the five owned
  directories, so installed third-party skills stay ignored.

Validated: markdownlint clean, check_eval_quality.py clean, all relative links
resolve, frontmatter parses, descriptions under 1024 chars, bodies under 500
lines. Claims fact-checked against the repo by a review pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 771c704d-9ac9-4d5f-8ccb-54c62066630f

* Address multi-model review: correct the sign-test arithmetic and eval exceptions

Three review rounds (Copilot, Claude Opus 4.8, GPT-5.6 Sol/Terra) against the
implementation in eng/vally-adapter/adapt.mjs.

Statistics. The docs conflated the gate's two independent bars. adapt.mjs gates
on (1) counted trials >= 5, else `underpowered`, and (2) p <= 0.05 on a sign
test over the *discordant* (non-tie) trials. So "at 5-7 trials one tie makes a
pass unreachable" was wrong: at 6 trials 5W/1T/0L is 5 discordant and passes at
p = 0.031, and at 7 trials two ties are survivable. Every trial table is now
keyed on discordant trials. eng/eval-quality/README.md carried the same
over-generalisation and is corrected too, since these docs restate it.

Eval coverage. "Every skill needs an eval" and "no direct eval for a
disable-model-invocation skill" contradicted each other. Reference skills are
now explicitly exempt and covered through their consumers, and filter-syntax is
no longer described as the sole answer-graded exception - platform-detection
uses the same pattern.

Fixtures. "Every buildable fixture must build" forbade the intentionally broken
fixtures the same document demonstrates. Now: healthy fixtures build,
deliberately broken ones fail only for the reason their stimulus is about.

Triage routing. The first-match classifier sent fixture failures to the harness
row before the fixture row could match, and treated any positive-but-failing
record as a power problem even though adapt.mjs requires `conclusive` first.
Added an inconclusive row, moved the fixture row up, and scoped "quote a losing
trial" to content fixes so it no longer blocks the no-results path.

Also: expect_tools does not assert activation (the harness reports it
separately); CONTRIBUTING claimed evaluations run automatically on plugin
changes when a maintainer must trigger /evaluate against a bound commit; the
minimal eval example is labelled as a sub-floor skeleton; create-skill scaffolds
into plugins/<plugin>/skills/ rather than a non-existent top-level skills/.

Validated: markdownlint clean, check_eval_quality.py clean, adapt.test.mjs 21
pass, links and anchors resolve.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 771c704d-9ac9-4d5f-8ccb-54c62066630f

* Address review: document that agent evals sit outside the verdict flow

Abhitej's review found one actionable gap plus three nits; all four are small
enough to land here rather than as follow-ups.

Agent evals. create-skill-test told authors to write tests/<plugin>/agent.<name>/
eval.yaml, size it for the trial floor, and validate with run-skill-evals.sh --
none of which applies. The canonical experiment declares
`evals: tests/*/!(agent.*)/eval.yaml`, so agent specs are excluded: no verdict is
computed, the floor does not apply, and the runner drops them even when named
explicitly because --eval-filter is intersected with that glob. Step 1 now says
so, Step 10 shows the EXPERIMENT_FILE route for actually exercising one, and the
checklist and pitfalls are scoped accordingly.

Nits: aligned eng/eval-quality/README.md check 8 on "stimuli" (it still said
"scenarios" while every authoring doc says stimuli); added a .gitignore comment
so the next repo-owned authoring skill gets a negation instead of being silently
untracked; and noted in improve-skill-quality that
eng/skill-validator/src/docs/InvestigatingResults.md documents the retired
skill-validator evaluate schema, so the vally-adapter one is the guide to read.

Validated: markdownlint clean, check_eval_quality.py clean, 22/22 eval-quality
self-tests, gitignore negations still exact.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 771c704d-9ac9-4d5f-8ccb-54c62066630f

* Generalize two triage rows that were written in test-skill vocabulary

The triage catalogue is meant to apply to every plugin, but two rows stated
their rule in dotnet-test terms, so an author on another plugin would read past
them:

- the split-brain fixture row was framed entirely around Cobertura `line-rate`;
  the rule is that a fixture must never state the same fact twice in two places
  that disagree, whatever the format. Cobertura is now the named example.
- the volume-comparison row said "both arms write tests" / "do not reward test
  count"; the rule is that when both arms produce the same kind of artifact the
  judge falls back on comparing quantity.

Also generalized the one-line summary of the first rule in SKILL.md Step 4 so
the detail and the summary do not disagree.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 771c704d-9ac9-4d5f-8ccb-54c62066630f

* Correct the environment.skills guidance and a stale cross-reference

Two suppressed findings from the Copilot review summaries, both real.

environment.skills. The Step 5 skeleton documented it as "extra skills loaded
in the isolated arm only", which is wrong in both halves. The experiment
declares `vary: /environment/skills` and supplies the value for each arm itself
(`[]` for baseline, `plugins/<plugin>/skills/<skill>` for skilled), so whatever
a skill eval declares is replaced in every arm -- it cannot add a skill to one
arm, and in a skill eval it does nothing at all. The key is meaningful only in
`agent.*` evals, which the experiment does not vary. Removed it from the generic
skeleton, documented where it does apply, and pointed at an existing agent eval
to copy rather than reproducing a spelling: the three agent specs in this repo
disagree about whether entries are bare names or relative paths, so the doc
should not assert one.

Stale cross-reference. Moving the fixture row to the top of the triage table
last round left the sentence below it still saying a fixture failure "also
matches the two rows above it", when it is now the first row. Reworded to say
why it leads.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 771c704d-9ac9-4d5f-8ccb-54c62066630f

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 771c704d-9ac9-4d5f-8ccb-54c62066630f
2026-08-03 08:17:31 +00:00

452 lines
8.3 KiB
Plaintext
Raw Blame History

## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
##
## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore
# Installed skills (other authors' content). The repo's own authoring skills are
# re-included below; a directory pattern would make that impossible, so this
# ignores the *entries* under .agents/skills/ rather than the directory itself.
# Adding a new repo-owned authoring skill? Add a negation for it here too, or it
# will be silently untracked.
.agents/skills/*
!.agents/skills/authoring-github-workflows/
!.agents/skills/create-custom-agent/
!.agents/skills/create-skill/
!.agents/skills/create-skill-test/
!.agents/skills/improve-skill-quality/
# User-specific files
*.rsuser
*.suo
*.user
*.userosscache
*.sln.docstates
*.env
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Mono auto generated files
mono_crash.*
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
[Ww][Ii][Nn]32/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
[Aa][Rr][Mm]64[Ee][Cc]/
bld/
[Oo]bj/
[Oo]ut/
[Ll]og/
[Ll]ogs/
# Build results on 'Bin' directories
**/[Bb]in/*
# Uncomment if you have tasks that rely on *.refresh files to move binaries
# (https://github.com/github/gitignore/pull/3736)
#!**/[Bb]in/*.refresh
# Language server cache files
*.lscache
# Visual Studio 2015/2017 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# Visual Studio 2017 auto generated files
Generated\ Files/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
*.trx
# NUnit
*.VisualState.xml
TestResult.xml
nunit-*.xml
# Approval Tests result files
*.received.*
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# Benchmark Results
BenchmarkDotNet.Artifacts/
# .NET Core
project.lock.json
project.fragment.lock.json
artifacts/
# ASP.NET Scaffolding
ScaffoldingReadMe.txt
# StyleCop
StyleCopReport.xml
# Files built by Visual Studio
*_i.c
*_p.c
*_h.h
*.ilk
*.meta
*.obj
*.idb
*.iobj
*.pch
*.pdb
*.ipdb
*.pgc
*.pgd
*.rsp
# but not Directory.Build.rsp, as it configures directory-level build defaults
!Directory.Build.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*_wpftmp.csproj
*.log
*.tlog
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
# Chutzpah Test files
_Chutzpah*
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio profiler
*.psess
*.vsp
*.vspx
*.sap
# Visual Studio Trace Files
*.e2e
# TFS 2012 Local Workspace
$tf/
# Guidance Automation Toolkit
*.gpState
# ReSharper is a .NET coding add-in
_ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# TeamCity is a build add-in
_TeamCity*
# DotCover is a Code Coverage Tool
*.dotCover
# AxoCover is a Code Coverage Tool
.axoCover/*
!.axoCover/settings.json
# Coverlet is a free, cross platform Code Coverage Tool
coverage*.json
coverage*.xml
coverage*.info
# ...but coverage files committed as eval fixtures are inputs, not build output.
# Without this negation `git add` silently skips them and the eval fails at setup.
!tests/**/fixtures/**/coverage*.xml
!tests/**/fixtures/**/coverage*.json
!tests/**/fixtures/**/coverage*.info
# Visual Studio code coverage results
*.coverage
*.coveragexml
# NCrunch
_NCrunch_*
.NCrunch_*
.*crunch*.local.xml
nCrunchTemp_*
# MightyMoose
*.mm.*
AutoTest.Net/
# Web workbench (sass)
.sass-cache/
# Installshield output folder
[Ee]xpress/
# DocProject is a documentation generator add-in
DocProject/buildhelp/
DocProject/Help/*.HxT
DocProject/Help/*.HxC
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/Html2
DocProject/Help/html
# Click-Once directory
publish/
# Publish Web Output
*.[Pp]ublish.xml
*.azurePubxml
# Note: Comment the next line if you want to checkin your web deploy settings,
# but database connection strings (with potential passwords) will be unencrypted
*.pubxml
*.publishproj
# Microsoft Azure Web App publish settings. Comment the next line if you want to
# checkin your Azure Web App publish settings, but sensitive information contained
# in these scripts will be unencrypted
PublishScripts/
# NuGet Packages
*.nupkg
# NuGet Symbol Packages
*.snupkg
# The packages folder can be ignored because of Package Restore
**/[Pp]ackages/*
# except build/, which is used as an MSBuild target.
!**/[Pp]ackages/build/
# Uncomment if necessary however generally it will be regenerated when needed
#!**/[Pp]ackages/repositories.config
# NuGet v3's project.json files produces more ignorable files
*.nuget.props
*.nuget.targets
# Microsoft Azure Build Output
csx/
*.build.csdef
# Microsoft Azure Emulator
ecf/
rcf/
# Windows Store app package directories and files
AppPackages/
BundleArtifacts/
Package.StoreAssociation.xml
_pkginfo.txt
*.appx
*.appxbundle
*.appxupload
# Visual Studio cache files
# files ending in .cache can be ignored
*.[Cc]ache
# but keep track of directories ending in .cache
!?*.[Cc]ache/
# Others
ClientBin/
~$*
*~
*.dbmdl
*.dbproj.schemaview
*.jfm
*.pfx
*.publishsettings
orleans.codegen.cs
# Including strong name files can present a security risk
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
#*.snk
# Since there are multiple workflows, uncomment next line to ignore bower_components
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
#bower_components/
# RIA/Silverlight projects
Generated_Code/
# Backup & report files from converting an old project file
# to a newer Visual Studio version. Backup files are not needed,
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
UpgradeLog*.XML
UpgradeLog*.htm
ServiceFabricBackup/
*.rptproj.bak
# SQL Server files
*.mdf
*.ldf
*.ndf
# Business Intelligence projects
*.rdl.data
*.bim.layout
*.bim_*.settings
*.rptproj.rsuser
*- [Bb]ackup.rdl
*- [Bb]ackup ([0-9]).rdl
*- [Bb]ackup ([0-9][0-9]).rdl
# Microsoft Fakes
FakesAssemblies/
# GhostDoc plugin setting file
*.GhostDoc.xml
# Node.js Tools for Visual Studio
.ntvs_analysis.dat
node_modules/
# Visual Studio 6 build log
*.plg
# Visual Studio 6 workspace options file
*.opt
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
*.vbw
# Visual Studio 6 auto-generated project file (contains which files were open etc.)
*.vbp
# Visual Studio 6 workspace and project file (working project files containing files to include in project)
*.dsw
*.dsp
# Visual Studio 6 technical files
*.ncb
*.aps
# Visual Studio LightSwitch build output
**/*.HTMLClient/GeneratedArtifacts
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Paket dependency manager
**/.paket/paket.exe
paket-files/
# FAKE - F# Make
**/.fake/
# CodeRush personal settings
**/.cr/personal
# Python Tools for Visual Studio (PTVS)
**/__pycache__/
*.pyc
# Cake - Uncomment if you are using it
#tools/**
#!tools/packages.config
# Tabs Studio
*.tss
# Telerik's JustMock configuration file
*.jmconfig
# BizTalk build output
*.btp.cs
*.btm.cs
*.odx.cs
*.xsd.cs
# OpenCover UI analysis results
OpenCover/
# Azure Stream Analytics local run output
ASALocalRun/
# MSBuild Binary and Structured Log
*.binlog
MSBuild_Logs/
# AWS SAM Build and Temporary Artifacts folder
.aws-sam
# NVidia Nsight GPU debugger configuration file
*.nvuser
# MFractors (Xamarin productivity tool) working folder
**/.mfractor/
# Local History for Visual Studio
**/.localhistory/
# Visual Studio History (VSHistory) files
.vshistory/
# BeatPulse healthcheck temp database
healthchecksdb
# Backup folder for Package Reference Convert tool in Visual Studio 2017
MigrationBackup/
# Ionide (cross platform F# VS Code tools) working folder
**/.ionide/
# Fody - auto-generated XML schema
FodyWeavers.xsd
# VS Code files for those working on multiple tools
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets
# Local History for Visual Studio Code
.history/
# Built Visual Studio Code Extensions
*.vsix
# Windows Installer files from build outputs
*.cab
*.msi
*.msix
*.msm
*.msp
.skill-validator-results/
.DS_Store
.nuget/
validation_report.md
eval-results/
# Roslyn / C# language server cache files
*.lscache
.dotnet/
# Local design plan kept out of the PR
docs/design/pr-triage-workflows-plan.md
Reference in New Issue View Git Blame Copy Permalink