mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
main
151 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
36222bf32d |
Add first-class custom-agent evaluation coverage (#1165)
* feat(evaluation): add custom agent coverage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): address agent review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): reject linked fixture sources Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): preserve agent result invariants Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): fail closed on agent errors Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): preserve completion regressions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): preserve nested command quotes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): harden native agent evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): honor declared agent layout Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): resolve declared agent sources Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): secure agent path discovery Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): reject linked dependencies Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): centralize path safety checks Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): diagnose ambiguous dependencies Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): reject linked allowed roots Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): preserve skill agent isolation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): normalize dashboard evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): preserve agent gate semantics Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): fail closed on incomplete evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): preserve completion evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): reject overflowing durations Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): stage verified plugin skills Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): block shell network access Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): reject linked MCP config files Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): trust manual dispatch path safety Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): keep agent plugin activation diagnostic Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): count failed tool completions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evaluation): synchronize agent event capture Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
460a01882f |
Define and validate native Codex plugin support (#1164)
* fix: validate native Codex plugin support Separate Codex-compatible plugin components from GitHub Copilot custom agents, validate Codex manifest fields and MCP shapes, and document the native agent installation boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: strengthen Codex plugin verification Validate Codex MCP per-tool settings and add a pinned native smoke lane that installs the marketplace, discovers skills and MCP, and calls binlog_overview. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: avoid restarting Codex MCP smoke client Use the real tool call as the app-server initialization so the Linux smoke test does not start and tear down a redundant status client first. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: harden Codex validation and smoke cleanup Constrain smoke artifacts to a dedicated child directory, validate Codex manifest field shapes and required skill paths, and document the separate Agents SDK handoff model. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: align Codex smoke and hooks parsing Use the clean Codex installation's default provider for the direct MCP call and reject heterogeneous hooks arrays like the runtime parser. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: verify every Codex plugin skill Compare native skill discovery with the full repository inventory, match Codex lexical path rejection, and allow null optional MCP tool settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix: address Codex compatibility review Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
6cf3a72f3e |
Correct merged plugin release checkpoints
Stamp versions from the latest main first-parent authority: dotnet-test 0.2.22 and dotnet-test-migration 0.1.9. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5d007947-f964-4809-a31c-2d9f457a2ba3 |
||
|
Amaury Levé
|
949995c8fe |
Add MTP hot reload coverage evaluations (#1137)
* Add MTP hot reload coverage evals Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten hot reload activation grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload evaluation guards Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Cover hot reload setup validation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Catch unsupported config suggestions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep Test Explorer eval dormant Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload eval matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Catch inline hot reload install commands Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Narrow hot reload negative matcher exemptions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refine hot reload negation handling Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Anchor hot reload package matcher Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten hot reload eval matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload eval guidance matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Correlate hot reload activation and relaunch Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Handle inline hot reload warning commands Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Narrow MTP hot reload skill routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix hot reload routing description Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Exclude negated console host questions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep hot reload menu within validator limit Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten unsupported configuration matcher Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Require command-shaped hot reload relaunch Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten hot reload dormancy matcher Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reject hot reload project mutation guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix hot reload warning matcher exemptions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload warning and env matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Accept inline hot reload environment commands Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Allow env configuration in hot reload grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reject unsupported Test Explorer config usage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Scope hot reload env matcher exemptions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e8ed8473d9 |
Improve GPT-focused dotnet-test eval outcomes (#1122)
* Improve GPT-focused dotnet-test evals Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify coverage risk routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix coverage analysis dormancy routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make test audits discover workspace inputs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce local calendar migration semantics Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Improve Codex workspace analysis reliability Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Exclude comments from UTC call count Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Allow migration evals enough execution time Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Keep static detection scoped to C# Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Give migration scenarios reliable headroom Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Make coverage dormancy prompt advisory Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Recover safely from workspace tool failures Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Refine Codex migration and tagging recovery Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Simplify Codex skill recovery paths Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Correct local time eval semantics Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Tighten dotnet-test skill activation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Clarify skill routing contracts Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Harden DateTime migration grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Require explicit skill activation evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Require CSharp routing evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Harden static scan fallback safety Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Harden coverage reader fallback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Remove static detector routing trap Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 * Stabilize routing dormancy evaluations Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b62ad695-3f5f-482c-a4e6-6b47c654e366 |
||
|
Amaury Levé
|
0afcd3ee6d |
Add testability obstacle eval coverage (#1138)
* Add testability obstacle eval coverage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Avoid brittle testability eval graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Cover testability eval guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Check static seam eval behaviors Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Generalize static seam eval grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Require environment default invocation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Preserve static feature flag API Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Strengthen static seam eval checks Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Improve testability obstacle guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1eb71366fe |
Expand static test pairing to more languages (#1136)
* Expand static test pairing to more languages Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix PowerShell pairing eval path Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Cover PowerShell suggested test path Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Avoid false test detection in Kotlin and Swift Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Narrow pairing eval command matcher Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align polyglot pairing documentation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ac8f41264b |
Improve non-passing dotnet-test scenarios (#1114)
* Improve non-passing dotnet-test scenarios Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address dotnet-test review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ba527a4a-6047-485a-9c14-d81cad8772ef --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ba527a4a-6047-485a-9c14-d81cad8772ef |
||
|
Amaury Levé
|
cb9651deae |
Pilot repository-specific overlays for dotnet-test skills (#1115)
* Pilot repository overlays for test skills Add revisioned, repo-specific overlay bindings for selected dotnet-test skills and eval coverage for valid, scoped, and stale bindings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Allow attributed overlay references Add the JeremyKuhne source repositories to the reviewed external reference allowlist. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address overlay pilot review feedback Strengthen the MSTest eval against duplicate/default policy output and make the suite fixture parameter drive its test filter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify invalid overlay fallback Make invalid overlays fall back to portable guidance and accept natural revision mismatch explanations in the stale-binding eval. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix overlay precedence wording Restore the separator between repository and portable precedence rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Use a literal overlay project path check Prevent the scaffold eval grader from interpreting dots in the expected project path as regex wildcards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce command-only overlay evals Reject shell, edit, and create tools in both run-tests overlay prompts that prohibit execution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
57733bebc8 |
Keep test agent state out of commits (#1108)
* Keep test agent state out of commits Move broad test-generation pipeline state to host scratch storage, worktree-specific Git metadata, or OS temp, and enforce the exclusion in evals. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Clarify absolute test agent state path Use Git's explicit absolute path formatting in both test-generation entry points. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Prune Git metadata from test agent eval guards Avoid scanning nested repositories and align the remaining TESTAGENT_DIR placeholder with the documented format. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Clarify test agent command handoff Require callers to provide exact commands, excerpts, or absolute TESTAGENT_DIR document paths to command-running sub-agents. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Reject all repository-local testagent entries Match .testagent by name regardless of whether it is a directory, file, or symlink while continuing to prune Git metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Verify external test agent artifacts Restore broad-run artifact checks at the Git metadata path and pass the researched lint command and state directory to the linter agent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Make testagent eval guards constant time Check only the forbidden workspace-root path, including broken symlinks, instead of recursively traversing dependency trees. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Broaden comprehensive test generation Treat explicit requirements as the floor for broad suites and add mutation-relevant equivalence-partition and invariant coverage without test-count padding. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Fix external artifact grader quoting Run state checks directly in the harness shell so TESTAGENT_DIR expands after assignment, with an isolated command probe covering valid and forbidden states. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Run broad skill evals in Git worktrees Initialize the seven broad evaluation roots as Git repositories so TESTAGENT_DIR resolves deterministically and external artifacts remain verifiable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Clarify non-stageable test agent state Describe the real invariant across the pipeline: state may live under .git metadata but must never be version-controlled workspace content or appear in git status. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Standardize intermediate test state contract Use one TESTAGENT_DIR placeholder, clearer intermediate-state terminology, and detect stageable research, plan, or status files regardless of directory name. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Use one Git root in workspace integrity eval Baseline the fixture from the evaluation root so stageable intermediate-state files remain visible to the directory-independent guard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Prune Vitest dependencies from state scan Exclude node_modules through per-eval Git metadata so stageable state detection remains fast without modifying fixture content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 * Strengthen focused intermediate-state guards Separate shell execution, reject Git-metadata files on focused runs, include ignored state files, and prune node_modules with a pathspec exclusion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 --------- Copilot-Session: 35c50c03-2dda-4919-981e-fd5f6b7938f0 |
||
|
Amaury Levé
|
30b30efe04 |
Improve dotnet-test evaluation outcomes (#1106)
* Improve dotnet-test evaluation outcomes Refine routing and evidence-backed guidance for non-passing dotnet-test skills, and repair evaluation fixtures and prerequisites. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3e140b67-0d44-4c6c-81c0-2ec02ff78ef8 * Clarify eval dependency constraint Scope package-install prohibitions to project dependencies so harness analyzer setup is not contradictory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3e140b67-0d44-4c6c-81c0-2ec02ff78ef8 * Fix wrapper skill dormancy routing Front-load the already-abstracted exclusion so wrapper requests for existing IFileSystem or TimeProvider seams remain dormant. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3e140b67-0d44-4c6c-81c0-2ec02ff78ef8 * Harden payment fixture validation Add explicit null guards and report the precise amount property for invalid payment values in the well-written fixture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3e140b67-0d44-4c6c-81c0-2ec02ff78ef8 * Align TypeScript pairing eval path Expect the analyzer's sibling tests/cart convention after merging the updated path inference logic. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3e140b67-0d44-4c6c-81c0-2ec02ff78ef8 * Complete payment fixture contract Validate the supported currency set so the data-driven currency test exercises observable production behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3e140b67-0d44-4c6c-81c0-2ec02ff78ef8 --------- Copilot-Session: 3e140b67-0d44-4c6c-81c0-2ec02ff78ef8 |
||
|
Amaury Levé
|
7caacad2ce |
Harden CRAP score reliability (#1109)
Reject incomplete coverage evidence, prefer measured complexity, and correct score expectations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
00b3bac07d |
Improve dotnet-test results for GPT-5.6 Sol (#1105)
* Improve dotnet-test scoring for Sol Use retained judge evidence to sharpen model-facing decisions, fix static test-path suggestions, and remove misleading eval ambiguity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5ccef003-0815-4afb-a92f-23aeab5080b6 * Cache sibling test directory lookup Precompute source-family test conventions once so path suggestions stay linear across large repositories. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5ccef003-0815-4afb-a92f-23aeab5080b6 * Require exact VSTest project target Reject substring filenames in the coverage command grader while accepting the requested project path across path separators. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5ccef003-0815-4afb-a92f-23aeab5080b6 --------- Copilot-Session: 5ccef003-0815-4afb-a92f-23aeab5080b6 |
||
|
github-actions[bot]
|
6cc9e1e1fc | Weekly plugin version sync | ||
|
Amaury Levé
|
1dd5e2b7d2 |
Improve test gap correctness and cost (#1080)
* Improve test gap correctness and cost Require complete public-outcome inventories, suppress inert and unobservable mutation candidates, and bound focused execution and output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten test gap analysis execution scope Reserve mutation execution for explicit verification, require complete authorization outcome ledgers, and calibrate exact-value and suite-strength reporting from the latest Luna evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Suppress speculative test gap findings Keep focused analyses within the named risk, require concrete edits to existing code, and cover complete money-math composition without future-only or trivial state findings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify observable error semantics Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Resolve remaining review wording Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refine test gap scope and boundaries Convert named risks into observable outcome allowlists, enumerate ordered guard partitions, and align the advisory rubric with static source-to-assertion evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Align advisory evaluation and verification Exclude derived non-monetary predicates, require canonical verification scripts for test additions, and remove the advisory rubric's incentive for unnecessary mutation execution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Address test gap review clarity Keep the output contract limited to response components and use clearer mutation terminology in the advisory rubric. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Correct test gap evidence and calibration Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Cover retry classifier edge partitions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Reject already-killed mutation gaps Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Clarify retry cutoff rubric Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Keep test tagging prompts dormant Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e * Require production-change routing intent Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a626035-c70f-4ba6-97f0-e076b8a5fd1e |
||
|
Amaury Levé
|
2b9056bd91 |
Separate dormancy evidence and improve affected evals (#1079)
* Separate dormancy from preference evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Preserve legacy dashboard preference semantics Default scenarios without schema v4 preference eligibility metadata to eligible so historical schema v3 records remain consistent with their original gate evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f * Cover activation contract reporting precedence Exercise the combined activation-contract failure and underpowered preference state so consolidation keeps the routing failure as the headline result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f * Tighten dormancy scalar parsing Require complete YAML false tokens so prefix values cannot alter preference eligibility, and keep activation-contract summary grammar correct. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f * Preserve legacy dashboard gate labels Mark only schema v4 dashboard evidence with dormancy exclusions so historical schema v3 records retain their original all-stimulus vote labeling. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f * Improve skills from evaluation evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f * Harden dormancy evaluation contracts Match PyYAML dormancy parsing, preserve unmatched annotations, prioritize gate-eligible report evidence, and make the MTP Boolean grader exact without case sensitivity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f |
||
|
YuliiaKovalova
|
be7b56028c |
Fix Codex discovery of the dotnet-msbuild binlog MCP server (#1070)
* Fix Codex discovery of the dotnet-msbuild binlog MCP server .codex-plugin/plugin.json declared "mcpServers": "./.mcp.json", but the file was packaged at .codex-plugin/.mcp.json. Codex resolves manifest resource paths against the plugin root, so it looked for plugins/dotnet-msbuild/.mcp.json and never found the binlog server. Declare the server inline in .codex-plugin/plugin.json, matching the root plugin.json and .claude-plugin/plugin.json, and drop the unreachable file. Add a packaging regression check to skill-validator: every companion manifest must declare the same MCP servers as the root plugin.json, and a manifest referencing an external .mcp.json must resolve it from the plugin root the way hosts do. skill-check.yml already runs `skill-validator check` over plugins/* on every PR, so this now blocks in CI. A test also loads the shipped dotnet-msbuild manifests and asserts binlog is present in each. Fixes #1069 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Report non-object JSON roots as validation errors JsonElement.TryGetProperty throws InvalidOperationException when the root value is not an object, so a manifest or referenced .mcp.json that is valid JSON but not an object (null, array, string) crashed skill-validator instead of producing a validation error. Check the root kind while reading and surface it as a structured error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7e5b5989f6 |
Clarify platform detection and MTP hot reload boundaries (#1076)
* Clarify platform and hot reload boundaries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address platform and hot reload review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Tighten platform detection response scope Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Address follow-up platform review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Improve platform and hot reload eval discrimination Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Allow multiline platform causality evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Correct dotnet watch restart syntax Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Harden xUnit hot reload fixture Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Clarify watch project ownership Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Expand platform import-order coverage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb * Complete platform evidence explanations Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d3d8e36b-fdc3-470f-81c8-dc8d696590fb |
||
|
Amaury Levé
|
26caf0ecab |
Scale coverage analysis to request scope (#1078)
* Scale coverage analysis to request scope Gate CRAP and full reports behind explicit intent, reconcile coverage arithmetic, and strengthen only evidence-backed eval outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address coverage eval review feedback Broaden no-command guards for Markdown formatting and clarify plateau recommendations across both uncovered members. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8 * Improve coverage analysis recovery Use the latest cross-model evidence to recover from failed artifact reads, require source-safe branch interpretation, make target combinations concrete, and enforce CRAP-first refactoring risk ranking. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc01c723-1c85-446e-873a-c4bd26fdb0e8 |
||
|
Abhitej John
|
f65b81bd71 |
Stabilize skill menu and evaluation retries (#1072)
* Stabilize skill evaluation retries Reduce the dotnet-test skill menu below the runtime budget and retry only transient baseline/skilled session.idle timeouts without replacing successful evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden evaluation retry matching Normalize eval paths before retry matching and require stable shardKey identities so recovery fails closed when record identity is uncertain. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4c7484eb-1fa5-4332-841e-e924031c75a2 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4c7484eb-1fa5-4332-841e-e924031c75a2 |
||
|
Amaury Levé
|
47dd914532 |
Add Claude plugin manifests (#1071)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0be2ef95a5 |
Fix nested testability override restoration (#1077)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0c27460786 |
Clarify injected dependency scan exclusions (#1074)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
2c3d9807b0 |
Merge pull request #1059 from dotnet/abhitejjohn-skill-value-dashboard
Add per-skill "Skill Value" view to the evaluation dashboard |
||
|
Amaury Levé
|
a77e7da573 |
Fix MSTest version and retry guidance (#1075)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
09e5babf49 |
Reduce dotnet-test skill menu size
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 032f6f50-dd65-4805-b7af-78755f8bbc77 |
||
|
Amaury Levé
|
d3921f7418 |
Strengthen testability skill evaluations (#1057)
* Strengthen testability skill evaluations Raise four dotnet-test evals to eight independent stimuli, add validated fixtures, and resolve code-testing-agent orphan fixtures without speculative routing changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Relax promo-code eval grader Accept deterministic suffix values beyond one hard-coded literal and match common PascalCase test names. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84f7c88c-c8e7-4d8f-96c9-421de725ab31 * Improve testability skill reliability Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84f7c88c-c8e7-4d8f-96c9-421de725ab31 * Refine testability obstacle graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84f7c88c-c8e7-4d8f-96c9-421de725ab31 * Accept qualified Random seams Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84f7c88c-c8e7-4d8f-96c9-421de725ab31 * Improve ambient seam compatibility Replace the C# 12 primary constructor in the copyable Scope sample with a conventional constructor so the guidance works in projects using older language versions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84f7c88c-c8e7-4d8f-96c9-421de725ab31 * Improve testability skill reliability Refine routing and execution contracts from exact losing transcripts, strengthen behavioral eval checks, and add isolated C# fixtures without increasing repeated runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84f7c88c-c8e7-4d8f-96c9-421de725ab31 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 84f7c88c-c8e7-4d8f-96c9-421de725ab31 |
||
|
Amaury Levé
|
3cd1337923 |
Improve dotnet test skill routing and quality (#1051)
* Improve coverage and tagging routing Clarify two-sided sibling boundaries for coverage arithmetic, branch gaps, trait classification, and report-only frameworks. Replace prompt-echo graders while preserving outcome rubrics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address routing review feedback Make coverage and tagging boundaries reciprocal, preserve convention-based Go tagging, and make outcome graders robust to multiline answers and equivalent uncovered wording. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Harden routing and grader equivalence Preserve key anti-pattern triggers, complete coverage-prioritization handoffs, generalize convention-based tagging, and accept natural multiline and semantic grader equivalents. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Bound coverage outcome graders Prevent substring false positives, accept equivalent branch and zero-coverage wording, preserve multiline Go classification, and restore the migration routing boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Clarify project coverage outcomes Scope hotspot thresholds to project-wide coverage, preserve behavioral threshold routing, and accept equivalent arithmetic and classification wording without weakening deterministic checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Tie graders to evaluated outcomes Scope prioritization by project versus named target, require arithmetic context for 34 lines, preserve member identity checks, and bind Go semantic alternatives to divide-by-zero. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Refine CRAP routing and eval graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Tighten coverage evaluation contracts Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Bind coverage graders to evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Close remaining eval grader gaps Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Use positive coverage grader assertions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * De-cue semantic outcome graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Address routing review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Scope anti-pattern coverage handoff Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Complete test routing boundaries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Harden coverage and tagging graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Bound coverage grader literals Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Complete coverage routing constraints Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Require coverage and taxonomy evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Enforce report-only fixture scope Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Move coverage arithmetic to rubric Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Fix coverage routing and table graders Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Add reciprocal routing guards Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Test tagging handoffs in sibling skills Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Tighten coverage and distribution boundaries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Harden routing and report shape evals Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Resolve final coverage routing overlaps Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Require Go test rows in reports Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Improve cross-model test skill quality Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 * Remove cue-sensitive anti-pattern grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bfd14b28-8b12-4a2a-8735-bf4e91865e14 |
||
|
Amaury Levé
|
fb3d055a49 |
Improve .NET test project scaffolding quality (#1058)
* Improve .NET test project scaffolding quality Clarify create, repair, and no-op routing; broaden the eval with loss-tolerant existing-project scenarios while reducing repeated-run cost. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Improve test skill evaluation reliability Fix natural-language routing gaps and scaffold completion behavior. Replace process-coupled code-testing graders with outcome checks, add predeclared TypeScript breadth, and reduce repeated-run cost. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 06f20828-94e0-4f48-aaff-6a2f46103d23 * Verify solution-level test discovery Run the existing-project registration grader through Catalog.sln so it proves the CI entry point, not only direct project execution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 06f20828-94e0-4f48-aaff-6a2f46103d23 * Improve test skill evaluation discrimination Strengthen focused and broad test generation decisions, avoid xUnit v3 template churn, and add objective next-experiment breadth. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 06f20828-94e0-4f48-aaff-6a2f46103d23 * Fix scaffold workspace discovery Use the task workspace rather than the skill installation directory, preserve complete verification evidence, and retire the unproven adjacent test-generation experiment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 06f20828-94e0-4f48-aaff-6a2f46103d23 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 06f20828-94e0-4f48-aaff-6a2f46103d23 |
||
|
Amaury Levé
|
3adf7cbc91 |
Improve run-tests routing and eval coverage (#1052)
* Improve run-tests routing and eval coverage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address run-tests review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Clarify no-build test routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Address remaining run-tests review feedback Tighten deterministic eval guards, correct MTP fixture signals, and clarify runner-specific guidance so validator and routing behavior match the documented contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Tighten run-tests evaluation scenarios Make diagnostic and dump graders command-specific, keep advisory scenarios tool-free, and align routing and reference guidance with the actual consumer graph. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Cover remaining run-tests routing paths Add no-tool no-build, bridged NUnit, and coverage scenarios; enforce direct argument placement and make the imported-props MTP fixture executable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Address post-merge eval review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Align run-tests and hot-reload review coverage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Harden test runner boundaries and fixtures Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Complete test routing eval fixtures Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Align advisory routing assumptions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Cover incomplete test bridge configurations Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Guard read-only platform evaluation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Reduce test skill eval overfitting Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Scope VSTest TRX negative grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Tighten test skill response contracts Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Reject positional native MTP targets Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Focus platform detection responses Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Progressively disclose test command mode Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Require TUnit executable fallback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 * Harden test routing from rerun evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bde0fbcd-b442-4ab7-b628-4beb1574a597 |
||
|
Amaury Levé
|
98a6816daa |
Improve test gap analysis routing and efficiency (#1048)
* Improve test gap analysis efficiency Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Clarify polyglot test gap routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix mutation catalog markdown lint Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Render logical OR correctly in mutation table Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Correct test gap analysis selection Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Clarify test gap request classification Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Distinguish static mutation candidates Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Improve test gap analysis consistency Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Fix logical OR table rendering Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Clarify unverified mutation candidates Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Name reportable mutation gap states Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Improve mutation gap verification consistency Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Clarify test gap evidence states Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Align test gap routing markers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Align test writing gap criteria Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 * Clarify discovery guidance wording Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: db4db104-41a3-47d9-afba-24e260939c48 |
||
|
Amaury Levé
|
5055e54a57 |
Improve MSTest skill routing and eval quality (#1054)
* Improve MSTest skill routing and eval Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address MSTest eval review feedback Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb7d3fa4-c617-4522-ace3-eab65915af34 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb7d3fa4-c617-4522-ace3-eab65915af34 |
||
|
Amaury Levé
|
ba4f404911 |
Retire invalid filter-syntax direct eval (#1050)
* Retire invalid filter syntax eval Remove the direct identical-arm evaluation and document consumer-level coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align reference skill eval guidance Remove stale filter-syntax direct-eval guidance called out in review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Report reference skill coverage as N/A Avoid treating model-disabled skills without direct evals as uncovered. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea380edb-63a9-40dc-9162-938c4d7754dd --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea380edb-63a9-40dc-9162-938c4d7754dd |
||
|
Amaury Levé
|
8fb17964bc |
Improve test smell skill quality and eval power (#1056)
* Improve test smell skill quality Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Use conventional empty class bodies Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Improve test smell calibration Align workspace discovery and false-positive decisions with the losing eval transcripts, correct contradictory fixtures, and make graders outcome-focused. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Make eval regexes multiline-safe Allow outcome evidence to match across line breaks in generated review output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Make notification fixtures observable Record notification identifiers so post-wait assertions can fail, while preserving fixed sleeps as the intentional smell under evaluation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Strengthen test smell stop conditions Require workspace discovery, preserve formal skip and file classifications, prevent clean-suite false positives, and reduce lexical grader coupling. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Use conventional exception class body Keep the fixture compatible with compilers that do not accept semicolon-only class declarations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Remove brittle eval gates Rely on outcome rubrics instead of narrow lexical matches and keep the Sensitive Equality fixture culture-stable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Add JUnit eval exit check Fail fast on empty or failed trial output while dropping a redundant severity-word matcher. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Make remaining eval regexes multiline-safe Allow concise verdict and async-fix patterns to match wrapped model output across line breaks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b * Preserve non-catalog validity findings Keep formal smell classification while separately reporting proven test-validity defects that do not belong to the taxonomy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 718f824b-6b86-4d7f-9428-2d7a8908e95b |
||
|
github-actions[bot]
|
2d00e600ac | Weekly plugin version sync | ||
|
Amaury Levé
|
5a06b20cc9 |
Support classic .NET test projects in dotnet-test (#993)
* Support classic .NET test projects Teach dotnet-test skills and agents to preserve non-SDK projects, packages.config dependencies, explicit compile registration, legacy runners, and version-compatible MSTest APIs. Add regression evals for generation, execution, coverage, and authoring. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68 * Address classic test fixture review Tighten the MSTest version grader, make the runner fixture assertion behavioral, and use nameof for the guarded parameter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68 --------- Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68 |
||
|
github-actions[bot]
|
c43e900fb1 | Weekly plugin version sync | ||
|
Amaury Levé
|
f4f28415da |
Add golden end-to-end test generation scenarios (#997)
* Add golden test generation scenarios Add reusable scaffolding and testability skills plus eval fixtures for zero-to-one, partial-suite gap, and ambient-dependency workflows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 * Address golden scenario review feedback Use the SDK-style project type GUID in the solution fixture and keep the discount lower bound explicitly decimal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 * Fix eval grep alternation Use extended regular expressions so forbidden-call and fake-name graders reliably match each alternative without YAML escape ambiguity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 * Increase golden eval discrimination Bundle isolated-run guidance, sharpen plugin routing, and add solution-filter, nested-scope, and fake-time delay decisions. Repeat the expensive end-to-end scenarios twice to clear the observed tie rate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 * Specialize golden test workflows Restore code-testing and migration skills to main, promote zero-to-one scaffolding to its own skill, extend verified gap analysis through focused test additions, and concentrate testability evaluation on nested ambient-scope behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 * Increase specialized eval margin Fix xUnit v3 discovery and no-op scaffolding, constrain smoke-test scope, double the clean gap-analysis evidence, and add two deterministic static-identity scenarios matching the strongest testability delta. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 * Remove unstable eval scenarios Drop consistently losing or inert scenarios, add SDK solution registration and empirical retry-policy mutation coverage, and sharpen activation for specialized workflows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 * Align testability workflow routing Narrow scaffold no-op routing, separate targeted obstacle handling from migrated-code test generation, and add an agent scenario covering migration plus deterministic tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 --------- Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7 |
||
|
Amaury Levé
|
69fe56a74f |
Improve dotnet-test eval power and resilience (#989)
* Improve dotnet-test eval coverage and efficiency Address remaining high-confidence items from #899 by bounding the code-testing pipeline and adding eval coverage for grade-tests and find-untested-sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 * Fix dotnet-test eval activation and quality Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 * Strengthen dotnet-test skill activation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 * Improve dotnet-test eval signal and resilience Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Address eval review findings Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Remove non-measuring smell eval trials Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Fail evaluation on unexpected adapter errors Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Stabilize assertion quality evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Strengthen assertion quality routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a * Treat adapter SIGKILL as failure Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a --------- Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 Copilot-Session: a492694c-6e0f-4af4-8b5e-18331568499a |
||
|
github-actions[bot]
|
62ff166f2b | Weekly plugin version sync | ||
|
Amaury Levé
|
2124a6e351 |
Flag reference skills that carry a direct-activation eval (#899) (#983)
`disable-model-invocation: true` drops a skill from the CLI's `<available_skills>` menu, so the experiment's skilled arm -- which loads exactly one skill -- cannot reach it either. The gate already reasoned about this, but `report_uncovered()` skipped any skill that had an eval, so it only reported the *better* case (no eval, visibly zero evidence) and stayed silent on the worse one (an eval scoring baseline against baseline and labelling the result a pass or a fail). Two evals landed in that blind spot after the reasoning was written down: tests/dotnet-test/platform-detection (#974) and tests/dotnet-test/filter-syntax (#976). Both READMEs still described them as deliberately un-evaluated. - report reference skills that carry a direct eval, naming the spec - self-test both sides of the trigger via a new silent_case helper - correct eng/eval-quality/README.md and plugins/dotnet-test/README.md Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3d666b74-a00f-4ed0-a3ce-2d3427cf0d23 |
||
|
Amaury Levé
|
573938df09 |
Close the remaining dotnet-test eval follow-ups (#899) (#971)
* Close the remaining dotnet-test eval follow-ups from #899 Every non-agent dotnet-test eval now clears the 5-trial floor, the two cost P1s are addressed, and the reference-skill coverage gap is recorded as a decision instead of a standing warning. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee * Remove the duplicate block that cloned a grade-tests scenario, and gate it The 'production code available' scenario carried a leftover tail from the edit that moved the 'production code unavailable' one. YAML keeps the last duplicate key, so every field of the new scenario was silently overwritten: it shipped as a byte-identical rerun of its predecessor and never loaded the dotnet-production-available fixture it was built around. Parsing the spec and counting stimuli - which is what verified this PR - returns the intended 5 names either way, so only the parser can see it. The eval-quality gate now loads specs with a duplicate-key-strict loader (failing check 9), with a self-test case and the incident recorded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee |
||
|
Amaury Levé
|
f2eb897a12 |
Fix dotnet-test findings from the refreshed cross-family eval (#899) (#945)
* Fix dotnet-test findings from the refreshed cross-family eval (#899) Every change below is driven by judge evidence from the losing trials of the refreshed 5-family dotnet-test matrix (runs 30108473397 + recovery runs), not by style preference. Eval measurement fix — the "discovery" P2s were an artifact: - assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging each have a decline stimulus with `constraints.reject_skills: ["*"]`, so the skill cannot activate there by construction. Without `expect_activation: false` the adapter counted those dormant runs as missed activations, which is exactly the 75-88% invocation rates reported in the scorecard. Annotating them (the convention already used by agent.test-quality-auditor) removes the false signal; the non-activations were the only ones observed for these skills. Skill fixes: - test-gap-analysis: baselines won by actually running the suite while the skill reasoned statically and reported survivors that the tests in fact kill. Added Step 4b: confirm every reported survivor by applying it, re-running the covering tests, and reverting; fall back to reasoning only when the suite cannot run, labelled unverified. Calibrated severity down for strong suites. - test-anti-patterns: baselines won on depth, not polish. Added a depth bar — account for every test in scope, give exact expected values in fixes, name the adjacent error-path/boundary gaps, and keep counts consistent. Trimmed three pitfall rows that duplicated the calibration step so the skill stays under the profiler's "comprehensive" threshold. - detect-static-dependencies: losses were all counting accuracy. One authoritative total (no findings parked outside it), classify by the resource touched rather than by the `static` keyword, exclude pure helpers such as Path.Combine from the needs-wrapping total, require file:line, and add the missing randomness/culture/serialization categories. - test-smell-detection: the calibration rule told models to downgrade Sleepy Test for integration tests, which is what lost both losing scenarios. Fixed sleeps now stay High in any category; Mystery Guest and Eager Test still downgrade. - crap-score: losses came from estimating coverage after collection failed. Added the dotnet-coverage/ReportGenerator recovery path and a hard rule never to publish a CRAP score built on assumed coverage. - coverage-analysis: answer the asked question first, reconcile every number against the script output, and list every below-threshold member instead of declaring one method the entire gap. - migrate-static-to-wrapper: migrate exactly what was requested (no adjacent DateTime.Now rewrites, respect intentional-use comments) and never report "build succeeded" when the build or restore failed. - code-testing-agent: quote each requirement verbatim in the evidence table so multi-condition requirements map to a test that covers the whole combination, and cite a clean run rather than a coverage attempt that exited non-zero. Validation: skill-validator check passes (20 skills, 10 agents); markdownlint clean; eval specs parse and the adapter now reports all four decline stimuli as expect-dormant. Refs #899 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa * Strengthen underpowered dotnet-test evals from the PR 945 eval run The PR eval reported 5 of 10 skills as "no credible improvement". Reproducing the gate arithmetic from the artifacts shows the dominant cause is statistical power, not skill quality. The gate is `mean > 0 AND ci_low > 0` with a t-based CI over per-trial scores, which reduces to `sqrt(n) * (mean/sd) > t(n-1)`. The required mean/sd ratio is brutal at small n: n=2 -> 8.98 n=3 -> 2.48 n=4 -> 1.59 n=5 -> 1.24 n=6 -> 1.05 n=8 -> 0.84 Recomputing each reported CI from the per-trial scores reproduces the published numbers exactly, which confirms the mechanism: crap-score [0.4,1.0,0.4] n=3 CI [-0.261, 1.461] migrate-static-to-wrapper [1.0,0.4,0.4] n=3 CI [-0.261, 1.461] test-gap-analysis [0.4,0,0.4,0.4] n=4 CI [-0.018, 0.618] test-anti-patterns [0.4,0,0.4,0,0,0.4] n=6 CI [-0.030, 0.430] code-testing-agent [0,0.4] n=2 CI [-2.341, 2.741] crap-score and migrate-static-to-wrapper won 100% of their trials (3W/0T/0L) and still failed: at n=3 nothing short of three identically-sized wins can clear the gate. That is a property of a thin eval, not of the skill. Scenario counts are raised with discriminating cases, four of them by wiring up fixtures that were already committed but had no stimulus referencing them: - test-gap-analysis 4 -> 6, using the orphaned `report-quality` fixture (trivial auto-properties and an auto-generated .g.cs to skip, private helpers reachable only through the public API, and a deliberately weak Assert.IsTrue that cannot kill arithmetic mutations) and the orphaned `rust-error-propagation` fixture (an untested `?` propagation path and an untested `<=` boundary). - test-anti-patterns 6 -> 8, using the orphaned `pytest-mixed` fixture (which also checks the calibration rule that pytest's bare `assert` must not be flagged) and the orphaned `assertion-problems` fixture (which separates Critical false-confidence assertions from a Low-severity message nit). - crap-score 3 -> 6, with a new `refactor-required` fixture whose numbers are self-consistent: ApplySurcharges has complexity 13 behind a stale "Complexity: 4" comment (CRAP 28.4, needs 77.2% coverage), ClassifyAccount has complexity 17 so coverage alone can never reach CRAP < 15, and RoundToCurrency is 100% covered so its CRAP equals its complexity exactly. - migrate-static-to-wrapper 3 -> 5, adding a DateTimeKind-preservation scenario over the existing fixture and a new `static-helper` fixture where a static class must gain an ambient TimeProvider seam without breaking its callers. - code-testing-agent 2 -> 3, with a compact C# fixture that must extend an existing suite to the untested method only. This eval stays the weakest: each scenario is expensive, so raising `runs` is a better lever than adding more heavyweight scenarios. Verification: - every eval spec parses and all 254 fixture references resolve - the three new fixtures compile; the shipping-quotes fixture restores, builds and its three seed tests pass under `dotnet test` in a clean workspace - skill-validator check passes (20 skills, 10 agents) - markdownlint clean Refs #899 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa * Address review feedback on fixture and counting wording - BillableWeightTests: the ZeroOrNegative test only asserted the zero case, so its name overstated what it covered. Made it data-driven over 0 and -1 so the name matches the assertions. This matters more than usual here: the file is the seed suite for a test-quality eval, and a misleading test name is exactly what these skills are supposed to flag. - detect-static-dependencies: the Step 3 lead-in said to count each "static call pattern", which contradicted the rule immediately below it that instance members reaching the same untestable resource must also be counted. Reworded to "call site" and made the instance-member inclusion explicit. Verified: the fixture restores, builds and now passes 4 tests (was 3); skill-validator check passes; markdownlint clean. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa |
||
|
Amaury Levé
|
71414ce000 |
Improve dotnet-test eval coverage and efficiency (#917)
* Improve dotnet-test eval coverage and efficiency Address remaining high-confidence items from #899 by bounding the code-testing pipeline and adding eval coverage for grade-tests and find-untested-sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 * Fix dotnet-test eval activation and quality Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 * Strengthen dotnet-test skill activation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c5c1a52-4f99-49d6-b503-1bec713a6e98 |
||
|
Amaury Levé
|
4df4da469a |
Upgrade agentic workflows and fix stale PR cleanup (#916)
* chore: upgrade gh-aw runtime * fix: paginate stale pull request cleanup Upgrade the generated agentic workflow assets and ensure stale PR discovery includes every result page and draft pull requests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 968a22c2-327f-4d26-8f86-1c59bcd323ea * Improve dotnet-test eval coverage and efficiency Address remaining high-confidence items from #899 by bounding the code-testing pipeline and adding eval coverage for grade-tests and find-untested-sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 * fix: address agentic workflow review Pin the Copilot setup checkout action and include the cutoff date in stale PR search results. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 968a22c2-327f-4d26-8f86-1c59bcd323ea * Improve test migration skill guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5e19d263-02a6-45b1-9cfb-424fa4d10863 * fix: add fixture namespace imports Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a464e6e4-3e45-41fe-b17d-887c8cb8a448 * test: assert GetOrderById grade Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a464e6e4-3e45-41fe-b17d-887c8cb8a448 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Amaury Levé
|
0ca7f8921d |
Fix migrate-static-to-wrapper DateTimeKind regression (#899) (#906)
The replacement mapping instructed models to convert TimeProvider's DateTimeOffset back to DateTime via .DateTime, which yields Kind=Unspecified and silently changes behavior vs the original DateTime.UtcNow (Utc) / DateTime.Now (Local). The eval rubric penalizes exactly this behavioral change, a likely cause of the frontier-model regression flagged in the dotnet-test scorecard. Use .UtcDateTime / .LocalDateTime to preserve Kind, and add a guardrail plus validation/pitfall notes about matching the target member type and not forcing DateTime through DateTimeOffset. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 69261256-256e-4e4c-9bdc-932d5ca8e5df |
||
|
Amaury Levé
|
79a2ada302 |
Fix dotnet-test plugin timeout + skill activation issues (#875)
* Fix dotnet-test plugin timeout + skill activation issues Activation: - filter-syntax eval: both scenarios set expect_activation: false. The skill is a reference skill (disable-model-invocation/user-invocable false) loaded by run-tests/mtp-hot-reload; it can never self-activate — this was an eval mis-spec, not a skill bug. - writing-mstest-tests 'Fix swapped Assert.AreEqual arguments': expect_activation false. The rubric requires diagnosing the root cause unaided, so no invocation is expected (notActivated 11/11 both arms). - test-smell-detection: relax the over-restrictive 'INVOKE ONLY when explicitly asked for the testsmells.org catalog' guidance so nuanced named-smell reviews with justified severity also activate (both-arm notActivated gap). Funded within the 15k skill-menu budget by compressing redundant framework parentheticals to the compact style already used by sibling skills. Timeout: - run-tests 'Filter xUnit v3 tests by class pattern and trait using query filter language': raise 240 -> 360 (observed peak ~178s, closest to its cap). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address /evaluate: revert test-smell regression, remove untestable filter-syntax eval Log-driven iteration on the eval results (run 29019511076): Priority 1 - regression fix: revert the test-smell-detection description broadening. results.json shows it reliably regressed 'Decline request to write new tests from scratch' (perRunScores [-0.19,-0.10,-0.22]; plugin overallJudgment -0.28, quality 5.0 baseline -> 4.3 skilled) while NOT fixing its target 'Distinguish reasoned skips' (isolated 5.0 vs 5.0 tie failing on judge position-bias, CV=114% - not description-fixable). Net-negative, so restore the pristine description. Priority 3 - remove the structurally-untestable filter-syntax eval. filter-syntax is disable-model-invocation/user-invocable:false, so it can never activate and can never show a skilled-vs-baseline delta - both scenarios fail by construction regardless of expect_activation. Peer reference skills (platform-detection, code-testing-extensions) have NO eval.yaml; filter-syntax was the lone exception. Its filter functionality is already covered by run-tests scenarios (which load filter-syntax as an additional_required_skill). Following precedent, remove the eval rather than leave two permanently-red cases. Kept from the prior commit: run-tests query-filter timeout 240->360 (harmless hard-cap headroom) and writing-mstest 'Fix swapped Assert.AreEqual' expect_activation:false (documents the unaided-by-design scenario and prevents a spurious hard activation failure). Not done - plugin-arm sibling deflections: results.json shows every failing plugin-arm scenario has detectedSkills=[] (NO sibling captured the turn), the rendered menu is under the 15k budget (descriptions are shown), and the verdict failures are dominated by high-variance efficiency noise (CV 60-3000%, several with quality improved). Per docs/InvestigatingResults.md there is no captor to deflect and description edits would not help; with zero budget headroom, adding speculative deflections is unjustified. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests evals: flip expect_tools:[bash] to reject_tools to fix activation Root cause of run-tests plugin+isolated non-activation (skill_not_activated gate): every run-tests scenario is a 'show me the exact command' knowledge question, yet each carried expect_tools: [bash] — which is an ASSERTION that bash was used. That actively forced the model into empirical exploration (dotnet test --help, trial runs, web_fetch) instead of consulting the run-tests skill, and a clean knowledge-only answer would FAIL the assertion. Evidence from run 29253857976: run-tests failed to activate even in the ISOLATED arm (where it is the only skill loaded) on scenarios as easy as base=1.0 — proof the model was bypassing the skill via bash rather than not needing it. The forced bash use also drove the token/tool/ time overhead that produced the negative efficiency scores, and one scenario hit the 360s timeout doing web exploration. Fix: convert all 15 run-tests scenarios from expect_tools: [bash] to reject_tools: [bash, edit, create]. These scenarios only assert on the recommended command text (output_matches) — none run tests (no run_command_and_assert, no file assertions) — and the model can still inspect fixtures via view/glob. Forbidding bash makes them pure-knowledge comparisons, so the model must reason from knowledge and reaches for the run-tests skill (improving activation in both arms), while removing the tool-induced overhead and timeout risk. This is stricter, not weaker, gating and matches docs/InvestigatingResults.md sec.8. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert run-tests reject_tools change — did not affect activation gate The expect_tools->reject_tools flip was based on a wrong assumption: reject_tools is a POST-HOC scoring assertion only (AssertionEvaluator/BaselineStore), it does NOT remove bash from the agent toolset, so the agent kept using bash and skill activation was unchanged (12 activation-fail arms before AND after, runs 29253857976 vs 29256920057). Reverting to keep the PR minimal and avoid adding failing assertions that don't serve the activation goal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * writing-mstest 'Fix swapped AreEqual': validate activation instead of waiving it Addresses review feedback (r3551729171): setting expect_activation:false on an in-scope prompt (swapped Assert.AreEqual is explicitly in writing-mstest-tests' USE FOR) masks real routing/activation regressions. The waiver existed only because the task was trivially solvable unaided (baseline 5.0/5, zero delta), so the skill never needed to activate. Fix: make the scenario genuinely skill-dependent so activation is expected and validated. The prompt now also asks which analyzer catches this and how to enable it — skill-specific MSTest knowledge (SKILL.md Step 8: MSTEST0017 + MSTest.Analyzers packaging) that a vanilla model does not reliably produce. Added an output_matches 'MSTEST0017' assertion and two rubric items requiring the analyzer id and its enablement, and removed expect_activation:false. This both restores activation validation and gives the skill a real quality advantage over baseline. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests: strengthen activation triggers to fix systemic non-activation Radical fix for run-tests' systemic activation-gate failures. Root cause from artifact analysis: these are 'show me the exact dotnet test command' prompts, so the model answers directly from its own knowledge without invoking run-tests — even in the ISOLATED arm where run-tests is the only skill loaded (proving it is prompt/description-trigger driven, not just plugin-menu competition). It often answers WRONG on the subtle SDK 8/9 '--' separator vs SDK 10+ direct-args rule (baseline 1.0-1.7) yet still does not reach for the skill. At ~70% per-run activation the strict '.Any(3 runs) in BOTH arms' gate flakes red. Two-pronged fix to raise per-run activation probability: 1. run-tests description: lead with an imperative trigger ('ALWAYS use when the user asks ... wants the precise command, flags, or argument order') and name the model's blind spot ('the right syntax depends on the platform and SDK version and is easy to get wrong from memory'). This signals the model should consult rather than guess, and is more trigger-matched for the plugin arm. Kept under the 1024-char description limit and the 15000-char plugin skill-menu budget (verified: skill-validator check passes). 2. The four currently-flaking scenario prompts now state the user needs the command to be correct for THIS project's specific test platform and SDK version (a natural need that raises the stakes) without naming the skill or leaking the answer — the model must still detect the platform and know the version-specific syntax. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests Negative test: fix brittle assertion penalizing the better skilled answer After the activation fix, run-tests activates on the Negative VSTest scenario in both arms (activation solved), but it still failed with completion_regression. Root cause: the assertion output_not_contains '--report-trx' is too naive. The skill-informed answer is strictly better — it recommends 'dotnet test --logger trx' AND educates 'use --logger trx, not --report-trx (which is MTP-only)'. That correct, more helpful contrast trips the substring ban, marking the better answer as a task-completion regression versus a bare baseline command. Fix: forbid --report-trx only when it appears as an actual dotnet test command (output_not_matches 'dotnet test[^\r\n]*--report-trx'), not when the answer names it to warn against it. Verified against the real failing outputs: both skilled runs' --report-trx mentions are in explanatory prose and now pass, while an actual 'dotnet test ... --report-trx' recommendation is still rejected. Stricter-but-correct. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests Negative test: drop expect_tools:[bash] to remove efficiency-only failure After the activation + assertion fixes, the Negative VSTest scenario activates in both arms and passes all assertions, but still failed on a -2.8% weighted score: the baseline is already perfect (5.0/5, correctly uses --logger trx), so the skill can add no quality — and expect_tools:[bash] forced the skilled arm to actually run the tests, inflating tokens (40k->88k) and tool calls (5->8) versus a leaner baseline. That efficiency delta was the entire failure (docs sec.8 baseline-already-good). This is a 'recommend the exact command' negative test: exit_success only checks the agent produced output, and the other assertions validate the recommended command text, so requiring bash execution over-specified the scenario. Dropping expect_tools lets both arms answer from knowledge, removing the forced-execution token/tool overhead while still fully validating that the answer uses --logger trx and never a 'dotnet test ... --report-trx' command. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
ca1b47371f |
Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop (#813)
* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop WHY Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a plugin's version directly from its checked-in manifest. With no versioning discipline, a plugin's behavior can change while its advertised version stays flat, so clients never learn to re-pull, and there is no human-readable signal of what changed. We want correct, current versions in the repo with minimal manual work and without bloating the marketplace clone. WHAT - Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin owns a version.json whose pathFilters exclude the generated manifests and the version.json itself, so version height tracks real content changes only. - The computed version is materialized into the checked-in manifests (plugin.json and .codex-plugin/plugin.json) so every consumer reads a current value with no build step on their side. - eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the set of changed plugins from a git diff, computes each version with nbgv (predicting the squash-merge height for PRs), and either reports or stamps. AUTOMATIONS (two, low-touch by design) - /version-bump: an admin/maintainer comments the command on a PR and the affected plugins are stamped on the PR branch. Gated on collaborator permission (admin/write/maintain); forks are rejected before any privileged step. No other PRs are auto-modified. - weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any drift on main, opens/updates a single bot PR, and explains the per-plugin reason. This self-heals anything that merged without a bump. We deliberately did NOT auto-edit contributor PRs or add a noisy advisory comment bot; maintainers stay in control and the signal stays clean. SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro) - Supply chain (High, both models): dotnet tool restore would have honored a nuget.config authored in the PR tree, letting an attacker remap the nbgv package source to a malicious feed and run code in the privileged contents:write context. Mitigated with a trusted eng/version/nuget.config (clear + nuget.org-only + packageSourceMapping), overlaid from main and used via --configfile so PR-supplied configs are ignored. No nuget.config is tracked in the repo today, so this path was genuinely exploitable. - TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather than the mutable branch name; a racing push fails non-fast-forward, which is the safe outcome. - Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement string) so a "$"-bearing version cannot re-expand, plus a strict major.minor.patch guard that throws on a malformed base, leaving manifests untouched. - A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped. VERIFIED End-to-end against a real NBGV git harness: content-scoped predict, base-only bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard, and --configfile restore (exit 0). actionlint passes on both workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review feedback - Add missing plugins/dotnet-test-migration/version.json so it participates in versioning (it was the only plugin without one; manifests are at 0.1.0). - CONTRIBUTING: the two manifests are not byte-identical; say the version is duplicated across two manifest files instead. - weekly-version-sync: include version.json in commit attribution so a base-only bump is explained rather than showing 'no attributable commits'. - Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI failures are diagnosable, while keeping stdout clean for JSON parsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1270722c95 |
Fix migrate-static-to-wrapper skill activation on evals (#864)
Sharpen the description boundary between migrate-static-to-wrapper and generate-testability-wrappers so prompts that replace call sites when the abstraction is already registered in DI activate the migration skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e558acb8e8 |
dotnet-test: raise timeout for boundary comparison assertions eval scenario (#863)
The 'Use comparison assertions for boundary testing' scenario chronically hit its 240s wall-clock timeout. Raise it to 360s to match the other code-generation scenarios in this eval file, per InvestigatingResults.md timeout guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |