mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
74c58505e5011268ed6870a699872ea2aeff86fb
493 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 YuliiaKovalova
|
74c58505e5 |
Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills (#745)
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills Fix inaccurate reserved-shortName guidance, consolidate validation rules into a single skill, expand discovery mappings, add explicit CPM/version steps, and introduce template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .codex-plugin manifest indentation; add evals for new skills Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space indent on the agents line). Add eval.yaml + eval.vally.yaml capability evals for the new template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh - Clarify the reserved shortName set is the current dotnet new subcommands (authoritative source: dotnet new --help); create is verified as a real subcommand (alias behind dotnet new <template>). - template-discovery: note that some mapped short names (maui, winui3, aspire, func, orleans) need workloads/template packages, with fallback to dotnet new list/search. - template-instantiation: keep template versions by default; if refreshing, use dotnet list package --outdated + user confirmation and constrain to same major/minor rather than always latest stable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list - smart-defaults evals: enforce --no-https absence (auth scenario), absence of minimal-API flag (controllers scenario), and no newer --framework TFM when net8.0 is explicitly required, using output_not_contains/output_not_matches. - comparison eval: split the combined (auth|aot|docker|controllers) check into four separate output_matches assertions so partial comparisons fail. - template-validation/authoring: reframe the reserved shortName list as non-exhaustive examples and source the authoritative set from dotnet new --help; drop the specific create-alias assertion in favor of parsing-ambiguity wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make template-comparison evals robust to CI SDK currency Switch the Blazor comparison scenario from blazorserver (absent in the CI SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in .NET 8+, and instruct the agent to inspect each via --help. Update the SKILL.md example reference for currency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CLI-failure resilience guidance to discovery/comparison skills The isolated eval runs failed because the agent ran 'dotnet new <t> --help', hit the template engine's global-mutex/persistence error (common when the command runs concurrently in a sandbox), and then returned no answer at all. Instruct both skills to run 'dotnet new' calls sequentially, retry once on a transient mutex/persistence error, and fall back to the intent/parameter mapping so a concrete answer is always produced instead of empty output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: accurate flags + robust negative-assertion prompts - template-smart-defaults SKILL.md: drop the non-existent --publish-aot flag. Clarify --aot is a dotnet new flag only on templates that expose it (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild PublishAot=true property, not a dotnet new flag. - template-discovery SKILL.md: replace the hardcoded --enable-docker mapping (not a real flag on common templates) with generic 'confirm with --help'. - smart-defaults evals: tighten the negative-assertion prompts to output only the command line and not mention unused flags, so a negated explanation can't trip output_not_contains/output_not_matches. Switch the AOT scenario from webapi to worker (which actually supports --aot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: scope negative checks, de-emphasize stale lists - smart-defaults evals: anchor the negative assertions to the 'dotnet new' command line (same-line regex) instead of whole-output substring/regex, so a flag mentioned only in prose can't fail the test. - template-validation / template-authoring: mark the dotnet new subcommand examples as illustrative/version-dependent and tell readers not to hardcode them; the live 'dotnet new --help' output is canonical. - template-comparison: fix the example table's AOT row — webapi/webapp do not expose a --aot template flag; native AOT is publish-time via PublishAot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0a387fcdfe |
code-testing-implementer: add mandatory harness-discovery check (#757)
* code-testing-implementer: add mandatory harness-discovery check
Generic CI/benchmark verifiers run the framework's default discovery from the repo root. Tests that pass via a scoped command (e.g. 'dotnet test MyProject.Tests.csproj', 'bundle exec rspec subgem/spec', 'Invoke-Pester -Path ./tools') but are invisible to the harness count as 0 generated tests.
Repeatedly observed in msbench cta-nightly runs:
- new C# test project never 'dotnet sln add'ed (ocelot)
- Pester tests placed under custom directories invisible to default Invoke-Pester (homebrew, ruby/ruby)
- RSpec specs placed in a sub-gem's spec/ dir invisible from repo root (fastlane)
Changes:
- code-testing-implementer: capture baseline test count in Step 2, new Step 7 'Verify Harness Discovery (MANDATORY)' with concrete failure examples, HARNESS_DISCOVERY line in report template, reminder after Step 3 to revisit registration if Step 4 creates a new project.
- code-testing-researcher: instruct to record BOTH scoped test command and harness-equivalent discovery command in .testagent/research.md.
- dotnet.md: strengthen 'Registering a new test project' heading to MANDATORY when dotnet new was used; add 'Harness Discovery Check' section with 'dotnet test <solution> --list-tests' from repo root.
- powershell.md: add 'Harness Discovery Check' section with default-config Invoke-Pester from repo root.
- ruby.md: add gem-monorepo trap guidance (fastlane, ruby/ruby) in Test Placement Contract; add 'Harness Discovery Check' section with 'bundle exec rspec --dry-run' from repo root.
* Address review feedback on harness-discovery check
- dotnet.md: replace stale `Step 8 cleanup` reference with the actual Step 3 (`Register Test Project with Build System`).
- dotnet.md: demote `Harness Discovery Check` to `###` so it nests under the parent `## .csproj / .sln Handling` section alongside `### Registering...`.
- dotnet.md: replace `\s\{4\}` (non-POSIX) in the grep regex with 4 literal spaces so the count works under BRE.
- ruby.md: group the rake/rails fallback with `{ ...; } | wc -l` so the pipe applies to both branches of `||`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
9be44ad6e5 |
migrate-xunit-to-mstest: add learnings from dotnet/sdk migration (#756)
* migrate-xunit-to-mstest: add learnings from dotnet/sdk migration Refines the skill based on a real migration of 5 test projects in dotnet/sdk: - MSTest.Sdk implicit global using: do not add `<Using Include='Microsoft.VisualStudio.TestTools.UnitTesting' />` or per-file `using` in MSTest.Sdk projects (it's already in scope). Note Option A (`MSTest` metapackage) still needs the per-file using. - `Assert.IsExactInstanceOfType<T>` (MSTest 4.1+) is the proper single-call equivalent of xUnit's exact-type `Assert.IsType<T>`; previous guidance silently degraded it to assignable semantics (closes #755). - `Assert.AreSequenceEqual` (MSTest 4.3+) is the modern element-wise equivalent of xUnit's `Assert.Equal` on `IEnumerable<T>`, avoiding the `CollectionAssert` + `.ToList()` dance and the MSTEST0065 trap on plain `AreEqual`. - Clarify `AwesomeAssertions` ships in the `FluentAssertions` namespace, so it's a no-source-change swap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * SKILL.md: split IsType/IsAssignableFrom row to avoid implying drop-in equivalence Address dotnet/sdk#54727 review comment: the Step 6 summary table conflated xUnit's exact-type Assert.IsType<T> and assignable Assert.IsAssignableFrom<T> into a single row, which implies both map interchangeably even though their semantics differ. Split into three rows -- IsType<T> -> IsExactInstanceOfType<T>, IsNotType<T> -> IsNotExactInstanceOfType<T>, IsAssignableFrom<T> -> IsInstanceOfType<T> -- and explicitly call out the silent-weakening trap. The cheatsheet already maps these correctly; this aligns the SKILL.md summary with the cheatsheet so the at-a-glance table doesn't suggest a wrong mapping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * SKILL.md: resolve Step 3/Step 4 conflict on the MSTest using for Option B Address review feedback on dotnet/skills#756: Step 3 said to skip the per-file 'using Microsoft.VisualStudio.TestTools.UnitTesting;' on Option B (MSTest.Sdk provides it as an implicit global using), but Step 4 unconditionally instructed replacing xUnit usings with that MSTest using -- so an agent following Step 4 literally would re-add the redundant using on Option B projects. Also drop the dangling 'Step 4 rewriter' phrasing in Step 3 (there is no rewriter in Step 4, just a list of mechanical rewrites the agent applies). Step 4 now explicitly says to skip the MSTest using on Option B and only remove the 'using Xunit;' lines, matching Step 2's note and Step 3's branch instructions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Evangelink <amaury@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
8b7867331c |
Pin github/gh-aw-actions to commit SHA and add pinning guard (#749)
* Pin github/gh-aw-actions to commit SHA and add pinning guard The agentic workflows referenced github/gh-aw-actions/setup and setup-cli by mutable tag (@v0.77.5) because .github/aw/actions-lock.json still carried a stale pre-migration entry (github/gh-aw/actions/setup@v0.71.5) with no entry for the action the compiled workflows actually use. With no matching lock entry, gh aw compile fell back to emitting the bare tag. Refresh the lock with SHA-pinned entries for github/gh-aw-actions/setup@v0.77.5 and setup-cli@v0.77.5 (commit 3ea13c02...), and regenerate the workflows via gh aw compile so every uses: ref, decorative comment, and manifest sha is pinned. This mirrors how dotnet/msbuild pins the same action and unblocks enabling org-level "require actions pinned to a full-length commit SHA". Also add enforce-action-pinning.yml, a self-contained PR/push check that fails any workflow referencing an action by tag/branch. The agentic workflows are schedule/issue-triggered and never run on PRs, so neither /evaluate nor the runtime org policy gates them pre-merge; this check does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Remove enforce-action-pinning guard workflow Drop the CI guard in favor of relying on the org/repo 'Require actions pinned to a full-length commit SHA' setting, per review feedback. The lock-file fix and SHA pins remain the root-cause fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Aaron Powell
|
3134b816de |
Add JSON output for skill-validator check (#601)
* Add JSON output for skill-validator check Fixes #600 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix empty check discovery results Address PR feedback by failing when explicit skill or agent paths discover nothing, including the combined check path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review comments - Replace AddPlainError with AddGeneralError (remove duplicate method) - Key pluginSkills by DirectoryPath instead of display name to avoid mismatch when plugin.Name differs from directory name - Use OS-aware path comparison in IsPathWithin (Ordinal on Unix, OrdinalIgnoreCase on Windows) - Pre-build name lookup dictionaries in CreateJsonOutput to eliminate O(n*m) FirstOrDefault scans for external dependency attachment - Extract CheckJsonSerializerContext scoped to check JSON output so UseStringEnumConverter does not affect unrelated JSON payloads Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address latest Copilot PR feedback - Attach external dependency warnings by stable path identifiers (skill SKILL.md path, agent path, plugin directory path) - Use OS-aware comparer for plugin directory path matching - Restore console warning formatting via SkillProfiler.FormatProfileWarnings - Move CheckJsonSerializerContext into SkillValidator.Check namespace - Precompute reference-attachment targets to avoid repeated full-path normalization and container recalculation per finding - Add regression test covering duplicate skill names with external dependency warnings in JSON output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix flaky duplicate-skill JSON test path - Run duplicate-name external dependency assertion through plugin mode, which is where external dependency checks are executed - Add plugin fixture helper for JSON output tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address latest Copilot follow-up feedback - Reintroduce PluginValidationResult as obsolete compatibility type while keeping PluginCheckResult as the primary model - Avoid unnecessary profile-line formatting when verbose output is off - Replace string discriminators for external dependency kind with enum - Centralize JSON warning kind literals as constants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
118cc69319 |
Issue Triage: process issues from all authors (#741)
Lower the GitHub MCP integrity filter (min-integrity: none, allowed-repos: public) so the triage agent can read issue bodies/comments from external contributors and read-only org members instead of getting a [Filtered] placeholder. Harden the now-less-filtered path: disable issue-body edits on update-issue (body: false) and add an Untrusted content prompt section. Recompiled lock file. |
||
|
Jan Krivanek
|
78f3f755e7 |
Migrate Copilot PAT rotation to shared pat_pool import (#743)
Adopt the shared workflow import pattern from dotnet/runtime PR #127946, replacing the per-workflow select-copilot-pat action + custom job (shipped in #736) with a reusable shared/pat_pool.md import. What changed: - Add .github/workflows/shared/pat_pool.md: an import that defines a `pat_pool` job (inline bash, no separate action) exposing a `pat_number` output, plus an import-schema mapping COPILOT_PAT_0..7 to this repo's pool secrets (COPILOT_GITHUB_TOKEN, COPILOT_GITHUB_TOKEN_2..8). - Add .github/workflows/shared/pat_pool.README.md documenting the pattern. - Convert all 8 agentic workflows to `imports: - shared/pat_pool.md` + `engine.env` `case(needs.pat_pool.outputs.pat_number ...)`. - Delete the now-unused .github/actions/select-copilot-pat action. - Add .github/workflows/validate-pat-pool.yml: a daily standalone workflow that validates each pool PAT with a Copilot CLI request and summarizes pool health. Wiring note (adaptation from runtime): consuming workflows declare `on.needs: [pat_pool]` instead of runtime's `needs: [pre_activation]` + `on.permissions: {}`. This wires pat_pool ahead of the pre_activation and activation jobs so the selected PAT is validated by the activation job and used by the agent, and it works for `roles: all` workflows (issue-triage), which do not produce a pre_activation job for the runtime workaround to attach to. Compiled with gh-aw v0.77.5. Verified end-to-end with a temporary test-pat-rotation workflow (since removed): a run selected token #2 of the 3-token pool and the agent job observed pat_number='2', confirming the rotated PAT reaches the agent. |
||
|
Abhitej John
|
c85a2d8d9f |
Fix dotnet-msbuild Codex plugin install by externalizing mcpServers (#740)
* Fix dotnet-msbuild Codex plugin install by externalizing mcpServers Move the inline mcpServers configuration to a separate .mcp.json file inside .codex-plugin/, and update .codex-plugin/plugin.json to reference it via a relative path string. This matches the Codex plugin spec which requires mcpServers to be a path reference to a .mcp.json file rather than an embedded object. The root plugin.json retains the inline mcpServers format for the skill-validator. Both validator methods (FindPluginMcpServers and ExternalDependencyChecker.CheckPlugin) are updated to also handle the Codex string path format for forward compatibility. Fixes #738 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: validate mcpServers path and log parse errors - Reject rooted paths and paths with '..' segments to prevent directory traversal when resolving mcpServers string references. - Log parse errors in ResolveMcpFile to stderr (consistent with plugin.json parse error handling) instead of silently swallowing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
cc0621e2ac |
Fix Copilot PAT rotation for gh-aw v0.77.5 runtime (#736)
* Fix Copilot PAT rotation for gh-aw v0.77.5 runtime
The PAT-rotation stop-gap wired the rotated token into engine.env via needs.pre_activation.outputs.copilot_pat_number. Because the compiled agent job depends only on 'activation' (not the built-in 'pre_activation'), that needs reference evaluated to an empty string in the agent job, so the case() fell through to the default COPILOT_GITHUB_TOKEN and rotation never reached the agent. gh-aw v0.77.5 surfaces this as a compiler warning.
Replace the pre_activation step-injection with a 'select_copilot_pat' custom job wired via on.needs. As a user-defined job referenced in engine.env, the compiler makes it a direct dependency of the agent job, so needs.select_copilot_pat.outputs.copilot_pat_number resolves correctly in both the activation and agent jobs. Same action, same secret pool, same case() expression. Recompiled all workflows with gh-aw v0.77.5.
* Add temporary test-pat-rotation workflow to validate rotation
Non-destructive pull_request-triggered workflow that selects a pool token and asserts, in the agent job, that needs.select_copilot_pat.outputs.copilot_pat_number is non-empty (the exact value that was silently empty with the old pre_activation wiring). To be removed after validation.
* Work around gh-aw v0.77.5 invalid-YAML rendering of top-level if
gh-aw v0.77.5 emits the top-level frontmatter `if:` on the built-in
pre_activation job WITHOUT a ${{ }} wrapper. When the condition starts with
`!` (the fork guards), the emitted `if: !(...)` is invalid YAML (a leading `!`
starts a YAML tag), which GitHub rejects as a workflow-file startup failure.
v0.68.3 wrapped it (valid); v0.77.5 does not. Confirmed independent of the
PAT-rotation change via a minimal probe.
Wrap the fork-guard conditions in parentheses so the emitted scalar starts
with `(` instead of `!` (semantically identical). Affects close-stale-prs,
devops-health-check, devops-health-groom, markdown-linter, pr-malicious-scan.
* Update select-copilot-pat README for the custom-job + on.needs pattern
* Remove temporary test-pat-rotation workflow (rotation validated)
|
||
|
Jan Krivanek
|
7bd6628e91 |
fix(devops-health): resilient dashboard issue discovery (#737)
The health check located its dashboard issue solely via the devops-health label search. When GitHub silently dropped the pinned dashboard (#288) from its issue search/list index, the workflow could not find it, created a duplicate (#695), and abandoned the pinned issue -- leaving the pinned dashboard stale for days. Step 4.1 now resolves the dashboard by a cached issue number first (updating it directly by number, which works even when the issue is missing from search), persists that number to cache-memory every run, falls back to label + pinned-issue lookup, and consolidates duplicates. Documents the new health-dashboard-issue cache key. Lock file unchanged: the prompt body is runtime-imported from these .md files. |
||
|
Abhitej John
|
2ab3501158 |
Add .codex-plugin/plugin.json manifests for Codex CLI plugin install (#726)
The Codex CLI requires .codex-plugin/plugin.json as the plugin manifest entry point. Without it, 'codex plugin add' fails with 'missing plugin.json' even though the marketplace listing works. This adds .codex-plugin/plugin.json to all 14 plugin directories, with paths relative to the plugin root per the Codex docs. Also updates the agents marketplace to use dotnet-aspnetcore (per #711 rename) and adds missing dotnet-blazor and dotnet11 entries. Fixes #578 Fixes #724 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
2d163bb600 |
Rename ASP.NET Core plugin from dotnet-aspnet to dotnet-aspnetcore (#711)
* Initial plan * Rename dotnet-aspnet plugin to dotnet-aspnetcore --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Joey Robichaud
|
998ae28063 |
Update lsp.config to invoke dnx from the plugin directory (#607)
Due to the behavior of dotnet SDK resolution when running in repos which use a global.json, we are not gaurenteed that the choosen SDK will be new enough to support the dotnet dnx command which we were using to install and run the roslyn-language-server. Instead, we will ship our own global.json and configure the current working directory to be the plugin directory. |
||
|
Amaury Levé
|
803b91f167 |
Add grade-tests skill for per-test PR-comment quality grading (#727)
* Add grade-tests skill for per-test quality grading Adds a new polyglot skill that grades a provided list of test methods on a letter scale (A-F) with a score band and 1-line notes, suitable for posting in a PR comment table. Complements existing audit skills (assertion-quality, test-anti-patterns, test-smell-detection) which produce suite-wide reports. The skill: - Takes an explicit list of tests (refuses to grade a whole workspace silently) - Uses a 3-dimension rubric (Assertion strength, Structure & focus, Anti-pattern hygiene) - Combines via 0.45/0.30/0.25 weighting with safety caps - Starts at A and only deducts for observable issues - Does not flag idiomatic patterns (table-driven sub-tests, bare assert, etc.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback - OrderProcessorTests.cs: fix 4 invocations that mixed named arg customerId: with subsequent positional args -- invalid C# syntax (Copilot review on PR #727) - README.md: update 'five test-analysis skills' to 'six' in three places now that grade-tests is added to the polyglot list, and clarify that only the original 5 load from test-analysis-extensions while grade-tests has an inline rubric Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: drop 'grade-tests skill' wording from scenario prompts The skill-validator's evaluate pre-flight check ValidateEvalPrompts rejects any prompt that names the target skill, because mentioning the skill biases baseline (skill-not-loaded) runs and inflates measured lift. The two scenarios that asked the agent to 'use the grade-tests skill' now just ask it to grade test methods individually -- the agent is still expected to pick the right skill organically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix rubric consistency per review feedback - README: drop incorrect 'inline rubric instead' claim (grade-tests also loads test-analysis-extensions for language-specific guidance) - Assertion sub-grade: literal always-true assertions are now F (was D), matching the eval expectation that 'assert True' produces overall F - Anti-pattern catalog: split always-true literals (now -> F) from self-referential assertions (still -> D) — they were conflated - Combination rule: overall grade is now capped at the worst sub-grade (was capped at D), so an F in any one dimension yields overall F --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
faca87d669 |
Bump vite, @vitest/coverage-v8 and vitest (#731)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) to 8.0.16 and updates ancestor dependencies [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite), [@vitest/coverage-v8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8) and [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest). These dependencies need to be updated together. Updates `vite` from 5.4.21 to 8.0.16 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.0.16/packages/vite) Updates `@vitest/coverage-v8` from 2.1.9 to 4.1.8 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/coverage-v8) Updates `vitest` from 2.1.9 to 4.1.8 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/vitest) --- updated-dependencies: - dependency-name: vite dependency-version: 8.0.16 dependency-type: indirect - dependency-name: "@vitest/coverage-v8" dependency-version: 4.1.8 dependency-type: direct:development - dependency-name: vitest dependency-version: 4.1.8 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Amaury Levé
|
812565f8d1 |
Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest) (#709)
* Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest) Adds two non-.NET evaluation scenarios to the dotnet-test/code-testing-agent eval to measure how well the agent generates idiomatic tests in Python and TypeScript — the languages exercised by the msbench top5-* benchmarks. New fixtures (no pre-existing tests, so passing pytest/vitest proves the agent actually generated them): - fixtures/python-flask-tasks/ — small Flask app with TaskService (injected clock), TaskRepository Protocol + InMemoryTaskRepository, and a /tasks blueprint with HTTP 201/200/400/404/409 surfaces. Validated locally with `python -m pip install -e ".[test]" && python -m pytest`. - fixtures/typescript-vitest-cart/ — small shopping-cart library with an injectable DiscountPolicy seam (No/Percentage policies) and a Cart class with non-trivial merge / clamping semantics. Validated locally with `npm ci && npx vitest run`. package-lock.json committed so `npm ci` is reproducible in CI. New scenarios (added to both eval.yaml and eval.vally.yaml): - "Generate pytest tests for the Flask tasks API (Python polyglot)" — asserts that `python3 -m pip install -e '.[test]' && python3 -m pytest` exits 0 and that at least one `tests/**/test_*.py` file was produced. Rubric rewards using Flask's `test_client()`, mocking `TaskRepository` for service-level tests, asserting HTTP status + JSON body, and covering the empty-title / >200-char / not-found / already-done error paths. - "Generate Vitest tests for the shopping-cart library (TypeScript polyglot)" — asserts that `npm ci && npx vitest run` exits 0 and that at least one `tests/**/*.test.ts` file was produced. Rubric rewards mocking `DiscountPolicy` via `vi.fn()`, covering Cart merge semantics, `updateQuantity` zero-removes, `totals()` discount-clamping, and the `PercentageDiscountPolicy` constructor boundary. Companion to #708 (polyglot examples + sub-agent generification); these scenarios exercise the per-language guidance that PR adds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback for #709 - routes.py: guard `create_task` against non-dict JSON payloads (e.g. list) by returning 400 instead of letting `payload.get(...)` raise AttributeError and surface as a 500. This keeps the API behavior stable and makes it easier for generated tests to assert 400 on invalid input shapes. - tsconfig.json: drop `vitest/globals` from `compilerOptions.types`. The fixture's vitest.config.ts sets `globals: false`, so allowing TypeScript to assume globals (describe/it/expect) only enables tests that compile but fail at runtime. Removing it keeps the fixture aligned with the non-global Vitest API the eval is exercising. Both fixtures re-smoke-tested locally: vitest 1/1, pytest 2/2 (including a new test confirming the non-dict body returns 400). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill activation for polyglot scenarios in PR #709 Both polyglot scenarios (pytest + Flask, Vitest + TS) scored 5.0/5 quality but reported `⚠️ NOT ACTIVATED` because the SDK's skill router did not load `code-testing-agent` into the agent's context: `activated: false, detectedSkills: []` in skillActivationIsolated / skillActivationPlugin. The .NET ContosoUniversity scenario activates fine because its prompt has stronger `project-wide, multi-file ... high coverage` framing and SKILL.md already covers .NET keywords. Fixes per `eng/skill-validator/src/docs/InvestigatingResults.md` section "Skill not activated": - SKILL.md description: add framework-specific keywords (pytest, Flask/Django, Vitest, Jest, Mocha, JUnit, Node libraries, API, package, project-wide, multi-file) so the router has explicit hooks for polyglot prompts. Trimmed verbose `DO NOT USE FOR` clauses to stay under the 1,024-char skill spec limit (now 1,019 chars). - eval.yaml + eval.vally.yaml: rewrite the two polyglot prompts to mirror the .NET scenario's framing — add `project-wide, multi-file test generation task across the ... layers` and `achieve high coverage`, soften the library-prescriptive bullets (Mock(spec=...), vi.fn()) into capability-level requirements (`mocked or stubbed`, `mock or hand-written stub`). Rubric items unchanged — they remain flexible enough to grade either mock style. Validated locally: `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` => ✅ All checks passed (23 skill(s), 11 agent(s), 1 plugin(s)). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #709 review + sharpen polyglot skill activation **Activation fix for polyglot scenarios (Python + TypeScript)** The previous attempt ( |
||
|
Jan Krivanek
|
2604067914 | Fix the auto-evaluation triggering in pr review agentic workflows (#728) | ||
|
Amaury Levé
|
7fc6d472c3 |
Add migrate-xunit-to-mstest skill (#718)
* Add migrate-xunit-to-mstest skill Adds a skill to migrate .NET test projects from xUnit.net (v2 or v3) to MSTest v4. Covers package replacement, attribute/assertion/fixture/lifecycle translation, ITestOutputHelper -> TestContext, [Trait] -> [TestCategory]/ [TestProperty], and xUnit v3 TestContext.Current.CancellationToken. Front-loads parallelization handling because xUnit parallelizes test classes by default while MSTest serializes — the single largest source of post- migration regressions. Step 11 enumerates three explicit choices and how to translate CollectionBehavior/MaxParallelThreads/[Collection] settings. Preserves the existing test platform (VSTest stays VSTest; MTP stays MTP) by default — bundling a platform migration would violate the test-migration agent's 'never mix migration steps' rule. - plugins/dotnet-test/skills/migrate-xunit-to-mstest/SKILL.md (workflow) - plugins/dotnet-test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md - tests/dotnet-test/migrate-xunit-to-mstest/eval.yaml (12 scenarios) - tests/dotnet-test/migrate-xunit-to-mstest/eval.vally.yaml (parity with sibling skill) - plugins/dotnet-test/README.md (table entry) - plugins/dotnet-test/agents/test-migration.agent.md (triage routing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix step cross-references flagged in PR review - Step 1.5 'Custom DataAttribute' bullet pointed to Step 6 (assertions); custom DataAttribute mapping lives in Step 5 (data-driven tests). - Step 7 lifecycle table 'Constructor' row pointed to Step 8 for ITestOutputHelper; ITestOutputHelper conversion is in Step 9 (output). - Commit-strategy callout said 'after Step 5 (asserts fixed)' and 'after Step 9 (fixtures/lifecycle rewritten)'; asserts are in Step 6 and fixtures/lifecycle complete after Step 8. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Map [assembly: Xunit.Trait] to [assembly: TestCategory]/[TestProperty] Both TestCategoryAttribute and TestPropertyAttribute target Assembly, Class, and Method, so an xUnit [assembly: Trait] does not need to be demoted to per-class/per-method MSTest attributes — assembly scope is preserved 1:1. - references/mapping-cheatsheet.md §8 now maps [assembly: Trait] to [assembly: TestCategory] / [assembly: TestProperty] instead of 'Remove'. - SKILL.md Step 12 promoted to 'Convert' (was 'Remove'); the same mapping is documented inline. - Notes after the Step 4 and §1 trait tables now disclose the assembly target so users do not assume class/method-only scope. Addresses PR feedback from @Evangelink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address second round of PR review feedback From @Evangelink: - [Ignore]/[Timeout] do NOT discover a test on their own — they are modifiers. SKILL.md Step 4 + cheatsheet §1 now show [Fact(Skip)] -> [TestMethod] + [Ignore(...)] (and same for [Timeout]). Eval scenario 9 rubric updated to require both attributes. - TestProperty IS filterable (was incorrectly described as 'metadata that does not need filter integration'). Notes after the trait tables in both SKILL.md and cheatsheet §1 now show both filter syntaxes. - For environmental gates (OS-/CI-/arch-conditional), point at MSTest 3.10+'s condition attributes ([OSCondition], [CICondition], [ArchitectureCondition], [NonParallelizableCondition]) as the preferred alternative to overloading [TestCategory] or scattering Assert.Inconclusive through test bodies. Strengthened in cheatsheet §3.9, SKILL.md Step 6, Step 10 SkippableFact row, and the Common Pitfalls table. - TestDataRow<T> is also a supported MSTest data source (strongly typed with per-row DisplayName/Ignore metadata). Added to both SKILL.md Step 5 and cheatsheet §2 with link to the docs. - MSTest.Sdk + <UseVSTest>true</UseVSTest> pulls in Microsoft.NET.Test.Sdk automatically — the manual PackageReference was incorrect. Removed it from both files (Step 2, Step 3 'common mistakes', and Common Pitfalls). - Added the 'pin MSTest.Sdk in global.json msbuild-sdks' alternative as the recommended pattern for multi-project solutions. - Xunit.Combinatorial -> Combinatorial.MSTest (Youssef1313 community port). Updated both SKILL.md Step 10 and cheatsheet §10. From Copilot reviewer: - MSTest.Sdk example in SKILL.md Step 2 Option B hardcoded <TargetFramework>net9.0</TargetFramework>, contradicting the preserve-TFM rule. Replaced with a placeholder + explicit comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve xUnit-to-MSTest skill effectiveness based on eval feedback Eval results on PR #718 showed regressions on three rubric items where the agent did the migration correctly but did not COMMUNICATE the judgement-call decisions, so the judge could not verify the choice was deliberate: - ICollectionFixture scope decision (rubric: 'Explicitly explains the scope decision instead of silently widening' scored 1/5 because the agent said 'Let me convert everything' with zero scope discussion). Step 8 now requires a templated explanation of (source scope, target scope, what is shared, what is serialized, why the rejected alternative was rejected) before applying. - Parallelization 'MSTest default is serial' messaging (rubric scored 1.7/5 because the agent only implicitly conveyed it). Step 11 Choice A now demands a templated sentence explaining that the [assembly: Parallelize] is REQUIRED to match xUnit, otherwise the suite silently regresses to serial. - Response Guidelines now lists the three judgement-call decisions that MUST be communicated up-front: fixture scope, parallelization model, and the Throws/ThrowsExactly semantic flip. Additional fixes from this round: - Step 2: explicit guidance to default to Option A (MSTest metapackage) when the user says 'preserve VSTest', so the safer PackageReference path is picked over MSTest.Sdk + UseVSTest. - Step 4: pulled the '[Ignore]/[Timeout] are modifiers, need [TestMethod]' callout out of a table footer into a visible note above the table. - known-domains.txt: allowlist github.com/Youssef1313/Combinatorial.MSTest (referenced from SKILL.md Step 10 and cheatsheet §10 after the recent Xunit.Combinatorial -> Combinatorial.MSTest switch). Validator: green (24 skills, 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add concrete TestContext.Current.CancellationToken migration example The CancellationToken scenario was failing/low-scoring because the skill only contained a one-line mapping with no concrete example. The agent often missed the requirement to: - Add constructor TestContext injection (even when ITestOutputHelper is absent) - Map Assert.False -> Assert.IsFalse alongside the rename - NOT fabricate a CancellationTokenSource as a replacement Add a full xUnit v3 -> MSTest worked example next to the TestContext.Current mapping (Step 9), and reinforce the CancellationTokenSource warning in the cheatsheet. The example covers all 3 rubric points for the scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address third round of PR review feedback - Remove fictional MSTest condition attributes [ArchitectureCondition] and [NonParallelizableCondition] from SKILL.md and cheatsheet (only [OSCondition] and [CICondition] exist in MSTest 3.10+; for non-parallel intent use [DoNotParallelize], for architecture gating fall back to runtime Assert.Inconclusive). (Copilot) - TestPropertyAttribute targets only [Class|Method] (no AttributeTargets.Assembly), so [assembly: TestProperty(...)] does not compile. Update the trait notes in SKILL.md Step 4, cheatsheet section 1, Step 12, and cheatsheet section 8 to collapse a non-category xUnit [assembly: Trait] to [assembly: TestCategory] or push it down to per-class [TestProperty]. (@Evangelink) - Step 1 platform detection: stop inlining a buggy VSTest-vs-MTP matrix; delegate to the platform-detection skill. Note explicitly that <UseMicrosoftTestingPlatformRunner> only affects 'dotnet run' and is not a reliable runner signal. (@Youssef1313) - Step 2 Option A: add MTP code-coverage caveat -- Microsoft.NET.Test.Sdk pulls VSTest's Microsoft.CodeCoverage transitively, which can interfere with MTP's collector. Prefer Option B (MSTest.Sdk without UseVSTest) for MTP projects. (@Youssef1313) - Step 3: remove fictional MSBuild properties (<CaptureConsoleOutput>, <UseRoslynCompilers>); restate the bullet about xunit.runner.json -> Step 11 port instead. Restate the <UseMicrosoftTestingPlatformRunner> bullet to make clear it is not a runner switch. (@Youssef1313) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 4 - mapping-cheatsheet.md (Assert.Contains predicate): the suggested fallback Assert.Contains(collection.First(x => predicate), collection) is buggy -- First() throws InvalidOperationException on no-match (changing the failure mode), and feeding the result back into Assert.Contains is circular. Keep only the correct translation: Assert.IsTrue(collection.Any(x => predicate)). - SKILL.md Step 4 / cheatsheet table: stop instructing 'mark every class sealed'. xUnit projects commonly use base/derived test classes (shared setup, generic base fixtures); mechanically sealing them would break compilation. Sealing is now framed as an optional follow-up handled by writing-mstest-tests, not part of the mechanical migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
01050ea83c |
Fix pr-malicious-scan: repeat-spam + integrity-filter blocks (#722)
* Fix malicious-scan repeat-spam + integrity-filter blocks Root causes (observed on PR #237): 1. Agent's emitted add_comment body did not include the HTML marker line (<!-- pr-malicious-scan:fingerprint=... -->), so both the orchestrator's pre-dispatch check and the agent's own Step 1 idempotency lookup failed to find a prior scan for the same head SHA. Result: hourly re-dispatch. 2. The github MCP tools (pull_request_read, list_pull_requests, search_pull_requests) are blocked by the gh-aw integrity filter on PRs from non-approved authors -- exactly the population this scanner targets. Result: 'Integrity filter blocked N items' notes in every comment. Fixes: - pr-malicious-scan.agent.md: drop the github MCP toolset, add 'gh' to the bash allowlist, and instruct the agent to use 'gh api' for all PR data reads (PAT-authenticated, not subject to the integrity filter). - Strengthen Step 5: the HTML marker MUST be the first line of the comment body. Add a defense-in-depth note that the orchestrator also accepts the visible-body sentinel. - pr-triage-batch.yml + pr-triage-act.sh: match prior scans by EITHER the HTML marker OR the visible-body sentinel ('Automated diff scan' + backticked sha7), so a missing marker on a previously-emitted comment no longer triggers re-dispatch. Workflow disabled remotely while this lands. * Orchestrator-only dispatch + integrity-filter opt-out Replace the per-push pull_request_target trigger and the gh-api workaround with the documented gh-aw pattern: - pr-malicious-scan.agent.md: drop pull_request_target; trigger only via workflow_dispatch from the orchestrator. Restore the github MCP toolset with min-integrity: none (the documented level for spam-detection / analytics workflows; safe-outputs still gates every mutation). Drop the 'gh' bash hack and visible-body sentinel requirements. - pr-triage-batch.yml: orchestrator now posts a deterministic '<!-- pr-malicious-scan:dispatched=SHORT --> ' comment BEFORE calling gh workflow run. That comment is the source of truth for 'a scan has been initiated for this head SHA' and survives every agent-side failure mode (PAT outage, integrity block, dropped HTML marker). Dedup matches either that orchestrator marker OR the agent's own fingerprint marker. - pr-triage-act.sh: drop the visible-body-sentinel fallback; match the orchestrator dispatched marker plus the agent fingerprint marker. Validated: gh aw compile clean; bash -n clean for both worker script and orchestrator embedded script; markdownlint clean; dedup query and POST api tested live against PR #713. |
||
|
Amaury Levé
|
ecbecbc6f7 |
writing-mstest-tests: tighten description triggers (#712)
Refine the SKILL.md frontmatter description so the skill router has sharper hooks for common MSTest test-writing prompts: - Add concrete intent triggers (better MSTest assertion than Assert.IsTrue, replace hard cast with MSTest type assertion). - Expand the assertion API surface listed in USE FOR with the modern MSTest 3.x/4.x set (IsInstanceOfType, Contains, ContainsSingle, IsEmpty/IsNotEmpty, DoesNotContain). - Group related triggers (data-driven, lifecycle, parallelization) for readability. - Tighten DO NOT USE FOR (drop redundant clauses) and explicitly exclude xUnit/NUnit/TUnit so the router doesn't grab non-MSTest prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
a037cdcd87 |
Add Codex-native plugin marketplace manifest (#555)
* Initial plan * Add Codex-native plugin marketplace manifest Agent-Logs-Url: https://github.com/dotnet/skills/sessions/d109fe03-243d-450a-be68-fe03e9d74153 Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> |
||
|
Jan Krivanek
|
d9e4a5e113 |
PR triage workflows: orchestrator, worker, and evaluate-now label (#716)
* PR triage workflows: orchestrator, worker, and evaluate-now label Implements docs/design/pr-triage-workflows.md: - pr-triage-batch.yml: hourly orchestrator that classifies open PRs - pr-triage.yml + pr-triage-act.sh: per-PR worker (state recompute, label reconciliation, eval-trigger, ping comments with cool-down) - evaluation.yml: gate job now also handles pull_request_target [labeled] with the evaluate-now label as a second entry point alongside /evaluate * Add temporary push triggers for testing pr-triage workflows * test: live-run pr-triage worker once * test: re-run worker for cool-down check * fix: age gate uses created_at and applies only before first ping * Remove temporary test triggers and inline test marker * Add pr-malicious-scan agent workflow; replace design doc with brief overview - New: .github/workflows/pr-malicious-scan.agent.md + compiled .lock.yml. Static diff scanner for external (non-trusted) PR contributors. Triggers on pull_request_target [opened/synchronize/reopened] and workflow_dispatch. Surfaces findings as code-scanning alerts plus a single maintainer-ping comment per head SHA when high-severity / workflow-tamper / supply-chain hits. Never executes PR head code. - docs/design/pr-triage-workflows.md replaced with a brief overview + diagram. The full implementation plan is kept locally as docs/design/pr-triage-workflows-plan.md (gitignored). - pr-triage-batch.yml's existing dispatch-scanner branch now resolves to the new scanner; orchestrator unchanged. * Fix markdownlint MD038 (pipe inside code span) in malicious-scan agent |
||
|
Jan Krivanek
|
19f024ba2d |
Redirect session-data reads/writes to dotnet/skills-data repo (#717)
The dashboard-session-data branch on this repo had grown to ~480MB and was bloating clones. Session data has been migrated to the standalone dotnet/skills-data repository (branch of the same name). This PR redirects: - dashboard.js -> reads manifest from dotnet/skills-data - evaluation.yml PR-comment link -> points at dotnet/skills-data - evaluation.yml publish-session-data job -> clones/pushes to dotnet/skills-data using a new SKILLS_DATA_TOKEN secret (fine-grained PAT with contents:write on the data repo) Also adds --depth 1 to the deploy clone so future runs do not re-import history. AGENTVIZ replay UI requires no changes; it accepts an arbitrary manifest URL via query string. raw.githubusercontent.com serves the new repo with Access-Control-Allow-Origin: * so the dotnet.github.io/skills/replay page can load it cross-origin. The old branch on dotnet/skills is left in place for now and will be deleted manually after one successful run end-to-end. |
||
|
Jan Krivanek
|
510cbe1a40 | Adjust issues triage workflow (#714) | ||
|
Amaury Levé
|
49a77a90bf |
code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java (#708)
* code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java
The code-testing-agent skill family is polyglot in description but in
practice biased toward .NET because dotnet-examples.md was the only
filled-in pipeline walkthrough. The four sub-agents that participate in
the Research-Plan-Implement pipeline (researcher, planner, implementer,
generator) all pointed to dotnet-examples.md whenever they suggested a
concrete example, which made it harder for the agent to produce
idiomatic non-.NET tests (e.g. in msbench top5-* benchmarks for
Python/Flask, TypeScript/Express).
This change
* adds four new example files mirroring dotnet-examples.md format
(source → research → plan → generated test → fix cycle → final report):
- python-examples.md (pytest, unittest.mock, Mock(spec=...), parametrize)
- typescript-examples.md (Vitest with notes for Jest; it.each,
async tests, fake timers, ESM/CJS fix cycle)
- go-examples.md (standard testing package, table-driven subtests,
hand-written fake repository, injected clock)
- java-examples.md (JUnit 5 + Mockito on Maven, @ParameterizedTest +
@CsvSource, Clock.fixed, Surefire fix cycles)
* updates code-testing-extensions/SKILL.md TOC to list the new files
and clarifies usage instructions to read the matching <language>-
examples.md alongside the base extension
* makes the "Concrete example" pointers in code-testing-generator,
code-testing-implementer, code-testing-planner and
code-testing-researcher agents language-agnostic (list all available
example files instead of hard-coding dotnet-examples.md)
* expands code-testing-researcher project-structure detection list to
cover more Python (tox.ini, noxfile.py, requirements*.txt, uv.lock,
poetry.lock, pdm.lock), JS/TS (.mts/.cts/.jsx, vitest.config.*,
jest.config.*), C++ (CMakeLists.txt, BUILD.bazel, meson.build),
Java/Kotlin (pom.xml, build.gradle[.kts], wrappers), and other
ecosystem files; expands the Identify-Language section accordingly
* extends the "Language-Specific Examples" section in
code-testing-agent/SKILL.md to summarise each example file
Validated with: skill-validator check --plugin ./plugins/dotnet-test
(23 skills, 11 agents — all checks passed) and markdownlint-cli2.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #708
- go-examples.md: replace the hand-rolled `contains`/`stringIndex`
helpers with `strings.Contains` from the standard library. The hand-rolled
`contains` had subtly wrong semantics — `contains(""abc"", """")` returned
`false` while `strings.Contains` returns `true` — and the helpers are
unnecessary complexity for a code-generation example.
- go-examples.md: in the `go test -run` "wrong selection regex" sample fix
cycle, quote the test name and use `single_item` (matching the underscore
that the surrounding diagnosis text refers to) instead of the unquoted
`single item` which the shell would parse as two separate CLI arguments.
- java-examples.md: the source-tree file list described `Invoice.java` as a
`record` but the `InvoiceService.markAsPaid` example mutates the invoice
via `setStatus(...)` and `setPaidDate(...)` — records are immutable, so
the description was internally inconsistent. Re-describe it as a mutable
POJO with explicit mutators to match the service code.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
680292e04c |
Make dotnet-test analysis skills and auditor agent polyglot (#707)
* Make dotnet-test analysis skills and auditor agent polyglot Five test analysis skills and the test-quality-auditor agent in plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/ TUnit cues, even though the underlying techniques (assertion diversity, anti-pattern detection, pseudo-mutation gap analysis, academic smell catalog, trait tagging) apply equally to other test frameworks. This change makes them polyglot across .NET, Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest). Changes ------- * New reference skill `test-analysis-extensions` with per-language extension files (dotnet, python, typescript, java, go, ruby, rust, swift, kotlin, powershell, cpp). Each documents test markers, assertion APIs, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators, integration markers, and tag-support capability for the corresponding framework. * `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter now lists all polyglot frameworks (keeping .NET trigger words for backward compat); bodies reference `test-analysis-extensions` and add per-language calibration (e.g., Go/Rust table-driven loops are idiomatic and not Conditional Test Logic; pytest bare `assert` is canonical; missing-await on async assertions is a new Critical smell). * `test-quality-auditor` agent now runs language detection, gates .NET-only pipeline steps (coverage-analysis, CRAP, detect-static- dependencies, testability migration, experimental dotnet skills), and recommends native tooling for non-.NET projects via an explicit Capability Matrix. * README updated to split sections by polyglot vs .NET-only. * New polyglot eval scenarios added (Python/pytest, TS/Jest, Java/JUnit) alongside existing C# scenarios in `test-anti-patterns`, `assertion-quality`, `test-smell-detection`. The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis, crap-score, detect-static-dependencies, run-tests, filter-syntax, platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are unchanged and continue to activate only for .NET work. Validated with skill-validator `check --plugin ./plugins/dotnet-test`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #707 review feedback * typescript.md: replace xpect.fail() recommendation (Jest has no such API) with hrow new Error(...) / explicit failing assertion. * test-tagging SKILL.md description: clarify that unittest has no canonical tag syntax (pytest markers are the auto-edit mechanism; unittest is report-only). Switch doctest reference to "decorators" rather than implying a Catch2-style [tags] mechanism exists. * test-tagging SKILL.md auto-edit list and table: use `* doctest::test_suite("tag")` decorator chain syntax for doctest in both the bullet list and the per-framework table, matching the example already shown. * cpp.md extension: spell out the Catch2 vs doctest tagging distinction in the capability matrix (Catch2 [tag], doctest * doctest::test_suite("tag") decorator). Validated with `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` (23 skills + 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
972f4831fe |
Bump the all-other-nuget group with 3 updates (#697)
Bumps Nerdbank.MessagePack from 1.1.62 to 1.2.4 Bumps Vecc.YamlDotNet.Analyzers.StaticGenerator from 17.0.0 to 18.0.0 Bumps YamlDotNet from 17.1.0 to 18.0.0 --- updated-dependencies: - dependency-name: Nerdbank.MessagePack dependency-version: 1.2.4 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-other-nuget - dependency-name: Vecc.YamlDotNet.Analyzers.StaticGenerator dependency-version: 18.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: all-other-nuget - dependency-name: YamlDotNet dependency-version: 18.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: all-other-nuget - dependency-name: Nerdbank.MessagePack dependency-version: 1.2.4 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-other-nuget - dependency-name: Vecc.YamlDotNet.Analyzers.StaticGenerator dependency-version: 18.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: all-other-nuget - dependency-name: YamlDotNet dependency-version: 18.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: all-other-nuget ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
hngye02
|
df4578950d |
Add session viz link to evaluation.yml (#701)
Co-authored-by: t-hungnguyen <t-hungnguyen@microsoft.com> |
||
|
Amaury Levé
|
2b7fcdd7ab |
dotnet-test: add Rule #0 (Confirm the Test Target) to ruby/powershell extensions (#702)
* dotnet-test: add Rule #0 (Confirm the Test Target) to ruby/powershell extensions When the prompt does not name a specific file ("test the repository", "one core module", "comprehensive suite"), agents frequently target the wrong code — typically the largest upstream module that already has rich existing tests — instead of the newly-added module the user actually wants tested. Recent benchmark runs (top5-{ruby,powershell}-*-{simple,complex}) showed this is a dominant failure mode for Ruby/PowerShell: the agent burns 50+ turns writing tests for files the verifier never measures, while the real target (a small untracked `lib/string_utils.rb` or `tools/StringUtils.psm1`) sits one `git status` away. Adds a Rule #0 to both ruby.md and powershell.md, ahead of the existing "Rule #1: Investigate the Repo First". The rule: - Tells the agent to use git history (`git status -s`, `git ls-files --others --exclude-standard`, `git log --diff-filter=A --name-only -5`) to find the actual target rather than guessing from repo size. - Documents a Test Placement Contract — RSpec scopes to `spec/`, Pester scopes to whatever directory the harness passes to `Invoke-Pester -Path`; tests placed elsewhere are invisible. - Adds a First-Test Sanity Loop: write one test, run --dry-run / -PassThru to confirm discovery > 0, fix LoadError / Import-Module issues before expanding. Catches placement mistakes on turn 1. Both files validate cleanly under skill-validator. The existing Rule #1 and all subsequent sections are unchanged — this is a pure prepend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address PR review: clarify Rule #0 precedence and broaden require regex - Rephrase Rule #0's discovery preamble in ruby.md and powershell.md to call out the commands as the read-only exception to Rule #1, removing the apparent contradiction between 'before planning' and Rule #1's 'before writing any test or running any command'. - Broaden the spec_helper require grep to match leading whitespace and `require_relative`, avoiding false negatives that send the agent to the wrong target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
3123e607a1 |
dotnet-test: add explicit non-.NET guards to 5 skill descriptions (#705)
* dotnet-test: add explicit non-.NET guards to 5 skill descriptions
Five skills in plugins/dotnet-test have language-neutral trigger phrases
in their USE FOR sections that could plausibly mis-route on non-.NET
prompts in mixed-language workflows:
- writing-mstest-tests ("write unit tests for a class", "create test class")
- test-anti-patterns ("audit my tests", "test smell audit")
- assertion-quality ("identify assertion-free tests")
- test-gap-analysis ("find weak tests", "discover untested edge cases")
- test-tagging ("categorize, audit, or label tests with traits")
Each skill now has an explicit non-.NET exclusion clause in its
DO NOT USE FOR section, naming the most common languages it must NOT
match (Python, JS/TS, Go, Java, Rust, Ruby, PHP, Swift, Kotlin) and
restating that the skill only handles C#/F#/VB.NET test code.
This was surfaced by an MSBench investigation comparing a baseline
Copilot CLI run vs the same run with this pruned dotnet-test plugin
installed against the sweatlas-tw-unit benchmark (24 Python / 15 Go /
5 TS, 0 .NET). Both runs scored identically (15/44 resolved). Skills
correctly never fired (0 invocations across all 43 instances), so this
is a hardening change, not a bug fix — it makes the router's job easier
in heterogeneous repos and future benchmarks.
Validated with: dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin plugins/dotnet-test
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
4f78daf583 |
run-tests: add platform-detection keywords for activation (#700)
* run-tests: add platform-detection keywords for activation
The 'Detect test platform from Directory.Build.props' eval scenario
regressed from 2.7/5 to 2.0/5 with run-tests not activated (skill
loaded column shows NOT ACTIVATED). The scenario prompt asks 'figure
out which test platform it uses' but the previous description framed
detection only as 'chooses the correct platform/SDK/framework syntax'
and 'selecting VSTest vs Microsoft.Testing.Platform command syntax' --
both focused on syntax selection, not on platform detection from
project files.
Per InvestigatingResults.md guidance for 'skill not activated'
failures, surface the detection use case explicitly with the file
names the scenario hinges on:
detecting test platform/framework from \global.json\, \.csproj\,
\Directory.Build.props\
Trim the syntax clause ('command syntax' -> 'syntax', 'including' ->
'incl.') and shorten the multi-TFM example (drop redundant
\dotnet test\ prefix) to stay under the 1,024-char description cap.
Folded description: 948 -> 1,010 chars.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: rephrase platform detection with prompt-shaped language
The previous fix (
|
||
|
Tom Longhurst
|
36209c3b52 |
TUnit Fixes & Improvements (#677)
* Update target framework to net10.0 and upgrade TUnit package to version 1.45.8
* Fix and expand TUnit coverage in dotnet-test-frameworks reference
Cross-referenced against tunit.dev/llms.txt and the official attributes
comparison page. Corrects the test class marker (TUnit is convention-based,
not [ClassDataSource]), and fills in the previously-missing TUnit rows for
the assertion APIs, setup/teardown methods, exception-handling examples,
and integration test category marker.
* Refine TUnit assertion table accuracy
The Type row's TUnit cell now uses IsAssignableTo<T>() to match the
assignable-type semantics of the MSTest/xUnit/NUnit equivalents, with a
note pointing at IsTypeOf<T>() for exact-type checks. The Skip row now
documents that [Skip] also applies at class and assembly scope and that
Skip.Test("reason") handles dynamic in-test skipping.
Verified against the TUnit migration guides and the type-assertions docs.
* Spell out TUnit assertion alternatives as complete awaited expressions
The Boolean, Null, Exception, and Type rows in the assertions table
previously used a `await Assert.That(x).IsTrue()` / `.IsFalse()` shorthand
for the alternative form. A reader copy-pasting just the second fragment
would end up with an un-awaited assertion that silently passes — which
contradicts the note immediately below the table about always awaiting.
Each alternative is now a complete, independently copy-pasteable expression.
Addresses the inline review on dotnet/skills#677.
---------
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
|
||
|
Amaury Levé
|
b9db51de9d |
Add TUnit-focused eval scenarios for dotnet-test-frameworks (#699)
* Add TUnit-focused eval scenarios for dotnet-test-frameworks Adds 5 new evaluation scenarios and tightens 2 existing scenarios to exercise the expanded TUnit coverage in dotnet-test-frameworks/SKILL.md (see #677). New scenarios: - Convert cross-framework assertions to TUnit syntax (await / IsEqualTo / IsTrue / IsNull / IsAssignableTo / Contains / Throws<T>) - Diagnose silently-passing TUnit test with missing await (debugging mystery framed without naming the pitfall) - Refactor TUnit try/catch to native exception assertion (Throws<T>() / ThrowsExactly<T>() / WithMessage) - TUnit lifecycle hooks at test / class / assembly / session scope - TUnit skip mechanisms — attribute, assembly-wide [assembly: Skip], and dynamic Skip.Test(...) Updated scenarios: - Identify TUnit framework: rubric now distinguishes [ClassDataSource] as a fixture/data source rather than a class marker (TUnit classes are convention-based, like xUnit) - Identify integration tests: adds a TUnit Project D using live SqlConnection and requires recommending [Category("Integration")] for TUnit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Mirror TUnit eval scenarios into Vally spec Apply the same updated and new scenarios as in eval.yaml to the parallel eval.vally.yaml so both pipelines exercise the new TUnit-focused checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Amaury Levé <amauryleve@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
b2ccf6dcab |
Bump github/gh-aw-actions in the github-actions-dependencies group (#696)
Bumps the github-actions-dependencies group with 1 update: [github/gh-aw-actions](https://github.com/github/gh-aw-actions). Updates `github/gh-aw-actions` from 0.68.3 to 0.76.1 - [Release notes](https://github.com/github/gh-aw-actions/releases) - [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/gh-aw-actions/compare/abea67e08ee83539ea33aaae67bf0cddaa0b03b5...46d564922b082d0db93244972e8005ea6904ee5f) --- updated-dependencies: - dependency-name: github/gh-aw-actions dependency-version: 0.76.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
053792ce8e |
Aggregate sharded plugin results in publish-eval-data (#698)
The publish-eval-data job iterates discover.outputs.plugins and looks for "all-results/skill-validator-results-<plugin>", but when the discover job fans a plugin into multiple shards the artifacts are named "skill-validator-results-<plugin>--shard-<tag>". The exact-match lookup misses all of them and the step logs "No run results found for dotnet-msbuild, skipping" -- which is exactly what the 2026-05-27 scheduled run did (confirmed in the publish-eval-data logs of run 26484154365). Fix: locate every matching artifact dir (exact name OR --shard-* suffix), collect the latest run dir's results.json from each, and -- when there are multiple -- concat their verdicts arrays into a single synthetic results.json before calling generate-benchmark-data.ps1. Calling the script per-shard would have emitted N separate datapoints with the same commit/timestamp and skewed the dashboard. publish-token-data and publish-session-data were already shard-tolerant and need no changes. |
||
|
Jan Krivanek
|
bd5e7402db |
Restore investigation prompt to PR comment (#693)
PR #473 moved the failure investigation prompt out of the PR comment and only into the workflow run summary. This made it harder for users to discover — they had to click through to the run details to find it. Move it back to the PR comment so the copy-paste prompt is visible inline on the PR, while keeping it in the workflow summary as well. |
||
|
Amaury Levé
|
ac653c62e4 |
dotnet-test: add cross-language edit-boundary rules to code-testing-implementer (#689)
The code-testing-* agent family generates tests for 12 languages (not just .NET). Until now, no instruction told the implementer to keep its changes additive: it would routinely modify existing test files non-additively (deleting/reformatting lines) or edit non-test production code to make something easier to test. Both behaviors cause the change to be rejected by: - the msbench 'simple' test verifier (any 'dels > 0' on a test file, or any change to a non-test file, sets reward=0) - most real-world test-quality gates, code-review policies, and CI Add explicit cross-language rules in the implementer agent prompt: 1. Existing test files are append-only (no reformat/reorder/remove). 2. Do not modify non-test source files; surface untestable seams as follow-ups for the testability-migration agent. 3. Prefer new test files over edits to existing ones when equivalent. 4. Build-system manifests may be edited only for project/dependency registration. Rule 6 in the Rules section now points at the Step 4 detail so the implementer reads it on every phase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
a9c35454f2 |
Bump github/gh-aw-actions in the github-actions-dependencies group (#678)
Bumps the github-actions-dependencies group with 1 update: [github/gh-aw-actions](https://github.com/github/gh-aw-actions). Updates `github/gh-aw-actions` from ba90f2186d7ad780ec640f364005fa24e797b360 to abea67e08ee83539ea33aaae67bf0cddaa0b03b5 - [Release notes](https://github.com/github/gh-aw-actions/releases) - [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/gh-aw-actions/compare/ba90f2186d7ad780ec640f364005fa24e797b360...abea67e08ee83539ea33aaae67bf0cddaa0b03b5) --- updated-dependencies: - dependency-name: github/gh-aw-actions dependency-version: abea67e08ee83539ea33aaae67bf0cddaa0b03b5 dependency-type: direct:production dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
14ee3c90f9 |
Rework evaluation to allow sharding of resource intensive evaluations (#692)
* Shard plugin evaluation via executionShard tag in eval.yaml MCP-heavy plugins like dotnet-msbuild reliably exhaust the GitHub-hosted runner host after ~87 min of cumulative MCP traffic (Copilot CLI plus dotnet dnx Microsoft.AITools.BinlogMcp child processes), which has produced no dashboard data for dotnet-msbuild scheduled runs since 2026-05-20. Cutting per-runner concurrency alone was insufficient: at the default 18-way concurrency the runner OOMs in 17-30 min; at 3-way it survives longer but still hits a host-level termination at ~87 min. Two mitigations, both required: 1. Sharding (this change). The discover step reads an optional top-level 'executionShard:' string from each tests/<plugin>/<skill>/eval.yaml and groups skills by tag, emitting one matrix entry per shard so each shard runs on a fresh runner. Untagged skills fall into a 'default' bucket. Plugins where all skills resolve to a single bucket emit one entry (unchanged behavior). The validator already calls IgnoreUnmatchedProperties so no Models.cs change is needed. Shard names follow '<plugin>--shard-<tag>' so the existing 'skill-validator-results-<plugin>--*' artifact glob keeps working. 2. Per-MCP-plugin concurrency caps (already in place). When plugin.json declares mcpServers, parallelism is forced to 1x3x1 with --judge-timeout 600 to avoid acute OOM within a shard. Also adds a workflow_dispatch 'plugin' input for targeted re-runs of a single plugin, and tags 10 dotnet-msbuild eval.yamls (heavy/medium); the remaining 8 fall into 'default'. Empirically the three shards complete in parallel in ~30-40 min each, well inside the 180-min job timeout, and successfully upload dashboard data. * Address PR review: lenient shard regex and validate dispatch plugin input - executionShard regex now tolerates optional leading whitespace, so an accidentally indented key still groups correctly instead of silently falling back to the default bucket. - workflow_dispatch 'plugin' input is now validated against ^[a-zA-Z0-9._-]+$ before any Join-Path/Test-Path use, preventing path-traversal style values (e.g. '../foo') from being injected into filesystem paths. |
||
|
Javier Calvarro Nelson
|
198e58c983 |
Add blazor skills to dotnet-blazor plugin (#357)
* Add dotnet-blazor plugin and convert-blazor-server-to-webapp skill Add the dotnet-blazor plugin with 9 skills covering Blazor Web App development: - plan-ui-change: Plan and scaffold UI features in Blazor Web Apps - create-blazor-project: Create new Blazor projects with proper render mode setup - author-component: Author Razor components with parameters, events, lifecycle - coordinate-components: Share state across components using CascadingValueSource/scoped services - use-js-interop: Call JavaScript from Blazor and vice versa - fetch-and-send-data: HTTP data access with proper service patterns - configure-auth: Set up ASP.NET Core Identity and authorization in Blazor - collect-user-input: Build forms with EditForm, validation, and file uploads - support-prerendering: Handle prerendering lifecycle and state persistence Add convert-blazor-server-to-webapp skill to the dotnet-aspnet plugin for migrating .NET 7 Blazor Server apps to .NET 8+ Blazor Web App architecture. All 10 skills pass evaluation with quality improvements ranging from 12-50% and overfitting scores within acceptable thresholds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/allowed-external-deps.txt * Update .gitignore --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
da5e63e878 |
[dotnet-msbuild] Fix AP-13/AP-14 false positives and add source-vs-packed layout guidance (#687)
* [dotnet-msbuild] Fix AP-13/AP-14 false positives and add source-vs-packed layout guidance The msbuild-quality-review run on microsoft/testfx surfaced two confidently-stated false positives that the maintainer rightly pushed back on: 1. The reviewer claimed unguarded `<Import>` forwarders in MSTest's NuGet `build/<tfm>/` folders point at non-existent paths under `buildTransitive/<tfm>/`. In reality those folders are produced at pack time by the `.nuspec` `<file target="...">` mappings, so the forwarders resolve correctly in every restored package. 2. The reviewer flagged backslashes in `<Import Project="...\...\...">` as a cross-platform 🔴 error. MSBuild's evaluator normalizes `\` to `/` on Unix-like systems via `FileUtilities.MaybeAdjustFilePath` / `ConvertToUnixSlashes` before resolving the path, so existing backslash-style imports work everywhere — confirmed by years of shipping MSTest to Linux/macOS users. This change updates the rubric so the same false positives don't recur across every consumer of the dotnet-msbuild plugin: * `msbuild-antipatterns/SKILL.md` AP-13 — adds an explicit "NuGet package forwarders" exception and tells reviewers to consult `.nuspec` / `<PackagePath>` before flagging. * `msbuild-antipatterns/SKILL.md` AP-14 — rewritten to distinguish where backslashes are a real bug (raw `<Exec>` shell strings, CDATA, non-MSBuild consumers — keep 🔴) from where they're only style (`<Import>` and other evaluator-routed paths — 🔵 with a cite to the MSBuild source). * `extension-points/SKILL.md` — adds a "Source Tree vs Packed Layout" section documenting the three packaging mechanisms (.nuspec `<file>` mappings, csproj `<PackagePath>` metadata, SDK pack conventions) that legitimately reshape the layout, with the cross-check procedure reviewers must run before flagging "missing-file" imports. * `agents/msbuild-code-review.agent.md` — Discovery now records the projected packed layout; Category 4 references the new AP-13/AP-14 nuances; new "Veracity gate" step downgrades or drops 🔴 findings that would imply currently-shipping CI is broken. No fixtures change: the `tests/dotnet-msbuild/msbuild-antipatterns/eval.yaml` rubric does not enumerate AP-13/AP-14 by id, and the F#-specific scenarios are unaffected. Markdownlint passes on all edited files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - extension-points: rewrite .nuspec example so the source file is genuinely shared (single buildTransitive/common/MyAdapter.props fan-out instead of three already-per-TFM sources); clarify <file target=> folder-vs-rename semantics. - extension-points: replace semicolon-list PackagePath example with the unambiguous multi-<None> form, and note in prose that semicolon syntax is also supported. Avoids reader confusion noted in review. - msbuild-antipatterns AP-13 cross-check: align search scope with extension-points (project directory AND any parent directory) so reviewers don't miss shared mono-repo nuspecs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * AP-13: drop ../ link to extension-points to satisfy skill-validator Skill-validator rejects file references that traverse out of the skill directory (parent-directory traversal). Replace the markdown link with a plain-text reference to the dotnet-msbuild/extension-points skill — the prose is still discoverable to humans and to the agent that loads the plugin, but no longer trips the validator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address second round of PR review feedback - AP-13: replace <tfm> placeholders inside XML attribute strings with concrete TFM examples (net8.0) so the snippets are valid copy-pasteable XML. - AP-13 & extension-points: bound the .nuspec search to project directory + IMMEDIATE parent only (no unbounded walking); wording now identical in both docs. - AP-14: add explicit text severity next to severity emoji (🔴 Error / 🔵 Style) for accessibility — screen readers and plain-text copies now convey severity without relying on color/emoji rendering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ca21fd61c1 |
run-tests: prompt-shaped multi-TFM keywords for plugin activation (#688)
* run-tests: use prompt-shaped multi-TFM keywords for plugin activation The 'Run tests in a multi-TFM project targeting a specific framework' eval scenario kept failing plugin-mode activation after #667. The skill description had 'multi-TFM projects (--framework)' as a brief parenthetical, but the scenario prompt uses different language: 'This test project targets both net8.0 and net9.0. I only want to run the tests on net9.0 using dotnet test.' Per the InvestigatingResults.md guidance for 'skill not activated' failures, expand that clause to include keywords from the prompt — 'running tests against a single target framework when a project targets multiple TFMs' plus a concrete '<TargetFrameworks>net8.0;net9.0</TargetFrameworks>' example and the '--framework <TFM>' invocation. Per-skill description is 947 chars, well under the 1024 spec cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace simple multi-TFM scenario with a harder MTP+SDK 9 variant that requires skill knowledge The previous 'Run tests in a multi-TFM project targeting a specific framework' scenario tested a trivial prompt (single --framework flag) that the model already answers perfectly from training (baseline 4.7/5). This matches pattern #8 (baseline already good, no headroom) in InvestigatingResults.md, and explains why two prior attempts to fix plugin-mode activation via description tweaks (PR #667, #688) failed: the router rationally skips loading the skill for a trivial question. This change replaces the scenario with a meaningfully harder one that combines several signals the model often confuses: - Multi-TFM project (TargetFrameworks plural) - MTP runner (TestingPlatformDotnetTestSupport) - .NET SDK 9 (-- separator required for MTP extension args) - TRX report (MTP extension flag) The correct answer requires knowing that --framework is a dotnet test/MSBuild flag that always goes BEFORE --, while --report-trx is an MTP extension flag that goes AFTER -- on SDK 9: dotnet test --framework net9.0 -- --report-trx The ordering assertion in eval.yaml enforces this on a single line, so swapping --framework after the separator no longer satisfies the scenario. Also fix a documentation bug in SKILL.md surfaced by this work: the 'Common MTP flags > Built-in flags' table previously listed --framework and --no-build under 'On SDK 8/9, pass after --', which is wrong. Those are dotnet test/MSBuild flags consumed by dotnet test itself; they always go before --. The table now lists only flags genuinely consumed by MTP, and an Important note clarifies the rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5810d492c9 |
test-anti-patterns / test-smell-detection: narrow-specialist re-pitch (Variant B) (#671)
Both skills suffer plugin-mode activation failures: in plugin runs the model bypasses the skill tool entirely for ~5/6 of test-anti-patterns audit prompts and ~2/3 of test-smell-detection prompts, answering directly from training. Variant B repositions the two skills as umbrella + narrow specialist: * test-anti-patterns becomes THE umbrella audit skill for all pragmatic test-quality reviews (anti-patterns and smell-style prompts alike). Its description explicitly absorbs smell-audit triggers. * test-smell-detection is repositioned as a narrow niche that only fires when the user explicitly asks for the testsmells.org / academic 19-smell catalog with citable smell names from the research literature. Eval prompts for test-smell-detection are rewritten to match the narrow niche (explicit testsmells.org / 19-smell catalog requests) so we test specialist activation rather than the umbrella scenarios. Variant B of a 3-PR experiment to fix activation. Companion PRs: * Variant A = activation-only rewrite, both skills keep overlap; * Variant C = full merge into test-anti-patterns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
362b3f25ec |
Fix plugin-mode skill activation for 8 dotnet-test scenarios (#667)
* migrate-vstest-to-mtp: fix plugin activation for OutputType=Exe scenario The `Set OutputType=Exe only for test projects in Directory.Build.props` scenario was failing plugin-mode skill activation (skillActivationPlugin. activated=false, detectedSkills=[]) in the latest scheduled eval. The model answered from its own knowledge and proposed the wrong solution (`Condition="'$(IsTestProject)' == 'true'"` in Directory.Build.props, which the skill explicitly warns against). Two fixes: 1. SKILL.md description: add an explicit `conditioning OutputType=Exe to test projects when centralizing MTP properties in Directory.Build.props` USE FOR clause so the description surfaces this specific MTP migration pitfall to the skill router. Per-skill description stays at <= 1024 chars (trimmed `configuration` -> `config` and `MSTest version upgrades (use migrate-mstest-* skills)` -> `MSTest version upgrades` to keep within the spec limit). 2. eval.vally.yaml: add the missing scenario (the eval.yaml scenario was added in PR #556 but vally was never synced — vally had 10 stimuli vs eval.yaml's 11). Brings vally and skill-validator coverage to parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: strengthen description for severity-ranked audits The `Detect mixed severity anti-patterns in repository service tests` scenario was failing plugin-mode skill activation (skillActivationPlugin.activated=false, detectedSkills=[]) in the latest scheduled eval, even though the prompt explicitly says `Audit the test class ... for .NET test anti-patterns ... Give me a severity-ranked list (Critical / Warning / Info)`. The previous description only mentioned `USE FOR: audit test quality, review test code, find test anti-patterns, ...`, so the model could not distinguish this skill from sibling test-quality skills based on the `severity-ranked` and `Critical/Warning/Info` framing in the prompt. Surface the catalog shape (severity-ranked output, concrete code-level fixes) and add explicit `severity-ranked test audit`, `self-comparing assertions` and `broad exception types` triggers — these are the exact phrases used by failing scenarios. Description stays at 822 chars (<= 1024 limit). Aggregate dotnet-test description stays at 15,099 chars (<= 20,000 limit). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * run-tests: restructure description for higher-signal plugin activation Six run-tests scenarios were failing plugin-mode skill activation in the latest scheduled eval (skillActivationPlugin.activated=false, detectedSkills=[]) even though isolated activation worked for every scenario: - Run tests with trx reporting on MTP project (SDK 9) - Run tests in a multi-TFM project targeting a specific framework - Filter MSTest tests by category on VSTest - Filter NUnit tests by class name on VSTest - Filter TUnit tests by class using treenode-filter - Negative test: do not use MTP syntax for a VSTest project The previous description piled ~30 quoted trigger phrases (`run tests`, `run my tests`, `run these tests`, `execute tests`, `dotnet test`, `test filter`, `filter by category`, `filter by class`, `combine filters`, `run only specific tests`, `integration tests`, `unit tests`, `tests not running`, `hang timeout`, `blame-hang`, `blame-crash`, `crash dump`, `TRX report`, `TRX`, `test report`, `generate TRX`, `TUnit`, `treenode-filter`, `target framework`, `multi-TFM`, ...). In plugin mode this signal-poor keyword wall made the skill look generic enough that the router preferred to answer directly from model knowledge. Replace the keyword pile with five higher-signal scope clauses grouped by user-visible concern: 1. running, filtering, or troubleshooting `dotnet test` 2. VSTest vs Microsoft.Testing.Platform syntax (incl. `--` separator rules on SDK 8/9 vs 10+) 3. framework-specific filter syntax for MSTest / xUnit / NUnit / TUnit (--filter, --filter-class, --filter-trait, --filter-query, --treenode-filter) 4. TRX/reporting (--report-trx vs --logger trx), blame/hang/crash diagnostics, multi-TFM (--framework) 5. avoiding MTP/VSTest argument mixups These are the same concerns the previous description tried to cover, but framed as decisions the skill helps make (and that distinguish it from sibling test skills) rather than as a list of bare keywords. Per-skill description shrinks from 1,000 to 819 chars (still <= 1,024 limit). Aggregate dotnet-test description stays well under the 20,000-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: drop specific severity labels from description PR review (copilot-pull-request-reviewer on #667) caught that `Produces a severity-ranked (Critical / Warning / Info) catalog ...` in the new description contradicts the SKILL.md body, which actually teaches a 4-level Critical / High / Medium / Low taxonomy. The `Critical / Warning / Info` wording came from one eval scenario's prompt, but other prompts ask for different bucket schemes, and the skill itself does not enforce a fixed three-level mapping. Drop the specific labels from the description and keep just `severity-ranked` as the activation signal. The body remains unchanged (it teaches Critical / High / Medium / Low and the agent maps to the prompt's requested buckets at output time). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b24599aa36 |
test-smell-detection: trim to detailed tier and add prompt-shaped triggers (#653)
Multi-model skill-validator eval (claude-haiku-4.5, claude-opus-4.6, gpt-5.3-codex, gpt-5.4-mini) showed two issues with this skill: 1. Quality regression on stronger models (+0.15 haiku, -0.14 opus, -0.17 codex, -0.29 gpt-5.4-mini): the skill added tokens and tool calls without lifting rubric quality on larger models. 2. Plugin-mode activation loss: the skill activated reliably in isolated mode but was almost never picked in plugin mode where it competed with siblings (code-testing-agent, etc.). Changes: - Trim BPE tokens 2,487 -> 1,472 (still detailed tier, well within the 800-2,500 sweet spot per SkillsBench guidance). Collapsed long smell-by-smell prose, `Why Test Smells Matter`, and pitfalls into a compact taxonomy table plus a short decision procedure / output checklist. Removed bulky examples. Kept taxonomy, calibration, negative guidance, and references intact. - Add prompt-shaped USE FOR triggers drawn from the eval prompts (`review test smells`, `review these tests and tell me if there are` `any problematic patterns`, `check my tests for test design` `problems or anti-patterns`, `give us an objective assessment of` `our integration tests`, plus explicit smell names). - Strengthen DO NOT USE FOR clause: `write new tests / scaffold tests` `from scratch / generate a complete MSTest test suite` now defers cleanly to code-testing-generator (covers the negative eval scenario). Validation: skill-validator check --plugin clean for this skill; build + 545/545 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
0724ccf8ac | Update binlog mcp name (#684) | ||
|
Jan Krivanek
|
1a6e78c9c6 |
Add binlog MCP usage to other msbuild skills/agents (#683)
* Add binlog MCP usage to other msbuild skills/agents * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
5f1170152b |
Add MSBuild target authoring skills (target-authoring, property-patterns, item-management, extension-points) (#669)
* Add 4 MSBuild target authoring skills for dotnet-msbuild plugin New skills: - target-authoring: three-level target chain, DependsOn extension, naming conventions - property-patterns: conditional defaults, composition, path normalization, TFM helpers - item-management: Include/Remove/Update, batching, transforms, FileWrites registration - extension-points: CustomBefore/After hooks, import gating, NuGet build extensions Each skill includes eval.yaml tests with anti-pattern fixtures. Closes #668 * Address Copilot review feedback - target-authoring: Reword DO NOT USE clause to exclude only deep incremental-build diagnostics, not basic Inputs/Outputs usage - property-patterns: Close unclosed XML elements in String Functions snippet (PropertyGroup, TargetFrameworkMoniker) - extension-points: Add missing CustomAfterMySDK property definition to match the import at bottom of example * Remove accidental pr-body.md file * Fix wildcard import placeholder path and quote NormalizePath args - extension-points: Use full MSBuildExtensionsPath expression and consistent Exists() casing in wildcard import example - property-patterns: Quote property arguments in NormalizePath call * Tune eval prompts to reduce overfitting Rewrite all 4 eval prompts as natural developer problem descriptions instead of skill-aligned checklists. Softens technique-prescriptive rubric items. Result: overfitting scores drop from 0.15-0.31 to 0.06-0.08 (all green). property-patterns now passes eval. * Fix Copilot review round 3: quote property function args, fix fixture - property-patterns: Quote in IsPathRooted call - extension-points: Quote in GetDirectoryNameOfFileAbove - item-management fixture: Use literal semicolon instead of %3B in WriteLinesToFile * Add harder multi-file eval scenarios for all 4 skills Each skill now has a second scenario with multi-file setups containing interacting bugs that require cross-file analysis: - target-authoring: SDK .targets + Directory.Build.targets with 3 bugs (target redefinition, fragile BeforeTargets, duplicate DependsOn chain) - property-patterns: nested Directory.Build.props with 5 bugs (import order, unconditional overwrite, unquoted condition, missing trailing slash, NoWarn overwrite) - item-management: csproj with 5 interacting bugs (Include vs Update, cross-product batching, eval-time FileWrites, too-broad Remove glob) - extension-points: Directory.Build.props/targets + NuGet package with 6 bugs (inverted guard, late ImportByWildcard, CustomBefore overwrite, missing Exists guard, target name collision, internal target hook) Timeouts increased to 180s for all scenarios. * Fix review round 4: revert unintended README change, quote intrinsic call in condition * Improve eval quality for 4 new MSBuild authoring skills Eval improvements across target-authoring, property-patterns, item-management, and extension-points skills: - Rewrite prompts to describe observable symptoms without naming MSBuild concepts, inspired by real dotnet/msbuild and dotnet/sdk issues (msbuild#2470, #12894, #13056, #4109, sdk#43908) - Broaden assertion regex patterns to test for diagnostic outcomes rather than skill-specific vocabulary - Improve rubric items to test understanding and diagnosis flow (diagnosis -> root cause -> fix) instead of skill terminology - Add task-completion 'fix it' scenario to each skill with file_contains assertions that verify the agent applied correct patterns (CompileDependsOn append, FileWrites, Exists guard, etc.) - Remove leading 'Anti-pattern:', 'BUG N:' comments from fixture files that were giving away answers to the baseline agent, making scenarios more realistic and better at measuring skill value Each skill now has 3 scenarios: basic review, hard multi-file review, and a task-completion fix scenario. Total: 12 scenarios across 4 skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix review comments: quote TFM intrinsic, fix PrivateAssets example - property-patterns SKILL.md: Quote the GetTargetFrameworkIdentifier intrinsic call result in the Condition to match the 'always quote both sides' guidance and avoid brittle parse when empty - item-management SKILL.md: Replace PackageReference Update on Microsoft.NETCore.App (not a real user-added item) with Include on Microsoft.CodeAnalysis.NetAnalyzers (a concrete analyzer package) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval activation and timeouts based on eval run results - Add MSBuild domain terms to prompts (e.g., 'MSBuild targets', 'MSBuild property patterns', 'MSBuild item groups', 'MSBuild extension points') to improve skill activation rates — skills were NOT ACTIVATED in plugin mode due to prompts being too symptom-focused without domain keywords - Increase timeouts: review scenarios 180s -> 240s, fix-it scenarios 180s -> 300s to avoid timeout-related scoring penalties - Keep prompts symptom-driven but include enough domain context for the skill router to activate the correct skill Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments and improve eval quality Review comment fixes: - TargetAuthoring.csproj: Replace %22/%3B URL encoding with XML escaping ("/;) so generated C# code is valid - target-authoring eval: Fix prompt referencing 'schema files' when fixture has endpoint definitions — now says 'endpoint list' - property-patterns eval: Reword LangVersion rubric — unconditional assignment in .props IS overridable by csproj but prevents override from earlier imports and command-line properties - extension-points eval: Add stronger activation terms to Fix scenario prompt (explicit mention of CustomBeforeMicrosoftCommonTargets) Eval quality improvements based on CI results: - Shorten target-authoring rubric items to reduce overfitting (was 0.30) - Remove 'Focus on target authoring patterns' instruction from hard scenario prompt (was directing agent too specifically) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval: increase timeout, add activation keywords, explicit fix instructions - property-patterns 'Diagnose multi-level': timeout 240s -> 360s (was hitting limit at 249s) - target-authoring 'Diagnose custom target': add CompileDependsOn/incremental build keywords for better skill activation - property-patterns 'Fix': add 'edit ... directly' to prompt to encourage file edits - target-authoring 'Fix': add 'edit ... directly' to prompt - item-management 'Fix': add 'edit ... directly' to prompt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix target-authoring skill description and eval prompts for plugin activation The skill description said 'DO NOT USE FOR: deep incremental-build diagnostics' which caused the plugin agent to skip it for scenarios about build regressions. Clarified that the skill IS for diagnosing target authoring mistakes (missing Inputs/Outputs, broken dependency chains) that cause full rebuilds. Also updated eval prompts to use 'target authoring mistakes' framing and increased Fix scenario timeout to 360s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase eval timeouts: extension-points NuGet (240->360s), item-management cascading (240->360s) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 3 fresh PR review comments + update eval prompts for target-authoring - property-patterns/SKILL.md: add warning that TargetFramework-conditioned PropertyGroups must be in .targets not .props (empty in single-target .props) - extension-points/SKILL.md: add Exists() guard to GetPathOfFileAbove Import to prevent build failure when no parent file is found - target-authoring/SKILL.md: fix comment mismatch — validation target runs via dependency chain, not BeforeTargets; update comment accordingly - target-authoring/eval.yaml: reframe 'Diagnose' and 'Fix' prompts to use 'dependency chain' / 'authoring patterns' language (avoids routing to incremental-build skill which also claims Inputs/Outputs diagnostics) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve skill descriptions for reliable plugin-mode activation Add 'Only activate in MSBuild/.NET build context.' prefix to all 4 skills for consistency with existing skills in the plugin. Add explicit 'diagnosing and fixing' and 'reviewing' keywords to USE FOR sections so the SDK selects these skills over the broader msbuild-antipatterns skill when prompts ask to fix or review specific item/extension/property/target patterns. Add 'general MSBuild anti-pattern catalog (use msbuild-antipatterns)' to DO NOT USE FOR sections to help the SDK disambiguate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill descriptions: shorten to under 1024-char SDK limit All 4 new skills had descriptions exceeding the 1024-character maximum enforced by the skill-validator check command. The Copilot SDK silently ignores skills with over-length descriptions, causing 0% activation in both isolated and plugin evaluation modes. Shortened all 4 descriptions while preserving key activation keywords (USE FOR / DO NOT USE FOR / 'Only activate in MSBuild/.NET build context'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - target-authoring eval.yaml: fix prompt to say 'Outputs attribute' (not 'Returns') since the fixture uses Outputs on a query target, which is the actual bug that causes MSBuild to skip re-execution - property-patterns eval.yaml: reword prompt and rubric — the real issue is unconditional assignment breaking parent-child Directory.Build.props inheritance, not project files being unable to override properties - extension-points SKILL.md: fix GetPathOfFileAbove example to factor the path into a property so Project= and Condition= use the same value consistently (avoiding the ..\\ vs ..\ discrepancy) - CustomSdk.targets: add WriteLinesToFile to CoreCodeGen so it actually creates the .g.cs output files (prevents compilation failure in fixture) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix property-patterns fixtures to match rubric claims Scenario 1 (path merge bug): - PropertyPatterns.csproj: add OutputPath using CustomOutputDir without a trailing separator, making the merge bug observable (artifacts\binPropertyPatterns\ instead of artifacts\bin\PropertyPatterns\) Scenario 2 (multi-level hierarchy bugs): - hard/Directory.Build.props: make LangVersion unconditional so it overwrites the child src/Directory.Build.props LangVersion=preview, matching the rubric claim about parent unconditional assignments - hard/Directory.Build.props: add NoWarn=NU1702 (conditional default) so parent suppressions are visible but lost when the child's unconditional <NoWarn>CS1591;IDE0005</NoWarn> overwrites them after import Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Sayed Ibrahim Hashimi
|
3b0ece94ad |
Add web api skill and tests (#613)
* adding web api skill and tests * fixing link * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
e951d69ab9 |
dotnet-msbuild: move AITools.BinlogMcp source to dotnet-tools feed (#680)
The MCP package was relocated from the dotnet-eng public feed to the dotnet-tools public feed on dnceng. Update plugin.json --add-source URL and the SKILL.md fallback note accordingly. |