diff --git a/plugins/dotnet-test-migration/skills/migrate-mstest-v1v2-to-v3/SKILL.md b/plugins/dotnet-test-migration/skills/migrate-mstest-v1v2-to-v3/SKILL.md index b046ae52..4fb0f340 100644 --- a/plugins/dotnet-test-migration/skills/migrate-mstest-v1v2-to-v3/SKILL.md +++ b/plugins/dotnet-test-migration/skills/migrate-mstest-v1v2-to-v3/SKILL.md @@ -1,15 +1,16 @@ --- name: migrate-mstest-v1v2-to-v3 description: > - Upgrade, compare, or repair MSTest v1/v2 projects during migration to v3. - Use for QualityTools assembly references; MSTest.TestFramework/TestAdapter - 1.x-2.x; choosing the MSTest metapackage or MSTest.Sdk; "what breaking - changes should I expect?"; CS0411/CS1503 after a v3 package bump; DataRow - type mismatch or MSTEST0014; .testsettings/LegacySettings to .runsettings; - timeout changes; and dropped v3 TFMs such as net5.0. Also use when packages - already say 3.x but v1/v2 source or settings remain, and when asked whether - v1 and v2 migration steps differ. Preserve VSTest/MTP. Do not use for clean - v3 projects, v3-to-v4, another test framework, or runner-only migration. + Use this skill before answering or editing whenever an MSTest v1/v2 project + is being upgraded or repaired for v3. Triggers include QualityTools assembly + references; MSTest.TestFramework/TestAdapter 1.x-2.x; "upgrade to MSTest + v3"; comparing v1 and v2 migration paths; choosing MSTest or MSTest.Sdk; + CS0411/CS1503 after a v3 package bump; DataRow type mismatch, + MSTEST0014, or "Test data doesn't match method parameters"; + .testsettings/LegacySettings to .runsettings; timeout changes; and net5.0 or + other dropped v3 TFMs. Still use it when packages say 3.x but migration + source errors or legacy settings remain. Preserve VSTest/MTP. Do not use for + a clean v3 project, v3-to-v4, framework conversion, or runner-only migration. license: MIT --- @@ -17,6 +18,14 @@ license: MIT Migrate a test project from MSTest v1 (assembly references) or MSTest v2 (NuGet 1.x-2.x) to MSTest v3. MSTest v3 is **not binary compatible** with v1/v2 -- libraries compiled against v1/v2 must be recompiled. +## First Action + +Inspect the supplied workspace and classify v1, v2, partially migrated v3, or +already-complete v3 before answering. Do not search the web or answer from +memory first. For an edit request, continue through the requested source +changes and validation; for an advice request, answer directly after the +classification. + ## When to Use - Project references `Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll` (MSTest v1) @@ -70,7 +79,7 @@ This overrides all steps below. - The skill directory contains guidance, not the staged project. Search the current working directory, open the literal paths returned by the search, and retry with another available reader/editor if one tool rejects a valid path. - Never ask for a path while a glob or directory search can discover it. Ask only after an exhaustive current-workspace search finds no project, or multiple projects make the target genuinely ambiguous. - Classify the requested deliverable, not isolated verbs: "make the edits", "update this project", or "then build and run" means execute; "what do I need to change?", "what should I expect?", "are the steps the same?", or "show me" means answer, even if the prompt also says upgrade or migrate. -- After changing files, name the detected MSTest version and runner, the files changed, the exact source/settings decisions, and the clean test result. Do not claim VSTest preservation, a build, or passing tests without evidence from the project. +- After changing files, name the detected MSTest version and runner, every file changed, and the exact repaired calls/settings (for example, list each assertion changed to `AreEqual`, `AreNotEqual`, or `AreSame`). Report the clean test counts. Do not claim VSTest preservation, a build, or passing tests without evidence from the project. ## Breaking Changes Summary diff --git a/plugins/dotnet-test-migration/skills/migrate-mstest-v3-to-v4/SKILL.md b/plugins/dotnet-test-migration/skills/migrate-mstest-v3-to-v4/SKILL.md index 55d7e7e6..9aac69b0 100644 --- a/plugins/dotnet-test-migration/skills/migrate-mstest-v3-to-v4/SKILL.md +++ b/plugins/dotnet-test-migration/skills/migrate-mstest-v3-to-v4/SKILL.md @@ -1,15 +1,16 @@ --- name: migrate-mstest-v3-to-v4 description: > - Upgrade MSTest 3.x projects to v4 or fix v4 migration failures. Use for - "MSTest v4 breaking changes", tests that stop compiling or behave differently - after 3.x-to-4.x, CS0507/CS0103/CS1061/CS1615, ExecuteAsync or CallerInfo in - custom TestMethodAttribute, DisplayName, ClassCleanupBehavior, ContainsKey, - ThrowsExactly/ExpectedException, IsInstanceOfType out parameters, TestTimeout, - ManagedType, net6/net7 compatibility, TestCase.Id history, TestName in - ClassInitialize, TreatDiscoveryWarningsAsErrors, and MSTest.Sdk/MTP or - vstest.console discovery changes. Do not use for v1/v2-to-v3 leftovers, - framework conversion, runner-only migration, or a general .NET upgrade. + Use this skill before answering, planning, or editing any MSTest 3.x-to-4.x + upgrade or post-upgrade failure. Triggers include "MSTest v4 breaking + changes"; CS0507/CS0103/CS1061/CS1615; ExecuteAsync, CallerInfo, DisplayName, + or custom TestMethodAttribute; ClassCleanupBehavior; ContainsKey; + ThrowsExactly or ExpectedException; IsInstanceOfType out parameters; + TestTimeout.Infinite; ManagedType; net6/net7 compatibility; TestCase.Id + history; TestName in ClassInitialize; TreatDiscoveryWarningsAsErrors; + discovery errors after a clean build; and MSTest.Sdk/MTP or vstest.console + discovery changes. Do not use for v1/v2-to-v3 leftovers, framework + conversion, runner-only migration, or a general .NET upgrade. license: MIT --- @@ -17,6 +18,15 @@ license: MIT Migrate a test project from MSTest v3 to MSTest v4. The outcome is a project using MSTest v4 that builds cleanly, passes tests, and accounts for every source-incompatible and behavioral change. MSTest v4 is **not binary compatible** with MSTest v3 -- any library compiled against v3 must be recompiled against v4. +## First Action + +Inspect the supplied project and source before searching the web or answering +from memory. Classify the request as a focused source fix, runtime behavior +change, CI discovery issue, compatibility question, or full migration, then +follow the matching row below. A clean compile does not exclude this skill: +discovery failures, `TestContext` lifecycle exceptions, and test-history +changes are runtime migration failures. + ## When to Use - Upgrading `MSTest.TestFramework`, `MSTest.TestAdapter`, or `MSTest` metapackage from 3.x to 4.x @@ -49,13 +59,14 @@ Migrate a test project from MSTest v3 to MSTest v4. The outcome is a project usi | User asks "what should I expect?", "how do I fix these changes?", for compatibility advice, or for a plan | Answer directly from the actual project state even when source is visible. Keep a single-symptom answer focused; include only adjacent risks that change the decision. | | Unsupported TFM in a full migration | Update the TFM first, then update MSTest packages, then fix source breaks. Do not bury this order in a release-note inventory. | | Custom `TestMethodAttribute` subclass | Treat `ExecuteAsync`, CallerInfo propagation, display-name handling, and the subclass's retry/result semantics as one coupled migration. Fix the actual class, not a placeholder example. | +| `MSTest.Sdk` v4 source/API errors (`ManagedType`, `TestTimeout`, `Contains`) | Give the exact source replacements, then add the adjacent runner warning: MTP mode no longer supplies `Microsoft.NET.Test.Sdk`; add it only if VSTest discovery is still required. | | `MSTest.Sdk` v4 plus `vstest.console` | This is a v4 change: MTP mode no longer brings `Microsoft.NET.Test.Sdk`. Keep MTP and add that package for transitional VSTest discovery, opt into `UseVSTest`, or migrate CI to `dotnet test`; state which runner the choice preserves. | ## Response Guidelines - **Always identify the current version first**: Before recommending any migration steps, explicitly state the current MSTest version detected in the project (e.g., "Your project uses MSTest v3 (3.8.0)"). This confirms you've read the project files and grounds the migration advice. - **Resolve, do not assume, the target version**: When the user asks for "latest", query the project's configured package source and select the latest stable MSTest v4 version available at execution time. Never copy the example version from this skill into the result without checking it. Keep all MSTest packages on the same resolved version. -- **Focused fix requests** (user has specific compilation errors after upgrading): Address only the relevant breaking changes from Step 3. Make edits only when the requested deliverable is a source change; "how do I fix these?" remains an answer request. **Always provide concrete fixed code** using the user's actual types and method names. If the fixture still references v3, do not claim that a green v3 build verifies v4 compatibility; either update packages when requested or state the verification boundary. For custom `TestMethodAttribute` subclasses, show the full fixed class including CallerInfo propagation to the base constructor. Mention any related analyzer that could have caught this earlier (e.g., MSTEST0006 for ExpectedException). Do not walk through the entire migration workflow. +- **Focused fix requests** (user has specific compilation errors after upgrading): Address only the relevant breaking changes from Step 3. Make edits only when the requested deliverable is a source change; "how do I fix these?" remains an answer request. **Always provide concrete fixed code** using the user's actual types and method names. If the fixture still references v3, do not claim that a green v3 build verifies v4 compatibility; either update packages when requested or state the verification boundary. For custom `TestMethodAttribute` subclasses, show the full fixed class including CallerInfo propagation to the base constructor. Mention any related analyzer that could have caught this earlier (e.g., MSTEST0006 for ExpectedException). When the project uses `MSTest.Sdk`, also state that v4 MTP mode no longer supplies `Microsoft.NET.Test.Sdk` and whether that affects the visible runner. Do not walk through the entire migration workflow. - **"What to expect" questions** (user asks about breaking changes before upgrading): Present ALL major breaking changes from the Step 3 quick-lookup table -- not just the ones visible in the current code. For each, provide a one-line fix summary. Also mention key behavioral changes from Step 4 (especially TestCase.Id history impact and TreatDiscoveryWarningsAsErrors default). If project code is available, highlight which changes apply directly. - **Full migration requests** (user wants complete migration): Follow the complete workflow below. - **Behavioral/runtime symptom reports** (user describes test execution differences without build errors): Match described symptoms to the behavioral changes table in Step 4. Provide targeted, symptom-specific advice. Mention other behavioral changes the user should watch for. Do not walk through source breaking changes unless the user also has build errors. diff --git a/plugins/dotnet-test-migration/skills/migrate-vstest-to-mtp/SKILL.md b/plugins/dotnet-test-migration/skills/migrate-vstest-to-mtp/SKILL.md index 8f587408..fab185cb 100644 --- a/plugins/dotnet-test-migration/skills/migrate-vstest-to-mtp/SKILL.md +++ b/plugins/dotnet-test-migration/skills/migrate-vstest-to-mtp/SKILL.md @@ -1,16 +1,16 @@ --- name: migrate-vstest-to-mtp description: > - Migrate .NET test projects and CI from VSTest to Microsoft.Testing.Platform - (MTP), or fix MTP migration behavior. Use for "switch from VSTest", MTP - runner enablement for MSTest/NUnit/xUnit, OutputType=Exe conditions in - Directory.Build.props, EnableMSTestRunner/EnableNUnitRunner/ - UseMicrosoftTestingPlatformRunner, YTest.MTP.XUnit2, .NET 10 global.json - test.runner, --filter-class/--filter-method/--filter-trait/--filter-query translation, - VSTest@3 replacement, TRX/coverage/dump extensions, and exit code 8 or zero - tests. Supports xUnit v3 MTP filter work even during a v2-to-v3 upgrade. Do - not use for test-framework conversion, xUnit v2-to-v3 API migration, TFM, - UWP, or WinUI migration. + Use this skill before answering, planning, or editing whenever .NET tests or + CI are switching from VSTest to Microsoft.Testing.Platform (MTP), or an MTP + migration behaves differently. Triggers include "switch from VSTest"; + MSTest/NUnit/xUnit MTP enablement; OutputType=Exe only for test projects in + Directory.Build.props; EnableMSTestRunner, EnableNUnitRunner, + UseMicrosoftTestingPlatformRunner, or YTest.MTP.XUnit2; .NET 10 global.json + test.runner and TestingPlatformDotnetTestSupport; translating VSTest + filters, logger, coverage, blame, or dump arguments; replacing VSTest@3; + and exit code 8 or zero tests. Also use for xUnit v3 MTP filters during a + v2-to-v3 upgrade. Do not use for framework conversion, TFM, UWP, or WinUI. license: MIT --- @@ -18,6 +18,16 @@ license: MIT Migrate a .NET test solution from VSTest to Microsoft.Testing.Platform (MTP). The outcome is a solution where all test projects run on MTP, `dotnet test` works correctly, and CI/CD pipelines are updated. +## First Action + +Inspect the supplied project, `Directory.Build.props`, `global.json`, and CI +files before searching the web or answering from memory. Resolve the framework +and SDK mode first: .NET 9 and earlier use the compatibility property plus the +`--` separator; .NET 10 native MTP uses `global.json`, removes that property, +and passes MTP arguments without the separator. For central properties, never +condition on `IsTestProject` in `Directory.Build.props`; use a property already +available there, such as `MSBuildProjectName`. + > **Important**: Do not mix VSTest-based and MTP-based .NET test projects in the same solution or run configuration -- this is an unsupported scenario. ## When to Use diff --git a/tests/dotnet-test/test-gap-analysis/eval.yaml b/tests/dotnet-test/test-gap-analysis/eval.yaml index 297d770d..3d895703 100644 --- a/tests/dotnet-test/test-gap-analysis/eval.yaml +++ b/tests/dotnet-test/test-gap-analysis/eval.yaml @@ -241,13 +241,67 @@ stimuli: - For this advisory request, accurate source/assertion mapping after one green baseline is sufficient; do not reward applying mutations over an equally accurate static assessment + - name: Verify every meaningful outcome in a focused discount review + prompt: | + Review only DiscountRules.Apply under fixtures/close-partial-suite and its + existing tests. Verify which caller-visible discount and validation + outcomes could change without a test failing. Keep the review focused on + this public method, account for every meaningful high-risk outcome, and + leave all source and test files unchanged when you finish. Recommend tests + only for gaps you can demonstrate. The workspace also contains an unrelated + AccessControl fixture; do not expand the requested scope to analyze it. + environment: + files: + - src: fixtures/close-partial-suite + dest: fixtures/close-partial-suite + - src: fixtures/logic-gaps + dest: unrelated/logic-gaps + commands: + - mkdir -p .eval-baseline && cp fixtures/close-partial-suite/src/DiscountRules.cs .eval-baseline/DiscountRules.cs && cp fixtures/close-partial-suite/tests/DiscountRulesTests.cs .eval-baseline/DiscountRulesTests.cs + graders: + - type: output-matches + config: + pattern: (whitespace|IsNullOrWhiteSpace) + - type: output-matches + config: + pattern: (case.insensitive|lowercase|ToUpperInvariant) + - type: output-matches + config: + pattern: (FLAT5|floor|below five|below 5) + - type: output-matches + config: + pattern: (negative|ArgumentOutOfRangeException) + - type: output-matches + config: + pattern: (unknown|ArgumentException) + - type: output-not-matches + config: + pattern: (AccessControl|AccessChecker|CanWrite|ElevateRole|Guest access) + - type: run-command + config: + command: sh -c "diff -u .eval-baseline/DiscountRules.cs fixtures/close-partial-suite/src/DiscountRules.cs && diff -u .eval-baseline/DiscountRulesTests.cs fixtures/close-partial-suite/tests/DiscountRulesTests.cs" + expected_exit_code: 0 + timeout: 1m + - type: prompt + rubric: + - Scope stayed proportional to the request by analyzing only DiscountRules.Apply and excluding the unrelated + AccessControl fixture + - Accounted for every high-risk public outcome in scope, including whitespace handling, case normalization, + the FLAT5 floor, negative-subtotal rejection, and unknown-code rejection + - Verified each reported survivor against the existing suite rather than stopping after the first gap + - Restored every temporary production-code change and left the existing source and tests unchanged + - Recommended focused tests only for demonstrated gaps and did not recommend redundant coverage for null + codes or ordinary uppercase SAVE10 behavior + - name: Verify and close only the surviving gaps in a partial suite prompt: | DiscountRules under fixtures/close-partial-suite/ already has a passing xUnit suite. Find the meaningful behaviors those tests would fail to catch, verify the gaps, and add focused tests that close them. Leave production code and DiscountRulesTests.cs byte-for-byte unchanged. Do not add tests - for behavior the existing suite already protects. + for behavior the existing suite already protects. Run the supplied + verify_mutations.sh after adding tests and report its result; do not replace + it with an ad-hoc mutation check or a broad green test run. environment: files: - src: fixtures/close-partial-suite @@ -257,7 +311,7 @@ stimuli: graders: - type: run-command config: - command: sh -c "dotnet test fixtures/close-partial-suite/tests/DiscountRules.Tests.csproj" + command: sh -c "dotnet run --project fixtures/close-partial-suite/tests/DiscountRules.Tests.csproj" expected_exit_code: 0 timeout: 10m - type: run-command @@ -275,6 +329,12 @@ stimuli: command: sh -c "test $(grep -RhE '\\[(Fact|Theory)\\]' fixtures/close-partial-suite/tests --include='*.cs' | wc -l) -le 8" expected_exit_code: 0 timeout: 1m + - type: output-matches + config: + pattern: (verify_mutations\.sh|canonical mutation verifier) + - type: output-matches + config: + pattern: '(?m)^[^\r\n]*`?verify_mutations\.sh`?(?: command)? (passed|succeeded|completed successfully)\.?$' - type: prompt rubric: - Empirically verified candidate survivors against the original narrow suite before adding tests @@ -282,6 +342,7 @@ stimuli: - Left production and the existing test file byte-for-byte unchanged - Re-applied the original mutations after implementation and confirmed the added tests kill them - Did not duplicate the existing null-code or ordinary uppercase SAVE10 behavior + - Explicitly reported the successful supplied verifier result instead of substituting an ad-hoc proxy - Judge completeness by the five verified behaviors, not raw test count; extra zero-subtotal or redundant happy-path cases are not an advantage - name: Decline request to write new tests from scratch diff --git a/tests/dotnet-test/test-gap-analysis/fixtures/close-partial-suite/verify_mutations.sh b/tests/dotnet-test/test-gap-analysis/fixtures/close-partial-suite/verify_mutations.sh index 3972fc09..fdf2a6d6 100644 --- a/tests/dotnet-test/test-gap-analysis/fixtures/close-partial-suite/verify_mutations.sh +++ b/tests/dotnet-test/test-gap-analysis/fixtures/close-partial-suite/verify_mutations.sh @@ -1,23 +1,38 @@ #!/usr/bin/env bash set -euo pipefail -source_file="fixtures/close-partial-suite/src/DiscountRules.cs" -test_project="fixtures/close-partial-suite/tests/DiscountRules.Tests.csproj" +script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +source_file="$script_dir/src/DiscountRules.cs" +test_project="$script_dir/tests/DiscountRules.Tests.csproj" backup="$(mktemp)" cp "$source_file" "$backup" +if command -v python3 >/dev/null 2>&1 && python3 -c 'import sys; raise SystemExit(sys.version_info < (3,))' >/dev/null 2>&1; then + python_command="python3" +elif command -v python >/dev/null 2>&1 && python -c 'import sys; raise SystemExit(sys.version_info < (3,))' >/dev/null 2>&1; then + python_command="python" +else + echo "Python 3 is required to run the mutation verifier." >&2 + exit 1 +fi + restore() { cp "$backup" "$source_file" } trap 'restore; rm -f "$backup"' EXIT +if ! baseline_output="$(dotnet run --project "$test_project" 2>&1)"; then + printf '%s\n' "$baseline_output" >&2 + exit 1 +fi + expect_killed() { local old="$1" local new="$2" local label="$3" restore - python3 - "$source_file" "$old" "$new" <<'PY' + "$python_command" - "$source_file" "$old" "$new" <<'PY' import pathlib import sys @@ -30,9 +45,26 @@ if old not in content: path.write_text(content.replace(old, new, 1), encoding="utf-8") PY - if dotnet test "$test_project" --nologo -v:q >/dev/null 2>&1; then + local build_output + if ! build_output="$(dotnet build "$test_project" --nologo -v:q 2>&1)"; then + printf '%s\n' "$build_output" >&2 + exit 1 + fi + + local test_output + if test_output="$(dotnet run --project "$test_project" --no-build 2>&1)"; then echo "Mutation survived: $label" >&2 exit 1 + else + local test_exit_code=$? + if grep -q "=== TEST EXECUTION SUMMARY ===" <<<"$test_output" && + grep -Eq "Failed: [1-9][0-9]*" <<<"$test_output"; then + return + fi + + echo "Mutation test infrastructure failed for $label (exit code $test_exit_code)." >&2 + printf '%s\n' "$test_output" >&2 + exit "$test_exit_code" fi }