Fix the two skill defects behind the v1v2-to-v3 eval losses

The first eval run reached 7W/2T/2L, p=0.090, short of the p<=0.05 gate. Both
losses trace to skill content that actively misled the agent, and the session
transcripts show exactly how.

Loss 1 -- 'Migrate MSTest v1 project with assembly reference', skilled scored
0.00 against a 4.17 baseline. The transcript shows the skill loading correctly
and the agent then replying, in full: 'To give you specific migration steps, I
need to see your project file. Could you share the path to your .csproj?' The
project was already in the working directory. Cause: the Inputs table marked
'Project or solution path' as Required=Yes, which reads as a precondition the
agent must obtain before doing anything. This is the worst kind of failure for a
real user - they describe their project in prose and get a question back instead
of an answer. Path is now optional and discovered by globbing, Step 1 leads with
locating the project, and a note forbids opening with a request for the path.
The same Required=Yes trap was present in migrate-mstest-v3-to-v4 and
migrate-vstest-to-mtp, so both are corrected too.

Loss 2 -- 'Fix DataRow type mismatch errors', skilled 3.96 against a 5.00
baseline. The skill's breaking-change table said the 16-argument DataRow cap was
'fixed in later v3 versions' and suggested 'refactor test / wrap extra params in
array'. On a project already at MSTest 3.8, the agent concluded the valid
17-argument row exceeded the limit and rewrote it - first as new object[] { 17 },
which failed, then second-guessing itself mid-run ('let me check if the latest
3.x actually fixed the 16-arg limit'), finally settling on a (object)17 cast.
Churn plus wasted turns on code that was already correct.

The vague wording was the problem, so it is replaced with the fact: the cap was
introduced in 3.0.1 and removed again in 3.0.3 (microsoft/testfx#1554 and the
maintainer's 'please feel free to update to 3.0.3'). On 3.0.3+ a longer row is
valid and must be left alone. A general guideline is added alongside it -
confirm the diagnostic before editing, because rewriting valid code to dodge a
limit the project is not subject to is a defect rather than caution.

Both fixes are about what the skill tells a real user, not about the graders;
no eval prompt, fixture, or grader is touched. Skill grows ~480 tokens and stays
in the 'standard' tier, below the 5,000-token warning threshold.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
This commit is contained in:
Copilot App
2026-07-31 09:31:51 +02:00
parent 2fc8ab8f54
commit 6b11ad97fe
3 changed files with 25 additions and 9 deletions
@@ -52,10 +52,15 @@ This overrides all steps below.
| Input | Required | Description |
|-------|----------|-------------|
| Project or solution path | Yes | The `.csproj`, `.sln`, or `.slnx` entry point containing MSTest test projects |
| Project or solution path | No | The `.csproj`, `.sln`, or `.slnx` entry point. Glob the working directory for it; ask only if nothing is found or several test projects make the target ambiguous |
| Build command | No | How to build (e.g., `dotnet build`, a repo build script). Auto-detect if not provided |
| Test command | No | How to run tests (e.g., `dotnet test`). Auto-detect if not provided |
> **Never open by asking for the project path.** A user describing their project
> in prose is asking a question, not withholding a file -- look on disk first. If
> there is genuinely no project file, answer for the setup they described rather
> than replying with only a question.
## Breaking Changes Summary
MSTest v3 introduces these breaking changes from v1/v2. Address only the ones relevant to the project:
@@ -64,7 +69,7 @@ MSTest v3 introduces these breaking changes from v1/v2. Address only the ones re
|---|---|---|
| `Assert.AreEqual(object, object)` overload removed | Compile error on untyped assertions | Add generic type: `Assert.AreEqual<T>(expected, actual)`. Same for `AreNotEqual`, `AreSame`, `AreNotSame` |
| `DataRow` strict type matching | Runtime/compile errors when argument types don't match parameter types exactly | Change literals to exact types: `1` for int, `1L` for long, `1.0f` for float |
| `DataRow` max 16 constructor parameters (early v3) | Compile error if >16 args; fixed in later v3 versions | Update to latest 3.x, or refactor test / wrap extra params in array |
| `DataRow` limited to 16 arguments -- **3.0.1 and 3.0.2 only** | `CS1729` on those two versions; the limit was removed again in **3.0.3** | On 3.0.3+ (every current 3.x) a longer row is valid -- **leave it unchanged**. Do not wrap extras in an array, cast to `object`, or split the test. Only a project pinned to 3.0.1/3.0.2 needs action: update to 3.0.3+ |
| `.testsettings` / `<LegacySettings>` no longer supported | Settings silently ignored | Delete `.testsettings`, create `.runsettings` with equivalent config |
| Timeout behavior unified across .NET Core / Framework | Tests with `[Timeout]` may behave differently | Verify timeout values; adjust if needed |
| Dropped target frameworks: .NET 5, .NET Fx < 4.6.2, netstandard1.0, UWP < 16299, WinUI < 18362 | Build error | Update TFM: .NET 5 -> net8.0 (LTS) or net6.0+, netfx -> net462+, netstandard1.0 -> netstandard2.0. Note: net6.0, net8.0, net9.0 are all supported |
@@ -78,11 +83,12 @@ MSTest v3 introduces these breaking changes from v1/v2. Address only the ones re
## Response Guidelines
- **Always identify the current version first**: Before recommending any migration steps, explicitly state the current MSTest version detected in the project (e.g., "Your project uses MSTest v2 (2.2.10)" or "This is an MSTest v1 project using QualityTools assembly references"). This grounds the migration advice and confirms you've read the project files.
- **Require project evidence**: Do not assume v1/v2 from the wording alone. Read project or central package files and classify the source as QualityTools/v1, NuGet 1.x, or NuGet 2.x. If the project is already on v3+, stop and route to the appropriate skill.
- **Require project evidence, but gather it yourself**: Do not assume v1/v2 from the wording alone -- read the project or central package files and classify the source as QualityTools/v1, NuGet 1.x, or NuGet 2.x. Gather that evidence from the working directory rather than asking the user for it. If the project is already on v3+ with no v1/v2 leftovers, stop and route to the appropriate skill.
- **Preserve the test platform**: Keep VSTest or MTP unchanged during the framework upgrade unless the user separately requests a runner migration.
- **Execute full migrations**: When the user asks you to migrate or upgrade the project, edit the files, build, and run tests. Do not stop after listing breaking changes. Advice-only responses are appropriate only when the user asks what to expect.
- **Focused fix requests** (user has specific compilation errors after upgrading): Address only the relevant breaking change from the table above. Show a concise before/after fix. Do not walk through the full migration workflow.
- **DataRow fix requests**: Compare every supplied `DataRow` with its method signature. Mismatches can build with only `MSTEST0014` and fail during test execution. Preserve the method contract and normally fix the literal (`1L` -> `1` for `int`), then run the affected tests.
- **DataRow fix requests**: Compare every supplied `DataRow` with its method signature. Mismatches can build with only `MSTEST0014` and fail during test execution. Preserve the method contract and normally fix the literal (`1L` -> `1` for `int`), then run the affected tests. **Change only the rows that are actually wrong.** Argument count is not itself a defect on 3.0.3+, so leave a long row alone unless the compiler rejects it.
- **Change nothing on suspicion -- confirm the error first**: When you believe a construct is unsupported, build and read the actual diagnostic before editing it. If it compiles, this version supports it and it needs no change. Rewriting valid code to dodge a limit the project is not subject to is a defect, not caution.
- **Specific feature migration** (user asks about one aspect like .testsettings, DataRow, or assertions): Address only that feature, but handle every active setting or affected usage in the supplied files. For `.testsettings`, put all MSTest settings under one `<MSTest>` element, map requested deployment, per-test timeout, data collector, and other active configuration, and do not add a session-wide timeout. Do not walk through unrelated breaking changes.
- **"What to expect" questions** (user asks about breaking changes before upgrading): First state the concrete package update needed to reach v3, then summarize every category in the Breaking Changes Summary, marking which ones directly apply to the visible project. Keep each item to one line and do not expand into release-note history.
- **Full migration requests** (user wants complete migration): Follow the complete workflow below.
@@ -100,11 +106,14 @@ Both paths converge at Step 3 -- the same v3 packages and breaking changes apply
### Step 1: Assess the project
1. In one discovery pass, batch-read project and central configuration files, search for affected APIs/settings, and identify which MSTest version is currently in use:
1. Locate the project first: glob the working directory for `*.csproj`, `*.sln`,
`*.slnx`, `Directory.Build.props`, `Directory.Packages.props`, and
`*.testsettings`. Do this before asking the user anything.
2. In one discovery pass, batch-read project and central configuration files, search for affected APIs/settings, and identify which MSTest version is currently in use:
- **Assembly reference**: Look for `Microsoft.VisualStudio.QualityTools.UnitTestFramework` in project references -> MSTest v1
- **NuGet packages**: Check `MSTest.TestFramework` and `MSTest.TestAdapter` package versions -> v1 if 1.x, v2 if 2.x
2. Check whether the target framework is dropped in v3 (see Step 4).
3. Run the existing test command. Record discovered, passed, failed, and skipped counts as the parity baseline.
3. Check whether the target framework is dropped in v3 (see Step 4).
4. Run the existing test command. Record discovered, passed, failed, and skipped counts as the parity baseline.
### Step 2: Remove v1 assembly references (if applicable)
@@ -179,6 +188,11 @@ Assert.AreSame(expected, actual); -> Assert.AreSame<MyType>(expected, ac
Preserve method parameter types unless independently wrong. `dotnet build` may
succeed with `MSTEST0014`; run the test to prove each row binds and executes.
**Rows with more than 16 arguments** -- leave them alone unless the compiler
actually emits `CS1729`. The cap existed only in 3.0.1/3.0.2 (removed in 3.0.3),
so wrapping extras in an `object[]`, casting to `object`, or splitting the method
just rewrites a correct test.
**Timeout behavior** -- unified across .NET Core and .NET Framework. Verify `[Timeout]` values still work.
### Step 6: Replace .testsettings with .runsettings
@@ -218,5 +232,7 @@ After v3 migration, use `migrate-mstest-v3-to-v4` for MSTest v4.
| Pitfall | Solution |
|---------|----------|
| Replying with "which project?" when the workspace already holds one | Glob for `*.csproj`/`*.sln`/`*.slnx` and read what is there |
| Rewriting a `DataRow` with more than 16 arguments | Valid on 3.0.3+, which is every current 3.x. Only 3.0.1/3.0.2 ever rejected it |
| Non-MSTest.Sdk VSTest project missing `Microsoft.NET.Test.Sdk` | Add the package reference for VSTest discovery |
| MSTest.Sdk tests not found by `vstest.console` | Set `<UseVSTest>true</UseVSTest>`; MSTest.Sdk then supplies `Microsoft.NET.Test.Sdk` |
@@ -41,7 +41,7 @@ Migrate a test project from MSTest v3 to MSTest v4. The outcome is a project usi
| Input | Required | Description |
|-------|----------|-------------|
| Project or solution path | Yes | The `.csproj`, `.sln`, or `.slnx` entry point containing MSTest test projects |
| Project or solution path | No | The `.csproj`, `.sln`, or `.slnx` entry point containing MSTest test projects. **Discover it yourself** by globbing the working directory; ask only when nothing is found or the choice is genuinely ambiguous |
| Build command | No | How to build (e.g., `dotnet build`, a repo build script). Auto-detect if not provided |
| Test command | No | How to run tests (e.g., `dotnet test`). Auto-detect if not provided |
@@ -47,7 +47,7 @@ Migrate a .NET test solution from VSTest to Microsoft.Testing.Platform (MTP). Th
| Input | Required | Description |
|-------|----------|-------------|
| Project or solution path | Yes | The `.csproj`, `.sln`, or `.slnx` entry point containing test projects |
| Project or solution path | No | The `.csproj`, `.sln`, or `.slnx` entry point containing test projects. **Discover it yourself** by globbing the working directory; ask only when nothing is found or the choice is genuinely ambiguous |
| Test framework | No | MSTest, NUnit, xUnit.net v2, or xUnit.net v3. Auto-detected from package references |
| .NET SDK version | No | Determines `dotnet test` integration mode. Auto-detected via `dotnet --version` |
| CI/CD pipeline files | No | Paths to pipeline definitions that invoke `vstest.console` or `dotnet test` |