Correct the Assert/DataRow facts and make the fixtures reproduce them

The two eval runs on this PR compared byte-identical skill content: run 1 (f9dc25d) and run 2 (2fc8ab8) differ in tests/ and eng/ only, git diff on plugins/ between them is empty, and the run-2 artifact confirms the loaded SKILL.md lacks the fix from 6b11ad9. So 7W/2T/2L -> 4W/5T/2L is judge noise, not a regression. The v1-assembly-ref scenario scored 0.00 in BOTH runs with the same 'I need to see your project file' reply, which 6b11ad9 addresses. Investigating the remaining scenarios against a real MSTest 3.8 project turned up something worse than a scoring problem: the skill was teaching two things that are not true, and one eval fixture could not reproduce the bug it was named for. 1. Assert. The skill said the removal of Assert.AreEqual(object, object) causes 'compile error on untyped assertions'. It does not. MSTest v3 keeps AreEqual<T>(T?, T?), so two object-typed arguments infer T = object and compile untouched; verified by building the shipped fixture, which passes 3/3 tests unmodified. The break happens only where T cannot be inferred, and the real diagnostics are CS0411 and CS1503 - not the CS1501/CS0121 the earlier description claimed. Following the old text, an agent rewrites assertions that were already correct, which is the same over-application defect as the DataRow one. Table, Step 5 and the description now state the real trigger and codes, and say to fix only the call sites the compiler rejects. 2. DataRow. The skill implied compile errors. Verified: a mismatched row builds with analyzer warning MSTEST0014 and fails at run time with 'Test data doesn't match method parameters'. Widening (int -> long) still binds; narrowing does not. Stated explicitly, because a green build is exactly what misleads here. 3. Fixtures. fix-assert-.../ComparisonTests.cs compiled and passed as shipped, so its scenario could never discriminate - it scored baseline 5.00/5.00 in both runs. It now uses two unrelated interface-typed views of one instance, which genuinely fails with CS0411 on all three assertions and passes 4/4 once the <object> argument is added. It also gains two already-valid typed assertions that must be left alone; widening them still compiles, so only judgement prevents it, and two graders now check that. v2-nuget/UserServiceTests.cs and v2-complex/InventoryServiceTests.cs had the same problem: their graders demanded Assert.AreEqual<object> on assertions that never needed it, which now directly contradicts the corrected skill. Both fixtures were rebuilt the same way and verified in three states - build clean on MSTest 2.2.10 as shipped, fail with CS0411 after the v3 bump, pass (5/5 and 7/7) once migrated. Prompts for the two affected scenarios now describe the real symptoms (CS0411, and 'builds but fails at run time') instead of the invented CS1501 and 'no longer compile'. Every claim above was verified by building and running against MSTest 3.8 and 2.2.10 rather than inferred. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
2026-09-20 09:49:54 +08:00 · 2026-07-31 10:27:09 +02:00
parent 6b11ad97fe
commit 22103d260c
5 changed files with 127 additions and 60 deletions
@@ -6,17 +6,16 @@ description: >
  USE FOR: removing v1
  Microsoft.VisualStudio.QualityTools.UnitTestFramework assembly references;
  moving MSTest.TestFramework/TestAdapter 1.x-2.x to 3.x, the MSTest
-  metapackage, or MSTest.Sdk; tests that stopped compiling after a 2.x-to-3.x
-  bump -- CS1501/CS1503/CS0121 on Assert.AreEqual/AreNotEqual/AreSame object
-  overloads, DataRow strict type matching (1L vs 1), MSTEST0014, 16+ DataRow
-  arguments; converting .testsettings/LegacySettings to .runsettings
-  (DeploymentEnabled, per-test MSTest TestTimeout); v3 timeout behavior; TFMs
-  v3 dropped (net5.0, .NET Fx below 4.6.2, netstandard1.0). Applies even when
-  the project already references MSTest 3.x, if a v1/v2-era setting or error
-  remains. Keeps the current runner.
+  metapackage, or MSTest.Sdk; tests that broke after a 2.x-to-3.x bump --
+  CS0411/CS1503 on Assert.AreEqual/AreNotEqual/AreSame once the object
+  overloads became generic, and DataRow strict type matching (1L vs 1) that
+  builds with MSTEST0014 but fails at run time; .testsettings/LegacySettings
+  to .runsettings (DeploymentEnabled, per-test MSTest TestTimeout); v3 timeout
+  behavior; TFMs v3 dropped (net5.0, .NET Fx below 4.6.2, netstandard1.0).
+  Applies even when the project already references MSTest 3.x, if a v1/v2-era
+  setting or error remains. Keeps the current runner.
  DO NOT USE FOR: MSTest v4 (use migrate-mstest-v3-to-v4 next), clean v3
-  projects with no v1/v2 leftovers, converting between test frameworks, or
-  VSTest-to-MTP migration.
+  projects with no v1/v2 leftovers, other test frameworks, or VSTest-to-MTP.
 license: MIT
 ---

@@ -67,8 +66,8 @@ MSTest v3 introduces these breaking changes from v1/v2. Address only the ones re

 | Breaking Change | Impact | Fix |
 |---|---|---|
-| `Assert.AreEqual(object, object)` overload removed | Compile error on untyped assertions | Add generic type: `Assert.AreEqual<T>(expected, actual)`. Same for `AreNotEqual`, `AreSame`, `AreNotSame` |
-| `DataRow` strict type matching | Runtime/compile errors when argument types don't match parameter types exactly | Change literals to exact types: `1` for int, `1L` for long, `1.0f` for float |
+| `Assert.AreEqual(object, object)` overload removed; only `AreEqual<T>(T?, T?)` remains | **`CS0411`** (type arguments cannot be inferred) or `CS1503` -- but only where the two arguments have no common inferred type. Two `object`-typed arguments still infer `T = object` and **compile unchanged** | Add the explicit type argument on the failing call: `Assert.AreEqual<object>(expected, actual)`. Same for `AreNotEqual`, `AreSame`, `AreNotSame`. Leave assertions that already compile alone |
+| `DataRow` strict type matching | **Not a compile error.** Builds with analyzer warning `MSTEST0014` and fails at run time with "Test data doesn't match method parameters". Widening conversions (`int` -> `long`) still bind; narrowing or unrelated types (`1L` -> `int`, `1.0` -> `float`) do not | Change literals to the exact parameter type: `1` for int, `1L` for long, `1.0f` for float. Run the tests -- a green build proves nothing here |
 | `DataRow` limited to 16 arguments -- **3.0.1 and 3.0.2 only** | `CS1729` on those two versions; the limit was removed again in **3.0.3** | On 3.0.3+ (every current 3.x) a longer row is valid -- **leave it unchanged**. Do not wrap extras in an array, cast to `object`, or split the test. Only a project pinned to 3.0.1/3.0.2 needs action: update to 3.0.3+ |
 | `.testsettings` / `<LegacySettings>` no longer supported | Settings silently ignored | Delete `.testsettings`, create `.runsettings` with equivalent config |
 | Timeout behavior unified across .NET Core / Framework | Tests with `[Timeout]` may behave differently | Verify timeout values; adjust if needed |
@@ -169,20 +168,29 @@ Search the supplied files first and fix only breaking changes that are present.
 A successful build does not prove compatibility; some failures surface only as
 analyzer warnings or during test execution.

-**Assertion overloads** -- MSTest v3 removed `Assert.AreEqual(object, object)` and `Assert.AreNotEqual(object, object)`. Add explicit generic type parameters:
+**Assertion overloads** -- MSTest v3 replaced `Assert.AreEqual(object, object)` and `AreNotEqual(object, object)` with the generic `AreEqual<T>(T?, T?)`. This breaks **only** where `T` can no longer be inferred, which the compiler reports as `CS0411` (or `CS1503` for unrelated argument types):

 ```csharp
-// Before (v1/v2)                           // After (v3)
-Assert.AreEqual(expected, actual);        -> Assert.AreEqual<MyType>(expected, actual);
-Assert.AreNotEqual(a, b);                -> Assert.AreNotEqual<MyType>(a, b);
-Assert.AreSame(expected, actual);         -> Assert.AreSame<MyType>(expected, actual);
+// Breaks -- string and int have no common inferred type:
+Assert.AreEqual(referenceCode, numericId);   // CS0411
+// Fix -- name the type argument explicitly:
+Assert.AreEqual<object>(referenceCode, numericId);
 ```

-**DataRow strict type matching** -- argument types must exactly match parameter types. Implicit conversions that worked in v2 fail in v3:
+Two `object`-typed arguments still infer `T = object` and compile untouched, as do
+ordinary typed assertions like `Assert.AreEqual("A-3", order.Reference)`. **Fix only
+the call sites the compiler rejects.** Widening every assertion in the file to
+`<object>` also compiles, so nothing will flag it -- but it discards the type
+checking v3 added, which is the entire point of the change.
+
+**DataRow strict type matching** -- argument types must match parameter types
+exactly. This is **not** a compile error: the row builds (with `MSTEST0014`) and
+fails at run time with "Test data doesn't match method parameters".

 ```csharp
-// Error: 1L (long) won't convert to int parameter -> fix: use 1 (int)
-// Error: 1.0 (double) won't convert to float parameter -> fix: use 1.0f (float)
+// Fails at run time: 1L (long) does not bind to an int parameter -> use 1
+// Fails at run time: 1.0 (double) does not bind to a float parameter -> use 1.0f
+// Still binds: 1 (int) to a long parameter -- widening conversions are accepted
 ```

 Preserve method parameter types unless independently wrong. `dotnet build` may
@@ -54,10 +54,11 @@ stimuli:
      - Covers non-compilation risks such as test ID/history changes, exception unwrapping, lifecycle logging, or deployment paths
  - name: Fix Assert.AreEqual object overload errors after v3 upgrade
    prompt: |
-      I just upgraded my MSTest packages from v2 to v3 and now several of my
-      tests no longer compile. I'm seeing errors like CS1501 on Assert.AreEqual,
-      Assert.AreNotEqual, and Assert.AreSame calls. Fix the affected source,
-      then build and run the tests to verify the changes.
+      I just upgraded my MSTest packages from v2 to v3 and now my test project
+      no longer compiles. I'm seeing CS0411 "the type arguments for method
+      'Assert.AreEqual<T>(T?, T?)' cannot be inferred from the usage" on several
+      assertions. Fix the affected source, then build and run the tests to
+      verify the changes.
    environment:
      files:
        - src: ./fixtures/fix-assert-areequal-object-overload-errors-after-v3-upgrade/TestProject.csproj
@@ -77,6 +78,16 @@ stimuli:
        config:
          path: ComparisonTests.cs
          value: Assert.AreSame<object>
+      # The two already-valid typed assertions must not be widened to object --
+      # that compiles either way, so only judgement (not the compiler) prevents it.
+      - type: file-contains
+        config:
+          path: ComparisonTests.cs
+          value: Assert.AreEqual("A-3", order.Reference)
+      - type: file-contains
+        config:
+          path: ComparisonTests.cs
+          value: Assert.AreEqual(42, 42)
      - type: run-command
        config:
          command: dotnet test
@@ -85,11 +96,12 @@ stimuli:
          timeout: 3m
      - type: prompt
    rubric:
-      - Explains that MSTest v3 removed the Assert.AreEqual(object, object) overload
-      - "Recommends adding explicit generic type parameters: Assert.AreEqual<T>(expected, actual)"
-      - Applies the same generic fix pattern to every affected assertion in the supplied source
+      - Explains that MSTest v3 replaced the Assert.AreEqual(object, object) overload with the generic AreEqual<T>, so CS0411 appears where T can no longer be inferred from two unrelated argument types
+      - "Recommends adding explicit generic type parameters: Assert.AreEqual<object>(expected, actual)"
+      - Applies the same generic fix to all three failing assertions (AreEqual, AreNotEqual, AreSame)
+      - Leaves the two already-valid typed assertions unchanged rather than widening every assertion in the file to object
      - Leaves the already-upgraded MSTest v3 package and target framework unchanged
-      - Builds the project and runs all three tests successfully
+      - Builds the project and runs all four tests successfully
      - Does not suggest downgrading to MSTest v2 as a solution
  - name: Migrate from .testsettings to .runsettings
    prompt: |
@@ -140,9 +152,10 @@ stimuli:
      - Removes the obsolete .testsettings file instead of keeping both formats
  - name: Fix DataRow type mismatch errors after v3 upgrade
    prompt: |
-      I upgraded my MSTest packages from v2 to v3 and many of my data-driven tests no
-      longer compile. The errors are on the DataRow constructors — implicit type conversions
-      that worked in MSTest v2 now fail in v3. I also have a test with 17 parameters.
+      I upgraded my MSTest packages from v2 to v3 and my data-driven tests broke.
+      The project still builds, but several DataRow tests now fail at run time with
+      "Test data doesn't match method parameters" — implicit type conversions that
+      worked in MSTest v2 are rejected in v3. I also have a test with 17 parameters.
      The supplied TestProject.csproj and DataDrivenTests.cs are in the current
      working directory. Fix the project without changing the test method
      signatures, then run the tests with the current MSTest v3 version.
@@ -2,32 +2,54 @@ using Microsoft.VisualStudio.TestTools.UnitTesting;

 namespace MyApp.Tests;

+public interface IAuditable { }
+
+public interface IShippable { }
+
+public sealed class Order : IAuditable, IShippable
+{
+    public string Reference { get; init; } = "";
+}
+
 [TestClass]
 public class ComparisonTests
 {
    [TestMethod]
-    public void CompareObjects_AreEqual()
+    public void SameOrder_SeenThroughBothInterfaces_IsEqual()
    {
-        object expected = GetExpected();
-        object actual = GetActual();
-        Assert.AreEqual(expected, actual);
+        var order = new Order { Reference = "A-1" };
+        IAuditable auditable = order;
+        IShippable shippable = order;
+
+        Assert.AreEqual(auditable, shippable);
    }

    [TestMethod]
-    public void CompareObjects_AreNotEqual()
+    public void ReferenceCode_IsNotTheNumericId()
    {
-        object a = "hello";
-        object b = "world";
-        Assert.AreNotEqual(a, b);
+        string referenceCode = "42";
+        int numericId = 42;
+
+        Assert.AreNotEqual(referenceCode, numericId);
    }

    [TestMethod]
-    public void CompareReferences_AreSame()
+    public void BothInterfaceViews_AreTheSameInstance()
    {
-        object obj = new object();
-        Assert.AreSame(obj, obj);
+        var order = new Order { Reference = "A-2" };
+        IAuditable auditable = order;
+        IShippable shippable = order;
+
+        Assert.AreSame(auditable, shippable);
    }

-    private static object GetExpected() => 42;
-    private static object GetActual() => 42;
+    // These already compile against MSTest v3 and must be left alone.
+    [TestMethod]
+    public void TypedAssertions_AreAlreadyValid()
+    {
+        var order = new Order { Reference = "A-3" };
+
+        Assert.AreEqual("A-3", order.Reference);
+        Assert.AreEqual(42, 42);
+    }
 }
@@ -3,30 +3,42 @@ using Microsoft.VisualStudio.TestTools.UnitTesting;

 namespace Warehouse.Tests;

+public interface IStockItem { }
+
+public interface ICatalogEntry { }
+
+public sealed class InventoryItem : IStockItem, ICatalogEntry
+{
+    public string Sku { get; set; } = "";
+}
+
 [TestClass]
 public class InventoryServiceTests
 {
    [TestMethod]
-    public void GetItem_ReturnsCorrectItem()
+    public void GetItem_BothViewsOfSameItem_AreEqual()
    {
-        object expected = "Widget";
-        object actual = "Widget";
-        Assert.AreEqual(expected, actual);
+        var item = new InventoryItem { Sku = "Widget" };
+        IStockItem stock = item;
+        ICatalogEntry catalog = item;
+        Assert.AreEqual(stock, catalog);
    }

    [TestMethod]
-    public void GetItem_DifferentItems_AreNotEqual()
+    public void GetItem_SkuIsNotTheNumericId()
    {
-        object a = "Widget";
-        object b = "Gadget";
-        Assert.AreNotEqual(a, b);
+        string sku = "1001";
+        int numericId = 1001;
+        Assert.AreNotEqual(sku, numericId);
    }

    [TestMethod]
    public void GetItem_SameReference()
    {
-        object item = new object();
-        Assert.AreSame(item, item);
+        var item = new InventoryItem { Sku = "Gadget" };
+        IStockItem stock = item;
+        ICatalogEntry catalog = item;
+        Assert.AreSame(stock, catalog);
    }

    [TestMethod]
@@ -2,22 +2,34 @@ using Microsoft.VisualStudio.TestTools.UnitTesting;

 namespace MyApp.Tests;

+public interface IAuditable { }
+
+public interface IPrincipal { }
+
+public sealed class User : IAuditable, IPrincipal
+{
+    public string Name { get; set; } = "";
+}
+
 [TestClass]
 public class UserServiceTests
 {
    [TestMethod]
-    public void GetUser_ValidId_ReturnsUser()
+    public void GetUser_BothViewsOfSameUser_AreEqual()
    {
-        object expected = 1;
-        object actual = 1;
-        Assert.AreEqual(expected, actual);
+        var user = new User { Name = "Alice" };
+        IAuditable auditable = user;
+        IPrincipal principal = user;
+        Assert.AreEqual(auditable, principal);
    }

    [TestMethod]
    public void GetUser_SameReference_AreSame()
    {
-        object obj = new object();
-        Assert.AreSame(obj, obj);
+        var user = new User { Name = "Bob" };
+        IAuditable auditable = user;
+        IPrincipal principal = user;
+        Assert.AreSame(auditable, principal);
    }

    [TestMethod]