Clear the eval-quality gate's test-skill findings

The gate reported three classes of debt against the dotnet-test and
dotnet-test-migration plugins. All three are addressed here; the four ERRORS it
also reports are dotnet-maui allowlist lines and are untouched.

Underpowered evals (5). Below five trials the pass gate's sign test cannot reach
p <= 0.05 at any effect size, so these five evals could never return a verdict.
Each is now at or above the floor and its allowlist line is deleted in the same
change, as the ledger's shrink-only rule requires:

  - coverage-analysis 3 -> 5: adds a refactoring-safety question (the "is this
    safe to change?" use case named in the skill's Purpose but never exercised)
    and a branch-vs-line coverage question. Both reuse the existing
    partial-coverage fixture.
  - find-untested-sources 4 -> 5: adds a mixed C#/TypeScript repository, which
    is the only case that exercises the documented engine choice - polyglot
    tree-sitter rather than the C#-only Roslyn engine. Composed from the two
    existing fixtures.
  - generate-testability-wrappers 4 -> 5: adds the ambient-context path (Step 5)
    for a project with no DI container, where AsyncLocal<T> and scoped disposal
    are the distinguishing content.
  - grade-tests 4 -> 5: adds a C# case with the production code present. Every
    prior C# scenario hides it, so "Unverified" was never tested as a negative,
    and the D band and the swallowed-exception F were never graded at all. New
    production-available fixture.
  - code-testing-agent 3 -> 6 via defaults.runs=2. Scenarios are preferred over
    runs, but each of these drives a full generate-build-test pipeline (npm ci
    plus two Vitest runs, pip install plus pytest, a dotnet test build) under a
    60m budget, which is the documented case for buying trials with runs.

Orphaned fixtures (5). v3-sealed-timeout, mtp-mstest-sdk9, mtp-mstest-sdk10,
mtp-mstest-hotreload-installed and vstest-mstest are all superseded first-
generation copies: their per-scenario successors differ only in whitespace, a
dropped rollForward, or a package version. Both evals are already well above the
floor, so wiring them up would add no power. Deleted.

Skills with no eval (2 of 4). platform-detection and filter-syntax carry real
checkable rules that nothing measured, and several are counterintuitive enough
that a baseline is likely to get them wrong - global.json test.runner outranking
TestingPlatformDotnetTestSupport on .NET 10+, Microsoft.NET.Test.Sdk not being a
VSTest signal, MTP properties living in Directory.Build.props, xUnit v3 dropping
VSTest --filter while MSTest on MTP keeps it. Both get a 5-scenario eval with
small fixtures and no build step. code-testing-extensions and
test-analysis-extensions are left flagged on purpose: their bodies are tables of
paths to extension files, so a head-to-head eval would score path recall rather
than user value. The content those files hold is already exercised through
code-testing-agent's three-language pipeline.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740

eng/eval-quality/underpowered-allowlist.txt

@@ -63,8 +63,6 @@ tests/dotnet-msbuild/item-management/eval.yaml
 tests/dotnet-msbuild/property-patterns/eval.yaml
 tests/dotnet-msbuild/target-authoring/eval.yaml
 tests/dotnet-template-engine/template-comparison/eval.yaml
-tests/dotnet-test/code-testing-agent/eval.yaml
-tests/dotnet-test/coverage-analysis/eval.yaml
 tests/dotnet-upgrade/migrate-nullable-references/eval.yaml
 tests/dotnet/setup-local-sdk/eval.yaml
 tests/dotnet11/system-text-json-net11/eval.yaml
@@ -79,6 +77,3 @@ tests/dotnet-maui/maui-shell-navigation/eval.yaml
 tests/dotnet-maui/maui-theming/eval.yaml
 tests/dotnet-msbuild/msbuild-antipatterns/eval.yaml
 tests/dotnet-template-engine/template-smart-defaults/eval.yaml
-tests/dotnet-test/find-untested-sources/eval.yaml
-tests/dotnet-test/generate-testability-wrappers/eval.yaml
-tests/dotnet-test/grade-tests/eval.yaml

tests/dotnet-test-migration/migrate-mstest-v3-to-v4/fixtures/v3-sealed-timeout/PerformanceTests.cs

@@ -1,41 +0,0 @@
-using System;
-using System.Diagnostics;
-using Microsoft.VisualStudio.TestTools.UnitTesting;
-
-namespace MyApp.Tests;
-
-public sealed class TimedTestMethodAttribute : TestMethodAttribute
-{
-    public override TestResult[] Execute(ITestMethod testMethod)
-    {
-        var sw = Stopwatch.StartNew();
-        var results = base.Execute(testMethod);
-        sw.Stop();
-        Console.WriteLine($"{testMethod.TestMethodName} took {sw.ElapsedMilliseconds}ms");
-        return results;
-    }
-}
-
-[TestClass]
-public class PerformanceTests
-{
-    [TimedTestMethod]
-    [Timeout(TestTimeout.Infinite)]
-    public void HeavyComputation_Completes()
-    {
-        Assert.IsTrue(true);
-    }
-
-    [TimedTestMethod]
-    public void QuickCheck_Succeeds()
-    {
-        Assert.AreEqual(42, 42);
-    }
-
-    [TestMethod]
-    [Timeout(TestTimeout.Infinite)]
-    public void StressTest_DoesNotTimeout()
-    {
-        Assert.IsTrue(true);
-    }
-}

tests/dotnet-test/code-testing-agent/eval.yaml

@@ -3,6 +3,15 @@ description: Evaluates the dotnet-test/code-testing-agent skill
 type: capability
 config:
   timeout: 60m
+# Trials = scenarios x runs, and the pass gate needs at least 5 (see
+# eng/eval-quality/README.md). Adding scenarios is normally preferred because it
+# widens what the skill is measured on, but every scenario here drives the full
+# multi-file generate-build-test pipeline end to end — `npm ci` plus two Vitest
+# runs, `pip install` plus pytest, and a `dotnet test` build — under a 60m
+# budget. A fourth and fifth scenario of that shape costs more than it informs,
+# so this eval buys its trials with runs instead.
+defaults:
+  runs: 2
 stimuli:
   - name: Generate Vitest tests for the shopping-cart library (TypeScript polyglot)
     prompt: |

tests/dotnet-test/coverage-analysis/eval.yaml

@@ -101,3 +101,72 @@ stimuli:
       - Quantifies the impact — covering CalculateGpa would raise overall coverage significantly
       - Uses CRAP scores to distinguish the risky gap (CalculateGpa) from trivial ones
       - Provides specific test recommendations for the uncovered method
+  - name: Refactoring safety assessment from coverage data
+    prompt: I need to refactor StudentService before adding a feature. Which methods are safe to change and which
+      ones would I be refactoring blind? Coverage data is in TestResults/coverage.cobertura.xml.
+    environment:
+      files:
+        - src: fixtures/shared/ContosoUniversity.sln
+          dest: ContosoUniversity.sln
+        - src: fixtures/shared/ContosoUniversity/ContosoUniversity.csproj
+          dest: ContosoUniversity/ContosoUniversity.csproj
+        - src: fixtures/shared/ContosoUniversity/StudentService.cs
+          dest: ContosoUniversity/StudentService.cs
+        - src: fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
+          dest: ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
+        - src: fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs
+          dest: ContosoUniversity.UnitTests/StudentServiceTests.cs
+        - src: fixtures/partial-coverage/coverage.cobertura.xml
+          dest: TestResults/coverage.cobertura.xml
+    graders:
+      - type: output-matches
+        config:
+          pattern: CRAP|crap
+      - type: output-matches
+        config:
+          pattern: CalculateGpa|Calculate_?Gpa
+      - type: output-matches
+        config:
+          pattern: FindById|Find_?ById
+      - type: prompt
+    rubric:
+      - Ranks the methods by refactoring risk using CRAP rather than raw coverage percentage alone
+      - Names CalculateGpa as the dangerous one — high complexity combined with low coverage means a regression would not be caught
+      - Names the well-covered low-complexity methods (FindById, and Enroll or Search) as comparatively safe to change
+      - Explains why complexity and coverage together determine refactoring safety, not either number on its own
+      - Recommends adding tests to the risky method before refactoring it, rather than refactoring first
+  - name: Distinguish branch coverage from line coverage
+    prompt: My report says line coverage is 47% but I do not know which decision paths are actually untested. Which
+      branches are missing tests, and does the line number overstate how well tested this is? Coverage data is in
+      TestResults/coverage.cobertura.xml.
+    environment:
+      files:
+        - src: fixtures/shared/ContosoUniversity.sln
+          dest: ContosoUniversity.sln
+        - src: fixtures/shared/ContosoUniversity/ContosoUniversity.csproj
+          dest: ContosoUniversity/ContosoUniversity.csproj
+        - src: fixtures/shared/ContosoUniversity/StudentService.cs
+          dest: ContosoUniversity/StudentService.cs
+        - src: fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
+          dest: ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
+        - src: fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs
+          dest: ContosoUniversity.UnitTests/StudentServiceTests.cs
+        - src: fixtures/partial-coverage/coverage.cobertura.xml
+          dest: TestResults/coverage.cobertura.xml
+    graders:
+      - type: output-matches
+        config:
+          pattern: (?i)branch
+      - type: output-matches
+        config:
+          pattern: CalculateGpa|Calculate_?Gpa
+      - type: output-matches
+        config:
+          pattern: Enroll
+      - type: prompt
+    rubric:
+      - Reports branch coverage as a separate figure from line coverage instead of collapsing the two
+      - Notes that branch coverage is the lower of the two here, so the line figure is the more flattering number
+      - Points at the partially covered conditions — Enroll is executed but has decisions taken only one way
+      - Identifies CalculateGpa as having whole decision paths never exercised
+      - Recommends tests that target the missing branch outcomes specifically, not just more line hits

tests/dotnet-test/filter-syntax/eval.yaml

@@ -0,0 +1,153 @@
+name: filter-syntax
+description: Evaluates the dotnet-test/filter-syntax skill
+type: capability
+config:
+  timeout: 4m
+stimuli:
+  - name: xUnit v3 on MTP rejects the VSTest filter expression
+    prompt: |
+      Give me the exact command line to run only the tests in the
+      MyApp.Tests.LoginTests class in this project. Do not build or run
+      anything — I just want the command.
+    environment:
+      files:
+        - src: ./fixtures/xunit-v3-mtp-sdk9/global.json
+          dest: global.json
+        - src: ./fixtures/xunit-v3-mtp-sdk9/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: --filter-class
+      - type: output-matches
+        config:
+          pattern: dotnet test\s+--\s
+      - type: output-not-matches
+        config:
+          pattern: (?im)^\s*[$>]?\s*dotnet test\b[^\n]*--filter\s+"?FullyQualifiedName
+      - type: prompt
+    rubric:
+      - Used xUnit v3's --filter-class flag rather than a VSTest-style --filter expression
+      - Stated that xUnit v3 on MTP does not support the VSTest --filter syntax
+      - Passed the argument after a bare -- because the pinned SDK is 9.0.100
+      - Did not offer FullyQualifiedName, Name, or DisplayName as filter properties for this project
+
+  - name: MSTest on MTP keeps the VSTest filter syntax
+    prompt: |
+      Give me the exact command line to run only the tests categorised as
+      Integration in this project. Do not build or run anything — I just want
+      the command.
+    environment:
+      files:
+        - src: ./fixtures/mstest-mtp-sdk10/global.json
+          dest: global.json
+        - src: ./fixtures/mstest-mtp-sdk10/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: TestCategory=Integration
+      - type: output-matches
+        config:
+          pattern: --filter
+      - type: output-not-matches
+        config:
+          pattern: (?im)^\s*[$>]?\s*dotnet test\s+--\s+--filter
+      - type: prompt
+    rubric:
+      - Used the same --filter "TestCategory=Integration" expression that VSTest uses
+      - Explained that MSTest on MTP shares the VSTest filter properties, operators and combinators, so nothing needs translating
+      - Passed the flag directly without a bare -- separator, because the pinned SDK is 10.0.100
+      - Did not reach for xUnit-style --filter-trait or TUnit-style --treenode-filter
+
+  - name: TUnit uses treenode-filter path syntax
+    prompt: |
+      Give me the exact command line to run only the tests marked with the
+      Smoke category in this project. Do not build or run anything — I just
+      want the command.
+    environment:
+      files:
+        - src: ./fixtures/tunit-mtp/global.json
+          dest: global.json
+        - src: ./fixtures/tunit-mtp/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: --treenode-filter
+      - type: output-matches
+        config:
+          pattern: \[Category=Smoke\]
+      - type: output-not-matches
+        config:
+          pattern: (?im)^\s*[$>]?\s*dotnet (?:test|run)\b[^\n]*--filter\s+"?TestCategory
+      - type: prompt
+    rubric:
+      - Used --treenode-filter rather than --filter
+      - Produced a path-shaped filter with a segment per assembly, namespace, class and test name, using * for the segments that are not being constrained
+      - Expressed the category as a bracketed property match such as [Category=Smoke]
+      - Recognised TUnit as the framework and MTP as the platform before choosing the syntax
+
+  - name: Translate VSTest filters for an xUnit v3 migration
+    prompt: |
+      We are moving this suite off VSTest and our CI currently runs these two
+      filters:
+
+        --filter "FullyQualifiedName~CheckoutTests"
+        --filter "Category=Smoke"
+
+      What are the equivalents for this project, and does anything change in
+      matching behaviour? Do not build or run anything.
+    environment:
+      files:
+        - src: ./fixtures/xunit-v3-mtp-sdk9/global.json
+          dest: global.json
+        - src: ./fixtures/xunit-v3-mtp-sdk9/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: --filter-class
+      - type: output-matches
+        config:
+          pattern: \*CheckoutTests\*
+      - type: output-matches
+        config:
+          pattern: --filter-trait
+      - type: prompt
+    rubric:
+      - Translated FullyQualifiedName~CheckoutTests to --filter-class with wildcards on both sides, because the substring behaviour of ~ is not implied by --filter-class
+      - Translated the Category=Smoke trait filter to --filter-trait "Category=Smoke"
+      - Stated that xUnit v3 on MTP drops VSTest --filter support, which is why the CI filters have to be rewritten at all
+      - Mentioned --filter-query as the option for expressions too complex for the individual flags
+      - Did not claim the existing filters keep working unchanged
+
+  - name: Combine class and category filters on VSTest
+    prompt: |
+      Give me the exact command line to run only the tests in the
+      MyApp.Tests.CheckoutTests class that are also in the Unit category, and
+      tell me what the shorthand `dotnet test --filter "Checkout"` would match
+      by comparison. Do not build or run anything.
+    environment:
+      files:
+        - src: ./fixtures/mstest-vstest/global.json
+          dest: global.json
+        - src: ./fixtures/mstest-vstest/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: ClassName=MyApp\.Tests\.CheckoutTests
+      - type: output-matches
+        config:
+          pattern: TestCategory=Unit
+      - type: output-matches
+        config:
+          pattern: FullyQualifiedName~
+      - type: prompt
+    rubric:
+      - Combined the two conditions with & rather than issuing two separate commands
+      - Used ClassName= for the exact class match and TestCategory= for the category
+      - Explained that a bare value with no operator is treated as FullyQualifiedName~<value>, i.e. a substring match
+      - Noted that the shorthand is therefore broader — it matches anything whose fully qualified name contains Checkout, not just that class
+      - Kept the plain VSTest invocation without a -- separator, since this project is on VSTest

tests/dotnet-test/filter-syntax/fixtures/mstest-vstest/TestProject.csproj

@@ -3,13 +3,11 @@
   <PropertyGroup>
     <TargetFramework>net9.0</TargetFramework>
     <IsPackable>false</IsPackable>
-    <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
   </PropertyGroup>
 
   <ItemGroup>
-    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
     <PackageReference Include="MSTest" Version="3.8.0" />
-    <PackageReference Include="Microsoft.Testing.Extensions.HotReload" Version="1.5.3" />
   </ItemGroup>
 
 </Project>

tests/dotnet-test/filter-syntax/fixtures/mstest-vstest/global.json

@@ -0,0 +1,5 @@
+{
+  "sdk": {
+    "version": "9.0.100"
+  }
+}

tests/dotnet-test/filter-syntax/fixtures/tunit-mtp/TestProject.csproj

@@ -0,0 +1,13 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <IsPackable>false</IsPackable>
+    <OutputType>Exe</OutputType>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="TUnit" Version="0.6.0" />
+  </ItemGroup>
+
+</Project>

tests/dotnet-test/filter-syntax/fixtures/tunit-mtp/global.json

@@ -0,0 +1,5 @@
+{
+  "sdk": {
+    "version": "9.0.100"
+  }
+}

tests/dotnet-test/filter-syntax/fixtures/xunit-v3-mtp-sdk9/TestProject.csproj

@@ -0,0 +1,14 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <IsPackable>false</IsPackable>
+    <OutputType>Exe</OutputType>
+    <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="xunit.v3" Version="1.0.0" />
+  </ItemGroup>
+
+</Project>

tests/dotnet-test/filter-syntax/fixtures/xunit-v3-mtp-sdk9/global.json

@@ -0,0 +1,5 @@
+{
+  "sdk": {
+    "version": "9.0.100"
+  }
+}

tests/dotnet-test/find-untested-sources/eval.yaml

@@ -156,3 +156,45 @@ stimuli:
       - Suggested a concrete test path following the repository's existing tests/Store.Tests/ convention
       - Did not run dotnet build or dotnet test — this is static pairing analysis
       - Did not claim a line-coverage percentage; source-to-test pairing is not coverage
+
+  - name: Choose the polyglot engine for a mixed C# and TypeScript repository
+    prompt: >
+      This repository has a C# service under src/Store and a TypeScript cart
+      under src/cart, with tests in tests/. Which sources have no tests? Use one
+      analysis pass over the whole repository — do not build, install packages,
+      or run tests.
+    environment:
+      files:
+        - src: fixtures/pairing-repo
+          dest: .
+        - src: fixtures/typescript-pairing/src/cart/pricing.ts
+          dest: src/cart/pricing.ts
+        - src: fixtures/typescript-pairing/src/cart/tax.ts
+          dest: src/cart/tax.ts
+        - src: fixtures/typescript-pairing/tests/cart/pricing.test.ts
+          dest: tests/cart/pricing.test.ts
+    graders:
+      - type: output-matches
+        config:
+          pattern: OrderProcessor
+      - type: output-matches
+        config:
+          pattern: tax\.ts
+      - type: output-not-matches
+        config:
+          pattern: (?i)pricing\.ts.{0,100}(untested|unpaired|no test)
+      - type: output-not-matches
+        config:
+          pattern: (?im)^\s*(?:[$>]\s*)?(?:dotnet\s+(?:build|test)|npm|pnpm|yarn)\b
+      - type: output-matches
+        config:
+          pattern: (?is)(?=.*\bstatic\b)(?=.*\b(?:pairing|heuristic)\b)(?=.*\b(?:line|branch)\b)(?=.*\bcoverage\b)(?=.*\b(?:not|isn't|doesn't|unknown|unverified|cannot)\b)
+      - type: prompt
+    rubric:
+      - Recognised the repository is not C#-only and therefore used the polyglot tree-sitter engine rather than the C#-only Roslyn engine
+      - Covered both languages in a single analysis pass instead of reporting only the C# half or only the TypeScript half
+      - Reported OrderProcessor.cs and tax.ts as the unpaired sources
+      - Did not report CustomerService.cs or pricing.ts as untested, and named the tests that cover them
+      - Suggested test paths that follow each language's existing convention (tests/Store.Tests/ for C#, a .test.ts beside the other cart tests)
+      - Did not run dotnet build, dotnet test, or a package manager — this is static pairing analysis
+      - Labelled the result a static pairing heuristic rather than line or branch coverage

tests/dotnet-test/generate-testability-wrappers/eval.yaml

@@ -96,3 +96,33 @@ stimuli:
       - Recognized that IFileSystem is already in use and no wrapper generation is needed
       - Did not generate a redundant wrapper interface
       - May have suggested other improvements but did not create duplicate abstractions
+
+  - name: Make time testable without a DI container
+    prompt: >
+      ReportGenerator calls DateTime.UtcNow and I want to pin the clock in unit
+      tests. This is a plain console app with no dependency injection container
+      and I do not want to add one. What are my options, and what does the code
+      look like?
+    environment:
+      files:
+        - src: ./fixtures/needs-wrappers
+          dest: NeedsWrappers
+    graders:
+      - type: output-matches
+        config:
+          pattern: AsyncLocal
+      - type: output-matches
+        config:
+          pattern: (IDisposable|Dispose)
+      - type: output-not-matches
+        config:
+          pattern: ThreadStatic
+      - type: exit-success
+      - type: prompt
+    rubric:
+      - Offers the ambient context pattern rather than insisting on adding a DI container
+      - Uses AsyncLocal<T> for the override so parallel and async tests do not interfere
+      - Scopes the override with a disposable so one test cannot leak a pinned clock into another
+      - Falls back to the real clock (TimeProvider.System or DateTime.UtcNow) when no override is set
+      - States the trade-offs of ambient context versus injection instead of presenting it as strictly better
+      - Does not use [ThreadStatic], which breaks across await points

tests/dotnet-test/grade-tests/eval.yaml

@@ -189,3 +189,58 @@ stimuli:
       reject_tools:
         - edit
         - create
+  - name: Grade C# tests against available production code
+    prompt: |
+      Please grade each of the following test methods individually for test
+      quality and produce a compact per-test table (one row per test) plus a
+      short summary. They live in `Banking.Tests/BankAccountTests.cs` and the
+      code under test is in `Banking/BankAccount.cs`. Apply this repository's
+      established per-test grading policy and do not modify any files.
+
+      Test methods to grade:
+        - Banking.Tests.BankAccountTests.Withdraw_AmountExceedsBalance_ThrowsInsufficientFunds
+        - Banking.Tests.BankAccountTests.Withdraw_SufficientFunds_LeavesBalanceUnchangedFromItself
+        - Banking.Tests.BankAccountTests.ApplyInterest_RateDependsOnTier_UpdatesBalance
+        - Banking.Tests.BankAccountTests.Deposit_NonPositiveAmount_HandledGracefully
+    environment:
+      files:
+        - src: fixtures/production-available/Banking/Banking.csproj
+          dest: Banking/Banking.csproj
+        - src: fixtures/production-available/Banking/BankAccount.cs
+          dest: Banking/BankAccount.cs
+        - src: fixtures/production-available/Banking.Tests/Banking.Tests.csproj
+          dest: Banking.Tests/Banking.Tests.csproj
+        - src: fixtures/production-available/Banking.Tests/BankAccountTests.cs
+          dest: Banking.Tests/BankAccountTests.cs
+    graders:
+      - type: output-matches
+        config:
+          pattern: \|\s*Test\s*\|\s*Grade\s*\|\s*Band\s*\|\s*Notes\s*\|
+      - type: output-matches
+        config:
+          pattern: (Withdraw_AmountExceedsBalance_ThrowsInsufficientFunds.*\|\s*A\s*\|)
+      - type: output-matches
+        config:
+          pattern: (Withdraw_SufficientFunds_LeavesBalanceUnchangedFromItself.*\|\s*D\s*\|)
+      - type: output-matches
+        config:
+          pattern: (ApplyInterest_RateDependsOnTier_UpdatesBalance.*\|\s*D\s*\|)
+      - type: output-matches
+        config:
+          pattern: (Deposit_NonPositiveAmount_HandledGracefully.*\|\s*F\s*\|)
+      - type: output-not-matches
+        config:
+          pattern: (?i)Production-dependent behavior coverage:\s*(?:\*{1,2})?Unverified
+      - type: exit-success
+      - type: prompt
+    rubric:
+      - Graded `Withdraw_AmountExceedsBalance_ThrowsInsufficientFunds` as A — a specific exception type plus its documented message is a complete assertion
+      - Graded `Withdraw_SufficientFunds_LeavesBalanceUnchangedFromItself` as D — `Assert.AreEqual(account.Balance, account.Balance)` is self-referential and verifies nothing about the withdrawal
+      - Graded `ApplyInterest_RateDependsOnTier_UpdatesBalance` as D — conditional logic drives the assertions, so only one branch ever runs and the test mirrors the implementation
+      - Graded `Deposit_NonPositiveAmount_HandledGracefully` as F — the empty catch swallows the exception and the test asserts nothing
+      - Did not report production-dependent behavior as Unverified, because BankAccount.cs is present in the workspace
+      - Used the stable `Test | Grade | Band | Notes` table schema with score bands rather than point scores
+    constraints:
+      reject_tools:
+        - edit
+        - create

tests/dotnet-test/grade-tests/fixtures/production-available/Banking.Tests/BankAccountTests.cs

@@ -0,0 +1,66 @@
+using Banking;
+using Microsoft.VisualStudio.TestTools.UnitTesting;
+
+namespace Banking.Tests;
+
+[TestClass]
+public class BankAccountTests
+{
+    [TestMethod]
+    public void Withdraw_AmountExceedsBalance_ThrowsInsufficientFunds()
+    {
+        // Arrange
+        var account = new BankAccount(100m);
+
+        // Act
+        var ex = Assert.ThrowsException<InvalidOperationException>(() => account.Withdraw(500m));
+
+        // Assert
+        Assert.AreEqual("Insufficient funds.", ex.Message);
+    }
+
+    [TestMethod]
+    public void Withdraw_SufficientFunds_LeavesBalanceUnchangedFromItself()
+    {
+        // Arrange
+        var account = new BankAccount(100m);
+
+        // Act
+        account.Withdraw(40m);
+
+        // Assert
+        Assert.AreEqual(account.Balance, account.Balance);
+    }
+
+    [TestMethod]
+    public void ApplyInterest_RateDependsOnTier_UpdatesBalance()
+    {
+        var account = new BankAccount(1000m);
+        var rate = account.Tier == AccountTier.Premium ? 0.05m : 0.01m;
+
+        account.ApplyInterest(rate);
+
+        if (account.Tier == AccountTier.Premium)
+        {
+            Assert.AreEqual(1050m, account.Balance);
+        }
+        else
+        {
+            Assert.AreEqual(1010m, account.Balance);
+        }
+    }
+
+    [TestMethod]
+    public void Deposit_NonPositiveAmount_HandledGracefully()
+    {
+        var account = new BankAccount(100m);
+
+        try
+        {
+            account.Deposit(-5m);
+        }
+        catch
+        {
+        }
+    }
+}

tests/dotnet-test/grade-tests/fixtures/production-available/Banking.Tests/Banking.Tests.csproj

@@ -2,13 +2,21 @@
 
   <PropertyGroup>
     <TargetFramework>net8.0</TargetFramework>
+    <Nullable>enable</Nullable>
     <IsPackable>false</IsPackable>
   </PropertyGroup>
 
   <ItemGroup>
     <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
-    <PackageReference Include="MSTest.TestAdapter" Version="3.6.3" />
-    <PackageReference Include="MSTest.TestFramework" Version="3.6.3" />
+    <PackageReference Include="MSTest.TestFramework" Version="3.6.0" />
+    <PackageReference Include="MSTest.TestAdapter" Version="3.6.0" />
+  </ItemGroup>
+
+  <!-- NOTE: unlike the production-unavailable fixture, the code under test
+       (Banking/BankAccount.cs) IS present here, so behavioral concerns are
+       observable and must not be reported as Unverified. -->
+  <ItemGroup>
+    <ProjectReference Include="../Banking/Banking.csproj" />
   </ItemGroup>
 
 </Project>

tests/dotnet-test/grade-tests/fixtures/production-available/Banking/BankAccount.cs

@@ -0,0 +1,57 @@
+namespace Banking;
+
+public enum AccountTier
+{
+    Standard,
+    Premium
+}
+
+public class BankAccount
+{
+    public BankAccount(decimal openingBalance, AccountTier tier = AccountTier.Standard)
+    {
+        Balance = openingBalance;
+        Tier = tier;
+    }
+
+    public decimal Balance { get; private set; }
+
+    public AccountTier Tier { get; }
+
+    /// <summary>
+    /// Adds <paramref name="amount"/> to the balance.
+    /// Throws <see cref="ArgumentOutOfRangeException"/> when the amount is not positive.
+    /// </summary>
+    public void Deposit(decimal amount)
+    {
+        if (amount <= 0m)
+        {
+            throw new ArgumentOutOfRangeException(nameof(amount), "Deposit amount must be positive.");
+        }
+
+        Balance += amount;
+    }
+
+    /// <summary>
+    /// Removes <paramref name="amount"/> from the balance.
+    /// Throws <see cref="InvalidOperationException"/> with the message "Insufficient funds."
+    /// when the amount exceeds the current balance.
+    /// </summary>
+    public void Withdraw(decimal amount)
+    {
+        if (amount > Balance)
+        {
+            throw new InvalidOperationException("Insufficient funds.");
+        }
+
+        Balance -= amount;
+    }
+
+    /// <summary>
+    /// Adds interest at <paramref name="rate"/>, rounded to two decimal places.
+    /// </summary>
+    public void ApplyInterest(decimal rate)
+    {
+        Balance += decimal.Round(Balance * rate, 2);
+    }
+}

tests/dotnet-test/grade-tests/fixtures/production-available/Banking/Banking.csproj

@@ -0,0 +1,8 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net8.0</TargetFramework>
+    <Nullable>enable</Nullable>
+  </PropertyGroup>
+
+</Project>

tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-hotreload-installed/EmailValidatorTests.cs

@@ -1,32 +0,0 @@
-using Microsoft.VisualStudio.TestTools.UnitTesting;
-
-namespace Contoso.Validation.Tests;
-
-[TestClass]
-public class EmailValidatorTests
-{
-    [TestMethod]
-    public void Validate_ValidEmail_ReturnsTrue()
-    {
-        Assert.IsTrue(IsValidEmail("user@example.com"));
-    }
-
-    [TestMethod]
-    public void Validate_MissingAtSign_ReturnsFalse()
-    {
-        // BUG: validator doesn't check for @ sign
-        Assert.IsFalse(IsValidEmail("userexample.com"));
-    }
-
-    [TestMethod]
-    public void Validate_EmptyString_ReturnsFalse()
-    {
-        Assert.IsFalse(IsValidEmail(""));
-    }
-
-    private static bool IsValidEmail(string email)
-    {
-        // Buggy: only checks non-empty, doesn't validate format
-        return !string.IsNullOrEmpty(email);
-    }
-}

tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-hotreload-installed/global.json

@@ -1,6 +0,0 @@
-{
-  "sdk": {
-    "version": "9.0.200",
-    "rollForward": "latestFeature"
-  }
-}

tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk10/InventoryServiceTests.cs

@@ -1,46 +0,0 @@
-using Microsoft.VisualStudio.TestTools.UnitTesting;
-
-namespace Contoso.Inventory.Tests;
-
-[TestClass]
-public class InventoryServiceTests
-{
-    [TestMethod]
-    public void AddStock_ValidQuantity_IncreasesCount()
-    {
-        var service = new InventoryService();
-        service.AddStock("SKU-001", 10);
-        Assert.AreEqual(10, service.GetStock("SKU-001"));
-    }
-
-    [TestMethod]
-    public void RemoveStock_ExceedsAvailable_ThrowsException()
-    {
-        var service = new InventoryService();
-        service.AddStock("SKU-001", 5);
-        // BUG: RemoveStock doesn't throw when quantity exceeds available stock
-        Assert.ThrowsException<InvalidOperationException>(() => service.RemoveStock("SKU-001", 10));
-    }
-
-    [TestMethod]
-    public void GetStock_UnknownSku_ReturnsZero()
-    {
-        var service = new InventoryService();
-        Assert.AreEqual(0, service.GetStock("UNKNOWN"));
-    }
-}
-
-public class InventoryService
-{
-    private readonly Dictionary<string, int> _stock = new();
-
-    public void AddStock(string sku, int quantity) =>
-        _stock[sku] = _stock.GetValueOrDefault(sku) + quantity;
-
-    public void RemoveStock(string sku, int quantity) =>
-        // Missing: check if quantity exceeds available stock
-        _stock[sku] = _stock.GetValueOrDefault(sku) - quantity;
-
-    public int GetStock(string sku) =>
-        _stock.GetValueOrDefault(sku);
-}

tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk9/PaymentProcessorTests.cs

@@ -1,34 +0,0 @@
-using Microsoft.VisualStudio.TestTools.UnitTesting;
-
-namespace Contoso.Payments.Tests;
-
-[TestClass]
-public class PaymentProcessorTests
-{
-    [TestMethod]
-    public void ProcessPayment_ValidAmount_Succeeds()
-    {
-        var result = ProcessPayment(100.00m);
-        Assert.IsTrue(result);
-    }
-
-    [TestMethod]
-    public void ProcessPayment_ZeroAmount_ThrowsException()
-    {
-        Assert.ThrowsException<ArgumentException>(() => ProcessPayment(0));
-    }
-
-    [TestMethod]
-    public void ProcessPayment_NegativeAmount_ThrowsException()
-    {
-        // BUG: This test is failing because ProcessPayment doesn't validate negative amounts
-        Assert.ThrowsException<ArgumentException>(() => ProcessPayment(-50.00m));
-    }
-
-    private static bool ProcessPayment(decimal amount)
-    {
-        if (amount == 0) throw new ArgumentException("Amount cannot be zero");
-        // Missing: validation for negative amounts
-        return true;
-    }
-}

tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk9/global.json

@@ -1,6 +0,0 @@
-{
-  "sdk": {
-    "version": "9.0.200",
-    "rollForward": "latestFeature"
-  }
-}

tests/dotnet-test/mtp-hot-reload/fixtures/vstest-mstest/ShippingCalculatorTests.cs

@@ -1,30 +0,0 @@
-using Microsoft.VisualStudio.TestTools.UnitTesting;
-
-namespace Contoso.Shipping.Tests;
-
-[TestClass]
-public class ShippingCalculatorTests
-{
-    [TestMethod]
-    public void CalculateShipping_StandardDelivery_Returns5()
-    {
-        Assert.AreEqual(5.00m, CalculateShipping("standard", 1.0));
-    }
-
-    [TestMethod]
-    public void CalculateShipping_ExpressDelivery_Returns15()
-    {
-        // BUG: express should be 15.00 but returns 10.00
-        Assert.AreEqual(15.00m, CalculateShipping("express", 1.0));
-    }
-
-    private static decimal CalculateShipping(string method, double weightKg)
-    {
-        return method switch
-        {
-            "standard" => 5.00m,
-            "express" => 10.00m, // Bug: should be 15.00m
-            _ => throw new ArgumentException("Unknown shipping method")
-        };
-    }
-}

tests/dotnet-test/platform-detection/eval.yaml

@@ -0,0 +1,155 @@
+name: platform-detection
+description: Evaluates the dotnet-test/platform-detection skill
+type: capability
+config:
+  timeout: 4m
+stimuli:
+  - name: global.json runner outranks TestingPlatformDotnetTestSupport on SDK 10
+    prompt: |
+      Which test platform and which test framework does this project use? The
+      SDK in use is the one pinned in global.json. Do not build or run anything.
+
+      Answer with these two lines first, then explain briefly:
+      Platform: <VSTest or MTP>
+      Framework: <MSTest, xUnit, NUnit, or TUnit>
+    environment:
+      files:
+        - src: ./fixtures/sdk10-globaljson-vstest-wins/global.json
+          dest: global.json
+        - src: ./fixtures/sdk10-globaljson-vstest-wins/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Platform:\s*\**\s*VSTest
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Framework:\s*\**\s*MSTest
+      - type: prompt
+    rubric:
+      - Concluded VSTest, not MTP
+      - Treated the global.json test.runner setting as authoritative because the pinned SDK is 10.0.100
+      - Explicitly noted that TestingPlatformDotnetTestSupport=true does not switch the project to MTP on .NET 10+ when global.json says VSTest
+      - Identified the framework as MSTest from the MSTest metapackage reference
+      - Did not treat the presence of Microsoft.NET.Test.Sdk as the deciding signal
+
+  - name: global.json opts a plain xUnit v3 project into MTP on SDK 10
+    prompt: |
+      Which test platform and which test framework does this project use? The
+      SDK in use is the one pinned in global.json. Do not build or run anything.
+
+      Answer with these two lines first, then explain briefly:
+      Platform: <VSTest or MTP>
+      Framework: <MSTest, xUnit, NUnit, or TUnit>
+    environment:
+      files:
+        - src: ./fixtures/sdk10-globaljson-mtp/global.json
+          dest: global.json
+        - src: ./fixtures/sdk10-globaljson-mtp/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Platform:\s*\**\s*(MTP|Microsoft\.Testing\.Platform)
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Framework:\s*\**\s*xUnit
+      - type: prompt
+    rubric:
+      - Concluded MTP from the global.json test.runner setting
+      - Identified the framework as xUnit from the xunit.v3 package reference
+      - Did not require an MSBuild property such as TestingPlatformDotnetTestSupport to be present, since global.json settles it on .NET 10+
+      - Read global.json before the project file rather than deciding from the .csproj alone
+
+  - name: MTP signal set in Directory.Build.props rather than the project file
+    prompt: |
+      Which test platform and which test framework does this project use? The
+      SDK in use is the one pinned in global.json. Do not build or run anything.
+
+      Answer with these two lines first, then explain briefly:
+      Platform: <VSTest or MTP>
+      Framework: <MSTest, xUnit, NUnit, or TUnit>
+    environment:
+      files:
+        - src: ./fixtures/sdk9-props-mtp-signal/global.json
+          dest: global.json
+        - src: ./fixtures/sdk9-props-mtp-signal/Directory.Build.props
+          dest: Directory.Build.props
+        - src: ./fixtures/sdk9-props-mtp-signal/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Platform:\s*\**\s*(MTP|Microsoft\.Testing\.Platform)
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Framework:\s*\**\s*NUnit
+      - type: output-matches
+        config:
+          pattern: Directory\.Build\.props
+      - type: prompt
+    rubric:
+      - Read Directory.Build.props, not only the .csproj, and found TestingPlatformDotnetTestSupport there
+      - Concluded MTP even though the project file carries no MTP signal of its own
+      - Identified the framework as NUnit from the NUnit and NUnit3TestAdapter references
+      - Applied the SDK 8/9 property-based rules rather than looking for a global.json test.runner section that does not exist here
+
+  - name: Microsoft.NET.Test.Sdk alongside an MTP runner property
+    prompt: |
+      Which test platform and which test framework does this project use? The
+      SDK in use is the one pinned in global.json. Do not build or run anything.
+
+      Answer with these two lines first, then explain briefly:
+      Platform: <VSTest or MTP>
+      Framework: <MSTest, xUnit, NUnit, or TUnit>
+    environment:
+      files:
+        - src: ./fixtures/sdk9-mstest-runner-with-test-sdk/global.json
+          dest: global.json
+        - src: ./fixtures/sdk9-mstest-runner-with-test-sdk/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Platform:\s*\**\s*(MTP|Microsoft\.Testing\.Platform)
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Framework:\s*\**\s*MSTest
+      - type: output-matches
+        config:
+          pattern: EnableMSTestRunner
+      - type: prompt
+    rubric:
+      - Concluded MTP on the strength of EnableMSTestRunner=true
+      - Explicitly stated that Microsoft.NET.Test.Sdk being referenced does not make this a VSTest project, because frameworks pull it in for compatibility
+      - Identified the framework as MSTest
+      - Checked the MTP signals before reasoning from Microsoft.NET.Test.Sdk
+
+  - name: TUnit project is MTP-only
+    prompt: |
+      Which test platform and which test framework does this project use, and
+      could I switch this project to the other platform? The SDK in use is the
+      one pinned in global.json. Do not build or run anything.
+
+      Answer with these two lines first, then explain briefly:
+      Platform: <VSTest or MTP>
+      Framework: <MSTest, xUnit, NUnit, or TUnit>
+    environment:
+      files:
+        - src: ./fixtures/sdk9-tunit/global.json
+          dest: global.json
+        - src: ./fixtures/sdk9-tunit/TestProject.csproj
+          dest: TestProject.csproj
+    graders:
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Platform:\s*\**\s*(MTP|Microsoft\.Testing\.Platform)
+      - type: output-matches
+        config:
+          pattern: (?im)^\s*\**\s*Framework:\s*\**\s*TUnit
+      - type: prompt
+    rubric:
+      - Concluded MTP from the TUnit package reference alone
+      - Identified the framework as TUnit
+      - Answered the follow-up correctly — TUnit runs only on MTP, so there is no VSTest option to switch to
+      - Did not require an MTP MSBuild property or a global.json runner setting to reach that conclusion

tests/dotnet-test/platform-detection/fixtures/sdk10-globaljson-mtp/TestProject.csproj

@@ -1,12 +1,12 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFramework>net8.0</TargetFramework>
+    <TargetFramework>net10.0</TargetFramework>
     <IsPackable>false</IsPackable>
   </PropertyGroup>
 
   <ItemGroup>
-    <PackageReference Include="MSTest" Version="3.8.0" />
+    <PackageReference Include="xunit.v3" Version="1.0.0" />
   </ItemGroup>
 
 </Project>

tests/dotnet-test/platform-detection/fixtures/sdk10-globaljson-mtp/global.json

@@ -0,0 +1,8 @@
+{
+  "sdk": {
+    "version": "10.0.100"
+  },
+  "test": {
+    "runner": "Microsoft.Testing.Platform"
+  }
+}

tests/dotnet-test/platform-detection/fixtures/sdk10-globaljson-vstest-wins/TestProject.csproj

@@ -1,7 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFramework>net9.0</TargetFramework>
+    <TargetFramework>net10.0</TargetFramework>
     <IsPackable>false</IsPackable>
     <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
   </PropertyGroup>

tests/dotnet-test/platform-detection/fixtures/sdk10-globaljson-vstest-wins/global.json

@@ -0,0 +1,8 @@
+{
+  "sdk": {
+    "version": "10.0.100"
+  },
+  "test": {
+    "runner": "VSTest"
+  }
+}

tests/dotnet-test/platform-detection/fixtures/sdk9-mstest-runner-with-test-sdk/TestProject.csproj

@@ -0,0 +1,15 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <IsPackable>false</IsPackable>
+    <OutputType>Exe</OutputType>
+    <EnableMSTestRunner>true</EnableMSTestRunner>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
+    <PackageReference Include="MSTest" Version="3.8.0" />
+  </ItemGroup>
+
+</Project>

tests/dotnet-test/platform-detection/fixtures/sdk9-mstest-runner-with-test-sdk/global.json

@@ -0,0 +1,5 @@
+{
+  "sdk": {
+    "version": "9.0.100"
+  }
+}

tests/dotnet-test/platform-detection/fixtures/sdk9-props-mtp-signal/Directory.Build.props

@@ -0,0 +1,8 @@
+<Project>
+
+  <!-- The MTP switch lives here, not in the .csproj. -->
+  <PropertyGroup>
+    <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
+  </PropertyGroup>
+
+</Project>

tests/dotnet-test/platform-detection/fixtures/sdk9-props-mtp-signal/TestProject.csproj

@@ -0,0 +1,14 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <IsPackable>false</IsPackable>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
+    <PackageReference Include="NUnit" Version="4.2.2" />
+    <PackageReference Include="NUnit3TestAdapter" Version="4.6.0" />
+  </ItemGroup>
+
+</Project>

tests/dotnet-test/platform-detection/fixtures/sdk9-props-mtp-signal/global.json

@@ -0,0 +1,5 @@
+{
+  "sdk": {
+    "version": "9.0.100"
+  }
+}

tests/dotnet-test/platform-detection/fixtures/sdk9-tunit/TestProject.csproj

@@ -0,0 +1,13 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net9.0</TargetFramework>
+    <IsPackable>false</IsPackable>
+    <OutputType>Exe</OutputType>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="TUnit" Version="0.6.0" />
+  </ItemGroup>
+
+</Project>

tests/dotnet-test/platform-detection/fixtures/sdk9-tunit/global.json

@@ -0,0 +1,5 @@
+{
+  "sdk": {
+    "version": "9.0.100"
+  }
+}