mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Improve test-gap evals and MSTest migration activation (#1107)
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget The per-plugin aggregate description cap had been raised 15,000 -> 20,000 -> 22,000 under the belief that 15K was 'a local repo policy, NOT a documented Copilot constraint'. That belief was wrong: the GitHub Copilot CLI renders the model-facing <available_skills> menu under a hard 15,000- char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with their full <description> only until the budget is exhausted; every skill past the cut-off collapses to a bare name with no description and can no longer be reliably model-activated. Raising the validator cap merely masked this silent menu truncation — e.g. dotnet-test's run-tests and test-* skills stopped activating in plugin eval runs because they fell into the name-only overflow. Changes: - SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the comment rewritten to document the real Copilot CLI budget (and correct the prior 'not a documented constraint' claim). - CheckCommand aggregate now excludes skills marked 'disable-model-invocation: true' — the CLI drops those from the menu, so they do not consume the budget. This makes the cap satisfiable by hiding reference / agent-orchestrated primitives rather than only by trimming. - InvestigatingResults.md: document plugin-arm-only non-activation caused by skill-menu budget overflow, and how to fix it. Note: dotnet-test currently exceeds 15K and must be slimmed below it (via disable-model-invocation on reference/primitive skills plus description trims) before this cap can go green repo-wide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill-validator: use source-generated regex for disable-model-invocation check Address review: replace Regex.IsMatch(pattern-string) with a [GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup), matching FrontmatterParser's style. Runs once per skill during checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-gap-analysis: cover validation safeguards Add focused evaluation scenarios for proportional scope, complete outcome inventory, mutation restoration, demonstrated recommendations, and canonical verifier usage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: harden mutation verifier eval Anchor verifier paths to the script directory and require the eval response to report the supplied canonical verifier result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: distinguish verifier failures Require Python 3, preflight the test runner, build mutants explicitly, and treat only the MTP test-failure exit code as a killed mutation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: tighten eval evidence Challenge proportional scope with an unrelated fixture and require affirmative canonical-verifier success wording. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: name MTP failure exit code Document the Microsoft.Testing.Platform test-failure convention used by the mutation verifier. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: require affirmative verifier result Constrain the output grader to a direct successful result for verify_mutations.sh so negative mutation summaries cannot pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * Improve MSTest migration skill activation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316 * test-gap-analysis: cover proportional scope validation Name the validation outcome directly and prove the focused review excludes an unrelated AccessControl fixture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: classify killed mutants by test evidence Capture MTP output and require an xUnit execution summary with failed tests, avoiding runner-specific exit-code assumptions while preserving infrastructure diagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: preserve mutant build diagnostics Capture MSBuild output and print it when mutant compilation fails so verifier infrastructure failures remain actionable in CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316
This commit is contained in:
Amaury Levé
@@ -1,15 +1,16 @@
|
||||
---
|
||||
name: migrate-mstest-v1v2-to-v3
|
||||
description: >
|
||||
Upgrade, compare, or repair MSTest v1/v2 projects during migration to v3.
|
||||
Use for QualityTools assembly references; MSTest.TestFramework/TestAdapter
|
||||
1.x-2.x; choosing the MSTest metapackage or MSTest.Sdk; "what breaking
|
||||
changes should I expect?"; CS0411/CS1503 after a v3 package bump; DataRow
|
||||
type mismatch or MSTEST0014; .testsettings/LegacySettings to .runsettings;
|
||||
timeout changes; and dropped v3 TFMs such as net5.0. Also use when packages
|
||||
already say 3.x but v1/v2 source or settings remain, and when asked whether
|
||||
v1 and v2 migration steps differ. Preserve VSTest/MTP. Do not use for clean
|
||||
v3 projects, v3-to-v4, another test framework, or runner-only migration.
|
||||
Use this skill before answering or editing whenever an MSTest v1/v2 project
|
||||
is being upgraded or repaired for v3. Triggers include QualityTools assembly
|
||||
references; MSTest.TestFramework/TestAdapter 1.x-2.x; "upgrade to MSTest
|
||||
v3"; comparing v1 and v2 migration paths; choosing MSTest or MSTest.Sdk;
|
||||
CS0411/CS1503 after a v3 package bump; DataRow type mismatch,
|
||||
MSTEST0014, or "Test data doesn't match method parameters";
|
||||
.testsettings/LegacySettings to .runsettings; timeout changes; and net5.0 or
|
||||
other dropped v3 TFMs. Still use it when packages say 3.x but migration
|
||||
source errors or legacy settings remain. Preserve VSTest/MTP. Do not use for
|
||||
a clean v3 project, v3-to-v4, framework conversion, or runner-only migration.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
@@ -17,6 +18,14 @@ license: MIT
|
||||
|
||||
Migrate a test project from MSTest v1 (assembly references) or MSTest v2 (NuGet 1.x-2.x) to MSTest v3. MSTest v3 is **not binary compatible** with v1/v2 -- libraries compiled against v1/v2 must be recompiled.
|
||||
|
||||
## First Action
|
||||
|
||||
Inspect the supplied workspace and classify v1, v2, partially migrated v3, or
|
||||
already-complete v3 before answering. Do not search the web or answer from
|
||||
memory first. For an edit request, continue through the requested source
|
||||
changes and validation; for an advice request, answer directly after the
|
||||
classification.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Project references `Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll` (MSTest v1)
|
||||
@@ -70,7 +79,7 @@ This overrides all steps below.
|
||||
- The skill directory contains guidance, not the staged project. Search the current working directory, open the literal paths returned by the search, and retry with another available reader/editor if one tool rejects a valid path.
|
||||
- Never ask for a path while a glob or directory search can discover it. Ask only after an exhaustive current-workspace search finds no project, or multiple projects make the target genuinely ambiguous.
|
||||
- Classify the requested deliverable, not isolated verbs: "make the edits", "update this project", or "then build and run" means execute; "what do I need to change?", "what should I expect?", "are the steps the same?", or "show me" means answer, even if the prompt also says upgrade or migrate.
|
||||
- After changing files, name the detected MSTest version and runner, the files changed, the exact source/settings decisions, and the clean test result. Do not claim VSTest preservation, a build, or passing tests without evidence from the project.
|
||||
- After changing files, name the detected MSTest version and runner, every file changed, and the exact repaired calls/settings (for example, list each assertion changed to `AreEqual<object>`, `AreNotEqual<object>`, or `AreSame<object>`). Report the clean test counts. Do not claim VSTest preservation, a build, or passing tests without evidence from the project.
|
||||
|
||||
## Breaking Changes Summary
|
||||
|
||||
|
||||
@@ -1,15 +1,16 @@
|
||||
---
|
||||
name: migrate-mstest-v3-to-v4
|
||||
description: >
|
||||
Upgrade MSTest 3.x projects to v4 or fix v4 migration failures. Use for
|
||||
"MSTest v4 breaking changes", tests that stop compiling or behave differently
|
||||
after 3.x-to-4.x, CS0507/CS0103/CS1061/CS1615, ExecuteAsync or CallerInfo in
|
||||
custom TestMethodAttribute, DisplayName, ClassCleanupBehavior, ContainsKey,
|
||||
ThrowsExactly/ExpectedException, IsInstanceOfType out parameters, TestTimeout,
|
||||
ManagedType, net6/net7 compatibility, TestCase.Id history, TestName in
|
||||
ClassInitialize, TreatDiscoveryWarningsAsErrors, and MSTest.Sdk/MTP or
|
||||
vstest.console discovery changes. Do not use for v1/v2-to-v3 leftovers,
|
||||
framework conversion, runner-only migration, or a general .NET upgrade.
|
||||
Use this skill before answering, planning, or editing any MSTest 3.x-to-4.x
|
||||
upgrade or post-upgrade failure. Triggers include "MSTest v4 breaking
|
||||
changes"; CS0507/CS0103/CS1061/CS1615; ExecuteAsync, CallerInfo, DisplayName,
|
||||
or custom TestMethodAttribute; ClassCleanupBehavior; ContainsKey;
|
||||
ThrowsExactly or ExpectedException; IsInstanceOfType out parameters;
|
||||
TestTimeout.Infinite; ManagedType; net6/net7 compatibility; TestCase.Id
|
||||
history; TestName in ClassInitialize; TreatDiscoveryWarningsAsErrors;
|
||||
discovery errors after a clean build; and MSTest.Sdk/MTP or vstest.console
|
||||
discovery changes. Do not use for v1/v2-to-v3 leftovers, framework
|
||||
conversion, runner-only migration, or a general .NET upgrade.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
@@ -17,6 +18,15 @@ license: MIT
|
||||
|
||||
Migrate a test project from MSTest v3 to MSTest v4. The outcome is a project using MSTest v4 that builds cleanly, passes tests, and accounts for every source-incompatible and behavioral change. MSTest v4 is **not binary compatible** with MSTest v3 -- any library compiled against v3 must be recompiled against v4.
|
||||
|
||||
## First Action
|
||||
|
||||
Inspect the supplied project and source before searching the web or answering
|
||||
from memory. Classify the request as a focused source fix, runtime behavior
|
||||
change, CI discovery issue, compatibility question, or full migration, then
|
||||
follow the matching row below. A clean compile does not exclude this skill:
|
||||
discovery failures, `TestContext` lifecycle exceptions, and test-history
|
||||
changes are runtime migration failures.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Upgrading `MSTest.TestFramework`, `MSTest.TestAdapter`, or `MSTest` metapackage from 3.x to 4.x
|
||||
@@ -49,13 +59,14 @@ Migrate a test project from MSTest v3 to MSTest v4. The outcome is a project usi
|
||||
| User asks "what should I expect?", "how do I fix these changes?", for compatibility advice, or for a plan | Answer directly from the actual project state even when source is visible. Keep a single-symptom answer focused; include only adjacent risks that change the decision. |
|
||||
| Unsupported TFM in a full migration | Update the TFM first, then update MSTest packages, then fix source breaks. Do not bury this order in a release-note inventory. |
|
||||
| Custom `TestMethodAttribute` subclass | Treat `ExecuteAsync`, CallerInfo propagation, display-name handling, and the subclass's retry/result semantics as one coupled migration. Fix the actual class, not a placeholder example. |
|
||||
| `MSTest.Sdk` v4 source/API errors (`ManagedType`, `TestTimeout`, `Contains`) | Give the exact source replacements, then add the adjacent runner warning: MTP mode no longer supplies `Microsoft.NET.Test.Sdk`; add it only if VSTest discovery is still required. |
|
||||
| `MSTest.Sdk` v4 plus `vstest.console` | This is a v4 change: MTP mode no longer brings `Microsoft.NET.Test.Sdk`. Keep MTP and add that package for transitional VSTest discovery, opt into `UseVSTest`, or migrate CI to `dotnet test`; state which runner the choice preserves. |
|
||||
|
||||
## Response Guidelines
|
||||
|
||||
- **Always identify the current version first**: Before recommending any migration steps, explicitly state the current MSTest version detected in the project (e.g., "Your project uses MSTest v3 (3.8.0)"). This confirms you've read the project files and grounds the migration advice.
|
||||
- **Resolve, do not assume, the target version**: When the user asks for "latest", query the project's configured package source and select the latest stable MSTest v4 version available at execution time. Never copy the example version from this skill into the result without checking it. Keep all MSTest packages on the same resolved version.
|
||||
- **Focused fix requests** (user has specific compilation errors after upgrading): Address only the relevant breaking changes from Step 3. Make edits only when the requested deliverable is a source change; "how do I fix these?" remains an answer request. **Always provide concrete fixed code** using the user's actual types and method names. If the fixture still references v3, do not claim that a green v3 build verifies v4 compatibility; either update packages when requested or state the verification boundary. For custom `TestMethodAttribute` subclasses, show the full fixed class including CallerInfo propagation to the base constructor. Mention any related analyzer that could have caught this earlier (e.g., MSTEST0006 for ExpectedException). Do not walk through the entire migration workflow.
|
||||
- **Focused fix requests** (user has specific compilation errors after upgrading): Address only the relevant breaking changes from Step 3. Make edits only when the requested deliverable is a source change; "how do I fix these?" remains an answer request. **Always provide concrete fixed code** using the user's actual types and method names. If the fixture still references v3, do not claim that a green v3 build verifies v4 compatibility; either update packages when requested or state the verification boundary. For custom `TestMethodAttribute` subclasses, show the full fixed class including CallerInfo propagation to the base constructor. Mention any related analyzer that could have caught this earlier (e.g., MSTEST0006 for ExpectedException). When the project uses `MSTest.Sdk`, also state that v4 MTP mode no longer supplies `Microsoft.NET.Test.Sdk` and whether that affects the visible runner. Do not walk through the entire migration workflow.
|
||||
- **"What to expect" questions** (user asks about breaking changes before upgrading): Present ALL major breaking changes from the Step 3 quick-lookup table -- not just the ones visible in the current code. For each, provide a one-line fix summary. Also mention key behavioral changes from Step 4 (especially TestCase.Id history impact and TreatDiscoveryWarningsAsErrors default). If project code is available, highlight which changes apply directly.
|
||||
- **Full migration requests** (user wants complete migration): Follow the complete workflow below.
|
||||
- **Behavioral/runtime symptom reports** (user describes test execution differences without build errors): Match described symptoms to the behavioral changes table in Step 4. Provide targeted, symptom-specific advice. Mention other behavioral changes the user should watch for. Do not walk through source breaking changes unless the user also has build errors.
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
---
|
||||
name: migrate-vstest-to-mtp
|
||||
description: >
|
||||
Migrate .NET test projects and CI from VSTest to Microsoft.Testing.Platform
|
||||
(MTP), or fix MTP migration behavior. Use for "switch from VSTest", MTP
|
||||
runner enablement for MSTest/NUnit/xUnit, OutputType=Exe conditions in
|
||||
Directory.Build.props, EnableMSTestRunner/EnableNUnitRunner/
|
||||
UseMicrosoftTestingPlatformRunner, YTest.MTP.XUnit2, .NET 10 global.json
|
||||
test.runner, --filter-class/--filter-method/--filter-trait/--filter-query translation,
|
||||
VSTest@3 replacement, TRX/coverage/dump extensions, and exit code 8 or zero
|
||||
tests. Supports xUnit v3 MTP filter work even during a v2-to-v3 upgrade. Do
|
||||
not use for test-framework conversion, xUnit v2-to-v3 API migration, TFM,
|
||||
UWP, or WinUI migration.
|
||||
Use this skill before answering, planning, or editing whenever .NET tests or
|
||||
CI are switching from VSTest to Microsoft.Testing.Platform (MTP), or an MTP
|
||||
migration behaves differently. Triggers include "switch from VSTest";
|
||||
MSTest/NUnit/xUnit MTP enablement; OutputType=Exe only for test projects in
|
||||
Directory.Build.props; EnableMSTestRunner, EnableNUnitRunner,
|
||||
UseMicrosoftTestingPlatformRunner, or YTest.MTP.XUnit2; .NET 10 global.json
|
||||
test.runner and TestingPlatformDotnetTestSupport; translating VSTest
|
||||
filters, logger, coverage, blame, or dump arguments; replacing VSTest@3;
|
||||
and exit code 8 or zero tests. Also use for xUnit v3 MTP filters during a
|
||||
v2-to-v3 upgrade. Do not use for framework conversion, TFM, UWP, or WinUI.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
@@ -18,6 +18,16 @@ license: MIT
|
||||
|
||||
Migrate a .NET test solution from VSTest to Microsoft.Testing.Platform (MTP). The outcome is a solution where all test projects run on MTP, `dotnet test` works correctly, and CI/CD pipelines are updated.
|
||||
|
||||
## First Action
|
||||
|
||||
Inspect the supplied project, `Directory.Build.props`, `global.json`, and CI
|
||||
files before searching the web or answering from memory. Resolve the framework
|
||||
and SDK mode first: .NET 9 and earlier use the compatibility property plus the
|
||||
`--` separator; .NET 10 native MTP uses `global.json`, removes that property,
|
||||
and passes MTP arguments without the separator. For central properties, never
|
||||
condition on `IsTestProject` in `Directory.Build.props`; use a property already
|
||||
available there, such as `MSBuildProjectName`.
|
||||
|
||||
> **Important**: Do not mix VSTest-based and MTP-based .NET test projects in the same solution or run configuration -- this is an unsupported scenario.
|
||||
|
||||
## When to Use
|
||||
|
||||
@@ -241,13 +241,67 @@ stimuli:
|
||||
- For this advisory request, accurate source/assertion mapping after one green baseline is sufficient;
|
||||
do not reward applying mutations over an equally accurate static assessment
|
||||
|
||||
- name: Verify every meaningful outcome in a focused discount review
|
||||
prompt: |
|
||||
Review only DiscountRules.Apply under fixtures/close-partial-suite and its
|
||||
existing tests. Verify which caller-visible discount and validation
|
||||
outcomes could change without a test failing. Keep the review focused on
|
||||
this public method, account for every meaningful high-risk outcome, and
|
||||
leave all source and test files unchanged when you finish. Recommend tests
|
||||
only for gaps you can demonstrate. The workspace also contains an unrelated
|
||||
AccessControl fixture; do not expand the requested scope to analyze it.
|
||||
environment:
|
||||
files:
|
||||
- src: fixtures/close-partial-suite
|
||||
dest: fixtures/close-partial-suite
|
||||
- src: fixtures/logic-gaps
|
||||
dest: unrelated/logic-gaps
|
||||
commands:
|
||||
- mkdir -p .eval-baseline && cp fixtures/close-partial-suite/src/DiscountRules.cs .eval-baseline/DiscountRules.cs && cp fixtures/close-partial-suite/tests/DiscountRulesTests.cs .eval-baseline/DiscountRulesTests.cs
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (whitespace|IsNullOrWhiteSpace)
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (case.insensitive|lowercase|ToUpperInvariant)
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (FLAT5|floor|below five|below 5)
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (negative|ArgumentOutOfRangeException)
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (unknown|ArgumentException)
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: (AccessControl|AccessChecker|CanWrite|ElevateRole|Guest access)
|
||||
- type: run-command
|
||||
config:
|
||||
command: sh -c "diff -u .eval-baseline/DiscountRules.cs fixtures/close-partial-suite/src/DiscountRules.cs && diff -u .eval-baseline/DiscountRulesTests.cs fixtures/close-partial-suite/tests/DiscountRulesTests.cs"
|
||||
expected_exit_code: 0
|
||||
timeout: 1m
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Scope stayed proportional to the request by analyzing only DiscountRules.Apply and excluding the unrelated
|
||||
AccessControl fixture
|
||||
- Accounted for every high-risk public outcome in scope, including whitespace handling, case normalization,
|
||||
the FLAT5 floor, negative-subtotal rejection, and unknown-code rejection
|
||||
- Verified each reported survivor against the existing suite rather than stopping after the first gap
|
||||
- Restored every temporary production-code change and left the existing source and tests unchanged
|
||||
- Recommended focused tests only for demonstrated gaps and did not recommend redundant coverage for null
|
||||
codes or ordinary uppercase SAVE10 behavior
|
||||
|
||||
- name: Verify and close only the surviving gaps in a partial suite
|
||||
prompt: |
|
||||
DiscountRules under fixtures/close-partial-suite/ already has a passing
|
||||
xUnit suite. Find the meaningful behaviors those tests would fail to catch,
|
||||
verify the gaps, and add focused tests that close them. Leave production
|
||||
code and DiscountRulesTests.cs byte-for-byte unchanged. Do not add tests
|
||||
for behavior the existing suite already protects.
|
||||
for behavior the existing suite already protects. Run the supplied
|
||||
verify_mutations.sh after adding tests and report its result; do not replace
|
||||
it with an ad-hoc mutation check or a broad green test run.
|
||||
environment:
|
||||
files:
|
||||
- src: fixtures/close-partial-suite
|
||||
@@ -257,7 +311,7 @@ stimuli:
|
||||
graders:
|
||||
- type: run-command
|
||||
config:
|
||||
command: sh -c "dotnet test fixtures/close-partial-suite/tests/DiscountRules.Tests.csproj"
|
||||
command: sh -c "dotnet run --project fixtures/close-partial-suite/tests/DiscountRules.Tests.csproj"
|
||||
expected_exit_code: 0
|
||||
timeout: 10m
|
||||
- type: run-command
|
||||
@@ -275,6 +329,12 @@ stimuli:
|
||||
command: sh -c "test $(grep -RhE '\\[(Fact|Theory)\\]' fixtures/close-partial-suite/tests --include='*.cs' | wc -l) -le 8"
|
||||
expected_exit_code: 0
|
||||
timeout: 1m
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (verify_mutations\.sh|canonical mutation verifier)
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: '(?m)^[^\r\n]*`?verify_mutations\.sh`?(?: command)? (passed|succeeded|completed successfully)\.?$'
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Empirically verified candidate survivors against the original narrow suite before adding tests
|
||||
@@ -282,6 +342,7 @@ stimuli:
|
||||
- Left production and the existing test file byte-for-byte unchanged
|
||||
- Re-applied the original mutations after implementation and confirmed the added tests kill them
|
||||
- Did not duplicate the existing null-code or ordinary uppercase SAVE10 behavior
|
||||
- Explicitly reported the successful supplied verifier result instead of substituting an ad-hoc proxy
|
||||
- Judge completeness by the five verified behaviors, not raw test count; extra zero-subtotal or redundant happy-path cases are not an advantage
|
||||
|
||||
- name: Decline request to write new tests from scratch
|
||||
|
||||
+36
-4
@@ -1,23 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
source_file="fixtures/close-partial-suite/src/DiscountRules.cs"
|
||||
test_project="fixtures/close-partial-suite/tests/DiscountRules.Tests.csproj"
|
||||
script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source_file="$script_dir/src/DiscountRules.cs"
|
||||
test_project="$script_dir/tests/DiscountRules.Tests.csproj"
|
||||
backup="$(mktemp)"
|
||||
cp "$source_file" "$backup"
|
||||
|
||||
if command -v python3 >/dev/null 2>&1 && python3 -c 'import sys; raise SystemExit(sys.version_info < (3,))' >/dev/null 2>&1; then
|
||||
python_command="python3"
|
||||
elif command -v python >/dev/null 2>&1 && python -c 'import sys; raise SystemExit(sys.version_info < (3,))' >/dev/null 2>&1; then
|
||||
python_command="python"
|
||||
else
|
||||
echo "Python 3 is required to run the mutation verifier." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
restore() {
|
||||
cp "$backup" "$source_file"
|
||||
}
|
||||
trap 'restore; rm -f "$backup"' EXIT
|
||||
|
||||
if ! baseline_output="$(dotnet run --project "$test_project" 2>&1)"; then
|
||||
printf '%s\n' "$baseline_output" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
expect_killed() {
|
||||
local old="$1"
|
||||
local new="$2"
|
||||
local label="$3"
|
||||
|
||||
restore
|
||||
python3 - "$source_file" "$old" "$new" <<'PY'
|
||||
"$python_command" - "$source_file" "$old" "$new" <<'PY'
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
@@ -30,9 +45,26 @@ if old not in content:
|
||||
path.write_text(content.replace(old, new, 1), encoding="utf-8")
|
||||
PY
|
||||
|
||||
if dotnet test "$test_project" --nologo -v:q >/dev/null 2>&1; then
|
||||
local build_output
|
||||
if ! build_output="$(dotnet build "$test_project" --nologo -v:q 2>&1)"; then
|
||||
printf '%s\n' "$build_output" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
local test_output
|
||||
if test_output="$(dotnet run --project "$test_project" --no-build 2>&1)"; then
|
||||
echo "Mutation survived: $label" >&2
|
||||
exit 1
|
||||
else
|
||||
local test_exit_code=$?
|
||||
if grep -q "=== TEST EXECUTION SUMMARY ===" <<<"$test_output" &&
|
||||
grep -Eq "Failed: [1-9][0-9]*" <<<"$test_output"; then
|
||||
return
|
||||
fi
|
||||
|
||||
echo "Mutation test infrastructure failed for $label (exit code $test_exit_code)." >&2
|
||||
printf '%s\n' "$test_output" >&2
|
||||
exit "$test_exit_code"
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user