mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Add MTP hot reload coverage evaluations (#1137)
* Add MTP hot reload coverage evals Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten hot reload activation grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload evaluation guards Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Cover hot reload setup validation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Catch unsupported config suggestions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep Test Explorer eval dormant Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload eval matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Catch inline hot reload install commands Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Narrow hot reload negative matcher exemptions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refine hot reload negation handling Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Anchor hot reload package matcher Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten hot reload eval matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload eval guidance matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Correlate hot reload activation and relaunch Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Handle inline hot reload warning commands Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Narrow MTP hot reload skill routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix hot reload routing description Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Exclude negated console host questions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep hot reload menu within validator limit Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten unsupported configuration matcher Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Require command-shaped hot reload relaunch Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tighten hot reload dormancy matcher Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reject hot reload project mutation guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix hot reload warning matcher exemptions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden hot reload warning and env matchers Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Accept inline hot reload environment commands Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Allow env configuration in hot reload grader Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reject unsupported Test Explorer config usage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Scope hot reload env matcher exemptions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Amaury Levé
@@ -1,13 +1,14 @@
|
||||
---
|
||||
name: mtp-hot-reload
|
||||
description: >
|
||||
Set up or recover MTP hot reload for a long-lived edit/re-run loop. Use for
|
||||
"hot reload tests", "dotnet run or dotnet test for hot reload", a host that
|
||||
keeps running, unsupported/rude edits, or a watch-based VSTest fallback.
|
||||
Covers setup, run/watch, restarts, filters, and the VSTest no-mutation
|
||||
fallback. Never mutate VSTest for hot reload. For one-time runs, exact
|
||||
commands, filter errors, TRX/dumps, or merely a failing test, use run-tests.
|
||||
Excludes writing/debugging tests, CI, and Test Explorer hot reload.
|
||||
Set up or recover MTP hot reload for a long-lived console-host edit/re-run
|
||||
loop in a Microsoft Testing Platform project. Use for explicit MTP console
|
||||
requests such as "enable hot reload", "dotnet run to rerun on edit", or
|
||||
unsupported/rude edits. Covers setup, run/watch, restarts, filters, and the
|
||||
VSTest no-mutation fallback. For one-time runs, exact commands, filter
|
||||
errors, TRX/dumps, or merely a failing test, use run-tests. Do not use when
|
||||
Test Explorer or an IDE should rerun tests without an MTP console host.
|
||||
Excludes editor integration, CI, and writing/debugging tests.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
@@ -26,7 +27,8 @@ edits and automatically reruns tests.
|
||||
|
||||
- User needs to write new tests from scratch (use general coding assistance)
|
||||
- User needs to diagnose why a test is failing (use diagnostic skills)
|
||||
- User wants Visual Studio Test Explorer hot reload (different feature, built into VS)
|
||||
- User asks about Visual Studio or VS Code Test Explorer hot reload or an IDE-integrated rerun experience (different feature, not MTP console-host hot reload)
|
||||
- User asks whether the editor can auto-rerun affected tests after source edits without using an MTP console host
|
||||
- User wants one normal run without rebuilding (use `run-tests`)
|
||||
- User needs CI/CD pipeline configuration
|
||||
|
||||
|
||||
@@ -293,10 +293,80 @@ stimuli:
|
||||
pattern: DOTNET_WATCH_RESTART_ON_RUDE_EDIT|Ctrl\+R|restart prompt
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: (?im)(?:^\s*(?:[-*]\s*)?`?(?:[$>]\s*)?dotnet\s+add(?:\s+\S+)?\s+package\s+\S*HotReload\b|`[^`\r\n]*\bdotnet\s+add(?:\s+\S+)?\s+package\s+\S*HotReload\b[^`\r\n]*`)
|
||||
pattern: '(?im)^(?![^\r\n]*\b(?:(?:don''t|do not)(?:\s+need\s+to)?\s+(?:(?:re)?install|add(?:\s+\S+)?\s+package|run\s+`?dotnet\s+add)|no need(?: to)?\s+(?:(?:re)?install|add(?:\s+\S+)?\s+package|run\s+`?dotnet\s+add)|not needed|not necessary)\b)[^\r\n]*\bdotnet\s+add(?:\s+\S+)?\s+package\s+\S*HotReload\b'
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Correctly distinguishes changes the active session can apply from changes that require a rebuild
|
||||
- Gives a safe recovery sequence that preserves the same persistent workflow
|
||||
- Offers a valid watch-managed restart fallback for future unsupported edits
|
||||
- Preserves existing setup and does not reinstall present dependencies
|
||||
|
||||
- name: Enable a configured host that does not react to edits
|
||||
prompt: |
|
||||
My test project already references the hot-reload extension. I started it
|
||||
with `dotnet run --project TestProject.csproj`, and the console stays
|
||||
open, but test or production-code edits never cause another run. I have
|
||||
not configured any environment variables. What is missing, and how should
|
||||
I relaunch the same long-lived loop? Do not modify the project.
|
||||
environment:
|
||||
files:
|
||||
- src: ./fixtures/enable-hot-reload-when-package-already-installed/TestProject.csproj
|
||||
dest: TestProject.csproj
|
||||
- src: ./fixtures/enable-hot-reload-when-package-already-installed/EmailValidatorTests.cs
|
||||
dest: EmailValidatorTests.cs
|
||||
- src: ./fixtures/enable-hot-reload-when-package-already-installed/global.json
|
||||
dest: global.json
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: '(?is)(?:(?m)^[ \t]*(?:[-*]\s*)?(?:[$>]\s*)?`?dotnet\s+run\b.{0,500}TESTINGPLATFORM_HOTRELOAD_ENABLED\s*=\s*["'']?1["'']?(?!\d)|(?m)^[ \t]*(?:[-*]\s*)?(?:[$>]\s*)?(?:`?(?:\$env:|export\s+|set\s+)?TESTINGPLATFORM_HOTRELOAD_ENABLED\s*=\s*["'']?1["'']?).{0,500}^[ \t]*(?:[-*]\s*)?(?:[$>]\s*)?`?dotnet\s+run\b|(?m)^[ \t]*(?:[-*]\s*)?(?:[$>]\s*)?`?TESTINGPLATFORM_HOTRELOAD_ENABLED\s*=\s*["'']?1["'']?\s+`?dotnet\s+run\b)'
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: '(?im)^(?![^\r\n]*\b(?:(?:don''t|do not)(?:\s+need\s+to)?\s+(?:(?:re)?install|add(?:\s+\S+)?\s+package|run\s+`?dotnet\s+add)|no need(?: to)?\s+(?:(?:re)?install|add(?:\s+\S+)?\s+package|run\s+`?dotnet\s+add)|not needed|not necessary)\b)[^\r\n]*(?:\bdotnet\s+add(?:\s+\S+)?\s+package\s+\S*HotReload\b)'
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: '(?im)^(?![^\r\n]*\b(?:(?:don''t|do not)(?:\s+need\s+to)?|(?:can''t|cannot)|(?:no need(?: to)?|not needed|not necessary)|unsupported\s+(?:to\s+)?)(?:create|update|add|configure|edit|set up|use)\b)[^\r\n]*(?:create|update|add|configure|edit|set up|use)\b[^\r\n]*(?:launchSettings\.json|Microsoft\.Testing\.Extensions\.HotReload|\.vscode[\\/](?:settings|launch)\.json)'
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Identifies the missing activation setting as the reason edits are not picked up
|
||||
- Gives a valid way to set the activation setting before restarting the existing persistent host
|
||||
- Preserves the already installed dependency and the requested no-mutation workflow
|
||||
constraints:
|
||||
reject_tools:
|
||||
- bash
|
||||
- edit
|
||||
- create
|
||||
|
||||
- name: Decline Test Explorer hot reload integration
|
||||
prompt: |
|
||||
I only want Visual Studio Code Test Explorer to automatically rerun the
|
||||
affected tests after every source edit. Is that an editor feature, and
|
||||
can it be enabled without using or configuring an MTP console host?
|
||||
Do not modify the project.
|
||||
expect_activation: false
|
||||
environment:
|
||||
files:
|
||||
- src: ./fixtures/enable-hot-reload-when-package-already-installed/TestProject.csproj
|
||||
dest: TestProject.csproj
|
||||
- src: ./fixtures/enable-hot-reload-when-package-already-installed/EmailValidatorTests.cs
|
||||
dest: EmailValidatorTests.cs
|
||||
- src: ./fixtures/enable-hot-reload-when-package-already-installed/global.json
|
||||
dest: global.json
|
||||
graders:
|
||||
- type: output-matches
|
||||
config:
|
||||
pattern: (?is)(?=.*Test Explorer.{0,120}(not supported|does not support|no support))(?=.*(?:console(?:-based| mode| host)|supported console))
|
||||
- type: output-not-matches
|
||||
config:
|
||||
pattern: '(?im)^(?![^\r\n]*\b(?:(?:don''t|do not|can''t|cannot)\s+(?:create|update|add|configure|edit|set up|use)|(?:no need(?: to)?|not needed|not necessary)\s+(?:create|update|add|configure|edit|set up|use)|unsupported\s+(?:to\s+)?(?:create|update|add|configure|edit|set up|use))\b)[^\r\n]*(?:create|update|add|configure|edit|set up|use)\b[^\r\n]*(?:launchSettings\.json|TESTINGPLATFORM_HOTRELOAD_ENABLED|Microsoft\.Testing\.Extensions\.HotReload|\.vscode[\\/](?:settings|launch)\.json)'
|
||||
- type: prompt
|
||||
rubric:
|
||||
- Clearly explains that the requested Test Explorer integration is unavailable
|
||||
- The hot-reload setup workflow stayed dormant rather than treating the editor-integration request as setup work
|
||||
- Does not claim to have configured unsupported editor integration or alter project setup
|
||||
- Directs the user to the supported console-based persistent test-host workflow
|
||||
constraints:
|
||||
reject_tools:
|
||||
- bash
|
||||
- edit
|
||||
- create
|
||||
|
||||
Reference in New Issue
Block a user